diff options
221 files changed, 60627 insertions, 1488 deletions
@@ -159,3 +159,9 @@ userspace/sdks/ *.bak-* *.orig *.rej + +# Demo/release videos (task: "0.9.9 release preview" lives in the repo root, +# never tracked - large binary; the file is produced locally from the host +# KVMFR capture). +/*.mp4 +/open-iov-*.mp4 diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 9bfbff5..441ea85 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,118 @@ # Open-IOV Current State -> **Snapshot: 2026-10-16 ~15:30 (integration wave: bigpath+dual merged, five fixes -> landed through `7d123630`; live r161/r162 cycles on kernel 6.8.0-147).** +> **Snapshot: 2026-10-30 ~12:20 (SESSION 153, r1013 live; HEAD `a64169db`).** +> The arc since the SESSION 143 snapshot split in two. Guest CUDA on the +> physical engine: the post-reboot stall is root-caused and fixed - the GR +> ring-autopin fallback and the CE first-submission capture each guarded +> block ownership only within their own engine class, so a channel could +> latch a live foreign FB block (`41343c63`: cross-class block claims, +> pre-latch CE reservations, own-block latch semantics) - and the fixed +> build is green repeatedly (`victory_mail` `VADD bad=0` + `STATUS OK`; +> `vecadd_loop` 100 launches PASS). The fresh-boot battery's remaining red +> class is D2H-stage (release-before-work, a ledger-vs-writer page +> divergence and a memset readback/writeback race are all proven; +> release-discipline fixes are staged but uncommitted; the opdef-deadline +> slot-complete path is the next fix). X/GL on the emulated vGPU: the wall +> that stood since the early sessions is closed - `fffae7b1` gets Xorg `:0` +> through the nvidia-push DMA sanity push and ScreenInit (`glxinfo` = +> OpenGL 4.6.0 NVIDIA 570.144, direct rendering, GZDoom launches), with the +> RC watchdog channel registered/serviced (`8898bd00`+`42c9626b`) and +> per-client VAS resolution (`38ba1e2d`: a channel's VAs resolve through +> its own VAS first). The app-side 10 s/frame waits are the client's own +> give-up manager; the `sema_xw_mirror` pair completes presents (glxgears +> latches 128-146 FPS, 200+ in the later shim runs) but produces no +> rendered output - the GL draw stream is marshalled, not executed. The +> engine now executes the real GL method stream in two forms: the C597 +> section capture + synthesized compute launch retires its QMD on real +> sm_75 SASS (`0ee574a3`..`595e023a`), and the verbatim 3D method-stream +> replay lane retires real 3D methods with no QMD (r932); the hard limit +> is that rasterised GL pixels still need the real 3D pipeline. SESSION +> 153's headline: the direct GL client's device-init gate is root-caused to +> the driver-shared memfd slot word (bit 0 set = glcore skips its +> class-0xc46f channel creation); cleared, the client allocates its +> channels (0xbeef0100/0101/0102 plus 0x902d/0xc5b5/0xc597/0x9072/0xa140/ +> 0x90f1) and `glReadPixels` returns with GL error 0 - fixed field-side by +> the kernel keeper `sema_shared_hold_pa` (`a64169db`, holds bit0 clear +> deterministically) plus host hold/toggle automation (the MakeCurrent +> handshake needs the 2<->3 transition). Current walls (ranked): (a) the +> fresh-boot Xorg DMA-init failure in r1013 (a 10 s `Failed to initialize +> DMA` -> no X server, so no client; a different wait from the member-spin +> class); (b) the client PUT-visibility wall - the direct client's channels +> never submit (its process writes no GP_PUT anywhere, the BAR0/BAR1-shadow +> premise was falsified by direct measurement, and the submitting DDX-side +> channel's USERD is guest RAM already tracked by the RAM scan); (c) the +> pixel wall - no render content reaches any ring (X-side pushes are 32-64 +> B housekeeping BEGINs; every capture/readback is black). Landmine: +> killing/exiting a direct client can trip a 4 GB `nv_alloc_system_pages` +> guest-kernel oops - never kill clients; re-cycle instead. Delivered on +> the emulated vGPU: the R18 CUDA-animation video (`70134652`; 60/60 +> frames zero faults, frames exact vs CPU math, single-page kernels only - +> the window-0 code shadow carries one 4 KB page, so kernels >4 KB stay +> hostile until it grows) and two two-VM GZDoom videos (`375d627a` tools; +> the GPU-accelerated one runs on virtio-gpu/virgl on the host iGPU, not +> the emulated vGPU, which remains the core frontier). Host clean, pstore +> empty; live: r1013 mdev `95446a6c-d5da-4e74-920b-5ac0d7ccc303`, QEMU +> 2335256, guest no Xorg/no GL client, pin/sem helpers up. Also open: the +> SESSION 143 Vulkan-client completion regression (fence timeout with the +> push re-completed) - not re-measured since r816, priority moved to the +> GL path. Commits this arc: `41343c63`, `fffae7b1`, `8898bd00`, +> `42c9626b`, `38ba1e2d`, `6f2e6193`, `0ee574a3`..`595e023a`, `70134652`, +> `375d627a`, `1fd8d50a`, `f500779b`..`a64169db` (SESSION 153 ladder). +> +> **Snapshot: 2026-10-26 (SESSION 143, post-reboot).** The r800 walk guard +> holds and the `gsp_phys_gr_min_data=0` bypass reaches the real engine: the +> guest Vulkan compute launch is now *submitted* (no oops, no degenerate-set +> refusal), the engine fetches the host-deposited QMD and the firmware reports +> a real `MMU Fault ... @ 0xdf_d8feb000, FAULT_PDE` at the guest program VA - +> the channel/context are alive, the program page is empty and unmapped. The +> util-sweep channel-visibility work (per-channel begin/skip/read=none lines) +> proves the Vulkan copy-class channels (0xbeef0101/0xbeef0102, engines 11/12) +> are registered and swept but never pushed, and that no channel carries +> program bytes or a `LAUNCH_DMA`: the code page (0x62b000 / rotating 0xc8b000) +> never fills. The frontier is therefore (1) the driver's staging-ring -> +> code-heap upload copy, which is not among the decoded pushes, and (2) the +> engine page-table mapping for the program VA. Separately, the refusal path no +> longer reaches the earlier three-black-frame baseline (`SYNTH-FAULT` vs the +> software completion/fence ordering), which must be restored before further +> observation. Docs: `docs/04-implementation/04-real-render-walls.md` sections +> 20-21. Commits: `34e611f4`, `c5510744`, `28def33e`, `0ca8ff9f`. +> +> **Snapshot: 2026-11-06 ~01:00 (the guest's REAL Vulkan pipeline now +> renders on the emulated vGPU: the first `VKFRAME seq=1..3` / +> `VKFRAME_DONE frames=3 640x480 saved=3` / `VKFRAME_IDLE ok` runs +> end-to-end — instance, device, allocation, image/view/buffer/shader/ +> descriptor/pipeline/command-buffer/fence, queue submit, fence release +> and frame readback all complete with a zero-touch service pipeline. +> The chain of fixes behind it: the push-stream parser now accepts the +> driver's real shapes (the `0x0001fff0` SET_SUBDEVICE_MASK bracket and +> zero NOP gaps — the class that silently skipped every large push), the +> fence fold (`sema_gr_ahead`) sweeps the pre-built GPFIFO continuation +> beyond GP_PUT and applies the submit's INC-5 RELEASE groups +> forward-only, the host-triggered VAS page-directory anchoring verifies +> the parked `COPY_SERVER_PDES` candidate against a content-located ring +> pair so the walk resolves every push VA before its first read, the +> COMPR+2MB VidHeap class is downgraded at the ioctl so +> `vkAllocateMemory` succeeds, the SEC2 scrubber's finish payload is +> authored by the service, the `0x9096` VidHeap enumeration and the +> `FIFO_GET_ENGINE_CONTEXT_PROPERTIES` ZCULL geometry are answered, and +> the C5C0-section -> vendor-launch translator found the real dispatch +> (a `SET_INLINE_QMD_ADDRESS_B` + 64-word run building QMD `0x4b85f00`, +> grid 40x30, plus the C5B5 copy-out). The frames are still BLACK: the +> driver delivers SASS/descriptors through its MME program + inline-to- +> memory stream (decoded in full; the payload rides the next GPFIFO +> entry as a source-view call), and the utility/copy-channel +> `LOAD_INLINE_DATA` arm that must materialize those writes is the +> current frontier (arbitration + `sema-utildec` witness landed). CUDA +> meanwhile: the best fresh-boot battery is 8 pass / 3 fail / 1 hang +> (the Vulkan work does not regress CUDA; always battery-first on a +> fresh boot); GL runs on the guest GPU (next wall = the FBO class); the +> Looking Glass frame path is proven end-to-end (guest render -> KVMFR +> ivshmem -> host PNG frame); two guests with 512M + 1GiB emulated VRAM +> coexist at boot level; the 0.9.9 release-preview demo video v1 is +> synced to files.vgpu.io (the updated video with GZDOOM working is the +> next publish). Frontier = non-black frames, then Xorg -> GZDOOM +> (GL/Vulkan) -> KVMFR/LG frames -> CUDA 15/15 + soak -> the perfect +> 1.0.0 video -> Windows guests).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -253,6 +364,111 @@ isolation, performance, portability). contamination demonstrated live) and needs the per-guest ownership rework (owner-tagged pin/map/pending lookups) before any dual-CUDA claim. No reboot; host GPU green; both guests left running. + **Update 2026-10-16 ~22:35, extended ~23:05 (reliability wave 2, m163-m170; cycles r169-r177, no host reboots):** + the stall classes that made clients freeze at DEVICE are largely closed. Fixed and + verified across boots r172/r174/r175/r177: the session flush can no longer wipe a live + session (a deferred flush executed with the client's channels valid, dropped its + queued launch release, and froze it in the tracker wait); GR entries whose decode + sees QMD words but builds no launch candidate are held and re-captured instead of + consumed silently; a failed device-destination copy holds its entry instead of + silently dropping the data; pool releases apply at fresh-pin time instead of + registration+5 s; the timeout ladder fails fast when the engine is wedged; argument + spans whose first page resolves nothing hold the launch (bounded) instead of + forwarding stale operands; device-destination copies answered only by a page-table + walk wait (bounded 3 s) for a pin. `victory_mail`, `victory_matmul` and `smiley` + are green on every boot of the wave, including the user's exact repro and mail as + the eighth client of a used boot; r177 ran mail, matmul, smiley, matmul (repeat) + and mail GREEN back-to-back as its first five clients. streams2/memset2d/saxpy no + longer wedge - they + now fail fast (~35-60 s) on the remaining data class: an H2D into a device buffer + that does not land where the launch's data fill later reads the operand (the + vidmem write path vs the fill's read path divergence; analysis and the ranked + ledger fix are in the private handoff, cont. 23). + **Update 2026-10-17 ~01:00 (stall pinpointed on r179; diagnostics only, no tree change yet):** + a minimal repro (`cudaEventCreate` + `cudaEventRecord` on the default stream + + `cudaEventSynchronize`) sits forever after the record - the first CUDA call pinned down + as a hard hang. A 3-stream workload using event record/wait/elapsed time stalls even as + the very first client of a fresh boot (intrinsic, not position- or poison-dependent; + unclean client kills were tested and do not break the next client). On a used boot the + user's `victory_mail`/`victory_matmul` hit the same wait: the client spins in driver + userspace while a background worker polls the interrupt-service RM control once per + second. The host's event-registration table and post path exist (register on NV01_EVENT + alloc, retire on free, per-channel/all post helpers) and the stall windows still show + frequent post failures with EINVAL; the host sema side is verified healthy at the stall + (client pushes decode, pool releases apply to the correct page, then the machine idles). + Cross-boot matrix: mail/matmul/smiley green early; strided/atomics/memset2d/saxpy fail + fast on the remaining data class at any position; event-sync stalls always. Next: fix + the completion post so the recorded event wakes (details in the private handoff, cont. 24). + **Update 2026-10-16 ~18:40 (reliability wave m160-m162 + harness fixes; verified battery, no reboots):** + five cycles (r163-r167) landed and verified the bounds of the earlier "STATUS FAIL / + freeze" repro. Fixed and live-verified across boots: `victory_mail` and + `victory_matmul` (repeats), `smiley`, `vector_add`, `victory_math`, `victory_strided` + and `vecadd_loop` (100 launches) all green on the physical engine; the acked-no-execute + class is recovered by outcome retries (fresh release-word ack, fresh inline-QMD bank + + rebuilt method block: a launch with zero engine writes recovered on retry 2 with its + full 1024-page output). A hardware fetch witness (QMD sentinel) now distinguishes + fetched-vs-not in every retry log. The post-teardown DEVICE-freeze cascade was rooted in + the pin-feeder tooling: blocked guest-trace reads over the guest agent stopped the + pool/tracker pin flow after ~5 client cycles and every later client froze in + `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the + demand-pin content resolver searches a learned per-allocation offset. Remaining + characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch + class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out + mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, + retest pending). + **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):** + the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was + reproduced and root-caused: under the driver's status-queue SWAP_RX mode the + emulation's status free-space check read the status queue's mirrored RX header + (which the emulation publishes itself) instead of the peer consumption cursor in + the command queue's RX header. With the guest mid-drain the status ring was + declared full, replies were written anyway, unread records were overwritten, and + the guest RM read a record one full ring lap ahead + (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed + and the first client's `cudaMalloc` returned "device busy or unavailable" - + poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the + cursor is read from the command queue's RX header per the driver's swap-mode + semantics; per-client channel unregister and session flush work again (one + unregister in the poisoned boot; 79 with one flush per client after). Verified on + fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul` + (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`, + `FFN4 layers=4 bad=0` + `STATUS OK`) all green. + Remaining large-buffer class (current frontier): a `vector_add` cold-run content + gap (a staging read/write race between the emulated CE copy and the guest's + staging ping-pong: ~0.68 MB of the third MB reads the previous staging content), + `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop` + hang class correlated with a boot where the injected-pin stream never re-targeted + the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class + (RM object fields overwritten with client data; stale system-memory destination + resolution on freed staging pages) is characterized but open. Next: pin-stream + re-targeting, then the staging-race and stale-destination fixes; then the + boot-lottery re-measure over fresh boots, the full battery + soak, two guests, + multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live + with the three green kernels. + **Update 2026-10-20 ~20:00 (pin-wins corruption fix; multi-tenant live; battery frontier):** + a second, independent fix landed after the SWAP_RX cursor: the emulation's + write-side ledger could redirect a copy to the page an earlier write landed + on even when the guest had already remapped or freed that VA (4448 such + divergences in the failing run), so guest-kernel memory could receive client + copy data - observed as an RM object's pointer field holding client float + data, a free-path traversal faulting, and a guest panic + ("Attempted to kill init") on the `vector_add` client. With a pin-backed + translation (guest pagemap truth) now winning the divergence and the ledger + re-homing after the write (`720d3f5f`): `vector_add` cold AND warm run + `bad=0` + `STATUS OK` and the guest survives; the cold-run content gap + class is cured with it. Battery on fresh boots: `victory_mail`, + `vecadd_loop` (100 launches), `victory_matmul` (a first-run all-bad flake, + green on immediate retry), `victory_strided`, `victory_math`, + `vector_add`, and a transformer kernel (`dh=256`/`FFN4` `bad=0`) are green; + streams2's ordering class is root-caused (core CUDA starts compute-channel + pushes with a C56F host-semaphore acquire that the decoder ignored, so a + D2H readback ran before the scale launch it waits for) and the multi-pass + content class is the remaining frontier (multi-launch/accumulating kernels + wrong; `victory_reduce` is exactly 2x - the engine never wrote on five + launch-shaped pushes of a one-launch client). Multi-tenant: two mediated + devices (512M + 1GiB) and two guests coexist; guest B reports its 1GiB + VRAM with `nvidia-smi` green; guest-B CUDA awaits its own pin stream. Host + clean, no reboots. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -576,6 +792,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM | | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | @@ -612,6 +829,29 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX + cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20 + ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`, + `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM + logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed + (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable") + onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr + lives in the command queue's RX header (the driver's swap-mode semantics), + while the status queue's own RX header is the mirror the emulation publishes + (its own command-queue read position); the status free-space check read the + mirror, so with the guest mid-drain the ring was declared full and replies + were written over unread records. Fix (`c72f8948`): read the peer + consumption cursor from the command queue's RX header. Verified on fresh + boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256 + bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`) + all green with `STATUS OK`; per-client channel unregister + session flush + restored (1 unregister in the poisoned boot vs 79 after). Remaining + frontier: the large-buffer class (`vector_add` cold-run staging race, + `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with + an un-targeted injected-pin stream), then boot-lottery re-measure, full + battery + soak, two guests, multi-mdev, VRAM sizes. Evidence: + `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in + tmux session 0 (SSH to the guest, three green kernels). - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index ef4b258..242f470 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -131,20 +131,26 @@ document: * **Working today**: the virtual device, the GSP boot handshake, the RPC transport, the resource-manager emulation surface the guest driver needs to - probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the full CUDA - driver-API path through context creation, allocation and verified copies - (the driver-API smoke passes). -* **Needs improvement**: engine execution first — the backend is live and - CE-class copies and their releases are forwarded to the physical copy engine - (default off) with a fault relay, but for CUDA kernel execution the guest's - launch now reaches the front-end and the inline launch-descriptor install has - not been observed (context validity at dispatch is ruled out; see - [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) - — then the remaining reply fidelity (some consumed - fields are still generic fallbacks), the memory/page-table model, Windows - guest support, non-KVM hypervisors, and performance instrumentation. Each of - these is tracked with explicit "known-unknowns" in + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, the full CUDA + driver-API path, and **CUDA kernel execution on the physical GPU** — + verified guest clients (vector-add, matrix-multiply, transformer) execute on + the engine with engine-authored completions relayed to the guest. On the + graphics side, the X server and GLX run on the emulated vGPU, the engine + executes GL method streams (verbatim 3D replay and synthesized launches), + and the guest Vulkan pipeline runs end to end. +* **Needs improvement**: the staged-transfer class (some large multi-chunk + CUDA clients still mis-verify; failure modes proven), the every-boot + acceptance campaign, per-guest execution isolation for simultaneous guests, + the rendered-pixel frontier for OpenGL/Vulkan (program delivery plus + program-VA mapping), the remaining reply fidelity (some consumed fields are + still generic fallbacks), Windows guest support, non-KVM hypervisors, and + performance instrumentation. Each of these is tracked with explicit + "known-unknowns" in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); + execution detail: + [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) + and + [`../04-implementation/05-graphics-execution-plan.md`](../04-implementation/05-graphics-execution-plan.md); for the CUDA analysis area, start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index 733da13..3d97d07 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -568,6 +568,39 @@ forwards (local evidence and the prepared harness: forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. +### 3.5a Copy-out change detection: fingerprint rule + +A forwarded launch stages every referenced data page into the data-bounce +chunk, and after the retire wait the copy-out ships back only the pages the +engine actually modified. That decision used to be a 32-bit additive +checksum of the page's words, compared against a baseline captured at +copy-in. The additive form has a structural blind spot: doubling every word +of a 4 KB page raises each float exponent by one (+2^23 per word), and 1024 +words contribute `1024 * 2^23 = 2^33`, which is exactly `0 (mod 2^32)` — an +in-place power-of-two scale of a whole page (`c[i] *= 2.0f`) left the +checksum bit-identical, the page was classified "engine did not touch", +the write-back was skipped, and the client read the pre-scale values. The +same blindness applies to `*= 0.5f`, `* 4`, `* 8`, etc. Since `r452` +(commit `19399f4f`) every change decision in the copy path — the retire +wait's settle scan, the tracked and gap copy-outs, the window witness, the +late-store witness and the idempotent-launch fingerprint — uses a 64-bit +FNV-1a hash over the page instead; the additive sum survives only as the +all-zero filter. Policy rule for the plane: **a change-detection +fingerprint must not be an algebraic invariant of any plausible kernel +operation** — use a hash, never a sum. + +The reader side of the same decision is the landing table: a D2H copy of a +device VA resolves through the last recorded write of that VA. Since `r453` +the forwarded QMD's release0 is redirected to the module semaphore by +default (`gsp_phys_rmcall_gr_rel_mod=1`): the guest-visible release word is +authored by the host only on the completion edge, after the copy-out has +shipped the engine's stores, so a client's stream-ordered readback can no +longer run ahead of the write-back (the historical shape: the engine's own +release satisfied the guest while the copy-out was still pending). The +engine-completion observable remains the module semaphore, and the +per-outcome authoring path guarantees the guest word is applied on every +outcome (timeout, wedge, no-write). + ### 3.6 Citation map for this section | Claim | Authority | @@ -1038,6 +1071,53 @@ technical barrier between the project's architecture and its promise. --- +## m91-class I-fetch faults: capture and icache coherence (2026-10-18) + +The physical submit's remaining transient failure class ("m91 class" after the +trap-decode campaign that first named the SM instruction-fetch signature) has +two observable faces. In one, the first attempt of a mid-sequence launch +raises SM warp exceptions ("Illegal Instruction Encoding", "Out Of Range +Register") on garbage instruction bytes while the engine-authored release +stays at its failure value, and a retry of the *identical* QMD and shadow +content succeeds. In the other, the launch reports fetch success but writes +nothing (dispatch loss). The submit retry that recovers the first face is the +fast re-arm: the same method block is re-published on a fresh GPFIFO ring +entry and the mode-35 lane-enable tail plus channel/TSG schedules and the +doorbell are re-issued — no shadow re-copy, no QMD re-read, no content +change. Identical content flipping outcome across attempts is the signature +of stale engine-side fetch state, not a content or relocation error. + +Two code changes follow from that: + +* **FAULT-CAP (diagnostic).** When a forwarded launch's fault epoch advanced + across its window, the submit emits one `vgsp-m91: FAULT-CAP` line per + launch (capped at 20 per module-load cycle, then one suppression notice): + fault-epoch delta, the forwarded QMD address and first 64 B, the live + program pair (word 48/49 read) with its resolved engine-visible VA/PA, the + first 64 B at that code address, the retry decision (fast re-arm / + resched/fresh-entry / none) and outcome (ok vs timeout), and the newest + relay-decoded fault note (exceptType, chid, errString). It is inert on a + fault-free launch: one epoch read and one compare. +* **icache coherence (fix, default on).** The window-0 shadow pool alternates + two 8-page banks by submit epoch, so the bytes at a program VA change every + two launches, while the captured guest QMDs carry the QMD v2.3 + cache-control word (word 5) with the + `INVALIDATE_INSTRUCTION_CACHE` bit (bit 30) clear — the field the client + driver arms only under its forced-invalidate mode. The submit now sets + that bit in the forwarded QMD when the QMD carries the driver's standard + cache-control signature (word 5 bits 26..29 set), forcing an instruction + fetch at dispatch and removing the stale-line window; an unknown QMD + revision is never touched, and `gsp_phys_gr_icache_inv=0` restores the + exact pre-change QMD for a one-flip A/B. Setting the bit cannot alter + instruction bytes or data — it only invalidates cached fetch state — so + green launches keep their semantics. + +The next physical cycle's evidence is the FAULT-CAP lines (or their absence) +against the four-client class: zero FAULT-CAP lines with the four clients +green confirms the stale-fetch mechanism; FAULT-CAP content policies the +remaining candidates (shadow-VA reuse distance, cross-cycle context state +left by killed clients). + ## See also * [`01-system-overview.md`](01-system-overview.md) — principles and layer map @@ -1057,3 +1137,4 @@ technical barrier between the project's architecture and its promise. |---|---| | 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. | | 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (`svc_thread`, `svc_gr_request`, `svc_pending_add`, H3 guard, readiness gate); GR completion §3.4/§6.1 reconciled with FORENSICS-grv5 (wake-up fires; the counter publication is the leading reading); discovery cadence wording corrected (2-second paced passes, not "milliseconds"); mode-matrix source note corrected. | +| 2026-10-18 | m91-class I-fetch section: FAULT-CAP first-fault capture (`vgsp-m91: FAULT-CAP`, 20/cycle) and the default-on forwarded-QMD instruction-cache invalidate (`gsp_phys_gr_icache_inv`) with the retry-state analysis (identical content, fresh ring entry + lane-enable replay; stale engine-side fetch state at reused shadow VAs). | diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md index f5546ca..a8e03ee 100644 --- a/docs/04-implementation/03-execution-frontier.md +++ b/docs/04-implementation/03-execution-frontier.md @@ -1,11 +1,17 @@ # The Execution Frontier: The GR Compute-Launch Chain -> **Status: 2026-10-06.** This page describes the current leading edge of +> **Status: 2026-10-20.** This page describes the current state of > guest-facing execution: the chain that must fire for a stock guest CUDA -> kernel to run on the physical GPU, what we have proven about it, where it -> currently stops, how we know, and the open lines of investigation and -> theories. It is evidence-led: every statement carries either a live -> observation, a decompilation finding, or a clearly labelled theory. +> kernel to run on the physical GPU, what we have proven about it, how we +> know, and where the leading edge is now. The chain fires end to end: +> verified guest CUDA clients (vector-add, matrix-multiply and a transformer +> kernel) execute on the physical GPU, their results verify, and the +> engine-authored completions relay back to the guest. The leading edge has +> moved from "can a forwarded launch execute at all?" to breadth and +> hardening — the large-buffer staged-transfer class, reproducibility across +> boots, and per-guest execution isolation. It is evidence-led: every +> statement carries either a live observation, a decompilation finding, or a +> clearly labelled theory. > > Evidence sources for this page: supervised physical cycles (module-owned > GSP-RM on the reference TU104), offline firmware emulation/probing, the host @@ -40,40 +46,59 @@ flowchart TB J["Host polls release, relays to guest;<br/>CUDA wakes"] ``` -| # | Stage | State (2026-10-06) | +| # | Stage | State (2026-10-20) | |---|---|---| | 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) | | 2 | GPFIFO entry + doorbell | ✅ observed | | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) | | 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) | -| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** | -| 7 | **Engine reads the QMD** | ✗ **not observed** | -| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 | -| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack | -| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches | -| 11 | Host relays completion; guest wakes | ✗ guest waits and times out | - -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) | +| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed | +| 7 | Engine reads the QMD | ✅ observed | +| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 | +| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve | + +**One-line summary:** the whole chain is live — verified guest CUDA clients +execute on the physical GPU and wake up correctly. The leading edge is no +longer the chain itself but its breadth and hardening: the large-buffer +staged-transfer class, reproducibility across boots, and per-guest execution +isolation. --- ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire we observe the ring entry consumed and the -front-end methods run — but the launch descriptor is never materialised, the -engine never reads a QMD, no engine-authored completion appears, and the guest -blocks in its synchronisation wait. - -This is **not** a context-validity wall (that was the previous frontier and is -now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the -descriptor destination (that install is accepted). The wall is the **form of -the launch method stream**: what the front-end requires in order to install the -inline descriptor, and what the engine requires in order to read and dispatch -it. +A stock, unmodified guest CUDA program now executes its kernels on the +physical GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. The engine authors the completions, the host relays them to the +guest, and the guest's synchronisation waits resolve. + +The leading edge has therefore moved from the launch chain itself — it fires — +to breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise, the first + CUDA client would fail, and the rest of that boot was poisoned (the + user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement + across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. --- @@ -81,17 +106,17 @@ it. | Observable | What it proves | What it does **not** prove | |---|---|---| -| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine | -| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) | -| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started | -| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction | -| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path | - -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results | +| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches | +| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored | +| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress | +| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction | + +**Consequence:** acceptance is now measured by verified guest results plus the +engine-authored release; front-end signals remain necessary but not +sufficient. A green run counts as progress only when the same acceptance +reproduces across fresh boots. --- @@ -111,90 +136,94 @@ signals, which we have already learned to distinguish. * **Parenting/TSG for the fetch** — both parenting variants fetch identically. * **The copy-engine path** — CE-class work, its releases and its fault relay are live (default off); the GR path is a separate problem. +* **The descriptor-gate line of investigation (the old T1–T4 theories).** The + inline-descriptor install, the engine's QMD read, the grid dispatch, the + engine-authored release and the guest wake-up are all now observed; the + gate they questioned is closed by observation (§1, §7). +* **The old boot-poisoning defect.** Root-caused to the status-queue + consumption cursor and fixed; it is no longer the explanation for a failing + first client (§5, issue B). --- ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** -The forwarded stream includes a verbatim copy of the guest's prelude whose -inline-data writes target addresses that are not present in the engine's view -of the channel's page tables, and the referenced data is not relocated. The -shorter, proven stream shape fires its tail (channel semaphore) but still does -not schedule the descriptor. -*How we know:* the prelude stalls in the forwarded form; the shorter shape's -tail fires; the descriptor-install sentinel never changes in either case. -*If fixed:* the install path is reached and we can observe stage 6 directly. - -**Issue B — the descriptor-virtualization control method is absent.** -The method that arms the inline-descriptor install path (the QMD -virtualization control) is not present in the forwarded stream. -*How we know:* the method does not appear in the submitted stream, and the -install sentinel never changes. -*If fixed:* the front-end should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. - -**Issue C — diagnostic hygiene.** -The descriptor sentinel placed at word 0 overlaps real descriptor fields; it -must be moved to an unused word so diagnostics do not perturb what they -measure. +**Issue A — the large-buffer staged-transfer class.** +Multi-megabyte staged transfers between guest memory and device memory are +under repair: some large multi-chunk clients still mis-verify. +*How we know:* the small and medium clients (vector-add, matrix-multiply, the +transformer kernel) verify green while large multi-chunk transfers mis-compare +or stall in the same configurations. +*If fixed:* the client battery should accept the large-buffer class with the +same verification discipline as the green clients. + +**Issue B — reproducibility across boots (re-measurement).** +A boot-poisoning defect was root-caused and fixed: under the driver's +status-queue mode the emulation read the wrong queue cursor, so under load the +reply ring could desynchronise, the first CUDA client would fail, and the rest +of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report). +*How we know:* the failure was reproduced, the cursor rule was corrected, and +per-client unregister and session flush work again. +*If fixed:* green on every fresh boot — re-measurement is in progress, and no +"every boot" claim is made until it completes. + +**Issue C — per-guest execution isolation.** +Multiple mediated devices with different emulated VRAM sizes and multiple +guests already coexist at the device level; per-guest execution isolation for +multiple simultaneous CUDA guests is in progress. +*How we know:* the device-level coexistence is live; execution isolation is +not complete. +*If fixed:* multiple CUDA guests execute simultaneously without interference. --- ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline (the shorter stream that - fires its tail) and **re-add the virtualization control method** — one - variable at a time, with the install sentinel and the engine-read signal as - the readout. -2. **Relocate or drop the verbatim prelude** so the inline-data writes target - addresses present in the engine's view, or replace it with the equivalent - host-side staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference.** Two ways to obtain one: (a) make - our own ladder dispatch a descriptor built from the same source material - (in-house reference), or (b) capture a working launch of the stock stack on - the same GPU (real method stream + descriptor + engine-visible channel - state) and compare field by field. Either reference converts the remaining - work from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware emulator/probe tooling before spending a - supervised cycle. +1. **Repair the large-buffer staged-transfer class** — the multi-megabyte + guest↔device transfers that still mis-verify. The three green clients stay + the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility, and + keep the re-measured battery as the regression gate. +3. **Per-guest execution isolation** — build from the device-level + coexistence of multiple mediated devices and guests to per-guest execution + isolation. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel run as a battery on every re-measurement; each entry is + accepted only on its own verification output (`bad=0` + `STATUS OK`). --- -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the virtualization-control method - enables the install path; its absence alone explains the inert descriptor. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape - works, the prelude's refused writes leave the launch path in a state that - prevents the descriptor from being scheduled. -* **T4 — relocation is the missing transform.** The guest's inline descriptor - is valid, but it is built for the guest's view of memory; the forwarding path - must relocate referenced data (and any embedded pointers) into the channel's - view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed (§1), +so the launch chain they questioned is now proven end to end. + +The remaining leading-edge work is engineering rather than theory: the +large-buffer transfer class, boot-to-boot reproducibility, and per-guest +execution isolation. Each has an acceptance signal named in §5 before the +work. --- ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals (ring cursor, method execution, semaphores that the front-end can - author) are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end for the verified clients; the guest's own verification + (`bad=0` + `STATUS OK`) is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred; the current GR path runs without an error-reporting object, which is itself a known observability gap. * **Emulator verdicts are L0/L1.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** The + boot-poisoning defect is the worked example: one green boot is not + reproducibility. * **A wall is closed only by an acceptance signal**, and the acceptance signal for this frontier is named before the experiment. @@ -218,3 +247,4 @@ signal, and the engine-authored release, in that order. | Date | Change | |---|---| | 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. | +| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. | diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md new file mode 100644 index 0000000..f2ba0ab --- /dev/null +++ b/docs/04-implementation/04-real-render-walls.md @@ -0,0 +1,909 @@ +# Real-render walls: Vulkan and Xorg on the emulated vGPU + +Status: OPEN (2026-10-29, SESSION 64). This page records the measured state +of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg +server - on the emulated vGPU, with the exact evidence and the next +subsystem to implement. It complements +[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md) +(which covers the CUDA -> ivshmem -> host frame transport that is already +proven). + +## 1. What works today + +* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**: + `vkCreateInstance` succeeds and two physical devices are reported - + `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the + software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes: + `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`. +* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device + platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older + `egl_frame`). +* CUDA is unaffected (its clients run through the same emulated GSP; see + the battery evidence in the handoff). + +## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE + +`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An +`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the +ICD's first nvKms ioctl failing: + +``` +NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE +``` + +i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a +non-success status; the userspace frees everything and reports the +generic initialization error. Reading the driver's own modeset source +(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed: + +1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS` + (`0x730102`): a zero reply logs `No heads found on board!` and fails the + alloc. The emulated GSP RSP answered all NV0073 controls through the + generic zero-reply path. +2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED` + (`0x730107`) must report at least one display id, and the connector + path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA` + `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector. + +The host GSP-RM emulation now implements a minimal headless display: +one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`), +one static HDMI connector on display id 0 (SOR + single-link TMDS, which +avoids the DP-AUX library), plus DDC support. With this shim the alloc +gets past heads and `NoValidDisplays` and proceeds through the connector +queries and the display-object allocations (classes `0xc570`, `0xc57d`, +`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing +later in the display-core/modeset-channel path. That remaining subsystem +is the wall: the modeset core channel and the per-display dpy objects +must be emulated (a coupled object + channel + memory set whose exact +wire forms should be captured from a real-GPU session). + +A related experiment ruled out one shortcut: booting the guest with +`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does +not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and +fails, so the display path cannot be sidestepped by disabling KMS. + +## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test + +Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then: + +``` +(EE) NVIDIA(GPU-0): Failed to initialize DMA. +(EE) NVIDIA(0): Failed to allocate push buffer +``` + +`nvidia-push` logs `Failed to initialize DMA.` from +`__nvPushTestPushBuffer()`: right after allocating a push buffer the +driver writes a NOP and waits (10 s) for the channel to consume it. Host +evidence shows why it never does: + +* the X driver's channel registers (`class 0xc46f` GPFIFO channel, + `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with + **`submissions=0`** - its published GP_PUT never advances in the host's + view, because the host needs the channel's VA->PA translations to read + its USERD/GPFIFO and that resolution comes from the guest-side scanner + that currently follows *CUDA clients only* (`targeted resolve + va=0x20400f000: full RAM scan found nothing`); +* on a busy boot (Xorg started while a CUDA battery ran) the channel's + completion-counter slot is rejected: `counter auto h=0x1000013 rank=0 + pa=0x11c827ff0 rejected (slot owned by another channel)` - two + concurrent clients both derive rank 0's slot from the per-device + structural formula. + +Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution +path for the scanner or the host, and (2) counter-slot ownership keyed by +client identity (or a per-client slot domain), not just rank. + +## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged) + +`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init +entry returns a "busy" status once, sets a state bit, and the eglcore +caller does not retry. Ten in-process retries (`egl_retry`) all fail, so +the state machine fails on a real input - the return-value trace of its +callees remains the way in (see +[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)). + +## 5. Why this matters for the frame path + +Everything downstream of a working GL/Vulkan context is already in place: +a guest publisher writes frames into the kvmfr ivshmem BAR +(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout +as the proven CUDA `frame_producer`), the host reader decodes them +(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing +piece is a *working* graphics API on the guest GPU - the two subsystems +above. +## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected + +Three new instruments settled the remaining questions: + +* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`) + decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 = + `NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**. +* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope** + (the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`): + breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped + from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped + the driver's class list (42 classes; all `NVC57x` display classes are + present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no + HAL class list, so the "base channel" fallback path can never succeed). + Breaking at the capability parse showed the **capability notifier buffer + is all zeros** (nothing emulates the display engine that would fill it). +* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS` + and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the + parse, the driver flips from the dead base path to the **window path** + (`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0` + unblocks the flow is confirmed. + +The window path then fails one step later: `nvRmApiAlloc(display, +class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED` +(0x56) locally in the guest kernel** (it never reaches the emulated +GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU +architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the +display/modeset channel classes are rejected (the reference source's +`_rpcAllocObjectPrologue` documents "Display classes are added for DCE, +vGPU Guest does not support those") - a vGPU guest's display is meant to +be emulated by the **host-side vGPU display service** (the vGPU RPC +display path), which the emulation does not implement. + +**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated +vGPU needs (a) the display capability fill on the host side (mechanics +proven by the gdb experiment) and (b) a vGPU display service: the +display-engine MMIO/RPC backend the guest expects from its host (the +equivalent of NVIDIA's vGPU display plugin), or the specific vGPU +configuration that makes the kernel accept the display channel class. +Until (b) exists the guest's NVIDIA display path is closed by design, and +GZDOOM-class workloads cannot render on the guest GPU. +## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path + +A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is +**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`), +`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any +`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes +addressable for live gdb breakpoints on the QEMU stub. With it: + +* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for + 0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register + mapping `hClass = r8d` was verified live). +* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters** + with class 0xc57e, calls `serverAllocResourceUnderLock`, and that + function returns **0x56** - *before* the GSP-RM alloc RPC + (`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never + sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573 + return 0. +* So the last blocker for the window path is a **local RS-layer validation + inside `serverAllocResourceUnderLock`** (candidates: the inlined + privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup, + or `clientAllocResource`'s construct dispatch). The 0x56-producing + instruction is not yet pinned - a `stepi` loop over the QEMU stub is far + too slow and wedged the guest (recovered by a fresh cycle); the next + session should use conditional breakpoints on the function's call-return + sites (the approach in `gdbdiag19` works; the earlier run had truncated + addresses and needs the `0xffffffff00000000` base restored) or a + watchpoint on the status word. + +**gdb-stub field notes**: the stub is single-client; killing an attached +gdb leaves the VM stopped (recover with a detach-only connect); +`stepi` loops over the stub freeze the guest's userspace for minutes - +prefer breakpoints; every probe run with the caps-fill breakpoint changes +the driver's path, so keep the fill in each experiment. + +## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD) + +**The fence wall is closed.** The client's per-frame fence wait polls a +16 KB BO word that a submit-push INC-5 semaphore group targets +(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a +GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring +already carries the entries it pre-built for the next submits, idx12..46), +so the GP_PUT-bounded pump walk never decoded it and a host-side assist had +to write the word. The service now sweeps that pre-built continuation +itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the +ring's validated translation, accepts only pushes that pass the strict +method-stream gate, extracts the addr-0x5c RELEASE groups and applies them +forward-only through the standard pending ledger. Live evidence: the sweep +finds the groups and the ledger applies the fence word once per frame; +`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still +black - the shader work itself is unexecuted. + +**The compute-dispatch shape (why nothing executes).** The workload is a +single compute dispatch (640x480 storage image). Its push is a DIRECT +method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0` +on subchannel 1, then the context windows (`0x7b4` local-memory window B, +`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and +the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data +port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO +inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture +that feeds the physical-engine forward path never fires - and the CE-shaped +ops path is not even consulted for GR-class channels by construction. Real +pixels therefore require a translator that recognizes the C5C0 section, +builds the vendor launch the physical channel expects (the same prelude/QMD +shape the CUDA path already executes) from the section's fields and data +runs, and forwards it with the existing operand/relocation machinery. The +method vocabulary overlaps the physical prelude's existing knowledge +(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known. + +## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate + +**The arm was dead by branch order.** The utility channel's engine-side +uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and +their operand group (`0x188/0x18c` destination, `0x180` line length, +`0x184` line count) was parsed only by an `else if (subch == 1 || subch == +2)` branch that could never run: the preceding `if (xlate && subch >= 2)` +TURING-copy parser also matches subchannel 2 and swallowed the group +unread. With `have_out`/`line_len` unset, the `0x1b4 +LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now +parsed first for subchannel 1|2; the arm reaches its write decision and +authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem +0x1c0000). + +**The payloads are stably zero - and that is the driver's zero buffer.** +Four independent reads (the sweep's channel path, the direct +registered-VAS walk - 22k attempts, never non-zero - the guest's own +`/proc/<pid>/mem` view of the client's mapping, and the guest's +`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a +gdb hardware write-watchpoint that never fires agree: the driver never +writes the payload chunks for these entries. The 256 B size equals +`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()` +uploads exactly 256 B of zeros through this path. The arm therefore +applies zero payloads under a variant once-key (`h | 1`) - idempotent over +an empty destination - while a later non-zero payload for the same entry +still lands under its own key. + +**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900` +resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is +not delivered by any decoded entry. The utility channel's ring holds +exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA - +both the zero objects above. The Vulkan (NVN) download path uploads via a +separate download state and points its destination view at the pitch-pool +allocation, so the SASS/constants i2m shape is a different destination +family than the observed zero-buffer uploads. Next steps: find the SASS +upload shape (pitch-pool destination, possibly on another channel or a +non-LID copy) and then the `n_data` descriptor/operand wiring +(section 8); with both, the physical forward can execute the dispatch. + +## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem + +**The ring is in vidmem, and its content varies per client.** The +utility channel's ring VA (0x4020000) resolves through the service's +learned map to a **vidmem** offset (PA 0xa0000, next to the channel's +USERD at 0x8fe00), and the pump's own entry read shows the live ring's +content: on the r750-era clients `entry[0] va=0x120100000 len=0x28` +(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000 +va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000) +confirms exactly one entry there. + +**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are +from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA) +can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an +EARLIER client's ring - a coherent 14-entry ring whose LID uploads target +the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000, +dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and +12288 B). Those entries are real, but they belong to a dead session: the +payload chunks they point at (VA 0x4250000/0x4250110 -> the session's +1 MB staging BO) read zero through EVERY path - the sweep's channel read, +the direct registered-VAS walk, the learned map, the guest's own +`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run +high-frequency poller of every 1 MB BO the client maps, and a gdb +hardware write-watchpoint that never fires - because the session ended +and the BO's pages were freed/reused. **The payload is not missing from a +live upload; it is a stale-session artifact.** + +**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on +the same boot show the arm working with real payloads: +`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378 +tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0 +dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the +push and holds a real method stream, the arm materializes it into the +high vidmem family, and the CUDA battery passes. + +**Next steps (ranked).** +1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a + client run and decode its entries - that is the live upload path; the + sweep's `svc_read_va` source preference must not answer with a stale + RAM page (verify the ring page by content: the entries must lie in the + client's current VA allocations, and the ring must be the same page the + pump reads). +2. With the live ring decoded, the SASS upload shape follows; the + descriptor/operand wiring (`n_data=0`) remains the second gate. +3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets + were exhausted early in multi-client boots, hiding later rows; make + them per-session for diagnosis. + +## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not + +`dump_vidmem map` (new: one line per 1 MB block with non-zero content, +`nz` = non-zero dwords) on a fresh boot, same module: + +* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up: + `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`, + `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`, + `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's + uploads (LID i2m writes with real payloads + CE copies) materialize in + the emulated vidmem. +* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14 + baseline dwords (the fence/counter words at 0xfee88, the GR param bank + at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads + reaches the emulated vidmem at all.** + +That is the frontier in one measurement: the Vulkan (NVN/GL) driver's +upload path is not among the writes the emulation can see. Combined with +section 10 (the utility ring lives in vidmem via the learned map but the +sweep can read a stale RAM page for it; the live ring holds a single +`0x120100000/40` entry, not the 14-entry LID ring), the next session must: +(1) capture the LIVE ring through the same source the pump uses while the +client is up and decode its entries (the 14-entry LID ring is a +stale-session page); (2) trace where the live client's upload writes go +(the map shows they are NOT in the first 256 MB of the shadow - check the +guest's own view of the client's staging BOs during the upload window); +(3) the descriptor/operand wiring (`n_data=0`) after that. + +## 12. UPDATE 2026-10-26 (SESSION 132): THE LIVE RING IS PROVEN (guest RAM, walk-confirmed); the vidmem "ring" was a false-positive pin whose dead entry the pump skipped 39 times; the two LID uploads are the driver's zero buffers by design; the code VA's mapping rotates and the shadow's lat cache can go stale + +**The live ring is in guest RAM and the sweep's decode was right.** +`walk_probe 0x4020000` (the client's own page tables, vas[0] +`h=0xfade0002 pdb=0x114039000`) resolves the utility channel's GPFIFO VA +to **guest RAM** (PA `0x11623d000` on r759; the page moves per run: +`0x104b55000` on r761) and its content is the 14-entry ring the sweep +decodes: entry0 -> push `0x4000000` len `0x8bc` (the methods), entry1 -> +the 256 B payload slot `0x4250000`, entry8 -> the 12288 B payload slot +`0x4250110`, ... - all pushes inside THIS client's own allocation window +(`PREV149-OUT ... off=0x4000000/0x4020000/0x4060000` in the client's own +VIDHEAP log). The decoded ring is live content, not a stale page. + +**The vidmem "ring" (PA 0xa0000) is a false-positive autopin.** On r759 +the VRAM shape scan found the channel's REAL USERD at vidmem `0xb0000`; +the userd-derived candidate loop then tried `0xb0000 - 0x10000 = +0xa0000`, whose entry0 (`push 0x120100000/40` - the CE-pool push +pattern) passed the plausibility gate and whose USERD probe hit the real +0xb0000 block, so the pin landed. The pinned entry's push reads all-zero +through every source; the pump retried 39 times and logged `skipping +stale entry h=0xbeef0100 va=0x120100000 len=40 after 39 failed rescues`, +while `sema_probe 0x4020000` then answered the wrong page +(`pa=0xa0000 aperture=vidmem`) and poisoned every later read (the sweep +included). Section 10's "stale session page" conclusion was this pin, not +the sweep. + +**Fixes landed (commit `72de0bed`).** (1) `svc_ring_entry_dead` + +`svc_ring_walk_live`: a candidate whose entry0 push provably reads +all-zero through the client's page tables is rejected when the VAS walk +resolves the ring VA to a page with a live entry (applied at the +userd-derived, FB-convention and existing-pin paths; the walk cannot vote +for itself); (2) `svc_gr_ring_candidate` rejects dead-entry candidates +and accepts the channel's known USERD cross-aperture (the driver places +GPFIFO and USERD independently: ring in guest RAM, USERD in vidmem); (3) +the util sweep arbitrates a dead channel-path ring entry against the +direct VAS walk per slot; (4) the util-dec/util-sweep witness budgets and +their dedup keys are per-client now (reset in `svc_session_flush`); (5) +`sema-copyent` logs once per copy OP, not per row - one 480-row frame +copy had exhausted the 128/session budget and hidden every later copy. +Verified on r760/r761: `sema-chan: h=0xbeef0100 ... ring rc=0 +pa=0x108cba000 ap=1` (the live RAM page), `sema-rdw: h=0xbeef0100 +va=0x4000000 len=2236 rc=0`, the sweep decodes the live 14 entries, the +fence fold releases fire, and the client completes (`VKFRAME_DONE +frames=3 640x480 saved=3`, `VKFRAME_IDLE ok`). + +**The two utility-channel LID uploads are the driver's zero buffers by +design, not the SASS.** Live decode (r760): i=0 -> `dst=0x423f000 ll=256 +lc=1 zero=1` (256 B = `GF100_ZERO_BUFFER_SIZE`) and i=8 -> +`dst=0x4448000 ll=12288 lc=1 zero=1` (12288 B = the CE fault method +buffer size). The payload slots (`0x4250000` -> RAM `0x116219000`; +`0x4250110`) read zero for the whole client run - the driver never +writes them. The GR channel's own LID entry DOES carry a real payload +(`dst=0x4b86020 ll=1568 zero=0` - the launch param bank at `param= +0x4b86000 size=0x1900`), which proves the arm works with real bytes on +this boot too. + +**Where the code page stands.** `prog=0xdfd8feb900` (the captured +launch): `sema_probe` answered vidmem `0x62b900` early in the run while +`walk_probe` answered vidmem `0xc8b000` later - the driver's code-buffer +mapping ROTATES (a code pool), and the shadow's learned lat entry for the +VA can go stale. Both pages read zero in the shadow: the driver's code +upload has not been observed on any decoded path (no LID, no CE copy - +the only `sema-copyent` op of the whole run is the 480-row frame +copy-out `dst=0x50b0000 src=0xdfd8d60000`, which executes `rc=0 +bytes=0x12c000` and correctly copies the still-empty image). The +remaining question is the SASS delivery shape for the C5C0 compute path +(section 6's `0x3800/0x39C8` code-run/data-port semantics) and the stale +lat preference for the rotating code VA. + +**Next steps (ranked).** +1. Resolve the code VA through the VAS walk at launch time (the walk is + the live truth; the lat entry goes stale as the pool rotates) and + watch the code page the QMD actually names. +2. Find the SASS delivery: dump the C5C0 section's `0x3800/0x39C8` runs + with a bounded methods witness while the client is live (the params + arrive by LID; the code must arrive by a sibling path). +3. Then the `n_data=0` descriptor/operand wiring (section 8). + +## 13. UPDATE 2026-10-26 (SESSION 133): THE SASS DELIVERY SHAPE IS FOUND - the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push and copies them to vidmem; the CE channel's ring/USERD resolution is the boot-lottery gate that keeps the copies from running + +**The delivery shape (decoded from the CE channel).** A CE-class channel +(eng=11) carries, as entry[4] of its ring, a push whose method stream is a +LAUNCH_DMA copy with the SASS bytes INLINE: + +``` +sema-chan: entry[4] h=0xcaf00012 eng=11 raw=0021340120a00000 va=0x120a00000 len=8500 +sema-bigrun: h=0xcaf00012 push=0x120a00000 off=0x0 hdr=0x64000002 op=3 cnt=1024 subch=0 addr=0x8 +sema-lda: h=0xcaf00012 eng=11 push=0x120a00000 dma=0x00002181 xfer=1 dstphys=1 srcphys=0 out=0x205000 in=0x120a00004 line=4096 have_out=1 have_in=1 +sema-opadd: h=0xcaf00012 type=1 dst=0x205000 src=0x120a00004 bytes=4096 ap=0 src_ap=0 phys=1 src_phys=0 +``` + +The source is the push itself (`in=0x120a00004`, right after the method +header) and the destination is a vidmem VA (`out=0x205000`, then +`0x206000`, ... 4096 B per run) - the engine-side "download program +range" path. The Vulkan client's CE channels carry the SAME shaped +entries (r763: `sema-utildec: h=0xbeef0101 i=4 va=0x120a00000 len=0x2134`, +19 entries swept) - so the SASS delivery is a CE inline copy, not the +utility-channel LID entries (those are the driver's zero buffers, +section 12). + +**Why the copies never ran (three gates).** (1) The CE channel's ring and +USERD live in VIDMEM and resolve only by boot-lottery: the FB convention +matched on r763 (`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> +fb 0x80000 (entry0 va=0x120000000 len=40)`; entries then decoded and the +frame path ran) but read zeros on r765/r767/r768 - the USERD shape scan +instead finds the channel blocks at 0xfee00/0xff000/0xff200 (r767/r768), +so the rings sit at unaligned offsets like 0xefe00. (2) The CE push VAs +(0x120000000+) need the client's VAS registered: the parked +COPY_SERVER_PDES candidate only registered at +15 s (age-out, +`sema_vas_cand_ms`) while the entry hold grace (`sema_ce_grace_ms`, +8000 ms from the cycle) expired first - r763 logged `skipping stale +entry h=0xbeef0101 va=0x120000000 len=40 after 8000 ms grace` and +`sema-pushfail: va=0x120a00000 ... trusted_rc=-2 ipin=0` for the copy +pushes. Setting `sema_vas_cand_ms=5000` registers the VAS at +5 s +(verified: `VAS 0xfade0002 ... registered UNVERIFIED after 5000 ms +parked`) but the ring gate still blocks. (3) The shadow ring scan +fallback (commits `52ef3f9d` + `2735180a`: 0x100 stepping, rate-limited +retry, entry0 shape + USERD probe at +0x10000/+0x2000/+0x1000) found no +candidate on r768: the ring/USERD content is volatile at scan instants +(a `find_userd` read shows `off=0xfee00 get=11 put=11` while a +`dump_vidmem 0xfee00` one second later reads zero), so a periodic scan +misses the window. + +**Next steps (ranked).** 1. Derive the CE ring from a USERD found by its +own shape (put at +0x8c, get at +0x88 - the `find_userd` scan already +does this) instead of scanning for ring entries: for every USERD-shaped +block, try ring = userd - {0x10000, 0x2000, 0x1000} and validate entry0 ++ the probe. 2. Or latch the ring at the first CE submission (the ring +content appears exactly when the driver writes entry0). 3. With the ring +latched and the VAS at +5 s, verify the inline copies execute +(`sema-lda`/`sema-opadd`/`sema-copyent` with the code dsts) and the code +page gets content; then the `n_data` descriptor wiring (section 8). + +## 14. UPDATE 2026-10-26 (SESSION 134/135): the CE first-submission capture landed and its rank-blind first-hit caused a CUDA-client stall regression (fixed); the Vulkan client's CE rings are allocated ABOVE the capture scan's 64 MB bound and the frame chain reduces to that un-latched ring + +**SESSION 134 (commits `36642718`, `928c0730`).** A new instrument fires in +the notifier-completion path for CE channels *before* the clear write - the +one instant the ring content and the USERD are both live - reads entry0 +through the channel translation and the trusted walk, then scans the shadow +(0x100 step, first 64 MB) for a ring-shaped page (entry0 naming a 1 MB +aligned push VA in `[0x120000000, 0x121000000)` with a sane length) and runs +the standard USERD gate at `+0x10000/+0x2000/+0x1000/+0x0`; a validating +candidate is pinned/latched exactly like the regular CE autopin. `928c0730` +fixed the entry decode: the push VA is 40 bits, the scan had declared it +`u32`, so `scan_hits` was zero by construction. + +**SESSION 135 (commit `a674c21c`) - the rank-blind first-hit regression.** +The scan started at offset 0 and accepted the first ring-shaped page for +*every* CE channel. On r779 both `caf00012` (rank 0) and `caf0001d` +(rank 1) latched `off=0x80000 upa=0x90000`: rank 1 read rank 0's +ring/page, its first release decoded to the wrong semaphore word (+0 +instead of +4) and was dropped as stale (`cur=1`, already written by +rank 0), so the client's rank-1 tracker wait never completed. victory_mail +then spun at 100% CPU with `pub_calls=0` on three consecutive boots +(r777/r778/r779: battery pass=0 fail=7). A control run of the r746-era +module (`25523f84`) on a fresh boot passed the same client (VADD +`bad=0 checksum=12204516.000000`, `STATUS OK`), isolating this to the +SESSION 133/134 code. The fix tries the rank's convention offset +(`0x80000 + 0x20000 * rank`) first, scans the rest bounded as before, and +never accepts an offset another CE rank already latched this session; an +unlatched rank stays for the regular per-pass latch. Verified on r781: +ranks 0..5 latched correctly (0x80000/0xa0000/0xc0000/0xe0000/0x100000/ +0x120000, USERD +0x10000 each), rank 1's own semaphore word applied +(`pool-wr va=0x121000004 cur=0 -> val=1 ok`), and the client progressed +(`pub_calls=185132+`, no spin). + +**The Vulkan CE ring on the same boots.** r781 (classic autopin, rank 0): +`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> fb 0x80000 (entry0 +va=0x120000000 len=40)` - a live entry - but the client's pushes decoded +`unread=1 injected=0`: the guest pin scanner covered CUDA clients +(`cdt,va,victory`) and never `vk_frame`. Extending the scanner to `vk` +engaged the tools (demand pins active) but r782/r783's captures reported +`scan_hits=0 latch=0`: no ring-shaped page anywhere in the shadow's first +64 MB, and no classic autopin either. + +**Where the Vulkan rings actually are.** The client's own vidheap ioctl +log (`PREV149-OUT` lines, r783) names the CE/GR channel allocations: +`off=0x4020000/0x4100000/0x4160000` - matching the registered gpfifos +(`0x4020000/0x4100000/0x4160000`) and lying just ABOVE the capture's +`lim = min(shadow, 0x4000000)` (64 MB). Direct dumps of those offsets (and +of `0x80000`) read zero, and a byte search of the full 4 GB guest-RAM image +for the entry pattern (`00 00 00 20 01 28 00 00` / the `0x20100000` +sibling) finds nothing - the ring content reaches neither the shadow at its +allocation offset nor guest RAM. + +**The frame chain (r783, live).** The Vulkan client completes three frames +(`VKFRAME_DONE frames=3`, `VKFRAME_IDLE ok`) but all PPMs are pure black. +The live GR channel (`h=0xbeec0100`) shows the dependency explicitly: +`sema-gr: launch captured ... prog=0xdff0feb900 ... inline=8` together +with `sema-gr: copy pend h=0xbeec0100 push=0x4f4580c src=0xdff0d60000 +dst=0x50b0000 row=0xa00 rows=480 ...` repeated ~20/s. The trusted walk +maps the copy source to VIDMEM (`walk_probe 0xdff0d60000 -> rc=0 +pa=0x570000 aperture=vidmem`) and the destination `0x50b0000` is the +framebuffer. The shadow at `0x570000` reads zero: the data pages are the +ones the CE inline copies are supposed to fill, and the GR launch is held +on their absence (the designed `dstpend` dependency). So the whole Vulkan +chain reduces once more to the un-latched CE ring: no ring -> no inline +i2m copies -> no data/code pages -> held launches -> black frame. + +**Next steps (ranked).** 1. Cover the client's real allocation in the ring +capture: raise the capture scan bound above the recorded channel +allocations (e.g. `0x6000000`) and/or seed candidates from the client's +vidheap offsets, then re-check whether the entry arrives at the +submission instant (a live `sema_vidmem_dump` at `0x4100000` while the +client is streaming would settle it; the post-run dumps read zero). 2. If +the shadow still reads zero at the allocation offset during live +submissions, the client-side write watch is the remaining forensic: find +the CPU pointer the driver writes GPFIFO entries through, its PA via the +guest pagemap, and why the GPU PTE/walk does not name that PA. 3. With a +latched ring, verify the inline copies execute (`sema-lda`/`sema-opadd` +with the code dsts), the code/data pages fill, then the `n_data` +descriptor wiring (section 8) → non-black frames. 4. Re-run the CUDA +battery on the fixed module (the r781 boot was a slow-crawl boot; the +spin stall class is gone) and revisit `sema_gr_xlate`'s multikernel +forward timeout. + +## 15. UPDATE 2026-10-26 (SESSION 135b): every "Vulkan CE ring latch" on record was a STALE CUDA BLOCK - the Vulkan CE ring content has never been observed; the write path is not BAR1 and the content is in no shadow, no RAM, and not at the PTE target + +**Forensics on four fresh boots (r782-r786; commits `a674c21c` in tree, `c3cdcf4b` new).** + +*The write path works; the driver does not use it for the ring.* A direct +BAR1 test - the guest writing a known pattern at `BAR1+0x4100000` through +`/sys/bus/pci/devices/0000:01:00.0/resource1` - lands in the emulated +vidmem shadow verbatim (`RINGTEST` + `AAAA...` read back by both the guest +and `sema_vidmem_dump`), so the mdev's BAR1 region path (mmapped, direct +into `bars[1].vaddr`) is sound at those offsets and at `0x80000`. A +high-frequency watcher (12 candidate offsets, ~2.5 Hz, including +`0x80000/0xa0000/.../0x4020000/0x4100000/0x4160000` and the +0x10000 +USERDs) recorded ZERO nonzero dumps across the whole live streaming window +of a clean Vulkan run (r786; three frames, `VKFRAME_RATE fps=0.05`, +`VKFRAME_DONE frames=3`, black PPMs). The `shadow_scan` over all three BAR +shadows (0/1/3) for the CE push header `{0x120,0xb5c50000,...}` and the +ring-entry pair `{0x20000000,0x2801}` finds nothing, and a byte search of +the full 4 GB guest-RAM image for both patterns (little-endian, corrected +from an earlier reversed search) finds nothing either. The walk and the +learned map resolve the CE ring VA (`0x4100000`) to a guest-RAM page +(per-boot PA, e.g. `0x135726000`) that reads all zero, and the vidheap +ioctl log (`PREV149-OUT`) names the channel allocations +`0x4020000/0x4100000/0x4160000` on every boot - so the ring content is +written neither to the shadow at its allocation offsets nor anywhere in +RAM reachable through the guest's own page tables. + +*A stale-block trap explains every historical "latch".* On CUDA-first +boots the capture latched `fb 0x80000 / userd 0x90000` for the Vulkan +channel `beef0101` with `put=1 get=1` - the dead CUDA client's `caf00012` +block surviving in VRAM after its session. The decoded entry +(`va=0x120000000 len=40`) is the old session's (both clients' first CE +pushes are shape-identical), the expectations seed from the stale pool +words (`want=2/3` vs `pay=1`; the word page read 3-6), and the Vulkan +client stalls into its fence timeout (`VKFAIL vkWaitForFences -> 2`) on +every run. r763's and r781's "auto-pinned ring h=0xbeef0101 -> fb 0x80000" +rows are the same artefact: both were CUDA-first boots. All genuine +first-submission latches on record read `put=1 get=0` at the capture +instant. `c3cdcf4b` rejects a consumed-state candidate (`get == put > 0`) +in the first-submission capture - a just-submitted entry cannot be +consumed at the notifier-clear instant. + +*The frontier is now precise.* The Vulkan client's CE submissions complete +only through the fabricated completion machinery (the driver never reads +its ring back), its CE ring content is invisible to every host source, and +the GR launch for each frame is held on the unexecuted CE i2m copies +(`copy pend src=0xdff0d60000 dst=0x50b0000`; walk: src -> vidmem +`0x570000`, zero). Next: the client-side write watch - find the CPU +pointer the guest driver writes GPFIFO entries through, its PA via the +guest pagemap, and why neither the GPU page table target nor any shadow +holds the result (guest-kernel gdb via the QEMU gdbstub or a kprobe on +the driver's ring-write helper). + +## 16. UPDATE 2026-10-26 (SESSION 137): THE CE-RING HYPOTHESIS FOR THE VULKAN CLIENT IS DISPROVEN BY DIRECT WRITE WATCHES - no kernel and no client writer ever touches the ring page + +**The write-watch toolchain (all three instrument classes).** (1) kprobes: +`channelFillGpFifo` (the kernel's GPFIFO-entry writer, called from the CE +and SEC2 submission paths per the open driver sources) is marked +`notrace` - `trace_kprobe: Could not probe notrace function +channelFillGpFifo [nvidia]` - so kprobes and ftrace can never attach +(this is why the named helpers "never fired" in earlier sessions: they +could never register). (2) QEMU-gdbstub hardware breakpoints work: a +verification `hbreak *__x64_sys_ioctl` fired on guest ioctls +(`IOCTL#1 pc=0xffffffffbc316160`). An `hbreak` on the exact entry store +(`channelFillGpFifo+0xb5`, `mov %edi,(%rdx)`, where `%rdx = +pbCpuVA + channelPbSize + lastSubmittedEntry*8` per the disassembly and +the source) stayed silent across a complete Vulkan run - even though that +same run produced the CE channel's notifier clear +(`sema-chan: notifier cleared h=0xbeef0101 info32=0x5`) and the +first-submission capture (`sema-cefirst: h=0xbeef0101 rank=1 ... +scan_hits=0 latch=0`). (3) A client-side hardware watchpoint: the ring +VA resolves through the guest page tables to a guest-RAM page that is +mapped into the client process (`/dev/nvidiactl` mapping; e.g. PA +`0x101021000` -> client VA `0x7fffee69b000`; a later run PA +`0x10087b000` -> VA `0x7fffee7d8000`), and the walk and the emulation's +own read agree the page is ZERO. An in-client gdb (the t122 recipe plus +a file-fed watchpoint) armed `watch *(unsigned long*)0x7fffee7d8000` +from the client's FIRST ioctl (`Hardware watchpoint 2` + `RGWATCH-SET` +logged before `GLBASE`) caught ZERO writes across the whole run +(3 frames, `VKFRAME_DONE`, IDLE). + +**Consequences.** On vk-first boots the client creates its CE channel but +never submits CE work: no GPFIFO entry is written by the kernel +(`channelFillGpFifo` never executes), nor by the client (hardware watch, +zero hits), nor is any content found in any BAR shadow or in RAM. The +historical "Vulkan CE ring latches" were the dead CUDA client's stale +blocks (section 15). The image/upload path for the Vulkan client rides +its GR channels (`launch captured ... inline=8`; the frame copy held as +`sema-gr: copy pend ... src=0xdff0d60000 dst=0x50b0000 rows=480`), and +the non-black gate must be attacked on that path - the data/code pages +those launches depend on are never filled because no CE i2m copies are +ever queued, and the launch is held on their absence (dstpend/n_data). +Next: instrument the GR launch's dependency resolution (which pages the +dstpend gate waits for, and whether the inline records the capture holds +are sufficient to fill them) instead of hunting a CE ring that nothing +writes. + +## 17. UPDATE 2026-10-26 (SESSION 138): THE GR LAUNCH'S WIRING IS PRINTED - one QMD-window page carries prog+pointer data (deduped and never scanned), its only fill candidate is an unresolvable inline word, and the backend refuses `n_data=0 (min 16)` + +**The witness (commit `10b09217`, sema_dbg-gated).** Per launch, bounded: +the candidate handle, `param_va`, `prog_va`, `n_pointer` + QMD pointer +pairs, inline count, and the resolved `gr_pages` list; then the fill's +built candidate set with the arg flag. + +**First capture (r788, Vulkan `h=0xbeec0100`, push=0x4f4509c).** +``` +sema-gr: cand h=0xbeec0100 param=0x4b86000 prog=0xdff0feb900 n_ptr=1 inline=8 n_pages=2 push=0x4f4509c +sema-gr: ptr[0] va=0xdff0febc00 word=34 +sema-gr: page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 +sema-gr: page[1] va=0xdff0feb000 pa=0x56b000 ap=2 qmd_word=48 +sema-gr: fill cand n=1 pages=2 push=0x4f4509c +sema-gr: win[0] va=0x43f00000 arg=1 +``` +Reading: the launch's page list is the param page and the +prog/QMD-window page (both resolving as `ap=2`, i.e. vidmem). +`ptr[0]` (QMD word 34, `0xdff0febc00`) shares its 4K page with the program +page, so `svc_gr_page_note` dedupes it into the `qmd_word=48` entry - and +the fill's scan passes cover only `qmd_word==32` and `32..47 even`, so +that page's content is never scanned for data pointers. The fill's single +candidate comes from the inline bytes instead (`0x43f00000`, arg=1) and +resolves in **no** vaspace (`svc-gr arg-miss base=0x43f00000 probe=none +... vas[0..3] rc=-2`), so `n_data` stays 0 and the backend refuses: +`vgsp-gr: GR submit refused: degenerate data set n_data=0 (min 16) +launch_va=0x4f4509c`. + +**Content state.** The shadow at the param page (`0x386000`), the arg +region (`0x386150`), the program area (`0x56b900`) and the pointer/data +area (`0x56bc00/0x56bd00`) reads all zero on this boot - the launch's +data and code bytes exist in no host-visible source (consistent with +section 16: nothing queues the CE i2m copies for the Vulkan client). + +**Next.** (1) Scan the deduped QMD-window page (or scan a page once under +every qmd_word that names it) so the pointer-region data words enter the +candidate set; (2) resolve the inline/dst candidates through the vidmem +serving lane (the pages ARE resolved to vidmem PAs - the missing piece is +the candidate list, not the resolver); (3) drive `n_data` toward the +launch's `min 16` (the window's expected page count) - and where the +window content must come from is the open question the byte-state above +poses. + +## 18. UPDATE 2026-10-26 (SESSION 139): THE FILL'S CANDIDATE SOURCE IS DISPROVEN - `0x43f00000` IS THE FLOAT 480.0f; after the float-shape filter the candidate set is EMPTY and the wall is content, not wiring + +**The false candidate (commit `add22329`).** A full dump of the launch's +param page (vidmem `0x386000`, r789) holds exactly two nonzero spots: +`+0x30: 08 00 00 00` and `+0x630: cd cc 4c 3e | 00 00 20 44 | 00 00 f0 +43 | 00 00 00 00` - i.e. the f32 scalars **0.2f, 640.0f, 480.0f** (the +640x480 viewport!). `0x43f00000` is 480.0f: the fill's only candidate +("arg base=0x43f00000 resolved 0 pages") has been a **float scalar +misread as a device VA** by the scan's shape filter (`>= 0x10000000`, +4-aligned). The GRCE inline write that does fire (`dst=0x4b86020 -> +pa=0x386020 bytes=1568 data8=0000000000000000`) writes the driver's +zero stub (the push carries only an 8-byte nonzero inline fragment at +`+0xb0`; the rest reads zero - the SESSION 130 zero-buffer finding). + +**The fix and verification.** All four scan sites (the three page scans +plus `svc_gr_scan_cand`) now reject float-shaped words (f32 exponent byte +in ~`[0x6c..0x8f]` = 2^-19..2^16; device VAs as f32 land in subnormal +exponents), and the bank-page scan covers word 48 (the deduped program +page). r790: **`sema-gr: fill cand n=0`** - no false candidates, no false +demand-loop traffic; the launch still refuses (`GR submit refused: +degenerate data set n_data=0 (min 16)`). + +**The wall, precisely.** The launch's real data pointers are not in any +scanned source: the param page holds scalars; the QMD pointer word +(`0xdff0febc00`) is one window buffer whose content is never written; the +inline payload is the zero stub. The content could only arrive via the +driver's i2m/LOAD_INLINE_DATA copies (which the Vulkan client never +queues - SESSION 137) or from the real GPU (physical forward, which the +n_data gate refuses first). This is now a pure content-source problem: +either find/queue the i2m that fills the QMD window, or break the +chicken-and-egg by forwarding a launch with the window zero (letting the +real engine read it) behind an explicit parameter. + +## 19. UPDATE 2026-10-26 (SESSION 140): THE n_data WIRING - the QMD's own pointer values ARE the launch's data set; the fill now reaches them (candidates appear, the page is correctly launch-internal), and the degenerate-set guard's bypass reaches the backend only to oops in the empty-set walk + +**Three wiring corrections (commit `5c0fcc21`).** (1) The QMD page itself +is noted into the request page list (word 34) - the pointer values live +inside the QMD (r790: word 34 = `0xdff0febc00`) and no scan ever covered +it. (2) The fill seeds candidates directly from the launch's own QMD +pointer pairs (`gr_qmd` words 34..47, decoded from the push's inline-QMD +run): r793 `sema-gr: qmd-ptr seed va=0xdff0feb000 (word 34)` - the values +need no page scan (the page shadows read empty; the +invisible-vidmem-write class). (3) The scans read pin-aware +(`svc_read_va` first) - the staging copy the decode reads is a pinned +page; a raw mapping of the vidmem pa sees zeros. + +**Result.** `fill cand n=1` with the QMD window page seeded; +`sema_probe 0xdff0feb000 -> pa=0x56b000 aperture=vidmem` resolves - but +the R3 hygiene filter classifies the page launch-internal, so `n_data` +stays 0: **this launch's QMD declares exactly one pointer pair and it +names its own QMD window - a launch with no external data, which the +"degenerate set" refusal class wrongly covers.** + +**The egg-break test + the oops.** With the r564 guard bypassed +(`gsp_phys_gr_min_data=0` - and the value clamp removed for the test -) +the refusal disappears and the forward ENTERS the backend: +`Oops: 0000 [#2] PREEMPT SMP NOPTI`, `Workqueue: gsp-phys-work +svc_fwd_work_fn [vgsp]`, `RIP: gsp_phys_backend_guest_gr_submit+0x4394` +(NULL-base table walk: CR2=0x1e93c, R14=0, RAX=0x1e000, RSI=0xdff0feb900 +- the program VA). The empty-data walk needs that guard before +`min_data=0` can be used; the clamp revert was withdrawn (crash +enabler). The copy pipeline is unaffected: `sema-gr: copy done +h=0xbeec0100 push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0` +runs cleanly in the same runs. + +**Next.** Guard the empty-set walk at `gsp_phys_backend_guest_gr_submit` +(the struct walk reading 0x04/0x10/0x38/0x3c off a NULL base with the +program VA in RSI), then re-test `min_data=0` for the no-external-data +launch class: the forward should reach the real engine, and the copy-out +(and the frame copy that already executes) can carry rendered pixels +back. + +## 20. UPDATE 2026-10-26 (SESSION 143): THE r800 GUARD HOLDS AND THE `min_data=0` BYPASS REACHES THE REAL ENGINE - IT FETCHES THE PROGRAM AND MMU-FAULTS WITH `FAULT_PDE` AT THE PROGRAM VA; THE WALL IS NOW THE PROGRAM PAGE'S MAPPING AND CONTENT, NOT THE DATA-SET GATE + +**The test (r801b).** `gsp_phys_gr_min_data=0` is now an explicit bypass +(commit `34e611f4`; 0 previously clamped to 1). On a clean cycle +(module from `34e611f4`, full params, `sema_gr_xlate=1`, +`sema_gr_util_scan=0`, `sema_vas_cand_ms=5000`, `sema_ce_grace_ms=20000`, +`sema_ce_hold_ms=0`, `sema_ops_log=8`, `sema_dbg=Y`, `min_data=0`) the +Vulkan client ran once: + +* **No oops.** `grep -c Oops` = 0 for the whole run; the r794 + `gsp_phys_backend_guest_gr_submit+0x4394` crash class is gone. +* **The refusal is gone.** Zero `GR submit refused: degenerate data set` + lines; the launch is submitted: + `GR LAUNCH submitted h=0xbeec0100 ... pages=3 shadow=1 + qmd_va=0x4b85f00 slot=3 words=239 ... qmdwin=0x108000`. +* **The engine ran and fetched.** After ~2.5 s: + `GR LAUNCH FAILED: timeout after 2500 ms (qmd_rel=0x00000000 + want=0x600df00d ...)`, and the real firmware reported: + `vgsp-fault: func=0x1006 ... errString="Ch 00000001, intr 00000000. + MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_GCC faulted @ 0xdf_d8feb000. + Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ"`. +* **The copy-out still works:** `sema-gr: copy done h=0xbeec0100 + push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0`. +* **The client now fails earlier than the black-frame runs:** it reports + `VKDEVICE ok family=0` then `VKFAIL vkWaitForFences ... -> 2` and no + frame is saved (the service synthesizes a client-visible fault for the + failed forward, where the refusal path used to software-complete and + let three black frames through). + +**The launch wiring measured on this boot.** +``` +page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 (param) +page[1] va=0xdfd8feb000 pa=0x62b000 ap=2 qmd_word=48 (program) +page[2] va=0x4b85000 pa=0x385000 ap=2 qmd_word=34 (QMD window bank) +``` +The program page resolves in the emulated vidmem at `0x62b000`, whose +backing host page (`0x5e0504000`) reads ZERO; `CODEHASH ... h0=0 ... +SUSPECT (first 16B)` and `FAULT-CAP ... prog_w48=0xd8feb900 +prog_pa=0x5e0504000 code_src=guest-hpa off=0x900 code64=000...` confirm +the source is empty. The QMD deposit keeps the guest program VA +(`w48/49=d8feb900 000000df`); in the retry only the `qmd_word 34` page is +copied to the window-0 shadow (`qmd_word 34 page copied to window-0 +shadow VA=0x1e8000`), so the program pointer is left at the guest VA and +the engine's page tables have no PDE for it. The post-mortem program +target reads `slot=30 ... w0..15 = 0`. + +**Reading.** The data-set gate and the r800 guard are settled: the +no-external-data launch is legitimately forwarded and the engine's +channel/context are alive (it fetched the QMD from the host deposit and +issued a real MMU fault). The remaining wall is exactly two coupled +facts: +1. **Mapping** - the engine's page tables do not map the guest program VA + (`FAULT_PDE`), and the host keeps that VA because the code source is + an empty page (the shadow rewrite is only used for sources it can + trust). +2. **Content** - the program page is empty in every host-visible source. + The driver's program-upload path stages the ucode and copies it to the + code heap (the observed CE `LAUNCH_DMA`/inline shape); that copy has + not been observed executing for the Vulkan client. + +**Next.** (1) Find and execute the Vulkan client's program-upload copy +(the staging->code-heap CE copy) so the program page becomes non-zero - +the utility/GR sweeps are the instruments, and the code page becoming +non-zero is the milestone. (2) Then map/relocate the program VA for the +engine (or install the PDE) so the fetch succeeds. (3) Then non-black +frames -> Xorg/GZDOOM -> KVMFR/LG -> CUDA 15/15 + soak -> the 1.0.0 +video -> Windows. Until (1)/(2) land, keep `min_data` at its default 16: +the bypass forwards a launch the engine cannot complete, which is worse +for the client than the software-completion fallback. + +## 21. UPDATE 2026-10-26 (SESSION 143b): THE COPY-CLASS CHANNELS ARE REGISTERED, SWEPT AND NEVER PUSHED; NO CHANNEL CARRIES PROGRAM BYTES OR A LAUNCH_DMA; THE CODE PAGE NEVER FILLS - AND THE REFUSAL PATH NO LONGER PRODUCES THE BLACK-FRAME BASELINE + +**The instrument (commit `c5510744`).** The util sweep now logs one-shot +`sema-utilsweep: begin|skip|read=none` for every channel plus a 128-row +per-pass cap, so "absent from the log" can no longer be confused with +"absent from the channel table". + +**Channel visibility (r803, full params, `util_scan=64`, `min_data=16`).** +Ten channels in the table, all with gpfifo/entries, **all USERD +unresolved** at first sight: +``` +begin h=0xbeef0101 eng=11 gpfifo=0x4100000 entries=32768 userd=0 +begin h=0xbeef0100 eng=0 gpfifo=0x4020000 entries=32768 userd=0 +begin h=0xbeef0102 eng=12 gpfifo=0x4160000 entries=32768 userd=0 +begin h=0xbeee0101 eng=11 gpfifo=0x4500000 ... +begin h=0xbeee0102 eng=12 gpfifo=0x4560000 ... +begin h=0xbeed0101 eng=12 gpfifo=0x49c0000 ... +begin h=0xbeee0100 eng=0 gpfifo=0x44a0000 ... +begin h=0xbeec0104 eng=0 gpfifo=0x4d20000 ... +begin h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ... +``` +The Vulkan client's copy-class channels **0xbeef0101 (eng 11) and +0xbeef0102 (eng 12) are in the table and swept** - and their rings carry +nothing all boot: `read=none` on first sight, `swept=0` on every pass, no +`utildec`, no LID, no copy. The client posted one event on 0xbeef0101 at +registration (`post_event hSrc=0xbeef0101`) and never pushed. + +**What did decode.** All boot: 3 LID writes only - the driver's two zero +buffers (256 B -> vidmem 0x423f000, 12288 B -> 0x4448000, both +`zero=1`) and the launch's own param bank (1568 B -> 0x4b86020, +`zero=0`). No `sema-lda`/`sema-opadd` rows at all, no `copy-in` rows, no +`utildec` on the two CE channels. Global swept=1652, writes=3. + +**The code page never fills.** `0x62b000` (the launch's program page for +`prog=0xdfd8feb900`) and the rotating-pool page `0xc8b000` read ALL-ZERO +before the client, after it, and at a final re-dump; the whole 256 MB +aperture holds only 12 non-zero dwords boot-wide (the viewport scalars +`0x44200000`/`0x43f00000` at 0x386634/0x386638, the releases, the +baseline). Nothing anywhere writes program bytes. + +**Client outcome.** `VKDEVICE ok family=0` then +`VKFAIL vkWaitForFences ... -> 2`; no frames, no PPMs. The service +answered `GR-COMPLETE-SW` 10 347 times for the same refused push while +the client kept waiting. This is the same outcome as the r801b forward +test, and it differs from the earlier black-frame baseline +(`VKFRAME seq=1..3`, three black PPMs): on a refused/failed forward the +service authors a client-visible `SYNTH-FAULT` (commit `c6c152cc`, +`sema_refusal_fault=1` since r618), and the current completion/release +path does not get the client past its fence. `util_scan` is excluded as +the cause (r801b failed with it off). + +**Reading.** The Vulkan client *believes* it uploaded its compute shader +(it dispatches with a valid program VA), but no CPU write, no CE LID, no +CE copy and no GR i2m in any observed channel ever delivers the program +bytes to the emulated vidmem. The driver's upload path is a +`staging ring -> code heap` copy (CPU memcpy into a staging BO, then a +GPU copy to the code heap); the copy is not among the decoded pushes and +the staging BO is not visible in the emulated vidmem. Two separate +questions now stand: +1. **Delivery** - where the driver queues that copy (a CE channel it + never pushes, a GR subchannel/method shape the parser does not + decode, or a sysmem staging BO that a different read path would + expose). +2. **Completion regression** - restore the refusal-path software + completion (or gate the synthetic fault) so the client reaches the + fence-released black-frame baseline again; this is needed for every + downstream observation. + +**Next.** (a) Dump and decode the client's push method stream around the +upload (the `staging ring -> code heap` copy) rather than more ring +sweeps; the staging BO's CPU write is the most likely host-visible copy +of the SASS. (b) Instrument the completion path for one refused Vulkan +launch (release ordering vs the synth-fault) to restore the black-frame +baseline. (c) Only then resume the mapping work (the r801b `FAULT_PDE` at +the program VA). diff --git a/docs/04-implementation/05-graphics-execution-plan.md b/docs/04-implementation/05-graphics-execution-plan.md new file mode 100644 index 0000000..6089548 --- /dev/null +++ b/docs/04-implementation/05-graphics-execution-plan.md @@ -0,0 +1,721 @@ +# Graphics execution plan: real GL pixels through the emulated vGPU + +> **Status: OPEN (2026-10-28).** This is the implementation-ready plan for the +> **graphics execution layer**: the component that must execute the guest +> NVIDIA vGPU driver's OpenGL/3D command streams on the physical GPU so that +> glxgears and GZDoom's OpenGL renderer produce real pixels through the +> emulated mdev. Nothing here is implemented yet beyond the pieces cited; +> every statement carries a code location or a measured session finding. +> +> Evidence base: `docs/04-implementation/03-execution-frontier.md` (the CUDA +> compute chain, proven), `docs/04-implementation/04-real-render-walls.md` +> (§8-§21, all measured graphics walls), the SESSION 149 R12-R18 round log +> (`session-149-BRIEF.md`: the M1 C597 recognizer, M2 materialization, M3 +> SKEDCHECK clear, R17 real-SASS execution), and SESSION 144 addenda 9/10/16 +> (the X/GL wait anatomy). The vendored 570.144 wire protocol and the public +> class headers are the interface authority; no guest driver change is +> allowed. +> +> Hard rules carried from the project: **no other GPU** (virtio-gpu/virgl, +> VirtualGL, llvmpipe, any second device are banned for this work), **no +> fake/labelled software rendering** presented as GL, **host-only emulation** +> (guest runs stock `nvidia.ko`), and **no host reboot** (AGENTS.md hard +> rule; a reboot needs the handoff update plus explicit human approval). + +--- + +## 0. What "real pixels" means here + +The acceptance target is: a stock guest GL application's draw commands are +executed by the physical GPU's **fixed-function 3D pipeline** (vertex fetch, +primitive assembly, rasterization, fragment work, ROP), its render target is +readable by the guest, and that image reaches the capture path through the +emulated mdev. The pixels must not come from Mesa, llvmpipe, virgl, a +software rasterizer, or a host display round-trip. + +Two consequences drive every milestone below: + +1. **A compute launch is not rasterization.** The SESSION 149 M1/M3 work + proved the engine will execute a *synthesized* C5C0 compute launch built + from the GL stream and will retire it + (`poll +500ms qmd_rel=0x600df00d`, R17), but the GL client's draws need + the 3D pipeline and the client's own per-draw state; rasterization is + fixed-function GR (`session-149-BRIEF.md:288-291`). +2. **The GL client's own channel is the source of truth.** The C597 section + the emulation can capture today comes from a readable channel (the X + server side); the GL client's GPFIFO ring (`0xbeef0100` family, + `0xbfef0002` observed in SESSION 153) is where its context-init uploads + and per-frame draws live. Until that ring is readable, no approach can + derive a correct draw (`session-149-BRIEF.md:438-451`). + +--- + +## A. Gap list, ordered by dependency + +Each gap lists what exists (with the code location that implements it), what +is missing, and the measurable gate that closes it. A gap is closed only when +its gate has been met on a fresh cycle with evidence kept. + +### G-A Readability: the GL client's ring and pushes must read + +**Exists.** The pump resolves rings through an autopin ladder (notifier- +derived, USERD-derived, trusted walk, shadow scan, USERD-shape scan) with +dead-entry rejection: `kernel/gsp_sema_service.c:16534-16602` +(`svc_ring_entry_dead`, `svc_ring_walk_live`, `svc_gr_ring_candidate`), +`:17891`, `:17972`, `:18013`, `:18075`, `:18123`, `:18242`, `:18298`. The +failure is witnessed at `:17633` (`sema-chan: GR ring h=... RAM content scan +found no candidate in N slice(s)`). Push reads report through `sema-rdw` +(`:28622`) and `sema-pushdec` (`:28868-28882`). The guest pin stream's client +selection and windows are `tools/guest-diag/guest_pin_stream.py:1429-1475`, +`tools/guest-diag/pin_gr_stream.py:134-160`, and the supervisor already +defaults to the widened GL prefix set +(`tools/guest-diag/pin_supervisor.sh:70-83`: +`cdt,va,victory,Xorg,X,glxgears,gzdoom`). + +**Missing.** For the GL client channel the ring page is not found by content +at the scan instants ("GR ring h=0xbfef0002 rank=2 ... no candidate in 12 +slice(s) (va=0x1203cfff0)", SESSION 153 item 3). The client's allocations are +just above the old 64 MB capture bound in the Vulkan case +(`docs/04-implementation/04-real-render-walls.md` §14-§15), and the pump's +ring candidates are shape-based; a client ring written late (after the DMA +test window) is missed. The guest pagemap stream (`#pid` + `va,pa,n` lines, +`guest_pin_stream.py:8-12`) is the authoritative VA->PA source but was not +covering the GL client until SESSION 153; the supervisor default now does. + +**Gate G-A.** On a run with glxgears alive for 60 s and the standard X +recipe: (a) zero new `sema-chan: GR ring h=0x... found no candidate` rows for +the GL client's handle; (b) every client push logs `sema-pushdec ... src>0 +unreadable=0`; (c) at least one entry is consumed from the client ring +(pump advances `last_get`; `sema-chan: entry[...] h=0x... eng=0` with a sane +`va/len`); (d) the X channel (`0x1000013`) stays `plausible=1` (SESSION 153 +item 1). + +### G-B Capture: the client's C597 sections and per-frame draws must decode + +**Exists.** A gated recognizer (`sema_gl_xlate`, default 0, +`kernel/gsp_sema_service.c:6812-6815`) opens a section at a `SET_OBJECT` +class bind of `0xC597` (TURING_A) on the channel subchannel and collects +observed VA operands: program (`0x2014`/`0x2018` runs), +tex header/sampler pools (`0x1574`/`0x155c`), vertex substitute (`0x0f84`), +upload source/destination (`0x180`), and the LAUNCH_DMA/LOAD_INLINE_DATA +payload (`0x1b0`) — `:6945-7049`, `:11993-12032`. It synthesizes a 64-word +QMD and captures it through the unchanged GR candidate ring +(`:7060-7287`, `:6686`). The client's memory-writing uploads are applied by +a standalone subchannel-0 pair tracker (`:12045-12084`) and the section +walker (`:7194-7217`) through `svc_grce_inline_write` (`:8979-9060`). +Completion of the capture is gated by content trust: a plausible stream +admitted by `cap_trust` (`:28995`, `:29069-29080`), with hash-verified +rewind replays admitted (`:28513-28517`, `:29106-29119`); the flush guard +requires `seen/prog/param/rel/trusted` (`:7082-7084`). Witness lines: +`sema-gl: section captured ...` (`:7248`), `sema-gl: uploads applied=...` +(`:7213`), `sema-gl-lid: ...` (`:12073`), `sema-gl: M3 code src=...` +(`:7261`). + +**Missing.** +1. The recognizer's method vocabulary covers init-shaped operands only. The + per-draw 3D state and draw-issuing methods (vertex streams/index, render + target and ZS, viewport/scissor, blend/depth/ROP state, draw calls) are + not decoded (`svc_gl_field` handles six addresses, + `:6945-7049`; the C597 stream carries far more). +2. The GL program/constant delivery is a context-init upload on the client + channel and is lost today: R17 measured that no readable push ever writes + `prog=0x121201b00` (`session-149-BRIEF.md:276-285`). The synthesized + launch therefore only runs a 12-28 B fallback blob or an operator hex + program (`sema_gl_prog_hex`, `:6877-6913`) — a development crutch, not + pixels. +3. Capture must survive the "one-shot setup push rewritten in place" + class: re-captures come from hash-verified replays now + (`:29069-29080`), but the per-frame draws are separate entries and need + the same treatment per entry. + +**Gate G-B.** For ≥3 consecutive frames of a live GL client: `sema-gl: +section captured` on the **client's** handle with nonzero `prog/param/rel`, +one captured section per frame (bounded replays suppressed), and a decoded +draw record (new `sema-gl-draw` witness) whose vertex/primitive count is +consistent with the app (e.g. glxgears gear geometry). Program source +priority shows `trusted-prog` (real bytes) for the context-init upload. + +### G-C Execution semantics: the GL work must run on the real 3D engine + +This is section **B**; the gap gate is M-GL1/M-GL2/M-GL3. + +### G-D Memory and resource mapping: every operand must be reachable by the engine and readable by the guest + +**Exists.** The physical backend builds engine-side page tables and shadow +windows for launch data: Tier-1 identity install or Tier-2 shadow copy +(`kernel/gsp_phys_booter.c:19654-19724`, `:19734-19785`), a two-bank +window-0 shadow pool of 8 pages per bank (`:1174`, `:19406-19410`) with a +24-page shadow chunk (`:1317`), QMD-window relocation +(`gsp_phys_rmcall_gr_qmdwin`), and a verbatim-prelude replay path that +relocates the inline-QMD address words and data words +(`:16878-16921`). Data pages are copied in before the doorbell and copied +back after the retire wait (`:17693`, `:18243`, `:21966`, `:22242`). +`svc_grce_inline_write` writes upload bytes to vidmem shadow, guest RAM, or +the VA shadow with a once-only key (`kernel/gsp_sema_service.c:8979-9060`). + +**Missing.** +1. Address-word relocation is QMD-only (words 32-49 and the indirect-page + pointer fixup at `booter:19765-19779`). A 3D method stream names VAs at + class-specific offsets; each must be resolved to a shadowed page and the + word rewritten (or the page mapped at its guest VA). +2. The engine-visible page budget is small: 16 pages per request + (`kernel/gsp_phys_booter.h:355`), 8 inline records (`:356`), 8 pages per + win0 bank (`booter:1174`). A GL frame names program + constants + texture + pools + vertex stream + render target + ZS; the shadow pool and the + per-request page list must grow or pages must be grouped per buffer. +3. Render-target writeback: the copy-out tracks only the launch's own data + pages; a 3D draw writes the RT, which must be mapped into guest-visible + memory (BAR1 shadow or pinned guest RAM) and copied back on the + completion edge, exactly like the current bounce/copy-out but for the RT + and depth surfaces. +4. Program/constant bytes: R17's measurement stands — the program page is + empty in every host-visible source until the CE/LID delivery executes + (`docs/04-implementation/04-real-render-walls.md` §20-§21). + +**Gate G-D.** Engine fetch of the 3D program/params/vertex/RT pages with +`FAULT-PDE count=0` (`vgsp-m91: FAULT-CAP ... code_src=...`, +`booter:18997-18999`), and a measurable post-launch change in the emulated +vidmem at the RT VA (>0 changed dwords in the `dump_vidmem map` class, +`docs/04-implementation/04-real-render-walls.md` §11). + +### G-E Completion and sync: the guest must be released by real completions + +**Exists.** Engine completion is polled (`qmd_rel/ch_rel`, +`booter:20814-20825`), the launch's release0 can be redirected to the module +semaphore and relayed (`booter:19499-19520`), the hold edge and the +backend-outcome edge author the guest word (`kernel/gsp_sema_service.c: +21329-21345`, `:21298-21322`), and failures synthesize a client-visible +channel error (`SYNTH-FAULT`, `:21400-21448`, `GR-COMPLETE-SW`, `:21504`). +The pre-built ring continuation beyond GP_PUT is swept for releases +(`sema_gr_ahead`, `:26564-26566`, `svc_gr_ahead_push`, `:26809`). + +**Missing.** The guest GL client's waits are still satisfied by the give-up +ladder or by host pokes: the X server's present wait spins on the X channel +member/counter pair (S144 addendum 12; the mirror thread `sema_xw_mirror` +`kernel/main.c:3103-3129`), and the GL client's `PhaseWait` compares +`obj+0x558` against a word in the shared `memfd:/.nvidia_drv` (+0x8) that the +emulation never advances (SESSION 149 R3, `session-149-BRIEF.md:115-126`: +"one u32 write 3 → 128.8 FPS", but only via `/proc/<pid>/mem`). The correct +long-term fix is the one named in addendum 16b §C.4: **the emulation +completes the real syncs so the equality is satisfied, with no lying pokes**. + +**Gate G-E.** M-GL4 (below): ≥60 fps sustained with no forcing daemon, no +`sema_xw_mirror` target written, and the guest's waits exiting on a natural +completion edge. + +### G-F Present/frame path: the rendered image must land where the guest and the capture can read it + +**Exists.** Guest-side capture tools: `tools/guest-diag/sdl_cap.c` (SDL swap +hook → `glReadPixels` → GZF2 record stream, `:1-18`, `:188-229`), +`tools/guest-diag/fbcopy_pub.c` (BAR1 scanout copy → kvmfr publisher), +`tools/guest-diag/qga_fetch.py` (QGA file fetch), host decoder +`tools/lg/frame_reader.py` and the resample/encode scripts. + +**Missing.** On the emulated-vGPU stack the X screen is black for GL windows +(DRI3-present path; SESSION 152) and `xwd` hangs (R4/R5; no rendered pixels +exist anywhere). The present path (the C5B5 copy-class and 902D 2D parts of +the GL push, R13) is decoded for releases but not executed as a display +path, and the capture must come from the GL client's swap, not from X. + +**Gate G-F.** M-GL2 (glxgears non-black) and M-GL3 (GZDoom). + +### G-G Concurrency: two VMs, two mdevs + +**Exists (designed in).** The service thread iterates every open device each +pass (`kernel/gsp_sema_service.c:32119-32236`), with the explicit note that +"with two, both guests get a pump turn per iteration (hardware forwarding +stays serialized by the single-inflight slot)" (`:32161-32167`). Per-guest +sessions exist (`SVC_MAX_GUESTS = 8`, `:728-736`), and the guest-RAM ranges +are registered per device at boot (`/tmp/opencode/start_vm_keep.sh:210-213`). + +**Missing / to verify.** Single-inflight forwarding and holds +(`:19878`, `:25528-25545`); VA-keyed global tables that assume one guest's VAs +do not co-reside: the write/read ledger is keyed by VA page only and its own +comment says collisions "only occur ... across clients whose pages do not +co-reside in time" (`:7547-7559`), plus `svc_gl` (`:6863`), +`svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); +the win0 banks are module-global per submit epoch (`booter:19406-19410`); the +server-managed VAS relocation windows are a finite global set +(`SVC_GR_DATA_WINS_MAX 31`, `:6441`). SESSION 151 avoided a second mdev "to +avoid a second fragile RM init", so a second RM bootstrap on the same module +is unproven. + +**Gate G-G.** M-GL5 (below). + +--- + +## B. Execution-semantics approaches + +### (i) 3D method-stream replay on the real engine + +Forward the client's captured C597 method stream (class bind + 3D state + +upload/draw methods + in-stream releases) to the module's real GR channel +and let the fixed-function pipeline execute it. + +What must be mirrored/provided: + +| Requirement | Where it stands | +|---|---| +| Real execution channel/context | Mode 35 "real-client user lane" replays the 570.144 client's compute-context sequence and executes on V7 (`booter:924`, `:9022-9033`); the promote ladder builds MAIN/PM/PATCH and the golden image. | +| 3D class object (0xC597) | `GSP_PHYS_RMCALL_CLASS_GR_3D 0xc597` is defined (`booter:396`) and the alloc helper is generic (`gsp_phys_rmcall_alloc_gr_object_class`, `:8936-8964`); it is constructed on the golden lane in several modes (`:9035-9043`, `:9665`), but **not on V7** (mode 35 logs "the capture carries no 0xc597 on the user lane", `:9029-9033`) — a class object must be added on V7 and validated. | +| 3D context global buffers | Already promoted: BUNDLE_CB / PAGEPOOL / ATTRIBUTE_CB / RTV_CB_GLOBAL (`booter:531-534`, `:605-621`, golden9 sized buffers `:985-1008`). The engine context image therefore contains the 3D subcontext data. | +| Method block | `gsp_phys_rmcall_build_gr_launch` (`:16834-16947`) emits compute init + inline QMD + PCAS + report-sem; the verbatim path (`:16878-16921`) already replays a captured raw method window and rewrites its address/QMD words. A 3D mode adds `SET_OBJECT 0xC597` + the captured state/upload/draw methods instead of the compute prelude. Note the channel's GR subchannel is 1 (`:442`) and the compute init binds `0xC5C0` (`:16693-16698`); the 3D bind replaces it per section. | +| Operand pages | The page list is built from QMD words 32-49 (`:19678`, `:6673-6682`); it must be generalized to the recognized 3D operand VAs, with the shadow/alias machinery (`:19654-19785`) reused. | +| Completion/sync | Keep the stream's own releases and the module release0 redirect (`:19499-19520`); the hold/relay path already authors the guest word on the engine edge (`gsp_sema_service.c:21329-21345`). | + +Pros: authentic rasterization is done by the hardware pipeline; no state is +invented; the engine has already proven it fetches and executes forwarded +work on this lane (R17 `qmd_rel=0x600df00d`); the completion machinery is +unchanged. + +Cons/risks: the promoted context may still be compute-shaped (class object +construction on a wired golden channel deadlocked in earlier experiments, +`booter:816-822`; V7 is the tested path, so construct there); the 3D method +vocabulary is large (state validation can silently fault); address-word +relocation must cover every class encoding; ~6-12 operand pages per frame +exceed today's 8-page win0 bank, requiring grouping/expansion. + +Effort: L. Depends on G-A, G-B. + +### (ii) Synthesize a "graphics-shaped" launch from the captured C597 stream + +On Turing, a QMD is a *compute* launch descriptor; there is no QMD that +rasterizes. "Graphics-shaped" therefore means: derive the draw inputs and +author the state + a DRAW method ourselves — program (`0x2014/0x2018`, +already decoded), constant buffers (bind + LID uploads, partly decoded), +vertex buffers (SET_VERTEX_STREAM_SUBSTITUTE is decoded only as a pointer, +`gsp_sema_service.c:6979-6983`; the real stream/index methods are not), +render targets/ZS (not decoded; the only observed target is the release +target `0x120001000`, R13), viewport/scissor/blend/depth state (not decoded). +The vertex count for the draw would be inferred, not read. + +Pros: fewer pages to mirror than a full replay if only draw-relevant state is +bound; still uses the hardware rasterizer. + +Cons: we author state the driver owns — any wrong field silently +mis-renders; the per-draw state lives in the client ring, so G-A/G-B are +still prerequisites; the program bytes are still lost without G-B; it is a +subset of (i)'s work plus extra invention. + +Effort: L. Risk: high. Not first. + +### (iii) Bounded compute translation for the subset GZDoom uses + +Replace each GL draw with a compute kernel (or the existing synthesized +C5C0 launch carrying the client's real SASS) that rasterizes triangles into +the RT. + +What breaks correctness, concretely: +- **Fixed-function raster and ROP**: depth test, stencil, alpha blending, + culling, clipping, perspective-correct interpolation, MSAA — none exist in + a compute launch; reimplementing them is writing a GPU. +- **Sampling**: mipmaps, LOD selection, anisotropic filtering, wrap modes + and sRGB conversion are texture-unit behaviour. +- **Shader semantics**: the client's GLSL is already compiled to SASS by the + guest driver; executing that SASS as a whole-SM compute program does not + give it vertex inputs, varying interpolation, or per-fragment coverage — + the very things the 3D pipeline supplies. +- **The claim itself**: even GZDoom's first frame uses FBOs + (scene → post-process), depth testing, alpha blending and mipmapped + textures; output would be a lookalike, not the renderer's pixels, and + could not be verified against a reference. + +Verdict: acceptable only as a **diagnostic probe** (e.g. a known-pattern +kernel proving the lane), never as the milestone path, and it must be +labelled as such. Not first. + +### Recommendation + +**Implement (i) first.** It is the only approach that yields authentic GL +pixels; its difficult prerequisites (G-A ring readability, G-B stream +capture) are shared by every alternative; and the backend already provides +the three pieces that make it tractable: a real execution lane with a +validated context (mode 35), a generic 0xC597 class-object allocator, and a +verbatim method-window replay with address/QMD rewriting plus page shadowing +and completion relay. (ii) is (i)'s draw-issuing step taken prematurely; +(iii) trades correctness for nothing. + +Order of implementation inside (i): **G-A** → **G-B** → replay the captured +*context-init* section (M-GL1, proves class bind + uploads + completion on +the 3D lane) → replay the client's per-frame *draw* sections (M-GL2/3). + +--- + +## C. Milestone ladder + +All gates are measured with the artifacts listed in section D; a milestone is +met only when its gate reproduces on **two consecutive fresh cycles** (three +for M-GL1) with the module hash recorded. + +### M-GL1 — Client context-init work decodes and executes; the client leaves its retry loop + +Gate (all in one run): +1. GL client ring resolved: no `sema-chan: GR ring h=<client> ... found no + candidate` row after the first minute; an autopin row exists + (`sema-chan: auto-pinned GR ring ...`). +2. Client pushes read: `sema-pushdec` rows for the client handle with + `src>0 unreadable=0 plausible=1`; no `sema-rdw ... rc=-2` for them. +3. Section captured from the client: `sema-gl: section captured h=<client>` + with `prog/param/rel` nonzero and `replay=0` (or the hash-verified replay + admission), at least once per client lifetime. +4. Forwarded and executed: `vgsp-gr: GR LAUNCH submitted h=<client> + ... pages>=2 shadow=1`, followed by engine evidence (`vgsp-gr: poll +... + qmd_rel=0x600df00d`) and either a natural release or `GR-COMPLETE-SW`. +5. Client behaviour: glxgears no longer loops on context creation (no + repeated init-failure retry; the X server stays alive; `xdpyinfo` OK, + `DMAFAIL=0`), and the app reaches its first swap. +6. Bounded failure hygiene: no SYNTH-FAULT storm for the accepted client + (≤ a handful in the window), no guest oops. + +Expected failure signatures (test one at a time): +`GR ring ... found no candidate` (G-A); +`unreadable=1 first8=00000000` (G-A, the S147 addendum 9 zero-page class); +`GR request without candidate` / `cand_valid=0` +(`gsp_sema_service.c:25479-25485`; G-B ordering: capture must precede the +request); +`GR submit refused: degenerate data set n_data=N (min M)` +(`booter:19352`; operand list, G-D); +`FAULT-PDE` at the program VA + `CODEHASH h0=0` +(`booter:18806-18883`, `:18997`; G-B/G-D); +`GR LAUNCH FAILED: timeout ... qmd_rel=...` (`booter:20825`; engine did not +retire — class/sched fields or completion wiring). + +### M-GL2 — glxgears renders non-black frames + +Gate: +1. A GZF2 record stream is captured from the **GL client's swap** (see D for + the GLX shim; the repo's `sdl_cap.c` is SDL-only — glxgears does not use + SDL). +2. ≥1 frame with ≥5% of pixels above luminance 8/255 and ≥2 distinct + dominant colours; two frames ≥500 ms apart differ in ≥1% of pixels + (motion, not a static noise tile). +3. The same time window contains the per-frame GR launch + completion in + dmesg, and the guest's `glxinfo` renderer is the NVIDIA driver (not Mesa). +4. The first frames are reached without any sync poke. If a diagnostic poke + is used to unblock the run, that run is labelled diagnostics and does not + satisfy the gate. + +Failure signatures: swap hook fires but frames all zero → no rasterization +(G-C); `xwd`/X path black → expected, do not use; frames show only clear +colour → draws executed but state/vertex data wrong (G-D); +`FAULT-CAP ... code_src=guest-hpa code64=000...` → program source lost +(G-B). + +### M-GL3 — GZDoom GL opens a window and presents non-black frames through the mdev + +Gate: +1. GZDoom (`vid_rendermode=4`, `vid_preferbackend=0`) runs on `DISPLAY=:0` + backed by the emulated vGPU (no Xvfb, no Mesa overrides). +2. ≥300 consecutive non-black GZF2 frames from the SDL swap hook, including + gameplay (not only the menu); the in-frame FPS counter ≥30. +3. dmesg shows one GR launch per rendered frame (bounded), no SYNTH-FAULT + loop; the X server stays alive for the whole run. +4. Capture fetched via QGA and decoded offline; a screenshot + (`+screenshot`) matches the captured frame within a small pixel delta. + +Failure signatures: window opens, swap fires, all frames black → draws not +executed; `GR submit refused ... n_data` → operand pages; app crashes with +`Illegal Instruction Encoding` (ESR 0x504730=0x9) → program bytes are not +SASS (G-B; cf. R18's single-page kernel lesson); `SYNTH-FAULT` per frame → +forward failed or refused. + +### M-GL4 — 60 fps sustained in ONE VM, no forcing + +Gate: +1. glxgears: "N frames in T seconds = X FPS" with X ≥ 60 over a 60 s + measurement (`vblank_mode=0 __GL_SYNC_TO_VBLANK=0` for the measurement + itself), and GZDoom's counter ≈60. +2. **No** poke/force/mirror daemon is running and no runtime module param is + used to fake progress: `sema_xw_mirror` is `0,0` (stopped, + `kernel/main.c:3103-3129`), no `/proc/<pid>/mem` write helper, no + strike/deadline forcing. +3. GZF2 inter-frame deltas: p95 ≤ 17 ms over 10 s (capture path keeps up). +4. The X present waits exit with member advancing naturally (Xorg log WAIT + lines absent or resolving without the kickfix daemon), and the glcore + equality word no longer requires the bump. +5. CUDA regression: `victory_mail` `VADD n=32768 bad=0` / `STATUS OK` on the + same boot after the GL run (battery-first ordering). + +Failure signature: the two ~11 s glcore waits return (the app's own strike +ladder; S144 addendum 16b) → G-E; X WAIT with static member → present +completion not authored; `GR-STALL-backend` per frame → completion is +software-only and the app's ladder is what "ran". + +### M-GL5 — Two VMs, two mdevs concurrently + +Verification checklist before the gate: +1. Second mdev creates and its guest RM bootstraps (the S151 open question: + "a second fragile RM init"); both devices appear in the service loop + (`gsp_sema_service.c:32119-32236`). +2. Per-guest isolation of the shared tables: `svc_ledger` VA-only key + (`:7547-7559`), `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`), + `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); either prove guest VAs do + not collide in practice or key them by `(vgpu, va)`. +3. Guest RAM: both VMs' memory-backend ranges registered and resolving + (the cycle registers one file per guest; verify the per-device map). +4. Server VAS relocation: no two guests' data windows alias in the managed + region (`SVC_GR_DATA_WINS_MAX 31`, `:6441`; comment `:6436-6439`). +5. Win0 banks/epochs (`booter:19406-19410`) are safe under serialized + submissions; single-inflight holds (`:19878`) do not starve one guest. + +Gate: both VMs run glxgears and GZDoom for ≥10 minutes; both present +non-black frames; each ≥30 fps (60 target); `victory_mail` green in both +afterwards; no cross-guest data anomalies (`sema-ledger: DIVERGE` / +`LEDGER-RESET` absent), no host oops, no guest wedge. + +Failure signatures: one VM starves the other's forward slot (`GR request +busy: hold active`, `:25509`); second guest's launches resolve the first +guest's pages (wrong results / ledger rewrite storms, `:7584-7594`); second +RM init fails in the boot ladder. + +### M-GL6 — Flawless 60 fps split-screen video + +Gate: 600+ frames at 60/1 fps, ≥95% unique frames, both halves show real +GZDoom GL gameplay through the two mdevs, encoded with the existing +resample/compose pipeline, uploaded with the files-upload skill and +pull-back-md5 verified. Encode with `libx264` (the R18 path did: +`/tmp/opencode/r18_final.sh`); NVENC is unavailable while the module owns +the physical GPU (the host NVIDIA driver is displaced). + +--- + +## D. Experiment recipes + +### D.0 Standard cycle (all milestones) + +Host, one controlled attempt per cycle (script leaves the VM running): + +``` +nohup /tmp/opencode/start_vm_keep.sh rNNN "<EXTRA_ARGS>" \ + > /tmp/opencode/rNNN-cycle.log 2>&1 & +# poll until: DONE (no teardown; VM left running) (~70-90 s) +``` + +`EXTRA_ARGS` = the SESSION 149 standard profile +(`session-149-BRIEF.md:65-67`), verbatim: + +``` +gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1 +gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1 +gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F +gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1 +gsp_phys_rmcall_gr_shadow_win0=1 gsp_phys_rmcall_gr_lmem_en=1 +gsp_phys_rmcall_gr_skedrec=1 gsp_phys_rmcall_gr_inst_sys=1 +gsp_phys_rmcall_gr_r1_first=1 gsp_phys_gr_strict_reloc=0 +gsp_phys_fault_relay=1 sema_bigcopy_log=512 +``` + +GL overlay (M-GL1..M-GL3 runs): append + +``` +sema_gl_xlate=1 sema_push_raw=4 gsp_phys_gr_min_data=0 +sema_gr_util_scan=0 sema_dbg=Y +``` + +(`sema_gl_xlate` is off by default in-tree, `gsp_sema_service.c:6812`; +`min_data=0` is the explicit no-external-data bypass validated in S143 and +used by R15-R17; `util_scan>0` is a measured regression source, keep 0; +`sema_dbg=Y` raises the bounded witnesses). The base script's ladder already +passes `gr_promote=25`; the later `=35` wins at insmod and is snapshotted by +the boot ladder — verify the boot log names mode 35 +(`booter:8612-8613`). + +Post-load, before Xorg: + +``` +sudo sh -c 'echo 1000 > /sys/module/vgsp/parameters/sema_vas_cand_ms' +sudo nohup dmesg -w > /tmp/opencode/dmesg_live_rNNN.log 2>&1 & +``` + +Guest Xorg (one per boot; ~50 % of boots fail DMA and need a re-cycle): + +``` +echo ubuntu | sudo -S setsid bash -c \ + 'nohup Xorg :0 -ac -noreset -logfile /tmp/xorg.log >/tmp/xorg.out 2>&1 </dev/null &' +# require: DISPLAY=:0 xdpyinfo -> 1280x720; grep -c "Failed to initialize DMA" /tmp/xorg.log == 0 +``` + +After Xorg is up: + +``` +sudo sh -c 'echo 1 > /sys/module/vgsp/parameters/sema_force_flush' +``` + +Evidence per run, always kept: module md5 (`md5sum kernel/vgsp.ko`), the +echoed insmod args, the full dmesg capture, guest `/tmp/xorg.log`, +`/tmp/glx*.log`, the GZF2 file and its md5, the decoded frame PPMs, and the +acceptance-grep output. + +### D.1 M-GL1 recipe + +Guest: start glxgears and watch the client channel: + +``` +DISPLAY=:0 setsid nohup stdbuf -oL glxgears -geometry 800x600+0+0 \ + > /tmp/glx1.log 2>&1 </dev/null & +``` + +Host acceptance greps on the dmesg capture (client handle filled from the +first `sema-chan` row): + +``` +grep -E "auto-pinned GR ring h=0x<client>" # G-A pass +grep -E "sema-pushdec: h=0x<client>.*unreadable=0" # G-A pass +grep -E "sema-gl: section captured h=0x<client>" # G-B pass +grep -E "GR LAUNCH submitted h=0x<client>" # capture->forward +grep -E "qmd_rel=0x600df00d|GR-COMPLETE-SW" # engine executed/completed +``` + +If G-A fails: confirm the guest scanner is running the widened prefixes +(`/tmp/guest_pin_stream.log`; supervisor env at +`tools/guest-diag/pin_supervisor.sh:70-83`), force a demand resolve via +`sema_trace_h=<client handle>` (S148 addendum 10) to see which source fails, +and use `echo <va> >> /tmp/guest_want_vas.txt` for a targeted pin. If G-B +fails with `entry_replay=1` keeping `cap_trust=0`, the pass is an un-hashed +replay: wait one fresh push or use the hash-admitted path +(`gsp_sema_service.c:29069-29080`) — do not add a bypass. + +Expected time-to-first-signal: minutes. Do not proceed to M-GL2 until 3 +cycles pass. + +### D.2 M-GL2 recipe (glxgears pixels) + +The repo shim is SDL-only (`sdl_cap.c:188-229` hooks `SDL_GL_SwapWindow`, +`SDL_RenderPresent`, `SDL_UpdateWindowSurface`); glxgears is GLX and calls +neither. Add a GLX mode to the same tool (hooks `glXSwapBuffers` → +`glReadPixels`, same GZF2 record layout), build: + +``` +gcc -shared -fPIC -O2 -o /tmp/glx_cap.so glx_cap.c -ldl +``` + +Guest: + +``` +DISPLAY=:0 GLXCAP=/dev/shm/glxframes.bin GZCAP_MAXF=0 \ + LD_PRELOAD=/tmp/glx_cap.so setsid nohup stdbuf -oL \ + glxgears -geometry 800x600+0+0 > /tmp/glx2.log 2>&1 </dev/null & +``` + +Fetch + decode on the host: + +``` +python3 tools/guest-diag/qga_fetch.py /dev/shm/glxframes.bin /tmp/opencode/glxframes.bin +# parse GZF2 records (tools/lg/frame_reader.py / gz_final_gpu.py's reader), +# then measure: % pixels > 8/255, colour count, motion between frames. +``` + +Do not use `vblank_mode=0` for the first pixel check (sync-on is fine); keep +the shim's swap hook ordered as "capture then forward" (same as sdl_cap). + +### D.3 M-GL3 recipe (GZDoom GL) + +Guest (xorg.conf untouched, the emulated GPU as the only GPU): + +``` +VK_ICD_FILENAMES=/nonexistent.json VK_DRIVER_FILES=/nonexistent.json \ +GZCAP=/dev/shm/gzframes.bin GZCAP_MAXF=0 LD_PRELOAD=/tmp/sdl_cap.so \ +DISPLAY=:0 setsid nohup ./gzdoom -iwad freedoom2.wad \ + +vid_rendermode 4 +vid_preferbackend 0 +vid_activeinbackground true \ + +sv_cheats 1 +noclip > /tmp/gz.log 2>&1 </dev/null & +# window management with tools/guest-diag/{xresize,xfocus,xtest_move}.c as in +# SESSION 151; XTEST alone is ignored by SDL here - use the XSendEvent path. +``` + +Host: `qga_fetch.py /dev/shm/gzframes.bin /tmp/opencode/gz.bin`, decode, then +`/tmp/opencode/gz_final_gpu.py` (resample to natural-speed 60 fps) for the +video. **Do not** set `LIBGL_ALWAYS_SOFTWARE` or the Mesa EGL vendor list: +that is the S151 software path and would invalidate the milestone. + +### D.4 M-GL4 recipe (60 fps, no forcing) + +Same as D.1/D.3, plus: + +- ensure `sema_xw_mirror` is stopped (`echo 0,0 > .../sema_xw_mirror`) and no + forcing/kickfix daemon is running (`pgrep -af` exact names first); +- measure 60 s windows only: glxgears FPS line, GZDoom counter, GZF2 + timestamps; +- keep `sema_gr_grace_ms`/`sema_ce_grace_ms` at their defaults; the R1 A/B + proved they are not the pacer (`session-149-BRIEF.md:96-99`); +- battery-first CUDA regression in the same boot. + +### D.5 M-GL5 recipe (two VMs) + +Second mdev + second VM (own disk copy, own sockets, own SSH/QGA ports), both +with the vGPU device this time. Verify in order: second mdev appears; second +guest's `nvidia-smi`/`glxinfo` sees the vGPU; service log shows both device +handles per pump pass (`sema-chan` rows for both); then run the D.1/D.3 apps +in both. Keep per-VM captures separate (two GZF2 files, two QGA sockets). + +### D.6 M-GL6 recipe (video) + +Per-VM SDL capture → `qga_fetch.py` → `/tmp/opencode/gz_final_gpu.py` +(timestamp resample + split-screen compose) → `ffmpeg -c:v libx264` (R18 +pattern) → files-upload skill → pull-back md5 verification. 1280x440 +split (640x440 per guest) matches the two delivered videos. + +--- + +## E. What must NOT be done + +1. **No other GPUs or virtual GPUs.** No virtio-gpu/virgl, no VirtualGL, no + llvmpipe/Mesa software GL, no second physical GPU, no host display + round-trip. The SESSION 152 video is explicitly *not* evidence for this + plan. +2. **No fake or mislabelled rendering.** No software rasterizer or compute + lookalike presented as GZDoom's OpenGL renderer. A compute-translation + probe (approach iii) may exist only as a labelled diagnostic and is never + an acceptance artifact. +3. **No lying sync pokes in accepted runs.** The `/proc/<pid>/mem` counter + bump, strike forcing, kickfix and `sema_xw_mirror` are diagnostics; any + milestone evidence must state they were absent. +4. **No host reboot.** AGENTS.md hard rule: handoff update first, explicit + human approval, never automatic. Guest VM cycles and Xorg restarts on the + test VM are the allowed recovery paths. +5. **No guest modifications.** Stock guest `nvidia.ko` and stock apps; no + injected code, patched firmware or modified driver in the guest; capture + shims are LD_PRELOAD observation only and must not alter GL semantics. +6. **No time/timing claims from unsupported sources.** The physical backend + is the execution authority; simulator (L0/L1) findings never substitute + for a live engine result. +7. **No unguarded experiments on the physical GPU.** Every cycle is + supervised, single-attempt, evidence-captured (the cycle script's hard + abort signatures are not bypassed). + +--- + +## F. Open questions and note-vs-code discrepancies + +1. **`sdl_cap.c` is SDL-only.** The milestone recipe in the SESSION 149 brief + names it for glxgears, but glxgears is GLX and never calls + `SDL_GL_SwapWindow` (`tools/guest-diag/sdl_cap.c:6-9`, `:188`). A GLX + variant is required (D.2); `sdl_cap.c` remains the right shim for GZDoom + (M-GL3). +2. **The GL-client pin prefix fix already looks landed.** + `tools/guest-diag/pin_supervisor.sh:73-83` defaults + `CLIENT_PREFIXES=cdt,va,victory,Xorg,X,glxgears,gzdoom` and the cycle + starts the supervisor (`start_vm_keep.sh:242`), while SESSION 153 still + lists the "permanent fix" as pending. Verify the running guest scanner + actually received the widened env (`/tmp/guest_pin_stream.log`) before + treating G-A as closed. +3. **The logged failure strings differ slightly from the briefs.** The code + prints `vgsp-work: GR request without candidate` + (`gsp_sema_service.c:25481`), not bare "GR request without candidate"; the + ring failure is `sema-chan: GR ring h=... found no candidate` + (`:17633`). Grep acceptance scripts for the exact prefixes. +4. **`n_data` threshold history.** The default is 16 + (`booter:17709-17717`); S143+ cycles used `min_data=0` as an explicit + bypass for the no-external-data launch class, enabled by the r800 guard. + The brief's "min 1"/"min 16" variants refer to different experiments; the + GL profile below uses 0 and must say so in every evidence record. +5. **Stale project status row.** `docs/README.md:114` still lists CUDA + execution as "no grid has dispatched yet", which predates the proven + physical forward and the R18 CUDA video. This plan relies on the newer + code and `docs/04-implementation/03-execution-frontier.md`, not that row. +6. **The 3D context on the execution lane is the main unknown.** Mode 35's + capture "carries no 0xc597 on the user lane" + (`booter:9022-9033`); golden-lane 3D objects exist (`:9035-9043`) but the + execution channel is V7. The first implementation experiment must prove a + 0xC597 class object can be constructed on V7 after the accepted promote + (same helper, `:8936`) without the wired-golden deadlock class + (`:816-822`), before any stream replay work.
\ No newline at end of file diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 2f10b35..1391426 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,26 +17,32 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` -**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes, -`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes -(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation, -allocation, copies, free and destroy complete, and the copies are verified -byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical -copy engine (default off) with engine-authored releases and a fault relay; for -GR kernel execution a forwarded guest launch now reaches the front-end but the -inline launch-descriptor install has not been observed (no grid dispatched; -context validity at dispatch is ruled out — -[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). -Before Rung 3 -is claimed, the -**reply-fidelity gate** applies: a control whose reply fields the driver -consumes is not implemented until the reply is coherent (status-only → -coherent zeroed params → coherent values for consumed fields → exact values) — -see -[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). -New to CUDA analysis here: start at +**Current rung (2026-10-30):** Rungs 1–3 are green for the verified CUDA +clients: the stock driver probes, `nvidia-smi` enumerates, the full CUDA +driver-API smoke passes (`ALL OK` 64/64), and guest kernels **execute on the +physical engine** — vector-add, matrix-multiply and transformer clients verify +on fresh boots with engine-authored completions relayed to the guest. What +remains before Rung 3 is claimed outright: the staged-transfer class (some +large multi-chunk clients still mis-verify; failure modes proven) and the +every-boot acceptance campaign — plus the **reply-fidelity gate** below. Rung 4 +(sustained) has partial evidence (a 100-launch loop, long soak pending); rung 5 +(multi-tenant) is partial: two mediated devices and two guests coexist, and +per-guest execution isolation plus fairness measurement are in progress. On the +graphics side, the X server/GLX run on the emulated vGPU, the engine executes +GL method streams, and the guest Vulkan pipeline runs end to end; rendered +(non-black) frames are the frontier +([`../04-implementation/05-graphics-execution-plan.md`](../04-implementation/05-graphics-execution-plan.md)). +**No performance numbers are published yet**; the A/B programme in §3 is the +next measurement step, and the first scheduled public measurement is the +graphics 60 fps target (no forcing). New to CUDA analysis here: start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). +The **reply-fidelity gate** applies to every emulated control: a control whose +reply fields the driver consumes is not implemented until the reply is +coherent (status-only → coherent zeroed params → coherent values for consumed +fields → exact values) — see +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). + ## 2. Correctness ### 2.1 Test corpus diff --git a/docs/05-operations/05-gpubreach-audit.md b/docs/05-operations/05-gpubreach-audit.md new file mode 100644 index 0000000..3c60385 --- /dev/null +++ b/docs/05-operations/05-gpubreach-audit.md @@ -0,0 +1,95 @@ +# GPUBreach-class audit and threat model (vGPU host module) + +Status: audit performed 2026-10-21 against the host module at `229bec2c` +(`vgsp.ko`, `kernel/`), with the write-path guards landed in the same window +(`7d4919dd`, `7a1f4805`). Scope: every host-side write whose target address +derives from guest-supplied data. Companion reading: +[`01-performance-correctness-security.md`](01-performance-correctness-security.md). + +## 1. What the GPUBreach threat is + +GPUBreach (IEEE S&P 2026, U. Toronto) is a GPU Rowhammer chain: the attacker +flips bits in the physical GPU's page tables (PTEs), which turns the GPU's +MMU into an arbitrary *device-memory* read/write primitive; the paper then +escalates to CPU root by exploiting memory-safety bugs in the *driver* code +that consumes those corrupted translations. It works with the IOMMU on, +because the corruption happens inside the GPU, after translation. + +The relevant question for this project is therefore not "can a guest read +device memory" (device memory is the guest's own resource here) but: + +1. can guest-influenced data reach a *host* write without validation, and +2. what bounds a write that lands on a *wrong* target. + +## 2. Structural mitigations (why the host is not on that path) + +* **Guest GPU page tables are host-emulated.** The guest's GMMU page tables + live in the emulated vidmem shadow (BAR1 backing) or in guest RAM; the + host resolves guest GPU VAs through its own walk of the guest's published + page directories (`SET_PAGE_DIRECTORY`) or through content-verified pins. + There is no physical GPU MMU walk of guest-authored tables. +* **The physical GPU's PTEs are host-owned.** Backend jobs are submitted by + the host module on host-authored PTE lines; the guest never supplies a + descriptor that the real hardware MMU consumes for host memory. +* **Every write is aperture-bounded.** The one write primitive, + `svc_write_ap()`, sends SYS writes through `vfio_dma_rw()` on the mdev's + own VFIO container - the IOMMU/vfio layer only maps the guest's RAM - and + FB writes are bounds-checked against the emulated shadow size. A + guest-influenced target can therefore only ever land inside the guest's + own RAM or the vidmem shadow; a wrong target is a correctness bug, not a + host-memory write. +* **No guest DMA to host buffers.** The guest's drivers talk to the + emulated device; they cannot address host kernel buffers. IOMMU is on + (vfio), and the mdev exposes only guest RAM plus the emulated windows. +* **Guest descriptors are not trusted by construction.** Every path that + turns a guest VA into a host PA goes through the resolution chain + (guest-verified injected pins -> write/read landing ledger -> published + page-directory walk -> targeted record scan), and the chain's outputs are + re-validated per page (see §3). + +## 3. Write-path inventory and validation state + +| Write path | Target derivation | Validation today | +|---|---|---| +| CE fill / memset (`svc_fill`) | guest VA or physical FB/GPA from the push | per-page destination resolution (pin -> ledger -> walk), page-boundary clamped; a page with no resolution truncates loudly and demands the VA through the pin channel (`FILL-TRUNC`, `stalled VA`); zero fills onto page-table-shaped guest pages are refused (`FILL-PT-REFUSED`); below-floor window fills that resolve outside vidmem are refused (`LOWFILL-REFUSED`) | +| CE copy (`svc_copy`) | dst/src VAs from the push | per-page resolution with pin-agreement (a fresh pin beats a stale ledger entry and repairs it); device-dst copies without a pin-resolved translation are held (`sema_copy_dst_pin_guard`) | +| GRCE inline writes (`LOAD_INLINE_DATA`) | destination VA from the push | once-only identity gate keyed by (push VA, push content hash, dst, bytes), session-scoped and 8192 slots wide | +| GR launch operand staging (`svc_gr_data_fill`) | argument pointers scanned from the QMD | per-page pin/ledger resolution; the resolution is stored in the ledger as the landing authority; dropped pages are named (`GRFILL-DROP`) and demanded | +| Launch copy-out (engine results back to the shadow/guest page) | the launch's own landing records | per-page destination resolution; every write is witnessed (`OUT-PAGE-WIT`/`OUT-PAGE-POST`); staged-identity rule (only the page staged into a slot may receive the slot's content) | +| Release/counter writes (semaphores, completion counters) | guest semaphore VAs from pushes | pool-range writes go through a pin/record-verified cache (`pool-pa`), re-validated against the current pin before the 4-byte write; exact-slot fallback for the client's tracker; forward-only apply | +| GPFIFO/USERD writes (put/get bookkeeping) | channel registration data | the channel's own USERD/instance blocks from the alloc params, aperture-tagged | +| Window/PTE installs (forward PDE install) | the launch's windows | installs are refused loudly when the backend rejects the PDE; the regular path leaves guest VAs in place (no host-side table mutation) | + +## 4. Residual risks and findings + +* **Wrong-but-valid writes remain possible** when every resolver agrees on a + stale page (e.g. a guest freed and reallocated a VA). The r391/r398/r401 + fixes removed the known instances (contiguous fill past the first page, + replayed inline writes, capture-window eviction), but the class is bounded, + not eliminated: the guards convert silent corruption into loud + truncation/refusal where a page is unresolved, and the ledger is + session-scoped and pin-repaired where a fresh pin exists. +* **The boot-corruption lottery** (a zero fill landing on guest page-table + pages) is structurally mitigated by the page-table-shape refusal and the + per-page clamping; no instance has been observed since r391. +* **Diagnostics stay on**: `DST-DIVERGE` (copy-dst ledger vs probe), + `FILL-TRUNC`, `GRFILL-DROP`, `NOWRITE-*` and the stalled-VA demand rows + exist so a future divergence is visible in the boot log rather than + silent. +* **What this audit does not claim**: no isolation or side-channel claim. + GPU Rowhammer itself is a *physical* fault-injection threat that this + software emulation cannot mitigate; the mitigation argument here is only + that the host write paths cannot be steered outside the guest's own + memory, and that wrong targets are bounded by IOMMU-mapped guest RAM or + the emulated shadow. + +## 5. Recommended follow-ups + +1. Keep the copy-destination ledger under the `DST-DIVERGE` witness and + promote it to a bounded hold only if a failing run names it. +2. Extend the pin-demand channel to client device VAs (today the in-guest + scanner resolves only its own push/table windows), so an unresolved + operand page becomes a wait on real data instead of a truncated fill. +3. Re-run this audit when the write-path guards change (any new + `svc_write_ap()` caller with a guest-derived target is in scope by + default). diff --git a/docs/05-operations/06-guest-frame-path-and-looking-glass.md b/docs/05-operations/06-guest-frame-path-and-looking-glass.md new file mode 100644 index 0000000..bc5447a --- /dev/null +++ b/docs/05-operations/06-guest-frame-path-and-looking-glass.md @@ -0,0 +1,102 @@ +# Guest frame path and Looking Glass status (2026-10-21) + +This page records the state of the "show the guest's GPU output on the host" +goal: what is proven, what is built, and exactly which pieces remain. It is +written from live evidence on the vGPU-IO stack (guest CUDA green, ivshmem +devices present, kvmfr module loaded on the host). + +## 1. What is proven + +* **Guest GPU rendering works through CUDA.** `tools/cuda-ab/frame_producer.cu` + renders an animated Julia set on the guest GPU (800x600 RGBA, ~10 fps), + copies the frame to host memory and publishes it into the ivshmem BAR + (`/sys/bus/pci/devices/0000:00:03.0/resource2`, the 128 MiB kvmfr device + QEMU backs with `/dev/shm/kvmfr-<uuid>`), with a small header + (`magic/version/width/height/stride/format/seq/timestamp`) at offset 1 MiB. +* **The host can read those frames.** `tools/lg/frame_reader.py` maps the same + shm file and writes each new sequence number out as a PPM, i.e. a real + GPU-rendered guest frame lands on the host. This is the full transport path + (guest GPU -> guest sysmem -> ivshmem -> host), independent of any display + protocol. +* **The ivshmem devices are visible in the guest** (`lspci`: two + "Inter-VM shared memory" functions; `resource2` of `0000:00:03.0` is the + 128 MiB window, `0000:00:04.0` the 2 MiB cursor window), and the guest can + mmap them (root required for the sysfs resource files). +* **EGL enumerates the NVIDIA device in the guest.** With + `EGL_EXT_platform_device`/`EGL_EXT_device_enumeration` present, + `eglQueryDevicesEXT` returns 3 devices and `eglInitialize` on the NVIDIA + device succeeds (`vendor=NVIDIA version=1.5`). + +## 2. What is blocked, and why + +* **Guest GL contexts do not allocate.** `eglMakeCurrent` on an offscreen + pbuffer fails with `EGL_BAD_ALLOC` (0x3003) on every platform tried + (device, surfaceless, default). Xorg's own failure is the same class: + `(EE) NVIDIA(GPU-0): Failed to allocate VASpace object` then + `Failing initialization of X screen`. The driver cannot allocate GPU virtual + address space for a GL context/surface through the vGPU, so both the X + display path and the EGL offscreen path stop at the same wall. This is the + single remaining blocker for guest OpenGL (and therefore for GZDOOM / + doomsday rendering in the guest). +* **The Looking Glass client (B4) cannot consume our frames directly.** + `looking-glass-client` (installed at `/usr/local/bin/looking-glass-client`, + runs under Xvfb with the `libbfd` shim in `/tmp/opencode/lgcompat`) supports + two transports, LGMP and SPICE. Its `app:shmFile` option still expects an + **LGMP session** (the `KVMFR` session header plus the shared-memory message + queues), not a raw framebuffer. Upstream's frame *producer* (`host/`) is the + Windows DXGI/IDD path. So a Linux guest-side producer must either implement + the LGMP session protocol or we keep the frame reader as the host-side + substitute (which is what `frame_reader.py` is). +* **GZDOOM is not packaged for Ubuntu noble** (`apt-cache policy gzdoom` has + no candidate); `doomsday` + `freedoom` are installed as the fallback, but + doomsday cannot start a renderer without a working GL context (blocker + above). + +## 3. Exact remaining pieces (in order of leverage) + +1. **Fix the guest GL VASpace allocation.** This is an RM/vGPU-side gap: the + driver's `nv_vgpu` VASpace allocation for a GL context fails. Fixing it + unlocks Xorg, EGL offscreen GL, doomsday/GZDOOM, and the standard Looking + Glass guest path (the guest's GL capture driver would then work as + upstream expects). +2. **Implement the LGMP session producer** (or a small Linux capture client) + that writes frames into the ivshmem region with the KVMFR session header + and the frame queue protocol the client expects. The transport half is + already proven; this is protocol work in the guest (or a host-side + substitute that speaks LGMP). +3. **Packaging**: GZDOOM would need a source build or a flatpak/snap; the + doomsday fallback is installed and only blocked by (1). + +## 4. Reproduce + +```bash +# guest (root, because the sysfs resource is root-only): +sudo /tmp/frame_producer 60 800 600 10 # 60 frames, 800x600, 10 fps + +# host: +python3 tools/lg/frame_reader.py --frames 5 --out /tmp/opencode/frames +# -> SAVED seq=N 800x600 fmt=0 -> /tmp/opencode/frames/frame-NNNN.ppm + +# guest EGL probe (shows the enumeration success and the MakeCurrent wall): +gcc -O2 -o egl_frame tools/guest-diag/egl_frame.c -lEGL -lGLESv2 +./egl_frame +``` + +## 5. Session-13 update (2026-10-21, r432→r441) + +* The single-frame path is unchanged and still proven: guest `frame_producer` + (CUDA Julia) -> kvmfr ivshmem BAR -> host `frame_reader.py` -> + `/tmp/opencode/frames/frame-0001.png` (320x240, 1431 colours). +* `tools/lg/frame_sequence.sh` captures an **animated** sequence end to end + (reader first, then N frames from the guest). It is blocked by the + GR no-execute wedge: when the lane is wedged the render launches write zero + pages and the producer republishes stale identical frames (observed: 6/6 + identical MD5s on r438; r439 published 5 identical plus one frame left in + the BAR by the previous boot). A multi-frame animation needs a healthy + lane; the frame pipeline itself is not the blocker. +* The GL wall is untouched: `Xorg :0` -> `(EE) NVIDIA(GPU-0): Failed to + allocate VASpace object`, guest dmesg `NVRM: vaspaceapiConstruct_IMPL: + Could not construct VA space. Status ffff`, i.e. `vmmCreateVaspace` inside + the guest's own RM. Next step: reproduce with the guest RM's + `ResmanDebugLevel` raised (a guest module reload, the service's m177 path) + to capture which inner call returns the generic error. diff --git a/docs/README.md b/docs/README.md index 13f0ae5..95ef85c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,7 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- ## 3. Where is the project today? -**Honest status (2026-10-02):** +**Honest status (2026-10-30):** A dated snapshot of this status, refreshed on every progress check, is kept at [`../STATE-CURRENT.md`](../STATE-CURRENT.md). @@ -111,28 +111,30 @@ A dated snapshot of this status, refreshed on every progress check, is kept at | GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | | Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | -| CUDA kernel execution | 🟡 A stock guest program now reaches its kernel launch and submits it through the forwarding path; the front-end fetches the ring entry and executes methods, but the inline launch-descriptor (QMD) install has not been observed and no grid has dispatched yet. Context validity at dispatch is ruled out; the frontier is the launch method-stream form. Copy-engine-class work, its releases and its fault relay remain forwarded (parameters default off). See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | -| OpenGL / Vulkan | 🔴 Not yet | -| Windows guest | 🔴 Not yet | -| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | - -We also know exactly where the CUDA bring-up stands and what the next step is: -the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes -the request parameters instead of returning zeros), the engine inventory is -complete, and the driver-API smoke test now completes end to end — `cuCtxCreate` -returns and the 64-byte payload round-trips exactly (64/64, re-verified -2026-10-02 on the current build). The full -mechanism, with offsets and the decompiled call path, is in +| CUDA kernel execution | ✅ Working — verified guest clients execute kernels on the physical GPU with engine-authored completions relayed to the guest: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK` on fresh boots, and `vecadd_loop` passes 100 launches. Remaining: the staged-transfer class (some large multi-chunk clients still mis-verify; failure modes proven), the every-boot acceptance campaign, and per-guest execution isolation. See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | +| OpenGL (3D graphics) | 🟡 In progress — the X server and GLX run on the emulated vGPU with the stock driver; the engine executes real GL method streams (verbatim 3D replay and synthesized launches); a direct GL client creates its channels and its readback path returns clean. Rendered (non-black) frames are the frontier. See [`04-implementation/05-graphics-execution-plan.md`](04-implementation/05-graphics-execution-plan.md) | +| Vulkan | 🟡 In progress — the guest Vulkan pipeline runs end to end on the emulated vGPU (device, allocation, pipeline, command buffer, fence, queue submit, readback); frames are still black — program delivery and program-VA mapping are the frontier | +| Windows guest | 🔴 Not yet (target) | +| Multi-tenant isolation (mdev per VM) | 🟡 Two mediated devices with different emulated VRAM sizes and two guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | + +Guest CUDA now executes on the physical GPU: the driver-API path completes end +to end, forwarded launches reach the engine, the grid dispatches, the engine +authors the completion and the host relays it, and the guest's synchronisation +waits resolve (verified with vector-add, matrix-multiply and transformer +clients on fresh boots). The mechanism, with offsets and the decompiled call +path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). -The remaining frontier is guest-facing execution. Copy-engine-class copies and -their releases are forwarded to real hardware with release relay and fault -relay in place (parameters default off). For GR (graphics/compute), the guest -now reaches and submits a real kernel launch: the ring entry is fetched and -front-end methods execute, but the inline launch-descriptor install has not -been observed and no grid has dispatched yet. The wall is the form of the -forwarded launch method stream — not context validity, which is ruled out at -dispatch. The chain, the evidence and the live investigation lines are in -[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md). +The remaining CUDA work is breadth and hardening: the staged-transfer class +(some large multi-chunk clients still mis-verify; the failure modes are +proven), the every-boot acceptance campaign, and per-guest execution isolation +for simultaneous guests. Graphics shares the same submission machinery and is +the other active frontier: the X server and GLX run on the emulated vGPU, the +engine executes GL method streams, and the guest Vulkan pipeline runs end to +end — rendered (non-black) pixels are the wall. The chain, the evidence and the +live investigation lines are in +[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) +and +[`04-implementation/05-graphics-execution-plan.md`](04-implementation/05-graphics-execution-plan.md). The backend, a module-owned GPU running its real GSP firmware as the resource-manager server, is specified in [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md); @@ -183,6 +185,8 @@ Read in order if you are new. Jump directly if you know what you need. | 23 | [GSP in a RISC-V VM (risc-v-gsp-rm-debugger, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | +| 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min | +| 25b | [Real-render walls: Vulkan and Xorg on the emulated vGPU](04-implementation/04-real-render-walls.md) | developers, debuggers | 10 min | | 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | | 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | | 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | diff --git a/kernel/gsp_cuda_bridge.h b/kernel/gsp_cuda_bridge.h index 2fb63d7..472fafb 100644 --- a/kernel/gsp_cuda_bridge.h +++ b/kernel/gsp_cuda_bridge.h @@ -15,7 +15,15 @@ struct nv_vgpu_device; * targeted resolve). */ #define GSP_BRIDGE_INLINE_MAX 16 -#define GSP_BRIDGE_INLINE_BYTES 64 +/* + * SESSION 153-CONTINUED-88: 256 -> 2048. The GL program delivery needs the + * driver's REAL captured upload payloads (observed: a 1536 B SPH+SASS image + * at 0x4350900 and a 1408 B image at 0x4350f00, w0=0x020a0481) to reach the + * engine-visible page; the 256 B cap truncated them. Every user of this + * record type grows with it (rings, bridge queues - all static storage; the + * only stack user, svc_gl_flush's prs[], was moved to static storage). + */ +#define GSP_BRIDGE_INLINE_BYTES 2048 struct gsp_bridge_inline { u64 dst_va; diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 0f7e91c..3d0aae2 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -39,12 +39,14 @@ #include <linux/seq_file.h> #include <linux/uaccess.h> #include <linux/jiffies.h> +#include <linux/ktime.h> #include <linux/timekeeping.h> #include <linux/byteorder/generic.h> #include <linux/pci_regs.h> #include <linux/vgaarb.h> #include <linux/acpi.h> #include <linux/mm.h> +#include <linux/vmalloc.h> #include <asm/barrier.h> #include <asm/unaligned.h> @@ -73,6 +75,150 @@ module_param_named(gsp_phys_nocat, gsp_phys_nocat_param, bool, 0644); MODULE_PARM_DESC(gsp_phys_nocat, "Log every GSP-RM NOCAT journal record (status-queue function 0x1020) individually during the stage-3e RM-init poll (default 0; the unique-record summary is always accumulated and dumped when the poll ends)"); +/* + * GRFIX FIX3 (relocation coverage, default 0): the QMD/bank rewrite scan + * counts 8-byte words that pass the rewrite's own device-VA shape test + * (>= 0x10000000, < 2^47, 256-byte aligned) but match NO tracked data page + * (req->data[x].va). Such a word is left as an unrelocated guest VA for the + * engine to walk, which is the multi-pointer QMD failure signature. With the + * parameter clear the scan only logs the per-page count (RELOC-GAP, + * rate-limited); with it set a launch carrying any such word is held + * fail-closed before publish (same backend-refusal path as the copy-in hold; + * the sema hold applies the software releases - bounded, never a hang). + * The hold is opt-in because the shape heuristic can false-positive on + * legitimate non-pointer data words; the RELOC-GAP count measures that rate. + */ +static bool gsp_phys_gr_strict_reloc_param; +module_param_named(gsp_phys_gr_strict_reloc, gsp_phys_gr_strict_reloc_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_strict_reloc, + "Hold a GR launch fail-closed when the QMD rewrite finds device-VA-shaped words that match no tracked data page (default 0: RELOC-GAP log only; 1: hold the launch)"); + +/* + * relfix (agent-A FIX A, default 1): the GR QMD rewrite's device-VA-shaped + * words that match NO tracked data page (the RELOC-GAP class - the raw guest + * pointers the real engine then faults on; r216: MMU FAULT_PDE at a gap VA, + * SM warp exceptions) are re-resolved through the sema service's trusted + * resolver (gsp_sema_gr_reloc_resolve). A word that resolves is authored + * into the bounce window exactly like a late-tracked data page - it and the + * contiguous span the resolver also answers are copied in before the + * doorbell and copied back after the retire wait, the same m142 ordering as + * every tracked page - and the word is rewritten to the page's reloc VA plus + * its in-page offset, the exact encoding the matched-word relocation uses. + * Unresolved words keep the historical RELOC-GAP behaviour (counted and left + * as guest VAs; never guessed). 0 = exact pre-relfix behaviour for one-flip + * A/B. The rewrite runs on the single-inflight GR work item with + * gsp_phys_rm_lock held; the resolver is read-only w.r.t. the service tables + * but may sleep in the same vfio walks the fill's resolver uses. + */ +static unsigned int gsp_phys_gr_reloc_resolve_param = 1; +module_param_named(gsp_phys_gr_reloc_resolve, gsp_phys_gr_reloc_resolve_param, + uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_reloc_resolve, + "Resolve QMD words that match no tracked data page and author them as late data pages (default 1: RELOC-FIXED; 0: RELOC-GAP only)"); + +/* + * COPYWIT (2026-10-18 fail4 discrimination tooling, default 1): witness the + * copy-out of every tracked page the engine changed - device VA, guest HPA + * the results were written back to, and the first/last 8 bytes actually + * shipped. Correlating these lines with the sema-bigcopy D2H resolution lines + * (same VA -> pa) and the client's own check discriminates "engine result + * never produced" from "produced but landed on a different guest page than + * the client's read resolves" - the two classes the all-bad client results + * cannot separate from the aggregate counters alone. Bounded per launch + * (GSP_PHYS_GR_COPYWIT_MAX) and rate-limited; 0 disables both this and the + * per-launch RELOC-CENSUS line. + */ +#define GSP_PHYS_GR_COPYWIT_MAX 12u +static bool gsp_phys_gr_copyout_dump_param = true; +module_param_named(gsp_phys_gr_copyout_dump, gsp_phys_gr_copyout_dump_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_copyout_dump, + "Log per-changed-page copy-out witnesses (VA, HPA, first/last 8 bytes) and the per-launch relocation census (default 1)"); + +static inline bool gsp_phys_gr_copywit_enabled(void) +{ + return READ_ONCE(gsp_phys_gr_copyout_dump_param); +} + + +/* + * r405 late-write probes (the r405 "partial write" class: the scale launch + * of streams2 wrote 1 of its 32 expected in-place pages and every other + * launch settled with far fewer changed pages than the kernel should + * produce). Two knobs, both runtime-settable: + * + * - gsp_phys_gr_late_probe (default 0): after the retire wait concludes, + * sleep +500 ms and re-scan the tracked pages. Run-once diagnostic: + * "ENGINE STILL WRITING" proves the copy-out shipped before the stores + * landed (the fix is a longer settle window); "stable" proves the engine + * genuinely stopped, which needs a different treatment. + * - gsp_phys_gr_settle_ms (default 100): the content-stable window the + * partial-changed rule requires before it declares the stores settled. + * Raise it to give a still-writing engine its time (A/B in one cycle). + * + * r552 (2026-10-23): measured A/B on real boots. Every launch with a + * partial changed-page count (read-mostly kernels: 1..64 of 1000+ pages + * written, so the changed>=n_all shortcut can never fire) paid the full + * window plus scan overhead: 683-901 ms per launch at 300 ms. At 100 ms + * the same boots measured 231-247 ms of retire wait and the launch + * latency p50 dropped 1119 ms -> 651 ms (n=22 launches; baseline 684 ms) + * with the battery unchanged (12/15, 0 hangs) and a green 2-round soak + * on the next boot. The content-stability hash still gates the verdict; + * only the minimum stable duration shrank. + */ +static bool gsp_phys_gr_late_probe_param; +module_param_named(gsp_phys_gr_late_probe, gsp_phys_gr_late_probe_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_late_probe, + "Diagnostic: after the GR retire wait, re-scan the tracked pages +500 ms later and log whether the engine is still writing (default 0)"); + +static int gsp_phys_gr_settle_ms_param = 100; +module_param_named(gsp_phys_gr_settle_ms, gsp_phys_gr_settle_ms_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_settle_ms, + "GR retire wait: content-stable window (ms) required before a partial changed-page count counts as settled (default 100; 300 was the pre-r552 value)"); + +static inline u32 gsp_phys_gr_settle_iters(void) +{ + int ms = READ_ONCE(gsp_phys_gr_settle_ms_param); + + if (ms < 60) + ms = 60; + if (ms > 1400) + ms = 1400; + return (u32)(ms / 5); +} + +/* + * COPYWIT fail4 r224 follow-up: the r224 OUT-PAGE-WIT dumps proved the + * engine's outputs are structurally corrupted (row bands >= 16 replicate a + * single tile per buffer) while the copy-out write/read pairs agree + * byte-for-byte. Two witnesses make the next cycle name the source: + * + * - OUT-ALIAS-WIT: during copy-out, every changed page's first 8 bytes are + * matched against the earlier changed pages of the SAME launch; a hit + * logs the (va, bi) pair, i.e. the replicated-page structure and the rec + * partners - the direct evidence for "one tile replicated across the + * band" and against "pass-through of pre-launch content" (the pre/post + * csums in OUT-PAGE-WIT prove the engine changed the page). + * - IN-ALIAS-WIT: during the copy-in fill, consecutive recs with identical + * content (equal per-page csum) are logged - a replicated INPUT buffer + * (e.g. an X or weight region whose H2D/copy-in served one page many + * times) is caught here before any launch runs. + * Bounded boot-wide and per launch; same single-inflight static discipline + * as the other submit scratch. + */ +#define GSP_PHYS_GR_OUTWIT_SLOTS 16u +static u64 gsp_phys_gr_outwit_first8[GSP_PHYS_GR_OUTWIT_SLOTS]; +static u64 gsp_phys_gr_outwit_va[GSP_PHYS_GR_OUTWIT_SLOTS]; +static u32 gsp_phys_gr_outwit_bi[GSP_PHYS_GR_OUTWIT_SLOTS]; +static u32 gsp_phys_gr_outwit_n; +static u32 gsp_phys_gr_alias_log; /* boot-wide alias-line budget */ +static u32 gsp_phys_gr_recdump_budget; /* r233 REC-DUMP boot budget */ +static u32 gsp_phys_gr_rdw_budget; /* r234 RDW arming-line budget */ +static u32 gsp_phys_gr_atrace_budget; /* r234 A-TRACE boot budget */ + bool gsp_phys_boot_enabled(void) { return gsp_phys_boot_param; @@ -986,18 +1132,61 @@ bool gsp_phys_boot_requested(void) * from gpFifoOffset, the pushbuffer from the GPFIFO ring entry GET, the CE * operands from the method block): SET_SEMAPHORE_A/B are always GPU VAs with * no mode bit (clc5b5.h:39-44; uvm_pascal_ce.c:42-63) and the user channel's - * copy is virtual-mode on both sides (uvm_maxwell_ce.c:179-205). All five - * pages live in one 2 MB PD0 slot below the VAS vaSize limit: with + * copy is virtual-mode on both sides (uvm_maxwell_ce.c:179-205). All the + * ladder pages live in one 2 MB PD0 slot below the VAS vaSize limit: with * VA_BASE = 0x00100000 every parent index (48:47, 46:38, 37:29, 28:21) is 0 - * and the five 4 KB pages take small-PT slots 0x100..0x104 - * (kern_gmmu_fmt_gp10x.c:16-83; uvm_pascal_mmu.c:25-32,172-204). + * and the 4 KB pages take small-PT slots in the single 128 B PTE line + * 0x100..0x10f (kern_gmmu_fmt_gp10x.c:16-83; uvm_pascal_mmu.c:25-32,172-204). + * + * SESSION 153-CONTINUED-66: the module pushbuffer is TWO pages. The MME + * context-setup prelude matured to 901 words while the beef window runs 390 + * words and the SA-RAM stubs add 29..47, so the single-launch block needs + * ~1400 words = 5.6 KB: the 1-page pushbuffer made every prelude-bearing + * launch print "MME-PRELUDE deferred (window=390 prelude=901 sa=47 exceeds + * block; replaying window alone)" and go out prelude=0. The block now spans + * VA_PB..VA_PB+0x2000 with the second page at small-PT slot 0x102. The + * slot 0x102 page used to be CE SRC; SRC moved to 0x109 (it is only read by + * the CE copy path and the GR inline-QMD destination default). Slot 0x10a + * is the dedicated warm-semaphore probe page: the probe used to share the + * pushbuffer's upper half, which a >0x800-byte forwarded block now occupies. */ #define GSP_PHYS_RMCALL_VA_BASE 0x00100000ULL #define GSP_PHYS_RMCALL_VA_GPFIFO GSP_PHYS_RMCALL_VA_BASE #define GSP_PHYS_RMCALL_VA_PB (GSP_PHYS_RMCALL_VA_BASE + GSP_PHYS_PAGE_SIZE) -#define GSP_PHYS_RMCALL_VA_SRC (GSP_PHYS_RMCALL_VA_BASE + 2 * GSP_PHYS_PAGE_SIZE) -#define GSP_PHYS_RMCALL_VA_DST (GSP_PHYS_RMCALL_VA_BASE + 3 * GSP_PHYS_PAGE_SIZE) -#define GSP_PHYS_RMCALL_VA_SEM (GSP_PHYS_RMCALL_VA_BASE + 4 * GSP_PHYS_PAGE_SIZE) +/* SESSION 153-CONTINUED-66: page 2 of the module pushbuffer (same PT line). */ +#define GSP_PHYS_RMCALL_VA_PB2 (GSP_PHYS_RMCALL_VA_BASE + 2 * GSP_PHYS_PAGE_SIZE) +/* + * SESSION 153-CONTINUED-77: page 3 of the module pushbuffer (slot 0x103) + * so a matured MME prelude (cap 2048 words) plus the largest chunked + * window fit one fetch block. The block walk is linear in VA, so the + * three pushbuffer VAs must stay contiguous: VA_DST (previously at + * base+0x3000, slot 0x103) moves to base+0xB000 (slot 0x10b, free). + */ +#define GSP_PHYS_RMCALL_VA_PB3 (GSP_PHYS_RMCALL_VA_BASE + 3 * GSP_PHYS_PAGE_SIZE) +/* + * SESSION 153-CONTINUED-77: page 4 (slot 0x104) after the r1163 cycle + * showed the capture cap-limited at 2048 words (ir_end=0x265, 34 slots + * covered). VA_SEM moves 0x104 -> 0x10c (free) so the four pushbuffer + * pages stay contiguous for the linear fetch. + */ +#define GSP_PHYS_RMCALL_VA_PB4 (GSP_PHYS_RMCALL_VA_BASE + 4 * GSP_PHYS_PAGE_SIZE) +#define GSP_PHYS_RMCALL_VA_DST (GSP_PHYS_RMCALL_VA_BASE + 0xB000ULL) +#define GSP_PHYS_RMCALL_VA_SEM (GSP_PHYS_RMCALL_VA_BASE + 0xC000ULL) +/* SESSION 153-CONTINUED-66: CE SRC moved off the pushbuffer's second page. */ +#define GSP_PHYS_RMCALL_VA_SRC (GSP_PHYS_RMCALL_VA_BASE + 0x9000ULL) +/* SESSION 153-CONTINUED-66: warm-semaphore probe page (own PTE, no sharing). */ +#define GSP_PHYS_RMCALL_VA_GR_PROBE (GSP_PHYS_RMCALL_VA_BASE + 0xA000ULL) +/* + * SESSION 153-CONTINUED-96: secondary CE (COPY0) lane. A COPY0 channel on + * the same VAS translates through the same 2 MiB window PDB, but CE class + * methods are never legal on the GR0 ladder channel, so the ladder's own + * FBMEM readback (gsp_phys_ctx_readback_try) needs its own engine. The + * lane's GPFIFO ring and one-page pushbuffer get their own small-PT slots + * 0x10d/0x10e in the same 128 B PTE line the ladder's pages occupy, planted + * in pdb_build (stage 3, before any engine fetch; there is no invalidate). + */ +#define GSP_PHYS_RMCALL_VA_CE_GPFIFO (GSP_PHYS_RMCALL_VA_BASE + 0xD000ULL) +#define GSP_PHYS_RMCALL_VA_CE_PB (GSP_PHYS_RMCALL_VA_BASE + 0xE000ULL) /* * m69: window-0 shadow pool - 16 module pages holding the Tier-2 copies of * the QMD's program/param pages, so the GR engine never has to read through @@ -1283,6 +1472,24 @@ bool gsp_phys_boot_requested(void) #define GSP_PHYS_RMCALL_HANDLE_USER_LANE_PM_FB 0xcaf000b8u #define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM0 0xcaf000c0u #define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM1 0xcaf000c4u +/* + * SESSION 153-CONTINUED-96: the secondary COPY0 lane's TSG/channel handles + * (free in the ladder's client range: 0xcaf000c8/0xcc sit above the mode-35 + * fault-mem handles and below the registered-buffer base 0xcaf01000). + */ +#define GSP_PHYS_RMCALL_HANDLE_CE_TSG 0xcaf000c8u +#define GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL 0xcaf000ccu +/* + * Mode 35, graphics-execution plan §F first experiment: the TURING_A + * (0xc597, GR_OBJECT_TYPE_3D) class object constructed on the V7 + * user-lane channel AFTER its promote - the missing half the mode-35 + * capture never provided ("mode 35: ... the capture carries no 0xc597 on + * the user lane"). Best-effort like the TURING_COMPUTE_A: a rejection is + * logged and the lane keeps running; a construction on V7 must not hit + * the wired-golden deadlock class (m3a-m3g, that channel). 0xcaf00094 is + * free between 0xcaf00090 and 0xcaf000a0. + */ +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR3D 0xcaf00094u /* * Mode-35 V7 channel flags/privilege and fault/PM geometry, decoded from the @@ -1508,6 +1715,19 @@ enum gsp_phys_rmcall_buf { GSP_PHYS_RMCALL_BUF_GOLDEN_PT0, GSP_PHYS_RMCALL_BUF_GOLDEN_PT1, GSP_PHYS_RMCALL_BUF_GOLDEN_PT2, + /* + * SESSION 153-CONTINUED-96: secondary COPY0 lane buffers (one page + * each). The instance/USERD/RAMFC/mthdbuf are the lane's own memdescs + * in the channel alloc (a second channel must never share the GR + * lane's hardware USERD); the GPFIFO ring and pushbuffer page are + * engine-reached through their planted PTE slots. + */ + GSP_PHYS_RMCALL_BUF_CE_GPFIFO, + GSP_PHYS_RMCALL_BUF_CE_USERD, + GSP_PHYS_RMCALL_BUF_CE_INSTANCE, + GSP_PHYS_RMCALL_BUF_CE_RAMFC, + GSP_PHYS_RMCALL_BUF_CE_MTHDBUF, + GSP_PHYS_RMCALL_BUF_CE_PB, GSP_PHYS_RMCALL_BUF_COUNT }; @@ -1523,6 +1743,20 @@ enum gsp_phys_rmcall_buf { #define GSP_PHYS_RMCALL_TIMEOUT_MS 2000u #define GSP_PHYS_RMCALL_POLL_MS 5u /* + * SESSION 148 perf (termbin review, verified in-tree): every RPC reply wait + * polled with msleep(5). With several quanta per round trip and ~8 + * serialized PDE_UPDATE forwards per forwarded GR launch, the 5 ms quantum + * dominated per-launch latency (~95% of the ~189 ms host service time per + * launch). Wait with sub-millisecond quanta instead: the reply is observed + * within one quantum, cutting each round trip to ~1 ms. POLL_MS remains for + * the poll accounting prints. + */ +#define GSP_PHYS_RMCALL_POLL_US 150u +static inline void gsp_phys_rmcall_wait_quantum(void) +{ + usleep_range(GSP_PHYS_RMCALL_POLL_US, GSP_PHYS_RMCALL_POLL_US + 200u); +} +/* * GR launch release poll window. A CE copy completes in single-digit ms, but * a GR launch can force the engine's first context switch for the channel; * the FECS ctxsw path is measured in fractions of a second, so the window is @@ -1542,6 +1776,13 @@ enum gsp_phys_rmcall_buf { * past the existing hold budget. */ #define GSP_PHYS_RMCALL_GR_RETRY_MS 2000u +/* + * F2: the short re-poll after the first-timeout fast re-arm (extended + * re-arm + fresh ring entry). The engine ack for a recovered launch is + * observed within tens of ms (r198); 1500 ms covers the ctxsw path without + * burning another full window before the historical ladder continues. + */ +#define GSP_PHYS_RMCALL_GR_FAST_RETRY_MS 1500u #define GSP_PHYS_RMCALL_GR_ATTEMPTS 3u /* P6a guest-memory CE copies: same bounded poll cap as the P5 shadow path. */ #define GSP_PHYS_RMCALL_GUEST_POLL_MS 200u @@ -1567,6 +1808,19 @@ enum gsp_phys_rmcall_buf { #define GSP_PHYS_RMCALL_CE_SUBCHANNEL 4u /* + * SESSION 153-CONTINUED-68: SEC_OP_ONE_INC = 5 (class/cl906f.h:181 + * NV906F_DMA_SEC_OP_ONE_INC, NVC06F_DMA_ONEINCR_* field layout). The + * client emits its MME IR uploads (method 0x114) as op=5 runs whose first + * data word is the IR instruction address and whose remaining words feed + * the IR data port (the r1099/r1116 captures: hdr 0xa0100045 = op 5, + * cnt 16, method 0x114, data [pc, code...]). op=1 (INC_METHOD) walks the + * METHOD address per data word instead (0x114 -> 0x118 -> 0x11c -> + * 0x120), so an IR upload emitted with op=1 writes one stray IR word and + * then clobbers the SA pointer/data registers. + */ +#define GSP_PHYS_RMCALL_METHOD_ONEINC 5u + +/* * CE (TURING_DMA_COPY_A 0xc5b5) method offsets and field values, from * class/clc5b5.h: SET_OBJECT 0x0, SET_SEMAPHORE_A/B/PAYLOAD 0x240/0x244/0x248, * LAUNCH_DMA 0x300, OFFSET_IN_UPPER/LOWER 0x400/0x404, OFFSET_OUT_UPPER/ @@ -2214,6 +2468,16 @@ static struct gsp_phys_boot_stage { u32 rmcall_gr_promote_size; u32 rmcall_gr_hvirt; /* mode-6 NV01_MEMORY_VIRTUAL handle */ u32 rmcall_token; /* GET_WORK_SUBMIT_TOKEN result */ + /* + * SESSION 153-CONTINUED-96: secondary COPY0 lane (same VAS). Alloc'd + * and scheduled only on a GR0 ladder (CE class is illegal on the GR + * channel); rmcall_ce_lane_valid means the token was obtained and the + * lane can accept CE copy blocks. Zero on a COPY0 ladder. + */ + u32 rmcall_ce_tsg; /* KEPLER_CHANNEL_GROUP_A handle */ + u32 rmcall_ce_channel; /* TURING_CHANNEL_GPFIFO_A handle */ + u32 rmcall_ce_token; /* its GET_WORK_SUBMIT_TOKEN result */ + bool rmcall_ce_lane_valid; /* alloc + BIND/schedules + token done */ u32 rmcall_notifier_mem; /* FBMEM error-notifier object handle */ u64 rmcall_notifier_base; /* its returned frame-buffer address */ bool rmcall_notifier_valid; /* the FBMEM notifier alloc succeeded */ @@ -2306,6 +2570,18 @@ static struct gsp_phys_boot_stage { */ struct gsp_phys_boot_chunk rmcall_gr_qmdwin; u64 rmcall_gr_qmdwin_va; /* planted VA; 0 = guest destination */ + /* + * SESSION 153-CONTINUED-66: page 2 of the module pushbuffer plus the + * dedicated warm-semaphore probe page. Allocated lazily by + * gsp_phys_rmcall_pdb_build() and planted in the ladder's 128 B PTE + * line at slots 0x102 (VA_PB2) and 0x10a (VA_GR_PROBE), before the + * engine's first fetch. Page 0 continues the registered BUF_PB chunk + * so stage 2's registration and every existing pushbuffer user are + * unchanged; only the >4 KB GR launch block reaches into these pages. + */ + struct gsp_phys_boot_chunk rmcall_gr_pb2; + struct gsp_phys_boot_chunk rmcall_gr_probe; + bool rmcall_gr_pb2_planted; /* m104: module page holding the injected mail_kernel message. */ struct gsp_phys_boot_chunk rmcall_gr_mail; /* @@ -2482,6 +2758,18 @@ EXPORT_SYMBOL_GPL(gsp_phys_boot_rm_flags); static void *gsp_phys_chunk_page_va(const struct gsp_phys_boot_chunk *c, u32 page) { + /* + * SESSION 141: an unallocated chunk (no per-page array AND no base + * va) must answer NULL. It previously returned + * (u8 *)NULL + page*PAGE_SIZE - a small non-NULL garbage pointer - + * so the caller's NULL guard passed and the empty-set post-mortem + * print dereferenced it: Oops #2 RIP + * gsp_phys_backend_guest_gr_submit+0x4394, CR2=0x1e93c with slot 30 + * (0x1e000 = 30*0x1000 + 0x900 off). min_data=0's no-external-data + * forward hit this on its first pass. + */ + if (!c->page_va && !c->va) + return NULL; if (c->page_va) return c->page_va[page]; return (u8 *)c->va + (size_t)page * GSP_PHYS_PAGE_SIZE; @@ -3028,6 +3316,10 @@ static void gsp_phys_rmcall_reset(void) boot_stage.rmcall_gr_promote_size = 0; boot_stage.rmcall_gr_hvirt = 0; boot_stage.rmcall_token = 0; + boot_stage.rmcall_ce_tsg = 0; + boot_stage.rmcall_ce_channel = 0; + boot_stage.rmcall_ce_token = 0; + boot_stage.rmcall_ce_lane_valid = false; boot_stage.rmcall_notifier_mem = 0; boot_stage.rmcall_notifier_base = 0; boot_stage.rmcall_notifier_valid = false; @@ -3063,6 +3355,11 @@ static void gsp_phys_rmcall_reset(void) memset(&boot_stage.rmcall_gr_qmdwin, 0, sizeof(boot_stage.rmcall_gr_qmdwin)); boot_stage.rmcall_gr_qmdwin_va = 0; + memset(&boot_stage.rmcall_gr_pb2, 0, + sizeof(boot_stage.rmcall_gr_pb2)); + memset(&boot_stage.rmcall_gr_probe, 0, + sizeof(boot_stage.rmcall_gr_probe)); + boot_stage.rmcall_gr_pb2_planted = false; memset(&boot_stage.rmcall_gr_shadow_win0, 0, sizeof(boot_stage.rmcall_gr_shadow_win0)); boot_stage.rmcall_gr_shadow_win0_used = 0; @@ -4770,6 +5067,496 @@ static u32 gsp_phys_engine_type_from_errstr(const char *err) } /* + * SESSION 153-CONTINUED-12: last published GR launch identity, set by the + * single-inflight submit path so a PBDMA fault dump can name the entry the + * PBDMA is stuck on (block length in dwords, guest push VA, object handle). + */ +static u32 gsp_phys_gr_last_words; +static u32 gsp_phys_gr_last_h; +static u64 gsp_phys_gr_last_push_va; + +/* + * SESSION 153-CONTINUED-12: on a robust-channel PBDMA error (exceptType + * 0x20) dump every PBDMA's latched instruction state and, for the PBDMA + * whose PB_FETCH sits inside the module pushbuffer page, the raw block words + * around the fetch point plus the whole block with indices. The latched + * raw instruction (HDR_SHADOW) can be matched against the block dump to + * name the exact word the PBDMA rejected ("invalid method encoding"); the + * fetch position shows how far the walk got. Read-only MMIO + a racy read + * of the pb page (best effort, diagnostic only); bounded per boot. + */ +#define GSP_PHYS_PBDMA_DUMP_MAX 64u + +/* + * SESSION 153-CONTINUED-52: the published method block's DMA address, so the + * PBDMA fault dump can name the exact word the PBDMA stopped on + * (offset = PB_FETCH - last block DMA). Set beside gsp_phys_gr_last_words. + */ +static u64 gsp_phys_gr_last_pb_dma; + +static const char *gsp_phys_pbdma_type_name(u32 t) +{ + static const char *names[8] = { "SSDM", "INC", "STORE_SDM", "NON_INC", + "IMMD", "INC_ONCE", "USE_SDM", "END_SEG" }; + + return names[t & 7u]; +} + +/* + * SESSION 153-CONTINUED-52: MME fault forensics. The C597 replay's dominant + * failure on draw-bearing windows is a real MME exception + * (NV_PGRAPH_PRI_MME_HWW_ESR 0x404490 latched ILLEGAL_OPCODE / + * MISSING_MACRO_DATA / ... - "unknown instruction fetched from the MME + * Instruction RAM", see gr_error_gk104.c grErrorLogMmeException). INFO + * carries the faulting MME PC and the macro address; INFO2 the instruction + * register. Read them post-failure (the latch survives) so the exact macro + * fetch point is on record. Read-only; bounded by its call sites. + */ +static void gsp_phys_mme_regs_log(void __iomem *bar0, const char *tag) +{ + u32 esr, info, info2, trapped; + + if (!bar0) + return; + esr = ioread32(bar0 + 0x00404490u); + info = ioread32(bar0 + 0x00404494u); + info2 = ioread32(bar0 + 0x00404498u); + trapped = ioread32(bar0 + 0x00400704u); + pr_warn("vgsp-gr: MME-REGS %s esr=%08x(pc_valid=%u illegal=%u miss=%u extra=%u) pc=%u macro_addr=0x%x info2_ir=%08x trapped=%08x\n", + tag, esr, (info >> 28) & 1u, (esr >> 2) & 1u, + (esr >> 0) & 1u, (esr >> 1) & 1u, info & 0xfffu, + (info >> 12) & 0xffu, info2, trapped); +} + +/* + * SESSION 153-CONTINUED-69: pre-reset MME fault capture. Everything sampled + * so far was post-mortem: by the time the 5 ms launch poll ticks, the + * GSP-RM's FECS exception handler has already read NV_PGRAPH_PRI_MME_HWW_ESR + * (0x404490) - the relayed errString "ESR 0x404490=0x80000004" is the only + * surviving trace of the raw ILLEGAL_OPCODE value - and either cleared the + * latch back to 0x80000000 or left the whole PRI block in the 0xbadfxxxx + * priv-error poison. This bounded spin runs in the submit thread immediately + * after the publish/doorbell (execution starts within the first + * milliseconds) and records EVERY transition of the fault registers at + * MMIO read rate, so the faulting ESR (+INFO pc/macro when the valid bit is + * set, and the instruction register INFO2 that names the rejected + * instruction word) is on the record before any clear. Read-only; bounded + * per boot and per launch (the spin costs at most sema_mme_fast_ms of the + * launch thread's time and stops early once the poison is seen). + */ +static u32 gsp_phys_mme_fast_ms = 400u; +module_param_named(sema_mme_fast_ms, gsp_phys_mme_fast_ms, uint, 0644); +MODULE_PARM_DESC(sema_mme_fast_ms, + "Spin-sample the MME fault registers right after each GR launch doorbell for this many ms, logging every transition before the GSP-RM clears them (0 = off; default 400)"); + +/* + * SESSION 153-CONTINUED-83: store-side emission forensics. + * + * The store wall: draws retire but `output retirement wait: 0/N pages + * changed` and gen0=0 (no MME-generated method witnessed). The client's + * Turing draw is EMITTED BY AN MME MACRO (slot 34 = the draw macro: + * BEGIN(0x1618) -> SET_VERTEX_ARRAY_START(0xd74) -> END(0x1614), per the + * offline replay of sa_full_stream2.bin and the nvk reference builder), so + * "no stores" == "the draw macro never emitted". This probe answers three + * questions with one line, at the launch boundary: + * + * 1. DS_HWW_ESR (0x405840) + the per-stage SPH_INFO snapshot + * (0x405c00+si*4 START / 0x405c24+si*4 UPPER / 0x405c1c,c20 REGION): + * which pipeline stages failed SPH validation / what VA they fetched. + * 2. MME_HWW_ESR_INFO (0x404494) PC + MACRO_ADDRESS + PC_VALID, + * INFO_EX (0x4044a4) + IR words INFO2/3/4 (0x40449c/a8/ac): did the + * MME actually execute a macro, and where did it stop. + * 3. FE_DECODE_STATE_EX(i) (0x40a630+i*4): MME_PC 11:0, MACRO 21:12, + * GENERATED 31 - the generated-method witness, all four lanes, at + * the launch end (the retire-time read races the release writes). + * + * Bounded: 24 calls per boot, beef family only, one line per call. + */ +/* + * SESSION 153-CONTINUED-83: MME shadow-RAM read port (bar0 0x404488 index, + * 0x40448c data). The MME tracks class methods in its shadow RAM when the + * class's SET_MME_SHADOW_RAM_CONTROL mode selects tracking; reading the + * port tells us (a) whether the tracked window's methods ever reached the + * MME and (b) what values the MME holds for the draw-relevant methods. + * A read is: index = nvclass(15:0) | method_addr(27:16) | READ(30), then + * read data. Bounded by the caller; host-side diagnostic only. + */ +static u32 gsp_phys_mme_shadow_read(void __iomem *bar0, u32 nvclass, u32 mthd) +{ + iowrite32(nvclass | ((mthd & 0xfffu) << 16) | (1u << 30), + bar0 + 0x00404488u); + return ioread32(bar0 + 0x0040448cu); +} + +static void gsp_phys_store_side_log(void __iomem *bar0, const char *tag, + u32 h_object) +{ + static u32 ss_n; + u32 si, sph[6], sphu[6], rr_a, rr_b, ds, ds2, mminfo, mmex; + u32 mm2, mm3, mm4, fex[4], fed[4], fds[4]; + + if (!bar0 || ss_n >= 40u) + return; + ss_n++; + for (si = 0; si < 6u; si++) { + sph[si] = ioread32(bar0 + 0x00405c00u + si * 4u); + sphu[si] = ioread32(bar0 + 0x00405c24u + si * 4u); + } + for (si = 0; si < 4u; si++) { + fex[si] = ioread32(bar0 + 0x0040a630u + si * 4u); + fed[si] = ioread32(bar0 + 0x0040a620u + si * 4u); + fds[si] = ioread32(bar0 + 0x0040a640u + si * 4u); + } + rr_a = ioread32(bar0 + 0x00405c1cu); + rr_b = ioread32(bar0 + 0x00405c20u); + ds = ioread32(bar0 + 0x00405840u); + ds2 = ioread32(bar0 + 0x00405848u); + mminfo = ioread32(bar0 + 0x00404494u); + mmex = ioread32(bar0 + 0x004044a4u); + mm2 = ioread32(bar0 + 0x0040449cu); + mm3 = ioread32(bar0 + 0x004044a8u); + mm4 = ioread32(bar0 + 0x004044acu); + pr_info("vgsp-store: %s h=0x%x ds=%08x/%08x sph=%08x,%08x,%08x,%08x,%08x,%08x sphU=%08x,%08x,%08x,%08x,%08x,%08x rr=%08x,%08x\n", + tag, h_object, ds, ds2, + sph[0], sph[1], sph[2], sph[3], sph[4], sph[5], + sphu[0], sphu[1], sphu[2], sphu[3], sphu[4], sphu[5], + rr_a, rr_b); + pr_info("vgsp-store: %s h=0x%x mme_esr_info=%08x(pcv=%u pc=%u mac=0x%x) ex=%08x ir=%08x,%08x,%08x fed=%08x,%08x,%08x,%08x fex=%08x,%08x,%08x,%08x\n", + tag, h_object, mminfo, (mminfo >> 28) & 1u, mminfo & 0xfffu, + (mminfo >> 12) & 0xffu, mmex, mm2, mm3, mm4, + fed[0], fed[1], fed[2], fed[3], + fex[0], fex[1], fex[2], fex[3]); + /* + * MME shadow-RAM readback for the draw-relevant C597 methods (tracked + * methods show the last value the MME saw; zeros = never tracked). + * Also probe the MME setup pair and the draw trigger/vertex-start. + */ + pr_info("vgsp-store: %s h=0x%x shadow[c597] 0x114=%08x 0x11c=%08x 0x124=%08x 0x260=%08x 0x840=%08x 0xd74=%08x 0xd78=%08x 0x1618=%08x\n", + tag, h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x114u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x11cu), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x124u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x260u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x840u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd74u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd78u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x1618u)); + /* + * SESSION 153-CONTINUED-83 probe 2: the CALL trigger methods the + * window uses (are the calls even reaching the MME?), a repeat read + * (does a read clear?), and the dword-address interpretation of the + * draw/RT methods (0xd74>>2, 0x840>>2) to pin the index units. + */ + pr_info("vgsp-store: %s h=0x%x shadow2[c597] 0x3800=%08x 0x3810=%08x 0x3910=%08x 0x3938=%08x 0x3940=%08x 0x840r=%08x 0xd74dw=%08x 0x840dw=%08x 0x1618r=%08x\n", + tag, h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3800u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3810u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3910u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3938u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3940u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x840u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd74u >> 2), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x840u >> 2), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x1618u >> 2)); +} + +static bool gsp_phys_mme_fast_sample(void __iomem *bar0, const char *tag, + u32 h_object) +{ + static u32 fast_boot_n; + u32 ms = READ_ONCE(gsp_phys_mme_fast_ms); + u32 esr_p, ds_p, gf_p, tr_p, n = 0; + u32 fed0_p, fed1_p; + ktime_t t_end; + bool poisoned = false; + + if (!bar0 || !ms || fast_boot_n >= 48u) + return false; + /* + * Beef-family channels get the full budget (the draw-class windows + * are the ones with the MME fault); other channels are sampled only + * a few times so a fast X-channel launch train cannot consume it. + */ + if ((h_object & 0xfffff000u) != 0xbeef0000u && fast_boot_n >= 2u) + return false; + fast_boot_n++; + esr_p = ioread32(bar0 + 0x00404490u); + ds_p = ioread32(bar0 + 0x00405840u); + gf_p = ioread32(bar0 + 0x00400504u); + tr_p = ioread32(bar0 + 0x00400704u); + /* + * SESSION 153-CONTINUED-84: FE_DECODE_STATE_EX(0)/(1) also drive a + * transition log - the MME's macro/generated witness can flip + * there without touching ESR/DS/grfifo, which is exactly the + * "did the MME start a macro after this doorbell" question the + * reloaded windows keep failing. + */ + fed0_p = ioread32(bar0 + 0x0040a630u); + fed1_p = ioread32(bar0 + 0x0040a634u); + pr_info("vgsp-gr: MME-FAST begin %s h=0x%x esr=%08x ds=%08x grfifo=%08x trap=%08x fe0=%08x fe1=%08x (spin %u ms)\n", + tag, h_object, esr_p, ds_p, gf_p, tr_p, fed0_p, fed1_p, ms); + t_end = ktime_get() + (s64)ms * 1000000ll; + for (;;) { + u32 esr = ioread32(bar0 + 0x00404490u); + u32 ds = ioread32(bar0 + 0x00405840u); + u32 gf = ioread32(bar0 + 0x00400504u); + u32 tr = ioread32(bar0 + 0x00400704u); + u32 fed0 = ioread32(bar0 + 0x0040a630u); + u32 fed1 = ioread32(bar0 + 0x0040a634u); + + if (esr != esr_p || ds != ds_p || gf != gf_p || tr != tr_p || + fed0 != fed0_p || fed1 != fed1_p) { + if (n < 12u) { + u32 info = ioread32(bar0 + 0x00404494u); + u32 info2 = ioread32(bar0 + 0x00404498u); + u32 mmeir = ioread32(bar0 + 0x0040a630u); + u32 mmeir1 = ioread32(bar0 + 0x0040a634u); + u32 mmedata = ioread32(bar0 + 0x00400708u); + + /* + * r1151 (SESSION 153-CONTINUED-72): + * GR-STATE: the FE decode-state EX slot 0 + * records the last MME-GENERATED method + * write (bit31 generated, 21:12 macro + * address, 11:0 pc). gen0=1 with the + * kept slot's macro address is the + * "kept macro executes" witness during + * the launch; slot 1 is the second + * lane of the same record set. + */ + pr_warn("vgsp-gr: MME-FAST %s h=0x%x esr %08x->%08x info=%08x(pc_valid=%u pc=%u mac=0x%x) info2_ir=%08x ds %08x->%08x grfifo %08x->%08x trap=%08x data=%08x fed=%08x,%08x (pc0=%u mac0=0x%x gen0=%u) fe1=%08x (pc1=%u mac1=0x%x gen1=%u)\n", + tag, h_object, esr_p, esr, + info, (info >> 28) & 1u, + info & 0xfffu, + (info >> 12) & 0xffu, info2, + ds_p, ds, gf_p, gf, tr, + mmedata, mmeir, mmeir1, + mmeir & 0xfffu, + (mmeir >> 12) & 0x3ffu, + (mmeir >> 31) & 1u, + fed1, + fed1 & 0xfffu, + (fed1 >> 12) & 0x3ffu, + (fed1 >> 31) & 1u); + } + n++; + esr_p = esr; + ds_p = ds; + gf_p = gf; + tr_p = tr; + fed0_p = fed0; + fed1_p = fed1; + /* + * The priv-error poison is the terminal state: the + * FECS has already logged and the rest of the + * registers are 0xbadfxxxx. Stop early so the launch + * thread resumes its normal poll. + */ + if ((gf & 0xffff0000u) == 0xbadf0000u) { + poisoned = true; + break; + } + } + if (ktime_after(ktime_get(), t_end)) + break; + } + pr_info("vgsp-gr: MME-FAST end %s h=0x%x events=%u%s\n", + tag, h_object, n, poisoned ? " (poison)" : ""); + /* + * SESSION 153-CONTINUED-83 store-side emission forensics: right after + * the launch's retire spin, snapshot the DS shader-program-header + * latches (which pipeline stages failed SPH validation), the MME + * execution state (did the MME run a macro, and where is its PC), and + * the FE decode-state EX slots (did any method decode carry the MME + * GENERATED bit). Bounded per boot (beef-family only). + */ + if ((h_object & 0xfffff000u) == 0xbeef0000u) + gsp_phys_store_side_log(bar0, tag, h_object); + /* + * SESSION 153-CONTINUED-72: return the poison observation to the + * launch site. The r1149 race: the poison flooded during this spin + * and the FECS cleared it before the poll loop's WEDGE-RECOVER check + * read grfifo, so recovery missed, the prelude latch stayed set and + * the engine wedged for the boot. The caller now latches this flag + * into the recovery gate instead of relying on a live register read. + */ + return poisoned; +} + +static void gsp_phys_pbdma_fault_dump(u32 chid) +{ + static u32 dump_n; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + u64 pb_va = boot_stage.rmcall_va_pb; + u32 i; + + if (!bar0 || !gsp_phys_gr_last_words || dump_n >= GSP_PHYS_PBDMA_DUMP_MAX) + return; + dump_n++; + pr_info("vgsp-gr: PB-FAULT-DUMP chid=%u last_block words=%u h=0x%x push=0x%llx pb_va=0x%llx\n", + chid, gsp_phys_gr_last_words, gsp_phys_gr_last_h, + gsp_phys_gr_last_push_va, pb_va); + for (i = 0; i < 12; i++) { + u32 b = 0x40000u + i * 0x2000u; + u32 intr0 = readl(bar0 + b + 0x108u); + u32 gpf = readl(bar0 + b + 0x050u); + u32 pbf = readl(bar0 + b + 0x054u); + u32 pbfhi = readl(bar0 + b + 0x058u); + u32 pbhd = readl(bar0 + b + 0x084u); + u32 pbcnt = readl(bar0 + b + 0x088u); + u32 pbd0 = readl(bar0 + b + 0x08cu); + u32 hdrsh = readl(bar0 + b + 0x118u); + u32 fetch64 = ((u64)(pbfhi & 0xffu) << 32) | (pbf & ~3u); + u32 trig; + const char *trig_src; + s64 off_words = -1; + s64 voff = -1; + + if (!intr0 && !pbf && !hdrsh && !pbhd) + continue; + /* + * SESSION 153-CONTINUED-52: the real driver's rule for the + * instruction that triggered the PBDMA interrupt: if the + * DECODED header (PB_HEADER) type is IMMD or PB_COUNT's value + * field is zero, the raw instruction is PB_DATA0; otherwise it + * is HDR_SHADOW (fifo_tu102.c: "If the TYPE field of + * NV_PPBDMA_PB_HEADER is IMMD or the VALUE field of + * NV_PPBDMA_PB_COUNT is zero, then the raw PB instruction + * stored in NV_PPBDMA_PB_DATA0 is the one that triggered the + * interrupt. Otherwise ... HDR_SHADOW"). + */ + if (((pbhd >> 29) & 7u) == 4u || (pbcnt & 0x1fffu) == 0u) { + trig = pbd0; + trig_src = "DATA0"; + } else { + trig = hdrsh; + trig_src = "HDRSHADOW"; + } + if (gsp_phys_gr_last_pb_dma && + fetch64 >= gsp_phys_gr_last_pb_dma && + fetch64 < gsp_phys_gr_last_pb_dma + + (u64)gsp_phys_gr_last_words * 4u) + off_words = (s64)(fetch64 - gsp_phys_gr_last_pb_dma) / 4; + /* + * SESSION 153-CONTINUED-59: the SAME offset computed against + * the GPU VA the engine fetches (the r1102 WINPROBE/PB-FAULT + * agreement: pbfetch 0x101ebc with pb_va 0x101000 and 943 + * words = exactly the block end). voff >= words is the + * past-the-block-end prefetch position. + */ + voff = (s64)-1; + if (pb_va && fetch64 >= pb_va && + fetch64 < pb_va + (u64)GSP_PHYS_PAGE_SIZE) + voff = (s64)(fetch64 - pb_va) / 4; + pr_info("vgsp-gr: PB-FAULT-DUMP P%u intr0=%08x gpf=%u pbfetch=0x%llx len=%u off=%lld voff=%lld/%u hdr=%08x(%s m=0x%x cnt=%u sub=%u) cnt=%08x data0=%08x shadow=%08x pbhd=%08x(type=%s) TRIG=%08x(%s)\n", + i, intr0, gpf, fetch64, pbfhi >> 11, off_words, voff, + gsp_phys_gr_last_words, + hdrsh, gsp_phys_pbdma_type_name(hdrsh >> 29), + (hdrsh & 0x1fffu) << 2, + (hdrsh >> 16) & 0xfffu, (hdrsh >> 13) & 7u, + pbcnt, pbd0, hdrsh, pbhd, + gsp_phys_pbdma_type_name(pbhd >> 29), + trig, trig_src); + } + { + struct gsp_phys_boot_chunk *pbch = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; + u32 *pw = pbch->pages ? gsp_phys_chunk_page_va(pbch, 0) : NULL; + + if (pw && pb_va) { + u32 n = gsp_phys_gr_last_words < 384u ? + gsp_phys_gr_last_words : 384u; + u32 w; + + for (w = 0; w < n; w += 16) { + u32 m; + + pr_info("vgsp-gr: PB-FAULT-DUMP pb[%3u]= %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + w, pw[w], w + 1 < n ? pw[w + 1] : 0, + w + 2 < n ? pw[w + 2] : 0, w + 3 < n ? pw[w + 3] : 0, + w + 4 < n ? pw[w + 4] : 0, w + 5 < n ? pw[w + 5] : 0, + w + 6 < n ? pw[w + 6] : 0, w + 7 < n ? pw[w + 7] : 0, + w + 8 < n ? pw[w + 8] : 0, w + 9 < n ? pw[w + 9] : 0, + w + 10 < n ? pw[w + 10] : 0, w + 11 < n ? pw[w + 11] : 0, + w + 12 < n ? pw[w + 12] : 0, w + 13 < n ? pw[w + 13] : 0, + w + 14 < n ? pw[w + 14] : 0, w + 15 < n ? pw[w + 15] : 0); + } + /* + * SESSION 153-CONTINUED-59: the block TAIL and the + * words PAST the block end (the r1102 fault fetch + * sits exactly at the end - voff == words). The past + * words are what a prefetching PBDMA decodes next; + * the tail shows whether the block itself ends at an + * entry boundary. + */ + { + u32 words = gsp_phys_gr_last_words; + u32 lo = words > 48u ? words - 48u : 0u; + u32 hi = words + 32u; + + if (hi > 1024u) + hi = 1024u; + for (w = lo; w < hi; w += 16) { + pr_info("vgsp-gr: PB-FAULT-DUMP %s[%3u]= %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + w < words ? "tail" : "past", w, + pw[w], w + 1 < 1024u ? pw[w + 1] : 0, + w + 2 < 1024u ? pw[w + 2] : 0, w + 3 < 1024u ? pw[w + 3] : 0, + w + 4 < 1024u ? pw[w + 4] : 0, w + 5 < 1024u ? pw[w + 5] : 0, + w + 6 < 1024u ? pw[w + 6] : 0, w + 7 < 1024u ? pw[w + 7] : 0, + w + 8 < 1024u ? pw[w + 8] : 0, w + 9 < 1024u ? pw[w + 9] : 0, + w + 10 < 1024u ? pw[w + 10] : 0, w + 11 < 1024u ? pw[w + 11] : 0, + w + 12 < 1024u ? pw[w + 12] : 0, w + 13 < 1024u ? pw[w + 13] : 0, + w + 14 < 1024u ? pw[w + 14] : 0, w + 15 < 1024u ? pw[w + 15] : 0); + } + } + } + } +} + +/* + * SESSION 153-CONTINUED-86: the fault-time register witness. The relayed + * errString names the fault; these reads add the machine state behind it at + * the earliest host observation point (the 0x1006 consume), before the + * FECS/GSP-RM clear the latches: MME_HWW_ESR (0x404490) + INFO (0x404494: + * PC_VALID/PC/MACRO_ADDRESS) + INFO2 (IR word), DS_HWW_ESR (0x405840) + + * ESR_2 (0x405848) + the SPH_INFO stage starts (0x405c00+si*4) that name the + * exact VA each failing pipeline stage fetched its shader program header + * from, the SPH program-region pair (0x405c1c/c20), and the trapped method + * address/data (0x400704/8). GR-class faults only (0xd graphics exception, + * 0x45 class error; PBDMA 0x20 has its own dump), bounded per boot so a fault + * storm cannot flood dmesg. + */ +static void gsp_phys_fault_regs_log(u32 except_type) +{ + static u32 frl_n; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + u32 esr, info, info2, ds, ds2, trap_a, trap_d, rr_a, rr_b, sph[6], si; + + if (!bar0 || frl_n >= 48u) + return; + if (except_type != 0xdu && except_type != 0x45u) + return; + frl_n++; + esr = ioread32(bar0 + 0x00404490u); + info = ioread32(bar0 + 0x00404494u); + info2 = ioread32(bar0 + 0x00404498u); + ds = ioread32(bar0 + 0x00405840u); + ds2 = ioread32(bar0 + 0x00405848u); + for (si = 0; si < 6u; si++) + sph[si] = ioread32(bar0 + 0x00405c00u + si * 4u); + rr_a = ioread32(bar0 + 0x00405c1cu); + rr_b = ioread32(bar0 + 0x00405c20u); + trap_a = ioread32(bar0 + 0x00400704u); + trap_d = ioread32(bar0 + 0x00400708u); + pr_info("vgsp-fault: regs except=0x%x mme_esr=%08x(pcv=%u ill=%u miss=%u extra=%u) mme_info=%08x(pc=%u mac=0x%x) mme_ir=%08x ds=%08x/%08x sph=%08x,%08x,%08x,%08x,%08x,%08x rr=%08x,%08x trap=%08x data=%08x\n", + except_type, esr, (info >> 28) & 1u, (esr >> 2) & 1u, + esr & 1u, (esr >> 1) & 1u, info, info & 0xfffu, + (info >> 12) & 0xffu, info2, ds, ds2, + sph[0], sph[1], sph[2], sph[3], sph[4], sph[5], + rr_a, rr_b, trap_a, trap_d); +} + +/* * Note every consumed 0x1006 in the bounded last-4 ring, advance the fault * epoch (the relay submit snapshot compares against it) and log the designed * observation line when the fault relay is armed. Decoding and counting are @@ -4785,10 +5572,14 @@ static void gsp_phys_fault_observe(const struct gsp_phys_rx_record *rec, GSP_PHYS_FAULT_NOTE_MAX; gsp_phys_fault_seen++; gsp_phys_fault_epoch++; - if (gsp_phys_fault_relay_enabled()) + if (gsp_phys_fault_relay_enabled()) { pr_info_ratelimited("vgsp-fault: func=0x1006 seq=%u exceptType=0x%x runlistId=%u chid=%u prevXid=0x%x errString=\"%s\"\n", rec->seq_num, out->except_type, out->runlist_id, out->chid, out->prev_xid, out->err); + gsp_phys_fault_regs_log(out->except_type); + } + if (out->except_type == 0x20u) + gsp_phys_pbdma_fault_dump(out->chid); } /* Note every consumed 0x1020 for the C3 corroboration ring. */ @@ -6686,6 +7477,35 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); /* + * Mode-35 KGR static-data snapshots. The user-lane query set + * (gsp_phys_rmcall_user_lane_kgr_queries) is answered by the LIVE GSP with + * the device's true GR static data; the replies are kept here so the + * guest-side emulation can serve the identical bytes instead of fabricating + * GR values. The concrete consumer is the X driver's 3D-engine selection: + * nv3dAllocDevice maps GR_GET_INFO index SM_VERSION (0x0c) through a table + * and fails with "Failed to select a 3D engine" when it reads back as + * SM_VERSION_NONE (0) - the fabricated fill never set it, while the live + * GSP's KGR_GET_INFO reply carries the real value (0x75 for TU104) along + * with every other GR static field (warps, caps, topology, ctx buffers). + * + * One entry per captured cmd; the generation key ties the cache to the + * backend session exactly like gsp_phys_backend_static above, and the + * getter refuses entries once the session has moved on. + */ +#define GSP_PHYS_KGR_SNAP_ENTRIES 10 +#define GSP_PHYS_KGR_SNAP_MAX_SIZE 3712u /* KGR_GET_INFO: 8 engines x 58 x 8 */ + +struct gsp_phys_kgr_snap { + u32 cmd; + u32 size; + bool valid; + u32 generation; + u8 data[GSP_PHYS_KGR_SNAP_MAX_SIZE]; +}; + +static struct gsp_phys_kgr_snap gsp_phys_kgr_snaps[GSP_PHYS_KGR_SNAP_ENTRIES]; + +/* * Drop the cached static-config snapshot. Called when the backend session * leaves READY (or a new claim starts): the cache generation key is bumped * past the current backend generation so a concurrent getter cannot serve the @@ -6694,9 +7514,13 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); */ void gsp_phys_backend_static_invalidate(void) { + unsigned int i; + mutex_lock(&gsp_phys_rm_lock); gsp_phys_backend_static.valid = false; gsp_phys_backend_static.generation = gsp_phys_backend_generation() + 1; + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) + gsp_phys_kgr_snaps[i].valid = false; mutex_unlock(&gsp_phys_rm_lock); } @@ -6750,6 +7574,71 @@ int gsp_phys_backend_static_get(struct gsp_phys_backend_static_config *out) } EXPORT_SYMBOL_GPL(gsp_phys_backend_static_get); +/* + * Copy the captured reply for cmd into dst. Returns 0 on success (with + * *out_len set), -ENODATA when nothing was captured for cmd (or the entry's + * generation no longer matches the backend session), -ENOSPC when the + * snapshot does not fit dst_len, -EAGAIN while the backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len) +{ + unsigned int i; + + if (!dst || !out_len) + return -EINVAL; + + if (gsp_phys_backend_state() != GSP_PHYS_BACKEND_READY) + return -EAGAIN; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + struct gsp_phys_kgr_snap *s = &gsp_phys_kgr_snaps[i]; + + if (s->cmd != cmd) + continue; + if (!s->valid || + s->generation != gsp_phys_backend_generation()) + return -ENODATA; + if (s->size > dst_len) + return -ENOSPC; + memcpy(dst, s->data, s->size); + *out_len = s->size; + return 0; + } + return -ENODATA; +} +EXPORT_SYMBOL_GPL(gsp_phys_kgr_snap_get); + +/* + * Record one mode-35 KGR query reply. Called under the boot thread only + * (same thread as the query itself), so no lock is needed beyond the + * generation key. + */ +static void gsp_phys_kgr_snap_store(u32 cmd, const void *data, u32 size) +{ + unsigned int i; + struct gsp_phys_kgr_snap *slot = NULL; + + if (size > GSP_PHYS_KGR_SNAP_MAX_SIZE) + return; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + if (gsp_phys_kgr_snaps[i].cmd == cmd) { + slot = &gsp_phys_kgr_snaps[i]; + break; + } + if (!slot && !gsp_phys_kgr_snaps[i].valid) + slot = &gsp_phys_kgr_snaps[i]; + } + if (!slot) + return; + + slot->cmd = cmd; + slot->size = size; + slot->generation = gsp_phys_backend_generation(); + memcpy(slot->data, data, size); + slot->valid = true; +} + /* ============================================================ * Host RM client ladder (R1): gsp-phys-rm-call * ============================================================ */ @@ -9485,14 +10374,94 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0) * (swref/published/pascal/gp100/dev_mmu.h:97-119,157; * uvm_pascal_mmu.c:64-141,235-303). */ -static u64 gsp_phys_rmcall_pde(u64 pa) +/* + * r431/r436 COHERENCY ROOT-CAUSE CANDIDATE: the leaf-PTE aperture for every + * host page this backend maps (the ladder windows, the win0 shadow pool, the + * bounce/data regions, the release/scratch shadows) was hardcoded to + * aperture 3 = SYSTEM_NON_COHERENT_MEMORY. The reference RM derives the + * pair from the allocation's CPU cache attribute (fabric_vaspace.c:1123-1134 + * for the aperture, mem_desc.c memdescGetVolatility for VOL: NV_MEMORY_CACHED + * -> SYSTEM_COHERENT (2) + volatile FALSE, NV_MEMORY_UNCACHED -> NON_COHERENT + * (3) + volatile TRUE). The r431 experiment used aperture 2 with VOL still 1 + * (0xd) - a pair the reference RM never produces - and it made the engine's + * truncated writes visible sooner but worsened the wedge on a fresh boot. + * r436 turns the knob into the full 2x2 matrix so each consistent encoding + * can be A/B'd: + * 0 = aperture 3 + VOL 1 (0xf) - historical, NV_MEMORY_UNCACHED + * 1 = aperture 2 + VOL 1 (0xd) - r431 experiment (inconsistent pair) + * 2 = aperture 2 + VOL 0 (0x5) - NV_MEMORY_CACHED (coherent, non-volatile) + * 3 = aperture 3 + VOL 0 (0x7) - non-coherent, non-volatile + * The PTEs are planted before the engine's first fetch, so the flag is + * boot-time only. + */ +static int gsp_phys_rmcall_pte_coh_param; +module_param_named(gsp_phys_rmcall_pte_coh, gsp_phys_rmcall_pte_coh_param, + int, 0444); +MODULE_PARM_DESC(gsp_phys_rmcall_pte_coh, + "r436: sysmem leaf-PTE aperture/VOL mode: 0=0xf (aperture3+VOL, historical), 1=0xd (aperture2+VOL, r431), 2=0x5 (aperture2, non-volatile), 3=0x7 (aperture3, non-volatile)"); + +static u64 gsp_phys_rmcall_sysmem_base_low(void) { - return 0xeull | ((pa >> 12) << 8); + switch (gsp_phys_rmcall_pte_coh_param) { + case 1: + return 0xdull; /* aperture 2, VOL 1 */ + case 2: + return 0x5ull; /* aperture 2, VOL 0 */ + case 3: + return 0x7ull; /* aperture 3, VOL 0 */ + default: + return 0xfull; /* aperture 3, VOL 1 (historical) */ + } } static u64 gsp_phys_rmcall_pte(u64 pa) { - return 0xfull | ((pa >> 12) << 8); + /* + * Low nibble: VALID (1) | APERTURE<<1 | VOL (8). Aperture 2 = + * SYSTEM_COHERENT_MEMORY -> 1|4|8 = 0xd; aperture 3 = NON_COHERENT + * -> 1|6|8 = 0xf. r436: see gsp_phys_rmcall_pte_coh_param. + */ + u64 base = gsp_phys_rmcall_sysmem_base_low(); + + return base | ((pa >> 12) << 8); +} + +/* + * r436: the single PDE uses the same aperture/VOL pair as the leaf PTEs but + * with IS_PDE=0 in bit 0 (the low bit of the PTE base is VALID there), so the + * PDE base is the PTE base with bit 0 cleared: 0xe/0xc/0x4/0x6. + */ +static u64 gsp_phys_rmcall_pde(u64 pa) +{ + u64 base = gsp_phys_rmcall_sysmem_base_low() & ~1ull; + + return base | ((pa >> 12) << 8); +} + +/* + * SESSION 153-CONTINUED-92: leaf PTE for the engine-visible guest/scratch + * mappings with the configurable v2 KIND field (bits 63:56). + * + * m96 documented the symptom that motivates this: on Turing a kind=0 PTE is + * what the SM I-fetch refuses with MMU_NACK while the FE tolerates it, and + * `gsp_phys_rmcall_gr_pte_kind` was introduced to plant the win0/identity + * pool PTEs with a kind (UVM turing uses GENERIC_MEMORY 0x06). The knob was + * never applied to the scratch-PT pool mappings - every shadow VA the 3D + * window's program operands are relocated to (the VAs the DS fetches the + * SPH through) carried kind=0. The DS validates a program header, i.e. runs + * a program fetch; if that path is the SM-side one, kind=0 is a candidate + * cause for the SPH rejection that is independent of content (and explains + * why no content A/B ever moved the wall). With the knob 0 this is + * byte-identical to the previous PTE encoding; nonzero = experiment. + */ +static u64 gsp_phys_rmcall_gr_pte_leaf(u64 hpa) +{ + u64 pte = gsp_phys_rmcall_pte(hpa); + int kind = gsp_phys_rmcall_gr_pte_kind_value(); + + if (kind) + pte |= (u64)(u32)kind << 56; + return pte; } /* @@ -9625,6 +10594,26 @@ static void gsp_phys_rmcall_pdb_build(void) (u8 *)gsp_phys_chunk_page_va(pt, 0) + (size_t)sem_idx * 8); /* + * SESSION 153-CONTINUED-96: secondary CE lane GPFIFO ring (slot 0x10d) + * and its one-page pushbuffer (slot 0x10e), planted here in the same + * 128 B PTE line as the ladder's own pages - stage 3, before any + * engine fetch (no invalidate exists). Always planted (the lane's + * page/PTE cost is two pages; a disabled lane simply never rings its + * doorbell), so pdb_build stays deterministic. + */ + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_CE_GPFIFO], 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)((GSP_PHYS_RMCALL_VA_CE_GPFIFO >> 12) & 0x1ffu) * 8); + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_CE_PB], 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)((GSP_PHYS_RMCALL_VA_CE_PB >> 12) & 0x1ffu) * 8); + /* * Rank-2 qmdwin: plant the dedicated module page for the forwarded * inline-QMD destination in this same small-PT line (slot 0x107, VA * 0x107000; slots 0x105/0x106 stay the P6a guest slots below). This @@ -9671,6 +10660,66 @@ static void gsp_phys_rmcall_pdb_build(void) } } /* + * SESSION 153-CONTINUED-66: plant page 2 of the module pushbuffer + * (VA_PB2, small-PT slot 0x102) and the dedicated warm-semaphore + * probe page (VA_GR_PROBE, slot 0x10a) in the same 128 B PTE line, + * at stage 3 before the engine's first fetch (no invalidate exists). + * Page 0 stays the registered BUF_PB chunk (stage 2 unchanged); this + * chunk is engine-reached purely through these PTEs, exactly like + * the qmdwin pair above. Without it the GR launch block keeps the + * single-page bound (<= 4 KB). + */ + if (!boot_stage.rmcall_gr_pb2.pages && boot_stage.pdev) { + if (!gsp_phys_chunk_alloc(boot_stage.pdev, + &boot_stage.rmcall_gr_pb2, 3u)) + gsp_phys_chunk_zero(&boot_stage.rmcall_gr_pb2); + else + pr_warn("vgsp-boot: rm-call: pushbuffer page2/3/4 allocation failed; >4 KB launches keep the single-page block\n"); + } + if (!boot_stage.rmcall_gr_probe.pages && boot_stage.pdev) { + if (!gsp_phys_chunk_alloc(boot_stage.pdev, + &boot_stage.rmcall_gr_probe, 1u)) + gsp_phys_chunk_zero(&boot_stage.rmcall_gr_probe); + else + pr_warn("vgsp-boot: rm-call: warm-sem probe page allocation failed; the probe keeps the pushbuffer upper half\n"); + } + if (boot_stage.rmcall_gr_pb2.pages && boot_stage.rmcall_gr_probe.pages) { + u32 pb2_idx = (u32)((GSP_PHYS_RMCALL_VA_PB2 >> 12) & 0x1ffu); + u32 pb3_idx = (u32)((GSP_PHYS_RMCALL_VA_PB3 >> 12) & 0x1ffu); + u32 pb4_idx = (u32)((GSP_PHYS_RMCALL_VA_PB4 >> 12) & 0x1ffu); + u32 pr_idx = (u32)((GSP_PHYS_RMCALL_VA_GR_PROBE >> 12) & + 0x1ffu); + + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_pb2, 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pb2_idx * 8); + if (boot_stage.rmcall_gr_pb2.pages >= 2u) + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_pb2, 1)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pb3_idx * 8); + if (boot_stage.rmcall_gr_pb2.pages >= 3u) + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_pb2, 2)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pb4_idx * 8); + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_probe, 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pr_idx * 8); + dma_wmb(); + boot_stage.rmcall_gr_pb2_planted = true; + pr_info("vgsp-boot: rm-call: GR launch pb page2/3/4 + warm-sem probe planted VA=0x%llx/0x%llx/0x%llx/0x%llx ptSlots=0x%x/0x%x/0x%x/0x%x (up to four-page block; before first fetch)\n", + GSP_PHYS_RMCALL_VA_PB2, GSP_PHYS_RMCALL_VA_PB3, + GSP_PHYS_RMCALL_VA_PB4, GSP_PHYS_RMCALL_VA_GR_PROBE, + pb2_idx, pb3_idx, pb4_idx, pr_idx); + } + /* * m69: plant the window-0 shadow pool PTEs (current win0 slot range) in the * same proven small-PT line, before the engine's first fetch. The * launch copies the QMD's program/param page content into these pages @@ -13410,6 +14459,12 @@ static int gsp_phys_rmcall_user_lane_userd(void __iomem *bar0) * (CONTEXT_BUFFERS_INFO in particular) may be prepared by the query. All are * zero-in best-effort getters at the capture's payload sizes; a failure * warns and the lane continues. + * + * Every successful reply is retained via gsp_phys_kgr_snap_store(): the + * guest-side emulation serves these exact bytes for the same commands, so + * the guest's GL/3D stack reads the live GSP's true GR static values + * (SM_VERSION in particular - see the snapshot comment near + * gsp_phys_backend_static_get). */ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) { @@ -13448,6 +14503,8 @@ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) q[i].name); pr_info("vgsp-boot: rm-call: mode 35 KGR query %s: size=%u status=0x%x rpc_result=0x%x ret=%d\n", q[i].name, q[i].size, status, rpc_result, ret); + if (ret == 0 && rpc_result == 0 && status == 0) + gsp_phys_kgr_snap_store(q[i].cmd, buf, q[i].size); kvfree(buf); } } @@ -13720,6 +14777,16 @@ static int gsp_phys_rmcall_user_lane_main_promote(void __iomem *bar0) status, rpc_result, GSP_PHYS_RMCALL_HANDLE_USER_LANE, boot_stage.rmcall_ctx_pa, boot_stage.rmcall_ctx_va); kvfree(p); + /* + * SESSION 153-CONTINUED-96: the F2 mapped-MAIN promote is the server's + * acceptance of the ladder's own context buffer (PA+VA, bInitialize=1), + * i.e. exactly the condition rmcall_ctx_valid records. Modes 0-34 set it + * at the end of their promote step (line ~13675); mode 35 promotes here, + * so without this the state file read valid=0 and the engine-ctx + * readback gate silently skipped. + */ + if (!ret) + boot_stage.rmcall_ctx_valid = true; return ret; } @@ -14440,6 +15507,35 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0) boot_stage.rmcall_gr_object = GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ; } + /* + * Graphics-execution plan §F first 3D-lane experiment: + * construct the TURING_A (0xc597) class object on V7 + * after the accepted promote, through the same generic + * alloc helper the golden lanes use. The plan's open + * question is whether a 0xc597 construct on the + * execution lane is accepted without the wired-golden + * deadlock class; the result line below is the + * experiment's evidence either way, and the construct + * never gates the lane. + */ + { + u32 g3_rpc = 0, g3_status = 0; + int g3_ret = gsp_phys_rmcall_alloc_gr_object_class(bar0, + GSP_PHYS_RMCALL_CLASS_GR_3D, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR3D, + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + "mode 35 TURING_A on V7 (plan-F first 3D-lane experiment, after promote)", + &g3_rpc, &g3_status); + + pr_info("vgsp-boot: rm-call: mode 35 plan-F experiment: TURING_A (0xc597) on V7 hChannel=0x%x -> ret=%d hObject=0x%x status=0x%x rpc_result=0x%x (%s)\n", + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + g3_ret, + g3_ret ? 0 : + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR3D, + g3_status, g3_rpc, + g3_ret ? "rejected/absent; lane continues" : + "accepted; V7 carries a 3D class object for the method-stream replay"); + } pr_info("vgsp-boot: rm-call: mode 35: execution channel switched from V6 0x%x to V7 0x%x (captured flags=0x%x internalFlags=0x%x hContextShare=0x%x hVASpace=0, gpFifoVa=0x%llx entries=%u, sysmem USERD, sysmem instance/ramfc (m62 readback), TURING_COMPUTE_A 0x%x); V6 stays allocated and unscheduled, stage 4 + forwarding run on V7\n", boot_stage.rmcall_channel, GSP_PHYS_RMCALL_HANDLE_USER_LANE, @@ -14757,23 +15853,31 @@ static u32 gsp_phys_rmcall_userd_ctl_off(void) * launch path publish byte-identically; the CE caller's semaphore clear and * method-block build happen before this call. The caller must hold * gsp_phys_rm_lock. - */ -static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, - u32 words, - struct gsp_phys_rmcall_ce_submit *out) + * + * SESSION 153-CONTINUED-96: the lane carries the channel's GPFIFO ring, + * hardware USERD chunk, out-fill chunks and work-submit token, so the + * secondary COPY0 lane publishes through the identical body on its own + * ring/USERD/doorbell token (a second channel must never advance the GR + * lane's GP_PUT or ring the GR channel's doorbell). + */ +struct gsp_phys_rmcall_lane { + struct gsp_phys_boot_chunk *ring; + struct gsp_phys_boot_chunk *userd; + struct gsp_phys_boot_chunk *pb; + struct gsp_phys_boot_chunk *sem; + struct gsp_phys_boot_chunk *src; + struct gsp_phys_boot_chunk *dst; + u32 token; + const char *tag; +}; + +static int gsp_phys_rmcall_publish_lane_locked(void __iomem *bar0, + u64 pb_gpu_va, u32 words, + const struct gsp_phys_rmcall_lane *lane, + struct gsp_phys_rmcall_ce_submit *out) { - struct gsp_phys_boot_chunk *ring = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GPFIFO]; - struct gsp_phys_boot_chunk *userd_c = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; - struct gsp_phys_boot_chunk *pb = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; - struct gsp_phys_boot_chunk *sem = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM]; - struct gsp_phys_boot_chunk *src = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SRC]; - struct gsp_phys_boot_chunk *dst = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_DST]; + struct gsp_phys_boot_chunk *ring = lane->ring; + struct gsp_phys_boot_chunk *userd_c = lane->userd; u32 *ring_va, *userd; u32 gpput, gpget, slot, e0, e1; u32 runlist_id, ch_id, doorbell; @@ -14786,11 +15890,39 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, gpput = get_unaligned_le32((u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPPUT_OFF); gpget = get_unaligned_le32((u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPGET_OFF); if (gpput >= GSP_PHYS_RMCALL_GPFIFO_ENTRIES) { - pr_err("vgsp-boot: rm-call: USERD GP_PUT=%u is outside the %u-entry ring; refusing to submit\n", - gpput, GSP_PHYS_RMCALL_GPFIFO_ENTRIES); + pr_err("vgsp-boot: rm-call: %s USERD GP_PUT=%u is outside the %u-entry ring; refusing to submit\n", + lane->tag, gpput, GSP_PHYS_RMCALL_GPFIFO_ENTRIES); return -EIO; } - slot = gpput; + /* + * SESSION 153-CONTINUED-46: publish at the ENGINE's GP_GET, not at + * the raw GP_PUT. Observed on every stalled beef boot: the USERD + * drifts (GPGet=5 while GPPut cycles 1/2/4/7 - the engine's GET ran + * past the publisher's PUT, e.g. after a wrapped or otherwise + * deranged ring), and a ring is MODULO: PUT < GET numerically means + * (PUT-GET) mod ENTRIES pending entries - hundreds - so the PBDMA + * walks stale slots, executes garbage method blocks and the launch + * never retires (the trapped=0x8000022c / no-dispatch class). The + * forwarded channel is strictly single-inflight, so the engine's + * GET names exactly the next free slot: write our entry there and + * advance PUT to GET+1 so the PBDMA sees exactly one pending entry - + * ours. When GP_GET is out of range (uninitialised USERD), fall back + * to the old PUT-derived slot. + */ + slot = (gpget < GSP_PHYS_RMCALL_GPFIFO_ENTRIES) ? gpget : gpput; + { + static u32 slotfix_log; + + if (gpget != gpput && slotfix_log < 64) { + slotfix_log++; + pr_info("vgsp-boot: rm-call: %s publish at engine GP_GET=%u (raw GP_PUT=%u was %s; pending-would-be=%u)\n", + lane->tag, gpget, gpput, + gpput == gpget ? "equal" : + (gpput > gpget ? "ahead" : "behind"), + (gpput + GSP_PHYS_RMCALL_GPFIFO_ENTRIES - gpget) % + GSP_PHYS_RMCALL_GPFIFO_ENTRIES); + } + } /* * e0: bits 31:2 hold the GET field, which itself is (fetch address @@ -14807,8 +15939,8 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, */ e0 = (u32)pb_gpu_va & ~3u; if (unlikely(e0 != (u32)((pb_gpu_va >> 2) << 2))) - pr_err("vgsp-boot: rm-call: GPFIFO GET encode mismatch: e0=0x%08x decodes to fetch 0x%llx, expected pb VA 0x%llx\n", - e0, (u64)(e0 & ~3u), pb_gpu_va); + pr_err("vgsp-boot: rm-call: %s GPFIFO GET encode mismatch: e0=0x%08x decodes to fetch 0x%llx, expected pb VA 0x%llx\n", + lane->tag, e0, (u64)(e0 & ~3u), pb_gpu_va); e1 = (u32)((pb_gpu_va >> 32) & 0xffu) | (words << 10); put_unaligned_le32(e0, (u8 *)ring_va + slot * 8); put_unaligned_le32(e1, (u8 *)ring_va + slot * 8 + 4); @@ -14817,8 +15949,10 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, * GP_PUT advances on the client sysmem USERD chunk, which is the * channel's hardware USERD because the params carry that same chunk; * the host write is therefore what the engine polls (cycle 21). + * CONT-46: PUT = slot+1 (slot was chosen from GP_GET above), so the + * single new entry is the entire pending set. */ - put_unaligned_le32((gpput + 1) % GSP_PHYS_RMCALL_GPFIFO_ENTRIES, + put_unaligned_le32((slot + 1) % GSP_PHYS_RMCALL_GPFIFO_ENTRIES, (u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPPUT_OFF); /* * Doorbell value: workSubmitToken = ((runlistId << 16) & 0x7f00000) | @@ -14827,12 +15961,12 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, * write the formula's result (identical unless the token carries stray * high bits, which is logged). */ - runlist_id = (boot_stage.rmcall_token >> 16) & 0x7fu; - ch_id = boot_stage.rmcall_token & 0xfffu; + runlist_id = (lane->token >> 16) & 0x7fu; + ch_id = lane->token & 0xfffu; doorbell = ((runlist_id << 16) & 0x7f00000u) | (ch_id & 0xfffu); - if (doorbell != boot_stage.rmcall_token) - pr_warn("vgsp-boot: rm-call: token 0x%x carries bits outside runlistId[22:16]|chId[11:0]; writing the recomposed doorbell 0x%x\n", - boot_stage.rmcall_token, doorbell); + if (doorbell != lane->token) + pr_warn("vgsp-boot: rm-call: %s token 0x%x carries bits outside runlistId[22:16]|chId[11:0]; writing the recomposed doorbell 0x%x\n", + lane->tag, lane->token, doorbell); /* * C3: snapshot the fault epoch immediately before the doorbell, so a * 0x1006 consumed by this submission's timeout drain is accepted only @@ -14852,14 +15986,33 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, out->doorbell = doorbell; out->runlist_id = runlist_id; out->ch_id = ch_id; - out->pb_dma = (u64)gsp_phys_chunk_page_dma(pb, 0); - out->sem_dma = (u64)gsp_phys_chunk_page_dma(sem, 0); - out->src_dma = (u64)gsp_phys_chunk_page_dma(src, 0); - out->dst_dma = (u64)gsp_phys_chunk_page_dma(dst, 0); + out->pb_dma = (u64)gsp_phys_chunk_page_dma(lane->pb, 0); + out->sem_dma = (u64)gsp_phys_chunk_page_dma(lane->sem, 0); + out->src_dma = (u64)gsp_phys_chunk_page_dma(lane->src, 0); + out->dst_dma = (u64)gsp_phys_chunk_page_dma(lane->dst, 0); } return 0; } +static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, + u32 words, + struct gsp_phys_rmcall_ce_submit *out) +{ + const struct gsp_phys_rmcall_lane lane = { + .ring = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GPFIFO], + .userd = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD], + .pb = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB], + .sem = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM], + .src = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SRC], + .dst = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_DST], + .token = boot_stage.rmcall_token, + .tag = "ladder", + }; + + return gsp_phys_rmcall_publish_lane_locked(bar0, pb_gpu_va, words, + &lane, out); +} + /* * One CE copy block on the ladder channel: clear the module semaphore, build * the methods for bytes into the module pushbuffer, publish the GPFIFO entry @@ -14881,10 +16034,30 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, * site passes false, so the default COPY0 ladder emits the same bytes as * before. */ -static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, - u64 dst_va, u64 sem_va, u32 bytes, - u32 sem_payload, bool gr, - struct gsp_phys_rmcall_ce_submit *out) +/* + * SESSION 153-CONTINUED-12: pb_off selects where in the module pushbuffer + * page the method block is written and fetched from. The warm-semaphore + * probe (gsp_phys_rmcall_gr_sem_push_locked) used the upper half so it + * could never overwrite a forwarded launch's pending method block: + * measured with the PBDMA dump, the probe's 10-dword block landing on + * words 0..9 of a pending 139-word 3D replay made the PBDMA walk the + * probe's SET_OBJECT/REPORT_SEM tail and then decode the replay window's + * operand 0x00230200 as an instruction -> PBENTRY on chid 4 and no + * retire. Every forwarded GR/CE block stays at offset 0. + * + * SESSION 153-CONTINUED-66: pb_alt/pb_alt_va point the block at a + * dedicated one-page module buffer (the probe's VA_GR_PROBE page) instead + * of the registered pushbuffer; a forwarded GR block may now span both + * pushbuffer pages, so no fixed offset of that page is safe for the probe. + */ +static int gsp_phys_rmcall_ce_submit_off_locked(void __iomem *bar0, + u64 src_va, u64 dst_va, + u64 sem_va, u32 bytes, + u32 sem_payload, bool gr, + u32 pb_off, + struct gsp_phys_boot_chunk *pb_alt, + u64 pb_alt_va, + struct gsp_phys_rmcall_ce_submit *out) { struct gsp_phys_boot_chunk *pb = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; @@ -14913,11 +16086,28 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, if (!bar0 || !boot_stage.rmcall_token || !bytes || bytes > GSP_PHYS_RMCALL_COPY_SIZE) return -EINVAL; + if (pb_off & 3u) + return -EINVAL; pb_gpu_va = boot_stage.rmcall_va_pb; if (!pb_gpu_va || (pb_gpu_va & 3)) return -EIO; + /* + * SESSION 153-CONTINUED-66: pb_alt selects a dedicated one-page module + * buffer instead of the registered pushbuffer page (the warm-semaphore + * probe uses its own PTE-planted page now that a forwarded GR block may + * span BOTH pushbuffer pages). pb_off stays relative to that page. + */ + if (pb_alt && pb_alt->pages) { + pb = pb_alt; + pb_gpu_va = pb_alt_va; + if (!pb_gpu_va || (pb_gpu_va & 3)) + return -EIO; + } - pb_va = gsp_phys_chunk_page_va(pb, 0); + if (!gsp_phys_chunk_page_va(pb, 0) || pb_off > GSP_PHYS_PAGE_SIZE) + return -EIO; + pb_va = (u32 *)((u8 *)gsp_phys_chunk_page_va(pb, 0) + pb_off); + pb_gpu_va += pb_off; ring_va = gsp_phys_chunk_page_va(ring, 0); userd = gsp_phys_chunk_page_va(userd_c, 0); if (!pb_va || !ring_va || !userd || @@ -14938,9 +16128,82 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, dst_va, bytes, sem_payload); + if (pb_off + words * 4 > GSP_PHYS_PAGE_SIZE) + return -EIO; + return gsp_phys_rmcall_publish_locked(bar0, pb_gpu_va, words, out); } +static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, + u64 dst_va, u64 sem_va, u32 bytes, + u32 sem_payload, bool gr, + struct gsp_phys_rmcall_ce_submit *out) +{ + return gsp_phys_rmcall_ce_submit_off_locked(bar0, src_va, dst_va, + sem_va, bytes, sem_payload, + gr, 0u, NULL, 0, out); +} + +/* + * SESSION 153-CONTINUED-96: one CE copy block on the secondary COPY0 lane. + * Byte-for-byte the CE submission body (same method block, same guards, + * same module-semaphore clear) but on the lane's own GPFIFO ring, USERD + * chunk, one-page pushbuffer and work-submit token. Only legal when the + * lane's bringup obtained a token (rmcall_ce_lane_valid). The caller must + * hold gsp_phys_rm_lock and polls the module semaphore separately, exactly + * like the readback's bounded page loop. + */ +static int gsp_phys_rmcall_ce_lane_submit_locked( + void __iomem *bar0, u64 src_va, u64 dst_va, u64 sem_va, + u32 bytes, u32 sem_payload, + struct gsp_phys_rmcall_ce_submit *out) +{ + const struct gsp_phys_rmcall_lane lane = { + .ring = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_CE_GPFIFO], + .userd = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_CE_USERD], + .pb = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_CE_PB], + .sem = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM], + .src = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SRC], + .dst = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_DST], + .token = boot_stage.rmcall_ce_token, + .tag = "CE lane", + }; + struct gsp_phys_boot_chunk *sem = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM]; + resource_size_t bar0_len; + u32 *pb_va; + u32 words; + + if (!boot_stage.rmcall_ce_lane_valid) + return -EOPNOTSUPP; + if (!boot_stage.pdev) + return -EINVAL; + bar0_len = pci_resource_len(boot_stage.pdev, 0); + if (bar0_len < GSP_PHYS_RMCALL_DOORBELL_OFF + 4) + return -EIO; + if (!bar0 || !boot_stage.rmcall_ce_token || !bytes || + bytes > GSP_PHYS_RMCALL_COPY_SIZE) + return -EINVAL; + pb_va = gsp_phys_chunk_page_va(lane.pb, 0); + if (!pb_va || !gsp_phys_chunk_page_va(lane.ring, 0) || + !gsp_phys_chunk_page_va(lane.userd, 0) || + !gsp_phys_chunk_page_va(sem, 0)) + return -EIO; + + /* A stale release from a previous submission must not be observed. */ + WRITE_ONCE(*(u32 *)gsp_phys_chunk_page_va(sem, 0), 0); + wmb(); + + words = gsp_phys_rmcall_build_methods(pb_va, sem_va, src_va, dst_va, + bytes, sem_payload); + if (words * 4 > GSP_PHYS_PAGE_SIZE) + return -EIO; + + return gsp_phys_rmcall_publish_lane_locked(bar0, + GSP_PHYS_RMCALL_VA_CE_PB, + words, &lane, out); +} + /* * H1 stage-4 GR verification: one GR semaphore release on the ladder channel, * shared by stage 4's GR branch and the debugfs trigger 9 re-run. The caller @@ -14953,6 +16216,39 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, * semaphore. On timeout the existing status-queue drain runs and the helper * returns -ETIMEDOUT; it never retries. */ +static int gsp_phys_rmcall_gr_resched_locked(void __iomem *bar0); +static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0); + +/* + * r445 wedge hard re-arm: the outcome-retry's warm semaphore probe proves the + * engine refused the channel's proven GR block (the sticky post-dispatch + * no-store wedge). One bounded recovery attempt instead of an immediate + * break: force a context save/unload with a bWait TSG preempt, re-run the + * mode-35 lane-enable tail + schedules (the m155 extended re-arm), then + * repeat the warm probe. Caller holds gsp_phys_rm_lock. Returns 0 when the + * engine executed the probe after the re-arm. + */ +static int gsp_phys_gr_wedge_hard_rearm_locked(void __iomem *bar0) +{ + u32 pre[2] = { 1u, 0u }; /* bWait=1, bManualTimeout=0 */ + u32 rpc_result = 0, status = 0; + int ret; + + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_CTRL_TSG_PREEMPT, + pre, sizeof(pre), sizeof(pre), NULL, 0, + &rpc_result, &status, + "TSG PREEMPT bWait=1 (r445 wedge hard re-arm)"); + pr_warn("vgsp-gr: WEDGE-REARM preempt status=0x%x rpc=0x%x ret=%d\n", + status, rpc_result, ret); + ret = gsp_phys_rmcall_gr_resched_locked(bar0); + pr_warn("vgsp-gr: WEDGE-REARM resched ret=%d\n", ret); + ret = gsp_phys_rmcall_gr_sem_push_locked(bar0); + pr_warn("vgsp-gr: WEDGE-REARM probe rc=%d (engine %s after re-arm)\n", + ret, ret ? "still wedged" : "ALIVE"); + return ret; +} + static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) { struct gsp_phys_boot_chunk *userd_c = @@ -14965,11 +16261,33 @@ static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) unsigned long deadline; int ret; - ret = gsp_phys_rmcall_ce_submit_locked(bar0, GSP_PHYS_RMCALL_VA_SRC, + /* + * SESSION 153-CONTINUED-12: this probe runs while a forwarded launch + * may still be pending in the same pushbuffer page. It must not land + * on words 0.. of that page: measured PB-FAULT-DUMP shows the probe's + * 10-dword block overwriting a pending 139-word 3D replay's head, so + * the PBDMA walks the probe then decodes a replay operand as an + * instruction (PBENTRY, no retire). + * + * SESSION 153-CONTINUED-66: the forwarded GR block now spans BOTH + * pushbuffer pages (up to 2048 dwords = 0x2000 bytes), so the old + * upper-half placement (0x800) would clobber it. The probe has its + * own PTE-planted page (VA_GR_PROBE, small-PT slot 0x10a); when that + * plant is unavailable the probe falls back to 0x800 as before. + */ + ret = gsp_phys_rmcall_ce_submit_off_locked(bar0, + GSP_PHYS_RMCALL_VA_SRC, GSP_PHYS_RMCALL_VA_DST, GSP_PHYS_RMCALL_VA_SEM, GSP_PHYS_RMCALL_COPY_SIZE, GSP_PHYS_RMCALL_SEM_PAYLOAD, true, + boot_stage.rmcall_gr_probe.pages ? + 0u : 0x800u, + boot_stage.rmcall_gr_probe.pages ? + &boot_stage.rmcall_gr_probe : + NULL, + boot_stage.rmcall_gr_probe.pages ? + GSP_PHYS_RMCALL_VA_GR_PROBE : 0, &sub); if (ret) { pr_err("vgsp-boot: rm-call: GR submission refused before the doorbell (%d)\n", @@ -15014,7 +16332,7 @@ static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) gsp_phys_status_drain(NULL); return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } gpget = get_unaligned_le32((u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPGET_OFF); @@ -15114,7 +16432,7 @@ static void gsp_phys_rmcall_ctx_image_probe(void __iomem *bar0) offs[k], poll); break; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } if (poll != GSP_PHYS_RMCALL_SEM_PAYLOAD) break; @@ -15215,7 +16533,7 @@ static int gsp_phys_rmcall_ce_copy_va(void __iomem *bar0, u64 src_va, *sem_out = poll; return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } } @@ -15398,7 +16716,7 @@ int gsp_phys_backend_ce_copy_submit(u64 src_va, u64 dst_va, u64 sem_va, gsp_phys_work_timed_out); return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } /* @@ -15917,7 +17235,7 @@ static int gsp_phys_guest_ce_copy_common(u64 src_hpa, u32 src_off, u64 dst_hpa, ret = -ETIMEDOUT; goto out; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } /* @@ -16139,19 +17457,145 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_guest_ce_copy); #define GSP_PHYS_RMCALL_GR_CWD_REFCOUNT 1536u #define GSP_PHYS_RMCALL_GR_CWD_LEVELS 64u #define GSP_PHYS_RMCALL_GR_MAX_SM_COUNT 24u +/* + * r434 CWD-slot-count path: the falcon PRI write (CUDA's + * turingFalconPRIWriteMasked: flag/data/mask through SET_MME_SHADOW_SCRATCH + * 0/1/2, then SET_FALCON04 = register offset) and the gated method. Used only + * when gsp_phys_rmcall_gr_cwd_slot >= 0. + */ +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH0 0x3400u +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH1 0x3404u +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH2 0x3408u +#define GSP_PHYS_RMCALL_GR_SET_FALCON04 0x0510u +#define GSP_PHYS_RMCALL_GR_SET_CWD_SLOT_COUNT 0x02b0u +#define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1 0x00407040u +#define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT 0x800u /* Method block bound: init(152) + inline run(67) + PCAS(4) + REPORT_SEM(8) + - * the optional QMD-virt pair(2). */ -#define GSP_PHYS_RMCALL_GR_METHOD_WORDS 512u + * the optional QMD-virt pair(2). Raised 512 -> 1024 (SESSION 153- + * CONTINUED-53) so a 3D stream replay can carry the captured MME setup + * prelude (<=512 words) ahead of the largest chunked window (<=448) in one + * block. SESSION 153-CONTINUED-66: 1024 -> 2048 because the matured prelude + * reached 901 words and the deferral test (window=390 + prelude=901 + sa=47 + * vs the bound) dropped it on every launch; the block now spans the module + * pushbuffer's TWO 4 KB pages (VA_PB..VA_PB2) and is copied page-wise by + * gsp_phys_gr_pb_copy(). */ +#define GSP_PHYS_RMCALL_GR_METHOD_WORDS 4096u static u64 gsp_phys_gr_submitted; static u64 gsp_phys_gr_ok; static u64 gsp_phys_gr_refused; static u64 gsp_phys_gr_timed_out; +/* + * SESSION 153-CONTINUED-60: 3D-stream launches whose window was all-zero + * (the client's push page caught mid-rewrite). They are never handed to the + * engine: the zero method stream trips the PBDMA (trap 0x1168 -> priv-error + * poison, see the ZEROWIN-SKIP site). Counted so a boot's zero-window rate + * is visible without dmesg archaeology. + */ +static u64 gsp_phys_gr_zerowin_skips; static u64 gsp_phys_gr_retries; /* fresh-entry launch retries published */ static u64 gsp_phys_gr_pool_exhausted; /* m156 (E5): zero-write retries where the warm sem push proved the engine * wedged (the persistent-lottery class); further retries are skipped. */ static u64 gsp_phys_gr_sem_wedged; +/* + * STALLFREE: bounded-completion telemetry. stall_wait_max counts launches + * that crossed gsp_phys_gr_wait_max_ms and failed out of the submit ladder + * (the sema service then authors the QMD release in software); stall_retry + * counts launches whose retirement retry loop stopped early on the same + * budget (the launch proceeds with its current outcome). Both are zero on a + * green run. + */ +static u64 gsp_phys_gr_stall_wait_max; +static u64 gsp_phys_gr_stall_retry; +/* + * r410 wedge quiet window state. nw_streak counts consecutive accepted + * zero-write launches with tracked data (reset by any launch that wrote); + * the sticky wedge is the state where this grows without bound while the + * lane is kept busy. quiet counts the quiet holds issued; heals counts the + * quiet retries after which the engine wrote. Single-inflight submit path, + * so plain statics. + */ +static u32 gsp_phys_gr_nw_streak; +static u64 gsp_phys_gr_wedge_quiet; +static u64 gsp_phys_gr_wedge_heals; +/* + * r411: once a quiet heal has succeeded on this boot, the lane is in a + * wedge-prone episode. r418 showed the wedge can alternate one-no-write / + * one-heal, and the healed launch resets nw_streak, so a streak-only trigger + * quieted only every second wedged launch and the other was accepted. The + * heal count makes the episode sticky for the boot: any later zero-write + * launch gets the quiet path on its first retry, so every wedged launch + * either heals (engine writes, client data correct) or is loudly classified. + */ +static bool gsp_phys_gr_wedge_episode; +/* + * r420: true exactly while a WEDGE-QUIET hold sleeps with NO submission + * outstanding. The hold is a deliberate, bounded wait for the sticky wedge to + * clear (r410/r411); during it the in-flight forwarded request is NOT done. + * The sema service's completion deadlines - the 12 s GR release hold + * (SVC_FWD_GR_HOLD_TIMEOUT_MS) and the 3 s per-launch consumption bound + * (gsp_phys_gr_hold_max_ms) - are shorter than this hold, so without the + * publish they fire UNDER it and author the launch's release word before the + * post-hold copy-out: the client then verifies stale data (r419 soak: + * `victory_matmul bad=65536` while its launch was held; the authoring paths + * are GR-STALL-fwd-hold / the busy-defer inline software release). The + * accessor lets both deadlines pause instead; the flag is bounded by the + * msleep itself and by the pause caps on the sema side. Single-inflight + * submit path, so a plain static. + */ +static bool gsp_phys_gr_quiet_hold_on; + +bool gsp_phys_gr_quiet_hold_active(void) +{ + return READ_ONCE(gsp_phys_gr_quiet_hold_on); +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_quiet_hold_active); + +/* + * r425 experiment (the second-launch wedge): the stock guest driver emits the + * per-channel compute init block ONCE at channel creation (turing_init.c), + * long before any launch, while this backend re-emits it as the preamble of + * every forwarded launch. The wedge strikes the SECOND launch of a channel, so + * test whether the re-emission is the trigger: with gsp_phys_rmcall_gr_init_once + * set, the init preamble is emitted only on the first forwarded launch after + * the module load (the engine's compute-class/CWD state persists in the + * context; every launch still carries its QMD + report semaphore). 0 keeps the + * historical per-launch emission (default; runtime-writable A/B). + */ +static bool gsp_phys_rmcall_gr_init_once_param; +module_param_named(gsp_phys_rmcall_gr_init_once, gsp_phys_rmcall_gr_init_once_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_init_once, + "r425: emit the GR compute init preamble only on the first forwarded launch (0 = every launch, default)"); +static bool gsp_phys_rmcall_gr_init_emitted; + +/* + * r425 stale-QMD witness: the first engine-changed page of the LAST completed + * launch (bounce index + post-copy-out checksum). At a zero-write accept the + * page is re-scanned; a change proves a late store (a stuck/queued grid or a + * stale-QMD execution) landed after its launch was shipped. + */ +static u32 gsp_phys_gr_prev_out_bi; +static u64 gsp_phys_gr_prev_out_csum; +static bool gsp_phys_gr_prev_out_valid; + +/* r425: SKED/FECS/CWD register witness (one line; the healthy-vs-wedged + * comparison baseline lives in the same log). */ +static void gsp_phys_gr_sked_wit(void __iomem *bar0, const char *tag) +{ + if (!bar0) + return; + pr_info("vgsp-gr: SKED-WIT %s cwd0=%08x cwd1=%08x cwd2=%08x cwd3=%08x info1=%08x act=%08x grfifo=%08x fecs=%08x new=%08x ctxsw1=%08x qmdvirt0=%08x tpcm0=%08x\n", + tag, + readl(bar0 + 0x00407060u), readl(bar0 + 0x00407064u), + readl(bar0 + 0x00407068u), readl(bar0 + 0x0040706cu), + readl(bar0 + 0x00407028u), readl(bar0 + 0x00407054u), + readl(bar0 + 0x00400504u), readl(bar0 + 0x00409b00u), + readl(bar0 + 0x00409b04u), readl(bar0 + 0x00409400u), + readl(bar0 + 0x0040a500u), readl(bar0 + 0x0040a680u)); +} + /* m86: program-page override blob (see gsp-phys-gr-override). */ static u8 gsp_phys_gr_override[GSP_PHYS_PAGE_SIZE]; static u32 gsp_phys_gr_override_len; @@ -16261,10 +17705,837 @@ static u32 *gsp_phys_rmcall_build_gr_init(u32 *p) GSP_PHYS_RMCALL_GR_MAX_SM_COUNT); p = gsp_phys_rmcall_push_gr(p, GSP_PHYS_RMCALL_GR_INIT_SM_DISP_CTRL, 0u); + /* + * r434 CWD-slot-count path (gsp_phys_rmcall_gr_cwd_slot >= 0): masked + * PRI write of SKED_DEBUG_1.ENABLE_CWD_SLOT_COUNT (bit 11) through the + * falcon scratch registers, then the gated SET_CWD_SLOT_COUNT method - + * the CUDA driver's own turingLaunchSetCwdSlotCount sequence. -1 (the + * default) emits nothing, so the historical preamble is unchanged. + */ + if (gsp_phys_rmcall_gr_cwd_slot_value() >= 0) { + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH0, 0u); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH1, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH2, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_SET_FALCON04, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_SET_CWD_SLOT_COUNT, + (u32)gsp_phys_rmcall_gr_cwd_slot_value()); + } return p; } /* + * r597: extract the SEM_EXECUTE ACQUIRE groups from the captured guest + * prelude and copy them, verbatim, ahead of the rebuilt inline-QMD window. + * Two encodings are recognised: + * - C46F host-semaphore group: INC_METHOD addr 0x5c count>=5, words + * {LO, HI, PAYLOAD_LO, PAYLOAD_HI, EXECUTE}; EXECUTE & 7 != 1 is an + * acquire (the pump's own op decode). + * - C5C0 SET_REPORT_SEMAPHORE A..D: addr 0x1b00 count>=4, word D's low + * two bits select release (0) / acquire (non-zero). + * Only the wait methods are taken, so the proven window (init + inline QMD + * + PCAS + release) is untouched. Returns the number of words emitted. + */ +static u32 gsp_phys_rmcall_gr_emit_acquires(u32 *p, + const struct gsp_phys_gr_submit *req, + u32 budget) +{ + u32 off = 0, words = 0; + + if (!gsp_phys_rmcall_gr_acq_value() || !req || !req->raw) + return 0; + while (off < req->raw_qmd_hdr && words + 8 < budget) { + u32 hdr = req->raw[off]; + u32 op = (hdr >> 29) & 0x7; + u32 count = (hdr >> 16) & 0xfff; + u32 addr = (hdr & 0x1fff) << 2; + u32 n = 1 + count; + + if ((op != 1 && op != 3) || !count || + off + n > req->raw_qmd_hdr) { + off++; + continue; + } + if (addr == 0x5c && count >= 5) { + u32 v4 = req->raw[off + 5]; + u64 va = ((u64)(req->raw[off + 2] & 0xff) << 32) | + (u64)(req->raw[off + 1] & 0xfffffffc); + + if ((v4 & 0x7) != 1 && va) { + memcpy(p + words, &req->raw[off], n * 4); + words += n; + } + } else if (addr >= 0x1b00 && addr <= 0x1b0c && + count >= 4) { + u32 d = req->raw[off + 4] & 0x3; + + if (d != 0) { + memcpy(p + words, &req->raw[off], n * 4); + words += n; + } + } + off += n; + } + return words; +} + +/* + * r929 (GL 3D execution, Path A): replay a captured raw 3D method window. + * The sema service marks stream captures (the X server's C597 sections that + * carried their own operands) with a raw window (req->stream); when this is + * on, gsp_phys_rmcall_build_gr_launch emits SET_OBJECT TURING_A + the window + * verbatim + the channel release tail instead of the compute prelude/QMD, so + * the engine executes the REAL 3D methods (state binds, LINE_LENGTH_IN + + * LAUNCH_DMA uploads, program/tex binds) and, once draws are in the window, + * rasterizes. No effect on any submit without a stream window (all CUDA + * launches). Default on; A/B with gsp_phys_rmcall_gr_3d=0. + */ +static bool gsp_phys_rmcall_gr_3d = true; +module_param_named(gsp_phys_rmcall_gr_3d, gsp_phys_rmcall_gr_3d, bool, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_3d, + "Replay a captured raw C597 method window verbatim instead of the compute prelude (default 1)"); + +/* + * SESSION 153-CONTINUED-59: drop single-word SDM-type entries (SEC_OP 0 + * SSDM / 2 STORE_SDM / 6 USE_SDM, tu102 dev_esched_pbdma.h type map) from + * every replayed stream: the 3D window patch, the MME prelude and the attr + * prelude emission. The r1102 PBENTRY first-sighting forensics localized + * the fault decode point to the MME prelude head (WINPROBE/PB-FAULT-DUMP + * P0 HDR_SHADOW=04666d14 = prelude[2], PB_DATA0=04646d00 = prelude[3], + * pbfetch=0x101ebc = block end after the whole 943-word block was + * prefetched, intr0 bit 18 in the guest relay): the client's setup push + * carries a 21-word run of `04666d00 04666d10 04666d14 04646d00 ...` + * (SDM-family single-word encodings, method fields 0x3400/0x3440/0x3450) + * that svc_mme_prelude_capture stored verbatim; the physical PBDMA refuses + * them ("invalid method encoding used in method", NV_PPBDMA_INTR_0 + * PBENTRY), which also explains why the whole block was fetched but the + * REPORT_SEMAPHORE never retired. The guest-vGPU parts (subdevice masks) + * are meaningless on the single-subdevice ladder channel - the same call + * the sub!=0 multi-word drop makes. Zero words are kept (benign padding). + * Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_sdm_drop = true; +module_param_named(sema_gl_sdm_drop, gsp_phys_rmcall_gr_sdm_drop, bool, + 0644); +MODULE_PARM_DESC(sema_gl_sdm_drop, + "Drop single-word SDM-type entries (SEC_OP 0/2/6) from 3D window replays, the MME prelude and the attr prelude (default 1)"); + +/* + * SESSION 153-CONTINUED-53: MME context-setup prelude. The r1073-r1077 + * in-flight capture proved the draw-window replay raises a real MME + * ILLEGAL_OPCODE on CALL_MME_MACRO(2)=0x3810 (mme_pc=0, mme_ir=1, mme_mac + * =2) because the ladder context never received the client's macro + * program. The client does upload it (0x11c/0x114 pointer pairs, 0x118 + * code, 0x120 entry addresses, 0x124 shadow control; see + * svc_mme_prelude_capture); with this knob the launch builder preplays + * those captured runs verbatim - once per capture generation - right + * after the SET_OBJECT C597 bind and ahead of the window, so the macros + * the window calls are resident. Declared before build_gr_launch (its + * reader); the matching capture-side knob is sema_mme_prelude_cap. + * Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_mme_prelude = true; +module_param_named(sema_gl_mme_prelude, gsp_phys_rmcall_gr_mme_prelude, bool, + 0644); +MODULE_PARM_DESC(sema_gl_mme_prelude, + "Preplay the captured client MME context-setup runs once per capture generation ahead of the first 3D window replay (default 1)"); + +/* + * SESSION 153-CONTINUED-63: MME START-ADDRESS RAM completion + stub knob. + * The captured SA table (svc_mme_sa_*) may be incomplete (the prelude + * buffer is capped at 512 words); every CALLED slot the table does not + * cover is loaded with a harmless end-noop stub at a fixed IR address so + * the call becomes a no-op instead of a jump to IR 0 with garbage state + * (the r1116 0x238c poison). Default 1; A/B with 0. + */ +#define GSP_PHYS_RMCALL_GR_MME_STUB_PC 0xC0u +/* + * SESSION 153-CONTINUED-69: the stub body, take 3. The CONT-68 body + * (20208c00 38c40046 038c4301) was copied from macro(0) instruction 3 - + * but that is a MID-MACRO instruction: the offline MME64 simulator shows + * it carries two output writes with mthd=NONE, so in isolation (method + * register still 0) it EMITS SET_OBJECT <- 0 twice - exactly the relayed + * trap "ChID 0001, Class 0000c597, Offset 00000000, Data 00000000" - and + * its end_next=1 requires the following instruction, so the macro also + * runs past the stub into whatever the IR holds there. + * + * The real minimal terminator is what Mesa's mme_tu104_builder_finish() + * emits: two default NOPs with end_next set on the FIRST (end_next - + * "end after the next instruction" - so the second NOP executes and then + * the macro returns). Both NOPs encode as w2=0x318c0300 (pred=ZERO, + * ALU0/1 dst=ZERO src=ZERO,ZERO, outs NONE) and emit NOTHING. The + * end_next bit is w2 bit 0. + */ +static const u32 gsp_phys_gr_mme_stub_code[6] = { + 0x00000000u, 0x318c0000u, 0x318c0301u, /* NOP, end_next=1 */ + 0x00000000u, 0x318c0000u, 0x318c0300u, /* NOP tail (executed) */ +}; +static bool gsp_phys_rmcall_gr_sa_stub = true; +module_param_named(sema_gl_mme_sa_stub, gsp_phys_rmcall_gr_sa_stub, bool, + 0644); +MODULE_PARM_DESC(sema_gl_mme_sa_stub, + "Load a harmless end-noop stub into every MME macro slot the window calls but the captured START-ADDRESS RAM table does not cover (default 1; the r1116 0x238c lane poison class)"); + +/* + * SESSION 153-CONTINUED-68: stub EVERY called slot, covered or not + * (attribution: with all calls no-op'd the window either still faults + * (the fault is in non-call content) or faults differently (the fault is + * macro execution). Default 0; A/B knob. + */ +static bool gsp_phys_rmcall_gr_sa_stub_all; +module_param_named(sema_gl_mme_sa_stub_all, gsp_phys_rmcall_gr_sa_stub_all, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_sa_stub_all, + "Stub every macro slot the window calls, including slots the captured SA table covers (attribution knob; default 0)"); + +static inline bool gsp_sema_gr_sa_stub_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sa_stub); +} + +/* + * SESSION 153-CONTINUED-69: optional explicit stub IR address override. + * 0 = derive from the capture (ir_end, floor 0xC0). Nonzero pins the stub + * body/call target to this instruction address - the stub-landing A/B for + * the IR-boundary/ONEINC pointer-unit question. + */ +static u32 gsp_phys_rmcall_gr_mme_stub_pc; +module_param_named(sema_gl_mme_stub_pc, gsp_phys_rmcall_gr_mme_stub_pc, uint, + 0644); +MODULE_PARM_DESC(sema_gl_mme_stub_pc, + "Explicit MME stub IR instruction address (0 = derive from capture ir_end; try 0xf0/0x100 for the IR-boundary A/B)"); + +/* + * SESSION 153-CONTINUED-69: no-stall CALL drop. With the stub terminating + * macros as pure NOPs the macro consumes none of the CALL_MME_DATA + * parameter words the window pushes; the MME parameter port then + * backpressures on the window's larger calls - the r1142 stall parked the + * PBDMA mid-block with HDR_SHADOW = the 16-arg CALL_MME_MACRO(57) + * ONEINC run and the block never completed (while every <=4-arg window + * retired in ~100 us). Drop every CALL_MME_MACRO/CALL_MME_DATA run of a + * STUBBED slot from the emitted window instead: no call, no parameter + * words, no backpressure. Covered slots keep their calls (their real code + * consumes the parameters). Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_call_drop = true; +module_param_named(sema_gl_mme_call_drop, gsp_phys_rmcall_gr_call_drop, bool, + 0644); +MODULE_PARM_DESC(sema_gl_mme_call_drop, + "Drop CALL_MME_MACRO/DATA runs of stubbed slots from emitted 3D windows (prevents the noop-stub parameter-port stall; default 1)"); + +/* + * SESSION 153-CONTINUED-70: per-slot macro re-enable mask, one slot at a + * time. The r1142-r1144 state drops EVERY stubbed-slot CALL run: the window + * retires on real HW (~94 us) but the macros that expand the driver's + * per-draw state (vertex stream binds, constant binds, ...) never execute, + * so no state the draw needs exists and every launch still stores 0 pages. + * A slot named in this mask is never treated as stubbed: its + * CALL_MME_MACRO/DATA words stay in the emitted window and the engine + * executes the macro against the replayed context (SA RAM + IR program as + * the client's own uploads left them; a slot the client never uploaded + * reads SA=0 exactly like silicon and lands in macro 0). A/B one bit per + * cycle; keep the proven-good set empty until the offline MME64 sim has + * shown the slot's macro emits sane methods for its caller args. + */ +static u64 gsp_phys_rmcall_gr_keep_mask; +module_param_named(sema_gl_mme_keep_mask, gsp_phys_rmcall_gr_keep_mask, + ullong, 0644); +MODULE_PARM_DESC(sema_gl_mme_keep_mask, + "Slots whose CALL_MME_MACRO/DATA runs are never dropped/stubbed (u64 bitmask; one slot at a time for macro re-enable)"); + +/* + * SESSION 153-CONTINUED-79: per-launch method-block word cap + prelude drain. + * The r1172-r1175 A/B: every beef launch whose emitted block exceeded ~2000 + * words (3083-word full-SA prelude + window, 3315-3484 words) NEVER STARTED + * on the physical engine (MME-FAST events=0, grfifo frozen, ch_rel never + * written) while every block <= ~1300 words (1002-word partial capture + + * window) executed (retires in 90-170 us when the executed macro stayed + * inside the loaded IR). The failure class is fetch-side (a stuck GPFIFO + * entry), not content: once parked, every later entry also failed until a + * poison-latch TSG re-arm reset the PBDMA. The full-SA capture (55 slots, + * 3083 words, macros that chain past the first 15 slots e.g. macro 0 -> + * IR 0x201 = slot-25 entry) therefore cannot be replayed in ONE block. + * Emit at most gsp_phys_rmcall_gr_block_cap words per launch and drain the + * (SDM-sanitized, run-aligned) prelude across consecutive launches with a + * generation-keyed cursor; the emission latch is set only when the final + * part has been emitted, so a completed program is re-requested after the + * WEDGE-RECOVER latch clear. Default 2048; 0 = legacy 4096 module buffer. + */ +static u32 gsp_phys_rmcall_gr_block_cap = 2048u; +module_param_named(sema_gl_mme_block_cap, gsp_phys_rmcall_gr_block_cap, uint, + 0644); +MODULE_PARM_DESC(sema_gl_mme_block_cap, + "Max words emitted per GR launch method block (0 = 4096 module buffer; the >2048-word blocks never start on the physical engine)"); + +/* Generation-keyed drain cursor for the split MME prelude emission. */ +static u32 gsp_phys_gr_mme_pre_emit_gen; +static u32 gsp_phys_gr_mme_pre_emit_h; +static u32 gsp_phys_gr_mme_pre_emit_off; + +/* + * SESSION 153-CONTINUED-79: parameter-conserving NOP for kept parking slots. + * The r1177 bisect: keep={0,2,57} (full 3089-word program resident, split + * emission) parks the PBDMA on the emitted block (MME-FAST events=0, grfifo + * frozen, warm sem WEDGED) while keep={0,2} retires every launch - slot 57's + * CALL_MME_MACRO(57) run is the r1142 park class. Rather than drop the run + * (which also removes its parameter words from the stream), rewrite the call + * run's header to target the CALL_MME_DATA port (method+4) so the parameter + * words are still written (conserved) but no macro executes: the trigger + * word becomes a plain port write and the PBDMA never opens a macro context + * whose parameter drain can back up. A slot named in this mask must also be + * in sema_gl_mme_keep_mask (kept = retained in the window). Default 0. + */ +static u64 gsp_phys_rmcall_gr_nop_param_mask; +module_param_named(sema_gl_mme_nop_param_mask, gsp_phys_rmcall_gr_nop_param_mask, + ullong, 0644); +MODULE_PARM_DESC(sema_gl_mme_nop_param_mask, + "Kept slots whose CALL_MME_MACRO trigger is rewritten to a plain CALL_MME_DATA port write (params conserved, macro not executed; u64 bitmask)"); + +/* + * SESSION 153-CONTINUED-80: the r1181 live A/B validated the retarget above + * and it PARKS the PBDMA exactly like the raw kept call - with + * keep=5|1<<57 + nop=1<<57 the first post-change launch froze + * (MME-FAST begin grfifo=0x00000001, end events=0, no poison, warm sem + * WEDGED) while keep=5 (slot 57 call+params elided) retired in 94-106 us + * on the same boot (r1176/r1181 baseline). The bare DATA-port run without an + * open macro context is not a safe landing for the trigger. With this knob + * set, a nop-mask slot joins the drop path instead (the r1144-r1177 + * proven-safe behavior: whole run elided, retires clean, params NOT + * conserved). Use for the all-real-macro A/B: sa_stub_all=0 leaves + * 34/39/40/48/51 real while this drops only 57. Default 0 preserves the + * retarget behavior so the r1179 transform evidence stays reproducible. + */ +static u64 gsp_phys_rmcall_gr_nop_drop; +module_param_named(sema_gl_mme_nop_drop, gsp_phys_rmcall_gr_nop_drop, ullong, + 0644); +MODULE_PARM_DESC(sema_gl_mme_nop_drop, + "Slots whose call runs are dropped entirely (proven-safe retire, SA[slot] stubbed) instead of retargeted to the CALL_MME_DATA port (u64 bitmask; default 0)"); + +/* + * SESSION 153-CONTINUED-82: hold real CALL runs until the prelude latch. + * The r1220 (all-real, nop_drop=1<<57, sa_stub_all=N) wedge: the window's + * real CALL runs executed in the SAME launch that emitted only the first + * PART of a fresh capture generation (`MME-PRELUDE part gen=3 off=1776 + * words=1776/3083 ... (continuation next launch)`), so the called macros + * fetched IR code that was not resident yet - `MME-FAST launch ... + * pc_valid=1 pc=513`, priv-error poison, `Shader Program Header 18 Error`, + * `Class Error ... Offset 0000238c, Data 00020020` (the r1116 lane-poison + * class) and the boot wedged. The prelude latch (gsp_phys_gr_mme_prelude_* + * gen/h) is set only when the FINAL part has gone out; with this knob the + * launch builder holds (elides, the proven-safe drop shape) every remaining + * CALL_MME_MACRO/DATA run of the emitted window while EITHER the latch for + * this generation+channel is not set after this launch, OR this launch + * uploaded generation code for the first time (latch invalid before the + * emission). The r1222 all-real boot proved the second clause necessary: + * the drain-completing launch (part-2 IR uploads + window real calls in the + * SAME block, latch set by its own emission) still hit ILLEGAL_OPCODE - + * real calls resume on the next launch, when only the idempotent keep-forced + * re-emission of the already-resident prefix is in flight. Default 1 (the + * fix); A/B with 0 to reproduce the r1209/r1220/r1222 wedge. + */ +static bool gsp_phys_rmcall_gr_mme_hold_calls = true; +module_param_named(sema_gl_mme_hold_calls, gsp_phys_rmcall_gr_mme_hold_calls, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_hold_calls, + "Hold (elide) window CALL_MME_MACRO/DATA runs until the MME prelude latches for the current generation+channel (default 1; 0 = legacy, real calls run against a partial prelude)"); + +/* + * SESSION 153-CONTINUED-84: emit NVC597_SET_MME_VERSION=1 (0x234) with the + * prelude step. The channel-init defaults the client pushes once per channel + * (Turing+) are never replayed, so the replayed context's MME stays in its + * reset/disabled state and every CALL_MME_MACRO run parks the front-end + * waiting for a macro that cannot start (r1245: call-bearing launches hang + * with all MME registers idle). Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_mme_version = true; +module_param_named(sema_gl_mme_version, gsp_phys_rmcall_gr_mme_version, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_version, + "Emit SET_MME_VERSION=1 ahead of the replayed MME prelude (default 1; the Turing channel-init method the guest pushes once and the capture never carries)"); + +/* + * SESSION 153-CONTINUED-86: the missing SPH (shader-program-header) state. + * + * The executed beef launches fault `Graphics Exception: Shader Program + * Header 9/11/18 Error` + `DS_HWW_ESR 0x405840=0xa2040a00` (r1272): the DS + * unit validates the header of every pipeline program bound when a draw + * issues, including the SPH VERSION against the channel's programmed + * window. The Turing channel-init (the 3D driver's own + * _nv3dInitChannelTuring) writes the SPH version pair ONCE per channel: + * + * INC run @0x16a4, count 2: + * METHOD 0x16a4 (SET_SPH_VERSION) <- 4 | (4 << 16) = 0x00040004 + * METHOD 0x16a8 (CHECK_SPH_VERSION) <- 0x00040004 + * + * (fields: CURRENT 15:0, OLDEST_SUPPORTED 31:16; the value 4 is the SPH V4 + * the Turing shader images carry). The replayed context never receives it: + * the capture families (MME prelude 0x114..0x124/0x234/0x550..0x568/ + * 0x3400..0x34fc, attr prelude 0x1100..0x11fc/0x2000..0x205c) do not cover + * 0x16a4, and the captured 3D windows contain no such write (verified in the + * emitted-window decode and the raw guest-RAM stream). With the register + * unset (0/0) every fetched SPH fails its version window -> per-SPH error + * bits -> 0 GR-STATE retires and no stores. The client's own emitted init + * pair was located verbatim in the guest pushbuffer image (guest PA + * 0x8672d5b4: `200205a9 00040004 00040004`), so the launch builder emits + * exactly that run ahead of the window (right after SET_OBJECT, beside the + * MME version write). 3 words, idempotent; emitted on every replayed 3D + * window. Default 1; A/B with 0 reproduces the SPH fault. + */ +static bool gsp_phys_rmcall_gr_sph_version = true; +module_param_named(sema_gl_sph_version, gsp_phys_rmcall_gr_sph_version, + bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_version, + "Emit the Turing channel-init SET_SPH_VERSION/CHECK_SPH_VERSION pair (0x16a4/0x16a8 <- 0x00040004) ahead of each replayed 3D window (default 1; A/B with 0 reproduces the Shader Program Header 9/11/18 fault)"); + +/* + * SESSION 153-CONTINUED-93: forced SPH version-window value. 0 = auto (the + * blob-derived window, see the emission site); nonzero = emit exactly this + * 32-bit value for both 0x16a4/0x16a8. Live A/B for the DS version-window + * hypothesis: the emitted 4/4 window may reject the fetched SPHs while a + * wider/older window (e.g. 0x00040003) accepts them. + */ +static u64 gsp_phys_rmcall_gr_sph_version_val; +module_param_named(sema_sph_version_val, gsp_phys_rmcall_gr_sph_version_val, + ullong, 0644); +MODULE_PARM_DESC(sema_sph_version_val, + "Forced SET_SPH_VERSION/CHECK_SPH_VERSION value (0 = auto from the loaded blobs)"); + +/* + * SESSION 153-CONTINUED-90: SET_SHADER_EXCEPTIONS (0x1528 <- 0). + * + * NVIDIA's own Turing 3D channel init (_nv3dInitChannelTuring) pushes this + * FIRST, with the comment "Disable shader exceptions. This matches OpenGL + * driver behavior." The client's captured init stream carries the write too + * (sema-pushraw `off=0x5a4 op=4 addr=0x1528 v0=0`). The replayed context + * never receives it: the emitted push (PB-FAULT-DUMP) goes SET_OBJECT -> + * SPH version pair -> MME version with no 0x1528 run, so the DS keeps + * raising "Graphics Exception: Shader Program Header 5/9/10/11 Error" on + * every draw whose fetched SPH it would tolerate with exceptions disabled - + * and the exception poisons the channel before any store retires. An + * executing r1294 launch names the consequence: the first post-exception + * method write reports `ChID 0001, Class 0000c597, Offset 0000022c, Data + * 00000800` (SET_GO_IDLE_TIMEOUT poisoned by the exception state). + * Emitted right after SET_OBJECT, exactly like the reference init order. + * Default 1; A/B with 0 restores the old behavior. + */ +static bool gsp_phys_rmcall_gr_shader_exc = true; +module_param_named(sema_gl_shader_exc, gsp_phys_rmcall_gr_shader_exc, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shader_exc, + "Emit SET_SHADER_EXCEPTIONS (0x1528) <- 0 after SET_OBJECT like _nv3dInitChannelTuring (default 1)"); + +/* + * SESSION 153-CONTINUED-91: clear DS_HWW_REPORT_MASK/_2 (0x405844/0x40584c) + * ahead of every replayed 3D window (host write + in-stream MME PRI R/W + * macro 5) so a DS SPH validation miss latches without poisoning the PRI + * block and the GSP-RM exception handler killing the channel. See the + * emission-site comment. Default 1; A/B with 0 restores the old behavior + * (every SPH miss = priv-error poison + channel wedge). + */ +static bool gsp_phys_rmcall_gr_ds_mask_clear = true; +module_param_named(sema_ds_mask_clear, gsp_phys_rmcall_gr_ds_mask_clear, + bool, 0644); +MODULE_PARM_DESC(sema_ds_mask_clear, + "Clear DS_HWW_REPORT_MASK/_2 ahead of replayed 3D windows (default 1; 0 = leave the all-report mask)"); + +/* + * SESSION 153-CONTINUED-91: emit the Turing program-region pair (the MME + * shadow scratch 0x78/0x79 = NVC397_SET_MME_PROGRAM_REGION) with the + * launch's own program family page as the shader-heap base, so the DS's + * SPH validation has a region to accept the fetched SPHs against. See the + * emission-site comment. Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_sph_region = true; +module_param_named(sema_gl_sph_region, gsp_phys_rmcall_gr_sph_region, + bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_region, + "Emit the Volta+ program-region pair (SET_MME_SHADOW_SCRATCH 0x78/0x79 <- program family page) ahead of replayed 3D windows (default 1)"); + +/* + * Runtime-selectable program-region base. 0 = auto (the launch's own + * program family page, the guest-VA heap base the real driver would use); + * a nonzero value forces that base - the engine-side shadow-VA space needs + * its own experiment (the SPH_INFO snapshot shows the DS fetched through + * shadow VAs 0x620900/0x620200/0x620000, so a guest-VA base may be the + * mismatch). Writable at runtime for a live A/B. + */ +static u64 gsp_phys_rmcall_gr_sph_region_base; +module_param_named(sema_sph_region_base, gsp_phys_rmcall_gr_sph_region_base, + ullong, 0644); +MODULE_PARM_DESC(sema_sph_region_base, + "Program-region base override (0 = launch program family page; nonzero = explicit engine-side base)"); + +/* + * SESSION 153-CONTINUED-93: additionally emit the legacy program-region + * pair (0x1608/0x160c) beside the MME shadow pair (see the emission site). + * Default 0 (A/B; the DS snapshot registers share the legacy pair's names). + */ +static bool gsp_phys_rmcall_gr_sph_region_legacy; +module_param_named(sema_sph_region_legacy, gsp_phys_rmcall_gr_sph_region_legacy, + bool, 0644); +MODULE_PARM_DESC(sema_sph_region_legacy, + "Also emit the legacy SET_PROGRAM_REGION_A/B (0x1608/0x160c) pair with the same base (default 0)"); + +/* + * SESSION 153-CONTINUED-87: the missing TESS_CONTROL/PIXEL program binds. + * The client's real channel state binds every pipeline stage once per + * context; the captured init-push image (guest RAM scrape) carries + * `20020825 [0, 04400200]` (j=2 TESS_CONTROL) and + * `20020855 [0, 04400000]` (j=5 PIXEL) beside the SPH version pair - + * but the captured per-frame window carries only the j=1 (VTG) pair + * (0x4400900/0x4400300). The DS validates the SPH of every pipeline + * stage the draw activates, so the unbound stages latch their header + * errors (DS_HWW_ESR 0xa2040a00 = SPH9|SPH11|SPH18) while the V3 blob + * passes wherever a stage IS bound. Emit both binds ahead of the + * window at the window's own program-family page (all four VAs share + * one 4K page: 0x4400000 j=5, 0x4400200 j=2, 0x4400300/0x4400900 j=1), + * pushed through the same relocation pass as the window - a MISS skips + * the bind entirely (never a raw client VA). Beef client channels only + * (the family layout is the client's). Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_sph_bind = true; +module_param_named(sema_gl_sph_bind, gsp_phys_rmcall_gr_sph_bind, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_bind, + "Emit the client's missing TESS_CONTROL (j=2) and PIXEL (j=5) pipeline program binds ahead of replayed 3D windows (program-family page of the window's prog VA; default 1)"); + +bool gsp_phys_rmcall_gr_sph_bind_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_bind); +} + +/* + * SESSION 153-CONTINUED-92: leave program-family operands at their ORIGINAL + * guest VAs when the operand's page is resolved and identity-mappable. + * + * The DS I-fetch reaches the delivered images through the shadow VAs the + * window pass picks (r1298-r1306: stages fetched at 0x620900/0x620200/ + * 0x620000) and rejects them with the bytes provably correct at those VAs + * (INLINE-RB rb=020a0481; canary proves the fetch follows the mapped page's + * content). The one variable left is the address space the DS validates in: + * the client programmed its own guest-VA program heap, and the shadow VAs + * (0x2xxxxx..0x6xxxxx pools) sit outside any region the client's state + * describes - a region/space mismatch rejects all bound stages regardless + * of content (rr=0/0, ds=a400e20 at every retire). With this knob the + * window pass first tries the Tier-1 identity mapper on the operand's own + * guest VA (launch page set first, the sema resolver else): when the guest + * VA is mapped to the resolved HPA, the operand is left verbatim, so the + * DS fetches and validates the SPH at the same VA space the client's heap + * uses. A page without an identity mapping keeps the shadow rewrite exactly + * as before (fail-open), and the identity-kept pairs still count toward the + * pass's relocated census so the SPH-BIND guard (orel == nrun) is stable. + * r1307 RESULT (experiment executed, negative): with the knob on, the + * beef lane's program pairs stayed at their guest VAs (0x4350900/0x4350f00/ + * 0x4350200/0x4350000, PROG-GUEST-VA receipts) and the engine faulted + * `MMU Fault: ENGINE GRAPHICS HUBCLIENT_PD faulted @ 0x0_04350000. FAULT_PTE + * ACCESS_TYPE_VIRT_READ` within 2.5 s of every such launch - the Tier-1 + * identity map installs (the PDE window is walked: FAULT_PTE, not + * FAULT_PDE) yet the engine still cannot fetch at the client's own VA. + * This reconfirms the r929 v2 lesson: guest VAs are not engine-fetchable on + * this replay channel, so the "validate in guest VA space" hypothesis is + * not testable by operand rewriting (the fetch dies before any DS check). + * Default 0 (baseline shadow rewrite); 1 reproduces the FAULT_PTE experiment. + */ +static bool gsp_phys_rmcall_gr_prog_guest_va; +module_param_named(sema_gl_prog_guest_va, gsp_phys_rmcall_gr_prog_guest_va, + bool, 0644); +MODULE_PARM_DESC(sema_gl_prog_guest_va, + "Leave program-family (0x2014-family) operands at their original guest VAs when identity-mappable (default 0; 1 = the r1307 FAULT_PTE experiment - guest VAs are not engine-fetchable, do not keep)"); + +/* + * SESSION 153-CONTINUED-84: SET_MME_SHADOW_RAM_CONTROL mode before the + * window. The vendor comment (mesh shader bind path) is explicit: + * "MME filtering swallows them all if old and new state values are the + * same" - METHOD_TRACK_WITH_FILTER (1) suppresses every method write whose + * value equals the shadowed one. Our replay repeats the same window every + * launch, so after the first replay every redundant write - including the + * CALL_MME_MACRO/CALL_MME_DATA parameter writes - matches the shadow and + * is swallowed; the macro call never reaches the MME (r1236-r1250: calls + * in the emitted window, all MME registers idle, some launches park the + * front-end). The mode-bearing writes live in the captured PRELUDE + * (0x124 x4 in the client's setup push), which the win_patch window sweep + * does not touch; emit METHOD_TRACK (0 = filtering OFF, semantics + * unchanged: the write simply applies) right before the SA re-arm and the + * window. Value = sema_gl_mme_track (0 = TRACK; 1/2/3 = other modes; -1 = + * do not emit). + */ +static int gsp_phys_rmcall_gr_mme_track; +module_param_named(sema_gl_mme_track, gsp_phys_rmcall_gr_mme_track, + int, 0644); +MODULE_PARM_DESC(sema_gl_mme_track, + "SET_MME_SHADOW_RAM_CONTROL mode emitted before each replayed window (0=TRACK filtering off (default); 1=TRACK_WITH_FILTER; 2=PASSTHROUGH; 3=REPLAY; -1=off)"); + +/* + * SESSION 153-CONTINUED-85: A/B knob for the SA re-arm (the r1237 fix). + * Before CONT-84 the calls in the replayed window resolved to the SA state + * the prelude left behind (its last captured SA write re-clobbers the slot; + * r1231: SA[34]=0 lands last, every draw call resolved to macro 0 and + * emitted nothing - and the r1228-phase launches retired). With the re-arm + * the calls resolve to the captured IR entries; every call-bearing launch + * since (r1245/r1250/r1265/r1270) parks or poisons the front-end. This knob + * restores the pre-arm resolution in one boot so the two states can be A/B'd + * without a rebuild (0 = do not re-point; default 1 = re-arm). + */ +static bool gsp_phys_rmcall_gr_mme_sa_rearm = true; +module_param_named(sema_gl_mme_sa_rearm, gsp_phys_rmcall_gr_mme_sa_rearm, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_sa_rearm, + "Re-emit LOAD_MME_START_ADDRESS_RAM pairs before each replayed window so CALL_MME_MACRO resolves to the captured SA entries (default 1; 0 = pre-arm resolution, A/B)"); + +/* + * The same stubbed-slot predicate the launch builder applies (see the + * sa_stub emit decision): a slot is stubbed when stubbing is on and either + * the all-slots knob is set, the capture holds no SA entry, or the IR + * instruction the entry names never entered the replay. A kept slot + * (sema_gl_mme_keep_mask) is never stubbed. + */ +static bool gsp_phys_gr_slot_stubbed(u32 j, u64 sa_mask, const u32 *sa_ent) +{ + if (j >= 64u || !gsp_sema_gr_sa_stub_value()) + return false; + /* + * SESSION 153-CONTINUED-80: a nop-drop slot is force-stubbed even + * when it is also kept or covered - the r1186 evidence shows a kept + * slot 57 loses its SA end-noop stub (the stub builder skips kept + * slots) and any call-run encoding the drop filter misses then + * executes the real macro 57 (the park class). Dropping the runs and + * stubbing SA[57] together is the safe combination. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_nop_drop) & (1ull << j)) + return true; + if (READ_ONCE(gsp_phys_rmcall_gr_keep_mask) & (1ull << j)) + return false; + if (READ_ONCE(gsp_phys_rmcall_gr_sa_stub_all)) + return true; + if (!(sa_mask & (1ull << j))) + return true; + return !gsp_sema_mme_ir_covered(sa_ent[j]); +} + +/* + * In-place filter for the relocated 3D window: remove every run that + * writes the CALL_MME_MACRO(j)/CALL_MME_DATA(j) method pair of a stubbed + * slot. Pure run-level removal (runs are self-contained: op 1/3/5 carry + * 1+cnt data words; 0/2/4/6 are single words), so the surviving method + * sequence is unchanged and both the F2A window walk and the engine see + * the same stream shape with the calls elided. + */ +static u32 gsp_phys_gr_mme_drop_calls(u32 *win, u32 n, u64 sa_mask, + const u32 *sa_ent, u64 *dropped_mask) +{ + u32 r = 0, w = 0, dropped = 0, transform = 0; + u64 dmask = 0; + + while (r < n) { + u32 h = win[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u, k, m; + bool iscall = false; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + /* + * Decode the touched methods the way the engine walks them: + * multi-word runs land on m + 4k; single-word entries carry + * their method in the low bits (IMMD and the SDM family both + * put it at 11:0 for our purposes). + */ + for (k = 0; k < span; k++) { + m = ((h & 0xfffu) << 2) + 4u * k; + if (k && op != 1u && op != 3u) + break; + if (m >= 0x3800u && m <= 0x39fcu) { + u32 j = (m - 0x3800u) / 8u; + + if ((READ_ONCE(gsp_phys_rmcall_gr_keep_mask) & + (1ull << j)) && + (READ_ONCE(gsp_phys_rmcall_gr_nop_param_mask) & + (1ull << j))) { + if (READ_ONCE(gsp_phys_rmcall_gr_nop_drop)) { + /* + * SESSION 153-CONTINUED-80: the + * retarget parks the PBDMA + * live (see the + * sema_gl_mme_nop_drop + * comment) - drop the whole + * run instead. + */ + iscall = true; + dmask |= 1ull << j; + break; + } + /* + * SESSION 153-CONTINUED-79: kept + + * nop-param slot - keep the run but + * retarget its header to the DATA + * port so the params still land + * (see the sema_gl_mme_nop_param_mask + * comment). + */ + u32 nm = 0x3804u + 8u * j; + + win[r] = (h & ~0xfffu) | + ((nm >> 2) & 0xfffu); + dmask |= 1ull << j; + transform++; + } else if (gsp_phys_gr_slot_stubbed(j, sa_mask, + sa_ent)) { + iscall = true; + dmask |= 1ull << j; + break; + } + } + } + if (iscall) { + dropped += span; + r += span; + continue; + } + if (w != r) + memmove(&win[w], &win[r], (size_t)span * 4u); + w += span; + r += span; + } + if (transform) + pr_info("vgsp-gr: MME-CALL-NOP %u call run(s) retargeted to the CALL_MME_DATA port (params conserved)\n", + transform); + if (dropped_mask) + *dropped_mask = dmask; + return dropped; +} + +/* + * SESSION 153-CONTINUED-61: generation of the MME prelude already emitted + * into the engine context (0 = none). The r1110 evidence: the prelude was + * emitted once on the first beef launch (gen=1, 512 words), that launch + * priv-errored (grfifo=0xbadf1201) and WEDGE-RECOVER ran TSG preempt + + * resched - which resets the engine context and with it the resident MME + * program - yet the latch kept suppressing re-emission, so every later + * window replay ran its macros against an unloaded MME (mme_pc=513 class + * stall, no retire). The recovery clears the latch so the next launch + * re-preloads the context. + * + * SESSION 153-CONTINUED-68: the latch is keyed by (generation, channel). + * A bare generation latch is SHARED across channels: the X channel's + * first post-maturity launch consumed the emission for the whole system + * and every beef launch then shipped without the program (r1140: only + * h=0x1000013 logged "MME-PRELUDE emitted gen=3", every beef launch + * words=400 prelude-less). Re-emitting per channel is idempotent. + */ +static u32 gsp_phys_gr_mme_prelude_gen; +static u32 gsp_phys_gr_mme_prelude_h; +static u32 gsp_phys_gr_mme_pre_full_dump; + +/* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude. The r1091 + * full-MME-prelude boot cleared MME (mme_esr=80000000) but the first beef + * window then latched a draw-shader program-header error + * (NV_PGRAPH_PRI_DS_HWW_ESR 0x405840 = 0xa2040a00: SPH9|SPH11|SPH18, + * M_PIPE_STAGE = VSB|PS) and a class error at SET_VERTEX_ATTRIBUTE_A(2) + * (0x1168, INVALID_ENUM): the replayed window is a mid-stream state cut - + * the attribute/pipeline record the client set in EARLIER windows (the + * missing stages' SET_PIPELINE_PROGRAM_ADDRESS binds and the attribute + * format state) was never replayed, so the DS validated program headers + * that were never bound. With this knob the launch builder preplays the + * captured runs from the service's attr prelude buffer (see + * svc_attr_prelude_capture), relocated exactly like the window itself + * (the patched copy is computed by the caller - it needs the launch page + * set and the live shadow counter), run-aligned, drained across the first + * launches of a capture generation. Declared before build_gr_launch (its + * reader); the matching capture-side knob is sema_attr_prelude_cap. + * Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_attr_prelude = true; +module_param_named(sema_gl_attr_prelude, gsp_phys_rmcall_gr_attr_prelude, + bool, 0644); +MODULE_PARM_DESC(sema_gl_attr_prelude, + "Preplay the captured client attr/pipeline runs (0x1100..0x11fc, 0x2000..0x205c, relocation applied) ahead of the first 3D window replays of a capture generation (default 1)"); + +/* + * The patched (relocation-applied) attr prelude for the current capture + * generation, built by the submit path just before build_gr_launch. Single + * caller (the ordered single-inflight sema work item), so plain statics. + */ +#define GSP_PHYS_GR_ATTR_PRE_WORDS 512u +static u32 gsp_phys_gr_attr_pre[GSP_PHYS_GR_ATTR_PRE_WORDS]; +static u32 gsp_phys_gr_attr_pre_n; +static u32 gsp_phys_gr_attr_pre_gen; /* capture gen the patch is from */ +static u32 gsp_phys_gr_attr_emit_gen; /* gen whose runs are draining */ +static u32 gsp_phys_gr_attr_emit_off; /* word cursor into attr_pre */ + +/* + * SESSION 153-CONTINUED-62: surface (render-target) prelude. The beef + * client's per-frame draw windows are mid-stream cuts: the colour-target + * bind is sticky context state (the client and the X server write it once + * per context), so the replayed windows carry none - and every replayed + * draw rasterized into an unbound target and stored nothing (the observed + * 0/N retire). The service captures the last C597 colour-target run from + * any trusted push (svc_surf_prelude_capture; the observed carrier is the + * X server's 1280x720 screen bind) and this site replays it - aliased and + * relocated exactly like the window - ahead of any window that carries no + * usable target bind of its own. Emitted per capture generation (the + * patched buffer is reused while the captured run is unchanged, and the + * alias VA is stable per boot); a fresh generation re-patches. A/B via + * the sema_gl_surf_prelude knob (default 1). + */ +#define GSP_PHYS_GR_SURF_PRE_WORDS 32u +static u32 gsp_phys_gr_surf_pre[GSP_PHYS_GR_SURF_PRE_WORDS]; /* patched */ +static u32 gsp_phys_gr_surf_pre_in[GSP_PHYS_GR_SURF_PRE_WORDS]; +static u32 gsp_phys_gr_surf_pre_n; /* patched words, 0 = none */ +static u32 gsp_phys_gr_surf_pre_gen; /* capture gen the patch is from */ +static bool gsp_phys_rmcall_gr_surf_prelude = true; +module_param_named(sema_gl_surf_prelude, gsp_phys_rmcall_gr_surf_prelude, + bool, 0644); +MODULE_PARM_DESC(sema_gl_surf_prelude, + "Replay the captured C597 colour-target bind ahead of 3D windows that carry no render target of their own (default 1)"); + +/* + * SESSION 153-CONTINUED-62: does this window already carry a usable + * colour-target bind? A run with va == 0 (the all-zero degenerate bind + * seen on the client's 0x402094c push) does not count - the surface + * prelude is injected for it too. + */ +static bool gsp_phys_gr_window_has_surface(const u32 *win, u32 n) +{ + u32 off = 0; + + while (off < n) { + u32 hdr = win[off]; + u32 op = (hdr >> 29) & 7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 addr = (hdr & 0x1fffu) << 2; + u32 span = 1u; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + off + 1u + cnt <= n) + span = 1u + cnt; + else { + off++; + continue; + } + if (op == 1u && addr >= 0x800u && addr < 0xc00u && + (addr & 0x3fu) == 0u && cnt >= 2u) { + u64 va = ((u64)(win[off + 1] & 0xffu) << 32) | + (u64)(win[off + 2] & 0xfffff000u); + + if (va) + return true; + } + off += span; + } + return false; +} + + +/* * GR launch method block: the inline QMD run at its original shape and the * channel REPORT_SEMAPHORE release. The stock guest push does NOT emit * SET_OBJECT before the inline run (the compute class is bound on the @@ -16274,19 +18545,1087 @@ static u32 *gsp_phys_rmcall_build_gr_init(u32 *p) static u32 gsp_phys_rmcall_build_gr_launch(u32 *pb, const struct gsp_phys_gr_submit *req, const u32 *qmd, u64 qmd_va, - u64 sem_va, u32 sem_payload) + u64 sem_va, u32 sem_payload, + const u32 *win3d, u32 win3d_n, + const u32 *attr_pre, + u32 attr_pre_n) { u32 *p = pb; int prelude = gsp_phys_rmcall_gr_prelude_value(); int qmdvirt = gsp_phys_rmcall_gr_qmdvirt_value(); int two_step = gsp_phys_rmcall_gr_pcas_two_step_value(); - p = gsp_phys_rmcall_build_gr_init(p); + /* + * r929 (Path A): a captured 3D method window replays verbatim. Bind + * TURING_A (0xC597) on the channel subchannel (the same bind the + * window itself carries - belt and braces, a rebind is harmless), + * then the window words (address operands relocated to shadow-pool + * VAs by gsp_phys_rmcall_gr_3d_patch_window), then the channel + * release tail. No compute init, no QMD, no PCAS: the engine walks + * real 3D methods (SET_OBJECT C597, state binds, LINE_LENGTH_IN + + * LAUNCH_DMA inline uploads, program/tex binds). Completion is the + * trailing REPORT_SEMAPHORE alone (the submit path leaves qmd_rel + * off for this mode). + */ + if (gsp_phys_rmcall_gr_3d && req && win3d && win3d_n && + win3d_n + 2u + 3u + 8u <= GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + /* + * SESSION 153-CONTINUED-53: MME context-setup prelude (see the + * sema_gl_mme_prelude knob above). Preplay the captured client + * setup runs once per capture generation, right after the + * SET_OBJECT C597 bind and ahead of the window, so + * CALL_MME_MACRO() runs execute against a loaded macro + * program instead of pc=0 -> ILLEGAL_OPCODE. The read is + * seqchecked (generation before/after): a concurrent append + * only defers emission to the next launch. A window too large + * to also carry the prelude replays without it (logged) - the + * emission then happens on the next launch that fits. + */ + bool emit_prelude = false; + bool pre_done = false; + bool mme_hold_calls = false; + bool latch_before = false; + u32 pre_words_out = 0; + const u32 *pl = NULL; + u32 pl_n = 0, pl_gen = 0, pl_room = 0; + u32 block_cap = READ_ONCE(gsp_phys_rmcall_gr_block_cap); + + if (block_cap < 64u || + block_cap > GSP_PHYS_RMCALL_GR_METHOD_WORDS) + block_cap = GSP_PHYS_RMCALL_GR_METHOD_WORDS; + /* + * SESSION 153-CONTINUED-86: harden the block cap against the + * emissions the block-cap arithmetic below does not fully + * count (the SA re-arm, 3 words per slot, up to 32 slots - + * see the MME-SA-REARM site). Clamping the cap keeps every + * combination inside the 4096-word method buffer; with the + * default 2048 cap this changes nothing. + */ + if (block_cap > GSP_PHYS_RMCALL_GR_METHOD_WORDS - 128u) + block_cap = GSP_PHYS_RMCALL_GR_METHOD_WORDS - 128u; + /* + * SESSION 153-CONTINUED-63: MME START-ADDRESS RAM completion. + * The prelude capture is capped at 512 words, so the client's + * SA table (macro slot -> IR address) may be incomplete; on a + * fresh context the missing slots read 0 and CALL_MME_MACRO(j) + * jumps to IR 0 - the slot-0 macro - which with the caller's + * args emitted LOAD_CONSTANT_BUFFER_OFFSET <- 0x20020 and + * poisoned the lane (r1116; verified offline with the MME64 + * simulator). Stubbing every CALLED slot the capture's SA + * table does not cover with a harmless end-noop makes the + * unknown call a no-op instead of a lane-killer. The called + * slots come from the window being emitted (IMMD writes to + * 0x3800+j*8 and INC/ONEINC runs writing any 0x3800..0x39f8 + * method, the .Macro half of each pair). + */ + u32 sa_ent[64]; + u64 sa_mask = 0; + u32 sa_stub[40]; + u32 sa_stub_ent[40]; + u32 sa_cov[40]; + u32 sa_cov_ent[40]; + u32 sa_nstub = 0, sa_ncov = 0, sa_block = 0; + + if (gsp_sema_gr_sa_stub_value() && win3d && win3d_n) { + u32 so = 0; + + sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + while (so < win3d_n && (sa_nstub < 40u || sa_ncov < 40u)) { + u32 h = win3d[so]; + u32 sop = (h >> 29) & 7u; + u32 scnt = (h >> 16) & 0x1fffu; + u32 sspan = 1u; + u32 k, slo, shi; + + if ((sop == 1u || sop == 3u || sop == 5u) && + scnt && so + 1u + scnt <= win3d_n) { + sspan = 1u + scnt; + slo = 0; + /* + * SESSION 153-CONTINUED-68: ONE_INC + * (op 5) writes only data[0] to the + * header method; the rest go to + * method+4 (the data port, see + * nv3dLoadConstantsHeader). Walking + * mthd+4k for op=5 invented ghost + * CALL_MME_MACRO slots (r1134/1135 + * list). + */ + shi = (sop == 5u) ? 0u : scnt - 1u; + } else if (sop == 4u) { + slo = 0; + shi = 0; + } else { + so++; + continue; + } + for (k = slo; k <= shi; k++) { + u32 mthd = ((h & 0xfffu) << 2) + 4u * k; + u32 j, t; + bool have = false; + + if (mthd < 0x3800u || mthd >= 0x3a00u || + (mthd & 7u)) + continue; + j = (mthd - 0x3800u) / 8u; + if (j >= 64u) + continue; + /* + * SESSION 153-CONTINUED-70: a kept slot + * (sema_gl_mme_keep_mask) is never + * stubbed - its real macro must run + * against the context exactly as the + * client's own SA/IR uploads left it. + * Logged in the covered list for the + * census; no stub pair is emitted. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_keep_mask) & + (1ull << j)) { + for (t = 0; t < sa_ncov; t++) + if (sa_cov[t] == j) + have = true; + if (!have && sa_ncov < 40u) { + sa_cov[sa_ncov] = j; + sa_cov_ent[sa_ncov] = + sa_ent[j]; + sa_ncov++; + } + continue; + } + /* + * SESSION 153-CONTINUED-68: a slot is only + * usable when the capture holds its SA entry + * AND the IR instruction that entry names + * actually entered the replay (see + * gsp_sema_mme_ir_covered); a last-wins/ + * truncated mismatch makes the engine fetch + * macro data that is not there + * (MISSING_MACRO_DATA). Called-and-covered + * slots are logged too (r1134 ILLEGAL_OPCODE + * attribution: which macros actually run). + */ + if (sa_mask & (1ull << j)) { + if (!READ_ONCE(gsp_phys_rmcall_gr_sa_stub_all) && + gsp_sema_mme_ir_covered(sa_ent[j]) && + !(READ_ONCE(gsp_phys_rmcall_gr_nop_drop) & + (1ull << j))) { + for (t = 0; t < sa_ncov; t++) + if (sa_cov[t] == j) + have = true; + if (!have && sa_ncov < 40u) { + sa_cov[sa_ncov] = j; + sa_cov_ent[sa_ncov] = + sa_ent[j]; + sa_ncov++; + } + continue; + } + } + for (t = 0; t < sa_nstub; t++) + if (sa_stub[t] == j) + have = true; + if (!have && sa_nstub < 40u) { + sa_stub[sa_nstub] = j; + sa_stub_ent[sa_nstub] = + (sa_mask & (1ull << j)) ? + sa_ent[j] : 0u; + sa_nstub++; + } + } + so += sspan; + } + if (sa_nstub) + sa_block = 8u + 3u * sa_nstub; + } + + /* + * SESSION 153-CONTINUED-68: full run-header census of the 3D + * window being replayed (bounded; first 4 windows). The + * r1134 ILLEGAL_OPCODE attribution needs the real call + * structure: which methods the window writes, how many data + * words each run carries, and the CALL_MME_MACRO/DATA + * pattern (op 1 INC vs op 5 ONE_INC) around them. + */ + { + static u32 win_dump_n; + static u32 win_dump_beef; + + /* + * SESSION 153-CONTINUED-69: the first-4 budget was + * consumed by the X channel's early windows in every + * boot, so the beef draw-class window (the one that + * faults) never got its run-header census. Two extra + * dumps are reserved for beef-family windows. + */ + if (win3d && win3d_n && + (win_dump_n < 4u || + (((req->h_object & 0xfffff000u) == 0xbeef0000u) && + win_dump_beef < 2u))) { + u32 wo = 0, wr = 0; + + win_dump_n++; + if ((req->h_object & 0xfffff000u) == 0xbeef0000u) + win_dump_beef++; + pr_info("vgsp-gr: MME-WINRUN dump h=0x%x push=0x%llx window=%u\n", + req->h_object, req->push_va, win3d_n); + while (wo < win3d_n && wr < 200u) { + u32 h = win3d[wo]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0x1fffu; + u32 addr = (h & 0xfffu) << 2; + u32 span = 1u; + u32 d0 = (wo + 1u < win3d_n) ? + win3d[wo + 1u] : 0u; + u32 d1 = (wo + 2u < win3d_n) ? + win3d[wo + 2u] : 0u; + + if ((op == 1u || op == 3u || op == 5u) && + cnt && wo + 1u + cnt <= win3d_n) + span = 1u + cnt; + pr_info("vgsp-gr: MME-WINRUN off=%x op=%u cnt=%u mthd=%x d0=%x d1=%x\n", + wo * 4u, op, cnt, addr, d0, d1); + wo += span; + wr++; + } + } + } + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_prelude)) { + pl = gsp_sema_mme_prelude_get(&pl_n, &pl_gen); + /* + * SESSION 153-CONTINUED-72: with keep_mask bits set a + * kept macro can execute, and it needs the MME program + * resident. The r1152 evidence: the prelude was carried + * by one launch of the generation only; when that launch + * did not complete (engine fetch NONE / timeout), the + * later kept-macro launches ran an UNLOADED program and + * priv-errored (ILLEGAL_OPCODE -> poison). Force + * re-emission on every launch while any slot is kept + * (idempotent per the emission contract). + * + * SESSION 153-CONTINUED-79: the emission is drained in + * per-launch parts capped at block_cap words (see the + * sema_gl_mme_block_cap comment; oversized blocks never + * start). The latch (gsp_phys_gr_mme_prelude_gen/h) is + * set only when the final part has gone out. + */ + if (pl && pl_n && pl_gen) { + /* + * Fixed overhead: SET_OBJECT bind (2), SA-stub + * block, window, and the 4x2-word release tail. + */ + u32 fixed = 2u + sa_block + win3d_n + 8u; + + /* + * SESSION 153-CONTINUED-82: latch state BEFORE + * this launch's emission (see the hold decision + * below). A launch that first uploads a + * generation's MME code must not run real CALL + * runs in the same block (the r1222 evidence: + * the drain-completing launch bundled part-2's + * IR uploads with the window's real calls and + * hit ILLEGAL_OPCODE even though the latch was + * set by its own emission step). + */ + latch_before = + (pl_gen == READ_ONCE( + gsp_phys_gr_mme_prelude_gen) && + req->h_object == READ_ONCE( + gsp_phys_gr_mme_prelude_h)); + + pl_room = (fixed + 24u <= block_cap) ? + block_cap - fixed : 0u; + if (pl_room > pl_n) + pl_room = pl_n; + /* + * (Re)start the drain on a new generation/ + * channel, or when the request is forced (kept + * slots; a context-wiping wedge recovery clears + * the latch) and nothing is left to drain. + */ + if (pl_gen != + READ_ONCE(gsp_phys_gr_mme_pre_emit_gen) || + req->h_object != + READ_ONCE(gsp_phys_gr_mme_pre_emit_h) || + (READ_ONCE(gsp_phys_gr_mme_pre_emit_off) >= + pl_n && + (READ_ONCE(gsp_phys_rmcall_gr_keep_mask) || + pl_gen != READ_ONCE( + gsp_phys_gr_mme_prelude_gen)))) { + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_gen, + pl_gen); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_h, + req->h_object); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + } + if (pl_room >= 24u) { + emit_prelude = + (pl_gen != READ_ONCE( + gsp_phys_gr_mme_prelude_gen) || + req->h_object != READ_ONCE( + gsp_phys_gr_mme_prelude_h) || + READ_ONCE( + gsp_phys_gr_mme_pre_emit_off) < + pl_n); + } else { + static u32 pre_defer_log; + + if (pre_defer_log < 8) { + pre_defer_log++; + pr_info("vgsp-gr: MME-PRELUDE deferred (window=%u prelude=%u sa=%u cap=%u) h=0x%x\n", + win3d_n, pl_n, sa_block, + block_cap, + req->h_object); + } + } + } + } + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | (1u << 16) | + (0u << 13) | + (GSP_PHYS_RMCALL_GR_SET_OBJECT >> 2); + *p++ = GSP_PHYS_RMCALL_CLASS_GR_3D; + /* + * SESSION 153-CONTINUED-90: SET_SHADER_EXCEPTIONS (0x1528) + * <- 0, the first write of NVIDIA's own Turing 3D channel + * init (see the sema_gl_shader_exc knob): with shader + * exceptions enabled the DS raises "Shader Program Header + * 5/9/10/11" on the draws whose SPHs it would otherwise + * tolerate, and the raised exception poisons the channel + * before any store retires. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_shader_exc)) { + static u32 shader_exc_log; + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x1528u >> 2); + *p++ = 0u; + if (shader_exc_log < 8u) { + shader_exc_log++; + pr_info("vgsp-gr: SHADER-EXC emitted 0x1528 <- 0 (2 words; pos=%u h=0x%x push=0x%llx)\n", + (u32)(p - pb), req->h_object, + req->push_va); + } + } + /* + * SESSION 153-CONTINUED-91: DS HWW report suppression. + * + * The DS validates the SPH of every bound pipeline program at + * draw time and reports failures through DS_HWW_REPORT_MASK + * (0x405844, observed live = 0x00ffffff = every SPH<n>_ERR + * reported; mask_2 = 0x77). A reported failure poisons the + * whole PRI block (reads return 0xbadf1201 - the priv-error + * marker already documented for the MME latch) and the GSP-RM + * exception handler kills the channel: every draw attempt + * wedges at the first SPH error storm even when the SPH bytes + * at the bound VAs are the client's own real V4 images (r1298: + * INLINE-RB rb=020a0481 at the bind VA, DS still 5/9/10/11). + * The reference driver's own diagnostic path (Aftermath) + * explicitly ENABLES these reports to capture them, i.e. the + * reports are a diagnostic tap, not the shader-execute gate. + * Clear the mask so a SPH miss latches (readable between + * faults) without poisoning the channel. Host write only: + * an in-stream CALL_MME_MACRO(5) (the NVN PRI R/W macro) + * faulted with MME ILLEGAL_OPCODE (pc=513 mac=0xf1) on + * r1300 - the Turing MME's loaded macro program does not + * carry that GF100-era slot - so the register is written + * directly (it persists: the next beef launch reads 0/0). + * A/B with sema_ds_mask_clear=0. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_ds_mask_clear)) { + static u32 ds_mask_log; + void __iomem *ds_bar0 = gsp_phys_gpu_bar0(); + u32 m0 = 0, m2 = 0; + + if (ds_bar0) { + m0 = ioread32(ds_bar0 + 0x00405844u); + m2 = ioread32(ds_bar0 + 0x0040584cu); + if (m0 || m2) { + iowrite32(0u, ds_bar0 + 0x00405844u); + iowrite32(0u, ds_bar0 + 0x0040584cu); + } + /* + * SESSION 153-CONTINUED-92: clear the DS HWW + * ESR latch (W1C) ahead of every window so the + * retire readback names EXACTLY this launch's + * failed SPH checks instead of the union of + * every prior draw's. The latch was observed + * sticky across launches (r1308: a6000e20 + * constant over 25 s of draws), which made + * "did this window's checks change?" unreadable. + * Log the value being cleared (the previous + * window's set) so the A/B sequence is in the + * ring either way. + */ + { + u32 esr = ioread32(ds_bar0 + 0x00405840u); + + if (esr) + iowrite32(esr, ds_bar0 + 0x00405840u); + if (ds_mask_log < 24u) { + ds_mask_log++; + pr_info("vgsp-gr: DS-ESR pre-clear=%08x (prev window's latch; this window starts clean) h=0x%x push=0x%llx\n", + esr, req->h_object, + req->push_va); + } + } + } + if (ds_mask_log < 24u) { + ds_mask_log++; + pr_info("vgsp-gr: DS-MASK clear mask=%08x/%08x rb=%08x/%08x h=0x%x push=0x%llx\n", + m0, m2, + ds_bar0 ? ioread32(ds_bar0 + 0x00405844u) : 0u, + ds_bar0 ? ioread32(ds_bar0 + 0x0040584cu) : 0u, + req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-86: the Turing channel-init SPH version + * pair (see the sema_gl_sph_version knob). Emitted on EVERY + * replayed 3D window (3 words, idempotent): the DS validates + * each bound program's SPH version against this window at draw + * time, and the replay context never received the client's + * once-per-channel write. Emitted before the prelude-room + * computations below so the live (p - pb) accounting covers + * it exactly. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_version)) { + /* + * SESSION 153-CONTINUED-87: the window's accepted + * version range must COVER the images the engine + * will fetch. The client's own init push writes + * 4|(4<<16), but our authored (V3) images are + * rejected by a 4/4 window (SPH9|11|18) while the + * r950-era acceptance - same V3 image - ran with + * NO version window. Emit oldest=V(blob), + * current=max(V(blob),4) so both the V3 image and + * a future V4 re-tag stay inside the range. + */ + u32 bv = (u32)gsp_sema_gl_sph_blob_version(); + u32 val = 0x00040004u; + u64 ov = READ_ONCE(gsp_phys_rmcall_gr_sph_version_val); + + if (ov) + val = (u32)ov; + else if (bv >= 1u && bv <= 15u) + val = ((bv < 4u ? 4u : bv) << 16) | bv; + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (2u << 16) | (0x16a4u >> 2); + *p++ = val; + *p++ = val; + } + /* + * SESSION 153-CONTINUED-91: the Turing program region. + * + * The DS's SPH validation checks each fetched SPH against the + * channel's PROGRAM REGION. On pre-Volta that was + * SET_PROGRAM_REGION_A/B (0x1608/0x160c); on Volta+ those + * methods are obsolete and the driver writes the same 64-bit + * shader-code-heap base through the MME shadow scratch pair + * (nvTuring: NVC397_SET_MME_PROGRAM_REGION = SET_MME_SHADOW_ + * SCRATCH(MME_SCRATCH_PROGRAM_REGION_A=0x78/B=0x79), + * gv100ResendProgramAddresses/`gf100Init`; method 0x3400 + + * i*4 -> 0x35e0/0x35e4). Observed live at the faulting draw: + * DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_A/B = 0/0 and DS latches + * SPH errors for every bound stage whose bytes are provably + * correct (r1298 INLINE-RB rb=020a0481 at the bind VA; + * r1303 GR-STATE sph stages 1/2/5 = 0x620900/0x620200/0x620000 + * with ds=a4000e20 and rr=0/0) - a region mismatch rejects all + * of them regardless of content. Emit the pair (A = upper + * 7:0, B = lower 31:0) with the launch's own program family + * page as the heap base. A/B with sema_gl_sph_region=0. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_region) && req->prog_va) { + static u32 sph_reg_log; + u64 base = READ_ONCE(gsp_phys_rmcall_gr_sph_region_base); + + if (!base) + base = req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x35e0u >> 2); + *p++ = (u32)(base >> 32) & 0xffu; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x35e4u >> 2); + *p++ = (u32)base; + if (sph_reg_log < 16u) { + sph_reg_log++; + pr_info("vgsp-gr: SPH-REGION emitted base=0x%llx (scratch 0x78/0x79) h=0x%x push=0x%llx\n", + base, req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-93: the LEGACY program-region pair + * (0x1608/0x160c = the pre-Volta SET_PROGRAM_REGION_A/B whose + * names the DS's own snapshot registers + * DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_A/B carry). Those + * snapshots read 0/0 on every SPH-faulting window while the + * MME shadow pair (0x78/0x79) is provably emitted, so the + * DS's validation region may be the legacy pair's state, not + * the MME scratch. The methods are "obsolete" on Volta+ (not + * in the c597 class table), so this is an A/B: a PBDMA that + * refuses them faults the launch visibly, an accepting one + * tells us via the latch. Knob sema_sph_region_legacy + * (default 0). Emitted beside the MME pair with the same + * base. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_region) && req->prog_va && + READ_ONCE(gsp_phys_rmcall_gr_sph_region_legacy)) { + static u32 sph_reg_leg_log; + u64 base = READ_ONCE(gsp_phys_rmcall_gr_sph_region_base); + + if (!base) + base = req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x1608u >> 2); + *p++ = (u32)(base >> 32) & 0xffu; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x160cu >> 2); + *p++ = (u32)base; + if (sph_reg_leg_log < 16u) { + sph_reg_leg_log++; + pr_info("vgsp-gr: SPH-REGION-LEGACY emitted base=0x%llx (0x1608/0x160c) h=0x%x push=0x%llx\n", + base, req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-84: SET_MME_VERSION=1 (0x234). + * + * The Turing channel-init defaults push + * `nvcMethodPusher: NVC597_SET_MME_VERSION <- 1` once per + * channel (class c597, TURING_OR_NEWER), and the replayed + * context otherwise never sees it: the captured MME setup + * families carry IR/SA/shadow-scratch loads but no 0x234 run + * (verified in the injected full-SA stream too). With the + * version register unset the MME never starts a macro - the + * r1245 evidence: call-bearing launches (real {0,2,34,39,40}, + * full 55-slot prelude, SA re-armed before the window) hang + * the front-end (`MME-FAST ... events=0` -> `GR LAUNCH FAILED: + * timeout after 2500 ms` with qmd_rel=0/ch_rel=0) while every + * MME register stays idle (`pc_valid=0`, `gen0=0`, + * `mme_esr=80000000`): the FE waits forever for a macro the + * halted MME cannot run. Emitted with the prelude step (the + * resident program is (re)loaded right after), ahead of the + * window, so the version select has landed before the first + * CALL_MME_MACRO run. Knob: sema_gl_mme_version (default 1; + * A/B with 0 reproduces the parked launches). + */ + if (emit_prelude && READ_ONCE(gsp_phys_rmcall_gr_mme_version)) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | + (0x234u >> 2); + *p++ = 1u; + } + if (emit_prelude) { + u32 g0 = pl_gen, g1 = 0, sdrop = 0, pwords = 0; + u32 doff = READ_ONCE(gsp_phys_gr_mme_pre_emit_off); + u32 room = (block_cap > (u32)(p - pb) + win3d_n + 8u + + sa_block) ? + block_cap - (u32)(p - pb) - win3d_n - 8u - + sa_block : 0u; + + /* + * SESSION 153-CONTINUED-79: run-aligned drain at the + * generation cursor; whole runs only, SDM-family + * single-word entries skipped exactly like the legacy + * sanitize (see gsp_phys_rmcall_gr_sdm_drop). + */ + while (doff < pl_n && pwords + 1u <= room) { + u32 h = pl[doff]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt && + doff + 1u + cnt <= pl_n) ? + 1u + cnt : 1u; + + if (doff + span > pl_n) + span = pl_n - doff; + if (READ_ONCE(gsp_phys_rmcall_gr_sdm_drop) && + h && (op == 0u || op == 2u || op == 6u)) { + sdrop += span; + doff += span; + continue; + } + if (pwords + span > room) + break; + memcpy(p, &pl[doff], (size_t)span * 4u); + p += span; + pwords += span; + doff += span; + } + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, doff); + pre_done = (doff >= pl_n); + pre_words_out = pwords; + /* seqcheck: a mid-copy append must not be recorded as + * emitted (the copy may be a torn prefix). */ + gsp_sema_mme_prelude_get(NULL, &g1); + if (pre_done && g1 == g0) { + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, g0); + WRITE_ONCE(gsp_phys_gr_mme_prelude_h, + req->h_object); + pr_info("vgsp-gr: MME-PRELUDE emitted gen=%u words=%u window=%u h=0x%x push=0x%llx sdm_drop=%u cap=%u\n", + g0, pl_n, win3d_n, req->h_object, + req->push_va, sdrop, block_cap); + /* + * SESSION 153-CONTINUED-70: one full dump of the + * replayed MME context-setup program per boot + * (the offline MME64 sim needs the same words the + * engine will execute). Also list the SA table + * with per-entry IR coverage so the slot-by-slot + * macro re-enable has its proof in one place. + */ + if (!gsp_phys_gr_mme_pre_full_dump) { + u32 k, sj; + u32 sa_ent[64]; + u64 sa_mask; + u32 cov_n = 0; + + gsp_phys_gr_mme_pre_full_dump = 1; + sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + for (sj = 0; sj < 64u; sj++) + if ((sa_mask & (1ull << sj)) && + gsp_sema_mme_ir_covered(sa_ent[sj])) + cov_n++; + pr_info("vgsp-gr: MME-PRE-DUMP words=%u gen=%u h=0x%x ir_end=0x%x sa_mask=0x%llx covered=%u code_words=%u keep_mask=0x%llx\n", + pl_n, g0, req->h_object, + gsp_sema_mme_ir_end_get(), + (unsigned long long)sa_mask, + cov_n, gsp_sema_mme_code_words_get(), + (unsigned long long)READ_ONCE(gsp_phys_rmcall_gr_keep_mask)); + for (k = 0; k + 4u <= pl_n; k += 4u) + pr_info("vgsp-gr: MME-PRE[%03u] %08x %08x %08x %08x\n", + k, pl[k], pl[k + 1], + pl[k + 2], pl[k + 3]); + for (sj = 0; sj < 64u; sj++) + if (sa_mask & (1ull << sj)) + pr_info("vgsp-gr: MME-PRE-SA[%u] entry=0x%x cov=%u\n", + sj, sa_ent[sj], + gsp_sema_mme_ir_covered(sa_ent[sj])); + } + } else if (pre_done) { + pr_info("vgsp-gr: MME-PRELUDE copy raced an append (gen %u->%u); will re-emit next launch\n", + g0, g1); + } else { + pr_info("vgsp-gr: MME-PRELUDE part gen=%u off=%u words=%u/%u window=%u h=0x%x push=0x%llx sdm_drop=%u cap=%u (continuation next launch)\n", + g0, doff, pwords, pl_n, win3d_n, + req->h_object, req->push_va, sdrop, + block_cap); + } + } + /* + * SESSION 153-CONTINUED-82: hold real CALL runs until the + * prelude latch for this generation+channel is set (see the + * sema_gl_mme_hold_calls comment). Evaluated AFTER the + * emission step above, so a launch that completed the drain + * in this same block has just written the latch. Real calls + * stay held when EITHER + * - the latch for this generation+channel is not valid after + * this launch (partial drain, raced append, or a fresh + * generation whose code is not fully resident), or + * - this launch uploaded generation code for the FIRST time + * (emitted words with the latch invalid before the step) - + * the r1222 evidence: the drain-completing launch bundled + * part-2's IR uploads with the window's real calls and hit + * ILLEGAL_OPCODE even though its own emission set the + * latch. A later launch (latch already valid, only the + * idempotent keep-forced re-emission of the prefix) runs + * the real calls against the now-resident program. + */ + if (pl && pl_n && pl_gen && + READ_ONCE(gsp_phys_rmcall_gr_mme_prelude) && + READ_ONCE(gsp_phys_rmcall_gr_mme_hold_calls)) { + u32 lg = READ_ONCE(gsp_phys_gr_mme_prelude_gen); + u32 lh = READ_ONCE(gsp_phys_gr_mme_prelude_h); + bool latch_after = (pl_gen == lg && + req->h_object == lh); + + mme_hold_calls = !latch_after || + (pre_words_out && !latch_before); + } + /* + * SESSION 153-CONTINUED-63: emit the SA-RAM completion computed + * above - a one-instruction end-noop stub in the IR plus one + * ONE_INC 0x11c run loading [slot, stub] for every called slot + * the capture's SA table does not cover (see the sa_stub + * comment at the emit decision). Emitted with the prelude (same + * generation latch), so it lands once per capture generation + * and the context keeps it for every later window. + */ + if (emit_prelude && sa_block) { + u32 t, stub_pc = gsp_sema_mme_ir_end_get(); + + /* + * SESSION 153-CONTINUED-68: the stub PC must be past the + * replayed program's last instruction - the window also + * calls COVERED slots, and a fixed PC inside a captured + * macro body would clobber its code. Fall back to 0xC0 + * when the capture reports nothing yet. + */ + if (stub_pc < GSP_PHYS_RMCALL_GR_MME_STUB_PC) + stub_pc = GSP_PHYS_RMCALL_GR_MME_STUB_PC; + /* + * SESSION 153-CONTINUED-69: explicit stub-PC override + * (sema_gl_mme_stub_pc). The ir_end-derived PC sits at + * the first instruction past the replayed program; if + * the MME's IR allocation/ONEINC pointer semantics + * differ (instruction vs word units, allocation + * boundary), the stub body may land where the engine + * does not look. This knob moves the stub (e.g. to + * 0xf0/0x100, or back to the 0xC0 floor) so the + * landing can be A/B-tested without a rebuild. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_stub_pc)) + stub_pc = READ_ONCE(gsp_phys_rmcall_gr_mme_stub_pc); + /* + * The IR upload MUST use the client's own SEC_OP (5, + * ONE_INC): with op=1 the method auto-increment walked + * the data words onto 0x11c/0x120 and clobbered SA[0] + * while writing no usable stub code (r1131). count = 7 + * = 1 pointer + 6 code words (two 3-word NOPs; see the + * gsp_phys_gr_mme_stub_code comment). + */ + *p++ = (GSP_PHYS_RMCALL_METHOD_ONEINC << 29) | + (7u << 16) | (0x114u >> 2); + *p++ = stub_pc; + *p++ = gsp_phys_gr_mme_stub_code[0]; + *p++ = gsp_phys_gr_mme_stub_code[1]; + *p++ = gsp_phys_gr_mme_stub_code[2]; + *p++ = gsp_phys_gr_mme_stub_code[3]; + *p++ = gsp_phys_gr_mme_stub_code[4]; + *p++ = gsp_phys_gr_mme_stub_code[5]; + /* + * Each [slot, stub] pair is its own INC 0x11c run: the + * pair methods are re-written per slot (a single longer + * run would walk onto 0x124 SetMmeShadowRamControl and + * beyond). + */ + for (t = 0; t < sa_nstub; t++) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (2u << 16) | (0x11cu >> 2); + *p++ = sa_stub[t]; + *p++ = stub_pc; + } + pr_info("vgsp-gr: MME-SA-STUB slots=%u words=%u pc=0x%x h=0x%x push=0x%llx\n", + sa_nstub, sa_block, stub_pc, req->h_object, + req->push_va); + { + char sb[192]; + u32 k, o = 0; + + sb[0] = '\0'; + for (k = 0; k < sa_nstub && o + 8u < sizeof(sb); k++) + o += scnprintf(sb + o, sizeof(sb) - o, + "%s%u:%x", k ? "," : "", + sa_stub[k], + sa_stub_ent[k]); + pr_info("vgsp-gr: MME-SA-STUB list=[%s]\n", sb); + } + { + char sb[192]; + u32 k, o = 0; + + sb[0] = '\0'; + for (k = 0; k < sa_ncov && o + 8u < sizeof(sb); k++) + o += scnprintf(sb + o, sizeof(sb) - o, + "%s%u:%x", k ? "," : "", + sa_cov[k], + sa_cov_ent[k]); + pr_info("vgsp-gr: MME-SA-COVERED list=[%s]\n", + sb); + } + } + /* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude (see the + * sema_gl_attr_prelude knob above). Emit the patched captured + * runs after the MME prelude and ahead of the window, + * run-aligned, draining across launches when the remaining + * block budget cannot take the whole remainder. The cursor + * resets when a new capture generation is patched. A window + * with no room at all defers the whole step to the next launch + * (logged by the patch site). + */ + if (attr_pre && attr_pre_n && + READ_ONCE(gsp_phys_rmcall_gr_attr_prelude)) { + u32 avail = block_cap - + (u32)(p - pb) - (win3d_n + 2u + 8u); + u32 off, copied = 0; + + /* + * SESSION 153-CONTINUED-79: the attr prelude shares + * the same per-launch block cap (see + * sema_gl_mme_block_cap); unsigned underflow would + * hand it the whole module buffer on an oversized + * window. + */ + if ((u32)(p - pb) + win3d_n + 2u + 8u >= block_cap) + avail = 0; + + if (gsp_phys_gr_attr_emit_gen != + READ_ONCE(gsp_phys_gr_attr_pre_gen)) { + WRITE_ONCE(gsp_phys_gr_attr_emit_gen, + READ_ONCE(gsp_phys_gr_attr_pre_gen)); + WRITE_ONCE(gsp_phys_gr_attr_emit_off, 0); + } + off = READ_ONCE(gsp_phys_gr_attr_emit_off); + while (off < attr_pre_n) { + u32 hdr = attr_pre[off]; + u32 op = (hdr >> 29) & 7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt) ? + 1u + cnt : 1u; + + if (off + span > attr_pre_n) + span = attr_pre_n - off; + /* + * SESSION 153-CONTINUED-59: the same SDM + * sanitize as the MME prelude (see + * gsp_phys_rmcall_gr_sdm_drop); the cursor + * still advances so the drop is permanent. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sdm_drop) && + hdr && (op == 0u || op == 2u || + op == 6u)) { + off += span; + continue; + } + if (copied + span > avail) + break; + memcpy(&p[copied], &attr_pre[off], + (size_t)span * 4u); + copied += span; + off += span; + } + if (copied) { + p += copied; + WRITE_ONCE(gsp_phys_gr_attr_emit_off, off); + pr_info("vgsp-gr: ATTR-PRELUDE emitted gen=%u words=%u/%u window=%u h=0x%x push=0x%llx%s\n", + READ_ONCE(gsp_phys_gr_attr_pre_gen), + off, attr_pre_n, win3d_n, + req->h_object, req->push_va, + off < attr_pre_n ? + " (continuation next launch)" : ""); + } else if (off < attr_pre_n) { + static u32 attr_defer_log; + + if (attr_defer_log < 8) { + attr_defer_log++; + pr_info("vgsp-gr: ATTR-PRELUDE deferred (no room: window=%u avail=%u) h=0x%x\n", + win3d_n, avail, + req->h_object); + } + } + } + /* + * SESSION 153-CONTINUED-84: shadow-RAM mode before the window + * (see the sema_gl_mme_track comment). METHOD_TRACK (0) is + * emitted by default so the repeated CALL/DATA parameter + * writes are not swallowed by the redundant-method filter. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_track) >= 0 && + p + 2u <= pb + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x124u >> 2); + *p++ = (u32)READ_ONCE(gsp_phys_rmcall_gr_mme_track); + } + /* + * SESSION 153-CONTINUED-84: SA re-arm BEFORE the window. + * The r1236 evidence: this block was emitted AFTER the + * window copy (contradicting its own "before the window" + * log), so the window's CALL_MME_MACRO(j) runs executed + * against the SA table state left by the prelude - whose + * last captured SA write re-clobbers the slot (r1231: + * SA[34]=0 lands last, every draw call resolved to macro + * 0 and emitted nothing). Because the prelude is + * re-emitted on every launch while any slot is kept, the + * late re-arm was re-clobbered before the next window + * anyway; the re-point only ever held between a window + * and the next prelude re-emission. Emitting the re-arm + * ahead of the window makes every launch's calls resolve + * to the captured SA entry. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_sa_rearm)) { + u32 sa_ent[64]; + u64 sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + u32 r = 0, rearm = 0; + + while (r < win3d_n && rearm < 32u) { + u32 h = win3d[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt && + r + 1u + cnt <= win3d_n) ? + 1u + cnt : 1u; + u32 mb = (h & 0xfffu) << 2; + + if (mb >= 0x3800u && mb <= 0x39fcu) { + u32 j = (mb - 0x3800u) / 8u; + + if (j < 64u && + (sa_mask & (1ull << j)) && + sa_ent[j]) { + *p++ = ((u32) + GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (2u << 16) | + (0x11cu >> 2); + *p++ = j; + *p++ = sa_ent[j]; + rearm++; + } + } + r += span; + } + if (rearm) + pr_info("vgsp-gr: MME-SA-REARM %u slot(s) re-pointed before the window (h=0x%x push=0x%llx)\n", + rearm, req->h_object, req->push_va); + } + if (mme_hold_calls) { + /* + * SESSION 153-CONTINUED-82: the prelude latch for + * this generation+channel is not set - elide every + * remaining CALL_MME_MACRO/DATA run of the window + * (the proven-safe drop shape; see the + * sema_gl_mme_hold_calls comment). Runs touching + * the MME macro method space 0x3800..0x39fc are the + * ones that execute a macro; the drop decision is + * the same walk gsp_phys_gr_mme_drop_calls uses. + */ + u32 r = 0, w = 0, held = 0; + + while (r < win3d_n) { + u32 h = win3d[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u; + bool iscall = false; + u32 k; + + if ((op == 1u || op == 3u || op == 5u) && + cnt && r + 1u + cnt <= win3d_n) + span = 1u + cnt; + for (k = 0; k < span; k++) { + u32 m = ((h & 0xfffu) << 2) + + 4u * k; + + if (k && op != 1u && op != 3u) + break; + if (m >= 0x3800u && m <= 0x39fcu) { + iscall = true; + break; + } + } + if (iscall) { + held += span; + r += span; + continue; + } + memcpy(&p[w], &win3d[r], (size_t)span * 4u); + w += span; + r += span; + } + if (held) + pr_info("vgsp-gr: MME-CALL-HOLD %u word(s) of real CALL runs held (prelude gen=%u not latched for h=0x%x; window=%u->%u) push=0x%llx\n", + held, pl_gen, req->h_object, + win3d_n, w, req->push_va); + p += w; + } else { + memcpy(p, win3d, (size_t)win3d_n * 4); + p += win3d_n; + } + /* + * SESSION 153-CONTINUED-84: the SA re-arm block used to be + * emitted HERE (after the window). It was moved ahead of the + * window above; see that block's comment for the r1236 + * evidence. An SA[slot]=entry re-point behind the call that + * needs it cannot help that call, and the per-launch + * prelude re-emission re-clobbers it before the next window. + */ + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_A, + (u32)(sem_va >> 32)); + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_B, + (u32)sem_va); + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_C, + sem_payload); + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_D, + GSP_PHYS_RMCALL_GR_SEM_RELEASE_D); + { + static u32 gr3d_log; + + if (gr3d_log < 16) { + gr3d_log++; + pr_info("vgsp-gr: 3D STREAM replay h=0x%x push=0x%llx class=0xc597 window=%u words=%u verbatim (no prelude/QMD)\n", + req->h_object, req->push_va, + win3d_n, (u32)(p - pb)); + } + /* + * SESSION 153-CONTINUED-45 diag: the exact block the + * PBDMA fetches (first/last window words). The beef + * family's replay schedule-stalls (grfifo frozen) while + * the X family's retires; this dump is the content A/B + * behind that divergence. Bounded per boot. + */ + if (gr3d_log < 16) { + u32 wi; + + pr_info("vgsp-gr: 3DSTREAM hdr=%08x %08x win0..9=", + pb[0], pb[1]); + for (wi = 0; wi < 10 && wi < win3d_n; wi++) + pr_cont("%08x ", win3d[wi]); + pr_cont("| tail="); + for (wi = (win3d_n > 10 ? win3d_n - 10 : 0); + wi < win3d_n; wi++) + pr_cont("%08x ", win3d[wi]); + pr_cont("\n"); + } + } + return (u32)(p - pb); + } + + /* + * r425 A/B: with gsp_phys_rmcall_gr_init_once the compute init preamble + * is emitted only on the first forwarded launch (the stock driver's + * per-channel shape); subsequent launches start at the inline-QMD + * window. + */ + if (!READ_ONCE(gsp_phys_rmcall_gr_init_once_param) || + !READ_ONCE(gsp_phys_rmcall_gr_init_emitted)) { + p = gsp_phys_rmcall_build_gr_init(p); + WRITE_ONCE(gsp_phys_rmcall_gr_init_emitted, true); + } if (qmdvirt) p = gsp_phys_rmcall_push_gr(p, GSP_PHYS_RMCALL_GR_SET_QMD_VIRT_CONTROL, GSP_PHYS_RMCALL_GR_QMD_VIRT_IQ2M_SEND_PCAS); /* + * r597: the prelude's acquire groups (if any) go before the QMD run + * so the engine cannot read the launch's data pages until the copy + * channel's semaphore has advanced. + */ + { + u32 w = gsp_phys_rmcall_gr_emit_acquires(p, req, + 128u); + static u32 acq_log; + + if (w) { + p += w; + if (acq_log < 16) { + acq_log++; + pr_info("vgsp-gr: prelude acquire groups emitted words=%u h=0x%x push=0x%llx\n", + w, req->h_object, req->push_va); + } + } + } + /* * Verbatim replay: copy the captured GRCE prelude and the guest's * own inline-QMD run, keeping the method headers, and relocate only * the ADDRESS_A/B data words and the 64 QMD words exactly like the @@ -16461,7 +19800,7 @@ static int gsp_phys_rmcall_gr_map_va_locked(void __iomem *bar0, u64 guest_va, pt_va = gsp_phys_chunk_page_va(pt, 0); if (!pt_va) return -EIO; - put_unaligned_le64(gsp_phys_rmcall_pte(hpa), + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa), pt_va + (size_t)pte_idx * 8); dma_wmb(); boot_stage.rmcall_gr_win[slot].pte[i].idx = pte_idx; @@ -16629,7 +19968,7 @@ static int gsp_phys_rmcall_gr_map_shadow_locked(u32 slot, u64 hpa, pt = gsp_phys_rmcall_guest_pt_page_va(page); if (!pt) return -EIO; - put_unaligned_le64(gsp_phys_rmcall_pte(hpa), + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa), pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * line_in_page) * 8); dma_wmb(); @@ -16676,7 +20015,7 @@ static int gsp_phys_rmcall_gr_map_shadow_contig_locked(u32 line, u32 pte, pt = gsp_phys_rmcall_guest_pt_page_va(page); if (!pt) return -EIO; - put_unaligned_le64(gsp_phys_rmcall_pte(hpa), + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa), pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * line_in_page + pte) * 8); dma_wmb(); @@ -16743,14 +20082,23 @@ static void gsp_phys_rmcall_gr_gpc_mmu_init(void __iomem *bar0) * the affine-relocated server-managed VA (the engine cannot walk the * client-managed guest VAs: UPDATE_PDE_2 refuses them with 0x1f). */ -static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, - const struct gsp_phys_gr_submit *req, - u32 bank, u32 *mirror) +/* + * relfix (agent-A FIX A): resolve-and-author one unmatched device-VA word. + * Defined after gsp_phys_gr_data_bi (the bounce-index helpers it needs); + * returns true when the word was rewritten to a reloc VA (and the pages + * queued for copy-in/out), false to leave it to the RELOC-GAP accounting. + */ +static bool gsp_phys_gr_reloc_fix_word(const struct gsp_phys_gr_submit *req, + u32 *w, u32 k, u32 *mirror); +static u32 gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, + const struct gsp_phys_gr_submit *req, + u32 bank, u32 *mirror, u64 *gap_first, + bool fix_enabled) { - u32 rew = 0, k; + u32 rew = 0, skipped = 0, gap = 0, k; if (!req->n_data) - return; + return 0; /* * m117: 8-byte stride with a 256-byte alignment guard. The m103 * 4-byte stride mis-matched arbitrary adjacent 4-byte words (the @@ -16763,19 +20111,6 @@ static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, u64 wb; u32 x; - /* - * m135: never relocate below offset 0x100. The param-page - * header (grid/block dims and driver bookkeeping at 0x0..0xff) - * can contain server-region VAs (e.g. 0x100000100) that are - * NOT kernel parameters; relocating them corrupted the - * vadd launch's blockDim word (read as 0x170100 instead of - * 0x100, so every CTA past 0 computed out-of-range indices - * and exited: VADD bad=32512 exactly one block correct). - * Kernel parameters live at >= 0x140 (JIT sm_75 layout: - * a/b/c at 0x160/0x168/0x170, n at 0x178). - */ - if (k * 4u < 0x100u) - continue; if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 0xff)) continue; wb = v & ~0xfffULL; @@ -16783,6 +20118,34 @@ static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, if (req->data[x].va != wb || !req->data[x].reloc_va) continue; + /* + * m135: never relocate below offset 0x100. The + * param-page header (grid/block dims and driver + * bookkeeping at 0x0..0xff) can contain + * server-region VAs (e.g. 0x100000100) that are NOT + * kernel parameters; relocating them corrupted the + * vadd launch's blockDim word (read as 0x170100 + * instead of 0x100, so every CTA past 0 computed + * out-of-range indices and exited: VADD bad=32512 + * exactly one block correct). Kernel parameters live + * at >= 0x140 (JIT sm_75 layout: a/b/c at + * 0x160/0x168/0x170, n at 0x178). + * + * m181 diagnostic: a below-0x100 value whose page is + * an operand VA of this launch is still logged (the + * observed class is the driver control block at + * bank offset 0, present in green clients too); the + * guard itself is unchanged until a dereferenced + * pointer in that range is proven. + */ + if (k * 4u < 0x100u) { + if (skipped < 16) { + skipped++; + pr_info("vgsp-gr: bank %u RELOC-SKIP @0x%x: 0x%llx is operand page 0x%llx (below 0x100, left guest-VA)\n", + bank, k * 4, v, wb); + } + break; + } /* m120: log the hit site so coverage is provable. */ if (rew < 16) pr_info("vgsp-gr: bank %u hit @0x%x: 0x%llx -> 0x%llx\n", @@ -16816,10 +20179,30 @@ static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, rew++; break; } + /* + * GRFIX FIX3: the inner scan ran out (x reached n_data) + * without matching req->data[x].va - the word passes the + * exact shape heuristic the match test above uses, so it + * will be left as an unrelocated guest VA in this page. + * relfix: first give the sema service's trusted resolver a + * chance to translate it; a word that resolves is authored + * as a late data page and rewritten to its reloc VA (the + * exact matched-word encoding). Only an unresolved word is + * counted as a gap for the caller's RELOC-GAP log/hold. + */ + if (x == req->n_data) { + if (fix_enabled && gsp_phys_gr_reloc_resolve_param && + gsp_phys_gr_reloc_fix_word(req, w, k, mirror)) + continue; + gap++; + if (gap_first && !*gap_first) + *gap_first = v; + } } if (rew) pr_info("vgsp-gr: bank %u: %u data pointers relocated\n", bank, rew); + return gap; } /* @@ -16973,6 +20356,54 @@ static struct gsp_phys_boot_chunk *gsp_phys_gr_data_chunk(void) } /* + * r452: scan the whole active GR data-bounce chunk for an 8-byte pattern + * (write "0x<pattern>" to /sys/module/vgsp/parameters/gsp_phys_gr_scan). + * Verifies by content where a lost engine store actually landed: the chunk + * is the only bulk memory the operand window PTEs map, so a store that + * executed but is absent from the writer's tracked pages must be findable + * here (a stale-PTE landing) or nowhere (a dropped store). + */ +static int gsp_phys_gr_scan_set(const char *val, + const struct kernel_param *kp) +{ + struct gsp_phys_boot_chunk *c = gsp_phys_gr_data_chunk(); + u64 pattern = 0; + u32 i, hits = 0; + + if (kstrtoull(val, 0, &pattern)) + return -EINVAL; + if (!c->pages) + return -ENODEV; + for (i = 0; i < c->pages; i++) { + const u8 *p = gsp_phys_chunk_page_va(c, i); + u32 off; + + if (!p) + continue; + for (off = 0; off + 8 <= GSP_PHYS_PAGE_SIZE; off += 8) { + if (get_unaligned_le64(p + off) == pattern) { + if (hits < 32) + pr_info("vgsp-gr: CHUNK-SCAN found 0x%llx at bi=%u off=0x%x (va=0x%llx)\n", + pattern, i, off, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + ((u64)i << 12) + off); + hits++; + } + } + } + pr_info("vgsp-gr: CHUNK-SCAN 0x%llx done: %u hit(s) over %u pages\n", + pattern, hits, c->pages); + return 0; +} + +static const struct kernel_param_ops gsp_phys_gr_scan_ops = { + .set = gsp_phys_gr_scan_set, +}; +module_param_cb(gsp_phys_gr_scan, &gsp_phys_gr_scan_ops, NULL, 0200); +MODULE_PARM_DESC(gsp_phys_gr_scan, + "Scan the GR data-bounce chunk for an 8-byte pattern (hex)"); + +/* * B-2: page index of a relocated operand VA inside the engine-visible data * region, replacing the five inlined (reloc_va - 0x110000) >> 12 derivations * and their MAIL_SLOT bounds. Grown region: VAs from @@ -17006,6 +20437,430 @@ static u32 gsp_phys_gr_data_bi(u64 reloc_va) } /* + * relfix: late-resolved data pages. A QMD word that matches no tracked data + * page yet resolves at rewrite time becomes a real operand page of this + * launch: it gets the next bounce-region index after the fill's tracked set, + * is copied in before the doorbell and copied back after the retire wait + * (the same m142 ordering as every tracked page), and the word is rewritten + * to its reloc VA - so the engine stops walking an untranslatable guest VA + * (r216 MMU FAULT_PDE class). Bounds are per-launch and fail-open: at the + * caps, or when no source answers, the word simply stays a RELOC-GAP (the + * pre-relfix behaviour). Static scratch: the single-inflight ordered + * gsp-phys-work item is the only caller, reset per submission exactly like + * bounce_csum_in. + */ +/* + * r564: minimum data records for a forwardable launch. Healthy vadd-class + * launches resolve 968-969 records (the client's whole arena); a launch + * with only a handful after the fill's drops has an unresolved argument + * set and must not be handed to the engine (see the submit guard). + * + * SESSION 127: the Vulkan inline-QMD launch (the C5C0 translator) is a + * compute dispatch whose operand set is one descriptor page and one + * parameter bank - a handful of records is healthy for it. Tunable module + * parameter (default 16 preserves the CUDA-era threshold); the translator + * cycles pass gsp_phys_gr_min_data=2. + */ +#define GSP_PHYS_GR_MIN_DATA_RECS 16u +static unsigned int gsp_phys_gr_min_data_param = GSP_PHYS_GR_MIN_DATA_RECS; +module_param_named(gsp_phys_gr_min_data, gsp_phys_gr_min_data_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_min_data, + "GR submit minimum data records (degenerate-set refusal threshold, 0..4096; 0 = explicit bypass for the no-external-data launch class; default 16)"); + +/* + * r801 (SESSION 143): 0 is an explicit BYPASS, not a clamp to 1. The + * Vulkan launch's single QMD pointer pair names its own QMD window and its + * params are scalars, so the launch has no external data by construction; + * the bare n_data>=min gate wrongly classifies it degenerate. The bypass is + * safe now that gsp_phys_chunk_page_va answers NULL for an unallocated + * chunk (r800): before that guard the empty-set post-mortem print + * dereferenced a garbage pointer and oopsed at + * gsp_phys_backend_guest_gr_submit+0x4394. The proper predicate (all + * candidates launch-internal-filtered, none unresolved => not degenerate) + * replaces the bare threshold next; until then 0 stays an explicit, + * opt-in debug/test value and the default remains 16. + */ +static unsigned int gsp_phys_gr_min_data_value(void) +{ + unsigned int v = READ_ONCE(gsp_phys_gr_min_data_param); + + if (v > 4096) + v = 4096; + return v; +} +#define GSP_PHYS_GR_GAP_MAX 4096u /* total late pages per launch */ +/* + * relfix PART 3 (r217 span containment): the late path authors ONLY the + * pages actually needed by the fixed word. r217 showed two fixed words + * (the launch's own param/program descriptor fields, which the fill skips + * as launch-internal) walking ~1242 fully-mapped neighbour pages each - + * 2484 late pages, ~10 MB copied in and out per launch, for zero engine + * writes - and the clients converted from fast-fail (~60 s verdict) to + * killed at 250 s. The span is now a small fetch line, not a region walk: + * the multi-page operand class is owned by the fill's own span pipeline + * (which the PART 3 candidate superset now feeds), and this path is a + * last-resort proxy for words the fill could not own. + */ +#define GSP_PHYS_GR_GAP_SPAN_MAX 16u /* contiguous pages per word max */ +#define GSP_PHYS_GR_GAP_WORD_MAX 32u /* distinct fixed words per launch */ +#define GSP_PHYS_GR_GAP_LOG_MAX 16u /* distinct gap VAs stage-logged */ +struct gsp_phys_gr_gap_page { + u64 va; /* device VA of the page (unique-proxy key) */ + u64 hpa; /* guest page copied in (results copied back to) */ + u64 reloc_va; /* engine-visible bounce-region VA of that page */ + u32 bi; /* bounce page index (n_data + late index) */ +}; +static struct gsp_phys_gr_gap_page + gsp_phys_gr_gap[GSP_PHYS_GR_GAP_MAX]; +static u32 gsp_phys_gr_gap_n; +struct gsp_phys_gr_gap_word { + u64 va; /* device VA of the fixed word (page base) */ + u64 reloc_va; /* reloc VA of its first page */ + u64 pa; /* resolved guest PA (logging) */ +}; +static struct gsp_phys_gr_gap_word + gsp_phys_gr_gap_word[GSP_PHYS_GR_GAP_WORD_MAX]; +static u32 gsp_phys_gr_gap_word_n; +static u64 gsp_phys_gr_reloc_fixed; /* words rewritten this launch */ +static u64 gsp_phys_gr_reloc_fixed_first; /* first fixed device VA */ +static u64 gsp_phys_gr_gap_logged[GSP_PHYS_GR_GAP_LOG_MAX]; +static u32 gsp_phys_gr_gap_logged_n; + +/* + * relfix: reloc VA of the i-th bounce-region index, region mode exactly as + * gsp_phys_gr_data_bi; the legacy window's small bound fails open. + */ +static bool gsp_phys_gr_data_reloc_va(u32 idx, u64 *va_out) +{ + if (gsp_phys_gr_data_region_value()) { + if (idx >= GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES) + return false; + *va_out = GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + ((u64)idx << 12); + return true; + } + if (idx >= GSP_PHYS_RMCALL_GR_MAIL_SLOT) + return false; + *va_out = GSP_PHYS_RMCALL_GR_BOUNCE_VA0 + + ((u64)idx << 12); + return true; +} + +/* + * relfix: append one late page at the next free bounce index (n_data + + * gap_n, the slot right after the fill's tracked set). Fail-open on the + * caps or an absent bounce page. + */ +static bool gsp_phys_gr_gap_add(u32 n_data, u64 va, u64 hpa, u64 *reloc_out) +{ + u32 gi = gsp_phys_gr_gap_n; + u32 idx = n_data + gi; + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u64 reloc_va; + + if (gi >= GSP_PHYS_GR_GAP_MAX) + return false; + if (!gsp_phys_gr_data_reloc_va(idx, &reloc_va)) + return false; + if (!dchunk->pages || !gsp_phys_chunk_page_va(dchunk, idx)) + return false; + gsp_phys_gr_gap[gi].va = va; + gsp_phys_gr_gap[gi].hpa = hpa; + gsp_phys_gr_gap[gi].reloc_va = reloc_va; + gsp_phys_gr_gap[gi].bi = idx; + gsp_phys_gr_gap_n = gi + 1u; + *reloc_out = reloc_va; + return true; +} + +/* + * relfix: is this device page already covered by a proxy this launch? + * Either the fill's tracked set (req->data, each page exactly once - the + * fill dedupes) or a late page queued by this fix. A second proxy for one + * guest page is unsound: the engine may author results through either + * slot, and the two copy-outs would race to write back to the same guest + * page, so one would clobber the other's results. The fix therefore never + * creates a second proxy (see the span rollback in + * gsp_phys_gr_reloc_fix_word). + */ +static bool gsp_phys_gr_page_tracked(const struct gsp_phys_gr_submit *req, + u64 va) +{ + u32 i; + + for (i = 0; i < req->n_data; i++) + if (req->data[i].va == va) + return true; + for (i = 0; i < gsp_phys_gr_gap_n; i++) + if (gsp_phys_gr_gap[i].va == va) + return true; + return false; +} + +/* + * relfix PART 3: one rate-limited log per distinct gap VA per launch + * (the dedup helper both gap diagnostics below share). + */ +static bool gsp_phys_gr_gap_log_once(u64 va) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_gap_logged_n; i++) + if (gsp_phys_gr_gap_logged[i] == va) + return false; + if (gsp_phys_gr_gap_logged_n < ARRAY_SIZE(gsp_phys_gr_gap_logged)) + gsp_phys_gr_gap_logged[gsp_phys_gr_gap_logged_n++] = va; + return true; +} + +/* + * relfix PART 3/4: one rate-limited RELOC-GAP-STAGE line per distinct gap VA + * per launch. `stage` is the resolver source chain (r.stage, e.g. + * "ipin:-2,pin:-2,lat:-2,map:-2,vas:-2,disc:-2,ledger:-2," - the resolving + * source ends in ":0,") for a word no source answered, or a short authoring + * refusal token ("span-overlap", "region-full", "wordmax-full", + * "base-tracked") for a word that resolved but was deliberately not + * authored. A FULL chain of -2 is classified as RELOC-GAP-UNMAPPED instead + * (see gsp_phys_gr_gap_unmapped_log): the sources are exhausted, so the word + * cannot be authored this launch; note the map stage's population is + * range-gated to [SEMA_VA_LO 4 GiB, SEMA_VA_HI 128 GiB) so its -2 is + * uninformative for VAs above that window - the vas/disc WALKS (49-bit + * full-range) are the authoritative absence signal. + */ +static void gsp_phys_gr_gap_stage_log(u64 va, const char *stage, int rc) +{ + if (!gsp_phys_gr_gap_log_once(va)) + return; + pr_info_ratelimited("vgsp-gr: RELOC-GAP-STAGE va=0x%llx stage=%s rc=%d\n", + va, stage, rc); +} + +/* + * relfix PART 4: definitive classification for a shape-passing word that NO + * resolver source answers (chain fully -2, r218: 0x760275000000, + * 0x760273000000, 0x3000000000, 0x17400000000, 0x17c00000000 - every stage + * -ENOENT). The walks cover the full 49-bit range, so a walk miss + no + * pin/cache/ledger entry means no mapping we can see exists anywhere; + * r218 additionally shows these words produce NO engine MMU fault (unlike + * the dereferenced argument words the late path now fixes), i.e. the engine + * does not consume them: they are descriptor/metadata values that are merely + * VA-shaped ("junk" as pointers). Any remaining engine fault is therefore + * NOT from this class - it redirects diagnosis to the code-pointer/entry + * word and page-content paths. + */ +static void gsp_phys_gr_gap_unmapped_log(u64 va) +{ + if (!gsp_phys_gr_gap_log_once(va)) + return; + pr_info_ratelimited("vgsp-gr: RELOC-GAP-UNMAPPED va=0x%llx (all stages -2; VA-shape junk)\n", + va); +} + +/* + * relfix PART 4: bounded content bound for a late-resolved word whose page is + * one of the launch's OWN pages (req->pages[], the only class with ground + * truth). The late copy-in itself is a direct copy of the resolved source + * page (gsp_phys_gr_fill_gap_bounce: src = __va(r.hpa)), so a dst-vs-src + * compare is vacuous; the meaningful question is whether the RESOLVER named + * the same page the launch pipeline uses. Compare the resolved source page + * against req->pages[i].hpa content and log the mission's + * LATE-CONTENT-MISMATCH line once per launch when they differ: that is the + * only way the engine could have been handed wrong bytes through a late + * proxy. Words whose page is not launch-internal have no ground truth and + * are skipped (their resolution is validated by the same walk/pin sources as + * every tracked data page). + */ +static void gsp_phys_gr_late_content_check(const struct gsp_phys_gr_submit *req, + u64 wb, u64 resolved_hpa, u32 off) +{ + u32 pi; + + for (pi = 0; pi < req->n_pages; pi++) { + const u8 *gt, *src; + + if ((req->pages[pi].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != wb) + continue; + if (!req->pages[pi].hpa) + break; + gt = (const u8 *)__va(req->pages[pi].hpa); + src = (const u8 *)__va(resolved_hpa); + if (!gt || !src) + break; + if (memcmp(gt, src, GSP_PHYS_PAGE_SIZE) && + gsp_phys_gr_gap_log_once(wb)) + pr_info_ratelimited("vgsp-gr: LATE-CONTENT-MISMATCH word_va=0x%llx page=0x%llx off=%u src=0x%llx\n", + wb, (u64)req->pages[pi].hpa, off, + resolved_hpa); + break; + } +} + +/* + * relfix: resolve and author one unmatched word (see gsp_phys_gr_gap_page). + * The base VA resolves through the sema service's trusted resolver; on a hit + * the page is queued and the word rewritten to <reloc VA> | offset - the + * exact encoding the matched-word relocation uses - and up to + * GSP_PHYS_GR_GAP_SPAN_MAX contiguous pages the resolver also answers are + * queued behind it so a small operand's base+offset reads stay contiguous. + * PART 3 (r217): that span is a small fetch line (16 pages), NOT a region + * walk - r217's 1242-page spans turned fast fails into 250 s kills; the + * multi-page operand class belongs to the fill's own span pipeline, which + * the PART 3 candidate superset now feeds. A repeated word (same base VA + * authored in another QMD page) reuses its recorded reloc VA. NEVER guesses: + * only a successful resolve rewrites, and a page that already has a proxy + * (the fill's or an earlier fix's) stops the fix - the queued pages of THIS + * word are rolled back and the word stays a RELOC-GAP - so one guest page + * can never gain a second copy-out writer (the clobber class + * gsp_phys_gr_page_tracked describes). Every word that stays a gap is + * stage-logged once (RELOC-GAP-STAGE) with the resolver chain or the + * authoring refusal token. + */ +static bool gsp_phys_gr_reloc_fix_word(const struct gsp_phys_gr_submit *req, + u32 *w, u32 k, u32 *mirror) +{ + u64 v = (u64)w[k] | ((u64)w[k + 1] << 32); + u64 wb = v & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + struct gsp_sema_gr_reloc r; + struct gsp_phys_gr_gap_word *gw = NULL; + u32 gap_start = gsp_phys_gr_gap_n; + u64 reloc_va = 0; + u32 wi, i; + + for (wi = 0; wi < gsp_phys_gr_gap_word_n; wi++) { + if (gsp_phys_gr_gap_word[wi].va == wb) { + gw = &gsp_phys_gr_gap_word[wi]; + break; + } + } + if (!gw) { + bool overlap = false; + u64 base_pa; + + if (gsp_phys_gr_gap_word_n >= GSP_PHYS_GR_GAP_WORD_MAX) { + gsp_phys_gr_gap_stage_log(wb, "wordmax-full", + -ENOSPC); + return false; + } + if (gsp_phys_gr_page_tracked(req, wb)) { + gsp_phys_gr_gap_stage_log(wb, "base-tracked", -EEXIST); + return false; + } + if (gsp_sema_gr_reloc_resolve(wb, &r)) { + /* + * A FULL chain of misses is the definitive + * no-mapping-anywhere class (PART 4); anything else + * (hpa conversion failure) keeps the stage chain. + */ + if (r.stage_rc == -ENOENT) + gsp_phys_gr_gap_unmapped_log(wb); + else + gsp_phys_gr_gap_stage_log(wb, r.stage, + r.stage_rc); + return false; + } + base_pa = r.pa; + /* + * PART 4: the copy-in is a direct copy of this source page; + * for a launch-internal word, bound it against the launch's + * own page content (the LATE-CONTENT-MISMATCH witness). + */ + gsp_phys_gr_late_content_check(req, wb, r.hpa, + (u32)(v & (u64)(GSP_PHYS_PAGE_SIZE - 1))); + if (!gsp_phys_gr_gap_add(req->n_data, wb, r.hpa, &reloc_va)) { + gsp_phys_gr_gap_stage_log(wb, "region-full", -ENOSPC); + return false; + } + { + u32 span_done = 1; /* pages authored this word */ + bool span_capped = false; + + for (i = 1; i < GSP_PHYS_GR_GAP_SPAN_MAX; i++) { + u64 va = wb + ((u64)i << 12); + u64 ignored = 0; + + /* + * A page with an existing proxy cannot get a + * second one (two copy-out writers for one + * guest page would clobber each other); stop + * the span before it. A merely unresolved + * page (the common end-of-buffer case) + * truncates the span like the fill's own span + * walk does. + */ + if (gsp_phys_gr_page_tracked(req, va)) { + overlap = true; + break; + } + if (gsp_sema_gr_reloc_resolve(va, &r)) + break; + if (!gsp_phys_gr_gap_add(req->n_data, va, r.hpa, + &ignored)) { + /* capacity cut: tail cannot be + * authored though more mapping may + * exist - reported below. */ + span_capped = true; + break; + } + span_done++; + } + /* + * COPYAUDIT completeness probe (2026-10-18): the + * span cap is intentional (r217 containment), but a + * word cut at the cap with a still-resolvable next + * page means the engine's base+offset reads past + * the authored run into later operands' bounce + * pages - silent partial data. One probe page past + * the cut proves it and makes the truncation loud. + */ + if (!overlap && span_done == GSP_PHYS_GR_GAP_SPAN_MAX) { + struct gsp_sema_gr_reloc r2; + u64 next_va = wb + + ((u64)GSP_PHYS_GR_GAP_SPAN_MAX << 12); + + if (!gsp_sema_gr_reloc_resolve(next_va, &r2)) + span_capped = true; + } + if (span_capped) + pr_info_ratelimited("vgsp: COPY-TRUNC va=0x%llx want>=0x%llx done=0x%llx path=reloc-fix-span\n", + wb, + ((u64)span_done + 1) << 12, + (u64)span_done << 12); + } + if (overlap) { + /* + * Roll the whole word back (base included) and + * leave it a RELOC-GAP: a partial run would make + * the engine's next base+offset read land on a + * later word's page. + */ + gsp_phys_gr_gap_stage_log(wb, "span-overlap", -EEXIST); + gsp_phys_gr_gap_n = gap_start; + return false; + } + gw = &gsp_phys_gr_gap_word[gsp_phys_gr_gap_word_n]; + gw->va = wb; + gw->reloc_va = gsp_phys_gr_gap[gap_start].reloc_va; + gw->pa = base_pa; + gsp_phys_gr_gap_word_n++; + } + + v = gw->reloc_va | (v & (u64)(GSP_PHYS_PAGE_SIZE - 1)); + w[k] = (u32)v; + w[k + 1] = (u32)(v >> 32); + if (mirror) { + mirror[k] = (u32)v; + mirror[k + 1] = (u32)(v >> 32); + } + gsp_phys_gr_reloc_fixed++; + if (!gsp_phys_gr_reloc_fixed_first) + gsp_phys_gr_reloc_fixed_first = wb; + pr_info_ratelimited("vgsp-gr: RELOC-FIXED va=0x%llx pa=0x%llx off=%u\n", + wb, gw->pa, (u32)(v & (u64)(GSP_PHYS_PAGE_SIZE - 1))); + return true; +} + +/* * m151: output retirement wait. qmd_rel is a schedule/fetch ack (the front * end latched and fetched the QMD), not a data-path retirement: the SM's * stores into the bounce pages can still be in flight when the ack lands. @@ -17026,26 +20881,24 @@ static u32 gsp_phys_gr_data_bi(u64 reloc_va) * noinline so this rarely-taken wait does not grow the submit frame, which * is already near the warning bound. */ +/* + * m180: scan the request's unique tracked bounce pages and return how many + * differ from the copy-in checksum baseline. `hash_out` (optional) receives + * an FNV-1a fold of the per-page checksums - the content fingerprint the + * settle rule needs so a store ramp with a constant changed-page COUNT can + * never read as settled again. Static scratch: the only caller is the + * single-inflight ordered gsp-phys-work item (same discipline as the other + * submit-scope statics). + */ static noinline u32 -gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, - const u32 *csum_in) +gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, + const u64 *csum_in, u32 *n_tracked, u64 *hash_out) { - /* - * B-2: one entry per page of the grown region (64 KB), static like the - * other submit-scope scratch: the only caller is the single-inflight - * ordered work item (see the static scratch note in - * gsp_phys_backend_guest_gr_submit). 64 KB must not sit on the kernel - * stack (this frame was already near the warning bound at 205 pages). - */ static u32 checked[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); - u32 n_all = 0, di, iter; - u32 changed = 0, prev = 0xffffffffu; - const char *why = " (TIMEOUT)"; - unsigned long wait_start = jiffies; + u32 n_all = 0, di, j, changed = 0; + u64 hash = 1469598103934665603ULL; - if (!req->n_data) - return 0; for (di = 0; di < req->n_data; di++) { u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); u32 k; @@ -17061,29 +20914,167 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, if (!dup) checked[n_all++] = bi; } + /* + * relfix: late-resolved pages are tracked exactly like fill-tracked + * data pages, so the retire wait, the late-check and the NOWRITE + * classification see stores the engine authors into them. + */ + for (di = 0; di < gsp_phys_gr_gap_n; di++) { + u32 bi = gsp_phys_gr_gap[di].bi; + u32 k; + bool dup = false; + + if (bi >= GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES) + continue; + for (k = 0; k < n_all; k++) + if (checked[k] == bi) { + dup = true; + break; + } + if (!dup) + checked[n_all++] = bi; + } + if (n_tracked) + *n_tracked = n_all; + for (j = 0; j < n_all; j++) { + const u8 *src = gsp_phys_chunk_page_va(dchunk, checked[j]); + u64 fp; + + if (!src) + continue; + /* + * r452: FNV-1a per page, not a 32-bit additive sum. The + * add-sum was PROVABLY blind to a power-of-two in-place + * scale: doubling every word of a 4 KB page shifts each + * exponent by +1, i.e. +2^23 per word, and 1024 words give + * 2^33 == 0 (mod 2^32) - so c[i] *= 2.0f left the checksum + * bit-identical, the page classified "unchanged", and the + * client read the pre-scale values (the deterministic + * streams2/scale class). FNV-1a has no such structure. + */ + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp != csum_in[checked[j]]) + changed++; + hash ^= fp; + hash *= 1099511628211ULL; + } + if (hash_out) + *hash_out = hash; + return changed; +} + +/* + * r434: post-launch deposit dump (hardware-only words). The r430 QMD64 dump + * runs at submission, before the engine touches the deposit, so the + * hardware-only fields (w50 INNER_GET, w51 INNER_PUT, w52 span-list head, + * w53 next-QMD) are always zero there. Reading the same page AFTER the + * retire wait shows what the SKED actually consumed: w50/w51 are the CTA + * (INNER) get/put counters, so a wedged launch whose grid "completed" with + * zero stores is distinguishable from a genuinely dispatched grid by + * whether INNER_PUT advanced to the CTA count. One line per launch, gated + * by the skedrec param. + */ +static void gsp_phys_gr_qmd64_post(const u8 *deposit, const char *tag) +{ + char qb[64 * 9 + 1]; + int qo = 0; + u32 i; + + if (!deposit || !gsp_phys_rmcall_gr_skedrec_value()) + return; + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS && qo < (int)sizeof(qb) - 10; + i++) + qo += scnprintf(qb + qo, sizeof(qb) - qo, "%08x ", + get_unaligned_le32(deposit + i * 4)); + pr_info("vgsp-gr: QMD64-POST %s %s\n", tag, qb); +} + +static noinline u32 +gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, + const u64 *csum_in, u32 *n_tracked) +{ + u32 n_all = 0, iter; + u32 changed = 0; + u64 prev_hash = 0; + const char *why = " (TIMEOUT)"; + unsigned long wait_start = jiffies; + + /* + * m179: report how many pages were actually trackable. "wrote == 0" + * from a launch with n_tracked == 0 (no data entry resolves to a + * bounce page) is not a no-execution signature and must not trigger + * the no-write retry/accept path. + */ + if (n_tracked) + *n_tracked = 0; + if (!req->n_data) + return 0; + /* + * r343: scan_pages RETURNS the changed count and reports the tracked + * total through the out-param. The old call put the return (changed) + * into n_all while the tracker total went to *n_tracked, so the + * loop's `changed >= n_all` was `changed >= changed` - always true + * after the first scan (observed: "1/1 pages changed after 4 ms"), + * and the `!n_all` early-return also skipped the no-write settle + * when zero pages changed. Net effect: the copy-out shipped the + * bounce mid-ramp, so a cold engine (slowest store ramp) produced + * all-bad first runs (the matmul cold flake) and late stores could + * be dropped. Fix: tracked total into n_all; the content-stable + * rule below becomes the real settle bound. + */ + (void)gsp_phys_gr_scan_pages(req, csum_in, &n_all, NULL); + /* + * r405: the r343 fix moved the tracked total from the out-param to + * the local n_all but FORGOT to write it back, so *n_tracked stayed + * 0 for every launch and the whole no-write classification chain in + * the caller (`wrote == 0 && n_tracked`: first-attempt witness, + * outcome retries, NOWRITE accept + release restore, late-store + * check) has been dead since ed91540a/r345 - the r404 forensics + * show "RETIRE-WIT n_data=968 n_tracked=0" while the same wait + * reports n_all=1032 in the "0/1032 pages changed (no writes)" + * line. Propagate the tracked total so the classifier can fire. + */ + if (n_tracked) + *n_tracked = n_all; + /* + * RETIRE-WIT (r227): the r227 capture shows n_tracked far below the + * copy-in/copy-out rec counts for the same launch (4 tracked vs 1482 + * written back) - either the rec array genuinely aliases reloc VAs + * mid-launch or the observed kernel differs from this source. One + * line per launch proves which: n_data, the unique-bi count, and the + * first/last recs' va/reloc_va/bi must be sequential and in-region + * for a healthy launch. + */ + if (gsp_phys_gr_copywit_enabled()) + pr_info_ratelimited("vgsp-gr: RETIRE-WIT n_data=%u n_tracked=%u first_va=0x%llx first_rel=0x%llx first_bi=%u last_va=0x%llx last_rel=0x%llx last_bi=%u\n", + req->n_data, n_tracked ? *n_tracked : 0, + req->data[0].va, req->data[0].reloc_va, + gsp_phys_gr_data_bi(req->data[0].reloc_va), + req->data[req->n_data - 1].va, + req->data[req->n_data - 1].reloc_va, + gsp_phys_gr_data_bi(req->data[req->n_data - 1].reloc_va)); if (!n_all) return 0; for (iter = 0; iter < 300u; iter++) { - u32 j; + u64 hash = 0; - changed = 0; - for (j = 0; j < n_all; j++) { - const u8 *src = gsp_phys_chunk_page_va( - dchunk, checked[j]); - u32 ci, csum = 0; - - if (!src) - continue; - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != csum_in[checked[j]]) - changed++; - } + changed = gsp_phys_gr_scan_pages(req, csum_in, NULL, &hash); if (changed >= n_all) { why = ""; break; } - if (changed > 0 && changed == prev && iter >= 3u) { + /* + * m180 (H3): a partial count must be CONTENT-stable for + * ~300 ms before it counts as settled. The m162 count-only + * rule accepted a continuing store ramp whose page count + * happened to be constant (the always-failing class's + * "17/32 settled" copy-outs), and accepted an + * un-dispatched grid as settled. A genuinely masked/short + * kernel costs at most 300 ms extra; a ramping engine gets + * the time its stores need. + */ + if (changed > 0 && hash == prev_hash && + iter >= gsp_phys_gr_settle_iters()) { why = " (settled)"; break; } @@ -17091,32 +21082,225 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, why = " (no writes)"; break; } - prev = changed; + prev_hash = hash; usleep_range(4000, 6000); } pr_info("vgsp-gr: output retirement wait: %u/%u pages changed after %u ms%s\n", changed, n_all, jiffies_to_msecs(jiffies - wait_start), why); + /* + * r405: run-once late-write probe (see the param comment above). + */ + if (READ_ONCE(gsp_phys_gr_late_probe_param) && changed < n_all) { + u32 late_n = 0, late; + unsigned long lp_start = jiffies; + + msleep(500); + late = gsp_phys_gr_scan_pages(req, csum_in, &late_n, NULL); + pr_info("vgsp-gr: retire LATE-PROBE %u -> %u of %u after +%u ms%s\n", + changed, late, late_n, + jiffies_to_msecs(jiffies - lp_start), + late != changed ? + " (ENGINE STILL WRITING - stores landed late)" : + " (stable - writes are final)"); + } return changed; } /* + * m161: hardware fetch witness. The host writes GSP_PHYS_GR_QMD_SENTINEL + * into the inline-QMD page before every deposit; the engine's QMD fetch is + * the only actor that overwrites it (the post-mortem has used this as its + * ran/not-run marker since m110). Computing it as a standalone check lets + * the outcome-retry path classify a zero-write result beyond the release + * word, which the sema service's pending-release sweep can satisfy in + * software. + */ +static bool gsp_phys_gr_qmd_fetched(const u8 *qmd_mem) +{ + if (!qmd_mem) + return false; + return get_unaligned_le32(qmd_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4) != + GSP_PHYS_GR_QMD_SENTINEL; +} + +/* + * r427 idempotent fast-accept (CONT.86 §6.1). A zero-write launch is either + * the no-execute wedge OR a legitimate idempotent re-launch (the same QMD + * content with the same staged inputs, whose result is already in place) - + * the host cannot tell them apart, and both burn the 40 s + 80 s quiet holds. + * The launch identity below (QMD minus the per-launch release0 address and + * payload, folded with the staged content checksums of every tracked page) is + * recorded for every launch that WROTE; a zero-write launch whose identity + * matches an entry accepts immediately, with no retries and no quiet window. + * Safe by construction: identical code + identical staged content + a prior + * execution that produced this launch's own footprint means the correct + * result is already in the buffers; a side-effecting kernel's re-launch is + * equally wrong under the current accept (the client sees the stale result + * either way) - this only removes the stall. + */ +#define GSP_PHYS_GR_IDEM_SLOTS 64u +static u64 gsp_phys_gr_idem_fp[GSP_PHYS_GR_IDEM_SLOTS]; +static u32 gsp_phys_gr_idem_n; + +static u64 gsp_phys_gr_launch_fp(const u32 *qmd, + const struct gsp_phys_gr_submit *req, + const u64 *csum_in) +{ + u64 h = 1469598103934665603ULL; + u32 i; + + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS; i++) { + /* Skip release0's address (w23/w24) and payload (w25): the + * driver allocates a fresh release slot per launch, so they + * differ even for an identical re-launch. */ + u32 w; + + if (i == 23u || i == 24u || i == 25u) + continue; + w = qmd[i]; + h ^= w; + h *= 1099511628211ULL; + } + for (i = 0; i < req->n_data; i++) { + u32 bi = gsp_phys_gr_data_bi(req->data[i].reloc_va); + + if (bi == GSP_PHYS_GR_DATA_BI_NONE) + continue; + h ^= csum_in[bi]; + h *= 1099511628211ULL; + } + return h; +} + +static bool gsp_phys_gr_idem_match(u64 fp) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_idem_n; i++) + if (gsp_phys_gr_idem_fp[i] == fp) + return true; + return false; +} + +static void gsp_phys_gr_idem_record(u64 fp) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_idem_n; i++) + if (gsp_phys_gr_idem_fp[i] == fp) + return; + if (gsp_phys_gr_idem_n < GSP_PHYS_GR_IDEM_SLOTS) { + gsp_phys_gr_idem_fp[gsp_phys_gr_idem_n++] = fp; + } else { + /* ring: drop the oldest */ + memmove(&gsp_phys_gr_idem_fp[0], &gsp_phys_gr_idem_fp[1], + (GSP_PHYS_GR_IDEM_SLOTS - 1u) * sizeof(u64)); + gsp_phys_gr_idem_fp[GSP_PHYS_GR_IDEM_SLOTS - 1u] = fp; + } +} + +/* + * COPYAUDIT (2026-10-18 copy-completeness audit): per-copy sighting of the + * GR operand staging copies. Every staged page is spot-checked (16 bytes + * at the start and end of the page, source vs destination, right after the + * memcpy) and any difference logs COPY-MISMATCH; the callers log the + * per-launch COPYAUDIT aggregate when the staged set exceeds 32 KB, and + * COPY-TRUNC whenever fewer pages were staged than the launch declared + * (want>done). Purely diagnostic: no control flow, locking or timing + * change; the memcmp reads memory the memcpy just touched. + */ +#define GSP_PHYS_GR_AUDIT_MIN (32u << 10) /* >32 KB staged sets */ + +static void gsp_phys_gr_page_spot_check(const u8 *dst, const u8 *src, + u64 va, const char *path) +{ + if (memcmp(dst, src, 16) || + memcmp(dst + GSP_PHYS_PAGE_SIZE - 16, + src + GSP_PHYS_PAGE_SIZE - 16, 16)) + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x%llx path=%s\n", + va, 0ull, path); +} + +/* + * r452: the legacy 32-bit additive page sum, kept ONLY for the all-zero + * filter. It is never a change-detection fingerprint any more: doubling every + * word of a 4 KB page leaves the sum bit-identical (1024 x 2^23 == 2^33 == 0 + * mod 2^32), which silently classified an in-place x2 scale as "no engine + * store" and shipped stale values (the deterministic streams2 class). + */ +static inline u32 gsp_phys_gr_page_sum32(const u8 *p) +{ + u32 ci, csum = 0; + + for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) + csum += get_unaligned_le32(p + ci); + return csum; +} + +/* + * COPYWIT r226 zero filter: an all-zero page is never a meaningful alias + * partner. A fresh client's fill stages long runs of zero pages + * (uninitialized cudaMalloc buffers), and consecutive zero pages produced + * the IN-ALIAS-WIT false positives of the r226a green run (csum=0x00000000 + * sequences). A page counts as zero only when its csum is zero AND every + * word is zero (csum==0 alone can hide cancelling contents). + */ +static inline bool gsp_phys_gr_page_all_zero(u32 csum, const u8 *p) +{ + u32 i; + + if (csum) + return false; + for (i = 0; i < GSP_PHYS_PAGE_SIZE; i += 4) + if (get_unaligned_le32(p + i)) + return false; + return true; +} + +/* + * COPYWIT r22x INPUT-ALIAS-WIT: per-launch census of the copy-in recs (the + * exact VA -> HPA -> content triples the engine will read for this launch, + * inputs included). Each new non-zero rec is compared against the last + * GSP_PHYS_GR_IN_CENSUS recs: + * - distinct VA resolving to the SAME host page (hpa collision) is the + * resolver-aliasing smoking gun: several guest pages served by one host + * page would replicate those rows through every consumer (the observed + * band-replication signature); + * - distinct HPA with identical content is the upstream-write-clone class + * (an H2D that landed on one page, or guest pages that genuinely hold + * the same bytes). + * Zero pages are excluded on both sides (fresh-client zero runs are not + * aliasing evidence). Bounded boot-wide; same single-inflight discipline. + */ +#define GSP_PHYS_GR_IN_CENSUS 64u +static u32 gsp_phys_gr_in_census_log; + +/* * m153: fill the bounce window from the request's guest pages and capture a * per-page checksum baseline. Used before the first publish and again before * every outcome retry (a mis-executing first attempt can leave stale or * wrong-window content behind, so a retry must start from the guest inputs - * again). + * again). Returns the number of pages staged. */ -static void gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, - u32 *csum_in) +static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, + u64 *csum_in, u64 *staged_hpa) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); - u32 di; + struct { + u64 va; + u64 hpa; + u32 csum; + } census[GSP_PHYS_GR_IN_CENSUS]; + u32 n_census = 0; + u32 di, copied = 0; for (di = 0; di < req->n_data; di++) { u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); const u8 *src; u8 *dst; u32 ci, csum = 0; + bool zero; if (bi == GSP_PHYS_GR_DATA_BI_NONE || !req->data[di].hpa) continue; @@ -17125,11 +21309,2864 @@ static void gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, if (!dst) continue; memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + /* + * r378: record which guest page's content now occupies this + * bounce slot. The copy-out may only write the slot back to + * that page - a rec whose reloc VA aliases another rec's slot + * would otherwise ship the other page's content into this + * guest page (the multi-writer clobber class). + */ + staged_hpa[bi] = req->data[di].hpa; + gsp_phys_gr_page_spot_check(dst, src, req->data[di].va, + "gr-bounce-copy-in"); for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) csum += get_unaligned_le32(dst + ci); - csum_in[bi] = csum; + /* r452: the change baseline is the FNV-1a fingerprint (the + * add-sum above survives only as the all-zero test). */ + csum_in[bi] = gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); + zero = gsp_phys_gr_page_all_zero(csum, dst); + if (!zero && gsp_phys_gr_copywit_enabled()) { + u32 j; + + for (j = 0; j < n_census; j++) { + if (census[j].hpa == req->data[di].hpa && + census[j].va != req->data[di].va) { + if (gsp_phys_gr_in_census_log < 24) { + gsp_phys_gr_in_census_log++; + pr_info("vgsp-gr: INPUT-ALIAS-WIT kind=hpa-collision va=0x%llx hpa=0x%llx prev_va=0x%llx (two guest pages served by one host page)\n", + req->data[di].va, + req->data[di].hpa, + census[j].va); + } + } else if (census[j].csum == csum && + census[j].hpa != req->data[di].hpa) { + if (gsp_phys_gr_in_census_log < 24) { + gsp_phys_gr_in_census_log++; + pr_info("vgsp-gr: INPUT-ALIAS-WIT kind=content-clone va=0x%llx hpa=0x%llx prev_va=0x%llx prev_hpa=0x%llx csum=0x%08x\n", + req->data[di].va, + req->data[di].hpa, + census[j].va, + census[j].hpa, + csum); + } + } + } + if (n_census < GSP_PHYS_GR_IN_CENSUS) { + census[n_census].va = req->data[di].va; + census[n_census].hpa = req->data[di].hpa; + census[n_census].csum = csum; + n_census++; + } + } + copied++; + } + dma_wmb(); + return copied; +} + +/* + * relfix: copy the late-resolved pages' guest content into their bounce + * pages and capture the copy-in checksum baseline, exactly as + * gsp_phys_gr_fill_bounce does for the fill's tracked data pages. Called + * before the doorbell (initial copy-in) and again before every outcome + * retry, so a retried launch starts from the guest inputs in both sets. + * Returns the number of pages staged. + */ +static u32 gsp_phys_gr_fill_gap_bounce(u64 *csum_in, u64 *staged_hpa) +{ + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u32 gi, copied = 0; + + for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { + const u8 *src = (const u8 *)__va(gsp_phys_gr_gap[gi].hpa); + u8 *dst = gsp_phys_chunk_page_va(dchunk, gsp_phys_gr_gap[gi].bi); + + if (!src || !dst) + continue; + memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + /* r378: the slot now holds THIS page's content (see the + * tracked-page fill above). */ + staged_hpa[gsp_phys_gr_gap[gi].bi] = gsp_phys_gr_gap[gi].hpa; + gsp_phys_gr_page_spot_check(dst, src, gsp_phys_gr_gap[gi].va, + "gr-bounce-copy-in"); + /* r452: FNV-1a baseline (see gsp_phys_gr_scan_pages). */ + csum_in[gsp_phys_gr_gap[gi].bi] = + gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); + copied++; + } + dma_wmb(); + return copied; +} + +/* + * F3 retry hygiene: an outcome retry re-publishes the same QMD. If the QMD's + * pointer pairs name the win0 shadow bank the previous attempt used, move the + * already-prepared param/program/bank pages into the alternate bank and + * rewrite the pairs (and the program prefetch) to the fresh VAs. The copies + * come from the module shadow pages - after relocation, inline replay and + * pointer fix-ups - so the engine-visible content is byte-identical to the + * first attempt and only the VA changes; the engine therefore cannot serve + * the retry from a cached fetch line of the previous attempt's bank. With + * gsp_phys_rmcall_gr_ptr_guest the pairs hold guest VAs, match nothing here, + * and the engine keeps reading the guest VAs exactly as before. Returns true + * when at least one pair moved (the caller must rebuild the method block). + */ +static bool gsp_phys_gr_win0_retry_rearm(u32 *qmd, + const struct gsp_phys_gr_submit *req, + u32 old_bank_base, u32 new_bank_base, + u32 new_bank_limit) +{ + struct gsp_phys_boot_chunk *pool = &boot_stage.rmcall_gr_shadow_win0; + u64 old_lo = GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)old_bank_base * GSP_PHYS_PAGE_SIZE; + u64 old_hi = old_lo + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS * + GSP_PHYS_PAGE_SIZE; + u32 next_slot = new_bank_base; + u32 i; + bool moved = false; + + if (!pool->pages) + return false; + for (i = 0; i < req->n_pages; i++) { + const struct gsp_phys_gr_page *pg = &req->pages[i]; + u32 w = pg->qmd_word; + u64 pair_va, new_va; + u32 off, old_slot; + u8 *src, *dst; + + if (w != 32u && w != 48u && + !(w >= 34u && w <= 46u && !(w & 1u))) + continue; + pair_va = ((u64)(qmd[w + 1] & 0x1ffffu) << 32) | qmd[w]; + if (pair_va < old_lo || pair_va >= old_hi) + continue; + if (next_slot >= new_bank_limit) + break; + off = (u32)(pair_va & (GSP_PHYS_PAGE_SIZE - 1)); + old_slot = (u32)((pair_va - + GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE) / + GSP_PHYS_PAGE_SIZE); + src = gsp_phys_chunk_page_va(pool, old_slot); + dst = gsp_phys_chunk_page_va(pool, next_slot); + if (!src || !dst) + break; + memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + new_va = GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)next_slot * GSP_PHYS_PAGE_SIZE + off; + qmd[w] = (u32)new_va; + qmd[w + 1] = (qmd[w + 1] & ~0x1ffffu) | + ((u32)(new_va >> 32) & 0x1ffffu); + if (w == 48u) { + u64 pref = ((u64)(qmd[31] & 0x1ffu) << 40) | + ((u64)qmd[8] << 8); + + if ((pref & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == + (pair_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) { + u64 np = new_va | + (pref & (GSP_PHYS_PAGE_SIZE - 1)); + + qmd[8] = (u32)(np >> 8); + qmd[31] = (qmd[31] & ~0x1ffu) | + ((u32)(np >> 40) & 0x1ffu); + } + } + next_slot++; + moved = true; + } + if (moved) + dma_wmb(); + return moved; +} + +/* + * F4-a: the first poll break, rendered from a noinline helper so its + * addresses/counter scratch never grow the (already warning-bound) submit + * frame. The USERD GP_GET/GP_PUT at the break discriminate an engine ack + * (GP_GET advanced, break latency long enough for a real retirement) from a + * host-authored word (break within a few ms of the doorbell, GP_GET frozen). + */ +static noinline void +gsp_phys_gr_poll_break_log(u32 w0, u32 want_qmd, u32 w1, u32 since_us, + const struct gsp_phys_rmcall_ce_submit *sub) +{ + struct gsp_phys_boot_chunk *ud_c = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; + u8 *ud = ud_c->pages ? gsp_phys_chunk_page_va(ud_c, 0) : NULL; + u32 usr_gpget = ud ? + get_unaligned_le32(ud + gsp_phys_rmcall_userd_ctl_off() + + GSP_PHYS_RMCALL_USERD_GPGET_OFF) : 0; + /* + * SESSION 153-CONTINUED-47: re-read the ring slot the launch + * published (the entry AS THE ENGINE SEES IT) after the wait, so an + * X-vs-beef diff on one boot compares the exact e0/e1 the PBDMA + * fetched, plus the doorbell-visible GP_GET/GP_PUT movement. + */ + struct gsp_phys_boot_chunk *r_c = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GPFIFO]; + u8 *ring = r_c->pages ? gsp_phys_chunk_page_va(r_c, 0) : NULL; + u32 re0 = 0, re1 = 0; + + if (ring && sub->slot < GSP_PHYS_RMCALL_GPFIFO_ENTRIES) { + re0 = get_unaligned_le32(ring + sub->slot * 8); + re1 = get_unaligned_le32(ring + sub->slot * 8 + 4); + } + + pr_info("vgsp-gr: poll break w0=0x%x want=0x%x ch=0x%x after %u us usr_gpget=0x%x gpget=0x%x gpput=0x%x ring[%u]=%08x %08x (pub %08x %08x)\n", + w0, want_qmd, w1, + (u32)ktime_to_us(ktime_get()) - since_us, + usr_gpget, sub->gpget, sub->gpput, + sub->slot, re0, re1, sub->e0, sub->e1); + /* r425: the healthy-launch SKED/FECS baseline for the wedge compare. */ + gsp_phys_gr_sked_wit(gsp_phys_gpu_bar0(), "post-poll"); +} + +/* + * SESSION 153-CONTINUED-72: GR-STATE retire readback. The keep-mask A/B + * must answer "does the kept macro execute at all" before chasing the + * 0/903 stores. The FE decode-state EX records (0x40a630..3c) name the + * last MME-GENERATED method write: bit31 GENERATED, 21:12 macro address, + * 11:0 pc - gen0=1 with the kept slot's macro address at the retire is + * the live witness that the engine fetched and ran the kept macro. The + * MME HWW latch (0x404490 + INFO/IR), the trapped method (0x40704 offset + * + bit20 mmegen, data 0x40708) and the DS latch catch the write when it + * faulted. One line per retire; bounded per boot (beef up to 48, others + * 4) so a battery boot cannot flood. + */ +static noinline void gsp_phys_gr_state_rb_log(u32 h_object) +{ + static u32 gs_beef_n, gs_other_n; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + bool beef = (h_object & 0xfffff000u) == 0xbeef0000u; + u32 fed0, fed1, fed2, fed3, esr, info, ir2, trap, data, ds, gf; + /* + * SESSION 153-CONTINUED-91: with the DS HWW report mask cleared + * (sema_ds_mask_clear) a SPH validation miss latches in DS_HWW_ESR + * without the GSP-RM fault handler poisoning the PRI block, so the + * per-stage SPH_INFO snapshot is finally readable: STAGE0..5 START + * (0x405c00+si*4) / UPPER (0x405c24+si*4) name the exact SPH VA each + * pipeline stage fetched, and PROGRAM_REGION_A/B (0x405c1c/c20) the + * region the DS validated against (the r1277+ wall's missing datum). + */ + u32 sph[6], sphu[6], rr_a, rr_b; + u32 si; + + if (!bar0) + return; + if (beef ? gs_beef_n >= 48u : gs_other_n >= 4u) + return; + if (beef) + gs_beef_n++; + else + gs_other_n++; + for (si = 0; si < 6u; si++) { + sph[si] = ioread32(bar0 + 0x00405c00u + si * 4u); + sphu[si] = ioread32(bar0 + 0x00405c24u + si * 4u); + } + rr_a = ioread32(bar0 + 0x00405c1cu); + rr_b = ioread32(bar0 + 0x00405c20u); + fed0 = ioread32(bar0 + 0x0040a630u); + fed1 = ioread32(bar0 + 0x0040a634u); + fed2 = ioread32(bar0 + 0x0040a638u); + fed3 = ioread32(bar0 + 0x0040a63cu); + esr = ioread32(bar0 + 0x00404490u); + info = ioread32(bar0 + 0x00404494u); + ir2 = ioread32(bar0 + 0x00404498u); + trap = ioread32(bar0 + 0x00400704u); + data = ioread32(bar0 + 0x00400708u); + ds = ioread32(bar0 + 0x00405840u); + gf = ioread32(bar0 + 0x00400504u); + pr_info("vgsp-gr: GR-STATE retire h=0x%x fed=%08x,%08x,%08x,%08x (pc0=%u mac0=0x%x gen0=%u) mme_esr=%08x info=%08x ir=%08x trap=%08x(mthd=0x%x mmegen=%u) data=%08x ds=%08x grfifo=%08x\n", + h_object, fed0, fed1, fed2, fed3, + fed0 & 0xfffu, (fed0 >> 12) & 0x3ffu, (fed0 >> 31) & 1u, + esr, info, ir2, trap, trap & 0x3ffcu, (trap >> 20) & 1u, + data, ds, gf); + pr_info("vgsp-gr: GR-STATE sph h=0x%x ds2=%08x sph=%08x,%08x,%08x,%08x,%08x,%08x sphU=%02x,%02x,%02x,%02x,%02x,%02x rr=%08x,%08x\n", + h_object, ioread32(bar0 + 0x00405848u), + sph[0], sph[1], sph[2], sph[3], sph[4], sph[5], + sphu[0] & 0xffu, sphu[1] & 0xffu, sphu[2] & 0xffu, + sphu[3] & 0xffu, sphu[4] & 0xffu, sphu[5] & 0xffu, + rr_a, rr_b); + /* + * SESSION 153-CONTINUED-91: did the emitted program-region pair + * (SET_MME_SHADOW_SCRATCH 0x78/0x79 = method 0x35e0/0x35e4 through + * class 0xc597) actually land in the MME shadow? The shadow read + * port is the same one the store-side probe uses; a zero readback + * means the write never took effect and every region experiment so + * far was a no-op. + */ + if (beef) + pr_info("vgsp-gr: GR-STATE region-shadow h=0x%x scratch[0x78]=%08x scratch[0x79]=%08x\n", + h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x35e0u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x35e4u)); +} + +/* + * F4-b: one-shot deposit readback at the first timeout (noinline for the same + * frame reason as above). Reads the qmdwin/shadow destination (or the guest + * QMD page) with the booter's existing unaligned host readback; w63 is the + * engine fetch sentinel. + */ +static noinline void +gsp_phys_gr_deposit_rb_log(const struct gsp_phys_gr_submit *req, + u8 *qmd_dst_mem) +{ + u8 *qp = qmd_dst_mem ? qmd_dst_mem : + (req->qmd_hpa ? (u8 *)__va(req->qmd_hpa) : NULL); + + if (!qp) + return; + pr_info("vgsp-gr: deposit readback w0=0x%08x w4=0x%08x w12=0x%08x w13=0x%08x w18=0x%08x w19=0x%08x w23=0x%08x w32=0x%08x w33=0x%08x w48=0x%08x w49=0x%08x w63=0x%08x\n", + get_unaligned_le32(qp), + get_unaligned_le32(qp + 16), + get_unaligned_le32(qp + 48), + get_unaligned_le32(qp + 52), + get_unaligned_le32(qp + 72), + get_unaligned_le32(qp + 76), + get_unaligned_le32(qp + 92), + get_unaligned_le32(qp + 128), + get_unaligned_le32(qp + 132), + get_unaligned_le32(qp + 192), + get_unaligned_le32(qp + 196), + get_unaligned_le32(qp + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4)); +} + +/* + * CODEHASH (2026-10-18): code-fetch content verification. At author time, + * sample the host-readable bytes the engine is about to fetch for this + * launch's program (the QMD word 48/49 pair): the first 16 B at the entry + * offset plus 16 B at PA+0 / PA+mid / PA+end of the entry page, and the + * same three windows on up to three further consecutive tracked pages of + * the code run. One bounded line per launch, emitted when the content is + * suspicious (all-zero samples, or the first four entry words identical - + * a page that never landed or a repeated filler word) or for the first few + * healthy launches per boot (so the healthy signature is visible without + * flooding). A missing host-readable landing (hpa 0 on the entry page) + * logs CODEHASH-UNREADABLE. sh0 is the engine-visible shadow page's entry + * word when the Tier-2 copy exists (NULL on the Tier-1/guest path). + * noinline: keeps the (warning-bound) submit frame unchanged. + */ +static bool gsp_phys_gr_codehash_done; +static noinline void +gsp_phys_gr_codehash(const struct gsp_phys_gr_submit *req, + const struct gsp_phys_gr_page *pg, const u8 *sh_page) +{ + static u32 ch_checked, ch_logged, ch_unread; + u64 va, h0 = 0, ent = 0, sh0 = 0; + u32 off, n = 0, w[4] = { 0, 0, 0, 0 }; + bool zero3 = false, suspect = false; + const u8 *p = NULL; + u32 i; + + if (gsp_phys_gr_codehash_done) + return; + gsp_phys_gr_codehash_done = true; + ch_checked++; + off = (u32)(req->qmd[48] & (GSP_PHYS_PAGE_SIZE - 1)); + va = (u64)req->qmd[48] | ((u64)(req->qmd[49] & 0x1ffffu) << 32); + if (!pg->hpa) { + if (ch_unread < 32) { + ch_unread++; + pr_info("vgsp-gr: CODEHASH-UNREADABLE va=0x%llx off=0x%x (entry page has no host-readable landing)\n", + va, off); + } + return; + } + p = (const u8 *)__va(pg->hpa); + n = 1; + ent = get_unaligned_le64(p + off); + h0 = get_unaligned_le64(p); + for (i = 0; i < 4; i++) + w[i] = get_unaligned_le32(p + off + i * 4); + zero3 = !(get_unaligned_le64(p) | get_unaligned_le64(p + 8) | + get_unaligned_le64(p + 0x800) | + get_unaligned_le64(p + 0x808) | + get_unaligned_le64(p + 0xff0) | + get_unaligned_le64(p + 0xff8)); + for (i = 1; i < 4; i++) { + u64 tva = pg->guest_va + ((u64)i << 12); + u32 j; + + for (j = 0; j < req->n_pages; j++) + if (req->pages[j].guest_va == tva && + req->pages[j].hpa) + break; + if (j == req->n_pages) + break; + { + const u8 *q = (const u8 *)__va(req->pages[j].hpa); + + n++; + if (get_unaligned_le64(q) || get_unaligned_le64(q + 8) || + get_unaligned_le64(q + 0x800) || + get_unaligned_le64(q + 0x808) || + get_unaligned_le64(q + 0xff0) || + get_unaligned_le64(q + 0xff8)) + zero3 = false; + } + } + if (sh_page) + sh0 = get_unaligned_le64(sh_page + off); + /* + * Conservative plausibility test: all three sampled windows zero; or + * the first two entry words zero; or the first four entry words + * identical (an all-zero page also lands here). + */ + suspect = zero3 || (!w[0] && !w[1]) || + (w[0] == w[1] && w[1] == w[2] && w[2] == w[3]); + if (!suspect && ch_logged >= 3) + return; + if (ch_logged >= 64) + return; + ch_logged++; + pr_info("vgsp-gr: CODEHASH va=0x%llx pa=0x%llx n=%u h0=0x%016llx ent=0x%016llx off=0x%x w=%08x %08x %08x %08x sh0=0x%016llx %s (first 16B)\n", + va, pg->hpa, n, h0, ent, off, w[0], w[1], w[2], w[3], sh0, + suspect ? "SUSPECT" : "ok"); +} + +/* + * M91 FAULT-CAP (2026-10-18): first-fault capture for the transient SM + * instruction-fetch class ("m91 class": the first attempt of a mid-sequence + * launch faults on SM garbage - "Illegal Instruction Encoding" / "Out Of + * Range Register" - while a retry of the identical QMD and identical + * shadow content succeeds). The submit path calls this once per launch at + * the end of the launch window (poll break or timeout) with the retry + * levers that fired; it emits one vgsp-m91: FAULT-CAP line per faulting + * launch carrying: the fault-epoch delta since the launch started, the + * forwarded QMD's address and first 64 B, the live program pair (w48/49) + * with its resolved engine-visible VA/PA, the first 64 B at that code + * address (the bytes the SM will fetch), the retry decision and outcome, + * and the newest relay-decoded 0x1006 note (exceptType/chid/errString). + * Statically inert on a fault-free launch: one READ_ONCE of the fault + * epoch and one compare, no reads of content or engine state. Full dumps + * are capped at GSP_PHYS_M91_CAP_MAX per module-load cycle, followed by a + * single suppression notice; the existing vgsp-fault lines keep decoding + * every fault either way. Static state follows the other per-launch submit + * scratch: the single-inflight ordered work item is the only caller. + */ +#define GSP_PHYS_M91_CAP_MAX 20u +static u32 gsp_phys_m91_fault_start; +static u32 gsp_phys_m91_cap_used; +static bool gsp_phys_m91_cap_suppressed; +static bool gsp_phys_m91_icache_log; +static bool gsp_phys_r449_ccache_log; +static bool gsp_phys_gr_rel_membar_log; +static noinline void +gsp_phys_gr_m91_fault_cap(const struct gsp_phys_gr_submit *req, + const u32 *qmd, u64 qmd_dst, const u8 *qmd_dst_mem, + u32 attempts, bool rearmed, bool fast, bool ok) +{ + u32 faults = READ_ONCE(gsp_phys_fault_epoch) - gsp_phys_m91_fault_start; + const struct gsp_phys_fault_info *fn; + const u8 *code = NULL; + const char *src = "unresolved"; + u64 pva, ppa = 0; + u32 off = 0, cn = 0, i; + u8 qbuf[64], cbuf[64]; + char qhex[2 * 64 + 1], chex[2 * 64 + 1]; + + /* INERT until a fault: fault-free launches take one compare and return. */ + if (!faults || !req || !qmd) + return; + if (gsp_phys_m91_cap_used >= GSP_PHYS_M91_CAP_MAX) { + if (!gsp_phys_m91_cap_suppressed) { + gsp_phys_m91_cap_suppressed = true; + pr_info("vgsp-m91: FAULT-CAP suppressed: %u dumps already emitted this cycle (the existing vgsp-fault lines still decode every fault)\n", + GSP_PHYS_M91_CAP_MAX); + } + return; + } + gsp_phys_m91_cap_used++; + + /* First 64 B of the forwarded QMD (qmdwin copy preferred). */ + memset(qbuf, 0, sizeof(qbuf)); + if (qmd_dst_mem) { + memcpy(qbuf, qmd_dst_mem, sizeof(qbuf)); + } else if (req->qmd_hpa) { + memcpy(qbuf, (const u8 *)__va(req->qmd_hpa), sizeof(qbuf)); + } + bin2hex(qhex, qbuf, sizeof(qbuf)); + qhex[sizeof(qhex) - 1] = '\0'; + + /* Program pair (QMD w48/49) and its engine-visible resolution. */ + pva = ((u64)(qmd[49] & 0x1ffffu) << 32) | qmd[48]; + off = (u32)(pva & (GSP_PHYS_PAGE_SIZE - 1)); + if (boot_stage.rmcall_gr_shadow_win0.pages && + pva >= GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE && + pva < GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS * GSP_PHYS_PAGE_SIZE) { + u32 slot = (u32)((pva - GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE) / + GSP_PHYS_PAGE_SIZE); + + if (slot < boot_stage.rmcall_gr_shadow_win0.pages) { + code = gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_shadow_win0, slot); + ppa = (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_shadow_win0, slot); + src = "win0"; + } + } else { + u64 base = pva & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + for (i = 0; i < boot_stage.rmcall_gr_alias_count; i++) { + if (boot_stage.rmcall_gr_alias_va[i] == base) { + ppa = boot_stage.rmcall_gr_alias_hpa[i]; + code = (const u8 *)__va(ppa); + src = "shadow-line"; + break; + } + } + for (i = 0; req->pages && !code && i < req->n_pages; i++) { + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != base || + !req->pages[i].hpa) + continue; + ppa = req->pages[i].hpa; + code = (const u8 *)__va(ppa); + src = "guest-hpa"; + } + } + memset(cbuf, 0, sizeof(cbuf)); + memset(chex, 0, sizeof(chex)); + if (code) { + cn = min_t(u32, sizeof(cbuf), GSP_PHYS_PAGE_SIZE - off); + memcpy(cbuf, code + off, cn); + bin2hex(chex, cbuf, cn); + chex[sizeof(chex) - 1] = '\0'; + } + + /* Newest relay-decoded 0x1006 (valid whenever the epoch advanced). */ + fn = &gsp_phys_fault_notes[(gsp_phys_fault_note_next + + GSP_PHYS_FAULT_NOTE_MAX - 1u) % + GSP_PHYS_FAULT_NOTE_MAX]; + pr_info("vgsp-m91: FAULT-CAP ok=%u faults=%u retry=%s attempts=%u qmd=0x%llx qmd64=%s prog_w48=0x%08x w49=0x%08x prog_va=0x%llx prog_pa=0x%llx code_src=%s off=0x%x code64=%s fault_et=0x%x fault_chid=%u fault_err=\"%.200s\"\n", + ok ? 1u : 0u, faults, + fast ? "fast-rearm" : (rearmed ? "resched/fresh-entry" : "none"), + attempts, qmd_dst, qhex, qmd[48], qmd[49], pva, ppa, src, off, + cn ? chex : "unresolved", + fn->valid ? fn->except_type : 0u, + fn->valid ? fn->chid : 0u, + fn->valid ? fn->err : "no relayed 0x1006 note"); +} + +/* + * CODESRC (2026-10-18, r221 follow-up): conservative refusal of a + * non-plausible Tier-2 program-pair source. The program page (QMD w48/49) + * is copied verbatim into the shadow the engine fetches from (win0 or the + * Tier-2 line); a source that is empty, PTE-shaped or one of the engine + * page-tree's own module pages would execute PTE/filler bytes as SASS. + * Detection is deliberately narrow so it is statically inert on every + * observed launch (greens included): + * (a) empty: the entry 16 B AND all three sampled windows are zero + * (a page that never landed anywhere the engine starts); + * (b) PTE page: >=12 of the 16 8-byte entries at the entry offset are + * identically-flagged valid PTEs (dense table page), or the run is + * >=12 zero entries with only PTE-signed entries (sparse table); + * (c) module page: the source HPA is a page of one of our own page-tree + * chunks (main PT, guest PT pool, Tier-2 shadow pools, qmdwin) - a + * guest page can never legitimately resolve there. + * A refused source is re-resolved through the late resolver + * (gsp_sema_gr_reloc_resolve: pins/map/walks before the ledger, the + * guest-walk-first counterpart of the fill's ledger-first order). Only a + * different, plausibly-code page is adopted (mode ALT; the engine-visible + * copy is taken from it); anything else leaves the QMD pair at the guest + * VA (mode GUEST_VA, the ptr_guest behaviour for this pair). One + * rate-limited CODESRC-SUSPECT line per detection. + */ +#define GSP_PHYS_GR_CODESRC_OK 0 +#define GSP_PHYS_GR_CODESRC_ALT 1 +#define GSP_PHYS_GR_CODESRC_GUEST_VA 2 + +static bool +gsp_phys_gr_codesrc_chunk_has(const struct gsp_phys_boot_chunk *c, u64 hpa) +{ + u32 i, n; + + if (!c || !c->pages) + return false; + n = c->pages < 32u ? c->pages : 32u; + for (i = 0; i < n; i++) { + u64 d = (u64)gsp_phys_chunk_page_dma(c, i); + + if (d && (d & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == hpa) + return true; + } + return false; +} + +static bool gsp_phys_gr_codesrc_modpt(u64 hpa) +{ + u32 i; + + if (gsp_phys_gr_codesrc_chunk_has( + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PT], hpa)) + return true; + for (i = 0; i < GSP_PHYS_RMCALL_GR_PT_PAGES; i++) + if (gsp_phys_gr_codesrc_chunk_has(&boot_stage.rmcall_gr_pt[i], + hpa)) + return true; + return gsp_phys_gr_codesrc_chunk_has(&boot_stage.rmcall_gr_shadow, + hpa) || + gsp_phys_gr_codesrc_chunk_has( + &boot_stage.rmcall_gr_shadow_win0, hpa) || + gsp_phys_gr_codesrc_chunk_has(&boot_stage.rmcall_gr_qmdwin, hpa); +} + +/* Entry sampling window: 16 8-byte entries starting at the entry offset. */ +static bool gsp_phys_gr_codesrc_ptelike(const u8 *p, u32 off) +{ + u32 start = off & ~7u, i, hits = 0, zeros = 0, nz = 0, flag = 0; + bool flag_set = false, sig_ok = true; + + if (start + 128u > GSP_PHYS_PAGE_SIZE) + start = GSP_PHYS_PAGE_SIZE - 128u; + for (i = 0; i < 16u; i++) { + u64 v = get_unaligned_le64(p + start + i * 8); + u32 lo = (u32)(v & 0xffu); + /* VALID, aperture not PEER(1): the PTE shape family. */ + bool pte = (v & 1u) && (((v >> 1) & 0x3u) != 1u); + + if (!v) { + zeros++; + continue; + } + nz++; + if (!pte) { + sig_ok = false; + continue; + } + if (!flag_set) { + flag = lo; + flag_set = true; + } + if (lo == flag) + hits++; + } + if (hits >= 12u) /* dense table page: a uniform valid-PTE run */ + return true; + /* sparse table page: mostly zero, only PTE-signed entries */ + return sig_ok && nz >= 1u && nz <= 4u && zeros >= 12u; +} + +static bool gsp_phys_gr_codesrc_empty(const u8 *p, u32 off) +{ + const u8 *e = p + off; + + return !(get_unaligned_le64(e) | get_unaligned_le64(e + 8)) && + !(get_unaligned_le64(p) | get_unaligned_le64(p + 8) | + get_unaligned_le64(p + 0x800) | get_unaligned_le64(p + 0x808) | + get_unaligned_le64(p + 0xff0) | get_unaligned_le64(p + 0xff8)); +} + +static noinline int +gsp_phys_gr_code_src(const struct gsp_phys_gr_submit *req, + const struct gsp_phys_gr_page *pg, u64 *alt_hpa) +{ + u64 page = pg->guest_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u32 off = (u32)(req->qmd[48] & (GSP_PHYS_PAGE_SIZE - 1)); + const u8 *p = (const u8 *)__va(pg->hpa); + const char *why = NULL, *used; + struct gsp_sema_gr_reloc r; + int mode = GSP_PHYS_GR_CODESRC_GUEST_VA; + + *alt_hpa = 0; + if (off > GSP_PHYS_PAGE_SIZE - 16u) + off = GSP_PHYS_PAGE_SIZE - 16u; /* keep 16 B in-page */ + if (gsp_phys_gr_codesrc_empty(p, off)) + why = "empty-page"; + else if (gsp_phys_gr_codesrc_ptelike(p, off)) + why = "pte-page"; + else if (gsp_phys_gr_codesrc_modpt(pg->hpa)) + why = "module-pt-page"; + if (!why) + return GSP_PHYS_GR_CODESRC_OK; + /* + * Alternative resolution order; only a different page that itself + * passes the content checks is adopted. + */ + if (!gsp_sema_gr_reloc_resolve(page, &r) && r.hpa && + (r.hpa & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != + (pg->hpa & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) && + !gsp_phys_gr_codesrc_empty((const u8 *)__va(r.hpa), off) && + !gsp_phys_gr_codesrc_ptelike((const u8 *)__va(r.hpa), off)) { + *alt_hpa = r.hpa; + mode = GSP_PHYS_GR_CODESRC_ALT; + used = "alt"; + } else { + used = "guest-va"; + } + pr_info_ratelimited("vgsp-gr: CODESRC-SUSPECT va=0x%llx pa=0x%llx alt=%s used=%s why=%s\n", + page, pg->hpa, + mode == GSP_PHYS_GR_CODESRC_ALT ? r.stage : "none", + used, why); + return mode; +} + +/* + * STALLFREE: the absolute per-launch wait budget. Every ladder phase is + * individually bounded, but a launch can chain them (fast re-arm + fresh + * entries + retirement retries), so the sum - and with it the time the + * single-inflight work item holds the client's completion - is not. This + * predicate bounds the whole submit: once it is true, no further re-arm, + * re-publish or re-execution runs. Green launches complete in ~0.2-11 s and + * never approach the default 60 s. + */ +static bool gsp_phys_gr_wait_budget_over(unsigned long start) +{ + unsigned int budget = gsp_phys_gr_wait_max_ms_value(); + + return budget && time_after_eq(jiffies, + start + msecs_to_jiffies(budget)); +} + +/* + * STALLFREE: one distinguishable bounded-completion line per stall episode. + * `action=fail` means the submit gives up (the sema service's failure edge + * authors the QMD completion release in software, so the guest proceeds); + * `action=proceed` means the phase gave up but the launch continues with its + * current outcome. + */ +static void gsp_phys_gr_stall_log(const char *reason, u64 push_va, + unsigned long start, const char *action) +{ + pr_warn("vgsp-gr: GR-STALL-%s va=0x%llx waited_ms=%u action=%s\n", + reason, push_va, jiffies_to_msecs(jiffies - start), action); +} + +/* + * SESSION 153-CONTINUED-27: big-surface alias bookkeeping. A replayed + * 3D window's colour target (C597 INC 0x800+j*0x40 [upper, lower, w, h, + * fmt] and its TWOD twin INC 0x218 [w, h, upper, lower]) can be a surface + * far larger than the launch's 16-page set (the X screen 0x120400000, + * 1280x720x4 = 900 pages). Two earlier routes are dead ends for it: + * - one-page shadow relocation (r992: `pair addr=0x840 va=0x120400000 + * -> MISS` then FAULT_PDE @ +0xa000 - the raster walk crosses the + * single alias page); + * - Tier-1 windows at the surface's own VAs (CONT.27 r993: PDE_UPDATE + * for pdeIndex 2306/2307 status=0x0, yet GPC2/GPC5 still fault + * FAULT_PDE @ +0x4000/+0xa000 - same class as the r929 v2 FE faults; + * the GPC engines only see the scratch-PT pool). + * The surviving route is the proven one: a CONTIGUOUS run of never-used + * scratch-PT fetch lines (one line = 16 PTEs = 64 KB VA; lines are + * monotonic and consecutive lines are VA-contiguous inside each pool + * block), one PTE per surface page, so the engine reaches all 900 pages + * through one contiguous alias VA and the raster walk never crosses into + * an unmapped page. The pair operands are rewritten to <alias>|<offset> + * by gsp_phys_rmcall_gr_3d_reloc. The table is per-boot: the alias PTEs + * are never rewritten (fetch-line rule) and the screen's HPAs are fixed + * by its single allocation, so every later launch is a pure rewrite. + */ +struct gsp_phys_gr_surf_keep { + u64 va; /* guest surface base VA (page aligned) */ + u64 alias; /* engine-visible contiguous shadow alias base */ + u64 npages; /* pages aliased (declared + slack tail) */ +}; +static struct gsp_phys_gr_surf_keep gsp_phys_gr_surf[4]; +static u32 gsp_phys_gr_surf_n; + +/* + * SESSION 153-CONTINUED-27: slack tail for the alias. r994 measured the + * replay's colour writes landing past the declared w*h*4 footprint + * (1280x720x4 = 0x384000) by up to +0xE000 (11.2 rows) - FAULT_PTE on + * the last fetch line's unwritten PTE slots (0x5d6000..0x5e2000 with the + * alias at 0x250000). The tail pages are aliased too: a page that still + * resolves (inside the guest allocation's own padding - exactly where + * native overdraw would land) maps the real HPA; one that does not + * resolves to a dedicated zeroed module page so the stray writes vanish + * instead of faulting. The chunk is allocated lazily and lives until + * module unload (4 KB). + */ +#define GSP_PHYS_GR_SURF_SLACK_PAGES 32u +static struct gsp_phys_boot_chunk gsp_phys_gr_surf_dummy; + +static u64 gsp_phys_gr_surf_dummy_dma_locked(void) +{ + struct gsp_phys_boot_chunk *d = &gsp_phys_gr_surf_dummy; + + if (!d->pages) { + if (gsp_phys_chunk_alloc(boot_stage.pdev, d, 1u)) + return 0; + gsp_phys_chunk_zero(d); + } + return (u64)gsp_phys_chunk_page_dma(d, 0); +} + +/* + * r929 v3 (Path A): relocate one 3D-window address operand to a shadow-pool + * VA. The guest VAs the 3D methods name (uploads 0x1_214f8f00.., vertex, + * program, tex pools) are installed in the ladder page tree by the Tier-1 + * mapper yet the engine's FE/PD walks still fault FAULT_PDE on them (r929 + * v2 evidence: "ENGINE GRAPHICS HUBCLIENT_FE faulted @ 0x1_214f8000 ... + * FAULT_PDE"); the shadow pool lines (map_shadow_locked) ARE the proven + * engine-visible region (every CUDA data/page relocation lives there). The + * page offset is preserved; a VA whose 4 KB page has no resolved hpa stays + * verbatim (it faults later and is named in the drain). Called with + * gsp_phys_rm_lock held. + */ +/* + * SESSION 153-CONTINUED-59: window/attr operand-miss state for the + * pre-doorbell abort (see gsp_phys_rmcall_gr_3d_reloc's level-2 fallback + * and the MISS-ABORT site in gsp_phys_backend_guest_gr_submit). Reset at + * the start of each 3D-window patch pass; a nonzero count at the submit + * point means the emitted block still carries a raw client VA the physical + * engine cannot resolve, which is the r1104/r1105 lane-killing class. + * sema_gl_miss_abort=0 restores the historical submit-anyway behaviour. + */ +static u32 gsp_phys_rmcall_gr_3d_miss_n; +static u64 gsp_phys_rmcall_gr_3d_junk_n; +static bool gsp_phys_rmcall_gr_miss_abort = true; +static u64 gsp_phys_gr_miss_abort_n; +module_param_named(sema_gl_miss_abort, gsp_phys_rmcall_gr_miss_abort, bool, + 0644); +MODULE_PARM_DESC(sema_gl_miss_abort, + "Abort a 3D-window launch before the doorbell while a window/attr operand relocation MISSes (page demanded via the pin path; retried on the next forward; fail-open after 3000 aborts). Default 1"); + +/* + * SESSION 153-CONTINUED-73: direct-write witness over the launch's OWN + * relocated operand destinations. The retirement scan only sees stores into + * the data-rec bounce copies; a pair rewritten to a shadow or surface-alias + * VA makes the engine store STRAIGHT into the guest page behind it (the + * 3d_reloc alias comment calls this class "direct guest writes, invisible + * to the witness"). Every such rewrite records its destination guest page + * here (with the source VA for the report); the submit path fingerprints + * the set right before the doorbell and the retirement wait re-fingerprints + * it, so one healthy run answers: do the draw's stores land in the bounce + * (counted today), in guest pages through a relocated operand (this + * witness), or in NEITHER (the draw rasterized nothing and the missing + * pipeline/vertex state is the wall). The MME-family payload rewrites + * (sema_gl_mme_reloc -> mme-reloc-rsv) feed the same set, so the MME + * shadow-scratch destinations are covered too. Static scratch: the only + * caller is the single-inflight ordered gsp-phys-work item. + */ +#define GSP_PHYS_GR_WIT_MAX 4096u +static u64 gsp_phys_gr_wit_pa[GSP_PHYS_GR_WIT_MAX]; +static u64 gsp_phys_gr_wit_va[GSP_PHYS_GR_WIT_MAX]; +static u64 gsp_phys_gr_wit_csum[GSP_PHYS_GR_WIT_MAX]; +static u64 gsp_phys_gr_wit_first8[GSP_PHYS_GR_WIT_MAX]; +static u32 gsp_phys_gr_wit_n; +static u32 gsp_phys_gr_wit_rep_log; + +static void gsp_phys_gr_wit_reset(void) +{ + gsp_phys_gr_wit_n = 0; +} + +static void gsp_phys_gr_wit_note(u64 va, u64 hpa) +{ + u32 k; + + if (!hpa) + return; + hpa &= ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + for (k = 0; k < gsp_phys_gr_wit_n; k++) + if (gsp_phys_gr_wit_pa[k] == hpa) + return; + if (gsp_phys_gr_wit_n >= GSP_PHYS_GR_WIT_MAX) + return; + gsp_phys_gr_wit_pa[gsp_phys_gr_wit_n] = hpa; + gsp_phys_gr_wit_va[gsp_phys_gr_wit_n] = + va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + gsp_phys_gr_wit_n++; +} + +static void gsp_phys_gr_wit_capture(const struct gsp_phys_gr_submit *req) +{ + u32 k; + + /* + * Also witness the guest pages behind the launch's tracked data + * records (and the late-resolved gap pages): a store through a + * surface ALIAS VA lands in the surface's own guest HPA - invisible + * to the bounce scan even when the same page is a tracked record - + * so the direct-write pass needs these HPAs in the set. The + * pre-doorbell timing (after every module-side writer) makes the + * baseline the guest content the engine starts from. + */ + for (k = 0; req && k < req->n_data; k++) + gsp_phys_gr_wit_note(req->data[k].va, req->data[k].hpa); + for (k = 0; k < gsp_phys_gr_gap_n; k++) + gsp_phys_gr_wit_note(gsp_phys_gr_gap[k].va, + gsp_phys_gr_gap[k].hpa); + for (k = 0; k < gsp_phys_gr_wit_n; k++) { + const u8 *p = (const u8 *)__va(gsp_phys_gr_wit_pa[k]); + + gsp_phys_gr_wit_first8[k] = p ? + get_unaligned_le64(p) : 0; + gsp_phys_gr_wit_csum[k] = p ? + gsp_phys_gr_fnv1a64(p, GSP_PHYS_PAGE_SIZE) : 0; + } +} + +static u32 gsp_phys_gr_wit_report(const char *where) +{ + u32 k, changed = 0, shown = 0; + u64 first_va = 0, first_pa = 0; + + if (!gsp_phys_gr_wit_n) + return 0; + for (k = 0; k < gsp_phys_gr_wit_n; k++) { + const u8 *p = (const u8 *)__va(gsp_phys_gr_wit_pa[k]); + u64 fp = p ? + gsp_phys_gr_fnv1a64(p, GSP_PHYS_PAGE_SIZE) : 0; + + if (fp != gsp_phys_gr_wit_csum[k]) { + changed++; + if (!first_pa) { + first_pa = gsp_phys_gr_wit_pa[k]; + first_va = gsp_phys_gr_wit_va[k]; + } + /* + * CONT-73 attribution: the pre/post first word of up + * to four changed pages. A QMD/descriptor word or an + * MME-scratch datum names the writer; freshly built + * vertex/float data means the guest CPU rewrote its + * buffer while the engine ran. + */ + if (shown < 4) { + shown++; + pr_info("vgsp-gr: GUEST-WIT %s changed va=0x%llx pa=0x%llx pre=0x%016llx post=0x%016llx\n", + where, + gsp_phys_gr_wit_va[k], + gsp_phys_gr_wit_pa[k], + gsp_phys_gr_wit_first8[k], + p ? get_unaligned_le64(p) : 0); + } + } + } + /* + * One line per launch while the report budget lasts (a full boot can + * carry hundreds of X-channel launches; the beef rows are the ones + * being chased). + */ + if (gsp_phys_gr_wit_rep_log < 512u) { + gsp_phys_gr_wit_rep_log++; + pr_info("vgsp-gr: GUEST-WIT %s: %u/%u relocated-operand guest page(s) changed (direct shadow/alias stores%s)\n", + where, changed, gsp_phys_gr_wit_n, + changed ? "" : " - NONE (no direct guest writes)"); + if (changed) + pr_info("vgsp-gr: GUEST-WIT %s first changed va=0x%llx pa=0x%llx\n", + where, first_va, first_pa); + } + /* + * CONT-73 settle control: the engine is retired by now, so a second + * scan after a short quiet delay separates engine stores (one-shot, + * will not grow) from LIVE guest-CPU rewrites (still growing). A + * nonzero "late" count here means the changed rows above are guest + * noise, not engine writes. + */ + if (changed && gsp_phys_gr_wit_rep_log <= 128u) { + u32 late = 0; + + msleep(150); + for (k = 0; k < gsp_phys_gr_wit_n; k++) { + const u8 *p = + (const u8 *)__va(gsp_phys_gr_wit_pa[k]); + u64 fp = p ? + gsp_phys_gr_fnv1a64(p, + GSP_PHYS_PAGE_SIZE) : 0; + + if (fp != gsp_phys_gr_wit_csum[k]) + late++; + } + pr_info("vgsp-gr: GUEST-WIT %s settle: %u page(s) changed at +150 ms (was %u)%s\n", + where, late, changed, + late > changed ? + " - GUEST-ACTIVE (growing; noise)" : + " - quiescent (engine-final writes)"); + } + return changed; +} + +/* + * SESSION 153-CONTINUED-90: SPH-alias receipt. The DS I-fetch of a bound + * stage reaches the program-family VA through the shadow VA the relocation + * picked, and the PTE of that shadow VA names the physical page the engine + * reads. Log the mapping choice (launch page set vs trusted resolver), the + * shadow VA and the mapped HPA, then read the fetch words back through the + * host direct map of that very page (for a RAM-backed HPA the direct map is + * content-identical to the engine's PTE-based read), so "which physical page + * does the DS I-fetch actually read" is provable from the log alone. Bounded + * per boot; foreign (non-RAM) HPAs are named but not dereferenced. + */ +static void gsp_phys_gr_sph_alias_receipt(u64 page, u64 va, u64 sh, u64 hpa, + const char *via) +{ + static u32 sph_alias_n; + u32 w[8] = { 0 }; + u32 off = (u32)(va & (GSP_PHYS_PAGE_SIZE - 1u)); + bool rd = false; + + if (sph_alias_n >= 48u) + return; + sph_alias_n++; + if (pfn_valid((unsigned long)(hpa >> PAGE_SHIFT))) { + const u8 *p = (const u8 *)__va(hpa + off); + u32 i; + + for (i = 0; i < 8u; i++) + w[i] = get_unaligned_le32(p + (size_t)i * 4u); + rd = true; + } + pr_info("vgsp-gr: SPH-ALIAS-%s page=0x%llx va=0x%llx off=0x%x shadow=0x%llx hpa=0x%llx rd=%u w8=%08x %08x %08x %08x %08x %08x %08x %08x\n", + via, page, va, off, sh, hpa, rd ? 1u : 0u, + w[0], w[1], w[2], w[3], w[4], w[5], w[6], w[7]); +} + +/* + * SESSION 153-CONTINUED-90: A/B canary for the DS fetch path. When set, + * every program-family operand the window relocates maps a dedicated canary + * page (a deterministic 0xcc00cc00|idx pattern) instead of the resolved + * guest page. The DS error set either changes (the DS does fetch through + * this shadow VA) or stays byte-identical (something else supplies the + * stage start, e.g. the QMD program pointer path). Live-flippable. + */ +static bool gsp_phys_gr_sph_canary; +module_param_named(sema_gl_sph_canary, gsp_phys_gr_sph_canary, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_canary, + "Map program-family shadow VAs to a canary page (SPH fetch-path A/B)"); +static struct gsp_phys_boot_chunk sph_canary_chunk; +static bool sph_canary_ready; + +static u64 gsp_phys_gr_sph_canary_hpa(void) +{ + if (!sph_canary_ready) { + u8 *p; + u32 i; + + if (!boot_stage.pdev || + gsp_phys_chunk_alloc(boot_stage.pdev, &sph_canary_chunk, 1)) + return 0; + p = gsp_phys_chunk_page_va(&sph_canary_chunk, 0); + if (!p) + return 0; + for (i = 0; i < GSP_PHYS_PAGE_SIZE / 4u; i++) + ((u32 *)p)[i] = 0xcc00cc00u | (i & 0xffu); + sph_canary_ready = true; + } + return (u64)gsp_phys_chunk_page_dma(&sph_canary_chunk, 0); +} + +/* + * SESSION 153-CONTINUED-93: 8K program-pair shadow line (mission task 2 + * fallback). The DS validates a bound program through its fetched SPH; a + * per-page shadow line maps exactly one 4 KB page at PTE0 and leaves the + * line's PTE1 invalid, so any read past the first 4 KB of the program page + * (the SASS tail of a bind whose delivered length crosses 0x1000, or any + * DS full-program walk) leaves the authored bytes behind. Map the program + * family page (PTE0) and family+0x1000 (PTE1) into ONE fresh 128 B fetch + * line BEFORE the engine can fetch it (a line's contents are frozen once + * fetched), so a contiguous 8 KB is engine-visible from its first window + * on. The line is cached per hpa0 (pair cache) so every later window reuses + * the identical never-rewritten mapping; the alias cache is re-pointed at + * the pair line so every other consumer (program binds, code-fetch + * operands) sees the same 8 KB view. Live A/B via sema_gl_sph_8k. Called + * with gsp_phys_rm_lock held. + */ +static bool gsp_phys_rmcall_gr_sph_8k; +module_param_named(sema_gl_sph_8k, gsp_phys_rmcall_gr_sph_8k, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_8k, + "Map the program family page and family+0x1000 in one contiguous 8K shadow line (fresh fetch line; default 0)"); +#define GSP_PHYS_RMCALL_GR_PAIR_CACHE 64u +static u64 gsp_phys_gr_pair_hpa[GSP_PHYS_RMCALL_GR_PAIR_CACHE]; +static u64 gsp_phys_gr_pair_va[GSP_PHYS_RMCALL_GR_PAIR_CACHE]; +static u32 gsp_phys_gr_pair_n; + +/* + * Resolve the HPA of `page0 + 4K` through the same sources the relocation + * chain uses: the launch's own page set (the pages the copy-in authored), + * the tracked data recs, the sema service's trusted resolver, then the + * late-resolved gap pages. 0 = not resolvable. + */ +static u64 gsp_phys_rmcall_gr_prog_next_hpa_locked( + const struct gsp_phys_gr_submit *req, u64 page0) +{ + u64 page1 = page0 + GSP_PHYS_PAGE_SIZE; + u32 i; + + for (i = 0; i < req->n_pages; i++) + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page1 && + req->pages[i].hpa) + return req->pages[i].hpa; + for (i = 0; i < req->n_data; i++) + if ((req->data[i].va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page1 && + req->data[i].hpa) + return req->data[i].hpa; + { + struct gsp_sema_gr_reloc r; + + if (!gsp_sema_gr_reloc_resolve(page1, &r) && r.hpa) + return r.hpa; + } + for (i = 0; i < gsp_phys_gr_gap_n; i++) + if ((gsp_phys_gr_gap[i].va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page1 && + gsp_phys_gr_gap[i].hpa) + return gsp_phys_gr_gap[i].hpa; + return 0; +} + +static void gsp_phys_rmcall_gr_alias_cache_set_locked(u64 hpa, u64 va) +{ + u32 i; + + for (i = 0; i < boot_stage.rmcall_gr_alias_count; i++) + if (boot_stage.rmcall_gr_alias_hpa[i] == hpa) { + boot_stage.rmcall_gr_alias_va[i] = va; + return; + } + if (boot_stage.rmcall_gr_alias_count < GSP_PHYS_RMCALL_GR_ALIAS_CACHE) { + u32 ci = boot_stage.rmcall_gr_alias_count; + + boot_stage.rmcall_gr_alias_hpa[ci] = hpa; + boot_stage.rmcall_gr_alias_va[ci] = va; + boot_stage.rmcall_gr_alias_count = ci + 1u; + } +} + +/* + * Allocate (or reuse) the pair line. Returns the line base VA on success, + * 0 when the second half cannot be resolved or the pool is exhausted (the + * caller then falls through to the single-page path). The fresh line is + * allocated on first use for an hpa0; the pair cache reuses it afterwards + * (the PTEs are identical on every window by construction). + */ +static u64 gsp_phys_rmcall_gr_sph_pair_locked( + const struct gsp_phys_gr_submit *req, u64 page0, u64 hpa0, u32 slot) +{ + struct gsp_phys_boot_chunk *sh = &boot_stage.rmcall_gr_shadow; + u32 i, line, page, line_in_page; + u64 hpa1, va0; + u8 *pt; + int ret; + + for (i = 0; i < gsp_phys_gr_pair_n; i++) + if (gsp_phys_gr_pair_hpa[i] == hpa0) + return gsp_phys_gr_pair_va[i]; + hpa1 = gsp_phys_rmcall_gr_prog_next_hpa_locked(req, page0); + if (!hpa1) + return 0; + if (!sh->pages) { + ret = gsp_phys_chunk_alloc(boot_stage.pdev, sh, + GSP_PHYS_RMCALL_GR_SHADOW_PAGES); + if (ret) + return 0; + gsp_phys_chunk_zero(sh); } + if (slot >= sh->pages || !boot_stage.rmcall_va_guest_pt) + return 0; + line = boot_stage.rmcall_guest_pt_next_line; + if (line >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) + return 0; + page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + line_in_page = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + if (gsp_phys_rmcall_gr_scratch_win_install_locked(page)) + return 0; + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) + return 0; + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa0), + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + line_in_page) * 8); + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa1), + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + line_in_page + 1u) * 8); dma_wmb(); + boot_stage.rmcall_guest_pt_next_line = line + 1u; + va0 = gsp_phys_rmcall_guest_pt_win_va(page) + + ((u64)line_in_page << 16); + gsp_phys_rmcall_gr_alias_cache_set_locked(hpa0, va0); + gsp_phys_rmcall_gr_alias_cache_set_locked(hpa1, + va0 + GSP_PHYS_PAGE_SIZE); + if (gsp_phys_gr_pair_n < GSP_PHYS_RMCALL_GR_PAIR_CACHE) { + gsp_phys_gr_pair_hpa[gsp_phys_gr_pair_n] = hpa0; + gsp_phys_gr_pair_va[gsp_phys_gr_pair_n] = va0; + gsp_phys_gr_pair_n++; + } + pr_info("vgsp-gr: SPH-PAIR line=0x%llx hpa0=0x%llx hpa1=0x%llx (8K contiguous program page) h=0x%x\n", + va0, hpa0, hpa1, req->h_object); + return va0; +} + +/* + * SESSION 153-CONTINUED-92: identity-fetch keep test (see the + * sema_gl_prog_guest_va knob above). Resolve the operand's page (the + * launch's own page set first - the pages the copy-in authored - then the + * sema service's trusted resolver), install the Tier-1 identity PTE at the + * guest VA itself and report whether the operand may stay verbatim. The + * receipt names the exact VA the DS will now fetch through (the guest VA, + * shadow column = the guest VA) and reads the fetch words back through the + * host direct map of the mapped HPA, so "the DS reads the authored page at + * its own heap VA" is provable from the log alone. Anything the identity + * mapper refuses (no window slot, stale fetch line, unresolved HPA) returns + * false and the caller falls back to the shadow rewrite. Called with + * gsp_phys_rm_lock held. + */ +static bool gsp_phys_rmcall_gr_prog_keep_locked(void __iomem *bar0, + u32 epoch, u32 up_w, u32 lo_w, + const struct gsp_phys_gr_submit *req) +{ + u64 va = ((u64)(up_w & 0xffu) << 32) | (u64)(lo_w & 0xfffffffcu); + u64 page = va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u64 hpa = 0; + u32 i; + + if (!va || !page) + return false; + for (i = 0; i < req->n_pages; i++) { + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page && + req->pages[i].hpa) { + hpa = req->pages[i].hpa; + break; + } + } + if (!hpa) { + struct gsp_sema_gr_reloc r; + + if (!gsp_sema_gr_reloc_resolve(page, &r) && r.hpa) + hpa = r.hpa; + } + if (!hpa) + return false; + if (gsp_phys_rmcall_gr_map_va_locked(bar0, page, hpa, epoch)) + return false; + gsp_phys_gr_sph_alias_receipt(page, va, va, hpa, "prog-guest"); + return true; +} + +static bool gsp_phys_rmcall_gr_3d_reloc(u32 *up_w, u32 *lo_w, + const struct gsp_phys_gr_submit *req, + u32 *n_shadow) +{ + u64 va = ((u64)(*up_w & 0xffu) << 32) | (u64)(*lo_w & 0xfffffffcu); + u64 page = va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u64 sh = 0; + u32 line = 0, i; + + if (!va || !page) + return false; + /* + * SESSION 153-CONTINUED-90: must run before every resolution stage so + * the canary cannot be shadowed by an earlier (surd) alias hit. + */ + if (READ_ONCE(gsp_phys_gr_sph_canary) && req->prog_va && + page == (req->prog_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) { + u64 chpa = gsp_phys_gr_sph_canary_hpa(); + u32 line3 = 0; + + if (chpa && + *n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES && + !gsp_phys_rmcall_gr_map_shadow_locked(*n_shadow, chpa, + &sh, &line3)) { + if (line3 != 0xffffffffu) + (*n_shadow)++; + sh |= va & (GSP_PHYS_PAGE_SIZE - 1); + gsp_phys_gr_sph_alias_receipt(page, va, sh, chpa, + "canary"); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + /* + * SESSION 153-CONTINUED-63: rec-over-alias preference. When the fill + * tracked the render target through the bounce lane (every page + * resolved into contiguous data recs), rewrite the operand to the + * rec's bounce VA so the engine's stores land where the retire wait + * can count them and the copy-out can ship them back to the guest's + * own pages. The completeness gate (gsp_sema_gr_surf_recs) makes a + * partial run impossible to point at. Runs before the alias so the + * identity alias (direct guest writes, invisible to the witness) + * cannot win for a tracked target. + */ + { + u64 sva = 0; + u32 sn = 0; + + if (gsp_sema_gr_surf_recs(&sva, &sn) && + va >= sva && va < sva + ((u64)sn << 12)) { + for (i = 0; i < req->n_data; i++) { + if (req->data[i].va != page || + !req->data[i].reloc_va) + continue; + sh = req->data[i].reloc_va | + (va & (GSP_PHYS_PAGE_SIZE - 1)); + *lo_w = (*lo_w & 0x3u) | + ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + } + /* + * SESSION 153-CONTINUED-27: a big surface (the X screen) is aliased + * as ONE contiguous run by the pre-pass; rewrite the operand to the + * alias base | offset. A one-page shadow relocation of it would + * break the raster walk at the first page crossing (r992), and the + * raw VA has no PTE chain the GPC engines can see (r993). + */ + for (i = 0; i < gsp_phys_gr_surf_n; i++) { + if (va >= gsp_phys_gr_surf[i].va && + va < gsp_phys_gr_surf[i].va + + (gsp_phys_gr_surf[i].npages << 12)) { + /* + * CONT-73: the alias VA maps the surface's guest + * HPAs 1:1 (the alias PTEs were built from + * req->pages[] first, the trusted resolver else), so + * a store through it lands straight in that guest + * page - record it for the direct-write witness. + */ + { + u32 pi; + + for (pi = 0; pi < req->n_pages; pi++) { + if ((req->pages[pi].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != + page || + !req->pages[pi].hpa) + continue; + gsp_phys_gr_wit_note(page, + req->pages[pi].hpa); + break; + } + if (pi == req->n_pages) { + struct gsp_sema_gr_reloc ra; + + if (!gsp_sema_gr_reloc_resolve(page, + &ra) && + ra.hpa) + gsp_phys_gr_wit_note(page, + ra.hpa); + } + } + sh = gsp_phys_gr_surf[i].alias + + (va - gsp_phys_gr_surf[i].va); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + for (i = 0; i < req->n_pages; i++) { + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != page || + !req->pages[i].hpa) + continue; + if (*n_shadow >= GSP_PHYS_RMCALL_GR_SHADOW_PAGES) + return false; + /* + * SESSION 153-CONTINUED-93: 8K program-pair line (see the + * pair helper). Only the program family page rides this + * path; an unresolvable second half falls through to the + * single-page map below. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_8k) && req->prog_va && + page == (req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) { + u64 pv = gsp_phys_rmcall_gr_sph_pair_locked( + req, page, req->pages[i].hpa, *n_shadow); + + if (pv) { + (*n_shadow)++; + sh = pv | (va & + (GSP_PHYS_PAGE_SIZE - 1)); + gsp_phys_gr_wit_note(page, req->pages[i].hpa); + gsp_phys_gr_sph_alias_receipt(page, va, pv, + req->pages[i].hpa, "pair"); + *lo_w = (*lo_w & 0x3u) | + ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + if (gsp_phys_rmcall_gr_map_shadow_locked(*n_shadow, + req->pages[i].hpa, + &sh, &line)) + return false; + if (line != 0xffffffffu) + (*n_shadow)++; /* alias hit consumes no slot */ + /* CONT-73: shadow VA maps this guest HPA 1:1. */ + gsp_phys_gr_wit_note(page, req->pages[i].hpa); + sh |= va & (GSP_PHYS_PAGE_SIZE - 1); + if (req->prog_va && + page == (req->prog_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) + gsp_phys_gr_sph_alias_receipt(page, va, sh, + req->pages[i].hpa, + "pages"); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | ((u32)(sh >> 32) & 0xffu); + return true; + } + /* + * SESSION 153-CONTINUED-59: level-2 late resolution, the same trusted + * chain the MME payload branch uses (ipin/pin/lat/map/walk then shadow + * map). The r1104 first beat window MISSed the program pair 0x2054 + * (va=0x4350900) and the r1105 windows MISSed the index-buffer pairs + * (0x17c8, va=0x44fedf0...) whose pages were not in the launch page + * set at build time; the raw guest VA then let the engine fault the + * MMU (HUBCLIENT_PD) and the guest driver reset the channel. Resolve + * the page here if any stage knows it, so a MISS only survives when + * every source fails; a surviving MISS is recorded for the pin demand + * set and counts toward the pre-doorbell abort. + */ + { + struct gsp_sema_gr_reloc r; + u32 line2 = 0; + + if (!gsp_sema_gr_reloc_resolve(va, &r) && r.hpa && + *n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES && + !gsp_phys_rmcall_gr_map_shadow_locked(*n_shadow, r.hpa, + &sh, &line2)) { + if (line2 != 0xffffffffu) + (*n_shadow)++; + /* CONT-73: shadow VA maps this guest HPA 1:1. */ + gsp_phys_gr_wit_note(page, r.hpa); + sh |= va & (GSP_PHYS_PAGE_SIZE - 1); + if (req->prog_va && + page == + (req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) + gsp_phys_gr_sph_alias_receipt(page, va, sh, + r.hpa, + "resolve"); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | ((u32)(sh >> 32) & 0xffu); + return true; + } + } + /* + * SESSION 153-CONTINUED-61: junk-operand filter. The window + * recognizer is byte-shape based, so float/immediate DATA words in + * the method stream also parse as address pairs. The observed junk + * class sits in the 256 MB..4 GB numeric range (0x3d000000..0xc2...... + * float bit patterns: 0x3f800000 = 1.0f, 0xbf800000 = -1.0f, ...) + * and can NEVER resolve - r1109 measured miss=20 constant across + * every attempt, all of it this class, which pinned the abort census + * nonzero forever and blocked every submit. Real client VAs of this + * driver live below 0x5000000 (the 64-80 MB GL/heap range, e.g. + * 0x4250000/0x4350000/0x462fc40) or at/above 0x1_00000000 (the 4G+ + * push-buffer/UVM ranges). A miss outside both is a data-word + * misidentification, not a raw client VA the engine would read as an + * address: it is neither demanded (the demand set was 60 % junk by + * r1109, diverting the pin cycle) nor counted toward the pre-doorbell + * abort. Witnessed per class, bounded. + */ + if (page >= 0x10000000ULL && page < 0x100000000ULL) { + static u32 gr_missjunk_log; + + gsp_phys_rmcall_gr_3d_junk_n++; + if (gr_missjunk_log < 24 || + !(gsp_phys_rmcall_gr_3d_junk_n % 512u)) { + gr_missjunk_log++; + pr_info("vgsp-gr: MISS-JUNK va=0x%llx (data-word class; not demanded, not counted toward the abort) n=%llu\n", + page, (u64)gsp_phys_rmcall_gr_3d_junk_n); + } + return false; + } + /* + * Unresolvable: demand the page for the pin path and count the miss + * (the submit path aborts pre-doorbell while the count is nonzero). + */ + gsp_phys_rmcall_gr_3d_miss_n++; + gsp_sema_demand_note(page); + return false; +} + +/* + * r929 v3: copy the 3D window and relocate every address operand a 3D-class + * method run carries (same recognizers as the sema service's svc_gl_field: + * LINE_LENGTH_IN 0x180 OFFSET_OUT pair, SET_VERTEX_STREAM_SUBSTITUTE 0xf84, + * SET_TEX_SAMPLER_POOL 0x155c, SET_TEX_HEADER_POOL 0x1574 and the + * SET_PIPELINE_PROGRAM_ADDRESS 0x2014/0x2018+j*0x40 pairs). Returns the + * number of relocated pair(s). Run widths follow the service's decoder: + * op 1/3/5 are 1+count dwords, everything else resyncs one dword. + */ +/* + * SESSION 153-CONTINUED-13 (draw-window strip experiment): comma-separated + * hex NVC597 method byte-addresses to REMOVE from a captured 3D window + * before the verbatim replay (whole methods - a dropped run takes its + * count data words with it). Answers whether a DRAW window's non-retire + * lives in the draw content (BEGIN/END/immediate vertices/DRAW_ARRAY) or + * in the state binds/method-stream shape. Empty = off (historical + * behaviour); live-tunable via sysfs so one boot can sweep the variants. + */ +/* + * SESSION 153-CONTINUED-47 (A/B knob): gates the newly recognised pair + * classes (SET_INDEX_BUFFER_A/B 0x17c8 and the vertex-stream banks + * 0x1c20..0x1cf0) in the verbatim-replay relocation pass. Default 0 keeps + * the r1035-r1041 baseline relocation set exactly; flipping it live + * (`echo Y > /sys/module/vgsp/parameters/sema_gl_idx_reloc`) rewrites the + * next launches' index/vertex operands to shadow VAs, so one boot can A/B + * the effect instead of comparing across boots. + */ +static bool gsp_phys_rmcall_gr_idx_reloc; +module_param_named(sema_gl_idx_reloc, gsp_phys_rmcall_gr_idx_reloc, bool, 0644); +MODULE_PARM_DESC(sema_gl_idx_reloc, + "Relocate SET_INDEX_BUFFER (0x17c8) and vertex-stream banks 0x1c20..0x1cf0 operands to shadow VAs in 3D replays (default 0 = baseline set only)"); + +/* + * SESSION 153-CONTINUED-48: MME-family payload relocation. The C597 class + * carries CALL_MME_MACRO(j) = 0x3800+j*8 and CALL_MME_DATA(j) = 0x3804+j*8 + * (class/clc597.h); the GLX client's macro(2) run (observed at byte + * address 0x3810, cnt=5) inlines raw 32-bit client VAs as macro arguments + * (`... 0450f800 ... 0450f000` in the first beef window of r1042/r1043). + * Those words are the last raw-operand class left in the faulting window + * after the index/vertex relocation: the engine walks the macro data + * through its own fetch path, and a raw client VA has no PTE chain the + * GPC engines can see (the r993/r1043 FAULT_PDE->PRI-fault class). The + * pass rewrites any payload word of a run whose byte address lies in + * [0x3800, 0x3a00) through the same vetted shadow mapping the pair + * classes use: only a word whose page the launch's own page set (or the + * big-surface alias) resolves is rewritten (MISS stays verbatim), the + * rewrite maps the SAME HPA at a shadow VA so the value is + * content-equivalent, and a shadow VA that would need an upper byte (a + * 64-bit alias in a 32-bit slot) is refused. Default 0 (baseline windows + * byte-identical); live-tunable so one boot can A/B. + */ +static bool gsp_phys_rmcall_gr_mme_reloc; +module_param_named(sema_gl_mme_reloc, gsp_phys_rmcall_gr_mme_reloc, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_reloc, + "Relocate CALL_MME_MACRO/DATA(j) run payload words whose page resolves in the launch page set to shadow VAs in 3D replays (default 0 = off)"); + +/* + * SESSION 153-CONTINUED-58: torn-read zero guard for the vertex-attribute + * families. The r1098/r1099 first-launch class error + * (`Class Error: ChId 0001, Class 0000c597, Offset 00001168, + * Data 00000000, ErrorCode 00000005` = SET_VERTEX_ATTRIBUTE_A(2) written + * with NUMERICAL_TYPE=0, the header's UNUSED_ENUM_DO_NOT_USE value -> + * INVALID_ENUM) is written by a replayed window run + * `20030458 38400000 38200040 00000000`: a cnt=3 INC run at method 0x1160 + * whose THIRD data word is exactly zero and is immediately followed by a + * 52-word zero region - the read caught the client's push buffer after + * part of it had been reused/cleared. The same client writes valid A(2) + * values in the other windows of the same push (0x38200040/0x38200600/ + * 0x38400600), so the zero is a torn read, not client content; and + * NUMERICAL_TYPE=0 is never a legitimate client write. Guard: for INC + * runs in the SET_VERTEX_ATTRIBUTE_A/B families (0x1160..0x11dc) drop a + * data word that is exactly zero, splitting the run so the remaining + * words keep their methods (the attribute then keeps the value the + * ATTR-PRELUDE or the previous frame wrote). Default 1; live-tunable. + */ +static u32 gsp_phys_rmcall_gr_attr_zero_fix = 1; +module_param_named(sema_gl_attr_zero_fix, gsp_phys_rmcall_gr_attr_zero_fix, uint, 0644); +MODULE_PARM_DESC(sema_gl_attr_zero_fix, + "Drop torn-read zero data words in the SET_VERTEX_ATTRIBUTE_A/B runs (0x1160..0x11dc) of 3D replays, splitting the run (default 1)"); + +/* (SESSION 153-CONTINUED-53 MME prelude knob is declared before + * gsp_phys_rmcall_build_gr_launch, its reader.) */ + +/* + * SESSION 153-CONTINUED-48: ladder-wedge recovery. CONT-47 measured that + * the FIRST beef launch per boot whose window carries an unresolvable raw + * operand PRI-faults the GR: every SKED/FIFO register then reads the + * priv-error poison (cp. kernel_falcon_ga102.c `privErrVal = 0xbadf0000`) + * and the ladder channel stays wedged for the whole boot (engine GP_GET + * stuck; every later publish unconsumed, even an emptied window) - which + * makes content bisection a one-shot per boot. When enabled, the launch + * poll watches the GR FIFO status (0x400504, the sked recorder's + * `grfifo`) for the 0xbadf-tagged poison and performs ONE bounded + * recovery per launch: the r445 hard re-arm (TSG PREEMPT bWait=1 + the + * m155 lane-enable tail + GPFIFO/TSG schedules + doorbell) followed by + * the warm GR semaphore probe, exactly the mechanism behind + * gsp_phys_gr_wedge_rearm. Bounded: one detection per launch, poison + * gated, all RPCs keep their own timeouts. Default 1 (the poisoned state + * is already a guaranteed-dead boot; the recovery only adds the chance + * that later launches in the same boot still execute). + */ +static bool gsp_phys_gr_wedge_recover = true; +module_param_named(sema_gr_wedge_recover, gsp_phys_gr_wedge_recover, bool, + 0644); +MODULE_PARM_DESC(sema_gr_wedge_recover, + "On the 0xbadf priv-error poison in the GR FIFO status, run one bounded TSG-preempt+resched hard re-arm per launch (default 1)"); + +/* + * SESSION 153-CONTINUED-93: parked-engine recovery on the warm-sem ladder + * gate. The r1311-r1313 class: after the first few DS-validated draws the + * front-end stops fetching (grfifo=0x00000001, no poison, no trap) and the + * warm GR semaphore probe times out on every later launch - the ladder + * then fails fast and the whole boot is dead for bisection. When this knob + * is set, a wedged probe triggers ONE bounded hard re-arm (the proven + * r445 TSG-preempt bWait=1 + resched + probe) before giving up; an ALIVE + * probe after the re-arm lets the fresh-entry retries resume, and the MME + * prelude latch is cleared so the re-preload is owed (the preempt reset + * the context). Bounded: one per launch; all RPCs keep their timeouts. + * Default 1; A/B with 0 restores the fail-fast ladder. + */ +static bool gsp_phys_gr_park_recover = true; +module_param_named(sema_gr_park_recover, gsp_phys_gr_park_recover, bool, + 0644); +MODULE_PARM_DESC(sema_gr_park_recover, + "On a wedged warm-sem probe in the launch timeout ladder, run one bounded TSG-preempt+resched hard re-arm per launch and retry (default 1)"); + +static char *gsp_phys_rmcall_gr_win_drop; +module_param_named(sema_gl_win_drop, gsp_phys_rmcall_gr_win_drop, charp, 0644); +MODULE_PARM_DESC(sema_gl_win_drop, + "Hex method byte-addresses (comma-separated) to strip from a captured 3D window before replay (draw isolation; default empty)"); + +static bool gsp_phys_rmcall_gr_win_drop_match(u32 addr) +{ + const char *p = gsp_phys_rmcall_gr_win_drop; + u32 n = 0; + + if (!p || !*p) + return false; + while (*p && n < 64) { + u32 v = 0; + u32 digits = 0; + + while (*p == ' ' || *p == ',' || *p == ':' || *p == ';') + p++; + /* + * SESSION 153-CONTINUED-46: accept an optional 0x/0X prefix. + * The original parser scanned hex digits only, so the natural + * "0x3810,0x39c8" input parsed as 0 (the 'x' ended the digit + * run after the leading '0') and silently never matched - + * cost a full bisect round. + */ + if (p[0] == '0' && (p[1] == 'x' || p[1] == 'X')) + p += 2; + for (;;) { + char c = *p; + u32 d; + + if (c >= '0' && c <= '9') + d = c - '0'; + else if (c >= 'a' && c <= 'f') + d = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + d = c - 'A' + 10; + else + break; + if (v <= 0xfffffffu) + v = (v << 4) | d; + p++; + digits++; + } + if (!digits) + break; + /* + * SESSION 153-CONTINUED-47: `lo-hi` drops every method + * byte-address in the closed range (both ends accept the + * 0x prefix). The content bisect needs whole families + * (`0x3800-0x3a00` MME, `0x3400-0x3500` MME scratch) and + * the emptied-window control (`0-0x7ffc`): with the window + * stripped whole, a launch that still retires proves the + * SET_OBJECT + release tail path is healthy and that the + * wedge lives in the stripped content. + */ + if (*p == '-') { + const char *q = p + 1; + u32 hi = 0; + u32 hd = 0; + + if (q[0] == '0' && (q[1] == 'x' || q[1] == 'X')) + q += 2; + for (;;) { + char c = *q; + u32 h; + + if (c >= '0' && c <= '9') + h = c - '0'; + else if (c >= 'a' && c <= 'f') + h = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + h = c - 'A' + 10; + else + break; + if (hi <= 0xfffffffu) + hi = (hi << 4) | h; + q++; + hd++; + } + if (hd) { + p = q; + n++; + if (addr >= v && addr <= hi) + return true; + continue; + } + } + n++; + if (v == addr) + return true; + } + return false; +} + +/* + * SESSION 153-CONTINUED-27: auto-drop of the software-emulated lanes from + * every verbatim 3D replay. Three method triggers in a captured window + * cause the physical engine to repeat work the sema service already + * executes in software - and, on the replay channel, with the WRONG + * context state (no previous-window sticky registers): + * - 0x1b0 LAUNCH_DMA (C597 CE lane): triggers both the CE copy and the + * LOAD_INLINE_DATA upload. The service authors every LID payload into + * the guest pages itself (svc_gl_lid/svc_grce_inline_write) and + * emulates the CE strips (svc_op2d); physically re-running them with + * an unset source/context walks unbounded memory (r992-r996: the + * "march" - faults at whatever page the alias cap ended, e.g. + * 0x5d6000/0x5f4000, after having written every mapped page). + * - 0x5e0 (TWOD RECTS fill trigger): the second point pair of the + * client's screen clear. The service's FERMI_TWOD executor runs every + * fill (r986: 46x `sema-twod-fill ... done=3686400 rc=0`); the + * physical re-run uses the replay channel's unset dst PITCH and + * marches (same fault class). + * - 0x10c NOTIFY: the completion write; without the preceding + * SET_NOTIFY_A/B (which live in other pushes) the engine writes + * address 0 (r996: `faulted @ 0x0_00000000 FAULT_PTE VIRT_WRITE`). + * The kick-notify lane + the backend's release tail author every + * completion the client actually waits on. + * All three are pure triggers - their setup methods stay in the window, + * so dropping the run can never misalign display state. The emulated + * lanes are the live ones (the r986 pixel proof: the screen paints via + * svc_twod fills, and every CE strip retires rc=0 in software). Default + * on; set 0 to replay the raw window unmodified. + */ +static bool gsp_phys_gr_win_drop_auto = 1; +module_param_named(sema_gl_win_drop_auto, gsp_phys_gr_win_drop_auto, bool, + 0644); +MODULE_PARM_DESC(sema_gl_win_drop_auto, + "Drop the software-emulated trigger methods (0x10c NOTIFY, 0x1b0 LAUNCH_DMA, 0x5e0 TWOD fill) from verbatim 3D replays (default 1)"); + +static bool gsp_phys_rmcall_gr_win_drop_auto_match(u32 addr) +{ + if (!READ_ONCE(gsp_phys_gr_win_drop_auto)) + return false; + /* + * SESSION 153-CONTINUED-48: NVC597_SET_REPORT_SEMAPHORE_A=0x1b00 - + * the C597 twin of the 0x10c NOTIFY trigger. Every glcore draw + * window ends with 4-word runs [A=upper(7:0), B=lower(31:0), + * C=payload, D=flags] whose B word is a raw client VA (observed + * 0x41a5bb0/0x759e870/0x251800; payloads = frame counters 0xb4f3+). + * The replayed engine executing them writes a client VA the GPC + * engines cannot reach - the VIRT_WRITE fault class (r1061: the + * kept [0,0x2000) window whose only fetch/write triggers besides + * the index binds are these raw 0x1b00 writes is exactly the one + * that still faulted with grfifo=0xbadf1201; r1062: the full window + * stalls the front-end right after the PBDMA consumes it and + * fecs_cur never loads). Their page cannot join the launch page set + * (n_ptr is capped at 7 by the QMD pointer region and the draw's + * program/index/vertex/MME pages already fill it), so relocation is + * impossible; dropping is the same call the 0x10c NOTIFY drop + * already makes: the module appends its own SET_REPORT_SEMAPHORE + * tail and the host-side completion lanes author every completion + * the client actually waits on (the r1061 keep-prefix window + * retired with ch_rel=0xc0ffee02 and the client kept pushing + * frames without any 0x1b00 in the replay). + */ + return addr == 0x010cu || addr == 0x01b0u || addr == 0x05e0u || + addr == 0x1b00u; +} + +/* + * SESSION 153-CONTINUED-14 (FECS residency / draw-stall isolation): patch + * the DATA of one method inside a replayed 3D window before it reaches the + * engine. Format: comma-separated `addr:data` hex pairs, addr = method + * byte-address (e.g. 1618 = BEGIN, 1614 = END), data = the replacement + * value. Header-only methods (IMMD) get their 13-bit data field rewritten + * in place; count-carrying methods (INC/NON_INC/INC_ONCE) get their FIRST + * payload word replaced. Live-tunable via sysfs so one boot can sweep + * variants (BEGIN OP=7 QUADS -> TRIANGLES etc.) against the draw-stall. + */ +static char *gsp_phys_rmcall_gr_win_patch; +module_param_named(sema_gl_win_patch, gsp_phys_rmcall_gr_win_patch, charp, + 0644); +MODULE_PARM_DESC(sema_gl_win_patch, + "Hex addr:data pairs (comma-separated) patched into a captured 3D window method before replay (default empty)"); + +static bool gsp_phys_rmcall_gr_win_patch_find(u32 addr, u32 *out_data) +{ + const char *p = gsp_phys_rmcall_gr_win_patch; + u32 n = 0; + + if (!p || !*p) + return false; + while (*p && n < 64) { + u32 v = 0, d = 0; + u32 digits = 0; + bool have_data = false; + + while (*p == ' ' || *p == ',' || *p == ':' || *p == ';') + p++; + for (;;) { + char c = *p; + u32 h; + + if (c >= '0' && c <= '9') + h = c - '0'; + else if (c >= 'a' && c <= 'f') + h = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + h = c - 'A' + 10; + else + break; + if (v <= 0xfffffffu) + v = (v << 4) | h; + p++; + digits++; + } + if (!digits) + break; + if (*p == ':' || *p == '=') { + p++; + for (;;) { + char c = *p; + u32 h; + + if (c >= '0' && c <= '9') + h = c - '0'; + else if (c >= 'a' && c <= 'f') + h = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + h = c - 'A' + 10; + else + break; + if (d <= 0xfffffffu) + d = (d << 4) | h; + p++; + have_data = true; + } + } + n++; + if (have_data && v == addr) { + *out_data = d; + return true; + } + } + return false; +} + +static u32 gsp_phys_rmcall_gr_3d_last_subdrop; +static u32 gsp_phys_rmcall_gr_3d_last_sdmdrop; + +/* + * SESSION 153-CONTINUED-62: an MME-family payload word that neither the + * launch page set nor the level-2 resolver could translate has to stay + * verbatim in the emitted stream (never guessed) - but a raw client VA in + * an 0x3810 macro payload is exactly the operand the engine faults on + * (`MMU Fault HUBCLIENT_PD`). Until this helper the word was left silent: + * the abort census stayed 0 and the launch went out to fault seconds + * later (r1115: fault @0x0_04350000, the 0x4350900 program-page payload). + * Now the surviving miss counts toward the pre-doorbell MISS-ABORT and + * joins the pin demand set, so the launch is held (fail-closed, bounded) + * until the page resolves, exactly like a window pair miss. The float/ + * data-word class (0x10000000..0x100000000) is excluded - it can never + * resolve and must not pin the census nonzero (the r1109 lesson). + */ +static void gsp_phys_gr_3d_mme_miss(u32 wv) +{ + u64 page = (u64)wv & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + if (page >= 0x10000000ULL && page < 0x100000000ULL) + return; + gsp_phys_rmcall_gr_3d_miss_n++; + gsp_sema_demand_note(page); +} + + +static u32 gsp_phys_rmcall_gr_3d_last_progkeep; +static u32 gsp_phys_rmcall_gr_3d_patch_window(const u32 *in, u32 n, u32 *out, + u32 *out_n, u32 *n_dropped, + const struct gsp_phys_gr_submit *req, + u32 *n_shadow, + void __iomem *bar0, u32 epoch) +{ + u32 off = 0, wout = 0, reloc = 0, dropped = 0, subdrop = 0, sdm_drop = 0; + u32 wp_match = 0, wp_lowrun = 0, prog_keep = 0; + + *out_n = n; + *n_dropped = 0; + while (off < n) { + u32 hdr = in[off]; + u32 op = (hdr >> 29) & 0x7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 addr = (hdr & 0x1fffu) << 2; + u32 sub = (hdr >> 13) & 0x7u; + u32 span = 1u; + + if ((op == 1 || op == 3 || op == 5) && cnt && + off + 1u + cnt <= n) + span = 1u + cnt; + /* + * SESSION 153-CONTINUED-52: drop METHOD WRITES aimed at a + * non-zero subchannel. The glcore draw window ends with + * `2001c00d 70000000/68000000` = INC m=0x34 cnt=1 SUBCHANNEL 6 + * (channel-class MEM_OP_D lane) two-word runs. The ladder + * channel has nothing bound on subchannel 6, so the physical + * PBDMA hands those methods to its host-software lane and + * raises NV_PPBDMA_INTR_0_METHOD (bit 21): the r1075 relay + * records `Channel ID 00000004 intr 00200000` at every launch, + * and the real host driver's software-method handler answers + * "Invalid host sofware method received" -> rcRequired -> + * robust-channel error recovery kills the channel mid-window + * (fifo_gk104.c). The methods are channel-level barriers whose + * effects the module's own release/fence lanes already author, + * so dropping them from the replay is the same call the 0x10c + * NOTIFY / 0x1b0 LAUNCH_DMA auto-drops make. Only count- + * carrying writes (op 1/3/5) are candidates: the single-word + * TERT encodings (op 0/2/4/7) reuse bits 15:13 as mask/type + * payload. + */ + if ((op == 1u || op == 3u || op == 5u) && sub != 0u) { + dropped++; + subdrop++; + off += span; + continue; + } + /* + * SESSION 153-CONTINUED-59: single-word SDM-type entries + * (SEC_OP 0/2/6, see gsp_phys_rmcall_gr_sdm_drop). The + * retiring X-family windows never yield one; this guards a + * window that carries the client's SDM markers (the + * 0x1fff0-class marker words or the prelude 0x046x6dXX + * family), which the physical PBDMA refuses with PBENTRY. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sdm_drop) && + hdr && (op == 0u || op == 2u || op == 6u)) { + /* + * SESSION 153-CONTINUED-86: name every dropped SDM + * entry (bounded) so the A/B arm can be audited: the + * "keep the drop only for entries proven unrelated" + * decision needs the actual words, not the count. + */ + static u32 sdm_log; + + if (sdm_log < 24u) { + sdm_log++; + pr_info("vgsp-gr: SDM-DROP in[%u]=%08x op=%u addr=%#x sub=%u cnt/val=%#x (span=%u)\n", + off, hdr, op, addr, sub, + (cnt ? cnt : ((hdr >> 16) & 0x1fffu)), + span); + } + dropped++; + sdm_drop++; + off += span; + continue; + } + if (gsp_phys_rmcall_gr_win_drop_match(addr) || + gsp_phys_rmcall_gr_win_drop_auto_match(addr)) { + dropped++; + off += span; + continue; + } + memcpy(&out[wout], &in[off], (size_t)span * 4); + /* + * SESSION 153-CONTINUED-58: torn-read zero guard - see the + * sema_gl_attr_zero_fix comment. Only INC runs (op 1) whose + * data words advance one method each; the attribute runs are + * exactly this shape. Rebuild the run in place, skipping + * exactly-zero words and re-targeting the surviving words' + * methods (a zero word's method is simply never written, so + * the attribute keeps its prelude/previous-frame value). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_attr_zero_fix) && + op == 1u && cnt && addr >= 0x1160u && addr <= 0x11dcu) { + u32 k, k0 = 0, L = 0, wr = wout; + u32 subch = (hdr >> 13) & 0x7u; + + for (k = 0; k <= cnt; k++) { + u32 wv = (k < cnt) ? out[wout + 1u + k] : 0u; + /* + * SESSION 153-CONTINUED-62: the bad predicate + * also covers a torn ACTIVE attribute whose + * COMPONENT_BIT_WIDTHS enum is 0 (no such + * encoding exists): the r1113 setup-push + * capture carried + * `20030458 38400000 38200040 000081a4` + * (A(2)=0x81a4, widths=0) and the replayed + * prelude write trapped the engine - + * `Class Error: Class 0000c597, Offset + * 00001168, Data 000081a4, ErrorCode 0xc` + * -> grfifo=0xbadf1201 poison the moment the + * surf prelude let the draw execute. An + * inactive attribute (SOURCE=1, bit 6) keeps + * its word; only the invalid active encoding + * is dropped, so the attribute keeps its + * prelude/previous value until the window's + * own (valid) write lands. + */ + bool bad = (k < cnt) && + (wv == 0u || + (((wv >> 21) & 0x3fu) == 0u && + !(wv & 0x40u))); + + if (!bad && k < cnt) { + if (!L) + k0 = k; + out[wr + 1u] = out[wout + 1u + k]; + wr++; + L++; + continue; + } + if (L) { + u32 naddr = addr + 4u * k0; + u32 nhdr = (1u << 29) | + (L << 16) | + ((naddr >> 2) & 0x1fffu) | + (subch << 13); + + out[wr - L] = nhdr; + L = 0; + } + if (bad) { + static u32 az_log; + + if (az_log < 64) { + az_log++; + pr_info("vgsp-gr: ATTR-ZERO addr=0x%x word=%u/%u off=%u dropped (torn zero/invalid-width; attribute keeps its prelude value)\n", + addr + 4u * k, k, + cnt, off); + } + } + } + /* +1: the rebuilt run still needs its (new) header word */ + span = (wr - wout) + 1u; + } + { + /* + * SESSION 153-CONTINUED-14: live data patch (BEGIN + * OP sweep etc.). Header-only forms rewrite the 13-bit + * data field; count-carrying forms rewrite the first + * payload word. + */ + u32 pv; + + if (gsp_phys_rmcall_gr_win_patch_find(addr, &pv)) { + static u32 patch_log; + u32 old; + + wp_match++; + if (span > 1u) { + old = out[wout + 1]; + out[wout + 1] = pv; + } else { + old = (out[wout] >> 16) & 0x1fffu; + out[wout] = (out[wout] & ~0x1fff0000u) | + ((pv & 0x1fffu) << 16); + } + if (patch_log < 32) { + patch_log++; + pr_info("vgsp-gr: 3D win-patch addr=0x%x off=%u data 0x%x -> 0x%x\n", + addr, off, old, pv); + } + } else if (addr >= 0x100u && addr < 0x200u) { + /* + * SESSION 153-CONTINUED-84: the win_patch + * knob (sema_gl_win_patch=124:1) reaches + * this pass only if the window carries a + * run at the patch address; log the low-page + * runs so a decode mismatch is visible + * instead of silent. + */ + wp_lowrun++; + if (wp_lowrun <= 4u && + gsp_phys_rmcall_gr_win_patch && + *gsp_phys_rmcall_gr_win_patch) + pr_info("vgsp-gr: WIN-PATCH lowrun addr=0x%x op=%u cnt=%u hdr=%08x\n", + addr, op, cnt, hdr); + } + } + if ((op == 1 || op == 3 || op == 5) && cnt && + off + 1u + cnt <= n) { + u32 *d = &out[wout + 1]; + u32 lo_i = 0; + bool pair = false; + + if (op == 1 && addr == 0x0180 && cnt >= 4) { + lo_i = 2; /* OFFSET_OUT is v2/v3 */ + pair = true; + } else if (op == 1 && addr == 0x0f84 && cnt >= 2) { + pair = true; + } else if (op == 1 && addr == 0x155c && cnt >= 3) { + pair = true; + } else if (op == 1 && addr == 0x1574 && cnt >= 3) { + pair = true; + } else if (op == 1 && addr >= 0x2014 && + addr < 0x4000 && (addr & 0x3fu) == 0x14 && + cnt >= 2) { + pair = true; + } else if (op == 1 && (addr == 0x1c00 || addr == 0x1c10) && + cnt >= 3) { + /* SESSION 153-CONTINUED-10: SET_VERTEX_STREAM_ + * A_FORMAT(j)+BASE(j) run [fmt, upper, lower] + * (j*0x10 stride) - the vertex data the draw + * reads. Observed [00001000 1 2150b790]. */ + lo_i = 1; + pair = true; + } else if (READ_ONCE(gsp_phys_rmcall_gr_idx_reloc) && + op == 1 && addr >= 0x1c20 && addr <= 0x1cf0 && + (addr & 0xfu) == 0u && cnt >= 3) { + /* SESSION 153-CONTINUED-47 (A/B knob): widen + * the vertex-stream banks from j=0/1 to every + * FORMAT slot (0x1c20..0x1cf0). The GLX + * client's 3D window binds streams j>=2 that + * svc_gl_field already collects into the page + * set but the window never had rewritten. */ + lo_i = 1; + pair = true; + } else if (READ_ONCE(gsp_phys_rmcall_gr_idx_reloc) && + op == 1 && addr == 0x17c8 && cnt >= 2) { + /* SESSION 153-CONTINUED-47 (A/B knob): SET_ + * INDEX_BUFFER_A/B [ADDRESS_UPPER(7:0), + * ADDRESS_LOWER(31:0)] - the indexed draw + * fetches its indices through this VA. The + * r1042 boot showed an X window carrying + * index binds PRI-stalling once relocated + * (relocation of exactly this pair is the + * only kernel delta vs the r1035 baseline + * where index-bearing windows did not exist + * and every X window retired), so the + * rewrite is gated off by default and A/B-ed + * live per boot. Observed [00000000 + * 0450fcc8] -> VA 0x450fcc8. */ + lo_i = 0; + pair = true; + } else if (READ_ONCE(gsp_phys_rmcall_gr_idx_reloc) && + op == 1 && addr == 0x1b00 && cnt >= 2) { + /* SESSION 153-CONTINUED-48: NVC597_SET_REPORT_ + * SEMAPHORE_A/B..D (0x1b00) - the engine's own + * completion write [upper(7:0), lower(31:0), + * payload, flags] (observed [0, 0x4181bb0, + * 0x4f9f6, 0x1000f010] at the end of every + * glcore window). With the VA raw the replayed + * engine writes a client VA the GPC engines + * cannot reach - the VIRT_WRITE fault class + * (r1061: the kept [0,0x2000) window that + * faults is exactly the one still carrying the + * raw 0x1b00 VAs). Rewrite the pair to the + * shadow VA (same HPA, page collected by + * svc_gl_field 0x1b00), so the write lands in + * the client's own page - its wait then sees a + * real hardware completion. MISS stays + * verbatim like every other pair. */ + lo_i = 0; + pair = true; + } else if (op == 1 && addr == 0x2380 && cnt >= 3) { + /* SET_CONSTANT_BUFFER_SELECTOR_A/B: + * [size, upper, lower]; observed + * [00010000 1 2150a500]. */ + lo_i = 1; + pair = true; + } else if (op == 1 && addr >= 0x0800 && addr < 0x0c00 && + (addr & 0x3fu) == 0u && cnt >= 2) { + /* SET_COLOR_TARGET_A/B(j) offset pair: + * [upper, lower, w, h, fmt]; the surface the + * draw rasterizes into (must live in guest + * RAM for the present lane to see pixels). + * Observed [1 222fa200 40 4 f7 0]. */ + lo_i = 0; + pair = true; + } else if (op == 1 && addr == 0x218 && cnt >= 4 && + (d[2] & ~0xffu) == 0u && + (d[3] & 0xfffu) == 0u) { + /* SESSION 153-CONTINUED-27: the TWOD twin + * of the colour target - INC 0x218 cnt>=4 + * covers 0x218..0x224 = [w, h, upper, + * lower] (the emulator's SET_DST sequence; + * the glxgears client window fills the X + * screen through it). Shape-checked so a + * non-surface run can never be rewritten. */ + lo_i = 2; + pair = true; + } + if (pair) { + u64 pva = ((u64)(d[lo_i] & 0xffu) << 32) | + (u64)(d[lo_i + 1] & 0xfffffffcu); + static u32 pair_log; + bool ok = false, kept = false; + + /* + * SESSION 153-CONTINUED-92: the program-family + * space experiment (see sema_gl_prog_guest_va). + * The DS rejects every bound stage whose SPH it + * fetches through a relocated shadow VA while + * the bytes at those VAs are provably correct; + * leave the operand at its ORIGINAL guest VA + * whenever the page resolves + identity-maps, + * so the DS fetches/validates in the same VA + * space the client's program heap describes. + * Kept pairs count as relocated so the SPH-BIND + * guard (orel == nrun) is unaffected; a page the + * identity mapper refuses falls through to the + * shadow rewrite exactly as before. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_prog_guest_va) && + bar0 && req->prog_va && + addr >= 0x2014u && addr < 0x4000u && + (addr & 0x3fu) == 0x14u && + gsp_phys_rmcall_gr_prog_keep_locked( + bar0, epoch, d[lo_i], d[lo_i + 1], + req)) { + static u32 progva_log; + + kept = true; + prog_keep++; + if (progva_log < 24u) { + progva_log++; + pr_info("vgsp-gr: PROG-GUEST-VA keep addr=0x%x va=0x%llx (Tier-1 identity; operand left verbatim) h=0x%x\n", + addr, pva, + req->h_object); + } + } else { + ok = gsp_phys_rmcall_gr_3d_reloc( + &d[lo_i], &d[lo_i + 1], req, + n_shadow); + } + if (ok || kept) + reloc++; + if (!kept && pair_log < 48) { + pair_log++; + pr_info("vgsp-gr: pair addr=0x%x va=0x%llx -> %s0x%llx\n", + addr, pva, ok ? "" : + "MISS ", + ((u64)(d[lo_i] & 0xffu) << 32) | + (u64)(d[lo_i + 1] & 0xfffffffcu)); + } + } + /* + * SESSION 153-CONTINUED-48: MME-family payload + * relocation (see the sema_gl_mme_reloc param). The + * C597 MME macro/data family spans 0x3800..0x3a00; + * its runs carry the macro arguments as raw payload + * words, which include 32-bit client VAs + * (0x450f800/0x450f000 observed in the beef macro(2) + * run at 0x3810). Rewrite only a word whose page the + * launch's own page set resolves (the same vetted + * shadow mapping the pairs use); a MISS, an + * unaligned/too-small word, or a shadow VA that + * cannot fit the 32-bit slot stays verbatim. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_reloc) && + addr >= 0x3800u && addr < 0x3a00u) { + u32 wi; + + for (wi = 0; wi < cnt; wi++) { + u32 wv = d[wi]; + u32 up = 0, lo = wv; + static u32 mme_log; + + if (wv < 0x10000u || (wv & 3u)) + continue; + if (!gsp_phys_rmcall_gr_3d_reloc( + &up, &lo, req, n_shadow)) { + /* + * SESSION 153-CONTINUED-51: + * the payload page missed the + * launch's page set (the QMD + * pointer region caps the set + * far below a full window's + * operand count - r1071: the + * macro payload pages + * 0x44fd200..0x44fde00 were + * ~12 distinct pages while the + * set held 7; the words stayed + * RAW and the engine fetched an + * untranslatable guest VA on + * every full window - the + * 0xbadf1201 priv-error + * poison). Resolve the page + * through the SAME trusted + * ledger/pin/walk chain the + * launch pages use (the + * RELOC-FIXED resolver) and + * shadow-map it exactly like a + * matched page. A resolve/map + * miss or a shadow VA that + * cannot fit the 32-bit slot + * stays verbatim (never + * guessed) - and, SESSION + * 153-CONTINUED-62, it now + * COUNTS toward the abort + * census and joins the pin + * demand set: r1115's residual + * fault was exactly this class + * (`MMU Fault HUBCLIENT_PD @ + * 0x0_04350000` 2.5 s after a + * launch whose census read 0 + * because an 0x3810 payload + * word 0x4350900 missed here + * silently). + */ + struct gsp_sema_gr_reloc r; + u64 va2 = (u64)wv; + u64 sh2 = 0; + u32 line2 = 0; + + if (gsp_sema_gr_reloc_resolve( + va2, &r) || + !r.hpa) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + if (*n_shadow >= + GSP_PHYS_RMCALL_GR_SHADOW_PAGES) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + if (gsp_phys_rmcall_gr_map_shadow_locked( + *n_shadow, r.hpa, + &sh2, &line2)) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + if (line2 != 0xffffffffu) + (*n_shadow)++; + /* CONT-73: MME shadow-scratch + * destination maps this guest + * HPA 1:1; record it so a store + * through the MME path is + * witnessed too. */ + gsp_phys_gr_wit_note( + va2 & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1), + r.hpa); + sh2 |= va2 & + (GSP_PHYS_PAGE_SIZE - 1); + up = (u32)(sh2 >> 32) & 0xffu; + lo = (wv & 0x3u) | + ((u32)sh2 & 0xfffffffcu); + if (up != 0) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + d[wi] = lo; + reloc++; + if (mme_log < 32) { + mme_log++; + pr_info("vgsp-gr: mme-reloc-rsv addr=0x%x wi=%u w=0x%x -> 0x%x\n", + addr, wi, wv, + lo); + } + continue; + } + if (up != 0) { + gsp_phys_gr_3d_mme_miss(wv); + continue; + } + d[wi] = lo; + reloc++; + if (mme_log < 32) { + mme_log++; + pr_info("vgsp-gr: mme-reloc addr=0x%x wi=%u w=0x%x -> 0x%x\n", + addr, wi, wv, lo); + } + } + } + wout += span; + off += 1u + cnt; + continue; + } + off++; /* header-only / resync, like the service decoder */ + wout++; + } + /* + * SESSION 153-CONTINUED-84: prove the win_patch knob reached the + * pass (the r1236 "never logged" was a late write, not a decode + * gap - this line makes the distinction explicit on every boot). + */ + if (gsp_phys_rmcall_gr_win_patch && *gsp_phys_rmcall_gr_win_patch) { + static u32 wp_dbg; + + if (wp_dbg < 24u) { + wp_dbg++; + pr_info("vgsp-gr: WIN-PATCH sweep knob=%s runs=%u matched=%u lowruns=%u h=0x%x\n", + gsp_phys_rmcall_gr_win_patch, n, wp_match, + wp_lowrun, + req ? req->h_object : 0u); + } + } + *out_n = wout; + *n_dropped = dropped; + gsp_phys_rmcall_gr_3d_last_subdrop = subdrop; + gsp_phys_rmcall_gr_3d_last_sdmdrop = sdm_drop; + gsp_phys_rmcall_gr_3d_last_progkeep = prog_keep; + return reloc; +} + +/* + * SESSION 153-CONTINUED-27: alias-build for a big colour target. Consumes + * ceil(npages/16) never-used scratch-PT fetch lines and writes one PTE per + * surface page, so the whole surface is reachable through one CONTIGUOUS + * alias VA (lines are monotonic; consecutive lines are VA-contiguous + * inside a pool block - both pool blocks are checked). Every page's HPA + * comes from the launch's own page set when present, else from the + * service's trusted resolver - the identical vetted translation the + * shadow relocation uses. Lines are consumed even on failure (they are + * never reused, so a half-written line can never be shadowed later); + * the caller then falls back to the existing per-page path. Called with + * gsp_phys_rm_lock held. + */ +static bool gsp_phys_gr_surface_alias_locked(const struct gsp_phys_gr_submit *req, + u64 va, u64 npages, u64 *alias_out) +{ + u64 map_pages = npages + GSP_PHYS_GR_SURF_SLACK_PAGES; + u32 nlines; + u32 base_line; + u32 i, j; + u64 first_va, last_va; + + if (map_pages > 4096u) + map_pages = 4096u; + nlines = (u32)((map_pages + + GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES - 1) / + GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES); + base_line = boot_stage.rmcall_guest_pt_next_line; + if (!nlines || base_line + nlines > GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) + return false; + first_va = gsp_phys_rmcall_guest_pt_win_va( + base_line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + + ((u64)(base_line % + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) << 16); + last_va = gsp_phys_rmcall_guest_pt_win_va( + (base_line + nlines - 1u) / + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + + ((u64)((base_line + nlines - 1u) % + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) << 16); + /* one contiguous run: never straddle a pool block boundary */ + if (last_va != first_va + ((u64)(nlines - 1u) << 16)) + return false; + /* + * Consume the lines before writing: a failure mid-way leaves the + * partially written lines consumed (never reused), so the monotonic + * line invariant - a cached line can never shadow a later mapping - + * holds unconditionally. + */ + boot_stage.rmcall_guest_pt_next_line = base_line + nlines; + for (i = 0; i < nlines; i++) { + u32 line = base_line + i; + u32 page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + u32 lip = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + u8 *pt; + + if (gsp_phys_rmcall_gr_scratch_win_install_locked(page)) + return false; + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) + return false; + for (j = 0; j < GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES; j++) { + u64 k = (u64)i * GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES + + j; + u64 pv, hpa = 0; + u32 pi; + + if (k >= map_pages) + break; + pv = va + (k << 12); + for (pi = 0; pi < req->n_pages; pi++) { + if ((req->pages[pi].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == pv && + req->pages[pi].hpa) { + hpa = req->pages[pi].hpa; + break; + } + } + if (!hpa) { + struct gsp_sema_gr_reloc r; + + if (gsp_sema_gr_reloc_resolve(pv, &r) || + !r.hpa) { + /* + * A declared page must resolve; a + * slack-tail page past the guest + * allocation gets the dummy so the + * engine's overdraw vanishes. + */ + if (k < npages) + return false; + hpa = gsp_phys_gr_surf_dummy_dma_locked(); + if (!hpa) + return false; + } else { + hpa = r.hpa; + } + } + put_unaligned_le64( + gsp_phys_rmcall_pte(hpa), + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + lip + + j) * + 8); + } + dma_wmb(); + } + *alias_out = first_va; + return true; +} + +/* + * SESSION 153-CONTINUED-27: pre-pass over a raw 3D window. Scans the + * colour-target runs (C597 INC 0x800+j*0x40 [upper, lower, w, h, fmt, + * blksz] and - same surface, shape seen on the TWOD subchannel - INC + * 0x218 cnt>=4 [w, h, upper, lower]) and aliases each big surface once + * per boot so the relocation pass can rewrite its operands to the alias + * (r992/r993 evidence: the X screen 0x120400000, 1280x720 fmt 0xe6, is + * the draw's render target AND the TWOD fill's destination). Small + * surfaces keep the historical one-pair shadow relocation. Called with + * gsp_phys_rm_lock held. + */ +static void gsp_phys_gr_surface_alias_scan_locked( + const u32 *win, u32 n, const struct gsp_phys_gr_submit *req) +{ + u32 off = 0; + + while (off < n) { + u32 hdr = win[off]; + u32 op = (hdr >> 29) & 0x7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 addr = (hdr & 0x1fffu) << 2; + u32 span = 1u; + + if ((op == 1 || op == 3 || op == 5) && cnt && + off + 1u + cnt <= n) { + span = 1u + cnt; + } else { + off++; + continue; + } + if (op == 1 && ((addr >= 0x0800 && addr < 0x0c00 && + (addr & 0x3fu) == 0u && cnt >= 4) || + (addr == 0x218 && cnt >= 4))) { + const u32 *d = &win[off + 1u]; + u64 va; + u32 w, h, fmt; + u64 bytes, npages; + u32 sdi; + u64 alias = 0; + + if (addr == 0x218) { + va = ((u64)(d[2] & 0xffu) << 32) | + (u64)(d[3] & 0xfffff000u); + w = d[0]; + h = d[1]; + fmt = 0xe6u; /* TWOD run carries fmt on + * its own IMMD; the surface + * class here is 32bpp */ + } else { + va = ((u64)(d[0] & 0xffu) << 32) | + (u64)(d[1] & 0xfffff000u); + w = d[2]; + h = d[3]; + fmt = d[4]; + } + if (!va || (va & 0xfffu) || fmt != 0xe6u || + !w || !h || w > 8192u || h > 8192u) + goto next; + bytes = (u64)w * 4u * (u64)h; + npages = (bytes + GSP_PHYS_PAGE_SIZE - 1) >> 12; + if (npages < 2u || npages > 4096u) + goto next; + for (sdi = 0; sdi < gsp_phys_gr_surf_n; sdi++) + if (gsp_phys_gr_surf[sdi].va == va) + goto next; /* already aliased */ + if (gsp_phys_gr_surface_alias_locked(req, va, npages, + &alias)) { + if (gsp_phys_gr_surf_n < + ARRAY_SIZE(gsp_phys_gr_surf)) { + gsp_phys_gr_surf[gsp_phys_gr_surf_n].va = + va; + gsp_phys_gr_surf[gsp_phys_gr_surf_n] + .alias = alias; + gsp_phys_gr_surf[gsp_phys_gr_surf_n] + .npages = npages; + gsp_phys_gr_surf_n++; + } + pr_info("vgsp-gr: SURFACE alias va=0x%llx pages=%llu (%ux%u fmt=0x%x) -> 0x%llx (contiguous shadow run; operands rewritten)\n", + va, npages, w, h, fmt, alias); + } else { + pr_info("vgsp-gr: SURFACE alias FAILED va=0x%llx (%ux%u fmt=0x%x): operands fall to the per-page path\n", + va, w, h, fmt); + } + } +next: + off += span; + } +} + +/* + * SESSION 153-CONTINUED-66: write a built method block into the module + * pushbuffer. The block may exceed one 4 KB page (GSP_PHYS_RMCALL_GR_METHOD_ + * WORDS is 2048), so copy page-wise: page 0 is the registered BUF_PB chunk, + * page 1 is the PTE-planted rmcall_gr_pb2 page at VA_PB2. The two pages are + * contiguous in GPU VA (VA_PB..VA_PB+0x2000), so the GPFIFO entry's single + * fetch address + length walks both. A short/absent page-2 plant leaves the + * tail unwritten and the caller's bound check keeps such launches <= 4 KB. + */ +static void gsp_phys_gr_pb_copy(const u32 *methods, u32 words) +{ + struct gsp_phys_boot_chunk *pb = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; + u32 per = GSP_PHYS_PAGE_SIZE / 4u; + u32 done = 0; + + while (done < words) { + struct gsp_phys_boot_chunk *c; + u32 page = done / per; + u32 off = done % per; + u32 n = words - done; + u8 *dst; + + if (n > per - off) + n = per - off; + if (!page) + c = pb; + else if (page == 1u) + c = &boot_stage.rmcall_gr_pb2; + else + return; + dst = gsp_phys_chunk_page_va(c, 0); + if (page == 2u || page == 3u) { + /* + * SESSION 153-CONTINUED-77: pages 1 and 2 of the pb2 + * chunk are the third/fourth pushbuffer pages + * (VA_PB3 slot 0x103, VA_PB4 slot 0x104). + */ + if (boot_stage.rmcall_gr_pb2.pages < page) + return; + dst = gsp_phys_chunk_page_va(&boot_stage.rmcall_gr_pb2, + page - 1u); + } + if (!dst) + return; + memcpy(dst + (size_t)off * 4u, methods + done, (size_t)n * 4u); + done += n; + } +} + +/* + * SESSION 153-CONTINUED-95: CTX-BIND launch receipt - is the client's + * promoted GR context buffer engine-visible on the replay channel? + * + * The guest driver promotes its per-channel context buffers through + * NV2080_CTRL_CMD_GPU_PROMOTE_CTX entries (MAIN bufferId 0 and PATCH + * bufferId 2 carry bInitialize=1 on a fresh context). The emulated GSP-RM + * answers SUCCESS and records the statements (gsp_sema_service_ctx_ + * promote_note), but it does not install them anywhere: the replay + * channel's engine context is the buffer the kernel's own ladder built + * (server FBMEM MAIN, mapped at GSP_PHYS_RMCALL_VA_GR_CTX and committed by + * the mode-35 lane promote). This receipt logs, at the first launches of a + * boot, the exact promote statements next to the engine's binding so the + * "context state never replayed" hypothesis is decided by bytes: the + * client's MAIN/PATCH pa/va/size/bInitialize, and the first words of the + * client MAIN buffer read through the GPA->HPA map (a zeroed buffer is the + * expected state before a server-side ctx init that the emulation never + * performs). + */ +static void gsp_phys_ctx_bind_receipt(void) +{ + static u32 receipt_n; + u64 cpa = 0, cva = 0, csz = 0, ppa = 0, pva = 0, psz = 0; + u32 cattr = 0, cinit = 0, pattr = 0, pinit = 0; + bool have_main, have_patch; + + if (receipt_n >= 4u) + return; + receipt_n++; + have_main = gsp_sema_service_ctx_promote_get(GSP_PHYS_RMCALL_ENGINE_GR0, + 0u, &cpa, &cva, &csz, + &cattr, &cinit) == 0; + have_patch = gsp_sema_service_ctx_promote_get(GSP_PHYS_RMCALL_ENGINE_GR0, + 2u, &ppa, &pva, &psz, + &pattr, &pinit) == 0; + pr_info("vgsp-gr: CTX-BIND client-main=%s pa=0x%llx va=0x%llx size=0x%llx attr=0x%x binit=%u; client-patch=%s pa=0x%llx va=0x%llx size=0x%llx binit=%u; replay-ctx pa=0x%llx size=0x%x fbmem=%u va=0x%llx srv_ctxva=%u; installed=0 (engine ctx is the ladder's buffer, NOT the client promote)\n", + have_main ? "seen" : "none", cpa, cva, csz, cattr, cinit, + have_patch ? "seen" : "none", ppa, pva, psz, pinit, + boot_stage.rmcall_ctx_pa, boot_stage.rmcall_ctx_size, + boot_stage.rmcall_ctx_fbmem ? 1u : 0u, + boot_stage.rmcall_ctx_va, boot_stage.rmcall_srv_ctxva ? 1u : 0u); + if (have_main && cpa) { + u64 hpa = 0; + + if (gsp_phys_gpa_to_hpa_vgpu(NULL, cpa & ~(u64)(GSP_PHYS_PAGE_SIZE - 1), &hpa) == 0 && + pfn_valid(hpa >> PAGE_SHIFT)) { + const u32 *w = (const u32 *)__va(hpa); + + pr_info("vgsp-gr: CTX-BIND client-main content[0..7]=%08x %08x %08x %08x %08x %08x %08x %08x (gpa=0x%llx hpa=0x%llx)\n", + get_unaligned_le32((const u8 *)&w[0]), + get_unaligned_le32((const u8 *)&w[1]), + get_unaligned_le32((const u8 *)&w[2]), + get_unaligned_le32((const u8 *)&w[3]), + get_unaligned_le32((const u8 *)&w[4]), + get_unaligned_le32((const u8 *)&w[5]), + get_unaligned_le32((const u8 *)&w[6]), + get_unaligned_le32((const u8 *)&w[7]), + cpa, hpa); + } else { + pr_info("vgsp-gr: CTX-BIND client-main content: GPA 0x%llx -> HPA translation unavailable (no guest RAM registration yet)\n", + cpa); + } + } +} + +/* + * SESSION 153-CONTINUED-95: engine context readback (gsp-phys-ctx-dump). + * + * What the DS validates SPHs against is the *engine's* context buffer, not + * this host's view of anything the client promoted. The replay channel's + * context is the ladder's buffer (server FBMEM MAIN) mapped into the + * channel VAS at GSP_PHYS_RMCALL_VA_GR_CTX with VIDMEM PTEs. The host has + * no aperture to FBMEM (no BAR1 window), so the copy engine reads it: one + * 4 KB CE copy per page from the ctx VA to a host page placed on a fresh + * never-reused scratch fetch line, polled on the module semaphore exactly + * like the P6a guest copies. Runs once per boot at the first launch (the + * channel is idle: single-inflight work item), while the ladder's channel + * is not server-memory and the identity DMA domain holds. Bounded: stops + * at the first refused/timed-out page and records how many bytes landed. + */ +#define GSP_PHYS_CTX_RB_BYTES 0x140000u /* MAIN 0x133000 + PATCH 0x5000 + slack */ + +static u8 *gsp_phys_ctx_rb_buf; +static u32 gsp_phys_ctx_rb_len; +static bool gsp_phys_ctx_rb_tried; + +static const u8 *gsp_phys_ctx_readback_get(u32 *len) +{ + if (len) + *len = READ_ONCE(gsp_phys_ctx_rb_len); + return gsp_phys_ctx_rb_buf; +} + +/* + * SESSION 153-CONTINUED-96: re-arm the one-shot readback. A write to the + * gsp-phys-ctx-read debugfs file clears the tried flag (and the length), so + * the next forwarded launch re-runs the copy with the current server state. + * Safe from any context: the readback itself only runs in the single-inflight + * launch path and re-checks every gate. + */ +static void gsp_phys_ctx_readback_rearm(void) +{ + WRITE_ONCE(gsp_phys_ctx_rb_len, 0); + WRITE_ONCE(gsp_phys_ctx_rb_tried, false); +} + +static void gsp_phys_ctx_readback_try(void) +{ + struct gsp_phys_boot_chunk *sem = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM]; + void __iomem *bar0; + struct page *pg; + u64 hpa; + u32 i, bytes = 0; + u32 ctx_pages, patch_pages = 0; + + if (READ_ONCE(gsp_phys_ctx_rb_tried)) + return; + if (!boot_stage.rmcall_ctx_valid || !boot_stage.rmcall_ctx_pa) + return; + /* + * SESSION 153-CONTINUED-96: the copies run on the secondary COPY0 + * lane (gsp_phys_rmcall_ce_lane_bringup). CE class methods are never + * legal on the GR0 ladder channel, so the lane's readiness (alloc + + * BIND/schedules + its own token) is the hard requirement here; the + * GR channel pieces are not used by the copy. Gate on the pieces the + * lane submission needs: claimed, RM init-done, not unloading. + */ + if (!boot_stage.rmcall_ce_lane_valid || !boot_stage.rmcall_ce_token) + return; + if (gsp_phys_backend_state() != GSP_PHYS_BACKEND_READY) + return; + if (!gsp_phys_gpu_claimed() || !boot_stage.rm_init_done || + boot_stage.rm_unload_submitted) + return; + if (gsp_phys_dma_domain_identity() != 1) + return; + bar0 = gsp_phys_gpu_bar0(); + if (!bar0) + return; + if (!gsp_phys_ctx_rb_buf) { + gsp_phys_ctx_rb_buf = vmalloc(GSP_PHYS_CTX_RB_BYTES); + if (!gsp_phys_ctx_rb_buf) + return; + } + /* + * SESSION 153-CONTINUED-96: copy the WHOLE MAIN context (the golden + * register image the engine restores) and the PATCH buffer (the + * context patch area the DS's SPH region may live in), in that order, + * after the 64 KB r1323/r1324 first run only proved the head. The + * dump file serves MAIN first, then PATCH; lengths are logged. + */ + ctx_pages = DIV_ROUND_UP(boot_stage.rmcall_ctx_size, + GSP_PHYS_PAGE_SIZE); + if (ctx_pages > (GSP_PHYS_CTX_RB_BYTES / GSP_PHYS_PAGE_SIZE)) + ctx_pages = GSP_PHYS_CTX_RB_BYTES / GSP_PHYS_PAGE_SIZE; + if (boot_stage.rmcall_golden_fb_aux && boot_stage.rmcall_patch_size) { + patch_pages = DIV_ROUND_UP(boot_stage.rmcall_patch_size, + GSP_PHYS_PAGE_SIZE); + if (ctx_pages + patch_pages > + GSP_PHYS_CTX_RB_BYTES / GSP_PHYS_PAGE_SIZE) + patch_pages = GSP_PHYS_CTX_RB_BYTES / + GSP_PHYS_PAGE_SIZE - ctx_pages; + } + pg = alloc_page(GFP_KERNEL); + if (!pg) + return; + hpa = (u64)page_to_phys(pg); + for (i = 0; i < ctx_pages + patch_pages; i++) { + u64 src_base = (i < ctx_pages) ? + GSP_PHYS_RMCALL_VA_GR_CTX : + GSP_PHYS_RMCALL_VA_GR_PATCH; + u64 page_off = (i < ctx_pages) ? + (u64)i * GSP_PHYS_PAGE_SIZE : + (u64)(i - ctx_pages) * GSP_PHYS_PAGE_SIZE; + u64 copy_src_va = src_base + page_off; + u64 map_src_va = 0, dst_va = 0, sem_va = 0; + u32 poll = 0; + unsigned long deadline; + int ret; + + mutex_lock(&gsp_phys_rm_lock); + ret = gsp_phys_rmcall_guest_map_locked(hpa, hpa, 0, &map_src_va, + &dst_va, &sem_va); + if (ret) { + mutex_unlock(&gsp_phys_rm_lock); + break; + } + memset((u8 *)__va(hpa), 0xab, GSP_PHYS_PAGE_SIZE); + ret = gsp_phys_rmcall_ce_lane_submit_locked(bar0, + copy_src_va, dst_va, GSP_PHYS_RMCALL_VA_SEM, + GSP_PHYS_PAGE_SIZE, + GSP_PHYS_RMCALL_SEM_PAYLOAD, NULL); + mutex_unlock(&gsp_phys_rm_lock); + if (ret) { + pr_warn("vgsp-gr: CTX-READBACK page %u submit refused (%d) after %u bytes; stopping\n", + i, ret, bytes); + break; + } + deadline = jiffies + msecs_to_jiffies(250); + for (;;) { + poll = READ_ONCE(*(u32 *)gsp_phys_chunk_page_va(sem, 0)); + if (poll == GSP_PHYS_RMCALL_SEM_PAYLOAD) + break; + if (time_after_eq(jiffies, deadline)) { + poll = 0; + break; + } + gsp_phys_rmcall_wait_quantum(); + } + if (poll != GSP_PHYS_RMCALL_SEM_PAYLOAD) { + pr_warn("vgsp-gr: CTX-READBACK page %u (src VA 0x%llx) no CE release within 250 ms after %u bytes; stopping\n", + i, copy_src_va, bytes); + break; + } + memcpy(gsp_phys_ctx_rb_buf + (size_t)i * GSP_PHYS_PAGE_SIZE, + (const u8 *)__va(hpa), GSP_PHYS_PAGE_SIZE); + bytes += GSP_PHYS_PAGE_SIZE; + } + __free_page(pg); + WRITE_ONCE(gsp_phys_ctx_rb_len, bytes); + WRITE_ONCE(gsp_phys_ctx_rb_tried, true); + pr_info("vgsp-gr: CTX-READBACK engine ctx pa=0x%llx copied=%u bytes on the CE lane (MAIN %u pages then PATCH %u pages; server/own buffer, not the client promote; WPR2 untouched)\n", + boot_stage.rmcall_ctx_pa, bytes, ctx_pages, patch_pages); } int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, @@ -17141,15 +24178,17 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; void __iomem *bar0; struct gsp_phys_rmcall_ce_submit sub; - struct gsp_phys_fault_info fault = { }; /* * Static scratch: the only caller is the single-inflight sema work * item (ordered gsp-phys-work queue), so no concurrent submission can - * touch these. Keeping the 96-dword method block and the 64-word QMD - * copy off the stack keeps the frame under the warning bound. + * touch these. Keeping the 96-dword method block, the 64-word QMD + * copy and the ~370-byte fault record off the stack keeps the frame + * under the warning bound. */ static u32 methods[GSP_PHYS_RMCALL_GR_METHOD_WORDS]; + static u32 stream_patch[GSP_PHYS_RMCALL_GR_METHOD_WORDS]; static u32 qmd[GSP_PHYS_GR_QMD_WORDS]; + static struct gsp_phys_fault_info fault; unsigned long start = jiffies; unsigned long deadline; u8 *sem_va; @@ -17158,14 +24197,34 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, const u8 *guest_param_pg; u32 want_qmd = 0, want_ch = 0; bool qmd_rel = false, ch_rel = false; + /* + * r929 (Path A): this submission is a raw 3D method window. The QMD + * machinery (qmdwin relocation, QMD prewrite, release0 redirect, + * PCAS) does not apply: the block is SET_OBJECT C597 + the window + + * the channel release tail, and completion is the channel release + * alone (qmd_rel stays off so the poll does not wait for a QMD + * release word no engine will write). + */ + bool stream3d = gsp_phys_rmcall_gr_3d && req && req->stream_n; + /* + * CONT-73: fresh direct-write witness for this submission. The + * window/attr patch passes below record every relocated operand + * destination's guest page; the pre-doorbell capture and the retire + * report read exactly this launch's set (single-inflight, statics). + */ + gsp_phys_gr_wit_reset(); + const u32 *win3d = NULL; + u32 win3d_n = 0; + bool qmd_rel_redirected = false; /* r420b: release0 -> module sem */ u64 qmd_rel_hpa = 0, ch_rel_hpa = 0; u64 ch_sem_va = GSP_PHYS_RMCALL_VA_SEM + 4; u64 qmd_dst = GSP_PHYS_RMCALL_VA_SRC; u8 *qmd_dst_mem = NULL; u32 qmd_rel_off = 0, ch_rel_off = 0; u32 poll = 0, w0 = 0, w1 = 0, grpoll_n = 0; - static u32 sked_prev[11]; - static u32 sked_sample[11]; + bool sem_gated = false; + static u32 sked_prev[28]; + static u32 sked_sample[28]; static u32 sked_tick; static bool sked_first; /* @@ -17178,19 +24237,119 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * stack). Safe as static: the only caller is the single-inflight * ordered gsp-phys-work item. */ - static u32 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + static u64 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + /* + * r378: which guest page's content each bounce slot holds this + * launch (set by the copy-in fills, cleared per submission). The + * copy-out writes a slot back only to that exact page, so a rec + * aliasing another rec's slot - or a slot left over from a previous + * launch - can never ship foreign content into a guest page. + * Static like bounce_csum_in: the single-inflight ordered + * gsp-phys-work item is the only caller. + */ + static u64 bounce_staged_hpa[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; sked_tick = 0; sked_first = true; guest_param_pg = NULL; + /* + * SESSION 153-CONTINUED-95: prove the client's promoted GR context + * buffer binding at the first launches, then read the engine's own + * context buffer back for byte-level inspection (both bounded, both + * diagnostic; no behavior change to the launch itself). + */ + gsp_phys_ctx_bind_receipt(); + gsp_phys_ctx_readback_try(); u32 qmd_win_pde = 0, qmd_win_installed = 0; int qmd_win_ret = 0; bool qmd_win_diag = false; bool qmdwin_on = false; + u32 qw_bank = 0; char qmdwin_note[32] = ""; bool rearmed = false; u32 attempts = 0; + /* + * SESSION 153-CONTINUED-48: one bounded priv-error recovery attempt + * per launch (sema_gr_wedge_recover), with a per-boot tally for the + * report. Static like the other submit scratch: the single-inflight + * ordered work item is the only caller. + */ + bool wedge_recover_done = false; + static u32 wedge_recoveries; + /* + * SESSION 153-CONTINUED-72: the MME-FAST pre-reset spin can observe + * the 0xbadf poison and then the FECS clears it before the poll + * loop's first recovery check reads grfifo (the r1149 race: recovery + * missed -> prelude latch stuck -> engine wedged for the boot). The + * observation is latched here and OR-ed into the WEDGE-RECOVER gate. + */ + bool fast_poison = false; + /* + * F2/F4 state: the fast first-timeout re-arm runs once per launch; the + * deposit readback is one-shot per launch (the first timeout is the + * evidence window - the old r198 post-mortem only ran after attempt 3, + * which recovered launches never reached); poll_ms is the window in + * force for the current poll, so the timeout log reports it accurately. + * Static like the other submit scratch: the single-inflight ordered + * work item is the only caller, and statics keep the frame at its + * warning-bound size. + */ + static bool fast_rearmed; + static bool deposit_rb_done; + static u16 poll_ms; + static u32 poll_start_us; + /* + * GRFIX FIX3: per-launch relocation coverage. Static like the other + * submit scratch: the single-inflight ordered work item is the only + * caller, and statics keep the frame at its warning-bound size. + */ + static u32 reloc_gap; + static u64 reloc_gap_first; int ret = 0; + fast_rearmed = false; + deposit_rb_done = false; + gsp_phys_gr_codehash_done = false; /* CODEHASH: once per launch */ + /* + * M91 FAULT-CAP: snapshot the fault epoch per launch, so the capture + * at the launch's end attributes exactly the faults consumed inside + * this window (and a refusal path that never publishes compares equal + * and stays inert). + */ + gsp_phys_m91_fault_start = READ_ONCE(gsp_phys_fault_epoch); + reloc_gap = 0; + reloc_gap_first = 0; + /* + * relfix: the late-resolved page list is per-launch scratch exactly + * like bounce_csum_in; the single-inflight work item is the only + * caller. + */ + gsp_phys_gr_gap_n = 0; + gsp_phys_gr_gap_word_n = 0; + gsp_phys_gr_reloc_fixed = 0; + gsp_phys_gr_reloc_fixed_first = 0; + gsp_phys_gr_outwit_n = 0; /* COPYWIT: per-launch alias table */ + gsp_phys_gr_gap_logged_n = 0; + /* + * The fault record is static scratch (frame bound): reset it per + * submission so a previous launch's fault can never leak into this + * one's result - the old on-stack `= { }` semantics. + */ + memset(&fault, 0, sizeof(fault)); + /* + * r378: the bounce staging identity is per launch. A slot left over + * from a previous launch (or never staged this one) must not be + * treated as this launch's page even when the HPA matches. + */ + memset(bounce_staged_hpa, 0, sizeof(bounce_staged_hpa)); + /* + * F2: the shortened first window is part of the fast path; with + * gsp_phys_gr_retry_fast=0 the launch keeps today's exact behavior + * (the historical 5 s first window and ladder). + */ + poll_ms = gsp_phys_gr_retry_fast_value() ? + (u16)gsp_phys_gr_first_poll_ms_value() : + (u16)GSP_PHYS_RMCALL_GR_POLL_MS; + if (out) memset(out, 0, sizeof(*out)); if (!req || !req->qmd || @@ -17200,6 +24359,26 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, gsp_phys_gr_refused++; return -EINVAL; } + /* + * r564: refuse a degenerate data set BEFORE the engine ever sees it. + * r565 evidence: the fill dropped 4 argument bases (GRFILL-DROP), + * RDW showed nrecs=2 where healthy launches run 968-969, the launch + * was still submitted and the backend timed out after ~19 s + * ("GR forward failed rc=-110"), after which the tail authored the + * software release and the client kept spinning on a completion for a + * kernel that never ran - the worst of both (20 s of backend + * occupancy plus a false completion). Refusing up front fails fast + * into the same software-completion fallback without touching the + * engine, and cannot wedge it for the next client. + */ + if (req->n_data < gsp_phys_gr_min_data_value()) { + gsp_phys_gr_refused++; + pr_err_ratelimited("vgsp-gr: GR submit refused: degenerate data set n_data=%u (min %u) launch_va=0x%llx\n", + req->n_data, + gsp_phys_gr_min_data_value(), + req->push_va); + return -ENODATA; + } if (!gsp_phys_backend_gr_ready() || !gsp_phys_guest_mem_enabled() || gsp_phys_dma_domain_identity() != 1 || !gsp_phys_guest_ram_active()) { gsp_phys_gr_refused++; @@ -17219,6 +24398,41 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, ret = -EIO; goto out; } + /* + * SESSION 153-CONTINUED-60: all-zero 3D-window skip. The client's push + * page is cyclically zeroed/rewritten (CONT-45); a capture that lands + * on the zero phase still forwards an all-zero method window. Live + * r1107: such a window launched (3D raw dump words=252, every word 0), + * the engine trapped SET_VERTEX_ATTRIBUTE_A(2) (trap=00001168 + * data_lo=0x50) and escalated to a priv-error poison (grfifo= + * 0xbadf1201, mme_pc=513) that wedged the GR lane for the rest of the + * boot; r1031 had already seen the same class fault the PBDMA on the + * zero lead (PB-FAULT-DUMP P0). A zero window carries no method to + * execute and nothing to store: skip the hardware handoff entirely - + * no shadow relocation, no engine contact, no wedge - and return the + * distinct rc -EBADMSG so the service authors the software release and + * suppresses the synthetic fault (a frame with no content deserves no + * client-visible engine error). + */ + if (stream3d) { + u32 zwi; + bool zerowin = req->stream_n != 0; + + for (zwi = 0; zwi < req->stream_n; zwi++) { + if (req->stream[zwi]) { + zerowin = false; + break; + } + } + if (zerowin) { + gsp_phys_gr_zerowin_skips++; + pr_warn_ratelimited("vgsp-gr: ZEROWIN-SKIP h=0x%x push=0x%llx window=%u all-zero (no engine submit; software release) n=%llu\n", + req->h_object, req->push_va, req->stream_n, + gsp_phys_gr_zerowin_skips); + ret = -EBADMSG; + goto out; + } + } gsp_phys_gr_submitted++; gsp_phys_gr_override_apply_param(); @@ -17255,6 +24469,77 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; memcpy(qmd, req->qmd, sizeof(qmd)); /* + * M91 icache coherence (gsp_phys_gr_icache_inv, default 1): the + * window-0 shadow pool reuses code VAs across launches (two 8-page + * banks alternate per submit epoch), so the bytes at a program VA + * change between launches, while every captured guest QMD carries + * the QMDV02_03 cache-control block with INVALIDATE_INSTRUCTION_CACHE + * (word 5 bit 30) clear - the same field the CUDA driver arms only + * under its forceIcacheInvalidate knob. An SM instruction-fetch line + * fetched for launch N can then survive into launch N+2 and execute + * the previous kernel's bytes at the same VA, the observed + * "first attempt faults, identical retry succeeds" class. Set the + * bit only when the QMD carries the driver's standard cache-control + * signature (word 5 bits 26..29) so an unknown QMD revision is never + * touched; 0 restores the exact pre-change QMD for a one-flip A/B. + */ + if (gsp_phys_gr_icache_inv_value() && + (qmd[5] & 0x3c000000u) == 0x3c000000u && + !(qmd[5] & (1u << 30))) { + qmd[5] |= 1u << 30; + if (!gsp_phys_m91_icache_log) { + gsp_phys_m91_icache_log = true; + pr_info("vgsp-gr: M91 icache-invalidate bit set in the forwarded QMD (w5=0x%08x; QMDV02_03 word5 bit30, the guest driver's forceIcacheInvalidate field)\n", + qmd[5]); + } + } + /* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default + * 0 = off): the same cache-control word's bit 31 (the V01_06-layout + * SHADER_CONSTANT_CACHE position) is never set by the captured guest + * QMDs. The forwarded param page is rewritten in place between launches + * (the two-bank win0 pool), so a stale constant-cache line can serve + * the PREVIOUS launch's parameters and the current launch's output + * pages stay untouched - the post-dispatch no-store signature. Mode 1 + * sets bit 31; mode 2 forces the whole invalidate set (bits 24..31). + * Only when the QMD carries the driver's standard cache-control + * signature (bits 26..29) so an unknown QMD revision is never touched. + */ + if (gsp_phys_gr_ccache_inv_value() && + (qmd[5] & 0x3c000000u) == 0x3c000000u) { + u32 mode = gsp_phys_gr_ccache_inv_value(); + + if (mode == 2) + qmd[5] |= 0xff000000u; + else + qmd[5] |= 1u << 31; + if (!gsp_phys_r449_ccache_log) { + gsp_phys_r449_ccache_log = true; + pr_info("vgsp-gr: r449 constant-cache invalidate mode=%u applied (w5=0x%08x)\n", + mode, qmd[5]); + } + } + /* + * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar): force the + * forwarded QMD's RELEASE_MEMBAR_TYPE to FE_SYSMEMBAR (QMDV00_06 + * word 11 bit 14) so the front-end flushes its pending system-memory + * stores before writing the release semaphore. The captured guest + * QMDs carry FE_NONE, so a grid that completes with its data still + * in the GPU's L2 releases the client while no store is host-visible + * - the post-dispatch store-loss signature. The release ordering then + * guarantees the host's retire scan and copy-out can never observe a + * completion whose data has not landed. 0 (default) keeps the + * guest's captured value. + */ + if (gsp_phys_gr_rel_membar_value() && !(qmd[11] & (1u << 14))) { + qmd[11] |= 1u << 14; + if (!gsp_phys_gr_rel_membar_log) { + gsp_phys_gr_rel_membar_log = true; + pr_info("vgsp-gr: r433 RELEASE_MEMBAR_TYPE forced to FE_SYSMEMBAR in the forwarded QMD (w11=0x%08x)\n", + qmd[11]); + } + } + /* * Two-step PCAS (gsp_phys_rmcall_gr_pcas_two_step): set the QMD's * REQUIRE_SCHEDULING_PCAS (w4 bit12) so the copy PCAS does not * schedule and the trailing schedule-only PCAS does. @@ -17268,13 +24553,16 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * a fresh shadow line so the guest observes its own QMD release; else * point release0 at the module semaphore so the host observes engine * completion. A reduction release is never redirected (its arithmetic - * would change). + * would change). r929: skipped entirely for a 3D stream replay - + * there is no QMD, so there is no release0 to point anywhere. */ - if (((qmd[4] >> 10) & 1)) { + if (!stream3d && ((qmd[4] >> 10) & 1)) { u32 red_en = (qmd[24] >> 26) & 1; u32 one_word = (qmd[24] >> 31) & 1; bool rel_mod = gsp_phys_rmcall_gr_rel_mod_value() != 0; + int rel_map_ret = 0; + if (!rel_mod && req->rel_hpa && one_word && !red_en && n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES) { u64 sh_va = 0; @@ -17294,6 +24582,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, 0xffu); qmd[25] = req->rel_payload; } else { + rel_map_ret = ret; ret = 0; } } @@ -17313,6 +24602,92 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, */ qmd_rel_hpa = 0; qmd_rel_off = 0; + /* + * r420b: the ENGINE will not write the guest's QMD + * release word this launch (rel_mod, or no targetable + * guest release page). Tell the caller so it authors + * the word in software on every outcome - otherwise a + * launch without deferred release records is never + * completed on the guest word (the r414 saxpy hang). + */ + qmd_rel_redirected = true; + /* + * r633 diag: name the exact redirect cause. rel_hpa==0 + * means the caller could not resolve the release page + * (capture resolution or PA->HPA); map_ret!=0 means the + * page was resolved but its shadow mapping failed. + */ + { + static u32 qmdrel_redir_log; + + if (qmdrel_redir_log < 32) { + qmdrel_redir_log++; + pr_info("vgsp-gr: qmd-rel redirect: rel_hpa=0x%llx map_ret=%d one_word=%u red_en=%u rel_mod=%d n_shadow=%u pay=%u\n", + req->rel_hpa, rel_map_ret, + one_word, red_en, rel_mod ? 1 : 0, + n_shadow, req->rel_payload); + } + } + } + } + + /* + * GRID-WIT (r226): the engine reads CTA geometry from this verbatim + * QMD copy, and the copy-out witnesses show the corruption of the + * failing clients starts exactly at a CTA-band boundary (rows >= 16 + * replicate one tile). Dump the geometry-relevant words once per + * launch so the next capture proves what raster/thread dimensions the + * engine actually sees and lets them be compared against the client's + * own launch config: w12 = CTA_RASTER_WIDTH (gridDim.x), + * w13.lo = CTA_RASTER_HEIGHT (gridDim.y), w13.hi = DEPTH (z), + * w14/w15 = raster resume W/H, w18.hi = CTA_THREAD_DIMENSION0 + * (blockDim.x), w19.lo/hi = DIM1/DIM2 (blockDim.y/z), w18.lo = + * QMD_VERSION/MAJOR nibbles. w4/w5/w8/w17: PCAS/cache-control, + * prefetch pair low word, shared-memory size. + */ + if (gsp_phys_gr_copywit_enabled()) + pr_info("vgsp-gr: GRID-WIT w4=%08x w5=%08x w8=%08x w12=%08x w13=%08x w14=%08x w15=%08x w17=%08x w18=%08x w19=%08x\n", + qmd[4], qmd[5], qmd[8], qmd[12], qmd[13], qmd[14], + qmd[15], qmd[17], qmd[18], qmd[19]); + + /* + * r234 RDW arming witness: one line per launch with the rec count, + * so a capture missing REC-DUMP lines can never be confused with + * "the witness did not run" - probe-class launches must show + * nrecs>0 here. + */ + if (req->n_data && gsp_phys_gr_rdw_budget < 256u) { + gsp_phys_gr_rdw_budget++; + pr_info("vgsp-gr: RDW nrecs=%u launch_va=0x%llx\n", + req->n_data, req->push_va); + } + + /* + * REC-DUMP (r229): the probe-class launches (small data sets) get + * their full rec table logged - every (va, hpa, reloc_va) the engine + * will read/write - so the next capture can compare the resolved + * HPAs against the buffer the client actually uploaded (distinct + * VAs collapsing onto one HPA, or HPAs in a foreign range, are + * visible directly). Big launches log the first/last 8 recs only. + * r233: plain pr_info with a boot-wide line budget - the earlier + * pr_info_ratelimited version was suppressed wholesale during the + * probe's post-stall logging burst, so the probe-class recs never + * reached the ring. + */ + if (gsp_phys_gr_copywit_enabled() && req->n_data && + gsp_phys_gr_recdump_budget < 512u) { + u32 d; + + for (d = 0; d < req->n_data; d++) { + if (req->n_data > 64u && d >= 8u && + d < req->n_data - 8u) + continue; + if (gsp_phys_gr_recdump_budget >= 512u) + break; + gsp_phys_gr_recdump_budget++; + pr_info("vgsp-gr: REC-DUMP[%u/%u] va=0x%llx hpa=0x%llx reloc=0x%llx\n", + d, req->n_data, req->data[d].va, + req->data[d].hpa, req->data[d].reloc_va); } } @@ -17350,12 +24725,30 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, pg_va = pg->guest_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); pr_info("vgsp-gr: page[%u] va=0x%llx hpa=0x%llx qmd_word=%u\n", i, pg->guest_va, pg->hpa, pg->qmd_word); - if (pg->hpa) { + /* + * SESSION 149 R15 (M2-S1): with the window-0 shadow path + * enabled, force the program page (QMD word 48) OFF the + * post-commit Tier-1 install. r801b proved that install is + * ineffective for a late GR page (the engine kept reading + * the empty guest page: CODEHASH h0=0, FAULT-CAP + * code_src=guest-hpa); skipping it lets the page fall to + * the win0 shadow copy + w48/49 rewrite line, which is the + * proven one. Every other page keeps its Tier-1 route. + */ + if (pg->hpa && !(pg->qmd_word == 48u && + gsp_phys_rmcall_gr_shadow_win0_value())) { ret = gsp_phys_rmcall_gr_map_va_locked(bar0, pg_va, pg->hpa, epoch); - if (!ret) + if (!ret) { mapped = true; - else if (ret == -ENOSPC) + /* + * CODEHASH: on the Tier-1 identity path the + * engine reads the guest page itself; sample + * the entry before the QMD pair is authored. + */ + if (pg->qmd_word == 48u) + gsp_phys_gr_codehash(req, pg, NULL); + } else if (ret == -ENOSPC) gsp_phys_gr_pool_exhausted++; } /* @@ -17450,12 +24843,66 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, pg->guest_va, sh_va, slot); } } + /* + * SESSION 153-CONTINUED-89: inline records that land on a + * qmd_word-NONE page. The record-apply loop below belongs to + * the QMD-word shadow-*copy* path and never runs for these + * pages; they are resolved to the guest HPA directly (Tier-1 + * identity, contiguous shadow-line extension, or the alias + * above), so authoring the record into pg->hpa is exactly + * what the engine will read. The concrete case: the real + * captured upload at 0x4350f00 (1408 B) crosses its 4 KB + * page - the head (256 B) lands on the program page 0x4350000 + * and applies, but the tail page 0x4351000 is qmd_word=NONE + * and was silently dropped, leaving the draw's winning j=1 + * program (the later bind) truncated at 256 B at the engine + * (the DS rejects the SASS it cannot fetch: SPH 9/11). + */ + if (mapped && pg->hpa && + pg->qmd_word == GSP_PHYS_GR_QMD_WORD_NONE) { + u32 jx; + + for (jx = 0; jx < req->n_inl; jx++) { + const struct gsp_phys_gr_inline *inl = + &req->inl[jx]; + u32 len; + + if ((inl->dst_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != pg_va) + continue; + len = inl->len; + if (len > GSP_PHYS_GR_INLINE_BYTES) + len = GSP_PHYS_GR_INLINE_BYTES; + if ((u64)(inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1)) + len > + GSP_PHYS_PAGE_SIZE) + continue; + memcpy((u8 *)__va(pg->hpa) + + (inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1)), + inl->data, len); + { + static u32 inl_alias_log; + + if (inl_alias_log < 12u) { + inl_alias_log++; + pr_info("vgsp-gr: INLINE-APPLY-ALIAS dst=0x%llx len=%u pa=0x%llx first=%08x (qmd_word=NONE page authored)\n", + inl->dst_va, len, + pg->hpa, + len >= 4 ? + get_unaligned_le32(inl->data) : + 0u); + } + } + } + } if (!mapped && pg->qmd_word != GSP_PHYS_GR_QMD_WORD_NONE && pg->qmd_word + 1 < GSP_PHYS_GR_QMD_WORDS) { struct gsp_phys_boot_chunk *sh = &boot_stage.rmcall_gr_shadow; u8 *sh_page; u32 j, copied = 0; + bool code_keep_guest = false; if (n_shadow >= GSP_PHYS_RMCALL_GR_SHADOW_PAGES) { gsp_phys_gr_pool_exhausted++; @@ -17481,8 +24928,32 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * domain, copy the full guest page into the shadow and * rewrite the QMD pointer pair, so no inline coverage * is needed for pages the engine only reads. + * + * CODESRC: the program page (qmd_word 48) is the page + * the engine's I-fetch reads; screen its source + * before publishing it. A refused source is either + * replaced by the alternative resolver's page or the + * pair is left at the guest VA (no copy, no rewrite, + * no mapping - the page is fully withdrawn from the + * submission). */ - if (pg->hpa) { + if (pg->hpa && pg->qmd_word == 48u) { + u64 alt_hpa = 0; + int cs = gsp_phys_gr_code_src(req, pg, &alt_hpa); + + if (cs == GSP_PHYS_GR_CODESRC_GUEST_VA) { + code_keep_guest = true; + mapped = true; + } else { + const u8 *csrc = __va(cs == + GSP_PHYS_GR_CODESRC_ALT ? + alt_hpa : pg->hpa); + + memcpy(sh_page, csrc, + GSP_PHYS_PAGE_SIZE); + copied++; + } + } else if (pg->hpa) { memcpy(sh_page, (u8 *)__va(pg->hpa), GSP_PHYS_PAGE_SIZE); copied++; @@ -17500,10 +24971,11 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, } if (pg->hpa && pg->qmd_word >= 32 && pg->qmd_word <= 47 && !(pg->qmd_word & 1)) - gsp_phys_gr_rewrite_data_vars((u32 *)sh_page, + (void)gsp_phys_gr_rewrite_data_vars( + (u32 *)sh_page, GSP_PHYS_PAGE_SIZE / 4, req, (pg->qmd_word - 32) / 2, - (u32 *)__va(pg->hpa)); + (u32 *)__va(pg->hpa), NULL, false); if (pg->hpa && pg->qmd_word == 40u && gsp_phys_rmcall_gr_mail_patch_value()) { u32 *w = (u32 *)sh_page; @@ -17539,7 +25011,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * data-access instruction and the guest VA arguments it * fetches can be read from the log. */ - if (pg->hpa && pg->qmd_word == 48u) { + if (pg->hpa && pg->qmd_word == 48u && !code_keep_guest) { u32 po = (qmd[48] & (GSP_PHYS_PAGE_SIZE - 1)) / 4; const u32 *w = (const u32 *)sh_page + po; @@ -17554,6 +25026,12 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, w[27], w[28], w[29], w[30], w[31], w[32], w[33], w[34], w[35], w[36], w[37], w[38], w[39]); + /* + * CODEHASH: the engine-visible code copy (sh_page) + * is authored here - sample source + shadow + * before the QMD pair points at this page. + */ + gsp_phys_gr_codehash(req, pg, sh_page); } if (pg->hpa && pg->qmd_word == 32u) { const u32 *w = (const u32 *)sh_page; @@ -17599,6 +25077,74 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, memcpy(sh_page + (inl->dst_va & (GSP_PHYS_PAGE_SIZE - 1)), inl->data, len); + /* + * SESSION 153-CONTINUED-10: a stream3d (3D + * method-window) submission maps the engine's + * VAs straight at the GUEST page HPA + * (gsp_phys_rmcall_gr_map_shadow_locked + * installs the guest PTE - there is no + * shadow-chunk copy in that path), so the + * engine's I-fetch reads the guest page, not + * sh_page. The GLX program page is never + * written by the guest itself (the driver's + * uploads never ran), and the r945 evidence is + * exactly that: "Graphics Exception: Shader + * Program Header 11/18 Error" on the draw + * whose inline prog-hex blob had only reached + * this shadow copy. Author the inline bytes + * into the guest page too, emulating the + * upload that never executed. + */ + if (stream3d && pg->hpa) + memcpy((u8 *)__va(pg->hpa) + + (inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1)), + inl->data, len); + /* + * SESSION 153-CONTINUED-11: bounded witness of + * every inline record applied (which program + * page received which image), so the SPH + * delivery is auditable from the log. + */ + { + static u32 inl_apply_log; + + if (inl_apply_log < 24) { + inl_apply_log++; + pr_info("vgsp-gr: INLINE-APPLY dst=0x%llx len=%u pg_va=0x%llx guest=%u first=%08x %08x\n", + inl->dst_va, len, pg_va, + stream3d && pg->hpa ? + 1u : 0u, + len >= 4 ? get_unaligned_le32(inl->data) : 0u, + len >= 8 ? get_unaligned_le32(inl->data + 4) : 0u); + } + } + /* + * SESSION 153-CONTINUED-87: readback witness. + * For a stream3d window the engine's I-fetch + * walks the GUEST page (the inline bytes were + * just authored there above); read the exact + * engine-visible location back so the fetch + * content is provable from the log - if the DS + * still rejects the header, the bytes are known + * present at the VA it fetched. + */ + if (stream3d && pg->hpa) { + static u32 inl_rb_log; + const u32 *gw = (const u32 *) + ((u8 *)__va(pg->hpa) + + (inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1))); + + if (inl_rb_log < 12u) { + inl_rb_log++; + pr_info("vgsp-gr: INLINE-RB dst=0x%llx pa=0x%llx rb=%08x %08x %08x %08x %08x %08x %08x %08x\n", + inl->dst_va, pg->hpa, + gw[0], gw[1], gw[2], + gw[3], gw[4], gw[5], + gw[6], gw[7]); + } + } copied++; } /* @@ -17614,11 +25160,31 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * relocated values no longer match a guest data VA. */ if (pg->hpa && pg->qmd_word >= 32 && pg->qmd_word <= 47 && - !(pg->qmd_word & 1)) - gsp_phys_gr_rewrite_data_vars((u32 *)sh_page, + !(pg->qmd_word & 1)) { + u64 gf = 0; + u32 gg; + + /* + * GRFIX FIX3: this is the final authoring + * of the engine-visible page (after the + * inline replay), so its gap count is the + * launch's relocation-coverage measure. + * Log it (rate-limited) and accumulate it + * for the strict hold below. + */ + gg = gsp_phys_gr_rewrite_data_vars( + (u32 *)sh_page, GSP_PHYS_PAGE_SIZE / 4, req, (pg->qmd_word - 32) / 2, - (u32 *)__va(pg->hpa)); + (u32 *)__va(pg->hpa), &gf, true); + if (gg) { + reloc_gap += gg; + if (!reloc_gap_first) + reloc_gap_first = gf; + pr_info_ratelimited("vgsp-gr: RELOC-GAP words=%u first=0x%llx\n", + gg, gf); + } + } if (copied) { u32 sh_line = boot_stage.rmcall_guest_pt_next_line; @@ -17802,6 +25368,40 @@ gr_page_unmapped: } /* + * GRFIX FIX3 strict hold: with gsp_phys_gr_strict_reloc=1, a launch + * whose rewritten pages carry device-VA-shaped words that match no + * tracked data page is refused before publish - the word stays what + * it was (an unrelocated guest VA), but the engine never sees the + * QMD, so it can never walk an untranslatable pointer. The refusal + * rides the existing backend-failure path (caller bridge fallback + + * the sema service's recorded-release hold; bounded, never a hang). + * The hold is opt-in because the shape heuristic can false-positive + * on non-pointer data words; the RELOC-GAP line (emitted with the + * parameter off as well) is the false-positive evidence. + */ + if (reloc_gap && gsp_phys_gr_strict_reloc_param) { + mutex_unlock(&gsp_phys_rm_lock); + gsp_phys_gr_refused++; + pr_err("vgsp-gr: RELOC-GAP-STRICT words=%u first=0x%llx; launch held fail-closed (unrelocated device-VA words never reach the engine)\n", + reloc_gap, reloc_gap_first); + ret = -EAGAIN; + goto out; + } + /* + * COPYWIT fail4: one per-launch census of the rewrite outcome so the + * resolved/unresolved split of every bank pass is provable from a + * single line (gaps left as guest VAs vs late-resolved words authored + * via RELOC-FIXED). Rate-limited like the per-word lines. + */ + if (gsp_phys_gr_copywit_enabled() && + (reloc_gap || gsp_phys_gr_reloc_fixed)) + pr_info_ratelimited("vgsp-gr: RELOC-CENSUS gaps=%u first_gap=0x%llx fixed=%llu first_fixed=0x%llx late_pages=%u\n", + reloc_gap, reloc_gap_first, + gsp_phys_gr_reloc_fixed, + gsp_phys_gr_reloc_fixed_first, + gsp_phys_gr_gap_n); + + /* * Rank-2 qmdwin (gsp_phys_rmcall_gr_qmdwin=1): relocate the inline-QMD * destination to the dedicated module page planted in the ladder's * window-0 small-PT line (VA 0x107000). The PTE was written by @@ -17811,14 +25411,15 @@ gr_page_unmapped: * GPFIFO/PB/REPORT_SEM legs already use. The 2 MiB window is installed * by the ladder itself, so no window ensure is sent for it (a * pdeIndex-0 Tier-1 install would touch the ladder's own PDE). + * r929: a 3D stream replay has no inline QMD at all (skipped). */ - if (gsp_phys_rmcall_gr_qmdwin_value()) { + if (!stream3d && gsp_phys_rmcall_gr_qmdwin_value()) { u8 *qw = boot_stage.rmcall_gr_qmdwin.pages ? gsp_phys_chunk_page_va(&boot_stage.rmcall_gr_qmdwin, 0) : NULL; if (boot_stage.rmcall_gr_qmdwin_va && qw) { - u32 qw_bank = (epoch & 1u) % + qw_bank = (epoch & 1u) % GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS; qmdwin_on = true; @@ -17839,9 +25440,9 @@ gr_page_unmapped: * use it as the destination when the caller resolved it; a Tier-1 * install keeps the guest VA (the guest sees its own QMD), a Tier-2 * alias keeps the launch runnable, and an unresolved page keeps the - * module scratch VA. + * module scratch VA. r929: skipped for a 3D stream replay. */ - if (!qmdwin_on && req->qmd_hpa) { + if (!stream3d && !qmdwin_on && req->qmd_hpa) { u64 qmd_page = req->qmd_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); u32 qmd_off = (u32)(req->qmd_va & (GSP_PHYS_PAGE_SIZE - 1)); bool dst_ok = false; @@ -17919,6 +25520,27 @@ gr_page_unmapped: /* A stale release from a previous submission must not be observed. */ WRITE_ONCE(*(u32 *)sem_va, 0); WRITE_ONCE(*(u32 *)(sem_va + 4), 0); + /* + * r441: zero the GUEST-VISIBLE release words for the first attempt + * too, not only for outcome retries. The r439 evidence shows a first + * attempt whose poll broke on a STALE value (qmd_rel=0x00000005 from + * the previous launch's engine write, want=6) while the engine never + * fetched this launch's deposit (sentinel intact, "engine fetch + * NONE"): the host then shipped a completion for a launch that did + * not run. Zeroing forces the poll to wait for THIS launch's own + * engine write; when the engine does not write it, the timeout ladder + * and the software-authored completion take over (the honest path), + * and the wedge classification stops being satisfiable by a stale + * value. The write guard re-validates the target like the retry path. + */ + if (qmd_rel_hpa && qmd_rel_off < GSP_PHYS_PAGE_SIZE && + gsp_sema_gr_write_guard(req->rel_va, qmd_rel_hpa, + "gr-rel-zero-first")) + WRITE_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + qmd_rel_off), 0); + if (ch_rel_hpa && ch_rel_off < GSP_PHYS_PAGE_SIZE && + gsp_sema_gr_write_guard(req->rel_va, ch_rel_hpa, + "gr-chrel-zero-first")) + WRITE_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + ch_rel_off), 0); wmb(); /* @@ -17938,13 +25560,603 @@ gr_page_unmapped: GSP_PHYS_GR_QMD_SENTINEL); } + /* + * r929 v3 (Path A): relocate the 3D window's address operands to + * shadow-pool VAs the engine walkers can reach (see + * gsp_phys_rmcall_gr_3d_patch_window; the r929 v2 FE faulted + * FAULT_PDE on a Tier-1-mapped guest VA). Runs with the rm lock + * held, before the method block is built. + */ + if (stream3d) { + u32 win_n = req->stream_n; + u32 ndrop = 0; + u32 nrel; + + /* + * SESSION 153-CONTINUED-27: a big colour target (the X + * screen) is aliased as ONE contiguous shadow run BEFORE + * the relocation pass runs, so the pass rewrites its + * operands to the alias (see the table above). + */ + gsp_phys_gr_surface_alias_scan_locked(req->stream, + req->stream_n, req); + /* SESSION 153-CONTINUED-59: fresh miss census for this pass + * (window + attr patch below both feed it). */ + gsp_phys_rmcall_gr_3d_miss_n = 0; + nrel = gsp_phys_rmcall_gr_3d_patch_window( + req->stream, req->stream_n, stream_patch, &win_n, + &ndrop, req, &n_shadow, bar0, epoch); + + win3d = stream_patch; + win3d_n = win_n; + /* + * SESSION 153-CONTINUED-69: drop the CALL_MME_MACRO/DATA runs + * of stubbed slots (see gsp_phys_gr_mme_drop_calls). With the + * stub as a pure NOP terminator the parameter port no longer + * drains; the r1142 stall parked the PBDMA on the 16-arg + * CALL_MME_MACRO(57) run and the block never completed. The + * drop turns those calls into nothing instead of a call whose + * parameters have nowhere to go. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_call_drop) && + gsp_sema_gr_sa_stub_value()) { + u32 sa_ent[64]; + u64 sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + u64 dmask = 0; + u32 ndrop_c = gsp_phys_gr_mme_drop_calls( + stream_patch, win3d_n, sa_mask, sa_ent, + &dmask); + + if (ndrop_c) { + win3d_n -= ndrop_c; + pr_info("vgsp-gr: MME-CALL-DROP %u word(s) of stubbed-slot CALL runs (window=%u) dropped_mask=0x%llx h=0x%x push=0x%llx\n", + ndrop_c, win3d_n, + (unsigned long long)dmask, + req->h_object, req->push_va); + } + } + pr_info("vgsp-gr: 3D window relocated %u operand pair(s) to shadow VAs (window=%u shadow=%u dropped=%u subdrop=%u sdmdrop=%u progkeep=%u)\n", + nrel, win3d_n, n_shadow, ndrop, + gsp_phys_rmcall_gr_3d_last_subdrop, + gsp_phys_rmcall_gr_3d_last_sdmdrop, + gsp_phys_rmcall_gr_3d_last_progkeep); + /* + * SESSION 153-CONTINUED-87: emit the client's missing + * pipeline-stage program binds (see the sema_gl_sph_bind + * comment). The window carries only the j=1 (VTG) pair; the + * real client state also binds j=2 (TESS_CONTROL at + * family+0x200) and j=5 (PIXEL at family+0x0) - both present + * verbatim in the init-push image scraped from guest RAM + * (`20020825 [0,04400200]`, `20020855 [0,04400000]`). Build + * the two runs and push them through the same window + * relocation pass (miss -> the pair stays raw and the whole + * bind is skipped, never a raw client VA to the engine), + * then prepend them ahead of the window. Beef client + * channels only: the family layout is that client's. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_bind) && + (req->h_object & 0xfffff000u) == 0xbeef0000u && + req->prog_va && + win3d_n + 64u <= GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + static u32 sph_bind_log; + u64 fam = req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u32 in[64], out[64]; + u32 nfin = 0, nrun = 0, on = 0, odrop = 0, orel; + u32 bi; + + /* + * SESSION 153-CONTINUED-89 (2nd half): the client's + * full channel-init pipeline block, harvested from the + * guest-RAM init-push image (guest PA 0x81efc5b4, this + * boot) and identical in the X channel's own captured + * window (methods 0x21c, 0x155c, 0x1574 in that exact + * order). The per-frame window carries none of it - + * on real hardware the client wrote it ONCE at + * channel init, and the replay channel's fresh context + * never receives it, so the DS validates the fetched + * SPHs against unset context state (9/11). Emitted + * verbatim in the client's own order, ahead of the + * stage binds: + * 0x21c <- 0x1001 INVALIDATE_SHADER_CACHES + * (INSTRUCTION|CONSTANT) + * 0x155c <- 1, 0x214fa200, 0 + * TEX_SAMPLER_POOL A/B/C + * 0x1574 <- 1, 0x214fa220, 0x10 + * TEX_HEADER_POOL A/B/C + * 0x310 <- 0x705 SET_SPA_VERSION + * 0x22c <- 0x800 SET_GO_IDLE_TIMEOUT + * 0x1128 <- ffffffff, ffffffff + * DA_OUTPUT_ATTRIBUTE_SKIP_MASK + * ZCULL_BOUNDS(0x196c) is already carried by the + * window itself. All constant runs (no relocation). + */ + in[nfin++] = 0x20000000u | (1u << 16) | + (0x21cu >> 2); + in[nfin++] = 0x1001u; + { + /* + * The pool runs are address-bearing (the patch + * pass's own rules: addr 0x155c/0x1574, cnt >= + * 3) and count toward the relocated-pair + * guard, so emit them only when their page is + * in this launch's page set (r1290: emitted + * unconditionally -> unaccounted pairs -> + * `SPH-BIND skipped (reloc=6/4)` and the whole + * prelude was lost). Both VAs share one page. + */ + bool pool_pg = false; + u32 pk; + + for (pk = 0; pk < req->n_pages; pk++) + if ((req->pages[pk].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == + (0x214fa200u & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) + pool_pg = true; + if (pool_pg) { + in[nfin++] = 0x20000000u | (3u << 16) | + (0x155cu >> 2); + in[nfin++] = 0x1u; + in[nfin++] = 0x214fa200u; + in[nfin++] = 0x0u; + in[nfin++] = 0x20000000u | (3u << 16) | + (0x1574u >> 2); + in[nfin++] = 0x1u; + in[nfin++] = 0x214fa220u; + in[nfin++] = 0x10u; + nrun += 2; + } + } + in[nfin++] = 0x20000000u | (1u << 16) | + (0x310u >> 2); + in[nfin++] = 0x705u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x22cu >> 2); + in[nfin++] = 0x800u; + in[nfin++] = 0x20000000u | (2u << 16) | + (0x1128u >> 2); + in[nfin++] = 0xffffffffu; + in[nfin++] = 0xffffffffu; + + /* + * SESSION 153-CONTINUED-89: the client's init-push + * pipeline state for the window's own stage 1. The + * guest-RAM init-push image (live at guest PA + * 0x81efc5b4, this boot) carries, beside the SPH + * version pair: + * 0x2040 <- 0x11 SET_PIPELINE_SHADER(1) + * ENABLE|TYPE=VERTEX + * 0x2054 <- 1, 0x21201b00 program address pair + * 0x204c <- 0x1a, 0x0 REGISTER_COUNT(1)=26, + * BINDING(1)=0 + * The per-frame window carries only the address pair, + * so the replay channel's TYPE/COUNT/BINDING stage + * registers stay unset - the state the DS validates + * the fetched SPH against (9/11). Emit the TYPE run + * before the binds and the COUNT/BINDING pair after + * them, in the client's own order. Both are constant + * runs (no relocation). + */ + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2040u >> 2); + in[nfin++] = 0x11u; + + /* + * SESSION 153-CONTINUED-87 (second half): the window's + * OWN binds first. The captured per-frame window + * carries draws (BEGIN at word 93) BEFORE its program + * binds (j=1 pair at word 184): on real hardware the + * bind happened in an earlier push, but the replay + * channel's fresh context has nothing bound when the + * early draws validate their stages - the + * Shader-Program-Header poison (r1277+ 9/11). Re-emit + * every bind the window itself carries ahead of it. + */ + for (bi = 0; bi < req->n_prog_binds && bi < 8u; bi++) { + u64 va = req->prog_bind_va[bi]; + u32 sl = req->prog_bind_slot[bi] & 0x1fu; + + if (!va || nfin + 3u > 64u) + continue; + in[nfin++] = 0x20000000u | (2u << 16) | + ((0x2014u + sl * 0x40u) >> 2); + in[nfin++] = (u32)(va >> 32) & 0xffu; + in[nfin++] = (u32)va & 0xfffffffcu; + nrun++; + } + /* + * Then the client's missing stage binds (see the + * sema_gl_sph_bind comment): TESS_CONTROL (j=2 at + * family+0x200) and PIXEL (j=5 at family+0x0), both + * present verbatim in the init-push image scraped + * from guest RAM (`20020825 [0,04400200]`, + * `20020855 [0,04400000]`). + */ + if (nfin + 13u <= 64u) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2094u >> 2); + in[nfin++] = (u32)((fam + 0x200u) >> 32) & 0xffu; + in[nfin++] = (u32)(fam + 0x200u) & 0xfffffffcu; + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2154u >> 2); + in[nfin++] = (u32)(fam >> 32) & 0xffu; + in[nfin++] = (u32)fam & 0xfffffffcu; + /* + * SESSION 153-CONTINUED-88: the per-slot + * SET_PIPELINE_SHADER(j) TYPE state. The + * client's init push writes it once per + * channel (init-push image: `20010810 + * 00000011` = SET_PIPELINE_SHADER(1) <- + * ENABLE|TYPE=VTG); the captured per-frame + * windows carry only the program-address + * pairs, so the replay context's stage + * registers for the synthesized binds would + * stay unset. Emit them for the two + * synthesized slots with the known types + * (j=2 TESS_CONTROL -> 2, j=5 PIXEL -> 5). + */ + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2080u >> 2); + in[nfin++] = 1u | (2u << 4); + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2140u >> 2); + in[nfin++] = 1u | (5u << 4); + /* + * SESSION 153-CONTINUED-89: the client's + * init-push stage-1 REGISTER_COUNT/BINDING + * pair (`20020813 0000001a 00000000` in the + * guest-RAM init image): REGISTER_COUNT(1) = + * 26 (the window SPH's reg count), BINDING(1) + * = group 0. Emitted after the address pair, + * exactly as the client's own push orders it. + */ + in[nfin++] = 0x20000000u | (2u << 16) | + (0x204cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x0u; + /* + * nrun counts only the two ADDRESS-bearing + * binds above: the guard below matches it + * against the relocation pass's + * relocated-pair count, and the constant + * type runs can never be relocated. + */ + nrun += 2; + } + orel = gsp_phys_rmcall_gr_3d_patch_window( + in, nfin, out, &on, &odrop, req, &n_shadow, + bar0, epoch); + if (orel == nrun && on == nfin) { + memmove(&stream_patch[nfin], stream_patch, + (size_t)win3d_n * 4u); + memcpy(stream_patch, out, (size_t)nfin * 4u); + win3d_n += nfin; + if (sph_bind_log < 16u) { + sph_bind_log++; + pr_info("vgsp-gr: SPH-BIND emitted fam=0x%llx win-binds=%u j2=0x%llx j5=0x%llx j1-type=0x11 rc=0x1a initblk=17 win=%u (pre=%u) h=0x%x push=0x%llx\n", + fam, req->n_prog_binds, + fam + 0x200u, fam, + win3d_n, nfin, + req->h_object, + req->push_va); + } + } else if (sph_bind_log < 16u) { + sph_bind_log++; + pr_info("vgsp-gr: SPH-BIND skipped (reloc=%u/%u kept=%u/%u fam=0x%llx h=0x%x push=0x%llx)\n", + orel, nrun, on, nfin, fam, + req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-62: surface (render-target) prelude - + * see the gsp_phys_gr_surf_pre comment. Injected only when the + * window carries no usable colour-target bind of its own; the + * alias scan + relocation run exactly as they would for a + * window-carried bind (the captured VA is the surface the + * machine's draws target, X screen observed). The miss census + * feeds the MISS-ABORT below, so an unresolvable surface pair + * holds the launch (fail-closed) instead of faulting. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_surf_prelude) && + !gsp_phys_gr_window_has_surface(req->stream, + req->stream_n)) { + const u32 *sp; + u32 spn = 0, spg = 0; + + sp = gsp_sema_surf_prelude_get(&spn, &spg); + if (sp && spn && spg) { + if (spg != READ_ONCE(gsp_phys_gr_surf_pre_gen)) { + u32 sdrop = 0, sn = spn; + u32 srel; + + if (sn > GSP_PHYS_GR_SURF_PRE_WORDS) + sn = GSP_PHYS_GR_SURF_PRE_WORDS; + memcpy(gsp_phys_gr_surf_pre_in, sp, + (size_t)sn * 4u); + gsp_phys_gr_surface_alias_scan_locked( + gsp_phys_gr_surf_pre_in, sn, + req); + srel = gsp_phys_rmcall_gr_3d_patch_window( + gsp_phys_gr_surf_pre_in, sn, + gsp_phys_gr_surf_pre, + &gsp_phys_gr_surf_pre_n, + &sdrop, req, &n_shadow, + bar0, epoch); + WRITE_ONCE(gsp_phys_gr_surf_pre_gen, + spg); + pr_info("vgsp-gr: SURF-PRELUDE patched gen=%u raw=%u kept=%u dropped=%u reloc=%u h=0x%x push=0x%llx\n", + spg, sn, + gsp_phys_gr_surf_pre_n, + sdrop, srel, + req->h_object, + req->push_va); + } + if (gsp_phys_gr_surf_pre_n && + win3d_n + gsp_phys_gr_surf_pre_n + 2u + + 8u <= + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + static u32 surf_emit_log; + + memmove(&stream_patch[gsp_phys_gr_surf_pre_n], + stream_patch, + (size_t)win3d_n * 4u); + memcpy(stream_patch, + gsp_phys_gr_surf_pre, + (size_t)gsp_phys_gr_surf_pre_n * + 4u); + win3d_n += gsp_phys_gr_surf_pre_n; + if (surf_emit_log < 16) { + surf_emit_log++; + pr_info("vgsp-gr: SURF-PRELUDE emitted words=%u window=%u h=0x%x push=0x%llx\n", + gsp_phys_gr_surf_pre_n, + win3d_n, + req->h_object, + req->push_va); + } + } + } + } + /* + * SESSION 153-CONTINUED-57: patch the attr/pipeline prelude for + * this launch (relocation exactly like the window: the + * SET_PIPELINE_PROGRAM_ADDRESS pairs and any other recognised + * operand land in engine-visible shadow VAs; sub!=0 and + * win_drop runs are removed by the same pass). Only done when + * the capture generation moved - the shadow VAs are stable + * per HPA for the boot, so an older patch stays valid while it + * drains across launches. See gsp_phys_gr_attr_pre*. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_attr_prelude)) { + const u32 *ap; + u32 apn = 0, apg = 0; + + ap = gsp_sema_attr_prelude_get(&apn, &apg); + if (ap && apn && apg && + apg != READ_ONCE(gsp_phys_gr_attr_pre_gen)) { + u32 adrop = 0, an = apn; + + if (an > GSP_PHYS_GR_ATTR_PRE_WORDS) + an = GSP_PHYS_GR_ATTR_PRE_WORDS; + gsp_phys_rmcall_gr_3d_patch_window(ap, an, + gsp_phys_gr_attr_pre, + &gsp_phys_gr_attr_pre_n, + &adrop, req, &n_shadow, + bar0, epoch); + if (gsp_phys_gr_attr_pre_n) { + WRITE_ONCE(gsp_phys_gr_attr_pre_gen, + apg); + pr_info("vgsp-gr: ATTR-PRELUDE patched gen=%u raw=%u kept=%u dropped=%u h=0x%x push=0x%llx\n", + apg, an, gsp_phys_gr_attr_pre_n, + adrop, req->h_object, + req->push_va); + } + } + } + if (ndrop) { + static u32 gr3d_drop_log; + + if (gr3d_drop_log < 32) { + gr3d_drop_log++; + pr_info("vgsp-gr: 3D win-drop h=0x%x push=0x%llx raw=%u kept=%u dropped=%u list=[%s]\n", + req->h_object, req->push_va, + req->stream_n, win3d_n, ndrop, + gsp_phys_rmcall_gr_win_drop ? + gsp_phys_rmcall_gr_win_drop : ""); + } + } + /* + * SESSION 153-CONTINUED-59: pre-doorbell MISS abort (see the + * gsp_phys_rmcall_gr_miss_abort comment). A window/attr + * operand whose relocation stayed a raw client VA after the + * level-2 resolver would make the physical engine read an + * unmapped page (r1104 MMU Fault HUBCLIENT_PD @0x4350000 -> + * guest channel reset; r1105 index-buffer misses). Ride the + * same failure path as RELOC-GAP-STRICT: the launch is not + * submitted, the caller falls back and authors the software + * release, the page is already in the pin demand set, and + * the next forward retries. Bounded: after 3000 aborts the + * check fails open so a page the resolver can never see + * cannot livelock the lane forever. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_miss_abort) && + gsp_phys_rmcall_gr_3d_miss_n && + gsp_phys_gr_miss_abort_n < 3000u) { + static u32 miss_abort_log; + + gsp_phys_gr_miss_abort_n++; + if (miss_abort_log < 16u || + !(gsp_phys_gr_miss_abort_n % 256u)) { + miss_abort_log++; + pr_info("vgsp-gr: MISS-ABORT h=0x%x push=0x%llx miss=%u n=%llu (operand page unresolvable; demanded via pin path, launch not submitted - retry next forward)\n", + req->h_object, req->push_va, + gsp_phys_rmcall_gr_3d_miss_n, + gsp_phys_gr_miss_abort_n); + } + mutex_unlock(&gsp_phys_rm_lock); + gsp_phys_gr_refused++; + /* + * SESSION 153-CONTINUED-61: the dedicated verdict lets + * the sema service keep the captured candidate armed + * and re-forward it (re-arm) instead of consuming it + * with a software completion: the demanded pages + * resolve through the pin path and the retry then runs + * the real launch. + */ + ret = GSP_PHYS_GR_RC_MISSABORT; + goto out; + } + /* + * SESSION 153-CONTINUED-56: bounded RAW (pre-drop, pre-reloc) + * window dump so the emitted window can be diffed against the + * client's authentic bytes. r1091 faulted at method 0x1168 + * (SET_VERTEX_ATTRIBUTE_A(2)) with Data 00000000 in the emitted + * stream; this copy tells whether that zero is the client's own + * value or an emission artifact (drop/reloc/align). + */ + { + static u32 gr3d_raw_log; + u32 wi; + + if (gr3d_raw_log < 8) { + gr3d_raw_log++; + pr_info("vgsp-gr: 3D raw dump h=0x%x push=0x%llx words=%u\n", + req->h_object, req->push_va, + req->stream_n); + for (wi = 0; + wi + 16 <= req->stream_n && wi < 448; + wi += 16) + pr_info("vgsp-gr: 3D raw[%3u]: %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + wi, + req->stream[wi + 0], + req->stream[wi + 1], + req->stream[wi + 2], + req->stream[wi + 3], + req->stream[wi + 4], + req->stream[wi + 5], + req->stream[wi + 6], + req->stream[wi + 7], + req->stream[wi + 8], + req->stream[wi + 9], + req->stream[wi + 10], + req->stream[wi + 11], + req->stream[wi + 12], + req->stream[wi + 13], + req->stream[wi + 14], + req->stream[wi + 15]); + } + } + /* + * SESSION 153-CONTINUED-11: bounded raw window dump - the + * exact C597 method stream the engine walks (program binds + * 0x2014/0x2018+j*0x40, SET_PIPELINE_SHADER 0x2000+j*0x40, + * draws 0x1618/0xd74). Two windows per boot. + */ + { + static u32 gr3d_win_log; + u32 wi; + + if (gr3d_win_log < 48) { + gr3d_win_log++; + pr_info("vgsp-gr: 3D window dump h=0x%x push=0x%llx words=%u\n", + req->h_object, req->push_va, win3d_n); + for (wi = 0; wi < win3d_n && wi < 384; + wi += 16) + pr_info("vgsp-gr: 3D win[%3u]: %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + wi, + win3d[wi + 0], win3d[wi + 1], + win3d[wi + 2], win3d[wi + 3], + win3d[wi + 4], win3d[wi + 5], + win3d[wi + 6], win3d[wi + 7], + win3d[wi + 8], win3d[wi + 9], + win3d[wi + 10], win3d[wi + 11], + win3d[wi + 12], win3d[wi + 13], + win3d[wi + 14], win3d[wi + 15]); + } + } + } + words = gsp_phys_rmcall_build_gr_launch(methods, req, qmd, qmd_dst, - ch_sem_va, want_ch); - memcpy(gsp_phys_chunk_page_va(pb, 0), methods, words * 4); + ch_sem_va, want_ch, + win3d, win3d_n, + gsp_phys_gr_attr_pre, + gsp_phys_gr_attr_pre_n); + gsp_phys_gr_pb_copy(methods, words); + /* SESSION 153-CONTINUED-12: identify this block to a later PBDMA dump. */ + WRITE_ONCE(gsp_phys_gr_last_words, words); + WRITE_ONCE(gsp_phys_gr_last_h, req->h_object); + WRITE_ONCE(gsp_phys_gr_last_push_va, req->push_va); + WRITE_ONCE(gsp_phys_gr_last_pb_dma, + (u64)gsp_phys_chunk_page_dma(pb, 0)); + /* + * SESSION 153-CONTINUED-52: stream validator for the block the engine + * will walk. The r1072 fault class ("Channel ID 4 intr0 00040000" = + * PBDMA PBENTRY, "invalid method encoding used in method", + * fifo_tu102.c) is raised by a method header the PBDMA refuses, or by + * a run whose data is not fully present so the walk desyncs into + * following words. Walk the exact published dwords with the FIFO DMA + * grammar (dev_ram.ref NV_FIFO_DMA_*: SEC_OP 31:29; 1/3/5 carry 1+cnt + * data words, 4/7 and TERT mask ops are single-word, 6 is RESERVED) + * and flag the first occurrences of a reserved encoding or a run + * overrunning the block. Read-only; bounded per boot. + */ + { + static u32 blkval_log; + const u32 *bv = methods; + u32 bo = 0, nbad = 0; + + if (blkval_log < 48) { + while (bo < words && nbad < 4) { + u32 h = bv[bo]; + u32 bop = (h >> 29) & 7u; + u32 bcnt = (h >> 16) & 0xfffu; + u32 bspan = 1u; + + if (bop == 1u || bop == 3u || bop == 5u) + bspan = 1u + bcnt; + if (bop == 6u) { + nbad++; + pr_warn("vgsp-gr: BLOCKVAL reserved-encoding word=%u/%u val=%08x (h=%x push=0x%llx prev=%08x %08x)\n", + bo, words, h, req->h_object, + req->push_va, + bo ? bv[bo - 1] : 0, + bo + 1 < words ? + bv[bo + 1] : 0); + } + if (bo + bspan > words) { + nbad++; + pr_warn("vgsp-gr: BLOCKVAL run-overrun word=%u/%u hdr=%08x span=%u (needs %u more) (h=%x push=0x%llx)\n", + bo, words, h, bspan, + bo + bspan - words, + req->h_object, req->push_va); + } + bo += bspan; + } + blkval_log++; + if (nbad) + pr_warn("vgsp-gr: BLOCKVAL done words=%u bad=%u (h=%x push=0x%llx)\n", + words, nbad, req->h_object, + req->push_va); + } + } pr_info("vgsp-gr: qmd w0..3=%08x %08x %08x %08x w4..7=%08x %08x %08x %08x w23..25=%08x %08x %08x w32/33=%08x %08x w48/49=%08x %08x\n", qmd[0], qmd[1], qmd[2], qmd[3], qmd[4], qmd[5], qmd[6], qmd[7], qmd[23], qmd[24], qmd[25], qmd[32], qmd[33], qmd[48], qmd[49]); + /* + * r430: full 64-word deposit dump for the wedge's healthy-vs-wedged + * QMD diff (hardware-only fields w50/w51 INNER_GET/PUT, w52 span-list, + * w53 next-QMD pointer are not in the partial line above). Gated by + * the skedrec param (default on); one long line per launch. + */ + if (gsp_phys_rmcall_gr_skedrec_value()) { + char qb[64 * 9 + 1]; + int qo = 0; + + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS && qo < (int)sizeof(qb) - 10; i++) + qo += scnprintf(qb + qo, sizeof(qb) - qo, "%08x ", qmd[i]); + pr_info("vgsp-gr: QMD64 %s\n", qb); + } /* m120: dump the main PT page slots 0x100..0x11f so the PTE content * the walker would read (win0 + bounce window) is visible in the log. */ { @@ -17997,24 +26209,131 @@ gr_page_unmapped: struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); u32 dcopied = 0, dfailed = 0, di; - gsp_phys_gr_fill_bounce(req, bounce_csum_in); + /* + * GRFIX FIX2 (copy-in fail-closed): classify every data page + * BEFORE the bounce copy. An entry with no resolved HPA, no + * bounce index or no backing bounce page cannot be freshly + * filled for this launch; publishing it would hand the engine + * a QMD data pointer into a bounce page still holding a + * previous launch's bytes (the multi-page stale-output + * class). Hold on the existing backend-refusal path: the + * caller's bridge fallback and the sema service's recorded- + * release hold apply the software releases - bounded, never + * a hang - and the engine never sees the QMD. + */ for (di = 0; di < req->n_data; di++) { u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); if (bi == GSP_PHYS_GR_DATA_BI_NONE || - !req->data[di].hpa) + !req->data[di].hpa || + !gsp_phys_chunk_page_va(dchunk, bi)) dfailed++; - else if (gsp_phys_chunk_page_va(dchunk, bi)) - dcopied++; } + if (dfailed) { + mutex_unlock(&gsp_phys_rm_lock); + gsp_phys_gr_refused++; + pr_err("vgsp-gr: copy-in hold: %u/%u data pages unresolved at copy-in (no fresh bounce content); launch held fail-closed, no publish\n", + dfailed, req->n_data); + ret = -EAGAIN; + goto out; + } + dcopied = gsp_phys_gr_fill_bounce(req, bounce_csum_in, + bounce_staged_hpa); pr_info("vgsp-gr: bounce copy-in: %u pages, %u skipped (pre-publish)\n", dcopied, dfailed); - for (di = 0; di < req->n_data && di < 3; di++) - pr_info("vgsp-gr: data[%u] va=0x%llx hpa=0x%llx reloc=0x%llx\n", + /* + * r234 A-TRACE pt=P3 (engine-read point): small launches get + * the source-page content of their first recs - the exact + * bytes the copy-in stages and the engine will read. The probe + * A buffer is 16 pages of ints 0..16383 (w0=0 w1=1 w2=2 ...); + * any point in the pipeline where that pattern breaks names + * the bug site. + */ + if (req->n_data <= 1024u && gsp_phys_gr_atrace_budget < 128u) { + u32 d; + + for (d = 0; d < req->n_data && d < 16u; d++) { + const u32 *pp = + (const u32 *)__va(req->data[d].hpa); + + if (!pp || gsp_phys_gr_atrace_budget >= 128u) + break; + gsp_phys_gr_atrace_budget++; + pr_info("vgsp: A-TRACE pt=P3 va=0x%llx hpa=0x%llx w0..15=%08x %08x %08x %08x %08x %08x %08x %08x | %08x %08x %08x %08x %08x %08x %08x %08x\n", + req->data[d].va, req->data[d].hpa, + pp[0], pp[1], pp[2], pp[3], pp[4], + pp[5], pp[6], pp[7], pp[8], pp[9], + pp[10], pp[11], pp[12], pp[13], + pp[14], pp[15]); + } + } + if ((u64)req->n_data << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-bounce-copy-in va=0x%llx len=0x%llx done=0x%llx\n", + req->data[0].va, (u64)req->n_data << 12, + (u64)dcopied << 12); + if (dcopied < req->n_data) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-bounce-copy-in\n", + req->data[0].va, + (u64)req->n_data << 12, + (u64)dcopied << 12); + for (di = 0; di < req->n_data && di < 3; di++) { + u32 w0 = 0, w1 = 0; + + /* + * fail4 definitive witness: the bytes the engine will + * read for the first recs (the rec hpa's content) + * beside the rec identity - names a +2 MB rec-page + * phase directly (client a[0..3] = 00000000 + * 0000003f 0000003f... would appear as exp[2M] + * values if the hpa points 512 pages late). + */ + if (req->data[di].hpa) { + const u32 *p = (const u32 *)__va( + req->data[di].hpa); + + w0 = p[0]; + w1 = p[1]; + } + pr_info("vgsp-gr: data[%u] va=0x%llx hpa=0x%llx reloc=0x%llx first8=%08x %08x\n", di, req->data[di].va, req->data[di].hpa, - req->data[di].reloc_va); + req->data[di].reloc_va, w0, w1); + } } + /* + * relfix: the late-resolved pages copy in exactly like the tracked + * data pages (same m142 pre-doorbell ordering). The word rewrites + * already point the engine at these bounce pages; filling them here + * is what makes those pointers real. Still under gsp_phys_rm_lock, + * like the block above. + */ + if (gsp_phys_gr_gap_n && gsp_phys_gr_data_chunk()->pages) { + u32 gcopied = gsp_phys_gr_fill_gap_bounce(bounce_csum_in, + bounce_staged_hpa); + + pr_info("vgsp-gr: gap copy-in: %u late-resolved pages, %llu word(s) fixed first=0x%llx\n", + gsp_phys_gr_gap_n, gsp_phys_gr_reloc_fixed, + gsp_phys_gr_reloc_fixed_first); + if ((u64)gsp_phys_gr_gap_n << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-gap-copy-in va=0x%llx len=0x%llx done=0x%llx\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gcopied << 12); + if (gcopied < gsp_phys_gr_gap_n) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-gap-copy-in\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gcopied << 12); + } + + /* + * CONT-73: fingerprint the direct-write witness set (relocated + * operand destinations that map guest pages) immediately before the + * doorbell - after every pre-launch writer (copy-in, mirrors) has + * run, so the retire re-scan measures the ENGINE's stores alone. + */ + gsp_phys_gr_wit_capture(req); + ret = gsp_phys_rmcall_publish_locked(bar0, boot_stage.rmcall_va_pb, words, &sub); mutex_unlock(&gsp_phys_rm_lock); @@ -18148,20 +26467,55 @@ gr_page_unmapped: break; } } - pr_info("vgsp-gr: GR LAUNCH submitted h=0x%x push=0x%llx len=%u pages=%u shadow=%u qmd_va=0x%llx slot=%u words=%u prelude=%d qmdvirt=%d token=0x%x; polling qmd_rel=%u ch_rel=0x%08x (<=%u ms)%s\n", + pr_info("vgsp-gr: GR LAUNCH submitted h=0x%x push=0x%llx len=%u pages=%u shadow=%u qmd_va=0x%llx slot=%u words=%u e0=0x%08x e1=0x%08x prelude=%d qmdvirt=%d token=0x%x; polling qmd_rel=%u ch_rel=0x%08x (<=%u ms)%s\n", req->h_object, req->push_va, req->push_len, req->n_pages, n_shadow, req->qmd_va, sub.slot, words, + sub.e0, sub.e1, gsp_phys_rmcall_gr_prelude_value(), gsp_phys_rmcall_gr_qmdvirt_value(), boot_stage.rmcall_token, - qmd_rel, want_ch, GSP_PHYS_RMCALL_GR_POLL_MS, qmdwin_note); + qmd_rel, want_ch, poll_ms, qmdwin_note); + { + /* + * r596 diag: the raw-prelude geometry the verbatim gate tests. + * The verbatim replay needs + * raw && raw_words >= hdr + 67 && + * raw_qmd_end == hdr + 67 && + * INIT_WORDS + hdr + 67 + TAIL_WORDS <= METHOD_WORDS + * Bounded to the first eight launches so the geometry is on + * record without flooding a battery run. + */ + static u32 prelude_geom_log; + + if (gsp_phys_rmcall_gr_prelude_value() && prelude_geom_log < 8) { + prelude_geom_log++; + pr_info("vgsp-gr: prelude geom raw=%u raw_words=%u hdr=%u end=%u want_end=%u v=%d\n", + req->raw ? 1u : 0u, req->raw_words, + req->raw_qmd_hdr, req->raw_qmd_end, + req->raw_qmd_hdr + + GSP_PHYS_RMCALL_GR_QMD_WINDOW_WORDS, + gsp_phys_rmcall_gr_prelude_value()); + } + } if (qmd_win_diag) pr_info("vgsp-gr: qmd dest va=0x%llx pdeIndex=%u installed=%u win_mask=0x%llx ensure_ret=%d\n", qmd_dst, qmd_win_pde, qmd_win_installed, (unsigned long long)boot_stage.rmcall_guest_pt_win_mask, qmd_win_ret); - deadline = jiffies + msecs_to_jiffies(GSP_PHYS_RMCALL_GR_POLL_MS); + /* + * SESSION 153-CONTINUED-69: pre-reset MME fault sample. The engine + * faults within the first tens of ms of the launch; sample the MME + * ESR/INFO/INFO2 + DS ESR + GRFIFO/trap regs at MMIO rate from the + * moment the doorbell rang (in publish) so the faulting values are + * caught before the GSP-RM clears them. + */ + if (stream3d && bar0) + fast_poison = gsp_phys_mme_fast_sample(bar0, "launch", + req->h_object); + + poll_start_us = (u32)ktime_to_us(ktime_get()); + deadline = jiffies + msecs_to_jiffies(poll_ms); for (;;) { /* * m72: SKED transition recorder - sample the CWD/HWW registers @@ -18169,8 +26523,14 @@ gr_page_unmapped: * cancel or dispatch sequence of the launch window is on the * record even when the poll times out. */ - if (gsp_phys_rmcall_gr_skedrec_value() && bar0 && - (++sked_tick & 7u) == 0u) { + if (gsp_phys_rmcall_gr_skedrec_value() && bar0) { + /* + * r430: sample EVERY poll iteration (not every 8th): + * the wedge's release word lands ~7 ms after the + * doorbell, so the old 8-iteration gate saw at most one + * sample of the dispatch window. Changes only. + */ + sked_tick++; sked_sample[0] = ioread32(bar0 + 0x00407060u); sked_sample[1] = ioread32(bar0 + 0x00407064u); sked_sample[2] = ioread32(bar0 + 0x00407068u); @@ -18182,23 +26542,180 @@ gr_page_unmapped: sked_sample[8] = ioread32(bar0 + 0x00409b00u); sked_sample[9] = ioread32(bar0 + 0x00409b04u); sked_sample[10] = ioread32(bar0 + 0x00409400u); + /* + * SESSION 153-CONTINUED-14: FECS/FE ctxsw handshake + * lanes. fe0 = FECS_CTXSW_STATUS_FE_0 (0x409c00), + * fehost = FECS_HOST_INT_STATUS (0x409c18), fecsirq + * = FECS_IRQSTAT (0x409008). If the FECS dies in a + * switch, it should die visibly here first. + */ + sked_sample[11] = ioread32(bar0 + 0x00409c00u); + sked_sample[12] = ioread32(bar0 + 0x00409c18u); + sked_sample[13] = ioread32(bar0 + 0x00409008u); + /* + * SESSION 153-CONTINUED-52: in-flight fault registers. + * The DS shader-program-header error latch (0x405840), + * the MME exception latch (0x404490) with its PC/macro + * address (0x404494) and instruction register + * (0x404498) are cleared by the real RM within ~ms of + * the fault, so the post-timeout reads are all clean. + * Sampling them every poll tick records the faulting + * values at the first change. + */ + sked_sample[14] = ioread32(bar0 + 0x00405840u); + sked_sample[15] = ioread32(bar0 + 0x00404490u); + sked_sample[16] = ioread32(bar0 + 0x00404494u); + sked_sample[17] = ioread32(bar0 + 0x00404498u); + /* + * SESSION 153-CONTINUED-52: the PBDMA host-software + * METHOD latch (NV_PPBDMA_METHOD0 0x4000c0: ADDR + * 13:2, SUBCH 18:16, INCR 0) - the method whose + * delivery raises NV_PPBDMA_INTR_0_METHOD (bit 21). + * The real driver clears the entry when it processes + * the interrupt, so only an in-flight sample can name + * it. + */ + sked_sample[18] = ioread32(bar0 + 0x004000c0u); + sked_sample[19] = ioread32(bar0 + 0x004000c4u); + /* + * SESSION 153-CONTINUED-57: the class-error trapped + * register set (gr_error_gk104.c grGetExceptionChannel): + * NV_PGRAPH_TRAPPED_ADDR 0x40704 (MTHD offset + SUBCH + + * INVALID status bit), TRAPPED_DATA_LOW 0x40708 (the + * class error's "Data" field), TRAPPED_DATA_HIGH + * 0x4070c, and TRAPPED_DATA_MME 0x40710 (PC 11:0 + + * MACRO_ADDRESS 19:12) - the last one names whether the + * offending write came from MME macro execution. The + * real RM clears these within ~ms of the fault; a + * per-tick sample catches the first change. The + * persistent 0x1168/Data=0 class error (r1091/r1098) + * needs exactly this decode. + */ + sked_sample[20] = ioread32(bar0 + 0x00400704u); + sked_sample[21] = ioread32(bar0 + 0x00400708u); + sked_sample[22] = ioread32(bar0 + 0x0040070cu); + sked_sample[23] = ioread32(bar0 + 0x00400710u); + /* + * SESSION 153-CONTINUED-58: the front-end DECODE + * STATE EX registers (NV_PGRAPH_PRI_FE_DECODE_STATE_EX + * 0x40a630..0x40a63c, 4 slots): MME_PC 11:0, + * MME_MACRO_ADDR 21:12, MME_GENERATED bit 31 - the + * front end's own record of the last MME-generated + * method write, which names the macro (address) and + * its pc for the class error even when the trapped + * registers are already back to the 0xbadf priv + * poison. Sampled per tick like the trap set. + */ + sked_sample[24] = ioread32(bar0 + 0x0040a630u); + sked_sample[25] = ioread32(bar0 + 0x0040a634u); + sked_sample[26] = ioread32(bar0 + 0x0040a638u); + sked_sample[27] = ioread32(bar0 + 0x0040a63cu); if (sked_first || memcmp(sked_sample, sked_prev, sizeof(sked_sample))) { - pr_info("vgsp-gr: sked rec +%ums cwd0=%08x cwd1=%08x cwd2=%08x info1=%08x info2=%08x act=%08x esr=%08x grfifo=%08x fecs_cur=%08x fecs_new=%08x ctxsw1=%08x\n", - (sked_tick / 8u) * + pr_info("vgsp-gr: sked rec +%ums cwd0=%08x cwd1=%08x cwd2=%08x info1=%08x info2=%08x act=%08x esr=%08x grfifo=%08x fecs_cur=%08x fecs_new=%08x ctxsw1=%08x fe0=%08x fehost=%08x fecsirq=%08x ds_esr=%08x mme_esr=%08x mme_pc=%u mme_mac=0x%x mme_ir=%08x mth0=%08x mth1=%08x(pbaddr=0x%x sub=%u incr=%u) trap=%08x(data_lo=%08x hi=%08x sub=%u mmegen=%u dhv=%u) mmepc=%u mmemac=0x%x\n", + sked_tick * GSP_PHYS_RMCALL_POLL_MS, sked_sample[0], sked_sample[1], sked_sample[2], sked_sample[3], sked_sample[4], sked_sample[5], sked_sample[6], sked_sample[7], sked_sample[8], sked_sample[9], - sked_sample[10]); + sked_sample[10], sked_sample[11], + sked_sample[12], sked_sample[13], + sked_sample[14], sked_sample[15], + sked_sample[16] & 0xfffu, + (sked_sample[16] >> 12) & 0xffu, + sked_sample[17], + sked_sample[18], sked_sample[19], + (sked_sample[18] >> 2) << 2, + (sked_sample[18] >> 16) & 7u, + sked_sample[18] & 1u, + sked_sample[20], sked_sample[21], + sked_sample[22], + (sked_sample[20] >> 16) & 7u, + (sked_sample[20] >> 20) & 1u, + (sked_sample[20] >> 24) & 1u, + sked_sample[23] & 0xfffu, + (sked_sample[23] >> 12) & 0xffu); + if (sked_sample[24] || sked_sample[25] || + sked_sample[26] || sked_sample[27]) + pr_info("vgsp-gr: sked fed0=%08x fed1=%08x fed2=%08x fed3=%08x (pc0=%u mac0=0x%x gen0=%u)\n", + sked_sample[24], + sked_sample[25], + sked_sample[26], + sked_sample[27], + sked_sample[24] & 0xfffu, + (sked_sample[24] >> 12) & 0x3ffu, + (sked_sample[24] >> 31) & 1u); memcpy(sked_prev, sked_sample, sizeof(sked_sample)); sked_first = false; } } + /* + * SESSION 153-CONTINUED-48: priv-error (0xbadf-tagged) wedge + * recovery - see sema_gr_wedge_recover. The poison latch is + * checked before the completion break and the ladder so a + * launch whose window faulted the GR hands the engine to the + * next launch recovered instead of wedged. One attempt per + * launch (local flag); the hard re-arm is the r445 mechanism, + * bounded by its own RPC and sem-probe timeouts. + */ + if (READ_ONCE(gsp_phys_gr_wedge_recover) && bar0 && + !wedge_recover_done) { + u32 g0 = ioread32(bar0 + 0x00400504u); + + /* + * SESSION 153-CONTINUED-72: either the poison is still + * asserted here, or the MME-FAST spin caught it before + * the FECS cleared it (fast_poison latch). Both mean + * the window priv-errored and the engine context is + * gone; the earlier live-register-only check missed + * the self-cleared case (r1149) and wedged the boot. + */ + if ((g0 & 0xffff0000u) == 0xbadf0000u || fast_poison) { + u32 g1; + + wedge_recover_done = true; + wedge_recoveries++; + pr_warn("vgsp-gr: WEDGE-RECOVER priv-error poison seen (grfifo=0x%08x%s) h=0x%x push=0x%llx - TSG preempt + resched\n", + g0, + fast_poison && + (g0 & 0xffff0000u) != 0xbadf0000u ? + " via MME-FAST latch (self-cleared)" : + "", + req->h_object, req->push_va); + gsp_phys_mme_regs_log(bar0, "wedge-poison"); + mutex_lock(&gsp_phys_rm_lock); + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + g1 = ioread32(bar0 + 0x00400504u); + /* + * SESSION 153-CONTINUED-61: the TSG preempt + + * resched above resets the engine context, so + * the MME program the emitted prelude loaded is + * gone. Clear the emission latch: the next + * launch that can carry the prelude re-preloads + * it (without this every later replay ran its + * macros against an unloaded MME and stalled + * pre-retire - the r1110 mme_pc=513 class). + */ + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, 0); + /* + * SESSION 153-CONTINUED-79: also rewind the + * split-emission drain cursor, else a cleared + * latch with an exhausted cursor would skip + * the owed re-preload. + */ + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + pr_warn("vgsp-gr: WEDGE-RECOVER after re-arm grfifo=0x%08x (%s); recovery #%u this boot; MME prelude latch cleared (context re-preload owed)\n", + g1, + (g1 & 0xffff0000u) == 0xbadf0000u ? + "still poisoned" : "cleared", + wedge_recoveries); + } + } if (qmd_rel) { if (qmd_rel_hpa) w0 = READ_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + @@ -18206,21 +26723,218 @@ gr_page_unmapped: else w0 = READ_ONCE(*(u32 *)sem_va); } + /* + * SESSION 153-CONTINUED-59: PBENTRY first-sighting watch. + * The r1102 fault was only captured via the GSP's 0x1006 + * relay (seconds later, intr0 already cleared by the RM): + * here every 5 ms poll of a 3D-stream launch scans the 12 + * PBDMA instances for NV_PPBDMA_INTR_0 bit 18 (PBENTRY) and + * on the FIRST sighting dumps the live PB_HEADER / PB_COUNT + * / PB_DATA0 / HDR_SHADOW set plus the raw 0x100..0x120 + * block for the faulting instance, before any re-arm or RM + * clear can overwrite them. Bounded (8 lines per boot); + * read-only, no behavior change to the launch. + */ + if (stream3d && bar0) { + static u32 pbentry_log; + u32 pi2; + + for (pi2 = 0; pi2 < 12 && pbentry_log < 8; pi2++) { + u32 b2 = 0x40000u + pi2 * 0x2000u; + u32 i0 = ioread32(bar0 + b2 + 0x108u); + + if (!(i0 & 0x00040000u)) + continue; + pbentry_log++; + pr_warn("vgsp-gr: PBENTRY-FIRST-SIGHTING P%u intr0=%08x gpf=%u pbfetch=0x%llx hdr=%08x cnt=%08x data0=%08x pbhd=%08x h=%x push=0x%llx words=%u tick=%u\n", + pi2, i0, + ioread32(bar0 + b2 + 0x050u), + ((u64)(ioread32(bar0 + b2 + 0x058u) & + 0xffu) << 32) | + (ioread32(bar0 + b2 + 0x054u) & + ~3u), + ioread32(bar0 + b2 + 0x118u), + ioread32(bar0 + b2 + 0x088u), + ioread32(bar0 + b2 + 0x08cu), + ioread32(bar0 + b2 + 0x084u), + req->h_object, req->push_va, words, + sked_tick); + pr_info("vgsp-gr: PBENTRY-FIRST-SIGHTING P%u raw[0x100]=%08x [104]=%08x [10c]=%08x [110]=%08x [114]=%08x [11c]=%08x [120]=%08x\n", + pi2, + ioread32(bar0 + b2 + 0x100u), + ioread32(bar0 + b2 + 0x104u), + ioread32(bar0 + b2 + 0x10cu), + ioread32(bar0 + b2 + 0x110u), + ioread32(bar0 + b2 + 0x114u), + ioread32(bar0 + b2 + 0x11cu), + ioread32(bar0 + b2 + 0x120u)); + gsp_phys_pbdma_fault_dump(0xffffffffu); + break; + } + } + /* + * SESSION 153-CONTINUED-13: early PBDMA fetch probe for a 3D + * stream launch - where is the front end while a DRAW window + * does not retire? One shot per launch at ~150 ms and ~1.2 s + * (5 ms poll), bounded per boot. pbfetch inside the module + * pushbuffer page = the walk/exec is stuck in our block; + * pbfetch past it = the whole stream was decoded and the + * stall is in execution/completion. + */ + if (stream3d && bar0 && + (sked_tick == 30 || sked_tick == 240)) { + static u32 winprobe_log; + u32 pi; + + if (winprobe_log < 64) { + winprobe_log++; + pr_info("vgsp-gr: WINPROBE +%ums push=0x%llx words=%u block_dma=0x%llx grfifo=%08x trapped=%08x trapped_lo=%08x\n", + sked_tick * GSP_PHYS_RMCALL_POLL_MS, + req->push_va, words, sub.pb_dma, + ioread32(bar0 + 0x00400504u), + ioread32(bar0 + 0x00400704u), + ioread32(bar0 + 0x00400708u)); + for (pi = 0; pi < 12; pi++) { + u32 b = 0x40000u + pi * 0x2000u; + u32 intr0 = ioread32(bar0 + b + 0x108u); + u32 gpf = ioread32(bar0 + b + 0x050u); + u32 pbf = ioread32(bar0 + b + 0x054u); + u32 pbfhi = ioread32(bar0 + b + 0x058u); + u32 hdrsh = ioread32(bar0 + b + 0x118u); + u32 wbh = ioread32(bar0 + b + 0x084u); + u32 wbcnt = ioread32(bar0 + b + 0x088u); + u32 wbd0 = ioread32(bar0 + b + 0x08cu); + u64 fetch64 = + ((u64)(pbfhi & 0xffu) << 32) | + (u64)(pbf & ~3u); + s64 woff = -1; + + if (!intr0 && !gpf && !pbf && !hdrsh) + continue; + if (gsp_phys_gr_last_pb_dma && + fetch64 >= gsp_phys_gr_last_pb_dma && + fetch64 < + gsp_phys_gr_last_pb_dma + + (u64)gsp_phys_gr_last_words * + 4u) + woff = (s64)(fetch64 - + gsp_phys_gr_last_pb_dma) / + 4; + pr_info("vgsp-gr: WINPROBE P%u intr0=%08x gp_put=%u pbfetch=0x%llx off=%lld hdr=%08x(%s m=0x%x cnt=%u sub=%u) pbhd=%08x(type=%s) pbcnt=%08x pbd0=%08x\n", + pi, intr0, gpf, fetch64, woff, hdrsh, + gsp_phys_pbdma_type_name(hdrsh >> 29), + (hdrsh & 0x1fffu) << 2, + (hdrsh >> 16) & 0xfffu, + (hdrsh >> 13) & 7u, wbh, + gsp_phys_pbdma_type_name(wbh >> 29), + wbcnt, wbd0); + } + } + } if (ch_rel_hpa) w1 = READ_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + ch_rel_off)); else w1 = READ_ONCE(*(u32 *)(sem_va + 4)); - if ((!qmd_rel || w0 == want_qmd) && w1 == want_ch) + if ((!qmd_rel || w0 == want_qmd) && w1 == want_ch) { + /* F4-a: first poll break + USERD witnesses. */ + gsp_phys_gr_poll_break_log(w0, want_qmd, w1, + poll_start_us, &sub); + /* + * SESSION 153-CONTINUED-72: GR-STATE readback at the + * retire (fed MME-generated-write records + latches). + */ + gsp_phys_gr_state_rb_log(req->h_object); break; + } + /* + * STALLFREE: the whole-launch budget. Checked after the release + * acks (a completed launch always wins) and before any further + * ladder work; on expiry the submit fails out now and the sema + * service's failure edge authors the QMD completion release in + * software, so the guest client is never left waiting on a + * launch this host will not finish. + */ + if (gsp_phys_gr_wait_budget_over(start)) { + gsp_phys_gr_stall_wait_max++; + gsp_phys_gr_stall_log("wait-max", req->push_va, start, + "fail"); + pr_err("vgsp-gr: GR LAUNCH FAILED: wait budget crossed (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x want=0x%08x) h=0x%x push=0x%llx; ladder stopped\n", + w0, want_qmd, w1, want_ch, req->h_object, + req->push_va); + gsp_phys_mme_regs_log(bar0, "wait-max"); + ret = -ETIMEDOUT; + goto out; + } if ((++grpoll_n % 100) == 0) pr_info("vgsp-gr: poll +%ums qmd_rel=0x%08x ch_rel=0x%08x\n", grpoll_n * GSP_PHYS_RMCALL_POLL_MS, w0, w1); if (time_after_eq(jiffies, deadline)) { gsp_phys_gr_timed_out++; pr_err("vgsp-gr: GR LAUNCH FAILED: timeout after %u ms (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x want=0x%08x) h=0x%x push=0x%llx\n", - GSP_PHYS_RMCALL_GR_POLL_MS, w0, want_qmd, w1, + poll_ms, w0, want_qmd, w1, want_ch, req->h_object, req->push_va); + gsp_phys_mme_regs_log(bar0, "timeout"); + /* + * F4-b: one-shot deposit readback at the FIRST timeout + * (the old post-mortem only ran after attempt 3, which + * a recovered launch never reached). Reading the + * qmdwin/shadow destination here proves whether THIS + * attempt's inline-QMD copy landed and what the engine + * would fetch (w63 is the fetch sentinel). Bound: one + * line per launch. + */ + if (!deposit_rb_done) { + deposit_rb_done = true; + gsp_phys_gr_deposit_rb_log(req, qmd_dst_mem); + } + /* + * F2: fast re-arm at the first timeout. The m155 + * extended re-arm (lane-enable replay + GPFIFO/TSG + * schedule + doorbell) plus a FRESH ring entry recovers + * the r198 class in ~11 s only because it runs after two + * full 5 s windows. Do it now and re-poll briefly; if it + * still fails, fall through to the historical ladder + * (the fast entry counts against the same attempt + * budget, so the total number of host publishes and the + * ring wrap exposure stay as before). + */ + if (gsp_phys_gr_retry_fast_value() && !fast_rearmed) { + int fr; + + fast_rearmed = true; + attempts++; + mutex_lock(&gsp_phys_rm_lock); + fr = gsp_phys_rmcall_publish_locked( + bar0, boot_stage.rmcall_va_pb, words, + &sub); + if (!fr) + fr = gsp_phys_rmcall_gr_resched_locked( + bar0); + mutex_unlock(&gsp_phys_rm_lock); + if (!fr) { + gsp_phys_gr_retries++; + poll_ms = + GSP_PHYS_RMCALL_GR_FAST_RETRY_MS; + pr_info("vgsp-gr: fast re-arm after first timeout: extended re-arm + fresh ring entry slot=%u published; re-polling %u ms (attempts %u/%u)\n", + sub.slot, poll_ms, attempts, + GSP_PHYS_RMCALL_GR_ATTEMPTS); + deadline = jiffies + + msecs_to_jiffies(poll_ms); + /* + * The immediate m142 re-arm would + * only duplicate the resched that just + * ran; the ladder below still owns the + * warm-sem gate and the remaining + * fresh entries. + */ + rearmed = true; + grpoll_n = 0; + continue; + } + pr_info("vgsp-gr: fast re-arm failed rc=%d; falling through to the timeout ladder\n", + fr); + } /* * m142: one bounded re-arm before declaring the launch * dead. Re-issue the stage-4 schedule controls and re-ring @@ -18236,11 +26950,11 @@ gr_page_unmapped: rr = gsp_phys_rmcall_gr_resched_locked(bar0); mutex_unlock(&gsp_phys_rm_lock); if (!rr) { + poll_ms = GSP_PHYS_RMCALL_GR_POLL_MS; pr_info("vgsp-gr: rearm after timeout: schedule re-issued + doorbell re-rung; re-polling %u ms\n", - GSP_PHYS_RMCALL_GR_POLL_MS); + poll_ms); deadline = jiffies + - msecs_to_jiffies( - GSP_PHYS_RMCALL_GR_POLL_MS); + msecs_to_jiffies(poll_ms); grpoll_n = 0; continue; } @@ -18248,6 +26962,62 @@ gr_page_unmapped: rr); } /* + * m166: liveness gate before spending the fresh-entry + * retries. The m156 warm sem push (the stage-4 GR sem + * block) is the only proven "engine alive" probe; if + * it times out, the engine is wedged for the boot and + * every further retry is futile - fail fast instead of + * burning ~20 s with the single-inflight slot held + * (r167 memset2d: 8 internal ladder timeouts over 31 s + * for an entry the front-end never fetched). The probe + * itself is bounded by the sem push's own poll. + */ + if (!sem_gated) { + int sp_ret; + + sem_gated = true; + mutex_lock(&gsp_phys_rm_lock); + sp_ret = gsp_phys_rmcall_gr_sem_push_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + /* + * SESSION 153-CONTINUED-93: parked-engine + * recovery (see the sema_gr_park_recover + * knob). A wedged probe on a poison-free, + * trap-free front-end is the r1311-r1313 + * park class: the context is gone and only a + * context reload brings it back. One bounded + * hard re-arm per launch; ALIVE afterwards + * resumes the fresh-entry retries. + */ + if (sp_ret && bar0 && + READ_ONCE(gsp_phys_gr_park_recover)) { + static u32 park_rec_n; + + mutex_lock(&gsp_phys_rm_lock); + sp_ret = + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + park_rec_n++; + if (!sp_ret) { + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + pr_warn("vgsp-gr: PARK-RECOVER #%u warm sem ALIVE after TSG preempt + resched; fresh-entry retries resume push=0x%llx\n", + park_rec_n, req->push_va); + } else { + pr_warn("vgsp-gr: PARK-RECOVER #%u still wedged after hard re-arm (rc=%d) push=0x%llx\n", + park_rec_n, sp_ret, + req->push_va); + } + } + if (sp_ret) { + gsp_phys_gr_sem_wedged++; + pr_info("vgsp-gr: timeout ladder: warm sem WEDGED (rc=%d) - failing fast instead of fresh-entry retries\n", + sp_ret); + break; + } + pr_info("vgsp-gr: timeout ladder: warm sem ALIVE - continuing fresh-entry retries\n"); + } + /* * The re-arm above only replays the schedule verbs; if the * entry was already fetched or the front-end never latched * it, that cannot make the QMD run. Re-publish a FRESH ring @@ -18271,14 +27041,13 @@ gr_page_unmapped: mutex_unlock(&gsp_phys_rm_lock); if (!pr_ret) { gsp_phys_gr_retries++; + poll_ms = GSP_PHYS_RMCALL_GR_RETRY_MS; pr_info("vgsp-gr: retry %u/%u: fresh ring entry slot=%u published + rescheduled; re-polling %u ms\n", attempts, GSP_PHYS_RMCALL_GR_ATTEMPTS, - sub.slot, - GSP_PHYS_RMCALL_GR_RETRY_MS); + sub.slot, poll_ms); deadline = jiffies + - msecs_to_jiffies( - GSP_PHYS_RMCALL_GR_RETRY_MS); + msecs_to_jiffies(poll_ms); rearmed = false; grpoll_n = 0; continue; @@ -18534,6 +27303,16 @@ gr_page_unmapped: ioread32(bar0 + 0x00405b00u), ioread32(bar0 + 0x00400100u), ioread32(bar0 + 0x0040013cu)); + if (bar0) + pr_info("vgsp-gr: FECS readback: CUR=0x%08x NEW=0x%08x ST1=0x%08x FE0=0x%08x HOSTINT=0x%08x IRQSTAT=0x%08x CPUCTL=0x%08x OS=0x%08x\n", + ioread32(bar0 + 0x00409b00u), + ioread32(bar0 + 0x00409b04u), + ioread32(bar0 + 0x00409400u), + ioread32(bar0 + 0x00409c00u), + ioread32(bar0 + 0x00409c18u), + ioread32(bar0 + 0x00409008u), + ioread32(bar0 + 0x00409100u), + ioread32(bar0 + 0x00409080u)); if (bar0) { u32 g; @@ -18571,7 +27350,7 @@ gr_page_unmapped: ret = -ETIMEDOUT; goto out; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } gsp_phys_gr_ok++; @@ -18590,30 +27369,183 @@ gr_page_unmapped: * retirement wait is the only execution witness. On zero writes, * refill the bounce window from the guest inputs, re-publish the * same method block on a fresh ring entry, re-issue the schedule and - * re-poll, then re-wait - bounded by the same attempt budget as the - * timeout path. After the budget the launch proceeds to copy-out - * with whatever the engine produced: a genuinely read-only kernel is - * indistinguishable from a no-execution here and simply pays the - * bounded extra polls (safe to re-execute by construction - a kernel - * that writes nothing has no side effects to duplicate). A + * re-poll, then re-wait. + * + * m179: the budget is now DEDICATED (gsp_phys_gr_nowrite_retries, + * default 3) instead of sharing the timeout ladder's attempt counter - + * a launch that had already spent attempts on the ladder used to reach + * the accept path with no re-execution at all (the r200-r202 + * memcpy_roundtrip / multikernel all-wrong class). The trigger is + * specific: only a launch with n_tracked > 0 tracked data pages can be + * a no-execution, so a launch whose data entries resolve nothing + * legitimately reports zero writes without entering this path. After + * the budget the launch proceeds to copy-out with whatever the engine + * produced, and the accept is logged loudly (pr_warn) and classified + * by the QMD fetch sentinel (CONFIRMED lost vs AMBIGUOUS): a + * genuinely read-only kernel is indistinguishable from a + * no-execution here and simply pays the bounded extra polls (safe to + * re-execute by construction - a kernel that writes nothing has no + * side effects to duplicate). A * wrong-place write detector was tried and removed: the highest-reloc * data entry is often a UVM-internal buffer, not the output, so it * false-fired on correct launches and would re-execute * side-effecting kernels (atomics) up to 3x. */ if (req->n_data && gsp_phys_gr_data_chunk()->pages) { - u32 wrote = gsp_phys_gr_output_retire_wait(req, bounce_csum_in); + u32 n_tracked = 0; + u32 nw_attempts = 0; + bool idem = false; + u32 wrote = gsp_phys_gr_output_retire_wait(req, bounce_csum_in, + &n_tracked); - if (gsp_phys_rmcall_gr_nowrite_retry_value()) { - while (attempts < GSP_PHYS_RMCALL_GR_ATTEMPTS) { - if (wrote > 0) - break; - attempts++; - if (attempts > GSP_PHYS_RMCALL_GR_ATTEMPTS) + /* CONT-73: where did the draw write? bounce count above; + * direct guest stores (relocated alias/shadow operands and + * MME shadow-scratch destinations) here. */ + gsp_phys_gr_wit_report("retire"); + + /* + * r434: post-launch hardware-only QMD dump (w50/w51 INNER + * get/put = the SKED's CTA dispatch counters) so the + * healthy-vs-wedged diff has the fields the submission-time + * dump cannot show. + */ + gsp_phys_gr_qmd64_post(qmd_dst_mem, "retire"); + + /* + * m179: the zero-write test is specific to launches with a + * non-empty TRACKED data set. A launch whose data entries do + * not resolve to any bounce page legitimately reports zero + * writes; treating that as a lost launch would re-execute a + * kernel for no reason (a false positive). Only + * n_tracked > 0 makes "wrote == 0" a possible no-execution + * signature. + */ + if (wrote == 0 && n_tracked) { + bool ff = gsp_phys_gr_qmd_fetched(qmd_dst_mem); + + pr_info("vgsp-gr: first-attempt witness: engine fetch %s (qmd sentinel %s) tracked=%u\n", + ff ? "OK" : "NONE", + ff ? "overwritten" : "intact", n_tracked); + } + /* + * r427 idempotent fast-accept: a zero-write launch whose identity + * (QMD minus release0 + staged content checksums) matches an + * earlier WRITING launch of this boot is a legitimate idempotent + * re-launch - the result is already in place. Accept immediately: + * no retries, no quiet holds. (See gsp_phys_gr_launch_fp.) + */ + if (wrote == 0 && n_tracked) { + u64 fp = gsp_phys_gr_launch_fp(qmd, req, bounce_csum_in); + + if (gsp_phys_gr_idem_match(fp)) { + pr_warn("vgsp-gr: IDEMPOTENT-ACCEPT h=0x%x push=0x%llx fp=0x%llx tracked=%u (same QMD+content as an earlier writing launch; no retries, no hold)\n", + req->h_object, req->push_va, fp, + n_tracked); + idem = true; + } + } + if (wrote == 0 && n_tracked && !idem && + gsp_phys_rmcall_gr_nowrite_retry_value()) { + u32 nw_retries = + (u32)gsp_phys_gr_nowrite_retries_value(); + /* + * r410: the sticky per-lane wedge (previous launch + * also no-wrote while the lane stays busy) is only + * cleared by idleness (r409 ~37 s, r416 ~52 s), so a + * back-to-back retry is futile and keeps it alive. + * When the streak says the lane is wedged, hold a + * quiet window with NO submissions before the retry + * (base on the first retry, doubled on the second), + * then let the normal fresh-bank + resched retry run. + * The quiet path relaxes the whole-launch budget + * (its own hold is the bound: 2 holds max) - the + * alternative is accepting every launch silently. + */ + u32 quiet_used = 0; + + while (wrote == 0 && nw_attempts < nw_retries) { + u32 quiet_ms = 0; + + if ((READ_ONCE(gsp_phys_gr_nw_streak) > 0 || + READ_ONCE(gsp_phys_gr_wedge_episode)) && + nw_attempts <= 1) { + quiet_ms = + gsp_phys_gr_wedge_quiet_ms_value(); + if (quiet_ms && nw_attempts) + quiet_ms <<= 1; + if (quiet_ms > 240000u) + quiet_ms = 240000u; + } + /* + * STALLFREE: the same whole-launch budget caps + * the retirement retry chain. When it is crossed + * the loop stops and the launch PROCEEDS with + * its current outcome (the zero-write accept + * below restores the release words and logs + * loudly), so a chain of slow dud retries can + * never hold the client past the budget. The + * r410 quiet path is exempt: it is entered only + * for a launch whose previous launch also + * no-wrote, and its own holds are bounded. + */ + if (!quiet_ms && + gsp_phys_gr_wait_budget_over(start)) { + gsp_phys_gr_stall_retry++; + gsp_phys_gr_stall_log("retry", + req->push_va, start, + "proceed"); break; - pr_info("vgsp-gr: outcome retry %u/%u (wrote=%u): refill + re-publish\n", - attempts, GSP_PHYS_RMCALL_GR_ATTEMPTS, - wrote); + } + nw_attempts++; + pr_info("vgsp-gr: outcome retry %u/%u (wrote=%u tracked=%u): refill + re-publish\n", + nw_attempts, nw_retries, + wrote, n_tracked); + /* + * r445 wedge hard re-arm (gsp_phys_gr_wedge_rearm, + * default off): the r445 evidence shows every + * post-dispatch no-store episode leaves the warm + * ladder sem probe ALIVE (the ladder block executes) + * while the client launch keeps writing nothing, so + * the plain refill+re-publish retries are futile. + * On the FINAL retry attempt, force one context + * save/unload (TSG PREEMPT bWait=1) and re-run the + * mode-35 lane-enable tail + schedules before the + * re-publish, so the retried launch meets a freshly + * restored context. Bounded: one preempt per launch + * (2 s RPC timeout), only on the last retry. + */ + if (nw_attempts == nw_retries && + gsp_phys_gr_wedge_rearm_value()) { + mutex_lock(&gsp_phys_rm_lock); + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + } + if (quiet_ms) { + gsp_phys_gr_wedge_quiet++; + pr_warn("vgsp-gr: WEDGE-QUIET hold %u ms before outcome retry %u (push=0x%llx; %u consecutive zero-write launch(es); no submissions in between)\n", + quiet_ms, nw_attempts, + req->push_va, + READ_ONCE(gsp_phys_gr_nw_streak)); + /* + * r420: publish the deliberate hold so + * the sema service's completion + * deadlines (the 12 s GR release hold + * and the 3 s per-launch consumption + * bound) pause instead of authoring + * this launch's release word under it - + * the client must not be released + * before the post-hold copy-out + * (gsp_phys_gr_quiet_hold_active). + */ + WRITE_ONCE(gsp_phys_gr_quiet_hold_on, + true); + msleep(quiet_ms); + WRITE_ONCE(gsp_phys_gr_quiet_hold_on, + false); + quiet_used = quiet_ms; + pr_warn("vgsp-gr: WEDGE-QUIET hold done (push=0x%llx); lanes re-armed by the retry below\n", + req->push_va); + } /* * m156 (E5): on the first zero-write retry, push the * proven stage-4 GR semaphore block before re-publishing. @@ -18627,7 +27559,7 @@ gr_page_unmapped: * (r155: 3 outcome retries + rearm all dead after the * wedge). Both outcomes are logged distinctly. */ - if (attempts == 1) { + if (nw_attempts == 1) { int sp_ret; mutex_lock(&gsp_phys_rm_lock); @@ -18635,7 +27567,7 @@ gr_page_unmapped: bar0); mutex_unlock(&gsp_phys_rm_lock); pr_info("vgsp-gr: outcome retry %u: warm sem push rc=%d (engine %s)\n", - attempts, sp_ret, + nw_attempts, sp_ret, sp_ret ? "WEDGED - skipping further retries" : "ALIVE"); if (sp_ret) { @@ -18643,7 +27575,150 @@ gr_page_unmapped: break; } } - gsp_phys_gr_fill_bounce(req, bounce_csum_in); + /* + * m160 (fresh-ack + fresh-deposit retry): the m153 + * retry re-published the same PB and then polled + * the release words WITHOUT clearing them - the + * previous attempt's ack (the identical payload) + * satisfied the poll instantly, so every retry + * burned only the retirement wait and concluded + * "engine wrote 0" without ever having waited for + * the engine to consume the re-published entry + * (r162: 3 dud retries, ~870 ms each, then + * accepted). Clear the guest release words (and + * the module semaphore when the QMD won't reach a + * guest page) so the poll below waits for a FRESH + * ack, and move the inline-QMD deposit to the + * other qmdwin bank (fresh epoch) and rebuild the + * method block for it, so the engine cannot have + * a cached fetch line or a stale deposit for the + * retry's entry. The mechanical sequence is + * exactly the one r162's ok=7 saga used when it + * recovered the no-execute launch (fresh ring + * entry + resched + fresh deposit). + */ + /* + * r378: re-validate the guest release page + * before zeroing it - the HPA was resolved at + * submission and the retry runs seconds later. + */ + if (qmd_rel_hpa && + gsp_sema_gr_write_guard(req->rel_va, + qmd_rel_hpa, + "gr-rel-zero")) + WRITE_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + + qmd_rel_off), 0); + if (ch_rel_hpa) + WRITE_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + + ch_rel_off), 0); + WRITE_ONCE(*(u32 *)sem_va, 0); + WRITE_ONCE(*(u32 *)(sem_va + 4), 0); + wmb(); + /* + * F3 retry hygiene: recompute the win0 bank + * and slot limit for THIS attempt (they were + * fixed at the first submission's epoch) and + * re-copy the prepared param/program/bank + * shadow pages into the alternate bank before + * the re-publish. The qmdwin bank swap (below) + * is unchanged; when both are active they share + * the same fresh epoch. + */ + { + bool win0_on = + gsp_phys_rmcall_gr_shadow_win0_value() && + boot_stage.rmcall_gr_shadow_win0.pages; + bool win0_moved = false; + bool qmdwin_swap = qmdwin_on && + boot_stage.rmcall_gr_qmdwin.pages; + + if (qmdwin_swap || win0_on) { + epoch = ++boot_stage.rmcall_gr_submit_epoch; + if (!epoch) + epoch = ++boot_stage.rmcall_gr_submit_epoch; + boot_stage.rmcall_gr_shadow_win0_used = + (epoch & 1u) * + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; + if (win0_on) + win0_moved = + gsp_phys_gr_win0_retry_rearm( + qmd, req, + win0_bank_base, + boot_stage.rmcall_gr_shadow_win0_used, + boot_stage.rmcall_gr_shadow_win0_used + + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS); + win0_bank_base = + boot_stage.rmcall_gr_shadow_win0_used; + shadow_slot_limit = + win0_bank_base + + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; + } + if (qmdwin_swap) { + qw_bank = (epoch & 1u) % + GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS; + qmd_dst = + boot_stage.rmcall_gr_qmdwin_va + + (u64)qw_bank * + GSP_PHYS_PAGE_SIZE; + qmd_dst_mem = + gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_qmdwin, + qw_bank); + if (qmd_dst_mem) { + memcpy(qmd_dst_mem, qmd, + GSP_PHYS_GR_QMD_WORDS * 4); + put_unaligned_le32( + GSP_PHYS_GR_QMD_SENTINEL, + qmd_dst_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4); + /* + * m179: order the fresh + * deposit before the + * re-publish/doorbell + * below (the first-attempt + * path has the same + * barrier; the publish tail + * also carries dma_wmb + + * wmb, this keeps the + * deposit->kick order + * explicit on the retry + * path). + */ + wmb(); + } + } + if (qmdwin_swap || win0_moved) { + words = gsp_phys_rmcall_build_gr_launch( + methods, req, qmd, + qmd_dst, ch_sem_va, + want_ch, win3d, + win3d_n, + gsp_phys_gr_attr_pre, + gsp_phys_gr_attr_pre_n); + gsp_phys_gr_pb_copy(methods, + words); + } + if (qmdwin_swap) + pr_info("vgsp-gr: outcome retry %u: release words zeroed + fresh qmdwin bank %u dest 0x%llx + win0 bank %u%s + PB rebuilt\n", + nw_attempts, qw_bank, + qmd_dst, win0_bank_base, + win0_moved ? + " (shadow pages re-copied)" : + ""); + else if (win0_moved) + pr_info("vgsp-gr: outcome retry %u: release words zeroed + fresh win0 bank %u (shadow pages re-copied) + PB rebuilt\n", + nw_attempts, win0_bank_base); + else + pr_info("vgsp-gr: outcome retry %u: release words zeroed (win0/qmdwin off)\n", + nw_attempts); + } + gsp_phys_gr_fill_bounce(req, bounce_csum_in, + bounce_staged_hpa); + gsp_phys_gr_fill_gap_bounce(bounce_csum_in, + bounce_staged_hpa); + /* CONT-73: refingerprint the direct-write + * witness set for this retry publish. */ + gsp_phys_gr_wit_capture(req); mutex_lock(&gsp_phys_rm_lock); ret = gsp_phys_rmcall_publish_locked( bar0, boot_stage.rmcall_va_pb, @@ -18654,15 +27729,17 @@ gr_page_unmapped: mutex_unlock(&gsp_phys_rm_lock); if (ret) { pr_info("vgsp-gr: outcome retry %u publish/resched failed rc=%d (accepting current outcome)\n", - attempts, ret); + nw_attempts, ret); ret = 0; break; } { unsigned long dl = jiffies + msecs_to_jiffies( - GSP_PHYS_RMCALL_GR_RETRY_MS); + GSP_PHYS_RMCALL_GR_POLL_MS); + unsigned long ack_start = jiffies; u32 rw0 = 0, rw1 = 0; + bool acked = false; for (;;) { if (qmd_rel_hpa) @@ -18683,18 +27760,289 @@ gr_page_unmapped: *(u32 *)(sem_va + 4)); if ((!qmd_rel || rw0 == want_qmd) && - rw1 == want_ch) + rw1 == want_ch) { + acked = true; break; + } if (time_after_eq(jiffies, dl)) break; - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); + } + pr_info("vgsp-gr: outcome retry %u: fresh ack %s (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x after %u ms)\n", + nw_attempts, + acked ? "OK" : "TIMEOUT", + rw0, want_qmd, rw1, + jiffies_to_msecs(jiffies - + ack_start)); + } + /* + * m161: hardware fetch witness. The fresh-ack + * poll cannot distinguish the engine's release + * from the sema service's software release (its + * pending-release sweep re-applies the payload + * as soon as the word reads below it). The QMD + * page's sentinel word is written only by the + * engine's QMD fetch, so it is the one side + * effect the sweep cannot fake: intact after the + * wait = this deposit never reached the engine. + */ + if (qmd_dst_mem) { + unsigned long wdl = jiffies + + msecs_to_jiffies(1500); + bool fetched = false; + + while (time_before(jiffies, wdl)) { + if (gsp_phys_gr_qmd_fetched( + qmd_dst_mem)) { + fetched = true; + break; + } + msleep(5); } + pr_info("vgsp-gr: outcome retry %u: engine fetch witness %s (qmd sentinel %s)\n", + nw_attempts, + fetched ? "OK" : "NONE", + fetched ? "overwritten" : "intact"); } wrote = gsp_phys_gr_output_retire_wait( - req, bounce_csum_in); + req, bounce_csum_in, &n_tracked); + /* CONT-73: direct-write witness per retry. */ + gsp_phys_gr_wit_report("retry-retire"); + gsp_phys_gr_qmd64_post(qmd_dst_mem, + "retry-retire"); pr_info("vgsp-gr: outcome retry %u done: engine wrote %u page(s)\n", - attempts, wrote); + nw_attempts, wrote); + /* r410: the quiet-window verdict. */ + if (quiet_used) { + if (wrote) { + gsp_phys_gr_wedge_heals++; + WRITE_ONCE(gsp_phys_gr_wedge_episode, + true); + pr_warn("vgsp-gr: WEDGE-QUIET healed: engine wrote %u page(s) after the %u ms hold (streak was %u)\n", + wrote, quiet_used, + READ_ONCE( + gsp_phys_gr_nw_streak)); + } else { + pr_warn("vgsp-gr: WEDGE-QUIET no heal: %u ms quiet and retry %u still wrote 0 page(s)\n", + quiet_used, nw_attempts); + } + quiet_used = 0; + } + } + } + /* + * m160/m179: if every fresh-ack retry still produced no engine + * writes, the release words may be left zeroed from the last + * retry. Restore the expected payloads so the guest client + * observes its completion and proceeds to its own result check + * (a genuinely lost launch then surfaces as the client's own + * FAIL, never as a hang). This accept runs even when the retry + * loop is disabled (gsp_phys_rmcall_gr_nowrite_retry=0) so a + * zero-write outcome is never silent. + */ + if (wrote == 0 && n_tracked && !idem) { + bool restored = false; + bool fetched = gsp_phys_gr_qmd_fetched(qmd_dst_mem); + + /* + * m179: the no-write accept is the last resort after + * the dedicated re-execution budget + * (gsp_phys_gr_nowrite_retries). It stays + * unconditional - the alternative is stalling the + * client forever - but it is now LOUD and classified + * by the hardware fetch witness: an intact sentinel + * means the deposit never reached the engine (a + * CONFIRMED lost launch, the sweep/hold-faked-ack + * class); an overwritten sentinel means the engine + * consumed the deposit, so the zero tracked writes + * may also be a legitimately read-only kernel + * (AMBIGUOUS). + */ + pr_warn("vgsp-gr: NOWRITE-%s h=0x%x push=0x%llx n_data=%u tracked=%u wrote=0 retries=%u qmd_rel=0x%x (engine fetch %s) consec=%u quiet=%llu heals=%llu\n", + fetched ? "AMBIGUOUS" : "CONFIRMED", + req->h_object, req->push_va, req->n_data, + n_tracked, nw_attempts, want_qmd, + fetched ? "OK" : "NONE", + READ_ONCE(gsp_phys_gr_nw_streak) + 1u, + gsp_phys_gr_wedge_quiet, + gsp_phys_gr_wedge_heals); + + /* + * m180 (lottery discriminator, agent-order H1/H2). + * Read BOTH qmdwin banks' sentinel words. Facts only: + * the other bank is normally the previous launch's + * deposit (legitimately consumed), so only THIS + * launch's sentinel classifies the event: intact = + * this deposit never reached the engine; overwritten + * = the engine consumed it (or something zeroed the + * word), so the zero stores are an engine-side + * non-execution of a fetched deposit, not a lost + * deposit. + */ + if (qmdwin_on && boot_stage.rmcall_gr_qmdwin.pages && + qmd_dst_mem) { + u8 *om = gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_qmdwin, + qw_bank ^ 1u); + + pr_warn("vgsp-gr: zero-write deposit stamp: bank=%u sentinel=0x%08x (%s) other-bank=%u %s\n", + qw_bank, + get_unaligned_le32( + qmd_dst_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4), + fetched ? "this deposit fetched" : + "this deposit NOT fetched", + qw_bank ^ 1u, + om ? (gsp_phys_gr_qmd_fetched(om) ? + "sentinel overwritten (previous deposit)" : + "sentinel intact") : "n/a"); + } + /* + * (b) Deposit readback at the accept (m181): the same + * QMD-word dump the first-timeout path logs, so a + * zero-write launch's deposit content (w0/w4/w12/.../ + * w63) shows whether the engine consumed the header + * words or the page is untouched, without needing a + * timeout to have happened first. + */ + gsp_phys_gr_deposit_rb_log(req, qmd_dst_mem); + /* + * GRFIX FIX4 (agent-A decisive experiment): bounded + * real-GR register dump at the zero-write accept, one + * line per classified launch, using the same readl + * accessor and the same offsets as the read-only + * debugfs probe (gsp_phys_reg_probe_show): FE_HWW_ESR + * 0x404000, SKED_HWW_ESR 0x407020, PGRAPH_TRAPPED_ADDR + * 0x400704 + TRAPPED_DATA_LO/HI 0x400708/70c, + * FECS_IRQSTAT 0x409008, FECS_CUR/NEW_CTX 0x409b00/04. + * No new locks or sleeps: the submit context already + * holds gsp_phys_rm_lock exactly as the surrounding + * post-mortem reads do. Zero is a legitimate value for + * several of these (no context resident), so the line + * is classified with the sentinel/faultbuf evidence, + * not on its own. + */ + if (bar0) + pr_warn("vgsp-gr: NOWRITE-REGS fe_esr=0x%08x sked_esr=0x%08x trap=0x%08x/%08x/%08x fecs_irq=0x%08x ctx=0x%08x->0x%08x\n", + readl(bar0 + 0x00404000u), + readl(bar0 + 0x00407020u), + readl(bar0 + 0x00400704u), + readl(bar0 + 0x00400708u), + readl(bar0 + 0x0040070cu), + readl(bar0 + 0x00409008u), + readl(bar0 + 0x00409b00u), + readl(bar0 + 0x00409b04u)); + /* r425: the wedged-launch SKED/FECS baseline. */ + gsp_phys_gr_sked_wit(bar0, "accept"); + /* + * r425 stale-store witness: re-scan the last completed + * launch's first engine-changed page. A change since its + * copy-out proves a late store landed after the launch + * was shipped (a stuck/queued grid or a stale-QMD + * execution), the mechanism behind the "1 page" heals. + */ + if (gsp_phys_gr_prev_out_valid && + bounce_staged_hpa[gsp_phys_gr_prev_out_bi] == 0 && + gsp_phys_gr_data_chunk()->pages) { + const u8 *pp = gsp_phys_chunk_page_va( + gsp_phys_gr_data_chunk(), + gsp_phys_gr_prev_out_bi); + u64 pc; + + if (pp) { + pc = gsp_phys_gr_fnv1a64(pp, GSP_PHYS_PAGE_SIZE); + if (pc != gsp_phys_gr_prev_out_csum) + pr_warn("vgsp-gr: PREV-OUT-CHANGED bi=%u fp %016llx -> %016llx (late store after its launch shipped)\n", + gsp_phys_gr_prev_out_bi, + gsp_phys_gr_prev_out_csum, + pc); + } + } + /* + * (c) Late-store check: a store ramp that retires after + * the host's window is indistinguishable from a lost + * dispatch at the accept moment. Re-scan once at +1 s + * and report whether the tracked pages changed in the + * meantime; the copy-out below picks up any stores + * that arrived late, so this both classifies and heals + * the late sub-case. + */ + { + u32 late_tracked = 0; + u32 late; + + msleep(1000); + late = gsp_phys_gr_scan_pages(req, bounce_csum_in, + &late_tracked, NULL); + pr_warn("vgsp-gr: zero-write late-check: %u/%u pages changed after +1000 ms (%s)\n", + late, late_tracked, + late ? "LATE-STORES - copy-out will ship them" : + "NO-STORES - dispatch lost or read-only kernel"); + /* + * SESSION 153-CONTINUED-83: the no-writes moment + * is the store wall's fingerprint - snapshot the + * DS/SPH/MME/FE-decode evidence here so the + * stage that never received a program header (or + * the MME PC that never emitted the draw) is on + * the same line as the failing launch. + */ + if (!late) + gsp_phys_store_side_log(gsp_phys_gpu_bar0(), + "no-writes", + req->h_object); + } + + if (qmd_rel_hpa) { + u8 *rp = (u8 *)__va(qmd_rel_hpa) + qmd_rel_off; + + /* + * r378: same target re-validation as the + * retry zero above. + */ + if (READ_ONCE(*(u32 *)rp) != want_qmd && + gsp_sema_gr_write_guard(req->rel_va, + qmd_rel_hpa, + "gr-rel-restore")) { + WRITE_ONCE(*(u32 *)rp, want_qmd); + restored = true; + } + } + if (ch_rel_hpa) { + u8 *rp = (u8 *)__va(ch_rel_hpa) + ch_rel_off; + + if (READ_ONCE(*(u32 *)rp) != want_ch) { + WRITE_ONCE(*(u32 *)rp, want_ch); + restored = true; + } } + if (restored) + pr_warn("vgsp-gr: no-write accept: release words restored (qmd=0x%08x ch=0x%08x); client's own check decides\n", + want_qmd, want_ch); + } + /* + * r427: record the launch identity of every launch that WROTE, + * so a later idempotent re-launch (zero writes, same identity) + * can accept immediately (see gsp_phys_gr_launch_fp). + */ + if (wrote && n_tracked) + gsp_phys_gr_idem_record( + gsp_phys_gr_launch_fp(qmd, req, bounce_csum_in)); + /* + * r410: streak bookkeeping for the quiet-window trigger. A + * launch with tracked data that wrote nothing extends the + * zero-write streak; any launch that wrote resets it. Runs + * even when the retry loop is disabled so the streak always + * reflects the accepted-outcome history. + */ + if (n_tracked) { + WRITE_ONCE(gsp_phys_gr_nw_streak, + wrote ? 0u : + READ_ONCE(gsp_phys_gr_nw_streak) + 1u); + /* r411: arm the wedge episode on any accepted + * zero-write launch, so the next zero-write launch + * quiets even after a heal reset the streak. */ + if (!wrote) + WRITE_ONCE(gsp_phys_gr_wedge_episode, true); } } /* @@ -18705,21 +28053,81 @@ gr_page_unmapped: */ if (req->n_data && gsp_phys_gr_data_chunk()->pages) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); - u32 wbi, wrote = 0, changed = 0; + u32 wbi, wrote = 0, changed = 0, eligible = 0; + u32 aliased = 0, unchanged = 0, skipped = 0; u32 chg[8] = { }; + u32 outwit = 0; /* COPYWIT: bounded per-launch witness lines */ for (wbi = 0; wbi < req->n_data; wbi++) { u32 bi = gsp_phys_gr_data_bi(req->data[wbi].reloc_va); const u8 *src; - u32 ci, csum = 0; + u64 fp; if (bi == GSP_PHYS_GR_DATA_BI_NONE || !req->data[wbi].hpa) continue; src = gsp_phys_chunk_page_va(dchunk, bi); - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != bounce_csum_in[bi]) { + if (!src) + continue; + eligible++; + /* + * r378: only the page whose content was staged into + * this slot this launch may be written back. A rec + * aliasing another rec's reloc slot would otherwise + * ship the other page's content into this guest page + * (the multi-writer clobber class), and a slot left + * over from a previous launch likewise. + */ + if (bounce_staged_hpa[bi] != req->data[wbi].hpa) { + aliased++; + continue; + } + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { + /* + * r378: the engine did not touch this page, + * so the guest page still holds exactly this + * content (the slot was copied in from it). + * Writing it back is a no-op at best and + * stale-content corruption at worst - skip. + */ + unchanged++; + /* + * r452: witness the UNCHANGED page's chunk + * content (first 4 words) and the guest + * page's, bounded per launch. This is the + * only place the two can be compared for a + * page the csum rule classified as "engine + * did not touch" - it separates "the store + * never landed in the chunk" from "the + * store landed but the csum rule missed + * it" (or "the reader resolves a different + * page than the writer wrote"). + */ + if (gsp_phys_gr_copywit_enabled() && + outwit < GSP_PHYS_GR_COPYWIT_MAX) { + const u8 *gp = req->data[wbi].hpa ? + (const u8 *)__va(req->data[wbi].hpa) : + NULL; + + outwit++; + pr_info("vgsp-gr: OUT-UNCH bi=%u va=0x%llx fp=%016llx pre=%016llx chunk=%08x %08x %08x %08x guest=%08x %08x %08x %08x\n", + bi, req->data[wbi].va, fp, + bounce_csum_in[bi], + get_unaligned_le32(src), + get_unaligned_le32(src + 4), + get_unaligned_le32(src + 8), + get_unaligned_le32(src + 12), + gp ? get_unaligned_le32(gp) : 0, + gp ? get_unaligned_le32(gp + 4) : 0, + gp ? get_unaligned_le32(gp + 8) : 0, + gp ? get_unaligned_le32(gp + 12) : 0); + } + continue; + } + { + u64 f8 = (u64)get_unaligned_le64(src); + /* * m142: which pages the engine touched. bi is * the page index inside the active data region @@ -18738,14 +28146,156 @@ gr_page_unmapped: get_unaligned_le32(src + 4), get_unaligned_le32(src + 8), get_unaligned_le32(src + 12)); + /* + * COPYWIT fail4: per-changed-page copy-out + * witness - the exact (device VA -> guest + * HPA) pair the results were written back to + * plus the first/last 8 bytes shipped and the + * pre/post csum pair (proof the engine changed + * the page vs the copy-in baseline). The D2H + * side of the same VA is named by the + * sema-bigcopy resolution lines. + */ + if (gsp_phys_gr_copywit_enabled() && + outwit < GSP_PHYS_GR_COPYWIT_MAX) { + /* + * r391 diagnostic (sum=0 class): the + * landing-table resolution of this VA + * (the page the D2H reader will use) + * and its first 8 bytes beside the + * write target - a launch/readback + * disagreement in one line. + */ + u64 rpa = 0, rval = 0; + u32 rap = 0; + bool rhave = gsp_sema_gr_va_snap( + req->data[wbi].va, &rpa, + &rap, &rval); + + outwit++; + pr_info("vgsp-gr: OUT-PAGE-WIT va=0x%llx hpa=0x%llx bi=%u first8=%016llx last8=%016llx pre=%016llx post=%016llx res=%s pa=0x%llx ap=%u val8=%016llx vbase=%px\n", + req->data[wbi].va, + req->data[wbi].hpa, bi, f8, + (u64)get_unaligned_le64( + src + GSP_PHYS_PAGE_SIZE - + 8), + bounce_csum_in[bi], fp, + rhave ? "ledger" : "none", + rpa, rap, rval, + (void *)gsp_sema_gr_shadow_base()); + } + /* + * OUT-ALIAS-WIT: identical first 8 bytes as + * an earlier changed page of this launch = + * the replicated-tile structure, with both + * recs named. r226: all-zero pages are + * excluded on both sides (the fresh-client + * fill stages zero runs; they are not alias + * evidence). + */ + if (gsp_phys_gr_copywit_enabled() && + !gsp_phys_gr_page_all_zero( + gsp_phys_gr_page_sum32(src), src)) { + u32 wi; + + for (wi = 0; wi < gsp_phys_gr_outwit_n; + wi++) { + if (gsp_phys_gr_outwit_first8[wi] != + f8) + continue; + if (gsp_phys_gr_alias_log < 12) { + gsp_phys_gr_alias_log++; + pr_info("vgsp-gr: OUT-ALIAS-WIT va=0x%llx bi=%u partner_va=0x%llx partner_bi=%u first8=%016llx\n", + req->data[wbi].va, + bi, + gsp_phys_gr_outwit_va[wi], + gsp_phys_gr_outwit_bi[wi], + f8); + } + break; + } + if (gsp_phys_gr_outwit_n < + GSP_PHYS_GR_OUTWIT_SLOTS) { + gsp_phys_gr_outwit_first8[gsp_phys_gr_outwit_n] = f8; + gsp_phys_gr_outwit_va[gsp_phys_gr_outwit_n] = req->data[wbi].va; + gsp_phys_gr_outwit_bi[gsp_phys_gr_outwit_n] = bi; + gsp_phys_gr_outwit_n++; + } + } if (changed < 8) chg[changed] = bi; + /* + * r425: remember the first engine-changed page + * (post-copy-out content) for the next zero-write + * accept's late-store witness. + */ + if (changed == 0) { + gsp_phys_gr_prev_out_bi = bi; + gsp_phys_gr_prev_out_csum = fp; + gsp_phys_gr_prev_out_valid = true; + } changed++; } + /* + * r378: the target HPA was resolved at submission; + * the poll/retire/retry window is seconds wide and a + * guest buffer freed and re-homed in it would receive + * this full page through the stale target. Re-resolve + * the VA and skip (bounded log) on a positive + * disagreement. + */ + if (!gsp_sema_gr_write_guard(req->data[wbi].va, + req->data[wbi].hpa, + "gr-bounce-copy-out")) { + skipped++; + continue; + } memcpy((u8 *)__va(req->data[wbi].hpa), src, GSP_PHYS_PAGE_SIZE); + gsp_phys_gr_page_spot_check( + (const u8 *)__va(req->data[wbi].hpa), src, + req->data[wbi].va, "gr-bounce-copy-out"); + /* + * r391d post-write witness (sum=0 class): read the VA's + * landing page right after the write through the same + * path the D2H reader uses. If val8 is not first8 the + * write target and the reader's page are different + * (aperture/staleness disagreement), not a lost write. + */ + if (outwit < GSP_PHYS_GR_COPYWIT_MAX) { + u64 rpa2 = 0, rval2 = 0; + u32 rap2 = 0; + + if (gsp_sema_gr_va_snap(req->data[wbi].va, + &rpa2, &rap2, &rval2)) { + outwit++; + pr_info("vgsp-gr: OUT-PAGE-POST va=0x%llx hpa=0x%llx res_pa=0x%llx ap=%u val8=%016llx vbase=%px\n", + req->data[wbi].va, + req->data[wbi].hpa, rpa2, + rap2, rval2, + (void *)gsp_sema_gr_shadow_base()); + } + } wrote++; } + if ((u64)req->n_data << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-bounce-copy-out va=0x%llx len=0x%llx done=0x%llx\n", + req->data[0].va, (u64)req->n_data << 12, + (u64)wrote << 12); + /* + * r378: the copy-out is now selective (staged + changed + + * re-validated pages only), so completeness is an accounting + * check over the eligible recs: every one is either written, + * skipped as unchanged, skipped as aliased, or refused by the + * write guard (logged as WGUARD-SKIP). This can only fire if + * a rec vanished between the filters. + */ + if (wrote + unchanged + aliased + skipped < eligible) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-bounce-copy-out\n", + req->data[0].va, + (u64)eligible << 12, + (u64)(wrote + unchanged + aliased + + skipped) << 12); /* m143b: unconditional first words of the first page of each * 32-page relocation window after the run, so the engine's * read/write footprint per window is provable without relying @@ -18762,8 +28312,7 @@ gr_page_unmapped: if (bi == GSP_PHYS_GR_DATA_BI_NONE) continue; wp = gsp_phys_chunk_page_va(dchunk, bi); - for (k = 0; k < GSP_PHYS_PAGE_SIZE; k += 4) - wc += get_unaligned_le32(wp + k); + (void)k; (void)wc; pr_info("vgsp-gr: window[%u] reloc=0x%llx first=%08x %08x %08x %08x changed=%u\n", bi, req->data[wi].reloc_va, @@ -18771,13 +28320,84 @@ gr_page_unmapped: get_unaligned_le32(wp + 4), get_unaligned_le32(wp + 8), get_unaligned_le32(wp + 12), - wc != bounce_csum_in[bi]); + gsp_phys_gr_fnv1a64( + wp, GSP_PHYS_PAGE_SIZE) != + bounce_csum_in[bi]); } } - pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine (first bi: %u %u %u %u %u %u %u %u)\n", - wrote, changed, chg[0], chg[1], chg[2], chg[3], chg[4], + pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u alias-skipped, %u stale-skipped (first bi: %u %u %u %u %u %u %u %u)\n", + wrote, changed, unchanged, aliased, skipped, + chg[0], chg[1], chg[2], chg[3], chg[4], chg[5], chg[6], chg[7]); } + /* + * relfix: copy the late-resolved pages back to the guest pages the + * resolver named - the same identity the word rewrite handed the + * engine - after the retire wait, exactly like the tracked data + * pages above. + */ + if (gsp_phys_gr_gap_n && gsp_phys_gr_data_chunk()->pages) { + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u32 gi, gwrote = 0, gchanged = 0; + u32 gunchanged = 0, gskipped = 0; + u32 gchg[4] = { }; + + for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { + u32 bi = gsp_phys_gr_gap[gi].bi; + const u8 *src = gsp_phys_chunk_page_va(dchunk, bi); + u64 fp; + + if (!src) + continue; + /* + * r378: same staging-identity rule as the tracked + * copy-out above - only the page staged into this + * slot may receive the slot's content. + */ + if (bounce_staged_hpa[bi] != gsp_phys_gr_gap[gi].hpa) { + gskipped++; + continue; + } + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { + /* Engine did not touch it: nothing to ship. */ + gunchanged++; + continue; + } + if (gchanged < 4) + gchg[gchanged] = bi; + gchanged++; + if (!gsp_sema_gr_write_guard(gsp_phys_gr_gap[gi].va, + gsp_phys_gr_gap[gi].hpa, + "gr-gap-copy-out")) { + gskipped++; + continue; + } + memcpy((u8 *)__va(gsp_phys_gr_gap[gi].hpa), src, + GSP_PHYS_PAGE_SIZE); + gsp_phys_gr_page_spot_check( + (const u8 *)__va(gsp_phys_gr_gap[gi].hpa), src, + gsp_phys_gr_gap[gi].va, "gr-gap-copy-out"); + gwrote++; + } + pr_info("vgsp-gr: gap copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u stale-skipped (first bi: %u %u %u %u)\n", + gwrote, gchanged, gunchanged, gskipped, + gchg[0], gchg[1], gchg[2], gchg[3]); + if ((u64)gsp_phys_gr_gap_n << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-gap-copy-out va=0x%llx len=0x%llx done=0x%llx\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gwrote << 12); + /* + * r378: same accounting rule as the tracked copy-out above - + * the intentional unchanged/stale skips are not truncation. + */ + if (gwrote + gunchanged + gskipped < gsp_phys_gr_gap_n) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-gap-copy-out\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gwrote << 12); + } poll = w1; if (bar0) pr_info("vgsp-gr: SKED readback (ok): HWW_ESR=0x%08x INFO2=0x%08x CWD_T0=0x%08x CWD_T1=0x%08x CWD_T2=0x%08x\n", @@ -18808,8 +28428,20 @@ gr_page_unmapped: jiffies_to_msecs(jiffies - start), gsp_phys_gr_ok); /* m128: post-success qmdwin words so a probe that stores a value into * the qmdwin page (e.g. p6's constant-read dump at w48) is readable - * from the success log (the post-mortem only prints on timeout). */ - if (boot_stage.rmcall_gr_qmdwin.pages) { + * from the success log (the post-mortem only prints on timeout). + * r434: read the launch's OWN bank (qmd_dst_mem) when available - + * the old bank-0-only read showed a stale page for odd epochs. */ + if (qmd_dst_mem) { + const u32 *qw = (const u32 *)qmd_dst_mem; + + pr_info("vgsp-gr: post-OK qmdwin w48..59=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + get_unaligned_le32(qw + 48), get_unaligned_le32(qw + 49), + get_unaligned_le32(qw + 50), get_unaligned_le32(qw + 51), + get_unaligned_le32(qw + 52), get_unaligned_le32(qw + 53), + get_unaligned_le32(qw + 54), get_unaligned_le32(qw + 55), + get_unaligned_le32(qw + 56), get_unaligned_le32(qw + 57), + get_unaligned_le32(qw + 58), get_unaligned_le32(qw + 59)); + } else if (boot_stage.rmcall_gr_qmdwin.pages) { const u32 *qw = (const u32 *)gsp_phys_chunk_page_va( &boot_stage.rmcall_gr_qmdwin, 0); @@ -18821,11 +28453,19 @@ gr_page_unmapped: ret = 0; out: + /* + * M91 FAULT-CAP: one line per faulting launch, emitted at the end of + * the launch window (fault epoch compared against the snapshot taken + * at submission; inert - one compare - when no fault was seen). + */ + gsp_phys_gr_m91_fault_cap(req, qmd, qmd_dst, qmd_dst_mem, attempts, + rearmed, fast_rearmed, ret == 0); if (out) { out->sem_value = poll; out->elapsed_ms = jiffies_to_msecs(jiffies - start); out->rc = ret; out->fault = fault; + out->qmd_rel_redirected = qmd_rel_redirected; } return ret; } @@ -18943,6 +28583,181 @@ static void gsp_phys_rmcall_ctx_diag_decode(const char *tag) ramfc_valid); } +/* + * SESSION 153-CONTINUED-96: secondary COPY0 lane enable. On a GR0 ladder + * the ladder channel speaks GR and CE class methods are never legal on it, + * so the ladder's own FBMEM reads (gsp_phys_ctx_readback_try) need a second + * engine. Default 1 builds the lane (best-effort: a failed build only + * disables the readback, the GR ladder is unchanged); 0 skips it entirely. + */ +static bool gsp_phys_rmcall_ce_lane = 1; +module_param_named(gsp_phys_rmcall_ce_lane, gsp_phys_rmcall_ce_lane, bool, + 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_ce_lane, + "Session 153-CONTINUED-96: build and schedule the secondary COPY0 lane (COPY0 TSG + channel on the ladder VAS, engineType 9) on a GR0 ladder so the ladder's own CE copy blocks (engine FBMEM readback) have a legal engine (default 1; 0 disables the lane, the GR ladder is unchanged)"); + +static bool gsp_phys_rmcall_ce_lane_value(void) +{ + return gsp_phys_rmcall_ce_lane; +} + +/* + * SESSION 153-CONTINUED-96: secondary COPY0 lane bringup (GR0 ladders only). + * + * The ladder's own FBMEM readback (and any later diagnostic copy of engine + * state the host cannot aperture) needs a CE engine, but CE class methods + * are never legal on the GR0 ladder channel. This builds the smallest + * second lane on the SAME VAS: its own COPY0 TSG (parent device, same + * ladder VAS) and its own V5-shape channel (engineType 9, TSG parent, + * lane memdescs, GPFIFO at VA_CE_GPFIFO), then runs the historical + * COPY0 stage-4 sequence - BIND(COPY0) -> GPFIFO_SCHEDULE -> + * TSG_GPFIFO_SCHEDULE -> notif index (advisory) -> GET_WORK_SUBMIT_TOKEN - + * and stores the lane's own work-submit token. Best-effort by design: a + * failure is logged and the GR ladder continues unchanged (no stage-4 + * abort, no retry); rmcall_ce_lane_valid stays false and the readback is + * skipped. The PDB slots for the lane's GPFIFO/PB pages were planted in + * pdb_build (stage 3, before any engine fetch). + */ +static int gsp_phys_rmcall_ce_lane_bringup(void __iomem *bar0) +{ + struct gsp_phys_rmcall_tsg_params tsg; + struct gsp_phys_rmcall_channel_params ch; + struct gsp_phys_rmcall_channel_variant v; + u32 token = 0, rpc_result = 0, status = 0; + u8 sched_params[2] = { 1, 0 }; + u32 bind_params = GSP_PHYS_RMCALL_ENGINE_COPY0; + u32 notif_index = GSP_PHYS_RMCALL_NOTIF_INDEX; + int ret; + + if (!gsp_phys_rmcall_engine_is_gr()) + return 0; /* COPY0 ladder: the lane already is it */ + if (!gsp_phys_rmcall_ce_lane_value()) + return 0; + if (boot_stage.rmcall_ce_lane_valid) + return 0; + + /* (1) TSG on the ladder VAS bound to the COPY0 engine. */ + memset(&tsg, 0, sizeof(tsg)); + tsg.h_va_space = cpu_to_le32(boot_stage.rmcall_vas); + tsg.engine_type = cpu_to_le32(GSP_PHYS_RMCALL_ENGINE_COPY0); + ret = gsp_phys_rmcall_alloc(bar0, boot_stage.rmcall_client, + boot_stage.rmcall_device, + GSP_PHYS_RMCALL_HANDLE_CE_TSG, + GSP_PHYS_RMCALL_CLASS_TSG, &tsg, sizeof(tsg), + false, NULL, "KEPLER_CHANNEL_GROUP_A (CE lane)", + true, NULL, NULL); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: COPY0 TSG alloc failed (%d); the lane is not built and the GR ladder continues unchanged\n", + ret); + return ret; + } + boot_stage.rmcall_ce_tsg = GSP_PHYS_RMCALL_HANDLE_CE_TSG; + + /* (2) The lane channel: V5 shape, COPY0 engine, lane memdescs. */ + memset(&v, 0, sizeof(v)); + v.name = "ce-lane-tsg-vas-pdb-skip-scrubber"; + v.id = 5; + v.tsg_parent = true; + v.flags = GSP_PHYS_RMCALL_CHANNEL_FLAG_SKIP_SCRUBBER; + v.internal_flags = GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_PRIV_KERNEL; + v.server_mem = false; + v.handle = GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL; + gsp_phys_rmcall_channel_params_init(&ch, &v); + ch.engine_type = cpu_to_le32(GSP_PHYS_RMCALL_ENGINE_COPY0); + ch.h_context_share = 0; /* a CE channel never joins a GR share */ + ch.gp_fifo_offset = cpu_to_le64(GSP_PHYS_RMCALL_VA_CE_GPFIFO); + ch.instance_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_INSTANCE); + ch.userd_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_USERD); + ch.ramfc_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_RAMFC); + ch.mthdbuf_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_MTHDBUF); + ret = gsp_phys_rmcall_alloc(bar0, boot_stage.rmcall_client, + boot_stage.rmcall_ce_tsg, + GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL, + GSP_PHYS_RMCALL_CLASS_CHANNEL, &ch, sizeof(ch), + false, NULL, "CE lane channel", true, NULL, + &status); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: channel alloc failed (%d, status=0x%x); the lane is not built and the GR ladder continues unchanged\n", + ret, status); + return ret; + } + boot_stage.rmcall_ce_channel = GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL; + + /* (3) BIND to the COPY0 engine (the historical stage-4 sequence). */ + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_BIND_CHANNEL, + &bind_params, sizeof(bind_params), 16, NULL, 0, + &rpc_result, &status, + "CE lane BIND_CHANNEL COPY0"); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: BIND_CHANNEL (engineType=0x%x) failed (rpc_result=0x%x status=0x%x); lane unusable, GR ladder continues\n", + bind_params, rpc_result, status); + return ret; + } + + /* (4) GPFIFO_SCHEDULE + TSG_GPFIFO_SCHEDULE. */ + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_GPFIFO_SCHEDULE, + sched_params, sizeof(sched_params), 16, NULL, 0, + &rpc_result, &status, "CE lane GPFIFO_SCHEDULE"); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: GPFIFO_SCHEDULE failed (rpc_result=0x%x status=0x%x); lane unusable, GR ladder continues\n", + rpc_result, status); + return ret; + } + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_tsg, + GSP_PHYS_RMCALL_CTRL_TSG_GPFIFO_SCHEDULE, + sched_params, sizeof(sched_params), 16, NULL, 0, + &rpc_result, &status, "CE lane TSG_GPFIFO_SCHEDULE"); + if (ret) + pr_warn("vgsp-boot: rm-call: CE lane: TSG_GPFIFO_SCHEDULE advisory failure (rpc_result=0x%x status=0x%x); continuing\n", + rpc_result, status); + + /* (5) notif index (advisory) + the lane's own work-submit token. */ + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_SET_WORK_SUBMIT_TOKEN_NOTIF_INDEX, + ¬if_index, sizeof(notif_index), 16, NULL, 0, + &rpc_result, &status, + "CE lane SET_WORK_SUBMIT_TOKEN_NOTIF_INDEX"); + if (ret) + pr_info("vgsp-boot: rm-call: CE lane: SET_WORK_SUBMIT_TOKEN_NOTIF_INDEX advisory failure (rpc_result=0x%x status=0x%x); continuing to the token\n", + rpc_result, status); + token = 0; + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_GET_WORK_SUBMIT_TOKEN, + &token, sizeof(token), 28, &token, sizeof(token), + &rpc_result, &status, "CE lane GET_WORK_SUBMIT_TOKEN"); + if (ret && token != 0) { + pr_warn("vgsp-boot: rm-call: CE lane: GET_WORK_SUBMIT_TOKEN failed (rpc_result=0x%x status=0x%x) but the echoed params carry token=0x%x; proceeding\n", + rpc_result, status, token); + ret = 0; + } + if (ret || !token) { + pr_warn("vgsp-boot: rm-call: CE lane: no work-submit token (ret=%d rpc_result=0x%x status=0x%x token=0x%x); lane unusable, GR ladder continues\n", + ret, rpc_result, status, token); + return ret ? ret : -EIO; + } + boot_stage.rmcall_ce_token = token; + boot_stage.rmcall_ce_lane_valid = true; + pr_info("vgsp-boot: rm-call: CE lane READY (secondary COPY0 on the ladder VAS): hTsg=0x%x hChannel=0x%x token=0x%x (runlistId=%u chId=0x%x) gpFifoVa=0x%llx pbVa=0x%llx; CE copies on it read FBMEM the host cannot aperture\n", + boot_stage.rmcall_ce_tsg, boot_stage.rmcall_ce_channel, token, + (token >> 16) & 0x7fu, token & 0xfffu, + GSP_PHYS_RMCALL_VA_CE_GPFIFO, GSP_PHYS_RMCALL_VA_CE_PB); + return 0; +} + static int gsp_phys_rmcall_stage4(void __iomem *bar0) { struct gsp_phys_boot_chunk *userd_c = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; @@ -19124,6 +28939,22 @@ static int gsp_phys_rmcall_stage4(void __iomem *bar0) boot_stage.rmcall_token = token; /* + * SESSION 153-CONTINUED-96: on a GR0 ladder, build the secondary + * COPY0 lane (own TSG/channel on the same VAS, own token) so the + * ladder's own CE copy blocks - the engine-context FBMEM readback - + * have a legal engine. Best-effort: a failure disables the readback + * only; the GR ladder below is byte-identical. On a COPY0 ladder the + * bringup is a no-op (the ladder channel already is the CE lane). + */ + { + int ce_ret = gsp_phys_rmcall_ce_lane_bringup(bar0); + + if (ce_ret) + pr_warn("vgsp-boot: rm-call: CE lane bringup failed (%d); engine FBMEM readback unavailable this boot (GR ladder unchanged)\n", + ce_ret); + } + + /* * H1: a GR0 ladder releases a semaphore on the compute engine instead * of running a CE copy. The dedicated helper performs the same guarded * submission (gr=true), doorbell and module-sem poll, logs REAL GR SEM @@ -19193,7 +29024,7 @@ static int gsp_phys_rmcall_stage4(void __iomem *bar0) gsp_phys_status_drain(NULL); return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } if (memcmp(gsp_phys_chunk_page_va(src, 0), @@ -19670,6 +29501,8 @@ static void gsp_phys_rmcall_fini(struct pci_dev *pdev) gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_pt[i]); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_shadow); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_qmdwin); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_pb2); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_probe); /* * B-2: the old 208-page bounce window (previously leaked here) and the * grown data-bounce region chunks. The region's PT chunk and data chunk @@ -19814,6 +29647,17 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) boot_stage.rmcall_gr_promote_step, boot_stage.rmcall_gr_promote_size, boot_stage.rmcall_gr_hvirt); + /* SESSION 153-CONTINUED-96: secondary COPY0 lane (GR0 ladders only). */ + if (boot_stage.rmcall_ce_tsg || boot_stage.rmcall_ce_lane_valid) + seq_printf(m, + "rm_call_ce_lane: valid=%u hTsg=0x%x hChannel=0x%x token=0x%x gpFifoVa=0x%llx pbVa=0x%llx ceLaneParam=%u\n", + boot_stage.rmcall_ce_lane_valid, + boot_stage.rmcall_ce_tsg, + boot_stage.rmcall_ce_channel, + boot_stage.rmcall_ce_token, + GSP_PHYS_RMCALL_VA_CE_GPFIFO, + GSP_PHYS_RMCALL_VA_CE_PB, + gsp_phys_rmcall_ce_lane_value() ? 1u : 0u); seq_printf(m, "rm_call_guest: gpa=0x%llx hpa=0x%llx g2h_ok=%u w2g_ok=%u selftest_ok=%u gpa_valid=%u phys_domain_identity=%d\n", boot_stage.rmcall_guest_gpa, boot_stage.rmcall_guest_hpa, @@ -19929,12 +29773,18 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) gsp_phys_backend_gr_counters(&b_sub, &b_ok, &b_refused, &b_timeout, &b_pool); seq_printf(m, - "work_gr: req=%llu fallback=%llu ok=%llu refused=%llu timeout=%llu fault=%llu submitted=%llu backend_ok=%llu backend_refused=%llu backend_timeout=%llu pool_exhausted=%llu forward_param=%u exec_ok=%u sem_wedged=%llu\n", + "work_gr: req=%llu fallback=%llu ok=%llu refused=%llu timeout=%llu fault=%llu submitted=%llu backend_ok=%llu backend_refused=%llu backend_timeout=%llu pool_exhausted=%llu forward_param=%u exec_ok=%u sem_wedged=%llu stall_wait_max=%llu stall_retry=%llu wait_max_ms=%u nw_streak=%u wedge_quiet=%llu wedge_heals=%llu\n", gr_req, gr_fallback, gr_ok, gr_refused, gr_timeout, gr_fault, b_sub, b_ok, b_refused, b_timeout, b_pool, gsp_phys_gr_forward_enabled(), boot_stage.rmcall_gr_exec_ok, - READ_ONCE(gsp_phys_gr_sem_wedged)); + READ_ONCE(gsp_phys_gr_sem_wedged), + READ_ONCE(gsp_phys_gr_stall_wait_max), + READ_ONCE(gsp_phys_gr_stall_retry), + gsp_phys_gr_wait_max_ms_value(), + READ_ONCE(gsp_phys_gr_nw_streak), + READ_ONCE(gsp_phys_gr_wedge_quiet), + READ_ONCE(gsp_phys_gr_wedge_heals)); } { /* @@ -20697,6 +30547,91 @@ static const struct file_operations gsp_phys_guest_gpa_fops = { }; /* + * gsp-phys-shared-bit0 (write-only, mode 0200): development-harness assist + * for the NVIDIA X driver's per-client shared-data slot. + * + * Every direct GLX client receives a shared memory page from the X driver + * (nvidia_drv.so creates a memfd and passes it over a unix socket) whose + * first words are {u32 zero, u32 cookie, u32 state, u32 zero, u32 hClient, + * u32 hObject}; glcore's device init reads the state word and skips its own + * class-c46f channel creation while bit 0 of it is set. In this emulated + * environment the X driver publishes state=3 (bits 0+1) and direct clients + * therefore never allocate a channel and never render. The value is written + * by the X driver before the fd reaches the client, so it cannot be + * corrected from the client side. + * + * Input: "0x<hex GPA>" (4 KB aligned) naming the guest page that carries the + * slot. The GPA must fall inside a range registered through + * gsp-phys-guest-ram; it is translated with gsp_phys_gpa_to_hpa_vgpu() + * (first open mdev) and the state word is cleared of bit 0 through the + * kernel direct map (the translator validates pfn_valid, so __va() is a + * valid kernel mapping for the shmem-backed guest RAM). As a sanity guard + * the write is only performed when the page's +0x10 word looks like an RM + * client handle (0xc1d0xxxx / 0xc1e0xxxx) or the token carries a "force" + * suffix. Old and new values are always logged. + * + * This is a diagnostic aid for the development VM, matching the spirit of + * the existing guest-GPA pin; it is not part of the product behaviour. + */ +static ssize_t gsp_phys_shared_bit0_write(struct file *file, + const char __user *ubuf, size_t len, + loff_t *ppos) +{ + char buf[48]; + u64 gpa, hpa = 0; + u32 *word; + u32 h1032; + bool force = false; + int ret; + + if (len == 0 || len >= sizeof(buf)) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + buf[len] = '\0'; + + { + char *p = strim(buf); + char *tok = strsep(&p, " \t"); + + if (p && strstr(p, "force")) + force = true; + if (kstrtoull(tok, 0, &gpa) || !PAGE_ALIGNED(gpa)) { + pr_info("vgsp-boot: gsp-phys-shared-bit0: write ignored: expected \"0x<hex GPA>\" (4 KB aligned, optional 'force')\n"); + return -EINVAL; + } + } + + ret = gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, &hpa); + if (ret) { + pr_err("vgsp-boot: gsp-phys-shared-bit0: GPA 0x%llx translation failed (%d: no open mdev / GPA not inside a gsp-phys-guest-ram range / not RAM)\n", + gpa, ret); + return -ENOENT; + } + + word = (u32 *)__va(hpa + 8); + h1032 = *(u32 *)__va(hpa + 0x10); + if (!force && (h1032 >> 16) != 0xc1d0 && (h1032 >> 16) != 0xc1e0) { + pr_info("vgsp-boot: gsp-phys-shared-bit0: gpa=0x%llx word=0x%x but +0x10=0x%x does not look like an RM client handle; refusing (append 'force' to override)\n", + gpa, *word, h1032); + return -EINVAL; + } + + pr_info("vgsp-boot: gsp-phys-shared-bit0: gpa=0x%llx hpa=0x%llx hClient=0x%x word=0x%x -> 0x%x\n", + gpa, hpa, h1032, *word, *word & ~1u); + *word &= ~1u; + return len; +} + +static const struct file_operations gsp_phys_shared_bit0_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .write = gsp_phys_shared_bit0_write, + .llseek = no_llseek, +}; + + +/* * gsp-phys-guest-ram (write-only, mode 0200): configure a guest RAM backing * file that gsp_phys_gpa_to_hpa_vgpu() translates through (QEMU * memory-backend-file,...,share=yes). One write is @@ -20899,12 +30834,54 @@ static int gsp_phys_reg_probe_show(struct seq_file *m, void *unused) 0x0040070cu); fe_hww_esr = gsp_phys_reg_probe_line(m, bar0, "FE_HWW_ESR", 0x00404000u); + /* + * SESSION 153-CONTINUED-52: MME exception latches (see + * gsp_phys_mme_regs_log). + */ + (void)gsp_phys_reg_probe_line(m, bar0, "PGRAPH_PRI_MME_HWW_ESR", + 0x00404490u); + (void)gsp_phys_reg_probe_line(m, bar0, + "PGRAPH_PRI_MME_HWW_ESR_INFO", + 0x00404494u); + (void)gsp_phys_reg_probe_line(m, bar0, + "PGRAPH_PRI_MME_HWW_ESR_INFO2", + 0x00404498u); fe_chip_def = gsp_phys_reg_probe_line(m, bar0, "FE_CHIP_DEF_INFO", 0x00404030u); fe_go_idle = gsp_phys_reg_probe_line(m, bar0, "FE_GO_IDLE_INFO", 0x00404194u); sked_hww_esr = gsp_phys_reg_probe_line(m, bar0, "SKED_HWW_ESR", 0x00407020u); + /* + * SESSION 153-CONTINUED-11: the shader-program-header fault window. + * DS_HWW_ESR bit n = "the SPH with index n failed validation" (hwref: + * NV_PGRAPH_PRI_DS_HWW_ESR_SPH0_ERR..SPH23_ERR, ESR_2 for 24..30); + * the SPH_INFO registers snapshot the START address of each stage's + * faulting SPH, so a post-fault read names the exact VA the engine + * fetched. + */ + (void)gsp_phys_reg_probe_line(m, bar0, "DS_HWW_ESR", + 0x00405840u); + (void)gsp_phys_reg_probe_line(m, bar0, "DS_HWW_ESR_2", + 0x00405848u); + (void)gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_A", + 0x00405c1cu); + (void)gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_B", + 0x00405c20u); + { + u32 si; + + for (si = 0; si < 6u; si++) { + gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_STAGE_START", + 0x00405c00u + si * 4u); + gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_STAGE_START_UPPER", + 0x00405c24u + si * 4u); + } + } sked_activity = gsp_phys_reg_probe_line(m, bar0, "SKED_ACTIVITY", 0x00407054u); sked_cwd0 = gsp_phys_reg_probe_line(m, bar0, @@ -20913,6 +30890,30 @@ static int gsp_phys_reg_probe_show(struct seq_file *m, void *unused) sked_cwd1 = gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_1", 0x00407064u); + /* r425: the wedge diagnostics - the rest of the SKED CWD table, the + * per-8-group GR TPC masks (a zero compute mask would dispatch no + * TPCs), and the FE QMD-virtualization control for VEID 0. */ + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_2", + 0x00407068u); + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_3", + 0x0040706cu); + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_INFO_1", + 0x00407028u); + { + u32 gi; + + for (gi = 0; gi < 8u; gi++) { + gsp_phys_reg_probe_line(m, bar0, + "FE_TPCM_GR_TPC_MASK", + 0x0040a680u + gi * 4u); + gsp_phys_reg_probe_line(m, bar0, + "FE_LAST_GR_TPC_MASK", + 0x0040a6a0u + gi * 4u); + gsp_phys_reg_probe_line(m, bar0, + "FE_CWD_OWN_TPC_MASK", + 0x0040a6c0u + gi * 4u); + } + } fecs_irq = gsp_phys_reg_probe_line(m, bar0, "FECS_IRQSTAT", 0x00409008u); fecs_cpuctl = gsp_phys_reg_probe_line(m, bar0, "FECS_CPUCTL", @@ -21035,6 +31036,306 @@ static const struct file_operations gsp_phys_reg_probe_fops = { .release = single_release, }; +/* + * SESSION 153-CONTINUED-91: DS SPH fault spy. + * + * The DS_HWW_ESR SPH<n>_ERR bits (bit n = "the SPH with index n failed + * validation", hwref NV_PGRAPH_PRI_DS_HWW_ESR_SPH0_ERR..SPH23_ERR) and the + * per-stage SPH_INFO snapshot (STAGE0..STAGE5 START/UPPER at 0x405c00+si*4 / + * 0x405c24+si*4, PROGRAM_REGION_A/B at 0x405c1c/c20) are latched by the DS + * at fault time and consumed+cleared by the real GSP-RM exception handler + * before our relayed fault packet arrives (the fault-time register read in + * gsp_phys_fault_regs_log sees ESR=0x80000000 and all-zero stage starts + * one millisecond later). The RM's own diagnostic prints those STAGE + * registers (grErrorPrintDsSphInfo_GM200), so they name the exact SPH VAs + * the DS rejected - the missing datum behind the r1277+ Shader Program + * Header wall (CONT-90/91 could only move the fault SET by content, never + * identify the SPH entries). + * + * Reading this file runs a bounded tight poll of DS_HWW_ESR and snapshots + * the ESR pair, the six stage starts and the region pair on every sample + * that has any SPH bit set. Development-only, read-only, no state touched; + * run `cat /sys/kernel/debug/vgsp/gsp-phys-ds-spy` while the client draws. + * Duration via sema_ds_spy_ms (default 20 s). + */ +static u32 gsp_phys_ds_spy_ms = 20000; +module_param_named(sema_ds_spy_ms, gsp_phys_ds_spy_ms, uint, 0644); +MODULE_PARM_DESC(sema_ds_spy_ms, + "gsp-phys-ds-spy sampling duration in milliseconds (default 20000)"); + +#define GSP_PHYS_DS_ESR 0x00405840u +#define GSP_PHYS_DS_ESR_2 0x00405848u +#define GSP_PHYS_DS_REPORT_MASK 0x00405844u +#define GSP_PHYS_DS_REPORT_MASK_2 0x0040584cu +#define GSP_PHYS_DS_SPH_STAGE_LO 0x00405c00u +#define GSP_PHYS_DS_SPH_STAGE_UP 0x00405c24u +#define GSP_PHYS_DS_SPH_REGION_A 0x00405c1cu +#define GSP_PHYS_DS_SPH_REGION_B 0x00405c20u + +static int gsp_phys_ds_spy_show(struct seq_file *m, void *unused) +{ + void __iomem *bar0 = gsp_phys_gpu_bar0(); + u64 t0; + u32 hits = 0, samples = 0, maskholds = 0, mask0_last = 0, mask2_last = 0; + u32 ms = READ_ONCE(gsp_phys_ds_spy_ms); + + if (!bar0) + return -ENODEV; + if (!ms) + ms = 20000; + seq_printf(m, "# DS-SPY begin duration_ms=%u (poll DS_HWW_ESR 0x%08x)\n", + ms, GSP_PHYS_DS_ESR); + seq_printf(m, "# DS-SPY masks esr=%08x esr2=%08x mask=%08x mask2=%08x\n", + readl(bar0 + GSP_PHYS_DS_ESR), + readl(bar0 + GSP_PHYS_DS_ESR_2), + readl(bar0 + GSP_PHYS_DS_REPORT_MASK), + readl(bar0 + GSP_PHYS_DS_REPORT_MASK_2)); + t0 = ktime_get_ns(); + while (ktime_get_ns() - t0 < (u64)ms * 1000000ull) { + u32 esr = readl(bar0 + GSP_PHYS_DS_ESR); + u32 m0 = readl(bar0 + GSP_PHYS_DS_REPORT_MASK); + u32 m2 = readl(bar0 + GSP_PHYS_DS_REPORT_MASK_2); + + /* + * SESSION 153-CONTINUED-97: hold the HWW report masks + * cleared while sampling. A pending report POISONS the + * whole PRI block (every read returns 0xbadf1201), which + * hid the SPH stage snapshot at the validation instant + * (r1325 spy2: all 16+ hits badf1201, masks back at + * 00ffffff). The mask is (re)armed by the context load / + * the RM setup, not only by our replay path, so re-clear + * it every sample; the re-clear count is logged so the + * capture's provenance is visible. NOTE (r1326): poison + * still appeared on one pass despite 29 re-clears, so a + * clean latch is not guaranteed and the store-side read + * remains the authoritative fault-time datum. + * Development-only diagnostic tap. + */ + if (m0 || m2) { + iowrite32(0u, bar0 + GSP_PHYS_DS_REPORT_MASK); + iowrite32(0u, bar0 + GSP_PHYS_DS_REPORT_MASK_2); + maskholds++; + mask0_last = m0; + mask2_last = m2; + } + samples++; + if (esr & 0x7fffffffu) { + u32 esr2 = readl(bar0 + GSP_PHYS_DS_ESR_2); + u32 esr_a, s[6], u[6], ra, rb; + u32 si; + + for (si = 0; si < 6u; si++) { + s[si] = readl(bar0 + + GSP_PHYS_DS_SPH_STAGE_LO + + si * 4u); + u[si] = readl(bar0 + + GSP_PHYS_DS_SPH_STAGE_UP + + si * 4u); + } + ra = readl(bar0 + GSP_PHYS_DS_SPH_REGION_A); + rb = readl(bar0 + GSP_PHYS_DS_SPH_REGION_B); + esr_a = readl(bar0 + GSP_PHYS_DS_ESR); + hits++; + seq_printf(m, + "DS-SPY hit %u t=%lldus esr=%08x esr2=%08x esr_after=%08x region=%08x/%08x stage=%08x,%08x,%08x,%08x,%08x,%08x upper=%02x,%02x,%02x,%02x,%02x,%02x\n", + hits, div_u64(ktime_get_ns() - t0, 1000), + esr, esr2, esr_a, ra, rb, + s[0], s[1], s[2], s[3], s[4], s[5], + u[0] & 0xffu, u[1] & 0xffu, u[2] & 0xffu, + u[3] & 0xffu, u[4] & 0xffu, u[5] & 0xffu); + udelay(20); + } + } + seq_printf(m, "# DS-SPY end samples=%u hits=%u maskholds=%u last_mask=%08x/%08x\n", + samples, hits, maskholds, mask0_last, mask2_last); + return 0; +} + +static int gsp_phys_ds_spy_open(struct inode *inode, struct file *file) +{ + return single_open(file, gsp_phys_ds_spy_show, NULL); +} + +static const struct file_operations gsp_phys_ds_spy_fops = { + .owner = THIS_MODULE, + .open = gsp_phys_ds_spy_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* + * SESSION 153-CONTINUED-93: live PGRAPH register lab (debugfs + * gsp-phys-pgraph-rw). Read: a fixed snapshot of the DS/PD/FIFO registers + * the SPH investigation reads. Write: "0xADDR [0xVALUE]" (hex) inside the + * PGRAPH PRI window [0x400000,0x410000) - reads the old value, writes the + * new one (when given) and reads back, logging both. Diagnostic only; + * development bring-up tool. Writes outside the window are refused. + */ +static int gsp_phys_pgraph_show(struct seq_file *m, void *unused) +{ + void __iomem *bar0 = gsp_phys_gpu_bar0(); + + if (!bar0) + return -ENODEV; + seq_printf(m, "# PGRAPH register snapshot\n"); + seq_printf(m, "PD_CYA 0x00406020 %08x\n", readl(bar0 + 0x00406020u)); + seq_printf(m, "PD_HWW_ESR 0x00406018 %08x\n", readl(bar0 + 0x00406018u)); + seq_printf(m, "DS_HWW_ESR 0x00405840 %08x\n", readl(bar0 + 0x00405840u)); + seq_printf(m, "DS_HWW_ESR_2 0x00405848 %08x\n", readl(bar0 + 0x00405848u)); + seq_printf(m, "DS_REPORT_MASK 0x00405844 %08x\n", readl(bar0 + 0x00405844u)); + seq_printf(m, "DS_REPORT_MASK2 0x0040584c %08x\n", readl(bar0 + 0x0040584cu)); + seq_printf(m, "DS_SPH_REGION_A 0x00405c1c %08x\n", readl(bar0 + 0x00405c1cu)); + seq_printf(m, "DS_SPH_REGION_B 0x00405c20 %08x\n", readl(bar0 + 0x00405c20u)); + seq_printf(m, "DS_SPH_STAGE1 0x00405c04 %08x\n", readl(bar0 + 0x00405c04u)); + seq_printf(m, "DS_SPH_STAGE2 0x00405c08 %08x\n", readl(bar0 + 0x00405c08u)); + seq_printf(m, "DS_SPH_STAGE5 0x00405c14 %08x\n", readl(bar0 + 0x00405c14u)); + seq_printf(m, "PGRAPH_STATUS 0x00400700 %08x\n", readl(bar0 + 0x00400700u)); + seq_printf(m, "GR_FIFO_STATUS 0x00400504 %08x\n", readl(bar0 + 0x00400504u)); + return 0; +} + +static int gsp_phys_pgraph_open(struct inode *inode, struct file *file) +{ + return single_open(file, gsp_phys_pgraph_show, NULL); +} + +static ssize_t gsp_phys_pgraph_write(struct file *file, + const char __user *ubuf, size_t len, + loff_t *off) +{ + char buf[64]; + unsigned long long addr = 0, val = 0; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + int n, has_val; + u32 old, now; + + if (!bar0) + return -ENODEV; + if (len == 0 || len >= sizeof(buf)) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + buf[len] = '\0'; + n = sscanf(buf, "%llx %llx", &addr, &val); + if (n < 1) + return -EINVAL; + has_val = (n >= 2); + if (addr < 0x00400000ull || addr >= 0x00410000ull) + return -EINVAL; + old = readl(bar0 + addr); + if (has_val) { + writel((u32)val, bar0 + addr); + now = readl(bar0 + addr); + pr_warn("vgsp-boot: pgraph-rw 0x%08llx old=%08x new=%08x (wrote %08llx)\n", + addr, old, now, val); + } else { + pr_info("vgsp-boot: pgraph-rw 0x%08llx = %08x (read)\n", + addr, old); + } + return (ssize_t)len; +} + +static const struct file_operations gsp_phys_pgraph_fops = { + .owner = THIS_MODULE, + .open = gsp_phys_pgraph_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, + .write = gsp_phys_pgraph_write, +}; + +/* + * SESSION 153-CONTINUED-94: gsp-phys-init-push - the first large beef push + * captured at decode time (see svc_init_push_scan). Prints a header with + * the push address/length, the local-memory-run count (SET_SHADER_LOCAL_ + * MEMORY_A..D seen in ANY beef push so far; 0 = the client never emitted + * the download-state local-memory block) and the bytes as 16-byte rows. + */ +static int gsp_phys_init_push_show(struct seq_file *m, void *unused) +{ + const u8 *buf; + u32 len, h, lmem, off; + u64 va; + + buf = gsp_sema_init_cap_get(&len, &h, &va, &lmem); + if (!len) { + seq_puts(m, "vgsp-init-push: empty (no >=4 KiB beef push with non-zero head captured yet)\n"); + return 0; + } + seq_printf(m, "vgsp-init-push: h=0x%x va=0x%llx len=%u lmem_runs=%u\n", + h, va, len, lmem); + for (off = 0; off + 4u <= len; off += 16u) { + u32 k, n = min_t(u32, 4u, (len - off) / 4u); + + seq_printf(m, "%06x:", off); + for (k = 0; k < n; k++) { + u32 w; + + memcpy(&w, buf + off + k * 4u, 4); + seq_printf(m, " %08x", w); + } + seq_putc(m, '\n'); + } + return 0; +} + +static int gsp_phys_init_push_open(struct inode *inode, struct file *file) +{ + return single_open(file, gsp_phys_init_push_show, NULL); +} + +static const struct file_operations gsp_phys_init_push_fops = { + .owner = THIS_MODULE, + .open = gsp_phys_init_push_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* + * SESSION 153-CONTINUED-95: gsp-phys-ctx-dump - raw binary readback of the + * engine context buffer (see gsp_phys_ctx_readback_try). Empty until the + * first launch of the boot has run the CE readback. + */ +static ssize_t gsp_phys_ctx_dump_read(struct file *filp, char __user *ubuf, + size_t count, loff_t *ppos) +{ + const u8 *buf; + u32 len = 0; + + buf = gsp_phys_ctx_readback_get(&len); + if (!buf || !len) + return 0; + return simple_read_from_buffer(ubuf, count, ppos, buf, len); +} + +static const struct file_operations gsp_phys_ctx_dump_fops = { + .owner = THIS_MODULE, + .read = gsp_phys_ctx_dump_read, + .llseek = default_llseek, +}; + +/* + * SESSION 153-CONTINUED-96: gsp-phys-ctx-read - write-only re-arm for the + * one-shot engine-context readback. `echo 1 > .../gsp-phys-ctx-read` clears + * the tried flag; the next forwarded launch re-copies MAIN+PATCH with the + * current server state. The copy itself still runs only in the launch path. + */ +static ssize_t gsp_phys_ctx_read_write(struct file *filp, + const char __user *ubuf, + size_t count, loff_t *ppos) +{ + gsp_phys_ctx_readback_rearm(); + pr_info("vgsp-gr: CTX-READBACK re-armed; the next forwarded launch re-copies the engine ctx (MAIN+PATCH)\n"); + return count; +} + +static const struct file_operations gsp_phys_ctx_read_fops = { + .owner = THIS_MODULE, + .write = gsp_phys_ctx_read_write, +}; + int gsp_phys_boot_debugfs_init(void) { struct dentry *file; @@ -21127,6 +31428,20 @@ int gsp_phys_boot_debugfs_init(void) err); return (int)err; } + file = debugfs_create_file("gsp-phys-shared-bit0", 0200, + gsp_phys_debugfs_root, NULL, + &gsp_phys_shared_bit0_fops); + if (IS_ERR_OR_NULL(file)) { + long err = IS_ERR(file) ? PTR_ERR(file) : -ENODEV; + + /* + * The shared-slot bit0 assist is optional; the other + * triggers and the state snapshot stay registered. + */ + pr_warn("vgsp-boot: debugfs file gsp-phys-shared-bit0 creation failed: %ld (other triggers and state snapshot stay available)\n", + err); + return (int)err; + } file = debugfs_create_file("gsp-phys-guest-ram", 0200, gsp_phys_debugfs_root, NULL, &gsp_phys_guest_ram_fops); @@ -21183,7 +31498,57 @@ int gsp_phys_boot_debugfs_init(void) err); return (int)err; } - pr_info("vgsp-boot: debugfs files /sys/kernel/debug/vgsp/{gsp-phys-state read-only, gsp-phys-log-dump read-only, gsp-phys-status-drain read-only, gsp-phys-reg-probe read-only, gsp-phys-unload-rpc write-only, gsp-phys-rm-probe write-only, gsp-phys-rm-call write-only, gsp-phys-guest-gpa write-only, gsp-phys-guest-ram write-only} ready\n"); + /* + * SESSION 153-CONTINUED-91: the DS SPH fault spy (see the show fn). + * Diagnostic only; failure to create it must not fail the init. + */ + file = debugfs_create_file("gsp-phys-ds-spy", 0444, + gsp_phys_debugfs_root, NULL, + &gsp_phys_ds_spy_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-ds-spy creation failed (DS SPH fault spy unavailable)\n"); + /* + * SESSION 153-CONTINUED-93: live PGRAPH register lab (read snapshot + + * bounded writes inside the PGRAPH PRI window). Diagnostic only. + */ + file = debugfs_create_file("gsp-phys-pgraph-rw", 0644, + gsp_phys_debugfs_root, NULL, + &gsp_phys_pgraph_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-pgraph-rw creation failed (PGRAPH register lab unavailable)\n"); + /* + * SESSION 153-CONTINUED-94: the captured first large beef push (the + * GL client's channel-init stream) verbatim, so the local-memory + * census and any other method in it can be verified offline after + * the guest's transient pushbuffer is gone. Diagnostic only. + */ + file = debugfs_create_file("gsp-phys-init-push", 0444, + gsp_phys_debugfs_root, NULL, + &gsp_phys_init_push_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-init-push creation failed (init-push capture unavailable)\n"); + /* + * SESSION 153-CONTINUED-95: raw binary dump of the engine context + * buffer readback (64 KB from GSP_PHYS_RMCALL_VA_GR_CTX, taken once + * per boot at the first launch). 0444; empty until the first launch + * happens. Read with `dd if=...gsp-phys-ctx-dump of=/tmp/ctx.bin`. + */ + file = debugfs_create_file("gsp-phys-ctx-dump", 0444, + gsp_phys_debugfs_root, NULL, + &gsp_phys_ctx_dump_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-ctx-dump creation failed (engine ctx readback unavailable)\n"); + /* + * SESSION 153-CONTINUED-96: write-only re-arm for the readback (see + * gsp_phys_ctx_read_write). `echo 1 >` it to re-copy MAIN+PATCH at + * the next forwarded launch without a module reload. + */ + file = debugfs_create_file("gsp-phys-ctx-read", 0200, + gsp_phys_debugfs_root, NULL, + &gsp_phys_ctx_read_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-ctx-read creation failed (readback re-arm unavailable)\n"); + pr_info("vgsp-boot: debugfs files /sys/kernel/debug/vgsp/{gsp-phys-state read-only, gsp-phys-log-dump read-only, gsp-phys-status-drain read-only, gsp-phys-reg-probe read-only, gsp-phys-ds-spy read-only, gsp-phys-ctx-dump read-only, gsp-phys-ctx-read write-only, gsp-phys-unload-rpc write-only, gsp-phys-rm-probe write-only, gsp-phys-rm-call write-only, gsp-phys-guest-gpa write-only, gsp-phys-guest-ram write-only} ready\n"); return 0; } @@ -21537,8 +31902,8 @@ static_assert(offsetof(struct gsp_phys_rmcall_channel_params, process_id) == 296 "NV_CHANNEL_ALLOC_PARAMS.ProcessID must be at offset 296"); static_assert(offsetof(struct gsp_phys_rmcall_channel_params, tpc_config_id) == 360, "NV_CHANNEL_ALLOC_PARAMS.tpcConfigID must be at offset 360"); -static_assert(GSP_PHYS_RMCALL_BUF_COUNT == 81, - "the rm-call ladder registers exactly eighty-one DMA buffers (14 ladder + 16 single-page guest scratch PTs + 48 single-page extension scratch PTs + 3 golden9g VAS levels)"); +static_assert(GSP_PHYS_RMCALL_BUF_COUNT == 87, + "the rm-call ladder registers exactly eighty-seven DMA buffers (14 ladder + 16 single-page guest scratch PTs + 48 single-page extension scratch PTs + 3 golden9g VAS levels + 6 secondary CE lane buffers)"); static_assert((GSP_PHYS_RMCALL_HANDLE_MEM_BASE + (GSP_PHYS_RMCALL_BUF_COUNT - 1) * GSP_PHYS_RMCALL_HANDLE_MEM_STRIDE) < 0xcaf80000u, @@ -21552,11 +31917,26 @@ static_assert(((GSP_PHYS_RMCALL_VA_BASE >> 47) & 0x3) == 0 && ((GSP_PHYS_RMCALL_VA_BASE >> 21) & 0xff) == 0 && ((GSP_PHYS_RMCALL_VA_SEM >> 21) & 0xff) == 0, "the channel PDB VA window must sit in root/PD2/PD1/PD0 index 0 (GP10X 48:47/46:38/37:29/28:21 splits)"); -static_assert(((GSP_PHYS_RMCALL_VA_GPFIFO >> 12) & 0x1ff) != - ((GSP_PHYS_RMCALL_VA_PB >> 12) & 0x1ff) && - (((GSP_PHYS_RMCALL_VA_GPFIFO >> 12) & 0x1ff) + 4) == - ((GSP_PHYS_RMCALL_VA_SEM >> 12) & 0x1ff), - "the GPFIFO, pushbuffer and CE src/dst/semaphore VAs must take the five consecutive small-PT slots 0x100..0x104"); +static_assert(((GSP_PHYS_RMCALL_VA_GPFIFO >> 12) & 0x1ff) == 0x100 && + ((GSP_PHYS_RMCALL_VA_PB >> 12) & 0x1ff) == 0x101 && + ((GSP_PHYS_RMCALL_VA_PB2 >> 12) & 0x1ff) == 0x102 && + ((GSP_PHYS_RMCALL_VA_PB3 >> 12) & 0x1ff) == 0x103 && + ((GSP_PHYS_RMCALL_VA_PB4 >> 12) & 0x1ff) == 0x104, + "the GPFIFO + four pushbuffer pages must occupy the consecutive small-PT slots 0x100..0x104 for the engine's linear fetch walk (SESSION 153-CONTINUED-77: CE src/dst/sem moved off the walk to 0x109/0x10b/0x10c)"); +/* + * SESSION 153-CONTINUED-66: the two-page pushbuffer and the moved CE SRC / + * probe pages must stay inside the same 2 MB PD0 slot and the same 128 B PTE + * fetch line as the ladder, and must not collide with each other or with the + * QMD-window pair (0x107/0x108). + */ +static_assert(((GSP_PHYS_RMCALL_VA_PB2 >> 12) & 0x1ff) == + ((GSP_PHYS_RMCALL_VA_PB >> 12) & 0x1ff) + 1, + "the pushbuffer's second page must sit directly above the first (one contiguous GPU VA fetch)"); +static_assert(((GSP_PHYS_RMCALL_VA_SRC >> 21) & 0xff) == 0 && + ((GSP_PHYS_RMCALL_VA_GR_PROBE >> 21) & 0xff) == 0 && + ((GSP_PHYS_RMCALL_VA_SRC >> 12) & 0x1ff) == 0x109 && + ((GSP_PHYS_RMCALL_VA_GR_PROBE >> 12) & 0x1ff) == 0x10a, + "the moved CE SRC page and the warm-semaphore probe page must take the free slots 0x109/0x10a in the ladder's PTE line"); static_assert(((GSP_PHYS_RMCALL_VA_GUEST_PT_BASE >> 47) & 0x3) == 0 && ((GSP_PHYS_RMCALL_VA_GUEST_PT_BASE >> 38) & 0x1ff) == 0 && ((GSP_PHYS_RMCALL_VA_GUEST_PT_BASE >> 29) & 0x1ff) == 0 && @@ -21593,18 +31973,20 @@ static_assert(GSP_PHYS_RMCALL_VA_GR_CTX + static_assert(sizeof(struct gsp_phys_gr_page) == 24, "the GR forwarding page descriptor must stay {u64,u64,u32,u32} = 24 bytes"); static_assert(offsetof(struct gsp_phys_gr_inline, data) == 12 && - sizeof(struct gsp_phys_gr_inline) == 80, - "the GR forwarding inline record must stay {u64,u32,u8[64]} padded to 80 bytes"); + sizeof(struct gsp_phys_gr_inline) >= + 12 + GSP_PHYS_GR_INLINE_BYTES, + "the GR forwarding inline record must stay {u64,u32,u8[N]}"); static_assert((2u + 1u + GSP_PHYS_RMCALL_GR_QMD_INLINE_WORDS + 8u) <= GSP_PHYS_RMCALL_GR_METHOD_WORDS, "the GR launch method block (compute init + SET_OBJECT + inline-QMD run + PCAS + REPORT_SEMAPHORE) must fit the bound"); -static_assert(GSP_PHYS_RMCALL_GR_INIT_WORDS + GSP_PHYS_GR_RAW_WORDS + +static_assert(GSP_PHYS_RMCALL_GR_INIT_WORDS + + GSP_PHYS_RMCALL_GR_QMD_WINDOW_WORDS + GSP_PHYS_RMCALL_GR_TAIL_WORDS <= GSP_PHYS_RMCALL_GR_METHOD_WORDS, - "the verbatim GR launch block (compute init + raw tail + PCAS + REPORT_SEMAPHORE) must fit the bound"); + "the verbatim GR launch block minimum (compute init + window + PCAS + REPORT_SEMAPHORE) must fit the bound; the per-launch runtime check bounds raw_qmd_hdr for larger preludes"); static_assert(GSP_PHYS_RMCALL_GR_METHOD_WORDS * 4u <= - GSP_PHYS_RMCALL_COPY_SIZE, - "the GR launch method block must fit one module pushbuffer page"); + 4u * GSP_PHYS_RMCALL_COPY_SIZE, + "the GR launch method block may span the four module pushbuffer pages (SESSION 153-CONTINUED-77: prelude cap 3584 + largest chunked window)"); /* * GspSystemInfo wire ABI, pinned against the reference header compile diff --git a/kernel/gsp_phys_booter.h b/kernel/gsp_phys_booter.h index fb7b2a4..6f4d1ea 100644 --- a/kernel/gsp_phys_booter.h +++ b/kernel/gsp_phys_booter.h @@ -164,6 +164,16 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, void gsp_phys_backend_static_invalidate(void); /* + * Mode-35 KGR static-data snapshots (see gsp_phys_booter.c): the live GSP's + * replies to the real client's KGR query set, captured at backend boot and + * served byte-identically by the guest-side emulation. Returns 0 with + * *out_len set, -ENODATA when cmd was not captured for the current backend + * generation, -ENOSPC when the snapshot exceeds dst_len, -EAGAIN when the + * backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len); + +/* * Run one fn1+fn65 probe on a READY backend to fill the static snapshot * cache. Takes gsp_phys_rm_lock inside (via gsp_phys_rm_probe), so it must * run on a work item, never inline under a caller-held rm lock or on the mdev @@ -275,6 +285,17 @@ struct gsp_phys_backend_copy_result { u32 elapsed_ms; /* return-to-caller elapsed milliseconds */ int rc; /* duplicated function return code */ struct gsp_phys_fault_info fault; /* decoded engine fault, if any */ + /* + * r420b: the forwarded launch's QMD release0 was redirected to the + * module semaphore (gsp_phys_rmcall_gr_rel_mod=1, or an unresolved + * guest release page), so the ENGINE will never write the guest's own + * QMD release word. The caller must author it in software on every + * outcome (the same write the hold's success edge performs); without + * this a rel_mod=1 launch with no deferred release records is never + * completed on the guest word and the client hangs (the r414 saxpy + * f0 cur=5 stall). + */ + bool qmd_rel_redirected; }; int gsp_phys_backend_guest_ce_copy_ex(u64 src_hpa, u32 src_off, u64 dst_hpa, @@ -333,7 +354,13 @@ void gsp_phys_backend_relay_counters(u64 *requested, u64 *ok, u64 *verify_fail, */ #define GSP_PHYS_GR_MAX_PAGES 16u #define GSP_PHYS_GR_MAX_INLINE 8u -#define GSP_PHYS_GR_INLINE_BYTES 64u +/* + * SESSION 153-CONTINUED-88: 256 -> 2048 (must stay equal to the sema + * service's SVC_GR_INLINE_BYTES and the bridge's GSP_BRIDGE_INLINE_BYTES - + * static_asserts enforce it). The GL window's real program uploads are + * 1536 B / 1408 B and must be carried whole to the engine-visible page. + */ +#define GSP_PHYS_GR_INLINE_BYTES 2048u #define GSP_PHYS_GR_QMD_WORDS 64u /* * IQ2M sentinel: written to the inline-QMD destination's word 63 after the @@ -347,7 +374,26 @@ void gsp_phys_backend_relay_counters(u64 *requested, u64 *ok, u64 *verify_fail, #define GSP_PHYS_GR_QMD_SENTINEL 0xdeadbeefu #define GSP_PHYS_GR_QMD_SENTINEL_WORD 63u #define GSP_PHYS_GR_QMD_WORD_NONE 0xffffffffu -#define GSP_PHYS_GR_RAW_WORDS 256u +/* + * SESSION 153-CONTINUED-61: dedicated backend verdict for the 3D-window + * MISS-ABORT (gsp_phys_rmcall_gr_miss_abort): the launch was NOT submitted + * because a window/attr operand's page is still unresolved. The sema + * service uses this code to keep the captured candidate armed (re-arm) + * instead of consuming it with a software completion, so the forward path + * retries the same chunk once the pin demand resolves the page. -ENOMSG + * is returned by no other path on this interface. + */ +#define GSP_PHYS_GR_RC_MISSABORT (-ENOMSG) +/* + * r929: grown 256 -> 512 dwords so a whole GL 3D section push (the observed + * X server one-shot is 1524 B = 381 dwords) survives in the captured raw + * tail; the raw window is the source of the 3D method-stream replay. + * SESSION 153-CONTINUED-45: 512 -> 2048 dwords (8 KB) for the DIRECT GL + * client's 5016-7144 B render pushes (glcore); the raw window must carry + * the whole push for the section capture's `len == push_n*4` gate. Must + * equal SVC_GR_PUSH_WORDS (static_assert in gsp_sema_service.c). + */ +#define GSP_PHYS_GR_RAW_WORDS 2048u /* * B-2 (large-operand data path): the grown, boot-fixed GR data-bounce region. @@ -421,18 +467,64 @@ struct gsp_phys_gr_submit { u64 rel_hpa; /* guest-visible QMD release0 page, or 0 */ u32 rel_off; u32 rel_payload; + u64 rel_va; /* r378: device VA rel_hpa was resolved from, + * for write-target re-validation before the + * release word is zeroed/restored */ u64 sem_hpa; /* guest-visible channel release page, or 0 */ u32 sem_off; u32 sem_payload; u64 push_va; /* guest push identity (logging only) */ u32 push_len; u32 h_object; /* guest GR channel handle (logging only) */ + /* + * r929 (GL 3D path): the captured raw 3D method window - the + * SET_OBJECT C597 section from its class bind through the section + * close (state binds, LINE_LENGTH_IN + LAUNCH_DMA uploads, program/ + * tex binds; the prelude before the bind is excluded - its host-sem + * VAs belong to the client's VM and are unmapped in the replay + * channel, r929 v1 FAULT_PDE). When stream_n > 0 and + * gsp_phys_rmcall_gr_3d is on, the launch block replays the window + * verbatim after a SET_OBJECT TURING_A bind instead of the compute + * prelude + QMD; the engine then executes the REAL 3D methods. + * stream points into the caller's raw buffer; it is a separate + * pointer so a caller can pass any window. + */ + const u32 *stream; + u32 stream_n; /* dwords */ + /* + * SESSION 153-CONTINUED-87: the first program VA the captured + * window binds (SET_PIPELINE_PROGRAM_ADDRESS pair; 0 when none). + * The launch builder uses its 4K page as the program family for + * the client's missing stage binds (sema_gl_sph_bind). + */ + u64 prog_va; + /* + * SESSION 153-CONTINUED-87: every program bind the captured window + * itself carries (VA + pipeline slot). The launch builder re-emits + * them AHEAD of the window: the window's own draws (BEGIN 0x1618) + * precede its binds (measured word 93 vs 184), and on the replay + * channel's fresh context the early draws would validate an unbound + * stage and latch the Shader Program Header fault. + */ + u32 n_prog_binds; + u64 prog_bind_va[8]; + u32 prog_bind_slot[8]; }; /* True when gsp_phys_gr_forward is set and the ladder is a live GR channel * whose boot submission proved out (rmcall_gr_exec_ok). */ bool gsp_phys_backend_gr_ready(void); +/* + * r420: true exactly while the GR submit path sleeps in a WEDGE-QUIET hold + * (a deliberate, bounded wait for the sticky wedge to clear). The in-flight + * forwarded request is NOT done during it, so the sema service's completion + * deadlines (the 12 s GR release hold and the 3 s per-launch consumption + * bound) pause while this returns true instead of authoring the launch's + * release word under the hold (the r419 mid-boot stale-data escape). + */ +bool gsp_phys_gr_quiet_hold_active(void); + int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, struct gsp_phys_backend_copy_result *out); diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c index b1945ee..f1d72f7 100644 --- a/kernel/gsp_phys_gpu.c +++ b/kernel/gsp_phys_gpu.c @@ -617,6 +617,25 @@ int gsp_phys_rmcall_gr_prelude_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_prelude_value); /* + * r597: emit the SEM_EXECUTE ACQUIRE groups found in the captured guest + * prelude ahead of the rebuilt inline-QMD window (default 0). The verbatim + * prelude replay breaks the launch (qmd_rel=0 after an 8.9 s stall, r599), + * so the acquire cannot be taken by replaying the prelude; this gate takes + * only the wait method groups and leaves the proven window in place. + */ +static int gsp_phys_rmcall_gr_acq_param; +module_param_named(gsp_phys_rmcall_gr_acq, gsp_phys_rmcall_gr_acq_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_acq, + "Emit the captured prelude's SEM_EXECUTE ACQUIRE groups ahead of the rebuilt GR window (default 0)"); + +int gsp_phys_rmcall_gr_acq_value(void) +{ + return gsp_phys_rmcall_gr_acq_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_acq_value); + +/* * QMD virtualization control gate (default 0 after m26 falsified the * hypothesis: on Turing the context init value is 0, native CUDA never * writes 0x29c, and the nonzero enable alone does not schedule). When set, @@ -638,13 +657,20 @@ int gsp_phys_rmcall_gr_qmdvirt_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdvirt_value); /* - * Explicit-PCAS gate (default 1). The rebuilt launch carries the vendor's - * turingPushPcas(CU_PCAS_TYPE_COPY) pair (SEND_PCAS_A = QMD>>8 followed by - * SEND_SIGNALING_PCAS_B = INVALIDATE|SCHEDULE). The stock guest push does - * not emit these (it relies on the IQ2M window's implied copy PCAS); setting - * this to 0 emits the pure vendor window-only form for the A/B test. - */ -static int gsp_phys_rmcall_gr_pcas_param = 1; + * Explicit-PCAS gate (default 0 since r343 - the double-execution fix). + * The rebuilt launch used to carry the vendor's turingPushPcas + * (CU_PCAS_TYPE_COPY) pair (SEND_PCAS_A = QMD>>8 followed by + * SEND_SIGNALING_PCAS_B = INVALIDATE|SCHEDULE) ADDED to the inline-QMD run + * that the stock guest push already carries. The QMD's + * REQUIRE_SCHEDULING_PCAS bit is clear, so the inline run schedules by + * itself - the explicit pair was a SECOND scheduling edge and the same QMD + * was dispatched twice (launch_count: one atomicAdd +100 landed as +200; + * victory_reduce exactly 2x; scale/saxpy/memset2d doubled). The stock guest + * push has NO PCAS pair (captured push dumps); default 0 restores that + * single-edge form. 1 = re-add the pair (A/B debug only - with it on, the + * grid executes twice). + */ +static int gsp_phys_rmcall_gr_pcas_param; module_param_named(gsp_phys_rmcall_gr_pcas, gsp_phys_rmcall_gr_pcas_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_pcas, @@ -657,6 +683,29 @@ int gsp_phys_rmcall_gr_pcas_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_pcas_value); /* + * r434 CWD-slot-count path (gsp_phys_rmcall_gr_cwd_slot; default -1 = off). + * On Volta+ the SET_CWD_SLOT_COUNT method is gated by the falcon PRI bit + * NV_PGRAPH_PRI_SKED_DEBUG_1.ENABLE_CWD_SLOT_COUNT (the CUDA driver's + * turingLaunchSetCwdSlotCount: a masked PRI write via SET_MME_SHADOW_SCRATCH + * 0/1/2 + SET_FALCON04, then the method itself). When >= 0 the init preamble + * emits that PRI write (ENABLE_CWD_SLOT_COUNT=1) followed by + * SET_CWD_SLOT_COUNT with the given value, so the wedge's "grid truncated to + * ~4 CTAs" signature can be A/B'd against the CWD slot-count watermark. + * Read per launch; -1 restores the historical preamble byte-for-byte. + */ +static int gsp_phys_rmcall_gr_cwd_slot_param = -1; +module_param_named(gsp_phys_rmcall_gr_cwd_slot, + gsp_phys_rmcall_gr_cwd_slot_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_cwd_slot, + "r434: enable the CWD slot count PRI (SKED_DEBUG_1 bit11) and emit SET_CWD_SLOT_COUNT with this value in the GR init preamble (-1 = off, default)"); + +int gsp_phys_rmcall_gr_cwd_slot_value(void) +{ + return gsp_phys_rmcall_gr_cwd_slot_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_cwd_slot_value); + +/* * Two-step PCAS gate (default 0). With the explicit PCAS pair on, setting * this marks the QMD's REQUIRE_SCHEDULING_PCAS (w4 bit12) and emits the copy * PCAS (INVALIDATE only) followed by a schedule-only PCAS (SCHEDULE), the @@ -704,20 +753,30 @@ int gsp_phys_rmcall_gr_qmdwin_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdwin_value); /* - * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod; default 0). - * When set, a forwarded GR launch ignores the resolved guest release page and - * points QMD release0 at the module window-0 semaphore VA - * (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host VA. - * Discriminant for "engine completed but the shadow release write was lost" - * vs "engine never completed": with the module release, a completed QMD is - * always observable. Also makes the existing module-release fallback - * observable (it previously set no poll source, so it could never complete). - */ -static int gsp_phys_rmcall_gr_rel_mod_param; + * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod). + * + * DEFAULT 1 since r453. When set, a forwarded GR launch ignores the resolved + * guest release page and points QMD release0 at the module window-0 semaphore + * VA (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host + * VA; the GUEST-VISIBLE release word is authored by the host only on the + * completion edge - after the retire wait and after the bounce copy-out has + * shipped the engine's stores to the guest pages. Evidence (r453, the FNV + * change-detection fix on top): + * - with 0: the guest's release could be satisfied by the engine's own + * early write while the host copy-out was still pending, so a client's + * stream-ordered readback (streams2's c2, memcpy_roundtrip's memset + * phase, streams_events' chain) read PRE-launch content: 11/15 battery. + * - with 1: 13/15 battery (streams_events and memcpy_roundtrip flipped + * green), no stalls across the full battery, mail+saxpy clean. + * The r414 stall hole (a launch that never reached the hold edge left the + * guest word unauthored) is closed by the r420b per-outcome authoring path; + * one-line rollback: echo 0 > .../parameters/gsp_phys_rmcall_gr_rel_mod. + */ +static int gsp_phys_rmcall_gr_rel_mod_param = 1; module_param_named(gsp_phys_rmcall_gr_rel_mod, gsp_phys_rmcall_gr_rel_mod_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_rel_mod, - "Force the forwarded GR QMD release0 to the module semaphore (default 0; 1 = m64 engine-completion discriminant)"); + "Force the forwarded GR QMD release0 to the module semaphore; guest release authored only on the post-copy-out completion edge (default 1 since r453; 0 = historical engine-authored guest release)"); int gsp_phys_rmcall_gr_rel_mod_value(void) { @@ -974,6 +1033,261 @@ int gsp_phys_rmcall_gr_nowrite_retry_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_nowrite_retry_value); /* + * F2 fast re-arm (gsp_phys_gr_retry_fast; default 1). r198: the first + * forwarded-launch timeout costs 2x5 s before the fresh-entry ladder runs, + * yet the m155 extended re-arm (lane-enable replay + GPFIFO/TSG schedule + + * doorbell) plus one fresh ring entry recovers most of these launches. With + * this on, the first timeout immediately runs that re-arm + a fresh entry + * and re-polls with a short window; only if that still fails does the + * historical timeout ladder continue. Set to 0 for today's exact behavior. + */ +static int gsp_phys_gr_retry_fast_param = 1; +module_param_named(gsp_phys_gr_retry_fast, + gsp_phys_gr_retry_fast_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_retry_fast, + "On the first GR launch timeout, run the extended re-arm + a fresh ring entry immediately and re-poll briefly (default 1; 0 = legacy ladder timing)"); + +int gsp_phys_gr_retry_fast_value(void) +{ + return gsp_phys_gr_retry_fast_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_retry_fast_value); + +/* + * F2 first-poll window (gsp_phys_gr_first_poll_ms; default 2500, clamped to + * [1000, 30000]). With gsp_phys_gr_retry_fast=1 the FIRST poll window of a + * forwarded GR launch (before any re-arm) uses this instead of the fixed 5 s, + * so the fast re-arm path above starts earlier; with retry_fast=0 the launch + * keeps today's exact behavior and this knob is not read. The later ladder + * windows keep their own constants. + */ +static unsigned int gsp_phys_gr_first_poll_ms_param = 2500u; +module_param_named(gsp_phys_gr_first_poll_ms, + gsp_phys_gr_first_poll_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_first_poll_ms, + "First GR launch poll window in ms, clamped [1000,30000], read when gsp_phys_gr_retry_fast=1 (default 2500)"); + +unsigned int gsp_phys_gr_first_poll_ms_value(void) +{ + unsigned int v = gsp_phys_gr_first_poll_ms_param; + + if (v < 1000u) + v = 1000u; + if (v > 30000u) + v = 30000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_first_poll_ms_value); + +/* + * M91 icache coherence (gsp_phys_gr_icache_inv; default 1). The m91 class is + * the transient SM instruction-fetch lottery: the first attempt of a + * mid-sequence launch faults on SM garbage ("Illegal Instruction Encoding" / + * "Out Of Range Register") while a retry of the identical QMD and identical + * shadow content succeeds. The window-0 shadow pool reuses code VAs across + * launches (two 8-page banks alternate by submit epoch), so the bytes at a + * program VA change between launches, and every captured guest QMD carries + * the QMDV02_03 cache-control block with INVALIDATE_INSTRUCTION_CACHE (word 5 + * bit 30) CLEAR - the field the CUDA driver arms only under its + * forceIcacheInvalidate knob. An SM fetch line cached for launch N can then + * execute the previous kernel's bytes at the same VA in launch N+2. With this + * on, the forwarded QMD sets that bit (only when the QMD carries the driver's + * standard cache-control signature, bits 26..29 of word 5, so an unknown QMD + * revision is never touched), forcing a refetch at dispatch. 0 restores the + * exact pre-change QMD for a one-flip A/B. + */ +static int gsp_phys_gr_icache_inv_param = 1; +module_param_named(gsp_phys_gr_icache_inv, gsp_phys_gr_icache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_icache_inv, + "Set QMDV02_03 INVALIDATE_INSTRUCTION_CACHE (word5 bit30) on forwarded GR launches whose QMD has the driver cache-control signature (default 1; 0 = pre-m91 QMD words)"); + +int gsp_phys_gr_icache_inv_value(void) +{ + return gsp_phys_gr_icache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_icache_inv_value); + +/* + * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar; default 0). The + * captured guest QMDs carry RELEASE_MEMBAR_TYPE=FE_NONE (QMDV00_06 word 11 + * bit 14 clear, w11=0x44010000), so when the front-end completes a grid it + * writes the QMD release semaphore without first flushing its pending + * system-memory stores. The post-dispatch store-loss signature (the engine + * processes the QMD, writes the release, the grid completes and ZERO data + * pages are host-visible) is exactly what an unflushed write-back L2 looks + * like from the host: the release is host-visible while the data sits in the + * GPU's L2. Forcing RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR makes the release + * ordering guarantee the stores are in memory before the semaphore write, so + * the host's retire scan (and the copy-out) can never observe a completion + * whose data has not landed. 0 keeps the guest's captured value (the + * historical behaviour) for a one-flip A/B; read per launch. + */ +static int gsp_phys_gr_rel_membar_param; +module_param_named(gsp_phys_gr_rel_membar, gsp_phys_gr_rel_membar_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_rel_membar, + "r433: force QMD RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR (word11 bit14) on forwarded GR launches (default 0 = guest's captured FE_NONE)"); + +int gsp_phys_gr_rel_membar_value(void) +{ + return gsp_phys_gr_rel_membar_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_rel_membar_value); + +/* + * m179 no-write retry budget (gsp_phys_gr_nowrite_retries; default 3, clamped + * to [0, 4]). The m153 outcome retry loop re-executes a data-bearing launch + * whose retirement wait saw zero engine writes; it shared the timeout ladder's + * attempt counter, so a launch that had already spent attempts on the ladder + * (fast re-arm + fresh ring entries) reached the accept path with NO + * re-execution at all - the r200/r201/r202 memcpy_roundtrip / multikernel + * class (a lost launch silently accepted, client output all wrong). This + * budget is dedicated to the no-write loop and independent of the ladder; it + * is read once per submission. 0 keeps the accept-without-retry behavior + * (still logged loudly). The clamp keeps the worst-case host publishes per + * launch (initial + fast re-arm + 2 fresh entries + budget) within the + * 8-entry GPFIFO ring. + */ +static int gsp_phys_gr_nowrite_retries_param = 3; +module_param_named(gsp_phys_gr_nowrite_retries, + gsp_phys_gr_nowrite_retries_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_nowrite_retries, + "Dedicated re-execution budget for a data-bearing GR launch that wrote nothing tracked, clamped [0,4] (default 3)"); + +/* + * STALLFREE: absolute per-launch wall-clock budget (gsp_phys_gr_wait_max_ms; + * default 60000, clamped [10000, 600000]; 0 disables the extra cap and the + * built-in ladder bounds stand alone). The GR submit ladder and its retirement + * (no-write) retries are individually bounded, but their SUM grows with every + * re-arm/re-publish: a launch whose engine ack never arrives could hold the + * single-inflight work item for tens of seconds while the guest client waits + * on its QMD completion word. When this budget is crossed the submit stops its + * ladder immediately (no further re-arm/publish/retry), logs + * 'vgsp-gr: GR-STALL-wait-max ... action=fail' and returns -ETIMEDOUT; the + * sema service's failure edge then authors the QMD completion release in + * software, so the client always completes. A green launch (~0.2-11 s end to + * end, including the fast-re-arm recovery) never approaches the default, so + * the default does not change green behavior. + */ +static unsigned int gsp_phys_gr_wait_max_ms_param = 60000u; +module_param_named(gsp_phys_gr_wait_max_ms, + gsp_phys_gr_wait_max_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wait_max_ms, + "Absolute per-launch GR submit budget in ms, clamped [10000,600000]; 0 = only the built-in per-ladder bounds (default 60000)"); + +unsigned int gsp_phys_gr_wait_max_ms_value(void) +{ + unsigned int v = gsp_phys_gr_wait_max_ms_param; + + if (!v) + return 0; + if (v < 10000u) + v = 10000u; + if (v > 600000u) + v = 600000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wait_max_ms_value); + +int gsp_phys_gr_nowrite_retries_value(void) +{ + int v = gsp_phys_gr_nowrite_retries_param; + + if (v < 0) + v = 0; + if (v > 4) + v = 4; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_nowrite_retries_value); + +/* + * r445 wedge hard re-arm (gsp_phys_gr_wedge_rearm; default 0 = off). When the + * outcome-retry's warm GR semaphore probe proves the engine wedged for the + * context (the proven-sticky post-dispatch no-store state), this knob makes + * the submit attempt ONE bounded recovery instead of breaking immediately: + * a TSG PREEMPT(bWait=1) forces the FECS to save/unload the channel's + * context, then the mode-35 lane-enable tail + GPFIFO schedules are re-issued + * (the m155 extended re-arm) and the semaphore probe is repeated. A pass + * means the lane revived for this launch and the normal retry ladder + * continues; a fail leaves the historical WEDGED break. Read once per wedge + * episode. Off by default for A/B. + */ +static int gsp_phys_gr_wedge_rearm_param; +module_param_named(gsp_phys_gr_wedge_rearm, gsp_phys_gr_wedge_rearm_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wedge_rearm, + "r445: on the proven engine-wedged sem probe, attempt a bounded preempt+resched hard re-arm before giving up (default 0=off)"); + +int gsp_phys_gr_wedge_rearm_value(void) +{ + return gsp_phys_gr_wedge_rearm_param ? 1 : 0; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_rearm_value); + +/* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default 0). + * The forwarded QMD's cache-control block (QMDV02_03 word 5) carries the + * per-cache invalidate bits; the captured guest QMDs always set bits 26..29 + * (texture header/sampler/data + shader DATA) and the M91 fix adds bit 30 + * (INSTRUCTION), but bit 31 (SHADER CONSTANT - where a kernel's launch + * parameters live) is never set. The forwarded launch's param page is + * rewritten in place between launches (the two-bank win0 pool), so a stale + * constant-cache line can serve the PREVIOUS launch's parameters - the + * kernel then computes with the previous launch's pointers/sizes and the + * current launch's tracked output pages stay untouched ("no stores"). + * Mode: 0 = off, 1 = set bit 31 only, 2 = force the full 0xff000000 + * invalidate set. Read at submission time, so it can be A/B'd at runtime. + */ +static int gsp_phys_gr_ccache_inv_param; +module_param_named(gsp_phys_gr_ccache_inv, gsp_phys_gr_ccache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_ccache_inv, + "r449: forwarded-QMD shader-constant-cache invalidate: 0=off, 1=bit31, 2=all invalidate bits (default 0)"); + +unsigned int gsp_phys_gr_ccache_inv_value(void) +{ + if (gsp_phys_gr_ccache_inv_param < 0) + return 0; + if (gsp_phys_gr_ccache_inv_param > 2) + return 2; + return (unsigned int)gsp_phys_gr_ccache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_ccache_inv_value); + +/* + * r410 wedge quiet window (gsp_phys_gr_wedge_quiet_ms; default 40000, clamped + * [0,120000]; 0 disables). The sticky per-lane no-execute wedge (100% of + * launches no-write while kept busy) is proven to CLEAR when the lane stays + * quiet for tens of seconds (r409: a ~37 s client gap healed a wedged run; + * r416: 58 consecutive no-write launches during a 126 s client run, first + * launches green again after the lane sat ~52 s idle; r417: a 20 s host hold + * did not heal, a 40 s hold did - "WEDGE-QUIET healed: engine wrote 1 + * page(s) after the 40000 ms hold"). Immediate retries and + * re-arms do not heal it (r409/r154). So when a zero-write outcome follows a + * previous accepted zero-write launch, the retry loop first holds this many ms + * with NO submissions to the engine, then re-arms (lane enable + schedules) + * and re-publishes once. The value doubles per further quiet retry. The + * quiet window is recovery-only: a launch that wrote anything never enters it. + */ +static unsigned int gsp_phys_gr_wedge_quiet_ms_param = 40000u; +module_param_named(gsp_phys_gr_wedge_quiet_ms, + gsp_phys_gr_wedge_quiet_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wedge_quiet_ms, + "Quiet-window ms before retrying a launch on a lane whose previous launch also no-wrote (0=disable, default 40000; r417: a 20 s hold did not heal, 40 s healed)"); + +unsigned int gsp_phys_gr_wedge_quiet_ms_value(void) +{ + unsigned int v = gsp_phys_gr_wedge_quiet_ms_param; + + if (v > 120000u) + v = 120000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_quiet_ms_value); + +/* * m69 window-0 shadow pool (gsp_phys_rmcall_gr_shadow_win0; default 0). When * set, the forwarded GR launch copies the QMD's program/param pages into the * window-0 shadow pool (module pages at VAs 0x108000..0x10f000, PTEs planted @@ -1640,6 +1954,41 @@ bool gsp_phys_guest_ram_active(void) EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_active); /* + * Highest GPA any registered guest-RAM range covers. Conservative global + * bound (max across the legacy and every per-mdev record) for consumers that + * only need to reject translations pointing past RAM - e.g. the sema push + * read's PA ceiling (r1127: a stale VAS walk answered 0x23921d000 = 8.89 GB + * for the beef setup push; with the 8/10 GB guest layout the answer is + * inside the registered ranges and reads padding zeros, so no bound alone + * fixes it - the repeated-zero invalidation does - but any answer ABOVE + * this top can never read content). 0 when nothing is registered. + */ +u64 gsp_phys_guest_ram_top(void) +{ + u64 top = 0; + unsigned int i, j; + + mutex_lock(&guest_ram_lock); + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS + 1; i++) { + const struct gsp_phys_guest_ram_rec *rec = + (i == 0) ? &guest_ram_legacy : &guest_ram_mdevs[i - 1]; + + if (!rec->file) + continue; + for (j = 0; j < rec->ranges_n; j++) { + u64 end = rec->ranges[j].gpa_base + + rec->ranges[j].size; + + if (end > top) + top = end; + } + } + mutex_unlock(&guest_ram_lock); + return top; +} +EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_top); + +/* * Lock-safe one-line snapshot for the gsp-phys-state dump: active flag, * path, range count and every range (gpa/offset/size). With only the legacy * record configured (the one-mdev campaign) the output is byte-identical to diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h index a89493b..b8d92cd 100644 --- a/kernel/gsp_phys_gpu.h +++ b/kernel/gsp_phys_gpu.h @@ -295,6 +295,7 @@ int gsp_phys_rmcall_gr_promote_value(void); * init+window+PCAS+sem shape); 1 replays the verbatim prelude. */ int gsp_phys_rmcall_gr_prelude_value(void); +int gsp_phys_rmcall_gr_acq_value(void); /* * gsp_phys_rmcall_gr_qmdvirt (default 1): emit @@ -339,6 +340,14 @@ int gsp_phys_rmcall_faultbuf_sys_value(void); * (proven small-PT line) instead of the window-1 scratch range (default 0). */ int gsp_phys_rmcall_gr_shadow_win0_value(void); +/* + * SESSION 153-CONTINUED-87: emit the client's missing TESS_CONTROL (j=2) + * and PIXEL (j=5) pipeline program binds ahead of replayed 3D windows + * (sema_gl_sph_bind, default 1). The sema service uses the same value to + * author the matching program images at those family VAs. + */ +bool gsp_phys_rmcall_gr_sph_bind_value(void); + /* m91: keep the forwarded QMD pointer pairs at the guest VAs (default 0). */ int gsp_phys_rmcall_gr_ptr_guest_value(void); @@ -375,6 +384,52 @@ int gsp_phys_rmcall_gr_skedrec_value(void); * wrong-place engine writes (default 1 = enabled). */ int gsp_phys_rmcall_gr_nowrite_retry_value(void); +/* F2: on the first GR launch timeout, run the extended re-arm + a fresh ring + * entry immediately and re-poll briefly (default 1; 0 = legacy ladder). */ +int gsp_phys_gr_retry_fast_value(void); + +/* F2: first GR launch poll window in ms, clamped [1000,30000], read while + * gsp_phys_gr_retry_fast=1 (default 2500). */ +unsigned int gsp_phys_gr_first_poll_ms_value(void); + +/* M91: set QMDV02_03 INVALIDATE_INSTRUCTION_CACHE (word5 bit30) on forwarded + * GR launches whose QMD carries the driver cache-control signature + * (default 1; 0 = pre-m91 QMD words). */ +int gsp_phys_gr_icache_inv_value(void); + +/* r433: force QMD RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR (QMDV00_06 word11 bit14) + * on forwarded GR launches so the engine flushes pending sysmem stores + * before writing the release semaphore (default 0 = guest's FE_NONE). */ +int gsp_phys_gr_rel_membar_value(void); + +/* r434: enable the CWD slot count PRI (SKED_DEBUG_1 bit11) and emit + * SET_CWD_SLOT_COUNT with this value in the GR init preamble + * (-1 = off, default). */ +int gsp_phys_rmcall_gr_cwd_slot_value(void); + +/* m179: dedicated re-execution budget for a data-bearing GR launch whose + * retirement wait saw zero engine writes, independent of the timeout ladder's + * attempt counter (default 3, clamped [0,4]). */ +int gsp_phys_gr_nowrite_retries_value(void); +int gsp_phys_gr_wedge_rearm_value(void); +unsigned int gsp_phys_gr_ccache_inv_value(void); + +/* STALLFREE: absolute per-launch GR submit wall-clock budget in ms, clamped + * [10000,600000]; 0 = only the built-in per-ladder bounds (default 60000). + * On expiry the submit fails out immediately with a GR-STALL-wait-max line + * and the sema service authors the QMD completion release in software. */ +unsigned int gsp_phys_gr_wait_max_ms_value(void); + +/* r410 wedge quiet window: when a launch's zero-write outcome follows a + * previous accepted zero-write launch (the sticky per-lane no-execute wedge), + * hold the retry for this many ms with NO submissions to the engine before + * re-publishing. The wedge is proven to clear when the lane stays quiet for + * tens of seconds (r409 ~37 s, r416 ~52 s); immediate retries keep it alive. + * Base value for the first quiet retry, doubled per further attempt, clamped + * [0,120000]; 0 disables the quiet path (retries run back-to-back as before). + * Default 20000. */ +unsigned int gsp_phys_gr_wedge_quiet_ms_value(void); + /* m73: use host-visible sysmem chunks for V7 instance/RAMFC (m62 shape). */ int gsp_phys_rmcall_gr_inst_sys_value(void); @@ -469,6 +524,15 @@ bool gsp_phys_guest_ram_active(void); void gsp_phys_guest_ram_state(char *buf, size_t len); /* + * Highest GPA any registered guest-RAM range covers (max of gpa_base+size + * across every active record; 0 when nothing is registered). A GPA at or + * above this can never hold guest content - the backing file's padding + * reads as zeros - so translation consumers use it to reject walk/pin + * answers that point past the registered RAM. + */ +u64 gsp_phys_guest_ram_top(void); + +/* * Open-mdev helpers for the per-mdev debugfs selector and the single-guest * guards: count the mdevs whose VFIO device is open, or resolve one open * mdev by its device name (the mdev UUID). The returned device is not diff --git a/kernel/gsp_rm_emulation.c b/kernel/gsp_rm_emulation.c index 2ffc8ac..d0f9e61 100644 --- a/kernel/gsp_rm_emulation.c +++ b/kernel/gsp_rm_emulation.c @@ -452,8 +452,26 @@ static u32 gsp_rm_build_record(struct gsp_rm_emu *rm, u8 *buf, u32 buf_size, /* * Write a status queue record (1+ elements) and advance the writePtr. */ -static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, - u32 sequence, u32 result, const void *data, u32 data_len) +/* + * m171: the event burst budget is a rate over a short time window, not a + * monotonic session counter. Without the time reset a flood left the counter + * pinned at the cap and silently dropped the completion post of a client + * that only started waiting later - a minimal cudaEventSynchronize repro sat + * forever while its registrations were valid and 'completion hint rc=-22' + * repeated. Channel-completion posts get headroom over hint re-posts: a hint + * can be retried a moment later, a completion wake-up cannot. + */ +#define GSPRM_EVENT_BURST_CRIT_MAX 48 +#define GSPRM_EVENT_BURST_WINDOW (HZ) +/* Strictly shorter than the service's 0.5 s hint cadence: an equal window + * races the boundary and every later hint skips forever (observed r181: + * 'completion hint rc=-2' forever after one success and the stream client + * still stalled - the late-waiter wake-up never fired again). */ +#define GSPRM_EVENT_HINT_DEDUP (HZ / 4) + +static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, + u32 sequence, u32 result, const void *data, + u32 data_len, bool critical) { struct gsp_rm_emu *rm = ctx->rm_emu; u8 *buf; @@ -489,13 +507,22 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, * newer record and the client aborts with "Bad sequence number". */ if (function == GSPRM_RPC_EVENT_POST_EVENT) { - if (rm->event_burst >= GSPRM_EVENT_BURST_MAX) { + u32 cap = critical ? GSPRM_EVENT_BURST_CRIT_MAX + : GSPRM_EVENT_BURST_MAX; + + /* m171: expire the budget by wall time so a long-ago + * flood can never starve a fresh completion post. */ + if (time_after(jiffies, rm->event_burst_at + + GSPRM_EVENT_BURST_WINDOW)) + rm->event_burst = 0; + rm->event_burst_at = jiffies; + if (rm->event_burst >= cap) { rm->events_dropped++; if (rm->events_dropped <= 8 || (rm->events_dropped % 1000) == 0) nv_vgpu_info(ctx->vgpu, - "vgsp-rm: event burst capped (%u), dropped total=%llu", - GSPRM_EVENT_BURST_MAX, + "vgsp-rm: event burst capped (%u/%u), dropped total=%llu", + rm->event_burst, cap, rm->events_dropped); mutex_unlock(&rm->stat_lock); kfree(buf); @@ -504,6 +531,7 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, rm->event_burst++; } else { rm->event_burst = 0; + rm->event_burst_at = jiffies; } record_size = gsp_rm_build_record(rm, buf, buf_size, function, @@ -524,6 +552,106 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, return -EOVERFLOW; } + /* + * fail4: free-space discipline. The guest's consumption cursor for + * the status queue (msgqRxHeader.readPtr, at stat+rxHdrOff under the + * queue's SWAP_RX layout - same location msgqTxGetFreeSpace polls) + * gates this write. Overwriting unread records mixes entries in the + * guest's RPC stream: under a completion burst the guest then runs + * garbage RM callbacks (rm_run_rc_callback+0x0) and dies. Events are + * wake-ups and drop freely; a reply waits briefly for the ring to + * drain and is dropped (never overwritten) if it stays full. + */ + { + u32 peer_read = 0; + bool have_peer = false; + int w; + + for (w = 0; w < 51; w++) { + u32 used, space, rxoff = 0; + u8 shdr[32]; + + /* + * r332 SWAP_RX cursor fix: the guest's consumption + * cursor for the STATUS queue lives in the COMMAND + * queue's RX header, not in the status queue's own + * rxHdrOff. Under MSGQ_FLAGS_SWAP_RX (both sides set + * it: flags=1 in both headers) the driver's + * msgqRxMarkConsumed writes its status readPtr to + * pOurRxHdr->readPtr, and the driver's pOurRxHdr is + * the COMMAND-queue buffer's RX header (msgq.c + * msgqRxLink swap branch: pReadOutgoing = + * &pOurRxHdr->readPtr with pOurRxHdr from + * msgqTxCreate's pBackingStore = pCommandQueue). The + * status queue's own +rxHdrOff is the mirror we + * publish: OUR command-queue read position + * (written below after every consumed command + * record). Reading that mirror here made the + * free-space math track our own writes: the ring was + * declared "full" with the guest mid-drain, replies + * were written into it anyway, unread records were + * overwritten, and the guest RM read a newer record + * ("Bad sequence number. Expected N got N+63") - the + * first-client cudaMalloc then fails with "device + * busy or unavailable" (the boot lottery) and every + * later RPC fails until a fresh boot. + */ + if (gsp_rm_read_buf(ctx, rm->cmd_off, shdr, sizeof(shdr))) + break; + memcpy(&rxoff, shdr + 24, 4); + if (!rxoff || rxoff >= 0x1000) + break; + if (gsp_rm_read_buf(ctx, rm->cmd_off + rxoff, &peer_read, 4) || + peer_read >= rm->stat_msgcount) + break; /* no usable cursor: blind write */ + have_peer = true; + used = (rm->stat_write + rm->stat_msgcount - + peer_read) % rm->stat_msgcount; + space = rm->stat_msgcount - used - 1u; + if (elem_count <= space) + break; + /* + * Only event wake-ups drop freely; a reply waits + * (bounded) and then proceeds - dropping replies + * breaks the guest's RM init handshake (observed: + * two fn=0x1 replies dropped at boot -> the device + * never comes up). + */ + if (function == GSPRM_RPC_EVENT_POST_EVENT || w == 50) + break; + mutex_unlock(&rm->stat_lock); + msleep(1); + mutex_lock(&rm->stat_lock); + } + if (have_peer) { + u32 used = (rm->stat_write + rm->stat_msgcount - + peer_read) % rm->stat_msgcount; + u32 space = rm->stat_msgcount - used - 1u; + + if (elem_count > space && + function == GSPRM_RPC_EVENT_POST_EVENT) { + rm->events_dropped++; + if (rm->events_dropped <= 8 || + (rm->events_dropped % 1000) == 0) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: status ring full (free=%u need=%u peer=%u write=%u mc=%u); event dropped, total=%llu", + space, elem_count, + peer_read, rm->stat_write, + rm->stat_msgcount, + rm->events_dropped); + kfree(buf); + mutex_unlock(&rm->stat_lock); + return -EAGAIN; + } + if (elem_count > space) + nv_vgpu_warn(ctx->vgpu, + "vgsp-rm: status ring full for reply fn=0x%x (free=%u need=%u peer=%u write=%u mc=%u); writing anyway after wait", + function, space, elem_count, + peer_read, rm->stat_write, + rm->stat_msgcount); + } + } + /* Write all elements to consecutive queue slots. Page-aware: the * queue buffer is non-contiguous, so translate per page. */ slot = rm->stat_write % rm->stat_msgcount; @@ -589,6 +717,114 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, return 0; } +/* Non-event sends: the historical entry point, budget untouched. */ +static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, + u32 sequence, u32 result, const void *data, u32 data_len) +{ + return gsp_rm_send_flags(ctx, function, sequence, result, data, + data_len, false); +} + +/* + * r576 DIAG: event-post payload witness (gsp_rm_ev_log=N budget per boot). + * Every EVENT_POST_EVENT record actually sent is logged with its full + * payload and send result: when a client parks waiting for a completion + * wake-up that never arrives, this shows whether the post was generated, + * with which hClient/hEvent, and whether the status-queue write accepted it + * (rc=0) or dropped it (burst cap / ring full => rc<0). The companion + * sentinel fires only when post_all_events found valid events but could not + * post any of them while none were dedup-skipped - the "client waits, host + * thinks it posted" anomaly. 0 (default) = off. + */ +unsigned gsp_rm_ev_log_max; +static unsigned gsp_rm_ev_log; + +static void gsp_rm_evlog_post(struct nv_vgpu_device *vgpu, + const struct gsp_rm_emu *rm, u32 idx, + const char *src, int rc) +{ + if (!gsp_rm_ev_log_max || gsp_rm_ev_log >= gsp_rm_ev_log_max) + return; + gsp_rm_ev_log++; + nv_vgpu_info(vgpu, + "vgsp-rm: EVPOST %s ev[%u] cl=0x%x ev=0x%x nidx=0x%x data=0x%x rc=%d", + src, idx, rm->events[idx].h_client, + rm->events[idx].h_event, rm->events[idx].notify_index, + rm->events[idx].data, rc); +} + +/* + * r577 experiment knobs (runtime-writable; default 0 = legacy behaviour). + * The r576 live experiment showed the guest consumes our POST_EVENT records + * (rmstate cursors match) yet never enters nv_post_event (clean single-probe + * window, forced rmpost rounds -> entries-written 0): the guest RM drops + * them somewhere between the status-queue read and osNotifyEvent. These + * knobs let one boot A/B the record contents without rebuilding: + * gsp_rm_ev_seq - when nonzero, event records carry the status-queue + * sequence as the RPC-level sequence (legacy: 0) + * gsp_rm_ev_nlist - when nonzero, bNotifyList (payload byte 28) is set, + * selecting the driver's notifier-list dispatch path + * gsp_rm_ev_cl - when nonzero, overrides the payload hClient (e.g. to + * the kernel RM client) for the CliGetEventInfo lookup + */ +unsigned gsp_rm_ev_seq; +unsigned gsp_rm_ev_nlist; +unsigned gsp_rm_ev_cl; +/* + * r578: reachability probe for the guest-RM event drop. When nonzero, event + * records carry this RPC function instead of POST_EVENT (0x1003). Posting a + * function with an unmistakable guest-side side effect (e.g. + * NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT 0x100C with an unsupported ucodeEngDesc + * trips NV_ASSERT_FAILED in _kgspRpcUcodeLibosPrint, or + * NV_VGPU_MSG_EVENT_OS_ERROR_LOG 0x1006 logs an errString) proves whether + * the guest ever DISPATCHES our status-queue event records: visible side + * effect => the dispatch works and POST_EVENT is dropped inside + * _kgspRpcPostEvent; no side effect => the records are consumed without + * dispatch. + */ +unsigned gsp_rm_ev_func; + +static void gsp_rm_fill_ev_payload(const struct gsp_rm_emu *rm, u32 i, + u8 *payload) +{ + u32 h_client = rm->events[i].h_client; + /* + * r579 THE EVENT-DROP ROOT CAUSE: NV0005_ALLOC_PARAMETERS.notifyIndex + * is OVERLOADED (cl0005.h): INDEX=bits[15:0], SUBDEVICE=bits[23:16], + * FLAGS=bits[31:24]. The registration word (e.g. 0x3c00000c) must not + * be posted verbatim: the guest's _kgspRpcPostEvent hands the payload + * notifyIndex straight to gpuNotifySubDeviceEvent_IMPL, which indexes + * its notifier array by it - posting 0x3c00000c produced a wild + * pointer (base + 0x3c00000c*4 + 0x1508) and a kernel Oops + * ("BUG: unable to handle page fault ... gpuNotifySubDeviceEvent_IMPL + * +0x21f", RAX=0x3c00000c) on every dispatched event, which is why + * nv_post_event never fired and clients waiting on event wakes + * wedged. Post the INDEX field only. + */ + u32 notify_index = rm->events[i].notify_index & 0xffffu; + + memset(payload, 0, 32); + if (gsp_rm_ev_cl) + h_client = gsp_rm_ev_cl; + memcpy(payload + 0, &h_client, 4); + memcpy(payload + 4, &rm->events[i].h_event, 4); + memcpy(payload + 8, ¬ify_index, 4); + memcpy(payload + 12, &rm->events[i].data, 4); + /* info16@16, status@20, eventDataSize@24 = 0 */ + if (gsp_rm_ev_nlist) + payload[28] = 1; /* bNotifyList */ +} + +static u32 gsp_rm_ev_sequence(const struct gsp_rm_emu *rm) +{ + return gsp_rm_ev_seq ? rm->stat_seq : 0; +} + +static u32 gsp_rm_ev_function(void) +{ + return gsp_rm_ev_func ? gsp_rm_ev_func : GSPRM_RPC_EVENT_POST_EVENT; +} + /* * Post the GPU event that tells the guest's CPU-side RM a channel's * submitted work is complete. A real GPU raises an interrupt, the host GSP @@ -621,15 +857,18 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) if (!rm->events[i].valid || rm->events[i].h_src != h_src) continue; - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[i].h_client, 4); - memcpy(payload + 4, &rm->events[i].h_event, 4); - memcpy(payload + 8, &rm->events[i].notify_index, 4); - memcpy(payload + 12, &rm->events[i].data, 4); - /* info16@16, status@20, eventDataSize@24, bNotifyList@28 = 0 */ - if (gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload))) - continue; + gsp_rm_fill_ev_payload(rm, i, payload); + { + int rc = gsp_rm_send_flags(ctx, gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), 0, + payload, sizeof(payload), + true); + + gsp_rm_evlog_post(vgpu, rm, i, "chan", rc); + if (rc) + continue; + } + rm->events[i].last_post = jiffies; posted++; } @@ -643,14 +882,34 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) for (i = 0; i < rm->events_n; i++) { if (!rm->events[i].valid) continue; - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[i].h_client, 4); - memcpy(payload + 4, &rm->events[i].h_event, 4); - memcpy(payload + 8, &rm->events[i].notify_index, 4); - memcpy(payload + 12, &rm->events[i].data, 4); - if (gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload))) + /* + * r580: apply the hint dedup to the per-completion + * shotgun. Every completion posts ALL registered + * events (the ABI gives no source association), so a + * completion burst burns the burst budget 8 records + * per completion and real wake-ups get dropped + * (r579 boot: events_dropped=724). A redundant + * re-post within 250 ms adds no wake-up - the + * client's dispatcher re-reads channel state on any + * event - and the 0.5 s hint re-posts late waiters. + */ + if (time_before(jiffies, rm->events[i].last_post + + GSPRM_EVENT_HINT_DEDUP)) continue; + gsp_rm_fill_ev_payload(rm, i, payload); + { + int rc = gsp_rm_send_flags(ctx, + gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), + 0, payload, + sizeof(payload), + true); + + gsp_rm_evlog_post(vgpu, rm, i, "chanfb", rc); + if (rc) + continue; + } + rm->events[i].last_post = jiffies; posted++; } } @@ -667,6 +926,48 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) } /* + * r623 experiment: deliver a synthetic OS_ERROR_LOG (0x1006) record to the + * guest's RM queue - the error report a real GSP sends into the client's + * message queue when an engine faults. r618's synth-fault writes the channel + * notifier and posts a generic event; after it, the NEXT client stuck + * pre-submission (r620 atomics; r616b saxpy/math). If the guest's fault + * handling marks the device recoverable only when the 0x1006 report arrives, + * this record is the missing half. Payload mirrors the GSP ABI decode in + * gsp_phys_booter.c (GSP_PHYS_OSERR_OFF_*): exceptType u32, runlistId u32, + * chid u32, errString[256], prevXid u32. Manual/debug only. + */ +int gsp_rm_send_os_error_log(struct nv_vgpu_device *vgpu, u32 except_type, + u32 runlist_id, u32 chid, const char *engine) +{ + struct gsp_emu_context *ctx = vgpu ? vgpu->gsp_comm_ctx : NULL; + struct gsp_rm_emu *rm; + u8 payload[272] = {0}; + int rc; + + if (!ctx) + return -ENODEV; + rm = ctx->rm_emu; + if (!rm || !rm->parsed) + return -EAGAIN; + + memcpy(payload + 0, &except_type, 4); + memcpy(payload + 4, &runlist_id, 4); + memcpy(payload + 8, &chid, 4); + snprintf((char *)payload + 12, 255, "ENGINE %s", + engine && *engine ? engine : "GR"); + /* prevXid stays 0 */ + + rc = gsp_rm_send(ctx, 0x1006, (u32)rm->stat_seq + 1, 0, payload, + sizeof(payload)); + if (!rc) + nv_vgpu_info(vgpu, + "vgsp-rm: OS_ERROR_LOG sent (synthetic) except=0x%x rl=%u chid=%u engine=%s\n", + except_type, runlist_id, chid, + engine && *engine ? engine : "GR"); + return rc; +} + +/* * Debug: post the event record for one registered event (or all of them with * idx 0xffffffff). Used to test the guest's event path on demand without * waiting for a channel submission. @@ -675,18 +976,18 @@ static int gsp_rm_post_event_idx(struct gsp_emu_context *ctx, u32 idx) { struct gsp_rm_emu *rm = ctx->rm_emu; u8 payload[32]; + int rc; if (idx >= rm->events_n || !rm->events[idx].valid) return -EINVAL; - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[idx].h_client, 4); - memcpy(payload + 4, &rm->events[idx].h_event, 4); - memcpy(payload + 8, &rm->events[idx].notify_index, 4); - memcpy(payload + 12, &rm->events[idx].data, 4); + gsp_rm_fill_ev_payload(rm, idx, payload); - return gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload)); + rc = gsp_rm_send(ctx, gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), 0, payload, sizeof(payload)); + if (!rc) + rm->events[idx].last_post = jiffies; + return rc; } int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx) @@ -709,8 +1010,14 @@ int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx) } for (i = 0; i < rm->events_n; i++) { - int r = gsp_rm_post_event_idx(ctx, i); + int r; + /* m171: retired slots are not errors - skipping them + * keeps the hint's rc meaningful (a valid table must not + * report EINVAL because the last allocated slot is dead). */ + if (!rm->events[i].valid) + continue; + r = gsp_rm_post_event_idx(ctx, i); if (r) last = r; else @@ -725,10 +1032,68 @@ int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx) * guest allocates carry no source object, so a per-channel association is * not available in the ABI). Used by the service's completion hint so a * client that starts waiting after the event fired still gets a wake-up. + * + * m171: this is a HINT, and hints repeat every 0.5 s while the window is + * open. Re-posting an event that a real completion post already woke moments + * ago burns the burst budget and floods the status ring without adding a + * wake-up, so skip events posted within GSPRM_EVENT_HINT_DEDUP. A waiter + * that arrives after the edge still gets a re-post within half a second. */ int gsp_rm_post_all_events(struct nv_vgpu_device *vgpu) { - return gsp_rm_force_post(vgpu, 0xffffffffu); + struct gsp_emu_context *ctx = vgpu ? vgpu->gsp_comm_ctx : NULL; + struct gsp_rm_emu *rm; + u8 payload[32]; + u32 i; + int n = 0, skipped = 0; + + if (!ctx) + return -ENODEV; + rm = ctx->rm_emu; + if (!rm) + return -ENODEV; + + for (i = 0; i < rm->events_n; i++) { + if (!rm->events[i].valid) + continue; + if (time_before(jiffies, rm->events[i].last_post + + GSPRM_EVENT_HINT_DEDUP)) { + skipped++; + continue; + } + + gsp_rm_fill_ev_payload(rm, i, payload); + { + int rc = gsp_rm_send(ctx, gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), 0, + payload, sizeof(payload)); + + gsp_rm_evlog_post(vgpu, rm, i, "all", rc); + if (rc) + continue; + } + rm->events[i].last_post = jiffies; + n++; + } + + /* + * r576 DIAG: the anomaly sentinel - valid events existed, none were + * dedup-skipped, yet not one post could be sent. Every send rc is in + * the EVPOST lines above (budget permitting). + */ + if (!n && !skipped && gsp_rm_ev_log_max && + gsp_rm_ev_log < gsp_rm_ev_log_max) { + gsp_rm_ev_log++; + nv_vgpu_info(vgpu, + "vgsp-rm: EVPOST all: no postable event (events_n=%u dropped=%llu)", + rm->events_n, rm->events_dropped); + } + + if (n) + return 0; + /* -EAGAIN: valid events existed but every one was dedup-skipped + * (normal during an active post burst); -ENOENT: nothing to post. */ + return skipped ? -EAGAIN : -ENOENT; } /* @@ -977,6 +1342,7 @@ static const char *gsp_rm_rpc_name(u32 function) case GSPRM_RPC_FUNC_GSP_RM_CONTROL: return "GSP_RM_CONTROL"; case GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER: return "UNLOADING_GUEST_DRIVER"; case GSPRM_RPC_FUNC_UPDATE_BAR_PDE: return "UPDATE_BAR_PDE"; + case GSPRM_RPC_FUNC_DUP_OBJECT: return "DUP_OBJECT"; default: return "UNKNOWN"; } } @@ -1547,7 +1913,20 @@ static void gsp_rm_handle_bus_get_pci_bar_info(struct gsp_emu_context *ctx, * 36 bytes (observed paramsSize): feHwSysCap(0) windowPresentMask(4) * bFbRemapperEnabled(8) numHeads(12) i2cPort(16) internalDispActiveMask(20) * embeddedDisplayPortMask(24) muxSupported(28) numDispChannels(32). - * Report a minimal headless display: 1 head, 1 (core) channel. + * + * r651: this reply is also the gate for the WINDOW-channel alloc path. + * Live gdb on the guest nvidia.ko pinned the display path 0x56 to + * kdispGetChannelNum_v03_00: for dispChnClass_Win it requires + * pStaticInfo->windowPresentMask & NVBIT32(instance) and returns + * NV_ERR_NOT_SUPPORTED (0x56) when the bit is clear; the Curs class reads + * HEAD_EXISTS bits from feHwSysCap the same way (0x49+instance channel num). + * The previous reply left both masks zero, so nvRmApiAlloc(0xc57e) died in + * the guest's local resource construct (clientAllocResource -> + * resservResourceFactory -> dispchnConstruct -> kdispGetChannelNum) before + * any GSP-RM RPC was issued. Report the same one-head / 8-window model the + * NV0073 shim and the caps notifier expose (feHwSysCap head-0 exists, + * windowPresentMask bits 0..7, numDispChannels covering the cursor channel + * numbers 73..80, i.e. 81 entries). */ static void gsp_rm_handle_display_get_static_info(struct gsp_emu_context *ctx, u32 sequence, @@ -1559,15 +1938,205 @@ static void gsp_rm_handle_display_get_static_info(struct gsp_emu_context *ctx, memset(out_params, 0, 36); + /* + * feHwSysCap: NV_PDISP_FE_HW_SYS_CAP_HEAD_EXISTS(i) occupies bit i + * (dev_disp.h v03_00); head 0 exists on the emulated board. + */ + val32 = 0x1; + memcpy(out_params + 0, &val32, 4); + + /* + * windowPresentMask: bit i = window instance i present. Mirrors the + * caps notifier's SYS_CAPB=0xff and the NV0073 window assignment + * (windows 0/1 flexible) - eight usable windows. + */ + val32 = 0xff; + memcpy(out_params + 4, &val32, 4); + val32 = 1; /* numHeads */ memcpy(out_params + 12, &val32, 4); - val32 = 1; /* numDispChannels (core channel) */ + + /* + * numDispChannels sizes pClientChannelTable[], indexed by the packed + * channel number kdispGetChannelNum returns: CORE=0, WIN(i)=1+i, + * WINIM(i)=33+i, CURS(i)=73+i. Cover CURS(7)=80 -> 81 entries so any + * instance the guest allocates stays in bounds. + */ + val32 = 81; memcpy(out_params + 32, &val32, 4); *out_params_size = 36; *out_status = 0; - nv_vgpu_info(ctx->vgpu, "vgsp-rm: DISPLAY_GET_STATIC_INFO -> heads=1 dispChannels=1"); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DISPLAY_GET_STATIC_INFO -> heads=1 dispChannels=81 winMask=0xff feHwSysCap=0x1"); +} + +/* + * INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES (0x20800a5d) + * + * dispswConstruct_IMPL (disp_sw.c) asks for {displayMask, numHeads} and + * rejects every logicalHeadId >= numHeads with "invalid logical head + * number: N" + NV_ERR_INSUFFICIENT_RESOURCES; the generic all-zero reply + * made numHeads = 0, so nvkms' DISPLAY_SW objects could not be constructed + * on any head (observed r653: heads 0..3 all rejected -> vkCreateDevice -3). + * Report the same one-head model as DISPLAY_GET_STATIC_INFO: display id 0 + * active, one head. + * + * Params ABI (NV2080_CTRL_INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES_PARAMS, + * 8 bytes): displayMask(0) numHeads(4). + */ +static void gsp_rm_handle_display_active_devices(struct gsp_emu_context *ctx, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 displayMask = 0x1; /* display id 0 */ + u32 numHeads = 1; + + memcpy(out_params + 0, &displayMask, 4); + memcpy(out_params + 4, &numHeads, 4); + + *out_params_size = 8; + *out_status = 0; + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DISPLAY_GET_ACTIVE_DISPLAY_DEVICES -> displayMask=0x1 numHeads=1"); +} + +/* + * NVC370_CTRL_CMD_GET_CHANNEL_INFO (0xc3700104) + * + * nvRMSyncEvoChannel() (nvkms-rm.c) polls this control through the display + * object after every EVO channel kickoff and waits up to 2 s for the channel + * to report NVC370_CTRL_GET_CHANNEL_INFO_STATE_IDLE, otherwise it logs + * "Idling display engine timed out: <class>:<instance>:<sd>:<token>" and + * fails the whole modeset init (observed on the r652 window path: 0xc57d and + * 0xc57e instances 0/1). The emulated display engine has no pushbuffer + * consumer, so the state would never leave BUSY; report IDLE immediately. + * + * Params ABI (NVC370_CTRL_CMD_GET_CHANNEL_INFO_PARAMS, 20 bytes): + * subdeviceIndex(0) channelClass(4) channelInstance(8) + * IsChannelInDebugMode(12) channelState(16 = NVBIT(0) IDLE). + * Echo the request so class/instance round-trip and set the state word. + */ +static void gsp_rm_handle_display_channel_info(struct gsp_emu_context *ctx, + const u8 *in_params, u32 in_size, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 idle = 1; /* NVC370_CTRL_GET_CHANNEL_INFO_STATE_IDLE */ + u32 class_id = 0, instance = 0; + u32 size = in_size; + + if (size > 64) + size = 64; /* defensive cap; the real params are 20 bytes */ + if (size >= 20) { + memcpy(out_params, in_params, size); + memcpy(&class_id, in_params + 4, 4); + memcpy(&instance, in_params + 8, 4); + memcpy(out_params + 16, &idle, 4); + } else { + memset(out_params, 0, 20); + memcpy(out_params + 16, &idle, 4); + size = 20; + } + + *out_params_size = size; + *out_status = 0; + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: GET_CHANNEL_INFO class=0x%x instance=%u -> IDLE", + class_id, instance); +} + +/* + * NV0080_CTRL_CMD_FIFO_GET_ENGINE_CONTEXT_PROPERTIES (0x801707) + * + * The Vulkan/GL client queries the device for the size and alignment of an + * engine's context save region while building its GR channel context + * (observed: engineId 0x8 = GRAPHICS_ZCULL, one query at the end of every + * vkCreateDevice attempt). The generic all-zero reply handed back size = 0; + * the client's context-buffer allocation then failed (memdesc of size 0) + * and device creation aborted with VK_ERROR_INITIALIZATION_FAILED + * (observed r661: four create attempts, each ending at this control, then + * teardown and VKFAIL -3). + * + * The sizes below mirror the real 570.144 GSP-RM geometry captured on this + * TU104 (rm-call golden runs); entries marked "fill" were not covered by + * the capture and use nonzero page-aligned placeholders - revisit them when + * a real capture for the remaining engine ids exists. + * + * Params ABI (NV0080_CTRL_FIFO_GET_ENGINE_CONTEXT_PROPERTIES_PARAMS, + * 12 bytes): engineId(0, in) alignment(4, out) size(8, out). + */ +static const struct gsp_rm_engine_ctx_geom { + u32 engine_id; + u32 alignment; + u32 size; +} gsp_rm_engine_ctx_geoms[] = { + { 0x00, 0x1000, 0xf2d00 }, /* GRAPHICS (captured) */ + { 0x08, 0x1000, 0x10000 }, /* GRAPHICS_ZCULL (fill) */ + { 0x09, 0x1000, 0xd100 }, /* GRAPHICS_PM (captured) */ + { 0x0a, 0x1000, 0x1000 }, /* COMPUTE_PREEMPT (fill) */ + { 0x0b, 0x1000, 0x1000 }, /* GRAPHICS_PREEMPT (fill) */ + { 0x0c, 0x1000, 0x1000 }, /* GRAPHICS_SPILL (fill) */ + { 0x0d, 0x100, 0x20000 }, /* GRAPHICS_PAGEPOOL (fill) */ + { 0x0e, 0x100, 0x8000 }, /* GRAPHICS_BETACB (fill) */ + { 0x0f, 0x100, 0x80000 }, /* GRAPHICS_RTV (fill) */ + { 0x10, 0x1000, 0x5000 }, /* GRAPHICS_PATCH (captured) */ + { 0x11, 0x100, 0x3000 }, /* GRAPHICS_BUNDLE_CB (captured) */ + { 0x12, 0x100, 0x20000 }, /* GRAPHICS_PAGEPOOL_GLOBAL (captured) */ + { 0x13, 0x1000, 0x7d7400 }, /* GRAPHICS_ATTRIBUTE_CB (captured) */ + { 0x14, 0x100, 0x80000 }, /* GRAPHICS_RTV_CB_GLOBAL (captured) */ + { 0x15, 0x100, 0x20000 }, /* GRAPHICS_GFXP_POOL (fill) */ + { 0x16, 0x100, 0x1000 }, /* GRAPHICS_GFXP_CTRL_BLK (fill) */ + { 0x17, 0x1000, 0x10000 }, /* GRAPHICS_FECS_EVENT (captured) */ + { 0x18, 0x1000, 0x80000 }, /* GRAPHICS_PRIV_ACCESS_MAP (captured) */ + { 0x19, 0x100, 0x1000 }, /* GRAPHICS_SETUP (fill) */ +}; + +static void gsp_rm_handle_fifo_engine_ctx_props(struct gsp_emu_context *ctx, + const u8 *in_params, u32 in_size, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 engine_id = ~0u; + u32 alignment = 0x1000, size = 0x1000; + u32 i; + bool found = false; + + if (in_size >= 4) + memcpy(&engine_id, in_params + 0, 4); + + for (i = 0; i < ARRAY_SIZE(gsp_rm_engine_ctx_geoms); i++) { + if (gsp_rm_engine_ctx_geoms[i].engine_id == engine_id) { + alignment = gsp_rm_engine_ctx_geoms[i].alignment; + size = gsp_rm_engine_ctx_geoms[i].size; + found = true; + break; + } + } + + if (engine_id > 0x19) { + /* Same rejection the real RM's VF handler uses for ids + * outside the engine context properties range. */ + *out_status = 0x56; /* NV_ERR_NOT_SUPPORTED */ + *out_params_size = 0; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: FIFO_GET_ENGINE_CONTEXT_PROPERTIES engineId=0x%x -> NOT_SUPPORTED", + engine_id); + return; + } + + memcpy(out_params + 0, &engine_id, 4); + memcpy(out_params + 4, &alignment, 4); + memcpy(out_params + 8, &size, 4); + *out_params_size = 12; + *out_status = 0; + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: FIFO_GET_ENGINE_CONTEXT_PROPERTIES engineId=0x%x -> alignment=0x%x size=0x%x%s", + engine_id, alignment, size, found ? "" : " (default)"); } /* @@ -1593,6 +2162,103 @@ static void gsp_rm_handle_display_get_ip_version(struct gsp_emu_context *ctx, } /* + * NV0073 display-common heads + one static HDMI connector + * (0x730102 / 0x730287 / 0x7302ad / 0x730107 / 0x73028b / 0x730250 / + * 0x731140) + * + * nvidia-modeset's nvkms-rm.c ProbeHeadCountAndWindowAssignment asks the + * display-common object for the head count (NV0073_CTRL_CMD_SYSTEM_ + * GET_NUM_HEADS = 0x730102) and a zero reply makes it log "No heads found on + * board!" and fail -- which kills Xorg's ScreenInit and every Vulkan + * vkCreateDevice (measured: the emulated device replies all-zero through the + * generic path, so nvkms never opens). The kernel driver already sees 1 head + * (DISPLAY_GET_STATIC_INFO reports numHeads=1); report the same 1 head here, + * a single-head mask for GET_ALL_HEAD_MASK (0x730287), and a fully-flexible + * window assignment for windows 0/1 (0xFF = FLEXIBLE; windows above the head + * count stay 0 = not present). + * + * The next wall after the heads (measured r643): nvRmAllocDisplays fails on + * NoValidDisplays() - nvkms requires at least one valid display. Report one + * static TMDS/HDMI connector on display id 0: GET_SUPPORTED (0x730107) + * displayMask=1|DDC=1, SPECIFIC_OR_GET_INFO (0x73028b) SOR / + * SOR_SINGLE_TMDS_A (TMDS avoids the whole DP-AUX lib), SPECIFIC_ + * GET_CONNECTOR_DATA (0x730250) one HDMI_A entry, DFP_GET_INFO (0x731140) + * TMDS flags. Still a headless setup: no connect state or EDID is claimed + * (those are modeset-time queries). + */ +static void gsp_rm_handle_display_heads(struct gsp_emu_context *ctx, u32 cmd, + u32 req_size, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 one = 1; + u32 sz = req_size ? req_size : 72; + u32 v; + + memset(out_params, 0, sz); + switch (cmd) { + case 0x730102: /* SYSTEM_GET_NUM_HEADS: {subDev, flags, numHeads} */ + if (sz >= 12) + memcpy(out_params + 8, &one, 4); + break; + case 0x730287: /* SPECIFIC_GET_ALL_HEAD_MASK: {subDev, headMask} */ + if (sz >= 8) + memcpy(out_params + 4, &one, 4); + break; + case 0x7302ad: /* SPECIFIC_GET_VALID_HEAD_WINDOW_ASSIGNMENT: + * {subDev, windowHeadMask[32]} - windows 0 and 1 + * flexible (0xFF), the rest absent (0). */ + if (sz >= 36) { + out_params[4] = 0xff; + out_params[5] = 0xff; + } + break; + case 0x730107: /* SYSTEM_GET_SUPPORTED: {subDev, displayMask, + * displayMaskDDC} - display id 0 supported + DDC. */ + if (sz >= 12) { + memcpy(out_params + 4, &one, 4); + memcpy(out_params + 8, &one, 4); + } + break; + case 0x73028b: /* SPECIFIC_OR_GET_INFO: {subDev, displayId, index, + * type, protocol, ditherType, ditherAlgo, location, + * rootPortId, dcbIndex, vbiosAddress, bIsLitByVbios, + * bIsDispDynamic} */ + if (sz >= 52) { + v = 2; /* NV0073_CTRL_SPECIFIC_OR_TYPE_SOR */ + memcpy(out_params + 12, &v, 4); + v = 1; /* SOR_SINGLE_TMDS_A (no DP aux) */ + memcpy(out_params + 16, &v, 4); + /* location=0 (chip), remaining out-fields stay 0: + * not lit by vbios, not dynamic. */ + } + break; + case 0x730250: /* SPECIFIC_GET_CONNECTOR_DATA: {subDev, + * displayId, flags, DDCPartners, count, + * data[4]{index,type,location}, platform} */ + if (sz >= 72) { + v = 1; /* CONNECTOR_DATA_FLAGS_PRESENT_YES */ + memcpy(out_params + 8, &v, 4); + v = 1; /* count */ + memcpy(out_params + 16, &v, 4); + /* data[0].index = 0 */ + v = 0x61; /* CONNECTOR_DATA_TYPE_HDMI_A */ + memcpy(out_params + 24, &v, 4); + /* data[0].location = 0 */ + } + break; + case 0x731140: /* DFP_GET_INFO: {subDev, displayId, flags, + * UHBRSupportedByDfp} - flags signal bits = TMDS(0) */ + break; /* zeros are the TMDS signal value */ + } + *out_params_size = sz; + *out_status = 0; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DISPLAY shim cmd=0x%x -> 1 head / 1 HDMI connector (size %u)", + cmd, sz); +} + +/* * CE_GET_FAULT_METHOD_BUFFER_SIZE (0x20802a08) * * kchangrpInit allocates one fault method buffer per runqueue of this @@ -1999,6 +2665,20 @@ static void gsp_rm_handle_kgr_get_context_buffers_info( * L2/LTC topology from here. Echo the requested indices and fill in the * emulated TU104 values. */ +/* + * m113: FB_GET_INFO_V2 COMPRESSION_SIZE (index 0x01, bytes). The header: + * "A return value of 0 indicates the GPU does not support compression" - + * but the guest driver still attempts a COMPR-enabled, 64 KB-aligned + * comptag allocation (vendor VidHeap alloc, attr COMPR=2) and fails it + * NV_ERR_NO_MEMORY (0x51) when the comptag pool has no space, which + * libcuda maps to VK_ERROR_OUT_OF_DEVICE_MEMORY (the vkAllocateMemory -2 + * gate). Report a comptag pool sized like the real Turing parts (~fb/16). + * 0 = the historic "no compression" reply. + */ +static uint gsp_rm_fb_compr_mb = 128; +module_param_named(wall2_fb_compr_mb, gsp_rm_fb_compr_mb, uint, 0644); +MODULE_PARM_DESC(wall2_fb_compr_mb, "FB_GET_INFO_V2 COMPRESSION_SIZE in MiB (default 128, 0 = none)"); + static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) { u64 fb = nv_vgpu_fb_emul_size(ctx->vgpu); @@ -2007,7 +2687,8 @@ static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) case TU104_FB_INFO_IDX_TILE_REGION_COUNT: return 0; case TU104_FB_INFO_IDX_COMPRESSION_SIZE: - return 0; + /* m113: non-zero comptag pool (bytes) - see gsp_rm_fb_compr_mb. */ + return gsp_rm_fb_compr_mb << 20; case TU104_FB_INFO_IDX_PARTITION_COUNT: return 1; case TU104_FB_INFO_IDX_BAR1_SIZE: @@ -2264,6 +2945,7 @@ static void gsp_rm_handle_bus_get_info_v2(struct gsp_emu_context *ctx, #define GSPRM_COPY_PDES_MAX_LEVELS 6 static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, + u32 h_object, const u8 *params, u32 params_size) { struct gsp_rm_emu *rm = ctx->rm_emu; @@ -2314,6 +2996,46 @@ static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, "vgsp-rm: pt level[%u] phys=0x%llx size=0x%llx aperture=%u pageShift=%u", i, phys, size, aperture, page_shift); } + + /* + * r496j: this control IS the kernel RM publishing the page-table + * chain of its internal VAS (the level[0..] instances with their + * phys addresses and page shifts). The chain can name the X/GL + * channel's pushbuffer pages and progress semaphore, for which no + * other translation source exists (the fn-54 SET_PAGE_DIRECTORY + * path is external-VAS/UVM only). It is NOT registered directly: + * a blast registration of every client's directory let a stale + * kernel directory answer another client's VA in the walker's + * first-success order (the r459 stale-directory class; observed + * as a CUDA client hanging after DEVICE with r496g). Instead the + * root is offered as a CANDIDATE; the progress service registers + * it only after it has been VERIFIED against a content-validated + * pair of that very client (the ring found by the RAM content + * scan). + */ + if (num_levels >= 1 && rm->pt.levels[0].phys && + rm->pt.levels[0].size >= 16 && + rm->pt.levels[0].size <= 4096 * 8) { + /* + * The level phys values are guest PAs on this platform + * (observed 4+ GiB, above any FB shadow); aperture 3 in the + * message does not match the walker's FB/SYS encoding, so + * classify by address: values inside the emulated VRAM are + * FB, everything else sysmem. The walker also retries the + * other aperture, so a misclassification only costs a walk. + */ + bool vidmem = rm->pt.levels[0].phys < + (u64)ctx->vgpu->bars[1].size; + + gsp_sema_service_vaspace_candidate_add( + h_object, rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), vidmem); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: VAS 0x%x page directory offered as candidate pdb=0x%llx entries=%u (%s; registered only after a content-verified ring walk)", + h_object, (u64)rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), + vidmem ? "vidmem" : "sysmem"); + } } static bool gsp_rm_alloc_interesting(u32 class_id); @@ -2330,8 +3052,21 @@ static bool gsp_rm_is_channel_class(u32 class_id) case 0x906f: /* MAXWELL_CHANNEL_GPFIFO_A */ case 0xa06f: /* KEPLER_CHANNEL_GPFIFO_A */ case 0xb06f: /* PASCAL_CHANNEL_GPFIFO_A */ - case 0xc06f: /* VOLTA_CHANNEL_GPFIFO_A */ + case 0xc06f: /* VOLTA_CHANNEL_GPFIFO_A (PRE-TURING NAME) */ + case 0xc36f: /* VOLTA_CHANNEL_GPFIFO_A */ + /* + * r828: the RC watchdog's channel allocates 0xc36f on this + * device (the driver's gpfifoMapping loop picks the first + * class the gpu reports supported) - without it in this set + * the channel never reaches the progress service, its class2d + * notifier cycle (SET_NOTIFY_A/B + NOTIFY) never executes and + * the guest's krcWatchdog declares the GPU locked after its + * 7 s timeout as soon as robust channels are in play. + */ case 0xc46f: /* TURING_CHANNEL_GPFIFO_A */ + case 0xc56f: /* AMPERE_CHANNEL_GPFIFO_A */ + case 0xc86f: /* HOPPER_CHANNEL_GPFIFO_A */ + case 0xc96f: /* BLACKWELL_CHANNEL_GPFIFO_A */ return true; default: return false; @@ -2353,8 +3088,18 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, { struct sema_channel_desc desc; - if (params_size < GSPRM_CHAN_NOTIFIER_MEM + 8) + if (params_size < GSPRM_CHAN_NOTIFIER_MEM + 8) { + /* + * r828 diag: a channel-class alloc whose params are shorter + * than the notifier field is not registered, silently. The + * RC watchdog channel (hObject 0x31415900) never appeared in + * the sema table; name every such skip with its size so the + * shape question is answered by a log, not inference. + */ + pr_info("vgsp-rm: channel alloc hObject=0x%x params_size=%u too short (< %u); NOT registered\n", + h_object, params_size, GSPRM_CHAN_NOTIFIER_MEM + 8); return; + } memset(&desc, 0, sizeof(desc)); desc.vgpu = ctx->vgpu; @@ -2362,15 +3107,108 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, memcpy(&desc.gpfifo_va, params + GSPRM_CHAN_GPFIFO_OFFSET, 8); memcpy(&desc.gpfifo_entries, params + GSPRM_CHAN_GPFIFO_ENTRIES, 4); memcpy(&desc.engine_type, params + GSPRM_CHAN_ENGINE_TYPE, 4); + /* + * The wire carries NV2080_ENGINE_TYPE_*: NULL = 0, GR0 (== GRAPHICS) + * = 1, COPY0 = 9, NVDEC0 = 0x13, ... The progress service's + * convention is "engine_type == 0 means the GR channel" (CUDA's + * kernel-RM GR channels arrive with engineType NULL), so an X/GL 3D + * channel - which carries the real NV2080_ENGINE_TYPE_GRAPHICS = 1 - + * was misclassified as a CE and its push never executed ("Failed to + * initialize DMA" in the X driver's pushbuffer NOP test). Normalize + * GR0..GR7 (1..8) to 0: the emulated device has a single GR and every + * other engine id is >= COPY0 (9), so this cannot capture a CE. + */ + if (desc.engine_type >= 1 && desc.engine_type <= 8) + desc.engine_type = 0; + /* + * r835: the channel's own address space (NV_CHANNEL_ALLOC_PARAMS: + * hContextShare @24, hVASpace @28 - confirmed by this struct's own + * known offsets: engineType @128 = 32 + 8 handles + 8 subdevice + * handles + 8 u64 userd offsets). Two clients map different objects + * at overlapping VAs (everyone's RM window sits at 0x1200xxxxx), so + * the global VA-keyed translation maps conflate them: the X server's + * ring VA resolved through the GL client's page tables (walk-probe: + * vas h=0xcaf00005 answered 0x120090000 with a zero page) and its + * push stream decoded as empty, wedging X exactly like the RC + * watchdog's ring. The service resolves a channel's VAs through this + * VAS first. + */ + memcpy(&desc.h_vaspace, params + 28, 4); memcpy(&desc.cid, params + GSPRM_CHAN_CID, 4); memcpy(&desc.instance_pa, params + GSPRM_CHAN_INSTANCE_MEM, 8); memcpy(&desc.userd_pa, params + GSPRM_CHAN_USERD_MEM, 8); + /* + * SESSION 153-CONTINUED-38: the USERD control page sits at + * userdMem.address + userdOffset[subDeviceId] (NV_CHANNEL_ALLOC_PARAMS + * userdOffset[NV_MAX_SUBDEVICES], @64; subDeviceId @136, the + * one-hot bit index is normally 0 but the X client-share channel + * carries 1). The GL client's kernel channels declare userdMem fb + * 0x18fe00 with userdOffset[0]=0x7fe00, so the service was + * probing/monitoring 0x18fe00 while the real control page + * (GPGet/GPPut) lives at 0x20fc00 - both the registered candidate + * and every FB observation were at the wrong offset, which is why + * the direct client's channels never resolved a USERD (CONT-37 + * alloc/free retry loop). Index by the subdevice. + */ + { + u32 sd = 0; + u64 uo = 0; + + if (params_size >= 140) + memcpy(&sd, params + 136, 4); + if (sd >= 8) + sd = 0; + if (params_size >= 64 + (sd + 1) * 8) { + memcpy(&uo, params + 64 + (u64)sd * 8, 8); + desc.userd_pa += uo; + } + } memcpy(&desc.userd_as, params + GSPRM_CHAN_USERD_MEM + GSPRM_CHAN_MEM_APERTURE, 4); memcpy(&desc.notifier_pa, params + GSPRM_CHAN_NOTIFIER_MEM, 8); memcpy(&desc.notifier_as, params + GSPRM_CHAN_NOTIFIER_MEM + GSPRM_CHAN_MEM_APERTURE, 4); + /* + * SESSION 153-CONTINUED-15 (draw attribution): every channel-class + * allocation is named with its wire shape before registration, so a + * channel the service silently refuses (gpfifo VA 0 / entries < 2) + * cannot hide behind an absent "registered" line. Bounded. + */ + { + static u32 chan_alloc_log; + + if (chan_alloc_log < 32) { + chan_alloc_log++; + pr_info("vgsp-rm: chan alloc hObject=0x%x gpfifo=0x%llx ent=%u eng=%u vas=0x%x cid=%u userd=0x%llx/%u notif=0x%llx/%u\n", + h_object, desc.gpfifo_va, desc.gpfifo_entries, + desc.engine_type, desc.h_vaspace, desc.cid, + desc.userd_pa, desc.userd_as, + desc.notifier_pa, desc.notifier_as); + } + } + /* + * SESSION 153-CONTINUED-38: name the per-subdevice userdOffset (the + * kernel channels' control page may sit inside the declared userdMem + * at a nonzero offset; the service currently assumes address+0). + * Bounded. + */ + { + static u32 uo_log; + + if (uo_log < 8 && params_size >= 80) { + u64 uo0 = 0; + u32 sd = 0; + + memcpy(&uo0, params + 64, 8); + if (params_size >= 140) + memcpy(&sd, params + 136, 4); + uo_log++; + pr_info("vgsp-rm: chan userdOff[0]=0x%llx subDevId=0x%x hObject=0x%x (userd memdesc=0x%llx/%u)\n", + uo0, sd, h_object, desc.userd_pa, + desc.userd_as); + } + } gsp_sema_service_channel_add(&desc); } @@ -2439,56 +3277,98 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, gsp_rm_register_channel(ctx, ho, params, params_size); /* - * NV01_EVENT (allocated with class 0x79 in the guest ABI; params are - * NV0005_ALLOC_PARAMETERS {hParentClient, hSrcResource, hClass, - * notifyIndex, data}). The driver allocates one per channel and waits - * for the GPU to post the event before it considers submitted work - * complete - remember them so completed work can be signalled back. + * NV01_EVENT (params are NV0005_ALLOC_PARAMETERS {hParentClient, + * hSrcResource, hClass, notifyIndex, data}). The driver allocates one + * per channel and waits for the GPU to post the event before it + * considers submitted work complete - remember them so completed work + * can be signalled back. + * + * r682: the guest ABI uses THREE event classes and only the + * NV01_EVENT_OS_EVENT (0x79) shape was registered: libcuda's channel + * events arrive as 0x79, but the modern nvkms/Vulkan path allocates + * NV01_EVENT_KERNEL_CALLBACK_EX (0x7E, nvos.h) - observed live as + * `GSP_RM_ALLOC class=0x7e paramsSize=24 hParent=<subdevice> + * hObject=0xcaf00003` from the second RM client while its main + * thread sat in the vkCreateDevice wait loop (readiness word in a + * /dev/nvidiactl notification page, never signalled). Register 0x78 + * (KERNEL_CALLBACK) and 0x7E alongside 0x79 so the completion + * wake-ups reach those events too. + * + * The table is append+retire: FREE retires records (valid=false) but + * never shrinks events_n, so long-lived boots with many client + * create/destroy cycles would otherwise exhaust the 128 slots and + * SILENTLY drop new registrations - after which the completion + * wake-ups (post_channel_event / post_all_events) fail with EINVAL + * and a later client waits forever for a completion that can no + * longer be notified (observed r162: `completion hint rc=-22` + * repeating from the first post-teardown client onward). Reclaim the + * first retired slot for a new registration, and log loudly when the + * table is genuinely full of live entries. */ - if (class_id == 0x79 && params_size >= 24 && rm->events_n < GSPRM_MAX_EVENTS) { + if ((class_id == 0x79 || class_id == 0x7e || class_id == 0x78) && + params_size >= 24) { u32 h_src = 0, notify_index = 0, hclass = 0; u64 data64 = 0; int hc_off = -1, i; + u32 slot = rm->events_n; - /* - * Locate hClass (0x79) inside the params instead of trusting a - * fixed offset: NV0005_ALLOC_PARAMETERS is - * {hParentClient, hSrcResource, hClass, notifyIndex, data}, - * with hSrcResource immediately before hClass. - */ - for (i = 0; i + 4 <= (int)params_size; i += 4) { - u32 v; - - memcpy(&v, params + i, 4); - if (v == 0x79) { - hc_off = i; + for (i = 0; i < (int)rm->events_n; i++) { + if (!rm->events[i].valid) { + slot = (u32)i; break; } } - if (hc_off >= 4) { - memcpy(&h_src, params + hc_off - 4, 4); - memcpy(&hclass, params + hc_off, 4); - if (hc_off + 8 <= (int)params_size) - memcpy(¬ify_index, params + hc_off + 4, 4); - if (hc_off + 16 <= (int)params_size) - memcpy(&data64, params + hc_off + 8, 8); - } else { - memcpy(&h_src, params + 4, 4); - memcpy(¬ify_index, params + 12, 4); - memcpy(&data64, params + 16, 8); + if (slot >= GSPRM_MAX_EVENTS) + nv_vgpu_err(ctx->vgpu, + "vgsp-rm: event table full (%u live records); dropping registration hObject=0x%x - completion wake-ups will not reach this client", + rm->events_n, ho); + else { + /* + * Locate hClass (0x78/0x79/0x7e) inside the params + * instead of trusting a fixed offset: + * NV0005_ALLOC_PARAMETERS is {hParentClient, + * hSrcResource, hClass, notifyIndex, data}, with + * hSrcResource immediately before hClass. + */ + for (i = 0; i + 4 <= (int)params_size; i += 4) { + u32 v; + + memcpy(&v, params + i, 4); + if (v == 0x79 || v == 0x7e || v == 0x78) { + hc_off = i; + break; + } + } + if (hc_off >= 4) { + memcpy(&h_src, params + hc_off - 4, 4); + memcpy(&hclass, params + hc_off, 4); + if (hc_off + 8 <= (int)params_size) + memcpy(¬ify_index, + params + hc_off + 4, 4); + if (hc_off + 16 <= (int)params_size) + memcpy(&data64, + params + hc_off + 8, 8); + } else { + memcpy(&h_src, params + 4, 4); + memcpy(¬ify_index, params + 12, 4); + memcpy(&data64, params + 16, 8); + } + rm->events[slot].h_src = h_src; + rm->events[slot].h_event = ho; + rm->events[slot].h_client = hc; + rm->events[slot].notify_index = notify_index; + rm->events[slot].data = (u32)data64; + rm->events[slot].valid = true; + if (slot == rm->events_n) + rm->events_n++; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: event registered hEvent=0x%x hSrc=0x%x notifyIndex=%u (off %d, slot %u) params=%*phN", + ho, h_src, notify_index, hc_off, + slot, + (int)min(params_size, (u32)24), + params); } - rm->events[rm->events_n].h_src = h_src; - rm->events[rm->events_n].h_event = ho; - rm->events[rm->events_n].h_client = hc; - rm->events[rm->events_n].notify_index = notify_index; - rm->events[rm->events_n].data = (u32)data64; - rm->events[rm->events_n].valid = true; - rm->events_n++; - if (gsp_sema_dbg) - nv_vgpu_info(ctx->vgpu, - "vgsp-rm: event registered hEvent=0x%x hSrc=0x%x notifyIndex=%u (off %d) params=%*phN", - ho, h_src, notify_index, hc_off, - (int)min(params_size, (u32)24), params); } /* Echo the fixed alloc header + as many param bytes as are contiguous @@ -2680,6 +3560,217 @@ gsp_rm_handle_promote_fault_method_buffers(struct gsp_emu_context *ctx, } /* + * r519 experiment switch, independent of the P3 whitelist: forward ONLY the + * BINAPI reserved-interface query 0x20810107 to the live GSP session. The P3 + * whitelist may be off (its mixed real/emulated device answers perturb the + * boot); this one command is what the GL surface init needs answered. + */ +static bool gsp_rm_binapi07_fwd_param; +module_param_named(gsp_rm_binapi07_fwd, gsp_rm_binapi07_fwd_param, bool, 0644); +MODULE_PARM_DESC(gsp_rm_binapi07_fwd, + "Forward only the guest GL core's BINAPI 0x20810107 query to the live GSP session (default 0; diagnostic/experiment)"); + +/* + * SESSION 153-CONTINUED-17: reply word (little-endian u16) for BINAPI + * 0x20810107. The guest GL client reads reply byte[1] as a feature flag + * (glcore+0xFC74E1: setne <ctx>+0x203 after cmpb $0x0,reply[1]) and the + * flag gates method emission at glcore+0xB5B48B (see the control handler). + * 0x0101 sets both bytes; 0x0100 sets byte[1] only. Default 0 = the + * historical {0,0} reply. + */ +static uint gsp_rm_binapi07_rep; +module_param_named(binapi07_rep, gsp_rm_binapi07_rep, uint, 0644); +MODULE_PARM_DESC(binapi07_rep, + "reply word (LE u16) for BINAPI cmd 0x20810107; the GL client reads byte[1] as a feature flag (default 0)"); + +/* + * session 75: reply words for the client-pending control 0x2080a097 (the + * libnvidia-eglcore engine-readiness poll). Runtime-tunable so one boot can + * sweep reply variants: the client reads +0x204, +0x488 and +0x48c back + * (file offsets 0xecd45f..0xecd475) before its engine loop re-issues the + * control. Defaults: flag=1 (the request sets it to 0), the other words 0. + */ +static uint gsp_rm_a097_flag = 1; +module_param_named(wall2_a097_flag, gsp_rm_a097_flag, uint, 0644); +MODULE_PARM_DESC(wall2_a097_flag, "0x2080a097 reply word at params+0x204 (default 1)"); +static uint gsp_rm_a097_v488; +module_param_named(wall2_a097_v488, gsp_rm_a097_v488, uint, 0644); +MODULE_PARM_DESC(wall2_a097_v488, "0x2080a097 reply word at params+0x488 (default 0)"); +static uint gsp_rm_a097_v48c; +module_param_named(wall2_a097_v48c, gsp_rm_a097_v48c, uint, 0644); +MODULE_PARM_DESC(wall2_a097_v48c, "0x2080a097 reply word at params+0x48c (default 0)"); + +/* + * session 76: the 0x2080a097 reply's RECORD area (params+4..+0x23). The + * real host RM handler overwrites up to four {u32, u32} mode pairs there + * (per-mode engine status + a x1000 clock/percent value); the client + * (count=1) copies back the first 16 bytes into its per-engine query + * buffer, and the second word of the first pair is consumed as the + * "engine progress" word (libnvidia-eglcore d5e3e0: *(inner+4) -> + * session+0xe104, compared against per-boot config thresholds). + * Each word defaults to 0xffffffff = leave the echoed request word + * untouched; any other value replaces it, so one boot can sweep the + * record matrix without rebuilding. + */ +static uint gsp_rm_a097_count = ~0u; +module_param_named(wall2_a097_count, gsp_rm_a097_count, uint, 0644); +MODULE_PARM_DESC(wall2_a097_count, "0x2080a097 reply count word at params+0 (0xffffffff = echo)"); +static uint gsp_rm_a097_rec0 = ~0u, gsp_rm_a097_rec1 = ~0u; +static uint gsp_rm_a097_rec2 = ~0u, gsp_rm_a097_rec3 = ~0u; +static uint gsp_rm_a097_rec4 = ~0u, gsp_rm_a097_rec5 = ~0u; +static uint gsp_rm_a097_rec6 = ~0u, gsp_rm_a097_rec7 = ~0u; +module_param_named(wall2_a097_rec0, gsp_rm_a097_rec0, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec0, "0x2080a097 reply word at params+0x4 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec1, gsp_rm_a097_rec1, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec1, "0x2080a097 reply word at params+0x8 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec2, gsp_rm_a097_rec2, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec2, "0x2080a097 reply word at params+0xc (0xffffffff = echo)"); +module_param_named(wall2_a097_rec3, gsp_rm_a097_rec3, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec3, "0x2080a097 reply word at params+0x10 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec4, gsp_rm_a097_rec4, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec4, "0x2080a097 reply word at params+0x14 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec5, gsp_rm_a097_rec5, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec5, "0x2080a097 reply word at params+0x18 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec6, gsp_rm_a097_rec6, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec6, "0x2080a097 reply word at params+0x1c (0xffffffff = echo)"); +module_param_named(wall2_a097_rec7, gsp_rm_a097_rec7, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec7, "0x2080a097 reply word at params+0x20 (0xffffffff = echo)"); + +/* + * m112b: the NV04_VID_HEAP_CONTROL enumeration pair 0x90960106/0x90960107 + * (legacy NVOS32 VidHeap API, class 0x9096). libnvidia-eglcore (file offset + * 0x586d80) walks three heaps: per heap it queries 0x90960106 (12 bytes: + * {u32 firstIdx, u32 lastIdx}; the client then enumerates the heap's memory + * blocks by index via 0x90960107 (56 bytes) while firstIdx+1 <= idx <= + * lastIdx). The 56-byte block entry is read at +0x00/+0x08 (two u64s), +0x28 + * (a u32 type compared against a table at eglcore 0x101b1c0) and +0x30 (a + * flag BYTE: when 0 the heap/block is skipped). The generic zero reply makes + * every heap look empty, the client concludes out-of-device-memory and + * vkAllocateMemory fails -2 WITHOUT any alloc reaching the emulation + * (dmesg-corroborated: only 0x9096* controls + a class 0xc5c0 alloc). + * Runtime-tunable so one boot can sweep reply variants (same pattern as the + * wall2_a097_* words). + */ +static uint gsp_rm_9096_lo; /* 0x90960106 firstIdx */ +static uint gsp_rm_9096_hi = 1; /* 0x90960106 lastIdx */ +module_param_named(wall2_9096_lo, gsp_rm_9096_lo, uint, 0644); +MODULE_PARM_DESC(wall2_9096_lo, "0x90960106 reply firstIdx (default 0)"); +module_param_named(wall2_9096_hi, gsp_rm_9096_hi, uint, 0644); +MODULE_PARM_DESC(wall2_9096_hi, "0x90960106 reply lastIdx (default 1)"); +static uint gsp_rm_9096_ent_addr; /* 0x90960107 entry +0x00 (u64) */ +module_param_named(wall2_9096_ent_addr, gsp_rm_9096_ent_addr, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_addr, "0x90960107 entry word at +0x00 (default 0)"); +static uint gsp_rm_9096_ent_size = 0x80000000u; /* entry +0x08 (u64) */ +module_param_named(wall2_9096_ent_size, gsp_rm_9096_ent_size, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_size, "0x90960107 entry word at +0x08 (default 0x80000000)"); +static uint gsp_rm_9096_ent_type = 3; /* entry +0x28 (u32) */ +module_param_named(wall2_9096_ent_type, gsp_rm_9096_ent_type, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_type, "0x90960107 entry word at +0x28 (default 3)"); +static uint gsp_rm_9096_ent_flag = 1; /* entry +0x30 (u8) */ + +module_param_named(wall2_9096_ent_flag, gsp_rm_9096_ent_flag, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_flag, "0x90960107 entry flag byte at +0x30 (default 1)"); + + +/* + * The live session's oracle client/object for the 0x20810107 experiment, + * created on demand; 0 until the first query builds it. One device per module + * load, so file-statics are enough; reset by unload. + */ +static u32 gsp_rm_binapi07_be_client; +static u32 gsp_rm_binapi07_be_object; + +/* + * One GSP_RM_ALLOC (fn 103) against the live backend, built exactly like the + * rm-call ladder's helper (rpc_gsp_rm_alloc_v03_00: hClient, hParent, + * hObject, hClass, status, paramsSize, flags, reserved[4] + params). + * object_is_out selects the reply read: a ROOT alloc returns the + * server-chosen client handle in params[0] (payload +32), anything else + * echoes the client-chosen hObject at +8; status is at +16 in both cases. + */ +static int gsp_rm_binapi07_backend_alloc(u32 h_client, u32 h_parent, + u32 h_object, u32 h_class, + const void *params, u32 params_len, + bool object_is_out, + u32 *out_object, u32 *out_status) +{ + u8 *areq; + void *arsp = NULL; + u32 arsp_len = 0, st = 0xffffffff, ho = 0; + int ret; + + areq = kzalloc(32 + params_len, GFP_KERNEL); + if (!areq) + return -ENOMEM; + memcpy(areq + 0, &h_client, 4); + memcpy(areq + 4, &h_parent, 4); + memcpy(areq + 8, &h_object, 4); + memcpy(areq + 12, &h_class, 4); + memcpy(areq + 20, ¶ms_len, 4); + if (params_len) + memcpy(areq + 32, params, params_len); + + ret = gsp_phys_backend_rpc(GSPRM_RPC_FUNC_GSP_RM_ALLOC, areq, + 32 + params_len, &arsp, &arsp_len, 500); + kfree(areq); + if (!ret && arsp) { + if (arsp_len >= 20) + memcpy(&st, (u8 *)arsp + 16, 4); + if (object_is_out) { + if (arsp_len >= 36) + memcpy(&ho, (u8 *)arsp + 32, 4); + } else if (arsp_len >= 12) { + memcpy(&ho, (u8 *)arsp + 8, 4); + } + } + kfree(arsp); + if (out_object) + *out_object = ho; + if (out_status) + *out_status = st; + return ret; +} + +/* + * Build the oracle object on the live session: a class-0x2081 child of the + * session's internal subdevice, since the server dispatches the method from + * the OBJECT's class (the subdevice attempt alone answered + * NV_ERR_NOT_SUPPORTED, 0x56, and a fresh root client at runtime answered + * NV_ERR_INVALID_CLIENT, 0x23 - the ladder "adopts" the server's own + * internal client instead of creating one). A small candidate list of + * likely-free handles in that client's namespace is tried; the first + * success wins. Returns 0 with *out_object set. + */ +static int gsp_rm_binapi07_backend_setup(struct gsp_emu_context *ctx, + struct gsp_phys_backend_static_config *snap, + u32 *out_object) +{ + static const u32 cand[] = { 0xabcd2081u, 0xabcd2001u, 0xabcd2a2au, + 0xabcd5f5fu, 0xabcd1234u }; + u8 p4[4] = { 0, 0, 0, 0 }; + u32 st = 0xffffffff, obj = 0; + int i, ret; + + for (i = 0; i < (int)ARRAY_SIZE(cand); i++) { + u32 want = cand[i]; + + obj = want; + ret = gsp_rm_binapi07_backend_alloc(snap->h_internal_client, + snap->h_internal_subdevice, + want, 0x2081, p4, 4, + false, &obj, &st); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT oracle BINAPI alloc attempt %d: hObject=0x%x ret=%d status=0x%x", + i, want, ret, st); + if (!ret && st == 0 && obj) { + *out_object = obj; + return 0; + } + } + return -EIO; +} + +/* * Bounded logging for the P3 forward counters: the first four fallbacks (so a * session that cannot forward is visible) and every 64th event. Never called * when the parameter is off. @@ -2728,12 +3819,37 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, u32 out_params_size; int ret; + bool guest_handles = false; + /* Wave 1 whitelist: compile-time side-effect-free queries only. */ switch (req->cmd) { case TU104_CTRL_BUS_GET_PCI_BAR_INFO: case TU104_CTRL_BUS_GET_INFO_V2: case TU104_CTRL_FB_GET_INFO_V2: case TU104_CTRL_INTERNAL_GMMU_GET_STATIC_INFO: + if (!gsp_phys_ctrl_forward_enabled()) + return false; + break; + case 0x20810107: /* NV2081_BINAPI reserved-interface method 7 */ + /* + * EXPERIMENT (r519): the guest GL core allocates a BINAPI + * object (class 0x2081) and sends this 2-byte query during + * 3D-surface init; our {0,0} generic reply leaves the + * driver's feature flag clear and the EGL surface init tears + * down (EGL_BAD_ALLOC at eglMakeCurrent). The guest-side + * handles (hClient 0xc1d..., a 0xbeef object) have no backend + * counterpart: allocate a real BINAPI object (class 0x2081) on + * the live session once, then send the query to it. The first + * attempt (subdevice translation) came back NV_ERR_NOT_SUPPORTED + * (0x56) - the server dispatches the method by the OBJECT's + * class, so a subdevice cannot answer a BINAPI method. The + * reply is logged either way; an error falls back to the + * emulated generic success until the semantics are known. + * Gated by its own parameter so the P3 whitelist can stay off. + */ + if (!gsp_rm_binapi07_fwd_param) + return false; + guest_handles = true; break; default: return false; @@ -2745,24 +3861,44 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, if (req->paramsSize > GSPRM_MAX_SINGLE_ELEM_DATA - GSP_RM_CTRL_HDR_SIZE) goto fallback; + /* The experiment is scoped to the observed shape: 2-byte in/out. */ + if (guest_handles && req->paramsSize != 2) + goto fallback; + /* A valid snapshot implies a READY session and the adopted handles. */ if (gsp_phys_backend_static_get(&snap)) goto fallback; + if (guest_handles && !gsp_rm_binapi07_be_object) { + u32 obj = 0; + + if (gsp_rm_binapi07_backend_setup(ctx, &snap, &obj)) + goto fallback; + /* + * The object lives in the internal client's namespace; the + * control is sent as that client (snap.h_internal_client). + */ + gsp_rm_binapi07_be_client = snap.h_internal_client; + gsp_rm_binapi07_be_object = obj; + } + /* Translate the guest's emulated internal handles; any other handle * domain stays emulated. */ - if (req->hClient != TU104_H_INTERNAL_CLIENT || - (req->hObject != TU104_H_INTERNAL_DEVICE && - req->hObject != TU104_H_INTERNAL_SUBDEVICE)) + if (!guest_handles && + (req->hClient != TU104_H_INTERNAL_CLIENT || + (req->hObject != TU104_H_INTERNAL_DEVICE && + req->hObject != TU104_H_INTERNAL_SUBDEVICE))) goto fallback; fwd_req = kmalloc(GSP_RM_CTRL_HDR_SIZE + req->paramsSize, GFP_KERNEL); if (!fwd_req) goto fallback; memset(&fwd_hdr, 0, sizeof(fwd_hdr)); - fwd_hdr.hClient = snap.h_internal_client; - fwd_hdr.hObject = req->hObject == TU104_H_INTERNAL_DEVICE ? - snap.h_internal_device : snap.h_internal_subdevice; + fwd_hdr.hClient = guest_handles ? gsp_rm_binapi07_be_client : + snap.h_internal_client; + fwd_hdr.hObject = guest_handles ? gsp_rm_binapi07_be_object : + (req->hObject == TU104_H_INTERNAL_DEVICE ? + snap.h_internal_device : snap.h_internal_subdevice); fwd_hdr.cmd = req->cmd; fwd_hdr.paramsSize = req->paramsSize; fwd_hdr.flags = req->flags; @@ -2782,6 +3918,27 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, } /* + * r519 experiment evidence: log the live GSP's own answer to the + * BINAPI query (status + first params bytes) before deciding. + */ + if (req->cmd == 0x20810107) { + u32 st = 0xffffffff, ps = 0xffffffff; + u8 b[8] = { 0 }; + + if (rsp && rsp_len >= GSP_RM_CTRL_HDR_SIZE) { + memcpy(&st, (u8 *)rsp + 12, 4); + memcpy(&ps, (u8 *)rsp + 16, 4); + if (ps <= 8 && ps <= rsp_len - GSP_RM_CTRL_HDR_SIZE) + memcpy(b, (u8 *)rsp + GSP_RM_CTRL_HDR_SIZE, + ps); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT 0x20810107 forwarded to live GSP: rsp_len=%u status=0x%x outSize=%u bytes=%02x %02x %02x %02x %02x %02x %02x %02x", + rsp_len, st, ps, b[0], b[1], b[2], b[3], b[4], + b[5], b[6], b[7]); + } + + /* * The reply is rpc_gsp_rm_control_v03_00: status at +12, paramsSize * at +16, echoed params at +24 - the same 24-byte prefix the emulated * handler builds below. The RPC-level rpc_result was already checked @@ -2827,6 +3984,101 @@ fallback: return false; } +/* + * r489: a GSP_RM_CONTROL reply whose rpc record exceeds the driver's + * pRpc->maxRpcSize must be sent with the GSP's continuation-record protocol + * (nv gsp_rpc.c _rpcWriteContinuationEntries): the FIRST record carries + * exactly maxRpcSize bytes of [rpc header + ctrl header + params]; each + * following NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD record carries up to + * maxRpcSize - sizeof(hdr) bytes of the remainder. The driver's + * _issueRpcLarge consumes the first frame as min(bufSize, maxRpcSize) and + * then requires one continuation record per remaining chunk - without them + * it times out ("rpcRecvPoll timedout for fn 71 continuation record") and + * the control fails (NV_ERR_INVALID_PARAMETER), which is how the GL VASpace + * construction died on cmd 0x2080a0a4 (paramsSize 67392 > maxRpcSize). + * + * maxRpcSize is the guest driver's GSP_MSG_QUEUE_RPC_SIZE_MAX; the value is + * pinned by the driver's own arithmetic: for the 0x2080a0a4 reply + * (full_param_size = 32 + 24 + 67392 = 67448) it reported remainingSize + * 0x940 = 2368 after the first frame, i.e. maxRpcSize = 67448 - 2368 = + * 65080. A build whose constant differs shows up immediately as a + * mismatched continuation count in the guest log; adjust here. + */ +#define GSPRM_CRM_CONT_MAXRPC 65080u + +static int gsp_rm_send_rm_control(struct gsp_emu_context *ctx, u32 sequence, + const void *data, u32 data_len) +{ + u32 total = sizeof(struct gsprm_rpc_header) + data_len; + const u8 *p = data; + u32 off = 0; + int ret; + + if (total <= GSPRM_CRM_CONT_MAXRPC) + return gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, + sequence, 0, data, data_len); + + /* Frame 1: rpc.length = maxRpcSize. */ + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, + p, GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)); + if (ret) + return ret; + off = GSPRM_CRM_CONT_MAXRPC - sizeof(struct gsprm_rpc_header); + + /* Continuations: rpc.length = hdr + chunk. */ + while (off < data_len) { + u32 chunk = data_len - off; + + if (chunk > GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)) + chunk = GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header); + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_CONTINUATION_RECORD, + sequence, 0, p + off, chunk); + if (ret) + return ret; + off += chunk; + } + return 0; +} + +/* + * Real-GSP KGR static-data snapshots (mode-35 boot capture; see + * gsp_phys_kgr_snap_get in gsp_phys_booter.c). The physical backend asks the + * live GSP the real client's KGR query set at boot and retains the replies; + * serve the identical bytes for the same commands so the guest's GL/3D stack + * reads the device's true GR static values. Concrete consumer: the X + * driver's 3D-engine selection maps GR_GET_INFO index SM_VERSION (0x0c) + * through a table and fails with "Failed to select a 3D engine" when it + * reads back as SM_VERSION_NONE (0); the live GSP's KGR_GET_INFO reply + * carries the real value (0x75 on TU104). Returns true when the reply was + * served from the snapshot (out_params_size set); false falls through to the + * emulated handler unchanged. + */ +static bool gsp_rm_kgr_snap_serve(struct gsp_emu_context *ctx, u32 cmd, + u32 req_params_size, + u8 *out_params, u32 out_len, + u32 *out_params_size) +{ + u32 n = 0; + int ret; + + ret = gsp_phys_kgr_snap_get(cmd, out_params, out_len, &n); + if (ret || !n) + return false; + + /* Never answer with more bytes than the guest asked for. */ + if (n > req_params_size) + return false; + + *out_params_size = n; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: KGR cmd=0x%x served from the real-GSP snapshot (%u bytes)", + cmd, n); + return true; +} + static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, const u8 *elem, u32 sequence) { @@ -2861,13 +4113,26 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, * return means nothing was sent and the emulated switch below runs * byte-identically to the parameter-off path. */ - if (gsp_phys_ctrl_forward_enabled() && + if ((gsp_phys_ctrl_forward_enabled() || gsp_rm_binapi07_fwd_param) && gsp_rm_ctrl_forward(ctx, &req, req_data + GSP_RM_CTRL_HDR_SIZE, reply, reply_data_len, sequence)) { kfree(reply); return; } + /* + * KGR static-data snapshots (mode-35 boot capture): the live GSP's own + * replies to the real client's KGR query set are served byte-identically + * for the same commands, so the guest's GL/3D stack sees the device's + * true GR static values. Falls through to the emulated switch when no + * snapshot exists (backend not READY, other boot mode, cmd not captured). + */ + if (gsp_rm_kgr_snap_serve(ctx, req.cmd, req.paramsSize, + reply + GSP_RM_CTRL_HDR_SIZE, + reply_data_len - GSP_RM_CTRL_HDR_SIZE, + &out_params_size)) + goto rm_control_reply; + switch (req.cmd) { case TU104_CTRL_FIFO_GET_DEVICE_INFO_TABLE: gsp_rm_handle_fifo_device_table(ctx, sequence, &req, @@ -2877,7 +4142,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; case 0x90f10106: /* NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES */ - gsp_rm_handle_copy_server_pdes(ctx, req_data + GSP_RM_CTRL_HDR_SIZE, + gsp_rm_handle_copy_server_pdes(ctx, req.hObject, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize); out_status = 0; out_params_size = 0; @@ -2947,6 +4213,38 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); break; + case 0xc3700104: /* NVC370_CTRL_CMD_GET_CHANNEL_INFO */ + gsp_rm_handle_display_channel_info(ctx, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0x20800a5d: /* NV2080_CTRL_CMD_INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES */ + gsp_rm_handle_display_active_devices(ctx, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0x801707: /* NV0080_CTRL_CMD_FIFO_GET_ENGINE_CONTEXT_PROPERTIES */ + gsp_rm_handle_fifo_engine_ctx_props(ctx, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0x730102: /* NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS */ + case 0x730287: /* NV0073_CTRL_CMD_SPECIFIC_GET_ALL_HEAD_MASK */ + case 0x7302ad: /* NV0073_CTRL_CMD_SPECIFIC_GET_VALID_HEAD_WINDOW_ASSIGNMENT */ + case 0x730107: /* NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED */ + case 0x73028b: /* NV0073_CTRL_CMD_SPECIFIC_OR_GET_INFO */ + case 0x730250: /* NV0073_CTRL_CMD_SPECIFIC_GET_CONNECTOR_DATA */ + case 0x731140: /* NV0073_CTRL_CMD_DFP_GET_INFO */ + gsp_rm_handle_display_heads(ctx, req.cmd, req.paramsSize, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + case TU104_CTRL_CE_GET_FAULT_METHOD_BUFFER_SIZE: gsp_rm_handle_ce_get_fault_method_buffer_size(ctx, sequence, &out_status, @@ -3085,6 +4383,32 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, "vgsp-rm: promote[%u] bufferId=%u bInitialize=%u bNonmapped=%u physAttr=0x%x pa=0x%llx va=0x%llx size=0x%llx", i, bid, e[30], e[31], attr, pa, eva, esz); + /* + * r820: register the driver's own statement + * of this kernel buffer's backing (r817 + * Xorg GL init: the page-table walk answered + * a stale zero page for the pushbuffer VA + * while this entry named the real one; a + * userspace pin scanner can never cover + * kernel-owned buffers). See + * gsp_sema_service_promote_add(). + */ + gsp_sema_service_promote_add(eva, pa, esz, + attr, bid); + /* + * SESSION 153-CONTINUED-95: keep the latest + * MAIN (0) / PATCH (2) statement per engine + * for the launch-time CTX-BIND receipt (the + * bInitialize flag is the client's "server, + * initialize this buffer" request; the + * emulation answers OK but never installs + * the buffer on the engine channel - the + * receipt makes that visible). + */ + gsp_sema_service_ctx_promote_note(engine_type, + bid, pa, eva, + esz, attr, + e[30]); } } out_status = 0; @@ -3150,6 +4474,149 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + case 0x20800afe: { /* NV2080_CTRL_CMD_INTERNAL_INIT_USER_SHARED_DATA + * + * The guest kernel driver hands the GSP-RM the physical + * address of the RUSD (RM user shared data) page it + * allocated; on real hardware the GSP registers the + * memdesc and keeps the page's sequence-guarded fields + * (clocks, BAR1, utilisation, ...) populated, and 0 + * de-initialises it. Nothing here performs that upkeep, + * and consumers treat a zeroed sequence word as "data + * invalid" and skip the fields, so the correct emulated + * behaviour is to acknowledge the registration, echo the + * request (the caller may read the address back) and log + * the address once so the page can be correlated with + * guest RAM during diagnosis. */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 8) { + u64 phys_addr; + + memcpy(&phys_addr, pp, 8); + memcpy(rp, pp, req.paramsSize); + rm->rusd_phys_addr = phys_addr; + rm->rusd_init_seen = true; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: INIT_USER_SHARED_DATA physAddr=0x%llx (RUSD registered; fields left at sequence 0 = 'data invalid', consumers skip)", + phys_addr); + out_status = 0; + out_params_size = req.paramsSize; + } else { + out_status = 0; + out_params_size = 0; + } + break; + } + + case 0x2080a097: { /* client-pending engine-context query (session 75) + * + * libnvidia-eglcore issues this via NVOS54 and then + * POLLS the reply's params words: the request sets + * +0x204 = 0, and the caller re-issues the whole + * control every ~1.8 s while its readiness check + * stays negative (observed: 32-36 polls against the + * generic all-zero reply, then the client gives up + * and vkCreateDevice returns -4). The params are an + * in/out record block: u32 count at +0, then count + * 16-byte records, then +0x204, and the client reads + * back +0, +0x204, +0x488 and +0x48c after the call + * (libnvidia-eglcore.so.570.144 file offsets + * 0xecd45f..0xecd475). The readiness wait itself + * lives in the client's engine loop + * (0xd6ef40..0xd6f0ab polling byte +0x1b038 of the + * engine object, which a91ea0 recomputes from many + * state fields), so the reply words below are + * runtime-tunable for a single-cycle sweep without + * rebuilding: the wall2_a097_* module parameters. + * + * Reply: verbatim echo (the request records are the + * client's own query data) with the tunable words + * applied. Defaults keep the completion-style flag. */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 0x490 && + req.paramsSize + GSP_RM_CTRL_HDR_SIZE <= + GSPRM_MAX_SINGLE_ELEM_DATA) { + u32 count = 0, flag; + static const u32 *const recp[8] = { + &gsp_rm_a097_rec0, &gsp_rm_a097_rec1, + &gsp_rm_a097_rec2, &gsp_rm_a097_rec3, + &gsp_rm_a097_rec4, &gsp_rm_a097_rec5, + &gsp_rm_a097_rec6, &gsp_rm_a097_rec7, + }; + int i; + + memcpy(rp, pp, req.paramsSize); + memcpy(&count, pp, 4); + memcpy(&flag, pp + 0x204, 4); + if (gsp_rm_a097_count != ~0u) + *(u32 *)(rp + 0) = gsp_rm_a097_count; + for (i = 0; i < 8; i++) { + if (*recp[i] != ~0u) + *(u32 *)(rp + 4 + i * 4) = *recp[i]; + } + *(u32 *)(rp + 0x204) = gsp_rm_a097_flag; + *(u32 *)(rp + 0x488) = gsp_rm_a097_v488; + *(u32 *)(rp + 0x48c) = gsp_rm_a097_v48c; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x2080a097: echoed %u bytes (count=%u, req flag=%u) -> flag=%u v488=0x%x v48c=0x%x", + req.paramsSize, count, flag, + gsp_rm_a097_flag, gsp_rm_a097_v488, + gsp_rm_a097_v48c); + out_status = 0; + out_params_size = req.paramsSize; + } else { + out_status = 0; + out_params_size = 0; + } + break; + } + + case 0x90960106: { /* NV04_VID_HEAP_CONTROL block-index range (m112b) */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 12) { + memcpy(rp, pp, req.paramsSize); + *(u32 *)(rp + 0) = gsp_rm_9096_lo; + *(u32 *)(rp + 4) = gsp_rm_9096_hi; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x90960106 -> first=%u last=%u", + gsp_rm_9096_lo, gsp_rm_9096_hi); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + + case 0x90960107: { /* NV04_VID_HEAP_CONTROL block entry by index (m112b) */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 56) { + memcpy(rp, pp, req.paramsSize); + *(u64 *)(rp + 0x00) = (u64)gsp_rm_9096_ent_addr; + *(u64 *)(rp + 0x08) = (u64)gsp_rm_9096_ent_size; + *(u32 *)(rp + 0x28) = gsp_rm_9096_ent_type; + *(u8 *)(rp + 0x30) = (u8)gsp_rm_9096_ent_flag; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x90960107 -> addr=0x%x size=0x%x type=%u flag=%u", + gsp_rm_9096_ent_addr, + gsp_rm_9096_ent_size, + gsp_rm_9096_ent_type, + gsp_rm_9096_ent_flag); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + default: /* Unknown control command - return success with zeroed params. * Do NOT echo the request: pure queries pass uninitialized @@ -3172,6 +4639,52 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, out_params_size = req.paramsSize; break; + case 0x20810107: { /* NV2081_BINAPI reserved-interface method 7 */ + /* + * SESSION 153-CONTINUED-17: the guest GL client + * (libnvidia-glcore 570.144, file offset 0xFC74A6) sends this + * 2-byte query right after allocating its BINAPI object and + * CONSUMES REPLY BYTE [1] AS A FEATURE FLAG: + * + * glcore+0xFC74D9: cmpb $0x0, 0x131(%rsp) + * glcore+0xFC74E1: setne 0x203(%r14) ; flag = replies[1] + * + * and the flag gates 3D method emission into the command + * buffer: + * + * glcore+0xB5B48B: mov 0x1248(%r14),%rax + * cmpb $0x0,0x203(%rax) + * je <skip> + * glcore+0xB5B49B: movabs $0xf020010e66,%rdi + * mov %rdi,(0x80(%r14)) ; emit + advance + * + * With the historical {0,0} generic reply the flag stays 0 and + * the client's submission pipeline never activates (no GL + * channel, no pushes, black window). binapi07_rep selects the + * reply word (LE u16) at runtime so one boot can test the + * feature without a rebuild: 0x0101 = both bytes set, + * 0x0100 = byte[1] only. Default 0 reproduces {0,0}. + */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 2 && req.paramsSize <= 16) { + memcpy(rp, pp, req.paramsSize); + rp[0] = gsp_rm_binapi07_rep & 0xff; + rp[1] = (gsp_rm_binapi07_rep >> 8) & 0xff; + out_params_size = req.paramsSize; + } else { + out_params_size = 0; + } + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI 0x20810107 reply in=%u rep=0x%04x bytes=%02x %02x", + req.paramsSize, gsp_rm_binapi07_rep, + rp[0], rp[1]); + out_status = 0; + break; + } + case TU104_CTRL_I2C_TRANSACTION: /* I2C transactions have variable-length FINN-serialized * params: an all-zero reply fails the driver's DeserializeUp @@ -3188,6 +4701,27 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + /* + * r487/r489 THE GL/VULKAN VASPACE WALL (Xorg/EGL/GZDOOM, every + * session to date): the reply must follow the GSP's own large-reply + * framing. r487 clamped the echo to fit one 64 KB record; the guest + * then refused the record ("Incorrect message length 65528") and, + * when it fit, waited for CONTINUATION records that never came + * ("rpcRecvPoll timedout for fn 71 continuation record + * (remainingSize=0x940)") - rpc.c's _issueRpcLarge consumes the + * first frame up to pRpc->maxRpcSize (65080) and then requires + * NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD frames for the rest + * (the GSP's own _rpcWriteContinuationEntries). The size is left + * unclamped here; gsp_rm_send_rm_control() frames it per that + * protocol. Root evidence: cmd 0x2080a0a4 (PERF_VPSTATES_INFO, + * paramsSize 67392, FINN-serialized) sent during Xorg VASpace + * construction; its failure made vmmCreateVaspace return ffff and + * killed GL - and, after the length was fixed, the 3D-engine + * selection ("Failed to select a 3D engine") until the framing was + * correct. + */ + +rm_control_reply: /* Build the reply header */ memcpy(reply + 0, &req.hClient, 4); memcpy(reply + 4, &req.hObject, 4); @@ -3207,8 +4741,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, out_status, dt_us); } - if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, - reply, GSP_RM_CTRL_HDR_SIZE + out_params_size)) + if (gsp_rm_send_rm_control(ctx, sequence, reply, + GSP_RM_CTRL_HDR_SIZE + out_params_size)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: RM_CONTROL reply send failed cmd=0x%x", req.cmd); else @@ -3218,6 +4752,61 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, } /* + * r510gl: DUP_OBJECT (function 21) - NVOS55_PARAMETERS echo with status=NV_OK. + * + * The guest driver dups an object into a second client at EGL/GLX + * MakeCurrent time (memory and VASpace objects: the GL context's client + * shares the device's allocations), and the historical generic-SUCCESS + * default replied with an EMPTY payload - the driver's reply must carry the + * params struct back (rpc_dup_object_v03_00 = NVOS55_PARAMETERS + * {hClient, hParent, hObject, hClientSrc, hObjectSrc, flags, status}), so + * an empty reply failed the call and the driver tore the whole EGL session + * down (guest witness: 'RPC UNKNOWN(21) seq=0 len=60 -> generic SUCCESS', + * immediately followed by the session's FREE cascade and + * eglMakeCurrent -> EGL_BAD_ALLOC (0x3003), 2026-10-23 r510). + * + * The emulation does not model per-object state for these classes (controls + * and allocs are answered from the request), so the echo with NV_OK is the + * complete answer; the dup'd handle is used by later calls exactly like the + * source handle. + */ +static void gsp_rm_handle_dup_object(struct gsp_emu_context *ctx, + const u8 *elem, u32 sequence) +{ + struct gsp_rm_emu *rm = ctx->rm_emu; + const struct gsprm_rpc_header *hdr = + (const struct gsprm_rpc_header *)(elem + GSPRM_ELEM_HDR_SIZE); + const u8 *req = elem + GSPRM_ELEM_HDR_SIZE + sizeof(*hdr); + u8 params[28]; /* NVOS55_PARAMETERS_v03_00 */ + u32 hDstClient = 0, hDstObject = 0, hSrcClient = 0, hSrcObject = 0; + + if (hdr->length >= sizeof(*hdr) + sizeof(params)) + memcpy(params, req, sizeof(params)); + else + memset(params, 0, sizeof(params)); + + memcpy(&hDstClient, params + 0, 4); + memcpy(&hDstObject, params + 8, 4); + memcpy(&hSrcClient, params + 12, 4); + memcpy(&hSrcObject, params + 16, 4); + memset(params + 24, 0, 4); /* status = NV_OK */ + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DUP_OBJECT hDstClient=0x%x hDstObject=0x%x hSrcClient=0x%x hSrcObject=0x%x seq=%u -> NV_OK echo", + hDstClient, hDstObject, hSrcClient, hSrcObject, sequence); + + /* Small delay before the reply: same reply-matching race mitigation as + * FREE/UNLOADING. */ + udelay(2000); + + if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_DUP_OBJECT, sequence, 0, + params, sizeof(params))) + nv_vgpu_err(ctx->vgpu, "vgsp-rm: DUP_OBJECT reply send failed"); + else + rm->rpcs_replied++; +} + +/* * GSP_RM_FREE (function 10) - NVOS00_PARAMETERS echo with status=NV_OK. * * rpc_gsp_free: { hRoot, hObjectParent, hObjectOld, status }. Echo the @@ -3254,11 +4843,19 @@ static void gsp_rm_handle_free(struct gsp_emu_context *ctx, const u8 *elem, * client's notifier events (hSrcResource 0) linger across runs and the * fallback wake-up nudge would hit them. Also retire a client's events * when its root object is freed (hObjOld == hRoot). + * + * m176: the hEvent match must also match the owning client. Handle + * values are recycled across runs, so a late FREE of an old client's + * hEvent must never retire a new client's record with the same numeric + * handle - the new client's completion wake-up would be lost and it + * would wait forever (late-client stall class). hRoot is the freeing + * client's own handle in NVOS00_PARAMETERS. */ for (i = 0; i < rm->events_n; i++) { if (!rm->events[i].valid) continue; - if (rm->events[i].h_event == hObjOld || + if ((rm->events[i].h_event == hObjOld && + (!hRoot || rm->events[i].h_client == hRoot)) || (hObjOld && hObjOld == hRoot && rm->events[i].h_client == hRoot)) { rm->events[i].valid = false; @@ -3456,14 +5053,18 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) return; case GSPRM_RPC_FUNC_CONTINUATION_RECORD: - /* CONTINUATION_RECORD: driver expects a reply. Send SUCCESS. */ - nv_vgpu_info(ctx->vgpu, "vgsp-rm: async RPC %s(%u) seq=%u len=%u -> SUCCESS reply", - gsp_rm_rpc_name(function), function, sequence, - hdr->length); - if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) - nv_vgpu_err(ctx->vgpu, "vgsp-rm: reply send failed fn=%u", function); - else - rm->rpcs_replied++; + /* + * r489: a CONTINUATION_RECORD from the driver is ONE-WAY data + * - the tail of a multi-frame request (its _issueRpcLarge + * sends frame 1 then continuation records). The old reply + * here was a guess ("driver expects a reply"); it injected an + * extra fn-71 record into the reply stream that the driver + * drained as an event ("Unexpected RPC event from GPU0: 0x47") + * and it corrupted the framing of our own large replies + * (which use fn-71 continuations the other way). Consume + * silently: the driver waits only for the reply to frame 1. + */ + rm->rpcs_replied++; return; case GSPRM_RPC_FUNC_SET_GUEST_SYSTEM_INFO: @@ -3486,6 +5087,10 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) gsp_rm_handle_free(ctx, elem, sequence); return; + case GSPRM_RPC_FUNC_DUP_OBJECT: + gsp_rm_handle_dup_object(ctx, elem, sequence); + return; + case GSPRM_RPC_FUNC_SET_PAGE_DIRECTORY: gsp_rm_handle_set_page_directory(ctx, elem, sequence, hdr->length); return; @@ -3499,6 +5104,23 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) * For vGPU, we just acknowledge success. */ nv_vgpu_info(ctx->vgpu, "vgsp-rm: RPC UPDATE_BAR_PDE(%u) seq=%u len=%u -> SUCCESS", function, sequence, hdr->length); + { + const u8 *pl = (const u8 *)hdr + sizeof(*hdr); + u32 n = min_t(u32, hdr->length, 64); + char dump[256]; + int o = 0; + u32 i; + + for (i = 0; i < n; i += 4) { + u32 v = 0; + + memcpy(&v, pl + i, 4); + o += snprintf(dump + o, sizeof(dump) - o, + "%08x ", v); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: bar-pde payload: %s", dump); + } if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: UPDATE_BAR_PDE reply send failed"); else @@ -3527,6 +5149,31 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) nv_vgpu_info(ctx->vgpu, "vgsp-rm: RPC %s(%u) seq=%u len=%u -> generic SUCCESS reply (unknown #%llu)", gsp_rm_rpc_name(function), function, sequence, hdr->length, rm->unknown_rpcs); + /* + * Bounded payload dump for the first unknown RPCs: the vGPU + * guest maps its client VA spaces through some RPC; the + * mapping data (va pages / PAs / memdesc handles) must be + * visible here to be captured. Never echo it back (pure + * queries would eat their own stack garbage). + */ + if (rm->unknown_rpcs <= 96 && hdr->length) { + const u8 *pl = (const u8 *)hdr + sizeof(*hdr); + u32 n = min_t(u32, hdr->length, 64); + char dump[256]; + int o = 0; + u32 i; + + for (i = 0; i < n; i += 4) { + u32 v = 0; + + memcpy(&v, pl + i, 4); + o += snprintf(dump + o, sizeof(dump) - o, + "%08x ", v); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: unknown fn=%u seq=%u len=%u payload: %s", + function, sequence, hdr->length, dump); + } if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: reply send failed fn=%u", function); else @@ -3658,8 +5305,40 @@ void gsp_rm_emu_fini(struct gsp_emu_context *ctx) void gsp_rm_emu_reset(struct gsp_emu_context *ctx) { struct gsp_rm_emu *rm = ctx->rm_emu; + u32 i, stale; - if (!rm || !rm->parsed) + if (!rm) + return; + + /* + * m176: a reset means the emulated GSP is gone and every RM client + * with it (probe/unload, engine reset). Event records left valid from + * the previous session would be posted to with stale + * (hClient, hEvent) pairs - the guest RM asserts in CliGetEventInfo + * and drops the record - and a recycled hEvent can absorb a wake-up + * meant for the new client. Drop them regardless of parse state. + */ + stale = 0; + for (i = 0; i < GSPRM_MAX_EVENTS; i++) { + if (rm->events[i].valid) { + rm->events[i].valid = false; + stale++; + } + } + if (stale) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: reset dropped %u stale event record(s)", + stale); + + /* + * m177: RM reload means a new guest boot: the guest's direct-map base + * and every content-scanned translation may have changed. Raise the + * boot-scoped wipe (executed on the sema svc thread; client session + * flushes intentionally keep that state - see svc_session_flush). + */ + gsp_sema_service_guest_reset(); + + if (!rm->parsed) return; nv_vgpu_info(ctx->vgpu, "vgsp-rm: reset (rpcs=%llu replied=%llu unknown=%llu)", diff --git a/kernel/gsp_rm_emulation.h b/kernel/gsp_rm_emulation.h index e464809..c0eecf8 100644 --- a/kernel/gsp_rm_emulation.h +++ b/kernel/gsp_rm_emulation.h @@ -45,6 +45,7 @@ struct gsp_emu_context; #define GSPRM_RPC_FUNC_GSP_RM_CONTROL 76 #define GSPRM_RPC_FUNC_GSP_RM_ALLOC 103 #define GSPRM_RPC_FUNC_FREE 10 +#define GSPRM_RPC_FUNC_DUP_OBJECT 21 #define GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER 47 #define GSPRM_RPC_FUNC_UPDATE_BAR_PDE 70 #define GSPRM_RPC_EVENT_POST_EVENT 0x1003 @@ -183,6 +184,10 @@ struct gsp_rm_emu { u64 ctrl_fwd_fallback; /* forward attempt used emulation */ u64 ctrl_fwd_timeout; /* fallbacks caused by a backend timeout */ + /* NV2080_CTRL_CMD_INTERNAL_INIT_USER_SHARED_DATA bookkeeping */ + u64 rusd_phys_addr; /* last registered RUSD phys address */ + bool rusd_init_seen; /* a registration was acknowledged */ + /* * NV01_EVENT objects (class 0x5, allocated via class 0x79 in the * guest ABI). Each carries {hSrcResource, notifyIndex, data}; the @@ -197,17 +202,44 @@ struct gsp_rm_emu { u32 notify_index; u32 data; bool valid; + unsigned long last_post; /* m171: jiffies of the last posted record */ } events[GSPRM_MAX_EVENTS]; u32 events_n; u64 events_posted; u64 events_dropped; - u32 event_burst; /* event records written since the last reply */ + u32 event_burst; /* event records written in the current window */ + unsigned long event_burst_at; /* m171: burst window start (time-based reset) */ }; +/* + * r576 DIAG: per-boot witness budget for GSP event-post payloads (default 0 + * = off). Nonzero logs every EVENT_POST_EVENT record actually sent (slot, + * hClient, hEvent, notifyIndex, data, send rc) and the sentinel case of a + * post_all_events call that found nothing postable - the evidence needed + * when a client parks waiting for a completion wake-up that never arrives. + */ +extern unsigned gsp_rm_ev_log_max; + +/* + * r577 experiment knobs (runtime-writable; default 0 = legacy). See the + * definitions in gsp_rm_emulation.c: event record sequence, bNotifyList, + * hClient override - the A/B space for the guest-RM event drop. + */ +extern unsigned gsp_rm_ev_seq; +extern unsigned gsp_rm_ev_nlist; +extern unsigned gsp_rm_ev_cl; +extern unsigned gsp_rm_ev_func; + int gsp_rm_emu_init(struct nv_vgpu_device *vgpu, struct gsp_emu_context *ctx); void gsp_rm_emu_fini(struct gsp_emu_context *ctx); int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src); int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx); +/* + * r623: deliver a synthetic OS_ERROR_LOG (0x1006) record to the guest's RM + * queue (the engine-fault report a real GSP sends). Debug/experiment only. + */ +int gsp_rm_send_os_error_log(struct nv_vgpu_device *vgpu, u32 except_type, + u32 runlist_id, u32 chid, const char *engine); int gsp_rm_post_all_events(struct nv_vgpu_device *vgpu); int gsp_rm_dump_state(struct nv_vgpu_device *vgpu); int gsp_rm_scan_guest_ram(struct nv_vgpu_device *vgpu, u64 needle, diff --git a/kernel/gsp_rm_tu104.h b/kernel/gsp_rm_tu104.h index b5ee4d7..3382939 100644 --- a/kernel/gsp_rm_tu104.h +++ b/kernel/gsp_rm_tu104.h @@ -754,7 +754,18 @@ static const struct tu104_internal_device_info tu104_internal_device_table[] = { TU104_CE_CAPS0_SYSMEM | \ TU104_CE_CAPS0_P2P | \ TU104_CE_CAPS0_BL_GT_64K) -#define TU104_CE_CAPS1 0x00 +/* + * Caps byte 1: the 2D/3D copy path requires a CE that supports the + * blocklinear access modes (hardware bug 200288104): the X driver's + * nvCEChooseEngine skips every CE without CE_SUPPORTS_NONPIPELINED_BL + * (byte1 0x01; and PIPELINED_BL 0x02 for the preferred pass) and then + * fails with "Failed to find a suitable copy object" - the observed wall + * after the DMA pushbuffer test started passing. Turing's CEs are fully + * capable (pipelined + non-pipelined BL + pitchlinear): byte1 = 0x03. + * (Reference: ctrl2080ce.h NV2080_CTRL_CE_CAPS_CE_SUPPORTS_*_BL 1:0x01/ + * 1:0x02.) + */ +#define TU104_CE_CAPS1 0x03 /* ENGINE_TYPE_COPY0 = 0x9, COPY(i) = COPY0 + i (cl2080_notification.h) */ #define TU104_ENGINE_TYPE_COPY0 0x00000009 diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index 2019a35..234932c 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -44,9 +44,12 @@ #include <linux/workqueue.h> #include <linux/mm.h> #include <linux/vmalloc.h> +#include <linux/debugfs.h> +#include <linux/seq_file.h> #include <asm/unaligned.h> #include "nvidia-vgpu-vfio.h" +#include "bar_emulation.h" #include "gsp_sema_service.h" #include "gsp_rm_emulation.h" #include "gsp_phys_booter.h" @@ -80,19 +83,49 @@ static inline u64 svc_fb_shadow_size(const struct nv_vgpu_device *vgpu) * hole), so the top of RAM is around 6 GB, not 4 GB. Anything below this is * RAM; the emulated vidmem lives in its own aperture and must not be confused * with it. + * + * r933: the 8 GiB GL guest's top of RAM is 0x280000000, and the GL client's + * kick/push/ring pages were measured in the 6-8 GiB band (kick gpa + * 0x20ad9b000 > the old 0x180000000 6 GiB cap), so every walk leaf there was + * rejected by the aperture bound (walk-probe rc=-2 pa=0x229c1a000) and only + * per-session pin injection could read them. The compile-time constant sizes + * the fixed tables (hot-page bitmap); the live acceptance bound is + * svc_pa_max(), driven by the sema_gr_scan_top module parameter so a cycle on + * a smaller guest shrinks both together. */ -#define SEMA_PA_MAX 0x180000000ULL +#define SEMA_PA_MAX 0x280000000ULL #define SEMA_SCAN_PAGES_MAX 0x20000 /* 512 MB worth of pages per pass */ /* Plausible GPU VA window for the client's system-memory objects (Turing). */ #define SEMA_VA_LO 0x100000000ULL #define SEMA_VA_HI 0x2000000000ULL +/* + * session 79: the GR/GL ucode init releases target the RM's LOW aperture + * (the same 0x4000000-0x4ffffff VA family as the push/ring windows): the + * live GL init push carries `2005c017 04923060 ... 02000001` (0x5c-form, + * va=0x4923060, payload 1, RELEASE) and `200426c0 00000000 04b19b10 + * 00000001 10000000` (REPORT_SEM group, va=0x4b19b10, payload 1, + * RELEASE) - both rejected by the sysmem-only range check, so the GR + * completion never applied and the NvOgl engine-init wait spun forever. + * Accept the low aperture for release decode (the pushes are only + * decoded from trusted ring reads, so the false-positive risk the + * sysmem bound guarded against does not apply here). + */ +#define SVC_LOW_REL_VA_LO 0x4000000ULL + /* Canonical kernel virtual address range (x86_64). */ #define KVA_LO 0xffff800000000000ULL #define KVA_HI 0xffffc00000000000ULL /* + * r534: guest-read granularity for the full-RAM discovery scan + * (svc_scan_pass). Hoisted here because the scan is defined before the + * push-apply window that used to own the define. + */ +#define SVC_SCAN_CHUNK (64 * 1024) /* 16 pages per guest read */ + +/* * INC_METHOD header for the CE semaphore group. Observed value 0x20038090: * bits [31:28] opcode/const, [23:16] count = 3, lower bits encode * (address >> 2) == 0x240 plus subchannel flags (0x8000 in the observed @@ -160,8 +193,110 @@ struct sema_map_entry { #define SVC_USERD_GP_GET 0x88 #define SVC_USERD_GP_PUT 0x8c +/* + * r496b: guest-RAM ring content scan (see gr_ram_cursor in struct + * svc_channel) and the VRAM USERD shape scan (svc_chan_userd). One scan + * slice per service pass so the service thread stays responsive. + */ +#define SVC_GR_RAM_SLICE (512ULL << 20) /* guest RAM per pass */ +/* + * SESSION 149 R15 (M1.1b): 24 slices cover two full sweeps of the 6 GB + * guest before the boot-time budget retires; the put-triggered keep-alive + * in the pump grants up to SVC_GR_RAM_REFRESH_MAX further budgets. + */ +#define SVC_GR_RAM_PASS_MAX 24 /* slices before a budget retires */ +#define SVC_GR_RAM_REFRESH_MAX 3 /* keep-alive budget refreshes */ +#define SVC_GR_RAM_REFRESH_MS 10000 /* minimum spacing between them */ +#define SVC_GR_RAM_WIN (2ULL << 20) /* allocation window pinned */ +/* + * r979 (SESSION 153-CONTINUED-21): the kick family's per-frame submit + * record pushbuffer VA. Constant across every measured boot (r908 + * documented the GL client ring pair {push=0x1203c0000,len=64} then + * {push=0x1203c0000,len=32}; r966-r978 every kick exec and every + * canonical push read named exactly this VA). Used to discriminate the + * kick ring page from shape lookalikes in the RAM ring scan - r978 + * latched a foreign table page (pa=0x104fa9000) whose entry pair passed + * the shape gate and named a bogus 256 KB push (0x12048b824/len 0x40000) + * while the true ring (pa=0x1168ce000, found by scanning for the + * record's own entry pair) sat later in RAM scan order. + */ +#define SVC_GL_KICK_PUSH_VA 0x1203c0000ULL +/* SESSION 154 (M-GL1.2): watched USERD candidates for the liveness latch. */ +#define SVC_GR_USERD_WATCH 64 +/* + * SESSION 153-CONTINUED-20: a kick-family (h 0xbfef00xx) watched candidate + * whose PUT does not move for this long is a frozen pool lookalike (the + * r972/r973 descriptor-pool class: put=7 forever) - dropped so the ~50 + * frozen entries can never crowd out the real advancing control block. + */ +#define SVC_KICK_WATCH_STALE_MS 20000 + +/* + * r908: guest-RAM ring-scan top (guest physical). The guest's usable RAM + * can extend above the old 6 GB service cap: the 8 GiB guest's e820 top is + * 0x280000000, and the GL client channel's GPFIFO ring page was measured + * at gpa 0x231e5d000 (readable: svc_read -> vfio_dma_rw answered there). + * Reads past mapped RAM (the 0x80000000-0x100000000 PCI hole, or a smaller + * guest) fail fast, so a too-large top costs a few failed probes per sweep. + * The module parameter is the per-cycle guard: shrink it to the boot's real + * RAM top. + * + * r933: this parameter is now the single live PA acceptance bound for the + * whole service (svc_pa_max), not just the ring scan - the walk leaves in + * the 6-8 GiB band were exactly what the old separate 6 GB SEMA_PA_MAX + * rejected (the GL client's kick pages). SEMA_PA_MAX remains the + * compile-time maximum that sizes fixed tables; the param clamps it. + */ +unsigned long long gsp_sema_gr_scan_top = 0x280000000ULL; +module_param_named(sema_gr_scan_top, gsp_sema_gr_scan_top, ullong, 0644); +#define SVC_GR_SCAN_TOP_MAX 0x400000000ULL /* 16 GiB sanity clamp */ + +/* + * Live guest-PA acceptance bound: min(param, compile-time max). Every + * runtime range check that used to compare against SEMA_PA_MAX uses this + * instead, so raising the ceiling for a bigger guest (r933: 8 GiB guest, + * GL client's pages above 6 GiB) is one module parameter, and shrinking it + * for a smaller guest shrinks the scans and the walk acceptance together. + * Never 0 (a zero param falls back to the compile-time max). + */ +static inline u64 svc_pa_max(void) +{ + u64 top = READ_ONCE(gsp_sema_gr_scan_top); + + if (!top || top > SEMA_PA_MAX) + return SEMA_PA_MAX; + return top; +} + +/* + * Ring-scan slice budget: two full sweeps at the configured top, with the + * original 24-slice floor and a u8-safe cap (gr_ram_passes). + */ +static u32 svc_gr_ram_pass_budget(void) +{ + u64 top = gsp_sema_gr_scan_top; + u32 slices; + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top <= SVC_GR_RAM_SLICE) + return SVC_GR_RAM_PASS_MAX; + slices = (u32)(top / SVC_GR_RAM_SLICE) * 2; + if (slices < SVC_GR_RAM_PASS_MAX) + slices = SVC_GR_RAM_PASS_MAX; + if (slices > 250) + slices = 250; + return slices; +} + struct svc_channel { bool valid; + u32 session_gen; /* m176: session generation this entry was + * created in; learned PA state (counter_pa/ + * release_pa) is only inherited on a re- + * registration within the same generation, so + * a recycled handle from a previous client can + * never keep the dead session's write targets */ struct nv_vgpu_device *vgpu; /* owning mdev (guest session); the * table is keyed by (vgpu, h_object) * because two guests' stock allocators @@ -169,6 +304,18 @@ struct svc_channel { u32 h_object; u32 engine_type; u32 cid; /* RM channel id (doorbell token vector) */ + u32 h_vaspace; /* r835: the client VAS this channel was + * allocated in (NV_CHANNEL_ALLOC_PARAMS. + * hVASpace). Channel-scoped VA resolution + * consults this VAS's page tables FIRST: + * every client's RM window sits at the same + * VAs (0x1200xxxxx) and the global VA-keyed + * maps conflate them - the X server's ring + * VA resolved through the GL client's page + * tables (walk-probe vas h=0xcaf00005 + * answered 0x120090000) and its push stream + * decoded as zero pages, wedging X the same + * way the RC watchdog's ring was starved. */ u8 ce_index; /* rank among the live CE channels (creation order) */ u8 gr_index; /* rank among the live GR channels (creation order) */ u64 gpfifo_va; @@ -188,6 +335,28 @@ struct svc_channel { * reset (PUT dropping below put_prev) resyncs * last_get to slot 0 so the live first push is * processed instead of skipped. */ + bool gr_pub_armed; /* mm2 (r207/r208 second-client hang): a GR + * channel's completion word may only receive a + * GP_PUT publish after the channel has been + * proven to belong to the CURRENT session. A + * GR ring/USERD surviving in its fixed VRAM + * slot from the previous session reads a + * residual put at discovery (e.g. 109/112); + * publishing that residual put into the + * channel's own word (8c634144 made the target + * release_pa - the client's waited word) while + * the stale ring is consumed poisons the fresh + * client's UVM tracking semaphore: its + * completed_value observes a far-forward + * payload, the later reset/release sequence is + * a >MAX_JUMP (2*entries) downward observation, + * and the fresh session's first waits wedge + * (matmul: one GR push per channel then 242 s + * silent). Armed when a GR channel is seen with + * put <= 1 (pristine ring) or when the fresh- + * put-drop rewind proves a session switch; the + * rewind also voids any publish state the + * stale pass advanced. */ bool logged; /* one state log per channel */ bool pool_rel_seen; /* ISSUE-0002 r70: a pool release word for * this channel has been read through a @@ -232,6 +401,10 @@ struct svc_channel { * window (observed: one 8 s grace per push, * serializing the runtime init) */ DECLARE_BITMAP(slot_done_epoch, SVC_GPFIFO_MAX_BITS); + u8 slot_pend_tries[SVC_GPFIFO_MAX_BITS]; /* fail4 r308: bounded + * re-holds of a denied device-dst copy slot */ + u32 counter_gen; /* r311: session generation that authored the + * completion word (guards the residue zero) */ /* GPFIFO slots already consumed in the current * put epoch (the client's put value since it * last wrapped). A pin-batch rewind replays @@ -270,6 +443,21 @@ struct svc_channel { * the end with no operand left pending, so a * replay suppresses every op. When clear, the * done bitmap decides per op (retry path). */ + bool ever_ops; /* r611: this channel has decoded at least one + * transfer op (copy/fill), i.e. it carries the + * client's stream-ordered memory work. The r607 + * stream-order watermark considers only such + * channels: a sibling GR LAUNCH channel's ring + * legitimately holds future launches, and + * deferring on those creates mutual deferral + * between the client's own launch channels + * (streams2: launches on h=0x5c00001f/23 each + * saw the other's unconsumed entry, burned the + * 15 s cap and were refused fail-closed - the + * r610c stream class red). Measured r606/r609/ + * r610c: op-decoding channels are exactly the + * copy ranks (0x5c000059 family) and the CE + * ranks; launch channels decode no ops. */ u64 slot_ops_hash[SVC_GPFIFO_MAX_BITS]; /* Content witness for slot_ops_ident: the * FNV-1a hash of the push bytes the identity @@ -290,6 +478,15 @@ struct svc_channel { unsigned long push_hold_until; /* absolute deadline of the bounded * work-before-release hold for an * unreadable CE entry (0 = none) */ + unsigned long gr_setup_hold_until; /* SESSION 153-CONTINUED-54: absolute + * deadline of the GR setup-push hold: + * an unreadable/all-zero MME-setup + * (or >= 12 KB) GR push stays pending + * past the 1-failed-rescue consume + * until the demanded guest pin lands + * (0 = none) */ + u64 gr_setup_hold_va; /* entry VA the setup hold was armed for */ + u32 gr_setup_hold_idx; /* GPFIFO slot the setup hold was armed for */ unsigned long defer_since; /* first failure of the pending GR entry * (0 = none): the entry is left pending * for the GR grace window so a late @@ -302,6 +499,10 @@ struct svc_channel { * publish targets counter_pa instead, so both * families of the counter page get written * (SA-COUNTER FORENSICS-grv7-counter F1) */ + u64 release_va; /* r378: the release VA release_pa was learned + * from, kept so a late write (stale-recover + * zero) can re-validate the target through + * gsp_sema_write_guard before it lands */ u32 counter_last; /* last value published to counter_pa */ u64 pushpg_va; /* last push page logged as pin-vs-window * divergent (rate limit: one line per page) */ @@ -349,12 +550,252 @@ struct svc_channel { * even if still unresolved - bounded zeros, never * a hang); m156: re-held while the window is * live so a busy forward gets its real retry */ + /* + * r509: one-shot verdict of the absolute bound. The expiry call + * pushes the deadline (so the hold window restarts) and latches this + * flag; the r351 retry-forward block consumes it within the same + * pass. Without the latch the two call sites in one pass could not + * both observe the expiry (the first push of the deadline made the + * second call see a fresh window - r213-r215's arming bug in + * reverse). + */ + bool gr_hold_bound_hit; + unsigned long gr_hold_deadline; /* GRFIX FIX1b: hard expiry of the data- + * stalled hold for gr_hold_dead_va. + * Fixed once on the first hold of that + * launch push VA and NEVER moved while + * the launch stays held; when it passes + * the launch proceeds (pre-8b058884 + * behaviour), so the guest can never + * wait unboundedly. Keyed by the launch + * VA, not the GPFIFO slot: pin-batch + * rewinds re-decode one launch at + * different slot indices (r213: + * entry 107 -> 107 -> 99), which + * re-armed a slot-keyed deadline on + * every pass. */ + u64 gr_hold_dead_va; /* launch push VA the bound belongs to */ + unsigned long gr_hold_dead_since; /* first-hold stamp (waited_ms) */ + bool gr_hold_exp_logged; /* one GRFILL-EXPIRED line per launch */ + u64 opdef_hold_dead_va; /* r361: launch push VA the opdef bound + * belongs to (slot-keyed `since` is + * defeated by rewind replays) */ + unsigned long opdef_hold_deadline; /* absolute opdef hold expiry */ + bool opdef_hold_exp_logged; /* one OPDEF-EXPIRED line per launch */ u32 gr_rehold_idx; /* m156: slot whose next decode must force a * fresh launch capture (a held entry is * re-decoded as a replay, which suppresses * the capture; forcing it lets a deferred * launch be forwarded once the engine's * in-flight work completes) */ + u32 gr_nocap_idx; /* m164 CLASS-2: slot in the no-capture hold + * (a readable launch push whose decode saw + * QMD words but built no candidate) */ + u32 gr_nocap_replay;/* 1 = window opened on a replay decode */ + u64 gr_nocap_va; /* push VA the window was opened for */ + unsigned long gr_nocap_since; /* absolute first-seen stamp */ + /* + * r496b (X/GL kernel-RM sysmem channel): ring discovery by content in + * guest RAM. The CPU RM maps sysmem objects locally (no + * SET_PAGE_DIRECTORY, no rm_mem record carrying the ring VA), so + * svc_lookup_trusted cannot name the ring page; the driver's kickoff + * GPFIFO entries {pushbuffer VA, len} identify it instead. The scan + * is sliced (SVC_GR_RAM_SLICE per pass) and cursor-resumed so the + * service thread stays responsive. + */ + u64 gr_ram_cursor; /* resume point of the content scan */ + u8 gr_ram_passes; /* slices walked so far */ + bool gr_ram_logged; /* one discovery line per channel */ + /* + * SESSION 149 R15 (M1.1b keep-alive): the boot-time slice budget can + * exhaust long before the client creates its ring objects (r886: 16 + * slices spent, the GL client's ring never scanned, no pairing). A + * GPFIFO put advance proves the client is live, so grant a fresh + * bounded budget - spaced and capped so a never-found ring cannot + * keep the scan hot. + */ + u32 gr_ram_kick; /* put value at the last budget refresh */ + u8 gr_ram_refreshes; /* budget refreshes granted so far */ + unsigned long gr_ram_refresh_at; /* next eligible refresh */ + /* + * r912 (M-GL1.2): guest-RAM USERD shape scan for a GR channel whose + * ring was content-validated in RAM (gr_ram_ring). Such a client's + * USERD is a system-memory object like its ring (the kernel-RM/X GL + * path), so the FB-shadow shape scan can never reach it and the pump + * never iterates its entries. Same sliced/cursor discipline as the + * ring scan; the cursor is persisted immediately because the + * stalled-userd pump path skips the end-of-pass write-back. + */ + u64 gr_ram_userd_cursor; /* resume point of the USERD scan */ + u8 gr_ram_userd_passes; /* slices walked so far */ + bool gr_ram_userd_logged; /* one discovery/give-up line */ + /* + * SESSION 154 (M-GL1.2): liveness confirmation for the RAM USERD + * latch. The shape+entry gates alone admitted frozen or slowly + * churning lookalike pages (r913: pa 0x7c000 adopted at put=1 only + * survived by luck; r914: pa 0x9e200 adopted at put=12 get=8 and + * its put%2==0 collapsed the walk - submissions stayed 0 while the + * client retried for minutes). A candidate is WATCHED across passes + * instead: only a block whose PUT takes the client's own single +1 + * step is adopted as the live control block - the measured live + * block advances 1->2->3 while the lookalikes jump by larger + * amounts or churn to zero. + */ + u64 gr_ram_userd_watch[SVC_GR_USERD_WATCH]; /* watched PAs */ + u32 gr_ram_userd_watch_put[SVC_GR_USERD_WATCH]; /* last PUT seen */ + u8 gr_ram_userd_watch_hits[SVC_GR_USERD_WATCH]; /* +1/+2 streak */ + /* + * SESSION 153-CONTINUED-20: jiffies when the entry was added or last + * stepped; the stale-entry timeout drops a candidate whose PUT has + * not moved for SVC_KICK_WATCH_STALE_MS (the frozen descriptor-pool + * class the kick-family watch exists to defeat). + */ + unsigned long gr_ram_userd_watch_at[SVC_GR_USERD_WATCH]; + u8 gr_ram_userd_watch_n; + u32 gr_ram_userd_watch_log; /* bounded add-log counter */ + /* + * SESSION 153-CONTINUED-34 (BAR0-shadow USERD): the GL client's channel + * allocation declares its USERD in the FB aperture (0x18fc00-class + * offsets) but the guest's GP_PUT writes land in the emulated BAR0 + * shadow at that offset. Watch that block across passes: a nonzero PUT + * (or a PUT step) is the client's own live write; adopt it as the + * channel's USERD in the BAR0 aperture so the pump iterates the ring. + * bar0_userd_at is the first-sighting stamp; a stable nonzero PUT is + * adopted after a short patience window so a single-submission client + * (waiting on that submission) cannot deadlock the watch. + */ + u64 bar0_userd_watch_pa; /* watched offset (0 = none) */ + u32 bar0_userd_watch_put; /* PUT at the first sighting */ + unsigned long bar0_userd_watch_at; /* jiffies at the first sighting */ + u8 bar0_userd_log; /* bounded per-channel log counter */ + /* + * SESSION 153-CONTINUED-20: bounded budget re-arms for the + * kick-family watch sweep. When every watch entry has gone stale + * (frozen pool) and the slice budget is spent, one re-arm keeps the + * sweep alive so the real block can still be found; capped like the + * reject counter so a dead channel stops sweeping in bounded time. + */ + u8 gr_ram_userd_kick_sweeps; + u32 gr_ram_put_seeded; /* last PUT the walk seed consumed */ + /* + * SESSION 154 (M-GL1.2): speculative-latch confirmation state. The + * RAM USERD scan latches a gated candidate immediately; the pump + * confirms it from its own per-pass PUT read (a +1/+2 step), and + * unlatches a churning/frozen block (see svc_pump_channels). + */ + u32 gr_ram_userd_latch_put; /* PUT at the speculative latch */ + unsigned long gr_ram_userd_latch_at; /* jiffies at the latch */ + bool gr_ram_userd_confirmed; /* +1/+2 step observed */ + /* + * SESSION 153-CONT-5: rejected-latch memory. A speculative latch the + * pump rejects (frozen or jumped PUT) used to exhaust the RAM scan + * budget permanently: r936's kick channel latched a FROZEN lookalike + * (put=7 get=1 on a driver struct page), the pump rejected it twice, + * the 20-slice scan budget retired, and the channel went dark with no + * re-latch path - kick retires (and their publish writes) stopped for + * the rest of the session. Remember the rejected PA and its PUT: the + * scan skips that exact (pa, put) pair silently, re-adopts the block + * the moment its PUT moves (content change = new evidence), and each + * rejection re-arms the scan budget so one stale window can never kill + * the pipeline. The re-arm count is bounded (64: the r937 rate of + * ~4 rejections/min makes that >15 min of continuous retire activity + * on a client that only ever wrote one kick, while a truly dead + * channel still stops sweeping in bounded time). + */ + u64 gr_ram_userd_reject_pa; /* PA of the rejected latch */ + u32 gr_ram_userd_reject_put; /* its PUT at rejection time */ + u8 gr_ram_userd_rejects; /* rejections seen (bound) */ + /* + * SESSION 154 (M-GL1.2): content anchor for a client push whose + * translation went stale (r913: a validated pin kept answering for + * a reused page and the wrong-VAS walk answer was implausible while + * the real push bytes sat in RAM on a page no directory covered). + * Scan guest RAM for the client's release-group signature (sliced, + * cursor-resumed, like the ring scans), validate the page and pin + * it so the pump's retry decodes the real bytes. + */ + u64 gr_anchor_cursor; /* resume point of the anchor scan */ + u64 gr_anchor_va; /* entry VA the sweep is for */ + u8 gr_anchor_sweeps; /* completed sweeps for that VA */ + bool gr_anchor_logged; /* one start line per channel */ + /* + * r496e: true when the ring was found by the RAM content scan (a + * kernel-RM sysmem channel - the X/GL 3D channel). Such a channel's + * pushes are 2D/3D method streams the emulation does not execute; + * the correct model is consume-and-apply-releases: no QMD capture, + * no launch forward, no no-capture hold. Also allows an all-zero + * NOP push (the driver's pushbuffer sanity test writes exactly one + * zero dword) to be consumed as the no-op it is. + */ + bool gr_ram_ring; + /* + * SESSION 153-CONTINUED-19 (kick-family USERD acceptance): ring + * liveness re-validation pace for the kick channel (h 0xbfef00xx). + * While the kick USERD stays unresolved this channel's ring latch is + * re-checked each pass (at most once a second) and re-armed when it + * no longer decodes entries - see svc_kick_ring_revalidate. + */ + unsigned long gr_kick_ring_reval_at; + /* + * r496b: the FB-scan USERD (see svc_userd_fb_scan). Latched when the + * registered/ring-derived candidates all fail; the scan validates the + * hit against the ring entries the put claims, so a foreign channel's + * USERD cannot be adopted. Bounded retries: the driver kicks off a + * few passes after channel registration, so a scan before the first + * put must not retire the discovery. + */ + u8 userd_scans; + /* + * SESSION 153-CONTINUED-37: time-paced re-run state for the FB USERD + * shape scan (r496b) and its pristine-block fallback. The old + * count-only cap spent every retry within ~64 ms of registration + * (pump pass cadence), while the X driver's pushbuffer test kicks its + * first GP_PUT hundreds of ms later and its VAS registers at ~1 s - + * so on unlucky timings the budget was gone before any candidate + * existed and the 10 s DMA test read GET=0 forever ("Failed to + * initialize DMA", r1013). One scan per 25 ms up to 512 keeps the + * discovery alive across the whole client window. + */ + unsigned long fb_userd_scan_at; /* jiffies of the last scan */ + u16 fb_userd_scan_n; /* scans spent since registration */ + /* + * r828: the RC watchdog's notifier VA, captured from the class2d + * SET_NOTIFY_A/B methods on subchannel 3 and zeroed at the NOTIFY + * (see svc_notify2d_write). Survives the pass snapshot via the + * write-back list. + */ + u64 notify2d_va; + /* r829: bounded count of zero-entry (bad-translation) holds; see the + * zero-entry hold in svc_pump_channels. Persists via the write-back. */ + u32 zero_held; + /* r831: the registered userd value is a real guest PA (the driver's + * own declaration; e.g. the RC watchdog channel). The ring auto-pin's + * ring+0x2000 convention must not overwrite it. */ + bool userd_reg; + /* + * r471: the completion-counter publish must never advance the + * guest's word past a push whose content was still unwritten when + * the entry was consumed (the streams_events staging-reuse + * corruption: the publish satisfied the client's D2H wait before + * its copy ran, and the readback drained a staging buffer that + * still held the PREVIOUS copy's bytes). Lowest ring slot consumed + * with all-zero content; 0 = none. Cleared when a later decode of + * a GR push reads non-zero content (the content landed - the rewinds + * re-read entries constantly, so this heals within a pass or two). + */ + u32 gr_bad_put; + /* + * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. A + * channel that cannot make forward progress (last_get pinned behind + * put, or a residual session put that never drops) is detected here + * and recovered instead of spinning. All fields MUST be in the pump + * snapshot write-back list (GRFIX NOTE). + */ + u64 wd_last_get; /* last_get at the window start */ + u64 wd_put; /* put at the window start */ + u64 wd_sub; /* submissions at the window start */ + unsigned long wd_since; /* window start (0 = no window) */ + unsigned long wd_rewind_off_until; /* pin-batch rewind cooldown */ }; /* @@ -410,6 +851,12 @@ struct svc_vaspace { u64 pdb; u32 entries; /* root entry count */ int ap; /* SVC_APERTURE_* of the directory itself */ + unsigned long stamp; /* r459: last registration/use; LRU evict + * when the table is full (a stale entry from + * a dead client otherwise blocks every new + * client after the 8th registration: the + * mid-boot walk failure that made the 10th+ + * client's copies fail rc=-2) */ }; struct svc_lat { @@ -439,6 +886,12 @@ static void svc_mark_hot(u64 pa) * push-buffer base) and the channel (its slot). */ #define SVC_PB_MAX 32 +/* + * R10 (patch A): svc_pb[] slot namespace for GR (eng==0) channels' + * content-anchored ring windows (svc_pb_pair_gr). CE slots are + * 4*ce_index (0..124), so the high bit keeps the namespaces apart. + */ +#define SVC_PB_GR_SLOT_BASE 0x80000000u struct svc_pb_hit { u64 pa_page; u64 va; /* push-buffer base VA, once paired with a channel */ @@ -471,18 +924,52 @@ static u32 svc_ring_n; * guest's last-channel close is also the global empty transition. */ static u8 svc_sess_flush; +static u8 svc_guest_reset_req; /* m177: guest RM reload; wiped on svc thread */ +static u8 svc_sess_flush_forced; /* execute the next flush past the live-channel skip + * (ISSUE-0002 client-session boundary aid; see + * gsp_sema_service_sess_flush_force) */ +/* + * SESSION 148 perf tracing: sema_trace_h selects one channel handle; the + * pump then logs a compact state line for it once a second, and + * svc_counter_publish logs why it skipped (last_get gate / arming gate) or + * the value it wrote. This is the instrument for locating the 22.4 s/frame + * glxgears pacer without guessing at windows. + */ +unsigned long long gsp_sema_trace_h; +module_param_named(sema_trace_h, gsp_sema_trace_h, ullong, 0644); +MODULE_PARM_DESC(sema_trace_h, + "Trace one channel handle (0x...; 0=off): 1/s state + publish gates"); +static unsigned long svc_perf_last; static u8 svc_sess_active; /* a channel of a session has registered (until flush) */ +static u32 svc_sess_gen; /* m176: bumped by every executed session flush; + * stamps channels/scrubbers so per-session learned + * state is never inherited by a recycled handle */ static struct completion svc_sess_flush_done; /* a pending flush has executed */ static unsigned long svc_ring_done; /* CE ranks whose ring is pinned (session state) */ /* Per-guest session records (one per open mdev; see SVC_MAX_CHANNELS for the channel cap). */ #define SVC_MAX_GUESTS 8 +/* + * SESSION 153-CONTINUED-34: PCFG doorbell window in the emulated BAR0 shadow. + * Work-submission doorbells the guest pokes after writing a GPFIFO entry can + * land anywhere in the PCFG window (0x88000..0x89000), not only at the + * virtual-function doorbell the trap path used to dispatch (0x88704). Because + * the BAR0 shadow is direct-mapped, most writes never reach the trap path at + * all, so the service thread also watches the window for changed words and + * feeds each new value to gsp_sema_service_doorbell (token format unchanged; + * the call only wakes the pump earlier - the pump itself is per-pass). + */ +#define SVC_BAR0_PCFG_LO 0x88000u +#define SVC_BAR0_PCFG_HI 0x89000u +#define SVC_BAR0_PCFG_WORDS ((SVC_BAR0_PCFG_HI - SVC_BAR0_PCFG_LO) / 4u) struct svc_guest_sess { struct nv_vgpu_device *vgpu; /* NULL = free slot */ bool active; /* a channel of this guest is * registered until this guest's * last channel closes */ u64 last_build; /* per-guest svc_rebuild freshness */ + u32 pcfg_snap[SVC_BAR0_PCFG_WORDS]; + bool pcfg_have; /* snapshot taken for this guest */ }; static struct svc_guest_sess svc_guests[SVC_MAX_GUESTS]; static DEFINE_SPINLOCK(svc_guest_lock); @@ -555,6 +1042,7 @@ struct svc_scrub { u64 submitted; /* blocks submitted since seeding */ u64 authored; /* last authored finish value */ u32 pa_gen; /* scanner registration generation (logged) */ + u32 session_gen; /* m176: session generation of the last registration */ u32 log_budget; }; /* @@ -600,7 +1088,15 @@ static u64 svc_pb_last; static u32 svc_pb_idle; /* consecutive passes with no new base */ static u32 svc_pb_last_n; /* bases before the active pass */ static unsigned long svc_pb_done; /* CE ranks whose push buffer is pinned */ +/* R10 (patch A): GR ranks whose ring-anchored window is paired. */ +static unsigned long svc_pb_gr_done; static unsigned long svc_gr_ring_done; /* GR ranks whose ring is pinned */ +/* SESSION 133: ranks that already consumed their one shadow ring scan. */ +static unsigned long svc_gr_ring_scan_done; +static unsigned long svc_gr_ring_scan_last; /* rate limit for the scan */ +/* SESSION 138: bounded witness for the GR request builder's page list and + * the fill's candidate set (one launch's wiring printed per budget slice). */ +static u32 svc_gr_pages_log; static bool svc_pb_first_push(const u8 *buf) { @@ -722,14 +1218,45 @@ struct sema_svc { u64 sem256_log; /* 256-byte op trace budget */ u64 op_tr_log; /* operand targeted-resolve budget */ u32 op_tr_pending; /* an operand's targeted scan is in progress */ + u32 op_owed; /* r540: a denied transfer op + * was never re-run because the + * per-slot pend cap was hit; the + * slot must stay INCOMPLETE so a + * later trusted/injected pass + * runs it (the r539 5-minute + * device-init stall class: the + * client's GRCE H2D was denied + * while untrusted, capped, then + * replay-suppressed forever and + * the copy never executed) */ + u64 op_owed_log; /* r540: bounded owe witness */ u64 copy_watch_log; /* sema_copy_watch log budget */ + u64 copy_dst_hold; /* m169: device-dst copies held for + * want of a pin (stale-table guard) */ + u64 copy_dst_relaxed; /* m169b: guards released without + * a pin after the bounded wait */ u64 copy_pat_log; /* sema_copy_pat log budget */ u64 copy_dst_log; /* sema_copy_dst log budget */ u32 op_tr_src; /* provenance of last svc_op_translate */ u32 sem_sw_log; /* sema-sw release-write log budget */ + u32 sem_sw_xg_log; /* sema-sw X/GL-window budget */ u64 push_pat_log; /* push-content pattern budget */ u64 dma_skip_log; /* unrecognized-DMA budget */ u64 xfer_raw_log; /* operandless transfer raw-dump budget */ + u64 fill_ram_log; /* r361 physical-RAM fill build budget */ + u64 pushtrace_all; /* fail4: de-windowed push decode + * trace (every push, bounded) */ + u64 opadd_log; /* fail4: every op appended */ + u64 op2d_raw_log; /* r458: raw 2D-shaped copy pushes */ + u64 dmatrace_log; /* fail4: denied-transfer trace */ + u64 pend_dbg_log; /* fail4 r305: pending/hold witness */ + u64 snap_gate_log; /* fail4 r311: capture-gate witness */ + u64 copyent_log; /* fail4: svc_copy entry trace */ + u64 bigrun_log; /* fail4: big method-data run headers */ + u64 lid_hdr_log; /* fail4: LOAD_INLINE_DATA headers */ + u64 skiprun_log; /* fail4: silently skipped data runs */ + u64 mystery_log; /* fail4: moderate op=1 runs on big pushes */ + u64 next_hdr_log; /* fail4: header after each big LID run */ u64 grce_log; /* GRCE inline-write budget */ u32 cur_ch; /* channel of the op being run */ u64 entry_log; /* per-push summary budget */ @@ -743,9 +1270,20 @@ struct sema_svc { u64 entry_norel_log; u64 ops_denied; /* work refused: implausible stream */ u64 false_trans; /* resolvable VA mapped to non-push */ + u64 kick_badpg; /* r977: kick-family push reads + * that lacked the canonical + * submit-record prefix (wrong + * page; forced the rescue) */ + u64 rescue_map_ok; /* SESSION 148: content-verified + * map-rescue successes (GR + CE) */ u64 sem_write_reresolve; /* writes re-resolved after read-back */ u64 pool_alt_applied; /* pool releases fanned to a * displaced pin (ISSUE-0002) */ + u64 pool_dual_applied; /* r539: pool releases fanned + * to the secondary candidate page + * (pin/walk disagreement; the + * r538 init-stall class) */ + u64 pool_dual_log; /* r539 fan-out log budget */ u64 pool_residue; /* fresh-channel first releases whose * pool word read far ahead of the * payload (previous session's @@ -775,10 +1313,19 @@ struct sema_svc { * expectation advanced, write queued */ u64 gpfifo_held; /* passes an entry was left pending * for a translation/grace window */ + u64 gr_nocap_held; /* m164: GR holds opened for a + * launch push whose decode saw QMD + * words but built no candidate */ + u64 gr_nocap_consumed; /* m164: no-capture windows that + * expired and consumed the entry */ + u64 gr_owed_resolved; /* r548: unwound deferred launches + * software-completed at hold expiry + * instead of consumed silently */ u64 gpfifo_starved; /* entries consumed without a * deliverable exact slot */ u64 counter_rel_applied; /* releases to registered counters */ u64 counter_rel_dbg; /* rel-apply diagnostic budget */ + u64 crel_entry_dbg; /* C1: rel-apply entry budget, 512/session */ u64 counter_noslot_dbg; /* "no slot registered" budget, * separate so it cannot hide the * written/skip lines */ @@ -800,9 +1347,13 @@ struct sema_svc { u64 opdef_held; u64 opdef_applied; u64 opdef_expired; + u64 opdef_delivered_late; /* stall fix: set delivered at consume */ + u64 opdef_drop_lost; /* withheld releases dropped unapplied */ u64 vidmem_writes; /* semaphores written in vidmem */ u64 gpfifo_log; /* channel-state log budget */ u64 gpfifo_entry_log; /* per-entry log budget */ + u64 walk_dbg; /* C4: GR put/walk witness budget */ + u64 gr_rel_auth_log; /* F1: GR-REL-AUTHOR budget, 256/session */ u64 doorbell_count; /* NV_VIRTUAL_FUNCTION_DOORBELL pokes */ u32 doorbell_token; /* last token written */ u32 doorbell_log; /* token log budget */ @@ -903,7 +1454,43 @@ struct sema_svc { u64 fwd_gr_ok; u64 fwd_gr_fallback; u64 fwd_gr_refused; + u32 fwd_gr_nocand_log; /* SESSION 127: first-N + * request-without-candidate + * witnesses */ + u64 fwd_gr_nordy; /* SESSION 127: backend-not- + * ready request refusals */ + u64 fwd_gr_holdbusy;/* SESSION 127: hold-active + * request busy verdicts */ + u64 fwd_gr_inflight;/* SESSION 127: inflight-bit + * request busy verdicts */ + u32 fwd_gr_enoe_log;/* SESSION 127: first-N fill + * ENOENT page witnesses */ u64 fwd_gr_defer; /* m150: held for data-arg retry */ + u64 fwd_gr_dstpend; /* r541: held for an unexecuted + * device write on a data page */ + u32 fwd_gr_dstpend_log; + u64 fwd_gr_closed; /* r583: fail-closed at the defer + * cap (dstpend still live) */ + u32 fwd_gr_closed_log; + u64 fwd_gr_argdefer; /* r543: held for a partial + * argument set (unresolved arg) */ + u32 fwd_gr_argdefer_log; + u64 fwd_gr_argfw; /* r612: non-degenerate fill + * forwarded at the cap (stalled + * candidates are decoys) */ + u32 fwd_gr_argfw_log; + u64 fwd_gr_cewm; /* r590: held on the CE ring's + * live submission watermark */ + u64 fwd_gr_cewm_cap; /* r590: watermark still + * backlogged at the defer cap */ + u32 fwd_gr_cewm_log; + u64 ce_mutual_skips; /* r638: watermark + * skipped a channel whose + * head entry is wait-gate + * held (a consumer-side + * mutual citation, not a + * producer backlog) */ + u32 ce_mutual_log; u64 fwd_gr_timeout; u64 fwd_gr_fault; u64 fwd_gr_bridge; @@ -914,11 +1501,67 @@ struct sema_svc { * abort, or a release0 page the backend could not observe). */ u64 fwd_gr_sw_rel; + /* + * STALLFREE: failed forwarded launches the pass-top outcome check + * completed in software because no hold owned their release (the + * bounded-completion guarantee; zero on a green run). + */ + u64 fwd_gr_stall_swrel; + /* + * STALLFREE r225: GR channels recovered by the stale-epoch / + * stale-replay wedge watchdog (zero on a green run). + */ + u64 gr_stale_wedge; + u64 fwd_gr_hold_exp; /* GRFIX FIX1: data-stalled holds + * that reached the hard bound and + * were consumed with the software + * release (proof the bound fires + * instead of a stall) */ /* Distinct stalled VA tracking for rate-limited logging. */ #define SVC_STALLED_MAX 256 u64 stalled_vas[SVC_STALLED_MAX]; u32 stalled_va_count; + + /* + * SESSION 153-CONTINUED-56: the armed setup-hold's demand VA set + * (page-aligned, deduped per boot), exposed read-only to the demand + * tool via /sys/kernel/debug/vgsp/sema_setup_demand. State, not a + * log row: under vgsp-sema spam the `stalled VA #N` dmesg rows + * rotate out of the ring before the 3 s harvest reads them, so the + * setup-push pages were lost (r1090 needed a manual demand); the pin + * tool now reads this file directly each pass instead. + * + * SESSION 153-CONTINUED-60: the set is a RING. At 64 entries the + * r1106 MISS wall (push=0xcf3d724 miss=11..17/pass, abort loop + * n=2..9) never converged because every later gsp_sema_demand_note + * (the MISS pages themselves, the attr-prelude program pages) was + * silently dropped once the 64 slots were exhausted -> the pin tool + * never demanded them -> never resolved -> perpetual pre-doorbell + * abort. Capacity is raised to 512 and a full set overwrites the + * oldest slot instead of refusing the note, so the resident set is + * the demand-paging working set and a later note is never lost; + * total/evicted counters are published for observability. + */ +#define SVC_SETUP_DEMAND_MAX 512 + u64 setup_demand_vas[SVC_SETUP_DEMAND_MAX]; + /* + * SESSION 153-CONTINUED-64: the held entry's exact read window. + * The armed setup hold records its entry page with (off = entry va's + * in-page offset, len = the GPFIFO entry length); every other page + * keeps len = 0. The guest pin resolver uses the pair to validate a + * candidate page RANGE against the exact bytes the host will read + * [va, va+len) with the host's own full method-stream parse, so a pin + * is only injected when the host read is guaranteed to parse (kills + * the content-heuristic family learn that served a wrong area in + * r1120: `first8=00040000 00001000 ... plausible=0`). + */ + u8 setup_demand_off[SVC_SETUP_DEMAND_MAX]; + u32 setup_demand_len[SVC_SETUP_DEMAND_MAX]; + u32 setup_demand_count; + u32 setup_demand_head; /* ring write cursor (valid once full) */ + u64 setup_demand_total; /* notes accepted (dedup hits excluded) */ + u64 setup_demand_evicted; /* oldest-slot overwrites */ }; static struct sema_svc svc; @@ -952,6 +1595,180 @@ static void svc_log_stalled_va(u64 va) } /* + * SESSION 153-CONTINUED-56: record one page-aligned VA of the armed + * setup-hold demand set. Same caller as svc_log_stalled_va (the service + * thread); the debugfs reader publishes entries with a write barrier on the + * count, so a reader can never observe a torn entry (the count never + * decreases and an entry is stable once its slot is covered by the count). + * + * SESSION 153-CONTINUED-60: full-set behaviour is ring overwrite (oldest + * slot) instead of drop, so the r1106 MISS pages can never be refused a + * slot by a stale 64-entry set; see the member comment. Entry stores are + * aligned u64 on a 64-bit kernel, so the in-place overwrite cannot tear for + * the seq reader (worst case it sees the previous resident VA once more). + */ +static void svc_setup_demand_record_ex(u64 va, u32 off, u32 len) +{ + u32 i, n = svc.setup_demand_count; + + if (!va) + return; + for (i = 0; i < n; i++) + if (svc.setup_demand_vas[i] == va) { + /* + * SESSION 153-CONTINUED-64: upgrade a plain entry to the + * anchored (off,len) window when the hold re-records it. + * off/len publish order keeps the reader consistent for + * new slots; an in-place upgrade may be seen one pass + * torn (off new, len old) - the guest then validates the + * wrong window once and retries, which is harmless. + */ + if (len && !svc.setup_demand_len[i]) { + svc.setup_demand_off[i] = (u8)off; + smp_wmb(); + svc.setup_demand_len[i] = len; + } + return; + } + if (n < SVC_SETUP_DEMAND_MAX) { + svc.setup_demand_vas[n] = va; + svc.setup_demand_off[n] = (u8)off; + svc.setup_demand_len[n] = len; + smp_wmb(); + svc.setup_demand_count = n + 1; + } else { + static u32 ring_log; + u32 w = svc.setup_demand_head; + + svc.setup_demand_vas[w] = va; + svc.setup_demand_off[w] = (u8)off; + svc.setup_demand_len[w] = len; + svc.setup_demand_head = (w + 1u) % SVC_SETUP_DEMAND_MAX; + svc.setup_demand_evicted++; + if (ring_log < 4u || !(svc.setup_demand_evicted % 4096u)) { + ring_log++; + pr_info("sema-svc: demand ring full; evicting oldest slot %u for va=0x%llx (evicted=%llu total=%llu)\n", + w, va, svc.setup_demand_evicted, + svc.setup_demand_total + 1u); + } + } + svc.setup_demand_total++; +} + +static void svc_setup_demand_record(u64 va) +{ + svc_setup_demand_record_ex(va, 0, 0); +} + +/* + * SESSION 153-CONTINUED-59: cross-file entry for the GR launch builder (see + * gsp_sema_service.h). Records the page of a VA whose window/attr operand + * relocation MISSed, so the pin demand path fetches it before the next + * forward; the builder then either resolves it (RELOC level-2 fallback) or + * aborts the launch pre-doorbell instead of letting the engine read a raw + * unmapped guest VA (the r1104 MMU Fault HUBCLIENT_PD class). + */ +void gsp_sema_demand_note(u64 va) +{ + if (va) + svc_setup_demand_record(va & ~(u64)0xfffu); +} + +/* + * Read-only view of the setup-hold demand set for the pin tool: + * # sema setup-hold demand VAs (page-aligned): N total=T evicted=E + * 0x4021000 + * 0x4001004 0x4 60216 <- anchored entry: entry va, in-page offset, len + * ... + * `total` counts every accepted note (dedup hits excluded); `evicted` counts + * ring overwrites of the oldest slot (SESSION 153-CONTINUED-60). An anchored + * line carries the exact [va, va+len) read window of the held GPFIFO entry + * (SESSION 153-CONTINUED-64). + */ +static int svc_setup_demand_show(struct seq_file *m, void *v) +{ + u32 i, n = READ_ONCE(svc.setup_demand_count); + + seq_printf(m, "# sema setup-hold demand VAs (page-aligned): %u total=%llu evicted=%llu\n", + n, (unsigned long long)svc.setup_demand_total, + (unsigned long long)svc.setup_demand_evicted); + for (i = 0; i < n && i < SVC_SETUP_DEMAND_MAX; i++) { + u32 dl = svc.setup_demand_len[i]; + + if (dl) + seq_printf(m, "0x%llx 0x%x %u\n", + (unsigned long long)svc.setup_demand_vas[i] + + svc.setup_demand_off[i], + (unsigned)svc.setup_demand_off[i], dl); + else + seq_printf(m, "0x%llx\n", + (unsigned long long)svc.setup_demand_vas[i]); + } + return 0; +} + +static int svc_setup_demand_open(struct inode *inode, struct file *file) +{ + return single_open(file, svc_setup_demand_show, NULL); +} + +static const struct file_operations svc_setup_demand_fops = { + .owner = THIS_MODULE, + .open = svc_setup_demand_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* True when this module created the vgsp debugfs root (mirrors cuda bridge). */ +static bool svc_setup_demand_root_owned; + +static int svc_setup_demand_debugfs_init(void) +{ + struct dentry *root, *f; + + svc_setup_demand_root_owned = false; + root = debugfs_lookup("vgsp", NULL); + if (IS_ERR(root)) + root = NULL; + if (!root) { + root = debugfs_create_dir("vgsp", NULL); + if (IS_ERR_OR_NULL(root)) { + pr_warn("sema-svc: debugfs dir vgsp unavailable; sema_setup_demand not exposed\n"); + return -ENOENT; + } + svc_setup_demand_root_owned = true; + } + f = debugfs_create_file("sema_setup_demand", 0444, root, NULL, + &svc_setup_demand_fops); + if (IS_ERR_OR_NULL(f)) { + pr_warn("sema-svc: debugfs sema_setup_demand creation failed\n"); + if (svc_setup_demand_root_owned) + debugfs_lookup_and_remove("vgsp", NULL); + svc_setup_demand_root_owned = false; + return -ENOMEM; + } + dput(root); + pr_info("sema-svc: debugfs /sys/kernel/debug/vgsp/sema_setup_demand ready\n"); + return 0; +} + +static void svc_setup_demand_debugfs_fini(void) +{ + struct dentry *root; + + root = debugfs_lookup("vgsp", NULL); + if (IS_ERR_OR_NULL(root)) + return; + debugfs_lookup_and_remove("sema_setup_demand", root); + dput(root); + if (svc_setup_demand_root_owned) { + debugfs_lookup_and_remove("vgsp", NULL); + svc_setup_demand_root_owned = false; + } +} + +/* * Deferred readability probe for unreadable CE pushes that were consumed via * the fake-release fallback. The pump fakes immediately (work-before-release * cannot hold the channel); this queue re-attempts the read later in the @@ -1012,6 +1829,17 @@ static int svc_write(struct nv_vgpu_device *vgpu, u64 gpa, void *buf, size_t len /* NV_ADDR_SYSMEM / NV_ADDR_FBMEM as carried in the channel alloc params. */ #define SVC_APERTURE_SYS 1 #define SVC_APERTURE_FB 2 +/* + * SESSION 153-CONTINUED-34: the emulated BAR0 (register) shadow as a third + * aperture. The GL client's channel allocation reports its USERD block in the + * FB aperture (NV_ADDR_FBMEM, offset 0x18fc00/0x18fe00/0xfbfc00...) but the + * writes the guest actually makes to its GP_PUT land in the emulated BAR0 + * shadow at that same offset (the vGPU guest driver programs its usermode + * block through the register window). Every other source reads the FB shadow + * at those offsets and finds nothing, so the pump never iterates the + * channel's GPFIFO and the channel retires with submissions=0. + */ +#define SVC_APERTURE_BAR0 3 /* * Aperture selectors are codes (1=SYS, 2=FB), never booleans. The idiom @@ -1023,7 +1851,31 @@ static int svc_write(struct nv_vgpu_device *vgpu, u64 gpa, void *buf, size_t len */ static inline u32 svc_ap_of(int ap) { - return ap == SVC_APERTURE_FB ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + if (ap == SVC_APERTURE_FB) + return SVC_APERTURE_FB; + /* SESSION 153-CONT-34: the BAR0 selector must survive normalisation. */ + if (ap == SVC_APERTURE_BAR0) + return SVC_APERTURE_BAR0; + return SVC_APERTURE_SYS; +} + +/* + * SESSION 153-CONT-34: the emulated BAR0 shadow (the buffer the guest's + * register writes actually land in - it is mmap'd into QEMU and also the + * trap-path write target). NULL when the device has no emulation shadow. + */ +static void *svc_bar0_shadow(struct nv_vgpu_device *vgpu, size_t *size) +{ + struct nv_vgpu_bar_emul_ctx *bctx; + + if (!vgpu) + return NULL; + bctx = vgpu->bar_emul_ctx; + if (!bctx || !bctx->bar0_shadow) + return NULL; + if (size) + *size = bctx->bar0_shadow_size; + return bctx->bar0_shadow; } /* @@ -1043,6 +1895,16 @@ static int svc_read_ap(struct nv_vgpu_device *vgpu, u64 addr, u32 aperture, memcpy(buf, base + addr, len); return 0; } + /* SESSION 153-CONT-34: emulated BAR0 shadow (register window). */ + if (aperture == SVC_APERTURE_BAR0) { + size_t size = 0; + u8 *base = svc_bar0_shadow(vgpu, &size); + + if (!base || addr + len > size) + return -EINVAL; + memcpy(buf, base + addr, len); + return 0; + } return svc_read(vgpu, addr, buf, len); } @@ -1058,6 +1920,16 @@ static int svc_write_ap(struct nv_vgpu_device *vgpu, u64 addr, u32 aperture, memcpy(base + addr, buf, len); return 0; } + /* SESSION 153-CONT-34: emulated BAR0 shadow (register window). */ + if (aperture == SVC_APERTURE_BAR0) { + size_t size = 0; + u8 *base = svc_bar0_shadow(vgpu, &size); + + if (!base || addr + len > size) + return -EINVAL; + memcpy(base + addr, buf, len); + return 0; + } return svc_write(vgpu, addr, buf, len); } @@ -1103,6 +1975,14 @@ static u32 svc_ring_pin_skips; * content-validated (entry0 must parse; a discovered base must carry this * channel's slot), so re-arming is safe and idempotent within a session. */ +/* + * SESSION 153-CONTINUED-76: the GR stale-ring heal timestamps live here so + * the channel-add path can force an immediate heal for a re-registered + * channel (see gsp_sema_service_channel_add). Declared before its use in + * the add path; the heal itself is defined with the ring-heal section. + */ +static unsigned long svc_gr_heal_at[32]; + static void svc_chan_rearm_rank(struct svc_channel *ch) { if (!ch || !ch->valid) @@ -1112,10 +1992,46 @@ static void svc_chan_rearm_rank(struct svc_channel *ch) clear_bit(ch->ce_index, &svc_pb_done); } else { clear_bit(ch->gr_index, &svc_gr_ring_done); + /* + * r914: re-arm the ring-anchored window pairing too. The GL + * client re-registers with a NEW ring PA on every attempt, + * and with only the ring bit cleared the stale first pair + * (svc_pb_win_lookup returns the first slot match) kept + * answering the push reads with the previous attempt's + * pages - which the guest had since reused for kernel + * structures (measured r913: the push read resolved to a + * page of kernel pointers and every entry went unreadable). + */ + clear_bit(ch->gr_index, &svc_pb_gr_done); } } /* + * SESSION 153-CONTINUED-75: does the service hold a page directory for this + * VA-space handle? The channel-registration log uses this to name the + * discovery gap on the line itself (a nonzero channel hVASpace absent from + * svc.vas[] is exactly the "undiscovered VAS" condition the CONT-74 + * investigation looked for by hand). + */ +static bool svc_vaspace_known(u32 h_vaspace) +{ + unsigned long flags; + bool found = false; + int i; + + if (!h_vaspace) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].h_vaspace == h_vaspace) { + found = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + return found; +} + +/* * Register a guest channel. Called from the GSP-RM emulation when the guest * driver allocates a channel (the allocation carries the GPFIFO GPU VA, its * entry count and the guest-physical USERD/instance blocks), and dropped when @@ -1128,8 +2044,27 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) int i; if (!desc || !desc->h_object || !desc->gpfifo_va || - desc->gpfifo_entries < 2) + desc->gpfifo_entries < 2) { + /* + * SESSION 153-CONTINUED-15 (draw attribution): a refused + * channel registration used to be silent, so a real client + * channel with an empty gpfifo VA looked identical to a + * channel that was never allocated. Name every refusal. + */ + static u32 refuse_log; + + if (refuse_log < 32) { + refuse_log++; + pr_info("sema-svc: chan add REFUSED h=0x%x gpfifo=0x%llx ent=%u eng=%u userd=0x%llx/%u\n", + desc ? desc->h_object : 0, + desc ? desc->gpfifo_va : 0, + desc ? desc->gpfifo_entries : 0, + desc ? desc->engine_type : 0, + desc ? desc->userd_pa : 0, + desc ? desc->userd_as : 0); + } return; + } if (!desc->engine_type) { WRITE_ONCE(svc_gr_content_rearm, true); svc_cepin_log = 0; @@ -1170,10 +2105,21 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) int k; u64 keep_counter_pa = 0; u64 keep_release_pa = 0; + u64 keep_release_va = 0; u8 keep_gr_index = 0, keep_ce_index = 0; + bool keep_pub_armed = false; + u32 keep_counter_gen = 0; DECLARE_BITMAP(keep_rel_done, SVC_GPFIFO_MAX_BITS); bool same = ch->valid && ch->vgpu == desc->vgpu && ch->h_object == desc->h_object; + /* m176: only a same-generation re-registration may inherit the + * learned write targets. After a session flush the same numeric + * handle belongs to a fresh client whose counter/release slots + * live in a brand-new page; inheriting the dead session's PAs + * made every release land on the old page (and be skipped + * against its stale contents) while the new client polled its + * own slot forever. */ + bool same_gen = same && ch->session_gen == svc_sess_gen; /* * A re-registration of the same channel must not lose the @@ -1182,21 +2128,43 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * counter layout does not change). Same for the increment * release marks: losing them would re-apply pushes. */ - if (same) { + if (same_gen) { keep_counter_pa = ch->counter_pa; keep_release_pa = ch->release_pa; + keep_release_va = ch->release_va; keep_gr_index = ch->gr_index; keep_ce_index = ch->ce_index; + /* + * r311: a same-session re-registration must not + * forget that this session already proved itself + * (gr_pub_armed) - losing it re-arms the residual + * watchdog, and its recovery zeroes a completion + * word this live session authored, which the + * client's UVM reads as a downward jump and never + * completes on (r311 vector_add: pub-adv/zero + * 3 s cycles with the client polling forever). + */ + keep_pub_armed = ch->gr_pub_armed; + keep_counter_gen = ch->counter_gen; bitmap_copy(keep_rel_done, ch->rel_done, SVC_GPFIFO_MAX_BITS); } memset(ch, 0, sizeof(*ch)); ch->gr_hold_idx = ~0u; /* m156: no hold armed yet */ + ch->gr_hold_dead_va = 0; /* GRFIX: no launch bound armed */ + ch->gr_hold_exp_logged = false; + ch->opdef_hold_dead_va = 0; /* r361: no opdef bound armed */ + ch->opdef_hold_exp_logged = false; ch->gr_rehold_idx = ~0u; - if (same) + ch->gr_nocap_idx = ~0u; /* m164: no no-capture window armed */ + ch->session_gen = svc_sess_gen; /* m176 */ + if (same_gen) { bitmap_copy(ch->rel_done, keep_rel_done, SVC_GPFIFO_MAX_BITS); + ch->gr_pub_armed = keep_pub_armed; + ch->counter_gen = keep_counter_gen; + } /* * Rank among the live CE channels OF THIS GUEST. The RM hands * out object handles in creation order and the guest's UVM @@ -1234,10 +2202,21 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) ch->h_object = desc->h_object; ch->engine_type = desc->engine_type; ch->cid = desc->cid; + ch->h_vaspace = desc->h_vaspace; ch->gpfifo_va = desc->gpfifo_va; ch->gpfifo_entries = desc->gpfifo_entries; ch->userd_pa = desc->userd_pa; ch->userd_as = desc->userd_as ? desc->userd_as : SVC_APERTURE_SYS; + /* + * r831: a registered userd value that looks like a real guest + * PA (the RC watchdog's channel carries its own userd block) + * is the driver's own declaration and outranks the ring+0x2000 + * convention guess the ring auto-pin applies (that guess is + * right for the CUDA GR layout, where the registered value is + * pool-relative - observed 0x10000). Tiny registered values + * stay guesses. + */ + ch->userd_reg = desc->userd_pa >= 0x10000000ULL; ch->instance_pa = desc->instance_pa; ch->notifier_pa = desc->notifier_pa; ch->notifier_as = desc->notifier_as ? desc->notifier_as @@ -1245,6 +2224,7 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) if (same) { ch->counter_pa = keep_counter_pa; ch->release_pa = keep_release_pa; + ch->release_va = keep_release_va; ch->gr_index = keep_gr_index; ch->ce_index = keep_ce_index; } @@ -1267,6 +2247,18 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * the previous session's one-shot state (second-client wedge). */ svc_chan_rearm_rank(ch); + /* + * SESSION 153-CONTINUED-76: a re-registered GR channel starts + * the stale-ring heal immediately instead of waiting out the + * previous instance's 1 s gate. The client churns its channels + * every ~12 s; a ring pin inherited from the dead session (the + * session flush may have been raced by this registration, see + * the add-vs-flush skip) can keep answering with the previous + * ring's pages for the new instance, and the ~40 passes the + * USERD/FB sweeps need must not also wait on the heal. + */ + if (!ch->engine_type && ch->gr_index < 32) + svc_gr_heal_at[ch->gr_index] = 0; svc.last_chan_reg = jiffies; /* * CeUtils scrubber channel: NUM_COPY_BLOCKS = 4096 GPFIFO @@ -1278,19 +2270,39 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * channel; the stashed geometry survives, so the scrub path * does not depend on the channel-table identity at all. */ - if (desc->gpfifo_entries >= 4096 && desc->gpfifo_va && + /* + * m112b: the CeUtils scrubber is the ONLY exactly-4096-entry + * channel (NUM_COPY_BLOCKS). The old `>= 4096` test also + * matched every client channel (this driver version: 32768 + * entries, not the 1024 the comment below assumed), so the + * single scrub slot bound to whichever client channel + * registered last (observed r701: the slot held h=0xbeee0102 + * userd=0x8f400/entries=32768 and the pass authored put=0 for + * the wrong channel while the real scrubber's payload stayed + * zero - the vkAllocateMemory -2 gate). + */ + if (desc->gpfifo_entries == 4096 && desc->gpfifo_va && desc->userd_pa) { struct svc_scrub *s = svc_scrub_get(desc->vgpu, true); if (!s) goto scrub_done; /* table full: max guests */ - /* Re-seed the counting on a fresh scrubber object. */ + pr_info("sema-svc: scrub stash bind vgpu=%px slot=%d h=0x%x entries=%u\n", + (void *)desc->vgpu, (int)(s - svc_scrubs), + desc->h_object, desc->gpfifo_entries); + /* Re-seed the counting on a fresh scrubber object or a + * new client session (m176: the counter/submitted state + * is per-session; a recycled scrubber handle after a + * flush must start from the new channel's GPPut, not + * carry the dead session's counts). */ if (!s->userd_ok || + s->session_gen != svc_sess_gen || s->h_object != desc->h_object || s->gpfifo_va != desc->gpfifo_va) { s->seeded = false; s->submitted = 0; } + s->session_gen = svc_sess_gen; s->userd_ok = true; s->h_object = desc->h_object; s->userd_pa = desc->userd_pa; @@ -1302,12 +2314,34 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) desc->h_object, desc->gpfifo_va, desc->gpfifo_entries, desc->userd_pa, s->userd_as); + } else if (desc->gpfifo_entries >= 4096 && + desc->gpfifo_entries != 32768) { + /* m112b: a non-32768 channel with >=4096 entries that + * fails the stash condition (missing VA or USERD PA) + * is a scrubber candidate the service cannot author + * for - log it so a silent no-authoring boot is + * diagnosable. */ + pr_info_ratelimited("sema-svc: scrubber-stash MISS h=0x%x entries=%u va=0x%llx userd_pa=0x%llx\n", + desc->h_object, desc->gpfifo_entries, + desc->gpfifo_va, desc->userd_pa); } scrub_done: - pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx\n", + /* + * SESSION 153-CONTINUED-75: name the channel's declared VA + * space and whether the service has a page directory for it. + * hVASpace==0 is the driver's "attach to the TSG address + * space" (nv_gpu_ops.c: "If zero then it will attach to the + * TSG address space"), which is why the GR channel's + * low-aperture ring/push VAs resolve through the TSG/client + * VAS published by its CE siblings, not through a handle of + * its own. A nonzero handle that never appears in svc.vas[] + * names the discovery gap directly. + */ + pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx vas=0x%x vas_known=%d\n", desc->h_object, desc->gpfifo_va, desc->gpfifo_entries, desc->userd_pa, desc->engine_type, desc->cid, - desc->notifier_pa); + desc->notifier_pa, desc->h_vaspace, + svc_vaspace_known(desc->h_vaspace)); } else pr_warn_ratelimited("sema-svc: channel 0x%x dropped: channel table full (%d)\n", @@ -1316,6 +2350,7 @@ scrub_done: /* P6b wave-1.5 hold state is defined with the forwarding section below. */ static void svc_fwd_hold_abort(struct nv_vgpu_device *vgpu, u32 h_object); +static void svc_fwd_hold_clear(void); /* m176: session flush drops a stale hold */ /* * P6b wave-2 inline software fallback for a withheld (skipped) candidate; * defined with the forwarding section below. Service-thread only. @@ -1333,6 +2368,45 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, static void svc_opdef_drop(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u32 entry_idx); static void svc_opdef_reset_all(void); +static void svc_opdef_finish_ch(struct svc_channel *ch, u32 idx); +static u32 svc_opdef_log; /* m176: forward decl for the session flush */ +/* r633: per-session synthetic shadow budget reset (defined with the fill). */ +static void svc_gr_synth_ep_reset(void); +/* + * One-cycle GAP-1/GAP-2 instrumentation budgets (agent-datapath plan C1-C10): + * small, reset per session where they are service-thread state, so a later + * client's strand is never blacked out by the first client's spend. + */ +static u32 svc_relx_log; /* C2: deferred-launch apply entry witness */ +static u32 svc_opdef_clr_log; /* C3: opdef-clear caller witness */ +static u32 svc_busy_defer_log; /* C8: GR busy-defer timeout witness */ +static u32 svc_hold_fb_log; /* C7: hold failure/timeout fallback witness */ +static u32 svc_cand_ovw_log; /* C9: candidate overwrite witness */ +static u32 svc_gr_push_dump; /* SESSION 127: bounded capture tail dump + * (sema_dbg only, 4 per boot) */ +static u32 svc_inline_recs_log; /* C10: inline-recs digest witness */ +static u32 svc_gr_await_log; /* r351: launch-held-for-trust witness */ +static u32 svc_gr_retry_log; /* r351: retry-forward/proceed witness */ +static bool svc_hold_bypass(void); /* r563: stall self-heal forward decl */ + +/* + * r551: releases DELIVERED by the opdef hold-expiry paths (flush/sweep) are + * remembered by (channel, entry, va, payload) for a short window, so a + * re-decode of the parked entry does not withhold the (monotonic, + * forward-only) release again - see the helpers with the opdef machinery. + * Declared here because svc_session_flush() resets them per session. + */ +#define SVC_OPDEF_DONE_SLOTS 256 +#define SVC_OPDEF_DONE_TTL_MS 60000u +struct svc_opdef_done_rec { + u32 h_object; + u32 entry_idx; + u64 va; + u32 payload; + unsigned long j; +}; +static struct svc_opdef_done_rec svc_opdef_done[SVC_OPDEF_DONE_SLOTS]; +static u32 svc_opdef_done_next; void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) { @@ -1401,6 +2475,191 @@ void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) * the page tables itself and hands their base address to the GPU). Store it so * the service can translate that space's VAs directly. */ +/* + * r496j: unverified page-directory candidates from COPY_SERVER_PDES. A + * kernel-RM internal VAS publishes its chain there, but registering every + * client's directory directly lets a stale kernel directory answer another + * client's VA in the walker's first-success order (the r459 stale-directory + * class; observed as a CUDA client hanging after DEVICE). Candidates are + * parked here and registered only after a walk of the client's own + * content-validated ring VA through the candidate reproduces that ring's PA + * (svc_gr_ram_ring_scan's latch calls svc_vas_candidates_verify). + */ +#define SVC_VAS_CAND_MAX 16 +static struct { + bool valid; + u32 h_vaspace; + u64 pdb; + u32 entries; + bool vidmem; + unsigned long stamp; /* r537: park time for the bounded fallback */ +} svc_vas_cand[SVC_VAS_CAND_MAX]; + +static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, + u64 va, int tap, u64 *addr, int *ap, bool trace); + +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem) +{ + int i; + + if (!pdb || entries < 2 || entries > 4096) + return; + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + if (svc_vas_cand[i].valid && + svc_vas_cand[i].h_vaspace == h_vaspace && + svc_vas_cand[i].pdb == pdb) + return; /* already parked */ + if (!svc_vas_cand[i].valid) { + svc_vas_cand[i].valid = true; + svc_vas_cand[i].h_vaspace = h_vaspace; + svc_vas_cand[i].pdb = pdb; + svc_vas_cand[i].entries = entries; + svc_vas_cand[i].vidmem = vidmem; + svc_vas_cand[i].stamp = jiffies; + return; + } + } +} + +/* + * r537 bounded fallback: register a parked candidate that has waited for + * its content-verified ring walk longer than SVC_VAS_CAND_AGE_MS. The + * verification (svc_vas_candidates_verify) needs the client's ring to be + * found by the RAM content scan; on a boot where that scan never anchors + * the ring (observed: pd_found=0 and every launch operand walk rc=-2 for + * minutes, the client spinning with its data VAs unresolvable) the + * candidate stays parked forever and the client can never proceed. The + * candidate itself came from the client's own RM publish (its VAS + * handle), and the r459 newest-first walk order keeps an older stale + * directory from winning, so after the bounded wait the registration is + * the lesser evil; it is logged loudly for the record. + */ +#define SVC_VAS_CAND_AGE_MS 15000u +/* + * r705h: the candidate age-out deadline is runtime-tunable. The parked + * COPY_SERVER_PDES candidate for the GL/Vulkan client ages out (UNVERIFIED) + * at exactly the moment the client's channels first register, so the + * content-verified walk can never anchor it; raising the deadline keeps the + * candidate parked until a host tool supplies a verified ring pair via + * sema_pdb_discover. 0 disables the age-out entirely. + */ +static u32 svc_vas_cand_age_ms = SVC_VAS_CAND_AGE_MS; +module_param_named(sema_vas_cand_ms, svc_vas_cand_age_ms, uint, 0644); + +/* + * SESSION 127 C5C0 translator gate. When 1, the sweep additionally + * translates continuation entries into launch work (tail-aware method + * parse, inline-QMD capture, LOAD_INLINE_DATA materialization, C5B5 + * copy-out capture, candidate-owned release withholding). + * + * SESSION 130 re-test of the r727 "translator regressions" (translator on, + * utility sweep OFF, r746): victory_saxpy and victory_math now PASS with + * checksums identical to the gate-off baseline (saxpy + * csum=981440890.500000), so the saxpy/math half of the r727 regression + * is FIXED (the SESSION 129/130 sweep changes); victory_mail PASSES. The + * remaining xlate-on regression is victory_multikernel: its launch is + * captured and submitted to the physical channel but the forward times + * out repeatedly (`GR LAUNCH FAILED: timeout after 2000 ms + * (qmd_rel=0xc0ffee01 want=0x600df00d ...) h=0x5c000019 + * push=0x20043c394`, `hold active`), so the client hangs - the default + * stays 0 until that forward path is fixed. The UTILITY SWEEP + * (`sema_gr_util_scan`) is the OTHER regression source: at 24 it broke + * saxpy (`bad=1047527`) and hung math on r745, so it stays opt-in + * (default 0) until its channel reads prefer the direct walk over pins. + */ +static bool gsp_sema_gr_xlate; +module_param_named(sema_gr_xlate, gsp_sema_gr_xlate, bool, 0644); +MODULE_PARM_DESC(sema_gr_xlate, + "C5C0 translator: sweep continuation entries into launch work (default 0 = releases-only sweep)"); +/* + * SESSION 128 utility-channel inline-download materialization + * (sema_gr_util_scan, default 0 = off): the driver's non-GR upload + * channels (the MEM2MEM/A140 + 2D utilities, registered but with an + * unresolved USERD so the pump's put-bounded entry loop never decodes + * them) carry the engine-side code/data downloads: the + * GRCE/INLINE_TO_MEMORY operand group on the mem2mem subchannel (0x180.. + * 0x18c), LAUNCH_DMA (0x1b0) and a LOAD_INLINE_DATA (0x1b4) run whose + * payload the engine fetches from the next GPFIFO entry (the + * source-view call). Without this pass those destinations stay zero in + * the vidmem shadow - observed: the Vulkan client's code-heap/SASS + * upload (12 KB) and command-stream staging (256 B) never land, so the + * forwarded launch has no program content. Bounded: scan up to N + * entries per channel per pass, materialize once-only keyed by + * (push_va, content hash, dst, bytes) via the same + * svc_grce_inline_write the GR sweep uses. Requires sema_gr_xlate. + */ +static u32 gsp_sema_gr_util_scan; +module_param_named(sema_gr_util_scan, gsp_sema_gr_util_scan, uint, 0644); +MODULE_PARM_DESC(sema_gr_util_scan, + "Utility-channel i2m sweep: entries scanned per pass (default 0 = off; requires sema_gr_xlate)"); +MODULE_PARM_DESC(sema_vas_cand_ms, "Parked-VAS-candidate age-out (ms, default 15000; 0 = never)"); + +static void svc_vas_candidates_age_out(struct nv_vgpu_device *vgpu) +{ + int i; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + struct sema_vaspace_desc d; + + if (!svc_vas_cand[i].valid) + continue; + if (!svc_vas_cand_age_ms || + time_before(jiffies, svc_vas_cand[i].stamp + + msecs_to_jiffies(svc_vas_cand_age_ms))) + continue; + memset(&d, 0, sizeof(d)); + d.h_vaspace = svc_vas_cand[i].h_vaspace; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + pr_warn("sema-svc: VAS 0x%x page directory registered UNVERIFIED after %u ms parked (pdb=0x%llx entries=%u; the content-verified ring walk never anchored this boot)\n", + d.h_vaspace, svc_vas_cand_age_ms, d.pdb, + d.num_entries); + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + } +} + +/* + * Verify each parked candidate against one content-validated translation + * (the ring VA -> PA the RAM scan just proved): walk the ring page through + * the candidate and register only the directory that reproduces it. Runs on + * the service thread from the ring-scan latch; the walk itself is cheap (a + * few table reads). + */ +static void svc_vas_candidates_verify(struct nv_vgpu_device *vgpu, u64 va, + u64 pa) +{ + int i; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + u64 wpa = 0; + int wap = 0; + int ap = svc_vas_cand[i].vidmem ? SVC_APERTURE_FB + : SVC_APERTURE_SYS; + struct sema_vaspace_desc d; + + if (!svc_vas_cand[i].valid) + continue; + if (svc_gmmu_walk(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, va & ~0xfffULL, + ap, &wpa, &wap, false) || + (wpa & ~0xfffULL) != (pa & ~0xfffULL)) + continue; + memset(&d, 0, sizeof(d)); + d.h_vaspace = svc_vas_cand[i].h_vaspace; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + pr_info("sema-svc: VAS 0x%x page directory registered after verified ring walk (pdb=0x%llx entries=%u ap=%d)\n", + d.h_vaspace, d.pdb, d.num_entries, ap); + break; + } +} + void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) { struct svc_vaspace *vs = NULL; @@ -1419,12 +2678,40 @@ void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) if (!vs && !svc.vas[i].valid) vs = &svc.vas[i]; } + /* + * r459: the table is 8 slots and the guest driver does not always + * send a teardown for a client's VA space (observed: 23 distinct + * registrations, 0 drops in one boot). Once all slots hold a dead + * client's directory, a NEW client's registration found no slot and + * every walk of its VAs failed (copies rc=-2, launches reading + * stale pages - the 10th-client mid-boot degradation). Evict the + * least-recently-registered entry instead of refusing the new one; + * the old client is gone and its directory pointer is dead guest + * memory, so the eviction can only remove stale authority. + */ + if (!vs) { + unsigned long oldest = 0; + int oi = 0; + + for (i = 0; i < SVC_MAX_VASPACES; i++) { + if (!svc.vas[i].valid) + continue; + if (!oldest || time_before(svc.vas[i].stamp, oldest)) { + oldest = svc.vas[i].stamp; + oi = i; + } + } + vs = &svc.vas[oi]; + pr_info("sema-svc: VA space table full; evicting 0x%x (pdb=0x%llx) for 0x%x\n", + vs->h_vaspace, vs->pdb, desc->h_vaspace); + } if (vs) { vs->valid = true; vs->h_vaspace = desc->h_vaspace; vs->pdb = desc->pdb; vs->entries = desc->num_entries; vs->ap = desc->b_vidmem ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + vs->stamp = jiffies; /* A new directory invalidates what we learned from the old one. */ memset(svc.lat, 0, sizeof(svc.lat)); } @@ -1553,7 +2840,7 @@ static bool svc_ap_ok(struct nv_vgpu_device *vgpu, u64 addr, int ap) { if (ap == SVC_APERTURE_FB) return vgpu->bars[1].vaddr && addr + 8 <= svc_fb_shadow_size(vgpu); - return addr + 8 <= SEMA_PA_MAX; + return addr + 8 <= svc_pa_max(); } static int svc_read_ent(struct nv_vgpu_device *vgpu, u64 table, u64 off, @@ -1648,8 +2935,18 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, return -ENOENT; } - /* PD0: one dual entry per index (big half then small half). */ + /* + * PD0: one dual entry per index (big half then small half). + * Geometry per the driver's own GMMU format (gp10x family): + * PD3 [48:47], PD2 [46:38], PD1 [37:29], PD0 [28:21], PT + * [20:12]. PD0 is 8 bits - an earlier 0x1ff experiment here was + * reverted; the observed +2 MB client-page phase is NOT a PD0 + * index error. + */ idx = (va >> 21) & 0xff; + if (trace) + pr_info("sema-walk: PD0 table=0x%llx idx=%u va=0x%llx tap=%d\n", + table, idx, va, tap); if (svc_read_ap(vgpu, table + 16 * idx, tap == SVC_APERTURE_FB ? SVC_APERTURE_FB : SVC_APERTURE_SYS, @@ -1659,6 +2956,9 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, : SVC_APERTURE_SYS, &small, sizeof(small))) return -ENOENT; + if (trace) + pr_info("sema-walk: PD0 big=0x%016llx small=0x%016llx\n", + big, small); /* Small half: 4 KB page table under it. */ sap = svc_pde_aperture(((small >> 1) & 0x3)); @@ -1670,6 +2970,12 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, *addr += va & 0xfff; return 0; } + if (trace) + pr_info("sema-walk: small miss sap=%d t=0x%llx\n", + sap, t); + } else if (trace) { + pr_info("sema-walk: small skip sap=%d small&1=%llu\n", + sap, small & 1); } /* Big half: a 2 MB page here, or a large (64 KB) page table. */ @@ -1702,6 +3008,9 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, return 0; } } + if (trace) + pr_info("sema-walk: no leaf va=0x%llx big=0x%llx small=0x%llx\n", + va, big, small); return -ENOENT; } @@ -1730,6 +3039,26 @@ static int svc_walk_vaspaces(struct nv_vgpu_device *vgpu, u64 va, } spin_unlock_irqrestore(&svc.vas_lock, flags); + /* + * r459: try the NEWEST registration first. A dead client's stale + * directory (the table outlives clients; only 8 slots) can still + * resolve a VA - through guest memory the driver has since reused - + * to a wrong page, and the first success wins. Ordering by the + * registration stamp makes the live client's directory the + * authority and only falls through to older ones when it cannot + * answer. + */ + for (i = 1; i < n; i++) { + struct svc_vaspace k = local[i]; + int j = i - 1; + + while (j >= 0 && time_before(local[j].stamp, k.stamp)) { + local[j + 1] = local[j]; + j--; + } + local[j + 1] = k; + } + for (i = 0; i < n; i++) { int alt; @@ -1855,8 +3184,23 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) } if (n < SVC_PD_MIN_HITS) { n = 0; - if (svc.map_n < SVC_PD_SAMPLES) - return; /* no evidence yet: retry when pairs appear */ + /* + * r510: a LARGE map is a mixed set of VA spaces cached over + * the whole guest boot (the comment above already names the + * consequence: no single directory reproduces a mixed set), so + * scanning with its stride subsample burns the attempts budget + * on evidence that cannot succeed - and by the time the + * push-buffer pairs (the strongest samples) exist, the budget + * is spent (r508/r509 witness: 'not found (base=0x0 ap=ram + * cands=39621 best=0/8)' during the first client, eight RAM + * walks consumed, every later client's high argument pointers + * unresolvable - the terminal class). Only a SMALL map, which + * is one session's own pairs, still falls back; otherwise wait + * for the push-buffer pairs and keep the attempts. + */ + if (svc.map_n > (u32)(SVC_PD_SAMPLES * 2) || + svc.map_n < SVC_PD_SAMPLES) + return; /* no coherent evidence yet: retry later */ step = svc.map_n / SVC_PD_SAMPLES; for (i = 0; i < SVC_PD_SAMPLES; i++) { sva[i] = svc.map[i * step].gpu_page; @@ -1892,7 +3236,7 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) * matters is the first one of the page. */ for (pass = 0; pass < 2 && !svc_pd_found; pass++) { int ap = pass == 0 ? SVC_APERTURE_SYS : SVC_APERTURE_FB; - u64 limit = pass == 0 ? SEMA_PA_MAX : svc_fb_shadow_size(vgpu); + u64 limit = pass == 0 ? svc_pa_max() : svc_fb_shadow_size(vgpu); u8 *fb = (u8 *)vgpu->bars[1].vaddr; u64 gpa; @@ -1929,13 +3273,133 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) svc_pd_cands, svc_pd_best, n); } +/* + * m177c: a SECOND discovered-directory slot for the UVM SYS VA space (the + * 0x2xxxxxxx family: GR rings/pushes, QMD windows, counter pages). The + * primary slot (above) is validated against rm_mem-content pairs - the + * 0x1xxxxxxx pool family - and cannot walk the UVM space, so without pins + * the counter page (0x20400fXXX) never resolved: the guest's KMD/UVM + * epoch waited on completions that could never be written and re-ran its + * ~103 s retry loop forever (r194/r195: 'rel-apply ... slot pending' x8, + * then silence). Samples for this slot are content-VERIFIED + * (push_va -> push_pa) pairs from the GR content discovery and the CE + * push pairing: the push bytes themselves are the proof, and they live in + * exactly the space this directory maps. + */ +#define SVC_PD_RING_SAMPLES 8 +static u64 svc_pd_r_v[SVC_PD_RING_SAMPLES]; +static u64 svc_pd_r_pa[SVC_PD_RING_SAMPLES]; +static u32 svc_pd_r_n; +static u64 svc_pd_r_base; +static int svc_pd_r_ap; +static u8 svc_pd_r_found; +static u8 svc_pd_r_attempts; +static u64 svc_pd_r_ev_hash; + +static void svc_pd_ring_note(u64 va, u64 pa) +{ + u32 i; + + if (!va || !pa) + return; + va &= ~0xfffULL; + pa &= ~0xfffULL; + for (i = 0; i < svc_pd_r_n; i++) + if (svc_pd_r_v[i] == va) + return; + if (svc_pd_r_n < SVC_PD_RING_SAMPLES) { + svc_pd_r_v[svc_pd_r_n] = va; + svc_pd_r_pa[svc_pd_r_n] = pa; + svc_pd_r_n++; + } +} + +static void svc_pd_ring_discover(struct nv_vgpu_device *vgpu) +{ + u64 chunk = 64 * 1024; + u8 *buf; + u64 gpa; + u64 ev = svc_pd_r_n; + u32 i; + + if (svc_pd_r_found || svc_pd_r_attempts >= SVC_PD_ATTEMPTS_MAX) + return; + if (svc_pd_r_n < SVC_PD_MIN_HITS) + return; + for (i = 0; i < svc_pd_r_n; i++) + ev = (ev + svc_pd_r_v[i] + (svc_pd_r_pa[i] << 12)) * 1000003; + if (svc_pd_r_attempts && ev == svc_pd_r_ev_hash) + return; + svc_pd_r_ev_hash = ev; + svc_pd_r_attempts++; + + buf = kmalloc(chunk, GFP_KERNEL); + if (!buf) + return; + for (gpa = 0; gpa < svc_pa_max() && !svc_pd_r_found; gpa += chunk) { + u32 len = (u32)min_t(u64, chunk, svc_pa_max() - gpa); + u64 off; + + if (svc_read(vgpu, gpa, buf, len)) + continue; + for (off = 0; off + 8 <= len && !svc_pd_r_found; + off += SEMA_PAGE_SIZE) { + u64 e; + u64 a = 0; + int eap = 0; + u32 hits = 1; + + memcpy(&e, buf + off, sizeof(e)); + if ((e & 1) || !((e >> 1) & 0x3)) + continue; + /* First sample rejects almost every candidate. */ + if (svc_gmmu_walk(vgpu, gpa + off, 4, svc_pd_r_v[0], + SVC_APERTURE_SYS, &a, &eap, false) || + (a & ~0xfffULL) != svc_pd_r_pa[0]) + continue; + for (i = 1; i < svc_pd_r_n; i++) { + if (!svc_gmmu_walk(vgpu, gpa + off, 4, + svc_pd_r_v[i], + SVC_APERTURE_SYS, &a, &eap, + false) && + (a & ~0xfffULL) == svc_pd_r_pa[i]) + hits++; + } + if (hits >= SVC_PD_MIN_HITS) { + svc_pd_r_base = gpa + off; + svc_pd_r_ap = SVC_APERTURE_SYS; + svc_pd_r_found = 1; + pr_info("sema-svc: discovered UVM ring-space page directory 0x%llx (ram), %u/%u content-verified samples\n", + svc_pd_r_base, hits, svc_pd_r_n); + } + } + } + kfree(buf); +} + static int svc_walk_discovered(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, int *ap) { - if (!svc_pd_found) - return -ENOENT; - return svc_gmmu_walk(vgpu, svc_pd_base, svc_pd_entries, va, - svc_pd_ap, addr, ap, false); + int rc = -ENOENT; + + if (svc_pd_found) + rc = svc_gmmu_walk(vgpu, svc_pd_base, svc_pd_entries, va, + svc_pd_ap, addr, ap, false); + if (rc && svc_pd_r_found) + rc = svc_gmmu_walk(vgpu, svc_pd_r_base, 4, va, + svc_pd_r_ap, addr, ap, false); + /* + * m177c: the ring-space directory discovery runs on new samples + * only (fingerprint + attempt cap inside). A first call with + * enough verified ring/push samples performs one bounded scan; + * after that this is a flag check. + */ + if (rc && !svc_pd_r_found && svc_pd_r_n >= SVC_PD_MIN_HITS) + svc_pd_ring_discover(vgpu); + if (rc && svc_pd_r_found) + rc = svc_gmmu_walk(vgpu, svc_pd_r_base, 4, va, + svc_pd_r_ap, addr, ap, false); + return rc; } /* Learned translations: a walk costs a handful of guest reads, so cache the @@ -1968,6 +3432,261 @@ static void svc_lat_store(u64 va, u64 addr, int ap) } /* + * SESSION 153-CONTINUED-67: invalidate a learned-cache entry so the next + * lookup of this page re-runs the source chain underneath it (discovered + * map, registered-VAS walk, discovered walk, targeted resolve) instead of + * reusing a cached answer. Used by the repeated all-zero read + * invalidation in svc_read_va_push_ex: every walk success re-stores the + * answer here (svc_lat_store after each svc_lookup_trusted sub-source), + * so a stale walk result otherwise shadows a fresh walk for the rest of + * the boot (r1127: the beef setup push kept answering the same zero page + * while the client's content was live elsewhere). + */ +static void svc_lat_drop(u64 va) +{ + struct svc_lat *l = &svc.lat[(va >> SEMA_PAGE_SHIFT) % SVC_LAT_MAX]; + + if (l->valid && + (l->va & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (va & ~(u64)(SEMA_PAGE_SIZE - 1))) + l->valid = false; +} + +/* + * r820: driver-declared kernel-buffer translations (PROMOTE_CTX). + * + * The guest RM announces the physical backing of a channel's context + * buffers through NV2080_CTRL_CMD_GPU_PROMOTE_CTX: every entry carries + * (gpuVirtAddr, gpuPhysAddr, size, physAttr, bufferId). These are + * KERNEL-owned buffers - the nvidia-push pushbuffer, the progress-tracker + * segment, the FECS/patch/access-map blocks - so the userspace pin + * scanner can never cover them (/proc/<pid>/pagemap has no kernel VAs) + * and the page-table walk can answer with a stale/wrong page. Observed + * r817 (Xorg GL init): the walk resolved the 64 KB pushbuffer VA + * 0x120010000 to pa=0x103fed000, a page reading all zeros, while the + * driver's own promote entry said pa=0x11e120000. The NOP sanity push of + * __nvPushTestPushBuffer then never decoded, GP_GET never advanced and + * the X server died with "Failed to initialize DMA". + * + * The registry is a small table keyed by the entry VA, newest wins: a + * client that re-promotes the same buffer - observed between two Xorg + * starts, the same channel object announced a NEW pa for the same VA - + * replaces the old record, so a stale backing is never preferred over + * the live statement. VA-only bookings (bInitialize=0 re-promotes of + * already-initialized buffers, zero PA/size) and bNonmapped entries + * (va=0) carry no backing and must not erase the last statement. + * + * Lookups run before the learned cache in svc_lookup_trusted: the wrong + * r817 walk answer had been cached in the learned table (svc_lat_store + * after every walk success), so a promote hit must both pre-empt and + * overwrite that entry; the hit re-stores the corrected page into the + * learned table so the direct svc_lat_lookup callers heal too. + */ +#define SVC_PROMO_MAX 256 +struct svc_promo_rec { + u64 va; + u64 pa; + u64 size; + unsigned long seq; + u32 phys_attr; + u16 buffer_id; + bool valid; +}; +static struct svc_promo_rec svc_promo[SVC_PROMO_MAX]; +static unsigned long svc_promo_seq; +static DEFINE_SPINLOCK(svc_promo_lock); +static u32 svc_promo_hit_log; /* bounded "answered from PROMOTE_CTX" witness */ + +void gsp_sema_service_promote_add(u64 va, u64 pa, u64 size, u32 phys_attr, + u32 buffer_id) +{ + struct svc_promo_rec *rec = NULL, *free_rec = NULL, *oldest = NULL; + unsigned long flags; + int i; + + if (!va || !pa || !size) + return; + + spin_lock_irqsave(&svc_promo_lock, flags); + for (i = 0; i < SVC_PROMO_MAX; i++) { + if (svc_promo[i].valid && svc_promo[i].va == va) { + rec = &svc_promo[i]; + break; + } + if (!svc_promo[i].valid && !free_rec) + free_rec = &svc_promo[i]; + if (svc_promo[i].valid && + (!oldest || svc_promo[i].seq < oldest->seq)) + oldest = &svc_promo[i]; + } + if (!rec) + rec = free_rec ? free_rec : oldest; + if (!rec) { + spin_unlock_irqrestore(&svc_promo_lock, flags); + return; + } + rec->va = va; + rec->pa = pa; + rec->size = size; + rec->phys_attr = phys_attr; + rec->buffer_id = (u16)buffer_id; + rec->seq = ++svc_promo_seq; + rec->valid = true; + spin_unlock_irqrestore(&svc_promo_lock, flags); +} + +static int svc_promo_lookup(u64 va, u64 *addr, int *ap) +{ + unsigned long flags; + u64 best_seq = 0; + int rc = -ENOENT; + int i; + + spin_lock_irqsave(&svc_promo_lock, flags); + for (i = 0; i < SVC_PROMO_MAX; i++) { + const struct svc_promo_rec *rec = &svc_promo[i]; + + /* + * r823: answer ONLY the base page. The promoted (va, size) + * pair is a VA-space booking, not a contiguous mapping: the + * ctx MAIN buffer's span [0x120020000, 0x120153000) contains + * OTHER buffers' VAs (the channel GPFIFO at 0x120090000), and + * the range arithmetic answered such an interior VA with + * MAIN.pa + offset - a zero page (observed r823: the GPFIFO + * VA resolved to 0x11ce70000, the ring-entry validation read + * va=0x0 len=0 and the latch never fired). Only the page at + * the declared VA has a proven claim to the declared PA; the + * pushbuffer's NOP sits exactly there (nvPushHeader writes at + * the segment base). + */ + if (!rec->valid || va < rec->va || + va - rec->va >= SEMA_PAGE_SIZE) + continue; + if (best_seq && rec->seq <= best_seq) + continue; + best_seq = rec->seq; + *addr = rec->pa + (va - rec->va); + /* + * physAttr 0x4 was observed only on vidmem entries (pa in + * the emulated FB aperture, e.g. 0x1f4000000), 0x5/0x6 on + * system memory. Unknown values fall to SYS, whose read + * guard (pa < SEMA_PA_MAX) still rejects nonsense. + */ + *ap = (rec->phys_attr == 0x4) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + rc = 0; + } + spin_unlock_irqrestore(&svc_promo_lock, flags); + if (!rc && svc_promo_hit_log < 64) { + svc_promo_hit_log++; + pr_info("sema-promo: va=0x%llx -> pa=0x%llx ap=%d (driver PROMOTE_CTX statement)\n", + va, *addr, *ap); + } + return rc; +} + +/* + * SESSION 153-CONTINUED-95: per-(engine,bufferId) record of the guest + * driver's GR context-buffer promote statements, for the launch-time + * CTX-BIND receipt. The response-free sema promote table above is keyed by + * VA and answers translations; this one keeps the latest PROMOTE_CTX entry + * for the context buffers the mission asks about - MAIN (bufferId 0) and + * PATCH (bufferId 2) - including the bInitialize flag, so a launch can log + * whether the client's promoted GR context buffer is installed on the + * engine's replay channel (it is not: the channel's context is the + * server/own buffer the ladder built; this record proves which buffer the + * client believes is its context, with pa/va/size/init as sent). + */ +#define SVC_CTXP_MAX 8 +struct svc_ctxp_rec { + u32 engine; + u32 buffer_id; + u64 pa; + u64 va; + u64 size; + u32 phys_attr; + u32 b_init; + unsigned long seq; + bool valid; +}; +static struct svc_ctxp_rec svc_ctxp[SVC_CTXP_MAX]; +static unsigned long svc_ctxp_seq; +static DEFINE_SPINLOCK(svc_ctxp_lock); + +void gsp_sema_service_ctx_promote_note(u32 engine, u32 buffer_id, u64 pa, + u64 va, u64 size, u32 phys_attr, + u32 b_init) +{ + struct svc_ctxp_rec *rec = NULL, *free_rec = NULL, *oldest = NULL; + unsigned long flags; + int i; + + spin_lock_irqsave(&svc_ctxp_lock, flags); + for (i = 0; i < SVC_CTXP_MAX; i++) { + if (svc_ctxp[i].valid && + svc_ctxp[i].engine == engine && + svc_ctxp[i].buffer_id == buffer_id) { + rec = &svc_ctxp[i]; + break; + } + if (!svc_ctxp[i].valid && !free_rec) + free_rec = &svc_ctxp[i]; + if (svc_ctxp[i].valid && + (!oldest || svc_ctxp[i].seq < oldest->seq)) + oldest = &svc_ctxp[i]; + } + if (!rec) + rec = free_rec ? free_rec : oldest; + if (rec) { + rec->engine = engine; + rec->buffer_id = buffer_id; + rec->pa = pa; + rec->va = va; + rec->size = size; + rec->phys_attr = phys_attr; + rec->b_init = b_init; + rec->seq = ++svc_ctxp_seq; + rec->valid = true; + } + spin_unlock_irqrestore(&svc_ctxp_lock, flags); +} + +int gsp_sema_service_ctx_promote_get(u32 engine, u32 buffer_id, u64 *pa, + u64 *va, u64 *size, u32 *phys_attr, + u32 *b_init) +{ + unsigned long flags; + u64 best_seq = 0; + int rc = -ENOENT; + int i; + + spin_lock_irqsave(&svc_ctxp_lock, flags); + for (i = 0; i < SVC_CTXP_MAX; i++) { + const struct svc_ctxp_rec *rec = &svc_ctxp[i]; + + if (!rec->valid || rec->engine != engine || + rec->buffer_id != buffer_id) + continue; + if (best_seq && rec->seq <= best_seq) + continue; + best_seq = rec->seq; + if (pa) + *pa = rec->pa; + if (va) + *va = rec->va; + if (size) + *size = rec->size; + if (phys_attr) + *phys_attr = rec->phys_attr; + if (b_init) + *b_init = rec->b_init; + rc = 0; + } + spin_unlock_irqrestore(&svc_ctxp_lock, flags); + return rc; +} + +/* * VA shadow: page-granular backing for device VAs whose VA->PA translation * cannot be derived from the guest's page tables. * @@ -1994,6 +3713,16 @@ static void svc_lat_store(u64 va, u64 addr, int ap) struct svc_vas_rec { u64 va_page; /* 0 = empty */ u8 *data; + /* + * r635: the page holds bytes a client/host WRITE actually landed + * (an H2D shadow-lane write, an inline write, a bridge write, an + * engine writeback). Synthetic pages minted by the GR fill serve + * zeros and must never be mistaken for the client's own content: + * only real content anchors a span or arms the launch-level anchor + * (without this flag the fill would re-anchor on its own minted + * pages and mint without bound across retry passes). + */ + u8 content; }; static struct svc_vas_rec svc_vas[SVC_VAS_SLOTS]; @@ -2029,6 +3758,14 @@ static bool svc_vas_has(u64 va) return svc_vas_find(va) != NULL; } +/* r635: the page holds REAL written content (not a synthetic zero page). */ +static bool svc_vas_has_content(u64 va) +{ + struct svc_vas_rec *r = svc_vas_find(va); + + return r && r->content; +} + /* * A VA may receive a new shadow record only when it is engine-only device * memory: high in the client VA space, where the guest's own sysmem objects @@ -2036,8 +3773,23 @@ static bool svc_vas_has(u64 va) * 100 TiB). Explicit sources (GRCE inline writes, the bridge's output write) * shadow any VA because their target is device memory by construction. */ +bool gsp_sema_no_shadow; /* r626 A/B: disable the shadow lane */ +module_param_named(sema_no_shadow, gsp_sema_no_shadow, bool, 0644); +MODULE_PARM_DESC(sema_no_shadow, + "Disable the VA shadow lane (copy-dst and fill fallbacks) for an A/B boot; unresolved device VAs hold instead (default 0)"); + static bool svc_vas_can_shadow(u64 va) { + /* + * r626 A/B: with the knob set, every shadow fallback is refused and + * the caller takes its -ENOENT/hold path. The r613c/r616/r623c + * corruption class (nvidia channel structures overwritten with + * PTE-like 0x444../0x4456.. patterns, fatal panic in + * kchannelGetGfid_IMPL via nvGpuOpsStopChannel) follows the + * shadow-lane era; this knob isolates the lane as a variable. + */ + if (gsp_sema_no_shadow) + return false; return va >= SVC_VAS_FLOOR; } @@ -2087,6 +3839,7 @@ static int svc_vas_write(u64 va, const u8 *buf, size_t len) if (!r) return -ENOMEM; memcpy(r->data + (va & (SEMA_PAGE_SIZE - 1)), buf, chunk); + r->content = 1; /* r635: real bytes landed here */ svc.vas_writes += chunk; buf += chunk; va += chunk; @@ -2199,6 +3952,37 @@ static void pin_store_tab(struct svc_pin *tab, u32 slots, u32 probe_max, (*drops)++; } +/* + * SESSION 153-CONTINUED-64d: same store, but reports a probe-window + * exhaustion back to the caller so it can evict instead of losing the pin + * (see svc_ipin_store). Returns true when the slot was NOT stored. + */ +static bool pin_store_tab_nodrop(struct svc_pin *tab, u32 slots, + u32 probe_max, u32 *n, u64 *drops, u64 va, + u64 pa, int ap, u8 win) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 mask = slots - 1; + u32 h = (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & mask; + u32 probe; + + for (probe = 0; probe < probe_max; probe++) { + struct svc_pin *p = &tab[(h + probe) & mask]; + + if (!p->va_page || p->va_page == page) { + if (!p->va_page) + (*n)++; + p->va_page = page; + p->pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + p->ap = ap; + p->win = win; + return false; + } + } + (*drops)++; + return true; +} + static int svc_pin_lookup(u64 va, u64 *pa, int *ap) { return pin_lookup_tab(svc.pin, SVC_PIN_SLOTS, SVC_PIN_PROBE, va, pa, ap, @@ -2230,6 +4014,36 @@ static void svc_pin_store_win(u64 va, u64 pa, int ap) } /* + * SESSION 154 (M-GL1.2): drop a validated pin whose content no longer + * parses as the page's kind of stream (a stale translation kept answering + * for a page the guest has since reused - r913: va 0x1203c0000 read + * `first8=00000001` through a validated pin while the real push bytes sat + * elsewhere). The next read then consults the window guess, the trusted + * walk and the content sources instead of the dead pin. + */ +static void svc_pin_drop(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 mask = SVC_PIN_SLOTS - 1; + u32 h = (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & mask; + u32 probe; + + for (probe = 0; probe < SVC_PIN_PROBE; probe++) { + struct svc_pin *p = &svc.pin[(h + probe) & mask]; + + if (!p->va_page) + return; + if (p->va_page == page) + break; + } + if (probe == SVC_PIN_PROBE) + return; + memset(&svc.pin[(h + probe) & mask], 0, sizeof(struct svc_pin)); + if (svc.pin_n) + svc.pin_n--; +} + +/* * Injected (host-tool) translations. Kept separate from the discovered pin * table so a hand-verified VA->PA pair can never be shadowed, evicted or * silently dropped by the self-discovery stream: in the shared table a hot @@ -2239,6 +4053,7 @@ static void svc_pin_store_win(u64 va, u64 pa, int ap) * every push was "implausible"). */ static inline bool svc_pool_sem_va(u64 va); +static bool svc_kick_family(const struct svc_channel *ch); static void svc_pool_pa_forget(u64 va); /* @@ -2267,6 +4082,55 @@ struct svc_ipin_alt { static struct svc_ipin_alt svc_ipin_alt[SVC_IPIN_ALT_SLOTS]; static u64 svc_pushalt_hits; /* push reads served by a displaced pair */ static u32 svc_pushalt_log; +/* r707x: per-client push-read attempt witness (pump read site). */ +static u32 svc_rdw_log; +/* r820: bounded per-session USERD-scan decision witness (svc_chan_userd). + * GR channels only and capped: a hot CE channel re-running this every + * ~16 ms burned the whole budget before the channel under diagnosis ever + * appeared. Each scan reports its decision, a scan that found nothing + * reportable logs a sweep summary. */ +static u32 svc_userdp_log; +/* SESSION 153-CONTINUED-44 (r1026): bounded witness for zero-put USERD + * candidate skips (svc_chan_userd; the screen-pattern false block). */ +static u32 svc_zero_put_skip_log; +/* r829: bounded witness for the zero-entry holds (bad ring translations). */ +static u32 svc_zero_entry_log; +static u8 svc_read_last_src; +/* r715: PA the winning source of the last push read resolved to. */ +static u64 svc_read_last_pa; +/* + * r820: set when the last svc_read_va_push_ex chunk was answered by a + * driver-declared PROMOTE_CTX kernel buffer. The push decode uses it to + * accept the NOP sanity push: nvidia-push's __nvPushTestPushBuffer writes + * NVA16F_DMA_NOP (0x00000000) into a promoted kernel pushbuffer, so the + * whole small push is zeros - unparseable as a method stream, but exactly + * right from a translation the driver itself declared. + */ +static bool svc_read_last_promote; +/* r715: per-client push-decode branch witness (pump read site). */ +static u32 svc_pushdec_log; +/* r707d: bounded push-read failure witness (see svc_read_va_push_ex). */ +static u32 svc_pushfail_log; +/* + * SESSION 132: the util-sweep witnesses are per-CLIENT budgets (reset in + * svc_session_flush), not per-boot: on a multi-client boot the driver-init + * reads of an earlier client exhausted the boot budget and hid the later + * clients' rows (the same fix the r707x/r715 pump witnesses got). + */ +static u32 svc_util_dec_log; +static u32 svc_gr_util_log; +static struct { u64 va; u64 h; } svc_util_dec_keys[32]; +/* + * SESSION 143 (r803): per-channel sweep VISIBILITY. The r802 evidence could + * not say whether the Vulkan client's copy-class channels (0xbeef0101/2, + * eng 11/12) were swept at all: the per-pass log capped at 24 rows and the + * early return (no gpfifo/entries) prints nothing. These one-shot (per + * client, reset in svc_session_flush) keys name the first sweep of EVERY + * channel - begin or skip + reason - so "channel absent from the log" can + * never again be confused with "channel not in the table". + */ +static u32 svc_util_seen_keys[32]; +static u32 svc_util_seen_n; static int svc_ipin_alt_lookup(u64 va, u64 *pa, int *ap) { @@ -2333,8 +4197,32 @@ static void svc_ipin_store(u64 va, u64 pa, int ap) (old_pa & ~(u64)(SEMA_PAGE_SIZE - 1)) != (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) svc_ipin_alt_store(va, old_pa, old_ap); - pin_store_tab(svc.ipin, SVC_IPIN_SLOTS, SVC_IPIN_PROBE, &svc.ipin_n, - &svc.ipin_drop, va, pa, ap, 0); + /* + * SESSION 153-CONTINUED-64d: the table SATURATES under client map + * churn (r1122: `ipins=32768/148777` - the old store DROPPED every + * pin past the 32768 slots, so the demand stream's setup-push pages + * could never land and the held read never flipped). Slots: 1<<15. + * When the probe window is exhausted, evict the home slot's + * occupant instead of dropping: the evicted pair is preserved in + * the alt table (where a reader may still fall back to it) and both + * the scanner's periodic re-assert and the demand resend cadences + * restore any evicted entry that is still needed. A dropped demand + * pin, by contrast, is unrecoverable within the hold window. + */ + if (pin_store_tab_nodrop(svc.ipin, SVC_IPIN_SLOTS, SVC_IPIN_PROBE, + &svc.ipin_n, &svc.ipin_drop, va, pa, ap, 0)) { + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 mask = SVC_IPIN_SLOTS - 1; + u32 h = (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & mask; + struct svc_pin *p = &svc.ipin[h]; + + if (p->va_page && p->va_page != page) + svc_ipin_alt_store(p->va_page, p->pa_page, p->ap); + p->va_page = page; + p->pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + p->ap = ap; + p->win = 0; + } /* * A fresh injection supersedes every older translation for this * page: drop the verified pool cache entry so a subsequent pool @@ -2464,8 +4352,8 @@ static int svc_targeted_resolve(struct nv_vgpu_device *vgpu, u64 gpu_va, return -ENOMEM; end = tr_cursor + SVC_TR_SLICE; - if (end > SEMA_PA_MAX) - end = SEMA_PA_MAX; + if (end > svc_pa_max()) + end = svc_pa_max(); for (gpa = tr_cursor; gpa < end; gpa += SVC_TR_CHUNK) { u64 chunk = min_t(u64, SVC_TR_CHUNK, end - gpa); @@ -2494,7 +4382,7 @@ static int svc_targeted_resolve(struct nv_vgpu_device *vgpu, u64 gpu_va, if (cpu_va < direct_base) continue; pa = cpu_va - direct_base; - if (pa >= SEMA_PA_MAX || (pa & 0xfff)) + if (pa >= svc_pa_max() || (pa & 0xfff)) continue; *pa_out = pa | (gpu_va & (SEMA_PAGE_SIZE - 1)); @@ -2512,7 +4400,7 @@ static int svc_targeted_resolve(struct nv_vgpu_device *vgpu, u64 gpu_va, tr_cursor = end; kfree(buf); - if (tr_cursor >= SEMA_PA_MAX) { + if (tr_cursor >= svc_pa_max()) { pr_info("sema-svc: targeted resolve va=0x%llx: full RAM scan found nothing (scan #%llu)\n", page_va, tr_scans); svc_tr_note_failure(page_va); @@ -2548,30 +4436,94 @@ static bool svc_resolve_in_progress(u64 va) * VA above the window base, including sysmem-backed VAs, so callers that must * not act on a guess (push execution) use this variant. */ +/* + * r835: the VAS handle of the channel currently being serviced (0 = none). + * Channel-scoped VA resolution consults this VAS's page tables FIRST: the + * global pin/lat/map layers are VA-keyed and every client's RM window sits + * at the same VAs (0x1200xxxxx), so a lookup outside the owning client's + * page tables can answer ANOTHER client's page - the X server's ring + * resolved to the GL client's (zero) page and its pushes decoded empty, + * wedging X's readback wait exactly like the RC watchdog's ring starvation. + */ +static u32 svc_cur_vas; + +static int svc_chanvas_lookup(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, + int *ap) +{ + int i; + + if (!svc_cur_vas) + return -ENOENT; + for (i = 0; i < SVC_MAX_VASPACES; i++) { + u64 pa = 0; + int a = SVC_APERTURE_SYS; + + if (!svc.vas[i].valid || + svc.vas[i].h_vaspace != svc_cur_vas) + continue; + if (svc_gmmu_walk(vgpu, svc.vas[i].pdb, svc.vas[i].entries, + va & ~(u64)(SEMA_PAGE_SIZE - 1), + svc.vas[i].ap, &pa, &a, false)) + continue; + *addr = pa + (va & (SEMA_PAGE_SIZE - 1)); + *ap = a; + return 0; + } + return -ENOENT; +} + +/* + * SESSION 153-CONTINUED-67: which sub-source of svc_lookup_trusted answered + * the last call (1=channel VAS 2=pin 3=learned 4=discovered map 5=VAS walk + * 6=discovered walk 7=targeted 8=promote 0=miss). Printed as `sub=` by the + * pushdec witness so a repeated-zero read can be attributed to the exact + * layer (the r1127 investigation could not tell a stale learned entry from + * a stale registered-VAS walk). + */ +static u8 svc_trust_src; + static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, int *ap) { u64 pa = 0; + svc_trust_src = 0; + /* + * r835: the decoded channel's own page tables answer first. When a + * channel is being serviced (svc_cur_vas set) its VAS is the only + * authoritative source for its VAs; the VA-keyed layers below may + * hold another client's pages for the same numeric VA. A failed + * walk falls through to the old chain unchanged. + */ + if (!svc_chanvas_lookup(vgpu, va, addr, ap)) { + svc_trust_src = 1; + return 0; + } if (!svc_pin_lookup_any(va, addr, ap)) { + svc_trust_src = 2; return 0; } - if (!svc_lat_lookup(va, addr, ap)) + if (!svc_lat_lookup(va, addr, ap)) { + svc_trust_src = 3; return 0; + } if (!svc_map_lookup(va, &pa)) { *addr = pa; *ap = SVC_APERTURE_SYS; svc_lat_store(va, pa, SVC_APERTURE_SYS); + svc_trust_src = 4; return 0; } if (!svc_walk_vaspaces(vgpu, va & ~0xfffULL, addr, ap)) { *addr += va & 0xfff; svc_lat_store(va, *addr, *ap); + svc_trust_src = 5; return 0; } if (!svc_walk_discovered(vgpu, va & ~0xfffULL, addr, ap)) { *addr += va & 0xfff; svc_lat_store(va, *addr, *ap); + svc_trust_src = 6; return 0; } if (!svc_pd_found) { @@ -2586,6 +4538,7 @@ static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, if (!svc_walk_discovered(vgpu, va & ~0xfffULL, addr, ap)) { *addr += va & 0xfff; svc_lat_store(va, *addr, *ap); + svc_trust_src = 6; return 0; } } @@ -2594,6 +4547,25 @@ static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, *addr = pa; *ap = SVC_APERTURE_SYS; svc_lat_store(va, pa, SVC_APERTURE_SYS); + svc_trust_src = 7; + return 0; + } + /* + * r826: driver-declared PROMOTE_CTX buffers are the LAST resort, not + * the first. The promoted (va, pa) pair names a BOOKING, and even its + * base page can belong to a different buffer than the one mapped at + * that VA (r823: the ctx-MAIN booking's span swallowed the GPFIFO VA; + * r826: the PATCH booking's base VA 0x120001000 hosts nvidia-push's + * separately mapped 4 KB progress semaphore - writing the release to + * the booking's pa left the driver's own polled page at zero and the + * X server failed at its 10 s wait while the release value sat in the + * wrong page). The page-table walks describe the real mappings and + * must answer first; the promote record is only a fallback for VAs + * the walks cannot resolve at all. + */ + if (!svc_promo_lookup(va, addr, ap)) { + svc_lat_store(va, *addr, *ap); + svc_trust_src = 8; return 0; } return -ENOENT; @@ -2699,6 +4671,29 @@ static inline bool svc_pool_hold_active(void) } /* + * m165 (D-F2a): the blanket pool hold exists to stop a release decoded + * before the guest's own injection lands from being applied through a + * stale translation. Once a host-verified injected pin for THIS VA is + * present, that premise is gone: the pin source is the resolver's first + * choice (read-back proven) and the ipin table is wiped at session flush, + * so it is this session's ground truth. r156 measured every pool release + * waiting the full registration+5.00 s with the pin ready all along. + */ +static bool svc_pool_hold_active_va(u64 va) +{ + u64 ipa = 0; + int iap = 0; + + if (!svc_pool_hold_active()) + return false; + if (!gsp_sema_pool_hold_fresh) + return true; + if (svc_ipin_lookup(va, &ipa, &iap) == 0) + return false; + return true; +} + +/* * Verified pool-semaphore translations, keyed by page VA. An entry is only * created from an injected pin or from a targeted rm_mem resolve whose page * was read back successfully; the learned cache and the discovered map are @@ -2722,6 +4717,16 @@ static u32 svc_pool_pa_n; static u32 svc_pool_pa_next; static u64 svc_pool_pa_logged[SVC_POOL_PA_LOG_MAX]; static u32 svc_pool_pa_logged_n; +/* + * fail4: the completion-hint window was 10 s. It outlived short-lived + * clients: after a client freed its RM events the window kept re-posting + * hint events every 0.5 s, and a post already queued in the status ring + * could be read after the guest's own free processing - the guest ISR + * then ran a freed RC callback and the guest panicked in interrupt + * (rm_run_rc_callback+0x0). 1.5 s still covers a waiter that arrives + * after the edge while shrinking the post-free exposure ~7x. + */ +#define SVC_POST_HINT_MS 1500 static u64 svc_pool_pa_miss_log; static u32 svc_pool_pa_hit_log; static u32 svc_pool_wr_log; @@ -2770,6 +4775,14 @@ static void svc_pool_wr_note(u32 h, u64 va, u64 pa, const char *src, } } +/* r372: landing-table entries repaired from a fresh pin (stale-ledger class). */ +static u64 svc_ledger_pin_repairs; +/* r372: pool alt fan-outs skipped because the primary pin moved on. */ +static u64 svc_pool_alt_skipped; +/* r372: bounded page-note resolution diagnostics (va/pa/ap/branch). */ +static u32 svc_gr_pagenote_log; +static u32 svc_fbhpa_log; + /* * Fan a pool release out to the last displaced translation of the same page * (see svc_ipin_alt): the primary target may be the stale one when two host @@ -2793,6 +4806,27 @@ static void svc_pool_alt_apply(struct nv_vgpu_device *vgpu, u32 h, u64 va, return; if ((pa & ~(u64)(SEMA_PAGE_SIZE - 1)) == page) return; /* never write an identity translation */ + /* + * r372 (stale-ledger corruption class): fan out ONLY while the + * current primary pin for this VA still names the displaced page. + * Once a newer pin has displaced it, the old page belongs to a + * buffer the guest has freed - writing it lands on whatever the + * guest kernel reused the page for. A missing current pin is + * equally unprovable: skip. The primary release write already + * succeeded either way, so this is a lost best-effort, not a + * failure. + */ + { + u64 cpa = 0; + int cap = SVC_APERTURE_SYS; + + if (svc_ipin_lookup(va, &cpa, &cap) || + ((cpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + svc_pool_alt_skipped++; + return; + } + } if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &cur, sizeof(cur))) return; if (cur >= val) @@ -2825,6 +4859,15 @@ static inline bool svc_pool_pa_identity(u64 va, u64 pa) (va & ~(u64)(SEMA_PAGE_SIZE - 1)); } +static void svc_pool_boot_store(u64 va, u64 pa, int ap); /* m168 */ +static bool svc_dst_guard_hold(u64 va); /* m169b */ +static bool svc_dst_guard_should_hold(u64 va); /* r609 */ +static void svc_dst_guard_age(void); /* r609 */ +static void svc_dst_guard_reset(void); /* r609 */ +static bool svc_dst_guard_pending(u64 va); /* r231 */ +static u32 svc_read_order_log; /* r231 budget */ +static u32 svc_atrace_log; /* r234 A-TRACE budget */ + static int svc_pool_pa_lookup(u64 va, u64 *pa, int *ap) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); @@ -2855,6 +4898,8 @@ static void svc_pool_pa_store(u64 va, u64 pa, int ap) /* Never cache an identity translation (see svc_pool_pa_identity). */ if (svc_pool_pa_identity(page, pa)) return; + /* m168: every proven resolution also feeds the boot-scoped cache. */ + svc_pool_boot_store(va, pa, ap); for (i = 0; i < svc_pool_pa_n; i++) { if (svc_pool_pa[i].va_page == page) { svc_pool_pa[i].pa_page = @@ -2884,6 +4929,207 @@ static void svc_pool_pa_forget(u64 va) } /* + * m168 (D-F2c): boot-scoped verified pool-page cache. Within one guest boot + * the UVM semaphore pool (gpu->semaphore_pool, a 4 KB sysmem rm_mem at + * 0x121000000 whose 1024 words are the per-channel tracker slots) belongs + * to the shared GPU object; clients of the same boot reuse the pages while + * their per-process vaspaces come and go. The session flush wipes + * svc_pool_pa deliberately (a VA may name a new allocation for the next + * client), but wiping the POOL page forces every new client through the + * slow resolver: r170's clients 4-6 starved 2+ minutes on + * "pending release starved ... pool/waiter page" while the scanner learned + * the live tracker sems. This cache survives the flush; it is filled ONLY + * from read-back-proven resolutions (pin/cache/walk/vaspaces/targeted) and + * every use re-proves the exact offset readable, so a page that moved + * fails the proof and is dropped. A fresh injected pin always overwrites + * it (pins are the resolver's first source and run before this cache). + */ +struct svc_pool_boot_pa { + u64 va_page; + u64 pa_page; + int ap; + /* + * r539: secondary candidate. The pool VA can be answered by two + * read-back-proven sources that disagree (the injected pin - the + * guest kernel's own tracker->page->rm_mem chain - and the host + * GMMU walk of the client's page tables). Either can be the page + * the client actually polls: the r536 fix preferred the walk and + * the r538 init-stall boot showed the pin and the walk flapping + * every resolve while a client sat in uvm channel_pool_add -> + * uvm_push_end_and_wait. A release written only to the losing page + * strands the tracker. Keep BOTH pages and fan every pool release + * out to the secondary as well: the word is a forward-only + * monotonic counter, so a duplicate write is absorbed. The + * secondary is dropped when it stops being readable and ages out + * after SVC_POOL_DUAL_TTL_MS of not being re-proven by any source + * (a page the guest has since freed must not be written). + */ + u64 pa2_page; + int ap2; + unsigned long seen; /* last proof time of the primary */ + unsigned long seen2; /* last proof time of the secondary */ +}; + +static struct svc_pool_boot_pa svc_pool_boot[SVC_POOL_PA_MAX]; +static u32 svc_pool_boot_n; +static u32 svc_pool_boot_next; +static u32 svc_pool_boot_conflicts; +static u32 svc_pool_boot_hits; +/* r539: fresh-secondary window; the flapping sources (the watcher's ~2 s + * pool re-inject, the per-resolve walk) re-prove both candidates well inside + * this, while a page the guest has since freed and reused stops being + * re-proven and ages out before any write can land on it (the r372 + * stale-page corruption rule). */ +#define SVC_POOL_DUAL_TTL_MS 10000u + +static int svc_pool_boot_lookup(u64 va, u64 *pa, int *ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_pool_boot_n; i++) { + if (svc_pool_boot[i].va_page == page) { + if (svc_pool_pa_identity(page, + svc_pool_boot[i].pa_page)) + return -ENOENT; + *pa = svc_pool_boot[i].pa_page | + (va & (SEMA_PAGE_SIZE - 1)); + *ap = svc_pool_boot[i].ap; + return 0; + } + } + return -ENOENT; +} + +static void svc_pool_boot_store(u64 va, u64 pa, int ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + if (svc_pool_pa_identity(page, pa)) + return; + for (i = 0; i < svc_pool_boot_n; i++) { + if (svc_pool_boot[i].va_page == page) { + if (svc_pool_boot[i].pa_page == pa_page) { + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].seen = jiffies; + return; + } + if (svc_pool_boot[i].pa2_page == pa_page) { + /* The previously-secondary page is the + * freshest proof: promote it, demote the + * old primary. No conflict line: both + * candidates stay live and both receive + * the fan-out. */ + u64 op = svc_pool_boot[i].pa_page; + int oap = svc_pool_boot[i].ap; + unsigned long oseen = svc_pool_boot[i].seen; + + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].seen = jiffies; + svc_pool_boot[i].pa2_page = op; + svc_pool_boot[i].ap2 = oap; + svc_pool_boot[i].seen2 = oseen; + return; + } + /* + * r539: a second, DIFFERENT page. Keep the old + * primary as the secondary candidate instead of + * discarding it - the r538 init-stall boot had the + * pin and the walk flapping forever, and the client + * polls whichever page its own mapping names. + */ + svc_pool_boot_conflicts++; + pr_info_ratelimited("sema-svc: pool-boot second candidate va=0x%llx primary=0x%llx -> 0x%llx (old primary kept as secondary)\n", + page, svc_pool_boot[i].pa_page, pa_page); + svc_pool_boot[i].pa2_page = svc_pool_boot[i].pa_page; + svc_pool_boot[i].ap2 = svc_pool_boot[i].ap; + svc_pool_boot[i].seen2 = svc_pool_boot[i].seen; + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].seen = jiffies; + return; + } + } + if (svc_pool_boot_n < SVC_POOL_PA_MAX) { + i = svc_pool_boot_n++; + } else { + i = svc_pool_boot_next++ % SVC_POOL_PA_MAX; + } + svc_pool_boot[i].va_page = page; + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].pa2_page = 0; + svc_pool_boot[i].ap2 = 0; + svc_pool_boot[i].seen = jiffies; + svc_pool_boot[i].seen2 = 0; +} + +/* + * r539: fan one applied pool release out to the secondary candidate page + * (see struct svc_pool_boot_pa). Forward-only and read-back proven, exactly + * like the primary write; a failure is counted, never propagated - the + * primary write already landed. A secondary that stopped being readable is + * dropped; one that was not re-proven by any source for + * SVC_POOL_DUAL_TTL_MS is left alone (the guest may have reused the page). + */ +static void svc_pool_boot_fanout(struct nv_vgpu_device *vgpu, u32 h, u64 va, + u32 val) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + u32 cur = 0; + u32 back = 0; + u32 i; + + for (i = 0; i < svc_pool_boot_n; i++) + if (svc_pool_boot[i].va_page == page) + break; + if (i == svc_pool_boot_n || !svc_pool_boot[i].pa2_page) + return; + if (!svc_pool_boot[i].seen2 || + time_after(jiffies, svc_pool_boot[i].seen2 + + msecs_to_jiffies(SVC_POOL_DUAL_TTL_MS))) + return; + pa = svc_pool_boot[i].pa2_page | (va & (SEMA_PAGE_SIZE - 1)); + ap = svc_pool_boot[i].ap2; + if (svc_pool_pa_identity(va, pa)) + return; + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &cur, sizeof(cur))) { + /* The page is gone: drop the stale secondary. */ + svc_pool_boot[i].pa2_page = 0; + return; + } + if (cur >= val) + return; /* forward-only: nothing to add */ + if (svc_write_ap(vgpu, pa, svc_ap_of(ap), &val, sizeof(val))) + return; + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &back, sizeof(back))) + return; + if (back != val) + return; + svc.pool_dual_applied++; + if (svc.pool_dual_log < 128) { + svc.pool_dual_log++; + pr_info("vgsp-sema: pool-fanout ch=0x%x va=0x%llx pa2=0x%llx cur=%u -> val=%u (secondary candidate received the release)\n", + h, va, pa, cur, val); + } +} + +static void svc_pool_boot_forget(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_pool_boot_n; i++) + if (svc_pool_boot[i].va_page == page) + svc_pool_boot[i].va_page = 0; +} + +/* * Resolve one pool-semaphore target the way the successful deferred apply did: * injected pin (host-verified ground truth), the verified cache, then the * targeted rm_mem record scan, whose result is read back before it is cached. @@ -2916,19 +5162,139 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, * the injected page is readable before trusting it; a failed * proof falls through to the read-back-validated cache, the * structural walk and the targeted rm_mem record. + * + * r536: the read-back proof does NOT catch a moved pool whose + * OLD page is still readable - the write then lands on the old + * page while the client polls the new one and the tracker + * starves (observed: "pending release starved va=0x12100000x + * tries=64" for ~45 s after the pool moved, with the stale pin + * pa=0x120b30000 still readable; the fresh pin pa=0x130e9d000 + * arrived 45 s later). The client's OWN page tables are the + * GPU/UVM ground truth for where the tracker reads, so when a + * structural walk resolves a DIFFERENT page than the pin, the + * walk wins (both are read-back proven at the exact offset). */ u32 probe = 0; if (!svc_read_ap(vgpu, pa, svc_ap_of(a), &probe, sizeof(probe))) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + bool walk_hit = false; + + if (!svc_walk_discovered(vgpu, page, &wpa, &wap) || + !svc_walk_vaspaces(vgpu, page, &wpa, &wap)) + walk_hit = true; + if (walk_hit && + (wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + !svc_pool_pa_identity(va, wpa)) { + u32 wprobe = 0; + u64 exact = (wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) | + (va & (SEMA_PAGE_SIZE - 1)); + + if (!svc_read_ap(vgpu, exact, svc_ap_of(wap), + &wprobe, sizeof(wprobe))) { + /* + * r539: record the PIN as the + * secondary candidate before the walk + * page becomes the primary. The walk + * is not infallible (the r538 boot: + * pin and walk flapped on every + * resolve and the client polled the + * pin's page); keeping both means the + * release fan-out reaches whichever + * page the client actually polls. + */ + svc_pool_boot_store(va, pa, a); + svc_pool_pa_store(va, exact, wap); + svc_tr_clear_failure(page); + if (src) + *src = "walk-over-pin"; + svc_pool_pa_note(h, va, exact, + "walk-over-pin"); + *addr = exact; + *ap = wap; + return 0; + } + } svc_tr_clear_failure(page); if (src) *src = "pin"; svc_pool_pa_note(h, va, pa, "pin"); + svc_pool_boot_store(va, pa, a); + *addr = pa; + *ap = a; + return 0; + } + } + /* + * m168: boot-scoped page, survives the session flush. Every use + * re-proves the exact offset readable: a page that moved fails the + * proof and is dropped (the fresh pin path above always wins when a + * pin exists). + */ + if (!svc_pool_boot_lookup(va, &pa, &a)) { + u32 bprobe = 0; + + /* + * r580: the boot cache survives session flushes, but a pool + * page MOVES between sessions (the client's new allocation) + * while the old page often stays readable - the read-back + * proof cannot tell. The client's own page tables are the + * ground truth for where ITS tracker reads, so try the + * structural walk BEFORE trusting the cache: a differing, + * read-proven walk page wins and the cached page stays as the + * fan-out's secondary candidate. Live evidence (r579 boot): + * releases were written to the stale cache pa=0x11e6d9000 + * while the current page was 0x134730000 (the pin) and the + * client's tracker starved; the pool-wr line said + * src=bootcache. + */ + { + u64 wpa2 = 0; + int wap2 = SVC_APERTURE_SYS; + bool walk_hit = false; + + if (!svc_walk_discovered(vgpu, page, &wpa2, &wap2) || + !svc_walk_vaspaces(vgpu, page, &wpa2, &wap2)) + walk_hit = true; + if (walk_hit && + (wpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + !svc_pool_pa_identity(va, wpa2)) { + u64 exact = (wpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) | + (va & (SEMA_PAGE_SIZE - 1)); + + if (!svc_read_ap(vgpu, exact, svc_ap_of(wap2), + &bprobe, sizeof(bprobe))) { + svc_pool_boot_store(va, pa, a); + svc_pool_pa_store(va, exact, wap2); + svc_tr_clear_failure(page); + if (src) + *src = "walk-over-boot"; + svc_pool_pa_note(h, va, exact, + "walk-over-boot"); + *addr = exact; + *ap = wap2; + return 0; + } + } + } + + if (!svc_read_ap(vgpu, pa, svc_ap_of(a), &bprobe, + sizeof(bprobe))) { + svc_tr_clear_failure(page); + svc_pool_boot_hits++; + if (src) + *src = "bootcache"; *addr = pa; *ap = a; return 0; } + pr_info("sema-svc: pool-boot stale va=0x%llx pa=0x%llx (read-back failed; dropping)\n", + va, pa); + svc_pool_boot_forget(va); } if (!svc_pool_pa_lookup(va, &pa, &a)) { svc_tr_clear_failure(page); @@ -3104,6 +5470,17 @@ static int svc_sem_read(struct nv_vgpu_device *vgpu, u64 va, u32 *val) } static void svc_pending_add(u64 va, u32 payload, u32 op); +/* r34x: defined with the pending ring / opdef slots (later in the file); + * the acquire gate drains/appraises withheld releases before it holds. */ +static void svc_apply_pending(struct nv_vgpu_device *vgpu); +static bool svc_wait_pend_release(u64 va); + +/* + * F1: GR release-word authorship checks. Defined next to the svc_fwd_req + * slot; forward-declared because the sweep writer runs earlier in the file. + */ +static bool svc_gr_rel_match(u64 va, u32 *payload); +static bool svc_gr_rel_inflight(u64 va, u32 *payload); static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) { @@ -3116,6 +5493,7 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) int wap, rc; bool verified = false; bool rb_ok = false; + static u32 sem_sw_dev_log; /* r619: non-pool device-VA write budget */ /* * Pool semaphores resolve pin -> verified cache -> targeted rm_mem @@ -3175,10 +5553,37 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) * sysmem semaphore range (0x792..), with the resolver provenance and * the read-back, so the second session's translations can be * compared with the first session's for the same VA. Bounded. + * + * r619 diag: the same proof for NON-pool device-VA writes (the VAS + * window >= 0x200000000; e.g. 0x20480fef0 push releases, 0x204827f7c + * QMD releases). The r618 hung math client read 0 at 0x20480fef0 + * while the host's deferred releases logged `cur` 97..105 on its own + * target - a write-target divergence that was invisible because this + * range never printed `pa`. r620: the first cut used the 1 TiB + * SVC_VAS_FLOOR and excluded this very window (no lines ever); + * bounded per boot (512). */ - if (va >= 0x79240000000ULL && va < 0x79260000000ULL && - svc.sem_sw_log < 8192) { - svc.sem_sw_log++; + if (((va >= 0x79240000000ULL && va < 0x79260000000ULL && + svc.sem_sw_log < 8192) || + (va >= 0x200000000ULL && !is_pool && sem_sw_dev_log < 4096) || + /* + * SESSION 153-CONT-7 (GL client attribution): the X/GL lanes' + * semaphore window. The X server's progress-tracker release + * (va=0x120001000, kernel progress semaphore) and the GL + * channels' low-aperture objects sit below 0x200000000, so + * their write attempts were invisible (no sema-sw line) while + * the client's polled word stayed zero - the r939 boot left + * no trace of WHERE such a write landed. Bounded like the + * other windows so a hot loop cannot flood dmesg. + */ + (va >= 0x120000000ULL && va < 0x120600000ULL && + svc.sem_sw_xg_log < 2048))) { + if (va >= 0x79240000000ULL && va < 0x79260000000ULL) + svc.sem_sw_log++; + else if (va >= 0x120000000ULL && va < 0x120600000ULL) + svc.sem_sw_xg_log++; + else + sem_sw_dev_log++; pr_info("sema-sw: h=0x%x va=0x%llx pa=0x%llx ap=%d src=%s val=0x%x rc=%d rb_ok=%d back=0x%x pool=%d\n", h, va, addr, wap, src, val, rc, rb_ok, back, is_pool); } @@ -3186,6 +5591,7 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) if (is_pool) { svc_pool_wr_note(h, va, addr, src, cur, val, "ok"); svc_pool_alt_apply(vgpu, h, va, val); + svc_pool_boot_fanout(vgpu, h, va, val); } return 0; } @@ -3212,6 +5618,7 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) svc_pool_wr_note(h, va, addr, "targeted", cur, val, "reresolve"); svc_pool_alt_apply(vgpu, h, va, val); + svc_pool_boot_fanout(vgpu, h, va, val); } svc.sem_write_reresolve++; pr_info_ratelimited("vgsp-sema: sem-write re-resolve va=0x%llx pa=0x%llx old=0x%llx new=0x%x\n", @@ -3224,6 +5631,7 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) svc_pool_wr_note(h, va, old_addr, src, cur, val, "advisory"); svc_pool_alt_apply(vgpu, h, va, val); + svc_pool_boot_fanout(vgpu, h, va, val); } return 0; /* the original write landed; read-back is advisory */ } @@ -3260,76 +5668,92 @@ static void svc_scan_pass(struct nv_vgpu_device *vgpu, u8 *buf, u64 gpa; u32 nonz = 0; - for (gpa = 0; gpa < SEMA_PA_MAX; gpa += SEMA_PAGE_SIZE) { - u32 off; + /* + * r534: read guest RAM in SVC_SCAN_CHUNK (64 KB) blocks instead of + * one 4 KB page per vfio_dma_rw call. The full 6 GB scan is the + * rebuild's whole cost (1.5 M guest reads ~= 10-11 s per rebuild, + * once per 10 s: the pump got one pass per rebuild and the second + * client's resolutions could wait minutes). A 64 KB read amortizes + * the iommu walk + pin over 16 pages, cutting the scan to well under + * a second with identical pair/base detection (the inner page loop + * is byte-for-byte the old per-page body). + */ + for (gpa = 0; gpa < svc_pa_max(); gpa += SVC_SCAN_CHUNK) { + u32 po; if (READ_ONCE(svc_sess_flush)) break; /* ISSUE-0002: a flush is waiting on this scan */ - if (svc_read(vgpu, gpa, buf, SEMA_PAGE_SIZE)) + if (svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) continue; - for (off = 0; off + 16 <= SEMA_PAGE_SIZE; off += 8) { - u64 a, b; + for (po = 0; po < SVC_SCAN_CHUNK; po += SEMA_PAGE_SIZE) { + u8 *page = buf + po; + u64 pg = gpa + po; + u32 off; - memcpy(&a, buf + off, 8); - memcpy(&b, buf + off + 8, 8); + for (off = 0; off + 16 <= SEMA_PAGE_SIZE; off += 8) { + u64 a, b; - /* (a) self-pointer -> base candidate */ - if (a >= KVA_LO && a < KVA_HI && - (a & 0xfff) == ((gpa + off) & 0xfff)) { - u64 cb = a - gpa - off; - u32 i; + memcpy(&a, page + off, 8); + memcpy(&b, page + off + 8, 8); - if ((cb & 0x3fffffffULL) == 0) { - for (i = 0; i < *ncand; i++) { - if (cands[i].base == cb) { - cands[i].count++; - break; + /* (a) self-pointer -> base candidate */ + if (a >= KVA_LO && a < KVA_HI && + (a & 0xfff) == ((pg + off) & 0xfff)) { + u64 cb = a - pg - off; + u32 i; + + if ((cb & 0x3fffffffULL) == 0) { + for (i = 0; i < *ncand; i++) { + if (cands[i].base == cb) { + cands[i].count++; + break; + } + } + if (i == *ncand && *ncand < 8) { + cands[*ncand].base = cb; + cands[*ncand].count = 1; + (*ncand)++; } - } - if (i == *ncand && *ncand < 8) { - cands[*ncand].base = cb; - cands[*ncand].count = 1; - (*ncand)++; } } - } - /* (b) {cpu_va, gpu_va} pair (page aligned: real mappings) */ - if (a >= KVA_LO && a < KVA_HI && - b >= SEMA_VA_LO && b < SEMA_VA_HI && - (a & 0xfff) == 0 && (b & 0xfff) == 0 && - *npairs < PAIR_MAX) { - u8 ok = 0; + /* (b) {cpu_va, gpu_va} pair (page aligned: real mappings) */ + if (a >= KVA_LO && a < KVA_HI && + b >= SEMA_VA_LO && b < SEMA_VA_HI && + (a & 0xfff) == 0 && (b & 0xfff) == 0 && + *npairs < PAIR_MAX) { + u8 ok = 0; - /* - * UVM's uvm_rm_mem_t stores vas[UVM_ID_CPU] - * (offset 48) directly before vas[UVM_ID_GPU0] - * (offset 56), so a pair taken from a live rm_mem - * is preceded by the header {u32 type; bitmap - * mapped_on}. Validate it when it fits in the same - * page: freed allocations and unrelated tables - * that happen to hold KVA/VA pairs do not match, - * and a stale entry for the same VA must not - * shadow the live one. - */ - if (off >= 48) { - u32 type; - u64 mapped_on; - - memcpy(&type, buf + off - 48, 4); - memcpy(&mapped_on, buf + off - 40, 8); - if ((type == 0 || type == 1) && - (mapped_on & 0x3) == 0x3) - ok = 1; + /* + * UVM's uvm_rm_mem_t stores vas[UVM_ID_CPU] + * (offset 48) directly before vas[UVM_ID_GPU0] + * (offset 56), so a pair taken from a live rm_mem + * is preceded by the header {u32 type; bitmap + * mapped_on}. Validate it when it fits in the same + * page: freed allocations and unrelated tables + * that happen to hold KVA/VA pairs do not match, + * and a stale entry for the same VA must not + * shadow the live one. + */ + if (off >= 48) { + u32 type; + u64 mapped_on; + + memcpy(&type, page + off - 48, 4); + memcpy(&mapped_on, page + off - 40, 8); + if ((type == 0 || type == 1) && + (mapped_on & 0x3) == 0x3) + ok = 1; + } + pairs[*npairs].cpu_va = a; + pairs[*npairs].gpu_va = b; + pairs[*npairs].valid_hdr = ok; + (*npairs)++; } - pairs[*npairs].cpu_va = a; - pairs[*npairs].gpu_va = b; - pairs[*npairs].valid_hdr = ok; - (*npairs)++; } + nonz++; } - nonz++; } } @@ -3378,7 +5802,7 @@ static void svc_add_pair_merge(u64 base, const struct raw_pair *p, u32 sorted_n) if (p->cpu_va < base) return; pa = p->cpu_va - base; - if (pa >= SEMA_PA_MAX || (pa & 0xfff)) + if (pa >= svc_pa_max() || (pa & 0xfff)) return; if (p->gpu_va < SEMA_VA_LO || p->gpu_va >= SEMA_VA_HI) return; @@ -3489,6 +5913,157 @@ static int svc_map_lookup_range(u64 gpu_va, u32 from, u32 to, u64 *pa) static void svc_session_flush(void); /* defined next to the pending queue */ +/* + * Ledger sizing + counters hoisted above svc_rebuild so the periodic diag + * can report the table's fill state: a full table degrades the write/read + * authority silently (home replacement evicts another VA's landing), which + * is exactly the D2H "reads back prior content" window. + */ +#define SVC_LEDGER_MAX (1u << 14) +#define SVC_LEDGER_MASK (SVC_LEDGER_MAX - 1u) +#define SVC_LEDGER_PROBE 16u +/* + * STALLFREE: eviction burst threshold for the cap-and-degrade reset. A + * cluster keeps 16 same-home VA pages; EXIT_RESET evictions inside one session + * means the table is thrashing (pin churn / leftover crashed clients), and + * the half-evicted table silently strands the evicted pages' landing + * authority. At the threshold the cache is cleared and rebuilt from live + * writes (uniform resolver fallback instead of silent partial authority). + */ +#define SVC_LEDGER_EVICT_RESET 256u +static u32 svc_ledger_n; +static u64 svc_ledger_diverges; +static u64 svc_ledger_pinwins; /* r336: pin-backed divergence wins */ +static u32 svc_d2hmap_log; /* D2H-MAP per-page destination budget */ +/* + * r336/r34x: host-semaphore acquire (C56F SEM_EXECUTE op 0/2/3/4/5) wait + * state. The decoder captures EVERY acquire of the ring entry currently + * being decoded (svc_waits[], reset per entry at the top of + * svc_decode_stream), and svc_run_ops honors them before running any of + * the entry's work, so a copy cannot execute ahead of the dependency its + * push waits on (the streams2 readback-before-scale class). Per-entry + * capture is essential: the wait state used to leak across entries (a + * later launch's self-release wait, e.g. need=85 while the held push + * releases 84, gated an earlier entry that carries no acquire at all - + * the r340 over-hold). The re-decode re-parses the entry, so the state + * is refreshed on every pass. + * + * Semantics per op (class contract, see svc_wait_satisfied): + * 0 ACQUIRE = (v == p); 2 ACQ_STRICT_GEQ = v >= p; 3 ACQ_CIRC_GEQ = + * circular (s32)(v - p) >= 0; 4 ACQ_AND = (v & p) != 0; + * 5 ACQ_NOR = ~(v | p) != 0. + */ +#define SVC_WAIT_MAX 8 +/* + * r401: bounded acquire wait (fail-open), now a module parameter. The + * original 100 ms bound was shorter than a single GR launch's completion + * latency in this emulation (observed 0.5-1.6 s per launch), so a copy + * that waited on a launch's tracker (streams2: the D2H readback's + * ACQ_CIRC_GEQ on the scale channel's progress counter) gave up and ran + * BEFORE the launch wrote its output - the host read pre-scale data + * (bad=32767, c2 all wrong). 4 s covers several launch latencies; the + * bound still guarantees the entry is never parked forever (a genuinely + * unsatisfiable acquire fails open late, after the dependency's own work + * would have landed - late-but-correct instead of early-and-wrong). + */ +unsigned gsp_sema_wait_hold_ms = 4000; +struct svc_wait_op { + u64 va; + u32 payload; + u8 op; /* C56F SEM_EXECUTE operation */ + u32 off; /* method offset in the entry (order) */ +}; +static struct svc_wait_op svc_waits[SVC_WAIT_MAX]; +static u32 svc_wait_n; /* acquires captured this entry */ +/* + * r484 THE STAGING-DRAIN COMPLETION EDGE: when the acquire gate holds an + * entry, the op can still have executed on an earlier decode whose gate did + * not trigger (svc_wait_n was 0 on that pass), and the once-only done + * bitmap then suppresses it on the first pass whose gate finally passes - + * the pass at which the dependency just became installed and the op's + * inputs changed. The release then applies and the guest drains a staging + * buffer holding the PRE-dependency bytes (streams_events s=2: the D2H + * copy read dc2 before the vadd_into wrote it, ran early on gate-less + * passes, was once-only-skipped on the passing pass, and the client + * drained a2+b2 instead of a2+b2+c1). Remember the last held (channel, + * entry) and force exactly ONE re-run of its ops on the first passing + * pass. + */ +static u32 svc_wait_hold_h; +static u32 svc_wait_hold_idx = 0xffffffffu; +static bool svc_wait_repass; +static u32 svc_wait_ovf; /* acquires dropped past the array */ +static u64 svc_wait_held; +static u64 svc_wait_pass; +static u64 svc_wait_giveup; +static u64 svc_wait_extended; /* r613: give-up deferred while a + * GR forward is owed */ +static u32 svc_wait_ext_log; +/* + * r613: episode extension bound while a GR forward is owed. Comfortably + * above wait-hold (default 4000 ms) + the absolute launch defer cap + * (SVC_GR_DEFER_CAP_MS, 15000 ms) + margin, and far below every client-side + * timeout (>= 60 s). + * + * r640: raised 25 s -> 45 s. The r639 streams2 transcript measured the + * readback's episode give up at waited_ms=28378 while the launch whose + * release it waits for only forwarded (GR-STALL-backend, software release + * authored) at 419322.99 - ~9 s AFTER the give-up - because the launch's + * own path paid the watermark defer (15 s cap) + arg-partial defer (8 s) + + * backend poll before it moved. The readback then ran out of order and the + * client read zeros (bad=65534). 45 s still leaves >15 s of headroom under + * the clients' >= 60 s timeouts, and only an episode with an unoffered + * candidate ever reaches it. + */ +#define SVC_WAIT_EXT_CAP_MS 45000u +static u32 svc_wait_log; /* per-session held-row print budget */ +static u32 svc_wait_pass_log; /* per-session resolved-row budget */ +static u32 svc_remapconst_log; /* r343 FILL: SET_REMAP_CONST_A rows */ +static u32 svc_pitch_log; /* r402: 2D (pitched) op builds */ +static u32 svc_fill_raw_log; /* r365 FILL-RAW: raw dropped-fill dumps */ +static u32 svc_lowfill_log; /* r365b LOWFILL-WIT: below-floor fills */ +static u32 svc_tabcopy_log; /* r367 TABLE-COPY: table-page copies */ +static u32 svc_fill_noremap; /* r343 FILL: fail-closed holds */ +static u64 svc_wait_rel_va; /* this entry's own release (mirror) */ +static u32 svc_wait_rel_payload; +static u32 svc_wait_rel_off; /* its method offset (order check) */ +static bool svc_wait_rel_have; +/* + * Per-channel first-hold clock for the bounded wait: a gate evaluation + * that cannot satisfy the acquire arms a 100 ms budget per hold episode; + * expiry fails open (ops run) with a loud 'give-up' row. The table is + * tiny (a handful of channels), service-thread only, cleared per pass + * resolution / session flush. + */ +struct svc_wait_hold { + bool active; + u32 h; + u32 idx; /* r638: GPFIFO entry index this hold owns + * (0xffffffff when unset); the CE stream-order + * watermark uses it to detect a mutual citation + * (a held consumer entry, not a producer backlog) */ + unsigned long since; +}; +#define SVC_WAIT_HOLD_MAX 16 +static struct svc_wait_hold svc_wait_holds[SVC_WAIT_HOLD_MAX]; +static u32 svc_dst_div_log; /* r402: copy-dst ledger/probe divergence */ +static u64 svc_ledger_stores; +static u64 svc_ledger_evictions; +static u64 svc_ledger_evict_since_reset; +static u64 svc_ledger_evict_resets; +static u64 svc_ledger_rewrites; +static u64 svc_ledger_reloc_diverges; +static u32 svc_ledger_store_log; +/* + * STALLFREE: pending-ring overflow telemetry (the negative queue silently + * overwrote its oldest entry when full). Hoisted here so svc_rebuild's diag + * can print it; the ring itself is defined with the pending queue below. + */ +static u64 svc_pending_ovf; +static u32 svc_pending_ovf_log; +/* STALLFREE: rescue scans that hit their wall-clock cap (bounded service). */ +static u64 svc_rescue_scan_capped; + static void svc_rebuild(struct nv_vgpu_device *vgpu) { static u8 *buf; @@ -3511,7 +6086,8 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) return; if (!buf) { - buf = kmalloc(SEMA_PAGE_SIZE, GFP_KERNEL); + /* r534: sized for svc_scan_pass's 64 KB chunked guest reads. */ + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); if (!buf) return; } @@ -3591,18 +6167,25 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) svc.false_trans, svc.gpfifo_skipped, svc.gpfifo_rescued, svc.releases_faked, svc.releases_recovered, svc.counter_rel_applied); - pr_info("sema-svc: drain held=%llu starved=%llu forwarded=%llu faked=%llu rec=%llu ops_once=%llu reident=%llu opdef=%llu/%llu/%llu\n", + pr_info("sema-svc: drain held=%llu starved=%llu forwarded=%llu faked=%llu rec=%llu ops_once=%llu reident=%llu opdef=%llu/%llu/%llu gr_nocap=%llu/%llu gr_owed=%llu gr_hold_exp=%llu gr_swrel=%llu gr_wedge=%llu pend_ovf=%llu\n", svc.gpfifo_held, svc.gpfifo_starved, svc.releases_forwarded, svc.releases_faked, svc.releases_recovered, svc.ops_once_skipped, svc.ops_reident, - svc.opdef_held, svc.opdef_applied, svc.opdef_expired); - pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu altw=%llu residue=%llu\n", + svc.opdef_held, svc.opdef_applied, svc.opdef_expired, + svc.gr_nocap_held, svc.gr_nocap_consumed, + svc.gr_owed_resolved, + svc.fwd_gr_hold_exp, svc.fwd_gr_stall_swrel, + svc.gr_stale_wedge, svc_pending_ovf); + pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu kbpg=%llu altw=%llu residue=%llu wait=%llu/%llu giveup=%llu ovf=%u\n", svc.counter_pub_calls, svc.gr_rel_applied, svc.sem_write_reresolve, svc.pushpg_fallback, - svc.pool_alt_applied, svc.pool_residue); - if (svc.resc_run) - pr_info("sema-svc: rescue_run=%llu (multi-page content pins)\n", - svc.resc_run); + svc.kick_badpg, + svc.pool_alt_applied, svc.pool_residue, + svc_wait_held, svc_wait_pass, svc_wait_giveup, + svc_wait_ovf); + if (svc.resc_run || svc_rescue_scan_capped) + pr_info("sema-svc: rescue_run=%llu scan_capped=%llu (multi-page content pins)\n", + svc.resc_run, svc_rescue_scan_capped); pr_info("sema-svc: gpfifo done=%llu nomap=%llu vidmem_writes=%llu ring_auto=%u pb=%u/%u pins=%u/%llu ipins=%u/%llu\n", svc.gpfifo_entries_done, svc.gpfifo_nomap, svc.vidmem_writes, svc_ring_n, svc_pb_n, svc_pb_passes, @@ -3611,6 +6194,20 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) svc_vas_n, SVC_VAS_SLOTS, svc.vas_writes, svc.vas_reads, svc.vas_shadowed); /* + * D2H-source authority visibility (r220): a full ledger degrades to + * home-slot replacement, which silently evicts another VA's landing + * and lets readers fall back to the resolver - the "reads back prior + * content" window. n approaching SVC_LEDGER_MAX or evict > 0 in a + * client's window is the tell. + */ + pr_info("sema-svc: ledger n=%u/%u stores=%llu evict=%llu resets=%llu diverges=%llu pinwins=%llu reloc_div=%llu rewrites=%llu pinrep=%llu alt_skip=%llu dual=%llu/%u\n", + svc_ledger_n, SVC_LEDGER_MAX, svc_ledger_stores, + svc_ledger_evictions, svc_ledger_evict_resets, + svc_ledger_diverges, svc_ledger_pinwins, + svc_ledger_reloc_diverges, svc_ledger_rewrites, + svc_ledger_pin_repairs, svc_pool_alt_skipped, + svc.pool_dual_applied, svc_pool_boot_conflicts); + /* * P6b wave 1: guest-forwarding summary. Printed whenever any counter * moved (not only on requests) so a run that captured no candidate is * still diagnosable from the log. @@ -3668,6 +6265,13 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) #define CE_M_OFFSET_OUT_UPPER 0x408 #define CE_M_OFFSET_OUT_LOWER 0x40c #define CE_M_LINE_LENGTH_IN 0x418 +#define CE_M_LINE_COUNT 0x41c +#define CE_M_PITCH_IN 0x410 +#define CE_M_PITCH_OUT 0x414 +#define CE_M_SRC_ORIGIN_X 0x744 +#define CE_M_SRC_ORIGIN_Y 0x748 +#define CE_M_DST_ORIGIN_X 0x74c +#define CE_M_DST_ORIGIN_Y 0x750 #define CE_M_SET_REMAP_CONST_A 0x700 #define CE_M_SET_REMAP_CONST_B 0x704 #define CE_M_SET_REMAP_COMPONENTS 0x708 @@ -3742,6 +6346,19 @@ struct svc_memset_op { u64 bytes; /* copy length */ u32 src_phys; u32 src_aperture; + /* + * r402: 2D (pitched) transfers. rows > 1 selects row-wise mode: + * the op runs `rows` times with the destination advancing by + * dpitch (and the source by spitch) per row, each row carrying + * `bytes` (copy) or elem*count (fill) bytes. cudaMemset2D and + * cudaMemcpy2D issue exactly this shape (LINE_COUNT rows, + * PITCH_OUT/PITCH_IN in bytes); without it the fill/copy ran flat + * and the partial-width rows corrupted the neighbouring columns + * (victory_memset2d bad>12M, memcpy_roundtrip pitched class). + */ + u32 rows; + u32 dpitch; + u32 spitch; }; /* Decoder state carried across the methods of one push. */ @@ -3753,9 +6370,31 @@ struct svc_push_state { u32 have_in, have_out; u32 line_length; u32 gr_line_count; /* GRCE LINE_COUNT */ + /* + * r402: 2D transfer geometry (CE class): PITCH_IN/PITCH_OUT in + * bytes and LINE_COUNT rows. A pitched fill/copy whose pitch + * exceeds the row length must run row-wise; a flat transfer leaves + * pitch == line_length (or no pitch method at all). + */ + u32 pitch_in, pitch_out; + u32 have_pitch_in, have_pitch_out; + u32 line_count; + /* + * SESSION 153 CONT.25: C5B5 SRC_ORIGIN_X/Y (0x744/0x748) and + * DST_ORIGIN_X/Y (0x74c/0x750). The X DDX's read/present paths + * position each strip with SRC_ORIGIN (x in bytes, y in rows for + * compSizeFactor == Bpp == 4; class C5B5 has no null remapper) + * instead of folding it into OFFSET_IN. The linear shadow holds + * the pixels linearly, so the effective operand becomes + * base + origin_y * pitch + origin_x. + */ + u32 src_origin_x, src_origin_y; + u32 dst_origin_x, dst_origin_y; + u32 have_src_origin, have_dst_origin; u32 sem_va_hi, sem_va_lo, sem_payload; u32 have_sem; u32 have_remap; + u32 have_remap_a, have_remap_b, have_remap_c; u32 release_dma; u32 sem_inc; u32 rs_upper, rs_lower, rs_payload; @@ -3775,6 +6414,66 @@ struct svc_push_state { u32 qmd[SVC_QMD_WORDS]; }; +/* + * SESSION 153 CONT.25: CE register state persists across pushes on the + * same channel/subchannel in hardware. The X DDX's read-image loop pushes + * the full setup once (SET_SRC_BLOCK_SIZE / OFFSET_IN / PITCH / + * LINE_LENGTH, a 112-byte push) and then per-strip DELTA pushes that only + * carry SRC_ORIGIN, SET_SEMAPHORE_A/PAYLOAD, OFFSET_OUT, (LINE_COUNT) and + * LAUNCH_DMA (a 52-byte push; raw witness: the delta launch decodes + * line=0/in=0/have_out=1/have_sem=1 on h=0x1000013). Decoding each push + * in isolation left line_length 0, so the transfer-shaped delta launch + * built NO op: every other 9-row strip never executed and the readback + * kept half the rows black (the visible even/odd banding). Remember the + * last full setup per channel and fill a delta launch's missing fields + * from it. + */ +struct svc_ce_saved { + u64 h_object; + u64 offset_in; + u32 pitch_in, pitch_out, line_length, line_count; + u32 src_phys_mode, dst_phys_mode; + u32 have_in, have_pitch_in, have_pitch_out; + u32 have_src_mode, have_dst_mode; +}; +#define SVC_CE_SAVED_N 8 +static struct svc_ce_saved svc_ce_saved[SVC_CE_SAVED_N]; +static u32 svc_ce_saved_next; +static u32 svc_cedelta_log; + +static struct svc_ce_saved *svc_ce_saved_slot(u64 h) +{ + u32 i; + + for (i = 0; i < SVC_CE_SAVED_N; i++) + if (svc_ce_saved[i].h_object == h) + return &svc_ce_saved[i]; + return NULL; +} + +static void svc_ce_saved_store(const struct svc_push_state *st, u64 h) +{ + struct svc_ce_saved *s = svc_ce_saved_slot(h); + + if (!s) { + s = &svc_ce_saved[svc_ce_saved_next++ % SVC_CE_SAVED_N]; + memset(s, 0, sizeof(*s)); + s->h_object = h; + } + s->offset_in = st->offset_in; + s->pitch_in = st->pitch_in; + s->pitch_out = st->pitch_out; + s->line_length = st->line_length; + s->line_count = st->line_count; + s->src_phys_mode = st->src_phys_mode; + s->dst_phys_mode = st->dst_phys_mode; + s->have_in = st->have_in; + s->have_pitch_in = st->have_pitch_in; + s->have_pitch_out = st->have_pitch_out; + s->have_src_mode = st->have_src_mode; + s->have_dst_mode = st->have_dst_mode; +} + /* Host CUDA bridge (gsp_cuda_bridge.c): launch transport + service hook * (prototypes in gsp_cuda_bridge.h; helpers exported to it below). */ @@ -3821,6 +6520,22 @@ static int svc_op_translate(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, svc.op_tr_src = 1; return 0; } + /* + * SESSION 153-CONTINUED-44 (r1027): the CONT-43 walk-first for + * below-floor CE operands is REVERTED. With it live, the moment the + * client's channels actually ran (submissions 33646 via the slot + * hold + zero-put-skip), the guest kernel oopsed repeatedly + * (`clear_page_erms` page fault from `rm_ioctl [nvidia]`, then + * `_mapInsertBase` general protection, Xorg exited) - i.e. the + * walk-first resolutions for below-floor COPY operands clobbered + * guest kernel memory, the r365/r369 corruption class the fill + * guard exists for. The DDX screen-repaint poison that CONT-43 + * fixed is a FILL class and stays fixed by svc_fill_floor_walk in + * svc_fill_1d; the copy path returns to the historical order + * (pins -> scanned map -> walk -> discovered -> targeted scan), + * which is the order every pre-r1025 boot (including the working + * r1024) ran with. + */ if (!svc_map_lookup(va, &pa)) { *addr = pa; *aperture = CE_APERTURE_SYS; @@ -3886,7 +6601,18 @@ static int svc_op_translate(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, * withholding the release corrupted the next session (r59: the late * memset overwrote refilled data after the client had passed the * release), so the two must move together. + * + * r383 (streams2 big-copy class): name the missed VA through the + * same stalled-VA demand channel the GR fill uses. The userspace pin + * stream watches the "stalled VA" row and pushes the VA to the + * in-guest scanner, so the pin lands on the next scanner pass + * instead of waiting out a full scan cadence. The hold itself is + * unchanged; this only shortens the window in which the entry sits + * op-pending (r377: a held H2D copy executed ~1.5 s late and read a + * staging page the client had since refilled with the next copy's + * bytes - the neighbor-data class). */ + svc_log_stalled_va(va & ~(u64)(SEMA_PAGE_SIZE - 1)); svc.op_tr_pending = 1; return -ENOENT; } @@ -3911,19 +6637,312 @@ static bool svc_dst_watched(u64 dst) } /* + * COPYAUDIT shared budget (see the svc_copy comment for the contract): + * every copy at/above SVC_COPY_AUDIT_MIN is logged once with want/done and + * gets a first/last-16-bytes destination readback vs the written bytes + * (COPY-MISMATCH). One budget covers every copy path in this file so a + * whole cycle stays visible. + */ +#define SVC_COPY_AUDIT_MIN (32u << 10) /* >32 KB copies audited */ +#define SVC_COPY_AUDIT_MAX 2048u /* per-boot line budget */ +#define SVC_CEAUDIT_MAX 4096u /* per-boot CEAUDIT budget */ +static u64 svc_copy_audit_n; +static u64 svc_ceaudit_n; + +/* COPYAUDIT for a finished fill: log want/done and check the first 16 + * destination bytes against the repeated pattern. The readback uses the + * accessor matching the aperture the fill wrote through. */ +static void svc_fill_audit(struct nv_vgpu_device *vgpu, u64 va0, u64 addr, + u32 aperture, size_t total, const u8 *pat, u32 elem) +{ + u8 exp[16], got[16] = {0}; + size_t i; + bool ok = false; + + if (total < SVC_COPY_AUDIT_MIN || svc_copy_audit_n >= SVC_COPY_AUDIT_MAX) + return; + /* + * The fill writes the 8-byte pattern group repeatedly (pat is the + * element zero-padded to 8 bytes), so the expectation mirrors that: + * byte i of the destination equals pat[i % 8]. + */ + for (i = 0; i < sizeof(exp); i++) + exp[i] = pat[i % 8]; + if (aperture == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + + if (base && addr + sizeof(got) <= svc_fb_shadow_size(vgpu)) { + memcpy(got, base + addr, sizeof(got)); + ok = true; + } + } else if (!svc_read(vgpu, addr, got, sizeof(got))) { + ok = true; + } + svc_copy_audit_n++; + pr_info("vgsp: COPYAUDIT path=sema-fill va=0x%llx len=0x%llx done=0x%llx\n", + va0, (u64)total, (u64)total); + if (ok && memcmp(exp, got, sizeof(exp))) + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x0 path=sema-fill want=%*phN got=%*phN\n", + va0, 16, exp, 16, got); +} + +/* * Write `count` elements of `value` (element size `elem`) starting at * `addr`, in vidmem or guest RAM depending on the aperture DST_TYPE * selected. Returns 0 on success. */ +static void svc_ledger_store(u64 va, u64 pa, u32 ap); /* R2 ledger, defined below */ +static int svc_ledger_lookup(u64 va, u64 *pa, u32 *ap); /* R2 ledger, defined below */ +static void svc_ledger_clear(void); /* STALLFREE: cap-and-degrade reset */ + +/* + * r384 (the guest boot-corruption lottery, root-caused 2026-10-20): + * page-table pages of the guest kernel's direct map were found ZEROED + * (two adjacent 4 KiB PTE pages at gpa 0x12d3c2000/0x3000 => two 2 MiB + * holes in the guest physmap, kernel objects inside them unreachable, + * multi-oops panic). The only emulation writers of page-sized ZERO runs + * are the virtual FILL executor (svc_fill / CE memsets), whose large + * >= SVC_VAS_FLOOR destinations resolve through the walk/scan when no + * pin and no ledger answer exists - and that resolver is demonstrably + * non-deterministic (four executions of the same 8 KiB engine-window + * clear resolved to four different guest pages, all refused by the + * below-floor guard). + * + * Two nets, mirroring the m169/r378 copy guards: + * 1. svc_fill_guard_hold(): a LARGE (>= 64 KiB) virtual fill whose only + * authority is the walk/scan (no pin, no ledger) is held through a + * short bounded window and its VA pushed to the pin-demand channel + * (the userspace pin stream injects the guest-pagemap truth), so the + * retried decode resolves through a real pin. Fail-open after the + * bound with the historical behaviour, like the copy guard. + * 2. svc_fill_target_is_pt(): a ZERO fill about to land in guest RAM on + * a page whose first 64 qwords are all PTE-shaped (present, NX, + * physical address in range) is REFUSED loudly. A page table page is + * never a legitimate zero-fill destination; this directly blocks the + * observed corruption shape even when some other resolver approved + * the target. Pin-backed resolutions are exempt (the guest's own + * pagemap is truth - if it says the VA maps the page, obey it). + */ +unsigned gsp_sema_fill_guard_ms = 800; +bool gsp_sema_fill_guard = true; +#define SVC_FILL_GUARD_SLOTS 16 +static struct { + u64 va_page; + unsigned long first; +} svc_fill_guard[SVC_FILL_GUARD_SLOTS]; +static u32 svc_fill_guard_next; +static u32 svc_fill_held; +static u32 svc_fill_pt_refused; +static u32 svc_fill_exec_log; /* r384 FILL-EXEC witness budget */ + +/* true = the guard may hold this fill; false = fail-open, run it. */ +static bool svc_fill_guard_hold(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i, slot = SVC_FILL_GUARD_SLOTS; + + for (i = 0; i < SVC_FILL_GUARD_SLOTS; i++) { + if (svc_fill_guard[i].va_page == page) { + if (time_before(now, svc_fill_guard[i].first + + msecs_to_jiffies( + gsp_sema_fill_guard_ms))) + return true; + if (svc_fill_guard[i].first) { + svc_fill_guard[i].first = 0; + pr_info("sema-svc: fill guard released without a pin va=0x%llx (using the walk translation)\n", + page); + } + return false; + } + if (!svc_fill_guard[i].va_page) + slot = i; + } + if (slot == SVC_FILL_GUARD_SLOTS) + slot = svc_fill_guard_next++ % SVC_FILL_GUARD_SLOTS; + svc_fill_guard[slot].va_page = page; + svc_fill_guard[slot].first = now; + return true; +} + +/* + * A guest-kernel page table page: the first SVC_PT_SCAN qwords are all + * present PTEs (bit 0), non-executable (bit 63 - kernel data mappings), + * with a physical address inside the 34-bit guest range. Random payload + * qwords pass all three conditions with probability far below the 75% + * threshold at SVC_PT_SCAN=64; a genuine table passes at 100%. + */ +#define SVC_PT_SCAN 64u +#define SVC_PT_MIN 48u +static bool svc_fill_target_is_pt(struct nv_vgpu_device *vgpu, u64 addr) +{ + u64 ents[SVC_PT_SCAN]; + u32 i, hits = 0; + + if (addr & (SEMA_PAGE_SIZE - 1)) + return false; /* only a page-aligned table shape */ + if (svc_read(vgpu, addr, ents, sizeof(ents))) + return false; + for (i = 0; i < SVC_PT_SCAN; i++) { + u64 e = ents[i]; + + if (!(e & 1ull)) + continue; + if (!(e & (1ull << 63))) + continue; + if ((e & 0x000ffffffffff000ull) == 0) + continue; + if ((e & 0x000ffffffffff000ull) >= (1ull << 34)) + continue; + hits++; + } + return hits >= SVC_PT_MIN; +} + +/* + * r391 per-page destination resolution for a VIRTUAL fill (the r390 + * guest-panic class). A device VA's pages are independent guest allocations + * - the pin stream shows split runs (va=0x7c36c7400000 -> 29 pages at + * 0x131b63000, the next run at 0x131a80000) - so resolving only the fill's + * first page and writing `total` bytes contiguously past it writes over + * whatever memory follows the first run. On r390 a 383620-byte zero fill + * did exactly that: the contiguous write covered the guest kernel's + * page-table pages (0x131ba1000/0x131ba2000/0x131bab000), the direct map + * grew holes, and the guest died with "Fatal exception in interrupt" + * (unmap_page_range / cpuacct / psi faults). svc_copy has always translated + * virtual operands per 4 KB page; the fill must do the same. + * + * Authority order mirrors the top-level resolution (and svc_copy's per-page + * dst): a fresh pin (guest pagemap truth) wins and repairs the landing + * table, then the landing table (the page a previous write of this VA + * landed on), then the resolver chain. A page with no resolution returns + * -ENOENT: the caller fails the fill loudly instead of writing through a + * contiguous guess. + */ +static int svc_fill_dst_page_pa(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out, u32 *prov_out) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0, lpa = 0; + u32 aper = 0, lap = 0; + + if (!svc_pin_lookup_any(page, &pa, &aper)) { + *prov_out = 1; + if (!svc_ledger_lookup(page, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != page) || + (u32)lap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill-page va=0x%llx ledger=0x%llx/%u pin=0x%llx/%u\n", + page, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, pa, aper); + svc_ledger_store(page, pa, aper); + } + *pa_out = pa; + *ap_out = aper; + return 0; + } + if (!svc_ledger_lookup(page, &pa, &aper)) { + *prov_out = 2; + *pa_out = pa; + *ap_out = aper; + return 0; + } + if (!svc_op_translate(vgpu, page, &pa, &aper)) { + *prov_out = 3; + *pa_out = pa; + *ap_out = aper; + return 0; + } + return -ENOENT; +} + +static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, + u32 elem, u32 count, u32 phys, u32 aperture, + u64 *done_out); +static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, + size_t total, u32 dst_phys, u32 dst_aperture, + u32 src_phys, u32 src_aperture, u64 *done_out, + u32 owner_ch, u32 owner_idx, u64 owner_hash); + +/* + * r402 row-wise wrapper: a 2D (pitched) fill runs the 1D fill once per + * row with the destination advancing by dpitch. Each row keeps the whole + * per-page resolution, ledger-store and guard machinery of the 1D path + * (no new resolution semantics), and a failed row's retry simply re-runs + * the already-written rows with the same pattern - idempotent for fills. + */ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, - u32 count, u32 phys, u32 aperture) + u32 count, u32 phys, u32 aperture, u64 *done_out, + u32 rows, u32 dpitch) +{ + if (rows > 1 && dpitch) { + u64 done = 0; + int rc = 0; + u32 r; + + for (r = 0; r < rows; r++) { + u64 row_done = 0; + + rc = svc_fill_1d(vgpu, addr + (u64)r * dpitch, + value, elem, count, phys, aperture, + &row_done); + done += row_done; + if (rc) + break; + } + if (done_out) + *done_out = done; + return rc; + } + return svc_fill_1d(vgpu, addr, value, elem, count, phys, aperture, + done_out); +} + +/* + * SESSION 153-CONT-43: resolve a below-floor fill destination through the + * guest's published page directories before the pin/ledger/map chain. + * + * Below SVC_VAS_FLOOR the driver maps its ABI/MMIO window (the X server's + * screen VA 0x120400000 family, the client pushbuffer region, ...). The + * content-scanned map and the landing ledger are heuristics built for the + * above-floor client VAs; for these low VA numbers a page-table page + * discovered by the scanner can hold a false identity pair (va == pa), and + * the LOWFILL guard then refuses every fill that the pair shadows - the + * observed X server repaint death (`LOWFILL-REFUSED va=0x120458c00 + * pa=0x120458c00`, fill done=52800 rc=-2, replayed ~15 Hz forever). The + * guest's page tables are the truth for this region: resolve through them + * first. Returns true when a translation was applied. + */ +static bool svc_fill_floor_walk(struct nv_vgpu_device *vgpu, u64 va0, + u64 *addr, u32 *aperture) +{ + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + + if (va0 >= SVC_VAS_FLOOR) + return false; + if (svc_walk_vaspaces(vgpu, va0 & ~0xfffULL, &wpa, &wap)) + return false; + *addr = wpa + (va0 & 0xfff); + *aperture = (wap == SVC_APERTURE_FB) ? CE_APERTURE_VID + : CE_APERTURE_SYS; + return true; +} + +static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, + u32 count, u32 phys, u32 aperture, u64 *done_out) { u8 pat[8]; u8 bounce[256]; size_t total = (size_t)elem * count; size_t o; bool shadow = false; + u64 va0 = addr; /* original destination VA (fills, C2) */ + if (done_out) + *done_out = 0; if (elem != 1 && elem != 2 && elem != 4 && elem != 8) return -EINVAL; if (!count || total > (256ULL << 20)) @@ -3934,21 +6953,199 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, if (!phys) { /* Virtual destination: the VA shadow first (its pages stay - * authoritative), then the pinned map and the VA->PA map. */ + * authoritative), then a live pin (the guest's newest + * statement - r372: a re-pinned VA must not be written + * through a stale landing), then the unified landing table + * (the page a previous write of this VA landed on - always + * the authoritative location of the content), then the + * pinned map and the VA->PA map. */ u64 pa; u32 aper; + u32 prov = 0; /* r384: 1 pin, 2 ledger, 3 walk/scan */ - if (svc_vas_has(addr)) { + /* + * r584: a VID-aperture fill whose destination is a vidmem + * WINDOW VA (the guest's 0x2xxxxxxxx family) is offset- + * mapped by construction: VA - SVC_VIDMEM_VA_BASE is the + * FB offset into the emulated BAR1 shadow (the svc_lookup + * window convention; see also the 0x200080000 rationale at + * SVC_VIDMEM_VA_BASE). The generic chain below (pin -> + * ledger -> translate) answered SYS for the r583 boot's CE + * channel-setup fill to 0x204820000 (a stale landing), the + * r369 below-floor guard refused it, the fill never ran, + * and its DSTPEND note stalled GR launches to the fail- + * closed cap. Window VAs resolve structurally - no guest + * page table can express an "engineering window" offset. + */ + if (aperture == CE_APERTURE_VID && + va0 >= SVC_VIDMEM_VA_BASE && + va0 - SVC_VIDMEM_VA_BASE + total <= + svc_fb_shadow_size(vgpu)) { + addr = va0 - SVC_VIDMEM_VA_BASE; + /* aperture stays CE_APERTURE_VID: the whole-range + * contiguity block below writes it into the shadow */ + prov = 4; /* structural window convention */ + if (svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill-window va=0x%llx -> fb offset 0x%llx (vidmem window convention) elem=%u count=%u value=0x%llx\n", + va0, addr, elem, count, value); + } + } else if (svc_fill_floor_walk(vgpu, va0, &addr, &aperture)) { + /* + * SESSION 153-CONT-43: below-floor VA (the driver's + * ABI/MMIO mapping window). The guest's published + * page directory is the truth there; a pin or a stale + * landing-table entry can hold a false identity pair + * (va==pa), observed live as the X server's screen + * repaint dying ~44 rows in (`LOWFILL-REFUSED + * va=0x120458c00 pa=0x120458c00` repeating at + * ~15 Hz, the screen frozen on a stale fill + * artifact). The walk resolves the same VA through + * the guest's 2 MB big page to vidmem + * (pa=0x200000+off), which is what the driver meant. + */ + prov = 6; + if (svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill-walk va=0x%llx -> pa=0x%llx ap=%u elem=%u count=%u value=0x%llx\n", + va0, addr, aperture, elem, count, + value); + } + } else if (svc_vas_has(addr)) { shadow = true; + } else if (!svc_pin_lookup_any(addr, &pa, &aper)) { + u64 lpa = 0; + u32 lap = 0; + + prov = 1; + /* + * r372 pin-wins: repair the landing table when it + * still names the pre-re-pin page, so every later + * reader follows the pin too. + */ + if (!svc_ledger_lookup(addr, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill-dst va=0x%llx ledger=0x%llx/%u pin=0x%llx/%u\n", + addr, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, pa, aper); + svc_ledger_store(addr, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper); + } + addr = pa; + aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (!svc_ledger_lookup(addr, &pa, &aper)) { + prov = 2; + addr = pa; + aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; } else if (svc_op_translate(vgpu, addr, &pa, &aper)) { if (!svc_vas_can_shadow(addr)) return -ENOENT; shadow = true; svc.vas_shadowed++; } else { + prov = 3; addr = pa; aperture = aper; } + /* + * r365b LOWFILL-WIT: a below-floor virtual fill (the guest's + * vidmem-window page-table writes) is a corruption risk: its + * resolution must be proven before the op is allowed to run + * (r365: 16-byte and 32 KB zero fills landed in guest RAM and + * killed the guest kernel). Log every resolution bounded. + */ + if (va0 < SVC_VAS_FLOOR && svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill: va=0x%llx shadow=%d resolved=0x%llx ap=%u elem=%u count=%u value=0x%llx\n", + va0, shadow, addr, aperture, elem, count, + value); + } + /* + * r369: a below-floor fill may ONLY land in vidmem. The + * 0x204020000 engine-window clear resolved to a DIFFERENT + * guest-RAM page on every execution (9 runs, 9 pages) and + * the guest died with its task_struct corrupted (gmain NULL + * deref in __fdget) - each wrong resolution zeroed 8 KB of + * guest kernel memory. Vidmem resolutions (the page-table + * region, ap=VID) are the emulation's own shadow and stay + * allowed; a SYS resolution for a window VA is refused and + * logged loudly. + */ + if (va0 < SVC_VAS_FLOOR && !shadow && + aperture != CE_APERTURE_VID) { + pr_err_ratelimited("vgsp-sema: LOWFILL-REFUSED va=0x%llx pa=0x%llx ap=%u elem=%u count=%u (below-floor fill resolved outside vidmem)\n", + va0, addr, aperture, elem, count); + return -ENOENT; + } + /* + * r384 net 1 (pin-demand hold for unproven large fills): a + * >= 64 KiB virtual fill whose only authority is the + * walk/scan resolver (no pin, no landing-table entry) AND + * which would land in GUEST RAM (SYS aperture) is the class + * that zeroed the guest's page-table pages (see the guard + * commentary above svc_fill_guard_hold). Demand the VA + * through the pin channel and hold the op through a short + * bounded window so the retried decode resolves through the + * guest's own pagemap; fail open with the historical + * behaviour after the bound. Vidmem-shadow resolutions are + * the emulation's own memory and stay immediate. + */ + if (gsp_sema_fill_guard && va0 >= SVC_VAS_FLOOR && !shadow && + total >= (64u << 10) && prov >= 3 && + aperture == CE_APERTURE_SYS) { + svc_log_stalled_va(va0); + if (svc_fill_guard_hold(va0)) { + svc_fill_held++; + if (svc_fill_held <= 32) + pr_info("sema-svc: FILL-HOLD va=0x%llx resolved=0x%llx ap=%u total=%zu value=0x%llx (unproven large fill; pin demanded)\n", + va0, addr, aperture, total, + value); + return -ENOENT; + } + } + /* + * r384 net 2 (page-table-shape refusal): a ZERO fill about + * to land in guest RAM on a page whose first 64 qwords are + * all present+NX+in-range PTEs is refused loudly - a page + * table page is never a legitimate zero-fill destination. + * This blocks the observed corruption shape (two adjacent + * PTE pages zeroed => two 2 MiB holes in the guest physmap) + * even when some other resolver approved the target, and + * irrespective of how wrong the resolution is. Pin-backed + * resolutions are exempt: the guest's own pagemap is truth + * there. + */ + if (gsp_sema_fill_guard && va0 >= SVC_VAS_FLOOR && !shadow && + value == 0 && total >= SEMA_PAGE_SIZE && + aperture == CE_APERTURE_SYS && prov != 1 && + svc_fill_target_is_pt(vgpu, addr)) { + svc_fill_pt_refused++; + if (svc_fill_pt_refused <= 32) + pr_err("vgsp-sema: FILL-PT-REFUSED va=0x%llx pa=0x%llx prov=%u total=%zu (zero fill target is a page table)\n", + va0, addr, prov, total); + svc_log_stalled_va(va0); + return -ENOENT; + } + /* + * r384 witness: where every >= 4 KiB virtual fill actually + * lands (va, pa, aperture, resolution provenance, pattern) - + * the decisive line for the boot-corruption forensics (the + * large zero fills were invisible until now). Bounded. + */ + if (va0 >= SVC_VAS_FLOOR && !shadow && + total >= SEMA_PAGE_SIZE && svc_fill_exec_log < 64) { + svc_fill_exec_log++; + pr_info("sema-svc: FILL-EXEC va=0x%llx pa=0x%llx ap=%u prov=%u total=%zu value=0x%llx\n", + va0, addr, aperture, prov, total, value); + } } if (gsp_sema_dbg && !shadow && svc_dst_watched(addr) && @@ -3963,10 +7160,40 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, svc.op_tr_src, elem, pb, svc.cur_ch); } - if (shadow) - return svc_vas_fill(addr, pat, elem, total); + if (shadow) { + int rc = svc_vas_fill(addr, pat, elem, total); - if (aperture == CE_APERTURE_VID) { + if (!rc && done_out) + *done_out = (u64)total; + return rc; + } + + /* + * C2 (data class) + m180 unification: a large virtual fill is a + * write/read pair with the memset's readers like any H2D copy - + * record its landing pages so the engine fill and the D2H read of + * this VA use exactly the page the fill wrote. Only the VID + * (shadow) case is contiguous; a SYS landing is recorded per page + * in the loop below (r391: the contiguous `addr + po` store also + * poisoned the table with pages the fill never wrote). + */ + if (!phys && !shadow && va0 >= SVC_VAS_FLOOR && + total >= (64u << 10) && aperture == CE_APERTURE_VID) { + u64 po; + + for (po = 0; po < total; po += SEMA_PAGE_SIZE) + svc_ledger_store(va0 + po, addr + po, + SVC_APERTURE_FB); + } + + /* + * r391: a virtual device destination is translated per 4 KB page + * (see svc_fill_dst_page_pa) - never written through the first + * page's PA contiguously. The whole-range shadow write below only + * applies to a resolution that is contiguous by construction. + */ + if (aperture == CE_APERTURE_VID && + !(va0 >= SVC_VAS_FLOOR && !phys && !shadow)) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); @@ -3976,6 +7203,9 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, memcpy(base + addr + o, pat, 8); if (o < total) memcpy(base + addr + o, pat, total - o); + if (done_out) + *done_out = (u64)total; + svc_fill_audit(vgpu, va0, addr, aperture, total, pat, elem); return 0; } @@ -3983,17 +7213,107 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, for (o = 0; o < total; ) { size_t chunk = total - o; size_t i; + u64 wpa = addr; + u32 wap = aperture; + bool per_page = !phys && !shadow && va0 >= SVC_VAS_FLOOR; if (chunk > sizeof(bounce)) chunk = sizeof(bounce); + /* + * r391: resolve THIS page's own landing (see + * svc_fill_dst_page_pa) and clamp the chunk to the page + * boundary - a virtual VA's pages are independent guest + * allocations and must never be written through the first + * page's PA contiguously. + */ + if (per_page) { + u32 off = (u32)((va0 + o) & (SEMA_PAGE_SIZE - 1)); + u64 ppa = 0; + u32 pap = 0, prov = 0; + int prc; + + chunk = min_t(size_t, chunk, SEMA_PAGE_SIZE - off); + prc = svc_fill_dst_page_pa(vgpu, va0 + o, &ppa, + &pap, &prov); + if (prc) { + if (done_out) + *done_out = (u64)o; + /* + * r391: demand the exact page through the pin + * channel (the userspace stream pushes this VA + * to the in-guest scanner) and hold the fill + * for a bounded retry - never write through a + * contiguous guess past a missing page. + */ + svc_log_stalled_va((va0 + o) & + ~(u64)(SEMA_PAGE_SIZE - 1)); + pr_err_ratelimited("vgsp: FILL-TRUNC va=0x%llx want=0x%llx done=0x%llx path=sema-fill (page 0x%llx unresolved)\n", + va0, (u64)total, (u64)o, + (va0 + o) & + ~(u64)(SEMA_PAGE_SIZE - 1)); + return -ENOENT; + } + wap = (pap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wpa = (ppa & ~(u64)(SEMA_PAGE_SIZE - 1)) + off; + /* + * r391 net: the r384 page-table-shape refusal + * applied to EVERY page of a virtual zero fill. + * The r390 fill's first page was the client's + * buffer, the later pages were the guest's PTE + * pages - the first-page-only check could not see + * them. A page-table-shaped guest-RAM page is never + * a legitimate zero-fill target: skip it loudly + * instead of zeroing it. Pin-backed resolutions + * are exempt (the guest's pagemap is truth there). + */ + if (wap == CE_APERTURE_SYS && value == 0 && prov != 1 && + svc_fill_target_is_pt(vgpu, + wpa & ~(u64)(SEMA_PAGE_SIZE - 1))) { + svc_fill_pt_refused++; + if (svc_fill_pt_refused <= 32) + pr_err("vgsp-sema: FILL-PT-REFUSED va=0x%llx pa=0x%llx prov=%u total=%zu (zero fill page is a page table; page skipped)\n", + va0 + o, wpa, prov, total); + o += chunk; + continue; + } + } for (i = 0; i + 8 <= chunk; i += 8) memcpy(bounce + i, pat, 8); if (i < chunk) memcpy(bounce + i, pat, chunk - i); - if (svc_write(vgpu, addr + o, bounce, chunk)) + if (wap == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = svc_fb_shadow_size(vgpu); + + if (!base || wpa + chunk > size) { + if (done_out) + *done_out = (u64)o; + return -EINVAL; + } + memcpy(base + wpa, bounce, chunk); + } else if (svc_write(vgpu, wpa, bounce, chunk)) { + if (done_out) + *done_out = (u64)o; + if (total >= SVC_COPY_AUDIT_MIN) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=sema-fill rc=%d\n", + va0, (u64)total, (u64)o, -EIO); return -EIO; + } + /* + * r391: record this page's landing (the readers follow the + * table) - SYS and VID alike, exactly like the copy path. + */ + if (per_page) + svc_ledger_store(va0 + o, + wpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (wap == CE_APERTURE_VID) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS); o += chunk; } + if (done_out) + *done_out = (u64)total; + svc_fill_audit(vgpu, va0, addr, aperture, total, pat, elem); return 0; } @@ -4044,10 +7364,26 @@ static struct svc_fwd_cand svc_fwd_cand; */ #define SVC_GR_QMD_WORDS 64 #define SVC_GR_INLINE_MAX 8 -#define SVC_GR_INLINE_BYTES 64 +/* + * SESSION 153-CONTINUED-88: 256 -> 2048 (static_asserts below tie it to + * GSP_BRIDGE_INLINE_BYTES / GSP_PHYS_GR_INLINE_BYTES). The GL window's real + * program upload payloads are 1536 B / 1408 B; the 256 B cap truncated them + * to their first 256 bytes, so the engine never saw the full image. + */ +#define SVC_GR_INLINE_BYTES 2048 #define SVC_GR_POINTER_MAX 8 #define SVC_GR_PAGES_MAX 16 /* + * relfix PART 3 (fill candidate superset): maximum distinct candidate pages + * per collection pass. Raised from 16 because the full-page scans added + * below (aligned with the rewrite's own whole-page scan) can legitimately + * carry more pointer-shaped words than the argument region alone; the old + * cap silently dropped the extra words, which then resurfaced as RELOC-GAP + * at rewrite time. It remains a hard bound with the historical silent-drop + * semantics beyond it (n_cand is logged in the fill summary). + */ +#define SVC_GR_CAND_MAX 24u +/* * m99: guest data identity-map capacity. The forwarded kernel's global data * pages (discovered by scanning the param/const-0 page for device VAs) are * resolved on the service thread and relocated into the engine-visible @@ -4055,7 +7391,23 @@ static struct svc_fwd_cand svc_fwd_cand; * so the hand-off array is never the binding limit. */ #define SVC_GR_DATA_MAX 32768 -#define SVC_GR_PUSH_WORDS 256 +/* + * r929: 256 -> 512 dwords (2 KB) so a whole GL 3D section push (the X + * server's one-shot stream is 1524 B = 381 dwords) survives in raw[] - the + * 3D stream replay needs the window verbatim, not just its tail. Must equal + * GSP_PHYS_GR_RAW_WORDS (static_assert below). + * + * SESSION 153-CONTINUED-45: 512 -> 2048 dwords (8 KB). The DIRECT GL + * client (glcore, channel family 0xbeef) emits its per-frame render pushes + * at 5016-7144 B (1254-1786 dwords) with the draw state (program binds, + * vertex-array bases, constant uploads) spread across the whole push; the + * 512-dword tail kept only the release block, so the section capture could + * never satisfy `len == push_n*4` and the verbatim 3D replay never marked + * (CONT-44: zero src=stream captures on the client channel). The window + * slice still has to fit the backend's method block; only the raw[] + * retention is widened here. + */ +#define SVC_GR_PUSH_WORDS 2048 #define SVC_GR_QMD_WORD_NONE 0xffffffffu /* svc_gr_cand.rel_flags bits (decoded from qmd[24] and qmd[4]). */ #define SVC_GR_REL_REDUCTION 0x1u @@ -4082,6 +7434,13 @@ struct svc_gr_data { u64 pa; /* guest PA, page aligned (0 = unresolved) */ u64 reloc_va; /* m102: server-managed VA the page maps to */ u8 ap; /* SVC_APERTURE_* of pa */ + /* + * r632: the page is served from the VA shadow lane instead of a guest + * page (no resolver and no landing answered for it). The submit build + * hands the shadow page's host address to the bounce machinery - see + * the fill's shadow-serve branch and the r632 note there. + */ + u8 shadow; }; /* @@ -4098,9 +7457,21 @@ struct svc_gr_data_win { u64 reloc_base; /* relocated window base */ }; -struct svc_gr_cand { +/* CONT-87: max program binds carried from a captured window (mirrors + * SVC_GL_PROG_MAX; defined here because the candidate struct precedes it). */ +#define SVC_GR_CAND_PROG_BINDS 8 +struct svc_gr_cand_rec { bool valid; bool offered; /* consumed by a request or a fallback */ + /* + * STALLFREE: exactly-once latch for the candidate's QMD completion + * release when the launch is consumed without ever being forwarded + * (a data-stall/busy/nocap hold fell through). Zeroed with the rest + * of the candidate at capture. + */ + bool rel_done; + unsigned long captured_jiffies; /* STALLFREE: capture stamp for the + * GR-STALL waited_ms accounting */ u32 h_object; u64 push_va; u32 push_len; @@ -4108,6 +7479,31 @@ struct svc_gr_cand { u32 push_n; /* raw tail words kept in raw[] */ u32 push_qmd_hdr; /* raw[] offset of the 0x318 run header */ u32 push_qmd_end; /* raw[] offset just past that run */ + /* + * r929 (GL 3D path): this capture is a real C597 method stream (the + * section carried its own operands, op_src=stream) whose whole push + * fits in raw[]. The backend replays raw[stream_off..stream_off+ + * stream_n) verbatim as a 3D method window (SET_OBJECT TURING_A + + * the engine's own uploads/binds) instead of running the synthesized + * compute QMD. The window is sliced from the section's C597 bind. + */ + bool stream_3d; + u32 stream_off; /* dword offset of the window inside raw[] */ + u32 stream_n; /* dwords of the replayable window */ + /* + * SESSION 153-CONTINUED-87: every SET_PIPELINE_PROGRAM_ADDRESS pair + * the captured window itself binds (VA + pipeline slot), copied from + * svc_gl at capture. The launch builder re-emits them AHEAD of the + * window: the captured per-frame window carries draws (BEGIN 0x1618) + * BEFORE its own program binds (measured: draw at window word 93, + * j=1 pair at word 184), on the real hardware the bind already + * happened in an earlier push - on the replay channel's fresh + * context the early draws would validate an UNBOUND stage and latch + * the Shader Program Header fault (the r1277+ 9/11 wall). + */ + u32 n_prog_binds; + u64 prog_bind_va[SVC_GR_CAND_PROG_BINDS]; + u32 prog_bind_slot[SVC_GR_CAND_PROG_BINDS]; u32 raw[SVC_GR_PUSH_WORDS]; u32 qmd[SVC_GR_QMD_WORDS]; u64 qmd_dev_va; @@ -4125,7 +7521,135 @@ struct svc_gr_cand { u32 pointer_word[SVC_GR_POINTER_MAX]; /* QMD word of the pair */ }; -static struct svc_gr_cand svc_gr_cand; + +/* + * fail4: the capture was a single slot; two concurrent launches (streams2's + * vadd pair on two channels) overwrote each other's candidate while the + * first was kept for a retry (the candidate-overwrite witness) and the + * engines ran crossed parameters (c1 == a2+b2). Keep up to four candidates + * side by side in a ring; the accessor macro addresses the current slot, + * so every existing use keeps working on the selected candidate, and + * capture/consume select the slot instead of clobbering it. + * + * SESSION 153-CONTINUED-45: 4 -> 8 slots. The DIRECT GL client's draw + * sections are 1470-dword windows split into up to four <=448-dword chunks + * (see svc_gl_stream_chunks); with four slots a single big push could + * evict its own later chunks (or the X channel's pending candidate) before + * the pump drained them. + */ +#define SVC_GR_CAND_RING 8 +static struct svc_gr_cand_rec svc_gr_cand_ring[SVC_GR_CAND_RING]; +static u32 svc_gr_cand_pos; +#define svc_gr_cand (svc_gr_cand_ring[svc_gr_cand_pos]) + +/* + * r613: is a GR forward owed for this device - a captured candidate that has + * not been offered (submitted or refused) yet? The wait gate uses this to + * distinguish "the acquire's release can never be installed by firmware + * alone" from "the release is pending behind OUR OWN launch deferral": the + * r612 streams2 capture gave up a D2H readback's acquire (0x20480ffe0 pay=4, + * cur=2) at waited_ms=10333 while the vadd launches that author that + * semaphore sat deferred by the CE watermark for the full 15 s cap - the + * readback then ran BEFORE the kernels and the client read zeros + * (bad=65534). While a forward is owed, the episode extends (bounded by + * wait-hold + defer cap + margin) instead of failing open. + */ +static bool svc_gr_forward_pending(void) +{ + u32 i; + + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + !svc_gr_cand_ring[i].offered) + return true; + return false; +} +/* + * r383: push-VA history of recent captures, beyond the (four-slot) + * candidate ring's lifetime. A replayed decode of a complete QMD asks + * "has this launch ever been captured?"; the ring alone can have rotated + * past it, and a wrong "no" would re-capture (and re-execute) an already + * forwarded launch. Bounded, service thread only. + */ +#define SVC_GR_CAP_HIST 8 +static u64 svc_gr_cap_hist[SVC_GR_CAP_HIST]; +static u32 svc_gr_cap_hist_pos; + +/* + * r401: robust session-scoped "already captured" memory for the r383 + * replay-capture path. The 4-slot candidate ring + 8-VA history evicted a + * real capture within ~12 later captures, so a true replay of an + * already-forwarded launch (identical push VA and content) was + * re-captured and re-executed: the r399 soak log shows the same nine + * launch pushes re-submitted every ~10-11 s for minutes (idempotent + * kernels survive it; accumulators would not). The open-addressed table + * keeps the (push_va, push_hash) identity of every capture of the + * session, so a true replay always classifies as "seen" (LAUNCH-SKIP, + * exactly once) while a genuinely new push - even at the same VA with + * the same entry words - is captured through the normal + * `trusted && !replay` path and never consults this table. Cleared by + * svc_session_flush (a new client's identical-payload pushes restart + * their release counters, so a boot-scoped table would wrongly suppress + * them). Service-thread only. + */ +#define SVC_GR_SEEN_SLOTS 2048 +#define SVC_GR_SEEN_PROBE 8 +static struct { + u64 push_va; + u64 hash; +} svc_gr_seen[SVC_GR_SEEN_SLOTS]; +static u64 svc_gr_seen_stale; /* probe-chain evictions (telemetry) */ + +static u32 svc_gr_seen_mix(u64 push_va, u64 hash) +{ + u64 k = push_va * 0x9E3779B97F4A7C15ULL; + + k ^= hash + 0x9E3779B97F4A7C15ULL + (k << 6) + (k >> 2); + return (u32)(k % SVC_GR_SEEN_SLOTS); +} + +static bool svc_gr_seen_lookup(u64 push_va, u64 hash) +{ + u32 i, base = svc_gr_seen_mix(push_va, hash); + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_SEEN_PROBE; i++) { + u32 k = (base + i) % SVC_GR_SEEN_SLOTS; + + if (!svc_gr_seen[k].push_va) + return false; + if (svc_gr_seen[k].push_va == push_va && + svc_gr_seen[k].hash == hash) + return true; + } + return false; +} + +static void svc_gr_seen_store(u64 push_va, u64 hash) +{ + u32 i, base = svc_gr_seen_mix(push_va, hash); + + if (!push_va) + return; + for (i = 0; i < SVC_GR_SEEN_PROBE; i++) { + u32 k = (base + i) % SVC_GR_SEEN_SLOTS; + + if (!svc_gr_seen[k].push_va || + (svc_gr_seen[k].push_va == push_va && + svc_gr_seen[k].hash == hash)) { + svc_gr_seen[k].push_va = push_va; + svc_gr_seen[k].hash = hash; + return; + } + } + /* Full probe chain: replace the tail (oldest-ish) entry. */ + svc_gr_seen_stale++; + svc_gr_seen[(base + SVC_GR_SEEN_PROBE - 1) % SVC_GR_SEEN_SLOTS].push_va = + push_va; + svc_gr_seen[(base + SVC_GR_SEEN_PROBE - 1) % SVC_GR_SEEN_SLOTS].hash = + hash; +} static_assert(SVC_GR_QMD_WORDS == GSP_PHYS_GR_QMD_WORDS, "the sema GR QMD word count must match the backend ABI"); @@ -4133,8 +7657,8 @@ static_assert(SVC_GR_PUSH_WORDS == GSP_PHYS_GR_RAW_WORDS, "the sema raw push cap must match the backend ABI"); static_assert(SVC_GR_INLINE_MAX <= GSP_PHYS_GR_MAX_INLINE, "the sema GR inline capture must fit the backend ABI"); -static_assert(sizeof(struct svc_gr_inline) == 80 && - offsetof(struct svc_gr_inline, data) == 12, +static_assert(offsetof(struct svc_gr_inline, data) == 12 && + sizeof(struct svc_gr_inline) >= 12 + SVC_GR_INLINE_BYTES, "the sema GR inline record must stay layout-compatible with the backend"); static_assert(sizeof(struct svc_gr_inline) == sizeof(struct gsp_bridge_inline) && offsetof(struct svc_gr_inline, dst_va) == @@ -4159,9 +7683,33 @@ static_assert(sizeof(struct svc_gr_page) == 24, "the sema GR page descriptor must stay {u64,u64,u8,u32} = 24 bytes"); static u64 svc_push_hash(const u8 *p, u32 len); +/* SESSION 149 R15: exactly-once guard for the hash-verified replay capture. */ +static bool svc_gr_cand_push_seen(u64 push_va, u64 hash); +/* SESSION 149 R16 (M3): trusted-walk VA read, defined below the decode. */ +static int svc_read_va_trusted(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len); +/* + * SESSION 149 R17 (M4): the section's memory-writing upload commands are + * applied through the shared inline-write path (defined below the section + * code) with its (push, hash, dst, bytes) once-only guard. + */ +static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, + const u8 *data, u32 bytes, u32 ch_h); +static bool svc_inline_once_seen(u64 push_va, u64 hash, u64 dst, u32 bytes); +static void svc_inline_once_store(u64 push_va, u64 hash, u64 dst, u32 bytes); + +/* + * SESSION 153-CONTINUED-88: newest captured inline-write payload landing + * exactly at `dst_va` with at least `min_len` bytes (defined next to the + * destination ring below). Used by svc_gl_flush to deliver the window's REAL + * program uploads (the driver's own SPH+SASS images) to the engine-visible + * page instead of the minimal authored blob. + */ +static bool svc_grce_dst_newest(u64 dst_va, u32 min_len, u32 *len, u8 *buf, + u32 buf_sz); /* QMD field decode: the exact shape the host CUDA bridge uses. */ -static void svc_gr_qmd_decode(struct svc_gr_cand *c) +static void svc_gr_qmd_decode(struct svc_gr_cand_rec *c) { const u32 *q = c->qmd; u32 i; @@ -4200,13 +7748,50 @@ static void svc_gr_cand_capture(struct svc_channel *ch, { u32 i, n; + /* + * C9 (GAP-2 A4): a capture over a still-valid candidate (never + * offered) destroys an unforwarded launch - the single-slot + * overwrite hazard. Bounded, unconditional (gsp_sema_dbg is defined + * later in the file and this site may not reference it). + */ + /* + * fail4: never overwrite a DIFFERENT launch's unconsumed candidate - + * move to a free ring slot instead (same-push re-captures still + * refresh in place). + */ + if (svc_gr_cand.valid && !svc_gr_cand.offered && + svc_gr_cand.push_va != push_va) { + u32 ri; + + for (ri = 1; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % SVC_GR_CAND_RING; + + if (!svc_gr_cand_ring[rp].valid || + svc_gr_cand_ring[rp].offered) { + svc_gr_cand_pos = rp; + break; + } + } + } + if (svc_gr_cand.valid && svc_cand_ovw_log < 64) { + svc_cand_ovw_log++; + pr_info("sema-gr: CAND-OVERWRITE old_h=0x%x old_push=0x%llx new_push=0x%llx\n", + svc_gr_cand.h_object, svc_gr_cand.push_va, push_va); + } memset(&svc_gr_cand, 0, sizeof(svc_gr_cand)); svc_gr_cand.valid = true; + svc_gr_cand.captured_jiffies = jiffies; svc_gr_cand.h_object = ch->h_object; svc_gr_cand.push_va = push_va; svc_gr_cand.push_len = push_len; + /* r383: remember the identity beyond this ring slot's lifetime. */ + svc_gr_cap_hist[svc_gr_cap_hist_pos % SVC_GR_CAP_HIST] = push_va; + svc_gr_cap_hist_pos++; if (push && push_len) svc_gr_cand.push_hash = svc_push_hash(push, push_len); + /* r401: the session-wide exactly-once memory (survives ring rotation; + * see svc_gr_seen_lookup). */ + svc_gr_seen_store(push_va, svc_gr_cand.push_hash); if (push && push_len >= 4) { u32 total = push_len / 4; u32 n = total < SVC_GR_PUSH_WORDS ? total : SVC_GR_PUSH_WORDS; @@ -4240,10 +7825,11 @@ static void svc_gr_cand_capture(struct svc_channel *ch, svc_gr_cand.param_size, svc_gr_cand.rel_va, svc_gr_cand.rel_payload, svc_gr_cand.rel_flags, svc_gr_cand.n_pointer, svc_gr_cand.n_inline); - if (push && push_len >= 16) { + if (gsp_sema_dbg && svc_gr_push_dump < 4 && push && push_len >= 16) { u32 n = push_len / 4; u32 i, j; + svc_gr_push_dump++; if (n > 256) n = 256; for (i = 0; i < n; i += 16) { @@ -4260,6 +7846,2541 @@ static void svc_gr_cand_capture(struct svc_channel *ch, } /* + * SESSION 149 R14 (sema_gl_xlate, default 0 = off): GL/3D direct method + * stream -> synthesized GR candidate recognizer. + * + * The GL channel's large per-frame pushes bind the TURING_A 3D class + * (0xC597) on the channel subchannel and carry no CUDA QMD, so the + * inline-QMD capture above never fires for them (the victim-section + * classification of the r882 stream, push VA 0x120010004 / 1524 B: + * SET_OBJECT C597, SET_TEX_HEADER_POOL_A -> 0x1214fa220/0x10, 20x + * LINE_LENGTH_IN(0x180)+LAUNCH_DMA(0x1b0) inline uploads to + * 0x1214f8f00+0x100k, SET_PIPELINE_PROGRAM_ADDRESS_A/B(0x2014/0x2018 + * + j*64) -> 0x121201b00, BIND_GROUP_CONSTANT_BUFFER(0)). This section + * collects the OBSERVED VA operands (nothing is invented) and synthesizes + * the 64-word QMD those fields would carry; the candidate then flows + * through the unchanged svc_gr_request / svc_fwd_gr_work / + * gsp_phys_backend_guest_gr_submit machinery. + * + * Strictly additive: gated by sema_gl_xlate and inert for every stream + * without the C597 class bind (0xC5B5 copy-class and 902D 2D binds never + * open a section). Class numbers from the public class headers + * (clc597.h: TURING_A 0xC597). + */ +static bool gsp_sema_gl_xlate; +module_param_named(sema_gl_xlate, gsp_sema_gl_xlate, bool, 0644); +MODULE_PARM_DESC(sema_gl_xlate, + "GL/3D (C597) direct stream -> synthesized GR candidate (default 0 = off)"); + +/* + * SESSION 153-CONTINUED-4: native publish-counter advance (pace-wall fix). + * + * The GLX client's swap-completion ladder waits for the u32 at +0x8 of the + * driver-shared page whose struct magic is 0x0e836365 at +0x4 (the page is + * CPU-mapped from the X driver's memfd; r149-R3 watchpoint forensics named + * it "the publish counter our emulation never advances"). In a real stack + * the GPU completion of the client's kick channel (h=0xbfef00xx family) + * advances it to 3; ours never did, so every wait fell through the client's + * multi-second give-up ladder (140 FPS -> 0.045 FPS). A guest daemon write + * proved the value: u32 3 -> "702 frames in 5.0 seconds = 140.398 FPS". + * + * Native replacement: when the pump RETIRES a push of a kick-family channel + * (svc_replay_note consume path), write 3 to the page. The page moves in + * guest RAM per boot, so it is discovered once by content: the full 8 GiB + * RAM scan shows the magic page-aligned 16-byte signature + * (magic 0x0e836365 at +0x4, counter <= 3 at +0x8) matches exactly ONE page + * (verified unique on the r934 image before implementating). Discovery walks + * guest RAM descending (the driver's allocations live high), is rate-limited + * and attempt-capped, and every write is read back; a mismatch drops the + * discovered page so a stale/foreign target can never persist. Guarded by + * sema_pub_adv (default on; the family guard makes it inert for CUDA lanes) + * and fully logged. + */ +static bool gsp_sema_pub_adv = true; +module_param_named(sema_pub_adv, gsp_sema_pub_adv, bool, 0644); +MODULE_PARM_DESC(sema_pub_adv, + "Advance the GLX publish counter (u32 3) when a kick-channel push retires (default 1)"); +static u64 gsp_sema_pub_pa; +module_param_named(sema_pub_pa, gsp_sema_pub_pa, ullong, 0644); +MODULE_PARM_DESC(sema_pub_pa, + "Force the publish page PA for sema_pub_adv (0 = content-discover per boot)"); + +/* + * SESSION 153-CONTINUED-36: the GL shared-state keeper (automates the + * slot-word handshake that gsp-phys-shared-bit0 previously did by hand). + * + * Every direct GLX client receives a shared page from the X driver whose + * word at +8 is a state field; glcore's device init reads it and SKIPS its + * own class-0xc46f channel creation while bit 0 is set. The emulated X + * driver publishes 3 (bits 0+1), so direct clients never allocate a + * channel and never render (witness: no c46f alloc in the RM log, glClear + * a no-op, err@ReadPixels=0x502). Clearing bit 0 while the client inits + * lets the channel alloc proceed (r999-r1002 by hand, r1012 by the + * debugfs write). The page moves per boot and is named here by guest PA; + * the keeper clears bit 0 on every service pass (up to ~1 kHz while the + * pump is active) so the client cannot re-publish 3 fast enough to block + * a later client's init. The PA is discovered by content (cookie word + * 0xc0b43fec at +4 / word5 0xcaf00006) with the host helper slot_hold.py + * or by hand, then written to this parameter; witness is bounded. + */ +static u64 gsp_sema_shared_hold_pa; +module_param_named(sema_shared_hold_pa, gsp_sema_shared_hold_pa, ullong, 0644); +MODULE_PARM_DESC(sema_shared_hold_pa, + "Guest PA of the GL shared-state page; clear bit 0 of its +8 word every pass (0 = off)"); +static u32 svc_shared_hold_log; + +static void svc_shared_hold(struct nv_vgpu_device *vgpu) +{ + u32 w[2]; + + if (!gsp_sema_shared_hold_pa) + return; + if (svc_read(vgpu, gsp_sema_shared_hold_pa + 8, w, sizeof(w))) + return; + if (!(w[0] & 1u)) + return; + w[0] &= ~1u; + if (svc_write(vgpu, gsp_sema_shared_hold_pa + 8, w, 4)) + return; + svc_shared_hold_log++; + if (svc_shared_hold_log <= 16 || + (svc_shared_hold_log % 4096) == 0) + pr_info("sema-shared: held pa=0x%llx word -> 0x%x (n=%u)\n", + gsp_sema_shared_hold_pa, w[0], svc_shared_hold_log); +} + +/* + * SESSION 149 R17 (M4): one recorded section upload command. The 3D-class + * i2m shape is a 0x180 (LINE_LENGTH_IN) run naming the destination followed + * by a 0x1b0 (LAUNCH_DMA + LOAD_INLINE_DATA) run carrying the payload. + */ +#define SVC_GL_UPLOAD_MAX 24 +struct svc_gl_up { + u64 dst_va; /* from the 0x180 run's OFFSET_OUT_UPPER|OFFSET_OUT */ + u32 len; /* payload bytes copied (payload = run data minus cfg) */ + u8 data[SVC_GR_INLINE_BYTES]; +}; + +#define SVC_GL_PROG_MAX 8 +struct svc_gl_sec { + bool seen; + u32 subch; + u32 start_off; /* byte offset of the C597 SET_OBJECT that opened the + * section - the 3D replay window starts here (r929: + * the prelude before it carries the channel's own + * host-sem/2D ops whose VAs are not mapped in the + * replay channel and FAULT the method walk) */ + u64 prog_va; /* SET_PIPELINE_PROGRAM_ADDRESS_A/B(j) upper|lower */ + /* + * SESSION 153-CONTINUED-11: every SET_PIPELINE_PROGRAM_ADDRESS pair + * the window binds (the engine validates the SPH at EACH bound + * program VA when a draw issues - r944/r946 faults on the draw's + * program pages), with the pipeline slot's shader type collected + * from SET_PIPELINE_SHADER(j) (0x2000+j*0x40, TYPE bits 7:4; 0xff = + * unknown). The per-VA inline records author the program bytes the + * driver's upload never delivered. + */ + u64 progs[SVC_GL_PROG_MAX]; + u8 prog_slot[SVC_GL_PROG_MAX]; + u32 n_progs; + u64 param_va; /* SET_TEX_HEADER_POOL_A/B/C upper|lower */ + u32 param_size; /* observed pool value word (0x10 on the r882 stream) */ + u64 rel_va; /* last VALID release decoded on the channel */ + u32 rel_payload; + u64 ptr[SVC_GR_POINTER_MAX]; + u32 n_ptr; + /* + * SESSION 149 R17 (M4): the section's memory-writing upload commands + * collected in order. On real hardware the 3D engine executes these + * i2m copies (constants, tex pools, ...); in this emulation no engine + * ran them, so every destination stayed zero. svc_gl_flush applies + * the recorded bytes through svc_grce_inline_write (FB aperture -> + * vidmem shadow, sysmem -> guest RAM, unresolved -> VA shadow). + */ + u64 up_dst; /* destination of the last 0x180 run */ + u32 up_bytes; /* its length word (stream); witness only */ + u32 n_up; + struct svc_gl_up up[SVC_GL_UPLOAD_MAX]; + /* + * SESSION 149 R15 (M2-S1): bytes of the first LAUNCH_DMA / + * LOAD_INLINE_DATA run observed inside the section (the observed + * upload payload). Attached to the synthesized candidate as an + * inline record at the program VA so the backend's window-0 shadow + * copy has a non-empty source even when the guest program page + * resolves empty (the GL program page is not among the section's + * own upload destinations). + */ + u8 blob[SVC_GR_INLINE_BYTES]; + u32 blob_len; + /* + * SESSION 153-CONTINUED-18: the kick notification this push carried + * (class2d SET_NOTIFY_A/B + NOTIFY on subchannel 3, decoded after the + * C597 bind that opened this section). Executed at flush time by + * svc_kick_notify_exec when both the target and the NOTIFY are in + * this same push (the short variant carries the NOTIFY without the + * arm pair; it never executes and stays witness-only). + */ + u64 kick_notify_va; + u32 kick_notify_val; + bool kick_notify_seen; +}; +static struct svc_gl_sec svc_gl; /* service thread only */ +/* + * SESSION 153-CONTINUED-11: the channel's sticky per-slot shader types + * (SET_PIPELINE_SHADER(j) TYPE field). Sticky like the C597 bind: the init + * push binds the programs once; later draw pushes reuse the state. 0xff = + * never seen. File-static so a section reset cannot wipe it. + */ +static u8 svc_gl_shader_type[32] = { + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff }; +/* + * SESSION 153-CONTINUED-10 (task 1): the C597 class ATTACHMENT. + * + * A SET_OBJECT bind is sticky on real hardware: the class stays bound to + * the channel subchannel across pushes until another bind replaces it. The + * GLX server's C597 SET_OBJECT arrives once (the driver's init push at + * 0x120010004); every later per-frame CONTENT push (588/644/756 B, ~5/s) + * carries real C597 draws (NVC597_BEGIN 0x1618 + vertex streams + colour + * target + LOAD_CONSTANT_BUFFER payloads) with NO per-push bind, so the + * SET_OBJECT-only recognizer below never opens a section for them and the + * 3D replay lane never sees a draw. Remember (h_object, subch) when a C597 + * bind is seen - including catches of the init push - and treat any later + * method run on that pair, on engine_type==0, as C597 class space. + */ +static u64 svc_gl_att_h; /* channel whose subch carries the C597 bind */ +static u32 svc_gl_att_subch; +static u32 svc_gl_draw_log; /* bounded draw-section-open witness */ +/* + * SESSION 153-CONTINUED-13: the C597 attachment is per-channel. The single + * (h_object, subch) slot was overwritten by the RM channel's periodic + * C597-bind pushes (h=0xbfef0002, observed every ~7 s on this boot), after + * which the GLX client's draw pushes on 0x1000013 failed the BEGIN + * recogniser's `svc_gl_att_h == ch->h_object` test and no draw section was + * ever opened (a boot-level, order-dependent race: the r954 boot decoded + * no such pushes). A small table (first match wins, dedup on note) keeps + * every channel's bind sticky independently, so the draw channel's + * attachment cannot be stolen. + */ +#define SVC_GL_ATT_MAX 8 +static u64 svc_gl_att_tab_h[SVC_GL_ATT_MAX]; +static u8 svc_gl_att_tab_sub[SVC_GL_ATT_MAX]; + +static void svc_gl_att_note(u64 h, u32 subch) +{ + u32 i, free = SVC_GL_ATT_MAX; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) { + if (svc_gl_att_tab_h[i] == h && + svc_gl_att_tab_sub[i] == (u8)subch) { + /* refresh LRU position: move to slot 0 */ + if (i) { + memmove(&svc_gl_att_tab_h[1], + &svc_gl_att_tab_h[0], + (size_t)i * sizeof(u64)); + memmove(&svc_gl_att_tab_sub[1], + &svc_gl_att_tab_sub[0], + (size_t)i); + svc_gl_att_tab_h[0] = h; + svc_gl_att_tab_sub[0] = (u8)subch; + } + return; + } + if (!svc_gl_att_tab_h[i] && free == SVC_GL_ATT_MAX) + free = i; + } + if (free == SVC_GL_ATT_MAX) { + /* table full: evict the oldest (last) slot */ + free = SVC_GL_ATT_MAX - 1; + } + memmove(&svc_gl_att_tab_h[1], &svc_gl_att_tab_h[0], + (size_t)free * sizeof(u64)); + memmove(&svc_gl_att_tab_sub[1], &svc_gl_att_tab_sub[0], + (size_t)free); + svc_gl_att_tab_h[0] = h; + svc_gl_att_tab_sub[0] = (u8)subch; + /* keep the legacy single slot pointing at the newest note (logging) */ + svc_gl_att_h = h; + svc_gl_att_subch = subch; +} + +static bool svc_gl_att_has(u64 h, u32 subch) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h && + svc_gl_att_tab_sub[i] == (u8)subch) + return true; + return false; +} + +static u32 svc_gl_att_subch_of(u64 h) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h) + return svc_gl_att_tab_sub[i]; + return 0xffu; +} + +static void svc_gl_att_clear(u64 h, u32 subch) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h && + svc_gl_att_tab_sub[i] == (u8)subch) { + u32 n = SVC_GL_ATT_MAX - 1 - i; + + if (n) + memmove(&svc_gl_att_tab_h[i], + &svc_gl_att_tab_h[i + 1], + (size_t)n * sizeof(u64)); + if (n) + memmove(&svc_gl_att_tab_sub[i], + &svc_gl_att_tab_sub[i + 1], + (size_t)n); + svc_gl_att_tab_h[SVC_GL_ATT_MAX - 1] = 0; + svc_gl_att_tab_sub[SVC_GL_ATT_MAX - 1] = 0; + if (svc_gl_att_h == h) + svc_gl_att_h = svc_gl_att_tab_h[0]; + return; + } +} +/* + * SESSION 153 (G-B gate: the client's own C597 draws must open sections). + * The replay guard at svc_gl_flush consults svc_gr_cand_push_seen(), which + * returns true when the push VA sits in the GR candidate ring - even when + * no GL section was ever captured for that push. The GL client's draw + * pushes are admitted as GR candidates during driver init (they are read + * as ordinary GR pushes first), so every later replay of them is rejected + * as "replay already captured" and no GL section ever opens. Track GL + * section captures in their own table: a replay only skips when THIS table + * holds the exact (push_va, content hash) pair. The hash keeps a later, + * different push at the same VA (a frame loop: same slot, new content) + * capturable, matching the r383 reasoning for the GR seen table. + */ +#define SVC_GL_CAP_HIST 256 +static struct { + u64 push_va; + u64 hash; +} svc_gl_cap_hist[SVC_GL_CAP_HIST]; +static u32 svc_gl_cap_next; +static u32 svc_gl_cap_seen_log; + +static bool svc_gl_cap_seen(u64 push_va, u64 hash) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GL_CAP_HIST; i++) + if (svc_gl_cap_hist[i].push_va == push_va && + svc_gl_cap_hist[i].hash == hash) + return true; + return false; +} + +static void svc_gl_cap_store(u64 push_va, u64 hash) +{ + u32 i = svc_gl_cap_next++ % SVC_GL_CAP_HIST; + + svc_gl_cap_hist[i].push_va = push_va; + svc_gl_cap_hist[i].hash = hash; +} +/* + * SESSION 153 CONT.6 (task 1): the per-frame GL section-arrival ledger. + * svc_gl_flush is the ONLY point where a C597 section's whole push buffer + * (content included) reaches the GL machinery, so every arrival is + * recorded here: (push_va, content hash, arrival count). The question the + * ledger answers deterministically is whether a NEW (different-content) + * section ever arrives after the driver's init - the wall evidence says + * the X server's 1524 B section is byte-identical every capture ("replay + * already captured" by content hash), but that evidence only covered the + * budgeted replay/skip logs. Now: first arrival, every content change + * (with the first words and the bound-site words), and a periodic + * liveness row per push VA. A hash never seen before on ANY push VA is + * NEW-CONTENT - the moment a genuinely new section (a draw stream) + * appears, it is on record with its words. + */ +#define SVC_GL_ARRIVAL_MAX 64 +static struct { + u64 push_va; + u64 hash; + u32 arrivals; +} svc_gl_arrivals[SVC_GL_ARRIVAL_MAX]; +static u32 svc_gl_arrival_n; +static u64 svc_gl_arrival_total; +static u32 svc_gl_arrival_log; /* bounded new/changed/periodic rows */ + +static void svc_gl_arrival_note(u64 push_va, const u8 *buf, u32 len) +{ + u64 ph = svc_push_hash(buf, len); + u32 i, slot = SVC_GL_ARRIVAL_MAX; + bool known_hash = false; + + for (i = 0; i < svc_gl_arrival_n; i++) { + if (svc_gl_arrivals[i].push_va == push_va) { + slot = i; + break; + } + if (svc_gl_arrivals[i].hash == ph) + known_hash = true; + } + svc_gl_arrival_total++; + if (slot < SVC_GL_ARRIVAL_MAX) { + bool changed = svc_gl_arrivals[slot].hash != ph; + + svc_gl_arrivals[slot].arrivals++; + if (!changed && + svc_gl_arrivals[slot].arrivals % 1024 != 0) + return; + if (svc_gl_arrival_log < 512) { + svc_gl_arrival_log++; + pr_info("sema-glass: %s h=0x%x push=0x%llx len=%u hash=0x%llx n=%u total=%llu first=%08x %08x %08x %08x\n", + changed ? "CONTENT-CHANGED" : "arrival", + 0, push_va, len, ph, + svc_gl_arrivals[slot].arrivals, + svc_gl_arrival_total, + len >= 4 ? get_unaligned_le32(buf) : 0, + len >= 8 ? get_unaligned_le32(buf + 4) : 0, + len >= 12 ? get_unaligned_le32(buf + 8) : 0, + len >= 16 ? get_unaligned_le32(buf + 12) : 0); + } + svc_gl_arrivals[slot].hash = ph; + return; + } + if (svc_gl_arrival_n < SVC_GL_ARRIVAL_MAX) { + slot = svc_gl_arrival_n++; + svc_gl_arrivals[slot].push_va = push_va; + svc_gl_arrivals[slot].hash = ph; + svc_gl_arrivals[slot].arrivals = 1; + if (svc_gl_arrival_log < 512) { + svc_gl_arrival_log++; + pr_info("sema-glass: %s h=0x%x push=0x%llx len=%u hash=0x%llx n=1 total=%llu first=%08x %08x %08x %08x\n", + known_hash ? "KNOWN-CONTENT-OTHER-VA" : "NEW-CONTENT", + 0, push_va, len, ph, svc_gl_arrival_total, + len >= 4 ? get_unaligned_le32(buf) : 0, + len >= 8 ? get_unaligned_le32(buf + 4) : 0, + len >= 12 ? get_unaligned_le32(buf + 8) : 0, + len >= 16 ? get_unaligned_le32(buf + 12) : 0); + } + } +} +static u32 svc_gl_rep_log; /* bounded replay-already-captured witness */ +static u32 svc_gl_prog_log; /* R16 bounded program-read witness */ +static u32 svc_gl_up_log; /* R17 bounded section-upload-apply witness */ +/* + * SESSION 149 R17 (M4): explicit program bytes for the synthesized GL + * launch (hex string, <= GSP_BRIDGE_INLINE_BYTES). The GL program page + * (e.g. 0x121201b00) is empty in every buffer this emulation can see: no + * memory-writing command in any readable push targets it and the + * context-init upload never reached a decodable ring. When set, these + * bytes are attached as the program record (after a trusted VA read, so a + * real upload still wins). Development use only: a minimal real sm_75 + * SASS program (EXIT;NOP;...) proves the engine executes OUR bytes. + */ +static char *sema_gl_prog_hex; +module_param_named(sema_gl_prog_hex, sema_gl_prog_hex, charp, 0644); +MODULE_PARM_DESC(sema_gl_prog_hex, + "explicit program bytes (hex) for the synthesized GL launch (<=256 B)"); +/* + * SESSION 153-CONTINUED-11: per-stage program images. A Turing graphics + * program is [SPH V4 (128 B)][SASS]; the SPH tells the engine the stage + * (VTG vs PS) and the I/O mapping. The engine validates the SPH of every + * bound pipeline program at draw time, so every program VA the window + * binds needs bytes. sema_gl_vs_hex feeds VTG-typed slots, sema_gl_ps_hex + * pixel-typed slots, and sema_gl_prog_hex is the generic fallback (also + * used for the single-record compute path). All three re-parse whenever + * their string changes, so a running module can be iterated against the + * "Shader Program Header N Error" fault oracle without a reload. + */ +static char *sema_gl_vs_hex; +module_param_named(sema_gl_vs_hex, sema_gl_vs_hex, charp, 0644); +MODULE_PARM_DESC(sema_gl_vs_hex, + "explicit VTG program image (SPH+SASS hex, <=256 B) for vertex slots"); +static char *sema_gl_ps_hex; +module_param_named(sema_gl_ps_hex, sema_gl_ps_hex, charp, 0644); +MODULE_PARM_DESC(sema_gl_ps_hex, + "explicit PS program image (SPH+SASS hex, <=256 B) for pixel slots"); +static u8 svc_gl_prog[GSP_BRIDGE_INLINE_BYTES]; +static u32 svc_gl_prog_len; +static u8 svc_gl_vs[GSP_BRIDGE_INLINE_BYTES]; +static u32 svc_gl_vs_len; +static u8 svc_gl_ps[GSP_BRIDGE_INLINE_BYTES]; +static u32 svc_gl_ps_len; +static char svc_gl_prog_seen[GSP_BRIDGE_INLINE_BYTES * 2 + 1]; +static char svc_gl_vs_seen[GSP_BRIDGE_INLINE_BYTES * 2 + 1]; +static char svc_gl_ps_seen[GSP_BRIDGE_INLINE_BYTES * 2 + 1]; + +static void svc_gl_hex_parse(const char *h, const char *tag, u8 *dst, + u32 *dst_len, char *seen) +{ + u32 n, i; + + if (!strcmp(seen, h ? h : "")) + return; /* unchanged since the last parse */ + strscpy(seen, h ? h : "", GSP_BRIDGE_INLINE_BYTES * 2 + 1); + *dst_len = 0; + if (!h || !h[0]) + return; + n = (u32)strlen(h); + if (n % 2 || n / 2 > GSP_BRIDGE_INLINE_BYTES) { + pr_warn("sema-gl: %s param ignored (bad length %u)\n", + tag, n); + return; + } + for (i = 0; i < n; i += 2) { + unsigned int b; + + if (sscanf(h + i, "%2x", &b) != 1) { + pr_warn("sema-gl: %s param bad at %u\n", tag, i); + return; + } + dst[i / 2] = (u8)b; + } + *dst_len = n / 2; + pr_info("sema-gl: %s param: %u bytes stored\n", tag, *dst_len); +} + +static void svc_gl_prog_apply_param(void) +{ + svc_gl_hex_parse(sema_gl_prog_hex, "prog-hex", svc_gl_prog, + &svc_gl_prog_len, svc_gl_prog_seen); + svc_gl_hex_parse(sema_gl_vs_hex, "vs-hex", svc_gl_vs, + &svc_gl_vs_len, svc_gl_vs_seen); + svc_gl_hex_parse(sema_gl_ps_hex, "ps-hex", svc_gl_ps, + &svc_gl_ps_len, svc_gl_ps_seen); +} + +/* + * SESSION 153-CONTINUED-87: the SPH VERSION of the live program images. + * The launch builder emits SET_SPH_VERSION from this value so the + * window's accepted version range always covers the images the engine + * is about to fetch. The client's own init push carries 4|(4<<16), but + * the r950 gate proved the V3 image executes; a 4/4 window over a V3 + * SPH rejects every fetched header (SPH9|SPH11|SPH18, r1273-r1275) - + * the r950-era acceptance ran with no version window at all. Returns + * the max VERSION field of the loaded blobs (0 = none). + */ +int gsp_sema_gl_sph_blob_version(void) +{ + u32 v = 0, ver; + + if (svc_gl_vs_len >= 4) { + ver = (get_unaligned_le32(svc_gl_vs) >> 5) & 0x1fu; + if (ver > v) + v = ver; + } + if (svc_gl_ps_len >= 4) { + ver = (get_unaligned_le32(svc_gl_ps) >> 5) & 0x1fu; + if (ver > v) + v = ver; + } + return (int)v; +} + +/* + * SESSION 153 (GL client draw sections): the GL client's own per-frame + * pushes bind TURING_A (0xC597) exactly like the X server's one-shot + * section, but carry no program/param/release operands at all (measured: + * `0001fff0 20010000 0000c597 0001fff0 20016043 00000000 20016040 + * 00000000` on h=0xbfef0002 - the C597 SET_OBJECT plus 2D-class + * NoOp/Notify only). The old flush guard required the full operand trio + * and silently dropped every client section, so no client capture ever + * appeared. Two additive sources close that gap: + * + * 1. a per-channel operand HOLD: the last valid prog/param/rel operands + * seen in a section on this channel are remembered and adopted by a + * later draw section on the same channel (still marked src=hold); + * 2. a dev fallback (gsp_sema_gl_draw_synth): when the client + * never supplied operands at all, the draw section captures with the + * operand values the X server's one-shot GL section supplied in the + * same boot class (r891/r923), marked src=draw-synth. + * + * r929 (Path A): the dev fallback is now OFF by default. The stream + * captures (real operands, src=stream) are the primary 3D path and replay + * verbatim; a draw-synth capture synthesizes a compute QMD from fake/dev + * operands, and on this boot class its launches never completed + * (qmd_rel=0x600df00d / ch_rel miss -> SYNTH-FAULT + GR-COMPLETE-SW every + * ~19 s per kick, GZDoom wedged). Set sema_gl_draw_synth=1 to A/B the old + * fallback. + */ +#define SVC_GL_HOLD_MAX 8 +struct svc_gl_hold { + u32 h; + u64 prog_va; + u64 param_va; + u32 param_size; + u64 rel_va; + u32 rel_payload; +}; +static struct svc_gl_hold svc_gl_hold_tab[SVC_GL_HOLD_MAX]; +static u32 svc_gl_open_log; /* bounded section-open witness */ +static u32 svc_gl_skip_log; /* bounded flush-skip witness */ +static u32 svc_gl_np_log; /* CONT.45 bounded no-param stream witness */ +static u32 svc_gl_mark_log; /* CONT.45 bounded 3D-mark-failure witness */ +/* + * SESSION 153-CONTINUED-18 (task 1): the kick NOTIFY executor. + * + * The GL client's per-frame kick push on the kick-family channel + * (h=0xbfef0002) is [SET_OBJECT(C597)][SET_NOTIFY_A {1, 0x2000e0b0}] + * [NOTIFY 0x10C][NO_OP]: the class2d notification write at VA + * 0x12000e0b0 (the shared .nvidia_drv member, page offset 0xB0; its u16 + * status at +0xE). On real hardware the 2D engine writes a completed + * NvNotification there (nonzero timestamp at +0x0, status 0 at +0xE) - + * that write is the client's per-frame completion edge; the lane used to + * log "flush skip" and drop it, leaving the member all-zero forever. This + * executor authors the NvNotification through the same trusted VA walk + * the release path uses (svc_lookup_trusted + svc_write_ap) and verifies + * by readback. sema_kick_notify=0 disables (A/B); the witness line + * "sema-kick: exec ..." is bounded (first 48 + every 512th). + */ +static bool gsp_sema_kick_notify = true; +module_param_named(sema_kick_notify, gsp_sema_kick_notify, bool, 0644); +MODULE_PARM_DESC(sema_kick_notify, + "Execute the GL client's per-frame kick NOTIFY (author the NvNotification at the SET_NOTIFY target VA; default 1, session 153 CONT.18)"); +static u32 svc_kick_log; /* bounded kick-notify witness */ +static u64 svc_kick_count; /* executor invocations */ +/* + * r982: the notify page a pending-marker hit proved, per knowledge the + * exec picked up. Kept in the service's own scope (not the channel + * snapshot): the kick channel is the only consumer and the entry is + * re-validated by a read before every use, so a stale PA can only cost + * one failed probe, never a write into freed memory. + */ +static struct { + u64 h; /* channel handle that proved the page */ + u64 pa; /* PA of the marker-proven notify page */ +} svc_kick_pa_cache[4]; + +static void svc_kick_pa_cache_store(u64 h, u64 pa) +{ + u32 i; + + for (i = 0; i < ARRAY_SIZE(svc_kick_pa_cache); i++) { + if (svc_kick_pa_cache[i].h == h) { + svc_kick_pa_cache[i].pa = pa; + return; + } + } + for (i = 0; i < ARRAY_SIZE(svc_kick_pa_cache); i++) { + if (!svc_kick_pa_cache[i].h) { + svc_kick_pa_cache[i].h = h; + svc_kick_pa_cache[i].pa = pa; + return; + } + } + svc_kick_pa_cache[0].h = h; + svc_kick_pa_cache[0].pa = pa; +} + +static bool gsp_sema_gl_draw_synth; +module_param_named(sema_gl_draw_synth, gsp_sema_gl_draw_synth, bool, 0644); +MODULE_PARM_DESC(sema_gl_draw_synth, + "Capture a draw-shaped C597 section with the dev fallback operands when the client supplied none (default 0 since r929; witness marks src=draw-synth)"); +#define SVC_GL_SYNTH_PROG_VA 0x121201b00ULL +#define SVC_GL_SYNTH_PARAM_VA 0x1214fa220ULL +#define SVC_GL_SYNTH_PARAM_SIZE 0x10u +#define SVC_GL_SYNTH_REL_VA 0x120001000ULL +#define SVC_GL_SYNTH_REL_PAY 1u + +static void svc_gl_hold_store(u32 h, u64 prog, u64 param, u32 psize, + u64 rel, u32 rpay) +{ + u32 i, free = SVC_GL_HOLD_MAX; + + /* + * SESSION 153-CONTINUED-13: rel == 0 is admitted (a section that has + * real operands but flushed before the channel's release was decoded + * stashes them here); a later call with rel != 0 completes the entry + * and a rel == 0 call must NEVER wipe a stored release. + */ + if (!h || !prog || !param) + return; + for (i = 0; i < SVC_GL_HOLD_MAX; i++) { + if (svc_gl_hold_tab[i].h == h) { + free = i; + break; + } + if (!svc_gl_hold_tab[i].h && free == SVC_GL_HOLD_MAX) + free = i; + } + if (free == SVC_GL_HOLD_MAX) + free = h % SVC_GL_HOLD_MAX; + svc_gl_hold_tab[free].h = h; + svc_gl_hold_tab[free].prog_va = prog; + svc_gl_hold_tab[free].param_va = param; + svc_gl_hold_tab[free].param_size = psize; + if (rel) { + svc_gl_hold_tab[free].rel_va = rel; + svc_gl_hold_tab[free].rel_payload = rpay; + } +} + +static bool svc_gl_hold_load(u32 h, u64 *prog, u64 *param, u32 *psize, + u64 *rel, u32 *rpay) +{ + u32 i; + + for (i = 0; i < SVC_GL_HOLD_MAX; i++) { + if (svc_gl_hold_tab[i].h != h || !h) + continue; + if (prog) + *prog = svc_gl_hold_tab[i].prog_va; + if (param) + *param = svc_gl_hold_tab[i].param_va; + if (psize) + *psize = svc_gl_hold_tab[i].param_size; + if (rel) + *rel = svc_gl_hold_tab[i].rel_va; + if (rpay) + *rpay = svc_gl_hold_tab[i].rel_payload; + return true; + } + return false; +} + +/* + * SESSION 149 R17 (M4): standalone GL-class upload tracker. A context-init + * upload push (the one-shot program upload) is NOT required to sit inside a + * C597 section, but it carries the same subchannel-0 0x180/0x1b0 i2m shape. + * The tracker pairs the last 0x180 destination with each following 0x1b0 + * payload and applies it on sight (sema_gl_xlate, eng==0, trusted/allow, + * once-only) so the destination (e.g. the program page) gets its content + * even when the push carries no class bind this pass recognises. + */ +static u64 svc_gl_spot_dst; /* last 0x180 destination (per pass) */ +static u32 svc_gl_spot_bytes; /* its length word (witness) */ +static u32 svc_gl_lid_log; /* bounded standalone-apply witness */ + +static void svc_gl_add_ptr(u64 va) +{ + u32 k; + + if (!va) + return; + /* + * SESSION 153-CONTINUED-48: dedup by 4 KB page, not exact VA. + * The QMD pointer region holds only seven pairs (words 34..47, + * SVC_GR_POINTER_MAX - 1); a window that binds several different + * offsets of one buffer (r1058 chunk1: five index-buffer VAs + * across three pages) used to consume one slot per exact VA and + * hit the cap, so no later operand page (the raw CALL_MME_MACRO + * payload VAs 0x43545f0/0xb94f700) could join the set and the + * shadow relocation left them raw - the raw operand the engine + * faulted on (grfifo=0xbadf1201). One pointer per page carries + * the same page set with slots left for the remaining operand + * families; the first VA seen for a page is kept so the value + * stays a real client VA (the fill resolves pages, and the + * relocation matches by page). + */ + for (k = 0; k < svc_gl.n_ptr; k++) + if (svc_gl.ptr[k] == va || + (svc_gl.ptr[k] & ~(u64)0xfffu) == (va & ~(u64)0xfffu)) + return; + if (svc_gl.n_ptr < SVC_GR_POINTER_MAX - 1) + svc_gl.ptr[svc_gl.n_ptr++] = va; +} + +/* + * Collect one observed VA operand from the open C597 section. Every case + * is transcribed from the decoded r882 stream words; the upper-word masks + * follow the class definitions (ADDRESS_UPPER 7:0 / lower 31:0). + */ +static void svc_gl_field(u32 addr, u32 count, const u8 *data, u32 dlen) +{ + u32 v[4] = { 0, 0, 0, 0 }; + u32 n = count > 4 ? 4 : count, k; + + for (k = 0; k < n; k++) { + if (4 * (k + 1) > dlen) + break; + memcpy(&v[k], data + 4 * k, 4); + } + /* + * SESSION 153-CONTINUED-11: SET_PIPELINE_SHADER(j) = 0x2000+j*0x40, + * [ENABLE(0) | TYPE(7:4)]. Collected per slot so a bound program VA + * can be typed (VTG vs PS) for the program-bytes delivery. + */ + /* + * SESSION 153-CONTINUED-48: NVC597_SET_REPORT_SEMAPHORE_A..D = 0x1b00, + * observed as 4-word runs [A=upper(7:0), B=lower(31:0), C=payload, + * D=flags] at the end of every glcore draw window (payload = the + * client's frame counter, e.g. 0x4f9f6; VA e.g. 0x4181bb0/0xc91e6c0). + * These are the engine's own completion writes; with the VA raw the + * replayed engine writes a client VA the GPC engines cannot reach - + * the same VIRT_WRITE fault class the 0x10c NOTIFY drop exists for + * (r1061: the kept [0,0x2000) window that still faults is exactly + * the one carrying these raw 0x1b00 VAs). Collect the semaphore page + * so the window relocation can rewrite the pair to a shadow VA: the + * write then lands in the SAME HPA and the client's wait sees the + * real hardware completion. + */ + if (addr == 0x1b00) { + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xffu) << 32) | + (u64)(v[1] & 0xfffffffcu)); + return; + } + if (addr >= 0x2000 && addr < 0x2800 && (addr & 0x3f) == 0) { + u32 j = (addr - 0x2000) / 0x40; + + if (j < 32 && count >= 1) + svc_gl_shader_type[j] = (u8)((v[0] >> 4) & 0xf); + return; + } + /* + * SESSION 153-CONTINUED-48: CALL_MME_MACRO(j)=0x3800+j*8 / + * CALL_MME_DATA(j)=0x3804+j*8 runs carry raw 32-bit client VAs as + * macro arguments (r1058 chunk1: the macro(2) run payloads + * 0x44ff800/0x44ff000 relocated only because the page set already + * held 0x44ff000 via the index bind; the sibling payloads + * 0x43545f0/0x4353f60/0xb94f700/0xb941578 stayed raw because their + * pages missed the capped pointer list - the raw operand class the + * GR faulted on). Collect every VA-shaped payload word so the page + * set - and therefore the 3D-window shadow relocation - covers the + * macro arguments too. This must run before the 0x2014..0x3fff + * program-pair recognizer below, whose (addr & 0x3f) == 0x14 shape + * would otherwise swallow a CALL_MME_DATA(2)=0x3814 run. Shape + * gates: 4-byte aligned; >= 64 KB excludes the method/flag scalars + * (0x0c/0x53/0x338); < 512 MB excludes the float constants the + * macro(57) runs carry (1.0f=0x3f800000) while keeping every + * observed client operand VA (max 0xb94f700). + */ + if (addr >= 0x3800 && addr < 0x3a00) { + for (k = 0; k < count; k++) { + u32 w; + + if (4 * (k + 1) > dlen) + break; + memcpy(&w, data + 4 * k, 4); + if (w >= 0x10000u && w < 0x20000000u && + !(w & 3u)) + svc_gl_add_ptr((u64)w); + } + return; + } + /* SET_PIPELINE_PROGRAM_ADDRESS_A/B(j), A=upper(7:0), B=lower(31:0). + * Observed: 0x20020815 (j=1) [00000001 21201b00] -> 0x121201b00. + * SESSION 153-CONTINUED-11: EVERY pair is recorded (the engine + * validates each bound program's SPH at draw time) and the program + * page joins the candidate's page set so the per-VA inline records + * can land there. + */ + if (addr >= 0x2014 && addr < 0x4000 && (addr & 0x3f) == 0x14 && + count >= 2) { + u64 pva = ((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc); + u32 j = (addr - 0x2014) / 0x40, k; + + if (!svc_gl.prog_va) + svc_gl.prog_va = pva; + if (pva && j < 32) { + bool dup = false; + + for (k = 0; k < svc_gl.n_progs; k++) + if (svc_gl.progs[k] == pva) + dup = true; + if (!dup && svc_gl.n_progs < SVC_GL_PROG_MAX) { + svc_gl.progs[svc_gl.n_progs] = pva; + svc_gl.prog_slot[svc_gl.n_progs] = (u8)j; + svc_gl.n_progs++; + } + svc_gl_add_ptr(pva); + } + return; + } + switch (addr) { + case 0x1574: /* SET_TEX_HEADER_POOL_A/B/C: upper, lower, index. + * Observed: [00000001 214fa220 00000010]. + */ + if (count >= 3 && !svc_gl.param_va) { + svc_gl.param_va = ((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc); + svc_gl.param_size = v[2]; /* observed 0x10 */ + } + return; + case 0x155c: /* SET_TEX_SAMPLER_POOL_A/B/C: [1 214fa200 0] */ + if (count >= 3) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + case 0x0f84: /* SET_VERTEX_STREAM_SUBSTITUTE_A/B: [1 21200000] */ + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + case 0x0180: /* LINE_LENGTH_IN run: VALUE, LINE_COUNT, + * OFFSET_OUT_UPPER, OFFSET_OUT, PITCH_OUT. + * Observed: [0000000c 1 1 214f8f00 0000000c]. + */ + if (count >= 4) + svc_gl_add_ptr(((u64)(v[2] & 0xff) << 32) | + (u64)(v[3] & 0xfffffffc)); + /* R17: this run names the destination the next 0x1b0 + * LAUNCH_DMA/LOAD_INLINE_DATA run writes to. */ + if (count >= 5) { + svc_gl.up_dst = ((u64)(v[2] & 0xff) << 32) | + (u64)(v[3] & 0xfffffffc); + svc_gl.up_bytes = v[0]; + } + return; + case 0x01b0: /* LAUNCH_DMA + LOAD_INLINE_DATA (op3/op5): the run's + * data words ARE the observed upload payload. Keep the + * FIRST run's bytes for the M2-S1 inline attachment. + * Observed: a004006c -> [00000041 00000670 00000690 + * 000006b0] = LAUNCH_DMA cfg + 3 LID words = 12 B. + * SESSION 149 R16 (M3): keep only the LID payload (the + * cfg word is not code) and bound the copy by the run's + * OWN count - dlen reaches the end of the push, so the + * R15 copy over-read the run (blob=64 = 12 real bytes + + * 52 B of the following method words). These bytes are + * only the fallback: svc_gl_flush reads the real + * program bytes from prog_va via the trusted walk. + */ + if (!svc_gl.blob_len && count > 1 && dlen > 4) { + u32 n = (count - 1) * 4; + + if (n > dlen - 4) + n = dlen - 4; + if (n > sizeof(svc_gl.blob)) + n = (u32)sizeof(svc_gl.blob); + if (n) { + memcpy(svc_gl.blob, data + 4, n); + svc_gl.blob_len = n; + } + } + /* + * R17: record the run's payload at the destination the + * preceding 0x180 run named (the payload starts after the + * LAUNCH_DMA config word - same split as the blob above). + */ + if (svc_gl.up_dst && count > 1 && dlen > 4 && + svc_gl.n_up < SVC_GL_UPLOAD_MAX) { + struct svc_gl_up *u = &svc_gl.up[svc_gl.n_up]; + u32 n = (count - 1) * 4; + + if (n > dlen - 4) + n = dlen - 4; + if (n > sizeof(u->data)) + n = (u32)sizeof(u->data); + if (n) { + u->dst_va = svc_gl.up_dst; + u->len = n; + memcpy(u->data, data + 4, n); + svc_gl.n_up++; + } + } + return; + case 0x17c8: /* SET_INDEX_BUFFER_A/B: A=ADDRESS_UPPER(7:0), + * B=ADDRESS_LOWER(31:0). SESSION 153-CONTINUED-46: + * the indexed draws (glxgears' glDrawElements) fetch + * their indices from this VA; r1039 evidence: the + * client's 3D window binds it + * ("200205f2 ... data=00000000 0450fcc8" -> VA + * 0x0450fcc8) but the candidate page set held only the + * program page (n_pages=1) while X's retiring windows + * carry 3-6 pages - the index page is never mapped in + * the replay channel. Collect it like every other + * address operand. + */ + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + case 0x1c20: /* SET_VERTEX_STREAM_A_FORMAT/BASE(j) for j >= 2: + * same [fmt, BASE_UPPER, BASE_LOWER] shape as 0x1c00/ + * 0x1c10; the old switch only matched j=0/1. + */ + case 0x1c30: + case 0x1c40: + case 0x1c50: + case 0x1c60: + case 0x1c70: + case 0x1c80: + case 0x1c90: + case 0x1ca0: + case 0x1cb0: + case 0x1cc0: + case 0x1cd0: + case 0x1ce0: + case 0x1cf0: + case 0x1c00: /* SET_VERTEX_STREAM_A_FORMAT(j) + j*0x10: the run + * carries [fmt, BASE_UPPER, BASE_LOWER] - the vertex + * stream the draw reads positions/colours from. + * Observed 0x1c00 [00001000 00000001 2150b790] and + * 0x1c10 [00001008 00000001 215257c0] (SESSION + * 153-CONTINUED-10 draw-push decode). + */ + case 0x1c10: + if (count >= 3) + svc_gl_add_ptr(((u64)(v[1] & 0xff) << 32) | + (u64)(v[2] & 0xfffffffc)); + return; + case 0x2380: /* SET_CONSTANT_BUFFER_SELECTOR_A/B: [size, upper, + * lower]. Observed [00010000 00000001 2150a500]. + */ + if (count >= 3) + svc_gl_add_ptr(((u64)(v[1] & 0xff) << 32) | + (u64)(v[2] & 0xfffffffc)); + return; + case 0x0800: /* SET_COLOR_TARGET_A/B(j=0): [OFFSET_UPPER, + * OFFSET_LOWER, w, h, fmt, ...] - the surface the + * draw rasterizes into. Observed [00000001 222fa200 + * 00000040 00000004 000000f7 00000000]. + */ + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + default: + return; + } +} + +/* + * SESSION 153-CONTINUED-10: re-walk [start, end) of the push and collect + * the C597 operand fields of every method run on the section's subchannel. + * Used when a draw section opens at its BEGIN marker (0x1618): the state + * methods that precede the marker (vertex stream binds, colour target, + * constant buffer binds) were decoded before the section existed, so their + * VAs must be re-collected or the candidate's page set starves and the + * replay's shadow relocation leaves them pointing at unreachable guest + * VAs (the r929 v2 FAULT_PDE class). Method shapes mirror the main + * decoder: op 1/3/5 carry 1+count dwords, everything else is header-only. + */ +static void svc_gl_scan_window(u32 start, u32 end, const u8 *buf, u32 len, + u32 att_subch) +{ + u32 o = start; + + while (o + 4 <= end && o + 4 <= len) { + u32 hdr, op, count, subch, addr; + + memcpy(&hdr, buf + o, 4); + op = (hdr >> 29) & 0x7; + count = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + addr = (hdr & 0x1fff) << 2; + if ((op == 1 || op == 3 || op == 5) && count && + o + 4 + count * 4 <= len) { + if (subch == att_subch) + svc_gl_field(addr, count, buf + o + 4, + len - o - 4); + o += 4 + count * 4; + continue; + } + o += 4; + } +} + +/* + * SESSION 153-CONTINUED-18 (task 1): execute the GL client's per-frame + * kick NOTIFY (see the sema_kick_notify comment above). The GPU's write + * is a completed NvNotification: nonzero timestamp at +0x0 (ns since + * epoch), info32 at +0x8 (the NOTIFY payload), info16 at +0xC, status 0 + * at +0xE. Resolution is the release path's trusted walk + * (svc_lookup_trusted -> svc_write_ap), then readback verification. + */ +static bool svc_kick_family(const struct svc_channel *ch); +static u64 svc_push_pa_ceiling(void); +static bool svc_kick_notify_exec(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 va, u32 val, + u64 push_va) +{ + u64 pa = 0, ts = ktime_get_real_ns(); + int ap = SVC_APERTURE_SYS; + u8 old[16], nw[16], rb[16]; + bool ok = false, changed = false; + u32 o0 = 0, n0 = 0; + int rc = -ENOENT; + const char *rsrc = "trusted"; + u32 kc = 0; + + memset(nw, 0, sizeof(nw)); + memcpy(nw, &ts, sizeof(ts)); + memcpy(nw + 8, &val, sizeof(val)); + if (!svc_lookup_trusted(vgpu, va, &pa, &ap)) { + int r1, r2; + + /* + * SESSION 153-CONTINUED-20: prefer the client-visible sysmem + * mapping. The target is the completion word the client's + * flush loop polls through its own memfd; r974/r975 measured + * the serviced-channel VAS walk answering this VA with the FB + * shadow (ap=2, pa=0x20e0b0) - a write no client can observe + * (r973's SYS answer, pa=0x12ae340b0, was the readable one, + * and the ch=NULL dev probe on r975 re-resolved the same VA + * to SYS pa=0x104b640b0 = the kick ring page + 0x1000 + 0xb0, + * exactly the r973 ring-adjacent layout). When the biased + * chain lands in FB, re-resolve with the serviced-channel VAS + * bias cleared and take the sysmem answer when it exists. + */ + if (svc_ap_of(ap) == SVC_APERTURE_FB) { + u64 pa2 = 0; + int ap2 = SVC_APERTURE_FB; + u32 save = svc_cur_vas; + + svc_cur_vas = 0; + if (!svc_lookup_trusted(vgpu, va, &pa2, &ap2) && + svc_ap_of(ap2) == SVC_APERTURE_SYS) { + pa = pa2; + ap = ap2; + rsrc = "trusted-u"; + } + svc_cur_vas = save; + } + /* + * r978: r977's exec fires (27 calls) but authors the word in + * the vidmem shadow (ap=2 pa=0x20e0b0) that no client reads, + * while r973 - the boot where the kick consumption visibly + * advanced the client - wrote the RING-ADJACENT sysmem slot + * (exec pa=0x12ae340b0 over ring pa=0x12ae33000 = ring + + * 0x1000 + 0xb0; the r975 probe measured the identical rule, + * pa=0x104b640b0 over ring 0x104b63000). Both walk answers + * miss that page this boot, so before falling back to the + * shadow: (a) the general lookup (the discovered rm_mem map - + * r979's sema_probe proved it resolves this VA to SYS after + * the map rebuild, pa=0x1184a40b0), (b) a driver-declared pin + * for this VA (the guest pin stream is the client's own + * statement of the mapping), then (c) for a kick-family + * channel the ring-relative completion slot, computed from + * the live ring pin each call. + */ + if (svc_ap_of(ap) != SVC_APERTURE_SYS) { + u64 mpa = 0; + int map_ap = SVC_APERTURE_SYS; + + if (!svc_lookup(vgpu, va, &mpa, &map_ap) && + svc_ap_of(map_ap) == SVC_APERTURE_SYS) { + pa = mpa; + ap = map_ap; + rsrc = "map"; + } + } + if (svc_ap_of(ap) != SVC_APERTURE_SYS) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(va, &ipa, &iap) && + svc_ap_of(iap) == SVC_APERTURE_SYS) { + pa = ipa; + ap = iap; + rsrc = "ipin"; + } + } + if (svc_ap_of(ap) != SVC_APERTURE_SYS && ch && + svc_kick_family(ch)) { + u64 ring_pa = 0; + int ring_ap = SVC_APERTURE_SYS; + + if (!svc_pin_lookup_any(ch->gpfifo_va, &ring_pa, + &ring_ap) && + svc_ap_of(ring_ap) == SVC_APERTURE_SYS) { + u64 cand = (ring_pa & + ~(u64)(SEMA_PAGE_SIZE - 1)) + + SEMA_PAGE_SIZE + + (va & (SEMA_PAGE_SIZE - 1)); + + if (cand < svc_push_pa_ceiling()) { + pa = cand; + ap = SVC_APERTURE_SYS; + rsrc = "ring+10b0"; + } + } + } + /* + * r981: the client's pending notification marker. r979/r980 + * measured the REAL slot's +0xC u32 reading 0x80000000 + * while the client waited (and 0 once completed; the write + * below clears it) - on the page the discovered map names, + * ring + 0x20b0 on both boots, one page ABOVE the r973-era + * ring + 0x10b0 heuristic the exec had been writing + * (0x1184a40b0 vs 0x1184a30b0 on r979; 0x12e07a0b0 vs + * 0x12e0790b0 on r980 - the completion landed on the wrong + * page both boots). When the chosen candidate does not show + * the marker, probe the ring-relative alternatives and use + * the one that does; the readback below still verifies the + * write. A slot already completed (marker 0) keeps the + * resolved candidate unchanged. + * + * r982: remember the page a pending-marker hit named, per + * handle. r981's boot: exec n=1 found the marker page + * (pa=0x129c1b0b0, ring+0x20b0) and completed it, but the + * marker is 0 afterwards so every later exec fell back to + * ring+0x10b0 (0x129c1a0b0 - wrong page, 33 further writes) + * while the map/trusted stages were still catching up. The + * cached page is retried at the top of the next call (read + * first, used only while the read succeeds), so all later + * completions stay on the page the marker proved. + */ + if (ch && svc_kick_family(ch)) { + /* Retry the page a marker hit proved, if still readable. */ + for (kc = 0; kc < 4; kc++) { + u32 probe_buf = 0; + + if (svc_kick_pa_cache[kc].h != ch->h_object || + !svc_kick_pa_cache[kc].pa) + continue; + if (!svc_read_ap(vgpu, svc_kick_pa_cache[kc].pa, + SVC_APERTURE_SYS, &probe_buf, + sizeof(probe_buf)) && + probe_buf != 0xffffffffu) { + pa = svc_kick_pa_cache[kc].pa; + ap = SVC_APERTURE_SYS; + rsrc = "kickcache"; + } + break; + } + } + if (ch && svc_kick_family(ch)) { + u32 pend = 0; + + if (!svc_read_ap(vgpu, pa + 0xc, svc_ap_of(ap), + &pend, sizeof(pend)) && + pend != 0x80000000u) { + u64 ring_pa = 0; + int ring_ap = SVC_APERTURE_SYS; + + if (!svc_pin_lookup_any(ch->gpfifo_va, + &ring_pa, &ring_ap) && + svc_ap_of(ring_ap) == SVC_APERTURE_SYS) { + u64 base = ring_pa & + ~(u64)(SEMA_PAGE_SIZE - 1); + u32 off = (u32)(va & (SEMA_PAGE_SIZE - 1)); + u64 cands[2]; + int i; + + cands[0] = base + + 2 * SEMA_PAGE_SIZE + off; + cands[1] = base + + SEMA_PAGE_SIZE + off; + for (i = 0; i < 2; i++) { + u32 m = 0; + + if (cands[i] >= + svc_push_pa_ceiling()) + continue; + if (!svc_read_ap(vgpu, + cands[i] + 0xc, + SVC_APERTURE_SYS, + &m, sizeof(m)) && + m == 0x80000000u) { + pa = cands[i]; + ap = SVC_APERTURE_SYS; + rsrc = i ? + "ring+10b0p" : + "ring+20b0p"; + svc_kick_pa_cache_store( + ch->h_object, + cands[i]); + break; + } + } + } + } else if (pend == 0x80000000u && + svc_ap_of(ap) == SVC_APERTURE_SYS) { + /* + * The chosen candidate IS the marker page: + * prove it for the cache so later execs keep + * writing the same page (r981: after the + * single marker hit the fallback wrote two + * wrong ring-relative pages for 33 further + * execs). Only a SYS answer that showed the + * marker is cached; a zero read (already + * completed) caches nothing - unproven. + */ + svc_kick_pa_cache_store(ch->h_object, pa); + } + } + r1 = svc_read_ap(vgpu, pa, svc_ap_of(ap), old, sizeof(old)); + r1 = svc_read_ap(vgpu, pa, svc_ap_of(ap), old, sizeof(old)); + r2 = svc_write_ap(vgpu, pa, svc_ap_of(ap), nw, sizeof(nw)); + if (!r1 && !r2 && + !svc_read_ap(vgpu, pa, svc_ap_of(ap), rb, sizeof(rb))) { + rc = 0; + ok = true; + changed = memcmp(old, rb, sizeof(old)) != 0; + memcpy(&o0, old, 4); + memcpy(&n0, rb, 4); + } else { + /* CONT.18 probe: name the failing step. */ + rc = r1 ? (r1 < 0 ? r1 : -100) : + (r2 ? (r2 < 0 ? r2 : -101) : -102); + } + } + svc_kick_count++; + if (gsp_sema_dbg && (svc_kick_log < 48 || !(svc_kick_count & 127))) { + svc_kick_log++; + pr_info("sema-kick: exec h=0x%x push=0x%llx va=0x%llx pa=0x%llx ap=%d src=%s val=0x%x ts=0x%llx old=%08x new=%08x changed=%d n=%llu rc=%d\n", + ch ? ch->h_object : 0, push_va, va, pa, ap, rsrc, val, + (unsigned long long)ts, o0, n0, changed ? 1 : 0, + (unsigned long long)svc_kick_count, rc); + } + /* + * SESSION 153-CONTINUED-19: the completed NOTIFY is the submitting + * client's completion edge - post the channel event so its OS event + * fires (r934's lesson on the notify2d path: without the event the + * GL client's kick submit wait never completes and the app retries + * on a ~16-22 s timer; r972 measured exactly that cadence with the + * event missing here). Only for a real channel (the dev probe + * passes ch=NULL). + */ + if (ok && ch) + gsp_rm_post_channel_event(vgpu, ch->h_object); + return ok; +} + +/* + * SESSION 153-CONTINUED-18 (task 1, dev probe): run the kick-notify + * executor on an explicit VA against the service's own device. The + * execution MUST run in a kernel-thread context: vfio_iommu_type1's + * dma_rw only copies when current->mm == dma->mm or the caller is a + * kthread (kthread_use_mm); a sysfs write from a shell process has + * current->mm != QEMU's mm and gets -EFAULT (measured rc=-14). The + * workqueue item below is a kthread - the same context class as the + * sema service thread that runs the real kick lane. + * echo 0x12000e0b0 > .../parameters/sema_kick_probe + */ +static u64 svc_kick_probe_va; +static void svc_kick_probe_work_fn(struct work_struct *w); +static DECLARE_WORK(svc_kick_probe_work, svc_kick_probe_work_fn); + +static void svc_kick_probe_work_fn(struct work_struct *w) +{ + struct nv_vgpu_device *vgpu = NULL; + unsigned long flags; + int i; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu) { + vgpu = svc.chan[i].vgpu; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (vgpu) + svc_kick_notify_exec(vgpu, NULL, svc_kick_probe_va, 0, 0); +} + +static int nv_vgpu_sema_kick_probe_set(const char *val, + const struct kernel_param *kp) +{ + if (!val || kstrtoull(val, 0, &svc_kick_probe_va)) + return -EINVAL; + if (svc_kick_probe_va) + schedule_work(&svc_kick_probe_work); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_kick_probe_ops = { + .set = nv_vgpu_sema_kick_probe_set, +}; + +module_param_cb(sema_kick_probe, &nv_vgpu_sema_kick_probe_ops, NULL, 0200); +MODULE_PARM_DESC(sema_kick_probe, "Run the kick-notify executor on a VA (dev probe; session 153 CONT.18)"); + +/* + * SESSION 153-CONTINUED-44 (r1026): host-driven present blit. + * + * The client's rendered frame provably exists in guest RAM (its readback + * staging) while the X server scans the emulated screen surface in vidmem + * (base 0x200000, pitch 5120 for the 1280x720 4-bpp screen), so the window + * shows only the DDX's cleared fill artifact. This is a bounded, explicit + * present emulation: the host names the guest-PA staging of the CLIENT'S + * OWN GPU-rendered frame, and the kernel blits those rows into the scanned + * surface - never synthesizing pixels. The host drives it with: + * + * echo <src_gpa>,<src_pitch>,<w>,<h>[,<dst_off>,<dst_pitch>] \ + * > .../parameters/sema_screen_blit + * + * (all values 0x-hex or decimal; src_pitch and w/h in bytes/pixels, bpp=4; + * defaults dst_off=0x200000, dst_pitch=5120). Bounded: w,h <= 4096, + * pitch <= 65536, total <= 64 MiB, one row buffer. Runs on a workqueue + * (vfio_dma_rw for the guest read, bars[1] shadow for the vidmem write), + * logs one line with the byte/row counts. + */ +#define SVC_SCREEN_BLIT_DEF_OFF 0x200000ULL +#define SVC_SCREEN_BLIT_DEF_PITCH 5120u +static u64 svc_screen_blit_gpa; +static u32 svc_screen_blit_spitch, svc_screen_blit_w, svc_screen_blit_h; +static u64 svc_screen_blit_doff; +static u32 svc_screen_blit_dpitch; +static void svc_screen_blit_work_fn(struct work_struct *w); +static DECLARE_WORK(svc_screen_blit_work, svc_screen_blit_work_fn); + +static void svc_screen_blit_work_fn(struct work_struct *w) +{ + struct nv_vgpu_device *vgpu = NULL; + unsigned long flags; + u8 *row, *base; + size_t size; + u32 y, w4 = svc_screen_blit_w * 4; + u64 bytes = 0; + int i, rc = 0; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu) { + vgpu = svc.chan[i].vgpu; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!vgpu) { + pr_info("sema-screen-blit: no vgpu (no channel registered yet)\n"); + return; + } + base = vgpu->bars[1].vaddr; + size = svc_fb_shadow_size(vgpu); + if (!base || svc_screen_blit_doff + (u64)svc_screen_blit_dpitch * + svc_screen_blit_h > size) { + pr_info("sema-screen-blit: bounds (off=0x%llx pitch=%u h=%u size=%zu)\n", + svc_screen_blit_doff, svc_screen_blit_dpitch, + svc_screen_blit_h, size); + return; + } + row = kmalloc(svc_screen_blit_spitch, GFP_KERNEL); + if (!row) { + pr_info("sema-screen-blit: no memory for row (pitch=%u)\n", + svc_screen_blit_spitch); + return; + } + for (y = 0; y < svc_screen_blit_h; y++) { + rc = svc_read(vgpu, svc_screen_blit_gpa + (u64)y * + svc_screen_blit_spitch, row, + svc_screen_blit_spitch); + if (rc) + break; + memcpy(base + svc_screen_blit_doff + (u64)y * + svc_screen_blit_dpitch, row, w4); + bytes += w4; + } + kfree(row); + pr_info("sema-screen-blit: src=0x%llx spitch=%u w=%u h=%u -> off=0x%llx dpitch=%u rows_done=%u bytes=%llu rc=%d\n", + svc_screen_blit_gpa, svc_screen_blit_spitch, + svc_screen_blit_w, svc_screen_blit_h, + svc_screen_blit_doff, svc_screen_blit_dpitch, y, bytes, rc); +} + +static int nv_vgpu_sema_screen_blit_set(const char *val, + const struct kernel_param *kp) +{ + char buf[128]; + int nf = 0; + u64 v[6] = {0}; + + if (!val || strlen(val) >= sizeof(buf)) + return -EINVAL; + strscpy(buf, val, sizeof(buf)); + { + char *p = buf; + + while (nf < 6 && p && *p) { + char *c = strchr(p, ','); + + if (c) + *c++ = '\0'; + if (kstrtoull(p, 0, &v[nf])) + return -EINVAL; + nf++; + p = c; + } + } + if (nf < 4) + return -EINVAL; + svc_screen_blit_gpa = v[0]; + svc_screen_blit_spitch = (u32)v[1]; + svc_screen_blit_w = (u32)v[2]; + svc_screen_blit_h = (u32)v[3]; + svc_screen_blit_doff = nf > 4 ? v[4] : SVC_SCREEN_BLIT_DEF_OFF; + svc_screen_blit_dpitch = nf > 5 ? (u32)v[5] : + SVC_SCREEN_BLIT_DEF_PITCH; + if (!svc_screen_blit_gpa || !svc_screen_blit_w || + !svc_screen_blit_h) + return -EINVAL; + if (svc_screen_blit_w > 4096 || svc_screen_blit_h > 4096 || + svc_screen_blit_spitch > 0x10000 || + svc_screen_blit_spitch < svc_screen_blit_w * 4 || + svc_screen_blit_dpitch > 0x10000 || + svc_screen_blit_dpitch < svc_screen_blit_w * 4 || + (u64)svc_screen_blit_spitch * svc_screen_blit_h > (64u << 20)) + return -EINVAL; + schedule_work(&svc_screen_blit_work); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_screen_blit_ops = { + .set = nv_vgpu_sema_screen_blit_set, +}; + +module_param_cb(sema_screen_blit, &nv_vgpu_sema_screen_blit_ops, NULL, 0200); +MODULE_PARM_DESC(sema_screen_blit, + "Blit a guest-RAM frame into the scanned vidmem surface: src_gpa,src_pitch,w,h[,dst_off,dst_pitch] (session 153 CONT.44, dev present emulation)"); + + +/* + * Close the open section: synthesize the QMD from the collected fields and + * capture it as the forwarding candidate. The QMD word layout mirrors + * svc_gr_qmd_decode() (the CUDA bridge contract): 48/49 program, 32/33 + * parameter + size, 23..25 release, 34..47 pointer pairs, qmd[4] b10 + * REL_ENABLE. Every VA is copied from a decoded operand; qmd_dev_va stays 0 + * (synthesized - no QMD buffer exists in guest memory) so the backend uses + * its qmdwin/scratch path. + */ + +/* + * SESSION 153-CONTINUED-45 (task 1): big-window verbatim replay. + * + * The direct GL client (glcore, channel family 0xbeef) renders through + * 5016-7144 B pushes whose C597 section window is 1253-1470 dwords - far + * beyond the backend's 448-dword method-block window (any window slice goes + * verbatim through the module pushbuffer page). The engine executes 3D + * methods from ANY number of sequential ring entries with channel state + * persisting across them (that is how real drivers split work), and the + * forward pump already forwards ONE candidate per pass FIFO per channel + * (svc_gr_request picks THIS channel's oldest unconsumed candidate). So the + * window is split into method-aligned chunks of <=448 dwords, each enqueued + * as its own candidate in ring order; successive passes replay them in + * order and the physical channel accumulates the section's state, exactly + * as if the one big window had executed. + * + * Boundary rule: a chunk ends at the last METHOD boundary that fits the + * budget - methods are never split across chunks (a run's data words would + * otherwise be orphaned). The QMD/rel/inline records are cloned from the + * section's base candidate; only stream_off/stream_n differ, so every + * chunk launch carries the same release identity and program images. + */ +#define SVC_GL_STREAM_CHUNK_MAX 6 +/* + * Words per chunk: 384 is the largest window the physical engine is + * PROVEN to retire (r997: `3D stream marked h=0x1000013 + * push=0x120010004 window=384 words` -> `GUEST GR LAUNCH OK`), + * while the client's 394-word first window schedule-stalled its + * replay (r1030: grfifo frozen at 1, fecs idle, timeout + software + * release). Stay under the proven bound. + * SESSION 153-CONTINUED-45: runtime-settable for the bisect - the + * /sys/module/vgsp/parameters/sema_gl_chunk_words knob overrides it so a + * live client's window can be cut to 64/128/256 words to find the first + * length (or starting method) that schedules and retires. + */ +static unsigned svc_gl_chunk_words_param; +module_param_named(sema_gl_chunk_words, svc_gl_chunk_words_param, uint, 0644); +MODULE_PARM_DESC(sema_gl_chunk_words, + "3D stream chunk size in words for the chunked verbatim replay (0 = built-in 384)"); +#define SVC_GL_STREAM_CHUNK_WORDS 384u + +static u32 svc_gl_chunk_boundary(const u8 *buf, u32 len, u32 start, + u32 at_off, u32 max_words) +{ + u32 off = start; + u32 limit = start + max_words * 4u; + u32 last = start; + + while (off + 4 <= at_off && off + 4 <= len) { + u32 hdr, op, cnt, nxt; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 0x7; + cnt = (hdr >> 16) & 0xfff; + if (op == 4 || op == 7 || !cnt) + nxt = off + 4; + else if ((op == 1 || op == 3 || op == 5) && cnt <= 0xfff && + off + 4 + cnt * 4u <= len && + off + 4 + cnt * 4u <= at_off) + nxt = off + 4 + cnt * 4u; + else if ((op == 1 || op == 3 || op == 5) && cnt <= 0xfff && + off + 4 + cnt * 4u > at_off) + /* + * SESSION 153-CONTINUED-45: a run whose data the + * window edge would cut MUST NOT be a boundary - + * ending a chunk on its header alone makes the + * PBDMA fault mid-method (r1031 PB-FAULT-DUMP P0 + * hdr=20164010 cnt=22, its 22 data words absent) + * and the whole replay dies silently. Stop before + * it; the orphaned head is left out of the replay. + */ + break; + else + nxt = off + 4; /* resync word, as the decoders */ + if (nxt > limit) + break; + last = nxt; + off = nxt; + } + if (last == start) { + /* One method alone overruns the budget: take what fits so the + * window makes progress instead of wedging. */ + last = start + max_words * 4u; + if (last > at_off) + last = at_off; + if (last <= start) + last = start + 4u <= at_off ? start + 4u : at_off; + } + return last; +} + +static void svc_gl_stream_chunks(struct svc_channel *ch, const u8 *buf, + u32 len, u32 at_off) +{ + u32 cur = svc_gl.start_off; + u32 n = 0, pending = 0, ri; + + if (!svc_gr_cand.valid) + return; + /* + * Backpressure: one frame's window is self-contained (it re-emits the + * whole state), so while the previous frame's chunks are still being + * replayed there is no reason to queue another window - it would only + * evict in-flight chunks and execute a torn state sequence. Count + * this channel's unconsumed candidates (including the base just + * captured); at >= 8 (a full frame's 4 chunks plus slack) the new + * capture is dropped whole: the base candidate is invalidated so the + * pass-end request cannot forward it down the compute path either. + */ + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered && + svc_gr_cand_ring[ri].h_object == ch->h_object) + pending++; + if (pending >= SVC_GR_CAND_RING) { + if (gsp_sema_dbg && svc_gl_mark_log < 64) { + svc_gl_mark_log++; + pr_info("sema-gl: stream chunks backpressure h=0x%x push=0x%llx pending=%u -> capture dropped\n", + ch->h_object, svc_gr_cand.push_va, pending); + } + svc_gr_cand.valid = false; + return; + } + while (cur + 8u <= at_off && n < SVC_GL_STREAM_CHUNK_MAX) { + u32 cw = svc_gl_chunk_words_param ? + svc_gl_chunk_words_param : SVC_GL_STREAM_CHUNK_WORDS; + u32 end; + + if (cw > 448u) + cw = 448u; + end = svc_gl_chunk_boundary(buf, len, cur, at_off, cw); + struct svc_gr_cand_rec *c = &svc_gr_cand; + + if (n) { + u32 ri, found = SVC_GR_CAND_RING; + + for (ri = 1; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % + SVC_GR_CAND_RING; + + if (!svc_gr_cand_ring[rp].valid || + svc_gr_cand_ring[rp].offered) { + found = rp; + break; + } + } + if (found == SVC_GR_CAND_RING) { + pr_info("sema-gl: stream chunks: no free ring slot after %u chunks h=0x%x push=0x%llx (window truncated)\n", + n, ch->h_object, svc_gr_cand.push_va); + break; + } + svc_gr_cand_ring[found] = svc_gr_cand; + c = &svc_gr_cand_ring[found]; + } + c->valid = true; + c->offered = false; + c->stream_3d = true; + c->stream_off = cur / 4u; + c->stream_n = (end - cur) / 4u; + n++; + cur = end; + } + if (n) { + pr_info("sema-gl: 3D stream chunked h=0x%x push=0x%llx window=%u words -> %u chunk(s) (verbatim replay in ring order; method-aligned)\n", + ch->h_object, svc_gr_cand.push_va, + (at_off - svc_gl.start_off) / 4u, n); + } + /* + * SESSION 153-CONTINUED-46: the first chunk's full method stream. + * Every beef chunk launch stalls the physical front-end (grfifo + * frozen, fecs_cur=0, PBDMA stopped mid-block at a + * SET_VERTEX_ATTRIBUTE_A run while X windows retire) and the window + * tail (`3DSTREAM ... tail=`) alone cannot name the stalling method. + * Dump the first chunk's words (up to 96) as operand lines so the + * exact method list is on record; bounded per boot. + */ + { + static u32 chunkdump_log; + + if (chunkdump_log < 8 && svc_gl.start_off + 8u <= at_off) { + u32 cur2 = svc_gl.start_off, end2, k2, wcnt; + u32 cw2 = svc_gl_chunk_words_param ? + svc_gl_chunk_words_param : + SVC_GL_STREAM_CHUNK_WORDS; + + chunkdump_log++; + if (cw2 > 448u) + cw2 = 448u; + end2 = svc_gl_chunk_boundary(buf, len, cur2, at_off, + cw2); + wcnt = (end2 - cur2) / 4u; + if (wcnt > 96u) + wcnt = 96u; + pr_info("sema-gl: CHUNK1 h=0x%x push=0x%llx off=0x%x words=%u (dump up to 96)\n", + ch->h_object, svc_gr_cand.push_va, cur2, + (end2 - cur2) / 4u); + for (k2 = 0; k2 + 4 <= wcnt; k2 += 4) + pr_info("sema-gl: CHUNK1[%02u] %08x %08x %08x %08x\n", + k2, + get_unaligned_le32(buf + cur2 + 4u * k2), + get_unaligned_le32(buf + cur2 + 4u * (k2 + 1)), + get_unaligned_le32(buf + cur2 + 4u * (k2 + 2)), + get_unaligned_le32(buf + cur2 + 4u * (k2 + 3))); + for (; k2 < wcnt; k2++) + pr_info("sema-gl: CHUNK1[%02u] %08x\n", k2, + get_unaligned_le32(buf + cur2 + 4u * k2)); + } + } +} + +static void svc_gl_flush(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, + const struct svc_push_state *st, const u8 *buf, + u32 len, u64 push_va, bool trusted, bool replay, + u32 at_off) +{ + struct svc_push_state s2; + /* + * SESSION 153-CONTINUED-88: static (was stack). With the inline + * record grown to 2048 B the 8-record array is ~16 KB - too much for + * a kernel stack; this function runs on the single service thread + * (same reason as the static recs[] in svc_gr_forward and the ops + * array in svc_decode_stream), so a static array is safe. + */ + static struct gsp_bridge_inline prs[SVC_GL_PROG_MAX]; + u32 n_recs = 0, k; + /* + * SESSION 149 R16 (M3): the inline attachment's source. Priority: + * the REAL program bytes read from prog_va through the trusted walk + * (the GL program page), else the observed LID payload bounded to its + * own run. r888's blob=64 over-read the 12 B LID run. + */ + u8 blob[SVC_GR_INLINE_BYTES]; + u32 blob_n = 0; + int blob_rc = -1; + bool blob_trusted = false; + const char *blob_src = "none"; + const char *op_src = "stream"; + + /* + * SESSION 153 CONT.6 (task 1): record every section arrival (any + * trusted or replay state, before any early exit) so a content + * change after init cannot hide behind the bounded capture logs. + */ + if (gsp_sema_dbg) + svc_gl_arrival_note(push_va, buf, len); + svc_gl_prog_apply_param(); + if (!svc_gl.seen || !trusted) + goto out; + { + u64 hp = 0, hm = 0, hr = 0; + u32 hms = 0, hrp = 0; + bool held = false; + + if (!svc_gl.prog_va || !svc_gl.param_va || !svc_gl.rel_va) { + held = svc_gl_hold_load(ch->h_object, &hp, &hm, &hms, + &hr, &hrp); + if (!svc_gl.prog_va && held) { + svc_gl.prog_va = hp; + op_src = "hold"; + } + if (!svc_gl.param_va && held) { + svc_gl.param_va = hm; + svc_gl.param_size = hms; + op_src = "hold"; + } + if (!svc_gl.rel_va && held) { + svc_gl.rel_va = hr; + svc_gl.rel_payload = hrp; + op_src = "hold"; + } + if ((!svc_gl.prog_va || !svc_gl.param_va) && + gsp_sema_gl_draw_synth && !ch->engine_type) { + if (!svc_gl.prog_va) + svc_gl.prog_va = SVC_GL_SYNTH_PROG_VA; + if (!svc_gl.param_va) { + svc_gl.param_va = SVC_GL_SYNTH_PARAM_VA; + svc_gl.param_size = + SVC_GL_SYNTH_PARAM_SIZE; + } + op_src = "draw-synth"; + } + if (!svc_gl.rel_va && gsp_sema_gl_draw_synth && + !ch->engine_type) { + svc_gl.rel_va = SVC_GL_SYNTH_REL_VA; + svc_gl.rel_payload = SVC_GL_SYNTH_REL_PAY; + op_src = "draw-synth"; + } + if (!svc_gl.prog_va || !svc_gl.param_va || + !svc_gl.rel_va) { + /* + * SESSION 153-CONTINUED-45 (task 1): the + * DIRECT GL client's (glcore) per-frame draw + * sections bind a program (SET_PIPELINE_ + * PROGRAM_ADDRESS) and carry the bracketing + * release, but no tex-header pool - the + * sticky pool was bound in an earlier push. + * The verbatim 3D replay path never reads + * param_va (the synthesized compute fields + * are what stream_3d bypasses), so a + * prog+rel section whose window fits the + * replay method block is capturable as a + * real stream window with param=0. Bounded + * to the GL stream lane and the fit so the + * historical skip still guards everything + * else; CONT-44 evidence: every client draw + * flush-skipped here with prog/rel real + * (`flush skip h=0xbeef0100 ... prog=0x4390900 + * param=0x0 rel=0x519c0a0`). + */ + bool stream_noparam = false; + + /* + * SESSION 153-CONTINUED-45: the push page is + * cyclically zeroed/rewritten by the client + * (watched live: real -> ZERO -> new content + * on a ~6 s cadence); a capture landing in the + * zero window decodes garbage (prog found + * deeper, window head all zeros) and its + * replay makes the PBDMA fault on the zero + * lead (r1031 PB-FAULT-DUMP P0). Only capture + * when the window head is a real word - a + * later pass/entry on the filled page captures + * the frame. + * + * SESSION 153-CONTINUED-67: the w2/r1127 + * zero-head refusals were NOT mid-write + * windows that later filled: the same word at + * the same offset read zero across every pass + * of the boot (64 refusals, census content + * stable at 0x1f8..0x614), because the window + * start had been armed on a zero word (a + * no-op INC(0) slot) in the gap after the last + * foreign-subchannel run. A zero word is not a + * method: advance past the leading zero words + * to the first real word and capture the + * window from there. Only a window that is + * ENTIRELY zero is still refused (the r1031 + * protection). + */ + if (at_off > svc_gl.start_off && + svc_gl.start_off + 4u <= len) { + u32 w0h; + + memcpy(&w0h, buf + svc_gl.start_off, 4); + if (!w0h) { + u32 adv = svc_gl.start_off; + u32 lim = min_t(u32, at_off, + len); + + while (adv + 4u <= lim) { + memcpy(&w0h, + buf + adv, 4); + if (w0h) + break; + adv += 4u; + } + if (!w0h) { + if (gsp_sema_dbg && + svc_gl_np_log < 64) { + svc_gl_np_log++; + pr_info("sema-gl: zero-head window h=0x%x push=0x%llx start=0x%x win=%u words (whole window zero; not captured)\n", + ch->h_object, + push_va, + svc_gl.start_off, + (at_off - svc_gl.start_off) / 4u); + } + goto out; + } + if (svc_gl_np_log < 64) { + svc_gl_np_log++; + pr_info("sema-gl: zero-lead window h=0x%x push=0x%llx start=0x%x->0x%x win=%u words (skipped zero no-ops; capturing)\n", + ch->h_object, + push_va, + svc_gl.start_off, + adv, + (at_off - adv) / 4u); + } + svc_gl.start_off = adv; + } + } + /* + * SESSION 153-CONTINUED-46: the admission + * bound uses the BUILT-IN 384-word chunk + * budget, never the sema_gl_chunk_words + * bisect knob. r1033 evidence: with the knob + * at 64 the bound shrank to 6*64=384 words + * and the client's real 1253-word windows + * were rejected outright ("flush skip ... + * win=1253 words (no operands, no hold)") - + * the knob is meant to cut the replayed + * chunks smaller, not to stop the capture. + * With the full bound every real window is + * admitted and chunked (the knob then + * truncates the replayed words, logged). + */ + if (svc_gl.prog_va && svc_gl.rel_va && + !svc_gl.param_va && + gsp_sema_gl_xlate && !ch->engine_type && + op_src[0] != 'd' && + at_off > svc_gl.start_off && + (at_off - svc_gl.start_off) / 4u <= + SVC_GL_STREAM_CHUNK_MAX * + SVC_GL_STREAM_CHUNK_WORDS) + stream_noparam = true; + if (!stream_noparam) { + /* + * SESSION 153-CONTINUED-13: the release may be + * decoded AFTER this section in ring order + * (the r955 init: the 24 B release pushes sit + * behind the 1524 B init push in the ring, so + * every replay pass of the section flushed + * before svc_gl.rel_va was known). When the + * operands are real but only the release is + * missing, stash them for the channel so the + * next section/pass (by then the constant + * release stream has been decoded) captures + * with src=hold instead of losing the init + * capture - without which the guest's GLX + * init completion degrades and the client + * never emits its draw pushes at all. + */ + if (svc_gl.prog_va && svc_gl.param_va && + !svc_gl.rel_va) + svc_gl_hold_store(ch->h_object, + svc_gl.prog_va, + svc_gl.param_va, + svc_gl.param_size, + 0, 0); + /* + * SESSION 153-CONTINUED-18 (task 1): replace + * the old "flush skip" drop with NOTIFY + * EXECUTION when this push carried the kick + * member (SET_NOTIFY_A/B + NOTIFY in the same + * push). The section has no draw operands - + * it is the client's completion kick - so the + * GPU's notification write is the only work + * it asks for. + */ + if (gsp_sema_kick_notify && + svc_gl.kick_notify_seen && + svc_gl.kick_notify_va && + svc_kick_notify_exec(vgpu, ch, + svc_gl.kick_notify_va, + svc_gl.kick_notify_val, + push_va)) { + /* authored once; a replay pass must not + * re-author an unchanged member. */ + svc_gl.kick_notify_seen = false; + svc_gl.kick_notify_va = 0; + goto out; + } + if (gsp_sema_dbg && svc_gl_skip_log < 2048) { + svc_gl_skip_log++; + pr_info("sema-gl: flush skip h=0x%x push=0x%llx prog=0x%llx param=0x%llx rel=0x%llx win=%u words (no operands, no hold)\n", + ch->h_object, push_va, + svc_gl.prog_va, svc_gl.param_va, + svc_gl.rel_va, + at_off > svc_gl.start_off ? + (at_off - svc_gl.start_off) / 4u : 0u); + } + goto out; + } else if (gsp_sema_dbg && svc_gl_np_log < 64) { + svc_gl_np_log++; + pr_info("sema-gl: stream capture no-param h=0x%x push=0x%llx prog=0x%llx rel=0x%llx win=%u words (real draw operands; pool param bound elsewhere)\n", + ch->h_object, push_va, + svc_gl.prog_va, svc_gl.rel_va, + (at_off - svc_gl.start_off) / 4u); + } + } + } + } + /* + * SESSION 149 R15 (M1.1a): admit a rewind replay when the same slot + * re-read valid bytes whose content hash matches the recorded pass + * (the pump sets cap_trust only for such a hash-verified replay; see + * the trust admission at the decode call site). The exactly-once guard + * for GL capture is GL-specific (SESSION 153): svc_gr_cand_push_seen() + * also matches the GR candidate ring and the session seen table, and + * the GL client's C597 draw pushes are admitted as GR candidates + * during driver init - so using it here suppressed every later GL + * section capture on those push VAs ("replay already captured" for + * every client draw). svc_gl_cap_seen() only matches pushes whose GL + * section was actually captured, with the content hash so a later + * different push in the same slot (a frame loop) stays capturable. + */ + if (replay && svc_gl_cap_seen(push_va, svc_push_hash(buf, len))) { + if (svc_gl_rep_log < 16) { + svc_gl_rep_log++; + pr_info("sema-gl: replay already captured h=0x%x push=0x%llx (no re-capture)\n", + ch->h_object, push_va); + } + goto out; + } + s2 = *st; + memset(s2.qmd, 0, sizeof(s2.qmd)); + s2.qmd[48] = (u32)svc_gl.prog_va; + s2.qmd[49] = ((u32)(svc_gl.prog_va >> 32)) & 0x1ffff; + s2.qmd[32] = (u32)svc_gl.param_va; + s2.qmd[33] = (((u32)(svc_gl.param_va >> 32)) & 0x1ffff) | + ((svc_gl.param_size >> 4) << 19); + s2.qmd[23] = (u32)svc_gl.rel_va; + s2.qmd[24] = ((u32)(svc_gl.rel_va >> 32)) & 0xff; + s2.qmd[25] = svc_gl.rel_payload; + s2.qmd[4] |= 1u << 10; /* REL_ENABLE */ + /* + * qmd[24] b31 (REL_ONE_WORD) stays clear on the first launch: the + * backend then redirects release0 to the module semaphore and the + * sema service authors the guest word (set it once rel_hpa resolving + * is wanted). qmd[24] b26 (REDUCTION) likewise stays clear. + */ + /* + * SESSION 149 R16 (M3): the scheduling fields the r888 engine + * refused to schedule without. Word/field offsets from the public + * QMDV02_03 class header (nvidia/inc/class/clc5c0qmd.h; the booter's + * own 48/49 program pair matches its QMDV02_03 offsets, so the + * rest of the layout is the same class): + * w12 CTA_RASTER_WIDTH MW(415:384) + * w13 CTA_RASTER_HEIGHT MW(431:416) + * w14 CTA_RASTER_DEPTH MW(463:448) + * w18 QMD_VERSION MW(579:576), QMD_MAJOR_VERSION MW(583:580), + * CTA_THREAD_DIMENSION0 MW(607:592) -> w18 hi word + * w19 CTA_THREAD_DIMENSION1 MW(623:608), + * CTA_THREAD_DIMENSION2 MW(639:624) + * w20 REGISTER_COUNT_V MW(656:648) <- SKEDCHECK06 field + * w29 BARRIER_COUNT MW(959:955) + * w31 SASS_VERSION MW(1023:1016) + * SKEDCHECK16 fired on CTA_THREAD_DIMENSION0==0 and SKEDCHECK06 on + * REGISTER_COUNT_V==0. The register count is the stream's own value: + * REGISTER_COUNT(1) = 0x1a = 26 (method 0x200c + 1*0x40 = 0x204c, + * words [0x1a, 0]) - the same number the driver's C5C0 path writes + * ("a NULL QMD will result in SKEDCHECK06_REGISTER_COUNT", cui + * turing_launch.c:781). CTA dims: the 0xC597 stream carries no grid / + * CTA operands we can locate (the 3D-class runs are format/viewport/ + * shader config; the C597 SHADER_CONFIG "CTA raster" words are not in + * this push), so a minimal 1x1x1 grid of 1x1x1 CTAs is used and + * documented as the choice: the checks only reject ZERO dims and a + * zero total-thread count, both of which a 1x1x1 launch satisfies. + * SHARED_MEMORY_SIZE (w17) stays 0 = "no shared memory" (valid and + * NOT among the failed checks); the m72 local-memory prelude already + * covers SKEDCHECK05. The driver's cache-control block (w5 top bits) + * is set as every captured guest QMD carries it; the backend turns it + * into its icache-invalidate bit for the shadow line. + */ + s2.qmd[12] = 1; /* CTA raster width (CTAs) */ + s2.qmd[13] = 1; /* CTA raster height (CTAs) */ + s2.qmd[14] = 1; /* CTA raster depth (CTAs) */ + s2.qmd[18] = 0x23u | (1u << 16); /* QMD 2.3, CTA dim0 = 1 */ + s2.qmd[19] = 1u | (1u << 16); /* CTA dim1 = 1, dim2 = 1 */ + s2.qmd[20] = 0x1au << 8; /* REGISTER_COUNT_V = 26 */ + s2.qmd[29] |= 1u << 27; /* BARRIER_COUNT = 1 */ + s2.qmd[31] |= 0x75u << 24; /* SASS_VERSION 7.5 (sm_75) */ + s2.qmd[5] |= 0x3c000000u; /* driver cache-control block */ + for (k = 0; k < svc_gl.n_ptr && k < SVC_GR_POINTER_MAX - 1; k++) { + s2.qmd[34 + 2 * k] = (u32)svc_gl.ptr[k]; + s2.qmd[35 + 2 * k] = ((u32)(svc_gl.ptr[k] >> 32)) & 0x1ffff; + } + s2.qmd_dev_va = 0; + s2.qmd_hdr = 0; + s2.qmd_end = 0; + s2.qmd_have = 0; + /* + * SESSION 149 R15 (M2-S1) + R16 (M3): attach program-page content as + * an inline record at the synthesized program VA. The GL program page + * (0x121201b00) is NOT among the section's own upload destinations, + * so without this the backend's w48 code-source screen sees an empty + * page, withdraws to the guest VA and the launch dies at FAULT_PDE; + * with it the proven win0 shadow copy + w48/49 rewrite line runs. + * + * R16 source order: the REAL program bytes read from prog_va through + * the trusted walk (the read machinery translates GL VAs now; src=4 + * answered this channel's ring/pushes in r888), else the observed + * LID payload bounded to its own run (12/24/28 B - "materialize + * exactly that"). A trusted read that lands on an all-zero page is + * treated as "not real" and falls through to the LID bytes. + */ + /* + * SESSION 149 R17 (M4): apply the section's memory-writing upload + * commands BEFORE reading the program page, so an upload that + * targets the program VA is visible to the read below (the section's + * engine commands never ran: every destination - constants, tex + * pools, program - stayed zero). The shared inline-write path + * resolves each destination (FB aperture -> emulated vidmem shadow, + * sysmem -> guest RAM, unresolved -> VA shadow) and the once-only + * guard is keyed by (push, content hash, dst, bytes) so a + * replay/rescue pass cannot double-apply. + */ + { + u64 ph = svc_push_hash(buf, len); + u32 ap_n = 0, ap_b = 0; + + for (k = 0; k < svc_gl.n_up && k < SVC_GL_UPLOAD_MAX; k++) { + struct svc_gl_up *u = &svc_gl.up[k]; + + if (svc_inline_once_seen(push_va, ph, u->dst_va, + u->len)) + continue; + svc_inline_once_store(push_va, ph, u->dst_va, u->len); + if (!svc_grce_inline_write(vgpu, u->dst_va, u->data, + u->len, ch->h_object)) { + ap_n++; + ap_b += u->len; + } + } + if (gsp_sema_dbg && svc_gl_up_log < 8) { + svc_gl_up_log++; + pr_info("sema-gl: uploads applied=%u/%u bytes=%u h=0x%x push=0x%llx\n", + ap_n, svc_gl.n_up, ap_b, ch->h_object, + push_va); + } + } + memset(blob, 0, sizeof(blob)); + /* + * SESSION 153-CONTINUED-10: the explicit operator program override + * (sema_gl_prog_hex) wins over the trusted page read. The GLX + * server's per-frame CONTENT pushes bind their own program slots + * (observed 0x121201f00 / 0x121215280) whose pages hold unrelated + * nonzero bytes at that offset; the R16 trusted-read priority + * accepts any nonzero 16 bytes as "the real program" and attached + * garbage, and the engine rejected the draw with "Graphics + * Exception: Shader Program Header 11/18 Error" (r944 fault + * evidence, push 0x120014578). An unset override keeps the + * historical read priority unchanged. + */ + if (svc_gl_prog_len) { + blob_n = svc_gl_prog_len; + memcpy(blob, svc_gl_prog, blob_n); + blob_src = "prog-hex"; + } else { + blob_rc = svc_read_va_trusted(vgpu, svc_gl.prog_va, blob, + sizeof(blob)); + if (!blob_rc) { + u32 z = 0; + + for (k = 0; k < 16; k++) + z |= blob[k]; + if (z) + blob_trusted = true; + } + if (blob_trusted) { + blob_n = sizeof(blob); + blob_src = "trusted-prog"; + } else if (svc_gl.blob_len) { + blob_n = svc_gl.blob_len; + memcpy(blob, svc_gl.blob, blob_n); + blob_src = "lid-fallback"; + } + } + /* + * r929 (Path A): a stream-derived capture is a 3D method window - the + * backend replays the raw push verbatim and the engine runs the real + * uploads/binds, so the synthesized compute attachments (the inline + * program record, and with it the prog-hex fallback) do not apply. + * Draw-synth/hold captures keep the old compute path unchanged. + * + * SESSION 153-CONTINUED-11: exception - the GL program-bytes + * delivery. The engine validates the SPH of EVERY pipeline program + * the channel state binds when a draw issues, and the driver's + * upload of those bytes never ran (the program pages read empty). + * For a window that binds program slots (the 0x2014/0x2018 pairs + * collected above) attach one inline record per program VA with the + * stage-typed image (vs-hex for VTG slots, ps-hex for pixel slots, + * prog-hex fallback) - this authors the missing upload. Windows + * without program pairs keep the historical single-record path. + */ + if (svc_gl.n_progs) { + u32 pi; + + for (pi = 0; pi < svc_gl.n_progs && n_recs < GSP_BRIDGE_INLINE_MAX; + pi++) { + const u8 *b = NULL; + u32 bl = 0; + u8 t = svc_gl_shader_type[svc_gl.prog_slot[pi] < 32 ? + svc_gl.prog_slot[pi] : 0]; + /* + * SESSION 153-CONTINUED-88: deliver the REAL captured + * upload payload to the engine-visible page. The + * window's own program uploads (the client's 0x180 + * LINE_LENGTH_IN + 0x1b0 LOAD_INLINE_DATA runs - + * observed 1536 B at 0x4350900 and 1408 B at + * 0x4350f00, w0=0x020a0481 = the driver's real SPH + * V4/VTG image) were captured at decode time into the + * destination ring but landed only in the BAR1 shadow; + * the engine validates the SPH of every bound program + * at the FILL-resolved page. Attach the real payload + * as an inline record at its own destination VA - the + * backend authors it into the engine-visible page on + * the stream3d path - and let it win over the minimal + * authored image below. Exact-VA match: the bind VA + * IS the upload destination. + */ + { + u32 rlen = 0; + u32 ri0 = n_recs; + + memset(&prs[n_recs], 0, sizeof(prs[n_recs])); + if (svc_grce_dst_newest(svc_gl.progs[pi], 128u, + &rlen, prs[n_recs].data, + (u32)sizeof(prs[n_recs].data))) { + u64 d = svc_gl.progs[pi]; + u32 head = 4096u - (u32)(d & 0xfff); + + if (rlen <= head) { + prs[n_recs].dst_va = d; + prs[n_recs].len = rlen; + n_recs++; + } else { + /* + * The payload crosses its + * 4K page (observed: the + * 1408 B image at + * 0x4350f00). The backend + * applies a record only + * within one page, so + * split: page remainder + + * tail at the next page's + * VA (skipped by the + * backend when that page + * is not in the request - + * the SPH head still + * lands). + */ + prs[n_recs].dst_va = d; + prs[n_recs].len = head; + n_recs++; + if (n_recs < + GSP_BRIDGE_INLINE_MAX) { + memset(&prs[n_recs], 0, + sizeof(prs[n_recs])); + prs[n_recs].dst_va = + (d & ~(u64)0xfff) + + 0x1000; + prs[n_recs].len = + rlen - head; + memcpy(prs[n_recs].data, + prs[n_recs - 1].data + head, + rlen - head); + n_recs++; + } + /* + * SESSION 153-CONTINUED-88: + * the tail record is applied + * only when its page is in the + * launch's page set. Add the + * tail page as a candidate + * operand page (the engine + * will fetch the program's + * second half from it), so + * the request carries it and + * the split delivery is + * complete instead of + * head-only. + */ + svc_gl_add_ptr((d & ~(u64)0xfff) + + 0x1000); + } + if (gsp_sema_dbg && + svc_gl_prog_log < 64) { + svc_gl_prog_log++; + pr_info("sema-gl: real-upload record va=0x%llx len=%u first=%08x (engine-visible delivery)\n", + d, rlen, + get_unaligned_le32(prs[ri0].data)); + } + continue; + } + } + /* + * SESSION 153-CONTINUED-87: the REAL program upload + * wins. When the window carries its own upload of a + * program-sized payload (>= 128 B) to this same 4K + * page, the minimal authored image must NOT overwrite + * the driver's bytes at the engine-visible page (the + * upload now lands there through the pin-aware + * svc_grce_inline_write). Keep an explicit image only + * where no real upload exists (the synthesized stage + * binds below). + */ + { + u32 uu; + bool real_up = false; + + for (uu = 0; uu < svc_gl.n_up; uu++) { + if (svc_gl.up[uu].len < 128u) + continue; + if ((svc_gl.up[uu].dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1)) == + (svc_gl.progs[pi] & + ~(u64)(SEMA_PAGE_SIZE - 1))) { + real_up = true; + break; + } + } + if (real_up) { + if (gsp_sema_dbg && + svc_gl_prog_log < 64) + pr_info("sema-gl: real-upload wins va=0x%llx (authored image skipped)\n", + svc_gl.progs[pi]); + continue; + } + } + + if (t == 5 && svc_gl_ps_len) { + b = svc_gl_ps; + bl = svc_gl_ps_len; + } else if (t != 5 && t != 0xff && svc_gl_vs_len) { + b = svc_gl_vs; + bl = svc_gl_vs_len; + } else if (svc_gl_prog_len) { + b = svc_gl_prog; + bl = svc_gl_prog_len; + } else if (blob_n) { + b = blob; + bl = blob_n; + } + if (!b || !bl) + continue; + memset(&prs[n_recs], 0, sizeof(prs[n_recs])); + prs[n_recs].dst_va = svc_gl.progs[pi]; + prs[n_recs].len = bl; + memcpy(prs[n_recs].data, b, bl); + n_recs++; + } + /* + * SESSION 153-CONTINUED-87: author the program images for the + * client's missing stage binds. The replayed window binds only + * the j=1 (VTG) pair; the booter emits the client's own + * TESS_CONTROL (j=2, family+0x200) and PIXEL (j=5, family+0x0) + * binds ahead of the window (sema_gl_sph_bind), so their SPH + * bytes must exist at those VAs too - otherwise the DS fetches + * an empty/foreign page and still latches its header error. + * Observed family layout (guest-RAM scrape of the init push + * and the window itself): 0x4400000 j=5, 0x4400200 j=2, + * 0x4400300/0x4400900 j=1 - all in the window prog VA's own + * 4K page. The TESS image is the VTG image with the SPH + * SHADER_TYPE field (bits 13:10) re-pointed to 2, exactly + * like the driver's own placeholder bind. Beef client + * channels only, and only when the window does not already + * bind that exact VA. + */ + if (gsp_phys_rmcall_gr_sph_bind_value() && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + svc_gl.progs[0]) { + u64 fam = svc_gl.progs[0] & + ~(u64)(SEMA_PAGE_SIZE - 1); + u32 k; + + for (k = 0; k < 2u && n_recs < SVC_GL_PROG_MAX; k++) { + u64 dst = fam + (k ? 0x200u : 0x0u); + const u8 *b = k ? svc_gl_vs : svc_gl_ps; + u32 bl = k ? svc_gl_vs_len : svc_gl_ps_len; + bool dup = false; + u32 pi3; + + if (!b || bl < 4u) + continue; + for (pi3 = 0; pi3 < svc_gl.n_progs; pi3++) + if (svc_gl.progs[pi3] == dst) + dup = true; + if (dup) + continue; + memset(&prs[n_recs], 0, sizeof(prs[n_recs])); + prs[n_recs].dst_va = dst; + prs[n_recs].len = bl; + memcpy(prs[n_recs].data, b, bl); + if (k) { + u32 w0 = get_unaligned_le32( + prs[n_recs].data); + + w0 = (w0 & ~(0xfu << 10)) | + (2u << 10); /* TESS_CONTROL */ + put_unaligned_le32(w0, prs[n_recs].data); + } + n_recs++; + if (gsp_sema_dbg && svc_gl_prog_log < 64) { + pr_info("sema-gl: stage-image dst=0x%llx slot=%u len=%u first=%08x\n", + dst, k ? 2u : 5u, bl, + get_unaligned_le32(prs[n_recs - 1].data)); + } + } + } + if (n_recs && gsp_sema_dbg && svc_gl_prog_log < 64) { + u32 pi2; + + svc_gl_prog_log++; + pr_info("sema-gl: prog-images h=0x%x push=0x%llx recs=%u n_progs=%u vs=%u ps=%u gen=%u\n", + ch->h_object, push_va, n_recs, + svc_gl.n_progs, + svc_gl_vs_len, svc_gl_ps_len, svc_gl_prog_len); + for (pi2 = 0; pi2 < svc_gl.n_progs && pi2 < 8; + pi2++) + pr_info("sema-gl: prog-slot[%u] va=0x%llx slot=%u type=%02x\n", + pi2, svc_gl.progs[pi2], + svc_gl.prog_slot[pi2], + svc_gl_shader_type[svc_gl.prog_slot[pi2] < 32 ? + svc_gl.prog_slot[pi2] : 0]); + } + } else if (blob_n && op_src[0] != 's') { + memset(&prs[0], 0, sizeof(prs[0])); + prs[0].dst_va = svc_gl.prog_va; + prs[0].len = blob_n; + memcpy(prs[0].data, blob, blob_n); + n_recs = 1; + } + /* + * SESSION 153: record this GL section capture in the GL-specific + * exactly-once table so only an actual capture suppresses later + * replays of the same (push_va, content hash). + */ + svc_gl_cap_store(push_va, svc_push_hash(buf, len)); + pr_info("sema-gl: section captured h=0x%x push=0x%llx len=%u off=0x%x prog=0x%llx param=0x%llx/0x%x rel=0x%llx/%u ptr=%u blob=%u replay=%u src=%s nprog=%u recs=%u\n", + ch->h_object, push_va, len, at_off, svc_gl.prog_va, + svc_gl.param_va, svc_gl.param_size, svc_gl.rel_va, + svc_gl.rel_payload, svc_gl.n_ptr, svc_gl.blob_len, + replay ? 1u : 0u, op_src, svc_gl.n_progs, n_recs); + /* + * SESSION 153: remember real operands for this channel so a later + * draw-only section on the same channel can still capture. A + * draw-synth capture carries no channel truth and is never stored. + */ + if (op_src[0] != 'd') + svc_gl_hold_store(ch->h_object, svc_gl.prog_va, + svc_gl.param_va, svc_gl.param_size, + svc_gl.rel_va, svc_gl.rel_payload); + /* R16 bounded witness: what the synthesized QMD now schedules and + * where the code bytes came from. */ + if (gsp_sema_dbg && svc_gl_prog_log < 8) { + svc_gl_prog_log++; + pr_info("sema-gl: M3 sched w12=%u w13=%u w14=%u w18=0x%08x w19=0x%08x w20=0x%08x w29=0x%08x w31=0x%08x w5=0x%08x\n", + s2.qmd[12], s2.qmd[13], s2.qmd[14], s2.qmd[18], + s2.qmd[19], s2.qmd[20], s2.qmd[29], s2.qmd[31], + s2.qmd[5]); + pr_info("sema-gl: M3 code src=%s read_rc=%d len=%u lid=%u first=%08x %08x %08x %08x\n", + blob_src, + blob_rc, blob_n, svc_gl.blob_len, + get_unaligned_le32(blob), + get_unaligned_le32(blob + 4), + get_unaligned_le32(blob + 8), + get_unaligned_le32(blob + 12)); + } + svc_gr_cand_capture(ch, &s2, n_recs ? prs : NULL, n_recs, buf, len, + push_va); + /* + * CONT-87: carry the window's own program binds so the launch + * builder can re-emit them ahead of the window (the window's early + * draws precede its binds; see svc_gr_cand_rec). + */ + { + u32 pb; + + svc_gr_cand.n_prog_binds = 0; + for (pb = 0; pb < svc_gl.n_progs && + pb < SVC_GR_CAND_PROG_BINDS; pb++) { + if (!svc_gl.progs[pb]) + continue; + svc_gr_cand.prog_bind_va[svc_gr_cand.n_prog_binds] = + svc_gl.progs[pb]; + svc_gr_cand.prog_bind_slot[svc_gr_cand.n_prog_binds] = + svc_gl.prog_slot[pb]; + svc_gr_cand.n_prog_binds++; + } + } + /* + * r929 (Path A): a stream-derived capture (op_src=stream: the section + * carried its own operands) is a raw 3D method window. Mark it for + * verbatim replay - but only when the WHOLE push survived in raw[] + * (len == push_n*4): a truncated window would feed the engine a + * half-method stream. The window is SLICED from the section's own + * C597 SET_OBJECT (svc_gl.start_off) through the section close: the + * prelude before the bind carries the X server channel's own + * host-sem + 2D/CE ops, whose semaphore VAs belong to that client's + * VM and are unmapped in the replay channel (r929 v1: the first + * host-sem write faulted FAULT_PDE @ 0x1_2000b000 and the method + * walk died before any 3D method). Every VA the sliced window + * references is covered by the candidate's own mapped pages. The + * 448-word cap leaves headroom for the backend's 512-word method + * block (2 bind words + the window + the 8-word release tail). + * Draw-synth/hold captures are never marked. + * + * SESSION 153-CONTINUED-10: op_src=hold is included. The GLX + * server's per-frame CONTENT pushes carry the draw window but no + * program/param operands (the class bind is sticky; no per-push + * SET_OBJECT), so their sections capture with the channel's real + * operands adopted from the hold table (filled by the init + * capture). The window itself is still the REAL push bytes sliced + * from its own first C597 state method to its END - exactly the + * raw 3D method window the r929 replay lane wants; the synthesized + * compute fields are what stream_3d bypasses. + */ + if ((op_src[0] == 's' || op_src[0] == 'h') && svc_gr_cand.valid && + len == svc_gr_cand.push_n * 4 && + svc_gl.start_off + 8u <= at_off && + (at_off - svc_gl.start_off) / 4u <= 448u) { + svc_gr_cand.stream_3d = true; + svc_gr_cand.stream_off = svc_gl.start_off / 4u; + svc_gr_cand.stream_n = (at_off - svc_gl.start_off) / 4u; + pr_info("sema-gl: 3D stream marked h=0x%x push=0x%llx window=%u words @off=0x%x (verbatim replay from the C597 bind)\n", + ch->h_object, push_va, svc_gr_cand.stream_n, + svc_gl.start_off); + } else if ((op_src[0] == 's' || op_src[0] == 'h') && + svc_gr_cand.valid) { + /* + * SESSION 153-CONTINUED-45: a REAL-operand section whose + * window cannot ride the single 448-dword block - split it + * into method-aligned chunks, each a verbatim replay in ring + * order (channel state persists across launches). The raw[] + * window must still hold the WHOLE push (the slices index + * into it), which is why the per-record raw bound was + * widened; if the raw geometry does not hold, the bounded + * diagnostic below names the failing condition (the next + * bound to size). + */ + if (len == svc_gr_cand.push_n * 4 && + svc_gl.start_off + 8u <= at_off) { + svc_gl_stream_chunks(ch, buf, len, at_off); + } else if (gsp_sema_dbg && svc_gl_mark_log < 64) { + svc_gl_mark_log++; + pr_info("sema-gl: 3D mark failed h=0x%x push=0x%llx win_start=0x%x at=0x%x win=%u words raw_ok=%d fit_ok=%d (rawlen=%u pushn=%u len=%u)\n", + ch->h_object, push_va, svc_gl.start_off, + at_off, + at_off > svc_gl.start_off ? + (at_off - svc_gl.start_off) / 4u : 0u, + len == svc_gr_cand.push_n * 4, + svc_gl.start_off + 8u <= at_off && + (at_off - svc_gl.start_off) / 4u <= 448u, + len / 4u, svc_gr_cand.push_n, len); + } + } +out: + /* + * SESSION 149 R15 (M1.1a): preserve the last valid release across + * the reset - it belongs to the channel (decoded from the bracketing + * 24 B release pushes), not to this section; wiping it here and at + * the section open starved the flush guard of rel_va on the trusted + * first pass (the r883/r886 one-shot decode). + */ + { + u64 keep_rel = svc_gl.rel_va; + u32 keep_pay = svc_gl.rel_payload; + + memset(&svc_gl, 0, sizeof(svc_gl)); + svc_gl.rel_va = keep_rel; + svc_gl.rel_payload = keep_pay; + } +} + +/* * Consume the current GR candidate without forwarding it: hand the launch to * the host CUDA bridge when that path is enabled, then clear the candidate. * Exactly-once via `offered`. Called on the service thread (capture/refusal) @@ -4299,12 +10420,43 @@ static bool svc_fwd_defer_arm(void) { if (!gsp_phys_guest_mem_enabled()) return false; - if (svc_gr_cand.valid && gsp_phys_gr_forward_enabled()) - return true; + if (gsp_phys_gr_forward_enabled()) { + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered) + return true; + } return svc_fwd_cand.valid && gsp_phys_work_defer_release_enabled(); } +/* + * r824: the release deferral must be owned by a candidate of THIS push. + * svc_fwd_defer_arm() answers globally (any unoffered candidate of the + * boot's ring); an unrelated lingering candidate then parked the release + * of a push that has no forwarding at all. Observed: the X/GL DMA test's + * progress-tracker release (the kernel's progress semaphore) was recorded + * into svc_fwd_rel and dropped at the next pass's reset, every pass, so + * the semaphore never advanced and Xorg failed at its 10 s wait - while + * every other stage of the chain (latch, NOP acceptance, decode) had + * just started working. A release whose push owns no candidate cannot + * wait for a forwarded copy that will never exist; it applies inline. + */ +static bool svc_fwd_cand_owns_push(u64 push_va) +{ + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered && + svc_gr_cand_ring[ri].push_va == push_va) + return true; + } + return false; +} + /* * P6b wave-1.5 deferred guest releases: every release group the current @@ -4350,6 +10502,106 @@ static u8 svc_fwd_rel_n; static bool svc_fwd_defer_armed; /* + * r633: PARKED push-level release records for a GR launch whose entry was + * HELD before it could be forwarded. The records above are per-decode state + * (reset at the top of every svc_decode_stream); the two GR hold paths + * (busy-defer hold, m150 data-stalled hold) break out of the entry loop + * BEFORE the release block runs, so the entry's records would be lost at + * the next decode - and the eventual forward (queued from the capture ring + * in a later pass, which does not re-decode the entry) would engage the + * release hold with whatever records that pass happens to carry (often + * none, or another entry's), leaving the push's tracking release unapplied + * and the client spinning in cudaDeviceSynchronize forever (the r632 mail + * MALLOC hang: push release 0x20480fff0 pay=112 never landed while the + * client's word stayed at 111). Park keyed by (vgpu, channel, push VA); + * restored into svc_fwd_rel[] by svc_gr_request when the launch is finally + * queued, so the pump's hold-engage holds exactly the queued push's + * records; delivered inline by the paths that consume a capture without a + * forward. Service-thread only. + */ +#define SVC_GRPARK_SLOTS 8 +static struct svc_grpark { + struct nv_vgpu_device *vgpu; + u32 h_object; + u64 push_va; + u8 n; + struct svc_fwd_rel rel[SVC_FWD_REL_MAX]; +} svc_grpark[SVC_GRPARK_SLOTS]; +static u32 svc_grpark_pos; + +static void svc_grpark_put(struct nv_vgpu_device *vgpu, u32 h_object, + u64 push_va, const struct svc_fwd_rel *rel, u8 n) +{ + u32 i, slot = ~0u; + + if (!vgpu || !push_va || !rel || !n) + return; + if (n > SVC_FWD_REL_MAX) + n = SVC_FWD_REL_MAX; + for (i = 0; i < SVC_GRPARK_SLOTS; i++) { + if (svc_grpark[i].vgpu == vgpu && + svc_grpark[i].h_object == h_object && + svc_grpark[i].push_va == push_va) { + slot = i; + break; + } + if (slot == ~0u && !svc_grpark[i].push_va) + slot = i; + } + if (slot == ~0u) + slot = svc_grpark_pos++ % SVC_GRPARK_SLOTS; + svc_grpark[slot].vgpu = vgpu; + svc_grpark[slot].h_object = h_object; + svc_grpark[slot].push_va = push_va; + svc_grpark[slot].n = n; + memcpy(svc_grpark[slot].rel, rel, + sizeof(svc_grpark[slot].rel)); +} + +static bool svc_grpark_get(struct nv_vgpu_device *vgpu, u32 h_object, + u64 push_va, struct svc_fwd_rel *out, u8 *n) +{ + u32 i; + + for (i = 0; i < SVC_GRPARK_SLOTS; i++) { + if (svc_grpark[i].vgpu == vgpu && + svc_grpark[i].h_object == h_object && + svc_grpark[i].push_va == push_va) { + if (out) + memcpy(out, svc_grpark[i].rel, + sizeof(svc_grpark[i].rel)); + if (n) + *n = svc_grpark[i].n; + return true; + } + } + return false; +} + +static void svc_grpark_drop(struct nv_vgpu_device *vgpu, u32 h_object, + u64 push_va) +{ + u32 i; + + for (i = 0; i < SVC_GRPARK_SLOTS; i++) + if (svc_grpark[i].vgpu == vgpu && + svc_grpark[i].h_object == h_object && + svc_grpark[i].push_va == push_va) { + memset(&svc_grpark[i], 0, sizeof(svc_grpark[i])); + return; + } +} + +static void svc_grpark_reset(void) +{ + memset(svc_grpark, 0, sizeof(svc_grpark)); + svc_grpark_pos = 0; +} + +/* r633: bounded evidence that a parked set was restored/delivered. */ +static u32 svc_grpark_log; + +/* * P6b wave 2: positive sentinel returned by svc_copy() when the captured * forwarding candidate's software copy is deliberately withheld (the * hardware-primary path will have the real engine author the data). @@ -4359,8 +10611,448 @@ static bool svc_fwd_defer_armed; #define SVC_COPY_FORWARDED 1 /* Byte copy between two CE addresses (src/dst decoding mirrors svc_fill). */ +/* + * m171 (AGENT-F R2): write/read consistency ledger. A large H2D write to a + * device VA records {va_page -> pa, aperture} for every page it wrote; the + * later read of the same VA (the launch data fill, or a D2H copy) prefers + * the recorded page whenever its own resolution disagrees. The pipeline + * then reads exactly what was written regardless of which of the two + * resolutions is stale - the r174/r175 first-page divergence (the write + * landed on pa X, the fill read a stale page Y, and the engine computed on + * old bytes) can no longer split a write/read pair. Session-scoped: cleared + * with the flush; zero cost while empty. Apertures are stored in the SVC_ + * encoding, the canonical space for the svc accessors. + */ +struct svc_ledger_ent { + u64 va_page; + u64 pa_page; + u32 ap; +}; +/* + * Unified VA->landing-page authority ("R2 ledger"). + * + * m172 grew the original 256-entry ring to 2048, but a ring still evicts + * unrelated VAs wholesale: one 4 MiB H2D plus one launch's data-page + * record set is 4-6 K inserts, so an 8 MiB machine operand's early pages + * were evicted before the launch fill looked them up, and the fill fell + * back to its own (possibly stale) resolution - the exact "write path and + * read path pick different pages" class. + * + * m180: keyed hash table instead of a ring. One entry per VA page, O(1) + * update/lookup, no cross-VA eviction, and the insert of a page already + * present rewrites it in place (a write is a new landing and is always + * the truth). 16384 slots cover a full multi-buffer client's working set + * (8 MB in + 8 MB out + engine records ~ 12 K pages); hash collisions + * only occur between VA pages 64 MiB apart or across clients whose pages + * do not co-reside in time. + * + * COPYAUDIT r220: SVC_LEDGER_MAX/MASK/PROBE and the counters live above + * svc_rebuild (hoisted for the diag line); only the table and the home + * helper stay here. + */ +static struct svc_ledger_ent svc_ledger[SVC_LEDGER_MAX]; + +static inline u32 svc_ledger_home(u64 page) +{ + return (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & + SVC_LEDGER_MASK; +} + +static void svc_ledger_store(u64 va, u64 pa, u32 ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 home = svc_ledger_home(page); + u32 p; + + for (p = 0; p < SVC_LEDGER_PROBE; p++) { + struct svc_ledger_ent *e = &svc_ledger[(home + p) & + SVC_LEDGER_MASK]; + + if (e->va_page == page) { + /* + * A write path moving an existing page: its landing + * is the new truth. Logged (bounded) because a move + * is the write-side half of the DIVERGE class. + */ + if (e->pa_page != pa_page) { + svc_ledger_rewrites++; + if (svc_ledger_rewrites <= 16) + pr_info("sema-ledger: rewrite va=0x%llx old=0x%llx new=0x%llx ap=%u\n", + page, e->pa_page, pa_page, + ap); + } + e->pa_page = pa_page; + e->ap = ap; + return; + } + if (!e->va_page) { + e->va_page = page; + e->pa_page = pa_page; + e->ap = ap; + svc_ledger_n++; + svc_ledger_stores++; + if (svc_ledger_store_log < 64) { + svc_ledger_store_log++; + pr_info("sema-ledger: store va=0x%llx pa=0x%llx ap=%u n=%u\n", + page, pa_page, ap, svc_ledger_n); + } + return; + } + } + /* + * Cluster full (16 same-home VA pages co-resident): replace the home + * slot. Bounded, rare, and only affects VAs whose pages hash to the + * identical home slot. r220: this is a SILENT loss of the evicted + * VA's landing authority (its later readers fall back to the + * resolver, possibly stale), so the first occurrences are logged and + * counted into the diag line. + * + * STALLFREE cap-and-degrade: a sustained eviction burst means the + * table is being thrashed (a pin storm / leftover crashed client + * churning VA pages faster than the working set), and a half-evicted + * table silently strands authority for exactly the evicted pages. + * Past SVC_LEDGER_EVICT_RESET evictions in one session the cache is + * cleared wholesale and rebuilt from live writes: every reader + * uniformly falls back to the resolver until its page is rewritten, + * which removes the stranded-authority class at a known, logged + * cost. The clear is a cache operation only (no control flow, no + * wait); the current store then lands in the empty table. + */ + svc_ledger_evictions++; + svc_ledger_evict_since_reset++; + if (svc_ledger_evict_since_reset >= SVC_LEDGER_EVICT_RESET) { + svc_ledger_evict_since_reset = 0; + svc_ledger_evict_resets++; + pr_warn("sema-ledger: LEDGER-RESET evict=%llu resets=%llu (cluster thrash under churn; cache cleared, readers fall back to the resolver until rewritten)\n", + svc_ledger_evictions, svc_ledger_evict_resets); + svc_ledger_clear(); + } + if (svc_ledger_evictions <= 32) + pr_warn("sema-ledger: EVICT home=%u old_va=0x%llx new_va=0x%llx (cluster full; old landing authority lost until rewritten)\n", + home, (u64)svc_ledger[home].va_page, page); + svc_ledger[home].va_page = page; + svc_ledger[home].pa_page = pa_page; + svc_ledger[home].ap = ap; + svc_ledger_stores++; +} + +static int svc_ledger_lookup(u64 va, u64 *pa, u32 *ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 home = svc_ledger_home(page); + u32 p; + + for (p = 0; p < SVC_LEDGER_PROBE; p++) { + const struct svc_ledger_ent *e = + &svc_ledger[(home + p) & SVC_LEDGER_MASK]; + + if (!e->va_page) + return -ENOENT; + if (e->va_page == page) { + *pa = e->pa_page | (va & (SEMA_PAGE_SIZE - 1)); + *ap = e->ap; + return 0; + } + } + return -ENOENT; +} + +static void svc_ledger_clear(void) +{ + memset(svc_ledger, 0, sizeof(svc_ledger)); + svc_ledger_n = 0; + svc_ledger_rewrites = 0; + svc_ledger_reloc_diverges = 0; + svc_ledger_evict_since_reset = 0; +} + +/* + * COPYAUDIT (2026-10-18 copy-completeness audit): every copy at/above + * SVC_COPY_AUDIT_MIN is logged once with its want/done byte counts, and + * its first/last 16 destination bytes are read back immediately and + * compared against the bytes just written (COPY-MISMATCH on difference). + * COPY-TRUNC marks a copy that failed after partially writing (want>done), + * so a partial transfer can never be silent. Pure diagnostics: no control + * flow, locking or timing change; the readback uses the same accessor the + * write just used. Definitions live with svc_fill (first user). + */ +static void svc_copy_audit_chunk(struct nv_vgpu_device *vgpu, u64 dst_va, + u64 o, const u8 *bounce, size_t chunk, + bool shadow, u64 pa_eff, u32 aper, bool tail) +{ + size_t n = chunk < 16 ? chunk : 16; + size_t so = tail ? chunk - n : 0; /* true end sample */ + u8 rb[16] = {0}; + + if (shadow) { + if (svc_vas_read(dst_va + o + so, rb, n)) + return; + } else if (aper == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + + if (!base || pa_eff + so + n > svc_fb_shadow_size(vgpu)) + return; + memcpy(rb, base + pa_eff + so, n); + } else if (svc_read(vgpu, pa_eff + so, rb, n)) { + return; + } + if (memcmp(rb, bounce + so, n)) { + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x%llx path=ce-soft-copy want=%*phN got=%*phN\n", + dst_va + o + so, o + so, (int)n, + bounce + so, (int)n, rb); + pr_info_ratelimited("vgsp: CEAUDIT-MISMATCH op=copy va=0x%llx off=0x%llx path=ce-soft-copy want=%*phN got=%*phN\n", + dst_va + o + so, o + so, (int)n, + bounce + so, (int)n, rb); + } +} + +#define SVC_COPY_TRUNC_RET(rc_) \ + do { \ + if (done_out) \ + *done_out = (u64)o; \ + if (total >= SVC_COPY_AUDIT_MIN) \ + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=ce-soft-copy rc=%d\n", \ + dst_orig, (u64)total, (u64)o, (rc_)); \ + return (rc_); \ + } while (0) + +/* + * fail4: pipeline-order source snapshot for held copies. The m169 + * device-dst guard defers a staged H2D (host->device) for up to 3 s; + * the driver's ping-pong staging buffers are overwritten by the next + * chunk meanwhile, so the deferred read returns the content of a later + * chunk (vector_add: device a[i] pages held a[i+2 MB] - exactly the + * 2x1 MB staging stride). A held copy snapshots its source at hold + * time and the retried execution copies from the snapshot, matching + * the hardware's in-order read of the staging. + */ +#define SVC_COPY_SNAP_N 16 +#define SVC_COPY_SNAP_MAX (4u << 20) +#define SVC_COPY_SNAP_TTL_MS 30000u +struct svc_copy_snap { + u64 dst, src; + size_t total; + /* + * Owner identity (r444): the channel, ring entry and push-content + * hash of the decode pass that captured this snapshot. A snapshot + * exists to serve ONE entry's retried copy (the held staging read); + * keying it by (dst,src,total) alone let a snapshot captured for an + * earlier entry poison a later copy with the same operand triple - + * the wedge_ab "bad = mutated-element count" class: step N's H2D + * executed from step N-1's snapshot because the re-decode of the + * completed step N-1 entry re-captured its (unchanged) source after + * its copy had already run, and step N's copy then consumed it. + * The pass hash changes whenever the guest rewrites the push, so a + * ring slot reused by a fresh push can never match a stale owner. + */ + u64 owner_hash; + u32 owner_ch; + u32 owner_idx; + u8 *buf; + unsigned long stamp; +}; +static struct svc_copy_snap svc_copy_snaps[SVC_COPY_SNAP_N]; + +static u8 *svc_copy_snap_lookup(u64 dst, u64 src, size_t total, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].dst == dst && + svc_copy_snaps[i].src == src && + svc_copy_snaps[i].total == total && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash == owner_hash) + return svc_copy_snaps[i].buf; + return NULL; +} + +static void svc_copy_snap_free(u64 dst, u64 src, size_t total, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].dst == dst && + svc_copy_snaps[i].src == src && + svc_copy_snaps[i].total == total && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash == owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + return; + } +} + +/* + * r456/r457: the snapshot lifecycle is capture-once -> free on the copy's + * own successful consumption (svc_copy) -> free on ring-slot reuse with a + * new push hash (svc_copy_snap_capture) -> free when a re-decode skips an + * already-done op (svc_run_ops) -> TTL. The old per-entry owner free was + * removed from every opdef_clear caller: an entry whose release was + * delivered can still be REPLAYED (the pin-batch rewind) with its copy + * still owed, and freeing at that point let the replay re-capture the + * driver's refilled ping-pong staging (the r457 mode-C b1 H2D wrote b2's + * bytes into db1). A stale snapshot can only ever serve its own + * (ch, idx, hash) owner, so the r444 cross-entry poisoning stays closed. + */ +static void __maybe_unused svc_copy_snap_free_owner(u32 owner_ch, u32 owner_idx) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } +} + +/* Capture the source of a virtual-operand copy about to be held. */ +static void svc_copy_snap_capture(struct nv_vgpu_device *vgpu, u64 dst, + u64 src, size_t total, u32 owner_ch, + u32 owner_idx, u64 owner_hash) +{ + struct svc_copy_snap *s = NULL; + u8 *buf; + size_t o; + u32 i; + + if (!total || total > SVC_COPY_SNAP_MAX) + return; + if (svc_copy_snap_lookup(dst, src, total, owner_ch, owner_idx, + owner_hash)) + return; /* already captured */ + /* + * r456: evict any snapshot left by the SAME ring slot under a + * DIFFERENT push hash (the slot was reused by a fresh push). The + * surviving-across-retries snapshot (svc_opdef_clear APPLY no + * longer frees it) must never be readable by the next occupant of + * the slot; the pass hash changes whenever the guest rewrites the + * push, so a same-slot different-hash snapshot is by definition + * stale for the new push. + */ + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash != owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } + } + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (!svc_copy_snaps[i].buf) { + s = &svc_copy_snaps[i]; + break; + } + if (time_after(jiffies, svc_copy_snaps[i].stamp + + msecs_to_jiffies(SVC_COPY_SNAP_TTL_MS))) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + s = &svc_copy_snaps[i]; + break; + } + } + if (!s) + return; /* all slots live: keep the historical behaviour */ + buf = kmalloc(total, GFP_KERNEL); + if (!buf) + return; + for (o = 0; o < total; ) { + size_t chunk = min_t(size_t, 0x1000 - ((src + o) & 0xfff), + total - o); + u64 pa = 0; + u32 ap = 0; + + /* The source is a virtual host VA; resolve per page first. */ + if (svc_op_translate(vgpu, src + o, &pa, &ap) || + svc_read(vgpu, pa, buf + o, chunk)) { + kfree(buf); + return; + } + o += chunk; + } + s->dst = dst; + s->src = src; + s->total = total; + s->owner_ch = owner_ch; + s->owner_idx = owner_idx; + s->owner_hash = owner_hash; + s->stamp = jiffies; + s->buf = buf; + pr_info_ratelimited("vgsp-sema: copy snapshot captured dst=0x%llx src=0x%llx total=%zu owner=0x%x/%u/%016llx first4=%08x %08x %08x %08x\n", + dst, src, total, owner_ch, owner_idx, + owner_hash, + total >= 4 ? *(const u32 *)buf : 0, + total >= 8 ? *(const u32 *)(buf + 4) : 0, + total >= 12 ? *(const u32 *)(buf + 8) : 0, + total >= 16 ? *(const u32 *)(buf + 12) : 0); +} + +static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind, + u64 pa, u32 off, u32 bytes, u64 *hpa, + u32 *out_off); + +/* r402 row-wise wrapper: see svc_fill. The source advances by spitch. */ +/* r402: read-only resolver probe (defined with the GR fill diagnostics). */ +static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out); static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, - u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture) + u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture, + u64 *done_out, u32 rows, u32 dpitch, u32 spitch, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + /* + * fail4: name every copy the executor is asked to run (pre- + * resolution), so a copy that fails or is held before any witness + * still has an entry record. SESSION 132: one record per COPY OP, + * not per row - a 480-row frame copy logged 480 rows and exhausted + * the per-session budget (128) inside the first op, hiding every + * later copy (including any SASS upload). + */ + if (gsp_sema_dbg && svc.copyent_log < 128) { + svc.copyent_log++; + pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu rows=%u dpitch=0x%x spitch=0x%x dphys=%u dap=%u sphys=%u sap=%u\n", + svc.cur_ch, dst, src, total, rows, dpitch, spitch, + dst_phys, dst_aperture, src_phys, src_aperture); + } + if (rows > 1 && dpitch) { + u64 done = 0; + int rc = 0; + u32 r; + + for (r = 0; r < rows; r++) { + u64 row_done = 0; + + rc = svc_copy_1d(vgpu, dst + (u64)r * dpitch, + src + (u64)r * spitch, total, + dst_phys, dst_aperture, src_phys, + src_aperture, &row_done, + owner_ch, owner_idx, owner_hash); + done += row_done; + if (rc) + break; + } + if (done_out) + *done_out = done; + return rc; + } + return svc_copy_1d(vgpu, dst, src, total, dst_phys, dst_aperture, + src_phys, src_aperture, done_out, + owner_ch, owner_idx, owner_hash); +} + +static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, + u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture, + u64 *done_out, u32 owner_ch, u32 owner_idx, u64 owner_hash) { u8 bounce[256]; size_t o; @@ -4370,10 +11062,19 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u64 dst_orig = dst, src_orig = src; u32 dst_prov = 0, src_prov = 0; bool bigcopy_this = false; + bool audit = total >= SVC_COPY_AUDIT_MIN; + u8 *snap = NULL; static u64 bigcopy_log; + static u32 shadow_dst_log; /* r614 diag: shadow-lane dst witness */ + bool trace64 = (total == 65536u); + const char *dir = "?"; + if (done_out) + *done_out = 0; /* 0 until the first byte lands */ if (!total || total > (256ULL << 20)) return -EINVAL; + snap = svc_copy_snap_lookup(dst_orig, src_orig, total, owner_ch, + owner_idx, owner_hash); if (!dst_phys) { u64 pa; @@ -4381,17 +11082,116 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, if (svc_vas_has(dst)) { dst_shadow = true; + } else if (!svc_ledger_lookup(dst, &pa, &aper)) { + /* + * r402 diagnostic (bounded): name the case where the + * ledger's landing page for a copy DESTINATION + * disagrees with the live resolver (pin/walk). The + * D2H staging writes failed on the class this + * witness detects: the writer follows the ledger + * while the guest's reader holds a different page, + * so the client reads stale data (streams2 c1 + * zeros / smiley 3-of-4 passes). Read-only probe + * (svc_gr_src_probe stores nothing). + */ + if (gsp_sema_dbg && svc_dst_div_log < 64 && + dst >= SVC_VAS_FLOOR) { + u64 wpa = 0; + u32 wap = 0; + const char *src = + svc_gr_src_probe(vgpu, dst, &wpa, &wap); + + if (wpa && + (((wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + ((wap == SVC_APERTURE_FB) != + (aper == SVC_APERTURE_FB)))) { + svc_dst_div_log++; + pr_err_ratelimited("vgsp-sema: DST-DIVERGE va=0x%llx ledger=0x%llx/%u probe=%s 0x%llx/%u (write follows ledger)\n", + dst, pa, aper, src, wpa, wap); + } + } + /* + * m180 unification: a previous write of this VA is + * the authority for where the content lives (and so + * for where a new write lands); it also spares the + * m169 stale-walk hold below, which exists precisely + * because no pin was known. + * + * r376 pin-agreement: a FRESH pin that disagrees is + * newer than the recorded landing - the guest freed + * or moved the buffer - so the pin wins and the + * entry is repaired (the same rule the fills and the + * launch paths apply; without it a stale entry + * writes a freed page - the guest-corruption class). + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 64) + pr_info("sema-ledger: PIN-WINS copy-dst va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + dst, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper, ipa, iap); + svc_ledger_store(dst, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + pa = ipa; + aper = (u32)iap; + } + } + dst = pa; + dst_aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + dst_prov = 1; + dst_virt = true; } else if (svc_op_translate(vgpu, dst, &pa, &aper)) { - if (!svc_vas_can_shadow(dst)) { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(dst, &lpa, &lap)) { + dst = lpa; + dst_aperture = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + dst_prov = 1; + dst_virt = true; + } else if (!svc_vas_can_shadow(dst)) { if (gsp_sema_dbg && svc.op_tr_log < 64) { svc.op_tr_log++; pr_info("sema-tr-miss: DST va=0x%llx h=0x%x\n", dst, svc.cur_ch); } return -ENOENT; + } else { + dst_shadow = true; + svc.vas_shadowed++; + /* + * r614 diag: the r613b red boot's matmul + * wedge began here - the H2D into the + * client's cudaMalloc'd buffer found no + * ledger/pin/walk answer, took the shadow + * silently, and the later GR fill could not + * consume the shadowed bytes (probe=none for + * every arg base; the launch held 15 s, then + * the degenerate refusal consumed it and the + * client spun). Name the first shadow-lane + * device-dst copies of a boot (bounded), so + * the next red boot is greppable where it + * starts instead of only at the launch. + */ + if (shadow_dst_log < 64 && + dst_orig >= SVC_VAS_FLOOR) { + shadow_dst_log++; + pr_info("sema-shadow-dst: h=0x%x dst=0x%llx total=%zu (device VA with no ledger/pin/walk answer; copy runs into the shadow, a GR fill cannot consume it)\n", + owner_ch, dst_orig, total); + } } - dst_shadow = true; - svc.vas_shadowed++; } else { dst = pa; dst_aperture = aper; @@ -4399,24 +11199,177 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, dst_virt = true; } } + /* + * m169 (AGENT-F R1): a device-lane destination translated only by a + * page-directory walk (prov 3/4) or the RAM scan (5) can be a stale + * PTE from the previous session - the table CONTENT is not wiped by + * the session flush - and a wrong-but-valid vidmem write is silent. + * r172 streams2_diag: the stream-1 a H2D ran rc=0 through a + * walk-resolved destination while the launch (pin-resolved) read the + * true page still holding the previous client's h_a; the engine + * computed from stale inputs and the client verified 32768 bad. + * Since the resolver tries pins FIRST, prov >= 3 means no pin existed + * at resolve time: hold the copy (-ENOENT feeds the m163 bounded + * hold), which re-decodes after the client's pins land (seconds) and + * is loud on expiry - never silent corruption. Device destinations + * only; sysmem copies keep the historical sources. + */ + if (dst_virt && dst_aperture == CE_APERTURE_VID && + dst_prov >= 3 && dst_orig >= SVC_VAS_FLOOR && + gsp_sema_copy_dst_pin_guard && + svc_dst_guard_should_hold(dst_orig)) { + svc.copy_dst_hold++; + if (svc.copy_dst_hold <= 16) + pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u (no pin yet; stale-table guard)\n", + dst_orig, dst_prov); + /* + * fail4: the staging source is valid NOW; snapshot it so + * the deferred execution copies the pipeline-order bytes. + */ + if (!src_phys && src_orig >= SVC_VAS_FLOOR) + svc_copy_snap_capture(vgpu, dst_orig, src_orig, + total, owner_ch, owner_idx, + owner_hash); + return -ENOENT; + } + /* + * r378: the same stale-table class for SYS destinations resolved by + * the walk/scan (prov >= 3): a FRESH injected pin that disagrees is + * the guest's newest statement, so writing through the resolved page + * lands on a page the guest has freed/re-homed (the guest-corruption + * class). Hold through the same bounded re-decode path instead of + * writing; a missing pin keeps the historical behaviour (the + * resolution is then the write authority). + */ + if (dst_virt && dst_aperture == CE_APERTURE_SYS && + dst_prov >= 3 && dst_orig >= SVC_VAS_FLOOR && + gsp_sema_copy_dst_pin_guard) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_orig, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dst & ~(u64)(SEMA_PAGE_SIZE - 1))) || + svc_ap_of(iap) != SVC_APERTURE_SYS) && + svc_dst_guard_hold(dst_orig)) { + svc.copy_dst_hold++; + if (svc.copy_dst_hold <= 16) + pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u pin=0x%llx (stale SYS target)\n", + dst_orig, dst_prov, ipa); + if (!src_phys && src_orig >= SVC_VAS_FLOOR) + svc_copy_snap_capture(vgpu, dst_orig, + src_orig, total, + owner_ch, owner_idx, + owner_hash); + return -ENOENT; + } + } + /* + * r231 READ-ORDER guard/witness: a device-src copy whose source VA + * has a PENDING device-dst write hold reads the PRE-write content - + * the writer's bytes land only when the guard resolves while the + * reader executes immediately (the ctaid_probe RAWA round trip is + * the reproducer). Witnessed always (bounded); deferred only with + * sema_read_order_guard, through the same bounded-hold re-decode. + */ + if (src_orig >= SVC_VAS_FLOOR && svc_dst_guard_pending(src_orig)) { + if (svc_read_order_log < 16) { + svc_read_order_log++; + pr_info("sema-svc: READ-ORDER-WIT device-src copy va=0x%llx reads while a device-dst write to the same VA is still held%s\n", + src_orig, gsp_sema_read_order_guard ? + " (deferring)" : " (guard off)"); + } + if (gsp_sema_read_order_guard) + return -ENOENT; + } if (!src_phys) { u64 pa; u32 aper; if (svc_vas_has(src)) { src_shadow = true; + } else if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src, &pa, &aper)) { + /* + * r375 read-after-write authority: the landing table + * records where every write of this VA landed (writes + * store it, fresh pins repair it). A reader MUST read + * exactly that page or the engine's output is invisible + * (the sum=0 class). The walk/pin sources can answer + * differently - e.g. the same numeric pa in a different + * aperture - so the table wins for device VAs; only a + * VA with no recorded landing falls through to the + * resolver below. + * + * r376 pin-agreement: unless a FRESH pin disagrees - + * then the pin is newer (the guest re-pinned the VA) + * and wins, and the entry is repaired. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(src, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 64) + pr_info("sema-ledger: PIN-WINS copy-src va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + src, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper, ipa, iap); + svc_ledger_store(src, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + pa = ipa; + aper = (u32)iap; + } + } + src = pa; + src_aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + src_prov = 1; + src_virt = true; } else if (svc_op_translate(vgpu, src, &pa, &aper)) { - if (gsp_sema_dbg && svc.op_tr_log < 64) { - svc.op_tr_log++; - pr_info("sema-tr-miss: SRC va=0x%llx h=0x%x\n", - src, svc.cur_ch); + u64 lpa = 0; + u32 lap = 0; + + /* + * m180 unification: read the page the last host write + * of this VA landed on when the resolver can no longer + * answer (same rule as the per-page loop below), instead + * of failing the whole copy. + */ + if (!svc_ledger_lookup(src, &lpa, &lap)) { + src = lpa; + src_aperture = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + src_prov = 1; + src_virt = true; + } else { + if (gsp_sema_dbg && svc.op_tr_log < 64) { + svc.op_tr_log++; + pr_info("sema-tr-miss: SRC va=0x%llx h=0x%x\n", + src, svc.cur_ch); + } + return -ENOENT; } - return -ENOENT; } else { src = pa; src_aperture = aper; src_prov = svc.op_tr_src; src_virt = true; + /* + * r375b: do NOT record a resolver answer into the + * landing table here. The resolver can return a stale + * PTE page (the m169 class) and the table is the WRITE + * authority - a poisoned entry makes every later write + * of this VA land on the stale page (r375 boot: slab + * GPF in kmem_cache_alloc at guest t=77s, before any + * client). The table is fed by writes and fresh pins + * only. + */ } } @@ -4426,8 +11379,35 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, * 16 bytes on each side before the copy. The vadd data path is exactly * these copies; this separates a wrong resolution from a wrong copy. */ - if (total >= (1u << 20) && bigcopy_log < 24) { + if ((total >= SVC_COPY_AUDIT_MIN || + (dst_virt && dst_orig >= SVC_VAS_FLOOR)) && + (bigcopy_log < gsp_sema_bigcopy_log_max || total == 65536u)) { u8 sb[16] = {0}, db[16] = {0}; + /* + * r235: 64 KB copies are exempt from the bigcopy budget - the + * probe's A upload/readbacks must ALWAYS appear (the budget + * belongs to the multi-MB traffic and had absorbed every 64 KB + * probe copy in the r235 capture). + */ + bool trace64 = (total == 65536u); + const char *dir = + (dst_aperture == CE_APERTURE_VID && + src_aperture == CE_APERTURE_SYS) ? "H2D" : + (src_aperture == CE_APERTURE_VID && + dst_aperture == CE_APERTURE_SYS) ? "D2H" : + src_shadow ? "shadow->" : "phys"; + + /* + * r235: 64 KB copies are exempt from the bigcopy budget - the + * probe's A upload/readbacks must ALWAYS appear (the budget + * belongs to the multi-MB traffic and had absorbed every 64 KB + * probe copy in the r235 capture). + */ + dir = (dst_aperture == CE_APERTURE_VID && + src_aperture == CE_APERTURE_SYS) ? "H2D" : + (src_aperture == CE_APERTURE_VID && + dst_aperture == CE_APERTURE_SYS) ? "D2H" : + src_shadow ? "shadow->" : "phys"; bigcopy_log++; bigcopy_this = true; @@ -4451,12 +11431,55 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, } else { svc_read(vgpu, dst, db, sizeof(db)); } - pr_info("sema-bigcopy: dst_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | src_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | bytes=%zu\n", + pr_info("sema-bigcopy: dst_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | src_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | bytes=%zu vbase=%px\n", dst_orig, dst, dst_aperture, dst_prov, dst_shadow, src_orig, src, src_aperture, src_prov, src_shadow, - total); + total, (void *)vgpu->bars[1].vaddr); + /* + * r348 diagnostic: for a high-VA SYS source name the exact + * host PA this read will touch (gpa_to_hpa) - the direct + * comparison against the svc-fwd-hpa writeback line. + */ + if (src_orig >= SVC_VAS_FLOOR && + src_aperture == CE_APERTURE_SYS) { + u64 shpa = 0; + u32 soff = 0; + + if (!svc_fwd_operand_hpa(vgpu, SVC_APERTURE_SYS, + src, 0, SEMA_PAGE_SIZE, + &shpa, &soff) && !soff) + pr_info("sema-bigcopy: src-hpa va=0x%llx gpa=0x%llx -> hpa=0x%llx\n", + src_orig, src, shpa); + else + pr_info("sema-bigcopy: src-hpa va=0x%llx gpa=0x%llx -> UNMAPPED\n", + src_orig, src); + } pr_info("sema-bigcopy: src16=%16phN dst16_before=%16phN\n", sb, db); + /* + * r234 A-TRACE: one labeled line per 64 KB copy so the probe A + * upload/readback is greppable end-to-end. pt=P1 = H2D source + * read (the uploaded bytes as ints), pt=P4 = D2H source read + * (the engine-authored page as ints). The expected content is + * ints 0,1,2,3,... for the probe A buffer. + */ + if (trace64 && svc_atrace_log < 128u) { + u32 i0, i1, i2, i3, j0 = 0, j1 = 0, j2 = 0, j3 = 0; + + svc_atrace_log++; + memcpy(&i0, sb, 4); + memcpy(&i1, sb + 4, 4); + memcpy(&i2, sb + 8, 4); + memcpy(&i3, sb + 12, 4); + memcpy(&j0, db, 4); + memcpy(&j1, db + 4, 4); + memcpy(&j2, db + 8, 4); + memcpy(&j3, db + 12, 4); + pr_info("vgsp: A-TRACE pt=%s dir=%s dst_va=0x%llx dst_pa=0x%llx dst_prov=%u | src_va=0x%llx src_pa=0x%llx src_prov=%u | src_int4=%d %d %d %d dst_int4_before=%d %d %d %d\n", + !strcmp(dir, "H2D") ? "P1" : "P4", dir, + dst_orig, dst, dst_prov, src_orig, src, + src_prov, i0, i1, i2, i3, j0, j1, j2, j3); + } } /* @@ -4505,6 +11528,17 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, svc_fwd_cand.bytes = (u32)total; /* + * m180 unification: record the destination landing + * even when the copy itself is deferred to the + * forwarded-request path (the software copy below + * never runs then, so the in-loop store would be + * skipped and the page would have no authority for + * the fill/D2H readers). + */ + if (dst_orig >= SVC_VAS_FLOOR) + svc_ledger_store(dst_orig, dst_page, dst_kind); + + /* * P6b wave 2: with the hardware-primary path armed * (parameter on and all three prerequisites on, so the * request can queue as a guest copy and its releases @@ -4638,64 +11672,267 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, chunk = min_t(size_t, chunk, 0x1000 - ((dst_orig + o) & 0xfff)); - if (src_shadow) { + if (snap) { + /* + * r314: consume the decode-time staging snapshot. + * The capture machinery existed since cb50282b but + * the execution never read it (lookup at svc_copy's + * top, free on success, NO consumer) - every copy + * read the live staging, which the driver had + * refilled once the push's fence fired (the + * deterministic 786335-bad vector_add output, + * identical across boots). Reading from the + * snapshot also needs no live source translation, + * so the copies execute on their first allowed + * pass instead of storming 60+ rc=-2 retries. + */ + memcpy(bounce, snap + o, chunk); + } else if (src_shadow) { if (svc_vas_read(src_orig + o, bounce, chunk)) - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); } else if (src_virt) { u64 pa = 0; u32 aper = 0; + u64 lpa = 0; + u32 lap = 0; - if (svc_op_translate(vgpu, src_orig + o, &pa, &aper)) - return -ENOENT; + /* + * r375: the landing table is the read-after-write + * authority for device VAs (see the top-level src + * resolve). Per page, the table wins whenever it has + * an entry; the resolver below only answers a VA the + * table has never seen. r376: a fresh disagreeing pin + * is newer than the table and wins (repaired). + */ + if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src_orig + o, &lpa, &lap)) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(src_orig + o, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (lpa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)lap)) { + svc_ledger_pin_repairs++; + svc_ledger_store(src_orig + o, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + lpa = ipa & ~(u64)(SEMA_PAGE_SIZE - 1); + lap = (u32)iap; + } + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (svc_op_translate(vgpu, src_orig + o, &pa, &aper)) { + /* + * m180 unification: a per-page resolution miss + * must not fail the whole copy when the unified + * table knows where this VA's content landed. + */ + if (svc_ledger_lookup(src_orig + o, &lpa, &lap)) + SVC_COPY_TRUNC_RET(-ENOENT); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src_orig + o, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (lap == SVC_APERTURE_FB) != + (aper == CE_APERTURE_VID))) { + /* + * m171 (AGENT-F R2): read-after-write - when + * a large H2D wrote this device VA page + * through a different translation, read the + * page the write landed on. + */ + svc_ledger_diverges++; + if (svc_ledger_diverges <= 32) + pr_info("sema-ledger: DIVERGE read va=0x%llx wrote=0x%llx read=0x%llx\n", + src_orig + o, lpa, pa); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } if (aper == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; if (!base || pa + chunk > svc_fb_shadow_size(vgpu)) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(bounce, base + pa, chunk); } else if (svc_read(vgpu, pa, bounce, chunk)) { - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); } } else if (src_aperture == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); if (!base || src + o + chunk > size) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(bounce, base + src + o, chunk); } else { if (svc_read(vgpu, src + o, bounce, chunk)) - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); + } + + /* + * r367 TABLE-COPY witness: a page-table page/entry copy is + * the guest's authoritative table content; dump the carried + * bytes so a table that later walks as ZERO can be traced to + * the copy that built it (or proven the source was already + * zero). Bounded; table-shaped sizes only. + */ + if (!o && (total == 4096 || total == 16) && + svc_tabcopy_log < 64) { + svc_tabcopy_log++; + pr_info("sema-tabcopy: dst_orig=0x%llx src_orig=0x%llx total=%zu dphys=%u dvirt=%d svirt=%d r_dst=0x%llx r_src=0x%llx first16=%16phN\n", + dst_orig, src_orig, total, dst_phys, dst_virt, + src_virt, dst, src, bounce); } if (dst_shadow) { if (svc_vas_write(dst_orig + o, bounce, chunk)) - return -ENOMEM; + SVC_COPY_TRUNC_RET(-ENOMEM); + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, true, 0, 0, + (o != 0)); } else if (dst_virt) { u64 pa = 0; u32 aper = 0; + u64 lpa = 0; + u32 lap = 0; + u32 wprov = 0; - if (svc_op_translate(vgpu, dst_orig + o, &pa, &aper)) - return -ENOENT; + if (svc_op_translate(vgpu, dst_orig + o, &pa, &aper)) { + /* + * m180 unification: a per-page resolution miss + * lands on the known page instead of failing + * the whole copy (readers already use it). + */ + if (svc_ledger_lookup(dst_orig + o, &lpa, &lap)) + SVC_COPY_TRUNC_RET(-ENOENT); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wprov = 0xF1; + } else if (dst_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(dst_orig + o, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (lap == SVC_APERTURE_FB) != + (aper == CE_APERTURE_VID))) { + svc_ledger_diverges++; + if (svc.op_tr_src == 1) { + /* + * r336 PIN-WINS (the r335 vector_add + * guest-panic class): a pin-backed + * translation (guest pagemap truth, + * op_tr_src == 1) that disagrees + * with the ledger means the guest + * remapped or reused this VA. The + * ledger page is where an earlier + * write landed; the guest has since + * freed or moved it, so writing it + * can land on memory the guest + * kernel now owns (observed: RM + * object fields overwritten with + * client c-data, slab freelist + * corruption, guest panic). Trust + * the pin for the write and re-home + * the ledger (the store after the + * write records the fresh page); + * readers then converge on the same + * page too. + */ + svc_ledger_pinwins++; + if (svc_ledger_diverges <= 512) + pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx (pin wins)\n", + dst_orig + o, lpa, pa); + wprov = svc.op_tr_src; + } else { + /* + * m180 unification: keep ONE + * authoritative page per VA. A new + * write goes to the page the + * previous content landed on, and + * the fill/D2H readers (which + * prefer the table) stay consistent + * with it. Non-pin translations + * (2..5) can carry stale PTEs + * (m169), so the write history + * stays the authority there. + */ + if (svc_ledger_diverges <= 512) + pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx\n", + dst_orig + o, lpa, pa); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wprov = 0xF0; + } + } else { + wprov = svc.op_tr_src; + } + /* + * D2H-MAP (diagnostic): per-page destination + * resolution of large host-dest copies - the direct + * witness for which guest pages a readback lands on. + * Bounded; one line per 4 KB page of copies >= 1 MB. + */ + if (total >= (1u << 20) && !(o & (SEMA_PAGE_SIZE - 1)) && + svc_d2hmap_log < 4096u) { + svc_d2hmap_log++; + pr_info("sema-d2hmap: va=0x%llx o=%zu pa=0x%llx ap=%u prov=%u\n", + dst_orig + o, o, pa, aper, wprov); + } if (aper == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; if (!base || pa + chunk > svc_fb_shadow_size(vgpu)) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(base + pa, bounce, chunk); } else if (svc_write(vgpu, pa, bounce, chunk)) { - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); } + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, false, pa, aper, + (o != 0)); + /* + * m171/m180: a large device-dest write records its + * landing page - SYS and VID alike, because the + * sysmem-backed UVM buffers are the dominant + * device-visible case and were the half the table + * never saw. The later kernel fill / D2H read of the + * same VA then uses exactly this page even if its own + * resolution is stale. + */ + if (dst_orig >= SVC_VAS_FLOOR && total >= (64u << 10)) + svc_ledger_store(dst_orig + o, pa, + (aper == CE_APERTURE_VID) ? + SVC_APERTURE_FB : + SVC_APERTURE_SYS); } else if (dst_aperture == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); if (!base || dst + o + chunk > size) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(base + dst + o, bounce, chunk); + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, false, dst + o, + CE_APERTURE_VID, + (o != 0)); } else { if (svc_write(vgpu, dst + o, bounce, chunk)) - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, false, dst + o, + CE_APERTURE_SYS, + (o != 0)); } (void)i; o += chunk; @@ -4721,6 +11958,23 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, } pr_info("sema-bigcopy: dst16_after=%16phN (dst_va=0x%llx pa=0x%llx)\n", da, dst_orig, dst); + /* + * r234 A-TRACE pt=P2/P4R: the destination content right after + * the copy - for H2D this is the landed upload (P2), for D2H + * the host-side readback result (P4R). + */ + if (trace64 && svc_atrace_log < 128u) { + u32 i0, i1, i2, i3; + + svc_atrace_log++; + memcpy(&i0, da, 4); + memcpy(&i1, da + 4, 4); + memcpy(&i2, da + 8, 4); + memcpy(&i3, da + 12, 4); + pr_info("vgsp: A-TRACE pt=%s dir=%s dst_va=0x%llx dst_pa=0x%llx dst_int4_after=%d %d %d %d\n", + !strcmp(dir, "H2D") ? "P2" : "P4R", dir, + dst_orig, dst, i0, i1, i2, i3); + } } if (gsp_sema_dbg && !src_shadow && !dst_shadow && @@ -4757,6 +12011,56 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, pr_info("sema-copy-dst: after dst=0x%llx dst16=%*phN\n", dst, n, da); } + if (audit && svc_copy_audit_n < SVC_COPY_AUDIT_MAX) { + svc_copy_audit_n++; + pr_info("vgsp: COPYAUDIT path=ce-soft-copy va=0x%llx len=0x%llx done=0x%llx\n", + dst_orig, (u64)total, (u64)o); + } + /* + * m180 parity with svc_fill: a large virtual-destination copy is a + * write/read pair with the launch rec-build readers - record its + * landing pages so the GR fill follows the page the copy actually + * wrote. Without this, a rec table built before the pins landed + * kept the stale pre-pin pages while the copy landed on the pinned + * page (streams2 r254: the b1 rec range carried the a2 pages and + * the vadd read a2's pattern as its second input). + * + * r627: NEVER overwrite a per-page landing the loop above already + * stored. That loop stores each page's true resolved PA; this + * pass's `dst + po` is a contiguity guess valid only while the VA + * range maps contiguously. For a scattered UVM host buffer (r623c + * saxpy: page 0 at pa 0x115781000, page 0x7f000 at pa 0x12e700000) + * the guess clobbered the true entries - + * `sema-ledger: rewrite va=0x70e64e77f000 old=0x12e700000 + * new=0x115800000` - and the NEXT copy of the same VA then + * followed the poisoned ledger: 504x `sema-ledger: DIVERGE write + * va=0x70e64e7xxxxx prev=0x1158xxxxx cur=0x12e7xxxxx`; the + * non-pin branch keeps the ledger, so ~2 MiB of client c-data + * landed in guest pages the kernel had since re-used and the + * guest died in kchannelGetGfid_IMPL (r623c panic). The loop's + * per-page entries are the authority; this pass only seeds pages + * the loop did not reach. + */ + if (!dst_phys && dst_virt && dst_orig >= SVC_VAS_FLOOR && + total >= (64u << 10)) { + u32 cap = (dst_aperture == CE_APERTURE_VID) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + size_t po; + + for (po = 0; po < total; po += SEMA_PAGE_SIZE) { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(dst_orig + po, &lpa, &lap)) + continue; + svc_ledger_store(dst_orig + po, dst + po, cap); + } + } + if (snap) + svc_copy_snap_free(dst_orig, src_orig, total, owner_ch, + owner_idx, owner_hash); + if (done_out) + *done_out = (u64)o; /* o == total on the success path */ return 0; } @@ -4766,6 +12070,44 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, * VA through the trusted sources (pins, discovered map, published page * directories, targeted rm_mem resolve), pin the page and write the bytes. */ +/* + * SESSION 153-CONTINUED-88: write an inline payload into one landing page + * (page-aligned PA, aperture) at the destination VA's in-page offset. Shared + * by the primary landing write and the engine-visible pin dual-write. The + * write is clamped to the page end: a crossing payload's tail belongs to the + * NEXT page's own landing, not to whatever follows this page in the aperture. + */ +static int svc_grce_land_write(struct nv_vgpu_device *vgpu, u64 pa_page, + int ap, u64 dst_va, const u8 *data, u32 bytes) +{ + u64 pa = pa_page + (dst_va & 0xfff); + u32 room = 4096u - (u32)(dst_va & 0xfff); + + if (bytes > room) + bytes = room; + if (!bytes) + return 0; + if (ap == SVC_APERTURE_FB) { + u8 *base = vgpu->bars[1].vaddr; + + if (!base || pa + bytes > svc_fb_shadow_size(vgpu)) + return -EINVAL; + memcpy(base + pa, data, bytes); + return 0; + } + { + u32 o; + + for (o = 0; o < bytes; o += 4096) { + u32 chunk = bytes - o > 4096 ? 4096 : bytes - o; + + if (svc_write(vgpu, pa + o, (void *)(data + o), chunk)) + return -EIO; + } + } + return 0; +} + static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, const u8 *data, u32 bytes, u32 ch_h) { @@ -4787,6 +12129,83 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, svc.memsets_done++; return 0; } + /* + * m180 unification: a page with a known landing (previous H2D/fill or + * engine write-back) is authoritative - write the inline data there + * so the engine fill / D2H readers that follow the table see it. + */ + { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(dst_va, &lpa, &lap)) { + pa = lpa & ~0xfffULL; + ap = (lap == SVC_APERTURE_FB) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + /* + * SESSION 153-CONTINUED-88: dual-write to the + * engine-visible page. The launch fill resolves a + * page through the LIVE PINS first ("pin wins", see + * svc_gr_page_note) and the engine reads exactly + * that page - a ledger landing in the BAR1 shadow + * (ap=vidmem) is NOT what the engine fetches. The + * observed case: the real program uploads (1536 B at + * 0x4350900, 1408 B at 0x4350f00) landed at pa + * 0x1b0900/0x1b0f00 (vidmem) while the engine read + * the pinned sysmem page (0x529ff4000). When an + * injected pin names a different page, author the + * bytes there too (the ledger landing stays written + * for its existing consumers). + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_va & ~0xfffULL, + &ipa, &iap) && + ipa && (ipa & ~0xfffULL) != + (pa & ~0xfffULL)) { + if (!svc_grce_land_write(vgpu, + ipa & ~0xfffULL, + iap, dst_va, + data, bytes) && + gsp_sema_dbg && + svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: inline dual-write dst=0x%llx pa=0x%llx ap=%s bytes=%u (engine-visible pin) h=0x%x\n", + dst_va, + ipa & ~0xfffULL, + iap == SVC_APERTURE_FB ? + "vidmem" : "ram", + bytes, ch_h); + } + } + } + svc_pin_store(dst_va & ~0xfffULL, pa, ap); + goto write_landing; + } + } + /* + * SESSION 153-CONTINUED-87: the launch fill resolves a page through + * the LIVE PINS first ("pin wins", see svc_gr_page_note) and the + * engine reads exactly that page. An upload whose trusted landing is + * a VIDMEM aperture page must also reach the pinned engine-visible + * page, or the engine validates stale content: the real program + * uploads (1536 B at 0x4350900, 1408 B at 0x4350f00) currently land + * only in the BAR1 shadow while the fill hands the engine the pinned + * sysmem page. Write to the pin when it names a different page. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_va & ~0xfffULL, &ipa, &iap) && + iap != SVC_APERTURE_FB && ipa) { + pa = ipa & ~0xfffULL; + ap = SVC_APERTURE_SYS; + goto write_landing; + } + } if (svc_lookup_trusted(vgpu, dst_va & ~0xfffULL, &pa, &ap)) { if (!svc_vas_write(dst_va, data, bytes)) { svc.vas_shadowed++; @@ -4806,6 +12225,7 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, return -ENOENT; } svc_pin_store(dst_va & ~0xfffULL, pa, ap); +write_landing: pa += dst_va & 0xfff; if (gsp_sema_dbg && svc.grce_log < (gsp_sema_ops_log > 64 ? gsp_sema_ops_log : 64)) { @@ -4828,24 +12248,537 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, for (o = 0; o < bytes; o += 4096) { u32 chunk = bytes - o > 4096 ? 4096 : bytes - o; - if (svc_write(vgpu, pa + o, (void *)(data + o), chunk)) + if (svc_write(vgpu, pa + o, (void *)(data + o), chunk)) { + if (bytes >= SVC_COPY_AUDIT_MIN) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=grce-inline rc=%d\n", + dst_va, (u64)bytes, + (u64)o, -EIO); return -EIO; + } + } + } + /* + * COPYAUDIT (see the svc_copy comment): log the big inline write and + * read the first 16 destination bytes back through the aperture it + * was written to, comparing against the source bytes. + */ + if (bytes >= SVC_COPY_AUDIT_MIN && svc_copy_audit_n < SVC_COPY_AUDIT_MAX) { + u8 rb[16] = {0}; + bool ok = false; + + if (ap == SVC_APERTURE_FB) { + u8 *base = vgpu->bars[1].vaddr; + + if (base && pa + sizeof(rb) <= svc_fb_shadow_size(vgpu)) { + memcpy(rb, base + pa, sizeof(rb)); + ok = true; + } + } else if (!svc_read(vgpu, pa, rb, sizeof(rb))) { + ok = true; } + svc_copy_audit_n++; + pr_info("vgsp: COPYAUDIT path=grce-inline va=0x%llx len=0x%llx done=0x%llx\n", + dst_va, (u64)bytes, (u64)bytes); + if (ok && memcmp(rb, data, sizeof(rb))) + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x0 path=grce-inline want=%*phN got=%*phN\n", + dst_va, 16, data, 16, rb); } + /* + * m180: record the landing so the fill/D2H readers of this VA follow + * the bytes the engine's inline path wrote, not their own resolution. + */ + if (dst_va >= SVC_VAS_FLOOR) + svc_ledger_store(dst_va, pa, (u32)ap); svc.memsets_done++; return 0; } +/* defined below (svc_handle_map_add); the CE decode path drains the queued + * scanner injections before resolving copy operands, exactly like the GR + * path's m143 drain. */ +static void svc_handle_map_add(void); + /* Execute a decoded operation list (fills and copies). `ops_done` (may be * NULL) is the per-entry bitmap of ops that already ran for the entry's * identity; `ops_base` is the sequence number of ops[0] within the entry. */ + +/* + * r34x: evaluate a C56F SEM_EXECUTE acquire against the semaphore word. + * The class defines the opcodes (clc56f.h NVC56F_SEM_EXECUTE_OPERATION: + * 0 ACQUIRE, 1 RELEASE, 2 ACQ_STRICT_GEQ, 3 ACQ_CIRC_GEQ, 4 ACQ_AND, + * 5 ACQ_NOR, 6 REDUCTION) and the CUDA driver documents the wait + * conditions each op must implement (cuda_device.h CUIsemaphoreAquire- + * Flags_enum: GEQ default = circular >=, EQUAL = ==, AND = (v & p) != 0, + * NOR = ~(v | p) != 0, STRICT_GEQ = >= with no wrap; ampere_semaphore.c + * maps those flags onto the opcodes: EQUAL->0, STRICT_GEQ->2, GEQ->3, + * AND->4, NOR->5). Note op 0 is EQUALITY, not a >=: only op 2/3 are + * ordering compares. 32-bit payloads: every observed CUDA acquire is + * PAYLOAD_SIZE 32BIT (SEM_PAYLOAD_HI == 0); a 64-bit acquire is captured + * and judged on the low word only (its high word is not modelled). + */ +static bool svc_wait_satisfied(u32 cur, u8 op, u32 need) +{ + switch (op) { + case 0: /* ACQUIRE: wait for v == p */ + return cur == need; + case 2: /* ACQ_STRICT_GEQ: v >= p, no wrap */ + return cur >= need; + case 3: /* ACQ_CIRC_GEQ: v in [p, p + 2^31 - 1] (wrap allowed) */ + return (s32)(cur - need) >= 0; + case 4: /* ACQ_AND: (v & p) != 0 */ + return (cur & need) != 0; + case 5: /* ACQ_NOR: ~(v | p) != 0 */ + return ~(cur | need) != 0; + default: + return true; + } +} + +/* + * r638: is (h, idx) currently owned by the bounded wait gate? The CE + * stream-order watermark consults this to tell a producer backlog (a CE + * copy the client ordered before the launch - the reason the watermark + * exists) from a consumer-side wait the forward itself will unblock (the + * mutual citation measured on r612/r636: the deferred launch's release is + * the only value that can satisfy the held copy's acquire, and the copy's + * unconsumed entry is the only reason the launch is deferred). + */ +static bool svc_wait_held_entry(u32 h, u32 idx) +{ + int i; + + if (idx == 0xffffffffu) + return false; + for (i = 0; i < SVC_WAIT_HOLD_MAX; i++) { + const struct svc_wait_hold *e = &svc_wait_holds[i]; + + if (e->active && e->h == h && e->idx == idx) + return true; + } + return false; +} + +/* r34x: the per-channel first-hold clock for the bounded wait. */ +static struct svc_wait_hold *svc_wait_hold_get(u32 h, bool create) +{ + struct svc_wait_hold *free_slot = NULL; + int i; + + for (i = 0; i < SVC_WAIT_HOLD_MAX; i++) { + struct svc_wait_hold *e = &svc_wait_holds[i]; + + if (e->active && e->h == h) + return e; + if (!e->active && !free_slot) + free_slot = e; + } + if (!create || !free_slot) + return NULL; + memset(free_slot, 0, sizeof(*free_slot)); + free_slot->active = true; + free_slot->h = h; + free_slot->idx = 0xffffffffu; /* r638: unset until a hold lands */ + free_slot->since = jiffies; + return free_slot; +} + +/* + * r506w/r507a: bounded per-VA failure latch for virtual ops. A virtual op + * that keeps failing -ENOENT (an unresolvable VA) re-runs on every decode + * pass and sets svc.op_tr_pending each time, so its push holds for the full + * SVC_OPDEF_HOLD_MS opdef bound - a client whose stream carries such an op + * (r505/r506 witness: the RM's channel-progress zeroing at 0x2000c0xxx + * after the first client, dst never translated) ground its whole stream + * at ~10 s per push ("hang after DEVICE" for matmul/multikernel). + * + * The window must still cover the legitimate pin-late retry (observed + * pin latency 0.5-2 s: the H2D copy's operands land a moment after the + * push, r507 matmul's 1 MB UVA copy at 0x7136... needed >8 fast passes), + * so the latch is TIME-based: the FIRST failure of a VA arms a + * SVC_OPDEF_HOLD_MS window during which the op holds its entry exactly as + * before; after a full bound with no success the op still attempts the + * write but no longer forces the hold - the entry consumes promptly and + * its withheld release applies, exactly the outcome the expired bound + * already sanctioned (op stays skipped, release delivered), minus the + * 10 s tax per subsequent pass. A success clears the record (a later + * failure gets a fresh window). Service-thread only, like every other op + * path. + */ +#define SVC_OPFAIL_MAX 32 +/* + * r507a: the op-failure window rides the opdef hold bound (SVC_OPDEF_HOLD_MS + * defined further down). Keep the literal here so the latch (used by + * svc_run_ops) precedes that define. + * + * r549: 10000 -> 3000. The window is the SECOND wait an unresolvable op + * costs: fail #1 arms this record, the entry is held for the opdef bound, + * the retry's fail #2 latches the op out of the hold. r548 evidence: a + * client whose init sequence carries ~65 device-dst ops with destinations + * no resolver covers walked one entry per 10 s (OPDEF-EXPIRED every + * 10.5 s, idx 7 -> 18) - the client spun at 100 % CPU for 10+ minutes and + * the battery timed out. Ops that DO resolve show 0.5-2 s pin latency, so + * 3 s keeps the m163 protection and cuts the dead-op cost to ~3 s per + * distinct destination. + */ +#define SVC_OPFAIL_WINDOW_MS 3000u +static struct { + u64 va; + unsigned long since; +} svc_opfail[SVC_OPFAIL_MAX]; +static u32 svc_opfail_log; + +static void svc_opfail_clear(u64 va) +{ + u32 i; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) + if (svc_opfail[i].va == va) + svc_opfail[i].va = 0; +} + +/* + * r541: device-destination copies that decoded but have not executed yet. + * + * smiley live evidence (r539): passes 2 and 4 verified a + (previous b) - + * the client's cudaMemcpy(b) completed (its release landed) while the H2D + * op was still held, the client submitted the launch, the GR forward read + * the shadow page BEFORE the H2D executed (b's copy ran at 294026.818, + * launch 2's engine read at ~294026.3), and the engine computed with the + * PREVIOUS pass's b (bad=65024). The same shape is the streams2 / + * streams_events readback lottery: a device page whose writer has not run. + * + * The guard: when a device-dst copy (or fill) op fails to execute this + * pass, its destination VA page is recorded here with a stamp. A GR launch + * whose data pages overlap a fresh record is NOT forwarded (the pump's + * bounded busy-defer retries it; the record clears when the op finally + * executes or the SVC_DSTPEND_MS window expires, at which point the + * historical forward stands). Keyed by VA page: the page identity is what + * the launch operand and the copy share, and it works even when neither + * side resolves. + */ +#define SVC_DSTPEND_MAX 64 +#define SVC_DSTPEND_MS 10000u +static struct { + u64 va_page; + unsigned long since; +} svc_dstpend[SVC_DSTPEND_MAX]; +static u32 svc_dstpend_n; +static u32 svc_dstpend_log; + +static void svc_dstpend_note(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + if (!page) + return; + for (i = 0; i < svc_dstpend_n; i++) { + if (svc_dstpend[i].va_page == page) { + /* + * r553: the window is ABSOLUTE from the first failed + * attempt for this page, never re-freshened by the + * retries. The retried decode of a copy that cannot + * resolve re-notes its destination on every pass; with + * a sliding stamp the record never expired and the r541 + * launch guard deferred the launch forever - r551 soak: + * victory_strided repeated "GR busy-defer timeout -> + * hold 8000 ms" every ~4 s for 427 s with the launch + * never forwarded (the copies' -ENOENT retries kept the + * note fresh). A record older than the window is + * refreshed only when a NEW failure episode starts. + */ + if (time_before(jiffies, svc_dstpend[i].since + + msecs_to_jiffies(SVC_DSTPEND_MS))) + return; + svc_dstpend[i].since = jiffies; + return; + } + } + if (svc_dstpend_n < SVC_DSTPEND_MAX) + i = svc_dstpend_n++; + else + i = (u32)(jiffies % SVC_DSTPEND_MAX); + svc_dstpend[i].va_page = page; + svc_dstpend[i].since = jiffies; +} + +/* + * r566: note the destinations of every COLLECTED device-dst copy that did + * not execute this pass, whenever the entry is being held (op_tr_pending). + * The r541 launch guard could only see copies that FAILED with -ENOENT + * (svc_dstpend_note at the failure site); the r568 mail failure is the + * missing case: the b H2D's COPY op was collected but never attempted while + * its entry was held for another operand, the launch's copy-in ran 0.6 s + * before the copy executed, and the engine computed c = a + 0 (the + * readback page held a's values). With the destination noted, the launch + * guard defers until the copy runs (its success path clears the record), + * bounded by the same dstpend TTL + busy-defer machinery as every other + * guard. + */ +static void svc_dstpend_note_ops(struct svc_memset_op *ops, u32 n, u32 base, + u64 *ops_done) +{ + u32 i; + + for (i = 0; i < n; i++) { + struct svc_memset_op *m = &ops[i]; + u32 seq = base + i; + + if (m->type != 1 || m->phys || m->dst < SVC_VAS_FLOOR) + continue; + if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) + continue; /* executed (this pass or earlier) */ + svc_dstpend_note(m->dst); + } +} + +static void svc_dstpend_clear(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_dstpend_n; i++) + if (svc_dstpend[i].va_page == page) + svc_dstpend[i].va_page = 0; +} + +static bool svc_dstpend_hit(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_dstpend_n; i++) { + if (svc_dstpend[i].va_page != page) + continue; + if (time_after_eq(jiffies, svc_dstpend[i].since + + msecs_to_jiffies(SVC_DSTPEND_MS))) { + svc_dstpend[i].va_page = 0; + return false; + } + return true; + } + return false; +} + +static bool svc_opfail_latched(u64 va) +{ + u32 i; + u32 free_i = SVC_OPFAIL_MAX; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) { + if (svc_opfail[i].va == va) { + if (time_after_eq(jiffies, + svc_opfail[i].since + + msecs_to_jiffies(SVC_OPFAIL_WINDOW_MS))) { + if (svc_opfail_log < 32) { + svc_opfail_log++; + pr_info("sema-svc: OPFAIL-LATCH va=0x%llx (>=%u ms of failures; op no longer holds its entry)\n", + va, SVC_OPFAIL_WINDOW_MS); + } + return true; + } + return false; + } + if (!svc_opfail[i].va && free_i == SVC_OPFAIL_MAX) + free_i = i; + } + if (free_i < SVC_OPFAIL_MAX) { + svc_opfail[free_i].va = va; + svc_opfail[free_i].since = jiffies; + } + return false; +} + static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, - u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base) + u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base, + u32 entry_idx, u64 pass_hash) { u32 i; static u64 bigop_log; svc.cur_ch = ch_h; + /* r484: one-shot per call; set only by the hold->pass transition. */ + svc_wait_repass = false; + /* + * m163: apply every queued scanner injection before resolving this + * pass's copy operands. Without this, a CE copy decoded in the window + * between the client's push and its pins being processed resolves + * through the previous client's stale translation (or fails), while + * its release is still applied - the client's cudaMemcpy "completes" + * with no data and the next GR launch computes from the stale vidmem + * page (streams2 r163: a1 held the previous client's h_a; the vadd + * output was a_stale + b_fresh). + */ + svc_handle_map_add(); + /* + * r336/r34x: honor every host-semaphore acquire (C56F SEM_EXECUTE + * op 0/2/3/4/5) captured for THIS ring entry before any of its work + * runs, so a copy cannot execute ahead of the dependency its push + * waits on (the streams2 readback-before-scale class). Decision + * ladder per wait: + * 1. saved value satisfies the op's condition -> pass; + * 2. the entry's own release, when its method offset precedes the + * acquire's (release-then-acquire, channel.c crossEngineSync), + * counts as the candidate value - the release method has + * already executed by the time the acquire is evaluated; + * an acquire that precedes its own trailing tracking release + * does NOT count it (nothing has installed that value yet); + * 3. drain the pending-translation release queue (releases OUR + * machinery is merely late about) and re-read once; + * 4. still unsatisfied -> hold the entry (release withheld through + * the op-pending machinery) for at most gsp_sema_wait_hold_ms + * per episode, then fail open loudly ('give-up': the value + * cannot be installed by firmware alone - it sits behind an + * upstream opdef/operand hold - so the old pass-through + * behaviour is safer than parking the channel forever). + * Unreadable semaphore: fail-open (the pre-r336 behaviour). + * The pass/hold/give-up decisions are logged as 'sema-wait' rows + * (op, payload, observed value, decision). + */ + if (svc_wait_n && gsp_sema_wait_gate) { + struct svc_wait_hold *gh; + bool hold = false; + u32 wi; + /* + * Kill switch: a wait that never satisfies (a counter our + * emulation does not advance the way the driver expects) + * would otherwise park entries until the sweep expires each + * one. After 2048 holds in a boot the gate fails open + * permanently (the pre-r336 behaviour) and says so once. + */ + if (svc_wait_held > 2048) { + if (svc_wait_log < 4096) { + svc_wait_log++; + pr_info("sema-wait: gate disabled after %llu holds (fail-open)\n", + svc_wait_held); + } + svc_wait_pass++; + goto svc_wait_done; + } + gh = svc_wait_hold_get(ch_h, true); + for (wi = 0; wi < svc_wait_n; wi++) { + const struct svc_wait_op *w = &svc_waits[wi]; + u32 cur = 0, waited_ms = 0; + bool ok = true, same_rel = false, pend = false; + + if (gh) + waited_ms = jiffies_to_msecs(jiffies - gh->since); + if (svc_sem_read(vgpu, w->va, &cur)) + continue; /* unreadable: fail-open */ + ok = svc_wait_satisfied(cur, w->op, w->payload); + if (!ok && svc_wait_rel_have && + svc_wait_rel_va == w->va && + svc_wait_rel_off < w->off) { + same_rel = true; + ok = svc_wait_satisfied(svc_wait_rel_payload, + w->op, w->payload); + } + if (!ok) { + svc_apply_pending(vgpu); + if (!svc_sem_read(vgpu, w->va, &cur)) + ok = svc_wait_satisfied(cur, w->op, + w->payload); + } + if (ok) { + if (svc_wait_pass_log < 256) { + svc_wait_pass_log++; + pr_info("sema-wait: pass ch=0x%x va=0x%llx op=%u pay=%u cur=%u same_rel=%u waited_ms=%u -> ops run this pass\n", + ch_h, w->va, w->op, w->payload, + cur, same_rel, waited_ms); + } + continue; + } + pend = svc_wait_pend_release(w->va); + if (waited_ms >= gsp_sema_wait_hold_ms) { + /* + * r613: a deferred GR launch extends the + * episode. The release this acquire waits for + * may be authored by a launch we are holding + * through our own bounded deferral (the r612 + * streams2 capture: the D2H readback's + * acquire on 0x20480ffe0 was given up at + * waited_ms=10333 while the vadd launches + * that release it sat deferred by the CE + * watermark for 15 s; the readback then ran + * before the kernels and the client read + * zeros). While a captured candidate is + * unoffered, "the value cannot be installed + * by firmware alone" is not yet true - keep + * holding, bounded by the defer cap plus + * margin. The client-side timeouts are all + * >= 60 s, so a <= 24 s episode cannot + * strand it. + */ + if (waited_ms < SVC_WAIT_EXT_CAP_MS && + svc_gr_forward_pending()) { + svc_wait_extended++; + if (svc_wait_ext_log < 32) { + svc_wait_ext_log++; + pr_info("sema-wait: extend ch=0x%x va=0x%llx op=%u pay=%u cur=%u waited_ms=%u -> hold (a GR forward is owed; the release is behind our own defer)\n", + ch_h, w->va, w->op, + w->payload, cur, + waited_ms); + } + } else { + /* + * Bounded episode expired: fail open for the + * WHOLE entry (one row, then the ops run). A + * per-wait continue here would ping-pong + * with a second unsatisfied wait: the first + * gave up and reset the shared clock before + * the second was judged, and the pair would + * hold forever 100 ms at a time. + */ + svc_wait_giveup++; + if (svc_wait_log < 2048) { + svc_wait_log++; + pr_info("sema-wait: give-up ch=0x%x va=0x%llx op=%u pay=%u cur=%u pend=%u waited_ms=%u -> fail-open (cannot be satisfied by firmware alone; ops run)\n", + ch_h, w->va, w->op, w->payload, + cur, pend, waited_ms); + } + if (gh) + gh->active = false; + hold = false; + svc_wait_pass++; + goto svc_wait_done; + } + } + hold = true; + if (svc_wait_log < 512) { + svc_wait_log++; + pr_info("sema-wait: hold ch=0x%x va=0x%llx op=%u pay=%u cur=%u same_rel=%u pend=%u waited_ms=%u -> entry held (acquire not satisfied)\n", + ch_h, w->va, w->op, w->payload, cur, + same_rel, pend, waited_ms); + } + break; /* one unsatisfied wait holds the entry */ + } + if (hold) { + svc_wait_held++; + svc_wait_hold_h = ch_h; + svc_wait_hold_idx = entry_idx; + /* r638: record the held entry for the CE watermark's + * mutual-cycle check. */ + if (gh) + gh->idx = entry_idx; + svc.op_tr_pending = 1; + return; + } + /* + * r484: the gate just PASSED for an entry it previously held - + * the dependency is installed now, so this pass's ops must + * re-run even if the once-only bitmap says they already did + * (they ran against the pre-dependency state). + */ + if (ch_h == svc_wait_hold_h && + entry_idx == svc_wait_hold_idx) { + svc_wait_repass = true; + svc_wait_hold_idx = 0xffffffffu; + } + svc_wait_pass++; + if (gh) + gh->active = false; + } +svc_wait_done: for (i = 0; i < n_ops; i++) { struct svc_memset_op *m = &ops[i]; u32 seq = ops_base + i; @@ -4858,18 +12791,70 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * ran; re-running a CE copy would read a staging buffer the * client has since refilled and overwrite finished data. */ - if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) { + if (!svc_wait_repass && ops_done && seq < 64 && + ((*ops_done >> seq) & 1)) { svc.ops_once_skipped++; + /* + * r456: this op already ran in an earlier pass. Any + * staging snapshot a later re-decode pass re-captured + * for it (the capture sites run on every decode) is + * a leak that the r444 APPLY-free used to sweep; with + * that free removed (r456 keeps the snapshot alive + * across retries) drop it here so it can never serve + * a re-push of the ring slot. + */ + if (m->type == 1 && m->bytes <= SVC_COPY_SNAP_MAX) + svc_copy_snap_free(m->dst, m->src, m->bytes, + ch_h, entry_idx, pass_hash); continue; } - if (m->type == 1) - rc = svc_copy(vgpu, m->dst, m->src, m->bytes, - m->phys, m->aperture, - m->src_phys, m->src_aperture); - else - rc = svc_fill(vgpu, m->dst, m->value, m->elem, - m->count, m->phys, m->aperture); + { + u64 ce_len = (m->type == 1) ? m->bytes : + (u64)m->elem * m->count; + u64 ce_done = 0; + + if (m->type == 1) + rc = svc_copy(vgpu, m->dst, m->src, m->bytes, + m->phys, m->aperture, + m->src_phys, m->src_aperture, + &ce_done, m->rows, m->dpitch, + m->spitch, ch_h, entry_idx, + pass_hash); + else + rc = svc_fill(vgpu, m->dst, m->value, m->elem, + m->count, m->phys, m->aperture, + &ce_done, m->rows, m->dpitch); + /* + * CEAUDIT (2026-10-18 r220): per-op audit of the CE + * data path - the software executor of every guest + * cudaMemcpy - at/above SVC_COPY_AUDIT_MIN, including + * the rc != 0 attempts the copy-level COPYAUDIT cannot + * see (the pre-loop resolution misses return before + * any byte moves: done=0x0). src/dst are the guest + * VAs of the op; done is the byte count that landed. + * Bounded per boot. + */ + if (ce_len >= SVC_COPY_AUDIT_MIN && + svc_ceaudit_n < SVC_CEAUDIT_MAX) { + svc_ceaudit_n++; + /* + * A failing op is retried by the hold and would + * spam: throttle the rc != 0 repeats; successful + * big ops log unconditionally (budget-bounded). + */ + if (rc) + pr_info_ratelimited("vgsp: CEAUDIT op=%s src=0x%llx dst=0x%llx len=0x%llx done=0x%llx rc=%d\n", + m->type == 1 ? "copy" : "fill", + m->type == 1 ? m->src : 0ull, + m->dst, ce_len, ce_done, rc); + else + pr_info("vgsp: CEAUDIT op=%s src=0x%llx dst=0x%llx len=0x%llx done=0x%llx rc=%d\n", + m->type == 1 ? "copy" : "fill", + m->type == 1 ? m->src : 0ull, + m->dst, ce_len, ce_done, rc); + } + } if ((rc == 0 || rc == SVC_COPY_FORWARDED) && ops_done && seq < 64) *ops_done |= 1ULL << seq; @@ -4896,11 +12881,28 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, */ if (rc == SVC_COPY_FORWARDED) continue; - if (rc == 0) + if (rc == 0) { svc.memsets_done++; - else if (rc == -ENOENT) + /* r507a: a success re-arms the failure window. */ + svc_opfail_clear(m->dst); + /* r541: the device page's writer has run; lift the + * launch-ordering guard for it. */ + svc_dstpend_clear(m->dst); + } else if (rc == -ENOENT) { + /* + * r541: a copy/fill that cannot run yet leaves its + * target page stale; a GR launch must not read it + * (see svc_dstpend_note). Record every unexecuted + * target - the VA-page key keeps the guard narrow. + */ + svc_dstpend_note(m->dst); + if (svc_dstpend_log < 64) { + svc_dstpend_log++; + pr_info("sema-svc: DSTPEND note va=0x%llx type=%u ap=%u (device write unexecuted; GR launches touching this page hold)\n", + m->dst, m->type, m->aperture); + } svc.memsets_nomap++; - else + } else svc.memsets_fail++; /* * ISSUE-0002 (r71/r72): a failed LARGE op leaves work owed on @@ -4917,9 +12919,20 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * (observed r72 c2) must not withhold a release that gates a GR * launch with a 5 s poll deadline. On expiry the release is * flushed and the op stays skipped (the pre-r62 fallback). + * m163 extends the hold to copy ops whose destination is a + * device buffer: a lost small H2D that silently "completes" + * corrupts the next launch's input (streams2 r163: the 128 KB + * a1 copy's release was applied with no copy op ever run; the + * vadd computed a_stale + b and the client read the wrong c). + * A small failing device copy now costs a bounded hold and the + * retried decode runs it once the pins land - never silent + * corruption. Small failing memsets keep the old trade. */ if (rc != 0 && rc != SVC_COPY_FORWARDED && - m->bytes >= (256u << 10)) + (m->bytes >= (256u << 10) || + (m->type == 1 && m->dst >= SVC_VAS_FLOOR) || + (m->type == 0 && !m->phys && m->dst >= SVC_VAS_FLOOR)) && + !svc_opfail_latched(m->dst)) svc.op_tr_pending = 1; if (gsp_sema_dbg && m->bytes == 256 && svc.sem256_log < 64) { svc.sem256_log++; @@ -4930,7 +12943,7 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, } if (gsp_sema_dbg && ((rc == 0 && svc.memsets_done <= gsp_sema_ops_log) || - (rc != 0 && svc.memsets_fail + svc.memsets_nomap <= 64))) + (rc != 0 && svc.memsets_fail + svc.memsets_nomap <= 2048))) pr_info("sema-svc: %s dst=0x%llx value=0x%llx elem=%u count=%u bytes=%llu phys=%u ap=%u src=0x%llx srcphys=%u srcap=%u rc=%d h=0x%x\n", m->type == 1 ? "memcopy" : "memset", m->dst, m->value, m->elem, m->count, @@ -4948,7 +12961,7 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * read-back signature: always keep room for it past the * general trace budget so the DtoH's src/dst/rc is on record. */ - if ((svc.op_trace_log < 160 && + if ((svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && (time_before(jiffies, svc.op_trace_until) || m->dst >= SVC_VAS_FLOOR || (m->type == 1 && !m->src_phys && m->src >= SVC_VAS_FLOOR))) || @@ -4992,18 +13005,20 @@ static bool svc_stream_parse(const u8 *buf, u32 len) memcpy(&hdr, buf + off, 4); if (hdr == 0) { /* - * UVM's large page-table pushes begin with a - * zero-filled header region (observed 0x1004 - * bytes) before the first method, so zeros before - * the first method are leading padding; zeros - * after one are trailing padding and end the - * stream. + * NVC06F_DMA_NOP is 0x00000000: a zero word is + * NOP padding. UVM's large page-table pushes begin + * with a zero-filled header region (observed 0x1004 + * bytes) and the GR submit pushes carry zero gaps + * between method groups (observed r715: the 1432- + * and 17316-byte pushes pad between the context + * setup and the dispatch groups), so zeros are + * skipped anywhere in the stream, not only before + * the first method or at the tail. The exact-end + * requirement and the valid-header walk below are + * the false-positive arbiters. */ - if (!methods) { - off += 4; - continue; - } - break; /* trailing zero padding */ + off += 4; + continue; } op = (hdr >> 29) & 0x7; count = (hdr >> 16) & 0xfff; @@ -5023,16 +13038,27 @@ static bool svc_stream_parse(const u8 *buf, u32 len) return false; off += 4 + count * 4; break; + case 0: /* GRP0_USE_TERT: SET/STORE/USE_SUB_DEV_MASK */ + case 2: /* GRP2_USE_TERT */ case 4: /* IMMD_DATA_METHOD: header only */ - off += 4; - break; case 7: /* END_PB_SEGMENT: header only */ + /* + * r715: the class header defines op 0 as + * GRP0_USE_TERT (0) and op 2 as GRP2_USE_TERT - + * 4-byte header-only formats. The GR submit + * pushes emit 0x0001fff0 (SET_SUBDEVICE_MASK, + * value 0xfff) around their semaphore groups; the + * old parser rejected the first one on every big + * push, so the whole push read implausible and + * the entries were skipped without decode + * (session 124's consume-without-decode). + */ off += 4; break; default: return false; } - if (++methods > 1024) + if (++methods > 8192) return false; } while (off < len && buf[off] == 0) @@ -5077,6 +13103,8 @@ static u64 svc_rel_va_seen; * on every decoded release regardless of its target. */ static u32 svc_rel_count; +static u32 svc_fake_rel_log; /* r576b DIAG: sema-fake witness budget */ +static u32 svc_ce_foreign_log; /* r577: foreign-release refusal witness */ static u32 svc_dbg_streams; static u32 svc_chan_put_log; /* ISSUE-0002 (r55): per-rank put-snapshot throttle (ch is a local copy). */ @@ -5089,6 +13117,127 @@ static unsigned long svc_put_log_at[32]; */ static u32 svc_userd_dbg; static u32 svc_userd_dbg_ch[32]; +/* SESSION 134: per-CE-channel budget of the first-submission ring captures + * (see svc_ce_firstsub_capture) and the start of an unlatched CE channel's + * completion hold (see svc_ce_hold_ms). Session state; reset by + * svc_session_flush. */ +static u32 svc_ce_firstsub_n[32]; +static unsigned long svc_ce_hold_since[32]; +/* + * SESSION 134c: the FB offset each CE rank's first-submission capture latched + * (0 = none). The shadow scan is rank-blind on its own - it takes the first + * ring-shaped page in the shadow - and on r779 that latched rank 0's block + * (fb 0x80000, userd 0x90000) for rank 1 as well (both channels reported + * scan_hits=1 latch=1 off=0x80000). The rank-1 channel's first release was + * then decoded from rank 0's ring/page and stale-dropped (cur=1 already from + * rank 0's applied release), so the client's rank-1 tracker wait never + * completed and the client spun forever (victory_mail stall, three boots). + * The capture now prefers the rank's convention offset and refuses any + * offset another CE rank already latched. Session state; reset by + * svc_session_flush. + */ +static u64 svc_ce_firstsub_off[32]; +/* + * SESSION 144: FB ring-block ownership ACROSS engine classes. The GR + * autopin fallback scan (SESSION 133c) and the CE first-submission + * capture (SESSION 134c) each solved the same problem only within their + * own class; a live-but-foreign ring block passes every shape gate, so a + * channel whose own ring is not written yet latches another channel's + * block permanently and never reads its own entries. Observed on + * r805-r808: every CUDA GR rank's userd-shape scan latch resolved the CE + * rank-0 block (fb 0x80000, userd 0x90000; e.g. + * "auto-pinned GR ring h=0x5c000019 rank=0 ... -> fb 0x80000 (userd-shape + * scan ... entry0 va=0x120000000 len=40 put=35 get=35)" while the CE + * handle decoded the identical entry[35] raw words from pa=0x10350641c), + * so no GR launch was ever captured and victory_mail hung after DEVICE. + * A channel that cannot see its own block must stay unlatched (and retry + * next pass) rather than steal a foreign one. Keyed by the FB ring + * offset; claims are session state, reset by svc_session_flush. + */ +struct svc_fb_claim { + u32 h; /* owning channel handle */ + u64 ring_off; /* FB ring offset */ + u64 userd_off; /* FB USERD offset (0 = unknown) */ +}; +#define SVC_CE_RING_FB_BASE 0x80000ULL +#define SVC_CE_RING_FB_STRIDE 0x20000ULL +static struct svc_fb_claim svc_fb_claim_tab[40]; +static u32 svc_fb_claim_n; + +static void svc_fb_claim_add(u32 h, u64 ring_off, u64 userd_off) +{ + u32 i; + + for (i = 0; i < svc_fb_claim_n; i++) { + if (svc_fb_claim_tab[i].h == h) { + svc_fb_claim_tab[i].ring_off = ring_off; + svc_fb_claim_tab[i].userd_off = userd_off; + return; + } + } + if (svc_fb_claim_n < ARRAY_SIZE(svc_fb_claim_tab)) { + svc_fb_claim_tab[svc_fb_claim_n].h = h; + svc_fb_claim_tab[svc_fb_claim_n].ring_off = ring_off; + svc_fb_claim_tab[svc_fb_claim_n].userd_off = userd_off; + svc_fb_claim_n++; + } +} + +/* + * true = the candidate FB block (ring at ring_off, USERD at userd_off if + * known) is already claimed by a DIFFERENT channel, or is a live CE + * rank's convention block even before that rank latched, i.e. it must not + * be latched or stolen. Same-channel re-latches pass. + */ +static bool svc_fb_claim_foreign(struct nv_vgpu_device *vgpu, u32 self_h, + u64 ring_off, u64 userd_off) +{ + u32 i; + unsigned long flags; + int k; + + for (i = 0; i < svc_fb_claim_n; i++) { + if (svc_fb_claim_tab[i].h == self_h) + continue; + if (svc_fb_claim_tab[i].ring_off == ring_off) + return true; + if (userd_off && svc_fb_claim_tab[i].userd_off == userd_off) + return true; + } + /* + * SESSION 144b: reserve every live CE rank's convention block even + * before that rank has latched. r809: CE rank 1's block at fb + * 0xa0000 was still unclaimed when GR rank 0's userd-shape scan ran + * (rank 1 latched through the plain autopin path, not the + * first-submission capture) and the GR channel stole it again. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (k = 0; k < SVC_MAX_CHANNELS; k++) { + u64 conv; + + if (!svc.chan[k].valid || svc.chan[k].vgpu != vgpu || + !svc.chan[k].engine_type || + svc.chan[k].h_object == self_h || + svc.chan[k].ce_index >= 32) + continue; + conv = SVC_CE_RING_FB_BASE + + SVC_CE_RING_FB_STRIDE * svc.chan[k].ce_index; + if (ring_off == conv || + (userd_off && userd_off == conv + 0x10000)) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + return true; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +static void svc_fb_claim_reset(void) +{ + svc_fb_claim_n = 0; + memset(svc_fb_claim_tab, 0, sizeof(svc_fb_claim_tab)); +} + /* * ISSUE-0002 (r47): deferred FB sweep for a CE channel whose layout USERD * probe fails at discovery. Session/reset-scoped, scheduled on the first @@ -5131,6 +13280,33 @@ struct svc_grce_rec { static struct svc_grce_rec svc_grce_dst_ring[SVC_GRCE_DST_MAX]; static u32 svc_grce_dst_ring_next; +/* + * SESSION 153-CONTINUED-88: newest captured inline-write payload landing + * exactly at `dst_va` with at least `min_len` bytes. Newest-first scan so a + * fresh upload always wins over a stale ring entry. Copies the payload into + * the caller's buffer (buf_sz bounds the copy) and returns true on a hit. + */ +static bool svc_grce_dst_newest(u64 dst_va, u32 min_len, u32 *len, u8 *buf, + u32 buf_sz) +{ + u32 ri; + + for (ri = 0; ri < SVC_GRCE_DST_MAX; ri++) { + u32 idx = (svc_grce_dst_ring_next + SVC_GRCE_DST_MAX - 1u - + ri) % SVC_GRCE_DST_MAX; + const struct svc_grce_rec *r = &svc_grce_dst_ring[idx]; + + if (r->len < min_len || !r->dst_va) + continue; + if (r->dst_va != dst_va) + continue; + *len = min_t(u32, r->len, buf_sz); + memcpy(buf, r->data, *len); + return true; + } + return false; +} + static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va); static void svc_pending_add(u64 va, u32 payload, u32 op); static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, @@ -5177,6 +13353,8 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, bool skip; int i; u32 dbg = 0; + u64 rel_pa = 0; + u64 cnt_pa = 0; if (!h || !payload) { /* @@ -5214,12 +13392,23 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, * FORENSICS-grv7-counter F1). Fall back to counter_pa * when the release slot could not be resolved. */ - pa = svc.chan[i].release_pa ? - svc.chan[i].release_pa : svc.chan[i].counter_pa; + rel_pa = svc.chan[i].release_pa; + cnt_pa = svc.chan[i].counter_pa; + pa = rel_pa ? rel_pa : cnt_pa; break; } } spin_unlock_irqrestore(&svc.chan_lock, flags); + /* + * C1 (GAP-1 H3/H4): was the apply entered, and with which slot? + * Budget 512/session (reset in svc_session_flush); the per-branch + * prints below stay on their own counter_rel_dbg budget. + */ + if (gsp_sema_dbg && svc.crel_entry_dbg < 512) { + svc.crel_entry_dbg++; + pr_info("sema-diag: crel-entry h=0x%x va=0x%llx pay=%u inc=%d rel_pa=0x%llx cnt_pa=0x%llx\n", + h, sem_va, payload, inc ? 1 : 0, rel_pa, cnt_pa); + } if (!pa) { /* * Pool (UVM tracking) semaphores have no counter slot. The @@ -5256,7 +13445,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, wrc = svc_sem_write(vgpu, h, sem_va, payload); if (!wrc) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-apply h=0x%x va=0x%llx pay=%u cur=%u -> pool rescue\n", @@ -5330,6 +13519,15 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, return; } nv = inc ? cur + payload : payload; + /* + * r378: the slot PA was learned from the release VA (or registered + * from the client's pagemap) earlier; re-validate it against the + * current pin before the 4-byte release lands, so a slot the guest + * has since freed/re-homed is never written through the stale page. + */ + if (sem_va && !gsp_sema_write_guard(sem_va, pa, SVC_APERTURE_SYS, + "counter-rel")) + return; if (svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, &nv, sizeof(nv))) { if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; @@ -5341,7 +13539,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, svc.releases_applied++; svc.counter_rel_applied++; svc.gr_rel_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-apply h=0x%x pa=0x%llx pay=%u cur=%u -> written\n", @@ -5424,7 +13622,7 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu, svc.releases_applied++; svc.counter_rel_applied++; svc.gr_rel_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-flush h=0x%x pa=0x%llx pay=%u cur=%u -> written\n", @@ -5464,34 +13662,108 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu, static struct { u64 va_page; u64 pa_page; + /* + * SESSION 153-CONTINUED-38: the VAS the page was derived for. The + * counter VA window (0x20400fXXX) is a fixed per-client layout, so + * two clients (the RC scrubber and the DDX) register the SAME VA and + * a VA-keyed cache/derive hands the second one the first one's page; + * that is the slot-ownership collision CONT-37 measured. 0 = any. + */ + u32 h_vas; } svc_cnt_der[SVC_CNT_DER_MAX]; static u32 svc_cnt_der_n; +static u32 svc_cnt_der_log; static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u64 off = va & (SEMA_PAGE_SIZE - 1); u64 pa = 0; - int ap = SVC_APERTURE_SYS; u32 probe = 0; u32 i; bool fresh = true; for (i = 0; i < svc_cnt_der_n; i++) { - if (svc_cnt_der[i].va_page == page) { + if (svc_cnt_der[i].va_page == page && + (!svc_cur_vas || !svc_cnt_der[i].h_vas || + svc_cnt_der[i].h_vas == svc_cur_vas)) { pa = svc_cnt_der[i].pa_page | off; fresh = false; break; } } if (fresh) { - if (svc_walk_vaspaces(vgpu, page, &pa, &ap) && - svc_walk_discovered(vgpu, page, &pa, &ap) && - svc_targeted_resolve(vgpu, page, &pa)) - return -ENOENT; - if (ap != SVC_APERTURE_SYS) + u64 p = 0; + int a = SVC_APERTURE_SYS; + bool ok = false; + const char *src = "none"; + + /* + * m177b: first READABLE source wins. The previous chain kept + * the first source that merely PRODUCED a translation and then + * failed -EIO on an unreadable page without trying the rest: + * the published directory can yield a stale/incorrect chain + * for a counter VA while the rm_mem-content scan would resolve + * it (r194: the counter page 0x20400f000 never resolved; its + * walk hit probed unreadable and targeted was never reached). + */ + /* + * SESSION 153-CONTINUED-38: the channel's OWN page tables + * answer first. The global walk sources can only name one + * page for the shared counter VA (0x20400fXXX is a fixed + * per-client layout), so a second client's slot derivation + * collided with the first client's page and its registration + * was rejected forever (CONT-37: "slot owned by another + * channel"). The per-channel walk (svc_cur_vas, set by the + * pump for the channel being serviced) names the page the + * client itself polls. + */ + if (!ok && svc_cur_vas && + !svc_chanvas_lookup(vgpu, page, &p, &a) && + a == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "chanvas"; + } + if (!ok && !svc_walk_vaspaces(vgpu, page, &p, &a) && + a == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "vaspaces"; + } + if (!ok && !svc_walk_discovered(vgpu, page, &p, &a) && + a == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "discovered"; + } + if (!ok && !svc_targeted_resolve(vgpu, page, &p) && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "targeted"; + } + if (!ok) { + if (svc_cnt_der_log < 16) { + svc_cnt_der_log++; + pr_info("sema-chan: counter page 0x%llx unreadable via vaspaces/discovered/targeted; slot stays pending\n", + page); + } return -ENOENT; - pa = (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + } + pa = (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + if (svc_cnt_der_log < 16) { + svc_cnt_der_log++; + pr_info("sema-chan: counter page 0x%llx resolved via %s (pa=0x%llx)\n", + page, src, pa); + } } /* Read the slot before trusting the mapping: an unreadable page is * not registered, so the caller keeps the "slot pending" fallback. */ @@ -5501,6 +13773,7 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) svc_cnt_der[svc_cnt_der_n].va_page = page; svc_cnt_der[svc_cnt_der_n].pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_cnt_der[svc_cnt_der_n].h_vas = svc_cur_vas; svc_cnt_der_n++; } *pa_out = pa; @@ -5508,15 +13781,73 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) } /* + * SESSION 153-CONTINUED-38: counter-slot disambiguation for a channel with + * no usable hVASpace (kernel channels register vas=0x0, so svc_cur_vas + * cannot answer). Two clients' channels can share a rank - the rank formula + * then names the SAME slot VA in each client's own address space - and the + * global walk sources answer with only one client's page, so the second + * client's autoreg collided with the first client's registered slot and was + * rejected forever (CONT-37: the GL client's beef0100 rank=2 vs the X + * client's bfef0002, both 0x2132acfd0). Walk EVERY registered VAS and + * return the first page that is readable AND different from avoid_pa: that + * is the second client's own page for the same VA. + */ +static int svc_counter_derive_not(struct nv_vgpu_device *vgpu, u64 va, + u64 avoid_pa, u64 *pa_out) +{ + unsigned long flags; + struct svc_vaspace local[SVC_MAX_VASPACES]; + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 off = va & (SEMA_PAGE_SIZE - 1); + int i, n = 0; + + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid) + local[n++] = svc.vas[i]; + spin_unlock_irqrestore(&svc.vas_lock, flags); + + for (i = 0; i < n; i++) { + u64 p = 0; + int a = SVC_APERTURE_SYS; + u32 probe; + + if (svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, + page, local[i].ap, &p, &a, false)) + continue; + if (a != SVC_APERTURE_SYS) + continue; + p = (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + if ((p & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (avoid_pa & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + if (svc_read_ap(vgpu, p, SVC_APERTURE_SYS, &probe, + sizeof(probe))) + continue; + *pa_out = p; + return 0; + } + return -ENOENT; +} + +/* * Auto-register a GR channel's completion-counter slot from the observed - * layout, without waiting for a readable push: the counter records form a - * 16-byte-strided array in the client's counter page, allocated in channel - * order from the low end (SA-COUNTER FORENSICS-grv7-counter: the ascending - * slot fec0 + r*0x10 is the polled completion word; the push's REPORT_SEM - * release writes a DIFFERENT (descending) family and is tracked separately - * in release_pa). The slot PA comes from a host-verified pin or from the - * in-kernel structural derivation (svc_counter_derive), so the write target - * is a page the client actually polls. + * layout, without waiting for a readable push: the client zips its GR + * channels in ascending handle order onto DESCENDING 16-byte slots, so the + * rank-r channel's own completion word - the slot its pushes' REPORT_SEM + * release names, verified for all 20 ranks - is + * 0x20400fec0 + (SVC_GR_COUNTER_TOTAL-1-r)*0x10. The ascending formula + * (F1/eea5ea93) registered the completion word of the PAIRED rank (19-r) + * instead, so svc_counter_publish wrote each channel's ring PUT into + * another channel's waited word: the busy ctx-init channel rank 16 + * (h=0x5c000059, PUT~94) overwrote rank 3's completion word 0x20400ffc0 + * from 1 to 94 (witness `pub-adv h=0x5c000059 pa=...fc0 val=94 cur=1`), + * after which the rank-3 stream's own release (payload 1) was + * forward-only-skipped and its sync never completed (evtest3 + * cudaStreamSynchronize(st[2]); victory_streams_events). The slot PA comes + * from a host-verified pin or from the in-kernel structural derivation + * (svc_counter_derive), so the write target is a page the client actually + * polls. */ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, struct svc_channel *ch, int n_gr) @@ -5529,7 +13860,8 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, return; if (ch->gr_index >= n_gr) return; - va = 0x20400fec0ULL + (u64)ch->gr_index * 0x10ULL; + va = 0x20400fec0ULL + + (u64)(SVC_GR_COUNTER_TOTAL - 1 - ch->gr_index) * 0x10ULL; if (ch->counter_pa) { /* * FORENSICS-v2 anomaly 6: a slot derived in an earlier pass @@ -5613,6 +13945,47 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, break; } } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (collision) { + /* + * SESSION 153-CONTINUED-38: the slot VA is shared by + * rank across clients (each client's rank-r channel + * polls 0x20400fec0+(TOTAL-1-r)*0x10 in its OWN VAS), + * so a collision with another client's registered slot + * means the global derive answered with the OTHER + * client's page. Re-resolve through every registered + * VAS and keep the first readable page that differs - + * that page is this client's own slot. Only when no + * such page exists is the collision real and the + * registration must stay pending. + */ + u64 pa2 = 0; + + if (!svc_counter_derive_not(vgpu, va, pa, &pa2)) { + bool coll2 = false; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (j = 0; j < SVC_MAX_CHANNELS; j++) { + if (svc.chan[j].valid && + svc.chan[j].vgpu == vgpu && + svc.chan[j].h_object != + ch->h_object && + svc.chan[j].counter_pa == + (pa2 & ~0x3ULL)) { + coll2 = true; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!coll2) { + pr_info("sema-chan: counter auto h=0x%x rank=%u pa=0x%llx (own-VAS page; contender owner page 0x%llx kept)\n", + ch->h_object, ch->gr_index, + pa2 & ~0x3ULL, pa); + pa = pa2 & ~0x3ULL; + collision = false; + } + } + } if (!collision) { /* * Persist the registration into the authoritative @@ -5629,6 +14002,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, * hung. The authoritative store makes the * registration durable and the log one-shot. */ + spin_lock_irqsave(&svc.chan_lock, flags); ch->counter_pa = pa; for (j = 0; j < SVC_MAX_CHANNELS; j++) { if (svc.chan[j].valid && @@ -5638,8 +14012,8 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, break; } } + spin_unlock_irqrestore(&svc.chan_lock, flags); } - spin_unlock_irqrestore(&svc.chan_lock, flags); if (collision) { pr_info_ratelimited("sema-chan: counter auto h=0x%x rank=%u pa=0x%llx rejected (slot owned by another channel; not installed)\n", ch->h_object, ch->gr_index, pa); @@ -5658,12 +14032,14 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, /* * Learn a GR channel's REPORT_SEM release slot from a release VA the push * carried (0x20400fXX observed). This is the slot the release write - * targets (release_pa); it is deliberately NOT the completion-counter slot - * (counter_pa), which the publish advances and which the client polls - * (SA-COUNTER FORENSICS-grv7-counter F1: green wrote both families of the - * counter page). An injected pin is the authoritative source; on a miss the - * slot is derived in-kernel (svc_counter_derive) so the release is authored - * immediately rather than waiting for the external pin watcher. + * targets (release_pa) and - per the descending layout the autoreg above + * documents - the channel's own polled completion word, the same slot the + * autoreg registers as counter_pa; keeping it in its own field preserves + * the push-named truth even when the registration derive fails, and + * svc_counter_publish prefers it. An injected pin is the authoritative + * source; on a miss the slot is derived in-kernel (svc_counter_derive) so + * the release is authored immediately rather than waiting for the external + * pin watcher. */ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va) { @@ -5708,6 +14084,13 @@ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va) pr_info("sema-chan: release learned h=0x%x pa=0x%llx (from release va=0x%llx)\n", h, pa & ~0x3ULL, sem_va); } + /* + * r378: keep the VA the slot was learned from so a + * late write to release_pa (the stale-recover zero) + * can re-validate the target through + * gsp_sema_write_guard before it lands. + */ + svc.chan[i].release_va = sem_va; break; } } @@ -5740,7 +14123,7 @@ static bool svc_rel_apply_906f(struct nv_vgpu_device *vgpu, cur = (op == HOST_SEM_OP_REDUCTION) ? cur + payload : payload; if (!svc_sem_write(vgpu, ch->h_object, va, cur)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } } @@ -5768,7 +14151,7 @@ static bool svc_rel_apply_reportsem(struct nv_vgpu_device *vgpu, svc.pushes_pending++; if (!svc_sem_write(vgpu, ch->h_object, va, payload)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } return false; @@ -5779,7 +14162,33 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, u32 payload, u32 cur, bool pending, bool inc, bool inc_apply) { - if (pending || inc_apply) { + /* + * P4 (GAP-1 H3): the exact-slot counter delivery runs before every + * path that can early-return, so a GR completion-counter release can + * never be skipped by the generic write's outcome. The generic + * write/learn tail below is skipped entirely for counter VAs: an + * increment release is not idempotent, so it must apply exactly once + * (the early return here is that exactly-once edge). + */ + if (ch->engine_type == 0 && svc_counter_va(sem_va)) { + if (pending || inc_apply) + svc.pushes_pending++; + svc_counter_learn(vgpu, ch->h_object, sem_va); + svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, + inc && inc_apply); + return false; + } + /* + * r506w: generic absolute releases are forward-only by VALUE, whichever + * pass discovers the gap. The recorded pending bit came from the + * decode's read of the release word, and that read can be blinded by a + * stale/crossed landing (r506 witness: a withheld tracker release went + * to the flush with pending=0 while the fresh read still showed 0, and + * the release was then dropped by the counter fallback below - the + * client's tracker never advanced and its stream ground at the 10 s + * bound per push). Increments keep the one-shot claim gate (`inc_apply`). + */ + if (pending || inc_apply || (!inc && payload > cur)) { svc.pushes_pending++; if (inc) cur += payload; @@ -5790,19 +14199,41 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, * generic translation can be a stale content-scanned page; * those releases go through the injected-pin-backed exact slot * (svc_counter_learn / svc_counter_release_apply) below. + * Counter VAs are handled by the early block above; the check + * stays as belt-and-braces for the generic write. */ if (ch->engine_type == 0 && svc_counter_va(sem_va)) { /* handled by the exact-slot path */ } else if (!svc_sem_write(vgpu, ch->h_object, sem_va, cur)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } } - if (ch->engine_type == 0) - svc_counter_learn(vgpu, ch->h_object, sem_va); - svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, - inc && inc_apply); + /* + * r506w: the exact-slot counter fallback is for COUNTER VAs only. Its + * `release_pa ?: counter_pa` fallback writes (and forward-only-skips + * against) the channel's REGISTERED slot, which belongs to a different + * word - the r506 witness: releases to the tracker VA 0x20480fef0 were + * skipped against the polluted PUT counter (cur=96) and lost. For a + * non-counter VA the generic write above is the delivery path. + */ + if (svc_counter_va(sem_va)) { + if (ch->engine_type == 0) + svc_counter_learn(vgpu, ch->h_object, sem_va); + svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, + inc && inc_apply); + } + /* + * Stall-class fix (R2): when the decode said this absolute release is + * genuinely pending and every write attempt above failed without + * queueing, the release must not vanish - retry it through the pending + * pass. Increments stay excluded (their one-shot claim makes + * re-queueing unsafe until a diag run proves otherwise). + */ + if (pending && !inc && !(ch->engine_type == 0 && + svc_counter_va(sem_va))) + svc_pending_add(sem_va, payload, 0); return false; } @@ -5906,6 +14337,93 @@ static void svc_fwd_rel_direct_note(u32 h, u64 va, u32 payload) * mapped to its register offset. Each completed QMD resets the accumulator, * so a push may carry more than one launch. */ +/* + * m164: per-decode QMD witness for the CLASS-2 no-capture hold. Set to the + * max number of QMD words any accumulator saw during the current + * svc_decode_stream() call; reset at its top and read by the entry loop's + * no-capture check. Service-thread only. + */ +static u32 svc_gr_qmd_seen; +static u64 svc_gr_nocap_log; + +/* + * r383: has a launch with this push VA ever been captured? Scans the live + * candidate ring, the recent-capture history and the r401 session-wide + * (push_va, hash) seen table. Used by the replayed-QMD dispatch below: a + * replay whose push was never captured is a launch that would otherwise be + * silently dropped (the streams2 second-vadd class); a replay of a + * captured/offered push is the exactly-once no-op. The hash is required + * for the seen table so a later, DIFFERENT push at the same VA (a loop + * iteration: same entry words, new content) is never misclassified as + * seen - the caller only consults this on a true replay anyway. + */ +static bool svc_gr_cand_push_seen(u64 push_va, u64 hash) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + svc_gr_cand_ring[i].push_va == push_va) + return true; + for (i = 0; i < SVC_GR_CAP_HIST; i++) + if (svc_gr_cap_hist[i] == push_va) + return true; + return svc_gr_seen_lookup(push_va, hash); +} + +/* + * r391f: once-only identity for GRCE inline writes. A push's + * LOAD_INLINE_DATA runs author bytes into the vidmem shadow; a re-decode of + * the same push (replay/rescue/rehold) must not re-run them - the target + * page may have been superseded in between (victory_reduce: the client's + * cudaMemset inline write re-zeroed the accumulator after the launch's + * copy-out, the sum=0 class). Keyed by (push_va, push content hash, dst, + * bytes): the release payload inside every push makes the hash unique per + * push, so a new client's identical-looking write never collides. Session + * scoped (cleared by svc_session_flush). Service-thread only. + * + * Sizing (r398): the ring must outlive the gap between a push's first + * execution and its next re-decode. The observed failing case had 76 + * inline writes between the two, which evicted a 64-slot table and let + * the memset re-run; 8192 slots (256 KB) gives a ~100x margin over the + * busiest observed burst. + */ +#define SVC_INLINE_ONCE_SLOTS 8192 +static struct { + u64 push_va; + u64 hash; + u64 dst; + u32 bytes; +} svc_inline_once[SVC_INLINE_ONCE_SLOTS]; +static u32 svc_inline_once_next; + +static bool svc_inline_once_seen(u64 push_va, u64 hash, u64 dst, u32 bytes) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_INLINE_ONCE_SLOTS; i++) + if (svc_inline_once[i].push_va == push_va && + svc_inline_once[i].hash == hash && + svc_inline_once[i].dst == dst && + svc_inline_once[i].bytes == bytes) + return true; + return false; +} + +static void svc_inline_once_store(u64 push_va, u64 hash, u64 dst, u32 bytes) +{ + u32 i = svc_inline_once_next++ % SVC_INLINE_ONCE_SLOTS; + + svc_inline_once[i].push_va = push_va; + svc_inline_once[i].hash = hash; + svc_inline_once[i].dst = dst; + svc_inline_once[i].bytes = bytes; +} + static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, struct svc_push_state *st, u32 addr, u32 count, const u8 *data, const u8 *push, u32 push_len, @@ -5935,8 +14453,21 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } } } - if (st->qmd_have == 3) - st->qmd_dev_va = (((u64)st->qmd_hi << 32) | st->qmd_lo) << 8; + if (st->qmd_n > svc_gr_qmd_seen) + svc_gr_qmd_seen = st->qmd_n; + /* + * SESSION 127 C5C0 translator: the Vulkan driver writes + * SET_INLINE_QMD_ADDRESS_B only (the upper half stays absent for a + * <4G QMD; observed beec0100 idx23: `0x31c = 0x4b85f` then a 64-word + * 0x320 run). Accept the B-only form with an implicit zero upper + * half - the old A+B requirement left qmd_dev_va at 0 and the + * candidate logged with qmd=0x0 (the field is identity/logging + * only, but the 0x0 read confused every prior session's forensics). + */ + if (st->qmd_have & 2) + st->qmd_dev_va = + (((u64)((st->qmd_have & 1) ? st->qmd_hi : 0) << 32) | + st->qmd_lo) << 8; if (st->qmd_n == SVC_QMD_WORDS) { /* * Static: a 16-entry record array is ~1.2 KB and this is @@ -5958,23 +14489,150 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, svc_grce_dst_ring_next : SVC_GRCE_DST_MAX; if (have > GSP_BRIDGE_INLINE_MAX) have = GSP_BRIDGE_INLINE_MAX; - for (i = 0; i < have; i++) { - u32 idx = (svc_grce_dst_ring_next - 1 - i) % - SVC_GRCE_DST_MAX; - const struct svc_grce_rec *rec = &svc_grce_dst_ring[idx]; - - recs[n].dst_va = rec->dst_va; - recs[n].len = rec->len; - memcpy(recs[n].data, rec->data, rec->len); - n++; + /* + * SESSION 127 C5C0 translator: the rec ring is session-wide + * and mixes every client's LOAD_INLINE_DATA destinations. A + * stale rec whose page this QMD does not name made the fill + * refuse the whole launch (GR fill ENOENT + * va=0x80412000 qmd_word=NONE covered=1 - the Vulkan + * client's cancel page inherited from the driver-init + * phase). Keep only recs whose page is one of the pages the + * captured QMD names (the QMD buffer, the parameter bank, + * the program, the pointer pairs) - exactly the pages the + * backend's inline-served fill can use; keep everything when + * the QMD names none (fail-open). + */ + { + u64 keep[12]; + u32 nkeep = 0, k; + + if (st->qmd_dev_va) + keep[nkeep++] = st->qmd_dev_va & + ~(u64)(SEMA_PAGE_SIZE - 1); + keep[nkeep++] = + ((u64)st->qmd[32] | + ((u64)(st->qmd[33] & 0x1ffff) << 32)) & + ~(u64)(SEMA_PAGE_SIZE - 1); + keep[nkeep++] = + ((u64)st->qmd[48] | + ((u64)(st->qmd[49] & 0x1ffff) << 32)) & + ~(u64)(SEMA_PAGE_SIZE - 1); + for (k = 0; k < SVC_GR_POINTER_MAX - 1 && nkeep < 12; k++) { + u32 w = 34 + 2 * k; + u64 pv = (u64)st->qmd[w] | + ((u64)(st->qmd[w + 1] & 0x1ffff) + << 32); + + if (pv) + keep[nkeep++] = pv & + ~(u64)(SEMA_PAGE_SIZE - 1); + } + for (i = 0; i < have; i++) { + u32 idx = (svc_grce_dst_ring_next - 1 - i) % + SVC_GRCE_DST_MAX; + const struct svc_grce_rec *rec = + &svc_grce_dst_ring[idx]; + u64 rpage = rec->dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1); + bool keepit = !nkeep; + u32 j; + + for (j = 0; j < nkeep; j++) + if (keep[j] == rpage) { + keepit = true; + break; + } + /* + * SESSION 127 (gated): a stale session-wide + * rec whose page resolves nowhere made the + * fill refuse the whole launch; keep it when + * it resolves (the CUDA inline-served + * pages), drop only unresolvable unnamed + * ones. With the translator off, the + * historical keep-all behaviour stands. + */ + if (!keepit && gsp_sema_gr_xlate) { + u64 rpa = 0; + u32 rap = SVC_APERTURE_SYS; + + if (svc_resolve_va(vgpu, rpage, &rpa, + &rap) && + svc_walk_vaspaces(vgpu, rpage, + &rpa, &rap)) + continue; + } else if (!keepit) { + continue; + } + recs[n].dst_va = rec->dst_va; + recs[n].len = rec->len; + memcpy(recs[n].data, rec->data, rec->len); + n++; + } + } + /* + * C10 (GAP-2 A5): duplicate destinations in the newest-first + * inline capture decide which record wins the backend's + * overlay; bounded and unconditional (gsp_sema_dbg is defined + * later in the file and this site may not reference it). + */ + if (svc_inline_recs_log < 64) { + u32 dup = 0, a2, b2; + + for (a2 = 0; a2 < n; a2++) + for (b2 = a2 + 1; b2 < n; b2++) + if (recs[a2].dst_va == + recs[b2].dst_va) + dup++; + svc_inline_recs_log++; + pr_info("sema-gr: inline recs n=%u dup_dst=%u\n", + n, dup); } - if (gsp_phys_gr_forward_enabled() && trusted && !replay) - svc_gr_cand_capture(ch, st, recs, n, push, push_len, - st->push_va); - else if (!replay || !gsp_phys_gr_forward_enabled()) + if (gsp_phys_gr_forward_enabled()) { + u64 push_hash = (push && push_len) ? + svc_push_hash(push, push_len) : 0; + + if (trusted && !replay) { + svc_gr_cand_capture(ch, st, recs, n, push, + push_len, st->push_va); + } else if (trusted && replay && + !svc_gr_cand_push_seen(st->push_va, + push_hash)) { + /* + * r383 (streams2 second-vadd class): a + * complete QMD first seen on a replay pass + * fell through BOTH arms below - the + * forward path skips replays and the + * bridge arm needs !replay - so the + * launch was silently dropped (zero + * engine writes, client read stale/zero + * outputs). The push VA was never + * captured, so capture it now (trusted + * content only, as the first-sight path); + * a replay of an already captured/offered + * launch stays the exactly-once no-op. + */ + pr_info("sema-gr: REPLAY-CAPTURE h=0x%x push=0x%llx (complete QMD first seen on a replay pass; no capture on record)\n", + ch->h_object, st->push_va); + svc_gr_cand_capture(ch, st, recs, n, push, + push_len, st->push_va); + } else if (replay && svc_gr_nocap_log < 64) { + svc_gr_nocap_log++; + pr_info("sema-gr: LAUNCH-SKIP h=0x%x push=0x%llx (replay; already captured=%d trusted=%d)\n", + ch->h_object, st->push_va, + svc_gr_cand_push_seen(st->push_va, + push_hash), + trusted); + } else if (!replay) { + gsp_cuda_bridge_launch(vgpu, ch->h_object, + st->qmd, + st->qmd_dev_va, + st->push_va, recs, n); + } + } else if (!replay) { gsp_cuda_bridge_launch(vgpu, ch->h_object, st->qmd, st->qmd_dev_va, st->push_va, recs, n); + } st->qmd_n = 0; st->qmd_bits = 0; st->qmd_have = 0; @@ -5983,6 +14641,1515 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } } +static u64 svc_pass_hash; /* r391f: this decode pass's push content hash */ + +/* + * r481 RISK-REDUCTION diagnostic (default 0): dump the full decoded method + * stream of the first few large GR-class pushes (len >= 1024) so the launch + * push's acquire/release method groups (or their absence) are visible + * verbatim. The streams_events class evidence shows the st2 vadd's copy-in + * running ~115 ms BEFORE its stream's H2D soft copies; the ordering the + * driver relied on is only provable by reading the push's own methods. + * Bounded: gsp_sema_push_raw dumps, at most once per push VA. + * SESSION 153: gsp_sema_push_raw_small extends the same dump to the GL + * client's small pushes (16..511 B - the C597 draw/notify pushes are 32/64 B + * and were excluded by the >= 512 gate), with its own budget and a per + * (VA,len) once-only guard so a re-read of the same push shape does not + * flood. + */ +unsigned gsp_sema_push_raw; +unsigned gsp_sema_push_raw_small; +static u32 svc_mme_census_flag; +module_param_named(sema_push_raw_small, gsp_sema_push_raw_small, uint, 0644); +MODULE_PARM_DESC(sema_push_raw_small, + "Verbatim decode dump for small GR pushes (16..511 B), bounded count (default 0)"); +module_param_named(sema_mme_census, svc_mme_census_flag, uint, 0644); +MODULE_PARM_DESC(sema_mme_census, + "Census MME-family method ids (0x114..0x124, 0x234, 0x550..0x568, 0x3400..0x34fc, 0x3800..0x39fc) in beef pushes, bounded (default 0)"); +static u64 svc_pushraw_last_va; +static u32 svc_pushraw_last_len; +static u32 svc_pushraw_n; +static u32 svc_pushraw_small_n; + +/* + * SESSION 153-CONTINUED-52: MME-family method census for the beef (GL 3D + * client) channel. The replayed draw windows raise a real MME ILLEGAL_OPCODE + * (NV_PGRAPH_PRI_MME_HWW_ESR bit 2) - the MME executed a macro whose code it + * could not fetch. The macro calls (CALL_MME_MACRO/DATA 0x3800..0x39fc) carry + * macro code addresses, so the macro CONTEXT state - the MME Instruction/ + * Start-address RAM loads (0x114/0x118/0x11c/0x120), the shadow RAM control + * (0x124), MME version (0x234), the MME DMA window (0x550..0x568) and the + * shadow scratch (0x3400..0x34fc) - must have reached the ladder context too. + * If those setup methods never appear in ANY client push the MME can never + * have its macro library and the replay is doomed; if they do appear, the + * next question is which push forwards them. This census answers that: it + * scans every beef push for those method ids and logs a bounded one-line + * summary (first-hit offsets + first data word). Read-only; bounded. + */ +static u32 svc_mme_census_n; + +static void svc_mme_census(const u8 *buf, u32 len, u64 push_va, u32 h_object) +{ + u32 off = 0; + u32 nhit = 0; + char line[512]; + int lc = 0; + + if (!READ_ONCE(svc_mme_census_flag) || svc_mme_census_n >= 128) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4 <= len && nhit < 16) { + u32 hdr, op, cnt, addr; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if (op == 1u || op == 3u || op == 5u) + /* multi-word run: classify by its method id */ + ; + else if (op == 4u || op == 7u) + cnt = 0; + else + cnt = 0; + if ((addr >= 0x114u && addr <= 0x124u) || + addr == 0x234u || + (addr >= 0x550u && addr <= 0x568u) || + (addr >= 0x3400u && addr <= 0x34fcu) || + (addr >= 0x3800u && addr <= 0x39fcu)) { + u32 v0 = 0; + + if (off + 8 <= len) + memcpy(&v0, buf + off + 4, 4); + lc += scnprintf(line + lc, sizeof(line) - lc, + "%s0x%x@0x%x(%08x)", + nhit ? "," : "", addr, off, v0); + nhit++; + } + off += (op == 1u || op == 3u || op == 5u) && cnt ? + (1u + cnt) * 4u : 4u; + } + if (nhit) { + svc_mme_census_n++; + pr_info("sema-mme-census: h=0x%x va=0x%llx len=%u n=%u hits=[%s]\n", + h_object, push_va, len, nhit, line); + } +} + +/* + * SESSION 153-CONTINUED-94: client init-push capture + local-memory census. + * + * CONT-93's "single most concrete next fix" was that the GL driver's + * download-state LOCAL MEMORY push (SET_SHADER_LOCAL_MEMORY_A/B/C/D, + * 0x790-0x79c) is the last channel-init class missing from the replayed + * context. The claim is directly decidable at this decode site: every + * client push that reaches the service is walked once, so any INC run at + * 0x790..0x79c is counted and logged with its values (bounded). The + * guest-RAM scrape alone cannot settle it: the client's pushbuffer is + * transient (the staging copies observed at init read back all-zero + * minutes later), so the first large beef push of a boot is preserved + * host-side and dumped through the gsp-phys-init-push debugfs file. + * Read-only instrumentation; beef channels only. + */ +#define SVC_INIT_CAP_BYTES (64u * 1024u) +static u8 svc_init_cap[SVC_INIT_CAP_BYTES]; +static u32 svc_init_cap_len; +static u32 svc_init_cap_h; +static u64 svc_init_cap_va; +static bool svc_init_cap_done; +static u32 svc_lmem_hits; +static u32 svc_lmem_seen; + +static void svc_init_push_scan(const u8 *buf, u32 len, u64 push_va, u32 h_object) +{ + u32 off = 0; + + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, m; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + m = (hdr & 0x1fffu) << 2; + if (m >= 0x790u && m <= 0x79cu && + (op == 1u || op == 3u || op == 5u) && + cnt >= 1u && cnt <= 8u && off + 4u + cnt * 4u <= len) { + svc_lmem_seen++; + if (svc_lmem_hits < 16u) { + u32 k, v[4] = {0, 0, 0, 0}; + + for (k = 0; k < cnt && k < 4u; k++) + memcpy(&v[k], buf + off + 4u + k * 4u, 4); + svc_lmem_hits++; + pr_info("vgsp-sema: INIT-LMEM h=0x%x va=0x%llx m=0x%x cnt=%u vals=%08x,%08x,%08x,%08x\n", + h_object, push_va, m, cnt, + v[0], v[1], v[2], v[3]); + } + } + if ((op == 1u || op == 3u || op == 5u) && cnt && cnt <= 0xfffu) + off += 4u + cnt * 4u; + else + off += 4u; + } + if (!svc_init_cap_done && len >= 4096u && + memchr_inv(buf, 0, min_t(u32, len, 64u))) { + svc_init_cap_len = min_t(u32, len, SVC_INIT_CAP_BYTES); + memcpy(svc_init_cap, buf, svc_init_cap_len); + svc_init_cap_h = h_object; + svc_init_cap_va = push_va; + svc_init_cap_done = true; + pr_info("vgsp-sema: INIT-CAP h=0x%x va=0x%llx len=%u lmem_runs=%u (gsp-phys-init-push)\n", + h_object, push_va, svc_init_cap_len, svc_lmem_seen); + } +} + +const u8 *gsp_sema_init_cap_get(u32 *len, u32 *h, u64 *va, u32 *lmem_seen) +{ + *len = svc_init_cap_len; + *h = svc_init_cap_h; + *va = svc_init_cap_va; + *lmem_seen = svc_lmem_seen; + return svc_init_cap; +} + +/* + * SESSION 153-CONTINUED-53: MME context-setup prelude buffer. + * + * The draw-window replay raises a real MME ILLEGAL_OPCODE + * (NV_PGRAPH_PRI_MME_HWW_ESR bit 2) on CALL_MME_MACRO(2)=0x3810 with + * mme_pc=0/mme_ir=1: the MME executed a macro call with no macro program + * loaded. The client DOES upload the program (the census records the + * 0x11c/0x114 pointer pairs and the 0x124 shadow-control runs), but the + * replay only ever forwards draw-window slices, so the ladder context + * never sees the setup. This buffer captures the setup runs verbatim - + * every method run whose id is in the C597 MME setup families: + * + * 0x114 LOAD_MME_INSTRUCTION_RAM_POINTER + * 0x118 LOAD_MME_INSTRUCTION_RAM (the macro code words) + * 0x11c LOAD_MME_START_ADDRESS_RAM_POINTER + * 0x120 LOAD_MME_START_ADDRESS_RAM (per-macro entry pc) + * 0x124 SET_MME_SHADOW_RAM_CONTROL (METHOD_TRACK[_WITH_FILTER]) + * 0x234 SET_MME_VERSION + * 0x550..0x568 MME mem address / data RAM / DMA / reduction + * 0x3400..0x34fc SET_MME_SHADOW_SCRATCH + * + * - in first-encounter order, so the booter can preplay them ahead of the + * first 3D window replay of a capture generation (see + * gsp_sema_mme_prelude_get and the gsp_phys_rmcall_gr_mme_prelude knob). + * Identical runs (a re-upload of the same program) are skipped by an + * exact-run signature table, so repeated setup pushes do not grow the + * buffer; genuinely new content appends and bumps the generation so the + * next window re-preplays. The writer is the single service thread and + * appends strictly in order; the generation only increments after a + * capture that appended, giving the booter a seqcheck against torn views. + * + * SESSION 153-CONTINUED-68: the capture can DROP runs once the buffer is + * full ("buffer full=N words=1023" on r1126) - e.g. a second GL program + * whose content never entered the replay - while the SA table (recorded + * unbounded, last-wins) still points slots at those never-replayed IR + * addresses; a call then fetches macro data that is not there (Graphics + * Exception: MISSING_MACRO_DATA). The coverage bitmap + * (svc_mme_ir_cov, instruction granularity) records exactly which IR + * instructions the REPLAY holds so the launch builder can stub every + * called slot whose entry is outside that set (see + * gsp_sema_mme_ir_covered). SESSION 153-CONTINUED-77: the buffer cap + * rose 1024 -> 2048 -> 3584 words (with the launch method block raised + * to 4096 words / four pushbuffer pages) because the fully captured + * client prelude exceeded each cap and was silently truncated; + * truncation leaves called slots outside the coverage set and keeps + * them stubbed. 3584 keeps the worst-case block (cap + 448-word chunk + * + init + SA stubs) inside 4096. + */ +#define SVC_MME_PRELUDE_WORDS 3584u +#define SVC_MME_PRELUDE_SIGS 2048u +static u32 svc_mme_prelude_buf[SVC_MME_PRELUDE_WORDS]; +static u32 svc_mme_prelude_n; +static u32 svc_mme_prelude_gen; +static u64 svc_mme_prelude_sigs[SVC_MME_PRELUDE_SIGS]; +static u32 svc_mme_prelude_sig_n; +static u32 svc_mme_prelude_dump_n; +static u32 svc_mme_prelude_flag = 1; +/* + * SESSION 153-CONTINUED-57: macro-program payload words seen by the + * capture (sum of the 0x114 LOAD_MME_INSTRUCTION_RAM run counts). The + * r1094 failure mode was a launch emitted with the capture still at 9 + * words (scratch only, zero code words): the MME executed macro(2) with + * pc=0 -> ILLEGAL_OPCODE -> poison 0.6 s before the client's real setup + * push became readable. gsp_sema_mme_prelude_mature() lets the forward + * path gate the first beef launch on the program actually being present. + */ +static u32 svc_mme_prelude_code_words; +module_param_named(sema_mme_prelude_cap, svc_mme_prelude_flag, uint, 0644); +MODULE_PARM_DESC(sema_mme_prelude_cap, + "Capture the client's MME context-setup method runs (0x114..0x124, 0x234, 0x550..0x568, 0x3400..0x34fc) from trusted beef pushes for the GR launch builder's one-time preplay (default 1)"); + +/* + * SESSION 153-CONTINUED-63: the MME START-ADDRESS RAM table (macro slot -> + * IR instruction address), captured unbounded by the 512-word prelude + * buffer. The r1116 0x238c wall proved the replay's SA table was + * incomplete (the capture buffer filled with the first 11 macros' code and + * 28 further setup runs - the remaining slots - never entered the prelude): + * the window's CALL_MME_MACRO(0x22) then read SA[0x22] = 0 on the fresh + * context, the MME jumped to IR 0 (the slot-0 macro, simulating with the + * caller's args emits LOAD_CONSTANT_BUFFER_OFFSET <- 0x20020; verified + * with the offline MME64 simulator), and the class error poisoned the + * lane. The builder uses this table to complete the SA RAM it replays and + * to stub every called-but-unknown slot with a harmless end-noop. + */ +static u64 svc_mme_sa_present; +static u32 svc_mme_sa_entry[64]; +static u32 svc_mme_sa_log; +/* + * SESSION 153-CONTINUED-70: last LOAD_MME_START_ADDRESS_RAM_POINTER (0x11c) + * value; the data port (0x120) writes entries for consecutive slots from + * here (see the 0x120 case in svc_mme_prelude_capture). + */ +static u32 svc_mme_sa_ptr; + +/* + * SESSION 153-CONTINUED-68: furthest IR instruction any CAPTURED 0x114 + * upload covers (pc + ceil(code words / 3); the client's upload addresses + * are instruction units - slot N's pc equals slot N-1's pc + its code + * instructions, verified across the r1099/r1116 captures). The launch + * builder's MME stub must land BEYOND the replayed program: a fixed stub + * PC inside a captured macro's body would clobber that macro's code (the + * window calls covered slots too). Recorded only for runs that entered the + * replay buffer, so the value always describes the replayed IR extent. + */ +static u32 svc_mme_ir_end; + +u32 gsp_sema_mme_ir_end_get(void) +{ + return READ_ONCE(svc_mme_ir_end); +} + +/* + * SESSION 153-CONTINUED-70: IR code words captured so far (the maturity + * gate's measure; the MME-PRE-DUMP prints it for the offline sim). + */ +u32 gsp_sema_mme_code_words_get(void) +{ + return READ_ONCE(svc_mme_prelude_code_words); +} + +/* + * SESSION 153-CONTINUED-68: instruction-granular coverage of the IR the + * REPLAY buffer actually holds (marked for every 0x114 run that entered + * svc_mme_prelude_buf). Values >= SVC_MME_IR_BITS are treated as covered + * "elsewhere" so an overflow cannot force every slot to be stubbed. + */ +#define SVC_MME_IR_BITS 8192u +static u8 svc_mme_ir_cov[SVC_MME_IR_BITS / 8u]; + +bool gsp_sema_mme_ir_covered(u32 pc) +{ + if (pc >= SVC_MME_IR_BITS) + return true; + return !!(svc_mme_ir_cov[pc >> 3] & (1u << (pc & 7u))); +} + +u64 gsp_sema_mme_sa_get(u32 *entries, u32 n) +{ + u32 i; + + if (entries) + for (i = 0; i < n && i < 64u; i++) + entries[i] = svc_mme_sa_entry[i]; + return READ_ONCE(svc_mme_sa_present); +} + +static void svc_mme_sa_record(u32 slot, u32 entry) +{ + if (slot >= 64u) + return; + if (!(svc_mme_sa_present & (1ull << slot)) && svc_mme_sa_log < 64) { + svc_mme_sa_log++; + pr_info("sema-mme-sa: slot[%u] entry=0x%x\n", slot, entry); + } + svc_mme_sa_entry[slot] = entry; + svc_mme_sa_present |= 1ull << slot; +} + +static bool svc_mme_setup_run_addr(u32 addr) +{ + return (addr >= 0x114u && addr <= 0x124u) || addr == 0x234u || + (addr >= 0x550u && addr <= 0x568u) || + (addr >= 0x3400u && addr <= 0x34fcu); +} + +static u64 svc_mme_run_sig(const u8 *p, u32 span, u32 hdr) +{ + u64 h = 1469598103934665603ULL ^ hdr; + u32 i; + + for (i = 1u; i < span; i++) { + u32 v; + + memcpy(&v, p + 4u * i, 4); + h ^= v; + h *= 1099511628211ULL; + } + return h; +} + +const u32 *gsp_sema_mme_prelude_get(u32 *n, u32 *gen) +{ + if (n) + *n = READ_ONCE(svc_mme_prelude_n); + if (gen) + *gen = READ_ONCE(svc_mme_prelude_gen); + return svc_mme_prelude_buf; +} + +static void svc_mme_prelude_capture(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, added = 0, dups = 0, full = 0; + u32 words0 = svc_mme_prelude_n; + /* + * SESSION 153-CONTINUED-84: which 4K pages of the push delivered at + * least one setup run. r1240's carrier read decoded only pages 0 of + * 0x4001004 (15 slots, 933 code words) while r1236's identical + * stream decoded all 16 (55 slots, 2854 code words) - the bitmap + * names the truncation page in one line. + */ + u64 page_bits = 0; + + if (!READ_ONCE(svc_mme_prelude_flag)) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, i; + u64 sig; + bool dup = false; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt) { + if ((u64)off + 4u + (u64)cnt * 4u > (u64)len) + break; + span = 1u + cnt; + } + /* + * SESSION 153-CONTINUED-63: record the START-ADDRESS RAM + * pairs independent of the prelude buffer's 512-word bound + * (see svc_mme_sa_present). The client's LoadMmeStartAddress + * Ram pair run is INC 0x11c with [slot, IR-address] pairs; a + * second run for the same slots overwrites (last wins), the + * same order the engine would see. + */ + if (addr == 0x11cu && cnt >= 2u && span > 1u) { + u32 k; + + for (k = 0; k + 1u < cnt; k += 2u) { + u32 sv, ev; + + memcpy(&sv, buf + off + 4u + 4u * k, 4); + memcpy(&ev, buf + off + 8u + 4u * k, 4); + svc_mme_sa_record(sv, ev); + /* the pointer follows the pair (see the + * 0x120 data-port case below). */ + svc_mme_sa_ptr = sv + 1u; + } + } else if (addr == 0x11cu && cnt == 1u && span > 1u) { + /* pointer-only write (LOAD_MME_START_ADDRESS_RAM_ + * POINTER <- slot) */ + u32 sv; + + memcpy(&sv, buf + off + 4u, 4); + svc_mme_sa_ptr = sv; + } + /* + * SESSION 153-CONTINUED-70: SA data-port form. The pair form + * above only catches [slot,entry] runs AT 0x11c; a client that + * seeds the pointer once and then streams entries through + * LOAD_MME_START_ADDRESS_RAM (0x120) left every slot beyond + * the seeded range unrecorded, so the builder stubbed macros + * the client had really uploaded (r1143/r1144: only slots + * 0..12 known while the window calls 34/39/40/48/51/57). The + * data port takes consecutive slots from the last 0x11c + * pointer value; whether the hardware auto-increments is not + * externally documented, so the host-side table is treated as + * a hint - the builder's keep/stub decision still A/Bs one + * slot at a time and the engine only ever sees the client's + * own runs. + */ + if (addr == 0x120u && cnt && span > 1u) { + u32 k; + + for (k = 0; k < cnt; k++) { + u32 ev; + + memcpy(&ev, buf + off + 4u + 4u * k, 4); + svc_mme_sa_record(svc_mme_sa_ptr + k, ev); + } + svc_mme_sa_ptr += cnt; + } + if (!svc_mme_setup_run_addr(addr)) { + off += span * 4u; + continue; + } + page_bits |= 1ull << ((off >> 12) & 63u); + sig = svc_mme_run_sig(buf + off, span, hdr); + for (i = 0; i < svc_mme_prelude_sig_n; i++) { + if (svc_mme_prelude_sigs[i] == sig) { + dup = true; + break; + } + } + if (dup) { + dups++; + off += span * 4u; + continue; + } + if (svc_mme_prelude_n + span > SVC_MME_PRELUDE_WORDS || + svc_mme_prelude_sig_n >= SVC_MME_PRELUDE_SIGS) { + full++; + off += span * 4u; + continue; + } + memcpy(&svc_mme_prelude_buf[svc_mme_prelude_n], buf + off, + span * 4u); + /* + * SESSION 153-CONTINUED-57: count the macro-program payload - + * the 0x114 instruction-RAM uploads are the only runs that + * make the prelude functionally complete; the maturity gate + * keys on them. + * + * SESSION 153-CONTINUED-68: also track the furthest covered + * IR instruction (see svc_mme_ir_end) so the builder's stub + * lands beyond the replayed program. + */ + if (addr == 0x114u && span > 1u) { + u32 pc = 0, code_words = span - 2u, ir_end, k; + + if (span >= 2u) + memcpy(&pc, buf + off + 4u, 4); + ir_end = pc + (code_words + 2u) / 3u; + if (ir_end > READ_ONCE(svc_mme_ir_end)) + WRITE_ONCE(svc_mme_ir_end, ir_end); + /* + * SESSION 153-CONTINUED-68: mark the instructions this + * replayed run covers (see svc_mme_ir_cov). + */ + for (k = 0; k < (code_words + 2u) / 3u; k++) { + u32 a = pc + k; + + if (a < SVC_MME_IR_BITS) + svc_mme_ir_cov[a >> 3] |= 1u << (a & 7u); + } + svc_mme_prelude_code_words += span - 1u; + } + svc_mme_prelude_n += span; + svc_mme_prelude_sigs[svc_mme_prelude_sig_n++] = sig; + added++; + off += span * 4u; + } + if (added) { + WRITE_ONCE(svc_mme_prelude_gen, svc_mme_prelude_gen + 1); + if (svc_mme_prelude_dump_n < 4) { + u32 k; + + svc_mme_prelude_dump_n++; + pr_info("sema-mme-prelude: capture h=0x%x va=0x%llx len=%u added=%u words=%u gen=%u (was %u) pages=0x%llx\n", + h_object, push_va, len, added, + svc_mme_prelude_n, svc_mme_prelude_gen, + words0, (unsigned long long)page_bits); + for (k = 0; k + 4u <= svc_mme_prelude_n && + k < 512u; k += 4u) + pr_info("sema-mme-prelude[%03u] %08x %08x %08x %08x\n", + k, svc_mme_prelude_buf[k], + svc_mme_prelude_buf[k + 1], + svc_mme_prelude_buf[k + 2], + svc_mme_prelude_buf[k + 3]); + } + if (full) + pr_info("sema-mme-prelude: buffer full=%u words=%u dup=%u\n", + full, svc_mme_prelude_n, dups); + } else if (dups && svc_mme_prelude_dump_n < 8) { + svc_mme_prelude_dump_n++; + pr_info("sema-mme-prelude: re-upload all-duplicate h=0x%x va=0x%llx dup=%u words=%u\n", + h_object, push_va, dups, svc_mme_prelude_n); + } +} + +/* + * SESSION 153-CONTINUED-57: is the captured MME setup complete enough to + * launch against? The r1091 (works) / r1094 (poisons) A/B: a mature capture + * holds the client's macro program (>= 64 code words = at least four + * 0x114 uploads); the premature r1094 capture held 9 scratch-only words + * and the launch raised MME ILLEGAL_OPCODE within milliseconds. The gate + * in svc_gr_request defers beef forwards until this reads true (bounded by + * sema_gr_prelude_gate_ms so a client that never uploads cannot livelock + * the forward path). + */ +bool gsp_sema_mme_prelude_mature(void) +{ + return READ_ONCE(svc_mme_prelude_code_words) >= 64u; +} + +/* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude. + * + * The full-MME-prelude boot (r1091) cleared the MME ILLEGAL_OPCODE but the + * first beef window then faulted the draw shader: NV_PGRAPH_PRI_DS_HWW_ESR + * (0x405840) latched 0xa2040a00 = SPH9|SPH11|SPH18 pending with + * M_PIPE_STAGE = VSB|PS, i.e. draws launched against shader-program-header + * slots whose SET_PIPELINE_PROGRAM_ADDRESS binds live in windows that were + * NEVER replayed (mid-stream state cut), and the same window then raised a + * class error at SET_VERTEX_ATTRIBUTE_A(2) (0x1168, INVALID_ENUM) - the + * window rebinds SOME attribute/pipeline state (byte-faithful, verified) + * but the surrounding record the client set earlier is missing. This + * buffer captures every trusted beef push's method runs in the C597 + * attribute/pipeline families, first-encounter order: + * + * 0x1100..0x11fc attribute formats, skip masks, vertex attribute A/B, + * anti-alias sample positions + * 0x2000..0x205c SET_PIPELINE_SHADER/REGISTER_COUNT/PROGRAM_ADDRESS/ + * BINDING - the per-stage pipeline record + * + * The launch builder preplays the captured runs (run-aligned, relocation + * applied exactly like the window itself) ahead of the first 3D window + * replay of a capture generation. The SET_PIPELINE_PROGRAM_ADDRESS_A/B + * operand VAs the runs bind are also recorded (svc_attr_prelude_ptr) so + * svc_gr_req_fill can put their pages into the launch page set - without + * them the shadow relocation MISSes the pair, the emitted address stays a + * raw client VA, and the DS cannot fetch the SPH through it. Identical + * re-uploads are deduped by an exact-run signature table (same scheme as + * the MME prelude). + */ +#define SVC_ATTR_PRELUDE_WORDS 512u +#define SVC_ATTR_PRELUDE_SIGS 256u +#define SVC_ATTR_PRELUDE_PTRS 8u +static u32 svc_attr_prelude_buf[SVC_ATTR_PRELUDE_WORDS]; +static u32 svc_attr_prelude_n; +static u32 svc_attr_prelude_gen; +static u64 svc_attr_prelude_sigs[SVC_ATTR_PRELUDE_SIGS]; +static u32 svc_attr_prelude_sig_n; +static u32 svc_attr_prelude_dump_n; +static u32 svc_attr_prelude_flag = 1; +static u64 svc_attr_prelude_ptr[SVC_ATTR_PRELUDE_PTRS]; +static u32 svc_attr_prelude_ptr_n; +module_param_named(sema_attr_prelude_cap, svc_attr_prelude_flag, uint, 0644); +MODULE_PARM_DESC(sema_attr_prelude_cap, + "Capture the client's attr/pipeline method runs (0x1100..0x11fc, 0x2000..0x21fc) from trusted beef pushes for the GR launch builder's preplay ahead of the first 3D window (default 1)"); + +static bool svc_attr_setup_run_addr(u32 addr) +{ + /* + * SESSION 153-CONTINUED-86: the pipeline family extends across ALL + * eight pipeline records (SET_PIPELINE_SHADER/BINDING/PROGRAM_ADDRESS + * for j=0..7 = 0x2000..0x21fc), not just j=0..1. The r1273 scrape of + * the client's own push image proved the guest driver binds stage 2 + * (TESS_CONTROL, VA 0x4400200) and stage 5 (PIXEL, VA 0x4400000) with + * the same pair runs; the old 0x2060 cut-off dropped those binds, the + * replay channel's stages 2/5 kept stale/zero program addresses, and + * the draw's SPH validation failed for the stages it could not fetch + * ("Shader Program Header 9/11/18 Error", DS_HWW_ESR 0xa2040a00). + */ + return (addr >= 0x1100u && addr < 0x1200u) || + (addr >= 0x2000u && addr < 0x2200u); +} + +const u32 *gsp_sema_attr_prelude_get(u32 *n, u32 *gen) +{ + if (n) + *n = READ_ONCE(svc_attr_prelude_n); + if (gen) + *gen = READ_ONCE(svc_attr_prelude_gen); + return svc_attr_prelude_buf; +} + +const u64 *gsp_sema_attr_prelude_ptrs(u32 *n) +{ + if (n) + *n = READ_ONCE(svc_attr_prelude_ptr_n); + return svc_attr_prelude_ptr; +} + +static void svc_attr_prelude_note_ptr(u64 va) +{ + u32 i; + + if (!va || svc_attr_prelude_ptr_n >= SVC_ATTR_PRELUDE_PTRS) + return; + for (i = 0; i < svc_attr_prelude_ptr_n; i++) + if (svc_attr_prelude_ptr[i] == va || + (svc_attr_prelude_ptr[i] & ~(u64)0xfffu) == + (va & ~(u64)0xfffu)) + return; + svc_attr_prelude_ptr[svc_attr_prelude_ptr_n++] = va; + /* + * SESSION 153-CONTINUED-59: demand the program page NOW. The r1104 + * first beat window MISSed its 0x2054 pair operand (va=0x4350900) + * because the page did not resolve at the launch fill; the engine + * read it raw -> MMU Fault HUBCLIENT_PD @0x4350000 -> the guest + * driver reset the channel and the lane died for the boot. Recording + * the page in the pin-demand set here (at capture time, often + * minutes before the first launch) lets pin_gr_stream demand it in + * the guest so the launch fill's note resolves. See also the + * sema_attr_settle_ms launch gate. + */ + svc_setup_demand_record(va & ~(u64)0xfffu); +} + +static void svc_attr_prelude_capture(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, added = 0, dups = 0, full = 0; + u32 words0 = svc_attr_prelude_n; + + if (!READ_ONCE(svc_attr_prelude_flag)) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, i; + u64 sig; + bool dup = false; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt) { + if ((u64)off + 4u + (u64)cnt * 4u > (u64)len) + break; + span = 1u + cnt; + } + if (!svc_attr_setup_run_addr(addr)) { + off += span * 4u; + continue; + } + /* + * SET_PIPELINE_PROGRAM_ADDRESS_A/B(j) pair [upper(7:0), + * lower(31:0)]: remember the program VA so its page can join + * the launch page set (see the header comment). + */ + if (addr >= 0x2014u && addr < 0x2200u && + (addr & 0x3fu) == 0x14u && span >= 3u) { + u32 v0 = 0, v1 = 0; + + memcpy(&v0, buf + off + 4, 4); + memcpy(&v1, buf + off + 8, 4); + svc_attr_prelude_note_ptr(((u64)(v0 & 0xffu) << 32) | + (u64)(v1 & 0xfffffffcu)); + } + sig = svc_mme_run_sig(buf + off, span, hdr); + for (i = 0; i < svc_attr_prelude_sig_n; i++) { + if (svc_attr_prelude_sigs[i] == sig) { + dup = true; + break; + } + } + if (dup) { + dups++; + off += span * 4u; + continue; + } + if (svc_attr_prelude_n + span > SVC_ATTR_PRELUDE_WORDS || + svc_attr_prelude_sig_n >= SVC_ATTR_PRELUDE_SIGS) { + full++; + off += span * 4u; + continue; + } + memcpy(&svc_attr_prelude_buf[svc_attr_prelude_n], buf + off, + span * 4u); + svc_attr_prelude_n += span; + svc_attr_prelude_sigs[svc_attr_prelude_sig_n++] = sig; + added++; + off += span * 4u; + } + if (added) { + WRITE_ONCE(svc_attr_prelude_gen, svc_attr_prelude_gen + 1); + if (svc_attr_prelude_dump_n < 4) { + u32 k; + + svc_attr_prelude_dump_n++; + pr_info("sema-attr-prelude: capture h=0x%x va=0x%llx len=%u added=%u words=%u gen=%u ptr_n=%u (was %u)\n", + h_object, push_va, len, added, + svc_attr_prelude_n, svc_attr_prelude_gen, + svc_attr_prelude_ptr_n, words0); + for (k = 0; k + 4u <= svc_attr_prelude_n && + k < 512u; k += 4u) + pr_info("sema-attr-prelude[%03u] %08x %08x %08x %08x\n", + k, svc_attr_prelude_buf[k], + svc_attr_prelude_buf[k + 1], + svc_attr_prelude_buf[k + 2], + svc_attr_prelude_buf[k + 3]); + } + if (full) + pr_info("sema-attr-prelude: buffer full=%u words=%u dup=%u\n", + full, svc_attr_prelude_n, dups); + } +} + +/* + * SESSION 153-CONTINUED-62: surface (render-target) setup capture. The + * replayed beef 3D windows are mid-stream cuts of the client's per-frame + * pushes: the C597 colour-target bind (INC 0x800+j*0x40, [upper, lower, + * w, h, fmt, blksz]) is sticky context state the client (and the X + * server) writes once per context/window, so the draw windows themselves + * carry no render target and the replayed draw rasterizes into an + * unbound target - every launch retires with 0/N pages changed while + * every other witness (index buffers, programs, macros) is in place. + * The observed carrier is the X server's 1280x720 X-screen bind (VA + * 0x120400000, fmt 0xe6, 900 pages) on its own channel, so this capture + * is channel-agnostic (any trusted push) and keeps the LAST surface run + * whose shape the booter's alias scan can use. The launch builder + * replays it - aliased/relocated exactly like the window - ahead of any + * window that carries no usable render-target bind of its own; see + * gsp_phys_gr_surf_pre in gsp_phys_booter.c. A/B via sema_surf_prelude_cap. + */ +#define SVC_SURF_PRE_WORDS 16u +static u32 svc_surf_pre_buf[SVC_SURF_PRE_WORDS]; +static u32 svc_surf_pre_n; +static u32 svc_surf_pre_gen; +static u64 svc_surf_pre_sig; +static u32 svc_surf_pre_log; +static u32 svc_surf_pre_flag = 1; +/* + * SESSION 153-CONTINUED-63: the captured colour target's geometry, for the + * fill's render-target tracking (see svc_gr_data_fill and + * gsp_sema_gr_surf_recs): the fill seeds the surface base VA as a tracked + * data candidate so the engine's stores into the target become a retire + * witness and the copy-out ships them back to the guest's own pages. + */ +static u64 svc_surf_pre_va; +static u32 svc_surf_pre_npages; +module_param_named(sema_surf_prelude_cap, svc_surf_pre_flag, uint, 0644); +MODULE_PARM_DESC(sema_surf_prelude_cap, + "Capture the last C597 colour-target bind (INC 0x800+j*0x40, fmt 0xe6) from trusted pushes so the GR launch builder can replay it ahead of 3D windows that carry no render target (default 1)"); + +const u32 *gsp_sema_surf_prelude_get(u32 *n, u32 *gen) +{ + if (n) + *n = READ_ONCE(svc_surf_pre_n); + if (gen) + *gen = READ_ONCE(svc_surf_pre_gen); + return svc_surf_pre_buf; +} + +bool gsp_sema_surf_va_get(u64 *va, u32 *npages) +{ + u32 n = READ_ONCE(svc_surf_pre_npages); + + if (!n) + return false; + if (va) + *va = svc_surf_pre_va; + if (npages) + *npages = n; + return true; +} + +static void svc_surf_prelude_capture(const u8 *buf, u32 len) +{ + u32 off = 0; + + if (!READ_ONCE(svc_surf_pre_flag)) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, k, n; + u32 v[5] = { 0, 0, 0, 0, 0 }; + u64 va, bytes, npages, sig; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt) { + if ((u64)off + 4u + (u64)cnt * 4u > (u64)len) + break; + span = 1u + cnt; + } + if (op != 1u || cnt < 5u || cnt >= SVC_SURF_PRE_WORDS || + addr < 0x800u || addr >= 0xc00u || + (addr & 0x3fu) != 0u) { + off += span * 4u; + continue; + } + for (k = 0; k < 5u; k++) + memcpy(&v[k], buf + off + 4u + 4u * k, 4); + va = ((u64)(v[0] & 0xffu) << 32) | (u64)(v[1] & 0xfffff000u); + bytes = (u64)v[2] * 4ull * (u64)v[3]; + npages = (bytes + 4095ull) >> 12; + if (!va || (va & 0xfffu) || v[4] != 0xe6u || !v[2] || !v[3] || + v[2] > 8192u || v[3] > 8192u || npages < 2ull || + npages > 4096ull) { + off += span * 4u; + continue; + } + n = span; + if (n > SVC_SURF_PRE_WORDS) + n = SVC_SURF_PRE_WORDS; + sig = svc_mme_run_sig(buf + off, n, hdr); + if (n == svc_surf_pre_n && sig == svc_surf_pre_sig) { + off += span * 4u; + continue; + } + for (k = 0; k < n; k++) + memcpy(&svc_surf_pre_buf[k], buf + off + 4u * k, 4); + WRITE_ONCE(svc_surf_pre_n, n); + svc_surf_pre_sig = sig; + svc_surf_pre_va = va; + WRITE_ONCE(svc_surf_pre_npages, (u32)npages); + WRITE_ONCE(svc_surf_pre_gen, svc_surf_pre_gen + 1); + if (svc_surf_pre_log < 8) { + svc_surf_pre_log++; + pr_info("sema-surf-pre: captured gen=%u words=%u va=0x%llx %ux%u fmt=0x%x hdr=%08x\n", + svc_surf_pre_gen, n, va, v[2], v[3], v[4], + hdr); + } + off += span * 4u; + } +} + +static void svc_push_raw_dump(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, gi = 0; + + if (len > 8192) + return; + if (len < 512) { + if (!gsp_sema_push_raw_small || + svc_pushraw_small_n >= gsp_sema_push_raw_small) + return; + } else { + if (!gsp_sema_push_raw || svc_pushraw_n >= gsp_sema_push_raw) + return; + } + if (push_va == svc_pushraw_last_va && len == svc_pushraw_last_len) + return; + svc_pushraw_last_va = push_va; + svc_pushraw_last_len = len; + if (len < 512) { + svc_pushraw_small_n++; + pr_info("sema-pushraw: BEGIN h=0x%x va=0x%llx len=%u n=%u (small)\n", + h_object, push_va, len, svc_pushraw_small_n); + } else { + svc_pushraw_n++; + pr_info("sema-pushraw: BEGIN h=0x%x va=0x%llx len=%u n=%u\n", + h_object, push_va, len, svc_pushraw_n); + } + while (off + 4 <= len && gi < 160) { + u32 hdr, op, cnt, subch, addr, v[4] = {0,0,0,0}, k, n; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 0x7; + cnt = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + addr = (hdr & 0x1fff) << 2; + n = cnt > 4 ? 4 : cnt; + for (k = 0; k < n && off + 8 + 4 * k <= len; k++) + memcpy(&v[k], buf + off + 4 + 4 * k, 4); + pr_info("sema-pushraw: [%u] off=0x%x op=%u sub=%u addr=0x%x cnt=%u v0=%08x v1=%08x v2=%08x v3=%08x\n", + gi, off, op, subch, addr, cnt, + v[0], v[1], v[2], v[3]); + if (op == 3 && cnt) { + off += 4 + cnt * 4; + gi++; + continue; + } + if (!cnt) { + gi++; + off += 4; + continue; + } + if (op == 1) { + off += 4 + cnt * 4; + gi++; + continue; + } + { + u32 p; + bool found = false; + + for (p = off + 4; p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 1 && + ((h2 >> 16) & 0xfff)) { + off = p; + found = true; + break; + } + } + if (!found) + off = len; + gi++; + } + } + pr_info("sema-pushraw: END h=0x%x va=0x%llx consumed=0x%x/%u\n", + h_object, push_va, off, len); +} + +/* + * r828: the guest kernel's RC watchdog (krcWatchdog_IMPL, 7 s notify + * timeout) posts a notifier cycle on its own channel (object 0x31415900) + * through the class2d subchannel (NVA06F_SUBCHANNEL_2D = 3): SET_NOTIFY_A/B + * name the notifier VA, NOTIFY (WRITE_ONLY) is the GPU's "write a zeroed + * NvNotification" - the driver set the status word to 0xFFFF and declares + * "GPU is probably locked" when it never comes back as 0. Observed: robust + * channels enabled by the GL client (GZDOOM), then NVRM krcWatchdog every + * 8 s and Xorg dead on its screen readback. The emulation must author the + * notification write, exactly like svc_notify_complete does for the + * work-submit token. + */ +#define SVC_2D_SUBCH 3 /* NVA06F_SUBCHANNEL_2D */ +#define SVC_2D_SET_NOTIFY_A 0x0104 /* payload: addr bits [56:32] */ +#define SVC_2D_SET_NOTIFY_B 0x0108 /* payload: addr bits [31:0] */ +#define SVC_2D_NOTIFY 0x010c /* payload: 0 = WRITE_ONLY */ + +static u64 svc_notify2d_done; +static u32 svc_notify2d_log; + +static bool svc_notify2d_write(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, bool completion) +{ + /* 16 bytes = the NvNotification (status word at +14). */ + u8 zero[16] = { 0 }; + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (!ch->notify2d_va) + return false; + if (svc_lookup(vgpu, ch->notify2d_va, &pa, &ap)) { + if (gsp_sema_dbg && (svc_notify2d_log < 64 || + ch->h_object == 0xbfef0002)) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x va=0x%llx unresolved (RC watchdog notifier; guest re-posts every second)\n", + ch->h_object, ch->notify2d_va); + } + return false; + } + /* + * SESSION 153-CONTINUED-18: read the first word for the witness + * (the kick member's timestamp slot: 0 = never GPU-written; the + * completed write makes it nonzero - see svc_kick_notify_exec). + */ + { + u32 o0 = 0; + + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &o0, sizeof(o0))) + o0 = 0xffffffff; + if (svc_write_ap(vgpu, pa, svc_ap_of(ap), zero, sizeof(zero))) + return false; + svc_notify2d_done++; + if (gsp_sema_dbg && (svc_notify2d_log < 64 || + ch->h_object == 0xbfef0002)) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x va=0x%llx -> pa=0x%llx zeroed (w0 old=%08x; ack)\n", + ch->h_object, ch->notify2d_va, pa, o0); + } + } + /* + * r834: pin the page that accepted the ack; the same stale-map + * defense as the push-side pin in the decode. + */ + svc_pin_store(ch->notify2d_va & ~(u64)(SEMA_PAGE_SIZE - 1), + pa & ~(u64)(SEMA_PAGE_SIZE - 1), svc_ap_of(ap)); + /* + * r934: a NOTIFY decoded from a push is that push's completion + * edge on real hardware - post the channel event so the submitting + * client's OS event fires (without it the GL client's kick that + * arms 0x12000e0b0 and NOTIFYs never completes its submit wait + * and the app retries every ~22 s). The periodic idempotent ack + * (r832) passes completion=false so it stays silent. + */ + if (completion) + gsp_rm_post_channel_event(vgpu, ch->h_object); + return true; +} + +/* ------------------------------------------------------------------ */ +/* FERMI_TWOD (0x902d) executor */ +/* ------------------------------------------------------------------ */ +/* + * SESSION 153-CONTINUED-26: the X DDX paints its surfaces (the screen + * 0x120400000 1280x720 X8R8G8B8 and the cursor 0x120370000 256x256 + * A8R8G8B8) exclusively through subchannel-3 FERMI_TWOD methods: the + * SET_DST_* group, then RENDER_SOLID_PRIM (mode RECTS, color format, + * solid color, RENDER_SOLID_PRIM_POINT_X_Y packed pairs). Without this + * executor every 2D surface paint was a no-op and the surfaces the CE + * readback reads stayed unwritten (the screen byte pattern is all + * 0x00000000 - "the X server's drawing never executes"). + * + * Method map: cl902d.h (FERMI_TWOD_A); encodings: NVA06F_DMA_* + * (1 INC / 3 NON_INC / 4 IMMD_DATA, data in header bits 28:16 / 5 + * ONE_INC). Live-decoded per-frame dialog (guest RAM dump at + * 0x8d30c5f8): NON_INC 0x100 cnt17 (MME/global-render-enable window), + * INC 0x280 clip {0,0,0x7fff,0x7fff}, IMMD 0x204 layout, INC 0x214 + * pitch, INC 0x218 {w,h,upper,lower}, IMMD 0x200 fmt, IMMD 0x2ac + * operation, INC 0x580 {mode,fmt,color}, NON_INC 0x5e0 {packed x/y + * point pairs}, IMMD 0x10c NOTIFY, IMMD 0x100 NOP. + * + * The DST VA is (upper<<32)|lower resolved through the SAME operand + * machinery every other engine op uses (svc_fill), so the 2D painting + * and the DDX's own CE readback of the same VA always agree by + * construction. + */ +struct svc_twod_state { + u64 h_object; + u32 dst_fmt, dst_layout, dst_bsize, dst_pitch, dst_w, dst_h; + u64 dst_off; + u32 have_dst; + u32 operation, rop; + u32 prim_mode, prim_fmt, prim_color; + u32 pt_x, pt_y, have_pt; + u64 fills, fill_bytes, misses; + u64 last_fill_hash, last_fill_va; + u32 fill_skips; + u32 dst_log, fill_log, unk_log; +}; + +#define SVC_TWOD_STATES 4 +static struct svc_twod_state svc_twod_tab[SVC_TWOD_STATES]; + +unsigned gsp_sema_twod = 1; +module_param_named(sema_twod, gsp_sema_twod, uint, 0644); +MODULE_PARM_DESC(sema_twod, + "Execute FERMI_TWOD (0x902d) fills on subchannel 3 (default 1)"); + +static struct svc_twod_state *svc_twod_get(struct svc_channel *ch) +{ + u32 i, free_i = SVC_TWOD_STATES; + + for (i = 0; i < SVC_TWOD_STATES; i++) { + if (svc_twod_tab[i].h_object == ch->h_object) + return &svc_twod_tab[i]; + if (!svc_twod_tab[i].h_object && free_i == SVC_TWOD_STATES) + free_i = i; + } + if (free_i == SVC_TWOD_STATES) + free_i = 0; /* bounded table: steal slot 0 */ + memset(&svc_twod_tab[free_i], 0, sizeof(svc_twod_tab[0])); + svc_twod_tab[free_i].h_object = ch->h_object; + return &svc_twod_tab[free_i]; +} + +static u32 svc_twod_bpp(u32 fmt) +{ + switch (fmt) { + case 0xe8: case 0xf8: case 0xe9: case 0xfb: case 0xfc: case 0xee: + return 2; + case 0xf3: case 0xf4: case 0xf7: case 0x1c: + return 1; + default: + return 4; + } +} + +/* One RECTS primitive: fill the (xa,ya)-(xb,yb) axis-aligned rectangle. */ +static void svc_twod_fill(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + struct svc_twod_state *tw, u32 xa, u32 ya, u32 xb, + u32 yb, u64 push_va) +{ + u32 bpp = svc_twod_bpp(tw->dst_fmt); + u32 pitch, x0, x1, y0, y1, w, h; + u64 done = 0; + int rc; + + if (!tw->have_dst) { + if (tw->fill_log < 32) { + tw->fill_log++; + pr_info("sema-twod-fill: h=0x%x push=0x%llx SKIP (no DST set)\n", + ch->h_object, push_va); + } + return; + } + /* + * Pitch: the PITCH layout carries it explicitly (the cursor dialog + * pushes 0x400 for a 256-wide 4-bpp surface). The BLOCKLINEAR + * screen dialog sets only w/h; the DDX's own CE readback of the + * same surface rides pitch = w*bpp (5120 = 1280*4), so the linear + * approximation uses w*bpp there - painting and readback agree. + */ + pitch = (tw->dst_layout == 1 && tw->dst_pitch) ? tw->dst_pitch : + tw->dst_w * bpp; + if (!bpp || !pitch || !tw->dst_w || !tw->dst_h) { + if (tw->fill_log < 32) { + tw->fill_log++; + pr_info("sema-twod-fill: h=0x%x SKIP (bpp=%u pitch=%u w=%u h=%u)\n", + ch->h_object, bpp, pitch, tw->dst_w, + tw->dst_h); + } + return; + } + x0 = xa < xb ? xa : xb; + x1 = xa < xb ? xb : xa; + y0 = ya < yb ? ya : yb; + y1 = ya < yb ? yb : ya; + /* Surface-bound clamp (the DDX keeps CLIP at 0x7fff). */ + if (tw->dst_w >= 4 && tw->dst_w <= 0x8000) { + if (x0 >= tw->dst_w) + return; + if (x1 > tw->dst_w) + x1 = tw->dst_w; + } + if (tw->dst_h >= 4 && tw->dst_h <= 0x8000) { + if (y0 >= tw->dst_h) + return; + if (y1 > tw->dst_h) + y1 = tw->dst_h; + } + w = x1 - x0; + h = y1 - y0; + if (!w || !h) + return; + /* + * Replay dedupe: the DDX re-submits the same dialog push buffer + * for every repaint (the pump re-decodes ring entries), and the + * first pass of a reused slot is often a replay (allow=0). The + * fill is idempotent, so re-running is safe - but re-running on + * every replay pass would write megabytes per second for zero + * change. Execute once per (push VA, content hash); a changed + * push (new points/colors/surface) hashes differently and runs. + */ + if (tw->fills && push_va == tw->last_fill_va && + tw->last_fill_hash == svc_pass_hash) { + tw->fill_skips++; + return; + } + tw->last_fill_va = push_va; + tw->last_fill_hash = svc_pass_hash; + /* + * Row-wise fill through the shared operand machinery: the same + * resolution chain (shadow -> pin -> landing -> walk -> scan) the + * CE pitched fill uses, so the operand guards (LOWFILL / + * FILL-PT / FILL-HOLD) apply to 2D painting as well. + */ + rc = svc_fill(vgpu, tw->dst_off + (u64)y0 * pitch + (u64)x0 * bpp, + tw->prim_color, bpp, w, 0, CE_APERTURE_SYS, &done, + h, pitch); + tw->fills++; + if (!rc) + tw->fill_bytes += (u64)w * bpp * h; + else + tw->misses++; + if (tw->fill_log < 256) { + tw->fill_log++; + pr_info("sema-twod-fill: h=0x%x va=0x%llx lay=%u fmt=0x%x pitch=%u rect=%u,%u-%u,%u color=0x%x bpp=%u rows=%u done=%llu rc=%d push=0x%llx\n", + ch->h_object, tw->dst_off, tw->dst_layout, + tw->dst_fmt, pitch, x0, y0, x1, y1, tw->prim_color, + bpp, h, done, rc, push_va); + } +} + +static void svc_twod_method(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 m, u32 v, bool exec, + u64 push_va) +{ + struct svc_twod_state *tw = svc_twod_get(ch); + + switch (m) { + case 0x200: + tw->dst_fmt = v; + break; + case 0x204: + tw->dst_layout = v; + break; + case 0x208: + tw->dst_bsize = v; + break; + case 0x214: + tw->dst_pitch = v; + break; + case 0x218: + tw->dst_w = v; + break; + case 0x21c: + tw->dst_h = v; + break; + case 0x220: + tw->dst_off = (tw->dst_off & 0xffffffffULL) | + ((u64)(v & 0x1ffffff) << 32); + break; + case 0x224: + tw->dst_off = (tw->dst_off & ~0xffffffffULL) | v; + tw->have_dst = 1; + if (tw->dst_log < 64) { + tw->dst_log++; + pr_info("sema-twod-dst: h=0x%x va=0x%llx pitch=%u w=%u h=%u fmt=0x%x lay=%u bsz=0x%x\n", + ch->h_object, tw->dst_off, tw->dst_pitch, + tw->dst_w, tw->dst_h, tw->dst_fmt, + tw->dst_layout, tw->dst_bsize); + } + break; + case 0x2ac: + tw->operation = v; + break; + case 0x2a0: + tw->rop = v; + break; + case 0x580: + tw->prim_mode = v; + break; + case 0x584: + tw->prim_fmt = v; + break; + case 0x588: + tw->prim_color = v; + break; + case 0x5e0: { + u32 x = v & 0xffff, y = (v >> 16) & 0xffff; + + if (tw->prim_mode == 4 /* RECTS: point pairs */) { + if (!tw->have_pt) { + tw->pt_x = x; + tw->pt_y = y; + tw->have_pt = 1; + } else { + tw->have_pt = 0; + if (exec && gsp_sema_twod) + svc_twod_fill(vgpu, ch, tw, + tw->pt_x, tw->pt_y, + x, y, push_va); + } + } else if (tw->unk_log < 256) { + tw->unk_log++; + pr_info("sema-twod-m: h=0x%x m=0x%x v=0x%x (point mode=%u)\n", + ch->h_object, m, v, tw->prim_mode); + } + break; + } + default: + if (tw->unk_log < 256) { + tw->unk_log++; + pr_info("sema-twod-m: h=0x%x m=0x%x v=0x%x\n", + ch->h_object, m, v); + } + break; + } +} + +/* + * Dispatch a whole method run's data words. `inc` selects the NVA06F + * DMA semantics: INC_METHOD/ONE_INC (op 1/5) advance the method address + * per word (a contiguous register run); NON_INC (op 3) does NOT - every + * data word goes to the SAME method, which is how the DDX streams the + * RECTS point list into RENDER_SOLID_PRIM_POINT_X_Y (0x5e0; the 2-point + * rect and the 4-point double-band fills are one NON_INC run each, and + * its 17-word 0x100 run is a NO_OPERATION padding block). + */ +static void svc_twod_run(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + u32 addr, bool inc, u32 count, const u8 *data, + u32 avail, bool exec, u64 push_va) +{ + u32 k; + + for (k = 0; k < count && (u64)(k + 1) * 4 <= avail; k++) { + u32 v; + + memcpy(&v, data + 4 * k, 4); + svc_twod_method(vgpu, ch, inc ? addr + 4 * k : addr, v, exec, + push_va); + } +} + +/* + * SESSION 148 perf (r862/r864): the RC watchdog's class2d notify cycle + * keep-alive that needs no push decode and no captured VA. Under the + * mirrored frame flow the watchdog ring reads go unreadable (rc=-2) for + * stretches, SET_NOTIFY never re-captures, notify2d_va stays 0, and the + * put != get gate of the r832 path misses - the guest's krcWatchdog then + * declares "GPU is probably locked" (Notify Timeout Seconds: 7) every + * ~8.2 s and the whole X/GL stack wedges (glxgears poll() never returns, + * late XOOM death as buffers pile up). + * + * Ground truth (r864, live RAM scan while wedged): BOTH the watchdog + * channel (0x31415900) and the X/GL channel (0x1000013) sit with their + * class2d notification ARMED - u16 status 0xFFFF at notifier_pa + 0x3E, + * i.e. the 16-byte NvNotification struct at slot 3 (notifier_pa + 0x30, + * status at +14 within the struct) - and NOTHING ever clears them + * (X channel notified=0 after 459 submissions). Slots 0..3 cover the + * error notifier (0), the work-submit token (1) and the class2d notify + * slot (3); the armed test is EXACTLY 0xFFFF so the fault-notifier + * payload slots (nonzero engine/exception codes) are never touched. + * + * Every pass, for every GR channel with a registered notifier (driver + * truth, always known): read slots 0..3; for each slot whose status is + * armed, write the zeroed 16-byte NvNotification (the GPU's completion + * write) and post the channel event - the same two effects + * svc_notify_complete() produces, but reached without any ring read. + * Idempotent: only an armed slot is written. + */ +#define SVC_NOTIFY_SCAN_SLOTS 4 + +static void svc_notify2d_keepalive(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 slots[16 * SVC_NOTIFY_SCAN_SLOTS]; + u8 zero[16] = { 0 }; + int slot; + + if (!ch->notifier_pa) + return; + if (svc_read_ap(vgpu, ch->notifier_pa, ch->notifier_as, + slots, sizeof(slots))) + return; + for (slot = 0; slot < SVC_NOTIFY_SCAN_SLOTS; slot++) { + u16 st; + + memcpy(&st, slots + slot * 16 + 14, 2); + if (st != 0xffff) + continue; /* not armed - nothing to ack */ + if (svc_write_ap(vgpu, ch->notifier_pa + slot * 16, + ch->notifier_as, zero, sizeof(zero))) + continue; + svc_notify2d_done++; + /* + * The GPU consumed the submission: post the channel event so + * the client's CPU-side RM signals the OS event the + * submitting thread waits on (a real GPU would raise an + * interrupt here). Mirrors svc_notify_complete(). + */ + gsp_rm_post_channel_event(vgpu, ch->h_object); + if (gsp_sema_dbg && svc_notify2d_log < 16) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x notifier_pa=0x%llx slot=%d armed -> zeroed (registration-based ack)\n", + ch->h_object, ch->notifier_pa, slot); + } + } +} + +/* + * SESSION 153 CONT.6 (task 2): serve a DENIED DMA copy. + * + * The X server's GL loop re-pushes transfer-shaped launches whose copy + * lands on a pass the trust/replay rules deny (witness: + * `sema-dmatrace: denied h=0x1000013 push=0x12001c64c dma=0x0000030d ... + * in=0x122281200 out=0x120350000 ... replay=1 eng=0`). Both operands of + * this class sit inside the identity guest-RAM window (< svc_pa_max()), + * so the emulator can execute the copy itself: read the source GPA, write + * the destination GPA, verify the landing bytes. The copy is exactly the + * work the hardware copy engine was asked to perform - refusing it only + * stalls whatever consumes the destination (the driver's own fence/state + * progression). Exactly-once per (push_va, content hash); a refused + * shape (operand outside guest RAM, oversized, or failing the readback) + * falls through to the existing deny/hold machinery unchanged. + */ +#define SVC_SERVE_MAX_BYTES (4u << 20) +#define SVC_SERVE_HIST 64 +static bool gsp_sema_serve_denied = true; +module_param_named(sema_serve_denied, gsp_sema_serve_denied, bool, 0644); +MODULE_PARM_DESC(sema_serve_denied, + "Serve a denied transfer-shaped DMA copy between guest-RAM operands by executing it in the emulator (default 1; session 153 CONT.6)"); +static struct { + u64 push_va; + u64 hash; +} svc_serve_hist[SVC_SERVE_HIST]; +static u32 svc_serve_hist_next; +static u32 svc_serve_log; /* bounded served witness */ +static u32 svc_serve_ref_log; /* bounded refusal witness */ +static u64 svc_serve_bytes; +static u64 svc_serve_ops; + +static bool svc_serve_seen(u64 push_va, u64 hash) +{ + u32 i; + + for (i = 0; i < SVC_SERVE_HIST; i++) + if (svc_serve_hist[i].push_va == push_va && + svc_serve_hist[i].hash == hash) + return true; + return false; +} + +static void svc_serve_store(u64 push_va, u64 hash) +{ + u32 i = svc_serve_hist_next++ % SVC_SERVE_HIST; + + svc_serve_hist[i].push_va = push_va; + svc_serve_hist[i].hash = hash; +} + +static bool svc_serve_denied_copy(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, + const struct svc_push_state *st, u32 dma, + u64 push_va, u64 pass_hash) +{ + u8 bounce[256]; + u8 head[32]; + u64 total = st->line_length; + u64 off, bytes = 0; + u64 hash = 1469598103934665603ULL; + int rc = 0; + + if (!total || total > SVC_SERVE_MAX_BYTES) + return false; + if (st->offset_in + total > svc_pa_max() || + st->offset_out + total > svc_pa_max()) { + if (gsp_sema_dbg && svc_serve_ref_log < 16) { + svc_serve_ref_log++; + pr_info("sema-serve: refused h=0x%x push=0x%llx in=0x%llx out=0x%llx line=%u dma=0x%08x (operand outside guest RAM)\n", + ch->h_object, push_va, st->offset_in, + st->offset_out, st->line_length, dma); + } + return false; + } + for (off = 0; off < total; off += sizeof(bounce)) { + size_t n = (size_t)min_t(u64, sizeof(bounce), total - off); + size_t b; + + rc = svc_read(vgpu, st->offset_in + off, bounce, n); + if (rc) + break; + for (b = 0; b < n; b++) { + hash ^= bounce[b]; + hash *= 1099511628211ULL; + } + rc = svc_write(vgpu, st->offset_out + off, bounce, n); + if (rc) + break; + bytes += n; + } + if (rc) { + if (gsp_sema_dbg && svc_serve_ref_log < 16) { + svc_serve_ref_log++; + pr_info("sema-serve: refused h=0x%x push=0x%llx in=0x%llx out=0x%llx line=%u rc=%d (read/write failed)\n", + ch->h_object, push_va, st->offset_in, + st->offset_out, st->line_length, rc); + } + return false; + } + /* + * Verified landing: read the destination back and compare its first + * bytes with the source's. A mismatch means the GPA model of the + * operand pair is wrong for this push - say so and do not latch it + * as served (the deny path then keeps its historical behaviour). + */ + rc = svc_read(vgpu, st->offset_in, bounce, sizeof(head)); + if (!rc) + rc = svc_read(vgpu, st->offset_out, head, sizeof(head)); + if (rc || memcmp(head, bounce, sizeof(head))) { + if (gsp_sema_dbg && svc_serve_ref_log < 16) { + svc_serve_ref_log++; + pr_info("sema-serve: declined h=0x%x push=0x%llx out=0x%llx (readback mismatch)\n", + ch->h_object, push_va, st->offset_out); + } + return false; + } + svc_serve_store(push_va, pass_hash); + svc_serve_bytes += bytes; + svc_serve_ops++; + if (svc_serve_log < 128) { + svc_serve_log++; + pr_info("sema-serve: served h=0x%x push=0x%llx dma=0x%08x in=0x%llx out=0x%llx bytes=%llu hash=0x%llx verify=ok n=%llu\n", + ch->h_object, push_va, dma, st->offset_in, + st->offset_out, bytes, hash, svc_serve_ops); + } + return true; +} + static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 len, u64 page_pa, bool allow_ops, struct svc_channel *ch, u32 entry_idx, @@ -5990,6 +16157,41 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u64 *ops_done) { struct svc_push_state st; + + /* r391f: the inline-write once-only key needs the push's identity. */ + svc_pass_hash = svc_push_hash(buf, len); + /* SESSION 149 R17 (M4): per-pass reset of the standalone 0x180/0x1b0 + * upload pair (the tracker pairs one run with its successor). */ + svc_gl_spot_dst = 0; + svc_gl_spot_bytes = 0; + + /* + * r316: pending releases record the channel they were decoded for; + * the verified pool resolver is channel-keyed (see svc_pending_add + * and the matmul r322 starved/hung note). + */ + svc.cur_ch = ch->h_object; + /* r835: decode-scoped VAS for channel-first VA resolution. */ + svc_cur_vas = ch->h_vaspace; + /* + * r34x: the acquire-wait capture is PER RING ENTRY. It used to be + * reset only at SET_OBJECT, so a decode pass over a later entry (or + * a set of entries, since a held entry is re-decoded after the + * sweep walks ahead) leaked its acquire into an earlier entry's + * release-site gate: r340 held a D2H copy entry that carries no + * acquire at all on a later launch's 'wait for the next tracking + * value' (need=85 while the held entry releases 84) - a circular + * wait only our own withheld release could ever satisfy, so every + * hold expired via the 10 s fallback (wait=4542 etc.). Reset here, + * per entry; SET_OBJECT still resets at the push boundary, and the + * re-decode re-captures the same waits, so held entries re-evaluate + * identically. + */ + svc_wait_n = 0; + svc_wait_rel_va = 0; + svc_wait_rel_payload = 0; + svc_wait_rel_off = 0; + svc_wait_rel_have = false; /* * Static for the same reason as svc_handle_chan_dump: a 32-entry ops * array is a ~2 KB frame, and this is called from the single service @@ -6000,6 +16202,14 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 ops_runs = 0; u32 ops_seq = 0; /* ops appended this pass (sequence numbers) */ u32 off = 0; + /* SESSION 153-CONTINUED-10: draw-window tracker. First method of + * the current C597 run on the attached subchannel after the last + * foreign-subchannel (2D/present) run; see the BEGIN recognizer + * below. Per pass, like the rest of the decode state. */ + u32 gl_win_start = 0; + + /* m164: fresh CLASS-2 QMD witness for this decode pass. */ + svc_gr_qmd_seen = 0; /* * A false VA->PA pair (the discovered map is content-scanned) can @@ -6015,6 +16225,47 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, allow_ops = false; } + /* + * r481: verbatim method-stream dump for large GR-class pushes + * (default 0 = off). See svc_push_raw_dump. + * SESSION 153: the small-push variant (sema_push_raw_small) covers the + * GL client's 16..511 B C597 pushes with its own budget. + */ + if (gsp_sema_push_raw && ch && !ch->engine_type && len >= 512) + svc_push_raw_dump(buf, len, push_va, ch->h_object); + else if (gsp_sema_push_raw_small && ch && !ch->engine_type && + len >= 16 && len < 512) + svc_push_raw_dump(buf, len, push_va, ch->h_object); + + /* SESSION 153-CONTINUED-52: MME-family census for the beef channel. */ + if (ch) + svc_mme_census(buf, len, push_va, ch->h_object); + + /* SESSION 153-CONTINUED-94: init-push capture + local-memory census. */ + if (ch) + svc_init_push_scan(buf, len, push_va, ch->h_object); + + /* + * SESSION 153-CONTINUED-53: MME context-setup prelude capture. Only a + * trusted resolution feeds the buffer (the bytes are replayed into a + * hardware context); identical re-uploads are deduped inside. + */ + if (ch && trusted) + svc_mme_prelude_capture(buf, len, push_va, ch->h_object); + /* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude capture (same + * trusted-only rule; the runs are replayed into a hardware context). + */ + if (ch && trusted) + svc_attr_prelude_capture(buf, len, push_va, ch->h_object); + /* + * SESSION 153-CONTINUED-62: surface (render-target) setup capture - + * channel-agnostic, the X-screen bind lives on the X server's own + * channel. See svc_surf_prelude_capture. + */ + if (ch && trusted) + svc_surf_prelude_capture(buf, len); + memset(&st, 0, sizeof(st)); st.push_va = push_va; /* @@ -6033,21 +16284,52 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * capture: the pump consumes a capture immediately after the pass that * built it, so anything still here is stale. Replay/rescue decodes * must not wipe a capture the pump has not consumed yet. + * + * r548: a valid, never-offered candidate is NOT stale - it is a launch + * whose forward was DEFERRED (r541/r543 guards, backend busy): the + * pump deliberately left it for a bounded retry. Wiping it here let + * the retry passes re-decode the entry as a replay (candidate already + * captured) and then consume it without any forward or completion + * (the r547 saxpy stall: both launches deferred, ring wiped by an + * interleaved fresh decode, zero GR LAUNCH / GR-COMPLETE-SW for 16+ + * minutes). Keep it; the capture path (svc_gr_cand_capture) rotates to + * a free ring slot when a different launch needs the current one. */ - if (!replay && trusted && ch->engine_type == 0) + if (!replay && trusted && ch->engine_type == 0 && + (!svc_gr_cand.valid || svc_gr_cand.offered)) memset(&svc_gr_cand, 0, sizeof(svc_gr_cand)); svc_fwd_rel_n = 0; svc_fwd_defer_armed = false; svc.pages_decoded++; /* Diagnostic: name every push decoded in the post-completion window. */ - if (svc.op_trace_log < 160 && + if (svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && time_before(jiffies, svc.op_trace_until)) { svc.op_trace_log++; pr_info("sema-pushtrace: h=0x%x eng=%u va=0x%llx len=%u allow=%d trusted=%d\n", ch->h_object, ch->engine_type, push_va, len, allow_ops, trusted); } - if (gsp_sema_dbg && ch->engine_type && svc_dbg_streams < 96) { + /* + * fail4: de-windowed decode trace - every push the service looks at, + * bounded per session. The windowed/identified-channel diagnostics + * above can miss a whole client's push stream (unidentified channel, + * no bridge completion to arm the window); this record cannot. + */ + /* + * fail4 r302: with sema_dbg on the boot-time driver init spends the + * 512-line budget long before the run under diagnosis, leaving the + * interesting push stream (a later client's copies) invisible. The + * debug flag raises the bound so one boot covers init + run. + */ + if (svc.pushtrace_all < (gsp_sema_dbg ? 32768 : 512)) { + svc.pushtrace_all++; + pr_info("sema-pushall: h=0x%x eng=%u va=0x%llx pa=0x%llx len=%u allow=%d trusted=%d first8=%08x %08x\n", + ch->h_object, ch->engine_type, push_va, page_pa, len, + allow_ops, trusted, + len >= 4 ? *(const u32 *)(buf + 0) : 0, + len >= 8 ? *(const u32 *)(buf + 4) : 0); + } + if (gsp_sema_dbg && svc_dbg_streams < 96) { svc_dbg_streams++; pr_info("sema-dbg: stream h=0x%x eng=%u va=0x%llx len=%u allow=%d w=%08x %08x %08x %08x %08x %08x %08x %08x\n", ch->h_object, ch->engine_type, push_va, len, allow_ops, @@ -6084,6 +16366,55 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } } + /* + * SESSION 153-CONTINUED-12: attach on the draw marker itself. + * + * The section recogniser below needs the sticky C597 bind + * (svc_gl_att_h) that the boot-class GLX init push carries. On a + * boot where the class is bound channel-side (inside the emulated + * RM channel setup) and no decoded push re-emits SET_OBJECT C597, + * every draw-shaped push is dropped and no 3D replay ever runs. + * Adopt the first trusted push that carries an NVC597_BEGIN IMMD + * (op 4, addr 0x1618, subchannel 0) as the attached channel; the + * per-pass gl_win_start tracker then runs over that same push from + * its start, exactly as it does after an explicit bind, so the + * window keeps the pre-BEGIN state methods (vertex stream bases, + * colour target, constant-buffer selectors). + */ + if (gsp_sema_gl_xlate && !ch->engine_type && trusted && + !svc_gl_att_has(ch->h_object, 0)) { + u32 o2 = 0; + + while (o2 + 4 <= len) { + u32 h2, op2, cnt2, sc2, ad2; + + memcpy(&h2, buf + o2, 4); + op2 = (h2 >> 29) & 0x7; + cnt2 = (h2 >> 16) & 0xfff; + sc2 = (h2 >> 13) & 0x7; + ad2 = (h2 & 0x1fff) << 2; + if (op2 == 4 && ad2 == 0x1618 && sc2 == 0) { + svc_gl_att_note(ch->h_object, 0); + if (gsp_sema_dbg && svc_gl_draw_log < 64) { + svc_gl_draw_log++; + pr_info("sema-gl: attach-on-BEGIN h=0x%x push=0x%llx off=0x%x (no explicit C597 bind seen)\n", + ch->h_object, push_va, o2); + } + break; + } + if (op2 == 4 || op2 == 7) { + o2 += 4; + continue; + } + if ((op2 == 1 || op2 == 3 || op2 == 5) && cnt2 && + cnt2 <= 0xfff && o2 + 4 + cnt2 * 4 <= len) { + o2 += 4 + cnt2 * 4; + continue; + } + o2 += 4; + } + } + while (off + 4 <= len) { u32 hdr = 0, op, count, subch, addr; u32 v0 = 0, v1 = 0, v2 = 0, v3 = 0, v4 = 0; @@ -6095,28 +16426,219 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, addr = (hdr & 0x1fff) << 2; /* + * SESSION 153-CONTINUED-10 (task 1): draw-window tracking on + * the attached C597 channel. The per-frame CONTENT pushes + * interleave the 2D/present ops (subchannel 3) and the 3D + * block (subchannel 0); the replayable 3D window starts at + * the first subchannel-0 method after the last subchannel-3 + * run. The SET_REPORT_SEMAPHORE arms (0x1b00..0x1b0c) are + * excluded: they name client semaphore VAs (0x1_2000c000) + * the replay channel cannot translate (the r929 v1 fault + * class), so they must stay outside the sliced window. + */ + if (gsp_sema_gl_xlate && !ch->engine_type) { + u32 as = svc_gl_att_subch_of(ch->h_object); + + if (as != 0xffu) { + if (subch != as) + gl_win_start = 0; + else if (!gl_win_start && hdr && + !(op == 1 && addr >= 0x1b00 && + addr <= 0x1b0c)) + gl_win_start = off; + } + } + + /* + * fail4: name every big method-data run and every run whose + * address is the LOAD_INLINE_DATA method (any op/subch - the + * encoding of the 64 KB H2D upload is what we are pinning). + */ + if (gsp_sema_dbg && len >= 8192 && count >= 1024 && + svc.bigrun_log < 128) { + svc.bigrun_log++; + pr_info("sema-bigrun: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x have_out=%u out=0x%llx line=%u lcnt=%u w=%08x %08x\n", + ch->h_object, push_va, off, hdr, op, count, subch, + addr, st.have_out, st.offset_out, st.line_length, + st.gr_line_count, + off + 8 <= len ? *(const u32 *)(buf + off + 4) : 0, + off + 12 <= len ? *(const u32 *)(buf + off + 8) : 0); + } + if (gsp_sema_dbg && len >= 8192 && + (addr == GRCE_M_LOAD_INLINE_DATA || addr == 0x1b8) && + svc.lid_hdr_log < 64) { + svc.lid_hdr_log++; + pr_info("sema-lid-hdr: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x trusted=%d allow=%d\n", + ch->h_object, push_va, off, hdr, op, count, subch, + addr, trusted, allow_ops); + } + + /* * GRCE LOAD_INLINE_DATA: a data run (opcode 3) that carries * the DMA source bytes in the push itself. Run the write to * the destination the preceding methods selected. */ if (op == 3 && subch == 1 && addr == GRCE_M_LOAD_INLINE_DATA && count && off + 4 + count * 4 <= len) { - if ((trusted || allow_ops) && !gsp_sema_skip_ops && - st.have_out && st.line_length) { - u32 bytes = st.line_length * - (st.gr_line_count ? st.gr_line_count : 1); - u32 have = count * 4; + /* + * fail4 slot rule: in the UVM upload pushes each + * LOAD_INLINE_DATA run fills a whole 32 KB slot - its + * data runs from the header to the next run header + * (or to the next 0x8000 boundary when this is the + * last run), regardless of the count field. A compact + * push (no room for a following slot) keeps the + * count*4 reading (the small parameter banks). The + * old count*4 reading truncated the big runs to half + * their slot and left the tail to be misparsed as + * methods (the split-upload bug). + */ + bool slot_run = (off + 0x8000 < len); + u32 slot_end = off + 0x8000; + u32 slot_bytes = 0; + /* + * fail4: the bridge's argument-block search needs the + * run's START as its destination; the cursor advance + * below must not leak into the rec (an end-of-run + * dst_va makes the QMD param patch miss and the + * engine never consumes the launch). + */ + u64 run_dst = st.offset_out; + + if (slot_run) { + u32 p; + + for (p = off + 4 + count * 4; + p + 4 <= slot_end && p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 3 && + ((h2 >> 13) & 0x7) == 1 && + ((h2 & 0x1fff) << 2) == + GRCE_M_LOAD_INLINE_DATA && + ((h2 >> 16) & 0xfff)) { + slot_end = p; + break; + } + } + slot_bytes = slot_end - (off + 4); + if (slot_bytes > 0x8000 - 4) + slot_bytes = 0x8000 - 4; + } + /* + * r391f once-only (the sum=0 class): the same + * inline write must not run twice for one push + * identity. The ops are gated by allow_ops + * (entry_replay clears it), but the inline-write + * gate `trusted || allow_ops` re-ran the write on + * every re-decode (a replay is trusted by + * construction): victory_reduce's + * cudaMemset(d_sum,0,8) is exactly such an inline + * write (observed: `inline write + * dst=0x7845ef000000 -> pa=0x7c00000 bytes=8 + * data8=0` firing 86 us before the D2H readback, + * zeroing the accumulator the launch had just + * written - the client then read zeros). A coarse + * !replay gate is wrong (a push whose first pass + * was untrusted must still author its bytes on the + * later trusted pass); key the write by + * (push_va, push content hash, dst, bytes) - a new + * push's release payload makes the hash unique - + * and skip only a genuine duplicate. The table is + * session-scoped (a new client starts clean). + */ + { + u32 bytes = slot_run ? slot_bytes : + st.line_length * + (st.gr_line_count ? + st.gr_line_count : 1); + u32 have = slot_run ? slot_bytes : count * 4; if (bytes > have) bytes = have; - svc_grce_inline_write(vgpu, st.offset_out, - buf + off + 4, bytes, - ch->h_object); - } else if (gsp_sema_dbg && svc.grce_log < 16) { - svc.grce_log++; - pr_info("sema-grce: inline data not run (trusted=%d allow=%d have_out=%d line=%u) h=0x%x\n", - trusted, allow_ops, st.have_out, - st.line_length, ch->h_object); + /* + * r391f once-only (the sum=0 class): the same + * inline write must not run twice for one push + * identity. The ops are gated by allow_ops + * (entry_replay clears it), but the + * inline-write gate `trusted || allow_ops` + * re-ran the write on every re-decode (a replay + * is trusted by construction): victory_reduce's + * cudaMemset(d_sum,0,8) is exactly such an + * inline write (observed: `inline write + * dst=0x7845ef000000 -> pa=0x7c00000 bytes=8 + * data8=0` firing 86 us before the D2H + * readback, zeroing the accumulator the launch + * had just written - the client then read + * zeros). A coarse !replay gate is wrong (a push + * whose first pass was untrusted must still + * author its bytes on the later trusted pass); + * key the write by (push_va, push content hash, + * dst, bytes) - the release payload inside every + * push makes the hash unique per push - and skip + * only a genuine duplicate. Session-scoped: a new + * client starts clean. + */ + if ((trusted || allow_ops) && !gsp_sema_skip_ops && + st.have_out && st.line_length && bytes && + !svc_inline_once_seen(push_va, svc_pass_hash, + st.offset_out, + bytes)) { + svc_inline_once_store(push_va, + svc_pass_hash, + st.offset_out, + bytes); + svc_grce_inline_write(vgpu, st.offset_out, + buf + off + 4, + bytes, + ch->h_object); + /* + * fail4: name the header that follows + * each big LID run - the split 64 KB + * upload's other payload carriers sit + * exactly there. + */ + if (gsp_sema_dbg && len >= 4096 && + svc.next_hdr_log < 24) { + u32 noff = off + 4 + count * 4; + u32 nh = 0, nw = 0; + + if (noff + 4 <= len) + memcpy(&nh, buf + noff, 4); + if (noff + 8 <= len) + memcpy(&nw, buf + noff + 4, + 4); + svc.next_hdr_log++; + pr_info("sema-next-hdr: h=0x%x push=0x%llx after=0x%x cnt=%u nxt_hdr=0x%08x nxt_w0=%08x cursor=0x%llx\n", + ch->h_object, push_va, + off, count, nh, nw, + st.offset_out); + } + } else if (gsp_sema_dbg && svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: inline data not run (trusted=%d allow=%d have_out=%d line=%u) h=0x%x\n", + trusted, allow_ops, st.have_out, + st.line_length, ch->h_object); + } + /* + * fail4 (r526): hardware semantics - a + * LOAD_INLINE_DATA run advances the destination + * cursor by the bytes it carries whether or not + * the once-only gate let the write run. Without + * this, the consecutive runs of a split upload + * all land at the first destination (the old + * RAWA bug), and worse: on a re-decode pass where + * the gate suppresses the FIRST run, the cursor + * stays at the first run's destination and the + * SECOND run executes there - its data (the + * buffer's bytes 8..16) overwriting offset 0 + * (the observed tgemm XDIFF: X[0..1] <- + * X[2..3], deterministic, every pass). The gate + * suppresses the write, not the method stream's + * positional semantics. + */ + if (bytes) + st.offset_out += bytes; } /* * Record the destination and the payload bytes for @@ -6126,21 +16648,157 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * kernel arguments in the QMD parameter bank, and the * destination VA may not be readable later. */ - if (st.have_out && st.offset_out) { + if (st.have_out && run_dst) { struct svc_grce_rec *rec = &svc_grce_dst_ring[svc_grce_dst_ring_next % SVC_GRCE_DST_MAX]; - rec->dst_va = st.offset_out; - rec->len = min_t(u32, count * 4, + rec->dst_va = run_dst; + rec->len = min_t(u32, + slot_run ? slot_bytes : count * 4, GSP_BRIDGE_INLINE_BYTES); memcpy(rec->data, buf + off + 4, rec->len); svc_grce_dst_ring_next++; } - off += 4 + count * 4; + /* + * fail4 slot rule: hop to the next slot's run header + * instead of walking the slot's raw tail as methods. + */ + off += slot_run ? (slot_end - off) : 4 + count * 4; continue; } + /* + * fail4: name the moderate op=1 method runs on big pushes - + * the split 64 KB upload's payload carriers beyond the LID + * class land here (their addr selects the write port). + */ + if (gsp_sema_dbg && len >= 4096 && op == 1 && + count >= 16 && count <= 1023 && svc.mystery_log < 48) { + svc.mystery_log++; + pr_info("sema-mystery: h=0x%x push=0x%llx off=0x%x hdr=0x%08x cnt=%u subch=%u addr=0x%x have_out=%u out=0x%llx line=%u lcnt=%u w=%08x %08x %08x\n", + ch->h_object, push_va, off, hdr, count, subch, + addr, st.have_out, st.offset_out, + st.line_length, st.gr_line_count, + off + 8 <= len ? *(const u32 *)(buf + off + 4) : 0, + off + 12 <= len ? *(const u32 *)(buf + off + 8) : 0, + off + 16 <= len ? *(const u32 *)(buf + off + 12) : 0); + } + + /* + * SESSION 153-CONTINUED-10 (task 1): the client's per-frame + * CONTENT pushes carry real C597 draws but no per-push + * SET_OBJECT (the class bind is sticky), so the section + * recognizer above never opened for them and the 3D replay + * lane never saw a draw. Recognise the draw marker itself: + * NVC597_BEGIN (0x1618, an IMMD method - its immediate is + * the topology in the count field) on the attached + * subchannel of the attached channel opens the section. + * SESSION 153-CONTINUED-26: the glxgears-era client draw + * pushes carry no 0x1618 marker - their first run is the + * C597 GL render-target setup (INC 0x840, {1, rtVA, w, h, + * fmt, blksz}) followed by the 0x1b0 draw list - so the RT + * setup opens the section too. Both shapes are gated on + * the sticky attachment, the channel class, and a closed + * section. + * The window starts at gl_win_start (the first C597 state + * method after the last foreign-subchannel run); the state + * methods before the marker are re-scanned so their operand + * VAs (vertex stream bases 0x1c00/0x1c10, colour target + * 0x0800, constant buffer selectors 0x2380) enter the + * candidate's page set - without them the backend's shadow + * relocation cannot rewrite the window and the engine + * faults on an unreachable guest VA (the r929 v2 class). + * The window then runs from gl_win_start to the pass end + * (the section close at the END method / push tail), which + * svc_gl_flush slices into the verbatim 3D replay. + */ + if (gsp_sema_gl_xlate && !ch->engine_type && + ((op == 4 && addr == 0x1618) || + (op == 1 && addr == 0x840 && count >= 4)) && + !svc_gl.seen && + svc_gl_att_has(ch->h_object, subch)) { + u64 keep_rel = svc_gl.rel_va; + u32 keep_pay = svc_gl.rel_payload; + u32 wstart = gl_win_start ? gl_win_start : off; + + memset(&svc_gl, 0, sizeof(svc_gl)); + svc_gl.seen = true; + svc_gl.subch = subch; + svc_gl.start_off = wstart; + svc_gl.rel_va = keep_rel; + svc_gl.rel_payload = keep_pay; + /* + * SESSION 153-CONTINUED-26: when the RT-setup run is + * the opener, its address word IS the draw's render + * target - put it into the candidate page set, or the + * forwarded replay's shadow relocation cannot rewrite + * the RT VA and the physical launch faults/times out + * (r990: `GR forward failed rc=-110` on every client + * draw while the X section's 7-pointer capture ran). + * Must run AFTER the section reset above. + */ + if (op == 1 && addr == 0x840 && off + 12 <= len) { + /* + * SESSION 153-CONTINUED-27: the run is + * [upper, lower, w, h, fmt, blksz] - the VA + * is ((upper & 0xff) << 32) | lower exactly + * like every other pair recognizer in + * svc_gl_field. Reading only the low word + * added 0x20400000 for a surface whose true + * VA is 0x120400000 (the X screen), so the + * backend's shadow relocation MISSed it and + * the physical replay wrote the raw VA with + * no PTE chain (r992 FAULT_PDE @ +0xa000). + */ + u32 v0 = 0, v1 = 0; + + memcpy(&v0, buf + off + 4, 4); + memcpy(&v1, buf + off + 8, 4); + svc_gl_add_ptr(((u64)(v0 & 0xff) << 32) | + (u64)(v1 & 0xfffffffc)); + } + if (wstart < off) + svc_gl_scan_window(wstart, off, buf, len, + subch); + if (gsp_sema_dbg && svc_gl_draw_log < 64) { + svc_gl_draw_log++; + pr_info("sema-gl: section open (draw) h=0x%x push=0x%llx subch=%u start=0x%x begin=0x%x len=%u trusted=%d\n", + ch->h_object, push_va, subch, wstart, + off, len, trusted); + } + } + + /* + * SESSION 153-CONTINUED-11: SET_PIPELINE_SHADER(j) as an IMMD + * (op 4) method - the GL driver emits it as a 1U method, so + * the TYPE field rides in the header's immediate (bits 27:16, + * value = [ENABLE(0) | TYPE(7:4)]). Sticky per slot, like the + * run-shaped case in svc_gl_field. + */ + if (gsp_sema_gl_xlate && op == 4 && addr >= 0x2000 && + addr < 0x2800 && (addr & 0x3f) == 0 && + ((svc_gl.seen && subch == svc_gl.subch) || + svc_gl_att_has(ch->h_object, subch))) { + u32 j = (addr - 0x2000) / 0x40; + + if (j < 32) + svc_gl_shader_type[j] = + (u8)((((hdr >> 16) & 0xfff) >> 4) & 0xf); + } + /* + * SESSION 153-CONTINUED-26: FERMI_TWOD (0x902d) IMMD_DATA + * methods on subchannel 3 (SET_DST_FORMAT 0x200, layout, + * SET_OPERATION 0x2ac, NOTIFY 0x10c, ...). The value rides + * in the header's bits 28:16. State only; no execution. + */ + if (gsp_sema_twod && !ch->engine_type && + subch == SVC_2D_SUBCH && op == 4 && + addr >= 0x0100 && addr <= 0x1400) + svc_twod_method(vgpu, ch, addr, + (hdr >> 16) & 0xfff, + allow_ops || trusted, push_va); + if (op == 4 || op == 7) { /* Header-only methods (immediate data in the header * for op 4; end-of-push segment for op 7). */ @@ -6196,6 +16854,155 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, continue; } + /* + * SESSION 149 R14 (sema_gl_xlate): GL/3D direct-stream section + * recognition. A SET_OBJECT (op 1, addr 0, one class word) + * binds a class to a subchannel; C597 (TURING_A, the observed + * GL class) on the channel subchannel opens the section. Any + * other class bind closes it (the copy-class C5B5 and 2D 902D + * binds surrounding the GL push must not open sections). The + * collected VA operands live in svc_gl until the section closes + * at the end of the pass or at the next class bind. + */ + if (gsp_sema_gl_xlate && op == 1 && addr == 0 && count == 1 && + off + 8 <= len) { + u32 cls; + + memcpy(&cls, buf + off + 4, 4); + if (cls == 0xc597) { + /* + * SESSION 149 R15 (M1.1a): carry the last + * valid release across the section reset. + * It was decoded from the bracketing 24 B + * release pushes BEFORE this class bind; + * the old wipe-and-open here left rel_va + * at 0 at flushtime, so the trusted first + * pass of the one-shot setup push was + * rejected by the flush guard and the + * section was never captured (r883/r886: + * exactly one trusted decode of the 1524 B + * push, zero captures). + */ + u64 keep_rel = svc_gl.rel_va; + u32 keep_pay = svc_gl.rel_payload; + + if (svc_gl.seen) + svc_gl_flush(vgpu, ch, &st, buf, + len, push_va, trusted, + replay, off); + memset(&svc_gl, 0, sizeof(svc_gl)); + svc_gl.seen = true; + svc_gl.subch = subch; + svc_gl.start_off = off; + svc_gl.rel_va = keep_rel; + svc_gl.rel_payload = keep_pay; + /* SESSION 153-CONTINUED-10: the bind is sticky - + * remember it so the class-space recognizer + * (BEGIN 0x1618, below) accepts this channel's + * later per-push-CONTENT pushes that carry no + * SET_OBJECT of their own. + * SESSION 153-CONTINUED-13: per-channel table - + * a foreign channel's periodic C597 bind must + * not steal this channel's attachment. */ + svc_gl_att_note(ch->h_object, subch); + /* + * SESSION 153: witness every C597 section open + * (bounded) - it is the recognizer-side proof + * that the client's own draw pushes are seen; + * the capture itself is witnessed by + * "sema-gl: section captured". + */ + if (gsp_sema_dbg && svc_gl_open_log < 32) { + svc_gl_open_log++; + pr_info("sema-gl: section open h=0x%x push=0x%llx subch=%u trusted=%d\n", + ch->h_object, push_va, subch, + trusted); + } + } else { + if (svc_gl.seen) + svc_gl_flush(vgpu, ch, &st, buf, + len, push_va, trusted, + replay, off); + /* A different class bound over the attached + * subchannel replaces the C597 bind: drop the + * attachment so the class-space recognizer + * cannot misread methods of the new class. + * SESSION 153-CONTINUED-13: clears only this + * channel's entry. */ + svc_gl_att_clear(ch->h_object, subch); + } + } else if (gsp_sema_gl_xlate && svc_gl.seen && + subch == svc_gl.subch) { + svc_gl_field(addr, count, buf + off + 4, + len - off - 4); + } + + /* + * SESSION 149 R17 (M4): standalone GL-class upload apply. The + * context-init upload push (the one-shot program upload) need + * not sit inside a recognised C597 section, but it carries + * the same subchannel-0 0x180 (LINE_LENGTH_IN) + 0x1b0 + * (LAUNCH_DMA + LOAD_INLINE_DATA) i2m shape. Pair the last + * 0x180 destination with each following 0x1b0 payload and + * apply it through the shared inline-write path (the engine + * never ran these commands). Once-only by (push, content + * hash, dst, bytes); the witness line is bounded. + */ + if (gsp_sema_gl_xlate && ch->engine_type == 0 && subch == 0) { + if (op == 1 && addr == 0x180 && count >= 4) { + u32 w2, w3; + + memcpy(&w2, buf + off + 12, 4); + memcpy(&w3, buf + off + 16, 4); + svc_gl_spot_dst = + ((u64)(w2 & 0xff) << 32) | + (u64)(w3 & 0xfffffffc); + memcpy(&svc_gl_spot_bytes, buf + off + 4, 4); + } else if (op == 5 && addr == 0x1b0 && count > 1 && + svc_gl_spot_dst && + (trusted || allow_ops)) { + u32 bytes = (count - 1) * 4; + + if (bytes && !svc_inline_once_seen(push_va, + svc_pass_hash, svc_gl_spot_dst, + bytes)) { + svc_inline_once_store(push_va, + svc_pass_hash, + svc_gl_spot_dst, bytes); + (void)svc_grce_inline_write(vgpu, + svc_gl_spot_dst, + buf + off + 8, bytes, + ch->h_object); + if (gsp_sema_dbg && + svc_gl_lid_log < 24) { + svc_gl_lid_log++; + pr_info("sema-gl-lid: h=0x%x push=0x%llx dst=0x%llx bytes=%u len=%u trusted=%d first=%08x %08x %08x\n", + ch->h_object, push_va, + svc_gl_spot_dst, bytes, + svc_gl_spot_bytes, + trusted, + *(const u32 *)(buf + off + 8), + off + 16 <= len ? *(const u32 *)(buf + off + 12) : 0, + off + 20 <= len ? *(const u32 *)(buf + off + 16) : 0); + } + } + } + } + + /* + * SESSION 153-CONTINUED-26: FERMI_TWOD NON_INC/ONE_INC runs + * on subchannel 3. The dialogs' whole body is NON_INC (the + * 17-word context window and the point pairs); decode the + * data words as cl902d methods before the generic + * whole-run skip below consumes them. + */ + if (gsp_sema_twod && !ch->engine_type && + subch == SVC_2D_SUBCH && (op == 3 || op == 5) && + addr >= 0x0100 && addr <= 0x1400) + svc_twod_run(vgpu, ch, addr, op == 5 ? true : false, + count, buf + off + 4, len - off - 4, + allow_ops || trusted, push_va); + if (op == 3 || op == 5) { /* * NON_INC / ONE_INC methods carry their own data @@ -6205,6 +17012,13 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * held INC-shaped words queued a bogus LAUNCH_DMA). * No release is decoded from these encodings. */ + if (gsp_sema_dbg && len >= 8192 && count >= 1024 && + svc.skiprun_log < 64) { + svc.skiprun_log++; + pr_info("sema-skiprun: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x trusted=%d allow=%d\n", + ch->h_object, push_va, off, hdr, op, + count, subch, addr, trusted, allow_ops); + } off += 4 + count * 4; continue; } @@ -6221,6 +17035,27 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, memcpy(&v4, buf + off + 20, 4); /* + * SESSION 153-CONTINUED-26: FERMI_TWOD INC runs on + * subchannel 3 (the RENDER_SOLID_PRIM mode/format/color + * triplet at 0x580, the DST group, clip). The three + * notify shapes (0x100 cnt<=1, 0x104, 0x108, 0x10c) stay + * with the watchdog handler below; everything else in the + * cl902d space is decoded here and must not fall into the + * CE method chain. + */ + if (gsp_sema_twod && !ch->engine_type && + subch == SVC_2D_SUBCH && op == 1 && + addr >= 0x0100 && addr <= 0x1400 && + !(addr == 0x0100 && count <= 1) && + addr != 0x0104 && addr != 0x0108 && addr != 0x010c) { + svc_twod_run(vgpu, ch, addr, true, count, + buf + off + 4, len - off - 4, + allow_ops || trusted, push_va); + off += 4 + count * 4; + continue; + } + + /* * OFFSET_IN_UPPER/LOWER and OFFSET_OUT_UPPER/LOWER are * contiguous (0x400..0x40c) and the driver pushes them as a * single INC_METHOD run (count 2 or 4). Apply every word of @@ -6267,10 +17102,106 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, continue; } + if (subch == SVC_2D_SUBCH && count >= 1 && addr >= 0x0100 && + addr <= 0x010c) { + /* + * r828: the RC watchdog's notifier cycle (class2d + * methods on subchannel 3). See svc_notify2d_write: + * without the zeroed notification the guest's + * krcWatchdog declares the GPU locked after 7 s and + * the X/GL stack wedges on screen readback. + */ + switch (addr) { + case SVC_2D_SET_NOTIFY_A: + ch->notify2d_va = (u64)(v0 & 0x1ffffff) << 32; + /* + * r934: the GL client's kick pushes the + * pair as ONE count-2 run at 0x104 (the + * methods 0x104 + 0x108 are contiguous): + * `20026041 00000001 2000e0b0` = + * SET_NOTIFY_A=1, SET_NOTIFY_B=0x2000e0b0 + * -> notification VA 0x12000e0b0. Fold the + * second data word in; the watchdog's + * separate count-1 runs are unchanged. + */ + if (count >= 2) + ch->notify2d_va |= (u64)v1; + /* + * SESSION 153-CONTINUED-18 (task 1): when + * this push also carries the C597 kick + * section, remember the arm so the section + * flush can execute the NOTIFY write (the + * 2D ack below is the watchdog's zero-ack; + * the kick member needs the completed + * NvNotification authored). + */ + if (gsp_sema_gl_xlate && svc_gl.seen) + svc_gl.kick_notify_va = + ch->notify2d_va; + /* + * r834: this method stream reached us through + * a REAL read (svc_read_last_pa); pin its page + * so the watchdog's push translation cannot go + * stale later - the content-scanned map answers + * a zero page once the injected pin ages out, + * staging exactly the starvation the ack must + * not depend on. + */ + if (svc_read_last_pa) + svc_pin_store(push_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + svc_read_last_pa & + ~(u64)(SEMA_PAGE_SIZE - 1), + SVC_APERTURE_SYS); + break; + case SVC_2D_SET_NOTIFY_B: + ch->notify2d_va |= (u64)v0; + break; + case SVC_2D_NOTIFY: + if (count == 1 && v0 == 0) { /* WRITE_ONLY */ + /* + * SESSION 153-CONTINUED-18: mark the + * kick section's NOTIFY (payload 0) so + * svc_gl_flush executes the completion + * write at the armed VA - the executor + * authors the NvNotification the 2D + * zero-ack alone does not. + */ + if (gsp_sema_gl_xlate && svc_gl.seen) { + svc_gl.kick_notify_val = v0; + svc_gl.kick_notify_seen = true; + } + if (svc_notify2d_write(vgpu, ch, + true)) { + /* + * r934: the notifier write IS + * this push's release edge + * (the GL client's kick car- + * ries no semaphore form); + * mark it so the push line + * shows have_rel=1. + */ + st.have_rel = 1; + } + } + break; + default: + break; + } + off += 4 + count * 4; + continue; + } + if (subch == 0 && addr == 0) { /* SET_OBJECT on the channel subchannel: a push starts */ n_ops = 0; memset(&st, 0, sizeof(st)); + /* r336/r34x: new push, new acquire set + release mirror */ + svc_wait_n = 0; + svc_wait_rel_va = 0; + svc_wait_rel_payload = 0; + svc_wait_rel_off = 0; + svc_wait_rel_have = false; off += 4 + count * 4; continue; } @@ -6283,12 +17214,26 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, if ((op == HOST_SEM_OP_RELEASE || op == HOST_SEM_OP_REDUCTION) && payload && - va >= SEMA_VA_LO && va < SEMA_VA_HI) { + (va >= SEMA_VA_LO || va >= SVC_LOW_REL_VA_LO) && + va < SEMA_VA_HI) { if (page_pa) svc_mark_hot(page_pa); svc.pushes_seen++; st.rel_va = va; st.rel_payload = payload; + /* SESSION 149 R14: keep the last VALID release + * decoded on this channel for the GL section + * synthesizer (the 24 B release pushes bracket + * the large GL launch push in ring order). + */ + if (gsp_sema_gl_xlate) { + svc_gl.rel_va = va; + svc_gl.rel_payload = payload; + } + svc_wait_rel_va = va; + svc_wait_rel_payload = payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; svc_rel_va_seen = va; svc_rel_count++; @@ -6296,10 +17241,17 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * REPORT_SEM comment below. */ if (allow_ops && n_ops && !gsp_sema_skip_ops) { svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } + /* r566: a held entry's unexecuted device-dst + * copies must be visible to the launch guard. */ + if (svc.op_tr_pending) + svc_dstpend_note_ops(ops, n_ops, + ops_seq - n_ops, + ops_done); n_ops = 0; /* * A release to a GR completion counter must go @@ -6341,7 +17293,17 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, or.va = va; or.payload = payload; or.entry_idx = entry_idx; - if (svc_fwd_defer_arm()) { + /* + * r824: the deferral needs a + * candidate that OWNS this push + * (see svc_fwd_cand_owns_push) - + * a release from a push with no + * forwarding has nothing to wait + * for and applies inline. + */ + if (svc_fwd_defer_arm() && + svc_fwd_cand_owns_push(push_va) && + !svc.op_tr_pending) { svc_opdef_drop(vgpu, ch, entry_idx); if (svc_fwd_rel_n < SVC_FWD_REL_MAX) { @@ -6386,6 +17348,42 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, va, payload, op); } } + } else if (op != HOST_SEM_OP_RELEASE && + va >= SEMA_VA_LO && va < SEMA_VA_HI) { + /* + * r336/r34x: C56F SEM_EXECUTE acquire ops + * (0 = ACQUIRE/equal, 2 = strict GEQ, + * 3 = circular GEQ, 4 = AND, 5 = NOR). + * Core CUDA starts its compute-channel + * pushes with one of these to wait for a + * progress counter (observed: + * [LO=0x0400ffd0][HI=2][pay=4][0][0x1003] on + * the D2H readback push - ACQ_CIRC_GEQ on + * the channel counter 0x20400ffd0). + * svc_run_ops honors every captured acquire + * of this entry before running its work, so + * a copy cannot execute ahead of the + * dependency it waits on (the streams2 + * readback-before-scale class). Capturing + * into the per-entry list (instead of a + * single last-wait global) is what stops a + * later entry's acquire from gating an + * earlier entry (the r340 over-hold). + * PAYLOAD_SIZE 64BIT (execute bit 24) is + * captured on the low word only; every + * observed CUDA acquire is 32-bit. + */ + if (svc_wait_n < SVC_WAIT_MAX) { + struct svc_wait_op *w = + &svc_waits[svc_wait_n++]; + + w->va = va; + w->payload = payload; + w->op = (u8)op; + w->off = off; + } else { + svc_wait_ovf++; + } } off += 4 + count * 4; continue; @@ -6428,13 +17426,19 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, break; va = ((u64)st.rs_upper << 32) | st.rs_lower; - if (va < SEMA_VA_LO || va >= SEMA_VA_HI) + if ((va < SEMA_VA_LO && + va < SVC_LOW_REL_VA_LO) || + va >= SEMA_VA_HI) break; if (page_pa) svc_mark_hot(page_pa); svc.pushes_seen++; st.rel_va = va; st.rel_payload = st.rs_payload; + svc_wait_rel_va = va; + svc_wait_rel_payload = st.rs_payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; svc_rel_va_seen = va; svc_rel_count++; @@ -6455,10 +17459,15 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, */ if (allow_ops && n_ops && !gsp_sema_skip_ops) { svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } + if (svc.op_tr_pending) + svc_dstpend_note_ops(ops, n_ops, + ops_seq - n_ops, + ops_done); n_ops = 0; /* * ISSUE-0002 (r62/r65): a captured @@ -6570,7 +17579,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * operand state, so one run covers both the * C5B5 and the GRCE copy shapes. */ - if (ch->h_object && gsp_sema_dbg && + if (gsp_sema_dbg && svc.xfer_raw_log < 48) { u32 dl = len < 192 ? len : 192; @@ -6596,12 +17605,29 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } switch (addr) { case CE_M_SET_SEMAPHORE_A: + /* + * SESSION 153 CONT.8: the X DDX's CE release pushes + * the semaphore pair as ONE contiguous count-2 run at + * 0x240 (raw witness, h=0x1000013 va=0x120012434: + * `addr=0x240 cnt=2 v0=00000001 v1=2000c010` = the + * hi/lo halves of VA 0x12000c010, followed by a + * count-1 run at 0x248 with the payload). The old + * count>=3-only fold left sem_va_lo at 0, so every + * such release assembled the bogus VA 0x100000000 + * (hi<<32) and was deferred forever on an + * untranslatable address while the DDX's nvSemWait + * on the real slot spun (xwd -root / server-side + * readback wedged the whole Xorg). Fold the low + * half for count>=2; the payload stays on the 0x248 + * run (or v2 when a count-3 triple arrives). + */ st.sem_va_hi = v0; - if (count >= 3) { + if (count >= 2) { st.sem_va_lo = v1; - st.sem_payload = v2; st.have_sem = 1; } + if (count >= 3) + st.sem_payload = v2; break; case CE_M_SET_SEMAPHORE_B: st.sem_va_lo = v0; @@ -6612,11 +17638,61 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.have_sem = 1; break; case CE_M_SET_REMAP_CONST_A: + /* + * r343 FILL instrument: a fill-shaped LAUNCH_DMA + * (REMAP_ENABLE + destination PITCH, OFFSET_OUT and + * no OFFSET_IN) needs this constant to build its op. + * The 8MB-memset class (memcpy_roundtrip all-bad, + * victory_memset2d) shows fill-shaped launches with + * have_remap=0 that decode nothing and are consumed + * silently. Log every decode (bounded) so one run + * captures the real run shape even when the count + * guard below cannot use it. + */ + if (svc_remapconst_log < 256) { + svc_remapconst_log++; + pr_info("sema-remapconst: h=0x%x push=0x%llx off=0x%x count=%u v0=0x%x v1=0x%x v2=0x%x trusted=%d allow=%u\n", + ch->h_object, push_va, off, count, + v0, count >= 2 ? v1 : 0u, + count >= 3 ? v2 : 0u, trusted, allow_ops); + } if (count >= 3) { st.remap_a = v0; st.remap_b = v1; st.remap_comp = v2; st.have_remap = 1; + st.have_remap_a = 1; + st.have_remap_b = 1; + st.have_remap_c = 1; + } else if (count >= 1) { + /* + * r365: the driver also pushes the fill constants + * as SEPARATE count=1 methods (the CE + * page-table fills carry only SET_REMAP_CONST_A + * with A=0). Capture the individual parts but do + * NOT set have_remap: r365b proved that letting + * these pass the fill guard executes below-floor + * window fills (0x2000c0xxx, 0x204020000) whose + * svc_fill resolution landed in GUEST RAM and + * zeroed slab pages (guest kernel GPF in + * kmem_cache_alloc -> panic at ~75 s). The + * window fills only run behind gsp_sema_winfill + * with the correct resolution proven first. + */ + st.remap_a = v0; + st.have_remap_a = 1; + } + break; + case CE_M_SET_REMAP_CONST_B: + if (count >= 1) { + st.remap_b = v0; + st.have_remap_b = 1; + } + break; + case CE_M_SET_REMAP_COMPONENTS: + if (count >= 1) { + st.remap_comp = v0; + st.have_remap_c = 1; } break; case CE_M_SET_SRC_PHYS_MODE: @@ -6629,12 +17705,172 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, break; case CE_M_LINE_LENGTH_IN: st.line_length = v0; + /* + * r458 THE memset2d root cause: the driver encodes + * LINE_LENGTH_IN (0x418) and the adjacent LINE_COUNT + * (0x41c) as ONE count=2 method group - observed raw + * push bytes `20028106 00001000 00000100` = header + * (subch 4, count 2, addr 0x418), line length 4096, + * line count 0x100 = 256. Reading only v0 left + * line_count = 0, so the 2D-copy builder below took + * the single-row path: one 4 KB op per push, and a + * 4 MB cudaMemcpy2D delivered exactly one page per + * 1 MB block (memset2d probe: per-MB ok=4096, + * bad=12.5M/12.58M). The GRCE variant below has + * always decoded the count=2 form; the CE path did + * not. + */ + if (count >= 2) + st.line_count = v1; + break; + /* r402: 2D geometry (cudaMemset2D / cudaMemcpy2D). */ + case CE_M_LINE_COUNT: + st.line_count = v0; + break; + case CE_M_PITCH_IN: + /* + * SESSION 153 CONT.9 (r943): the X DDX's per-frame + * present blit pushes PITCH_IN+PITCH_OUT+ + * LINE_LENGTH_IN as ONE count-3 INC run at 0x410 + * (raw witness, h=0x1000013 va=0x120015578: + * `addr=0x410 cnt=3 v0=00001400 v1=00000c80 + * v2=00000c80` = pitch_in 5120, pitch_out 3200, + * line_length 3200, followed by LINE_COUNT 10 at + * 0x41c). Reading only v0 left line_length 0, so + * the transfer-shaped LAUNCH_DMA (dma=0x30d xfer= + * pipelined multi-line) built NO op - both the + * allow-path builder and the r939 denied-copy serve + * gate on st.line_length - and every per-frame + * present copy was dropped (`sema-ldatrace ... + * line=0`). Same batched-run fix as r458 did for + * LINE_LENGTH_IN/LINE_COUNT at 0x418. + */ + st.pitch_in = v0 & 0xffffff; + st.have_pitch_in = 1; + if (count >= 2) { + st.pitch_out = v1 & 0xffffff; + st.have_pitch_out = 1; + } + if (count >= 3) + st.line_length = v2; + break; + case CE_M_PITCH_OUT: + st.pitch_out = v0 & 0xffffff; + st.have_pitch_out = 1; + break; + /* + * SESSION 153 CONT.25: SRC_ORIGIN_X/Y (0x744/0x748) and + * DST_ORIGIN_X/Y (0x74c/0x750), pushed as a count-2 (X,Y) + * or count-4 (X,Y,X,Y) INC run (nv_xaa_kepler-style + * SetSrcOrigin: x already scaled by compSizeFactor == Bpp + * for this class). No other class uses these method + * offsets (cl902d/cla06f clear). + */ + case CE_M_SRC_ORIGIN_X: + case CE_M_SRC_ORIGIN_Y: + case CE_M_DST_ORIGIN_X: + case CE_M_DST_ORIGIN_Y: + { + u32 vv[4] = { v0, v1, v2, v3 }; + u32 k, n = count > 4 ? 4 : count; + + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case CE_M_SRC_ORIGIN_X: + st.src_origin_x = vv[k]; + st.have_src_origin = 1; + break; + case CE_M_SRC_ORIGIN_Y: + st.src_origin_y = vv[k]; + st.have_src_origin = 1; + break; + case CE_M_DST_ORIGIN_X: + st.dst_origin_x = vv[k]; + st.have_dst_origin = 1; + break; + case CE_M_DST_ORIGIN_Y: + st.dst_origin_y = vv[k]; + st.have_dst_origin = 1; + break; + } + } + } break; case CE_M_LAUNCH_DMA: { u32 dma = v0; u32 transfer = DMA_DATA_TRANSFER_TYPE(dma); + u32 n_ops_before = n_ops; + bool dma_served = false; + + /* + * SESSION 153 CONT.25: remember a full setup (this is + * the push that carried LINE_LENGTH + OFFSET_IN) as + * the channel's persistent CE state, then fill a + * transfer-shaped DELTA launch (the driver's per-strip + * follow-up: no LINE_LENGTH/OFFSET_IN of its own, + * OFFSET_OUT and the semaphore release present - the + * witnessed 52-byte pushes on h=0x1000013) from it so + * it builds its op instead of being silently dropped. + */ + if (allow_ops && st.line_length && st.have_in) + svc_ce_saved_store(&st, ch->h_object); + if (transfer != 0 && allow_ops && st.have_out && + st.have_sem && + (!st.line_length || !st.have_in)) { + struct svc_ce_saved *s = + svc_ce_saved_slot(ch->h_object); + + if (s && s->line_length && s->have_in) { + if (!st.line_length) { + st.line_length = + s->line_length; + st.line_count = + s->line_count; + } + if (!st.have_in) { + st.offset_in = + s->offset_in; + st.have_in = 1; + } + if (!st.have_pitch_in && + s->have_pitch_in) { + st.pitch_in = s->pitch_in; + st.have_pitch_in = 1; + } + if (!st.have_pitch_out && + s->have_pitch_out) { + st.pitch_out = s->pitch_out; + st.have_pitch_out = 1; + } + if (!st.have_src_mode && + s->have_src_mode) { + st.src_phys_mode = + s->src_phys_mode; + st.have_src_mode = 1; + } + if (!st.have_dst_mode && + s->have_dst_mode) { + st.dst_phys_mode = + s->dst_phys_mode; + st.have_dst_mode = 1; + } + if (gsp_sema_dbg && + svc_cedelta_log < 64) { + svc_cedelta_log++; + pr_info("sema-cedelta: h=0x%x push=0x%llx dma=0x%08x filled line=%u lc=%u in=0x%llx pi=%u po=%u sx=%u sy=%u (engine state inherited)\n", + ch->h_object, push_va, + dma, st.line_length, + st.line_count, + st.offset_in, + st.pitch_in, + st.pitch_out, + st.src_origin_x, + st.src_origin_y); + } + } + } - if (gsp_sema_dbg && ch->h_object && + if (gsp_sema_dbg && svc.lda_log < (gsp_sema_ops_log > 48 ? gsp_sema_ops_log : 48)) { svc.lda_log++; pr_info("sema-lda: h=0x%x eng=%u push=0x%llx inj=%u allow=%u dma=0x%08x xfer=%u remap=%u dpitch=%u dstphys=%u srcphys=%u sem=%u out=0x%llx in=0x%llx line=%u have_out=%u have_in=%u have_remap=%u have_sem=%u ops=%u\n", @@ -6651,7 +17887,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* Diagnostic: full operand state in the post-completion * window (or for high device VAs), bounded. */ - if (svc.op_trace_log < 160 && ch->h_object && + if (svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && ch->h_object && (time_before(jiffies, svc.op_trace_until) || st.offset_out >= SVC_VAS_FLOOR || st.offset_in >= SVC_VAS_FLOOR)) { @@ -6675,7 +17911,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * bounded number of such launches so the operand * encoding (or its absence) is on record. */ - if (transfer != 0 && ch->h_object && gsp_sema_dbg && + if (transfer != 0 && gsp_sema_dbg && svc.xfer_raw_log < 48 && (!st.have_out || !st.line_length)) { u32 dl = len < 192 ? len : 192; @@ -6690,25 +17926,195 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, if (allow_ops && transfer != 0 && st.line_length && st.have_out) { - if (DMA_REMAP_ENABLE(dma) && st.have_remap && + /* + * r361: a fill-shaped transfer whose + * destination is PHYSICAL guest RAM (the + * driver zeroes freshly-allocated PTE / + * page-directory pages with exactly this + * push shape) needs no translation: the + * destination already IS the GPA. The old + * `offset_out >= SVC_VAS_FLOOR` guard + * compared the GPA against the VA floor + * and blocked every one of these, so the + * PD0 pages the GMMU walk later reads + * stayed stale and the first CUDA client + * of a fresh boot dropped operand pages + * (GRFILL-DROP). SEMA_PA_MAX bounds the + * range so a ramfill can only touch guest + * RAM; virtual engine-window fills + * (0x2040xxxxx) stay excluded. + */ + bool ramfill = DMA_DST_TYPE_PHYSICAL(dma) && + st.have_dst_mode && + st.dst_phys_mode == CE_APERTURE_SYS && + st.offset_out + (u64)st.line_length <= + svc_pa_max(); + /* r365: vidmem-window fill (see gsp_sema_winfill). */ + bool winfill = gsp_sema_winfill && + st.offset_out >= SVC_VIDMEM_VA_LO && + st.offset_out < SVC_VIDMEM_VA_BASE + + svc_fb_shadow_size(vgpu); + + if (DMA_REMAP_ENABLE(dma) && + (st.have_remap || + (st.offset_out >= SVC_VAS_FLOOR) || + ramfill || winfill) && DMA_DST_MEMORY_LAYOUT_PITCH(dma) && n_ops < SVC_MAX_MEMSETS) { - u32 csz = (st.remap_comp >> 16) & 0x3; - u32 ndst = (st.remap_comp >> 24) & 0x3; + u32 csz = st.have_remap ? + ((st.remap_comp >> 16) & 0x3) : 0; + u32 ndst = st.have_remap ? + ((st.remap_comp >> 24) & 0x3) : 0; + u32 dst_x = st.have_remap ? + (st.remap_comp & 0x7) : 4; struct svc_memset_op *m = &ops[n_ops]; m->type = 0; m->dst = st.offset_out; - m->value = ndst >= 1 ? - (((u64)st.remap_b << 32) | - st.remap_a) : st.remap_b; - m->elem = (csz + 1) * (ndst + 1); - m->count = st.line_length; + /* SESSION 153 CONT.25: blocklinear + * destination strip position. */ + if (st.have_dst_origin) { + u32 dp = (st.have_pitch_out && + st.pitch_out) ? + st.pitch_out : + st.line_length; + + m->dst += + (u64)st.dst_origin_y * + dp + + st.dst_origin_x; + } + /* + * r367: the driver's own memset push + * (channel_utils.c) proves the + * encoding: LINE_LENGTH_IN is the + * fill SIZE IN BYTES, and + * SET_REMAP_COMPONENTS selects + * COMPONENT_SIZE ([17:16], 0..3 -> + * 1/2/3/4 bytes) and + * NUM_DST_COMPONENTS ([25:24], n+1 + * components). elem = + * (csz+1)*(ndst+1) is the pattern + * width; count must therefore be + * line_length/elem - the old + * count=line_length wrote + * elem x too much (r365: a 383620- + * byte memset wrote 1534480 bytes, + * clobbering the bytes after it). + * The value comes from the mapped + * constants: X<-CONST_A (4) and + * Y<-CONST_B (5); with two + * components the pattern is + * B:A 64-bit. + */ + if (st.have_remap) + m->value = ndst >= 1 ? + (((u64)st.remap_b << 32) | + st.remap_a) : + (dst_x == 5 ? st.remap_b : + st.remap_a); + else + /* + * r357: a fill-shaped transfer + * (REMAP_ENABLE + DST PITCH, + * no source) to a CLIENT DEVICE + * VA whose value constant did + * not decode is a ZERO fill: + * the driver's page-table + * clears and cudaMemset(0) + * issue exactly this shape with + * no SET_REMAP_CONST method. + * Building no op left the + * guest's GPU page tables + * stale (the GMMU walk then + * missed freshly-mapped + * operand pages and the first + * CUDA client of every fresh + * boot ran with dropped + * operands). The >= floor + * guard keeps engine-window + * fills (0x2040xxxxx init + * clears) OUT of the executor: + * executing those clobbered + * emulated engine state and + * hung client init (r356). + */ + m->value = 0; + m->elem = st.have_remap ? + (csz + 1) * (ndst + 1) : 1; + if (m->elem > 1 && + st.line_length % m->elem) { + /* not pattern-divisible: exact + * byte count wins */ + m->elem = 1; + m->count = st.line_length; + } else { + m->count = st.line_length / + m->elem; + } + /* + * r402: a 2D fill (cudaMemset2D): + * LINE_COUNT rows of + * LINE_LENGTH_IN bytes at + * PITCH_OUT stride. A pitch WIDER + * than the row selects row mode; a + * flat fill carries pitch == row + * length (or no pitch method) and + * keeps the historical single-run + * path. + * + * r420c: LINE_COUNT is + * authoritative for the flat case + * too - cudaMemset2D with pitch == + * width (victory_memset2d, + * pitch=4096/w=4096) is CONTIGUOUS + * rows, so the single run must + * cover rows*line bytes. The + * historical path filled one row + * and the readback lost the rest + * (bad ~= 12.5M of 12.58M). + */ + if (st.line_count > 1) { + u32 fpitch = st.have_pitch_out ? + st.pitch_out : + st.line_length; + + if (fpitch > st.line_length) { + m->rows = + st.line_count; + m->dpitch = fpitch; + } else if (st.line_length % + m->elem) { + m->elem = 1; + m->count = + st.line_length * + st.line_count; + } else { + m->count = + (st.line_length / + m->elem) * + st.line_count; + } + } + if (m->rows > 1 && svc_pitch_log < 32) { + svc_pitch_log++; + pr_info("sema-2d: fill rows=%u dpitch=%u line=%u dst=0x%llx h=0x%x push=0x%llx\n", + m->rows, m->dpitch, + st.line_length, m->dst, + ch->h_object, push_va); + } m->phys = DMA_DST_TYPE_PHYSICAL(dma); m->aperture = st.have_dst_mode ? st.dst_phys_mode : CE_APERTURE_VID; n_ops++; ops_seq++; + if (ramfill && svc.fill_ram_log < 24) { + svc.fill_ram_log++; + pr_info("vgsp-sema: RAMFILL-EXEC push=0x%llx dst=0x%llx line=%u ap=%u value=0x%llx (physical guest-RAM fill built)\n", + push_va, st.offset_out, + st.line_length, + m->aperture, m->value); + } } else if (!DMA_REMAP_ENABLE(dma) && st.have_in && n_ops < SVC_MAX_MEMSETS) { @@ -6725,6 +18131,120 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, DMA_SRC_TYPE_PHYSICAL(dma); m->src_aperture = st.have_src_mode ? st.src_phys_mode : CE_APERTURE_VID; + /* + * SESSION 153 CONT.25: a BLOCKLINEAR + * operand positions each strip with + * SRC/DST_ORIGIN (x in bytes, y in + * rows) instead of folding it into + * OFFSET_IN/OUT. The linear shadow + * holds the pixels linearly, so the + * effective operand is + * base + origin_y*pitch + origin_x. + */ + if (st.have_src_origin) { + u32 sp = (st.have_pitch_in && + st.pitch_in) ? + st.pitch_in : + st.line_length; + + m->src += + (u64)st.src_origin_y * + sp + + st.src_origin_x; + } + if (st.have_dst_origin) { + u32 dp = (st.have_pitch_out && + st.pitch_out) ? + st.pitch_out : + st.line_length; + + m->dst += + (u64)st.dst_origin_y * + dp + + st.dst_origin_x; + } + /* + * r402: a 2D copy (cudaMemcpy2D): + * LINE_COUNT rows of + * LINE_LENGTH_IN bytes with + * PITCH_IN/PITCH_OUT strides. + * Row mode when a pitch is + * wider than the row. + * + * r420c: pitch == row length + * (or no pitch method) makes the + * rows CONTIGUOUS, so the single + * flat run must cover + * rows*line bytes - the + * historical path copied one row + * (victory_memset2d's pitched + * D2H/H2D at pitch 4096 = width + * 4096 lost all but the first + * row). + */ + if (st.line_count > 1) { + u32 dpitch = st.have_pitch_out ? + st.pitch_out : + st.line_length; + u32 spitch = st.have_pitch_in ? + (st.pitch_in ? + st.pitch_in : + dpitch) : + dpitch; + + if (dpitch == st.line_length && + spitch == st.line_length) { + m->bytes = + st.line_length * + st.line_count; + } else { + m->rows = + st.line_count; + m->dpitch = dpitch; + m->spitch = spitch; + } + } + /* + * r458 memset2d forensics: the + * driver's cudaMemcpy2D pushes decode + * LINE_LENGTH=4096 with PITCH_IN/OUT + * present but LINE_COUNT absent + * (lc=0), so only one 4 KB row per + * push executes and a 4 MB copy + * delivers 4 pages. Dump the raw push + * (bounded) for every pitch-shaped + * copy so the method stream that + * carries the row count is on record. + */ + if (gsp_sema_dbg && + (st.have_pitch_out || + st.have_pitch_in) && + svc.op2d_raw_log < 32) { + u32 dl = len < 220 ? len : 220; + + svc.op2d_raw_log++; + pr_info("sema-op2d-raw: h=0x%x push=0x%llx len=%u off=0x%x line=%u lc=%u po=%u/%u pi=%u/%u out=0x%llx in=0x%llx bytes=%u dma=0x%08x\n", + ch->h_object, push_va, + len, off, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in, + m->dst, m->src, + (u32)m->bytes, dma); + /* %*phN caps at 64 bytes; + * print the push whole in + * 64-byte halves. */ + pr_info("sema-op2d-raw: raw[0] %*phN\n", + dl > 64 ? 64 : dl, + buf); + if (dl > 64) + pr_info("sema-op2d-raw: raw[1] %*phN\n", + dl - 64, + buf + 64); + } n_ops++; ops_seq++; } else if (gsp_sema_dbg && svc.dma_skip_log < 32) { @@ -6753,6 +18273,445 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* + * SESSION 153 CONT.6 (task 2): execute the copy a + * denied pass would otherwise refuse (see + * svc_serve_denied_copy). Placement: after the + * allow-path builder (a built op means the copy is + * already owned by svc_run_ops), before the owed/ + * pending marks. A successful serve means nothing is + * owed - the dstpend note and the op_tr_pending mark + * below are suppressed via dma_served so the slot + * completes and later passes classify as replays. + */ + if (gsp_sema_serve_denied && transfer != 0 && + n_ops == n_ops_before && trusted && !replay && + !ch->engine_type && st.line_length && st.have_out && + st.have_in && !DMA_REMAP_ENABLE(dma) && + !svc_serve_seen(push_va, svc_pass_hash) && + svc_serve_denied_copy(vgpu, ch, &st, dma, push_va, + svc_pass_hash)) { + dma_served = true; + svc_dstpend_clear(st.offset_out); + } + + /* + * r567: a trusted, transfer-shaped push that decoded + * NO op this pass (its operands are not resolved yet: + * the r569 mail b-copy needed have_in, which the + * staging upload had not produced) must (a) keep its + * slot incomplete so a later pass runs the copy + * instead of replay-suppressing it forever (op_owed, + * the r540 rule), and (b) make its stale destination + * visible to the r541 launch guard, which defers any + * launch reading that page until the copy runs or the + * dstpend window expires. The release still applies + * (no hold is armed: a push whose trust never arrives + * must not wedge the pump) - the guard, not the + * release, provides the ordering. + */ + if (allow_ops && transfer != 0 && trusted && ch && + n_ops == n_ops_before && st.have_out && + st.offset_out >= SVC_VAS_FLOOR) { + svc_dstpend_note(st.offset_out); + svc.op_owed = 1; + if (svc.op_owed_log < 64) { + svc.op_owed_log++; + pr_info("sema-pend: OWED-PUSH h=0x%x push=0x%llx out=0x%llx line=%u (transfer-shaped, no op; dst noted for the launch guard)\n", + ch->h_object, push_va, + st.offset_out, + st.line_length); + } + } + + /* + * r365 FILL-RAW: the fill class that still decodes no + * op (window VA, no remap const decoded) - dump the + * raw push and decode its method stream so the true + * value/component encoding is on record. Bounded. + */ + if (allow_ops && transfer != 0 && + DMA_REMAP_ENABLE(dma) && + DMA_DST_MEMORY_LAYOUT_PITCH(dma) && + st.have_out && !st.have_in && !st.have_remap && + svc_fill_raw_log < 48) { + u32 dl = len < 160 ? len : 160; + u32 mo = 0, mn = 0; + + svc_fill_raw_log++; + pr_info("sema-fill-raw: h=0x%x push=0x%llx out=0x%llx line=%u dma=0x%08x dmode=%u have_dstmode=%u trusted=%d allow=%d\n", + ch->h_object, push_va, st.offset_out, + st.line_length, dma, st.dst_phys_mode, + st.have_dst_mode, trusted, allow_ops); + pr_info("sema-fill-raw: raw len=%u %*phN\n", + len, dl, buf); + /* decode the first method pairs */ + while (mo + 8 <= len && mn < 14) { + u32 h2, c2, a2; + + memcpy(&h2, buf + mo, 4); + c2 = (h2 >> 16) & 0xfff; + a2 = (h2 & 0x1fff) << 2; + if ((h2 >> 29) != 1 || !c2 || + mo + 4 + c2 * 4 > len) + break; + if (c2 >= 2) + pr_info("sema-fill-raw: m[%u] addr=0x%x cnt=%u v0=0x%08x v1=0x%08x\n", + mn, a2, c2, + *(const u32 *)(buf + mo + 4), + *(const u32 *)(buf + mo + 8)); + else + pr_info("sema-fill-raw: m[%u] addr=0x%x cnt=%u v0=0x%08x\n", + mn, a2, c2, + *(const u32 *)(buf + mo + 4)); + mo += 4 + c2 * 4; + mn++; + } + } + + /* + * r343 FILL diagnostic (LOG-ONLY, no hold): a + * fill-shaped transfer (REMAP_ENABLE + destination + * PITCH, OFFSET_OUT and no OFFSET_IN) whose value + * constant did not decode currently decodes no op + * and is consumed silently - the guest's cudaMemset + * then reports success with zero bytes written + * (r341 memcpy_roundtrip: 8MB memset readback + * all-bad; victory_memset2d). An op_tr_pending hold + * here WEDGED client init (r343: first cudaMalloc + * retried forever - driver init issues fill-shaped + * CE pushes, the hold re-armed every decode pass + * and never expired). So: log loudly only; the + * sema-remapconst rows above capture the real + * decode shape to build the proper fill decoder. + */ + if (allow_ops && transfer != 0 && + st.line_length >= 1024 && st.have_out && + !st.have_in && DMA_REMAP_ENABLE(dma) && + DMA_DST_MEMORY_LAYOUT_PITCH(dma) && + !st.have_remap && n_ops == n_ops_before && + svc_fill_noremap < 8) { + svc_fill_noremap++; + pr_err("vgsp: FILL-NOREMAP h=0x%x push=0x%llx dma=0x%08x line=%u out=0x%llx phys=%u dstmode=%u: fill-shaped launch, value source not decoded (log-only diagnostic)\n", + ch->h_object, push_va, dma, + st.line_length, st.offset_out, + DMA_DST_TYPE_PHYSICAL(dma), + st.have_dst_mode ? st.dst_phys_mode : 99); + } + + /* + * fail4: name a transfer-shaped launch that decoded NO + * op because the pass was denied (allow_ops=0) - the + * op-chain above (and its dma-skip dump) is skipped + * wholesale in that case, so this class was invisible. + */ + if (gsp_sema_dbg && transfer != 0 && !allow_ops && + svc.dmatrace_log < (gsp_sema_dbg ? 4096 : 64)) { + svc.dmatrace_log++; + pr_info("sema-dmatrace: denied h=0x%x push=0x%llx dma=0x%08x line=%u in=0x%llx out=0x%llx have_in=%u have_out=%u have_remap=%u trusted=%d replay=%d eng=%u pend_would=%d\n", + ch->h_object, push_va, dma, + st.line_length, st.offset_in, + st.offset_out, st.have_in, st.have_out, + st.have_remap, trusted, replay, + ch->engine_type, + (!allow_ops && trusted && + !ch->engine_type && transfer != 0 && + st.line_length && st.have_out && + st.have_in && + !DMA_REMAP_ENABLE(dma) && + st.offset_out >= SVC_VAS_FLOOR && + entry_idx != SVC_ENTRY_NONE && + entry_idx < SVC_GPFIFO_MAX_BITS && + ch->slot_pend_tries[entry_idx] < 4) ? 1 : 0); + } + /* + * fail4 r302: a denied device-destination copy that + * WOULD have built a type=1 op (the exact branch shape + * above: no remap, both offsets, device dst) must not + * be consumed silently. This is the sw-copy client + * mode: the GR-classified GRCE channel's ops run only + * once the pin/trust stream (injected) reaches the + * entry, and the first pass over a chunk push lands + * before that - the copy then never ran and the guest + * computed on stale VRAM (vector_add r302: chunk + * copies cc..d1 skipped; only the tail entries were + * re-decoded by a rewind and ran). Mark the pass + * op-pending so the pump holds the entry through the + * m163/opdef machinery and the retry runs the copy as + * soon as the trust (or the operand translation) + * arrives; the bounded SVC_OPDEF_HOLD_MS expiry keeps + * the pre-existing skip as the last resort. + * + * r610: a REPLAY pass must be exempt. A replay is the + * pump re-walking an entry whose ops were already + * recorded complete for this identity (entry_replay = + * same && slot_ops_complete): re-marking it as pending + * re-arms a 3 s opdef hold for a copy that HAS run, + * and the fresh pass that follows re-executes the op - + * a replay/fresh/replay cycle (r609 boot, idx=94: the + * d_b H2D re-marked and re-run every ~0.3 s from + * 368841 while the vadd launch waited; the r606 + * captured held chain is the same shape). Measured + * r604-r609: 100% of op_tr_pending marks (27/27, + * 71/71, 72/72) came from replay passes of already- + * executed copies - each one stalled the whole copy + * channel behind it, which is the held-chain class. + * The r308 rationale (pre-r595 the eng==0 copy + * channel's fresh passes never admitted ops, so the + * replay re-mark WAS the retry hatch) no longer + * applies: r595's cap_trust admission runs a plausible + * copy on its fresh pass, and a genuinely failed copy + * keeps its slot incomplete through op_owed / the + * rc!=0 pending rule. The whole accounting block is + * skipped on replay: no dstpend note (the write ran), + * no staging re-snapshot, no hold. A fresh + * (non-replay) denied pass marks pending exactly as + * before. + */ + if (!allow_ops && trusted && !replay && !dma_served && + !ch->engine_type && transfer != 0 && + st.line_length && st.have_out && st.have_in && + !DMA_REMAP_ENABLE(dma) && + st.offset_out >= SVC_VAS_FLOOR) { + /* + * r541: the denied device-dst copy has NOT + * run; whatever release edge later consumes + * this entry, a GR launch must not read the + * target page until the copy executes (see + * svc_dstpend_note). + */ + svc_dstpend_note(st.offset_out); + /* + * r314 (CONT.57 lead 2): capture the staging + * source on the DENIED pass too. This is the + * pipeline-order moment: the push was just + * decoded, the driver has not seen this + * push's fence (it applies later in this same + * pass at the earliest), so the staging still + * holds this chunk's bytes. The old capture + * gate lived in the op-build block, which + * only runs on ALLOWED passes - by then the + * fence had long fired and the staging was + * refilled (r302-r313: captures 0-3/8, the + * late executions read refilled staging, the + * deterministic 786335-bad output). Same + * translation-probe guard as the build-time + * gate: only an SYS-resolving source is + * host staging. + */ + if (st.line_length <= SVC_COPY_SNAP_MAX) { + u64 spa = 0; + u32 sap = 0; + + if (!svc_op_translate(vgpu, + st.offset_in, + &spa, &sap) && + sap == CE_APERTURE_SYS) + svc_copy_snap_capture(vgpu, + st.offset_out, + st.offset_in, + st.line_length, + ch->h_object, + entry_idx, + svc_pass_hash); + } + /* + * r308: the `!replay` gate above kept this + * dead on the sw-copy channel: after the first + * consumed pass every later pass classifies as + * a replay, so the denied chunk copies were + * never retried (vector_add: 6 of 8 chunks + * skipped). A replay pass that marks the slot + * pending un-completes it (slot_ops_complete + * is written from svc.op_tr_pending below), so + * the next pass is treated fresh and the copy + * runs once trust/translation arrives. The + * per-slot counter caps the re-holds so a slot + * whose trust never arrives cannot wedge the + * channel in 10 s hold cycles. + */ + if (entry_idx != SVC_ENTRY_NONE && + entry_idx < SVC_GPFIFO_MAX_BITS) { + if (ch->slot_pend_tries[entry_idx] < 4) { + ch->slot_pend_tries[entry_idx]++; + svc.op_tr_pending = 1; + if (gsp_sema_dbg && + svc.pend_dbg_log < 512) { + svc.pend_dbg_log++; + pr_info("sema-pend: h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u replay=%d trusted=%d tries=%u -> op_tr_pending\n", + ch->h_object, push_va, + entry_idx, st.offset_out, + st.offset_in, + st.line_length, + replay, trusted, + ch->slot_pend_tries[entry_idx]); + } + } else { + /* + * r540: the cap stopped the + * re-hold, but the op is still + * owed. Leaving the slot + * complete would classify every + * later pass as a replay and + * suppress the copy forever - + * the client waits on a copy + * that never ran (the r539 + * ~5-minute device-init stall: + * a GRCE H2D denied while + * untrusted, capped at 4 tries, + * then replay-suppressed after + * the pin arrived). Mark the + * pass owed so the slot stays + * INCOMPLETE and the next + * fresh pass (injected or + * translated) executes the + * copy. No hold is armed, so a + * slot whose trust never arrives + * still cannot wedge the pump. + */ + svc.op_owed = 1; + if (svc.op_owed_log < 64) { + svc.op_owed_log++; + pr_info("sema-pend: OWED h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u trusted=%d (cap hit; slot stays incomplete for a fresh retry)\n", + ch->h_object, push_va, + entry_idx, st.offset_out, + st.offset_in, + st.line_length, + trusted); + } + } + } + } + /* + * fail4: name every op appended to this pass (copy or + * fill) with its resolved operand fields, so the + * executor's population is on record even when the + * 64 KB copy witness below never fires for it. + */ + if (n_ops != n_ops_before) { + struct svc_memset_op *lm = &ops[n_ops - 1]; + + /* r611: this channel carries transfer work + * (see ever_ops) - the stream-order + * watermark's scope. */ + ch->ever_ops = true; + + /* + * fail4 CONT.53: H2D-shaped copies (virtual + * device destination, virtual host source) + * read the driver's ping-pong staging. The + * driver paces restaging on the copy's + * completion fence; our execution may run + * later than the push's processing position, + * so capture the source NOW (bounded pool) + * and let svc_copy execute from the capture - + * the hardware's in-order staging read. + */ + /* + * H2D shape ONLY: device destination (VID + * aperture), host source (SYS). A D2H readback + * (host dst) captured here would freeze + * pre-kernel zeros (observed in the r300 + * captures) and break readbacks. + */ + if (lm->type == 1 && !lm->phys && + lm->dst >= SVC_VAS_FLOOR && + lm->aperture == CE_APERTURE_VID && + !lm->src_phys && + lm->src >= SVC_VAS_FLOOR && + lm->bytes <= SVC_COPY_SNAP_MAX) { + /* + * r311: the src_aperture equality + * gate never matched the sw-copy + * client (the driver sends no + * SET_SRC_PHYS_MODE, so src_ap + * defaults to VID=0 while the + * source is host staging; r302-r310 + * logged zero captures and the + * late executions read refilled + * staging). The translation is the + * truth: capture only when the + * first source page resolves + * through the SYS aperture. + */ + u64 spa = 0; + u32 sap = 0; + + if (!svc_op_translate(vgpu, lm->src, + &spa, &sap) && + sap == CE_APERTURE_SYS) + svc_copy_snap_capture(vgpu, + lm->dst, + lm->src, + lm->bytes, + ch->h_object, + entry_idx, + svc_pass_hash); + if (gsp_sema_dbg && + svc.snap_gate_log < 256) { + svc.snap_gate_log++; + pr_info("sema-snapgate: dst=0x%llx src=0x%llx trc=%d sap=%u\n", + lm->dst, lm->src, + svc_op_translate(vgpu, lm->src, &spa, &sap), + sap); + } + } + + /* + * fail4 CE linear auto-advance, BOTH sides: the + * driver re-sets only ONE offset per op and the + * copy engine advances the other by the transfer + * size. H2D ops show an explicit advancing DST + * with a FIXED SRC base (every 1 MB chunk read + * the source base again: vector_add's device a + * held h_a[0..1MB) x4 -> 75% mismatch; the + * readbacks are the mirror image). Advance both + * registers; an explicit re-set simply overrides. + */ + { + u64 adv = (lm->type == 1) ? lm->bytes : + (u64)lm->elem * lm->count; + + st.offset_out += adv; + if (lm->type == 1) + st.offset_in += adv; + } + if (svc.opadd_log < 2048) { + svc.opadd_log++; + pr_info("sema-opadd: h=0x%x type=%u dst=0x%llx src=0x%llx bytes=%llu elem=%u count=%u ap=%u src_ap=%u phys=%u src_phys=%u\n", + ch->h_object, lm->type, + lm->dst, lm->src, lm->bytes, + lm->elem, lm->count, + lm->aperture, lm->src_aperture, + lm->phys, lm->src_phys); + /* + * r420d: 2D geometry of every copy op + * (the memset2d readback forensics - + * the driver decomposes cudaMemcpy2D + * into per-push ops whose LINE_COUNT/ + * PITCH decode must be visible to + * tell "one row per op" from "N rows + * in one op"). Bounded. + */ + if (lm->type == 1) + pr_info("sema-op2d: h=0x%x dst=0x%llx src=0x%llx bytes=%llu rows=%u dpitch=%u spitch=%u line=%u lc=%u have_po=%u po=%u have_pi=%u pi=%u so=%u/%u do=%u/%u\n", + ch->h_object, lm->dst, + lm->src, lm->bytes, + lm->rows, lm->dpitch, + lm->spitch, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in, + st.have_src_origin, + st.src_origin_y, + st.have_dst_origin, + st.dst_origin_y); + } + } + + /* * Both release encodings write the payload word first * (NvGpuSemaphore.data[0] for the four-word format, the * single word for the one-word format), and the client's @@ -6778,6 +18737,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.sem_va_lo; st.rel_va = sem_va; st.rel_payload = st.sem_payload; + svc_wait_rel_va = sem_va; + svc_wait_rel_payload = st.sem_payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; if (st.have_sem && st.sem_payload && @@ -6875,6 +18838,22 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, ch->entry_want = st.sem_payload; have_cur = true; pending = 1; + /* + * fail4: apply this seed ABSOLUTE. + * The generic applier still sees the + * increment flag (inc), so leaving + * the residue in cur makes it write + * cur+payload (observed 134+1=135) + * - a completed value far past the + * guest's queued count, which + * trips the guest UVM assert + * (uvm_channel.c:204 completed <= + * queued) and kills the device. + * cur=0 makes inc-apply land the + * payload itself, exactly the + * re-seed the client polls. + */ + cur = 0; svc.pool_residue++; if (svc.pool_residue_log < 32) { svc.pool_residue_log++; @@ -6991,7 +18970,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, gsp_sema_skip_ops); } svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -7087,7 +19067,44 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, */ svc.fwd_rel_overflow++; svc_fwd_soft_copy_inline(vgpu); - if (svc_rel_apply_launch(vgpu, ch, + /* + * fail4 CONT.53: never clear a + * push's fence before its ops + * read their source. The pass's + * ops run at the pass end; the + * driver's ping-pong staging + * restages as soon as the fence + * clears, and a later op read + * then returns chunk+N bytes + * (vector_add +2 MB). Queue the + * release through the pending + * pass instead. + */ + if (n_ops > 0 || + svc.op_tr_pending) { + /* + * r316 (CONT.60): + * op_tr_pending covers + * the denied pass - the + * pass built no op + * (allow=0) but a copy is + * still owed; applying + * the fence here (the old + * n_ops==0 path) let the + * driver refill the + * ping-pong staging before + * the copy ever ran - the + * cold-run capture gap + * (vector_add run 1 + * bad=157696/193536 vs + * bad=0 warm). + */ + svc_pending_add(sem_va, + st.sem_payload, + inc ? 1u : 0u); + svc_rel_va_seen = sem_va; + svc_rel_count++; + } else if (svc_rel_apply_launch(vgpu, ch, sem_va, st.sem_payload, cur, pending, @@ -7121,7 +19138,20 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc_fwd_rel_direct_note(ch->h_object, sem_va, st.sem_payload); - if (svc_rel_apply_launch(vgpu, ch, sem_va, + /* + * fail4 CONT.53: same ops-order + * guard as above - a pass with + * pending copy ops must run + * them before its fence clears. + */ + if (n_ops > 0) { + svc_pending_add(sem_va, + st.sem_payload, + inc ? 1u : 0u); + svc_rel_va_seen = sem_va; + svc_rel_count++; + } else if (svc_rel_apply_launch(vgpu, ch, + sem_va, st.sem_payload, cur, pending, inc, inc_apply)) { @@ -7141,7 +19171,16 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, off += 4 + count * 4; } - if (gsp_sema_dbg && ch->h_object && svc.entry_log < 512) { + /* + * SESSION 149 R14: close an open GL/3D section when the pass ends - + * this is the normal capture point (one section per large GL push). + */ + if (gsp_sema_gl_xlate && svc_gl.seen) + svc_gl_flush(vgpu, ch, &st, buf, len, push_va, trusted, + replay, off); + + if (gsp_sema_dbg && ch->h_object && + svc.entry_log < 16384) { svc.entry_log++; { u8 tail[24]; @@ -7207,6 +19246,12 @@ struct svc_pending_release { u64 va; u32 payload; u32 op; /* 0 = RELEASE, 1 = REDUCTION (add payload) */ + u32 h; /* r316: channel the release was decoded for; the + * verified pool resolver is channel-keyed and h=0 + * never matches its per-channel entries (matmul + * r322: pool-rel cand resolved at decode (rrc=0, + * h=channel) while svc_apply_pending with h=0 + * starved forever and hung the client). */ /* * Resolution attempts that found no translation for this VA. * Telemetry for ISSUE-0001: a pending release that keeps failing to @@ -7223,25 +19268,67 @@ static struct svc_pending_release svc_pending[SVC_PENDING_MAX]; static u32 svc_pending_n; static u32 svc_pending_next; +static u64 svc_pending_rescued; /* stall-class: entries the old >= dedup would have dropped */ +static u32 svc_pending_rescued_log; +/* + * STALLFREE: the pending ring silently overwrote its oldest entry when full + * (an evicted release is a client wait that never clears). The counters are + * hoisted next to the ledger counters so the periodic diag (above) can print + * them; the ring is a bounded queue, so under churn this is the back-pressure + * signal, not an error path. + */ + static void svc_pending_add(u64 va, u32 payload, u32 op) { u32 i; if (!payload) return; - for (i = 0; i < svc_pending_n; i++) - if (svc_pending[i].va == va && - svc_pending[i].payload >= payload) + for (i = 0; i < svc_pending_n; i++) { + if (svc_pending[i].va != va) + continue; + /* + * Stall-class fix (R1): only an EXACT duplicate may be + * dropped. The old rule dropped every release behind an + * already-queued higher payload; in a burst the queue could + * end up holding only the last value (observed: payloads 7..34 + * decoded, queue held {7,34}, the guest word stopped short of + * the waiter's value and the client spun forever with + * starved=0). Distinct releases must always queue - the apply + * pass already stale-drops values the word has passed. + */ + if (svc_pending[i].payload == payload && + svc_pending[i].op == op) return; + if (svc_pending[i].payload >= payload) { + svc_pending_rescued++; + if (svc_pending_rescued_log < 32) { + svc_pending_rescued_log++; + pr_info("sema-svc: pending rescue va=0x%llx pay=%u op=%u (queued; ahead entry pay=%u op=%u)\n", + va, payload, op, + svc_pending[i].payload, + svc_pending[i].op); + } + } + } if (svc_pending_n < SVC_PENDING_MAX) { i = svc_pending_n++; } else { i = svc_pending_next++ % SVC_PENDING_MAX; + svc_pending_ovf++; + if (svc_pending_ovf_log < 8) { + svc_pending_ovf_log++; + pr_warn("sema-svc: PENDING-OVF ring full (%u); overwriting oldest entry va=0x%llx pay=%u op=%u (n=%llu)\n", + SVC_PENDING_MAX, svc_pending[i].va, + svc_pending[i].payload, svc_pending[i].op, + svc_pending_ovf); + } } svc_pending[i].va = va; svc_pending[i].payload = payload; svc_pending[i].op = op; svc_pending[i].tries = 0; + svc_pending[i].h = svc.cur_ch; } /* @@ -7254,21 +19341,94 @@ static void svc_pending_add(u64 va, u32 payload, u32 op) * exist; drop them instead of resolving them against the next session. */ static void svc_gr_diag_reset(void); /* m150: defined with the fill diag */ +/* r604: defined next to the per-VA defer tables; clears both on session end + * (they are keyed by push VA alone and the UVM pool reuses those VAs). */ +static void svc_gr_defer_reset(void); static void svc_session_flush(void) { + /* r707x: per-client witness budgets. */ + svc_pushfail_log = 0; + svc_rdw_log = 0; + svc_pushdec_log = 0; + /* SESSION 132: util-sweep witnesses + their dedup keys are per-client. */ + svc_util_dec_log = 0; + svc_gr_util_log = 0; + memset(svc_util_dec_keys, 0, sizeof(svc_util_dec_keys)); + /* SESSION 143: the per-channel sweep-visibility keys are per-client. */ + svc_util_seen_n = 0; + memset(svc_util_seen_keys, 0, sizeof(svc_util_seen_keys)); unsigned long flags; u32 i; + /* + * m167: never wipe a LIVE session. The flush is requested when a + * channel close leaves the table empty, but that transition fires + * during ordinary UVM channel cycling and the deferred execution can + * land after the same client (or the next client, when its chan_add + * raced the requested flush) has registered channels again - r169: + * the flush executed 2.5 s after smiley's launch with its channels + * live (registered 7 s earlier), wiped the queued completion releases, + * and the client spun forever in uvm_channel_is_value_completed + * (release never applied; pool word frozen). A live channel's session + * owns the tables - skipping is always safe because a genuine session + * end re-arms the flush when its own channels close. + */ + { + bool live = false; + bool forced = READ_ONCE(svc_sess_flush_forced); + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) + if (svc.chan[i].valid) { + live = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (live && !forced) { + pr_info("sema-svc: session flush skipped: live channel(s) present (deferred flush raced a new session registration)\n"); + return; + } + if (forced) { + svc_sess_flush_forced = 0; + pr_info("sema-svc: session flush (forced) executing with live=%d channel(s)\n", + live); + } + } + svc_gr_diag_reset(); /* m150: per-session diagnostic budgets */ + svc_gr_defer_reset(); /* r604: per-session defer latches */ + svc_grpark_reset(); /* r633: per-session parked GR releases */ + svc_gr_synth_ep_reset(); /* r633: per-session synthetic budgets */ + svc_dst_guard_reset(); /* r609: per-session device-dst windows */ svc.map_n = 0; svc_pb_n = 0; + svc_pb_gr_done = 0; svc_pd_found = 0; svc_pd_attempts = 0; svc_pd_best = 0; svc_pd_cands = 0; svc_pd_ev_hash = 0; - svc.direct_base = 0; + /* + * m177c: the UVM ring-space directory is per-client guest memory + * (and its samples come from this session's rings): reset with the + * session like the primary slot. + */ + svc_pd_r_found = 0; + svc_pd_r_attempts = 0; + svc_pd_r_base = 0; + svc_pd_r_ev_hash = 0; + svc_pd_r_n = 0; + /* + * m177: svc.direct_base is NOT cleared here. It is the guest kernel's + * direct-map base - a per-BOOT constant, identical for every client + * session of the same guest boot - and it is the gate and input of + * the pin-free resolution paths (svc_targeted_resolve). Clearing it + * per session forced every client to wait for a rebuild scan while + * its first cudaMalloc polled, and made the module depend on the + * host pin tools entirely. It is wiped on a guest reboot by + * svc_guest_reset_wipe() (gsp_sema_service_guest_reset()). + */ svc_pending_n = 0; svc_pending_next = 0; /* @@ -7279,6 +19439,18 @@ static void svc_session_flush(void) svc_opdef_reset_all(); svc_pool_pa_n = 0; svc_pool_pa_next = 0; + /* + * m168 addendum (r171): the UVM semaphore pool is per-CLIENT, not + * per-boot. The boot cache's conflict line proved it: the same VA + * resolved to a different PA for the next client + * (cached=0x1247b5000 new=0x1321fa000). A cache surviving the flush + * therefore served the previous client's dead page: the release was + * applied (read-back passes on a dead-but-readable page), dropped + * from pending, and the live client spun forever. Session-scope the + * cache; the scanner's per-client resolution is the source of truth. + */ + svc_pool_boot_n = 0; + svc_pool_boot_next = 0; svc_retry_next = 0; svc_fwd_rel_n = 0; svc_grce_dst_ring_next = 0; @@ -7298,6 +19470,7 @@ static void svc_session_flush(void) svc_pb_last = 0; svc_pb_last_n = 0; svc_gr_ring_done = 0; + svc_gr_ring_scan_done = 0; svc_ring_n = 0; /* * The learned tables are session state too: the primary VA->PA pin @@ -7323,8 +19496,37 @@ static void svc_session_flush(void) * name the previous client's pages (ISSUE-0002 fan-out). */ memset(svc_ipin_alt, 0, sizeof(svc_ipin_alt)); + /* + * r820: PROMOTE_CTX records name one client's kernel buffers, and a + * new client's channels re-announce their own (the same VAs can back + * with different PAs across sessions): session state. A flush skips + * entirely while any channel is live, so a running client's records + * cannot be wiped under it. + */ + memset(svc_promo, 0, sizeof(svc_promo)); + svc_promo_seq = 0; + svc_promo_hit_log = 0; memset(svc_cnt_der, 0, sizeof(svc_cnt_der)); svc_cnt_der_n = 0; + /* r551: delivered-release marks are per-session (same handles reused). */ + memset(svc_opdef_done, 0, sizeof(svc_opdef_done)); + svc_opdef_done_next = 0; + /* + * r565: the GR exactly-once memories are boot-scoped caches keyed + * (partly) by push VA, and the UVM pool reuses the same launch push + * VAs for every client of the boot. svc_gr_cap_hist matches on the VA + * ALONE (8 entries, no hash), so a previous client's capture could + * classify a later client's fresh launch as "already captured" and + * skip it (`LAUNCH-SKIP (replay; already captured=1)`); the hash-keyed + * svc_gr_seen and the content-keyed svc_inline_once are lower risk but + * share the same lifetime. Reset all three per session - they refill + * within the session from the live client's own pushes. + */ + memset(svc_gr_cap_hist, 0, sizeof(svc_gr_cap_hist)); + svc_gr_cap_hist_pos = 0; + memset(svc_gr_seen, 0, sizeof(svc_gr_seen)); + memset(svc_inline_once, 0, sizeof(svc_inline_once)); + svc_inline_once_next = 0; /* * Re-arm the diagnostic log budgets per session. The first client of * a boot would otherwise spend every budget and the next session's @@ -7333,9 +19535,32 @@ static void svc_session_flush(void) * because the counters were already exhausted by the first client). */ svc.counter_rel_dbg = 0; + svc.crel_entry_dbg = 0; + svc.walk_dbg = 0; + svc.gr_rel_auth_log = 0; svc.counter_noslot_dbg = 0; svc.sem256_log = 0; svc.xfer_raw_log = 0; + svc.pushtrace_all = 0; + svc.opadd_log = 0; + svc.dmatrace_log = 0; + svc.copyent_log = 0; + svc_wait_log = 0; + svc_wait_pass_log = 0; + svc_remapconst_log = 0; + svc_pitch_log = 0; + svc_fill_raw_log = 0; + svc_lowfill_log = 0; + svc_tabcopy_log = 0; + svc_fill_noremap = 0; + svc_wait_n = 0; + svc_wait_rel_have = false; + memset(svc_wait_holds, 0, sizeof(svc_wait_holds)); + svc.bigrun_log = 0; + svc.lid_hdr_log = 0; + svc.skiprun_log = 0; + svc.mystery_log = 0; + svc.next_hdr_log = 0; /* * The push decode budget (`sema-push` lines) is per-session too: the * first client of a boot otherwise spends it and the next session's @@ -7346,6 +19571,14 @@ static void svc_session_flush(void) svc.entry_log = 0; svc_pushalt_log = 0; /* + * m169: the op-trace budget is session state too. Left un-reset, the + * first client of a boot spends it and the failing client's copy ops + * are invisible in exactly the runs that need them (r172: streams2's + * H2D evidence was blacked out because the shared budget was spent at + * 29704, before the client ran). + */ + svc.op_trace_log = 0; + /* * Per-entry/stream decode diagnostics are per-session: the first * client otherwise spends them and the second client's CE->GR * transition is invisible (r24-r30: c2's follow-up pushes never @@ -7357,6 +19590,10 @@ static void svc_session_flush(void) memset(svc_put_log_at, 0, sizeof(svc_put_log_at)); svc_userd_dbg = 0; memset(svc_userd_dbg_ch, 0, sizeof(svc_userd_dbg_ch)); + memset(svc_ce_firstsub_n, 0, sizeof(svc_ce_firstsub_n)); + memset(svc_ce_hold_since, 0, sizeof(svc_ce_hold_since)); + memset(svc_ce_firstsub_off, 0, sizeof(svc_ce_firstsub_off)); + svc_fb_claim_reset(); svc_userd_sweep_log = 0; memset(svc_userd_sweep_at, 0, sizeof(svc_userd_sweep_at)); memset(svc_userd_sweep_n, 0, sizeof(svc_userd_sweep_n)); @@ -7364,6 +19601,8 @@ static void svc_session_flush(void) memset(svc_retry_q, 0, sizeof(svc_retry_q)); memset(svc_pool_pa, 0, sizeof(svc_pool_pa)); memset(svc_fwd_rel, 0, sizeof(svc_fwd_rel)); + memset(svc_inline_once, 0, sizeof(svc_inline_once)); + svc_inline_once_next = 0; memset(svc_grce_dst_ring, 0, sizeof(svc_grce_dst_ring)); for (i = 0; i < SVC_VAS_SLOTS; i++) { if (svc_vas[i].data) @@ -7373,11 +19612,118 @@ static void svc_session_flush(void) } svc_vas_n = 0; memset(svc.lat, 0, sizeof(svc.lat)); + /* m171 (AGENT-F R2): the write/read ledger is per-session. */ + svc_ledger_clear(); spin_lock_irqsave(&svc.vas_lock, flags); for (i = 0; i < SVC_MAX_VASPACES; i++) svc.vas[i].valid = false; spin_unlock_irqrestore(&svc.vas_lock, flags); - pr_info("sema-svc: session flush (all channels closed): discovery state reset for the next client\n"); + /* + * m176: carryover state that previously survived the flush and could + * poison the next client: + * - user-verified scan failures (tr_failed) held the pool VA page for + * 60 s even after the new client's page was reachable by targeted + * scan - clear them so the next session starts unblocked; + * - a forwarding hold whose channel is gone is resolved against a + * recycled handle with stale entry indices - drop it (no channels + * are live by definition of this flush); + * - the captured GR candidate belongs to the dead session's QMD/push; + * offering it to a recycled channel would replay a dead launch. + */ + memset(tr_failed, 0, sizeof(tr_failed)); + tr_fail_next = 0; + svc_fwd_hold_clear(); + memset(&svc_gr_cand, 0, sizeof(svc_gr_cand)); + /* m176: re-arm the opdef diagnostic budget (it was never per-session, + * so a late client's strand could be silent). */ + svc_opdef_log = 0; + /* One-cycle diag budgets (C2/C3/C7/C8/C9/C10) are per-session too. */ + svc_relx_log = 0; + svc_opdef_clr_log = 0; + svc_busy_defer_log = 0; + svc_hold_fb_log = 0; + svc_cand_ovw_log = 0; + svc_inline_recs_log = 0; + /* + * r401: the exactly-once capture memory is session-scoped. A new + * client's pushes restart their release counters, so their content + * (and hence hash) can collide with the previous client's - keeping + * the table would wrongly classify a fresh launch of the new + * session as already captured (LAUNCH-SKIP, silent drop). + */ + memset(svc_gr_seen, 0, sizeof(svc_gr_seen)); + svc_gr_seen_stale = 0; + /* m176: new session generation - channels and scrubbers re-stamp on + * registration and never inherit per-session learned state. */ + svc_sess_gen++; + pr_info("sema-svc: session flush (all channels closed): discovery state reset for the next client (gen %u)\n", + svc_sess_gen); +} + +/* + * m177: boot-scoped wipe for a GUEST boot (RM reload). Unlike the + * per-client session flush this clears the state that is valid for one + * guest boot only: the direct-map base and every content-scanned table. + * Runs on the svc thread (svc_guest_reset_wipe is invoked from the thread + * loop below), the only writer of this state. + */ +static void svc_guest_reset_wipe(void) +{ + svc.direct_base = 0; + svc.map_n = 0; + svc_pd_found = 0; + svc_pd_attempts = 0; + svc_pd_best = 0; + svc_pd_cands = 0; + svc_pd_ev_hash = 0; + svc_pd_r_found = 0; + svc_pd_r_attempts = 0; + svc_pd_r_base = 0; + svc_pd_r_ev_hash = 0; + svc_pd_r_n = 0; + svc_pb_n = 0; + svc_pb_gr_done = 0; + svc_pool_boot_n = 0; + svc_pool_boot_next = 0; + svc_pool_pa_n = 0; + svc_pool_pa_next = 0; + memset(svc_pool_pa, 0, sizeof(svc_pool_pa)); + memset(tr_failed, 0, sizeof(tr_failed)); + tr_fail_next = 0; + memset(svc.lat, 0, sizeof(svc.lat)); + pr_info("sema-svc: guest reset: boot-scoped discovery state wiped (direct_base, map, caches)\n"); +} + +/* + * m177: called by gsp_rm_emu_reset() when the guest's GSP-RM is (re)loaded + * - a new guest boot, where every VA->PA relation can have changed. The + * request is latched and executed on the svc thread, which owns the + * discovery state (a cross-thread wipe here would race svc_rebuild). + */ +void gsp_sema_service_guest_reset(void) +{ + WRITE_ONCE(svc_guest_reset_req, 1); + if (svc.task) + wake_up_process(svc.task); +} + +/* + * r317: queue a pass's recorded forward releases through the pending + * machinery instead of applying them at a hold break / fallback while an + * op of that pass is still owed. Applying the fence early let the driver + * refill the ping-pong staging before the copy read it (the cold-run + * capture gap: vector_add run 1 of every cold boot, last ~2 chunks). The + * pending store applies each release once the word resolves (the copy has + * run by then, since the op execution is what clears the pending state on + * the retry). + */ +static void svc_fwd_rel_queue_pending(struct svc_fwd_rel *rel, u8 n) +{ + u8 i; + + for (i = 0; i < n; i++) + svc_pending_add(rel[i].va, rel[i].payload, + rel[i].inc_claimed ? 1u : 0u); } /* @@ -7411,8 +19757,11 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) * resolver: the learned map can answer with a stale * page whose value makes this release look already * delivered and drops it (see svc_sem_write). + * r316: resolve with the channel the release was + * decoded for (h=0 never matches the per-channel + * pool entries - matmul r322 starved/hung). */ - if (svc_pool_pa_resolve(vgpu, 0, p->va, &ppa, &pap, + if (svc_pool_pa_resolve(vgpu, p->h, p->va, &ppa, &pap, NULL)) { /* not translatable yet: keep (ISSUE-0001) */ p->tries++; @@ -7440,7 +19789,7 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) * a pool VA the base may not have landed: applying while * the word is still 0 is absorbed, so hold and retry. */ - if (svc_pool_sem_va(p->va) && svc_pool_hold_active()) { + if (svc_pool_sem_va(p->va) && svc_pool_hold_active_va(p->va)) { p->tries++; svc_pending_note_starved(p); i++; @@ -7466,7 +19815,7 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) if (!svc_sem_write(vgpu, 0, p->va, p->op ? cur + p->payload : p->payload)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg) pr_info("sema-svc: deferred release va=0x%llx payload=%u (was cur=%u)\n", p->va, p->payload, cur); @@ -7598,6 +19947,25 @@ static u32 __maybe_unused __svc_apply_page_data_disabled(struct nv_vgpu_device * } /* + * F1-b: while the forwarded GR launch this release belongs to + * is still in flight, the sweep must not author its QMD + * completion word - the backend poll would break on a + * host-authored ack (r198: qmd_rel satisfied ~6 ms after the + * doorbell, zero engine writes). Defer: nothing is written or + * consumed here; the release stays in the guest page and later + * sweep passes re-see it, while the hold/fallback path owns a + * failed hardware copy. + */ + if (gsp_sema_gr_swrel_gate && svc_gr_rel_inflight(va, NULL)) { + if (svc.gr_rel_auth_log < 256) { + svc.gr_rel_auth_log++; + pr_info("sema-diag: GR-REL-AUTHOR src=sweep va=0x%llx pay=%u h=0x%x DEFER\n", + va, payload, 0u); + } + continue; + } + + /* * A release at page offset 0x10 with payload 1 in front of a * channel's first push identifies the page as that channel's * push-buffer base (see svc_pb_add). @@ -7606,9 +19974,21 @@ static u32 __maybe_unused __svc_apply_page_data_disabled(struct nv_vgpu_device * (lo & 0xfff) < 0x20 && svc_pb_first_push(buf)) svc_pb_add(pa_page, lo & 0xfff); + /* + * F1-a: tag software authorship of the forwarded launch's + * release word (gate off, or the request already resolved and + * the forward progress rule still allows the write). + */ + if (svc_gr_rel_match(va, NULL) && + svc.gr_rel_auth_log < 256) { + svc.gr_rel_auth_log++; + pr_info("sema-diag: GR-REL-AUTHOR src=sweep va=0x%llx pay=%u h=0x%x\n", + va, payload, 0u); + } + if (svc_sem_write(vgpu, 0, va, payload) == 0) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); applied++; if (payload != cur + 1) { svc.releases_jumped++; @@ -7639,7 +20019,6 @@ static u32 svc_apply_page(struct nv_vgpu_device *vgpu, u8 *buf, u64 pa_page) * *semaphore* address inside a release needs to resolve through the map. */ #define SVC_SCAN_WINDOW (32ULL << 20) /* 32 MB per pass */ -#define SVC_SCAN_CHUNK (64 * 1024) /* 16 pages per guest read */ #define SVC_MBUF_MAX 6 /* rescue pin candidates */ /* @@ -7688,8 +20067,8 @@ static void svc_apply_pushes(struct nv_vgpu_device *vgpu) if (!buf) return; } - if (end > SEMA_PA_MAX) - end = SEMA_PA_MAX; + if (end > svc_pa_max()) + end = svc_pa_max(); svc_apply_hot(vgpu); @@ -7707,7 +20086,7 @@ static void svc_apply_pushes(struct nv_vgpu_device *vgpu) } } - svc_scan_cursor = (end >= SEMA_PA_MAX) ? 0 : end; + svc_scan_cursor = (end >= svc_pa_max()) ? 0 : end; } /* @@ -7734,9 +20113,9 @@ static void svc_apply_sweep_slice(struct nv_vgpu_device *vgpu) u32 pages = 0; bool complete; - if (end > SEMA_PA_MAX) - end = SEMA_PA_MAX; - complete = (end >= SEMA_PA_MAX); + if (end > svc_pa_max()) + end = svc_pa_max(); + complete = (end >= svc_pa_max()); /* * Back off after a full pass that applied nothing: a stalled channel @@ -7787,6 +20166,52 @@ static void svc_apply_sweep_slice(struct nv_vgpu_device *vgpu) } } +/* + * SESSION 153-CONT-34: watch the emulated BAR0 shadow's PCFG window + * (0x88000..0x89000) for doorbell writes the guest made directly to the + * mmap'd shadow (those never reach the trap path). Each changed word is fed + * to the doorbell path, whose only effect is waking the pump earlier - the + * pump itself stays per-pass. The first call for a guest snapshots the window + * without firing, so pre-existing register values (the emulation's own init + * writes) cannot ring. + */ +static u32 svc_bar0_pcfg_log; + +static void svc_bar0_pcfg_scan(struct nv_vgpu_device *vgpu) +{ + struct svc_guest_sess *sess; + size_t bsz = 0; + u8 *base = svc_bar0_shadow(vgpu, &bsz); + u32 i; + + if (!base || bsz < SVC_BAR0_PCFG_HI) + return; + sess = svc_guest_get(vgpu, true); + if (!sess) + return; + for (i = 0; i < SVC_BAR0_PCFG_WORDS; i++) { + u32 v; + + memcpy(&v, base + SVC_BAR0_PCFG_LO + i * 4, 4); + if (!sess->pcfg_have) { + sess->pcfg_snap[i] = v; + continue; + } + if (v == sess->pcfg_snap[i]) + continue; + sess->pcfg_snap[i] = v; + if (!v) + continue; /* a clear, not a doorbell */ + gsp_sema_service_doorbell(v); + if (svc_bar0_pcfg_log < 32) { + svc_bar0_pcfg_log++; + pr_info("vgsp-sema: PCFG doorbell (BAR0 shadow scan) off=0x%x val=0x%x\n", + SVC_BAR0_PCFG_LO + i * 4, v); + } + } + sess->pcfg_have = true; +} + static void svc_pb_note(struct nv_vgpu_device *vgpu, const u8 *buf, u64 pa_page) { u32 hdr, hi, lo, payload; @@ -7881,7 +20306,7 @@ static void svc_pb_discover(struct nv_vgpu_device *vgpu) svc_pb_passes++; svc_pb_last = jiffies; svc_pb_last_n = svc_pb_n; - for (gpa = 0; gpa < SEMA_PA_MAX; gpa += SVC_SCAN_CHUNK) { + for (gpa = 0; gpa < svc_pa_max(); gpa += SVC_SCAN_CHUNK) { u32 o; if (READ_ONCE(svc_sess_flush)) @@ -8111,9 +20536,23 @@ static int svc_pb_win_lookup(const struct svc_channel *ch, u64 va_page, { u32 slot, i; - if (!ch || !ch->engine_type || ch->ce_index >= 32) + if (!ch) return -ENOENT; - slot = 4 * ch->ce_index; + /* + * R10 (patch A): GR (eng==0) channels whose ring was content- + * validated own a ring-anchored window pair (svc_pb_pair_gr). + * The CE path is unchanged; a GR channel without a paired + * window is rejected exactly as before. + */ + if (!ch->engine_type) { + if (ch->gr_index >= 32 || !ch->gr_ram_ring) + return -ENOENT; + slot = SVC_PB_GR_SLOT_BASE | ch->gr_index; + } else { + if (ch->ce_index >= 32) + return -ENOENT; + slot = 4 * ch->ce_index; + } for (i = 0; i < svc_pb_n; i++) { u64 pa; @@ -8124,7 +20563,7 @@ static int svc_pb_win_lookup(const struct svc_channel *ch, u64 va_page, va_page >= svc_pb[i].va + 0x100000ULL) continue; pa = svc_pb[i].pa_page + (va_page - svc_pb[i].va); - if (pa >= SEMA_PA_MAX) + if (pa >= svc_pa_max()) return -ENOENT; *pa_page = pa; return 0; @@ -8182,7 +20621,14 @@ static bool svc_push_prefix_ok(const u8 *buf, size_t len) if (off + 4 > len) return true; /* all zeros: not refutable here */ op = (hdr >> 29) & 0x7; - if (op == 4 || op == 7) /* header-only methods */ + /* + * r715: op 0 (GRP0_USE_TERT: SET/STORE/USE_SUB_DEV_MASK) and op 2 + * (GRP2_USE_TERT) are 4-byte header-only formats per the class + * header; the GR submit pushes start with 0x0001fff0 + * (SET_SUBDEVICE_MASK), so a correct injected pin for such a page + * was rejected here and the read fell through to the walk. + */ + if (op == 0 || op == 2 || op == 4 || op == 7) /* header-only */ return true; if (op != 1 && op != 3 && op != 5) return false; @@ -8231,12 +20677,214 @@ static u32 svc_pushpg_split_log; * validated candidate). Reads stay defensively copied through * svc_read_ap()/vfio_dma_rw(), which enforce the aperture bounds. */ +/* r707d: bounded push-read failure witness (see below). */ +static u32 svc_pushfail_log; +/* + * r707e: last content-backed ring pair resolved by any channel. The VAS + * candidate verify is retried with it on EVERY pump pass: the parked + * COPY_SERVER_PDES candidate for the client's user VAS parks at + * device-create and must register before the submit's first push read + * (a failed GR entry read is never retried with a fresh VAS), but the ring + * resolution and the park race within one pass - the every-pass retry + * closes that race in a single boot. + */ +static u64 svc_lastgood_ring_va; +static u64 svc_lastgood_ring_pa; + + +/* + * SESSION 154 (M-GL1.2): the PA ceiling for the push read. The GL client's + * sysmem allocations live ABOVE SEMA_PA_MAX (the ring scan already reads to + * gsp_sema_gr_scan_top for exactly this client - r908: its ring page at gpa + * 0x231e5d000); the content anchor pinned its push page at pa 0x2328e8000 + * (r920, measured) and the old 6 GiB read bound then rejected the freshly + * pinned page (src=2 -> -EIO) while the stale window answered instead + * (src=3 garbage, every push "unreadable"). Use the same widened ceiling + * the scans use. + */ +static u64 svc_push_pa_ceiling(void) +{ + u64 top = READ_ONCE(gsp_sema_gr_scan_top); + u64 reg = gsp_phys_guest_ram_top(); + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PA_MAX) + top = SEMA_PA_MAX; + /* + * SESSION 153-CONTINUED-67: never let the ceiling exceed the highest + * GPA any registered guest-RAM range covers - a read above it can + * only return the backing file's padding zeros, never content. With + * the 8/10 GB guest layout (0-2 GB + 4-10 GB) the registered top is + * what it is and inside-range answers (like the r1127 stale walk at + * 0x23921d000) stay readable-but-zero; the repeated-zero + * invalidation above handles those, this bounds the rest. + */ + if (reg && reg < top) + top = reg; + return top; +} + +/* + * SESSION 153-CONTINUED-51: deterministic push-page resolution. + * + * The direct GL client's 5-7 KB window pushes routinely read back all + * zero through the trusted chain on some boots ("sema-pushzero ... + * rescued=0", r1033-class) while the SAME VA pages answer real content + * on other boots - the learned cache (lat) can shadow a page-table walk + * that has since moved, and the client reuses/reallocates its push VAs + * across frames. The launch-page chain (svc_gr_page_note) resolves + * through the ledger then a FRESH walk; the push read chain consults the + * learned cache FIRST, so a stale lat/pin entry shadows the fresh walk + * forever. When the trusted read answered an all-zero page, consult the + * same ledger/walk chain the launch pages use and, when a DIFFERENT page + * with non-zero content answers, adopt it AND heal the read sources + * (validated pin + lat) so every later read of this page is deterministic + * instead of a per-boot lottery. + * + * Only a page whose content differs from all-zero is adopted: a + * legitimately-zero page (padding) must never steal a correct + * translation. Returns 0 when a replacement page was adopted (pa_out / + * ap_out updated and `p` refilled), -ENOENT when nothing better exists. + */ +static int svc_push_page_heal(struct nv_vgpu_device *vgpu, u32 h_object, + u64 va, u64 net_pa, int net_ap, u8 *p, + size_t chunk, u64 *pa_out, int *ap_out) +{ + u64 page_va = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 off = va & (SEMA_PAGE_SIZE - 1); + u64 cur_page = net_pa & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 lpa = 0, wpa = 0; + u32 lap = 0; + int wap = SVC_APERTURE_SYS; + static u32 heal_log; + + /* 1. The ledger: the launch-page chain's landing authority. A page a + * host write or an earlier launch resolution landed on is the + * page the client's own chain agreed about - the read chain does + * not consult it at all. */ + if (!svc_ledger_lookup(page_va, &lpa, &lap) && + (lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != cur_page) { + u32 map_ap = svc_ap_of((int)lap); + + if (svc_read_ap(vgpu, lpa + off, + map_ap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + p, chunk) == 0 && + memchr_inv(p, 0, chunk)) { + svc_pin_store(page_va, lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)map_ap); + svc_lat_store(va, lpa + off, (int)map_ap); + *pa_out = lpa + off; + *ap_out = (int)map_ap; + if (heal_log < 64) { + heal_log++; + pr_info("sema-pushheal: h=0x%x va=0x%llx ledger pa=0x%llx (was 0x%llx, zero)\n", + h_object, (unsigned long long)va, + (unsigned long long)(lpa + off), + (unsigned long long)net_pa); + } + return 0; + } + } + + /* 2. A FRESH walk, bypassing the learned cache: the guest may have + * remapped the VA after the lat entry was written. */ + if (svc_walk_vaspaces(vgpu, page_va, &wpa, &wap) == 0 && + wpa != 0 && (wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != cur_page) { + u32 map_ap = svc_ap_of(wap); + + if (svc_read_ap(vgpu, wpa + off, + map_ap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + p, chunk) == 0 && + memchr_inv(p, 0, chunk)) { + svc_pin_store(page_va, wpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)map_ap); + svc_lat_store(va, wpa + off, (int)map_ap); + *pa_out = wpa + off; + *ap_out = (int)map_ap; + if (heal_log < 64) { + heal_log++; + pr_info("sema-pushheal: h=0x%x va=0x%llx walk pa=0x%llx (was 0x%llx, zero)\n", + h_object, (unsigned long long)va, + (unsigned long long)(wpa + off), + (unsigned long long)net_pa); + } + return 0; + } + } + /* Nothing better: restore the zero state exactly (the caller's read + * filled p with zeros and must see zeros back). */ + memset(p, 0, chunk); + return -ENOENT; +} + +/* + * SESSION 153-CONTINUED-67: repeated all-zero read invalidation. + * + * The beef (direct GL 3D client) channel's pushes are re-read every pump + * pass; on the r1127/w2 boots the setup push and every draw window kept + * answering ALL-ZERO from the same cache-derived translation for the whole + * boot: "sema-pushdec: h=0xbeef0100 va=0x4001004 len=60216 src=4 + * pa=0x23921d000 ... unreadable=1 first8=00000000 00000000" and 64 + * "zero-head window ... (page mid-write; not captured)" rows - while the + * SAME VAs answered real content on other boots (r1126). A single zero + * read can be a legitimately-zero page (padding, a zeroed push slot), but + * the SAME page answering all-zero TWICE in a row through a cache-derived + * source is the stale-translation signature: the client zeroed, + * reallocated or rewrote the page and the cached pin/learned/walk answer + * keeps resolving to the dead page - every walk success re-stores its + * answer into the learned table, so nothing ever re-resolves. When the + * streak reaches two, drop the learned-cache entry (and a validated pin + * when that was the source) so the NEXT read re-runs the source chain + * (discovered map, registered-VAS walk, discovered walk, targeted + * resolve) instead of reusing the same answer; this read still returns + * what it read. Beef channels only (the GL client's push family this + * work targets); the drop is bounded and the witness bounded per boot. + */ +#define SVC_ZRW_MAX 256 +struct svc_zrw_rec { + u64 page_va; + u8 streak; +}; +static struct svc_zrw_rec svc_zrw_tab[SVC_ZRW_MAX]; +static u32 svc_zrw_next; +static u32 svc_zrw_log; + +static struct svc_zrw_rec *svc_zrw_slot(u64 page_va) +{ + u32 i; + + for (i = 0; i < SVC_ZRW_MAX; i++) + if (svc_zrw_tab[i].page_va == page_va) + return &svc_zrw_tab[i]; + i = svc_zrw_next++ % SVC_ZRW_MAX; + svc_zrw_tab[i].page_va = page_va; + svc_zrw_tab[i].streak = 0; + return &svc_zrw_tab[i]; +} + +static void svc_zrw_reset(u64 page_va) +{ + u32 i; + + for (i = 0; i < SVC_ZRW_MAX; i++) + if (svc_zrw_tab[i].page_va == page_va) { + svc_zrw_tab[i].streak = 0; + return; + } +} + static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u64 va, void *buf, size_t len, bool skip_ipin) { u8 *p = buf; + /* r820: per-read; the decode reads it right after the call. */ + svc_read_last_promote = false; + while (len) { u64 page_va = va & ~(u64)(SEMA_PAGE_SIZE - 1); size_t chunk = min_t(size_t, len, @@ -8246,6 +20894,7 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, bool have_win, done = false; bool first = (p == buf); int last_rc = -ENOENT; + int trusted_rc = 0; u32 src; have_win = !svc_pb_win_lookup(ch, page_va, &win_pa); @@ -8299,12 +20948,15 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, default: /* learned cache / discovered map / page * directories / targeted rm_mem resolve */ - if (svc_lookup_trusted(vgpu, va, &pa, &ap)) + trusted_rc = svc_lookup_trusted(vgpu, va, + &pa, &ap); + if (trusted_rc) continue; break; } - if (ap == SVC_APERTURE_SYS && pa >= SEMA_PA_MAX) { + if (ap == SVC_APERTURE_SYS && + pa >= svc_push_pa_ceiling()) { last_rc = -EIO; continue; } @@ -8360,10 +21012,194 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, */ if (src == 3 && first && !memchr_inv(p, 0, chunk)) continue; + svc_read_last_src = (u8)src; + svc_read_last_pa = pa; + /* + * SESSION 153-CONTINUED-46: zero-content rescue for the + * direct GL client's push pages. r1032/r1033 evidence: + * the beef family's 5-7 KB draws routinely read back + * ALL ZERO from the trusted chain (latch/map/walk) - + * "sema-pushdec: h=0xbeef0100 va=0xcae2894 len=7148 + * rc=0 src=4 pa=0x13d181000 ... first8=00000000 + * 00000000" - and the capture is then skipped as a + * zero-head window, so no 3D replay ever launches + * (42 such skips on one boot). This is the r817 class + * of stale/wrong kernel-buffer translations: the + * driver's OWN PROMOTE_CTX statement holds the real + * backing. When the first chunk of a big GL push reads + * all zero from the trusted chain (src >= 2; the + * host-verified injected pins are ground truth and + * stay authoritative), probe the promote booking: a + * DIFFERENT page with non-zero content is used in + * place, the learned cache is healed with it, and the + * read is tagged src=9 so the pushdec witness shows + * the rescue. Bounded per boot. + */ + if (src >= 2 && first && len >= 512u && + (ch->h_object & 0xffff0000u) == 0xbeef0000u && + !memchr_inv(p, 0, chunk)) { + u64 ppa = 0; + int pap = SVC_APERTURE_SYS; + static u32 pushzero_log; + + if (!svc_promo_lookup(page_va, &ppa, &pap) && + ((ppa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + u64 rpa = ppa + + (va & (SEMA_PAGE_SIZE - 1)); + int rrc = svc_read_ap(vgpu, rpa, pap, + p, chunk); + + if (!rrc && memchr_inv(p, 0, chunk)) { + pa = rpa; + svc_read_last_src = 9; + svc_read_last_pa = rpa; + svc_lat_store(page_va, ppa, + pap); + if (pushzero_log < 1024) { + pushzero_log++; + pr_info("sema-pushzero: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx promo pa=0x%llx rescued=1\n", + ch->h_object, + (unsigned long long)va, + (unsigned)len, + src, + (unsigned long long)pa, + (unsigned long long)ppa); + } + } else { + /* promo page unusable: put the + * zero state back exactly. */ + memset(p, 0, chunk); + if (!rrc && pushzero_log < 1024) { + pushzero_log++; + pr_info("sema-pushzero: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx promo pa=0x%llx rescued=0 (promo content zero)\n", + ch->h_object, + (unsigned long long)va, + (unsigned)len, + src, + (unsigned long long)pa, + (unsigned long long)ppa); + } + } + } else if (pushzero_log < 1024) { + pushzero_log++; + pr_info("sema-pushzero: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx rescued=0 (no distinct promo booking)\n", + ch->h_object, + (unsigned long long)va, + (unsigned)len, src, + (unsigned long long)pa); + } + } + /* + * SESSION 153-CONTINUED-51: when the zero-content + * rescue above did not produce non-zero bytes, the + * page must still be resolved deterministically - + * run the SAME ledger/fresh-walk chain the launch + * pages use (svc_gr_page_note) for this page. A + * different page with non-zero content is adopted, + * tagged src=10, and pinned so every later read of + * the page (continuation chunks and future frames) + * answers from the healed source instead of the + * learned cache. This is the r1069 class: every + * window push read zeros for a whole boot while the + * launch pages resolved fine. + */ + if (src >= 2 && + (ch->h_object & 0xffff0000u) == 0xbeef0000u && + (first ? len >= 512u : chunk >= 512u) && + !memchr_inv(p, 0, chunk)) { + u64 hpa = pa; + int hap = ap; + + if (!svc_push_page_heal(vgpu, ch->h_object, + va, pa, ap, p, + chunk, &hpa, &hap)) { + pa = hpa; + svc_read_last_src = 10; + svc_read_last_pa = hpa; + } + /* Not healed: the original (zero) source and + * its label stay; nothing was adopted. */ + } + /* + * r820: name whether the trusted answer came from a + * driver-declared PROMOTE_CTX buffer; the decode uses + * it for the zero-NOP acceptance (see the flag). + */ + if (src == 4) { + u64 promo_pa = 0; + int promo_ap = SVC_APERTURE_SYS; + + svc_read_last_promote = + !svc_promo_lookup(page_va, &promo_pa, + &promo_ap); + } + /* + * SESSION 153-CONTINUED-67: repeated all-zero read + * invalidation (see the svc_zrw block before this + * function). Count consecutive all-zero answers per + * page from cache-derived sources (validated pin / + * window / trusted chain); on the second in a row + * drop the learned-cache answer (and a validated pin + * when that answered) so the next read re-walks + * instead of reusing the same dead translation. The + * current read still returns what it read. + */ + if ((ch->h_object & 0xffff0000u) == 0xbeef0000u && + src >= 2) { + if (!memchr_inv(p, 0, chunk)) { + struct svc_zrw_rec *z = + svc_zrw_slot(page_va); + + if (z->streak < 0xff) + z->streak++; + if (z->streak >= 2) { + z->streak = 0; + svc_lat_drop(page_va); + if (src == 2) + svc_pin_drop(page_va); + if (svc_zrw_log < 64) { + svc_zrw_log++; + pr_info("sema-zrw: h=0x%x va=0x%llx page=0x%llx src=%u pa=0x%llx sub=%u all-zero x2 -> drop lat%s, re-walk next read\n", + ch->h_object, + (unsigned long long)va, + (unsigned long long)page_va, + src, + (unsigned long long)pa, + svc_trust_src, + src == 2 ? "+pin" : ""); + } + } + } else { + svc_zrw_reset(page_va); + } + } done = true; } - if (!done) + if (!done) { + svc_read_last_src = 0xff; + if (svc_pushfail_log < 96) { + u64 ipa = 0, apa = 0; + int iap = 0, aap = 0; + bool has_ipin = + (svc_ipin_lookup(va, &ipa, &iap) == 0); + bool has_alt = + (svc_ipin_alt_lookup(va, &apa, &aap) == 0); + + svc_pushfail_log++; + pr_info("sema-pushfail: va=0x%llx chunk=0x%llx page=0x%llx last_rc=%d trusted_rc=%d ipin=%d pa=0x%llx alt=%d pa=0x%llx win=%d pa=0x%llx\n", + (unsigned long long)va, + (unsigned long long)chunk, + (unsigned long long)page_va, + last_rc, trusted_rc, has_ipin, + (unsigned long long)ipa, + has_alt, + (unsigned long long)apa, + have_win, + (unsigned long long)win_pa); + } return last_rc; + } p += chunk; va += chunk; len -= chunk; @@ -8382,8 +21218,8 @@ static int svc_read_va_push(struct nv_vgpu_device *vgpu, * Read the GPFIFO get/put pair from a candidate USERD. Returns 0 when the * values are plausible for a channel with `entries` GPFIFO slots. */ -static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, - u32 entries, u32 *put, u32 *get) +static int svc_userd_probe_ex(struct nv_vgpu_device *vgpu, u64 pa, u32 as, + u32 entries, u32 *put, u32 *get, bool loose_get) { u32 p = 0, g = 0; @@ -8395,13 +21231,186 @@ static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, return -ENOENT; *put = p; *get = g; - if (p > entries || g > p) + /* + * SESSION 153-CONTINUED-38: the kernel channels' control page can + * carry an uninitialised GP_GET (measured 0x201400 at fb 0x18fe00+0x88 + * while GPPut was a live 3) - GP_GET is GPU-owned ("read only" in the + * Kepler control page) and the driver does not zero it, exactly the + * CE-layout lesson (ISSUE-0002 r47: "the GET word is GPU-owned and its + * initial content is not coherent"). The pump tracks its own position + * in last_get and rewrites GP_GET as it consumes, so `loose_get` + * callers (large GR kernel rings only) accept the block on the PUT + * alone. + */ + if (p > entries || (!loose_get && g > p)) return -EINVAL; if (!p && !g) return -EAGAIN; /* plausible but idle: callers decide */ return 0; } +static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, + u32 entries, u32 *put, u32 *get) +{ + return svc_userd_probe_ex(vgpu, pa, as, entries, put, get, false); +} + +/* + * SESSION 153-CONTINUED-34 (BAR0-shadow USERD): the direct GL client's + * channel allocation declares its USERD in the FB aperture at a 0x18fc00/ + * 0x18fe00-class offset, but the guest's GP_PUT/GP_GET writes for those + * channels land in the EMULATED BAR0 register shadow at that same offset + * (the vGPU guest driver programs its usermode/userd block through the + * register window; the FB shadow at the same offset holds nothing). Every + * existing source reads the FB/SYS shadows or scans guest RAM, so the block + * is invisible: svc_chan_userd returns -ENOENT, the pump never iterates the + * channel's GPFIFO ring and the channel retires with submissions=0. + * + * Watch policy (same spirit as the RAM-scan liveness latch): read the pair + * at bar0_shadow + userd_pa. A nonzero, coherent PUT is the guest's own + * write - the shadow is zeroed at device creation and no other writer + * targets that offset - so: + * - PUT >= 2 (two publishes already): adopt immediately; + * - PUT == 1 with a different value later (movement): adopt; + * - PUT == 1 stable past SVC_BAR0_USERD_PATIENT_MS: adopt anyway, so a + * client waiting on its single first submission cannot deadlock the + * watch (the pump then consumes that very submission); + * - unreadable/zero/incoherent: keep waiting (other sources unchanged). + * Adoption sets the BAR0 aperture so every later USERD read/write + * (svc_read_ap/svc_write_ap) and the pump's GP_GET write-back hit the same + * block the guest is updating. The walk seed mirrors the RAM-scan latch: + * the newest entries the client is waiting on, bounded to entries-1 slots. + */ +#define SVC_BAR0_USERD_PATIENT_MS 3000u + +static bool svc_bar0_userd_try(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 *put, u32 *get) +{ + size_t bsz = 0; + u8 *base = svc_bar0_shadow(vgpu, &bsz); + u64 off = ch->userd_pa; + u32 p = 0, g = 0, span; + + if (!base || !off) + return false; + if (off + SVC_USERD_GP_PUT + 4 > bsz) + return false; + /* + * Only the FB-declared family is retried here: SYS-declared offsets + * are real guest PAs the ordinary paths already read, and the CUDA/ + * kick channels resolve through their own sources before this point. + */ + if (ch->userd_as != SVC_APERTURE_FB) + return false; + + memcpy(&p, base + off + SVC_USERD_GP_PUT, 4); + memcpy(&g, base + off + SVC_USERD_GP_GET, 4); + + if (gsp_sema_dbg && (p || g) && ch->bar0_userd_log < 8) { + ch->bar0_userd_log++; + pr_info("sema-chan: BAR0 userd probe h=0x%x rank=%u off=0x%llx put=%u get=%u\n", + ch->h_object, ch->engine_type ? ch->ce_index + : ch->gr_index, + off, p, g); + } + /* Idle, incoherent or an implausible register-looking value. */ + if (!p || g > p || p > 0x200000u) + return false; + + if (ch->bar0_userd_watch_pa != off) { + ch->bar0_userd_watch_pa = off; + ch->bar0_userd_watch_put = p; + ch->bar0_userd_watch_at = jiffies; + if (p >= 2) + goto adopt; + return false; /* first sighting, p == 1: patience */ + } + if (p != ch->bar0_userd_watch_put) + goto adopt; /* movement: live writer */ + if (time_after(jiffies, ch->bar0_userd_watch_at + + msecs_to_jiffies(SVC_BAR0_USERD_PATIENT_MS))) + goto adopt; /* stable nonzero past the patience window */ + return false; + +adopt: + ch->userd_ok = true; + ch->userd_as = SVC_APERTURE_BAR0; + if (ch->gpfifo_entries > 1 && p) { + span = p < ch->gpfifo_entries ? p : ch->gpfifo_entries - 1; + ch->last_get = (p % ch->gpfifo_entries + + ch->gpfifo_entries - span) % ch->gpfifo_entries; + } else { + ch->last_get = 0; + } + ch->bar0_userd_watch_pa = 0; + *put = p; + *get = g; + pr_info("sema-chan: BAR0 USERD h=0x%x eng=%u ADOPT off=0x%llx put=%u get=%u ap=bar0 (walk seed last_get=%u; guest GP_PUT lives in the emulated BAR0 shadow)\n", + ch->h_object, ch->engine_type, off, p, g, + (u32)ch->last_get); + return true; +} + +/* r912: guest-RAM USERD shape scan (defined next to the ring RAM scan). */ +static bool svc_gr_ram_userd_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out, + u32 *put_out, u32 *get_out); + +/* + * SESSION 153-CONTINUED-37: pristine-block USERD fallback for a GR channel + * whose ring does not translate yet. + * + * Measured (r1013 fresh boot): the X driver's pushbuffer-test channel + * (class 0xc46f, 8192 entries, VA 0x120090000) reports its USERD as the FB + * memdesc base 0x30000, but the guest's GPPut write lands in emulated vidmem + * at 0xa0000 - the params' base is relative to a client pool the emulation + * cannot see (same class of problem the r496b sweep solves once the ring + * resolves). The block is pristine in the test window: the only non-zero + * bytes in [0x0, 0x100) are the GP_PUT dword at +0x8c (GP_GET at +0x88 is + * still 0, because the driver zeroes it before the NOP test). When the + * channel's VAS page directory has not registered yet, the r496b sweep + * cannot run (it validates a hit by reading the ring entries the put + * claims), so this bounded shape scan keeps the discovery alive: pending + * only (get < put), pristine page, unclaimed by any other channel, then + * adopt it exactly like the ring-validated sweep does. The r496b sweep + * still runs later and may re-adopt a ring-validated block; both paths + * claim their latch, so a foreign channel cannot take it in between. + */ +static bool svc_fb_userd_pristine_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *off_out, + u32 *put_out, u32 *get_out) +{ + u8 *base = vgpu->bars[1].vaddr; + u64 lim, off; + + if (!base || !ch->gpfifo_entries) + return false; + lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x2000000ULL); + for (off = 0; off + 0x100 <= lim; off += 0x1000) { + u32 hp, hg; + + memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); + if (!hp || hp > ch->gpfifo_entries) + continue; + memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); + /* Only a posed kick (get < put) - an idle/stale block stays + * with its owner. */ + if (hg >= hp) + continue; + if (memchr_inv(base + off, 0, SVC_USERD_GP_GET) || + memchr_inv(base + off + SVC_USERD_GP_GET + 8, 0, + 0x100 - SVC_USERD_GP_GET - 8)) + continue; + if (svc_fb_claim_foreign(vgpu, ch->h_object, 0, off)) + continue; + *off_out = off; + *put_out = hp; + *get_out = hg; + return true; + } + return false; +} + /* * Locate a channel's USERD. The allocation parameters carry an offset that is * relative to a client pool the emulation cannot see, so it is only one @@ -8419,6 +21428,12 @@ static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, * aperture stays the channel's own; a wrong second aperture only costs a * failed probe (the values must still read as a coherent put/get pair). */ +/* + * SESSION 153-CONTINUED-38: the shared entry-window gate used by every GR + * ring/entry validator (defined next to svc_gr_ring_candidate below). + */ +static bool svc_gr_entry_win_ok(u64 ring_va, u64 eva); + static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u64 ring_pa, int ring_ap, u32 *put, u32 *get, int rrc) { @@ -8428,6 +21443,22 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, : SVC_APERTURE_FB; /* + * r820 diag: the Xorg DMA-test window is 10 s and the latch hinges on + * this function's bounded scan re-runs (userd_scans < 8); a pass + * cadence stretched by a rebuild can spend every scan before the VAS + * candidate registers (r820: the ring resolved at +1 s but no latch + * line followed). One bounded row per call names the state the scan + * decides on. + */ + if (gsp_sema_dbg && !ch->engine_type && svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp h=0x%x eng=%u scans=%u rrc=%d ring_pa=0x%llx ring_ap=%d userd_pa=0x%llx as=%d ok=%u\n", + ch->h_object, ch->engine_type, ch->userd_scans, + rrc, ring_pa, ring_ap, ch->userd_pa, ch->userd_as, + ch->userd_ok ? 1 : 0); + } + + /* * ISSUE-0002 (r40): prefer the layout-derived CE USERD and accept it * while it is still idle. In a second session the registered userdMem * offset (pool-relative, see below) aliases a DIFFERENT rank's real @@ -8557,6 +21588,14 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * at the layout offset. */ if (ch->engine_type != 0) { + /* + * SESSION 153-CONT-34: before giving up on a CE channel, try + * the registered offset read as a BAR0-shadow USERD - the GL + * client's copy/2D channels declare the same FB offsets as its + * GR channel and their GP_PUT writes land in the same shadow. + */ + if (svc_bar0_userd_try(vgpu, ch, put, get)) + return 0; svc_userd_sweep_sched(ch); return -ENOENT; } @@ -8588,10 +21627,42 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } for (i = 0; i < nc; i++) { + bool loose = !ch->engine_type && ch->gpfifo_entries > 1024; + if (!cand[i]) continue; - if (!svc_userd_probe(vgpu, cand[i], as[i], ch->gpfifo_entries, - put, get)) { + if (!svc_userd_probe_ex(vgpu, cand[i], as[i], + ch->gpfifo_entries, put, get, loose)) { + /* + * SESSION 153-CONTINUED-44 (r1026): a GR kernel + * channel's candidate read of (put == 0, arbitrary + * get) is not a block the client publishes to. The + * cleared-screen fill pattern (alternating + * 0xff000000 / 0x00000000 pixels) yields exactly + * (put=0, get=0xff000000) at every 8-byte aligned + * offset, and the loose large-ring acceptance + * (g > p allowed) latched it: r1025's every + * beef0100 cycle adopted fb 0x20fc00 with put=0 + * forever (`submissions=0`, client recycles at + * 12 s) while the pre-walk-first boot's FB shape + * scan adopted the block carrying the client's + * real puts (`put=3/14/6`, submissions 29->208). + * Skip zero-put candidates for GR channels: the + * resolution re-runs every pass, and the FB/RAM + * shape scans (which require put >= 1) then adopt + * the block the client actually publishes on its + * first push. CE channels keep their layout rules + * (idle 0/0 acceptance is a documented CE case). + */ + if (!ch->engine_type && !*put) { + if (svc_zero_put_skip_log < 32) { + svc_zero_put_skip_log++; + pr_info("sema-chan: userd candidate 0x%llx/%u skipped for h=0x%x (put=0 get=%u; not a published block)\n", + cand[i], as[i], + ch->h_object, *get); + } + continue; + } /* ISSUE-0002 r46: name a fallback latch (CE wrong block). */ if (gsp_sema_dbg && ch->engine_type && ch->ce_index < 32 && @@ -8609,6 +21680,411 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, return 0; } } + + /* + * r912 (M-GL1.2): a GR channel whose ring was content-validated in + * guest RAM (gr_ram_ring) has its USERD in guest RAM too - the + * FB-shadow sweep below can never reach it. The guest-RAM shape + * scan is sliced and budgeted like the ring scan; it validates + * every entry the candidate's put claims against the ring window + * and admits the block only after its PUT advances across sightings + * (see svc_gr_ram_userd_scan; SESSION 154 liveness). The budget + * gate stays open while candidates are under watch, so the + * liveness confirmation cannot be starved by slice accounting. + * SESSION 153-CONTINUED-20: for the kick family it also stays open + * while the bounded sweep re-arm budget lasts - the watch set can + * go empty for stretches (all frozen pool entries timed out), and + * the in-scan re-arm needs a next call to re-run the sweep that + * finds the advancing block. + */ + /* + * SESSION 153-CONTINUED-38: kernel-VA rings (the direct GL client's + * 0x4000000-class channels) resolve through the channel's own VAS + * walk, never the guest-RAM content scan, so gr_ram_ring stays false + * and the RAM USERD discovery below would never run for exactly the + * channels that need it (CONT-37: beef0100-0102 ok=0, sub=0, then + * the alloc/free retry loop). Anchor the walk-resolved ring as a pin + * - only after entry0 decodes plausibly, so a stale walk answer is + * not cemented - and let the scan run on either anchor. The scan's + * entry validation still proves ownership against the ring. + */ + if (!ch->engine_type && ring_pa && !ch->gr_ram_ring && !ch->userd_ok) { + u8 re[SVC_GPFIFO_ENTRY_SIZE]; + u32 r0, r1; + u64 rva; + u32 rlen; + + if (!svc_read_ap(vgpu, ring_pa, ring_ap, re, sizeof(re))) { + memcpy(&r0, re, 4); + memcpy(&r1, re + 4, 4); + rva = ((u64)(r1 & 0xff) << 32) | + (u64)(r0 & 0xfffffffc); + rlen = ((r1 >> 10) & 0x1fffff) << 2; + if (!(rva & 0xfffff) && rlen >= 4 && rlen <= 0x10000 && + svc_gr_entry_win_ok(ch->gpfifo_va, rva)) { + svc_pin_store(ch->gpfifo_va, ring_pa, ring_ap); + svc_lat_store(ch->gpfifo_va, ring_pa, ring_ap); + } + } + } + if (!ch->engine_type && (ch->gr_ram_ring || ring_pa) && !ch->userd_ok && + (ch->gr_ram_userd_passes < svc_gr_ram_pass_budget() || + ch->gr_ram_userd_watch_n || + ((ch->h_object >= 0xbfef0000u && + ch->h_object <= 0xbfef00ffu) && + ch->gr_ram_userd_kick_sweeps < 64))) { + u64 upa = 0; + u32 uput = 0, uget = 0; + + if (svc_gr_ram_userd_scan(vgpu, ch, &upa, &uput, &uget)) { + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_SYS; + ch->userd_ok = true; + /* + * SESSION 154 (M-GL1.2): seed the walk at the live + * block's PUT. The client's put is free-running + * (measured 0->3 with entries=2), so the pump's + * `put %= entries` alone can collapse onto + * last_get and consume NOTHING - r914 stayed at + * submissions=0 for minutes with put=12 % 2 = 0 and + * last_get=0. Consume up to entries-1 positions + * ENDING at the PUT: the newest slots the client is + * waiting on; normal slot-space tracking resumes + * from there. + */ + if (ch->gpfifo_entries > 1 && uput) { + u32 span = uput < ch->gpfifo_entries ? + uput : ch->gpfifo_entries - 1; + + ch->last_get = (uput % ch->gpfifo_entries + + ch->gpfifo_entries - span) % + ch->gpfifo_entries; + } else { + ch->last_get = 0; + } + ch->gr_ram_put_seeded = uput; + ch->gr_ram_userd_latch_put = uput; + ch->gr_ram_userd_latch_at = jiffies; + ch->gr_ram_userd_confirmed = false; + *put = uput; + *get = uget; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by RAM scan -> pa=0x%llx (put=%u get=%u%s; entries ring-validated; speculative, pump confirms; walk seed last_get=%u)\n", + ch->h_object, ch->gr_index, upa, uput, uget, + uget < uput ? "" : ", idle", + (u32)ch->last_get); + return 0; + } + } + + /* + * SESSION 153-CONTINUED-76: name the gated-out case too. The churning + * GL client's failing instance showed neither a sweep line nor an + * adoption; if the ring parameter never resolves post-re-registration + * the whole r496b sweep is skipped and the failure is invisible. + */ + if (gsp_sema_dbg && !ch->engine_type && !ring_pa && !ch->userd_ok && + (ch->fb_userd_scan_n == 0u || ch->fb_userd_scan_n == 64u) && + (ch->h_object >= 0xbeef0000u && ch->h_object <= 0xbeefffffu)) { + ch->fb_userd_scan_n++; + pr_info("sema-chan: userdp no-sweep h=0x%x ring_pa=0 n=%u gr_ram_ring=%d (r496b gated on an unresolved ring)\n", + ch->h_object, ch->fb_userd_scan_n, + ch->gr_ram_ring ? 1 : 0); + } + + /* + * r496b: no candidate matched. For a GR channel whose ring is known + * (translated and pinned - the sysmem content scan or the VRAM + * conventions), the USERD may still be elsewhere: the X/GL driver + * allocates its USERD in VRAM and the CPU RM maps it through a BAR1 + * window, so the driver's GP_PUT write lands at a BAR1 offset that is + * neither the registered memdesc base nor ring-derived. Scan the + * emulated VRAM for the USERD shape and validate each hit against the + * ring: the put it claims must correspond to entries whose pushbuffer + * VAs all sit in the ring's allocation window. A foreign channel's + * USERD cannot pass that test. One scan per channel (the result + * persists through the pump's write-back), logged on latch. + */ + /* + * r820: the latch scan is capped per channel so a channel that + * never anchors cannot sweep forever. But the resolution the scan + * needs can arrive LATE: the Xorg GL channel's ring resolves only + * when the client's VAS page directory registers (COPY_SERVER_PDES + * age-out), and r820 showed every one of the 8 pre-resolution scans + * spent before that moment (the pass cadence stretched by a + * rebuild), so the 10 s DMA-test window closed with the ring + * resolved and no retry left. Once ring_pa is non-zero the scan is + * cheap (the sweep validates against a resolved ring), so it may + * keep retrying up to a wider cap; the client's own timeout still + * bounds the channel's life. + */ + if (!ch->engine_type && ring_pa && + ch->fb_userd_scan_n < 512 && + (!ch->fb_userd_scan_at || + time_after_eq(jiffies, + (unsigned long)ch->fb_userd_scan_at + + msecs_to_jiffies(25)))) { + u64 off, lim; + u8 *base = vgpu->bars[1].vaddr; + u64 best = 0, best_dist = ~0ULL; + u32 best_put = 0, best_get = 0; + bool best_pending = false; + u32 gate_n = 0, vfail_n = 0; + /* + * SESSION 153-CONTINUED-76: post-re-registration ring reads. + * The client churns its channels every ~12 s and a ring pin + * inherited from the dead instance can shadow the walk in + * svc_lookup; validated FELL through the stale pin made every + * candidate fail and the re-registered channel never got its + * USERD (top-level contention.75). Resolve the ring page + * through the client's OWN VAS walk once per sweep and retry + * each failed entry read there: the walk is the authority for + * the live session, the pin only a cache. + */ + u64 wp = 0; + int wap = SVC_APERTURE_SYS; + bool wtry = false, wused = false; + u32 woff = (u32)(ch->gpfifo_va & (SEMA_PAGE_SIZE - 1)); + + ch->fb_userd_scan_at = jiffies; + ch->fb_userd_scan_n++; + ch->userd_scans++; + lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + if (base && lim) { + for (off = 0; off + 0xa0 <= lim; off += 0x200) { + u32 hp, hg; + bool ok = true; + u32 k; + u64 dist; + bool pending; + int rck = -1; + u64 rkva = 0; + u32 rklen = 0; + + memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); + memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); + /* CONT-38: large GR kernel rings may carry an + * uninitialised GPU-owned GP_GET; the PUT alone + * gates there (see svc_userd_probe_ex). */ + if (!hp || hp > ch->gpfifo_entries || + (hg > hp && ch->gpfifo_entries <= 1024)) + continue; + gate_n++; + /* Validate: every claimed entry decodes to a + * pushbuffer VA in the ring's window. */ + for (k = 0; k < hp && k < 64; k++) { + u8 e[8]; + u64 eva; + u32 elen, a0, a1; + int rdrc; + + rdrc = svc_read_va(vgpu, + ch->gpfifo_va + + (u64)k * 8, + e, sizeof(e)); + if (rdrc) { + /* + * CONT-76 walk fallback: a stale + * or missing pin must not veto a + * candidate the live session's own + * page tables can answer. Resolve + * the ring page once per sweep and + * re-read the entry from it. + */ + if (!wtry && !wp) { + wtry = true; + if (svc_walk_vaspaces(vgpu, + ch->gpfifo_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + &wp, &wap)) + wp = 0; + } + if (wp && + (u64)woff + (u64)k * 8 + 8 <= + SEMA_PAGE_SIZE) { + rdrc = svc_read_ap(vgpu, + wp + woff + + (u64)k * 8, + wap, e, sizeof(e)); + if (!rdrc) + wused = true; + } + } + if (rdrc) { + ok = false; + rck = (int)k; + /* -1: read failure rc; else the + * window check below. */ + rkva = (u64)rdrc; + break; + } + memcpy(&a0, e, 4); + memcpy(&a1, e + 4, 4); + eva = ((u64)(a1 & 0xff) << 32) | + (u64)(a0 & 0xfffffffc); + elen = ((a1 >> 10) & 0x1fffff) << 2; + /* + * Every entry the put claims must name a + * VA in the ring's allocation window - + * pushbuffers below the ring, the + * progress tracker above it; the + * kernel-VA family is accepted in its + * own low window (CONT-38). + */ + if (!eva || !svc_gr_entry_win_ok( + ch->gpfifo_va, eva) || + elen < 4 || elen > 0x100000) { + ok = false; + rck = (int)k; + rkva = eva; + rklen = elen; + break; + } + } + if (!ok) { + /* + * r820 diag: name why a scanned + * block was rejected (read rc or + * the entry VA/len window). + */ + vfail_n++; + if (gsp_sema_dbg && !ch->engine_type && + svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp scan off=0x%llx put=%u rejected k=%d (read_rc=%lld va=0x%llx len=%u)\n", + off, hp, rck, + (long long)rkva, + rkva, rklen); + } + continue; + } + /* + * Several blocks can validate: a previous + * session leaves its old USERD holding the same + * put/entry values while the new channel's block + * sits elsewhere (observed r504: the re- + * registered channel re-scanned, adopted the + * stale block, and its DMA test timed out). + * Prefer a block with work PENDING (get < put); + * among equals prefer the one nearest the + * registered memdesc offset (the driver's own + * hint). The pump's GP_GET write-back goes to + * the adopted block, so a stale adoption strands + * the client's poll. + */ + dist = off > ch->userd_pa ? off - ch->userd_pa + : ch->userd_pa - off; + pending = hg < hp; + if (best && + (best_pending > pending || + (best_pending == pending && dist >= best_dist))) + continue; + best = off; + best_dist = dist; + best_put = hp; + best_get = hg; + best_pending = pending; + } + if (best) { + ch->userd_pa = best; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + *put = best_put; + *get = best_get; + /* + * SESSION 153-CONTINUED-68: deliberately do NOT + * seed the speculative latch here. Seeding + * armed the pump's judge on channels that + * previously accepted the VRAM candidate + * quietly; a pool block whose PUT advances in + * jumps (r1133: 219 -> 996) or a stale seed + * from an earlier RAM-scan candidate (r1132: + * latch=2 vs put=6) then rejected on every + * pass - adopt/reject spin at ~10 Hz (4638 + * REJECTED lines on r1132, the guest's Xorg + * soft-locked, glxgears starved). The r1129- + * r1131 good boots show the intended shape: + * VRAM adopts logged, no rejects, ring + * consumption via the walk. + */ + ch->gr_ram_userd_latch_at = 0; + ch->gr_ram_userd_confirmed = false; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by VRAM shape scan at fb 0x%llx (put=%u get=%u%s, ring-validated)\n", + ch->h_object, ch->gr_index, best, + best_put, best_get, + best_pending ? "" : ", idle"); + return 0; + } + /* + * r820 diag: the sweep found nothing adoptable. gates + * counts put/get-coherent blocks seen, vfails the ones + * rejected by the ring-entry validation. + * + * SESSION 153-CONTINUED-76: the global r820 budget + * (svc_userdp_log < 512) was spent by other channels + * before the churning GL client's ranks ever swept, so + * its post-re-registration failure was invisible. Log + * the sweep result for GR ranks at fixed per-instance + * scan numbers (3 lines per instance max) regardless of + * the global budget, with the walk answer included so + * the stale-pin vs dead-walk distinction is in the + * record. + */ + if (gsp_sema_dbg && + (svc_userdp_log < 512 || + (!ch->engine_type && + (ch->fb_userd_scan_n == 4u || + ch->fb_userd_scan_n == 64u || + ch->fb_userd_scan_n == 256u)))) { + svc_userdp_log++; + pr_info("sema-chan: userdp sweep h=0x%x scans=%u gates=%u vfails=%u best=0x%llx ring_pa=0x%llx walk=0x%llx used=%d\n", + ch->h_object, ch->userd_scans, + gate_n, vfail_n, best, ring_pa, wp, + wused ? 1 : 0); + } + } + } + /* + * SESSION 153-CONTINUED-37: the channel's ring VA does not translate + * yet (the client's VAS page directory registers ~1 s after channel + * creation; until then svc_lookup cannot resolve it and the + * ring-validated r496b sweep above never runs). The driver's DMA + * test only needs the block it is polling: a freshly kicked pristine + * FB block (see svc_fb_userd_pristine_scan). Adopt it so the pump + * starts iterating entries as soon as the ring walk lands, inside + * the driver's 10 s window. + */ + if (!ch->engine_type && !ring_pa && + ch->fb_userd_scan_n < 512 && + (!ch->fb_userd_scan_at || + time_after_eq(jiffies, + (unsigned long)ch->fb_userd_scan_at + + msecs_to_jiffies(25)))) { + u64 off = 0; + u32 p = 0, g = 0; + + ch->fb_userd_scan_at = jiffies; + ch->fb_userd_scan_n++; + if (svc_fb_userd_pristine_scan(vgpu, ch, &off, &p, &g)) { + ch->userd_pa = off; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_fb_claim_add(ch->h_object, 0, off); + *put = p; + *get = g; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by pristine-FB scan at fb 0x%llx (put=%u get=%u; ring 0x%llx not translated yet)\n", + ch->h_object, ch->gr_index, off, p, g, + ch->gpfifo_va); + return 0; + } + } + /* + * SESSION 153-CONT-34: last resort for the GL client family - retry the + * registered offset as a BAR0-shadow USERD. Every other source (layout, + * ring-derived, RAM scan, FB sweep) has already missed the block whose + * GP_PUT writes land in the emulated register window. + */ + if (svc_bar0_userd_try(vgpu, ch, put, get)) + return 0; return -ENOENT; } @@ -8635,6 +22111,316 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * pump's completion clear before the normal path resumes. */ #define SVC_NOTIFIER_ERR_HOLD (2 * HZ) +/* + * SESSION 134: CE first-submission capture. + * + * The CE (copy-engine) rings are placed by the guest's allocator at a + * location that varies per boot and is NOT reliably described by the + * channel's registered gpfifo VA through the discovered page tables (r763 + * matched the fixed FB convention and the sweep decoded the inline-SASS + * entry; r765-r772 read zeros everywhere, the CE channels never latched, + * the driver's inline LAUNCH_DMA code downloads never executed and the + * frames stayed black). The logged instance where ring content and USERD + * were both live is the channel's first notifier clear - the guest's + * doorbell for its first submission, processed by the pump in the same + * pass that writes the ring. At that instant this hook: + * 1. reads entry0 through the channel's own translation (svc_lookup), + * 2. reads entry0 through the trusted walk (the VAS may have just + * landed since the registration read), + * 3. scans the shadow (0x100 step, first 64 MB, the GR-scan bound) for + * a ring-shaped page whose entry0 names a 1MB-aligned push VA in the + * CE window [0x120000000,0x121000000), and for every hit runs the + * standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0; a candidate + * whose USERD reads coherent (or idle 0/0) is PINNED and LATCHED + * exactly like svc_ring_autopin's CE path. + * Bounded: at most 4 captures per CE channel, 16 witness rows per boot; + * resets with the session (svc_session_flush). + */ +static u64 svc_ce_firstsub_log; + +/* + * SESSION 134: bounded completion hold for unlatched CE channels. + * + * The clear write in svc_notify_complete is what makes the guest treat the + * submission as consumed; the driver then recycles the ring and its content + * is gone before the client's VAS lands (sema_vas_cand_ms), leaving the CE + * ring latch with nothing readable (r769-r774: rrc=-2 at the first + * submission, the walk answering a zero page later). Holding the notifier + * busy for a bounded window keeps the driver's submission in flight - and + * with it the ring content that the regular per-pass latch path + * (svc_chan_userd through svc_lookup) needs - until the VAS registers and + * the ring reads coherently. 0 = off (the pre-SESSION-134 behaviour). + */ +static u64 svc_ce_hold_ms; +module_param_named(sema_ce_hold_ms, svc_ce_hold_ms, ullong, 0644); +MODULE_PARM_DESC(sema_ce_hold_ms, "Hold an unlatched CE channel's completion open for up to N ms so the ring stays live for the latch (0 = off)"); + +/* + * SESSION 134c rank-aware first-submission capture (see the capture below). + * The CE block convention: ring at 0x80000 + 0x20000 * rank, USERD at + * ring + 0x10000 (observed for every CE channel across boots; the same + * convention svc_chan_userd applies in its regular per-pass latch). + */ +#define SVC_CE_RING_FB_BASE 0x80000ULL +#define SVC_CE_RING_FB_STRIDE 0x20000ULL + +/* + * One candidate offset for the CE first-submission capture: validate the + * ring shape (`scan_hits` counts shape hits) and, on a shape hit, run the + * standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0; a candidate whose + * USERD reads coherent (or idle 0/0) is PINNED and LATCHED exactly like + * svc_ring_autopin's CE path. Returns true when latched. + */ +static bool svc_ce_firstsub_try_off(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u8 *sbase, + u64 lim, u64 off, u32 *scan_hits, + u64 *latch_off, u64 *latch_upa, + u32 *latch_put, u32 *latch_get, + bool own_block) +{ + static const u64 udeltas[] = { 0x10000, 0x2000, 0x1000, 0x0 }; + u32 h0, h1, hlen; + u64 hva; + unsigned int ui; + + memcpy(&h0, sbase + off, 4); + memcpy(&h1, sbase + off + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if ((hva & 0xfffff) || hva < 0x120000000ULL || + hva >= 0x121000000ULL || hlen < 4 || hlen > 0x10000) + return false; + (*scan_hits)++; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 upa = off + udeltas[ui]; + u32 p3 = 0, g3 = 0; + int prc; + + if (upa + 4 > lim) + continue; + prc = svc_userd_probe(vgpu, upa, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3); + /* + * SESSION 144d: the rank's OWN convention block (own_block, + * the +0x10000 candidate at the rank's convention offset) is + * latched even when its stale content probes incoherent + * (g > p) or idle - the guest rewrites put/get on its next + * submission and the pump's per-channel last_get processes + * entry0 exactly once. r813: client #2's convention block at + * fb 0x90000 read put=1/get=146 (g > p -> -EINVAL) at the + * capture instant, the coherence gate rejected it, and the + * never-used 0x82000 zero block was latched instead, so the + * client's first push never drained and it hung. The gates + * below stay for every heuristic candidate (other deltas and + * the fallback scan), preserving the SESSION 135 protection. + */ + if (!own_block || ui != 0) { + if (prc && prc != -EAGAIN) + continue; + } + /* + * SESSION 135: at the FIRST-SUBMISSION instant the channel's + * submission is by definition unconsumed - the notifier clear + * this capture runs before is what would let the driver treat + * it as consumed. A candidate whose USERD reads a consumed + * state (get >= put && put > 0) cannot be this channel's live + * just-submitted work; it is a PREVIOUS session's leftover + * block surviving in VRAM (r785: the capture latched the dead + * CUDA client's caf00012 block at fb 0x80000, put=1 get=1, + * for the Vulkan channel beef0101 - the decoded entry was the + * old session's, the client stalled on its expectations). + * Idle 0/0 stays acceptable (svc_userd_probe's own rule). + */ + /* + * SESSION 144c: this consumed-state rejection is for FOREIGN + * blocks found by the heuristic deltas. The rank's OWN + * convention block (+0x10000) legitimately carries the + * previous session's consumed state until the guest + * re-initialises the reused allocation - rejecting it made + * the r812 second client fall through to a never-used + * 0x82000 block and stall forever (cefirst row: off=0x80000 + * upa=0x82000 put=0 get=0 while the client waited for the + * first push's completion). The pump's per-channel last_get + * still processes entry0 exactly once, so latching the + * convention block with stale put/get is correct. + */ + if (!own_block && p3 && p3 == g3) + continue; + /* latch exactly like the CE autopin */ + svc_pin_store(ch->gpfifo_va, off, SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, off, SVC_APERTURE_FB); + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + set_bit(ch->ce_index, &svc_ring_done); + svc_ring_n++; + *latch_off = off; + *latch_upa = upa; + *latch_put = p3; + *latch_get = g3; + return true; + } + return false; +} + +static void svc_ce_firstsub_capture(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 *sbase = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL); + u64 ring_pa = 0, wpa = 0; + u32 e0 = 0, e1 = 0, walk_e0 = 0, walk_e1 = 0; + u32 scan_hits = 0, latched = 0; + u64 latch_off = 0, latch_upa = 0; + u32 latch_put = 0, latch_get = 0; + int ring_ap = 0, wap = SVC_APERTURE_SYS; + int rrc = -1, wrc = -1; + + if (ch->engine_type == 0 || ch->ce_index >= 32) + return; + if (svc_ce_firstsub_n[ch->ce_index] >= 4) + return; + svc_ce_firstsub_n[ch->ce_index]++; + + if (sbase && lim >= 0x1000) { + /* 1. the channel's own translation at this instant. */ + rrc = svc_lookup(vgpu, ch->gpfifo_va, &ring_pa, &ring_ap); + if (!rrc && ring_pa) { + u8 e[8]; + + if (!svc_read_ap(vgpu, ring_pa, ring_ap, e, + sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + } + } + /* 2. the trusted walk at this instant. */ + if (!svc_lookup_trusted(vgpu, ch->gpfifo_va, &wpa, &wap)) { + u8 e[8]; + + wrc = 0; + if (!svc_read_ap(vgpu, wpa, svc_ap_of(wap), e, + sizeof(e))) { + memcpy(&walk_e0, e, 4); + memcpy(&walk_e1, e + 4, 4); + } + } + /* 3. shadow scan for a CE-window ring page. */ + { + /* + * SESSION 134c rank-aware order: try the rank's + * convention offset first, then the bounded scan, + * and never accept an offset another CE rank already + * latched. The rank-blind first-hit latched rank 0's + * block for rank 1 on r779 (both rows read + * off=0x80000 upa=0x90000), so the rank-1 channel's + * first release decoded from rank 0's ring/page and + * was stale-dropped (cur=1 already) - the client's + * rank-1 tracker wait never completed and it spun + * forever. A rank whose block is not in the shadow + * at this instant now simply stays unlatched for the + * regular per-pass CE latch instead of stealing a + * foreign block. + */ + u64 exp = SVC_CE_RING_FB_BASE + + SVC_CE_RING_FB_STRIDE * ch->ce_index; + u64 off; + + latched = svc_ce_firstsub_try_off(vgpu, ch, sbase, + lim, exp, &scan_hits, + &latch_off, &latch_upa, + &latch_put, &latch_get, + true); + for (off = 0; !latched && off + 0x90 <= lim; + off += 0x100) { + unsigned int ri; + bool foreign = false; + + if (off == exp) + continue; + for (ri = 0; ri < 32; ri++) { + if (ri == ch->ce_index || + !svc_ce_firstsub_off[ri]) + continue; + if (svc_ce_firstsub_off[ri] == off) { + foreign = true; + break; + } + } + if (foreign) + continue; + /* + * SESSION 144: symmetric protection - a CE + * fallback scan must not steal a block another + * channel (CE rank or GR) already claims. + */ + if (svc_fb_claim_foreign(vgpu, ch->h_object, off, + off + 0x10000)) + continue; + latched = svc_ce_firstsub_try_off(vgpu, ch, + sbase, lim, + off, &scan_hits, + &latch_off, + &latch_upa, + &latch_put, + &latch_get, + false); + } + if (latched) { + svc_ce_firstsub_off[ch->ce_index] = latch_off; + /* + * SESSION 144: publish the claim so a GR rank + * whose own ring is not written yet cannot + * latch this CE block (r805-r808: every CUDA + * GR rank stole the CE rank-0 block and the + * launch was never captured). + */ + svc_fb_claim_add(ch->h_object, latch_off, + latch_upa); + } + } + } + + if (gsp_sema_dbg && svc_ce_firstsub_log < 16) { + svc_ce_firstsub_log++; + pr_info("sema-cefirst: h=0x%x rank=%u n=%u rrc=%d rpa=0x%llx rap=%d e0=0x%x e1=0x%x wrc=%d wpa=0x%llx wap=%d we0=0x%x we1=0x%x scan_hits=%u latch=%u off=0x%llx upa=0x%llx put=%u get=%u\n", + ch->h_object, ch->ce_index, + svc_ce_firstsub_n[ch->ce_index], rrc, ring_pa, + ring_ap, e0, e1, wrc, wpa, wap, walk_e0, walk_e1, + scan_hits, latched, latch_off, latch_upa, + latch_put, latch_get); + } + if (latched) + pr_info("sema-cefirst: LATCHED h=0x%x rank=%u va=0x%llx -> fb 0x%llx (userd 0x%llx put=%u get=%u; first-submission capture)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, + latch_off, latch_upa, latch_put, latch_get); +} + +/* + * SESSION 153-CONTINUED-41 (mission: advance the client's completed side + * host-side): the direct GL client's kernel channels (h=0xbeef01xx) arm + * their OWN completion sentinel. Glcore's submit pump writes 0xffff into + * a u16 at notifier-page+0x0e+i*16 (i=0..5) and spins until the word + * changes (glcore+0xB1A313 scan loop; live: armed at +0x2e during the + * yield wait, SESSION 40/41). On real hardware the RM completion + * overwrites that word; the emulation never did, so every ladder attempt + * timed out (~11 s), the client fell back to no-op frames and GP_PUT was + * never stored. Proof by host-side emission: blindly writing 0 over the + * armed word took a live client from ~23.4 s/frame to ~11.7 s/frame. + * The channel notifier proper starts at `base` (page+0x60 for this + * family), so the client's six 16-byte records are just below it; clear + * any that read 0xffff. Only the client family is touched - the X DDX's + * bfef-family member page uses a different all-zero wait discipline and + * a 0xffff there means a delivered error, so it must not be wiped. + */ +static bool gsp_sema_client_slot = true; +module_param_named(sema_client_slot, gsp_sema_client_slot, bool, 0644); +MODULE_PARM_DESC(sema_client_slot, + "Clear the GL client's armed 0xffff completion slots (beef-family notifier pages)"); +static u32 svc_client_slot_n; + static void svc_notify_complete(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -8646,6 +22432,32 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, if (!ch->notifier_pa) return; + if (gsp_sema_client_slot && + (ch->h_object & 0xfffff000u) == 0xbeef0000u) { + u64 page = base & ~((u64)SEMA_PAGE_SIZE - 1); + int i; + + for (i = 0; i < 6; i++) { + u64 pa = page + 0x0e + (u64)i * SVC_NOTIFY_SIZE; + u16 st = 0, z = 0; + + if (pa + sizeof(st) > base) + break; + if (svc_read_ap(vgpu, pa, ch->notifier_as, + &st, sizeof(st))) + continue; + if (st != 0xffff) + continue; + if (!svc_write_ap(vgpu, pa, ch->notifier_as, + &z, sizeof(z))) { + svc_client_slot_n++; + if (gsp_sema_dbg && svc_client_slot_n <= 64) + pr_info("sema-chan: client-slot cleared h=0x%x slot=%d pa=0x%llx\n", + ch->h_object, i, pa); + } + } + } + /* * P6c C3: while a delivered fault notifier is latched the pump must * not clear the error slot. The latch clears when the guest ack reads @@ -8687,6 +22499,41 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, return; /* + * SESSION 134: a visible doorbell means a submission just landed - + * for a CE-class channel this is the one instant its ring content + * and USERD are both live. Capture/snapshot it BEFORE the clear + * write below (the clear is what makes the guest treat the + * submission as complete and recycle the ring), see + * svc_ce_firstsub_capture. + */ + if (ch->engine_type) + svc_ce_firstsub_capture(vgpu, ch); + + /* + * SESSION 134: keep an unlatched CE submission open so the ring + * content survives until the latch can read it (see + * svc_ce_hold_ms). The regular per-pass latch path retries while + * the hold runs; the hold expires un-latched into the old + * behaviour, so a channel the emulation can never resolve cannot + * strand the guest. + */ + if (ch->engine_type && !ch->userd_ok && svc_ce_hold_ms && + ch->ce_index < 32) { + if (!svc_ce_hold_since[ch->ce_index]) + svc_ce_hold_since[ch->ce_index] = jiffies; + if (time_before(jiffies, + svc_ce_hold_since[ch->ce_index] + + msecs_to_jiffies(svc_ce_hold_ms))) + return; + if (svc_ce_firstsub_log < 16) { + svc_ce_firstsub_log++; + pr_info("sema-cefirst: h=0x%x rank=%u completion hold expired (%llu ms) unlatched; clearing\n", + ch->h_object, ch->ce_index, svc_ce_hold_ms); + } + svc_ce_hold_since[ch->ce_index] = jiffies; + } + + /* * Clear both slots: they are adjacent, and whichever one the client is * polling must read as completed for a healthy channel. */ @@ -8706,7 +22553,90 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, /* Keep the completion visible to clients that start waiting * after the event fired. */ - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); + } +} + +/* + * SESSION 153-CONTINUED-42 (mission: converge the client-slot clear so a + * ladder attempt COMPLETES instead of recycling). + * + * CONT-41 found the protocol: glcore's submit pump arms a 0xffff sentinel in + * one of six u16 records just below the beef-family channel-notifier base + * (page+0x0e+i*16) and treats ANY other value as the completion. The kernel + * clear landed inside svc_notify_complete, but that hook only runs on passes + * where a completion is being processed - a client whose sentinel is armed + * while the pump has nothing else to complete still sat out its ~11 s + * deadline (r1023: attempt 2/3 had the clears logged and still retired + * submissions=0; only attempt 1 - cleared while its pushes were live - + * submitted). This sweep runs once per service pass for every live + * beef-family channel and clears ANY armed 0xffff in the six records, so + * every ladder phase releases within a pass regardless of what else the + * pump is doing. The bfef-family (X DDX member page) keeps its 0xffff + * delivered-error semantics and is not touched. + * + * Bounded: 16 channels per pass, first 96 clear events logged with the slot + * index (arm-phase tracing), then every 4096th. The log's slot index is + * what identifies which phase the client is in when it arms each record. + */ +static u32 svc_client_slot_log_n; +static unsigned long svc_client_slot_at; + +static void svc_client_slot_sweep(struct nv_vgpu_device *vgpu) +{ + struct { + u32 h; + u64 base; + u32 as; + } tab[16]; + unsigned long flags; + int i, n = 0; + + if (!gsp_sema_client_slot) + return; + /* ~1 pass per tick while the pump runs tight; plenty for an ~11 s wait. */ + if (time_before(jiffies, svc_client_slot_at)) + return; + svc_client_slot_at = jiffies + msecs_to_jiffies(2); + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS && n < ARRAY_SIZE(tab); i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu || !c->notifier_pa) + continue; + if ((c->h_object & 0xfffff000u) != 0xbeef0000u) + continue; + tab[n].h = c->h_object; + tab[n].base = c->notifier_pa; + tab[n].as = c->notifier_as; + n++; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + + for (i = 0; i < n; i++) { + u64 page = tab[i].base & ~((u64)SEMA_PAGE_SIZE - 1); + int k; + + for (k = 0; k < 6; k++) { + u64 pa = page + 0x0e + (u64)k * SVC_NOTIFY_SIZE; + u16 st = 0, z = 0; + + if (pa + sizeof(st) > tab[i].base) + break; + if (svc_read_ap(vgpu, pa, tab[i].as, &st, sizeof(st))) + continue; + if (st != 0xffff) + continue; + if (svc_write_ap(vgpu, pa, tab[i].as, &z, sizeof(z))) + continue; + svc_client_slot_n++; + svc_client_slot_log_n++; + if (gsp_sema_dbg && (svc_client_slot_log_n <= 96 || + (svc_client_slot_log_n % 4096) == 0)) + pr_info("sema-chan: client-slot cleared(sweep) h=0x%x slot=%d pa=0x%llx n=%u\n", + tab[i].h, k, pa, svc_client_slot_log_n); + } } } @@ -8729,8 +22659,6 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, * against the ring's first GPFIFO entry, so a layout change costs a failed * pin, not a wrong one. */ -#define SVC_CE_RING_FB_BASE 0x80000ULL -#define SVC_CE_RING_FB_STRIDE 0x20000ULL static unsigned long svc_ring_done; /* CE ranks whose ring is pinned */ static bool svc_fbs_scan_done; /* one-shot FB entry scan diagnostic */ @@ -8766,6 +22694,86 @@ static int svc_chan_entry0(struct nv_vgpu_device *vgpu, struct svc_channel *ch, bool gsp_sema_gr_rings = true; /* + * SESSION 132 dead-entry guard for ring candidates. + * + * The r759 false positive: a vidmem candidate (0xa0000) whose first GPFIFO + * entry (push 0x120100000/40) passed the plausibility gate - the USERD + * probe hit the channel's REAL USERD at 0xb0000, which the VRAM shape scan + * had just found 0x10000 after the candidate by coincidence - was pinned + * as the GR ring while the client's own page tables resolve the ring VA to + * a guest-RAM page (0x11623d000) whose entry0 is live (push 0x4000000/ + * 0x8bc, decoded by the util sweep). The pump then read the pinned page, + * could not fetch the entry's push (all-zero through every source), and + * skipped the entry after 39 failed rescues - the live ring stayed + * undrained and every upload behind it invisible. + * + * A candidate whose first entry cannot deliver a method stream is not this + * channel's ring. The push head is read through the client's own page + * tables (guest truth); a push that cannot be resolved at all does NOT + * mark the entry dead (the VAS may not be registered yet on an early + * pass, and vidmem-backed pushes are not in the client's tables at all). + */ +static bool svc_ring_entry_dead(struct nv_vgpu_device *vgpu, u32 e0, u32 e1) +{ + u64 va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + u32 len = ((e1 >> 10) & 0x1fffff) << 2; + u8 head[16]; + + if (!va || len < 8 || len > 0x10000) + return false; /* unusable entry; the gate rejects it */ + if (svc_read_va_trusted(vgpu, va, head, sizeof(head))) + return false; /* cannot tell: do not reject */ + return !memchr_inv(head, 0, sizeof(head)); +} + +/* + * SESSION 132: does the client's own page table (VAS walk only - never the + * pin/lat/map sources, so a bad pin cannot vote for itself) resolve the + * ring VA to a page whose first entry is LIVE? Used to reject a dead + * candidate only when a live alternative provably exists. + */ +static bool svc_ring_walk_live(struct nv_vgpu_device *vgpu, u64 gpfifo_va) +{ + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u8 we[8]; + u32 w0, w1; + + if (svc_walk_vaspaces(vgpu, gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1), + &wpa, &wap)) + return false; /* walk cannot resolve: no verdict */ + if (svc_read_ap(vgpu, wpa, wap, we, sizeof(we))) + return false; + memcpy(&w0, we, 4); + memcpy(&w1, we + 4, 4); + return !svc_ring_entry_dead(vgpu, w0, w1); +} + +/* + * SESSION 153-CONTINUED-38: kernel-VA ring entry window. The direct GL + * client's kernel channels (h 0xbeef01xx) place their GPFIFO ring and + * pushbuffers in the driver's LOW aperture (0x4000000-class, + * SVC_LOW_REL_VA_LO) instead of the userspace GPU-VA window (>= 4 GiB, + * SEMA_VA_LO) every GR gate below assumed, so their entries could never + * validate and the channels never resolved a USERD (the CONT-37 alloc/free + * retry loop). An entry VA in the low family is this channel's push only + * when it sits in the same contiguous allocation as the ring VA; accept it + * within a 256 MB window around the ring, never outside the low family. + */ +static bool svc_gr_entry_win_ok(u64 ring_va, u64 eva) +{ + if (eva + SVC_GR_RAM_WIN / 2 >= ring_va && + eva < ring_va + SVC_GR_RAM_WIN / 2) + return true; + if (eva >= SVC_LOW_REL_VA_LO && + eva < SVC_LOW_REL_VA_LO + 0x10000000ULL && + eva + 0x10000000ULL > ring_va && + eva < ring_va + 0x10000000ULL) + return true; + return false; +} + +/* * Validate a candidate translation for a GR ring: the first GPFIFO entry * must decode to a 1 MB aligned push in the client VA window with a sane * length, and the USERD at +SVC_GR_USERD_OFF must be readable through the @@ -8788,8 +22796,19 @@ static bool svc_gr_ring_candidate(struct nv_vgpu_device *vgpu, memcpy(&e1, e + 4, 4); *va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); *len = ((e1 >> 10) & 0x1fffff) << 2; - if ((*va & 0xfffff) || *va < SEMA_VA_LO || *va >= SEMA_VA_HI || - *len < 4 || *len > 0x10000) + if ((*va & 0xfffff) || *len < 4 || *len > 0x10000) + return false; + /* SESSION 153-CONTINUED-38: low-aperture kernel-VA ring entries. */ + if ((*va < SEMA_VA_LO || *va >= SEMA_VA_HI) && + !svc_gr_entry_win_ok(ch->gpfifo_va, *va)) + return false; + /* + * SESSION 132: a candidate whose entry0 push provably reads all-zero + * through the client's own page tables cannot deliver a method + * stream (the pump's rescue path eventually skips exactly these) - + * reject it so the autopin falls through to a live candidate. + */ + if (svc_ring_entry_dead(vgpu, e0, e1)) return false; /* * Gate: the USERD must be readable. rc==0 is an active get/put @@ -8801,6 +22820,20 @@ static bool svc_gr_ring_candidate(struct nv_vgpu_device *vgpu, *get = 0; prc = svc_userd_probe(vgpu, cand_pa + SVC_GR_USERD_OFF, cand_ap, ch->gpfifo_entries, put, get); + /* + * SESSION 132: the driver places the GPFIFO and the USERD + * independently (r759: ring in guest RAM 0x11623d000, USERD in + * vidmem 0xb0000), so a walk-resolved ring cannot pass a + * same-aperture USERD probe. Accept the channel's own USERD when + * one is already known - the put/get liveness gate is unchanged. + */ + if (prc != 0 && prc != -EAGAIN && ch->userd_ok && ch->userd_pa && + ch->userd_pa != cand_pa + SVC_GR_USERD_OFF) { + *put = 0; + *get = 0; + prc = svc_userd_probe(vgpu, ch->userd_pa, ch->userd_as, + ch->gpfifo_entries, put, get); + } return prc == 0 || prc == -EAGAIN; } @@ -9003,6 +23036,9 @@ static bool svc_gr_content_pin(struct nv_vgpu_device *vgpu, u32 rank, svc_pin_store(push_va, push_pa, SVC_APERTURE_SYS); svc_lat_store(push_va, push_pa, SVC_APERTURE_SYS); svc_mark_hot(push_pa); + /* m177c: a content-verified UVM-space pair - feed the ring-space + * directory discovery (counter pages resolve through it). */ + svc_pd_ring_note(push_va, push_pa); if (ring_ap == SVC_APERTURE_FB) pr_info("sema-chan: content-discovered GR ring rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u; push pa=0x%llx; userd rc=%d put=%u get=%u)\n", rank, ring_va, ring_pa, push_va, len, push_pa, prc, @@ -9251,7 +23287,7 @@ static void svc_gr_content_discover(struct nv_vgpu_device *vgpu) if (!buf) return; - for (gpa = SEMA_PA_MAX - SVC_SCAN_CHUNK;; gpa -= SVC_SCAN_CHUNK) { + for (gpa = svc_pa_max() - SVC_SCAN_CHUNK;; gpa -= SVC_SCAN_CHUNK) { if (svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) goto next_chunk; for (o = 0; o + 24 <= SVC_SCAN_CHUNK; o += 4) { @@ -9339,6 +23375,303 @@ bool gsp_sema_counter_derive = true; unsigned gsp_sema_gr_grace_ms = 8000; /* + * GRFIX FIX1b (r213): absolute per-LAUNCH bound (ms) for the data-stalled + * hold of svc_gr_data_fill. The identity is the launch push VA, not the + * GPFIFO slot: pin-batch rewinds/replays re-decode the same launch at + * different slot indices (r213: entry 107 -> 107 -> 99 -> ... on one + * channel), so an idx-keyed deadline re-armed on every pass and never + * fired - 410+ holds over 216 s, gr_hold_exp=0, client killed at its 240 s + * timeout. The deadline is fixed at the first hold of a launch; when it + * passes, svc_gr_request proceeds with the launch exactly as the + * pre-8b058884 path did (forward best-effort - never block the guest). + */ +unsigned gsp_phys_gr_hold_max_ms = 3000; + +/* + * GRFIX FIX1e (r215): master gate for the GR fill hold on dropped operand + * pages. Default 0 = LOG-ONLY: svc_gr_data_fill never holds on dropped + * pages - the drops are logged (GRFILL-DROP) and the launch takes the + * historical forward path immediately (exactly the pre-8b058884 behaviour); + * the GRFILL-HOLD/GRFILL-EXPIRED machinery and the per-launch absolute + * bound are skipped entirely (the pump's own m150/m156 hold and its + * absolute bound are NOT - they run regardless of this gate; the record + * fields are now in the snapshot write-back list, see the GRFIX NOTE). + * When set to 1 the page-count hold + absolute bound apply on the fill + * path too. + */ +unsigned gsp_phys_gr_fill_hold; /* default 0: log-only, never hold */ + +/* + * r346: trace GMMU walks for unresolved high-VA operands (the arg-miss + * diagnostic re-walks each published vaspace for a failing base). When + * nonzero, those walks print the per-level entries (L0..L2, PD0 halves) + * so a failing VA shows exactly where the walk dies. + */ +unsigned gsp_sema_walk_trace; + +/* + * F1-b: GR release-word authorship gate (default 1). While a forwarded GR + * launch is in flight, the RAM-content sweep must not author the very QMD + * completion release the backend poll is waiting for: a sweep write could + * satisfy the poll before the engine retires the QMD (false completion, + * silent corruption). With the gate on, such a write is deferred - the + * release stays in RAM and later sweep passes re-see it; when the hardware + * copy resolves, the hold/fallback path applies it. 0 restores the + * unconditional sweep apply for an A/B. + */ +bool gsp_sema_gr_swrel_gate = true; + +/* + * m164 CLASS-2: hold window (ms) for a readable GR launch push whose decode + * saw QMD words but built no launch candidate for its channel (a replay- + * suppressed capture, a partial/split QMD, or a push page the pin stream has + * not reached yet). The entry is left unconsumed for this window so the next + * pass re-decodes it with a forced fresh capture (gr_rehold_idx); on expiry + * the historical consume applies. Without this, such entries were consumed + * silently and the client's launch never ran (streams2 second vadd: 56 + * consumes / 2 captures on one channel; the client read zeros). + */ +unsigned gsp_sema_gr_nocap_hold_ms = 6000; + +/* + * m165 (D-F2a): when set (default), a pool release is only deferred by the + * post-registration hold while NO host-verified injected pin exists for its + * VA; a fresh pin is this session's ground truth and lets the apply land at + * pin time (~0.5-2 s) instead of registration+5 s. Set 0 to restore the + * legacy blanket 5 s hold. + */ +bool gsp_sema_pool_hold_fresh = 1; + +/* + * m169 (AGENT-F R1): hold device-destination copies whose only + * translation is a page-directory walk or RAM scan (no pin). Default on; + * 0 restores the historical immediate-write behavior. + */ +bool gsp_sema_copy_dst_pin_guard = 1; +/* + * r336/r34x wait gate: honor the C56F host-semaphore acquire ops captured + * for a ring entry before running its work. DEFAULT ON since r408: the + * r408 streams2d forensics reproduced the class the gate was built for + * with the gate off - the c2 readback (C56F ACQ_CIRC_GEQ pay=4 on the + * scale channel's progress counter 0x20400ffd0) ran at 106617.279 with + * cur=2, i.e. BEFORE the c2-add launch was even submitted (106617.411), + * read the not-yet-produced buffer (zeros) and the client failed with + * 'c2 all zeros'. With the gate on the same readback holds (sema-wait: + * hold ... cur=2 waited_ms=353) until the counter reaches 4 (pass ... + * cur=4 at 106812.259) and the ordering is correct. The semantics are + * the class contract's: op 0 = equal, 2 = strict GEQ, 3 = circular GEQ, + * 4 = AND, 5 = NOR; the wait set is captured PER ENTRY (the old + * single-last-wait global leaked a later entry's acquire into an earlier + * entry - the r340 over-hold); the entry's own release counts only when + * its method order precedes the acquire; pending releases our machinery + * is merely late about are drained first; and every hold is bounded at + * gsp_sema_wait_hold_ms (default 4000 ms, r401) then fails open loudly + * ('give-up') so a value that can only be installed behind an upstream + * opdef hold cannot park a channel. Diagnostics: 'sema-wait: + * pass/hold/give-up' rows carry op, payload, observed value and decision. + * sema_wait_gate=0 restores the pre-r336 pass-through for an A/B. + */ +bool gsp_sema_wait_gate = 1; + +/* + * m169b: the guard above must never deadlock the boot's first client. The + * first device-destination H2D of a client runs before any launch, and the + * pins for device VAs are (re)discovered by the launch/fill paths - so at + * that instant NO pin can exist. r173: mail's H2D was held forever + * (opdef_held ~6000 passes, six expiry cycles, the entry re-held on every + * re-decode because the retry keeps resolving prov>=3). Bound the wait per + * destination page: after this many ms the guard releases and the copy runs + * with the walk translation (the historical behavior; the walk of a + * just-built client table is fresh), logged loudly. + */ +unsigned gsp_sema_copy_dst_guard_ms = 3000; + +/* + * r609: the m169 first-touch window is per PAGE, but the stale-table risk it + * covers is per CLIENT TABLE BUILD. The driver's H2D relocation burst on the + * eng==0 copy channel writes N distinct first-write pages back to back, and + * because the pump holds the FIRST held entry and the rest of the chain + * decodes only after it drains, each page paid its own full window, + * serialized (r606 red draw: `sema-pend: hold h=0x5c000059 idx=90` held + * 362167.7->362170.8, idx=92 held 362177.7->362182.9; the b H2D entry ran + * 362182.882, 326 ms AFTER the launch forward -> `VADD bad=32735 + * csum=8139264`, c = a + 0). The guard's bet - a fresh pin lands inside the + * window - is a property of the client's table state, not of one page: once + * ONE device-dst page of a channel has waited the full window with NO pin + * arriving (every observed release logs "without a pin"), the same bet for + * the next page of the same burst is already lost. Age the channel for + * gsp_sema_copy_dst_age_ms after such a release; further first-write + * device-dst pages of that channel inside the window skip the hold and run + * with the walk translation - the exact translation the expired guard would + * have used anyway. The first page of every burst still pays the full + * window, so the anti-stale bet is unchanged; only the N x window + * serialization is removed (r606 class: ~20 s of chain -> one window). + * Keyed by h_object like every channel table; the session flush clears it, + * so a recycled handle can never inherit the previous client's ageing + * (the r604 lifetime lesson). 0 disables (= per-page windows, pre-r609). + */ +unsigned gsp_sema_copy_dst_age_ms = 3000; +#define SVC_DST_AGE_SLOTS 16 +static u32 svc_dst_age_ch[SVC_DST_AGE_SLOTS]; +static unsigned long svc_dst_age_j[SVC_DST_AGE_SLOTS]; +static u32 svc_dst_age_pos; +static u32 svc_dst_age_log; + +/* r609: the current decode channel's guard window expired without a pin. */ +static void svc_dst_guard_age(void) +{ + u32 i; + + if (!gsp_sema_copy_dst_age_ms) + return; + for (i = 0; i < SVC_DST_AGE_SLOTS; i++) + if (svc_dst_age_ch[i] == svc.cur_ch) { + svc_dst_age_j[i] = jiffies; + return; + } + i = svc_dst_age_pos++ % SVC_DST_AGE_SLOTS; + svc_dst_age_ch[i] = svc.cur_ch; + svc_dst_age_j[i] = jiffies; +} + +/* r609: is the current decode channel inside its post-expiry ageing window? */ +static bool svc_dst_guard_aged(void) +{ + unsigned long now = jiffies; + u32 i; + + if (!gsp_sema_copy_dst_age_ms || !svc.cur_ch) + return false; + for (i = 0; i < SVC_DST_AGE_SLOTS; i++) + if (svc_dst_age_ch[i] == svc.cur_ch && + time_before(now, svc_dst_age_j[i] + + msecs_to_jiffies(gsp_sema_copy_dst_age_ms))) + return true; + return false; +} + +/* + * r609: decide whether a first-write device-dst page pays the m169 window. + * An aged channel skips it (the burst's first page already paid and no pin + * arrived); otherwise the normal per-page window runs. + */ +static bool svc_dst_guard_should_hold(u64 va) +{ + if (svc_dst_guard_aged()) { + if (svc_dst_age_log < 16) { + svc_dst_age_log++; + pr_info("sema-svc: device-dst guard age-skipped dst=0x%llx h=0x%x (same-channel burst after an un-pinned release; walk translation)\n", + va, svc.cur_ch); + } + return false; + } + return svc_dst_guard_hold(va); +} + +/* + * r231 READ-ORDER guard (default 0: witness only). A device-src copy whose + * source VA has a PENDING device-dst write hold (the m169 guard window is + * still open) reads the PRE-write content: the writer's bytes land only + * when the guard resolves, while the reader can execute immediately. The + * ctaid_probe RAWA round trip is the reproducer - upload a fresh 64 KB + * buffer, read it straight back with no kernel - and the readback came + * back as foreign deterministic content. With this on, such a read is + * deferred through the existing bounded-hold machinery (-ENOENT) so it + * re-decodes after the write resolves; with it off the condition is + * logged (READ-ORDER-WIT) so the next capture proves the coincidence. + * Opt-in until an A/B proves no green regression. + */ +bool gsp_sema_read_order_guard; + +/* + * COPYWIT fail4: per-boot budget for the sema-bigcopy resolution witnesses + * (svc_copy's large-copy proof: both operands' VA -> pa/aperture/provenance + * plus the first 16 bytes each side). The shipped default kept the boot's + * head only; a failing-client run happens minutes later, so the budget is a + * module parameter now - raise it for a discrimination cycle and correlate + * each D2H's resolved source PA with the booter's OUT-PAGE-WIT copy-out PA + * for the same device VA. 0 disables the lines. + */ +unsigned gsp_sema_bigcopy_log_max = 24; +#define SVC_DST_GUARD_SLOTS 16 +struct svc_dst_guard { + u64 va_page; + unsigned long first; +}; +static struct svc_dst_guard svc_dst_guard[SVC_DST_GUARD_SLOTS]; +static u32 svc_dst_guard_next; + +/* true = the guard may hold this destination; false = release it. */ +static bool svc_dst_guard_hold(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i, slot = SVC_DST_GUARD_SLOTS; + + for (i = 0; i < SVC_DST_GUARD_SLOTS; i++) { + if (svc_dst_guard[i].va_page == page) { + if (time_before(now, svc_dst_guard[i].first + + msecs_to_jiffies( + gsp_sema_copy_dst_guard_ms))) + return true; + if (svc_dst_guard[i].first) { + svc_dst_guard[i].first = 0; + svc.copy_dst_relaxed++; + svc_dst_guard_age(); /* r609: burst ageing */ + pr_info("sema-svc: device-dst guard released without a pin va=0x%llx (using the walk translation)\n", + page); + } + return false; + } + if (!svc_dst_guard[i].va_page) + slot = i; + } + if (slot == SVC_DST_GUARD_SLOTS) + slot = svc_dst_guard_next++ % SVC_DST_GUARD_SLOTS; + svc_dst_guard[slot].va_page = page; + svc_dst_guard[slot].first = now; + return true; +} + +/* + * r609: clear the m169 page windows and the r609 ageing ring with the + * session. Both are keyed by VA page / h_object alone, and both would + * otherwise pre-age the next client's first touch (a page slot still inside + * its window makes the new client's first device-dst write skip the + * anti-stale wait; an ageing stamp makes the new channel's whole first + * burst skip it). The stale-table risk this guard covers is precisely the + * new-session condition, so the windows must start fresh. + */ +static void svc_dst_guard_reset(void) +{ + memset(svc_dst_guard, 0, sizeof(svc_dst_guard)); + svc_dst_guard_next = 0; + memset(svc_dst_age_ch, 0, sizeof(svc_dst_age_ch)); + memset(svc_dst_age_j, 0, sizeof(svc_dst_age_j)); + svc_dst_age_pos = 0; +} + +/* + * r231 READ-ORDER guard: true while a held device-dst write for this VA + * page is still inside its bounded window (the writer's bytes have not + * landed yet). See the gsp_sema_read_order_guard declaration. + */ +static bool svc_dst_guard_pending(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i; + + for (i = 0; i < SVC_DST_GUARD_SLOTS; i++) + if (svc_dst_guard[i].va_page == page && + svc_dst_guard[i].first && + time_before(now, svc_dst_guard[i].first + + msecs_to_jiffies(gsp_sema_copy_dst_guard_ms))) + return true; + return false; +} + +/* * CE release fallback: when a push carrying a tracking-semaphore release * cannot be read (and the content rescue failed), advance the channel's * tracking semaphore to the expected payload directly. The push's data @@ -9369,6 +23702,24 @@ bool gsp_sema_rel_recover; bool gsp_sema_dbg; /* + * r365: execute fill-shaped CE transfers whose destination is a VIDMEM + * WINDOW VA (>= SVC_VIDMEM_VA_LO, below the 1 TiB VA floor). The guest's + * page-table writes arrive exactly this way (dma=0x586, REMAP+PITCH, a + * count=1 SET_REMAP_CONST_A, line=4, out=0x2000c0xxx) and previously + * decoded no op, so the shadow never saw them and the first client of a + * fresh boot dropped operand pages (GRFILL-DROP -> launch held/fail-open + * with 8 dropped pages -> wrong results or a client wedge). + * + * The op is built with the VA unchanged (phys=0): svc_fill's virtual path + * resolves it through the guest's own published page tables + * (svc_op_translate) and only falls back to the shadow for floor-range + * VAs - never through the window convention (r362 proved a forced + * `addr - SVC_VIDMEM_VA_BASE` writes the wrong page and corrupts results: + * reduce sum=0 + mail hang). Default off for A/B. + */ +bool gsp_sema_winfill; + +/* * Skip executing a push's collected work (CE memsets/copies into the vidmem * shadow) while still applying its semaphore release. The client only ever * observes the release; the shadow writes exist so the emulation can walk @@ -9425,8 +23776,1421 @@ unsigned gsp_sema_ce_grace_ms = 25000; */ unsigned gsp_sema_push_hold_ms = 2500; +/* + * SESSION 153-CONTINUED-54: GR setup-push hold. + * + * The client's 12288 B C597 MME-setup push ("LOAD_MME_* program upload") + * reads unreadable/all-zero on the first-read lottery boots and used to be + * consumed ("skipping stale entry ... after 1 failed rescues") before the + * demand channel's injected pin could land, so the MME prelude capture + * missed the macro program for the whole boot. When such a push's first + * read fails, demand every 4K VA page of the push through the stalled-VA + * demand channel (pin_gr_stream harvests the rows and the guest resolves + + * injects the pages) and hold the entry pending PAST the rescue-consume + * until the pin lands (the cheap-wait guard re-reads when it does) or this + * window expires. Bounded by an absolute per-(va,slot) deadline; 0 + * disables the hold (the demand rows are still emitted). + * + * r1083-r1087: DEFAULT 0 (off). A 20 s hold blocks the whole ring behind + * the held entry, and on the three hold-active boots (r1083/r1084/r1085) + * the GL client derailed into the black-dud/churn class, while the r1087 + * boot (hold=0) read the beef windows fine (the wall moved back to the + * incomplete MME prelude + dispatch stall). The demand channel also cannot + * yet resolve the GL pushbuffer VA families (0x4000000+ / 0x42xxxxx), so + * the pin the hold waits for never lands and the hold only delays. Re-enable + * (e.g. sema_gr_setup_hold_ms=20000) once the guest content resolver covers + * those VA families. + */ +unsigned gsp_sema_gr_setup_hold_ms; + static unsigned long svc_gr_ring_tried; /* ranks that got a diagnostic line */ +/* + * SESSION 153-CONTINUED-65: per-boot map of every DISTINCT beacon + * (0xbeef0xxx) GPFIFO entry the pump walks - (idx, va, len), first + * sighting order, dedup table of 512 triples (the ring re-walks entries + * every pass, so without dedup this floods). The r1124 setup-hold bound + * itself to entries 8 (0x4001004, 60216 B) and 15 (0x3200000, 153600 B); + * the client's 0x114 instruction-RAM program lives in the 0x3200000 + * family window, so binding the hold/demand/capture class to the ENTRY + * (not a size heuristic) starts from this map. Service-thread only. + */ +static void svc_beef_entry_note(u32 h, u32 idx, u64 va, u32 len) +{ + static u64 sva[512]; + static u32 sidx[512]; + static u32 slen[512]; + static u32 n; + u32 i; + + for (i = 0; i < n; i++) + if (sva[i] == va && sidx[i] == idx && slen[i] == len) + return; + if (n >= 512u) + return; + sva[n] = va; + sidx[n] = idx; + slen[n] = len; + n++; + pr_info("sema-chan: beef entry h=0x%x idx=%u va=0x%llx len=%u\n", + h, idx, va, len); +} + +/* + * Is this a GR push whose readability the MME-setup capture depends on? + * The client's C597 setup push is 12288 B; any GR push of that size is a + * candidate. A smaller push qualifies only when it is READABLE and its + * method runs hit the MME setup families (the unreadable case has no bytes + * to scan, hence the length rule alone). + * + * SESSION 153-CONTINUED-65: the MME macro-upload family (0x3200000..) + * is setup-class by VA alone. The r1124 entry 15 (va=0x3200000, + * len=153600) carries the client's 0x114 instruction-RAM program - the + * guest RAM image of its window measures code=712 words with an exact-end + * stream walk - and the ubounded-tail demand must follow the ENTRY, not a + * size heuristic: the class match is what keeps the hold, the 4K-page + * demand and the strict pin anchored on the entry that actually feeds + * svc_mme_prelude_capture the 0x114 runs. + */ +static bool svc_gr_setup_push(u64 va, u32 len, const u8 *buf) +{ + u32 off = 0; + + if (va >= 0x3200000u && va < 0x3300000u) + return true; + if (len >= 12288u) + return true; + if (!buf) + return false; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt && + svc_mme_setup_run_addr(addr)) + return true; + off += ((op == 1u || op == 3u || op == 5u) && cnt) ? + (1u + cnt) * 4u : 4u; + } + return false; +} + +/* + * Demand every 4K page of the push and arm the per-(va,slot) hold. The + * demand is the existing stalled-VA channel: svc_log_stalled_va rows are + * deduped and harvested by tools/guest-diag/pin_gr_stream.py, which asks + * the guest scanner to resolve the VA set and inject the pages. Only one + * arm line per (va,slot) is logged. The loop is bounded by `page_cap` (the + * 12288 B setup push needs 3; the cap keeps a bulk push from eating the + * 256-entry stalled-VA budget). + * + * SESSION 153-CONTINUED-55: the cap was 0x8000 (8 pages). The GL client's + * observed setup push is 60216 B (16 pages); pages 1..8 were demanded but + * pages 9..15 never were, so the host's per-chunk read of the entry had no + * source for the tail chunks and the whole read failed (unreadable=1, the + * hold re-armed forever). 0x10000 covered that push within the stalled-VA + * budget. + * + * SESSION 153-CONTINUED-65: the macro-upload carrier (r1124 entry 15, + * va=0x3200000) is 153600 B = 38 pages; 0x10000 demanded only 16 of them, + * so even a resolved window could never be read whole. Raise to 0x40000 + * (256 pages) - bounded by the 512-slot demand ring and the 256-entry + * stalled-VA table, both of which keep their own eviction. + */ +static void svc_gr_setup_hold_arm(struct svc_channel *ch, u64 va, u32 idx, + u32 len) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 poff; + + if (!va) + return; + /* + * SESSION 153-CONTINUED-76: the page demand is DECOUPLED from the + * hold. The r1157 live A/B proved the 20 s hold derails the client + * (the ring stalls behind the unreadable setup entry and the client's + * ~10 s fence timeout churns the channel), while with the demand also + * disabled (early return) the unreadable setup push's pages are never + * demanded, never pinned, and the 0x114/0x11c/0x120 macro-program + * runs stay zeros. Keep emitting the demand rows for the entry with + * the hold off; one emission per (va,slot). + */ + if (ch->gr_setup_hold_va != va || ch->gr_setup_hold_idx != idx) { + ch->gr_setup_hold_va = va; + ch->gr_setup_hold_idx = idx; + if (gsp_sema_gr_setup_hold_ms) + ch->gr_setup_hold_until = jiffies + + msecs_to_jiffies(gsp_sema_gr_setup_hold_ms); + else + ch->gr_setup_hold_until = 0; + pr_info("sema-chan: GR setup hold h=0x%x entry=%u va=0x%llx len=%u armed (demanding 4K pages; hold=%u ms; pending until the pin lands)\n", + ch->h_object, idx, va, len, + gsp_sema_gr_setup_hold_ms); + } + for (poff = 0; poff < len && poff < 0x40000u; poff += SEMA_PAGE_SIZE) { + svc_log_stalled_va(page + poff); + /* + * SESSION 153-CONTINUED-56: also publish the page in the + * debugfs demand set the pin tool reads directly, so the + * demand no longer depends on the dmesg row surviving the + * ring rotation under vgsp-sema spam. + * + * SESSION 153-CONTINUED-64: the entry's own page (poff == 0) + * additionally carries (off, len) - the exact [va, va+len) + * window the host read will parse - so the guest resolver + * can validate a candidate page range with the host's own + * full method-stream parse before injecting the pin. + */ + svc_setup_demand_record_ex(page + poff, + poff == 0 ? + (u32)(va & (SEMA_PAGE_SIZE - 1)) : 0u, + poff == 0 ? len : 0u); + } +} + +static bool svc_gr_setup_hold_active(const struct svc_channel *ch, u64 va, + u32 idx) +{ + return ch->gr_setup_hold_until && + ch->gr_setup_hold_va == va && + ch->gr_setup_hold_idx == idx && + time_before(jiffies, ch->gr_setup_hold_until); +} + +static void svc_gr_setup_hold_clear(struct svc_channel *ch) +{ + ch->gr_setup_hold_until = 0; + ch->gr_setup_hold_va = 0; + ch->gr_setup_hold_idx = ~0u; +} + +static bool svc_gr_ram_ring_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out) +{ + static u8 *buf; + u64 top = gsp_sema_gr_scan_top; + u64 start = ch->gr_ram_cursor; + u64 end; + u64 gpa; + /* + * r908: the GPFIFO entries start AT the registered ring VA, so their + * in-page offset is the ring VA's own offset (X server ring + * 0x120090000 -> offset 0; RC watchdog 0x120000400 -> 0x400; the GL + * client 0x1203cfff0 -> 0xff0, measured). The scan used to test only + * page offset 0, so any ring whose VA is not page-aligned could never + * match no matter how good the entry contents were. + */ + u32 eoff = (u32)(ch->gpfifo_va & (SEMA_PAGE_SIZE - 1)); + int i; + unsigned long flags; + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PAGE_SIZE) + return false; + if (start >= top) + start = 0; + end = start + SVC_GR_RAM_SLICE; + if (end > top) + end = top; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + for (gpa = start; gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + eoff + 2 * SVC_GPFIFO_ENTRY_SIZE <= chunk; + o += SEMA_PAGE_SIZE) { + u32 e0, e1, f0, f1; + u64 va0, va1; + u32 len0, len1; + + memcpy(&e0, buf + o + eoff, 4); + memcpy(&e1, buf + o + eoff + 4, 4); + va0 = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len0 = ((e1 >> 10) & 0x1fffff) << 2; + /* + * entry0: a pushbuffer VA in the ring's allocation + * window - just below the ring for the long-standing + * convention (X ring: 0x120010000 for ring + * 0x120090000), or just above it (r908: the GL + * client's PROMOTE_CTX ordering shows the main + * pushbuffer above the ring on this boot; accept + * both sides, still bounded to +/-1 MB of the ring). + */ + if (va0 + SVC_GR_RAM_WIN / 2 < ch->gpfifo_va || + va0 > ch->gpfifo_va + SVC_GR_RAM_WIN / 2) + continue; + if (va0 < SEMA_VA_LO || va0 >= SEMA_VA_HI) + continue; + if (len0 < 4 || len0 > 0x100000) + continue; + memcpy(&f0, buf + o + eoff + 8, 4); + memcpy(&f1, buf + o + eoff + 12, 4); + va1 = ((u64)(f1 & 0xff) << 32) | + (u64)(f0 & 0xfffffffc); + len1 = ((f1 >> 10) & 0x1fffff) << 2; + /* + * entry1: the progress tracker above entry0 - or the + * SAME push submitted again (r908 measured on the GL + * client's ring: {push=0x1203c0000,len=64} then + * {push=0x1203c0000,len=32}). Never below entry0; + * still bounded by the window above the ring. + */ + if (va1 < va0 || + va1 > ch->gpfifo_va + SVC_GR_RAM_WIN / 2) + continue; + if (len1 < 4 || len1 > 0x100000) + continue; + /* + * r979: kick-family ring discrimination. Shape alone + * admitted a foreign table page ahead of the real ring + * on r978 (pa=0x104fa9000, entries naming a bogus + * 0x12048b824/0x40000 push) while the true ring's + * entry pair names the client's per-frame record VA + * (SVC_GL_KICK_PUSH_VA, the r908-documented GL + * pushbuffer, constant across r908-r978). A + * kick-family candidate must name the record; the + * lookalike class then never latches. Non-kick + * channels keep the shape gate unchanged. + */ + if (svc_kick_family(ch) && + va0 != SVC_GL_KICK_PUSH_VA && + va1 != SVC_GL_KICK_PUSH_VA) + continue; + *pa_out = gpa + o; + return true; + } + } + ch->gr_ram_cursor = (end >= top) ? 0 : end; + ch->gr_ram_passes++; + if (ch->gr_ram_cursor == 0 && !ch->gr_ram_logged) { + ch->gr_ram_logged = true; + pr_info("sema-chan: GR ring h=0x%x rank=%u RAM content scan found no candidate in %u slice(s) (va=0x%llx)\n", + ch->h_object, ch->gr_index, ch->gr_ram_passes, + ch->gpfifo_va); + } + /* + * Persist the cursor immediately: the pump's stalled-userd path + * `continue`s past the end-of-pass write-back, so without this the + * scan would restart from slice 0 every pass and never reach the + * ring's page. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[i].gr_ram_passes = ch->gr_ram_passes; + svc.chan[i].gr_ram_logged = ch->gr_ram_logged; + /* SESSION 149 R15: the M1.1b keep-alive budget state. */ + svc.chan[i].gr_ram_kick = ch->gr_ram_kick; + svc.chan[i].gr_ram_refreshes = ch->gr_ram_refreshes; + svc.chan[i].gr_ram_refresh_at = ch->gr_ram_refresh_at; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +/* + * SESSION 153-CONTINUED-19 (kick-family USERD acceptance). + * + * The glxgears/X GLX client's kick channel (h 0xbfef00xx, GR engine, ring + * found by the guest-RAM content scan) posts a per-frame completion kick + * the client's flush loop paces on. Its USERD is a sysmem control block + * the driver allocates far from the ring - r971 measured the ring at pa + * 0x12f9ed000 and the live USERD at pa 0x1052ae000 (~681 MB away; the + * same nomination was +0.6..+2.3 GB across r966-r971) - while the general + * RAM USERD scan's allocation-locality gate only admits candidates within + * +/-64 MB of the ring (SESSION 153-CONT-7, built for the r936-r939 + * nine-decoy class). That gate rejected the channel's real USERD on every + * boot since the gate landed, the scan retired ("RAM scan found no + * candidate in 20 slice(s)"), the pump never iterated the kick entries + * and the kick-notify lane never ran. For this family the ring-entry + * match (every PUT-claimed ring entry must decode to a pushbuffer VA + * inside the ring's own allocation window) is the ownership proof; the + * locality distance is not - so the gate is waived for these channels and + * the distance is logged instead. The pump's speculative-latch + * confirmation (a +1/+2 PUT step, svc_pump_channels) remains the final + * judge, and the rejected-latch memory prevents a frozen block from ever + * pinning the scan. + */ +static bool svc_kick_family(const struct svc_channel *ch) +{ + return ch && !ch->engine_type && + ch->h_object >= 0xbfef0000u && ch->h_object <= 0xbfef00ffu; +} + +/* + * Re-validate the kick channel's ring latch each pass (bounded to once a + * second per channel). The RAM content scan latches a ring page once; if + * the driver later zeroes or reuses that page (the r966-r971 class the + * design brief describes: the USERD scan then validates every candidate + * against a dead ring and adopts none), the latch is dropped here so the + * content scan re-runs and re-latches the channel's current ring page. + * The stale entry pair is logged on re-arm so the transition is visible + * in the record. A live ring (a first entry that still decodes to a + * window-bounded pushbuffer VA) is a cheap no-op. + */ +static void svc_kick_ring_revalidate(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0; + unsigned long flags; + int i; + + if (!svc_kick_family(ch) || !ch->gr_ram_ring || ch->userd_ok) + return; + if (!time_after(jiffies, ch->gr_kick_ring_reval_at)) + return; + ch->gr_kick_ring_reval_at = jiffies + msecs_to_jiffies(1000); + if (!svc_read_va(vgpu, ch->gpfifo_va, e, sizeof(e))) { + u64 eva; + u32 elen; + + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + elen = ((e1 >> 10) & 0x1fffff) << 2; + if (eva && elen >= 4 && elen <= 0x100000) + return; /* ring page live; nothing to do */ + } + /* + * The latched page no longer decodes entries: drop the one-shot + * latch state so the RAM ring content scan re-runs. The pin is left + * in place - a stale pin only shadows VA reads, while the content + * scan validates against raw RAM and re-pins on a hit. + */ + clear_bit(ch->gr_index, &svc_gr_ring_done); + clear_bit(ch->gr_index, &svc_pb_gr_done); + ch->gr_ram_cursor = 0; + ch->gr_ram_passes = 0; + ch->gr_ram_logged = false; + ch->gr_ram_ring = false; + pr_info("sema-chan: kick ring re-arm h=0x%x va=0x%llx (latched entry e0=0x%x e1=0x%x no longer decodes; content scan re-runs)\n", + ch->h_object, ch->gpfifo_va, e0, e1); + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[i].gr_ram_passes = ch->gr_ram_passes; + svc.chan[i].gr_ram_logged = ch->gr_ram_logged; + svc.chan[i].gr_ram_ring = ch->gr_ram_ring; + svc.chan[i].gr_kick_ring_reval_at = + ch->gr_kick_ring_reval_at; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); +} + +/* + * SESSION 153-CONTINUED-75: GR ring stale-pin self-heal. + * + * The GR ring pin can outlive the page it names. The client churns its + * channels every ~12 s (fence timeouts), frees the ring allocation, and the + * driver's allocator reuses the PA. Live evidence (r1156): the beef ring + * pin 0x238041000 later held glxgears vertex/matrix floats and every + * "entry" the pump decoded from it was float data - the phantom held push + * va=0x3e4ccccc len=123916 is exactly the float pair (0x3e4ccccd = 0.2f, + * 0x81e40e00). A stale pin shadows the client's own page tables in + * svc_lookup, so the channel runs on phantom entries and never retires. + * + * Once per second, when the current translation's entry0 no longer decodes + * as a ring entry (dead, zeroed or float junk), ask the client's own VAS + * walk for the ring VA. If the walk resolves a DIFFERENT page whose entry0 + * DOES decode through the same window/length gate every adoption path uses, + * re-pin to that page and log it. With no walk, or a walk page that also + * fails the gate, nothing is touched: the heal can only replace a + * provably-dead translation with a provably-plausible one. + */ +static u32 svc_gr_heal_log; +static u64 svc_gr_heal_n; + +static bool svc_gr_entry_plausible(const struct svc_channel *ch, u32 e0, + u32 e1) +{ + u64 eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + u32 elen = ((e1 >> 10) & 0x1fffff) << 2; + + if ((eva & 0xfffff) || elen < 4 || elen > 0x10000) + return false; + if ((eva < SEMA_VA_LO || eva >= SEMA_VA_HI) && + !svc_gr_entry_win_ok(ch->gpfifo_va, eva)) + return false; + return true; +} + +static void svc_gr_ring_stale_heal(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0; + u64 wpa = 0, cpa = 0; + int wap = SVC_APERTURE_SYS, cap = SVC_APERTURE_SYS; + + if (!ch->gpfifo_va || ch->engine_type || ch->gr_index >= 32) + return; + if (!time_after(jiffies, svc_gr_heal_at[ch->gr_index])) + return; + svc_gr_heal_at[ch->gr_index] = jiffies + msecs_to_jiffies(1000); + + /* + * Entry0 through the CURRENT translation (the pin shadows the walk in + * svc_lookup). An unreadable read gets the walk check too - the pin + * may name freed memory the read path rejects. + */ + if (!svc_read_va(vgpu, ch->gpfifo_va, e, sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + if (svc_gr_entry_plausible(ch, e0, e1)) + return; /* live ring; nothing to do */ + } else { + e0 = 0; + e1 = 0; + } + + if (svc_pin_lookup_any(ch->gpfifo_va, &cpa, &cap)) + cpa = 0; + + /* The client's own page tables answer the ring VA right now? */ + if (svc_walk_vaspaces(vgpu, ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1), + &wpa, &wap)) + return; /* no walk: no verdict, leave the pin alone */ + if (cpa && (wpa & ~0xfffULL) == (cpa & ~0xfffULL)) + return; /* walk agrees with the implausible pin: dead either way */ + if (svc_read_ap(vgpu, wpa, wap, e, sizeof(e))) + return; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + if (!svc_gr_entry_plausible(ch, e0, e1)) + return; /* the walk page is not a live ring either */ + + svc_pin_store(ch->gpfifo_va, wpa, wap); + svc_lat_store(ch->gpfifo_va, wpa, wap); + svc_gr_heal_n++; + if (svc_gr_heal_log < 16) { + svc_gr_heal_log++; + pr_info("sema-chan: GR ring re-pin h=0x%x va=0x%llx -> pa=0x%llx ap=%d (stale translation 0x%llx no longer decodes; walk entry0 va=0x%llx len=%u)\n", + ch->h_object, ch->gpfifo_va, wpa, wap, cpa, + ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc), + ((e1 >> 10) & 0x1fffff) << 2); + } +} + +/* + * r912 (M-GL1.2): guest-RAM USERD shape scan for a GR channel whose ring + * was content-validated in guest RAM (gr_ram_ring; svc_gr_ram_ring_scan). + * Such a client's USERD is a system-memory object like its ring, so the + * FB-shadow shape scan inside svc_chan_userd can never reach it: the + * registered userd value is pool-relative, every FB candidate reads + * garbage, and the pump never iterates the GPFIFO entries ("unregistered + * (submissions=0)"). Scan guest RAM for the USERD control block shape + * (GET at +0x88, PUT at +0x8c - SVC_USERD_GP_*), with the same sliced, + * cursor-resumed budget as the ring scan (one SVC_GR_RAM_SLICE slice per + * pass, svc_gr_ram_pass_budget() slices before giving up). + * + * A candidate passes the shape and entry gates first (every entry its PUT + * claims must decode to a pushbuffer VA inside the ring's allocation + * window: entry k at ring VA + (k % entries)*8, within +/-SVC_GR_RAM_WIN/2 + * of the ring VA with a sane length), but admission is by LIVENESS, not by + * shape (SESSION 154): the GET/PUT gate alone admitted frozen or churning + * lookalike pages (r914: a low-RAM table whose +0x88/+0x8c words happened + * to read 12/8 and later 8/4 - adopted, the pump then read PUT=12 with + * entries=2, the modulo collapsed to 0 and NOTHING was consumed while the + * client retried for minutes). A passing candidate is WATCHED across + * passes; only a block whose PUT takes the client's own small step (+1 or + * +2) on two separate sightings is adopted as the live control block (the + * measured live block advances 1->2->3; the lookalikes sit still or jump + * by 4). The watch runs every call, so it keeps working past the slice + * budget (the gate in svc_chan_userd keeps calling while a watch is live). + */ +static bool svc_gr_ram_userd_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out, + u32 *put_out, u32 *get_out) +{ + static u8 *buf; + u64 top = gsp_sema_gr_scan_top; + u64 ring_pa = 0; + u64 start = ch->gr_ram_userd_cursor; + u64 end, gpa; + u32 entries = ch->gpfifo_entries; + u64 live_pa = 0; + u32 live_put = 0, live_get = 0, live_prev = 0; + int ring_ap = SVC_APERTURE_SYS; + bool kick; + u32 wi; + int i; + unsigned long flags; + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PAGE_SIZE || !entries) + return false; + if (start >= top) + start = 0; + end = start + SVC_GR_RAM_SLICE; + if (end > top) + end = top; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + kick = svc_kick_family(ch); + /* + * SESSION 153-CONTINUED-19: re-check the kick channel's ring latch + * before validating candidates against it (svc_kick_ring_revalidate + * re-latches a ring page the driver has zeroed since the latch; see + * its comment). + */ + svc_kick_ring_revalidate(vgpu, ch); + (void)svc_pin_lookup_any(ch->gpfifo_va, &ring_pa, &ring_ap); + /* + * SESSION 154 (M-GL1.2): liveness re-check of the watched candidates. + * Cheap (one 4-byte PUT read each, every pass) and the primary adoption + * site: a watched block whose PUT takes the client's own single +1 step + * is the live control block (the measured live block advances 1->2->3 + * in 2-8 s steps; the registration window is 25-45 s, so one step must + * suffice). A block whose PUT jumps by a larger amount or decreases is + * a churning lookalike (r914's pa 0x9e200 marched -4; the dense + * lookalike pools churn +14) and its streak resets; one that goes + * unreadable or zero is dropped. + */ + for (wi = 0; wi < ch->gr_ram_userd_watch_n; ) { + u64 wpa = ch->gr_ram_userd_watch[wi]; + u32 hp = 0, hg = 0, last; + + /* + * SESSION 153-CONT-7 (ownership by allocation locality): + * drop a watched candidate that sits far outside the + * channel's ring allocation. r939 measured: the ring at pa + * 0x1225ac000, the client's sysmem blocks (memfd) at pa + * 0x136b9c000..0x136eca000 (~21 MB away), while the + * latched pair at pa 0x106e8xxxx sat ~460 MB from the ring, + * mapped by NO process, and never advanced on any pass + * (two frozen snapshots that cost a 15 s reject/re-latch + * cycle all boot). A control block this far from its own + * ring is not this channel's USERD. SESSION 153-CONT-19: + * waived for the kick family - its real USERD is measured + * 0.6-2.3 GB from the ring, and the ring-entry match (not + * proximity) is the ownership proof there (see + * svc_kick_family). + */ + if (ring_pa && !kick && + (wpa < ring_pa - (SVC_GR_RAM_WIN << 5) || + wpa > ring_pa + (SVC_GR_RAM_WIN << 5))) { + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + continue; + } + if (svc_read(vgpu, wpa + SVC_USERD_GP_PUT, &hp, + sizeof(hp)) || !hp || hp > 0x100000) { + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + continue; + } + last = ch->gr_ram_userd_watch_put[wi]; + if (hp == last) { + /* + * SESSION 153-CONTINUED-20: stale-entry timeout. A + * kick-family watched block whose PUT has not moved + * for the bound is a frozen lookalike (the r972/r973 + * descriptor-pool class, put=7 forever) - drop it so + * the watch set keeps room for the advancing control + * block. Non-kick entries keep the old semantics + * (they are only added by the speculative latch's + * own re-arm today). + */ + if (kick && time_after(jiffies, + ch->gr_ram_userd_watch_at[wi] + + msecs_to_jiffies(SVC_KICK_WATCH_STALE_MS))) { + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_at[wi] = + ch->gr_ram_userd_watch_at[ch->gr_ram_userd_watch_n]; + continue; + } + wi++; + continue; + } + if (kick && hp < last) { + /* A reset is churn: drop (same class as the jump). */ + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_at[wi] = + ch->gr_ram_userd_watch_at[ch->gr_ram_userd_watch_n]; + continue; + } + if (!kick && hp != last + 1) { + /* + * SESSION 154: a jump or a reset is the churn + * signature (the r916-r919 lookalike pools move by + * +4/-4/+14 or collapse to zero between sightings) - + * drop the entry instead of keeping a dead watch slot + * that would evict newer, better candidates. + */ + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_at[wi] = + ch->gr_ram_userd_watch_at[ch->gr_ram_userd_watch_n]; + continue; + } + ch->gr_ram_userd_watch_put[wi] = hp; + ch->gr_ram_userd_watch_at[wi] = jiffies; + /* + * SESSION 153-CONTINUED-20: THE kick-family adoption rule - + * any positive PUT step across passes. The real control block + * is authored by the client's frame loop (measured 140/s), so + * between two sightings its PUT steps by several positions; + * the frozen descriptor-pool entries never step and time out + * above. Do NOT require the +1 step here: for this family a + * free-running put is the LIVE signature, a stall is the + * lookalike. + */ + if (kick) { + (void)svc_read(vgpu, wpa + SVC_USERD_GP_GET, &hg, + sizeof(hg)); + if (hg > hp) + continue; /* impossible; keep watching */ + live_pa = wpa; + live_put = hp; + live_get = hg; + live_prev = last; + ch->gr_ram_userd_watch_n = 0; + pr_info("sema-chan: GR USERD h=0x%x rank=%u kick-family ADOPT on PUT step (put %u -> %u across passes) -> pa=0x%llx put=%u get=%u\n", + ch->h_object, ch->gr_index, live_prev, + live_put, live_pa, live_put, live_get); + break; + } + /* + * SESSION 154: ONE observed +1 step adopts. The live block + * advances ~2-8 s per step (measured) while the client's + * channel registration window is ~25-45 s, so waiting for + * a second step can outlive the window; a churning lookalike + * moves in larger jumps (measured +4/-4/+14) and a random + * page cannot produce this exact one-step signature with + * the shape+entry gates already applied. + */ + ch->gr_ram_userd_watch_hits[wi]++; + (void)svc_read(vgpu, wpa + SVC_USERD_GP_GET, &hg, + sizeof(hg)); + if (!hg || hg > hp) { + ch->gr_ram_userd_watch_hits[wi] = 0; + wi++; + continue; + } + live_pa = wpa; + live_put = hp; + live_get = hg; + live_prev = last; + ch->gr_ram_userd_watch_n = 0; + pr_info("sema-chan: GR USERD h=0x%x rank=%u LIVE (put %u -> %u across passes) -> pa=0x%llx put=%u get=%u\n", + ch->h_object, ch->gr_index, live_prev, live_put, + live_pa, live_put, live_get); + break; + } + if (!live_pa) + for (gpa = start; gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + 0x90 <= chunk; o += 0x100) { + u32 hp, hg, k; + bool ok = true; + + memcpy(&hg, buf + o + SVC_USERD_GP_GET, 4); + memcpy(&hp, buf + o + SVC_USERD_GP_PUT, 4); + /* + * r914: the control block lives in a near-empty page + * (only the put/get pair and a few engine-owned words + * are written - the measured X-channel FB block and + * the GL client's live sysmem block both leave the + * leading 0x80 bytes zero). The shape gate alone is + * far too weak: a low-RAM page at pa 0x7c000 carrying + * coincidental small numbers at +0x88/+0x8c was + * latched on r913 (the pass then read its put + * forever and never advanced). SESSION 154: the put + * cap is 8 - the GL client's live block cycles 0..3 + * with entries=2 (measured), while the r914 false + * latches read 12/8 and 16/16 and every previously + * measured lookalike counted 16 or more. Also accept + * a free-running put (the old put <= entries bound + * would reject the live block). SESSION 153- + * CONTINUED-20: for the kick family the cap is + * LIFTED to the watch loop's sanity bound (0x100000) + * - its control block is authored by the client's + * frame loop and its PUT free-runs with the frame + * rate; the step-based watch adoption (not the cap) + * is the discriminator there. + */ + /* + * SESSION 153-CONTINUED-38: the cap is also lifted for + * LARGE rings (the kernel channels' 32768-entry + * GPFIFOs): their put is a real free-running count and + * can sit far above 8 while the client waits (measured + * put=16 on beef0100). The remaining gates (near-empty + * page, entry validation against the ring, and the + * pump's confirm/reject) carry the ownership proof. + */ + if (!hp || + hp > (kick || entries > 1024 ? 0x100000 : 8) || + (hg > hp && entries <= 1024)) + continue; + { + u32 z = 0, zi; + + for (zi = 0; zi < 0x80; zi++) + if (!buf[o + zi]) + z++; + if (z < 0x60) + continue; + } + /* + * Never adopt the ring page itself: the ring VA's + * own page holds GPFIFO entries and ring metadata, + * and a latch there would make the pump's GP_GET + * write-back corrupt it. + */ + if (ring_pa && (gpa + o) == + (ring_pa & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + /* + * SESSION 153-CONT-7: allocation-locality gate - the + * same 64 MB ring neighbourhood as the watch loop + * above. Excludes the r936-r939 nine-decoy pair + * (~460 MB from the ring) before it can occupy a + * watch slot and burn a reject/re-latch cycle. + * SESSION 153-CONT-19: waived for the kick family + * (h 0xbfef00xx) - its real USERD is measured far + * outside this window, so admission there is by the + * ring-entry match alone (the entry loop below) and + * the pump's +1/+2-step confirmation; the distance + * is logged at the latch. + */ + if (ring_pa && !kick && + (gpa + o < ring_pa - (SVC_GR_RAM_WIN << 5) || + gpa + o > ring_pa + (SVC_GR_RAM_WIN << 5))) + continue; + for (k = 0; k < hp && k < 8; k++) { + u8 e[8]; + u64 eva; + u32 elen, a0, a1; + /* + * r914: entry slot j of a free-running put is + * j % entries, not j. + */ + u32 slot = k % entries; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)slot * SVC_GPFIFO_ENTRY_SIZE, + e, sizeof(e))) { + ok = false; + break; + } + memcpy(&a0, e, 4); + memcpy(&a1, e + 4, 4); + eva = ((u64)(a1 & 0xff) << 32) | + (u64)(a0 & 0xfffffffc); + elen = ((a1 >> 10) & 0x1fffff) << 2; + if (!eva || + !svc_gr_entry_win_ok(ch->gpfifo_va, eva) || + elen < 4 || elen > 0x100000) { + ok = false; + break; + } + } + if (!ok) + continue; + /* + * SESSION 154: the live control block lives in a + * NEAR-EMPTY page (measured: put/get plus two more + * bytes non-zero - r914's live block; the X-channel FB + * block and the GL sysmem block both leave the leading + * 0x80 bytes zero). The r916-r919 candidate floods all + * came from dense pages (a dozen+ shape hits per page + * in tables and pools), so a page with more than 32 + * non-zero bytes is rejected before it can occupy a + * watch slot. + */ + { + u32 pstart = o & ~0xfffu; + u32 zc = 0, zi2; + + for (zi2 = 0; zi2 < SEMA_PAGE_SIZE; zi2++) + if (!buf[pstart + zi2]) + zc++; + if (zc < SEMA_PAGE_SIZE - 32) + continue; + } + /* + * SESSION 153-CONT-5: a previously rejected block at + * the same PUT is a frozen lookalike - do not spend a + * (re-armed) scan budget re-latching it, or the budget + * loops on the same stale window forever. A DIFFERENT + * PUT is content change (the block moved): adopt it + * again so a real control block that appears late is + * still picked up. + */ + if ((gpa + o) == ch->gr_ram_userd_reject_pa && + hp == ch->gr_ram_userd_reject_put) + continue; + /* + * SESSION 153-CONTINUED-20 (watch-adopt-on-PUT- + * step): a kick-family candidate is NOT adopted on + * sight. This family's RAM is littered with frozen + * descriptor-pool lookalikes whose +0x88/+0x8c + * words happen to form the put/get shape (r972/973: + * put=7 frozen, 65 pump rejections in one boot, + * the latch alternating between two pool entries). + * Add the candidate to the watch set (deduped, LRU + * replace when full) and KEEP scanning the slice; + * the watch loop at the top adopts the block only + * when its PUT takes a step across passes, and a + * stale-entry timeout (SVC_KICK_WATCH_STALE_MS) + * evicts frozen entries so the pool can never crowd + * out the real control block. The r971-r973 + * immediate latch is retired for this family. + */ + if (kick) { + u32 wn = ch->gr_ram_userd_watch_n; + bool known = false; + u32 wj; + + for (wj = 0; wj < wn; wj++) + if (ch->gr_ram_userd_watch[wj] == + gpa + o) { + known = true; + break; + } + if (!known) { + if (wn >= SVC_GR_USERD_WATCH) { + u32 old = 0, si; + + for (si = 1; si < wn; si++) + if (time_before( + ch->gr_ram_userd_watch_at[si], + ch->gr_ram_userd_watch_at[old])) + old = si; + wn = old; + } + ch->gr_ram_userd_watch[wn] = gpa + o; + ch->gr_ram_userd_watch_put[wn] = hp; + ch->gr_ram_userd_watch_hits[wn] = 0; + ch->gr_ram_userd_watch_at[wn] = jiffies; + if (wn == ch->gr_ram_userd_watch_n) + ch->gr_ram_userd_watch_n++; + if (ch->gr_ram_userd_watch_log < 48) { + ch->gr_ram_userd_watch_log++; + pr_info("sema-chan: GR USERD h=0x%x rank=%u kick-family watch[%u] pa=0x%llx put=%u get=%u (ring_pa=0x%llx delta=%lld MB; ring-entry validated; adopt on PUT step)\n", + ch->h_object, + ch->gr_index, wn, + gpa + o, hp, hg, + ring_pa, + (long long)(gpa + o - ring_pa) / + (1024 * 1024)); + } + } + continue; /* keep scanning; adopt is deferred */ + } + /* + * SESSION 154 (M-GL1.2): a candidate that passed the + * shape gate, the entry validation, the page-density + * gate and the put cap is latched SPECULATIVELY. The + * old code latched on shape alone and a frozen/ + * churning lookalike could hold the walk forever + * (r913: pa 0x7c000; r914: pa 0x9e200 at put=12); + * the strengthened gates reject those classes, and the + * pump now CONFIRMS the latch from its own per-pass + * read: a +1/+2 step confirms the block as the live + * one, a reset/jump or no movement within the bound + * unlatches it so the scan can adopt a better + * candidate (see svc_pump_channels). + */ + live_pa = gpa + o; + live_put = hp; + live_get = hg; + ch->gr_ram_userd_watch_n = 0; + pr_info("sema-chan: GR USERD h=0x%x rank=%u LIVE (put step across passes) -> pa=0x%llx put=%u get=%u\n", + ch->h_object, ch->gr_index, live_pa, + live_put, live_get); + break; + } + if (live_pa) + break; + } + ch->gr_ram_userd_cursor = (end >= top) ? 0 : end; + ch->gr_ram_userd_passes++; + /* + * SESSION 153-CONTINUED-20: kick-family sweep re-arm. When every + * watched candidate has gone stale (frozen pool) and the slice + * budget is spent, one bounded re-arm keeps the sweep alive so the + * real advancing block can still be found; capped like the reject + * counter so a truly dead channel stops sweeping in bounded time. + */ + if (kick && !live_pa && !ch->gr_ram_userd_watch_n && + ch->gr_ram_userd_passes >= svc_gr_ram_pass_budget() && + ch->gr_ram_userd_kick_sweeps < 64) { + ch->gr_ram_userd_kick_sweeps++; + ch->gr_ram_userd_passes = 0; + ch->gr_ram_userd_logged = false; + pr_info("sema-chan: GR USERD h=0x%x rank=%u kick-family sweep re-arm %u/64 (watch empty, budget spent; content scan re-runs)\n", + ch->h_object, ch->gr_index, + ch->gr_ram_userd_kick_sweeps); + } + if (ch->gr_ram_userd_cursor == 0 && !ch->gr_ram_userd_logged && + !ch->gr_ram_userd_watch_n) { + ch->gr_ram_userd_logged = true; + pr_info("sema-chan: GR USERD h=0x%x rank=%u RAM scan found no candidate in %u slice(s) (top 0x%llx)\n", + ch->h_object, ch->gr_index, ch->gr_ram_userd_passes, + top); + } + /* + * Persist the cursor immediately: the stalled-userd pump path + * `continue`s past the end-of-pass write-back (the same reason + * svc_gr_ram_ring_scan persists its cursor). + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + u32 wj; + + svc.chan[i].gr_ram_userd_cursor = + ch->gr_ram_userd_cursor; + svc.chan[i].gr_ram_userd_passes = + ch->gr_ram_userd_passes; + svc.chan[i].gr_ram_userd_logged = + ch->gr_ram_userd_logged; + svc.chan[i].gr_ram_userd_reject_pa = + ch->gr_ram_userd_reject_pa; + svc.chan[i].gr_ram_userd_reject_put = + ch->gr_ram_userd_reject_put; + svc.chan[i].gr_ram_userd_rejects = + ch->gr_ram_userd_rejects; + svc.chan[i].gr_ram_userd_watch_n = + ch->gr_ram_userd_watch_n; + svc.chan[i].gr_ram_userd_watch_log = + ch->gr_ram_userd_watch_log; + svc.chan[i].gr_ram_userd_kick_sweeps = + ch->gr_ram_userd_kick_sweeps; + for (wj = 0; wj < SVC_GR_USERD_WATCH; wj++) { + svc.chan[i].gr_ram_userd_watch[wj] = + ch->gr_ram_userd_watch[wj]; + svc.chan[i].gr_ram_userd_watch_put[wj] = + ch->gr_ram_userd_watch_put[wj]; + svc.chan[i].gr_ram_userd_watch_hits[wj] = + ch->gr_ram_userd_watch_hits[wj]; + svc.chan[i].gr_ram_userd_watch_at[wj] = + ch->gr_ram_userd_watch_at[wj]; + } + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!live_pa) + return false; + *pa_out = live_pa; + *put_out = live_put; + *get_out = live_get; + return true; +} + +/* + * SESSION 154 (M-GL1.2): content anchor for a GR client push whose + * translation went stale. r913: the entry's read resolved through a + * VALIDATED pin that kept answering for a page the guest had since reused + * (`first8=00000001` through src=2) while the trusted walk answered from a + * wrong VASPACE (`pa=0x5c0000 aperture=vidmem plaus=0`) and the client's + * real push bytes sat in RAM on a page no registered directory covered. + * The client's submit pushes carry two stable anchors - the first method + * pair (SET_SUBDEVICE_MASK 0x0001fff0, measured on every GR submit push of + * this driver) and the release-group signature pair measured deep in the + * same pushes - so sweep guest RAM for them (one SVC_GR_RAM_SLICE slice per + * call, cursor-resumed, a few sweeps per entry VA), validate the page by + * re-reading it (prefix at the entry's own page offset + full method-stream + * parse + the signature present in the same page - a stale fragment page + * carrying only the signature fails), then pin it with svc_pin_store so the + * pump's retry reads the REAL bytes like any validated page. + */ +static const u8 svc_gr_anchor_pre[8] = { + 0xf0, 0xff, 0x01, 0x00, 0x00, 0x00, 0x01, 0x20, +}; +static const u8 svc_gr_anchor_sig[16] = { + 0x43, 0x60, 0x01, 0x20, 0x00, 0x00, 0x00, 0x00, + 0x40, 0x60, 0x01, 0x20, 0x00, 0x00, 0x00, 0x00, +}; +#define SVC_GR_ANCHOR_SWEEPS 3 + +/* + * r977 (SESSION 153-CONTINUED-21): the kick family's per-frame submit + * record is a fixed method stream opening with the SET_SUBDEVICE_MASK + * pair 0x0001fff0 0x20010000 (LE bytes f0 ff 01 00 00 00 01 20; the + * bytes r975's anchor pin answered and r973 decoded to n=512 execs). + * Every boot r966-r976 read that VA through some untrusted translation + * and, because a plausible-looking garbage page (r976: first8=9864be60 + * ffffffff at pa=0x10353b000 via the src=3 window guess) passes the + * generic method-stream parse, the entry was consumed with the wrong + * bytes and the kick lane stayed silent for the whole session. For + * this family the canonical prefix IS the content identity: a read + * that lacks it came from a stale/foreign page no matter how + * plausible it parses, and must take the rescue chain (VAS walk -> + * content anchor -> pin). + */ +static bool svc_kick_canonical(const u8 *buf, size_t len) +{ + return len >= sizeof(svc_gr_anchor_pre) && + !memcmp(buf, svc_gr_anchor_pre, sizeof(svc_gr_anchor_pre)); +} + +static bool svc_gr_anchor_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 va, u32 len) +{ + static u8 *buf; + u64 top = gsp_sema_gr_scan_top; + u64 start = ch->gr_anchor_cursor; + u64 end, gpa, page_pa = 0; + u32 poff = (u32)(va & (SEMA_PAGE_SIZE - 1)); + int i; + unsigned long flags; + + if (len > SEMA_PAGE_SIZE) + return false; + if (ch->gr_anchor_va != va) { + ch->gr_anchor_va = va; + ch->gr_anchor_sweeps = 0; + ch->gr_anchor_cursor = 0; + start = 0; + } + if (ch->gr_anchor_sweeps >= SVC_GR_ANCHOR_SWEEPS) + return false; + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PAGE_SIZE || !len) + return false; + if (start >= top) + start = 0; + end = start + SVC_GR_RAM_SLICE; + if (end > top) + end = top; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + if (!ch->gr_anchor_logged) { + ch->gr_anchor_logged = true; + pr_info("sema-anchor: h=0x%x va=0x%llx len=%u content anchor scan started\n", + ch->h_object, va, len); + } + for (gpa = start; !page_pa && gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + 24 <= chunk; o += 4) { + /* + * (a) the push's own first method pair - it sits at + * the entry VA's page offset, so only a hit whose + * page offset matches the entry's is a candidate. + */ + if (!memcmp(buf + o, svc_gr_anchor_pre, + sizeof(svc_gr_anchor_pre)) && + ((gpa + o) & (SEMA_PAGE_SIZE - 1)) == poff) { + page_pa = gpa + o - poff; + break; + } + /* + * (b) the release-group signature measured deep in + * the client's submit pushes; the page is validated + * below before it is pinned. + */ + if (!memcmp(buf + o, svc_gr_anchor_sig, + sizeof(svc_gr_anchor_sig))) { + page_pa = (gpa + o) & + ~(u64)(SEMA_PAGE_SIZE - 1); + break; + } + } + } + ch->gr_anchor_cursor = (end >= top) ? 0 : end; + if (ch->gr_anchor_cursor == 0) + ch->gr_anchor_sweeps++; + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_anchor_cursor = ch->gr_anchor_cursor; + svc.chan[i].gr_anchor_va = ch->gr_anchor_va; + svc.chan[i].gr_anchor_sweeps = ch->gr_anchor_sweeps; + svc.chan[i].gr_anchor_logged = ch->gr_anchor_logged; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!page_pa) + return false; + /* + * Validate the candidate BEFORE pinning: the push's first method + * pair at the entry's own page offset, a full method-stream parse, + * and the release-group signature present in the same page (the + * remnant fragment at r914's pa 0x84aef000 carries the signature + * but fails the prefix/parse gate). + */ + { + u8 *pb = kmalloc(SEMA_PAGE_SIZE, GFP_KERNEL); + u32 n = min_t(u32, len, SEMA_PAGE_SIZE - poff); + bool ok = false; + + if (pb) { + const u8 *sig = NULL; + + ok = !svc_read(vgpu, page_pa, pb, SEMA_PAGE_SIZE); + if (ok) { + u32 z; + + for (z = 0; z + 16 <= SEMA_PAGE_SIZE; z++) + if (!memcmp(pb + z, svc_gr_anchor_sig, + sizeof(svc_gr_anchor_sig))) { + sig = pb + z; + break; + } + } + ok = ok && sig && + !memcmp(pb + poff, svc_gr_anchor_pre, + sizeof(svc_gr_anchor_pre)) && + svc_push_prefix_ok(pb + poff, n) && + svc_method_stream_plausible(pb + poff, n); + kfree(pb); + } + if (!ok) { + pr_info("sema-anchor: h=0x%x va=0x%llx candidate pa=0x%llx rejected (content gate)\n", + ch->h_object, va, page_pa); + return false; + } + } + svc_pin_store(va & ~(u64)(SEMA_PAGE_SIZE - 1), page_pa, + SVC_APERTURE_SYS); + svc_lat_store(va & ~(u64)(SEMA_PAGE_SIZE - 1), page_pa, + SVC_APERTURE_SYS); + pr_info("sema-anchor: h=0x%x va=0x%llx -> pa=0x%llx (release-group signature anchored; pinned)\n", + ch->h_object, va, page_pa); + return true; +} + +/* + * r705h: host-triggered pdb discovery. The automatic latch + * (svc_vas_candidates_verify + svc_xvas_pdb_discover) only runs from the RAM + * ring scan, which sweeps [0,6GB) before the client's pushes exist and then + * gives up for the boot; and per-client VAS state is flushed at session end. + * This entry point lets a host tool that CONTENT-LOCATED a verified + * {ring VA -> ring PA} pair (the ring page's GPFIFO entries are its + * fingerprint) hand the pair to the service while the client is alive, so: + * 1. every parked COPY_SERVER_PDES candidate is verified against the pair + * (fast: a walk each), and + * 2. if none verifies, the vidmem walk-root sweep anchors on the pair. + * Registered VASes make the client's pushbuffer VAs walkable and the pump + * can then decode the submit's pushes (the fence release among them). + */ +static u64 svc_pdb_disc_va; +static u64 svc_pdb_disc_pa; +static bool svc_pdb_disc_pending; + +/* + * r707r: bounded channel rewind (debug). After the VAS pdb anchor lands, + * entries skipped before it never re-decode on their own (their unread + * window expires into a skip), so the release they carry would be lost for + * that put epoch. A host tool asks for a rewind (sema_chan_rewind: a hex + * h_object, or "all"); the pump re-processes the channel from slot 0 with + * the consumed-slot bitmap and the per-entry rescue state cleared. + */ +static u32 svc_rewind_h; +static bool svc_rewind_all; +static bool svc_rewind_pending; + +int gsp_sema_service_chan_rewind(u32 h_object) +{ + if (!svc.task) + return -ENODEV; + svc_rewind_h = h_object; + svc_rewind_all = (h_object == 0xffffffffu); + svc_rewind_pending = true; + wake_up_process(svc.task); + return 0; +} + +int gsp_sema_service_pdb_discover(u64 ring_va, u64 ring_pa) +{ + if (!svc.task) + return -ENODEV; + svc_pdb_disc_va = ring_va & ~0xfffULL; + svc_pdb_disc_pa = ring_pa & ~0xfffULL; + svc_pdb_disc_pending = true; + wake_up_process(svc.task); + return 0; +} + +/* + * R10 (patch B): one candidate walk-root probe. Shared by the vidmem + * and guest-sysmem phases of svc_xvas_pdb_discover: a candidate + * registers only when its walk reproduces the content-verified + * ring_va -> ring_pa pair, so a wrong candidate costs a rejected walk, + * never a wrong registration. + */ +static bool svc_xvas_try_root(struct nv_vgpu_device *vgpu, u64 pdb, + u32 entries, bool vidmem, u64 ring_va, + u64 ring_pa) +{ + struct sema_vaspace_desc d; + u64 pa = 0; + int ap = 0; + + if (svc_gmmu_walk(vgpu, pdb, entries, ring_va & ~0xfffULL, + vidmem ? SVC_APERTURE_FB : SVC_APERTURE_SYS, + &pa, &ap, false) || + (pa & ~0xfffULL) != (ring_pa & ~0xfffULL)) + return false; + + memset(&d, 0, sizeof(d)); + d.h_vaspace = 0x5a5a0000u | (u32)(ring_va >> 20); + d.pdb = pdb; + d.num_entries = entries; + d.b_vidmem = vidmem; + gsp_sema_service_vaspace_add(&d); + pr_info("sema-chan: client VAS page directory discovered pdb=0x%llx entries=%u (%s root; ring VA 0x%llx -> pa 0x%llx verified; the client's other VAs are now walkable)\n", + pdb, entries, vidmem ? "vidmem" : "sysmem", ring_va, ring_pa); + return true; +} + +/* + * r496f: discover a kernel-RM client VAS's page directory by verification. + * The CPU RM builds internal-VAS page tables and never publishes them + * (SET_PAGE_DIRECTORY is an externally owned / UVM-only RPC), so the + * pushbuffer pages and the progress semaphore of an X/GL channel cannot + * be translated from any existing source. Try each 4 KB page as the + * walk root and accept the first whose walk reproduces the ring VA -> + * ring PA pair we already validated by content; registering it with the + * VAS table makes every other VA of the same client resolvable through + * the existing walker. + * + * R10 (patch B): bounded guest-sysmem phase. A sysmem client (the X/GL + * kernel-RM channel whose ring is guest RAM) can keep its internal-VAS + * page tables in guest RAM too, where the vidmem sweep cannot see them + * (r880: ring VA 0x120090000 -> pa 0x10643d000 content-validated, no + * root ever reproduced it). Candidates come only from guest-sysmem + * ranges this module already knows: + * - the ring's allocation window, +/- SVC_GR_RAM_WIN/2 around the + * caller's content-verified ring PA (SVC_GR_RAM_WIN, the span + * svc_gr_ram_ring_scan validated the client's ring/pushes in), and + * - the merged guest-allocation map svc.map[]/svc.map_n (struct + * sema_map_entry - "the allocations the guest has described to us", + * see svc_apply_mapped), capped to SVC_XVAS_SYS_MAP_MAX entries. + * Both are additionally bounded by SEMA_PA_MAX (guest RAM); every + * candidate passes the identical verification (svc_xvas_try_root), so a + * wrong candidate cannot be registered. + */ +#define SVC_XVAS_SYS_MAP_MAX 16384 +static void svc_xvas_pdb_discover(struct nv_vgpu_device *vgpu, u64 ring_va, + u64 ring_pa) +{ + static const u32 ent_tab[] = { 4, 8, 16, 32, 64, 2, 128, 256, 512 }; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + unsigned int ei; + + /* Vidmem shadow sweep (r496f; unchanged behaviour). */ + if (vgpu->bars[1].vaddr && lim) { + for (off = 0; off + 0x1000 <= lim; off += 0x1000) + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) + if (svc_xvas_try_root(vgpu, off, + ent_tab[ei], true, + ring_va, ring_pa)) + return; + } + + /* + * R10 (patch B): bounded guest-sysmem phase. The ring allocation + * window first (the strongest guess - the ring scan's own + * convention), then the merged allocation map, skipping pages the + * window phase already covered. + */ + if (ring_pa >= SVC_GR_RAM_WIN / 2 && + ring_pa + SVC_GR_RAM_WIN / 2 <= svc_pa_max()) { + u64 lo = (ring_pa - SVC_GR_RAM_WIN / 2) & ~0xfffULL; + u64 hi = (ring_pa + SVC_GR_RAM_WIN / 2) & ~0xfffULL; + + for (off = lo; off < hi; off += 0x1000) + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) + if (svc_xvas_try_root(vgpu, off, + ent_tab[ei], false, + ring_va, ring_pa)) + return; + } + { + u32 nmap = min_t(u32, svc.map_n, SVC_XVAS_SYS_MAP_MAX); + u32 mi; + + for (mi = 0; mi < nmap; mi++) { + off = svc.map[mi].pa_page & ~0xfffULL; + if (!off || off >= svc_pa_max()) + continue; + if (ring_pa >= SVC_GR_RAM_WIN / 2 && + off >= ring_pa - SVC_GR_RAM_WIN / 2 && + off < ring_pa + SVC_GR_RAM_WIN / 2) + continue; /* window phase covered it */ + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) + if (svc_xvas_try_root(vgpu, off, + ent_tab[ei], false, + ring_va, ring_pa)) + return; + } + } +} + static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -9460,6 +25224,42 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, return; /* wait until the CE channels are known */ /* + * SESSION 148 perf (r858/r859 evidence): a GR channel's ring page + * sits at (notifier_pa page + 0x400). Verified for the RC watchdog + * across r848/r853/r858; extended to all GR channels after r859 + * showed the X channel (0x1000013) NEVER resolving its ring (no + * auto-pin all boot, entries decoded from garbage: va=0x41c000/ + * 0x2017a20/0x10104 with lens 20464..49152, unreadable, skipped; + * ~11 s per frame). The candidate probe (entry0 in the channel's + * own alloc + USERD shape) content-gates every derivation, so a + * wrong guess costs a failed probe - exactly the current behaviour. + */ + if (!ch->engine_type && ch->notifier_pa) { + u64 rpa = (ch->notifier_pa & ~(u64)(SEMA_PAGE_SIZE - 1)) + + 0x400; + + if (svc_gr_ring_candidate(vgpu, ch, rpa, SVC_APERTURE_SYS, + &va, &len, &put, &get)) { + svc_pin_store(ch->gpfifo_va, rpa, SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, rpa, SVC_APERTURE_SYS); + if (ch->userd_reg) + ch->userd_ok = true; + else { + ch->userd_pa = rpa + SVC_GR_USERD_OFF; + ch->userd_as = SVC_APERTURE_SYS; + ch->userd_ok = true; + } + svc_gr_ring_base = rpa; + set_bit(ch->gr_index, &svc_gr_ring_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=1 (notifier-derived; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, ch->gpfifo_va, + rpa, va, len, put, get); + return; + } + } + + /* * Guest truth first: the channel registration carries the USERD * memdesc base for THIS session's allocation, and the ring sits one * USERD block before it (observed for every GR channel across @@ -9514,11 +25314,23 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, ch->gpfifo_entries, &put, &get)) continue; + /* + * SESSION 132: reject a dead candidate when + * the client's own page tables provably + * resolve the ring VA to a live page (the + * r759 vidmem false positive). + */ + if (svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va)) + continue; svc_pin_store(ch->gpfifo_va, cand, cap); svc_lat_store(ch->gpfifo_va, cand, cap); ch->userd_as = cap; ch->userd_ok = true; svc_gr_ring_base = cand; + if (cap == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, cand, + ch->userd_pa); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (userd-derived d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -9549,13 +25361,17 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, len >= 4 && len <= 0x10000 && !svc_userd_probe(vgpu, pa + SVC_GR_USERD_OFF, SVC_APERTURE_FB, ch->gpfifo_entries, - &put, &get)) { + &put, &get) && + !(svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va))) { svc_pin_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); svc_lat_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); ch->userd_pa = pa + SVC_GR_USERD_OFF; ch->userd_as = SVC_APERTURE_FB; ch->userd_ok = true; svc_gr_ring_base = base; + svc_fb_claim_add(ch->h_object, pa, + pa + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -9587,7 +25403,9 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, (u64)(e0 & 0xfffffffc); len = ((e1 >> 10) & 0x1fffff) << 2; if (!(va & 0xfffff) && va >= SEMA_VA_LO && - va < SEMA_VA_HI && len >= 4 && len <= 0x10000) { + va < SEMA_VA_HI && len >= 4 && len <= 0x10000 && + !(svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va))) { /* * Best effort only: a wrapped GP_PUT * legitimately exceeds the entry count, so a @@ -9602,10 +25420,20 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, &get); svc_pin_store(ch->gpfifo_va, pin_pa, pin_ap); svc_lat_store(ch->gpfifo_va, pin_pa, pin_ap); - ch->userd_pa = pin_pa + SVC_GR_USERD_OFF; - ch->userd_as = svc_ap_of(pin_ap); - ch->userd_ok = true; + /* r831: keep a registered real-PA userd. */ + if (ch->userd_reg) { + ch->userd_ok = true; + } else { + ch->userd_pa = + pin_pa + SVC_GR_USERD_OFF; + ch->userd_as = svc_ap_of(pin_ap); + ch->userd_ok = true; + } svc_gr_ring_base = base; + if (svc_ap_of(pin_ap) == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, pin_pa, + pin_pa + + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -9637,10 +25465,23 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, &get)) { svc_pin_store(ch->gpfifo_va, wpa, wap); svc_lat_store(ch->gpfifo_va, wpa, wap); - ch->userd_pa = wpa + SVC_GR_USERD_OFF; - ch->userd_as = wap; - ch->userd_ok = true; + /* + * r831: the ring+0x2000 userd convention is a guess + * for the CUDA GR layout; a channel whose REGISTERED + * userd is a real PA (the RC watchdog) keeps its own + * (the candidate gate above probes it as fallback). + */ + if (ch->userd_reg) { + ch->userd_ok = true; + } else { + ch->userd_pa = wpa + SVC_GR_USERD_OFF; + ch->userd_as = wap; + ch->userd_ok = true; + } svc_gr_ring_base = wpa; + if (wap == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, wpa, + wpa + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (trusted walk; entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -9650,6 +25491,185 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, } } + /* + * SESSION 133: bounded shadow ring scan fallback. + * + * The CE-class rings live in VIDMEM and the FB convention only + * matches when the guest's VRAM allocator happens to place them at + * the conventional offsets (r763: matched, fb 0x80000/0xa0000, the + * channel decoded; r765: read zeros, the ring VA resolved to a + * stale RAM page whose entry0 was dead, the CE channel never + * latched and its inline SASS copies never decoded - black + * frames). When every deterministic candidate has failed, scan the + * shadow for a ring-shaped page: entry0 must name a 1 MB aligned + * push VA in the client window with a sane length, must not be + * provably dead (svc_ring_entry_dead - the r759 false-positive + * guard), and the USERD at the observed deltas (+0x10000 for the + * CE family, +0x2000 for GR) must read a coherent put/get. One + * bounded scan per rank per boot; a false candidate still costs + * only a failed probe. + */ + if (!test_bit(ch->gr_index, &svc_gr_ring_scan_done)) { + u8 *sbase = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL); + u64 off; + + /* + * SESSION 133b: the rings are NOT page-aligned (r767: the + * USERD shape scan found the channel blocks at 0xfee00/ + * 0xff000/0xff200, so the rings sit at unaligned offsets like + * 0xefe00); step by 0x100 and RETRY (rate-limited) - a first + * pass can run before the client has written the ring or the + * USERD, and a consumed once-bit would never rescan. + */ + if (time_before(jiffies, (unsigned long)svc_gr_ring_scan_last + + msecs_to_jiffies(1500))) + return; + svc_gr_ring_scan_last = jiffies; + if (sbase && lim > 0x1000) { + static const u64 udeltas[] = { 0x10000, SVC_GR_USERD_OFF, + 0x1000 }; + + /* + * SESSION 133c pass 1: find the USERD by its OWN + * shape (put@+0x8c, get@+0x88 - the find_userd + * layout) and derive the ring from it. The USERD + * words appear at channel creation and stay; the + * ring entry may be written later, so this pass is + * the stabler trigger. Each hit is validated by the + * derived ring's entry0 (shape + not-dead), so a + * coincidental put/get pair costs a failed check. + */ + for (off = 0; off + 0x90 <= lim; off += 0x100) { + u32 up, ug; + unsigned int ui; + + memcpy(&up, sbase + off + 0x8c, 4); + if (up < 1 || up > 65535) + continue; + memcpy(&ug, sbase + off + 0x88, 4); + if (ug > up) + continue; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 roff; + u32 h0, h1, p3 = 0, g3 = 0; + u64 hva; + u32 hlen; + + if (off < udeltas[ui]) + continue; + roff = off - udeltas[ui]; + memcpy(&h0, sbase + roff, 4); + memcpy(&h1, sbase + roff + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | + (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if (!hva || (hva & 0xfffff) || + hva < SEMA_VA_LO || + hva >= SEMA_VA_HI || hlen < 4 || + hlen > 0x10000) + continue; + if (svc_ring_entry_dead(vgpu, h0, h1)) + continue; + /* + * SESSION 144: never latch a foreign + * block. Without this every CUDA GR + * rank stole the CE rank-0 block while + * its own ring was unwritten + * (r805-r808); staying unlatched here + * lets the FB-convention path latch the + * real ring once the client writes it. + */ + if (svc_fb_claim_foreign(vgpu, + ch->h_object, + roff, off)) + continue; + if (svc_userd_probe(vgpu, off, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3)) + continue; + svc_pin_store(ch->gpfifo_va, roff, + SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, roff, + SVC_APERTURE_FB); + ch->userd_pa = off; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_gr_ring_base = roff; + svc_fb_claim_add(ch->h_object, roff, off); + set_bit(ch->gr_index, + &svc_gr_ring_done); + set_bit(ch->gr_index, + &svc_gr_ring_scan_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (userd-shape scan d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, + ch->gpfifo_va, roff, + udeltas[ui], hva, hlen, p3, + g3); + return; + } + } + /* pass 2: ring-entry-shaped pages (below) */ + for (off = 0; off + 8 <= lim; off += 0x100) { + u32 h0, h1; + u64 hva; + u32 hlen; + unsigned int ui; + + memcpy(&h0, sbase + off, 4); + memcpy(&h1, sbase + off + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | + (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if (!hva || (hva & 0xfffff) || + hva < SEMA_VA_LO || hva >= SEMA_VA_HI || + hlen < 4 || hlen > 0x10000) + continue; + if (svc_ring_entry_dead(vgpu, h0, h1)) + continue; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 upa = off + udeltas[ui]; + u32 p3 = 0, g3 = 0; + + if (upa + 4 > lim) + continue; + if (svc_fb_claim_foreign(vgpu, + ch->h_object, + off, upa)) + continue; + if (svc_userd_probe(vgpu, upa, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3)) + continue; + svc_pin_store(ch->gpfifo_va, off, + SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, off, + SVC_APERTURE_FB); + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_gr_ring_base = off; + svc_fb_claim_add(ch->h_object, off, + upa); + set_bit(ch->gr_index, + &svc_gr_ring_done); + set_bit(ch->gr_index, + &svc_gr_ring_scan_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (shadow scan d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, + ch->gpfifo_va, off, + udeltas[ui], hva, hlen, p3, + g3); + return; + } + } + } + } + if (!test_and_set_bit(ch->gr_index, &svc_gr_ring_tried)) { u8 d1[8] = { 0 }, d2[8] = { 0 }; int cap = ch->userd_as ? ch->userd_as : SVC_APERTURE_SYS; @@ -9703,6 +25723,145 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, pr_info("sema-chan: fbs scan done h=0x%x hits=%d (lim 0x%llx)\n", ch->h_object, hits, lim); } + + /* + * r495 diagnostic (X/GL channel): the X driver's GR channel + * registers its USERD at a memdesc base that may be an FB + * offset or a BAR1 window. The driver's kickoff writes GP_PUT + * through its CPU mapping of that object; if the write landed + * in the emulated FB shadow, scanning for the USERD shape + * (GP_PUT in [1..entries] at +0x8c, GP_GET <= GP_PUT at +0x90, + * both nonzero somewhere in the 0x200 block) names the real + * offset. One-shot per channel, diagnostic only. + */ + if (gsp_sema_dbg && vgpu->bars[1].vaddr && + svc_fb_shadow_size(vgpu) && ch->gpfifo_entries <= 0x20000) { + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + int hits = 0; + + for (off = 0; off + 0xa0 <= lim && hits < 8; + off += 0x200) { + u32 hp, hg; + + memcpy(&hp, vgpu->bars[1].vaddr + off + 0x8c, 4); + memcpy(&hg, vgpu->bars[1].vaddr + off + 0x90, 4); + if (!hp || hp > ch->gpfifo_entries || hg > hp) + continue; + pr_info("sema-chan: userd scan hit off=0x%llx put=%u get=%u (blk %08x %08x %08x %08x)\n", + off, hp, hg, + *(u32 *)(vgpu->bars[1].vaddr + off + 0x80), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x84), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x88), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x94)); + hits++; + } + pr_info("sema-chan: userd scan done h=0x%x hits=%d (lim 0x%llx)\n", + ch->h_object, hits, lim); + } + } + + /* + * r496b: guest-RAM content scan for the ring (see the gr_ram_cursor + * field comment). The driver's kickoff writes GPFIFO entries whose + * pushbuffer VAs sit just BELOW the ring VA in the same allocation + * (main pushbuffer first, GPFIFO after - nvidia-push-priv.h): entry0 + * in (ring_va - 0x100000, ring_va), entry1 above entry0 and up to + * ring_va + 0x100000 (the progress tracker). r908 relaxations, + * measured on the GL client ring (va=0x1203cfff0, gpa 0x231e5d000): + * the entry pair is tested at the ring VA's own in-page offset (the + * entries start AT the registered VA), entry0 may sit either side of + * the ring within the window, and entry1 may repeat entry0's VA + * (that client queues {push,64} then {push,32}). A page whose entry + * pair both name such VAs with sane lengths is the ring; pin it + * (validated) and its allocation window (guesses) so the ring entry + * and pushbuffer reads resolve. Sliced so the service thread stays + * responsive; gives up per channel after svc_gr_ram_pass_budget() + * slices (two sweeps of sema_gr_scan_top). + * + * SESSION 147 (r840 evidence): the scan must NOT run for the kernel + * RC-watchdog channel (h=0x31415900). Its ring is a plain kernel + * allocation whose VA is always resolvable through the driver- + * published VAS walk - the walk is the correct, deterministic + * source. On r840 the walk lost a registration race, the scan ran + * first and latched a FOREIGN page (pa=0x101cc3000, garbage + * entries: raw=0xffff88f10403e000), and the bad latch poisoned + * every later entry/push read: the driver's DMA test never saw its + * push complete and Xorg died with "Failed to initialize DMA" + * (the same boot passed on r835/836/838 where the trusted walk + * won the race). Gating the scan keeps the per-pass walk retry + * alive for the watchdog; the walk landing late is indistinguishable + * from the r838 flow, and never landing is no worse than a poisoned + * latch. + */ + if (ch->h_object != 0x31415900u && + ch->gr_ram_passes < svc_gr_ram_pass_budget() && + !test_bit(ch->gr_index, &svc_gr_ring_done)) { + u64 hit_pa = 0; + + if (svc_gr_ram_ring_scan(vgpu, ch, &hit_pa)) { + /* + * Pin ONLY the content-validated ring page (validated, + * not a window guess). The pushbuffer pages and the + * tracker push of this client resolve through the VAS + * page directory registered from COPY_SERVER_PDES + * (r496g) - the surface is NONCONTIGUOUS, so a linear + * window guess is wrong for them and, being a + * pin_lookup_any hit, would shadow the correct walk in + * svc_lookup_trusted (observed r502: the tracker push + * read zeros through the wrong window page and the + * driver's DMA poll never completed). + */ + svc_pin_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + ch->gr_ram_logged = true; + ch->gr_ram_ring = true; + set_bit(ch->gr_index, &svc_gr_ring_done); + svc_ring_n++; + /* + * r496j: with the ring content-validated, verify the + * parked COPY_SERVER_PDES candidates against it and + * register the directory that reproduces the ring's + * PA; its walk then resolves the client's other VAs + * (pushbuffer pages, the progress semaphore the + * driver polls). r496f's blind vidmem sweep stays as + * a fallback for directories the control never + * published. + */ + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + hit_pa); + svc_xvas_pdb_discover(vgpu, ch->gpfifo_va, hit_pa); + /* + * Persist the latch immediately: a later stalled + * path in this pass may `continue` past the + * end-of-pass write-back, and the next pass's + * snapshot must carry gr_ram_ring (the zero-NOP + * exception and the no-capture-hold skip key on + * it). + */ + { + unsigned long flags2; + int i2; + + spin_lock_irqsave(&svc.chan_lock, flags2); + for (i2 = 0; i2 < SVC_MAX_CHANNELS; i2++) + if (svc.chan[i2].valid && + svc.chan[i2].vgpu == vgpu && + svc.chan[i2].h_object == + ch->h_object) { + svc.chan[i2].gr_ram_ring = true; + svc.chan[i2].gr_ram_logged = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags2); + } + pr_info("sema-chan: GR ring h=0x%x rank=%u found by RAM content scan va=0x%llx -> pa=0x%llx (allocation window pinned; entry/push reads resolve through it)\n", + ch->h_object, ch->gr_index, ch->gpfifo_va, + hit_pa); + return; + } } } @@ -9710,8 +25869,7 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { u64 pa = 0, va = 0; - int ap = SVC_APERTURE_SYS; - u8 e[SVC_GPFIFO_ENTRY_SIZE]; + int ap = SVC_APERTURE_SYS; u8 e[SVC_GPFIFO_ENTRY_SIZE]; u32 e0, e1, len; if (!ch->engine_type) { @@ -9741,6 +25899,13 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, svc_pin_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); svc_lat_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); + /* + * SESSION 144b: record the block ownership here too - CE ranks that + * latch through this plain autopin path (not the first-submission + * capture) were invisible to the GR fallback scan, which then stole + * their block (r809: CE rank 1 at fb 0xa0000). + */ + svc_fb_claim_add(ch->h_object, pa, pa + 0x10000); set_bit(ch->ce_index, &svc_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u)\n", @@ -9748,6 +25913,90 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, } /* + * R10 (patch A): content-anchored ring window for a GR (eng==0) channel. + * + * A GR channel whose ring was found by the guest-RAM content scan + * (gr_ram_ring; svc_gr_ram_ring_scan) has exactly one content-verified + * anchor: the ring page itself, pinned by the scan. The scan's contract + * bounds the channel's pushbuffer VAs to +/- SVC_GR_RAM_WIN/2 around the + * ring VA (entry0 just below the ring, the progress tracker above), so + * install the window as two one-megabyte svc_pb[] entries with the same + * +1 slope convention CE uses (cross-checked on the RC watchdog r880: + * ring 0x120000400 -> pa 0x10567a400, its push 0x120000000 -> + * pa 0x10567a000). + * + * Strictly additive: only eng==0 channels whose ring was content- + * validated are eligible - the kernel-RM sysmem clients the scan exists + * for. CUDA channels keep their verified VAS/walk sources; the RC + * watchdog is excluded from the scan gate. The entries are consumed + * only by svc_pb_win_lookup, i.e. as an untrusted last-resort source in + * the push read, after every per-page validated source missed, with the + * existing all-zero first-chunk rejection still applied. No pin, lat + * or other lookup consumer sees these guesses. + */ +static void svc_pb_pair_gr(struct svc_channel *ch) +{ + u64 ring_pa = 0, ring_va; + int ring_ap = SVC_APERTURE_SYS; + u32 want, n = 0; + + if (ch->gr_index >= 32 || !ch->gr_ram_ring || !ch->gpfifo_va) + return; + if (test_bit(ch->gr_index, &svc_pb_gr_done)) + return; + if (svc_pin_lookup_any(ch->gpfifo_va, &ring_pa, &ring_ap)) + return; + ring_pa &= ~(u64)(SEMA_PAGE_SIZE - 1); + if (!ring_pa || ring_pa >= svc_pa_max()) + return; + ring_va = ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1); + want = SVC_PB_GR_SLOT_BASE | ch->gr_index; + + /* + * r914: replace this rank's previous pair instead of appending. + * svc_pb_win_lookup returns the FIRST matching slot entry, so a + * stale earlier pair (previous attempt's ring PA) would keep + * winning over the refreshed one; svc_chan_rearm_rank re-arms this + * pairing per (re-)registration for exactly that reason. + */ + { + u32 i; + + for (i = 0; i < svc_pb_n; i++) { + if (svc_pb[i].slot == want) { + svc_pb[i] = svc_pb[svc_pb_n - 1]; + svc_pb_n--; + i--; + } + } + } + + /* Lower half: [ring - 1 MB, ring) -> [ring_pa - 1 MB, ring_pa). */ + if (svc_pb_n < SVC_PB_MAX && ring_va >= SVC_GR_RAM_WIN / 2 && + ring_pa >= SVC_GR_RAM_WIN / 2) { + svc_pb[svc_pb_n].va = ring_va - SVC_GR_RAM_WIN / 2; + svc_pb[svc_pb_n].pa_page = ring_pa - SVC_GR_RAM_WIN / 2; + svc_pb[svc_pb_n].slot = want; + svc_pb_n++; + n++; + } + /* Upper half: [ring, ring + 1 MB) -> [ring_pa, ring_pa + 1 MB). */ + if (svc_pb_n < SVC_PB_MAX && + ring_pa + SVC_GR_RAM_WIN / 2 < svc_pa_max()) { + svc_pb[svc_pb_n].va = ring_va; + svc_pb[svc_pb_n].pa_page = ring_pa; + svc_pb[svc_pb_n].slot = want; + svc_pb_n++; + n++; + } + if (!n) + return; + set_bit(ch->gr_index, &svc_pb_gr_done); + pr_info("sema-svc: GR ring window paired h=0x%x rank=%u ring va=0x%llx -> pa=0x%llx (%u entries; untrusted last-resort push window)\n", + ch->h_object, ch->gr_index, ring_va, ring_pa, n); +} + +/* * Pair a channel's push buffer with its base page: the ring's first entry * carries the push-buffer VA, the base page was found by content and its * release names the channel's slot (4 bytes per rank). @@ -9757,7 +26006,11 @@ static void svc_pb_pair(struct nv_vgpu_device *vgpu, struct svc_channel *ch) u64 hit_pa = 0, va = 0; u32 len = 0, i, hit_i = 0, want, off; - if (!ch->engine_type || ch->ce_index >= 32) + if (!ch->engine_type) { + svc_pb_pair_gr(ch); + return; + } + if (ch->ce_index >= 32) return; if (test_bit(ch->ce_index, &svc_pb_done)) return; @@ -9794,6 +26047,9 @@ static void svc_pb_pair(struct nv_vgpu_device *vgpu, struct svc_channel *ch) } svc_pb[hit_i].va = va; set_bit(ch->ce_index, &svc_pb_done); + /* m177c: the base page was content-read (svc_chan_entry0) - a + * verified UVM-space sample for the ring-space directory. */ + svc_pd_ring_note(va, hit_pa); /* * The pinned pairs are verified samples of UVM's own page directory; @@ -10116,7 +26372,7 @@ static int svc_rescue_run_try(struct nv_vgpu_device *vgpu, break; } pa = (u64)((s64)anchor_pa + delta); - if (pa + SEMA_PAGE_SIZE > SEMA_PA_MAX) { + if (pa + SEMA_PAGE_SIZE > svc_pa_max()) { rc = -EINVAL; break; } @@ -10160,6 +26416,19 @@ static int svc_rescue_run_try(struct nv_vgpu_device *vgpu, } /* + * STALLFREE: wall-clock cap for one content-rescue scan range. The scan is + * bounded by the guest-RAM range, not by time: the full fallback walks up to + * 6 GB in 4 KB vfio reads, and under a pin storm / leftover crashed client's + * churn that walk can run for tens of seconds on the service thread, + * delaying every channel (GR launches included). Green scans answer from the + * hot set or the near window in milliseconds, so a 3 s cap never fires on a + * healthy boot; when it does fire the entry falls through its existing grace + * path (hold, then consume with the software release) instead of pinning the + * whole pump. + */ +#define SVC_RESCUE_SCAN_MAX_MS 3000u + +/* * Scan a guest-RAM range descending for a page that validates as this entry's * push (method stream + expected release payload), pinning it and returning 0 * on the first hit. `hi` and `lo` are inclusive chunk start addresses. @@ -10172,10 +26441,20 @@ static int svc_rescue_scan_range(struct nv_vgpu_device *vgpu, { u64 gpa = hi; unsigned long t0 = jiffies; + unsigned long deadline = + t0 + msecs_to_jiffies(SVC_RESCUE_SCAN_MAX_MS); u32 total = 0; u64 bytes = 0; for (;;) { + if (time_after_eq(jiffies, deadline)) { + svc_rescue_scan_capped++; + if (svc_rescue_scan_capped <= 8) + pr_warn("sema-svc: RESCUE-SCAN-CAP h=0x%x va=0x%llx [0x%llx..0x%llx] %llu MB in %u ms (capped; entry keeps its grace path)\n", + ch->h_object, entry_va, hi, lo, + bytes >> 20, SVC_RESCUE_SCAN_MAX_MS); + break; + } if (!svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) { u32 ncand, n; @@ -10239,6 +26518,65 @@ static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, u32 ncand = 0, i; /* + * SESSION 148 perf (measured on r843): content-verified full-chain + * rescue, usable by GR channels too. The CE-only gate below refused + * every GR push outright, yet the same svc_lookup the diagnostic + * probe uses resolves the stalled entries correctly (X's frame + * pushes at 0x120000514 -> vidmem shadow pa=0x200514 with plausible + * stream content) while the pump's own read went through a stale + * shadowing translation, failed plausibility, and skipped the entry + * - one full retry window per push, the measured 22.4 s/frame + * glxgears cadence. Resolve via svc_lookup, verify the bytes parse + * as a method stream, then pin the verified page so the pump's + * re-read lands on it. Cheap: one chain lookup + one page read. + */ + if (entry_va && entry_len && entry_len <= SEMA_PAGE_SIZE) { + u64 mpa = 0; + int map = SVC_APERTURE_SYS; + u8 *vb = kmalloc(entry_len, GFP_KERNEL); + int lrc = -ENOMEM, rrc = -ENOMEM; + bool plaus = false; + + if (vb) { + lrc = svc_lookup(vgpu, entry_va, &mpa, &map); + if (!lrc) + rrc = svc_read_ap(vgpu, mpa, map, vb, + entry_len); + if (!rrc) + plaus = svc_method_stream_plausible(vb, + entry_len); + if (gsp_sema_trace_h && + (u32)ch->h_object == (u32)gsp_sema_trace_h && + time_after(jiffies, svc_perf_last)) { + svc_perf_last = jiffies + HZ; + pr_info("sema-perf: maprescue h=0x%x va=0x%llx len=%u lookup=%d pa=0x%llx ap=%d read=%d plaus=%d\n", + ch->h_object, entry_va, entry_len, + lrc, mpa, map, rrc, plaus); + } + if (!lrc && !rrc && plaus) { + svc_pin_store(entry_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + mpa & + ~(u64)(SEMA_PAGE_SIZE - 1), + map); + svc_lat_store(entry_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + mpa & + ~(u64)(SEMA_PAGE_SIZE - 1), + map); + svc.rescue_map_ok++; + if (svc.rescue_map_ok <= 8) + pr_info("sema-svc: rescue via map h=0x%x va=0x%llx -> pa=0x%llx ap=%d (verified stream)\n", + ch->h_object, entry_va, mpa, + map); + kfree(vb); + return 0; + } + kfree(vb); + } + } + + /* * The matcher needs only the channel rank (slot) and the release-push * shape; it does NOT need the pool semaphore translation that * expected_sem_payload is learned from. Requiring the learned payload @@ -10370,8 +26708,8 @@ static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, lo = ALIGN(svc_pb[i].pa_page > (512ULL << 20) ? svc_pb[i].pa_page - (512ULL << 20) : (u64)SVC_SCAN_CHUNK, SVC_SCAN_CHUNK); - if (hi > SEMA_PA_MAX - SVC_SCAN_CHUNK) - hi = SEMA_PA_MAX - SVC_SCAN_CHUNK; + if (hi > svc_pa_max() - SVC_SCAN_CHUNK) + hi = svc_pa_max() - SVC_SCAN_CHUNK; if (lo <= hi && !svc_rescue_scan_range(vgpu, ch, entry_va, entry_len, slot, hi, lo, buf, cand, cand_va)) { @@ -10396,7 +26734,7 @@ static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, * few hundred MB. */ i = svc_rescue_scan_range(vgpu, ch, entry_va, entry_len, slot, - SEMA_PA_MAX - SVC_SCAN_CHUNK, 0, buf, cand, + svc_pa_max() - SVC_SCAN_CHUNK, 0, buf, cand, cand_va); kfree(buf); if (!i) @@ -10525,20 +26863,106 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, { u32 val; u32 cur = 0; + /* + * STALL FIX (rank-3 completion-word pollution): the publish must + * target the channel's OWN completion word - the descending slot its + * pushes release, which the autoreg above registers and which the + * decoded releases write (release_pa). Writing the ring PUT into the + * ascending rank slot (F1/eea5ea93) made every channel's publish land + * in the paired rank (19-r)'s waited word: the busy ctx-init channel + * rank 16 (h=0x5c000059, PUT~94) overwrote rank 3's word 0x20400ffc0 + * from 1 to 94 (`pub-adv h=0x5c000059 pa=...fc0 val=94 cur=1`), the + * rank-3 stream's own release (payload 1) was then forward-only- + * skipped, and its exact-value sync wait never completed (evtest3 + * cudaStreamSynchronize(st[2]); victory_streams_events). release_pa + * is the push-named truth and wins when known; counter_pa stays the + * fallback for a channel whose pushes are unreadable. + */ + u64 pa = ch->release_pa ? ch->release_pa : ch->counter_pa; - if (!ch->counter_pa) + if (gsp_sema_trace_h && (u32)ch->h_object == (u32)gsp_sema_trace_h && + time_after(jiffies, svc_perf_last)) { + svc_perf_last = jiffies + HZ; + pr_info("sema-perf: pub h=0x%x put=%u last=%u armed=%d pa=0x%llx cnt=%u gate=%s\n", + ch->h_object, put, (u32)ch->last_get, ch->gr_pub_armed, + pa, ch->counter_last, + !pa ? "no-pa" : + (ch->last_get != put) ? "outstanding" : + (!ch->gr_pub_armed ? "unarmed" : "publish")); + } + if (!pa) + return; + /* + * COMPLETION GATE (r205 mail/matmul regression): GP_PUT is the + * client's SUBMISSION position; the pump's last_get is its next + * unconsumed entry. While any submitted entry is outstanding + * (last_get != put) the pushes up to put have NOT all completed, so + * the put must never be published into the completion word: doing so + * short-circuits every client wait (r205 mail: rank-0's word jumped + * to the submitted 108 at last_get=104 - `pub-adv h=0x5c000019 + * pa=...ff0 val=108 cur=104` - ~1 s before the engine ran + * `output retirement wait: 1/968 ... settled`, so the host's + * sub-put release payloads were then forward-only-skipped and the + * client read a stale buffer and failed; matmul's stage ordering + * broke the same way). Only the release applies deliver completions; + * this publish is the catch-up for entries the pump consumed WITHOUT + * a decoded release, so it may run solely when nothing is + * outstanding - the same equality the entry loop consumes to + * (wrap-safe for the 1024-entry rings). + */ + if (ch->last_get != put) return; svc.counter_pub_calls++; if (gsp_sema_dbg && !ch->engine_type && svc.counter_pub_dbg < 96) { svc.counter_pub_dbg++; pr_info("sema-chan: pub-info h=0x%x rank=%u pa=0x%llx put=%u last=%u\n", - ch->h_object, ch->gr_index, ch->counter_pa, put, + ch->h_object, ch->gr_index, pa, put, ch->counter_last); } svc_counter_flush_pending(vgpu, ch); + /* + * mm2 (r207/r208 second-client hang): suppress the GR publish until + * the channel is proven to belong to the current session. During a + * fresh session's first pass the pump consumes the PREVIOUS + * session's residual ring; the completion gate above passes exactly + * then (the stale consumption satisfies last_get == put) and the + * residual put would land in the client's own completion word + * (release_pa). The fresh client's UVM then sees a far-forward + * payload followed by its real release sequence - a downward + * observation larger than UVM_GPU_SEMAPHORE_MAX_JUMP that corrupts + * the tracking counter and wedges the session's first waits (r207 + * victory_matmul: 1 GR push per channel, zero forwards, 242 s). + * gr_pub_armed is set at the fresh-put-drop rewind (session switch + * proven) or when a GR channel is seen with put <= 1 (pristine + * ring, nothing stale to mispublish); a live mid-stream discovery + * (put high, never dropping) loses only this catch-up publish, + * which the release applies already cover. + */ + if (!ch->engine_type && !ch->gr_pub_armed) + return; + val = put; + /* + * r471: never advance the guest's completion word past a push whose + * content was still unwritten when its entry was consumed. The + * clamp position is the lowest all-zero GR slot (gr_bad_put = slot + * + 1; 0 = none). Without it the publish can satisfy a D2H wait + * before its copy ran and the client drains a reused staging buffer + * holding the previous copy's bytes (streams_events h_c[2] = c0). + * The clamp heals when the rewinds re-read the slot with real + * content; the real completions still arrive through the decoded + * releases. + */ + if (!ch->engine_type && ch->gr_bad_put && val >= ch->gr_bad_put) { + val = ch->gr_bad_put - 1; + if (gsp_sema_dbg && svc.counter_pub_dbg < 96) { + svc.counter_pub_dbg++; + pr_info("sema-diag: pub-clamp h=0x%x pa=0x%llx val=%u (bad=%u) cur=%u\n", + ch->h_object, pa, val, ch->gr_bad_put, cur); + } + } if (val == ch->counter_last) return; /* @@ -10546,22 +26970,40 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, * advanced the slot past the put position the client last submitted; * the client polls this u32, so never write it backwards. */ - if (svc_read_ap(vgpu, ch->counter_pa, SVC_APERTURE_SYS, &cur, - sizeof(cur))) + if (svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, &cur, sizeof(cur))) return; if (val <= cur) { if (cur > ch->counter_last) ch->counter_last = cur; return; } - if (svc_write_ap(vgpu, ch->counter_pa, SVC_APERTURE_SYS, - &val, sizeof(val))) - return; + /* + * C5 (GAP-1 H4 secondary): the put-based publish is about to advance + * the same word the decoded releases target; a pre-advance here can + * make a later absolute release a silent skip. Witness every advance. + */ + if (gsp_sema_dbg) + pr_info("sema-diag: pub-adv h=0x%x pa=0x%llx val=%u cur=%u last=%u\n", + ch->h_object, pa, val, cur, + ch->counter_last); + { + int wrc = svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, + &val, sizeof(val)); + if (wrc) { + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: pub-adv-wrc h=0x%x pa=0x%llx val=%u wrc=%d (write did not land)\n", + ch->h_object, pa, val, wrc); + } + return; + } + } ch->counter_last = val; + ch->counter_gen = svc_sess_gen; if (gsp_sema_dbg && svc.counter_pub_dbg < 48) { svc.counter_pub_dbg++; pr_info("sema-chan: counter publish h=0x%x pa=0x%llx val=%u (was %u)\n", - ch->h_object, ch->counter_pa, val, cur); + ch->h_object, pa, val, cur); } } @@ -10649,13 +27091,19 @@ static bool svc_fake_release(struct nv_vgpu_device *vgpu, struct svc_channel *ch sizeof(back)) && back == nv) { delivered = true; - if (svc_pool_sem_va(pool_va)) + if (svc_pool_sem_va(pool_va)) { svc_pool_pa_store(pool_va, spa, svc_ap_of(sap)); + /* r539: reach the secondary candidate + * page too (pin/walk disagreement). */ + svc_pool_boot_fanout(vgpu, + ch->h_object, + pool_va, nv); + } svc.releases_applied++; if (!gsp_sema_fake_rel) svc.releases_recovered++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); } } } @@ -10668,6 +27116,22 @@ static bool svc_fake_release(struct nv_vgpu_device *vgpu, struct svc_channel *ch svc_pending_add(pool_va, deliver, 0); svc.releases_forwarded++; } + /* + * r576b DIAG: witness every authored/queued recovery release and the + * target it chose. The r575 atomics wedge (a UVM tracker wait for + * payload 34 that stalled at 25 while the ring was consumed) needs + * this to prove which VA the fakes targeted when the client's pool + * stopped advancing - last_rel_va is learned from readable pushes and + * a foreign release VA there would send every later recovery to the + * wrong semaphore. + */ + if (gsp_sema_dbg && svc_fake_rel_log < 256) { + svc_fake_rel_log++; + pr_info("sema-fake: h=0x%x pool=0x%llx deliver=%u delivered=%d have_pa=%d last_rel=0x%llx exp=%u\n", + ch->h_object, pool_va, deliver, delivered ? 1 : 0, + have_pa ? 1 : 0, ch->last_rel_va, + ch->expected_sem_payload); + } svc.releases_faked++; svc_rel_count++; ch->expected_sem_payload++; @@ -10706,12 +27170,216 @@ static bool svc_entry_want_reached(struct nv_vgpu_device *vgpu, return cur >= ch->entry_want; } +/* ------------------------------------------------------------------ */ +/* Native publish-counter advance (SESSION 153-CONTINUED-4, pace wall) */ +/* ------------------------------------------------------------------ */ + +/* + * Two CPU-polled publish words gate the GLX client's per-frame pace; both + * live in the X driver's shared pages (CPU-mapped from its memfd) and both + * were proven to be needed: a guest daemon poking only the 0x44000-offset + * page sustained 140 FPS while the other word was already latched at 3; + * resetting both to 2 collapsed to the 22.4 s give-up ladder, and an + * ordered re-poke showed each word releasing part of the stall (page2 3 + + * main 2 -> still stalled; main 3 -> "603 frames in 5.0 seconds"). Each + * page is discovered by content and the counter word is at +0x8. + * + * Page signatures: a boot-varying magic at +0x4 (observed 0x0e836365 and + * 0xc228e806) is useless, but the handle words around it are stable: + * +0x10 family 0xc1d0xxxx and +0x14 == 0xcaf00006 on both observed boots. + * Page B's sparse notifier state is +0x208==0x80, +0x20c==1, +0x210==0. + * Both signatures were verified unique over the full 8 GiB RAM scan on the + * r935 image (and the legacy magic on r934) - exactly one match each. + */ +#define SVC_PUB_A_HANDLE_LO 0x14 /* stable client handle word */ +#define SVC_PUB_A_HANDLE_LO_V 0xcaf00006u +#define SVC_PUB_A_HANDLE_HI 0x10 /* 0xc1d0xxxx family */ +#define SVC_PUB_A_HANDLE_MASK 0xfffff000u +#define SVC_PUB_A_HANDLE_HI_V 0xc1d00000u +#define SVC_PUB_A_MAGIC_OFF 0x4 /* legacy boot-varying magic */ +#define SVC_PUB_A_MAGIC 0x0e836365u +#define SVC_PUB_B_OFF 0x208 /* sparse 2D notifier state */ +#define SVC_PUB_B_208 0x80u +#define SVC_PUB_B_20C 0x1u +#define SVC_PUB_WORD_OFF 0x8 +#define SVC_PUB_VALUE 3u +#define SVC_PUB_SANE_MAX 0x100u /* counter plausibility bound */ +#define SVC_PUB_SCAN_CAP 16 /* attempts before giving up per boot */ + +static u64 svc_pub_pa_a; +static u64 svc_pub_pa_b; +static unsigned long svc_pub_next_scan; +static u32 svc_pub_attempts; +static u32 svc_pub_writes; + +/* + * Content discovery for both publish pages. The pages are created per + * session by the X driver and their GPAs move per boot, so find them by + * structure (the r934 full-RAM scan proved each signature matches exactly + * one page in all of guest RAM). Descending walk: the driver allocates in + * the high band, so the pages are reached within the first chunks. + */ +static void svc_pub_scan(struct nv_vgpu_device *vgpu) +{ + u8 *buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + u64 gpa; + + if (!buf) + return; + svc_pub_attempts++; + for (gpa = svc_pa_max() - SVC_SCAN_CHUNK;; gpa -= SVC_SCAN_CHUNK) { + if (!svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) { + u32 o; + + for (o = 0; o + SEMA_PAGE_SIZE <= SVC_SCAN_CHUNK; + o += SEMA_PAGE_SIZE) { + u32 w4, w8, w10, w14, w208, w20c, w210; + + if (!svc_pub_pa_a) { + memcpy(&w8, buf + o + SVC_PUB_WORD_OFF, 4); + if (w8 <= SVC_PUB_SANE_MAX) { + memcpy(&w4, buf + o + SVC_PUB_A_MAGIC_OFF, 4); + memcpy(&w10, buf + o + SVC_PUB_A_HANDLE_HI, 4); + memcpy(&w14, buf + o + SVC_PUB_A_HANDLE_LO, 4); + if (w4 == SVC_PUB_A_MAGIC || + (w14 == SVC_PUB_A_HANDLE_LO_V && + (w10 & SVC_PUB_A_HANDLE_MASK) == + SVC_PUB_A_HANDLE_HI_V)) + svc_pub_pa_a = gpa + o; + } + } + if (!svc_pub_pa_b) { + memcpy(&w208, buf + o + SVC_PUB_B_OFF, 4); + if (w208 == SVC_PUB_B_208) { + memcpy(&w20c, buf + o + SVC_PUB_B_OFF + 4, 4); + memcpy(&w210, buf + o + SVC_PUB_B_OFF + 8, 4); + memcpy(&w8, buf + o + SVC_PUB_WORD_OFF, 4); + if (w20c == SVC_PUB_B_20C && + w210 == 0 && + w8 <= SVC_PUB_SANE_MAX) + svc_pub_pa_b = gpa + o; + } + } + if (svc_pub_pa_a && svc_pub_pa_b) + goto out; + } + } + if (gpa == 0) + break; + } +out: + kfree(buf); + if (svc_pub_pa_a || svc_pub_pa_b) + pr_info("sema-pubadv: publish pages discovered a=0x%llx b=0x%llx (attempt %u)\n", + svc_pub_pa_a, svc_pub_pa_b, svc_pub_attempts); + else if (svc_pub_attempts <= 8) + pr_info("sema-pubadv: publish pages not found yet (attempt %u)\n", + svc_pub_attempts); +} + +/* + * Write-3 + readback for one target. Returns false when the target is + * unreadable or non-coherent so a stale/foreign page can be dropped and + * re-discovered (never persist a wrong write target). + */ +static bool svc_pub_write_one(struct nv_vgpu_device *vgpu, u64 pa, u32 *back) +{ + u32 v = SVC_PUB_VALUE; + int rc; + + *back = 0; + rc = svc_write(vgpu, pa + SVC_PUB_WORD_OFF, &v, sizeof(v)); + if (!rc) + rc = svc_read(vgpu, pa + SVC_PUB_WORD_OFF, back, sizeof(*back)); + return !rc && *back == v; +} + +/* + * Called from svc_replay_note (pump retire path) for every consumed slot. + * Only kick-family channels (the GLX client's user channels, h=0xbfef00xx) + * drive the publish counters; every other lane is untouched. + */ +static void svc_pub_advance(struct svc_channel *ch) +{ + u32 back_a = 0, back_b = 0; + bool ok_a, ok_b; + + if (!gsp_sema_pub_adv || !ch || !ch->vgpu) + return; + if (ch->h_object < 0xbfef0000u || ch->h_object > 0xbfef00ffu) + return; + if (!svc_pub_pa_a && !svc_pub_pa_b) { + if (gsp_sema_pub_pa) { + svc_pub_pa_a = gsp_sema_pub_pa & + ~((u64)SEMA_PAGE_SIZE - 1); + } else if (svc_pub_attempts < SVC_PUB_SCAN_CAP && + time_after_eq(jiffies, svc_pub_next_scan)) { + svc_pub_scan(ch->vgpu); + svc_pub_next_scan = jiffies + 30 * HZ; + } + if (!svc_pub_pa_a && !svc_pub_pa_b) + return; + } + ok_a = !svc_pub_pa_a || + svc_pub_write_one(ch->vgpu, svc_pub_pa_a, &back_a); + ok_b = !svc_pub_pa_b || + svc_pub_write_one(ch->vgpu, svc_pub_pa_b, &back_b); + svc_pub_writes++; + if (!ok_a && !gsp_sema_pub_pa) { + pr_warn("sema-pubadv: page a write rejected (pa=0x%llx back=%u); dropping page\n", + svc_pub_pa_a, back_a); + svc_pub_pa_a = 0; + svc_pub_next_scan = jiffies; + } + if (!ok_b && !gsp_sema_pub_pa) { + pr_warn("sema-pubadv: page b write rejected (pa=0x%llx back=%u); dropping page\n", + svc_pub_pa_b, back_b); + svc_pub_pa_b = 0; + svc_pub_next_scan = jiffies; + } + if (svc_pub_writes <= 8 || (svc_pub_writes & 0x3ff) == 0) + pr_info("sema-pubadv: h=0x%x a=0x%llx/%s b=0x%llx/%s n=%u\n", + ch->h_object, svc_pub_pa_a, ok_a ? "3" : "ERR", + svc_pub_pa_b, ok_b ? "3" : "ERR", svc_pub_writes); +} + /* Mark a consumed GPFIFO slot for the current put epoch. */ static void svc_replay_note(struct svc_channel *ch, u32 idx, u32 put) { + static u32 consume_log; + + /* + * fail4 r303: name every consumed slot (all channels). Paired with + * the sema-entry decode rows this shows exactly which entries were + * consumed without ops and when (the vector_add chunk-drop chase: + * a consumed slot whose decode row carried a device-dst copy but + * whose op never ran is the silent skip under diagnosis). + */ + if (gsp_sema_dbg && consume_log < 16384) { + consume_log++; + pr_info("sema-consume: h=0x%x idx=%u put=%u\n", + ch->h_object, idx, put); + } (void)put; + /* + * STALL FIX (entry=82/pay=84 class, r186): the entry leaving the + * decode path must never strand a release set that svc_opdef_release() + * withheld for it while an operand was pending. svc_opdef_flush() is + * only reachable from the entry decode path; once the entry is + * consumed (e.g. via the entry_want shortcut) the withheld set sat + * until a reset/drop cleared it WITHOUT applying, and the guest's + * tracker word stopped short of the waiter's value (evtest3 + * cudaStreamSynchronize(st[2]) hung with starved=0). Deliver the set + * here, forward-only, before the slot is marked done. + */ + svc_opdef_finish_ch(ch, idx); if (idx < SVC_GPFIFO_MAX_BITS) set_bit(idx, ch->slot_done_epoch); + /* + * SESSION 153-CONTINUED-4: retirement of a kick-family slot advances + * the GLX client's publish counter (see svc_pub_advance). + */ + svc_pub_advance(ch); } /* Was this slot already consumed in the current put epoch? */ @@ -10836,12 +27504,50 @@ struct svc_fwd_req { bool gr; bool gr_bridge_done; bool gr_page_overflow; + /* + * STALLFREE: bounded-completion state for the forwarded launch. The + * request's backend outcome is the only host edge that can complete a + * launch whose release records engage no hold (svc_fwd_rel_n == 0); + * gr_swrel_done is the exactly-once latch for the software authorship + * of the QMD completion word, and gr_fill_jiffies stamps the request + * so the GR-STALL line reports the real waited time. + */ + bool gr_swrel_done; + /* + * STALLFREE: set by the work item when release0 could not target the + * guest page (unresolved release PA / HPA), so a successful launch's + * engine completion lands only on the module semaphore and the guest + * QMD word stays behind. The pump's bounded outcome check then + * authors it in software (the same write the hold's success edge + * performs), so the client always completes. + */ + bool gr_rel_unobserved; + unsigned long gr_fill_jiffies; u32 gr_h_object; + u32 gr_dropped; /* GRFIX: pages the fill's last pass could not + * resolve (0 = launch fully resolved); read + * by the absolute-bound expiry log */ + bool gr_arg_partial; /* r543: an ARGUMENT pointer resolved 0 + * pages while other arguments resolved - the + * launch would run with a stale/zero operand + * (the r542 matmul NOWRITE wedge: fe_esr= + * 0x80000000, engine fetch OK, wrote 0, + * client read zeros). The request path holds + * such a launch once per push VA through the + * bounded busy-defer retry. */ u64 gr_push_va; u32 gr_push_len; u32 gr_push_n; u32 gr_push_qmd_hdr; u32 gr_push_qmd_end; + /* + * r929 (GL 3D path): the captured raw 3D method window (gr_raw + * words gr_stream_off..gr_stream_off+gr_stream_n) replays verbatim; + * see struct gsp_phys_gr_submit.stream. + */ + bool gr_stream_3d; + u32 gr_stream_off; + u32 gr_stream_n; u32 gr_raw[SVC_GR_PUSH_WORDS]; u64 gr_qmd_dev_va; /* @@ -10865,6 +27571,10 @@ struct svc_fwd_req { u64 gr_rel_pa; u8 gr_rel_ap; /* SVC_APERTURE_* of gr_rel_pa */ u64 gr_prog_va; + /* CONT-87: the window's own program binds (see svc_gr_cand_rec). */ + u32 gr_n_prog_binds; + u64 gr_prog_bind_va[SVC_GR_CAND_PROG_BINDS]; + u32 gr_prog_bind_slot[SVC_GR_CAND_PROG_BINDS]; u64 gr_param_va; u32 gr_param_size; u32 gr_qmd[SVC_GR_QMD_WORDS]; @@ -10880,10 +27590,222 @@ struct svc_fwd_req { static struct svc_fwd_req svc_fwd_req; +/* + * SESSION 153-CONTINUED-61: MISS-ABORT re-arm. The pre-doorbell abort + * (GSP_PHYS_GR_RC_MISSABORT) fires when a 3D-window operand's page is still + * unresolved; the page is demanded through the pin path and resolves within + * the pin cycle (tens of seconds). The historical handling consumed the + * captured candidate and software-completed the client (releases + QMD + * word), so no later forward ever re-ran that chunk and a store-bearing + * window never reached the engine (CONT-60). With the re-arm the candidate + * is copied back into the candidate ring unoffered, the client's releases + * are held back, and a bounded retry tick (svc_gr_missre_tick, called from + * svc_pump_channels) re-offers the same chunk; once the pins land the real + * launch runs and the engine (hold success edge) authors the releases. + * Bounded: after SVC_GR_MISSRE_DEAD_MS / SVC_GR_MISSRE_MAX_TRIES the + * historical consume-with-software-completion path stands so a page that + * never resolves cannot hang the client. Service-thread only. + */ +#define SVC_GR_MISSRE_INT_MS 1500u +#define SVC_GR_MISSRE_DEAD_MS 55000u +#define SVC_GR_MISSRE_MAX_TRIES 64u +static struct { + bool active; + u64 push_va; + u32 h_object; + u32 tries; + unsigned long first_at; + unsigned long last_try; +} svc_gr_missre; +static u32 svc_gr_missre_log; +static u64 svc_gr_missre_arm_n, svc_gr_missre_ok_n, svc_gr_missre_exp_n; + +static bool svc_gr_request(struct nv_vgpu_device *vgpu, + const struct svc_channel *ch); + +/* + * Episode complete: the re-armed launch ran for real. Clears the retry + * state so the tick stops re-offering. Called from the success edges + * (svc_fwd_outcome_check / svc_fwd_hold_poll). + */ +static void svc_gr_missre_done(void) +{ + if (!svc_gr_missre.active || + svc_gr_missre.push_va != svc_fwd_req.gr_push_va) + return; + svc_gr_missre.active = false; + svc_gr_missre_ok_n++; + pr_info("vgsp-gr: MISS-RETRY converged push=0x%llx h=0x%x tries=%u ok=%llu (real launch ran after the pins landed)\n", + svc_fwd_req.gr_push_va, svc_fwd_req.gr_h_object, + svc_gr_missre.tries, svc_gr_missre_ok_n); +} + +/* + * Copy the just-aborted request's captured launch back into the candidate + * ring as an unoffered candidate (the re-arm). Returns false when the + * episode bound has expired or the ring has no free slot; the caller then + * falls through to the historical software completion. `rel`/`nrel` are + * the attempt's push-level release records (restored from the park table + * by svc_gr_request and drained by its caller): they are re-parked so the + * retried request restores the same set - the GPFIFO entry is consumed + * with this edge and will not re-decode. + */ +static bool svc_gr_missre_arm(const struct svc_fwd_rel *rel, u8 nrel) +{ + struct svc_gr_cand_rec *c = NULL; + unsigned long now = jiffies; + u32 slot; + + if (!svc_gr_missre.active || + svc_gr_missre.push_va != svc_fwd_req.gr_push_va || + svc_gr_missre.h_object != svc_fwd_req.gr_h_object) { + svc_gr_missre.active = true; + svc_gr_missre.push_va = svc_fwd_req.gr_push_va; + svc_gr_missre.h_object = svc_fwd_req.gr_h_object; + svc_gr_missre.tries = 0; + svc_gr_missre.first_at = now; + } + if (svc_gr_missre.tries >= SVC_GR_MISSRE_MAX_TRIES || + time_after(now, svc_gr_missre.first_at + + msecs_to_jiffies(SVC_GR_MISSRE_DEAD_MS))) { + u32 ri; + + svc_gr_missre_exp_n++; + pr_warn("vgsp-gr: MISS-RETRY episode expired push=0x%llx h=0x%x tries=%u waited_ms=%lu (falling back to software completion)\n", + svc_gr_missre.push_va, svc_gr_missre.h_object, + svc_gr_missre.tries, + jiffies_to_msecs(now - svc_gr_missre.first_at)); + /* + * Drop the lingering armed slot(s) for this launch: an + * unoffered candidate surviving the episode could be offered + * much later with a stale window snapshot (the client has + * since reused its buffers - the double-run class). + */ + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + struct svc_gr_cand_rec *cc = &svc_gr_cand_ring[ri]; + + if (cc->valid && !cc->offered && + cc->push_va == svc_gr_missre.push_va && + cc->h_object == svc_gr_missre.h_object) { + cc->offered = true; + cc->valid = false; + } + } + svc_gr_missre.active = false; + return false; + } + for (slot = 0; slot < SVC_GR_CAND_RING; slot++) { + if (!svc_gr_cand_ring[slot].valid || + svc_gr_cand_ring[slot].offered) { + c = &svc_gr_cand_ring[slot]; + break; + } + } + if (!c) + return false; /* ring full: fall back historically */ + memset(c, 0, sizeof(*c)); + c->valid = true; + c->offered = false; + c->rel_done = false; + c->captured_jiffies = now; + c->h_object = svc_fwd_req.gr_h_object; + c->push_va = svc_fwd_req.gr_push_va; + c->push_len = svc_fwd_req.gr_push_len; + c->push_n = svc_fwd_req.gr_push_n; + c->push_qmd_hdr = svc_fwd_req.gr_push_qmd_hdr; + c->push_qmd_end = svc_fwd_req.gr_push_qmd_end; + c->stream_3d = svc_fwd_req.gr_stream_3d; + c->stream_off = svc_fwd_req.gr_stream_off; + c->stream_n = svc_fwd_req.gr_stream_n; + memcpy(c->raw, svc_fwd_req.gr_raw, sizeof(c->raw)); + c->qmd_dev_va = svc_fwd_req.gr_qmd_dev_va; + memcpy(c->qmd, svc_fwd_req.gr_qmd, sizeof(c->qmd)); + c->n_inline = svc_fwd_req.gr_n_inline; + if (c->n_inline > SVC_GR_INLINE_MAX) + c->n_inline = SVC_GR_INLINE_MAX; + memcpy(c->inline_recs, svc_fwd_req.gr_inline, + sizeof(c->inline_recs)); + c->prog_va = svc_fwd_req.gr_prog_va; + c->param_va = svc_fwd_req.gr_param_va; + c->param_size = svc_fwd_req.gr_param_size; + c->rel_va = svc_fwd_req.gr_rel_va; + c->rel_payload = svc_fwd_req.gr_rel_payload; + c->rel_flags = svc_fwd_req.gr_rel_flags; + if (rel && nrel) + svc_grpark_put(svc_fwd_req.vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va, rel, nrel); + svc_gr_missre_arm_n++; + if (svc_gr_missre_log < 96) { + svc_gr_missre_log++; + pr_info("vgsp-gr: MISS-RETRY re-arm push=0x%llx h=0x%x slot=%u window=%u nrel=%u arm=%llu (candidate kept unoffered; retry tick re-offers after the pins)\n", + c->push_va, c->h_object, slot, c->stream_n, + (unsigned)nrel, svc_gr_missre_arm_n); + } + return true; +} + +/* + * The retry tick (called per pass from svc_pump_channels for the tracked + * channel): re-offer the re-armed candidate on a bounded cadence. Parks + * svc_gr_cand_pos on the tracked slot so svc_gr_request's oldest-unoffered + * scan selects exactly this candidate; svc_gr_request itself defers on + * busy/inflight and no-ops for channels without a matching candidate. + */ +static void svc_gr_missre_tick(struct nv_vgpu_device *vgpu, + const struct svc_channel *ch) +{ + u32 ti; + + if (ch->engine_type || !svc_gr_missre.active || + svc_gr_missre.h_object != ch->h_object || + !gsp_phys_gr_forward_enabled() || + !time_after_eq(jiffies, svc_gr_missre.last_try + + msecs_to_jiffies(SVC_GR_MISSRE_INT_MS))) + return; + if (svc_gr_missre.tries >= SVC_GR_MISSRE_MAX_TRIES || + time_after(jiffies, svc_gr_missre.first_at + + msecs_to_jiffies(SVC_GR_MISSRE_DEAD_MS))) + return; /* the next abort edge expires the episode */ + svc_gr_missre.last_try = jiffies; + svc_gr_missre.tries++; + for (ti = 0; ti < SVC_GR_CAND_RING; ti++) { + struct svc_gr_cand_rec *cc = &svc_gr_cand_ring[ti]; + + if (cc->valid && !cc->offered && + cc->push_va == svc_gr_missre.push_va && + cc->h_object == svc_gr_missre.h_object) { + svc_gr_cand_pos = ti; + break; + } + } + if (svc_gr_missre_log < 192) { + svc_gr_missre_log++; + pr_info("vgsp-gr: MISS-RETRY tick try=%u push=0x%llx h=0x%x (re-offering the armed candidate)\n", + svc_gr_missre.tries, svc_gr_missre.push_va, + svc_gr_missre.h_object); + } + if (svc_gr_request(vgpu, ch)) { + /* + * A tick-queued request never gets an entry hold (the GPFIFO + * entry was consumed with the first abort - there is nothing + * left for the entry walk to hold). The per-push release + * records svc_gr_request just restored are re-parked so they + * survive the work item's lifetime; svc_fwd_outcome_check + * applies them when the retried launch's outcome lands. + */ + if (svc_fwd_defer_armed && svc_fwd_rel_n > 0) + svc_grpark_put(vgpu, ch->h_object, + svc_gr_missre.push_va, + svc_fwd_rel, svc_fwd_rel_n); + } +} + static struct workqueue_struct *svc_fwd_wq; static unsigned long svc_fwd_state; static u64 svc_fwd_requests; static u64 svc_fwd_dropped; +static u32 svc_grvaudit_launches; /* GRVAUDIT: per-launch audit counter */ +static u32 svc_readdiv_log; /* r233 READ-DIVERGE-WIT budget */ /* * Pause gate shared by the pump's request path and the backend release path: * quiesce takes this lock, sets paused and waits for the in-flight work item; @@ -10895,6 +27817,34 @@ static DEFINE_MUTEX(svc_fwd_lock); static bool svc_fwd_paused; /* + * F1 helpers: does a software release write target the release word of the + * forwarded GR launch the request slot holds? The slot is written only by the + * service thread (this code also runs there) and is never reused while the + * hold is active, so no lock is needed - same convention as the existing + * svc_fwd_hold_gr_release()/svc_fwd_hold_expired() reads. svc_gr_rel_match() + * covers the just-resolved hold (done may already be true); the inflight + * variant additionally requires the SVC_FWD_INFLIGHT bit and an unresolved + * work item, i.e. the hardware copy the poll is waiting on is still running. + */ +static bool svc_gr_rel_match(u64 va, u32 *payload) +{ + if (!va || !svc_fwd_req.gr || svc_fwd_req.gr_rel_va != va) + return false; + if (payload) + *payload = svc_fwd_req.gr_rel_payload; + return true; +} + +static bool svc_gr_rel_inflight(u64 va, u32 *payload) +{ + if (!svc_gr_rel_match(va, payload)) + return false; + if (!test_bit(SVC_FWD_INFLIGHT, &svc_fwd_state)) + return false; + return !smp_load_acquire(&svc_fwd_req.done); +} + +/* * Resolve one captured operand to an HPA. SYS (guest RAM) operands resolve * the page-aligned GPA through the guest-RAM backing file; VID operands * resolve the shadow byte offset through the vmalloc backing of BAR1 (the @@ -10924,6 +27874,24 @@ static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind, return -EFAULT; *hpa = (u64)page_to_phys(pg); *out_off = page_off; + /* + * r372 two-path identity check: the copy-out writes through + * __va(hpa) while every reader reads base+pa - if those are + * not the same memory the engine's output silently vanishes + * (the sum=0 class). Prove it empirically, bounded. + */ + if (svc_fbhpa_log < 16) { + u64 at_base = 0, at_va = 0; + + svc_fbhpa_log++; + memcpy(&at_base, base + page, sizeof(at_base)); + memcpy(&at_va, (u8 *)__va(*hpa), sizeof(at_va)); + pr_info("sema-fbhpa: pa=0x%llx base=%px pg=%px hpa=0x%llx shadow=0x%llx at_base=%016llx at___va=%016llx %s\n", + page, base, pg, *hpa, + (unsigned long long)svc_fb_shadow_size(vgpu), + at_base, at_va, + at_base == at_va ? "SAME" : "DIFFER"); + } return 0; } if (kind != SVC_APERTURE_SYS) @@ -11043,6 +28011,19 @@ static void svc_fwd_soft_copy_inline(struct nv_vgpu_device *vgpu) * hardware has not produced yet. */ #define SVC_FWD_GR_HOLD_TIMEOUT_MS 12000u +/* + * r420: hard cap on the WEDGE-QUIET deferral of the GR hold deadline. The + * backend's quiet hold is a bounded msleep (its own caps: <=240 s per hold, + * two holds per launch), so the pause can only last a few minutes; the cap is + * a defence-in-depth bound so a stuck quiet-hold flag can never defer the + * client's completion indefinitely. + */ +#define SVC_FWD_GR_HOLD_PAUSE_MAX_MS 600000u +/* + * r420: same defence for the per-launch consumption bound pause in + * svc_gr_hold_bound_expired(). + */ +#define SVC_GR_QUIET_PAUSE_MAX_MS 600000u struct svc_fwd_hold { bool active; @@ -11078,9 +28059,9 @@ struct svc_fwd_hold { /* * P6c completion relay: the request selected record relay_index and * the hardware copy was asked to author it into the guest-visible - * semaphore page. On the success edge the hold re-reads the guest - * word and, when it carries the payload, skips that one record's - * software apply. Cleared with the hold. + * semaphore page. On the success edge the hold re-reads the guest word + * and, when it carries the payload, skips that one record's software + * apply. Cleared with the hold. */ bool relay_armed; u8 relay_index; @@ -11093,6 +28074,18 @@ struct svc_fwd_hold { struct svc_fwd_rel rel[SVC_FWD_REL_MAX]; u64 last_rel_va; unsigned long since; + /* + * r420: WEDGE-QUIET deferral state. While the backend deliberately + * holds the in-flight launch in its bounded quiet window + * (gsp_phys_gr_quiet_hold_active), the GR deadline is pushed forward + * (paused_until) so the hold cannot author the launch's release word + * under the hold - the client would then verify data the post-hold + * copy-out has not written yet (the r419 mid-boot escape). The + * deferral is itself bounded by SVC_FWD_GR_HOLD_PAUSE_MAX_MS from + * `since` and is announced once per hold. + */ + unsigned long paused_until; + bool pause_logged; }; static struct svc_fwd_hold svc_fwd_hold; @@ -11144,8 +28137,19 @@ static void svc_fwd_rel_apply_launch_deferred(struct nv_vgpu_device *vgpu, bool pending = r->op != 0; bool inc = DMA_SEM_REDUCTION_EN(r->dma) && DMA_SEM_REDUCTION(r->dma) == SEM_REDUCTION_INC; + int rd = svc_sem_read(vgpu, r->va, &cur); - if (svc_sem_read(vgpu, r->va, &cur)) { + /* + * C2 (GAP-1 H2/H3): prove the opdef apply handed the record to the + * helper and what the software read of the release word returned. + */ + if (gsp_sema_dbg && svc_relx_log < 128) { + svc_relx_log++; + pr_info("sema-diag: relx kind=%u op=%u inc_claim=%u va=0x%llx pay=%u rd=%d\n", + (unsigned)r->kind, (unsigned)r->op, + (unsigned)r->inc_claimed, r->va, r->payload, rd); + } + if (rd) { svc_pending_add(r->va, r->payload, inc ? 1 : 0); svc_rel_apply_launch(vgpu, ch, r->va, r->payload, 0, pending, inc, r->inc_claimed); @@ -11230,8 +28234,17 @@ static void svc_fwd_rel_apply_all(struct nv_vgpu_device *vgpu, * Bounded hold: comfortably above the observed ~1 s pin latency and well * inside the 25 s CE grace, so a genuinely dead operand delays its * channel by this much and then falls back to the old skip-and-release. + * + * r549: 10000 -> 3000. r548 evidence: a reused channel handle whose new + * client submits ~65 init entries with device-dst ops no resolver covers + * walked one entry per 10 s (the journal shows OPDEF-EXPIRED exactly + * every 10.5 s, idx 7 -> 18) while the client spun at 100 % CPU - the + * release set the client waits on advances one entry per hold, so the + * bound is the stall multiplier. Ops that do resolve land in 0.5-2 s; + * 3 s keeps the m163 guarantee (a failed device copy is still retried + * once after its pins arrive) and caps the dead-op walk 3.3x shorter. */ -#define SVC_OPDEF_HOLD_MS 10000u +#define SVC_OPDEF_HOLD_MS 3000u struct svc_opdef_ent { bool active; @@ -11282,11 +28295,110 @@ static struct svc_opdef_ent *svc_opdef_alloc(struct nv_vgpu_device *vgpu, return NULL; } -static void svc_opdef_clear(struct svc_opdef_ent *e) +/* C3 caller tags for the opdef-clear witness (H2: which path cleared). */ +#define SVC_OPDEF_CLR_APPLY 0u +#define SVC_OPDEF_CLR_FINISH 1u +#define SVC_OPDEF_CLR_FLUSH 2u +#define SVC_OPDEF_CLR_DROP 3u +#define SVC_OPDEF_CLR_STALE 4u +#define SVC_OPDEF_CLR_COUNTED 5u +#define SVC_OPDEF_CLR_SWEEP 6u +#define SVC_OPDEF_CLR_STOP 7u + +static void svc_opdef_clear(struct svc_opdef_ent *e, u32 caller) { + /* + * C3 (GAP-1 H2): every clear is witnessed with its caller and the + * slot contents, so a "silent" clear (n=0, or a kind the apply + * switch ignores) is distinguishable from a delivery. + */ + if (gsp_sema_dbg && e->active && svc_opdef_clr_log < 128) { + svc_opdef_clr_log++; + pr_info("sema-diag: opdef-clear caller=%u h=0x%x entry=%u n=%u kind0=%u va0=0x%llx pay0=%u\n", + caller, e->h_object, e->entry_idx, (unsigned)e->n, + e->n ? (unsigned)e->rel[0].kind : 0u, + e->n ? e->rel[0].va : 0ull, + e->n ? e->rel[0].payload : 0u); + } + /* + * r456/r457: the snapshot lifecycle is now "capture once, consume + * on the copy's success, evict on ring-slot reuse, TTL". Every + * opdef_clear caller can still re-decode the entry: + * - APPLY (0): the pass's own op execution (which consumes the + * snapshot on success) follows the apply; + * - FINISH (1): the entry can be REPLAYED after a consume (the + * pin-batch rewind replays entries while mappings stream in), + * and an owed copy still needs the pipeline-order bytes; + * - FLUSH (2): the op stays skipped this pass but a later replay + * may run it. + * The r457 mode-C evidence: the b1 H2D's snapshot was captured at + * 191607.816388 with b1's bytes (first4=00000000 3e800000 ...), + * FINISH freed it on the entry's consume, the rewind replay + * re-captured at 191608.798838 from the REFILLED staging + * (first4=00000000 3d800000 ... = b2) and the deferred execution + * wrote b2's bytes into db1 (dst16_after=000000000000803d...). + * The r444 leak the old free guarded against is already covered + * by the owner key (a stale snapshot can only ever serve its own + * (ch, idx, hash)), the slot-reuse eviction in + * svc_copy_snap_capture, the ops_done skip free in svc_run_ops + * and the 30 s TTL. + */ memset(e, 0, sizeof(*e)); } +/* + * r551: remember releases DELIVERED by the hold-expiry paths (flush and + * sweep) by (channel, entry, va, payload) for a short window. The pump + * re-decodes a parked entry on every pass; without this mark each re-decode + * recorded a fresh opdef slot and held the entry again - r550 + * victory_strided: SWEEP-EXPIRED every ~3 s for the SAME entry six times + * (entries 41..43, ~18-21 s per entry, OPDEF-EXPIRED only after six + * cycles), so a ~40-entry init sequence walked for 12+ minutes with the + * client at 100 % CPU. The release is a monotonic completion signal and + * every delivery path is forward-only, so re-applying it is a no-op: a + * re-decode of an already-delivered entry must neither withhold the + * release again nor re-arm the hold. + */ +static void svc_opdef_done_note(u32 h, u32 entry_idx, const struct svc_fwd_rel *r) +{ + u32 i = svc_opdef_done_next++ % SVC_OPDEF_DONE_SLOTS; + + svc_opdef_done[i].h_object = h; + svc_opdef_done[i].entry_idx = entry_idx; + svc_opdef_done[i].va = r->va; + svc_opdef_done[i].payload = r->payload; + svc_opdef_done[i].j = jiffies; +} + +static bool svc_opdef_done_seen(u32 h, u32 entry_idx, const struct svc_fwd_rel *r) +{ + u32 i; + + for (i = 0; i < SVC_OPDEF_DONE_SLOTS; i++) + if (svc_opdef_done[i].h_object == h && + svc_opdef_done[i].entry_idx == entry_idx && + svc_opdef_done[i].va == r->va && + svc_opdef_done[i].payload == r->payload && + time_before(jiffies, svc_opdef_done[i].j + + msecs_to_jiffies(SVC_OPDEF_DONE_TTL_MS))) + return true; + return false; +} + +/* Any release of this entry delivered recently? (hold gate) */ +static bool svc_opdef_done_entry(u32 h, u32 entry_idx) +{ + u32 i; + + for (i = 0; i < SVC_OPDEF_DONE_SLOTS; i++) + if (svc_opdef_done[i].h_object == h && + svc_opdef_done[i].entry_idx == entry_idx && + time_before(jiffies, svc_opdef_done[i].j + + msecs_to_jiffies(SVC_OPDEF_DONE_TTL_MS))) + return true; + return false; +} + static bool svc_opdef_contains(const struct svc_opdef_ent *e, const struct svc_fwd_rel *r) { @@ -11315,13 +28427,78 @@ static void svc_opdef_apply(struct nv_vgpu_device *vgpu, if (e->n) { svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); svc.opdef_applied++; - if (gsp_sema_dbg && svc_opdef_log < 32) { + if (gsp_sema_dbg && svc_opdef_log < (gsp_sema_dbg ? 512 : 32)) { svc_opdef_log++; pr_info("sema-opdef: apply h=0x%x entry=%u n=%u (ops complete)\n", ch->h_object, entry_idx, e->n); } } - svc_opdef_clear(e); + svc_opdef_clear(e, SVC_OPDEF_CLR_APPLY); +} + +/* + * r34x: is a release to this VA currently withheld by our own machinery? + * True when the pending-translation ring holds one (translation late) or + * an opdef slot has one recorded (operand hold). The acquire gate calls + * this to classify an unsatisfied wait in its diagnostics: pend=1 names + * the 'value sits behind our own withhold' case - the bounded 100 ms + * give-up is exactly for it (waiting longer cannot install the value + * until the upstream entry's ops complete). Never forces an opdef release + * here: those releases are withheld because their ops have NOT completed, + * and delivering early re-opens the work-before-fence hole they close. + */ +static bool svc_wait_pend_release(u64 va) +{ + u32 i; + + for (i = 0; i < svc_pending_n; i++) + if (svc_pending[i].va == va) + return true; + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + u8 k; + + if (!e->active) + continue; + for (k = 0; k < e->n; k++) + if (e->rel[k].va == va) + return true; + } + return false; +} + +/* + * STALL FIX (r186 entry=82/pay=84): called from svc_replay_note() on every + * entry-consume path. If a release set was withheld for this entry while an + * operand was pending and the entry is now leaving the decode path (so the + * 10 s flush can never run for it), deliver the set forward-only and clear + * the slot. A drop without delivery is the stall class: the guest word + * stops short of the value its tracker waits for. + */ +static void svc_opdef_finish_ch(struct svc_channel *ch, u32 idx) +{ + u32 i; + + if (!ch || idx == SVC_ENTRY_NONE) + return; + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + + if (!e->active || e->h_object != ch->h_object || + e->vgpu != ch->vgpu || e->entry_idx != idx) + continue; + if (e->n) { + svc_fwd_rel_apply_all(e->vgpu, ch, e->rel, e->n); + svc.opdef_delivered_late++; + if (svc_opdef_log < 64) { + svc_opdef_log++; + pr_info("sema-opdef: LATE-FLUSH h=0x%x entry=%u n=%u (entry consumed before deadline; releases delivered)\n", + ch->h_object, idx, e->n); + } + } + svc_opdef_clear(e, SVC_OPDEF_CLR_FINISH); + return; + } } /* @@ -11350,6 +28527,16 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, if (svc.op_tr_pending) { if (rescue) return e != NULL; + /* + * r551: this release was already delivered forward-only by the + * hold-expiry flush/sweep; the pump's re-decode of the parked + * entry must not withhold it (or re-arm the hold) again. + */ + if (svc_opdef_done_seen(ch->h_object, entry_idx, r)) { + svc_rel_va_seen = r->va; + svc_rel_count++; + return true; + } if (!e) { e = svc_opdef_alloc(vgpu, ch->h_object, entry_idx); if (!e) @@ -11358,7 +28545,7 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, if (!svc_opdef_contains(e, r)) { if (r->payload && e->n < SVC_FWD_REL_MAX) { e->rel[e->n++] = *r; - if (gsp_sema_dbg && svc_opdef_log < 32) { + if (gsp_sema_dbg && svc_opdef_log < (gsp_sema_dbg ? 512 : 32)) { svc_opdef_log++; pr_info("sema-opdef: hold h=0x%x entry=%u kind=%u va=0x%llx pay=%u (operand pending)\n", ch->h_object, entry_idx, @@ -11400,20 +28587,58 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, * flushes the slot and consumes the entry. */ static bool svc_opdef_hold(struct nv_vgpu_device *vgpu, struct svc_channel *ch, - u32 entry_idx) + u32 entry_idx, u64 launch_va) { struct svc_opdef_ent *e = svc_opdef_find(vgpu, ch->h_object, entry_idx); + /* r551: a delivered entry is consumed, never held again. */ + if (svc_opdef_done_entry(ch->h_object, entry_idx)) + return false; if (!e) { e = svc_opdef_alloc(vgpu, ch->h_object, entry_idx); if (!e) return false; } + /* + * r361: absolute per-launch bound keyed to the LAUNCH PUSH VA + * (the GRFIX lesson, r213). The slot's `since` restarts whenever + * a pin-batch rewind replays the entry at a different GPFIFO + * index, so a slot-keyed-only bound can be re-armed forever - + * observed as opdef=3301/32/0 (held climbing, expired 0) with the + * guest wedged. The push VA never changes across replays, so the + * first hold of a launch arms an ABSOLUTE deadline nothing moves. + */ + if (!launch_va) + launch_va = 1; /* anonymous launch: still absolutely bound */ + /* + * r368: an ACTIVE bound is never re-armed. The pump walks up to + * SVC_OPDEF_SLOTS held entries, each with its OWN push VA; the r361 + * arm-on-VA-change rule restarted the deadline on every entry + * change, so with several held launches the deadline moved forever + * (streams2 wedge: opdef=955/13/0, held climbing, expired 0). The + * first hold arms the absolute deadline; entries held while it is + * active ride it and flush together when it fires. A fresh bound is + * armed once the previous one has passed. + */ + if (!ch->opdef_hold_dead_va || + time_after_eq(jiffies, ch->opdef_hold_deadline)) { + ch->opdef_hold_dead_va = launch_va; + ch->opdef_hold_deadline = jiffies + + msecs_to_jiffies(SVC_OPDEF_HOLD_MS); + ch->opdef_hold_exp_logged = false; + } if (time_before(jiffies, - e->since + msecs_to_jiffies(SVC_OPDEF_HOLD_MS))) { + e->since + msecs_to_jiffies(SVC_OPDEF_HOLD_MS)) && + time_before(jiffies, ch->opdef_hold_deadline)) { svc.opdef_held++; return true; } + if (!ch->opdef_hold_exp_logged) { + ch->opdef_hold_exp_logged = true; + pr_err_ratelimited("vgsp-sema: OPDEF-EXPIRED h=0x%x idx=%u va=0x%llx (absolute per-launch bound %u ms; rewind replays no longer re-arm)\n", + ch->h_object, entry_idx, launch_va, + SVC_OPDEF_HOLD_MS); + } return false; } @@ -11426,12 +28651,16 @@ static void svc_opdef_flush(struct nv_vgpu_device *vgpu, struct svc_channel *ch, if (!e) return; if (e->n) { + u8 i; + svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); + for (i = 0; i < e->n; i++) + svc_opdef_done_note(ch->h_object, entry_idx, &e->rel[i]); pr_info("sema-opdef: EXPIRED h=0x%x entry=%u n=%u applied (operand never translated; op stays skipped)\n", ch->h_object, entry_idx, e->n); } svc.opdef_expired++; - svc_opdef_clear(e); + svc_opdef_clear(e, SVC_OPDEF_CLR_FLUSH); } /* @@ -11445,17 +28674,67 @@ static void svc_opdef_drop(struct nv_vgpu_device *vgpu, struct svc_channel *ch, { struct svc_opdef_ent *e = svc_opdef_find(vgpu, ch->h_object, entry_idx); - if (e) - svc_opdef_clear(e); + if (!e) + return; + /* + * STALL FIX (r186/r187 entry=82, va=0x20400fef0 pay=84): when the + * forwarding machinery takes ownership of an entry, the withheld set + * may NOT ride the forwarded candidate's rel[] (svc_opdef_release() + * returned before the release was appended), so a blind clear strands + * the guest's tracker word with NO LATE-FLUSH/LOSS/EXPIRED ever + * logged (crel freezes right after the launches; evtest3 + * cudaStreamSynchronize(st[2]) hangs). Counter releases are + * monotonic completion signals: apply them forward-only here - a + * record at/ahead of the word is a no-op, so the fwd edge cannot + * double-apply and an additive reduction is skipped. + */ + { + u8 i; + u32 delivered = 0; + + for (i = 0; i < e->n; i++) { + if (!svc_counter_va(e->rel[i].va)) + continue; + if (e->rel[i].kind == SVC_FWD_REL_906F && + e->rel[i].op == HOST_SEM_OP_REDUCTION) + continue; /* additive: not idempotent */ + svc_fwd_rel_apply_all(vgpu, ch, &e->rel[i], 1); + svc.opdef_delivered_late++; + delivered++; + pr_info_ratelimited("sema-opdef: DROP-APPLY h=0x%x entry=%u va=0x%llx pay=%u (counter release delivered at fwd-ownership handoff)\n", + ch->h_object, entry_idx, + e->rel[i].va, e->rel[i].payload); + } + if (delivered < e->n) { + svc.opdef_drop_lost += (u32)(e->n - delivered); + pr_info_ratelimited("sema-opdef: LOSS h=0x%x entry=%u %u of %u withheld release(s) dropped at fwd-ownership handoff\n", + ch->h_object, entry_idx, + (u32)(e->n - delivered), e->n); + } + } + svc_opdef_clear(e, SVC_OPDEF_CLR_DROP); } /* * The ring advanced past a slot's entry (a slot for another entry of the * same channel is being processed): its set belongs to a consumed entry. * Deliver late (forward-only, so it cannot double-apply) and drop it. + * + * r406: an entry is only "passed" if the ring actually CONSUMED it + * (slot_done_epoch). Without that proof the pin-batch rewind breaks the + * invariant: it re-opens lower indices (e.g. 83) while a held entry + * (e.g. 91) sits AHEAD in the same epoch, and the old unconditional drop + * treated the held entry as consumed and delivered its withheld releases + * early. Observed (r406 inplace_scale): the H2D copy's operand was + * unresolved (rc=-2), the entry held with its release withheld; a rewind + * pass at idx=83 fired drop-stale caller=4 for idx=91 and the release + * went out at 104955.751 - while the copy only executed at 104957.418. + * The client raced past its blocking memcpy, launched, the launch read + * the stale buffer (previous run's data), and the copy landed after the + * launch's copy-out: wrong results with 32/1032 "changed". */ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu, - struct svc_channel *ch, u32 keep_idx) + struct svc_channel *ch, u32 keep_idx, u32 put) { int i; @@ -11465,15 +28744,27 @@ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu, if (!e->active || e->vgpu != vgpu || e->h_object != ch->h_object || e->entry_idx == keep_idx) continue; + if (!svc_replay_seen(ch, e->entry_idx, put)) + continue; /* never consumed: not stale (rewind) */ if (e->n) svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); - svc_opdef_clear(e); + svc_opdef_clear(e, SVC_OPDEF_CLR_STALE); } } /* A slot of THIS pump's guest whose channel is gone from the pass snapshot: * drop it. Slots of other guests are never touched (their channels are not in * this snapshot by construction). */ +static void svc_opdef_drop_counted(struct svc_opdef_ent *e) +{ + if (e->n) { + svc.opdef_drop_lost += e->n; + pr_info_ratelimited("sema-opdef: LOSS entry h=0x%x idx=%u dropped with %u undelivered release(s)\n", + e->h_object, e->entry_idx, e->n); + } + svc_opdef_clear(e, SVC_OPDEF_CLR_COUNTED); +} + static void svc_opdef_drop_gone(struct nv_vgpu_device *vgpu, const struct svc_channel *local, int n) { @@ -11489,16 +28780,132 @@ static void svc_opdef_drop_gone(struct nv_vgpu_device *vgpu, local[k].h_object == e->h_object) break; if (k == n) - svc_opdef_clear(e); + svc_opdef_drop_counted(e); } } static void svc_opdef_reset_all(void) { + int i; + + /* STALL FIX: a non-applying clear is a silent release loss - count it. */ + for (i = 0; i < SVC_OPDEF_SLOTS; i++) + if (svc_opdef[i].active && svc_opdef[i].n) + svc_opdef_drop_counted(&svc_opdef[i]); memset(svc_opdef, 0, sizeof(svc_opdef)); } /* + * STALL FIX (r189 entry=82/83 va=0x20400fef0 pay=84/85): the 10 s deadline + * flush (svc_opdef_flush) is only reachable while the pump keeps RE-DECODING + * the entry. When a withheld entry stops being revisited (ring advanced, the + * channel went quiet, the entry was parked while the pump moved on), the + * deadline never runs and the withheld counter release strands forever - + * the guest's tracker word stops short (evtest3 cudaStreamSynchronize(st[2]) + * hangs; opdef=531/8/0 static, no EXPIRED/LOSS lines). Sweep expired slots + * once per service iteration: deliver forward-only to the still-live owner + * channel, count a LOSS when the owner is gone. + */ +/* + * r563b: forced variant of the expiry sweep for the stall self-heal. The + * regular sweep skips slots younger than SVC_OPDEF_HOLD_MS; when the pump + * has already been stuck for SVC_STALL_WATCH_MS, age no longer matters - + * deliver every withheld release forward-only (recording the delivered + * marks so a re-decode cannot re-hold) and clear the slots. This is the + * same delivery the slot's own 3 s bound would perform, just executed at + * the watchdog's discretion. + */ +static void svc_opdef_force_flush(void) +{ + int i; + + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + u8 k; + + if (!e->active || !e->n) + continue; + for (k = 0; k < e->n; k++) + svc_opdef_done_note(e->h_object, e->entry_idx, + &e->rel[k]); + { + static struct svc_channel ch_scratch; + unsigned long flags; + struct svc_channel *local = NULL; + int c; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) + if (svc.chan[c].valid && + svc.chan[c].vgpu == e->vgpu && + svc.chan[c].h_object == e->h_object) { + ch_scratch = svc.chan[c]; + local = &ch_scratch; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (local) + svc_fwd_rel_apply_all(e->vgpu, local, + e->rel, e->n); + } + svc.opdef_expired++; + svc_opdef_clear(e, SVC_OPDEF_CLR_SWEEP); + } +} + +static void svc_opdef_expire_sweep(void) +{ + /* + * struct svc_channel is >25 KB (per-slot op bitmaps); a stack copy + * overflows the 16 KB kernel stack. Only the single service thread + * calls this, so a static scratch is safe (same pattern as + * svc_handle_chan_dump / svc_pump_channels). + */ + static struct svc_channel ch_scratch; + int i; + + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + struct svc_channel *local = NULL; + unsigned long flags; + int c; + + if (!e->active || !e->n) + continue; + if (time_before(jiffies, e->since + + msecs_to_jiffies(SVC_OPDEF_HOLD_MS))) + continue; + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) + if (svc.chan[c].valid && + svc.chan[c].vgpu == e->vgpu && + svc.chan[c].h_object == e->h_object) { + ch_scratch = svc.chan[c]; + local = &ch_scratch; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (local) { + u8 i; + + svc_fwd_rel_apply_all(e->vgpu, local, e->rel, e->n); + for (i = 0; i < e->n; i++) + svc_opdef_done_note(e->h_object, + (u32)e->entry_idx, + &e->rel[i]); + svc.opdef_expired++; + pr_info_ratelimited("sema-opdef: SWEEP-EXPIRED h=0x%x entry=%u n=%u applied forward-only (entry parked past the hold window)\n", + e->h_object, e->entry_idx, e->n); + } else { + svc.opdef_drop_lost += e->n; + pr_info_ratelimited("sema-opdef: SWEEP-LOSS h=0x%x entry=%u n=%u owner channel gone (releases dropped)\n", + e->h_object, e->entry_idx, e->n); + } + svc_opdef_clear(e, SVC_OPDEF_CLR_SWEEP); + } +} + +/* * Proof line for a held record set resolved without a successful forwarded * copy (failure/timeout/abort): the releases below are applied directly, so * none is lost to a hardware copy that never ran. Called with svc_fwd_hold @@ -11515,6 +28922,225 @@ static void svc_fwd_hold_direct_note(void) } /* + * SESSION 127 C5C0 translator: pending GRCE copy-out list. + * + * The Vulkan compute client submits its frame work in the pre-built + * continuation beyond GP_PUT: beec0100 idx23 carries the inline-QMD launch + * (0x31c = 0x4b85f, 64-word 0x320 run) and idx24 carries the TURING_DMA_COPY_A + * (0xC5B5, subch 4) image->buffer copies that un-tile the storage image into + * the client's sysmem readback buffer. Both pushes are only visible to the + * sema_gr_ahead sweep (see svc_gr_ahead_push), not to the put-bounded main + * walk. + * + * Copy ordering: the copy must read the image AFTER the forwarded launch + * completed (the engine's output is copied back into the VA shadow on the + * completion edge). The sweep only records the copy; it is executed inside + * svc_fwd_hold_gr_release, i.e. after the backend published done and after + * the copy-out, immediately before the launch's guest release word is + * authored - so the client's fence never observes a pre-render image. + * A bounded fallback run (sweep, when no forward is owed) keeps the list + * from stalling when the launch was refused. + */ +#define SVC_GR_COPY_PEND 64 +struct svc_gr_copy_pend { + bool valid; + u32 h_object; + u64 push_va; + u64 hash; + u64 src; + u64 dst; + u32 bytes; /* per-row byte count */ + u32 rows; + u32 dpitch; + u32 spitch; + u32 dma; /* LAUNCH_DMA word (remap/transfer bits, logging) */ + u32 remap; /* SET_REMAP_COMPONENTS (logging) */ +}; +static struct svc_gr_copy_pend svc_gr_copy_pends[SVC_GR_COPY_PEND]; +static u32 svc_gr_copy_pend_next; +static u64 svc_gr_copy_run_total; +static u64 svc_gr_copy_run_log; + +/* + * SESSION 153 CONT.25: recently-executed copy identities. The sweep + * re-decodes the ring continuation every pass, so an entry it just ran + * would be re-added (the valid-dedup above only covers live entries) and + * re-run forever - with a bounded per-pass run count that pins the runner + * on the same first slots. The content hash changes whenever the driver + * rewrites the push (new frame: the semaphore payload advances), so a + * (push_va, hash, src, dst, bytes, rows) match means "this exact strip + * already ran this frame". + */ +#define SVC_GR_COPY_DONE_N 512 +struct svc_gr_copy_done_rec { + u64 push_va, hash, src, dst; + u32 bytes, rows; +}; +static struct svc_gr_copy_done_rec svc_gr_copy_done[SVC_GR_COPY_DONE_N]; +static u32 svc_gr_copy_done_next; + +static bool svc_gr_copy_done_lookup(u64 push_va, u64 hash, u64 src, + u64 dst, u32 bytes, u32 rows) +{ + u32 i; + + for (i = 0; i < SVC_GR_COPY_DONE_N; i++) + if (svc_gr_copy_done[i].push_va == push_va && + svc_gr_copy_done[i].hash == hash && + svc_gr_copy_done[i].src == src && + svc_gr_copy_done[i].dst == dst && + svc_gr_copy_done[i].bytes == bytes && + svc_gr_copy_done[i].rows == rows) + return true; + return false; +} + +static void svc_gr_copy_done_store(u64 push_va, u64 hash, u64 src, + u64 dst, u32 bytes, u32 rows) +{ + struct svc_gr_copy_done_rec *d = + &svc_gr_copy_done[svc_gr_copy_done_next++ % + SVC_GR_COPY_DONE_N]; + + d->push_va = push_va; + d->hash = hash; + d->src = src; + d->dst = dst; + d->bytes = bytes; + d->rows = rows; +} + +static void svc_gr_copy_pend_add(u32 h_object, u64 push_va, u64 hash, + u64 src, u64 dst, u32 bytes, u32 rows, + u32 dpitch, u32 spitch, u32 dma, u32 remap) +{ + u32 i; + + if (!bytes || !rows || !src || !dst) + return; + if (svc_gr_copy_done_lookup(push_va, hash, src, dst, bytes, rows)) + return; /* this exact strip already ran for this content */ + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = &svc_gr_copy_pends[i]; + + if (p->valid && p->push_va == push_va && p->src == src && + p->dst == dst && p->bytes == bytes && p->rows == rows) + return; /* already recorded (idempotent re-sweep) */ + } + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = + &svc_gr_copy_pends[svc_gr_copy_pend_next++ % + SVC_GR_COPY_PEND]; + + if (p->valid) + continue; /* keep live ones; take a free slot */ + p->valid = true; + p->h_object = h_object; + p->push_va = push_va; + p->hash = hash; + p->src = src; + p->dst = dst; + p->bytes = bytes; + p->rows = rows; + p->dpitch = dpitch ? dpitch : bytes; + p->spitch = spitch ? spitch : bytes; + p->dma = dma; + p->remap = remap; + pr_info("sema-gr: copy pend h=0x%x push=0x%llx src=0x%llx dst=0x%llx row=0x%x rows=%u dpitch=0x%x spitch=0x%x dma=0x%x remap=0x%x\n", + h_object, push_va, src, dst, bytes, rows, p->dpitch, + p->spitch, dma, remap); + return; + } + /* All slots live: overwrite the oldest position. */ + { + struct svc_gr_copy_pend *p = + &svc_gr_copy_pends[svc_gr_copy_pend_next++ % + SVC_GR_COPY_PEND]; + + pr_warn_ratelimited("sema-gr: copy pend overflow (dropping push=0x%llx)\n", + p->push_va); + memset(p, 0, sizeof(*p)); + } +} + +/* + * Execute and retire every pending GRCE copy. Called from the completion + * edge (svc_fwd_hold_gr_release) and, as a bounded fallback, from the sweep + * when no forward is owed. The executor is the same svc_copy() the CE/GCE + * op path uses (handles vidmem shadow sources and sysmem destinations). + */ +/* + * SESSION 153 CONT.25: bounded per-call run count for the pending + * copy-out list. The X DDX readback keeps at most two strips in flight + * (the two scratch halves), so running the whole queue in one sweep pass + * overruns the server's consumption and every frame reads the newest + * strips only. Pace at a few per pass; 0 = unlimited (historical). + */ +static uint gsp_sema_gr_copy_run_max; +module_param_named(sema_gr_copy_run_max, gsp_sema_gr_copy_run_max, uint, 0644); +MODULE_PARM_DESC(sema_gr_copy_run_max, + "Max pending GRCE copy-outs executed per sweep pass (0 = unlimited)"); + +static void svc_gr_copy_pend_run(struct nv_vgpu_device *vgpu) +{ + u32 i, ran = 0; + + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + if (gsp_sema_gr_copy_run_max && + ran >= gsp_sema_gr_copy_run_max) + break; + struct svc_gr_copy_pend *p = &svc_gr_copy_pends[i]; + u64 done = 0; + int rc; + + if (!p->valid) + continue; + ran++; + rc = svc_copy(vgpu, p->dst, p->src, p->bytes, 0, + SVC_APERTURE_SYS, 0, SVC_APERTURE_SYS, &done, + p->rows, p->dpitch, p->spitch, p->h_object, 0, + p->hash); + svc_gr_copy_run_total++; + if (rc == 0 || rc == SVC_COPY_FORWARDED) { + p->valid = false; + svc_gr_copy_done_store(p->push_va, p->hash, p->src, + p->dst, p->bytes, p->rows); + if (svc_gr_copy_run_log < 64) { + svc_gr_copy_run_log++; + pr_info("sema-gr: copy done h=0x%x push=0x%llx dst=0x%llx bytes=0x%llx rows=%u rc=%d\n", + p->h_object, p->push_va, p->dst, + (u64)p->bytes * p->rows, p->rows, rc); + } + } else if (svc_gr_copy_run_log < 64) { + svc_gr_copy_run_log++; + pr_info("sema-gr: copy retry h=0x%x push=0x%llx dst=0x%llx bytes=0x%llx rc=%d done=0x%llx\n", + p->h_object, p->push_va, p->dst, + (u64)p->bytes * p->rows, rc, done); + } + } +} + +/* + * Is this release VA owned by a live, not-yet-authored launch candidate + * captured from THIS push? (Only the launch's own push's release is + * withheld; a continuation fence entry elsewhere in the ring keeps the + * historical immediate-forward behaviour.) + */ +static bool svc_gr_ahead_owned_rel_va(u64 va, u64 push_va) +{ + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].rel_done && + svc_gr_cand_ring[ri].rel_va == va && + svc_gr_cand_ring[ri].push_va == push_va) + return true; + } + return false; +} + +/* * Author a GR launch completion release (the QMD-level semaphore the * physical engine writes when the launch completes) in software, * forward-only. Used when a hold resolves without a successful hardware @@ -11535,6 +29161,16 @@ static void svc_gr_sw_release(struct nv_vgpu_device *vgpu, u32 h, u64 va, return; if (!svc_sem_read(vgpu, va, &cur) && cur >= payload) return; + /* + * F1-a: this is the hold/fallback authorship of the forwarded launch's + * release word (src="hold"), budgeted with the sweep tag so one + * 256-line session window covers both authors. + */ + if (svc_gr_rel_match(va, NULL) && svc.gr_rel_auth_log < 256) { + svc.gr_rel_auth_log++; + pr_info("sema-diag: GR-REL-AUTHOR src=hold va=0x%llx pay=%u h=0x%x\n", + va, payload, h); + } if (!svc_sem_write(vgpu, h, va, payload)) { svc.fwd_gr_sw_rel++; pr_info("vgsp-work: GR launch release software-applied h=0x%x va=0x%llx pay=0x%x (no successful hardware copy; forward-only)\n", @@ -11552,16 +29188,277 @@ static void svc_fwd_hold_gr_release(struct nv_vgpu_device *vgpu, { if (!svc_fwd_req.gr) return; + /* + * SESSION 127 C5C0 translator: the launch is complete and its output + * pages were copied back into the VA shadow on this edge - run any + * pending GRCE copy-outs (image -> linear readback buffer) BEFORE + * authoring the client's release word, so the fence can never clear + * on a pre-render image. + */ + svc_gr_copy_pend_run(vgpu); svc_gr_sw_release(vgpu, ch ? ch->h_object : 0, svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload); } +/* + * STALLFREE: bounded completion for a failed forwarded GR launch that no hold + * completes. A GR push whose decode recorded no push-level release records + * (svc_fwd_rel_n == 0) engages no release hold, so when the backend fails + * (ladder timeout, fault abort, refusal) nothing authors the QMD completion + * word the guest client polls after the kick: the client's launch wait then + * has no host edge and lasts until the guest's own timeout (the r219/r222 + * stall class, qmd_rel frozen at the previous launch's value). + * + * This runs on the service thread at the top of every pump pass, before any + * request fill of that pass (svc_fwd_request requires SVC_FWD_INFLIGHT free, + * which the work item only releases after publishing done, and the fill's + * first write is done=false), so a pass that sees done also sees this + * request's fields, and every finished request is observed before its slot + * can be reused. Authors the release once (forward-only, full resolver - the + * same write the hold's failure edge performs), latches gr_swrel_done, and + * logs the distinguishable GR-STALL line. + * + * While a hold is active the hold owns the completion (its failure edge + * applies the recorded releases AND the QMD word, bounded by the existing + * hold timeout), so this check stands down for it. + */ + +/* + * r618: synth-fault for a refused/failed GR launch. The r616 experiment + * (manual grfaultpoke on the live r616b matmul wedge - fill `0/5 high + * argument ptr(s)`, `GR submit refused: degenerate data set n_data=2 + * (min 16)`, sw release authored, client spun in cudaDeviceSynchronize + * forever) proved the edge: the channel error-notifier write + * (status=0xffff + channel event) ends the client's wait with + * `CUDA_ERROR unspecified launch failure` in seconds. Deliver the same + * edge automatically on every FAILED forward so a refused launch is an + * observable error, never an unbounded hang. One-line rollback: + * gsp_sema_refusal_fault=0. + */ +bool gsp_sema_refusal_fault = true; +module_param_named(sema_refusal_fault, gsp_sema_refusal_fault, bool, 0644); +MODULE_PARM_DESC(sema_refusal_fault, + "Deliver a synthetic channel error edge on a failed/refused GR forward (default 1; 0 = legacy silent sw release)"); + +/* + * r638: break the CE-watermark/wait-gate mutual citation (see + * svc_wait_held_entry and svc_ce_stream_backlog): a channel whose + * unconsumed head entry is wait-gate held is a consumer-side wait, not a + * producer backlog, and is skipped while the candidate's operands are fully + * resolved and no decoded device write is pending on a data page. Rollback + * to the r636 behaviour: sema_mutual_skip=0. + */ +bool gsp_sema_mutual_skip = true; +module_param_named(sema_mutual_skip, gsp_sema_mutual_skip, bool, 0644); +MODULE_PARM_DESC(sema_mutual_skip, + "Skip a wait-held consumer head in the CE stream-order watermark (default 1; 0 = r636 behaviour)"); + +static void svc_gr_fault_synth(struct nv_vgpu_device *vgpu, u32 h_object) +{ + u8 notif[SVC_NOTIFY_SIZE] = {0}; + u16 status = SVC_NOTIFY_ERROR_STATUS; + u32 info32 = 0x1; /* generic engine exception, as proven live */ + unsigned long flags; + struct svc_channel *ch = NULL; + u64 pa; + u32 as, eng; + int i; + + if (!gsp_sema_refusal_fault || !h_object) + return; + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == h_object && + svc.chan[i].notifier_pa) { + ch = &svc.chan[i]; + break; + } + } + if (!ch) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + return; + } + pa = ch->notifier_pa; + as = ch->notifier_as; + eng = ch->engine_type; + ch->notifier_err = true; + ch->notifier_err_until = jiffies + SVC_NOTIFIER_ERR_HOLD; + ch->notifier_err_put = ch->put_prev; + spin_unlock_irqrestore(&svc.chan_lock, flags); + + memcpy(notif + SVC_NOTIFY_INFO32_OFF, &info32, 4); + { + u16 info16 = (u16)eng; + + memcpy(notif + SVC_NOTIFY_INFO16_OFF, &info16, 2); + } + memcpy(notif + SVC_NOTIFY_STATUS_OFF, &status, 2); + if (svc_write_ap(vgpu, pa, as, notif, sizeof(notif))) { + pr_info_ratelimited("vgsp-fault: SYNTH-FAULT notifier write failed ch=0x%x pa=0x%llx as=%u\n", + h_object, pa, as); + return; + } + pr_warn("vgsp-fault: SYNTH-FAULT ch=0x%x engine=%u pa=0x%llx status=0x%x except=0x%x (refused forwarded launch -> client-visible error)\n", + h_object, eng, pa, status, info32); + gsp_rm_post_channel_event(vgpu, h_object); +} +static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) +{ + bool done = smp_load_acquire(&svc_fwd_req.done); + unsigned long waited; + + if (!svc_fwd_req.gr || svc_fwd_req.vgpu != vgpu || + svc_fwd_req.gr_swrel_done || svc_fwd_hold.active || !done) + return; + /* + * Complete when the backend failed (the never-completing r219/r222 + * class) OR when a successful launch's release0 could not target the + * guest page, so the engine's completion landed only on the module + * semaphore and the client's QMD word is still behind (the same edge + * the hold's success path covers for held requests). + */ + if (svc_fwd_req.hw_rc == GSP_PHYS_GR_RC_MISSABORT) { + /* + * SESSION 153-CONTINUED-61: MISS-ABORT without a hold - keep + * the candidate armed for a real re-forward and hold the + * client's release back (it completes on the retried + * launch's own edges). Only the expired episode falls + * through to the historical software completion below. + */ + if (svc_gr_missre_arm(NULL, 0)) { + svc_fwd_req.gr_swrel_done = true; + return; + } + } + /* + * SESSION 153-CONTINUED-61: a tick-queued retried launch carries no + * entry hold, so its per-push release records survive in the park + * table (re-parked by svc_gr_missre_tick). Deliver them here on ANY + * final outcome - success, fault or the expired retry episode - so + * the client's tracking words always complete. Exactly-once: the + * park entry is dropped on delivery and the latch below stops any + * later pass from re-running this outcome. + */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + static struct svc_channel shim; /* single-threaded service + * path; BSS keeps the kernel + * stack safe */ + u8 pn = 0; + + if (svc_grpark_get(vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va, parked, &pn)) { + memset(&shim, 0, sizeof(shim)); + shim.valid = true; + shim.h_object = svc_fwd_req.gr_h_object; + svc_gr_copy_pend_run(vgpu); + svc_fwd_rel_apply_all(vgpu, &shim, parked, pn); + svc_grpark_drop(vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va); + svc.fwd_rel_applied++; + pr_info("vgsp-gr: MISS-RETRY records delivered h=0x%x push=0x%llx nrel=%u (no-entry retry completion)\n", + svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va, (unsigned)pn); + } + } + if (svc_fwd_req.hw_rc == 0) { + svc_gr_missre_done(); + if (!svc_fwd_req.gr_rel_unobserved) { + svc_fwd_req.gr_swrel_done = true; + return; + } + } + /* + * The outcome is final; latch it first so no later pass, hold edge or + * consume tail accounts it again (the release write itself is + * forward-only, so even a duplicate could not corrupt the word). + */ + svc_fwd_req.gr_swrel_done = true; + waited = jiffies_to_msecs(jiffies - svc_fwd_req.gr_fill_jiffies); + if (!svc_fwd_req.gr_rel_va || !svc_fwd_req.gr_rel_payload) { + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (launch carries no release record; no word to author)\n", + svc_fwd_req.gr_push_va, waited); + return; + } + /* + * SESSION 127 C5C0 translator: this edge authors the guest release + * without a hold - run any pending GRCE copy-outs first, exactly like + * the hold's success edge, so the fence cannot clear on a pre-render + * image. + */ + svc_gr_copy_pend_run(vgpu); + svc_gr_sw_release(vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_rel_va, + svc_fwd_req.gr_rel_payload); + svc.fwd_gr_stall_swrel++; + if (svc_fwd_req.hw_rc == -EBADMSG) { + /* + * SESSION 153-CONTINUED-60: the backend skipped an all-zero 3D + * window (ZEROWIN-SKIP) - nothing was submitted, so there is no + * engine error to report. Author the software release and do + * NOT raise the synthetic fault: the client sees a clean + * completion for a frame whose window had no content, and the + * engine is never handed the zero method stream that used to + * poison it (trap 0x1168 -> grfifo=0xbadf1201, r1107). + */ + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (zero window skipped; QMD release authored in software, no synth fault)\n", + svc_fwd_req.gr_push_va, waited); + } else if (svc_fwd_req.hw_rc != 0) { + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (failed forwarded launch, rc=%d; QMD release authored in software)\n", + svc_fwd_req.gr_push_va, waited, svc_fwd_req.hw_rc); + /* + * r618: the software release alone completes only launches the + * engine actually ran; a refused/failed one left the client in + * an unbounded cudaDeviceSynchronize. Deliver the synthetic + * engine-error edge (proven live by the r616 grfaultpoke test) + * so the failed launch is client-observable. + */ + svc_gr_fault_synth(vgpu, svc_fwd_req.gr_h_object); + } else + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (successful launch whose release0 could not target the guest page; QMD release authored in software)\n", + svc_fwd_req.gr_push_va, waited); + pr_info("vgsp-gr: GR-COMPLETE-SW h=0x%x push=0x%llx rel=0x%llx pay=0x%x waited_ms=%lu n=%llu (backend outcome without a hold)\n", + svc_fwd_req.gr_h_object, svc_fwd_req.gr_push_va, + svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload, waited, + svc.fwd_gr_stall_swrel); +} + static bool svc_fwd_hold_expired(void) { - if (svc_fwd_req.gr && !smp_load_acquire(&svc_fwd_req.done)) + if (svc_fwd_req.gr && !smp_load_acquire(&svc_fwd_req.done)) { + /* + * r420: while the backend deliberately holds the in-flight + * launch in its bounded WEDGE-QUIET window, the GR deadline + * must not fire - expiring here authors the launch's release + * word (svc_fwd_hold_gr_release) and applies the recorded + * releases before the post-hold copy-out, so the client + * verifies stale data (the r419 soak's mid-boot + * "matmul fails after time"). Pause instead, bounded by + * SVC_FWD_GR_HOLD_PAUSE_MAX_MS from the hold's start; the + * pause extends the deadline a full GR window past the last + * quiet-hold pass, so the backend's retry/accept can finish + * before the fallback would author. Announced once per hold. + */ + if (gsp_phys_gr_quiet_hold_active() && + time_before(jiffies, svc_fwd_hold.since + + msecs_to_jiffies(SVC_FWD_GR_HOLD_PAUSE_MAX_MS))) { + if (!svc_fwd_hold.pause_logged) { + svc_fwd_hold.pause_logged = true; + pr_warn("vgsp-gr: GR-HOLD-PAUSE va=0x%llx (backend wedge-quiet hold active; release authorship deferred under the bounded hold)\n", + svc_fwd_req.gr_push_va); + } + svc_fwd_hold.paused_until = jiffies + + msecs_to_jiffies(SVC_FWD_GR_HOLD_TIMEOUT_MS); + return false; + } + if (svc_fwd_hold.paused_until && + time_before(jiffies, svc_fwd_hold.paused_until)) + return false; return time_after(jiffies, svc_fwd_hold.since + msecs_to_jiffies(SVC_FWD_GR_HOLD_TIMEOUT_MS)); + } return time_after(jiffies, svc_fwd_hold.since + msecs_to_jiffies(SVC_FWD_HOLD_TIMEOUT_MS)); } @@ -11595,6 +29492,22 @@ static void svc_fwd_hold_engage(struct nv_vgpu_device *vgpu, svc_fwd_hold.relay_armed = svc_fwd_req.relay; svc_fwd_hold.relay_index = svc_fwd_req.relay_index; svc_fwd_hold.fault_delivered = false; + /* + * r633 diag: the held record set, so a later release-delivery gap + * (the streams2 spin class) can be told apart from a hold that + * engaged empty / with another entry's records. Bounded. + */ + if (svc_grpark_log < 64) { + svc_grpark_log++; + pr_info("vgsp-work: hold engage h=0x%x entry=%u nrel=%u va0=0x%llx pay0=%u gr=%d\n", + ch->h_object, entry_idx, n, + n ? svc_fwd_hold.rel[0].va : 0ull, + n ? svc_fwd_hold.rel[0].payload : 0u, + svc_fwd_req.gr ? 1 : 0); + } + /* r420: a fresh hold starts with no WEDGE-QUIET deferral. */ + svc_fwd_hold.paused_until = 0; + svc_fwd_hold.pause_logged = false; svc.fwd_hold_engaged++; svc.fwd_rel_pending = n; } @@ -11621,6 +29534,8 @@ static void svc_fwd_hold_clear(void) svc_fwd_hold.relay_armed = false; svc_fwd_hold.relay_index = 0; svc_fwd_hold.fault_delivered = false; + svc_fwd_hold.paused_until = 0; + svc_fwd_hold.pause_logged = false; svc.fwd_rel_pending = 0; } @@ -11765,6 +29680,20 @@ static void svc_fwd_hold_expire_check(struct nv_vgpu_device *vgpu, done = smp_load_acquire(&svc_fwd_req.done); hv = svc_fwd_hold.vgpu ? svc_fwd_hold.vgpu : vgpu; /* + * SESSION 153-CONTINUED-61: a MISS-ABORT hold expiring with the + * outcome known takes the re-arm edge instead of the software + * completion (the same verdict svc_fwd_hold_poll applies); only an + * episode past its bound falls through to the historical authorship + * below. + */ + if (done && svc_fwd_req.gr && + svc_fwd_req.hw_rc == GSP_PHYS_GR_RC_MISSABORT && + svc_gr_missre_arm(svc_fwd_hold.rel, svc_fwd_hold.nrel)) { + svc_fwd_req.gr_swrel_done = true; + svc_fwd_hold_clear(); + return; + } + /* * P6c C3: a relayed hold whose work item already published a timeout * gets the decoded engine fault delivered before the software releases * land (the late-release recheck inside the helper protects against a @@ -11774,6 +29703,15 @@ static void svc_fwd_hold_expire_check(struct nv_vgpu_device *vgpu, svc_fwd_hold.relay_armed) svc_fwd_fault_deliver(hv, ch); /* + * STALLFREE: the unpumpable-channel expiry edge (the entry loop never + * ran for this hold). The releases and the QMD completion word are + * software-applied below; exactly one line per hold. + */ + if (svc_fwd_req.gr) + pr_warn("vgsp-gr: GR-STALL-fwd-hold va=0x%llx waited_ms=%u action=proceed (hold expired on an unpumpable channel; releases + QMD word software-applied)\n", + svc_fwd_req.gr_push_va, + jiffies_to_msecs(jiffies - svc_fwd_hold.since)); + /* * P6b wave 2: a skipped candidate whose hardware copy did not complete * successfully is authored by one software copy here, BEFORE the * recorded releases below. A successful hardware copy needs none. @@ -11827,6 +29765,7 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, if (!done && !expired && !aborted) return false; if (done && svc_fwd_req.hw_rc == 0) { + svc_gr_missre_done(); /* * P6c: on the relay success edge, re-read the guest-visible * semaphore through the software view. Only a word that @@ -11874,6 +29813,23 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, svc_fwd_hold.nrel); svc.fwd_rel_applied++; } + } else if (done && svc_fwd_req.gr && + svc_fwd_req.hw_rc == GSP_PHYS_GR_RC_MISSABORT && + svc_gr_missre_arm(svc_fwd_hold.rel, svc_fwd_hold.nrel)) { + /* + * SESSION 153-CONTINUED-61: the launch aborted pre-doorbell + * because a window operand's page is still unresolved. Do NOT + * software-complete it: the candidate is re-armed for a real + * re-forward once the pin demand resolves the page (the retry + * tick re-offers it) and the client's releases stay unauthored + * (the retried launch's own edges author them on success). + * The attempt's push-level records were re-parked by the arm. + * Bounded by the episode deadline; an expired episode falls + * into the historical failure path below. gr_swrel_done + * latches the request outcome so the pass-top check cannot + * author the release later. + */ + svc_fwd_req.gr_swrel_done = true; } else { /* * P6b wave 2: failure/timeout/abort fallback - author the @@ -11884,9 +29840,36 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, * done && hw_rc == -ETIMEDOUT; abort/cancel paths never write * guest memory). */ + /* + * C7 (GAP-2 A1): the false-completion edge - a hold whose + * hardware copy failed/timed out still software-applies the + * release set here. + */ + if (gsp_sema_dbg && svc_hold_fb_log < 64) { + svc_hold_fb_log++; + pr_info("sema-diag: hold-fallback h=0x%x gr=%d done=%d hw_rc=%d nrel=%u rel0=0x%llx/%u -> sw apply\n", + ch->h_object, ch->engine_type == 0 ? 1 : 0, + done ? 1 : 0, svc_fwd_req.hw_rc, + (unsigned)svc_fwd_hold.nrel, + svc_fwd_hold.nrel ? + svc_fwd_hold.rel[0].va : 0ull, + svc_fwd_hold.nrel ? + svc_fwd_hold.rel[0].payload : 0u); + } if (done && svc_fwd_req.hw_rc == -ETIMEDOUT && svc_fwd_hold.relay_armed) svc_fwd_fault_deliver(hv, ch); + /* + * STALLFREE: the bounded failure edge of a held forwarded GR + * launch - the releases AND the QMD completion word are + * software-applied below, so the client always completes. + * Exactly one line per hold (the hold is cleared at the end). + */ + if (svc_fwd_req.gr) + pr_warn("vgsp-gr: GR-STALL-fwd-fail va=0x%llx waited_ms=%u action=proceed (hold resolved on backend failure/expiry; releases + QMD word software-applied)\n", + svc_fwd_req.gr_push_va, + jiffies_to_msecs(jiffies - + svc_fwd_hold.since)); svc_fwd_hold_soft_copy(hv); if (svc_fwd_hold.relay_armed) svc.fwd_relay_fallback++; @@ -11912,6 +29895,14 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, ch->last_get = (idx + 1) % ch->gpfifo_entries; ch->rescue_fail = 0; ch->defer_since = 0; + /* + * GRFIX FIX1c (r214): do NOT reset the per-launch bound record + * here. This consume can belong to a different entry than the held + * launch, and resetting on every consume re-armed the bound before + * it could pass (r214: 420 re-holds, GRFILL-EXPIRED=0 over 220 s). + * The record is keyed by the launch push VA; it resets only on + * channel (re)registration or the session diag reset. + */ ch->submissions++; svc.gpfifo_entries_done++; if (ch->engine_type && ch->expected_sem_payload) @@ -12101,6 +30092,10 @@ static void svc_fwd_gr_work(void) struct gsp_phys_backend_copy_result cp = { 0 }; u64 rel_hpa = 0; u32 off = 0, i; + /* GRVAUDIT: aggregated from the data[]-build loop below (first + * problem page in ga_va/ga_used/ga_led; 0 while clean). */ + u32 ga_unlanded = 0, ga_ptebad = 0; + u64 ga_va = 0, ga_used = 0, ga_led = 0; int hw_rc; memset(&gs, 0, sizeof(gs)); @@ -12133,6 +30128,7 @@ static void svc_fwd_gr_work(void) (SEMA_PAGE_SIZE - 1)); u8 rel_kind = (svc_fwd_req.gr_rel_ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + bool rel_unobserved = false; if (svc_fwd_req.gr_rel_pa && (svc_fwd_operand_hpa(svc_fwd_req.vgpu, rel_kind, @@ -12145,11 +30141,44 @@ static void svc_fwd_gr_work(void) * means the engine would author nothing this launch; point * release0 at the module semaphore instead so the host still * observes completion. + * + * STALLFREE: when the guest page could not be targeted at all + * (unresolved release PA or HPA), the engine's completion will + * land only on the module semaphore and the guest QMD word + * stays behind - a launch whose client would otherwise wait + * forever even on a SUCCESSFUL submit. Record it so the pump's + * bounded outcome check authors the word in software (the + * same write the hold's success edge performs). A resolved + * page whose word is already ahead needs nothing. */ - if (rel_hpa && - READ_ONCE(*(u32 *)((u8 *)__va(rel_hpa) + roff)) >= + if (rel_hpa) { + if (READ_ONCE(*(u32 *)((u8 *)__va(rel_hpa) + roff)) >= svc_fwd_req.gr_rel_payload) - rel_hpa = 0; + rel_hpa = 0; + } else { + rel_unobserved = true; + } + svc_fwd_req.gr_rel_unobserved = rel_unobserved; + /* + * r633 diag: the release0 resolution chain, so a redirect + * ("release0 could not target the guest page") names its cause + * (rel_pa unresolved at capture vs HPA unresolved here vs the + * forward-only zeroing). Bounded. + */ + { + static u32 gr_relhpa_log; + + if (gr_relhpa_log < 32) { + gr_relhpa_log++; + pr_info("vgsp-work: gr-rel0 va=0x%llx pa=0x%llx ap=%u hpa=0x%llx off=%u word=%08x pay=%u unobs=%d\n", + svc_fwd_req.gr_rel_va, + svc_fwd_req.gr_rel_pa, + svc_fwd_req.gr_rel_ap, rel_hpa, roff, + rel_hpa ? READ_ONCE(*(u32 *)((u8 *)__va(rel_hpa) + roff)) : 0, + svc_fwd_req.gr_rel_payload, + rel_unobserved ? 1 : 0); + } + } } gs.qmd = svc_fwd_req.gr_qmd; @@ -12175,16 +30204,129 @@ static void svc_fwd_gr_work(void) u64 dhpa = 0; u32 doff = 0; - if (!dg->pa) - continue; - if (svc_fwd_operand_hpa(svc_fwd_req.vgpu, dkind, dg->pa, 0, - SEMA_PAGE_SIZE, &dhpa, &doff) || doff) - continue; + if (dg->shadow) { + /* + * r632: the page was served from the VA shadow lane + * (no resolver and no landing answered for it). The + * shadow page is the page's storage: hand its host + * address to the bounce machinery exactly like a + * resolved guest page. The copy-in stages the client's + * own bytes (H2D'd or zeroed) from it, and the + * copy-out writes the engine's results back into it - + * the same page the D2H copies read through the + * shadow-first rule. Module-owned memory throughout; + * no guest page is involved. + */ + struct svc_vas_rec *vr = svc_vas_find(dg->va); + + if (!vr || !vr->data) + continue; + dhpa = (u64)virt_to_phys(vr->data) & + ~(u64)(SEMA_PAGE_SIZE - 1); + dkind = SVC_APERTURE_SYS; + } else { + if (!dg->pa) + continue; + if (svc_fwd_operand_hpa(svc_fwd_req.vgpu, dkind, + dg->pa, 0, SEMA_PAGE_SIZE, + &dhpa, &doff) || doff) + continue; + } if (gs.n_data >= SVC_GR_DATA_MAX) break; data[gs.n_data].va = dg->va; data[gs.n_data].hpa = dhpa; data[gs.n_data].reloc_va = dg->reloc_va; + /* + * r348 diagnostic: the last 32 high-VA data pages (the output + * operands live there) with the exact (gpa, ap) -> hpa the + * writeback will use - comparable against the D2H read's + * resolution line (sema-bigcopy) for the same VA. + */ + if (dg->va >= SVC_VAS_FLOOR && + i + 32 >= svc_fwd_req.gr_n_data) + pr_info("svc-fwd-hpa: va=0x%llx gpa=0x%llx ap=%u hpa=0x%llx bi=%u\n", + dg->va, dg->pa, dg->ap, dhpa, i); + /* + * GRVAUDIT (diagnostic only): audit the m180 landing + * authority against the PA this submit will put on the wire + * for every tracked data page - the data[]/n_data set the + * backend's NOWRITE analysis later enumerates. The lookup + * runs BEFORE the store underneath re-records this launch's + * own landing, so it reads the pre-existing authority: + * unlanded = no landing recorded for this VA at submit; + * pte_bad = a landing exists on a different page/aperture + * than the PA the submit path computed/used. Pages below + * SVC_VAS_FLOOR are outside the ledger's domain (all store + * sites gate on the floor) and are not classified. + */ + if (dg->va >= SVC_VAS_FLOOR) { + u64 lpa = 0; + u32 lap = 0; + + if (svc_ledger_lookup(dg->va, &lpa, &lap)) { + ga_unlanded++; + if (!ga_va) { + ga_va = dg->va; + ga_used = dg->pa; + ga_led = 0; + } + } else if ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dg->pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != (u32)dkind) { + ga_ptebad++; + if (!ga_va) { + ga_va = dg->va; + ga_used = dg->pa; + ga_led = lpa; + } + } + } + /* + * r233 READ-DIVERGE-WIT: the engine's read of this VA uses + * exactly what the fill resolved (dg->pa: the page whose + * content was copied into the bounce). The copy paths (H2D + * writes, D2H reads) resolve independently through + * svc_op_translate/ledger. For small launches (the probe + * class) compare the three for the first recs and flag any + * page-level split - the direct witness for a read/write + * permutation. Bounded: first 8 recs, 4 lines per launch, + * boot budget 128. + */ + if (svc_fwd_req.gr_n_data <= 1024u && i < 8u && + svc_readdiv_log < 128u) { + u64 opa = 0, lpa2 = 0; + u32 oap = 0, lap2 = 0; + bool op_miss, have_led; + + op_miss = svc_op_translate(svc_fwd_req.vgpu, dg->va, + &opa, &oap) != 0; + have_led = !svc_ledger_lookup(dg->va, &lpa2, &lap2); + if (op_miss || (!op_miss && + (opa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dg->pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (have_led && + (lpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dg->pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + svc_readdiv_log++; + pr_info("vgsp-gr: READ-DIVERGE-WIT va=0x%llx fill_pa=0x%llx op_pa=0x%llx op_src=%u op_rc=%d ledger_pa=0x%llx%s\n", + dg->va, dg->pa, opa, + op_miss ? 0xffffffffu : svc.op_tr_src, + op_miss ? -ENOENT : 0, + have_led ? lpa2 : 0, + op_miss ? " (op-miss)" : ""); + } + } + /* + * C3 (data class): after the run the backend copies this + * page's results back to dg->pa (gsp_phys_booter.c copy-out). + * Record that page as the truth for this VA so the guest's + * D2H read of an engine-authored output follows the page the + * results actually landed on, whatever a later resolver says + * (atomics: engine ran, readback read a different page). + */ + if (dg->va >= SVC_VAS_FLOOR) + svc_ledger_store(dg->va, dg->pa, dkind); gs.n_data++; } gs.data = data; @@ -12204,6 +30346,12 @@ static void svc_fwd_gr_work(void) gs.rel_hpa = rel_hpa; gs.rel_off = (u32)(svc_fwd_req.gr_rel_va & (SEMA_PAGE_SIZE - 1)); gs.rel_payload = svc_fwd_req.gr_rel_payload; + /* + * r378: carry the release VA so the backend can re-validate the + * release-word write targets (zero-on-retry, restore-on-accept) + * through gsp_sema_gr_write_guard before they land. + */ + gs.rel_va = svc_fwd_req.gr_rel_va; gs.sem_hpa = 0; gs.sem_off = 0; gs.sem_payload = 0; @@ -12213,9 +30361,54 @@ static void svc_fwd_gr_work(void) gs.raw_words = svc_fwd_req.gr_push_n; gs.raw_qmd_hdr = svc_fwd_req.gr_push_qmd_hdr; gs.raw_qmd_end = svc_fwd_req.gr_push_qmd_end; + /* + * r929 (Path A): a 3D stream capture replays its raw window verbatim; + * the backend emits SET_OBJECT C597 + the window + the release tail + * instead of the compute prelude/QMD. The window is sliced from the + * section's C597 bind; see svc_gl_flush. + */ + if (svc_fwd_req.gr_stream_3d) { + gs.stream = svc_fwd_req.gr_raw + svc_fwd_req.gr_stream_off; + gs.stream_n = svc_fwd_req.gr_stream_n; + } + gs.prog_va = svc_fwd_req.gr_prog_va; /* CONT-87: family binds */ + gs.n_prog_binds = svc_fwd_req.gr_n_prog_binds; + { + u32 pb; + + for (pb = 0; pb < svc_fwd_req.gr_n_prog_binds && pb < 8u; + pb++) { + gs.prog_bind_va[pb] = + svc_fwd_req.gr_prog_bind_va[pb]; + gs.prog_bind_slot[pb] = + svc_fwd_req.gr_prog_bind_slot[pb]; + } + } gs.h_object = svc_fwd_req.gr_h_object; + /* + * GRVAUDIT: one line per launch (n_data matches the backend + * NOWRITE line's req->n_data); nonzero counts always log, an + * all-clean audit logs once per 50 launches. + */ + svc_grvaudit_launches++; + if (ga_unlanded || ga_ptebad || !(svc_grvaudit_launches % 50u)) + pr_info("vgsp-gr: GRVAUDIT n_data=%u unlanded=%u pte_bad=%u sample_va=0x%llx pa_used=0x%llx ledger_pa=0x%llx\n", + gs.n_data, ga_unlanded, ga_ptebad, + ga_va, ga_used, ga_led); + hw_rc = gsp_phys_backend_guest_gr_submit(&gs, &cp); + /* + * r420b: the backend redirected the launch's QMD release0 to the + * module semaphore (rel_mod=1 or no targetable guest release page), so + * the ENGINE never writes the guest's own QMD release word. Treat the + * word as unobserved so the bounded outcome check (and the hold's + * edges) author it in software on every outcome; without this a + * rel_mod=1 launch with no deferred release records is never completed + * on the guest word and the client hangs (the r414 saxpy stall). + */ + if (cp.qmd_rel_redirected) + svc_fwd_req.gr_rel_unobserved = true; if (!hw_rc) { svc.fwd_gr_ok++; } else { @@ -12486,12 +30679,16 @@ static void svc_fwd_relay_select(struct svc_fwd_req *req) * svc_gr_req_fill from its QMD pointer association and the captured inline * bytes. Overflow is a failure (fail-closed), not a silent drop. */ +static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out); static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u64 pa = 0; u32 ap = 0; u32 i; + /* r372 diagnostic: which branch resolved this launch page. */ + const char *src = "?"; if (!page) return; @@ -12510,10 +30707,158 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) i = svc_fwd_req.gr_n_pages++; svc_fwd_req.gr_pages[i].va = page; svc_fwd_req.gr_pages[i].qmd_word = qmd_word; - if (svc_resolve_va(vgpu, page, &pa, &ap)) - return; + /* + * m181 (relocation/ledger unification): the unified landing table is + * the authority for a page a host write already landed on. The CE + * inline writes that author these launch pages (param, program, + * constant banks, inline destinations) go through + * svc_grce_inline_write(), which records its landing. The backend's + * QMD relocation reads the page content through pg->pa (win0 shadow + * copy) and the m129 mirror patches that same page, so resolving + * independently here could hand the engine a stale page whose + * pointer values are not this launch's VAs: the relocation then + * matches nothing, the engine executes against unrelocated guest VAs + * it cannot walk, and the grid aborts silently (zero stores, ack + * still delivered) - the cross-session zero-write class - or runs on + * the previous client's parameters (the multi-page wrongness). + * Resolve once, from the table, and record the independent answer as + * stale when it differs. + */ + { + u64 lpa = 0; + u32 lap = 0; + + /* + * r372 pin-wins: a live injected pin is the guest's newest + * statement about this VA. If it disagrees with the landing + * table, the pin wins and the table is repaired (see the + * PIN-REPAIR note in svc_handle_map_add): the old landing was + * a freed page, and resolving the launch there wrote engine + * output onto memory the guest kernel had reused. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(page, &ipa, &iap)) { + u64 lpa2 = 0; + u32 lap2 = 0; + + if (svc_ledger_lookup(page, &lpa2, &lap2) || + ((lpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (ipa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap2 != (u32)iap) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS note va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + page, lpa2, lap2, ipa, iap); + svc_ledger_store(page, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + } + svc_fwd_req.gr_pages[i].pa = + ipa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)iap; + svc_pin_store(page, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + iap); + src = "ipin"; + goto note_done; + } + } + if (!svc_ledger_lookup(page, &lpa, &lap)) { + u64 rpa = 0; + u32 rap = 0; + const char *rsrc = svc_gr_src_probe(vgpu, page, &rpa, + &rap); + + if (rpa && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (rpa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != (u32)rap)) { + svc_ledger_reloc_diverges++; + if (svc_ledger_reloc_diverges <= 16) + pr_info("sema-ledger: RELOC-DIVERGE va=0x%llx ledger_pa=0x%llx %s_pa=0x%llx (ledger wins)\n", + page, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + rsrc, rpa & + ~(u64)(SEMA_PAGE_SIZE - 1)); + } + svc_fwd_req.gr_pages[i].pa = + lpa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)lap; + /* + * Keep the pin table consistent with the authority so + * the win0 copy path and later resolves agree (and no + * stale self-pin is planted for this VA). + */ + svc_pin_store(page, lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)lap); + pa = lpa; + ap = lap; + src = "ledger"; + goto note_done; + } + } + if (svc_resolve_va(vgpu, page, &pa, &ap)) { + /* + * SESSION 127 C5C0 translator (gated): a client-VAS page. The + * Vulkan + * QMD's program, parameter and descriptor pages live in the + * client's own page tables (vidmem aperture) and the trusted + * sources - pins, learned cache, discovered map - do not + * carry them, which made the fill refuse the launch with + * -ENOENT. Resolve through the registered-VAS walk instead: + * the exact path walk_probe proves (r707w). + */ + u64 wpa = 0; + u32 wap = SVC_APERTURE_SYS; + + if (!gsp_sema_gr_xlate || + svc_walk_vaspaces(vgpu, page, &wpa, &wap)) + return; + pa = wpa; + ap = svc_ap_of(wap); + svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); + svc_ledger_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)svc_ap_of(ap)); + svc_pin_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)svc_ap_of(ap)); + src = "vas"; + goto note_done; + } svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); + /* + * r370: store the landing table for EVERY launch page, not just the + * data-fill spans. An indirect/bank operand (the kernel's output + * buffer, qmd_word NONE) was resolved HERE, the engine wrote it via + * this hpa, the copy-out wrote it back to this hpa - and the client's + * D2H readback then resolved the same VA through the ledger/pin to a + * DIFFERENT page (an older landing), read stale zeros and the kernel + * looked wrong (victory_reduce sum=0 with the correct sum proven in + * the bounce page by OUT-PAGE-WIT). Recording this resolution makes + * the readback read exactly the page the engine wrote. + */ + svc_ledger_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)svc_ap_of(ap)); + svc_pin_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)svc_ap_of(ap)); + src = "walk"; +note_done: + /* + * r372 diagnostic: the exact (pa, ap) and branch the launch page + * resolved through - the value the engine's hpa and the copy-out use. + * A D2H readback that lands elsewhere (the sum=0 class) names its + * disagreement against these lines. + */ + if (svc_gr_pagenote_log < 512) { + svc_gr_pagenote_log++; + pr_info("sema-gr: page-note va=0x%llx pa=0x%llx ap=%u src=%s qmd=%u\n", + page, svc_fwd_req.gr_pages[i].pa, + svc_fwd_req.gr_pages[i].ap, src, qmd_word); + } } /* Does any captured inline write land in this page? */ @@ -12526,6 +30871,17 @@ static bool svc_gr_page_inline_covers(u64 page) ~(u64)(SEMA_PAGE_SIZE - 1)) == page) return true; } + /* + * SESSION 153-CONTINUED-91: the request's records may have been + * re-rooted onto this launch's bind targets (svc_gr_inline_retarget), + * so a page the request names can be covered by an inline write even + * when the candidate's original record page differs. + */ + for (i = 0; i < svc_fwd_req.gr_n_inline; i++) { + if ((svc_fwd_req.gr_inline[i].dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1)) == page) + return true; + } return false; } @@ -12575,11 +30931,57 @@ static bool svc_gr_page_inline_covers(u64 page) * reach the engine. The legacy value is the pre-B-2 window below the mail * page (GSP_PHYS_RMCALL_GR_MAIL_SLOT = 206). */ -#define SVC_GR_DATA_BOUNCE_MAX GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES +/* + * fail4: each launch may relocate up to half of the grown region - the + * other half is the alternate base of a concurrently in-flight launch + * (see the per-capture base flip in svc_gr_data_fill). + */ +#define SVC_GR_DATA_BOUNCE_MAX (GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / 2u) #define SVC_GR_DATA_BOUNCE_MAX_OLD 205u +/* + * SESSION 153-CONTINUED-63: render-target page tracking (see + * gsp_sema_surf_va_get above). The captured colour target is seeded as a + * tracked data candidate so (a) GR-DEC n_data >= 1, (b) the retire wait + * counts the engine's stores into the target ("N/N pages changed") and + * (c) the copy-out ships the changed pages back to the guest's own screen + * pages. Bounded to one full 1280x720 RGB surface plus slack; the + * completeness record (va + n) arms the booter's rec-over-alias operand + * preference only when EVERY page resolved, so a partial span can never + * hand the engine a base pointer walking past the tracked run. + */ +#define SVC_GR_SURF_TRACK_MAX 1024u +static u64 svc_gr_surf_rec_va; +static u32 svc_gr_surf_rec_n; +static u32 svc_gr_surf_rec_ok; +static u32 svc_gr_surf_rec_log; + +bool gsp_sema_gr_surf_recs(u64 *va, u32 *n) +{ + if (!READ_ONCE(svc_gr_surf_rec_ok)) + return false; + if (va) + *va = svc_gr_surf_rec_va; + if (n) + *n = svc_gr_surf_rec_n; + return true; +} + static u32 svc_gr_data_log; /* m142: bounded per-page attribution log */ static u32 svc_gr_src_log; /* m150: bounded per-candidate source log */ +static u32 svc_gr_shadow_fill_log; /* r632: bounded shadow-serve log */ +/* + * r632 (the red-draw resolution class): caps on SYNTHETIC shadow pages a + * launch may create for argument candidates whose VAs no resolver answers + * (the output-buffer class). Existing shadow content (the client's own H2D + * writes) serves without a quota. The per-candidate cap bounds how far a + * span may guess past a buffer's true end (the first page past the cap ends + * the span through the historical drop path); the per-launch cap keeps the + * bounce window (16384 pages) safe. The real output operands are argument + * candidates and are collected first. + */ +#define SVC_GR_SHADOW_SYNTH_PER_CAND 1024u +#define SVC_GR_SHADOW_SYNTH_MAX 4096u static bool svc_gr_data_stalled; /* m150: high data args unresolved */ /* * m156 (R1): the GR request was refused because a forward was already @@ -12590,12 +30992,17 @@ static bool svc_gr_data_stalled; /* m150: high data args unresolved */ */ static bool svc_gr_busy_defer; static u64 svc_gr_hold_log; /* m150: bounded GR-hold proof lines */ +static u64 svc_gr_drop_log_va; /* GRFIX: launch whose dropped-page detail + * (GRFILL-DROP) was already logged - the + * detail prints once per launch episode, + * not once per ~100 ms re-decode */ /* m150: re-arm both fill diagnostics for each client session. */ static void svc_gr_diag_reset(void) { svc_gr_data_log = 0; svc_gr_src_log = 0; + svc_gr_drop_log_va = 0; } /* @@ -12661,6 +31068,265 @@ static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, } /* + * relfix PART 3: append one attempt to the resolver's diagnostic chain + * ("name:rc,"). Bounded; the tail is simply dropped once full. + */ +static void svc_gr_reloc_stage_note(struct gsp_sema_gr_reloc *out, + const char *name, int rc) +{ + size_t n = strlen(out->stage); + + if (n + 16 >= sizeof(out->stage)) + return; + scnprintf(out->stage + n, sizeof(out->stage) - n, "%s:%d,", name, rc); +} + +/* + * relfix (agent-A FIX A): rewrite-time GR relocation resolver (see + * gsp_sema_service.h). Mirrors the per-page source order of the fill + * (svc_gr_data_fill: svc_ipin_lookup -> svc_resolve_va's trusted chain -> + * ledger) and of the read-only attribution probe svc_gr_src_probe above, + * then applies the m180 ledger in the fill's two roles: fallback when + * nothing above answered (the C1 landing path), and the m171 landing + * preference for VAs at/above the ledger floor when the ledger disagrees + * with the resolver (the skip-ahead class where a walk result is stale but + * the landing is authoritative). + * + * Deliberately excluded versus svc_resolve_va: pin_store/lat_store writes, + * svc_pd_discover and the sliced targeted RAM scan - those mutate + * service-thread state or can block for seconds, and this runs on the GR + * work item (process context; the page-table walks take svc.vas_lock only + * for their publication snapshot and may sleep in vfio_dma_rw, exactly the + * discipline the walk comment above svc_walk_vaspaces mandates). + * + * Every attempt is recorded in out->stage ("name:rc," chain; the resolving + * source ends in ":0,") for the booter's RELOC-GAP-STAGE diagnostic. + * + * Returns 0 with out filled on a successful translation, -ENOENT on a miss + * (caller must leave the word untouched), or a conversion errno. + */ +int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out) +{ + struct nv_vgpu_device *vgpu = svc_fwd_req.vgpu; + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0, lpa = 0, hpa = 0; + int ap = SVC_APERTURE_SYS; + u32 lap = 0, off = 0; + bool resolved = false; + int rc, ret; + + if (!vgpu || !out) + return -EINVAL; + + out->stage[0] = '\0'; + out->stage_rc = 0; + + rc = svc_ipin_lookup(page, &pa, &ap); + svc_gr_reloc_stage_note(out, "ipin", rc); + if (!rc) + resolved = true; + if (!resolved) { + rc = svc_pin_lookup(page, &pa, &ap); + svc_gr_reloc_stage_note(out, "pin", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_lat_lookup(page, &pa, &ap); + svc_gr_reloc_stage_note(out, "lat", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_map_lookup(page, &pa); + svc_gr_reloc_stage_note(out, "map", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_walk_vaspaces(vgpu, page, &pa, &ap); + svc_gr_reloc_stage_note(out, "vas", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_walk_discovered(vgpu, page, &pa, &ap); + svc_gr_reloc_stage_note(out, "disc", rc); + if (!rc) + resolved = true; + } + + if (!resolved) { + /* C1 fallback: a landing is a page the client already backed + * with content even when every resolver missed (r174). */ + rc = svc_ledger_lookup(page, &lpa, &lap); + svc_gr_reloc_stage_note(out, "ledger", rc); + if (rc) { + out->stage_rc = -ENOENT; + return -ENOENT; + } + pa = lpa; + ap = (int)lap; + } else if (page >= SVC_VAS_FLOOR) { + /* m171: the landing table wins over a disagreeing resolver. */ + if (!svc_ledger_lookup(page, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != svc_ap_of(ap))) { + pa = lpa; + ap = (int)lap; + } + } + + ret = svc_fwd_operand_hpa(vgpu, + svc_ap_of(ap) == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + pa, 0, SEMA_PAGE_SIZE, &hpa, &off); + if (ret || off) { + svc_gr_reloc_stage_note(out, "hpa", ret ? ret : -EFAULT); + out->stage_rc = ret ? ret : -EFAULT; + return ret ? ret : -EFAULT; + } + + out->pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + out->ap = svc_ap_of(ap); + out->hpa = hpa; + out->stage_rc = 0; + return 0; +} + +/* + * r378 write guard, GR backend half (see gsp_sema_service.h). The GR submit + * resolves every operand page once and the copy-out/release writes run + * seconds later (the bounded poll, the retire wait, the outcome retries); a + * guest buffer freed and re-homed inside that window would receive a full + * page of another buffer's content through the stale HPA - the guest + * boot-corruption class (slab freelist clobbered by a foreign page write). + * + * Authority order mirrors the fill's own choice (r372/r376 pin-wins): a + * FRESH injected pin is the guest's newest statement, so the guard compares + * the write target against the pin's page first; only a pin-less VA falls + * back to the trusted resolver chain. The landing table is deliberately NOT + * the authority here - our own launches feed it, so it would confirm the + * target by construction. A miss on every source keeps the historical + * behaviour (the recorded landing is the write authority), so the guard can + * only remove provably stale writes. Bounded diagnostic budget. + */ +static u32 svc_wguard_log; /* r378 WGUARD-SKIP budget (256/boot) */ +bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 ipa = 0, cur_hpa = 0; + int iap = SVC_APERTURE_SYS; + u32 off = 0; + int ret; + + if (!va || !hpa) + return true; + /* + * r632: a VA with a shadow page uses the shadow as its storage (the + * copy paths resolve it there before any other source, and the launch + * path serves its pages from it). The engine's writeback targets + * exactly that page, so a later pin or discovery answer must not skip + * the write - skipping it would strand the engine's output in the + * bounce while the D2H copies still read the shadow. + */ + if (svc_vas_has(va)) + return true; + if (!svc_ipin_lookup(page, &ipa, &iap)) { + u32 kind = svc_ap_of(iap) == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + + ret = svc_fwd_operand_hpa(svc_fwd_req.vgpu, kind, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + 0, SEMA_PAGE_SIZE, &cur_hpa, &off); + if (ret || off) + return true; /* pin page unresolvable: keep */ + } else { + struct gsp_sema_gr_reloc r; + + if (gsp_sema_gr_reloc_resolve(va, &r)) + return true; /* no current answer: keep */ + cur_hpa = r.hpa; + } + if ((cur_hpa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (hpa & ~(u64)(SEMA_PAGE_SIZE - 1))) + return true; + if (svc_wguard_log < 256) { + svc_wguard_log++; + pr_warn("vgsp-sema: WGUARD-SKIP path=%s va=0x%llx used=0x%llx now=0x%llx (stale write target)\n", + path, va, hpa, cur_hpa); + } + return false; +} + +/* + * r378 write guard, service half (see gsp_sema_service.h). A fresh injected + * pin is the guest's newest statement about a VA; a pin that disagrees with + * the recorded write target means the page was freed/re-homed (the r372/r376 + * pin-wins rule applied at the write instead of only at the resolution). A + * missing pin keeps the historical behaviour - the recorded landing is then + * the write authority - so no green path loses its write. + */ +bool gsp_sema_write_guard(u64 va, u64 pa, u32 ap, const char *path) +{ + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!va || !pa || va < SVC_VAS_FLOOR) + return true; + if (svc_ipin_lookup(va, &ipa, &iap)) + return true; /* no fresh pin: nothing newer to obey */ + if ((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + svc_ap_of(iap) == svc_ap_of((int)ap)) + return true; + if (svc_wguard_log < 256) { + svc_wguard_log++; + pr_warn("vgsp-sema: WGUARD-SKIP path=%s va=0x%llx used=0x%llx/%u pin=0x%llx/%d (stale write target)\n", + path, va, pa, ap, ipa, iap); + } + return false; +} + +/* + * r391 diagnostic (sum=0 class): the landing-table resolution of a GR data VA + * plus the first 8 bytes read through it - the exact page the D2H reader's + * ledger-first branch will use. Printed by the GR copy-out next to the write + * target, so a launch/readback disagreement is visible in one line. + */ +bool gsp_sema_gr_va_snap(u64 va, u64 *pa, u32 *ap, u64 *val8) +{ + u64 lpa = 0; + u32 lap = 0; + + if (svc_ledger_lookup(va, &lpa, &lap)) + return false; + *pa = lpa & ~(u64)(SEMA_PAGE_SIZE - 1); + *ap = lap; + *val8 = 0; + if (!svc_fwd_req.vgpu) + return true; + if (lap == SVC_APERTURE_FB) { + u8 *base = svc_fwd_req.vgpu->bars[1].vaddr; + size_t size = svc_fb_shadow_size(svc_fwd_req.vgpu); + + if (base && *pa + 8 <= size) + memcpy(val8, base + *pa, 8); + } else { + svc_read(svc_fwd_req.vgpu, *pa, val8, 8); + } + return true; +} + +/* r391 diagnostic: the shadow base this service resolves through (compare + * against the CE copy path's own vgpu->bars[1].vaddr in the same run). */ +u8 *gsp_sema_gr_shadow_base(void) +{ + return svc_fwd_req.vgpu ? svc_fwd_req.vgpu->bars[1].vaddr : NULL; +} + +/* * m144 (second-client launch-state divergence): collect device-VA-shaped * candidate pointers from the bytes the ENGINE will actually execute. * @@ -12688,11 +31354,19 @@ static void svc_gr_scan_cand(u64 *wins, u32 *n_cand, const u8 *data, u32 bytes) if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* SESSION 139: float-shaped words are scalars, not device VAs + * (see the scan-site note; r789's 480.0f = 0x43f00000). */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < *n_cand; w++) if (wins[w] == wv) break; - if (w == *n_cand && *n_cand < 16u) + if (w == *n_cand && *n_cand < SVC_GR_CAND_MAX) wins[(*n_cand)++] = wv; } } @@ -12710,10 +31384,387 @@ static void svc_gr_scan_inline(u64 *wins, u32 *n_cand, u64 page_va) } } -static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) +/* + * STALLFREE: identity of the launch currently held/retried on this channel - + * the capture's push VA when it belongs to this channel (the busy-refusal + * path never reaches the fill, so the request slot can still hold a previous + * launch's VA), else the request slot's when it names this channel. The + * absolute hold bound is keyed to this identity so a pin-batch rewind that + * replays the entry at a different slot index can never re-arm it. + */ +static u64 svc_gr_launch_va(const struct svc_channel *ch) +{ + if (svc_gr_cand.valid && svc_gr_cand.h_object == ch->h_object) + return svc_gr_cand.push_va; + if (svc_fwd_req.gr && svc_fwd_req.gr_h_object == ch->h_object) + return svc_fwd_req.gr_push_va; + return 0; +} + +/* + * GRFIX FIX1b (r213): absolute per-launch hold bound. The m150/m156 data + * hold had no working bound: the 8b058884 deadline was keyed by the GPFIFO + * slot index, and a pin-batch rewind/replay re-decodes the same launch push + * at different slots (r213: entry 107 -> 107 -> 99 on one channel, other + * passes at yet other indices), so "first arm" re-armed on every pass; the + * launch held for 216+ s (410+ GRFILL-HOLD lines) until the client's 240 s + * timeout killed it (gr_hold_exp=0). The identity is now the launch push VA: + * the first hold of a VA arms an ABSOLUTE deadline that nothing moves while + * the launch is held. Returns true when the bound has passed (caller must + * let the launch proceed); false leaves the record armed. + */ +static bool svc_gr_hold_bound_expired(struct svc_channel *ch, u64 launch_va) +{ + u32 cap_ms = gsp_phys_gr_hold_max_ms; + + /* + * STALLFREE (r213-r215 follow-up): the absolute per-launch bound is + * now effective on the default path. Before the write-back fix the + * bound's record fields were excluded from the pump snapshot and + * zeroed every pass, so this returned false forever and the legacy + * m150/m156 hold window (gr_hold_until) was the only expiry - and + * that window is keyed to the GPFIFO slot index, which a pin-batch + * rewind changes on every replay (the r213 identity bug). Under pin + * churn the same launch was therefore re-held indefinitely: r220 + * held one launch from 155221 to 155417 (196 s, 1631 re-fills of + * entry 106) until the guest's 240 s kill. The bound is keyed to the + * LAUNCH PUSH VA and nothing re-arms it, so a replayed slot no + * longer extends the hold; at the cap the pump falls through to the + * historical consume, where STALLFREE authors the QMD completion + * release and logs GR-STALL-hold action=proceed. Default cap 3000 ms + * (gsp_phys_gr_hold_max_ms) sits above the observed 0.5-2 s pin + * latency, so the legitimate pin-late retry still wins. + */ + if (!launch_va) + launch_va = 1; /* anonymous launch: still absolutely bound */ + if (!cap_ms) + cap_ms = 3000; + /* + * r368: mirror of the opdef fix - an ACTIVE bound is never re-armed + * by a different launch VA. The pump can hold several launches whose + * push VAs differ; arming on every VA change moved the deadline + * forever (the same identity bug class as the opdef=955/13/0 + * streams2 wedge). The first hold arms; others ride it; a fresh + * bound arms once the previous one has passed. + * + * r509: the bound must actually REPORT expiry. The historical arming + * rule re-armed the deadline on the very call that detected expiry + * and then compared against the NEW deadline - so the function could + * never return true, and a data-stalled launch whose arguments never + * resolved held its entry forever (r508 witness: matmul plus ten + * sibling channels in "GR hold ... data args unresolved" with zero + * GR-STALL-hold-bound lines for 10+ minutes; the designed + * retry-forward + consume never ran). The expiry is now one-shot: + * push the deadline, latch gr_hold_bound_hit for the r351 block of + * the same pass, return true. A fresh arm clears the latch (and the + * once-per-launch retry log). + */ + if (!ch->gr_hold_dead_va || + time_after_eq(jiffies, ch->gr_hold_deadline)) { + ch->gr_hold_dead_va = launch_va; + ch->gr_hold_dead_since = jiffies; + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + ch->gr_hold_exp_logged = false; + ch->gr_hold_bound_hit = false; + return false; + } + /* + * r420: pause the per-launch consumption bound while the backend + * deliberately holds the in-flight launch in its bounded WEDGE-QUIET + * window. At the bound the pump falls through to the historical + * consume, which authors the capture's QMD completion release in + * software (GR-STALL-hold) - but the backend is still going to + * submit/accept this very launch after the hold and copy its data + * out; authoring now releases the client onto stale data (the r419 + * mid-boot escape). The quiet hold is a bounded msleep (the + * backend's own caps), and the deferral here is bounded by + * SVC_GR_QUIET_PAUSE_MAX_MS from the bound's arm time as + * defence-in-depth. Rate-limited announcement. + */ + if (ch->gr_hold_dead_va && gsp_phys_gr_quiet_hold_active() && + time_before(jiffies, ch->gr_hold_dead_since + + msecs_to_jiffies(SVC_GR_QUIET_PAUSE_MAX_MS))) { + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + pr_info_ratelimited("vgsp-gr: GR-HOLD-BOUND-PAUSE launch=0x%llx (backend wedge-quiet hold active; consumption bound deferred)\n", + launch_va); + return false; + } + if (time_after_eq(jiffies, ch->gr_hold_deadline)) { + /* + * r509: one-shot expiry. Push the deadline so the hold window + * restarts for the caller's retry, latch the verdict for the + * second call site of this pass (the r351 retry-forward + * block), and report expired. + */ + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + ch->gr_hold_bound_hit = true; + return true; + } + return false; +} + +/* + * r632 (the red-draw resolution class): serve a data page from the VA shadow + * lane when the resolution chain (ipin > resolver > ledger) misses. + * + * The red draw's chain is: a client session whose device VAs no source + * answers (the FB-aperture directory walk returns rc=-2 for every published + * VA space, the pin stream delivers no pair, the discovery re-validation + * fails) - the client's H2D copies then land in the shadow lane (the copy's + * documented fallback), and the launch fill drops every operand, the hold + * expires, the data set is degenerate and the refusal faults the client. + * + * Serving the page from the shadow instead keeps one story across the whole + * pipeline: the copy paths already read/write a shadowed VA in the shadow + * (svc_vas_has is checked before every resolver there), so the fill hands the + * engine the client's own H2D bytes (or a zeroed page for an output the + * client never wrote), and the submit build passes the shadow page's host + * address to the bounce machinery. The engine's writeback lands back in that + * same page, which is exactly what the D2H copies read. The shadow is + * module-owned memory: no guest page is ever touched on this path, and the + * pages are freed with the session like every other shadow page. + * + * Existing shadow content serves unconditionally (the client wrote it). + * A synthetic page (an output buffer no copy has touched) is created only + * for ANCHORED argument candidates - candidates whose span already has at + * least one page the resolvers answered - under both a per-candidate cap + * (the span has no other way to know where the buffer ends - the cap bounds + * the guess and the first page failing the cap ends the span through the + * historical drop path) and a per-launch cap (the bounce window must + * survive). With sema_no_shadow=1 every fallback is refused and the caller + * keeps its historical -ENOENT/drop path. + * + * r633 (the r632 cap validation): the r632 caps were per-FILL-CALL, and the + * fill re-runs on every retry pass of a held launch - one probe=none junk + * candidate could therefore mint another 1024 synthetic pages per pass and + * the 8192-page shadow table filled within seconds (r632 mail: va-shadow + * 8192/8192 with all pages synthetic, from the two 16 MB guessed spans of + * candidates no resolver ever answered). Two changes bound this: + * - the ANCHOR GATE: synthetic pages exist only to extend a span that + * real evidence has already anchored (>= 1 page resolved through + * ipin/resolve/ledger). A candidate whose every page is answerless is + * the historical drop path - its guessed span is not a buffer; + * - the budget is PERSISTED PER LAUNCH EPISODE, keyed by (push VA, span + * base), so the caps hold across the episode's passes, not per call. + * The anchor gate is what keeps the genuinely useful case working: the + * red-draw chain's H2D-shadowed operand pages serve through the CONTENT + * path (unconditionally), and a real buffer's unresolved tail is anchored + * by its resolved head. + */ +#define SVC_GR_SYNTH_EP_SLOTS 16 +static struct svc_gr_synth_ep { + u64 push_va; + u64 base; /* span base; 0 = the launch-level total record */ + u32 n; + /* + * r635: the cross-mint counter (launch-level anchor, see + * svc_gr_shadow_fill_serve). Tracked separately from n so a cross + * mint can never consume the anchored budgets and vice versa. + */ + u32 cross; +} svc_gr_synth_ep[SVC_GR_SYNTH_EP_SLOTS]; +static u32 svc_gr_synth_ep_pos; + +static struct svc_gr_synth_ep *svc_gr_synth_ep_get(u64 push_va, u64 base, + bool create) +{ + u32 i, slot = ~0u; + + if (!push_va) + return NULL; + for (i = 0; i < SVC_GR_SYNTH_EP_SLOTS; i++) { + if (svc_gr_synth_ep[i].push_va == push_va && + svc_gr_synth_ep[i].base == base) + return &svc_gr_synth_ep[i]; + if (slot == ~0u && !svc_gr_synth_ep[i].push_va) + slot = i; + } + if (!create) + return NULL; + if (slot == ~0u) + slot = svc_gr_synth_ep_pos++ % SVC_GR_SYNTH_EP_SLOTS; + memset(&svc_gr_synth_ep[slot], 0, sizeof(svc_gr_synth_ep[slot])); + svc_gr_synth_ep[slot].push_va = push_va; + svc_gr_synth_ep[slot].base = base; + return &svc_gr_synth_ep[slot]; +} + +static u32 svc_gr_synth_ep_launch_n(u64 push_va) +{ + struct svc_gr_synth_ep *lep = + svc_gr_synth_ep_get(push_va, 0, false); + + return lep ? lep->n : 0; +} + +static void svc_gr_synth_ep_reset(void) +{ + memset(svc_gr_synth_ep, 0, sizeof(svc_gr_synth_ep)); + svc_gr_synth_ep_pos = 0; +} + +/* + * r635 (host-side operand resolution without the scanner): the r634 red + * draws measured the remaining shape - a client whose INPUT buffers are + * shadow-CONTENT served (its own H2D bytes, the copy took the shadow lane) + * while its OUTPUT buffer (an argument candidate the client never wrote) + * still failed closed, because the r633 anchor gate only counted a page + * answered through ipin/resolve/ledger and every page of the output span + * was answerless. Two additions, both host-side (no scanner/pin needed): + * + * - a CONTENT-served page anchors its span exactly like a trusted answer: + * the client's own H2D bytes are first-hand truth, and the output tail + * of a contiguous allocation (matmul d_a|d_b|d_c, streams2's buffers) + * then mints synthetic pages through the unchanged per-candidate / + * per-launch budgets instead of dropping; + * - a LAUNCH-LEVEL anchor (cross): once any argument page of the launch + * is served from content or trust, an answerless ARG candidate whose + * span holds nothing at all (a non-contiguous output allocation) may + * still mint a strictly smaller cross-budget page set. This is what + * makes output buffers resolvable regardless of candidate order or + * allocation placement. + * + * Guards kept from r632/r633: only argument candidates mint, budgets are + * per launch episode, and every synthetic mint additionally respects + * SVC_GR_SHADOW_TABLE_GUARD - once the shared session shadow table is + * three-quarters full, no guess may displace the headroom real client + * content (H2D writes) still needs; content serves unconditionally. + */ +#define SVC_GR_SHADOW_NONE 0 +#define SVC_GR_SHADOW_CONTENT 1 +#define SVC_GR_SHADOW_SYNTH 2 +#define SVC_GR_SHADOW_SYNTH_CROSS 3 +/* + * Cross budgets: the r633 proven caps (1024 per candidate / 4096 per + * launch). They apply to CONTENT-anchored spans; the flood class they + * must never re-open (r632: 8192/8192 synthetic pages from answerless + * junk spans) is excluded by the r635 content gate - a green boot has + * no written shadow content at all, so the cross mint is off there, and + * an answerless junk span on a red boot has no head to anchor it. + */ +#define SVC_GR_SHADOW_CROSS_PER_CAND 1024u +#define SVC_GR_SHADOW_CROSS_MAX 4096u +#define SVC_GR_SHADOW_TABLE_GUARD (SVC_VAS_SLOTS * 3u / 4u) + +static int svc_gr_shadow_fill_serve(u64 va, bool is_arg, bool trusted, + bool span_content, bool launch_content, + u64 push_va, u64 base, bool *real_content) +{ + struct svc_gr_synth_ep *ep = NULL; + bool have; + + if (real_content) + *real_content = false; + if (!svc_vas_can_shadow(va)) + return SVC_GR_SHADOW_NONE; + have = svc_vas_has(va); + if (!have) { + if (!is_arg) + return SVC_GR_SHADOW_NONE; + if (trusted) { + /* + * r633 path: a trusted answer (ipin/resolver/ledger) + * anchors the span; the large r633 budgets apply. + */ + if (svc_vas_n >= SVC_GR_SHADOW_TABLE_GUARD) + return SVC_GR_SHADOW_NONE; + ep = svc_gr_synth_ep_get(push_va, base, true); + if (!ep || ep->n >= SVC_GR_SHADOW_SYNTH_PER_CAND) + return SVC_GR_SHADOW_NONE; + if (svc_gr_synth_ep_launch_n(push_va) >= + SVC_GR_SHADOW_SYNTH_MAX) + return SVC_GR_SHADOW_NONE; + } else if (span_content || launch_content) { + /* + * r635 path: the anchor is the client's own written + * content (its H2D took the shadow lane), not a + * resolver answer. The cross budgets apply - smaller, + * because this serves the output tail of a red-draw + * client, not a trusted span. + */ + struct svc_gr_synth_ep *lep; + + if (svc_vas_n >= SVC_GR_SHADOW_TABLE_GUARD) + return SVC_GR_SHADOW_NONE; + ep = svc_gr_synth_ep_get(push_va, base, true); + if (ep && ep->cross >= SVC_GR_SHADOW_CROSS_PER_CAND) + return SVC_GR_SHADOW_NONE; + lep = svc_gr_synth_ep_get(push_va, 0, false); + if (lep && lep->cross >= SVC_GR_SHADOW_CROSS_MAX) + return SVC_GR_SHADOW_NONE; + } else { + return SVC_GR_SHADOW_NONE; + } + } + { + struct svc_vas_rec *r = svc_vas_get(va, true); + + if (!r) + return SVC_GR_SHADOW_NONE; + if (have) { + /* + * r635: real_content tells the caller whether these + * bytes are the client's own WRITES (anchoring + * material) or a synthetic zero page minted by an + * earlier pass (servable, but never an anchor - the + * guard against the self-feeding mint loop). + */ + if (real_content) + *real_content = r->content != 0; + if (svc_gr_shadow_fill_log < 32) { + svc_gr_shadow_fill_log++; + pr_info("sema-svc: gr data page served from the shadow lane va=0x%llx (content%s)\n", + va, r->content ? "" : ", synthetic"); + } + return SVC_GR_SHADOW_CONTENT; + } + } + if (trusted) { + struct svc_gr_synth_ep *lep = + svc_gr_synth_ep_get(push_va, 0, true); + + ep->n++; + if (lep) + lep->n++; + if (svc_gr_shadow_fill_log < 32) { + svc_gr_shadow_fill_log++; + pr_info("sema-svc: gr data page served from the shadow lane va=0x%llx (synthetic; cand=%u launch=%u)\n", + va, ep->n, + svc_gr_synth_ep_launch_n(push_va)); + } + return SVC_GR_SHADOW_SYNTH; + } + { + struct svc_gr_synth_ep *lep = svc_gr_synth_ep_get(push_va, 0, + true); + + if (ep) + ep->cross++; + if (lep) + lep->cross++; + if (svc_gr_shadow_fill_log < 32) { + svc_gr_shadow_fill_log++; + pr_info("sema-svc: gr data page served from the shadow lane va=0x%llx (synthetic cross; cand=%u launch=%u)\n", + va, ep ? ep->cross : 0, lep ? lep->cross : 0); + } + } + return SVC_GR_SHADOW_SYNTH_CROSS; +} + +static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) { - u64 wins[16]; - bool is_arg[16]; /* m150: candidate came from the argument scan */ + u64 wins[SVC_GR_CAND_MAX]; + bool is_arg[SVC_GR_CAND_MAX]; /* m150: candidate came from the arg scan */ + /* + * r635: launch-level CONTENT anchor for the host-side operand + * fallback (see the pre-pass and svc_gr_shadow_fill_serve). Armed + * only by the client's own WRITTEN shadow content (H2D bytes that + * took the shadow lane) - a resolver/pin/ledger answer is NOT + * enough, so a green boot cannot enable the cross mint; consumed + * by the cross-mint for answerless argument candidates. + */ + bool launch_arg_content = false; u32 n_cand; u32 off = 0, i, w; u8 kind; @@ -12724,8 +31775,32 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * window below the mail page for A/B trialing. */ bool region_on = gsp_phys_gr_data_region_value() != 0; - u64 reloc_base = region_on ? SVC_GR_DATA_RELOC_BASE : - SVC_GR_DATA_RELOC_BASE_OLD; + /* + * fail4: two concurrent forwarded launches (streams2's vadd pair) + * both relocated their operands from the same base, so the second + * launch's copy-in overwrote the first's bounce windows and each + * engine read the other's inputs (c1 came out as a2+b2). Alternate + * the per-launch base across the grown region's two halves (32 MB + * each, above any single launch's span), keyed to the capture push + * so a retried launch keeps its half, so concurrent launches + * cannot collide. + */ + static u64 svc_gr_reloc_half_key; + static u8 svc_gr_reloc_half; + u64 reloc_off = 0; + + if (region_on) { + if (svc_gr_cand.push_va != svc_gr_reloc_half_key) { + svc_gr_reloc_half_key = svc_gr_cand.push_va; + svc_gr_reloc_half ^= 1; + } + if (svc_gr_reloc_half) + reloc_off = (u64)(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / + 2u) * SEMA_PAGE_SIZE; + } + u64 reloc_base = region_on ? + SVC_GR_DATA_RELOC_BASE + reloc_off : + SVC_GR_DATA_RELOC_BASE_OLD; u32 per_ptr = region_on ? SVC_GR_DATA_PAGES_PER_PTR : SVC_GR_DATA_PAGES_PER_PTR_OLD; u32 bounce_max = region_on ? SVC_GR_DATA_BOUNCE_MAX : @@ -12738,8 +31813,36 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * retry instead of forwarded (svc_gr_data_stalled). */ u32 n_arg_hi = 0, n_arg_hi_res = 0; + /* + * GRFIX FIX1 (full-page fail-closed): every page of every tracked + * span whose resolution chain (ipin > resolve > ledger) misses is + * counted here, first page or mid-span alike. Any drop holds the + * launch through the same bounded m150/m156 retry path as the + * first-page guard below, so a QMD is never forwarded with a data + * pointer left as an unresolvable guest VA (the zero-store class) + * and the bounce window is never served stale for this launch. + * FIX1b: the hold carries an absolute per-launch bound + * (gsp_phys_gr_hold_max_ms) - when it passes the launch proceeds. + */ + u32 dropped_pages = 0; + bool bound_hit = false; /* GRFIX FIX1c: the absolute bound fired this + * pass - the launch proceeds, no hold */ + /* + * GRFIX FIX1c (r213): detail for the first (up to 4) dropped pages, + * so the cycle records WHICH VAs and which candidate class dropped. + * There is no read/write direction anywhere in the candidate or + * data[] structures (both are va/pa/reloc_va only), so kind is the + * truthful candidate origin: 'a' = argument page (c[0x0] param / + * inline), 'b' = const-bank page; idx is the candidate index (the + * same index the per-candidate probe lines use). + */ + u64 drop_va[4]; + char drop_kind[4]; + u32 drop_idx[4]; svc_fwd_req.gr_n_data = 0; + svc_fwd_req.gr_dropped = 0; + svc_fwd_req.gr_arg_partial = false; svc_fwd_req.gr_n_data_wins = 0; svc_gr_data_stalled = false; memset(is_arg, 0, sizeof(is_arg)); @@ -12751,6 +31854,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) */ n_cand = 0; /* + * SESSION 153-CONTINUED-63: the captured colour target (the surface + * the replayed draw rasterizes into) becomes a tracked data + * candidate FIRST, so its pages are in the launch's data set even + * when the window/content scans fill every other slot (the beef + * windows' 24 scan candidates are all unresolvable junk). Odd: the + * copy-in stages the guest page into the bounce, the engine's + * stores land in the bounce, the retire wait sees them (GR-DEC + * n_data>=1 -> "N/N pages changed") and the copy-out writes the + * changed pages back to the guest's own pages. Completeness is + * recorded for the booter's rec-over-alias preference. + */ + u64 surf_cand_va = 0; + u32 surf_cand_span = 0; + bool surf_cand_used = false; + + WRITE_ONCE(svc_gr_surf_rec_ok, 0); + if (gsp_sema_surf_va_get(&surf_cand_va, &surf_cand_span)) { + if (surf_cand_span > SVC_GR_SURF_TRACK_MAX) + surf_cand_span = SVC_GR_SURF_TRACK_MAX; + if (surf_cand_va && !(surf_cand_va & (SEMA_PAGE_SIZE - 1)) && + surf_cand_span && n_cand < SVC_GR_CAND_MAX) { + wins[n_cand] = surf_cand_va; + is_arg[n_cand] = false; + n_cand++; + } + } + /* * m121: kernel-argument pointers first. The launched kernel's device * pointers live in the param page (qmd_word 32, c[0x0]) at byte * offsets >= 0x150 (m120: mail_kernel out@0x160, msg@0x168; vadd @@ -12764,18 +31894,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; if (pp->qmd_word != 32u || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } for (k = 0x150u / 8u; k < 256; k++) { u64 v = get_unaligned_le64(page + k * 8); u64 wv; @@ -12783,6 +31928,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -12800,30 +31962,149 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * actually dereferences. */ for (w = 0; w < n_cand; w++) is_arg[w] = true; + /* + * relfix PART 3 (fill candidate superset): the rewrite scans the WHOLE + * authored param page (0x0..0xfff) and re-resolves every + * device-VA-shaped word it finds; any word this fill does not collect + * resurfaces as a RELOC-GAP (r217: the two words the late path fixed + * were param/program descriptor fields below 0x150 / past 0x800 that + * the argument scan never looked at). Collect the page remainder here, + * AFTER the argument marking, so these page words keep is_arg=false - + * the per-argument hold and diagnostic semantics stay exactly as + * before. Budget priority is also unchanged: the argument candidates + * above are resolved first. Launch-internal pages remain filtered at + * resolution time (R3 hygiene below). + */ for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; - if (pp->qmd_word < 32 || pp->qmd_word > 47 || - (pp->qmd_word & 1) || !pp->pa) - continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) + if (pp->qmd_word != 32u || !pp->pa) continue; - page = (const u8 *)__va(hpa); - if (!page) + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } + for (k = 0; k < SEMA_PAGE_SIZE / 8; k++) { + u64 v, wv; + + if (k >= 0x150u / 8u && k < 256u) + continue; /* arg region: scanned above */ + v = get_unaligned_le64(page + k * 8); + if (v < 0x10000000ULL || v >= (1ULL << 47) || + (v & 3)) + continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } + wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); + for (w = 0; w < n_cand; w++) + if (wins[w] == wv) + break; + if (w == n_cand && n_cand < ARRAY_SIZE(wins)) + wins[n_cand++] = wv; + } + } + for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { + const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; + u64 hpa = 0; + const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; + u32 k; + + /* SESSION 139: include word 48 (see the r788 finding: the + * word-34 pointer dedupes into the program's 4K page whose + * entry carries qmd_word=48; the old 32..47-even filter left + * that page unscanned). */ + if (pp->qmd_word < 32 || pp->qmd_word > 48 || + (pp->qmd_word & 1) || !pp->pa) continue; - for (k = 0; k < 256; k++) { + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } + /* + * relfix PART 3: whole page, matching the rewrite's scan + * range (it walks 0x0..0xfff qword-pairs); the old 0..0x7ff + * bound left the second half as permanent RELOC-GAPs. + */ + for (k = 0; k < SEMA_PAGE_SIZE / 8; k++) { u64 v = get_unaligned_le64(page + k * 8); u64 wv; if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -12835,6 +32116,90 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * inline-captured bytes as well (see svc_gr_scan_inline). */ svc_gr_scan_inline(wins, &n_cand, pp->va); } + /* + * SESSION 140 (the n_data wiring): seed the candidate set with the + * launch's OWN QMD pointer pairs (words 34..47, already decoded into + * gr_qmd from the push's inline-QMD run). These ARE the launch's + * data-window pointers - the r792 evidence: the capture holds + * ptr[0]=0xdff0febc00 but the fill's page scans read only empty + * shadow pages (the QMD page's vidmem shadow is zero; its content + * never arrives - the invisible-vidmem-write class), so the pointer + * values could never become candidates and n_data stayed 0 (refused + * degenerate). The values need no page scan - use them directly. + */ + { + u32 w2; + + for (w2 = 0; w2 + 1 < 16; w2 += 2) { + u64 pv = (u32)svc_fwd_req.gr_qmd[34 + w2] | + ((u64)(svc_fwd_req.gr_qmd[35 + w2] & + 0x1ffffu) << 32); + u64 pw; + u32 fexp; + + if (pv < 0x10000000ULL || pv >= (1ULL << 47) || + (pv & 3)) + continue; + fexp = (u32)(pv >> 23) & 0xffu; + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + pw = pv & ~(u64)(SEMA_PAGE_SIZE - 1); + for (w = 0; w < n_cand; w++) + if (wins[w] == pw) + break; + if (w == n_cand && n_cand < ARRAY_SIZE(wins)) { + wins[n_cand++] = pw; + is_arg[n_cand - 1] = true; + if (svc_gr_pages_log < 64) { + svc_gr_pages_log++; + pr_info("sema-gr: qmd-ptr seed va=0x%llx (word %u) h=0x%x\n", + pw, 34 + w2, + svc_fwd_req.gr_h_object); + } + } + } + } + if (gsp_sema_dbg && svc_gr_pages_log < 48) { + svc_gr_pages_log++; + pr_info("sema-gr: fill cand n=%u pages=%u push=0x%llx\n", + n_cand, svc_fwd_req.gr_n_pages, + svc_fwd_req.gr_push_va); + for (w = 0; w < n_cand && w < 12; w++) + pr_info("sema-gr: win[%u] va=0x%llx arg=%u\n", + w, wins[w], is_arg[w]); + } + /* + * r635: the launch-level CONTENT anchor for the host-side fallback, + * computed BEFORE the span loop so candidate order cannot decide. + * An argument candidate whose span already holds the client's own + * WRITTEN shadow bytes (r->content; a synthetic zero page does not + * count) marks the launch's operands as host-visible; an answerless + * ARG candidate (the output buffer the client never wrote, possibly + * not contiguous with the inputs) may then mint a strictly bounded + * cross-budget page set instead of failing the whole launch closed. + * Only the first SVC_GR_CROSS_ANCHOR_PROBE pages of each span are + * consulted - a real H2D'd buffer always has its head written, and + * the probe is a pure hash lookup. Trusted answers (pin/walk/ledger) + * do NOT arm this: on a green boot the cross mint stays off. + */ +#define SVC_GR_CROSS_ANCHOR_PROBE 64u + { + bool found = false; + + for (w = 0; w < n_cand && !found; w++) { + u32 p; + if (!is_arg[w] || wins[w] < 0x40000000000ULL) + continue; + for (p = 0; p < SVC_GR_CROSS_ANCHOR_PROBE; p++) { + if (svc_vas_has_content(wins[w] + + (u64)p * SEMA_PAGE_SIZE)) { + found = true; + break; + } + } + } + launch_arg_content = found; + } /* m110/B-2: one contiguous run of pages per candidate pointer, * relocated into the active engine-visible data region (grown region * 0x0a000000+ with gsp_phys_gr_data_region=1 - the default - or the @@ -12844,6 +32209,16 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) u64 base = wins[w]; u32 p; u32 before = svc_fwd_req.gr_n_data; + bool cand_anchored = false; /* r633: >= 1 page of this span + * resolved through a trusted + * source (ipin/resolve/ledger); + * the r633 large synthetic + * budgets only apply here */ + bool cand_content = false; /* r635: >= 1 page of this span + * holds the client's own + * WRITTEN shadow bytes; the + * cross budgets extend it */ + bool first_unresolved = false; /* m170: span's first page failed */ u64 pba = 0; u32 pba_ap = 0; u32 span; @@ -12870,6 +32245,8 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * in every launch and starved the real operands under the old * window. Skip them as data candidates; they are * shadowed/rewritten through gr_pages/Tier 2, not this path. + * P6: the QMD program pair (word 48/49) is launch-internal in + * exactly the same way and is skipped with them. */ if (svc_fwd_req.gr_param_va && (base & ~(u64)(SEMA_PAGE_SIZE - 1)) == @@ -12883,7 +32260,7 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) &svc_fwd_req.gr_pages[i]; if (pp->qmd_word >= 32u && - pp->qmd_word <= 47u && + pp->qmd_word <= 49u && (base & ~(u64)(SEMA_PAGE_SIZE - 1)) == (pp->va & ~(u64)(SEMA_PAGE_SIZE - 1))) { launch_internal = true; @@ -12894,6 +32271,18 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) continue; } span = (base >= 0x40000000000ULL) ? per_ptr : 1u; + /* + * SESSION 153-CONTINUED-63: the render-target candidate keeps + * its full surface span (it is a low VA, so the generic span + * rule above would give it ONE page and the engine's raster + * walk would leave the tracked run at the first page + * crossing). surf_cand_used marks the pass so the tail can + * record completeness only for the run that actually filled. + */ + if (surf_cand_va && base == surf_cand_va) { + span = surf_cand_span; + surf_cand_used = true; + } if (is_arg[w] && base >= 0x40000000000ULL) n_arg_hi++; @@ -12914,6 +32303,9 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) u64 pa = 0; u32 ap = 0, x; struct svc_gr_data *rec; + bool shadow_serve = false; + int serve = SVC_GR_SHADOW_NONE; + bool real_content = false; /* r635 */ if (va >= (1ULL << 47)) break; @@ -12938,8 +32330,220 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) if (!svc_ipin_lookup(va, &ipa, &iap)) { pa = ipa; ap = (u32)iap; + cand_anchored = true; } else if (svc_resolve_va(vgpu, va, &pa, &ap)) { - break; + /* + * C1 (stall/data class): the page may + * still be readable - the ledger holds + * the landing page of the H2D that + * wrote it even when every resolver + * misses (r174 cand[] probe=none, 0 + * pages, bad=65536). Use it before + * declaring the page dead; only a page + * with no ledger entry can truncate + * the span. + */ + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(va, &lpa, &lap)) { + pa = lpa; + ap = lap; + cand_anchored = true; + } else if ((serve = svc_gr_shadow_fill_serve( + va, is_arg[w], + cand_anchored, + cand_content, + launch_arg_content, + svc_gr_cand.push_va, + wins[w], + &real_content)) != + SVC_GR_SHADOW_NONE) { + /* + * r632/r633: no resolver, no + * landing - serve the page + * from the shadow lane + * instead of dropping it. + * Existing H2D shadow + * content serves always; a + * SYNTHETIC page only when + * the span is anchored by a + * resolved page and the + * per-episode budgets hold. + * The rec is created with + * pa=0 + shadow=1 below; + * the submit build hands + * the shadow page to the + * bounce machinery. Never + * breaks the span: a + * shadow-served page is a + * resolved page. + */ + shadow_serve = true; + /* + * r635: REAL content (the + * client's own written H2D + * bytes, r->content) anchors + * the span for the cross + * budgets and arms the + * launch-level content + * anchor. A synthetic zero + * page served from an + * earlier pass is NOT an + * anchor (the self-feeding + * mint guard). + */ + if (serve == + SVC_GR_SHADOW_CONTENT && + real_content) { + cand_content = true; + if (is_arg[w]) + launch_arg_content = + true; + } + } else { + /* + * m170 (R4a): the very first + * page of this span failed to + * resolve - the fail-closed + * trigger below distinguishes + * it from the benign "0 new + * pages" of a span whose + * pages were already mapped. + * + * GRFIX FIX1: every failed + * page counts (first or mid- + * span) for the launch-level + * hold; the span still stops + * here exactly as before. + * FIX1c: capture the first 4 + * (VA + candidate origin + + * candidate index) for the + * GRFILL-DROP detail. + */ + if (p == 0 && + svc_fwd_req.gr_n_data == before) + first_unresolved = true; + /* + * COPYAUDIT (2026-10-18): a + * span cut after at least one + * authored page is the + * completeness hazard - the + * rewritten base pointer is + * still handed to the engine, + * whose base+offset reads then + * run past the authored pages + * into later operands' bounce + * pages. Make the truncation + * loud (want is a lower bound: + * the guest asked for >= p+1 + * pages; done is the authored + * page count before the cut). + */ + if (p > 0) + pr_info_ratelimited("vgsp: COPY-TRUNC va=0x%llx want>=0x%llx done=0x%llx path=gr-fill-span\n", + wins[w], + ((u64)p + 1) << 12, + (u64)p << 12); + /* + * r377 pin-demand: name every + * unresolved operand page as a + * stalled VA. The userspace pin + * stream watches dmesg for this + * exact line and pushes the VA + * to the in-guest scanner's + * demand file, so the page gets + * pinned and the bounded launch + * hold (gsp_phys_gr_fill_hold) + * resumes on real data instead + * of waiting out its bound (the + * r345 wedge: held forever + * because nobody demanded the + * page). + */ + svc_log_stalled_va(va); + if (is_arg[w]) + svc_fwd_req.gr_arg_partial = true; + /* + * SESSION 153-CONTINUED-63: a + * render-target page that does + * not resolve must NOT hold + * the launch (the target is + * an output, not an operand; + * its alias fallback keeps + * today's behaviour), and the + * broken span must not be + * recorded as complete. + */ + if (surf_cand_va && wins[w] == surf_cand_va) + break; + if (dropped_pages < 4) { + drop_va[dropped_pages] = va; + drop_kind[dropped_pages] = + is_arg[w] ? 'a' : 'b'; + drop_idx[dropped_pages] = w; + } + dropped_pages++; + break; + } + } else { + /* r633: the trusted resolver answered + * this page - the span is anchored + * (the r633 large budgets apply; + * the r635 cross anchor stays + * content-only). */ + cand_anchored = true; + } + } + /* + * m171 (AGENT-F R2): prefer the write/read ledger's + * landing page for this operand VA when it disagrees + * with the resolution - the kernel then reads exactly + * the page a large H2D wrote (r174: write landed on + * pa X while the fill read a stale page Y). Compare + * aperture-aware (a SYS page and an FB page with the + * same numeric address are not the same page). + */ + if (va >= SVC_VAS_FLOOR) { + u64 lpa = 0; + u32 lap = 0; + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + /* + * r372 pin-wins: when a live pin exists it is the + * newest truth; the landing table may name a page + * the guest has since freed (VA reuse across + * clients). Keep the pin and repair the table. + * Only a pin-less VA still follows the table. + */ + if (!svc_ipin_lookup(va, &ipa, &iap)) { + pa = ipa; + ap = (u32)iap; + if (!svc_ledger_lookup(va, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (ipa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)iap)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + va, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, ipa, iap); + svc_ledger_store(va, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + } + } else if (!svc_ledger_lookup(va, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != (u32)ap)) { + svc_ledger_diverges++; + if (svc_ledger_diverges <= 32) + pr_info("sema-ledger: DIVERGE fill va=0x%llx wrote=0x%llx read=0x%llx\n", + va, lpa, pa); + pa = lpa; + ap = lap; } } if (svc_fwd_req.gr_n_data >= bounce_max) { @@ -12960,21 +32564,44 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) } rec = &svc_fwd_req.gr_data[svc_fwd_req.gr_n_data]; rec->va = va; - rec->pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + rec->pa = shadow_serve ? 0 : + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)); rec->reloc_va = reloc_base + ((u64)svc_fwd_req.gr_n_data << 12); - rec->ap = (u8)svc_ap_of(ap); + rec->ap = shadow_serve ? SVC_APERTURE_SYS : + (u8)svc_ap_of(ap); + rec->shadow = shadow_serve ? 1 : 0; svc_fwd_req.gr_n_data++; /* + * r349: record the launch's own resolution as the + * ledger's landing authority for this VA. The read + * path's m171 read-after-write rule follows the + * ledger when it disagrees with the live translation, + * and the engine writeback below writes exactly this + * (pa, ap). Without the store the ledger kept a stale + * page for engine-written outputs (victory_reduce: + * the launch wrote FB 0x7c00000 while the ledger + * said SYS 0x7c00000; the D2H read followed the + * ledger and returned zeros while the correct sum + * sat on the written page). + * + * r632: a shadow-served rec has no guest page to + * record - the shadow is its storage, and the copy + * paths already prefer the shadow for these VAs. + */ + if (!shadow_serve && va >= SVC_VAS_FLOOR) + svc_ledger_store(va, rec->pa, rec->ap); + /* * m142: bounded per-page attribution log. A wrong data * page is invisible in the aggregate counters; these * lines let a post-mortem compare the chosen PA against * the guest's own mapping. */ - if (svc_gr_data_log < 16) { + if (svc_gr_data_log < 192) { svc_gr_data_log++; - pr_info("vgsp-sema: svc-gr data page va=0x%llx -> pa=0x%llx ap=%u\n", - va, rec->pa, rec->ap); + pr_info("vgsp-sema: svc-gr data page va=0x%llx -> pa=0x%llx ap=%u%s\n", + va, rec->pa, rec->ap, + shadow_serve ? " (shadow lane)" : ""); } } (void)base; @@ -12984,6 +32611,24 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) pr_info("vgsp-sema: svc-gr candidate[%u] base=0x%llx -> %u new pages (total %u)\n", w, base, svc_fwd_req.gr_n_data - before, svc_fwd_req.gr_n_data); + if (is_arg[w] && svc_fwd_req.gr_n_data == before && + first_unresolved) { + /* + * m170 (AGENT-F R4a): a high argument pointer whose + * FIRST page resolved nothing at fill time is the + * partial-operand signature - the launch would forward + * with stale/zero data for that operand (r174: matmul + * cand[2]/[3] probe=none, 0 pages, bad=65536; the + * other operands resolved via pins). Fail closed: mark + * the launch stalled so the m150 hold keeps the entry + * and retries the fill while the pins land (bounded, + * then forwards anyway - never a hang, never silent + * truncation). + */ + svc_gr_data_stalled = true; + pr_err_ratelimited("vgsp-sema: svc-gr data fill: arg base=0x%llx resolved 0 pages; launch held fail-closed\n", + base); + } /* * m150: an argument pointer that resolves zero pages is the * second-client zeros signature; record the walk result per @@ -12994,7 +32639,14 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) if (is_arg[w] && svc_fwd_req.gr_n_data == before && svc_gr_src_log < 64) { int vi; - char vb[240]; + /* r614 diag: the r613b red-boot post-mortem could not + * tell which registered pdb would have answered an + * unresolved arg base (the 240-byte buffer truncated + * the walk list at three entries, and only the red + * boot's misses were ever printed). A larger buffer + * with a clamped append keeps every registered + * VASpace's walk verdict visible. */ + char vb[512]; int o = 0; svc_gr_src_log++; @@ -13008,27 +32660,253 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) wrc = svc_gmmu_walk(vgpu, svc.vas[vi].pdb, svc.vas[vi].entries, base, svc.vas[vi].ap, &wa, &wap, - false); - o += snprintf(vb + o, sizeof(vb) - o, - " vas[%d] pdb=0x%llx ent=%u ap=%d rc=%d pa=0x%llx", - vi, svc.vas[vi].pdb, - svc.vas[vi].entries, - svc.vas[vi].ap, wrc, wa); + gsp_sema_walk_trace != 0); + if (o < (int)sizeof(vb) - 1) + o += snprintf(vb + o, sizeof(vb) - o, + " vas[%d] pdb=0x%llx ent=%u ap=%d rc=%d pa=0x%llx", + vi, svc.vas[vi].pdb, + svc.vas[vi].entries, + svc.vas[vi].ap, wrc, wa); } pr_info("vgsp-sema: svc-gr arg-miss base=0x%llx probe=%s n_cand=%u pd_found=%u ipin_n=%u disc=0x%llx/%d%s\n", base, pba_src, n_cand, svc_pd_found, svc.ipin_n, svc_pd_base, svc_pd_ap, vb); } } - svc_gr_data_stalled = (n_arg_hi > 0 && n_arg_hi_res == 0); + /* + * m170 revert: the per-argument fail-closed signal is kept as + * DIAGNOSTIC only (the held fail-closed line above identifies the + * exact unresolved operand); the historical aggregate rule held only + * a launch with NO resolvable high argument, because a fresh client's + * launch stream carries BENIGN 0-page candidates (junk pointer words + * that never translate) and persisting the per-arg flag held mail's + * first launch forever (r176). + * + * GRFIX FIX1: the stall is now page-count based - ANY page in a + * tracked span that failed resolution (or any tracked data VA whose + * resolution returned nothing) holds the launch, so a QMD can never + * be forwarded with a data pointer left as an unresolvable guest VA + * and no bounce page is ever served stale for this launch. The + * response is the unchanged bounded m150/m156 hold (leave the GPFIFO + * entry unconsumed, retry the fill while the pins land); the pump + * side now carries a hard wall-clock bound (gr_hold_deadline) so a + * word that never resolves can no longer re-arm the window forever: + * at the bound the entry is consumed with the historical software + * release - bounded, never a hang, never a silent stale-operand + * submission. + */ + svc_gr_data_stalled = (dropped_pages > 0) || + (n_arg_hi > 0 && n_arg_hi_res == 0); + svc_fwd_req.gr_dropped = dropped_pages; + /* + * SESSION 153-CONTINUED-63: record the render-target tracking run - + * only when the pass actually seeded it AND every requested page + * produced a data rec contiguous from the base (the booter then + * rewrites the target's operand pair to the bounce base; a partial + * run falls back to the identity alias so the engine can never walk + * past the tracked pages). + */ + if (surf_cand_used) { + u32 hit = 0; + + for (i = 0; i < svc_fwd_req.gr_n_data; i++) { + u64 rva = svc_fwd_req.gr_data[i].va; + + if (rva >= surf_cand_va && + rva < surf_cand_va + + ((u64)surf_cand_span << 12)) + hit++; + } + if (hit == surf_cand_span) { + svc_gr_surf_rec_va = surf_cand_va; + svc_gr_surf_rec_n = surf_cand_span; + WRITE_ONCE(svc_gr_surf_rec_ok, 1); + } else if (svc_gr_surf_rec_log < 16) { + svc_gr_surf_rec_log++; + pr_info("vgsp-sema: surf-rec incomplete va=0x%llx want=%u got=%u (target falls back to the identity alias)\n", + surf_cand_va, surf_cand_span, hit); + } + } + /* + * GRFIX FIX1e (r215): gsp_phys_gr_fill_hold is the master gate. + * Default 0 = LOG-ONLY: the fill NEVER holds on dropped pages - the + * stalled verdict keeps only the historical pre-8b058884 aggregate + * rule (all high argument pointers unresolved) and the HOLD/EXPIRED + * machinery below is skipped entirely; the launch takes the forward + * path immediately. When 1 the page-count hold applies, bounded by + * the per-launch absolute deadline below. + * + * GRFIX FIX1c/FIX1d (live only with fill_hold=1): the decision point + * is self-enforcing - the absolute per-launch bound is consulted at + * the only place the stalled verdict is produced, before the + * GRFILL-HOLD line can be emitted. When the bound passes the launch + * PROCEEDS via the historical forward path. The benign class + * (probe=none / walk rc=-2) is indistinguishable at fill time from a + * page whose pin has not landed yet, so a "never hold" predicate + * would also disable the pin-late retry; the bound is the only + * guard. NOTE (r213-r215): the bound never fired in the live cycles + * because these per-channel record fields are not in the + * svc_pump_channels snapshot write-back list (see the GRFIX NOTE + * there) and were zeroed on every pass - which is why the gate now + * defaults to 0. + */ + if (!gsp_phys_gr_fill_hold) { + svc_gr_data_stalled = (n_arg_hi > 0 && n_arg_hi_res == 0); + } else if (svc_gr_data_stalled && ch && + svc_gr_hold_bound_expired(ch, svc_fwd_req.gr_push_va)) { + svc.fwd_gr_hold_exp++; + bound_hit = true; + if (!ch->gr_hold_exp_logged) { + ch->gr_hold_exp_logged = true; + pr_info("vgsp-gr: GRFILL-EXPIRED launch=0x%llx dropped=%u waited_ms=%u\n", + svc_fwd_req.gr_push_va, dropped_pages, + jiffies_to_msecs(jiffies - + ch->gr_hold_dead_since)); + } + svc_gr_data_stalled = false; /* proceed: forward */ + } + if (dropped_pages > 0) { + if (gsp_phys_gr_fill_hold && !bound_hit) + pr_info_ratelimited("vgsp-gr: GRFILL-HOLD launch=0x%llx dropped=%u kept=%u\n", + svc_fwd_req.gr_push_va, dropped_pages, + svc_fwd_req.gr_n_data); + /* + * GRFIX FIX1c: name the dropped pages once per launch + * episode (the fill re-runs every pass for a held launch; + * the marker makes the detail a first-pass-only record). + * kind 'a' = argument-page candidate, 'b' = bank candidate; + * idx = candidate index (same indexing as the probe lines). + */ + if (svc_gr_drop_log_va != svc_fwd_req.gr_push_va) { + u32 dl; + + svc_gr_drop_log_va = svc_fwd_req.gr_push_va; + for (dl = 0; dl < 4 && dl < dropped_pages; dl++) { + /* + * r585: DEMAND the dropped pages through the + * pin channel. The host pin stream parses the + * "stalled VA #N: 0x..." lines (svc_log_stalled_va) + * and pushes each VA to the guest scanner within + * ~3 s, whose pin then lands inside the launch's + * bounded argument-defer window - the r584 matmul + * draw dropped 18 staging pages that were NEVER + * demanded (the GRFILL-DROP line format is + * invisible to the host's demand regex), the r543 + * defer window expired, and the r564 degenerate + * refusal SW-completed a launch that wrote + * nothing (MATMUL bad=65536 csum=0). One demand + * per page per boot (svc_log_stalled_va dedups); + * one episode per launch push VA (the marker + * above). + */ + svc_log_stalled_va(drop_va[dl]); + pr_info("vgsp-gr: GRFILL-DROP va=0x%llx kind=%c idx=%u\n", + drop_va[dl], drop_kind[dl], + drop_idx[dl]); + } + } + } if (svc_gr_data_stalled) - pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u high argument ptr(s), 0 resolved -> launch held for a bounded retry\n", - n_cand, svc_fwd_req.gr_n_data, n_arg_hi); + pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u high argument ptr(s), %u resolved; %u dropped page(s) -> launch held for a bounded retry\n", + n_cand, svc_fwd_req.gr_n_data, n_arg_hi, n_arg_hi_res, + dropped_pages); + else if (bound_hit) + pr_info_ratelimited("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u dropped page(s); hold bound expired -> launch proceeds\n", + n_cand, svc_fwd_req.gr_n_data, dropped_pages); + else if (dropped_pages > 0 && !gsp_phys_gr_fill_hold) + pr_info_ratelimited("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u dropped page(s); fill hold disabled (log-only) -> launch proceeds\n", + n_cand, svc_fwd_req.gr_n_data, dropped_pages); else pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved (reloc 0x%llx+)\n", n_cand, svc_fwd_req.gr_n_data, reloc_base); } +static u32 svc_gr_inline_retarget_log; + +/* + * SESSION 153-CONTINUED-91: delivery follows the launch. + * + * The captured program-family inline records (the real-upload payloads and + * the authored stage images) are keyed to the program family of the section + * that captured them, but that family is sticky guest state: the client + * cycles push slots whose content names a different program family per slot + * (observed live: records at 0x4350000/0x4300000 while the launched window + * binds 0x42d0000/0x41c0000/0x4400000). The record pages are noted into the + * launch page set, so the delivery lands on pages no bind names, the + * engine's own bind VAs keep whatever stale content the guest memory holds, + * and the DS latches `Shader Program Header N Error` (r1297: zero + * INLINE-APPLY, fetch page all zeros). + * + * Re-root every record onto the launch's own bind targets by in-page + * offset: the window's own binds (prog_bind_va) first, then the launch's + * program family at the record's own offset (which is also where the + * synthesized j=5/j=2 stage binds sit, family+0x0 / family+0x200). A record + * that continues the previous record's page (the page-crossing tail of a + * split real-upload) follows its head's new target so the split delivery + * stays contiguous. A record already on a build-bind page is left alone; + * the pass is idempotent and a no-op whenever the families already agree. + */ +static void svc_gr_inline_retarget(u64 prog_va, const u64 *binds, u32 n_binds) +{ + u64 lfam = prog_va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 prev_page = 0, prev_new = 0; + u32 i, moved = 0; + + if (!lfam || !svc_gr_cand.n_inline) + return; + for (i = 0; i < svc_gr_cand.n_inline && i < SVC_GR_INLINE_MAX; i++) { + struct svc_gr_inline *inl = &svc_gr_cand.inline_recs[i]; + u64 page = inl->dst_va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 off = inl->dst_va & (SEMA_PAGE_SIZE - 1); + u64 target = 0; + u32 b; + + if (!page) + continue; + /* Page-crossing tail: follow the head's new location. */ + if (prev_page && page == prev_page + SEMA_PAGE_SIZE && !off) { + u64 nv = (prev_new & ~(u64)(SEMA_PAGE_SIZE - 1)) + + SEMA_PAGE_SIZE; + + if (nv != inl->dst_va) { + inl->dst_va = nv; + moved++; + } + prev_page = page; + prev_new = inl->dst_va; + continue; + } + /* A build bind whose in-page offset matches this record. */ + for (b = 0; b < n_binds && b < SVC_GR_CAND_PROG_BINDS; b++) { + if (binds[b] && + (binds[b] & (SEMA_PAGE_SIZE - 1)) == off) { + target = binds[b]; + break; + } + } + /* Else the launch's own family at the record's offset. */ + if (!target) + target = lfam + off; + if ((target & ~(u64)(SEMA_PAGE_SIZE - 1)) != page) { + inl->dst_va = (target & ~(u64)(SEMA_PAGE_SIZE - 1)) + + (inl->dst_va - page); + moved++; + } + prev_page = page; + prev_new = inl->dst_va; + } + if (moved && svc_gr_inline_retarget_log < 16u) { + svc_gr_inline_retarget_log++; + pr_info("sema-gr: INLINE-RETARGET prog=0x%llx lfam=0x%llx binds=%u moved=%u/%u\n", + prog_va, lfam, n_binds, moved, + svc_gr_cand.n_inline); + for (i = 0; i < svc_gr_cand.n_inline && i < 8u; i++) + pr_info("sema-gr: inl[%u] dst=0x%llx len=%u\n", i, + svc_gr_cand.inline_recs[i].dst_va, + svc_gr_cand.inline_recs[i].len); + } +} + static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, const struct svc_channel *ch) { @@ -13040,6 +32918,11 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.guest = false; svc_fwd_req.gr = true; svc_fwd_req.gr_bridge_done = false; + /* STALLFREE: fresh request - re-arm the software-completion latch and + * stamp the fill time for the GR-STALL waited_ms accounting. */ + svc_fwd_req.gr_swrel_done = false; + svc_fwd_req.gr_rel_unobserved = false; + svc_fwd_req.gr_fill_jiffies = jiffies; svc_fwd_req.defer = true; svc_fwd_req.relay = false; svc_fwd_req.relay_index = 0; @@ -13050,6 +32933,22 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_push_n = svc_gr_cand.push_n; svc_fwd_req.gr_push_qmd_hdr = svc_gr_cand.push_qmd_hdr; svc_fwd_req.gr_push_qmd_end = svc_gr_cand.push_qmd_end; + /* r929 (Path A): carry the 3D method-window identity to the backend. */ + svc_fwd_req.gr_stream_3d = svc_gr_cand.stream_3d; + svc_fwd_req.gr_stream_off = svc_gr_cand.stream_off; + svc_fwd_req.gr_stream_n = svc_gr_cand.stream_n; + svc_fwd_req.gr_n_prog_binds = svc_gr_cand.n_prog_binds; + { + u32 pb; + + for (pb = 0; pb < svc_gr_cand.n_prog_binds && + pb < SVC_GR_CAND_PROG_BINDS; pb++) { + svc_fwd_req.gr_prog_bind_va[pb] = + svc_gr_cand.prog_bind_va[pb]; + svc_fwd_req.gr_prog_bind_slot[pb] = + svc_gr_cand.prog_bind_slot[pb]; + } + } memcpy(svc_fwd_req.gr_raw, svc_gr_cand.raw, sizeof(svc_fwd_req.gr_raw)); svc_fwd_req.gr_qmd_dev_va = svc_gr_cand.qmd_dev_va; @@ -13064,7 +32963,12 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, u64 qpa = 0; u32 qap = SVC_APERTURE_SYS; - if (!svc_resolve_va(vgpu, svc_gr_cand.qmd_dev_va, &qpa, &qap)) { + if (!svc_resolve_va(vgpu, svc_gr_cand.qmd_dev_va, &qpa, &qap) || + (gsp_sema_gr_xlate && + !svc_walk_vaspaces(vgpu, + svc_gr_cand.qmd_dev_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + &qpa, &qap))) { svc_fwd_req.gr_qmd_pa = qpa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_qmd_ap = (u8)svc_ap_of(qap); @@ -13112,6 +33016,15 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_param_size = svc_gr_cand.param_size; memcpy(svc_fwd_req.gr_qmd, svc_gr_cand.qmd, sizeof(svc_fwd_req.gr_qmd)); + /* + * SESSION 153-CONTINUED-91: re-root the captured program-family inline + * records onto this launch's own bind targets before they are copied + * into the request (see svc_gr_inline_retarget). The candidate's + * records are retargeted in place so the page-note loop below and + * svc_gr_scan_inline() both follow the delivered VAs. + */ + svc_gr_inline_retarget(svc_gr_cand.prog_va, svc_gr_cand.prog_bind_va, + svc_gr_cand.n_prog_binds); svc_fwd_req.gr_n_inline = svc_gr_cand.n_inline; if (svc_fwd_req.gr_n_inline > SVC_GR_INLINE_MAX) svc_fwd_req.gr_n_inline = SVC_GR_INLINE_MAX; @@ -13124,12 +33037,75 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_gr_page_note(vgpu, svc_gr_cand.param_va, 32); if (svc_gr_cand.prog_va) svc_gr_page_note(vgpu, svc_gr_cand.prog_va, 48); + /* + * SESSION 140: the QMD page itself carries the launch's pointer + * VALUES (QMD words 32..49 live inside it - the param, the + * pointer-region pairs and the program). Without this note the fill + * never scans it, so the pointer values (r790: word 34 = + * 0xdff0febc00) can never become candidates and n_data stays 0 + * (refused: degenerate data set). Noted under word 34 so the + * 32..48-even bank-page scan covers it. + */ + if (svc_gr_cand.qmd_dev_va) + svc_gr_page_note(vgpu, svc_gr_cand.qmd_dev_va, 34); for (i = 0; i < svc_gr_cand.n_pointer; i++) svc_gr_page_note(vgpu, svc_gr_cand.pointer_va[i], svc_gr_cand.pointer_word[i]); for (i = 0; i < svc_gr_cand.n_inline; i++) svc_gr_page_note(vgpu, svc_gr_cand.inline_recs[i].dst_va, SVC_GR_QMD_WORD_NONE); + /* + * SESSION 153-CONTINUED-57: attr-prelude operand pages. The captured + * attr/pipeline prelude (see svc_attr_prelude_capture) binds + * SET_PIPELINE_PROGRAM_ADDRESS pairs whose VAs were set in earlier + * windows; the launch builder relocates the emitted pairs to + * shadow-pool VAs - which requires each page to be IN this page set, + * or gsp_phys_rmcall_gr_3d_reloc MISSes and the emitted address stays + * a raw client VA the DS cannot fetch an SPH through. Append them + * (deduped by page inside svc_gr_page_note), soft: an unresolved or + * overflowing page is rolled back so the fill's ENOENT/overflow + * checks cannot refuse the launch over a prelude nicety. + */ + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u) { + u32 apn = 0, ai; + const u64 *aps = gsp_sema_attr_prelude_ptrs(&apn); + static u32 svc_attr_ptr_log; + + for (ai = 0; ai < apn; ai++) { + u32 before = svc_fwd_req.gr_n_pages; + + if (svc_fwd_req.gr_n_pages + 2u >= SVC_GR_PAGES_MAX) + break; + svc_gr_page_note(vgpu, aps[ai], SVC_GR_QMD_WORD_NONE); + if (svc_fwd_req.gr_n_pages > before && + !svc_fwd_req.gr_pages[svc_fwd_req.gr_n_pages - 1].pa) + svc_fwd_req.gr_n_pages = before; + else if (svc_attr_ptr_log < 16) { + svc_attr_ptr_log++; + pr_info("sema-gr: attr-prelude page va=0x%llx noted (n_pages=%u)\n", + aps[ai], svc_fwd_req.gr_n_pages); + } + } + } + if (gsp_sema_dbg && svc_gr_pages_log < 24) { + svc_gr_pages_log++; + pr_info("sema-gr: cand h=0x%x param=0x%llx prog=0x%llx n_ptr=%u inline=%u n_pages=%u push=0x%llx\n", + svc_gr_cand.h_object, svc_gr_cand.param_va, + svc_gr_cand.prog_va, svc_gr_cand.n_pointer, + svc_gr_cand.n_inline, svc_fwd_req.gr_n_pages, + svc_gr_cand.push_va); + for (i = 0; i < svc_gr_cand.n_pointer && i < 8; i++) + pr_info("sema-gr: ptr[%u] va=0x%llx word=%u\n", i, + svc_gr_cand.pointer_va[i], + svc_gr_cand.pointer_word[i]); + for (i = 0; i < svc_fwd_req.gr_n_pages && i < 12; i++) { + const struct svc_gr_page *pg = + &svc_fwd_req.gr_pages[i]; + + pr_info("sema-gr: page[%u] va=0x%llx pa=0x%llx ap=%u qmd_word=%u\n", + i, pg->va, pg->pa, pg->ap, pg->qmd_word); + } + } if (svc_fwd_req.gr_page_overflow || !svc_fwd_req.gr_n_pages) return -E2BIG; for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { @@ -13137,12 +33113,28 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, if (pg->pa) continue; - if (pg->qmd_word == SVC_GR_QMD_WORD_NONE) - return -ENOENT; - if (!svc_gr_page_inline_covers(pg->va)) + if (pg->qmd_word == SVC_GR_QMD_WORD_NONE || + !svc_gr_page_inline_covers(pg->va)) { + if (svc.fwd_gr_enoe_log < 8) { + svc.fwd_gr_enoe_log++; + pr_info("vgsp-work: GR fill ENOENT va=0x%llx qmd_word=%u inline=%u covered=%d n_pages=%u push=0x%llx\n", + pg->va, pg->qmd_word, + svc_gr_cand.n_inline, + svc_gr_page_inline_covers(pg->va), + svc_fwd_req.gr_n_pages, + svc_gr_cand.push_va); + } return -ENOENT; + } } - svc_gr_data_fill(vgpu); + /* + * GRFIX FIX1c: the fill hosts the self-enforcing decision point and + * therefore needs the channel's per-launch bound record. The const + * view exists for the read-only gates; the channel is the pump's + * mutable live bookkeeping object and this runs on the same + * service thread. + */ + svc_gr_data_fill(vgpu, (struct svc_channel *)ch); return 0; } @@ -13181,6 +33173,635 @@ static void svc_gr_req_bridge_fallback(struct nv_vgpu_device *vgpu) */ static void svc_handle_map_add(void); +/* + * r543: one bounded deferral per push VA for a partially-resolved argument + * set (see svc_fwd_req.gr_arg_partial). The latch keeps a genuine junk + * pointer word from adding a retry window to every launch: the first time a + * push is seen with a partial argument set it is deferred (pins land within + * ~1-2 s in the healthy case); the same push VA seen again inside the + * window proceeds through the historical forward. A re-used ring slot after + * the window gets its own deferral. + * + * r548: the latch is a bounded per-VA table, not a single slot. A client + * with TWO in-flight launches (victory_saxpy r547: push VAs A and B + * alternating re-decodes) made the single slot ping-pong: every call saw + * "a different VA than last time" and deferred again - both launches + * deferred forever, the client's cudaDeviceSynchronize never completed + * (held launches, zero GR LAUNCH lines, zero GR-COMPLETE). The table gives + * each VA exactly one deferral inside the window; the second sighting of + * any VA proceeds (bounded, cannot hang), interleaved launches included. + */ +#define SVC_GR_ARG_DEFER_MS 30000u +/* + * r554: 64, not 8. The latch must remember every launch push VA of the + * current client (the UVM pool allocates ~8-10 channels and each command + * buffer reuses two launch push slots, so >8 distinct VAs are in rotation); + * with 8 slots each new VA evicted another and the "one deferral per VA" + * guarantee thrashed - r553 soak: victory_reduce re-deferred the same push + * VA (0x20043be64) every pass for 430 s ("partial argument set: 7-8 pages + * unresolved") and never forwarded. 64 covers a full client with margin; + * the table is per-boot tiny (64 * 16 B). + */ +#define SVC_GR_ARG_DEFER_SLOTS 64 +static u64 svc_gr_arg_defer_va[SVC_GR_ARG_DEFER_SLOTS]; +static unsigned long svc_gr_arg_defer_j[SVC_GR_ARG_DEFER_SLOTS]; +static u32 svc_gr_arg_defer_pos; + +static bool svc_gr_arg_defer_ok(u64 push_va) +{ + u32 i; + + for (i = 0; i < SVC_GR_ARG_DEFER_SLOTS; i++) { + if (svc_gr_arg_defer_va[i] == push_va && + time_before(jiffies, svc_gr_arg_defer_j[i] + + msecs_to_jiffies(SVC_GR_ARG_DEFER_MS))) + return false; /* deferred once already: forward */ + } + i = svc_gr_arg_defer_pos++ % SVC_GR_ARG_DEFER_SLOTS; + svc_gr_arg_defer_va[i] = push_va; + svc_gr_arg_defer_j[i] = jiffies; + return true; +} + +/* + * r556: the ABSOLUTE per-launch deferral cap. Every defer path (r541 + * dstpend, r543 partial args, m150 data-stalled hold, busy-defer re-holds) + * is individually bounded, but the bounds interact and re-arm across + * re-decode passes; the r553/r555 boots show clients still walking these + * loops for 400+ s (victory_reduce 430 s, victory_saxpy 254 s at write + * time, work_guard deferred climbing by the thousand). This is the single + * backstop that makes "a launch can never be deferred for more than + * SVC_GR_DEFER_CAP_MS" true regardless of which guard keeps winning: the + * first sighting of a push VA arms an absolute deadline; once it passes, + * svc_gr_defer_expired() returns true and every defer site forwards + * best-effort. 15 s is well above the 0.5-2 s pin latency the deferrals + * exist to cover and well below every client-side timeout (>= 60 s). + */ +#define SVC_GR_DEFER_CAP_MS 15000u +static u64 svc_gr_defer_cap_va[SVC_GR_ARG_DEFER_SLOTS]; +static unsigned long svc_gr_defer_cap_j[SVC_GR_ARG_DEFER_SLOTS]; +static u32 svc_gr_defer_cap_pos; + +static bool svc_gr_defer_expired(u64 push_va) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_ARG_DEFER_SLOTS; i++) { + if (svc_gr_defer_cap_va[i] == push_va) + return !time_before(jiffies, svc_gr_defer_cap_j[i] + + msecs_to_jiffies(SVC_GR_DEFER_CAP_MS)); + } + i = svc_gr_defer_cap_pos++ % SVC_GR_ARG_DEFER_SLOTS; + svc_gr_defer_cap_va[i] = push_va; + svc_gr_defer_cap_j[i] = jiffies; + return false; +} + +/* + * SESSION 153-CONTINUED-57: first-launch prelude-maturity gate. + * + * r1094's second failure mode: the FIRST beef launch of the boot ran with + * `MME-PRELUDE emitted gen=2 words=9` (the client's setup push had not + * become readable yet; the capture matured to 443 words only ~90 s later, + * after the premature launch had already poisoned the lane with MME + * ILLEGAL_OPCODE). The 5 ms sked sampler recorded `mme_esr=80000004 + * mme_mac=0x30` at +5 ms. The mirror of that boot (r1091) read the setup + * push BEFORE the first draw and launched with words=512: MME clean. + * + * This gate withholds a beef forward while the captured MME prelude is + * immature (no macro program yet), so the first window that reaches the + * engine carries the client's real setup. The pump's busy-defer machinery + * holds the entry and retries; once the setup push lands (the setup-hold + * demand/pin path is already armed for it) the capture matures and the + * launch proceeds. Bounded: after sema_gr_prelude_gate_ms the gate stops + * firing for the boot (fail-open, one loud log) so a client that never + * uploads a macro program cannot livelock the forward path. + */ +static unsigned gsp_sema_gr_prelude_gate_ms = 60000u; +module_param_named(sema_gr_prelude_gate_ms, gsp_sema_gr_prelude_gate_ms, + uint, 0644); +MODULE_PARM_DESC(sema_gr_prelude_gate_ms, + "Withhold the first beef GR forwards while the captured MME prelude is immature (no 0x114 macro program yet); fail-open after this many ms (default 60000, 0 = off)"); +static unsigned long svc_gr_prelude_gate_j; +static u64 svc_gr_prelude_gate_defer; +static u64 svc_gr_prelude_gate_exp; + +/* + * SESSION 153-CONTINUED-62: immature-MME skip. The prelude gate above + * HOLDS the forward while the client's macro program is missing, but its + * bounded fail-open window used to forward anyway - and a window whose + * CALL_MME_MACRO runs have no program poisons the engine on contact + * (r1094 and r1114: `MME-PRELUDE emitted gen=1 words=9` -> + * mme_esr=0x80000004 ILLEGAL_OPCODE -> grfifo=0xbadf1201, whole lane + * wedged for the boot). The client needs no engine action to advance + * (the software release lanes author every completion it waits on), so an + * immature-MME beef window is CONSUMED with the software release instead + * of being forwarded into the poison. The moment the setup push lands the + * capture matures and the launches proceed normally. Default 1; A/B with + * 0 (the historical fail-open forward). + */ +static unsigned gsp_sema_mme_immature_skip = 1u; +module_param_named(sema_mme_immature_skip, gsp_sema_mme_immature_skip, + uint, 0644); +MODULE_PARM_DESC(sema_mme_immature_skip, + "Software-complete a beef 3D window instead of forwarding it while the captured MME prelude has no macro program yet (avoids the ILLEGAL_OPCODE engine poison the fail-open forward causes; default 1)"); + +/* + * SESSION 153-CONTINUED-59: attr-program settle gate. The first beat + * window of a capture generation binds program pairs (method 0x2014/0x2054) + * whose operand pages must be IN the launch page set or the builder's + * shadow relocation MISSes and the emitted address stays a raw client VA + * the engine cannot fetch (r1104: `pair addr=0x2054 va=0x4350900 -> MISS` + * -> MMU Fault HUBCLIENT_PD @0x4350000 -> guest channel reset). The pages + * resolve through the pin demand path with a multi-second latency, so a + * launch that runs within a second of the attr prelude's program-VA set + * last growing races that resolution. This gate withholds beef forwards + * for sema_attr_settle_ms after the set last grew (bounded by the same + * gate window as the prelude gate), giving the pins time; the busy-defer + * keeps the candidate so the launch goes out once settled. + */ +static unsigned gsp_sema_attr_settle_ms = 8000u; +module_param_named(sema_attr_settle_ms, gsp_sema_attr_settle_ms, uint, 0644); +MODULE_PARM_DESC(sema_attr_settle_ms, + "Withhold beef GR forwards for this long after the attr prelude program-VA set last grew, so window pair pages resolve before the first launch (default 8000, 0 = off)"); +static unsigned long svc_attr_settle_j; +static u32 svc_attr_settle_n; + +static bool svc_gr_attr_settle_wait(void) +{ + u32 apn = 0; + + if (!gsp_sema_attr_settle_ms) + return false; + (void)gsp_sema_attr_prelude_ptrs(&apn); + if (!apn) + return false; + if (apn != svc_attr_settle_n) { + svc_attr_settle_n = apn; + svc_attr_settle_j = jiffies; + } + if (!svc_gr_prelude_gate_j) + svc_gr_prelude_gate_j = jiffies; + if (time_after(jiffies, svc_gr_prelude_gate_j + + msecs_to_jiffies(gsp_sema_gr_prelude_gate_ms))) + return false; /* overall fail-open with the prelude gate */ + return time_before(jiffies, svc_attr_settle_j + + msecs_to_jiffies(gsp_sema_attr_settle_ms)); +} + +static bool svc_gr_prelude_gate_active(const struct svc_channel *ch) +{ + if ((ch->h_object & 0xfffff000u) != 0xbeef0000u) + return false; + /* + * SESSION 153-CONTINUED-59: attr-program settle first (fires even + * with the MME prelude already mature - that is the r1104 state). + */ + if (svc_gr_attr_settle_wait()) + return true; + if (!READ_ONCE(svc_mme_prelude_flag) || !gsp_sema_gr_prelude_gate_ms) + return false; + if (gsp_sema_mme_prelude_mature()) + return false; + if (!svc_gr_prelude_gate_j) + svc_gr_prelude_gate_j = jiffies; + if (time_after(jiffies, svc_gr_prelude_gate_j + + msecs_to_jiffies(gsp_sema_gr_prelude_gate_ms))) { + if (!svc_gr_prelude_gate_exp) { + svc_gr_prelude_gate_exp = 1; + pr_warn("vgsp-work: prelude gate EXPIRED h=0x%x (MME prelude still immature after %u ms; forwarding anyway)\n", + ch->h_object, gsp_sema_gr_prelude_gate_ms); + } + return false; + } + return true; +} + +/* + * r604: the r543 arg-defer latch and the r556 absolute defer cap are keyed + * by push VA ALONE, and the UVM pool reuses the same launch push VAs for + * every client of a boot - the exact lifetime bug the session flush already + * fixes for the r565 exactly-once memories (svc_gr_cap_hist, svc_gr_seen, + * svc_inline_once: "matches on the VA ALONE ... a previous client's + * capture could classify a later client's fresh launch"). Carried across + * sessions, the r556 stamp arms once - at the previous client's first + * sighting - so a new client's first decision for a reused VA already reads + * defer_expired()=true: every partial-argument defer fails closed (r586 + * path), the engine never launches, the QMD release is authored in + * software, and the client reads stale/zero buffers. Witnessed r603 family + * boot: the first mail client (fresh VAs 0x20043be94/0x20043c364) deferred + * and launched (`GUEST GR LAUNCH OK`), while every later mail client (VAs + * 0x20043beac/0x20043c37c reused from earlier boot runs) saw defer_exp=1 + * at its FIRST decision and `GR forward REFUSED at defer cap ... argument + * pages still unresolved` for both launches; its "green" was the previous + * client's results still sitting in the reused device pages. + * + * Reset both tables with the session: within a session the original + * bounds stand unchanged (one deferred sighting per VA in the arg window, + * one absolute 15 s cap per VA lifetime); a new session starts with fresh + * push generations and must not inherit either stamp. + */ +static void svc_gr_defer_reset(void) +{ + memset(svc_gr_arg_defer_va, 0, sizeof(svc_gr_arg_defer_va)); + memset(svc_gr_arg_defer_j, 0, sizeof(svc_gr_arg_defer_j)); + svc_gr_arg_defer_pos = 0; + memset(svc_gr_defer_cap_va, 0, sizeof(svc_gr_defer_cap_va)); + memset(svc_gr_defer_cap_j, 0, sizeof(svc_gr_defer_cap_j)); + svc_gr_defer_cap_pos = 0; +} + +/* + * r548: is a forward still OWED for this push VA - a captured, never + * offered candidate sitting in the candidate ring? The r547 saxpy stall: + * both launches were deferred by the r541/r543 guards, the candidate ring + * held them as unoffered, and the later passes re-decoded the entries as + * REPLAYS (LAUNCH-SKIP; already captured=1). The replay consume then + * dropped the entries without forwarding them and without authoring any + * completion (no GR LAUNCH, no GR-COMPLETE-SW) - the client's + * cudaDeviceSynchronize waited forever while the pump re-walked the ring. + * The pump's no-capture hold must therefore also fire for a replay whose + * forward is owed, forcing one fresh capture + request per bounded window. + */ +static bool svc_gr_cand_push_owed(u64 push_va) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + !svc_gr_cand_ring[i].offered && + svc_gr_cand_ring[i].push_va == push_va) + return true; + return false; +} + +/* + * r548: last-resort resolution of an owed forward when the bounded + * no-capture hold expires without the launch ever being submitted - the + * client's completion word must not be left to a launch that never ran. + * Author the candidate's QMD release in software (forward-only) and mark + * the candidate consumed so a later pass can never forward it late onto + * refilled buffers (the r59 double-run class). Returns true when a pending + * candidate for this channel+push VA was resolved. + */ +static bool svc_gr_owed_resolve(struct nv_vgpu_device *vgpu, + const struct svc_channel *ch, u64 push_va) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) { + struct svc_gr_cand_rec *c = &svc_gr_cand_ring[i]; + + if (!c->valid || c->offered || + c->push_va != push_va || + c->h_object != ch->h_object) + continue; + svc_gr_sw_release(vgpu, ch->h_object, c->rel_va, + c->rel_payload); + c->offered = true; + svc.gr_owed_resolved++; + return true; + } + return false; +} + +/* + * r541: does any of the filled launch's data pages carry an unexecuted + * device write (see svc_dstpend_note)? Runs on the service thread right + * after the fill, so svc_fwd_req.gr_data holds this launch's records. + */ +static bool svc_gr_dstpend_overlap(void) +{ + u32 i; + + for (i = 0; i < svc_fwd_req.gr_n_data; i++) + if (svc_dstpend_hit(svc_fwd_req.gr_data[i].va)) + return true; + return false; +} + +/* + * r568: a CE channel entry with a withheld release is a device write that + * has NOT executed yet. A GR launch must not read any of the client's + * device pages while such a write is pending. The page-level dstpend guard + * cannot cover this when the launch's argument base was dropped + * (GRFILL-DROP): the launch's data set then does not contain the buffer's + * pages, so the overlap check is blind to them - the r571 mail VADD read b + * 1.5 s before the b H2D executed (copy-in 319818.68, b copy 319820.16) and + * computed c = a + 0. Gate the launch on the coarser but complete + * condition: any active opdef slot whose channel is a CE channel means at + * least one copy/fill is unexecuted. Bounded by the same busy-defer + + * absolute 15 s cap machinery as every other guard, so a genuinely stuck + * copy delays a launch by the cap and no more. GR channels are excluded + * (a launch's own withheld release must not defer itself). + */ +static bool svc_copy_pending(void) +{ + unsigned long flags; + bool pending = false; + u32 i, c; + + for (i = 0; i < SVC_OPDEF_SLOTS && !pending; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + + if (!e->active || !e->n) + continue; + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) + if (svc.chan[c].valid && + svc.chan[c].vgpu == e->vgpu && + svc.chan[c].h_object == e->h_object) { + if (svc.chan[c].engine_type) + pending = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + } + return pending; +} + +/* + * r590: this pass's channel snapshot, so the CE watermark guard can read the + * pump's CURRENT consumption position (local.last_get) instead of the live + * table's previous-pass merge. The pump is single-threaded and walks CE + * channels before GR ones, so a GR decision always sees every CE channel's + * consumed position for this pass. + */ +static struct svc_channel *svc_pass_local; +static u32 svc_pass_local_n; + +/* + * r590: does any ring of this device still hold entries this pass's pump has + * not consumed? Reads the ring's live GP_PUT from its own USERD and compares + * it (modulo the ring size) with the snapshot's last_get. A mismatch means + * the client submitted copies/fills the pump has not decoded yet - exactly + * the stream-order backlog the r541/r568 guards cannot see. + * + * r607 (r606 red-draw capture, final form of r590b): the engine_type==0 + * filter skipped the channel the guard exists for. The driver's H2D + * relocation copies ride a COMPUTE-class channel (h=0x5c000059, eng=0 - + * the r595 finding), and the captured red draw shows the launch forwarding + * while that channel held three published, unconsumed entries behind an + * opdef operand hold: + * 362177.927 GR-DEC push=0x20043c394 dropped=9 partial=1 dstpend=0 + * copypend=0 ceback=0 + * 362178.001 peek-backlog h=0x5c000059 eng=0 put=95 last=92 + * 362182.145 peek-backlog h=0x5c000059 eng=0 put=95 last=92 + * 362182.556 GR LAUNCH submitted (forwarded at the one-shot arg-defer + * retry) -> VADD bad=32735 csum=8139264 (c = a + 0) + * 362182.882 ce-soft-copy va=d_b executed, 326 ms AFTER the forward + * The chain was held by `sema-pend: hold h=0x5c000059 idx=90 ... (operand + * pending)`; the wait group (0x20480fff0 pay=84) passed every pass, so the + * only blindness was the class filter - ceback=0 while the copy channel + * was three entries behind. Consider every other channel of the device; + * the launching channel itself is excluded (its own ring legitimately runs + * ahead with future entries). r590's validated part is unchanged: same + * pass snapshot, same bounded busy-defer, historical forward at the + * absolute cap (this guard itself never refuses; the r586 fail-closed + * path is a separate branch that fires only when argument pages never + * resolve). r595's GR-class ops admission has since cut the hold + * resolution to 0.5-4.7 s (the captured 4.7 s), well inside the 15 s cap. + * + * Channels whose USERD is unreadable are skipped (their work cannot be + * ordered by this mechanism anyway). + * + * r611: only channels that carry transfer ops are considered (ever_ops). + * The r607 blanket "every other ring" caught sibling GR LAUNCH channels + * whose rings hold the client's own future launches: streams2's two stream + * channels (h=0x5c00001f/23) each deferred on the other's published entry + * for the full 15 s cap, then the r586 fail-closed refused both kernels + * (measured r610c: every streams2 launch REFUSED, bad=65534). A launch + * channel decodes no ops (measured r606/r609/r610c: ops appear only on the + * 0x5c000059 copy family and the CE ranks), so the flag separates them + * cleanly while keeping the r606 red-draw copy channel in scope. + */ +static bool svc_ce_stream_backlog(struct nv_vgpu_device *vgpu, + const struct svc_channel *self) +{ + u32 k; + + if (!svc_pass_local || !svc_pass_local_n) + return false; + for (k = 0; k < svc_pass_local_n; k++) { + const struct svc_channel *c = &svc_pass_local[k]; + u32 put = 0; + + if (!c->valid || (vgpu && c->vgpu != vgpu)) + continue; + if (self && c->h_object == self->h_object) + continue; + if (!c->ever_ops) + continue; + if (!c->gpfifo_entries || !c->userd_pa) + continue; + if (svc_read_ap(vgpu ? vgpu : c->vgpu, + c->userd_pa + SVC_USERD_GP_PUT, + c->userd_as, &put, sizeof(put))) + continue; + put %= c->gpfifo_entries; + if (put != ((u32)c->last_get % c->gpfifo_entries)) { + /* + * r638 mutual-cycle gate. The r612/r636 captures + * showed a provable deadlock: a launch L is deferred + * by this watermark because channel X holds an + * unconsumed entry; X's head entry is held by the + * bounded wait gate because its acquire's release is + * authored by L itself, which cannot forward until + * this watermark clears - both halves cite each + * other, and only the 15 s defer cap / the wait's + * fail-open break it (the copy runs ~20 s late; the + * launch finds its operand pages empty). Such a head + * is a consumer-side wait, not a producer backlog: + * skip it and let the forward install the release. + * Bounded by the same guards the watermark's + * consumers already trust: skip only while the + * candidate's operands are fully resolved (no + * stalled fill pages - a launch that still needs + * producer data stays protected) and no decoded + * device write is pending on a data page. Rollback: + * sema_mutual_skip=0. + */ + if (gsp_sema_mutual_skip && + svc_gr_forward_pending() && + !svc_gr_data_stalled && + !svc_gr_dstpend_overlap() && + svc_wait_held_entry(c->h_object, + (u32)c->last_get % c->gpfifo_entries)) { + svc.ce_mutual_skips++; + if (svc.ce_mutual_log < 32) { + svc.ce_mutual_log++; + pr_info("vgsp-work: watermark skip h=0x%x head=%u (head is wait-gate held; forward unblocks its release)\n", + c->h_object, + (u32)c->last_get % + c->gpfifo_entries); + } + continue; + } + return true; + } + } + return false; +} + +/* + * r598 diag: the driver's ring WRITE pointer, not the GPFIFO put. The r568 + * draw needs the copy entries that are already written into the ring but + * whose GP_PUT has not been published yet. When the put says the ring is + * drained, peek at the slots immediately after the consumption position and + * log their {va,len} plus whether the slot already executed that exact + * entry. Bounded per launch, gated by gsp_phys_gr_peek. + */ +u32 gsp_phys_gr_peek; + +static void svc_gr_peek_lookahead(struct nv_vgpu_device *vgpu, + const struct svc_channel *self) +{ + u32 k, logged = 0, backlog_logged = 0; + + if (!gsp_phys_gr_peek || !svc_pass_local) + return; + /* + * r604 diag: 8 was too small. svc_pass_local carries the six CE ranks + * (engine_type 11/12/13) FIRST and the GR-class channels after, so the + * cap was always spent before the copy-carrying GR-class channels + * (0x5c000059 family - the r595 finding: "the driver's relocation / + * H2D copies live on a compute channel") could be logged. Session 48 + * concluded "the copy channel's lookahead read empty" from captures + * where that channel was never in the logged set at all. 24 covers + * every drained channel of a live client with margin; the lookahead + * depth 6 covers the copy behind the launch's own small pushes. + * + * r604: also name every channel the drained-only filter SKIPS because + * its live GP_PUT reads ahead of the pass consumption position. Those + * are exactly the channels the r590 CE watermark watches - but that + * guard skips engine_type==0, so a published backlog on a GR-class + * copy channel is invisible to every existing guard; the capture must + * show it so a red decision can be attributed to write-ahead (put == + * last, entries logged below) vs published-backlog (put != last, + * logged here) without guessing. + */ + for (k = 0; k < svc_pass_local_n && logged < 24; k++) { + const struct svc_channel *c = &svc_pass_local[k]; + u32 put = 0, j; + + if (!c->valid || c->h_object == self->h_object || + !c->gpfifo_va || !c->gpfifo_entries || !c->userd_pa) + continue; + if (svc_read_ap(vgpu, c->userd_pa + SVC_USERD_GP_PUT, + c->userd_as, &put, sizeof(put))) + continue; + put %= c->gpfifo_entries; + if (put != ((u32)c->last_get % c->gpfifo_entries)) { + /* real published backlog */ + if (backlog_logged < 12) { + backlog_logged++; + pr_info("vgsp-gr: peek-backlog h=0x%x eng=%u put=%u last=%u\n", + c->h_object, c->engine_type, put, + (u32)c->last_get); + } + continue; + } + pr_info("vgsp-gr: peek h=0x%x eng=%u put=%u last=%u\n", + c->h_object, c->engine_type, put, (u32)c->last_get); + for (j = 1; j <= 6; j++) { + u32 idx = ((u32)c->last_get + j) % c->gpfifo_entries; + u8 ent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len; + u64 va; + + if (svc_read_va(vgpu, + c->gpfifo_va + + (u64)idx * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va || !len || len > SVC_PUSH_MAX) + break; + pr_info("vgsp-gr: peek +%u idx=%u va=0x%llx len=%u ident_match=%d\n", + j, idx, va, len, + c->slot_ops_ident[idx] == + ((u64)e0 | ((u64)e1 << 32))); + } + logged++; + } +} + +/* + * r576 DIAG: the GR launch-decision witness (gsp_phys_gr_dec_log=N budget + * per boot). Logged from svc_gr_request right after the fill, for launches + * whose fill saw dropped pages / a partial argument set or whose decision + * point carries a dstpend/copy-pending verdict - i.e. exactly the draws the + * r566/r567/r568 guards can defer or (on their blind spots) forward. The + * data-page list is the operand evidence: each VA -> guest PA the engine + * will read; a wrong-but-valid PA here (vs the page the client's H2D wrote + * for the same VA) is the failure mechanism the readback lottery draws. + */ +unsigned gsp_phys_gr_dec_log_max; +static unsigned svc_gr_dec_log; + +static void svc_gr_dec_dump(const struct svc_channel *ch) +{ + bool dstpend, copypend, ceback; + u32 i, n = 0; + + if (svc_gr_dec_log >= gsp_phys_gr_dec_log_max) + return; + svc_gr_dec_log++; + + dstpend = svc_gr_dstpend_overlap(); + copypend = svc_copy_pending(); + ceback = svc_ce_stream_backlog(NULL, ch); + + pr_info("vgsp-gr: GR-DEC h=0x%x push=0x%llx n_data=%u dropped=%u partial=%d stalled=%d dstpend=%d copypend=%d ceback=%d defer_exp=%d\n", + ch->h_object, svc_gr_cand.push_va, svc_fwd_req.gr_n_data, + svc_fwd_req.gr_dropped, + svc_fwd_req.gr_arg_partial ? 1 : 0, + svc_gr_data_stalled ? 1 : 0, + dstpend ? 1 : 0, copypend ? 1 : 0, + ceback ? 1 : 0, + svc_gr_defer_expired(svc_gr_cand.push_va) ? 1 : 0); + + /* + * r577: raised from 32 to 256 pages - the mail VADD's fill carried + * n_data=969 pages and the red-draw divergence (a wrong-but-similar + * input page) can sit beyond the first 32 pages. + */ + for (i = 0; i < svc_fwd_req.gr_n_data && i < 256; i++) + pr_info("vgsp-gr: GR-DEC data[%u] va=0x%llx pa=0x%llx ap=%u reloc=0x%llx\n", + i, svc_fwd_req.gr_data[i].va, + svc_fwd_req.gr_data[i].pa, svc_fwd_req.gr_data[i].ap, + svc_fwd_req.gr_data[i].reloc_va); + + for (i = 0, n = 0; i < svc_dstpend_n && n < 16; i++) { + if (!svc_dstpend[i].va_page) + continue; + n++; + pr_info("vgsp-gr: GR-DEC dstpend va=0x%llx age_ms=%u\n", + svc_dstpend[i].va_page, + jiffies_to_msecs(jiffies - svc_dstpend[i].since)); + } + + for (i = 0, n = 0; i < SVC_OPDEF_SLOTS && n < 8; i++) { + if (!svc_opdef[i].active || !svc_opdef[i].n) + continue; + n++; + pr_info("vgsp-gr: GR-DEC opdef h=0x%x entry=%u n=%u age_ms=%u\n", + svc_opdef[i].h_object, svc_opdef[i].entry_idx, + svc_opdef[i].n, + jiffies_to_msecs(jiffies - svc_opdef[i].since)); + } +} + static bool svc_gr_request(struct nv_vgpu_device *vgpu, const struct svc_channel *ch) { @@ -13193,14 +33814,43 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * The candidate is per-channel: a capture left unconsumed by an * earlier pass (e.g. the entry was held before its request) must never * be queued against a different channel's handle by that channel's - * own end-of-pass request call. + * own end-of-pass request call. fail4: with several reserved ring + * slots, select THIS channel's oldest unconsumed candidate first. */ + if (gsp_phys_gr_forward_enabled()) { + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % SVC_GR_CAND_RING; + + if (svc_gr_cand_ring[rp].valid && + !svc_gr_cand_ring[rp].offered && + svc_gr_cand_ring[rp].h_object == ch->h_object) { + svc_gr_cand_pos = rp; + break; + } + } + } if (!gsp_phys_gr_forward_enabled() || !svc_gr_cand.valid || - svc_gr_cand.h_object != ch->h_object) + svc_gr_cand.h_object != ch->h_object) { + if (svc.fwd_gr_nocand_log < 8) { + svc.fwd_gr_nocand_log++; + pr_info("vgsp-work: GR request without candidate h=0x%x (forward=%d cand_valid=%d cand_h=0x%x offered=%d)\n", + ch->h_object, gsp_phys_gr_forward_enabled(), + svc_gr_cand.valid, svc_gr_cand.h_object, + svc_gr_cand.offered); + } return false; + } svc_gr_data_stalled = false; /* m150: only the fill sets this */ svc.fwd_gr_req++; if (!gsp_phys_backend_gr_ready() || !gsp_phys_guest_mem_enabled()) { + svc.fwd_gr_nordy++; + if ((svc.fwd_gr_nordy & 0x3F) == 1) + pr_info("vgsp-work: GR request refused: backend not ready h=0x%x ready=%d gmem=%d n=%llu\n", + ch->h_object, gsp_phys_backend_gr_ready(), + gsp_phys_guest_mem_enabled(), + svc.fwd_gr_nordy); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; svc_gr_bridge_fallback(vgpu); @@ -13211,6 +33861,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * request's done/hw_rc are still being read. The capture falls back. */ if (svc_fwd_hold.active) { + svc.fwd_gr_holdbusy++; + if ((svc.fwd_gr_holdbusy & 0x3F) == 1) + pr_info("vgsp-work: GR request busy: hold active h=0x%x (hold ch=0x%x n=%llu)\n", + ch->h_object, svc_fwd_hold.ch_handle, + svc.fwd_gr_holdbusy); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; /* @@ -13231,6 +33886,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, if (svc_fwd_paused || test_and_set_bit_lock(SVC_FWD_INFLIGHT, &svc_fwd_state)) { mutex_unlock(&svc_fwd_lock); + svc.fwd_gr_inflight++; + if ((svc.fwd_gr_inflight & 0x3F) == 1) + pr_info("vgsp-work: GR request busy: inflight h=0x%x (state=0x%lx n=%llu)\n", + ch->h_object, svc_fwd_state, + svc.fwd_gr_inflight); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; if (!gsp_cuda_bridge_enabled()) { @@ -13241,6 +33901,52 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, return false; } /* + * SESSION 153-CONTINUED-57: first-launch prelude-maturity gate (see + * svc_gr_prelude_gate_active). Defer the forward while the captured + * MME setup is incomplete: the candidate survives (busy-defer), the + * pump re-decodes the entry and retries, and the launch goes out + * with the client's real macro program instead of poisoning the + * lane. Skipped for every non-beef channel and after the bounded + * fail-open window. + */ + if (svc_gr_prelude_gate_active(ch)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc_gr_prelude_gate_defer++; + if (svc_gr_prelude_gate_defer <= 16 || + (svc_gr_prelude_gate_defer & 0x3F) == 1) + pr_info("vgsp-work: prelude gate h=0x%x push=0x%llx deferred (MME prelude immature; capture will gate this launch until the setup push lands) n=%llu\n", + ch->h_object, svc_gr_cand.push_va, + svc_gr_prelude_gate_defer); + return false; + } + /* + * SESSION 153-CONTINUED-62: the gate's fail-open edge (see + * gsp_sema_mme_immature_skip). Past the gate window a beef window + * whose macro program is STILL missing must not be forwarded - the + * engine poisons on the first CALL_MME_MACRO without a program. + * Consume it with the software release instead; the client advances, + * and the lane survives until the setup push matures the capture. + */ + if (READ_ONCE(gsp_sema_mme_immature_skip) && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + READ_ONCE(svc_mme_prelude_flag) && + !gsp_sema_mme_prelude_mature()) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc.fwd_gr_refused++; + svc_gr_prelude_gate_defer++; + if (svc_gr_prelude_gate_defer <= 24 || + (svc_gr_prelude_gate_defer & 0xFF) == 1) + pr_info("vgsp-work: MME-IMMATURE-SKIP h=0x%x push=0x%llx (macro program not captured yet; window software-completed, no engine contact) n=%llu\n", + ch->h_object, svc_gr_cand.push_va, + svc_gr_prelude_gate_defer); + svc_gr_bridge_fallback(vgpu); + return false; + } + /* * m143: drain the host injection queue before resolving the launch's * pages. The watcher's pins for the client's a/b/c buffers can land * between the service loop's map_add drain and this capture; the fill @@ -13257,13 +33963,272 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, mutex_unlock(&svc_fwd_lock); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; - if ((svc.fwd_gr_fallback & SVC_FWD_LOG_MASK) == 1) + if (svc.fwd_gr_refused < 8 || + (svc.fwd_gr_fallback & SVC_FWD_LOG_MASK) == 1) pr_info("vgsp-work: GR forward request refused rc=%d h=0x%x push=0x%llx; bridge fallback\n", rc, ch->h_object, svc_gr_cand.push_va); svc_gr_bridge_fallback(vgpu); return false; } /* + * r576 DIAG: witness the decision inputs of the draws the guards + * care about (see svc_gr_dec_dump). + */ + if (gsp_phys_gr_dec_log_max && + (svc_fwd_req.gr_dropped > 0 || svc_fwd_req.gr_arg_partial || + svc_gr_data_stalled || svc_gr_dstpend_overlap() || + svc_copy_pending() || svc_ce_stream_backlog(vgpu, ch))) + svc_gr_dec_dump(ch); + /* + * r541: launch-ordering guard. The fill's data pages are what the + * engine will read; if any of them is the target of a device copy + * that decoded but has NOT executed yet (svc_dstpend_note - the + * smiley/streams2 class: the client's memcpy completed on its + * release while the H2D op was still held, and the launch read the + * PREVIOUS page), do not forward: return the busy-defer verdict so + * the pump holds the entry through its bounded retry window and + * re-captures once the copy has run (or the dstpend window expired, + * which restores the historical forward). Bounded by the same + * busy-defer + gr_hold machinery as every other busy refusal, so it + * can never hang the launch. + */ + if (svc_gr_dstpend_overlap()) { + if (!svc_gr_defer_expired(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_dstpend++; + if (svc.fwd_gr_dstpend_log < 32) { + svc.fwd_gr_dstpend_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (data page has an unexecuted device write; bounded retry)\n", + ch->h_object, svc_gr_cand.push_va); + } + return false; + } + /* + * r583: the absolute defer cap expired while a data page still + * overlaps a pending device write (r581 saxpy live: GR-DEC + * dstpend=1 defer_exp=1 -> forwarded anyway -> bad=524121 + * with dstpend ages 9.5-85 s). Never forward a KNOWN-stale + * operand: fail-closed - author the candidate's QMD + * completion in software (forward-only, latched so the + * consume tail cannot author it twice) and refuse the + * engine launch. The client completes; the engine stays + * away from a page whose writer never ran. Witnessed. + */ + svc.fwd_gr_refused++; + svc.fwd_gr_closed++; + if (svc_gr_cand.rel_va) { + svc_gr_sw_release(vgpu, ch->h_object, + svc_gr_cand.rel_va, + svc_gr_cand.rel_payload); + svc_gr_cand.rel_done = true; + } + /* + * r633: deliver the parked push-level release records too - a + * fail-closed refusal still completes the client's tracking + * word (forward-only; a record already applied is skipped). + */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + u8 parked_n = 0; + + if (svc_grpark_get(vgpu, ch->h_object, + svc_gr_cand.push_va, parked, + &parked_n)) { + svc_fwd_rel_apply_all(vgpu, ch, parked, + parked_n); + svc_grpark_drop(vgpu, ch->h_object, + svc_gr_cand.push_va); + } + } + svc_gr_cand.offered = true; + svc_gr_cand.valid = false; + if (svc.fwd_gr_closed_log < 32) { + svc.fwd_gr_closed_log++; + pr_warn("vgsp-work: GR forward REFUSED at defer cap h=0x%x push=0x%llx (data page still has an unexecuted device write; fail-closed, QMD release authored in software)\n", + ch->h_object, svc_gr_cand.push_va); + } + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + return false; /* no busy-defer: the pump consumes; the + * candidate is owned and sw-completed */ + } + /* + * r590: the CE submission watermark. A launch must not be forwarded + * while a CE ring still holds entries the pump has not consumed yet: + * those are the client's stream-ordered copies, and an entry the + * pump has not decoded cannot be seen by the r541 dstpend guard + * (decoded copies only) nor by the r568 opdef gate (decoded holds + * only). The r589 mail VADD drew exactly this: the h_b H2D entries + * sat behind a translation-held CE entry, the pump's last read of the + * ring's GP_PUT was 92 while the guest had already submitted 95, the + * GR channel was processed later in the same pass and the launch was + * forwarded with the b page still zero (c = a + 0). + * + * The watermark is read LIVE (USERD GP_PUT) against THIS pass's + * consumption position (the pump runs CE channels before GR ones, so + * local.last_get already includes everything this pass consumed). + * r588's proxy failed because it compared the live table's stale + * last_get against a put sampled in a different epoch; both sides of + * this comparison come from the same channel and the same pass. + * + * Bounded by the normal busy-defer window; at the absolute cap the + * historical forward stands (witnessed) - the guard delays a launch, + * it never refuses one, so it cannot recreate r588's mail refusal. + */ + svc_gr_peek_lookahead(vgpu, ch); + if (svc_ce_stream_backlog(vgpu, ch)) { + if (!svc_gr_defer_expired(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_dstpend++; + if (svc.fwd_gr_dstpend_log < 32) { + svc.fwd_gr_dstpend_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (CE ring has unconsumed entries; stream-order watermark)\n", + ch->h_object, svc_gr_cand.push_va); + } + return false; + } + svc.fwd_gr_cewm_cap++; + if (svc.fwd_gr_cewm_log < 32) { + svc.fwd_gr_cewm_log++; + pr_warn("vgsp-work: GR forward at defer cap h=0x%x push=0x%llx (CE ring still has unconsumed entries; historical forward)\n", + ch->h_object, svc_gr_cand.push_va); + } + } + /* + * r568: the coarse companion gate - any unexecuted CE write at all + * (see svc_copy_pending). Covers the dropped-argument-base case the + * page-level guard cannot see (the r571 mail b-copy). + */ + if (svc_copy_pending() && + (svc_fwd_req.gr_arg_partial || svc_fwd_req.gr_dropped > 0) && + !svc_gr_defer_expired(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_dstpend++; + if (svc.fwd_gr_dstpend_log < 32) { + svc.fwd_gr_dstpend_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (a CE write is unexecuted; bounded retry)\n", + ch->h_object, svc_gr_cand.push_va); + } + return false; + } + /* + * r543: a partially-resolved ARGUMENT set must not forward either - + * the engine would run with a stale/zero operand (the r542 matmul + * wedge: 2 of 13 high arg bases resolved 0 pages, the fill proceeded + * "fill hold disabled (log-only)", the engine fetched the QMD and + * wrote nothing: NOWRITE-AMBIGUOUS fe_esr=0x80000000, client read + * zeros). Defer once per push VA through the bounded busy-defer + * retry so the pins can land; the latch (svc_gr_arg_defer_ok) keeps + * a genuinely junk pointer word from paying the window on every + * launch. After the retry the historical forward stands - the guard + * delays, it cannot hang. + */ + if (svc_fwd_req.gr_arg_partial && !svc_gr_data_stalled && + !svc_gr_defer_expired(svc_gr_cand.push_va) && + svc_gr_arg_defer_ok(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_argdefer++; + if (svc.fwd_gr_argdefer_log < 32) { + svc.fwd_gr_argdefer_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (partial argument set: %u page(s) unresolved; bounded retry)\n", + ch->h_object, svc_gr_cand.push_va, + svc_fwd_req.gr_dropped); + } + return false; + } + /* + * r586: the defer cap expired with argument pages still unresolved - + * fail-closed like the r583 dstpend cap. The historical path + * forwarded the launch with the partial set and the engine read + * stale/zero operands (r585 boot saxpy: n_data=969 dropped=7 + * partial=1 -> forwarded at the cap -> bad=1047527 csum=130523290; + * r584 matmul: n_data=2 dropped=18 -> r564 degenerate refusal -> + * no-write). Refuse the forward, author the completion in software, + * consume; the dropped pages were already demanded through the pin + * channel (r585) so the retry path had its chance first. + * + * r612: refuse only a DEGENERATE fill - one whose dropped set is at + * least as large as the resolved set (r584's 2 resolved / 18 + * dropped). A launch with a full resolved window (969 resolved / + * 9 dropped) whose unresolved candidates are still unresolved at + * the absolute cap has them stalled in EVERY source for the whole + * window; they are the param page's decoy/foreign pointers (the + * r611 streams2 first-client capture: candidates 0x783299000000 / + * 0x783297000000, `stalled VA` from the first fill, no pin, no + * walk, no ledger, never resolving in the run) - waiting longer + * cannot help, and refusing converts a launch whose real operands + * all resolved into a guaranteed no-write client failure (streams2 + * bad=65534). At the cap the copy-related guards (dstpend, CE + * watermark) have already cleared or the refusal branches above + * win first, so a non-degenerate fill forwards historically, + * loudly witnessed. + */ + if (svc_fwd_req.gr_arg_partial && !svc_gr_data_stalled && + svc_gr_defer_expired(svc_gr_cand.push_va) && + svc_fwd_req.gr_n_data < svc_fwd_req.gr_dropped) { + svc.fwd_gr_refused++; + svc.fwd_gr_closed++; + if (svc_gr_cand.rel_va) { + svc_gr_sw_release(vgpu, ch->h_object, + svc_gr_cand.rel_va, + svc_gr_cand.rel_payload); + svc_gr_cand.rel_done = true; + } + /* r633: deliver the parked push-level release records too. */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + u8 parked_n = 0; + + if (svc_grpark_get(vgpu, ch->h_object, + svc_gr_cand.push_va, parked, + &parked_n)) { + svc_fwd_rel_apply_all(vgpu, ch, parked, + parked_n); + svc_grpark_drop(vgpu, ch->h_object, + svc_gr_cand.push_va); + } + } + svc_gr_cand.offered = true; + svc_gr_cand.valid = false; + if (svc.fwd_gr_closed_log < 32) { + svc.fwd_gr_closed_log++; + pr_warn("vgsp-work: GR forward REFUSED at defer cap h=0x%x push=0x%llx (argument pages still unresolved: %u; fail-closed, QMD release authored in software)\n", + ch->h_object, svc_gr_cand.push_va, + svc_fwd_req.gr_dropped); + } + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + return false; /* no busy-defer: the pump consumes; the + * candidate is owned and sw-completed */ + } + /* + * r612: see the refusal note above - a non-degenerate fill at the cap + * forwards historically (witnessed once per budget slot). + */ + if (svc_fwd_req.gr_arg_partial && !svc_gr_data_stalled && + svc_gr_defer_expired(svc_gr_cand.push_va)) { + svc.fwd_gr_argfw++; + if (svc.fwd_gr_argfw_log < 32) { + svc.fwd_gr_argfw_log++; + pr_info("vgsp-work: GR forward at defer cap h=0x%x push=0x%llx (partial argument set: %u dropped of %u resolved; stalled candidates never resolve, forwarding historically)\n", + ch->h_object, svc_gr_cand.push_va, + svc_fwd_req.gr_dropped, + svc_fwd_req.gr_n_data); + } + } + /* * m150: the fill found high argument pointers that no source can * translate (the second-client zeros class). Do NOT forward a launch * whose kernel would dereference unresolved pointers: leave the @@ -13272,7 +34237,50 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * the channel's gr_hold window at the call site. Once the window * expires the entry is consumed exactly as before. */ - if (svc_gr_data_stalled) { + if (svc_gr_data_stalled && gsp_phys_gr_fill_hold) { + u64 lva = svc_fwd_req.gr_push_va; + /* + * The channel is the pump's mutable live bookkeeping object + * (the const view exists for the read-only gates); these + * record fields are the same ones the pump's hold writes, + * on this same service thread. + */ + struct svc_channel *mch = (struct svc_channel *)ch; + + /* + * GRFIX FIX1b (r213): the ABSOLUTE per-launch bound. The + * first hold of this push VA arms a fixed deadline + * (svc_gr_hold_bound_expired) that nothing re-arms; once it + * passes, the launch PROCEEDS exactly as the pre-8b058884 + * path - the candidate is consumed below and forwarded + * best-effort - so the guest is never blocked by a page that + * never resolves. r213: a slot-keyed deadline was re-armed + * on every re-decode (the same launch is replayed at + * different GPFIFO slots after rewinds) and held one launch + * for 216+ s until the client's 240 s timeout killed it. + * GRFIX FIX1e: gated by gsp_phys_gr_fill_hold (default 0) - + * with the gate off the fill never stalls on drops, so this + * block only ever sees the historical aggregate stall. + */ + if (svc_gr_hold_bound_expired(mch, lva)) { + svc.fwd_gr_hold_exp++; + if (!mch->gr_hold_exp_logged) { + mch->gr_hold_exp_logged = true; + pr_info("vgsp-gr: GRFILL-EXPIRED launch=0x%llx dropped=%u waited_ms=%u\n", + lva, svc_fwd_req.gr_dropped, + jiffies_to_msecs(jiffies - + mch->gr_hold_dead_since)); + } + svc_gr_data_stalled = false; /* proceed: forward */ + if (svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-gr: GRFILL-PROCEED launch=0x%llx (bound expired; forwarding best-effort)\n", + lva); + } + } + } + if (svc_gr_data_stalled && + !svc_gr_defer_expired(svc_gr_cand.push_va)) { clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); mutex_unlock(&svc_fwd_lock); svc.fwd_gr_refused++; @@ -13284,6 +34292,38 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, return false; /* candidate kept: the pump retries the entry */ } /* + * r633: this launch may have been held in an earlier pass whose decode + * recorded its push-level release records - records that are per-decode + * state and were parked at the hold's break. Restore them now so the + * caller's hold-engage holds THIS push's records (a queued launch from + * the capture ring is not accompanied by a re-decode, so the current + * svc_fwd_rel[] can be empty or belong to another entry; the r632 mail + * hang: the vadd push's tracking release never landed and the client + * spun in cudaDeviceSynchronize). + */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + u8 parked_n = 0; + + if (svc_grpark_get(vgpu, ch->h_object, + svc_gr_cand.push_va, parked, &parked_n)) { + if (parked_n > SVC_FWD_REL_MAX) + parked_n = SVC_FWD_REL_MAX; + memcpy(svc_fwd_rel, parked, sizeof(svc_fwd_rel)); + svc_fwd_rel_n = parked_n; + svc_fwd_defer_armed = true; + svc_grpark_drop(vgpu, ch->h_object, + svc_gr_cand.push_va); + if (svc_grpark_log < 32) { + svc_grpark_log++; + pr_info("vgsp-work: GR parked releases restored h=0x%x push=0x%llx n=%u va0=0x%llx pay0=%u\n", + ch->h_object, svc_gr_cand.push_va, + parked_n, parked[0].va, + parked[0].payload); + } + } + } + /* * The candidate is consumed: the request copy owns it from here, so a * later capture/replay cannot re-offer it and the bridge is only * reached through the request's own latch. @@ -13657,7 +34697,7 @@ static bool svc_push_read_displaced(struct nv_vgpu_device *vgpu, u32 h, static void svc_entry_outcome(const char *what, const struct svc_channel *ch, u64 idx, u64 va, u32 len) { - if (!gsp_sema_dbg || !ch->engine_type || svc.gpfifo_entry_log >= 512) + if (!gsp_sema_dbg || svc.gpfifo_entry_log >= 16384) return; svc.gpfifo_entry_log++; pr_info("sema-entry: %s h=0x%x idx=%llu va=0x%llx len=%u want=%u exp=%u unread=%d\n", @@ -13725,8 +34765,1425 @@ static void svc_userd_sweep(struct nv_vgpu_device *vgpu, struct svc_channel *ch, ch->h_object, hi, lo); } +/* + * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. + * + * r225 evidence (witness-r225b.log): after a client was killed mid-run, a + * fresh client's GR channel (0x5c000059) was registered with a RESIDUAL + * USERD put=101 left by the dead session. Its fresh session wrote no put + * (it was stuck at ctx-init, polling MC_SERVICE_INTERRUPTS once a second), + * so the m181 fresh-put-drop recovery - which requires put < put_prev - + * never fired, and the host-side completion-counter publish stayed + * withheld (gr_pub_armed false). Meanwhile the pin-batch rewind + * (svc_handle_map_add, SVC_PIN_REWIND=8) kept yanking last_get from 101 + * back to 93 on every injection batch, so the entry loop re-walked slots + * 93..100 forever: 8 stale entries per batch, all decoding to nothing + * (rel=0, no capture) and consuming with zero delivery. The whole stall + * window shows zero captures, zero GR requests and therefore zero + * GR-STALL lines - the launch never reached the bounded submit path + * because the session epoch itself was wedged. + * + * Two signatures are detected, both keyed to NO FORWARD PROGRESS for + * SVC_GR_STALE_WEDGE_MS (the (last_get, put) pair unchanged across the + * window): + * + * 1. stale-epoch: a GR channel that still carries a residual put + * (put > 1 at this pass, publish withheld) and whose ring the host + * believes is fully consumed (last_get == put) - the fresh session + * never proved its switch with a put drop. The host applies the + * m181 recovery on a TIMEOUT instead: arm the counter publish and + * void the withheld publish state (no last_get rewind - a timeout is + * not proof the ring is the dead session's; the proven m181 rewind + * stays reserved for its put-drop signature). This is the recovery + * the killed-client rerun needs. + * + * 2. stale-replay: last_get pinned behind put while put == get (the + * producer has nothing pending) AND the loop consumed entries during + * the window (submissions moved) - i.e. the rewind is re-opening an + * already-consumed window faster than the loop closes it. The window + * is dropped (last_get = put) and the pin-batch rewind is put on a + * cooldown (wd_rewind_off_until) so the futile retry cannot pin the + * pump; the rewind budget returns later and a translation that lands + * can still retry (bounded duty cycle). + * + * Nothing here can fire on a green channel: a fresh channel has put <= 1 + * (signature 1) or consumes its first entries within milliseconds + * (submissions > 0); signature 2 additionally requires BOTH a stalled + * (last_get, put) pair for 3 s with no pending producer work AND consumed + * entries during that window, which only the rewind re-opening of a + * consumed window produces. + */ +#define SVC_GR_STALE_WEDGE_MS 3000u +static void svc_gr_stale_wedge_check(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 put, u32 get) +{ + bool residual, replay; + const char *reason = ""; + unsigned long waited; + + if (ch->engine_type || !ch->gpfifo_entries) + return; + /* New work (put advanced) clears the replay cooldown immediately. */ + if (ch->wd_rewind_off_until && ch->wd_put != put) + ch->wd_rewind_off_until = 0; + if (ch->wd_since && + (ch->wd_put != put || ch->wd_last_get != ch->last_get)) + ch->wd_since = 0; /* progress (or a new position): re-arm */ + residual = (!ch->gr_pub_armed && put > 1 && ch->last_get == put); + replay = (ch->last_get != put && put == get && + ch->wd_since && ch->submissions != ch->wd_sub); + if (!residual && !replay) { + ch->wd_since = 0; + ch->wd_put = put; + ch->wd_last_get = ch->last_get; + return; + } + if (!ch->wd_since) { + ch->wd_since = jiffies; + ch->wd_put = put; + ch->wd_last_get = ch->last_get; + ch->wd_sub = ch->submissions; + return; + } + if (time_before(jiffies, ch->wd_since + + msecs_to_jiffies(SVC_GR_STALE_WEDGE_MS))) + return; + waited = jiffies_to_msecs(jiffies - ch->wd_since); + if (residual) { + /* + * m181 recovery on timeout: the fresh session never dropped + * the residual put, so prove the epoch switch here. Arm the + * publish (the withheld completion counter was the plausible + * ctx-init wait - r225's client polled service-interrupts 1/s + * with no host edge) and void any publish state the stale + * pass advanced, exactly like m181; the guest word itself is + * only zeroed when the host authored it. No last_get rewind: + * a timeout is not proof that the ring is the dead session's + * (re-walking it could re-capture a stale launch), so the + * proven m181 rewind stays reserved for its put-drop + * signature; the client's own first put write will trigger it. + */ + ch->gr_pub_armed = true; + if (ch->counter_last) { + u64 upa = ch->release_pa ? ch->release_pa : + ch->counter_pa; + u32 zero = 0; + + /* + * r311: zero only a word authored by a DEAD + * generation. Zeroing a word this live session + * authored makes the client's UVM observe a + * downward jump (its tracker then never completes) + * while the publish re-advances it - the r311 + * vector_add wedge (pub-adv/zero cycles, client + * polling cur=0 forever). + */ + if (ch->counter_gen == svc_sess_gen) { + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: stale-recover-zero-skip h=0x%x upa=0x%llx last=%u (authored by live gen)\n", + ch->h_object, upa, + ch->counter_last); + } + ch->counter_last = 0; + } else if (upa) { + int zrc; + + /* + * r378: the zero is authored from a PA + * learned earlier (release_va -> release_pa); + * re-validate it before the 4-byte write + * lands so a freed/re-homed slot is never + * zeroed through the stale page. + */ + if (!gsp_sema_write_guard(ch->release_va, + upa, + SVC_APERTURE_SYS, + "stale-recover-zero")) + zrc = -ESTALE; + else + zrc = svc_write_ap(vgpu, upa, + SVC_APERTURE_SYS, + &zero, sizeof(zero)); + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: stale-recover-zero h=0x%x upa=0x%llx last=%u zrc=%d\n", + ch->h_object, upa, + ch->counter_last, zrc); + } + } + ch->counter_last = 0; + } + reason = "stale-epoch"; + } else { + /* Drop the rewind-reopened consumed window; cool the rewind. */ + ch->last_get = put; + ch->wd_rewind_off_until = jiffies + + msecs_to_jiffies(8u * SVC_GR_STALE_WEDGE_MS); + reason = "stale-replay"; + } + ch->wd_since = 0; + ch->wd_put = put; + ch->wd_last_get = ch->last_get; + svc.gr_stale_wedge++; + pr_warn_ratelimited("vgsp-gr: GR-STALL-stale-wedge h=0x%x va=0x%llx last=%u put=%u get=%u sub=%llu waited_ms=%u action=recover reason=%s\n", + ch->h_object, ch->gpfifo_va, (u32)ch->last_get, + put, get, ch->submissions, waited, reason); +} + +/* + * --------------------------------------------------------------------- + * SESSION 126 - FENCE FOLD: channel-anchored sweep of a GR-class + * channel's PRE-BUILT ring continuation. + * + * The Vulkan client's fence waits never clear unaided because the + * driver's kick boundary (GP_PUT) stops at the first batch while the ring + * already carries the continuation entries it pre-built for the next + * submits - and the fence's own INC-5 RELEASE group (observed + * [2005c017][0x45ba090][0][payload 1][0][0x02100001]) lives in one of + * those entries, so the pump's GP_PUT-bounded walk never decodes it + * (r715/r716: put=11, entries 12..47 carried content, no dmesg row ever + * mentioned 0x45ba). The host-side fenceword_assist wrote the wait word + * directly; this fold makes the service author it from the driver's own + * push bytes. + * + * Doctrine: unlike the disabled page-content sweep (svc_apply_page_data, + * off since the false-positive corruption incidents), this sweep is + * CHANNEL-ANCHORED: it only walks entries reachable through the ring's + * validated GPFIFO translation, only accepts entries whose bytes pass the + * strict method-stream gate, and only applies the semaphore release + * groups it finds - forward-only, through the standard pending ledger + * (svc_pending_add). No consumption bookkeeping is touched (last_get, + * put_prev, slot_done_epoch and the replay state stay as the pump left + * them), so a later real kick of the same entries decodes them normally; + * repeated sweeps are idempotent (the pending ledger dedups (va,payload) + * and the apply path is forward-only). + * + * Gated by the same live-epoch arm as the counter publish + * (ch->gr_pub_armed) so a residual ring from a previous session cannot + * author releases before the session switch is proven. + * --------------------------------------------------------------------- + */ +unsigned gsp_sema_gr_ahead_max = 64; +module_param_named(sema_gr_ahead, gsp_sema_gr_ahead_max, uint, 0644); +MODULE_PARM_DESC(sema_gr_ahead, + "GR-class channels: max GPFIFO entries beyond GP_PUT whose channel-anchored pushes are swept for semaphore releases (0=off)"); + + +static u32 svc_gr_ahead_log; +static u64 svc_gr_ahead_entries; +static u64 svc_gr_ahead_rels; +static u64 svc_gr_ahead_skips; +/* SESSION 127: releases withheld because a live candidate owns the word. */ +static u64 svc_gr_ahead_owned; +/* SESSION 127: bounded witness for the sweep-caller's launch request. */ +static u32 svc_gr_ahead_req_log; +/* SESSION 128: inline (LOAD_INLINE_DATA) writes authored by the sweep. */ +static u64 svc_gr_ahead_inline_writes; + +/* + * SESSION 129 util-channel witness: the utility channel's subchannel-2 + * LOAD_INLINE_DATA arm never fired (writes=0 on r737) while the same arm + * fired for in-push payloads on the GR channel - i.e. the failure sits in + * the split (tail-served) shape. This struct carries the per-entry story + * out of the translator for one bounded, deduplicated witness line: + * which read source answered (attempt A = the channel path, 1 = the direct + * registered-VAS walk), whether the strict parse accepted the entry, and + * what the LID arm did (saw/fired/refused a still-zero payload). + */ +struct svc_util_wit { + u32 p_src; /* attempt that won: 0 = channel, 1 = VAS walk, 0xff none */ + u32 pa_ok; /* attempt-A parse (method count) */ + u32 pb_ok; /* attempt-B parse (method count) */ + u32 t_src; /* tail read source: 0 = channel, 1 = walk, 0xff none */ + u32 t_len; /* tail length served */ + u32 t0, t1; /* SESSION 130: first two words of the served tail */ + bool saw_lid; /* a LOAD_INLINE_DATA arm was reached */ + bool fired; /* ... and authored its write */ + bool zero; /* ... but the tail payload was still all-zero (pending) */ + u32 dst; + u32 line_len; + u32 line_lc; + u32 count; +}; +static struct svc_util_wit svc_util_wit; +static u64 svc_util_pending_zero; + +/* + * SESSION 127 C5C0 translator, bounded tail materialization buffer: a + * LOAD_INLINE_DATA run whose payload overruns the entry end is served from + * the next ring entry (the driver splits large inline runs into a methods + * entry plus a dedicated data slot entry). 4 KB covers the observed + * param-bank write (1568 B) in full and the first page of the big uploads. + */ +#define SVC_AHEAD_TAIL_INLINE_MAX 32768 +static u8 svc_ahead_inline_buf[SVC_AHEAD_TAIL_INLINE_MAX]; +static u64 svc_ahead_inline_trunc; + +/* + * SESSION 128: one-copy materialization bound for the utility/CE sweep. + * The engine-side copy-in the sweep replays (a TURING_DMA_COPY_A pitch + * copy from a client-owned staging buffer into the vidmem shadow) moves + * the driver's code/constant uploads; the observed SASS/code-heap class is + * tens of KB, so 4 MB is a generous cap that still refuses a bogus + * geometry before it can walk the shadow. + */ +#define SVC_AHEAD_COPY_IN_MAX (4ULL << 20) + +/* + * SESSION 128: materialize one TURING_DMA_COPY_A pitch copy-in that an + * unprocessed copy-class channel carried - src bytes live in the client's + * staging buffer (guest RAM, resolved through the trusted walk like any + * operand page), dst is the vidmem page the driver staged the upload for + * (code heap / constant store). This is the emulated engine write for the + * copy the real CE would perform; it is once-only keyed by (push_va, + * content hash, dst, bytes) so re-scans of a stable ring are free. + */ +static u64 svc_ahead_copy_in_skip; +static void svc_ahead_copy_in(struct nv_vgpu_device *vgpu, u32 ch_h, + u64 push_va, u64 hash, u64 src_va, u64 dst_va, + u32 row, u32 rows, u32 dma) +{ + static u8 cb[65536]; + u64 bytes = (u64)row * (rows ? rows : 1); + u64 done = 0; + u64 spa = 0, dpa = 0; + int sap = SVC_APERTURE_SYS, dap = SVC_APERTURE_SYS; + int rc = 0; + bool src_ok, dst_ok; + + if (!bytes || src_va < SVC_LOW_REL_VA_LO || src_va >= SEMA_VA_HI || + dst_va < SVC_LOW_REL_VA_LO || dst_va >= SEMA_VA_HI) + return; + /* + * Only the upload direction is materialized: the source must be a + * resolved client page in sysmem/guest RAM (the staging buffer the + * client CPU-wrote), and the destination must be vidmem (resolved to + * the FB aperture) or a not-yet-resolved VA whose bytes the VA + * shadow will hold. A resolved SYS destination is a copy-OUT (the + * engine reading vidmem into a client buffer): replaying it here + * would overwrite the client's buffer with shadow garbage before the + * producing launch ran, so it is skipped - the GR channels' copy-out + * machinery owns that direction. + */ + src_ok = !svc_lookup(vgpu, src_va, &spa, &sap) && sap != SVC_APERTURE_FB; + dst_ok = svc_lookup(vgpu, dst_va, &dpa, &dap) || + dap == SVC_APERTURE_FB; + if (!src_ok || !dst_ok) { + svc_ahead_copy_in_skip++; + if (svc_ahead_copy_in_skip <= 8) + pr_info("sema-utilsweep: copy-in skipped (src_ok=%d dst_ok=%d) push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx\n", + src_ok, dst_ok, push_va, src_va, dst_va, + bytes); + return; + } + if (bytes > SVC_AHEAD_COPY_IN_MAX) { + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-utilsweep: copy-in too large push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx cap=0x%llx\n", + push_va, src_va, dst_va, bytes, + SVC_AHEAD_COPY_IN_MAX); + return; + } + if (svc_inline_once_seen(push_va, hash, dst_va, (u32)bytes)) + return; + svc_inline_once_store(push_va, hash, dst_va, (u32)bytes); + while (done < bytes) { + u32 chunk = bytes - done > sizeof(cb) ? + (u32)sizeof(cb) : (u32)(bytes - done); + + rc = svc_read_va(vgpu, src_va + done, cb, chunk); + if (rc) { + rc = -EIO; + break; + } + rc = svc_grce_inline_write(vgpu, dst_va + done, cb, chunk, + ch_h); + if (rc) + break; + done += chunk; + } + svc_gr_ahead_inline_writes++; + pr_info("sema-utilsweep: copy-in src=0x%llx -> dst=0x%llx bytes=0x%llx dma=0x%x done=0x%llx rc=%d h=0x%x\n", + src_va, dst_va, bytes, dma, done, rc, ch_h); +} + +/* One 4-byte word of a method stream that may continue into the tail. */ +static u32 svc_ahead_w32(const u8 *buf, u32 len, const u8 *tail, u32 taillen, + u32 off) +{ + u32 v = 0; + + if (off + 4 <= len) + memcpy(&v, buf + off, 4); + else if (tail && off >= len && (u64)(off - len) + 4 <= taillen) + memcpy(&v, tail + (off - len), 4); + return v; +} + +/* + * Strict method-stream parse with a one-entry tail (see the sweep comment). + * A run whose payload overruns this entry's end may be completed from the + * next ring entry; anything else invalid means the page is not a method + * stream (the same false-positive arbiter as svc_stream_full_ok). Returns + * the method count, or 0; *used_tail reports that the tail served a + * trailing data run (nothing after it is parsed). + */ +static u32 svc_ahead_parse(const u8 *buf, u32 len, const u8 *tail, u32 taillen, + bool *used_tail) +{ + u32 off = 0, methods = 0; + + *used_tail = false; + for (;;) { + u32 hdr, op, count; + u64 end; + + if (off + 4 > len) + break; + memcpy(&hdr, buf + off, 4); + if (hdr == 0) { + off += 4; + continue; + } + op = (hdr >> 29) & 0x7; + count = (hdr >> 16) & 0xfff; + switch (op) { + case 1: /* INC_METHOD */ + case 3: /* NON_INC_METHOD */ + case 5: /* ONE_INC */ + if (!count || count > 0xfff) + return 0; + end = (u64)off + 4 + (u64)count * 4; + if (end > len) { + if (!tail || (u64)(end - len) > taillen) + return 0; + *used_tail = true; + off = len; + goto done; + } + off = (u32)end; + break; + case 0: /* GRP0_USE_TERT */ + case 2: /* GRP2_USE_TERT */ + case 4: /* IMMD_DATA_METHOD */ + case 7: /* END_PB_SEGMENT */ + off += 4; + break; + default: + return 0; + } + if (++methods > 8192) + return 0; + } +done: + while (off < len && buf[off] == 0) + off++; + return (methods > 0 && (off == len || *used_tail)) ? methods : 0; +} + +/* + * SESSION 127 C5C0 translator sweep pass over one coherent push. Besides the + * original release extraction (svc_gr_ahead_push, below), it recognizes, for + * pre-built continuation entries: + * + * - the C5C0 inline-QMD window (SET_INLINE_QMD_ADDRESS_A/B + the 64-word + * LOAD_INLINE_QMD_DATA run) and feeds it to the standard svc_qmd_accum + * capture, so the physical GR forward (the CUDA QMD path) fires for a + * launch that never reaches the GP_PUT-bounded main walk - observed + * beec0100 idx23: `0x31c = 0x4b85f` + `0x320 x 64`; + * - GRCE / KEPLER_INLINE_TO_MEMORY_B LOAD_INLINE_DATA runs (op 3 at + * 0x1b4, offsets from the 0x180/0x184/0x188/0x18c group), materialized + * into the VA shadow exactly like the main decode does - observed + * beef0100 idx00 (256 B -> 0x423f000, payload in idx01) and idx08 + * (12288 B -> 0x4448000, payload in idx09), and beec0100 idx23 + * (1568 B -> 0x4b86020, payload in-push); + * - TURING_DMA_COPY_A (0xC5B5, subch >= 2) copy operands + * (0x400/0x404 in, 0x408/0x40c out, 0x410/0x414 pitch, 0x418/0x41c + * line length/count, 0x300 LAUNCH_DMA, 0x700..0x708 remap) recorded as + * pending copy-outs for the completion edge (svc_gr_copy_pend_run) - + * observed beec0100 idx24: image 0xdfd8d60000 -> readback 0x050b0000, + * 640x480 rows. + * + * Returns the number of release groups queued (as before). The buffer must + * pass the strict tail-aware parse first; a page that is not a method + * stream is not processed at all. + */ +static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + const u8 *buf, u32 len, const u8 *tail, + u32 taillen, u64 push_va, bool inline_only) +{ + static struct svc_push_state stq; /* service thread only */ + static u32 dw[66]; /* QMD window words */ + u32 off = 0, nrel = 0, methods, i; + u64 off_in = 0, off_out = 0; + u32 have_in = 0, have_out = 0; + u32 pitch_in = 0, pitch_out = 0; + u32 line_len = 0, line_count = 0; + u32 src_origin_x = 0, src_origin_y = 0; + u32 dst_origin_x = 0, dst_origin_y = 0; + u32 have_src_origin = 0, have_dst_origin = 0; + u32 last_dma = 0, last_remap = 0; + u32 cp_subch = ~0u; + bool used_tail = false; + bool forward = gsp_phys_gr_forward_enabled(); + bool xlate = gsp_sema_gr_xlate; + + if (!xlate) { + /* Historical sweep: releases from exactly-covered entries. */ + tail = NULL; + taillen = 0; + } + methods = svc_ahead_parse(buf, len, tail, taillen, &used_tail); + if (!methods) + return 0; + + memset(&stq, 0, sizeof(stq)); + stq.push_va = push_va; + + while (off + 4 <= len) { + u32 hdr, op, addr, count, subch; + u32 w0, w1, w2, w3, w4; + u64 end; + + memcpy(&hdr, buf + off, 4); + if (!hdr) { + off += 4; + continue; + } + op = (hdr >> 29) & 0x7; + addr = (hdr & 0x1fff) << 2; + count = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + if (op == 0 || op == 2 || op == 4 || op == 7) { + off += 4; + continue; + } + if (op != 1 && op != 3 && op != 5) + break; + if (!count || count > 0xfff) + break; + end = (u64)off + 4 + (u64)count * 4; + /* + * Tail-served trailing run: process from the head word, then + * stop (the tail is a data slot, nothing parses after it). + */ + w0 = svc_ahead_w32(buf, len, tail, taillen, off + 4); + w1 = svc_ahead_w32(buf, len, tail, taillen, off + 8); + w2 = svc_ahead_w32(buf, len, tail, taillen, off + 12); + w3 = svc_ahead_w32(buf, len, tail, taillen, off + 16); + w4 = svc_ahead_w32(buf, len, tail, taillen, off + 20); + + /* + * C5C0 inline-QMD window (mirrors the main decode's intercept): + * only accumulate once the run declares itself a QMD window. + */ + if (xlate && forward && !inline_only && subch >= 1 && + addr >= SVC_QMD_ADDR_A && + addr <= SVC_QMD_DATA + (SVC_QMD_WORDS - 1) * 4 && + (stq.qmd_have || + (addr <= SVC_QMD_ADDR_B && + (u64)addr + (u64)count * 4 > SVC_QMD_ADDR_A))) { + u32 n = count > 66 ? 66 : count; + + for (i = 0; i < n; i++) + dw[i] = svc_ahead_w32(buf, len, tail, taillen, + off + 4 + 4 * i); + svc_qmd_accum(vgpu, ch, &stq, addr, n, + (const u8 *)dw, buf, len, true, false); + if (end > len) + break; + off += 4 + count * 4; + continue; + } + + /* + * GRCE / INLINE_TO_MEMORY_B LOAD_INLINE_DATA: materialize the + * DMA write (the engine would do this; the emulation authors + * the bytes into the VA shadow so the forwarded launch's + * operand pages - parameter banks - are real). + */ + if (xlate && op == 3 && subch >= 1 && + addr == GRCE_M_LOAD_INLINE_DATA && + have_out && line_len) { + u64 nbytes = (u64)line_len * + (line_count ? line_count : 1); + u32 cap = count * 4; + u32 bytes, k, words; + u64 h = svc_push_hash(buf, len); + bool tail_served = end > len; + bool refuse_zero = false; + + if (nbytes > cap) + nbytes = cap; + bytes = (u32)nbytes; + if (bytes > SVC_AHEAD_TAIL_INLINE_MAX) { + bytes = SVC_AHEAD_TAIL_INLINE_MAX; + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-ahead: inline write truncated push=0x%llx dst=0x%llx want=%llu cap=%u\n", + push_va, off_out, nbytes, + SVC_AHEAD_TAIL_INLINE_MAX); + } + words = (bytes + 3) / 4; + for (k = 0; k < words; k++) + memcpy(svc_ahead_inline_buf + 4 * k, + &(u32){ svc_ahead_w32(buf, len, tail, + taillen, + off + 4 + 4 * k) }, + 4); + /* + * SESSION 129: a tail-served payload whose head is still + * all zeros means the tail source answered with an empty + * page (a stale pin shadow, or the driver has not staged + * the bytes yet) - authoring zeros over the destination + * would corrupt the code heap. Leave the write pending + * (no once-store, the sweep retries every pass) until a + * nonzero payload arrives. In-push payloads are never + * gated: the bytes are part of the flushed submit. + */ + if (inline_only && tail_served && bytes && + !memchr_inv(svc_ahead_inline_buf, 0, + min_t(u32, bytes, 16))) + refuse_zero = true; + if (inline_only) { + svc_util_wit.saw_lid = true; + svc_util_wit.dst = (u32)off_out; + svc_util_wit.line_len = line_len; + svc_util_wit.line_lc = line_count; + svc_util_wit.count = count; + if (refuse_zero) + svc_util_wit.zero = true; + } + if (refuse_zero) { + /* + * SESSION 130: a zero-headed tail-served + * payload is applied under a VARIANT once-key + * (h | 1) instead of being refused forever: + * the destination is currently empty, so + * authoring zeros is idempotent and correct + * for the driver's internal zero buffer (the + * observed 256 B utility i2m = exactly + * GF100_ZERO_BUFFER_SIZE), while a later + * non-zero payload for the same entry still + * matches its own (h) key and lands - the + * refusal can no longer stall the arm + * (r737/r739: `writes=0` with the payload + * page stably zero). + */ + svc_util_pending_zero++; + if (bytes && !svc_inline_once_seen(push_va, + h | 1, + off_out, + bytes)) { + svc_inline_once_store(push_va, h | 1, + off_out, bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, + off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + } else if (bytes && !svc_inline_once_seen(push_va, h, + off_out, + bytes)) { + svc_inline_once_store(push_va, h, off_out, + bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + /* + * Record the destination and the fresh payload bytes + * for the launch fill's inline-served pages, exactly + * like the main decode's LOAD_INLINE_DATA arm: the + * QMD capture later in this entry collects the ring + * and the fill can serve the parameter bank from it + * even while its shadow page is still empty. + */ + { + struct svc_grce_rec *rec = + &svc_grce_dst_ring[svc_grce_dst_ring_next % + SVC_GRCE_DST_MAX]; + + /* + * A refused (still-zero) tail payload is not + * recorded either: the next pass re-reads the + * tail and the real bytes land then. + */ + if (!refuse_zero) { + rec->dst_va = off_out; + rec->len = min_t(u32, bytes, + GSP_BRIDGE_INLINE_BYTES); + memcpy(rec->data, svc_ahead_inline_buf, + rec->len); + svc_grce_dst_ring_next++; + } + } + if (end > len) + break; + off += 4 + count * 4; + continue; + } + + /* + * GRCE / INLINE_TO_MEMORY operand group (0x180..0x18c), for + * subchannel 1 AND 2. The mem2mem subchannel (2, the + * A140/KEPLER_INLINE_TO_MEMORY_B upload lane the utility + * channels drive) carries the same operand methods as the + * compute subchannel (1). This group MUST be parsed BEFORE + * the TURING-copy branch below: that branch's `subch >= 2` + * guard also matches subchannel 2 and would swallow the + * 0x180..0x18c methods unread, leaving have_out/line_len + * unset - the exact reason the utility channel's subch-2 + * LOAD_INLINE_DATA arm never fired (r737/r739 witness: + * `lidn=1 subch=2 ... saw=0 dst=0 ll=0`). + */ + if (xlate && (subch == 1 || subch == 2) && + addr >= 0x180 && addr <= 0x18c && count) { + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x180: + line_len = vv[k]; + break; + case 0x184: + line_count = vv[k]; + break; + case 0x188: + off_out = (off_out & 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x18c: + off_out = (off_out & + ~0xffffffffULL) | vv[k]; + have_out = 1; + break; + default: + break; + } + } + } else if (xlate && subch >= 2) { + /* TURING_DMA_COPY_A operand runs (subch >= 2). */ + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + if (addr >= 0x400 && addr <= 0x41c && n) { + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x400: + off_in = (off_in & + 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_in = 1; + break; + case 0x404: + off_in = (off_in & + ~0xffffffffULL) | + vv[k]; + have_in = 1; + break; + case 0x408: + off_out = (off_out & + 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x40c: + off_out = (off_out & + ~0xffffffffULL) | + vv[k]; + have_out = 1; + break; + case 0x410: + pitch_in = vv[k]; + break; + case 0x414: + pitch_out = vv[k]; + break; + case 0x418: + line_len = vv[k]; + break; + case 0x41c: + line_count = vv[k]; + break; + default: + break; + } + } + } else if (addr >= 0x700 && addr <= 0x708 && n) { + for (k = 0; k < n; k++) { + if (addr + 4 * k == 0x700) + last_remap = (last_remap & + ~0xffffu) | + (vv[k] & 0xffffu); + else if (addr + 4 * k == 0x708) + last_remap = (last_remap & + 0xffffu) | + ((vv[k] & 0xffffu) + << 16); + } + } else if (addr >= CE_M_SRC_ORIGIN_X && + addr <= CE_M_DST_ORIGIN_Y && n) { + /* + * SESSION 153 CONT.25: the X DDX readback + * pair positions each 6-row strip with + * SRC_ORIGIN (x bytes / y rows) instead of + * folding it into OFFSET_IN - the same + * methods the main walk now decodes. The + * strip y steps 6 per push (A: 120, B: 126, + * A: 132, ...) and the copy must read + * OFFSET_IN + origin_y*PITCH_IN. + */ + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case CE_M_SRC_ORIGIN_X: + src_origin_x = vv[k]; + have_src_origin = 1; + break; + case CE_M_SRC_ORIGIN_Y: + src_origin_y = vv[k]; + have_src_origin = 1; + break; + case CE_M_DST_ORIGIN_X: + dst_origin_x = vv[k]; + have_dst_origin = 1; + break; + case CE_M_DST_ORIGIN_Y: + dst_origin_y = vv[k]; + have_dst_origin = 1; + break; + } + } + } else if (addr == 0x300 && count) { + u32 dma = w0; + + /* + * DATA_TRANSFER_TYPE bits 1:0; 0 = NONE (a + * release-only / semaphore op - no data + * movement, nothing to record). + * + * SESSION 153 CONT.25: remember a full setup + * (LINE_LENGTH + OFFSET_IN present) as the + * channel's persistent CE state and fill a + * DELTA launch (the X DDX readback pair's + * second push: no setup of its own, only + * SRC_ORIGIN + SEM_A/PAYLOAD + OFFSET_OUT + + * LAUNCH_DMA) from it, exactly like the main + * walk - without this every odd strip of the + * frame is dropped. + */ + last_dma = dma; + if (have_in && off_in && line_len) { + struct svc_push_state tst; + + memset(&tst, 0, sizeof(tst)); + tst.offset_in = off_in; + tst.pitch_in = pitch_in; + tst.pitch_out = pitch_out; + tst.line_length = line_len; + tst.line_count = line_count; + svc_ce_saved_store(&tst, + ch->h_object); + } + if ((dma & 0x3) && have_out && off_out && + (!have_in || !line_len)) { + struct svc_ce_saved *sv = + svc_ce_saved_slot( + ch->h_object); + + if (sv && sv->line_length && + sv->have_in) { + if (!line_len) { + line_len = + sv->line_length; + line_count = + sv->line_count; + } + if (!have_in) { + off_in = + sv->offset_in; + have_in = 1; + } + if (!pitch_in) + pitch_in = + sv->pitch_in; + if (!pitch_out) + pitch_out = + sv->pitch_out; + } + } + if ((dma & 0x3) && have_in && have_out && + line_len && line_count && + off_in && off_out) { + /* + * SESSION 153 CONT.25: fold the + * per-strip SRC/DST_ORIGIN into the + * operands (bytes on X, rows on Y; + * the linear shadow makes + * base + y*pitch + x the truth). + * line_len is BYTES for C5B5 + * (LINE_LENGTH_IN semantics), not + * dwords: the old *4 inflated every + * strip 4x. + */ + u64 src_f = off_in, dst_f = off_out; + + if (have_src_origin) + src_f += + (u64)src_origin_y * + (pitch_in ? + pitch_in : + line_len) + + src_origin_x; + if (have_dst_origin) + dst_f += + (u64)dst_origin_y * + (pitch_out ? + pitch_out : + line_len) + + dst_origin_x; + if (inline_only) + svc_ahead_copy_in(vgpu, + ch->h_object, push_va, + svc_push_hash(buf, + len), + src_f, dst_f, + line_len, + line_count, dma); + else if (forward) { + svc_gr_copy_pend_add(ch->h_object, + push_va, svc_push_hash(buf, + len), + src_f, dst_f, + line_len, + line_count, + pitch_out ? pitch_out : + line_len, + pitch_in, dma, + last_remap); + cp_subch = subch; + } + } + } + } + + if (!inline_only && addr == HOST_M_SEM_ADDR_LO && count >= 5) { + u64 va; + + va = ((u64)(w1 & 0xff) << 32) | + (u64)(w0 & 0xfffffffc); + op = w4 & 0x7; + if ((op == HOST_SEM_OP_RELEASE || + op == HOST_SEM_OP_REDUCTION) && w2 && + va >= SVC_LOW_REL_VA_LO && va < SEMA_VA_HI) { + /* + * A release targeting a live launch candidate's + * own QMD release word is owned by the + * completion edge (post-copy-out); sweeping it + * here would clear the client's fence before + * the engine output landed. + */ + if (xlate && forward && + svc_gr_ahead_owned_rel_va(va, push_va)) { + svc_gr_ahead_owned++; + } else { + svc_pending_add(va, w2, + op == HOST_SEM_OP_REDUCTION ? + 1 : 0); + nrel++; + } + } + } + if (end > len) + break; + off += 4 + count * 4; + } + (void)cp_subch; + return nrel; +} + +static void svc_gr_ahead_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 put) +{ + static u8 abuf[2048]; + static u8 nbuf[2048]; + u32 i, maxent, nrel = 0, swept = 0, nskip = 0; + + if (!put || !ch->gpfifo_entries) + return; + maxent = min_t(u32, gsp_sema_gr_ahead_max, 512); + for (i = 0; i < maxent; i++) { + u32 idx = (put + i) % ch->gpfifo_entries; + u8 ent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen, tail_len = 0; + u64 va; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)idx * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + break; /* unwritten slot: end of the continuation */ + if (!va || !len || len > SVC_PUSH_MAX) + break; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + break; + rlen = min_t(u32, len, sizeof(abuf)); + if (svc_read_va_push(vgpu, ch, va, abuf, rlen)) { + /* + * Unreadable here (pre-anchor translation) is a + * reason to stop this pass: the continuation is not + * proven coherent, so nothing is authored. + */ + nskip++; + break; + } + /* + * SESSION 127 C5C0 translator: read the next ring entry as + * the tail of a stream whose final inline run overruns this + * entry (the driver splits large LOAD_INLINE_DATA payloads + * into a dedicated data slot entry; observed beef0100 + * idx00->idx01 and idx08->idx09). + */ + { + u32 nidx = (put + i + 1) % ch->gpfifo_entries; + u8 nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 ne0, ne1, nlen; + u64 nva; + + if (!svc_read_va(vgpu, + ch->gpfifo_va + + (u64)nidx * SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent))) { + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + if (nva && nlen < SVC_PUSH_MAX) { + tail_len = min_t(u32, nlen, + sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, + nbuf, tail_len)) + tail_len = 0; + } + } + } + swept++; + nrel += svc_gr_ahead_push(vgpu, ch, abuf, rlen, nbuf, + tail_len, va, false); + } + svc_gr_ahead_entries += swept; + svc_gr_ahead_rels += nrel; + svc_gr_ahead_skips += nskip; + if (swept && (nrel || svc_gr_ahead_log < 64)) { + svc_gr_ahead_log++; + pr_info("sema-ahead: h=0x%x put=%u swept=%u rels=%u owned=%llu skipped=%u (total rels=%llu)\n", + ch->h_object, put, swept, nrel, svc_gr_ahead_owned, + nskip, svc_gr_ahead_rels); + } +} + +/* + * SESSION 129: read a VA straight through the registered-VAS walks only + * (newest registration first), one page at a time - the exact path + * walk_probe and sema_probe prove. The util sweep's channel read can be + * shadowed by a stale injected pin / window guess for the utility VAs + * (observed r737: the subchannel-2 LID arm never fired although the push + * page content is real and the VAS walk resolves both the methods entry + * and its payload slot); this helper is the arbitration fallback that + * bypasses those sources entirely. + */ +static int svc_util_walk_read(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len) +{ + u8 *p = buf; + + while (len) { + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + size_t off = va & (SEMA_PAGE_SIZE - 1); + size_t chunk = min_t(size_t, len, SEMA_PAGE_SIZE - off); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (svc_walk_vaspaces(vgpu, page, &pa, &ap)) + return -ENOENT; + if (svc_read_ap(vgpu, pa + off, svc_ap_of(ap), p, chunk)) + return -EIO; + p += chunk; + va += chunk; + len -= chunk; + } + return 0; +} + +/* + * SESSION 129: does this entry window carry a LOAD_INLINE_DATA header? Used + * by the bounded util-sweep witness to select only the entries that can + * exercise the arm (one per distinct content, so a stable ring logs once). + */ +static u32 svc_util_lid_scan(const u8 *buf, u32 len, u32 *first_subch) +{ + u32 k, n = 0; + + *first_subch = 0xff; + for (k = 0; k + 4 <= len; k += 4) { + u32 hdr; + + memcpy(&hdr, buf + k, 4); + if (((hdr >> 29) & 0x7) == 3 && + ((hdr & 0x1fff) << 2) == GRCE_M_LOAD_INLINE_DATA) { + if (*first_subch == 0xff) + *first_subch = (hdr >> 13) & 0x7; + n++; + } + } + return n; +} + +/* + * SESSION 128: utility-channel inline-download materialization. + * + * The driver's upload/utility channels (the MEM2MEM/A140 + C5B5 + 2D + * composite: observed h=0xbeef0100 with gpfifo=0x4020000) are registered + * with an unresolved USERD (userd_ok=0), so the put-bounded entry loop in + * svc_pump_channels never decodes their pushes and the LOAD_INLINE_DATA + * downloads they carry (the client's code-heap/SASS upload, command-stream + * staging) never land anywhere the emulation can see: the destinations + * stay zero in the vidmem shadow. + * + * This pass scans a bounded window of the channel's GPFIFO directly (the + * gpfifo VA resolves through the trusted walk even when the USERD does + * not) and feeds each entry + its successor (the source-view call carrying + * the payload) through the proven sweep decoder in inline-only mode: the + * GRCE/INLINE_TO_MEMORY operand group (0x180..0x18c on subch 1/2), + * LAUNCH_DMA and the LOAD_INLINE_DATA run are materialized by + * svc_grce_inline_write; releases, QMD windows and copy-pend records are + * skipped so nothing about the client's fence/completion state changes. + * Once-only keyed by (push_va, content hash, dst, bytes), so a stable ring + * costs only the reads. Requires sema_gr_xlate; bounded by + * sema_gr_util_scan entries per channel per pass (0 = off). + * + * SESSION 129 read arbitration: attempt A is the channel path (unchanged); + * when A fails to read, or reads content the strict parse rejects, attempt + * B re-reads the push and the tail through svc_util_walk_read and the + * parse arbitrates - a source that yields a method stream wins. The r737 + * evidence (arm never fired while the content is real) pointed at exactly + * this: a shadowed channel read consumed the entry without a parseable + * buffer while the walk resolves the true pages. + */ +static void svc_gr_util_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ +/* + * The observed utility-channel pushes are the ~17 KB setup block and the + * two split uploads (256 B command stream + 12 KB code/heap payload); a + * 64 KB window covers every one of them with room to spare while keeping + * the two static buffers modest. + */ +#define SVC_UTIL_BUF (64 * 1024) + static u8 ubuf[SVC_UTIL_BUF]; + static u8 nbuf[SVC_UTIL_BUF]; + static u8 bbuf[SVC_UTIL_BUF]; + static u8 tbuf[SVC_UTIL_BUF]; + static u64 svc_gr_util_entries; + static u64 svc_gr_util_writes; + static u64 svc_util_tail_alt; /* SESSION 130: tails re-read through + * the direct walk because the + * channel read was zero-headed */ + static u64 svc_util_tail_try; /* ... and the walk attempts (alt is + * the subset that yielded non-zero) */ + u32 i, maxent = min_t(u32, gsp_sema_gr_util_scan, 64); + u32 swept = 0, nmade = 0; + bool first_sight = false; + + if (!gsp_sema_gr_xlate || !gsp_sema_gr_util_scan) + return; + if (!ch->gpfifo_va || !ch->gpfifo_entries) { + /* SESSION 143: name every channel the sweep cannot cover. */ + u32 k; + + for (k = 0; k < svc_util_seen_n; k++) + if (svc_util_seen_keys[k] == ch->h_object) + break; + if (k == svc_util_seen_n && + svc_util_seen_n < ARRAY_SIZE(svc_util_seen_keys)) { + svc_util_seen_keys[svc_util_seen_n++] = ch->h_object; + pr_info("sema-utilsweep: skip h=0x%x eng=%u reason=no-gpfifo gpfifo=0x%llx entries=%u userd=%d\n", + ch->h_object, ch->engine_type, ch->gpfifo_va, + ch->gpfifo_entries, ch->userd_ok ? 1 : 0); + } + return; + } + { + u32 k; + + for (k = 0; k < svc_util_seen_n; k++) + if (svc_util_seen_keys[k] == ch->h_object) + break; + if (k == svc_util_seen_n && + svc_util_seen_n < ARRAY_SIZE(svc_util_seen_keys)) { + svc_util_seen_keys[svc_util_seen_n++] = ch->h_object; + first_sight = true; + pr_info("sema-utilsweep: begin h=0x%x eng=%u gpfifo=0x%llx entries=%u userd=%d cid=%u\n", + ch->h_object, ch->engine_type, ch->gpfifo_va, + ch->gpfifo_entries, ch->userd_ok ? 1 : 0, + ch->cid); + } + } + for (i = 0; i < maxent && i < ch->gpfifo_entries; i++) { + u8 ent[SVC_GPFIFO_ENTRY_SIZE], nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen; + u64 va, before; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + continue; /* unwritten slot */ + if (!va || !len || len > SVC_PUSH_MAX) + continue; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + continue; + /* + * SESSION 132 stale-translation arbitration: when the + * channel-path entry is dead (its push provably reads + * all-zero through the client's page tables) while the + * direct VAS walk answers a live entry on the same ring + * slot, the channel path is reading a stale page (the r759 + * pin false positive) - the walk's entry is the channel's + * ring. A dead entry with no live walk alternative is left + * to the existing push arbitration below. + */ + if (svc_ring_entry_dead(vgpu, e0, e1)) { + u8 went[SVC_GPFIFO_ENTRY_SIZE]; + + if (!svc_util_walk_read(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + went, sizeof(went))) { + u32 we0, we1; + u64 wva; + u32 wlen; + + memcpy(&we0, went, 4); + memcpy(&we1, went + 4, 4); + wva = ((u64)(we1 & 0xff) << 32) | + (u64)(we0 & 0xfffffffc); + wlen = ((we1 >> 10) & 0x1fffff) << 2; + if (!svc_ring_entry_dead(vgpu, we0, we1) && + wva && wlen && wlen <= SVC_PUSH_MAX && + wva >= SVC_LOW_REL_VA_LO && + wva < SEMA_VA_HI) { + e0 = we0; + e1 = we1; + va = wva; + len = wlen; + } + } + } + rlen = min_t(u32, len, sizeof(ubuf)); + /* + * SESSION 129 read arbitration + bounded witness (see the + * sweep's header comment). Attempt A: the channel path, + * unchanged. Attempt B: the direct registered-VAS walk, used + * when A cannot read or cannot parse; the strict parse then + * arbitrates which content is the method stream. + */ + { + u32 push_rc, parse_a = 0, parse_b = 0; + u32 tl_a = 0, tl_b = 0; + u32 lidn, lidsc = 0xff, slot; + u64 nva = 0; + u32 nlen = 0; + u64 hkey; + u8 *pbuf = ubuf; + bool ut = false, seen = false; + + push_rc = svc_read_va_push(vgpu, ch, va, ubuf, + rlen); + /* + * The payload of a trailing LOAD_INLINE_DATA run rides + * the next GPFIFO entry (the driver's source-view + * GPFIFO call, observed beef0100 idx00->idx01 and + * idx08->idx09): read it as the tail exactly like + * svc_gr_ahead_sweep does. + */ + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)((i + 1) % ch->gpfifo_entries) * + SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent)) == 0) { + u32 ne0, ne1; + + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + } + if (!push_rc && nva && nlen && nlen < SVC_PUSH_MAX) { + tl_a = min_t(u32, nlen, sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, nbuf, + tl_a)) + tl_a = 0; + } + if (!push_rc) + parse_a = svc_ahead_parse(ubuf, rlen, nbuf, + tl_a, &ut); + if (push_rc || !parse_a) { + bool bpush = false; + + if (nva && nlen && nlen < SVC_PUSH_MAX) { + u32 want = min_t(u32, nlen, + sizeof(nbuf)); + + if (!svc_util_walk_read(vgpu, nva, + nbuf, + want)) + tl_b = want; + } + if (!svc_util_walk_read(vgpu, va, bbuf, + rlen)) { + bpush = true; + parse_b = svc_ahead_parse(bbuf, rlen, + nbuf, tl_b, + &ut); + /* + * A's buffer is authoritative only + * when A actually read it; otherwise + * (or when only B parses) B's walk + * content is the stream. + */ + if (push_rc || + (parse_b && !parse_a)) { + pbuf = bbuf; + tl_a = tl_b; + } + } + if (push_rc && !bpush) + continue; /* unreadable both ways */ + } + /* + * SESSION 130 tail-content arbitration: a tail-served + * payload whose channel head is still all-zero is not + * the staged bytes (the page the channel translation + * points at is empty, or the read is shadowed by a + * stale pin). Try the direct registered-VAS walk - + * the same path attempt B proves - before handing + * the tail to the LID arm, whose zero-headed refusal + * would otherwise stall the upload every pass. + */ + if (tl_a && nva && + !memchr_inv(nbuf, 0, min_t(u32, tl_a, 16))) { + u32 want = tl_a; + + svc_util_tail_try++; + if (!svc_util_walk_read(vgpu, nva, tbuf, + want) && + memchr_inv(tbuf, 0, + min_t(u32, want, 16))) { + memcpy(nbuf, tbuf, want); + svc_util_tail_alt++; + } + } + memset(&svc_util_wit, 0, sizeof(svc_util_wit)); + if (tl_a >= 4) + memcpy(&svc_util_wit.t0, nbuf, 4); + if (tl_a >= 8) + memcpy(&svc_util_wit.t1, nbuf + 4, 4); + before = svc_gr_ahead_inline_writes; + (void)svc_gr_ahead_push(vgpu, ch, pbuf, rlen, nbuf, + tl_a, va, true); + if (svc_gr_ahead_inline_writes != before) + nmade++; + swept++; + /* + * Bounded witness, one line per distinct LID-carrying + * (or unparseable) entry content: which attempt was + * used, the parses, the tail resolution and what the + * LID arm did. + */ + lidn = svc_util_lid_scan(pbuf, rlen, &lidsc); + if (!lidn && parse_a) + continue; + hkey = svc_push_hash(pbuf, rlen); + for (slot = 0; slot < 32; slot++) + if (svc_util_dec_keys[slot].h == hkey && + svc_util_dec_keys[slot].va == va) { + seen = true; + break; + } + if (seen || svc_util_dec_log >= 96) + continue; + slot = svc_util_dec_log & 31; + svc_util_dec_keys[slot].va = va; + svc_util_dec_keys[slot].h = hkey; + svc_util_dec_log++; + pr_info("sema-utildec: h=0x%x i=%u va=0x%llx len=0x%x p_src=%u pa=%u pb=%u tail_va=0x%llx tail=0x%x t8=%08x %08x alt=%llu/%llu lidn=%u subch=%u saw=%d fired=%d zero=%d dst=0x%x ll=%u lc=%u f8=%08x %08x\n", + ch->h_object, i, va, rlen, + pbuf == bbuf ? 1u : 0u, parse_a, parse_b, + nva, tl_a, svc_util_wit.t0, svc_util_wit.t1, + svc_util_tail_alt, svc_util_tail_try, + lidn, lidsc, + svc_util_wit.saw_lid ? 1 : 0, + svc_util_wit.fired ? 1 : 0, + svc_util_wit.zero ? 1 : 0, + svc_util_wit.dst, svc_util_wit.line_len, + svc_util_wit.line_lc, + rlen >= 4 ? *(const u32 *)pbuf : 0, + rlen >= 8 ? *(const u32 *)(pbuf + 4) : 0); + } + } + svc_gr_util_entries += swept; + svc_gr_util_writes += nmade; + if (first_sight && !swept) + pr_info("sema-utilsweep: read=none h=0x%x gpfifo=0x%llx entries=%u\n", + ch->h_object, ch->gpfifo_va, ch->gpfifo_entries); + if (swept && (nmade || svc_gr_util_log < 128)) { + svc_gr_util_log++; + pr_info("sema-utilsweep: h=0x%x swept=%u writes=%u (total %llu/%llu)\n", + ch->h_object, swept, nmade, svc_gr_util_entries, + svc_gr_util_writes); + } +} + static void svc_pump_channels(struct nv_vgpu_device *vgpu) { + if (svc_pdb_disc_pending) { + svc_pdb_disc_pending = false; + pr_info("sema-pdb-disc: requested va=0x%llx pa=0x%llx\n", + svc_pdb_disc_va, svc_pdb_disc_pa); + svc_vas_candidates_verify(vgpu, svc_pdb_disc_va, + svc_pdb_disc_pa); + svc_xvas_pdb_discover(vgpu, svc_pdb_disc_va, + svc_pdb_disc_pa); + } /* Only ever called from the service thread. */ static struct svc_channel local[SVC_MAX_CHANNELS]; /* @@ -13740,15 +36197,124 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) bool stalled = false; spin_lock_irqsave(&svc.chan_lock, flags); + if (svc_rewind_pending) { + svc_rewind_pending = false; + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu) + continue; + if (!svc_rewind_all && c->h_object != svc_rewind_h) + continue; + pr_info("sema-chan: rewind h=0x%x last_get=%u put=%u (slot bitmap + rescue state cleared)\n", + c->h_object, + (unsigned int)c->last_get, + (unsigned int)c->put_prev); + bitmap_zero(c->slot_done_epoch, SVC_GPFIFO_MAX_BITS); + c->last_get = 0; + c->rescue_fail = 0; + c->unread_va = 0; + c->unread_until = 0; + c->defer_since = 0; + svc_gr_setup_hold_clear(c); + } + } + /* + * r707e: every-pass candidate verify with the last-good ring pair. + * Cheap (<=16 parked candidates, one walk each) and idempotent; the + * first pass after the user-VAS candidate parks registers it, so the + * client's very first push reads resolve. + */ + if (svc_lastgood_ring_pa) + svc_vas_candidates_verify(vgpu, svc_lastgood_ring_va, + svc_lastgood_ring_pa); + /* + * r507o stream-order: walk the CE channels BEFORE the GR channels. + * The client's stream order places its H2D copies before the kernel + * that consumes them, but the channels register in handle order (the + * RM's GR channel early, the copy-ce channels later), and a pass that + * decoded the launch before the CE channel's entries ran let the + * forward workqueue execute the launch's copy-in against pre-copy + * data (r480-sep: copy-in 115 ms before its stream's H2D soft + * copies; r481 proved no host semaphore links the two, so the + * emulation must enforce stream order itself). The CE copies execute + * on this same service thread, so consuming the CE entries first + * fixes the inversion directly - no cross-channel hold, no latency + * tax. Stable partition; relative order within each class preserved. + */ for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].engine_type) local[n++] = svc.chan[i]; } + /* + * r605 stream-order, the eng==0 half of the r507o fix. The same + * argument that made CE channels walk before GR ones applies WITHIN + * the GR class: the driver's relocation/H2D copies ride a + * later-registered compute-class channel (h=0x5c000059 family, + * eng=0 - the r595 finding), while the launch channel registers + * early (0x5c000019). The stable partition walked the launch first, + * so at a launch decision the copy channel's PUBLISHED backlog was + * unconsumed and invisible to every guard: the r590 watermark skips + * engine_type==0, and the r541/r568 gates only see DECODED copies. + * r604 capture (fresh boot, peek armed): at the vadd's second + * decision `peek-backlog h=0x5c000059 eng=0 put=95 last=94` - the + * b-copy entry was already in the ring with GP_PUT ahead of the + * pump, executed 8 ms after the decision and 320 ms before the + * launch's copy-in, and the run was green only because the + * partial-argument defer had stretched the window; pins landing one + * pass earlier would have forwarded against the pre-copy page + * (c = a + 0, the r568 red draw). Newest-registered first + * (descending h_object) walks the copy / push-library channels + * before the compute channels, so every launch decision sees the + * copy ring consumed to its live write pointer (GP_PUT): the + * write-pointer drain, by ordering. No deferral, no cap + * interaction, no speculative execution - the pump still consumes + * each entry exactly once through its normal per-slot identity + * bookkeeping (slot_ops_ident/done/complete/hash), so nothing needs + * mirroring. A client whose compute channel registers last simply + * keeps the historical order (no worse). + */ + { + u32 prev_h = 0xffffffffu; + int picked; + + for (picked = 0; picked < SVC_MAX_CHANNELS; picked++) { + int best = -1; + u32 best_h = 0; + + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu || + c->engine_type) + continue; + if (c->h_object >= prev_h) + continue; + if (best < 0 || c->h_object > best_h) { + best = i; + best_h = c->h_object; + } + } + if (best < 0) + break; + local[n++] = svc.chan[best]; + prev_h = best_h; + } + } spin_unlock_irqrestore(&svc.chan_lock, flags); n_gr = SVC_GR_COUNTER_TOTAL; /* + * STALLFREE: complete a failed forwarded GR launch no hold owns (see + * svc_fwd_outcome_check). Runs before this pass's channel loop, so a + * finished request is always observed before any fill can reuse its + * slot. + */ + svc_fwd_outcome_check(vgpu); + + /* * H1 last resort: one self-gated content-discovery pass for the GR * ctx-init rings before this pass's channel loop. The pins it * installs (ring VA + push VA pages, both validated from content and @@ -13779,6 +36345,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc_opdef_drop_gone(vgpu, local, n); + /* + * r590: publish this pass's snapshot for the CE submission-watermark + * guard (svc_ce_stream_backlog). CE channels were copied first, so a + * GR launch's decision sees the pass-current consumed positions. + */ + svc_pass_local = local; + svc_pass_local_n = (u32)n; + for (i = 0; i < n; i++) { struct svc_channel *ch = &local[i]; u64 ring_pa = 0; @@ -13786,6 +36360,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) int rrc; u32 put = 0, get = 0, guard = 0; u32 get_before; + u32 put_prev_wit; u64 applied_before; bool held = false; @@ -13803,6 +36378,16 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!ch->engine_type) { u32 pput = 0; + /* + * SESSION 153-CONTINUED-38: set the channel's VAS + * BEFORE the autoreg derive, not after the ring + * lookup. The counter slot must be disambiguated + * through THIS channel's page tables (two clients + * register the same counter VA; CONT-37 collision), + * and the ring lookup + USERD discovery below must + * also resolve through the channel's own VAS first. + */ + svc_cur_vas = ch->h_vaspace; svc_counter_autoreg(vgpu, ch, n_gr); svc_counter_flush_pending(vgpu, ch); /* @@ -13832,14 +36417,30 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * svc_counter_autoreg). Skip while a relay hold owns * the channel, exactly like the end-of-pass call. */ - if (ch->counter_pa && ch->userd_pa && + if ((ch->release_pa || ch->counter_pa) && + ch->userd_pa && !(svc_fwd_hold.active && svc_fwd_hold.vgpu == vgpu && svc_fwd_hold.ch_handle == ch->h_object) && !svc_read_ap(vgpu, ch->userd_pa + SVC_USERD_GP_PUT, - ch->userd_as, &pput, sizeof(pput))) + ch->userd_as, &pput, sizeof(pput))) { + if (gsp_sema_trace_h && + (u32)ch->h_object == + (u32)gsp_sema_trace_h && + time_after(jiffies, svc_perf_last)) { + svc_perf_last = jiffies + HZ; + pr_info("sema-perf: pass h=0x%x pput=%u last=%u armed=%d cnt=%u sub=%llu held=%llu starved=%llu\n", + ch->h_object, pput, + (u32)ch->last_get, + ch->gr_pub_armed, + ch->counter_last, + ch->submissions, + svc.gpfifo_held, + svc.gpfifo_starved); + } svc_counter_publish(vgpu, ch, pput); + } } /* @@ -13860,15 +36461,52 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (svc_fwd_hold.active && svc_fwd_hold.vgpu == vgpu) svc_fwd_hold_expire_check(vgpu, ch); + /* + * SESSION 153-CONTINUED-61: MISS-ABORT retry tick. A re-armed + * candidate (svc_gr_missre) is re-offered on a bounded cadence + * until the pin demand resolves its pages; the full decode path + * is deliberately NOT involved (the candidate ring slot is the + * queue). svc_gr_request no-ops for other channels and defers on + * busy/inflight. Placed before the early `continue`s so the retry + * still runs when the USERD read happens to fail this pass. + */ + svc_gr_missre_tick(vgpu, ch); + if (!ch->gpfifo_va || !ch->gpfifo_entries) continue; svc_ring_autopin(vgpu, ch); + /* + * SESSION 153-CONTINUED-75: heal a GR ring whose translation + * has gone stale (the client's churn frees the ring page and + * the allocator reuses it for data; the stale pin then feeds + * the pump phantom entries - see svc_gr_ring_stale_heal). + */ + svc_gr_ring_stale_heal(vgpu, ch); rrc = svc_lookup(vgpu, ch->gpfifo_va, &ring_pa, &ring_ap); if (!rrc) svc_pb_pair(vgpu, ch); + if (!rrc && ring_pa) { + svc_lastgood_ring_va = ch->gpfifo_va; + svc_lastgood_ring_pa = ring_pa; + } + /* + * r707f (FOLD): the moment ANY channel's ring resolves through + * the channel object, verify the parked COPY_SERVER_PDES VAS + * candidates against that content-backed pair. The candidates + * are parked at device-create time, before the client's compute + * channels first submit, so registering here lands the VAS page + * directory BEFORE the submit's entries get their first push + * read - the window the host-triggered path could not win (the + * entry's first read and the ring's first resolution happen in + * the same pass; a pair that arrives later cannot help, because + * a failed GR entry read is never retried with the fresh VAS). + */ + if (!rrc) + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + ring_pa); /* * ISSUE-0002 (r47 diag): run the deferred CE USERD sweep for a @@ -13889,6 +36527,39 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) jiffies + 27 * HZ; } + /* + * SESSION 128: utility/upload channels carry the engine's + * inline code/data downloads (LOAD_INLINE_DATA on the + * mem2mem subchannel) and the copy-class channels (eng=11/12) + * carry the driver's pitch copy-ins (staging -> code heap / + * constant store). With an unresolved USERD the put-bounded + * entry loop below never decodes them (the !userd_ok branch + * continues before it); with a resolved USERD the main + * decode's LOAD_INLINE_DATA/operand arms still only accept + * subchannel 1, so the subchannel-2 downloads are missed + * either way. Scan every channel's GPFIFO directly here and + * materialize the downloads (see svc_gr_util_sweep); the + * inline-only mode plus the once-only key make the overlap + * with the main loop and the GR sweep a no-op. + */ + if (gsp_sema_gr_util_scan && gsp_sema_gr_xlate) + svc_gr_util_sweep(vgpu, ch); + + /* + * SESSION 153-CONT-7: a kick-family channel whose USERD + * stays unresolved (the r939/r940 decoys are excluded by + * the allocation-locality gate and no in-window control + * block exists) still owes its pace publication: the GL + * client polls the publish words to pace its render loop, + * and the publish advance was otherwise reachable only + * through the decoy consume path. Author it per pass while + * the USERD is missing; once a userd exists the normal + * retire path carries it again. + */ + if (!ch->engine_type && !ch->userd_ok && + ch->h_object >= 0xbfef0000u && + ch->h_object <= 0xbfef00ffu) + svc_pub_advance(ch); if (!ch->userd_ok) { if (svc_chan_userd(vgpu, ch, rrc ? 0 : ring_pa, rrc ? -1 : ring_ap, &put, &get, @@ -13913,6 +36584,69 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * r832: the RC watchdog's notifier ack must not depend on the + * ring reads. Its VA is known from the channel's first decode + * (ch->notify2d_va, captured from SET_NOTIFY_A/B) and the ack + * is the same zeroed NvNotification the decode path writes: + * the guest re-arms status=0xFFFF every cycle and checks it + * back to 0 within 7 s. Under heavy GL traffic the watchdog + * ring's translation can go stale for stretches (its entries + * then read as zeros and are consumed as empty), which starved + * exactly this write and wedged the guest with "GPU is + * probably locked" while every other channel kept running. + * Writing the ack on every pass that sees work pending is + * idempotent. + * + * SESSION 153-CONTINUED-19: NOT for the kick family. Their + * notify2d_va is the kick record's own NOTIFY target (the + * shared completion word the GL client polls), and the kick + * lane authors that word (a completed NvNotification with a + * nonzero timestamp, svc_kick_notify_exec). The per-pass ack + * zeroed it 14029 times on r972 - the last kick write + * survived at most one pass (~60 ms) before being wiped, so + * the client's poll kept seeing "not fired". The GPU never + * clears this notification; the client does (re-arm). + */ + if (!ch->engine_type && ch->notify2d_va && put != get && + !svc_kick_family(ch)) + svc_notify2d_write(vgpu, ch, false); + /* + * SESSION 148 (r864): registration-based keep-alive that + * needs no decode - ack the driver-registered notifier when + * it is armed. Catches the stretches where the watchdog ring + * reads go unreadable and notify2d_va was never captured. + */ + if (!ch->engine_type) + svc_notify2d_keepalive(vgpu, ch); + + /* + * SESSION 149 R15 (M1.1b): GR RAM-scan keep-alive. The boot + * budget can retire before the client creates its ring (r886: + * 16 slices spent, no pairing). A GPFIFO put advance proves + * the client is live NOW, so grant a fresh bounded budget - + * capped and spaced so an unfound ring cannot keep the scan + * hot. The scan itself stops on its own once the ring is + * content-validated (gr_ram_ring). + */ + if (!ch->engine_type && ch->h_object != 0x31415900u && + ch->gr_index < 32 && + !test_bit(ch->gr_index, &svc_gr_ring_done) && + ch->gr_ram_passes >= svc_gr_ram_pass_budget() && + ch->gr_ram_refreshes < SVC_GR_RAM_REFRESH_MAX && + time_after_eq(jiffies, ch->gr_ram_refresh_at) && + ch->gr_ram_kick != put) { + ch->gr_ram_kick = put; + ch->gr_ram_refreshes++; + ch->gr_ram_refresh_at = jiffies + + msecs_to_jiffies(SVC_GR_RAM_REFRESH_MS); + ch->gr_ram_passes = 0; + pr_info("sema-chan: GR ring h=0x%x rank=%u RAM content scan keep-alive %u/%u (put=%u moved; budget reset)\n", + ch->h_object, ch->gr_index, + ch->gr_ram_refreshes, SVC_GR_RAM_REFRESH_MAX, + put); + } + + /* * ISSUE-0002 diag: put snapshots for CE channels, bounded per * session. The throttle is indexed by rank (the channel struct * iterated here is a local copy, so a per-channel stamp never @@ -13950,11 +36684,174 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->logged = true; } + /* + * mm2: a GR channel whose USERD already reads PUT <= 1 cannot + * hold a stale consumed prefix, so its publishes (the catch-up + * for releases missing from consumed entries) are safe to + * enable immediately - this is the pristine first-boot ring. + * Channels discovered with a residual put stay unarmed until + * the fresh-put-drop rewind proves the session switch, so the + * residual put can never reach the fresh client's completion + * word (see gr_pub_armed and svc_counter_publish). + */ + if (!ch->engine_type && !ch->gr_pub_armed && put <= 1 && + !ch->userd_wait_fresh) + ch->gr_pub_armed = true; + /* Initialize expected semaphore payload from the pool (if mapped). */ svc_init_expected_payload(vgpu, ch); - if (put >= ch->gpfifo_entries) + /* SESSION 153-CONTINUED-20: the family test shared by the + * kick-confirm relaxation and the churn monitor below. */ + bool kfam = svc_kick_family(ch); + + /* + * SESSION 154 (M-GL1.2): speculative-latch confirmation for + * the RAM USERD. The scan latches a gated candidate right + * away (fast walk start); the pump is the judge: a +1/+2 + * step of the block's PUT confirms it as the client's live + * control block, while a jump, a reset or no movement within + * the bound rejects it so the scan can adopt a better + * candidate. The live block advances 1->2->3 in 2-10 s + * (measured); the client's own release waits are ~11 s, so + * the no-advance bound must sit above those. + * + * SESSION 153-CONTINUED-20: for the kick family ANY positive + * PUT step confirms (its put is authored by the client's + * frame loop), only a decrease or the no-advance bound + * rejects - and a CONFIRMED kick latch is not exempt from the + * churn rule: the r974 false confirm (pa=0x11c62e000, put + * walked 1->2->3, then reset to 0/1 and toggled for minutes) + * shows a slowly resetting pool block can pass any + * single-step test. Track its PUT while confirmed and + * unlatch on the first decrease - a client-authored + * free-running put never decreases (the rewind path owns the + * session-reset case). + */ + if (kfam && ch->gr_ram_userd_confirmed && + put > ch->gr_ram_userd_latch_put) + ch->gr_ram_userd_latch_put = put; + /* + * SESSION 153-CONTINUED-38: the latch confirmation must also + * judge a block adopted through the RAM scan for a + * walk-resolved ring (gr_ram_ring false; gr_ram_userd_latch_at + * stamps the latch), or a wrong low-aperture latch would stick + * forever. + */ + if (!ch->engine_type && ch->userd_ok && + (ch->gr_ram_ring || ch->gr_ram_userd_latch_at) && + (!ch->gr_ram_userd_confirmed || + (kfam && put < ch->gr_ram_userd_latch_put))) { + bool stepped; + + if (!ch->gr_ram_userd_latch_at) + ch->gr_ram_userd_latch_at = jiffies; + stepped = kfam ? + put > ch->gr_ram_userd_latch_put : + (put == ch->gr_ram_userd_latch_put + 1 || + put == ch->gr_ram_userd_latch_put + 2); + if (stepped) { + ch->gr_ram_userd_confirmed = true; + pr_info("sema-chan: GR USERD h=0x%x pa=0x%llx speculative latch CONFIRMED (put %u -> %u)\n", + ch->h_object, ch->userd_pa, + ch->gr_ram_userd_latch_put, put); + } else if (put != ch->gr_ram_userd_latch_put || + time_after(jiffies, + ch->gr_ram_userd_latch_at + + msecs_to_jiffies(15000))) { + unsigned long lflags; + int li; + + pr_info("sema-chan: GR USERD h=0x%x pa=0x%llx speculative latch REJECTED (put=%u latch=%u%s)\n", + ch->h_object, ch->userd_pa, put, + ch->gr_ram_userd_latch_put, + put != ch->gr_ram_userd_latch_put ? + "" : " no advance in 15 s"); + /* + * SESSION 153-CONT-5: one stale window must not + * kill the pipeline. Remember the rejected + * (pa, put) so the scan never re-latches the same + * frozen block, re-arm the scan budget (bounded) + * so a REAL control block that appears later can + * still be adopted, and only then drop the USERD. + * r936: the rejection retired the 20-slice budget + * and the kick channel never re-latched anything. + */ + ch->gr_ram_userd_reject_pa = ch->userd_pa; + ch->gr_ram_userd_reject_put = + ch->gr_ram_userd_latch_put; + if (ch->gr_ram_userd_rejects < 64) { + ch->gr_ram_userd_rejects++; + ch->gr_ram_userd_passes = 0; + ch->gr_ram_userd_logged = false; + } + ch->userd_ok = false; + ch->userd_pa = 0; + ch->userd_as = 0; + ch->gr_ram_userd_latch_at = 0; + ch->gr_ram_userd_confirmed = false; + ch->gr_ram_put_seeded = 0; + ch->last_get = 0; + /* + * Persist immediately: the pump `continue`s + * here, skipping the end-of-pass write-back + * for this channel. + */ + spin_lock_irqsave(&svc.chan_lock, lflags); + for (li = 0; li < SVC_MAX_CHANNELS; li++) + if (svc.chan[li].valid && + svc.chan[li].vgpu == vgpu && + svc.chan[li].h_object == + ch->h_object) { + svc.chan[li].userd_ok = false; + svc.chan[li].userd_pa = 0; + svc.chan[li].userd_as = 0; + svc.chan[li].gr_ram_userd_passes = + ch->gr_ram_userd_passes; + svc.chan[li].gr_ram_userd_logged = + ch->gr_ram_userd_logged; + svc.chan[li].gr_ram_userd_reject_pa = + ch->gr_ram_userd_reject_pa; + svc.chan[li].gr_ram_userd_reject_put = + ch->gr_ram_userd_reject_put; + svc.chan[li].gr_ram_userd_rejects = + ch->gr_ram_userd_rejects; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, lflags); + continue; + } + } + if (put >= ch->gpfifo_entries) { + u32 put_raw = put; + put %= ch->gpfifo_entries; + /* + * SESSION 154 (M-GL1.2): a free-running producer (the + * GL client's control block counts past any small ring + * - put=12 with entries=2, measured) can collapse + * onto the pump's own position by the modulo: an even + * number of positions ahead and last_get == put, so + * the walk consumes NOTHING and the client's release + * is never applied (r914: submissions stayed 0 while + * the client retried for minutes). One-shot per put + * value: step the walk back one slot so it consumes + * the newest entry; normal slot-space tracking + * resumes from there and the next put value walks + * on. + */ + if (ch->gpfifo_entries > 1 && + put == (u32)ch->last_get && + put_raw != ch->gr_ram_put_seeded) { + ch->gr_ram_put_seeded = put_raw; + ch->last_get = (put + ch->gpfifo_entries - 1) % + ch->gpfifo_entries; + pr_info_ratelimited("sema-chan: GR free-running put h=0x%x put=%u get=%u ent=%u -> walk seed last=%u (modulo collapse consumed nothing)\n", + ch->h_object, put_raw, get, + ch->gpfifo_entries, + (u32)ch->last_get); + } + } /* * A wrapped put means every slot may hold a new entry: clear @@ -14003,12 +36900,157 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->h_object, ch->ce_index, put, (u32)ch->last_get); ch->last_get = put; + } else if (!ch->engine_type && ch->gpfifo_entries && + ch->last_get > put && + ch->submissions < ch->gpfifo_entries / 2) { + /* + * m181 (r203 GR ctx-init stall): a GR channel's + * ring/USERD live in fixed VRAM slots + * (fb 0x140000 + rank*0x3000) that keep the + * PREVIOUS session's final state across the + * session flush (m176 reset learned counter + * state but not the ring-consumption + * bookkeeping). A new session's first pump + * pass therefore sees the residual put (e.g. + * 107 = the last session's final GP_PUT) and + * consumes the stale ring up to it, advancing + * last_get before the client has written its + * own entries. The client's fresh put then + * drops below put_prev, and without this + * rewind last_get stays ahead of the producer: + * the live ctx-init entries are never decoded, + * their REPORT_SEM releases are never applied + * (the completion word freezes at the first + * stale-decoded payload), and the client's + * ctx-init wait polls forever - r203 rank 0 + * hung exactly so in multikernel / streams2 / + * victory_mail / smiley whenever the prior + * session had reached a high put (the sessions + * after a hung one, whose final put is 1, + * proceeded - a perfect alternation). Rewind + * to slot 0 like the r48 resync: a fresh + * session restarts its push sequence at slot + * 0, and re-decoding slot 0 is idempotent + * (the stale pass could only have authored the + * same payload, and the forward-only apply + * drops a duplicate). A genuine wrap cannot be + * this fresh (submissions guard), mirroring + * the r73 CE rewind. + */ + pr_info_ratelimited("sema-chan: fresh put drop (GR) h=0x%x rank=%u put=%u last=%u -> rewind to 0\n", + ch->h_object, ch->gr_index, put, + (u32)ch->last_get); + ch->last_get = 0; + /* + * mm2: the session switch is now proven - arm + * the publish for the live entries that follow, + * and void any publish state the stale pass + * advanced. Everything consumed to reach + * last_get was the dead session's ring; a + * publish issued on it wrote the residual put + * into the very word the fresh client waits on + * (its tracking semaphore; pub target is + * release_pa since 8c634144) and the + * forward-only apply then skips the real + * release payloads that follow. Re-arm the + * word to 0 and reset the catch-up bookkeeping + * so the re-decoded live releases re-advance + * it from the client's own sequence (1, 2, + * ...). Idempotent when no stale publish + * happened (counter_last == 0). + */ + ch->gr_pub_armed = true; + if (ch->counter_last) { + u64 upa = ch->release_pa ? + ch->release_pa : ch->counter_pa; + u32 zero = 0; + + if (upa) + (void)svc_write_ap(vgpu, upa, + SVC_APERTURE_SYS, + &zero, sizeof(zero)); + ch->counter_last = 0; + } } bitmap_zero(ch->slot_done_epoch, SVC_GPFIFO_MAX_BITS); + memset(ch->slot_pend_tries, 0, + sizeof(ch->slot_pend_tries)); + /* GRFIX FIX1b: a recycled slot in the next put epoch + * gets a fresh absolute-bound record; without this a + * wrapped slot would inherit a stale expired launch + * record. */ + ch->gr_hold_dead_va = 0; + ch->gr_hold_exp_logged = false; + /* r361: same for the opdef absolute bound. */ + ch->opdef_hold_dead_va = 0; + ch->opdef_hold_exp_logged = false; } + put_prev_wit = ch->put_prev; ch->put_prev = put; /* + * SESSION 126 fence fold: sweep the ring continuation beyond + * GP_PUT for semaphore releases (see svc_gr_ahead_sweep). + * GR-class channels only, live-armed epochs only, and only + * with the ring translation proven through the channel + * object (rrc == 0). + */ + if (!ch->engine_type && ch->gr_pub_armed && + gsp_sema_gr_ahead_max && !rrc && ring_pa) { + svc_gr_ahead_sweep(vgpu, ch, put); + /* + * SESSION 127 C5C0 translator: offer a launch the + * sweep captured from the continuation (the Vulkan + * client's inline-QMD push lives beyond GP_PUT, so + * the main walk's per-entry request never sees it). + * svc_gr_request is a no-op without a matching + * unoffered candidate and keeps it on a busy + * verdict, so a bounded per-pass retry here is the + * sweep-captured launch's entire dispatch path. + */ + if (gsp_sema_gr_xlate && + gsp_phys_gr_forward_enabled()) { + if (svc_gr_cand.valid && + svc_gr_cand.h_object == ch->h_object && + svc_gr_ahead_req_log < 8) { + svc_gr_ahead_req_log++; + pr_info("sema-ahead: request h=0x%x push=0x%llx offered=%d\n", + ch->h_object, + svc_gr_cand.push_va, + svc_gr_cand.offered); + } + (void)svc_gr_request(vgpu, ch); + } + /* + * Fallback copy-out retirement: when no forward is + * owed (the launch was already consumed or the + * backend is off), run any pending GRCE copies so + * they cannot sit forever (the completion edge is + * the primary path). + * + * SESSION 153 CONT.25: also run when a forward IS + * owed - the X DDX readback pair's copies have no + * forwarded launch behind them, and a stuck + * unoffered candidate (valid && !offered) blocked + * this fallback forever: 1443 copy pends, 0 runs, + * the server's per-copy completion wait timing out + * on every strip. The completion edge still runs + * them first when it fires; the runner retires + * each entry once, so running early is bounded. + */ + if (gsp_sema_gr_xlate) + svc_gr_copy_pend_run(vgpu); + } + + /* + * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. + * Runs before the entry loop so a wedged session epoch is + * recovered (and logged) instead of spinning: see + * svc_gr_stale_wedge_check. + */ + svc_gr_stale_wedge_check(vgpu, ch, put, get); + + /* * m136 (SA-STALL M2): liveness watchdog for a held CE entry. * When this channel's entry loop is not re-entered (early * continue paths above, or no fresh pushes), a deferred entry @@ -14034,6 +37076,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since = 0; ch->unread_va = 0; ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); svc_replay_note(ch, (u32)ch->last_get, put); ch->last_get = (ch->last_get + 1) % ch->gpfifo_entries; @@ -14042,6 +37085,24 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) get_before = ch->last_get; applied_before = svc.releases_applied; + /* r835: this channel's VAS answers its VAs first (see + * svc_chanvas_lookup). Refreshed per channel turn; a channel + * without a known VAS clears it back to the old chain. */ + svc_cur_vas = ch->h_vaspace; + + /* + * C4 (GAP-1 H1): the GR walk witness - does put advance, and + * does last_get reach it? Budget 512/session; fires on a put + * change or while last_get is in the stalled 80..100 band. + */ + if (gsp_sema_dbg && svc.walk_dbg < 512 && + (put != put_prev_wit || + ((u32)ch->last_get >= 80 && (u32)ch->last_get <= 100))) { + svc.walk_dbg++; + pr_info("sema-diag: walk h=0x%x put=%u get=%u last=%u prev=%u\n", + ch->h_object, put, get, (u32)ch->last_get, + put_prev_wit); + } while (ch->last_get != put && guard++ < ch->gpfifo_entries) { u8 entry[SVC_GPFIFO_ENTRY_SIZE]; @@ -14075,7 +37136,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * (forward-only, so it cannot double-apply) and drop * it before this entry is examined. */ - svc_opdef_drop_stale(vgpu, ch, (u32)idx); + svc_opdef_drop_stale(vgpu, ch, (u32)idx, put); /* * A pin-batch rewind replays recently consumed @@ -14145,10 +37206,17 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ { u64 ident = (u64)e0 | ((u64)e1 << 32); - bool same = ident && + /* + * r505b: the per-slot op bookkeeping arrays are + * SVC_GPFIFO_MAX_BITS (1024) entries; the X/GL + * channel's ring is 8192 slots. Bound every slot + * index here or a large ring writes out of + * bounds. + */ + bool same = ident && idx < SVC_GPFIFO_MAX_BITS && ch->slot_ops_ident[idx] == ident; - entry_ident = ident; + entry_ident = (idx < SVC_GPFIFO_MAX_BITS) ? ident : 0; /* * A complete prior pass suppresses every op. * A partial pass (an operand's targeted scan @@ -14156,6 +37224,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * leaves the per-op done bitmap to decide: * work that already ran must not run again, * the pending op must get its retry. + * + * r354: the content witness decides (the hash + * override sits below, after the push bytes + * are read): entry words alone cannot tell a + * true rewind replay (identical bytes - must + * not run twice) from a REWRITTEN push that + * reuses the same push-buffer VA and length + * with new methods (the driver's repeated + * page-table/fill pushes at the same ring + * slots - suppressing those left the guest's + * GPU page tables stale, the GMMU walk + * missing operand pages, and the first CUDA + * client of every fresh boot failing with + * dropped operands). */ entry_replay = same && ch->slot_ops_complete[idx]; @@ -14171,6 +37253,49 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * SESSION 153-CONTINUED-65: per-boot map of every + * distinct beacon entry (see svc_beef_entry_note) - + * the setup-hold/capture binding reads this list. + */ + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u) + svc_beef_entry_note(ch->h_object, (u32)idx, + va, len); + + /* + * r829: a zero-length / zero-VA entry is a bad read + * (a stale or wrong translation), never work. The old + * path consumed it as an empty push and advanced + * GPGet; observed r830 on the RC watchdog channel: + * two bogus zero entries consumed through a + * mis-resolved ring page pushed the userd's GPGet past + * the driver's own GPPut, every later userd probe then + * read get>put and the channel's REAL userd was + * rejected - the channel stopped being serviced and + * the guest declared the GPU locked. Hold the entry + * instead (bounded; the per-channel counter persists + * through the pass snapshot); a corrected translation + * (the ring auto-pin lands within seconds) re-reads + * it. Past the cap the old consume-as-empty behavior + * resumes so an unresolvable ring cannot livelock the + * channel. + */ + if (va == 0 || len == 0) { + if (ch->zero_held < 512 && !svc_hold_bypass()) { + ch->zero_held++; + svc.gpfifo_held++; + if (gsp_sema_dbg && + svc_zero_entry_log < 16) { + svc_zero_entry_log++; + pr_info("sema-chan: zero-entry hold h=0x%x idx=%llu raw=%08x%08x (bad ring translation; not consumed)\n", + ch->h_object, idx, + e1, e0); + } + stalled = true; + continue; + } + } + + /* * Cheap wait for an entry already inside its * expensive-wait window: read + rescue + hold were * already paid for this VA, and a pin-batch rewind @@ -14183,7 +37308,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * exists for the page, the put advances (a new * push at the same VA), or the window expires. */ - if (ch->engine_type && ch->unread_va == va && + if ((ch->engine_type || + svc_gr_setup_hold_active(ch, va, (u32)idx)) && + ch->unread_va == va && ch->unread_idx == idx && time_before(jiffies, ch->unread_until)) { u64 tpa = 0, vpa = 0; @@ -14218,11 +37345,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (pb) { int rescue_rc; - bool unreadable = - svc_read_va_push(vgpu, ch, va, - pb, len) != 0; + int rd_rc; + bool unreadable; + bool zero_nop = false; + bool displaced = false; + bool have_ipin = false; + + svc_read_last_src = 0xff; + svc_read_last_pa = 0; + rd_rc = svc_read_va_push(vgpu, ch, va, + pb, len); + unreadable = rd_rc != 0; + if (gsp_sema_dbg && svc_rdw_log < 6000 && + !ch->engine_type) { + svc_rdw_log++; + pr_info("sema-rdw: h=0x%x va=0x%llx len=%u rc=%d src=%u\n", + ch->h_object, va, len, + rd_rc, + (unsigned)svc_read_last_src); + } bool injected = false; bool allow_ops = (ch->engine_type != 0); + bool kick_badpg = false; /* * A resolved buffer that does not start @@ -14245,9 +37389,126 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) u64 ipa; int iap; - injected = (svc_ipin_lookup(va, &ipa, - &iap) == 0); - if (!svc_method_stream_plausible(pb, len)) { + have_ipin = + (svc_ipin_lookup(va, &ipa, + &iap) == 0); + /* + * r715: the entry's treatment + * follows the source that + * ACTUALLY answered the read, + * not a fresh page-granularity + * ipin lookup. The read falls + * through to the trusted walk + * (src 4) whenever an ipin's + * page fails its first-chunk + * guard, and taking the + * injected branch then skipped + * the only source that could + * answer: the entry was + * consumed without ever + * consulting the registered + * VAS walk (session 124). + */ + injected = have_ipin && + (svc_read_last_src == 0); + /* + * r496e: an all-zero push on a + * content-scanned-ring channel is the + * driver's NOP sanity push (one zero + * dword - NVA06F_DMA_NOP), not a bad + * translation: accept it and let the + * decode find no ops. + * r820/r826: the same push arrives on + * KERNEL channels (nvidia-push's + * __nvPushTestPushBuffer writes the + * NOP into the kernel pushbuffer); its + * promote entry is a booking, so the + * trusted walk answers - a SMALL + * all-zero GR push resolved by the + * trusted chain (src=4) is the NOP. + * Real launches are large (a QMD push + * is hundreds of bytes), so the size + * bound keeps an unread legitimate + * push from being consumed as empty. + */ + /* + * r934 (SESSION 153-CONT-3): + * bound the gr_ram_ring + * zero-NOP acceptance by + * SIZE. The NOP sanity push + * is one zero dword + * (NVA06F_DMA_NOP; observed + * len=4 on h=0x1000013); + * without the bound a + * 64-byte ALL-ZERO read + * from a stale validated pin + * (the GL client's kick page + * after the guest reused it, + * r933: src=2 + * pa=0x1285a3000 zeros while + * the real kick 0001fff0 + * 20010000 sat at + * pa=0x12c94b000) was + * consumed as empty and the + * entry never fell through + * to the VAS walk that + * answers correctly. With + * the bound the implausible + * push takes the r707v + * viawalk retry (and, if + * that fails, the stale pin + * is dropped by the + * rescue path). + */ + zero_nop = (ch->gr_ram_ring || + (!ch->engine_type && + svc_read_last_src == 4)) && + len >= 4 && len <= 16 && + !memchr_inv(pb, 0, len); + + /* + * r977/r979: a read of the + * kick family's per-frame + * record (SVC_GL_KICK_PUSH_VA; + * r978 showed the channel also + * carries bulk pushes that + * must NOT be held to this + * content identity - a 256 KB + * push at 0x12048b824) whose + * bytes lack the canonical + * submit-record prefix is a + * wrong page no matter how + * plausible it parses (r976: + * src=3 window garbage + * 9864be60 ffffffff consumed + * as a valid push and the kick + * lane ran zero execs). Force + * the implausible path so the + * VAS walk / content anchor / + * pin-drop chain below + * re-anchors the page. + */ + kick_badpg = !zero_nop && + svc_kick_family(ch) && + va == SVC_GL_KICK_PUSH_VA && + len <= 128 && + !svc_kick_canonical(pb, len); + if (kick_badpg) { + svc.kick_badpg++; + if (svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-kickpg: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx first8=%08x %08x (non-canonical kick page; rescue chain)\n", + ch->h_object, va, len, + (unsigned)svc_read_last_src, + svc_read_last_pa, + len >= 4 ? *(const u32 *)pb : 0, + len >= 8 ? *(const u32 *)(pb + 4) : 0); + } + } + + if ((!svc_method_stream_plausible(pb, len) || + kick_badpg) && + !zero_nop) { svc.false_trans++; if (injected) { u8 *fb = kmalloc(len, @@ -14275,7 +37536,115 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(fb); } } else { - unreadable = true; + /* + * r707v: a primary-source + * hit (injected pin, window + * guess, lat cache) whose + * content is implausible + * must not shadow the + * registered VAS walk - a + * stale pin otherwise makes + * every push read "resolve" + * to the wrong page, fail + * plausibility and turn the + * entry unreadable forever. + * Retry once with the pins + * bypassed so the trusted + * walk (svc_walk_vaspaces) + * answers; accept only a + * plausible stream. + */ + u8 *vb = kmalloc(len, + GFP_KERNEL); + + if (vb) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + bool ok = false; + + /* + * r707w: consult ONLY the + * registered VAS walk (the + * exact path walk_probe + * proves) - not + * svc_lookup_trusted, whose + * pin/lat hits would answer + * with the same wrong page + * again and keep the entry + * unreadable. + */ + int wr = svc_walk_vaspaces(vgpu, + va & ~0xfffULL, + &wpa, &wap); + int rr = wr ? wr : svc_read_ap(vgpu, + wpa + (va & 0xfff), + wap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : + SVC_APERTURE_SYS, + vb, len); + bool pl = !rr ? svc_method_stream_plausible(vb, len) + : false; + + /* + * r977: for a kick-family + * entry a plausible walk + * answer still has to carry + * the canonical prefix - a + * wrong-VAS garbage page + * parses plausibly (r976) and + * would re-shadow the real + * page here. + */ + if (pl && kick_badpg && + !svc_kick_canonical(vb, len)) + pl = false; + + if (gsp_sema_dbg && + svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx viawalk rc=%d pa=0x%llx plaus=%d\n", + ch->h_object, va, + wr ? wr : rr, + wpa + (va & 0xfff), + pl ? 1 : 0); + } + if (!wr && !rr && pl) { + memcpy(pb, vb, len); + svc.pushpg_fallback++; + ok = true; + /* + * r977: a kick-family + * entry whose page the + * walk re-anchored is + * pinned so later passes + * read the canonical page + * directly (src=2) instead + * of re-paying the walk + * while the content anchor + * catches up (r975 needed + * ~78 s for the anchor; the + * walk answered in seconds). + */ + if (kick_badpg && + wap == SVC_APERTURE_SYS && + svc_kick_canonical(vb, len)) + svc_pin_store(va & ~(u64)(SEMA_PAGE_SIZE - 1), + wpa, + SVC_APERTURE_SYS); + if (gsp_sema_dbg && + svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx viawalk ok pa=0x%llx\n", + ch->h_object, va, + wpa + (va & 0xfff)); + } + } + kfree(vb); + if (!ok) + unreadable = true; + } else { + unreadable = true; + } } /* * Still implausible (or @@ -14291,18 +37660,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) u8 *db = kmalloc(len, GFP_KERNEL); - if (db && - svc_push_read_displaced(vgpu, - ch->h_object, va, - db, len)) { - memcpy(pb, db, len); - injected = false; - unreadable = false; - } - kfree(db); - } - } - } + if (db && + svc_push_read_displaced(vgpu, + ch->h_object, va, + db, len)) { + memcpy(pb, db, len); + injected = false; + unreadable = false; + displaced = true; + } + kfree(db); + } + } + } /* * The entry's page was unreadable through @@ -14321,11 +37691,87 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) memcpy(pb, db, len); injected = false; unreadable = false; + displaced = true; } kfree(db); } /* + * SESSION 154 (M-GL1.2): a GR RAM-ring + * client push whose every primary + * source failed (r913: a validated pin + * kept answering for a page the guest + * reused and the wrong-VASPACE walk + * answer was implausible). The + * client's submit pushes carry stable + * content anchors, so sweep RAM for + * the page, validate it, pin it and + * re-read the entry through the fresh + * pin. When the anchor has not found + * the page yet, drop a stale validated + * pin (the read that failed came from + * src=2) so later reads prefer the + * walk and the anchor over it. + */ + if (unreadable && !ch->engine_type && + ch->gr_ram_ring && va && len) { + if (svc_gr_anchor_scan(vgpu, ch, + va, len) && + !svc_read_va_push(vgpu, ch, + va, pb, + len) && + svc_method_stream_plausible(pb, + len) && + (!(svc_kick_family(ch) && + va == SVC_GL_KICK_PUSH_VA && + len <= 128) || + svc_kick_canonical(pb, + len))) { + ch->rescue_last = 0; + ch->push_hold_until = 0; + svc.gpfifo_rescued++; + unreadable = false; + } else if (svc_read_last_src == 2) { + svc_pin_drop(va); + if (svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx stale validated pin dropped (content failed every source)\n", + ch->h_object, + va); + } + } + } + + /* + * r715 pushdec witness: name the branch + * the entry takes straight after the + * read, before the rescue/hold paths. + * Session 124 had an entry consumed + * between the read (rc=0 src=4) and the + * decode with no row naming why; this + * line makes the branch and the content + * (first 8 bytes) visible on every read. + * Budget: 512 rows per client (reset in + * svc_session_flush). + */ + if (gsp_sema_dbg && svc_pushdec_log < 2048) { + svc_pushdec_log++; + pr_info("sema-pushdec: h=0x%x va=0x%llx len=%u src=%u sub=%u pa=0x%llx plausible=%d injected=%d ipin=%d zero_nop=%d displaced=%d unreadable=%d first8=%08x %08x\n", + ch->h_object, va, len, + (unsigned)svc_read_last_src, + svc_trust_src, + svc_read_last_pa, + svc_method_stream_plausible(pb, len) ? 1 : 0, + injected ? 1 : 0, + have_ipin ? 1 : 0, + zero_nop ? 1 : 0, + displaced ? 1 : 0, + unreadable ? 1 : 0, + len >= 4 ? *(const u32 *)pb : 0, + len >= 8 ? *(const u32 *)(pb + 4) : 0); + } + + /* * Content rescue before any fake-release * fallback (CE channels): the push bytes * are in guest RAM even when the VA->PA @@ -14364,6 +37810,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!unreadable) { ch->push_hold_until = 0; + svc_gr_setup_hold_clear(ch); /* * The entry words repeated, but the * push bytes may not have: a client @@ -14414,19 +37861,149 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) !ch->engine_type && ch->gr_rehold_idx == (u32)idx; + /* + * m160: the capture gate used to + * require a host-injected pin. A + * fresh ring entry whose push read + * succeeded and whose bytes parse + * as a plausible method stream is + * the client's actual work - the + * same evidence the untrusted ops + * gate already accepts - and a GR + * capture built from it is exactly + * what the m156 rehold forces for + * deferred launches. Without this, + * a channel whose pages the pin + * stream had not reached yet (the + * streams2 second-channel case: + * entry consumed, no capture, the + * client read zeros) never gets + * its launch forwarded at all. + * Replays stay suppressed: a + * consumed entry must not run its + * work twice. + */ + bool cap_trust = injected || rehold; + + /* + * r354: the content witness for + * the once-only op gate. Entry + * words alone cannot tell a true + * rewind replay (identical bytes) + * from a rewritten push that + * reuses the same push VA and + * length with new methods (the + * driver's repeated fill / + * page-table pushes at the same + * ring slots). Suppressing the + * rewritten push left the GPU + * page tables stale and the first + * client's operands unresolved. + * slot_ops_hash has been stored + * since r62 for this comparison. + */ + if (entry_replay && + ch->slot_ops_hash[idx] != + svc_push_hash(pb, len)) { + entry_replay = false; + if (svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-sema: rewritten push runs (idx=%llu va=0x%llx len=%u)\n", + (unsigned long long)idx, va, len); + } + } + if (rehold && !injected && + svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-gr: GR rehold capture idx=%llu va=0x%llx\n", + (unsigned long long)idx, va); + } + /* + * SESSION 149 R15 + * (M1.1a): a rewind + * replay whose slot + * identity AND content + * hash both match the + * recorded pass is the + * same push re-read + * (entry words alone + * cannot tell it from a + * rewritten push - the + * hash can, and the + * r354 checks above + * cleared entry_replay + * on every mismatch, so + * a surviving replay + * plus the explicit hash + * compare here is the + * verified case). + * Admit it as trusted so + * the GL section / + * synthesis capture can + * still be made when the + * first pass never + * captured (the r883 + * one-shot setup push + * decoded exactly once, + * as a replay, and the + * old suppression threw + * that single chance + * away). Ops stay + * suppressed: + * entry_replay keeps + * them off at the + * decode call, and the + * capture sites hold + * their own exactly-once + * guards. + */ + if (!cap_trust && + !ch->engine_type && + gsp_phys_gr_forward_enabled() && + !unreadable && + svc_method_stream_plausible(pb, + len)) { + if (!entry_replay || + (entry_ident && + ch->slot_ops_hash[idx] == + svc_push_hash(pb, len))) + cap_trust = true; + } svc_entry_outcome("decode", ch, idx, va, len); svc.op_tr_pending = 0; + svc.op_owed = 0; + /* + * r595: a GR-class channel + * (engine_type 0) carries the + * driver's own relocation / + * H2D copies as well as its + * launches, but ops were only + * allowed for CE channels (or + * an injected push page). The + * copies then never executed + * and their releases were + * applied by the hold path - + * exactly the r568 b-copy + * class. Admit a GR push's + * ops when its content is a + * plausible method stream + * (cap_trust is that same + * content arbiter, set just + * above), never on a replay + * or a rehold. + */ svc_decode_stream(vgpu, pb, len, 0, - (allow_ops || injected) && + (allow_ops || injected || + cap_trust) && !entry_replay && !rehold, ch, idx, va, - injected || rehold, + cap_trust, entry_replay && !rehold, (entry_replay && !rehold) ? NULL : @@ -14452,8 +38029,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) entry_ident; ch->slot_ops_done[idx] = entry_done; + /* + * r540: an owed + * denied op keeps + * the slot + * incomplete too - + * otherwise every + * later pass is a + * replay and the + * op can never run + * (see op_owed). + */ ch->slot_ops_complete[idx] = - svc.op_tr_pending ? + (svc.op_tr_pending || + svc.op_owed) ? 0 : 1; ch->slot_ops_hash[idx] = svc_push_hash(pb, @@ -14478,10 +38067,37 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * data), so the hold and the * deferral move together. */ + /* + * r558: the opdef hold is the one + * pump hold path without the + * absolute per-launch bound. r557 + * STALL-DUMP evidence: the pump + * consumed NOTHING for 20+ s while + * held/opdef counters climbed + * (opdef=3880/16/64, held=155) - + * a held entry whose operand never + * resolves was re-held every pass + * and the whole pump froze on it. + * Consult the same absolute + * per-launch-VA bound the GR hold + * paths use (svc_gr_hold_bound_ + * expired, one-shot r509): once it + * fires the entry flushes and is + * consumed instead of held again. + */ if (svc.op_tr_pending && + !svc_gr_hold_bound_expired(ch, + svc_gr_launch_va(ch)) && + !svc_hold_bypass() && svc_opdef_hold(vgpu, ch, - (u32)idx)) { + (u32)idx, + va)) { svc.gpfifo_nomap++; + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-pend: hold h=0x%x idx=%u va=0x%llx\n", + ch->h_object, (u32)idx, va); + } /* * P6b wave 1.5: * releases recorded @@ -14505,7 +38121,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(pb); break; } - if (svc.op_tr_pending) + if (svc.op_tr_pending) { /* * Deadline hit: the * operand never @@ -14517,6 +38133,34 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc_opdef_flush(vgpu, ch, (u32)idx); + /* + * r505b: mark the slot's + * ops complete. The op was + * permanently skipped, but + * the decode above stored + * complete=0 (op_tr_pending + * was set), so every pin- + * batch rewind replay of + * this slot re-decoded it, + * re-set op_tr_pending and + * re-held the entry in a + * livelock (multikernel on + * r505: opdef held=18359, + * expired=61, the client + * waiting forever while the + * pump looped on slot 1). + * Nothing is owed after the + * flush, so a replay must + * skip the ops. + */ + if ((u32)idx < + SVC_GPFIFO_MAX_BITS) { + ch->slot_ops_complete[idx] = + 1; + ch->slot_ops_done[idx] = + entry_done; + } + } rel_seen = (svc_rel_count != rel_n_before); { bool fwd_guest_queued; @@ -14602,28 +38246,121 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) fwd_guest_queued = svc_fwd_request(vgpu, ch); - if (fwd_guest_queued) + if (fwd_guest_queued) { + svc.fwd_gr_defer++; + } else if (!svc_hold_bypass() && + !svc_gr_hold_bound_expired(ch, + svc_gr_launch_va(ch)) && + (ch->gr_hold_idx != (u32)idx || + time_before(jiffies, + ch->gr_hold_until) || + /* + * r420: the backend's + * bounded WEDGE-QUIET + * hold outlives this + * one-shot window; keep + * re-holding (never + * consume-with-sw-release) + * while it is active, so + * the launch is still + * forwarded and copied out + * after the hold instead of + * being released onto stale + * data. + */ + gsp_phys_gr_quiet_hold_active())) { + u32 hold_ms = gsp_sema_gr_grace_ms ? + gsp_sema_gr_grace_ms : 8000; + + /* + * P2 (GAP-2 A3): the wait + * expired without queueing. + * Do NOT fall through to the + * software-release consume - + * the launch was never + * submitted, so authoring + * its release would be a + * false completion. Hold + * exactly like the + * data-stalled path + * (m150/m156); the one-shot + * window lets the historical + * consume resume after it + * expires. + */ + ch->gr_rehold_idx = (u32)idx; + ch->gr_hold_idx = (u32)idx; + ch->gr_hold_until = jiffies + + msecs_to_jiffies(hold_ms); + /* + * r633: this break leaves + * before the release block - + * park this pass's records + * with the launch so the + * eventual forward's hold + * delivers them. + */ + if (svc_fwd_defer_armed && + svc_fwd_rel_n > 0) + svc_grpark_put(vgpu, + ch->h_object, + svc_gr_cand.push_va, + svc_fwd_rel, + svc_fwd_rel_n); svc.fwd_gr_defer++; + if (svc_busy_defer_log < 64) { + svc_busy_defer_log++; + pr_info("vgsp-work: GR busy-defer timeout h=0x%x idx=%llu -> hold %u ms\n", + ch->h_object, + (unsigned long long)idx, + hold_ms); + } + kfree(pb); + break; + } else if (svc_busy_defer_log < 64) { + /* + * C8 (GAP-2 A3): the bounded + * hold window is over; the + * historical consume with the + * software release resumes. + */ + svc_busy_defer_log++; + pr_info("vgsp-work: GR busy-defer timeout h=0x%x idx=%llu -> consume w/ sw release\n", + ch->h_object, + (unsigned long long)idx); + } } /* * m150: the GR fill held this - * launch because its high - * argument pointers do not - * translate yet. Leave the - * entry unconsumed (do not - * write GP_GET) so the pump - * re-captures and retries it; - * one bounded window per entry, - * then fall through and consume - * exactly as before. The slot's - * identity makes the hold - * one-shot per entry: after - * expiry the same entry is - * never re-held. + * launch because it carries + * data pages that do not + * resolve yet. Leave the entry + * unconsumed (do not write + * GP_GET) so the pump + * re-captures and retries it. + * GRFIX FIX1b (r213): the hold + * is bounded ABSOLUTELY per + * launch push VA + * (svc_gr_hold_bound_expired + * fixes the deadline on the + * first hold and nothing + * re-arms it; svc_gr_request + * enforces it and forwards the + * launch once it passes). The + * extra check here is the + * defence for a stale stalled + * verdict when the request + * path did not run this pass - + * past the bound the entry + * falls through to the + * historical consume. */ if (!fwd_guest_queued && !ch->engine_type && svc_gr_data_stalled && + !svc_hold_bypass() && + !svc_gr_hold_bound_expired(ch, + svc_gr_launch_va(ch)) && (ch->gr_hold_idx != (u32)idx || time_before(jiffies, ch->gr_hold_until))) { @@ -14647,6 +38384,25 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gr_hold_idx = (u32)idx; ch->gr_hold_until = jiffies + msecs_to_jiffies(hold_ms); + /* + * r633: this break leaves + * before the release block - + * park this pass's records + * with the launch so the + * eventual forward's hold + * delivers them (the r632 + * mail hang: the vadd push's + * tracking release was lost + * here and the client spun + * at cudaDeviceSynchronize). + */ + if (svc_fwd_defer_armed && + svc_fwd_rel_n > 0) + svc_grpark_put(vgpu, + ch->h_object, + svc_gr_cand.push_va, + svc_fwd_rel, + svc_fwd_rel_n); svc.fwd_gr_defer++; if (svc_gr_hold_log < 16) { svc_gr_hold_log++; @@ -14662,6 +38418,65 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) break; } /* + * STALLFREE: the absolute per-launch + * hold bound just fired - the entry + * falls through to the historical + * consume, where the tail authors the + * QMD completion release and logs + * GR-STALL-hold. This line pins the + * bound expiry itself (the r220 hold + * storm's fix: one line per launch, + * nothing re-arms it across rewind + * replays). + */ + if (!fwd_guest_queued && + !ch->engine_type && + svc_gr_data_stalled && + ch->gr_hold_bound_hit && + !ch->gr_hold_exp_logged) { + ch->gr_hold_exp_logged = true; + /* r509: the latch is this pass's only + * report of the expiry; clear it so the + * NEXT expiry cycle (gr_hold_exp_logged + * now set) falls through to the + * historical consume. */ + ch->gr_hold_bound_hit = false; + svc.fwd_gr_hold_exp++; + /* + * r351: the bound firing used to + * fall through to the historical + * consume (entry consumed, QMD + * release authored) WITHOUT ever + * forwarding the launch - the + * client's kernel never ran and + * the client hung (victory_reduce + * with gsp_phys_gr_fill_hold=1: + * bound fired every 3 s, zero GR + * LAUNCH lines, 110 s+ hang). + * Instead, force ONE more fresh + * capture (rehold) with the stall + * verdict cleared: the request's + * own bound is expired too, so it + * proceeds and forwards the launch + * best-effort with whatever + * resolved. If that retry cannot + * run, the next pass (flag set) + * falls through to the historical + * consume as before. + */ + pr_warn("vgsp-gr: GR-STALL-hold-bound va=0x%llx waited_ms=%u action=retry-forward (bound expired; one best-effort forward pass)\n", + svc_gr_launch_va(ch), + jiffies_to_msecs(jiffies - + ch->gr_hold_dead_since)); + svc_gr_data_stalled = false; + ch->gr_rehold_idx = (u32)idx; + ch->gr_hold_idx = (u32)idx; + ch->gr_hold_until = jiffies + + msecs_to_jiffies(200); + kfree(pb); + break; + } + /* * P6b wave 1.5: when * this pass recorded * releases and the @@ -14737,12 +38552,202 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * bridge job owns the release. */ if (!ch->engine_type && - !gsp_cuda_bridge_enabled()) + !gsp_cuda_bridge_enabled()) { svc_gr_sw_release(vgpu, ch->h_object, svc_gr_cand.rel_va, svc_gr_cand.rel_payload); + /* STALLFREE: latch so the + * consume tail cannot author + * the same release twice. */ + svc_gr_cand.rel_done = true; + } + } + } + /* + * m164 CLASS-2 no-capture hold: a + * readable GR launch push whose + * decode saw QMD words but built + * no launch candidate for this + * channel (replay-suppressed + * capture, partial/split QMD, or + * a page the pin stream has not + * reached) must not be consumed + * at the tail - consuming drops + * the client's launch and only + * the RAM sweep's release reaches + * it (r161/r167 streams2 second + * vadd: 56 consumes / 2 captures + * on channel 0x5c000023). Hold + * the entry and force a fresh + * capture on the next pass + * (gr_rehold_idx makes the replay + * decode rebuild the candidate); + * one bounded window per + * (slot, push VA), then fall + * through to the historical + * consume so the channel can + * never livelock. + * + * r470: an ALL-ZERO push page is + * the same "not the client's work + * yet" condition - the ring entry + * is written (the previous + * session's or the client's own + * slot reuse) but the push + * CONTENT has not landed, so the + * decode sees no QMD and the + * entry would be consumed as a + * no-op; the end-of-pass counter + * publish then advances the + * guest's completion word while + * the real push was never read, + * and the client drains a + * staging buffer that still holds + * the PREVIOUS copy's bytes + * (streams_events: h_c[2] = c0 + * because the c2 D2H's + * completion was published 4.5 s + * before its copy ran, the ring + * slots having been decoded as + * all-zero pushes). Deliberately + * NARROWER than the CE-shaped + * plausibility verdict: compute + * pushes legitimately fail that + * matcher, so only an entirely + * zero page counts as unwritten. + */ + { + bool all_zero = false; + + if (!ch->engine_type && + va && len) { + u32 zi; + + all_zero = true; + for (zi = 0; zi < len; + zi++) + if (pb[zi]) { + all_zero = + false; + break; + } + } + /* + * r471: publish clamp bookkeeping. + * A zero-content GR entry that is + * held or consumed must never let + * the end-of-pass counter publish + * advance the guest's completion + * word past it (the client would + * satisfy its readback wait before + * the real push/copy ran - the + * staging buffer still holds the + * previous copy's bytes). Record + * the lowest zero slot; a decode + * that reads non-zero content + * clears the clamp. + */ + if (!ch->engine_type) { + if (all_zero) { + if (!ch->gr_bad_put || + (u32)idx + 1 < + ch->gr_bad_put) + ch->gr_bad_put = + (u32)idx + 1; + } else if (va && len) { + ch->gr_bad_put = 0; + } } + if (!ch->engine_type && va && len && + gsp_phys_gr_forward_enabled() && + (svc_gr_qmd_seen > 0 || + (all_zero && !zero_nop)) && + (!entry_replay || + svc_gr_cand_push_owed(va)) && + !(svc_gr_cand.valid && + svc_gr_cand.h_object == + ch->h_object && + svc_gr_cand.push_va == va)) { + if (ch->gr_nocap_idx != + (u32)idx || + ch->gr_nocap_va != va) { + ch->gr_nocap_idx = + (u32)idx; + ch->gr_nocap_va = va; + ch->gr_nocap_replay = + entry_replay ? + 1 : 0; + ch->gr_nocap_since = + jiffies; + /* + * SESSION 153-CONTINUED-54: an + * all-zero setup push gets the + * same 4K-page demand + pending + * hold as the unreadable one. + */ + if (svc_gr_setup_push(va, len, pb)) + svc_gr_setup_hold_arm(ch, + va, + (u32)idx, + len); + } + { + u32 nocap_ms = + gsp_sema_gr_nocap_hold_ms; + + if (svc_gr_setup_push(va, len, pb) && + gsp_sema_gr_setup_hold_ms > + nocap_ms) + nocap_ms = + gsp_sema_gr_setup_hold_ms; + if (nocap_ms && + !ch->gr_ram_ring && + time_before(jiffies, + ch->gr_nocap_since + + msecs_to_jiffies( + nocap_ms)) && + !svc_hold_bypass()) { + /* Force a fresh capture on + * the retry: a replay decode + * would suppress it. */ + ch->gr_rehold_idx = + (u32)idx; + svc.gpfifo_held++; + svc.gr_nocap_held++; + if (svc_gr_nocap_log < 16) { + svc_gr_nocap_log++; + pr_info("sema-chan: GR no-capture hold h=0x%x entry=%u va=0x%llx len=%u replay=%d owed=%d qmd_seen=%u (forcing capture; window %u ms)\n", + ch->h_object, + (u32)idx, va, + len, + ch->gr_nocap_replay, + svc_gr_cand_push_owed(va) ? + 1 : 0, + svc_gr_qmd_seen, + nocap_ms); + } + kfree(pb); + break; + } + /* Window expired: consume + * exactly as the pre-change + * tail does, with a proof + * line. r548: a forward + * still owed at expiry is + * software-completed first so + * the entry is never consumed + * without a completion. */ + int owed_done = + svc_gr_owed_resolve(vgpu, ch, va); + + svc.gr_nocap_consumed++; + pr_info_ratelimited("sema-chan: GR no-capture hold expired h=0x%x entry=%u va=0x%llx len=%u owed_resolved=%d (consuming without a launch)\n", + ch->h_object, + (u32)idx, va, len, + owed_done); + } + } } /* * A CE push that should carry this @@ -14800,8 +38805,21 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch, idx, va, true, true, NULL); + /* + * r577: for a CE channel only a + * pool-shaped release target is + * this channel's tracking slot. A + * rescued stale page can decode a + * release to a foreign VA (r575: + * 0x12006c004 pay=60); counting + * that as the tracking release + * advanced the expectation without + * delivering to the pool. + */ rel_seen = - (svc_rel_count != rel_n2); + (svc_rel_count != rel_n2) && + (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)); /* * The rescue re-synced the * expectation down to the RAM @@ -14828,16 +38846,66 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) true, NULL); rel_seen = - (svc_rel_count != rel_n2); + (svc_rel_count != rel_n2) && + (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)); } + /* + * r581: the rescue decoded a + * release but the r577 gate + * refused it as a tracking slot + * (rel_seen false while the + * counter moved) - the content + * is provably stale/foreign, no + * work may run from it, and + * waiting the 8 s translation + * grace per entry crawls the + * client's init for minutes + * (r577/r580 boots: 'skipping + * stale entry ... after 8000 ms + * grace' every ~8 s). Backdate + * the deferral so the consume + * path delivers the pool step + * through the exact-slot + * fallback immediately. + */ + if (!rel_seen && + svc_rel_count != rel_n2 && + ch->engine_type && + ch->expected_sem_payload) + ch->defer_since = jiffies - + msecs_to_jiffies(gsp_sema_ce_grace_ms + 100); } } if (rel_seen) { - ch->last_rel_va = - svc_rel_va_seen; + /* + * r577: the exact slot every + * recovery release targets; only + * a pool-shaped VA may claim it + * for a CE channel. The r575 + * wedge adopted a foreign rescued + * release (0x12006c004) here and + * every later recovery went away + * from the client's pool + * (0x121000000 stayed at 25 while + * the client waited for 34; the + * exact-slot poke recovered it). + */ + if (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)) + ch->last_rel_va = + svc_rel_va_seen; + else if (svc_ce_foreign_log < 16) { + svc_ce_foreign_log++; + pr_info("sema-chan: foreign release VA 0x%llx refused as tracking slot h=0x%x eng=%u (recovery stays on the pool slot)\n", + svc_rel_va_seen, + ch->h_object, + ch->engine_type); + } ch->unread_va = 0; ch->unread_until = 0; } + ch->submissions++; svc.gpfifo_entries_done++; if (ch->engine_type && ch->expected_sem_payload) { @@ -14923,6 +38991,41 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) idx, va, len); svc_log_stalled_va(va); /* + * SESSION 153-CONTINUED-54: + * demand the setup push's 4K + * pages and hold it pending. + */ + if (!ch->engine_type && + svc_gr_setup_push(va, len, NULL)) + svc_gr_setup_hold_arm(ch, va, + (u32)idx, + len); + /* + * GR stale-skip clock: the + * rescue-failure skip gate below + * requires defer_since, yet every + * GR arming site is gated on a + * nonzero expected_sem_payload. A + * zero-expectation GR entry (its + * release was already delivered) + * therefore never satisfied the + * gate and was re-read on every + * pass while the channel ring + * stalled behind it (live + * evidence: one 96-byte entry + * re-read 362 times over ~18 s; + * ~8 s of ring stall per frame + * with frames at 0.045 FPS). Arm + * the clock at first sighting; the + * skip window itself stays + * governed by gsp_sema_gr_grace_ms. + */ + if (!ch->engine_type && + !ch->expected_sem_payload && + !ch->defer_since) + ch->defer_since = jiffies - + msecs_to_jiffies(gsp_sema_gr_grace_ms + 1); + /* * Open the cheap-wait window * for this entry (see the * loop-top guard) so rewind @@ -14946,6 +39049,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->unread_ipin_pa = ipa; } /* + * SESSION 153-CONTINUED-54: a held + * GR setup push waits cheaply for + * the whole hold window (the fresh- + * pin check still releases it the + * moment the demanded pin lands). + */ + if (svc_gr_setup_hold_active(ch, va, + (u32)idx) && + time_before(ch->unread_until, + ch->gr_setup_hold_until)) + ch->unread_until = + ch->gr_setup_hold_until; + /* * Bounded work-before-release * hold (CE): the deferred retry * probe shows the page's @@ -15071,6 +39187,8 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) /* Rescue succeeded, retry the read. */ if (!svc_read_va_push(vgpu, ch, va, pb, len)) { + bool track_rel = false; + /* * Never run work decoded from a * rescued page: the rescue matches @@ -15095,14 +39213,43 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) false, false, NULL); - if (svc_rel_count != rel_n_before) + /* + * r577: only a pool-shaped + * release counts as this CE + * channel's tracking release + * (see the norel-path comment). + * A foreign target must not + * claim the exact slot, and the + * pool step must still be + * delivered. + */ + track_rel = + (svc_rel_count != rel_n_before) && + (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)); + if (track_rel) ch->last_rel_va = svc_rel_va_seen; + else if (svc_rel_count != rel_n_before && + svc_ce_foreign_log < 16) { + svc_ce_foreign_log++; + pr_info("sema-chan: foreign release VA 0x%llx refused (rescue) h=0x%x eng=%u (pool step via fallback)\n", + svc_rel_va_seen, + ch->h_object, + ch->engine_type); + } } ch->submissions++; svc.gpfifo_entries_done++; - if (ch->engine_type && ch->expected_sem_payload) - ch->expected_sem_payload++; + if (ch->engine_type && + ch->expected_sem_payload) { + if (track_rel) + ch->expected_sem_payload++; + else if (!gsp_sema_fake_rel) + svc_fake_release(vgpu, ch); + else + ch->expected_sem_payload++; + } } else { svc.gpfifo_nomap++; kfree(pb); @@ -15235,7 +39382,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since && time_after(jiffies, ch->defer_since + - msecs_to_jiffies(gsp_sema_gr_grace_ms))) { + msecs_to_jiffies(gsp_sema_gr_grace_ms)) && + !svc_gr_setup_hold_active(ch, va, + (u32)idx)) { /* * The slot no longer holds this * push (overwritten buffer): the @@ -15280,10 +39429,21 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since = 0; ch->unread_va = 0; ch->unread_until = 0; - if (gsp_sema_fake_rel || - gsp_sema_rel_recover) - svc_replay_note(ch, idx, - put); + svc_gr_setup_hold_clear(ch); + /* + * m177: deliver any + * withheld set even + * on the GR stale-skip + * path. GR channels + * have no rescue and + * their consume-time + * delivery gate is + * below; without this + * the slot strands + * until the sweep. + */ + svc_replay_note(ch, idx, + put); ch->last_get = (idx + 1) % ch->gpfifo_entries; svc.gpfifo_skipped++; @@ -15319,13 +39479,62 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } } - if (ch->engine_type && svc_rel_count != rel_n_iter) + /* + * m177: GR channels (engine_type == 0) have no rescue and + * no CE-shaped delivery branches above; deliver any + * withheld set for this entry whenever the entry is + * consumed so a held release can never strand until the + * 10 s sweep (evtest3's st[2] sync hang). + */ + if (svc_rel_count != rel_n_iter || !ch->engine_type) svc_replay_note(ch, idx, put); + /* + * STALLFREE: bounded completion for a GR capture that was + * consumed without ever being forwarded - a data-stall, + * busy-defer or no-capture hold window expired and the + * entry fell through to the historical consume. No + * hardware copy is coming and no release hold owns the + * launch, so without this the QMD completion word the + * client polls is never authored by the host and the + * launch wait runs to the guest's own timeout. Author + * it now, forward-only, latched once per capture. + * svc_gr_cand.push_va is the entry's push VA, so a + * candidate captured for a later entry of the same + * channel can never be completed by this entry's + * consume. + */ + if (!ch->engine_type && svc_gr_cand.valid && + svc_gr_cand.h_object == ch->h_object && + !svc_gr_cand.offered && !svc_gr_cand.rel_done && + svc_gr_cand.rel_va && + svc_gr_cand.push_va == va) { + svc_gr_cand.rel_done = true; + svc_gr_sw_release(vgpu, ch->h_object, + svc_gr_cand.rel_va, + svc_gr_cand.rel_payload); + svc.fwd_gr_stall_swrel++; + pr_warn("vgsp-gr: GR-STALL-hold va=0x%llx waited_ms=%u action=proceed (unforwarded GR capture consumed; QMD release authored in software)\n", + va, + jiffies_to_msecs(jiffies - + svc_gr_cand.captured_jiffies)); + } ch->last_get = (idx + 1) % ch->gpfifo_entries; ch->rescue_fail = 0; ch->defer_since = 0; + /* m164: the entry was consumed (or fell through the + * no-capture window); clear the hold identity so a + * future push on this slot opens a fresh window. */ + ch->gr_nocap_idx = ~0u; + ch->gr_nocap_since = 0; + ch->gr_nocap_va = 0; ch->unread_va = 0; ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); + /* GRFIX FIX1c (r214): do NOT reset the per-launch + * bound record here - a consume of an unrelated entry + * would re-arm the held launch's bound every pass + * (see svc_fwd_hold_poll). Reset happens on channel + * (re)registration and the session diag reset. */ /* Publish progress the way the GPU would. */ g = ch->last_get; @@ -15395,6 +39604,81 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].userd_pa = ch->userd_pa; svc.chan[k].userd_as = ch->userd_as; svc.chan[k].userd_ok = ch->userd_ok; + /* r496b: persist the RAM-scan cursor and the + * one-shot USERD-scan latch so they survive + * the per-pass snapshot copy. */ + svc.chan[k].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[k].gr_ram_passes = ch->gr_ram_passes; + svc.chan[k].gr_ram_logged = ch->gr_ram_logged; + /* r912: the guest-RAM USERD scan cursor (also + * persisted immediately inside the scan - the + * stalled path skips this write-back). */ + svc.chan[k].gr_ram_userd_cursor = + ch->gr_ram_userd_cursor; + svc.chan[k].gr_ram_userd_passes = + ch->gr_ram_userd_passes; + svc.chan[k].gr_ram_userd_logged = + ch->gr_ram_userd_logged; + /* SESSION 154 (M-GL1.2): liveness watch + the + * free-running-put seed + the push anchor + * cursor all survive the snapshot copy. */ + { + u32 wj; + + svc.chan[k].gr_ram_userd_watch_n = + ch->gr_ram_userd_watch_n; + svc.chan[k].gr_ram_userd_watch_log = + ch->gr_ram_userd_watch_log; + svc.chan[k].gr_ram_userd_kick_sweeps = + ch->gr_ram_userd_kick_sweeps; + for (wj = 0; wj < SVC_GR_USERD_WATCH; + wj++) { + svc.chan[k].gr_ram_userd_watch[wj] = + ch->gr_ram_userd_watch[wj]; + svc.chan[k].gr_ram_userd_watch_put[wj] = + ch->gr_ram_userd_watch_put[wj]; + svc.chan[k].gr_ram_userd_watch_hits[wj] = + ch->gr_ram_userd_watch_hits[wj]; + svc.chan[k].gr_ram_userd_watch_at[wj] = + ch->gr_ram_userd_watch_at[wj]; + } + } + svc.chan[k].gr_ram_put_seeded = + ch->gr_ram_put_seeded; + svc.chan[k].gr_ram_userd_latch_put = + ch->gr_ram_userd_latch_put; + svc.chan[k].gr_ram_userd_latch_at = + ch->gr_ram_userd_latch_at; + svc.chan[k].gr_ram_userd_confirmed = + ch->gr_ram_userd_confirmed; + /* SESSION 153-CONT-5: rejected-latch memory. */ + svc.chan[k].gr_ram_userd_reject_pa = + ch->gr_ram_userd_reject_pa; + svc.chan[k].gr_ram_userd_reject_put = + ch->gr_ram_userd_reject_put; + svc.chan[k].gr_ram_userd_rejects = + ch->gr_ram_userd_rejects; + svc.chan[k].gr_anchor_cursor = ch->gr_anchor_cursor; + svc.chan[k].gr_anchor_va = ch->gr_anchor_va; + svc.chan[k].gr_anchor_sweeps = ch->gr_anchor_sweeps; + svc.chan[k].gr_anchor_logged = ch->gr_anchor_logged; + /* SESSION 149 R15: M1.1b keep-alive state. */ + svc.chan[k].gr_ram_kick = ch->gr_ram_kick; + svc.chan[k].gr_ram_refreshes = + ch->gr_ram_refreshes; + svc.chan[k].gr_ram_refresh_at = + ch->gr_ram_refresh_at; + svc.chan[k].gr_ram_ring = ch->gr_ram_ring; + svc.chan[k].gr_kick_ring_reval_at = + ch->gr_kick_ring_reval_at; + svc.chan[k].userd_scans = ch->userd_scans; + svc.chan[k].fb_userd_scan_at = + ch->fb_userd_scan_at; + svc.chan[k].fb_userd_scan_n = + ch->fb_userd_scan_n; + svc.chan[k].notify2d_va = ch->notify2d_va; + svc.chan[k].zero_held = ch->zero_held; + svc.chan[k].userd_reg = ch->userd_reg; svc.chan[k].logged = ch->logged; svc.chan[k].logged_expected = ch->logged_expected; svc.chan[k].notified = ch->notified; @@ -15421,6 +39705,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].push_hold_until = ch->push_hold_until; svc.chan[k].defer_since = ch->defer_since; svc.chan[k].put_prev = ch->put_prev; + /* r471: the publish clamp watermark is set + * and cleared on the pump thread; persist it + * so the next pass still clamps. */ + svc.chan[k].gr_bad_put = ch->gr_bad_put; /* * Persist the cheap-wait window and the content * watermark: the pump works on a snapshot, so @@ -15441,6 +39729,80 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc.chan[k].gr_hold_until = ch->gr_hold_until; svc.chan[k].gr_hold_idx = ch->gr_hold_idx; + /* + * GRFIX NOTE (r213-r215): ANY new per-channel + * record field MUST be added to this snapshot + * write-back list or it is zeroed on every pass + * (the pump works on the local[] snapshot copied + * at the top of svc_pump_channels). + * + * STALLFREE (r213-r215 follow-up): the GRFIX + * absolute-bound fields were the proven record + * destroyer - they were intentionally left out, + * which is why gsp_phys_gr_fill_hold defaulted + * to 0 (the bound could never fire). They are now + * written back, so the per-launch bound survives + * the snapshot and is effective on the default + * path: a data-stalled launch is held for at most + * gsp_phys_gr_hold_max_ms (default 3000) and then + * is consumed with a software completion, instead + * of being re-held forever by a rewind replay at + * a different slot index (the r220 196 s hold). + */ + svc.chan[k].gr_hold_dead_va = ch->gr_hold_dead_va; + svc.chan[k].gr_hold_dead_since = + ch->gr_hold_dead_since; + svc.chan[k].gr_hold_deadline = + ch->gr_hold_deadline; + svc.chan[k].gr_hold_exp_logged = + ch->gr_hold_exp_logged; + /* r509: the one-shot expiry latch is snapshot + * state too. */ + svc.chan[k].gr_hold_bound_hit = + ch->gr_hold_bound_hit; + /* r361: the opdef absolute-bound record is + * snapshot state too (the GRFIX lesson) - + * without the write-back the deadline would + * be zeroed every pass and the bound could + * never fire. */ + svc.chan[k].opdef_hold_dead_va = + ch->opdef_hold_dead_va; + svc.chan[k].opdef_hold_deadline = + ch->opdef_hold_deadline; + svc.chan[k].opdef_hold_exp_logged = + ch->opdef_hold_exp_logged; + /* m164: the CLASS-2 no-capture window is snapshot + * state too - without the write-back every pass + * would re-arm it and rehold the entry forever + * (the m150 lesson). */ + svc.chan[k].gr_nocap_idx = ch->gr_nocap_idx; + svc.chan[k].gr_nocap_replay = + ch->gr_nocap_replay; + svc.chan[k].gr_nocap_va = ch->gr_nocap_va; + svc.chan[k].gr_nocap_since = ch->gr_nocap_since; + /* + * STALLFREE r225: the stale-wedge watchdog + * state is snapshot state too - without the + * write-back the window would restart every + * pass and never expire. + */ + svc.chan[k].wd_last_get = ch->wd_last_get; + svc.chan[k].wd_put = ch->wd_put; + svc.chan[k].wd_sub = ch->wd_sub; + svc.chan[k].wd_since = ch->wd_since; + /* + * r312: gr_pub_armed / counter_gen live only on + * the snapshot copy if not persisted. Losing the + * armed bit re-arms the residual watchdog every + * pass; its recovery then keeps zeroing the + * completion word and re-publishing - the r311/ + * r312 vector_add wedge loop (66 stale-wedge + * rows with the client polling forever). + */ + svc.chan[k].gr_pub_armed = ch->gr_pub_armed; + svc.chan[k].counter_gen = ch->counter_gen; + svc.chan[k].wd_rewind_off_until = + ch->wd_rewind_off_until; svc.chan[k].pushpg_va = ch->pushpg_va; svc.chan[k].pushpg_n = ch->pushpg_n; svc.chan[k].content_max = ch->content_max; @@ -15466,6 +39828,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) memcpy(svc.chan[k].slot_ops_ident, ch->slot_ops_ident, sizeof(ch->slot_ops_ident)); + memcpy(svc.chan[k].slot_pend_tries, + ch->slot_pend_tries, + sizeof(ch->slot_pend_tries)); memcpy(svc.chan[k].slot_ops_done, ch->slot_ops_done, sizeof(ch->slot_ops_done)); @@ -15529,6 +39894,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ if (stalled) svc_apply_sweep_slice(vgpu); + + /* r590: the pass snapshot is only valid while the loop runs. */ + svc_pass_local = NULL; + svc_pass_local_n = 0; } /* ------------------------------------------------------------------ */ @@ -15723,12 +40092,13 @@ static void svc_handle_chan_dump(struct nv_vgpu_device *vgpu) svc_read_ap(vgpu, ch->notifier_pa + 16 + 8, ch->notifier_as, &info1, 4); } - pr_info("sema-chan-dump: h=0x%x eng=%u gpfifo=0x%llx ent=%u userd=0x%llx as=%u ok=%d get=%u sub=%llu exp=%u rel=0x%llx crel=0x%llx notif=0x%llx as=%u slot0{info=0x%x st=0x%x} slot1{info=0x%x st=0x%x} notified=%llu\n", + pr_info("sema-chan-dump: h=0x%x eng=%u gpfifo=0x%llx ent=%u userd=0x%llx as=%u ok=%d get=%u sub=%llu exp=%u rel=0x%llx crel=0x%llx armed=%d notif=0x%llx as=%u slot0{info=0x%x st=0x%x} slot1{info=0x%x st=0x%x} notified=%llu\n", ch->h_object, ch->engine_type, ch->gpfifo_va, ch->gpfifo_entries, ch->userd_pa, ch->userd_as, ch->userd_ok, ch->last_get, ch->submissions, ch->expected_sem_payload, ch->last_rel_va, ch->counter_pa, + ch->gr_pub_armed, ch->notifier_pa, ch->notifier_as, info0, st0, info1, st1, ch->notified); @@ -15770,6 +40140,23 @@ int gsp_sema_service_dump_channels(void) #define SVC_PIN_REWIND 8 #define SVC_MAP_ADD_MAX 4096 +/* + * SESSION 153-CONTINUED-84: optional length guard for injected mappings. + * A stale producer on the host (observed 2026-11-01: a periodic writer + * enqueueing (0x4001004,0x11f168000,0x10000) every ~2 s from an unknown + * leftover process) re-applies a whole-64K contiguous mapping over the + * per-page SA-carrier pins and can win the read of the client's push + * region, leaving every page past the first reading zeros (r1240/r1260: + * the replayed prelude captured only the first 15 SA slots while r1236/ + * r1245/r1250 captured all 55). When set, requests longer than this many + * bytes are refused with a bounded log, so only per-page (4K) pins land. + * 0 = accept any length (the historical behaviour). + */ +static u32 gsp_sema_map_add_max; +module_param_named(sema_map_add_max, gsp_sema_map_add_max, uint, 0644); +MODULE_PARM_DESC(sema_map_add_max, + "Refuse injected mapping requests longer than this many bytes (0 = off; set 4096 to defeat stale whole-window injections)"); + struct svc_map_add_req { u64 va; u64 pa; @@ -15817,17 +40204,32 @@ static bool svc_va_hits_chan_ring(u64 va) return hit; } +/* + * r561: the drain is BOUNDED per call. r559 soak evidence (victory_saxpy, + * 167 s at 100% CPU, pump alive but consuming nothing): the guest pin + * stream flooded the injection queue (svc_handle_map_add: 1867-6101 + * callbacks suppressed per burst) and the old unbounded for(;;) drain + * spent every service pass processing injections - the ring walk never + * ran, no GPFIFO entry was consumed, and the client spun. The queue is + * drained again on every pass (and from svc_gr_request / svc_run_ops), so + * a bounded batch loses nothing; it just guarantees the ring walk gets + * its cycles under an injection flood. + */ +#define SVC_MAP_ADD_BATCH 64u static void svc_handle_map_add(void) { unsigned long flags; bool rewound = false; u32 i; + u32 batch = 0; for (;;) { struct svc_map_add_req req; u32 ap; u64 off; + if (batch++ >= SVC_MAP_ADD_BATCH) + break; spin_lock_irqsave(&svc_map_add_lock, flags); if (svc_map_add_tail == svc_map_add_head) { spin_unlock_irqrestore(&svc_map_add_lock, flags); @@ -15838,6 +40240,24 @@ static void svc_handle_map_add(void) svc_map_add_tail++; spin_unlock_irqrestore(&svc_map_add_lock, flags); + /* + * SESSION 153-CONTINUED-84: refuse oversized requests when + * the guard is set (see the sema_map_add_max comment) - the + * page pins from pin_loop2 (len 0x1000) are unaffected. + */ + if (READ_ONCE(gsp_sema_map_add_max) && + req.len > (u64)READ_ONCE(gsp_sema_map_add_max)) { + static u32 map_rej_log; + + if (map_rej_log < 8) { + map_rej_log++; + pr_info("sema-svc: injected mapping va=0x%llx pa=0x%llx len=0x%llx REFUSED (len > sema_map_add_max=%u)\n", + req.va, req.pa, req.len, + READ_ONCE(gsp_sema_map_add_max)); + } + continue; + } + ap = req.ap ? req.ap : SVC_APERTURE_SYS; for (off = 0; off < (req.len ? req.len : 1); off += SEMA_PAGE_SIZE) { @@ -15861,13 +40281,46 @@ static void svc_handle_map_add(void) */ if (svc_va_hits_chan_ring(va)) { svc_ring_pin_skips++; - if (svc_ring_pin_skips <= 8) + if (svc_ring_pin_skips <= 64) pr_info("sema-svc: injected pin va=0x%llx DROPPED (live channel ring range; ring reads stay on the channel's own translation)\n", va); continue; } svc_ipin_store(va, req.pa + off, ap); svc_lat_store(va, req.pa + off, ap); + /* + * r372 (stale-ledger corruption class): a fresh pin is the + * guest's newest statement of where this VA's content + * lives. The landing table (ledger) otherwise only moves + * on host writes, so a VA re-pinned to a NEW page (the + * guest freed the old buffer and reallocated it - CUDA + * reuses VA ranges across processes) kept resolving to + * the OLD, freed page: every later write through that + * ledger entry landed on memory the guest kernel had + * since reused (proven class: 8 zero bytes into an + * nv_free_system_pages page array -> guest oops at + * client teardown). Repair the entry only when it + * disagrees with the pin (a no-op otherwise, so the + * table is not flooded with pin pages). + */ + { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(va, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + ((req.pa + off) & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)ap)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-REPAIR va=0x%llx old=0x%llx ap=%u -> new=0x%llx ap=%u\n", + va, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, + req.pa + off, ap); + svc_ledger_store(va, req.pa + off, ap); + } + } } /* * m149: bounded per-session evidence that the CE push-buffer @@ -15910,6 +40363,20 @@ static void svc_handle_map_add(void) if (!ch->valid || !ch->gpfifo_entries) continue; /* + * STALLFREE r225: a channel under the stale-replay + * watchdog's rewind cooldown is NOT rewound. Without + * this the rewind re-opened the same consumed window + * every batch while the producer had nothing pending, + * pinning last_get and re-walking stale entries + * forever (r225: slots 93..100 of 0x5c000059 for + * 42+ s). The cooldown expires on its own (bounded + * duty cycle) and is cleared by new work, so a + * translation landing later can still retry. + */ + if (ch->wd_rewind_off_until && + time_before(jiffies, ch->wd_rewind_off_until)) + continue; + /* * Clamp the rewind at 0: wrapping below zero (get=0 -> * entries - N) makes the pump read ring slots past the * producer that hold stale entries, so replay only @@ -15968,6 +40435,262 @@ int gsp_sema_service_add_map(u64 gpu_va, u64 pa, u64 len, u32 ap) } /* + * X/GL frame-pacer mirror (SESSION 148 perf, proven on r861): the X + * server's present path waits for a GPU-progress word - a lone u32 + * (format 0x14 | base nibble at bits 16..19 | consumed-dword index in + * bits 0..17) in its own guest-RAM page - to reach its per-frame + * submitted-byte counter ((word & 0x3ffff) << 2 == counter). The + * emulation does not execute the 2D method streams, so nothing ever + * advances the word: every frame paid the driver's full ~10 s wait + * timeout (xorg.log "(EE) NVIDIA(GPU-0): WAIT (2, 8, 0x8000, 0x8d4, + * 0x<growing>)" every 10 s -> glxgears 0.09 FPS). A gdb poke of the + * word to (0x140000 | counter >> 2) made the wait complete in 3 ms + * ("WAIT (0, ... <equal>, <equal>)" appears ms after the timeout print) + * and silenced all further timeouts for the session. + * + * This kthread mirrors the driver's counter word (guest PA cnt_pa, a + * u32 in the X process's channel object that advances by the frame's + * submitted bytes) into the progress word (guest PA mem_pa) every few + * ms, preserving the word's high bits. Both PAs are discovered per boot + * (content/handle search in guest RAM) and injected with the + * sema_xw_mirror parameter; "0,0" stops the thread. Pure tuning aid for + * the 1.0.0 demo path - no product logic depends on it. + */ +static u64 svc_xw_cnt_pa; +static u64 svc_xw_mem_pa; +static struct task_struct *svc_xw_task; + +static int svc_xw_mirror_loop(void *unused) +{ + u32 orig_top = 0; + u32 ticks = 0; + + while (!kthread_should_stop()) { + struct nv_vgpu_device *d, *vgpu = NULL; + u32 c = 0, cur = 0; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (vgpu && + !svc_read_ap(vgpu, svc_xw_cnt_pa, SVC_APERTURE_SYS, + &c, sizeof(c)) && + !svc_read_ap(vgpu, svc_xw_mem_pa, SVC_APERTURE_SYS, + &cur, sizeof(cur))) { + /* + * SESSION 148 (r867): member = counter (dword index, + * high bits preserved). The plain tracking law is + * the correct one: the driver's re-check cadence + * (~3 ms-scale re-probes) gives the mirror room to + * settle on the new counter between the driver's + * increment and its first probe. + * + * SESSION 148 r873 page-reuse guard: the word page + * is only ours while the driver's progress word + * lives there. When the page is freed/reused (X + * server died, batch BO recycled) its content + * changes and the base bits (bits 16..23 = 0x10 or + * 0x14) vanish. r872 evidence: the mirror kept + * writing 0x26d into a page that had become another + * process's PTE table (guest "Bad page map ... + * pmd:102656067" pointing at exactly the mirrored + * page's PA), corrupting unrelated guest memory. + * NEVER write once the base bits have changed; stop + * the thread so nothing more touches the page. + */ + u32 top = cur & 0xfffc0000u; + + if (!orig_top) { + if (!top) { + /* not a progress word: write nothing */ + msleep(3); + continue; + } + orig_top = top; + } + if (top != orig_top) { + pr_warn("sema-xw: mirror STOP: word page 0x%llx repurposed (top 0x%x -> 0x%x)\n", + (unsigned long long)svc_xw_mem_pa, + orig_top, cur); + break; + } + if (!(++ticks & 63)) { + /* counter-side guard: channel signature */ + u64 sig = 0; + + if (svc_read_ap(vgpu, svc_xw_cnt_pa - 0x50, + SVC_APERTURE_SYS, &sig, + sizeof(sig)) || + sig != 0x0100001300000001ULL) { + pr_warn("sema-xw: mirror STOP: counter object 0x%llx repurposed (sig=0x%llx)\n", + (unsigned long long)svc_xw_cnt_pa, + (unsigned long long)sig); + break; + } + } + { + u32 m = top | ((c >> 2) & 0x3ffffu); + u16 st = 0; + + if (m != cur) + svc_write_ap(vgpu, svc_xw_mem_pa, + SVC_APERTURE_SYS, &m, + sizeof(m)); + /* + * SESSION 148 r873: submit-kick completion. + * The X submit path arms member status + * 0x8000 at value_word+0xE (gdb-proven layout: + * value @member+0, status @member+0xE) and + * polls until it reads 0; nothing else ever + * clears it, so each kick paid the driver's + * full timeout. Clear it here, right after + * the value write, so the poll passes ms + * after the frame is submitted. + */ + if (!svc_read_ap(vgpu, svc_xw_mem_pa + 0xE, + SVC_APERTURE_SYS, &st, + sizeof(st)) && + st == 0x8000) { + u16 z = 0; + + svc_write_ap(vgpu, svc_xw_mem_pa + 0xE, + SVC_APERTURE_SYS, &z, + sizeof(z)); + if (!(++ticks % 256)) + pr_info("sema-xw: mirror alive (clears=%u cnt=0x%x)\n", + ticks, c); + } + } + } + msleep(3); + } + svc_xw_task = NULL; + return 0; +} + +/* + * X/GL submit-kick completion clearer (SESSION 148 r873). The X server's + * submit path (decoded at nvidia_drv dc170) arms a 16-byte member - + * u32 value at +0x0 (0 while pending) and u16 status 0x8000 at +0xE - and + * then polls dc330 until EVERY member status reads 0; nothing in the + * emulation ever writes that completion, so each kick paid the driver's + * full retry/timeout cycle. The member lives in a page whose page offset + * is 0xB0 (membase is page-aligned), so the status word is at page + * offset 0xBE. The finder in guest RAM reports that PA during an armed + * window; this thread clears status 0x8000 -> 0 (and only that exact + * pattern) so the kick poll passes within a few ms. Pure tuning aid - + * no product logic depends on it. + */ +static u64 svc_xw_comp_pa; /* PA of the u16 status word (page off 0xBE) */ +static struct task_struct *svc_xw_comp_task; + +static int svc_xw_comp_loop(void *unused) +{ + u64 n = 0; + + while (!kthread_should_stop()) { + struct nv_vgpu_device *d, *vgpu = NULL; + u32 val = 1; + u16 st = 0; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (vgpu && + !svc_read_ap(vgpu, svc_xw_comp_pa - 0xE, + SVC_APERTURE_SYS, &val, sizeof(val)) && + !svc_read_ap(vgpu, svc_xw_comp_pa, SVC_APERTURE_SYS, + &st, sizeof(st))) { + /* + * Clear ONLY the exact pending pattern (value still + * 0, status 0x8000). Anything else is either a + * completed member (status 0) or a repurposed page - + * leave it alone. + */ + if (st == 0x8000 && val == 0) { + u16 z = 0; + + svc_write_ap(vgpu, svc_xw_comp_pa, + SVC_APERTURE_SYS, &z, sizeof(z)); + if (!(++n % 25)) + pr_info("sema-xw: comp clears=%llu\n", + (unsigned long long)n); + } + } + msleep(2); + } + svc_xw_comp_task = NULL; + return 0; +} + +int gsp_sema_service_xw_comp(u64 comp_pa) +{ + if (!comp_pa) { + if (svc_xw_comp_task) { + kthread_stop(svc_xw_comp_task); + svc_xw_comp_task = NULL; + pr_info("sema-xw: comp stopped\n"); + } + return 0; + } + if (comp_pa & 1) + return -EINVAL; + svc_xw_comp_pa = comp_pa; + if (!svc_xw_comp_task) { + svc_xw_comp_task = kthread_run(svc_xw_comp_loop, NULL, + "vgsp-xwcomp"); + if (IS_ERR(svc_xw_comp_task)) { + int err = PTR_ERR(svc_xw_comp_task); + + svc_xw_comp_task = NULL; + return err; + } + } + pr_info("sema-xw: comp status_pa=0x%llx running\n", + (unsigned long long)comp_pa); + return 0; +} + +int gsp_sema_service_xw_mirror(u64 cnt_pa, u64 mem_pa) +{ + if (!cnt_pa || !mem_pa) { + if (svc_xw_task) { + kthread_stop(svc_xw_task); + svc_xw_task = NULL; + pr_info("sema-xw: mirror stopped\n"); + } + return 0; + } + if ((cnt_pa | mem_pa) & 0x3) + return -EINVAL; + svc_xw_cnt_pa = cnt_pa; + svc_xw_mem_pa = mem_pa; + if (!svc_xw_task) { + svc_xw_task = kthread_run(svc_xw_mirror_loop, NULL, + "vgsp-xwmirror"); + if (IS_ERR(svc_xw_task)) { + int err = PTR_ERR(svc_xw_task); + + svc_xw_task = NULL; + return err; + } + } + pr_info("sema-xw: mirror cnt=0x%llx -> mem=0x%llx running\n", + (unsigned long long)cnt_pa, (unsigned long long)mem_pa); + return 0; +} + +/* * Dump a range of the emulated vidmem (BAR1 shadow) into the kernel log. * Debug aid for the CE data path: verifies whether a decoded fill/copy * actually landed where the CE operand said it should. @@ -16065,7 +40788,7 @@ int gsp_sema_service_counter_add(u32 h_object, u64 pa) if (!svc.task) return -ENODEV; /* The counter is a u32: require a 4-byte aligned guest PA. */ - if (!h_object || !pa || (pa & 0x3) || pa >= SEMA_PA_MAX) + if (!h_object || !pa || (pa & 0x3) || pa >= svc_pa_max()) return -EINVAL; spin_lock_irqsave(&svc.chan_lock, flags); @@ -16110,9 +40833,9 @@ int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa) if (!svc.task) return -ENODEV; /* Both words are u32: require 4-byte alignment and a RAM-shaped PA. */ - if (!fin_pa || (fin_pa & 0x3) || fin_pa >= SEMA_PA_MAX) + if (!fin_pa || (fin_pa & 0x3) || fin_pa >= svc_pa_max()) return -EINVAL; - if (!hsem_pa || (hsem_pa & 0x3) || hsem_pa >= SEMA_PA_MAX) + if (!hsem_pa || (hsem_pa & 0x3) || hsem_pa >= svc_pa_max()) return -EINVAL; mutex_lock(&nv_vgpu_driver.lock); @@ -16168,19 +40891,43 @@ static void svc_scrub_pass(struct nv_vgpu_device *vgpu) u32 put = 0; u64 delta; - if (!s || !s->userd_ok || !s->pa_ok || !s->entries) + if (!s) + return; + if (!s->userd_ok || !s->pa_ok || !s->entries) { + /* m112b: silent-return diagnostic (bounded per boot). */ + if (s->log_budget < 8) { + s->log_budget++; + pr_info("sema-svc: scrub pass idle vgpu=%px slot=%d userd_ok=%d pa_ok=%d entries=%u gen=%u\n", + (void *)vgpu, (int)(s - svc_scrubs), + s->userd_ok, s->pa_ok, s->entries, + s->session_gen); + } return; + } if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as, &put, sizeof(put))) { /* Retry through the alternate aperture (as registration did). */ if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as == SVC_APERTURE_FB ? SVC_APERTURE_SYS : SVC_APERTURE_FB, - &put, sizeof(put))) + &put, sizeof(put))) { + if (s->log_budget < 12) { + s->log_budget++; + pr_info("sema-svc: scrub pass read-fail userd=0x%llx as=%u\n", + s->userd_pa, s->userd_as); + } return; + } } - if (put >= s->entries) + if (put >= s->entries) { + /* m112b: junk-translation diagnostic (bounded). */ + if (s->log_budget < 12) { + s->log_budget++; + pr_info("sema-svc: scrub pass junk put=%u entries=%u userd=0x%llx\n", + put, s->entries, s->userd_pa); + } return; /* junk translation; retry next pass */ + } if (!s->seeded) { s->seeded = true; @@ -16223,6 +40970,185 @@ static void svc_scrub_pass(struct nv_vgpu_device *vgpu) s->authored = s->submitted; } +/* + * r539: init-stall watchdog. The r530-r538 stall boots all share one shape: + * the pump keeps looping (rebuilds, pending retries) but the entry loop stops + * consuming anything, a client spins at DEVICE (uvm channel_pool_add -> + * uvm_push_end_and_wait) and every counter the guest waits on is frozen. The + * historical evidence for that state was spread over many bounded logs; this + * dump collects the whole stall state in one place when the entry loop makes + * no progress for SVC_STALL_WATCH_MS while withheld work exists. Bounded: + * SVC_STALL_WATCH_DUMPS per boot, one per re-armed window. + */ +#define SVC_STALL_WATCH_MS 20000u +#define SVC_STALL_WATCH_DUMPS 4u + +static unsigned long svc_stall_watch_last; +static u64 svc_stall_watch_done; +static u32 svc_stall_watch_dumps; + +/* + * r563: stall-watchdog self-heal. The r562 STALL-DUMP shape (pump alive, + * no entry consumed for 20 s, held/opdef counters frozen) has been closed + * one hole at a time (r548..r558), but every remaining interaction hole + * has the same end state: some hold condition keeps winning and the walk + * breaks on the same entry forever. When the watchdog fires, arm a small + * budget of forced consumes: the next hold decisions that would otherwise + * park an entry are bypassed, the entry is consumed with its withheld + * releases applied forward-only (and the GR QMD completion authored by the + * existing tail), and the client's wait advances. Bounded by the dump + * cadence (<= 4 per boot) and by the budget (4 hold bypasses per arming), + * so a healthy boot never sees it and a stalled one cannot livelock. + */ +static u32 svc_force_consume; + +static inline bool svc_hold_bypass(void) +{ + if (!svc_force_consume) + return false; + svc_force_consume--; + pr_info_ratelimited("vgsp-sema: STALL-HEAL: hold bypassed (forced consume, budget left %u)\n", + svc_force_consume); + return true; +} + +static void svc_stall_watch_dump(void) +{ + u32 i; + + pr_err("vgsp-sema: STALL-DUMP #%u: no GPFIFO entry consumed for %us (done=%llu held=%llu opdef=%llu/%llu/%llu pending=%u starved=%llu)\n", + svc_stall_watch_dumps, SVC_STALL_WATCH_MS / 1000u, + svc.gpfifo_entries_done, svc.gpfifo_held, + svc.opdef_held, svc.opdef_applied, svc.opdef_expired, + svc_pending_n, svc.gpfifo_starved); + for (i = 0; i < svc_pending_n && i < 16; i++) + pr_err("vgsp-sema: STALL-DUMP pending[%u] h=0x%x va=0x%llx pay=%u op=%u tries=%u\n", + i, svc_pending[i].h, svc_pending[i].va, + svc_pending[i].payload, svc_pending[i].op, + svc_pending[i].tries); + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + + if (!e->active) + continue; + pr_err("vgsp-sema: STALL-DUMP opdef[%u] h=0x%x entry=%u n=%u age_ms=%u first_va=0x%llx pay=%u\n", + i, e->h_object, e->entry_idx, (unsigned)e->n, + jiffies_to_msecs(jiffies - e->since), + e->n ? e->rel[0].va : 0ull, + e->n ? e->rel[0].payload : 0u); + } + for (i = 0; i < svc_pool_boot_n; i++) { + struct svc_pool_boot_pa *b = &svc_pool_boot[i]; + + if (!b->va_page) + continue; + pr_err("vgsp-sema: STALL-DUMP pool[%u] va=0x%llx pa=0x%llx ap=%d pa2=0x%llx ap2=%d age_ms=%u\n", + i, b->va_page, b->pa_page, b->ap, b->pa2_page, + b->ap2, jiffies_to_msecs(jiffies - b->seen)); + } + pr_err("vgsp-sema: STALL-DUMP pool_dual=%llu alt=%llu conflicts=%u residue=%llu\n", + svc.pool_dual_applied, svc.pool_alt_applied, + svc_pool_boot_conflicts, svc.pool_residue); +} + +static void svc_stall_watch_check(void) +{ + if (svc.gpfifo_entries_done != svc_stall_watch_done) { + svc_stall_watch_done = svc.gpfifo_entries_done; + svc_stall_watch_last = jiffies; + return; + } + if (!svc_stall_watch_last) + svc_stall_watch_last = jiffies; + if (!time_after(jiffies, svc_stall_watch_last + + msecs_to_jiffies(SVC_STALL_WATCH_MS))) + return; + svc_stall_watch_last = jiffies; /* re-arm the window */ + if (svc_stall_watch_dumps >= SVC_STALL_WATCH_DUMPS) + return; + /* + * r563c: gate on ACTIVE withheld work, not on the cumulative counters. + * r566 evidence: the dump fired four times with done=8929 and + * opdef=215/7/5 frozen - but the ring was EMPTY (every channel + * last_get == put: the clients were simply waiting and no entry was + * pending), and the forced flush had nothing to deliver. `opdef_held` + * is a lifetime counter (215 holds applied/expired long ago), not + * proof of a current hold. A real stall has either a pending + * translation (svc_pending_n) or an ACTIVE opdef slot with releases + * recorded (e->active && e->n) - only then is there something the + * pump is withholding. + */ + { + bool work = svc_pending_n != 0; + u32 wi; + + for (wi = 0; wi < SVC_OPDEF_SLOTS && !work; wi++) + if (svc_opdef[wi].active && svc_opdef[wi].n) + work = true; + if (!work) + return; + } + svc_stall_watch_dumps++; + svc_stall_watch_dump(); + /* + * r563: self-heal after the dump. The r564 boot showed the force + * budget alone is not enough: the walk broke BEFORE the instrumented + * hold sites (no STALL-HEAL lines while held=727 climbed), so clear + * the state blockers every break site consults, and force-deliver + * the withheld opdef slots: + * - per channel: the cheap-wait unread window (its break at the + * loop top) and the GR hold/nocap indices; + * - the opdef slots: deliver forward-only and mark delivered, so + * the next decode cannot re-hold (the same edge the 3 s bound + * would run); + * - the force-consume budget stays as the belt for the hold sites. + */ + svc_opdef_force_flush(); + { + unsigned long flags; + u32 c; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) { + struct svc_channel *ch = &svc.chan[c]; + + if (!ch->valid) + continue; + ch->unread_va = 0; + ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); + ch->gr_hold_idx = ~0u; + ch->gr_hold_until = 0; + ch->gr_nocap_idx = ~0u; + ch->gr_nocap_va = 0; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + } + pr_err("vgsp-sema: STALL-HEAL: forced opdef flush + walk-state clear (dumps=%u)\n", + svc_stall_watch_dumps); + svc_force_consume = 4; +} + +/* + * ISSUE-0002 client-session boundary aid: force a session flush that + * executes even while channels are live. The channel-close trigger only + * fires when the table empties, which never happens on a guest where the + * driver-global kernel channels (RC watchdog 0x31415900, the tiny notifier + * channel) stay valid; later client sessions then inherit the previous + * session's map/pin/pool discovery state and wedge (false translations -> + * failed rescues -> driver DMA-init timeout). Raised through the + * sema_force_flush module parameter so the boundary can be tested before + * wiring automatic detection. + */ +void gsp_sema_service_sess_flush_force(void) +{ + WRITE_ONCE(svc_sess_flush_forced, 1); + WRITE_ONCE(svc_sess_flush, 1); + if (READ_ONCE(svc.task)) + wake_up_process(svc.task); + pr_info("sema-svc: forced session flush requested\n"); +} + static int svc_thread(void *arg) { struct nv_vgpu_device *vgpus[SVC_MAX_GUESTS]; @@ -16248,6 +41174,24 @@ static int svc_thread(void *arg) } /* + * m177: a guest RM (re)load means a new guest boot: wipe the + * boot-scoped discovery state here, on the thread that owns + * it. Raised by gsp_sema_service_guest_reset(). + */ + if (READ_ONCE(svc_guest_reset_req)) { + svc_guest_reset_req = 0; + svc_guest_reset_wipe(); + } + + /* + * STALL FIX (r189): expire withheld release sets whose entry + * stopped being re-decoded (the deadline flush is only + * reachable from the entry path). Forward-only delivery to a + * live owner; counted LOSS otherwise. Cheap: 8 slots. + */ + svc_opdef_expire_sweep(); + + /* * Service EVERY open mdev each pass, in list order. With one * mdev this is exactly the historical single-vgpu pass; with * two, both guests get a pump turn per iteration (hardware @@ -16281,6 +41225,11 @@ static int svc_thread(void *arg) u64 done_before; vgpu = vgpus[g]; + svc_shared_hold(vgpu); /* CONT.36: GL slot-word keeper */ + /* CONT.42: release any armed client completion sentinel + * on every pass, not only when a completion is being + * processed (see svc_client_slot_sweep). */ + svc_client_slot_sweep(vgpu); svc_handle_probe(vgpu); svc_handle_walk_probe(vgpu); svc_handle_chan_dump(vgpu); @@ -16306,6 +41255,13 @@ static int svc_thread(void *arg) } /* + * r537: bounded fallback for a parked VAS candidate + * whose content-verified ring walk never anchored + * (see svc_vas_candidates_age_out). + */ + svc_vas_candidates_age_out(vgpu); + + /* * Channel-driven execution: notice submissions as they * are made and run the pushed work immediately, so the * guest's UVM progress waits complete in milliseconds @@ -16345,6 +41301,9 @@ static int svc_thread(void *arg) svc_apply_sweep_slice(vgpu); } + /* r539: one stall dump when the entry loop stops progressing. */ + svc_stall_watch_check(); + /* * While the pump is making progress the client is streaming * pushes and waiting on their releases: loop tightly and skip @@ -16354,6 +41313,32 @@ static int svc_thread(void *arg) * progress. (With two guests, any guest's progress keeps the * loop tight; the next iteration pumps both.) */ + /* + * r580: the completion hint must also run while the pump is + * making progress - the fast path below skips it, and a + * completion burst saturates the event burst budget and drops + * wake-ups (r579 boot: events_dropped=724; the sentinel + * "no postable event (events_n=8 dropped=724)" repeated every + * 0.5 s). A client that starts waiting mid-burst needs the + * 0.5 s re-post even under load; the per-event dedup keeps the + * extra volume bounded. + */ + for (g = 0; g < n_vgpus; g++) { + vgpu = vgpus[g]; + if (time_before(jiffies, + (unsigned long)svc.post_hint_until) && + time_after(jiffies, + (unsigned long)svc.last_hint + HZ / 2)) { + int rc; + + svc.last_hint = jiffies; + rc = gsp_rm_post_all_events(vgpu); + if (gsp_sema_dbg) + pr_info_ratelimited("sema-svc: completion hint rc=%d\n", + rc); + } + } + if (pumped) { msleep(1); continue; @@ -16365,30 +41350,22 @@ static int svc_thread(void *arg) /* * Push-buffer base discovery: a chunked walk of guest * RAM finds the pages whose first push names a - * channel, so a channel that appears with work - * already waiting is pinned and pumped without - * waiting for the rolling sweep. */ svc_pb_discover(vgpu); /* - * Completion hint: re-post the events for a few - * seconds after a completion, so a client that starts - * waiting after the edge-triggered wake-up still - * observes it. + * SESSION 153-CONT-34: doorbell writes the guest made + * directly into the mmap'd BAR0 shadow (PCFG window) - + * feed them to the doorbell path so the pump is woken + * without waiting for the next pass boundary. */ - if (time_before(jiffies, - (unsigned long)svc.post_hint_until) && - time_after(jiffies, - (unsigned long)svc.last_hint + HZ / 2)) { - int rc; + svc_bar0_pcfg_scan(vgpu); - svc.last_hint = jiffies; - rc = gsp_rm_post_all_events(vgpu); - if (gsp_sema_dbg) - pr_info_ratelimited("sema-svc: completion hint rc=%d\n", - rc); - } + /* + * r580: the completion hint was hoisted above the + * pumped fast path (it must run under load too); it is + * not repeated here. + */ /* * Fast path for the pushes themselves: every mapped @@ -16498,6 +41475,8 @@ int gsp_sema_service_start(void) } if (gsp_cuda_bridge_init()) pr_warn("sema-svc: cuda bridge debugfs unavailable\n"); + if (svc_setup_demand_debugfs_init()) + pr_warn("sema-svc: setup-demand debugfs unavailable\n"); return 0; } @@ -16533,6 +41512,7 @@ void gsp_sema_service_stop(void) } /* Thread is gone: release the bridge's pending jobs and debugfs files. */ gsp_cuda_bridge_fini(); + svc_setup_demand_debugfs_fini(); /* A fresh session must not inherit shadowed bytes from the last one. */ svc_vas_free_all(); } @@ -16612,7 +41592,7 @@ void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu) /* Held-operand release slots of this guest cannot be delivered. */ for (i = 0; i < SVC_OPDEF_SLOTS; i++) if (svc_opdef[i].active && svc_opdef[i].vgpu == vgpu) - svc_opdef_clear(&svc_opdef[i]); + svc_opdef_clear(&svc_opdef[i], SVC_OPDEF_CLR_STOP); /* Free the per-guest session and scrubber records. */ spin_lock_irqsave(&svc_guest_lock, flags); @@ -16823,6 +41803,71 @@ int gsp_sema_service_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, * after the write still receives the completion event. */ if (!rc) - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return rc; } + +/* + * r616 diag: manual fault poke for the refused-launch hang experiment. A GR + * push that the fill cannot resolve is refused (degenerate data set) and the + * entry is consumed with a software release; the authored release lands in + * the guest page (measured: kcore read 0x6 == pay on r613d) yet the guest + * driver never completes cudaDeviceSynchronize for a launch that never ran. + * This writes the channel's error notifier in the exact shape the relayed- + * fault edge uses (info32=exceptType, info16=engine, status=0xffff, latched + * by notifier_err) and posts the channel event, so a hung client's kernel + * can be poked into observing an engine error and ending its wait. Debug + * only; harmless when no client is hung (the pump's completion clear resumes + * after SVC_NOTIFIER_ERR_HOLD). h_object 0 = every valid channel. + */ +int gsp_sema_channel_fault_poke(struct nv_vgpu_device *vgpu, u32 h_object) +{ + u8 notif[SVC_NOTIFY_SIZE] = {0}; + u16 status = SVC_NOTIFY_ERROR_STATUS; + u32 info32 = 0x1; /* generic engine exception */ + int i, done = 0; + + if (!vgpu) + return -ENODEV; + memcpy(notif + SVC_NOTIFY_INFO32_OFF, &info32, 4); + memcpy(notif + SVC_NOTIFY_STATUS_OFF, &status, 2); + + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *ch = &svc.chan[i]; + unsigned long flags; + u64 pa; + u32 as, eng, h; + u16 info16; + + spin_lock_irqsave(&svc.chan_lock, flags); + if (!ch->valid || ch->vgpu != vgpu || !ch->notifier_pa || + (h_object && ch->h_object != h_object)) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + continue; + } + h = ch->h_object; + pa = ch->notifier_pa; + as = ch->notifier_as; + eng = ch->engine_type; + /* Latch exactly like the relayed-fault edge so the pump's + * completion clear cannot wipe the poke before the guest + * reads it. */ + ch->notifier_err = true; + ch->notifier_err_until = jiffies + SVC_NOTIFIER_ERR_HOLD; + ch->notifier_err_put = ch->put_prev; + spin_unlock_irqrestore(&svc.chan_lock, flags); + + info16 = (u16)eng; + memcpy(notif + SVC_NOTIFY_INFO16_OFF, &info16, 2); + if (svc_write_ap(vgpu, pa, as, notif, sizeof(notif))) { + pr_info("vgsp-fault: POKE ch=0x%x notifier write failed pa=0x%llx as=%u\n", + h, pa, as); + continue; + } + pr_info("vgsp-fault: POKE ch=0x%x engine=%u pa=0x%llx as=%u status=0x%x except=0x%x (manual experiment)\n", + h, eng, pa, as, status, info32); + gsp_rm_post_channel_event(vgpu, h); + done++; + } + return done ? done : -ENOENT; +} diff --git a/kernel/gsp_sema_service.h b/kernel/gsp_sema_service.h index c1dd763..f2a40ee 100644 --- a/kernel/gsp_sema_service.h +++ b/kernel/gsp_sema_service.h @@ -31,6 +31,7 @@ struct sema_channel_desc { u32 h_object; u32 engine_type; u32 cid; /* RM channel id (the doorbell token vector) */ + u32 h_vaspace; /* r835: VAS handle the channel was allocated in */ u64 gpfifo_va; u32 gpfifo_entries; u64 userd_pa; /* NvA06FControl: GPGet @0x88, GPPut @0x8c */ @@ -44,6 +45,14 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc); void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object); /* + * r616 diag: manually write a channel's error notifier (engine-exception + * shape) and post its channel event, to test whether a refused launch's + * hung client can be ended by a fault-conversion edge. h_object 0 = every + * valid channel of the device. Debug only. + */ +int gsp_sema_channel_fault_poke(struct nv_vgpu_device *vgpu, u32 h_object); + +/* * Release every piece of service state that belongs to one mdev before its * guest session disappears: drop the mdev's channels (per-vgpu session end), * clear a release hold that names it (never dereference a removed device), @@ -82,6 +91,54 @@ struct sema_vaspace_desc { void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc); void gsp_sema_service_vaspace_del(u32 h_vaspace); +/* + * r820: a driver-declared kernel-buffer translation from an + * NV2080_CTRL_CMD_GPU_PROMOTE_CTX entry (gpuVirtAddr/gpuPhysAddr/size/ + * physAttr of one context buffer). These buffers - the nvidia-push + * pushbuffer, the progress-tracker segment, the FECS/patch/access-map + * blocks - are kernel-owned, so the userspace pin scanner cannot cover + * them and the page-table walk can answer with a stale/wrong page + * (r817 Xorg GL init: the walk resolved the 64 KB pushbuffer VA to a + * zero page while the promote entry named the real one). The sema + * service consults these records before the learned cache and the walks + * in svc_lookup_trusted; newest statement per VA wins. VA-only bookings + * (zero pa/size) and bNonmapped entries (va=0) are ignored. Records are + * session state, cleared by svc_session_flush. + */ +void gsp_sema_service_promote_add(u64 va, u64 pa, u64 size, u32 phys_attr, + u32 buffer_id); +/* + * SESSION 153-CONTINUED-95: latest PROMOTE_CTX statement per + * (engine, bufferId), kept for the launch-time CTX-BIND receipt. MAIN is + * bufferId 0, PATCH bufferId 2 (NV2080_CTRL_GPU_PROMOTE_CTX_BUFFER_ID_*). + * The note side never fails; the get side returns -ENOENT until the guest + * driver has promoted that buffer. + */ +void gsp_sema_service_ctx_promote_note(u32 engine, u32 buffer_id, u64 pa, + u64 va, u64 size, u32 phys_attr, + u32 b_init); +int gsp_sema_service_ctx_promote_get(u32 engine, u32 buffer_id, u64 *pa, + u64 *va, u64 *size, u32 *phys_attr, + u32 *b_init); +/* + * r496j: park a page-directory candidate (from COPY_SERVER_PDES) until a + * content-validated walk proves it belongs to a live client; only then is + * it registered (a blind registration let stale kernel directories answer + * other clients' VAs in the walker's first-success order). + */ +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem); +/* m177: guest RM reload = new guest boot; wipe boot-scoped discovery state + * (direct-map base, content-scanned tables) on the svc thread. Client + * session flushes keep that state - see svc_session_flush. */ +void gsp_sema_service_guest_reset(void); + +/* + * ISSUE-0002 aid: force a client-session flush that executes even while + * channels are live (see addendum 8 and the sema_force_flush parameter). + */ +void gsp_sema_service_sess_flush_force(void); + int gsp_sema_service_start(void); void gsp_sema_service_stop(void); int gsp_sema_service_probe(u64 gpu_va, u32 count); @@ -89,7 +146,11 @@ int gsp_sema_service_walk_probe(u64 gpu_va); int gsp_sema_service_dump_channels(void); int gsp_sema_service_add_map(u64 gpu_va, u64 pa, u64 len, u32 ap); int gsp_sema_service_dump_vidmem(u64 addr, u64 len); +int gsp_sema_service_xw_mirror(u64 cnt_pa, u64 mem_pa); +int gsp_sema_service_xw_comp(u64 comp_pa); int gsp_sema_service_find_vidmem(u64 pattern, u64 max_hits); +int gsp_sema_service_pdb_discover(u64 ring_va, u64 ring_pa); +int gsp_sema_service_chan_rewind(u32 h_object); int gsp_sema_service_counter_add(u32 h_object, u64 pa); void gsp_sema_service_doorbell(u32 token); @@ -187,6 +248,16 @@ void gsp_sema_service_fwd_gr_stats(u64 *req, u64 *ok, u64 *fallback, u64 *refused, u64 *timeout, u64 *fault); /* + * SESSION 153-CONTINUED-94: the first large beef-channel push of the boot, + * preserved verbatim for the gsp-phys-init-push debugfs dump (the client's + * pushbuffer is transient in guest RAM). Returns the buffer pointer and + * fills len/h/va; lmem_seen counts any SET_SHADER_LOCAL_MEMORY_A..D + * (0x790..0x79c) runs seen in beef pushes so far (0 = the GL client never + * emitted the download-state local-memory block). + */ +const u8 *gsp_sema_init_cap_get(u32 *len, u32 *h, u64 *va, u32 *lmem_seen); + +/* * Discover and pin GR (compute) channel rings after the CE region of the * channel video-memory allocation. Set false to leave GR rings untranslated * (the pump then only runs the CE channels). @@ -247,6 +318,7 @@ extern u64 gsp_sema_copy_dst2; /* CE pending-entry grace window in ms before the release fallback fires. */ extern unsigned gsp_sema_ce_grace_ms; +extern u32 gsp_phys_gr_peek; /* * Bounded work-before-release hold (ms) for an unreadable CE entry whose @@ -254,6 +326,8 @@ extern unsigned gsp_sema_ce_grace_ms; * retry probe). Absolute deadline per entry, never reset; 0 disables. */ extern unsigned gsp_sema_push_hold_ms; +/* SESSION 153-CONTINUED-54: GR setup-push (MME-setup / >=12 KB) hold. */ +extern unsigned gsp_sema_gr_setup_hold_ms; /* * Grace window (ms) an untranslatable GR GPFIFO entry is left pending before @@ -263,4 +337,297 @@ extern unsigned gsp_sema_push_hold_ms; */ extern unsigned gsp_sema_gr_grace_ms; +/* + * GRFIX FIX1b (r213): absolute per-launch bound (ms) for the GR + * data-stalled hold. Keyed by the launch push VA (slot indices are not + * stable across pin-batch rewinds); first hold fixes the deadline, nothing + * re-arms it, and when it passes the launch proceeds (the pre-8b058884 + * behaviour) so the guest can never wait unboundedly. + */ +extern unsigned gsp_phys_gr_hold_max_ms; + +/* + * GRFIX FIX1e (r215): master gate for the GR fill hold (default 0 = + * log-only, never hold). At 0 the fill logs dropped operand pages + * (GRFILL-DROP) and takes the historical forward path immediately; the + * GRFILL-HOLD/GRFILL-EXPIRED machinery is skipped. At 1 the page-count + * hold and its absolute bound apply. + */ +extern unsigned gsp_phys_gr_fill_hold; + +/* + * r576 DIAG: per-boot witness budget for the GR launch-decision dump + * (default 0 = off). Nonzero logs, for the first N launches whose fill saw + * dropped pages / a partial argument set or whose decision point had a + * dstpend/copy-pending verdict: the decision verdicts, the launch's + * resolved data pages (VA -> guest PA), the live deferred-write records and + * the pending CE opdef entries - the evidence needed to explain a red draw + * of the readback lottery. + */ +extern unsigned gsp_phys_gr_dec_log_max; +extern unsigned gsp_sema_walk_trace; + +/* + * r365: execute fill-shaped CE transfers to VIDMEM-window VAs (the guest's + * page-table writes). Off by default; see the definition in the .c. + */ +extern bool gsp_sema_winfill; + +/* + * m164 CLASS-2: hold window (ms) for a GR launch push whose decode saw QMD + * words but built no launch candidate; 0 disables the hold. + */ +extern unsigned gsp_sema_gr_nocap_hold_ms; + +/* + * m165 (D-F2a): skip the post-registration pool hold when a verified + * injected pin exists for the VA (default on); 0 restores the blanket 5 s. + */ +extern bool gsp_sema_pool_hold_fresh; + +/* + * m169 (AGENT-F R1): hold device-destination copies translated only by a + * walk/RAM scan until a pin confirms (default on). + */ +extern bool gsp_sema_copy_dst_pin_guard; +/* r609: after a device-dst guard window expires WITHOUT a pin, skip the + * window for further first-write device-dst pages of the same channel for + * this long (ms). 0 restores the per-page windows (pre-r609). */ +extern unsigned gsp_sema_copy_dst_age_ms; +extern bool gsp_sema_wait_gate; +/* r401: bounded acquire-wait hold (fail-open) in ms; must exceed the + * emulated launch completion latency for copy-after-launch ordering. */ +extern unsigned gsp_sema_wait_hold_ms; +/* r481: verbatim method-stream dump for the first N large GR pushes + * (default 0 off). Diagnostic only; no behaviour change. */ +extern unsigned gsp_sema_push_raw; + +/* + * r384 (guest boot-corruption lottery): fill-destination guards (default + * on). Large virtual fills whose only authority is the walk/scan resolver + * are held through a short bounded window while their VA is demanded + * through the pin channel (fail-open after gsp_sema_fill_guard_ms); zero + * fills targeting a page-table-shaped guest page are refused loudly. + */ +extern bool gsp_sema_fill_guard; +extern unsigned gsp_sema_fill_guard_ms; + +/* + * r231 READ-ORDER guard (default 0: witness only). Defer a device-src copy + * whose source VA has a pending device-dst write hold (READ-ORDER-WIT) so + * the read re-decodes after the write resolves. Opt-in pending an A/B. + */ +extern bool gsp_sema_read_order_guard; + +/* + * COPYWIT fail4: per-boot line budget for the sema-bigcopy resolution + * witnesses (svc_copy's large-copy VA -> pa proof). Raise it for a + * discrimination cycle so the failing clients' H2D/D2H resolutions are + * visible; 0 disables the lines. + */ +extern unsigned gsp_sema_bigcopy_log_max; + +/* + * F1-b: authorship gate for the GR completion release the forwarded launch's + * poll waits on. When on (default), a RAM-sweep release whose VA equals the + * in-flight forward request's gr_rel_va is deferred (skipped, not applied) + * while the hardware copy is still in flight, so the poll can only break on + * an engine-authored word; the deferred release is re-seen by later sweep + * passes and the hold/fallback paths apply it if the hardware copy fails. + * 0 restores the unconditional sweep apply. + */ +extern bool gsp_sema_gr_swrel_gate; + +/* + * relfix (agent-A FIX A): rewrite-time GR relocation resolver. Resolves one + * device VA through the same trusted source chain the GR fill's per-page + * resolution uses (ipin -> self pin -> learned cache -> discovered map -> + * published vaspaces -> discovered directory), then the m180 ledger in the + * fill's two roles (fallback when nothing else answered; m171 landing + * preference above the ledger floor when it disagrees). The result is + * converted to a host page the caller can copy/map. Deliberately read-only + * w.r.t. the resolver tables - no pin or cache stores, no directory + * discovery, no targeted RAM scan: the caller is the GR work item, not the + * service thread, and must not write service state or block on a full-RAM + * scan. The page-table walks may sleep in vfio_dma_rw exactly like the + * fill's walks; callers run in process context (the GR work item, holding + * gsp_phys_rm_lock) with no spinlocks held. A live forwarded GR request + * (svc_fwd_req.vgpu) is required. A successful return (0) means the VA is a + * translated page of that request's client - the caller may act on it; a + * miss returns -ENOENT and the caller must change nothing. + * + * `stage` carries the per-source diagnostic chain of THIS call + * ("ipin:-2,pin:-2,...,ledger:-2," with the resolving source ending in + * ":0,") and `stage_rc` the overall result (0 on success, else the miss + * errno). relfix PART 3: the booter's RELOC-GAP-STAGE line prints them for + * words that stayed unrelocated, so the failing source is on the record. + */ +struct gsp_sema_gr_reloc { + u64 pa; /* resolved guest physical page base */ + u32 ap; /* aperture: 1 = SYS (guest RAM), 2 = FB (vidmem) */ + u64 hpa; /* host physical address of that guest page */ + char stage[96]; /* per-attempt "name:rc," source chain (diagnostic) */ + int stage_rc; /* 0 on success; else the failure errno */ +}; + +int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out); + +/* + * SESSION 153-CONTINUED-59: record one VA (page-aligned internally) in the + * pin-demand set exposed at /sys/kernel/debug/vgsp/sema_setup_demand, so + * pin_gr_stream demands its page in the guest. The GR launch builder calls + * this when a window/attr operand page cannot be resolved at build time + * (the r1104/r1105 MISS -> MMU-fault class); the demand gives the next + * forward a resolvable page instead of a lane-killing raw guest VA. + */ +void gsp_sema_demand_note(u64 va); + +/* + * SESSION 153-CONTINUED-53: MME context-setup prelude accessor for the GR + * launch builder. Returns the captured setup runs (C597 MME family + * 0x114..0x124/0x234/0x550..0x568/0x3400..0x34fc, verbatim, first- + * encounter order), the word count in *n and the capture generation in + * *gen. The builder preplays the words once per generation, ahead of the + * first 3D window replay, so the ladder context's MME has its macro + * program before CALL_MME_MACRO runs. Reading the generation before and + * after the copy detects a concurrent append (the service thread only + * appends; a mismatch just defers emission to the next launch). *n is 0 + * until the client's setup push has been decoded. + */ +const u32 *gsp_sema_mme_prelude_get(u32 *n, u32 *gen); + +/* + * SESSION 153-CONTINUED-57: is the captured MME prelude mature enough to + * launch against - i.e. does it hold the client's macro program (>= 64 + * captured 0x114 code words)? The r1094 premature-emission poison + * (MME ILLEGAL_OPCODE at +5 ms, capture still at 9 scratch words) is + * gated on this: the sema forward path withholds beef launches while it + * reads false, bounded by sema_gr_prelude_gate_ms. + */ +bool gsp_sema_mme_prelude_mature(void); + +/* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude accessor (C597 + * families 0x1100..0x11fc and 0x2000..0x205c, verbatim, first-encounter + * order, deduped). Same contract as gsp_sema_mme_prelude_get: preplay once + * per generation ahead of the first 3D window replay. The runs carry the + * pipeline program binds whose SPH the draw shader validates, so the + * builder also registers the program VAs (gsp_sema_attr_prelude_ptrs) in + * the launch page set to make the address relocation land. + */ +const u32 *gsp_sema_attr_prelude_get(u32 *n, u32 *gen); +const u64 *gsp_sema_attr_prelude_ptrs(u32 *n); + +/* + * SESSION 153-CONTINUED-62: surface (render-target) setup accessor. Holds + * the last captured C597 colour-target bind run (INC 800+j*40, [upper, + * lower, w, h, fmt, blksz], fmt 0xe6, 2..4096 pages) from any trusted + * push. The GR launch builder replays it - aliased/relocated exactly like + * the window - ahead of a 3D window that carries no usable render-target + * bind of its own; without it the replayed draw rasterizes into an + * unbound target and stores nothing (the observed 0/N retire). + */ +const u32 *gsp_sema_surf_prelude_get(u32 *n, u32 *gen); + +/* + * SESSION 153-CONTINUED-63: surface geometry for render-target tracking + * (base VA + page count of the captured colour target). + */ +bool gsp_sema_surf_va_get(u64 *va, u32 *npages); + +/* + * SESSION 153-CONTINUED-63: the fill's completed render-target tracking run + * (base VA + page count), true only when every page of the surface resolved + * into contiguous data recs this pass. The booter prefers these recs over + * the identity surface alias when rewriting the target's operand pair, so + * the engine's stores land in the bounce (witness + copy-out) instead of + * the guest pages directly. + */ +bool gsp_sema_gr_surf_recs(u64 *va, u32 *n); + +/* + * SESSION 153-CONTINUED-63: the captured MME START-ADDRESS RAM table + * (macro slot -> IR instruction address; present-bitmask returned, entries + * copied into `entries`). The prelude buffer is capped at 512 words, so + * the SA table is recorded independently; the GR launch builder completes + * the SA RAM it replays and stubs every called-but-unknown slot with a + * harmless end-noop (the r1116 0x238c wall: SA[0x22] = 0 on a fresh + * context made CALL_MME_MACRO(0x22) jump to IR 0 and the slot-0 macro's + * emission trapped the lane). + */ +u64 gsp_sema_mme_sa_get(u32 *entries, u32 n); + +/* + * SESSION 153-CONTINUED-87: the SPH VERSION of the live program images + * (sema_gl_vs/ps_hex blobs), so the GR launch builder emits a + * SET_SPH_VERSION window that MATCHES the images the engine will fetch + * (the client's own init push says 4/4, but the r950 gate proved the V3 + * images execute; a 4/4 window over a V3 SPH reproduces the + * SPH9|SPH11|SPH18 fault). 0 when no image is loaded. + */ +int gsp_sema_gl_sph_blob_version(void); + +/* + * SESSION 153-CONTINUED-68: furthest IR instruction covered by the CAPTURED + * 0x114 uploads (pc + ceil(code words / 3)). The GR launch builder places + * its end-noop macro stub at this address (aligned past the replayed + * program) so the stub cannot clobber a captured macro body; 0 until the + * first 0x114 upload enters the prelude buffer. + */ +u32 gsp_sema_mme_ir_end_get(void); + +/* + * SESSION 153-CONTINUED-70: IR code words captured so far (the maturity + * gate's measure; printed by the MME-PRE-DUMP for the offline sim). + */ +u32 gsp_sema_mme_code_words_get(void); + +/* + * SESSION 153-CONTINUED-68: is the IR instruction at address `pc` present + * in the replay buffer (i.e. was it uploaded by a captured run that was + * NOT dropped as a duplicate/over-cap)? The launch builder stubs every + * called macro slot whose SA entry sits outside this set, so a truncated + * or generation-mismatched capture becomes a no-op instead of a macro + * fetch into never-replayed IR (Graphics Exception: MISSING_MACRO_DATA). + */ +bool gsp_sema_mme_ir_covered(u32 pc); + +/* + * r378 write guards. Both re-validate a guest-memory write target that was + * resolved earlier (ledger/pin/walk/counter registration, or the GR submit's + * per-page resolution) against the CURRENT resolution before the bytes land, + * and return false when a positive disagreement proves the target stale: + * + * gsp_sema_gr_write_guard(va, hpa, path): GR backend pages. A fresh + * injected pin is checked first (the r372/r376 pin-wins rule - the guest's + * newest statement), then the trusted resolver chain for a pin-less VA. + * The submit-to-copy-out window is seconds wide (the GR poll), and a guest + * buffer freed/re-homed in it would otherwise receive a full page of + * another buffer's content (the boot-corruption class). The landing table + * is deliberately not the authority: our own launches feed it, so it would + * confirm the target by construction. + * + * gsp_sema_write_guard(va, pa, ap, path): service-side SYS/FB targets. A + * fresh injected pin is the guest's newest statement about the VA; a pin + * that disagrees with the recorded target means the page moved (the + * r372/r376 pin-wins rule), so the stale write is skipped. + * + * Both fail OPEN when no current answer exists (the recorded landing is then + * the write authority) and log every skip (bounded, 256 lines) with the path + * and the VA/target/current values. + */ +bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path); +bool gsp_sema_write_guard(u64 va, u64 pa, u32 ap, const char *path); + +/* + * r391 diagnostic (sum=0 class): resolve a GR data VA through the landing + * table (the exact source the D2H reader's ledger-first branch uses) and + * return the resolution plus the first 8 bytes read through it. The GR + * copy-out prints it next to the write target it used, so a launch/readback + * disagreement (the write went to another page/aperture than the reader + * resolves) is one line instead of a forensics session. + */ +bool gsp_sema_gr_va_snap(u64 va, u64 *pa, u32 *ap, u64 *val8); +u8 *gsp_sema_gr_shadow_base(void); + #endif /* _GSP_SEMA_SERVICE_H_ */ diff --git a/kernel/main.c b/kernel/main.c index 257638a..c96abf8 100644 --- a/kernel/main.c +++ b/kernel/main.c @@ -849,25 +849,36 @@ static ssize_t nv_vgpu_vfio_dev_write(struct vfio_device *vdev, const char __use if (copy_from_user(tmp, buf, count)) return -EFAULT; - if ((u64)pos == 0x88704 || (gsp_sema_dbg && bar0_log < 64)) { + if ((u64)pos >= 0x88000 && (u64)pos < 0x89000) { + /* + * SESSION 153-CONTINUED-34: ANY write in the PCFG + * window is a submission doorbell, not only the + * virtual-function doorbell at 0x88704 the old code + * dispatched. Feed every 4-byte write to the doorbell + * path so the pump is woken for the client's channel + * kicks no matter which register the driver poked + * (the service thread also watches the mmap'd shadow + * for writes that never trap). Logging stays bounded. + */ static u32 db_log; u64 v = 0; memcpy(&v, tmp, count); - - if ((u64)pos == 0x88704) { - if (gsp_sema_dbg && db_log < 512) { - pr_info("vgsp: DOORBELL write pos=0x%llx size=%zu val=0x%llx\n", - (u64)pos, count, v); - db_log++; - } - if (count == 4) - gsp_sema_service_doorbell((u32)v); - } else { - pr_info("vgsp: BAR0 write pos=0x%llx size=%zu val=0x%llx\n", + if (gsp_sema_dbg && db_log < 512) { + pr_info("vgsp: PCFG write pos=0x%llx size=%zu val=0x%llx\n", (u64)pos, count, v); - bar0_log++; + db_log++; } + if (count == 4) + gsp_sema_service_doorbell((u32)v); + } else if ((u64)pos == 0x88704 || + (gsp_sema_dbg && bar0_log < 64)) { + u64 v = 0; + + memcpy(&v, tmp, count); + pr_info("vgsp: BAR0 write pos=0x%llx size=%zu val=0x%llx\n", + (u64)pos, count, v); + bar0_log++; } ret = nv_vgpu_bar_emul_write(vgpu, 0, (u64)pos, tmp, count); @@ -2554,6 +2565,25 @@ module_param_cb(sema_probe, &nv_vgpu_sema_probe_ops, NULL, 0200); MODULE_PARM_DESC(sema_probe, "Dump sema-service map + slot values for a GPU VA"); /* + * ISSUE-0002 aid: force a client-session flush past the live-channel skip. + * Write anything (echo 1 > .../parameters/sema_force_flush). + */ +static int nv_vgpu_sema_force_flush_set(const char *val, + const struct kernel_param *kp) +{ + gsp_sema_service_sess_flush_force(); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_force_flush_ops = { + .set = nv_vgpu_sema_force_flush_set, +}; + +module_param_cb(sema_force_flush, &nv_vgpu_sema_force_flush_ops, NULL, 0200); +MODULE_PARM_DESC(sema_force_flush, + "Force a client-session flush even with live channels (ISSUE-0002 aid)"); + +/* * Register the guest PMA scrubber channel buffer words (finish payload PA, * host-semaphore PA) the guest scanner resolved: * echo 0x11a8ff004,0x11a8ff000 > .../parameters/sema_scrubbuf_add @@ -2752,6 +2782,10 @@ MODULE_PARM_DESC(sema_copy_dst2, "Second destination window for sema_copy_dst (d * the release the client synchronizes on; the translation usually lands within * a service pass. */ +module_param_named(gsp_phys_gr_peek, gsp_phys_gr_peek, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_peek, + "Peek the ring slots past GP_PUT at each GR launch decision (diagnostic, default 0)"); + module_param_named(sema_ce_grace_ms, gsp_sema_ce_grace_ms, uint, 0644); MODULE_PARM_DESC(sema_ce_grace_ms, "CE pending-entry grace window in ms (default 25000)"); @@ -2764,6 +2798,17 @@ module_param_named(sema_push_hold_ms, gsp_sema_push_hold_ms, uint, 0644); MODULE_PARM_DESC(sema_push_hold_ms, "CE unreadable-entry work-before-release hold in ms (default 2500, 0 disables)"); /* + * SESSION 153-CONTINUED-54: GR setup-push (client MME-setup upload / any + * >= 12 KB GR push) pending hold (ms). An unreadable/all-zero setup push + * keeps its ring entry pending past the failed-rescue consume until the + * demanded 4K-page guest pin lands; 0 disables (demand rows still emitted). + * Default 0: the demand channel cannot yet resolve the GL pushbuffer VA + * families, so the hold only delays the ring (see gsp_sema_service.c). + */ +module_param_named(sema_gr_setup_hold_ms, gsp_sema_gr_setup_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_gr_setup_hold_ms, "GR setup-push (MME-setup/>=12KB) pending hold in ms (default 0 = off; 20000 = aggressive, needs a resolvable demand family)"); + +/* * Grace window (ms) an untranslatable GR entry stays pending before the pump * consumes it as stale, so a late push-buffer pin can still make it readable. */ @@ -2771,6 +2816,149 @@ module_param_named(sema_gr_grace_ms, gsp_sema_gr_grace_ms, uint, 0644); MODULE_PARM_DESC(sema_gr_grace_ms, "GR pending-entry grace window in ms (default 8000)"); /* + * GRFIX FIX1b (r213): absolute per-launch bound in ms for the GR fill + * data-stalled hold (the FIX1 page-drop hold). The deadline is keyed by the + * launch push VA, fixed on the first hold and never re-armed; when it passes + * the launch proceeds (forwarded best-effort) so a never-resolving page can + * never block the guest. 0 keeps the 3000 ms default. + */ +module_param_named(gsp_phys_gr_hold_max_ms, gsp_phys_gr_hold_max_ms, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_hold_max_ms, "Absolute GR fill hold bound in ms, keyed by launch push VA (default 3000; launch proceeds when it passes)"); + +/* + * GRFIX FIX1e (r215): master gate for the GR fill hold. At 0 (default) the + * fill logs dropped operand pages and takes the historical forward path + * immediately - it can never hold or delay a launch. At 1 the page-count + * hold and the absolute bound apply. + */ +module_param_named(gsp_phys_gr_fill_hold, gsp_phys_gr_fill_hold, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_fill_hold, "GR fill hold on unresolved operand pages (default 0: log-only, never hold)"); + +/* + * r576 DIAG: per-boot witness budget for the GR launch-decision dump (see + * svc_gr_dec_dump in gsp_sema_service.c). Nonzero logs, for the first N + * launches whose fill saw dropped pages / a partial argument set or whose + * decision point carries a dstpend/copy-pending verdict: the decision + * verdicts, the launch's resolved data pages (VA -> guest PA), the live + * deferred-write records and the pending CE opdef entries. 0 (default) = + * off. + */ +module_param_named(gsp_phys_gr_dec_log, gsp_phys_gr_dec_log_max, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_dec_log, "GR launch-decision dump budget per boot (default 0: off)"); + +/* + * r576 DIAG: per-boot witness budget for GSP event-post payloads (see + * gsp_rm_evlog_post in gsp_rm_emulation.c). Nonzero logs every + * EVENT_POST_EVENT record actually sent, with its full payload + * (hClient/hEvent/notifyIndex/data) and send result - the evidence needed + * when a client parks waiting for a completion wake-up that never arrives. + * 0 (default) = off. + */ +module_param_named(gsp_rm_ev_log, gsp_rm_ev_log_max, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_log, "GSP event-post payload witness budget per boot (default 0: off)"); + +/* + * r577 experiment knobs for the guest-RM event drop (runtime-writable so + * one boot can A/B). See gsp_rm_emulation.c. All default 0 = legacy. + */ +module_param_named(gsp_rm_ev_seq, gsp_rm_ev_seq, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_seq, "event records carry the status-queue sequence as RPC sequence (default 0: legacy 0)"); +module_param_named(gsp_rm_ev_nlist, gsp_rm_ev_nlist, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_nlist, "set bNotifyList on event records (default 0: specific-hEvent dispatch)"); +module_param_named(gsp_rm_ev_cl, gsp_rm_ev_cl, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_cl, "override event record hClient (default 0: the allocating client)"); +module_param_named(gsp_rm_ev_func, gsp_rm_ev_func, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_func, "reachability probe: event records carry this RPC function instead of POST_EVENT 0x1003 (default 0)"); + +module_param_named(gsp_sema_walk_trace, gsp_sema_walk_trace, uint, 0644); +MODULE_PARM_DESC(gsp_sema_walk_trace, "Trace GMMU walks for unresolved high-VA operands (default 0: off)"); +module_param_named(gsp_sema_winfill, gsp_sema_winfill, bool, 0644); +MODULE_PARM_DESC(gsp_sema_winfill, "Execute fill-shaped CE transfers to VIDMEM-window VAs (default 0: off)"); + +/* + * F1-b: while a forwarded GR launch is in flight, defer (do not apply) a + * RAM-sweep release whose VA is that launch's QMD completion word, so the + * backend poll can only break on an engine-authored ack. Deferred releases + * are re-seen by later sweep passes; the hold/fallback path applies them when + * the hardware copy fails. 0 restores the unconditional sweep apply. + */ +module_param_named(gsp_sema_gr_swrel_gate, gsp_sema_gr_swrel_gate, bool, 0644); +MODULE_PARM_DESC(gsp_sema_gr_swrel_gate, "Defer a sweep-authored in-flight GR release word (default 1; 0 = unconditional sweep apply)"); + +/* + * m164 CLASS-2: hold window (ms) for a readable GR launch push whose decode + * saw QMD words but built no launch candidate (replay-suppressed capture, + * partial/split QMD, or an unreadable push page). The entry stays unconsumed + * for this window and the next pass forces a fresh capture; on expiry the + * historical consume applies. 0 disables. + */ +module_param_named(sema_gr_nocap_hold_ms, gsp_sema_gr_nocap_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_gr_nocap_hold_ms, "GR no-capture entry hold in ms (default 6000, 0 disables)"); + +/* + * m165 (D-F2a): skip the post-registration pool-release hold when a + * host-verified injected pin for the VA exists (default on). 0 restores the + * legacy blanket 5 s hold. + */ +module_param_named(sema_pool_hold_fresh, gsp_sema_pool_hold_fresh, bool, 0644); +MODULE_PARM_DESC(sema_pool_hold_fresh, "apply pool releases at fresh-pin time instead of registration+5s (default 1)"); + +/* + * m169 (AGENT-F R1): hold device-destination copies translated only by a + * page-directory walk until a pin confirms (default on). A wrong-but-valid + * stale table entry silently writes the wrong vidmem page; the hold lets + * the m163 retry path re-resolve with pins. + */ +module_param_named(sema_copy_dst_pin_guard, gsp_sema_copy_dst_pin_guard, bool, 0644); +module_param_named(sema_wait_gate, gsp_sema_wait_gate, bool, 0644); +MODULE_PARM_DESC(sema_wait_gate, "honor C56F host-semaphore acquires before running a ring entry's work (default 1 since r408; 0 restores pass-through)"); +module_param_named(sema_wait_hold_ms, gsp_sema_wait_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_wait_hold_ms, "bounded acquire-wait hold before fail-open, ms (r401, default 4000; must exceed the emulated launch latency)"); +module_param_named(sema_push_raw, gsp_sema_push_raw, uint, 0644); +MODULE_PARM_DESC(sema_push_raw, "r481 diagnostic: dump the decoded method stream of the first N large GR pushes (default 0 off)"); +MODULE_PARM_DESC(sema_copy_dst_pin_guard, "hold device-dst copies without a pin-resolved translation (default 1)"); +/* + * r609: after a device-dst guard window expires WITHOUT a pin, skip the + * window for further first-write device-dst pages of the same channel for + * this long (ms) - the r606 held-chain serialization fix. 0 restores the + * per-page windows (pre-r609). + */ +module_param_named(sema_copy_dst_age_ms, gsp_sema_copy_dst_age_ms, uint, 0644); +MODULE_PARM_DESC(sema_copy_dst_age_ms, "device-dst guard burst-ageing window after an un-pinned release, ms (r609, default 3000; 0 off)"); + +/* + * r384 (guest boot-corruption lottery): fill-destination guards. A large + * virtual fill whose only authority is the walk/scan resolver is held for + * gsp_sema_fill_guard_ms while its VA is demanded through the pin channel + * (fail-open, mirrors sema_copy_dst_pin_guard); a zero fill targeting a + * page-table-shaped guest page is refused loudly (FILL-PT-REFUSED). + */ +module_param_named(sema_fill_guard, gsp_sema_fill_guard, bool, 0644); +MODULE_PARM_DESC(sema_fill_guard, "guard virtual fill destinations (pin-demand hold + page-table-shape refusal, default 1)"); +module_param_named(sema_fill_guard_ms, gsp_sema_fill_guard_ms, uint, 0644); +MODULE_PARM_DESC(sema_fill_guard_ms, "fill pin-demand hold window in ms (default 800)"); + +/* + * r231 READ-ORDER guard: defer a device-src copy whose source VA has a + * pending device-dst write hold so the read re-decodes after the write + * resolves (fixes the ctaid_probe RAWA upload/readback inversion). + * Default 0: witness only (READ-ORDER-WIT). Opt-in pending an A/B proving + * no green regression. + */ +module_param_named(sema_read_order_guard, gsp_sema_read_order_guard, bool, 0644); +MODULE_PARM_DESC(sema_read_order_guard, "defer device-src reads of a VA with a pending held device-dst write (default 0: witness only)"); + +/* + * COPYWIT fail4 (2026-10-18): per-boot budget for the sema-bigcopy resolution + * witnesses. svc_copy's large-copy proof logs both operands' resolved + * VA -> pa/aperture/provenance, which is the read side of the client data + * path; raise this for a failing-client discrimination cycle so the lines + * survive past the boot's head. 0 disables. + */ +module_param_named(sema_bigcopy_log, gsp_sema_bigcopy_log_max, uint, 0644); +MODULE_PARM_DESC(sema_bigcopy_log, "sema-bigcopy resolution witness budget per boot (default 24; 0 disables)"); + +/* * Debug: inject a verified {GPU VA -> guest PA} translation into the * semaphore service (used when the client's mapping cannot be discovered * from guest structures): @@ -2813,6 +3001,61 @@ module_param_cb(sema_map_add, &nv_vgpu_sema_map_add_ops, NULL, 0200); MODULE_PARM_DESC(sema_map_add, "Inject a GPU VA -> guest PA mapping (hex va,hex pa)"); /* + * Debug: retrigger the client-VAS page-directory discovery with a host + * content-verified {ring VA -> ring PA} pair (the ring page's GPFIFO + * entries are the fingerprint): + * echo 0x<va>,0x<pa> > .../parameters/sema_pdb_discover + */ +static int nv_vgpu_sema_pdb_discover_set(const char *val, + const struct kernel_param *kp) +{ + char buf[64]; + char *comma; + u64 va = 0, pa = 0; + + strscpy(buf, val, sizeof(buf)); + comma = strchr(buf, ','); + if (!comma) + return -EINVAL; + *comma = '\0'; + if (kstrtoull(buf, 0, &va) || kstrtoull(comma + 1, 0, &pa)) + return -EINVAL; + + return gsp_sema_service_pdb_discover(va, pa); +} + +static const struct kernel_param_ops nv_vgpu_sema_pdb_discover_ops = { + .set = nv_vgpu_sema_pdb_discover_set, +}; + +module_param_cb(sema_pdb_discover, &nv_vgpu_sema_pdb_discover_ops, NULL, 0200); +MODULE_PARM_DESC(sema_pdb_discover, "Retrigger the VAS pdb discovery with a verified ring pair (hex va,hex pa)"); + +/* + * Debug: bounded channel rewind. Clears the consumed-slot bitmap and the + * per-entry rescue state for one channel (hex h_object) or every channel + * ("all"), so the pump re-processes its ring from slot 0 - used after the + * VAS pdb anchor lands so entries skipped before it re-decode (the release + * they carry would otherwise be lost for that put epoch). + */ +static int nv_vgpu_sema_chan_rewind_set(const char *val, + const struct kernel_param *kp) +{ + u32 h = 0xffffffffu; + + if (strncmp(val, "all", 3) && kstrtouint(val, 0, &h)) + return -EINVAL; + return gsp_sema_service_chan_rewind(h); +} + +static const struct kernel_param_ops nv_vgpu_sema_chan_rewind_ops = { + .set = nv_vgpu_sema_chan_rewind_set, +}; + +module_param_cb(sema_chan_rewind, &nv_vgpu_sema_chan_rewind_ops, NULL, 0200); +MODULE_PARM_DESC(sema_chan_rewind, "Rewind a channel's ring (hex h_object or 'all')"); + +/* * Debug: register a channel's GPU completion counter (guest PA of the u32 * tracking counter the GPU would write when a push completes): * echo <h_object>,<pa> > .../parameters/sema_counter @@ -2876,6 +3119,62 @@ module_param_cb(sema_vidmem_dump, &nv_vgpu_sema_vidmem_dump_ops, NULL, 0200); MODULE_PARM_DESC(sema_vidmem_dump, "Dump emulated vidmem (addr,len) to the kernel log"); /* + * X/GL frame-pacer mirror (SESSION 148 perf): mirror the X server's + * submitted-byte counter (guest PA) into its GPU-progress word (guest PA) + * so the driver's per-frame 10 s wait completes in milliseconds. + * echo 0x112345678,0x112345000 > .../parameters/sema_xw_mirror + * echo 0,0 > .../parameters/sema_xw_mirror (stop) + */ +static int nv_vgpu_sema_xw_mirror_set(const char *val, + const struct kernel_param *kp) +{ + char buf[64]; + char *comma; + u64 cnt = 0, mem = 0; + + strscpy(buf, val, sizeof(buf)); + comma = strchr(buf, ','); + if (!comma) + return -EINVAL; + *comma = '\0'; + if (kstrtoull(buf, 0, &cnt) || kstrtoull(comma + 1, 0, &mem)) + return -EINVAL; + + return gsp_sema_service_xw_mirror(cnt, mem); +} + +static const struct kernel_param_ops nv_vgpu_sema_xw_mirror_ops = { + .set = nv_vgpu_sema_xw_mirror_set, +}; + +module_param_cb(sema_xw_mirror, &nv_vgpu_sema_xw_mirror_ops, NULL, 0200); +MODULE_PARM_DESC(sema_xw_mirror, "Mirror X frame counter (cnt_pa,mem_pa; 0,0=stop)"); + +/* + * X/GL submit-kick completion clearer (SESSION 148 r873): PA of the u16 + * status word (member page offset 0xBE) that the X submit path arms to + * 0x8000 and polls until 0. Cleared host-side so the kick settles in ms. + * echo 0x1123456be > .../parameters/sema_xw_comp + * echo 0 > .../parameters/sema_xw_comp (stop) + */ +static int nv_vgpu_sema_xw_comp_set(const char *val, + const struct kernel_param *kp) +{ + u64 pa = 0; + + if (kstrtoull(val, 0, &pa)) + return -EINVAL; + return gsp_sema_service_xw_comp(pa); +} + +static const struct kernel_param_ops nv_vgpu_sema_xw_comp_ops = { + .set = nv_vgpu_sema_xw_comp_set, +}; + +module_param_cb(sema_xw_comp, &nv_vgpu_sema_xw_comp_ops, NULL, 0200); +MODULE_PARM_DESC(sema_xw_comp, "Clear X submit member status at PA (0=stop)"); + +/* * Debug: search the whole emulated vidmem for an 8-byte pattern: * echo 0x0000000400000001,16 > .../parameters/sema_vidmem_find */ @@ -3280,6 +3579,47 @@ static int nv_vgpu_dump_vidmem_set(const char *val, const struct kernel_param *k mutex_unlock(&nv_vgpu_driver.lock); return 0; } + if (!strncmp(buf, "map", 3)) { + /* + * Coarse non-zero map: one line per 1 MB block that holds + * any non-zero dword (SESSION 131: locate the live client's + * uploaded content - the image BO, the code heap, the rings - + * inside the 256 MB aperture without 65536 range dumps). + */ + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = vgpu->bars[1].size; + size_t blk, nblk = (size + (1 << 20) - 1) >> 20; + size_t o; + + if (!base || !size) + continue; + for (blk = 0; blk < nblk; blk++) { + size_t start = blk << 20; + size_t end = start + (1 << 20); + u32 nz = 0, first = 0; + + if (end > size) + end = size; + for (o = start; o + 4 <= end; o += 4) { + u32 v; + + memcpy(&v, base + o, 4); + if (v) { + if (!nz) + first = (u32)(o & 0xFFFFF); + nz++; + } + } + if (nz) + pr_info("vgsp: vidmem map blk 0x%zx nz=%u first=0x%x\n", + start, nz, first); + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return 0; + } comma = strchr(buf, ','); if (comma) @@ -3746,5 +4086,143 @@ static const struct kernel_param_ops nv_vgpu_rmpost_ops = { module_param_cb(rmpost, &nv_vgpu_rmpost_ops, NULL, 0200); MODULE_PARM_DESC(rmpost, "Post a GSP-RM event to the guest by event index (debug)"); +/* + * Debug: poke a channel's error notifier with an engine-exception shape and + * post its channel event (the relayed-fault delivery edge, manually): + * echo 0x5c000019 > .../parameters/grfaultpoke + * echo 0 > .../parameters/grfaultpoke (every valid channel) + * Experiment: a refused/degenerate GR launch ends its client's + * cudaDeviceSynchronize spin if a synthetic engine error completes the wait + * (r616; see gsp_sema_channel_fault_poke). + */ +struct nv_vgpu_faultpoke_req { + u32 h_object; + int rc; + struct completion done; +}; + +static int nv_vgpu_faultpoke_thread(void *arg) +{ + struct nv_vgpu_faultpoke_req *req = arg; + struct nv_vgpu_device *vgpu; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + if (atomic_read(&vgpu->vfio_dev_open_count) <= 0) + continue; + req->rc = gsp_sema_channel_fault_poke(vgpu, req->h_object); + pr_info("vgsp: grfaultpoke h=0x%x -> %d\n", + req->h_object, req->rc); + } + mutex_unlock(&nv_vgpu_driver.lock); + + complete(&req->done); + return 0; +} + +static int nv_vgpu_faultpoke_set(const char *val, const struct kernel_param *kp) +{ + static struct nv_vgpu_faultpoke_req req; + struct task_struct *task; + + memset(&req, 0, sizeof(req)); + req.rc = -ENODEV; + if (kstrtou32(val, 0, &req.h_object)) + return -EINVAL; + + init_completion(&req.done); + task = kthread_run(nv_vgpu_faultpoke_thread, &req, "vgpu-faultpoke"); + if (IS_ERR(task)) + return PTR_ERR(task); + wait_for_completion(&req.done); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_faultpoke_ops = { + .set = nv_vgpu_faultpoke_set, +}; + +module_param_cb(grfaultpoke, &nv_vgpu_faultpoke_ops, NULL, 0200); +MODULE_PARM_DESC(grfaultpoke, "Write a channel error notifier + post its event (r616 experiment)"); + +/* + * Debug r623: deliver a synthetic OS_ERROR_LOG (0x1006) engine-fault report + * to the guest's RM queue: + * echo 1 > .../parameters/oserrlog + * echo 0x1,0,0 > .../parameters/oserrlog (exceptType,runlistId,chid) + * Experiment: after a synth-fault, the next client sticks pre-submission; if + * the guest's fault handling completes only when the 0x1006 report arrives, + * this record is the missing recovery half. + */ +struct nv_vgpu_oserrlog_req { + u32 except_type, runlist_id, chid; + int rc; + struct completion done; +}; + +static int nv_vgpu_oserrlog_thread(void *arg) +{ + struct nv_vgpu_oserrlog_req *req = arg; + struct nv_vgpu_device *vgpu; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + if (atomic_read(&vgpu->vfio_dev_open_count) <= 0) + continue; + req->rc = gsp_rm_send_os_error_log(vgpu, req->except_type, + req->runlist_id, req->chid, + "GR"); + pr_info("vgsp: oserrlog except=0x%x rl=%u chid=%u -> %d\n", + req->except_type, req->runlist_id, req->chid, req->rc); + } + mutex_unlock(&nv_vgpu_driver.lock); + + complete(&req->done); + return 0; +} + +static int nv_vgpu_oserrlog_set(const char *val, const struct kernel_param *kp) +{ + static struct nv_vgpu_oserrlog_req req; + struct task_struct *task; + char buf[64]; + char *c1, *c2; + + memset(&req, 0, sizeof(req)); + req.rc = -ENODEV; + strscpy(buf, val, sizeof(buf)); + req.except_type = 1; /* generic engine exception, as the synth-fault */ + { + c1 = strchr(buf, ','); + if (c1) { + *c1++ = '\0'; + c2 = strchr(c1, ','); + if (c2) { + *c2++ = '\0'; + if (kstrtou32(c2, 0, &req.chid)) + return -EINVAL; + } + if (kstrtou32(c1, 0, &req.runlist_id)) + return -EINVAL; + } + if (kstrtou32(buf, 0, &req.except_type)) + return -EINVAL; + } + + init_completion(&req.done); + task = kthread_run(nv_vgpu_oserrlog_thread, &req, "vgpu-oserrlog"); + if (IS_ERR(task)) + return PTR_ERR(task); + wait_for_completion(&req.done); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_oserrlog_ops = { + .set = nv_vgpu_oserrlog_set, +}; + +module_param_cb(oserrlog, &nv_vgpu_oserrlog_ops, NULL, 0200); +MODULE_PARM_DESC(oserrlog, "Send a synthetic OS_ERROR_LOG (0x1006) RM record to the guest (r623 experiment)"); + module_init(nv_vgpu_init); module_exit(nv_vgpu_exit); diff --git a/tools/cuda-ab/battery_resume.sh b/tools/cuda-ab/battery_resume.sh new file mode 100755 index 0000000..33bbe86 --- /dev/null +++ b/tools/cuda-ab/battery_resume.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# battery_resume2.sh - resume a battery after a partial stop, with a +# zombie-tolerant completion check: a client is DONE when its .bat.out holds +# a verdict line (STATUS OK|FAIL|CUDA_ERROR) even if a zombie with the same +# comm lingers (the pgrep-x poll in battery_smileylast falsely hangs on +# those). A client with an alive non-zombie process and no verdict after +# HANG_S is a real hang. +# Usage: battery_resume2.sh <outfile> <client...> +set -u +cd /home/rmcore/Git/integdev_gpu_drv/open-iov +OUT="${1:?usage: battery_resume2.sh <out> <client...>}"; shift +CLIENTS="$*" +HANG_S=600 +: >> "$OUT" +pass=0; fail=0; hang=0 +for c in $CLIENTS; do + echo "=== $c ===" | tee -a "$OUT" + timeout 40 scripts/ssh_guest.sh "test -x /tmp/$c || exit 9; cd /tmp && rm -f /tmp/$c.bat.out && setsid nohup ./$c > /tmp/$c.bat.out 2>&1 < /dev/null & echo LAUNCHED" >/dev/null 2>&1 + t0=$(date +%s); done_ok=0 + while :; do + sleep 5 + block=$(timeout 25 scripts/ssh_guest.sh "cat /tmp/$c.bat.out 2>/dev/null" 2>/dev/null | grep -v "Permanently added") + if printf '%s\n' "$block" | grep -qE "STATUS OK|STATUS FAIL|CUDA_ERROR"; then done_ok=1; break; fi + # alive non-zombie process with no verdict? + alive=$(printf '%s\n' "$block"; timeout 25 scripts/ssh_guest.sh "ps -eo stat=,comm= | awk -v n='$c' '\$2==n && \$1 !~ /^Z/' | wc -l" 2>/dev/null | tail -1) + now=$(date +%s) + if [ $((now - t0)) -ge $HANG_S ]; then + echo "HANG: $c no verdict after ${HANG_S}s" | tee -a "$OUT" + hang=$((hang+1)) + echo "BATTERY RESUME STOPPED at $c (hang; boot preserved, do not kill)" | tee -a "$OUT" + echo "BATTERY RESUME pass=$pass fail=$fail hang=$hang" | tee -a "$OUT" + exit 0 + fi + done + printf '%s\n' "$block" | tail -8 | tee -a "$OUT" + if printf '%s\n' "$block" | grep -q "STATUS OK"; then pass=$((pass+1)); else fail=$((fail+1)); fi + echo "--- done ---" | tee -a "$OUT" +done +echo "BATTERY RESUME pass=$pass fail=$fail hang=$hang" | tee -a "$OUT"
\ No newline at end of file diff --git a/tools/cuda-ab/evhold.cu b/tools/cuda-ab/evhold.cu new file mode 100644 index 0000000..3817ff3 --- /dev/null +++ b/tools/cuda-ab/evhold.cu @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * evhold.cu - deterministic event-holder / event-wait probe for the guest + * event-delivery class (r577). Opens a CUDA context (so the driver + * registers its RM events), runs one small op, records and synchronizes a + * CUDA event (which exercises the client's wait path), then holds the + * context open for N seconds so host-side experiments (rmpost rounds with + * the gsp_rm_ev_* knobs, ev_post kprobe) have a live event table to post + * into. + * + * Usage: evhold [hold_seconds] + * Output lines: + * EVHOLD ctx ok + * EVHOLD event sync ok + * EVHOLD hold done + */ +#include <cstdio> +#include <cstdlib> +#include <unistd.h> +#include <cuda_runtime.h> + +static void check(const char *what, cudaError_t e) +{ + if (e != cudaSuccess) { + printf("EVHOLD FAIL %s: %s\n", what, cudaGetErrorString(e)); + fflush(stdout); + exit(1); + } +} + +int main(int argc, char **argv) +{ + int hold = (argc > 1) ? atoi(argv[1]) : 120; + float *d = NULL; + cudaEvent_t ev; + + check("freectx", cudaFree(0)); + printf("EVHOLD ctx ok\n"); + fflush(stdout); + + check("malloc", cudaMalloc(&d, 4096)); + check("memset", cudaMemset(d, 0, 4096)); + check("evcreate", cudaEventCreate(&ev)); + check("evrecord", cudaEventRecord(ev, 0)); + check("evsync", cudaEventSynchronize(ev)); + printf("EVHOLD event sync ok\n"); + fflush(stdout); + + for (int i = 0; i < hold; i++) + sleep(1); + + cudaEventDestroy(ev); + cudaFree(d); + printf("EVHOLD hold done\n"); + return 0; +} diff --git a/tools/cuda-ab/frame_producer.cu b/tools/cuda-ab/frame_producer.cu new file mode 100644 index 0000000..609114c --- /dev/null +++ b/tools/cuda-ab/frame_producer.cu @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * frame_producer.cu - render an animated frame on the guest GPU and publish + * it into the ivshmem (kvmfr) BAR so the host can see a real GPU-rendered + * frame from the guest. + * + * Path: CUDA kernel -> device buffer -> host buffer -> ivshmem BAR + * (/sys/bus/pci/devices/0000:00:03.0/resource2, the 128 MiB kvmfr + * device QEMU backs with /dev/shm/kvmfr-<uuid>). + * + * Layout in the BAR (offset 0x100000, clear of the kvmfr protocol header): + * struct frame_hdr { magic, version, width, height, stride, format, seq } + * then width*height*4 bytes of RGBA8. + * + * Usage: sudo ./frame_producer [frames] [width] [height] [fps] + * Prints: FRAME seq=<n> ms=<render+copy ms> (one line per frame, first 5 and + * every 30th) and FRAME_DONE frames=<n>. + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <ctime> +#include <fcntl.h> +#include <sys/mman.h> +#include <unistd.h> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + return 1; \ + } \ + } while (0) + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" little-endian-ish */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +__global__ void render_julia(uchar4 *fb, int w, int h, float t) +{ + int x = blockIdx.x * blockDim.x + threadIdx.x; + int y = blockIdx.y * blockDim.y + threadIdx.y; + + if (x >= w || y >= h) + return; + + float fx = (x - w * 0.5f) / (h * 0.45f); + float fy = (y - h * 0.5f) / (h * 0.45f); + float cr = 0.7885f * cosf(t); + float ci = 0.7885f * sinf(t); + float zr = fx, zi = fy; + int it = 0; + const int maxit = 96; + + while (zr * zr + zi * zi < 4.0f && it < maxit) { + float nzr = zr * zr - zi * zi + cr; + float nzi = 2.0f * zr * zi + ci; + + zr = nzr; + zi = nzi; + it++; + } + { + float v = (float)it / (float)maxit; + unsigned char r = (unsigned char)(255.0f * v); + unsigned char g = (unsigned char)(255.0f * (1.0f - v) * v * 4.0f); + unsigned char b = (unsigned char)(255.0f * (1.0f - v)); + unsigned char a = 255; + + fb[y * w + x] = make_uchar4(r, g, b, a); + } +} + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +int main(int argc, char **argv) +{ + int frames = argc > 1 ? atoi(argv[1]) : 60; + int w = argc > 2 ? atoi(argv[2]) : 800; + int h = argc > 3 ? atoi(argv[3]) : 600; + int fps = argc > 4 ? atoi(argv[4]) : 10; + const char *bar_path = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + size_t fb_bytes = (size_t)w * h * 4; + uchar4 *d_fb = NULL, *h_fb = NULL; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + int fd, i; + + setvbuf(stdout, NULL, _IONBF, 0); + + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + printf("OPEN_FAIL %s\n", bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (bar == MAP_FAILED) { + printf("MMAP_FAIL\n"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + CHECK(cudaMalloc(&d_fb, fb_bytes)); + h_fb = (uchar4 *)malloc(fb_bytes); + if (!h_fb) { + printf("HOST_ALLOC_FAIL\n"); + return 1; + } + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + + for (i = 0; i < frames; i++) { + uint64_t t0 = now_ns(); + float phase = (float)i * 0.15f; + dim3 block(16, 16); + dim3 grid((w + 15) / 16, (h + 15) / 16); + + render_julia<<<grid, block>>>(d_fb, w, h, phase); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_fb, d_fb, fb_bytes, + cudaMemcpyDeviceToHost)); + memcpy(bar + FRAME_OFFSET + sizeof(struct frame_hdr), h_fb, + fb_bytes); + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + if (i < 5 || (i % 30) == 29) + printf("FRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d %dx%d\n", frames, w, h); + CHECK(cudaFree(d_fb)); + free(h_fb); + munmap(bar, 128UL << 20); + close(fd); + return 0; +} diff --git a/tools/cuda-ab/inplace_scale.cu b/tools/cuda-ab/inplace_scale.cu new file mode 100644 index 0000000..e83a039 --- /dev/null +++ b/tools/cuda-ab/inplace_scale.cu @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * inplace_scale.cu - minimal in-place scale repro for the streams2 + * "scale wrote 1 page" class. + * + * One H2D, one in-place scale launch (c[i] *= 2 over 128 KB = 32 pages), + * one D2H, per-page diff. Run it repeatedly: if the launch writes only + * part of the buffer the per-page report names how many 4 KB blocks + * changed and the host log carries the launch's retire wait. + * + * Build: nvcc -O2 --cudart=static -o inplace_scale inplace_scale.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +int main(void) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *h, *d; + int i, bad = 0, first = -1, last = -1; + int nb = (n + 1023) / 1024, blk_bad = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + h = (float *)malloc(bytes); + if (!h) + return 1; + for (i = 0; i < n; i++) + h[i] = (float)(i % 1000) * 0.5f; + + CHECK(cudaMalloc(&d, bytes)); + CHECK(cudaMemcpy(d, h, bytes, cudaMemcpyHostToDevice)); + printf("PTR d=%p\n", (void *)d); + + scale<<<(n + 255) / 256, 256>>>(d, 2.0f, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + + CHECK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost)); + for (i = 0; i < n; i++) { + float exp = (float)(i % 1000) * 0.5f * 2.0f; + + if (h[i] != exp) { + bad++; + if (first < 0) + first = i; + last = i; + } + } + for (i = 0; i < nb; i++) { + int j, hit = 0; + + for (j = i * 1024; j < (i + 1) * 1024 && j < n; j++) + if (h[j] != (float)(j % 1000) * 0.5f * 2.0f) { + hit = 1; + break; + } + if (hit) + blk_bad++; + } + printf("SCALE n=%d bad=%d first=%d last=%d blocks=%d/%d\n", + n, bad, first, last, blk_bad, nb); + printf("STATUS %s\n", bad ? "FAIL" : "OK"); + return bad ? 1 : 0; +} diff --git a/tools/cuda-ab/ip_probe.cu b/tools/cuda-ab/ip_probe.cu new file mode 100644 index 0000000..366fbc8 --- /dev/null +++ b/tools/cuda-ab/ip_probe.cu @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe.cu - how many CTAs actually executed for an in-place kernel? + * + * scale_mark marks progress[blockIdx.x]=0xCAFEBABE after the c store; D2H of + * progress counts executed CTAs exactly. c (the in-place buffer) is read back + * and verified. seed (argv[1]) makes the run content-unique. + * + * Build: nvcc -O2 --cudart=static -o ip_probe ip_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_mark(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g grid=%d\n", prop.name, (double)k, grid); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + /* in-place scale with CTA markers */ + scale_mark<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0, first_missing = -1; + + for (i = 0; i < grid; i++) { + if (h_prog[i] == 0xCAFEBABEu) + marked++; + else if (first_missing < 0) + first_missing = i; + } + printf("CTA-MARK marked=%d/%d first_missing=%d\n", marked, grid, + first_missing); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + if (first >= 0) + printf("SCALE firstbad got=%g want=%g\n", hc[first], + (ha[first] + hb[first]) * 2.0f); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe2.cu b/tools/cuda-ab/ip_probe2.cu new file mode 100644 index 0000000..a5fa2ec --- /dev/null +++ b/tools/cuda-ab/ip_probe2.cu @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe2.cu - why do in-place c stores vanish while prog stores land? + * + * After a vadd that fills c, three variants run on the SAME c: + * 1. scale_if : if (i<n) c[i] *= s; (control, like ip_probe) + * 2. scale_u : c[i] *= s; (unconditional, grid exact) + * 3. fill_c : c[i] = 7.0f; (store only, no load) + * 4. add_c : c[i] = c[i] + 1.0f; (load + store) + * Each stage reads c back and compares against the expected value at that + * stage. A separate prog buffer marks CTA execution. + * + * Build: nvcc -O2 --cudart=static -o ip_probe2 ip_probe2.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_if(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +__global__ void scale_u(float *c, float s, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xDEADBEEFu; +} + +__global__ void fill_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = 7.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xFEEDFACu; +} + +__global__ void add_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] + 1.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0x0DDF00Du; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i, stage; + float mult; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + mult = 2.0f; + for (stage = 0; stage < 4; stage++) { + unsigned mark = 0; + + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + switch (stage) { + case 0: + scale_if<<<grid, 256>>>(dc, 2.0f, n, d_prog); + mult = 2.0f; + mark = 0xCAFEBABEu; + break; + case 1: + scale_u<<<grid, 256>>>(dc, 2.0f, d_prog); + mult = 4.0f; + mark = 0xDEADBEEFu; + break; + case 2: + fill_c<<<grid, 256>>>(dc, d_prog); + mark = 0xFEEDFACu; + break; + default: + add_c<<<grid, 256>>>(dc, d_prog); + mult = -1.0f; + mark = 0x0DDF00Du; + break; + } + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0; + + for (i = 0; i < grid; i++) + if (h_prog[i] == mark) + marked++; + printf("S%d CTA-MARK marked=%d/%d\n", stage + 1, + marked, grid); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + if (stage < 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) { + b++; + if (first < 0) + first = i; + } + printf("S%d readback bad=%d/%d first=%d\n", + stage + 1, b, n, first); + } else if (stage == 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 7.0f) { + b++; + if (first < 0) + first = i; + } + printf("S3 readback bad=%d/%d first=%d (fill 7.0)\n", + b, n, first); + } else { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 8.0f) { + b++; + if (first < 0) + first = i; + } + printf("S4 readback bad=%d/%d first=%d (add 1.0 to fill)\n", + b, n, first); + } + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe3.cu b/tools/cuda-ab/ip_probe3.cu new file mode 100644 index 0000000..a50df0e --- /dev/null +++ b/tools/cuda-ab/ip_probe3.cu @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe3.cu - what does the SM actually READ for the scale kernel? + * + * scale_report writes, per CTA: prog[2b] = bits of the float param s, + * prog[2b+1] = bits of c[b*256] (the load). + * The D2H of prog then shows exactly what the SM saw, independent of any + * store-visibility path for c. + * + * Build: nvcc -O2 --cudart=static -o ip_probe3 ip_probe3.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_report(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) { + prog[2 * blockIdx.x] = __float_as_uint(s); + prog[2 * blockIdx.x + 1] = __float_as_uint(c[blockIdx.x * 256]); + } +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[256]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 256 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 256 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + scale_report<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 256 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int s_ok = 0, s_1 = 0, c_ok = 0, n_seen = 0; + unsigned first_s = 0, first_c = 0; + + for (i = 0; i < grid; i++) { + unsigned sb = h_prog[2 * i]; + unsigned cb = h_prog[2 * i + 1]; + + if (i == 0) { + first_s = sb; + first_c = cb; + } + if (sb == 0x40000000u) + s_ok++; + if (sb == 0x3F800000u) + s_1++; + { + float want = ha[i * 256] + hb[i * 256]; + + if (cb == *(unsigned *)&want) + c_ok++; + } + n_seen++; + } + printf("REPORT ctas=%d s_eq_2.0=%d s_eq_1.0=%d c_load_ok=%d first_s=0x%08x first_c=0x%08x\n", + n_seen, s_ok, s_1, c_ok, first_s, first_c); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/launch_count.cu b/tools/cuda-ab/launch_count.cu new file mode 100644 index 0000000..02db1d6 --- /dev/null +++ b/tools/cuda-ab/launch_count.cu @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: BSD-3-Clause +// launch_count.cu - single-launch execution-count repro (r342). +// +// One launch, one atomic increment. The engine-written page witness showed the +// increment applied TWICE (1 -> 2; the 100-increment variant -> 200): every +// forwarded launch executes its engine work twice (one submission, one ring +// publish, completion OK). This is the root cause of the multi-pass content +// class (victory_reduce exactly 2x, scale/saxpy/smiley/multikernel wrong). +// Run in the guest; expected LAUNCH_COUNT=1. +// for a single launch, the launch path executes work more than once. +#include <cstdio> +#include <cuda_runtime.h> +__global__ void inc(unsigned long long *c) { if (threadIdx.x == 0 && blockIdx.x == 0) atomicAdd(c, 1ULL); } +int main(void) { + unsigned long long *d; unsigned long long h = 0; + if (cudaMalloc(&d, 8) != cudaSuccess) { printf("CUDA_ERROR malloc\nSTATUS FAIL\n"); return 1; } + if (cudaMemset(d, 0, 8) != cudaSuccess) { printf("CUDA_ERROR memset\nSTATUS FAIL\n"); return 1; } + inc<<<1, 32>>>(d); + if (cudaGetLastError() != cudaSuccess) { printf("CUDA_ERROR launch\nSTATUS FAIL\n"); return 1; } + if (cudaDeviceSynchronize() != cudaSuccess) { printf("CUDA_ERROR sync\nSTATUS FAIL\n"); return 1; } + if (cudaMemcpy(&h, d, 8, cudaMemcpyDeviceToHost) != cudaSuccess) { printf("CUDA_ERROR d2h\nSTATUS FAIL\n"); return 1; } + printf("LAUNCH_COUNT=%llu (expect 1)\n", h); + printf("STATUS %s\n", h == 1 ? "OK" : "FAIL"); + return h == 1 ? 0 : 1; +} diff --git a/tools/cuda-ab/memset2d_probe.cu b/tools/cuda-ab/memset2d_probe.cu new file mode 100644 index 0000000..acf43d4 --- /dev/null +++ b/tools/cuda-ab/memset2d_probe.cu @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset2d_probe.cu - per-phase diagnostics for victory_memset2d. + * + * victory_memset2d reports one aggregate bad count (~12.5M) that mixes all + * three phases. This probe runs the SAME three phases and reports each + * phase's bad count, the first mismatching byte (offset, got, want), and a + * per-phase 1MB-chunk histogram so the failing phase and the failing region + * are attributable in one run. + * + * Phases: + * P1 memset : cudaMemset(d_img, 0xA5, pitch*h); D2H 2D readback + * P2 roundtrip: H2D 2D pattern; D2H 2D readback + * P3 memset2d : cudaMemset2D quadrant 0x3C; D2H 2D readback + * + * Output lines: + * DEVICE ... + * PHASE p=1 bad=<n> first_off=<i> got=<g> want=<w> + * PHIST p=1 mb=<m> ok=<n> bad=<n> + * MEMSET2D2 bad_total=<n> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define IMG_W 4096 +#define IMG_H 1024 +#define FILL_BYTE 0xA5 +#define QUAD_BYTE 0x3C + +static int phase_check(const char *tag, const unsigned char *back, + const unsigned char *want, size_t n, int p) +{ + size_t i, first = (size_t)-1; + int bad = 0, got = 0, wantv = 0; + size_t mb; + + for (i = 0; i < n; i++) { + if (back[i] != want[i]) { + if (first == (size_t)-1) { + first = i; + got = back[i]; + wantv = want[i]; + } + bad++; + } + } + printf("PHASE p=%d bad=%d first_off=%zu got=0x%02x want=0x%02x\n", + p, bad, first, got, wantv); + if (bad) { + size_t nb = (n + (1u << 20) - 1) >> 20; + + for (mb = 0; mb < nb && mb < 8; mb++) { + size_t lo = mb << 20, hi = lo + (1u << 20); + size_t j; + int bok = 0, bbad = 0; + + if (hi > n) + hi = n; + for (j = lo; j < hi; j++) { + if (back[j] == want[j]) + bok++; + else + bbad++; + } + printf("PHIST p=%d mb=%zu ok=%d bad=%d\n", + p, mb, bok, bbad); + } + } + (void)tag; + return bad; +} + +int main(void) +{ + cudaDeviceProp prop; + const size_t w = IMG_W, h = IMG_H; + unsigned char *h_img, *h_back, *h_want; + unsigned char *d_img; + size_t pitch = 0; + int bad = 0, row, col; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_img = (unsigned char *)malloc(w * h); + h_back = (unsigned char *)malloc(w * h); + h_want = (unsigned char *)malloc(w * h); + if (!h_img || !h_back || !h_want) + return 1; + + CHECK(cudaMallocPitch(&d_img, &pitch, w, h)); + printf("PITCH pitch=%zu\n", pitch); + printf("DPTRS h_img=%p h_back=%p h_want=%p d_img=%p\n", (void *)h_img, + (void *)h_back, (void *)h_want, (void *)d_img); + + /* P1: full cudaMemset over the pitched allocation, 2D readback. */ + CHECK(cudaMemset(d_img, FILL_BYTE, pitch * h)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + memset(h_want, FILL_BYTE, w * h); + bad += phase_check("P1", h_back, h_want, w * h, 1); + + /* P2: pitched H2D pattern -> D2H round trip. */ + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) + h_img[row * w + col] = (unsigned char)((row + col) & 0xFF); + CHECK(cudaMemcpy2D(d_img, pitch, h_img, w, w, h, + cudaMemcpyHostToDevice)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P2", h_back, h_img, w * h, 2); + + /* P3: cudaMemset2D of the upper-left quadrant only. */ + CHECK(cudaMemset2D(d_img, pitch, QUAD_BYTE, w / 2, h / 2)); + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) { + unsigned char expect = h_img[row * w + col]; + + if (row < (int)(h / 2) && col < (int)(w / 2)) + expect = QUAD_BYTE; + h_want[row * w + col] = expect; + } + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P3", h_back, h_want, w * h, 3); + + printf("MEMSET2D2 bad_total=%d\n", bad); + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/memset_probe.cu b/tools/cuda-ab/memset_probe.cu new file mode 100644 index 0000000..a6a12e0 --- /dev/null +++ b/tools/cuda-ab/memset_probe.cu @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset_probe.cu - isolate the memcpy_roundtrip memset-phase failure. + * + * cudaMemset(8 MB, 0x5A) -> D2H -> byte verify, with a run-length report: + * the first bad offset, and a coverage map at 4 KB granularity (correct / + * wrong / untouched-malloc-prefill) so "fill under-delivered" is + * distinguishable from "readback delivered wrong pages". + * + * Build: nvcc -O2 --cudart=static -o memset_probe memset_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define PAGE 4096 +#define MemsetByte 0x5A + +int main(void) +{ + const size_t bytes = 8u << 20; + unsigned char *h; + unsigned char *d; + size_t i; + int bad = 0; + size_t first_bad = (size_t)-1; + size_t n_pages, p; + + setvbuf(stdout, NULL, _IONBF, 0); + h = (unsigned char *)malloc(bytes); + if (!h) + return 1; + memset(h, 0xEE, bytes); + + CHECK(cudaMalloc(&d, bytes)); + printf("DPTR d=%p bytes=%zu\n", (void *)d, bytes); + + CHECK(cudaMemset(d, MemsetByte, bytes)); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < bytes; i++) { + if (h[i] != (unsigned char)MemsetByte) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("FIRST_BAD off=%zu (0x%zx) bad=%d total=%zu\n", + first_bad, first_bad, bad, bytes); + n_pages = bytes / PAGE; + for (p = 0; p < n_pages; p++) { + int good = 1, prefill = 1; + + for (i = 0; i < PAGE; i++) { + if (h[p * PAGE + i] != (unsigned char)MemsetByte) + good = 0; + if (h[p * PAGE + i] != 0xEE) + prefill = 0; + if (!good && !prefill) + break; + } + if (!good) { + printf("PAGE %zu %s b0=%02x b1=%02x b2=%02x b3=%02x b4095=%02x\n", + p, prefill ? "PREFILL" : "OTHER", + h[p * PAGE], h[p * PAGE + 1], + h[p * PAGE + 2], h[p * PAGE + 3], + h[p * PAGE + 4095]); + if (p > 24) + break; + } + } + printf("STATUS %s\n", bad == 0 ? "OK" : "FAIL"); + return 0; +} diff --git a/tools/cuda-ab/push_battery.sh b/tools/cuda-ab/push_battery.sh index 644beba..7bd3df9 100755 --- a/tools/cuda-ab/push_battery.sh +++ b/tools/cuda-ab/push_battery.sh @@ -6,7 +6,7 @@ set -u HERE="$(cd "$(dirname "$0")" && pwd)" RUN="$HERE/run" GUEST_DIR="${1:-/tmp}" -CLIENTS="victory_mail smiley vecadd_loop streams2 memcpy_roundtrip multikernel victory_saxpy victory_reduce victory_matmul victory_strided victory_memset2d victory_atomics victory_streams_events victory_math vector_add" +CLIENTS="victory_mail smiley vecadd_loop streams2 memcpy_roundtrip multikernel victory_saxpy victory_reduce victory_matmul victory_strided victory_memset2d victory_atomics victory_streams_events victory_math vector_add inplace_scale streams2_probe streams_events_probe memset2d_probe tgemm_probe transformer transformer_stages ip_probe ip_probe2 ip_probe3 scale_probe rt_probe launch_count egl_frame frame_producer" export SSHPASS="${VGPU_GUEST_PASS:-ubuntu}" FILES="" for c in $CLIENTS; do [ -x "$RUN/$c" ] && FILES="$FILES $RUN/$c"; done diff --git a/tools/cuda-ab/r18_anim.cu b/tools/cuda-ab/r18_anim.cu new file mode 100644 index 0000000..89111be --- /dev/null +++ b/tools/cuda-ab/r18_anim.cu @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * r18_anim.cu - GPU-computed animation for R18 (v3: single-page kernel). + * + * Renders NFRAMES frames of an animated plasma with one launch per frame. + * The emulated forward path delivers the kernel through ONE 4 KB code page + * (the window shadow copies the program page; multi-page kernels execute off + * the end of the mapped page - the 904-instruction v2 kernel faulted with + * "Illegal Instruction Encoding" on the real engine). This version keeps the + * SASS text under 4096 bytes: 512x360 RGBA8 (power-of-two width for cheap + * index math), grid-stride loop, three MUFU.SIN under --use_fast_math. + * + * ONE 0.9 MB device frame buffer is reused for every frame (small-alloc + * regime, matching the proven CUDA path); after each launch the frame is + * copied back (DtoH) and APPENDED to /dev/shm with a magic header so the + * host can find it by scanning the guest RAM file. The file is + * + * "R18FRM01" u32 count u32 width u32 height u32 stride + * then count*height*stride bytes of RGBA8 + * + * The header is written first with the final count and bytes are appended + * (flushed) per frame, so a host-side harvest can recover every completed + * frame even if the client or guest wedges mid-run. + * + * Build (host): + * nvcc -O2 --use_fast_math --cudart=static -arch=sm_75 -o r18_anim r18_anim.cu + * Usage: r18_anim [frames] [outpath] [blocks] + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <ctime> +#include <cuda_runtime.h> + +#define WIDTH 512 +#define HEIGHT 360 +#define NFRAMES 60 +#define OUTPATH "/dev/shm/r18frames.bin" + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +/* 3 MUFU.SIN only - the text must stay under one 4 KB page. */ +__global__ void plasma_kernel(uchar4 *fb, int w, int h, int frame) +{ + unsigned stride = gridDim.x * blockDim.x; + unsigned xmask = (unsigned)w - 1u; /* w is a power of two */ + int idx; + + for (idx = blockIdx.x * blockDim.x + threadIdx.x; idx < w * h; + idx += stride) { + float u = (float)((unsigned)idx & xmask) * (1.0f / 512.0f); + float v = (float)((unsigned)idx >> 9) * (1.0f / 360.0f); + float t = (float)frame * 0.10f; + float p = sinf(u * 9.0f + t * 2.0f) + + sinf(v * 7.0f - t * 2.0f); + float s = sinf(p * 1.5f); + + p *= 0.5f; /* -1 .. 1 */ + fb[idx] = make_uchar4((unsigned char)(127.5f + 127.5f * s), + (unsigned char)(127.5f + 127.5f * p), + (unsigned char)(127.5f - 127.5f * s), + 255); + } +} + +static uint64_t fnv1a(const unsigned char *p, size_t n) +{ + uint64_t h = 1469598103934665603ull; + + while (n--) { + h ^= *p++; + h *= 1099511628211ull; + } + return h; +} + +static double now_ms(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (double)ts.tv_sec * 1e3 + (double)ts.tv_nsec / 1e6; +} + +int main(int argc, char **argv) +{ + int nframes = NFRAMES; + const char *outpath = OUTPATH; + size_t fbytes = (size_t)WIDTH * HEIGHT * 4; + uchar4 *d_fb = NULL; + unsigned char *h_fb = NULL; + double km = 0.0, cm = 0.0, t0, t1, wall0 = now_ms(); + int blocks = 720; + int f, dev = -1; + FILE *fp; + uint32_t hdr[4]; + + if (argc > 1) + nframes = atoi(argv[1]); + if (nframes <= 0 || nframes > 60000) + nframes = NFRAMES; + if (argc > 2) + outpath = argv[2]; + if (argc > 3) + blocks = atoi(argv[3]); + if (blocks <= 0 || blocks > 4096) + blocks = 720; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDevice(&dev)); + { + cudaDeviceProp prop; + + if (cudaGetDeviceProperties(&prop, dev) == cudaSuccess) + printf("DEVICE %s cc=%d.%d\n", prop.name, + prop.major, prop.minor); + } + printf("ANIM start frames=%d %dx%d blocks=%d out=%s\n", + nframes, WIDTH, HEIGHT, blocks, outpath); + + CHECK(cudaMalloc(&d_fb, fbytes)); + h_fb = (unsigned char *)malloc(fbytes); + if (!h_fb) { + printf("HOST_ALLOC_FAIL\nSTATUS FAIL\n"); + return 1; + } + + fp = fopen(outpath, "wb"); + if (!fp) { + printf("OPEN_FAIL %s\nSTATUS FAIL\n", outpath); + return 1; + } + hdr[0] = (uint32_t)nframes; + hdr[1] = WIDTH; + hdr[2] = HEIGHT; + hdr[3] = WIDTH * 4; + if (fwrite("R18FRM01", 1, 8, fp) != 8 || + fwrite(hdr, 1, sizeof(hdr), fp) != sizeof(hdr) || fflush(fp)) { + printf("WRITE_FAIL header\nSTATUS FAIL\n"); + fclose(fp); + return 1; + } + + for (f = 0; f < nframes; f++) { + uchar4 *dst = d_fb; + uint64_t sum = 0; + size_t i; + + t0 = now_ms(); + plasma_kernel<<<blocks, 256>>>(dst, WIDTH, HEIGHT, f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + t1 = now_ms(); + km += t1 - t0; + CHECK(cudaMemcpy(h_fb, dst, fbytes, cudaMemcpyDeviceToHost)); + cm += now_ms() - t1; + + for (i = 0; i < fbytes; i++) + sum += h_fb[i]; + if (fwrite(h_fb, 1, fbytes, fp) != fbytes || fflush(fp)) { + printf("WRITE_FAIL frame %d\nSTATUS FAIL\n", f); + fclose(fp); + return 1; + } + printf("FRAME %03d fnv=0x%016llx mean=%u kms=%.1f cms=%.1f wall=%.1f\n", + f, (unsigned long long)fnv1a(h_fb, fbytes), + (unsigned)(sum / fbytes), km, cm, now_ms() - wall0); + } + + fclose(fp); + printf("KERNEL frames=%d %dx%d blocks=%d launches=%d total_ms=%.2f per_frame_ms=%.2f\n", + nframes, WIDTH, HEIGHT, blocks, nframes, km, km / nframes); + printf("COPY frames=%d total_ms=%.2f per_frame_ms=%.2f\n", nframes, + cm, cm / nframes); + printf("WRITE path=%s bytes=%zu magic=R18FRM01 count=%d %dx%d stride=%d wall_ms=%.1f\n", + outpath, 24 + fbytes * (size_t)nframes, nframes, WIDTH, HEIGHT, + WIDTH * 4, now_ms() - wall0); + printf("STATUS OK\n"); + + CHECK(cudaFree(d_fb)); + free(h_fb); + return 0; +} diff --git a/tools/cuda-ab/rt_probe.cu b/tools/cuda-ab/rt_probe.cu new file mode 100644 index 0000000..025301d --- /dev/null +++ b/tools/cuda-ab/rt_probe.cu @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * rt_probe.cu - instrumented memcpy_roundtrip variant isolating the + * memset-phase D2H readback failure. + * + * Phases (argv[1]): + * "full" : H2D -> kernel -> D2H -> memset -> D2H (the original order) + * "sync" : same but cudaDeviceSynchronize() before the second D2H + * "memset" : memset -> D2H only (memset_probe shape, 3-buffer alloc) + * "rev" : memset -> D2H first, then H2D -> kernel -> D2H + * + * Forensics on the memset readback: first bad offset, byte histogram + * (0x5A / 0x00 / other), per-1MB chunk histogram, and the first 32 bytes. + * + * Build: nvcc -O2 --cudart=static -o rt_probe rt_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define MemsetByte 0x5A + +__global__ void bump(const float *in, float *out, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + out[i] = in[i] + 0.5f; +} + +static void memset_forensics(const unsigned char *h, size_t bytes, + const char *tag) +{ + size_t i, first_bad = (size_t)-1; + int bad = 0, n5a = 0, n00 = 0, nother = 0; + size_t chunk; + + for (i = 0; i < bytes; i++) { + if (h[i] == 0x5A) + n5a++; + else if (h[i] == 0x00) + n00++; + else + nother++; + if (h[i] != 0x5A) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("MEMSET-READ %s bad=%d first_bad=%zu n5a=%d n00=%d nother=%d\n", + tag, bad, first_bad, n5a, n00, nother); + printf("MEMSET-HEAD %s", tag); + for (i = 0; i < 32; i++) + printf(" %02x", h[i]); + printf("\n"); + if (first_bad != (size_t)-1) { + printf("MEMSET-ATBAD %s off=%zu:", tag, first_bad); + for (i = first_bad; i < first_bad + 16 && i < bytes; i++) + printf(" %02x", h[i]); + printf("\n"); + } + for (chunk = 0; chunk < bytes / (1u << 20); chunk++) { + size_t lo = chunk << 20, hi = lo + (1u << 20); + int c5a = 0, c00 = 0, cother = 0; + + for (i = lo; i < hi; i++) { + if (h[i] == 0x5A) + c5a++; + else if (h[i] == 0x00) + c00++; + else + cother++; + } + printf("MEMSET-MB %s mb=%zu 5a=%d 00=%d other=%d\n", + tag, chunk, c5a, c00, cother); + } +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "full"; + cudaDeviceProp prop; + const size_t bytes = 8u << 20; + const int n = (int)(bytes / sizeof(float)); + unsigned char *h_m; + float *h_in, *h_out; + float *d_in, *d_out; + unsigned char *d_m; + double csum = 0.0; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu mode=%s\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem, mode); + + h_in = (float *)malloc(bytes); + h_out = (float *)malloc(bytes); + h_m = (unsigned char *)malloc(bytes); + if (!h_in || !h_out || !h_m) + return 1; + + for (i = 0; i < n; i++) + h_in[i] = (float)(i % 1024) * 0.25f; + + CHECK(cudaMalloc(&d_in, bytes)); + CHECK(cudaMalloc(&d_out, bytes)); + CHECK(cudaMalloc(&d_m, bytes)); + printf("DPTRS d_in=%p d_out=%p d_m=%p\n", (void *)d_in, + (void *)d_out, (void *)d_m); + + if (!strcmp(mode, "memset") || !strcmp(mode, "rev")) { + /* memset phase first */ + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "early"); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync") || + !strcmp(mode, "rev")) { + CHECK(cudaMemcpy(d_in, h_in, bytes, cudaMemcpyHostToDevice)); + bump<<<grid, 256>>>(d_in, d_out, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < n; i++) { + if (h_out[i] != h_in[i] + 0.5f) + bad++; + csum += (double)h_out[i]; + } + printf("BUMP bad=%d csum=%.6f\n", bad, csum); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync")) { + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "late"); + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/run/egl_frame b/tools/cuda-ab/run/egl_frame Binary files differnew file mode 100755 index 0000000..130f032 --- /dev/null +++ b/tools/cuda-ab/run/egl_frame diff --git a/tools/cuda-ab/run/evhold b/tools/cuda-ab/run/evhold Binary files differnew file mode 100755 index 0000000..de4448c --- /dev/null +++ b/tools/cuda-ab/run/evhold diff --git a/tools/cuda-ab/run/frame_producer b/tools/cuda-ab/run/frame_producer Binary files differnew file mode 100755 index 0000000..a202ac7 --- /dev/null +++ b/tools/cuda-ab/run/frame_producer diff --git a/tools/cuda-ab/run/inplace_scale b/tools/cuda-ab/run/inplace_scale Binary files differnew file mode 100755 index 0000000..bb47f42 --- /dev/null +++ b/tools/cuda-ab/run/inplace_scale diff --git a/tools/cuda-ab/run/ip4 b/tools/cuda-ab/run/ip4 Binary files differnew file mode 100755 index 0000000..54937e6 --- /dev/null +++ b/tools/cuda-ab/run/ip4 diff --git a/tools/cuda-ab/run/ip5 b/tools/cuda-ab/run/ip5 Binary files differnew file mode 100755 index 0000000..c4b4a39 --- /dev/null +++ b/tools/cuda-ab/run/ip5 diff --git a/tools/cuda-ab/run/ip6 b/tools/cuda-ab/run/ip6 Binary files differnew file mode 100755 index 0000000..56815c2 --- /dev/null +++ b/tools/cuda-ab/run/ip6 diff --git a/tools/cuda-ab/run/ip_probe b/tools/cuda-ab/run/ip_probe Binary files differnew file mode 100755 index 0000000..08fb86e --- /dev/null +++ b/tools/cuda-ab/run/ip_probe diff --git a/tools/cuda-ab/run/ip_probe2 b/tools/cuda-ab/run/ip_probe2 Binary files differnew file mode 100755 index 0000000..dfe6ab9 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe2 diff --git a/tools/cuda-ab/run/ip_probe3 b/tools/cuda-ab/run/ip_probe3 Binary files differnew file mode 100755 index 0000000..8a34c05 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe3 diff --git a/tools/cuda-ab/run/launch_count b/tools/cuda-ab/run/launch_count Binary files differnew file mode 100755 index 0000000..0c67f45 --- /dev/null +++ b/tools/cuda-ab/run/launch_count diff --git a/tools/cuda-ab/run/memset2d_probe b/tools/cuda-ab/run/memset2d_probe Binary files differnew file mode 100755 index 0000000..f5bd071 --- /dev/null +++ b/tools/cuda-ab/run/memset2d_probe diff --git a/tools/cuda-ab/run/memset_probe b/tools/cuda-ab/run/memset_probe Binary files differnew file mode 100755 index 0000000..bd9df5e --- /dev/null +++ b/tools/cuda-ab/run/memset_probe diff --git a/tools/cuda-ab/run/r18_anim b/tools/cuda-ab/run/r18_anim Binary files differnew file mode 100755 index 0000000..6a944f9 --- /dev/null +++ b/tools/cuda-ab/run/r18_anim diff --git a/tools/cuda-ab/run/rt_probe b/tools/cuda-ab/run/rt_probe Binary files differnew file mode 100755 index 0000000..48d866f --- /dev/null +++ b/tools/cuda-ab/run/rt_probe diff --git a/tools/cuda-ab/run/scale_probe b/tools/cuda-ab/run/scale_probe Binary files differnew file mode 100755 index 0000000..ca3fc20 --- /dev/null +++ b/tools/cuda-ab/run/scale_probe diff --git a/tools/cuda-ab/run/streams2_probe b/tools/cuda-ab/run/streams2_probe Binary files differnew file mode 100755 index 0000000..206d6d6 --- /dev/null +++ b/tools/cuda-ab/run/streams2_probe diff --git a/tools/cuda-ab/run/streams2d b/tools/cuda-ab/run/streams2d Binary files differnew file mode 100755 index 0000000..90b6df8 --- /dev/null +++ b/tools/cuda-ab/run/streams2d diff --git a/tools/cuda-ab/run/streams_events_probe b/tools/cuda-ab/run/streams_events_probe Binary files differnew file mode 100755 index 0000000..00f3539 --- /dev/null +++ b/tools/cuda-ab/run/streams_events_probe diff --git a/tools/cuda-ab/run/tgemm_probe b/tools/cuda-ab/run/tgemm_probe Binary files differnew file mode 100755 index 0000000..d9c9aa3 --- /dev/null +++ b/tools/cuda-ab/run/tgemm_probe diff --git a/tools/cuda-ab/run/transformer b/tools/cuda-ab/run/transformer Binary files differnew file mode 100755 index 0000000..bcde39e --- /dev/null +++ b/tools/cuda-ab/run/transformer diff --git a/tools/cuda-ab/run/transformer_async b/tools/cuda-ab/run/transformer_async Binary files differnew file mode 100755 index 0000000..2911b5c --- /dev/null +++ b/tools/cuda-ab/run/transformer_async diff --git a/tools/cuda-ab/run/transformer_stages b/tools/cuda-ab/run/transformer_stages Binary files differnew file mode 100755 index 0000000..1461a3e --- /dev/null +++ b/tools/cuda-ab/run/transformer_stages diff --git a/tools/cuda-ab/run/wedge_ab b/tools/cuda-ab/run/wedge_ab Binary files differnew file mode 100755 index 0000000..23a90aa --- /dev/null +++ b/tools/cuda-ab/run/wedge_ab diff --git a/tools/cuda-ab/run/wedge_bisect b/tools/cuda-ab/run/wedge_bisect Binary files differnew file mode 100755 index 0000000..5898fcf --- /dev/null +++ b/tools/cuda-ab/run/wedge_bisect diff --git a/tools/cuda-ab/run/wedge_recover b/tools/cuda-ab/run/wedge_recover Binary files differnew file mode 100755 index 0000000..3d9351c --- /dev/null +++ b/tools/cuda-ab/run/wedge_recover diff --git a/tools/cuda-ab/scale_probe.cu b/tools/cuda-ab/scale_probe.cu new file mode 100644 index 0000000..8a0df59 --- /dev/null +++ b/tools/cuda-ab/scale_probe.cu @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * scale_probe.cu - localize the deterministic "in-place scale writes only the + * first 4 KB page" failure. + * + * Modes (argv[1]): + * V : vadd(a,b->c); sync; D2H+verify; vadd again; D2H+verify + * I : vadd(a,b->c); sync; D2H+verify; scale(c,*2 in place); sync; + * D2H+verify (with page map); scale again; sync; D2H+verify; D2H again + * O : vadd(a,b->c); sync; D2H+verify; scale_out(c->d,*2); sync; + * D2H+verify (with page map) + * + * Every stage also runs a GPU-side verify kernel that counts matching + * elements reading the device buffers (engine's own view) and writes the + * counts to a small buffer, D2H'd afterwards: separates "device pages never + * written" (GPU count low) from "D2H readback stale" (GPU count high, host + * count low). + * + * Build: nvcc -O2 --cudart=static -o scale_probe scale_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +__global__ void scale_out(const float *c, float *d, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + d[i] = c[i] * s; +} + +/* verify kernel: out[0]=count(c==a+b), out[1]=count(c==2*(a+b)) */ +__global__ void vdigest(const float *a, const float *b, const float *c, int n, + unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c[i] == a[i] + b[i]) + atomicAdd(&out[0], 1ull); + if (c[i] == (a[i] + b[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "V"; + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hd; + float *da, *db, *dc, *dd; + unsigned long long *d_out, h_out[2]; + int i; + float k = argc > 2 ? atof(argv[2]) : 0.0f; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + { + FILE *f = fopen("/proc/uptime", "r"); + double up = -1.0; + + if (f) { if (fscanf(f, "%lf", &up) != 1) up = -1.0; fclose(f); } + printf("UPTIME %.3f\n", up); + } + printf("DEVICE %s mode=%s seed=%g\n", prop.name, mode, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hd = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + memset(hd, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&dd, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da=%p db=%p dc=%p dd=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)dd); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S1 vadd readback bad=%d/%d\n", b, n); + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S1 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", h_out[0], + h_out[1]); + } + + if (!strcmp(mode, "V")) { + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S2 vadd-again readback bad=%d/%d\n", b, n); + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "I")) { + int stage; + + for (stage = 0; stage < 2; stage++) { + float mult = stage == 0 ? 2.0f : 4.0f; + + scale<<<grid, 256>>>(dc, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) + b++; + printf("S%d inplace-scale readback bad=%d/%d (mult=%g)\n", + 2 + stage, b, n, mult); + if (stage == 0) + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hc[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * mult) + pb++; + if (pb) + printf("S%d PAGE %d bad=%d/256 first=%g want=%g\n", + 2 + stage, page, + pb, hc[page * 256], + (ha[page * 256] + + hb[page * 256]) * mult); + } + } + if (stage == 0) { + CHECK(cudaMemset(d_out, 0, + 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", + h_out[0], h_out[1]); + } + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "O")) { + scale_out<<<grid, 256>>>(dc, dd, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hd, dd, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hd[i] != (ha[i] + hb[i]) * 2.0f) + b++; + printf("S2 outofplace-scale readback bad=%d/%d\n", b, n); + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hd[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * 2.0f) + pb++; + if (pb) + printf("S2 PAGE %d bad=%d/256\n", page, + pb); + } + } + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dd, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST d_eq_ab=%llu d_eq_2ab=%llu\n", h_out[0], + h_out[1]); + printf("STATUS DONE\n"); + return 0; + } + return 1; +} diff --git a/tools/cuda-ab/soak.sh b/tools/cuda-ab/soak.sh index f870fda..4f1430f 100755 --- a/tools/cuda-ab/soak.sh +++ b/tools/cuda-ab/soak.sh @@ -135,6 +135,19 @@ run_one() { r="$(gssh "cat $out 2>/dev/null")" || true [ -n "$r" ] && text="$r" printf '%s' "$text" | grep -q 'EXIT=' && break + # The guest wrapper's EXIT= marker can be lost when the guest's + # process reaping stalls (observed: the client reached STATUS OK + # but its zombie was never reaped and `timeout` never wrote the + # marker). A terminal STATUS line is proof of completion by + # itself - accept it after a short grace so the EXIT= marker + # still wins when it does arrive. + if printf '%s' "$text" | grep -qE 'STATUS (OK|FAIL)'; then + sleep 10 + r="$(gssh "cat $out 2>/dev/null")" || true + [ -n "$r" ] && text="$r" + printf '%s' "$text" | grep -q 'EXIT=' && break + break + fi tnow=$(date +%s) if [ $((tnow - t0)) -gt $((TMO + 120)) ]; then if client_alive "$bin"; then @@ -179,7 +192,7 @@ log "SOAK START mode=$MODE seconds=$SECS guest_dir=$GUEST_DIR timeout=${TMO}s lo # cannot silently shrink the battery. found="" for i in 1 2 3; do - found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done")" || found="" + found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done; true")" || found="" [ -n "$found" ] && break sleep 3 done diff --git a/tools/cuda-ab/streams2_probe.cu b/tools/cuda-ab/streams2_probe.cu new file mode 100644 index 0000000..5bebab4 --- /dev/null +++ b/tools/cuda-ab/streams2_probe.cu @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2_probe.cu - instrumented streams2 to localize the deterministic + * failure: per-array mismatch counts, first-bad indices with values, and a + * GPU-side verify-kernel digest of the device buffers (to separate "device + * pages never written" from "D2H readback wrong"). + * + * Modes (argv[1]): + * A : exactly streams2's shape (control) + * B : like A but the two D2H copy-backs use synchronous cudaMemcpy + * C : like A but no events cross the streams; both streams are joined + * with cudaStreamSynchronize before any copy-back + * D : single stream (everything on s1) + * + * Build: nvcc -O2 --cudart=static -o streams2_probe streams2_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +/* GPU-side digest: counts of matches for c1==a1+b1 and c2==(a2+b2)*2 */ +__global__ void digest(const float *a1, const float *b1, const float *c1, + const float *a2, const float *b2, const float *c2, + int n, unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c1[i] == a1[i] + b1[i]) + atomicAdd(&out[0], 1ull); + if (c2[i] == (a2[i] + b2[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +static int report(const char *tag, const float *h_c, const float *h_a, + const float *h_b, const float *h_a2, const float *h_b2, + int n, float scale2) +{ + int bad = 0, i, shown = 0; + + for (i = 0; i < n; i++) { + float want = h_a[i] + h_b[i]; + + if (scale2 != 1.0f) + want = (h_a2[i] + h_b2[i]) * scale2; + if (h_c[i] != want) { + bad++; + if (shown < 4) { + printf("FIRSTBAD %s i=%d got=%g want=%g\n", + tag, i, h_c[i], want); + shown++; + } + } + } + printf("BAD %s n=%d bad=%d\n", tag, n, bad); + return bad; +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "A"; + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + unsigned long long *d_out, h_out[2]; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s mode=%s\n", prop.name, mode); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + memset(h_c1, 0xCD, bytes); + memset(h_c2, 0xCD, bytes); + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da1=%p db1=%p dc1=%p da2=%p db2=%p dc2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, (void *)d_a2, + (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + if (!strcmp(mode, "D")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s1)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s1>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + scale<<<grid, 256, 0, s1>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + goto verify; + } + + if (!strcmp(mode, "S")) { + /* staged verification: check device content after every step */ + float *hb1 = (float *)malloc(bytes); + float *hb2 = (float *)malloc(bytes); + float *hb3 = (float *)malloc(bytes); + + CHECK(cudaMemcpy(d_a1, h_a1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b1, h_b1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_a2, h_a2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b2, h_b2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(hb1, d_a1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a1[i]) b++; + printf("STAGE-IN da1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b1[i]) b++; + printf("STAGE-IN db1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_a2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a2[i]) b++; + printf("STAGE-IN da2 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b2[i]) b++; + printf("STAGE-IN db2 bad=%d\n", b); } + + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; + if (first < 0) { first = i; + printf("S1-FIRSTBAD i=%d got=%g want=%g a2b2=%g\n", + i, hb1[i], h_a1[i]+h_b1[i], h_a2[i]+h_b2[i]); } } + printf("STAGE vadd1->c1 readback bad=%d first=%d\n", b, first); } + + CHECK(cudaDeviceSynchronize()); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb2, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb2[i] != h_a2[i] + h_b2[i]) { b++; if (first < 0) first = i; } + printf("STAGE vadd2->c2 readback bad=%d first=%d\n", b, first); } + + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb3, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb3[i] != (h_a2[i] + h_b2[i]) * 2.0f) { b++; if (first < 0) first = i; } + printf("STAGE scale->c2 readback bad=%d first=%d\n", b, first); } + /* second readback of c1 AFTER everything: does it change? */ + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; if (first < 0) first = i; } + printf("STAGE c1-late readback bad=%d first=%d\n", b, first); } + printf("STATUS OK\n"); + return 0; + } + + if (!strcmp(mode, "C")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + goto verify; + } + + /* A and B: original shape */ + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + if (!strcmp(mode, "B")) { + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + } else { + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + } + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + +verify: + { + int bad1, bad2; + + bad1 = report("C1", h_c1, h_a1, h_b1, h_a2, h_b2, n, 1.0f); + bad2 = report("C2", h_c2, h_a2, h_b2, h_a2, h_b2, n, s2); + printf("STREAMS n=%d bad=%d\n", n, bad1 + bad2); + printf("TIMING ms=%.3f\n", ms); + + /* GPU-side digest of the device buffers */ + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + digest<<<grid, 256>>>(d_a1, d_b1, d_c1, d_a2, d_b2, d_c2, n, + d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("GPU-DIGEST c1_ok=%llu c2_ok=%llu\n", h_out[0], + h_out[1]); + printf("STATUS %s\n", (bad1 | bad2) ? "FAIL" : "OK"); + } + return 0; +} diff --git a/tools/cuda-ab/streams2d.cu b/tools/cuda-ab/streams2d.cu new file mode 100644 index 0000000..04a0866 --- /dev/null +++ b/tools/cuda-ab/streams2d.cu @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2d.cu - streams2 with per-array / per-range diagnostic reporting. + * + * Same workload as streams2.cu (two streams, cross-stream event ordering) + * but instead of one "bad" count it reports, per array and per 4 KB block, + * how many elements mismatched, and prints the device pointers so a host + * log can be correlated with what the engine read/wrote. + * + * Output lines: + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * PTR <name>=0x<hex> (d_a1 d_b1 d_c1 d_a2 d_b2 d_c2) + * DIFF c1 total=<n> blocks=<first block index with mismatches> + * DIFF c2 total=<n> blocks=<...> + * STREAMS n=<count> bad=<mismatches> + * TIMING ms=<elapsed> + * STATUS OK|FAIL + * + * Build: nvcc -O2 --cudart=static -o streams2d streams2d.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +static void report_diff(const char *name, const float *got, + const float *exp, int n) +{ + int i, bad = 0, first = -1, last = -1; + int blk_hits = 0, blk_first = -1; + int nb = (n + 1023) / 1024; + int b; + + for (i = 0; i < n; i++) { + if (got[i] != exp[i]) { + bad++; + if (first < 0) + first = i; + last = i; + } + } + for (b = 0; b < nb; b++) { + int j, hit = 0; + + for (j = b * 1024; j < (b + 1) * 1024 && j < n; j++) + if (got[j] != exp[j]) { + hit = 1; + break; + } + if (hit) { + blk_hits++; + if (blk_first < 0) + blk_first = b; + } + } + printf("DIFF %s total=%d first=%d last=%d blocks=%d/%d blk_first=%d\n", + name, bad, first, last, blk_hits, nb, blk_first); + if (first >= 0) + printf("DIFF %s sample got[%d]=%a exp[%d]=%a got[%d]=%a exp[%d]=%a\n", + name, first, got[first], first, exp[first], last, + got[last], last, exp[last]); +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + if (!h_a1 || !h_b1 || !h_c1 || !h_a2 || !h_b2 || !h_c2) + return 1; + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + printf("PTR d_a1=%p d_b1=%p d_c1=%p d_a2=%p d_b2=%p d_c2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, + (void *)d_a2, (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + report_diff("c1", h_c1, h_a1, n); + { + float *exp2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) + exp2[i] = (h_a2[i] + h_b2[i]) * s2; + report_diff("c2", h_c2, exp2, n); + for (i = 0; i < n; i++) { + if (h_c1[i] != h_a1[i] + h_b1[i]) + bad++; + if (h_c2[i] != exp2[i]) + bad++; + } + free(exp2); + } + + printf("STREAMS n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/streams_events_probe.cu b/tools/cuda-ab/streams_events_probe.cu new file mode 100644 index 0000000..94aef69 --- /dev/null +++ b/tools/cuda-ab/streams_events_probe.cu @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams_events_probe.cu - instrumented victory_streams_events. + * + * Same 3-stream event chain (c0 = a0+b0; c1 = (a1+b1)+c0; c2 = (a2+b2)+c1) + * but with per-buffer diagnostics: device pointers, per-buffer bad counts, + * and the first 8 expected/actual values of every output so a stale-input + * vs stale-readback failure is distinguishable in one run. + * + * Output lines: + * DEVICE ... + * DPTRS da0..dc2 (device), ha0..hc2 (host) + * BUF s=<n> bad=<n> + * FIRST s=<n> exp=<v0..v7> got=<v0..v7> + * STREAMS3 n=<count> bad=<mismatches> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define NSTREAMS 3 + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vadd_into(float *c, const float *s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] + s[i]; +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 16; + size_t bytes = (size_t)n * sizeof(float); + float *h_a[NSTREAMS], *h_b[NSTREAMS], *h_c[NSTREAMS]; + float *d_a[NSTREAMS], *d_b[NSTREAMS], *d_c[NSTREAMS]; + cudaStream_t st[NSTREAMS]; + cudaEvent_t ev[NSTREAMS], t0, t1; + float ms = 0.0f; + int bad = 0, i, s; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + for (s = 0; s < NSTREAMS; s++) { + h_a[s] = (float *)malloc(bytes); + h_b[s] = (float *)malloc(bytes); + h_c[s] = (float *)malloc(bytes); + if (!h_a[s] || !h_b[s] || !h_c[s]) + return 1; + for (i = 0; i < n; i++) { + h_a[s][i] = (float)((i + s) % 1000) * 0.5f; + h_b[s][i] = (float)((i + 2 * s) % 997) * 0.25f; + } + CHECK(cudaMalloc(&d_a[s], bytes)); + CHECK(cudaMalloc(&d_b[s], bytes)); + CHECK(cudaMalloc(&d_c[s], bytes)); + CHECK(cudaStreamCreate(&st[s])); + CHECK(cudaEventCreate(&ev[s])); + } + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + printf("DPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" da%d=%p db%d=%p dc%d=%p", s, (void *)d_a[s], s, + (void *)d_b[s], s, (void *)d_c[s]); + printf("\n"); + printf("HPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" ha%d=%p hb%d=%p hc%d=%p", s, (void *)h_a[s], s, + (void *)h_b[s], s, (void *)h_c[s]); + printf("\n"); + + CHECK(cudaEventRecord(t0)); + + for (s = 0; s < NSTREAMS; s++) { + CHECK(cudaMemcpyAsync(d_a[s], h_a[s], bytes, + cudaMemcpyHostToDevice, st[s])); + CHECK(cudaMemcpyAsync(d_b[s], h_b[s], bytes, + cudaMemcpyHostToDevice, st[s])); + vadd<<<grid, 256, 0, st[s]>>>(d_a[s], d_b[s], d_c[s], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[s], st[s])); + } + + CHECK(cudaStreamWaitEvent(st[1], ev[0], 0)); + vadd_into<<<grid, 256, 0, st[1]>>>(d_c[1], d_c[0], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[1], st[1])); + + CHECK(cudaStreamWaitEvent(st[2], ev[1], 0)); + vadd_into<<<grid, 256, 0, st[2]>>>(d_c[2], d_c[1], n); + CHECK(cudaGetLastError()); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaMemcpyAsync(h_c[s], d_c[s], bytes, + cudaMemcpyDeviceToHost, st[s])); + CHECK(cudaEventRecord(t1, st[2])); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaStreamSynchronize(st[s])); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + for (s = 0; s < NSTREAMS; s++) { + int b = 0; + float exp0[8], got0[8]; + + for (i = 0; i < n; i++) { + float c0 = h_a[0][i] + h_b[0][i]; + float c1 = h_a[1][i] + h_b[1][i] + c0; + float c2 = h_a[2][i] + h_b[2][i] + c1; + float want = (s == 0) ? c0 : (s == 1) ? c1 : c2; + + if (i < 8) { + exp0[i] = want; + got0[i] = h_c[s][i]; + } + if (h_c[s][i] != want) + b++; + } + bad += b; + printf("BUF s=%d bad=%d\n", s, b); + printf("FIRST s=%d exp=", s); + for (i = 0; i < 8; i++) + printf("%.4f ", exp0[i]); + printf(" got="); + for (i = 0; i < 8; i++) + printf("%.4f ", got0[i]); + printf("\n"); + } + + printf("STREAMS3 n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/tgemm_probe.cu b/tools/cuda-ab/tgemm_probe.cu new file mode 100644 index 0000000..f0a1084 --- /dev/null +++ b/tools/cuda-ab/tgemm_probe.cu @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * tgemm_probe.cu - minimal isolation for the transformer K/V divergence. + * + * Reproduces exactly the transformer's Q/K/V GEMM shape (X[64x256] x + * W[256x256], tile 16) for three weight matrices uploaded back-to-back + * (wq, wk, wv), then verifies each against a CPU reference and prints + * per-GEMM bad counts plus a bad-position histogram (first bad index per + * 16-row band and per 16-col band) so a copy-side hole is distinguishable + * from a kernel-side error. + * + * Output: + * DEVICE ... + * GEMM name=<q|k|v> bad=<n> first=(row,col) got=.. ref=.. + * BADPOS name=.. rows=<r0,r1,..> cols=<c0,c1,..> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define SEQ 64 +#define DH 256 +#define TILE 16 + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static float xval(int i, int j) +{ + return (float)(((i * 7 + j * 3 + 1) % 9) - 4) * 0.25f; +} + +int main(void) +{ + cudaDeviceProp prop; + float *h_x, *h_w, *h_c, *h_ref; + float *d_x, *d_w[3], *d_c[3]; + const int seqdh = SEQ * DH, dh2 = DH * DH; + int g, i, j, k; + + setvbuf(stdout, NULL, _IONBF, 0); + if (cudaGetDeviceProperties(&prop, 0) != cudaSuccess) { + printf("STATUS FAIL\n"); + return 1; + } + printf("DEVICE %s cc=%d.%d\n", prop.name, prop.major, prop.minor); + + h_x = (float *)malloc(seqdh * 4); + h_w = (float *)malloc(dh2 * 4); + h_c = (float *)malloc(seqdh * 4); + h_ref = (float *)malloc(seqdh * 4); + + if (cudaMalloc(&d_x, seqdh * 4) != cudaSuccess) + return 1; + for (g = 0; g < 3; g++) { + if (cudaMalloc(&d_w[g], dh2 * 4) != cudaSuccess) + return 1; + if (cudaMalloc(&d_c[g], seqdh * 4) != cudaSuccess) + return 1; + } + + for (i = 0; i < seqdh; i++) + h_x[i] = xval(i / DH, i % DH); + + /* Upload X once (the transformer's shared input). */ + if (cudaMemcpy(d_x, h_x, seqdh * 4, cudaMemcpyHostToDevice) != + cudaSuccess) + return 1; + /* Verify the X upload itself (first row + full buffer). */ + { + float *h_xb = (float *)malloc(seqdh * 4); + int xbad = 0, xr0 = 0; + + if (!h_xb) + return 1; + if (cudaMemcpy(h_xb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (h_xb[i] != h_x[i]) { + xbad++; + if (i < DH) + xr0++; + } + printf("XVERIFY bad=%d row0_bad=%d\n", xbad, xr0); + free(h_xb); + } + + for (g = 0; g < 3; g++) { + int bad = 0, first_r = -1, first_c = -1; + float got = 0, ref = 0; + int rows[64] = {0}, cols[64] = {0}; + int nr = 0, nc = 0; + int pass; + + for (i = 0; i < dh2; i++) + h_w[i] = wval(i, g + 1); + if (cudaMemcpy(d_w[g], h_w, dh2 * 4, + cudaMemcpyHostToDevice) != cudaSuccess) + return 1; + + dim3 blk(TILE, TILE); + dim3 grd(DH / TILE, SEQ / TILE); + + for (pass = 0; pass < 3; pass++) { + float *hc = h_c, *hxb = h_ref; + int xbad = 0; + + gemm_nn<<<grd, blk>>>(d_x, d_w[g], d_c[g], SEQ, DH, DH); + if (cudaDeviceSynchronize() != cudaSuccess) { + printf("SYNC FAIL g=%d\n", g); + return 1; + } + if (cudaMemcpy(hc, d_c[g], seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + /* re-verify X after the launch */ + if (cudaMemcpy(hxb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (hxb[i] != h_x[i]) { + printf("XDIFF pass=%d i=%d got=%.6f want=%.6f\n", + pass, i, hxb[i], h_x[i]); + xbad++; + } + bad = 0; + nr = 0; + nc = 0; + for (i = 0; i < SEQ; i++) + for (j = 0; j < DH; j++) { + double acc = 0.0; + float r, a, diff; + + for (k = 0; k < DH; k++) + acc += (double)h_x[i * DH + k] * + (double)h_w[k * DH + j]; + r = (float)acc; + a = hc[i * DH + j]; + diff = a - r; + if (diff < 0) + diff = -diff; + if (diff > 1e-4f) { + int rr = i, cc = j; + + if (bad == 0) { + first_r = rr; + first_c = cc; + got = a; + ref = r; + } + bad++; + if (nr < 64 && (bad == 1 || + rr != rows[nr - 1])) + rows[nr++] = rr; + if (nc < 64 && (bad == 1 || + cc != cols[nc - 1])) + cols[nc++] = cc; + } + } + printf("PASS name=%c pass=%d bad=%d xbad=%d first=(%d,%d) got=%.6f ref=%.6f\n", + "qkv"[g], pass, bad, xbad, first_r, first_c, + got, ref); + if (bad) { + printf("ROW0 name=%c got=", "qkv"[g]); + for (i = 0; i < 8; i++) + printf("%.6f ", hc[i]); + printf(" ref="); + for (i = 0; i < 8; i++) { + double acc = 0.0; + + for (k = 0; k < DH; k++) + acc += (double)h_x[k] * + (double)h_w[k * DH + i]; + printf("%.6f ", (float)acc); + } + printf("\n"); + } + } + (void)got; + (void)ref; + (void)first_r; + (void)first_c; + (void)nr; + (void)nc; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/transformer.cu b/tools/cuda-ab/transformer.cu new file mode 100644 index 0000000..3044522 --- /dev/null +++ b/tools/cuda-ab/transformer.cu @@ -0,0 +1,472 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward ---- */ + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/transformer_async.cu b/tools/cuda-ab/transformer_async.cu new file mode 100644 index 0000000..213363a --- /dev/null +++ b/tools/cuda-ab/transformer_async.cu @@ -0,0 +1,522 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward (async chain, stage readbacks) ---- */ + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + { + float *hs = (float *)malloc(b_seqdh); + float *hs2 = (float *)malloc(b_seq2); + double *rw = (double *)malloc((size_t)dh2 * sizeof(double)); + double *rq = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rk = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rv = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rs = (double *)malloc((size_t)seq2 * sizeof(double)); + double *ra = (double *)malloc((size_t)seq2 * sizeof(double)); + double *rav = (double *)malloc((size_t)seqdh * sizeof(double)); + double *ry = (double *)malloc((size_t)seqdh * sizeof(double)); + int st; + + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 1); + ref_gemm(r_x, rw, rq, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 2); + ref_gemm(r_x, rw, rk, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 3); + ref_gemm(r_x, rw, rv, SEQ, DH, DH); + ref_gemm_nt(rq, rk, rs, SEQ, SEQ, DH); + ref_softmax(rs, ra, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(ra, rv, rav, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 4); + ref_gemm(rav, rw, ry, SEQ, DH, DH); + + CHECK(cudaMemcpy(hs, d_q, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rq, seqdh); + printf("ASYNC Q bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_k, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rk, seqdh); + printf("ASYNC K bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_v, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rv, seqdh); + printf("ASYNC V bad=%d\n", st); + CHECK(cudaMemcpy(hs2, d_s, b_seq2, cudaMemcpyDeviceToHost)); + st = count_bad(hs2, rs, seq2); + printf("ASYNC S bad=%d\n", st); + CHECK(cudaMemcpy(hs2, d_a, b_seq2, cudaMemcpyDeviceToHost)); + st = count_bad(hs2, ra, seq2); + printf("ASYNC A bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_av, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rav, seqdh); + printf("ASYNC AV bad=%d\n", st); + st = count_bad(h_y, ry, seqdh); + printf("ASYNC Y bad=%d\n", st); + } + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/transformer_stages.cu b/tools/cuda-ab/transformer_stages.cu new file mode 100644 index 0000000..d7f67c0 --- /dev/null +++ b/tools/cuda-ab/transformer_stages.cu @@ -0,0 +1,553 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +static void dump_bad(const char *tag, const float *gpu, const double *ref, + int n) +{ + int i, shown = 0; + + for (i = 0; i < n && shown < 8; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) { + printf("BAD %s i=%d gpu=%.6f ref=%.6f (tile %d,%d row %d col %d)\n", + tag, i, g, r, i / 16, (i % 256) / 16, + i / 256, i % 256); + shown++; + } + } +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward, stage-by-stage verify ---- */ + { + float *h_stage = (float *)malloc(b_seqdh); + float *h_s2 = (float *)malloc(b_seq2); + double *r_stage = (double *)malloc((size_t)seqdh * sizeof(double)); + double *r_s2 = (double *)malloc((size_t)seq2 * sizeof(double)); + double *r_w2 = (double *)malloc((size_t)dh2 * sizeof(double)); + int st; + + if (!h_stage || !h_s2 || !r_stage || !r_s2 || !r_w2) + return 1; + + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_q, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE Q bad=%d\n", st); + dump_bad("Q", h_stage, r_stage, seqdh); + + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_k, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE K bad=%d\n", st); + dump_bad("K", h_stage, r_stage, seqdh); + + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_v, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE V bad=%d\n", st); + dump_bad("V", h_stage, r_stage, seqdh); + + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_s2, d_s, b_seq2, cudaMemcpyDeviceToHost)); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + st = count_bad(h_s2, r_s, seq2); + printf("STAGE S bad=%d\n", st); + dump_bad("S", h_s2, r_s, seq2); + + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_s2, d_a, b_seq2, cudaMemcpyDeviceToHost)); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + st = count_bad(h_s2, r_a, seq2); + printf("STAGE A bad=%d\n", st); + dump_bad("A", h_s2, r_a, seq2); + + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_av, b_seqdh, cudaMemcpyDeviceToHost)); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + st = count_bad(h_stage, r_av, seqdh); + printf("STAGE AV bad=%d\n", st); + + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w2, r_y, SEQ, DH, DH); + st = count_bad(h_y, r_y, seqdh); + printf("STAGE Y bad=%d\n", st); + } + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/wedge_ab.cu b/tools/cuda-ab/wedge_ab.cu new file mode 100644 index 0000000..2d88f92 --- /dev/null +++ b/tools/cuda-ab/wedge_ab.cu @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_ab.cu - discriminate the wedge trigger: is it the second launch of + * the SAME program, or the second launch of the context regardless of + * program? + * + * Sequence (each launch verified against mutated inputs): + * 1. vadd (program A) + * 2. vsub (program B, different code) + * 3. vadd (program A again) + * 4. vsub (program B again) + * + * If launch 2 (B) executes -> the trigger is same-program reuse. + * If launch 2 (B) wedges -> the trigger is the second launch in the context. + * + * Build: nvcc -O2 --cudart=static -o wedge_ab wedge_ab.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vsub(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] - b[i]; +} + +int main(void) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc; + float *da, *db, *dc; + int i, bad, step; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + if (!ha || !hb || !hc) + return 1; + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + + for (step = 0; step < 4; step++) { + const char *which = (step & 1) ? "vsub" : "vadd"; + int is_sub = step & 1; + + if (step > 0) { + ha[step] += 1.0f; + hb[step * 7 + 3] += 2.0f; + } + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + if (is_sub) + vsub<<<(n + 255) / 256, 256>>>(da, db, dc, n); + else + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + bad = 0; + for (i = 0; i < n; i++) { + float expect = is_sub ? ha[i] - hb[i] : ha[i] + hb[i]; + + if (hc[i] != expect) + bad++; + } + printf("STEP %d %s bad=%d/%d\n", step, which, bad, n); + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/wedge_bisect.cu b/tools/cuda-ab/wedge_bisect.cu new file mode 100644 index 0000000..e062a2b --- /dev/null +++ b/tools/cuda-ab/wedge_bisect.cu @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_bisect.cu - bisect the wedge trigger: which inter-launch activity + * makes the SECOND vadd launch no-execute? + * + * Every iteration is: mutate host inputs -> H2D da/db -> launch -> sync -> + * D2H dc -> verify (the wedge_recover shape). Modes add activity between + * the verify and the next launch: + * + * mode A: none (pure second-launch class) + * mode B: an extra H2D of the same buffers (CE write before the launch) + * mode C: an extra D2H readback (CE read before the launch) + * mode D: a 500 ms sleep (no CE traffic) + * + * A no-execute launch is reported (bad>0) not masked: the inputs mutate every + * iteration. + * + * Build: nvcc -O2 --cudart=static -o wedge_bisect wedge_bisect.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> +#include <unistd.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +int main(int argc, char **argv) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc; + float *da, *db, *dc; + const char *mode = argc > 1 ? argv[1] : "A"; + int iters = argc > 2 ? atoi(argv[2]) : 4; + int i, bad = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + if (!ha || !hb || !hc) + return 1; + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + for (i = 0; i < iters; i++) { + int j, itbad = 0; + + if (i > 0) { + ha[i % n] += 1.0f; + hb[(i * 7 + 3) % n] += 2.0f; + } + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + for (j = 0; j < n; j++) + if (hc[j] != ha[j] + hb[j]) + itbad++; + printf("MODE %s iter=%d bad=%d/%d\n", mode, i, itbad, n); + bad += itbad; + if (i + 1 < iters) { + if (mode[0] == 'B') { + CHECK(cudaMemcpy(da, ha, bytes, + cudaMemcpyHostToDevice)); + } else if (mode[0] == 'C') { + CHECK(cudaMemcpy(hc, dc, bytes, + cudaMemcpyDeviceToHost)); + } else if (mode[0] == 'D') { + usleep(500000); + } + } + } + printf("STATUS %s\n", bad ? "FAIL" : "OK"); + return bad ? 1 : 0; +} diff --git a/tools/cuda-ab/wedge_recover.cu b/tools/cuda-ab/wedge_recover.cu new file mode 100644 index 0000000..fd8f3f9 --- /dev/null +++ b/tools/cuda-ab/wedge_recover.cu @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_recover.cu - reproduce the per-launch no-execute (wedge) and test + * whether an idle gap heals a wedged context. + * + * v2 (session 10, r416 finding): the v1 loop reused one deterministic + * kernel+buffers, so a no-write launch was MASKED - the stale buffer held + * exactly the previous iteration's (identical) values and the verify passed. + * r416 ran 60/60 "OK" while 58 consecutive launches no-wrote underneath. + * + * v2 mutates the inputs every iteration (H2D of fresh ha/hb), so a stale + * readback cannot match the new expected values: a no-write launch must + * report BAD. On BAD the same iteration is retried after a sleep, and the + * readback is classified: STALE-PREV (byte-identical to the previous + * iteration's output = launch never executed) vs PARTIAL/OTHER (engine wrote + * something). Latency and per-iteration outcome are printed. + * + * Build: nvcc -O2 --cudart=static -o wedge_recover wedge_recover.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> +#include <unistd.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +int main(int argc, char **argv) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hc_prev; + float *da, *db, *dc; + int iters = argc > 1 ? atoi(argv[1]) : 40; + int sleep_s = argc > 2 ? atoi(argv[2]) : 20; + int fails = 0, heals = 0, i; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hc_prev = (float *)malloc(bytes); + if (!ha || !hb || !hc || !hc_prev) + return 1; + /* iteration 0 baseline; mutated each iteration below (v2) */ + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + memset(hc_prev, 0, bytes); + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + for (i = 0; i < iters; i++) { + int attempt = 0; + long long ms[2] = { 0, 0 }; + + /* + * v2: fresh inputs every iteration so a stale (never + * executed) launch cannot match. Mutate after iteration 0. + */ + if (i > 0) { + ha[i % n] += 1.0f; + hb[(i * 7 + 3) % n] += 2.0f; + CHECK(cudaMemcpy(da, ha, bytes, + cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, + cudaMemcpyHostToDevice)); + } + + for (;;) { + int bad = 0, stale_prev = 1, j; + cudaEvent_t e0, e1; + float ems = 0; + + cudaEventCreate(&e0); + cudaEventCreate(&e1); + cudaEventRecord(e0); + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + cudaEventRecord(e1); + CHECK(cudaDeviceSynchronize()); + cudaEventElapsedTime(&ems, e0, e1); + cudaEventDestroy(e0); + cudaEventDestroy(e1); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + for (j = 0; j < n; j++) { + float expect = ha[j] + hb[j]; + + if (hc[j] != expect) { + bad++; + if (hc[j] != hc_prev[j]) + stale_prev = 0; + } + } + ms[attempt & 1] = (long long)(ems * 1000.0f); + if (bad == 0) { + if (attempt > 0) + heals++; + printf("ITER %d OK attempt=%d launch_us=%lld\n", + i, attempt, ms[attempt & 1]); + memcpy(hc_prev, hc, bytes); + break; + } + fails++; + printf("ITER %d BAD bad=%d/%d attempt=%d launch_us=%lld class=%s\n", + i, bad, n, attempt, ms[attempt & 1], + stale_prev ? "STALE-PREV(NOWRITE)" + : "OTHER/PARTIAL"); + if (attempt >= 3) { + printf("ITER %d GIVEUP\n", i); + break; + } + attempt++; + printf("ITER %d SLEEP %ds before retry\n", i, sleep_s); + sleep(sleep_s); + } + } + printf("SUMMARY iters=%d fails=%d heals=%d sleep_s=%d\n", + iters, fails, heals, sleep_s); + printf("STATUS %s\n", fails == 0 ? "OK" : "FAIL"); + return fails == 0 ? 0 : 1; +} diff --git a/tools/guest-diag/egl_frame.c b/tools/guest-diag/egl_frame.c new file mode 100644 index 0000000..ed10e39 --- /dev/null +++ b/tools/guest-diag/egl_frame.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * egl_frame.c - minimal EGL/GLES2 offscreen render test for the vGPU guest. + * + * Tries, in order: the EGL_EXT_platform_device platform with each enumerated + * EGLDeviceEXT, EGL_MESA_platform_surfaceless, then EGL_DEFAULT_DISPLAY. + * For the first display that initializes it creates an offscreen pbuffer + * context, renders colour bands into an FBO, reads them back and writes a + * PPM. One greppable line per stage. + * + * Build (guest): gcc -O2 -o egl_frame egl_frame.c -lEGL -lGLESv2 + */ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <EGL/egl.h> +#include <EGL/eglext.h> +#include <GLES2/gl2.h> + +#define W 320 +#define H 240 + +static EGLDisplay try_display(const char *name, EGLDisplay dpy, + const char *dev_name) +{ + EGLint major = 0, minor = 0; + + if (dpy == EGL_NO_DISPLAY) { + printf("EGL_PLATFORM %s -> no display\n", name); + return EGL_NO_DISPLAY; + } + if (!eglInitialize(dpy, &major, &minor)) { + printf("EGL_PLATFORM %s -> init fail 0x%x\n", name, + eglGetError()); + return EGL_NO_DISPLAY; + } + printf("EGL_PLATFORM %s ok %d.%d vendor=%s%s%s\n", name, major, minor, + eglQueryString(dpy, EGL_VENDOR), + dev_name ? " device=" : "", dev_name ? dev_name : ""); + return dpy; +} + +static EGLDisplay get_display(void) +{ + PFNEGLGETPLATFORMDISPLAYEXTPROC gp = + (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress( + "eglGetPlatformDisplayEXT"); + PFNEGLQUERYDEVICESEXTPROC qd = + (PFNEGLQUERYDEVICESEXTPROC)eglGetProcAddress( + "eglQueryDevicesEXT"); + EGLDisplay dpy; + + if (gp && qd) { + EGLDeviceEXT devs[8]; + EGLint n = 0; + int i; + + if (qd(8, devs, &n) && n > 0) { + printf("EGL_DEVICES n=%d\n", n); + for (i = 0; i < n; i++) { + EGLDisplay d = gp(EGL_PLATFORM_DEVICE_EXT, + devs[i], NULL); + + dpy = try_display("device", d, "enum"); + if (dpy != EGL_NO_DISPLAY) + return dpy; + } + } else { + printf("EGL_DEVICES n=0\n"); + } + if (gp) { + dpy = try_display("surfaceless", + gp(EGL_PLATFORM_SURFACELESS_MESA, + NULL, NULL), NULL); + if (dpy != EGL_NO_DISPLAY) + return dpy; + } + } + return try_display("default", eglGetDisplay(EGL_DEFAULT_DISPLAY), + NULL); +} + +static int render(EGLDisplay dpy) +{ + EGLConfig cfg; + EGLint ncfg = 0; + EGLSurface surf; + EGLContext ctx; + GLuint fbo = 0, tex = 0; + unsigned char *px; + FILE *f; + int i; + EGLint attrs[] = { + EGL_SURFACE_TYPE, EGL_PBUFFER_BIT, + EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT, + EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, + EGL_BLUE_SIZE, 8, EGL_ALPHA_SIZE, 8, + EGL_NONE + }; + EGLint pattrs[] = { EGL_WIDTH, W, EGL_HEIGHT, H, EGL_NONE }; + EGLint cattrs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; + + if (!eglChooseConfig(dpy, attrs, &cfg, 1, &ncfg) || ncfg < 1) { + printf("EGL_CONFIG_FAIL err=0x%x\n", eglGetError()); + return 1; + } + surf = eglCreatePbufferSurface(dpy, cfg, pattrs); + if (surf == EGL_NO_SURFACE) { + printf("EGL_PBUFFER_FAIL err=0x%x\n", eglGetError()); + return 1; + } + ctx = eglCreateContext(dpy, cfg, EGL_NO_CONTEXT, cattrs); + if (ctx == EGL_NO_CONTEXT) { + printf("EGL_CONTEXT_FAIL err=0x%x\n", eglGetError()); + return 1; + } + if (!eglMakeCurrent(dpy, surf, surf, ctx)) { + printf("EGL_CURRENT_FAIL err=0x%x\n", eglGetError()); + return 1; + } + printf("GL_RENDERER %s\n", glGetString(GL_RENDERER)); + printf("GL_VERSION %s\n", glGetString(GL_VERSION)); + + glGenTextures(1, &tex); + glBindTexture(GL_TEXTURE_2D, tex); + glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, W, H, 0, GL_RGBA, + GL_UNSIGNED_BYTE, NULL); + glGenFramebuffers(1, &fbo); + glBindFramebuffer(GL_FRAMEBUFFER, fbo); + glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, + GL_TEXTURE_2D, tex, 0); + if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != + GL_FRAMEBUFFER_COMPLETE) { + printf("GL_FBO_FAIL\n"); + return 1; + } + for (i = 0; i < 16; i++) { + glEnable(GL_SCISSOR_TEST); + glScissor(0, i * H / 16, W, H / 16); + glClearColor((float)i / 16.0f, 1.0f - (float)i / 16.0f, + 0.5f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT); + } + glFinish(); + printf("GL_ERR 0x%x\n", glGetError()); + px = (unsigned char *)malloc(W * H * 4); + glReadPixels(0, 0, W, H, GL_RGBA, GL_UNSIGNED_BYTE, px); + { + const char *out = getenv("EGL_FRAME_OUT"); + + if (!out) + out = "/tmp/egl_frame.ppm"; + f = fopen(out, "wb"); + if (!f) { + printf("PPM_OPEN_FAIL\n"); + return 1; + } + fprintf(f, "P6\n%d %d\n255\n", W, H); + for (i = 0; i < W * H; i++) + fwrite(px + i * 4, 1, 3, f); + fclose(f); + printf("EGL_FRAME_WRITTEN %s\n", out); + } + printf("EGL_FRAME_OK\n"); + return 0; +} + +int main(void) +{ + EGLDisplay dpy; + const char *ext; + + setvbuf(stdout, NULL, _IONBF, 0); + ext = eglQueryString(EGL_NO_DISPLAY, EGL_EXTENSIONS); + printf("EGL_CLIENT_EXT %s\n", ext ? ext : "(none)"); + dpy = get_display(); + if (dpy == EGL_NO_DISPLAY) { + printf("EGL_ALL_PLATFORMS_FAILED\n"); + return 1; + } + return render(dpy); +} diff --git a/tools/guest-diag/egl_retry.c b/tools/guest-diag/egl_retry.c new file mode 100644 index 0000000..ce45d45 --- /dev/null +++ b/tools/guest-diag/egl_retry.c @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * egl_retry.c - EGL/GLES2 offscreen probe with an explicit eglMakeCurrent + * retry loop. The r581-era trace showed the glsi engine-init returning + * 0xee00001 ("busy") exactly once and setting the device state bit 0x20; + * eglcore does not retry. This probe retries the make-current (and the whole + * display/context creation) up to N times, printing the precise EGL error of + * every attempt, so we can tell "one-shot busy, retry succeeds" from + * "permanent failure". + * + * Build (guest): gcc -O2 -o egl_retry egl_retry.c -lEGL -lGLESv2 + */ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <EGL/egl.h> +#include <EGL/eglext.h> +#include <GLES2/gl2.h> + +static EGLDisplay get_device_display(void) +{ + PFNEGLGETPLATFORMDISPLAYEXTPROC gp = + (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress( + "eglGetPlatformDisplayEXT"); + PFNEGLQUERYDEVICESEXTPROC qd = + (PFNEGLQUERYDEVICESEXTPROC)eglGetProcAddress( + "eglQueryDevicesEXT"); + EGLDeviceEXT devs[8]; + EGLint n = 0; + + if (!gp || !qd || !qd(8, devs, &n) || n <= 0) + return EGL_NO_DISPLAY; + printf("EGL_DEVICES n=%d\n", n); + return gp(EGL_PLATFORM_DEVICE_EXT, devs[0], NULL); +} + +int main(int argc, char **argv) +{ + int tries = argc > 1 ? atoi(argv[1]) : 10; + EGLDisplay dpy; + EGLint major = 0, minor = 0; + EGLConfig cfg; + EGLint ncfg = 0; + EGLSurface surf; + EGLContext ctx = EGL_NO_CONTEXT; + EGLint attrs[] = { + EGL_SURFACE_TYPE, EGL_PBUFFER_BIT, + EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT, + EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, EGL_BLUE_SIZE, 8, + EGL_ALPHA_SIZE, 8, EGL_NONE + }; + EGLint pb[] = { EGL_WIDTH, 64, EGL_HEIGHT, 64, EGL_NONE }; + EGLint cattr[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + + dpy = get_device_display(); + if (dpy == EGL_NO_DISPLAY) { + printf("NO_DEVICE_DISPLAY\n"); + return 1; + } + if (!eglInitialize(dpy, &major, &minor)) { + printf("EGL_INIT_FAIL 0x%x\n", eglGetError()); + return 1; + } + printf("EGL_INIT ok %d.%d vendor=%s\n", major, minor, + eglQueryString(dpy, EGL_VENDOR)); + if (!eglBindAPI(EGL_OPENGL_ES_API)) { + printf("BIND_API_FAIL 0x%x\n", eglGetError()); + return 1; + } + if (!eglChooseConfig(dpy, attrs, &cfg, 1, &ncfg) || ncfg < 1) { + printf("CHOOSE_CONFIG_FAIL 0x%x n=%d\n", eglGetError(), ncfg); + return 1; + } + for (i = 0; i < tries; i++) { + surf = eglCreatePbufferSurface(dpy, cfg, pb); + if (surf == EGL_NO_SURFACE) { + printf("TRY %d CREATE_SURFACE_FAIL 0x%x\n", i + 1, + eglGetError()); + usleep(200000); + continue; + } + if (ctx == EGL_NO_CONTEXT) { + ctx = eglCreateContext(dpy, cfg, EGL_NO_CONTEXT, + cattr); + if (ctx == EGL_NO_CONTEXT) { + printf("TRY %d CREATE_CTX_FAIL 0x%x\n", i + 1, + eglGetError()); + eglDestroySurface(dpy, surf); + usleep(200000); + continue; + } + } + if (eglMakeCurrent(dpy, surf, surf, ctx)) { + const char *rend = (const char *)glGetString( + GL_RENDERER); + const char *ver = (const char *)glGetString( + GL_VERSION); + printf("TRY %d CURRENT OK renderer=%s version=%s\n", + i + 1, rend ? rend : "?", ver ? ver : "?"); + /* draw one clear + readback so the pipeline runs */ + glClearColor(0.1f, 0.4f, 0.9f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT); + { + unsigned char px[4]; + glReadPixels(0, 0, 1, 1, GL_RGBA, + GL_UNSIGNED_BYTE, px); + printf("PIXEL %02x%02x%02x%02x\n", px[0], + px[1], px[2], px[3]); + } + printf("EGL_RETRY_OK on try %d\n", i + 1); + return 0; + } + printf("TRY %d MAKE_CURRENT_FAIL 0x%x\n", i + 1, + eglGetError()); + eglDestroySurface(dpy, surf); + usleep(200000); + } + printf("EGL_RETRY_EXHAUSTED tries=%d\n", tries); + return 1; +}
\ No newline at end of file diff --git a/tools/guest-diag/fbcopy_pub.c b/tools/guest-diag/fbcopy_pub.c new file mode 100644 index 0000000..a064882 --- /dev/null +++ b/tools/guest-diag/fbcopy_pub.c @@ -0,0 +1,718 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * fbcopy_pub.c - capture the vGPU framebuffer WITHOUT X and publish frames + * into the kvmfr ivshmem BAR. + * + * The X server's screen has 1280x720 pixels in the emulated VRAM, which the + * guest maps as the vGPU's BAR1 (resource1). The X SHM readback path can + * stall behind the X server's own push wait; this tool reads the framebuffer + * pixels straight out of the BAR1 mapping instead (host RAM backing, no GPU + * or X round-trip), swizzles BGRA -> RGBA and writes the kvmfr frame + * protocol the host reader already decodes. + * + * Usage (guest, as root): + * ./fbcopy_pub test one solid red frame (pipeline check) + * ./fbcopy_pub find [passes] [ms] motion-scan first 64 MB of BAR1 + * ./fbcopy_pub findp [off] [sz] [p] [ms] + * pread motion-scan of any BAR1 range + * (mmap only reaches the first 64 MB) + * ./fbcopy_pub pub <off> [n] [fps] [w] [h] [pitch] + * copy w*h from BAR1@off (pitch), + * mmap path (off < 64 MB) + * ./fbcopy_pub pubp <off> [n] [fps] [w] [h] [pitch] + * same but pread path (any offset) + * + * Find/build on the host (same glibc as the guest): gcc -O2 -o fbcopy_pub + * fbcopy_pub.c + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <dirent.h> +#include <errno.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <time.h> + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" */ +#define DEF_W 1280 +#define DEF_H 720 +#define DEF_PITCH 5120 /* xorg-vgpu.conf: 1280*4 rounded */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +static const char *find_vgpu_resource1(void) +{ + static char path[320]; + DIR *d = opendir("/sys/bus/pci/devices"); + struct dirent *de; + + if (!d) + return NULL; + while ((de = readdir(d))) { + char p[340], buf[32]; + FILE *f; + int n; + + if (de->d_name[0] == '.') + continue; + snprintf(p, sizeof(p), "/sys/bus/pci/devices/%s/vendor", + de->d_name); + f = fopen(p, "r"); + if (!f) + continue; + n = fread(buf, 1, sizeof(buf) - 1, f); + fclose(f); + if (n <= 0) + continue; + buf[n] = 0; + if (strtoul(buf, NULL, 16) != 0x10de) + continue; + snprintf(path, sizeof(path), + "/sys/bus/pci/devices/%s/resource1", de->d_name); + closedir(d); + return path; + } + closedir(d); + return NULL; +} + +/* + * The vfio mdev region mmap is limited: only file offset 0 with len <= 64 MB + * works (nonzero offsets and 256 MB both EINVAL). Map the first 64 MB once at + * offset 0 and index into the mapping for every access. + */ +#define CHUNK_MAX (64UL << 20) + +static unsigned char *map_vram(int fd, size_t want, size_t *got) +{ + size_t len = want > CHUNK_MAX ? CHUNK_MAX : want; + unsigned char *p; + + p = mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0); + if (p == MAP_FAILED) + return NULL; + *got = len; + return p; +} + +static double t_ms(const struct timespec *a, const struct timespec *b) +{ + return (double)(b->tv_sec - a->tv_sec) * 1e3 + + (double)(b->tv_nsec - a->tv_nsec) / 1e6; +} + +/* + * Find the largest contiguous run of "dense" pages (>= 64 of 256 sampled + * bytes nonzero) - an image in VRAM looks like one long dense run. Small + * gaps (<= 16 pages) are tolerated. + */ +static void report_cluster(const uint32_t *nz, size_t pages, + unsigned long long base_off, const char *tag) +{ + size_t p, run_start = 0, last_dense = 0, best_start = 0, best_len = 0; + int in_run = 0; + + for (p = 0; p <= pages; p++) { + int dense = p < pages && nz[p] >= 64; + + if (dense) { + if (!in_run) { + run_start = p; + in_run = 1; + } + last_dense = p; + } else if (in_run && p - last_dense > 16) { + size_t len = last_dense - run_start + 1; + + if (len > best_len) { + best_len = len; + best_start = run_start; + } + in_run = 0; + } + } + if (best_len >= 8) + printf("%s_CLUSTER off=0x%llx pages=%zu bytes=%zu\n", tag, + base_off + ((unsigned long long)best_start << 12), + best_len, best_len << 12); +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "find"; + const char *bar_path = getenv("FBCOPY_BAR") ? getenv("FBCOPY_BAR") : + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + const char *res1 = getenv("FBCOPY_RES1") ? getenv("FBCOPY_RES1") : + find_vgpu_resource1(); + struct stat st; + size_t map_size, i; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + int fd, vram_fd; + + setvbuf(stdout, NULL, _IONBF, 0); + if (!res1) { + printf("NO_VGPU\n"); + return 1; + } + printf("VGPU_RES1 %s\n", res1); + if (stat(res1, &st) || !st.st_size) { + printf("STAT_FAIL\n"); + return 1; + } + map_size = (size_t)st.st_size; + printf("VRAM_SIZE 0x%zx (%zu MB)\n", map_size, map_size >> 20); + vram_fd = open(res1, O_RDONLY | O_SYNC); + if (vram_fd < 0) { + perror(res1); + return 1; + } + + /* kvmfr BAR next: every mode needs it except find. */ + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + perror(bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + close(fd); + if (bar == MAP_FAILED) { + perror("mmap kvmfr"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + if (!strcmp(mode, "zero")) { + /* Reset the kvmfr header seq so a reader seeding from it will + * accept the next pub run's frame 1. */ + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = DEF_W; + hdr->height = DEF_H; + hdr->stride = DEF_W * 4; + hdr->format = 0; + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = 0; + printf("SEQ_ZEROED\n"); + return 0; + } + + if (!strcmp(mode, "test")) { + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + size_t n = (size_t)DEF_W * DEF_H * 4; + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = DEF_W; + hdr->height = DEF_H; + hdr->stride = DEF_W * 4; + hdr->format = 0; + for (i = 0; i < n; i += 4) { + dst[i + 0] = (unsigned char)(argc > 2 ? atoi(argv[2]) + : 255); + dst[i + 1] = (unsigned char)(argc > 3 ? atoi(argv[3]) + : 0); + dst[i + 2] = (unsigned char)(argc > 4 ? atoi(argv[4]) + : 0); + dst[i + 3] = 0xff; + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = 1; + printf("TEST_FRAME_PUBLISHED\n"); + return 0; + } + + if (!strcmp(mode, "find")) { + /* + * Motion + content scan of the first 64 MB of BAR1 (the only + * window the vfio mdev mmap accepts). Sample 256 B of every + * 4 KB page over `passes` passes `interval_ms` apart; score + * pages by total differing bytes (motion) and by nonzero + * density (content). Rendering can be slower than one frame + * per second, so the interval defaults to 3 s and is + * configurable: find [passes] [interval_ms] + */ + int passes = argc > 2 ? atoi(argv[2]) : 3; + int interval_ms = argc > 3 ? atoi(argv[3]) : 3000; + size_t scan_size = map_size > CHUNK_MAX ? CHUNK_MAX : map_size; + size_t pages = scan_size >> 12; + unsigned char *s0, *s1, *tmp, *w; + uint32_t *motion, *nz; + size_t p, wlen; + int top, pass; + + if (passes < 2) + passes = 2; + w = map_vram(vram_fd, scan_size, &wlen); + if (!w) { + printf("CHUNK_MAP_FAIL\n"); + return 1; + } + printf("SCAN_WINDOW 0x%zx pages=%zu passes=%d interval=%dms\n", + wlen, pages, passes, interval_ms); + s0 = malloc(pages * 256); + s1 = malloc(pages * 256); + motion = calloc(pages, sizeof(uint32_t)); + nz = calloc(pages, sizeof(uint32_t)); + if (!s0 || !s1 || !motion || !nz) + return 1; + for (p = 0; p < pages; p++) + memcpy(s0 + p * 256, w + (p << 12), 256); + for (pass = 1; pass < passes; pass++) { + struct timespec a, b; + uint64_t moved = 0; + + usleep((useconds_t)interval_ms * 1000); + clock_gettime(CLOCK_MONOTONIC, &a); + for (p = 0; p < pages; p++) + memcpy(s1 + p * 256, w + (p << 12), 256); + clock_gettime(CLOCK_MONOTONIC, &b); + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + unsigned char *y = s1 + p * 256; + uint32_t d = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j] != y[j]) + d++; + motion[p] += d; + moved += d; + } + printf("PASS %d moved_bytes=%llu dt=%.0fms\n", + pass, (unsigned long long)moved, t_ms(&a, &b)); + tmp = s0; s0 = s1; s1 = tmp; + } + /* nonzero density of the last sample */ + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + uint32_t c = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j]) + c++; + nz[p] = c; + } + report_cluster(nz, pages, 0, "FIND"); + /* per-16MB summary */ + for (i = 0; i + 4096 <= pages; i += 4096) { + uint64_t s = 0, c = 0; + size_t j; + + for (j = i; j < i + 4096; j++) { + s += motion[j]; + c += nz[j]; + } + if (s || c) + printf("BLOCK off=0x%zx moving_bytes=%llu nonzero_bytes=%llu\n", + i << 12, (unsigned long long)s, + (unsigned long long)c); + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (motion[p] > bs) { + bs = motion[p]; + best = p; + } + if (!bs) + break; + printf("HOT off=0x%zx score=%u\n", best << 12, bs); + motion[best] = 0; + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (nz[p] > bs) { + bs = nz[p]; + best = p; + } + if (!bs) + break; + printf("CONTENT off=0x%zx nonzero=%u\n", + best << 12, bs); + nz[best] = 0; + } + printf("FIND_DONE\n"); + return 0; + } + + if (!strcmp(mode, "probe")) { + /* + * Ground truth on what BAR1 access actually works: try mmap + * windows at several offsets/sizes and pread at several + * offsets, report each result. Read-only. + */ + struct { + unsigned long long off, len; + } mm[] = { + { 0, 64UL << 20 }, + { 0, 256UL << 20 }, + { 0, 4UL << 20 }, + { 4UL << 20, 4UL << 20 }, + { 64UL << 20, 64UL << 20 }, + { 128UL << 20, 4UL << 20 }, + { 192UL << 20, 4UL << 20 }, + { 60UL << 20, 8UL << 20 }, + }; + unsigned long long po[] = { + 0, 0x100000, 4UL << 20, 60UL << 20, + 64UL << 20, 128UL << 20, 252UL << 20, + }; + unsigned char buf[64]; + size_t k; + + for (k = 0; k < sizeof(mm) / sizeof(mm[0]); k++) { + unsigned char *p; + + if (mm[k].off + mm[k].len > map_size) { + printf("MMAP off=0x%llx len=0x%llx SKIP(>bar)\n", + mm[k].off, mm[k].len); + continue; + } + errno = 0; + p = mmap(NULL, (size_t)mm[k].len, PROT_READ, + MAP_SHARED, vram_fd, (off_t)mm[k].off); + if (p == MAP_FAILED) + printf("MMAP off=0x%llx len=0x%llx FAIL errno=%d\n", + mm[k].off, mm[k].len, errno); + else { + printf("MMAP off=0x%llx len=0x%llx OK first=%02x%02x%02x%02x\n", + mm[k].off, mm[k].len, p[0], p[1], p[2], p[3]); + munmap(p, (size_t)mm[k].len); + } + } + for (k = 0; k < sizeof(po) / sizeof(po[0]); k++) { + ssize_t r; + + errno = 0; + r = pread(vram_fd, buf, 16, (off_t)po[k]); + if (r != 16) + printf("PREAD off=0x%llx FAIL r=%zd errno=%d\n", + po[k], r, errno); + else + printf("PREAD off=0x%llx OK %02x%02x%02x%02x\n", + po[k], buf[0], buf[1], buf[2], buf[3]); + } + printf("PROBE_DONE\n"); + return 0; + } + + if (!strcmp(mode, "findp")) { + /* + * Same motion + content scan as `find`, but via pread so the + * whole 256 MB BAR is reachable (mmap stops at 64 MB). + * findp [start_off] [size] [passes] [interval_ms] + */ + unsigned long long start = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + unsigned long long size = argc > 3 ? + strtoull(argv[3], NULL, 0) : map_size - start; + int passes = argc > 4 ? atoi(argv[4]) : 3; + int interval_ms = argc > 5 ? atoi(argv[5]) : 3000; + size_t pages = size >> 12; + unsigned char *s0, *s1, *tmp; + uint32_t *motion, *nz; + size_t p; + int top, pass; + + if (passes < 2) + passes = 2; + if (start + size > map_size) { + printf("RANGE_FAIL start=0x%llx size=0x%llx bar=0x%zx\n", + start, size, map_size); + return 1; + } + printf("SCANP start=0x%llx size=0x%llx pages=%zu passes=%d interval=%dms\n", + start, size, pages, passes, interval_ms); + s0 = malloc(pages * 256); + s1 = malloc(pages * 256); + motion = calloc(pages, sizeof(uint32_t)); + nz = calloc(pages, sizeof(uint32_t)); + if (!s0 || !s1 || !motion || !nz) + return 1; + for (pass = 0; pass < passes; pass++) { + struct timespec a, b; + uint64_t moved = 0; + + if (pass) + usleep((useconds_t)interval_ms * 1000); + clock_gettime(CLOCK_MONOTONIC, &a); + for (p = 0; p < pages; p++) { + unsigned char *dst = pass ? s1 : s0; + ssize_t r = pread(vram_fd, dst + p * 256, 256, + (off_t)(start + (p << 12))); + + if (r != 256) { + printf("PREAD_FAIL off=0x%llx r=%zd errno=%d\n", + start + (p << 12), r, errno); + return 1; + } + } + clock_gettime(CLOCK_MONOTONIC, &b); + if (pass) { + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + unsigned char *y = s1 + p * 256; + uint32_t d = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j] != y[j]) + d++; + motion[p] += d; + moved += d; + } + printf("PASS %d moved_bytes=%llu dt=%.0fms\n", + pass, (unsigned long long)moved, + t_ms(&a, &b)); + tmp = s0; s0 = s1; s1 = tmp; + } else { + printf("PASS 0 sampled dt=%.0fms\n", + t_ms(&a, &b)); + } + } + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + uint32_t c = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j]) + c++; + nz[p] = c; + } + report_cluster(nz, pages, start, "FINDP"); + for (i = 0; i + 4096 <= pages; i += 4096) { + uint64_t s = 0, c = 0; + size_t j; + + for (j = i; j < i + 4096; j++) { + s += motion[j]; + c += nz[j]; + } + if (s || c) + printf("BLOCK off=0x%llx moving_bytes=%llu nonzero_bytes=%llu\n", + start + ((unsigned long long)i << 12), + (unsigned long long)s, + (unsigned long long)c); + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (motion[p] > bs) { + bs = motion[p]; + best = p; + } + if (!bs) + break; + printf("HOT off=0x%llx score=%u\n", + start + ((unsigned long long)best << 12), bs); + motion[best] = 0; + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (nz[p] > bs) { + bs = nz[p]; + best = p; + } + if (!bs) + break; + printf("CONTENT off=0x%llx nonzero=%u\n", + start + ((unsigned long long)best << 12), bs); + nz[best] = 0; + } + printf("FINDP_DONE\n"); + return 0; + } + + if (!strcmp(mode, "pub")) { + unsigned long long off = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + int frames = argc > 3 ? atoi(argv[3]) : 30; + int fps = argc > 4 ? atoi(argv[4]) : 5; + int w = argc > 5 ? atoi(argv[5]) : DEF_W; + int h = argc > 6 ? atoi(argv[6]) : DEF_H; + int pitch = argc > 7 ? atoi(argv[7]) : DEF_PITCH; + unsigned long long span = (unsigned long long)(h - 1) * pitch + + (unsigned long long)w * 4; + unsigned long long base = off & ~0xfffULL; + size_t wlen = (size_t)((off - base) + span); + unsigned char *wmap; + int f, y, x; + + if (off + span > map_size) { + printf("OFF_RANGE_FAIL need=0x%llx bar=0x%zx\n", + off + span, map_size); + return 1; + } + /* 2026-10-28 (session 149): nonzero-offset mmaps of BAR1 work + * on this mdev (earlier builds assumed only offset 0 <= 64 MB + * was mappable). Map directly at the requested offset; fall + * back to the old offset-0 64 MB window if that ever fails. */ + wlen = (wlen + 0xfff) & ~0xfffUL; + wmap = mmap(NULL, wlen, PROT_READ, MAP_SHARED, vram_fd, + (off_t)base); + if (wmap == MAP_FAILED) { + size_t got = 0; + wmap = map_vram(vram_fd, map_size, &got); + if (wmap) { + base = 0; + wlen = got; + } + if (!wmap || off + span > wlen) { + printf("WIN_MAP_FAIL off=0x%llx len=0x%zx errno=%d\n", + base, wlen, errno); + return 1; + } + } + printf("PUB off=0x%llx base=0x%llx len=0x%zx frames=%d fps=%d %dx%d pitch=%d\n", + off, base, wlen, frames, fps, w, h, pitch); + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + for (f = 0; f < frames; f++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + unsigned char *src = wmap + (size_t)(off - base); + + for (y = 0; y < h; y++) { + unsigned char *s = src + (size_t)y * pitch; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + d[4 * x + 0] = s[4 * x + 2]; + d[4 * x + 1] = s[4 * x + 1]; + d[4 * x + 2] = s[4 * x + 0]; + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(f + 1); + __sync_synchronize(); + printf("FRAME seq=%d ms=%.1f\n", f + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d\n", f); + return 0; + } + + if (!strcmp(mode, "pubp")) { + unsigned long long off = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + int frames = argc > 3 ? atoi(argv[3]) : 30; + int fps = argc > 4 ? atoi(argv[4]) : 5; + int w = argc > 5 ? atoi(argv[5]) : DEF_W; + int h = argc > 6 ? atoi(argv[6]) : DEF_H; + int pitch = argc > 7 ? atoi(argv[7]) : DEF_PITCH; + unsigned long long span = (unsigned long long)(h - 1) * pitch + + (unsigned long long)w * 4; + unsigned long long base = off & ~0xfffULL; + size_t len = (size_t)((off - base) + span); + unsigned char *buf; + int f, y, x; + + len = (len + 0xfff) & ~0xfffUL; + if (off + span > map_size) { + printf("OFF_RANGE_FAIL need=0x%llx bar=0x%zx\n", + off + span, map_size); + return 1; + } + buf = malloc(len); + if (!buf) + return 1; + printf("PUBP off=0x%llx frames=%d fps=%d %dx%d pitch=%d len=0x%zx\n", + off, frames, fps, w, h, pitch, len); + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + for (f = 0; f < frames; f++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + unsigned char *src; + ssize_t r = pread(vram_fd, buf, len, (off_t)base); + + if (r != (ssize_t)len) { + printf("PREAD_FAIL off=0x%llx r=%zd errno=%d\n", + base, r, errno); + return 1; + } + src = buf + (size_t)(off - base); + for (y = 0; y < h; y++) { + unsigned char *s = src + (size_t)y * pitch; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + d[4 * x + 0] = s[4 * x + 2]; + d[4 * x + 1] = s[4 * x + 1]; + d[4 * x + 2] = s[4 * x + 0]; + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(f + 1); + __sync_synchronize(); + printf("FRAME seq=%d ms=%.1f\n", f + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d\n", f); + return 0; + } + + printf("USAGE: fbcopy_pub test|find|findp|pub|pubp " + "<off> [n] [fps] [w] [h] [pitch]\n"); + return 1; +} diff --git a/tools/guest-diag/fenceword_assist.py b/tools/guest-diag/fenceword_assist.py new file mode 100644 index 0000000..9e207e2 --- /dev/null +++ b/tools/guest-diag/fenceword_assist.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# fenceword_assist.py - keep the Vulkan fence wait word of a live vk_frame120 +# client released (session 125). +# +# SUPERSEDED (SESSION 126, commit 95e7f57e): the service now authors these +# releases itself. The `sema_gr_ahead` fence fold sweeps the channel's +# pre-built ring continuation beyond GP_PUT through the ring's validated +# GPFIFO translation, decodes the INC-5 RELEASE groups it finds (addr 0x5c +# count>=5) and queues them in the standard pending ledger; r717 evidence: +# `sema-ahead: h=0xbeec0100 put=11 swept=10 rels=4` followed 30 ms later by +# `sema-svc: deferred release va=0x45ba090 payload=1 (was cur=0)`, with +# VKFRAME seq/DONE and NO assist running (see +# backups/evidence-20261002-victory/r717-fencefold/). Keep this tool only as +# the A/B fallback for boots that predate the fold (or with +# sema_gr_ahead=0). +# +# The client's fence wait polls a word in a 16 KB /dev/nvidiactl BO mapped +# read-write into the client: the init wait word lives at +0x80 (the t122 +# "BW" chaser writes it once) and the per-submit fence word at +0x90 of the +# SAME BO (GPU VA <base>+0x90, delivered by the submit push's INC-5 semaphore +# group). While the emulation does not apply that group, this host assist +# writes 1 to the +0x90 slot whenever it reads 0 - the direct predecessor of +# the service-side fold (see Handoff SESSION 125/126). +# +# Usage (host, while a t122-driven client runs): +# tools/guest-diag/fenceword_assist.py [seconds] [guest-pid] +import os +import subprocess +import sys +import time + +VMEXEC = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), + "..", "..", "scripts", "vgpu_vm.py")) +PY = r''' +import struct, sys, time +pids = [p for p in sys.argv[1:] if p] +maps_of = {} +def vma(pid, addr): + for line in open("/proc/%d/maps" % pid): + rng, perms = line.split(None, 2)[:2] + a, b = [int(x, 16) for x in rng.split("-")] + if a <= addr < b: + return line.strip() + return None +found = {} +for pid in pids: + try: + maps = [l for l in open("/proc/%d/maps" % pid) if "nvidia" in l and "rw-s" in l] + except OSError: + continue + for l in maps: + rng, perms, off = l.split(None, 3)[:3] + a, b = [int(x, 16) for x in rng.split("-")] + if b - a != 0x4000: + continue + try: + mem = open("/proc/%d/mem" % pid, "r+b", 0) + mem.seek(a + 0x80); init = mem.read(4) + mem.seek(a + 0x90); fw = mem.read(4) + except Exception: + continue + if init == b"\x01\x00\x00\x00" and fw == b"\x00\x00\x00\x00": + found[pid] = (a + 0x90, mem) + print("found pid=%s word=%#x" % (pid, a + 0x90), flush=True) +end = time.time() + float(sys.argv[1]) +while time.time() < end: + for pid, (w, mem) in list(found.items()): + try: + mem.seek(w) + if mem.read(4) == b"\x00\x00\x00\x00": + mem.seek(w); mem.write(struct.pack("<I", 1)); mem.flush() + print("released pid=%s word=%#x" % (pid, w), flush=True) + except Exception: + del found[pid] + time.sleep(0.5) +print("assist done", flush=True) +''' + + +def main(): + dur = sys.argv[1] if len(sys.argv) > 1 else "300" + pid = sys.argv[2] if len(sys.argv) > 2 else "" + b64 = __import__("base64").b64encode(PY.encode()).decode() + subprocess.run([VMEXEC, "exec", "--timeout", str(int(dur) + 60), + "echo %s | base64 -d > /tmp/fwa.py; sudo python3 /tmp/fwa.py %s %s" + % (b64, dur, pid)]) + + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/guest_pin_stream.py b/tools/guest-diag/guest_pin_stream.py index 3f4183c..a645a86 100644 --- a/tools/guest-diag/guest_pin_stream.py +++ b/tools/guest-diag/guest_pin_stream.py @@ -211,7 +211,11 @@ def read_u32(kc, va): return struct.unpack("<I", d)[0] if d else 0xFFFFFFFF -DBG_BUDGET = int(os.environ.get("STREAM_DBG_BUDGET", "600")) +DBG_BUDGET = int(os.environ.get("STREAM_DBG_BUDGET", "2400")) +# SESSION 153-CONTINUED-65: minimum seconds between full-offset sweeps of +# one strict anchor (the demand file re-sends every ~2 s; the sweep can cost +# tens of seconds - see the throttle comment at the strict processing loop). +ANCHOR_RETRY_S = float(os.environ.get("STREAM_ANCHOR_RETRY_S", "20")) _dbg_count = [0] @@ -300,50 +304,702 @@ def _nv_vmap_areas(kc): return areas +# Learned page-offset per nv_vm_map_pages area for the demand content match +# (2026-10-16b): the first successful resolution records which page index the +# VA maps to, and later pushes in the same allocation retry that offset first +# instead of walking the whole +/-16 neighbourhood. +_area_off = {} + +# SESSION 153-CONTINUED-55: per-GL-family learned {family_base: (area_lo, +# adj)}. Once one page of a GL push-buffer family is content-verified the +# whole family resolves through the learned area/offset (continuation and +# zero pages included); cleared when the area disappears (context rebuild). +_gl_fam = {} + + def resolve_va_by_content(kc, va, out): """Demand pin: find the page carrying GPU VA 'va' by content match. The host reports the CE push-buffer VAs whose GPFIFO entries it cannot read (dmesg 'stalled VA #N: 0x...'). The guest walks every nv_vm_map_pages vmap area and tests the candidate leaf page at the VA's - in-window offset (plus one neighbour on either side, for the per-mem - header-page offset ambiguity) against the host's method-stream prefix - check. On a match it emits the exact pair, so the host can translate - and consume the entry - no kprobe event, no object-chain discovery, no - resolution lottery. Only the requested VA is pinned, so a wrong - candidate cannot poison other VAs. + in-window offset (plus a neighbourhood, for the per-mem header-page + offset ambiguity) against the host's method-stream prefix check. On a + match it emits the exact pair, so the host can translate and consume the + entry - no kprobe event, no object-chain discovery, no resolution + lottery. Only the requested VA is pinned, so a wrong candidate cannot + poison other VAs. + + SESSION 153-CONTINUED-55: the same rule now covers the GL push-buffer + families (PB_GL_FAMILIES). The setup-push holds (va=0x4001xxx) and the + draw pushes (0x402xxxx) are nv_vm_map_pages allocations exactly like the + CE window; without this they were rejected here silently and the held + setup push was never pinned, so its 4K pages read zeros at the host. + + 2026-10-16b: the offset search window was +/-1 page. Pushes of a client + whose pushbuffer vmap carries a larger per-allocation header (observed + with multi-MB CE staging: pushes at 0x120000f58+ never resolved and the + demand loop re-sent the same VA for minutes while the client crawled) + fall outside it. Widen to +/-16 pages and LEARN the winning offset per + vmap area, because consecutive pushes advance sequentially inside the + same allocation and share the offset. """ - if not (PB_CE_LO <= va < PB_CE_HI): - return False - # 2026-10-16 fix: the CE push-buffer window is 16 MB (PB_CE_LO..PB_CE_HI) - # and each rank is a 1 MB slice of ONE nv_vm_map_pages vmap. The old - # `& 0xFFFFF` folded every 1 MB boundary to offset 0, so ranks 0x120700000, - # 0x120800000 and 0x120a00000 all resolved to page 0 of the first area - - # the same wrong PA - and the host read garbage for every rank but the - # first. Keep the full 16 MB offset: the page index is the true offset - # into the mem, and the per-area pages bound rejects areas too small. - off = (va - PB_CE_LO) & 0xFFFFFF - k = off >> 12 - for (lo, _hi, pages) in _nv_vmap_areas(kc): - for adj in (0, 1, -1): + if PB_CE_LO <= va < PB_CE_HI: + k = ((va - PB_CE_LO) & 0xFFFFFF) >> 12 + gl_base = None + else: + gl_base = next((lo for lo, hi in PB_GL_FAMILIES if lo <= va < hi), + None) + if gl_base is None: + return False + k = (va - gl_base) >> 12 + areas = _nv_vmap_areas(kc) + # SESSION 153-CONTINUED-55: a GL push-buffer family is one allocation + # whose vmap offset is IDENTITY with the family offset (verified: the + # setup push's bytes at va 0x4001004 read back byte-exact at the vmap + # page's +0x1004). Once one page of the family resolves, every page of + # the family is known: continuation pages of a stream do NOT start with + # a method header (the 60216-byte setup push crosses page boundaries + # mid-run, so pages 2/3/5/7 fail the prefix test) and unwritten pages + # read all zero (pages 8..15 of the same push) yet are real translations + # the host read still needs. Use the learned (area, adj) for the whole + # family; only the FIRST learn pays the content test. + learned_fam = _gl_fam.get(gl_base) if gl_base else None + if learned_fam is not None: + lo, adj = learned_fam + if any(a_lo == lo for (a_lo, _h, _p) in areas): + idx = k + adj + pages = next(p for (a_lo, _h, p) in areas if a_lo == lo) + if 0 <= idx < pages: + cpu = lo + idx * 0x1000 + pa = kc.va_to_pa(cpu) + d = kc.read(cpu, 64) if pa is not None else None + if pa is not None and d is not None: + if gl_base is not None or (_push_prefix_ok(d) and + any(d[:32])): + out.append("0x%x,0x%x,1\n" % (va & ~0xFFF, pa)) + dbg("demand-va: 0x%x -> pa=0x%x (gl family learned" + " area=0x%x adj=%+d)" + % (va & ~0xFFF, pa, lo, adj)) + return True + # The learned area is gone (context rebuilt): fall through to a + # fresh content match and re-learn. + _gl_fam.pop(gl_base, None) + for (lo, _hi, pages) in areas: + learned = _area_off.get(lo) + if learned is not None: + adjs = [learned] + else: + adjs = [0, 1, -1] + adjs += [d for m in (2, 4, 8, 16) for d in (m, -m)] + for adj in adjs: idx = k + adj if idx < 0 or idx >= pages: continue cpu = lo + idx * 0x1000 - d = kc.read(cpu, 64) - if not d or not any(d[:32]): - continue - if not _push_prefix_ok(d): - continue pa = kc.va_to_pa(cpu) if pa is None: continue + # 0x200 (not 64): a GL family's page 0 can carry its first + # method header at offset ~0x68 behind a zero-filled prefix + # (observed on the 60216 B setup push), invisible to a 64-byte + # read. + d = kc.read(cpu, 0x200) + if not d or not any(d): + continue + if not _push_prefix_ok(d): + continue + _area_off[lo] = adj + if gl_base is not None: + # First content-verified page of this GL family: the + # identity mapping (area, adj) now serves every page of + # the family, including continuation and zero pages. + _gl_fam[gl_base] = (lo, adj) out.append("0x%x,0x%x,1\n" % (va & ~0xFFF, pa)) - dbg("demand-va: 0x%x -> pa=0x%x (area off=%d adj=%+d)" - % (va & ~0xFFF, pa, idx, adj)) + dbg("demand-va: 0x%x -> pa=0x%x (area off=%d adj=%+d%s)" + % (va & ~0xFFF, pa, idx, adj, + " gl-learn" if gl_base is not None else "")) return True dbg("demand-va: 0x%x not found by content (%d areas)" - % (va & ~0xFFF, len(_nv_vmap_areas(kc)))) + % (va & ~0xFFF, len(areas))) + return False + + +# SESSION 153-CONTINUED-64: candidate adjacencies for the strict window scan. +_STRICT_ADJ = (0, 1, -1, 2, -2, 3, -3, 4, -4, 6, -6, 8, -8, 14, -14, 16, -16) + + +def _stream_parse_full(buf): + """Exact port of the host's svc_stream_parse (gsp_sema_service.c), plus + the MME-setup content requirements the bare parse lacks. + + Zero words are NOP padding (skipped anywhere in the stream), ops 1/3/5 + are INC/NON_INC/ONE_INC runs with a 12-bit count (1..0xfff) and count + data words, ops 0/2/4/7 are header-only, anything else fails. The buffer + parses only when at least one method was seen and the walk lands exactly + on the end (trailing zero padding allowed). + + SESSION 153-CONTINUED-64b: the bare exact-end parse accepted a GMMU + page-table window - PTEs are {pfn<<12, flags} u32 pairs whose values sit + below 0x20000000, so every word reads as op 0 (header-only) and the walk + lands exactly on the end (r1122 pinned pa=0x11dbb0000: all 32 words of + every page op 0, zero data runs, host read `sema-bigrun hdr=0x1dbb7000`). + A window is now only accepted when it carries at least one data run AND + at least one run addressing an MME-setup method (0x114..0x124 / 0x234 / + 0x550..0x568 / 0x3400..0x34fc - the same windows the host's + svc_mme_setup_run_addr classifies with). A page table can never satisfy + either; the held setup push always carries both (the r1116 capture's + first run `20020047` = INC run @0x11c). + """ + off = 0 + methods = 0 + data_runs = 0 + mme_hits = 0 + code_words = 0 + n = len(buf) + while off + 4 <= n: + hdr = struct.unpack_from('<I', buf, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count or off + 4 + count * 4 > n: + return False + data_runs += 1 + addr = (hdr & 0x1FFF) << 2 + if (0x114 <= addr <= 0x124 or addr == 0x234 or + 0x550 <= addr <= 0x568 or 0x3400 <= addr <= 0x34FC): + mme_hits += 1 + if addr == 0x114 and count >= 1: + code_words += count + off += 4 + count * 4 + elif op in (0, 2, 4, 7): + off += 4 + else: + return False + methods += 1 + if methods > 8192: + return False + while off < n and buf[off] == 0: + off += 1 + return (methods > 0 and off == n and data_runs > 0 and mme_hits > 0 + and code_words >= 8) + + +def _walk_measure(buf): + """Method-shape walk returning (end_ok, data_runs, mme_hits, code_words, + pair_runs, parsed_bytes). 'end_ok' mirrors _stream_parse_full's + exact-end + zero padding landing; 'pair_runs' counts runs addressing the + SA-pair method 0x11c (the client push signature); 'parsed_bytes' is how + far the walk got before a shape failure (or the window end).""" + off = 0 + methods = 0 + data_runs = 0 + mme_hits = 0 + code_words = 0 + pair_runs = 0 + n = len(buf) + end_ok = False + while off + 4 <= n: + hdr = struct.unpack_from('<I', buf, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count or off + 4 + count * 4 > n: + break + data_runs += 1 + addr = (hdr & 0x1FFF) << 2 + if (0x114 <= addr <= 0x124 or addr == 0x234 or + 0x550 <= addr <= 0x568 or 0x3400 <= addr <= 0x34FC): + mme_hits += 1 + if addr == 0x114: + code_words += count + if addr == 0x11c and count >= 2: + pair_runs += 1 + off += 4 + count * 4 + elif op in (0, 2, 4, 7): + off += 4 + else: + break + methods += 1 + if methods > 8192: + break + else: + o2 = off + while o2 < n and buf[o2] == 0: + o2 += 1 + end_ok = (methods > 0 and o2 == n) + return end_ok, data_runs, mme_hits, code_words, pair_runs, off + + +def _window_accept(buf, ln): + """Acceptance for a client-mapping candidate window. + + SESSION 153-CONTINUED-65: the full-offset sweep of the client-mem + resolver needs a false-positive gate that survives pointer/event data. + The probe of the r1125 boot found a 60216-byte window of pointer junk + that walked to an exact end with one misinterpreted 0x114 run + (mme=1): the old accept (data>0, mme>0, code>=8) would have pinned it. + Large windows therefore require the client push signature - at least + two MME-setup runs or two SA-pair (0x11c) runs - in addition to the + counts; small windows keep the historical strict rule. The relaxed + path additionally accepts a window whose tail is unmapped/unwritten + (walk parses >= 0x1000) when it carries >= 64 code words (the macro + upload signature). + + Returns (ok, mode). + """ + if ln < 0x8000: + return _stream_parse_full(buf), "strict" + end_ok, data, mme, code, pairs, parsed = _walk_measure(buf) + # The client-push signature: SA-pair (0x11c) runs or 0x114 code runs. + # The r1125 probe of the live client mappings measured 10k+ pointer/float + # windows with data/mme hits but pairs=0, code=0 - MME-setup-family hits + # alone (0x124/0x34xx) are NOT a discriminator; the pair/code runs are. + sig = (pairs >= 2 or code >= 8) + if end_ok and data >= 2 and sig: + return True, "strict" + if data >= 2 and sig and parsed >= 0x1000: + return True, "relaxed" + return False, None + + +def _prefix_walk_ok(buf, off): + """Cheap first-page pre-filter for a strict window candidate. + + Walks the method shapes from `off` to the end of `buf` WITHOUT requiring + the end to land on a boundary (runs may cross into the next page and a + run header near the page end declares data we cannot see yet). Fails on + the first header no method shape can produce; this kills arbitrary data + pages after a few words so the full parse only runs on survivors. + """ + n = len(buf) + while off + 4 <= n: + hdr = struct.unpack_from('<I', buf, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count: + return False + off += 4 + count * 4 + elif op in (0, 2, 4, 7): + off += 4 + else: + return False + return True + + +def _read_window(kc, cpu, off, ln): + """Read [cpu+off, cpu+off+ln) page-by-page through kcore. + + A vmap area's pages are contiguous in the kernel VA but arbitrary in + physical memory, so the window is assembled one page at a time. + """ + out = b"" + a = cpu + off + left = ln + while left > 0: + n = min(left, 0x1000 - (a & 0xFFF)) + d = kc.read(a, n) + if not d: + return None + out += d + a += n + left -= n + return out + + +def _mme_run_in_page(d): + """True when a 4K page contains a run addressing an MME-setup method. + + The strict-window prefilter for client-mapping candidates (cheap: one + page read rejects the ~511/512 wrong shifts before the full parse). + """ + n = len(d) + off = 0 + while off + 4 <= n: + hdr = struct.unpack_from('<I', d, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count: + off += 4 + continue + addr = (hdr & 0x1FFF) << 2 + if (0x114 <= addr <= 0x124 or addr == 0x234 or + 0x550 <= addr <= 0x568 or 0x3400 <= addr <= 0x34FC): + return True + off += 4 + count * 4 + else: + off += 4 + return False + + +def resolve_va_client_mem(pid, ent_va, off, ln, out): + """Strict pin for a setup push whose pages live in CLIENT sysmem. + + SESSION 153-CONTINUED-64c: when the setup push's allocation is UVM + sysmem it has NO kernel vmap (r1122: no nv_vm_map_pages area held the + `a0100045` 0x114 upload runs anywhere on the boot - the content exists + only in the client's `rw-s` /dev/nvidia mappings, and the host's + GPU-page-table walk answered an unrelated page-table buffer for the + held VA). The client's CPU mappings + /proc/<pid>/pagemap are the only + ground truth for those pages. Scan the client's readable+writable + mappings: for each candidate page shift, read the exact [va, va+len) + window via /proc/<pid>/mem, validate with the host's own full parse, + and emit pins for every page of the validated window from the + pagemap. The cheap per-page MME-run prefilter rejects wrong shifts. + """ + if not pid: + return False + try: + pfd = os.open('/proc/%d/pagemap' % pid, os.O_RDONLY) + mfd = os.open('/proc/%d/mem' % pid, os.O_RDONLY) + except OSError: + return False + try: + gl_base = next((lo for lo, hi in PB_GL_FAMILIES + if lo <= (ent_va & ~0xFFF) < hi), None) + k = ((ent_va & ~0xFFF) - gl_base) >> 12 if gl_base is not None else None + if ln <= 0 or ln > 0x40000: + return False + need = (off + ln + 0xFFF) // 0x1000 + if need > 64: + return False + try: + maps = open('/proc/%d/maps' % pid).read().splitlines() + except OSError: + return False + tried = 0 + for line in maps: + parts = line.split() + if len(parts) < 5 or 'rw' not in parts[1]: + continue + # SESSION 153-CONTINUED-64e: any rw mapping may carry the + # sysmem push (observed as pathless rw-s /dev/nvidia mappings; + # memfd-backed buffers cannot be excluded either - the strict + # parse + MME prefilter are the safety gates, not the path). + try: + lo, hi = [int(x, 16) for x in parts[0].split('-')] + except ValueError: + continue + if hi - lo < (need + 4) * 0x1000 or hi - lo > (64 << 20): + continue + pages = (hi - lo) >> 12 + if k is None or k == 0: + # Unknown family, or the FAMILY-BASE page (k == 0): neither + # has an identity page index to key on, so sweep every page + # start of the mapping (bounded). SESSION 153-CONTINUED-65: + # the 0x3200000 macro-upload push's window sits at an + # arbitrary page offset inside the client's mapping (the + # r1124 power-of-two neighbourhood never tried it; the + # window content measures code=712, exact-end, under the + # strict walk). + adjs = range(0, max(1, pages - need + 1)) + else: + # SESSION 153-CONTINUED-65: neighbourhood FIRST, then the + # FULL page sweep. The r1125 setup push (family pages k=1 + # and k=33, 60216 B) failed every power-of-two offset while + # a matching window could sit at an arbitrary mapping page; + # every candidate still passes _window_accept before any + # pin is emitted, so the widened sweep cannot pin a window + # the parse rejects. + near = [0, 1, -1, 2, -2, 4, -4, 8, -8, 16, -16, 32, -32, + 64, -64, 128, -128, 256, -256] + near_set = set(near) + rest = [a for a in range(-k, pages - need + 1 - k) + if a not in near_set] + adjs = near + rest + for adj in adjs: + idx = (k + adj) if k is not None else adj + if idx < 0 or idx + need > pages: + continue + win = lo + idx * 0x1000 + off + tried += 1 + if tried > 8192: + break + try: + first = os.pread(mfd, 0x1000 - off, win) + except OSError: + continue + if not first: + continue + # An all-zero first page is a legitimate push head (the + # method stream may start later); only a page with non-zero + # content and no MME-setup run is rejected outright. + if not _mme_run_in_page(first) and any(first): + continue + buf = b'' + left = ln + a = win + ok = True + while left > 0: + n = min(left, 0x1000 - (a & 0xFFF)) + try: + d = os.pread(mfd, n, a) + except OSError: + ok = False + break + if not d or len(d) != n: + ok = False + break + buf += d + a += n + left -= n + if not ok: + continue + acc, mode = _window_accept(buf, ln) + if not acc: + continue + pas = [] + for j in range(need): + vpg = (ent_va & ~0xFFF) + j * 0x1000 + cpg = lo + (idx + j) * 0x1000 + os.lseek(pfd, (cpg >> 12) * 8, 0) + e = int.from_bytes(os.read(pfd, 8), 'little') + if not (e >> 63) & 1: + pas = None + break + pas.append(((e & ((1 << 55) - 1)) << 12)) + if not pas: + continue + for j, pa in enumerate(pas): + out.append("0x%x,0x%x,1\n" + % ((ent_va & ~0xFFF) + j * 0x1000, pa)) + dbg("demand-va: 0x%x -> pa=0x%x (%s client-mem parse" + " %u B ok pages=%u map=0x%x idx=%d)" + % (ent_va & ~0xFFF, pas[0], mode, ln, need, lo, idx)) + return True + if tried > 8192: + break + return False + finally: + os.close(pfd) + os.close(mfd) + + +def _strict_kvm_sweep(kc, ent_va, off, ln, out): + """Kernel-vmap sweep for an anchor with no known-family page index. + + SESSION 153-CONTINUED-74: mirrors the k == 0 known-family path (the + resolver that found the r1126 0x3200000 macro window) but works for ANY + anchor outside PB_GL_FAMILIES. Every candidate window passes the host's + own full parse (`_window_accept`) before a single pin is emitted, so a + false positive is impossible by construction; the cheap per-page prefix + walk rejects arbitrary data after a few words. Bounded: at most + 8192 candidate windows and at most 64 pages of a window (the same caps + as resolve_va_client_mem). + + Areas swept: the nv_vm_map_pages vmap set (the historical resolver) plus + the nvidia os_alloc_mem vmalloc areas (`vpages`), which the r1156 probe + showed can be large (571 pages in one area) and are completely outside + _nv_vmap_areas' filter. A window is only emitted for an area whose pages + are all present in the CPU alias. + """ + import re as _re + + page = ent_va & ~0xFFF + need = (off + ln + 0xFFF) // 0x1000 + if ln <= 0 or ln > 0x40000 or need > 64: + return False + areas = list(_nv_vmap_areas(kc)) + try: + for line in open("/proc/vmallocinfo"): + if "[nvidia]" not in line or "vmalloc" not in line: + continue + if "nv_vm_map_pages" in line or "phys=" in line: + continue + if "os_map_kernel_space" in line or "ioremap" in line: + continue + m = _re.match(r'0x([0-9a-f]+)-0x([0-9a-f]+)\s+(\d+)', line) + if not m: + continue + lo = int(m.group(1), 16) + hi = int(m.group(2), 16) + pages = (hi - lo) >> 12 + if pages >= need and lo < hi: + areas.append((lo, hi, pages)) + except OSError: + pass + tried = 0 + for (lo, _hi, pages) in areas: + for adj in range(0, max(1, pages - need + 1)): + tried += 1 + if tried > 8192: + return False + cpu = lo + adj * 0x1000 + first = kc.read(cpu, 0x1000) + if not first: + continue + if any(first) and not _prefix_walk_ok(first, off): + continue + buf = _read_window(kc, cpu, off, ln) + if buf is None: + continue + acc, mode = _window_accept(buf, ln) + if not acc: + continue + pas = [] + for j in range(need): + vpa = kc.va_to_pa(cpu + j * 0x1000) + if vpa is None: + pas = None + break + pas.append(vpa) + if not pas: + continue + for j, vpa in enumerate(pas): + out.append("0x%x,0x%x,1\n" % (page + j * 0x1000, vpa)) + dbg("demand-va: 0x%x -> pa=0x%x (%s vmap parse %u B ok" + " pages=%u area=0x%x adj=%+d)" + % (page, pas[0], mode, ln, need, lo, adj)) + return True + dbg("demand-va: 0x%x strict vmap sweep failed (%u B, %u windows)" + % (page, ln, tried)) + return False + + +def resolve_va_strict(kc, ent_va, off, ln, out, pid=None): + """Demand pin validated by the host's exact full-parse of [ent_va, +ln). + + SESSION 153-CONTINUED-64: the content heuristic learned a GL family from + ONE page's weak 0x200-byte prefix test and then served every family page + from that (area, adj) with NO content check at all; r1120 pinned + 0x4001000 -> a page whose bytes (`00040000 00001000...`) failed the + host's stream parse for the whole boot (`plausible=0 unreadable=1`), so + the setup push was never captured and the prelude gate deferred every + launch. This path runs the HOST'S OWN full parse on the exact bytes the + host will read before any pin is emitted, then pins EVERY page of the + validated window and learns the family from it. A family is never + served from an unvalidated learn. + """ + page = ent_va & ~0xFFF + gl_base = next((lo for lo, hi in PB_GL_FAMILIES if lo <= page < hi), None) + if ln <= 0 or ln > 0x40000: + return False + if gl_base is None: + # SESSION 153-CONTINUED-64e: an anchor outside the known families + # (r1124: the hold moved to a 153600-byte push at 0x3200000) has no + # identity page index; only the client-mapping sweep can resolve it. + # + # SESSION 153-CONTINUED-74: the unknown-family branch skipped the + # kernel-vmap sweep entirely - the known-family path has always tried + # it, and it is the resolver that found the r1126 0x3200000 macro + # window (a KERNEL-mapped allocation no process pagemap covers). The + # r1156 held push (va=0x3e4ccccc, 123916 B) failed here after only + # the client-mem attempt ("strict unknown-family client-mem failed"), + # so a kernel-resident window could never be found for any anchor + # outside PB_GL_FAMILIES. Try the vmap sweep first (same full-parse + # gate; strictly additive - a failed sweep costs the bounded reads + # and falls through to the client-mem path unchanged). + try: + if _strict_kvm_sweep(kc, ent_va, off, ln, out): + return True + except Exception: + pass + try: + if resolve_va_client_mem(pid, ent_va, off, ln, out): + return True + except Exception: + pass + dbg("demand-va: 0x%x strict unknown-family (vmap+client-mem) failed" + % page) + return False + gl_hi = next(hi for lo, hi in PB_GL_FAMILIES if lo <= page < hi) + k = (page - gl_base) >> 12 + need = (off + ln + 0xFFF) // 0x1000 + if need > 64 or page + need * 0x1000 > gl_hi: + return False + areas = _nv_vmap_areas(kc) + cands = [] + fam = _gl_fam.get(gl_base) + if fam is not None and any(a_lo == fam[0] for (a_lo, _h, _p) in areas): + cands.append(fam) + if k == 0: + # SESSION 153-CONTINUED-65: a family-BASE anchor (the 0x3200000 + # macro-upload push) has no identity page index to key on. The + # r1126 macro window lives in a KERNEL-mapped allocation (no + # process pagemap maps it: the whole-process reverse scan found + # none), so the kcore vmap sweep is the only resolver - and the + # power-of-two neighborhood missed the window's arbitrary page + # index. Sweep every page of every nv_vm_map_pages area; the + # _window_accept parse gates each candidate. + for (lo, _hi, pages) in areas: + for adj in range(0, max(1, pages - need + 1)): + c = (lo, adj) + if c not in cands: + cands.append(c) + else: + for (lo, _hi, pages) in areas: + for adj in _STRICT_ADJ: + c = (lo, adj) + if c not in cands: + cands.append(c) + tried = 0 + for (lo, adj) in cands: + pages = next(p for (a_lo, _h, p) in areas if a_lo == lo) + idx = k + adj + if idx < 0 or idx + need > pages: + continue + tried += 1 + if tried > (8192 if k == 0 else 128): + break + cpu = lo + idx * 0x1000 + first = kc.read(cpu, 0x1000) + if not first: + continue + # SESSION 153-CONTINUED-65: an all-zero first page is a legitimate + # push head - the r1126 macro-upload window (0x3200000, 153600 B) + # starts with a fully zero page (the method stream lands later in + # the window); the old `not any(first)` skip threw exactly that + # candidate away (the r1128 sweep tried 166 kernel-vmap windows and + # rejected the true one on this check). The prefix walk trivially + # passes an all-zero page; the full-window accept is the gate. + if any(first) and not _prefix_walk_ok(first, off): + continue + buf = _read_window(kc, cpu, off, ln) + if buf is None: + continue + acc, mode = _window_accept(buf, ln) + if not acc: + continue + pas = [] + for j in range(need): + vpa = kc.va_to_pa(cpu + j * 0x1000) + if vpa is None: + pas = None + break + pas.append(vpa) + if not pas: + continue + for j, vpa in enumerate(pas): + out.append("0x%x,0x%x,1\n" % (page + j * 0x1000, vpa)) + _area_off[lo] = adj + _gl_fam[gl_base] = (lo, adj) + dbg("demand-va: 0x%x -> pa=0x%x (%s parse %u B ok pages=%u" + " area=0x%x adj=%+d)" % (page, pas[0], mode, ln, need, lo, adj)) + return True + dbg("demand-va: 0x%x strict parse failed (%u B, %u windows tried)" + % (page, ln, tried)) + # SESSION 153-CONTINUED-64c: no kernel vmap held the content - the + # allocation is UVM sysmem visible only in the client's own mappings. + try: + if resolve_va_client_mem(pid, ent_va, off, ln, out): + return True + except Exception: + pass + dbg("demand-va: 0x%x strict client-mem parse failed too" % page) return False @@ -718,6 +1374,49 @@ def resolve_sem_pins(kc, sem, out): return resolve_mem_pins(kc, mem, out, default_pages=16) +def resolve_pool_va(kc, va, out, sems): + """Pool/waiter VA -> PA via tracker-sem -> page -> mem; NO content match. + + The word at POOL base + 4*idx is a slot in a UVM semaphore-pool page: a + sysmem uvm_rm_mem whose PA is only reachable through its kernel CPU + alias (the client pagemap has no present entry for it). Same kcore walk + the CE pushbuffer uses; the chain is shorter (sem+0 = page, page+0 = + mem). The page sits 16 MB above the CE push buffer base, outside + resolve_sem_pins' default_pages window, which is why the host saw + 'pending release starved' for it on every client's first seconds. + """ + page = va & ~0xFFF + if not (POOL_SEM_LO <= page < POOL_SEM_HI): + return False + for sem in list(sems): + if not is_kern_ptr(sem): + continue + try: + pg = kc.read64(sem) + if not is_kern_ptr(pg): + continue + mem = kc.read64(pg) + if not is_kern_ptr(mem): + continue + got = [] + resolve_mem_pins(kc, mem, got, default_pages=1, + va_lo=page, va_hi=page + 0x1000) + except Exception: + continue + for line in got: + try: + lva = int(line.split(',')[0], 16) + except Exception: + continue + if lva == page: + out.append(line) + dbg('pool-va: 0x%x -> %s (mem=0x%x)' + % (page, line.strip(), mem)) + return True + dbg('pool-va: 0x%x unresolved (%d known sems)' % (page, len(sems))) + return False + + def resolve_pb_pins(kc, pb, out): mem = kc.read64(pb + 8) if not mem: @@ -751,6 +1450,27 @@ def resolve_pb_pins(kc, pb, out): # hold the ring-entry pushes is chosen (content anchor, _mem_content_ok). PB_CE_LO = 0x120000000 PB_CE_HI = 0x121000000 +# SESSION 153-CONTINUED-55: GL push-buffer VA families. The GL client's +# setup/MME-prelude push buffers (observed hold va=0x4001004 len=60216) and +# the draw pushes (observed va=0x4021004) are separate nv_vm_map_pages +# allocations in two fixed windows; the same family-relative page index +# rule applies: k = (va - base) >> 12. +# SESSION 153-CONTINUED-63: the r1117 boot's setup push lives at +# va=0x4200110 len=12288 - a THIRD window (0x4200000..0x4210000) the two +# original families did not cover, so the content resolver answered +# nothing and the read lottery could never be won for it (the +# 0x4020000-family hold re-armed for its whole 20 s window). +PB_GL_FAMILIES = ((0x3200000, 0x3300000), (0x4000000, 0x4100000), + (0x4200000, 0x4210000), + (0x42000000, 0x43000000)) +# m165 (D-F1a): UVM semaphore-pool page range. The pool is a 4 KB sysmem +# uvm_rm_mem placed by the UVM arena right after the 16 MB CE push buffer; +# the client's tracker words (0x121000000 + 4*idx) live in its first page. +# The client pagemap has no present entry for it, so only the kcore +# tracker-sem -> page -> mem walk can translate it. +POOL_SEM_LO = 0x121000000 +POOL_SEM_HI = 0x122000000 +KC_POOL_REEMIT_S = 2.0 # re-assert the pool page pair this often too KC_CHAIN_BACK = 0x2000 KC_CHAIN_SPAN = 0xA000 KC_CHAIN_MAX = 400 @@ -1331,13 +2051,49 @@ def nvidia_fds(pid): return kind +def pid_starttime(pid): + """Starttime (clock ticks, field 22) of /proc/<pid>, or None. + + Part of the client identity (pid, starttime): pids are recycled, so a + per-client reset keyed on the pid alone can mis-fire on a recycled pid. + Parsed after the last ')' so comm names with spaces/parens are safe.""" + try: + with open('/proc/%d/stat' % pid) as f: + st = f.read() + rp = st.rfind(')') + if rp < 0: + return None + toks = st[rp + 2:].split() + if len(toks) < 20: # starttime = field 22 overall = index 19 + return None + return int(toks[19]) + except (OSError, ValueError): + return None + + +def pid_line(tag, pid, start): + """'#<tag> <pid> [starttime]' - the optional starttime keeps the line + backward compatible with older hosts (they parse the pid only).""" + if start is None: + return "#%s %d\n" % (tag, pid) + return "#%s %d %d\n" % (tag, pid, start) + + def cdt_pid(): prefixes = tuple(p for p in os.environ.get("CLIENT_PREFIXES", - "cdt,va,victory").split(",") if p) + "cdt,va,victory,Xorg,X,glxgears," + "gzdoom").split(",") if p) # Processes that hold a /dev/nvidia* fd but are not CUDA clients; the # scanner must never commit to them (a transient nvidia-smi at boot was # picked once and its exited pagemap read back EOF forever). - non_clients = ("nvidia-smi", "nvidia-persistenced", "Xorg", "X") + # + # SESSION 146: Xorg/X used to be excluded here too, but the GL/GZDOOM + # path needs the X server's pages pinned - Xorg screen init fails with + # "Failed to initialize DMA / Failed to allocate push buffer" when the + # nvidia-push buffers cannot be resolved (observed r816). Xorg is a + # long-lived process (unlike the nvidia-smi transient the exclusion + # guarded against), so it is a valid, stable pin target. + non_clients = ("nvidia-smi", "nvidia-persistenced") cands = [] for d in os.listdir('/proc'): if not d.isdigit(): @@ -1449,6 +2205,18 @@ def scan_vmas(fd, pid, out): parts = line.split() if len(parts) < 5: continue + # Skip mappings with no read permission: a CUDA device buffer + # is a PROT_NONE reservation (e.g. `---p 00000000 00:00 0`), + # potentially hundreds of MB. Its pages are never present in + # the pagemap, so scanning it produces no pairs - but it + # consumed the VMA_MAX_PAGES budget and the scan never reached + # the client's present staging buffers that sit ABOVE the + # device reservation (observed: a client whose device range + # started below its staging buffer got no pins at all and its + # H2D copies failed with rc=-2 forever). + perms = parts[1] if len(parts) >= 2 else '' + if 'r' not in perms: + continue # Anonymous mappings (no path field) are the client's UVM sysmem # buffers when the driver maps them without a /dev/zero backing: # the second client's cudaMalloc'd data buffers show up exactly @@ -1550,16 +2318,35 @@ def main(): except OSError: pass last_pid = None + last_start = None last_log = 0.0 scrub_next = 0.0 demand_next = 0.0 - _want_state = {"mtime": None, "done": set()} + # m165 (D-F1a): pool page VAs learned from demand, re-asserted every + # KC_POOL_REEMIT_S. Boot-stable (UVM frees pool pages only at GPU + # deinit), so the set deliberately persists across client switches. + pool_pages = {} + pool_reemit_next = 0.0 + _want_state = {"mtime": None, "done": set(), "strict_pending": {}, + "strict_try": {}} loop = 0 while True: pid = cdt_pid() loop += 1 if pid is None: + if last_pid is not None and sock is not None: + # In-band client-death signal: the host tools purge the dead + # client's per-client state on this line, so the NEXT client + # works without a VM cycle (the kill cascade). Best-effort - + # if the socket is gone the host still has its own probe. + try: + sock.sendall(pid_line("nopid", last_pid, + last_start).encode()) + except OSError: + sock.close() + sock = None last_pid = None + last_start = None seen = {} seen_sems = set() seen_pbs = set() @@ -1568,6 +2355,7 @@ def main(): continue if pid != last_pid: last_pid = pid + last_start = pid_starttime(pid) seen = {} seen_sems = set() seen_pbs = set() @@ -1598,7 +2386,7 @@ def main(): dbg("switch seed: +%d event kptr(s)" % _seeded) if sock is not None: try: - sock.sendall(("#pid %d\n" % pid).encode()) + sock.sendall(pid_line("pid", pid, last_start).encode()) except OSError: sock.close() sock = None @@ -1646,23 +2434,203 @@ def main(): st_d = os.stat("/tmp/guest_want_vas.txt") if st_d.st_mtime != _want_state.get("mtime"): with open("/tmp/guest_want_vas.txt") as fw: - wvas = [int(t, 16) for t in fw.read().split() - if t.startswith("0x")] + # r582: tolerate malformed historical lines. The + # r581 boot's scanner died here: a transport-escaped + # append merged VAs into one token containing + # literal backslash-n ('0x20400f000\\n0x121000000\\n + # ...') and int(t, 16) raised ValueError out of + # main(), stopping ALL new pins (the observed + # pool-pin latency). Extract every 0x token instead; + # a bad token is skipped, never fatal. + text = fw.read() _want_state["mtime"] = st_d.st_mtime - for wv in wvas[-64:]: - if wv in _want_state["done"]: + # SESSION 153-CONTINUED-64: strict demand anchors first. + # A line `S 0x<entry_va> 0x<off> <len>` (written by + # pin_gr_stream from the service's anchored debugfs + # demand record) asks for the exact [entry_va, +len) + # read window to be validated with the host's own full + # stream parse before a pin is injected. Loose lines + # (plain `0x<va>`) keep the content/pagemap paths, but + # a GL family with a pending anchor is deferred so its + # learn can only come from the strict validation. + strict = [] + loose_tokens = [] + for line in text.splitlines(): + ms = re.match( + r"\s*S\s+0x([0-9a-fA-F]+)\s+0x([0-9a-fA-F]+)" + r"\s+(\d+)\s*$", line) + if ms: + try: + strict.append((int(ms.group(1), 16), + int(ms.group(2), 16), + int(ms.group(3)))) + except ValueError: + pass + continue + loose_tokens += re.findall(r"0x[0-9a-fA-F]+", line) + now_d = time.time() + for epage in list(_want_state["strict_pending"]): + if now_d - _want_state["strict_pending"][epage] > 300.0: + del _want_state["strict_pending"][epage] + for (ev, eoff, eln) in reversed(strict): + epage = ev & ~0xFFF + if epage in _want_state["done"]: continue + # SESSION 153-CONTINUED-65: throttle anchor + # re-resolution. The full-offset sweep can read + # thousands of windows (tens of seconds of CPU); + # the demand file re-sends the anchor every ~2 s, + # so without this throttle the scanner starves the + # client (both measured at ~91% CPU at r1125). One + # attempt per anchor per window is enough - content + # that lands late is picked up on the next window. + lt = _want_state["strict_try"].get(epage, 0.0) + if now_d - lt < ANCHOR_RETRY_S: + continue + _want_state["strict_try"][epage] = now_d got = [] try: - ok = resolve_va_by_content(kc, wv, got) + ok = resolve_va_strict(kc, ev, eoff, eln, got, + pid) except Exception: ok = False got = [] + if not ok and eoff: + # The demand's off is the entry va's in-page + # offset; the push content can be page-aligned + # instead (r1123 vs r1125 parse behaviour). A + # second try at page alignment is cheap next to + # the completed sweep and the same pins result. + got2 = [] + try: + ok = resolve_va_strict(kc, ev & ~0xFFF, 0, + eln, got2, pid) + except Exception: + ok = False + if ok: + got = got2 + if ok and got: + lines.extend(got) + for g in got: + try: + _want_state["done"].add( + int(g.split(",", 1)[0], 16)) + except ValueError: + pass + _want_state["done"].add(epage) + _want_state["strict_pending"].pop(epage, None) + else: + _want_state["strict_pending"].setdefault( + epage, now_d) + # Loose VAs in a family whose strict anchor is still + # pending are deferred to the strict path only. + pend_bases = set() + for epage in _want_state["strict_pending"]: + for lo, hi in PB_GL_FAMILIES: + if lo <= epage < hi: + pend_bases.add((lo, hi)) + # SESSION 153-CONTINUED-55: process the NEWEST VAs first, + # not the deduped head. The file is append-only and grows + # to hundreds of lines (every stalled-VA row is appended + # each demand cycle, ~48/4 s under load), so both the + # literal tail (`wvas[-64:]`) and a first-occurrence + # dedupe sorted older pending VAs out of the window (the + # 16-page GL setup-push demand fell out between its + # appends and the next mtime change). Walk the raw tail + # backwards, newest instruction first, skipping resolved + # VAs, up to a bounded batch. + wvas = [] + for t in reversed(loose_tokens): + try: + v = int(t, 16) + except ValueError: + continue + if v not in _want_state["done"] and v not in wvas: + wvas.append(v) + if len(wvas) >= 256: + break + for wv in wvas: + if wv in _want_state["done"]: + continue + if any(lo <= (wv & ~0xFFF) < hi + for lo, hi in pend_bases): + continue + got = [] + if POOL_SEM_LO <= (wv & ~0xFFF) < POOL_SEM_HI: + # m165 (D-F1a): pool/waiter VAs have no method + # stream content and the CE offset math does not + # apply; route them to the structural tracker-sem + # walk instead. m168: the tracker sems are + # long-lived objects (allocated once) whose page + # pointer follows the pool across clients, so try + # every sem we have ever resolved a chain for - + # seen_sems alone can miss the live generation. + _psems = set(seen_sems) | set(KC_PB_STATE.keys()) \ + | set(resolved_kc_pb) + try: + ok = resolve_pool_va(kc, wv, got, + _psems) + except Exception: + ok = False + got = [] + dbg('demand-pool-va: 0x%x %s' + % (wv, 'ok' if ok else 'pending')) + if ok and got: + pool_pages.setdefault(wv & ~0xFFF, True) + else: + try: + ok = resolve_va_by_content(kc, wv, got) + except Exception: + ok = False + got = [] + # r586: a demanded VA outside the CE + # pushbuffer window (the GRFILL staging pages, + # high client VAs) is a plain client mapping + # page: resolve it through the client's own + # pagemap. Without this the demand for those + # VAs was a silent no-op (resolve_va_by_content + # rejects anything outside its windows) and the + # fills kept dropping them. + if not ok and not got and pid: + try: + pfd = os.open('/proc/%d/pagemap' % pid, + os.O_RDONLY) + try: + sub = {} + vpg = wv & ~0xFFF + scan_range(pfd, vpg, vpg + 0x1000, + sub) + finally: + os.close(pfd) + if sub: + for v, p in sub.items(): + got.append('0x%x,0x%x,1\n' + % (v, p)) + ok = True + except OSError: + pass if ok and got: lines.extend(got) _want_state["done"].add(wv) except OSError: pass + # m165 (D-F1a): re-assert the pool page pair every KC_POOL_REEMIT_S. + # The pool page is boot-stable (UVM frees it only at GPU deinit) but + # the host pin table is wiped at every session flush, so a one-shot + # pin leaves later clients starving until the slow structural walks + # answer; the periodic re-assert heals the wipe within seconds (the + # same argument as the CE mem sweep above). + if kc_ok and pool_pages and time.time() >= pool_reemit_next: + pool_reemit_next = time.time() + KC_POOL_REEMIT_S + for pv in list(pool_pages): + got = [] + try: + ok = resolve_pool_va(kc, pv, got, seen_sems) + except Exception: + ok = False + got = [] + if ok and got: + lines.extend(got) # PMA scrubber buffer discovery (throttled): resolve the finish/host- # semaphore word PAs and stream them for host registration. if kc_ok and time.time() >= scrub_next: @@ -1914,7 +2882,8 @@ def main(): sock.connect((CID, PORT)) sock.settimeout(None) if last_pid is not None: - sock.sendall(("#pid %d\n" % last_pid).encode()) + sock.sendall(pid_line("pid", last_pid, + last_start).encode()) except OSError: if sock: sock.close() diff --git a/tools/guest-diag/launch_qemu_rootport.sh b/tools/guest-diag/launch_qemu_rootport.sh index 444bdfc..d38321b 100755 --- a/tools/guest-diag/launch_qemu_rootport.sh +++ b/tools/guest-diag/launch_qemu_rootport.sh @@ -1,4 +1,4 @@ #!/bin/bash # SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2026 Open-IOV contributors -exec /bin/qemu-system-x86_64 -D /home/rmcore/.local/libvf.io/logs/qemu/e726ba0b-204f-4ec9-91da-983ed7f41d06-session.txt -no-hpet -mem-prealloc -qmp unix:/tmp/qemu-qmp.sock,server,nowait -chardev socket,path=/tmp/qemu-guest-agent.sock,server=on,wait=off,id=qga0 -device virtio-serial-pci -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -serial file:/tmp/vgpu-guest-serial.log -gdb tcp::1234 -display none -device ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr -object memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=128M,share=yes -device ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr -object memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=2M,share=yes -object memory-backend-file,id=pc.ram,size=4096M,mem-path=/dev/shm/vgpu-guest-ram,share=yes,prealloc=on -uuid e726ba0b-204f-4ec9-91da-983ed7f41d06 -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -cpu host,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=on,topoext=on -rtc clock=host,base=localtime -m 4096 -smp cores=4,threads=1,sockets=1 -hda /home/rmcore/.local/libvf.io/kernel/linux.qcow2 --enable-kvm -device pcie-root-port,id=rp0,chassis=1,slot=30,bus=pcie.0 -device vfio-pci,id=hostdev0,bus=rp0,sysfsdev=/sys/bus/mdev/devices/<MDEV-UUID>,display=off -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22 -device vhost-vsock-pci,guest-cid=50718 -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/main.sock,server,nowait -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/master.sock,server,nowait +exec /bin/qemu-system-x86_64 -D /home/rmcore/.local/libvf.io/logs/qemu/e726ba0b-204f-4ec9-91da-983ed7f41d06-session.txt -no-hpet -mem-prealloc -qmp unix:/tmp/qemu-qmp.sock,server,nowait -chardev socket,path=/tmp/qemu-guest-agent.sock,server=on,wait=off,id=qga0 -device virtio-serial-pci -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -serial file:/tmp/vgpu-guest-serial.log -gdb tcp::1234 -display none -device ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr -object memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=128M,share=yes -device ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr -object memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=2M,share=yes -object memory-backend-file,id=pc.ram,size=8192M,mem-path=/dev/shm/vgpu-guest-ram,share=yes,prealloc=on -uuid e726ba0b-204f-4ec9-91da-983ed7f41d06 -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -cpu host,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=on,topoext=on -rtc clock=host,base=localtime -m 8192 -smp cores=4,threads=1,sockets=1 -hda /home/rmcore/.local/libvf.io/kernel/linux.qcow2 --enable-kvm -device pcie-root-port,id=rp0,chassis=1,slot=30,bus=pcie.0 -device vfio-pci,id=hostdev0,bus=rp0,sysfsdev=/sys/bus/mdev/devices/<MDEV-UUID>,display=off -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22 -device vhost-vsock-pci,guest-cid=50718 -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/main.sock,server,nowait -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/master.sock,server,nowait diff --git a/tools/guest-diag/nvkmstrace.c b/tools/guest-diag/nvkmstrace.c new file mode 100644 index 0000000..ca72e46 --- /dev/null +++ b/tools/guest-diag/nvkmstrace.c @@ -0,0 +1,62 @@ +// nvkmstrace.c - LD_PRELOAD shim: log NVKMS_IOCTL_IOWR commands +#define _GNU_SOURCE +#include <stdio.h> +#include <stdarg.h> +#include <stdint.h> +#include <string.h> +#include <dlfcn.h> +#include <sys/ioctl.h> + +struct nvkms_ioctl_params { + uint32_t cmd; + uint32_t size; + uint64_t address; +}; + +#define NVKMS_REQ 0xC0106D00UL + +static int (*real_ioctl)(int, unsigned long, ...); + +int ioctl(int fd, unsigned long req, ...) +{ + va_list ap; + void *arg; + + va_start(ap, req); + arg = va_arg(ap, void *); + va_end(ap); + if (!real_ioctl) + real_ioctl = dlsym(RTLD_NEXT, "ioctl"); + if (req == NVKMS_REQ && arg) { + struct nvkms_ioctl_params *p = arg; + int rc; + + rc = real_ioctl(fd, req, arg); + fprintf(stderr, "NVKMS_IOCTL cmd=%u size=%u addr=%p rc=%d\n", + p->cmd, p->size, (void *)(uintptr_t)p->address, rc); + if (p->address && p->size) { + unsigned char *q = (unsigned char *)(uintptr_t)p->address; + unsigned n = p->size < 64 ? p->size : 64, i; + + fprintf(stderr, " data:"); + for (i = 0; i < n; i++) + fprintf(stderr, " %02x", q[i]); + fprintf(stderr, "\n"); + if (p->size >= 700) { + /* reply.status candidates: request = 32 + 4 + + * 4*1 (NvBool=NvU8) + 16*(32+4) = 616 bytes */ + unsigned o; + + for (o = 600; o <= 700; o += 4) { + unsigned v; + memcpy(&v, q + o, 4); + if (v <= 16) + fprintf(stderr, " small@%u=%u\n", + o, v); + } + } + } + return rc; + } + return real_ioctl(fd, req, arg); +} diff --git a/tools/guest-diag/pin_gr_stream.py b/tools/guest-diag/pin_gr_stream.py index e1f5195..35c2ea3 100644 --- a/tools/guest-diag/pin_gr_stream.py +++ b/tools/guest-diag/pin_gr_stream.py @@ -32,12 +32,44 @@ LO = os.environ.get("PINWIN_LO", "0x200000000") HI = os.environ.get("PINWIN_HI", "0x206000000") PAMAX = os.environ.get("PINWIN_PA_MAX", "0x180000000") SEMSTREAM = os.environ.get("SEM_STREAM_FILE", "/tmp/opencode/sem_stream.txt") +# Liveness file checked by pin_supervisor.sh (mtime must stay < 75 s; the +# supervisor grants a 120 s startup grace on top). +HEARTBEAT_FILE = os.environ.get("PIN_GR_HEARTBEAT", "/tmp/opencode/pin_gr_heartbeat") +# Inter-tool re-assert request (written by semfix_uvmt.py's stall watchdog): +# when its mtime moves, re-assert every pair this tool has ever pinned. +REASSERT_REQ = os.environ.get("PIN_REASSERT_FILE", + "/tmp/opencode/pin_reassert_request") +# Stall watchdog: a known client with no pin/data activity for this long gets +# an automatic recovery (full re-assert + demand of the fixed counter/pool +# VAs). Pre-capture (no pin yet for this client) uses the tighter limit. +STALL_PRE_S = float(os.environ.get("PIN_STALL_PRE", "60")) +STALL_POST_S = float(os.environ.get("PIN_STALL_POST", "120")) +STALL_RETRY_S = float(os.environ.get("PIN_STALL_RETRY", "30")) +STALL_ESCALATE_S = float(os.environ.get("PIN_STALL_ESCALATE", "180")) def log(msg): print("[%s] %s" % (time.strftime("%H:%M:%S"), msg), flush=True) +def _heartbeat(): + try: + with open(HEARTBEAT_FILE, "w") as f: + f.write("%.3f\n" % time.time()) + except OSError: + pass + + +def _hb_loop(): + """Refresh the heartbeat on a fixed cadence, independent of data-path + activity: recv can be legally idle for minutes between client phases and + the supervisor used to read that silence as a hung tool (r198 restarts at + 34 s staleness while injection bursts were running).""" + while True: + _heartbeat() + time.sleep(5) + + def _mirror(line): """Append guest control lines (#pid/#sem) to the shared host file so the single-QMP watcher can consume them without guest-agent round trips.""" @@ -57,11 +89,73 @@ def vexec(cmd, timeout=40): return "" -def _start_cmd(): +def _scanner_alive(): + """True when the in-guest scanner process still exists. + + install_and_start() pkill's and restarts the scanner, which resets every + in-guest latch; probing first avoids killing a healthy scanner that is + merely silent because no client exists yet (r198 pre-client churn).""" + out = vexec("sudo sh -c 'kill -0 $(cat /tmp/guest_pin_stream.pid) " + "2>/dev/null && echo ALIVE'") + return "ALIVE" in (out or "") + + +def _demand_write_cmd(vas): + """Build a transport-proof append command for the guest demand file. + + r582: the old `printf '%s' '<lines with real LF>'` form was mangled by + the guest-agent transport under load - the newlines arrived as LITERAL + backslash-n text and merged VAs onto one line, and the guest scanner's + parser then crashed on `int('0x20400f000\\\\n0x121000000...', 16)` + (ValueError in main; r581 boot scanner log). The scanner's death stopped + ALL new pins while the host only re-asserted known pairs - the observed + pool-pin-latency (0x121000000 never arrived while other injections + flowed). One `echo <va> >> file` per VA carries no newlines and no + backslashes at all, so no transport layer can escape them.""" + return "; ".join("echo 0x%x >> /tmp/guest_want_vas.txt" % va + for va in vas) + + +def _demand_send(vas): + """Append VA hints to the guest scanner's demand file (best-effort). + + Same channel pin_gr_stream._demand_loop uses for host-dmesg stalls; the + stall watchdog uses it to ask the scanner to re-resolve the fixed + pre-capture VAs (counter page / UVM pool) by content/pagemap.""" + cmd = _demand_write_cmd(vas) + if not cmd: + return + try: + vexec(cmd, 30) + except Exception: + pass + + +def _push_guest_script(): + """Push the scanner into the guest in argv-safe chunks. + + SESSION 153-CONTINUED-64: the one-shot `echo <b64> | base64 -d >` form + put the whole ~135 KB base64 of the (grown) scanner on vgpu_vm.py's + argv, and the host's MAX_ARG_STRLEN (128 KB) rejected the exec before + it ever reached the guest ('OSError: Argument list too long'), so no + scanner started and the boot streamed zero pins. Push 60 KB chunks and + print the guest-side md5 for verification. + """ with open(GUEST_SCRIPT) as f: b64 = subprocess.run(["base64", "-w0"], stdin=f, - capture_output=True, text=True).stdout - prefixes = os.environ.get("CLIENT_PREFIXES", "cdt,va,victory") + capture_output=True, text=True, + timeout=10).stdout.strip() + vexec("rm -f /tmp/gps.b64", 30) + for off in range(0, len(b64), 60000): + vexec("printf %%s '%s' >> /tmp/gps.b64" % b64[off:off + 60000], 30) + return vexec("base64 -d /tmp/gps.b64 > /tmp/guest_pin_stream.py && " + "rm -f /tmp/gps.b64 && md5sum /tmp/guest_pin_stream.py " + "&& echo PUSH_OK", 60) or "" + + +def _start_cmd(): + prefixes = os.environ.get("CLIENT_PREFIXES", + "cdt,va,victory,Xorg,X,glxgears,gzdoom") vma_every = os.environ.get("STREAM_VMA_EVERY", "1") # Kill EVERY existing scanner, not just the pid-file one: a retry after # a partially-failed first attempt (pid file not written yet) used to @@ -75,14 +169,13 @@ def _start_cmd(): # failed silently (r149). return ("P=$(printf '^python3 /tmp/guest_pin_%%s.py' stream); " "sudo pkill -f \"$P\" 2>/dev/null; sleep 1; " - "echo '%s' | base64 -d > /tmp/guest_pin_stream.py; " "sudo rm -f /tmp/guest_pin_stream.pid; " "sudo setsid nohup env CLIENT_PREFIXES='%s' " "STREAM_VMA_EVERY='%s' " "python3 /tmp/guest_pin_stream.py %d %d %s %s %s " ">/tmp/guest_pin_stream.log 2>&1 & sleep 2; " "sudo cat /tmp/guest_pin_stream.pid 2>/dev/null" - % (b64, prefixes, vma_every, 2, PORT, LO, HI, PAMAX)) + % (prefixes, vma_every, 2, PORT, LO, HI, PAMAX)) def install_and_start(): @@ -90,6 +183,15 @@ def install_and_start(): # or a stale qga client); retry a few times instead of giving up, because # a boot without the scanner streams no pins and the client then hangs. for attempt in range(5): + _heartbeat() + try: + out_push = _push_guest_script() + if "PUSH_OK" not in (out_push or ""): + log("WARN: scanner push attempt %d incomplete: %s" + % (attempt + 1, (out_push or "")[-120:])) + except Exception as e: + log("WARN: scanner push attempt %d failed: %s" + % (attempt + 1, e)) out = vexec(_start_cmd(), 60) if any(ch.isdigit() for ch in out): return True @@ -101,8 +203,12 @@ def install_and_start(): def inject(va, pa, ln): cmd = "printf '0x%x,0x%x,0x%x,1' > %s/sema_map_add" % (va, pa, ln, PARAM) for _ in range(60): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=10) + except subprocess.TimeoutExpired: + time.sleep(0.05) + continue if r.returncode == 0: return True time.sleep(0.05) @@ -115,8 +221,12 @@ def scrub_register(fin_pa, hsem_pa): cmd = ("printf '0x%x,0x%x' > %s/sema_scrubbuf_add" % (fin_pa, hsem_pa, PARAM)) for _ in range(60): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=10) + except subprocess.TimeoutExpired: + time.sleep(0.05) + continue if r.returncode == 0: log("scrub registered fin=0x%x hsem=0x%x" % (fin_pa, hsem_pa)) return True @@ -135,112 +245,361 @@ def _demand_loop(): that was not yet written when first asked is picked up on a retry. This closes the stall class where the one-shot #sem kprobe event was lost and the chain resolver never learns the client. + + r908: the GR ring VA itself ('GR ring h=0x... ... (va=0x...)', the + path-B "RAM content scan found no candidate" and "not at fb" rows) is + harvested too - the ring page is a kernel allocation the userspace + pagemap cannot cover, so the guest's content resolver is the only + source, and the pump needs the ring readable to decode its entries. + Same bounded/deduped tracking (distinct VAs, 20 s resend, 300 s + expiry) as the stalled-VA class. """ track = {} # va -> [first_seen, last_sent] + # SESSION 153-CONTINUED-64: strict demand anchors (entry va -> [first, + # last_sent]) from the anchored debugfs lines. The guest validates the + # exact [entry_va, +len) read window with the host's full stream parse + # before pinning, so the setup-push pin can never land on a page whose + # bytes the host's read will reject. Resent fast (2 s) while the hold + # is up; the loose cadence (20 s) is too slow for the client's churn. + strict_track = {} + it = 0 while True: - time.sleep(3) - try: - r = subprocess.run(["sudo", "-n", "dmesg"], - capture_output=True, text=True, timeout=20) - except Exception: - continue + # SESSION 153-CONTINUED-57: the 3 s pass was the losing half of the + # r1096 race - the beef setup-hold's demanded pages resolved in the + # guest ~2-4 s after the arm, while the pump re-read the entry + # every ~110 ms and moved on. Poll fast (0.75 s) so a newly armed + # demand goes to the guest within a second; the expensive dmesg + # harvest stays on a slower cadence (every 4th pass) because the + # debugfs setup-demand set is the time-critical source. + time.sleep(0.75) + it += 1 + r = None + if it % 4 == 1: + try: + r = subprocess.run(["sudo", "-n", "dmesg"], + capture_output=True, text=True, timeout=20) + except Exception: + continue now = time.time() - for m in re.finditer(r"stalled VA #\d+: (0x[0-9a-f]+)", - (r.stdout or "")[-200000:]): - va = int(m.group(1), 16) + for m in re.finditer( + r"(?:stalled VA #\d+[:=]\s*" + r"|pending release starved va=" + r"|GR ring h=0x[0-9a-f]+ rank=\d+[^\n]*?\(va=)" + r"(0x[0-9a-f]+)" + # r914: an unreadable GR push names its pushbuffer VA - + # the GL client's push page is a kernel allocation the + # userspace pagemap cannot cover, so the guest's content + # resolver is the only source (same class as the ring VA + # demand above). Harvest it so the guest resolves and + # injects the page. + r"|sema-pushdec: h=0x[0-9a-f]+ va=(0x[0-9a-f]+)" + r"[^\n]*unreadable=1" + # SESSION 153-CONTINUED-46: a big GL push whose trusted + # chain answered an all-zero page is the beef draw that + # never captures ("sema-pushzero: ... rescued=0"). The + # promote rescue inside the read may heal it; when it + # cannot, the guest content resolver is the last source - + # harvest the VA so it resolves and injects the real page. + r"|sema-pushzero: h=0x[0-9a-f]+ va=(0x[0-9a-f]+)", + (r.stdout or "")[-2000000:] if r else ""): + va = int(m.group(1) or m.group(2) or m.group(3), 16) track.setdefault(va, [now, 0.0]) + # SESSION 153-CONTINUED-51: the pushzero line now carries the + # push length; a zero-head WINDOW push spans 1-2 pages, and the + # demand of a single page leaves the capture zero - expand the + # range and demand every page so the guest pagemap resolves the + # whole push at once (bulk demand instead of one page per cycle). + for m in re.finditer( + r"sema-pushzero: h=0x[0-9a-f]+ va=(0x[0-9a-f]+) len=(\d+)", + (r.stdout or "")[-2000000:] if r else ""): + zva = int(m.group(1), 16) + zlen = int(m.group(2)) + for zoff in range(0, min(zlen, 0x8000), 0x1000): + track.setdefault(zva + zoff, [now, 0.0]) + # SESSION 153-CONTINUED-56: the armed setup-hold's demand VA set is + # exposed by the service as STATE (debugfs), not as a dmesg row, so + # it cannot race the ring rotation under vgsp-sema spam (r1090 lost + # the setup-push pages that way). Read it every pass and feed the + # same track/resend machinery: new VAs are sent this pass (last=0), + # existing ones keep the 20 s resend / 300 s expiry cadence. + try: + rr = subprocess.run( + ["sudo", "-n", "cat", + "/sys/kernel/debug/vgsp/sema_setup_demand"], + capture_output=True, text=True, timeout=10) + if rr.returncode == 0: + for line in (rr.stdout or "").splitlines(): + ma = re.match( + r"\s*0x([0-9a-f]+)\s+0x([0-9a-f]+)\s+(\d+)\s*$", + line) + if ma: + ev = int(ma.group(1), 16) + off = int(ma.group(2), 16) + ln = int(ma.group(3)) + if ev and 0 < ln <= 0x40000: + strict_track.setdefault(ev, [now, 0.0, + off, ln]) + continue + for m in re.finditer(r"0x[0-9a-f]+", line): + sva = int(m.group(0), 16) & ~0xfff + if sva: + track.setdefault(sva, [now, 0.0]) + except Exception: + pass for va in list(track): first, last = track[va] if now - first > 300.0: del track[va] + for ev in list(strict_track): + first, last, off, ln = strict_track[ev] + if now - first > 300.0: + del strict_track[ev] + cmds = [] todo = [va for va, (f, l) in track.items() if now - l > 20.0] - if not todo: + if todo: + todo = todo[:48] + cmds.append(_demand_write_cmd(todo)) + stodo = [ev for ev, (f, l, o, ln) in strict_track.items() + if now - l > 2.0] + if stodo: + stodo = stodo[:16] + cmds.append("; ".join( + "echo S 0x%x 0x%x %d >> /tmp/guest_want_vas.txt" + % (ev, strict_track[ev][2], strict_track[ev][3]) + for ev in stodo)) + if not cmds: continue - lines = "".join("0x%x\n" % va for va in todo) try: - vexec("printf '%%s' '%s' >> /tmp/guest_want_vas.txt" % lines, 30) + vexec("; ".join(cmds), 30) except Exception: continue for va in todo: track[va][1] = now - log("demand: sent %d stalled VA(s) to guest" % len(todo)) + for ev in stodo: + strict_track[ev][1] = now + log("demand: sent %d stalled VA(s)%s to guest" % + (len(todo), + " + %d strict anchor(s)" % len(stodo) if stodo else "")) + + +_DEMAND_STARTED = False +_HB_STARTED = False +_REASSERT_REQ_SEEN = 0.0 def main(): + global _DEMAND_STARTED, _HB_STARTED + _heartbeat() + if not _HB_STARTED: + _HB_STARTED = True + try: + threading.Thread(target=_hb_loop, daemon=True).start() + except Exception as e: + log("WARN: heartbeat thread failed to start: %s" % e) if not install_and_start(): log("WARN: could not start guest scanner (will retry via vsock wait)") - try: - threading.Thread(target=_demand_loop, daemon=True).start() - except Exception as e: - log("WARN: demand thread failed to start: %s" % e) + if not _DEMAND_STARTED: + _DEMAND_STARTED = True + try: + threading.Thread(target=_demand_loop, daemon=True).start() + except Exception as e: + log("WARN: demand thread failed to start: %s" % e) srv = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM) srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) srv.bind((socket.VMADDR_CID_ANY, PORT)) + srv.settimeout(10) srv.listen(4) log("listening on vsock port %d (window %s..%s)" % (PORT, LO, HI)) seen = {} pin_total = 0 - buf = b"" + last_conn = time.time() + last_reinstall = 0.0 + # Stall-watchdog state for the connected client. client_pid is the + # scanner-reported pid (None until a client appears); last_activity marks + # the last pin/data/control event; pins_since_client counts pages pinned + # for the current client (0 = pre-capture). + client_pid = None + client_start = None + last_activity = time.time() + pins_since_client = 0 + stall_since = None + recover_next = 0.0 + escalated = False + no_conn_warn_next = 0.0 + + def _activity(): + nonlocal last_activity, stall_since, escalated + last_activity = time.time() + stall_since = None + escalated = False + while True: try: conn, _ = srv.accept() + except socket.timeout: + now = time.time() + # No guest scanner connection for 30 s: it died or never started + # (a boot without it streams no pins and every client then hangs). + # Probe the scanner first instead of reinstalling blindly: + # install_and_start() pkill's a live scanner and resets all + # in-guest latches, and pre-client silence is normal (r198 churn). + if now - last_conn > 30.0 and now - last_reinstall >= 60.0: + last_reinstall = now + if _scanner_alive(): + log("no guest connection for %.0fs but scanner alive; " + "not reinstalling" % (now - last_conn)) + if now - last_conn > 180.0 and now >= no_conn_warn_next: + no_conn_warn_next = now + 60.0 + log("STALL WATCHDOG: no guest scanner connection for " + "%.0fs (scanner process alive) - pins/demand may " + "be stranded" % (now - last_conn)) + continue + log("reinstall attempt: no guest connection for %.0fs" + % (now - last_conn)) + try: + install_and_start() + except Exception as e: + log("reinstall failed: %s" % e) + continue except OSError as e: log("accept failed: %s" % e) time.sleep(1) continue + last_conn = time.time() + _heartbeat() conn.settimeout(5) log("guest connected") buf = b"" runs = [] - while True: - try: - chunk = conn.recv(1 << 20) - except socket.timeout: - if runs: - _flush(runs, seen) - runs = [] - continue - except OSError: - break - if not chunk: - break - buf += chunk - while b"\n" in buf: - line, buf = buf.split(b"\n", 1) - text = line.decode(errors="replace").strip() - if not text: - continue - if text.startswith("#pid"): - m = re.search(r"(\d+)", text) - seen = {} - log("guest reports cdt pid %s" % (m.group(1) if m else "?")) - _mirror(text) - continue - if text.startswith("#sem"): - _mirror(text) - continue - if text.startswith("#pb"): - _mirror(text) - continue - if text.startswith("#put"): - log(text) - _mirror(text) + try: + while True: + try: + chunk = conn.recv(1 << 20) + except socket.timeout: + if runs: + novel = _novel_pages(runs, seen) + pages = _flush(runs, seen) + pin_total += pages + pins_since_client += pages + if novel: + _activity() + runs = [] + _maybe_reassert_request(seen) + now = time.time() + # Stall watchdog: a known client with no pin/data + # activity gets a bounded, idempotent recovery (re-assert + # everything + demand the fixed counter/pool VAs), then a + # loud escalation if the stall persists. This is the + # "first cudaMalloc / first GR capture must not stall + # silently" guard. recover_next rate-limits attempts (the + # loop wakes every 5 s on the recv timeout). + if client_pid is not None: + limit = STALL_PRE_S if pins_since_client == 0 \ + else STALL_POST_S + idle = now - last_activity + if idle >= limit and now >= recover_next: + recover_next = now + STALL_RETRY_S + if stall_since is None: + stall_since = now + log("STALL WATCHDOG: client pid %d - no pin " + "activity for %.0fs (pre-capture=%s) - " + "re-asserting all known pairs + demanding " + "counter/pool VAs" + % (client_pid, idle, pins_since_client == 0)) + if seen: + _flush([], seen, full=True) + _demand_send([0x20400f000, 0x121000000]) + if not escalated and stall_since is not None \ + and now - stall_since >= STALL_ESCALATE_S: + escalated = True + log("STALL WATCHDOG: recovery FAILED for pid %d " + "after %.0fs - stall persists, needs manual " + "attention" % (client_pid, now - stall_since)) continue - if text.startswith("#scrub"): - m = re.search(r"0x([0-9a-fA-F]+),0x([0-9a-fA-F]+)", text) - _mirror(text) + except OSError: + break + if not chunk: + break + _heartbeat() + buf += chunk + while b"\n" in buf: + line, buf = buf.split(b"\n", 1) + text = line.decode(errors="replace").strip() + if not text: + continue + if text.startswith("#pid"): + m = re.match(r"#pid\s+(\d+)(?:\s+(\d+))?", text) + seen = {} + client_pid = int(m.group(1)) if m else None + client_start = (int(m.group(2)) if m and m.group(2) + else None) + pins_since_client = 0 + _activity() + log("guest reports cdt pid %s%s" + % (client_pid if client_pid is not None else "?", + (" start=%s" % client_start + if client_start is not None else ""))) + _mirror(text) + continue + if text.startswith("#nopid"): + # In-band client death: the guest scanner tells us the + # client is gone (with its identity). Drop the per-pair + # dedupe map so the next client's pairs are injected + # unconditionally; semfix_uvmt consumes the mirrored + # line and purges its per-client state too. + client_pid = None + client_start = None + seen = {} + pins_since_client = 0 + _activity() + log("guest reports client gone (%s)" % text) + _mirror(text) + continue + if text.startswith("#sem"): + _mirror(text) + _activity() + continue + if text.startswith("#pb"): + _mirror(text) + _activity() + continue + if text.startswith("#put"): + log(text) + _mirror(text) + _activity() + continue + if text.startswith("#scrub"): + m = re.search(r"0x([0-9a-fA-F]+),0x([0-9a-fA-F]+)", + text) + _mirror(text) + if m: + scrub_register(int(m.group(1), 16), + int(m.group(2), 16)) + continue + m = re.match(r"^(0x[0-9a-f]+),(0x[0-9a-f]+)(?:,(\d+))?$", + text) if m: - scrub_register(int(m.group(1), 16), - int(m.group(2), 16)) - continue - m = re.match(r"^(0x[0-9a-f]+),(0x[0-9a-f]+)(?:,(\d+))?$", text) - if m: - runs.append((int(m.group(1), 16), int(m.group(2), 16), - int(m.group(3)) if m.group(3) else 1)) - if runs: - pin_total += _flush(runs, seen) - runs = [] + runs.append((int(m.group(1), 16), + int(m.group(2), 16), + int(m.group(3)) if m.group(3) else 1)) + if runs: + novel = _novel_pages(runs, seen) + pages = _flush(runs, seen) + pin_total += pages + pins_since_client += pages + if novel: + _activity() + runs = [] + # Poll the semfix re-assert request on every recv cycle, not + # only on recv-timeouts: sustained scanner traffic starves the + # timeout branch, so the request fired post-mortem in r203 + # (119 s late, `seen` already cleared by #nopid). + _maybe_reassert_request(seen) + except Exception as e: + # A transient error on one connection must not kill the process: + # log it, drop the connection and go back to accept(). + log("connection handler error: %s (continuing)" % e) log("guest disconnected (pins this session: %d)" % pin_total) try: conn.close() @@ -248,27 +607,82 @@ def main(): pass -def _flush(runs, seen): +def _maybe_reassert_request(seen): + """Honour a full-re-assert request written by semfix_uvmt's watchdog.""" + global _REASSERT_REQ_SEEN + try: + m = os.stat(REASSERT_REQ).st_mtime + except OSError: + return + if m > _REASSERT_REQ_SEEN: + _REASSERT_REQ_SEEN = m + if seen: + log("semfix requested a full pin re-assert - re-asserting %d " + "pair(s)" % len(seen)) + _flush([], seen, full=True) + + +def _coalesce_seen(seen): + """Coalesce the seen {va: pa} per-page map into (va, pa, npages) runs.""" + items = sorted(seen.items()) + runs = [] + i = 0 + while i < len(items): + va, pa = items[i] + n = 1 + while (i + n < len(items) + and items[i + n][0] == va + n * 0x1000 + and items[i + n][1] == pa + n * 0x1000): + n += 1 + runs.append((va, pa, n)) + i += n + return runs + + +def _novel_pages(runs, seen): + """Count pages in runs whose translation is new or changed. + + Progress accounting for the stall watchdog: the guest re-streams the CE + window and UVM pool every couple of seconds even when nothing moves, so a + pure repeat of an already-known pair must not count as pin activity + (r203: those repeats masked the watchdog through every hang).""" + return sum(1 for va, pa, n in runs for i in range(n) + if seen.get(va + i * 0x1000) != pa + i * 0x1000) + + +def _flush(runs, seen, full=False): """Inject run lines (one batched sudo pass), return number of pages pinned. The old implementation spawned one sudo process per run; with ~400 runs that cost ~4 s, which landed the GR pins after the client's context-init deadline. The batch writes all runs in a single shell, retrying only the ones the kernel queue rejected (ENOSPC) individually. + + full=True re-asserts every pair ever pinned (coalesced), not just this + batch's runs: the kernel session flush wipes the pin table when a client's + channels close, and the stall watchdog / semfix re-assert request use this + to heal it without touching the guest. Writes are idempotent. """ - todo = [] + todo = {} for va, pa, n in runs: - # The CE window (0x120000000..0x121000000) is re-asserted by the + # The CE window (0x120000000..0x121000000) and the UVM semaphore + # pool page range (0x121000000..0x122000000) are re-asserted by the # guest every couple of seconds on purpose: the host's session flush # wipes the whole pin table when the previous client's last channel # closes, and a pin batch that raced in for the new client must be # allowed to land again even when the pair is unchanged. Skipping # repeats there stranded the second client's CE pushes (the r144/ - # r145 stall: sema-probe "NO ENTRY" for the live push pages). - ce_win = 0x120000000 <= va < 0x121000000 - if not ce_win and seen.get(va) == pa and n == 1: + # r145 stall: sema-probe "NO ENTRY" for the live push pages) and the + # pool/tracker page (pending release starved, every client's first + # seconds). + ce_win = 0x120000000 <= va < 0x122000000 + if not full and not ce_win and seen.get(va) == pa and n == 1: continue - todo.append((va, pa, n)) + todo[(va, pa, n)] = True + if full: + for va, pa, n in _coalesce_seen(seen): + todo[(va, pa, n)] = True + todo = list(todo) if not todo: return 0 @@ -286,6 +700,7 @@ def _flush(runs, seen): CHUNK = 200 for start in range(0, len(script), CHUNK): batch = script[start:start + CHUNK] + _heartbeat() try: r = subprocess.run(["sudo", "-n", "sh", "-c", "\n".join(batch)], capture_output=True, text=True, timeout=60) @@ -319,4 +734,12 @@ def _flush(runs, seen): if __name__ == "__main__": - main() + # main() loops forever; only an unexpected exception exits it. Restart + # instead of dying silently - a dead listener means no pins and every + # later CUDA client hangs at first cudaMalloc. + while True: + try: + main() + except Exception as e: + log("FATAL: main() exited: %r - restarting in 2s" % (e,)) + time.sleep(2) diff --git a/tools/guest-diag/pin_supervisor.sh b/tools/guest-diag/pin_supervisor.sh new file mode 100755 index 0000000..72f588f --- /dev/null +++ b/tools/guest-diag/pin_supervisor.sh @@ -0,0 +1,176 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2026 Open-IOV contributors +# pin_supervisor.sh - supervise the host-side pin tools (pin_gr_stream.py and +# semfix_uvmt.py) so a silent death cannot strand every later CUDA client at +# first cudaMalloc. +# +# Both tools write a heartbeat file; this script checks, every 5 s, that each +# process is alive (exact script path) and that its heartbeat mtime is < 75 s. +# A startup grace (120 s, skipped early once the tool log shows it has done +# first useful work) keeps a slow cold start from being restarted. On failure +# it kills the tool (exact path match) and restarts it, appending RESTART +# lines to the supervisor log. After 10 restarts in one hour for a tool it +# logs GIVEUP and exits 1. Only one semfix_uvmt instance is ever started +# (QEMU's QMP socket accepts a single client). +# +# Liveness policy (do not tighten without re-checking the soak evidence): +# * the tools refresh their heartbeats from dedicated loops/threads even +# while idle (pin_gr_stream: 5 s thread; semfix: per loop + around every +# blocking guest-agent/dmesg call), so 75 s never trips on a healthy +# client between phases; +# * a DEAD process is restarted on the next 5 s tick regardless of grace; +# * dead-pid/state recovery (the kill cascade) is handled INSIDE the tools +# (in-band #nopid + idle liveness probe + stall watchdog), NOT by this +# supervisor - a log-based restart here could fire mid-run of a healthy +# client (the m178 lesson). +# +# Usage: pin_supervisor.sh LOGDIR TAG +# Logs: LOGDIR/12-pin-supervisor-TAG.log (supervisor) +# LOGDIR/12-pin_gr_stream-TAG.log, LOGDIR/12-semfix-TAG.log (tools) +set -uo pipefail + +LOGDIR="${1:?usage: pin_supervisor.sh LOGDIR TAG}" +TAG="${2:?usage: pin_supervisor.sh LOGDIR TAG}" + +HERE=$(cd "$(dirname "$0")" && pwd) +PIN="$HERE/pin_gr_stream.py" +SEMFIX="$HERE/semfix_uvmt.py" +PIN_HB="${PIN_GR_HEARTBEAT:-/tmp/opencode/pin_gr_heartbeat}" +SEMFIX_HB="${SEMFIX_HEARTBEAT_FILE:-/tmp/opencode/semfix_heartbeat}" +SUPLOG="$LOGDIR/12-pin-supervisor-$TAG.log" +PIN_LOG="$LOGDIR/12-pin_gr_stream-$TAG.log" +SEMFIX_LOG="$LOGDIR/12-semfix-$TAG.log" +WINDOW=3600 +CAP=10 +STALE=75 +GRACE=120 +mkdir -p "$LOGDIR" /tmp/opencode + +log() { echo "[$(date +%H:%M:%S)] $*" | tee -a "$SUPLOG"; } + +tool_alive() { pgrep -f "python3 $1" >/dev/null 2>&1; } + +hb_age() { # age in seconds of the heartbeat file, 9999 if missing + local f="$1" m + m=$(stat -c %Y "$f" 2>/dev/null) || { echo 9999; return; } + echo $(( $(date +%s) - m )) +} + +kill_tool() { + pkill -f "python3 $1" 2>/dev/null + sleep 1 + if pgrep -f "python3 $1" >/dev/null 2>&1; then + pkill -9 -f "python3 $1" 2>/dev/null + sleep 1 + fi + return 0 +} + +start_pin() { + if tool_alive "$PIN"; then return 0; fi + touch "$PIN_HB" 2>/dev/null || true + # CLIENT_PREFIXES threads through pin_gr_stream.py into the guest + # scanner launch (its _start_cmd). GL clients (glxgears, gzdoom) and + # the X server must be covered or the GL client's ring/mappings are + # never pinned (SESSION 149: the GR ring scan then finds no candidate + # for h=0xbfef0002, va=0x1203cfff0). + ( setsid nohup env \ + CLIENT_PREFIXES="${CLIENT_PREFIXES:-cdt,va,victory,Xorg,X,glxgears,gzdoom}" \ + SEMFIX_COUNTER_RETRY=10 \ + python3 "$PIN" >> "$PIN_LOG" 2>&1 & ) + pin_started=$(date +%s) + log "START pin_gr_stream" +} + +start_semfix() { + # pgrep exact-count check before start: if an instance already exists + # (ours or a leftover), never start a second one. + if tool_alive "$SEMFIX"; then + log "semfix already running - not starting another" + return 0 + fi + touch "$SEMFIX_HB" 2>/dev/null || true + ( setsid nohup env SEMFIX_SKIP_PB=0 SEMFIX_COUNTER_RETRY=10 \ + python3 "$SEMFIX" 0.5 >> "$SEMFIX_LOG" 2>&1 & ) + semfix_started=$(date +%s) + log "START semfix_uvmt" +} + +prune() { # $1 now, $2 list -> list with entries older than WINDOW dropped + local now="$1" t out="" + for t in $2; do + [ $(( now - t )) -lt "$WINDOW" ] && out="$out $t" + done + printf '%s' "$out" +} + +count_list() { # $1 list -> element count + local n=0 t + for t in $1; do n=$(( n + 1 )); done + echo "$n" +} + +restarts_pin="" +restarts_semfix="" +pin_started=$(date +%s) +semfix_started=$(date +%s) + +restart() { # $1 label, $2 script path, $3 start function + local now list n + now=$(date +%s) + if [ "$1" = pin_gr_stream ]; then + restarts_pin=$(prune "$now" "$restarts_pin") + n=$(count_list "$restarts_pin") + else + restarts_semfix=$(prune "$now" "$restarts_semfix") + n=$(count_list "$restarts_semfix") + fi + if [ "$n" -ge "$CAP" ]; then + log "GIVEUP $1: $n restarts within ${WINDOW}s" + exit 1 + fi + kill_tool "$2" + if [ "$1" = pin_gr_stream ]; then + restarts_pin="$restarts_pin $now" + else + restarts_semfix="$restarts_semfix $now" + fi + log "RESTART $(( n + 1 )) $1 $(date +%Y-%m-%dT%H:%M:%S)" + "$3" +} + +check_tool() { # $1 label, $2 path, $3 heartbeat file, $4 start function, + # $5 tool start TS, $6 tool log file, $7 first-work pattern + local age runtime + age=$(hb_age "$3") + if tool_alive "$2" && [ "$age" -le "$STALE" ]; then + return 0 + fi + if tool_alive "$2"; then + # Startup grace: a cold tool can spend its first ~2 min installing the + # guest scanner or connecting QMP with a lagging heartbeat; only + # enforce staleness once the grace has passed or the log shows the + # tool has already done successful work. + runtime=$(( $(date +%s) - $5 )) + if [ "$runtime" -lt "$GRACE" ] && \ + ! grep -q -- "$7" "$6" 2>/dev/null; then + return 0 + fi + log "$1 heartbeat stale (${age}s) - restarting" + else + log "$1 not running (heartbeat age ${age}s) - restarting" + fi + restart "$1" "$2" "$4" +} + +log "pin_supervisor up (LOGDIR=$LOGDIR TAG=$TAG)" +start_pin +start_semfix +while true; do + sleep 5 + check_tool pin_gr_stream "$PIN" "$PIN_HB" start_pin "$pin_started" \ + "$PIN_LOG" "listening on vsock" + check_tool semfix_uvmt "$SEMFIX" "$SEMFIX_HB" start_semfix \ + "$semfix_started" "$SEMFIX_LOG" "semfix_uvmt: watching" +done diff --git a/tools/guest-diag/pool_page.py b/tools/guest-diag/pool_page.py new file mode 100755 index 0000000..a9f7b15 --- /dev/null +++ b/tools/guest-diag/pool_page.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +# pool_page.py <pa-hex> - dump the UVM semaphore-pool page's u32 slots via +# /proc/kcore (guest root). The pool VA 0x121000000 maps to the PA the +# scanner+host pinned; nonzero slots are the applied release values. Compare +# against the tracker's expected payload (tracker_walk.py) and the driver's +# uvsv wait value to find write-target divergence / starvation. +import sys +import struct + +pa = int(sys.argv[1], 16) if len(sys.argv) > 1 else 0x107fd8000 +sys.argv = ["pool_page"] +sys.path.insert(0, "/tmp") +import guest_pin_stream as g # noqa: E402 + +kc = g.Kcore() +if not kc.load(): + print("KCORE LOAD FAILED") + sys.exit(1) + +nz = [] +for off in range(0, 0x1000, 4): + d = kc.read(kc.page_offset_base + pa + off, 4) + v = struct.unpack("<I", d)[0] if d else 0 + if v: + nz.append((off, v)) +print("pa=0x%x nonzero slots=%d" % (pa, len(nz))) +for off, v in nz[:64]: + print("off=0x%x val=%u" % (off, v)) diff --git a/tools/guest-diag/qga_fetch.py b/tools/guest-diag/qga_fetch.py new file mode 100755 index 0000000..5a940fc --- /dev/null +++ b/tools/guest-diag/qga_fetch.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# Fetch a file from the guest via the QEMU guest agent protocol +# (/tmp/qemu-guest-agent.sock, the virtio-serial QGA chardev - the same +# socket vgpu_vm.py exec uses). Usage: qga_fetch.py <guest_path> <host_path> [chunk] +import base64 +import json +import socket +import sys + +GUEST = sys.argv[1] +HOST = sys.argv[2] +CHUNK = int(sys.argv[3]) if len(sys.argv) > 3 else 1 << 20 +SOCK = "/tmp/qemu-guest-agent.sock" + + +def recv_line(s): + buf = b"" + while b"\n" not in buf: + chunk = s.recv(65536) + if not chunk: + break + buf += chunk + return buf.split(b"\n", 1)[0] + + +def cmd(s, obj): + s.sendall((json.dumps(obj) + "\r\n").encode()) + line = recv_line(s) + return json.loads(line) + + +s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) +s.settimeout(30.0) +s.connect(SOCK) +r = cmd(s, {"execute": "guest-file-open", + "arguments": {"path": GUEST, "mode": "r"}}) +h = r["return"] +print(f"handle={h}") +total = 0 +with open(HOST, "wb") as out: + while True: + r = cmd(s, {"execute": "guest-file-read", + "arguments": {"handle": h, "count": CHUNK}}) + d = r["return"] + buf = base64.b64decode(d.get("buf-b64", "")) + if buf: + out.write(buf) + total += len(buf) + print(f"\r{total}", end="", flush=True) + if d.get("eof"): + break +cmd(s, {"execute": "guest-file-close", "arguments": {"handle": h}}) +print(f"\nFETCHED {total} bytes -> {HOST}") diff --git a/tools/guest-diag/qga_snip.py b/tools/guest-diag/qga_snip.py new file mode 100644 index 0000000..cd3b992 --- /dev/null +++ b/tools/guest-diag/qga_snip.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +# Fetch a byte range from a guest file via the QEMU guest agent. +# Usage: qga_snip.py <guest_path> <offset> <length> <host_out> +import base64 +import json +import socket +import sys + +GUEST, OFF, LEN, HOST = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), sys.argv[4] +import os +SOCK = os.environ.get("QGA_SOCK", "/tmp/qemu-guest-agent.sock") + + +def recv_line(s): + buf = b"" + while b"\n" not in buf: + c = s.recv(65536) + if not c: + break + buf += c + return buf.split(b"\n", 1)[0] + + +def cmd(s, obj): + s.sendall((json.dumps(obj) + "\r\n").encode()) + return json.loads(recv_line(s)) + + +s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) +s.settimeout(30.0) +s.connect(SOCK) +h = cmd(s, {"execute": "guest-file-open", + "arguments": {"path": GUEST, "mode": "r"}})["return"] +cmd(s, {"execute": "guest-file-seek", + "arguments": {"handle": h, "offset": OFF, "whence": 0}}) +r = cmd(s, {"execute": "guest-file-read", + "arguments": {"handle": h, "count": LEN}}) +data = base64.b64decode(r["return"].get("buf-b64", "")) +open(HOST, "wb").write(data) +cmd(s, {"execute": "guest-file-close", "arguments": {"handle": h}}) +print(f"GOT {len(data)} bytes @ {OFF} -> {HOST}") diff --git a/tools/guest-diag/ram_content_pin.py b/tools/guest-diag/ram_content_pin.py new file mode 100755 index 0000000..d521d91 --- /dev/null +++ b/tools/guest-diag/ram_content_pin.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Host-side CONTENT-based resolver for demanded GR setup-push windows. + +SESSION 153-CONTINUED-66. The client's GL setup pushes (and its 0x3200000 / +0x6f00000 macro-upload windows) live in allocations that have NO kernel vmap +and NO client CPU mapping on the failing boots: the guest scanner's area+idx +identity sweep and its client-mem sweep both fail (`strict unknown-family +client-mem failed`, `strict parse failed (N B, M windows tried)`), so the +armed GR setup hold never lands its pin and the MME prelude stays 0. + +The bytes ARE in guest RAM - a RAM-wide pattern scan finds the MME/SA +prelude runs (`0xa0100045` = INC 0x114 x16 args, `0x20020047` = INC 0x11c x2 +args). This tool therefore resolves a demanded window from CONTENT instead of +identity: + + 1. read the demand set the service publishes (debugfs sema_setup_demand: + "0xVA 0xOFF LEN" per line); + 2. locate every physical page in the guest RAM image that carries an + MME/SA prelude run signature; + 3. for each candidate physical page P and each window page index j, read + the EXACT window bytes the host would read - [P - j*0x1000 + OFF, +LEN) + - straight out of the RAM image and run the host's own method-stream + acceptance on them; + 4. inject the validated page run into the service through the + sema_map_add module parameter (0xVA,0xPA,LEN,1), so the host read + resolves the demanded window from real content. + +Nothing is injected unless the host's parser accepts the exact bytes, so a +wrong candidate cannot mis-resolve a demand VA (the r1120 lesson). + +Usage: ram_content_pin.py [ram-image] [seconds] +""" +import importlib.util +import mmap +import os +import re +import subprocess +import sys +import time + +import numpy as np + +RAM = sys.argv[1] if len(sys.argv) > 1 else '/dev/shm/vgpu-guest-ram' +SECS = float(sys.argv[2]) if len(sys.argv) > 2 else 120.0 +DEMAND = '/sys/kernel/debug/vgsp/sema_setup_demand' +MAPADD = '/sys/module/vgsp/parameters/sema_map_add' +GPS = ('/home/rmcore/Git/integdev_gpu_drv/open-iov/tools/guest-diag/' + 'guest_pin_stream.py') +SIGS = (0xa0100045, 0x20020047) # INC 0x114 x16, INC 0x11c x2 +PAGE = 0x1000 +MAXHITS = 4096 + + +def log(msg): + print('[%s] %s' % (time.strftime('%H:%M:%S'), msg), flush=True) + + +def load_accept(): + spec = importlib.util.spec_from_file_location('gps', GPS) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod._window_accept + + +def read_demand(): + out = [] + try: + with open(DEMAND) as f: + for ln in f: + m = re.match(r'0x([0-9a-fA-F]+)\s+0x([0-9a-fA-F]+)\s+(\d+)', + ln.strip()) + if not m: + continue + out.append((int(m.group(1), 16), int(m.group(2), 16), + int(m.group(3)))) + except OSError: + pass + return out + + +def inject(va_page, pa_page, ln): + cmd = "printf '0x%x,0x%x,0x%x,1' > %s" % (va_page, pa_page, ln, MAPADD) + try: + r = subprocess.run(['sudo', '-n', 'sh', '-c', cmd], + capture_output=True, text=True, timeout=10) + except subprocess.TimeoutExpired: + return False + return r.returncode == 0 + + +def main(): + accept = load_accept() + f = open(RAM, 'rb') + mm = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) + size = len(mm) + a32 = np.frombuffer(mm, dtype=np.uint32) + hits = [] + for sig in SIGS: + idx = np.flatnonzero(a32 == sig) + log('sig 0x%08x: %d hit(s)' % (sig, idx.size)) + for i in idx[:MAXHITS]: + hits.append(int(i) * 4) + del a32 + hits = sorted(set(h >> 12 for h in hits)) + log('candidate physical pages: %d (ram image %d MB)' + % (len(hits), size >> 20)) + if not hits: + return 1 + seen = set() + t0 = time.time() + while time.time() - t0 < SECS: + dem = read_demand() + for (va, off, ln) in dem: + key = (va & ~(PAGE - 1), ln) + if key in seen: + continue + if ln <= 0 or ln > 0x40000: + continue + need = (off + ln + PAGE - 1) // PAGE + if need > 64: + continue + va_page = va & ~(PAGE - 1) + found = None + for P in hits: + for j in range(need): + P0 = P - j * PAGE + if P0 < 0 or P0 + off + ln > size: + continue + buf = mm[P0 + off:P0 + off + ln] + try: + ok, mode = accept(buf, ln) + except Exception: + ok = False + if ok: + found = (P0, mode) + break + if found: + break + if not found: + log('demand va=0x%x off=0x%x len=%d: no content candidate' + % (va, off, ln)) + seen.add(key) + continue + P0, mode = found + if inject(va_page, P0, need * PAGE): + log('demand va=0x%x off=0x%x len=%d -> pa=0x%x (%s parse, ' + 'pages=%d) INJECTED' + % (va, off, ln, P0, mode, need)) + seen.add(key) + else: + log('demand va=0x%x: injection refused by the module param' + % va) + seen.add(key) + time.sleep(1.0) + log('done') + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/guest-diag/read_proc_va.py b/tools/guest-diag/read_proc_va.py new file mode 100644 index 0000000..c035840 --- /dev/null +++ b/tools/guest-diag/read_proc_va.py @@ -0,0 +1,16 @@ +import os, sys +pat = sys.argv[1] +vas = [int(a, 16) for a in sys.argv[2:]] +pids = os.popen("pgrep -x " + pat).read().split() +if not pids: + print("NO-PID"); sys.exit(0) +pid = int(pids[0]) +print("pid", pid) +fd = os.open("/proc/%d/mem" % pid, os.O_RDONLY) +for a in vas: + try: + os.lseek(fd, a, 0) + d = os.read(fd, 4) + print(hex(a), d.hex()) + except OSError as e: + print(hex(a), "ERR", e) diff --git a/tools/guest-diag/redboot-triage.sh b/tools/guest-diag/redboot-triage.sh new file mode 100755 index 0000000..041e9a6 --- /dev/null +++ b/tools/guest-diag/redboot-triage.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# redboot-triage.sh <live-log> [base-va] - pull the r613b/r613d wedge-class +# evidence from a boot's dmesg capture in one pass. +# usage: redboot-triage.sh /tmp/opencode/rNNN-live.log [0x7fc716c00000] +set -u +LOG="${1:?usage: redboot-triage.sh <log> [base-va]}" +VA="${2:-}" +echo "=== $LOG ===" +echo "--- sema-shadow-dst (r614 witness; the red-boot entry point)" +grep -h "sema-shadow-dst" "$LOG" | head -12 +echo "--- GRFILL-DROP + held-for-retry + degenerate refusal" +grep -hE "GRFILL-DROP|launch held for a bounded retry|GR submit refused|GR forward failed|GR-STALL-fwd-fail|GR-COMPLETE-SW" "$LOG" | tail -24 +echo "--- arg-miss walk lists (r614: full, untruncated)" +grep -h "arg-miss" "$LOG" | tail -6 +echo "--- cand[0] arg base probes (first resolution of each client)" +grep -h "cand\[0\] arg base" "$LOG" | tail -12 +echo "--- EPOST drop signature (status-ring starvation rounds)" +grep -h "no postable event" "$LOG" | tail -6 +if [ -n "$VA" ]; then + echo "--- all lines for base $VA" + grep -h "$VA" "$LOG" | grep -vE "ldatrace|dmatrace" | head -30 +fi +echo "=== done ==="
\ No newline at end of file diff --git a/tools/guest-diag/sdl_cap.c b/tools/guest-diag/sdl_cap.c new file mode 100644 index 0000000..23c83ab --- /dev/null +++ b/tools/guest-diag/sdl_cap.c @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: BSD-3-Clause +// sdl_cap.c - LD_PRELOAD capture shim: records every frame GZDoom presents +// through SDL into an append-only file, so the host can harvest the guest's +// real rendered frames without any X/driver readback path. +// +// Hooks: +// SDL_RenderPresent(SDL_Renderer*) -> SDL_RenderReadPixels(ARGB8888) +// SDL_UpdateWindowSurface(SDL_Window*) -> SDL_GetWindowSurface pixels +// Both forward to the real SDL after capturing. +// +// File layout (little endian): +// [0] 8B magic "GZCAPV01" (+ 4088 B reserved zero) +// [4096] frame records: +// 4B "GZF2" | u32 w | u32 h | u32 stride | u32 bpp +// u64 ts_ns (CLOCK_MONOTONIC) | u64 seq | stride*h bytes (top-down) +// A trailing record with seq = 0xFFFFFFFFFFFFFFFF marks clean stop. +// +// Env: GZCAP (path, default /dev/shm/gzframes.bin), GZCAP_MAXF (0=unlimited) +#define _GNU_SOURCE +#include <dlfcn.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include <unistd.h> + +#define ARGB8888 0x16362004u + +typedef struct SDL_Renderer SDL_Renderer; +typedef struct SDL_Window SDL_Window; + +typedef struct SDL_Surface { + uint32_t flags; + void *format; + int w, h; + int pitch; + void *pixels; + void *userdata; + int locked; + void *list_blitmap; + struct { int x, y, w, h; } clip_rect; + void *map; + int refcount; +} SDL_Surface; + +static int (*real_RenderReadPixels)(SDL_Renderer *, const void *, uint32_t, + void *, int); +static int (*real_RenderPresent)(SDL_Renderer *); +static SDL_Surface *(*real_GetWindowSurface)(SDL_Window *); +static int (*real_UpdateWindowSurface)(SDL_Window *); +static int (*real_GetRendererOutputSize)(SDL_Renderer *, int *, int *); +static void (*real_GL_SwapWindow)(SDL_Window *); +static int (*real_GL_GetDrawableSize)(SDL_Window *, int *, int *); +static void (*real_glReadPixels)(int, int, int, int, unsigned, unsigned, void *); + +static FILE *cap_fp; +static unsigned long long cap_count, cap_max = 0; +static int cap_stop; + +__attribute__((constructor)) static void cap_init(void) +{ + const char *p = getenv("GZCAP"); + const char *m = getenv("GZCAP_MAXF"); + char hdr[4096]; + + if (m) + cap_max = strtoull(m, NULL, 0); + if (!p) + p = "/dev/shm/gzframes.bin"; + real_RenderReadPixels = dlsym(RTLD_NEXT, "SDL_RenderReadPixels"); + real_RenderPresent = dlsym(RTLD_NEXT, "SDL_RenderPresent"); + real_GetWindowSurface = dlsym(RTLD_NEXT, "SDL_GetWindowSurface"); + real_UpdateWindowSurface = dlsym(RTLD_NEXT, "SDL_UpdateWindowSurface"); + real_GetRendererOutputSize = + dlsym(RTLD_NEXT, "SDL_GetRendererOutputSize"); + real_GL_SwapWindow = dlsym(RTLD_NEXT, "SDL_GL_SwapWindow"); + real_GL_GetDrawableSize = dlsym(RTLD_NEXT, "SDL_GL_GetDrawableSize"); + real_glReadPixels = dlsym(RTLD_NEXT, "glReadPixels"); + if (!real_glReadPixels) + real_glReadPixels = dlsym(RTLD_DEFAULT, "glReadPixels"); + if (!real_glReadPixels) { + void *h = dlopen("libGL.so.1", RTLD_NOW | RTLD_GLOBAL); + + if (!h) + h = dlopen("libGLESv2.so.2", RTLD_NOW | RTLD_GLOBAL); + if (h) + real_glReadPixels = dlsym(h, "glReadPixels"); + } + fprintf(stderr, "[gzcap] init path=%s present=%p updwin=%p swap=%p read=%p gros=%p\n", + p, (void *)real_RenderPresent, (void *)real_UpdateWindowSurface, + (void *)real_GL_SwapWindow, (void *)real_RenderReadPixels, + (void *)real_GetRendererOutputSize, (void *)real_glReadPixels); + cap_fp = fopen(p, "wb"); + if (!cap_fp) + return; + memset(hdr, 0, sizeof(hdr)); + memcpy(hdr, "GZCAPV01", 8); + fwrite(hdr, 1, sizeof(hdr), cap_fp); + fflush(cap_fp); +} + +static void cap_record(const void *px, int w, int h, int stride, int bpp) +{ + struct timespec ts; + uint64_t tns, seq; + uint32_t rh[4]; + char m[4] = { 'G', 'Z', 'F', '2' }; + + if (!cap_fp || cap_stop) + return; + if (cap_max && cap_count >= cap_max) { + uint64_t endseq = 0xFFFFFFFFFFFFFFFFull; + + fwrite(m, 1, 4, cap_fp); + fwrite(&endseq, 8, 1, cap_fp); + fflush(cap_fp); + cap_stop = 1; + return; + } + clock_gettime(CLOCK_MONOTONIC, &ts); + tns = (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec; + seq = cap_count; + rh[0] = (uint32_t)w; + rh[1] = (uint32_t)h; + rh[2] = (uint32_t)stride; + rh[3] = (uint32_t)bpp; + fwrite(m, 1, 4, cap_fp); + fwrite(rh, 4, 4, cap_fp); + fwrite(&tns, 8, 1, cap_fp); + fwrite(&seq, 8, 1, cap_fp); + fwrite(px, 1, (size_t)stride * h, cap_fp); + fflush(cap_fp); + cap_count++; + if ((cap_count & 31) == 0) + fprintf(stderr, "[gzcap] frames=%llu\n", cap_count); +} + +int SDL_RenderPresent(SDL_Renderer *r) +{ + static int hit; + + if (!hit++) + fprintf(stderr, "[gzcap] hook RenderPresent\n"); + if (cap_fp && !cap_stop && real_RenderReadPixels && + real_GetRendererOutputSize) { + int w = 0, h = 0; + + if (real_GetRendererOutputSize(r, &w, &h) == 0 && w > 0 && h > 0 && + w <= 4096 && h <= 4096) { + static unsigned char *buf; + static size_t bufsz; + size_t need = (size_t)w * 4 * h; + + if (bufsz < need) { + free(buf); + buf = malloc(need); + bufsz = buf ? need : 0; + } + if (buf && real_RenderReadPixels(r, NULL, ARGB8888, buf, + w * 4) == 0) + cap_record(buf, w, h, w * 4, 32); + } + } + if (real_RenderPresent) + return real_RenderPresent(r); + return 0; +} + +int SDL_UpdateWindowSurface(SDL_Window *win) +{ + static int hit; + + if (!hit++) + fprintf(stderr, "[gzcap] hook UpdateWindowSurface\n"); + if (cap_fp && !cap_stop && real_GetWindowSurface) { + SDL_Surface *s = real_GetWindowSurface(win); + + if (s && s->pixels && s->w > 0 && s->h > 0 && s->pitch > 0 && + s->w <= 4096 && s->h <= 4096) + cap_record(s->pixels, s->w, s->h, s->pitch, 32); + } + if (real_UpdateWindowSurface) + return real_UpdateWindowSurface(win); + return 0; +} + +void SDL_GL_SwapWindow(SDL_Window *win) +{ + static int hit; + static unsigned char *gbuf; + static size_t gbufsz; + + if (!hit++) + fprintf(stderr, "[gzcap] hook GLSwapWindow\n"); + if (cap_fp && !cap_stop && real_glReadPixels && real_GL_GetDrawableSize) { + int w = 0, h = 0; + + if (real_GL_GetDrawableSize(win, &w, &h) == 0 && w > 0 && h > 0 && + w <= 4096 && h <= 4096) { + size_t need = (size_t)w * 4 * h; + + if (gbufsz < need) { + free(gbuf); + gbuf = malloc(need); + gbufsz = gbuf ? need : 0; + } + if (gbuf) { + real_glReadPixels(0, 0, w, h, 0x1908, 0x1401, gbuf); + /* GL rows are bottom-up: flip into a scratch record */ + { + unsigned char *tmp = malloc(need); + int y; + + if (tmp) { + for (y = 0; y < h; y++) + memcpy(tmp + (size_t)y * w * 4, + gbuf + (size_t)(h - 1 - y) * w * 4, + (size_t)w * 4); + cap_record(tmp, w, h, w * 4, 32); + free(tmp); + } + } + } + } + } + if (real_GL_SwapWindow) + real_GL_SwapWindow(win); +} diff --git a/tools/guest-diag/semfix_uvmt.py b/tools/guest-diag/semfix_uvmt.py index cf54445..11c191c 100755 --- a/tools/guest-diag/semfix_uvmt.py +++ b/tools/guest-diag/semfix_uvmt.py @@ -53,14 +53,47 @@ SKIP_PB = os.environ.get("SEMFIX_SKIP_PB", "0") == "1" # Seconds between counter-page pin attempts (the guest-agent exec blocks for # its full timeout when the client saturates the guest). COUNTER_RETRY_S = float(os.environ.get("SEMFIX_COUNTER_RETRY", "10")) +# Liveness file checked by pin_supervisor.sh (mtime must stay < 75 s; the +# supervisor grants a 120 s startup grace on top). +SEMFIX_HB_FILE = os.environ.get("SEMFIX_HEARTBEAT_FILE", + "/tmp/opencode/semfix_heartbeat") +# Dead-pid/stall watchdog tuning (seconds). The tracked client is probed in +# the guest only when it has been idle this long, on a cadence, so live +# clients are never probed and a killed client is detected within ~1 min. +LIVENESS_IDLE_S = float(os.environ.get("SEMFIX_LIVENESS_IDLE", "15")) +LIVENESS_EVERY_S = float(os.environ.get("SEMFIX_LIVENESS_EVERY", "25")) +# Pre-capture stall: client alive, counter page not pinned, no progress for +# this long -> recovery attempt; escalate loudly if it persists. +STALL_PRECAP_S = float(os.environ.get("SEMFIX_STALL_PRECAP", "60")) +STALL_RETRY_S = float(os.environ.get("SEMFIX_STALL_RETRY", "30")) +STALL_ESCALATE_S = float(os.environ.get("SEMFIX_STALL_ESCALATE", "180")) +# Inter-tool re-assert request: written here, consumed by pin_gr_stream.py +# (full ping re-assert of every pair it has ever pinned). +REASSERT_REQ = os.environ.get("PIN_REASSERT_FILE", + "/tmp/opencode/pin_reassert_request") + + +def _hb(): + try: + with open(SEMFIX_HB_FILE, "w") as f: + f.write("%.3f\n" % time.time()) + except OSError: + pass class Stream: """Tail the guest scanner's control lines mirrored by pin_gr_stream.py. - The guest emits "#pid <n>" and "#sem 0x<k>" over vsock; the listener - appends them here. This replaces per-loop guest-agent trace reads, which - were too slow to catch short-lived tracking semaphores. + The guest emits "#pid <n> [starttime]" and "#sem 0x<k>" over vsock; the + listener appends them here. This replaces per-loop guest-agent trace + reads, which were too slow to catch short-lived tracking semaphores. + + Replay safety (the m178 regression): the file persists across tool + restarts, so a fresh process re-reads the whole history. Lines written + before this process started (byte offset < init_size) are absorbed as + HISTORY - they seed the dedupe/identity state and the sem/pb lists but + can never trigger a per-client reset. Only lines appended afterwards + are live. This makes a supervisor restart mid-run harmless. """ def __init__(self, path): @@ -70,6 +103,15 @@ class Stream: self.pids = [] self.pbs = [] self.new_pbs = [] + self.new_nopids = [] + self.history_pids = [] # (pid, start) identities from history + self.pid_keys = set() # (pid, start) dedupe, per generation + try: + # Everything already in the file when this process starts is + # history (a restart replays it); never reset on it. + self._init_size = os.path.getsize(path) + except OSError: + self._init_size = 0 def reset(self): """Forget everything learned about the previous client. A new client @@ -80,6 +122,16 @@ class Stream: self.pbs = [] self.pids = [] self.new_pbs = [] + self.new_nopids = [] + self.pid_keys = set() + + @staticmethod + def _ident(line): + """Parse '#pid 1234' or '#pid 1234 5678900' -> (pid, start|None).""" + nums = re.findall(r"\d+", line) + if not nums: + return None, None + return int(nums[0]), (int(nums[1]) if len(nums) > 1 else None) def poll(self): try: @@ -90,12 +142,19 @@ class Stream: self.off = 0 f.seek(self.off) data = f.read() + base = self.off self.off = f.tell() except OSError: - return [], [] + return [], [], [] new_sems, new_pids = [], [] self.new_pbs = [] + self.new_nopids = [] + pos = base for line in data.splitlines(): + # A line is live when it ends past the startup boundary; history + # lines are absorbed (identity/dedupe) but never emitted as live. + live = pos + len(line) + 1 > self._init_size + pos += len(line) + 1 if line.startswith("#sem"): m = re.search(r"0x([0-9a-fA-F]+)", line) if m: @@ -111,13 +170,116 @@ class Stream: self.pbs.append(v) self.new_pbs.append(v) elif line.startswith("#pid"): - m = re.search(r"(\d+)", line) - if m: - v = int(m.group(1)) - if v not in self.pids: - self.pids.append(v) - new_pids.append(v) - return new_sems, new_pids + pid, start = self._ident(line) + if pid is not None and (pid, start) not in self.pid_keys: + self.pid_keys.add((pid, start)) + self.pids.append(pid) + if live: + new_pids.append((pid, start)) + else: + self.history_pids.append((pid, start)) + elif line.startswith("#nopid"): + # In-band client-death signal (new guest scanner; older ones + # simply never send it and the host probe covers the case). + pid, start = self._ident(line) + if live and pid is not None: + self.new_nopids.append((pid, start)) + return new_sems, new_pids, self.new_nopids + + +class ClientTracker: + """Monotonic, replay-proof client-identity tracking. + + A client's identity is (pid, starttime). Rules, in priority order: + * an identity adopted once is never reset a second time (duplicate + lines, trace/stream replays, supervisor restarts); + * a backwards pid, or the same pid with an older starttime, is a + replay - never reset; + * a pid recycled with a NEWER starttime is a real new client - reset + immediately (the previous holder is definitionally gone); + * a newer unseen identity only resets the per-client state once the + CURRENT client is CONFIRMED gone - a reset fired while the previous + client was still active is the m178 third-client hang regression. + + Pure logic: the caller feeds guest liveness results in and performs the + actual state purge when an action says so. + """ + + def __init__(self): + self.seen = set() + self.current = None + self.pending = None + self.dead = set() + + @staticmethod + def _start_match(a, b): + # Unknown starttimes match conservatively (legacy scanner lines). + return a[1] == b[1] or a[1] is None or b[1] is None + + def observe_history(self, pid, start): + """Absorb an identity found in pre-restart history: the process has + no per-client state yet, so history only seeds identity tracking and + the freshest identity becomes current - no reset is ever requested + for history lines.""" + key = (pid, start) + if key in self.seen: + return + self.seen.add(key) + self.current = key + + def note_dead(self, pid, start): + """A #nopid or a successful dead probe: identity is gone.""" + key = (pid, start) + self.dead.add(key) + if self.current and self.current[0] == pid \ + and self._start_match(self.current, key): + self.current = None + if self.pending and self.pending[0] == pid \ + and self._start_match(self.pending, key): + self.pending = None + + def is_current(self, pid, start): + """True when (pid, start) is the currently tracked identity (used to + decide whether a death signal may purge - a stale #nopid for a + recycled pid must never purge the new holder).""" + return self.current is not None and self.current[0] == pid \ + and self._start_match(self.current, (pid, start)) + + def decide(self, pid, start): + """Classify a live #pid line: adopt | skip | skip_backwards | + maybe_reset | reset.""" + key = (pid, start) + if key in self.seen: + return "skip" + self.seen.add(key) + cur = self.current + if cur is None: + self.current = key + return "adopt" + if pid == cur[0] and self._start_match(cur, key): + return "skip" + if pid == cur[0] and start is not None and cur[1] is not None \ + and start > cur[1]: + # Recycled pid, newer process: safe to reset now. + self.current = key + return "reset" + if pid < cur[0] or (pid == cur[0] and start is not None + and cur[1] is not None and start < cur[1]): + return "skip_backwards" + self.pending = key + return "maybe_reset" + + def resolve(self, old_dead): + """Caller learned whether the CURRENT client is dead. Returns + 'reset' (purge + adopt the pending client) or 'defer'.""" + if self.pending is None: + return None + if old_dead: + key = self.pending + self.pending = None + self.current = key + return "reset" + return "defer" def log(msg): @@ -196,7 +358,9 @@ def dmesg_releases(seen): Only lines after the last "sema-svc: thread started" marker count: dmesg keeps the previous module load's lines and the old client's releases. + Returns the number of newly printed lines (used as watchdog progress). """ + n = 0 try: r = subprocess.run( ["sudo", "-n", "sh", "-c", @@ -205,13 +369,15 @@ def dmesg_releases(seen): "| grep 'sema-push' | tail -300"], capture_output=True, text=True, timeout=10) except Exception: - return + return 0 for line in r.stdout.splitlines(): line = line.strip() if not line or line in seen: continue seen.add(line) + n += 1 log("push: %s" % line.replace("vgsp: ", "")) + return n def cdt_pid(): @@ -224,6 +390,34 @@ def cdt_pid(): return int(pids[-1]) if pids else None +def _pid_alive_cmd(pid): + """Guest shell that reports ALIVE / DEAD for a pid. Zombie-aware: a + killed (SIGKILLed) client lingers as a defunct /proc entry, and a plain + `test -d /proc/<pid>` would latch it as alive forever (the m178 + dead-pid latch).""" + return ("f=/proc/%d/stat; if [ -r \"$f\" ]; then " + "s=$(sed -e 's/^[^)]*) //' -e 's/ .*//' \"$f\"); " + "if [ \"$s\" = Z ] || [ -z \"$s\" ]; then echo DEAD; " + "else echo ALIVE; fi; else echo DEAD; fi" % pid) + + +def pid_alive(pid): + """Guest liveness of a client pid: True alive, False dead/zombie/gone, + None unknown (agent error). Callers must treat None as alive (never + purge a client on an inconclusive probe).""" + try: + r = subprocess.run([VMEXEC, "exec", _pid_alive_cmd(pid)], + capture_output=True, text=True, timeout=20) + except Exception: + return None + out = r.stdout or "" + if "ALIVE" in out: + return True + if "DEAD" in out: + return False + return None + + def pin_counter_page(pid, pinned_pids): """Pin the client's completion-counter page from its own pagemap. @@ -268,10 +462,87 @@ def main(): dseen = set() counter_pids = set() counter_next_try = {} + # Current client pid/starttime, mirrored from the tracker (starttime + # disambiguates recycled pids). last_pid = None + last_start = None + # Stale-sem tracking: consecutive unreadable reads, first-unreadable time, + # and the per-sem read skip deadline once a sem has failed 3 times. + unread_count = {} + first_unreadable_t = {} + skip_until = {} + # Failed pool-pin tracking: retry with a 15 s backoff after 2 failures. + sem_pin_fail = {} + sem_pin_skip_until = {} + qmp_next_try = 0.0 stream = Stream(SEMSTREAM) + tracker = ClientTracker() + # Dead-pid/backwards-transition proofing and the stall watchdog. + history_absorbed = False + adopted_from_history = False + probe_next = 0.0 # next allowed idle-liveness probe + recover_next = 0.0 # next watchdog recovery attempt + stall_since = None # first observation of the pre-capture stall + escalated = False + progress_t = time.time() # last host-side progress for the client loop = 0 + def _clear_client_state(): + """Purge every per-client latch. ONLY valid when the previous client + is CONFIRMED dead (or at fresh process start): the m178 third-client + hang came from wiping the state of an ACTIVE client on a replayed + (#pid) identity.""" + stream.reset() + known_sems.clear() + seen_sem_pin.clear() + last_state.clear() + last_seen_t.clear() + pinned_ranges.clear() + seen_ch.clear() + seen_pb.clear() + counter_pids.clear() + counter_next_try.clear() + sem_pin_fail.clear() + sem_pin_skip_until.clear() + unread_count.clear() + first_unreadable_t.clear() + skip_until.clear() + + def _adopt(pid, start, why): + nonlocal last_pid, last_start, progress_t, adopted_from_history + nonlocal stall_since, escalated + last_pid, last_start = pid, start + progress_t = time.time() + adopted_from_history = False + stall_since = None + escalated = False + log("semfix: tracking client pid %d%s (%s)" + % (pid, " start=%s" % start if start is not None else "", why)) + + def _watchdog_reset(): + nonlocal stall_since, escalated + stall_since = None + escalated = False + + def _watchdog_recover(): + """Idempotent pre-capture stall recovery: re-arm every sem latch so + the next loop re-resolves and re-pins, force an immediate counter + retry, and request a full pin re-assert from pin_gr_stream.""" + for sem in set(known_sems) | set(stream.sems): + skip_until.pop(sem, None) + sem_pin_skip_until.pop(sem, None) + sem_pin_fail.pop(sem, None) + unread_count.pop(sem, None) + first_unreadable_t.pop(sem, None) + seen_sem_pin.discard(sem) + if last_pid is not None: + counter_next_try[last_pid] = 0.0 + try: + with open(REASSERT_REQ, "w") as f: + f.write("%.3f\n" % time.time()) + except OSError: + pass + def pin_range_once(gpu_vas, cpu, gpa, size=0): key = tuple(gpu_vas), cpu, size st = pinned_ranges.get(key) @@ -293,40 +564,104 @@ def main(): while True: loop += 1 + _hb() sems = set() pbs = set() chans = set() - new_sems, new_pids = stream.poll() + new_sems, new_pids, new_nopids = stream.poll() for s in new_sems: log("stream: new uvmt sem 0x%x" % s) - for p in new_pids: - log("stream: client pid %d" % p) - if last_pid is not None and p != last_pid: - stream.reset() - known_sems.clear() - seen_sem_pin.clear() - last_state.clear() - last_seen_t.clear() - pinned_ranges.clear() - seen_ch.clear() - seen_pb.clear() - counter_pids.clear() - log("semfix: new client pid %d - reset per-client pin state" % p) - last_pid = p + if new_sems and last_pid is not None: + progress_t = time.time() + + # In-band client-death signals (new guest scanner): purge the dead + # client's per-client state immediately so the NEXT client works + # without a VM cycle (the kill cascade). + for p, st in new_nopids: + log("stream: client gone (%d%s)" + % (p, " start=%s" % st if st is not None else "")) + was_current = tracker.is_current(p, st) + tracker.note_dead(p, st) + if was_current and last_pid == p: + _clear_client_state() + last_pid = last_start = None + _watchdog_reset() + log("semfix: purged per-client state of exited pid %d" % p) + # The old client is gone: adopt any client whose reset was deferred + # while it was still (assumed) alive. + if last_pid is None and tracker.pending is not None \ + and tracker.current is None: + if tracker.resolve(old_dead=True) == "reset": + p, st = tracker.current + _clear_client_state() + _adopt(p, st, "deferred reset") + # Pre-restart history (a supervisor restart re-reads the file): absorb + # identities WITHOUT any reset - the fresh process already has empty + # state; the freshest identity is the client that was running. + if stream.history_pids and not history_absorbed: + for p, st in stream.history_pids: + tracker.observe_history(p, st) + history_absorbed = True + if tracker.current is not None and last_pid is None: + p, st = tracker.current + _adopt(p, st, "resumed from pre-restart history") + adopted_from_history = True + + for p, st in new_pids: + cur_before = tracker.current + action = tracker.decide(p, st) + tag = " start=%s" % st if st is not None else "" + if action == "skip": + log("stream: client pid %d%s seen before - no reset" % (p, tag)) + elif action == "skip_backwards": + log("semfix: client pid %d%s backwards/replay - no reset" + % (p, tag)) + elif action == "adopt": + log("stream: client pid %d%s" % (p, tag)) + _adopt(p, st, "new client") + elif action == "reset": + # Recycled pid, newer starttime: the previous holder of this + # pid is definitionally gone, safe to reset now. + log("semfix: pid %d recycled (start=%s) - reset per-client " + "pin state" % (p, st)) + _clear_client_state() + _adopt(p, st, "pid recycled") + elif action == "maybe_reset": + old = cur_before + alive = pid_alive(old[0]) if old else False + if alive is False: + log("semfix: new client pid %d%s (pid %d gone) - reset " + "per-client pin state" + % (p, tag, old[0] if old else -1)) + _clear_client_state() + tracker.resolve(old_dead=True) + _adopt(p, st, "client switch") + else: + # NEVER clear the active client's state on the account of + # a newer pid: the m178 hang. Deferred until the old pid + # is confirmed gone (periodic idle probe below). + log("semfix: client pid %d%s appeared while pid %s is " + "alive%s - reset deferred" + % (p, tag, old[0] if old else "?", + "" if alive else " (probe unknown)")) for pb in getattr(stream, "new_pbs", []): log("stream: new pushbuffer 0x%x" % pb) pbs.add(pb) + if getattr(stream, "new_pbs", []) and last_pid is not None: + progress_t = time.time() sems.update(stream.sems) # The guest-agent trace read is slow under client load; use it only # as a fallback for channel/pushbuffer discovery and to recover sems # the scanner may have missed. if loop % TRACE_EVERY == 0: + _hb() try: t_sems, t_pbs, t_chans = guest_trace() except Exception: t_sems, t_pbs, t_chans = set(), set(), set() + _hb() sems.update(t_sems) pbs.update(t_pbs) chans.update(t_chans) @@ -336,16 +671,54 @@ def main(): for sem in sorted(sems - known_sems): known_sems.add(sem) - for sem in sorted(known_sems): + for hb_i, sem in enumerate(sorted(known_sems)): + if hb_i % 16 == 0: + # A full pass can run 60-90 s (thousands of QMP reads); keep + # the heartbeat fresh inside it (r203: 79 s gap -> supervisor + # restarted the tool mid-pass). + _hb() + now = time.time() + if now < skip_until.get(sem, 0): + continue try: st = uvmt_state(q, sem) except Exception: st = None + if st is None: + # Stale-sem hygiene: the trace ring replays dead clients' + # kernel pointers forever; without eviction every loop burns + # QMP reads on them and the log floods with "unreadable". + cnt = unread_count.get(sem, 0) + 1 + unread_count[sem] = cnt + first_unreadable_t.setdefault(sem, now) + if cnt >= 3: + skip_until[sem] = now + 30.0 + if cnt == 3: + log("semfix: sem=0x%x unreadable x3 - retrying every " + "30s" % sem) + age = now - first_unreadable_t[sem] + if age >= 60.0 and sem not in stream.sems: + known_sems.discard(sem) + unread_count.pop(sem, None) + first_unreadable_t.pop(sem, None) + skip_until.pop(sem, None) + last_state.pop(sem, None) + last_seen_t.pop(sem, None) + seen_sem_pin.discard(sem) + sem_pin_fail.pop(sem, None) + sem_pin_skip_until.pop(sem, None) + log("semfix: evicting stale sem=0x%x (unreadable %.0fs, " + "absent from stream)" % (sem, age)) + continue + else: + unread_count.pop(sem, None) + first_unreadable_t.pop(sem, None) + skip_until.pop(sem, None) key = state_key(st) - now = time.time() if key != last_state.get(sem): last_state[sem] = key last_seen_t[sem] = now + progress_t = time.time() log("uvmt %s" % state_str(sem, st)) elif now - last_seen_t.get(sem, 0) >= HEARTBEAT: last_seen_t[sem] = now @@ -354,18 +727,34 @@ def main(): # Pin the pool page once per (sem, word) resolution. The payload # word is what the releasing push writes; pinning the page makes # the host able to apply that release. - if st and sem not in seen_sem_pin: + if (st and sem not in seen_sem_pin + and now >= sem_pin_skip_until.get(sem, 0)): try: pairs = resolve_sem_pool(q, sem) except Exception as e: pairs = [] log("semfix: sem=0x%x resolve failed: %s" % (sem, e)) if pairs: - seen_sem_pin.add(sem) + ok_any = False for gv, gpa in pairs: ok = pin(gv, gpa) + ok_any = ok_any or ok log("semfix: sem=0x%x pool va=0x%x -> gpa=0x%x pin=%s" % (sem, gv, gpa, "ok" if ok else "FAILED")) + if not ok_any: + # Mark pinned only after a successful pin(): a failed + # pin used to be recorded as done and never retried. + fails = sem_pin_fail.get(sem, 0) + 1 + sem_pin_fail[sem] = fails + if fails >= 2: + sem_pin_skip_until[sem] = now + 15.0 + log("semfix: sem=0x%x pool pin failed x%d - retry " + "in 15s" % (sem, fails)) + continue + seen_sem_pin.add(sem) + sem_pin_fail.pop(sem, None) + sem_pin_skip_until.pop(sem, None) + progress_t = time.time() if SKIP_PB: continue d = {} @@ -385,6 +774,7 @@ def main(): d["exception"] = str(e)[:40] if res: pin_range_once(*res) + progress_t = time.time() log("semfix: sem=0x%x pushbuf -> %s" % (sem, diag_str(d))) else: @@ -393,7 +783,22 @@ def main(): else: log("semfix: sem=0x%x pool unresolved (retry)" % sem) + if getattr(q, "_fails", 0) >= 5: + # QMP went away (QEMU restart, monitor died): all reads have been + # empty for 5 consecutive replies. Recreate the client loudly + # instead of reading empty forever ("uvmt(hb) ... unreadable"). + now = time.time() + if now >= qmp_next_try: + log("semfix: QMP unreadable (%d consecutive empty replies) - " + "recreating connection" % q._fails) + try: + q = Qmp(QMP_SOCK) + except Exception as e: + log("semfix: QMP recreate failed: %s" % e) + qmp_next_try = now + 5.0 + for ch in sorted(chans - seen_ch): + _hb() # a resolve can block for tens of seconds d = {} try: res = resolve_from_channel(q, ch, d) @@ -403,10 +808,12 @@ def main(): if res: seen_ch.add(ch) pin_range_once(*res) + progress_t = time.time() log("semfix: ch=0x%x -> %s" % (ch, diag_str(d))) else: log("semfix: ch=0x%x pushbuf miss [%s]" % (ch, diag_str(d))) for pb in sorted(pbs - seen_pb): + _hb() # a resolve can block for tens of seconds d = {} try: res = resolve_from_pushbuffer(q, pb, d) @@ -416,12 +823,15 @@ def main(): if res: seen_pb.add(pb) pin_range_once(*res) + progress_t = time.time() log("semfix: pb=0x%x -> %s" % (pb, diag_str(d))) else: log("semfix: pb=0x%x pushbuf miss [%s]" % (pb, diag_str(d))) if DLOG: - dmesg_releases(dseen) + if dmesg_releases(dseen): + progress_t = time.time() + _hb() # Retry incomplete push-buffer pins: gva2gpa of the kernel vmalloc # alias fails while the sampled vCPU runs in user mode (PTI), so a @@ -430,24 +840,89 @@ def main(): if loop % 8 == 0: for st in list(pinned_ranges.values()): if not st["done"] and st["tries"] > 0: + _hb() # pin_range_once can walk thousands of QMP reads pin_range_once(*st["args"]) + # Dead-pid detection and the pre-capture stall watchdog. The guest is + # probed ONLY when the tracked client has been quiet for a while (a + # live, pushing client is never probed), so the agent is not burdened + # during healthy runs. A confirmed-dead client is purged immediately: + # the NEXT client then starts with clean state - no VM cycle needed + # (the kill cascade). + now = time.time() + if adopted_from_history and last_pid is not None: + # The client resumed from pre-restart history may have died while + # this process was down; do not latch a dead pid into the counter + # retries. + alive = pid_alive(last_pid) + adopted_from_history = False + if alive is False: + log("semfix: resumed pid %d already gone - dropping" % last_pid) + tracker.note_dead(last_pid, last_start) + last_pid = last_start = None + _watchdog_reset() + if last_pid is not None and now >= probe_next \ + and now - progress_t >= LIVENESS_IDLE_S: + probe_next = now + LIVENESS_EVERY_S + alive = pid_alive(last_pid) + if alive is False: + log("semfix: client pid %d gone (probe) - purging per-client " + "state" % last_pid) + tracker.note_dead(last_pid, last_start) + _clear_client_state() + last_pid = last_start = None + _watchdog_reset() + if tracker.pending is not None and tracker.current is None \ + and tracker.resolve(old_dead=True) == "reset": + p, st = tracker.current + log("semfix: adopting deferred client pid %d" % p) + _adopt(p, st, "deferred reset") + elif alive is True and last_pid not in counter_pids: + # Pre-capture stall: the client is alive but has no pinned + # counter page and nothing progressed. Recover (idempotent), + # then escalate loudly if the stall persists - the goal is + # that a first cudaMalloc / first GR capture never stalls + # silently. + idle = now - progress_t + if idle >= STALL_PRECAP_S: + if stall_since is None: + stall_since = now + if now >= recover_next: + recover_next = now + STALL_RETRY_S + log("STALL WATCHDOG: client pid %d alive, no " + "pre-capture progress for %.0fs - recovery " + "attempt (re-arm sems, re-pin counter, request " + "pin re-assert)" % (last_pid, idle)) + _watchdog_recover() + if not escalated and now - stall_since >= STALL_ESCALATE_S: + escalated = True + log("STALL WATCHDOG: recovery FAILED for pid %d " + "after %.0fs (alive, counter page not pinned) - " + "pre-capture hang persists, needs manual " + "attention" % (last_pid, now - stall_since)) + # Pin the completion-counter page of a newly started client, before its # first GR pushes decode their releases. The pid comes from the guest # scanner; the guest-agent fallback is only a safety net. The exec can # block for its whole timeout under client load, so retry on a slow # cadence: hammering the agent here starves it and wedges the run. if last_pid is None and loop % TRACE_EVERY == 0: - last_pid = cdt_pid() - if last_pid: - log("counter: client pid %d (agent)" % last_pid) + p = cdt_pid() + _hb() + if p: + if tracker.current is None and (p, None) not in tracker.seen: + tracker.observe_history(p, None) + last_pid, last_start = p, None + log("counter: client pid %d (agent)" % p) if last_pid and last_pid not in counter_pids: now = time.time() if now >= counter_next_try.get(last_pid, 0): try: - pin_counter_page(last_pid, counter_pids) + if pin_counter_page(last_pid, counter_pids): + progress_t = time.time() except Exception as e: log("counter: pin failed: %s" % e) + _hb() counter_next_try[last_pid] = now + COUNTER_RETRY_S time.sleep(interval) diff --git a/tools/guest-diag/semfix_watch.py b/tools/guest-diag/semfix_watch.py index 5124805..4b60fd6 100644 --- a/tools/guest-diag/semfix_watch.py +++ b/tools/guest-diag/semfix_watch.py @@ -58,12 +58,47 @@ CHAN_MAX_CANDS = int(os.environ.get("CHAN_MAX_CANDS", "256")) class Qmp: def __init__(self, path): + self.path = path + self._cache = {} + self._fails = 0 + self._last_reconnect = 0.0 + self._reconnecting = False self.s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) self.s.connect(path) self.s.settimeout(15) self._rd() self.cmd("qmp_capabilities") - self._cache = {} + + def reconnect(self): + """Re-establish the QMP connection after the socket died. + + QEMU's QMP socket accepts a single client and closes when the guest + reboots; cmd() used to swallow every exception and return {}, so all + reads stayed empty forever ("uvmt(hb) sem=... unreadable") and + pinning stopped. Rate-limited to one attempt per second so a failing + monitor does not turn every read into a connect storm.""" + if self._reconnecting or time.time() - self._last_reconnect < 1.0: + return + self._reconnecting = True + self._last_reconnect = time.time() + try: + try: + self.s.close() + except Exception: + pass + self.s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self.s.connect(self.path) + self.s.settimeout(15) + self._rd() + self.cmd("qmp_capabilities") + self._cache = {} + self._ncpu = 0 + self._cpu = 0 + self._fails = 0 + except Exception: + pass + finally: + self._reconnecting = False def _rd(self): """Read one QMP reply (JSON object ends with a newline). Poll with a @@ -91,8 +126,17 @@ class Qmp: def cmd(self, name, **args): try: self.s.sendall(json.dumps({"execute": name, "arguments": args}).encode() + b"\n") - return json.loads(self._rd() or "{}") + reply = self._rd() + if not reply: + raise OSError("empty QMP reply") + self._fails = 0 + return json.loads(reply) except Exception: + # Broken socket (QEMU restart, monitor hiccup): try exactly one + # reconnect, rate-limited inside reconnect(). + self._fails = getattr(self, "_fails", 0) + 1 + if not getattr(self, "_reconnecting", False): + self.reconnect() return {} def hmp(self, line): @@ -143,8 +187,14 @@ class Qmp: break m = re.search(r":(.*)$", out, re.S) if not m: + if not out.strip(): + # Empty monitor output: the socket is broken, not the address. + self.reconnect() return [] vals = [int(x, 16) for x in re.findall(r"0x([0-9a-fA-F]+)", m.group(1))][:n] + if not vals: + self.reconnect() + return [] if cache: self._cache[key] = vals return vals @@ -189,18 +239,46 @@ def guest_trace(): Read *before* clearing: the create-time probes fire once per client and would be wiped by a clear that races them. + + 2026-10-16 robustness fix (the cascade root): this used to run a bare + `tail` over the guest agent. A tracefs read that blocks under a busy + guest left *uninterruptible* tail processes behind (observed dozens in + D state), the guest-agent exec then blocked for its full 90 s timeout + every iteration, and the pool/tracker pins this tool supplies stopped + flowing - after ~5 clients every new client froze at DEVICE waiting on + its UVM tracker with `pending release starved ... pool/waiter page`. + Now the read is bounded remotely (`timeout -s KILL 12`) so no tail can + outlive its call, and SSH (which stays healthy when the guest agent + wedges) is preferred with the agent as fallback. """ - inner = ("tail -c 131072 /sys/kernel/tracing/trace " + inner = ("timeout -s KILL 12 tail -c 131072 /sys/kernel/tracing/trace " "| grep -E \"uvmt|uvch|uvbp|uvpbf\" " "| tail -400") if not os.environ.get("KEEP_TRACE"): inner += "; echo > /sys/kernel/tracing/trace" - cmd = "sudo sh -c '%s'" % inner + outer = "sudo sh -c '%s'" % inner + out = "" + ssh = os.environ.get("SEMFIX_SSH_PORT", "2222") + ssh_cmd = [ + "sshpass", "-p", os.environ.get("VGPU_GUEST_PASS", "ubuntu"), + "ssh", "-o", "StrictHostKeyChecking=no", + "-o", "UserKnownHostsFile=/dev/null", + "-o", "PubkeyAuthentication=no", "-o", "ConnectTimeout=6", + "-p", ssh, "ubuntu@127.0.0.1", outer, + ] try: - out = subprocess.run([VMEXEC, "exec", cmd], capture_output=True, - text=True, timeout=90).stdout + r = subprocess.run(ssh_cmd, capture_output=True, text=True, + timeout=25) + out = r.stdout or "" except Exception: - return set(), set(), set() + out = "" + if not out: + try: + out = subprocess.run([VMEXEC, "exec", outer], + capture_output=True, text=True, + timeout=25).stdout or "" + except Exception: + return set(), set(), set() sems = {int(m, 16) for m in re.findall(r"sem=0x([0-9a-fA-F]+)", out)} pbs = {int(m, 16) for m in re.findall(r"pb=0x([0-9a-fA-F]+)", out)} chans = {int(m, 16) for m in re.findall(r"ch=0x([0-9a-fA-F]+)", out)} @@ -369,8 +447,11 @@ def resolve_pushbuffer_wide(q, sem, d): def pin(va, pa): cmd = "printf '0x%x,0x%x' > %s/sema_map_add" % (va, pa, PARAM) for _ in range(40): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=5) + except subprocess.TimeoutExpired: + continue if r.returncode == 0 and "timed out" not in (r.stderr or ""): return True time.sleep(0.05) # async queue full: drain and retry @@ -380,8 +461,11 @@ def pin(va, pa): def pin_bulk(va, pa, nbytes): cmd = "printf '0x%x,0x%x,0x%x' > %s/sema_map_add" % (va, pa, nbytes, PARAM) for _ in range(40): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=5) + except subprocess.TimeoutExpired: + continue if r.returncode == 0 and "timed out" not in (r.stderr or ""): return True time.sleep(0.05) diff --git a/tools/guest-diag/stuck_client_capture.sh b/tools/guest-diag/stuck_client_capture.sh new file mode 100755 index 0000000..6acd8b5 --- /dev/null +++ b/tools/guest-diag/stuck_client_capture.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# stuck_client_capture.sh - one-shot capture for a stuck CUDA client in the +# vGPU guest (run on the HOST; talks to the guest over ssh_guest.sh). +# +# The in-guest pin scanner consumes trace_pipe continuously, so kprobe reads +# return entries=0 while it runs. This tool: +# 1. pauses the scanner (kill -STOP), +# 2. reads the retained kprobe trace tail for the client's UVM waits +# (uvtw/uvsv/uvmt) - the tracker semaphore kptr and the value waited, +# 3. walks the tracker object (tracker_walk.py) for its expected payload, +# 4. dumps the semaphore-pool page (pool_page.py) for applied values, +# 5. resumes the scanner. +# Usage: stuck_client_capture.sh <client-comm> [pool-pa-hex] +# e.g. stuck_client_capture.sh victory_matmul 0x107fd8000 +set -u +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +SSH="$REPO/scripts/ssh_guest.sh" +CLIENT="${1:?usage: stuck_client_capture.sh <client-comm> [pool-pa]}" +POOL="${2:-0x107fd8000}" + +echo "=== pushing walk helpers to the guest ===" +"$SSH" "cat > /tmp/tracker_walk.py" < "$REPO/tools/guest-diag/tracker_walk.py" +"$SSH" "cat > /tmp/pool_page.py" < "$REPO/tools/guest-diag/pool_page.py" + +echo "=== pausing scanner ===" +timeout 30 "$SSH" "echo ubuntu | sudo -S sh -c 'pkill -STOP -f guest_pin_stream.py; echo STOPPED'" 2>/dev/null | tail -1 + +echo "=== kprobe trace tail (client $CLIENT) ===" +timeout 60 "$SSH" "echo ubuntu | sudo -S sh -c 'tail -c 300000 /sys/kernel/tracing/trace | grep -a \"\$CLIENT\" | grep -aE \"uvtw|uvsv|uvmt\" | tail -8'" 2>/dev/null | tail -10 + +echo "=== tracker walk (kptr from the uvsv line above) ===" +echo " (run: sudo python3 /tmp/tracker_walk.py 0x<sem-kptr>)" + +echo "=== pool page $POOL ===" +timeout 90 "$SSH" "echo ubuntu | sudo -S python3 /tmp/poolpage.py $POOL" 2>/dev/null | tail -12 + +echo "=== resuming scanner ===" +timeout 30 "$SSH" "echo ubuntu | sudo -S sh -c 'pkill -CONT -f guest_pin_stream.py; echo RESUMED'" 2>/dev/null | tail -1 +echo "=== done ===" diff --git a/tools/guest-diag/tracker_walk.py b/tools/guest-diag/tracker_walk.py new file mode 100755 index 0000000..3846fa1 --- /dev/null +++ b/tools/guest-diag/tracker_walk.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +# tracker_walk.py - read a UVM tracking semaphore's expected payload and walk +# its semaphore object via /proc/kcore (run in the guest as root). +# +# Use with a live stuck client: the udvt/uvtw kprobes or the scanner's #sem +# lines give the tracker kptr (e.g. 0xffff899a885b5480); this prints +# tracker+0x0 = semaphore kptr +# tracker+0x28 = expected payload (the value the client waits to observe) +# semaphore+0.. = its pointers (rm_mem/page) and candidate value words +# Pair it with the host's pool-wr/deferred-release logs and a pool page dump +# (pool_page.py) to compare what the driver expects vs what the host wrote. +import sys +import struct + +sys.argv = ["tracker_walk"] +sys.path.insert(0, "/tmp") +import guest_pin_stream as g # noqa: E402 + +kc = g.Kcore() +if not kc.load(): + print("KCORE LOAD FAILED") + sys.exit(1) + + +def r64(a): + d = kc.read(a, 8) + return struct.unpack("<Q", d)[0] if d else None + + +def r32(a): + d = kc.read(a, 4) + return struct.unpack("<I", d)[0] if d else None + + +for arg in sys.argv[1:] or ["0xffff899a885b5480"]: + SEM = int(arg, 16) + print("tracker 0x%x:" % SEM) + for off in range(0, 0x40, 8): + print(" +0x%x = 0x%x" % (off, r64(SEM + off) or 0)) + sp = r64(SEM) + print("semaphore=0x%x" % (sp or 0)) + if sp and g.is_kern_ptr(sp): + for off in range(0, 0x80, 8): + v = r64(sp + off) + print(" sem+0x%x = 0x%x" % (off, v or 0)) + for off in range(0, 0x80, 8): + v = r64(sp + off) + if v and g.is_kern_ptr(v): + w = r32(v) + print(" *sem+0x%x -> dword=0x%x (%u)" % (off, w or 0, w or 0)) diff --git a/tools/guest-diag/wall2/README-session74.md b/tools/guest-diag/wall2/README-session74.md new file mode 100644 index 0000000..58587eb --- /dev/null +++ b/tools/guest-diag/wall2/README-session74.md @@ -0,0 +1,66 @@ +# WALL #2 / device-init completion — session 74 findings (r675) + +## The client's "11 s stalls" fully root-caused (this is NOT the channel-init release) +strace (ioctl-only) shows THREE exact 11.00 s gaps with NO ioctl in flight; a +full strace shows the client doing a pure `sched_yield` spin (~57K/s) + +`clock_gettime` (perf) inside **libnvidia-eglcore.so.570.144** — i.e. a +userspace timeout loop. gdb backtraces place the spin inside the Vulkan ICD's +EGL/GL init (`vkCreateInstance` -> libGLX_nvidia vk_icdNegotiate... -> +libEGL -> eglcore wait loop). + +## The wait target: completion slots in /dev/nvidiactl mappings +- The wait loop (eglcore file offsets ~0xb284b0 delay, ~0xb33d00 wait loop, + ~0xb1c750 slot check) polls a slot table on the client's heap object: + table at object+0x1428, count at object+0x22efc. +- Slot check: `esi` is offset by 0x43; for esi-0x43 > 4 it reads a **u16 at + slot+0xe** and treats `0xffff` as "empty/not complete". +- gdb sampling during the stall: 13164 checks with `u16@slot+0xe = 0xffff`, + 1 with `0` — the client spins until the flag changes. +- The slot pointer = a **/dev/nvidiactl rw-s mapping** (page-sized; base+0x20 + = slot). These are the client's mappings of RM memory (the channel + notifier pages: service logs `notif=0x11627f060` per channel). + +## Filling the slots (guest-side) WORKS mechanically but is not the -4 cause +- `slotfill.sh` / `slotfill2.sh`: while vk_frame runs, write 0 to + `base+0x2e` of every small /dev/nvidiactl rw-s mapping when it reads 0xffff. +- Result: the 11 s gaps vanish (5 fills within ~2 s; the client proceeds + immediately) — **but vkCreateDevice still fails -4**. So the 11 s waits are + not the failure; they are an init-side slowdown the client tolerates. + +## The next wall: the polled RM control 0x2080a097 +- With the slots filled, the client settles into **NVOS54 (ioctl nr=0x2a) + polling every ~1.25 s** (strace), and the host dmesg shows + `GSP_RM_CONTROL cmd=0x2080a097 hClient=0xc1d000bb hObject=0xbeef0004 + paramsSize=1168 flags=0x0` -> `unknown cmd=0x2080a097 -> generic SUCCESS` + (all-zero reply), 36 polls in 45 s. +- The NVOS54 struct (gdb dump): {hClient=0xc1d000c0, hObject=0xbeef0004, + cmd=0x2080a097, flags=0, params=ptr, paramsSize=0x490, status=ptr, ...}. +- The params buffer = 1168 bytes of 32-byte records with a `0x25` field at + +0x10 (dump in /tmp/opencode evidence; gdbparam.sh captures it live). +- The client polls until the control's status/reply signals completion; our + generic all-zero reply never does. **This is the wall that must be solved to + get vkCreateDevice past -4** (either a real reply for 0x2080a097 or making + the polled status field change). + +## Other session-74 results +- The GR ring-entry writer was NOT found by disassembly-guessing: nvidia.ko + `channelFillGpFifo` and nvidia-modeset `nvWriteGpEntry` never fire for this + channel's kick (gdb hbreaks); nvidia/modeset kprobes are `notrace`; QEMU gdb + hbreak works. +- `sema_push_raw` dumps (module param) captured the service's reads of the + push: entry0 decodes only 2 methods then zeros (`consumed=0x288/1432`), + entry2 starts `SET_OBJECT 0xc597` then zeros; **no release in the push** + (`have_rel=0` for all three entries) — the channel-init release theory for + these waits is dead. +- `find_ring_va.py` / `vmap_watch2.py`: the ring entries are NOT in any + vmallocinfo area during the run; the nv_vm_map_pages areas hold the CE + pushes; the GR push content is not found by content searches. + +## Files +- gdbpoll6.gdb, gdbu16.gdb, gdbfill.gdb, gdbwatch.gdb, gdbparam.sh, + gdbgap3.sh, gdbspin.sh — the gdb probes (slot check, u16 flag, fill, + NVOS54 param dump, backtraces). +- slotfill.sh / slotfill2.sh — the guest-side slot-fill watcher. +- stracevk.sh, fastrun.sh, fastrun2.sh — the strace/perf recipes. +- perfstall5.sh — perf sampling with map-relative offsets. +- find_ring_va.py, vmap_watch2.py — the vmap/kcore watchers. diff --git a/tools/guest-diag/wall2/README-session76.md b/tools/guest-diag/wall2/README-session76.md new file mode 100644 index 0000000..828b3b9 --- /dev/null +++ b/tools/guest-diag/wall2/README-session76.md @@ -0,0 +1,134 @@ +# session 76 notes (r680) — the 0x2080a097 engine-monitor re-examined + +## What the firmware says (ground truth for the a097 family) + +The shipped GSP-RM 570.144 image embeds the host-side control +descriptor table (file offset 0x41df8..0x42450, entry = 0x20 bytes +{u32 cmd, u32 params_size, u32 0x4049580, u32 0, u32 handler_ptr, +u32 0, u32 out_size, u32 0}). Entries seen: + + 0x2080a07f size 0x408 handler vaddr 0x551b2c0 + 0x2080a079 size 0x14804 + 0x2080a097 size 0x490 handler vaddr 0x55166f4 + 0x2080a0d1 size 0x7e8 + ... + +The real 0x2080a097 handler (0x55166f4) writes up to FOUR {u32,u32} +mode pairs into params+4..+0x23 (params[0] low byte = the request's +count flag): for i=1..4 it calls an engine-status helper +(0x5516334 / 0x55d7f04 / 0x55d7c1c / 0x55d7d60) with mode=i and emits +{pair} = {engine value, 1000 * u16(engine+0x1bc0)}; the loop stops at +the first helper error. So the real reply overwrites the client's +request record {1,0,0,0}. + +## What the client does with the reply + +libnvidia-eglcore's local handler (file offset 0xecd2e0, the userspace +implementation of the internal cmd 0x2080a040) copies back: + + result+0 = reply+0 (count) + result+0x10 = reply+0x204 (flag) + result+0x20 = reply+0x488 + result+0x24 = reply+0x48c + count*16 bytes from reply+4 -> *(result+8) + +For the engine tick (file offset 0xd5e3e0, called from the wait loop +0xd6ef40 per iteration) the result struct IS the caller's request at +rsp+0x10, and *(result+8) points at a 16-byte "inner" buffer +(rsp+0). The tick then reads inner+4 and stores it to session+0xe104 +(the per-engine "progress" cache; poll interval ~1.25 s from the same +function). A second consumer of the tick's return value compares it +against per-boot config at lib+0x2019320: + + 0xfc8 = 1000, 0xffc = 95 (progress threshold), + 0xfec = 10, 0xfed = 20 (bit-history cutoffs), + 0x1000 = 500 + +and ORs the comparison bit into a round-robin bitfield (+0xb4) of the +state machine object. + +## Live evidence on r680 (fresh boot, full-param recipe) + +* The 0x2080a097 polls (NVOS54 0x2a, 1.25 s cadence) come from a + WORKER thread (gdb bt: start_thread -> eglcore a11b2a -> wait loop + 0xd6ef40 -> tick 0xd5e3e0). The engine object E (0x604101365e90) + sits at E+0x150 = 0x00200000 (bit 21 only), E+0x1b038 = 1, + E+0x1b039 = 1, *(E+0x1b030)+8 != 0 — all static for 5+ minutes. +* MAIN thread (vkCreateDevice) is in a poll(fd=/dev/nvidia0, events + POLLIN|POLLPRI, timeout=10s) loop inside eglcore (bt: poll <- + eglcore+0xa12de4 <- 0xd69159 <- 0xd6a02b <- 0xdcebc6 <- ... <- + libvulkan vkCreateDevice). No -4 is ever printed in this state: the + client waits indefinitely (killed by its own timeout in runs). +* GR channel kicks continue concurrently (sema-entry lines for + h=0xbeec0100 va=0x4ca0000 len=1432 etc.) and end as + "skipping stale entry ... after N failed rescues" (gr_wedge>0). +* Reply sweep on r680 (module params) — flag=0/2/1/1..0xffffffff, + v488, v48c AND the new record matrix count/rec0..rec7 = + 0/{1,0,0,0}/100/1000/100000/1000000/0xdeadbeef — produced ZERO + observable change: same a097 cadence, no writers to + E+0x150/E+0x1b038/E+0x1b039 ever caught, no client progress. The + hw watchpoints on those words fired zero times over ~5 min. + +## Tools + +* t75g.py — in-guest gdb python: dispatcher-type log, a097 issuer + dump, tick (d5e449) state + cfg dump, E/S capture, hw watchpoints on + E+0x150 / E+0x1b038 / E+0x1b039 / S+0x120, POST a040 consumption + dump. +* diag76.py — in-guest gdb python: thread-apply bt + map-relative + address resolution (used to find the main-thread poll wait). +* slots_probe.sh — read the u16 completion-slot words of every small + /dev/nvidiactl rw-s mapping (SESSION 74's slotfill ABI). +* run75g.sh — trace + record-word sweep driver. + +## Verdict + +The a097 monitor is a background worker, not an obviously blocking +gate: every knob the client reads from it can be varied with no +observable state change, while the main thread blocks elsewhere +(/dev/nvidia0 poll loop) and the GR channel-init pushes still cannot +be read by the sema service. Next lever is the main thread's wait +(what would make the /dev/nvidia0 poll return readable) or the GR +channel-init completion — not the a097 reply words. + +## ADDENDUM (r681) — THE MAIN-THREAD vkCreateDevice WAIT DECODED + +Live gdb on the main thread (r681, fresh boot) resolved the loop at +eglcore 0xdceb80 (frame chain: poll <- a12de4 <- d69159 <- d6a02b <- +dcebc6 <- d3838a <- ... <- libvulkan vkCreateDevice): + +* rbx (waitable, vt 0x...3f58f8): vt+0x18 = d69ee0 = "should wait" + (returns `(*0x48(*(rbx+0x70)) != 1)`); vt+0x28 = d69f50 = "wait" + (builds a pollfd array from the object's fd holder and poll()s up to + 16 fds with the 10 s timeout). +* rbp (loop object, vt 0x...3f71f8): vt+0x38 = dcee20 = the give-up + predicate. dcee20 returns nonzero (-> return -4 = the client's + VKFAIL -4 origin) when either object at rbp+0x110 / rbp+0x120 shows + an error (b29d60(...) != 0 or *(obj+0x70cc) != 0); otherwise 0 and + the loop repeats. It also throttles itself at 1e6 (units) via + rbp+0x1b8. +* The readiness value: sub = *(rbx+0x70) (0x...f7f0c90); sub+0x58 -> + obj2; obj2's vt+0x10 = 0xce71f0 = `read u32 at (*(obj2+0x58)) + + *(u32*)(obj2+0x60)` — i.e. a u32 word at holder_base + 0x80 (the + live object had obj2+0x60 = 0x80, obj2+0x64 = 0x1e). Observed: 0x0 + every poll (39/39) -> should-wait = 1 forever. +* The word lives in a **/dev/nvidiactl rw-s mapping** + (0x7623d0480000-0x7623d0484000) — a VM_IO driver page, not plain + guest RAM: /proc/PID/mem reads of it return zeros, and writing the + guest-RAM file at the pagemap-derived PAs (0x1036dd080 and both + hole-shifted candidates) did NOT change the client's behaviour. + The writer is therefore the guest kernel driver/RM when the awaited + operation completes (the same class as the SESSION-74 completion + slots) — not a host RAM poke. +* All 20 sampled polls of the wait helper returned 0 (timeout): the + fd (/dev/nvidia0) never becomes readable (nv_poll reports + POLLPRI|POLLIN only when an nv_event is pending on the file). + +So vkCreateDevice waits for (a) the readiness word == 1, and the wait +ends only when the driver/RM signals that word (or an nv_event arrives +on /dev/nvidia0). The give-up path (-4) fires only if the RM error +flags appear. Next: identify the operation whose completion writes the +word (the object chain sub -> obj2 -> holder is a "signal/semaphore" +object; dump the holder object and its creating control), then make +the emulation drive the guest kernel to complete it (the engine/ +channel-init completion the sema service already tries to deliver). diff --git a/tools/guest-diag/wall2/README-session77.md b/tools/guest-diag/wall2/README-session77.md new file mode 100644 index 0000000..d830bef --- /dev/null +++ b/tools/guest-diag/wall2/README-session77.md @@ -0,0 +1,1195 @@ +# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion + +## The two-part unblock + +1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the + nvkms/Vulkan path allocates its completion event as + NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now + registers it alongside 0x79. Live proof (r683, sema_dbg=1): + + GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011 + hParent=0xcaf00001 hObject=0xcaf00003 + event registered hEvent=0xcaf00003 hSrc=0x0 + notifyIndex=1006632972 (off 8, slot 13) + params=1100e0c1000000007e0000000c00003c0000000000000000 + EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0 + + (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc = + NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records + (`guest_read == our_write` in the rmstate dump), but the guest's + event dispatch does NOT write the event's notification word — the + residual delivery gap. + +2. **The completion word found and driven (the actual unblock)**. The + main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads + a u32 readiness word: `should-wait = (word != 1)`; the word lives at + `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event + notification buffer; holder chain sub -> obj2 -> holder, resolved + live by t79.py). The host can write it directly in the guest RAM + file: + + PA = pagemap PFN << 12 (+0x80); file offset = PA - 0x80000000 + (above-4G RAM; below-4G would be PA itself per the cycle's + guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000) + + Writing 1 flipped the wait and **vkCreateDevice returned success** + (`VKDEVICE ok family=0`) — reproduced three times: + * r683: manual write after a t79 probe; + * r684: nudger (all 16 KB mapping words) + slotfill restart; + * r684d: probe -> single-word write (the reliable recipe). + +## The reliable recipe (per run) + +1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional. +2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the + slots re-arm, so re-run slotfill if the client is stuck in the + sched_yield spin before the word-wait). +3. Wait for the main thread in `do_poll` (the word-wait). +4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA; + host write `01 00 00 00` at `PA-0x80000000`. +5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to + image/memory/pipeline setup; its next wait is NOT the same word + (the word stays 1) — next session: probe the follow-on state + (attach fails while the client spins in the post-device phase; + use the /proc-only checks + a longer observation window). + +## Cautions learned + +* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the + -4 give-up path (an RM error flag word gets written): restrict nudges + to the single probed word. +* Attaching gdb to the client while it is in the tight post-device spin + can hang the attach (leave a `timeout`-wrapped gdb, then kill that + gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a + dead tracer otherwise wedges it). +* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in + python, not zsh. + +## Tools + +* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA) + and prints the client's nvidiactl rw-s mapping sizes. +* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB + nvidiactl mapping (pagemap; run in the guest as root). +* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any + zero word (multi-page variant; prefer the single-word recipe). + +## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6) + +* The post-device spin is the **GL module bring-up**: the main thread's + A8-layer retry loop (eglcore 0xa85d24) works on a work item whose + embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6 + submit layer (0xd624e0) ORs **0x200000** into its pending word (the + same engine-pending bit SESSION 75/76 watched on E+0x150), then the + A8 loop yields via the B284B0 spin helper (sched_yield) waiting for + the GL ucode init to complete on the GR engine. +* **The push windows ARE now readable** (first time): the resolver6 + recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} -> + same-page 0x20000 memdesc PA -> `sema_map_add`) injected + `0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d + 68000000` = the 2-method semaphore-like push) and `0x4ca0000 -> + 0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5, + `0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d + 70/68`). The service now decodes the streams (`sema-push ... tail=0dc0 + 012000000070...`) where SESSION 73-74 saw "unread". + CAUTION: the resolver injects every candidate; a zero-page candidate + injected last replaces the good mapping for that VA (re-inject the + good page after the scan). +* **Remaining gate**: the service still reports `have_rel=0` for every + GR push (the CE-shaped release matcher) and — decisively — the + GR-forward machinery is NOT engaging: `drain held=4 starved=149 + forwarded=0 ... gr_nocap=4`. The GL ucode init work is never + forwarded/executed, so the completion (and the 0x200000 bit clear) + never arrives. Next: the GR forward/capture path for these channels + (operand capture + release decode for the GR shape), then the client + proceeds to VKFRAME. + +## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe + +* The GR/GL init releases decode only with the low-aperture acceptance + (`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms, + commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group + `200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0 + pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ... + 02000001`); `have_rel=1`, `applied` rising, un-translated releases + queue in svc_pending (417 observed). +* Corrected pair-specific push injections for r686: + `0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc), + `0x4ca0000 -> 0x131649000` (page 0x122f20000), plus + `0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window). + CAUTION: resolver7 injects each page's descs for BOTH pairs (union + loop) - fix it to keep per-pair pages or inject manually from the + per-pair scan. The window-record layout is {tag, VA, SIZE, 0, + "tail", ...} stride 0x40 (search the VA u64 with size at +8 and + "tail" at +0x10). +* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no + window records in the known record pages; the service cannot resolve + them (targeted rm_mem miss) so the releases stay pending and the + NvOgl wait spins. Find their PAs (client mapping pages / RM + VA-tracker / live poll-watch), inject, and the pending releases + should apply. + +## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying + +* **The record scan fix**: the RM window record is 0x40 bytes + {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is + stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT + "tail" (the hex display `7461696c` reads as "tail" only as a number). + Searching for b'tail' finds nothing; `wall2_recscan.py` searches for + the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at + -0x20/-0x18/-0x10. It found **all 75 records** in one pass. +* **The GR init semaphore windows** (r686): + `va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content: + +0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 / + rel=0x4b17e30 pay=1 payloads) and + `va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the + release applied there: +0xb10 became 2). Injections: + `echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and + `echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`. +* **State after the injections**: the low-VA releases decode + (`have_rel=1`) and apply to the semaphore pages; `pending` hovers at + ~418 (the client re-kicks continuously); the NvOgl wait still spins + (the A8 loop compares per-entry counter tables r14+0x5e8 vs the + wanted set - the completion is not just the semaphore value). Next: + the A8 counter tables' source (what fills r14+0x5e8 / the "wanted" + table - likely the GR channel's own progress counters) or the GR + forward/ucode execution (`forwarded=0 gr_nocap=7` still). + +## ADDENDUM (r686, session 81) — the counter-autoreg gap analysed: the counter page 0x20400f000 is unresolvable; the manual `sema_counter` registration is the bypass + +* `svc_counter_autoreg` (per-pass for eng==0 channels, n_gr=20) needs a + translation for the rank slot VA `0x20400fec0 + (19-r)*0x10` + (r = gr_index = rank by ascending handle among the guest's GR + channels): either an ipin pin (`svc_ipin_lookup`, 9275 entries live + from the guest pin stream) or `svc_counter_derive` (the + vaspaces/discovered/targeted walk chain, `sema_counter_derive=1` + default). Both miss for the counter page -> no "counter auto" log, + `pub_calls=0` -> the A8 loop's per-entry counter tables never fill and + the NvOgl wait spins. +* The client does NOT map the counter pool: `/proc/<pid>/pagemap` for + 0x20400f000/0x20400ffd0/0x20400ffc0/0x20400ffb0 is all-zero (not + present) and no VMA covers it - it is a UVM kernel-side object, so the + pin stream cannot resolve it through the client's page tables. The + stream's log shows `sem=0` (zero semaphore resolutions) even though + the want-list (`/tmp/guest_want_vas.txt`, host-written) contains 44 + `20400f` entries; the stream's "channel-anchored expansion" for the + UVM tracking sems (guest_pin_stream.py ~line 950-1010) is the + machinery that must find them (it is written for the CUDA second-client + stall - "the only live uvmt event is the spinning sem"). +* Bypass available now: the `sema_counter` parameter registers a channel + slot manually - `echo <h_object>,<pa> > .../parameters/sema_counter`. + Once the counter page's PA is known (via the guest kcore/UVM walk), + this wires the publish path without a kernel rebuild. +* The rank slots for the GL client's channels (handles 0xbeec0100, + 0xbeec0104, 0xbeee0100 -> ranks 0,1,2): 0x20400ffd0 / 0x20400ffc0 / + 0x20400ffb0. + +## ADDENDUM (r686, session 82) — the counter-page PA hunt: pagemap route confirmed dead, vidmem-window hypothesis tested and not supported + +* The client's own pagemap does NOT contain the counter pool: the + semfix_uvmt tool's resolver (`guest_pmap_win.py <pid> 0x20400f000 + 0x204010000 64`) returns NOTHING, as does the UVM pool range + (`0x121000000 0x121100000`). The tool's `pin_counter_page` has + therefore never succeeded for these clients (the STALL WATCHDOG's + "re-pin counter" attempts across sessions produced no "counter: + pinned" line). +* VIDMEM-window hypothesis tested: the counter VA 0x20400f000 is in the + emulated vidmem window (>= SVC_VIDMEM_VA_BASE 0x200080000), so its + window offset is 0x3F87000 (0x20400f000 - 0x200080000). The shadow + there is reachable and all-zero (`sema_vidmem_dump 0x3f87000,64`); + poking the rank slots (+0xfd0/+0xfc0/+0xfb0 = ranks 0/1/2) with 6 and + then 0x40 via `poke_vidmem` changed the shadow (verified by dump) but + did NOT move the client's wait. The vidmem route is not (or not only) + the completion path; the A8 loop's tables have another source. +* gdb-attach hazard (again): attaches to the client in its post-device + spin hang with no output; two gdb chains had to be killed by PID + (never a pattern) and the client SIGCONT'd. Probe only while the + client is in do_poll. + +## ADDENDUM (r687, session 83) — THE A8 COUNTER SOURCE DECODED: a client readback window the host can write (first live read-back confirmed) + +* The A8 loop calls `a82f30(obj)` per entry; a82f30 calls the object's + method pointer at ***(obj+0x19b8)** = eglcore+0xb250f0, which returns + `(hi<<32)|count`. The count comes from **b1bf00**: + - fast path (obj+0x224 != 0): `*(u32*)(obj+0x1078) - 1`; + - slow path (observed: obj+0x224 == 0): **b27c90(obj, 0x800000bb/0xbc)** + which reads `*(u32*)(*(*(obj+0x1480)+0x58) + idx*16)` - a **client + READBACK WINDOW** (idx = 0xbb..0xbb+n, n = *(u32*)(obj+0x22efc)); + the loop takes the MIN of those reads. +* Live (r687, object handle 0xbeec0100): the window = VA + 0x7827e6c12000 (a /dev/nvidiactl mapping), all zeros; obj+0x224=0 + (slow path active), obj+0x1078=7 (fast-path field unused). +* **The host CAN write it**: resolving the window's PA via the client's + pagemap and writing 0x100 into the window (offsets 0xbb0..0x1ff0, via + the RAM file) made the client read it - **obj+0x1070 = 0x100** (the + method stores the returned count there) and the A8 "actual" table + entry became `0xffffffff00000100`. FIRST LIVE PROOF that the client's + completion counter can be driven from the host. +* The wait still spins: the A8 loop's first scan (compare + a85e1d: actual vs want, want observed 0) passes with our value, but + the loop continues into a second scan (compare a85ec1, driven by + b29dc0) and the `*(rsp+0x38)` branch (a85e60). NEXT: decode those + (the second scan's actual source and the rsp+0x38 flag), and whether + the other objects (entries 0-2 of the A8 table) need the same window + writes. +* The other objects' windows were not observed (only 0xbeec0100 called + a82f30 in the sampled window); the nvidia0-side window (obj+0x1478) + is NOT present in the pagemap (VM_IO; host cannot write it via the + RAM file). + +## ADDENDUM (r687, session 84) — the a86038 want-scan mapped: ALL WANTS PASS with the driven window value; the residual gate is the A8 loop's internal callback/retry machinery + +* Live dump of the a86038 scan (BP at 0xa8608e, per-index): + i=0 want=0x0 actual=0x0 pend=no + i=1 want=0x0 actual=0x0 pend=no + i=2 want=0x0 actual=0x0 pend=no + i=3 want=0x6 actual=0xffffffff00000100 pend=no + (want from a82fd0(job_obj, i) = the job object's +0x98 array maxed + against global+0x600; actual from r14+0x5e8 = our driven counter). + NO pending entries -> the scan completes. +* Writing 0x100 to the ENTIRE readback window (offsets 0x0..0x1ff0 of VA + 0x7827e6c12000) did not clear the wait either. +* The loop still cycles: the bt stays at the a85f08-a85f1a yield call + (b284b0(*(rsp+0x10), 0xb) - a bounded pause/yield spin of ~0x14 + units, hence the slow cadence), and the exit-attempt path a85f30 + decrements *(rbp) (w0=1) while firing the global callbacks + (0x206f790/0x206f7a0 vtables: *(%rax), *0x10, *0x18) with args from + rbp+0x30/0x38/0x40. The remaining gate is this callback/retry + machinery (or another wait object), not the readback values. +* NEXT: resolve the global callback targets (0x206f790/0x206f7a0 + vtables at runtime) and trace the a85f30 exit path (what the + callbacks do and what makes *(rbp) reach 0); or find how the real GPU + writes the readback window (the op-4 methods 0x800000bb/0xbc the + service skips) and emulate that write so the client's own machinery + completes. + +## ADDENDUM (r687, session 85) — **THE NvOgl WAIT CLEARED** (the session-count gate) and the client advanced to vkAllocateMemory (new wall: -2 OUT_OF_DEVICE_MEMORY) + +* The A8 loop's exit requires the session count `*(u32*)(r14+0x5bc)` + (= lib_base + 0x2070b7c, the global at +0x20705c0+0x5bc) to reach 0; + the loop head increments `*(rbp)` and the entry scans pass with the + driven window value, but with count=4 the exit checks + (a85b48/a85e78: `if (r12d == 0)`) are never taken - the loop waits for + the engine sessions to complete (count -> 0). +* **DIRECT TEST (succeeded)**: `set {unsigned int}0x<base+0x2070b7c> = 0` + via gdb -> the loop exited, the NvOgl init completed, and the client + ADVANCED past the post-device wait for the first time: + + VKDEVICE ok family=0 + VKFAIL vkAllocateMemory(dev, &ma, NULL, &img_mem) -> -2 (line 204) + +* **NEW WALL**: `vkAllocateMemory` for the image (DEVICE_LOCAL) returns + -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY) at vk_frame.c:204 - the guest + driver's heap/type selection fails before any alloc RPC (no + GSP_RM_ALLOC in dmesg). Next: the memory-heap reporting + (FB_GET_INFO_V2 0x20801303 is handled; check the heap-size/type + queries the driver uses for the DEVICE_LOCAL type, and the emulated + VRAM size the driver sees). +* Recipe for the next run: slotfill + word nudge -> VKDEVICE ok; then + write 0 to lib_base+0x2070b7c (gdb; the harness) once the NvOgl wait + starts (the A8 loop's count gate); then the memory-alloc wall. + +## ADDENDUM (r689, session 86) — **THE DEVICE_LOCAL -2 WALL CRACKED**: the 512 MiB emulated VRAM was the gate; the "nvidia-nvidia-vgpu-turing-2g" mdev type (2 GiB) lets the allocation proceed + +* Evidence chain: the ICD hooks (t91.py) captured the client's allocation + `ALLOC sType=5 size=0x4000 typeIndex=1` (a valid 16 KB DEVICE_LOCAL + request); the standalone memory-properties dump (vk_memprops.py) + showed device-local heap[0] = 512 MiB with a separate 2936 MiB + non-local heap. With 512 MiB the driver failed the allocation + (VK_ERROR_OUT_OF_DEVICE_MEMORY, no RM RPC). +* **FIX TESTED**: the cycle with `VGPU_MDEV_TYPE=nvidia-nvidia-vgpu-turing-2g` + (start_vm_keep.sh honours the env var) -> `heap[0] size=0x80000000 + (2048 MiB)` (vk_memprops.py) -> the SAME client with the SAME recipe + (slotfill + word nudge + count auto-clear) reaches `VKDEVICE ok` and + the allocation NO LONGER fails -2: the client proceeds into + `os_schedule` (a deeper wait). +* REMAINING in the new state: the client's later pushes (va=0x44852d4 / + 0x4ca52d4, then +0x586c) extend beyond the first window page and read + `pa=0x0`/zeros (the sema injections covered page 1 only; the window's + page list is scattered - the memdesc carries only the FIRST page). + NEXT: extend the window injections to the full scattered page list + (from the memdesc's page array) or accept 1-page pushes; then the + allocation completes and the client proceeds to the buffer/pipeline. +* Tools: vk_memprops.py (standalone memory-properties dump; safe + alongside other clients), vk_alloc_test.py (standalone allocation + reproducer - NOTE: it hits vkCreateDevice -4 without the harness + (word nudge + count clear), so it cannot replace the vk_frame run), + t91.py (auto-clear + ICD memprops/alloc hooks). + +## ADDENDUM (r689, session 87) — the window page-list hunt: the pages are scattered; content matches found but the VA<->page relation is unresolved + +* The 2 GiB cycle's window records (recscan89): `0x4480000` descs = + {0x107c47000 (live init stream), 0x103c06000, 0x12ecc2000}; + `0x4ca0000` descs = {0x11dbfa000, 0x12f398000 (live init stream)}. + The window's later offsets (va+0x52d4, +0x586c, +0x9000) at + first_PA+offset are ALL ZEROS -> the windows are NOT contiguous + (scattered pages). +* The memdesc record (page 0x106128000) is {size=0x20000, flags + 0x03008000, PA=0x107c47000, kernel pointers 0xffff8d2e5e21ec50 / + 0x4fe7ec50 / 0x461287e8 ... , "head"/"tail"} — the page list hangs off + the kernel pointers. A guest kcore walk of those pointers returned + "unreadable" (stale addresses by the time of the walk; the walk must + happen on FRESH pointers in the same pass as the dump). +* Content search for the distinctive init-stream prefix (`00 80 01 20 + b5 c5 00 00 04 00 04 20`) found the window pages: + 0x107c47000+0x0 (the injected init page), 0x103c06000+0x858, + 0x11dbfa000+0x858, 0x126d69000+0x59c — the later pages hold the same + engine-init streams at arbitrary offsets; the VA<->page relation is + NOT linear (the +0x59c push appears in pages whose +0x0 is zero), so + the mapping cannot be reconstructed by offset alone. +* NEXT: (a) dump the memdesc + kcore-walk its page-list pointers in ONE + guest pass (fresh addresses); or (b) inject the content-matched pages + at trial VAs and watch the service's reads; or (c) the service-side + decode of the window page lists. The client (2632) blocks in + os_schedule at the allocation — the unread later pushes are the + suspected gate. + +## Session 88 addendum - the push-page fix, the gdb-kill wedge, the ring-mapping gap + +**1. PATH-5 RULE (the push pages).** A GR window's pushes do NOT live at +the window base page: the r689-family windows carry their method streams in +WINDOW PAGE 5 (va offsets 0x52d4/0x559c/0x586c/0x596c all live in +0x5000-0x5fff). Standing injections at the base page resolve pa=0x0 for the +pushes (the service's `sema-pushall ... pa=0x0` lines) and the channel +holds/skips entries as "GR no-capture". The fix: page PA = +match_filoff - (va & 0xfff) + 0x80000000 for a content match of the push's +first8 bytes constrained to the same in-page offset; then +`echo 0x<va&~0xfff>,0x<pa>,0x1000,1 > .../sema_map_add`. Proven on r689: +channel beec0104's pushes at va=0x4d00000 (page -> PA 0x12f398000) went +from first8=00000000 to first8=20012000 0000c5c0, and `fast sweep` +applied 3281 -> 3809 (+30/sweep sustained). + +**2. host_resolve.py** automates 1: it polls dmesg for sema-pushall lines +with pa=0x0 and non-zero first8, searches /dev/shm/vgpu-guest-ram +(offset-constrained), and injects the page. The service's 6 s no-capture +hold is the race window - run it BEFORE the client and keep it polling. + +**3. NEVER SIGKILL AN ATTACHED GDB.** On r689, killing the stale t91.py +gdb (kill -9 on PID 5334) killed its tracees (client threads +2883/2884/2885/3048); one held the guest RM API rwsem write lock; the lock +leaked and every later RM call blocked (`[vkps] Update blocked for more +than 1228 seconds` in rmapiLockAcquire -> RmGssLegacyRpcCmd -> +_nv04ControlWithSecInfo). The t91 harness design is correct: `timeout 500` +SIGTERMs the gdb and it detaches cleanly. If a gdb must go, use SIGTERM +and wait; SIGKILL only when the guest is about to be cycled anyway. A +leaked kernel rwsem cannot be repaired live. + +**4. The r690 gap: the GR channel ring mapping.** Fresh cycle r690 (mdev +f5a796b0): every RM RPC of the client init is answered (GR ctx commit, +channel allocs class 0x9067 hObject 0xcef00000/0xcef00080, EVPOSTs), but +the service reports `gpfifo done=0 nomap=938 pushes=0` - it cannot read +any GPFIFO ring, so no entry decodes, no pushall line appears, and the +guest's completion poll loops on RM control 0x20800a70 until the ~10 s RM +timeout -> vkCreateDevice returns -4 (VK_ERROR_DEVICE_LOST). r689's +service had done=19108 (rings mapped) - so the ring mapping is the +missing ingredient for a fresh client in r690; candidate paths: the FIFO +alloc params (12 bytes, GPFIFO VA inside) are not logged; the scanner saw +the CE window (ipins 4592) but no GR ring this boot. Next: log the 0x9067 +alloc params (or scan the guest for the ring VA) and inject the ring page. + +**5. gr_walk.py / gr_walk2.py** are the guest-side kcore walkers (fresh-pass +memdesc discovery; 12 records found, kptrs readable in-pass) kept as the +alternate grounding path. + +## Session 89 addendum - THE PT WALK WORKS (live); the completion-word gap + +**1. GMMU PT walk resolved LIVE (the mission's PT-binding step: DONE).** +The service's own walker math (svc_gmmu_walk: root_bits=ilog2(entries), +shift=49-root_bits; L1 shift38, L2 shift29, PD0 idx=(va>>21)&0xff with 16-byte +dual big/small entries, PT idx=(va>>12)&0x1ff; PDE bit0=0, PTE bit0=1; +pte_addr=(e>>8)<<12) was replicated host-side over /dev/shm/vgpu-guest-ram +(walk.py / live_walk.py). The LIVE published pdb (dmesg: "page directory +... pdb=0x... entries=4" - a NEW pdb per client, e.g. 0x105f0a000 / +0x12a359000 / 0x107012000) resolves every client VA while the client runs: +ring 0x44a0000 -> PA, windows 0x4480000/0x4485000/0x4ca0000..., releases +0x4213000/0x45a0bb0, and the injection via sema_map_add (ap=1 sysmem) makes +the service read them. VALIDATED BY CONTENT: the ring page held exactly the +GPFIFO entries the service logged (entry[1] raw=8000040004480598 va=0x4480598 +len=4), and the push page at va=0x4480598 held the method stream +(...0x20020047...). Post-mortem reads fail: the guest frees/zeros the +structures and pdb pages when the client dies - ALL resolution must run +LIVE during the client's ~11 s life (live_walk.py polls dmesg every 0.4 s, +parses pdb + VAs, walks, injects; static family list pre-seeds the +deterministic VAs). + +**2. Progress with the walk+inject live: gpfifo done=0 -> 6-9.** +With the live injections the service decoded and CONSUMED the whole GR ring +(entries raw logged, releases decoded: rel=0x4213000 pay=1, rel=0x45a0bb0 +pay=3, have_rel=1) and submissions reached 6. The client STILL ends +vkCreateDevice -4 (~11 s, RM 10 s poll of control 0x20800a70) because the +releases are decoded but NOT APPLIED and the client's completion counter +page is unresolvable. + +**3. Why releases do not apply (two gates found).** +(a) svc_fwd_defer_arm(): with gsp_phys_gr_forward=1 a GR ring candidate that +is valid but not "offered" keeps defer armed - every GR release goes to the +forwarded-hardware path (which never runs) instead of the inline software +apply. Cycling with gsp_phys_gr_forward=0 removes the arm, but the release +still does not apply - the decode's apply path is additionally gated by +svc.op_tr_pending (a global "operand targeted scan in progress" state) via +svc_opdef_release: while op_tr_pending=1 every release is recorded into an +opdef slot and withheld, silently when the slot already contains it. The +live symptom: `counter page 0x20400f000 unreadable via +vaspaces/discovered/targeted; slot stays pending` repeated at the client's +poll rate plus repeated `targeted resolve va=0x20400f000: full RAM scan +found nothing`. + +**4. The counter/notifier page record tables (host-side resolve).** +Searching the RAM for the counter-page VA as u64 finds 16-byte {pa,va} +record arrays, e.g. {0x5FEDC000,0x20400D800},{0x5FEDD000,0x20400E000}, +{0x5FEDE000,0x20400E800},{0x5FEDF000,0x20400F000},... (VA step 0x800, +PA step 0x1000) and a second variant ({0x5FEEE000,0x20400E000}, +{0x5FEEF000,0x20400F000}). These are how the counter page (and probably +the notifier, notif=0x1xxxxxxx0 per registration) can be resolved WITHOUT +the VA-space walk: search the VA u64, parse the neighbouring u64 as PA, +inject. Must be done live (records vanish post-mortem). NOTE: the service's +own targeted scan reports "found nothing" for this VA - the record layout +the host parses is not what that scan looks for. + +**5. Next steps (ranked) for session 90.** +1. Extend live_walk.py with a record-resolve fallback: for any VA the + service logs as unreadable/pending/stalled (regex set already widens), + search the RAM for its u64, parse {pa,va} records (both orders), inject + candidates; feed it counter-page (0x20400f000 family), notifier + (notif=... per registration) and release VAs BEFORE the decode (static + list already covers the deterministic ones: add 0x20400f000 with the + record-resolve because the pdb walk cannot reach the UVM/counter space). +2. Or fix the two release gates kernel-side: (a) don't arm the GR deferral + when the forwarding engine has no proven channel (offered flag); (b) + treat op_tr_pending as per-request state (clear when no scan is actually + in progress for the entry) so a stuck global does not withhold every + release. Both are small, testable changes (gsp_sema_service.c). +3. Watch for the first live sign: `rel-apply ... written` / `pub-adv` / + `counter auto/derived` lines on the client's channel; then VKDEVICE + should pass and the flow proceeds to vkAllocateMemory. + +**6. Counter-page record resolve PROVEN LIVE (session 89 end).** +`echo 0x20400f000,0x5FEDF000,0x1000,1 > sema_map_add` (the {pa,va} record +found by the u64 search) made the service learn the slot: +`sema-chan: counter auto h=0xbeef0100 rank=0 pa=0x5fedfff0 (slot +va=0x20400fff0)` and the completion publish went from ZERO to 172 calls +(`counter pub_calls=72 -> 172` over the client's life; previously 0 on every +boot of this session). The client still ends vkCreateDevice -4, so the +remaining unmet wait is the GR ctx-init release word (rel=0x4213000 pay=1, +rel=0x45a0bb0 pay=3 - decoded, have_rel=1, never applied: no rel-apply / +pub-adv line for either). With the counter now advancing, the next +instrumented step is naming the exact GR release-handling branch that +swallows the apply (rebuild with a bounded print at each branch of +svc_rel_apply_906f / svc_opdef_release / svc_rel_apply_launch), then fix +that gate. The gr_forward defer-arm gate is already understood (see 3a) and +gr_forward=0 is the correct test configuration; the remaining suspect is +the service-global svc.op_tr_pending path in svc_opdef_release (it records +the release into an opdef slot and returns true silently when the slot +already contains it). + +**7. Session 90: the CURRENT-boot wait loop site (the old recipe no longer matches).** +The client's long spin on this driver/boot sits INSIDE vkCreateInstance (bt: +libEGL -> eglcore -> 0xB2852D helper), a bounded yield loop at +eglcore+0xB2852D: `mov (%r11),%r12d / test $1,%r12b / jne <exit> / +mov 0x4(%r11),%eax / cmp $1,%eax / jne <retry>` - the loop waits for an +IN-PROCESS word pair at r11 (e.g. 0x7ffc2a1c5080, stack/heap; guest-written +only, the host RAM file cannot feed it because it is not a GPU buffer). +Evidence the old site moved: t91's breakpoint at eglcore+0xa85d24 never +fires this boot (`clears=0`) and a read-watchpoint (t80r.py) on the +0x80 +words of the two 16 KB rw-s nvidiactl mappings does not fire either (the +16K event-buffer wait of sessions 77/78 is not the live target now; the +spin stack is +0xB2852D <- +0xA5325A <- +0xA539AD <- +0xA47491 <- +0x9FF515 +<- +0xA20D04 <- libEGL, NOT the 0xA85F1A/A8 path). A widened multi-word +nudge (nudge2.sh/guest_words2.sh: all rw-s nvidia mappings <=256K, +0x80) +DID fire on 20+ words but the client still returned -4 - the session-77 +caution stands: multi-word writes trip the error path; single-word only, +after identifying the true target. +NEXT: (a) rwatch WRITE on r11+4 to name the writer of the in-process word +(the component that must run; if nothing ever writes it, disassemble the +exit branch 0x...ebc9 vs retry 0x...eb93 to see the pass/fail contract); +(b) the same for the 0xA53xxx caller chain - one of these callers is the +A8/NvOgl GL-ucode wait of sessions 78/79 (its completion comes from the +emulated GR engine path); (c) only then re-test the count-clear + single +word nudge recipes. + +**8. Session 91: THE vkCreateInstance WALL BREACHED (the marker-clear recipe).** +The client's retry loop (eglcore+0xB33D30) waits until NO event-slot entry has +the pending marker 0xffff: per entry ptr = *(u64*)(obj+0x1428+i*8), +cnt = *(u32*)(obj+0x22efc), the checked slot = ptr + n*0x10 (n = the call's +esi), marker = u16@+0xe == 0xffff (or u32@+0 when (esi-0x43)<=4) via the +check at +0xB1C750 (OR of all pending markers). The guest's event dispatch +NEVER clears it, so the client burns its 10 s -4 timeout. PROVEN FIX: +clear {u16@slot+0xe, u16@slot+0xc, u32@slot+0} to 0 at every hit of ++0xB33D30 (t93 single-shot; t94 auto-clearer, 4000 hits) -> the client +passes vkCreateInstance AND device enumeration FOR THE FIRST TIME +(`VKINSTANCE ok / VKDEVS 2 / VKPICK`, previously the 10 s -4 fired during +this phase). t94 cleared 5+ distinct module objects (obj=0x5d67063a84a0, +0x5d670661ad80, 0x5d670673eed0, 0x5d67067ebb50, ...) over the client's life. +The client then sat in vkCreateDevice's poll(/dev/nvidia0, nfds=1, +timeout=10 ms) loop at eglcore+0xA12DDF..+0xA12DE4 WITHOUT any -4 for 5+ +minutes (vs the old ~11 s death) - the longest the client has ever run. + +**9. The remaining gap (single): the event-fd signal.** The device-phase wait +is a poll() on /dev/nvidia0 (fd 27) - nvidia_poll wakes only when the guest +RM signals the client's event state; the guest consumes our POST_EVENT +records but never performs the event signaling (session 78's "residual +delivery gap"). Writing to /proc/PID/fd/<nvidia0> is not a signal. The +session-84 count write (base+0x2070b7c, read 4, wrote 0, t97) did not flip +this wait alone. NEXT: fix the host POST_EVENT -> guest dispatch association +(gsp_rm_emulation.c r579 area: the payload fields hClient/hEvent/notifyIndex/ +data/info/status; compare against the guest's _kgspRpcPostEvent -> +gpuNotifySubDeviceEvent expectations) so the guest signals the event and +nvidia_poll returns POLLIN; then the same t94-pattern (or a service-side +equivalent) durably unblocks both waits per boot. + +**10. Session 92: THE AUTOMATED PIPELINE - fresh boot to VKDEVICE with ZERO manual steps.** +On a fresh r692 boot (cycle params unchanged, gr_forward=0) a single gdb +script (t99.py) carrying two breakpoint handlers drove the client end-to-end: +(a) the instance-phase marker cleaner at eglcore+0xB33D30 (t94 pattern: clear +u16@slot+0xe / u16@slot+0xc / u32@slot to 0 per entry at ptr+n*0x10 for +entries *(obj+0x1428+i*8), cnt *(obj+0x22efc)) - fired CLEAR#1..#5 across +five module objects; (b) the device-phase wait-word chaser at +eglcore+0xDCEBD9 (t79 chain: rbx -> *(rbx+0x70) -> *(sub+0x58)=obj2 -> + off=*(u32)(obj2+0x60), holder=*(obj2+0x58), base=*(holder+0x58), +word=base+off) - found WORD addr=0x734781a86080 val=0x0 and wrote 1. +Result: `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok family=0` with no +manual intervention (r692 client 1725). The client then sat in the +POST-DEVICE wait (the next phase - allocation/engine-init), which t99 does +not cover. The live_walk PT-walk injector ran alongside and injected the +window/ring VAs (0x4d20000, 0x4923000...; note 0x4b19b10 resolved ap=FB - +the GL init release is vidmem-mapped and skipped by the SYS-only injector). + +**11. HARD LESSON (attach discipline, again).** A gdb attached to a client +that never hits its breakpoints wedges: `continue` never returns, the +timeout's SIGTERM does not tear it down, and the ptrace slot blocks all +later attaches (hit 3x this session: 146084, the t91-on-1725 attach, both +left holding). RULE: only attach with breakpoints that are KNOWN to fire +for the client's current phase, keep windows <= ~60 s, and prefer +self-terminating handlers (hit caps). A wedged attach leaves the client +RUNNING (state R) but unattachable; cycle the VM for a clean environment +when more attaches are needed. + +**12. Continuing the chain (next session).** (1) Identify the post-device +wait of the VKDEVICE-ok client (bt sample once attachable; likely the +allocation/UVM phase or the A8/NvOgl engine-init with its count gate at +lib_base+0x2070b7c=4 -> 0 and/or the 0xdceb80 wait loop - add the t91 +count-clear BP to the t99 bundle if the A8 path is live). (2) Extend t99 +into the "durable harness bundle": marker-clean + word-chase + count-clear ++ (once VKFRAME) the release/alloc phases. (3) The bNotifyList=1 +(gsp_rm_ev_nlist) path stays armed for the event-signal experiment; +post/EVPOST logging re-budgeted (gsp_rm_ev_log=2000 set). + +**13. Session 93: THE ZERO-PTRACE PIPELINE (r693) - VKDEVICE with no gdb-wedge risk.** +Split the harness by risk: (a) ONE gdb (t101.py) for the INSTANCE-phase +marker clears only (base+0xB33D30 handler, clears u16@+0xe/+0xc and u32@+0 +of each entry; self-disables after 24 clears; NO other BPs so it cannot +fire constantly); (b) the WAIT WORDS handled by /tmp/unwait.py - a guest +python poller with ZERO ptrace that scans the newest vk_frame's rw-s +nvidia mappings of exactly 0x4000 bytes and writes 1 to any zero u32 at +mapping+0x80 via /proc/PID/mem (the session-77 geometry, confirmed by t79 +chains: word = the +0x80 of a 16 KB rw-s nvidiactl mapping). Result on +fresh r693 (mdev 261980af): t101 CLEAR#1..#5 then idle; unwait wrote FIVE +wait words (0x7876ead57080, 0x7876f1816080, 0x7876ea47f080, +0x7876ea81a080, 0x7876ead51080); client reached `VKINSTANCE ok / VKDEVS 2 +/ VKPICK / VKDEVICE ok family=0` with NO wedge exposure. live_walk (PT-walk +injector) ran alongside. The client then sat in the post-device wait that +the +0x80 poller does not cover (t102.py = the one-shot safe diagnostic: +attach -> print PC/base/A8-count(base+0x2070b7c)/bt -> detach, NO continue +and NO breakpoints - use it when the slot is free). + +**14. THE RM-LOCK WEDGE ROOT CAUSE + RULE.** A gdb killed (SIGTERM) while +the tracee sits AT a breakpoint leaves the tracee STOPPED; if that stop is +inside an RM api critical section (the rwlock held across an RM call!), the +RM write lock stays held and EVERY other client blocks in +os_acquire_rwlock_write (state D) - observed directly: old client 1725 +stopped by its wedged t99 gdb -> new client 7967 hung `state=D wchan= +os_acquire_rwlock_write` on ioctl 0xc020462b; killing the gdb + `kill -CONT` +resumed 1725 out of the ptrace stop but it resumed INSIDE the same RM wait, +so the lock stayed held until the VM was cycled. RULES: never let a gdb +timeout fire while a tracee is at a frequent-firing BP; prefer BPs that +fire once or self-disable; prefer no-ptrace pollers for anything with a +repeating pattern; the safe diagnostic is attach -> read -> detach without +continue. + +**15. Session 94: THE INSTANCE CLEARS ARE REQUIRED (zero-gdb disproved); t104 is the safe form.** +Pure zero-gdb run (fresh r694, client + unwait.py ONLY, no gdb): unwait +wrote three +0x80 words and the client still ended `VKFAIL vkCreateDevice +... -4` - WITHOUT the instance-phase marker clears the device gate fails +even with the words. With t104.py (single BP at +0xB33D30; clears +u16@+0xe/+0xc/u32@+0; self-exits on cleared>=8 OR chits>=40 OR 25 s since +the last clear -> returns True so the script tail runs and batch gdb exits +cleanly, no outer-timeout dependence) + unwait + live_walk: CLEAR#1..#5, +two words written, client reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / +VKDEVICE ok family=0` (r694 client 3725). The client then sat in the +post-device wait again; the t102 diagnostic could not attach because the +t104 gdb still held the slot (its 25 s-idle exit never fired - no +post-instance B33D30 hits - and its 150 s timeout likely fired at a stop += the known wedge risk). NEXT FORM: make t104 detach ITSELF from inside +the handler once the clears are done (`gdb.execute("detach")` from the +stop handler; the tracee resumes; the script ends and gdb exits) so NO +gdb is ever held while the client runs - then t102 always has the slot +and the wedge class disappears end-to-end. + +**16. The allocation/post-device phase is the remaining gate.** With +VKDEVICE green the client waits in a site that is neither the +0x80 word +pattern nor (likely) the A8 count path; the next session should (a) get +the detach-first t104 live, (b) run t102 immediately after VKDEVICE to +name the wait (read PC/bt/A8 count = base+0x2070b7c), and (c) inspect the +service for this client (pushes decoded / releases applied / counter +pub_calls) to see whether the allocation needs the live injections only +or a service-side gap. + +**17. Session 95: THE GDB THREAD-STOP INTERFERENCE + the SIGKILL nuance.** +The t105 attach (t=1.5 s, before the client's worker threads spawned) left +four worker threads in `ptrace_stop.part.0` (futex) - the client's main +thread span waiting on them, so the client froze at the post-device wait +with the gdb holding the trace. TERM/INT/HUP on the wedged gdb were all +ignored (batch gdb stuck in continue). SIGKILL on the gdb (1584) was tried +with the reasoning that all stopped threads sat at futex (NO RM lock held): +result - the tracer died, the tracees DETACHED+RESUMED (client 1518 +SURVIVED with its workers back in futex_wait_queue; contrast session 88 +where SIGKILLed-tracer tracees were KILLED mid-syscall). So the SIGKILL +outcome depends on the tracee's stop context; the mission's never-SIGKILL +rule stands, but this datapoint matters when judging recoverability. +After the gdb died, 1518 still held the RM write lock from its own +post-device wait (its wait word was no longer polled because unwait.py +tracks the NEWEST vk_frame and a fresh test client 6555 was launched and +blocked D-state on os_acquire_rwlock_write) - the unwait-newest coupling +must be replaced by ALL-clients polling in the next form. + +**18. HARNESS DESIGN RULES (accumulated).** (1) A gdb attached across +thread-spawn windows stops worker threads and freezes the client - attach +only AFTER all threads exist, or ensure the gdb explicitly resumes new +threads, or do the work with NO gdb. (2) Any gdb that must run through +thread-spawn windows MUST self-detach quickly (the instance phase spans +~10-20 s and the clears arrive in ~5 bursts) and NEVER rely on outer +timeouts. (3) unwait.py should poll EVERY live vk_frame (not just the +newest). (4) The durable answer for all of this is the service-side +folding: the instance markers and the +0x80 words are guest-RAM writes a +kernel-side service can perform on the same triggers (the marker semantics +and the word semantics are understood) - that removes gdb and /proc +helpers from the loop entirely. + +**19. Session 96: THE DETACH-FIRST CLEARER PROVEN (t106) + the orphan-tracer plague.** +t106.py (single +0xB33D30 BP; clears markers; detaches FROM THE HANDLER as +soon as cleared>=5 or chits>=14: `gdb.execute("detach")` + return True so +the batch gdb exits) on fresh r696 (mdev 241af1c1): CLEAR#1..#5, +`BC-DETACHED cleaned=5 chits=5`, `T106-DONE chits=5 cleared=5`, +`TracerPid: 0` - THE SLOT IS LEFT FREE. unwait.py wrote five words and the +client (1549) reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok +family=0`. THE CLEAN FORM OF THE CLEARER EXISTS NOW. + +**20. THE ORPHAN-TRACER PLAGUE (root cause of every t102 failure).** Every +INTERRUPTED gdb attempt leaves an attached orphan: (a) piping gdb to +`head` closes the pipe early and the gdb stays attached (hit twice); (b) an +ssh command dying/timing out mid-gdb-add leaves the gdb running (hit +again). Symptoms: later attaches fail with `Could not attach ... already +traced by process N` / `ptrace: Inappropriate ioctl for device` and the +t102 log stays empty. RECOVERY (session-95 evidence): TERM is usually +ignored (wedged in continue); SIGKILL the orphan + `kill -CONT <client>`; +the tracee SURVIVES when it was running/at-futex and detaches cleanly +(TracerPid 0). STRICT DISCIPLINE: never pipe gdb output (redirect to a +file, read it after); never let an ssh command end while a gdb may be +attached (generous timeouts); after every gdb run verify TracerPid 0. + +**21. THE NEXT FORM (t107, one gdb, one window).** Combine the proven +clearer and the diagnostic in ONE batch gdb: attach -> clears -> detach -> +`python time.sleep(...)` idle -> RE-ATTACH (gdb can attach again after +detach) -> read PC / bt / A8 count (t102 content) -> detach -> print - +one ssh session, one window, everything self-contained, no orphan risk +(every path ends in an explicit detach; wrap each phase in try/except with +a final detach). Expected: names the post-device wait (allocation/UVM vs +A8 count base+0x2070b7c) so the allocation can be passed and the first +VKFRAME reached. + +**22. Session 97: THE DEEP SUBMISSION STATE + THE COUNTER-SLOT WEB.** +The r696 client (1549) reached the deepest state yet: the service reported +`gpfifo done=11576 pushes=10932 counter pub_calls=1356` and climbing +(gpfifo done was 0 in the pre-pipeline boots) - the client submits the full +work stream and the service decodes/processes it (pairs=14395 map=14318, +ipins=9240 from live_walk). The client then sits in the KNOWN userspace +post-device spin (main thread `running` in all samples; one tid in +os_acquire_rwlock_read). NEW: a counter-slot collision spam +(`counter auto ... pa=0x5fedfff0 rejected (slot owned by another channel; +not installed)`, 232+ callbacks suppressed) - the 0x20400f000 injection +resolved the WHOLE counter VA family to ONE page, so both live channels +collided; the manual bypass `echo <h>,<pa> > .../parameters/sema_counter` +(h_object,pa syntax, main.c:2965) was applied for both handles +(0xbeee0100,0xbeec0100 -> 0x5fedfff0) and pub_calls jumped 1356 -> 3739. +The client gate (VKDEVICE -> frames) persists: its remaining wait is the +userspace spin word NOT covered by the +0x80/16K poller (candidate: the +tracker/counter the service now advances - watch whether pub traffic +alone resolves the spin, else t107-GDB the spin site). ENVIRONMENT: one +orphan gdb (10092 on 1549) + stale channels (beec0100) from prior +attempts; the orphan-kill discipline applies (SIGKILL by PID + CONT + +verify TracerPid). + +**23. THE IOCTL-ARG SAMPLING AXIS (no ptrace).** /proc/<tid>/syscall shows +the running ioctl's ARGS (fd, cmd, arg) when caught - sampling the main +thread repeatedly during an ioctl-loop phase names the exact stuck RM +control without any gdb. Use it before ptrace-based diagnostics when the +thread does syscalls at a catchable rate. + +**24. Session 98: THE POST-DEVICE SPIN NAMED - THE SEC2 SCRUBBER AUTH-TAG WAIT.** +perf (no ptrace!) on the spun client: top symbol `channelReadChannelMemdesc` +24.33% (then scheduler internals) - the guest's nvidia driver hot-loops +reading the CHANNEL MEMDESC. Source decode (nvidia-570.144-reference, +sec2_utils.c): the only reference caller is +`_sec2utilsUpdateGetPtr` -> `channelReadChannelMemdesc(pChannel, +authTagBufSemaOffset)` inside `_sec2utilsGetNextAuthTagSlot`, whose loop +waits for the scrub auth-tag slots to free; `authTagBufSemaOffset = +finishPayloadOffset + CHANNEL_ENGINE_SEMAPHORE_SIZE` (4). So the spin = +the SEC2 MEMORY-SCRUBBER channel waiting for its auth-tag semaphore (a u32 +inside the channel memdesc buffer) to advance - the GPU never delivers it +in our emulation, so the client waits forever (each poll = one full memdesc +transfer = the 24% cost). THE FIX DIRECTION: the service must complete the +scrubber's semaphore updates - the scrubber channel registers in the +service (`scrubber channel 0x... detected (gpfifo=... userd=...)`) and its +submits carry semaphore releases targeting the auth-tag VA +(pbGpuVA + authTagBufSemaOffset) in the channel buffer; on decode, write +the release value to that VA's page (the channel buffer is a guest-RAM +allocation the service can resolve/pin; the pending queue holding 17 +entries is the suspect pool for the unresolvable ones - inspect +svc_pending[]). Service-side counter to watch after the fix: the scrub +loop exits, the client proceeds to the allocation/pipeline. Tool: perf +record/report on the guest is the no-ptrace spin-pointer (`perf record -p +<pid> -- sleep 3; perf report --stdio --sort=symbol,dso`). + +**25. Session 99: THE SCRUBBER GATE CONNECTED TO THE CE CHANNELS' ok=0.** +chan_dump on r696 (client 1549 spun): six channels - +`h=0xbeee0100 eng=0 gpfifo=0x44a0000 userd=0xfee00 ok=1 get=6 sub=6354 rel=0x4213020 crel=0x5fedfff0 notif=0x11cd51060` +`h=0xbeec0100 eng=0 gpfifo=0x4cc0000 userd=0xfee00 ok=1 rel=0x4923020 crel=0x5fedfff0` +`h=0xbeec0104 eng=0 gpfifo=0x4d20000 userd=0xfee00 ok=1 rel=0x4b17e30 crel=0x5fedffe0` +and the CE-class channels `beee0101 (eng=11, gpfifo=0x4500000 userd=0x8f600 as=2 ok=0)`, +`beee0102 (eng=12, gpfifo=0x4560000 userd=0x8f400 ok=0)`, `beed0101 (eng=12, +gpfifo=0x49c0000 userd=0x8f200 ok=0)` - ALL ok=0: the CE userd discovery +(failed probe: EAGAIN 0/0 or EINVAL) means no CE pump, no CE sema applies, +so the SEC2 scrubber channel (a CE-class channel) never sees its auth-tag +semaphore advance and the client spins in _sec2utilsGetNextAuthTagSlot. +The service's candidates (registered PA, ring+0x10000 in both apertures, +8-slot list) all miss for the CE channels. `find_userd` (FB scan for a +live put@0x8c/get@0x88 pair) found ZERO candidates - the CE userds are not +in FB with live puts (either the client's PUT writes go to a page the +service's candidates miss, or the CE userds genuinely idle). A raw host-RAM +shape scan (put in 1..65535, get<=put) is too noisy (24 hits in the first +28KB of BIOS-era low RAM) - needs tighter validation. NEXT APPROACH: find +the REAL userd page by DIFF-WATCH - sample the candidate pages (and the +CE ring pages) twice around a scrub submit and diff for the page whose ++0x88/+0x8c pair changes; the guest kernel writes GP_PUT there, so the +changing page is the true userd. Then fix the discovery (or add a +sema_userd-style injection param) so the CE pump engages and the scrubber +completes. NOTE: the GR channels' userd=0xfee00 (as=2, FB) and ok=1 with +get=6/sub=6354 - the GR machinery runs fine; only the CE/scrubber path is +dark. + +**26. Session 100: THE CE USERD HUNT EXHAUSTED THE STANDARD LAYOUTS; THE CIRCULAR WAIT.** +Candidates examined for the CE userds this session: (a) the registered FB +offsets (0x8f600/0x8f400/0x8f200 as=2) - ALL ZERO in the FB shadow +(sema_vidmem_dump 0x8f100,0x600 twice); (b) the ring-PA + 0x10000 linear +guess - random/zero/table garbage (the allocations are NON-CONTIGUOUS); +(c) the PDB walks of gpfifo+0x10000 (0x4501000->pa 0x125e91000, +0x4561000->pa 0x132032000, 0x49c1000->pa 0x11bddd000 via pdb 0x1152f4000) +- ALL THREE PAGES ALL ZERO. The CE userds were never written - and the +deeper reading: the SEC2 scrub flow is a CIRCULAR WAIT: the scrubber waits +for its previous scrub pushes to be CONSUMED (the auth-tag sema advances +only when the GPU runs the scrub) -> the consumption needs the CE pump -> +the CE pump needs a live userd -> the userd is written BY SUBMITS -> the +submits are held by the wait. In a working boot the guest's first scrub +push happens BEFORE the wait loop (the loop only waits when the slots are +full) - so in our boots the FIRST scrub push must have gone into the CE +ring, the service failed to consume it (ok=0/userd dark), and the client +now waits for a completion that will never come. + +**27. THE DIRECT BYPASS (next session, ranked).** (1) Write the auth-tag +semaphore in the CHANNEL BUFFER (memdesc) page directly: the scrub loop's +`_sec2utilsUpdateGetPtr` reads `channelReadChannelMemdesc(pChannel, +authTagBufSemaOffset)` = a u32 at memdesc+finishPayloadOffset+4; advancing +that value frees the slots and unblocks the client WITHOUT the CE pump. +The memdesc = the channel buffer allocation (a VA in the client's space); +resolve it via the PDB (the guest's channel struct gives the buffer VA; +the service's channel registration/alloc params carry it) and write the +incremented value from the service (or the host RAM file). (2) OR decode +the scrub pushes the service already sees (check sema-push for the SCRUBBER +channel - its submits carry the auth-tag VA; the release decode should +target it; the service's release applies would write the sema if the VA +resolves). (3) The registered FB userd offsets being zero suggests the +client's kernel maps the userd through a VA whose PA the discovery misses; +the diff-watch (two samples around activity) remains the fallback proof. + +**28. Session 101: THE LOOP SEMANTICS + perf trace (NO-PTRACE SYSCALL TRACING).** +The wait loop decoded exactly (sec2_utils.c:420-441): +`while (nextPut == pSec2Utils->authTagGetIndex) { gpuCheckTimeout(); +authTagGetIndex = _sec2utilsUpdateGetPtr(); osSpinLoop(); }` - it exits +when the memdesc sema (u32 at pChannelBufferMemdesc + +authTagBufSemaOffset; authTagBufSemaOffset = finishPayloadOffset + 4) +advances past nextPut; it also has a GPU timeout that would error out. +perf trace on the spun client (no ptrace!) revealed the live pattern: +tid 1793 = a 10ms-timed futex loop (WAKE val=1 then +WAIT_BITSET|CLOCK_REALTIME 10ms - the paced retry); tid 1788 = repeated +`openat -> read(fd, 299 bytes) -> close -> futex-wait` - a transient-fd +POLLING READER of a small blob (the fd is opened and closed each cycle, so +fd 28 no longer exists; the client's persistent fds: /dev/nvidia-modeset +(10), /dev/nvidia0 (11/12/14/15), GLCache .toc/.bin (16/17), memfd +\"allocation fd\" (18)). The 299-byte read is the likely userland face of +the memdesc read path (the 24% channelReadChannelMemdesc hot loop). +NEXT: (a) catch the transient fd path by sampling `ls -l /proc/<pid>/fd` +every ~0.2 s in a loop around the burst (or perf trace -s during a burst); +(b) if it is a driver read() of the memdesc, resolve its buffer and write +the auth-tag sema value (advance past nextPut) via the RAM file; (c) the +loop timeout (gpuCheckTimeout) may eventually error the client - the write +must land first. + +**29. Session 102: fd-sampler miss, no nvidia debugfs; the memdesc hunt narrows.** +The 0.15 s /proc/1549/fd sampler over 20 s caught NO transient path (the +open/read(299)/close reader is bursty or dormant between bursts; the static +fds are /dev/nvidiactl, /dev/nvidia0, /dev/nvidia-modeset, a socket, the +GLCache .toc/.bin pair, and a memfd "allocation fd"). No nvidia debugfs +tree exists in the guest. Remaining ranked paths to the auth-tag sema +(memdesc + finishPayloadOffset + 4): (1) content-locate the scrubber's +channel buffer by its method stream (the C5B5/C56F shapes: SET_OBJECT +0xc5c0/0xc5b5, headers 0x200121ed/0x200120a9 observed in the session-78 +window dumps) and read/write the sema inside; (2) the scrubber channel's +own sema-push decodes (its pushes carry the auth-tag VA = pbGpuVA + +authTagBufSemaOffset; grep the boot's sema-pushall for the scrubber +channel id and read the rel= VA - then PDB-walk it); (3) sample the fd +burst narrower (0.02 s) right after a client restart when the burst is +hot. The service-side ideal: on detecting the scrubber channel, resolve +its channel buffer and pre-advance the auth-tag sema (a durable fold). + +**30. Session 103: THE SCRUB STREAM FOUND; ONE RELEASE DELIVERED; THE SEMA PAGE STILL AMBIGUOUS.** +Content-located the CE/scrub init method stream (headers 0x200121ed/ +0x200120a9 searched): full stream at `pa=0x114e98000` = SET_OBJECT +0xC5C0/0xC5B5 (0x20012000/0x20012080 headers), engine-init methods +0x200121ed(3)/0x200120a9(1)/0x2001209c(0x0c000001), bulk methods +(0x8000xxxx), then TWO REPORT_SEM releases in one buffer: +`0x200426c0 00000000 04b19b10 ...` (va=0x4b19b10) and +`0x200426c0 00000000 04b17e30 ...` (va=0x4b17e30), tail 0x2001c00d +70/68. Twins at pa=0x1026ad000+0x87c and pa=0x12afba000+0x87c (identical +content at identical offsets - multi-mapped copies). Release resolution: +`0x4b17e30 -> pa=0x114efbe30 (SYS) cur=1` - THAT release WAS delivered (the +service applied it, cur=1). `0x4b19b10` resolves via the walker's big-half +2MB branch to FB 0x660b10 holding garbage (0xD9F26F94) - poke_vidmem +tests ([0x660b10]=1 then =0x40000000) did NOT flip the client, and the +linear-block guesses (buffer VA 0x4b00000 <-> stream pa 0x114e98000, sema +candidate 0x114eb1b10) read all zero. NEXT: the auth-tag sema page must be +picked by VALIDATION (the existing value must be a small plausible +GetPtr): walk 0x4b19000 returning ALL chains (small/64K/2MB both +apertures), pick the page whose u32 at +0x1b10 (or the offset the memdesc +actually uses) is a small number; or take the buffer aperture from the +channel's alloc params (the service's registration) and walk with that +aperture exclusively. The fix remains: advance that u32 (write via the +host RAM file if SYS, poke_vidmem if FB). + +**31. Session 104: THE SEMA VA 0x4b19b10 IS NOT THE LOOP'S PAGE; THE KPROBE IS THE ANSWER.** +The validated multi-chain walk (small-4KB and big chains BOTH) resolves +VA 0x4b19b10 -> FB 0x660b10 (both chains converge; the guest's own PTs say +the object lives in vidmem). poke_vidmem writes at 0x660b10 (0x1, +0x40000000) and 0x660b14 (0x1, 0x40000000) LANDED (verified via +sema_vidmem_dump: [0x660b10]=0x11112222) yet the client does not flip and +perf still shows channelReadChannelMemdesc at 24.52%. CONCLUSION: the +REPORT_SEM VA 0x4b19b10 (and 0x4b17e30) are NOT the auth-tag sema the loop +reads - they are the GL-init release objects; the real auth-tag sema is at +pChannelBufferMemdesc + finishPayloadOffset + 4 (a VA and page we do not +yet know; finishPayloadOffset is NOT 0x19b10). THE DEFINITIVE INSTRUMENT: +a guest kprobe on channelReadChannelMemdesc (the offset argument = %si = +the authTagBufSemaOffset VALUE; and the pChannel pointer lets us read the +channel struct -> pChannelBufferMemdesc): `echo 'p:crcd +channelReadChannelMemdesc offset=%si' >> /sys/kernel/tracing/kprobe_events` ++ enable the event + read tracing pipe WHILE the loop runs. This names the +offset and the memdesc in one shot, then the sema page = walk(memdesc+off) +and the advance (RAM file if SYS; FB + poke if the converged mapping). + +**32. Session 105: KPROBE BLOCKED (notrace); THE BAR1-PATH ANALYSIS; THE CHANNEL-FIELD OFFSETS.** +(1) The kprobe on channelReadChannelMemdesc is IMPOSSIBLE: the guest's +dmesg says `trace_kprobe: Could not probe notrace function +channelReadChannelMemdesc [nvidia]` - the function is annotated notrace +(ftrace/perf-probe/BPF-kprobe all blocked). The tracefs filesystem had to +be mounted first (mount -t tracefs tracefs /sys/kernel/tracing) and it +already carries UVM kprobes from earlier sessions (uvmt/uvch/uvbp/uvtw/ +uvsv...). The static sec2utils functions (the callers) are absent from +kallsyms (inlined) - no probe target on that path. +(2) BAR1-path analysis: the guest driver's memdesc read uses +TRANSFER_FLAGS_USE_BAR1|SHADOW_ALLOC|SHADOW_INIT_MEM - a CPU read of the +GUEST'S BAR1 mapping, which the mdev mmap maps onto the HOST's vidmem +shadow (the same buffer poke_vidmem writes). The pokes at FB 0x660b10/ +0x660b14 LANDED (verified) yet the loop did not advance, so the memdesc +page is NOT the FB 0x660b10 the PT walk converged on (VA 0x4b19b10 belongs +to the GL-init release object, not the channel-buffer memdesc). +(3) The channel registration parse exposes the guest-visible channel +fields and their alloc-param offsets (gsp_rm_emulation.c +gsp_rm_register_channel: GSPRM_CHAN_GPFIFO_OFFSET/ENTRIES, ENGINE_TYPE, +CID, INSTANCE_MEM, USERD_MEM(+APERTURE), NOTIFIER_MEM(+APERTURE)) - the +channel-buffer/memdesc field (pChannelBufferMemdesc) is not among them; +the next session should dump a CE channel's FULL alloc params (the +service's rate-limited alloc dumps or a fresh boot) and look for the +channel-buffer VA/PA field, then sema = buffer + finishPayloadOffset + 4 +(per-channel offset, e.g. the 0x19b10-style for the GL channel). +(4) Fallback if the field is not in the params: per-VA content hunt for +the auth-tag sema VALUE (the retprobe would have given it; without +kprobes: the sema value = a small slot counter - scan the CE channel's +allocation candidates for a plausible counter that CHANGES across two +samples). + +**33. Session 106: THE CHANNEL BUFFER IS NOT A CLIENT PARAM; THE TAIL SWEEP AND HASH SCAN NEGATIVE.** +NV_CHANNEL_ALLOC_PARAMS (alloc_channel.h:300-341) layout mapped to the +service offsets: gpFifoOffset@8/entries@16, engineType@128, cid@132, +instanceMem@144, userdMem@168, ramfcMem@192, mthdbufMem@216, +hPhysChannelGroup/internalFlags, errorNotifierMem@248... - THE CHANNEL +BUFFER (pChannelBufferMemdesc) IS NOT IN THE PARAMS: it is a +driver-internal memdesc created by channelAlloc, so the alloc-params path +cannot name it (the kprobe/debuginfo path was the right one; it is +notrace-blocked). The finish-payload tail of the stream page +(0x114e98000 +0xd4..+0x118) was swept with 0x40000000 (18 u32s; the +pre-sweep values: +0xd8=0x68000000 finish marker, +0xe0=0x901025a6, ++0xe4..+0xec=0x8000xxxx methods) - no flip. A 16-page high-entropy scan of +0x114e98000..0x114ea8000 for the auth-tag SHA-256 array: ZERO high-entropy +32-byte blocks (the hash array is elsewhere / another allocation). NEXT: +(1) the memmgrMemDescBeginTransfer SHADOW insight - the read happens from +a FRESH kmalloc'd shadow copied from the memdesc each call; the shadow's +source = the memdesc's backing page; the memdesc page must be found by +TRANSFER-TIME properties (not the VA walk); (2) or a guest perf hardware +watchpoint on the driver's BAR1 ioremap VA + the (unknown) memdesc offset; +(3) or the service-side broad assist: increment a RANGE of plausible FB +sema cells periodically (blind but bounded), watching for the flip. + +**34. Session 107: FB ASSIST SWEEPS NEGATIVE - THE READ IS NOT FB-SOURCED; THE TWINS ARE THE NEXT TARGET.** +Two bounded FB "sema assist" sweeps (distinct values per cell, +0x40000000|idx): (a) the walked page 0x660b00..0x661000 (320 cells) - no +flip; (b) the CE-channel allocation cluster 0x8f000..0x90000 (1024 cells, +where the registered userds live) - no flip; perf re-confirmed +channelReadChannelMemdesc 23.13% (the loop still runs). CONCLUSION: the +memdesc read does NOT come from the FB shadow the CPU can poke - the +transfer's SHADOW_INIT must read the memdesc through a path that bypasses +these bytes, or the memdesc backing is SYSTEM memory via an unknown VA. +NEXT (one-minute test): the STREAM TWINS - pa 0x1026ad000 and +0x12afba000 hold BYTE-IDENTICAL copies of the CE method stream (verified at ++0x7c0/+0x880: same 0x68000000 finish marker area, same 0x200120a9/9c +methods) - one of them is very likely the memdesc BACKING in sysmem (the +driver copies the same stream per channel/boot). Sweep BOTH twins' +stream-area pages (reuse the assist code; 1 page each) and watch for the +flip; if it flips, bisect the winning page to the sema cell and make it +durable (host RAM file writes = zero-cost, no poke_vidmem needed for +sysmem). Failing that: content-diff the twins across a driver transition +(the memdesc copy updates during submits; the copy that CHANGES is the +backing). + +**35. Session 108: THE CONTENT-DIFF - 9 ACTIVE PAGES + THE CONSTANT-READ DEDUCTION.** +Page-hash diff over three +/-256KB windows (twins 0x1026ad000/0x12afba000, +stream 0x114e98000) across 15 s: NINE pages change: +0x102690000, 0x102698000 (440 word diffs - a buffer written continuously = +the push copy!), 0x1026a7000 (+0xe8: 934 -> 940 - a slow clean counter), +0x1026b3000 (+0x48: 32768 -> 1 transition), 0x1026b5000 (+0x244/+0x24c: +24987 -> 32765, two u16s at ~650/s), 0x1026c2000 (+0xe8: 232 -> 3, a +wrapping counter mod ~233), 0x1026e6000 (+0x18: 2764 -> 1930, +0x88/+0x90 +decreasing - countdowns), 0x114e6b000, 0x114e6e000. THE DEDUCTION: the +stuck loop's read value must be CONSTANT (its page shows no writes; the +GPU never advances the sema), so the auth-tag sema is a static cell - +likely zero - and the fix is to make the read NONZERO (advance it) at the +right page. None of the 9 changing pages is it (their cells move on their +own). NEXT: (1) restrict the search to CONSTANT pages among the CE +channel's candidate backing (the twins region pages that DID NOT change +but sit adjacent to the active set); (2) target the sema cell: the cell +that equals nextPut (small constant, e.g. 0) at the memdesc+off; sweep the +CONSTANT pages' zero u32s in a bounded ring around the twins/active set +with 1 (the sysmem write is free); (3) the durable service fold once +found. + +**36. Session 109: THE BURST vs HEARTBEAT DISCOVERY + THE 200ms READER.** +perf trace of the whole process over 10 s with -e ioctl: ZERO ioctls from +ANY thread; meanwhile the process shows a strict 200 ms pattern on thread +1788: `openat(filename ptr 0x5eaec740) -> read(fd 28, count 299) -> close` +(the transient fd the fd-sampler missed!) and thread 1793's 10 ms +futex re-arms. The perf-reported 24% channelReadChannelMemdesc (a kernel +function only reachable via RM calls) CONTRADICTS the zero-ioctl window: +the hot kernel work is BURSTY (periodic rounds between the 200 ms +heartbeats), not continuous. The 299-byte blob read every 200 ms = the +client's health poll (unnamed path; candidate: /proc/driver/nvidia/... +informational node - find it by reading /proc/PID/io or by narrowing the +sampler to the 200 ms phase). THE MODEL NOW: the client idles (futex +waits) + wakes every 200 ms to poll a node + periodically runs RM bursts +(the burst = a full retry round hitting channelReadChannelMemdesc). NEXT: +(1) name the 299-byte node (it may BE the wait's condition - a status the +service must change); (2) capture the BURST with perf (a long record + +per-thread breakdown) to see which thread runs the memdesc reads (the RM +burst thread) and correlate with the host-side RPC traffic in the same +window (the host dmesg RPC lines during a burst = the exact RM call!). + +**37. Session 110: THE LOOP IS CONTINUOUS INSIDE ONE LONG IOCTL (THE LOCK MODEL CLOSED).** +(1) The 299-byte/200ms reader: the filename pointer 0x5eaec740 is NEVER in +the client's maps (30 Hz sampler over 15 s, zero hits) yet openat returns +28 every cycle - a decode artifact/transience, and at 5 reads/s it is +irrelevant to the 24% blocker. Treat it as a RED HERRING. (2) A 20-second +perf record (80K samples): channelReadChannelMemdesc 24.65% + a scheduler +family (pick_next_task_fair/update_curr/pick_eevdf/__schedule/ +schedule_timeout/os_schedule ~30% total) = the SEC2 wait loop running +CONTINUOUSLY. The earlier zero-ioctl windows are explained: THE MAIN +THREAD SITS INSIDE ONE LONG-LIVED ioctl (entered hours ago; the loop runs +in the kernel inside it) - and that ioctl HOLDS THE RM WRITE LOCK, which +closes the lock model: tid 1791 waits on the RM read lock, new clients +block D-state (os_acquire_rwlock_write), all consistent. (3) PEBS/ +mem-loads events are unavailable in the guest perf (no memory-address +sampling) - the read address cannot be sampled. Instruments for the sema +address are now exhausted short of the notrace blocker or the blind +service-side assist. +NEXT: (1) the blind assist, refined: the loop's READ is CONTINUOUS and +the sema is CONSTANT - sweep ONLY constant small-value cells in the CE +candidate pages (see session 108's active set for the region), one value +at a time, 2 s per batch, watching the flip; (2) or a graft: since the +loop runs inside the client's blocked ioctl, unloading the CLIENT (kill) +releases the RM lock and the fresh client restarts the same phase - use +fresh-client runs for any A/B of service-side fixes; (3) the durable +service-side fold remains the endgame once the cell is known. + +**38. Session 111: THE FB IS FULLY DISPROVED (WHOLE-FB FILL, NO FLIP); THE SYS-FILL WEDGED THE GUEST (CYCLE NEXT).** +(1) scan_vidmem for the stream signatures (0x200121ed, 0x200120a9) and even +0x68: ZERO hits across 256MB - THE FB SHADOW WAS ALL ZEROS (before any +pokes): the GPU-side objects are never written in our emulation (the +driver->GPU DMA never runs), so the driver's memdesc SHADOW_INIT copies +zeros and the sec2 loop reads 0 forever (nextPut==authTagGetIndex==0 -> +infinite wait; the complete model). (2) The refined assist: swept FB +0x80000..0x90000 (16384 cells, distinct values; 60s/32KB via poke_vidmem) +- no flip. (3) THE DEFINITIVE TEST: `fill_vidmem 0x40000000` filled the +WHOLE 256MB FB shadow in-kernel (`vgsp: fill_vidmem value=0x40000000 +size=0x10000000 done`) - STILL NO FLIP: THE LOOP'S READ DOES NOT COME FROM +THE FB SHADOW AT ALL; the channel-buffer memdesc is SYS-BACKED (or another +backing) in our emulation, not the FB. (4) The follow-up SYS fill (128 +candidate pages near the active set overwritten with 0x40000000 via the +RAM file) CLOBBERED LIVE DRIVER DATA and WEDGED THE GUEST (SSH banner +timeout; QEMU alive) - the next session MUST CYCLE (fresh boot); lesson: +never bulk-overwrite pages adjacent to the identified active set (0x102698000 +was a hot staging buffer). +NEXT: (1) fresh cycle; (2) the sema is a SYS page - apply the full-page +marker fill ONLY to genuinely-constant, non-active pages (re-run the +content-diff first, then fill the constant set in small batches with a +client check after each, NOT the active neighborhoods); (3) or the +service-side durable route: have the SERVICE write the marker into the +channel-buffer page once it can resolve it (the memdesc VA = the CE +channel's buffer; the service's channel registration fields + the PT walk +give the page candidates). + +**39. Sessions 112b-114: THE SCRUB GATE FELL, THE COMPR WALL WAS DECODED AND FLIPPED, AND THE FIRST VULKAN SUBMISSION LANDED (VKFENCES IS THE NEW WALL).** +Context first (sessions 112b/112c, commits `97489187`/`cb7d8ce7`/`752e2b30`/ +`d6ccdaf3`/`8700f65a`): the CeUtils scrubber's finish payload is authored +in-service (bound to the EXACT 4096-entry channel), the client then leaves +`scrubCheckAndWaitForSize` permanently and runs +vkCreateImage -> vkGetImageMemoryRequirements -> vkAllocateMemory; the +remaining `-2` was decomposed to the guest-internal VidHeap `#149`-shape +call (attr COMPR_ANY + 2MB page + owner 0x474C0000) returning 0x51 +NV_ERR_NO_MEMORY; session 113 read that as a 64KB comptag/CBM alloc. This +session REFINED it: the COMPR+2MB class is not always 64KB - the r705 +runs show the *image data* alloc itself (size 0x140000 = 1.25 MB for +640x480x4 + pad, attr 0x1002000, flags 0xc10d) in exactly the same shape. +The "separate 64KB CBM" reading was too narrow; every alloc of the class +{attr&0xF000 == 0x2000, 2MB page, owner 0x474C0000} fails 0x51. + +**A. THE ZERO-TOUCH CREATE RECIPE (NEW, SCRIPTED).** Two live assists are +required per client, both proven again this session: (1) the +instance-phase marker clears at eglcore+0xB33D30 (t106/t99 `BC`, 5 +clears); (2) THE DEVICE-PHASE WAIT-WORD CHASER at eglcore+0xDCEBD9 +(t99 `BW`: word = *(holder+0x58)+*(u32)(obj2+0x60) with +obj2 = *(*(rbx+0x70)+0x58); write 1 when 0). Without (2) the client +reaches `VKDEVICE`'s poll(`/dev/nvidia0`, 10 ms) retry loop and sits +there INDEFINITELY (session-76 state; the a097 engine-monitor worker +keeps polling 0x2080a097 every 1.25 s regardless - its reply words are +still a red herring for the main-thread gate). With (1)+(2) the run is +`VKINSTANCE ok / VKDEVS 2 / VKDEVICE ok family=0` then straight to the +image path. `t99.py`-style attach works too: attaching `BW` to an +already-stuck client writes the word and the client immediately +proceeds (proven on a stuck pid 15797: WORDWROTE x2 -> VKDEVICE ok -> +VKFAIL -2 at the alloc). +NEW TOOLS: `t120.py` (BC+BW+single-shot COMPR->non-COMPR flip), +`t121.py` (BC+BW+full VidHeap trace from #120), `t122.py` (BC+BW+flip +for ANY COMPR+2MB internal alloc, size >= 0x10000), `a8clear.py` (the +no-ptrace count-gate clearer for base+0x2070b7c, reconstructed). + +**B. THE FLIP (THE DECISIVE EXPERIMENT).** With `t122.py`: at the +vidheap ioctl whose params match (attr&0xF000)==0x2000, owner +0x474C0000, size>=0x10000, the script clears attr bits 13:12 +(COMPR_ANY -> NONE) in the client's own params before the kernel sees +them. Result (r705c, reproducible twice on the same boot): +`HIT149 size=0x140000 attr=0x1002000 -> 0x1000000` (image data alloc +SUCCEEDS), a second `HIT149 size=0x140000 attr=0x1022004 -> 0x1020004` +(buffer-class COMPR alloc) also succeeds, and the client runs the WHOLE +pipeline for the first time in project history: +vkCreateImageView -> buffer -> shader module -> descriptor set +layout/pool -> pipeline layout -> compute pipeline -> command pool -> +command buffer -> fence -> vkResetFences -> **vkQueueSubmit OK** -> +`VKFAIL vkWaitForFences(dev, 1, &fence, VK_TRUE, 30000000000ull) -> 2 +(line 459)` - THE FIRST VULKAN SUBMISSION, then the FIRST FENCE TIMEOUT +(30 s, the client's own bound; the run exits rc=1 cleanly). + +**C. THE NEW WALL = THE FIRST FENCE RELEASE.** Host-side evidence in the +submit window: the client's GR channel registers +(`sema-chan: h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ent=32768 +userd=0xfee00 put=11 get=2 ring rc=0 pa=0x1217e3000 ap=1`) - the guest +driver published 11 GPFIFO entries, the emulation's reader consumed 2; +the service still reports the channel-init pushes unread +(`sema-entry: unread h=0xbeec0100 idx=2 va=0x4ca059c len=976 want=0 +exp=0 unread=1`) and `gr_wedge` grew 3 -> 9 over the runs. So the fence's +release is never authored after the submit - this is the session-76/93 +"wall #2" class (push-page readability/completion), now the ONLY gate +between the flipped client and the first `VKFRAME seq/DONE` + PPM. + +**D. DURABLE-FIX LEADS (for the COMPR class, no guest patching).** +(1) memmgrAllocHal_GM107 (570.144) downgrades COMPR_ANY -> NONE when +`pMemorySystemConfig->bDisableCompbitBacking` is set - the guest's +MEMORY_SYSTEM_STATIC_CONFIG comes from the GSP (the MEMSYS_GET_STATIC_ +CONFIG control 0x20800a1c / the GSP static info path); answering it (or +the same knob wherever the guest's config actually comes from) is the +host-side durable equivalent of the flip. 0x20800a1c was NEVER observed +in the RM_CONTROL log, so first find where the guest's config is fed +(control cache? GSP init args?) before coding the handler. +(2) Alternatively locate glcore's COMPR decision (attr construction at +eglcore+0x2c83fb/0x2c8465 and +0x50e038/0x50e894/0x50e8bd per session +113) and answer the capability that drives it. (3) The session-85 +device-local -2 story (2g mdev type) remains a prerequisite context. + +NEXT (ranked): (1) author the fence release for the first submit (the +push-page-5 content-match / pinning machinery; or have the service +complete the fence when it consumes the submit's GPFIFO entries); (2) +then the first VKFRAME seq/DONE + PPM; (3) fold the COMPR flip into a +host-side answer (see D); (4) the mission series (Xorg/GZDOOM, KVMFR, +CUDA 15/15 + soak, test series, video, Windows). + +**40. Session 115: THE FENCE WALL DECOMPOSED - THE CLIENT VAS PAGE DIRECTORY IS UNDISCOVERABLE AND THE PUSHES ARE NONCONTIGUOUS; RING PAGES ARE CONTENT-LOCATABLE AND INJECTABLE, BUT THE RING PAIR ALONE DOES NOT UNLOCK THE PUSH DECODE.** +(1) The fence timeout is upstream of everything else: the sema service cannot +decode the submit's GR pushes (`sema-entry: unread ... va=0x4ca0000 +len=1432` -> `skipping stale entry ... after 185-194 failed rescues`; +`work_gr submitted=0` - no GR forward ever starts). (2) The pump's push +reads resolve ONLY through the client VAS page directory (or pins). The +service has NO pdb for this client: `walk_probe` of the ring and push VAs +prints `discovered pdb=0x0 found=0` and no registered VASes; the r496f +vidmem pdb discovery never ran (it is only called from the RAM ring-scan +latch), and the r496j COPY_SERVER_PDES candidate (0x109b9c000) is garbage +(its content is not a page table). (3) THE RAM RING SCAN MISSES BY +CONSTRUCTION for this client: it sweeps [0, 6 GB) in 12 slices and gives +up (`found no candidate in 12 slice(s)`), but the sweep runs BEFORE the +client's pushes exist and never re-runs - while the ring page content is +provably in RAM (content-located below). (4) CONTENT LOCATION WORKS: the +GR ring page (ring VA 0x4cc0000, first entries {va=0x4ca0000 len=1432, +va=0x4ca0598 len=4}) was found in the guest RAM file by its first two GPFIFO +entries (at GPA 0x12C600000 then 0x12898c000 on later runs - per-run +allocations), and `sema_map_add` injection of the TRUE ring pair is +accepted (`INJECT va=0x4d20000 pa=0x13619d000 : ok`). BUT the pushes do not +become readable from the ring pair alone: the pushbuffer is NONCONTIGUOUS +(the window guess `ring_pa - 0x20000` reads zeros for both the GR and the +CE channels), the pushes are not CPU-mapped in the client process (the +ring GPA is absent from /proc/PID/pagemap of every rw-s nvidia mapping), +and a bottom-up search for the level-3 table (a 4 KB page whose index-2 +big/small entry maps into the ring's 2 MB window) found ZERO candidates in +RAM - the page tables are almost certainly in VIDMEM (r496f's note) or use +a shape the walker does not model. (5) The service's OWN successful read of +one CE init push (`sema-push: h=0xbeec0104 va=0x4d00000 ... rel=0x4b19b10 +pay=1 have_rel=1`, `ops_runs=0`) did NOT apply its release either - +whatever resolved that one read is not reproducible for the GR pushes. +NEW TOOLS: `fence_assist.py` (host: watches a `dmesg -w` capture for the +pump's entry decodes, content-locates the ring pages by their first two +entries in the RAM file, injects the pairs via sema_map_add - proven to +find and inject, but the ring pair alone is insufficient) and +`find_push.py` (guest: tests whether the client CPU-maps the ring GPA). +NEXT (ranked, concrete): (1) find the client VAS page directory: extract the +pdb from the GUEST kernel's RM objects (kcore walk of the VAS/GPU object +for this channel - the GPFIFO ring page content anchors the search), then +register it via the service's VAS table (a new sysfs/debugfs entry point +or the existing candidate path) - this is the "PDB injection" the mission +names; (2) or scan VIDMEM for the pdb with the ring pair as the verifier +(extend `svc_xvas_pdb_discover`'s trigger: call it from the ring-injection +path, not only from the RAM scan latch); (3) make the RAM ring scan re-arm +per channel with the pushes present (its 12-slice give-up is the reason +the auto-discovery never latches on this client); (4) then the submit +decodes, the release lands and `vkWaitForFences` returns -> VKFRAME. diff --git a/tools/guest-diag/wall2/README.md b/tools/guest-diag/wall2/README.md new file mode 100644 index 0000000..1ef9558 --- /dev/null +++ b/tools/guest-diag/wall2/README.md @@ -0,0 +1,85 @@ +# WALL #2 (GR channel-init release) — session 73 findings + tools + +## Goal +The vk_frame Vulkan client fails `vkCreateDevice -> VKFAIL -4` because the +kernel RM kicks its GR channel (h=0xbeee0100, ring VA 0x44a0000, push VA +0x4480000) and waits ~11 s per attempt for a completion the sema service +cannot apply: it cannot translate the push VA ("false translation for push +va=0x4480000 ... rescuing"), so no release is decoded and the wait times out. + +## Findings (r674/r675, 8+ client runs) +1. The push is RM-built (nvidia.ko), NOT UVM: a kprobe on + `uvm_hal_turing_host_semaphore_release` never fired during a client run. +2. nvidia.ko / nvidia-modeset.ko functions are `notrace` -> `kprobe_events` + refuses them ("Could not probe notrace function ... [nvidia]"). UVM module + probes DO work. QEMU gdbstub hardware breakpoints (hbreak) work fine. +3. The service's rescue matcher is CE-shaped only (hdr 0x20038090, hi==1, + lo&0xfff==slot, w5/v5 shape) -> it can never match a GR push. Entries 0/1/2 + are consumed as "skipping stale entry ... after ~150 failed rescues"; + gr_wedge=1. +4. Host-side content searches of /dev/shm/vgpu-guest-ram during the stall found + NO push data: neither the CE release group (0x20038090) nor the host-class + release header (0x20050017) appears outside driver code pages. +5. Ring entry encoding (from `channelFillGpFifo` in nvidia.ko and + `nvWriteGpEntry` in nvidia-modeset.ko): [0] = va & ~3, [1] = + ((va>>32)&0xff) | ((len<<8)&0x7ffffc00) | 0x80000000 (flag bit seen on + entry[1]). +6. Neither `channelFillGpFifo` (nvidia.ko) nor `nvWriteGpEntry` + (nvidia-modeset.ko) is called during the client run (gdb hbreaks, 0 hits + for the GR channel) -> the GR channel's ring entries are written by an + unidentified writer (possibly the emulation/GSP path); the push content + was never located in guest RAM. +7. RM window records + memdescs (host-readable, stable layout): + - a page holds the client channel's window records + {VA=0x4480000, SIZE=0x20000} (push) and {VA=0x44a0000, SIZE=0x40000} + (ring), each inside head/tail list nodes; + - memdescs `{SIZE, 0x03008000, FIRST_PA, ptr}` live in the same page: + the ring memdesc's FIRST_PA == the service's validated ring pa + (verified twice: 0x1158ce000, 0x129415000) -> memdesc PA == the PA of the + window's first VA page; + - the push memdesc `{0x20000, 0x03008000, PA}` was injected as + 0x4480000 -> PA (resolver4, r675: PA 0x11595c000; "injected mapping + va=0x4480000 ... (sysmem)" in dmesg) and the service then read the page + WITHOUT the "false translation" line - but decoded no release (the page + content does not look like the method stream; later candidates read as + zero). +8. The kernel-side stall shows no GSP RPCs and no ioctls; the wait function + `channelWaitForFinishPayload` (polls `channelReadChannelMemdesc(channel, + channel->0x50)`, which reads `*(channel+0xe8)+off`) was NEVER called during + a client run (gdb hbreak) -> the RM's wait path for this kick is elsewhere. + +## Tools (this directory) +- wall2_resolver2.py — v2: CE release-group content search + tuple arrays + (pa&~0xfff, strides 0x20/0x40/0x60). No candidates live. +- wall2_resolver3.py — v3: ring memdesc -> same-page push memdesc recipe. +- wall2_resolver4.py — v4: window-record pair + ring memdesc cross-check + (this one injected successfully; rdesc[0] bug fixed in v5). +- wall2_resolver5.py — v5: continuous scan (no ring-line gate), injects every + 0x20000 memdesc PA found in pages holding both window records. +- wall2_livecap{,2,3}.py — live stall-time scanners for the CE group and the + host release header (both negative; useful for re-checks). +- scan_r673*.py — RAM structure analysis (ring-PA refs, record pages, memdesc + scans). +- trace_push674.py — VA-reference tracer with direct-map pointer following. +- gdb675*.gdb — QEMU gdbstub scripts (channelFillGpFifo / nvWriteGpEntry + probes, channel struct dumps). + +## Next steps (ranked) +1. Find the GR ring-entry writer (gdb watchpoint on the ring page's CPU VA; + get the VA from the channel struct; find the channel struct via the ring + memdesc page's head/tail pointers). The writer's caller holds the push's + CPU mapping -> PA -> inject -> service decodes the release. +2. Or: gdb-break at the stall on the RM's wait (sample the sleeping task's + kernel stack; the current candidates never fired) and read the semaphore + value it polls; bump it host-side via the ring-page PA (the finish payload + may live at ring_cpu + channel->0x50). +3. Or: fix the service's rescue matcher to accept GR-shaped releases once the + push content is captured (see item 1). + +## Ops +- pkill -f self-matches: the bracket trick fails when the pattern text appears + elsewhere in the agent's own command line; kill by explicit PID from + `ps -eo pid,comm,args | awk '$2=="python3" && /<script>/ {print $1}'`. +- r674 wedged after a gdb-interrupted run; fresh cycle r675 used (no host + reboot). gdb674d/gdb675d sessions left attached (never kill gdbs). +- Never reload the guest nvidia modules; never kill clients mid-flight. diff --git a/tools/guest-diag/wall2/a8clear.py b/tools/guest-diag/wall2/a8clear.py new file mode 100644 index 0000000..261312a --- /dev/null +++ b/tools/guest-diag/wall2/a8clear.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +# a8clear.py - no-ptrace A8/NvOgl count clearer: finds each live vk_frame's +# libnvidia-eglcore base (the off==0 mapping) and writes 0 to the u32 at +# base+0x2070b7c whenever it is nonzero (the count gate t91 cleared via gdb). +import os, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def pids(): + try: + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + except Exception: + return [] +log = set() +end = time.time() + DUR +while time.time() < end: + for pid in pids(): + base = None + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or "libnvidia-eglcore" not in p[-1]: + continue + try: + off = int(p[2], 16) + except Exception: + continue + if off == 0: + base = int(p[0].split("-")[0], 16) + break + except Exception: + continue + if not base: + continue + w = base + 0x2070b7c + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + os.lseek(fd, w, 0) + v = os.read(fd, 4) + if len(v) == 4: + n = int.from_bytes(v, "little") + if n: + os.lseek(fd, w, 0) + os.write(fd, b"\x00\x00\x00\x00") + k = (pid, n) + if len(log) < 40: + log.add(k) + print("A8CLEAR pid=%d count=%u -> 0" % (pid, n), flush=True) + os.close(fd) + except Exception: + pass + time.sleep(0.5) +print("A8CLEAR-DONE", flush=True) diff --git a/tools/guest-diag/wall2/diag76.py b/tools/guest-diag/wall2/diag76.py new file mode 100644 index 0000000..989abd3 --- /dev/null +++ b/tools/guest-diag/wall2/diag76.py @@ -0,0 +1,47 @@ +import gdb + +inf = gdb.selected_inferior() +pid = inf.pid +maps = [] +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + maps.append((lo, hi, off, parts[5] if len(parts) > 5 else "")) + except Exception: + pass + +def resolve(addr): + for lo, hi, off, name in maps: + if lo <= addr < hi: + return "%s+0x%x" % (name.split('/')[-1], off + (addr - lo)) + return "?" + +gdb.execute("set pagination off") +print("=== THREADS ===") +for t in inf.threads(): + try: + t.switch() + except Exception as e: + print("switch err", e) + continue + pc = int(gdb.parse_and_eval("$pc")) + print("TID %s pc=%s" % (t.ptid, resolve(pc))) + try: + bt = gdb.execute("bt 12", to_string=True) + except Exception as e: + bt = "bt err %s" % e + for ln in bt.splitlines(): + ln = ln.strip() + if ln.startswith("#"): + try: + a = int(ln.split()[1], 16) + print(" ", ln, "=>", resolve(a)) + except Exception: + print(" ", ln) + else: + print(" ", ln) +print("DIAG-DONE") +gdb.execute("detach") +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/fastrun.sh b/tools/guest-diag/wall2/fastrun.sh new file mode 100644 index 0000000..73b9d80 --- /dev/null +++ b/tools/guest-diag/wall2/fastrun.sh @@ -0,0 +1,10 @@ +#!/bin/bash +cd /tmp +sudo dmesg -C +( sudo bash /tmp/slotfill2.sh > /tmp/slotfill2.log 2>&1 & ) +sleep 0.5 +sudo timeout 60 strace -f -T -tt -e trace=ioctl -o /tmp/vk_strace3.out /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk77.out 2>&1 +echo "=== client ===" +cat /tmp/vk77.out +echo "=== last ioctls ===" +tail -12 /tmp/vk_strace3.out diff --git a/tools/guest-diag/wall2/fastrun2.sh b/tools/guest-diag/wall2/fastrun2.sh new file mode 100644 index 0000000..3c7b943 --- /dev/null +++ b/tools/guest-diag/wall2/fastrun2.sh @@ -0,0 +1,12 @@ +#!/bin/bash +cd /tmp +sudo dmesg -C +( sudo bash /tmp/slotfill2.sh > /dev/null 2>&1 & ) +sleep 0.5 +sudo timeout 45 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk78.out 2>&1 +echo "=== client ===" +cat /tmp/vk78.out +echo "=== repeated controls (top) ===" +sudo dmesg -T | grep -oE "GSP_RM_CONTROL cmd=0x[0-9a-f]+" | sort | uniq -c | sort -rn | head -12 +echo "=== last controls ===" +sudo dmesg -T | grep "GSP_RM_CONTROL cmd" | tail -8 diff --git a/tools/guest-diag/wall2/fasttick.py b/tools/guest-diag/wall2/fasttick.py new file mode 100644 index 0000000..413d3ba --- /dev/null +++ b/tools/guest-diag/wall2/fasttick.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +# fasttick.py - watch the service log for the GR0 channel's ring resolution +# (sema-chan: h=0xbeec0100 ... ring rc=0 pa=0x...) and immediately hand the +# verified {ring VA -> ring PA} pair to the service (sema_pdb_discover), so +# the parked VAS pdb candidate can be verified before/at age-out. +import re, subprocess, sys, time + +PARAM = "/sys/module/vgsp/parameters/sema_pdb_discover" +RING_VA = 0x4cc0000 + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + end = time.time() + dur + fired = set() + seen_pdbs = set() + print("# fasttick start (%.0fs)" % dur, flush=True) + while time.time() < end: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + tail = out.splitlines()[-120:] + for ln in tail: + m = re.search(r"sema-chan: h=0xbeec0100 eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+) ent=\d+ userd=0x[0-9a-f]+ as=\d+ put=\d+ get=\d+ ring rc=0 pa=0x([0-9a-f]+)", ln) + if m: + ring_va = int(m.group(1), 16) + ring_pa = int(m.group(2), 16) + key = (ring_va, ring_pa) + if key not in fired: + fired.add(key) + cmd = "echo 0x%x,0x%x > %s" % (ring_va, ring_pa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("[%s] TRIGGER va=0x%x pa=0x%x rc=%d" % ( + time.strftime("%H:%M:%S"), ring_va, ring_pa, r.returncode), flush=True) + m2 = re.search(r"page directory (?:registered UNVERIFIED after \d+ ms parked \()?pdb=0x([0-9a-f]+) entries=(\d+)", ln) + if not m2: + m2 = re.search(r"VA space 0x[0-9a-f]+ page directory 0x([0-9a-f]+) entries=(\d+)", ln) + if m2: + key = int(m2.group(1), 16) + if key not in seen_pdbs: + seen_pdbs.add(key) + print("[%s] PDB candidate 0x%x entries=%s" % ( + time.strftime("%H:%M:%S"), key, m2.group(2)), flush=True) + time.sleep(0.3) + print("# fasttick end", flush=True) + +main() diff --git a/tools/guest-diag/wall2/fence_assist.py b/tools/guest-diag/wall2/fence_assist.py new file mode 100644 index 0000000..4ea0b0c --- /dev/null +++ b/tools/guest-diag/wall2/fence_assist.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +# fence_assist.py - content-locate the client's ring pages in guest RAM from the +# service's own GPFIFO entry decodes (dmesg) and inject the TRUE VA->PA pairs +# via sema_map_add, so the pump's push reads (and its pushbuffer pairing) have +# a correct anchor. +import mmap, re, struct, subprocess, sys, threading, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" + +def gpa_of(fo): + return fo if fo < 0x80000000 else fo - 0x80000000 + 0x100000000 + +def inject(va, pa, ln=0x1000, flag=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va & ~0xfff, pa & ~0xfff, ln, flag, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("INJECT va=0x%x pa=0x%x : %s" % (va & ~0xfff, pa & ~0xfff, + "ok" if r.returncode == 0 else "FAIL"), flush=True) + +def search(mm, pats, results): + CH = 8 << 20 + size = len(mm) + off = 0 + while off < size: + d = mm[off:off+CH] + for (name, pat) in pats.items(): + if name in results: + continue + i = d.find(pat) + if i >= 0: + results[name] = gpa_of(off + i) + off += CH - 32 + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + end = time.time() + dur + pats = {} # name -> pattern bytes + ring_va = {} # name -> ring VA + seen = set() # raw entry lines processed + done = set() # injected names + print("# fence_assist start (%.0fs)" % dur, flush=True) + while time.time() < end: + try: + with open("/tmp/opencode/dmsg_w.txt", "r", errors="ignore") as df: + tail = df.read().splitlines()[-6000:] + out = "\n".join(tail) + except OSError: + out = "" + tail = [] + tail = [ln for ln in tail if "sema-chan:" in ln or "sema-push" in ln] + cur = {} + for ln in tail: + m = re.search(r"sema-chan: entry\[(\d+)\] h=0x([0-9a-f]+) eng=\d+ raw=([0-9a-f]+) va=0x([0-9a-f]+)", ln) + if m: + idx, h, raw, va = int(m.group(1)), m.group(2), int(m.group(3), 16), int(m.group(4), 16) + cur.setdefault(h, {})[idx] = raw + if idx == 0: + ring_va[h] = va # entry[0] va is the push VA (ring_va - 0x20000) + for h, ents in cur.items(): + key = (h, tuple(sorted(ents.items()))) + if key in seen or len(ents) < 2: + continue + seen.add(key) + name = h + pat = b"".join(struct.pack("<Q", ents[i]) for i in sorted(ents)[:2]) + pats[name] = pat + print("# channel h=0x%s entries=%s pattern ready" % (h, sorted(ents)), flush=True) + if pats: + results = {} + threads = [] + # split the map among 4 threads + LO = 0x100000000; HI = 0x180000000 + size = len(mm) + step = size // 4 + class Sl(shim := object): + def __init__(self, a, b): + self.a, self.b = a, b + def worker(a, b): + CH = 8 << 20 + off = a + while off < b: + if off < 0x80000000 and b <= 0x80000000: pass + d = mm[off:off+CH] + for (nm, p) in list(pats.items()): + if nm in results: continue + i = d.find(p) + if i >= 0: + results[nm] = gpa_of(off + i) + off += CH - 32 + for k in range(4): + a = 0x80000000 + k * (0x80000000 // 4) + b = 0x80000000 + (k + 1) * (0x80000000 // 4) + t = threading.Thread(target=worker, args=(a, b)) + t.start(); threads.append(t) + for t in threads: t.join() + for nm, gpa in results.items(): + if nm in done: continue + done.add(nm) + # ring page content starts with entry[0]; the ring VA is the + # channel's gpfifo VA = we reconstruct: entry0 va is the push, + # ring = first "gpfifo=0x..." dmesg line for this h + m = re.search(r"sema-chan: h=0x%s eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+)" % nm, out) + rva = int(m.group(1), 16) if m else None + if rva: + print("RING FOUND h=0x%s gpfifo_va=0x%x -> gpa=0x%x" % (nm, rva, gpa), flush=True) + inject(rva, gpa, 0x1000) + time.sleep(0.6) + print("# fence_assist done, injected %d" % len(done), flush=True) + +main() diff --git a/tools/guest-diag/wall2/find_push.py b/tools/guest-diag/wall2/find_push.py new file mode 100644 index 0000000..4bf8ccc --- /dev/null +++ b/tools/guest-diag/wall2/find_push.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +# find_push.py - locate the client's CPU mapping that owns the GR ring page +# (GPA 0x12C600000 as content-located on the host) and print the CPU VA<-> +# GPU VA correspondence for the low GPU VAs (0x4ca0000 etc). +import os, struct, sys, subprocess, mmap, time + +RING_GPA = 0x12C600000 +PUSH_GVA = 0x4ca0000 +RING_GVA = 0x4cc0000 + +def pids(): + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + +def read_pagemap(pid, va): + try: + with open("/proc/%d/pagemap" % pid, "rb") as f: + f.seek((va // 4096) * 8) + d = f.read(8) + if len(d) < 8: + return None + e = int.from_bytes(d, "little") + if not (e >> 63) & 1: + return None + return (e & ((1 << 55) - 1)) << 12 + except OSError: + return None + +def main(): + for pid in pids(): + print("pid=%d" % pid) + try: + maps = open("/proc/%d/maps" % pid).read().splitlines() + except OSError: + continue + for line in maps: + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1]: + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo > 0x1000000: + continue + # scan pages; look for RING_GPA + for va in range(lo, hi, 0x10000): + for off in range(0, 0x10000, 0x1000): + pa = read_pagemap(pid, va + off) + if pa == RING_GPA: + print("FOUND ring page: cpu_va=0x%x (map 0x%x-0x%x %s)" % (va+off, lo, hi, p[-1])) + delta_cpu = (va + off) - RING_GVA + push_cpu = PUSH_GVA + delta_cpu + print(" delta_cpu = 0x%x" % delta_cpu) + for gva in (PUSH_GVA, PUSH_GVA+0x1000, 0x4ca059c, 0x4ca0980, 0x4ca5870, 0x4903000): + cva = gva + delta_cpu + pa2 = read_pagemap(pid, cva & ~0xfff) + print(" gva=0x%x -> cpu=0x%x -> pa=%s" % (gva, cva, hex(pa2) if pa2 else None)) + return + print(" ring GPA 0x%x not found in this pid's mappings" % RING_GPA) + +main() diff --git a/tools/guest-diag/wall2/find_ring_va.py b/tools/guest-diag/wall2/find_ring_va.py new file mode 100644 index 0000000..c02ea40 --- /dev/null +++ b/tools/guest-diag/wall2/find_ring_va.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +# Guest-side: read the nv_vm_map_pages vmap areas via /proc/kcore and find the +# GPFIFO entry pattern (entry0 raw 0x0005980004480000 for va=0x4480000 len=1432). +import struct, sys +pat = bytes.fromhex("0000480400980500") # 0x0005980004480000 LE +class Kcore: + def __init__(self): + self.fd = open("/proc/kcore", "rb") + hdr = self.fd.read(64) + e_phoff = struct.unpack_from("<Q", hdr, 32)[0] + e_phentsize, e_phnum = struct.unpack_from("<HH", hdr, 54) + self.segs = [] + for i in range(e_phnum): + self.fd.seek(e_phoff + i * e_phentsize) + ph = self.fd.read(56) + p_type, _fl, p_off, p_va, _pa, p_sz = struct.unpack_from("<IIQQQQ", ph, 0) + if p_type == 1: + self.segs.append((p_va, p_va + p_sz, p_off)) + def read(self, va, n): + for lo, hi, off in self.segs: + if lo <= va and va + n <= hi: + self.fd.seek(off + (va - lo)) + return self.fd.read(n) + return None +kc = Kcore() +areas = [] +for line in open("/proc/vmallocinfo"): + p = line.split() + if len(p) < 2: continue + if "nv_vm_map_pages" not in line: continue + lo, hi = p[0].split("-") + areas.append((int(lo, 16), int(hi, 16))) +print("nv_vm_map_pages areas:", len(areas)) +for lo, hi in areas: + size = hi - lo + d = kc.read(lo, size) + if not d: + print(" 0x%x-0x%x unreadable" % (lo, hi)); continue + idx = d.find(pat) + print(" 0x%x-0x%x size=0x%x hit=%s" % (lo, hi, size, ("+0x%x -> VA 0x%x" % (idx, lo + idx)) if idx >= 0 else "no")) + if idx >= 0: + # dump the page around the hit + base = lo + (idx & ~0xfff) + page = kc.read(base, 0x100) + print(" page head: %s" % page.hex()) diff --git a/tools/guest-diag/wall2/gdb675.gdb b/tools/guest-diag/wall2/gdb675.gdb new file mode 100644 index 0000000..e4b73ce --- /dev/null +++ b/tools/guest-diag/wall2/gdb675.gdb @@ -0,0 +1,21 @@ +set pagination off +set confirm off +set print elements 0 +target remote :1234 +hbreak *0xffffffffc0d0b9f0 +condition 1 *(unsigned int*)($rdi+0x110) == 0xc46f +commands 1 +silent +printf "GRFILL ch=%p idx=%d len=%d\n", $rdi, $esi, $edx +printf " +0x048=%p +0x0b8=%p +0x0e8=%p\n", *(unsigned long*)($rdi+0x48), *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0xe8) +printf " +0x060=%p +0x0c0=%p +0x0c8=%p +0x0d0=%p +0x0d8=%p\n", *(unsigned long*)($rdi+0x60), *(unsigned long*)($rdi+0xc0), *(unsigned long*)($rdi+0xc8), *(unsigned long*)($rdi+0xd0), *(unsigned long*)($rdi+0xd8) +printf " +0x0e0=%p +0x0f0=%p +0x0f8=%p +0x108=%p +0x140=%p +0x148=%p\n", *(unsigned long*)($rdi+0xe0), *(unsigned long*)($rdi+0xf0), *(unsigned long*)($rdi+0xf8), *(unsigned long*)($rdi+0x108), *(unsigned long*)($rdi+0x140), *(unsigned long*)($rdi+0x148) +x/8gx $rdi+0x40 +x/8gx $rdi+0x80 +x/8gx $rdi+0xc0 +x/8gx $rdi+0xe0 +x/8gx $rdi+0x100 +x/8gx $rdi+0x140 +continue +end +continue diff --git a/tools/guest-diag/wall2/gdb675c.gdb b/tools/guest-diag/wall2/gdb675c.gdb new file mode 100644 index 0000000..25b5ca0 --- /dev/null +++ b/tools/guest-diag/wall2/gdb675c.gdb @@ -0,0 +1,12 @@ +set pagination off +set confirm off +set print elements 0 +target remote :1234 +hbreak *0xffffffffc0d0b9f0 +commands 1 +silent +printf "FILL ch=%p cls100=0x%x b8=0x%x stride48=0x%x idx=%d len=%d\n", $rdi, *(unsigned int*)($rdi+0x100), *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0x48), $esi, $edx +printf " d0=%p d8=%p e8=%p f8=%p 60=%p\n", *(unsigned long*)($rdi+0xd0), *(unsigned long*)($rdi+0xd8), *(unsigned long*)($rdi+0xe8), *(unsigned long*)($rdi+0xf8), *(unsigned long*)($rdi+0x60) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdb675d.gdb b/tools/guest-diag/wall2/gdb675d.gdb new file mode 100644 index 0000000..f9dc460 --- /dev/null +++ b/tools/guest-diag/wall2/gdb675d.gdb @@ -0,0 +1,11 @@ +set pagination off +set confirm off +set print elements 0 +target remote :1234 +hbreak *0xffffffffc1605fd0 +commands 1 +silent +printf "NVWRITE ch=%p idx=%u len=%u b8=0x%x f68=%p f50=0x%x fb0=0x%x fc0=0x%x f198=%p\n", $rdi, $esi, $edx, *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0x68), *(unsigned int*)($rdi+0x50), *(unsigned int*)($rdi+0xb0), *(unsigned int*)($rdi+0xc0), *(unsigned long*)($rdi+0x198) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbfill.gdb b/tools/guest-diag/wall2/gdbfill.gdb new file mode 100644 index 0000000..41d1ea3 --- /dev/null +++ b/tools/guest-diag/wall2/gdbfill.gdb @@ -0,0 +1,13 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +set $e = *(unsigned long*)($rdi+0x1428) +if *(unsigned short*)($e+0x2e) == 0xffff + set {unsigned short}($e+0x2e) = 0 + printf "FILLED slot at %p+0x2e -> 0\n", $e +end +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbgap3.sh b/tools/guest-diag/wall2/gdbgap3.sh new file mode 100644 index 0000000..bf19370 --- /dev/null +++ b/tools/guest-diag/wall2/gdbgap3.sh @@ -0,0 +1,28 @@ +#!/bin/bash +cd /tmp +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk65.out 2>&1 &) +sleep 5 +PID=$(pgrep -x vk_frame | head -1) +cp /proc/$PID/maps /tmp/maps_d.txt +gdb -p $PID -batch -ex "set pagination off" -ex "bt 12" 2>&1 | grep -E "^#" > /tmp/bt_gap3.txt +python3 - <<'PY' +import re +maps=open('/tmp/maps_d.txt').read().splitlines() +segs=[] +for m in maps: + p=m.split() + if len(p)<6: continue + lo,hi=[int(x,16) for x in p[0].split('-')] + off=int(p[2],16) + segs.append((lo,hi,off,p[5])) +for line in open('/tmp/bt_gap3.txt'): + m=re.match(r'#(\d+)\s+0x([0-9a-f]+)', line) + if not m: continue + a=int(m.group(2),16) + for lo,hi,off,name in segs: + if lo<=a<hi: + print('#%s %s + 0x%x' % (m.group(1), name.split('/')[-1], off+(a-lo))) + break + else: + print('#%s 0x%x unmapped' % (m.group(1), a)) +PY diff --git a/tools/guest-diag/wall2/gdbparam.sh b/tools/guest-diag/wall2/gdbparam.sh new file mode 100644 index 0000000..3f30d49 --- /dev/null +++ b/tools/guest-diag/wall2/gdbparam.sh @@ -0,0 +1,16 @@ +#!/bin/bash +cd /tmp +sudo dmesg -C +( sudo bash /tmp/slotfill2.sh > /dev/null 2>&1 & ) +sleep 0.5 +(timeout 60 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk79.out 2>&1 &) +sleep 5 +PID=$(pgrep -x vk_frame | head -1) +gdb -p $PID -batch \ + -ex "set pagination off" \ + -ex "catch syscall ioctl" \ + -ex "condition 1 \$rsi == 0xc020462a" \ + -ex "continue" \ + -ex "printf \"ioctl nr=0x2a arg=%p\n\", \$rdx" \ + -ex "x/40wx \$rdx" \ + -ex "detach" 2>/dev/null | tail -45 diff --git a/tools/guest-diag/wall2/gdbpoll6.gdb b/tools/guest-diag/wall2/gdbpoll6.gdb new file mode 100644 index 0000000..d7fccfd --- /dev/null +++ b/tools/guest-diag/wall2/gdbpoll6.gdb @@ -0,0 +1,9 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +printf "CHK rsi=%d cnt=%d e0=%p v=%x\n", $rsi, *(int*)($rdi+0x22efc), *(unsigned long*)($rdi+0x1428), *(unsigned int*)(*(unsigned long*)($rdi+0x1428)+($rsi<<4)) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbspin.sh b/tools/guest-diag/wall2/gdbspin.sh new file mode 100644 index 0000000..989e584 --- /dev/null +++ b/tools/guest-diag/wall2/gdbspin.sh @@ -0,0 +1,19 @@ +#!/bin/bash +cd /tmp +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk57.out 2>&1 &) +sleep 25 +PID=$(pgrep -x vk_frame | head -1) +echo "pid=$PID" +BASE=$(grep 'eglcore' /proc/$PID/maps | head -1 | cut -d- -f1) +echo "base=$BASE" +SPIN=$(printf '0x%x' $((0x$BASE + 0xa10e90))) +echo "spin=$SPIN" +gdb -p $PID -batch \ + -ex "set pagination off" \ + -ex "break *$SPIN" \ + -ex "continue" \ + -ex "printf \"HIT rip=%p rdi=%p rsi=%p rdx=%p rbx=%p rbp=%p\n\", \$rip, \$rdi, \$rsi, \$rdx, \$rbx, \$rbp" \ + -ex "bt" \ + -ex "info registers r12 r13 r14 r15" \ + -ex "x/12i \$rip-24" \ + -ex "detach" 2>&1 | tail -40 diff --git a/tools/guest-diag/wall2/gdbu16.gdb b/tools/guest-diag/wall2/gdbu16.gdb new file mode 100644 index 0000000..61f335b --- /dev/null +++ b/tools/guest-diag/wall2/gdbu16.gdb @@ -0,0 +1,9 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +printf "CHK rsi=%d cnt=%d e0=%p u16@2e=%x u32@20=%x u32@0=%x\n", $rsi, *(int*)($rdi+0x22efc), *(unsigned long*)($rdi+0x1428), *(unsigned short*)(*(unsigned long*)($rdi+0x1428)+0x2e), *(unsigned int*)(*(unsigned long*)($rdi+0x1428)+0x20), *(unsigned int*)(*(unsigned long*)($rdi+0x1428)) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbwatch.gdb b/tools/guest-diag/wall2/gdbwatch.gdb new file mode 100644 index 0000000..36b3339 --- /dev/null +++ b/tools/guest-diag/wall2/gdbwatch.gdb @@ -0,0 +1,18 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +set $e = *(unsigned long*)($rdi+0x1428) +watch *(unsigned short*)($e+0x2e) +printf "WATCH set at %p+0x2e\n", $e +disable 1 +continue +end +commands 2 +silent +printf "WRITE to slot+0x2e: old=%x new=%x by pc=%p\n", *(unsigned short*)($e+0x2e), $eax, $pc +bt 3 +continue +end +continue diff --git a/tools/guest-diag/wall2/gr_walk.py b/tools/guest-diag/wall2/gr_walk.py new file mode 100644 index 0000000..cbdb02d --- /dev/null +++ b/tools/guest-diag/wall2/gr_walk.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +# gr_walk.py - find the RM window memdesc records in the guest's own RAM +# (via /proc/kcore direct map) and walk their kernel page-list pointers in +# the SAME pass (addresses go stale otherwise). Prints VA->PA candidates. +import sys, struct +sys.path.insert(0, "/tmp") +import guest_pin_stream as gps + +PAT_SIZE = struct.pack("<Q", 0x20000) +PAT_FLAGS = struct.pack("<Q", 0x03008000) + +def main(): + kc = gps.Kcore() + if not kc.load(): + print("# kcore load failed"); return 1 + pov = kc.page_offset_base + print("# page_offset_base=0x%x" % pov) + found = [] + CH = 1 << 20 + pa = 0x100000000 + end = 0x180000000 + while pa < end and len(found) < 12: + va = pov + pa + d = kc.read(va, CH) + if d: + st = 0 + while True: + i = d.find(PAT_SIZE, st) + if i < 0: + break + st = i + 1 + if i + 16 <= len(d) and d[i+8:i+16] == PAT_FLAGS: + found.append((pa + i, d[i:i+0x60])) + pa += CH + print("# memdesc records: %d" % len(found)) + for (gpa, rec) in found: + size, flags = struct.unpack_from("<QQ", rec, 0) + wpa = struct.unpack_from("<Q", rec, 0x10)[0] + kptrs = [struct.unpack_from("<Q", rec, o)[0] for o in (0x18, 0x20, 0x28, 0x30, 0x38)] + print("rec@0x%x size=0x%x pa=0x%x kptrs=%s" % + (gpa, size, wpa, ["0x%x" % k for k in kptrs])) + # walk each kptr for PA-like lists + for kp in kptrs: + if not gps.is_kern_ptr(kp): + continue + dd = kc.read(kp, 0x80) + if not dd: + print(" kptr 0x%x: unreadable" % kp); continue + words = struct.unpack_from("<%dQ" % (len(dd)//8), dd, 0) + pas = [w for w in words if 0x100000000 <= w < 0x180000000] + npas = [w for w in words if 0x100000000 <= (w & ~0xfff) < 0x180000000] + print(" kptr 0x%x: %s" % (kp, " ".join("0x%x" % w for w in words[:10]))) + if len(npas) >= 2: + print(" PA-like: %s" % ["0x%x" % w for w in npas[:16]]) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/gr_walk2.py b/tools/guest-diag/wall2/gr_walk2.py new file mode 100644 index 0000000..7a50bc8 --- /dev/null +++ b/tools/guest-diag/wall2/gr_walk2.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# gr_walk2.py - deeper walk: from the window memdesc records, follow the +# kernel pointers 2-3 levels and search for page-array records +# {vmap_cpu, flags, PA, 0x1000} (2+ consecutive) to recover the page lists. +import sys, struct +sys.path.insert(0, "/tmp") +import guest_pin_stream as gps + +PAT_SIZE = struct.pack("<Q", 0x20000) +PAT_FLAGS = struct.pack("<Q", 0x03008000) + +def is_pa(v): + return 0x100000000 <= v < 0x180000000 + +def page_array_at(kc, va, n=0x400): + """Return [(off, [PA,...])] for runs of {ptr,flags,PA,0x1000} records.""" + d = kc.read(va, n) + if not d: + return [] + out = [] + i = 0 + while i + 32 <= len(d): + p, fl, pa, z = struct.unpack_from("<4Q", d, i) + if z == 0x1000 and is_pa(pa) and gps.is_kern_ptr(p): + run = [pa] + j = i + 32 + while j + 32 <= len(d): + p2, fl2, pa2, z2 = struct.unpack_from("<4Q", d, j) + if z2 == 0x1000 and is_pa(pa2): + run.append(pa2) + j += 32 + else: + break + if len(run) >= 2: + out.append((i, run)) + i = j + else: + i += 8 + return out + +def main(): + kc = gps.Kcore() + if not kc.load(): + print("# kcore load failed"); return 1 + pov = kc.page_offset_base + # find memdesc records (fast direct-map scan) + found = [] + CH = 1 << 20 + pa = 0x100000000 + while pa < 0x180000000 and len(found) < 12: + d = kc.read(pov + pa, CH) + if d: + st = 0 + while True: + i = d.find(PAT_SIZE, st) + if i < 0: + break + st = i + 1 + if i + 16 <= len(d) and d[i+8:i+16] == PAT_FLAGS: + found.append((pa + i, d[i:i+0x40])) + pa += CH + print("# memdesc records: %d" % len(found)) + for (gpa, rec) in found: + size, flags = struct.unpack_from("<QQ", rec, 0) + wpa = struct.unpack_from("<Q", rec, 0x10)[0] + print("rec@0x%x size=0x%x pa=0x%x" % (gpa, size, wpa)) + kptrs = [struct.unpack_from("<Q", rec, o)[0] for o in range(0x18, 0x40, 8)] + for kp in kptrs: + if not gps.is_kern_ptr(kp): + continue + # level 1: read the target, look for page arrays directly + for arr in page_array_at(kc, kp): + print(" L1 kptr 0x%x: page-array run at +0x%x: %s" % + (kp, arr[0], ["0x%x" % x for x in arr[1][:12]])) + # level 2: follow pointers in the target + d = kc.read(kp, 0x100) + if not d: + continue + for o in range(0, 0x100, 8): + p2 = struct.unpack_from("<Q", d, o)[0] + if not gps.is_kern_ptr(p2): + continue + for arr in page_array_at(kc, p2): + print(" L2 0x%x -> 0x%x: page-array at +0x%x: %s" % + (kp, p2, arr[0], ["0x%x" % x for x in arr[1][:12]])) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/guest_words.sh b/tools/guest-diag/wall2/guest_words.sh new file mode 100755 index 0000000..6ac200c --- /dev/null +++ b/tools/guest-diag/wall2/guest_words.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# guest_words.sh - print "VA PA" for the +0x80 word of every 16 KB rw-s +# /dev/nvidiactl mapping of the vk_frame client (pagemap PFN -> PA). +PID=$(pgrep -x vk_frame | head -1) +[ -z "$PID" ] && exit 0 +sudo grep -E 'rw-s' /proc/$PID/maps | grep nvidiactl | while read -r line; do + RNG=$(echo "$line" | awk '{print $1}') + LO=${RNG%-*}; HI=${RNG#*-} + SZ=$((0x$HI - 0x$LO)) + [ "$SZ" -ne 16384 ] && continue + VP=$(( (0x$LO + 0x80) / 4096 )) + ENT=$(sudo dd if=/proc/$PID/pagemap bs=8 skip=$VP count=1 2>/dev/null | xxd -e -g8 | awk '{print $2}') + [ -z "$ENT" ] && continue + PFN=$(( 0x$ENT & 0x7fffffffffffff )) + PA=$(( PFN * 4096 + 0x80 )) + echo "0x$LO 0x$(printf %x $PA)" +done diff --git a/tools/guest-diag/wall2/host_resolve.py b/tools/guest-diag/wall2/host_resolve.py new file mode 100644 index 0000000..258d994 --- /dev/null +++ b/tools/guest-diag/wall2/host_resolve.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +# host_resolve.py - host-side push resolver: read the service's sema-pushall +# lines (pa=0x0, non-zero first8), find the content page in +# /dev/shm/vgpu-guest-ram by the push's first8 bytes, and inject the page +# mapping via sema_map_add. Replaces the guest-side kcore walker: the +# service already tells us push VA + content prefix, the host RAM file has +# the bytes. +import mmap, os, re, subprocess, sys, time + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +BASE = 0x80000000 # file offset = pa - BASE + +def inject(va_page, pa): + cmd = "echo 0x%x,0x%x,0x1000,1 > %s" % (va_page, pa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + ok = r.returncode == 0 + print("inject 0x%x -> 0x%x : %s" % (va_page, pa, "ok" if ok else r.stderr.decode().strip())) + return ok + +def main(): + seen = set() + f = open(RAM, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + injected = 0 + deadline = time.time() + (float(sys.argv[1]) if len(sys.argv) > 1 else 600) + while time.time() < deadline and injected < 40: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + lines = [l for l in out.splitlines() if "sema-pushall" in l] + for l in lines[-800:]: + m = re.search(r"va=(0x[0-9a-f]+) pa=0x0 len=\d+ allow=\d trusted=\d first8=([0-9a-f]{8}) ([0-9a-f]{8})", l) + if not m: + continue + va = int(m.group(1), 16) + w0, w1 = int(m.group(2), 16), int(m.group(3), 16) + if (w0, w1) == (0, 0) and (w0 == 0 or w1 == 0): + pass + if w0 == 0 and w1 == 0: + continue + key = (va & ~0xFFF, w0, w1) + if key in seen: + continue + pat = w0.to_bytes(4, "little") + w1.to_bytes(4, "little") + inpage = va & 0xFFF + # search with the in-page offset constraint + start = 0 + hit = None + while True: + i = mm.find(pat, start) + if i < 0: + break + if (i & 0xFFF) == inpage: + hit = i + break + start = i + 1 + if hit is None: + # fallback: any hit within the same page offset modulo + start = 0 + while True: + i = mm.find(pat, start) + if i < 0: + break + hit = i + break + # proceed only if the constraint is loose? no - skip + hit = None + if hit is None: + seen.add(key) + print("no match for va=0x%x first8=%08x %08x" % (va, w0, w1)) + continue + pa = (hit & ~0xFFF) + BASE + seen.add(key) + if inject(va & ~0xFFF, pa): + injected += 1 + time.sleep(0.5) + mm.close() + f.close() + print("done: injected=%d" % injected) + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/wall2/kc_dump_fn.py b/tools/guest-diag/wall2/kc_dump_fn.py new file mode 100644 index 0000000..e481d24 --- /dev/null +++ b/tools/guest-diag/wall2/kc_dump_fn.py @@ -0,0 +1,21 @@ +import struct, sys +VA = int(sys.argv[1], 16) +N = int(sys.argv[2], 16) if len(sys.argv) > 2 else 0x800 +f = open("/proc/kcore","rb") +hdr = f.read(64) +e_phoff = struct.unpack_from("<Q", hdr, 0x20)[0] +e_phentsize = struct.unpack_from("<H", hdr, 0x36)[0] +e_phnum = struct.unpack_from("<H", hdr, 0x38)[0] +segs = [] +for i in range(e_phnum): + f.seek(e_phoff + i*e_phentsize) + ph = f.read(e_phentsize) + p_type, p_flags, p_offset, p_vaddr, p_paddr, p_filesz, p_memsz = struct.unpack_from("<IIQQQQQ", ph, 0) + if p_type == 1: + segs.append((p_vaddr, p_memsz, p_offset)) +for v, sz, off in segs: + if v <= VA < v+sz: + f.seek(off + (VA - v)) + d = f.read(min(N, v+sz-VA)) + sys.stdout.write(d.hex()) + break diff --git a/tools/guest-diag/wall2/live_walk.py b/tools/guest-diag/wall2/live_walk.py new file mode 100644 index 0000000..48f131f --- /dev/null +++ b/tools/guest-diag/wall2/live_walk.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +# live_walk.py - live PT-walk + inject loop for the running client. +# Polls dmesg for the published page directory (pdb) and for the VAs the +# service is failing to read (sema-entry: unread / entry[] lines / registered +# gpfifo), walks each VA through the pdb with the service's own GMMU math, +# and injects resolved pairs via sema_map_add (ap=1 sysmem). +import mmap, re, subprocess, sys, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +SEMA_PA_MAX = 0x180000000 + +WINDOW_VAS = [0x44a0000, 0x4500000, 0x4560000, # rings + 0x4480000, 0x4485000, 0x4ca0000, 0x4ca5000, + 0x4d00000, 0x4b17000, 0x4020000, 0x4100000, 0x4160000, + 0x4213000, 0x45a0bb0, # ctx-init releases (deterministic) + 0x124935060, 0x102e34060, 0x108d4f060, 0x12a3a4060, 0x116c0b060, # notif families + 0x20400f000, 0x20400ffc0, 0x20480fef0, 0x204827f7c] # counter pages / tracker VAs + +mm = None +def off(pa): return pa if pa < 0x80000000 else pa - 0x80000000 +def rd64(pa): + if pa + 8 > SEMA_PA_MAX: return None + d = mm[off(pa):off(pa)+8] + return int.from_bytes(d, "little") if len(d) == 8 else None +def pde_ap(ap): return "FB" if ap == 1 else ("SYS" if ap in (2, 3) else None) +def pte_ap(ap): return "FB" if ap == 0 else (None if ap == 1 else "SYS") +def pte_addr(e, fb): return ((e >> 8) & 0x1FFFFFF) << 12 if fb else ((e >> 8) & 0x3FFFFFFFFFFF) << 12 +def pte_leaf(table, idx): + e = rd64(table + 8 * idx) + if e is None or not (e & 1): return None + ap = pte_ap((e >> 1) & 3) + if ap is None: return None + a = pte_addr(e, ap == "FB") + return (a, ap) if a + 8 <= SEMA_PA_MAX else None + +def walk(pdb, va, entries=4): + import math + root_bits = max(1, min(12, int(math.log2(entries or 1)))) + table = pdb + for level in range(3): + if level == 0: + bits, shift = root_bits, 49 - root_bits + else: + bits, shift = 9, 38 if level == 1 else 29 + idx = (va >> shift) & ((1 << bits) - 1) + e = rd64(table + 8 * idx) + if e is None or (e & 1): return None + ap = pde_ap((e >> 1) & 3) + if ap is None: return None + table = pte_addr(e, ap == "FB") + idx = (va >> 21) & 0xff + big = rd64(table + 16 * idx); small = rd64(table + 16 * idx + 8) + if big is None or small is None: return None + if not (small & 1): + sap = pde_ap((small >> 1) & 3) + if sap is not None: + r = pte_leaf(pte_addr(small, sap == "FB"), (va >> 12) & 0x1ff) + if r: return r[0] + (va & 0xfff), r[1] + if big & 1: + bap = pte_ap((big >> 1) & 3) + if bap is not None: + a = pte_addr(big, bap == "FB") + (va & ((1 << 21) - 1)) + if a + 8 <= SEMA_PA_MAX: return a, bap + else: + bap = pde_ap((big >> 1) & 3) + if bap is not None: + r = pte_leaf(pte_addr(big, bap == "FB"), (va >> 16) & 0x1f) + if r: return r[0] + (va & 0xffff), r[1] + return None + +def inject(va, pa, ln=0x1000): + cmd = "echo 0x%x,0x%x,0x%x,1 > %s" % (va & ~0xfff, pa & ~0xfff, ln, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print(" INJECT 0x%x -> 0x%x len=0x%x : %s" % + (va & ~0xfff, pa & ~0xfff, ln, "ok" if r.returncode == 0 else "FAIL"), flush=True) + return r.returncode == 0 + +def main(): + global mm + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 300 + end = time.time() + dur + pdbs = [] # (pdb_pa, entries) + done = set() # (pdb, va_page) + seen_va = set() + last_lines = 0 + print("# live_walk start (%.0fs)" % dur, flush=True) + while time.time() < end: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + lines = out.splitlines()[-500:] + out = "\n".join(lines) + # pdb offers/registrations + for m in re.finditer(r"page directory (?:offered as candidate )?pdb=0x([0-9a-f]+) entries=(\d+)", out): + pdb, ent = int(m.group(1), 16), int(m.group(2)) + if (pdb, ent) not in pdbs: + pdbs.append((pdb, ent)) + print("# pdb 0x%x entries=%u" % (pdb, ent), flush=True) + # VAs the service is failing on + vas = [] + for m in re.finditer(r"sema-entry: unread h=0x[0-9a-f]+ idx=\d+ va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"sema-chan: entry\[\d+\] h=0x[0-9a-f]+ eng=\d+ raw=[0-9a-f]+ va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"registered gpfifo=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"GR ring h=0x[0-9a-f]+ rank=\d+ not at fb 0x[0-9a-f]+ yet \(e0=0x[0-9a-f]+ e1=0x[0-9a-f]+ va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"rel=0x([0-9a-f]+) ", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"notif=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"counter page 0x([0-9a-f]+) unreadable", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"release (?:learned|derived) h=0x[0-9a-f]+ pa=0x[0-9a-f]+ \(from release va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + vas = [v for v in vas if 0x4000000 <= v < 0x2000000000] + for va in WINDOW_VAS + vas: + seen_va.add(va) + # resolve + inject through every pdb + for (pdb, ent) in pdbs: + for va in list(seen_va): + key = (pdb, va & ~0xfff) + if key in done: + continue + r = walk(pdb, va, ent) + if not r: + continue + done.add(key) + pa, ap = r + if ap != "SYS": + print("# va=0x%x -> pa=0x%x ap=%s (skip non-sys)" % (va, pa, ap), flush=True) + continue + print("# walk ok pdb=0x%x va=0x%x -> pa=0x%x" % (pdb, va, pa), flush=True) + inject(va, pa) + time.sleep(0.4) + mm.close(); f.close() + print("# live_walk done", flush=True) + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/wall2/nudge.sh b/tools/guest-diag/wall2/nudge.sh new file mode 100755 index 0000000..143b1cd --- /dev/null +++ b/tools/guest-diag/wall2/nudge.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# nudge.sh - host-side readiness-word nudger for the vk_frame wait. +# Polls the guest for the 16 KB nvidiactl mapping words (guest_words.sh), +# and writes 1 to any that are still 0 in the guest RAM file. +# Usage: nudge.sh <duration_seconds> +DUR=${1:-300} +F=/dev/shm/vgpu-guest-ram +END=$(( $(date +%s) + DUR )) +sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/guest_words.sh ubuntu@127.0.0.1:/tmp/guest_words.sh >/dev/null 2>&1 +while [ "$(date +%s)" -lt "$END" ]; do + OUT=$(timeout 20 sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 "sudo bash /tmp/guest_words.sh" 2>/dev/null) + echo "[$(date +%H:%M:%S)] words:"; echo "$OUT" + echo "$OUT" | while read -r VA PA; do + [ -z "$PA" ] && continue + PAD=$((PA)); FOD=$((PAD - 0x80000000)) + [ $FOD -lt 0 ] && continue + CUR=$(sudo dd if=$F bs=4 skip=$((FOD/4)) count=1 2>/dev/null | xxd -p) + if [ "$CUR" = "00000000" ]; then + printf '\x01\x00\x00\x00' | sudo dd of=$F bs=1 seek=$FOD count=4 conv=notrunc 2>/dev/null + echo " NUDGE $PA (va=$VA) -> 1" + else + echo " skip $PA (va=$VA) = $CUR" + fi + done + sleep 2 +done +echo NUDGE-DONE diff --git a/tools/guest-diag/wall2/nudge2.sh b/tools/guest-diag/wall2/nudge2.sh new file mode 100755 index 0000000..16ebcd8 --- /dev/null +++ b/tools/guest-diag/wall2/nudge2.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# nudge2.sh - widened readiness-word nudger: ALL rw-s nvidia mappings <=256K, +# writes 1 to any +0x80 u32 that is 0. +DUR=${1:-300} +F=/dev/shm/vgpu-guest-ram +END=$(( $(date +%s) + DUR )) +sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/guest_words2.sh ubuntu@127.0.0.1:/tmp/guest_words2.sh >/dev/null 2>&1 +while [ "$(date +%s)" -lt "$END" ]; do + OUT=$(timeout 12 sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 "sudo bash /tmp/guest_words2.sh" 2>/dev/null) + [ -n "$OUT" ] && echo "[$(date +%H:%M:%S)] words:" && echo "$OUT" + echo "$OUT" | while read -r VA PA SZ; do + [ -z "$PA" ] && continue + PAD=$((PA)); FOD=$((PAD - 0x80000000)); [ $FOD -lt 0 ] && FOD=$((PAD)) + CUR=$(sudo dd if=$F bs=4 skip=$((FOD/4)) count=1 2>/dev/null | xxd -p) + if [ "$CUR" = "00000000" ]; then + printf '\x01\x00\x00\x00' | sudo dd of=$F bs=1 seek=$FOD count=4 conv=notrunc 2>/dev/null + echo " NUDGE va=$VA pa=$PA -> 1" + fi + done + sleep 1.5 +done +echo NUDGE2-DONE diff --git a/tools/guest-diag/wall2/pdbtick.py b/tools/guest-diag/wall2/pdbtick.py new file mode 100644 index 0000000..d65b79d --- /dev/null +++ b/tools/guest-diag/wall2/pdbtick.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +# pdbtick.py - content-locate the GR0 ring page (deterministic first three +# GPFIFO entries) in the guest RAM file, then hand the verified +# {ring VA -> ring PA} pair to the service (sema_pdb_discover), which +# verifies the parked COPY_SERVER_PDES candidates and falls back to the +# vidmem walk-root sweep. Runs while the client is alive. +import mmap, struct, subprocess, sys, threading, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_pdb_discover" +PAT = struct.pack("<QQQ", 0x0005980004ca0000, 0x8000040004ca0598, + 0x0003d00004ca059c) +RING_VA = 0x4cc0000 + +def gpa_of(x): + return x if x < 0x80000000 else x - 0x80000000 + 0x100000000 + +def main(): + fobj = open(F, "rb") + mm = mmap.mmap(fobj.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + nthr = 8 + step = size // nthr + res = [] + lock = threading.Lock() + def worker(a, b): + CH = 8 << 20 + off = a + while off < b: + d = mm[off:off + CH] + i = d.find(PAT) + while i >= 0: + with lock: + res.append(off + i) + i = d.find(PAT, i + 1) + off += CH - 64 + ts = [] + for k in range(nthr): + a = k * step + b = (k + 1) * step if k < nthr - 1 else size + t = threading.Thread(target=worker, args=(a, b)); t.start(); ts.append(t) + for t in ts: t.join() + if not res: + print("RING NOT FOUND") + return None + hits = [gpa_of(x) for x in res] + print("ring candidates:", [hex(h) for h in hits], flush=True) + for gpa in hits: + cmd = "echo 0x%x,0x%x > %s" % (RING_VA, gpa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("sema_pdb_discover 0x%x,0x%x rc=%d" % (RING_VA, gpa, r.returncode), flush=True) + return hits + + +def loop(): + import time as _t + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + end = _t.time() + dur + fired = set() + while _t.time() < end: + hits = main() + if hits: + new = [h for h in hits if h not in fired] + if new: + fired.update(new) + _t.sleep(1.0) + print("# pdbtick end", flush=True) + +loop() diff --git a/tools/guest-diag/wall2/perfstall5.sh b/tools/guest-diag/wall2/perfstall5.sh new file mode 100644 index 0000000..ae1c161 --- /dev/null +++ b/tools/guest-diag/wall2/perfstall5.sh @@ -0,0 +1,9 @@ +#!/bin/sh +cd /tmp +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk55.out 2>&1 &) +sleep 25 +PID=$(pgrep -x vk_frame | head -1) +cp /proc/$PID/maps /tmp/vk_maps2.txt +sudo perf record -p $PID -g -o /tmp/perf6.data -- sleep 3 2>/dev/null +sudo perf script -i /tmp/perf6.data -F ip,dso 2>/dev/null | grep eglcore | awk '{print $1}' | sort | uniq -c | sort -rn | head -8 > /tmp/spin_addrs.txt +cat /tmp/spin_addrs.txt diff --git a/tools/guest-diag/wall2/pt_bind.py b/tools/guest-diag/wall2/pt_bind.py new file mode 100644 index 0000000..126ed01 --- /dev/null +++ b/tools/guest-diag/wall2/pt_bind.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +# pt_bind.py - bind GMMU PT pages to VA bases and read out VA->PA for the +# client's ring/window family. Strategy: +# 1. ring-page candidates: pages whose 8B-aligned u64 pairs look like +# GPFIFO entries (e0 = push VA & ~3 in the 0x44x-0x4ex families with +# va<4G; e1 top byte 0; length bits sane). +# 2. anchor: for each candidate PA and each family VA, check every 4KB +# page for entry idx=(VA>>12)&0x1ff whose PA part == candidate PA. +# A hit binds (PT page, VA, PA). +# 3. read out: from bound PT pages, decode entries for the whole VA list. +import mmap, struct, sys + +F = "/dev/shm/vgpu-guest-ram" +FAMILY = [ + 0x4020000, 0x4100000, 0x4160000, # r689 eng rings + 0x44a0000, 0x4500000, 0x4560000, # r690 ring VAs + 0x4480000, 0x4485000, 0x4480980, 0x4480f1c, + 0x4ca0000, 0x4ca5000, 0x4d00000, 0x4b17000, +] + +def gpa_to_off(pa): + return pa if pa < 0x80000000 else pa - 0x80000000 + +def off_to_gpa(off): + return off if off < 0x80000000 else off + 0x80000000 + +def pte_pa(v): + return (v >> 8) << 12 + +def main(): + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + print("# file size 0x%x" % size) + + # ---- pass 1: ring-page candidates by content ---- + cands = [] + pos = 0 + step = 4096 + while pos + step <= size: + d = mm[pos:pos+step] + hits = [] + for i in range(0, step, 8): + e0 = int.from_bytes(d[i:i+4], "little") + e1 = int.from_bytes(d[i+4:i+8], "little") + if 0x4400000 <= e0 <= 0x4e00000 and (e0 & 3) == 0 and (e1 & 0xff) == 0: + lw = (e1 >> 10) & 0x1fffff + if 0x40 <= lw <= 0x100000: + hits.append((i, e0, lw << 2)) + if len(hits) >= 3: + cands.append((off_to_gpa(pos), hits)) + if len(cands) >= 12: + break + pos += step + print("# ring-page candidates: %d" % len(cands)) + for pa, hits in cands: + print("cand pa=0x%x hits=%d first=%s" % + (pa, len(hits), ["0x%x/0x%x" % (h[1], h[2]) for h in hits[:4]])) + if not cands: + return 1 + + # ---- pass 2: anchor PT pages for candidate PAs x family VAs ---- + idxs = {} + for va in FAMILY: + idxs.setdefault((va >> 12) & 0x1ff, []).append(va) + cand_pas = set(pa for pa, _ in cands) + bound = [] + pos = 0 + while pos + step <= size: + pg = off_to_gpa(pos) + # skip scanning the candidate pages themselves + if pg not in cand_pas: + d = mm[pos:pos+step] + for idx, vas in idxs.items(): + o = idx * 8 + v = int.from_bytes(d[o:o+8], "little") + if v & 1: + pa = pte_pa(v) + if pa in cand_pas: + bound.append((pg, idx, vas, pa, v & 0xff)) + pos += step + print("# PT anchors found: %d" % len(bound)) + for pg, idx, vas, pa, fl in bound[:20]: + print("PT pa=0x%x entry[0x%x] flags=0x%x -> PA=0x%x (VA cands %s)" % + (pg, idx, fl, pa, ["0x%x" % x for x in vas])) + + # ---- pass 3: read out PT entries for the whole family from each bound PT page ---- + seen_pt = set() + for pg, idx, vas, pa, fl in bound: + if pg in seen_pt: + continue + seen_pt.add(pg) + off = gpa_to_off(pg) + d = mm[off:off+4096] + print("== PT page pa=0x%x family readout:" % pg) + for va in FAMILY: + i = (va >> 12) & 0x1ff + v = int.from_bytes(d[i*8:i*8+8], "little") + if v & 1: + print(" va=0x%x idx=0x%x -> pa=0x%x (raw=0x%x fl=0x%x)" % + (va, i, pte_pa(v), v, v & 0xff)) + else: + print(" va=0x%x idx=0x%x -> absent (raw=0x%x)" % (va, i, v)) + mm.close(); f.close() + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/run75g.sh b/tools/guest-diag/wall2/run75g.sh new file mode 100755 index 0000000..001a80f --- /dev/null +++ b/tools/guest-diag/wall2/run75g.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# run75g.sh - trace + record-word sweep on r680. +cd /tmp/opencode +rm -f /tmp/opencode/t75g-sweep.log +sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/t75g.py ubuntu@127.0.0.1:/tmp/t75g.py >/dev/null 2>&1 +( + sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 \ + 'sudo bash /tmp/slotfill2.sh > /tmp/slotfill75g.log 2>&1 & sleep 0.5; timeout 400 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk_t75g.out 2>&1 & sleep 8; PID=$(pgrep -x vk_frame | head -1); echo "PID=$PID"; sudo timeout 380 gdb -q -batch -p $PID -x /tmp/t75g.py > /tmp/t75g.out 2>&1; echo GDB_DONE rc=$?' +) > /tmp/opencode/t75g-ssh.log 2>&1 & +GPID=$! +sleep 35 +setvar() { echo "$2" | sudo tee /sys/module/vgsp/parameters/$1 >/dev/null; } +mark() { echo "[$(date +%s.%N)] $*" >> /tmp/opencode/t75g-sweep.log; } +# phase 1: baseline echo (client already polling) +mark "baseline echo" +sleep 20 +# phase 2: all-zero records +setvar wall2_a097_rec0 0; setvar wall2_a097_rec1 0; setvar wall2_a097_rec2 0; setvar wall2_a097_rec3 0 +setvar wall2_a097_rec4 0; setvar wall2_a097_rec5 0; setvar wall2_a097_rec6 0; setvar wall2_a097_rec7 0 +mark "records all zero"; sleep 20 +# phase 3: progress=100 +setvar wall2_a097_rec1 100 +mark "rec1=100"; sleep 20 +# phase 4: progress=1000 +setvar wall2_a097_rec1 1000 +mark "rec1=1000"; sleep 20 +# phase 5: progress=100000 +setvar wall2_a097_rec1 100000 +mark "rec1=100000"; sleep 20 +# phase 6: progress=1000000 + rec0=0 +setvar wall2_a097_rec1 1000000 +setvar wall2_a097_rec0 0 +mark "rec1=1000000 rec0=0"; sleep 20 +# phase 7: all ones +setvar wall2_a097_rec0 3735928559; setvar wall2_a097_rec1 3735928559; setvar wall2_a097_rec2 3735928559; setvar wall2_a097_rec3 3735928559 +mark "rec0-3=0xdeadbeef"; sleep 20 +# phase 8: back to echo +setvar wall2_a097_rec0 4294967295; setvar wall2_a097_rec1 4294967295; setvar wall2_a097_rec2 4294967295; setvar wall2_a097_rec3 4294967295 +mark "baseline echo again" +wait $GPID +echo SWEEP-DONE diff --git a/tools/guest-diag/wall2/runbook.py b/tools/guest-diag/wall2/runbook.py new file mode 100644 index 0000000..256d255 --- /dev/null +++ b/tools/guest-diag/wall2/runbook.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +# runbook.py - watch for the GR0 channel ring registration, hand the pair to +# sema_pdb_discover, then (after the pdb anchor) issue bounded +# sema_chan_rewind calls so entries skipped before the anchor re-decode. +import re, subprocess, sys, time + +PDB = "/sys/module/vgsp/parameters/sema_pdb_discover" +REW = "/sys/module/vgsp/parameters/sema_chan_rewind" + +def dmesg_tail(n=150): + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + return out.splitlines()[-n:] + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 180 + end = time.time() + dur + fired = set() + rewinds = {} + print("# runbook start (%.0fs)" % dur, flush=True) + while time.time() < end: + for ln in dmesg_tail(): + m = re.search(r"sema-chan: h=0x([0-9a-f]+) eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+) ent=\d+ userd=0x[0-9a-f]+ as=\d+ put=\d+ get=\d+ ring rc=0 pa=0x([0-9a-f]+)", ln) + if not m: + continue + h = int(m.group(1), 16) + gv = int(m.group(2), 16) + rp = int(m.group(3), 16) + key = (h, gv, rp) + if key in fired: + # after the anchor has had time to land, rewind this channel + if h == 0xbeec0100: + t0 = rewinds.setdefault(key, time.time()) + n = int((time.time() - t0) // 8) + done = rewinds.get(("done", key), 0) + if n > done: + subprocess.run(["sudo", "sh", "-c", + "echo all > %s" % REW], capture_output=True) + rewinds[("done", key)] = n + print("[%s] REWIND all (h=0x%x, n=%d)" % ( + time.strftime("%H:%M:%S"), h, n), flush=True) + continue + fired.add(key) + r = subprocess.run(["sudo", "sh", "-c", + "echo 0x%x,0x%x > %s" % (gv, rp, PDB)], + capture_output=True) + print("[%s] TRIGGER h=0x%x va=0x%x pa=0x%x rc=%d" % ( + time.strftime("%H:%M:%S"), h, gv, rp, r.returncode), flush=True) + # report decodes/releases + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + for ln in out.splitlines()[-8:]: + if ("sema-push: h=0xbeec0100" in ln and "have_rel=1" in ln) or \ + "registered after verified" in ln or "rewind" in ln or \ + ("sema-push: h=0xbeec0100" in ln and "len=1432" in ln): + print("LOG %s" % ln.strip()[-170:], flush=True) + time.sleep(0.4) + print("# runbook end", flush=True) + +main() diff --git a/tools/guest-diag/wall2/scan_r673.py b/tools/guest-diag/wall2/scan_r673.py new file mode 100644 index 0000000..71a0128 --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +# Analyze the live r673 guest RAM: locate every reference to the validated ring +# PA (0x12a783000) including flag-bit variants, and characterize the record +# arrays around each hit. Also dump the two pages named in the handoff. +import numpy as np, struct, sys, time + +RAM="/dev/shm/vgpu-guest-ram" +RING=0x12a783000 +MASK=~0xfff & ((1<<64)-1) + +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) + +img=open(RAM,"rb") +import os +fsz=os.path.getsize(RAM) +print("ram size 0x%x" % fsz) + +t0=time.time() +hits=[] +CH=64<<20 +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero((a & np.uint64(MASK))==np.uint64(RING))[0] + for i in idx: + off=pos+int(i)*8 + val=int(a[i]) + hits.append((off,val)) +print("hits=%d scan=%.1fs" % (len(hits), time.time()-t0)) +for off,val in hits: + print(" off=0x%x gpa=0x%x val=0x%x (low=0x%x)" % (off,off2gpa(off),val,val&0xfff)) + +# characterize each hit page +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +def plausible_pa(v): return 0x100000000 <= (v & MASK) < 0x180000000 +def ptrish(v): + return (0xffff800000000000 <= v < 0xffffffffffffffff) + +for off,val in hits: + gpa=off2gpa(off) + page=gpa & MASK + rec_off=gpa-page + d=rd(page,0x1000) + print("=== hit gpa=0x%x page=0x%x rec_off=0x%x ===" % (gpa,page,rec_off)) + for stride in (0x20,0x40,0x60,0x30,0x50,0x28,0x48): + # record start such that the hit qword falls inside the record + for rel in range(0,stride,8): + start=rec_off-rel + if start<0: continue + # walk records; count plausible PA fields anywhere in each record + n_pa=0; n_ptr=0; n_1000=0 + for o in range(start,0x1000-stride,stride): + rec=d[o:o+stride] + qs=struct.unpack_from("<%dQ"%(stride//8),rec,0) + if any(plausible_pa(q) for q in qs): n_pa+=1 + if any(ptrish(q) for q in qs): n_ptr+=1 + if 0x1000 in qs: n_1000+=1 + if n_pa>=8: + print(" stride=0x%x start=0x%x: PA-records=%d ptr-records=%d 0x1000=%d" % (stride,start,n_pa,n_ptr,n_1000)) + # dump the first 0x100 bytes of the page and around the hit + print(" page head: %s" % d[:0x60].hex()) + s=max(0,rec_off-0x40); e=min(0x1000,rec_off+0x80) + print(" around hit [0x%x..0x%x]: %s" % (s,e,d[s:e].hex())) + +# dump the two named pages +for pa in (0x101499000,0x127cf2000): + d=rd(pa,0x1000) + print("=== page 0x%x ===" % pa) + if d is None: print(" unreadable"); continue + for o in range(0,0x120,0x20): + qs=struct.unpack_from("<4Q",d,o) + print(" +0x%03x: %016x %016x %016x %016x" % ((o,)+qs)) diff --git a/tools/guest-diag/wall2/scan_r673b.py b/tools/guest-diag/wall2/scan_r673b.py new file mode 100644 index 0000000..11613b1 --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673b.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# r673: dump the full record page 0x101499000 and search RAM for the ring/push +# VA structures to identify which record is the 16-page push (VA 0x4500000). +import numpy as np, struct, os, time + +RAM="/dev/shm/vgpu-guest-ram" +MASK=~0xfff & ((1<<64)-1) +img=open(RAM,"rb") +fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +print("== full record page 0x101499000 ==") +d=rd(0x101499000,0x1000) +nrec=0 +for o in range(0,0x1000,32): + p,f,pa,sz=struct.unpack_from("<QQQQ",d,o) + if p==0 and f==0 and pa==0 and sz==0: continue + nrec+=1 + print(" rec[%3d] vmap=%016x flags=%016x pa=%012x size=0x%x" % (o//32,p,f,pa,sz)) +print("nonempty records:",nrec) + +print() +print("== search RAM for qword 0x4520000 (ring VA) ==") +t0=time.time(); CH=64<<20 +for pat_name,pat in (("0x4520000",0x4520000),("0x4500000",0x4500000)): + print("-- pattern %s --" % pat_name) + hits=[] + for pos in range(0,fsz,CH): + img.seek(pos); dd=img.read(min(CH,fsz-pos)) + n=len(dd)//8 + a=np.frombuffer(dd[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(pat))[0] + for i in idx: + hits.append(pos+int(i)*8) + print(" hits=%d (%.1fs)" % (len(hits),time.time()-t0)) + for off in hits[:40]: + gpa=off2gpa(off); page=gpa&MASK + # show 32 bytes before/after + dd=rd(page,0x1000) + if not dd: continue + ro=gpa-page + s=max(0,ro-32); e=min(0x1000,ro+64) + print(" gpa=0x%x page=0x%x off=0x%x ctx=%s" % (gpa,page,ro,dd[s:e].hex())) + if len(hits)>40: print(" ... %d more" % (len(hits)-40)) diff --git a/tools/guest-diag/wall2/scan_r673c.py b/tools/guest-diag/wall2/scan_r673c.py new file mode 100644 index 0000000..db69b5d --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673c.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +# r673: find the push page. Patterns: ring entry raw qwords, the sem VA, and +# method-stream signatures. Dump contexts. +import numpy as np, struct, os, time + +RAM="/dev/shm/vgpu-guest-ram" +img=open(RAM,"rb"); fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +pats=[ + ("entry0raw", 0x0005980004500000), + ("entry1raw", 0x0000040004500598), + ("entry2raw", 0x0003d0000450059c), + ("sem_va", 0x4500598), + ("sem_va2", 0x450059c), + ("m20038090", 0x20038090), + ("push_va", 0x4500000), +] +CH=64<<20 +for name,pat in pats: + hits=[] + t0=time.time() + for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(pat))[0] + for i in idx: hits.append(pos+int(i)*8) + print("== %s pat=0x%x hits=%d (%.1fs)" % (name,pat,len(hits),time.time()-t0)) + for off in hits[:12]: + gpa=off2gpa(off); page=gpa&~0xfff + d=rd(page,0x1000) + ro=gpa-page + s=max(0,ro-24); e=min(0x1000,ro+40) + print(" gpa=0x%x page=0x%x off=0x%x ctx=%s" % (gpa,page,ro,d[s:e].hex() if d else "?")) + if len(hits)>12: print(" ... %d more" % (len(hits)-12)) diff --git a/tools/guest-diag/wall2/scan_r673d.py b/tools/guest-diag/wall2/scan_r673d.py new file mode 100644 index 0000000..1fa89ca --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673d.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +# r673 deep-dive: all ring-PA refs with low bits, ring page dump, RM record +# pages, and a RAM-wide scan for 0xa3bf20 VA-table records mentioning 0x4500000. +import numpy as np, struct, os, time + +RAM="/dev/shm/vgpu-guest-ram" +RING=0x12a783000; TAG=0xa3bf20 +img=open(RAM,"rb"); fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +MASK=~0xfff & ((1<<64)-1) +print("== all qwords (v & ~0xfff)==ring_pa ==") +CH=64<<20 +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero((a & np.uint64(MASK))==np.uint64(RING))[0] + for i in idx: + off=pos+int(i)*8 + gpa=off2gpa(off); page=gpa&~0xfff + dd=rd(page,0x1000); ro=gpa-page + s=max(0,ro-32); e=min(0x1000,ro+48) + print(" gpa=0x%x low=0x%x ctx=%s" % (gpa,int(a[i])&0xfff,dd[s:e].hex() if dd else "?")) + +print() +print("== ring page 0x%x first 0x120 ==" % RING) +d=rd(RING,0x120) +for o in range(0,0x120,0x20): + print(" +0x%03x: %s" % (o,d[o:o+0x20].hex())) + +for pa in (0x1013a6000,0x104acb000,0x114760000,0x1202cd000,0x13e2c1000,0x13e62e000): + print() + print("== page 0x%x first 0x140 ==" % pa) + d=rd(pa,0x140) + for o in range(0,0x140,0x20): + q=struct.unpack_from("<4Q",d,o) + print(" +0x%03x: %016x %016x %016x %016x" % ((o,)+q)) + +print() +print("== RAM-wide: 32-byte records {va, cpu, 0xa3bf20, pages} with va==0x4500000 ==") +t0=time.time() +found=0 +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(TAG))[0] + for i in idx: + off=pos+int(i)*8 + gpa=off2gpa(off); page=gpa&~0xfff + dd=rd(page,0x1000) + ro=gpa-page + if ro>=16 and ro+8<=0x1000: + va,cpu=struct.unpack_from("<QQ",dd,ro-16) + pgs=struct.unpack_from("<Q",dd,ro+8)[0] + if va==0x4500000 or (0x4400000<=va<=0x4600000): + print(" rec gpa=0x%x va=0x%x cpu=0x%x pages=0x%x" % (gpa,va,cpu,pgs)) + found+=1 +print("total", found, "(%.1fs)" % (time.time()-t0)) diff --git a/tools/guest-diag/wall2/session73-notes.md b/tools/guest-diag/wall2/session73-notes.md new file mode 100644 index 0000000..d2079d3 --- /dev/null +++ b/tools/guest-diag/wall2/session73-notes.md @@ -0,0 +1,51 @@ +# SESSION 73 notes (wall #2 live validation, r674 + r675) + +## r674 cycle results (8 client runs, all VKFAIL -4) +- Client vk_frame reproducibly fails at vkCreateDevice (-4) as before. +- wall2_resolver2 (content-first CE group 0x20038090 + tuple arrays): no candidates found + during the stall; tuple arrays contain the ring PA but not the push. +- livecap/livecap2/livecap3 (0x20038090 shape / 0x20050017 host-release shape): NO data + hits in guest RAM during the stall; only driver CODE pages match. +- Conclusion: the GR channel-init push content is NOT findable in /dev/shm/vgpu-guest-ram + by these patterns, even live. (The ring page content IS there and matches the service's + decode.) + +## Structural findings +- The push is RM-built (nvidia.ko), NOT UVM: kprobe on + uvm_hal_turing_host_semaphore_release never fired during the client run. +- nvidia.ko functions are `notrace` -> kprobe_events refuses them + ("Could not probe notrace function ... [nvidia]"). UVM module probes DO work. + QEMU gdbstub hardware breakpoints work fine (hbreak *addr). +- Service rescue matcher is CE-shaped only (hdr 0x20038090, hi==1, lo&0xfff==slot, + w5/v5 shape) -> can never match a GR push. Entries 0/1/2 are consumed as "stale" + after ~150 failed rescues each; gr_wedge=1. +- channelFillGpFifo(channel, idx, len) disasm: rdi=channel; +0x178=gpu; + +0xe8=cached ring CPU VA; +0xb8=push GPU VA base; +0x48=stride; + +0x110=class (0xc5b5 seen for a CE channel; GR = 0xc46f). + Entry encoding: [0]=va&~3 (u32), [1]=(va>>32 &0xff)|((len<<8)&0x7ffffc00). +- r674 ring PAs per run: 0x12a783000(?) no: run1 0x1050b6000, run2 0x10297d000, + later 0x102317000 (ring line format: "h=0x... gpfifo=0x44a0000 ent=32768 userd=0xffe00 as=2 put=3 get=0 ring rc=0 pa=0x... ap=1"). +- Push VA = 0x4480000, ring VA = 0x44a0000 (delta 0x20000); entry0 len=1432, + entry1 {va=0x4480598,len=4}, entry2 {va=0x448059c,len=976}. +- The pushbuffer (UVM) = 0x20000 (32 pages) aligned; UVM_PUSHBUFFER_SIZE=0x20000 + (UVM_MAX_PUSH_SIZE=0x400). UVM_RM_MEM_TYPE_SYS with CPU map. +- RM memdesc records {size=0x20000, flags=0x03008000, first_PA, ptr} exist in RAM + (11 found post-run) but their PAs did not contain the release group. +- RM window record {0x4480000, 0x20000} + head/tail list nodes exist in RAM. +- The r673 VA-table record {0x10, 0x4500000, 0x786fd9598ab0, 0xa3bf20} has a + "cpu" field that may be a user VA of the client (0x64..-0x78.. range). + +## Ops +- r674 wedged after the gdb-interrupted vk31 run (SSH + guest agent dead, vCPUs idle). + Fresh cycle r675 started. gdb674d still attached to the dead QEMU (never kill gdbs). +- /dev/shm 53%. + +## r675 plan +1. gdb675.gdb: hbreak channelFillGpFifo if class(+0x110)==0xc46f; on hit dump the + channel struct pointer fields (see gdb675.gdb) -> identify the push's CPU mapping + (direct-map or vmalloc) by content. +2. Compute PA (direct-map: VA - 0xffff8916c0000000 style base) and read the push + content from the RAM file; find the release group. +3. Inject 0x4480000 -> PA via sema_map_add BEFORE/DURING the stall; watch the service + apply the release and vkCreateDevice pass. +4. Durability: integrate into pin_gr_stream.py demand path or a service hook. diff --git a/tools/guest-diag/wall2/slotfill.sh b/tools/guest-diag/wall2/slotfill.sh new file mode 100644 index 0000000..a3416a9 --- /dev/null +++ b/tools/guest-diag/wall2/slotfill.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Watch the vk_frame client and fill the completion-slot u16 (base+0x2e) in +# every small /dev/nvidiactl rw-s mapping while it waits. +cd /tmp +rm -f /tmp/slotfill.log +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk76.out 2>&1 &) +PID="" +for i in $(seq 1 100); do PID=$(pgrep -x vk_frame | head -1); [ -n "$PID" ] && break; sleep 0.1; done +echo "pid=$PID" >> /tmp/slotfill.log +END=$((SECONDS+60)) +while [ $SECONDS -lt $END ]; do + [ -d /proc/$PID ] || break + for m in $(grep 'nvidiactl' /proc/$PID/maps | awk '$2 ~ /rw-s/ {print $1}'); do + lo=${m%-*}; hi=${m#*-} + size=$((0x$hi - 0x$lo)) + [ $size -gt 65536 ] && continue + off=$((0x$lo + 0x2e)) + v=$(dd if=/proc/$PID/mem bs=1 skip=$off count=2 2>/dev/null | xxd -p) + if [ "$v" = "ffff" ]; then + printf '\x00\x00' | dd of=/proc/$PID/mem bs=1 seek=$off count=2 conv=notrunc 2>/dev/null + echo "$(date +%s.%N) FILL $m+0x2e was ffff" >> /tmp/slotfill.log + fi + done + sleep 0.2 +done +echo "done" >> /tmp/slotfill.log +echo "=== client ===" +cat /tmp/vk76.out +echo "=== fills ===" +grep -c FILL /tmp/slotfill.log diff --git a/tools/guest-diag/wall2/slotfill2.sh b/tools/guest-diag/wall2/slotfill2.sh new file mode 100644 index 0000000..33ad702 --- /dev/null +++ b/tools/guest-diag/wall2/slotfill2.sh @@ -0,0 +1,19 @@ +#!/bin/bash +cd /tmp +PID="" +for i in $(seq 1 200); do PID=$(pgrep -x vk_frame | head -1); [ -n "$PID" ] && break; sleep 0.05; done +END=$((SECONDS+45)) +while [ $SECONDS -lt $END ]; do + [ -d /proc/$PID ] || break + for m in $(grep 'nvidiactl' /proc/$PID/maps 2>/dev/null | awk '$2 ~ /rw-s/ {print $1}'); do + lo=${m%-*}; hi=${m#*-} + size=$((0x$hi - 0x$lo)) + [ $size -gt 65536 ] && continue + off=$((0x$lo + 0x2e)) + v=$(dd if=/proc/$PID/mem bs=1 skip=$off count=2 2>/dev/null | xxd -p) + if [ "$v" = "ffff" ]; then + printf '\x00\x00' | dd of=/proc/$PID/mem bs=1 seek=$off count=2 conv=notrunc 2>/dev/null + fi + done + sleep 0.1 +done diff --git a/tools/guest-diag/wall2/slots_probe.sh b/tools/guest-diag/wall2/slots_probe.sh new file mode 100644 index 0000000..71e6fb7 --- /dev/null +++ b/tools/guest-diag/wall2/slots_probe.sh @@ -0,0 +1,11 @@ +#!/bin/bash +PID=$1 +sudo grep -E 'rw-s' /proc/$PID/maps | grep nvidiactl | while read -r line; do + RNG=$(echo "$line" | awk '{print $1}') + LO=${RNG%-*}; HI=${RNG#*-} + SZ=$((0x$HI-0x$LO)) + [ $SZ -gt 262144 ] && continue + V2E=$(sudo dd if=/proc/$PID/mem bs=1 skip=$((0x$LO+0x2e)) count=2 2>/dev/null | xxd -p) + V0E=$(sudo dd if=/proc/$PID/mem bs=1 skip=$((0x$LO+0xe)) count=2 2>/dev/null | xxd -p) + echo "$RNG sz=$SZ w2e=$V2E w0e=$V0E" +done diff --git a/tools/guest-diag/wall2/stracevk.sh b/tools/guest-diag/wall2/stracevk.sh new file mode 100644 index 0000000..b2752c4 --- /dev/null +++ b/tools/guest-diag/wall2/stracevk.sh @@ -0,0 +1,7 @@ +#!/bin/sh +cd /tmp +sudo timeout 90 strace -f -T -tt -e trace=ioctl -o /tmp/vk_strace.out /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk58.out 2>&1 +echo "=== client out ===" +cat /tmp/vk58.out +echo "=== ioctl tail ===" +tail -30 /tmp/vk_strace.out diff --git a/tools/guest-diag/wall2/t101.py b/tools/guest-diag/wall2/t101.py new file mode 100644 index 0000000..436c609 --- /dev/null +++ b/tools/guest-diag/wall2/t101.py @@ -0,0 +1,55 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception as e: + if state["chits"] <= 4: + print("chitfail %s" % e, flush=True) + if state["cleared"] >= 24: + self.enabled = False + print("BC-DISABLED", flush=True) + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("bpfail %s" % e, flush=True) +print("T101-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t102.py b/tools/guest-diag/wall2/t102.py new file mode 100644 index 0000000..759d974 --- /dev/null +++ b/tools/guest-diag/wall2/t102.py @@ -0,0 +1,31 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +try: + pc = int(gdb.parse_and_eval("$pc")) + print("PC 0x%x off=0x%x" % (pc, pc - base if base else 0)) +except Exception as e: + print("pcfail %s" % e) +try: + C = base + 0x2070B7C + print("A8COUNT %u" % int(gdb.parse_and_eval("*(unsigned int*)0x%x" % C))) +except Exception as e: + print("countfail %s" % e) +try: + print(gr("bt 6", to_string=True)) +except Exception as e: + print("btfail %s" % e) +gr("detach") +gr("quit") +print("T102-DONE") diff --git a/tools/guest-diag/wall2/t104.py b/tools/guest-diag/wall2/t104.py new file mode 100644 index 0000000..5a4c059 --- /dev/null +++ b/tools/guest-diag/wall2/t104.py @@ -0,0 +1,56 @@ +import gdb, time +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid), flush=True) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "t0":time.time(), "last":time.time()} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + state["last"] = time.time() + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + if state["cleared"] >= 8 or state["chits"] >= 40 or (time.time() - state["last"]) > 25.0: + self.enabled = False + print("BC-EXIT cleaned=%d chits=%d" % (state["cleared"], state["chits"]), flush=True) + return True + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("bpfail %s" % e, flush=True) +print("T104-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t106.py b/tools/guest-diag/wall2/t106.py new file mode 100644 index 0000000..ceaac44 --- /dev/null +++ b/tools/guest-diag/wall2/t106.py @@ -0,0 +1,61 @@ +import gdb, time +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid), flush=True) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "done":False} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + if state["done"]: + return False + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + if state["cleared"] >= 5 or state["chits"] >= 14: + state["done"] = True + try: + gr("detach") + print("BC-DETACHED cleaned=%d chits=%d" % (state["cleared"], state["chits"]), flush=True) + except Exception as e: + print("detachfail %s" % e, flush=True) + return True + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("contend %s" % e, flush=True) +print("T106-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t112c-gdbstub.py b/tools/guest-diag/wall2/t112c-gdbstub.py new file mode 100644 index 0000000..27d0b9d --- /dev/null +++ b/tools/guest-diag/wall2/t112c-gdbstub.py @@ -0,0 +1,42 @@ +import gdb + +gr = gdb.execute + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +state = {"hits": {}} + +class HB(gdb.Breakpoint): + def __init__(self, pa, tag, quiet_after=8): + super().__init__("*0x%x" % pa, type=gdb.BP_BREAKPOINT, internal=False) + self.tag = tag + self.quiet_after = quiet_after + self.silent = True + def stop(self): + n = state["hits"].get(self.tag, 0) + 1 + state["hits"][self.tag] = n + try: + rdi = int(gdb.parse_and_eval("$rdi")) + rsi = int(gdb.parse_and_eval("$rsi")) + rdx = int(gdb.parse_and_eval("$rdx")) + rcx = int(gdb.parse_and_eval("$rcx")) + r8 = int(gdb.parse_and_eval("$r8")) + r9 = int(gdb.parse_and_eval("$r9")) + if n <= self.quiet_after or self.tag in ("PRED", "COMPR"): + print("HIT %s #%d rdi=0x%x rsi=0x%x rdx=0x%x rcx=0x%x r8=0x%x r9=0x%x" % + (self.tag, n, rdi, rsi, rdx, rcx, r8, r9), flush=True) + except Exception as e: + print("err", e, flush=True) + return False + +HB(0xffffffffc0c403b0, "VIDH") +HB(0xffffffffc0d667a0, "PMA", 4) +HB(0xffffffffc0d6ae30, "PRED") +HB(0xffffffffc0d57d60, "COMPR", 16) +HB(0xffffffffc0d607f0, "SCRUB", 4) +print("HB-SET", flush=True) +gr("continue") +print("T112B-DONE %s" % state["hits"], flush=True) diff --git a/tools/guest-diag/wall2/t120.py b/tools/guest-diag/wall2/t120.py new file mode 100644 index 0000000..68a1b8b --- /dev/null +++ b/tools/guest-diag/wall2/t120.py @@ -0,0 +1,148 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2 and size == 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + state["n"] += 1 + print("HIT149 n=%d attr=0x%x flags=0x%x owner=0x%x align=0x%x" % ( + state["n"], attr, flags, owner, u64(arg + 0x60)), flush=True) + if state["n"] == 1: + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t121.py b/tools/guest-diag/wall2/t121.py new file mode 100644 index 0000000..1ae25ed --- /dev/null +++ b/tools/guest-diag/wall2/t121.py @@ -0,0 +1,151 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2: + state["n"] += 1 + if state["n"] >= 120: + print("VH#%d size=0x%x align=0x%x attr=0x%x attr2=0x%x flags=0x%x owner=0x%x range=0x%x-0x%x" % ( + state["n"], size, u64(arg + 0x60), attr, u32(arg + 0x90), flags, owner, + u64(arg + 0x80), u64(arg + 0x88)), flush=True) + if size == 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + print("HIT149 attr=0x%x flags=0x%x" % (attr, flags), flush=True) + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t122.py b/tools/guest-diag/wall2/t122.py new file mode 100644 index 0000000..8c9c6fd --- /dev/null +++ b/tools/guest-diag/wall2/t122.py @@ -0,0 +1,151 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2: + state["n"] += 1 + if state["n"] >= 120: + print("VH#%d size=0x%x align=0x%x attr=0x%x attr2=0x%x flags=0x%x owner=0x%x range=0x%x-0x%x" % ( + state["n"], size, u64(arg + 0x60), attr, u32(arg + 0x90), flags, owner, + u64(arg + 0x80), u64(arg + 0x88)), flush=True) + if size >= 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + print("HIT149 size=0x%x attr=0x%x flags=0x%x" % (size, attr, flags), flush=True) + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t75g.py b/tools/guest-diag/wall2/t75g.py new file mode 100644 index 0000000..2a0881a --- /dev/null +++ b/tools/guest-diag/wall2/t75g.py @@ -0,0 +1,170 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid + +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found") + gdb.execute("detach") + sys.exit(1) +print("EGLBASE 0x%x PID %d" % (base, pid)) + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') +def u64(a): + return int.from_bytes(rd(a, 8), 'little') + +state = {"out": 0, "E": 0, "S": 0, "armed150": False, "armedS": False, + "bt_done": False, "post_done": False} + +class B(gdb.Breakpoint): + def __init__(self, spec, fn, limit=100000): + super().__init__(spec, internal=False) + self.fn = fn; self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + stop = bool(self.fn(self)) + except Exception as e: + print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e)) + stop = False + return stop or self.hits >= self.limit + +def a97(bp): + rbp = int(gdb.parse_and_eval("$rbp")) + r14 = int(gdb.parse_and_eval("$r14")) + out = u64(rbp+0x10) + state["out"] = out + q = rd(out, 0x28) + rec_ptr = int.from_bytes(q[8:16], 'little') + print("A97 out=0x%x count=%d f10=%08x f20=%08x f24=%08x recp=0x%x" % + (out, int.from_bytes(q[0:4], 'little'), int.from_bytes(q[0x10:0x14], 'little'), + int.from_bytes(q[0x20:0x24], 'little'), int.from_bytes(q[0x24:0x28], 'little'), rec_ptr)) + if rec_ptr and rec_ptr > 0x1000: + try: + rec = rd(rec_ptr, 16) + print("A97 rec=%08x %08x %08x %08x" % + tuple(int.from_bytes(rec[i*4:i*4+4], 'little') for i in range(4))) + except Exception as e: + print("A97 rec-unreadable", e) + return False + +def post(bp): + if state["post_done"]: + return False + state["post_done"] = True + rsp = int(gdb.parse_and_eval("$rsp")) + q = rd(rsp, 0x40) + print("POST rsp=0x%x out=0x%x" % (rsp, state["out"])) + for i in range(0, 0x40, 8): + v = int.from_bytes(q[i:i+8], 'little') + print("POST rsp+%02x = 0x%016x" % (i, v)) + if state["out"]: + oq = rd(state["out"], 0x40) + for i in range(0, 0x40, 8): + v = int.from_bytes(oq[i:i+8], 'little') + print("POST out+%02x = 0x%016x" % (i, v)) + return False + +def t150(bp): + rbp = int(gdb.parse_and_eval("$rbp")) + if not state.get("vt_done"): + state["vt_done"] = True + rdi = u64(rbp+0x148) + vt = u64(rdi) if rdi else 0 + m118 = u64(vt+0x118) if vt else 0 + m170 = u64(vt+0x170) if vt else 0 + print("VT E148=0x%x vt=0x%x m118=0x%x m170=0x%x" % (rdi, vt, m118, m170)) + if not state.get("eng_done"): + state["eng_done"] = True + n = u32(rbp+8) + ids = [u32(rbp+0x54+i*4) for i in range(min(n,8))] + print("ENG n=%d ids=%s" % (n, ",".join("0x%x" % x for x in ids))) + + # cfg area pointer is rip-relative 0x2019320 at runtime = base + (0x2019320 - 0) + # eglcore base == min load addr; file offset == vaddr for the .text; data seg present? + try: + vals = rd(base + 0x2019320 + 0xf80, 0xe0) + for i in range(0, 0xe0, 0x10): + v4 = [int.from_bytes(vals[i+j:i+j+4],'little') for j in (0,4,8,12)] + print("CFG +0x%03x = %s" % (0xf80+i, ["0x%x"%x for x in v4])) + except Exception as e: + print("CFG dump err", e) + v = u32(rbp+0x150) + b38 = rd(rbp+0x1b038, 1)[0]; b39 = rd(rbp+0x1b039, 1)[0] + p = u64(rbp+0x1b030); s8 = u64(p+8) if p else 0 + print("T150 E=0x%x v=0x%08x b038=%d b039=%d p=0x%x p8=0x%x" % + (rbp, v, b38, b39, p, s8)) + if not state["bt_done"]: + state["bt_done"] = True + print(gdb.execute("bt 8", to_string=True)) + if not state["armed150"]: + state["armed150"] = True + state["E"] = rbp + try: + gdb.execute("frame 1") + S = int(gdb.parse_and_eval("$rbp")) + gdb.execute("frame 0") + print("S=0x%x S120=%08x S128=%08x" % (S, u32(S+0x120), u32(S+0x128))) + state["S"] = S + except Exception as e: + print("S capture err", e) + S = 0 + W("*(unsigned int *)0x%x" % (rbp+0x150), rbp, 0x150, 4, "W150") + W("*(unsigned char *)0x%x" % (rbp+0x1b038), rbp, 0x1b038, 1, "W038") + W("*(unsigned char *)0x%x" % (rbp+0x1b039), rbp, 0x1b039, 1, "W039") + if S: + W("*(unsigned int *)0x%x" % (S+0x120), S, 0x120, 4, "W120") + print("WATCHES armed E=0x%x S=0x%x" % (rbp, S)) + return False + +def sl(bp): + rbp = int(gdb.parse_and_eval("$rbp")) + print("SLOOP S=0x%x S120=%08x" % (rbp, u32(rbp+0x120))) + if not state["armedS"]: + state["armedS"] = True + W("*(unsigned int *)0x%x" % (rbp+0x120), rbp, 0x120, 4, "W120") + print("W120 armed on S+0x120") + return False + +class W(gdb.Breakpoint): + def __init__(self, spec, E, off, size, tag): + super().__init__(spec, type=gdb.BP_WATCHPOINT) + self.silent = True; self.E = E; self.off = off; self.size = size + self.tag = tag; self.hits = 0 + def stop(self): + self.hits += 1 + cur = int.from_bytes(rd(self.E+self.off, self.size), 'little') + pc = int(gdb.parse_and_eval("$pc")) + print("%s v=0x%x pc=0x%x (rel 0x%x)" % (self.tag, cur, pc, pc-base)) + if self.hits <= 8: + print(gdb.execute("bt 10", to_string=True)) + return self.hits >= 400 + +B("*0x%x" % (base+0xecd43d), a97, 220) +B("*0x%x" % (base+0xd5e4d3), post, 100000) +B("*0x%x" % (base+0xd5e449), t150, 220) +B("*0x%x" % (base+0xd6ef76), sl, 240) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +print("TRACE-END") +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t79.py b/tools/guest-diag/wall2/t79.py new file mode 100644 index 0000000..c2fe381 --- /dev/null +++ b/tools/guest-diag/wall2/t79.py @@ -0,0 +1,99 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1) +print("EGLBASE 0x%x PID %d" % (base, pid)) + +maps = [] +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x, 16) for x in parts[0].split('-')] + maps.append((lo, hi, parts[1], parts[-1] if len(parts) > 5 else "anon")) + except Exception: + pass + +def resolve(addr): + for lo, hi, perm, name in maps: + if lo <= addr < hi: + return "0x%x-0x%x %s %s" % (lo, hi, perm, name) + return "?" + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') +def u64(a): + return int.from_bytes(rd(a, 8), 'little') + +state = {"done": False} + +class B(gdb.Breakpoint): + def __init__(self, spec, fn, limit=100000): + super().__init__(spec, internal=False) + self.fn = fn; self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + stop = bool(self.fn(self)) + except Exception as e: + print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e)) + stop = False + return stop or self.hits >= self.limit + +def loophead(bp): + if state["done"]: + return False + state["done"] = True + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx+0x70) + print("WAIT rbx=0x%x sub=0x%x" % (rbx, sub)) + try: + obj2 = u64(sub+0x58) + off = u32(obj2+0x60) + holder = u64(obj2+0x58) + base2 = u64(holder+0x58) + word_addr = base2 + off + val = u32(word_addr) + print("WAIT obj2=0x%x off=0x%x holder=0x%x base=0x%x word=0x%x val=0x%x" % + (obj2, off, holder, base2, word_addr, val)) + print("WAIT word map:", resolve(word_addr)) + print("WAIT holder obj2+0x50=0x%x obj2+0x64=0x%x" % (u64(obj2+0x50), u32(obj2+0x64))) + try: + hd = rd(holder, 0x30) + print("WAIT holder dump:", " ".join("%02x" % b for b in hd)) + except Exception as e: + print("WAIT holder dump err", e) + except Exception as e: + print("WAIT chain err", e) + # dump the client's nvidiactl rw-s mappings + print("WAIT nvidiactl maps:") + for lo, hi, perm, name in maps: + if "nvidiactl" in name and "rw-s" in perm: + print(" 0x%x-0x%x size=0x%x" % (lo, hi, hi-lo)) + return True + +B("*0x%x" % (base+0xdcebd9), loophead, 1) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +print("TRACE-END") +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t80r.py b/tools/guest-diag/wall2/t80r.py new file mode 100644 index 0000000..ac2263b --- /dev/null +++ b/tools/guest-diag/wall2/t80r.py @@ -0,0 +1,32 @@ +import gdb, sys +inf = gdb.selected_inferior() +pid = inf.pid +cands = [] +for line in open("/proc/%d/maps" % pid): + if "nvidiactl" in line and "rw-s" in line: + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if hi - lo == 0x4000: + cands.append(lo + 0x80) +print("CANDIDATES %s" % ["0x%x" % c for c in cands]) +class W(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*(unsigned int*)0x%x" % addr, gdb.BP_WATCHPOINT, gdb.WP_READ, internal=False) + self.addr = addr + self.silent = True + def stop(self): + print("READWORD 0x%x" % self.addr) + return True +armed = 0 +for c in cands: + try: + W(c); armed += 1 + except Exception as e: + print("watchfail 0x%x: %s" % (c, e)) +print("ARMED %d" % armed) +if armed: + gdb.execute("continue") +print("T80R-DONE") diff --git a/tools/guest-diag/wall2/t83.py b/tools/guest-diag/wall2/t83.py new file mode 100644 index 0000000..5e3cce2 --- /dev/null +++ b/tools/guest-diag/wall2/t83.py @@ -0,0 +1,25 @@ +import gdb +from collections import Counter +gr = gdb.execute +pcs = Counter() +inf = gdb.selected_inferior() +for i in range(300): + try: + gr("stepi", to_string=True) + except Exception as e: + print("stepfail %s" % e); break + try: + pcs[int(gdb.parse_and_eval("$pc"))] += 1 + except Exception: + break +print("HOTPCS") +top = pcs.most_common(4) +for pc, n in top: + print("PC 0x%x hits=%d" % (pc, n)) +for pc, n in top[:1]: + try: + print(gr("x/10i 0x%x" % pc, to_string=True)) + print(gr("info registers rax rbx rcx rdx rsi rdi rbp r8 r9 r10 r11 r12 r13 r14 r15", to_string=True)) + except Exception as e: + print("fail %s" % e) +print("T83-DONE") diff --git a/tools/guest-diag/wall2/t89.py b/tools/guest-diag/wall2/t89.py new file mode 100644 index 0000000..92c1de3 --- /dev/null +++ b/tools/guest-diag/wall2/t89.py @@ -0,0 +1,17 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +print(gr("x/70i 0x%x" % (base + 0xB28519), to_string=True)) +print(gr("x/40i 0x%x" % (base + 0xB28560), to_string=True)) +print("T89-DONE") diff --git a/tools/guest-diag/wall2/t90.py b/tools/guest-diag/wall2/t90.py new file mode 100644 index 0000000..8518235 --- /dev/null +++ b/tools/guest-diag/wall2/t90.py @@ -0,0 +1,57 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1) +COUNT = base + 0x2070b7c +print("EGLBASE 0x%x PID %d COUNT_ADDR 0x%x" % (base, pid, COUNT)) + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') + +state = {"n": 0} + +class B(gdb.Breakpoint): + def __init__(self, spec, limit=100000): + super().__init__(spec, internal=False) + self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + v = u32(COUNT) + if v: + state["n"] += 1 + if state["n"] <= 8: + print("CLEAR count %u -> 0 (hit %d)" % (v, self.hits)) + gdb.execute("set {unsigned int}0x%x = 0" % COUNT) + except Exception as e: + print("ERR", e) + return self.hits >= 400 + +B("*0x%x" % (base+0xa85d24), 400) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +except KeyboardInterrupt: + pass +print("TRACE-END clears=%d" % state["n"]) +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t91.py b/tools/guest-diag/wall2/t91.py new file mode 100644 index 0000000..5c84ba9 --- /dev/null +++ b/tools/guest-diag/wall2/t91.py @@ -0,0 +1,95 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1) +COUNT = base + 0x2070b7c +print("EGLBASE 0x%x PID %d COUNT 0x%x" % (base, pid, COUNT)) + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') +def u64(a): + return int.from_bytes(rd(a, 8), 'little') + +state = {"clears": 0, "mp": 0, "am": 0} + +class B(gdb.Breakpoint): + def __init__(self, spec, fn, limit=100000): + super().__init__(spec, internal=False) + self.fn = fn; self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + stop = bool(self.fn(self)) + except Exception as e: + print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e)) + stop = False + return stop or self.hits >= self.limit + +def clear(bp): + try: + v = u32(COUNT) + if v: + state["clears"] += 1 + if state["clears"] <= 5: + print("CLEAR count %u -> 0" % v) + gdb.execute("set {unsigned int}0x%x = 0" % COUNT) + except Exception as e: + print("CLEAR err", e) + return False + +def mp(bp): + rsi = int(gdb.parse_and_eval("$rsi")) + state["mp"] += 1 + if state["mp"] <= 3: + print("MEMPROPS ptr=0x%x" % rsi) + return False + +def am(bp): + rsi = int(gdb.parse_and_eval("$rsi")) + state["am"] += 1 + if state["am"] <= 10: + try: + st = u32(rsi) + size = u64(rsi+0x10) + tidx = u32(rsi+0x18) + print("ALLOC sType=%d size=0x%x typeIndex=%d" % (st, size, tidx)) + except Exception as e: + print("ALLOC err", e) + return False + +B("*0x%x" % (base+0xa85d24), clear, 100000) +try: + B("vkGetPhysicalDeviceMemoryProperties", mp, 5) +except Exception as e: + print("MP BP err", e) +try: + B("vkAllocateMemory", am, 20) +except Exception as e: + print("AM BP err", e) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +except KeyboardInterrupt: + pass +print("TRACE-END clears=%d" % state["clears"]) +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t93.py b/tools/guest-diag/wall2/t93.py new file mode 100644 index 0000000..6a0c23d --- /dev/null +++ b/tools/guest-diag/wall2/t93.py @@ -0,0 +1,57 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +class B(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True; self.done = False + def stop(self): + if self.done: + return True + self.done = True + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + print("RETRYFN obj=0x%x n=0x%x" % (obj, n)) + cnt = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % (obj + 0x22efc))) + print("CNT %d" % cnt) + wrote = 0 + for i in range(0, min(cnt, 32)): + p = int(gdb.parse_and_eval("*(unsigned long*)0x%x" % (obj + 0x1428 + i*8))) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + v0 = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % tgt)) + print("E%d ptr=0x%x tgt=0x%x w0=0x%x w16e=0x%x" % (i, p, tgt, v0, v16)) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + wrote += 1 + except Exception as e: + print("E%d fail %s" % (i, e)) + print("CLEARED %d" % wrote) + gr("detach") + except Exception as e: + print("fail %s" % e) + return True +try: + B() + print("BP-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T92-DONE") diff --git a/tools/guest-diag/wall2/t94.py b/tools/guest-diag/wall2/t94.py new file mode 100644 index 0000000..0d2aa27 --- /dev/null +++ b/tools/guest-diag/wall2/t94.py @@ -0,0 +1,54 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +state = {"hits":0, "cleared":0} +class B(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["hits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % (obj + 0x22efc))) + if state["hits"] <= 6: + print("HIT %d obj=0x%x n=0x%x cnt=%d" % (state["hits"], obj, n, cnt)) + for i in range(0, min(cnt, 32)): + p = int(gdb.parse_and_eval("*(unsigned long*)0x%x" % (obj + 0x1428 + i*8))) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + if state["cleared"] <= 12: + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt)) + except Exception as e: + if state["hits"] <= 6: + print("E%d fail %s" % (i, e)) + except Exception as e: + if state["hits"] <= 6: + print("hitfail %s" % e) + return state["hits"] >= 4000 +try: + B(); print("BP-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T94-DONE hits=%d cleared=%d" % (state["hits"], state["cleared"])) diff --git a/tools/guest-diag/wall2/t95.py b/tools/guest-diag/wall2/t95.py new file mode 100644 index 0000000..8daf6ca --- /dev/null +++ b/tools/guest-diag/wall2/t95.py @@ -0,0 +1,19 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +print("--- wait site +0xA12D90..:") +print(gr("x/60i 0x%x" % (base + 0xA12D90), to_string=True)) +print("--- caller +0xD690D0..:") +print(gr("x/40i 0x%x" % (base + 0xD690D0), to_string=True)) +print("T95-DONE") diff --git a/tools/guest-diag/wall2/t96b.py b/tools/guest-diag/wall2/t96b.py new file mode 100644 index 0000000..cd3b7ab --- /dev/null +++ b/tools/guest-diag/wall2/t96b.py @@ -0,0 +1,40 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +class B(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xA12DDF), internal=False) + self.silent = True; self.done = False + def stop(self): + if self.done: + return True + self.done = True + try: + rdi = int(gdb.parse_and_eval("$rdi")) + rsi = int(gdb.parse_and_eval("$rsi")) + rdx = int(gdb.parse_and_eval("$rdx")) + print("POLL nfds=%d timeout=%d arr=0x%x" % (rsi, rdx, rdi)) + fd = int(gdb.parse_and_eval("*(int*)0x%x" % rdi)) + ev = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (rdi + 4))) + rv = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (rdi + 6))) + print("FDPOLL fd=%d events=0x%x revents=0x%x" % (fd, ev, rv)) + except Exception as e: + print("fail %s" % e) + return True +try: + B(); print("BP-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T96-DONE") diff --git a/tools/guest-diag/wall2/t97.py b/tools/guest-diag/wall2/t97.py new file mode 100644 index 0000000..5a12348 --- /dev/null +++ b/tools/guest-diag/wall2/t97.py @@ -0,0 +1,22 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +C = base + 0x2070b7c +v = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % C)) +print("COUNT %u" % v) +print(gr("x/8wx 0x%x" % (C - 0x10), to_string=True)) +if v: + gr("set {unsigned int}0x%x = 0" % C) + print("WROTE 0") +print("T97-DONE") diff --git a/tools/guest-diag/wall2/t99.py b/tools/guest-diag/wall2/t99.py new file mode 100644 index 0000000..ed665fb --- /dev/null +++ b/tools/guest-diag/wall2/t99.py @@ -0,0 +1,79 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "whits":0, "wrote":0} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + if state["cleared"] <= 10: + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt)) + except Exception: + pass + except Exception as e: + if state["chits"] <= 4: + print("chitfail %s" % e) + return state["chits"] >= 6000 +class BW(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xDCEBD9), internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val)) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + if state["wrote"] <= 10: + print("WORDWROTE 0x%x" % word) + except Exception as e: + if state["whits"] <= 4: + print("whitfail %s" % e) + return False +try: + BC(); BW(); print("BPS-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T99-DONE chits=%d cleared=%d whits=%d wrote=%d" % (state["chits"], state["cleared"], state["whits"], state["wrote"])) diff --git a/tools/guest-diag/wall2/trace_push674.py b/tools/guest-diag/wall2/trace_push674.py new file mode 100644 index 0000000..ced4168 --- /dev/null +++ b/tools/guest-diag/wall2/trace_push674.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +# r674: find every reference to the push VA 0x4480000 and follow RM structures +# (direct-map pointers: PA = ptr - page_offset_base) toward a page list. +import numpy as np, struct, os, time, sys + +RAM="/dev/shm/vgpu-guest-ram" +VA=0x4480000 +BASE=0xffff8916c0000000 # r674 page_offset_base (from dmesg rebuild #15) +img=open(RAM,"rb"); fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa<0x80000000: return pa + if pa<0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off<0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) +def ptr2pa(v): + if BASE<=v<BASE+0x180000000: + return v-BASE + return None + +CH=64<<20 +hits=[] +t0=time.time() +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(VA))[0] + for i in idx: hits.append(off2gpa(pos+int(i)*8)) +print("VA hits:", len(hits), "(%.1fs)" % (time.time()-t0)) + +def tag(v): + bs=struct.pack("<Q",v) + return bs.decode() if all(32<=c<127 for c in bs) else "" + +for gpa in hits: + page=gpa&~0xfff + d=rd(page,0x1000) + if not d: continue + ro=gpa-page + print("== hit gpa=0x%x page=0x%x off=0x%x ==" % (gpa,page,ro)) + # print 0x60 before, 0x80 after + s=max(0,ro-0x60); e=min(0x1000,ro+0x80) + for o in range(s,e,0x10): + a,b=struct.unpack_from("<QQ",d,o) + print(" +0x%03x: %016x %016x %s %s" % (o,a,b,tag(a),tag(b))) + # follow direct-map pointers in this page + for o in range(0,0x1000-8,8): + v=struct.unpack_from("<Q",d,o)[0] + pa=ptr2pa(v) + if pa is None: continue + dd=rd(pa&~0xfff,0x1000) + if not dd: continue + # search that page for PA runs (>=8 page-aligned RAM PAs ascending/descending) + runs=[]; cur=[] + for oo in range(0,0xff8,8): + vv=struct.unpack_from("<Q",dd,oo)[0] + if 0x100000000<=vv<0x180000000 and (vv&0xfff)==0: + cur.append((oo,vv)) + else: + if len(cur)>=8: runs.append(cur) + cur=[] + if len(cur)>=8: runs.append(cur) + if runs: + print(" ptr +0x%03x -> pa=0x%x has %d PA-run(s):" % (o,pa&~0xfff,len(runs))) + for r in runs[:4]: + print(" run at +0x%x: %d pages first=0x%x last=0x%x" % (r[0][0],len(r),r[0][1],r[-1][1])) diff --git a/tools/guest-diag/wall2/unwait.py b/tools/guest-diag/wall2/unwait.py new file mode 100644 index 0000000..94e677c --- /dev/null +++ b/tools/guest-diag/wall2/unwait.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +# unwait.py - guest-side, zero-ptrace wait-word poller: writes 1 to the +0x80 +# u32 of every zero 16 KB rw-s nvidiactl mapping of the newest vk_frame. +import os, re, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def newest_pid(): + try: + out = subprocess.run(["pgrep", "-n", "-x", "vk_frame"], capture_output=True).stdout + return int(out.decode().strip()) + except Exception: + return None +def maps(pid): + try: + return open("/proc/%d/maps" % pid).read() + except Exception: + return "" +logged = set() +end = time.time() + DUR +while time.time() < end: + pid = newest_pid() + if pid: + for line in maps(pid).splitlines(): + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1] or "rw-s" not in p[1]: + continue + try: + lo, hi = [int(x, 16) for x in p[0].split("-")] + except Exception: + continue + if hi - lo != 0x4000: + continue + wa = lo + 0x80 + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + os.lseek(fd, wa, 0) + v = os.read(fd, 4) + if len(v) == 4 and v == b"\x00\x00\x00\x00": + os.lseek(fd, wa, 0) + os.write(fd, b"\x01\x00\x00\x00") + key = (pid, wa) + if key not in logged: + logged.add(key) + print("WORDWROTE pid=%d va=0x%x" % (pid, wa), flush=True) + os.close(fd) + except Exception: + pass + time.sleep(0.4) +print("UNWAIT-DONE", flush=True) diff --git a/tools/guest-diag/wall2/unwait2.py b/tools/guest-diag/wall2/unwait2.py new file mode 100644 index 0000000..cd4169b --- /dev/null +++ b/tools/guest-diag/wall2/unwait2.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +# unwait2.py - no-ptrace guest-side harness: +# (a) writes 1 to any zero u32 at +0x80 of every 0x4000-byte rw-s nvidia +# mapping of EVERY live vk_frame; +# (b) scans each live vk_frame's anonymous/heap rw-p mappings (16-byte +# aligned) for marker slots {u32@0==0, u16@+0xe==0xffff} and clears +# u16@+0xe to 0 (the instance-phase pending markers the t105 gdb used +# to clear). No ptrace anywhere. +import os, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def pids(): + try: + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + except Exception: + return [] +def rd(fd, off, n): + try: + os.lseek(fd, off, 0) + return os.read(fd, n) + except Exception: + return b"" +def wr(fd, off, data): + try: + os.lseek(fd, off, 0) + os.write(fd, data) + return True + except Exception: + return False +wlog = set(); mlog = set() +end = time.time() + DUR +last_scan = 0.0 +while time.time() < end: + for pid in pids(): + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + except Exception: + continue + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1] or "rw-s" not in p[1]: + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo != 0x4000: + continue + wa = lo + 0x80 + v = rd(fd, wa, 4) + if len(v) == 4 and v == b"\x00\x00\x00\x00": + if wr(fd, wa, b"\x01\x00\x00\x00"): + k = (pid, wa) + if k not in wlog: + wlog.add(k) + print("WORDWROTE pid=%d va=0x%x" % (pid, wa), flush=True) + except Exception: + pass + if time.time() - last_scan > 1.5: + budget = 64 + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or p[1] != "rw-p": + continue + path = p[-1] + if path.startswith("/") and "nvidia" not in path and path != "[heap]": + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo > 0x10000000: + hi = lo + 0x10000000 + off = lo + while off < hi: + chunk = rd(fd, off, 0x100000) + if len(chunk) < 32: + break + j = 0 + while j + 16 <= len(chunk): + if chunk[j:j+4] == b"\x00\x00\x00\x00" and chunk[j+0xe:j+0x10] == b"\xff\xff": + wa = off + j + 0xe + if (pid, wa) not in mlog: + if wr(fd, wa, b"\x00\x00"): + mlog.add((pid, wa)) + print("MARKCLR pid=%d va=0x%x" % (pid, wa), flush=True) + j += 16 + off += 0x100000 + budget -= 1 + if budget <= 0: + break + if budget <= 0: + break + except Exception: + pass + last_scan = time.time() + os.close(fd) + time.sleep(0.4) +print("UNWAIT2-DONE", flush=True) diff --git a/tools/guest-diag/wall2/vk_alloc_test.py b/tools/guest-diag/wall2/vk_alloc_test.py new file mode 100644 index 0000000..0bcf35a --- /dev/null +++ b/tools/guest-diag/wall2/vk_alloc_test.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +# vk_alloc_test.py - reproduce vk_frame's DEVICE_LOCAL image allocation: +# instance -> device -> OPTIMAL R8G8B8A8 storage image -> mem reqs -> +# pick first DEVICE_LOCAL type -> vkAllocateMemory -> report rc/type/size. +import ctypes, sys + +vk = ctypes.CDLL("libvulkan.so.1") + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32)] + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("pApplicationInfo", ctypes.c_void_p), + ("enabledLayerCount", ctypes.c_uint32), ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), ("ppEnabledExtensionNames", ctypes.c_void_p)] + +class VkMemoryType(ctypes.Structure): + _fields_ = [("propertyFlags", ctypes.c_uint32), ("heapIndex", ctypes.c_uint32)] +class VkMemoryHeap(ctypes.Structure): + _fields_ = [("size", ctypes.c_uint64), ("flags", ctypes.c_uint32)] +class VkPhysicalDeviceMemoryProperties(ctypes.Structure): + _fields_ = [("memoryTypeCount", ctypes.c_uint32), + ("memoryTypes", VkMemoryType * 32), + ("memoryHeapCount", ctypes.c_uint32), + ("memoryHeaps", VkMemoryHeap * 16)] + +class VkDeviceQueueCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("queueFamilyIndex", ctypes.c_uint32), + ("queueCount", ctypes.c_uint32), ("pQueuePriorities", ctypes.c_void_p)] +class VkDeviceCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("queueCreateInfoCount", ctypes.c_uint32), + ("pQueueCreateInfos", ctypes.c_void_p), + ("enabledLayerCount", ctypes.c_uint32), ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), ("ppEnabledExtensionNames", ctypes.c_void_p), + ("pEnabledFeatures", ctypes.c_void_p)] + +class VkExtent3D(ctypes.Structure): + _fields_ = [("width", ctypes.c_uint32), ("height", ctypes.c_uint32), ("depth", ctypes.c_uint32)] +class VkImageCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), ("flags", ctypes.c_uint32), + ("imageType", ctypes.c_uint32), ("format", ctypes.c_uint32), + ("extent", VkExtent3D), ("mipLevels", ctypes.c_uint32), ("arrayLayers", ctypes.c_uint32), + ("samples", ctypes.c_uint32), ("tiling", ctypes.c_uint32), ("usage", ctypes.c_uint32), + ("sharingMode", ctypes.c_uint32), ("queueFamilyIndexCount", ctypes.c_uint32), + ("pQueueFamilyIndices", ctypes.c_void_p), ("initialLayout", ctypes.c_uint32)] +class VkMemoryRequirements(ctypes.Structure): + _fields_ = [("size", ctypes.c_uint64), ("alignment", ctypes.c_uint64), ("memoryTypeBits", ctypes.c_uint32)] +class VkMemoryAllocateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("allocationSize", ctypes.c_uint64), ("memoryTypeIndex", ctypes.c_uint32)] + +app = VkApplicationInfo(0, None, b"alloctest", 1, b"alloctest", 1, 0x401000) +ci = VkInstanceCreateInfo(1, None, 0, ctypes.cast(ctypes.pointer(app), ctypes.c_void_p), 0, None, 0, None) +inst = ctypes.c_void_p() +print("vkCreateInstance rc=%d" % vk.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst))) + +n = ctypes.c_uint32(0) +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +devs = (ctypes.c_void_p * n.value)() +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) +phys = devs[0] +print("phys=0x%x" % (phys or 0)) + +prio = ctypes.c_float(1.0) +qci = VkDeviceQueueCreateInfo(2, None, 0, 0, 1, ctypes.cast(ctypes.pointer(prio), ctypes.c_void_p)) +dci = VkDeviceCreateInfo(3, None, 0, 1, ctypes.cast(ctypes.pointer(qci), ctypes.c_void_p), 0, None, 0, None, None) +dev = ctypes.c_void_p() +rc = vk.vkCreateDevice(phys, ctypes.byref(dci), None, ctypes.byref(dev)) +print("vkCreateDevice rc=%d dev=0x%x" % (rc, dev or 0)) +if rc != 0: + sys.exit(1) + +mp = VkPhysicalDeviceMemoryProperties() +vk.vkGetPhysicalDeviceMemoryProperties(phys, ctypes.byref(mp)) + +img_ci = VkImageCreateInfo(14, None, 0, 1, 37, # VK_IMAGE_TYPE_2D, R8G8B8A8_UNORM + VkExtent3D(64, 64, 1), 1, 1, 1, 0, # OPTIMAL + 0x2 | 0x1, 0, 0, None, 0) # STORAGE | TRANSFER_SRC, UNDEFINED +img = ctypes.c_void_p() +rc = vk.vkCreateImage(dev, ctypes.byref(img_ci), None, ctypes.byref(img)) +print("vkCreateImage rc=%d img=0x%x" % (rc, img or 0)) +if rc != 0: + sys.exit(1) + +mr = VkMemoryRequirements() +vk.vkGetImageMemoryRequirements(dev, img, ctypes.byref(mr)) +print("memreqs size=0x%x align=0x%x typeBits=0x%x" % (mr.size, mr.alignment, mr.memoryTypeBits)) + +tidx = 0 +found = False +for k in range(mp.memoryTypeCount): + if (mr.memoryTypeBits & (1 << k)) and (mp.memoryTypes[k].propertyFlags & 0x1): + tidx = k + found = True + break +print("selected type=%d found_device_local=%s (types: %s)" % + (tidx, found, ", ".join("%d:0x%x" % (k, mp.memoryTypes[k].propertyFlags) for k in range(mp.memoryTypeCount)))) + +ma = VkMemoryAllocateInfo(5, None, mr.size, tidx) +mem = ctypes.c_void_p() +rc = vk.vkAllocateMemory(dev, ctypes.byref(ma), None, ctypes.byref(mem)) +print("vkAllocateMemory size=0x%x type=%d rc=%d" % (mr.size, tidx, rc)) diff --git a/tools/guest-diag/wall2/vk_memprops.py b/tools/guest-diag/wall2/vk_memprops.py new file mode 100644 index 0000000..3cf6ba2 --- /dev/null +++ b/tools/guest-diag/wall2/vk_memprops.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +# vk_memprops.py - enumerate Vulkan devices and dump their memory properties +# via ctypes (no vkCreateDevice needed; safe alongside other clients). +import ctypes, ctypes.util, sys + +vk = ctypes.CDLL("libvulkan.so.1") + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32)] + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("pApplicationInfo", ctypes.c_void_p), + ("enabledLayerCount", ctypes.c_uint32), ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), ("ppEnabledExtensionNames", ctypes.c_void_p)] + +class VkMemoryType(ctypes.Structure): + _fields_ = [("propertyFlags", ctypes.c_uint32), ("heapIndex", ctypes.c_uint32)] + +class VkMemoryHeap(ctypes.Structure): + _fields_ = [("size", ctypes.c_uint64), ("flags", ctypes.c_uint32)] + +class VkPhysicalDeviceMemoryProperties(ctypes.Structure): + _fields_ = [("memoryTypeCount", ctypes.c_uint32), + ("memoryTypes", VkMemoryType * 32), + ("memoryHeapCount", ctypes.c_uint32), + ("memoryHeaps", VkMemoryHeap * 16)] + +app = VkApplicationInfo(0, None, b"memprops", 1, b"memprops", 1, 0x401000) +ci = VkInstanceCreateInfo(1, None, 0, ctypes.cast(ctypes.pointer(app), ctypes.c_void_p), 0, None, 0, None) +inst = ctypes.c_void_p() +rc = vk.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst)) +print("vkCreateInstance rc=%d" % rc) +if rc != 0: + sys.exit(1) + +n = ctypes.c_uint32(0) +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +print("devices=%d" % n.value) +devs = (ctypes.c_void_p * n.value)() +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) + +for i in range(n.value): + props = VkPhysicalDeviceMemoryProperties() + vk.vkGetPhysicalDeviceMemoryProperties(devs[i], ctypes.byref(props)) + print("=== DEV %d phys=0x%x ===" % (i, devs[i] or 0)) + print(" memoryTypeCount=%d memoryHeapCount=%d" % (props.memoryTypeCount, props.memoryHeapCount)) + for t in range(props.memoryTypeCount): + mt = props.memoryTypes[t] + print(" type[%d] flags=0x%08x heap=%d" % (t, mt.propertyFlags, mt.heapIndex)) + for h in range(props.memoryHeapCount): + mh = props.memoryHeaps[h] + print(" heap[%d] size=0x%x (%u MiB) flags=0x%x" % (h, mh.size, mh.size >> 20, mh.flags)) diff --git a/tools/guest-diag/wall2/vmap_watch2.py b/tools/guest-diag/wall2/vmap_watch2.py new file mode 100644 index 0000000..194b0d6 --- /dev/null +++ b/tools/guest-diag/wall2/vmap_watch2.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +# Guest-side live watcher v2: scan ALL vmallocinfo areas via /proc/kcore for +# (a) GR ring entry patterns, (b) pages starting with a small INC_METHOD header. +import struct, time +ENTRIES = [bytes.fromhex("0000480400980500"), bytes.fromhex("8000040004480598"), bytes.fromhex("0003d0000448059c")] +class Kcore: + def __init__(self): + self.fd = open("/proc/kcore", "rb") + hdr = self.fd.read(64) + e_phoff = struct.unpack_from("<Q", hdr, 32)[0] + e_phentsize, e_phnum = struct.unpack_from("<HH", hdr, 54) + self.segs = [] + for i in range(e_phnum): + self.fd.seek(e_phoff + i * e_phentsize) + ph = self.fd.read(56) + p_type, _fl, p_off, p_va, _pa, p_sz = struct.unpack_from("<IIQQQQ", ph, 0) + if p_type == 1: + self.segs.append((p_va, p_va + p_sz, p_off)) + def read(self, va, n): + for lo, hi, off in self.segs: + if lo <= va and va + n <= hi: + self.fd.seek(off + (va - lo)) + return self.fd.read(n) + return None +kc = Kcore() +def areas(): + out = [] + for line in open("/proc/vmallocinfo"): + p = line.split() + if len(p) < 2 or "-" not in p[0]: continue + try: + lo, hi = p[0].split("-") + lo, hi = int(lo,16), int(hi,16) + except Exception: + continue + out.append((lo, hi, line.strip()[-60:])) + return out +def hdr_ok(page): + v = struct.unpack_from("<I", page, 0)[0] + op = (v >> 29) & 7 + cnt = (v >> 16) & 0xfff + return op == 1 and 1 <= cnt <= 8 +print("vmap_watch2 start", flush=True) +found = set() +t0 = time.time() +it = 0 +while time.time() - t0 < 300: + it += 1 + nareas = 0 + for lo, hi, tag in areas(): + size = hi - lo + if size > 8 << 20: continue + nareas += 1 + d = kc.read(lo, size) + if not d: continue + for pat in ENTRIES: + i = d.find(pat) + if i >= 0: + key = (lo, i & ~0xfff, pat.hex()) + if key not in found: + found.add(key) + print("%.2f RING %s VA 0x%x (+0x%x) %s" % (time.time()-t0, pat.hex(), lo+(i&~0xfff), i&~0xfff, tag), flush=True) + for off in range(0, size - 0x1000, 0x1000): + page = d[off:off+0x1000] + if hdr_ok(page): + key = (lo, off, "h") + if key not in found: + found.add(key) + print("%.2f HDR VA 0x%x (+0x%x) %s head=%s" % (time.time()-t0, lo+off, off, tag, page[:0x30].hex()), flush=True) + if it % 10 == 0: + print("%.1f iter %d areas %d" % (time.time()-t0, it, nareas), flush=True) + time.sleep(0.2) +print("vmap_watch2 done", flush=True) diff --git a/tools/guest-diag/wall2/walk.py b/tools/guest-diag/wall2/walk.py new file mode 100644 index 0000000..3624007 --- /dev/null +++ b/tools/guest-diag/wall2/walk.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +# walk.py - host-side replica of the service's svc_gmmu_walk + pd_discover, +# run against /dev/shm/vgpu-guest-ram. Discovers the page directory that +# reproduces the service's map[] sample pairs, then walks the ring/window +# VAs through it. +import mmap, struct, sys + +F = "/dev/shm/vgpu-guest-ram" +SEMA_PA_MAX = 0x180000000 + +# samples from the r690 dmesg map[] lines +SAMPLES = [ + (0x100000000, 0x100326000), + (0x100e2f000, 0x10096e000), + (0x100e33000, 0x10096b000), + (0x100e34000, 0x10096a000), + (0x100e35000, 0x100969000), + (0x100e37000, 0x102dcb000), + (0x100e38000, 0x102dca000), + (0x100e39000, 0x102dc9000), +] + +TARGET_VAS = [0x44a0000, 0x4500000, 0x4560000, 0x4480000, 0x4485000, + 0x4ca0000, 0x4ca5000, 0x4d00000, 0x4b17000, 0x4020000, + 0x4100000, 0x4160000] + +mm = None + +def off(pa): + return pa if pa < 0x80000000 else pa - 0x80000000 + +def rd(pa, n=8): + if pa + n > SEMA_PA_MAX: + return None + try: + d = mm[off(pa):off(pa)+n] + return d if len(d) == n else None + except Exception: + return None + +def rd64(pa): + d = rd(pa, 8) + return None if d is None else int.from_bytes(d, "little") + +def pde_aperture(ap): + if ap == 1: return "FB" + if ap in (2, 3): return "SYS" + return None + +def pte_aperture(ap): + if ap == 0: return "FB" + if ap == 1: return None + return "SYS" + +def pte_addr(e, fb): + if fb: + return ((e >> 8) & 0x1FFFFFF) << 12 + return ((e >> 8) & 0x3FFFFFFFFFFF) << 12 + +def pte_leaf(table, idx): + e = rd64(table + 8 * idx) + if e is None or not (e & 1): + return None + ap = pte_aperture((e >> 1) & 0x3) + if ap is None: + return None + a = pte_addr(e, ap == "FB") + if not fb_ok(a): + return None + return a, ap + +def fb_ok(a): + return a + 8 <= SEMA_PA_MAX + +def walk(va): + table = None + # root_bits = ilog2(entries=4) = 2; shift = 49-2 = 47 + e = rd64(table) if False else None + # level 0 + idx = (va >> 47) & 0x3 + # root table is the candidate itself (passed separately) + return None + +def walk_from(pdb, va, trace=False): + table = pdb + for level in range(3): + if level == 0: + bits, shift = 2, 47 + else: + bits, shift = 9, 38 if level == 1 else 29 + idx = (va >> shift) & ((1 << bits) - 1) + e = rd64(table + 8 * idx) + if e is None: + return None + if trace: + print(" L%d table=0x%x idx=%u e=0x%016x" % (level, table, idx, e)) + if e & 1: + return None + ap = pde_aperture((e >> 1) & 0x3) + if ap is None: + return None + table = pte_addr(e, ap == "FB") + if not fb_ok(table): + return None + idx = (va >> 21) & 0xff + big = rd64(table + 16 * idx) + small = rd64(table + 16 * idx + 8) + if big is None or small is None: + return None + if trace: + print(" PD0 table=0x%x idx=%u big=0x%016x small=0x%016x" % + (table, idx, big, small)) + if not (small & 1): + sap = pde_aperture((small >> 1) & 0x3) + if sap is not None: + t = pte_addr(small, sap == "FB") + r = pte_leaf(t, (va >> 12) & 0x1ff) + if r: + return r[0] + (va & 0xfff), r[1] + if big & 1: + bap = pte_aperture((big >> 1) & 0x3) + if bap is not None: + a = pte_addr(big, bap == "FB") + (va & ((1 << 21) - 1)) + if fb_ok(a): + return a, bap + else: + bap = pde_aperture((big >> 1) & 0x3) + if bap is not None: + t = pte_addr(big, bap == "FB") + r = pte_leaf(t, (va >> 16) & 0x1f) + if r: + return r[0] + (va & 0xffff), r[1] + return None + +def main(): + global mm + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + print("# scanning for directory candidates...") + cands = [] + for fo in range(0, size, 4096): + e = int.from_bytes(mm[fo:fo+8], "little") + if e and not (e & 1): + ap = (e >> 1) & 0x3 + if ap in (1, 2, 3): + pa = fo if fo < 0x80000000 else fo + 0x80000000 + if pa < SEMA_PA_MAX: + cands.append(pa) + print("# candidates: %d" % len(cands)) + best = [] + for c in cands: + hits = 0 + for (sva, spa) in SAMPLES: + r = walk_from(c, sva) + if r and (r[0] & ~0xfff) == (spa & ~0xfff): + hits += 1 + if hits >= 2: + best.append((hits, c)) + best.sort(reverse=True) + print("# candidates with >=2 hits: %d" % len(best)) + for hits, c in best[:10]: + print(" pdb=0x%x hits=%d/8" % (c, hits)) + if not best: + return 1 + pdb = best[0][1] + print("== walking targets through pdb=0x%x:" % pdb) + for va in TARGET_VAS: + r = walk_from(pdb, va) + if r: + print(" va=0x%x -> pa=0x%x ap=%s" % (va, r[0], r[1])) + else: + print(" va=0x%x -> no translation" % va) + mm.close(); f.close() + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_livecap.py b/tools/guest-diag/wall2/wall2_livecap.py new file mode 100644 index 0000000..af68c03 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_livecap.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +# wall2_livecap.py - live capture of the GR channel-init push page. +# +# The release group in the push is {0x20038090, sem_va_hi, sem_va_lo, payload, +# continuation}: sem_va = (hi<<32)|lo, and for the channel-init entry the sem +# sits at push_va + entry_len (entry0: len=1432=0x598, sem at +0x598), with the +# group itself at push_va + entry_len - 24 = +0x580. +# +# This watcher scans guest RAM for 0x20038090 groups during the client's stall, +# classifies them, and injects push_va -> page_pa via sema_map_add when the +# group's sem VA and offset identify the push page. Prints full evidence. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_livecap.log" +MASK = ~0xfff & ((1 << 64) - 1) + +ENTRY_LEN = int(os.environ.get("W2_LEN", "1432")) +WATCH_S = int(os.environ.get("W2_WATCH", "120")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def cont_ok(v): + op = (v >> 29) & 7 + cnt = (v >> 16) & 0xfff + return (op in (1, 3, 5) and cnt != 0) or op in (4, 7) + +def scan(): + img.seek(0, 2); fsz = img.tell() + pat = b"\x90\x80\x03\x20" + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + g = d[i:i+24] + hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g) + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, hi, lo, pay, w5, v5, g)) + pos += CH + return out + +def main(): + log("=== livecap start len=%d watch=%ds ===" % (ENTRY_LEN, WATCH_S)) + injected = {} + deadline = time.time() + WATCH_S + while time.time() < deadline: + hits = scan() + cands = [] + for (page, off, hi, lo, pay, w5, v5, g) in hits: + sem = (hi << 32) | lo + # classify + pool = (hi == 1 and lo < 0x10000 and 0 < pay < (1 << 24) + and (w5 >> 29) == 1 and ((w5 >> 13) & 7) == 4 + and ((w5 & 0x1fff) << 2) == 0x300 and (v5 & 3) == 0) + if pool: + continue # CE pool release family, not the push + if not (0x4000000 <= sem < 0x100000000): + continue + if not (0 < pay < (1 << 24)): + continue + if not cont_ok(w5): + continue + push_va = sem - ENTRY_LEN + cands.append((page, off, sem, pay, push_va, w5, v5, g)) + if cands: + # prefer the canonical group offset 0x580 / sem-consistent push VA + cands.sort(key=lambda c: (c[1] != (ENTRY_LEN - 24), -c[3])) + page, off, sem, pay, push_va, w5, v5, g = cands[0] + d = rd(page, 0x100) + log("livecap: %d cand(s); page=0x%x off=0x%x sem=0x%x pay=%u push_va=0x%x w5=0x%08x v5=0x%08x" + % (len(cands), page, off, sem, pay, push_va, w5, v5)) + log(" page head: %s" % d[:0x40].hex()) + if (push_va & 0xfff) == 0 and push_va not in injected: + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,1\n" % (push_va, page, 0x1000)) + log(" INJECT 0x%x -> 0x%x (off=0x%x)" % (push_va, page, off)) + injected[push_va] = page + except OSError as e: + log(" inject FAIL", e) + time.sleep(1) + log("=== livecap done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_livecap2.py b/tools/guest-diag/wall2/wall2_livecap2.py new file mode 100644 index 0000000..a33b25b --- /dev/null +++ b/tools/guest-diag/wall2/wall2_livecap2.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +# wall2_livecap2.py - capture the GR channel-init push page by its release +# group {0x20038090, hi, lo, payload, continuation} at page offset 0x598. +# The push VA comes from the service's own entry[0] decode in dmesg. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_livecap2.log" +MASK = ~0xfff & ((1 << 64) - 1) +WATCH_S = int(os.environ.get("W2_WATCH", "180")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def prefix_ok(d): + off = 0 + while off + 4 <= len(d): + hdr = struct.unpack_from("<I", d, off)[0] + if hdr: break + off += 4 + if off + 4 > len(d): return False + op = (hdr >> 29) & 7 + if op in (4, 7): return True + if op not in (1, 3, 5): return False + return ((hdr >> 16) & 0xfff) != 0 + +def cont_ok(v): + op = (v >> 29) & 7 + cnt = (v >> 16) & 0xfff + return (op in (1, 3, 5) and cnt != 0) or op in (4, 7) + +def scan(): + img.seek(0, 2); fsz = img.tell() + pat = b"\x90\x80\x03\x20" + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + g = d[i:i+24] + hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g) + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, hi, lo, pay, w5, v5)) + pos += CH + return out + +def main(): + log("=== livecap2 start watch=%ds ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + seen_pages = {} + while time.time() < deadline: + hits = scan() + for (page, off, hi, lo, pay, w5, v5) in hits: + if not (0 < pay < (1 << 24)): + continue + if not cont_ok(w5): + continue + d = rd(page, 0x1000) + if not d: continue + pok = prefix_ok(d[:0x100]) + key = (page, off) + if key in seen_pages: + continue + seen_pages[key] = 1 + log("hit page=0x%x off=0x%x sem=0x%x pay=%u w5=0x%08x v5=0x%08x prefix_ok=%s" + % (page, off, (hi << 32) | lo, pay, w5, v5, pok)) + log(" head: %s" % d[:0x30].hex()) + log(" @0x590: %s" % d[0x590:0x5c0].hex()) + if pok and off in (0x598, 0x590, 0x580, 0x588): + va = 0x4480000 + if va not in injected: + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,1\n" % (va, page, 0x1000)) + log(" INJECT 0x%x -> 0x%x" % (va, page)) + injected.add(va) + except OSError as e: + log(" inject FAIL", e) + time.sleep(0.5) + log("=== livecap2 done: %d injections ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_livecap3.py b/tools/guest-diag/wall2/wall2_livecap3.py new file mode 100644 index 0000000..db28bc2 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_livecap3.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +# wall2_livecap3.py - capture the UVM GR channel-init push page via the host +# (C46F) semaphore release group emitted by uvm_hal_turing_host_semaphore_release: +# {0x20050017 (INC_METHOD count=5 addr=0x5c), sem_lo, sem_hi, payload, 0, exec} +# The group sits at push_va + entry_len - 24 = push_va + 0x580 (entry0 len 1432), +# with the header at +0x598 (entry1's 4-byte push). The push VA comes from the +# service's own entry[0] decode in dmesg. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_livecap3.log" +MASK = ~0xfff & ((1 << 64) - 1) +WATCH_S = int(os.environ.get("W2_WATCH", "240")) +PUSH_VA_DEFAULT = 0x4480000 + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def push_va_now(): + # newest entry[0] line for the Vulkan client's channel + va = None + for ln in dmesg().splitlines(): + m = re.search(r"entry\[0\] h=0x([0-9a-f]+) eng=0 raw=\S+ va=0x([0-9a-f]+) len=(\d+)", ln) + if m: + va = int(m.group(2), 16) + return va + +def scan(): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<I", 0x20050017) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + lo, hi, pay, w4, w5 = struct.unpack_from("<5I", d, i + 4) + # data-word sanity: payload_hi==0, RELEASE op, sane payload + if w4 != 0: continue + if (w5 & 0x7) != 1: continue + if not (0 < pay < (1 << 24)): continue + sem = ((hi & 0xffffffff) << 32) | ((lo & 0x3fffffff) << 2) + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, sem, pay, w5)) + pos += CH + return out + +def main(): + log("=== livecap3 start watch=%ds ===" % WATCH_S) + injected = {} + seen = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + hits = scan() + if hits: + for (page, off, sem, pay, w5) in hits: + key = (page, off) + if key in seen: continue + seen.add(key) + d = rd(page, 0x1000) + head = d[:0x20].hex() if d else "?" + log("hit page=0x%x off=0x%x sem=0x%x pay=%u exec=0x%08x head=%s" + % (page, off, sem, pay, w5, head)) + log(" @0x580: %s" % (d[0x580:0x5b0].hex() if d else "?")) + if off in (0x598, 0x580, 0x590): + pv = push_va_now() or PUSH_VA_DEFAULT + if pv not in injected: + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,1\n" % (pv, page, 0x1000)) + log(" INJECT 0x%x -> 0x%x" % (pv, page)) + injected[pv] = page + except OSError as e: + log(" inject FAIL", e) + time.sleep(0.5) + log("=== livecap3 done: %d injections ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_recscan.py b/tools/guest-diag/wall2/wall2_recscan.py new file mode 100644 index 0000000..7bc4e2c --- /dev/null +++ b/tools/guest-diag/wall2/wall2_recscan.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +# wall2_recscan.py - scan the guest RAM for the RM's low-aperture window +# records and their memdesc PAs. +# +# Record layout (verified r685/r686): 0x40-byte stride +# {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c ("tail" as a u32 - +# on disk the BYTES are "liat"); u32 0} +# The tag is NOT the byte string "tail" - searching for b'tail' finds +# nothing; search for the 4-byte pattern 6c 69 61 74 (struct '<I' +# 0x7461696c) at record+0x20 and decode {tag,VA,SIZE} at -0x20/-0x18/-0x10. +# +# Usage: wall2_recscan.py [VA_prefix_or_full_VA] +# Prints every record and the size-matching memdesc {SIZE,0x03008000,PA} +# found in the same page, and dumps the first words of each memdesc page so +# semaphore windows can be identified by the release payload values. +import struct, sys + +RAM = "/dev/shm/vgpu-guest-ram" +TAG = struct.pack("<I", 0x7461696c) + +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) + +def gpa2off(pa): + if pa < 0x80000000: + return pa + if pa < 0x180000000: + return 0x80000000 + (pa - 0x100000000) + return None + +def main(): + want = None + if len(sys.argv) > 1: + want = int(sys.argv[1], 0) + img = open(RAM, "rb", buffering=0) + img.seek(0, 2); fsz = img.tell() + recs = [] + pos = 0; CH = 64 << 20 + while pos < fsz: + img.seek(pos); buf = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = buf.find(TAG, st) + if i < 0: + break + st = i + 1 + if i >= 0x20: + tag, va, sz, z = struct.unpack_from("<QQQQ", buf, i - 0x20) + if 0x4000000 <= va < 0x100000000 and \ + 0x1000 <= sz <= 0x1000000 and (sz & 0xfff) == 0 and z == 0: + if want is None or va == want or (va & ~0xfffff) == (want & ~0xfffff): + recs.append((off2gpa(pos + i - 0x20), tag, va, sz)) + pos += CH + for gpa, tag, va, sz in sorted(recs, key=lambda r: r[2]): + page = gpa & ~0xfff + img.seek(gpa2off(page)); d = img.read(0x1000) + descs = [] + for o in range(0, 0x1000 - 24, 8): + s2, f2 = struct.unpack_from("<QQ", d, o) + pa = struct.unpack_from("<Q", d, o + 16)[0] + if f2 == 0x03008000 and s2 == sz and 0x100000000 <= pa < 0x180000000: + descs.append(pa) + print("rec gpa=0x%x tag=0x%x va=0x%x size=0x%x" % (gpa, tag, va, sz)) + for pa in descs: + img.seek(gpa2off(pa)); pd = img.read(0x20) + print(" desc pa=0x%x head=%s" % (pa, pd.hex(" "))) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver2.py b/tools/guest-diag/wall2/wall2_resolver2.py new file mode 100644 index 0000000..785e5e2 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver2.py @@ -0,0 +1,274 @@ +#!/usr/bin/env python3 +# wall2_resolver2.py - HOST-SIDE wall#2 resolver v2 (r674+). +# +# Why: the vk_frame client's channel-init push (va=0x4500000 len=1432) is +# kernel-built and kernel-only; the sema service cannot translate it +# ("false translation for push ... rescuing"), the client times out (~11 s) +# per attempt, retries, then VKFAIL -4. Guest RAM is host-readable, so the +# push page can be found and injected via sema_map_add. +# +# v2 recipe (per the r672/r673 structure analysis): +# 1. wait for the channel ring line in dmesg (validated RING_PA + handle) +# 2. scan guest RAM for the push page: +# a) CONTENT: the 24-byte release group the service's own rescue matches +# (hdr=0x20038090, hi=1, payload sane, w5/v5 shape). The channel-init +# entry (va=0x4500000 len=1432) puts that group at page offset 0x580. +# b) TUPLES: pages of 32-byte records {vmap_ptr; flags; PA; size} (both +# observed layouts) matched on (PA & ~0xfff) == RING_PA with strides +# 0x20/0x40/0x60; entries below the ring entry in VA order map +# 0x4500000.. (push VA = ring VA 0x4520000 - 0x20000 = 32 pages). +# 3. inject (va -> pa, len) through /sys/module/vgsp/parameters/sema_map_add +# 4. keep scanning/re-injecting for the watch window (retries allocate new +# pages) and log everything. +# +# Run this FIRST (foreground), then launch the client in the guest. +import subprocess, struct, time, sys, re, os, signal + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver2.log" +RING_VA = 0x4520000 +PUSH_VA = 0x4500000 +DELTA = (RING_VA - PUSH_VA) // 0x1000 # 32 pages +REL_OFF = 0x580 # group offset in the push page +MASK = ~0xfff & ((1 << 64) - 1) + +WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420")) +WATCH_S = int(os.environ.get("W2_WATCH", "150")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def inject(va, pa, ln=0x1000, ap=1): + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,%d\n" % (va, pa, ln, ap)) + log("INJECT va=0x%x -> pa=0x%x len=0x%x" % (va, pa, ln)) + return True + except OSError as e: + log("inject FAIL", e) + return False + +def prefix_ok(d): + off = 0 + while off + 4 <= len(d): + hdr = struct.unpack_from("<I", d, off)[0] + if hdr: + break + off += 4 + if off + 4 > len(d): + return False + op = (hdr >> 29) & 7 + if op in (4, 7): + return True + if op not in (1, 3, 5): + return False + return ((hdr >> 16) & 0xfff) != 0 + +def rel_shape(g): + hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g) + return (hdr == 0x20038090 and hi == 1 and 0 < pay < (1 << 24) + and (w5 >> 29) == 1 and ((w5 >> 13) & 7) == 4 + and ((w5 & 0x1fff) << 2) == 0x300 and (v5 & 3) == 0), lo & 0xfff, pay + +img = None +def open_ram(): + global img + if img is None: + img = open(RAM, "rb", buffering=0) + return img + +def gpa2off(pa): + if pa < 0x80000000: + return pa + if pa < 0x180000000: + return 0x80000000 + (pa - 0x100000000) + return None + +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) + +def rd(pa, n): + off = gpa2off(pa) + if off is None: + return None + f = open_ram() + try: + f.seek(off) + return f.read(n) + except OSError: + return None + +def find_content_candidates(): + """Pages containing the release group; returns [(page_pa, off, slot, pay)].""" + f = open_ram() + f.seek(0, 2) + fsz = f.tell() + pat = b"\x90\x80\x03\x20" + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + f.seek(pos) + d = f.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: + break + st = i + 1 + if i + 24 > len(d): + continue + ok, slot, pay = rel_shape(d[i:i + 24]) + if not ok: + continue + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, slot, pay)) + pos += CH + return out + +def find_tuple_arrays(ring_pa): + """Find record pages containing the ring PA; returns list of + (page_gpa, stride, ring_index, entries[list of pa]).""" + f = open_ram() + f.seek(0, 2) + fsz = f.tell() + pat = struct.pack("<Q", ring_pa) + arrays = [] + CH = 64 << 20 + pos = 0 + hits = [] + while pos < fsz: + f.seek(pos) + d = f.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: + break + hits.append(off2gpa(pos + i)) + st = i + 1 + pos += CH + seen = set() + for gpa in hits: + page = gpa & MASK + if page in seen: + continue + seen.add(page) + d = rd(page, 0x1000) + if not d: + continue + rec_off = gpa - page + for stride in (0x20, 0x40, 0x60): + for rel in range(0, min(stride, rec_off + 1), 8): + start = rec_off - rel + ents = [] + for o in range(start, 0x1000 - stride + 1, stride): + # PA field anywhere in the record: try +0x10 first, then any qword + vals = struct.unpack_from("<%dQ" % (stride // 8), d, o) + pa = None + for cand_off in (0x10, 0x8, 0x18, 0x0, 0x20, 0x28): + if cand_off + 8 <= stride: + v = vals[cand_off // 8] + if (v & MASK) == ring_pa: + pa = v & MASK + break + if pa is None: + continue + ents.append((o, pa)) + if len(ents) >= 4: + idx = [e[1] for e in ents].index(ring_pa) + arrays.append((page, stride, start, idx, [e[1] for e in ents])) + break + if arrays and arrays[-1][0] == page: + break + return arrays + +def main(): + log("=== wall2_resolver2 start (wait_ring=%ss watch=%ss) ===" % (WAIT_RING_S, WATCH_S)) + ring_pa = None + ring_h = None + deadline = time.time() + WAIT_RING_S + while time.time() < deadline: + for ln in dmesg().splitlines(): + m = re.search(r"h=(0x[0-9a-f]+) .*ring rc=0 pa=0x([0-9a-f]+)", ln) + if m: + ring_h = m.group(1) + ring_pa = int(m.group(2), 16) + break + if ring_pa: + break + time.sleep(1) + if not ring_pa: + log("no ring line seen in %ss; exiting" % WAIT_RING_S) + return 1 + log("RING %s PA=0x%x" % (ring_h, ring_pa)) + + injected = set() + watch_deadline = time.time() + WATCH_S + last_pass = 0 + while time.time() < watch_deadline: + # ---- content pass + try: + cands = find_content_candidates() + except Exception as e: + log("content scan error:", e) + cands = [] + # prefer the canonical group offset 0x580, then highest payload + cands.sort(key=lambda c: (c[1] != REL_OFF, -c[3])) + if cands: + page, off, slot, pay = cands[0] + d = rd(page, 0x1000) + pok = bool(d) and prefix_ok(d[:0x100]) + log("content: %d candidate(s); best page=0x%x off=0x%x slot=0x%x pay=%u prefix_ok=%s" + % (len(cands), page, off, slot, pay, pok)) + key = ("C", PUSH_VA, page) + if key not in injected and pok: + if inject(PUSH_VA, page, 0x1000): + injected.add(key) + # also cover the following page (entry2 tail) if non-contiguous unknown; skip + else: + # ---- tuple fallback (only every ~10 s to save time) + if time.time() - last_pass > 10: + last_pass = time.time() + try: + arrs = find_tuple_arrays(ring_pa) + except Exception as e: + log("tuple scan error:", e) + arrs = [] + if arrs: + page, stride, start, idx, ents = arrs[0] + log("tuple: array page=0x%x stride=0x%x ring_index=%d entries=%d" + % (page, stride, idx, len(ents))) + for k in range(idx - DELTA, idx): + if k < 0 or k >= len(ents): + continue + va = PUSH_VA + (k - (idx - DELTA)) * 0x1000 + pa = ents[k] + key = ("T", va, pa) + if key in injected: + continue + d = rd(pa, 0x40) + if d and any(d): + inject(va, pa, 0x1000) + injected.add(key) + else: + log("tuple: no array with ring PA found") + # stop early when the service confirms our mapping + if len(injected) and "injected mapping va=0x%x" % PUSH_VA in dmesg(): + log("service confirmed the injected mapping; done") + break + time.sleep(1) + log("=== done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver3.py b/tools/guest-diag/wall2/wall2_resolver3.py new file mode 100644 index 0000000..4b16686 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver3.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +# wall2_resolver3.py - HOST-SIDE wall#2 resolver v3: the RM window-memdesc recipe. +# +# Proven structure (r674/r675): the RM keeps per-window records +# {..., VA, SIZE, ...} (head/tail list nodes) +# and per-window memdescs in the same page: +# {SIZE, 0x03008000, FIRST_PA, ptr} +# where FIRST_PA is the PA of the window's FIRST VA page. Verified: the ring +# window {VA=0x44a0000, SIZE=0x40000} -> memdesc PA == the service's validated +# ring pa (0x1158ce000 r675). Therefore the push window +# {VA=0x4480000, SIZE=0x20000} -> memdesc PA == PA of VA 0x4480000 (the push +# page the sema service cannot translate). +# +# Recipe: wait for the ring line -> find the ring window memdesc +# {0x40000, 0x03008000, RING_PA} -> in the same page find the push window +# memdesc {0x20000, 0x03008000, PUSH_PA} -> inject 0x4480000 -> PUSH_PA. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver3.log" +PUSH_VA = 0x4480000 +WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420")) +WATCH_S = int(os.environ.get("W2_WATCH", "150")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def inject(va, pa, ln=0x1000, ap=1): + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,%d\n" % (va, pa, ln, ap)) + log("INJECT va=0x%x -> pa=0x%x len=0x%x" % (va, pa, ln)) + return True + except OSError as e: + log("inject FAIL", e) + return False + +def find_memdescs(size): + """Find {size, 0x03008000, PA} patterns; returns [(page, off, pa)].""" + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", size, 0x03008000) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + pa = struct.unpack_from("<Q", d, i + 16)[0] + gpa = off2gpa(pos + i) + if 0x100000000 <= pa < 0x180000000: + out.append((gpa & ~0xfff, gpa - (gpa & ~0xfff), pa)) + pos += CH + return out + +def main(): + log("=== resolver3 start ===") + # count pre-existing ring lines; wait for a NEW one (fresh client kick) + def ring_lines(): + out = [] + for ln in dmesg().splitlines(): + m = re.search(r"ring rc=0 pa=0x([0-9a-f]+) ap=", ln) + if m: + out.append(int(m.group(1), 16)) + return out + base = ring_lines() + log("pre-existing ring lines: %d" % len(base)) + ring_pa = None + deadline = time.time() + WAIT_RING_S + while time.time() < deadline: + cur = ring_lines() + if len(cur) > len(base): + ring_pa = cur[-1] + break + time.sleep(1) + if not ring_pa: + log("no NEW ring line; exit"); return 1 + log("NEW RING PA = 0x%x" % ring_pa) + + injected = set() + watch = time.time() + WATCH_S + while time.time() < watch: + # ring window memdesc {0x40000, 0x03008000, ring_pa} + rd_hits = [x for x in find_memdescs(0x40000) if x[2] == ring_pa] + if not rd_hits: + time.sleep(0.5); continue + for (page, off, _pa) in rd_hits: + d = rd(page, 0x1000) + if not d: continue + # find sibling push memdesc in the same page + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if sz == 0x20000 and fl == 0x03008000: + pa = struct.unpack_from("<Q", d, o + 16)[0] + if not (0x100000000 <= pa < 0x180000000): + continue + key = (page, pa) + if key in injected: continue + log("memdesc page=0x%x ring_desc@0x%x push_desc@0x%x push_pa=0x%x" + % (page, off, o, pa)) + # content sanity: read the push page + pd = rd(pa, 0x40) + log(" push page head: %s" % (pd.hex() if pd else "?")) + if inject(PUSH_VA, pa, 0x1000): + injected.add(key) + if injected and ("injected mapping va=0x%x" % PUSH_VA) in dmesg(): + log("service confirmed; keep watching") + time.sleep(0.5) + log("=== resolver3 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver4.py b/tools/guest-diag/wall2/wall2_resolver4.py new file mode 100644 index 0000000..140aaa7 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver4.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# wall2_resolver4.py - HOST-SIDE wall#2 resolver v4 (final recipe). +# +# Association proven on r675: the RM keeps, in ONE page, the client's channel +# window records and their memdescs: +# push window record {VA=0x4480000, SIZE=0x20000} +# ring window record {VA=0x44a0000, SIZE=0x40000} +# push memdesc {0x20000, 0x03008000, FIRST_PA, ptr} +# ring memdesc {0x40000, 0x03008000, FIRST_PA, ptr} +# where FIRST_PA is the PA of the window's FIRST VA page. Verified: the ring +# memdesc's PA == the service's validated ring pa. So the push memdesc's PA == +# the PA of VA 0x4480000 (the push page the service cannot translate). +# +# Recipe: wait for a NEW ring line -> find the page containing the ring window +# record {0x44a0000, 0x40000} -> require the sibling push window record +# {0x4480000, 0x20000} and the memdescs in the same page -> cross-check the +# ring memdesc PA == ring_pa -> inject 0x4480000 -> push memdesc PA (sudo). +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver4.log" +PUSH_VA = 0x4480000 +RING_VA = 0x44a0000 +WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420")) +WATCH_S = int(os.environ.get("W2_WATCH", "180")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x len=0x%x OK" % (va, pa, ln)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + """Find qword pairs {a,b}; returns [(page, off)].""" + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + gpa = off2gpa(pos + i) + out.append((gpa & ~0xfff, gpa - (gpa & ~0xfff))) + pos += CH + return out + +def scan_page_descs(page): + d = rd(page, 0x1000) + if not d: return [] + out = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if fl != 0x03008000: continue + if sz not in (0x20000, 0x40000): continue + pa = struct.unpack_from("<Q", d, o + 16)[0] + if 0x100000000 <= pa < 0x180000000: + out.append((sz, o, pa)) + return out + +def main(): + log("=== resolver4 start ===") + def ring_lines(): + return re.findall(r"ring rc=0 pa=0x([0-9a-f]+) ap=", dmesg()) + base = ring_lines() + log("pre-existing ring lines: %d" % len(base)) + ring_pa = None + deadline = time.time() + WAIT_RING_S + while time.time() < deadline: + cur = ring_lines() + if len(cur) > len(base): + ring_pa = int(cur[-1], 16) + break + time.sleep(1) + if not ring_pa: + log("no NEW ring line; exit"); return 1 + log("NEW RING PA = 0x%x" % ring_pa) + + injected = set() + LEARNED = [0x101e39000, 0x102a62000] # RM window-record pages (stable across runs) + def try_page(page): + descs = scan_page_descs(page) + rdesc = [x for x in descs if x[0] == 0x40000] + pdesc = [x for x in descs if x[0] == 0x20000] + if not rdesc or not pdesc: + return False + if rdesc[0][2] != ring_pa: + return False + ok = False + for (_sz, o, pa) in pdesc: + key = (page, o) + if key in injected: continue + pd = rd(pa, 0x40) + log("page=0x%x ring_desc@0x%x(pa=0x%x OK) push_desc@0x%x pa=0x%x head=%s" + % (page, rdesc[0][1], rdesc[0][2], o, pa, + pd.hex() if pd else "?")) + if inject(PUSH_VA, pa, 0x1000): + injected.add(key); ok = True + return ok + log("learned pages: %s" % [hex(p) for p in LEARNED]) + watch = time.time() + WATCH_S + last_full = 0 + while time.time() < watch: + hit = False + for page in LEARNED: + if try_page(page): + hit = True + if not hit and time.time() - last_full > 5: + last_full = time.time() + for (page, _off) in find_qpair(RING_VA, 0x40000): + if try_page(page): + if page not in LEARNED: + LEARNED.append(page) + hit = True + time.sleep(0.25) + log("=== resolver4 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver5.py b/tools/guest-diag/wall2/wall2_resolver5.py new file mode 100644 index 0000000..faec902 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver5.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +# wall2_resolver5.py - final: continuous scan, all-candidate injection. +# +# Structure (r675, verified): an RM page holds the client channel's window +# records and memdescs: +# {0x4480000, 0x20000} push window record +# {0x44a0000, 0x40000} ring window record +# {0x20000, 0x03008000, PUSH_PA} push window memdesc +# {0x40000, 0x03008000, RING_PA} ring window memdesc (PA == validated ring pa) +# The resolver scans CONTINUOUSLY (no ring-line gate), finds pages holding the +# pair of window records, and injects 0x4480000 -> every 0x20000 memdesc PA in +# the same page (last write wins in the service; content check arbitrates). +import subprocess, struct, time, sys, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver5.log" +PUSH_VA = 0x4480000 +RING_VA = 0x44a0000 +WATCH_S = int(os.environ.get("W2_WATCH", "600")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x" % (va, pa)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + gpa = off2gpa(pos + i) + out.append(gpa & ~0xfff) + pos += CH + return out + +def main(): + log("=== resolver5 start (watch=%ds) ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + try: + push_pages = set(find_qpair(PUSH_VA, 0x20000)) + ring_pages = set(find_qpair(RING_VA, 0x40000)) + except Exception as e: + log("scan error:", e); time.sleep(2); continue + for page in sorted(push_pages & ring_pages): + d = rd(page, 0x1000) + if not d: continue + # collect 0x20000 descs in this page + cands = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if sz == 0x20000 and fl == 0x03008000: + pa = struct.unpack_from("<Q", d, o + 16)[0] + if 0x100000000 <= pa < 0x180000000: + cands.append((o, pa)) + for (o, pa) in cands: + key = (page, o) + if key in injected: continue + pd = rd(pa, 0x40) + nz = sum(1 for b in pd if b) if pd else -1 + log("page=0x%x desc@0x%x pa=0x%x live_nz=%d head=%s" + % (page, o, pa, nz, pd.hex()[:64] if pd else "?")) + if inject(PUSH_VA, pa): + injected.add(key) + time.sleep(1) + log("=== resolver5 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver6.py b/tools/guest-diag/wall2/wall2_resolver6.py new file mode 100644 index 0000000..4653efa --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver6.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +# wall2_resolver6.py - resolver5 logic for BOTH current push windows: +# pair A: push 0x4480000 (0x20000) + ring 0x44a0000 (0x40000) +# pair B: push 0x4ca0000 (0x20000) + ring 0x4cc0000 (0x40000) +# Scans guest RAM continuously; injects PUSH_VA -> every 0x20000 memdesc PA +# found in a page holding both window records. +import subprocess, struct, time, sys, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver6.log" +WATCH_S = int(os.environ.get("W2_WATCH", "420")) +PAIRS = [(0x4480000, 0x44a0000), (0x4ca0000, 0x4cc0000)] + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x" % (va, pa)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + out.append(off2gpa(pos + i) & ~0xfff) + pos += CH + return out + +def main(): + log("=== resolver6 start (watch=%ds) ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + for (pva, rva) in PAIRS: + try: + pages = set(find_qpair(pva, 0x20000)) & set(find_qpair(rva, 0x40000)) + except Exception as e: + log("scan error:", e); continue + for page in sorted(pages): + d = rd(page, 0x1000) + if not d: continue + cands = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if sz == 0x20000 and fl == 0x03008000: + pa = struct.unpack_from("<Q", d, o + 16)[0] + if 0x100000000 <= pa < 0x180000000: + cands.append((o, pa)) + for (o, pa) in cands: + key = (pva, page, o) + if key in injected: continue + pd = rd(pa, 0x40) + nz = sum(1 for b in pd if b) if pd else -1 + log("pair=0x%x page=0x%x desc@0x%x pa=0x%x live_nz=%d head=%s" + % (pva, page, o, pa, nz, pd.hex()[:64] if pd else "?")) + if inject(pva, pa): + injected.add(key) + time.sleep(1) + log("=== resolver6 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver7.py b/tools/guest-diag/wall2/wall2_resolver7.py new file mode 100644 index 0000000..5d9e58e --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver7.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +# wall2_resolver7.py - resolver6 + release-window injection + live-first ordering. +# Fixes the resolver6 zero-page overwrite: candidates are injected good-first +# (live_nz>0) and zero pages last, so the readable mapping wins by order if the +# service keeps the newest write. +# New: the GR init release windows (0x4b16000 size 0x1000 from the record page, +# and any {VA,0x1000} record alongside) are injected from the size-0x1000 +# memdescs of the same page. +import subprocess, struct, time, sys, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver7.log" +WATCH_S = int(os.environ.get("W2_WATCH", "420")) +PAIRS = [(0x4480000, 0x44a0000), (0x4ca0000, 0x4cc0000)] +REL_WINS = [0x4b16000] + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def off2gpa(off): return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x" % (va, pa)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + out.append(off2gpa(pos + i) & ~0xfff) + pos += CH + return out + +def collect(page): + d = rd(page, 0x1000) + if not d: return None + descs = [] + wins = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + pa = struct.unpack_from("<Q", d, o + 16)[0] + if fl == 0x03008000 and 0x1000 <= sz <= 0x400000 and \ + 0x100000000 <= pa < 0x180000000: + descs.append((o, sz, pa)) + va2 = sz + if 0x4000000 <= va2 < 0x100000000 and (va2 & 0xfff) == 0: + sz2 = fl + if 0x1000 <= sz2 <= 0x1000000 and (sz2 & 0xfff) == 0: + wins.append((va2, sz2)) + return descs, wins + +def main(): + log("=== resolver7 start (watch=%ds) ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + pages = set() + for (pva, rva) in PAIRS: + try: + pages |= set(find_qpair(pva, 0x20000)) & set(find_qpair(rva, 0x40000)) + except Exception as e: + log("scan error:", e) + for page in sorted(pages): + got = collect(page) + if not got: continue + descs, wins = got + # push windows: inject live-first (nz>0 first, zeros last) + for (pva, _rva) in PAIRS: + cands = [(o, sz, pa) for (o, sz, pa) in descs if sz == 0x20000] + order = [] + for (o, sz, pa) in cands: + pd = rd(pa, 0x40) + nz = sum(1 for b in pd if b) if pd else -1 + log("pair=0x%x page=0x%x desc@0x%x pa=0x%x live_nz=%d" % (pva, page, o, pa, nz)) + order.append((nz, o, pa)) + order.sort(key=lambda t: t[0]) # zeros first, live last + for (nz, o, pa) in order: + key = (pva, page, o) + if key in injected: continue + if inject(pva, pa): + injected.add(key) + # release windows: any {VA, 0x1000} record on the page -> size-0x1000 descs + for (va, sz) in wins: + if va not in REL_WINS and not (0x4b00000 <= va <= 0x4b20000): + continue + for (o, dsz, pa) in descs: + if dsz != sz: + continue + key = (va, page, o) + if key in injected: continue + if inject(va, pa): + injected.add(key) + time.sleep(1) + log("=== resolver7 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/xfocus.c b/tools/guest-diag/xfocus.c new file mode 100644 index 0000000..e0ec05d --- /dev/null +++ b/tools/guest-diag/xfocus.c @@ -0,0 +1,69 @@ +// xfocus.c - print the current X input focus and the gzdoom window, and try +// to focus the gzdoom window; then report again. Diagnostic for WM-less X. +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <stdio.h> +#include <string.h> + +static Window find_named(Display *d, Window root, const char *sub, + char *title, size_t tl) +{ + Window r, parent, *kids = NULL; + unsigned int nk = 0; + + title[0] = 0; + if (!XQueryTree(d, root, &r, &parent, &kids, &nk)) + return 0; + for (unsigned int i = 0; i < nk; i++) { + char *name = NULL; + + if (XFetchName(d, kids[i], &name) && name) { + if (strstr(name, sub)) { + snprintf(title, tl, "%s", name); + Window w = kids[i]; + + XFree(name); + if (kids) + XFree(kids); + return w; + } + XFree(name); + } + } + if (kids) + XFree(kids); + return 0; +} + +int main(void) +{ + Display *d = XOpenDisplay(NULL); + Window root, focus, w, croot; + int rx, ry, wx, wy; + unsigned int mask; + char title[256]; + + if (!d) { + printf("no display\n"); + return 1; + } + root = DefaultRootWindow(d); + w = find_named(d, root, "Freedoom", title, sizeof(title)); + XGetInputFocus(d, &focus, &rx); + printf("gzdoom window: 0x%lx '%s'\n", w, title); + printf("input focus: 0x%lx\n", focus); + printf("pointer window: "); + if (XQueryPointer(d, root, &croot, &rx /*reuse*/, &wx, &wy, &rx, &ry, + &mask)) + printf("0x%lx at %d,%d mask=0x%x\n", rx, wx, wy, mask); + else + printf("query failed\n"); + if (w) { + XSetInputFocus(d, w, RevertToParent, CurrentTime); + XFlush(d); + XGetInputFocus(d, &focus, &rx); + printf("after set: 0x%lx\n", focus); + } + XCloseDisplay(d); + return 0; +} diff --git a/tools/guest-diag/xgrab.c b/tools/guest-diag/xgrab.c new file mode 100644 index 0000000..eb605d6 --- /dev/null +++ b/tools/guest-diag/xgrab.c @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: BSD-3-Clause +// xgrab.c - capture the X screen (root window) into an append-only frame file. +// Pure client-side XGetImage on a software X server (Xvfb): no GL, no driver, +// no present-path hooks. Same record format as sdl_cap.c: +// [0] 8B "GZCAPV01" + zero pad to 4096 +// [4096..] records: 4B "GZF2" | u32 w | u32 h | u32 stride | u32 bpp | +// u64 ts_ns | u64 seq | stride*h bytes (top-down BGRA) +// Trailing record seq=0xFFFFFFFFFFFFFFFF on clean stop. +// Usage: xgrab <display> <out> <maxframes|0> <pace_us|0> +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include <unistd.h> + +int main(int argc, char **argv) +{ + const char *dpy = argc > 1 ? argv[1] : ":1"; + const char *out = argc > 2 ? argv[2] : "/dev/shm/gzframes.bin"; + long maxf = argc > 3 ? atol(argv[3]) : 0; + long pace = argc > 4 ? atol(argv[4]) : 0; + int r_x = argc > 5 ? atoi(argv[5]) : -1; + int r_y = argc > 6 ? atoi(argv[6]) : 0; + int r_w = argc > 7 ? atoi(argv[7]) : 0; + int r_h = argc > 8 ? atoi(argv[8]) : 0; + Display *d; + Window root; + XWindowAttributes wa; + FILE *f; + char hdr[4096]; + uint64_t seq = 0; + + d = XOpenDisplay(dpy); + if (!d) { + fprintf(stderr, "xgrab: cannot open display %s\n", dpy); + return 1; + } + root = DefaultRootWindow(d); + if (!XGetWindowAttributes(d, root, &wa)) { + fprintf(stderr, "xgrab: no root attrs\n"); + return 1; + } + if (r_x < 0 || r_w <= 0 || r_h <= 0 || r_x + r_w > wa.width || + r_y + r_h > wa.height) { + r_x = 0; r_y = 0; r_w = wa.width; r_h = wa.height; + } + fprintf(stderr, "xgrab: root %dx%d depth=%d region %d,%d %dx%d\n", + wa.width, wa.height, wa.depth, r_x, r_y, r_w, r_h); + f = fopen(out, "wb"); + if (!f) { + perror("xgrab: open out"); + return 1; + } + memset(hdr, 0, sizeof(hdr)); + memcpy(hdr, "GZCAPV01", 8); + fwrite(hdr, 1, sizeof(hdr), f); + fflush(f); + + while (!maxf || (long)seq < maxf) { + XImage *im = XGetImage(d, root, r_x, r_y, r_w, r_h, AllPlanes, + ZPixmap); + struct timespec ts; + uint64_t tns; + uint32_t rh[4]; + char m[4] = { 'G', 'Z', 'F', '2' }; + int y; + + if (!im) { + usleep(10000); + continue; + } + if (im->bits_per_pixel != 32) { + fprintf(stderr, "xgrab: unexpected bpp %d\n", + im->bits_per_pixel); + XDestroyImage(im); + break; + } + clock_gettime(CLOCK_MONOTONIC, &ts); + tns = (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec; + rh[0] = (uint32_t)r_w; + rh[1] = (uint32_t)r_h; + rh[2] = (uint32_t)r_w * 4; + rh[3] = 32; + fwrite(m, 1, 4, f); + fwrite(rh, 4, 4, f); + fwrite(&tns, 8, 1, f); + fwrite(&seq, 8, 1, f); + for (y = 0; y < r_h; y++) + fwrite(im->data + (size_t)y * im->bytes_per_line, 1, + (size_t)r_w * 4, f); + fflush(f); + seq++; + XDestroyImage(im); + if (pace) + usleep(pace); + if ((seq & 63) == 0) + fprintf(stderr, "xgrab: frames=%llu\n", + (unsigned long long)seq); + } + { + char m[4] = { 'G', 'Z', 'F', '2' }; + uint64_t end = 0xFFFFFFFFFFFFFFFFull; + + fwrite(m, 1, 4, f); + fwrite(&end, 8, 1, f); + fflush(f); + } + fclose(f); + XCloseDisplay(d); + fprintf(stderr, "xgrab: done frames=%llu\n", (unsigned long long)seq); + return 0; +} diff --git a/tools/guest-diag/xorg-vgpu.conf b/tools/guest-diag/xorg-vgpu.conf new file mode 100644 index 0000000..4d4c07d --- /dev/null +++ b/tools/guest-diag/xorg-vgpu.conf @@ -0,0 +1,31 @@ +# Minimal Xorg config for the emulated vGPU (nvidia driver), headless. +# AllowEmptyInitialConfiguration lets the X server start with no connected +# display; UseDisplayDevice "None" keeps the server from demanding one. +Section "ServerLayout" + Identifier "Layout0" + Screen 0 "Screen0" +EndSection + +Section "Device" + Identifier "Device0" + Driver "nvidia" + BusID "PCI:1:0:0" + Option "AllowEmptyInitialConfiguration" "True" + Option "UseDisplayDevice" "None" + Option "ConnectToAcpid" "False" +EndSection + +Section "Screen" + Identifier "Screen0" + Device "Device0" + DefaultDepth 24 + SubSection "Display" + Depth 24 + Virtual 1280 720 + EndSubSection +EndSection + +Section "ServerFlags" + Option "AutoAddDevices" "False" + Option "AutoEnableDevices" "False" +EndSection
\ No newline at end of file diff --git a/tools/guest-diag/xresize.c b/tools/guest-diag/xresize.c new file mode 100644 index 0000000..877f0e9 --- /dev/null +++ b/tools/guest-diag/xresize.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: BSD-3-Clause +// xresize.c - find a window by name substring and resize/move it (Xvfb has +// no WM, so a client-side XMoveResizeWindow is the only way to force the +// game window onto the screen). +// Usage: xresize <display> <name-substr> <w> <h> <x> <y> +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <stdio.h> +#include <string.h> +#include <stdlib.h> + +static Window find_named(Display *d, Window root, const char *sub) +{ + Window r, parent, *kids = NULL; + unsigned int nkids = 0; + + if (!XQueryTree(d, root, &r, &parent, &kids, &nkids)) + return 0; + for (unsigned int i = 0; i < nkids; i++) { + char *name = NULL; + + if (XFetchName(d, kids[i], &name) && name) { + int hit = strstr(name, sub) != NULL; + + XFree(name); + if (hit) { + Window w = kids[i]; + + if (kids) + XFree(kids); + return w; + } + } + } + if (kids) + XFree(kids); + return 0; +} + +int main(int argc, char **argv) +{ + Display *d; + Window w; + + if (argc < 6) { + fprintf(stderr, "usage: xresize <display> <name> <w> <h> <x> <y>\n"); + return 2; + } + d = XOpenDisplay(argv[1]); + if (!d) { + fprintf(stderr, "no display\n"); + return 1; + } + w = find_named(d, DefaultRootWindow(d), argv[2]); + if (!w) { + fprintf(stderr, "window '%s' not found\n", argv[2]); + return 1; + } + XMoveResizeWindow(d, w, atoi(argv[5]), atoi(argv[6]), atoi(argv[3]), + atoi(argv[4])); + XFlush(d); + printf("resized 0x%lx to %sx%s at %s,%s\n", w, argv[3], argv[4], + argv[5], argv[6]); + XCloseDisplay(d); + return 0; +} diff --git a/tools/guest-diag/xshm_pub.c b/tools/guest-diag/xshm_pub.c new file mode 100644 index 0000000..98fb539 --- /dev/null +++ b/tools/guest-diag/xshm_pub.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * xshm_pub.c - capture an X11 screen and publish frames into the kvmfr + * ivshmem BAR so the host can display them (the Looking-Glass-style path: + * the guest renders with its real graphics stack - X on the passed GPU / + * GLX / Vulkan surface - and this tool copies the framebuffer into the + * shared BAR; the host reader decodes it). + * + * Layout in the BAR (same as tools/cuda-ab/frame_producer.cu): + * offset 0x100000: struct frame_hdr { magic, version, width, height, + * stride, format, seq } then width*height*4 RGBA8 bytes. + * + * Usage (guest, with DISPLAY set): + * ./xshm_pub [frames] [fps] [x] [y] [w] [h] + * Prints FRAME lines (first 5 and every 30th) then FRAME_DONE. + * Build (host, same glibc): gcc -O2 -o xshm_pub xshm_pub.c -lX11 -lXext + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <sys/mman.h> +#include <time.h> +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <X11/extensions/XShm.h> +#include <sys/ipc.h> +#include <sys/shm.h> + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +int main(int argc, char **argv) +{ + int frames = argc > 1 ? atoi(argv[1]) : 120; + int fps = argc > 2 ? atoi(argv[2]) : 10; + int gx = argc > 3 ? atoi(argv[3]) : 0; + int gy = argc > 4 ? atoi(argv[4]) : 0; + int gw = argc > 5 ? atoi(argv[5]) : 0; + int gh = argc > 6 ? atoi(argv[6]) : 0; + const char *bar_path = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + Display *dpy; + Window root; + XImage *img = NULL; + XShmSegmentInfo shm; + int use_shm = 1, fd, i, w, h; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + uint64_t t_first = 0; + uint64_t t_last = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + + dpy = XOpenDisplay(NULL); + if (!dpy) { + printf("XOPEN_FAIL DISPLAY=%s\n", + getenv("DISPLAY") ? getenv("DISPLAY") : "(unset)"); + return 1; + } + root = DefaultRootWindow(dpy); + { + XWindowAttributes wa; + + XGetWindowAttributes(dpy, root, &wa); + w = gw > 0 ? gw : wa.width; + h = gh > 0 ? gh : wa.height; + printf("XSCREEN %dx%d root=%dx%d\n", w, h, wa.width, + wa.height); + } + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + printf("OPEN_FAIL %s\n", bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (bar == MAP_FAILED) { + printf("MMAP_FAIL\n"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + memset(&shm, 0, sizeof(shm)); + if (use_shm) { + img = XShmCreateImage(dpy, DefaultVisual(dpy, + DefaultScreen(dpy)), + DefaultDepth(dpy, DefaultScreen(dpy)), + ZPixmap, NULL, &shm, w, h); + if (!img) { + printf("XSHM_CREATE_FAIL\n"); + return 1; + } + shm.shmid = shmget(IPC_PRIVATE, + (size_t)img->bytes_per_line * h, + IPC_CREAT | 0600); + if (shm.shmid < 0) { + printf("SHMGET_FAIL\n"); + return 1; + } + shm.shmaddr = img->data = shmat(shm.shmid, NULL, 0); + if (shm.shmaddr == (char *)-1) { + printf("SHMAT_FAIL\n"); + return 1; + } + shm.readOnly = False; + if (!XShmAttach(dpy, &shm)) { + printf("XSHM_ATTACH_FAIL (falling back)\n"); + XDestroyImage(img); + img = NULL; + } else { + XSync(dpy, False); + printf("XSHM OK %dx%d bpl=%d\n", w, h, + img->bytes_per_line); + } + } + if (!img) { + img = XGetImage(dpy, root, gx, gy, w, h, AllPlanes, ZPixmap); + if (!img) { + printf("XGETIMAGE_FAIL\n"); + return 1; + } + printf("XGETIMAGE OK %dx%d bpl=%d\n", w, h, + img->bytes_per_line); + } + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + + for (i = 0; i < frames; i++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + int x, y; + + if (shm.shmaddr) { + XShmGetImage(dpy, root, img, gx, gy, AllPlanes); + } else { + XImage *n = XGetImage(dpy, root, gx, gy, w, h, + AllPlanes, ZPixmap); + + if (!n) + break; + XDestroyImage(img); + img = n; + } + /* X ZPixmap: 0x00RRGGBB little-endian -> RGBA bytes */ + for (y = 0; y < h; y++) { + unsigned char *s = (unsigned char *)img->data + + (size_t)y * img->bytes_per_line; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + uint32_t p; + + memcpy(&p, s + 4 * x, 4); + d[4 * x + 0] = (unsigned char)(p >> 16); + d[4 * x + 1] = (unsigned char)(p >> 8); + d[4 * x + 2] = (unsigned char)(p); + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + if (!t_first) + t_first = t0; + t_last = t0; + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + if (i < 5 || (i % 30) == 29) + printf("FRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + if (t_first && t_last > t_first) + printf("XPUB_RATE fps=%.2f frames=%d\n", + (double)(i - 1) * 1e9 / + (double)(t_last - t_first), i); + printf("FRAME_DONE frames=%d %dx%d\n", i, w, h); + if (shm.shmaddr) + XShmDetach(dpy, &shm); + XCloseDisplay(dpy); + return 0; +}
\ No newline at end of file diff --git a/tools/guest-diag/xtest_move.c b/tools/guest-diag/xtest_move.c new file mode 100644 index 0000000..8331487 --- /dev/null +++ b/tools/guest-diag/xtest_move.c @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: BSD-3-Clause +// xtest_move.c - drive the game with XSendEvent-synthetic keys (SDL2 on this +// Xorg ignores XTEST-faked keys; it accepts send_event KeyPress/KeyRelease). +// Finds the game window by name, then holds forward+fire and alternates +// turning left/right so the captured footage has constant motion. +// Usage: xtest_move <display> <seconds> +#include <X11/Xlib.h> +#include <X11/keysym.h> +#include <X11/Xutil.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> + +static Display *d; +static Window game; + +static Window find_named(const char *sub) +{ + Window root = DefaultRootWindow(d), r, parent, *kids = NULL; + unsigned int nk = 0; + + if (!XQueryTree(d, root, &r, &parent, &kids, &nk)) + return 0; + for (unsigned int i = 0; i < nk; i++) { + char *name = NULL; + + if (XFetchName(d, kids[i], &name) && name) { + if (strstr(name, sub)) { + Window w = kids[i]; + + XFree(name); + if (kids) + XFree(kids); + return w; + } + XFree(name); + } + } + if (kids) + XFree(kids); + return 0; +} + +static void send_key(KeySym ks, int down) +{ + XEvent e; + Window root = DefaultRootWindow(d); + int rx, ry, wx, wy; + unsigned int mask; + Window cr, pr; + + if (!game) + return; + memset(&e, 0, sizeof(e)); + e.xkey.type = down ? KeyPress : KeyRelease; + e.xkey.display = d; + e.xkey.window = game; + e.xkey.root = root; + e.xkey.subwindow = None; + e.xkey.time = CurrentTime; + XQueryPointer(d, root, &cr, &pr, &rx, &ry, &wx, &wy, &mask); + e.xkey.x = wx; + e.xkey.y = wy; + e.xkey.x_root = rx; + e.xkey.y_root = ry; + e.xkey.same_screen = True; + e.xkey.keycode = XKeysymToKeycode(d, ks); + e.xkey.state = 0; + XSendEvent(d, game, True, down ? KeyPressMask : KeyReleaseMask, &e); + XFlush(d); +} + +int main(int argc, char **argv) +{ + int secs = argc > 2 ? atoi(argv[2]) : 12; + int t; + + d = XOpenDisplay(argc > 1 ? argv[1] : ":1"); + if (!d) { + fprintf(stderr, "xtest_move: no display\n"); + return 1; + } + game = find_named("Freedoom"); + if (!game) + game = find_named("Outer Prison"); + if (!game) + game = find_named("Doom"); + fprintf(stderr, "xtest_move: game window 0x%lx\n", game); + if (!game) + return 1; + XSetInputFocus(d, game, RevertToParent, CurrentTime); + XFlush(d); + + /* Hold forward only (no fire: fire pulls enemies and the player can + * die mid-capture; with +noclip from the command line this is a smooth + * fly-through). */ + send_key(XK_Up, 1); + + for (t = 0; t < secs * 10; t++) { + static int dir; /* 0 = left held, 1 = right held */ + + XSetInputFocus(d, game, RevertToParent, CurrentTime); /* keep focus */ + /* Re-assert the held forward key: SDL's held state drops between + * synthetic events, so forward motion stalls unless re-sent. */ + if ((t % 4) == 0) + send_key(XK_Up, 1); + if ((t % 9) == 8) { + KeySym off = dir ? XK_Right : XK_Left; + KeySym on = dir ? XK_Left : XK_Right; + + send_key(off, 0); + usleep(30000); + send_key(on, 1); + dir = !dir; + } + XFlush(d); + usleep(100000); + } + send_key(XK_Left, 0); + send_key(XK_Right, 0); + send_key(XK_Up, 0); + XCloseDisplay(d); + fprintf(stderr, "xtest_move: done\n"); + return 0; +} diff --git a/tools/lg/build_video.sh b/tools/lg/build_video.sh new file mode 100755 index 0000000..4f80be8 --- /dev/null +++ b/tools/lg/build_video.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# build_video.sh - build the final open-iov 1.0.0 demo video: a drawtext +# title card, the split-screen main segment (two GZDOOM VMs, one per pane, +# captured through the KVMFR ivshmem path), and an optional end card, +# concatenated into one 1280x720@30 libx264 yuv420p mp4. +# +# Usage: +# build_video.sh <title-text> <main.mp4> [end-text] +# build_video.sh <title-text> <dirA> <dirB> [end-text] +# title-text text for the title card (required, non-empty) +# main.mp4 pre-built split-screen video (e.g. from split_screen.sh) +# dirA dirB two frame-%04d.ppm dirs; split_screen.sh (next to this +# script) is invoked first at FPS to build the main segment +# end-text optional text for the end card (omitted = no end card) +# +# Output: $OUT if set, else ./open-iov-1.0.0-demo.mp4 in the current dir. +# Env knobs: FPS (default 30), CARD_SECS (default 5), FONTSIZE (default 48). +# +# Approach: title/end cards are generated with ffmpeg drawtext (white on +# dark 0x101010, 1280x720); the main segment is normalized to 1280x720@30 +# (aspect-preserving fit + letterbox pad, audio dropped - the demo video is +# video-only); the three segments are concatenated with the concat demuxer +# followed by ONE re-encode. Stream-copy concat is brittle across +# separately-generated mp4s (timestamp/param mismatches); the re-encode is +# cheap at these lengths and guarantees a clean single-stream output. +# The title font is auto-detected (DejaVuSans/NotoSans); the script fails +# with a clear message if no TTF font exists. +# +# Requires: ffmpeg with libx264 + drawtext (libfreetype); split_screen.sh +# next to this script for the two-dirs form. +set -euo pipefail + +usage() { + sed -n '3,31p' "$0" | sed 's/^# \{0,1\}//' +} + +die() { echo "build_video.sh: ERROR: $*" >&2; exit 1; } + +# grep -q would SIGPIPE the ffmpeg producer under pipefail; capture first. +ffmpeg_has() { # $1 = encoder/filter token + local out + out=$(ffmpeg -hide_banner "$2" 2>/dev/null) || true + case "$out" in *" $1 "*) return 0 ;; *) return 1 ;; esac +} + +command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg not found in PATH" +ffmpeg_has libx264 -encoders || die "ffmpeg has no libx264 encoder" +ffmpeg_has drawtext -filters || die "ffmpeg has no drawtext filter (needs libfreetype)" + +# --- locate a usable TTF font ------------------------------------------- +FONT="" +for f in \ + /usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf \ + /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf \ + /usr/share/fonts/truetype/noto/NotoSans-Regular.ttf \ + /usr/share/fonts/dejavu/DejaVuSans.ttf; do + if [ -f "$f" ]; then FONT=$f; break; fi +done +if [ -z "$FONT" ] && command -v fc-match >/dev/null 2>&1; then + f=$(fc-match -f '%{file}' sans 2>/dev/null || true) + [ -n "$f" ] && [ -f "$f" ] && FONT=$f +fi +[ -n "$FONT" ] || die "no usable TTF font found (looked for DejaVuSans, NotoSans; \ +install fonts-dejavu-core or set up fontconfig)" +echo "build_video.sh: font=$FONT" >&2 + +# --- args ---------------------------------------------------------------- +[ "$#" -ge 1 ] || { usage; exit 2; } +TITLE=$1; shift +[ -n "$TITLE" ] || die "title text must be non-empty" +MAIN="" DIR_A="" DIR_B="" END="" +if [ "$#" -ge 1 ] && [ -d "$1" ]; then + DIR_A=$1 + [ "$#" -ge 2 ] && [ -d "$2" ] || die "two-dirs form needs two directories: <dirA> <dirB> [end-text]" + DIR_B=$2 + END=${3:-} +elif [ "$#" -ge 1 ]; then + [ -f "$1" ] || die "main input not found: $1" + MAIN=$1 + END=${2:-} +else + die "missing main input (mp4 file or two frame dirs)" +fi + +FPS=${FPS:-30} +CARD_SECS=${CARD_SECS:-5} +FONTSIZE=${FONTSIZE:-48} +OUT=${OUT:-$PWD/open-iov-1.0.0-demo.mp4} +HERE=$(cd "$(dirname "$0")" && pwd) + +TMP=$(mktemp -d "${TMPDIR:-/tmp}/build_video.XXXXXX") +trap 'rm -rf "$TMP"' EXIT + +# --- cards (drawtext; text via textfile so ':' ',' '%' etc. are literal) -- +make_card() { + local text=$1 out=$2 tf="$TMP/card-$$.txt" + printf '%s' "$text" > "$tf" + ffmpeg -hide_banner -loglevel warning -y \ + -f lavfi -i "color=c=0x101010:s=1280x720:r=$FPS:d=$CARD_SECS" \ + -vf "drawtext=fontfile=$FONT:textfile=$tf:fontcolor=white:fontsize=$FONTSIZE:expansion=none:x=(w-text_w)/2:y=(h-text_h)/2" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + "$out" +} + +echo "build_video.sh: title card (${CARD_SECS}s)..." >&2 +make_card "$TITLE" "$TMP/title.mp4" + +# --- main segment -------------------------------------------------------- +if [ -n "$DIR_A" ]; then + [ -x "$HERE/split_screen.sh" ] || die "split_screen.sh not found/executable next to build_video.sh" + echo "build_video.sh: building split-screen main from $DIR_A + $DIR_B..." >&2 + "$HERE/split_screen.sh" "$DIR_A" "$DIR_B" "$TMP/split.mp4" "$FPS" + MAIN=$TMP/split.mp4 +fi +echo "build_video.sh: normalizing main segment to 1280x720@$FPS..." >&2 +ffmpeg -hide_banner -loglevel warning -y -i "$MAIN" \ + -vf "scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=$FPS" \ + -an -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + "$TMP/main.mp4" + +# --- concat [title, main, end] ------------------------------------------ +printf "file '%s'\n" "$TMP/title.mp4" "$TMP/main.mp4" > "$TMP/list.txt" +if [ -n "$END" ]; then + echo "build_video.sh: end card (${CARD_SECS}s)..." >&2 + make_card "$END" "$TMP/end.mp4" + printf "file '%s'\n" "$TMP/end.mp4" >> "$TMP/list.txt" +fi + +echo "build_video.sh: concatenating..." >&2 +ffmpeg -hide_banner -loglevel warning -y -f concat -safe 0 -i "$TMP/list.txt" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + -r "$FPS" -movflags +faststart \ + "$OUT" + +[ -s "$OUT" ] || die "ffmpeg produced no output: $OUT" +if command -v ffprobe >/dev/null 2>&1; then + ffprobe -v error -select_streams v:0 \ + -show_entries stream=width,height,r_frame_rate,duration \ + -show_entries format=duration,size \ + -of default=nw=1 "$OUT" | sed 's/^/build_video.sh: /' >&2 +fi +echo "VIDEO_DONE $OUT" >&2 diff --git a/tools/lg/frame_reader.py b/tools/lg/frame_reader.py new file mode 100644 index 0000000..783fcaf --- /dev/null +++ b/tools/lg/frame_reader.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# frame_reader.py - read the GPU-rendered frames the guest publishes into the +# kvmfr ivshmem BAR (see tools/cuda-ab/frame_producer.cu) and save them on the +# host as PPM/PNG. This is the host half of the guest frame-producer path: +# the QEMU ivshmem-plain device is backed by /dev/shm/kvmfr-<uuid>, which the +# guest maps through /sys/bus/pci/devices/0000:00:03.0/resource2. +# +# Usage: frame_reader.py [--shm /dev/shm/kvmfr-<uuid>] [--frames N] [--out DIR] +import argparse +import mmap +import os +import struct +import sys +import time + +MAGIC = 0x4B564D46 +FRAME_OFFSET = 0x100000 +# magic, version, width, height, stride, format, seq, reserved, timestamp_ns +HDR = struct.Struct("<IIIIIIIIQ") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--shm", default=None) + ap.add_argument("--frames", type=int, default=5) + ap.add_argument("--out", default="/tmp/opencode/frames") + ap.add_argument("--timeout", type=float, default=120.0) + args = ap.parse_args() + + if not args.shm: + import glob + cands = [p for p in glob.glob("/dev/shm/kvmfr-*") + if os.path.getsize(p) > 0] + if not cands: + print("NO_SHM_FILE", file=sys.stderr) + return 1 + args.shm = max(cands, key=os.path.getsize) + os.makedirs(args.out, exist_ok=True) + print("SHM %s" % args.shm) + + fd = os.open(args.shm, os.O_RDWR) + size = os.fstat(fd).st_size + mm = mmap.mmap(fd, size, mmap.MAP_SHARED, + mmap.PROT_READ | mmap.PROT_WRITE) + + deadline = time.time() + args.timeout + saved = 0 + mm.seek(FRAME_OFFSET) + magic, _v, _w, _h, _s, _f, last_seq, _r, _t = HDR.unpack(mm.read(HDR.size)) + if magic != MAGIC: + last_seq = 0 + print("SEED last_seq=%d" % last_seq) + while saved < args.frames and time.time() < deadline: + mm.seek(FRAME_OFFSET) + magic, ver, w, h, stride, fmt, seq, _res, ts = HDR.unpack( + mm.read(HDR.size)) + if magic != MAGIC or seq == 0 or seq == last_seq or w == 0: + time.sleep(0.05) + continue + fb = mm.read(w * h * 4) + last_seq = seq + path = os.path.join(args.out, "frame-%04d.ppm" % seq) + with open(path, "wb") as f: + f.write(b"P6\n%d %d\n255\n" % (w, h)) + # RGBA -> RGB + rgb = bytearray(w * h * 3) + rgb[0::3] = fb[0::4] + rgb[1::3] = fb[1::4] + rgb[2::3] = fb[2::4] + f.write(bytes(rgb)) + saved += 1 + print("SAVED seq=%d %dx%d fmt=%d -> %s" % (seq, w, h, fmt, path)) + mm.close() + os.close(fd) + print("READER_DONE saved=%d last_seq=%d" % (saved, last_seq)) + return 0 if saved else 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/lg/frame_sequence.sh b/tools/lg/frame_sequence.sh new file mode 100755 index 0000000..958a34a --- /dev/null +++ b/tools/lg/frame_sequence.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# frame_sequence.sh - capture an animated GPU-rendered frame sequence end to +# end: run the guest's CUDA frame producer into the kvmfr ivshmem BAR and +# save every published frame on the host as PPM/PNG (the proven +# CUDA -> ivshmem -> host path; see docs/05-operations/06). +# +# Usage: tools/lg/frame_sequence.sh [frames] [width] [height] [fps] [outdir] +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +FRAMES="${1:-40}" +W="${2:-800}" +H="${3:-600}" +FPS="${4:-10}" +OUT="${5:-/tmp/opencode/frames-seq}" + +mkdir -p "$OUT" +rm -f "$OUT"/frame-*.ppm "$OUT"/frame-*.png + +# Start the host reader first so no published frame is missed. +python3 "$REPO/tools/lg/frame_reader.py" --frames "$FRAMES" --out "$OUT" \ + --timeout 240 > "$OUT/reader.log" 2>&1 & +READER=$! + +sleep 1 +"$REPO/scripts/ssh_guest.sh" \ + "sudo /tmp/frame_producer $FRAMES $W $H $FPS" 2>&1 | grep -v "Permanently added" \ + | tail -5 + +wait "$READER" +echo "reader rc=$?" +ls -1 "$OUT"/frame-*.ppm 2>/dev/null | wc -l diff --git a/tools/lg/split_screen.sh b/tools/lg/split_screen.sh new file mode 100755 index 0000000..55e11ee --- /dev/null +++ b/tools/lg/split_screen.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# split_screen.sh - compose a side-by-side (split-screen) mp4 from two PPM +# frame sequences captured through the KVMFR ivshmem path (guest publishers: +# tools/cuda-ab/frame_producer.cu or tools/guest-diag/xshm_pub.c; host reader: +# tools/lg/frame_reader.py, which writes frame-%04d.ppm into a directory). +# +# Usage: split_screen.sh <dirA> <dirB> <out.mp4> [fps] [left_w] [left_h] +# dirA left-side frame-%04d.ppm directory (any starting index) +# dirB right-side frame-%04d.ppm directory +# out.mp4 output file (libx264, yuv420p) +# fps output frame rate (default 30; pass the measured capture rate, +# e.g. 10, for real-time playback of ~10 fps captures) +# left_w optional left-pane width; the left pane is fitted into +# left_w x H with letterbox padding (aspect preserved) +# left_h optional pane height H; BOTH panes are scaled to height H +# (default: native height of dirA's first frame). Widths follow +# each input's aspect ratio (forced even for yuv420p). +# +# Unequal sequence lengths: the shorter side is padded by repeating its last +# frame. hstack is a framesync filter whose repeatlast=1 default holds the +# shorter input's last frame until the longer input ends; shortest=0 is +# passed explicitly to make that the contract (verified on ffmpeg 6.1.1 for +# both input orders). No symlink/tpad workaround is needed. +# +# Fails loudly (exit 1) on: missing ffmpeg/libx264, missing input dirs, or an +# input dir with no frame-*.ppm files. Requires: ffmpeg (libx264); ffprobe is +# used for the final summary when present. +set -euo pipefail + +usage() { + sed -n '3,28p' "$0" | sed 's/^# \{0,1\}//' +} + +die() { echo "split_screen.sh: ERROR: $*" >&2; exit 1; } + +# grep -q would SIGPIPE the ffmpeg producer under pipefail; capture first. +ffmpeg_has() { # $1 = encoder/filter token + local out + out=$(ffmpeg -hide_banner "$2" 2>/dev/null) || true + case "$out" in *" $1 "*) return 0 ;; *) return 1 ;; esac +} + +# Read width/height from the PPM P6 header (line 2: "<w> <h>"). +ppm_dims() { + local line w h + line=$(sed -n '2p' "$1") + read -r w h _ <<< "$line" + [ -n "${w:-}" ] && [ -n "${h:-}" ] || return 1 + echo "$w $h" +} + +# Scan a frame dir: sets PD_COUNT PD_FIRST PD_LAST PD_W PD_H PD_FILE +probe_dir() { + local dir="$1" label="$2" files f dim + [ -d "$dir" ] || die "$label directory not found: $dir" + mapfile -t files < <(find "$dir" -maxdepth 1 -name 'frame-*.ppm' -type f \ + -printf '%f\n' | sort) + PD_COUNT=${#files[@]} + [ "$PD_COUNT" -gt 0 ] || die "$label directory has no frame-*.ppm files: $dir" + f=${files[0]} + PD_FIRST=${f#frame-}; PD_FIRST=${PD_FIRST%.ppm}; PD_FIRST=$((10#$PD_FIRST)) + f=${files[$((PD_COUNT - 1))]} + PD_LAST=${f#frame-}; PD_LAST=${PD_LAST%.ppm}; PD_LAST=$((10#$PD_LAST)) + if [ $((PD_LAST - PD_FIRST + 1)) -ne "$PD_COUNT" ]; then + echo "split_screen.sh: WARN: $label sequence has gaps ($PD_COUNT files," \ + "indices $PD_FIRST..$PD_LAST); ffmpeg stops at the first gap" >&2 + fi + PD_FILE=$(printf '%s/frame-%04d.ppm' "$dir" "$PD_FIRST") + dim=$(ppm_dims "$PD_FILE") || die "$label: cannot read PPM header of $PD_FILE" + PD_W=${dim% *}; PD_H=${dim#* } +} + +[ "$#" -ge 3 ] || { usage; exit 2; } +DIR_A=$1 DIR_B=$2 OUT=$3 FPS=${4:-30} LW=${5:-} LH=${6:-} + +command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg not found in PATH" +ffmpeg_has libx264 -encoders || die "ffmpeg has no libx264 encoder" + +probe_dir "$DIR_A" left; A_COUNT=$PD_COUNT A_FIRST=$PD_FIRST A_H=$PD_H +probe_dir "$DIR_B" right; B_COUNT=$PD_COUNT B_FIRST=$PD_FIRST + +# Pane height H: left_h if given, else dirA's native height. Keep even. +H=${LH:-$A_H} +if [ $((H % 2)) -ne 0 ]; then + echo "split_screen.sh: WARN: pane height $H rounded down to even for yuv420p" >&2 + H=$((H / 2 * 2)) +fi +[ "$H" -ge 2 ] || die "pane height must be >= 2" + +# Left pane: fit into left_w x H (letterbox) when left_w given, else scale +# to height H with aspect preserved. Right pane: always scale to height H. +if [ -n "$LW" ]; then + if [ $((LW % 2)) -ne 0 ]; then + echo "split_screen.sh: WARN: left_w $LW rounded down to even for yuv420p" >&2 + LW=$((LW / 2 * 2)) + fi + [ "$LW" -ge 2 ] || die "left_w must be >= 2" + LEFT="[0:v]scale=${LW}:${H}:force_original_aspect_ratio=decrease,"\ +"pad=${LW}:${H}:(ow-iw)/2:(oh-ih)/2,setsar=1[l]" +else + LEFT="[0:v]scale=-2:${H},setsar=1[l]" +fi +RIGHT="[1:v]scale=-2:${H},setsar=1[r]" +# shortest=0 + framesync repeatlast=1 (default): shorter side's last frame is +# repeated until the longer side ends (see header comment). +FILTER="${LEFT};${RIGHT};[l][r]hstack=shortest=0" + +echo "split_screen.sh: left=$DIR_A ($A_COUNT frames from $A_FIRST)" \ + "right=$DIR_B ($B_COUNT frames from $B_FIRST) pane_h=$H fps=$FPS" >&2 + +ffmpeg -hide_banner -loglevel warning -y \ + -thread_queue_size 512 -framerate "$FPS" -start_number "$A_FIRST" -i "$DIR_A/frame-%04d.ppm" \ + -thread_queue_size 512 -framerate "$FPS" -start_number "$B_FIRST" -i "$DIR_B/frame-%04d.ppm" \ + -filter_complex "$FILTER" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + -r "$FPS" -movflags +faststart \ + "$OUT" + +[ -s "$OUT" ] || die "ffmpeg produced no output: $OUT" +if command -v ffprobe >/dev/null 2>&1; then + ffprobe -v error -count_frames -select_streams v:0 \ + -show_entries stream=width,height,nb_read_frames \ + -of default=nw=1 "$OUT" | sed 's/^/split_screen.sh: /' >&2 +fi +echo "SPLIT_DONE $OUT" >&2 diff --git a/tools/site/pages/Documentation.wiki b/tools/site/pages/Documentation.wiki index 79f2cec..9d3a9dc 100644 --- a/tools/site/pages/Documentation.wiki +++ b/tools/site/pages/Documentation.wiki @@ -3,10 +3,12 @@ This is the index of the Open-IOV wiki. If you are new, read '''[[Project Status]]''' first, then '''[[System Overview]]'''. The wiki is a curated public view of the project's maintained documentation; each page names its canonical document in the [https://git.vgpu.io/open-iov.git/ repository]. == Start here == +* [[Project Goals]] — what Open-IOV is for, and the status of each goal +* [[Project Plan]] — the phases, gates, workstreams and what is next * [[Project Status]] — what works today, what is in progress, what is not started * [[Install and First Run]] — build, load, and a stock guest driver in about half an hour * [[System Overview]] — the architecture in one page -* [[FAQ]] — common questions, including "why is nvidia-smi working but CUDA not?" +* [[FAQ]] — common questions, including the current state of CUDA and 3D == Architecture == * [[System Overview]] — principle, layer map, data flows @@ -19,10 +21,18 @@ This is the index of the Open-IOV wiki. If you are new, read '''[[Project Status * [[GMMU Multi-Tenant Protection]] — isolation design for multiple VMs and tenants * [[Physical GPU Backend and Guest Wiring]] — how guest work reaches real hardware * [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA, with real control IDs and log signatures +* [[Performance]] — the cost model, tracing, gaps and improvement programme +* [[Interfaces and Stability]] — which surfaces are contracts, which may change == CUDA == * [[CUDA Userspace Driver]] — the <code>libcuda.so</code> deep dive -* [[The Execution Frontier]] — the GR compute-launch chain and its current wall +* [[The Execution Frontier]] — the GR compute-launch chain and its leading edge + +== Graphics == +* [[Graphics Execution]] — how OpenGL/Vulkan draw commands reach the physical engines +* [[OpenGL]] — the stock guest GL stack on the emulated vGPU +* [[Vulkan]] — the guest Vulkan pipeline on the emulated vGPU +* [[DirectX]] — the Windows/DirectX target == Firmware interoperability == * [[GSP 570.144W Firmware Analysis]] — how the 28.5 MB image was decompiled and mapped diff --git a/tools/site/pages/Main-Page.wiki b/tools/site/pages/Main-Page.wiki index 9a80009..572b365 100644 --- a/tools/site/pages/Main-Page.wiki +++ b/tools/site/pages/Main-Page.wiki @@ -12,13 +12,15 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i </div> == Start here == +* '''[[Project Goals]]''' — what Open-IOV is for, and the status of each goal. +* '''[[Project Plan]]''' — how the work is sequenced, gated, and what is next. * '''[[Project Status]]''' — the honest, dated state of the project. * '''[[Install and First Run]]''' — fresh host to a stock guest driver in about half an hour. * '''[[System Overview]]''' — how Open-IOV works, in one page. * '''[[Documentation]]''' — the full index of this wiki. * '''[[FAQ]]''' — the questions we get asked most. -== Project status (2026-10-12) == +== Project status (2026-10-30) == {| class="wikitable" ! Area !! State |- @@ -30,15 +32,24 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i |- | CUDA userspace init and the driver-API smoke (ALL OK, 64-byte payload round-trips) || Working |- -| Guest CUDA kernel execution on real hardware || In progress — a forwarded launch reaches the engine queue; engine execution is the frontier +| Guest CUDA kernel execution on real hardware || Working — vector-add, matrix-multiply and transformer kernels run green on fresh boots; the battery's staged-transfer class and per-guest isolation are in progress |- -| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet +| 3D graphics: Xorg + OpenGL, Vulkan || In progress — the X server and GLX run on the emulated vGPU, the engine executes real GL method streams, and the guest Vulkan pipeline runs end to end; rendered (non-black) pixels are the current frontier — see [[Graphics Execution]], [[OpenGL]], [[Vulkan]] +|- +| Windows guests, multi-tenancy, migration, performance || Not yet — multiple mediated devices (different emulated VRAM sizes) and multiple guests already coexist; per-guest execution isolation is in progress |} The full story, with evidence and the current wall: [[Project Status]] and [[The Execution Frontier]]. +== Graphics == +* [[Graphics Execution]] — how guest 3D work reaches the physical engines +* [[OpenGL]] — the stock guest GL stack on the emulated vGPU +* [[Vulkan]] — the guest Vulkan pipeline on the emulated vGPU +* [[DirectX]] — the Windows/DirectX target + == Deep dives == * [[System Overview]] — the architecture before any subsystem +* [[Project Goals]] — mission, scope and goal-by-goal status * [[GSP and GSP-RM]] — the firmware interface we emulate * [[The vGSP Execution Plane (vEP)]] — how guest work becomes execution and completion * [[The Host Module (vgsp.ko)]] — the module internals @@ -48,6 +59,8 @@ The full story, with evidence and the current wall: [[Project Status]] and [[The * [[UVM in the Guest]] — unified memory as a correctness oracle * [[Physical GPU Backend and Guest Wiring]] — how work reaches real hardware * [[The Execution Frontier]] — the leading edge, evidence-led +* [[Performance]] — the cost model, tracing and improvement programme +* [[Interfaces and Stability]] — which surfaces are contracts, which may change * [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA with real IDs and log signatures == Firmware interoperability == diff --git a/tools/site/pages/MediaWiki-Sidebar.wiki b/tools/site/pages/MediaWiki-Sidebar.wiki index 4e6ab70..0085b42 100644 --- a/tools/site/pages/MediaWiki-Sidebar.wiki +++ b/tools/site/pages/MediaWiki-Sidebar.wiki @@ -1,5 +1,7 @@ * navigation ** mainpage|mainpage-description +** Project Goals +** Project Plan ** Project Status ** Documentation ** FAQ @@ -15,6 +17,13 @@ ** UVM in the Guest ** Physical GPU Backend and Guest Wiring ** The Execution Frontier +** Performance +** Interfaces and Stability +* Graphics +** Graphics Execution +** OpenGL +** Vulkan +** DirectX * Project ** Open-IOV:About|About Open-IOV ** Install and First Run diff --git a/tools/site/pages/Open-IOV-About.wiki b/tools/site/pages/Open-IOV-About.wiki index 0195c91..5fcad03 100644 --- a/tools/site/pages/Open-IOV-About.wiki +++ b/tools/site/pages/Open-IOV-About.wiki @@ -1,6 +1,6 @@ {{About message}} -'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs. It runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with GPU compute (CUDA), and eventually 3D graphics and video, on hardware you already own: without a proprietary vGPU license, without patching the guest driver, the host driver, the hypervisor or the GPU firmware. +'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs. It runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with GPU compute (CUDA) working today, 3D graphics in active development, and video on the roadmap, on hardware you already own: without a proprietary vGPU license, without patching the guest driver, the host driver, the hypervisor or the GPU firmware. == Mission == Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without modifying NVIDIA software, firmware or hardware. @@ -10,7 +10,7 @@ The only component the project adds is a host kernel module (<code>vgsp.ko</code ::'''''Emulate the interface. Pass through the work. Never patch anyone.''''' -See [[System Overview]] for the architecture and [[Project Status]] for the honest current state. +See [[System Overview]] for the architecture, [[Project Goals]] for the mission and the status of each goal, and [[Project Status]] for the honest current state. == Principles == * '''No patches anywhere.''' Guest drivers, host drivers, firmware and hypervisors are all used unmodified; interoperability is the goal, not counterfeiting. diff --git a/tools/site/titles.json b/tools/site/titles.json index b193342..ed2c588 100644 --- a/tools/site/titles.json +++ b/tools/site/titles.json @@ -16,6 +16,10 @@ "vGPU-IO-Integration-Guide.md": "VGPU-IO Integration Guide", "WPR-Management.md": "WPR Management", "Project-Status.md": "Project Status", + "Project-Goals.md": "Project Goals", + "Project-Plan.md": "Project Plan", + "Performance.md": "Performance", + "Interfaces-and-Stability.md": "Interfaces and Stability", "Install-and-First-Run.md": "Install and First Run", "FAQ.md": "FAQ", "System-Overview.md": "System Overview", @@ -23,6 +27,10 @@ "Glossary.md": "Glossary", "Reference-Glossary.md": "Glossary", "Execution-Frontier.md": "The Execution Frontier", + "Graphics-Execution.md": "Graphics Execution", + "OpenGL.md": "OpenGL", + "Vulkan.md": "Vulkan", + "DirectX.md": "DirectX", "Host-Module.md": "The Host Module (vgsp.ko)", "Guest-Stack-and-CUDASW.md": "The Guest Stack and CUDASW", "UVM-in-the-Guest.md": "UVM in the Guest", @@ -33,7 +41,9 @@ "HAAEEP-and-Collaboration.md": "HAAEEP and Collaboration" }, "mermaid_images": { - "Execution-Plane.md": "[[File:The-vGSP-Execution-Plane-diagram.png|frameless|center|upright=2.2|The vGSP execution plane (vEP): one CUDA launch, end to end.]]" + "Execution-Plane.md": "[[File:The-vGSP-Execution-Plane-diagram.png|frameless|center|upright=2.2|The vGSP execution plane (vEP): one CUDA launch, end to end.]]", + "Graphics-Execution.md": "[[File:Open-IOV-graphics-execution-paths.png|frameless|center|upright=2.2|The graphics execution paths: guest OpenGL/Vulkan/CUDA commands to the physical engines, and completions back.]]", + "Vulkan.md": "[[File:Open-IOV-vulkan-submit-path.png|frameless|center|upright=2.2|One Vulkan queue submit end to end on the emulated vGPU.]]" }, "doc_pages": { "docs/README.md": "Documentation", "docs/03-architecture/01-system-overview.md": "System Overview", diff --git a/tools/vk/vk_frame.c b/tools/vk/vk_frame.c new file mode 100644 index 0000000..62548be --- /dev/null +++ b/tools/vk/vk_frame.c @@ -0,0 +1,510 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * vk_frame.c - headless Vulkan render probe for the vGPU guest. + * + * Creates a real Vulkan device on the NVIDIA ICD, runs a compute shader that + * paints an animated plasma into an RGBA8 storage image, copies the image to + * a host-visible buffer, saves a PPM, and (optionally) publishes each frame + * into the kvmfr ivshmem BAR in the frame_producer layout so the host can + * display it. No window system involved (pure headless), so it works even + * while the GL/GLX walls are being fixed. + * + * Usage: ./vk_frame [frames] [width] [height] [fps] [outdir] [publish] + * Build: gcc -O2 -o vk_frame vk_frame.c -lvulkan (same glibc as the guest) + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <time.h> +#include <sys/mman.h> +#include <vulkan/vulkan.h> + +#include "vk_plasma_spv.h" + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u + +static VkImageTiling g_tiling = VK_IMAGE_TILING_OPTIMAL; + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +#define VKC(x) \ + do { \ + VkResult r_ = (x); \ + if (r_ != VK_SUCCESS) { \ + printf("VKFAIL %s -> %d (line %d)\n", #x, r_, \ + __LINE__); \ + return 1; \ + } \ + } while (0) + +int main(int argc, char **argv) +{ + if (argc > 7 && atoi(argv[7]) == 1) { + g_tiling = VK_IMAGE_TILING_LINEAR; + printf("TILING linear\n"); + } + + int frames = argc > 1 ? atoi(argv[1]) : 60; + uint32_t W = argc > 2 ? (uint32_t)atoi(argv[2]) : 640; + uint32_t H = argc > 3 ? (uint32_t)atoi(argv[3]) : 480; + int fps = argc > 4 ? atoi(argv[4]) : 10; + const char *outdir = argc > 5 ? argv[5] : NULL; + int publish = argc > 6 ? atoi(argv[6]) : 1; + VkApplicationInfo app = {0}; + VkInstanceCreateInfo ici = {0}; + VkInstance inst; + uint32_t ndev = 0; + VkPhysicalDevice devs[16]; + VkPhysicalDevice phys = VK_NULL_HANDLE; + char picked[256] = ""; + uint32_t nfam = 0, fam = ~0u; + VkQueueFamilyProperties fams[16]; + VkDevice dev; + VkQueue queue; + VkImage image; + VkDeviceMemory img_mem; + VkImageView view; + VkBuffer buf; + VkDeviceMemory buf_mem; + VkShaderModule shader; + VkDescriptorSetLayout dsl; + VkPipelineLayout pl; + VkPipeline pipe; + VkDescriptorPool pool; + VkDescriptorSet dset; + VkCommandPool cpool; + VkCommandBuffer cb; + VkFence fence; + uint32_t i; + float prio = 1.0f; + VkDeviceSize imgsz = (VkDeviceSize)W * H * 4; + unsigned char *pix; + struct frame_hdr *hdr = NULL; + volatile uint32_t *seqp = NULL; + unsigned char *bar = NULL; + int bfd = -1; + int saved = 0; + uint64_t t_first = 0, t_last = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + + app.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO; + app.pApplicationName = "vk_frame"; + app.pEngineName = "vk_frame"; + app.apiVersion = VK_API_VERSION_1_2; + ici.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO; + ici.pApplicationInfo = &app; + VKC(vkCreateInstance(&ici, NULL, &inst)); + printf("VKINSTANCE ok\n"); + + VKC(vkEnumeratePhysicalDevices(inst, &ndev, NULL)); + if (ndev > 16) + ndev = 16; + VKC(vkEnumeratePhysicalDevices(inst, &ndev, devs)); + printf("VKDEVS %u\n", ndev); + for (i = 0; i < ndev; i++) { + VkPhysicalDeviceProperties pp; + + vkGetPhysicalDeviceProperties(devs[i], &pp); + printf("VKDEV %u: %s vendor=0x%04x\n", i, pp.deviceName, + pp.vendorID); + if (pp.vendorID == 0x10de || strstr(pp.deviceName, "NVIDIA")) { + phys = devs[i]; + snprintf(picked, sizeof(picked), "%s", + pp.deviceName); + } + } + if (!phys) { + printf("VK_NO_NVIDIA\n"); + return 2; + } + printf("VKPICK %s\n", picked); + vkGetPhysicalDeviceQueueFamilyProperties(phys, &nfam, NULL); + if (nfam > 16) + nfam = 16; + vkGetPhysicalDeviceQueueFamilyProperties(phys, &nfam, fams); + for (i = 0; i < nfam; i++) + if ((fams[i].queueFlags & VK_QUEUE_COMPUTE_BIT) && + fams[i].queueCount > 0) { + fam = i; + break; + } + if (fam == ~0u) { + printf("VK_NO_COMPUTE_QUEUE\n"); + return 3; + } + { + VkDeviceQueueCreateInfo qci = {0}; + VkDeviceCreateInfo dci = {0}; + + qci.sType = VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO; + qci.queueFamilyIndex = fam; + qci.queueCount = 1; + qci.pQueuePriorities = &prio; + dci.sType = VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO; + dci.queueCreateInfoCount = 1; + dci.pQueueCreateInfos = &qci; + VKC(vkCreateDevice(phys, &dci, NULL, &dev)); + } + printf("VKDEVICE ok family=%u\n", fam); + vkGetDeviceQueue(dev, fam, 0, &queue); + + /* storage image */ + { + VkImageCreateInfo ci = {0}; + VkMemoryRequirements mr; + VkMemoryAllocateInfo ma = {0}; + + ci.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO; + ci.imageType = VK_IMAGE_TYPE_2D; + ci.format = VK_FORMAT_R8G8B8A8_UNORM; + ci.extent.width = W; + ci.extent.height = H; + ci.extent.depth = 1; + ci.mipLevels = 1; + ci.arrayLayers = 1; + ci.samples = VK_SAMPLE_COUNT_1_BIT; + ci.tiling = g_tiling; + ci.usage = VK_IMAGE_USAGE_STORAGE_BIT | + VK_IMAGE_USAGE_TRANSFER_SRC_BIT; + ci.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; + VKC(vkCreateImage(dev, &ci, NULL, &image)); + vkGetImageMemoryRequirements(dev, image, &mr); + ma.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; + ma.allocationSize = mr.size; + /* find a device-local memory type */ + { + VkPhysicalDeviceMemoryProperties mp; + uint32_t k; + + vkGetPhysicalDeviceMemoryProperties(phys, &mp); + for (k = 0; k < mp.memoryTypeCount; k++) + if ((mr.memoryTypeBits & (1u << k)) && + (mp.memoryTypes[k].propertyFlags & + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)) { + ma.memoryTypeIndex = k; + break; + } + } + VKC(vkAllocateMemory(dev, &ma, NULL, &img_mem)); + VKC(vkBindImageMemory(dev, image, img_mem, 0)); + } + { + VkImageViewCreateInfo ci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO; + ci.image = image; + ci.viewType = VK_IMAGE_VIEW_TYPE_2D; + ci.format = VK_FORMAT_R8G8B8A8_UNORM; + ci.subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT; + ci.subresourceRange.levelCount = 1; + ci.subresourceRange.layerCount = 1; + VKC(vkCreateImageView(dev, &ci, NULL, &view)); + } + /* host-visible readback buffer */ + { + VkBufferCreateInfo ci = {0}; + VkMemoryRequirements mr; + VkMemoryAllocateInfo ma = {0}; + + ci.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO; + ci.size = imgsz; + ci.usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT; + ci.sharingMode = VK_SHARING_MODE_EXCLUSIVE; + VKC(vkCreateBuffer(dev, &ci, NULL, &buf)); + vkGetBufferMemoryRequirements(dev, buf, &mr); + ma.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; + ma.allocationSize = mr.size; + { + VkPhysicalDeviceMemoryProperties mp; + uint32_t k; + + vkGetPhysicalDeviceMemoryProperties(phys, &mp); + for (k = 0; k < mp.memoryTypeCount; k++) + if ((mr.memoryTypeBits & (1u << k)) && + (mp.memoryTypes[k].propertyFlags & + (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | + VK_MEMORY_PROPERTY_HOST_COHERENT_BIT))) { + ma.memoryTypeIndex = k; + break; + } + } + { + int rc_ = vkAllocateMemory(dev, &ma, NULL, &buf_mem); + if (rc_ != VK_SUCCESS) { + VkPhysicalDeviceMemoryProperties mp2; + uint32_t k2; + vkGetPhysicalDeviceMemoryProperties(phys, &mp2); + for (k2 = 0; k2 < mp2.memoryTypeCount && rc_ != VK_SUCCESS; k2++) { + if (!(mr.memoryTypeBits & (1u << k2))) + continue; + ma.memoryTypeIndex = k2; + rc_ = vkAllocateMemory(dev, &ma, NULL, &buf_mem); + printf("BUF retry type=%u rc=%d\n", k2, rc_); + } + } + if (rc_ != VK_SUCCESS) { + printf("VKFAIL vkAllocateMemory(buf) -> %d (line %d)\n", rc_, __LINE__); + return 1; + } + } + VKC(vkBindBufferMemory(dev, buf, buf_mem, 0)); + } + /* shader + pipeline */ + { + VkShaderModuleCreateInfo ci = {0}; + VkDescriptorSetLayoutBinding b = {0}; + VkDescriptorSetLayoutCreateInfo dci = {0}; + VkPushConstantRange pcr = {0}; + VkPipelineLayoutCreateInfo plci = {0}; + VkComputePipelineCreateInfo pci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO; + ci.codeSize = vk_plasma_spv_len; + ci.pCode = (const uint32_t *)vk_plasma_spv; + VKC(vkCreateShaderModule(dev, &ci, NULL, &shader)); + b.binding = 0; + b.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE; + b.descriptorCount = 1; + b.stageFlags = VK_SHADER_STAGE_COMPUTE_BIT; + dci.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO; + dci.bindingCount = 1; + dci.pBindings = &b; + VKC(vkCreateDescriptorSetLayout(dev, &dci, NULL, &dsl)); + pcr.stageFlags = VK_SHADER_STAGE_COMPUTE_BIT; + pcr.size = 16; + plci.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO; + plci.setLayoutCount = 1; + plci.pSetLayouts = &dsl; + plci.pushConstantRangeCount = 1; + plci.pPushConstantRanges = &pcr; + VKC(vkCreatePipelineLayout(dev, &plci, NULL, &pl)); + pci.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO; + pci.stage.sType = + VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO; + pci.stage.stage = VK_SHADER_STAGE_COMPUTE_BIT; + pci.stage.module = shader; + pci.stage.pName = "main"; + pci.layout = pl; + VKC(vkCreateComputePipelines(dev, VK_NULL_HANDLE, 1, &pci, + NULL, &pipe)); + } + { + VkDescriptorPoolSize ps = {0}; + VkDescriptorPoolCreateInfo pci = {0}; + VkDescriptorSetAllocateInfo ai = {0}; + VkDescriptorImageInfo ii = {0}; + VkWriteDescriptorSet w = {0}; + + ps.type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE; + ps.descriptorCount = 1; + pci.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO; + pci.maxSets = 1; + pci.poolSizeCount = 1; + pci.pPoolSizes = &ps; + VKC(vkCreateDescriptorPool(dev, &pci, NULL, &pool)); + ai.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO; + ai.descriptorPool = pool; + ai.descriptorSetCount = 1; + ai.pSetLayouts = &dsl; + VKC(vkAllocateDescriptorSets(dev, &ai, &dset)); + ii.imageView = view; + ii.imageLayout = VK_IMAGE_LAYOUT_GENERAL; + w.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET; + w.dstSet = dset; + w.dstBinding = 0; + w.descriptorCount = 1; + w.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE; + w.pImageInfo = ⅈ + vkUpdateDescriptorSets(dev, 1, &w, 0, NULL); + } + { + VkCommandPoolCreateInfo ci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO; + ci.queueFamilyIndex = fam; + VKC(vkCreateCommandPool(dev, &ci, NULL, &cpool)); + VkCommandBufferAllocateInfo ai = {0}; + + ai.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO; + ai.commandPool = cpool; + ai.level = VK_COMMAND_BUFFER_LEVEL_PRIMARY; + ai.commandBufferCount = 1; + VKC(vkAllocateCommandBuffers(dev, &ai, &cb)); + } + { + VkFenceCreateInfo ci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_FENCE_CREATE_INFO; + VKC(vkCreateFence(dev, &ci, NULL, &fence)); + } + if (publish) { + const char *bp = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + + bfd = open(bp, O_RDWR | O_SYNC); + if (bfd >= 0) { + bar = mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, bfd, 0); + if (bar != MAP_FAILED) { + hdr = (struct frame_hdr *)(bar + + FRAME_OFFSET); + seqp = &hdr->seq; + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = W; + hdr->height = H; + hdr->stride = W * 4; + hdr->format = 0; + printf("PUBLISH on\n"); + } else { + bar = NULL; + } + } + if (!bar) + printf("PUBLISH off (BAR open/mmap failed)\n"); + } + VKC(vkMapMemory(dev, buf_mem, 0, imgsz, 0, (void **)&pix)); + + for (i = 0; i < (uint32_t)frames; i++) { + VkCommandBufferBeginInfo bi = {0}; + VkImageMemoryBarrier ib = {0}; + VkBufferImageCopy bc = {0}; + VkBufferMemoryBarrier bb = {0}; + float pc[4]; + uint64_t t0 = now_ns(); + VkSubmitInfo si = {0}; + uint32_t gw = (W + 15) / 16, gh = (H + 15) / 16; + + VKC(vkResetCommandBuffer(cb, 0)); + bi.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO; + VKC(vkBeginCommandBuffer(cb, &bi)); + ib.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER; + ib.srcAccessMask = 0; + ib.dstAccessMask = VK_ACCESS_SHADER_WRITE_BIT; + ib.oldLayout = VK_IMAGE_LAYOUT_UNDEFINED; + ib.newLayout = VK_IMAGE_LAYOUT_GENERAL; + ib.image = image; + ib.subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT; + ib.subresourceRange.levelCount = 1; + ib.subresourceRange.layerCount = 1; + vkCmdPipelineBarrier(cb, + VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, + VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 0, + NULL, 0, NULL, 1, &ib); + vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, pipe); + vkCmdBindDescriptorSets(cb, VK_PIPELINE_BIND_POINT_COMPUTE, pl, + 0, 1, &dset, 0, NULL); + pc[0] = (float)i * 0.1f; + pc[1] = (float)W; + pc[2] = (float)H; + pc[3] = 0.0f; + vkCmdPushConstants(cb, pl, VK_SHADER_STAGE_COMPUTE_BIT, 0, 16, + pc); + vkCmdDispatch(cb, gw, gh, 1); + ib.srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT; + ib.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT; + ib.oldLayout = VK_IMAGE_LAYOUT_GENERAL; + ib.newLayout = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL; + vkCmdPipelineBarrier(cb, + VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, NULL, + 0, NULL, 1, &ib); + bc.imageSubresource.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT; + bc.imageSubresource.layerCount = 1; + bc.imageExtent.width = W; + bc.imageExtent.height = H; + bc.imageExtent.depth = 1; + vkCmdCopyImageToBuffer(cb, image, + VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, + buf, 1, &bc); + bb.sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER; + bb.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT; + bb.dstAccessMask = VK_ACCESS_HOST_READ_BIT; + bb.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED; + bb.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED; + bb.buffer = buf; + bb.size = VK_WHOLE_SIZE; + vkCmdPipelineBarrier(cb, VK_PIPELINE_STAGE_TRANSFER_BIT, + VK_PIPELINE_STAGE_HOST_BIT, 0, 0, NULL, 1, + &bb, 0, NULL); + VKC(vkEndCommandBuffer(cb)); + si.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO; + si.commandBufferCount = 1; + si.pCommandBuffers = &cb; + VKC(vkResetFences(dev, 1, &fence)); + VKC(vkQueueSubmit(queue, 1, &si, fence)); + VKC(vkWaitForFences(dev, 1, &fence, VK_TRUE, 30000000000ull)); + vkUnmapMemory(dev, buf_mem); + vkMapMemory(dev, buf_mem, 0, imgsz, 0, (void **)&pix); + + if (outdir) { + char path[512]; + FILE *f; + + snprintf(path, sizeof(path), "%s/vk-%04d.ppm", + outdir, i + 1); + f = fopen(path, "wb"); + if (f) { + unsigned char *d = malloc((size_t)W * H * 3); + uint32_t p; + + for (p = 0; p < W * H; p++) { + d[p * 3 + 0] = pix[p * 4 + 0]; + d[p * 3 + 1] = pix[p * 4 + 1]; + d[p * 3 + 2] = pix[p * 4 + 2]; + } + fprintf(f, "P6\n%u %u\n255\n", W, H); + fwrite(d, 1, (size_t)W * H * 3, f); + fclose(f); + free(d); + saved++; + } + } + if (bar) { + memcpy(bar + FRAME_OFFSET + sizeof(struct frame_hdr), + pix, imgsz); + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + } + if (!t_first) + t_first = t0; + t_last = t0; + if (i < 5 || (i % 30) == 29) + printf("VKFRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + if (t_first && t_last > t_first) + printf("VKFRAME_RATE fps=%.2f frames=%d\n", + (double)(i - 1) * 1e9 / (double)(t_last - t_first), i); + printf("VKFRAME_DONE frames=%d %ux%u saved=%d\n", i, W, H, saved); + vkDeviceWaitIdle(dev); + printf("VKFRAME_IDLE ok\n"); + return 0; +}
\ No newline at end of file diff --git a/tools/vk/vk_plasma.comp b/tools/vk/vk_plasma.comp new file mode 100644 index 0000000..fe6b3ac --- /dev/null +++ b/tools/vk/vk_plasma.comp @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: BSD-3-Clause +// vk_plasma.comp - compute-shader frame painter for the Vulkan guest test. +#version 450 +layout(local_size_x = 16, local_size_y = 16) in; +layout(set = 0, binding = 0, rgba8) uniform writeonly image2D fb; +layout(push_constant) uniform PC { float t; float w; float h; float pad; } pc; +void main() { + ivec2 p = ivec2(gl_GlobalInvocationID.xy); + if (p.x >= int(pc.w) || p.y >= int(pc.h)) + return; + vec2 uv = vec2(p) / vec2(pc.w, pc.h); + float v = sin(uv.x * 12.0 + pc.t * 2.0) * cos(uv.y * 9.0 - pc.t * 1.7); + float r = 0.5 + 0.5 * sin(pc.t + uv.x * 6.2831); + float g = 0.5 + 0.5 * sin(pc.t * 1.3 + uv.y * 6.2831); + float b = 0.5 + 0.5 * v; + imageStore(fb, p, vec4(r, g, b, 1.0)); +} diff --git a/tools/vk/vk_plasma.spv b/tools/vk/vk_plasma.spv Binary files differnew file mode 100644 index 0000000..7c12b51 --- /dev/null +++ b/tools/vk/vk_plasma.spv diff --git a/tools/vk/vk_plasma_spv.h b/tools/vk/vk_plasma_spv.h new file mode 100644 index 0000000..49c16f1 --- /dev/null +++ b/tools/vk/vk_plasma_spv.h @@ -0,0 +1,237 @@ +unsigned char vk_plasma_spv[] = { + 0x03, 0x02, 0x23, 0x07, 0x00, 0x00, 0x01, 0x00, 0x0b, 0x00, 0x08, 0x00, + 0x7c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x11, 0x00, 0x02, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x06, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x47, 0x4c, 0x53, 0x4c, 0x2e, 0x73, 0x74, 0x64, 0x2e, 0x34, 0x35, 0x30, + 0x00, 0x00, 0x00, 0x00, 0x0e, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x06, 0x00, 0x05, 0x00, 0x00, 0x00, + 0x04, 0x00, 0x00, 0x00, 0x6d, 0x61, 0x69, 0x6e, 0x00, 0x00, 0x00, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x10, 0x00, 0x06, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x11, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x03, 0x00, 0x03, 0x00, 0x02, 0x00, 0x00, 0x00, + 0xc2, 0x01, 0x00, 0x00, 0x05, 0x00, 0x04, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x6d, 0x61, 0x69, 0x6e, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x70, 0x00, 0x00, 0x00, 0x05, 0x00, 0x08, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x67, 0x6c, 0x5f, 0x47, 0x6c, 0x6f, 0x62, 0x61, + 0x6c, 0x49, 0x6e, 0x76, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x49, + 0x44, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x50, 0x43, 0x00, 0x00, 0x06, 0x00, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x74, 0x00, 0x00, 0x00, 0x06, 0x00, 0x04, 0x00, + 0x18, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x77, 0x00, 0x00, 0x00, + 0x06, 0x00, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x68, 0x00, 0x00, 0x00, 0x06, 0x00, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x03, 0x00, 0x00, 0x00, 0x70, 0x61, 0x64, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x1a, 0x00, 0x00, 0x00, 0x70, 0x63, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x75, 0x76, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x3c, 0x00, 0x00, 0x00, 0x76, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x53, 0x00, 0x00, 0x00, 0x72, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x5f, 0x00, 0x00, 0x00, 0x67, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x6b, 0x00, 0x00, 0x00, 0x62, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x66, 0x62, 0x00, 0x00, 0x47, 0x00, 0x04, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x47, 0x00, 0x03, 0x00, 0x18, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x48, 0x00, 0x05, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x23, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x48, 0x00, 0x05, 0x00, + 0x18, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x23, 0x00, 0x00, 0x00, + 0x04, 0x00, 0x00, 0x00, 0x48, 0x00, 0x05, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x23, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, + 0x48, 0x00, 0x05, 0x00, 0x18, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x23, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x47, 0x00, 0x03, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x47, 0x00, 0x04, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x47, 0x00, 0x04, 0x00, 0x71, 0x00, 0x00, 0x00, 0x22, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x47, 0x00, 0x04, 0x00, 0x7b, 0x00, 0x00, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x13, 0x00, 0x02, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x21, 0x00, 0x03, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x15, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x08, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x15, 0x00, 0x04, 0x00, 0x0a, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x0c, 0x00, 0x00, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, + 0x0e, 0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x14, 0x00, 0x02, 0x00, 0x12, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, + 0x0a, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x14, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x06, 0x00, 0x00, 0x00, 0x16, 0x00, 0x03, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x1e, 0x00, 0x06, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x20, 0x00, 0x04, 0x00, 0x19, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, + 0x19, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, + 0x2b, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x1b, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x20, 0x00, 0x04, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, + 0x0a, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x2b, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x27, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x20, 0x00, 0x04, 0x00, + 0x31, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x3f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x41, 0x2b, 0x00, 0x04, 0x00, + 0x06, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x44, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x40, 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x4a, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x41, 0x2b, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x4e, 0x00, 0x00, 0x00, 0x9a, 0x99, 0xd9, 0x3f, + 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x3f, 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x59, 0x00, 0x00, 0x00, 0x28, 0x0f, 0xc9, 0x40, 0x2b, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x62, 0x00, 0x00, 0x00, 0x66, 0x66, 0xa6, 0x3f, + 0x19, 0x00, 0x09, 0x00, 0x6f, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x70, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x6f, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x70, 0x00, 0x00, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x77, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x3f, + 0x17, 0x00, 0x04, 0x00, 0x78, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x04, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, 0x0a, 0x00, 0x00, 0x00, + 0x7a, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x2c, 0x00, 0x06, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x7b, 0x00, 0x00, 0x00, 0x7a, 0x00, 0x00, 0x00, + 0x7a, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x36, 0x00, 0x05, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x03, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, + 0x3b, 0x00, 0x04, 0x00, 0x08, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x31, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, + 0x3b, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x3b, 0x00, 0x04, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x53, 0x00, 0x00, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x3b, 0x00, 0x00, 0x00, + 0x5f, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, + 0x3b, 0x00, 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x07, 0x00, 0x0e, 0x00, 0x00, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x7c, 0x00, 0x04, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x11, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, + 0x3e, 0x00, 0x03, 0x00, 0x09, 0x00, 0x00, 0x00, 0x11, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x14, 0x00, 0x00, 0x00, 0x15, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x06, 0x00, 0x00, 0x00, 0x16, 0x00, 0x00, 0x00, 0x15, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x1d, 0x00, 0x00, 0x00, + 0x1a, 0x00, 0x00, 0x00, 0x1b, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x1d, 0x00, 0x00, 0x00, + 0x6e, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x1f, 0x00, 0x00, 0x00, + 0x1e, 0x00, 0x00, 0x00, 0xaf, 0x00, 0x05, 0x00, 0x12, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x16, 0x00, 0x00, 0x00, 0x1f, 0x00, 0x00, 0x00, + 0xa8, 0x00, 0x04, 0x00, 0x12, 0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0xf7, 0x00, 0x03, 0x00, 0x23, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0xfa, 0x00, 0x04, 0x00, 0x21, 0x00, 0x00, 0x00, + 0x22, 0x00, 0x00, 0x00, 0x23, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x02, 0x00, + 0x22, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, 0x14, 0x00, 0x00, 0x00, + 0x25, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x26, 0x00, 0x00, 0x00, + 0x25, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x28, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x27, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x29, 0x00, 0x00, 0x00, + 0x28, 0x00, 0x00, 0x00, 0x6e, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, + 0x2a, 0x00, 0x00, 0x00, 0x29, 0x00, 0x00, 0x00, 0xaf, 0x00, 0x05, 0x00, + 0x12, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x00, 0x00, 0x26, 0x00, 0x00, 0x00, + 0x2a, 0x00, 0x00, 0x00, 0xf9, 0x00, 0x02, 0x00, 0x23, 0x00, 0x00, 0x00, + 0xf8, 0x00, 0x02, 0x00, 0x23, 0x00, 0x00, 0x00, 0xf5, 0x00, 0x07, 0x00, + 0x12, 0x00, 0x00, 0x00, 0x2c, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00, + 0x05, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x00, 0x00, 0x22, 0x00, 0x00, 0x00, + 0xf7, 0x00, 0x03, 0x00, 0x2e, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0xfa, 0x00, 0x04, 0x00, 0x2c, 0x00, 0x00, 0x00, 0x2d, 0x00, 0x00, 0x00, + 0x2e, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x02, 0x00, 0x2d, 0x00, 0x00, 0x00, + 0xfd, 0x00, 0x01, 0x00, 0xf8, 0x00, 0x02, 0x00, 0x2e, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x07, 0x00, 0x00, 0x00, 0x33, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x6f, 0x00, 0x04, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x34, 0x00, 0x00, 0x00, 0x33, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x35, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x1b, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x36, 0x00, 0x00, 0x00, 0x35, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x27, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x38, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x50, 0x00, 0x05, 0x00, + 0x30, 0x00, 0x00, 0x00, 0x39, 0x00, 0x00, 0x00, 0x36, 0x00, 0x00, 0x00, + 0x38, 0x00, 0x00, 0x00, 0x88, 0x00, 0x05, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x3a, 0x00, 0x00, 0x00, 0x34, 0x00, 0x00, 0x00, 0x39, 0x00, 0x00, 0x00, + 0x3e, 0x00, 0x03, 0x00, 0x32, 0x00, 0x00, 0x00, 0x3a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, + 0x3e, 0x00, 0x00, 0x00, 0x3f, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x43, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, 0x43, 0x00, 0x00, 0x00, + 0x44, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x46, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, + 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, 0x47, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00, 0x00, 0x46, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x48, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x49, 0x00, 0x00, 0x00, 0x48, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x4b, 0x00, 0x00, 0x00, + 0x49, 0x00, 0x00, 0x00, 0x4a, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x4c, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x4d, 0x00, 0x00, 0x00, 0x4c, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x4d, 0x00, 0x00, 0x00, + 0x4e, 0x00, 0x00, 0x00, 0x83, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x50, 0x00, 0x00, 0x00, 0x4b, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, + 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0e, 0x00, 0x00, 0x00, 0x50, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x52, 0x00, 0x00, 0x00, + 0x47, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, + 0x3c, 0x00, 0x00, 0x00, 0x52, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x55, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x56, 0x00, 0x00, 0x00, 0x55, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x3b, 0x00, 0x00, 0x00, 0x57, 0x00, 0x00, 0x00, 0x32, 0x00, 0x00, 0x00, + 0x13, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x58, 0x00, 0x00, 0x00, 0x57, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x5a, 0x00, 0x00, 0x00, 0x58, 0x00, 0x00, 0x00, + 0x59, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x5b, 0x00, 0x00, 0x00, 0x56, 0x00, 0x00, 0x00, 0x5a, 0x00, 0x00, 0x00, + 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, 0x5c, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00, 0x00, 0x5b, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x5d, 0x00, 0x00, 0x00, + 0x54, 0x00, 0x00, 0x00, 0x5c, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x5e, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, + 0x5d, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, 0x53, 0x00, 0x00, 0x00, + 0x5e, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x60, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x61, 0x00, 0x00, 0x00, + 0x60, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x63, 0x00, 0x00, 0x00, 0x61, 0x00, 0x00, 0x00, 0x62, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x65, 0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x66, 0x00, 0x00, 0x00, + 0x65, 0x00, 0x00, 0x00, 0x59, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x67, 0x00, 0x00, 0x00, 0x63, 0x00, 0x00, 0x00, + 0x66, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x68, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00, 0x00, + 0x67, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x69, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, 0x68, 0x00, 0x00, 0x00, + 0x81, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x6a, 0x00, 0x00, 0x00, + 0x54, 0x00, 0x00, 0x00, 0x69, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, + 0x5f, 0x00, 0x00, 0x00, 0x6a, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x6c, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x6d, 0x00, 0x00, 0x00, + 0x54, 0x00, 0x00, 0x00, 0x6c, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x6e, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, + 0x6d, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, 0x6b, 0x00, 0x00, 0x00, + 0x6e, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x6f, 0x00, 0x00, 0x00, + 0x72, 0x00, 0x00, 0x00, 0x71, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x73, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x74, 0x00, 0x00, 0x00, + 0x53, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x75, 0x00, 0x00, 0x00, 0x5f, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x76, 0x00, 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, + 0x50, 0x00, 0x07, 0x00, 0x78, 0x00, 0x00, 0x00, 0x79, 0x00, 0x00, 0x00, + 0x74, 0x00, 0x00, 0x00, 0x75, 0x00, 0x00, 0x00, 0x76, 0x00, 0x00, 0x00, + 0x77, 0x00, 0x00, 0x00, 0x63, 0x00, 0x04, 0x00, 0x72, 0x00, 0x00, 0x00, + 0x73, 0x00, 0x00, 0x00, 0x79, 0x00, 0x00, 0x00, 0xfd, 0x00, 0x01, 0x00, + 0x38, 0x00, 0x01, 0x00 +}; +unsigned int vk_plasma_spv_len = 2800; diff --git a/tools/vk/vk_probe.py b/tools/vk/vk_probe.py new file mode 100644 index 0000000..3113c3f --- /dev/null +++ b/tools/vk/vk_probe.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +# vk_probe.py - enumerate real Vulkan physical devices via the loader (ctypes, +# no dev headers needed). Prints one DEV line per physical device. +import ctypes +import sys + +VK_SUCCESS = 0 +VK_STRUCTURE_TYPE_APPLICATION_INFO = 0 +VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO = 1 +VK_API_VERSION_1_2 = (1 << 22) | (2 << 12) + +try: + vulkan = ctypes.CDLL("libvulkan.so.1") +except OSError as e: + print("LOAD_FAIL %s" % e) + sys.exit(1) + + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), + ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), + ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32), + ] + + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("pApplicationInfo", ctypes.POINTER(VkApplicationInfo)), + ("enabledLayerCount", ctypes.c_uint32), + ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), + ("ppEnabledExtensionNames", ctypes.c_void_p), + ] + + +app = VkApplicationInfo( + VK_STRUCTURE_TYPE_APPLICATION_INFO, None, b"vk_probe", 1, b"vk_probe", 1, + VK_API_VERSION_1_2) +ci = VkInstanceCreateInfo( + VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, None, 0, ctypes.pointer(app), 0, + None, 0, None) +inst = ctypes.c_void_p() +r = vulkan.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst)) +print("vkCreateInstance rc=%d (0x%x)" % (r, r & 0xfffffffff if r < 0 else r)) +if r != VK_SUCCESS: + sys.exit(1) + +n = ctypes.c_uint32(0) +r = vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +print("vkEnumeratePhysicalDevices rc=%d count=%d" % (r, n.value)) +if r != VK_SUCCESS or n.value == 0: + sys.exit(2) + +devs = (ctypes.c_void_p * n.value)() +r = vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) +ok = 0 +for i in range(n.value): + buf = ctypes.create_string_buffer(4096) + vulkan.vkGetPhysicalDeviceProperties(devs[i], buf) + raw = buf.raw + api = int.from_bytes(raw[0:4], "little") + drv = int.from_bytes(raw[4:8], "little") + vendor = int.from_bytes(raw[8:12], "little") + devid = int.from_bytes(raw[12:16], "little") + dtype = int.from_bytes(raw[16:20], "little") + name = raw[20:20 + 256].split(b"\x00")[0].decode("utf-8", "replace") + print("DEV %d: %s vendor=0x%04x device=0x%04x type=%d api=%d.%d.%d drv=%d" + % (i, name, vendor, devid, dtype, api >> 22, + (api >> 12) & 0x3ff, api & 0xfff, drv)) + if vendor == 0x10de: + ok = 1 +print("VK_NVIDIA=%d" % ok) +sys.exit(0 if ok else 3)
\ No newline at end of file diff --git a/tools/vk/vk_probe2.py b/tools/vk/vk_probe2.py new file mode 100644 index 0000000..557b888 --- /dev/null +++ b/tools/vk/vk_probe2.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +# vk_probe2.py - Vulkan probe stage 2: create a logical device + queue on the +# NVIDIA physical device. Prints the queue-family table and the create +# result. ctypes only, no dev headers. +import ctypes +import sys + +VK_SUCCESS = 0 +VK_STRUCTURE_TYPE_APPLICATION_INFO = 0 +VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO = 1 +VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO = 2 +VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO = 3 +VK_API_VERSION_1_2 = (1 << 22) | (2 << 12) + +vulkan = ctypes.CDLL("libvulkan.so.1") + + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), + ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), + ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32), + ] + + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("pApplicationInfo", ctypes.POINTER(VkApplicationInfo)), + ("enabledLayerCount", ctypes.c_uint32), + ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), + ("ppEnabledExtensionNames", ctypes.c_void_p), + ] + + +class VkQueueFamilyProperties(ctypes.Structure): + _fields_ = [ + ("queueFlags", ctypes.c_uint32), + ("queueCount", ctypes.c_uint32), + ("timestampValidBits", ctypes.c_uint32), + ("minImageTransferGranularity", ctypes.c_uint8 * 12), + ] + + +class VkDeviceQueueCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("queueFamilyIndex", ctypes.c_uint32), + ("queueCount", ctypes.c_uint32), + ("pQueuePriorities", ctypes.POINTER(ctypes.c_float)), + ] + + +class VkDeviceCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("queueCreateInfoCount", ctypes.c_uint32), + ("pQueueCreateInfos", ctypes.POINTER(VkDeviceQueueCreateInfo)), + ("enabledLayerCount", ctypes.c_uint32), + ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), + ("ppEnabledExtensionNames", ctypes.c_void_p), + ("pEnabledFeatures", ctypes.c_void_p), + ] + + +app = VkApplicationInfo(VK_STRUCTURE_TYPE_APPLICATION_INFO, None, b"vk_probe2", + 1, b"vk_probe2", 1, VK_API_VERSION_1_2) +ci = VkInstanceCreateInfo(VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, None, 0, + ctypes.pointer(app), 0, None, 0, None) +inst = ctypes.c_void_p() +r = vulkan.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst)) +print("INSTANCE rc=%d" % r) +if r != 0: + sys.exit(1) + +n = ctypes.c_uint32(0) +vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +devs = (ctypes.c_void_p * n.value)() +vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) + +# pick the NVIDIA device by name +best = None +for i in range(n.value): + buf = ctypes.create_string_buffer(4096) + vulkan.vkGetPhysicalDeviceProperties(devs[i], buf) + raw = buf.raw + name = raw[20:276].split(b"\x00")[0].decode("utf-8", "replace") + print("PROPS dev=%d head=%s name=%r" % (i, raw[:40].hex(), name)) + if "NVIDIA" in name or "llvmpipe" in name: + if "NVIDIA" in name: + best = devs[i] + print("PICK NVIDIA dev=%d name=%s" % (i, name)) + break +if best is None: + print("NO_NVIDIA") + sys.exit(2) + +nfam = ctypes.c_uint32(0) +vulkan.vkGetPhysicalDeviceQueueFamilyProperties(best, ctypes.byref(nfam), + None) +fams = (VkQueueFamilyProperties * nfam.value)() +vulkan.vkGetPhysicalDeviceQueueFamilyProperties(best, ctypes.byref(nfam), + fams) +gfam = None +for j in range(nfam.value): + print("QFAM %d flags=0x%x count=%d" % (j, fams[j].queueFlags, + fams[j].queueCount)) + if (fams[j].queueFlags & 0x1) and fams[j].queueCount > 0 and gfam is None: + gfam = j +if gfam is None: + print("NO_GRAPHICS_QUEUE") + sys.exit(3) +prio = ctypes.c_float(1.0) +qci = VkDeviceQueueCreateInfo(VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO, + None, 0, gfam, 1, ctypes.pointer(prio)) +dci = VkDeviceCreateInfo(VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, None, 0, 1, + ctypes.pointer(qci), 0, None, 0, None, None) +dev = ctypes.c_void_p() +r = vulkan.vkCreateDevice(best, ctypes.byref(dci), None, ctypes.byref(dev)) +print("vkCreateDevice rc=%d (0x%x)" % (r, r if r < 0 else 0)) +if r != 0: + sys.exit(4) +queue = ctypes.c_void_p() +vulkan.vkGetDeviceQueue(dev, gfam, 0, ctypes.byref(queue)) +print("QUEUE device=%s queue=%s" % (bool(dev), bool(queue))) +# optional: wait-idle (exercises the queue) +vulkan.vkDeviceWaitIdle(dev) +print("VK2_DEVICE_OK") +sys.exit(0)
\ No newline at end of file diff --git a/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md b/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md new file mode 100644 index 0000000..48e9f35 --- /dev/null +++ b/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md @@ -0,0 +1,156 @@ +# ANALYSIS - EGL 3D-engine selection wall (570.144 guest userspace) + +Status: OPEN (2026-10-23). The EGL-device path initialises (devices enumerate, +`eglInitialize` succeeds, `vendor=NVIDIA`), but `eglMakeCurrent` fails with +`EGL_BAD_ALLOC (0x3003)` because the per-device EGL-core client creator +returns NULL after a lower-layer engine-init call fails. + +This document records the *shipped-binary* evidence only (disassembly + +live gdb traces). Method: run the probe under gdb with ASLR disabled +(gdb's default) so library offsets are stable, set breakpoints at the +recorded offsets, and read the arguments/returns. + +## 1. The failing chain (libnvidia-eglcore.so.570.144) + +``` +eglMakeCurrent + -> NvGlEglApiInit family: per-device EGL-core client creator (+0xa246f0) + -> +0xa06ed0 (device/engine selection wrapper) + -> +0x9fe790 (engine-list walk) + list head: *( *(device + 0x28738) + 8 ) + filters per engine: engine[0x59] & 0x40 + engine[0x70] == 0 + per-engine callback: *0x70( *(*(engine+0x28738)+0x10) ) + -> +0xa22000 (callback) + rbx = *(reqs + 0xa0) (device object) + mode = *(reqs + 0x20) + mode 6 -> +0xa09830 + mode 7 -> +0xa0a4d0 + else -> +0x9fdfd0(device, engine, obj, reqs) + NULL -> EGL_BAD_ALLOC +``` + +`+0x9fdfd0` ends in the rejection site: + +``` +9fe146: rax = *(r14 + 0x10) ; r14 = *(engine + 0x28738) +9fe14d: rdi = *(rbx + 0x20) ; rbx = *(reqs + 0xa0) (device object) +9fe151: call *0x30(rax) ; vtable slot 0x30 +9fe154: test eax,eax +9fe156: je 9fe170 ; 0 == continue (success) +9fe158: free(tmp); return 0 ; non-zero == reject +``` + +## 2. The rejecting method (libnvidia-glsi.so.570.144) + +The vtable at `*(*(engine+0x28738)+0x10)` is a *glsi* device-engine vtable; +its slot 0x30 resolves (live gdb: vtable `0x7ffff71ddfe0`, method +`+0xa29440` in eglcore terms = glsi `+0x5f480` base-adjusted) to the glsi +engine-init entry: + +``` +glsi +0x5f480(device, reqs): + 5f4c0: fd = *(*(*(dev+0x108)+0x58)) ; int fd (nvidia device) + 5f4c8: rsi = *(dev+0x118) + 5f4dc: call +0x1aba0(fd, rsi, reqs) ; FIRST call + 5f52e: call +0x1af20(...) ; second call +``` + +Live trace (probe `egl_frame` under gdb, breakpoint at the callback's call +site): the **first** glsi call returns `0xee00001`; the caller propagates it +and the engine is rejected. `0xee00001` is a glsi-internal status produced by +two sites (`glsi +0x31f84`, `+0x33910`) that set flag bit `0x20` in the +device's `+0xb84` word and return "busy"-style; the engine-init state machine +(`glsi +0x301b0` and callees `+0x2fbe0` -> `+0x29760` -> `+0x32110`/`+0x2dc70`) +is what fails. + +## 3. What is ruled out + +* **Not an RM-control error.** During the failing probe the emulated GSP + answers every `GSP_RM_CONTROL` with `status=0x0` (including the + `0x20810107` BINAPI query that was previously suspected; that one is a red + herring - a real class-0x2081 object on the live session answers + `NV_OK`/`{0,0}` exactly like the emulation). +* **Not the engine-list shape alone.** The single candidate engine passes the + walk's own filters (`flag59 & 0x40` set, `f70 == 0`), so the list contains + a usable engine; the failure is inside the per-engine init. +* **Not ASLR-dependent.** All offsets above are stable under gdb + (`set disable-randomization on`). + +## 4. Entry-trace update (2026-10-23, r543 boot) + +A batch-gdb entry trace (breakpoints on the glsi engine-init entry and its +inner functions, breakpoint set after `eglInitialize` so the vendor libraries +are mapped; ASLR disabled reproduces glsi base `0x7ffff7e2a000`) captured the +failing call order: + +``` +glsi+0x5f480 (engine-init) + glsi+0x1aba0 (first inner call) + glsi+0x22f30 device lookup by GPU id (miss -> 0xee00002) + glsi+0x2c360 object lookup (miss -> 0xee00003) + glsi+0x33b80 device-flags + state <-- the failing layer + glsi+0x301b0 (state machine, entered twice) + glsi+0x3a6f0 (entered ~7x) + glsi+0x2ac10 (entered twice) + -> EGL_CURRENT_FAIL err=0x3003 (eglcore rejects the engine) +``` + +`+0x5f480` returns as soon as `+0x1aba0` reports non-zero; `+0x1aba0` +propagates the first failure of `+0x22f30` / `+0x2c360` / `+0x33b80`. In the +trace all three were entered, and the failure lands inside `+0x33b80`'s +state-machine call. `+0x33b80` reads the device flags word at `+0xb84`; on a +state-machine failure with the retry bit set it marks the device busy +(`or 0x20` into the state record at `device + idx*0xdc8 + 0x291b4`) and +returns `0xee00001` once (`glsi+0x30ef2` inside its callee `+0x30b20`); +the higher layer reads `+0xb84` bit 0x20 and rejects the engine for this +attempt without retry. `+0x22f30` is called ~1300 times during the probe +(per-device/per-engine enumeration) - the failure is not a lookup miss. + +The state machine (`+0x301b0`) consumes per-engine context records; its +inputs are the fields the caller passes in (`+0x95` mode byte, `+0x38` +record array, `+0x74` id, `+0x98` index) plus the device flags. The next +step is a return-value trace of `+0x3a6f0`/`+0x2ac10` (their failure +conditions) or a differential state dump against a physical-GPU run. + +Note (same day): a control experiment was attempted - Xorg (GLX path) on +the same boot died in `Failed to initialize DMA` / `Failed to allocate push +buffer`, but that boot had a stalled CUDA client holding the device, so the +result is inconclusive; retry on a clean boot. + +## 5. Next steps (for the successor) + +1. Return-value trace of `glsi +0x33b80`'s callees (`+0x301b0`'s + `+0x3a6f0` / `+0x2ac10` results): the first non-zero return there is the + true failing layer. +2. Compare the glsi device-engine state with the same probe on a physical + Turing GPU (the host's own 570 driver runs the identical userspace; a + host-side `egl_frame` run with the real GPU succeeds, so a differential + dump of the engine records / the `+0xb84` flags word will show what the + emulated device is missing). +3. The engine records glsi consumes come from the kernel driver's device + data (engine table + class list). The CUDA path builds the analogous + records from `GET_ENGINES_V2`; the CUDA engine list on this guest is + known to be complete for GR + CE (see the private analysis area). The + glsi path may require an entry (e.g. a specific engine kind or a caps + bit) that the emulated device does not report. + +## 5. Reproduction + +``` +# in the guest (570.144 stock driver): +gcc -O2 -o /tmp/egl_frame /tmp/egl_frame.c -lEGL -lGLESv2 # tools/guest-diag/egl_frame.c +timeout 90 /tmp/egl_frame # EGL_CURRENT_FAIL err=0x3003 + +# under gdb (offsets stable, ASLR disabled): +gdb -q -batch -x egl-gdb11.txt ./egl_frame # catches the vtable call site +``` + +Breakpoints used (no-ASLR addresses with eglcore base 0x7ffff5200000, +glsi base 0x7ffff7e2a000): + +| address | meaning | +|---|---| +| `0x7ffff5bfe151` | eglcore rejection call `*0x30(vtable)` | +| `0x7ffff7e89480` | glsi engine-init entry (+0x5f480) | +| `0x7ffff7e894e1` | after glsi's first inner call (returns 0xee00001) | diff --git a/wiki/.publish-manifest.json b/wiki/.publish-manifest.json index 20e9722..98ed37a 100644 --- a/wiki/.publish-manifest.json +++ b/wiki/.publish-manifest.json @@ -19,11 +19,11 @@ }, "CUDA Userspace Driver": { "file": "CUDA-Userspace-Driver.wiki", - "new_revid": 23963, - "old_revid": 23963, - "source_sha": "31d77a3a672b2943", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:06Z", + "new_revid": 24014, + "old_revid": 23998, + "source_sha": "8703a12fa11674f0", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:25Z", "title": "CUDA Userspace Driver" }, "Community": { @@ -35,13 +35,22 @@ "timestamp": "2026-10-12T04:40:27Z", "title": "Community" }, + "DirectX": { + "file": "DirectX.wiki", + "new_revid": 24007, + "old_revid": null, + "source_sha": "0b5540b83a100148", + "summary": "New page (2026-10-30 wiki refresh): DirectX", + "timestamp": "2026-10-30T17:05:03Z", + "title": "DirectX" + }, "Documentation": { "file": "Documentation.wiki", - "new_revid": 23974, - "old_revid": 23930, - "source_sha": "28a5145b53d94c28", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:25Z", + "new_revid": 24031, + "old_revid": 24026, + "source_sha": "04de62980311a5e4", + "summary": "2026-10-30 wiki refresh: add Project Plan page (phases, gates, workstreams, next steps)", + "timestamp": "2026-10-30T17:12:05Z", "title": "Documentation" }, "Error Codes": { @@ -55,11 +64,11 @@ }, "FAQ": { "file": "FAQ.wiki", - "new_revid": 23968, + "new_revid": 24017, "old_revid": 23968, - "source_sha": "02b90d88b6c94c84", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:08Z", + "source_sha": "c1c84bf09feb6acb", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:29Z", "title": "FAQ" }, "GMMU Multi-Tenant Protection": { @@ -100,11 +109,11 @@ }, "GSP and GSP-RM": { "file": "GSP-and-GSP-RM.wiki", - "new_revid": 23977, - "old_revid": 23943, - "source_sha": "e9bd4bdc8de69c62", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:55Z", + "new_revid": 24010, + "old_revid": 23977, + "source_sha": "0019a098ff128830", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:19Z", "title": "GSP and GSP-RM" }, "Glossary": { @@ -116,31 +125,40 @@ "timestamp": "2026-10-12T04:41:01Z", "title": "Glossary" }, + "Graphics Execution": { + "file": "Graphics-Execution.wiki", + "new_revid": 24004, + "old_revid": null, + "source_sha": "c3418a7220f62d10", + "summary": "New page (2026-10-30 wiki refresh): Graphics Execution", + "timestamp": "2026-10-30T17:04:57Z", + "title": "Graphics Execution" + }, "Guest Driver Lifecycle: A Field Guide": { "file": "Driver-Lifecycle-Field-Guide.wiki", - "new_revid": 23964, + "new_revid": 24019, "old_revid": 23964, - "source_sha": "518c07ce20d3b840", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:06Z", + "source_sha": "f24d66bd99dce76f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:32Z", "title": "Guest Driver Lifecycle: A Field Guide" }, "HAAEEP and Collaboration": { "file": "HAAEEP-and-Collaboration.wiki", - "new_revid": 23979, - "old_revid": 23945, - "source_sha": "cd1d9dfd41b1e6b6", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:56Z", + "new_revid": 24023, + "old_revid": 23979, + "source_sha": "20990217e179badc", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:38Z", "title": "HAAEEP and Collaboration" }, "Hardware and Software Matrix": { "file": "Hardware-Software-Matrix.wiki", - "new_revid": 23980, - "old_revid": 23946, - "source_sha": "899c86d3ae612537", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:57Z", + "new_revid": 24022, + "old_revid": 23980, + "source_sha": "fc7ff074320feafb", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:37Z", "title": "Hardware and Software Matrix" }, "Install and First Run": { @@ -152,6 +170,15 @@ "timestamp": "2026-10-12T04:40:58Z", "title": "Install and First Run" }, + "Interfaces and Stability": { + "file": "Interfaces-and-Stability.wiki", + "new_revid": 24009, + "old_revid": null, + "source_sha": "5e16ac6f098129ce", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:17Z", + "title": "Interfaces and Stability" + }, "Introduction to GMMU": { "file": "Introduction-to-GMMU.wiki", "new_revid": 23983, @@ -163,20 +190,20 @@ }, "Main Page": { "file": "Main-Page.wiki", - "new_revid": 23972, - "old_revid": 23960, - "source_sha": "1b53ef37afca940e", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:23Z", + "new_revid": 24030, + "old_revid": 24025, + "source_sha": "73ae8e2c1f093423", + "summary": "2026-10-30 wiki refresh: add Project Plan page (phases, gates, workstreams, next steps)", + "timestamp": "2026-10-30T17:12:04Z", "title": "Main Page" }, "MediaWiki:Sidebar": { "file": "MediaWiki-Sidebar.wiki", - "new_revid": 23976, - "old_revid": 23906, - "source_sha": "73777ab1026c6dc6", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:28Z", + "new_revid": 24032, + "old_revid": 24027, + "source_sha": "290bc8a24b8ab97c", + "summary": "2026-10-30 wiki refresh: add Project Plan to sidebar", + "timestamp": "2026-10-30T17:12:10Z", "title": "MediaWiki:Sidebar" }, "Migration Protocol": { @@ -190,29 +217,65 @@ }, "Open-IOV:About": { "file": "Open-IOV-About.wiki", - "new_revid": 23973, - "old_revid": 23929, - "source_sha": "2cba9bbf3ab93ecf", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:24Z", + "new_revid": 24028, + "old_revid": 23973, + "source_sha": "78307e495e4241ab", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:46Z", "title": "Open-IOV:About" }, + "OpenGL": { + "file": "OpenGL.wiki", + "new_revid": 24005, + "old_revid": null, + "source_sha": "9b1bdc8f09c43277", + "summary": "New page (2026-10-30 wiki refresh): OpenGL", + "timestamp": "2026-10-30T17:04:59Z", + "title": "OpenGL" + }, + "Performance": { + "file": "Performance.wiki", + "new_revid": 24008, + "old_revid": null, + "source_sha": "f2b44ad5e7ccaeb0", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:16Z", + "title": "Performance" + }, "Physical GPU Backend and Guest Wiring": { "file": "Physical-GPU-Backend.wiki", - "new_revid": 23985, - "old_revid": 23951, - "source_sha": "4bff4dbfa8b4644e", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:59Z", + "new_revid": 24015, + "old_revid": 23999, + "source_sha": "059030d7cb01f248", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:26Z", "title": "Physical GPU Backend and Guest Wiring" }, + "Project Goals": { + "file": "Project-Goals.wiki", + "new_revid": 24003, + "old_revid": null, + "source_sha": "8392f8beadbe9b45", + "summary": "New page (2026-10-30 wiki refresh): Project Goals", + "timestamp": "2026-10-30T17:04:55Z", + "title": "Project Goals" + }, + "Project Plan": { + "file": "Project-Plan.wiki", + "new_revid": 24029, + "old_revid": null, + "source_sha": "0440c98d1b4927fa", + "summary": "2026-10-30 wiki refresh: add Project Plan page (phases, gates, workstreams, next steps)", + "timestamp": "2026-10-30T17:12:02Z", + "title": "Project Plan" + }, "Project Status": { "file": "Project-Status.wiki", - "new_revid": 23986, - "old_revid": 23952, - "source_sha": "6d07f2a0c6f524a9", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:00Z", + "new_revid": 24011, + "old_revid": 23997, + "source_sha": "eb7a5ff2ceed506d", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:21Z", "title": "Project Status" }, "RPC Message Format": { @@ -226,11 +289,11 @@ }, "System Overview": { "file": "System-Overview.wiki", - "new_revid": 23990, - "old_revid": 23956, - "source_sha": "f2658a051178f477", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:02Z", + "new_revid": 24016, + "old_revid": 23990, + "source_sha": "262f756d3638e4a4", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:28Z", "title": "System Overview" }, "Template:About message": { @@ -262,47 +325,47 @@ }, "The Execution Frontier": { "file": "Execution-Frontier.wiki", - "new_revid": 23966, - "old_revid": 23966, - "source_sha": "b26ce9bb6fe63f6a", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:07Z", + "new_revid": 24012, + "old_revid": 23996, + "source_sha": "b204f91275834f9f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:22Z", "title": "The Execution Frontier" }, "The Guest Stack and CUDASW": { "file": "Guest-Stack-and-CUDASW.wiki", - "new_revid": 23978, - "old_revid": 23944, - "source_sha": "40480b8806fd7de5", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:56Z", + "new_revid": 24018, + "old_revid": 23978, + "source_sha": "2683360ca15b4995", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:31Z", "title": "The Guest Stack and CUDASW" }, "The Host Module (vgsp.ko)": { "file": "Host-Module.wiki", - "new_revid": 23981, - "old_revid": 23947, - "source_sha": "ede82920d8920b8f", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:57Z", + "new_revid": 24021, + "old_revid": 23981, + "source_sha": "e2af9d5d4ca0d65f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:35Z", "title": "The Host Module (vgsp.ko)" }, "The vGSP Execution Plane (vEP)": { "file": "Execution-Plane.wiki", - "new_revid": 23967, + "new_revid": 24013, "old_revid": 23967, - "source_sha": "f2b7e219c1b3b18a", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:08Z", + "source_sha": "f6df28da809fbc57", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:23Z", "title": "The vGSP Execution Plane (vEP)" }, "UVM in the Guest": { "file": "UVM-in-the-Guest.wiki", - "new_revid": 23991, - "old_revid": 23957, - "source_sha": "46ea1e4d85c342e5", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:02Z", + "new_revid": 24020, + "old_revid": 23991, + "source_sha": "70774393afbe8e1f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:34Z", "title": "UVM in the Guest" }, "VGPU-IO Integration Guide": { @@ -314,6 +377,15 @@ "timestamp": "2026-10-12T04:41:03Z", "title": "VGPU-IO Integration Guide" }, + "Vulkan": { + "file": "Vulkan.wiki", + "new_revid": 24006, + "old_revid": null, + "source_sha": "d94fa2d2b85e4057", + "summary": "New page (2026-10-30 wiki refresh): Vulkan", + "timestamp": "2026-10-30T17:05:01Z", + "title": "Vulkan" + }, "WPR Management": { "file": "WPR-Management.wiki", "new_revid": 23992, @@ -325,11 +397,11 @@ }, "risc-v-gsp-rm-debugger": { "file": "Riscv-GSP-RM-Debugger.wiki", - "new_revid": 23989, - "old_revid": 23955, - "source_sha": "5482a72b81b0e4e6", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:01Z", + "new_revid": 24024, + "old_revid": 23989, + "source_sha": "fcd6dc67e49c8d1f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:40Z", "title": "risc-v-gsp-rm-debugger" } }
\ No newline at end of file diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md index 9a041ef..c18a6ab 100644 --- a/wiki/CUDA-Userspace-Driver.md +++ b/wiki/CUDA-Userspace-Driver.md @@ -1,16 +1,23 @@ # CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive -> **Status (2026-10-12, honest).** The analysis covers the whole library for +> **Status (2026-10-30, honest).** The analysis covers the whole library for > driver 570.144 and the tooling is version-agnostic. The *control plane* gets > the stock guest stack through `cuInit`, device enumeration, `cuCtxCreate` > and the full driver-API smoke test — allocation, copies, free and destroy > complete end to end (`ALL OK`; the 64-byte payload round-trips exactly). > The *data plane* decodes guest push buffers and executes the memory and > semaphore work it understands; CUDA kernel launches are forwarded to the -> physical execution path, where a launch reaches the engine queue -> (front-end deposit and scheduling are observed) but an engine-authored -> release is not yet observed. Green runs are not yet reproducible on every -> boot — an open issue, tracked in the repository. Canonical detail: +> physical execution path and execute on the physical GPU — the +> engine-authored completions relay to the guest and its synchronisation +> waits resolve. The vector-add (`VADD n=32768 bad=0`), matrix-multiply +> (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, +> `FFN4 layers=4 bad=0`) clients verify green on fresh boots, and a +> ring-block ownership defect that could stall the first client after a +> reboot was root-caused and fixed. The staged-transfer class (multi-megabyte +> transfers between guest memory and device memory) is under repair with its +> failure modes proven. Graphics (OpenGL/Vulkan) shares this driver's object +> machinery on a parallel frontier +> ([`Graphics-Execution.md`](Graphics-Execution.md)). Canonical detail: > [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) > through `08`; reading order > [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); @@ -172,19 +179,18 @@ the data plane (`docs/08`). emulation and object/control replies are implemented; the stock guest stack completes `cuInit`, device enumeration, `cuCtxCreate` and the driver-API smoke test (allocation, copies, free and destroy). -* **Current frontier:** a guest CUDA kernel launch is forwarded and reaches - the physical engine queue — the front end deposits the launch and the - scheduling path accepts it — but the engine does not yet execute the - forwarded launch descriptor: an engine-authored release is the only proof - of execution and it is not observed. The per-boot green/stall variance on - the emulation path is an open issue, tracked in the repository. The chain, - the evidence and the live state are on +* **Current frontier:** guest CUDA kernel launches execute on the physical + GPU with engine-authored completions relayed to the guest; the verified + clients are green on fresh boots. The leading edge is the staged-transfer + class (some large multi-chunk clients still mis-verify, with the failure + modes proven), the every-boot acceptance campaign, and per-guest execution + isolation. The chain, the evidence and the live state are on [`Execution-Frontier.md`](Execution-Frontier.md). * **Data plane today:** the host module decodes guest push buffers as method streams and executes the memory and semaphore work it understands (memsets, copies, semaphore releases), and forwards engine-class work to - the physical backend; kernel launches are forwarded but not yet executed - by the engine. + the physical backend; kernel launches now execute on the engine through + the forwarded path, with completions relayed back to the guest. ## 5. The crash-path lesson: replies are consumed data diff --git a/wiki/DirectX.md b/wiki/DirectX.md new file mode 100644 index 0000000..9b531dc --- /dev/null +++ b/wiki/DirectX.md @@ -0,0 +1,100 @@ +# DirectX + +> **Scope.** The Windows guest path — Direct3D and the rest of the DirectX +> family — as a **project target**: how the Windows NVIDIA stack relates to +> the interface Open-IOV already emulates for Linux, what a bring-up would +> require, and what has *not* happened yet (no Windows guest has been booted +> on the emulated vGPU). Linux graphics is covered on +> [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) and +> [`Graphics-Execution.md`](Graphics-Execution.md). Canonical sources: +> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) +> (guest stack) and +> [`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md) +> (support ladder). The docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[target]` designed for, +bring-up not started; `[planned]` roadmap; no Windows-specific code exists in +the project today. + +## 1. The Windows graphics stack + +```text +Direct3D application (D3D12 / D3D11 / D3D9) or CUDA application + -> DXGI (swap chains, adapters, presentation) + -> user-mode display driver (UMD): nvd3dumx / nvwgf2umx [D3D] + -> kernel-mode driver (KMD): nvlddmkm.sys, under WDDM + -> GPU +Windows kernel graphics subsystem (dxgkrnl) + -> GPU scheduler, video-memory manager (VidMm), residency, TDR +``` + +Two structural differences from Linux matter for this project: + +* **WDDM owns more of the pipeline.** The OS schedules command buffers + submitted by the UMD, manages video-memory residency, and runs the GPU + scheduler in the kernel; the driver's KMD programs hardware contexts and + fences beneath that. (D3D12 exposes queues and fences with an explicit + model that is closer to Vulkan than older APIs were.) +* **The device bring-up is different.** A Windows display adapter starts + through the OS's display/PDO path (adapter enumeration, ROM/VBIOS views, + VIDPN topology, power transitions), not the Linux DRM/modeset path. On + Turing and later, however, the driver still talks to the **same GSP + firmware resource manager** — the interface `vgsp.ko` emulates. + +CUDA on Windows rides the same KMD with its own userspace driver, so the +device-facing requirements overlap almost entirely with the Linux CUDA work +already proven ([`Execution-Frontier.md`](Execution-Frontier.md)). + +## 2. What this means for Open-IOV + +The project's architecture does not change for Windows: **emulate the +firmware interface, pass the work through, patch nobody.** The Windows KMD +would be stock (signed, as shipped); the emulation must satisfy its +device-view and its GSP-RM usage. The measured bring-up items, from +[`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md): + +* WDDM device start on the emulated device (PDO/display path, MSI-X and + interrupt semantics). +* ROM/VBIOS provisioning in the device profile (Windows adapters expect one). +* Power-management transitions (Dx states) and reset behaviour. +* D3D/DXGI device creation, then compute (CUDA on Windows), then rendering + through the same execution lanes as Linux. + +## 3. Status + +| Item | State | +|---|---| +| Windows guest boots on the emulated vGPU | `[planned]` — not started | +| KMD (`nvlddmkm.sys`) initialises on the emulated device | `[target]` | +| CUDA on Windows (driver-API smoke) | `[target]` | +| Direct3D device creation / rendering | `[target]` | +| Multi-VM Windows guests | `[target]` | + +The support matrix lists **Windows x86_64 (WDDM driver, CUDA, 3D)** as +**Target**: the architecture is designed for it; the bring-up is ahead. The +proposed ladder mirrors the Linux milestones: KMD initialises → adapter +enumeration healthy → CUDA-on-Windows driver-API smoke → D3D device creation +→ first compute execution → first rendered pixels → multiple VMs. + +## 4. What the project will not do + +* No Windows-specific code inside the guest — no patched `nvlddmkm`, no + shims in the guest kernel, no guest agent. Any Windows-only quirk belongs + in the host module's device model. +* No reliance on the vendor vGPU host stack; the same mediator answers both + guest OS families. +* The two-VM GPU demo videos published so far used **Linux** guests (and, for + the `virgl` variant, a labelled non-vGPU path); neither is DirectX. + +## 5. Related pages + +* [`Graphics-Execution.md`](Graphics-Execution.md) — the execution lanes + Direct3D would use. +* [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) — the Linux API paths, + further along. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — the support + ladder, including the Windows target row. +* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md) — §7 covers + the Windows kernel-mode driver and the no-guest-code rule. +* [`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md) — + the canonical matrix and what each support level means. diff --git a/wiki/Driver-Lifecycle-Field-Guide.md b/wiki/Driver-Lifecycle-Field-Guide.md index 1115be9..9d3fde8 100644 --- a/wiki/Driver-Lifecycle-Field-Guide.md +++ b/wiki/Driver-Lifecycle-Field-Guide.md @@ -5,7 +5,7 @@ > unmodified 570.144 guest. Canonical source: > [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md); if this page and the docs tree disagree, the docs tree wins. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival. +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival. ## 1. How to read this page @@ -107,7 +107,7 @@ vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject **Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. **Emulation today:** the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet. -**Progress:** context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted `CTXNOTVALID`, and a following `GPU_PROMOTE_CTX` wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [`Execution-Frontier.md`](Execution-Frontier.md). The roadmap from here is: +**Progress:** context creation returns (the driver-API smoke completes end to end) and guest CUDA kernel launches now execute on the physical engine with engine-authored completions relayed to the guest — verified green clients on fresh boots. Earlier walls are closed: the host GR ladder's `CTXNOTVALID` fault and the `GPU_PROMOTE_CTX` wedge led to the commit-chain work, and the forwarded guest launch that once reached the front-end without a dispatched grid now executes end to end. The current leading edge is the staged-transfer class, the every-boot acceptance campaign, per-guest isolation, and the graphics pixel frontier. See [`Execution-Frontier.md`](Execution-Frontier.md) and [`Graphics-Execution.md`](Graphics-Execution.md). The roadmap from here is: ```text UVM registers → cuInit returns → context creation → memory allocation diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md index fe21fda..c8d42cc 100644 --- a/wiki/Execution-Frontier.md +++ b/wiki/Execution-Frontier.md @@ -1,15 +1,20 @@ # The Execution Frontier -> **What this page is.** The current leading edge of guest-facing execution: -> the chain that must fire for a stock guest CUDA kernel to run on the -> physical GPU, what has been proven about it, where it stops, how we know, -> and the open lines of investigation and theories. It is evidence-led: every -> statement carries a live observation, a decompilation finding, or a clearly -> labelled theory. Condensed from +> **What this page is.** The state of guest-facing execution: the chain that +> must fire for a stock guest CUDA kernel to run on the physical GPU, what has +> been proven about it, how we know, and where the leading edge is now. The +> chain fires end to end: verified guest CUDA clients execute on the physical +> GPU and wake up correctly; the leading edge is now breadth and hardening. A +> **parallel graphics frontier** (OpenGL/Vulkan execution) shares the same +> submission machinery at a different execution layer — +> [`Graphics-Execution.md`](Graphics-Execution.md), +> [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md). It +> is evidence-led: every statement carries a live observation, a +> decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. -Status: 2026-10-12 (source page dated 2026-10-06), tree `master @ 2f066727`. +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. @@ -45,46 +50,77 @@ guest: builds launch + method stream | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed | | 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) | | 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised) | `[wall]` not observed | -| 7 | **Engine reads the QMD** | `[wall]` not observed | +| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed | +| 7 | **Engine reads the QMD** | `[live]` observed | | 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in | -| 9 | GR dispatches the grid to the SMs | never observed on this stack | -| 10 | Engine authors release + channel semaphore | absent for forwarded launches | -| 11 | Host relays completion; guest wakes | the guest waits and times out | +| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve | -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +**One-line summary:** the whole chain is live; the leading edge is now breadth +and hardening — the large-buffer staged-transfer class, reproducibility across +boots, and per-guest execution isolation. ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire the ring entry is consumed and the front-end -methods run — but the launch descriptor is never materialised, the engine -never reads a QMD, no engine-authored completion appears, and the guest blocks -in its synchronisation wait. - -This is **not** a context-validity wall (ruled out at dispatch) and **not** a -window-mapping wall for the descriptor destination (that install is accepted). -The wall is the **form of the launch method stream**: what the front-end -requires in order to install the inline descriptor, and what the engine -requires in order to read and dispatch it. +A stock, unmodified guest CUDA program executes its kernels on the physical +GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. Engine-authored completions relay to the guest and its +synchronisation waits resolve. + +The leading edge has moved from the launch chain itself — it fires — to +breadth and hardening around it: + +* **The staged-transfer class.** Multi-megabyte transfers between guest memory + and device memory are under repair: some large multi-chunk clients still + mis-verify, with the failure modes proven (a release applied before the op + it owes, a ledger-vs-writer page divergence, and a readback racing a + writeback). +* **Reproducibility across boots.** Two defects landed and were fixed: the + boot-poisoning defect (under the driver's status-queue mode the emulation + read the wrong queue cursor, so under load the reply ring could + desynchronise and the first CUDA client would fail) and a ring-block + ownership defect that could stall the first client after a reboot. Green + clients reproduce across fresh boots; a formal every-boot campaign remains. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level + (two guests demonstrated); per-guest execution isolation for multiple + simultaneous CUDA guests is in progress. + +### A parallel frontier: 3D graphics + +Graphics execution shares this page's submission and completion machinery but +runs a different engine workload. The story so far: the X server and GLX run +on the emulated vGPU; the emulation captures the X server's graphics-class +stream and the engine executes it — both as a verbatim 3D method-stream replay +and as a synthesized launch that retires real `sm_75` SASS — and the guest +Vulkan pipeline runs end to end (device → command buffer → fence → readback). +What is missing is **rendered pixels**: every frame is still black, and the +program-delivery/program-VA-mapping pair is the working frontier. The full +picture lives on [`Graphics-Execution.md`](Graphics-Execution.md). + +No breadth beyond these is claimed: Windows guests, migration and performance +numbers remain planned. ## 3. How we know (observables and their limits) | Observable | What it proves | What it does not prove | |---|---|---| -| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | nothing about the engine | -| Method execution on the ladder channel | the front-end can execute the method vocabulary | **not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten with a detectable sentinel | whether the descriptor was materialised after submission | a read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | the inline-install method did not run | it does not say why | -| Engine-authored release word absent; poll times out | the engine did not complete the launch | it does not say whether the engine started | +| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results | +| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches | +| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient | +| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress | | No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction | -| Channel instance context flag set | the channel is context-valid at dispatch | nothing about the descriptor path | -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +**Consequence:** acceptance is verified guest results plus the engine-authored +release; a green run counts as progress only when it reproduces across fresh +boots. ## 4. What is exonerated (do not relitigate) @@ -98,68 +134,75 @@ signals, which we have already learned to distinguish. already initialised in the same cycle; not the dispatch delta. * Parenting/TSG for the fetch — both variants fetch identically. * The copy-engine path — CE-class work, releases and fault relay are live - (default off); the GR path is a separate problem. + (default off); the GR path is separate. +* The descriptor-gate line of investigation (the old T1–T4 theories) — the + inline-descriptor install, the engine read, the dispatch, the + engine-authored release and the guest wake-up are all observed; closed by + observation. +* The old boot-poisoning defect — root-caused to the status-queue consumption + cursor and fixed. ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** The forwarded stream includes a verbatim copy of the guest's -prelude whose inline-data writes target addresses not present in the engine's -view of the channel's page tables, and the referenced data is not relocated. -The shorter, proven stream shape fires its tail (channel semaphore) but still -does not schedule the descriptor. +**Issue A — the staged-transfer class.** Multi-megabyte transfers between +guest memory and device memory are under repair: some large multi-chunk +clients still mis-verify, with the failure modes proven and named. + +**Issue B — every-boot reproducibility (acceptance campaign).** The +boot-poisoning defect and the post-reboot ring-block ownership defect were +root-caused and fixed, and green clients reproduce on fresh boots; the formal +every-boot acceptance campaign is the remaining work. -**Issue B — the descriptor-virtualization control method is absent.** The -method that arms the inline-descriptor install path is not present in the -forwarded stream. Fixing it should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. +**Issue C — per-guest execution isolation.** Multiple mediated devices with +different emulated VRAM sizes and multiple guests coexist at the device level +(two guests demonstrated); per-guest execution isolation for simultaneous +CUDA guests is in progress. -**Issue C — diagnostic hygiene.** The sentinel placed at word 0 overlaps real -descriptor fields; it must be moved to an unused word so diagnostics do not -perturb what they measure. +**Issue D — the graphics pixel frontier.** The engine executes graphics method +streams, but no rendered (non-black) frame has been produced yet; program +delivery and program-VA mapping are the named blockers +([`Graphics-Execution.md`](Graphics-Execution.md)). ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline and **re-add the - virtualization control method** — one variable at a time, with the install - sentinel and the engine-read signal as the readout. -2. **Relocate or drop the verbatim prelude** so inline-data writes target - addresses present in the engine's view, or replace it with the host-side - staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference** — our own ladder dispatching a - descriptor built from the same source material, or a captured working - launch of the stock stack on the same GPU — converting the remaining work - from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware debugger before spending a supervised cycle - ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)). - -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the control method enables the - install path. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Its refused writes leave - the launch path unable to schedule the descriptor. -* **T4 — relocation is the missing transform.** The guest's descriptor is - valid for the guest's view of memory; the forwarding path must relocate - referenced data into the channel's view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +1. **Repair the staged-transfer class** — the green clients + stay the regression battery and must remain green throughout. +2. **Complete the every-boot acceptance campaign** — demonstrate the fixed boot + path across a defined run of fresh boots before claiming boot-to-boot + reproducibility. +3. **Per-guest execution isolation** — build from the device-level coexistence + to isolation for simultaneous CUDA guests. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel; each entry is accepted only on its own verification + output (`bad=0` + `STATUS OK`). +5. **The graphics pixel frontier** — materialise the driver's program + uploads and map the program VA so draws execute with real state; the lanes, + the walls and the milestone ladder are on + [`Graphics-Execution.md`](Graphics-Execution.md). + +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed, so +the launch chain they questioned is proven end to end. The remaining work is +engineering (staged transfers, an every-boot acceptance campaign, per-guest +isolation, the graphics pixel frontier), each with an acceptance signal named +before the work. ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end; the guest's own verification (`bad=0` + `STATUS OK`) + is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred. * **Emulator verdicts are instruction/memory-level.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** * **A wall is closed only by an acceptance signal**, named before the experiment. @@ -170,6 +213,8 @@ signal, and the engine-authored release, in that order. client path. * [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work is forwarded to. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the parallel graphics + frontier (OpenGL/Vulkan), sharing this machinery. * [`Execution-Plane.md`](Execution-Plane.md) — the execution layer's design. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page tables, QMDs and the engines. diff --git a/wiki/Execution-Plane.md b/wiki/Execution-Plane.md index 5d1052f..10e3379 100644 --- a/wiki/Execution-Plane.md +++ b/wiki/Execution-Plane.md @@ -8,7 +8,7 @@ > when the two disagree, the docs tree wins. Front door: > [`../docs/README.md`](../docs/README.md). -Status: 2026-10-08, tree `master @ 1f571dda`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or parameter-gated; `[wip]` partial; `[wall]` current blocker; `[target]` named success latch; `[planned]` designed, not built. @@ -37,7 +37,7 @@ would collide with existing project vocabulary. | Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request | | Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow | | Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix | -| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned | +| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; two guests with different emulated VRAM sizes coexist at the device level, per-guest execution isolation is in progress | ## 2. The shape of the plane @@ -93,18 +93,19 @@ guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to - the ladder channel's GPFIFO and rings the real doorbell. `[gated]` - default-off; latched on a proven real GR submission. -5. **Engine.** The front-end fetches the entry and executes methods; then the - **inline-QMD install** must materialise the descriptor for the engine to - read, currently **`[wall]`** — not observed. Context validity, destination - mapping and fetch/alignment are exonerated; the recovered-method-stream - form is the investigation. + the ladder channel's GPFIFO and rings the real doorbell. `[live]` — + exercised on every verified guest launch (default-off while the seams are + validated). +5. **Engine.** The front-end fetches the entry and executes methods; the + inline launch descriptor installs and the engine reads it, the grid + dispatches to the SMs, and the engine authors the release. `[live]` — + context validity, destination mapping and fetch/alignment were exonerated + on the way. 6. **Completion.** The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X - wakes the guest. CE work is live in this shape `[gated]`; for GR the - completion-counter publication is the current second face of the wall. + wakes the guest. `[live]` for CE and GR: engine-authored completions relay + to the guest and its synchronisation waits resolve. ## 4. How work is found: discovery, truth, completion @@ -129,27 +130,24 @@ guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) ## 5. Challenges, honestly -* **GR dispatch wall.** The forward path runs front-end methods, the context - is valid and the destination window is accepted — and the descriptor - install is still not observed. The mode matrix carries dozens of - single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory - block, lane controls, shadow windows, pointer relocation). -* **Completion publication.** A boot where GR rings were discovered and - releases applied still saw the guest re-queue the same push and give up. A - later `sema_dbg=Y` run settled it: the release was written to the exact - slot, the notifier cleared and the event posted, while the completion - counter publication never wrote because the fixed-total autoreg mapping - makes it a structural no-op. The leading reading is the counter-slot - publication; the GP_GET write-back is the secondary, still-open candidate. - Both are debug-gated today. -* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The - first fix wave landed and the earlier CE freeze is gone; the stall moved to - GR ring discovery, which the latest discovery work addresses. Acceptance - is 10 cold boots green, pending. -* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest - UVM because host per-client sema/GR state is not reset when the guest - re-runs its per-client boot handshake. Per-client state containers and - reset are the designed path. +* **The staged-transfer class (compute).** Some large multi-chunk CUDA clients + still mis-verify; the failure modes are proven (a release applied before + the op it owes, a ledger-vs-writer page divergence, a readback racing a + writeback) and the fixes are in flight. +* **Every-boot reproducibility.** The boot-poisoning defect (wrong + status-queue consumption cursor under the driver's swap mode) and the + post-reboot ring-block ownership defect were root-caused and fixed; green + clients reproduce across fresh boots, and the formal every-boot acceptance + campaign is the remaining work. +* **Multiple clients and per-guest isolation.** Two guests with different + emulated VRAM sizes coexist at the device level; per-guest execution + isolation for simultaneous CUDA guests is in progress. +* **The graphics pixel frontier.** Graphics work now rides this plane in two + lanes (verbatim 3D method-stream replay and synthesized launches) and the + engine retires real graphics streams — but no rendered (non-black) frame + has been produced yet. Program delivery and program-VA mapping are the + named blockers; the lanes and the ladder are on + [`Graphics-Execution.md`](Graphics-Execution.md). ## 6. Portability @@ -164,15 +162,16 @@ human approval gate. ## 7. Future -The planned next steps are reproducibility acceptance, the descriptor-install -and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger -where possible), the arbitrary-kernel acceptance suite, then the near-native -performance program. The unsolved future classes — multi-tenant fairness, -fault replay/cancel, context switching between tenants, hostile-guest +The planned next steps are the staged-transfer fixes, the every-boot +acceptance campaign, per-guest isolation, and the graphics pixel frontier +(offline-first with the RISC-V GSP-RM debugger where possible), then the +near-native performance program. The unsolved future classes — multi-tenant +fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction in the -canonical document. 3D graphics reuses this pipeline once compute dispatch is -proven; it adds engine breadth (video/image), more channels and synchronisation -volume, and a much larger method vocabulary, but not a new plane. +canonical document. 3D graphics reuses this pipeline and now executes across +two lanes ([`Graphics-Execution.md`](Graphics-Execution.md)); it adds engine +breadth (video/image), more channels and synchronisation volume, and a much +larger method vocabulary, but not a new plane. --- diff --git a/wiki/FAQ.md b/wiki/FAQ.md index 9472740..35554e5 100644 --- a/wiki/FAQ.md +++ b/wiki/FAQ.md @@ -5,7 +5,7 @@ > [`../docs/09-reference/glossary-and-faq.md`](../docs/09-reference/glossary-and-faq.md); > the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. +Status: 2026-10-30, tree `master @ b4930168`. ## Project and positioning @@ -27,26 +27,36 @@ not for the QEMU/emulated path ([`Install-and-First-Run.md`](Install-and-First-Run.md) §2). **Why is `nvidia-smi` working but my CUDA kernel not running?** -Because execution is the frontier, not the setup. Today, a stock guest driver -probes, `nvidia-smi` works, device enumeration works, and the full CUDA -**driver-API** path (context creation, allocation, copies, free, destroy) -completes; a guest **kernel launch** is forwarded to the physical engine queue -but the engine does not yet execute the forwarded launch descriptor -(`[wall]`). This is the project's known status, not a misconfiguration — see +It probably is running: guest CUDA kernels execute on the physical GPU and +verify — the vector-add, matrix-multiply and transformer clients are green on +fresh boots. If a specific workload fails, the known classes are: multi-megabyte +staged transfers between guest and device memory (under repair, failure modes +proven), and simultaneous CUDA guests (per-guest execution isolation is in +progress). This is the project's known status, not a misconfiguration — see [`Project-Status.md`](Project-Status.md) and [`Execution-Frontier.md`](Execution-Frontier.md). +**Can I run OpenGL or Vulkan applications in the guest?** +Partially, and it is an active frontier. The X server and GLX come up on the +emulated vGPU with the stock driver, the engine executes real GL method +streams (verbatim 3D replay plus synthesized launches), and the guest Vulkan +pipeline runs end to end (device → command buffer → fence → readback). +Rendered (non-black) frames are the remaining wall — see +[`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) and +[`Graphics-Execution.md`](Graphics-Execution.md). + **How fast is it?** -No published numbers yet. Copy-engine-class forwarding and its completion and -fault relay are implemented (default off) and GR execution is the open -frontier; even the initialization path has not yet been measured under the -project's methodology. Any performance claim will ship together with its -methodology and measurements. +No published numbers yet. Kernel launches execute on the physical engine; +what remains unmeasured is the project's own methodology — nothing ships as a +performance claim without its measurements. The first scheduled measurement +is the graphics target (60 fps sustained in one VM, no forcing). ## Compatibility **Will it support Windows guests?** -Yes, it is a target: same architecture, measured bring-up. +Yes, it is a target: same architecture, measured bring-up. No Windows guest +has been booted on the emulated vGPU yet; the Windows/DirectX path (WDDM, +`nvlddmkm`, Direct3D) is described on [`DirectX.md`](DirectX.md). [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) lists the support ladder. diff --git a/wiki/GSP-and-GSP-RM.md b/wiki/GSP-and-GSP-RM.md index c714612..651259b 100644 --- a/wiki/GSP-and-GSP-RM.md +++ b/wiki/GSP-and-GSP-RM.md @@ -7,7 +7,7 @@ > [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md); > the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` designed end state. @@ -40,6 +40,26 @@ it is written to talk to an RM, and on modern GPUs a copy of the RM runs on the GSP. The driver is therefore an **RM client**: it issues requests, receives callbacks, and observes state through a well-defined protocol. +### What the GSP does on bare metal + +* **It is the device's manager processor.** Since Turing, the driver does not + program most of the GPU. The GSP runs NVIDIA's signed firmware — a small + RTOS plus the RM server — and owns GPU-global state: initialisation, the + memory-management unit's deeper page-table levels, engine configuration, + context scheduling, interrupts and power/clock management (PMU). The + driver's kernel RM boots it, then becomes its client. +* **It boots from driver-provided memory.** At power-on the driver carves a + firmware-protected region of video memory (WPR2), copies the firmware image + there, has the device verify signatures (FWSEC), hands control to the boot + ROM ("booter"), which starts the RISC-V core; the firmware runs its RTOS and + RM initialisation and signals readiness. Everything the driver does + afterwards is a request to that server. +* **Why this matters for virtualization.** Because the management path is a + *firmware interface* — shared-memory queues plus a small RPC protocol, + not a pile of register lore — it is small, stable, and observable. That is + the interface Open-IOV emulates; it is the reason a virtual NVIDIA GPU is + practical at all. + ## 2. The boot handshake we emulate The guest driver boots the GSP exactly as it would on bare metal: @@ -130,7 +150,32 @@ vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM → guest vector, with the pending-bit array emulated so a masked vector's interrupt is not lost. -## 7. What we emulate today +## 7. How Open-IOV deals with the GSP — two modes + +The project handles the GSP in two ways, and they complement each other: + +1. **Guest-facing GSP emulation** (`vgsp.ko` *is* the virtual GSP). For each + mediated device, the module presents a device whose GSP behaviour is + software: it implements the boot state machine the driver polls (stage + completion, not silicon), answers the WPR/BAR0 register protocol, serves + the message-queue transport and every RPC verb the driver uses during + probe, initialisation and runtime, and delivers events via emulated MSI-X. + The guest's RM believes it is talking to firmware; it is talking to the + emulation. +2. **The real GSP as the backend server.** When the module owns the physical + GPU, it loads the *matching real firmware image* into the device, runs the + actual boot (FWSEC → booter → RISC-V → RM init-done), and becomes the RM + client itself — "be the host driver's RM" becomes "be the RM client". The + same protocol the guest uses one layer up is used here one layer down, and + it is what the work path submits through + ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). + +Keeping the two in sync is a project discipline: what the guest's RM is +promised (objects, controls, page tables, completions) must be answerable by +the emulation, and what executes on hardware must round-trip through the real +firmware session. + +### What is emulated today | Layer | State | Notes | |---|---|---| @@ -140,9 +185,10 @@ interrupt is not lost. | Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list | | Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving | | Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet | -| Page tables for server-owned ranges | `[wip]` | contract received and recorded; leaf population is the execution work | -| Event/notification delivery | `[wip]` | MSI-X works; richer event semantics land with forwarding | -| Firmware behaviour (routing, scheduling, real execution) | `[wall]` | the execution frontier — see [`Execution-Frontier.md`](Execution-Frontier.md) | +| Page tables for server-owned ranges | `[wip]` | contract received and recorded; forwarded paths run on module-built scratch tables while full server-owned leaf population is the remaining execution work | +| Event/notification delivery | `[live]` | MSI-X works; completion counters, notifiers and channel events are serviced and relayed to the guest | +| Real GSP as the backend server | `[live]` | the module boots the real firmware, serves its RM client ladder, and executes forwarded work through it | +| Firmware behaviour (routing, scheduling, real execution) | `[live]` | forwarded guest CUDA launches and graphics method streams execute on the physical engines — see [`Execution-Frontier.md`](Execution-Frontier.md) | ## 8. Where the firmware knowledge lives diff --git a/wiki/Graphics-Execution.md b/wiki/Graphics-Execution.md new file mode 100644 index 0000000..48423f8 --- /dev/null +++ b/wiki/Graphics-Execution.md @@ -0,0 +1,182 @@ +# Graphics Execution + +> **Scope.** How the emulated vGPU runs **3D graphics** work — the OpenGL and +> Vulkan command streams a stock guest submits — on the physical GPU, and what +> is proven about it. This is the shared machinery behind +> [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) and the Windows/DirectX +> target ([`DirectX.md`](DirectX.md)); CUDA compute uses the same submission and +> completion path for a different engine workload +> ([`Execution-Frontier.md`](Execution-Frontier.md)). Canonical sources: +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) +> (the plan and milestone ladder) and +> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) +> (the measured walls); the docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed end to +end; `[wip]` partially built; `[wall]` current blocker; `[target]` defined +success state; `[planned]` designed, not built. + +## 1. What "real pixels" means + +The acceptance target is strict: a **stock** guest GL or Vulkan application's +draw commands are executed by the physical GPU's fixed-function 3D pipeline +(vertex fetch, primitive assembly, rasterisation, fragment work, ROP), the +render target is readable by the guest, and that image reaches the capture +path through the emulated mdev. The pixels must not come from another GPU, a +software rasteriser, or a host display round-trip. + +(The project has also published two deliberately-labelled two-VM demo videos +that use the standard `virtio-gpu`/`virgl` virtual GPU on a host integrated +GPU, and a CUDA-animation video that *does* run on the emulated vGPU through +the compute path. The `virgl` videos are honest demonstrations of the guest +experience; they are not this execution path, and their captions say so.) + +## 2. The path a graphics command takes + +```mermaid +graph TD + APP["Guest applications<br/>OpenGL · Vulkan · CUDA"] --> LIB["Guest user space<br/>GL: GLX client + the driver's GL core<br/>Vulkan: the NVIDIA ICD<br/>CUDA: libcuda"] + APP --> X["X server (Linux GL)<br/>GLX module + the NVIDIA DDX"] + LIB --> KMD["Guest kernel: nvidia.ko<br/>RM core, GSP client, channels, modeset/drm"] + X --> KMD + KMD -- "GPFIFO rings + doorbell;<br/>method streams in push buffers" --> VG["Host module vgsp.ko<br/>device model + emulation core"] + VG --> L1["Lane 1: verbatim 3D<br/>method-stream replay"] + VG --> L2["Lane 2: synthesized launch<br/>from a captured section"] + VG --> L3["Lane 3: forwarded CUDA<br/>launch (compute)"] + L1 --> FE["Physical GPU<br/>front-end fetch -> GR engine"] + L2 --> FE + L3 --> FE + FE -- "engine-authored releases<br/>(semaphores in guest memory)" --> VG + VG -- "completion relay: MSI-X +<br/>poller/event wake the guest" --> KMD +``` + +```text +guest apps (GL / Vulkan / CUDA) + -> guest user space (GLX client + GL core, Vulkan ICD, libcuda) + -> guest X server (GLX module + NVIDIA DDX) [OpenGL] + -> nvidia.ko: channels, GPFIFO rings, doorbells [all APIs] + -> vgsp.ko: capture sections, decode pushes, forward + -> physical GPU front-end -> GR engine (3D + compute) / copy engines + <- engine-authored releases written into guest memory + <- completion relay (MSI-X + wakes) back to the guest +ASCII fallback above; the diagram shows the three execution lanes. +``` + +## 3. The three execution lanes + +Graphics work reaches the engines today in one of three shapes, in increasing +fidelity: + +1. **Verbatim 3D method-stream replay** `[live]`. The emulation captures the + X server's graphics-class (`0xC597`, `TURING_A`) section from its channel, + relocates the operand addresses into host-built shadow pages the engine can + reach, and submits the stream unchanged. The engine retires it: the uploads + (`LINE_LENGTH_IN`/`LAUNCH_DMA`), the state, program and texture binds, and + the release all execute with no MMU fault. This proves the *mechanics* of + 3D execution — object bind, window placement, operand relocation, + completion — on real hardware. +2. **Synthesized launch from a captured section** `[live]`. A compute-shaped + launch built from the captured section (operand list, program, a 64-word + launch descriptor) reaches the engine and retires real `sm_75` SASS; the + Turing shader-program-header form (`V3`) is accepted with zero header + faults across repeated launches. This proved engine-side dispatch before + the verbatim lane existed. The program itself is still a development + stand-in at this rung — execution proof, not pixels. +3. **Forwarding the client's own pushes** `[wall]`. The goal state: the GL + client's own GPFIFO ring and method streams are read, decoded and executed + with the client's real per-draw state. The GL client's channel is created + today, but it never submits — its pushes have not been made readable — and + this lane is gated on the ring/push-readability work (§6). + +## 4. The channel and object picture + +* **Every submission lives on a channel**: a GPFIFO ring plus engine objects, + with a USERD the driver advances its put pointer in and a doorbell the host + module traps. Channels are typed by engine: a graphics channel carries 3D + state and draws; copy-class channels carry memory movements. +* **The graphics object class** the guest's 3D stream binds is the Turing + graphics class, `0xC597`. The stream the emulation can capture today comes + from the **X server's channel** (`OpenGL`); the 2D present/readback work + rides lighter 2D-class and copy-engine shapes. +* **Cells of state**: 3D streams are long—hundreds of method words per draw — + and mix state binds (viewport, render target, ZS, blend/ROP), program and + constant binds, vertex/index streams and draw-issuing methods. The capture + machinery recognises the init-shaped operands today; the per-draw state + vocabulary is being extended ([G-B in the plan]( + ../docs/04-implementation/05-graphics-execution-plan.md)). +* **Synchronisation**: tracking semaphores, notifications and fence values + written by the engine; the guest's swap/present waits sit on top of those. + +## 5. Completion, fences and presents + +The acceptance signal is the **engine-authored release**: a semaphore the +engine writes when the work completes. For GL, completion flows through the +channel's semaphores/notifications to the X server and the app's swap wait; +for Vulkan, the client's fence value is released; for CUDA, the launch's +release group wakes the client. The emulation relays engine completions into +guest-observable memory (MSI-X plus a poller), and the guest resumes from +there. + +During bring-up the module also carries a **diagnostics-only mirror** for apps +that spin on a shared progress word; it exists so a run can proceed while the +natural completion path is being finished. The 60-fps milestone explicitly +requires it switched off, along with any external poke. + +## 6. Where the frontier is (ranked walls) + +1. **The GL client's submissions** `[wall]`. The client's channel exists but + never submits: direct measurement shows the client process writes no put + pointer in any mapping the service can see, and its pushes are not yet + reachable. Everything downstream waits on this. +2. **Program (SASS) delivery** `[wall]`. Graphics programs reach the device + through the driver's copy/utility-channel uploads; those inline-data + writes are not yet materialised for the GL/Vulkan clients, and the program + page the engine fetches is empty. With the data-set gate bypassed, the + engine fetches the launch and MMU-faults (`FAULT_PDE`) at the program VA — + proof the channel and context are alive, and that mapping + content are + the missing pair. +3. **Non-black frames** `[wall]`. Once draws execute with real state and real + programs, the capture must show motion; every frame captured so far is + black. +4. **Sustained frame rate, and two VMs** `[target]`. The defined end state: + 60 fps in one VM with no forcing, then two VMs on two mediated devices + with non-black frames in both. + +## 7. What is proven today + +| Capability | State | +|---|---| +| X server initialises on the emulated vGPU (driver DMA sanity push, ScreenInit) | `[live]` | +| `glxinfo` enumerates the emulated GPU with the NVIDIA renderer | `[live]` | +| GZDoom launches and runs on the emulated vGPU | `[live]` (pixels frontier) | +| GL section capture: the X server's `0xC597` stream decoded | `[live]` | +| Verbatim 3D method-stream replay retired by the engine (no MMU fault) | `[live]` | +| Synthesized launch executes real `sm_75` SASS; Turing `V3` program header accepted | `[live]` | +| 2D screen fills and copy-engine present ops execute | `[live]` | +| Direct GL client: device-init gate handled, channels allocate, readback path returns clean | `[wip]` (client submissions not yet) | +| Non-black pixels from the emulated vGPU's 3D pipeline | `[wall]` | +| Windows/DirectX guests | `[planned]` | + +## 8. What is not claimed + +* No non-black pixel has been produced by the emulated vGPU's 3D pipeline yet. +* No frame-rate or latency numbers: the scheduled measurement is defined + (60 fps over 60 s, no forcing) but not yet met. +* The `virgl`-based demo videos are a different, labelled path — not this one. + +## 9. Related pages + +* [`OpenGL.md`](OpenGL.md) — the guest OpenGL path and its status. +* [`Vulkan.md`](Vulkan.md) — the guest Vulkan path and its status. +* [`DirectX.md`](DirectX.md) — Windows guests and the DirectX target. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the CUDA compute chain, + proven end to end. +* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work + is forwarded to. +* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — channels, GPFIFO, + engines, and page tables. +* [`Host-Module.md`](Host-Module.md) — where capture, decode and forwarding + live. +* [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) — + the canonical plan: gap list, three approaches, and the M-GL1..M-GL6 + milestone ladder with gates. diff --git a/wiki/Guest-Stack-and-CUDASW.md b/wiki/Guest-Stack-and-CUDASW.md index 1548f63..eca21a3 100644 --- a/wiki/Guest-Stack-and-CUDASW.md +++ b/wiki/Guest-Stack-and-CUDASW.md @@ -6,7 +6,7 @@ > [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md); > the docs tree wins on disagreement with this page. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with > the stock guest driver; `[gated]` implemented default-off behind supervised > gates; `[wall]` known blocking limit; `[target]` designed, not built; > `[planned]` roadmap. @@ -31,9 +31,12 @@ nvidia.ko (RM + UVM + modeset) Adjacent members of the family: **libnvidia-ptxjitcompiler** (PTX→SASS JIT — usually "just works" once the driver initializes), **libnvidia-fatbinaryloader / nvrtc**, **UVM** (`nvidia_uvm.ko` — [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)), -the **graphics stack** (`libGLX_nvidia`, `libnvidia-glcore`, Vulkan ICD, -`nvidia_modeset`/`nvidia_drm`) and the **video stack** (NVENC/NVDEC) — both -roadmapped. +the **graphics stack** (`libGLX_nvidia`, the driver's GL core, the Vulkan ICD, +`nvidia_modeset`/`nvidia_drm` — partially live on the emulated vGPU: +Xorg/GLX run, the engine executes GL method streams, and the Vulkan pipeline +runs end to end; rendered pixels are the frontier — +[`Graphics-Execution.md`](Graphics-Execution.md)) and the **video stack** +(NVENC/NVDEC) — roadmap. ## 2. Layering rules the emulation must respect @@ -91,14 +94,18 @@ open /dev/nvidia* → cuInit → UVM registration (per process) → faults, completions, events ``` -Context creation and the full CUDA driver-API smoke now pass (`ALL OK`, -copies matched) [live]. The current focus is execution: the backend is a -**module-owned GPU running its real GSP firmware as the resource-manager -server** ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)), with -CE-class copies forwarded to the physical copy engine default-off [gated]. -For GR, a forwarded guest launch reaches the front-end but the inline -launch-descriptor install has not been observed — the open execution wall -([`Execution-Frontier.md`](Execution-Frontier.md)) [wall]. +Context creation and the full CUDA driver-API smoke pass (`ALL OK`, copies +matched) [live], and guest kernel launches now **execute on the physical +engine** with engine-authored completions relayed to the guest [live]. The +backend is a **module-owned GPU running its real GSP firmware as the +resource-manager server** +([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). The leading edge is +the staged-transfer class, the every-boot acceptance campaign and per-guest +isolation ([`Execution-Frontier.md`](Execution-Frontier.md)). Graphics work +reuses the same submission machinery on its own frontier: the X server and +GLX run, the engine executes GL method streams, and the Vulkan pipeline runs +end to end; rendered pixels are the wall +([`Graphics-Execution.md`](Graphics-Execution.md)). ## 5. What the guest expects from the device, per phase @@ -135,7 +142,9 @@ underlying GSP device/firmware model but different host interfaces (WDDM, MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the firmware interface, forward the work. Windows support is a roadmap item ([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) [planned], and -the design goal is that no Windows-specific code lives in the guest. +the design goal is that no Windows-specific code lives in the guest. The +Windows/DirectX path — WDDM, the Direct3D user-/kernel-mode drivers, and what +a bring-up requires — is described on [`DirectX.md`](DirectX.md). ## 8. Compatibility notes for versions diff --git a/wiki/HAAEEP-and-Collaboration.md b/wiki/HAAEEP-and-Collaboration.md index 201d430..0e12636 100644 --- a/wiki/HAAEEP-and-Collaboration.md +++ b/wiki/HAAEEP-and-Collaboration.md @@ -8,7 +8,7 @@ > not a shipped product. When this page and the docs tree disagree, the docs > tree wins. Front door: [`../docs/README.md`](../docs/README.md). -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[proposal]` architecture and roadmap; `[target]` milestone to reach; `[goal]` end state; `[design]` integration point designed, not built. +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[proposal]` architecture and roadmap; `[target]` milestone to reach; `[goal]` end state; `[design]` integration point designed, not built. ## 1. What HAAEEP is @@ -97,7 +97,7 @@ isolation model is described in | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | rung 2 (runtime start) is green; CE-class work is forwarded with release and fault relay in place (default off); for GR the guest launch reaches the front-end but no grid has dispatched yet — the frontier is tracked in [`Execution-Frontier.md`](Execution-Frontier.md) | +| CUDA execution in a guest (rung 3) | green: verified guest CUDA clients execute on the physical engine with engine-authored completions relayed to the guest; the staged-transfer class, every-boot acceptance, per-guest isolation and the graphics pixel frontier remain — tracked in [`Execution-Frontier.md`](Execution-Frontier.md) and [`Graphics-Execution.md`](Graphics-Execution.md) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | diff --git a/wiki/Hardware-Software-Matrix.md b/wiki/Hardware-Software-Matrix.md index 0363636..697d92e 100644 --- a/wiki/Hardware-Software-Matrix.md +++ b/wiki/Hardware-Software-Matrix.md @@ -9,7 +9,7 @@ > When this page and the docs tree disagree, the docs tree wins. Front door: > [`../docs/README.md`](../docs/README.md). -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[reference]` continuously tested, bugs block releases; `[supported]` tested per release; `[experimental]` partial or bring-up; `[target]` designed for, bring-up in progress; `[design]` design direction, not the current build. +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[reference]` continuously tested, bugs block releases; `[supported]` tested per release; `[experimental]` partial or bring-up; `[target]` designed for, bring-up in progress; `[design]` design direction, not the current build. ## 1. Support ladder @@ -43,9 +43,9 @@ selection is compile-time today and the module is built for the TU104 profile. | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** — probe, `nvidia-smi` and the driver-API smoke pass; CE-class copies are forwarded default-off; for GR the guest launch reaches the front-end but no grid has dispatched yet (see [`Execution-Frontier.md`](Execution-Frontier.md)); no continuous guest testing yet | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** — probe, `nvidia-smi`, the driver-API smoke and CUDA kernel execution on the physical engine pass (verified clients green on fresh boots); the staged-transfer class and per-guest isolation are in progress; graphics partially: the X server and GLX run, the engine executes GL method streams, rendered pixels are the frontier (see [`Graphics-Execution.md`](Graphics-Execution.md)); no continuous guest testing yet | | Linux aarch64 | **Target** | -| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | +| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** — see [`DirectX.md`](DirectX.md) | | Other Unix (FreeBSD, ...) | Out of scope until demand | **Guest rule:** the guest never contains Open-IOV code — no guest agent, no diff --git a/wiki/Host-Module.md b/wiki/Host-Module.md index acf98ed..dbbe68c 100644 --- a/wiki/Host-Module.md +++ b/wiki/Host-Module.md @@ -6,7 +6,7 @@ > source: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md); > the docs tree wins on disagreement with this page. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with > the stock guest driver; `[wip]` demonstrated live, being productized; > `[gated]` implemented default-off behind supervised gates; `[wall]` known > blocking limit; `[target]` designed, not built; `[planned]` roadmap. @@ -106,9 +106,10 @@ Proxying through the distribution host driver has **no callable entry point from kernel context**, so the module **owns the device and runs its real GSP firmware as the resource-manager server**, with emulation as the control plane. CE copies, release and fault relay are implemented default-off and -verified live [gated]; GR execution is the open wall [wall] — a forwarded -launch reaches the front-end; the inline launch-descriptor install is unobserved -([`Execution-Frontier.md`](Execution-Frontier.md)). +verified live [gated]; forwarded guest CUDA launches execute on the engine +with engine-authored completions relayed to the guest, and graphics method +streams execute in two lanes ([`Execution-Frontier.md`](Execution-Frontier.md), +[`Graphics-Execution.md`](Graphics-Execution.md)). ## 6. Debug parameters diff --git a/wiki/Interfaces-and-Stability.md b/wiki/Interfaces-and-Stability.md new file mode 100644 index 0000000..44cade2 --- /dev/null +++ b/wiki/Interfaces-and-Stability.md @@ -0,0 +1,117 @@ +# Interfaces and Stability + +> **Scope.** The interfaces Open-IOV exposes or consumes, which of them are +> **contracts** (stable, versioned, safe to build on), which are **not** +> (bring-up and debug surfaces, free to change), and *why* the stable set is +> stable. This is the compatibility policy for the project's own software — the +> interfaces of *other people's* software are fixed by the project's rules +> (nothing is patched). Canonical sources: +> [`../docs/eng-ops.md`](../docs/eng-ops.md) (engineering rules), +> [`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md) +> (version/device profiles) and the headers/schemas named below; the docs tree +> wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. + +## 1. Why interfaces are the product + +Open-IOV's durability comes from a single idea: **own the interface, not a +snapshot of it**, and **never patch anyone**. The interfaces the project +*consumes* — the stock NVIDIA driver's device expectations, the kernel's VFIO/ +mdev UAPI, the hypervisor's DMA and interrupt interfaces — are therefore +immutable by construction: if the project changed what the guest driver sees, +the stock driver would break, and the project would have failed its own rule. + +The interfaces the project *exposes* are split deliberately into three +classes, and everything below is one of them: + +| Class | Promise | Examples | +|---|---|---| +| **Consumer contracts** | stable; versioned; changes are additive or bump a version | the guest-visible device interface, the mdev lifecycle surface, the userspace management library, the configuration schemas, tooling evidence formats | +| **Bring-up and debug surfaces** | *no* promise — names and semantics may change between builds | module parameters, debug filesystem files, internal state shapes | +| **Internal surfaces** | no promise, no consumers outside the module | kernel functions, structs, heuristics | + +## 2. Stable interfaces (contracts) + +| Interface | Consumers | Why it is stable | Change policy | +|---|---|---|---| +| **The guest-visible device interface**: PCI configuration space, BAR0 register interface, BAR1/BAR3 apertures, MSI-X; the GSP boot protocol; the GSP-RM message-queue transport and RPC functions; RM object/control semantics; events and completions | stock NVIDIA guest drivers (Linux and Windows) | *It is NVIDIA's interface*, fixed by the driver version the profile targets. The no-patch rule means changing it is not an option: any update must keep the unmodified driver working | Never changed by us. New driver/firmware versions enter as **profiles with measured deltas** ([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)); the emulation answers the version the driver negotiates | +| **The host/platform surface of `vgsp.ko`**: the VFIO **mdev lifecycle** (mdev type names such as `nvidia-vgpu-turing`, `-1g`, `-2g`; sysfs create/remove; the VFIO region, DMA and IRQ UAPI) | QEMU/libvirt and management tooling; any conforming VFIO consumer | It is the kernel's mdev/VFIO contract plus a small, documented set of type names that tooling (and user scripts) reference by name | Additive by default; renaming or removing a type requires a migration note and a supported overlap window | +| **The userspace management library** (`libvgpu` / `libiov`; header `userspace/include/libvgpu.h`): error codes, memory/type enums, VF, migration and WPR operations; versioned soname | project tooling and integrators writing management workflows | A small C API with a versioned shared-object name; its consumers link rather than copy | Additive changes keep the soname; breaking changes bump it. The surface is small on purpose — it is a management API, not the data path | +| **Configuration profiles** (`config/versions/*.yaml`, `config/hardware/*.yaml`): per-driver and per-device adaptation data | the module's version adaptation and the porting workflows | Data contract: the porting promise is "a new version is a profile plus a measured delta", which only holds if the schema is stable | Additive fields; semantic changes are versioned with the profile | +| **Developer-tooling evidence formats**: the recorded-session format (`greeneye-session/1`), the symbol database, the injection ABI (v1) | recorded evidence, replays and future sessions of the firmware workbench | Evidence must remain interpretable years later, so the formats are declared back-compatibility items even though the tooling itself is development-only | Version bumps with migration notes; additive fields preferred; a breaking change needs a shim or a format-version bump | +| **The documentation canon** | users, integrators, future contributors | The docs tree is the declared source of truth: where a wiki page or summary disagrees with the repository documents, the repository wins | Corrections update the docs tree first, then the derived pages | + +## 3. Explicitly not stable (may change) + +Nothing in this list is safe to build on; treat it as subject to change +between builds, without notice: + +* **Module parameters** (`/sys/module/vgsp/parameters/*`) — bring-up and debug + knobs (targeted probes, forced behaviours, diagnostic mirrors). They exist to + run experiments; names and semantics change as the canonical paths replace + them. +* **Debug filesystem surfaces** (the module's `gsp-phys-*` controls and + counters) — observability only; no format or lifetime promise. +* **`vgsp.ko` internals** — functions, structures, state layouts; there are no + external consumers and no stability promise. +* **Emulation internals** — ring/USERD discovery heuristics, shadow windows, + scratch mappings, scan cadences, and the parameter-gated forwarding and + graphics seams while they are validated. When a seam graduates, it becomes a + documented configuration surface and leaves this list. +* **Scripts, harnesses and test clients** (`scripts/`, `tools/`, + `tools/guest-diag/`) — working material, not products. +* **The firmware workbench's CLI, plugins and probe files** + ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) — development-only + by declaration; interfaces and file formats may change without notice. + (Exception: the *session format* above is versioned precisely so recorded + evidence survives tool change.) +* **This wiki and the site** — a curated, dated documentation view, refreshed + with status changes; it is not an API. + +## 4. Why the stable set is stable + +Three kinds of reason, and each stable interface has at least one of them: + +1. **It is externally specified.** The guest-visible device interface is + NVIDIA's; the mdev lifecycle is the kernel's. Stability is inherited, and + the project's rules forbid "fixing" either side. +2. **It has declared consumers.** The user library, the mdev type names, the + configuration schemas and the evidence formats have real consumers + (tooling, scripts, evidence archives). A contract exists because someone + else depends on it — not because it was convenient to freeze. +3. **The project's own rules make it so.** *No patches* + *profiles, not + rewrites* means the device-facing behaviour must evolve the way NVIDIA's + driver evolves — by version, never by surprise. + +**Change discipline, in one sentence:** prefer additive; version what +consumers link against or parse; never break evidence or an unmodified +driver; document the migration when something must move. + +## 5. Open items (stability questions we expect to close) + +* **Userspace library freeze point.** When the management workflows stabilise, + declare an API freeze (a 1.0 surface) so integrators can rely on it. +* **Graduating debug seams.** The parameter-gated forwarding and graphics + seams will migrate from debug knobs to documented configuration as they are + validated; the graduating list is tracked on + [`Project-Status.md`](Project-Status.md). +* **Schema formality.** The configuration profiles are documented data today; + a formal schema would make the "profile + measured delta" promise checkable + by tooling. +* **Evidence-format externalisation.** If parties outside the project archive + workbench sessions, the session and symbol formats need the same public + versioning treatment as the user library. + +## 6. Related pages + +* [`Project-Goals.md`](Project-Goals.md) — the goals this compatibility + policy serves. +* [`Host-Module.md`](Host-Module.md) — where the mdev surface and the debug + parameters live. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — how + profiles, versions and devices are handled. +* [`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md) — the development-only + tooling and its versioned session format. +* [`Performance.md`](Performance.md) — why debug surfaces are explicitly not + configuration. diff --git a/wiki/OpenGL.md b/wiki/OpenGL.md new file mode 100644 index 0000000..8264620 --- /dev/null +++ b/wiki/OpenGL.md @@ -0,0 +1,122 @@ +# OpenGL + +> **Scope.** How a stock guest OpenGL application's commands travel through the +> unmodified NVIDIA stack on the emulated vGPU, and the honest state of running +> OpenGL on it. The host-side execution machinery shared by all graphics APIs +> is on [`Graphics-Execution.md`](Graphics-Execution.md); CUDA compute is on +> [`Execution-Frontier.md`](Execution-Frontier.md). Canonical sources: +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) +> (the plan and milestones), +> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) +> (the measured walls) and +> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) +> (the guest stack). The docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the +stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` +defined success state. + +## 1. The guest OpenGL stack + +On Linux, an OpenGL application does not talk to the kernel driver directly: +it talks to the GL implementation, and — when a display is involved — through +the X server's GLX extension: + +```text +application (toolkit / SDL / GLFW / raw GLX) + -> libGL (dispatch) -> libGLX_nvidia (NVIDIA GLX client) + -> libnvidia-glcore (the vendor GL engine: state, programs, shaders, draws) + -> ioctls + mappings to nvidia.ko -> channels, buffers, synchronization +X server (Xorg) + -> GLX extension module (glxserver_nvidia) and the NVIDIA DDX (X driver) + -> owns the screen, drawables and the present path +``` + +Two forms of operation exist, and both are visible in the guest: + +* **Direct rendering** — the normal desktop case. The GL client builds GPU + commands in its own process and the X server coordinates buffers and + presents. The client's channel is the source of truth for per-draw state. +* **Indirect GLX** — the client's GL protocol is serialised to the X server, + which executes it against its own GL context. Here the X server's channel + carries the graphics stream. + +On the emulated vGPU the readable command source today is the **X server's +channel**, whose graphics-class (`0xC597`, `TURING_A`) sections the emulation +captures and executes ([`Graphics-Execution.md`](Graphics-Execution.md) §3). + +## 2. What an NVIDIA GL frame involves (observed shapes) + +* **Context and state setup** — shader programs (SASS) uploaded to device + memory, constant/descriptor banks, render targets and depth surfaces + allocated and mapped. +* **Per-draw work** — a graphics-class method stream: state binds (viewport, + render target, ZS, blend/ROP), program and constant binds, vertex/index + data, and draw-issuing methods that drive the fixed-function pipeline + (vertex fetch, primitive assembly, rasterisation, fragment work, ROP). +* **Program/descriptor delivery** — the driver uploads shader code and + descriptors through its copy/utility-channel inline-data path; this upload + is the piece the emulation must materialise for the engine to run *real* + programs ([`Vulkan.md`](Vulkan.md) has the mirror-image of the same gap). +* **Present** — the X server's present/readback path runs as light 2D-class + and copy-engine operations; the emulator decodes and executes those shapes. + +## 3. Status on the emulated vGPU + +Rated honestly, against the milestone ladder in the plan: + +| Item | State | +|---|---| +| X server initialises on the emulated vGPU (driver push-DMA sanity test, ScreenInit) | `[live]` | +| `glxinfo -B` reports the NVIDIA renderer (OpenGL 4.6.0, driver 570.144) | `[live]` | +| The X driver's RC-watchdog channel registers and its notifier is authored | `[live]` | +| GZDoom launches and runs on the emulated vGPU | `[live]` (menu/frames black) | +| X server's `0xC597` graphics section captured and decoded | `[live]` | +| Engine executes a launch built from the captured section (real `sm_75` SASS retired) | `[live]` | +| Verbatim 3D method-stream replay retired by the engine, no MMU fault | `[live]` | +| Direct GL client: device-init gate handled; client channels allocate; readback path returns with no GL error | `[wip]` | +| Direct GL client submits its per-frame draws | `[wall]` (no submissions observed) | +| Non-black rendered frames through the emulated vGPU | `[wall]` | +| 60 fps sustained, no forcing (M-GL4) | `[target]` | + +## 4. How OpenGL runs are verified + +Instruments that matter, in order of usefulness: the X server log and +`xdpyinfo` (did the server initialise?), `glxinfo -B` (which renderer is +real), the application's own progress output, and frame capture at the GL +source (a swap-hook capture is the ground truth for pixels — the X screen +itself is not, on some stacks). Captures are decoded offline and scored: +fraction of pixels above a luminance threshold, colour count, and +frame-to-frame motion. A run counts only when the capture shows motion with +the NVIDIA renderer in use. + +## 5. Milestones on this path + +The graphics plan defines M-GL1 (client context-init decodes and executes; the +client leaves its retry loop), M-GL2 (`glxgears` non-black), M-GL3 (GZDoom GL +non-black through the mdev), M-GL4 (60 fps, no forcing), M-GL5 (two VMs, two +mdevs), M-GL6 (split-screen video). The project sits **between M-GL1 and +M-GL2**: the interface work is done — the client creates its channels and its +readback path is clean — and pixels are the gate. + +## 6. What is not claimed + +* No OpenGL frame from the emulated vGPU has produced real rendered pixels + yet; every capture so far is black. +* No frame-rate or latency numbers. +* The published two-VM GZDoom demo videos used the labelled `virtio-gpu`/ + `virgl` path, not the emulated vGPU. + +## 7. Related pages + +* [`Graphics-Execution.md`](Graphics-Execution.md) — the shared execution + machinery and the ranked walls. +* [`Vulkan.md`](Vulkan.md) — the sibling API path, further along in the + pipeline lifecycle. +* [`DirectX.md`](DirectX.md) — the Windows/DirectX target. +* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — how the NVIDIA + userspace stack is analysed. +* [`Install-and-First-Run.md`](Install-and-First-Run.md) — bringing the stack + up. +* [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) — + the canonical plan, gap list and gates. diff --git a/wiki/Performance.md b/wiki/Performance.md new file mode 100644 index 0000000..b62d6a7 --- /dev/null +++ b/wiki/Performance.md @@ -0,0 +1,192 @@ +# Performance + +> **Scope.** How Open-IOV's performance is modelled, traced, and planned: the +> cost structure of the system, the metric set, what is observed today +> (honestly — there are no published numbers yet), the known gaps with their +> improvement plans, and the architecture questions the performance work will +> have to answer. Canonical sources: +> [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) +> (the measurement contract), [`../STATE-CURRENT.md`](../STATE-CURRENT.md) (the +> dated snapshot) and [`RUNBOOK.md`](../docs/RUNBOOK.md); the docs tree wins on +> disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` measured with +working code; `[wip]` measurement framework partial; `[planned]` designed, +not built. + +## 1. The performance model + +The cost structure of a vGPU session splits into four layers, and each is +measured separately: + +| Layer | What it costs | Representative metric | +|---|---|---| +| **Initialization** | VM boot → driver probe → `cuInit` | `T_init` | +| **Control plane** | Resource-manager request/reply round trips (objects, controls, memory) | RPC latency p50/p99 | +| **Steady-state data path** | Submission → decode → forward → engine → completion → guest wake-up | push overhead; completion latency; kernel throughput | +| **Data movement** | Copies between guest memory and device memory | copy bandwidth (H2D/D2H/D2D) | + +Presentation (3D frames) sits on top of the data path, with its own budget +(frame pacing, not just frame rate). + +Three project rules govern every number: + +1. **Correctness before speed.** A change that makes the vGPU faster but + breaks the acceptance corpus does not count. +2. **Proxy first, optimise by measurement.** The architecture deliberately + prototyped the safe path (emulate the interface, forward the work through + the module's own GSP client) before considering a hot path; whether to move + the hot path closer to direct submission is a *measurement decision*, not a + design preference. +3. **No number without methodology.** Numbers are only quoted with their + harness, versions, clocks and warm/cold state, and ideally as a ratio + against a same-guest, same-hardware baseline (the A/B method below). + +## 2. The metric set + +| Metric | Definition | Why it matters | +|---|---|---| +| `T_init` | VM boot → `cuInit` returns | user-visible startup | +| RPC latency p50/p99 | request write → reply visible in the status queue | init and management overhead | +| Push overhead | guest submission → engine start (forwarded) | steady-state latency | +| Semaphore completion latency | release written → tracker observes | per-operation sync cost | +| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality | +| Kernel throughput | kernels/second for a fixed workload | launch overhead | +| Multi-tenant fairness | per-tenant progress under contention | isolation quality | +| Frame pacing | inter-frame deltas under load | graphics smoothness (p95 is the budget) | + +## 3. How performance is traced today + +* **Host-side telemetry.** The module timestamps its own work: per-run + counters in the kernel log (accepted / failed / retried / refused work, + decode outcomes, forward outcomes), `trace_printk`/tracepoints at dispatch + entry/exit for distributions, and counter surfaces readable through the + kernel's tracing and debug filesystems. Module parameters exist for targeted + probes during bring-up — they are debug surfaces, not configuration + ([`Interfaces-and-Stability.md`](Interfaces-and-Stability.md)). +* **Guest-side instruments.** The guest test clients print their own + acceptance *and* timing lines (they are the ground truth for what the + application observed); guest-side kernel probes and ptrace tools attribute + driver-side waits when a client stalls. +* **The A/B harness.** Every measurement runs twice on the same machine and + preferably the same boot: **A** = the workload on the physical GPU with the + stock driver; **B** = the identical workload in the guest on the emulated + vGPU. Results are reported as **B/A ratios**; warm and cold, with variance + and run counts. One variable at a time; the harness is published so results + are comparable. +* **Offline tools — with a hard boundary.** The risc-v-gsp-rm-debugger + workbench models *instruction semantics and memory behaviour only*; it makes + **no timing claims** (its fidelity levels explicitly exclude time and + concurrency). Timing questions are answered on hardware, with + instrumentation. +* **The evaluation ladder.** Performance is only meaningful from the + *execution* rung upwards: initialization → runtime start → execution → + sustained → multi-tenant. Claims are made per rung, never across it. + +## 4. Where it stands today (honest) + +**There are no published performance numbers yet.** The framework above is +the contract; the first formally recorded measurements are scheduled, not +done. What *is* observed today, clearly labelled as internal observation +rather than benchmark: + +* **Forwarded CUDA launch latency is on the order of hundreds of + milliseconds** end-to-end on the development setup (one ten-client arc + measured p50 ≈ 650 ms). This is the dominant known cost, dominated by the + submission trap → decode → forward chain and by polled completion waits — + not by the engine itself. +* **Data movement is verified correct, not yet measured for throughput**: + copies through the forwarded path round-trip byte-exact; the A/B bandwidth + runs (the `nvbandwidth`-class suite) are pending. +* **Graphics frame delivery is far below interactive rates** in the recorded + pipeline runs — the defined target is 60 fps *sustained with no forcing* + ([`Graphics-Execution.md`](Graphics-Execution.md)); today's frame numbers + come from development instrumentation and are explicitly not a benchmark. +* **Initialization is functional**: the backend's GSP boot reaches init-done + in about a second on the reference hardware; `T_init` as a guest-visible + measurement has not been taken. +* **The software path is already cheap where it is work-shaped**: decoded + memory operations execute at host memory speed; control-plane replies are + produced inline in the emulation thread; the expensive parts are the + *control-heavy* structures (per-push passes, scanning, polling). + +## 5. Observed gaps and the plan for each + +| Gap | Observation | Plan | +|---|---|---| +| Forwarded-launch latency | hundreds of ms (internal); dominated by the multi-step forward chain and completion polling | instrument the distribution end to end; reduce per-push passes; move completion to event-driven paths; re-measure per step | +| Completion latency | releases are detected by polling the engine/guest memory | prefer engine-authored release paths already used by verified launches; doorbell/event wake-ups instead of periodic scans where possible | +| RPC dispatch cost | all management funnels through one emulation thread | build the RPC round-trip microbenchmark first; batch and prioritise; reduce wake-ups | +| Scanning overhead | per-pass discovery/scan work (guest RAM, shadow apertures) exists for correctness while registrations are incomplete | convert discovery to registered/learned state so per-pass cost scales with *work*, not with memory size | +| BAR0 shadow reads | uncached to the guest, as MMIO must be; bulk reads would be slow | if a consumer bulk-reads, introduce cache-safe sub-regions rather than a slower guest | +| Memory-model gap | guest VRAM is backed by host RAM today | real VRAM backing is a target; DMA-heavy workloads are not representative until it exists | +| Graphics frame path | far below 60 fps; forcing/diagnostic assistance excluded by the target definition | the graphics milestone ladder: natural completions with no mirror, then measure pacing | +| Multi-tenant fairness | two guests coexist; fairness under load is untested | define the fairness metric, run two-VM contention, then add scheduling policy | +| Initialization | functional but unmeasured | take `T_init` and the reply-fidelity tail as the next control-plane measurement | + +## 6. The improvement programme + +1. **Build the microbenchmarks first** — the four the measurement contract + names: RPC round-trip, MSI-X delivery latency, BAR0 shadow read latency, + and doorbell-polling cost. These attribute overhead to components before + any optimisation. +2. **Run and publish the A/B corpus** — the canonical suite (vector-add, + SGEMM, copy bandwidth, a GL frame-rate app, a framework op loop) with the + contract's pass criteria (e.g. ≥95% B/A for compute, ≥90% for GL and + framework loops) and full environment capture. +3. **Optimise the observed hot structures** — fewer per-push passes, batched + decode, event-driven completion, shadow-window reuse — each as an A/B + change with its numbers kept. +4. **Evaluate the hot-path placement decision** — keep the module as a GSP + client proxy, or move submissions closer to direct channels — using the + measured budgets as the criterion. +5. **Publish the first graphics measurement** — 60 fps sustained, no forcing, + p95 inter-frame pacing — as the first end-to-end performance claim. +6. **Then multi-tenant**: define fairness, measure it, and only then claim it. + +## 7. Architecture areas that may need to improve — open questions + +These are the structural questions the performance programme exists to close: + +* **One ordered workqueue, one in-flight request.** The current design + serialises forwarded work for ordering safety. Is that a throughput ceiling + for multi-channel/multi-launch workloads, and how much pipelining is + possible without weakening the ordering guarantees the guest observes? +* **Per-pass scanning vs registered state.** Ring/USERD discovery, content + scans and shadow diffs are correctness mechanisms; can they be replaced (or + narrowed) by registrations learned on first sight, so steady-state cost does + not scale with aperture size? +* **Completion publication machinery.** Which of poll, engine-authored + release, notifier and event paths should carry the steady state — and what + is the p99, not just the median? +* **MMIO trap frequency.** Doorbells and shadow writes pay a VM-exit cost; + which guest patterns produce the most traps, and can they be coalesced or + served with less work? +* **The transport itself.** Guest-system-memory message queues plus doorbells + are deliberately conservative; would a shared-ring fast path change the + budget materially? +* **Hot-path placement.** Proxy through the module's own GSP session vs + moving hot submissions closer to the hardware — decided by the A/B numbers, + not by preference. +* **Multi-tenant scheduling.** Time-slicing exists only as a default; the + fairness policy (and MIG-backed partitioning where available) is a design + area, with fairness measurements defined before the claim. +* **Fidelity boundary.** The offline simulator can never answer timing or + concurrency questions; the performance programme therefore owns all timing + claims, and no timing claim may cite a simulator session. + +## 8. Related pages + +* [`Interfaces-and-Stability.md`](Interfaces-and-Stability.md) — which + surfaces are contracts and which are free to change. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the compute path whose + latency this page budgets. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the 60 fps target and its + milestone ladder. +* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work + is forwarded to. +* [`Host-Module.md`](Host-Module.md) — the module whose threads, queues and + scans carry the costs above. +* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — + the measurement contract: ladder, metrics, A/B method, microbenchmarks, + targets. diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md index 1b83203..976430d 100644 --- a/wiki/Physical-GPU-Backend.md +++ b/wiki/Physical-GPU-Backend.md @@ -5,7 +5,7 @@ > guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) > and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partially built; `[wall]` current blocker; `[target]` planned end state. @@ -64,13 +64,15 @@ independently verified: 5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]` 6. **Work path** — decoded guest pushes become real push streams; resolved host pages - map through module-built scratch page tables; the firmware schedules the work. `[gated]` + map through module-built scratch page tables; the firmware schedules the work. + `[live]` observed end to end for the verified guest kernel clients. 7. **Completion relay** — hardware-completed semaphores land in guest-polled memory; - events and faults relay through the existing MSI-X injection path. `[gated]` + events and faults relay through the existing MSI-X injection path. + `[live]` engine-authored completions relay to the guest and its waits resolve. ## 5. The two planes and the wiring -The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software. +The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software. ```text GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU @@ -85,8 +87,9 @@ INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live] RM battery, UVM registration [live] WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell host pump decodes; eligible CE copies forwarded via module-built scratch - page tables; GR: launch reaches the front-end, descriptor not seen [wall] -DONE engine authors the release in guest memory; host relays via MSI-X [gated] + page tables; GR: launches execute on the engine, completions relay + to the guest [live] +DONE engine authors the release in guest memory; host relays via MSI-X [live] TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown -> protected region down -> distribution driver rebound [live] FAIL a failed mutating operation parks the device in a locked fail-safe; @@ -104,17 +107,25 @@ FAIL a failed mutating operation parks the device in a locked fail-safe; | Power domain held active across the ownership window | verified across supervised cycles `[live]` | | Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | | Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | -| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` | -| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` | +| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the staged-transfer class is under repair `[live]` | +| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest | +| Graphics method-stream execution (verbatim 3D replay, synthesized launches) | `[wip]` — the engine retires real graphics streams and launched kernels; rendered pixels are the frontier ([`Graphics-Execution.md`](Graphics-Execution.md)) | ## 7. The execution frontier The RM client ladder reaches the compute engine (a GR0 channel plus a -compute-class object). A forwarded guest CUDA launch now -reaches the front-end (ring entry fetched, methods executed) but the inline -launch-descriptor install has not been observed and no grid has dispatched; -context validity at dispatch is ruled out. The wall is the form of the -forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md). +compute-class object). Forwarded guest CUDA launches execute on the +engine: the inline launch descriptor installs and is read, the grid +dispatches, the engine authors the completion, the host relays it, and the +guest's synchronisation wait resolves — verified on fresh boots with the +vector-add, matrix-multiply and transformer clients. Graphics work rides the +same machinery in two lanes (a verbatim 3D method-stream replay and +synthesized launches) and executes on the engine; producing rendered pixels +is the graphics frontier. The remaining compute work is the staged-transfer +class (under repair), the every-boot acceptance campaign, and per-guest +execution isolation — see +[`Execution-Frontier.md`](Execution-Frontier.md) and +[`Graphics-Execution.md`](Graphics-Execution.md). A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step. @@ -147,6 +158,9 @@ fail-safe that is never auto-retried; recovery is a supervised step. access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). +* Multiple mediated devices with different emulated VRAM sizes and multiple + guests already coexist at the device level (two guests demonstrated); + per-guest execution isolation for simultaneous CUDA guests is in progress. ## 10. Open questions @@ -159,7 +173,7 @@ fail-safe that is never auto-retried; recovery is a supervised step. * [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up. * [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design. -* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane. +* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane. * [`Host-Module.md`](Host-Module.md) — the module that owns both planes. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings. * [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture. diff --git a/wiki/Project-Goals.md b/wiki/Project-Goals.md new file mode 100644 index 0000000..bbce3e3 --- /dev/null +++ b/wiki/Project-Goals.md @@ -0,0 +1,106 @@ +# Project Goals + +> **Scope.** What Open-IOV is for, what it is not for, and the dated status of +> each goal. Condensed from +> [`../docs/01-orientation/01-purpose-and-scope.md`](../docs/01-orientation/01-purpose-and-scope.md) +> (mission, scope and success criteria) and +> [`../docs/README.md`](../docs/README.md) (the project front door); the docs +> tree wins on disagreement. The live milestone ladder is on +> [`Project-Status.md`](Project-Status.md), and the current execution detail on +> [`Execution-Frontier.md`](Execution-Frontier.md) and +> [`Graphics-Execution.md`](Graphics-Execution.md). + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` working with the +stock guest driver; `[wip]` in progress; `[planned]` designed, not built; +`[target]` bring-up ahead; `[non-goal]` explicitly out of scope. + +## 1. Mission + +**Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without +modifying NVIDIA software, firmware or hardware.** + +Concretely, Open-IOV builds an independent, maintained IP block that: + +* presents a virtual NVIDIA GPU to a guest operating system; +* lets the guest run the **stock NVIDIA driver** (Linux and Windows); +* supports **compute (CUDA)** first, then **3D graphics** and **video**; +* runs on ordinary NVIDIA GPUs already deployed in the field; +* isolates guests from each other and from the host; +* and requires no license key, no proprietary host service, no patched guest + driver, and no specific hypervisor. + +The guiding sentence is: ***Emulate the interface. Pass through the work. +Never patch anyone.*** + +## 2. The gap it fills + +| Approach | Limitation | +|---|---| +| Discrete device assignment (VFIO passthrough) | One GPU per VM; no sharing; fragile on consumer/laptop systems. | +| NVIDIA proprietary vGPU | Requires a license and a vendor host stack; driver-version locked. | +| Software GPU emulation (QEMU virtual GPUs, `virgl`) | No real CUDA; no real 3D performance; not usable for AI. | +| Cloud-only inference | Requires an external service; unacceptable for secure edge, offline, or private data. | + +Open-IOV occupies the gap: **real GPU acceleration, multiple VMs, stock +drivers, no license.** + +## 3. Goals and their status + +| Goal | What success looks like | Status (2026-10-30) | +|---|---|---| +| Stock guest driver initialisation | The unmodified driver probes, initialises and enumerates as on bare metal (`nvidia-smi`) | `[live]` — TU104 reference platform | +| Full CUDA driver-API surface | Context creation, allocation, copies, free, destroy complete with correct results | `[live]` — the driver-API smoke prints `ALL OK`; the 64-byte payload round-trips | +| CUDA kernel execution on real hardware | Verified guest kernels execute on the physical GPU, with engine-authored completions waking the guest | `[live]` — vector-add, matrix-multiply and transformer clients green on fresh boots; the staged-transfer class, the every-boot acceptance campaign and per-guest isolation are in progress ([`Execution-Frontier.md`](Execution-Frontier.md)) | +| OpenGL (3D graphics) | Stock guest GL applications render real pixels through the emulated vGPU | `[wip]` — Xorg/GLX run, the engine executes GL method streams (verbatim 3D replay + synthesized launches); rendered pixels are the frontier ([`OpenGL.md`](OpenGL.md)) | +| Vulkan | Stock guest Vulkan applications render real pixels | `[wip]` — the pipeline runs end to end (device → command buffer → fence → readback); frames are black — program delivery and program-VA mapping are the frontier ([`Vulkan.md`](Vulkan.md)) | +| Windows guests (DirectX, CUDA) | A stock WDDM driver and stock D3D/CUDA applications run in Windows guests | `[target]` — not started; same emulation target ([`DirectX.md`](DirectX.md)) | +| Video (NVENC/NVDEC) | Encode/decode through the emulated device | `[planned]` | +| Multi-tenant isolation | Multiple VMs on one GPU, isolated by the IOMMU and the device model, with explicit scheduling | `[wip]` — two mediated devices (different emulated VRAM sizes) and two guests coexist; per-guest execution isolation is in progress ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)) | +| Hypervisor portability | The same IP block attaches to KVM, Xen, Argo/OpenXT, Qubes OS Air, ... | `[planned]` — the emulation core is hypervisor-independent by construction; KVM/QEMU with VFIO mdev is the reference transport ([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) | +| Version and hardware portability | New driver, firmware or GPU generations enter by configuration plus measurement | `[wip]` — 570.144/TU104 is the anchor; per-version evidence workflows exist; newer versions are gated work | +| Live migration | A VM migrates with its GPU state | `[planned]` — the protocol analysis is kept for the record ([`Migration-Protocol.md`](Migration-Protocol.md)) | +| Performance | Within an agreed factor of native for the supported cases, with published methodology | `[planned]` — no published numbers yet; the first scheduled measurement is the 60 fps graphics target ([`Graphics-Execution.md`](Graphics-Execution.md)) | +| Security posture | Isolation survives adversarial review; the device fails closed | `[wip]` — fail-closed design and audit practice; per-tenant execution isolation is in progress ([`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md)) | + +## 4. How success is judged + +The project evaluates itself on five axes, each with a methodology document +in [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md): + +1. **Initialization** — the driver reaches bare-metal parity through boot, + probe and initialisation. +2. **Runtime** — the guest runs real work: CUDA kernels with correct results, + OpenGL/Vulkan rendering, encode/decode; repeated, sustained, resilient. +3. **Interoperability** — no changes anywhere: guest driver, host driver, + firmware, hypervisor. +4. **Stability** — crash-free under stress: start/stop cycles, VM kill, driver + reload, suspend/resume, migration. +5. **Performance and security** — within an agreed factor of native, with + isolation that survives adversarial review. + +## 5. Non-goals + +* **Replacing bare-metal inference serving.** Continuous batching on + bare-metal hosts remains the right answer for centralised datacenter + inference; Open-IOV is for compute that must live *inside* guests. +* **Forking or patching** the NVIDIA driver (guest or host), the GPU firmware, + or hypervisors to make the device work. +* **Anti-cheat / DRM circumvention.** The project interoperates with + interfaces and implements **its own** device; it does not counterfeit + vendor licensing. +* **Emulating GPU instruction execution in software.** Software emulation of + CUDA kernels is orders of magnitude too slow; work must run on real silicon. + +## 6. Related pages + +* [`Project-Status.md`](Project-Status.md) — the dated milestone ladder. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the compute execution + frontier. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the 3D execution + frontier. +* [`HAAEEP-and-Collaboration.md`](HAAEEP-and-Collaboration.md) — the + agent-edge use case and collaboration models. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — the support + ladder. +* [`../docs/01-orientation/01-purpose-and-scope.md`](../docs/01-orientation/01-purpose-and-scope.md) — + the canonical purpose-and-scope document. diff --git a/wiki/Project-Plan.md b/wiki/Project-Plan.md new file mode 100644 index 0000000..697e95f --- /dev/null +++ b/wiki/Project-Plan.md @@ -0,0 +1,136 @@ +# Project Plan + +> **Scope.** The sequence of work: the phases, the gates that close them, what +> comes next, and how the plan itself is maintained. This is the *roadmap* +> view — [`Project-Goals.md`](Project-Goals.md) says what the project is for, +> [`Project-Status.md`](Project-Status.md) says where it stands today, and this +> page says in what order the remaining work happens and what must be true +> before each step. Canonical sources: the compute milestone ladder in +> [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md), +> the graphics ladder in +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md), +> the five-rung evaluation ladder in +> [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md), +> and the porting sequence in +> [`../docs/06-portability/03-device-validation-and-porting-sequence.md`](../docs/06-portability/03-device-validation-and-porting-sequence.md); +> the docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[done]`, `[active]`, +`[ahead]` — the plan's own state vocabulary, distinct from the goal tags. + +## 1. How the plan works + +Five rules shape every phase below, and they are the project's contract with +itself: + +1. **Every milestone has a gate and an acceptance signal, named before the + work starts.** A wall is closed only by its acceptance signal — not by an + intention, a decompiler reading, or a lucky run. +2. **The evaluation ladder is the spine.** A platform advances + initialization → runtime start → execution → sustained → multi-tenant, and + nothing claims a rung it has not reached and reproduced. +3. **Correctness before speed; no number without methodology.** Performance + claims carry their harness, versions and A/B baseline. +4. **Default-off until validated.** New mechanisms land behind parameters + (the forwarding and graphics seams today) and graduate only after their + gate passes; graduation replaces the knob with documented configuration. +5. **No patches anywhere, and profiles over rewrites.** Guest drivers, host + drivers, firmware and hypervisors stay stock; a new driver, firmware or GPU + enters the plan as a *profile plus a measured delta*. + +## 2. The phases + +| Phase | What it delivers | Exit gate | State (2026-10-30) | +|---|---|---|---| +| **P0 — Platform** | Emulated vGPU device, GSP boot and GSP-RM emulation, stock guest driver initialisation | Driver-API smoke green (`ALL OK`) = evaluation rung 2 | `[done]` | +| **P1 — First execution (compute)** | Guest CUDA kernels executing on the physical GPU, with engine-authored completions; then hardening | Verified clients + engine-authored completion = rung 3 core; hardening: staged-transfer class, every-boot acceptance, per-guest isolation | `[active]` — execution green; hardening in progress | +| **P2 — Graphics execution** | OpenGL/Vulkan draw commands executed by the physical 3D pipeline and delivered as readable pixels | The graphics ladder: M-GL1 client context-init executes; M-GL2 `glxgears` non-black; M-GL3 GZDoom GL non-black through the mdev | `[active]` — between M-GL1 and M-GL2; the pixel walls are named | +| **P3 — Sustained and multi-tenant** | Frame rates without assistance; multiple VMs sharing the GPU with measured fairness; soak-stable operation | Graphics M-GL4 (60 fps, no forcing) and M-GL5 (two VMs, two mdevs); CUDA battery + soak; per-guest execution isolation = rungs 4/5 | `[ahead]` | +| **P4 — Breadth** | Video (NVENC/NVDEC), Windows/DirectX guests, live migration, non-KVM hypervisor transports | Per-case bring-up gates plus the matrix's evidence expectations | `[ahead]` | +| **P5 — Scale and portability** | New GPU generations and driver families carried forward; published performance programme | A profile + measured delta per version, validated on hardware; A/B numbers published per the measurement contract | `[ahead]` | + +The phases overlap on purpose: the graphics workstream is what unblocks P3's +first exit gate, and portability work starts from whichever hardware becomes +available (P5 runs across the whole plan). + +## 3. Where the plan stands, and what is next + +Ranked, in dependency order: + +1. **Graphics pixels (P2).** Three walls, each with a named next fix: the GL + client's submissions; program (SASS) delivery plus program-VA mapping; then + non-black frames. Full detail: + [`Graphics-Execution.md`](Graphics-Execution.md) §6. +2. **Compute hardening (P1).** The staged-transfer class (failure modes proven, + fixes in flight); the every-boot acceptance campaign; per-guest execution + isolation. Detail: + [`Execution-Frontier.md`](Execution-Frontier.md). +3. **The visible sequence after these**: first non-black frame → 60 fps in one + VM with no forcing → two VMs on two mdevs → the CUDA battery 15/15 plus a + soak → video → Windows. +4. **Then P4/P5 row by row**: each item ships with its own gate, profile and + evidence, and only then moves in the matrix. + +Recently closed, as evidence the gates work: the boot-poisoning defect and the +post-reboot client-stall defect (compute reproducibility); the X/GL wall (the +X server and GLX run on the emulated vGPU); the engine executing the real GL +method stream; and the guest Vulkan pipeline running end to end. + +## 4. The gates + +Each gate's full text lives in its canonical document; this is the map: + +| Gate | What must be true | Where it is defined | +|---|---|---| +| **Rung gate** | The previous rung is green and reproducible for this platform before the next is claimed | Evaluation ladder | +| **Execution gate** | Engine-authored release plus verified guest results, reproduced across fresh boots | Execution frontier | +| **Graphics gate** | Milestone definitions: capture shows motion, no forcing, no diagnostic mirror, renderer is the NVIDIA driver | Graphics plan (§C) | +| **Performance gate** | A/B ratios from the published harness; no absolute numbers; variance and run counts reported | Measurement contract | +| **Portability gate** | Profile fields recorded, firmware/device pairing hashed, first-touch differences explained, supervised physical validation done | Porting sequence + matrix | +| **Isolation gate** | Fail-closed enforcement; fairness measured before any multi-tenant claim | Security document | +| **Security gate** | Adversarial review before multi-tenant or production claims | Security document | + +## 5. Workstreams (always on) + +| Workstream | What it does | Entry point | +|---|---|---| +| Correctness and acceptance | The regression battery, battery-first ordering, regression gates | [`Execution-Frontier.md`](Execution-Frontier.md) | +| Graphics execution | The execution lanes, capture/decode, the M-GL ladder | [`Graphics-Execution.md`](Graphics-Execution.md) | +| Performance measurement | Microbenchmarks first, then the A/B corpus, then structural optimisation | [`Performance.md`](Performance.md) | +| Portability and versions | Profiles, measured deltas, the validation sequence; new driver/firmware families behind a review gate | [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) | +| Developer tooling | The firmware workbench: recorded sessions as evidence, offline-first investigation of walls | [`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md) | +| Security and isolation | Threat model, audits, fail-closed posture | [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) | +| Documentation and evidence | Docs tree canonical; dated status; replayable evidence for every claim | [`../docs/README.md`](../docs/README.md) | + +## 6. Contributing to the plan + +* **The unit of contribution is a profile plus evidence**, evaluated through + the five-rung ladder — see "Adding a new combination" in + [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) and the porting + sequence document. +* **Nasty bugs** (non-reproducible, cross-subsystem, blocking a gate) get a + tracked issue in the repository; ordinary work stays in the live handoffs. +* **Support claims** are made only by recording hardware + driver + firmware + + rung + date in the matrix with the evidence bundle behind them. + +## 7. How this plan is maintained + +The plan is a living sequence, not a dated roadmap: each milestone's gate is +owned by its canonical document, the current state is tracked on +[`Project-Status.md`](Project-Status.md), and the plan changes only when a gate +is named and its evidence rules are written down. When a wall closes, the +canonical doc is updated first and the site pages follow on the next refresh. + +## 8. Related pages + +* [`Project-Goals.md`](Project-Goals.md) — the goals and their status. +* [`Project-Status.md`](Project-Status.md) — the dated milestone ladder. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the compute frontier and + its gates. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the graphics frontier and + the M-GL ladder. +* [`Performance.md`](Performance.md) — the measurement programme. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — the + portability plan and support ladder. +* [`../docs/README.md`](../docs/README.md) — the documentation front door and + the project's own status table. diff --git a/wiki/Project-Status.md b/wiki/Project-Status.md index d6e60d1..944e525 100644 --- a/wiki/Project-Status.md +++ b/wiki/Project-Status.md @@ -10,7 +10,7 @@ > (the execution frontier). When this page and the docs disagree, the docs tree > wins. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or behind supervised gates; `[wip]` partially implemented; `[wall]` current blocker; `[target]` named success state; `[planned]` designed, not built. @@ -24,11 +24,13 @@ named success state; `[planned]` designed, not built. | GSP-RM message queues and RPC transport | `[live]` working | | Stock guest driver probe and `nvidia-smi` (TU104) | `[live]` working | | CUDA userspace init and the driver-API smoke (`cuInit`, device enumeration, `cuCtxCreate`, allocation, copies, free, destroy) | `[live]` working — the smoke prints `ALL OK` and the 64-byte payload round-trips exactly | -| Guest CUDA kernel execution on real hardware | `[wall]` a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue | -| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above | -| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` designed; the baseline service-state ownership work is in progress | -| OpenGL / Vulkan (3D), video | `[planned]` | -| Windows guests | `[planned]` (target) | +| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. A ring-block ownership defect that could stall the first client after a reboot was root-caused and fixed. Remaining: the staged-transfer class (some large multi-chunk clients still mis-verify; the failure modes are proven and under repair) and per-guest execution isolation for simultaneous CUDA guests | +| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend, and graphics method streams execute on the engine in two lanes ([`Graphics-Execution.md`](Graphics-Execution.md)) | +| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress | +| OpenGL (3D graphics) | `[wip]` — the X server and GLX run on the emulated vGPU with the stock driver, the engine executes real GL method streams (verbatim 3D replay plus synthesized launches), and a direct GL client creates its channels and readback path; rendered (non-black) frames are the frontier ([`OpenGL.md`](OpenGL.md)) | +| Vulkan (3D/compute) | `[wip]` — the guest Vulkan pipeline runs end to end on the emulated vGPU (device, allocation, pipeline, command buffer, fence, queue submit, readback) and the engine executes translated launches; frames are still black — program delivery and program-VA mapping are the frontier ([`Vulkan.md`](Vulkan.md)) | +| DirectX / Windows guests | `[planned]` (target) — no Windows guest booted yet; the architecture is the same emulation target ([`DirectX.md`](DirectX.md)) | +| Video (NVENC/NVDEC) | `[planned]` | | Xen / Argo / OpenXT / Qubes Air transports | `[planned]` (the emulation core is hypervisor-independent by construction) | | Live migration | `[planned]` | | Performance numbers | `[planned]` — no published numbers yet; any claim will ship with its measurement methodology | @@ -43,23 +45,36 @@ The project's definition of done for CUDA requires all of the following 2. Context creation and the full driver-API surface (`cuMemAlloc`, copies, free, destroy) — done. 3. A guest kernel launch that **executes on real hardware** and is proven by an - engine-authored completion — the current frontier. + engine-authored completion — demonstrated: the verified clients execute on + the physical engine and the engine-authored completions relay to the guest. 4. Memory the guest's work refers to resolving correctly end to end (the VA↔PA model) — partially, for the paths exercised so far. -5. Correct completion, semaphore and fault semantics back to the guest — CE - class work has the relay; GR is open. -6. Reproducibility: every boot, not a lucky one — an open issue. -7. No ad-hoc default-off parameters required for the canonical path. +5. Correct completion, semaphore and fault semantics back to the guest — the + GR completion path now relays (engine-authored releases wake the guest's + synchronisation waits); fault semantics remain partial. +6. Reproducibility: every boot, not a lucky one — two root causes landed: the + boot-poisoning defect (the wrong status-queue consumption cursor could + desynchronise the reply ring under load) and the ring-block ownership + defect that could stall the first client after a reboot. Green clients are + reproduced across fresh boots; a formal every-boot acceptance campaign is + the remaining work. +7. No ad-hoc default-off parameters required for the canonical path — still + open (the forwarding and graphics seams remain parameterised while they + are validated). -Until every gate holds, the honest formulation is: **initialization and the -driver-API path work; engine execution is in progress.** +Until every gate holds, the honest formulation is: **initialization, the +driver-API path and kernel execution on the physical GPU work and verify; +the staged-transfer class, per-guest isolation and the graphics pixel +frontier remain.** ## 3. How to see it yourself The reference setup takes about half an hour from a fresh host ([`Install-and-First-Run.md`](Install-and-First-Run.md)); the current expected outcome is exactly the ladder above — driver probe, `nvidia-smi`, the full -driver-API smoke, and the execution frontier where it stands. The +driver-API smoke, and guest CUDA kernels executing and verifying on the +physical GPU. On the graphics side the same setup brings up the X server and +GLX on the emulated vGPU ([`OpenGL.md`](OpenGL.md)). The risc-v-gsp-rm-debugger workbench ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) is the host-side tooling used to study the firmware side of that frontier. @@ -68,7 +83,7 @@ tooling used to study the firmware side of that frontier. Every claim above carries a date and a tag, maps to a maintained document in the repository, and is refreshed as milestones land. The execution story, with -its evidence and its open theories, is on +its evidence, its resolved theories and its open work, is on [`Execution-Frontier.md`](Execution-Frontier.md). ## 5. Related pages diff --git a/wiki/Riscv-GSP-RM-Debugger.md b/wiki/Riscv-GSP-RM-Debugger.md index 645f4a8..8155364 100644 --- a/wiki/Riscv-GSP-RM-Debugger.md +++ b/wiki/Riscv-GSP-RM-Debugger.md @@ -4,7 +4,7 @@ > > **Development-only.** Host-side tooling that is never shipped, never a driver dependency and never present in a guest; interfaces and file formats may change without notice. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` recorded QEMU sessions; `[design]` device model and custom simulator are roadmap. +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` recorded QEMU sessions; `[wip]` partial device model; `[design]` full device model and custom simulator are roadmap. ## 1. What it is @@ -21,7 +21,15 @@ Its purpose is **capturing how the firmware server behaves, repeatably and safel ## 2. What it is not -* **Not a GSP boot environment.** QEMU `virt` has no falcon windows, WPR2, CSB/TRANSCOM, PMU, timers or FB DMA. `probe` reports exactly where the real image first diverges (for the TU104 570.144 image: a load-access fault at firmware vaddr `0x4009014` reading `0x2000000000000A00`) — expected data, not a boot. A device model is a roadmap item, not a claim. +* **Not a complete GSP boot environment.** QEMU `virt` has no falcon windows, + WPR2, CSB/TRANSCOM, PMU, timers or FB DMA. `probe` reports exactly where the + real image first diverges (for the TU104 570.144 image: a load-access fault + at firmware vaddr `0x4009014` reading `0x2000000000000A00`) — expected data, + not a boot. A **partial device model** ("S1") exists and boots the real + image natively through its relocation copy on the simulator's boot ladder + ([`../debug/risc-v-gsp-rm-vm/docs/36-GSP-DEVICE-MODEL.md`](../debug/risc-v-gsp-rm-vm/docs/36-GSP-DEVICE-MODEL.md)); + the message-loop rungs and the remaining device environment are roadmap, + not claims. * **Not a timing model.** Do not infer latency, jitter or concurrency behaviour from it; its in-scope conclusions are execution flow and memory side effects. * **Not a replacement for Ghidra or the guest diagnostics.** Ghidra answers *what is the code*; risc-v-gsp-rm-debugger answers *what did it do*; the guest driver remains the oracle for guest-side behaviour. * **Not a hardware or security tool.** It never attaches to a real GPU, a guest, a kernel module or an mdev, and it is never a security or confidentiality boundary. @@ -112,7 +120,15 @@ Rules that make a session usable as evidence: * **Bounded everywhere.** Runs have durations and stop caps; reads and snapshots have size caps; a wedged target is left stopped and restarted clean rather than poked. * **Local only.** The Ghidra bridge is a localhost transport, unauthenticated by design; never expose it on a network interface. * **No confidentiality claims.** The tool is a study environment; never use injection to defeat a hardware security boundary. -* **Status.** The audited v0.1 surfaces are the CLI and the session format. Working and live-verified: hooks/watchpoints, memory tools, sessions, declarative probes, plugins, remote calls, run-to-cursor, the injection SDK, the Ghidra bridges, the first-fault probe and the instruction/access trace. The device model, the custom simulator and serving full host message traffic remain roadmap; porting and packaging scaffolds are pilot-status. Treat interface changes as reviewed events. +* **Status.** The audited surfaces are the CLI and the session format. + Working and live-verified: hooks/watchpoints, memory tools, sessions, + declarative probes, plugins, remote calls, run-to-cursor, the injection + SDK, the Ghidra bridges, the first-fault probe and the instruction/access + trace. A partial device model boots the real GSP image through its + relocation-copy rung deterministically; the message loop, the custom + simulator and serving full host message traffic remain roadmap; porting and + packaging scaffolds are pilot-status. Treat interface changes as reviewed + events. ## 9. Related documentation diff --git a/wiki/System-Overview.md b/wiki/System-Overview.md index 3f2d3e2..685926c 100644 --- a/wiki/System-Overview.md +++ b/wiki/System-Overview.md @@ -7,7 +7,7 @@ > and [`../docs/03-architecture/10-interaction-and-scheduling.md`](../docs/03-architecture/10-interaction-and-scheduling.md) > for the full picture); the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partial; `[wall]` current blocker; `[target]` designed end state. @@ -32,8 +32,10 @@ EMULATE (software; the guest expects virtual hardware) PASS THROUGH (real hardware; correctness and performance) GPU memory: guest VRAM objects backed by real VRAM [target] page tables / GMMU state used by the GPU [wip] - channel execution: guest push buffers on real channels [gated/wall] - compute (CUDA) and, later, 3D/video work [wip/planned] + channel execution: guest push buffers on real channels [wip] + compute (CUDA) kernel launches on the real engine [live] + 3D graphics method streams on the real engine [wip] + rendered 3D pixels / video work [wall/planned] DMA, completion and fault interrupts relayed to the guest ``` @@ -130,9 +132,15 @@ RPC from the guest to "firmware": payload round-trips). 5. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored semaphore releases and a fault relay. For GR - (graphics/compute), a forwarded guest launch reaches the front-end but the - engine has not executed it yet — the current frontier - ([`Execution-Frontier.md`](Execution-Frontier.md)). + (graphics/compute), forwarded guest CUDA launches execute on the engine + with engine-authored completions relayed to the guest, and graphics + method streams execute in two lanes + ([`Graphics-Execution.md`](Graphics-Execution.md)); rendered 3D pixels are + the graphics frontier ([`Execution-Frontier.md`](Execution-Frontier.md)). +6. On the graphics side, the X server and GLX come up on the emulated vGPU + with the stock driver (`glxinfo` reports the NVIDIA renderer); the guest + Vulkan pipeline runs end to end; rendered (non-black) frames are the open + frontier ([`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md)). ## 6. Where the detail lives diff --git a/wiki/UVM-in-the-Guest.md b/wiki/UVM-in-the-Guest.md index 53ca755..af1238f 100644 --- a/wiki/UVM-in-the-Guest.md +++ b/wiki/UVM-in-the-Guest.md @@ -5,7 +5,7 @@ > Canonical source: [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md); > where this page and the docs tree disagree, the docs tree wins. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the stock guest driver; `[wip]` partially working, software bridge in charge; `[gated]` implemented default-off; `[target]` planned end state. @@ -60,13 +60,13 @@ Each stage has a distinct failure signature — see | Completion semantics | tracker values must advance exactly | `[wip]` | **Current frontier.** Context creation and the full CUDA driver-API smoke pass -(`ALL OK`, copies matched) run end to end. CE-class copies are forwarded to the -physical copy engine (default off) with engine-authored releases and a fault -relay. For GR, a forwarded guest launch reaches the front-end but the inline -launch-descriptor install has not been observed; context validity at dispatch -is ruled out ([`Execution-Frontier.md`](Execution-Frontier.md)). The remaining -UVM work is completion translation and the guest-side fault replay/cancel -contract. +(`ALL OK`, copies matched) run end to end, and guest CUDA kernel launches now +execute on the physical engine with engine-authored completions relayed to the +guest. The remaining UVM work is completion translation for the staged-transfer +class, the guest-side fault replay/cancel contract, and per-guest execution +isolation ([`Execution-Frontier.md`](Execution-Frontier.md)). Graphics work +rides the same submission machinery on its own frontier +([`Graphics-Execution.md`](Graphics-Execution.md)). ## 4. The discovery that defines the work: how UVM waits @@ -164,5 +164,5 @@ behaving like a real GPU. * [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md) — canonical chapter (this page's source). * [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the guest stack and the RM objects UVM consumes. -* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace half of the contract; [`Execution-Plane.md`](Execution-Plane.md), [`Execution-Frontier.md`](Execution-Frontier.md) — how UVM's pushes execute, and the current wall. +* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace half of the contract; [`Execution-Plane.md`](Execution-Plane.md), [`Execution-Frontier.md`](Execution-Frontier.md) — how UVM's pushes execute, and the current leading edge. * [`../docs/04-implementation/01-known-knowns-and-unknowns.md`](../docs/04-implementation/01-known-knowns-and-unknowns.md) — server-owned page tables and the fault contract. diff --git a/wiki/Vulkan.md b/wiki/Vulkan.md new file mode 100644 index 0000000..d213136 --- /dev/null +++ b/wiki/Vulkan.md @@ -0,0 +1,127 @@ +# Vulkan + +> **Scope.** How a stock guest Vulkan application runs on the emulated vGPU: +> the guest-side stack, the submit/completion path, and the honest state of +> the work — including the one thing every Vulkan run on the emulated vGPU +> currently has in common: **the pipeline completes, and the frames are +> black**. The shared host machinery is on +> [`Graphics-Execution.md`](Graphics-Execution.md); the CUDA compute chain +> (further along) is on [`Execution-Frontier.md`](Execution-Frontier.md). +> Canonical sources: +> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) +> and +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md); +> the docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the +stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` +defined success state. + +## 1. The guest Vulkan stack + +A Vulkan application loads the **loader** (`libvulkan.so`), which dispatches to +an **installable client driver (ICD)** — for NVIDIA GPUs, the NVIDIA Vulkan +driver. On Linux, Vulkan compute does not require an X server; the display +path does (through `nvidia-modeset`/DRM for presentation). The kernel-side +requirements are the same as CUDA's: `nvidia.ko`, the GSP client, channels, +memory objects and fences. + +At device creation the driver also walks the **display/modeset device** +allocation path (the `ALLOC_DEVICE` class family) — on a vGPU-style device the +driver expects the host to provide those display classes, which the emulation +must answer for `vkCreateDevice` to succeed. Past that gate, the driver +behaves much like the compute stack: memory types through its VidHeap classes, +queues as channels, and work as method streams. + +## 2. The submit and completion path (observed) + +```mermaid +sequenceDiagram + participant App as Guest app + participant ICD as NVIDIA Vulkan driver + participant KMD as nvidia.ko + GSP client + participant VG as vgsp.ko emulation + participant GPU as Physical GPU (GR/CE) + App->>ICD: record command buffer, then vkQueueSubmit(fence) + ICD->>KMD: push buffer + GPFIFO entry + doorbell + KMD-->>VG: method stream (compute C5C0 shape + copy shapes) + VG->>VG: capture the section, translate to a vendor launch,<br/>relocate operands into reachable shadow pages + VG->>GPU: submit on a real channel (launch descriptor install) + GPU-->>VG: engine-authored release + VG-->>KMD: completion relay (MSI-X + poller) + KMD-->>App: fence released, vkWaitForFences returns + App->>App: reuse buffers, read back the frame +``` + +```text +app: vkQueueSubmit(fence) + -> NVIDIA driver: push buffer + GPFIFO entry + doorbell + -> vgsp.ko: capture section, translate to a vendor launch, + relocate operands into shadow pages, submit on a real channel + -> physical GPU: front-end fetch, engine executes, authors release + -> vgsp.ko: completion relay (MSI-X + poller) back to the guest + -> app: fence releases; frame readback proceeds +ASCII fallback above; the sequence shows one queue submit end to end. +``` + +Two mechanics make this work today: the emulation decodes the driver's +compute-dispatch shape — a **direct method stream** (object bind, window runs, +code runs, a subchannel data-port stream) with **no inline launch descriptor** +— and *translates* it into a launch the real engine accepts, and it derives +the submit's release groups from the driver's own push words so the fence +value is released when the engine completes (the "fence fold": the pre-built +GPFIFO continuation beyond the put pointer is swept and the submit's release +groups are applied forward-only). + +## 3. Status on the emulated vGPU + +| Item | State | +|---|---| +| `vkCreateDevice` on the emulated vGPU (display/modeset allocation path answered) | `[live]` | +| Full pipeline: instance, device, allocation, image/view/buffer/shader/descriptor, pipeline, command buffer, fence | `[live]` | +| Queue submit, engine-side execution of the translated launch, fence release, frame readback | `[live]` | +| The client saves frames end to end (`VKFRAME` evidence) | `[live]` | +| Non-black frames | `[wall]` | +| The driver's program (SASS) upload materialised by the emulation | `[wall]` | +| The engine's page tables mapping the program VA | `[wall]` | + +## 4. Where the wall is, precisely + +The client *believes* it uploaded its shader and dispatches with a valid +program address, but the bytes never appear in any source the emulation can +see: the driver delivers programmed SASS and descriptors through an **MME +program + inline-to-memory stream** on its copy/utility channel, and the +inline-data writes are not yet materialised for the graphics clients. Two +coupled facts define the remaining work: + +1. **Delivery** — decode and execute the driver's inline upload so the + program page becomes non-zero (the same machinery already works for CUDA's + uploads; the graphics clients' shapes are the extension). +2. **Mapping** — the engine's page tables must map the program VA. With the + data-set gate bypassed, the engine fetches the launch and raises a real + MMU fault (`FAULT_PDE`) at the program VA — proof the channel, context and + fetch path are alive, and that mapping plus content is the missing pair. + +When both land, the emulated vGPU should carry its first non-black Vulkan +frame — the milestone the whole graphics effort is converging on +([`Graphics-Execution.md`](Graphics-Execution.md) §6). + +## 5. What this page is not claiming + +* No rendered pixels yet — every frame saved so far is black. +* No performance numbers. +* `vkCreateDevice` success does not imply rendering; it means the device + model answers the driver's device-view correctly. + +## 6. Related pages + +* [`Graphics-Execution.md`](Graphics-Execution.md) — the shared lanes, the + `0xC597` graphics stream and the ranked walls. +* [`OpenGL.md`](OpenGL.md) — the GL path through the X server. +* [`DirectX.md`](DirectX.md) — the Windows/DirectX target. +* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace + driver model the Vulkan ICD mirrors in structure. +* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — translation and the + engines. +* [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) — + the measured wall ledger, with every experiment and its verdict. |