diff options
72 files changed, 11948 insertions, 916 deletions
@@ -159,3 +159,9 @@ userspace/sdks/ *.bak-* *.orig *.rej + +# Demo/release videos (task: "0.9.9 release preview" lives in the repo root, +# never tracked - large binary; the file is produced locally from the host +# KVMFR capture). +/*.mp4 +/open-iov-*.mp4 diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 5322e98..27295df 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,12 @@ # Open-IOV Current State -> **Snapshot: 2026-10-16 ~15:30 (integration wave: bigpath+dual merged, five fixes -> landed through `7d123630`; live r161/r162 cycles on kernel 6.8.0-147).** +> **Snapshot: 2026-10-20 ~20:00 (two more fixes landed: the SWAP_RX cursor +> and the pin-wins ledger divergence that cured a guest-kernel-corruption +> panic; guest CUDA kernels green on fresh boots - `victory_mail`, +> `victory_matmul` (first-run flake), transformer, `vector_add` (1M, +> `bad=0`), `vecadd_loop` (100 launches); two guests with two emulated VRAM +> sizes (512M + 1GiB) coexist live; frontier = the multi-pass content class +> and the streams2 launch-order wait).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -302,9 +307,62 @@ isolation, performance, portability). `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the demand-pin content resolver searches a learned per-allocation offset. Remaining characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch - class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out - mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, - retest pending). + class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out + mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, + retest pending). + **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):** + the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was + reproduced and root-caused: under the driver's status-queue SWAP_RX mode the + emulation's status free-space check read the status queue's mirrored RX header + (which the emulation publishes itself) instead of the peer consumption cursor in + the command queue's RX header. With the guest mid-drain the status ring was + declared full, replies were written anyway, unread records were overwritten, and + the guest RM read a record one full ring lap ahead + (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed + and the first client's `cudaMalloc` returned "device busy or unavailable" - + poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the + cursor is read from the command queue's RX header per the driver's swap-mode + semantics; per-client channel unregister and session flush work again (one + unregister in the poisoned boot; 79 with one flush per client after). Verified on + fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul` + (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`, + `FFN4 layers=4 bad=0` + `STATUS OK`) all green. + Remaining large-buffer class (current frontier): a `vector_add` cold-run content + gap (a staging read/write race between the emulated CE copy and the guest's + staging ping-pong: ~0.68 MB of the third MB reads the previous staging content), + `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop` + hang class correlated with a boot where the injected-pin stream never re-targeted + the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class + (RM object fields overwritten with client data; stale system-memory destination + resolution on freed staging pages) is characterized but open. Next: pin-stream + re-targeting, then the staging-race and stale-destination fixes; then the + boot-lottery re-measure over fresh boots, the full battery + soak, two guests, + multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live + with the three green kernels. + **Update 2026-10-20 ~20:00 (pin-wins corruption fix; multi-tenant live; battery frontier):** + a second, independent fix landed after the SWAP_RX cursor: the emulation's + write-side ledger could redirect a copy to the page an earlier write landed + on even when the guest had already remapped or freed that VA (4448 such + divergences in the failing run), so guest-kernel memory could receive client + copy data - observed as an RM object's pointer field holding client float + data, a free-path traversal faulting, and a guest panic + ("Attempted to kill init") on the `vector_add` client. With a pin-backed + translation (guest pagemap truth) now winning the divergence and the ledger + re-homing after the write (`720d3f5f`): `vector_add` cold AND warm run + `bad=0` + `STATUS OK` and the guest survives; the cold-run content gap + class is cured with it. Battery on fresh boots: `victory_mail`, + `vecadd_loop` (100 launches), `victory_matmul` (a first-run all-bad flake, + green on immediate retry), `victory_strided`, `victory_math`, + `vector_add`, and a transformer kernel (`dh=256`/`FFN4` `bad=0`) are green; + streams2's ordering class is root-caused (core CUDA starts compute-channel + pushes with a C56F host-semaphore acquire that the decoder ignored, so a + D2H readback ran before the scale launch it waits for) and the multi-pass + content class is the remaining frontier (multi-launch/accumulating kernels + wrong; `victory_reduce` is exactly 2x - the engine never wrote on five + launch-shaped pushes of a one-launch client). Multi-tenant: two mediated + devices (512M + 1GiB) and two guests coexist; guest B reports its 1GiB + VRAM with `nvidia-smi` green; guest-B CUDA awaits its own pin stream. Host + clean, no reboots. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -628,6 +686,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM | | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | @@ -664,6 +723,29 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX + cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20 + ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`, + `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM + logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed + (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable") + onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr + lives in the command queue's RX header (the driver's swap-mode semantics), + while the status queue's own RX header is the mirror the emulation publishes + (its own command-queue read position); the status free-space check read the + mirror, so with the guest mid-drain the ring was declared full and replies + were written over unread records. Fix (`c72f8948`): read the peer + consumption cursor from the command queue's RX header. Verified on fresh + boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256 + bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`) + all green with `STATUS OK`; per-client channel unregister + session flush + restored (1 unregister in the poisoned boot vs 79 after). Remaining + frontier: the large-buffer class (`vector_add` cold-run staging race, + `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with + an un-targeted injected-pin stream), then boot-lottery re-measure, full + battery + soak, two guests, multi-mdev, VRAM sizes. Evidence: + `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in + tmux session 0 (SSH to the guest, three green kernels). - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index caaad01..3d97d07 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -568,6 +568,39 @@ forwards (local evidence and the prepared harness: forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. +### 3.5a Copy-out change detection: fingerprint rule + +A forwarded launch stages every referenced data page into the data-bounce +chunk, and after the retire wait the copy-out ships back only the pages the +engine actually modified. That decision used to be a 32-bit additive +checksum of the page's words, compared against a baseline captured at +copy-in. The additive form has a structural blind spot: doubling every word +of a 4 KB page raises each float exponent by one (+2^23 per word), and 1024 +words contribute `1024 * 2^23 = 2^33`, which is exactly `0 (mod 2^32)` — an +in-place power-of-two scale of a whole page (`c[i] *= 2.0f`) left the +checksum bit-identical, the page was classified "engine did not touch", +the write-back was skipped, and the client read the pre-scale values. The +same blindness applies to `*= 0.5f`, `* 4`, `* 8`, etc. Since `r452` +(commit `19399f4f`) every change decision in the copy path — the retire +wait's settle scan, the tracked and gap copy-outs, the window witness, the +late-store witness and the idempotent-launch fingerprint — uses a 64-bit +FNV-1a hash over the page instead; the additive sum survives only as the +all-zero filter. Policy rule for the plane: **a change-detection +fingerprint must not be an algebraic invariant of any plausible kernel +operation** — use a hash, never a sum. + +The reader side of the same decision is the landing table: a D2H copy of a +device VA resolves through the last recorded write of that VA. Since `r453` +the forwarded QMD's release0 is redirected to the module semaphore by +default (`gsp_phys_rmcall_gr_rel_mod=1`): the guest-visible release word is +authored by the host only on the completion edge, after the copy-out has +shipped the engine's stores, so a client's stream-ordered readback can no +longer run ahead of the write-back (the historical shape: the engine's own +release satisfied the guest while the copy-out was still pending). The +engine-completion observable remains the module semaphore, and the +per-outcome authoring path guarantees the guest word is applied on every +outcome (timeout, wedge, no-write). + ### 3.6 Citation map for this section | Claim | Authority | diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md index f5546ca..a8e03ee 100644 --- a/docs/04-implementation/03-execution-frontier.md +++ b/docs/04-implementation/03-execution-frontier.md @@ -1,11 +1,17 @@ # The Execution Frontier: The GR Compute-Launch Chain -> **Status: 2026-10-06.** This page describes the current leading edge of +> **Status: 2026-10-20.** This page describes the current state of > guest-facing execution: the chain that must fire for a stock guest CUDA -> kernel to run on the physical GPU, what we have proven about it, where it -> currently stops, how we know, and the open lines of investigation and -> theories. It is evidence-led: every statement carries either a live -> observation, a decompilation finding, or a clearly labelled theory. +> kernel to run on the physical GPU, what we have proven about it, how we +> know, and where the leading edge is now. The chain fires end to end: +> verified guest CUDA clients (vector-add, matrix-multiply and a transformer +> kernel) execute on the physical GPU, their results verify, and the +> engine-authored completions relay back to the guest. The leading edge has +> moved from "can a forwarded launch execute at all?" to breadth and +> hardening — the large-buffer staged-transfer class, reproducibility across +> boots, and per-guest execution isolation. It is evidence-led: every +> statement carries either a live observation, a decompilation finding, or a +> clearly labelled theory. > > Evidence sources for this page: supervised physical cycles (module-owned > GSP-RM on the reference TU104), offline firmware emulation/probing, the host @@ -40,40 +46,59 @@ flowchart TB J["Host polls release, relays to guest;<br/>CUDA wakes"] ``` -| # | Stage | State (2026-10-06) | +| # | Stage | State (2026-10-20) | |---|---|---| | 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) | | 2 | GPFIFO entry + doorbell | ✅ observed | | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) | | 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) | -| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** | -| 7 | **Engine reads the QMD** | ✗ **not observed** | -| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 | -| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack | -| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches | -| 11 | Host relays completion; guest wakes | ✗ guest waits and times out | - -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) | +| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed | +| 7 | Engine reads the QMD | ✅ observed | +| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 | +| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve | + +**One-line summary:** the whole chain is live — verified guest CUDA clients +execute on the physical GPU and wake up correctly. The leading edge is no +longer the chain itself but its breadth and hardening: the large-buffer +staged-transfer class, reproducibility across boots, and per-guest execution +isolation. --- ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire we observe the ring entry consumed and the -front-end methods run — but the launch descriptor is never materialised, the -engine never reads a QMD, no engine-authored completion appears, and the guest -blocks in its synchronisation wait. - -This is **not** a context-validity wall (that was the previous frontier and is -now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the -descriptor destination (that install is accepted). The wall is the **form of -the launch method stream**: what the front-end requires in order to install the -inline descriptor, and what the engine requires in order to read and dispatch -it. +A stock, unmodified guest CUDA program now executes its kernels on the +physical GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. The engine authors the completions, the host relays them to the +guest, and the guest's synchronisation waits resolve. + +The leading edge has therefore moved from the launch chain itself — it fires — +to breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise, the first + CUDA client would fail, and the rest of that boot was poisoned (the + user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement + across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. --- @@ -81,17 +106,17 @@ it. | Observable | What it proves | What it does **not** prove | |---|---|---| -| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine | -| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) | -| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started | -| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction | -| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path | - -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results | +| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches | +| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored | +| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress | +| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction | + +**Consequence:** acceptance is now measured by verified guest results plus the +engine-authored release; front-end signals remain necessary but not +sufficient. A green run counts as progress only when the same acceptance +reproduces across fresh boots. --- @@ -111,90 +136,94 @@ signals, which we have already learned to distinguish. * **Parenting/TSG for the fetch** — both parenting variants fetch identically. * **The copy-engine path** — CE-class work, its releases and its fault relay are live (default off); the GR path is a separate problem. +* **The descriptor-gate line of investigation (the old T1–T4 theories).** The + inline-descriptor install, the engine's QMD read, the grid dispatch, the + engine-authored release and the guest wake-up are all now observed; the + gate they questioned is closed by observation (§1, §7). +* **The old boot-poisoning defect.** Root-caused to the status-queue + consumption cursor and fixed; it is no longer the explanation for a failing + first client (§5, issue B). --- ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** -The forwarded stream includes a verbatim copy of the guest's prelude whose -inline-data writes target addresses that are not present in the engine's view -of the channel's page tables, and the referenced data is not relocated. The -shorter, proven stream shape fires its tail (channel semaphore) but still does -not schedule the descriptor. -*How we know:* the prelude stalls in the forwarded form; the shorter shape's -tail fires; the descriptor-install sentinel never changes in either case. -*If fixed:* the install path is reached and we can observe stage 6 directly. - -**Issue B — the descriptor-virtualization control method is absent.** -The method that arms the inline-descriptor install path (the QMD -virtualization control) is not present in the forwarded stream. -*How we know:* the method does not appear in the submitted stream, and the -install sentinel never changes. -*If fixed:* the front-end should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. - -**Issue C — diagnostic hygiene.** -The descriptor sentinel placed at word 0 overlaps real descriptor fields; it -must be moved to an unused word so diagnostics do not perturb what they -measure. +**Issue A — the large-buffer staged-transfer class.** +Multi-megabyte staged transfers between guest memory and device memory are +under repair: some large multi-chunk clients still mis-verify. +*How we know:* the small and medium clients (vector-add, matrix-multiply, the +transformer kernel) verify green while large multi-chunk transfers mis-compare +or stall in the same configurations. +*If fixed:* the client battery should accept the large-buffer class with the +same verification discipline as the green clients. + +**Issue B — reproducibility across boots (re-measurement).** +A boot-poisoning defect was root-caused and fixed: under the driver's +status-queue mode the emulation read the wrong queue cursor, so under load the +reply ring could desynchronise, the first CUDA client would fail, and the rest +of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report). +*How we know:* the failure was reproduced, the cursor rule was corrected, and +per-client unregister and session flush work again. +*If fixed:* green on every fresh boot — re-measurement is in progress, and no +"every boot" claim is made until it completes. + +**Issue C — per-guest execution isolation.** +Multiple mediated devices with different emulated VRAM sizes and multiple +guests already coexist at the device level; per-guest execution isolation for +multiple simultaneous CUDA guests is in progress. +*How we know:* the device-level coexistence is live; execution isolation is +not complete. +*If fixed:* multiple CUDA guests execute simultaneously without interference. --- ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline (the shorter stream that - fires its tail) and **re-add the virtualization control method** — one - variable at a time, with the install sentinel and the engine-read signal as - the readout. -2. **Relocate or drop the verbatim prelude** so the inline-data writes target - addresses present in the engine's view, or replace it with the equivalent - host-side staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference.** Two ways to obtain one: (a) make - our own ladder dispatch a descriptor built from the same source material - (in-house reference), or (b) capture a working launch of the stock stack on - the same GPU (real method stream + descriptor + engine-visible channel - state) and compare field by field. Either reference converts the remaining - work from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware emulator/probe tooling before spending a - supervised cycle. +1. **Repair the large-buffer staged-transfer class** — the multi-megabyte + guest↔device transfers that still mis-verify. The three green clients stay + the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility, and + keep the re-measured battery as the regression gate. +3. **Per-guest execution isolation** — build from the device-level + coexistence of multiple mediated devices and guests to per-guest execution + isolation. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel run as a battery on every re-measurement; each entry is + accepted only on its own verification output (`bad=0` + `STATUS OK`). --- -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the virtualization-control method - enables the install path; its absence alone explains the inert descriptor. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape - works, the prelude's refused writes leave the launch path in a state that - prevents the descriptor from being scheduled. -* **T4 — relocation is the missing transform.** The guest's inline descriptor - is valid, but it is built for the guest's view of memory; the forwarding path - must relocate referenced data (and any embedded pointers) into the channel's - view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed (§1), +so the launch chain they questioned is now proven end to end. + +The remaining leading-edge work is engineering rather than theory: the +large-buffer transfer class, boot-to-boot reproducibility, and per-guest +execution isolation. Each has an acceptance signal named in §5 before the +work. --- ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals (ring cursor, method execution, semaphores that the front-end can - author) are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end for the verified clients; the guest's own verification + (`bad=0` + `STATUS OK`) is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred; the current GR path runs without an error-reporting object, which is itself a known observability gap. * **Emulator verdicts are L0/L1.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** The + boot-poisoning defect is the worked example: one green boot is not + reproducibility. * **A wall is closed only by an acceptance signal**, and the acceptance signal for this frontier is named before the experiment. @@ -218,3 +247,4 @@ signal, and the engine-authored release, in that order. | Date | Change | |---|---| | 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. | +| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. | diff --git a/docs/05-operations/05-gpubreach-audit.md b/docs/05-operations/05-gpubreach-audit.md new file mode 100644 index 0000000..3c60385 --- /dev/null +++ b/docs/05-operations/05-gpubreach-audit.md @@ -0,0 +1,95 @@ +# GPUBreach-class audit and threat model (vGPU host module) + +Status: audit performed 2026-10-21 against the host module at `229bec2c` +(`vgsp.ko`, `kernel/`), with the write-path guards landed in the same window +(`7d4919dd`, `7a1f4805`). Scope: every host-side write whose target address +derives from guest-supplied data. Companion reading: +[`01-performance-correctness-security.md`](01-performance-correctness-security.md). + +## 1. What the GPUBreach threat is + +GPUBreach (IEEE S&P 2026, U. Toronto) is a GPU Rowhammer chain: the attacker +flips bits in the physical GPU's page tables (PTEs), which turns the GPU's +MMU into an arbitrary *device-memory* read/write primitive; the paper then +escalates to CPU root by exploiting memory-safety bugs in the *driver* code +that consumes those corrupted translations. It works with the IOMMU on, +because the corruption happens inside the GPU, after translation. + +The relevant question for this project is therefore not "can a guest read +device memory" (device memory is the guest's own resource here) but: + +1. can guest-influenced data reach a *host* write without validation, and +2. what bounds a write that lands on a *wrong* target. + +## 2. Structural mitigations (why the host is not on that path) + +* **Guest GPU page tables are host-emulated.** The guest's GMMU page tables + live in the emulated vidmem shadow (BAR1 backing) or in guest RAM; the + host resolves guest GPU VAs through its own walk of the guest's published + page directories (`SET_PAGE_DIRECTORY`) or through content-verified pins. + There is no physical GPU MMU walk of guest-authored tables. +* **The physical GPU's PTEs are host-owned.** Backend jobs are submitted by + the host module on host-authored PTE lines; the guest never supplies a + descriptor that the real hardware MMU consumes for host memory. +* **Every write is aperture-bounded.** The one write primitive, + `svc_write_ap()`, sends SYS writes through `vfio_dma_rw()` on the mdev's + own VFIO container - the IOMMU/vfio layer only maps the guest's RAM - and + FB writes are bounds-checked against the emulated shadow size. A + guest-influenced target can therefore only ever land inside the guest's + own RAM or the vidmem shadow; a wrong target is a correctness bug, not a + host-memory write. +* **No guest DMA to host buffers.** The guest's drivers talk to the + emulated device; they cannot address host kernel buffers. IOMMU is on + (vfio), and the mdev exposes only guest RAM plus the emulated windows. +* **Guest descriptors are not trusted by construction.** Every path that + turns a guest VA into a host PA goes through the resolution chain + (guest-verified injected pins -> write/read landing ledger -> published + page-directory walk -> targeted record scan), and the chain's outputs are + re-validated per page (see §3). + +## 3. Write-path inventory and validation state + +| Write path | Target derivation | Validation today | +|---|---|---| +| CE fill / memset (`svc_fill`) | guest VA or physical FB/GPA from the push | per-page destination resolution (pin -> ledger -> walk), page-boundary clamped; a page with no resolution truncates loudly and demands the VA through the pin channel (`FILL-TRUNC`, `stalled VA`); zero fills onto page-table-shaped guest pages are refused (`FILL-PT-REFUSED`); below-floor window fills that resolve outside vidmem are refused (`LOWFILL-REFUSED`) | +| CE copy (`svc_copy`) | dst/src VAs from the push | per-page resolution with pin-agreement (a fresh pin beats a stale ledger entry and repairs it); device-dst copies without a pin-resolved translation are held (`sema_copy_dst_pin_guard`) | +| GRCE inline writes (`LOAD_INLINE_DATA`) | destination VA from the push | once-only identity gate keyed by (push VA, push content hash, dst, bytes), session-scoped and 8192 slots wide | +| GR launch operand staging (`svc_gr_data_fill`) | argument pointers scanned from the QMD | per-page pin/ledger resolution; the resolution is stored in the ledger as the landing authority; dropped pages are named (`GRFILL-DROP`) and demanded | +| Launch copy-out (engine results back to the shadow/guest page) | the launch's own landing records | per-page destination resolution; every write is witnessed (`OUT-PAGE-WIT`/`OUT-PAGE-POST`); staged-identity rule (only the page staged into a slot may receive the slot's content) | +| Release/counter writes (semaphores, completion counters) | guest semaphore VAs from pushes | pool-range writes go through a pin/record-verified cache (`pool-pa`), re-validated against the current pin before the 4-byte write; exact-slot fallback for the client's tracker; forward-only apply | +| GPFIFO/USERD writes (put/get bookkeeping) | channel registration data | the channel's own USERD/instance blocks from the alloc params, aperture-tagged | +| Window/PTE installs (forward PDE install) | the launch's windows | installs are refused loudly when the backend rejects the PDE; the regular path leaves guest VAs in place (no host-side table mutation) | + +## 4. Residual risks and findings + +* **Wrong-but-valid writes remain possible** when every resolver agrees on a + stale page (e.g. a guest freed and reallocated a VA). The r391/r398/r401 + fixes removed the known instances (contiguous fill past the first page, + replayed inline writes, capture-window eviction), but the class is bounded, + not eliminated: the guards convert silent corruption into loud + truncation/refusal where a page is unresolved, and the ledger is + session-scoped and pin-repaired where a fresh pin exists. +* **The boot-corruption lottery** (a zero fill landing on guest page-table + pages) is structurally mitigated by the page-table-shape refusal and the + per-page clamping; no instance has been observed since r391. +* **Diagnostics stay on**: `DST-DIVERGE` (copy-dst ledger vs probe), + `FILL-TRUNC`, `GRFILL-DROP`, `NOWRITE-*` and the stalled-VA demand rows + exist so a future divergence is visible in the boot log rather than + silent. +* **What this audit does not claim**: no isolation or side-channel claim. + GPU Rowhammer itself is a *physical* fault-injection threat that this + software emulation cannot mitigate; the mitigation argument here is only + that the host write paths cannot be steered outside the guest's own + memory, and that wrong targets are bounded by IOMMU-mapped guest RAM or + the emulated shadow. + +## 5. Recommended follow-ups + +1. Keep the copy-destination ledger under the `DST-DIVERGE` witness and + promote it to a bounded hold only if a failing run names it. +2. Extend the pin-demand channel to client device VAs (today the in-guest + scanner resolves only its own push/table windows), so an unresolved + operand page becomes a wait on real data instead of a truncated fill. +3. Re-run this audit when the write-path guards change (any new + `svc_write_ap()` caller with a guest-derived target is in scope by + default). diff --git a/docs/05-operations/06-guest-frame-path-and-looking-glass.md b/docs/05-operations/06-guest-frame-path-and-looking-glass.md new file mode 100644 index 0000000..bc5447a --- /dev/null +++ b/docs/05-operations/06-guest-frame-path-and-looking-glass.md @@ -0,0 +1,102 @@ +# Guest frame path and Looking Glass status (2026-10-21) + +This page records the state of the "show the guest's GPU output on the host" +goal: what is proven, what is built, and exactly which pieces remain. It is +written from live evidence on the vGPU-IO stack (guest CUDA green, ivshmem +devices present, kvmfr module loaded on the host). + +## 1. What is proven + +* **Guest GPU rendering works through CUDA.** `tools/cuda-ab/frame_producer.cu` + renders an animated Julia set on the guest GPU (800x600 RGBA, ~10 fps), + copies the frame to host memory and publishes it into the ivshmem BAR + (`/sys/bus/pci/devices/0000:00:03.0/resource2`, the 128 MiB kvmfr device + QEMU backs with `/dev/shm/kvmfr-<uuid>`), with a small header + (`magic/version/width/height/stride/format/seq/timestamp`) at offset 1 MiB. +* **The host can read those frames.** `tools/lg/frame_reader.py` maps the same + shm file and writes each new sequence number out as a PPM, i.e. a real + GPU-rendered guest frame lands on the host. This is the full transport path + (guest GPU -> guest sysmem -> ivshmem -> host), independent of any display + protocol. +* **The ivshmem devices are visible in the guest** (`lspci`: two + "Inter-VM shared memory" functions; `resource2` of `0000:00:03.0` is the + 128 MiB window, `0000:00:04.0` the 2 MiB cursor window), and the guest can + mmap them (root required for the sysfs resource files). +* **EGL enumerates the NVIDIA device in the guest.** With + `EGL_EXT_platform_device`/`EGL_EXT_device_enumeration` present, + `eglQueryDevicesEXT` returns 3 devices and `eglInitialize` on the NVIDIA + device succeeds (`vendor=NVIDIA version=1.5`). + +## 2. What is blocked, and why + +* **Guest GL contexts do not allocate.** `eglMakeCurrent` on an offscreen + pbuffer fails with `EGL_BAD_ALLOC` (0x3003) on every platform tried + (device, surfaceless, default). Xorg's own failure is the same class: + `(EE) NVIDIA(GPU-0): Failed to allocate VASpace object` then + `Failing initialization of X screen`. The driver cannot allocate GPU virtual + address space for a GL context/surface through the vGPU, so both the X + display path and the EGL offscreen path stop at the same wall. This is the + single remaining blocker for guest OpenGL (and therefore for GZDOOM / + doomsday rendering in the guest). +* **The Looking Glass client (B4) cannot consume our frames directly.** + `looking-glass-client` (installed at `/usr/local/bin/looking-glass-client`, + runs under Xvfb with the `libbfd` shim in `/tmp/opencode/lgcompat`) supports + two transports, LGMP and SPICE. Its `app:shmFile` option still expects an + **LGMP session** (the `KVMFR` session header plus the shared-memory message + queues), not a raw framebuffer. Upstream's frame *producer* (`host/`) is the + Windows DXGI/IDD path. So a Linux guest-side producer must either implement + the LGMP session protocol or we keep the frame reader as the host-side + substitute (which is what `frame_reader.py` is). +* **GZDOOM is not packaged for Ubuntu noble** (`apt-cache policy gzdoom` has + no candidate); `doomsday` + `freedoom` are installed as the fallback, but + doomsday cannot start a renderer without a working GL context (blocker + above). + +## 3. Exact remaining pieces (in order of leverage) + +1. **Fix the guest GL VASpace allocation.** This is an RM/vGPU-side gap: the + driver's `nv_vgpu` VASpace allocation for a GL context fails. Fixing it + unlocks Xorg, EGL offscreen GL, doomsday/GZDOOM, and the standard Looking + Glass guest path (the guest's GL capture driver would then work as + upstream expects). +2. **Implement the LGMP session producer** (or a small Linux capture client) + that writes frames into the ivshmem region with the KVMFR session header + and the frame queue protocol the client expects. The transport half is + already proven; this is protocol work in the guest (or a host-side + substitute that speaks LGMP). +3. **Packaging**: GZDOOM would need a source build or a flatpak/snap; the + doomsday fallback is installed and only blocked by (1). + +## 4. Reproduce + +```bash +# guest (root, because the sysfs resource is root-only): +sudo /tmp/frame_producer 60 800 600 10 # 60 frames, 800x600, 10 fps + +# host: +python3 tools/lg/frame_reader.py --frames 5 --out /tmp/opencode/frames +# -> SAVED seq=N 800x600 fmt=0 -> /tmp/opencode/frames/frame-NNNN.ppm + +# guest EGL probe (shows the enumeration success and the MakeCurrent wall): +gcc -O2 -o egl_frame tools/guest-diag/egl_frame.c -lEGL -lGLESv2 +./egl_frame +``` + +## 5. Session-13 update (2026-10-21, r432→r441) + +* The single-frame path is unchanged and still proven: guest `frame_producer` + (CUDA Julia) -> kvmfr ivshmem BAR -> host `frame_reader.py` -> + `/tmp/opencode/frames/frame-0001.png` (320x240, 1431 colours). +* `tools/lg/frame_sequence.sh` captures an **animated** sequence end to end + (reader first, then N frames from the guest). It is blocked by the + GR no-execute wedge: when the lane is wedged the render launches write zero + pages and the producer republishes stale identical frames (observed: 6/6 + identical MD5s on r438; r439 published 5 identical plus one frame left in + the BAR by the previous boot). A multi-frame animation needs a healthy + lane; the frame pipeline itself is not the blocker. +* The GL wall is untouched: `Xorg :0` -> `(EE) NVIDIA(GPU-0): Failed to + allocate VASpace object`, guest dmesg `NVRM: vaspaceapiConstruct_IMPL: + Could not construct VA space. Status ffff`, i.e. `vmmCreateVaspace` inside + the guest's own RM. Next step: reproduce with the guest RM's + `ResmanDebugLevel` raised (a guest module reload, the service's m177 path) + to capture which inner call returns the generic error. diff --git a/docs/README.md b/docs/README.md index 13f0ae5..27e04e9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -183,6 +183,7 @@ Read in order if you are new. Jump directly if you know what you need. | 23 | [GSP in a RISC-V VM (risc-v-gsp-rm-debugger, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | +| 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min | | 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | | 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | | 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 7766cb5..734ebf2 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -140,6 +140,45 @@ static inline bool gsp_phys_gr_copywit_enabled(void) return READ_ONCE(gsp_phys_gr_copyout_dump_param); } + +/* + * r405 late-write probes (the r405 "partial write" class: the scale launch + * of streams2 wrote 1 of its 32 expected in-place pages and every other + * launch settled with far fewer changed pages than the kernel should + * produce). Two knobs, both runtime-settable: + * + * - gsp_phys_gr_late_probe (default 0): after the retire wait concludes, + * sleep +500 ms and re-scan the tracked pages. Run-once diagnostic: + * "ENGINE STILL WRITING" proves the copy-out shipped before the stores + * landed (the fix is a longer settle window); "stable" proves the engine + * genuinely stopped, which needs a different treatment. + * - gsp_phys_gr_settle_ms (default 300): the content-stable window the + * partial-changed rule requires before it declares the stores settled. + * Raise it to give a still-writing engine its time (A/B in one cycle). + */ +static bool gsp_phys_gr_late_probe_param; +module_param_named(gsp_phys_gr_late_probe, gsp_phys_gr_late_probe_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_late_probe, + "Diagnostic: after the GR retire wait, re-scan the tracked pages +500 ms later and log whether the engine is still writing (default 0)"); + +static int gsp_phys_gr_settle_ms_param = 300; +module_param_named(gsp_phys_gr_settle_ms, gsp_phys_gr_settle_ms_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_settle_ms, + "GR retire wait: content-stable window (ms) required before a partial changed-page count counts as settled (default 300)"); + +static inline u32 gsp_phys_gr_settle_iters(void) +{ + int ms = READ_ONCE(gsp_phys_gr_settle_ms_param); + + if (ms < 60) + ms = 60; + if (ms > 1400) + ms = 1400; + return (u32)(ms / 5); +} + /* * COPYWIT fail4 r224 follow-up: the r224 OUT-PAGE-WIT dumps proved the * engine's outputs are structurally corrupted (row bands >= 16 replicate a @@ -6789,6 +6828,35 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); /* + * Mode-35 KGR static-data snapshots. The user-lane query set + * (gsp_phys_rmcall_user_lane_kgr_queries) is answered by the LIVE GSP with + * the device's true GR static data; the replies are kept here so the + * guest-side emulation can serve the identical bytes instead of fabricating + * GR values. The concrete consumer is the X driver's 3D-engine selection: + * nv3dAllocDevice maps GR_GET_INFO index SM_VERSION (0x0c) through a table + * and fails with "Failed to select a 3D engine" when it reads back as + * SM_VERSION_NONE (0) - the fabricated fill never set it, while the live + * GSP's KGR_GET_INFO reply carries the real value (0x75 for TU104) along + * with every other GR static field (warps, caps, topology, ctx buffers). + * + * One entry per captured cmd; the generation key ties the cache to the + * backend session exactly like gsp_phys_backend_static above, and the + * getter refuses entries once the session has moved on. + */ +#define GSP_PHYS_KGR_SNAP_ENTRIES 10 +#define GSP_PHYS_KGR_SNAP_MAX_SIZE 3712u /* KGR_GET_INFO: 8 engines x 58 x 8 */ + +struct gsp_phys_kgr_snap { + u32 cmd; + u32 size; + bool valid; + u32 generation; + u8 data[GSP_PHYS_KGR_SNAP_MAX_SIZE]; +}; + +static struct gsp_phys_kgr_snap gsp_phys_kgr_snaps[GSP_PHYS_KGR_SNAP_ENTRIES]; + +/* * Drop the cached static-config snapshot. Called when the backend session * leaves READY (or a new claim starts): the cache generation key is bumped * past the current backend generation so a concurrent getter cannot serve the @@ -6797,9 +6865,13 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); */ void gsp_phys_backend_static_invalidate(void) { + unsigned int i; + mutex_lock(&gsp_phys_rm_lock); gsp_phys_backend_static.valid = false; gsp_phys_backend_static.generation = gsp_phys_backend_generation() + 1; + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) + gsp_phys_kgr_snaps[i].valid = false; mutex_unlock(&gsp_phys_rm_lock); } @@ -6853,6 +6925,71 @@ int gsp_phys_backend_static_get(struct gsp_phys_backend_static_config *out) } EXPORT_SYMBOL_GPL(gsp_phys_backend_static_get); +/* + * Copy the captured reply for cmd into dst. Returns 0 on success (with + * *out_len set), -ENODATA when nothing was captured for cmd (or the entry's + * generation no longer matches the backend session), -ENOSPC when the + * snapshot does not fit dst_len, -EAGAIN while the backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len) +{ + unsigned int i; + + if (!dst || !out_len) + return -EINVAL; + + if (gsp_phys_backend_state() != GSP_PHYS_BACKEND_READY) + return -EAGAIN; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + struct gsp_phys_kgr_snap *s = &gsp_phys_kgr_snaps[i]; + + if (s->cmd != cmd) + continue; + if (!s->valid || + s->generation != gsp_phys_backend_generation()) + return -ENODATA; + if (s->size > dst_len) + return -ENOSPC; + memcpy(dst, s->data, s->size); + *out_len = s->size; + return 0; + } + return -ENODATA; +} +EXPORT_SYMBOL_GPL(gsp_phys_kgr_snap_get); + +/* + * Record one mode-35 KGR query reply. Called under the boot thread only + * (same thread as the query itself), so no lock is needed beyond the + * generation key. + */ +static void gsp_phys_kgr_snap_store(u32 cmd, const void *data, u32 size) +{ + unsigned int i; + struct gsp_phys_kgr_snap *slot = NULL; + + if (size > GSP_PHYS_KGR_SNAP_MAX_SIZE) + return; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + if (gsp_phys_kgr_snaps[i].cmd == cmd) { + slot = &gsp_phys_kgr_snaps[i]; + break; + } + if (!slot && !gsp_phys_kgr_snaps[i].valid) + slot = &gsp_phys_kgr_snaps[i]; + } + if (!slot) + return; + + slot->cmd = cmd; + slot->size = size; + slot->generation = gsp_phys_backend_generation(); + memcpy(slot->data, data, size); + slot->valid = true; +} + /* ============================================================ * Host RM client ladder (R1): gsp-phys-rm-call * ============================================================ */ @@ -9588,14 +9725,68 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0) * (swref/published/pascal/gp100/dev_mmu.h:97-119,157; * uvm_pascal_mmu.c:64-141,235-303). */ -static u64 gsp_phys_rmcall_pde(u64 pa) -{ - return 0xeull | ((pa >> 12) << 8); +/* + * r431/r436 COHERENCY ROOT-CAUSE CANDIDATE: the leaf-PTE aperture for every + * host page this backend maps (the ladder windows, the win0 shadow pool, the + * bounce/data regions, the release/scratch shadows) was hardcoded to + * aperture 3 = SYSTEM_NON_COHERENT_MEMORY. The reference RM derives the + * pair from the allocation's CPU cache attribute (fabric_vaspace.c:1123-1134 + * for the aperture, mem_desc.c memdescGetVolatility for VOL: NV_MEMORY_CACHED + * -> SYSTEM_COHERENT (2) + volatile FALSE, NV_MEMORY_UNCACHED -> NON_COHERENT + * (3) + volatile TRUE). The r431 experiment used aperture 2 with VOL still 1 + * (0xd) - a pair the reference RM never produces - and it made the engine's + * truncated writes visible sooner but worsened the wedge on a fresh boot. + * r436 turns the knob into the full 2x2 matrix so each consistent encoding + * can be A/B'd: + * 0 = aperture 3 + VOL 1 (0xf) - historical, NV_MEMORY_UNCACHED + * 1 = aperture 2 + VOL 1 (0xd) - r431 experiment (inconsistent pair) + * 2 = aperture 2 + VOL 0 (0x5) - NV_MEMORY_CACHED (coherent, non-volatile) + * 3 = aperture 3 + VOL 0 (0x7) - non-coherent, non-volatile + * The PTEs are planted before the engine's first fetch, so the flag is + * boot-time only. + */ +static int gsp_phys_rmcall_pte_coh_param; +module_param_named(gsp_phys_rmcall_pte_coh, gsp_phys_rmcall_pte_coh_param, + int, 0444); +MODULE_PARM_DESC(gsp_phys_rmcall_pte_coh, + "r436: sysmem leaf-PTE aperture/VOL mode: 0=0xf (aperture3+VOL, historical), 1=0xd (aperture2+VOL, r431), 2=0x5 (aperture2, non-volatile), 3=0x7 (aperture3, non-volatile)"); + +static u64 gsp_phys_rmcall_sysmem_base_low(void) +{ + switch (gsp_phys_rmcall_pte_coh_param) { + case 1: + return 0xdull; /* aperture 2, VOL 1 */ + case 2: + return 0x5ull; /* aperture 2, VOL 0 */ + case 3: + return 0x7ull; /* aperture 3, VOL 0 */ + default: + return 0xfull; /* aperture 3, VOL 1 (historical) */ + } } static u64 gsp_phys_rmcall_pte(u64 pa) { - return 0xfull | ((pa >> 12) << 8); + /* + * Low nibble: VALID (1) | APERTURE<<1 | VOL (8). Aperture 2 = + * SYSTEM_COHERENT_MEMORY -> 1|4|8 = 0xd; aperture 3 = NON_COHERENT + * -> 1|6|8 = 0xf. r436: see gsp_phys_rmcall_pte_coh_param. + */ + u64 base = gsp_phys_rmcall_sysmem_base_low(); + + return base | ((pa >> 12) << 8); +} + +/* + * r436: the single PDE uses the same aperture/VOL pair as the leaf PTEs but + * with IS_PDE=0 in bit 0 (the low bit of the PTE base is VALID there), so the + * PDE base is the PTE base with bit 0 cleared: 0xe/0xc/0x4/0x6. + */ +static u64 gsp_phys_rmcall_pde(u64 pa) +{ + u64 base = gsp_phys_rmcall_sysmem_base_low() & ~1ull; + + return base | ((pa >> 12) << 8); } /* @@ -13513,6 +13704,12 @@ static int gsp_phys_rmcall_user_lane_userd(void __iomem *bar0) * (CONTEXT_BUFFERS_INFO in particular) may be prepared by the query. All are * zero-in best-effort getters at the capture's payload sizes; a failure * warns and the lane continues. + * + * Every successful reply is retained via gsp_phys_kgr_snap_store(): the + * guest-side emulation serves these exact bytes for the same commands, so + * the guest's GL/3D stack reads the live GSP's true GR static values + * (SM_VERSION in particular - see the snapshot comment near + * gsp_phys_backend_static_get). */ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) { @@ -13551,6 +13748,8 @@ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) q[i].name); pr_info("vgsp-boot: rm-call: mode 35 KGR query %s: size=%u status=0x%x rpc_result=0x%x ret=%d\n", q[i].name, q[i].size, status, rpc_result, ret); + if (ret == 0 && rpc_result == 0 && status == 0) + gsp_phys_kgr_snap_store(q[i].cmd, buf, q[i].size); kvfree(buf); } } @@ -15056,6 +15255,39 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, * semaphore. On timeout the existing status-queue drain runs and the helper * returns -ETIMEDOUT; it never retries. */ +static int gsp_phys_rmcall_gr_resched_locked(void __iomem *bar0); +static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0); + +/* + * r445 wedge hard re-arm: the outcome-retry's warm semaphore probe proves the + * engine refused the channel's proven GR block (the sticky post-dispatch + * no-store wedge). One bounded recovery attempt instead of an immediate + * break: force a context save/unload with a bWait TSG preempt, re-run the + * mode-35 lane-enable tail + schedules (the m155 extended re-arm), then + * repeat the warm probe. Caller holds gsp_phys_rm_lock. Returns 0 when the + * engine executed the probe after the re-arm. + */ +static int gsp_phys_gr_wedge_hard_rearm_locked(void __iomem *bar0) +{ + u32 pre[2] = { 1u, 0u }; /* bWait=1, bManualTimeout=0 */ + u32 rpc_result = 0, status = 0; + int ret; + + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_CTRL_TSG_PREEMPT, + pre, sizeof(pre), sizeof(pre), NULL, 0, + &rpc_result, &status, + "TSG PREEMPT bWait=1 (r445 wedge hard re-arm)"); + pr_warn("vgsp-gr: WEDGE-REARM preempt status=0x%x rpc=0x%x ret=%d\n", + status, rpc_result, ret); + ret = gsp_phys_rmcall_gr_resched_locked(bar0); + pr_warn("vgsp-gr: WEDGE-REARM resched ret=%d\n", ret); + ret = gsp_phys_rmcall_gr_sem_push_locked(bar0); + pr_warn("vgsp-gr: WEDGE-REARM probe rc=%d (engine %s after re-arm)\n", + ret, ret ? "still wedged" : "ALIVE"); + return ret; +} + static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) { struct gsp_phys_boot_chunk *userd_c = @@ -16242,6 +16474,19 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_guest_ce_copy); #define GSP_PHYS_RMCALL_GR_CWD_REFCOUNT 1536u #define GSP_PHYS_RMCALL_GR_CWD_LEVELS 64u #define GSP_PHYS_RMCALL_GR_MAX_SM_COUNT 24u +/* + * r434 CWD-slot-count path: the falcon PRI write (CUDA's + * turingFalconPRIWriteMasked: flag/data/mask through SET_MME_SHADOW_SCRATCH + * 0/1/2, then SET_FALCON04 = register offset) and the gated method. Used only + * when gsp_phys_rmcall_gr_cwd_slot >= 0. + */ +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH0 0x3400u +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH1 0x3404u +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH2 0x3408u +#define GSP_PHYS_RMCALL_GR_SET_FALCON04 0x0510u +#define GSP_PHYS_RMCALL_GR_SET_CWD_SLOT_COUNT 0x02b0u +#define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1 0x00407040u +#define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT 0x800u /* Method block bound: init(152) + inline run(67) + PCAS(4) + REPORT_SEM(8) + * the optional QMD-virt pair(2). */ #define GSP_PHYS_RMCALL_GR_METHOD_WORDS 512u @@ -16265,6 +16510,94 @@ static u64 gsp_phys_gr_sem_wedged; */ static u64 gsp_phys_gr_stall_wait_max; static u64 gsp_phys_gr_stall_retry; +/* + * r410 wedge quiet window state. nw_streak counts consecutive accepted + * zero-write launches with tracked data (reset by any launch that wrote); + * the sticky wedge is the state where this grows without bound while the + * lane is kept busy. quiet counts the quiet holds issued; heals counts the + * quiet retries after which the engine wrote. Single-inflight submit path, + * so plain statics. + */ +static u32 gsp_phys_gr_nw_streak; +static u64 gsp_phys_gr_wedge_quiet; +static u64 gsp_phys_gr_wedge_heals; +/* + * r411: once a quiet heal has succeeded on this boot, the lane is in a + * wedge-prone episode. r418 showed the wedge can alternate one-no-write / + * one-heal, and the healed launch resets nw_streak, so a streak-only trigger + * quieted only every second wedged launch and the other was accepted. The + * heal count makes the episode sticky for the boot: any later zero-write + * launch gets the quiet path on its first retry, so every wedged launch + * either heals (engine writes, client data correct) or is loudly classified. + */ +static bool gsp_phys_gr_wedge_episode; +/* + * r420: true exactly while a WEDGE-QUIET hold sleeps with NO submission + * outstanding. The hold is a deliberate, bounded wait for the sticky wedge to + * clear (r410/r411); during it the in-flight forwarded request is NOT done. + * The sema service's completion deadlines - the 12 s GR release hold + * (SVC_FWD_GR_HOLD_TIMEOUT_MS) and the 3 s per-launch consumption bound + * (gsp_phys_gr_hold_max_ms) - are shorter than this hold, so without the + * publish they fire UNDER it and author the launch's release word before the + * post-hold copy-out: the client then verifies stale data (r419 soak: + * `victory_matmul bad=65536` while its launch was held; the authoring paths + * are GR-STALL-fwd-hold / the busy-defer inline software release). The + * accessor lets both deadlines pause instead; the flag is bounded by the + * msleep itself and by the pause caps on the sema side. Single-inflight + * submit path, so a plain static. + */ +static bool gsp_phys_gr_quiet_hold_on; + +bool gsp_phys_gr_quiet_hold_active(void) +{ + return READ_ONCE(gsp_phys_gr_quiet_hold_on); +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_quiet_hold_active); + +/* + * r425 experiment (the second-launch wedge): the stock guest driver emits the + * per-channel compute init block ONCE at channel creation (turing_init.c), + * long before any launch, while this backend re-emits it as the preamble of + * every forwarded launch. The wedge strikes the SECOND launch of a channel, so + * test whether the re-emission is the trigger: with gsp_phys_rmcall_gr_init_once + * set, the init preamble is emitted only on the first forwarded launch after + * the module load (the engine's compute-class/CWD state persists in the + * context; every launch still carries its QMD + report semaphore). 0 keeps the + * historical per-launch emission (default; runtime-writable A/B). + */ +static bool gsp_phys_rmcall_gr_init_once_param; +module_param_named(gsp_phys_rmcall_gr_init_once, gsp_phys_rmcall_gr_init_once_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_init_once, + "r425: emit the GR compute init preamble only on the first forwarded launch (0 = every launch, default)"); +static bool gsp_phys_rmcall_gr_init_emitted; + +/* + * r425 stale-QMD witness: the first engine-changed page of the LAST completed + * launch (bounce index + post-copy-out checksum). At a zero-write accept the + * page is re-scanned; a change proves a late store (a stuck/queued grid or a + * stale-QMD execution) landed after its launch was shipped. + */ +static u32 gsp_phys_gr_prev_out_bi; +static u64 gsp_phys_gr_prev_out_csum; +static bool gsp_phys_gr_prev_out_valid; + +/* r425: SKED/FECS/CWD register witness (one line; the healthy-vs-wedged + * comparison baseline lives in the same log). */ +static void gsp_phys_gr_sked_wit(void __iomem *bar0, const char *tag) +{ + if (!bar0) + return; + pr_info("vgsp-gr: SKED-WIT %s cwd0=%08x cwd1=%08x cwd2=%08x cwd3=%08x info1=%08x act=%08x grfifo=%08x fecs=%08x new=%08x ctxsw1=%08x qmdvirt0=%08x tpcm0=%08x\n", + tag, + readl(bar0 + 0x00407060u), readl(bar0 + 0x00407064u), + readl(bar0 + 0x00407068u), readl(bar0 + 0x0040706cu), + readl(bar0 + 0x00407028u), readl(bar0 + 0x00407054u), + readl(bar0 + 0x00400504u), readl(bar0 + 0x00409b00u), + readl(bar0 + 0x00409b04u), readl(bar0 + 0x00409400u), + readl(bar0 + 0x0040a500u), readl(bar0 + 0x0040a680u)); +} + /* m86: program-page override blob (see gsp-phys-gr-override). */ static u8 gsp_phys_gr_override[GSP_PHYS_PAGE_SIZE]; static u32 gsp_phys_gr_override_len; @@ -16374,6 +16707,29 @@ static u32 *gsp_phys_rmcall_build_gr_init(u32 *p) GSP_PHYS_RMCALL_GR_MAX_SM_COUNT); p = gsp_phys_rmcall_push_gr(p, GSP_PHYS_RMCALL_GR_INIT_SM_DISP_CTRL, 0u); + /* + * r434 CWD-slot-count path (gsp_phys_rmcall_gr_cwd_slot >= 0): masked + * PRI write of SKED_DEBUG_1.ENABLE_CWD_SLOT_COUNT (bit 11) through the + * falcon scratch registers, then the gated SET_CWD_SLOT_COUNT method - + * the CUDA driver's own turingLaunchSetCwdSlotCount sequence. -1 (the + * default) emits nothing, so the historical preamble is unchanged. + */ + if (gsp_phys_rmcall_gr_cwd_slot_value() >= 0) { + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH0, 0u); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH1, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH2, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_SET_FALCON04, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_SET_CWD_SLOT_COUNT, + (u32)gsp_phys_rmcall_gr_cwd_slot_value()); + } return p; } @@ -16394,7 +16750,17 @@ static u32 gsp_phys_rmcall_build_gr_launch(u32 *pb, int qmdvirt = gsp_phys_rmcall_gr_qmdvirt_value(); int two_step = gsp_phys_rmcall_gr_pcas_two_step_value(); - p = gsp_phys_rmcall_build_gr_init(p); + /* + * r425 A/B: with gsp_phys_rmcall_gr_init_once the compute init preamble + * is emitted only on the first forwarded launch (the stock driver's + * per-channel shape); subsequent launches start at the inline-QMD + * window. + */ + if (!READ_ONCE(gsp_phys_rmcall_gr_init_once_param) || + !READ_ONCE(gsp_phys_rmcall_gr_init_emitted)) { + p = gsp_phys_rmcall_build_gr_init(p); + WRITE_ONCE(gsp_phys_rmcall_gr_init_emitted, true); + } if (qmdvirt) p = gsp_phys_rmcall_push_gr(p, GSP_PHYS_RMCALL_GR_SET_QMD_VIRT_CONTROL, @@ -17130,6 +17496,54 @@ static struct gsp_phys_boot_chunk *gsp_phys_gr_data_chunk(void) } /* + * r452: scan the whole active GR data-bounce chunk for an 8-byte pattern + * (write "0x<pattern>" to /sys/module/vgsp/parameters/gsp_phys_gr_scan). + * Verifies by content where a lost engine store actually landed: the chunk + * is the only bulk memory the operand window PTEs map, so a store that + * executed but is absent from the writer's tracked pages must be findable + * here (a stale-PTE landing) or nowhere (a dropped store). + */ +static int gsp_phys_gr_scan_set(const char *val, + const struct kernel_param *kp) +{ + struct gsp_phys_boot_chunk *c = gsp_phys_gr_data_chunk(); + u64 pattern = 0; + u32 i, hits = 0; + + if (kstrtoull(val, 0, &pattern)) + return -EINVAL; + if (!c->pages) + return -ENODEV; + for (i = 0; i < c->pages; i++) { + const u8 *p = gsp_phys_chunk_page_va(c, i); + u32 off; + + if (!p) + continue; + for (off = 0; off + 8 <= GSP_PHYS_PAGE_SIZE; off += 8) { + if (get_unaligned_le64(p + off) == pattern) { + if (hits < 32) + pr_info("vgsp-gr: CHUNK-SCAN found 0x%llx at bi=%u off=0x%x (va=0x%llx)\n", + pattern, i, off, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + ((u64)i << 12) + off); + hits++; + } + } + } + pr_info("vgsp-gr: CHUNK-SCAN 0x%llx done: %u hit(s) over %u pages\n", + pattern, hits, c->pages); + return 0; +} + +static const struct kernel_param_ops gsp_phys_gr_scan_ops = { + .set = gsp_phys_gr_scan_set, +}; +module_param_cb(gsp_phys_gr_scan, &gsp_phys_gr_scan_ops, NULL, 0200); +MODULE_PARM_DESC(gsp_phys_gr_scan, + "Scan the GR data-bounce chunk for an 8-byte pattern (hex)"); + +/* * B-2: page index of a relocated operand VA inside the engine-visible data * region, replacing the five inlined (reloc_va - 0x110000) >> 12 derivations * and their MAIL_SLOT bounds. Grown region: VAs from @@ -17579,7 +17993,7 @@ static bool gsp_phys_gr_reloc_fix_word(const struct gsp_phys_gr_submit *req, */ static noinline u32 gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, - const u32 *csum_in, u32 *n_tracked, u64 *hash_out) + const u64 *csum_in, u32 *n_tracked, u64 *hash_out) { static u32 checked[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); @@ -17625,15 +18039,24 @@ gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, *n_tracked = n_all; for (j = 0; j < n_all; j++) { const u8 *src = gsp_phys_chunk_page_va(dchunk, checked[j]); - u32 ci, csum = 0; + u64 fp; if (!src) continue; - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != csum_in[checked[j]]) + /* + * r452: FNV-1a per page, not a 32-bit additive sum. The + * add-sum was PROVABLY blind to a power-of-two in-place + * scale: doubling every word of a 4 KB page shifts each + * exponent by +1, i.e. +2^23 per word, and 1024 words give + * 2^33 == 0 (mod 2^32) - so c[i] *= 2.0f left the checksum + * bit-identical, the page classified "unchanged", and the + * client read the pre-scale values (the deterministic + * streams2/scale class). FNV-1a has no such structure. + */ + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp != csum_in[checked[j]]) changed++; - hash ^= csum; + hash ^= fp; hash *= 1099511628211ULL; } if (hash_out) @@ -17641,9 +18064,35 @@ gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, return changed; } +/* + * r434: post-launch deposit dump (hardware-only words). The r430 QMD64 dump + * runs at submission, before the engine touches the deposit, so the + * hardware-only fields (w50 INNER_GET, w51 INNER_PUT, w52 span-list head, + * w53 next-QMD) are always zero there. Reading the same page AFTER the + * retire wait shows what the SKED actually consumed: w50/w51 are the CTA + * (INNER) get/put counters, so a wedged launch whose grid "completed" with + * zero stores is distinguishable from a genuinely dispatched grid by + * whether INNER_PUT advanced to the CTA count. One line per launch, gated + * by the skedrec param. + */ +static void gsp_phys_gr_qmd64_post(const u8 *deposit, const char *tag) +{ + char qb[64 * 9 + 1]; + int qo = 0; + u32 i; + + if (!deposit || !gsp_phys_rmcall_gr_skedrec_value()) + return; + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS && qo < (int)sizeof(qb) - 10; + i++) + qo += scnprintf(qb + qo, sizeof(qb) - qo, "%08x ", + get_unaligned_le32(deposit + i * 4)); + pr_info("vgsp-gr: QMD64-POST %s %s\n", tag, qb); +} + static noinline u32 gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, - const u32 *csum_in, u32 *n_tracked) + const u64 *csum_in, u32 *n_tracked) { u32 n_all = 0, iter; u32 changed = 0; @@ -17661,7 +18110,33 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, *n_tracked = 0; if (!req->n_data) return 0; - n_all = gsp_phys_gr_scan_pages(req, csum_in, n_tracked, NULL); + /* + * r343: scan_pages RETURNS the changed count and reports the tracked + * total through the out-param. The old call put the return (changed) + * into n_all while the tracker total went to *n_tracked, so the + * loop's `changed >= n_all` was `changed >= changed` - always true + * after the first scan (observed: "1/1 pages changed after 4 ms"), + * and the `!n_all` early-return also skipped the no-write settle + * when zero pages changed. Net effect: the copy-out shipped the + * bounce mid-ramp, so a cold engine (slowest store ramp) produced + * all-bad first runs (the matmul cold flake) and late stores could + * be dropped. Fix: tracked total into n_all; the content-stable + * rule below becomes the real settle bound. + */ + (void)gsp_phys_gr_scan_pages(req, csum_in, &n_all, NULL); + /* + * r405: the r343 fix moved the tracked total from the out-param to + * the local n_all but FORGOT to write it back, so *n_tracked stayed + * 0 for every launch and the whole no-write classification chain in + * the caller (`wrote == 0 && n_tracked`: first-attempt witness, + * outcome retries, NOWRITE accept + release restore, late-store + * check) has been dead since ed91540a/r345 - the r404 forensics + * show "RETIRE-WIT n_data=968 n_tracked=0" while the same wait + * reports n_all=1032 in the "0/1032 pages changed (no writes)" + * line. Propagate the tracked total so the classifier can fire. + */ + if (n_tracked) + *n_tracked = n_all; /* * RETIRE-WIT (r227): the r227 capture shows n_tracked far below the * copy-in/copy-out rec counts for the same launch (4 tracked vs 1482 @@ -17699,7 +18174,8 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, * kernel costs at most 300 ms extra; a ramping engine gets * the time its stores need. */ - if (changed > 0 && hash == prev_hash && iter >= 60u) { + if (changed > 0 && hash == prev_hash && + iter >= gsp_phys_gr_settle_iters()) { why = " (settled)"; break; } @@ -17712,6 +18188,22 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, } pr_info("vgsp-gr: output retirement wait: %u/%u pages changed after %u ms%s\n", changed, n_all, jiffies_to_msecs(jiffies - wait_start), why); + /* + * r405: run-once late-write probe (see the param comment above). + */ + if (READ_ONCE(gsp_phys_gr_late_probe_param) && changed < n_all) { + u32 late_n = 0, late; + unsigned long lp_start = jiffies; + + msleep(500); + late = gsp_phys_gr_scan_pages(req, csum_in, &late_n, NULL); + pr_info("vgsp-gr: retire LATE-PROBE %u -> %u of %u after +%u ms%s\n", + changed, late, late_n, + jiffies_to_msecs(jiffies - lp_start), + late != changed ? + " (ENGINE STILL WRITING - stores landed late)" : + " (stable - writes are final)"); + } return changed; } @@ -17734,6 +18226,82 @@ static bool gsp_phys_gr_qmd_fetched(const u8 *qmd_mem) } /* + * r427 idempotent fast-accept (CONT.86 §6.1). A zero-write launch is either + * the no-execute wedge OR a legitimate idempotent re-launch (the same QMD + * content with the same staged inputs, whose result is already in place) - + * the host cannot tell them apart, and both burn the 40 s + 80 s quiet holds. + * The launch identity below (QMD minus the per-launch release0 address and + * payload, folded with the staged content checksums of every tracked page) is + * recorded for every launch that WROTE; a zero-write launch whose identity + * matches an entry accepts immediately, with no retries and no quiet window. + * Safe by construction: identical code + identical staged content + a prior + * execution that produced this launch's own footprint means the correct + * result is already in the buffers; a side-effecting kernel's re-launch is + * equally wrong under the current accept (the client sees the stale result + * either way) - this only removes the stall. + */ +#define GSP_PHYS_GR_IDEM_SLOTS 64u +static u64 gsp_phys_gr_idem_fp[GSP_PHYS_GR_IDEM_SLOTS]; +static u32 gsp_phys_gr_idem_n; + +static u64 gsp_phys_gr_launch_fp(const u32 *qmd, + const struct gsp_phys_gr_submit *req, + const u64 *csum_in) +{ + u64 h = 1469598103934665603ULL; + u32 i; + + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS; i++) { + /* Skip release0's address (w23/w24) and payload (w25): the + * driver allocates a fresh release slot per launch, so they + * differ even for an identical re-launch. */ + u32 w; + + if (i == 23u || i == 24u || i == 25u) + continue; + w = qmd[i]; + h ^= w; + h *= 1099511628211ULL; + } + for (i = 0; i < req->n_data; i++) { + u32 bi = gsp_phys_gr_data_bi(req->data[i].reloc_va); + + if (bi == GSP_PHYS_GR_DATA_BI_NONE) + continue; + h ^= csum_in[bi]; + h *= 1099511628211ULL; + } + return h; +} + +static bool gsp_phys_gr_idem_match(u64 fp) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_idem_n; i++) + if (gsp_phys_gr_idem_fp[i] == fp) + return true; + return false; +} + +static void gsp_phys_gr_idem_record(u64 fp) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_idem_n; i++) + if (gsp_phys_gr_idem_fp[i] == fp) + return; + if (gsp_phys_gr_idem_n < GSP_PHYS_GR_IDEM_SLOTS) { + gsp_phys_gr_idem_fp[gsp_phys_gr_idem_n++] = fp; + } else { + /* ring: drop the oldest */ + memmove(&gsp_phys_gr_idem_fp[0], &gsp_phys_gr_idem_fp[1], + (GSP_PHYS_GR_IDEM_SLOTS - 1u) * sizeof(u64)); + gsp_phys_gr_idem_fp[GSP_PHYS_GR_IDEM_SLOTS - 1u] = fp; + } +} + +/* * COPYAUDIT (2026-10-18 copy-completeness audit): per-copy sighting of the * GR operand staging copies. Every staged page is spot-checked (16 bytes * at the start and end of the page, source vs destination, right after the @@ -17756,6 +18324,22 @@ static void gsp_phys_gr_page_spot_check(const u8 *dst, const u8 *src, } /* + * r452: the legacy 32-bit additive page sum, kept ONLY for the all-zero + * filter. It is never a change-detection fingerprint any more: doubling every + * word of a 4 KB page leaves the sum bit-identical (1024 x 2^23 == 2^33 == 0 + * mod 2^32), which silently classified an in-place x2 scale as "no engine + * store" and shipped stale values (the deterministic streams2 class). + */ +static inline u32 gsp_phys_gr_page_sum32(const u8 *p) +{ + u32 ci, csum = 0; + + for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) + csum += get_unaligned_le32(p + ci); + return csum; +} + +/* * COPYWIT r226 zero filter: an all-zero page is never a meaningful alias * partner. A fresh client's fill stages long runs of zero pages * (uninitialized cudaMalloc buffers), and consecutive zero pages produced @@ -17801,7 +18385,7 @@ static u32 gsp_phys_gr_in_census_log; * again). Returns the number of pages staged. */ static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, - u32 *csum_in) + u64 *csum_in, u64 *staged_hpa) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); struct { @@ -17826,11 +18410,21 @@ static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, if (!dst) continue; memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + /* + * r378: record which guest page's content now occupies this + * bounce slot. The copy-out may only write the slot back to + * that page - a rec whose reloc VA aliases another rec's slot + * would otherwise ship the other page's content into this + * guest page (the multi-writer clobber class). + */ + staged_hpa[bi] = req->data[di].hpa; gsp_phys_gr_page_spot_check(dst, src, req->data[di].va, "gr-bounce-copy-in"); for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) csum += get_unaligned_le32(dst + ci); - csum_in[bi] = csum; + /* r452: the change baseline is the FNV-1a fingerprint (the + * add-sum above survives only as the all-zero test). */ + csum_in[bi] = gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); zero = gsp_phys_gr_page_all_zero(csum, dst); if (!zero && gsp_phys_gr_copywit_enabled()) { u32 j; @@ -17879,7 +18473,7 @@ static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, * retry, so a retried launch starts from the guest inputs in both sets. * Returns the number of pages staged. */ -static u32 gsp_phys_gr_fill_gap_bounce(u32 *csum_in) +static u32 gsp_phys_gr_fill_gap_bounce(u64 *csum_in, u64 *staged_hpa) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); u32 gi, copied = 0; @@ -17887,16 +18481,18 @@ static u32 gsp_phys_gr_fill_gap_bounce(u32 *csum_in) for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { const u8 *src = (const u8 *)__va(gsp_phys_gr_gap[gi].hpa); u8 *dst = gsp_phys_chunk_page_va(dchunk, gsp_phys_gr_gap[gi].bi); - u32 ci, csum = 0; if (!src || !dst) continue; memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + /* r378: the slot now holds THIS page's content (see the + * tracked-page fill above). */ + staged_hpa[gsp_phys_gr_gap[gi].bi] = gsp_phys_gr_gap[gi].hpa; gsp_phys_gr_page_spot_check(dst, src, gsp_phys_gr_gap[gi].va, "gr-bounce-copy-in"); - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(dst + ci); - csum_in[gsp_phys_gr_gap[gi].bi] = csum; + /* r452: FNV-1a baseline (see gsp_phys_gr_scan_pages). */ + csum_in[gsp_phys_gr_gap[gi].bi] = + gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); copied++; } dma_wmb(); @@ -18006,6 +18602,8 @@ gsp_phys_gr_poll_break_log(u32 w0, u32 want_qmd, u32 w1, u32 since_us, w0, want_qmd, w1, (u32)ktime_to_us(ktime_get()) - since_us, usr_gpget, sub->gpget, sub->gpput); + /* r425: the healthy-launch SKED/FECS baseline for the wedge compare. */ + gsp_phys_gr_sked_wit(gsp_phys_gpu_bar0(), "post-poll"); } /* @@ -18157,6 +18755,8 @@ static u32 gsp_phys_m91_fault_start; static u32 gsp_phys_m91_cap_used; static bool gsp_phys_m91_cap_suppressed; static bool gsp_phys_m91_icache_log; +static bool gsp_phys_r449_ccache_log; +static bool gsp_phys_gr_rel_membar_log; static noinline void gsp_phys_gr_m91_fault_cap(const struct gsp_phys_gr_submit *req, const u32 *qmd, u64 qmd_dst, const u8 *qmd_dst_mem, @@ -18466,6 +19066,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, const u8 *guest_param_pg; u32 want_qmd = 0, want_ch = 0; bool qmd_rel = false, ch_rel = false; + bool qmd_rel_redirected = false; /* r420b: release0 -> module sem */ u64 qmd_rel_hpa = 0, ch_rel_hpa = 0; u64 ch_sem_va = GSP_PHYS_RMCALL_VA_SEM + 4; u64 qmd_dst = GSP_PHYS_RMCALL_VA_SRC; @@ -18487,7 +19088,17 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * stack). Safe as static: the only caller is the single-inflight * ordered gsp-phys-work item. */ - static u32 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + static u64 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + /* + * r378: which guest page's content each bounce slot holds this + * launch (set by the copy-in fills, cleared per submission). The + * copy-out writes a slot back only to that exact page, so a rec + * aliasing another rec's slot - or a slot left over from a previous + * launch - can never ship foreign content into a guest page. + * Static like bounce_csum_in: the single-inflight ordered + * gsp-phys-work item is the only caller. + */ + static u64 bounce_staged_hpa[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; sked_tick = 0; sked_first = true; guest_param_pg = NULL; @@ -18552,6 +19163,12 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, */ memset(&fault, 0, sizeof(fault)); /* + * r378: the bounce staging identity is per launch. A slot left over + * from a previous launch (or never staged this one) must not be + * treated as this launch's page even when the HPA matches. + */ + memset(bounce_staged_hpa, 0, sizeof(bounce_staged_hpa)); + /* * F2: the shortened first window is part of the fast path; with * gsp_phys_gr_retry_fast=0 the launch keeps today's exact behavior * (the historical 5 s first window and ladder). @@ -18649,6 +19266,52 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, } } /* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default + * 0 = off): the same cache-control word's bit 31 (the V01_06-layout + * SHADER_CONSTANT_CACHE position) is never set by the captured guest + * QMDs. The forwarded param page is rewritten in place between launches + * (the two-bank win0 pool), so a stale constant-cache line can serve + * the PREVIOUS launch's parameters and the current launch's output + * pages stay untouched - the post-dispatch no-store signature. Mode 1 + * sets bit 31; mode 2 forces the whole invalidate set (bits 24..31). + * Only when the QMD carries the driver's standard cache-control + * signature (bits 26..29) so an unknown QMD revision is never touched. + */ + if (gsp_phys_gr_ccache_inv_value() && + (qmd[5] & 0x3c000000u) == 0x3c000000u) { + u32 mode = gsp_phys_gr_ccache_inv_value(); + + if (mode == 2) + qmd[5] |= 0xff000000u; + else + qmd[5] |= 1u << 31; + if (!gsp_phys_r449_ccache_log) { + gsp_phys_r449_ccache_log = true; + pr_info("vgsp-gr: r449 constant-cache invalidate mode=%u applied (w5=0x%08x)\n", + mode, qmd[5]); + } + } + /* + * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar): force the + * forwarded QMD's RELEASE_MEMBAR_TYPE to FE_SYSMEMBAR (QMDV00_06 + * word 11 bit 14) so the front-end flushes its pending system-memory + * stores before writing the release semaphore. The captured guest + * QMDs carry FE_NONE, so a grid that completes with its data still + * in the GPU's L2 releases the client while no store is host-visible + * - the post-dispatch store-loss signature. The release ordering then + * guarantees the host's retire scan and copy-out can never observe a + * completion whose data has not landed. 0 (default) keeps the + * guest's captured value. + */ + if (gsp_phys_gr_rel_membar_value() && !(qmd[11] & (1u << 14))) { + qmd[11] |= 1u << 14; + if (!gsp_phys_gr_rel_membar_log) { + gsp_phys_gr_rel_membar_log = true; + pr_info("vgsp-gr: r433 RELEASE_MEMBAR_TYPE forced to FE_SYSMEMBAR in the forwarded QMD (w11=0x%08x)\n", + qmd[11]); + } + } + /* * Two-step PCAS (gsp_phys_rmcall_gr_pcas_two_step): set the QMD's * REQUIRE_SCHEDULING_PCAS (w4 bit12) so the copy PCAS does not * schedule and the trailing schedule-only PCAS does. @@ -18707,6 +19370,15 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, */ qmd_rel_hpa = 0; qmd_rel_off = 0; + /* + * r420b: the ENGINE will not write the guest's QMD + * release word this launch (rel_mod, or no targetable + * guest release page). Tell the caller so it authors + * the word in software on every outcome - otherwise a + * launch without deferred release records is never + * completed on the guest word (the r414 saxpy hang). + */ + qmd_rel_redirected = true; } } @@ -19466,6 +20138,27 @@ gr_page_unmapped: /* A stale release from a previous submission must not be observed. */ WRITE_ONCE(*(u32 *)sem_va, 0); WRITE_ONCE(*(u32 *)(sem_va + 4), 0); + /* + * r441: zero the GUEST-VISIBLE release words for the first attempt + * too, not only for outcome retries. The r439 evidence shows a first + * attempt whose poll broke on a STALE value (qmd_rel=0x00000005 from + * the previous launch's engine write, want=6) while the engine never + * fetched this launch's deposit (sentinel intact, "engine fetch + * NONE"): the host then shipped a completion for a launch that did + * not run. Zeroing forces the poll to wait for THIS launch's own + * engine write; when the engine does not write it, the timeout ladder + * and the software-authored completion take over (the honest path), + * and the wedge classification stops being satisfiable by a stale + * value. The write guard re-validates the target like the retry path. + */ + if (qmd_rel_hpa && qmd_rel_off < GSP_PHYS_PAGE_SIZE && + gsp_sema_gr_write_guard(req->rel_va, qmd_rel_hpa, + "gr-rel-zero-first")) + WRITE_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + qmd_rel_off), 0); + if (ch_rel_hpa && ch_rel_off < GSP_PHYS_PAGE_SIZE && + gsp_sema_gr_write_guard(req->rel_va, ch_rel_hpa, + "gr-chrel-zero-first")) + WRITE_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + ch_rel_off), 0); wmb(); /* @@ -19492,6 +20185,20 @@ gr_page_unmapped: pr_info("vgsp-gr: qmd w0..3=%08x %08x %08x %08x w4..7=%08x %08x %08x %08x w23..25=%08x %08x %08x w32/33=%08x %08x w48/49=%08x %08x\n", qmd[0], qmd[1], qmd[2], qmd[3], qmd[4], qmd[5], qmd[6], qmd[7], qmd[23], qmd[24], qmd[25], qmd[32], qmd[33], qmd[48], qmd[49]); + /* + * r430: full 64-word deposit dump for the wedge's healthy-vs-wedged + * QMD diff (hardware-only fields w50/w51 INNER_GET/PUT, w52 span-list, + * w53 next-QMD pointer are not in the partial line above). Gated by + * the skedrec param (default on); one long line per launch. + */ + if (gsp_phys_rmcall_gr_skedrec_value()) { + char qb[64 * 9 + 1]; + int qo = 0; + + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS && qo < (int)sizeof(qb) - 10; i++) + qo += scnprintf(qb + qo, sizeof(qb) - qo, "%08x ", qmd[i]); + pr_info("vgsp-gr: QMD64 %s\n", qb); + } /* m120: dump the main PT page slots 0x100..0x11f so the PTE content * the walker would read (win0 + bounce window) is visible in the log. */ { @@ -19572,7 +20279,8 @@ gr_page_unmapped: ret = -EAGAIN; goto out; } - dcopied = gsp_phys_gr_fill_bounce(req, bounce_csum_in); + dcopied = gsp_phys_gr_fill_bounce(req, bounce_csum_in, + bounce_staged_hpa); pr_info("vgsp-gr: bounce copy-in: %u pages, %u skipped (pre-publish)\n", dcopied, dfailed); /* @@ -19583,7 +20291,7 @@ gr_page_unmapped: * any point in the pipeline where that pattern breaks names * the bug site. */ - if (req->n_data <= 64u && gsp_phys_gr_atrace_budget < 128u) { + if (req->n_data <= 1024u && gsp_phys_gr_atrace_budget < 128u) { u32 d; for (d = 0; d < req->n_data && d < 16u; d++) { @@ -19610,10 +20318,28 @@ gr_page_unmapped: req->data[0].va, (u64)req->n_data << 12, (u64)dcopied << 12); - for (di = 0; di < req->n_data && di < 3; di++) - pr_info("vgsp-gr: data[%u] va=0x%llx hpa=0x%llx reloc=0x%llx\n", + for (di = 0; di < req->n_data && di < 3; di++) { + u32 w0 = 0, w1 = 0; + + /* + * fail4 definitive witness: the bytes the engine will + * read for the first recs (the rec hpa's content) + * beside the rec identity - names a +2 MB rec-page + * phase directly (client a[0..3] = 00000000 + * 0000003f 0000003f... would appear as exp[2M] + * values if the hpa points 512 pages late). + */ + if (req->data[di].hpa) { + const u32 *p = (const u32 *)__va( + req->data[di].hpa); + + w0 = p[0]; + w1 = p[1]; + } + pr_info("vgsp-gr: data[%u] va=0x%llx hpa=0x%llx reloc=0x%llx first8=%08x %08x\n", di, req->data[di].va, req->data[di].hpa, - req->data[di].reloc_va); + req->data[di].reloc_va, w0, w1); + } } /* @@ -19624,7 +20350,8 @@ gr_page_unmapped: * like the block above. */ if (gsp_phys_gr_gap_n && gsp_phys_gr_data_chunk()->pages) { - u32 gcopied = gsp_phys_gr_fill_gap_bounce(bounce_csum_in); + u32 gcopied = gsp_phys_gr_fill_gap_bounce(bounce_csum_in, + bounce_staged_hpa); pr_info("vgsp-gr: gap copy-in: %u late-resolved pages, %llu word(s) fixed first=0x%llx\n", gsp_phys_gr_gap_n, gsp_phys_gr_reloc_fixed, @@ -19796,8 +20523,14 @@ gr_page_unmapped: * cancel or dispatch sequence of the launch window is on the * record even when the poll times out. */ - if (gsp_phys_rmcall_gr_skedrec_value() && bar0 && - (++sked_tick & 7u) == 0u) { + if (gsp_phys_rmcall_gr_skedrec_value() && bar0) { + /* + * r430: sample EVERY poll iteration (not every 8th): + * the wedge's release word lands ~7 ms after the + * doorbell, so the old 8-iteration gate saw at most one + * sample of the dispatch window. Changes only. + */ + sked_tick++; sked_sample[0] = ioread32(bar0 + 0x00407060u); sked_sample[1] = ioread32(bar0 + 0x00407064u); sked_sample[2] = ioread32(bar0 + 0x00407068u); @@ -19813,7 +20546,7 @@ gr_page_unmapped: memcmp(sked_sample, sked_prev, sizeof(sked_sample))) { pr_info("vgsp-gr: sked rec +%ums cwd0=%08x cwd1=%08x cwd2=%08x info1=%08x info2=%08x act=%08x esr=%08x grfifo=%08x fecs_cur=%08x fecs_new=%08x ctxsw1=%08x\n", - (sked_tick / 8u) * + sked_tick * GSP_PHYS_RMCALL_POLL_MS, sked_sample[0], sked_sample[1], sked_sample[2], sked_sample[3], @@ -20349,10 +21082,19 @@ gr_page_unmapped: if (req->n_data && gsp_phys_gr_data_chunk()->pages) { u32 n_tracked = 0; u32 nw_attempts = 0; + bool idem = false; u32 wrote = gsp_phys_gr_output_retire_wait(req, bounce_csum_in, &n_tracked); /* + * r434: post-launch hardware-only QMD dump (w50/w51 INNER + * get/put = the SKED's CTA dispatch counters) so the + * healthy-vs-wedged diff has the fields the submission-time + * dump cannot show. + */ + gsp_phys_gr_qmd64_post(qmd_dst_mem, "retire"); + + /* * m179: the zero-write test is specific to launches with a * non-empty TRACKED data set. A launch whose data entries do * not resolve to any bounce page legitimately reports zero @@ -20368,12 +21110,55 @@ gr_page_unmapped: ff ? "OK" : "NONE", ff ? "overwritten" : "intact", n_tracked); } - if (wrote == 0 && n_tracked && + /* + * r427 idempotent fast-accept: a zero-write launch whose identity + * (QMD minus release0 + staged content checksums) matches an + * earlier WRITING launch of this boot is a legitimate idempotent + * re-launch - the result is already in place. Accept immediately: + * no retries, no quiet holds. (See gsp_phys_gr_launch_fp.) + */ + if (wrote == 0 && n_tracked) { + u64 fp = gsp_phys_gr_launch_fp(qmd, req, bounce_csum_in); + + if (gsp_phys_gr_idem_match(fp)) { + pr_warn("vgsp-gr: IDEMPOTENT-ACCEPT h=0x%x push=0x%llx fp=0x%llx tracked=%u (same QMD+content as an earlier writing launch; no retries, no hold)\n", + req->h_object, req->push_va, fp, + n_tracked); + idem = true; + } + } + if (wrote == 0 && n_tracked && !idem && gsp_phys_rmcall_gr_nowrite_retry_value()) { u32 nw_retries = (u32)gsp_phys_gr_nowrite_retries_value(); + /* + * r410: the sticky per-lane wedge (previous launch + * also no-wrote while the lane stays busy) is only + * cleared by idleness (r409 ~37 s, r416 ~52 s), so a + * back-to-back retry is futile and keeps it alive. + * When the streak says the lane is wedged, hold a + * quiet window with NO submissions before the retry + * (base on the first retry, doubled on the second), + * then let the normal fresh-bank + resched retry run. + * The quiet path relaxes the whole-launch budget + * (its own hold is the bound: 2 holds max) - the + * alternative is accepting every launch silently. + */ + u32 quiet_used = 0; while (wrote == 0 && nw_attempts < nw_retries) { + u32 quiet_ms = 0; + + if ((READ_ONCE(gsp_phys_gr_nw_streak) > 0 || + READ_ONCE(gsp_phys_gr_wedge_episode)) && + nw_attempts <= 1) { + quiet_ms = + gsp_phys_gr_wedge_quiet_ms_value(); + if (quiet_ms && nw_attempts) + quiet_ms <<= 1; + if (quiet_ms > 240000u) + quiet_ms = 240000u; + } /* * STALLFREE: the same whole-launch budget caps * the retirement retry chain. When it is crossed @@ -20381,9 +21166,13 @@ gr_page_unmapped: * its current outcome (the zero-write accept * below restores the release words and logs * loudly), so a chain of slow dud retries can - * never hold the client past the budget. + * never hold the client past the budget. The + * r410 quiet path is exempt: it is entered only + * for a launch whose previous launch also + * no-wrote, and its own holds are bounded. */ - if (gsp_phys_gr_wait_budget_over(start)) { + if (!quiet_ms && + gsp_phys_gr_wait_budget_over(start)) { gsp_phys_gr_stall_retry++; gsp_phys_gr_stall_log("retry", req->push_va, start, @@ -20395,6 +21184,52 @@ gr_page_unmapped: nw_attempts, nw_retries, wrote, n_tracked); /* + * r445 wedge hard re-arm (gsp_phys_gr_wedge_rearm, + * default off): the r445 evidence shows every + * post-dispatch no-store episode leaves the warm + * ladder sem probe ALIVE (the ladder block executes) + * while the client launch keeps writing nothing, so + * the plain refill+re-publish retries are futile. + * On the FINAL retry attempt, force one context + * save/unload (TSG PREEMPT bWait=1) and re-run the + * mode-35 lane-enable tail + schedules before the + * re-publish, so the retried launch meets a freshly + * restored context. Bounded: one preempt per launch + * (2 s RPC timeout), only on the last retry. + */ + if (nw_attempts == nw_retries && + gsp_phys_gr_wedge_rearm_value()) { + mutex_lock(&gsp_phys_rm_lock); + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + } + if (quiet_ms) { + gsp_phys_gr_wedge_quiet++; + pr_warn("vgsp-gr: WEDGE-QUIET hold %u ms before outcome retry %u (push=0x%llx; %u consecutive zero-write launch(es); no submissions in between)\n", + quiet_ms, nw_attempts, + req->push_va, + READ_ONCE(gsp_phys_gr_nw_streak)); + /* + * r420: publish the deliberate hold so + * the sema service's completion + * deadlines (the 12 s GR release hold + * and the 3 s per-launch consumption + * bound) pause instead of authoring + * this launch's release word under it - + * the client must not be released + * before the post-hold copy-out + * (gsp_phys_gr_quiet_hold_active). + */ + WRITE_ONCE(gsp_phys_gr_quiet_hold_on, + true); + msleep(quiet_ms); + WRITE_ONCE(gsp_phys_gr_quiet_hold_on, + false); + quiet_used = quiet_ms; + pr_warn("vgsp-gr: WEDGE-QUIET hold done (push=0x%llx); lanes re-armed by the retry below\n", + req->push_va); + } + /* * m156 (E5): on the first zero-write retry, push the * proven stage-4 GR semaphore block before re-publishing. * It is the discriminator the lottery report asked for: @@ -20445,7 +21280,15 @@ gr_page_unmapped: * recovered the no-execute launch (fresh ring * entry + resched + fresh deposit). */ - if (qmd_rel_hpa) + /* + * r378: re-validate the guest release page + * before zeroing it - the HPA was resolved at + * submission and the retry runs seconds later. + */ + if (qmd_rel_hpa && + gsp_sema_gr_write_guard(req->rel_va, + qmd_rel_hpa, + "gr-rel-zero")) WRITE_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + qmd_rel_off), 0); if (ch_rel_hpa) @@ -20549,8 +21392,10 @@ gr_page_unmapped: pr_info("vgsp-gr: outcome retry %u: release words zeroed (win0/qmdwin off)\n", nw_attempts); } - gsp_phys_gr_fill_bounce(req, bounce_csum_in); - gsp_phys_gr_fill_gap_bounce(bounce_csum_in); + gsp_phys_gr_fill_bounce(req, bounce_csum_in, + bounce_staged_hpa); + gsp_phys_gr_fill_gap_bounce(bounce_csum_in, + bounce_staged_hpa); mutex_lock(&gsp_phys_rm_lock); ret = gsp_phys_rmcall_publish_locked( bar0, boot_stage.rmcall_va_pb, @@ -20638,8 +21483,26 @@ gr_page_unmapped: } wrote = gsp_phys_gr_output_retire_wait( req, bounce_csum_in, &n_tracked); + gsp_phys_gr_qmd64_post(qmd_dst_mem, + "retry-retire"); pr_info("vgsp-gr: outcome retry %u done: engine wrote %u page(s)\n", nw_attempts, wrote); + /* r410: the quiet-window verdict. */ + if (quiet_used) { + if (wrote) { + gsp_phys_gr_wedge_heals++; + WRITE_ONCE(gsp_phys_gr_wedge_episode, + true); + pr_warn("vgsp-gr: WEDGE-QUIET healed: engine wrote %u page(s) after the %u ms hold (streak was %u)\n", + wrote, quiet_used, + READ_ONCE( + gsp_phys_gr_nw_streak)); + } else { + pr_warn("vgsp-gr: WEDGE-QUIET no heal: %u ms quiet and retry %u still wrote 0 page(s)\n", + quiet_used, nw_attempts); + } + quiet_used = 0; + } } } /* @@ -20652,7 +21515,7 @@ gr_page_unmapped: * loop is disabled (gsp_phys_rmcall_gr_nowrite_retry=0) so a * zero-write outcome is never silent. */ - if (wrote == 0 && n_tracked) { + if (wrote == 0 && n_tracked && !idem) { bool restored = false; bool fetched = gsp_phys_gr_qmd_fetched(qmd_dst_mem); @@ -20670,11 +21533,14 @@ gr_page_unmapped: * may also be a legitimately read-only kernel * (AMBIGUOUS). */ - pr_warn("vgsp-gr: NOWRITE-%s h=0x%x push=0x%llx n_data=%u tracked=%u wrote=0 retries=%u qmd_rel=0x%x (engine fetch %s)\n", + pr_warn("vgsp-gr: NOWRITE-%s h=0x%x push=0x%llx n_data=%u tracked=%u wrote=0 retries=%u qmd_rel=0x%x (engine fetch %s) consec=%u quiet=%llu heals=%llu\n", fetched ? "AMBIGUOUS" : "CONFIRMED", req->h_object, req->push_va, req->n_data, n_tracked, nw_attempts, want_qmd, - fetched ? "OK" : "NONE"); + fetched ? "OK" : "NONE", + READ_ONCE(gsp_phys_gr_nw_streak) + 1u, + gsp_phys_gr_wedge_quiet, + gsp_phys_gr_wedge_heals); /* * m180 (lottery discriminator, agent-order H1/H2). @@ -20741,6 +21607,32 @@ gr_page_unmapped: readl(bar0 + 0x00409008u), readl(bar0 + 0x00409b00u), readl(bar0 + 0x00409b04u)); + /* r425: the wedged-launch SKED/FECS baseline. */ + gsp_phys_gr_sked_wit(bar0, "accept"); + /* + * r425 stale-store witness: re-scan the last completed + * launch's first engine-changed page. A change since its + * copy-out proves a late store landed after the launch + * was shipped (a stuck/queued grid or a stale-QMD + * execution), the mechanism behind the "1 page" heals. + */ + if (gsp_phys_gr_prev_out_valid && + bounce_staged_hpa[gsp_phys_gr_prev_out_bi] == 0 && + gsp_phys_gr_data_chunk()->pages) { + const u8 *pp = gsp_phys_chunk_page_va( + gsp_phys_gr_data_chunk(), + gsp_phys_gr_prev_out_bi); + u64 pc; + + if (pp) { + pc = gsp_phys_gr_fnv1a64(pp, GSP_PHYS_PAGE_SIZE); + if (pc != gsp_phys_gr_prev_out_csum) + pr_warn("vgsp-gr: PREV-OUT-CHANGED bi=%u fp %016llx -> %016llx (late store after its launch shipped)\n", + gsp_phys_gr_prev_out_bi, + gsp_phys_gr_prev_out_csum, + pc); + } + } /* * (c) Late-store check: a store ramp that retires after * the host's window is indistinguishable from a lost @@ -20766,7 +21658,14 @@ gr_page_unmapped: if (qmd_rel_hpa) { u8 *rp = (u8 *)__va(qmd_rel_hpa) + qmd_rel_off; - if (READ_ONCE(*(u32 *)rp) != want_qmd) { + /* + * r378: same target re-validation as the + * retry zero above. + */ + if (READ_ONCE(*(u32 *)rp) != want_qmd && + gsp_sema_gr_write_guard(req->rel_va, + qmd_rel_hpa, + "gr-rel-restore")) { WRITE_ONCE(*(u32 *)rp, want_qmd); restored = true; } @@ -20783,6 +21682,31 @@ gr_page_unmapped: pr_warn("vgsp-gr: no-write accept: release words restored (qmd=0x%08x ch=0x%08x); client's own check decides\n", want_qmd, want_ch); } + /* + * r427: record the launch identity of every launch that WROTE, + * so a later idempotent re-launch (zero writes, same identity) + * can accept immediately (see gsp_phys_gr_launch_fp). + */ + if (wrote && n_tracked) + gsp_phys_gr_idem_record( + gsp_phys_gr_launch_fp(qmd, req, bounce_csum_in)); + /* + * r410: streak bookkeeping for the quiet-window trigger. A + * launch with tracked data that wrote nothing extends the + * zero-write streak; any launch that wrote resets it. Runs + * even when the retry loop is disabled so the streak always + * reflects the accepted-outcome history. + */ + if (n_tracked) { + WRITE_ONCE(gsp_phys_gr_nw_streak, + wrote ? 0u : + READ_ONCE(gsp_phys_gr_nw_streak) + 1u); + /* r411: arm the wedge episode on any accepted + * zero-write launch, so the next zero-write launch + * quiets even after a heal reset the streak. */ + if (!wrote) + WRITE_ONCE(gsp_phys_gr_wedge_episode, true); + } } /* * m114: the kernel ran against the pre-commit bounce window; copy the @@ -20792,22 +21716,79 @@ gr_page_unmapped: */ if (req->n_data && gsp_phys_gr_data_chunk()->pages) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); - u32 wbi, wrote = 0, changed = 0; + u32 wbi, wrote = 0, changed = 0, eligible = 0; + u32 aliased = 0, unchanged = 0, skipped = 0; u32 chg[8] = { }; u32 outwit = 0; /* COPYWIT: bounded per-launch witness lines */ for (wbi = 0; wbi < req->n_data; wbi++) { u32 bi = gsp_phys_gr_data_bi(req->data[wbi].reloc_va); const u8 *src; - u32 ci, csum = 0; + u64 fp; if (bi == GSP_PHYS_GR_DATA_BI_NONE || !req->data[wbi].hpa) continue; src = gsp_phys_chunk_page_va(dchunk, bi); - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != bounce_csum_in[bi]) { + if (!src) + continue; + eligible++; + /* + * r378: only the page whose content was staged into + * this slot this launch may be written back. A rec + * aliasing another rec's reloc slot would otherwise + * ship the other page's content into this guest page + * (the multi-writer clobber class), and a slot left + * over from a previous launch likewise. + */ + if (bounce_staged_hpa[bi] != req->data[wbi].hpa) { + aliased++; + continue; + } + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { + /* + * r378: the engine did not touch this page, + * so the guest page still holds exactly this + * content (the slot was copied in from it). + * Writing it back is a no-op at best and + * stale-content corruption at worst - skip. + */ + unchanged++; + /* + * r452: witness the UNCHANGED page's chunk + * content (first 4 words) and the guest + * page's, bounded per launch. This is the + * only place the two can be compared for a + * page the csum rule classified as "engine + * did not touch" - it separates "the store + * never landed in the chunk" from "the + * store landed but the csum rule missed + * it" (or "the reader resolves a different + * page than the writer wrote"). + */ + if (gsp_phys_gr_copywit_enabled() && + outwit < GSP_PHYS_GR_COPYWIT_MAX) { + const u8 *gp = req->data[wbi].hpa ? + (const u8 *)__va(req->data[wbi].hpa) : + NULL; + + outwit++; + pr_info("vgsp-gr: OUT-UNCH bi=%u va=0x%llx fp=%016llx pre=%016llx chunk=%08x %08x %08x %08x guest=%08x %08x %08x %08x\n", + bi, req->data[wbi].va, fp, + bounce_csum_in[bi], + get_unaligned_le32(src), + get_unaligned_le32(src + 4), + get_unaligned_le32(src + 8), + get_unaligned_le32(src + 12), + gp ? get_unaligned_le32(gp) : 0, + gp ? get_unaligned_le32(gp + 4) : 0, + gp ? get_unaligned_le32(gp + 8) : 0, + gp ? get_unaligned_le32(gp + 12) : 0); + } + continue; + } + { u64 f8 = (u64)get_unaligned_le64(src); /* @@ -20840,14 +21821,31 @@ gr_page_unmapped: */ if (gsp_phys_gr_copywit_enabled() && outwit < GSP_PHYS_GR_COPYWIT_MAX) { + /* + * r391 diagnostic (sum=0 class): the + * landing-table resolution of this VA + * (the page the D2H reader will use) + * and its first 8 bytes beside the + * write target - a launch/readback + * disagreement in one line. + */ + u64 rpa = 0, rval = 0; + u32 rap = 0; + bool rhave = gsp_sema_gr_va_snap( + req->data[wbi].va, &rpa, + &rap, &rval); + outwit++; - pr_info("vgsp-gr: OUT-PAGE-WIT va=0x%llx hpa=0x%llx bi=%u first8=%016llx last8=%016llx pre=%08x post=%08x\n", + pr_info("vgsp-gr: OUT-PAGE-WIT va=0x%llx hpa=0x%llx bi=%u first8=%016llx last8=%016llx pre=%016llx post=%016llx res=%s pa=0x%llx ap=%u val8=%016llx vbase=%px\n", req->data[wbi].va, req->data[wbi].hpa, bi, f8, (u64)get_unaligned_le64( src + GSP_PHYS_PAGE_SIZE - 8), - bounce_csum_in[bi], csum); + bounce_csum_in[bi], fp, + rhave ? "ledger" : "none", + rpa, rap, rval, + (void *)gsp_sema_gr_shadow_base()); } /* * OUT-ALIAS-WIT: identical first 8 bytes as @@ -20859,7 +21857,8 @@ gr_page_unmapped: * evidence). */ if (gsp_phys_gr_copywit_enabled() && - !gsp_phys_gr_page_all_zero(csum, src)) { + !gsp_phys_gr_page_all_zero( + gsp_phys_gr_page_sum32(src), src)) { u32 wi; for (wi = 0; wi < gsp_phys_gr_outwit_n; @@ -20888,24 +21887,78 @@ gr_page_unmapped: } if (changed < 8) chg[changed] = bi; + /* + * r425: remember the first engine-changed page + * (post-copy-out content) for the next zero-write + * accept's late-store witness. + */ + if (changed == 0) { + gsp_phys_gr_prev_out_bi = bi; + gsp_phys_gr_prev_out_csum = fp; + gsp_phys_gr_prev_out_valid = true; + } changed++; } + /* + * r378: the target HPA was resolved at submission; + * the poll/retire/retry window is seconds wide and a + * guest buffer freed and re-homed in it would receive + * this full page through the stale target. Re-resolve + * the VA and skip (bounded log) on a positive + * disagreement. + */ + if (!gsp_sema_gr_write_guard(req->data[wbi].va, + req->data[wbi].hpa, + "gr-bounce-copy-out")) { + skipped++; + continue; + } memcpy((u8 *)__va(req->data[wbi].hpa), src, GSP_PHYS_PAGE_SIZE); gsp_phys_gr_page_spot_check( (const u8 *)__va(req->data[wbi].hpa), src, req->data[wbi].va, "gr-bounce-copy-out"); + /* + * r391d post-write witness (sum=0 class): read the VA's + * landing page right after the write through the same + * path the D2H reader uses. If val8 is not first8 the + * write target and the reader's page are different + * (aperture/staleness disagreement), not a lost write. + */ + if (outwit < GSP_PHYS_GR_COPYWIT_MAX) { + u64 rpa2 = 0, rval2 = 0; + u32 rap2 = 0; + + if (gsp_sema_gr_va_snap(req->data[wbi].va, + &rpa2, &rap2, &rval2)) { + outwit++; + pr_info("vgsp-gr: OUT-PAGE-POST va=0x%llx hpa=0x%llx res_pa=0x%llx ap=%u val8=%016llx vbase=%px\n", + req->data[wbi].va, + req->data[wbi].hpa, rpa2, + rap2, rval2, + (void *)gsp_sema_gr_shadow_base()); + } + } wrote++; } if ((u64)req->n_data << 12 >= GSP_PHYS_GR_AUDIT_MIN) pr_info("vgsp: COPYAUDIT path=gr-bounce-copy-out va=0x%llx len=0x%llx done=0x%llx\n", req->data[0].va, (u64)req->n_data << 12, (u64)wrote << 12); - if (wrote < req->n_data) + /* + * r378: the copy-out is now selective (staged + changed + + * re-validated pages only), so completeness is an accounting + * check over the eligible recs: every one is either written, + * skipped as unchanged, skipped as aliased, or refused by the + * write guard (logged as WGUARD-SKIP). This can only fire if + * a rec vanished between the filters. + */ + if (wrote + unchanged + aliased + skipped < eligible) pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-bounce-copy-out\n", req->data[0].va, - (u64)req->n_data << 12, - (u64)wrote << 12); + (u64)eligible << 12, + (u64)(wrote + unchanged + aliased + + skipped) << 12); /* m143b: unconditional first words of the first page of each * 32-page relocation window after the run, so the engine's * read/write footprint per window is provable without relying @@ -20922,8 +21975,7 @@ gr_page_unmapped: if (bi == GSP_PHYS_GR_DATA_BI_NONE) continue; wp = gsp_phys_chunk_page_va(dchunk, bi); - for (k = 0; k < GSP_PHYS_PAGE_SIZE; k += 4) - wc += get_unaligned_le32(wp + k); + (void)k; (void)wc; pr_info("vgsp-gr: window[%u] reloc=0x%llx first=%08x %08x %08x %08x changed=%u\n", bi, req->data[wi].reloc_va, @@ -20931,11 +21983,14 @@ gr_page_unmapped: get_unaligned_le32(wp + 4), get_unaligned_le32(wp + 8), get_unaligned_le32(wp + 12), - wc != bounce_csum_in[bi]); + gsp_phys_gr_fnv1a64( + wp, GSP_PHYS_PAGE_SIZE) != + bounce_csum_in[bi]); } } - pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine (first bi: %u %u %u %u %u %u %u %u)\n", - wrote, changed, chg[0], chg[1], chg[2], chg[3], chg[4], + pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u alias-skipped, %u stale-skipped (first bi: %u %u %u %u %u %u %u %u)\n", + wrote, changed, unchanged, aliased, skipped, + chg[0], chg[1], chg[2], chg[3], chg[4], chg[5], chg[6], chg[7]); } /* @@ -20947,21 +22002,39 @@ gr_page_unmapped: if (gsp_phys_gr_gap_n && gsp_phys_gr_data_chunk()->pages) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); u32 gi, gwrote = 0, gchanged = 0; + u32 gunchanged = 0, gskipped = 0; u32 gchg[4] = { }; for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { u32 bi = gsp_phys_gr_gap[gi].bi; const u8 *src = gsp_phys_chunk_page_va(dchunk, bi); - u32 ci, csum = 0; + u64 fp; if (!src) continue; - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != bounce_csum_in[bi]) { - if (gchanged < 4) - gchg[gchanged] = bi; - gchanged++; + /* + * r378: same staging-identity rule as the tracked + * copy-out above - only the page staged into this + * slot may receive the slot's content. + */ + if (bounce_staged_hpa[bi] != gsp_phys_gr_gap[gi].hpa) { + gskipped++; + continue; + } + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { + /* Engine did not touch it: nothing to ship. */ + gunchanged++; + continue; + } + if (gchanged < 4) + gchg[gchanged] = bi; + gchanged++; + if (!gsp_sema_gr_write_guard(gsp_phys_gr_gap[gi].va, + gsp_phys_gr_gap[gi].hpa, + "gr-gap-copy-out")) { + gskipped++; + continue; } memcpy((u8 *)__va(gsp_phys_gr_gap[gi].hpa), src, GSP_PHYS_PAGE_SIZE); @@ -20970,14 +22043,19 @@ gr_page_unmapped: gsp_phys_gr_gap[gi].va, "gr-gap-copy-out"); gwrote++; } - pr_info("vgsp-gr: gap copy-out: %u pages written back, %u changed by engine (first bi: %u %u %u %u)\n", - gwrote, gchanged, gchg[0], gchg[1], gchg[2], gchg[3]); + pr_info("vgsp-gr: gap copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u stale-skipped (first bi: %u %u %u %u)\n", + gwrote, gchanged, gunchanged, gskipped, + gchg[0], gchg[1], gchg[2], gchg[3]); if ((u64)gsp_phys_gr_gap_n << 12 >= GSP_PHYS_GR_AUDIT_MIN) pr_info("vgsp: COPYAUDIT path=gr-gap-copy-out va=0x%llx len=0x%llx done=0x%llx\n", gsp_phys_gr_gap[0].va, (u64)gsp_phys_gr_gap_n << 12, (u64)gwrote << 12); - if (gwrote < gsp_phys_gr_gap_n) + /* + * r378: same accounting rule as the tracked copy-out above - + * the intentional unchanged/stale skips are not truncation. + */ + if (gwrote + gunchanged + gskipped < gsp_phys_gr_gap_n) pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-gap-copy-out\n", gsp_phys_gr_gap[0].va, (u64)gsp_phys_gr_gap_n << 12, @@ -21013,8 +22091,20 @@ gr_page_unmapped: jiffies_to_msecs(jiffies - start), gsp_phys_gr_ok); /* m128: post-success qmdwin words so a probe that stores a value into * the qmdwin page (e.g. p6's constant-read dump at w48) is readable - * from the success log (the post-mortem only prints on timeout). */ - if (boot_stage.rmcall_gr_qmdwin.pages) { + * from the success log (the post-mortem only prints on timeout). + * r434: read the launch's OWN bank (qmd_dst_mem) when available - + * the old bank-0-only read showed a stale page for odd epochs. */ + if (qmd_dst_mem) { + const u32 *qw = (const u32 *)qmd_dst_mem; + + pr_info("vgsp-gr: post-OK qmdwin w48..59=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + get_unaligned_le32(qw + 48), get_unaligned_le32(qw + 49), + get_unaligned_le32(qw + 50), get_unaligned_le32(qw + 51), + get_unaligned_le32(qw + 52), get_unaligned_le32(qw + 53), + get_unaligned_le32(qw + 54), get_unaligned_le32(qw + 55), + get_unaligned_le32(qw + 56), get_unaligned_le32(qw + 57), + get_unaligned_le32(qw + 58), get_unaligned_le32(qw + 59)); + } else if (boot_stage.rmcall_gr_qmdwin.pages) { const u32 *qw = (const u32 *)gsp_phys_chunk_page_va( &boot_stage.rmcall_gr_qmdwin, 0); @@ -21038,6 +22128,7 @@ out: out->elapsed_ms = jiffies_to_msecs(jiffies - start); out->rc = ret; out->fault = fault; + out->qmd_rel_redirected = qmd_rel_redirected; } return ret; } @@ -22141,7 +23232,7 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) gsp_phys_backend_gr_counters(&b_sub, &b_ok, &b_refused, &b_timeout, &b_pool); seq_printf(m, - "work_gr: req=%llu fallback=%llu ok=%llu refused=%llu timeout=%llu fault=%llu submitted=%llu backend_ok=%llu backend_refused=%llu backend_timeout=%llu pool_exhausted=%llu forward_param=%u exec_ok=%u sem_wedged=%llu stall_wait_max=%llu stall_retry=%llu wait_max_ms=%u\n", + "work_gr: req=%llu fallback=%llu ok=%llu refused=%llu timeout=%llu fault=%llu submitted=%llu backend_ok=%llu backend_refused=%llu backend_timeout=%llu pool_exhausted=%llu forward_param=%u exec_ok=%u sem_wedged=%llu stall_wait_max=%llu stall_retry=%llu wait_max_ms=%u nw_streak=%u wedge_quiet=%llu wedge_heals=%llu\n", gr_req, gr_fallback, gr_ok, gr_refused, gr_timeout, gr_fault, b_sub, b_ok, b_refused, b_timeout, b_pool, gsp_phys_gr_forward_enabled(), @@ -22149,7 +23240,10 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) READ_ONCE(gsp_phys_gr_sem_wedged), READ_ONCE(gsp_phys_gr_stall_wait_max), READ_ONCE(gsp_phys_gr_stall_retry), - gsp_phys_gr_wait_max_ms_value()); + gsp_phys_gr_wait_max_ms_value(), + READ_ONCE(gsp_phys_gr_nw_streak), + READ_ONCE(gsp_phys_gr_wedge_quiet), + READ_ONCE(gsp_phys_gr_wedge_heals)); } { /* @@ -23128,6 +24222,30 @@ static int gsp_phys_reg_probe_show(struct seq_file *m, void *unused) sked_cwd1 = gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_1", 0x00407064u); + /* r425: the wedge diagnostics - the rest of the SKED CWD table, the + * per-8-group GR TPC masks (a zero compute mask would dispatch no + * TPCs), and the FE QMD-virtualization control for VEID 0. */ + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_2", + 0x00407068u); + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_3", + 0x0040706cu); + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_INFO_1", + 0x00407028u); + { + u32 gi; + + for (gi = 0; gi < 8u; gi++) { + gsp_phys_reg_probe_line(m, bar0, + "FE_TPCM_GR_TPC_MASK", + 0x0040a680u + gi * 4u); + gsp_phys_reg_probe_line(m, bar0, + "FE_LAST_GR_TPC_MASK", + 0x0040a6a0u + gi * 4u); + gsp_phys_reg_probe_line(m, bar0, + "FE_CWD_OWN_TPC_MASK", + 0x0040a6c0u + gi * 4u); + } + } fecs_irq = gsp_phys_reg_probe_line(m, bar0, "FECS_IRQSTAT", 0x00409008u); fecs_cpuctl = gsp_phys_reg_probe_line(m, bar0, "FECS_CPUCTL", diff --git a/kernel/gsp_phys_booter.h b/kernel/gsp_phys_booter.h index fb7b2a4..af74f7c 100644 --- a/kernel/gsp_phys_booter.h +++ b/kernel/gsp_phys_booter.h @@ -164,6 +164,16 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, void gsp_phys_backend_static_invalidate(void); /* + * Mode-35 KGR static-data snapshots (see gsp_phys_booter.c): the live GSP's + * replies to the real client's KGR query set, captured at backend boot and + * served byte-identically by the guest-side emulation. Returns 0 with + * *out_len set, -ENODATA when cmd was not captured for the current backend + * generation, -ENOSPC when the snapshot exceeds dst_len, -EAGAIN when the + * backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len); + +/* * Run one fn1+fn65 probe on a READY backend to fill the static snapshot * cache. Takes gsp_phys_rm_lock inside (via gsp_phys_rm_probe), so it must * run on a work item, never inline under a caller-held rm lock or on the mdev @@ -275,6 +285,17 @@ struct gsp_phys_backend_copy_result { u32 elapsed_ms; /* return-to-caller elapsed milliseconds */ int rc; /* duplicated function return code */ struct gsp_phys_fault_info fault; /* decoded engine fault, if any */ + /* + * r420b: the forwarded launch's QMD release0 was redirected to the + * module semaphore (gsp_phys_rmcall_gr_rel_mod=1, or an unresolved + * guest release page), so the ENGINE will never write the guest's own + * QMD release word. The caller must author it in software on every + * outcome (the same write the hold's success edge performs); without + * this a rel_mod=1 launch with no deferred release records is never + * completed on the guest word and the client hangs (the r414 saxpy + * f0 cur=5 stall). + */ + bool qmd_rel_redirected; }; int gsp_phys_backend_guest_ce_copy_ex(u64 src_hpa, u32 src_off, u64 dst_hpa, @@ -421,6 +442,9 @@ struct gsp_phys_gr_submit { u64 rel_hpa; /* guest-visible QMD release0 page, or 0 */ u32 rel_off; u32 rel_payload; + u64 rel_va; /* r378: device VA rel_hpa was resolved from, + * for write-target re-validation before the + * release word is zeroed/restored */ u64 sem_hpa; /* guest-visible channel release page, or 0 */ u32 sem_off; u32 sem_payload; @@ -433,6 +457,16 @@ struct gsp_phys_gr_submit { * whose boot submission proved out (rmcall_gr_exec_ok). */ bool gsp_phys_backend_gr_ready(void); +/* + * r420: true exactly while the GR submit path sleeps in a WEDGE-QUIET hold + * (a deliberate, bounded wait for the sticky wedge to clear). The in-flight + * forwarded request is NOT done during it, so the sema service's completion + * deadlines (the 12 s GR release hold and the 3 s per-launch consumption + * bound) pause while this returns true instead of authoring the launch's + * release word under the hold (the r419 mid-boot stale-data escape). + */ +bool gsp_phys_gr_quiet_hold_active(void); + int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, struct gsp_phys_backend_copy_result *out); diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c index e845493..74062dd 100644 --- a/kernel/gsp_phys_gpu.c +++ b/kernel/gsp_phys_gpu.c @@ -638,13 +638,20 @@ int gsp_phys_rmcall_gr_qmdvirt_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdvirt_value); /* - * Explicit-PCAS gate (default 1). The rebuilt launch carries the vendor's - * turingPushPcas(CU_PCAS_TYPE_COPY) pair (SEND_PCAS_A = QMD>>8 followed by - * SEND_SIGNALING_PCAS_B = INVALIDATE|SCHEDULE). The stock guest push does - * not emit these (it relies on the IQ2M window's implied copy PCAS); setting - * this to 0 emits the pure vendor window-only form for the A/B test. - */ -static int gsp_phys_rmcall_gr_pcas_param = 1; + * Explicit-PCAS gate (default 0 since r343 - the double-execution fix). + * The rebuilt launch used to carry the vendor's turingPushPcas + * (CU_PCAS_TYPE_COPY) pair (SEND_PCAS_A = QMD>>8 followed by + * SEND_SIGNALING_PCAS_B = INVALIDATE|SCHEDULE) ADDED to the inline-QMD run + * that the stock guest push already carries. The QMD's + * REQUIRE_SCHEDULING_PCAS bit is clear, so the inline run schedules by + * itself - the explicit pair was a SECOND scheduling edge and the same QMD + * was dispatched twice (launch_count: one atomicAdd +100 landed as +200; + * victory_reduce exactly 2x; scale/saxpy/memset2d doubled). The stock guest + * push has NO PCAS pair (captured push dumps); default 0 restores that + * single-edge form. 1 = re-add the pair (A/B debug only - with it on, the + * grid executes twice). + */ +static int gsp_phys_rmcall_gr_pcas_param; module_param_named(gsp_phys_rmcall_gr_pcas, gsp_phys_rmcall_gr_pcas_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_pcas, @@ -657,6 +664,29 @@ int gsp_phys_rmcall_gr_pcas_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_pcas_value); /* + * r434 CWD-slot-count path (gsp_phys_rmcall_gr_cwd_slot; default -1 = off). + * On Volta+ the SET_CWD_SLOT_COUNT method is gated by the falcon PRI bit + * NV_PGRAPH_PRI_SKED_DEBUG_1.ENABLE_CWD_SLOT_COUNT (the CUDA driver's + * turingLaunchSetCwdSlotCount: a masked PRI write via SET_MME_SHADOW_SCRATCH + * 0/1/2 + SET_FALCON04, then the method itself). When >= 0 the init preamble + * emits that PRI write (ENABLE_CWD_SLOT_COUNT=1) followed by + * SET_CWD_SLOT_COUNT with the given value, so the wedge's "grid truncated to + * ~4 CTAs" signature can be A/B'd against the CWD slot-count watermark. + * Read per launch; -1 restores the historical preamble byte-for-byte. + */ +static int gsp_phys_rmcall_gr_cwd_slot_param = -1; +module_param_named(gsp_phys_rmcall_gr_cwd_slot, + gsp_phys_rmcall_gr_cwd_slot_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_cwd_slot, + "r434: enable the CWD slot count PRI (SKED_DEBUG_1 bit11) and emit SET_CWD_SLOT_COUNT with this value in the GR init preamble (-1 = off, default)"); + +int gsp_phys_rmcall_gr_cwd_slot_value(void) +{ + return gsp_phys_rmcall_gr_cwd_slot_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_cwd_slot_value); + +/* * Two-step PCAS gate (default 0). With the explicit PCAS pair on, setting * this marks the QMD's REQUIRE_SCHEDULING_PCAS (w4 bit12) and emits the copy * PCAS (INVALIDATE only) followed by a schedule-only PCAS (SCHEDULE), the @@ -704,20 +734,30 @@ int gsp_phys_rmcall_gr_qmdwin_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdwin_value); /* - * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod; default 0). - * When set, a forwarded GR launch ignores the resolved guest release page and - * points QMD release0 at the module window-0 semaphore VA - * (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host VA. - * Discriminant for "engine completed but the shadow release write was lost" - * vs "engine never completed": with the module release, a completed QMD is - * always observable. Also makes the existing module-release fallback - * observable (it previously set no poll source, so it could never complete). - */ -static int gsp_phys_rmcall_gr_rel_mod_param; + * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod). + * + * DEFAULT 1 since r453. When set, a forwarded GR launch ignores the resolved + * guest release page and points QMD release0 at the module window-0 semaphore + * VA (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host + * VA; the GUEST-VISIBLE release word is authored by the host only on the + * completion edge - after the retire wait and after the bounce copy-out has + * shipped the engine's stores to the guest pages. Evidence (r453, the FNV + * change-detection fix on top): + * - with 0: the guest's release could be satisfied by the engine's own + * early write while the host copy-out was still pending, so a client's + * stream-ordered readback (streams2's c2, memcpy_roundtrip's memset + * phase, streams_events' chain) read PRE-launch content: 11/15 battery. + * - with 1: 13/15 battery (streams_events and memcpy_roundtrip flipped + * green), no stalls across the full battery, mail+saxpy clean. + * The r414 stall hole (a launch that never reached the hold edge left the + * guest word unauthored) is closed by the r420b per-outcome authoring path; + * one-line rollback: echo 0 > .../parameters/gsp_phys_rmcall_gr_rel_mod. + */ +static int gsp_phys_rmcall_gr_rel_mod_param = 1; module_param_named(gsp_phys_rmcall_gr_rel_mod, gsp_phys_rmcall_gr_rel_mod_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_rel_mod, - "Force the forwarded GR QMD release0 to the module semaphore (default 0; 1 = m64 engine-completion discriminant)"); + "Force the forwarded GR QMD release0 to the module semaphore; guest release authored only on the post-copy-out completion edge (default 1 since r453; 0 = historical engine-authored guest release)"); int gsp_phys_rmcall_gr_rel_mod_value(void) { @@ -1050,6 +1090,33 @@ int gsp_phys_gr_icache_inv_value(void) EXPORT_SYMBOL_GPL(gsp_phys_gr_icache_inv_value); /* + * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar; default 0). The + * captured guest QMDs carry RELEASE_MEMBAR_TYPE=FE_NONE (QMDV00_06 word 11 + * bit 14 clear, w11=0x44010000), so when the front-end completes a grid it + * writes the QMD release semaphore without first flushing its pending + * system-memory stores. The post-dispatch store-loss signature (the engine + * processes the QMD, writes the release, the grid completes and ZERO data + * pages are host-visible) is exactly what an unflushed write-back L2 looks + * like from the host: the release is host-visible while the data sits in the + * GPU's L2. Forcing RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR makes the release + * ordering guarantee the stores are in memory before the semaphore write, so + * the host's retire scan (and the copy-out) can never observe a completion + * whose data has not landed. 0 keeps the guest's captured value (the + * historical behaviour) for a one-flip A/B; read per launch. + */ +static int gsp_phys_gr_rel_membar_param; +module_param_named(gsp_phys_gr_rel_membar, gsp_phys_gr_rel_membar_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_rel_membar, + "r433: force QMD RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR (word11 bit14) on forwarded GR launches (default 0 = guest's captured FE_NONE)"); + +int gsp_phys_gr_rel_membar_value(void) +{ + return gsp_phys_gr_rel_membar_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_rel_membar_value); + +/* * m179 no-write retry budget (gsp_phys_gr_nowrite_retries; default 3, clamped * to [0, 4]). The m153 outcome retry loop re-executes a data-bearing launch * whose retirement wait saw zero engine writes; it shared the timeout ladder's @@ -1117,6 +1184,91 @@ int gsp_phys_gr_nowrite_retries_value(void) EXPORT_SYMBOL_GPL(gsp_phys_gr_nowrite_retries_value); /* + * r445 wedge hard re-arm (gsp_phys_gr_wedge_rearm; default 0 = off). When the + * outcome-retry's warm GR semaphore probe proves the engine wedged for the + * context (the proven-sticky post-dispatch no-store state), this knob makes + * the submit attempt ONE bounded recovery instead of breaking immediately: + * a TSG PREEMPT(bWait=1) forces the FECS to save/unload the channel's + * context, then the mode-35 lane-enable tail + GPFIFO schedules are re-issued + * (the m155 extended re-arm) and the semaphore probe is repeated. A pass + * means the lane revived for this launch and the normal retry ladder + * continues; a fail leaves the historical WEDGED break. Read once per wedge + * episode. Off by default for A/B. + */ +static int gsp_phys_gr_wedge_rearm_param; +module_param_named(gsp_phys_gr_wedge_rearm, gsp_phys_gr_wedge_rearm_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wedge_rearm, + "r445: on the proven engine-wedged sem probe, attempt a bounded preempt+resched hard re-arm before giving up (default 0=off)"); + +int gsp_phys_gr_wedge_rearm_value(void) +{ + return gsp_phys_gr_wedge_rearm_param ? 1 : 0; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_rearm_value); + +/* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default 0). + * The forwarded QMD's cache-control block (QMDV02_03 word 5) carries the + * per-cache invalidate bits; the captured guest QMDs always set bits 26..29 + * (texture header/sampler/data + shader DATA) and the M91 fix adds bit 30 + * (INSTRUCTION), but bit 31 (SHADER CONSTANT - where a kernel's launch + * parameters live) is never set. The forwarded launch's param page is + * rewritten in place between launches (the two-bank win0 pool), so a stale + * constant-cache line can serve the PREVIOUS launch's parameters - the + * kernel then computes with the previous launch's pointers/sizes and the + * current launch's tracked output pages stay untouched ("no stores"). + * Mode: 0 = off, 1 = set bit 31 only, 2 = force the full 0xff000000 + * invalidate set. Read at submission time, so it can be A/B'd at runtime. + */ +static int gsp_phys_gr_ccache_inv_param; +module_param_named(gsp_phys_gr_ccache_inv, gsp_phys_gr_ccache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_ccache_inv, + "r449: forwarded-QMD shader-constant-cache invalidate: 0=off, 1=bit31, 2=all invalidate bits (default 0)"); + +unsigned int gsp_phys_gr_ccache_inv_value(void) +{ + if (gsp_phys_gr_ccache_inv_param < 0) + return 0; + if (gsp_phys_gr_ccache_inv_param > 2) + return 2; + return (unsigned int)gsp_phys_gr_ccache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_ccache_inv_value); + +/* + * r410 wedge quiet window (gsp_phys_gr_wedge_quiet_ms; default 40000, clamped + * [0,120000]; 0 disables). The sticky per-lane no-execute wedge (100% of + * launches no-write while kept busy) is proven to CLEAR when the lane stays + * quiet for tens of seconds (r409: a ~37 s client gap healed a wedged run; + * r416: 58 consecutive no-write launches during a 126 s client run, first + * launches green again after the lane sat ~52 s idle; r417: a 20 s host hold + * did not heal, a 40 s hold did - "WEDGE-QUIET healed: engine wrote 1 + * page(s) after the 40000 ms hold"). Immediate retries and + * re-arms do not heal it (r409/r154). So when a zero-write outcome follows a + * previous accepted zero-write launch, the retry loop first holds this many ms + * with NO submissions to the engine, then re-arms (lane enable + schedules) + * and re-publishes once. The value doubles per further quiet retry. The + * quiet window is recovery-only: a launch that wrote anything never enters it. + */ +static unsigned int gsp_phys_gr_wedge_quiet_ms_param = 40000u; +module_param_named(gsp_phys_gr_wedge_quiet_ms, + gsp_phys_gr_wedge_quiet_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wedge_quiet_ms, + "Quiet-window ms before retrying a launch on a lane whose previous launch also no-wrote (0=disable, default 40000; r417: a 20 s hold did not heal, 40 s healed)"); + +unsigned int gsp_phys_gr_wedge_quiet_ms_value(void) +{ + unsigned int v = gsp_phys_gr_wedge_quiet_ms_param; + + if (v > 120000u) + v = 120000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_quiet_ms_value); + +/* * m69 window-0 shadow pool (gsp_phys_rmcall_gr_shadow_win0; default 0). When * set, the forwarded GR launch copies the QMD's program/param pages into the * window-0 shadow pool (module pages at VAs 0x108000..0x10f000, PTEs planted diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h index c664e12..36c2220 100644 --- a/kernel/gsp_phys_gpu.h +++ b/kernel/gsp_phys_gpu.h @@ -388,10 +388,22 @@ unsigned int gsp_phys_gr_first_poll_ms_value(void); * (default 1; 0 = pre-m91 QMD words). */ int gsp_phys_gr_icache_inv_value(void); +/* r433: force QMD RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR (QMDV00_06 word11 bit14) + * on forwarded GR launches so the engine flushes pending sysmem stores + * before writing the release semaphore (default 0 = guest's FE_NONE). */ +int gsp_phys_gr_rel_membar_value(void); + +/* r434: enable the CWD slot count PRI (SKED_DEBUG_1 bit11) and emit + * SET_CWD_SLOT_COUNT with this value in the GR init preamble + * (-1 = off, default). */ +int gsp_phys_rmcall_gr_cwd_slot_value(void); + /* m179: dedicated re-execution budget for a data-bearing GR launch whose * retirement wait saw zero engine writes, independent of the timeout ladder's * attempt counter (default 3, clamped [0,4]). */ int gsp_phys_gr_nowrite_retries_value(void); +int gsp_phys_gr_wedge_rearm_value(void); +unsigned int gsp_phys_gr_ccache_inv_value(void); /* STALLFREE: absolute per-launch GR submit wall-clock budget in ms, clamped * [10000,600000]; 0 = only the built-in per-ladder bounds (default 60000). @@ -399,6 +411,16 @@ int gsp_phys_gr_nowrite_retries_value(void); * and the sema service authors the QMD completion release in software. */ unsigned int gsp_phys_gr_wait_max_ms_value(void); +/* r410 wedge quiet window: when a launch's zero-write outcome follows a + * previous accepted zero-write launch (the sticky per-lane no-execute wedge), + * hold the retry for this many ms with NO submissions to the engine before + * re-publishing. The wedge is proven to clear when the lane stays quiet for + * tens of seconds (r409 ~37 s, r416 ~52 s); immediate retries keep it alive. + * Base value for the first quiet retry, doubled per further attempt, clamped + * [0,120000]; 0 disables the quiet path (retries run back-to-back as before). + * Default 20000. */ +unsigned int gsp_phys_gr_wedge_quiet_ms_value(void); + /* m73: use host-visible sysmem chunks for V7 instance/RAMFC (m62 shape). */ int gsp_phys_rmcall_gr_inst_sys_value(void); diff --git a/kernel/gsp_rm_emulation.c b/kernel/gsp_rm_emulation.c index f6ca2c1..90556f2 100644 --- a/kernel/gsp_rm_emulation.c +++ b/kernel/gsp_rm_emulation.c @@ -552,6 +552,106 @@ static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, return -EOVERFLOW; } + /* + * fail4: free-space discipline. The guest's consumption cursor for + * the status queue (msgqRxHeader.readPtr, at stat+rxHdrOff under the + * queue's SWAP_RX layout - same location msgqTxGetFreeSpace polls) + * gates this write. Overwriting unread records mixes entries in the + * guest's RPC stream: under a completion burst the guest then runs + * garbage RM callbacks (rm_run_rc_callback+0x0) and dies. Events are + * wake-ups and drop freely; a reply waits briefly for the ring to + * drain and is dropped (never overwritten) if it stays full. + */ + { + u32 peer_read = 0; + bool have_peer = false; + int w; + + for (w = 0; w < 51; w++) { + u32 used, space, rxoff = 0; + u8 shdr[32]; + + /* + * r332 SWAP_RX cursor fix: the guest's consumption + * cursor for the STATUS queue lives in the COMMAND + * queue's RX header, not in the status queue's own + * rxHdrOff. Under MSGQ_FLAGS_SWAP_RX (both sides set + * it: flags=1 in both headers) the driver's + * msgqRxMarkConsumed writes its status readPtr to + * pOurRxHdr->readPtr, and the driver's pOurRxHdr is + * the COMMAND-queue buffer's RX header (msgq.c + * msgqRxLink swap branch: pReadOutgoing = + * &pOurRxHdr->readPtr with pOurRxHdr from + * msgqTxCreate's pBackingStore = pCommandQueue). The + * status queue's own +rxHdrOff is the mirror we + * publish: OUR command-queue read position + * (written below after every consumed command + * record). Reading that mirror here made the + * free-space math track our own writes: the ring was + * declared "full" with the guest mid-drain, replies + * were written into it anyway, unread records were + * overwritten, and the guest RM read a newer record + * ("Bad sequence number. Expected N got N+63") - the + * first-client cudaMalloc then fails with "device + * busy or unavailable" (the boot lottery) and every + * later RPC fails until a fresh boot. + */ + if (gsp_rm_read_buf(ctx, rm->cmd_off, shdr, sizeof(shdr))) + break; + memcpy(&rxoff, shdr + 24, 4); + if (!rxoff || rxoff >= 0x1000) + break; + if (gsp_rm_read_buf(ctx, rm->cmd_off + rxoff, &peer_read, 4) || + peer_read >= rm->stat_msgcount) + break; /* no usable cursor: blind write */ + have_peer = true; + used = (rm->stat_write + rm->stat_msgcount - + peer_read) % rm->stat_msgcount; + space = rm->stat_msgcount - used - 1u; + if (elem_count <= space) + break; + /* + * Only event wake-ups drop freely; a reply waits + * (bounded) and then proceeds - dropping replies + * breaks the guest's RM init handshake (observed: + * two fn=0x1 replies dropped at boot -> the device + * never comes up). + */ + if (function == GSPRM_RPC_EVENT_POST_EVENT || w == 50) + break; + mutex_unlock(&rm->stat_lock); + msleep(1); + mutex_lock(&rm->stat_lock); + } + if (have_peer) { + u32 used = (rm->stat_write + rm->stat_msgcount - + peer_read) % rm->stat_msgcount; + u32 space = rm->stat_msgcount - used - 1u; + + if (elem_count > space && + function == GSPRM_RPC_EVENT_POST_EVENT) { + rm->events_dropped++; + if (rm->events_dropped <= 8 || + (rm->events_dropped % 1000) == 0) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: status ring full (free=%u need=%u peer=%u write=%u mc=%u); event dropped, total=%llu", + space, elem_count, + peer_read, rm->stat_write, + rm->stat_msgcount, + rm->events_dropped); + kfree(buf); + mutex_unlock(&rm->stat_lock); + return -EAGAIN; + } + if (elem_count > space) + nv_vgpu_warn(ctx->vgpu, + "vgsp-rm: status ring full for reply fn=0x%x (free=%u need=%u peer=%u write=%u mc=%u); writing anyway after wait", + function, space, elem_count, + peer_read, rm->stat_write, + rm->stat_msgcount); + } + } + /* Write all elements to consecutive queue slots. Page-aware: the * queue buffer is non-contiguous, so translate per page. */ slot = rm->stat_write % rm->stat_msgcount; @@ -1069,6 +1169,7 @@ static const char *gsp_rm_rpc_name(u32 function) case GSPRM_RPC_FUNC_GSP_RM_CONTROL: return "GSP_RM_CONTROL"; case GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER: return "UNLOADING_GUEST_DRIVER"; case GSPRM_RPC_FUNC_UPDATE_BAR_PDE: return "UPDATE_BAR_PDE"; + case GSPRM_RPC_FUNC_DUP_OBJECT: return "DUP_OBJECT"; default: return "UNKNOWN"; } } @@ -2356,6 +2457,7 @@ static void gsp_rm_handle_bus_get_info_v2(struct gsp_emu_context *ctx, #define GSPRM_COPY_PDES_MAX_LEVELS 6 static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, + u32 h_object, const u8 *params, u32 params_size) { struct gsp_rm_emu *rm = ctx->rm_emu; @@ -2406,6 +2508,46 @@ static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, "vgsp-rm: pt level[%u] phys=0x%llx size=0x%llx aperture=%u pageShift=%u", i, phys, size, aperture, page_shift); } + + /* + * r496j: this control IS the kernel RM publishing the page-table + * chain of its internal VAS (the level[0..] instances with their + * phys addresses and page shifts). The chain can name the X/GL + * channel's pushbuffer pages and progress semaphore, for which no + * other translation source exists (the fn-54 SET_PAGE_DIRECTORY + * path is external-VAS/UVM only). It is NOT registered directly: + * a blast registration of every client's directory let a stale + * kernel directory answer another client's VA in the walker's + * first-success order (the r459 stale-directory class; observed + * as a CUDA client hanging after DEVICE with r496g). Instead the + * root is offered as a CANDIDATE; the progress service registers + * it only after it has been VERIFIED against a content-validated + * pair of that very client (the ring found by the RAM content + * scan). + */ + if (num_levels >= 1 && rm->pt.levels[0].phys && + rm->pt.levels[0].size >= 16 && + rm->pt.levels[0].size <= 4096 * 8) { + /* + * The level phys values are guest PAs on this platform + * (observed 4+ GiB, above any FB shadow); aperture 3 in the + * message does not match the walker's FB/SYS encoding, so + * classify by address: values inside the emulated VRAM are + * FB, everything else sysmem. The walker also retries the + * other aperture, so a misclassification only costs a walk. + */ + bool vidmem = rm->pt.levels[0].phys < + (u64)ctx->vgpu->bars[1].size; + + gsp_sema_service_vaspace_candidate_add( + h_object, rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), vidmem); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: VAS 0x%x page directory offered as candidate pdb=0x%llx entries=%u (%s; registered only after a content-verified ring walk)", + h_object, (u64)rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), + vidmem ? "vidmem" : "sysmem"); + } } static bool gsp_rm_alloc_interesting(u32 class_id); @@ -2454,6 +2596,19 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, memcpy(&desc.gpfifo_va, params + GSPRM_CHAN_GPFIFO_OFFSET, 8); memcpy(&desc.gpfifo_entries, params + GSPRM_CHAN_GPFIFO_ENTRIES, 4); memcpy(&desc.engine_type, params + GSPRM_CHAN_ENGINE_TYPE, 4); + /* + * The wire carries NV2080_ENGINE_TYPE_*: NULL = 0, GR0 (== GRAPHICS) + * = 1, COPY0 = 9, NVDEC0 = 0x13, ... The progress service's + * convention is "engine_type == 0 means the GR channel" (CUDA's + * kernel-RM GR channels arrive with engineType NULL), so an X/GL 3D + * channel - which carries the real NV2080_ENGINE_TYPE_GRAPHICS = 1 - + * was misclassified as a CE and its push never executed ("Failed to + * initialize DMA" in the X driver's pushbuffer NOP test). Normalize + * GR0..GR7 (1..8) to 0: the emulated device has a single GR and every + * other engine id is >= COPY0 (9), so this cannot capture a CE. + */ + if (desc.engine_type >= 1 && desc.engine_type <= 8) + desc.engine_type = 0; memcpy(&desc.cid, params + GSPRM_CHAN_CID, 4); memcpy(&desc.instance_pa, params + GSPRM_CHAN_INSTANCE_MEM, 8); memcpy(&desc.userd_pa, params + GSPRM_CHAN_USERD_MEM, 8); @@ -2801,6 +2956,115 @@ gsp_rm_handle_promote_fault_method_buffers(struct gsp_emu_context *ctx, } /* + * r519 experiment switch, independent of the P3 whitelist: forward ONLY the + * BINAPI reserved-interface query 0x20810107 to the live GSP session. The P3 + * whitelist may be off (its mixed real/emulated device answers perturb the + * boot); this one command is what the GL surface init needs answered. + */ +static bool gsp_rm_binapi07_fwd_param; +module_param_named(gsp_rm_binapi07_fwd, gsp_rm_binapi07_fwd_param, bool, 0644); +MODULE_PARM_DESC(gsp_rm_binapi07_fwd, + "Forward only the guest GL core's BINAPI 0x20810107 query to the live GSP session (default 0; diagnostic/experiment)"); + +/* + * The live session's oracle client/object for the 0x20810107 experiment, + * created on demand; 0 until the first query builds it. One device per module + * load, so file-statics are enough; reset by unload. + */ +static u32 gsp_rm_binapi07_be_client; +static u32 gsp_rm_binapi07_be_object; + +/* + * One GSP_RM_ALLOC (fn 103) against the live backend, built exactly like the + * rm-call ladder's helper (rpc_gsp_rm_alloc_v03_00: hClient, hParent, + * hObject, hClass, status, paramsSize, flags, reserved[4] + params). + * object_is_out selects the reply read: a ROOT alloc returns the + * server-chosen client handle in params[0] (payload +32), anything else + * echoes the client-chosen hObject at +8; status is at +16 in both cases. + */ +static int gsp_rm_binapi07_backend_alloc(u32 h_client, u32 h_parent, + u32 h_object, u32 h_class, + const void *params, u32 params_len, + bool object_is_out, + u32 *out_object, u32 *out_status) +{ + u8 *areq; + void *arsp = NULL; + u32 arsp_len = 0, st = 0xffffffff, ho = 0; + int ret; + + areq = kzalloc(32 + params_len, GFP_KERNEL); + if (!areq) + return -ENOMEM; + memcpy(areq + 0, &h_client, 4); + memcpy(areq + 4, &h_parent, 4); + memcpy(areq + 8, &h_object, 4); + memcpy(areq + 12, &h_class, 4); + memcpy(areq + 20, ¶ms_len, 4); + if (params_len) + memcpy(areq + 32, params, params_len); + + ret = gsp_phys_backend_rpc(GSPRM_RPC_FUNC_GSP_RM_ALLOC, areq, + 32 + params_len, &arsp, &arsp_len, 500); + kfree(areq); + if (!ret && arsp) { + if (arsp_len >= 20) + memcpy(&st, (u8 *)arsp + 16, 4); + if (object_is_out) { + if (arsp_len >= 36) + memcpy(&ho, (u8 *)arsp + 32, 4); + } else if (arsp_len >= 12) { + memcpy(&ho, (u8 *)arsp + 8, 4); + } + } + kfree(arsp); + if (out_object) + *out_object = ho; + if (out_status) + *out_status = st; + return ret; +} + +/* + * Build the oracle object on the live session: a class-0x2081 child of the + * session's internal subdevice, since the server dispatches the method from + * the OBJECT's class (the subdevice attempt alone answered + * NV_ERR_NOT_SUPPORTED, 0x56, and a fresh root client at runtime answered + * NV_ERR_INVALID_CLIENT, 0x23 - the ladder "adopts" the server's own + * internal client instead of creating one). A small candidate list of + * likely-free handles in that client's namespace is tried; the first + * success wins. Returns 0 with *out_object set. + */ +static int gsp_rm_binapi07_backend_setup(struct gsp_emu_context *ctx, + struct gsp_phys_backend_static_config *snap, + u32 *out_object) +{ + static const u32 cand[] = { 0xabcd2081u, 0xabcd2001u, 0xabcd2a2au, + 0xabcd5f5fu, 0xabcd1234u }; + u8 p4[4] = { 0, 0, 0, 0 }; + u32 st = 0xffffffff, obj = 0; + int i, ret; + + for (i = 0; i < (int)ARRAY_SIZE(cand); i++) { + u32 want = cand[i]; + + obj = want; + ret = gsp_rm_binapi07_backend_alloc(snap->h_internal_client, + snap->h_internal_subdevice, + want, 0x2081, p4, 4, + false, &obj, &st); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT oracle BINAPI alloc attempt %d: hObject=0x%x ret=%d status=0x%x", + i, want, ret, st); + if (!ret && st == 0 && obj) { + *out_object = obj; + return 0; + } + } + return -EIO; +} + +/* * Bounded logging for the P3 forward counters: the first four fallbacks (so a * session that cannot forward is visible) and every 64th event. Never called * when the parameter is off. @@ -2849,12 +3113,37 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, u32 out_params_size; int ret; + bool guest_handles = false; + /* Wave 1 whitelist: compile-time side-effect-free queries only. */ switch (req->cmd) { case TU104_CTRL_BUS_GET_PCI_BAR_INFO: case TU104_CTRL_BUS_GET_INFO_V2: case TU104_CTRL_FB_GET_INFO_V2: case TU104_CTRL_INTERNAL_GMMU_GET_STATIC_INFO: + if (!gsp_phys_ctrl_forward_enabled()) + return false; + break; + case 0x20810107: /* NV2081_BINAPI reserved-interface method 7 */ + /* + * EXPERIMENT (r519): the guest GL core allocates a BINAPI + * object (class 0x2081) and sends this 2-byte query during + * 3D-surface init; our {0,0} generic reply leaves the + * driver's feature flag clear and the EGL surface init tears + * down (EGL_BAD_ALLOC at eglMakeCurrent). The guest-side + * handles (hClient 0xc1d..., a 0xbeef object) have no backend + * counterpart: allocate a real BINAPI object (class 0x2081) on + * the live session once, then send the query to it. The first + * attempt (subdevice translation) came back NV_ERR_NOT_SUPPORTED + * (0x56) - the server dispatches the method by the OBJECT's + * class, so a subdevice cannot answer a BINAPI method. The + * reply is logged either way; an error falls back to the + * emulated generic success until the semantics are known. + * Gated by its own parameter so the P3 whitelist can stay off. + */ + if (!gsp_rm_binapi07_fwd_param) + return false; + guest_handles = true; break; default: return false; @@ -2866,24 +3155,44 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, if (req->paramsSize > GSPRM_MAX_SINGLE_ELEM_DATA - GSP_RM_CTRL_HDR_SIZE) goto fallback; + /* The experiment is scoped to the observed shape: 2-byte in/out. */ + if (guest_handles && req->paramsSize != 2) + goto fallback; + /* A valid snapshot implies a READY session and the adopted handles. */ if (gsp_phys_backend_static_get(&snap)) goto fallback; + if (guest_handles && !gsp_rm_binapi07_be_object) { + u32 obj = 0; + + if (gsp_rm_binapi07_backend_setup(ctx, &snap, &obj)) + goto fallback; + /* + * The object lives in the internal client's namespace; the + * control is sent as that client (snap.h_internal_client). + */ + gsp_rm_binapi07_be_client = snap.h_internal_client; + gsp_rm_binapi07_be_object = obj; + } + /* Translate the guest's emulated internal handles; any other handle * domain stays emulated. */ - if (req->hClient != TU104_H_INTERNAL_CLIENT || - (req->hObject != TU104_H_INTERNAL_DEVICE && - req->hObject != TU104_H_INTERNAL_SUBDEVICE)) + if (!guest_handles && + (req->hClient != TU104_H_INTERNAL_CLIENT || + (req->hObject != TU104_H_INTERNAL_DEVICE && + req->hObject != TU104_H_INTERNAL_SUBDEVICE))) goto fallback; fwd_req = kmalloc(GSP_RM_CTRL_HDR_SIZE + req->paramsSize, GFP_KERNEL); if (!fwd_req) goto fallback; memset(&fwd_hdr, 0, sizeof(fwd_hdr)); - fwd_hdr.hClient = snap.h_internal_client; - fwd_hdr.hObject = req->hObject == TU104_H_INTERNAL_DEVICE ? - snap.h_internal_device : snap.h_internal_subdevice; + fwd_hdr.hClient = guest_handles ? gsp_rm_binapi07_be_client : + snap.h_internal_client; + fwd_hdr.hObject = guest_handles ? gsp_rm_binapi07_be_object : + (req->hObject == TU104_H_INTERNAL_DEVICE ? + snap.h_internal_device : snap.h_internal_subdevice); fwd_hdr.cmd = req->cmd; fwd_hdr.paramsSize = req->paramsSize; fwd_hdr.flags = req->flags; @@ -2903,6 +3212,27 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, } /* + * r519 experiment evidence: log the live GSP's own answer to the + * BINAPI query (status + first params bytes) before deciding. + */ + if (req->cmd == 0x20810107) { + u32 st = 0xffffffff, ps = 0xffffffff; + u8 b[8] = { 0 }; + + if (rsp && rsp_len >= GSP_RM_CTRL_HDR_SIZE) { + memcpy(&st, (u8 *)rsp + 12, 4); + memcpy(&ps, (u8 *)rsp + 16, 4); + if (ps <= 8 && ps <= rsp_len - GSP_RM_CTRL_HDR_SIZE) + memcpy(b, (u8 *)rsp + GSP_RM_CTRL_HDR_SIZE, + ps); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT 0x20810107 forwarded to live GSP: rsp_len=%u status=0x%x outSize=%u bytes=%02x %02x %02x %02x %02x %02x %02x %02x", + rsp_len, st, ps, b[0], b[1], b[2], b[3], b[4], + b[5], b[6], b[7]); + } + + /* * The reply is rpc_gsp_rm_control_v03_00: status at +12, paramsSize * at +16, echoed params at +24 - the same 24-byte prefix the emulated * handler builds below. The RPC-level rpc_result was already checked @@ -2948,6 +3278,101 @@ fallback: return false; } +/* + * r489: a GSP_RM_CONTROL reply whose rpc record exceeds the driver's + * pRpc->maxRpcSize must be sent with the GSP's continuation-record protocol + * (nv gsp_rpc.c _rpcWriteContinuationEntries): the FIRST record carries + * exactly maxRpcSize bytes of [rpc header + ctrl header + params]; each + * following NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD record carries up to + * maxRpcSize - sizeof(hdr) bytes of the remainder. The driver's + * _issueRpcLarge consumes the first frame as min(bufSize, maxRpcSize) and + * then requires one continuation record per remaining chunk - without them + * it times out ("rpcRecvPoll timedout for fn 71 continuation record") and + * the control fails (NV_ERR_INVALID_PARAMETER), which is how the GL VASpace + * construction died on cmd 0x2080a0a4 (paramsSize 67392 > maxRpcSize). + * + * maxRpcSize is the guest driver's GSP_MSG_QUEUE_RPC_SIZE_MAX; the value is + * pinned by the driver's own arithmetic: for the 0x2080a0a4 reply + * (full_param_size = 32 + 24 + 67392 = 67448) it reported remainingSize + * 0x940 = 2368 after the first frame, i.e. maxRpcSize = 67448 - 2368 = + * 65080. A build whose constant differs shows up immediately as a + * mismatched continuation count in the guest log; adjust here. + */ +#define GSPRM_CRM_CONT_MAXRPC 65080u + +static int gsp_rm_send_rm_control(struct gsp_emu_context *ctx, u32 sequence, + const void *data, u32 data_len) +{ + u32 total = sizeof(struct gsprm_rpc_header) + data_len; + const u8 *p = data; + u32 off = 0; + int ret; + + if (total <= GSPRM_CRM_CONT_MAXRPC) + return gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, + sequence, 0, data, data_len); + + /* Frame 1: rpc.length = maxRpcSize. */ + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, + p, GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)); + if (ret) + return ret; + off = GSPRM_CRM_CONT_MAXRPC - sizeof(struct gsprm_rpc_header); + + /* Continuations: rpc.length = hdr + chunk. */ + while (off < data_len) { + u32 chunk = data_len - off; + + if (chunk > GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)) + chunk = GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header); + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_CONTINUATION_RECORD, + sequence, 0, p + off, chunk); + if (ret) + return ret; + off += chunk; + } + return 0; +} + +/* + * Real-GSP KGR static-data snapshots (mode-35 boot capture; see + * gsp_phys_kgr_snap_get in gsp_phys_booter.c). The physical backend asks the + * live GSP the real client's KGR query set at boot and retains the replies; + * serve the identical bytes for the same commands so the guest's GL/3D stack + * reads the device's true GR static values. Concrete consumer: the X + * driver's 3D-engine selection maps GR_GET_INFO index SM_VERSION (0x0c) + * through a table and fails with "Failed to select a 3D engine" when it + * reads back as SM_VERSION_NONE (0); the live GSP's KGR_GET_INFO reply + * carries the real value (0x75 on TU104). Returns true when the reply was + * served from the snapshot (out_params_size set); false falls through to the + * emulated handler unchanged. + */ +static bool gsp_rm_kgr_snap_serve(struct gsp_emu_context *ctx, u32 cmd, + u32 req_params_size, + u8 *out_params, u32 out_len, + u32 *out_params_size) +{ + u32 n = 0; + int ret; + + ret = gsp_phys_kgr_snap_get(cmd, out_params, out_len, &n); + if (ret || !n) + return false; + + /* Never answer with more bytes than the guest asked for. */ + if (n > req_params_size) + return false; + + *out_params_size = n; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: KGR cmd=0x%x served from the real-GSP snapshot (%u bytes)", + cmd, n); + return true; +} + static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, const u8 *elem, u32 sequence) { @@ -2982,13 +3407,26 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, * return means nothing was sent and the emulated switch below runs * byte-identically to the parameter-off path. */ - if (gsp_phys_ctrl_forward_enabled() && + if ((gsp_phys_ctrl_forward_enabled() || gsp_rm_binapi07_fwd_param) && gsp_rm_ctrl_forward(ctx, &req, req_data + GSP_RM_CTRL_HDR_SIZE, reply, reply_data_len, sequence)) { kfree(reply); return; } + /* + * KGR static-data snapshots (mode-35 boot capture): the live GSP's own + * replies to the real client's KGR query set are served byte-identically + * for the same commands, so the guest's GL/3D stack sees the device's + * true GR static values. Falls through to the emulated switch when no + * snapshot exists (backend not READY, other boot mode, cmd not captured). + */ + if (gsp_rm_kgr_snap_serve(ctx, req.cmd, req.paramsSize, + reply + GSP_RM_CTRL_HDR_SIZE, + reply_data_len - GSP_RM_CTRL_HDR_SIZE, + &out_params_size)) + goto rm_control_reply; + switch (req.cmd) { case TU104_CTRL_FIFO_GET_DEVICE_INFO_TABLE: gsp_rm_handle_fifo_device_table(ctx, sequence, &req, @@ -2998,7 +3436,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; case 0x90f10106: /* NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES */ - gsp_rm_handle_copy_server_pdes(ctx, req_data + GSP_RM_CTRL_HDR_SIZE, + gsp_rm_handle_copy_server_pdes(ctx, req.hObject, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize); out_status = 0; out_params_size = 0; @@ -3309,6 +3748,27 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + /* + * r487/r489 THE GL/VULKAN VASPACE WALL (Xorg/EGL/GZDOOM, every + * session to date): the reply must follow the GSP's own large-reply + * framing. r487 clamped the echo to fit one 64 KB record; the guest + * then refused the record ("Incorrect message length 65528") and, + * when it fit, waited for CONTINUATION records that never came + * ("rpcRecvPoll timedout for fn 71 continuation record + * (remainingSize=0x940)") - rpc.c's _issueRpcLarge consumes the + * first frame up to pRpc->maxRpcSize (65080) and then requires + * NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD frames for the rest + * (the GSP's own _rpcWriteContinuationEntries). The size is left + * unclamped here; gsp_rm_send_rm_control() frames it per that + * protocol. Root evidence: cmd 0x2080a0a4 (PERF_VPSTATES_INFO, + * paramsSize 67392, FINN-serialized) sent during Xorg VASpace + * construction; its failure made vmmCreateVaspace return ffff and + * killed GL - and, after the length was fixed, the 3D-engine + * selection ("Failed to select a 3D engine") until the framing was + * correct. + */ + +rm_control_reply: /* Build the reply header */ memcpy(reply + 0, &req.hClient, 4); memcpy(reply + 4, &req.hObject, 4); @@ -3328,8 +3788,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, out_status, dt_us); } - if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, - reply, GSP_RM_CTRL_HDR_SIZE + out_params_size)) + if (gsp_rm_send_rm_control(ctx, sequence, reply, + GSP_RM_CTRL_HDR_SIZE + out_params_size)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: RM_CONTROL reply send failed cmd=0x%x", req.cmd); else @@ -3339,6 +3799,61 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, } /* + * r510gl: DUP_OBJECT (function 21) - NVOS55_PARAMETERS echo with status=NV_OK. + * + * The guest driver dups an object into a second client at EGL/GLX + * MakeCurrent time (memory and VASpace objects: the GL context's client + * shares the device's allocations), and the historical generic-SUCCESS + * default replied with an EMPTY payload - the driver's reply must carry the + * params struct back (rpc_dup_object_v03_00 = NVOS55_PARAMETERS + * {hClient, hParent, hObject, hClientSrc, hObjectSrc, flags, status}), so + * an empty reply failed the call and the driver tore the whole EGL session + * down (guest witness: 'RPC UNKNOWN(21) seq=0 len=60 -> generic SUCCESS', + * immediately followed by the session's FREE cascade and + * eglMakeCurrent -> EGL_BAD_ALLOC (0x3003), 2026-10-23 r510). + * + * The emulation does not model per-object state for these classes (controls + * and allocs are answered from the request), so the echo with NV_OK is the + * complete answer; the dup'd handle is used by later calls exactly like the + * source handle. + */ +static void gsp_rm_handle_dup_object(struct gsp_emu_context *ctx, + const u8 *elem, u32 sequence) +{ + struct gsp_rm_emu *rm = ctx->rm_emu; + const struct gsprm_rpc_header *hdr = + (const struct gsprm_rpc_header *)(elem + GSPRM_ELEM_HDR_SIZE); + const u8 *req = elem + GSPRM_ELEM_HDR_SIZE + sizeof(*hdr); + u8 params[28]; /* NVOS55_PARAMETERS_v03_00 */ + u32 hDstClient = 0, hDstObject = 0, hSrcClient = 0, hSrcObject = 0; + + if (hdr->length >= sizeof(*hdr) + sizeof(params)) + memcpy(params, req, sizeof(params)); + else + memset(params, 0, sizeof(params)); + + memcpy(&hDstClient, params + 0, 4); + memcpy(&hDstObject, params + 8, 4); + memcpy(&hSrcClient, params + 12, 4); + memcpy(&hSrcObject, params + 16, 4); + memset(params + 24, 0, 4); /* status = NV_OK */ + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DUP_OBJECT hDstClient=0x%x hDstObject=0x%x hSrcClient=0x%x hSrcObject=0x%x seq=%u -> NV_OK echo", + hDstClient, hDstObject, hSrcClient, hSrcObject, sequence); + + /* Small delay before the reply: same reply-matching race mitigation as + * FREE/UNLOADING. */ + udelay(2000); + + if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_DUP_OBJECT, sequence, 0, + params, sizeof(params))) + nv_vgpu_err(ctx->vgpu, "vgsp-rm: DUP_OBJECT reply send failed"); + else + rm->rpcs_replied++; +} + +/* * GSP_RM_FREE (function 10) - NVOS00_PARAMETERS echo with status=NV_OK. * * rpc_gsp_free: { hRoot, hObjectParent, hObjectOld, status }. Echo the @@ -3585,14 +4100,18 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) return; case GSPRM_RPC_FUNC_CONTINUATION_RECORD: - /* CONTINUATION_RECORD: driver expects a reply. Send SUCCESS. */ - nv_vgpu_info(ctx->vgpu, "vgsp-rm: async RPC %s(%u) seq=%u len=%u -> SUCCESS reply", - gsp_rm_rpc_name(function), function, sequence, - hdr->length); - if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) - nv_vgpu_err(ctx->vgpu, "vgsp-rm: reply send failed fn=%u", function); - else - rm->rpcs_replied++; + /* + * r489: a CONTINUATION_RECORD from the driver is ONE-WAY data + * - the tail of a multi-frame request (its _issueRpcLarge + * sends frame 1 then continuation records). The old reply + * here was a guess ("driver expects a reply"); it injected an + * extra fn-71 record into the reply stream that the driver + * drained as an event ("Unexpected RPC event from GPU0: 0x47") + * and it corrupted the framing of our own large replies + * (which use fn-71 continuations the other way). Consume + * silently: the driver waits only for the reply to frame 1. + */ + rm->rpcs_replied++; return; case GSPRM_RPC_FUNC_SET_GUEST_SYSTEM_INFO: @@ -3615,6 +4134,10 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) gsp_rm_handle_free(ctx, elem, sequence); return; + case GSPRM_RPC_FUNC_DUP_OBJECT: + gsp_rm_handle_dup_object(ctx, elem, sequence); + return; + case GSPRM_RPC_FUNC_SET_PAGE_DIRECTORY: gsp_rm_handle_set_page_directory(ctx, elem, sequence, hdr->length); return; diff --git a/kernel/gsp_rm_emulation.h b/kernel/gsp_rm_emulation.h index 96ea16a..45fcc64 100644 --- a/kernel/gsp_rm_emulation.h +++ b/kernel/gsp_rm_emulation.h @@ -45,6 +45,7 @@ struct gsp_emu_context; #define GSPRM_RPC_FUNC_GSP_RM_CONTROL 76 #define GSPRM_RPC_FUNC_GSP_RM_ALLOC 103 #define GSPRM_RPC_FUNC_FREE 10 +#define GSPRM_RPC_FUNC_DUP_OBJECT 21 #define GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER 47 #define GSPRM_RPC_FUNC_UPDATE_BAR_PDE 70 #define GSPRM_RPC_EVENT_POST_EVENT 0x1003 diff --git a/kernel/gsp_rm_tu104.h b/kernel/gsp_rm_tu104.h index b5ee4d7..3382939 100644 --- a/kernel/gsp_rm_tu104.h +++ b/kernel/gsp_rm_tu104.h @@ -754,7 +754,18 @@ static const struct tu104_internal_device_info tu104_internal_device_table[] = { TU104_CE_CAPS0_SYSMEM | \ TU104_CE_CAPS0_P2P | \ TU104_CE_CAPS0_BL_GT_64K) -#define TU104_CE_CAPS1 0x00 +/* + * Caps byte 1: the 2D/3D copy path requires a CE that supports the + * blocklinear access modes (hardware bug 200288104): the X driver's + * nvCEChooseEngine skips every CE without CE_SUPPORTS_NONPIPELINED_BL + * (byte1 0x01; and PIPELINED_BL 0x02 for the preferred pass) and then + * fails with "Failed to find a suitable copy object" - the observed wall + * after the DMA pushbuffer test started passing. Turing's CEs are fully + * capable (pipelined + non-pipelined BL + pitchlinear): byte1 = 0x03. + * (Reference: ctrl2080ce.h NV2080_CTRL_CE_CAPS_CE_SUPPORTS_*_BL 1:0x01/ + * 1:0x02.) + */ +#define TU104_CE_CAPS1 0x03 /* ENGINE_TYPE_COPY0 = 0x9, COPY(i) = COPY0 + i (cl2080_notification.h) */ #define TU104_ENGINE_TYPE_COPY0 0x00000009 diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index 487bb4d..42e8d76 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -160,6 +160,15 @@ struct sema_map_entry { #define SVC_USERD_GP_GET 0x88 #define SVC_USERD_GP_PUT 0x8c +/* + * r496b: guest-RAM ring content scan (see gr_ram_cursor in struct + * svc_channel) and the VRAM USERD shape scan (svc_chan_userd). One scan + * slice per service pass so the service thread stays responsive. + */ +#define SVC_GR_RAM_SLICE (512ULL << 20) /* guest RAM per pass */ +#define SVC_GR_RAM_PASS_MAX 16 /* slices before giving up */ +#define SVC_GR_RAM_WIN (2ULL << 20) /* allocation window pinned */ + struct svc_channel { bool valid; u32 session_gen; /* m176: session generation this entry was @@ -260,6 +269,10 @@ struct svc_channel { * window (observed: one 8 s grace per push, * serializing the runtime init) */ DECLARE_BITMAP(slot_done_epoch, SVC_GPFIFO_MAX_BITS); + u8 slot_pend_tries[SVC_GPFIFO_MAX_BITS]; /* fail4 r308: bounded + * re-holds of a denied device-dst copy slot */ + u32 counter_gen; /* r311: session generation that authored the + * completion word (guards the residue zero) */ /* GPFIFO slots already consumed in the current * put epoch (the client's put value since it * last wrapped). A pin-batch rewind replays @@ -330,6 +343,10 @@ struct svc_channel { * publish targets counter_pa instead, so both * families of the counter page get written * (SA-COUNTER FORENSICS-grv7-counter F1) */ + u64 release_va; /* r378: the release VA release_pa was learned + * from, kept so a late write (stale-recover + * zero) can re-validate the target through + * gsp_sema_write_guard before it lands */ u32 counter_last; /* last value published to counter_pa */ u64 pushpg_va; /* last push page logged as pin-vs-window * divergent (rate limit: one line per page) */ @@ -377,6 +394,16 @@ struct svc_channel { * even if still unresolved - bounded zeros, never * a hang); m156: re-held while the window is * live so a busy forward gets its real retry */ + /* + * r509: one-shot verdict of the absolute bound. The expiry call + * pushes the deadline (so the hold window restarts) and latches this + * flag; the r351 retry-forward block consumes it within the same + * pass. Without the latch the two call sites in one pass could not + * both observe the expiry (the first push of the deadline made the + * second call see a fresh window - r213-r215's arming bug in + * reverse). + */ + bool gr_hold_bound_hit; unsigned long gr_hold_deadline; /* GRFIX FIX1b: hard expiry of the data- * stalled hold for gr_hold_dead_va. * Fixed once on the first hold of that @@ -394,6 +421,11 @@ struct svc_channel { u64 gr_hold_dead_va; /* launch push VA the bound belongs to */ unsigned long gr_hold_dead_since; /* first-hold stamp (waited_ms) */ bool gr_hold_exp_logged; /* one GRFILL-EXPIRED line per launch */ + u64 opdef_hold_dead_va; /* r361: launch push VA the opdef bound + * belongs to (slot-keyed `since` is + * defeated by rewind replays) */ + unsigned long opdef_hold_deadline; /* absolute opdef hold expiry */ + bool opdef_hold_exp_logged; /* one OPDEF-EXPIRED line per launch */ u32 gr_rehold_idx; /* m156: slot whose next decode must force a * fresh launch capture (a held entry is * re-decoded as a replay, which suppresses @@ -407,6 +439,49 @@ struct svc_channel { u64 gr_nocap_va; /* push VA the window was opened for */ unsigned long gr_nocap_since; /* absolute first-seen stamp */ /* + * r496b (X/GL kernel-RM sysmem channel): ring discovery by content in + * guest RAM. The CPU RM maps sysmem objects locally (no + * SET_PAGE_DIRECTORY, no rm_mem record carrying the ring VA), so + * svc_lookup_trusted cannot name the ring page; the driver's kickoff + * GPFIFO entries {pushbuffer VA, len} identify it instead. The scan + * is sliced (SVC_GR_RAM_SLICE per pass) and cursor-resumed so the + * service thread stays responsive. + */ + u64 gr_ram_cursor; /* resume point of the content scan */ + u8 gr_ram_passes; /* slices walked so far */ + bool gr_ram_logged; /* one discovery line per channel */ + /* + * r496e: true when the ring was found by the RAM content scan (a + * kernel-RM sysmem channel - the X/GL 3D channel). Such a channel's + * pushes are 2D/3D method streams the emulation does not execute; + * the correct model is consume-and-apply-releases: no QMD capture, + * no launch forward, no no-capture hold. Also allows an all-zero + * NOP push (the driver's pushbuffer sanity test writes exactly one + * zero dword) to be consumed as the no-op it is. + */ + bool gr_ram_ring; + /* + * r496b: the FB-scan USERD (see svc_userd_fb_scan). Latched when the + * registered/ring-derived candidates all fail; the scan validates the + * hit against the ring entries the put claims, so a foreign channel's + * USERD cannot be adopted. Bounded retries: the driver kicks off a + * few passes after channel registration, so a scan before the first + * put must not retire the discovery. + */ + u8 userd_scans; + /* + * r471: the completion-counter publish must never advance the + * guest's word past a push whose content was still unwritten when + * the entry was consumed (the streams_events staging-reuse + * corruption: the publish satisfied the client's D2H wait before + * its copy ran, and the readback drained a staging buffer that + * still held the PREVIOUS copy's bytes). Lowest ring slot consumed + * with all-zero content; 0 = none. Cleared when a later decode of + * a GR push reads non-zero content (the content landed - the rewinds + * re-read entries constantly, so this heals within a pass or two). + */ + u32 gr_bad_put; + /* * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. A * channel that cannot make forward progress (last_get pinned behind * put, or a residual session put that never drops) is detected here @@ -417,19 +492,7 @@ struct svc_channel { u64 wd_put; /* put at the window start */ u64 wd_sub; /* submissions at the window start */ unsigned long wd_since; /* window start (0 = no window) */ - bool wd_epoch_done; /* the stale-epoch reset ran for this put */ - u32 wd_nocap_put; /* force-replay (no capture) for slots below - * this while the stale range is re-walked; 0 = - * none */ - bool wd_chanswitch; /* STALLFREE: registered; first pump pass - * must reconcile the binding before the walk - * (drop dead opdef slots whose since predates - * the binding; log the transition) */ - unsigned long wd_newbind_until; /* STALLFREE: freshness window of a - * new binding; inside it an - * unmapped/undecoded GR entry is held - * (not silently consumed) and an - * immediate rebuild is requested */ + unsigned long wd_rewind_off_until; /* pin-batch rewind cooldown */ }; /* @@ -485,6 +548,12 @@ struct svc_vaspace { u64 pdb; u32 entries; /* root entry count */ int ap; /* SVC_APERTURE_* of the directory itself */ + unsigned long stamp; /* r459: last registration/use; LRU evict + * when the table is full (a stale entry from + * a dead client otherwise blocks every new + * client after the 8th registration: the + * mid-boot walk failure that made the 10th+ + * client's copies fail rc=-2) */ }; struct svc_lat { @@ -814,6 +883,20 @@ struct sema_svc { u64 push_pat_log; /* push-content pattern budget */ u64 dma_skip_log; /* unrecognized-DMA budget */ u64 xfer_raw_log; /* operandless transfer raw-dump budget */ + u64 fill_ram_log; /* r361 physical-RAM fill build budget */ + u64 pushtrace_all; /* fail4: de-windowed push decode + * trace (every push, bounded) */ + u64 opadd_log; /* fail4: every op appended */ + u64 op2d_raw_log; /* r458: raw 2D-shaped copy pushes */ + u64 dmatrace_log; /* fail4: denied-transfer trace */ + u64 pend_dbg_log; /* fail4 r305: pending/hold witness */ + u64 snap_gate_log; /* fail4 r311: capture-gate witness */ + u64 copyent_log; /* fail4: svc_copy entry trace */ + u64 bigrun_log; /* fail4: big method-data run headers */ + u64 lid_hdr_log; /* fail4: LOAD_INLINE_DATA headers */ + u64 skiprun_log; /* fail4: silently skipped data runs */ + u64 mystery_log; /* fail4: moderate op=1 runs on big pushes */ + u64 next_hdr_log; /* fail4: header after each big LID run */ u64 grce_log; /* GRCE inline-write budget */ u32 cur_ch; /* channel of the op being run */ u64 entry_log; /* per-push summary budget */ @@ -1019,19 +1102,6 @@ struct sema_svc { * stale-replay wedge watchdog (zero on a green run). */ u64 gr_stale_wedge; - /* - * STALLFREE r229/r230: set by the pump when a new binding's GR entry - * is unmapped/empty; the service loop consumes it by running - * svc_rebuild() before the next pump pass, so the page translations - * land within ~1 s and the pending entry re-evaluates. - */ - bool rebuild_now; - u32 gr_pending_map; /* GR-STALL-pending-map lines - * emitted (bounded proof) */ - u32 gr_newbind_log; /* GR-STALL-newbind window-armed - * lines emitted (bounded proof) */ - u32 gr_rebuild_log; /* GR-STALL-rebuild consumption - * lines emitted (bounded proof) */ u64 fwd_gr_hold_exp; /* GRFIX FIX1: data-stalled holds * that reached the hard bound and * were consumed with the software @@ -1293,7 +1363,10 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) int k; u64 keep_counter_pa = 0; u64 keep_release_pa = 0; + u64 keep_release_va = 0; u8 keep_gr_index = 0, keep_ce_index = 0; + bool keep_pub_armed = false; + u32 keep_counter_gen = 0; DECLARE_BITMAP(keep_rel_done, SVC_GPFIFO_MAX_BITS); bool same = ch->valid && ch->vgpu == desc->vgpu && ch->h_object == desc->h_object; @@ -1316,8 +1389,21 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) if (same_gen) { keep_counter_pa = ch->counter_pa; keep_release_pa = ch->release_pa; + keep_release_va = ch->release_va; keep_gr_index = ch->gr_index; keep_ce_index = ch->ce_index; + /* + * r311: a same-session re-registration must not + * forget that this session already proved itself + * (gr_pub_armed) - losing it re-arms the residual + * watchdog, and its recovery zeroes a completion + * word this live session authored, which the + * client's UVM reads as a downward jump and never + * completes on (r311 vector_add: pub-adv/zero + * 3 s cycles with the client polling forever). + */ + keep_pub_armed = ch->gr_pub_armed; + keep_counter_gen = ch->counter_gen; bitmap_copy(keep_rel_done, ch->rel_done, SVC_GPFIFO_MAX_BITS); } @@ -1326,24 +1412,17 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) ch->gr_hold_idx = ~0u; /* m156: no hold armed yet */ ch->gr_hold_dead_va = 0; /* GRFIX: no launch bound armed */ ch->gr_hold_exp_logged = false; + ch->opdef_hold_dead_va = 0; /* r361: no opdef bound armed */ + ch->opdef_hold_exp_logged = false; ch->gr_rehold_idx = ~0u; ch->gr_nocap_idx = ~0u; /* m164: no no-capture window armed */ ch->session_gen = svc_sess_gen; /* m176 */ - /* - * STALLFREE: a genuinely new binding - a fresh channel or a - * same-handle rebinding across a session/context boundary - * (!same_gen) - must reconcile on its first pump pass: drop - * opdef slots a dead context left under this handle (only - * ones whose first-pending stamp predates the binding, so a - * slot the fresh binding creates can never be touched) and - * log the transition when the ring already carries content. - * A same-generation RM re-issue preserves state deliberately - * (m148/m176) and is not reconciled. - */ - ch->wd_chanswitch = !same_gen; - if (same_gen) + if (same_gen) { bitmap_copy(ch->rel_done, keep_rel_done, SVC_GPFIFO_MAX_BITS); + ch->gr_pub_armed = keep_pub_armed; + ch->counter_gen = keep_counter_gen; + } /* * Rank among the live CE channels OF THIS GUEST. The RM hands * out object handles in creation order and the guest's UVM @@ -1392,6 +1471,7 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) if (same) { ch->counter_pa = keep_counter_pa; ch->release_pa = keep_release_pa; + ch->release_va = keep_release_va; ch->gr_index = keep_gr_index; ch->ce_index = keep_ce_index; } @@ -1500,6 +1580,8 @@ static u32 svc_busy_defer_log; /* C8: GR busy-defer timeout witness */ static u32 svc_hold_fb_log; /* C7: hold failure/timeout fallback witness */ static u32 svc_cand_ovw_log; /* C9: candidate overwrite witness */ static u32 svc_inline_recs_log; /* C10: inline-recs digest witness */ +static u32 svc_gr_await_log; /* r351: launch-held-for-trust witness */ +static u32 svc_gr_retry_log; /* r351: retry-forward/proceed witness */ void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) { @@ -1568,6 +1650,90 @@ void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) * the page tables itself and hands their base address to the GPU). Store it so * the service can translate that space's VAs directly. */ +/* + * r496j: unverified page-directory candidates from COPY_SERVER_PDES. A + * kernel-RM internal VAS publishes its chain there, but registering every + * client's directory directly lets a stale kernel directory answer another + * client's VA in the walker's first-success order (the r459 stale-directory + * class; observed as a CUDA client hanging after DEVICE). Candidates are + * parked here and registered only after a walk of the client's own + * content-validated ring VA through the candidate reproduces that ring's PA + * (svc_gr_ram_ring_scan's latch calls svc_vas_candidates_verify). + */ +#define SVC_VAS_CAND_MAX 16 +static struct { + bool valid; + u32 h_vaspace; + u64 pdb; + u32 entries; + bool vidmem; +} svc_vas_cand[SVC_VAS_CAND_MAX]; + +static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, + u64 va, int tap, u64 *addr, int *ap, bool trace); + +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem) +{ + int i; + + if (!pdb || entries < 2 || entries > 4096) + return; + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + if (svc_vas_cand[i].valid && + svc_vas_cand[i].h_vaspace == h_vaspace && + svc_vas_cand[i].pdb == pdb) + return; /* already parked */ + if (!svc_vas_cand[i].valid) { + svc_vas_cand[i].valid = true; + svc_vas_cand[i].h_vaspace = h_vaspace; + svc_vas_cand[i].pdb = pdb; + svc_vas_cand[i].entries = entries; + svc_vas_cand[i].vidmem = vidmem; + return; + } + } +} + +/* + * Verify each parked candidate against one content-validated translation + * (the ring VA -> PA the RAM scan just proved): walk the ring page through + * the candidate and register only the directory that reproduces it. Runs on + * the service thread from the ring-scan latch; the walk itself is cheap (a + * few table reads). + */ +static void svc_vas_candidates_verify(struct nv_vgpu_device *vgpu, u64 va, + u64 pa) +{ + int i; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + u64 wpa = 0; + int wap = 0; + int ap = svc_vas_cand[i].vidmem ? SVC_APERTURE_FB + : SVC_APERTURE_SYS; + struct sema_vaspace_desc d; + + if (!svc_vas_cand[i].valid) + continue; + if (svc_gmmu_walk(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, va & ~0xfffULL, + ap, &wpa, &wap, false) || + (wpa & ~0xfffULL) != (pa & ~0xfffULL)) + continue; + memset(&d, 0, sizeof(d)); + d.h_vaspace = svc_vas_cand[i].h_vaspace; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + pr_info("sema-svc: VAS 0x%x page directory registered after verified ring walk (pdb=0x%llx entries=%u ap=%d)\n", + d.h_vaspace, d.pdb, d.num_entries, ap); + break; + } +} + void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) { struct svc_vaspace *vs = NULL; @@ -1586,12 +1752,40 @@ void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) if (!vs && !svc.vas[i].valid) vs = &svc.vas[i]; } + /* + * r459: the table is 8 slots and the guest driver does not always + * send a teardown for a client's VA space (observed: 23 distinct + * registrations, 0 drops in one boot). Once all slots hold a dead + * client's directory, a NEW client's registration found no slot and + * every walk of its VAs failed (copies rc=-2, launches reading + * stale pages - the 10th-client mid-boot degradation). Evict the + * least-recently-registered entry instead of refusing the new one; + * the old client is gone and its directory pointer is dead guest + * memory, so the eviction can only remove stale authority. + */ + if (!vs) { + unsigned long oldest = 0; + int oi = 0; + + for (i = 0; i < SVC_MAX_VASPACES; i++) { + if (!svc.vas[i].valid) + continue; + if (!oldest || time_before(svc.vas[i].stamp, oldest)) { + oldest = svc.vas[i].stamp; + oi = i; + } + } + vs = &svc.vas[oi]; + pr_info("sema-svc: VA space table full; evicting 0x%x (pdb=0x%llx) for 0x%x\n", + vs->h_vaspace, vs->pdb, desc->h_vaspace); + } if (vs) { vs->valid = true; vs->h_vaspace = desc->h_vaspace; vs->pdb = desc->pdb; vs->entries = desc->num_entries; vs->ap = desc->b_vidmem ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + vs->stamp = jiffies; /* A new directory invalidates what we learned from the old one. */ memset(svc.lat, 0, sizeof(svc.lat)); } @@ -1815,8 +2009,18 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, return -ENOENT; } - /* PD0: one dual entry per index (big half then small half). */ + /* + * PD0: one dual entry per index (big half then small half). + * Geometry per the driver's own GMMU format (gp10x family): + * PD3 [48:47], PD2 [46:38], PD1 [37:29], PD0 [28:21], PT + * [20:12]. PD0 is 8 bits - an earlier 0x1ff experiment here was + * reverted; the observed +2 MB client-page phase is NOT a PD0 + * index error. + */ idx = (va >> 21) & 0xff; + if (trace) + pr_info("sema-walk: PD0 table=0x%llx idx=%u va=0x%llx tap=%d\n", + table, idx, va, tap); if (svc_read_ap(vgpu, table + 16 * idx, tap == SVC_APERTURE_FB ? SVC_APERTURE_FB : SVC_APERTURE_SYS, @@ -1826,6 +2030,9 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, : SVC_APERTURE_SYS, &small, sizeof(small))) return -ENOENT; + if (trace) + pr_info("sema-walk: PD0 big=0x%016llx small=0x%016llx\n", + big, small); /* Small half: 4 KB page table under it. */ sap = svc_pde_aperture(((small >> 1) & 0x3)); @@ -1837,6 +2044,12 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, *addr += va & 0xfff; return 0; } + if (trace) + pr_info("sema-walk: small miss sap=%d t=0x%llx\n", + sap, t); + } else if (trace) { + pr_info("sema-walk: small skip sap=%d small&1=%llu\n", + sap, small & 1); } /* Big half: a 2 MB page here, or a large (64 KB) page table. */ @@ -1869,6 +2082,9 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, return 0; } } + if (trace) + pr_info("sema-walk: no leaf va=0x%llx big=0x%llx small=0x%llx\n", + va, big, small); return -ENOENT; } @@ -1897,6 +2113,26 @@ static int svc_walk_vaspaces(struct nv_vgpu_device *vgpu, u64 va, } spin_unlock_irqrestore(&svc.vas_lock, flags); + /* + * r459: try the NEWEST registration first. A dead client's stale + * directory (the table outlives clients; only 8 slots) can still + * resolve a VA - through guest memory the driver has since reused - + * to a wrong page, and the first success wins. Ordering by the + * registration stamp makes the live client's directory the + * authority and only falls through to older ones when it cannot + * answer. + */ + for (i = 1; i < n; i++) { + struct svc_vaspace k = local[i]; + int j = i - 1; + + while (j >= 0 && time_before(local[j].stamp, k.stamp)) { + local[j + 1] = local[j]; + j--; + } + local[j + 1] = k; + } + for (i = 0; i < n; i++) { int alt; @@ -2022,8 +2258,23 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) } if (n < SVC_PD_MIN_HITS) { n = 0; - if (svc.map_n < SVC_PD_SAMPLES) - return; /* no evidence yet: retry when pairs appear */ + /* + * r510: a LARGE map is a mixed set of VA spaces cached over + * the whole guest boot (the comment above already names the + * consequence: no single directory reproduces a mixed set), so + * scanning with its stride subsample burns the attempts budget + * on evidence that cannot succeed - and by the time the + * push-buffer pairs (the strongest samples) exist, the budget + * is spent (r508/r509 witness: 'not found (base=0x0 ap=ram + * cands=39621 best=0/8)' during the first client, eight RAM + * walks consumed, every later client's high argument pointers + * unresolvable - the terminal class). Only a SMALL map, which + * is one session's own pairs, still falls back; otherwise wait + * for the push-buffer pairs and keep the attempts. + */ + if (svc.map_n > (u32)(SVC_PD_SAMPLES * 2) || + svc.map_n < SVC_PD_SAMPLES) + return; /* no coherent evidence yet: retry later */ step = svc.map_n / SVC_PD_SAMPLES; for (i = 0; i < SVC_PD_SAMPLES; i++) { sva[i] = svc.map[i * step].gpu_page; @@ -3032,6 +3283,16 @@ static u32 svc_pool_pa_n; static u32 svc_pool_pa_next; static u64 svc_pool_pa_logged[SVC_POOL_PA_LOG_MAX]; static u32 svc_pool_pa_logged_n; +/* + * fail4: the completion-hint window was 10 s. It outlived short-lived + * clients: after a client freed its RM events the window kept re-posting + * hint events every 0.5 s, and a post already queued in the status ring + * could be read after the guest's own free processing - the guest ISR + * then ran a freed RC callback and the guest panicked in interrupt + * (rm_run_rc_callback+0x0). 1.5 s still covers a waiter that arrives + * after the edge while shrinking the post-free exposure ~7x. + */ +#define SVC_POST_HINT_MS 1500 static u64 svc_pool_pa_miss_log; static u32 svc_pool_pa_hit_log; static u32 svc_pool_wr_log; @@ -3080,6 +3341,14 @@ static void svc_pool_wr_note(u32 h, u64 va, u64 pa, const char *src, } } +/* r372: landing-table entries repaired from a fresh pin (stale-ledger class). */ +static u64 svc_ledger_pin_repairs; +/* r372: pool alt fan-outs skipped because the primary pin moved on. */ +static u64 svc_pool_alt_skipped; +/* r372: bounded page-note resolution diagnostics (va/pa/ap/branch). */ +static u32 svc_gr_pagenote_log; +static u32 svc_fbhpa_log; + /* * Fan a pool release out to the last displaced translation of the same page * (see svc_ipin_alt): the primary target may be the stale one when two host @@ -3103,6 +3372,27 @@ static void svc_pool_alt_apply(struct nv_vgpu_device *vgpu, u32 h, u64 va, return; if ((pa & ~(u64)(SEMA_PAGE_SIZE - 1)) == page) return; /* never write an identity translation */ + /* + * r372 (stale-ledger corruption class): fan out ONLY while the + * current primary pin for this VA still names the displaced page. + * Once a newer pin has displaced it, the old page belongs to a + * buffer the guest has freed - writing it lands on whatever the + * guest kernel reused the page for. A missing current pin is + * equally unprovable: skip. The primary release write already + * succeeded either way, so this is a lost best-effort, not a + * failure. + */ + { + u64 cpa = 0; + int cap = SVC_APERTURE_SYS; + + if (svc_ipin_lookup(va, &cpa, &cap) || + ((cpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + svc_pool_alt_skipped++; + return; + } + } if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &cur, sizeof(cur))) return; if (cur >= val) @@ -3537,6 +3827,10 @@ static int svc_sem_read(struct nv_vgpu_device *vgpu, u64 va, u32 *val) } static void svc_pending_add(u64 va, u32 payload, u32 op); +/* r34x: defined with the pending ring / opdef slots (later in the file); + * the acquire gate drains/appraises withheld releases before it holds. */ +static void svc_apply_pending(struct nv_vgpu_device *vgpu); +static bool svc_wait_pend_release(u64 va); /* * F1: GR release-word authorship checks. Defined next to the svc_fwd_req @@ -3949,6 +4243,96 @@ static void svc_session_flush(void); /* defined next to the pending queue */ #define SVC_LEDGER_EVICT_RESET 256u static u32 svc_ledger_n; static u64 svc_ledger_diverges; +static u64 svc_ledger_pinwins; /* r336: pin-backed divergence wins */ +static u32 svc_d2hmap_log; /* D2H-MAP per-page destination budget */ +/* + * r336/r34x: host-semaphore acquire (C56F SEM_EXECUTE op 0/2/3/4/5) wait + * state. The decoder captures EVERY acquire of the ring entry currently + * being decoded (svc_waits[], reset per entry at the top of + * svc_decode_stream), and svc_run_ops honors them before running any of + * the entry's work, so a copy cannot execute ahead of the dependency its + * push waits on (the streams2 readback-before-scale class). Per-entry + * capture is essential: the wait state used to leak across entries (a + * later launch's self-release wait, e.g. need=85 while the held push + * releases 84, gated an earlier entry that carries no acquire at all - + * the r340 over-hold). The re-decode re-parses the entry, so the state + * is refreshed on every pass. + * + * Semantics per op (class contract, see svc_wait_satisfied): + * 0 ACQUIRE = (v == p); 2 ACQ_STRICT_GEQ = v >= p; 3 ACQ_CIRC_GEQ = + * circular (s32)(v - p) >= 0; 4 ACQ_AND = (v & p) != 0; + * 5 ACQ_NOR = ~(v | p) != 0. + */ +#define SVC_WAIT_MAX 8 +/* + * r401: bounded acquire wait (fail-open), now a module parameter. The + * original 100 ms bound was shorter than a single GR launch's completion + * latency in this emulation (observed 0.5-1.6 s per launch), so a copy + * that waited on a launch's tracker (streams2: the D2H readback's + * ACQ_CIRC_GEQ on the scale channel's progress counter) gave up and ran + * BEFORE the launch wrote its output - the host read pre-scale data + * (bad=32767, c2 all wrong). 4 s covers several launch latencies; the + * bound still guarantees the entry is never parked forever (a genuinely + * unsatisfiable acquire fails open late, after the dependency's own work + * would have landed - late-but-correct instead of early-and-wrong). + */ +unsigned gsp_sema_wait_hold_ms = 4000; +struct svc_wait_op { + u64 va; + u32 payload; + u8 op; /* C56F SEM_EXECUTE operation */ + u32 off; /* method offset in the entry (order) */ +}; +static struct svc_wait_op svc_waits[SVC_WAIT_MAX]; +static u32 svc_wait_n; /* acquires captured this entry */ +/* + * r484 THE STAGING-DRAIN COMPLETION EDGE: when the acquire gate holds an + * entry, the op can still have executed on an earlier decode whose gate did + * not trigger (svc_wait_n was 0 on that pass), and the once-only done + * bitmap then suppresses it on the first pass whose gate finally passes - + * the pass at which the dependency just became installed and the op's + * inputs changed. The release then applies and the guest drains a staging + * buffer holding the PRE-dependency bytes (streams_events s=2: the D2H + * copy read dc2 before the vadd_into wrote it, ran early on gate-less + * passes, was once-only-skipped on the passing pass, and the client + * drained a2+b2 instead of a2+b2+c1). Remember the last held (channel, + * entry) and force exactly ONE re-run of its ops on the first passing + * pass. + */ +static u32 svc_wait_hold_h; +static u32 svc_wait_hold_idx = 0xffffffffu; +static bool svc_wait_repass; +static u32 svc_wait_ovf; /* acquires dropped past the array */ +static u64 svc_wait_held; +static u64 svc_wait_pass; +static u64 svc_wait_giveup; +static u32 svc_wait_log; /* per-session held-row print budget */ +static u32 svc_wait_pass_log; /* per-session resolved-row budget */ +static u32 svc_remapconst_log; /* r343 FILL: SET_REMAP_CONST_A rows */ +static u32 svc_pitch_log; /* r402: 2D (pitched) op builds */ +static u32 svc_fill_raw_log; /* r365 FILL-RAW: raw dropped-fill dumps */ +static u32 svc_lowfill_log; /* r365b LOWFILL-WIT: below-floor fills */ +static u32 svc_tabcopy_log; /* r367 TABLE-COPY: table-page copies */ +static u32 svc_fill_noremap; /* r343 FILL: fail-closed holds */ +static u64 svc_wait_rel_va; /* this entry's own release (mirror) */ +static u32 svc_wait_rel_payload; +static u32 svc_wait_rel_off; /* its method offset (order check) */ +static bool svc_wait_rel_have; +/* + * Per-channel first-hold clock for the bounded wait: a gate evaluation + * that cannot satisfy the acquire arms a 100 ms budget per hold episode; + * expiry fails open (ops run) with a loud 'give-up' row. The table is + * tiny (a handful of channels), service-thread only, cleared per pass + * resolution / session flush. + */ +struct svc_wait_hold { + bool active; + u32 h; + unsigned long since; +}; +#define SVC_WAIT_HOLD_MAX 16 +static struct svc_wait_hold svc_wait_holds[SVC_WAIT_HOLD_MAX]; +static u32 svc_dst_div_log; /* r402: copy-dst ledger/probe divergence */ static u64 svc_ledger_stores; static u64 svc_ledger_evictions; static u64 svc_ledger_evict_since_reset; @@ -4076,10 +4460,12 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) svc.gr_nocap_held, svc.gr_nocap_consumed, svc.fwd_gr_hold_exp, svc.fwd_gr_stall_swrel, svc.gr_stale_wedge, svc_pending_ovf); - pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu altw=%llu residue=%llu\n", + pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu altw=%llu residue=%llu wait=%llu/%llu giveup=%llu ovf=%u\n", svc.counter_pub_calls, svc.gr_rel_applied, svc.sem_write_reresolve, svc.pushpg_fallback, - svc.pool_alt_applied, svc.pool_residue); + svc.pool_alt_applied, svc.pool_residue, + svc_wait_held, svc_wait_pass, svc_wait_giveup, + svc_wait_ovf); if (svc.resc_run || svc_rescue_scan_capped) pr_info("sema-svc: rescue_run=%llu scan_capped=%llu (multi-page content pins)\n", svc.resc_run, svc_rescue_scan_capped); @@ -4097,11 +4483,12 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) * content" window. n approaching SVC_LEDGER_MAX or evict > 0 in a * client's window is the tell. */ - pr_info("sema-svc: ledger n=%u/%u stores=%llu evict=%llu resets=%llu diverges=%llu reloc_div=%llu rewrites=%llu\n", + pr_info("sema-svc: ledger n=%u/%u stores=%llu evict=%llu resets=%llu diverges=%llu pinwins=%llu reloc_div=%llu rewrites=%llu pinrep=%llu alt_skip=%llu\n", svc_ledger_n, SVC_LEDGER_MAX, svc_ledger_stores, svc_ledger_evictions, svc_ledger_evict_resets, - svc_ledger_diverges, - svc_ledger_reloc_diverges, svc_ledger_rewrites); + svc_ledger_diverges, svc_ledger_pinwins, + svc_ledger_reloc_diverges, svc_ledger_rewrites, + svc_ledger_pin_repairs, svc_pool_alt_skipped); /* * P6b wave 1: guest-forwarding summary. Printed whenever any counter * moved (not only on requests) so a run that captured no candidate is @@ -4160,6 +4547,9 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) #define CE_M_OFFSET_OUT_UPPER 0x408 #define CE_M_OFFSET_OUT_LOWER 0x40c #define CE_M_LINE_LENGTH_IN 0x418 +#define CE_M_LINE_COUNT 0x41c +#define CE_M_PITCH_IN 0x410 +#define CE_M_PITCH_OUT 0x414 #define CE_M_SET_REMAP_CONST_A 0x700 #define CE_M_SET_REMAP_CONST_B 0x704 #define CE_M_SET_REMAP_COMPONENTS 0x708 @@ -4234,6 +4624,19 @@ struct svc_memset_op { u64 bytes; /* copy length */ u32 src_phys; u32 src_aperture; + /* + * r402: 2D (pitched) transfers. rows > 1 selects row-wise mode: + * the op runs `rows` times with the destination advancing by + * dpitch (and the source by spitch) per row, each row carrying + * `bytes` (copy) or elem*count (fill) bytes. cudaMemset2D and + * cudaMemcpy2D issue exactly this shape (LINE_COUNT rows, + * PITCH_OUT/PITCH_IN in bytes); without it the fill/copy ran flat + * and the partial-width rows corrupted the neighbouring columns + * (victory_memset2d bad>12M, memcpy_roundtrip pitched class). + */ + u32 rows; + u32 dpitch; + u32 spitch; }; /* Decoder state carried across the methods of one push. */ @@ -4245,9 +4648,19 @@ struct svc_push_state { u32 have_in, have_out; u32 line_length; u32 gr_line_count; /* GRCE LINE_COUNT */ + /* + * r402: 2D transfer geometry (CE class): PITCH_IN/PITCH_OUT in + * bytes and LINE_COUNT rows. A pitched fill/copy whose pitch + * exceeds the row length must run row-wise; a flat transfer leaves + * pitch == line_length (or no pitch method at all). + */ + u32 pitch_in, pitch_out; + u32 have_pitch_in, have_pitch_out; + u32 line_count; u32 sem_va_hi, sem_va_lo, sem_payload; u32 have_sem; u32 have_remap; + u32 have_remap_a, have_remap_b, have_remap_c; u32 release_dma; u32 sem_inc; u32 rs_upper, rs_lower, rs_payload; @@ -4378,7 +4791,18 @@ static int svc_op_translate(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, * withholding the release corrupted the next session (r59: the late * memset overwrote refilled data after the client had passed the * release), so the two must move together. + * + * r383 (streams2 big-copy class): name the missed VA through the + * same stalled-VA demand channel the GR fill uses. The userspace pin + * stream watches the "stalled VA" row and pushes the VA to the + * in-guest scanner, so the pin lands on the next scanner pass + * instead of waiting out a full scan cadence. The hold itself is + * unchanged; this only shortens the window in which the entry sits + * op-pending (r377: a held H2D copy executed ~1.5 s late and read a + * staging page the client had since refilled with the next copy's + * bytes - the neighbor-data class). */ + svc_log_stalled_va(va & ~(u64)(SEMA_PAGE_SIZE - 1)); svc.op_tr_pending = 1; return -ENOENT; } @@ -4461,8 +4885,213 @@ static void svc_ledger_store(u64 va, u64 pa, u32 ap); /* R2 ledger, defined belo static int svc_ledger_lookup(u64 va, u64 *pa, u32 *ap); /* R2 ledger, defined below */ static void svc_ledger_clear(void); /* STALLFREE: cap-and-degrade reset */ +/* + * r384 (the guest boot-corruption lottery, root-caused 2026-10-20): + * page-table pages of the guest kernel's direct map were found ZEROED + * (two adjacent 4 KiB PTE pages at gpa 0x12d3c2000/0x3000 => two 2 MiB + * holes in the guest physmap, kernel objects inside them unreachable, + * multi-oops panic). The only emulation writers of page-sized ZERO runs + * are the virtual FILL executor (svc_fill / CE memsets), whose large + * >= SVC_VAS_FLOOR destinations resolve through the walk/scan when no + * pin and no ledger answer exists - and that resolver is demonstrably + * non-deterministic (four executions of the same 8 KiB engine-window + * clear resolved to four different guest pages, all refused by the + * below-floor guard). + * + * Two nets, mirroring the m169/r378 copy guards: + * 1. svc_fill_guard_hold(): a LARGE (>= 64 KiB) virtual fill whose only + * authority is the walk/scan (no pin, no ledger) is held through a + * short bounded window and its VA pushed to the pin-demand channel + * (the userspace pin stream injects the guest-pagemap truth), so the + * retried decode resolves through a real pin. Fail-open after the + * bound with the historical behaviour, like the copy guard. + * 2. svc_fill_target_is_pt(): a ZERO fill about to land in guest RAM on + * a page whose first 64 qwords are all PTE-shaped (present, NX, + * physical address in range) is REFUSED loudly. A page table page is + * never a legitimate zero-fill destination; this directly blocks the + * observed corruption shape even when some other resolver approved + * the target. Pin-backed resolutions are exempt (the guest's own + * pagemap is truth - if it says the VA maps the page, obey it). + */ +unsigned gsp_sema_fill_guard_ms = 800; +bool gsp_sema_fill_guard = true; +#define SVC_FILL_GUARD_SLOTS 16 +static struct { + u64 va_page; + unsigned long first; +} svc_fill_guard[SVC_FILL_GUARD_SLOTS]; +static u32 svc_fill_guard_next; +static u32 svc_fill_held; +static u32 svc_fill_pt_refused; +static u32 svc_fill_exec_log; /* r384 FILL-EXEC witness budget */ + +/* true = the guard may hold this fill; false = fail-open, run it. */ +static bool svc_fill_guard_hold(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i, slot = SVC_FILL_GUARD_SLOTS; + + for (i = 0; i < SVC_FILL_GUARD_SLOTS; i++) { + if (svc_fill_guard[i].va_page == page) { + if (time_before(now, svc_fill_guard[i].first + + msecs_to_jiffies( + gsp_sema_fill_guard_ms))) + return true; + if (svc_fill_guard[i].first) { + svc_fill_guard[i].first = 0; + pr_info("sema-svc: fill guard released without a pin va=0x%llx (using the walk translation)\n", + page); + } + return false; + } + if (!svc_fill_guard[i].va_page) + slot = i; + } + if (slot == SVC_FILL_GUARD_SLOTS) + slot = svc_fill_guard_next++ % SVC_FILL_GUARD_SLOTS; + svc_fill_guard[slot].va_page = page; + svc_fill_guard[slot].first = now; + return true; +} + +/* + * A guest-kernel page table page: the first SVC_PT_SCAN qwords are all + * present PTEs (bit 0), non-executable (bit 63 - kernel data mappings), + * with a physical address inside the 34-bit guest range. Random payload + * qwords pass all three conditions with probability far below the 75% + * threshold at SVC_PT_SCAN=64; a genuine table passes at 100%. + */ +#define SVC_PT_SCAN 64u +#define SVC_PT_MIN 48u +static bool svc_fill_target_is_pt(struct nv_vgpu_device *vgpu, u64 addr) +{ + u64 ents[SVC_PT_SCAN]; + u32 i, hits = 0; + + if (addr & (SEMA_PAGE_SIZE - 1)) + return false; /* only a page-aligned table shape */ + if (svc_read(vgpu, addr, ents, sizeof(ents))) + return false; + for (i = 0; i < SVC_PT_SCAN; i++) { + u64 e = ents[i]; + + if (!(e & 1ull)) + continue; + if (!(e & (1ull << 63))) + continue; + if ((e & 0x000ffffffffff000ull) == 0) + continue; + if ((e & 0x000ffffffffff000ull) >= (1ull << 34)) + continue; + hits++; + } + return hits >= SVC_PT_MIN; +} + +/* + * r391 per-page destination resolution for a VIRTUAL fill (the r390 + * guest-panic class). A device VA's pages are independent guest allocations + * - the pin stream shows split runs (va=0x7c36c7400000 -> 29 pages at + * 0x131b63000, the next run at 0x131a80000) - so resolving only the fill's + * first page and writing `total` bytes contiguously past it writes over + * whatever memory follows the first run. On r390 a 383620-byte zero fill + * did exactly that: the contiguous write covered the guest kernel's + * page-table pages (0x131ba1000/0x131ba2000/0x131bab000), the direct map + * grew holes, and the guest died with "Fatal exception in interrupt" + * (unmap_page_range / cpuacct / psi faults). svc_copy has always translated + * virtual operands per 4 KB page; the fill must do the same. + * + * Authority order mirrors the top-level resolution (and svc_copy's per-page + * dst): a fresh pin (guest pagemap truth) wins and repairs the landing + * table, then the landing table (the page a previous write of this VA + * landed on), then the resolver chain. A page with no resolution returns + * -ENOENT: the caller fails the fill loudly instead of writing through a + * contiguous guess. + */ +static int svc_fill_dst_page_pa(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out, u32 *prov_out) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0, lpa = 0; + u32 aper = 0, lap = 0; + + if (!svc_pin_lookup_any(page, &pa, &aper)) { + *prov_out = 1; + if (!svc_ledger_lookup(page, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != page) || + (u32)lap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill-page va=0x%llx ledger=0x%llx/%u pin=0x%llx/%u\n", + page, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, pa, aper); + svc_ledger_store(page, pa, aper); + } + *pa_out = pa; + *ap_out = aper; + return 0; + } + if (!svc_ledger_lookup(page, &pa, &aper)) { + *prov_out = 2; + *pa_out = pa; + *ap_out = aper; + return 0; + } + if (!svc_op_translate(vgpu, page, &pa, &aper)) { + *prov_out = 3; + *pa_out = pa; + *ap_out = aper; + return 0; + } + return -ENOENT; +} + +static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, + u32 elem, u32 count, u32 phys, u32 aperture, + u64 *done_out); +static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, + size_t total, u32 dst_phys, u32 dst_aperture, + u32 src_phys, u32 src_aperture, u64 *done_out, + u32 owner_ch, u32 owner_idx, u64 owner_hash); + +/* + * r402 row-wise wrapper: a 2D (pitched) fill runs the 1D fill once per + * row with the destination advancing by dpitch. Each row keeps the whole + * per-page resolution, ledger-store and guard machinery of the 1D path + * (no new resolution semantics), and a failed row's retry simply re-runs + * the already-written rows with the same pattern - idempotent for fills. + */ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, - u32 count, u32 phys, u32 aperture, u64 *done_out) + u32 count, u32 phys, u32 aperture, u64 *done_out, + u32 rows, u32 dpitch) +{ + if (rows > 1 && dpitch) { + u64 done = 0; + int rc = 0; + u32 r; + + for (r = 0; r < rows; r++) { + u64 row_done = 0; + + rc = svc_fill_1d(vgpu, addr + (u64)r * dpitch, + value, elem, count, phys, aperture, + &row_done); + done += row_done; + if (rc) + break; + } + if (done_out) + *done_out = done; + return rc; + } + return svc_fill_1d(vgpu, addr, value, elem, count, phys, aperture, + done_out); +} + +static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, + u32 count, u32 phys, u32 aperture, u64 *done_out) { u8 pat[8]; u8 bounce[256]; @@ -4483,16 +5112,47 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, if (!phys) { /* Virtual destination: the VA shadow first (its pages stay - * authoritative), then the unified landing table (the page a - * previous write of this VA landed on - always the - * authoritative location of the content), then the pinned - * map and the VA->PA map. */ + * authoritative), then a live pin (the guest's newest + * statement - r372: a re-pinned VA must not be written + * through a stale landing), then the unified landing table + * (the page a previous write of this VA landed on - always + * the authoritative location of the content), then the + * pinned map and the VA->PA map. */ u64 pa; u32 aper; + u32 prov = 0; /* r384: 1 pin, 2 ledger, 3 walk/scan */ if (svc_vas_has(addr)) { shadow = true; + } else if (!svc_pin_lookup_any(addr, &pa, &aper)) { + u64 lpa = 0; + u32 lap = 0; + + prov = 1; + /* + * r372 pin-wins: repair the landing table when it + * still names the pre-re-pin page, so every later + * reader follows the pin too. + */ + if (!svc_ledger_lookup(addr, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill-dst va=0x%llx ledger=0x%llx/%u pin=0x%llx/%u\n", + addr, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, pa, aper); + svc_ledger_store(addr, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper); + } + addr = pa; + aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; } else if (!svc_ledger_lookup(addr, &pa, &aper)) { + prov = 2; addr = pa; aperture = (aper == SVC_APERTURE_FB) ? CE_APERTURE_VID : CE_APERTURE_SYS; @@ -4502,9 +5162,101 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, shadow = true; svc.vas_shadowed++; } else { + prov = 3; addr = pa; aperture = aper; } + /* + * r365b LOWFILL-WIT: a below-floor virtual fill (the guest's + * vidmem-window page-table writes) is a corruption risk: its + * resolution must be proven before the op is allowed to run + * (r365: 16-byte and 32 KB zero fills landed in guest RAM and + * killed the guest kernel). Log every resolution bounded. + */ + if (va0 < SVC_VAS_FLOOR && svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill: va=0x%llx shadow=%d resolved=0x%llx ap=%u elem=%u count=%u value=0x%llx\n", + va0, shadow, addr, aperture, elem, count, + value); + } + /* + * r369: a below-floor fill may ONLY land in vidmem. The + * 0x204020000 engine-window clear resolved to a DIFFERENT + * guest-RAM page on every execution (9 runs, 9 pages) and + * the guest died with its task_struct corrupted (gmain NULL + * deref in __fdget) - each wrong resolution zeroed 8 KB of + * guest kernel memory. Vidmem resolutions (the page-table + * region, ap=VID) are the emulation's own shadow and stay + * allowed; a SYS resolution for a window VA is refused and + * logged loudly. + */ + if (va0 < SVC_VAS_FLOOR && !shadow && + aperture != CE_APERTURE_VID) { + pr_err_ratelimited("vgsp-sema: LOWFILL-REFUSED va=0x%llx pa=0x%llx ap=%u elem=%u count=%u (below-floor fill resolved outside vidmem)\n", + va0, addr, aperture, elem, count); + return -ENOENT; + } + /* + * r384 net 1 (pin-demand hold for unproven large fills): a + * >= 64 KiB virtual fill whose only authority is the + * walk/scan resolver (no pin, no landing-table entry) AND + * which would land in GUEST RAM (SYS aperture) is the class + * that zeroed the guest's page-table pages (see the guard + * commentary above svc_fill_guard_hold). Demand the VA + * through the pin channel and hold the op through a short + * bounded window so the retried decode resolves through the + * guest's own pagemap; fail open with the historical + * behaviour after the bound. Vidmem-shadow resolutions are + * the emulation's own memory and stay immediate. + */ + if (gsp_sema_fill_guard && va0 >= SVC_VAS_FLOOR && !shadow && + total >= (64u << 10) && prov >= 3 && + aperture == CE_APERTURE_SYS) { + svc_log_stalled_va(va0); + if (svc_fill_guard_hold(va0)) { + svc_fill_held++; + if (svc_fill_held <= 32) + pr_info("sema-svc: FILL-HOLD va=0x%llx resolved=0x%llx ap=%u total=%zu value=0x%llx (unproven large fill; pin demanded)\n", + va0, addr, aperture, total, + value); + return -ENOENT; + } + } + /* + * r384 net 2 (page-table-shape refusal): a ZERO fill about + * to land in guest RAM on a page whose first 64 qwords are + * all present+NX+in-range PTEs is refused loudly - a page + * table page is never a legitimate zero-fill destination. + * This blocks the observed corruption shape (two adjacent + * PTE pages zeroed => two 2 MiB holes in the guest physmap) + * even when some other resolver approved the target, and + * irrespective of how wrong the resolution is. Pin-backed + * resolutions are exempt: the guest's own pagemap is truth + * there. + */ + if (gsp_sema_fill_guard && va0 >= SVC_VAS_FLOOR && !shadow && + value == 0 && total >= SEMA_PAGE_SIZE && + aperture == CE_APERTURE_SYS && prov != 1 && + svc_fill_target_is_pt(vgpu, addr)) { + svc_fill_pt_refused++; + if (svc_fill_pt_refused <= 32) + pr_err("vgsp-sema: FILL-PT-REFUSED va=0x%llx pa=0x%llx prov=%u total=%zu (zero fill target is a page table)\n", + va0, addr, prov, total); + svc_log_stalled_va(va0); + return -ENOENT; + } + /* + * r384 witness: where every >= 4 KiB virtual fill actually + * lands (va, pa, aperture, resolution provenance, pattern) - + * the decisive line for the boot-corruption forensics (the + * large zero fills were invisible until now). Bounded. + */ + if (va0 >= SVC_VAS_FLOOR && !shadow && + total >= SEMA_PAGE_SIZE && svc_fill_exec_log < 64) { + svc_fill_exec_log++; + pr_info("sema-svc: FILL-EXEC va=0x%llx pa=0x%llx ap=%u prov=%u total=%zu value=0x%llx\n", + va0, addr, aperture, prov, total, value); + } } if (gsp_sema_dbg && !shadow && svc_dst_watched(addr) && @@ -4530,20 +5282,29 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, /* * C2 (data class) + m180 unification: a large virtual fill is a * write/read pair with the memset's readers like any H2D copy - - * record its landing pages (SYS and VID alike) so the engine fill - * and the D2H read of this VA use exactly the page the fill wrote. + * record its landing pages so the engine fill and the D2H read of + * this VA use exactly the page the fill wrote. Only the VID + * (shadow) case is contiguous; a SYS landing is recorded per page + * in the loop below (r391: the contiguous `addr + po` store also + * poisoned the table with pages the fill never wrote). */ if (!phys && !shadow && va0 >= SVC_VAS_FLOOR && - total >= (64u << 10)) { - u32 cap = (aperture == CE_APERTURE_VID) ? - SVC_APERTURE_FB : SVC_APERTURE_SYS; + total >= (64u << 10) && aperture == CE_APERTURE_VID) { u64 po; for (po = 0; po < total; po += SEMA_PAGE_SIZE) - svc_ledger_store(va0 + po, addr + po, cap); + svc_ledger_store(va0 + po, addr + po, + SVC_APERTURE_FB); } - if (aperture == CE_APERTURE_VID) { + /* + * r391: a virtual device destination is translated per 4 KB page + * (see svc_fill_dst_page_pa) - never written through the first + * page's PA contiguously. The whole-range shadow write below only + * applies to a resolution that is contiguous by construction. + */ + if (aperture == CE_APERTURE_VID && + !(va0 >= SVC_VAS_FLOOR && !phys && !shadow)) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); @@ -4563,14 +5324,86 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, for (o = 0; o < total; ) { size_t chunk = total - o; size_t i; + u64 wpa = addr; + u32 wap = aperture; + bool per_page = !phys && !shadow && va0 >= SVC_VAS_FLOOR; if (chunk > sizeof(bounce)) chunk = sizeof(bounce); + /* + * r391: resolve THIS page's own landing (see + * svc_fill_dst_page_pa) and clamp the chunk to the page + * boundary - a virtual VA's pages are independent guest + * allocations and must never be written through the first + * page's PA contiguously. + */ + if (per_page) { + u32 off = (u32)((va0 + o) & (SEMA_PAGE_SIZE - 1)); + u64 ppa = 0; + u32 pap = 0, prov = 0; + int prc; + + chunk = min_t(size_t, chunk, SEMA_PAGE_SIZE - off); + prc = svc_fill_dst_page_pa(vgpu, va0 + o, &ppa, + &pap, &prov); + if (prc) { + if (done_out) + *done_out = (u64)o; + /* + * r391: demand the exact page through the pin + * channel (the userspace stream pushes this VA + * to the in-guest scanner) and hold the fill + * for a bounded retry - never write through a + * contiguous guess past a missing page. + */ + svc_log_stalled_va((va0 + o) & + ~(u64)(SEMA_PAGE_SIZE - 1)); + pr_err_ratelimited("vgsp: FILL-TRUNC va=0x%llx want=0x%llx done=0x%llx path=sema-fill (page 0x%llx unresolved)\n", + va0, (u64)total, (u64)o, + (va0 + o) & + ~(u64)(SEMA_PAGE_SIZE - 1)); + return -ENOENT; + } + wap = (pap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wpa = (ppa & ~(u64)(SEMA_PAGE_SIZE - 1)) + off; + /* + * r391 net: the r384 page-table-shape refusal + * applied to EVERY page of a virtual zero fill. + * The r390 fill's first page was the client's + * buffer, the later pages were the guest's PTE + * pages - the first-page-only check could not see + * them. A page-table-shaped guest-RAM page is never + * a legitimate zero-fill target: skip it loudly + * instead of zeroing it. Pin-backed resolutions + * are exempt (the guest's pagemap is truth there). + */ + if (wap == CE_APERTURE_SYS && value == 0 && prov != 1 && + svc_fill_target_is_pt(vgpu, + wpa & ~(u64)(SEMA_PAGE_SIZE - 1))) { + svc_fill_pt_refused++; + if (svc_fill_pt_refused <= 32) + pr_err("vgsp-sema: FILL-PT-REFUSED va=0x%llx pa=0x%llx prov=%u total=%zu (zero fill page is a page table; page skipped)\n", + va0 + o, wpa, prov, total); + o += chunk; + continue; + } + } for (i = 0; i + 8 <= chunk; i += 8) memcpy(bounce + i, pat, 8); if (i < chunk) memcpy(bounce + i, pat, chunk - i); - if (svc_write(vgpu, addr + o, bounce, chunk)) { + if (wap == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = svc_fb_shadow_size(vgpu); + + if (!base || wpa + chunk > size) { + if (done_out) + *done_out = (u64)o; + return -EINVAL; + } + memcpy(base + wpa, bounce, chunk); + } else if (svc_write(vgpu, wpa, bounce, chunk)) { if (done_out) *done_out = (u64)o; if (total >= SVC_COPY_AUDIT_MIN) @@ -4578,6 +5411,15 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, va0, (u64)total, (u64)o, -EIO); return -EIO; } + /* + * r391: record this page's landing (the readers follow the + * table) - SYS and VID alike, exactly like the copy path. + */ + if (per_page) + svc_ledger_store(va0 + o, + wpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (wap == CE_APERTURE_VID) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS); o += chunk; } if (done_out) @@ -4697,7 +5539,7 @@ struct svc_gr_data_win { u64 reloc_base; /* relocated window base */ }; -struct svc_gr_cand { +struct svc_gr_cand_rec { bool valid; bool offered; /* consumed by a request or a fallback */ /* @@ -4733,7 +5575,105 @@ struct svc_gr_cand { u32 pointer_word[SVC_GR_POINTER_MAX]; /* QMD word of the pair */ }; -static struct svc_gr_cand svc_gr_cand; +/* + * fail4: the capture was a single slot; two concurrent launches (streams2's + * vadd pair on two channels) overwrote each other's candidate while the + * first was kept for a retry (the candidate-overwrite witness) and the + * engines ran crossed parameters (c1 == a2+b2). Keep up to four candidates + * side by side in a ring; the accessor macro addresses the current slot, + * so every existing use keeps working on the selected candidate, and + * capture/consume select the slot instead of clobbering it. + */ +#define SVC_GR_CAND_RING 4 +static struct svc_gr_cand_rec svc_gr_cand_ring[SVC_GR_CAND_RING]; +static u32 svc_gr_cand_pos; +#define svc_gr_cand (svc_gr_cand_ring[svc_gr_cand_pos]) +/* + * r383: push-VA history of recent captures, beyond the (four-slot) + * candidate ring's lifetime. A replayed decode of a complete QMD asks + * "has this launch ever been captured?"; the ring alone can have rotated + * past it, and a wrong "no" would re-capture (and re-execute) an already + * forwarded launch. Bounded, service thread only. + */ +#define SVC_GR_CAP_HIST 8 +static u64 svc_gr_cap_hist[SVC_GR_CAP_HIST]; +static u32 svc_gr_cap_hist_pos; + +/* + * r401: robust session-scoped "already captured" memory for the r383 + * replay-capture path. The 4-slot candidate ring + 8-VA history evicted a + * real capture within ~12 later captures, so a true replay of an + * already-forwarded launch (identical push VA and content) was + * re-captured and re-executed: the r399 soak log shows the same nine + * launch pushes re-submitted every ~10-11 s for minutes (idempotent + * kernels survive it; accumulators would not). The open-addressed table + * keeps the (push_va, push_hash) identity of every capture of the + * session, so a true replay always classifies as "seen" (LAUNCH-SKIP, + * exactly once) while a genuinely new push - even at the same VA with + * the same entry words - is captured through the normal + * `trusted && !replay` path and never consults this table. Cleared by + * svc_session_flush (a new client's identical-payload pushes restart + * their release counters, so a boot-scoped table would wrongly suppress + * them). Service-thread only. + */ +#define SVC_GR_SEEN_SLOTS 2048 +#define SVC_GR_SEEN_PROBE 8 +static struct { + u64 push_va; + u64 hash; +} svc_gr_seen[SVC_GR_SEEN_SLOTS]; +static u64 svc_gr_seen_stale; /* probe-chain evictions (telemetry) */ + +static u32 svc_gr_seen_mix(u64 push_va, u64 hash) +{ + u64 k = push_va * 0x9E3779B97F4A7C15ULL; + + k ^= hash + 0x9E3779B97F4A7C15ULL + (k << 6) + (k >> 2); + return (u32)(k % SVC_GR_SEEN_SLOTS); +} + +static bool svc_gr_seen_lookup(u64 push_va, u64 hash) +{ + u32 i, base = svc_gr_seen_mix(push_va, hash); + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_SEEN_PROBE; i++) { + u32 k = (base + i) % SVC_GR_SEEN_SLOTS; + + if (!svc_gr_seen[k].push_va) + return false; + if (svc_gr_seen[k].push_va == push_va && + svc_gr_seen[k].hash == hash) + return true; + } + return false; +} + +static void svc_gr_seen_store(u64 push_va, u64 hash) +{ + u32 i, base = svc_gr_seen_mix(push_va, hash); + + if (!push_va) + return; + for (i = 0; i < SVC_GR_SEEN_PROBE; i++) { + u32 k = (base + i) % SVC_GR_SEEN_SLOTS; + + if (!svc_gr_seen[k].push_va || + (svc_gr_seen[k].push_va == push_va && + svc_gr_seen[k].hash == hash)) { + svc_gr_seen[k].push_va = push_va; + svc_gr_seen[k].hash = hash; + return; + } + } + /* Full probe chain: replace the tail (oldest-ish) entry. */ + svc_gr_seen_stale++; + svc_gr_seen[(base + SVC_GR_SEEN_PROBE - 1) % SVC_GR_SEEN_SLOTS].push_va = + push_va; + svc_gr_seen[(base + SVC_GR_SEEN_PROBE - 1) % SVC_GR_SEEN_SLOTS].hash = + hash; +} static_assert(SVC_GR_QMD_WORDS == GSP_PHYS_GR_QMD_WORDS, "the sema GR QMD word count must match the backend ABI"); @@ -4769,7 +5709,7 @@ static_assert(sizeof(struct svc_gr_page) == 24, static u64 svc_push_hash(const u8 *p, u32 len); /* QMD field decode: the exact shape the host CUDA bridge uses. */ -static void svc_gr_qmd_decode(struct svc_gr_cand *c) +static void svc_gr_qmd_decode(struct svc_gr_cand_rec *c) { const u32 *q = c->qmd; u32 i; @@ -4814,6 +5754,25 @@ static void svc_gr_cand_capture(struct svc_channel *ch, * overwrite hazard. Bounded, unconditional (gsp_sema_dbg is defined * later in the file and this site may not reference it). */ + /* + * fail4: never overwrite a DIFFERENT launch's unconsumed candidate - + * move to a free ring slot instead (same-push re-captures still + * refresh in place). + */ + if (svc_gr_cand.valid && !svc_gr_cand.offered && + svc_gr_cand.push_va != push_va) { + u32 ri; + + for (ri = 1; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % SVC_GR_CAND_RING; + + if (!svc_gr_cand_ring[rp].valid || + svc_gr_cand_ring[rp].offered) { + svc_gr_cand_pos = rp; + break; + } + } + } if (svc_gr_cand.valid && svc_cand_ovw_log < 64) { svc_cand_ovw_log++; pr_info("sema-gr: CAND-OVERWRITE old_h=0x%x old_push=0x%llx new_push=0x%llx\n", @@ -4825,8 +5784,14 @@ static void svc_gr_cand_capture(struct svc_channel *ch, svc_gr_cand.h_object = ch->h_object; svc_gr_cand.push_va = push_va; svc_gr_cand.push_len = push_len; + /* r383: remember the identity beyond this ring slot's lifetime. */ + svc_gr_cap_hist[svc_gr_cap_hist_pos % SVC_GR_CAP_HIST] = push_va; + svc_gr_cap_hist_pos++; if (push && push_len) svc_gr_cand.push_hash = svc_push_hash(push, push_len); + /* r401: the session-wide exactly-once memory (survives ring rotation; + * see svc_gr_seen_lookup). */ + svc_gr_seen_store(push_va, svc_gr_cand.push_hash); if (push && push_len >= 4) { u32 total = push_len / 4; u32 n = total < SVC_GR_PUSH_WORDS ? total : SVC_GR_PUSH_WORDS; @@ -4919,8 +5884,14 @@ static bool svc_fwd_defer_arm(void) { if (!gsp_phys_guest_mem_enabled()) return false; - if (svc_gr_cand.valid && gsp_phys_gr_forward_enabled()) - return true; + if (gsp_phys_gr_forward_enabled()) { + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered) + return true; + } return svc_fwd_cand.valid && gsp_phys_work_defer_release_enabled(); } @@ -5185,9 +6156,228 @@ static void svc_copy_audit_chunk(struct nv_vgpu_device *vgpu, u64 dst_va, return (rc_); \ } while (0) +/* + * fail4: pipeline-order source snapshot for held copies. The m169 + * device-dst guard defers a staged H2D (host->device) for up to 3 s; + * the driver's ping-pong staging buffers are overwritten by the next + * chunk meanwhile, so the deferred read returns the content of a later + * chunk (vector_add: device a[i] pages held a[i+2 MB] - exactly the + * 2x1 MB staging stride). A held copy snapshots its source at hold + * time and the retried execution copies from the snapshot, matching + * the hardware's in-order read of the staging. + */ +#define SVC_COPY_SNAP_N 16 +#define SVC_COPY_SNAP_MAX (4u << 20) +#define SVC_COPY_SNAP_TTL_MS 30000u +struct svc_copy_snap { + u64 dst, src; + size_t total; + /* + * Owner identity (r444): the channel, ring entry and push-content + * hash of the decode pass that captured this snapshot. A snapshot + * exists to serve ONE entry's retried copy (the held staging read); + * keying it by (dst,src,total) alone let a snapshot captured for an + * earlier entry poison a later copy with the same operand triple - + * the wedge_ab "bad = mutated-element count" class: step N's H2D + * executed from step N-1's snapshot because the re-decode of the + * completed step N-1 entry re-captured its (unchanged) source after + * its copy had already run, and step N's copy then consumed it. + * The pass hash changes whenever the guest rewrites the push, so a + * ring slot reused by a fresh push can never match a stale owner. + */ + u64 owner_hash; + u32 owner_ch; + u32 owner_idx; + u8 *buf; + unsigned long stamp; +}; +static struct svc_copy_snap svc_copy_snaps[SVC_COPY_SNAP_N]; + +static u8 *svc_copy_snap_lookup(u64 dst, u64 src, size_t total, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].dst == dst && + svc_copy_snaps[i].src == src && + svc_copy_snaps[i].total == total && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash == owner_hash) + return svc_copy_snaps[i].buf; + return NULL; +} + +static void svc_copy_snap_free(u64 dst, u64 src, size_t total, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].dst == dst && + svc_copy_snaps[i].src == src && + svc_copy_snaps[i].total == total && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash == owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + return; + } +} + +/* + * r456/r457: the snapshot lifecycle is capture-once -> free on the copy's + * own successful consumption (svc_copy) -> free on ring-slot reuse with a + * new push hash (svc_copy_snap_capture) -> free when a re-decode skips an + * already-done op (svc_run_ops) -> TTL. The old per-entry owner free was + * removed from every opdef_clear caller: an entry whose release was + * delivered can still be REPLAYED (the pin-batch rewind) with its copy + * still owed, and freeing at that point let the replay re-capture the + * driver's refilled ping-pong staging (the r457 mode-C b1 H2D wrote b2's + * bytes into db1). A stale snapshot can only ever serve its own + * (ch, idx, hash) owner, so the r444 cross-entry poisoning stays closed. + */ +static void __maybe_unused svc_copy_snap_free_owner(u32 owner_ch, u32 owner_idx) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } +} + +/* Capture the source of a virtual-operand copy about to be held. */ +static void svc_copy_snap_capture(struct nv_vgpu_device *vgpu, u64 dst, + u64 src, size_t total, u32 owner_ch, + u32 owner_idx, u64 owner_hash) +{ + struct svc_copy_snap *s = NULL; + u8 *buf; + size_t o; + u32 i; + + if (!total || total > SVC_COPY_SNAP_MAX) + return; + if (svc_copy_snap_lookup(dst, src, total, owner_ch, owner_idx, + owner_hash)) + return; /* already captured */ + /* + * r456: evict any snapshot left by the SAME ring slot under a + * DIFFERENT push hash (the slot was reused by a fresh push). The + * surviving-across-retries snapshot (svc_opdef_clear APPLY no + * longer frees it) must never be readable by the next occupant of + * the slot; the pass hash changes whenever the guest rewrites the + * push, so a same-slot different-hash snapshot is by definition + * stale for the new push. + */ + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash != owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } + } + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (!svc_copy_snaps[i].buf) { + s = &svc_copy_snaps[i]; + break; + } + if (time_after(jiffies, svc_copy_snaps[i].stamp + + msecs_to_jiffies(SVC_COPY_SNAP_TTL_MS))) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + s = &svc_copy_snaps[i]; + break; + } + } + if (!s) + return; /* all slots live: keep the historical behaviour */ + buf = kmalloc(total, GFP_KERNEL); + if (!buf) + return; + for (o = 0; o < total; ) { + size_t chunk = min_t(size_t, 0x1000 - ((src + o) & 0xfff), + total - o); + u64 pa = 0; + u32 ap = 0; + + /* The source is a virtual host VA; resolve per page first. */ + if (svc_op_translate(vgpu, src + o, &pa, &ap) || + svc_read(vgpu, pa, buf + o, chunk)) { + kfree(buf); + return; + } + o += chunk; + } + s->dst = dst; + s->src = src; + s->total = total; + s->owner_ch = owner_ch; + s->owner_idx = owner_idx; + s->owner_hash = owner_hash; + s->stamp = jiffies; + s->buf = buf; + pr_info_ratelimited("vgsp-sema: copy snapshot captured dst=0x%llx src=0x%llx total=%zu owner=0x%x/%u/%016llx first4=%08x %08x %08x %08x\n", + dst, src, total, owner_ch, owner_idx, + owner_hash, + total >= 4 ? *(const u32 *)buf : 0, + total >= 8 ? *(const u32 *)(buf + 4) : 0, + total >= 12 ? *(const u32 *)(buf + 8) : 0, + total >= 16 ? *(const u32 *)(buf + 12) : 0); +} + +static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind, + u64 pa, u32 off, u32 bytes, u64 *hpa, + u32 *out_off); + +/* r402 row-wise wrapper: see svc_fill. The source advances by spitch. */ +/* r402: read-only resolver probe (defined with the GR fill diagnostics). */ +static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out); static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture, - u64 *done_out) + u64 *done_out, u32 rows, u32 dpitch, u32 spitch, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + if (rows > 1 && dpitch) { + u64 done = 0; + int rc = 0; + u32 r; + + for (r = 0; r < rows; r++) { + u64 row_done = 0; + + rc = svc_copy_1d(vgpu, dst + (u64)r * dpitch, + src + (u64)r * spitch, total, + dst_phys, dst_aperture, src_phys, + src_aperture, &row_done, + owner_ch, owner_idx, owner_hash); + done += row_done; + if (rc) + break; + } + if (done_out) + *done_out = done; + return rc; + } + return svc_copy_1d(vgpu, dst, src, total, dst_phys, dst_aperture, + src_phys, src_aperture, done_out, + owner_ch, owner_idx, owner_hash); +} + +static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, + u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture, + u64 *done_out, u32 owner_ch, u32 owner_idx, u64 owner_hash) { u8 bounce[256]; size_t o; @@ -5198,12 +6388,28 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u32 dst_prov = 0, src_prov = 0; bool bigcopy_this = false; bool audit = total >= SVC_COPY_AUDIT_MIN; + u8 *snap = NULL; static u64 bigcopy_log; + bool trace64 = (total == 65536u); + const char *dir = "?"; if (done_out) *done_out = 0; /* 0 until the first byte lands */ if (!total || total > (256ULL << 20)) return -EINVAL; + /* + * fail4: name every copy the executor is asked to run (pre- + * resolution), so a copy that fails or is held before any witness + * still has an entry record. + */ + if (gsp_sema_dbg && svc.copyent_log < 128) { + svc.copyent_log++; + pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu dphys=%u dap=%u sphys=%u sap=%u\n", + svc.cur_ch, dst, src, total, dst_phys, dst_aperture, + src_phys, src_aperture); + } + snap = svc_copy_snap_lookup(dst_orig, src_orig, total, owner_ch, + owner_idx, owner_hash); if (!dst_phys) { u64 pa; @@ -5213,12 +6419,68 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, dst_shadow = true; } else if (!svc_ledger_lookup(dst, &pa, &aper)) { /* + * r402 diagnostic (bounded): name the case where the + * ledger's landing page for a copy DESTINATION + * disagrees with the live resolver (pin/walk). The + * D2H staging writes failed on the class this + * witness detects: the writer follows the ledger + * while the guest's reader holds a different page, + * so the client reads stale data (streams2 c1 + * zeros / smiley 3-of-4 passes). Read-only probe + * (svc_gr_src_probe stores nothing). + */ + if (gsp_sema_dbg && svc_dst_div_log < 64 && + dst >= SVC_VAS_FLOOR) { + u64 wpa = 0; + u32 wap = 0; + const char *src = + svc_gr_src_probe(vgpu, dst, &wpa, &wap); + + if (wpa && + (((wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + ((wap == SVC_APERTURE_FB) != + (aper == SVC_APERTURE_FB)))) { + svc_dst_div_log++; + pr_err_ratelimited("vgsp-sema: DST-DIVERGE va=0x%llx ledger=0x%llx/%u probe=%s 0x%llx/%u (write follows ledger)\n", + dst, pa, aper, src, wpa, wap); + } + } + /* * m180 unification: a previous write of this VA is * the authority for where the content lives (and so * for where a new write lands); it also spares the * m169 stale-walk hold below, which exists precisely * because no pin was known. + * + * r376 pin-agreement: a FRESH pin that disagrees is + * newer than the recorded landing - the guest freed + * or moved the buffer - so the pin wins and the + * entry is repaired (the same rule the fills and the + * launch paths apply; without it a stale entry + * writes a freed page - the guest-corruption class). */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 64) + pr_info("sema-ledger: PIN-WINS copy-dst va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + dst, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper, ipa, iap); + svc_ledger_store(dst, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + pa = ipa; + aper = (u32)iap; + } + } dst = pa; dst_aperture = (aper == SVC_APERTURE_FB) ? CE_APERTURE_VID : CE_APERTURE_SYS; @@ -5275,9 +6537,49 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, if (svc.copy_dst_hold <= 16) pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u (no pin yet; stale-table guard)\n", dst_orig, dst_prov); + /* + * fail4: the staging source is valid NOW; snapshot it so + * the deferred execution copies the pipeline-order bytes. + */ + if (!src_phys && src_orig >= SVC_VAS_FLOOR) + svc_copy_snap_capture(vgpu, dst_orig, src_orig, + total, owner_ch, owner_idx, + owner_hash); return -ENOENT; } /* + * r378: the same stale-table class for SYS destinations resolved by + * the walk/scan (prov >= 3): a FRESH injected pin that disagrees is + * the guest's newest statement, so writing through the resolved page + * lands on a page the guest has freed/re-homed (the guest-corruption + * class). Hold through the same bounded re-decode path instead of + * writing; a missing pin keeps the historical behaviour (the + * resolution is then the write authority). + */ + if (dst_virt && dst_aperture == CE_APERTURE_SYS && + dst_prov >= 3 && dst_orig >= SVC_VAS_FLOOR && + gsp_sema_copy_dst_pin_guard) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_orig, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dst & ~(u64)(SEMA_PAGE_SIZE - 1))) || + svc_ap_of(iap) != SVC_APERTURE_SYS) && + svc_dst_guard_hold(dst_orig)) { + svc.copy_dst_hold++; + if (svc.copy_dst_hold <= 16) + pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u pin=0x%llx (stale SYS target)\n", + dst_orig, dst_prov, ipa); + if (!src_phys && src_orig >= SVC_VAS_FLOOR) + svc_copy_snap_capture(vgpu, dst_orig, + src_orig, total, + owner_ch, owner_idx, + owner_hash); + return -ENOENT; + } + } + /* * r231 READ-ORDER guard/witness: a device-src copy whose source VA * has a PENDING device-dst write hold reads the PRE-write content - * the writer's bytes land only when the guard resolves while the @@ -5301,6 +6603,49 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, if (svc_vas_has(src)) { src_shadow = true; + } else if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src, &pa, &aper)) { + /* + * r375 read-after-write authority: the landing table + * records where every write of this VA landed (writes + * store it, fresh pins repair it). A reader MUST read + * exactly that page or the engine's output is invisible + * (the sum=0 class). The walk/pin sources can answer + * differently - e.g. the same numeric pa in a different + * aperture - so the table wins for device VAs; only a + * VA with no recorded landing falls through to the + * resolver below. + * + * r376 pin-agreement: unless a FRESH pin disagrees - + * then the pin is newer (the guest re-pinned the VA) + * and wins, and the entry is repaired. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(src, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 64) + pr_info("sema-ledger: PIN-WINS copy-src va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + src, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper, ipa, iap); + svc_ledger_store(src, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + pa = ipa; + aper = (u32)iap; + } + } + src = pa; + src_aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + src_prov = 1; + src_virt = true; } else if (svc_op_translate(vgpu, src, &pa, &aper)) { u64 lpa = 0; u32 lap = 0; @@ -5330,6 +6675,16 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, src_aperture = aper; src_prov = svc.op_tr_src; src_virt = true; + /* + * r375b: do NOT record a resolver answer into the + * landing table here. The resolver can return a stale + * PTE page (the m169 class) and the table is the WRITE + * authority - a poisoned entry makes every later write + * of this VA land on the stale page (r375 boot: slab + * GPF in kmem_cache_alloc at guest t=77s, before any + * client). The table is fed by writes and fresh pins + * only. + */ } } @@ -5341,8 +6696,33 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, */ if ((total >= SVC_COPY_AUDIT_MIN || (dst_virt && dst_orig >= SVC_VAS_FLOOR)) && - bigcopy_log < gsp_sema_bigcopy_log_max) { + (bigcopy_log < gsp_sema_bigcopy_log_max || total == 65536u)) { u8 sb[16] = {0}, db[16] = {0}; + /* + * r235: 64 KB copies are exempt from the bigcopy budget - the + * probe's A upload/readbacks must ALWAYS appear (the budget + * belongs to the multi-MB traffic and had absorbed every 64 KB + * probe copy in the r235 capture). + */ + bool trace64 = (total == 65536u); + const char *dir = + (dst_aperture == CE_APERTURE_VID && + src_aperture == CE_APERTURE_SYS) ? "H2D" : + (src_aperture == CE_APERTURE_VID && + dst_aperture == CE_APERTURE_SYS) ? "D2H" : + src_shadow ? "shadow->" : "phys"; + + /* + * r235: 64 KB copies are exempt from the bigcopy budget - the + * probe's A upload/readbacks must ALWAYS appear (the budget + * belongs to the multi-MB traffic and had absorbed every 64 KB + * probe copy in the r235 capture). + */ + dir = (dst_aperture == CE_APERTURE_VID && + src_aperture == CE_APERTURE_SYS) ? "H2D" : + (src_aperture == CE_APERTURE_VID && + dst_aperture == CE_APERTURE_SYS) ? "D2H" : + src_shadow ? "shadow->" : "phys"; bigcopy_log++; bigcopy_this = true; @@ -5366,10 +6746,29 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, } else { svc_read(vgpu, dst, db, sizeof(db)); } - pr_info("sema-bigcopy: dst_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | src_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | bytes=%zu\n", + pr_info("sema-bigcopy: dst_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | src_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | bytes=%zu vbase=%px\n", dst_orig, dst, dst_aperture, dst_prov, dst_shadow, src_orig, src, src_aperture, src_prov, src_shadow, - total); + total, (void *)vgpu->bars[1].vaddr); + /* + * r348 diagnostic: for a high-VA SYS source name the exact + * host PA this read will touch (gpa_to_hpa) - the direct + * comparison against the svc-fwd-hpa writeback line. + */ + if (src_orig >= SVC_VAS_FLOOR && + src_aperture == CE_APERTURE_SYS) { + u64 shpa = 0; + u32 soff = 0; + + if (!svc_fwd_operand_hpa(vgpu, SVC_APERTURE_SYS, + src, 0, SEMA_PAGE_SIZE, + &shpa, &soff) && !soff) + pr_info("sema-bigcopy: src-hpa va=0x%llx gpa=0x%llx -> hpa=0x%llx\n", + src_orig, src, shpa); + else + pr_info("sema-bigcopy: src-hpa va=0x%llx gpa=0x%llx -> UNMAPPED\n", + src_orig, src); + } pr_info("sema-bigcopy: src16=%16phN dst16_before=%16phN\n", sb, db); /* @@ -5379,7 +6778,7 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, * (the engine-authored page as ints). The expected content is * ints 0,1,2,3,... for the probe A buffer. */ - if (total == 65536u && svc_atrace_log < 128u) { + if (trace64 && svc_atrace_log < 128u) { u32 i0, i1, i2, i3, j0 = 0, j1 = 0, j2 = 0, j3 = 0; svc_atrace_log++; @@ -5391,9 +6790,8 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, memcpy(&j1, db + 4, 4); memcpy(&j2, db + 8, 4); memcpy(&j3, db + 12, 4); - pr_info("vgsp: A-TRACE pt=%s dst_va=0x%llx dst_pa=0x%llx dst_prov=%u | src_va=0x%llx src_pa=0x%llx src_prov=%u | src_int4=%d %d %d %d dst_int4_before=%d %d %d %d\n", - dst_shadow || dst_aperture == CE_APERTURE_VID ? - "P1" : "P4", + pr_info("vgsp: A-TRACE pt=%s dir=%s dst_va=0x%llx dst_pa=0x%llx dst_prov=%u | src_va=0x%llx src_pa=0x%llx src_prov=%u | src_int4=%d %d %d %d dst_int4_before=%d %d %d %d\n", + !strcmp(dir, "H2D") ? "P1" : "P4", dir, dst_orig, dst, dst_prov, src_orig, src, src_prov, i0, i1, i2, i3, j0, j1, j2, j3); } @@ -5589,7 +6987,22 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, chunk = min_t(size_t, chunk, 0x1000 - ((dst_orig + o) & 0xfff)); - if (src_shadow) { + if (snap) { + /* + * r314: consume the decode-time staging snapshot. + * The capture machinery existed since cb50282b but + * the execution never read it (lookup at svc_copy's + * top, free on success, NO consumer) - every copy + * read the live staging, which the driver had + * refilled once the push's fence fired (the + * deterministic 786335-bad vector_add output, + * identical across boots). Reading from the + * snapshot also needs no live source translation, + * so the copies execute on their first allowed + * pass instead of storming 60+ rc=-2 retries. + */ + memcpy(bounce, snap + o, chunk); + } else if (src_shadow) { if (svc_vas_read(src_orig + o, bounce, chunk)) SVC_COPY_TRUNC_RET(-EIO); } else if (src_virt) { @@ -5598,7 +7011,34 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u64 lpa = 0; u32 lap = 0; - if (svc_op_translate(vgpu, src_orig + o, &pa, &aper)) { + /* + * r375: the landing table is the read-after-write + * authority for device VAs (see the top-level src + * resolve). Per page, the table wins whenever it has + * an entry; the resolver below only answers a VA the + * table has never seen. r376: a fresh disagreeing pin + * is newer than the table and wins (repaired). + */ + if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src_orig + o, &lpa, &lap)) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(src_orig + o, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (lpa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)lap)) { + svc_ledger_pin_repairs++; + svc_ledger_store(src_orig + o, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + lpa = ipa & ~(u64)(SEMA_PAGE_SIZE - 1); + lap = (u32)iap; + } + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (svc_op_translate(vgpu, src_orig + o, &pa, &aper)) { /* * m180 unification: a per-page resolution miss * must not fail the whole copy when the unified @@ -5650,6 +7090,21 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, SVC_COPY_TRUNC_RET(-EIO); } + /* + * r367 TABLE-COPY witness: a page-table page/entry copy is + * the guest's authoritative table content; dump the carried + * bytes so a table that later walks as ZERO can be traced to + * the copy that built it (or proven the source was already + * zero). Bounded; table-shaped sizes only. + */ + if (!o && (total == 4096 || total == 16) && + svc_tabcopy_log < 64) { + svc_tabcopy_log++; + pr_info("sema-tabcopy: dst_orig=0x%llx src_orig=0x%llx total=%zu dphys=%u dvirt=%d svirt=%d r_dst=0x%llx r_src=0x%llx first16=%16phN\n", + dst_orig, src_orig, total, dst_phys, dst_virt, + src_virt, dst, src, bounce); + } + if (dst_shadow) { if (svc_vas_write(dst_orig + o, bounce, chunk)) SVC_COPY_TRUNC_RET(-ENOMEM); @@ -5662,6 +7117,7 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u32 aper = 0; u64 lpa = 0; u32 lap = 0; + u32 wprov = 0; if (svc_op_translate(vgpu, dst_orig + o, &pa, &aper)) { /* @@ -5674,26 +7130,76 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, pa = lpa; aper = (lap == SVC_APERTURE_FB) ? CE_APERTURE_VID : CE_APERTURE_SYS; + wprov = 0xF1; } else if (dst_orig >= SVC_VAS_FLOOR && !svc_ledger_lookup(dst_orig + o, &lpa, &lap) && ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || (lap == SVC_APERTURE_FB) != (aper == CE_APERTURE_VID))) { - /* - * m180 unification: keep ONE authoritative - * page per VA. A new write goes to the page - * the previous content landed on, and the - * fill/D2H readers (which prefer the table) - * stay consistent with it. - */ svc_ledger_diverges++; - if (svc_ledger_diverges <= 32) - pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx\n", - dst_orig + o, lpa, pa); - pa = lpa; - aper = (lap == SVC_APERTURE_FB) ? - CE_APERTURE_VID : CE_APERTURE_SYS; + if (svc.op_tr_src == 1) { + /* + * r336 PIN-WINS (the r335 vector_add + * guest-panic class): a pin-backed + * translation (guest pagemap truth, + * op_tr_src == 1) that disagrees + * with the ledger means the guest + * remapped or reused this VA. The + * ledger page is where an earlier + * write landed; the guest has since + * freed or moved it, so writing it + * can land on memory the guest + * kernel now owns (observed: RM + * object fields overwritten with + * client c-data, slab freelist + * corruption, guest panic). Trust + * the pin for the write and re-home + * the ledger (the store after the + * write records the fresh page); + * readers then converge on the same + * page too. + */ + svc_ledger_pinwins++; + if (svc_ledger_diverges <= 512) + pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx (pin wins)\n", + dst_orig + o, lpa, pa); + wprov = svc.op_tr_src; + } else { + /* + * m180 unification: keep ONE + * authoritative page per VA. A new + * write goes to the page the + * previous content landed on, and + * the fill/D2H readers (which + * prefer the table) stay consistent + * with it. Non-pin translations + * (2..5) can carry stale PTEs + * (m169), so the write history + * stays the authority there. + */ + if (svc_ledger_diverges <= 512) + pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx\n", + dst_orig + o, lpa, pa); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wprov = 0xF0; + } + } else { + wprov = svc.op_tr_src; + } + /* + * D2H-MAP (diagnostic): per-page destination + * resolution of large host-dest copies - the direct + * witness for which guest pages a readback lands on. + * Bounded; one line per 4 KB page of copies >= 1 MB. + */ + if (total >= (1u << 20) && !(o & (SEMA_PAGE_SIZE - 1)) && + svc_d2hmap_log < 4096u) { + svc_d2hmap_log++; + pr_info("sema-d2hmap: va=0x%llx o=%zu pa=0x%llx ap=%u prov=%u\n", + dst_orig + o, o, pa, aper, wprov); } if (aper == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; @@ -5772,7 +7278,7 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, * the copy - for H2D this is the landed upload (P2), for D2H * the host-side readback result (P4R). */ - if (total == 65536u && svc_atrace_log < 128u) { + if (trace64 && svc_atrace_log < 128u) { u32 i0, i1, i2, i3; svc_atrace_log++; @@ -5780,9 +7286,8 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, memcpy(&i1, da + 4, 4); memcpy(&i2, da + 8, 4); memcpy(&i3, da + 12, 4); - pr_info("vgsp: A-TRACE pt=%s dst_va=0x%llx dst_pa=0x%llx dst_int4_after=%d %d %d %d\n", - dst_shadow || dst_aperture == CE_APERTURE_VID ? - "P2" : "P4R", + pr_info("vgsp: A-TRACE pt=%s dir=%s dst_va=0x%llx dst_pa=0x%llx dst_int4_after=%d %d %d %d\n", + !strcmp(dir, "H2D") ? "P2" : "P4R", dir, dst_orig, dst, i0, i1, i2, i3); } } @@ -5826,6 +7331,27 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, pr_info("vgsp: COPYAUDIT path=ce-soft-copy va=0x%llx len=0x%llx done=0x%llx\n", dst_orig, (u64)total, (u64)o); } + /* + * m180 parity with svc_fill: a large virtual-destination copy is a + * write/read pair with the launch rec-build readers - record its + * landing pages so the GR fill follows the page the copy actually + * wrote. Without this, a rec table built before the pins landed + * kept the stale pre-pin pages while the copy landed on the pinned + * page (streams2 r254: the b1 rec range carried the a2 pages and + * the vadd read a2's pattern as its second input). + */ + if (!dst_phys && dst_virt && dst_orig >= SVC_VAS_FLOOR && + total >= (64u << 10)) { + u32 cap = (dst_aperture == CE_APERTURE_VID) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + size_t po; + + for (po = 0; po < total; po += SEMA_PAGE_SIZE) + svc_ledger_store(dst_orig + po, dst + po, cap); + } + if (snap) + svc_copy_snap_free(dst_orig, src_orig, total, owner_ch, + owner_idx, owner_hash); if (done_out) *done_out = (u64)o; /* o == total on the success path */ return 0; @@ -5970,13 +7496,145 @@ static void svc_handle_map_add(void); /* Execute a decoded operation list (fills and copies). `ops_done` (may be * NULL) is the per-entry bitmap of ops that already ran for the entry's * identity; `ops_base` is the sequence number of ops[0] within the entry. */ + +/* + * r34x: evaluate a C56F SEM_EXECUTE acquire against the semaphore word. + * The class defines the opcodes (clc56f.h NVC56F_SEM_EXECUTE_OPERATION: + * 0 ACQUIRE, 1 RELEASE, 2 ACQ_STRICT_GEQ, 3 ACQ_CIRC_GEQ, 4 ACQ_AND, + * 5 ACQ_NOR, 6 REDUCTION) and the CUDA driver documents the wait + * conditions each op must implement (cuda_device.h CUIsemaphoreAquire- + * Flags_enum: GEQ default = circular >=, EQUAL = ==, AND = (v & p) != 0, + * NOR = ~(v | p) != 0, STRICT_GEQ = >= with no wrap; ampere_semaphore.c + * maps those flags onto the opcodes: EQUAL->0, STRICT_GEQ->2, GEQ->3, + * AND->4, NOR->5). Note op 0 is EQUALITY, not a >=: only op 2/3 are + * ordering compares. 32-bit payloads: every observed CUDA acquire is + * PAYLOAD_SIZE 32BIT (SEM_PAYLOAD_HI == 0); a 64-bit acquire is captured + * and judged on the low word only (its high word is not modelled). + */ +static bool svc_wait_satisfied(u32 cur, u8 op, u32 need) +{ + switch (op) { + case 0: /* ACQUIRE: wait for v == p */ + return cur == need; + case 2: /* ACQ_STRICT_GEQ: v >= p, no wrap */ + return cur >= need; + case 3: /* ACQ_CIRC_GEQ: v in [p, p + 2^31 - 1] (wrap allowed) */ + return (s32)(cur - need) >= 0; + case 4: /* ACQ_AND: (v & p) != 0 */ + return (cur & need) != 0; + case 5: /* ACQ_NOR: ~(v | p) != 0 */ + return ~(cur | need) != 0; + default: + return true; + } +} + +/* r34x: the per-channel first-hold clock for the bounded wait. */ +static struct svc_wait_hold *svc_wait_hold_get(u32 h, bool create) +{ + struct svc_wait_hold *free_slot = NULL; + int i; + + for (i = 0; i < SVC_WAIT_HOLD_MAX; i++) { + struct svc_wait_hold *e = &svc_wait_holds[i]; + + if (e->active && e->h == h) + return e; + if (!e->active && !free_slot) + free_slot = e; + } + if (!create || !free_slot) + return NULL; + memset(free_slot, 0, sizeof(*free_slot)); + free_slot->active = true; + free_slot->h = h; + free_slot->since = jiffies; + return free_slot; +} + +/* + * r506w/r507a: bounded per-VA failure latch for virtual ops. A virtual op + * that keeps failing -ENOENT (an unresolvable VA) re-runs on every decode + * pass and sets svc.op_tr_pending each time, so its push holds for the full + * SVC_OPDEF_HOLD_MS opdef bound - a client whose stream carries such an op + * (r505/r506 witness: the RM's channel-progress zeroing at 0x2000c0xxx + * after the first client, dst never translated) ground its whole stream + * at ~10 s per push ("hang after DEVICE" for matmul/multikernel). + * + * The window must still cover the legitimate pin-late retry (observed + * pin latency 0.5-2 s: the H2D copy's operands land a moment after the + * push, r507 matmul's 1 MB UVA copy at 0x7136... needed >8 fast passes), + * so the latch is TIME-based: the FIRST failure of a VA arms a + * SVC_OPDEF_HOLD_MS window during which the op holds its entry exactly as + * before; after a full bound with no success the op still attempts the + * write but no longer forces the hold - the entry consumes promptly and + * its withheld release applies, exactly the outcome the expired bound + * already sanctioned (op stays skipped, release delivered), minus the + * 10 s tax per subsequent pass. A success clears the record (a later + * failure gets a fresh window). Service-thread only, like every other op + * path. + */ +#define SVC_OPFAIL_MAX 32 +/* + * r507a: the op-failure window rides the opdef hold bound (10000 ms, + * SVC_OPDEF_HOLD_MS defined further down). Keep the literal here so the + * latch (used by svc_run_ops) precedes that define. + */ +#define SVC_OPFAIL_WINDOW_MS 10000u +static struct { + u64 va; + unsigned long since; +} svc_opfail[SVC_OPFAIL_MAX]; +static u32 svc_opfail_log; + +static void svc_opfail_clear(u64 va) +{ + u32 i; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) + if (svc_opfail[i].va == va) + svc_opfail[i].va = 0; +} + +static bool svc_opfail_latched(u64 va) +{ + u32 i; + u32 free_i = SVC_OPFAIL_MAX; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) { + if (svc_opfail[i].va == va) { + if (time_after_eq(jiffies, + svc_opfail[i].since + + msecs_to_jiffies(SVC_OPFAIL_WINDOW_MS))) { + if (svc_opfail_log < 32) { + svc_opfail_log++; + pr_info("sema-svc: OPFAIL-LATCH va=0x%llx (>=%u ms of failures; op no longer holds its entry)\n", + va, SVC_OPFAIL_WINDOW_MS); + } + return true; + } + return false; + } + if (!svc_opfail[i].va && free_i == SVC_OPFAIL_MAX) + free_i = i; + } + if (free_i < SVC_OPFAIL_MAX) { + svc_opfail[free_i].va = va; + svc_opfail[free_i].since = jiffies; + } + return false; +} + static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, - u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base) + u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base, + u32 entry_idx, u64 pass_hash) { u32 i; static u64 bigop_log; svc.cur_ch = ch_h; + /* r484: one-shot per call; set only by the hold->pass transition. */ + svc_wait_repass = false; /* * m163: apply every queued scanner injection before resolving this * pass's copy operands. Without this, a CE copy decoded in the window @@ -5988,6 +7646,140 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * output was a_stale + b_fresh). */ svc_handle_map_add(); + /* + * r336/r34x: honor every host-semaphore acquire (C56F SEM_EXECUTE + * op 0/2/3/4/5) captured for THIS ring entry before any of its work + * runs, so a copy cannot execute ahead of the dependency its push + * waits on (the streams2 readback-before-scale class). Decision + * ladder per wait: + * 1. saved value satisfies the op's condition -> pass; + * 2. the entry's own release, when its method offset precedes the + * acquire's (release-then-acquire, channel.c crossEngineSync), + * counts as the candidate value - the release method has + * already executed by the time the acquire is evaluated; + * an acquire that precedes its own trailing tracking release + * does NOT count it (nothing has installed that value yet); + * 3. drain the pending-translation release queue (releases OUR + * machinery is merely late about) and re-read once; + * 4. still unsatisfied -> hold the entry (release withheld through + * the op-pending machinery) for at most gsp_sema_wait_hold_ms + * per episode, then fail open loudly ('give-up': the value + * cannot be installed by firmware alone - it sits behind an + * upstream opdef/operand hold - so the old pass-through + * behaviour is safer than parking the channel forever). + * Unreadable semaphore: fail-open (the pre-r336 behaviour). + * The pass/hold/give-up decisions are logged as 'sema-wait' rows + * (op, payload, observed value, decision). + */ + if (svc_wait_n && gsp_sema_wait_gate) { + struct svc_wait_hold *gh; + bool hold = false; + u32 wi; + /* + * Kill switch: a wait that never satisfies (a counter our + * emulation does not advance the way the driver expects) + * would otherwise park entries until the sweep expires each + * one. After 2048 holds in a boot the gate fails open + * permanently (the pre-r336 behaviour) and says so once. + */ + if (svc_wait_held > 2048) { + if (svc_wait_log < 4096) { + svc_wait_log++; + pr_info("sema-wait: gate disabled after %llu holds (fail-open)\n", + svc_wait_held); + } + svc_wait_pass++; + goto svc_wait_done; + } + gh = svc_wait_hold_get(ch_h, true); + for (wi = 0; wi < svc_wait_n; wi++) { + const struct svc_wait_op *w = &svc_waits[wi]; + u32 cur = 0, waited_ms = 0; + bool ok = true, same_rel = false, pend = false; + + if (gh) + waited_ms = jiffies_to_msecs(jiffies - gh->since); + if (svc_sem_read(vgpu, w->va, &cur)) + continue; /* unreadable: fail-open */ + ok = svc_wait_satisfied(cur, w->op, w->payload); + if (!ok && svc_wait_rel_have && + svc_wait_rel_va == w->va && + svc_wait_rel_off < w->off) { + same_rel = true; + ok = svc_wait_satisfied(svc_wait_rel_payload, + w->op, w->payload); + } + if (!ok) { + svc_apply_pending(vgpu); + if (!svc_sem_read(vgpu, w->va, &cur)) + ok = svc_wait_satisfied(cur, w->op, + w->payload); + } + if (ok) { + if (svc_wait_pass_log < 256) { + svc_wait_pass_log++; + pr_info("sema-wait: pass ch=0x%x va=0x%llx op=%u pay=%u cur=%u same_rel=%u waited_ms=%u -> ops run this pass\n", + ch_h, w->va, w->op, w->payload, + cur, same_rel, waited_ms); + } + continue; + } + pend = svc_wait_pend_release(w->va); + if (waited_ms >= gsp_sema_wait_hold_ms) { + /* + * Bounded episode expired: fail open for the + * WHOLE entry (one row, then the ops run). A + * per-wait continue here would ping-pong + * with a second unsatisfied wait: the first + * gave up and reset the shared clock before + * the second was judged, and the pair would + * hold forever 100 ms at a time. + */ + svc_wait_giveup++; + if (svc_wait_log < 2048) { + svc_wait_log++; + pr_info("sema-wait: give-up ch=0x%x va=0x%llx op=%u pay=%u cur=%u pend=%u waited_ms=%u -> fail-open (cannot be satisfied by firmware alone; ops run)\n", + ch_h, w->va, w->op, w->payload, + cur, pend, waited_ms); + } + if (gh) + gh->active = false; + hold = false; + svc_wait_pass++; + goto svc_wait_done; + } + hold = true; + if (svc_wait_log < 512) { + svc_wait_log++; + pr_info("sema-wait: hold ch=0x%x va=0x%llx op=%u pay=%u cur=%u same_rel=%u pend=%u waited_ms=%u -> entry held (acquire not satisfied)\n", + ch_h, w->va, w->op, w->payload, cur, + same_rel, pend, waited_ms); + } + break; /* one unsatisfied wait holds the entry */ + } + if (hold) { + svc_wait_held++; + svc_wait_hold_h = ch_h; + svc_wait_hold_idx = entry_idx; + svc.op_tr_pending = 1; + return; + } + /* + * r484: the gate just PASSED for an entry it previously held - + * the dependency is installed now, so this pass's ops must + * re-run even if the once-only bitmap says they already did + * (they ran against the pre-dependency state). + */ + if (ch_h == svc_wait_hold_h && + entry_idx == svc_wait_hold_idx) { + svc_wait_repass = true; + svc_wait_hold_idx = 0xffffffffu; + } + svc_wait_pass++; + if (gh) + gh->active = false; + } +svc_wait_done: for (i = 0; i < n_ops; i++) { struct svc_memset_op *m = &ops[i]; u32 seq = ops_base + i; @@ -6000,8 +7792,21 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * ran; re-running a CE copy would read a staging buffer the * client has since refilled and overwrite finished data. */ - if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) { + if (!svc_wait_repass && ops_done && seq < 64 && + ((*ops_done >> seq) & 1)) { svc.ops_once_skipped++; + /* + * r456: this op already ran in an earlier pass. Any + * staging snapshot a later re-decode pass re-captured + * for it (the capture sites run on every decode) is + * a leak that the r444 APPLY-free used to sweep; with + * that free removed (r456 keeps the snapshot alive + * across retries) drop it here so it can never serve + * a re-push of the ring slot. + */ + if (m->type == 1 && m->bytes <= SVC_COPY_SNAP_MAX) + svc_copy_snap_free(m->dst, m->src, m->bytes, + ch_h, entry_idx, pass_hash); continue; } @@ -6014,11 +7819,13 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, rc = svc_copy(vgpu, m->dst, m->src, m->bytes, m->phys, m->aperture, m->src_phys, m->src_aperture, - &ce_done); + &ce_done, m->rows, m->dpitch, + m->spitch, ch_h, entry_idx, + pass_hash); else rc = svc_fill(vgpu, m->dst, m->value, m->elem, m->count, m->phys, m->aperture, - &ce_done); + &ce_done, m->rows, m->dpitch); /* * CEAUDIT (2026-10-18 r220): per-op audit of the CE * data path - the software executor of every guest @@ -6075,9 +7882,11 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, */ if (rc == SVC_COPY_FORWARDED) continue; - if (rc == 0) + if (rc == 0) { svc.memsets_done++; - else if (rc == -ENOENT) + /* r507a: a success re-arms the failure window. */ + svc_opfail_clear(m->dst); + } else if (rc == -ENOENT) svc.memsets_nomap++; else svc.memsets_fail++; @@ -6107,7 +7916,9 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, */ if (rc != 0 && rc != SVC_COPY_FORWARDED && (m->bytes >= (256u << 10) || - (m->type == 1 && m->dst >= SVC_VAS_FLOOR))) + (m->type == 1 && m->dst >= SVC_VAS_FLOOR) || + (m->type == 0 && !m->phys && m->dst >= SVC_VAS_FLOOR)) && + !svc_opfail_latched(m->dst)) svc.op_tr_pending = 1; if (gsp_sema_dbg && m->bytes == 256 && svc.sem256_log < 64) { svc.sem256_log++; @@ -6118,7 +7929,7 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, } if (gsp_sema_dbg && ((rc == 0 && svc.memsets_done <= gsp_sema_ops_log) || - (rc != 0 && svc.memsets_fail + svc.memsets_nomap <= 64))) + (rc != 0 && svc.memsets_fail + svc.memsets_nomap <= 2048))) pr_info("sema-svc: %s dst=0x%llx value=0x%llx elem=%u count=%u bytes=%llu phys=%u ap=%u src=0x%llx srcphys=%u srcap=%u rc=%d h=0x%x\n", m->type == 1 ? "memcopy" : "memset", m->dst, m->value, m->elem, m->count, @@ -6136,7 +7947,7 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * read-back signature: always keep room for it past the * general trace budget so the DtoH's src/dst/rc is on record. */ - if ((svc.op_trace_log < 160 && + if ((svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && (time_before(jiffies, svc.op_trace_until) || m->dst >= SVC_VAS_FLOOR || (m->type == 1 && !m->src_phys && m->src >= SVC_VAS_FLOOR))) || @@ -6457,7 +8268,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, wrc = svc_sem_write(vgpu, h, sem_va, payload); if (!wrc) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-apply h=0x%x va=0x%llx pay=%u cur=%u -> pool rescue\n", @@ -6531,6 +8342,15 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, return; } nv = inc ? cur + payload : payload; + /* + * r378: the slot PA was learned from the release VA (or registered + * from the client's pagemap) earlier; re-validate it against the + * current pin before the 4-byte release lands, so a slot the guest + * has since freed/re-homed is never written through the stale page. + */ + if (sem_va && !gsp_sema_write_guard(sem_va, pa, SVC_APERTURE_SYS, + "counter-rel")) + return; if (svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, &nv, sizeof(nv))) { if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; @@ -6542,7 +8362,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, svc.releases_applied++; svc.counter_rel_applied++; svc.gr_rel_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-apply h=0x%x pa=0x%llx pay=%u cur=%u -> written\n", @@ -6625,7 +8445,7 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu, svc.releases_applied++; svc.counter_rel_applied++; svc.gr_rel_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-flush h=0x%x pa=0x%llx pay=%u cur=%u -> written\n", @@ -6964,6 +8784,13 @@ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va) pr_info("sema-chan: release learned h=0x%x pa=0x%llx (from release va=0x%llx)\n", h, pa & ~0x3ULL, sem_va); } + /* + * r378: keep the VA the slot was learned from so a + * late write to release_pa (the stale-recover zero) + * can re-validate the target through + * gsp_sema_write_guard before it lands. + */ + svc.chan[i].release_va = sem_va; break; } } @@ -6996,7 +8823,7 @@ static bool svc_rel_apply_906f(struct nv_vgpu_device *vgpu, cur = (op == HOST_SEM_OP_REDUCTION) ? cur + payload : payload; if (!svc_sem_write(vgpu, ch->h_object, va, cur)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } } @@ -7024,7 +8851,7 @@ static bool svc_rel_apply_reportsem(struct nv_vgpu_device *vgpu, svc.pushes_pending++; if (!svc_sem_write(vgpu, ch->h_object, va, payload)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } return false; @@ -7051,7 +8878,17 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, inc && inc_apply); return false; } - if (pending || inc_apply) { + /* + * r506w: generic absolute releases are forward-only by VALUE, whichever + * pass discovers the gap. The recorded pending bit came from the + * decode's read of the release word, and that read can be blinded by a + * stale/crossed landing (r506 witness: a withheld tracker release went + * to the flush with pending=0 while the fresh read still showed 0, and + * the release was then dropped by the counter fallback below - the + * client's tracker never advanced and its stream ground at the 10 s + * bound per push). Increments keep the one-shot claim gate (`inc_apply`). + */ + if (pending || inc_apply || (!inc && payload > cur)) { svc.pushes_pending++; if (inc) cur += payload; @@ -7069,14 +8906,24 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, /* handled by the exact-slot path */ } else if (!svc_sem_write(vgpu, ch->h_object, sem_va, cur)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } } - if (ch->engine_type == 0) - svc_counter_learn(vgpu, ch->h_object, sem_va); - svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, - inc && inc_apply); + /* + * r506w: the exact-slot counter fallback is for COUNTER VAs only. Its + * `release_pa ?: counter_pa` fallback writes (and forward-only-skips + * against) the channel's REGISTERED slot, which belongs to a different + * word - the r506 witness: releases to the tracker VA 0x20480fef0 were + * skipped against the polluted PUT counter (cur=96) and lost. For a + * non-counter VA the generic write above is the delivery path. + */ + if (svc_counter_va(sem_va)) { + if (ch->engine_type == 0) + svc_counter_learn(vgpu, ch->h_object, sem_va); + svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, + inc && inc_apply); + } /* * Stall-class fix (R2): when the decode said this absolute release is * genuinely pending and every write attempt above failed without @@ -7199,6 +9046,84 @@ static void svc_fwd_rel_direct_note(u32 h, u64 va, u32 payload) static u32 svc_gr_qmd_seen; static u64 svc_gr_nocap_log; +/* + * r383: has a launch with this push VA ever been captured? Scans the live + * candidate ring, the recent-capture history and the r401 session-wide + * (push_va, hash) seen table. Used by the replayed-QMD dispatch below: a + * replay whose push was never captured is a launch that would otherwise be + * silently dropped (the streams2 second-vadd class); a replay of a + * captured/offered push is the exactly-once no-op. The hash is required + * for the seen table so a later, DIFFERENT push at the same VA (a loop + * iteration: same entry words, new content) is never misclassified as + * seen - the caller only consults this on a true replay anyway. + */ +static bool svc_gr_cand_push_seen(u64 push_va, u64 hash) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + svc_gr_cand_ring[i].push_va == push_va) + return true; + for (i = 0; i < SVC_GR_CAP_HIST; i++) + if (svc_gr_cap_hist[i] == push_va) + return true; + return svc_gr_seen_lookup(push_va, hash); +} + +/* + * r391f: once-only identity for GRCE inline writes. A push's + * LOAD_INLINE_DATA runs author bytes into the vidmem shadow; a re-decode of + * the same push (replay/rescue/rehold) must not re-run them - the target + * page may have been superseded in between (victory_reduce: the client's + * cudaMemset inline write re-zeroed the accumulator after the launch's + * copy-out, the sum=0 class). Keyed by (push_va, push content hash, dst, + * bytes): the release payload inside every push makes the hash unique per + * push, so a new client's identical-looking write never collides. Session + * scoped (cleared by svc_session_flush). Service-thread only. + * + * Sizing (r398): the ring must outlive the gap between a push's first + * execution and its next re-decode. The observed failing case had 76 + * inline writes between the two, which evicted a 64-slot table and let + * the memset re-run; 8192 slots (256 KB) gives a ~100x margin over the + * busiest observed burst. + */ +#define SVC_INLINE_ONCE_SLOTS 8192 +static struct { + u64 push_va; + u64 hash; + u64 dst; + u32 bytes; +} svc_inline_once[SVC_INLINE_ONCE_SLOTS]; +static u32 svc_inline_once_next; + +static bool svc_inline_once_seen(u64 push_va, u64 hash, u64 dst, u32 bytes) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_INLINE_ONCE_SLOTS; i++) + if (svc_inline_once[i].push_va == push_va && + svc_inline_once[i].hash == hash && + svc_inline_once[i].dst == dst && + svc_inline_once[i].bytes == bytes) + return true; + return false; +} + +static void svc_inline_once_store(u64 push_va, u64 hash, u64 dst, u32 bytes) +{ + u32 i = svc_inline_once_next++ % SVC_INLINE_ONCE_SLOTS; + + svc_inline_once[i].push_va = push_va; + svc_inline_once[i].hash = hash; + svc_inline_once[i].dst = dst; + svc_inline_once[i].bytes = bytes; +} + static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, struct svc_push_state *st, u32 addr, u32 count, const u8 *data, const u8 *push, u32 push_len, @@ -7281,13 +9206,52 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, pr_info("sema-gr: inline recs n=%u dup_dst=%u\n", n, dup); } - if (gsp_phys_gr_forward_enabled() && trusted && !replay) - svc_gr_cand_capture(ch, st, recs, n, push, push_len, - st->push_va); - else if (!replay || !gsp_phys_gr_forward_enabled()) + if (gsp_phys_gr_forward_enabled()) { + u64 push_hash = (push && push_len) ? + svc_push_hash(push, push_len) : 0; + + if (trusted && !replay) { + svc_gr_cand_capture(ch, st, recs, n, push, + push_len, st->push_va); + } else if (trusted && replay && + !svc_gr_cand_push_seen(st->push_va, + push_hash)) { + /* + * r383 (streams2 second-vadd class): a + * complete QMD first seen on a replay pass + * fell through BOTH arms below - the + * forward path skips replays and the + * bridge arm needs !replay - so the + * launch was silently dropped (zero + * engine writes, client read stale/zero + * outputs). The push VA was never + * captured, so capture it now (trusted + * content only, as the first-sight path); + * a replay of an already captured/offered + * launch stays the exactly-once no-op. + */ + pr_info("sema-gr: REPLAY-CAPTURE h=0x%x push=0x%llx (complete QMD first seen on a replay pass; no capture on record)\n", + ch->h_object, st->push_va); + svc_gr_cand_capture(ch, st, recs, n, push, + push_len, st->push_va); + } else if (replay && svc_gr_nocap_log < 64) { + svc_gr_nocap_log++; + pr_info("sema-gr: LAUNCH-SKIP h=0x%x push=0x%llx (replay; already captured=%d trusted=%d)\n", + ch->h_object, st->push_va, + svc_gr_cand_push_seen(st->push_va, + push_hash), + trusted); + } else if (!replay) { + gsp_cuda_bridge_launch(vgpu, ch->h_object, + st->qmd, + st->qmd_dev_va, + st->push_va, recs, n); + } + } else if (!replay) { gsp_cuda_bridge_launch(vgpu, ch->h_object, st->qmd, st->qmd_dev_va, st->push_va, recs, n); + } st->qmd_n = 0; st->qmd_bits = 0; st->qmd_have = 0; @@ -7296,6 +9260,89 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } } +static u64 svc_pass_hash; /* r391f: this decode pass's push content hash */ + +/* + * r481 RISK-REDUCTION diagnostic (default 0): dump the full decoded method + * stream of the first few large GR-class pushes (len >= 1024) so the launch + * push's acquire/release method groups (or their absence) are visible + * verbatim. The streams_events class evidence shows the st2 vadd's copy-in + * running ~115 ms BEFORE its stream's H2D soft copies; the ordering the + * driver relied on is only provable by reading the push's own methods. + * Bounded: gsp_sema_push_raw dumps, at most once per push VA. + */ +unsigned gsp_sema_push_raw; +static u64 svc_pushraw_last_va; +static u32 svc_pushraw_n; + +static void svc_push_raw_dump(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, gi = 0; + + if (!gsp_sema_push_raw || svc_pushraw_n >= gsp_sema_push_raw) + return; + if (len < 512 || len > 8192) + return; + if (push_va == svc_pushraw_last_va) + return; + svc_pushraw_last_va = push_va; + svc_pushraw_n++; + pr_info("sema-pushraw: BEGIN h=0x%x va=0x%llx len=%u n=%u\n", + h_object, push_va, len, svc_pushraw_n); + while (off + 4 <= len && gi < 160) { + u32 hdr, op, cnt, subch, addr, v[4] = {0,0,0,0}, k, n; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 0x7; + cnt = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + addr = (hdr & 0x1fff) << 2; + n = cnt > 4 ? 4 : cnt; + for (k = 0; k < n && off + 8 + 4 * k <= len; k++) + memcpy(&v[k], buf + off + 4 + 4 * k, 4); + pr_info("sema-pushraw: [%u] off=0x%x op=%u sub=%u addr=0x%x cnt=%u v0=%08x v1=%08x v2=%08x v3=%08x\n", + gi, off, op, subch, addr, cnt, + v[0], v[1], v[2], v[3]); + if (op == 3 && cnt) { + off += 4 + cnt * 4; + gi++; + continue; + } + if (!cnt) { + gi++; + off += 4; + continue; + } + if (op == 1) { + off += 4 + cnt * 4; + gi++; + continue; + } + { + u32 p; + bool found = false; + + for (p = off + 4; p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 1 && + ((h2 >> 16) & 0xfff)) { + off = p; + found = true; + break; + } + } + if (!found) + off = len; + gi++; + } + } + pr_info("sema-pushraw: END h=0x%x va=0x%llx consumed=0x%x/%u\n", + h_object, push_va, off, len); +} + static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 len, u64 page_pa, bool allow_ops, struct svc_channel *ch, u32 entry_idx, @@ -7303,6 +9350,35 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u64 *ops_done) { struct svc_push_state st; + + /* r391f: the inline-write once-only key needs the push's identity. */ + svc_pass_hash = svc_push_hash(buf, len); + + /* + * r316: pending releases record the channel they were decoded for; + * the verified pool resolver is channel-keyed (see svc_pending_add + * and the matmul r322 starved/hung note). + */ + svc.cur_ch = ch->h_object; + /* + * r34x: the acquire-wait capture is PER RING ENTRY. It used to be + * reset only at SET_OBJECT, so a decode pass over a later entry (or + * a set of entries, since a held entry is re-decoded after the + * sweep walks ahead) leaked its acquire into an earlier entry's + * release-site gate: r340 held a D2H copy entry that carries no + * acquire at all on a later launch's 'wait for the next tracking + * value' (need=85 while the held entry releases 84) - a circular + * wait only our own withheld release could ever satisfy, so every + * hold expired via the 10 s fallback (wait=4542 etc.). Reset here, + * per entry; SET_OBJECT still resets at the push boundary, and the + * re-decode re-captures the same waits, so held entries re-evaluate + * identically. + */ + svc_wait_n = 0; + svc_wait_rel_va = 0; + svc_wait_rel_payload = 0; + svc_wait_rel_off = 0; + svc_wait_rel_have = false; /* * Static for the same reason as svc_handle_chan_dump: a 32-entry ops * array is a ~2 KB frame, and this is called from the single service @@ -7331,6 +9407,13 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, allow_ops = false; } + /* + * r481: verbatim method-stream dump for large GR-class pushes + * (default 0 = off). See svc_push_raw_dump. + */ + if (gsp_sema_push_raw && ch && !ch->engine_type && len >= 512) + svc_push_raw_dump(buf, len, push_va, ch->h_object); + memset(&st, 0, sizeof(st)); st.push_va = push_va; /* @@ -7356,14 +9439,34 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc_fwd_defer_armed = false; svc.pages_decoded++; /* Diagnostic: name every push decoded in the post-completion window. */ - if (svc.op_trace_log < 160 && + if (svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && time_before(jiffies, svc.op_trace_until)) { svc.op_trace_log++; pr_info("sema-pushtrace: h=0x%x eng=%u va=0x%llx len=%u allow=%d trusted=%d\n", ch->h_object, ch->engine_type, push_va, len, allow_ops, trusted); } - if (gsp_sema_dbg && ch->engine_type && svc_dbg_streams < 96) { + /* + * fail4: de-windowed decode trace - every push the service looks at, + * bounded per session. The windowed/identified-channel diagnostics + * above can miss a whole client's push stream (unidentified channel, + * no bridge completion to arm the window); this record cannot. + */ + /* + * fail4 r302: with sema_dbg on the boot-time driver init spends the + * 512-line budget long before the run under diagnosis, leaving the + * interesting push stream (a later client's copies) invisible. The + * debug flag raises the bound so one boot covers init + run. + */ + if (svc.pushtrace_all < (gsp_sema_dbg ? 32768 : 512)) { + svc.pushtrace_all++; + pr_info("sema-pushall: h=0x%x eng=%u va=0x%llx pa=0x%llx len=%u allow=%d trusted=%d first8=%08x %08x\n", + ch->h_object, ch->engine_type, push_va, page_pa, len, + allow_ops, trusted, + len >= 4 ? *(const u32 *)(buf + 0) : 0, + len >= 8 ? *(const u32 *)(buf + 4) : 0); + } + if (gsp_sema_dbg && svc_dbg_streams < 96) { svc_dbg_streams++; pr_info("sema-dbg: stream h=0x%x eng=%u va=0x%llx len=%u allow=%d w=%08x %08x %08x %08x %08x %08x %08x %08x\n", ch->h_object, ch->engine_type, push_va, len, allow_ops, @@ -7411,28 +9514,195 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, addr = (hdr & 0x1fff) << 2; /* + * fail4: name every big method-data run and every run whose + * address is the LOAD_INLINE_DATA method (any op/subch - the + * encoding of the 64 KB H2D upload is what we are pinning). + */ + if (gsp_sema_dbg && len >= 8192 && count >= 1024 && + svc.bigrun_log < 128) { + svc.bigrun_log++; + pr_info("sema-bigrun: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x have_out=%u out=0x%llx line=%u lcnt=%u w=%08x %08x\n", + ch->h_object, push_va, off, hdr, op, count, subch, + addr, st.have_out, st.offset_out, st.line_length, + st.gr_line_count, + off + 8 <= len ? *(const u32 *)(buf + off + 4) : 0, + off + 12 <= len ? *(const u32 *)(buf + off + 8) : 0); + } + if (gsp_sema_dbg && len >= 8192 && + (addr == GRCE_M_LOAD_INLINE_DATA || addr == 0x1b8) && + svc.lid_hdr_log < 64) { + svc.lid_hdr_log++; + pr_info("sema-lid-hdr: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x trusted=%d allow=%d\n", + ch->h_object, push_va, off, hdr, op, count, subch, + addr, trusted, allow_ops); + } + + /* * GRCE LOAD_INLINE_DATA: a data run (opcode 3) that carries * the DMA source bytes in the push itself. Run the write to * the destination the preceding methods selected. */ if (op == 3 && subch == 1 && addr == GRCE_M_LOAD_INLINE_DATA && count && off + 4 + count * 4 <= len) { - if ((trusted || allow_ops) && !gsp_sema_skip_ops && - st.have_out && st.line_length) { - u32 bytes = st.line_length * - (st.gr_line_count ? st.gr_line_count : 1); - u32 have = count * 4; + /* + * fail4 slot rule: in the UVM upload pushes each + * LOAD_INLINE_DATA run fills a whole 32 KB slot - its + * data runs from the header to the next run header + * (or to the next 0x8000 boundary when this is the + * last run), regardless of the count field. A compact + * push (no room for a following slot) keeps the + * count*4 reading (the small parameter banks). The + * old count*4 reading truncated the big runs to half + * their slot and left the tail to be misparsed as + * methods (the split-upload bug). + */ + bool slot_run = (off + 0x8000 < len); + u32 slot_end = off + 0x8000; + u32 slot_bytes = 0; + /* + * fail4: the bridge's argument-block search needs the + * run's START as its destination; the cursor advance + * below must not leak into the rec (an end-of-run + * dst_va makes the QMD param patch miss and the + * engine never consumes the launch). + */ + u64 run_dst = st.offset_out; + + if (slot_run) { + u32 p; + + for (p = off + 4 + count * 4; + p + 4 <= slot_end && p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 3 && + ((h2 >> 13) & 0x7) == 1 && + ((h2 & 0x1fff) << 2) == + GRCE_M_LOAD_INLINE_DATA && + ((h2 >> 16) & 0xfff)) { + slot_end = p; + break; + } + } + slot_bytes = slot_end - (off + 4); + if (slot_bytes > 0x8000 - 4) + slot_bytes = 0x8000 - 4; + } + /* + * r391f once-only (the sum=0 class): the same + * inline write must not run twice for one push + * identity. The ops are gated by allow_ops + * (entry_replay clears it), but the inline-write + * gate `trusted || allow_ops` re-ran the write on + * every re-decode (a replay is trusted by + * construction): victory_reduce's + * cudaMemset(d_sum,0,8) is exactly such an inline + * write (observed: `inline write + * dst=0x7845ef000000 -> pa=0x7c00000 bytes=8 + * data8=0` firing 86 us before the D2H readback, + * zeroing the accumulator the launch had just + * written - the client then read zeros). A coarse + * !replay gate is wrong (a push whose first pass + * was untrusted must still author its bytes on the + * later trusted pass); key the write by + * (push_va, push content hash, dst, bytes) - a new + * push's release payload makes the hash unique - + * and skip only a genuine duplicate. The table is + * session-scoped (a new client starts clean). + */ + { + u32 bytes = slot_run ? slot_bytes : + st.line_length * + (st.gr_line_count ? + st.gr_line_count : 1); + u32 have = slot_run ? slot_bytes : count * 4; if (bytes > have) bytes = have; - svc_grce_inline_write(vgpu, st.offset_out, - buf + off + 4, bytes, - ch->h_object); - } else if (gsp_sema_dbg && svc.grce_log < 16) { - svc.grce_log++; - pr_info("sema-grce: inline data not run (trusted=%d allow=%d have_out=%d line=%u) h=0x%x\n", - trusted, allow_ops, st.have_out, - st.line_length, ch->h_object); + /* + * r391f once-only (the sum=0 class): the same + * inline write must not run twice for one push + * identity. The ops are gated by allow_ops + * (entry_replay clears it), but the + * inline-write gate `trusted || allow_ops` + * re-ran the write on every re-decode (a replay + * is trusted by construction): victory_reduce's + * cudaMemset(d_sum,0,8) is exactly such an + * inline write (observed: `inline write + * dst=0x7845ef000000 -> pa=0x7c00000 bytes=8 + * data8=0` firing 86 us before the D2H + * readback, zeroing the accumulator the launch + * had just written - the client then read + * zeros). A coarse !replay gate is wrong (a push + * whose first pass was untrusted must still + * author its bytes on the later trusted pass); + * key the write by (push_va, push content hash, + * dst, bytes) - the release payload inside every + * push makes the hash unique per push - and skip + * only a genuine duplicate. Session-scoped: a new + * client starts clean. + */ + if ((trusted || allow_ops) && !gsp_sema_skip_ops && + st.have_out && st.line_length && bytes && + !svc_inline_once_seen(push_va, svc_pass_hash, + st.offset_out, + bytes)) { + svc_inline_once_store(push_va, + svc_pass_hash, + st.offset_out, + bytes); + svc_grce_inline_write(vgpu, st.offset_out, + buf + off + 4, + bytes, + ch->h_object); + /* + * fail4: name the header that follows + * each big LID run - the split 64 KB + * upload's other payload carriers sit + * exactly there. + */ + if (gsp_sema_dbg && len >= 4096 && + svc.next_hdr_log < 24) { + u32 noff = off + 4 + count * 4; + u32 nh = 0, nw = 0; + + if (noff + 4 <= len) + memcpy(&nh, buf + noff, 4); + if (noff + 8 <= len) + memcpy(&nw, buf + noff + 4, + 4); + svc.next_hdr_log++; + pr_info("sema-next-hdr: h=0x%x push=0x%llx after=0x%x cnt=%u nxt_hdr=0x%08x nxt_w0=%08x cursor=0x%llx\n", + ch->h_object, push_va, + off, count, nh, nw, + st.offset_out); + } + } else if (gsp_sema_dbg && svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: inline data not run (trusted=%d allow=%d have_out=%d line=%u) h=0x%x\n", + trusted, allow_ops, st.have_out, + st.line_length, ch->h_object); + } + /* + * fail4 (r526): hardware semantics - a + * LOAD_INLINE_DATA run advances the destination + * cursor by the bytes it carries whether or not + * the once-only gate let the write run. Without + * this, the consecutive runs of a split upload + * all land at the first destination (the old + * RAWA bug), and worse: on a re-decode pass where + * the gate suppresses the FIRST run, the cursor + * stays at the first run's destination and the + * SECOND run executes there - its data (the + * buffer's bytes 8..16) overwriting offset 0 + * (the observed tgemm XDIFF: X[0..1] <- + * X[2..3], deterministic, every pass). The gate + * suppresses the write, not the method stream's + * positional semantics. + */ + if (bytes) + st.offset_out += bytes; } /* * Record the destination and the payload bytes for @@ -7442,21 +9712,43 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * kernel arguments in the QMD parameter bank, and the * destination VA may not be readable later. */ - if (st.have_out && st.offset_out) { + if (st.have_out && run_dst) { struct svc_grce_rec *rec = &svc_grce_dst_ring[svc_grce_dst_ring_next % SVC_GRCE_DST_MAX]; - rec->dst_va = st.offset_out; - rec->len = min_t(u32, count * 4, + rec->dst_va = run_dst; + rec->len = min_t(u32, + slot_run ? slot_bytes : count * 4, GSP_BRIDGE_INLINE_BYTES); memcpy(rec->data, buf + off + 4, rec->len); svc_grce_dst_ring_next++; } - off += 4 + count * 4; + /* + * fail4 slot rule: hop to the next slot's run header + * instead of walking the slot's raw tail as methods. + */ + off += slot_run ? (slot_end - off) : 4 + count * 4; continue; } + /* + * fail4: name the moderate op=1 method runs on big pushes - + * the split 64 KB upload's payload carriers beyond the LID + * class land here (their addr selects the write port). + */ + if (gsp_sema_dbg && len >= 4096 && op == 1 && + count >= 16 && count <= 1023 && svc.mystery_log < 48) { + svc.mystery_log++; + pr_info("sema-mystery: h=0x%x push=0x%llx off=0x%x hdr=0x%08x cnt=%u subch=%u addr=0x%x have_out=%u out=0x%llx line=%u lcnt=%u w=%08x %08x %08x\n", + ch->h_object, push_va, off, hdr, count, subch, + addr, st.have_out, st.offset_out, + st.line_length, st.gr_line_count, + off + 8 <= len ? *(const u32 *)(buf + off + 4) : 0, + off + 12 <= len ? *(const u32 *)(buf + off + 8) : 0, + off + 16 <= len ? *(const u32 *)(buf + off + 12) : 0); + } + if (op == 4 || op == 7) { /* Header-only methods (immediate data in the header * for op 4; end-of-push segment for op 7). */ @@ -7521,6 +9813,13 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * held INC-shaped words queued a bogus LAUNCH_DMA). * No release is decoded from these encodings. */ + if (gsp_sema_dbg && len >= 8192 && count >= 1024 && + svc.skiprun_log < 64) { + svc.skiprun_log++; + pr_info("sema-skiprun: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x trusted=%d allow=%d\n", + ch->h_object, push_va, off, hdr, op, + count, subch, addr, trusted, allow_ops); + } off += 4 + count * 4; continue; } @@ -7587,6 +9886,12 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, /* SET_OBJECT on the channel subchannel: a push starts */ n_ops = 0; memset(&st, 0, sizeof(st)); + /* r336/r34x: new push, new acquire set + release mirror */ + svc_wait_n = 0; + svc_wait_rel_va = 0; + svc_wait_rel_payload = 0; + svc_wait_rel_off = 0; + svc_wait_rel_have = false; off += 4 + count * 4; continue; } @@ -7605,6 +9910,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc.pushes_seen++; st.rel_va = va; st.rel_payload = payload; + svc_wait_rel_va = va; + svc_wait_rel_payload = payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; svc_rel_va_seen = va; svc_rel_count++; @@ -7612,7 +9921,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * REPORT_SEM comment below. */ if (allow_ops && n_ops && !gsp_sema_skip_ops) { svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -7657,7 +9967,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, or.va = va; or.payload = payload; or.entry_idx = entry_idx; - if (svc_fwd_defer_arm()) { + if (svc_fwd_defer_arm() && + !svc.op_tr_pending) { svc_opdef_drop(vgpu, ch, entry_idx); if (svc_fwd_rel_n < SVC_FWD_REL_MAX) { @@ -7702,6 +10013,42 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, va, payload, op); } } + } else if (op != HOST_SEM_OP_RELEASE && + va >= SEMA_VA_LO && va < SEMA_VA_HI) { + /* + * r336/r34x: C56F SEM_EXECUTE acquire ops + * (0 = ACQUIRE/equal, 2 = strict GEQ, + * 3 = circular GEQ, 4 = AND, 5 = NOR). + * Core CUDA starts its compute-channel + * pushes with one of these to wait for a + * progress counter (observed: + * [LO=0x0400ffd0][HI=2][pay=4][0][0x1003] on + * the D2H readback push - ACQ_CIRC_GEQ on + * the channel counter 0x20400ffd0). + * svc_run_ops honors every captured acquire + * of this entry before running its work, so + * a copy cannot execute ahead of the + * dependency it waits on (the streams2 + * readback-before-scale class). Capturing + * into the per-entry list (instead of a + * single last-wait global) is what stops a + * later entry's acquire from gating an + * earlier entry (the r340 over-hold). + * PAYLOAD_SIZE 64BIT (execute bit 24) is + * captured on the low word only; every + * observed CUDA acquire is 32-bit. + */ + if (svc_wait_n < SVC_WAIT_MAX) { + struct svc_wait_op *w = + &svc_waits[svc_wait_n++]; + + w->va = va; + w->payload = payload; + w->op = (u8)op; + w->off = off; + } else { + svc_wait_ovf++; + } } off += 4 + count * 4; continue; @@ -7751,6 +10098,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc.pushes_seen++; st.rel_va = va; st.rel_payload = st.rs_payload; + svc_wait_rel_va = va; + svc_wait_rel_payload = st.rs_payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; svc_rel_va_seen = va; svc_rel_count++; @@ -7771,7 +10122,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, */ if (allow_ops && n_ops && !gsp_sema_skip_ops) { svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -7886,7 +10238,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * operand state, so one run covers both the * C5B5 and the GRCE copy shapes. */ - if (ch->h_object && gsp_sema_dbg && + if (gsp_sema_dbg && svc.xfer_raw_log < 48) { u32 dl = len < 192 ? len : 192; @@ -7928,11 +10280,61 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.have_sem = 1; break; case CE_M_SET_REMAP_CONST_A: + /* + * r343 FILL instrument: a fill-shaped LAUNCH_DMA + * (REMAP_ENABLE + destination PITCH, OFFSET_OUT and + * no OFFSET_IN) needs this constant to build its op. + * The 8MB-memset class (memcpy_roundtrip all-bad, + * victory_memset2d) shows fill-shaped launches with + * have_remap=0 that decode nothing and are consumed + * silently. Log every decode (bounded) so one run + * captures the real run shape even when the count + * guard below cannot use it. + */ + if (svc_remapconst_log < 256) { + svc_remapconst_log++; + pr_info("sema-remapconst: h=0x%x push=0x%llx off=0x%x count=%u v0=0x%x v1=0x%x v2=0x%x trusted=%d allow=%u\n", + ch->h_object, push_va, off, count, + v0, count >= 2 ? v1 : 0u, + count >= 3 ? v2 : 0u, trusted, allow_ops); + } if (count >= 3) { st.remap_a = v0; st.remap_b = v1; st.remap_comp = v2; st.have_remap = 1; + st.have_remap_a = 1; + st.have_remap_b = 1; + st.have_remap_c = 1; + } else if (count >= 1) { + /* + * r365: the driver also pushes the fill constants + * as SEPARATE count=1 methods (the CE + * page-table fills carry only SET_REMAP_CONST_A + * with A=0). Capture the individual parts but do + * NOT set have_remap: r365b proved that letting + * these pass the fill guard executes below-floor + * window fills (0x2000c0xxx, 0x204020000) whose + * svc_fill resolution landed in GUEST RAM and + * zeroed slab pages (guest kernel GPF in + * kmem_cache_alloc -> panic at ~75 s). The + * window fills only run behind gsp_sema_winfill + * with the correct resolution proven first. + */ + st.remap_a = v0; + st.have_remap_a = 1; + } + break; + case CE_M_SET_REMAP_CONST_B: + if (count >= 1) { + st.remap_b = v0; + st.have_remap_b = 1; + } + break; + case CE_M_SET_REMAP_COMPONENTS: + if (count >= 1) { + st.remap_comp = v0; + st.have_remap_c = 1; } break; case CE_M_SET_SRC_PHYS_MODE: @@ -7945,12 +10347,42 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, break; case CE_M_LINE_LENGTH_IN: st.line_length = v0; + /* + * r458 THE memset2d root cause: the driver encodes + * LINE_LENGTH_IN (0x418) and the adjacent LINE_COUNT + * (0x41c) as ONE count=2 method group - observed raw + * push bytes `20028106 00001000 00000100` = header + * (subch 4, count 2, addr 0x418), line length 4096, + * line count 0x100 = 256. Reading only v0 left + * line_count = 0, so the 2D-copy builder below took + * the single-row path: one 4 KB op per push, and a + * 4 MB cudaMemcpy2D delivered exactly one page per + * 1 MB block (memset2d probe: per-MB ok=4096, + * bad=12.5M/12.58M). The GRCE variant below has + * always decoded the count=2 form; the CE path did + * not. + */ + if (count >= 2) + st.line_count = v1; + break; + /* r402: 2D geometry (cudaMemset2D / cudaMemcpy2D). */ + case CE_M_LINE_COUNT: + st.line_count = v0; + break; + case CE_M_PITCH_IN: + st.pitch_in = v0 & 0xffffff; + st.have_pitch_in = 1; + break; + case CE_M_PITCH_OUT: + st.pitch_out = v0 & 0xffffff; + st.have_pitch_out = 1; break; case CE_M_LAUNCH_DMA: { u32 dma = v0; u32 transfer = DMA_DATA_TRANSFER_TYPE(dma); + u32 n_ops_before = n_ops; - if (gsp_sema_dbg && ch->h_object && + if (gsp_sema_dbg && svc.lda_log < (gsp_sema_ops_log > 48 ? gsp_sema_ops_log : 48)) { svc.lda_log++; pr_info("sema-lda: h=0x%x eng=%u push=0x%llx inj=%u allow=%u dma=0x%08x xfer=%u remap=%u dpitch=%u dstphys=%u srcphys=%u sem=%u out=0x%llx in=0x%llx line=%u have_out=%u have_in=%u have_remap=%u have_sem=%u ops=%u\n", @@ -7967,7 +10399,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* Diagnostic: full operand state in the post-completion * window (or for high device VAs), bounded. */ - if (svc.op_trace_log < 160 && ch->h_object && + if (svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && ch->h_object && (time_before(jiffies, svc.op_trace_until) || st.offset_out >= SVC_VAS_FLOOR || st.offset_in >= SVC_VAS_FLOOR)) { @@ -7991,7 +10423,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * bounded number of such launches so the operand * encoding (or its absence) is on record. */ - if (transfer != 0 && ch->h_object && gsp_sema_dbg && + if (transfer != 0 && gsp_sema_dbg && svc.xfer_raw_log < 48 && (!st.have_out || !st.line_length)) { u32 dl = len < 192 ? len : 192; @@ -8006,25 +10438,182 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, if (allow_ops && transfer != 0 && st.line_length && st.have_out) { - if (DMA_REMAP_ENABLE(dma) && st.have_remap && + /* + * r361: a fill-shaped transfer whose + * destination is PHYSICAL guest RAM (the + * driver zeroes freshly-allocated PTE / + * page-directory pages with exactly this + * push shape) needs no translation: the + * destination already IS the GPA. The old + * `offset_out >= SVC_VAS_FLOOR` guard + * compared the GPA against the VA floor + * and blocked every one of these, so the + * PD0 pages the GMMU walk later reads + * stayed stale and the first CUDA client + * of a fresh boot dropped operand pages + * (GRFILL-DROP). SEMA_PA_MAX bounds the + * range so a ramfill can only touch guest + * RAM; virtual engine-window fills + * (0x2040xxxxx) stay excluded. + */ + bool ramfill = DMA_DST_TYPE_PHYSICAL(dma) && + st.have_dst_mode && + st.dst_phys_mode == CE_APERTURE_SYS && + st.offset_out + (u64)st.line_length <= + SEMA_PA_MAX; + /* r365: vidmem-window fill (see gsp_sema_winfill). */ + bool winfill = gsp_sema_winfill && + st.offset_out >= SVC_VIDMEM_VA_LO && + st.offset_out < SVC_VIDMEM_VA_BASE + + svc_fb_shadow_size(vgpu); + + if (DMA_REMAP_ENABLE(dma) && + (st.have_remap || + (st.offset_out >= SVC_VAS_FLOOR) || + ramfill || winfill) && DMA_DST_MEMORY_LAYOUT_PITCH(dma) && n_ops < SVC_MAX_MEMSETS) { - u32 csz = (st.remap_comp >> 16) & 0x3; - u32 ndst = (st.remap_comp >> 24) & 0x3; + u32 csz = st.have_remap ? + ((st.remap_comp >> 16) & 0x3) : 0; + u32 ndst = st.have_remap ? + ((st.remap_comp >> 24) & 0x3) : 0; + u32 dst_x = st.have_remap ? + (st.remap_comp & 0x7) : 4; struct svc_memset_op *m = &ops[n_ops]; m->type = 0; m->dst = st.offset_out; - m->value = ndst >= 1 ? - (((u64)st.remap_b << 32) | - st.remap_a) : st.remap_b; - m->elem = (csz + 1) * (ndst + 1); - m->count = st.line_length; + /* + * r367: the driver's own memset push + * (channel_utils.c) proves the + * encoding: LINE_LENGTH_IN is the + * fill SIZE IN BYTES, and + * SET_REMAP_COMPONENTS selects + * COMPONENT_SIZE ([17:16], 0..3 -> + * 1/2/3/4 bytes) and + * NUM_DST_COMPONENTS ([25:24], n+1 + * components). elem = + * (csz+1)*(ndst+1) is the pattern + * width; count must therefore be + * line_length/elem - the old + * count=line_length wrote + * elem x too much (r365: a 383620- + * byte memset wrote 1534480 bytes, + * clobbering the bytes after it). + * The value comes from the mapped + * constants: X<-CONST_A (4) and + * Y<-CONST_B (5); with two + * components the pattern is + * B:A 64-bit. + */ + if (st.have_remap) + m->value = ndst >= 1 ? + (((u64)st.remap_b << 32) | + st.remap_a) : + (dst_x == 5 ? st.remap_b : + st.remap_a); + else + /* + * r357: a fill-shaped transfer + * (REMAP_ENABLE + DST PITCH, + * no source) to a CLIENT DEVICE + * VA whose value constant did + * not decode is a ZERO fill: + * the driver's page-table + * clears and cudaMemset(0) + * issue exactly this shape with + * no SET_REMAP_CONST method. + * Building no op left the + * guest's GPU page tables + * stale (the GMMU walk then + * missed freshly-mapped + * operand pages and the first + * CUDA client of every fresh + * boot ran with dropped + * operands). The >= floor + * guard keeps engine-window + * fills (0x2040xxxxx init + * clears) OUT of the executor: + * executing those clobbered + * emulated engine state and + * hung client init (r356). + */ + m->value = 0; + m->elem = st.have_remap ? + (csz + 1) * (ndst + 1) : 1; + if (m->elem > 1 && + st.line_length % m->elem) { + /* not pattern-divisible: exact + * byte count wins */ + m->elem = 1; + m->count = st.line_length; + } else { + m->count = st.line_length / + m->elem; + } + /* + * r402: a 2D fill (cudaMemset2D): + * LINE_COUNT rows of + * LINE_LENGTH_IN bytes at + * PITCH_OUT stride. A pitch WIDER + * than the row selects row mode; a + * flat fill carries pitch == row + * length (or no pitch method) and + * keeps the historical single-run + * path. + * + * r420c: LINE_COUNT is + * authoritative for the flat case + * too - cudaMemset2D with pitch == + * width (victory_memset2d, + * pitch=4096/w=4096) is CONTIGUOUS + * rows, so the single run must + * cover rows*line bytes. The + * historical path filled one row + * and the readback lost the rest + * (bad ~= 12.5M of 12.58M). + */ + if (st.line_count > 1) { + u32 fpitch = st.have_pitch_out ? + st.pitch_out : + st.line_length; + + if (fpitch > st.line_length) { + m->rows = + st.line_count; + m->dpitch = fpitch; + } else if (st.line_length % + m->elem) { + m->elem = 1; + m->count = + st.line_length * + st.line_count; + } else { + m->count = + (st.line_length / + m->elem) * + st.line_count; + } + } + if (m->rows > 1 && svc_pitch_log < 32) { + svc_pitch_log++; + pr_info("sema-2d: fill rows=%u dpitch=%u line=%u dst=0x%llx h=0x%x push=0x%llx\n", + m->rows, m->dpitch, + st.line_length, m->dst, + ch->h_object, push_va); + } m->phys = DMA_DST_TYPE_PHYSICAL(dma); m->aperture = st.have_dst_mode ? st.dst_phys_mode : CE_APERTURE_VID; n_ops++; ops_seq++; + if (ramfill && svc.fill_ram_log < 24) { + svc.fill_ram_log++; + pr_info("vgsp-sema: RAMFILL-EXEC push=0x%llx dst=0x%llx line=%u ap=%u value=0x%llx (physical guest-RAM fill built)\n", + push_va, st.offset_out, + st.line_length, + m->aperture, m->value); + } } else if (!DMA_REMAP_ENABLE(dma) && st.have_in && n_ops < SVC_MAX_MEMSETS) { @@ -8041,6 +10630,80 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, DMA_SRC_TYPE_PHYSICAL(dma); m->src_aperture = st.have_src_mode ? st.src_phys_mode : CE_APERTURE_VID; + /* + * r402: a 2D copy (cudaMemcpy2D): + * LINE_COUNT rows of + * LINE_LENGTH_IN bytes with + * PITCH_IN/PITCH_OUT strides. + * Row mode when a pitch is + * wider than the row. + * + * r420c: pitch == row length + * (or no pitch method) makes the + * rows CONTIGUOUS, so the single + * flat run must cover + * rows*line bytes - the + * historical path copied one row + * (victory_memset2d's pitched + * D2H/H2D at pitch 4096 = width + * 4096 lost all but the first + * row). + */ + if (st.line_count > 1) { + u32 dpitch = st.have_pitch_out ? + st.pitch_out : + st.line_length; + u32 spitch = st.have_pitch_in ? + (st.pitch_in ? + st.pitch_in : + dpitch) : + dpitch; + + if (dpitch == st.line_length && + spitch == st.line_length) { + m->bytes = + st.line_length * + st.line_count; + } else { + m->rows = + st.line_count; + m->dpitch = dpitch; + m->spitch = spitch; + } + } + /* + * r458 memset2d forensics: the + * driver's cudaMemcpy2D pushes decode + * LINE_LENGTH=4096 with PITCH_IN/OUT + * present but LINE_COUNT absent + * (lc=0), so only one 4 KB row per + * push executes and a 4 MB copy + * delivers 4 pages. Dump the raw push + * (bounded) for every pitch-shaped + * copy so the method stream that + * carries the row count is on record. + */ + if (gsp_sema_dbg && + (st.have_pitch_out || + st.have_pitch_in) && + svc.op2d_raw_log < 12) { + u32 dl = len < 220 ? len : 220; + + svc.op2d_raw_log++; + pr_info("sema-op2d-raw: h=0x%x push=0x%llx len=%u off=0x%x line=%u lc=%u po=%u/%u pi=%u/%u out=0x%llx in=0x%llx bytes=%u dma=0x%08x\n", + ch->h_object, push_va, + len, off, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in, + m->dst, m->src, + (u32)m->bytes, dma); + pr_info("sema-op2d-raw: raw %*phN\n", + dl, buf); + } n_ops++; ops_seq++; } else if (gsp_sema_dbg && svc.dma_skip_log < 32) { @@ -8069,6 +10732,317 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* + * r365 FILL-RAW: the fill class that still decodes no + * op (window VA, no remap const decoded) - dump the + * raw push and decode its method stream so the true + * value/component encoding is on record. Bounded. + */ + if (allow_ops && transfer != 0 && + DMA_REMAP_ENABLE(dma) && + DMA_DST_MEMORY_LAYOUT_PITCH(dma) && + st.have_out && !st.have_in && !st.have_remap && + svc_fill_raw_log < 48) { + u32 dl = len < 160 ? len : 160; + u32 mo = 0, mn = 0; + + svc_fill_raw_log++; + pr_info("sema-fill-raw: h=0x%x push=0x%llx out=0x%llx line=%u dma=0x%08x dmode=%u have_dstmode=%u trusted=%d allow=%d\n", + ch->h_object, push_va, st.offset_out, + st.line_length, dma, st.dst_phys_mode, + st.have_dst_mode, trusted, allow_ops); + pr_info("sema-fill-raw: raw len=%u %*phN\n", + len, dl, buf); + /* decode the first method pairs */ + while (mo + 8 <= len && mn < 14) { + u32 h2, c2, a2; + + memcpy(&h2, buf + mo, 4); + c2 = (h2 >> 16) & 0xfff; + a2 = (h2 & 0x1fff) << 2; + if ((h2 >> 29) != 1 || !c2 || + mo + 4 + c2 * 4 > len) + break; + if (c2 >= 2) + pr_info("sema-fill-raw: m[%u] addr=0x%x cnt=%u v0=0x%08x v1=0x%08x\n", + mn, a2, c2, + *(const u32 *)(buf + mo + 4), + *(const u32 *)(buf + mo + 8)); + else + pr_info("sema-fill-raw: m[%u] addr=0x%x cnt=%u v0=0x%08x\n", + mn, a2, c2, + *(const u32 *)(buf + mo + 4)); + mo += 4 + c2 * 4; + mn++; + } + } + + /* + * r343 FILL diagnostic (LOG-ONLY, no hold): a + * fill-shaped transfer (REMAP_ENABLE + destination + * PITCH, OFFSET_OUT and no OFFSET_IN) whose value + * constant did not decode currently decodes no op + * and is consumed silently - the guest's cudaMemset + * then reports success with zero bytes written + * (r341 memcpy_roundtrip: 8MB memset readback + * all-bad; victory_memset2d). An op_tr_pending hold + * here WEDGED client init (r343: first cudaMalloc + * retried forever - driver init issues fill-shaped + * CE pushes, the hold re-armed every decode pass + * and never expired). So: log loudly only; the + * sema-remapconst rows above capture the real + * decode shape to build the proper fill decoder. + */ + if (allow_ops && transfer != 0 && + st.line_length >= 1024 && st.have_out && + !st.have_in && DMA_REMAP_ENABLE(dma) && + DMA_DST_MEMORY_LAYOUT_PITCH(dma) && + !st.have_remap && n_ops == n_ops_before && + svc_fill_noremap < 8) { + svc_fill_noremap++; + pr_err("vgsp: FILL-NOREMAP h=0x%x push=0x%llx dma=0x%08x line=%u out=0x%llx phys=%u dstmode=%u: fill-shaped launch, value source not decoded (log-only diagnostic)\n", + ch->h_object, push_va, dma, + st.line_length, st.offset_out, + DMA_DST_TYPE_PHYSICAL(dma), + st.have_dst_mode ? st.dst_phys_mode : 99); + } + + /* + * fail4: name a transfer-shaped launch that decoded NO + * op because the pass was denied (allow_ops=0) - the + * op-chain above (and its dma-skip dump) is skipped + * wholesale in that case, so this class was invisible. + */ + if (gsp_sema_dbg && transfer != 0 && !allow_ops && + svc.dmatrace_log < (gsp_sema_dbg ? 4096 : 64)) { + svc.dmatrace_log++; + pr_info("sema-dmatrace: denied h=0x%x push=0x%llx dma=0x%08x line=%u in=0x%llx out=0x%llx have_in=%u have_out=%u have_remap=%u trusted=%d replay=%d eng=%u pend_would=%d\n", + ch->h_object, push_va, dma, + st.line_length, st.offset_in, + st.offset_out, st.have_in, st.have_out, + st.have_remap, trusted, replay, + ch->engine_type, + (!allow_ops && trusted && + !ch->engine_type && transfer != 0 && + st.line_length && st.have_out && + st.have_in && + !DMA_REMAP_ENABLE(dma) && + st.offset_out >= SVC_VAS_FLOOR && + entry_idx != SVC_ENTRY_NONE && + entry_idx < SVC_GPFIFO_MAX_BITS && + ch->slot_pend_tries[entry_idx] < 4) ? 1 : 0); + } + /* + * fail4 r302: a denied device-destination copy that + * WOULD have built a type=1 op (the exact branch shape + * above: no remap, both offsets, device dst) must not + * be consumed silently. This is the sw-copy client + * mode: the GR-classified GRCE channel's ops run only + * once the pin/trust stream (injected) reaches the + * entry, and the first pass over a chunk push lands + * before that - the copy then never ran and the guest + * computed on stale VRAM (vector_add r302: chunk + * copies cc..d1 skipped; only the tail entries were + * re-decoded by a rewind and ran). Mark the pass + * op-pending so the pump holds the entry through the + * m163/opdef machinery and the retry runs the copy as + * soon as the trust (or the operand translation) + * arrives; the bounded SVC_OPDEF_HOLD_MS expiry keeps + * the pre-existing skip as the last resort. + */ + if (!allow_ops && trusted && + !ch->engine_type && transfer != 0 && + st.line_length && st.have_out && st.have_in && + !DMA_REMAP_ENABLE(dma) && + st.offset_out >= SVC_VAS_FLOOR) { + /* + * r314 (CONT.57 lead 2): capture the staging + * source on the DENIED pass too. This is the + * pipeline-order moment: the push was just + * decoded, the driver has not seen this + * push's fence (it applies later in this same + * pass at the earliest), so the staging still + * holds this chunk's bytes. The old capture + * gate lived in the op-build block, which + * only runs on ALLOWED passes - by then the + * fence had long fired and the staging was + * refilled (r302-r313: captures 0-3/8, the + * late executions read refilled staging, the + * deterministic 786335-bad output). Same + * translation-probe guard as the build-time + * gate: only an SYS-resolving source is + * host staging. + */ + if (st.line_length <= SVC_COPY_SNAP_MAX) { + u64 spa = 0; + u32 sap = 0; + + if (!svc_op_translate(vgpu, + st.offset_in, + &spa, &sap) && + sap == CE_APERTURE_SYS) + svc_copy_snap_capture(vgpu, + st.offset_out, + st.offset_in, + st.line_length, + ch->h_object, + entry_idx, + svc_pass_hash); + } + /* + * r308: the `!replay` gate above kept this + * dead on the sw-copy channel: after the first + * consumed pass every later pass classifies as + * a replay, so the denied chunk copies were + * never retried (vector_add: 6 of 8 chunks + * skipped). A replay pass that marks the slot + * pending un-completes it (slot_ops_complete + * is written from svc.op_tr_pending below), so + * the next pass is treated fresh and the copy + * runs once trust/translation arrives. The + * per-slot counter caps the re-holds so a slot + * whose trust never arrives cannot wedge the + * channel in 10 s hold cycles. + */ + if (entry_idx != SVC_ENTRY_NONE && + entry_idx < SVC_GPFIFO_MAX_BITS && + ch->slot_pend_tries[entry_idx] < 4) { + ch->slot_pend_tries[entry_idx]++; + svc.op_tr_pending = 1; + if (gsp_sema_dbg && + svc.pend_dbg_log < 512) { + svc.pend_dbg_log++; + pr_info("sema-pend: h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u replay=%d trusted=%d tries=%u -> op_tr_pending\n", + ch->h_object, push_va, + entry_idx, st.offset_out, + st.offset_in, + st.line_length, + replay, trusted, + ch->slot_pend_tries[entry_idx]); + } + } + } + /* + * fail4: name every op appended to this pass (copy or + * fill) with its resolved operand fields, so the + * executor's population is on record even when the + * 64 KB copy witness below never fires for it. + */ + if (n_ops != n_ops_before) { + struct svc_memset_op *lm = &ops[n_ops - 1]; + + /* + * fail4 CONT.53: H2D-shaped copies (virtual + * device destination, virtual host source) + * read the driver's ping-pong staging. The + * driver paces restaging on the copy's + * completion fence; our execution may run + * later than the push's processing position, + * so capture the source NOW (bounded pool) + * and let svc_copy execute from the capture - + * the hardware's in-order staging read. + */ + /* + * H2D shape ONLY: device destination (VID + * aperture), host source (SYS). A D2H readback + * (host dst) captured here would freeze + * pre-kernel zeros (observed in the r300 + * captures) and break readbacks. + */ + if (lm->type == 1 && !lm->phys && + lm->dst >= SVC_VAS_FLOOR && + lm->aperture == CE_APERTURE_VID && + !lm->src_phys && + lm->src >= SVC_VAS_FLOOR && + lm->bytes <= SVC_COPY_SNAP_MAX) { + /* + * r311: the src_aperture equality + * gate never matched the sw-copy + * client (the driver sends no + * SET_SRC_PHYS_MODE, so src_ap + * defaults to VID=0 while the + * source is host staging; r302-r310 + * logged zero captures and the + * late executions read refilled + * staging). The translation is the + * truth: capture only when the + * first source page resolves + * through the SYS aperture. + */ + u64 spa = 0; + u32 sap = 0; + + if (!svc_op_translate(vgpu, lm->src, + &spa, &sap) && + sap == CE_APERTURE_SYS) + svc_copy_snap_capture(vgpu, + lm->dst, + lm->src, + lm->bytes, + ch->h_object, + entry_idx, + svc_pass_hash); + if (gsp_sema_dbg && + svc.snap_gate_log < 256) { + svc.snap_gate_log++; + pr_info("sema-snapgate: dst=0x%llx src=0x%llx trc=%d sap=%u\n", + lm->dst, lm->src, + svc_op_translate(vgpu, lm->src, &spa, &sap), + sap); + } + } + + /* + * fail4 CE linear auto-advance, BOTH sides: the + * driver re-sets only ONE offset per op and the + * copy engine advances the other by the transfer + * size. H2D ops show an explicit advancing DST + * with a FIXED SRC base (every 1 MB chunk read + * the source base again: vector_add's device a + * held h_a[0..1MB) x4 -> 75% mismatch; the + * readbacks are the mirror image). Advance both + * registers; an explicit re-set simply overrides. + */ + { + u64 adv = (lm->type == 1) ? lm->bytes : + (u64)lm->elem * lm->count; + + st.offset_out += adv; + if (lm->type == 1) + st.offset_in += adv; + } + if (svc.opadd_log < 2048) { + svc.opadd_log++; + pr_info("sema-opadd: h=0x%x type=%u dst=0x%llx src=0x%llx bytes=%llu elem=%u count=%u ap=%u src_ap=%u phys=%u src_phys=%u\n", + ch->h_object, lm->type, + lm->dst, lm->src, lm->bytes, + lm->elem, lm->count, + lm->aperture, lm->src_aperture, + lm->phys, lm->src_phys); + /* + * r420d: 2D geometry of every copy op + * (the memset2d readback forensics - + * the driver decomposes cudaMemcpy2D + * into per-push ops whose LINE_COUNT/ + * PITCH decode must be visible to + * tell "one row per op" from "N rows + * in one op"). Bounded. + */ + if (lm->type == 1) + pr_info("sema-op2d: h=0x%x dst=0x%llx src=0x%llx bytes=%u rows=%u dpitch=%u spitch=%u line=%u lc=%u have_po=%u po=%u have_pi=%u pi=%u\n", + ch->h_object, lm->dst, + lm->src, lm->bytes, + lm->rows, lm->dpitch, + lm->spitch, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in); + } + } + + /* * Both release encodings write the payload word first * (NvGpuSemaphore.data[0] for the four-word format, the * single word for the one-word format), and the client's @@ -8094,6 +11068,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.sem_va_lo; st.rel_va = sem_va; st.rel_payload = st.sem_payload; + svc_wait_rel_va = sem_va; + svc_wait_rel_payload = st.sem_payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; if (st.have_sem && st.sem_payload && @@ -8191,6 +11169,22 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, ch->entry_want = st.sem_payload; have_cur = true; pending = 1; + /* + * fail4: apply this seed ABSOLUTE. + * The generic applier still sees the + * increment flag (inc), so leaving + * the residue in cur makes it write + * cur+payload (observed 134+1=135) + * - a completed value far past the + * guest's queued count, which + * trips the guest UVM assert + * (uvm_channel.c:204 completed <= + * queued) and kills the device. + * cur=0 makes inc-apply land the + * payload itself, exactly the + * re-seed the client polls. + */ + cur = 0; svc.pool_residue++; if (svc.pool_residue_log < 32) { svc.pool_residue_log++; @@ -8307,7 +11301,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, gsp_sema_skip_ops); } svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -8403,7 +11398,44 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, */ svc.fwd_rel_overflow++; svc_fwd_soft_copy_inline(vgpu); - if (svc_rel_apply_launch(vgpu, ch, + /* + * fail4 CONT.53: never clear a + * push's fence before its ops + * read their source. The pass's + * ops run at the pass end; the + * driver's ping-pong staging + * restages as soon as the fence + * clears, and a later op read + * then returns chunk+N bytes + * (vector_add +2 MB). Queue the + * release through the pending + * pass instead. + */ + if (n_ops > 0 || + svc.op_tr_pending) { + /* + * r316 (CONT.60): + * op_tr_pending covers + * the denied pass - the + * pass built no op + * (allow=0) but a copy is + * still owed; applying + * the fence here (the old + * n_ops==0 path) let the + * driver refill the + * ping-pong staging before + * the copy ever ran - the + * cold-run capture gap + * (vector_add run 1 + * bad=157696/193536 vs + * bad=0 warm). + */ + svc_pending_add(sem_va, + st.sem_payload, + inc ? 1u : 0u); + svc_rel_va_seen = sem_va; + svc_rel_count++; + } else if (svc_rel_apply_launch(vgpu, ch, sem_va, st.sem_payload, cur, pending, @@ -8437,7 +11469,20 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc_fwd_rel_direct_note(ch->h_object, sem_va, st.sem_payload); - if (svc_rel_apply_launch(vgpu, ch, sem_va, + /* + * fail4 CONT.53: same ops-order + * guard as above - a pass with + * pending copy ops must run + * them before its fence clears. + */ + if (n_ops > 0) { + svc_pending_add(sem_va, + st.sem_payload, + inc ? 1u : 0u); + svc_rel_va_seen = sem_va; + svc_rel_count++; + } else if (svc_rel_apply_launch(vgpu, ch, + sem_va, st.sem_payload, cur, pending, inc, inc_apply)) { @@ -8457,7 +11502,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, off += 4 + count * 4; } - if (gsp_sema_dbg && ch->h_object && svc.entry_log < 512) { + if (gsp_sema_dbg && ch->h_object && + svc.entry_log < 16384) { svc.entry_log++; { u8 tail[24]; @@ -8523,6 +11569,12 @@ struct svc_pending_release { u64 va; u32 payload; u32 op; /* 0 = RELEASE, 1 = REDUCTION (add payload) */ + u32 h; /* r316: channel the release was decoded for; the + * verified pool resolver is channel-keyed and h=0 + * never matches its per-channel entries (matmul + * r322: pool-rel cand resolved at decode (rrc=0, + * h=channel) while svc_apply_pending with h=0 + * starved forever and hung the client). */ /* * Resolution attempts that found no translation for this VA. * Telemetry for ISSUE-0001: a pending release that keeps failing to @@ -8599,6 +11651,7 @@ static void svc_pending_add(u64 va, u32 payload, u32 op) svc_pending[i].payload = payload; svc_pending[i].op = op; svc_pending[i].tries = 0; + svc_pending[i].h = svc.cur_ch; } /* @@ -8756,6 +11809,26 @@ static void svc_session_flush(void) svc.counter_noslot_dbg = 0; svc.sem256_log = 0; svc.xfer_raw_log = 0; + svc.pushtrace_all = 0; + svc.opadd_log = 0; + svc.dmatrace_log = 0; + svc.copyent_log = 0; + svc_wait_log = 0; + svc_wait_pass_log = 0; + svc_remapconst_log = 0; + svc_pitch_log = 0; + svc_fill_raw_log = 0; + svc_lowfill_log = 0; + svc_tabcopy_log = 0; + svc_fill_noremap = 0; + svc_wait_n = 0; + svc_wait_rel_have = false; + memset(svc_wait_holds, 0, sizeof(svc_wait_holds)); + svc.bigrun_log = 0; + svc.lid_hdr_log = 0; + svc.skiprun_log = 0; + svc.mystery_log = 0; + svc.next_hdr_log = 0; /* * The push decode budget (`sema-push` lines) is per-session too: the * first client of a boot otherwise spends it and the next session's @@ -8792,6 +11865,8 @@ static void svc_session_flush(void) memset(svc_retry_q, 0, sizeof(svc_retry_q)); memset(svc_pool_pa, 0, sizeof(svc_pool_pa)); memset(svc_fwd_rel, 0, sizeof(svc_fwd_rel)); + memset(svc_inline_once, 0, sizeof(svc_inline_once)); + svc_inline_once_next = 0; memset(svc_grce_dst_ring, 0, sizeof(svc_grce_dst_ring)); for (i = 0; i < SVC_VAS_SLOTS; i++) { if (svc_vas[i].data) @@ -8833,6 +11908,15 @@ static void svc_session_flush(void) svc_hold_fb_log = 0; svc_cand_ovw_log = 0; svc_inline_recs_log = 0; + /* + * r401: the exactly-once capture memory is session-scoped. A new + * client's pushes restart their release counters, so their content + * (and hence hash) can collide with the previous client's - keeping + * the table would wrongly classify a fresh launch of the new + * session as already captured (LAUNCH-SKIP, silent drop). + */ + memset(svc_gr_seen, 0, sizeof(svc_gr_seen)); + svc_gr_seen_stale = 0; /* m176: new session generation - channels and scrubbers re-stamp on * registration and never inherit per-session learned state. */ svc_sess_gen++; @@ -8887,6 +11971,25 @@ void gsp_sema_service_guest_reset(void) } /* + * r317: queue a pass's recorded forward releases through the pending + * machinery instead of applying them at a hold break / fallback while an + * op of that pass is still owed. Applying the fence early let the driver + * refill the ping-pong staging before the copy read it (the cold-run + * capture gap: vector_add run 1 of every cold boot, last ~2 chunks). The + * pending store applies each release once the word resolves (the copy has + * run by then, since the op execution is what clears the pending state on + * the retry). + */ +static void svc_fwd_rel_queue_pending(struct svc_fwd_rel *rel, u8 n) +{ + u8 i; + + for (i = 0; i < n; i++) + svc_pending_add(rel[i].va, rel[i].payload, + rel[i].inc_claimed ? 1u : 0u); +} + +/* * Loud, bounded telemetry for a starved deferred release (ISSUE-0001). * Called from the two "no translation yet, keep" paths of * svc_apply_pending() with an incremented try count; logs at 4/64/1024 @@ -8917,8 +12020,11 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) * resolver: the learned map can answer with a stale * page whose value makes this release look already * delivered and drops it (see svc_sem_write). + * r316: resolve with the channel the release was + * decoded for (h=0 never matches the per-channel + * pool entries - matmul r322 starved/hung). */ - if (svc_pool_pa_resolve(vgpu, 0, p->va, &ppa, &pap, + if (svc_pool_pa_resolve(vgpu, p->h, p->va, &ppa, &pap, NULL)) { /* not translatable yet: keep (ISSUE-0001) */ p->tries++; @@ -8972,7 +12078,7 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) if (!svc_sem_write(vgpu, 0, p->va, p->op ? cur + p->payload : p->payload)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg) pr_info("sema-svc: deferred release va=0x%llx payload=%u (was cur=%u)\n", p->va, p->payload, cur); @@ -9145,7 +12251,7 @@ static u32 __maybe_unused __svc_apply_page_data_disabled(struct nv_vgpu_device * if (svc_sem_write(vgpu, 0, va, payload) == 0) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); applied++; if (payload != cur + 1) { svc.releases_jumped++; @@ -10146,6 +13252,119 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, return 0; } } + + /* + * r496b: no candidate matched. For a GR channel whose ring is known + * (translated and pinned - the sysmem content scan or the VRAM + * conventions), the USERD may still be elsewhere: the X/GL driver + * allocates its USERD in VRAM and the CPU RM maps it through a BAR1 + * window, so the driver's GP_PUT write lands at a BAR1 offset that is + * neither the registered memdesc base nor ring-derived. Scan the + * emulated VRAM for the USERD shape and validate each hit against the + * ring: the put it claims must correspond to entries whose pushbuffer + * VAs all sit in the ring's allocation window. A foreign channel's + * USERD cannot pass that test. One scan per channel (the result + * persists through the pump's write-back), logged on latch. + */ + if (!ch->engine_type && ch->userd_scans < 8 && ring_pa) { + u64 off, lim; + u8 *base = vgpu->bars[1].vaddr; + u64 best = 0, best_dist = ~0ULL; + u32 best_put = 0, best_get = 0; + bool best_pending = false; + + ch->userd_scans++; + lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + if (base && lim) { + for (off = 0; off + 0xa0 <= lim; off += 0x200) { + u32 hp, hg; + bool ok = true; + u32 k; + u64 dist; + bool pending; + + memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); + memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); + if (!hp || hp > ch->gpfifo_entries || hg > hp) + continue; + /* Validate: every claimed entry decodes to a + * pushbuffer VA in the ring's window. */ + for (k = 0; k < hp && k < 64; k++) { + u8 e[8]; + u64 eva; + u32 elen, a0, a1; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)k * 8, + e, sizeof(e))) { + ok = false; + break; + } + memcpy(&a0, e, 4); + memcpy(&a1, e + 4, 4); + eva = ((u64)(a1 & 0xff) << 32) | + (u64)(a0 & 0xfffffffc); + elen = ((a1 >> 10) & 0x1fffff) << 2; + /* + * Every entry the put claims must name a + * VA in the ring's allocation window - + * pushbuffers below the ring, the + * progress tracker above it. + */ + if (!eva || + eva + SVC_GR_RAM_WIN / 2 < + ch->gpfifo_va || + eva >= ch->gpfifo_va + + SVC_GR_RAM_WIN / 2 || + elen < 4 || elen > 0x100000) { + ok = false; + break; + } + } + if (!ok) + continue; + /* + * Several blocks can validate: a previous + * session leaves its old USERD holding the same + * put/entry values while the new channel's block + * sits elsewhere (observed r504: the re- + * registered channel re-scanned, adopted the + * stale block, and its DMA test timed out). + * Prefer a block with work PENDING (get < put); + * among equals prefer the one nearest the + * registered memdesc offset (the driver's own + * hint). The pump's GP_GET write-back goes to + * the adopted block, so a stale adoption strands + * the client's poll. + */ + dist = off > ch->userd_pa ? off - ch->userd_pa + : ch->userd_pa - off; + pending = hg < hp; + if (best && + (best_pending > pending || + (best_pending == pending && dist >= best_dist))) + continue; + best = off; + best_dist = dist; + best_put = hp; + best_get = hg; + best_pending = pending; + } + if (best) { + ch->userd_pa = best; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + *put = best_put; + *get = best_get; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by VRAM shape scan at fb 0x%llx (put=%u get=%u%s, ring-validated)\n", + ch->h_object, ch->gr_index, best, + best_put, best_get, + best_pending ? "" : ", idle"); + return 0; + } + } + } return -ENOENT; } @@ -10243,7 +13462,7 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, /* Keep the completion visible to clients that start waiting * after the event fired. */ - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); } } @@ -10906,6 +14125,14 @@ unsigned gsp_phys_gr_hold_max_ms = 3000; unsigned gsp_phys_gr_fill_hold; /* default 0: log-only, never hold */ /* + * r346: trace GMMU walks for unresolved high-VA operands (the arg-miss + * diagnostic re-walks each published vaspace for a failing base). When + * nonzero, those walks print the per-level entries (L0..L2, PD0 halves) + * so a failing VA shows exactly where the walk dies. + */ +unsigned gsp_sema_walk_trace; + +/* * F1-b: GR release-word authorship gate (default 1). While a forwarded GR * launch is in flight, the RAM-content sweep must not author the very QMD * completion release the backend poll is waiting for: a sweep write could @@ -10944,6 +14171,30 @@ bool gsp_sema_pool_hold_fresh = 1; * 0 restores the historical immediate-write behavior. */ bool gsp_sema_copy_dst_pin_guard = 1; +/* + * r336/r34x wait gate: honor the C56F host-semaphore acquire ops captured + * for a ring entry before running its work. DEFAULT ON since r408: the + * r408 streams2d forensics reproduced the class the gate was built for + * with the gate off - the c2 readback (C56F ACQ_CIRC_GEQ pay=4 on the + * scale channel's progress counter 0x20400ffd0) ran at 106617.279 with + * cur=2, i.e. BEFORE the c2-add launch was even submitted (106617.411), + * read the not-yet-produced buffer (zeros) and the client failed with + * 'c2 all zeros'. With the gate on the same readback holds (sema-wait: + * hold ... cur=2 waited_ms=353) until the counter reaches 4 (pass ... + * cur=4 at 106812.259) and the ordering is correct. The semantics are + * the class contract's: op 0 = equal, 2 = strict GEQ, 3 = circular GEQ, + * 4 = AND, 5 = NOR; the wait set is captured PER ENTRY (the old + * single-last-wait global leaked a later entry's acquire into an earlier + * entry - the r340 over-hold); the entry's own release counts only when + * its method order precedes the acquire; pending releases our machinery + * is merely late about are drained first; and every hold is bounded at + * gsp_sema_wait_hold_ms (default 4000 ms, r401) then fails open loudly + * ('give-up') so a value that can only be installed behind an upstream + * opdef hold cannot park a channel. Diagnostics: 'sema-wait: + * pass/hold/give-up' rows carry op, payload, observed value and decision. + * sema_wait_gate=0 restores the pre-r336 pass-through for an A/B. + */ +bool gsp_sema_wait_gate = 1; /* * m169b: the guard above must never deadlock the boot's first client. The @@ -11073,6 +14324,24 @@ bool gsp_sema_rel_recover; bool gsp_sema_dbg; /* + * r365: execute fill-shaped CE transfers whose destination is a VIDMEM + * WINDOW VA (>= SVC_VIDMEM_VA_LO, below the 1 TiB VA floor). The guest's + * page-table writes arrive exactly this way (dma=0x586, REMAP+PITCH, a + * count=1 SET_REMAP_CONST_A, line=4, out=0x2000c0xxx) and previously + * decoded no op, so the shadow never saw them and the first client of a + * fresh boot dropped operand pages (GRFILL-DROP -> launch held/fail-open + * with 8 dropped pages -> wrong results or a client wedge). + * + * The op is built with the VA unchanged (phys=0): svc_fill's virtual path + * resolves it through the guest's own published page tables + * (svc_op_translate) and only falls back to the shadow for floor-range + * VAs - never through the window convention (r362 proved a forced + * `addr - SVC_VIDMEM_VA_BASE` writes the wrong page and corrupts results: + * reduce sum=0 + mail hang). Default off for A/B. + */ +bool gsp_sema_winfill; + +/* * Skip executing a push's collected work (CE memsets/copies into the vidmem * shadow) while still applying its semaphore release. The client only ever * observes the release; the shadow writes exist so the emulation can walk @@ -11131,6 +14400,139 @@ unsigned gsp_sema_push_hold_ms = 2500; static unsigned long svc_gr_ring_tried; /* ranks that got a diagnostic line */ +static bool svc_gr_ram_ring_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out) +{ + static u8 *buf; + u64 start = ch->gr_ram_cursor; + u64 end = start + SVC_GR_RAM_SLICE; + u64 gpa; + int i; + unsigned long flags; + + if (end > SEMA_PA_MAX) + end = SEMA_PA_MAX; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + for (gpa = start; gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + 16 <= chunk; o += SEMA_PAGE_SIZE) { + u32 e0, e1, f0, f1; + u64 va0, va1; + u32 len0, len1; + + memcpy(&e0, buf + o, 4); + memcpy(&e1, buf + o + 4, 4); + va0 = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len0 = ((e1 >> 10) & 0x1fffff) << 2; + /* entry0: main pushbuffer, just below the ring */ + if (va0 >= ch->gpfifo_va || + va0 + SVC_GR_RAM_WIN / 2 < ch->gpfifo_va) + continue; + if (va0 < SEMA_VA_LO || va0 >= SEMA_VA_HI) + continue; + if (len0 < 4 || len0 > 0x100000) + continue; + memcpy(&f0, buf + o + 8, 4); + memcpy(&f1, buf + o + 12, 4); + va1 = ((u64)(f1 & 0xff) << 32) | + (u64)(f0 & 0xfffffffc); + len1 = ((f1 >> 10) & 0x1fffff) << 2; + /* entry1: progress tracker, above entry0 */ + if (va1 <= va0 || + va1 > ch->gpfifo_va + SVC_GR_RAM_WIN / 2) + continue; + if (len1 < 4 || len1 > 0x100000) + continue; + *pa_out = gpa + o; + return true; + } + } + ch->gr_ram_cursor = (end >= SEMA_PA_MAX) ? 0 : end; + ch->gr_ram_passes++; + if (ch->gr_ram_cursor == 0 && !ch->gr_ram_logged) { + ch->gr_ram_logged = true; + pr_info("sema-chan: GR ring h=0x%x rank=%u RAM content scan found no candidate in %u slice(s) (va=0x%llx)\n", + ch->h_object, ch->gr_index, ch->gr_ram_passes, + ch->gpfifo_va); + } + /* + * Persist the cursor immediately: the pump's stalled-userd path + * `continue`s past the end-of-pass write-back, so without this the + * scan would restart from slice 0 every pass and never reach the + * ring's page. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[i].gr_ram_passes = ch->gr_ram_passes; + svc.chan[i].gr_ram_logged = ch->gr_ram_logged; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +/* + * r496f: discover a kernel-RM client VAS's page directory in the emulated + * VRAM by verification. The CPU RM builds internal-VAS page tables in + * vidmem and never publishes them (SET_PAGE_DIRECTORY is an externally + * owned / UVM-only RPC), so the pushbuffer pages and the progress + * semaphore of an X/GL channel cannot be translated from any existing + * source. Try each 4 KB page as the walk root and accept the first whose + * walk reproduces the ring VA -> ring PA pair we already validated by + * content; registering it with the VAS table makes every other VA of the + * same client resolvable through the existing walker. Bounded to the + * vidmem shadow; verified, so a wrong candidate cannot be registered. + */ +static void svc_xvas_pdb_discover(struct nv_vgpu_device *vgpu, u64 ring_va, + u64 ring_pa) +{ + static const u32 ent_tab[] = { 4, 8, 16, 32, 64, 2, 128, 256, 512 }; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + unsigned int ei; + + if (!vgpu->bars[1].vaddr || !lim) + return; + for (off = 0; off + 0x1000 <= lim; off += 0x1000) { + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) { + u64 pa = 0; + int ap = 0; + + if (!svc_gmmu_walk(vgpu, off, ent_tab[ei], + ring_va & ~0xfffULL, + SVC_APERTURE_FB, &pa, &ap, false) && + (pa & ~0xfffULL) == (ring_pa & ~0xfffULL)) { + struct sema_vaspace_desc d; + + memset(&d, 0, sizeof(d)); + d.h_vaspace = 0x5a5a0000u | + (u32)(ring_va >> 20); + d.pdb = off; + d.num_entries = ent_tab[ei]; + d.b_vidmem = true; + gsp_sema_service_vaspace_add(&d); + pr_info("sema-chan: client VAS page directory discovered pdb=0x%llx entries=%u (ring VA 0x%llx -> pa 0x%llx verified; the client's other VAs are now walkable)\n", + off, ent_tab[ei], ring_va, ring_pa); + return; + } + } + } +} + static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -11407,6 +14809,124 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, pr_info("sema-chan: fbs scan done h=0x%x hits=%d (lim 0x%llx)\n", ch->h_object, hits, lim); } + + /* + * r495 diagnostic (X/GL channel): the X driver's GR channel + * registers its USERD at a memdesc base that may be an FB + * offset or a BAR1 window. The driver's kickoff writes GP_PUT + * through its CPU mapping of that object; if the write landed + * in the emulated FB shadow, scanning for the USERD shape + * (GP_PUT in [1..entries] at +0x8c, GP_GET <= GP_PUT at +0x90, + * both nonzero somewhere in the 0x200 block) names the real + * offset. One-shot per channel, diagnostic only. + */ + if (gsp_sema_dbg && vgpu->bars[1].vaddr && + svc_fb_shadow_size(vgpu) && ch->gpfifo_entries <= 0x20000) { + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + int hits = 0; + + for (off = 0; off + 0xa0 <= lim && hits < 8; + off += 0x200) { + u32 hp, hg; + + memcpy(&hp, vgpu->bars[1].vaddr + off + 0x8c, 4); + memcpy(&hg, vgpu->bars[1].vaddr + off + 0x90, 4); + if (!hp || hp > ch->gpfifo_entries || hg > hp) + continue; + pr_info("sema-chan: userd scan hit off=0x%llx put=%u get=%u (blk %08x %08x %08x %08x)\n", + off, hp, hg, + *(u32 *)(vgpu->bars[1].vaddr + off + 0x80), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x84), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x88), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x94)); + hits++; + } + pr_info("sema-chan: userd scan done h=0x%x hits=%d (lim 0x%llx)\n", + ch->h_object, hits, lim); + } + } + + /* + * r496b: guest-RAM content scan for the ring (see the gr_ram_cursor + * field comment). The driver's kickoff writes GPFIFO entries whose + * pushbuffer VAs sit just BELOW the ring VA in the same allocation + * (main pushbuffer first, GPFIFO after - nvidia-push-priv.h): entry0 + * in (ring_va - 0x100000, ring_va), entry1 above entry0 and up to + * ring_va + 0x100000 (the progress tracker). A page whose first two + * 8-byte words both name such VAs with sane lengths is the ring; pin + * it (validated) and its allocation window (guesses) so the ring + * entry and pushbuffer reads resolve. Sliced so the service thread + * stays responsive; gives up per channel after + * SVC_GR_RAM_PASS_MAX slices. + */ + if (ch->gr_ram_passes < SVC_GR_RAM_PASS_MAX && + !test_bit(ch->gr_index, &svc_gr_ring_done)) { + u64 hit_pa = 0; + + if (svc_gr_ram_ring_scan(vgpu, ch, &hit_pa)) { + /* + * Pin ONLY the content-validated ring page (validated, + * not a window guess). The pushbuffer pages and the + * tracker push of this client resolve through the VAS + * page directory registered from COPY_SERVER_PDES + * (r496g) - the surface is NONCONTIGUOUS, so a linear + * window guess is wrong for them and, being a + * pin_lookup_any hit, would shadow the correct walk in + * svc_lookup_trusted (observed r502: the tracker push + * read zeros through the wrong window page and the + * driver's DMA poll never completed). + */ + svc_pin_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + ch->gr_ram_logged = true; + ch->gr_ram_ring = true; + set_bit(ch->gr_index, &svc_gr_ring_done); + svc_ring_n++; + /* + * r496j: with the ring content-validated, verify the + * parked COPY_SERVER_PDES candidates against it and + * register the directory that reproduces the ring's + * PA; its walk then resolves the client's other VAs + * (pushbuffer pages, the progress semaphore the + * driver polls). r496f's blind vidmem sweep stays as + * a fallback for directories the control never + * published. + */ + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + hit_pa); + svc_xvas_pdb_discover(vgpu, ch->gpfifo_va, hit_pa); + /* + * Persist the latch immediately: a later stalled + * path in this pass may `continue` past the + * end-of-pass write-back, and the next pass's + * snapshot must carry gr_ram_ring (the zero-NOP + * exception and the no-capture-hold skip key on + * it). + */ + { + unsigned long flags2; + int i2; + + spin_lock_irqsave(&svc.chan_lock, flags2); + for (i2 = 0; i2 < SVC_MAX_CHANNELS; i2++) + if (svc.chan[i2].valid && + svc.chan[i2].vgpu == vgpu && + svc.chan[i2].h_object == + ch->h_object) { + svc.chan[i2].gr_ram_ring = true; + svc.chan[i2].gr_ram_logged = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags2); + } + pr_info("sema-chan: GR ring h=0x%x rank=%u found by RAM content scan va=0x%llx -> pa=0x%llx (allocation window pinned; entry/push reads resolve through it)\n", + ch->h_object, ch->gr_index, ch->gpfifo_va, + hit_pa); + return; + } } } @@ -11414,8 +14934,7 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { u64 pa = 0, va = 0; - int ap = SVC_APERTURE_SYS; - u8 e[SVC_GPFIFO_ENTRY_SIZE]; + int ap = SVC_APERTURE_SYS; u8 e[SVC_GPFIFO_ENTRY_SIZE]; u32 e0, e1, len; if (!ch->engine_type) { @@ -12326,6 +15845,25 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, return; val = put; + /* + * r471: never advance the guest's completion word past a push whose + * content was still unwritten when its entry was consumed. The + * clamp position is the lowest all-zero GR slot (gr_bad_put = slot + * + 1; 0 = none). Without it the publish can satisfy a D2H wait + * before its copy ran and the client drains a reused staging buffer + * holding the previous copy's bytes (streams_events h_c[2] = c0). + * The clamp heals when the rewinds re-read the slot with real + * content; the real completions still arrive through the decoded + * releases. + */ + if (!ch->engine_type && ch->gr_bad_put && val >= ch->gr_bad_put) { + val = ch->gr_bad_put - 1; + if (gsp_sema_dbg && svc.counter_pub_dbg < 96) { + svc.counter_pub_dbg++; + pr_info("sema-diag: pub-clamp h=0x%x pa=0x%llx val=%u (bad=%u) cur=%u\n", + ch->h_object, pa, val, ch->gr_bad_put, cur); + } + } if (val == ch->counter_last) return; /* @@ -12349,9 +15887,20 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, pr_info("sema-diag: pub-adv h=0x%x pa=0x%llx val=%u cur=%u last=%u\n", ch->h_object, pa, val, cur, ch->counter_last); - if (svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, &val, sizeof(val))) - return; + { + int wrc = svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, + &val, sizeof(val)); + if (wrc) { + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: pub-adv-wrc h=0x%x pa=0x%llx val=%u wrc=%d (write did not land)\n", + ch->h_object, pa, val, wrc); + } + return; + } + } ch->counter_last = val; + ch->counter_gen = svc_sess_gen; if (gsp_sema_dbg && svc.counter_pub_dbg < 48) { svc.counter_pub_dbg++; pr_info("sema-chan: counter publish h=0x%x pa=0x%llx val=%u (was %u)\n", @@ -12449,7 +15998,7 @@ static bool svc_fake_release(struct nv_vgpu_device *vgpu, struct svc_channel *ch svc.releases_applied++; if (!gsp_sema_fake_rel) svc.releases_recovered++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); } } } @@ -12503,6 +16052,20 @@ static bool svc_entry_want_reached(struct nv_vgpu_device *vgpu, /* Mark a consumed GPFIFO slot for the current put epoch. */ static void svc_replay_note(struct svc_channel *ch, u32 idx, u32 put) { + static u32 consume_log; + + /* + * fail4 r303: name every consumed slot (all channels). Paired with + * the sema-entry decode rows this shows exactly which entries were + * consumed without ops and when (the vector_add chunk-drop chase: + * a consumed slot whose decode row carried a device-dst copy but + * whose op never ran is the silent skip under diagnosis). + */ + if (gsp_sema_dbg && consume_log < 16384) { + consume_log++; + pr_info("sema-consume: h=0x%x idx=%u put=%u\n", + ch->h_object, idx, put); + } (void)put; /* * STALL FIX (entry=82/pay=84 class, r186): the entry leaving the @@ -12782,6 +16345,24 @@ static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind, return -EFAULT; *hpa = (u64)page_to_phys(pg); *out_off = page_off; + /* + * r372 two-path identity check: the copy-out writes through + * __va(hpa) while every reader reads base+pa - if those are + * not the same memory the engine's output silently vanishes + * (the sum=0 class). Prove it empirically, bounded. + */ + if (svc_fbhpa_log < 16) { + u64 at_base = 0, at_va = 0; + + svc_fbhpa_log++; + memcpy(&at_base, base + page, sizeof(at_base)); + memcpy(&at_va, (u8 *)__va(*hpa), sizeof(at_va)); + pr_info("sema-fbhpa: pa=0x%llx base=%px pg=%px hpa=0x%llx shadow=0x%llx at_base=%016llx at___va=%016llx %s\n", + page, base, pg, *hpa, + (unsigned long long)svc_fb_shadow_size(vgpu), + at_base, at_va, + at_base == at_va ? "SAME" : "DIFFER"); + } return 0; } if (kind != SVC_APERTURE_SYS) @@ -12901,6 +16482,19 @@ static void svc_fwd_soft_copy_inline(struct nv_vgpu_device *vgpu) * hardware has not produced yet. */ #define SVC_FWD_GR_HOLD_TIMEOUT_MS 12000u +/* + * r420: hard cap on the WEDGE-QUIET deferral of the GR hold deadline. The + * backend's quiet hold is a bounded msleep (its own caps: <=240 s per hold, + * two holds per launch), so the pause can only last a few minutes; the cap is + * a defence-in-depth bound so a stuck quiet-hold flag can never defer the + * client's completion indefinitely. + */ +#define SVC_FWD_GR_HOLD_PAUSE_MAX_MS 600000u +/* + * r420: same defence for the per-launch consumption bound pause in + * svc_gr_hold_bound_expired(). + */ +#define SVC_GR_QUIET_PAUSE_MAX_MS 600000u struct svc_fwd_hold { bool active; @@ -12936,9 +16530,9 @@ struct svc_fwd_hold { /* * P6c completion relay: the request selected record relay_index and * the hardware copy was asked to author it into the guest-visible - * semaphore page. On the success edge the hold re-reads the guest - * word and, when it carries the payload, skips that one record's - * software apply. Cleared with the hold. + * semaphore page. On the success edge the hold re-reads the guest word + * and, when it carries the payload, skips that one record's software + * apply. Cleared with the hold. */ bool relay_armed; u8 relay_index; @@ -12951,6 +16545,18 @@ struct svc_fwd_hold { struct svc_fwd_rel rel[SVC_FWD_REL_MAX]; u64 last_rel_va; unsigned long since; + /* + * r420: WEDGE-QUIET deferral state. While the backend deliberately + * holds the in-flight launch in its bounded quiet window + * (gsp_phys_gr_quiet_hold_active), the GR deadline is pushed forward + * (paused_until) so the hold cannot author the launch's release word + * under the hold - the client would then verify data the post-hold + * copy-out has not written yet (the r419 mid-boot escape). The + * deferral is itself bounded by SVC_FWD_GR_HOLD_PAUSE_MAX_MS from + * `since` and is announced once per hold. + */ + unsigned long paused_until; + bool pause_logged; }; static struct svc_fwd_hold svc_fwd_hold; @@ -13176,6 +16782,29 @@ static void svc_opdef_clear(struct svc_opdef_ent *e, u32 caller) e->n ? e->rel[0].va : 0ull, e->n ? e->rel[0].payload : 0u); } + /* + * r456/r457: the snapshot lifecycle is now "capture once, consume + * on the copy's success, evict on ring-slot reuse, TTL". Every + * opdef_clear caller can still re-decode the entry: + * - APPLY (0): the pass's own op execution (which consumes the + * snapshot on success) follows the apply; + * - FINISH (1): the entry can be REPLAYED after a consume (the + * pin-batch rewind replays entries while mappings stream in), + * and an owed copy still needs the pipeline-order bytes; + * - FLUSH (2): the op stays skipped this pass but a later replay + * may run it. + * The r457 mode-C evidence: the b1 H2D's snapshot was captured at + * 191607.816388 with b1's bytes (first4=00000000 3e800000 ...), + * FINISH freed it on the entry's consume, the rewind replay + * re-captured at 191608.798838 from the REFILLED staging + * (first4=00000000 3d800000 ... = b2) and the deferred execution + * wrote b2's bytes into db1 (dst16_after=000000000000803d...). + * The r444 leak the old free guarded against is already covered + * by the owner key (a stale snapshot can only ever serve its own + * (ch, idx, hash)), the slot-reuse eviction in + * svc_copy_snap_capture, the ops_done skip free in svc_run_ops + * and the 30 s TTL. + */ memset(e, 0, sizeof(*e)); } @@ -13207,7 +16836,7 @@ static void svc_opdef_apply(struct nv_vgpu_device *vgpu, if (e->n) { svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); svc.opdef_applied++; - if (gsp_sema_dbg && svc_opdef_log < 32) { + if (gsp_sema_dbg && svc_opdef_log < (gsp_sema_dbg ? 512 : 32)) { svc_opdef_log++; pr_info("sema-opdef: apply h=0x%x entry=%u n=%u (ops complete)\n", ch->h_object, entry_idx, e->n); @@ -13217,6 +16846,37 @@ static void svc_opdef_apply(struct nv_vgpu_device *vgpu, } /* + * r34x: is a release to this VA currently withheld by our own machinery? + * True when the pending-translation ring holds one (translation late) or + * an opdef slot has one recorded (operand hold). The acquire gate calls + * this to classify an unsatisfied wait in its diagnostics: pend=1 names + * the 'value sits behind our own withhold' case - the bounded 100 ms + * give-up is exactly for it (waiting longer cannot install the value + * until the upstream entry's ops complete). Never forces an opdef release + * here: those releases are withheld because their ops have NOT completed, + * and delivering early re-opens the work-before-fence hole they close. + */ +static bool svc_wait_pend_release(u64 va) +{ + u32 i; + + for (i = 0; i < svc_pending_n; i++) + if (svc_pending[i].va == va) + return true; + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + u8 k; + + if (!e->active) + continue; + for (k = 0; k < e->n; k++) + if (e->rel[k].va == va) + return true; + } + return false; +} + +/* * STALL FIX (r186 entry=82/pay=84): called from svc_replay_note() on every * entry-consume path. If a release set was withheld for this entry while an * operand was pending and the entry is now leaving the decode path (so the @@ -13284,7 +16944,7 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, if (!svc_opdef_contains(e, r)) { if (r->payload && e->n < SVC_FWD_REL_MAX) { e->rel[e->n++] = *r; - if (gsp_sema_dbg && svc_opdef_log < 32) { + if (gsp_sema_dbg && svc_opdef_log < (gsp_sema_dbg ? 512 : 32)) { svc_opdef_log++; pr_info("sema-opdef: hold h=0x%x entry=%u kind=%u va=0x%llx pay=%u (operand pending)\n", ch->h_object, entry_idx, @@ -13326,7 +16986,7 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, * flushes the slot and consumes the entry. */ static bool svc_opdef_hold(struct nv_vgpu_device *vgpu, struct svc_channel *ch, - u32 entry_idx) + u32 entry_idx, u64 launch_va) { struct svc_opdef_ent *e = svc_opdef_find(vgpu, ch->h_object, entry_idx); @@ -13335,11 +16995,46 @@ static bool svc_opdef_hold(struct nv_vgpu_device *vgpu, struct svc_channel *ch, if (!e) return false; } + /* + * r361: absolute per-launch bound keyed to the LAUNCH PUSH VA + * (the GRFIX lesson, r213). The slot's `since` restarts whenever + * a pin-batch rewind replays the entry at a different GPFIFO + * index, so a slot-keyed-only bound can be re-armed forever - + * observed as opdef=3301/32/0 (held climbing, expired 0) with the + * guest wedged. The push VA never changes across replays, so the + * first hold of a launch arms an ABSOLUTE deadline nothing moves. + */ + if (!launch_va) + launch_va = 1; /* anonymous launch: still absolutely bound */ + /* + * r368: an ACTIVE bound is never re-armed. The pump walks up to + * SVC_OPDEF_SLOTS held entries, each with its OWN push VA; the r361 + * arm-on-VA-change rule restarted the deadline on every entry + * change, so with several held launches the deadline moved forever + * (streams2 wedge: opdef=955/13/0, held climbing, expired 0). The + * first hold arms the absolute deadline; entries held while it is + * active ride it and flush together when it fires. A fresh bound is + * armed once the previous one has passed. + */ + if (!ch->opdef_hold_dead_va || + time_after_eq(jiffies, ch->opdef_hold_deadline)) { + ch->opdef_hold_dead_va = launch_va; + ch->opdef_hold_deadline = jiffies + + msecs_to_jiffies(SVC_OPDEF_HOLD_MS); + ch->opdef_hold_exp_logged = false; + } if (time_before(jiffies, - e->since + msecs_to_jiffies(SVC_OPDEF_HOLD_MS))) { + e->since + msecs_to_jiffies(SVC_OPDEF_HOLD_MS)) && + time_before(jiffies, ch->opdef_hold_deadline)) { svc.opdef_held++; return true; } + if (!ch->opdef_hold_exp_logged) { + ch->opdef_hold_exp_logged = true; + pr_err_ratelimited("vgsp-sema: OPDEF-EXPIRED h=0x%x idx=%u va=0x%llx (absolute per-launch bound %u ms; rewind replays no longer re-arm)\n", + ch->h_object, entry_idx, launch_va, + SVC_OPDEF_HOLD_MS); + } return false; } @@ -13416,9 +17111,22 @@ static void svc_opdef_drop(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * The ring advanced past a slot's entry (a slot for another entry of the * same channel is being processed): its set belongs to a consumed entry. * Deliver late (forward-only, so it cannot double-apply) and drop it. + * + * r406: an entry is only "passed" if the ring actually CONSUMED it + * (slot_done_epoch). Without that proof the pin-batch rewind breaks the + * invariant: it re-opens lower indices (e.g. 83) while a held entry + * (e.g. 91) sits AHEAD in the same epoch, and the old unconditional drop + * treated the held entry as consumed and delivered its withheld releases + * early. Observed (r406 inplace_scale): the H2D copy's operand was + * unresolved (rc=-2), the entry held with its release withheld; a rewind + * pass at idx=83 fired drop-stale caller=4 for idx=91 and the release + * went out at 104955.751 - while the copy only executed at 104957.418. + * The client raced past its blocking memcpy, launched, the launch read + * the stale buffer (previous run's data), and the copy landed after the + * launch's copy-out: wrong results with 32/1032 "changed". */ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu, - struct svc_channel *ch, u32 keep_idx) + struct svc_channel *ch, u32 keep_idx, u32 put) { int i; @@ -13428,21 +17136,8 @@ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu, if (!e->active || e->vgpu != vgpu || e->h_object != ch->h_object || e->entry_idx == keep_idx) continue; - /* - * STALLFREE r228 (entry=82 of 0x5c000059): a slot whose entry - * the ring has NOT consumed in this put epoch is NOT stale. A - * pin-batch rewind moves last_get backwards over an unconsumed - * entry and re-decodes it; clearing the slot here restarted the - * 10 s opdef deadline on every re-walk, so the hold re-armed - * forever (log: 'opdef-clear caller=4 entry=82' + 'opdef: hold - * entry=82' repeating while the 1 MiB op's operand never - * resolved, rc=-2; put=84 get=82 last=82 frozen for 10 s). The - * epoch bit is set by svc_replay_note on consume, so only an - * entry that was truly consumed has its withheld set - * delivered-late and dropped here. - */ - if (!test_bit(e->entry_idx, ch->slot_done_epoch)) - continue; + if (!svc_replay_seen(ch, e->entry_idx, put)) + continue; /* never consumed: not stale (rewind) */ if (e->n) svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); svc_opdef_clear(e, SVC_OPDEF_CLR_STALE); @@ -13481,31 +17176,6 @@ static void svc_opdef_drop_gone(struct nv_vgpu_device *vgpu, } } -/* - * STALLFREE: drop every opdef slot bound to one channel handle whose - * first-pending stamp PREDATES `before` (the rebinding's reconcile instant). - * A context switch/teardown invalidates the handle's withheld sets - their - * waiters are gone and the releases must not ride the reused channel (the - * opdef-hold class). The `before` filter is the provable-safety rule: a slot - * the fresh binding creates after its registration (the reconcile runs at - * the top of the first pump pass, before that pass's walk) always has - * since >= before and is NEVER touched; only the dead context's slots are - * dropped. Counted as a LOSS when the slot still held undelivered releases. - */ -static void svc_opdef_drop_channel(struct nv_vgpu_device *vgpu, u32 h_object, - unsigned long before) -{ - int i; - - for (i = 0; i < SVC_OPDEF_SLOTS; i++) { - struct svc_opdef_ent *e = &svc_opdef[i]; - - if (e->active && e->vgpu == vgpu && e->h_object == h_object && - time_before(e->since, before)) - svc_opdef_drop_counted(e); - } -} - static void svc_opdef_reset_all(void) { int i; @@ -13712,9 +17382,38 @@ static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) static bool svc_fwd_hold_expired(void) { - if (svc_fwd_req.gr && !smp_load_acquire(&svc_fwd_req.done)) + if (svc_fwd_req.gr && !smp_load_acquire(&svc_fwd_req.done)) { + /* + * r420: while the backend deliberately holds the in-flight + * launch in its bounded WEDGE-QUIET window, the GR deadline + * must not fire - expiring here authors the launch's release + * word (svc_fwd_hold_gr_release) and applies the recorded + * releases before the post-hold copy-out, so the client + * verifies stale data (the r419 soak's mid-boot + * "matmul fails after time"). Pause instead, bounded by + * SVC_FWD_GR_HOLD_PAUSE_MAX_MS from the hold's start; the + * pause extends the deadline a full GR window past the last + * quiet-hold pass, so the backend's retry/accept can finish + * before the fallback would author. Announced once per hold. + */ + if (gsp_phys_gr_quiet_hold_active() && + time_before(jiffies, svc_fwd_hold.since + + msecs_to_jiffies(SVC_FWD_GR_HOLD_PAUSE_MAX_MS))) { + if (!svc_fwd_hold.pause_logged) { + svc_fwd_hold.pause_logged = true; + pr_warn("vgsp-gr: GR-HOLD-PAUSE va=0x%llx (backend wedge-quiet hold active; release authorship deferred under the bounded hold)\n", + svc_fwd_req.gr_push_va); + } + svc_fwd_hold.paused_until = jiffies + + msecs_to_jiffies(SVC_FWD_GR_HOLD_TIMEOUT_MS); + return false; + } + if (svc_fwd_hold.paused_until && + time_before(jiffies, svc_fwd_hold.paused_until)) + return false; return time_after(jiffies, svc_fwd_hold.since + msecs_to_jiffies(SVC_FWD_GR_HOLD_TIMEOUT_MS)); + } return time_after(jiffies, svc_fwd_hold.since + msecs_to_jiffies(SVC_FWD_HOLD_TIMEOUT_MS)); } @@ -13748,6 +17447,9 @@ static void svc_fwd_hold_engage(struct nv_vgpu_device *vgpu, svc_fwd_hold.relay_armed = svc_fwd_req.relay; svc_fwd_hold.relay_index = svc_fwd_req.relay_index; svc_fwd_hold.fault_delivered = false; + /* r420: a fresh hold starts with no WEDGE-QUIET deferral. */ + svc_fwd_hold.paused_until = 0; + svc_fwd_hold.pause_logged = false; svc.fwd_hold_engaged++; svc.fwd_rel_pending = n; } @@ -13774,6 +17476,8 @@ static void svc_fwd_hold_clear(void) svc_fwd_hold.relay_armed = false; svc_fwd_hold.relay_index = 0; svc_fwd_hold.fault_delivered = false; + svc_fwd_hold.paused_until = 0; + svc_fwd_hold.pause_logged = false; svc.fwd_rel_pending = 0; } @@ -14401,6 +18105,16 @@ static void svc_fwd_gr_work(void) data[gs.n_data].hpa = dhpa; data[gs.n_data].reloc_va = dg->reloc_va; /* + * r348 diagnostic: the last 32 high-VA data pages (the output + * operands live there) with the exact (gpa, ap) -> hpa the + * writeback will use - comparable against the D2H read's + * resolution line (sema-bigcopy) for the same VA. + */ + if (dg->va >= SVC_VAS_FLOOR && + i + 32 >= svc_fwd_req.gr_n_data) + pr_info("svc-fwd-hpa: va=0x%llx gpa=0x%llx ap=%u hpa=0x%llx bi=%u\n", + dg->va, dg->pa, dg->ap, dhpa, i); + /* * GRVAUDIT (diagnostic only): audit the m180 landing * authority against the PA this submit will put on the wire * for every tracked data page - the data[]/n_data set the @@ -14446,7 +18160,7 @@ static void svc_fwd_gr_work(void) * permutation. Bounded: first 8 recs, 4 lines per launch, * boot budget 128. */ - if (svc_fwd_req.gr_n_data <= 64u && i < 8u && + if (svc_fwd_req.gr_n_data <= 1024u && i < 8u && svc_readdiv_log < 128u) { u64 opa = 0, lpa2 = 0; u32 oap = 0, lap2 = 0; @@ -14499,6 +18213,12 @@ static void svc_fwd_gr_work(void) gs.rel_hpa = rel_hpa; gs.rel_off = (u32)(svc_fwd_req.gr_rel_va & (SEMA_PAGE_SIZE - 1)); gs.rel_payload = svc_fwd_req.gr_rel_payload; + /* + * r378: carry the release VA so the backend can re-validate the + * release-word write targets (zero-on-retry, restore-on-accept) + * through gsp_sema_gr_write_guard before they land. + */ + gs.rel_va = svc_fwd_req.gr_rel_va; gs.sem_hpa = 0; gs.sem_off = 0; gs.sem_payload = 0; @@ -14522,6 +18242,17 @@ static void svc_fwd_gr_work(void) ga_va, ga_used, ga_led); hw_rc = gsp_phys_backend_guest_gr_submit(&gs, &cp); + /* + * r420b: the backend redirected the launch's QMD release0 to the + * module semaphore (rel_mod=1 or no targetable guest release page), so + * the ENGINE never writes the guest's own QMD release word. Treat the + * word as unobserved so the bounded outcome check (and the hold's + * edges) author it in software on every outcome; without this a + * rel_mod=1 launch with no deferred release records is never completed + * on the guest word and the client hangs (the r414 saxpy stall). + */ + if (cp.qmd_rel_redirected) + svc_fwd_req.gr_rel_unobserved = true; if (!hw_rc) { svc.fwd_gr_ok++; } else { @@ -14800,6 +18531,8 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) u64 pa = 0; u32 ap = 0; u32 i; + /* r372 diagnostic: which branch resolved this launch page. */ + const char *src = "?"; if (!page) return; @@ -14839,6 +18572,44 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) u64 lpa = 0; u32 lap = 0; + /* + * r372 pin-wins: a live injected pin is the guest's newest + * statement about this VA. If it disagrees with the landing + * table, the pin wins and the table is repaired (see the + * PIN-REPAIR note in svc_handle_map_add): the old landing was + * a freed page, and resolving the launch there wrote engine + * output onto memory the guest kernel had reused. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(page, &ipa, &iap)) { + u64 lpa2 = 0; + u32 lap2 = 0; + + if (svc_ledger_lookup(page, &lpa2, &lap2) || + ((lpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (ipa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap2 != (u32)iap) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS note va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + page, lpa2, lap2, ipa, iap); + svc_ledger_store(page, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + } + svc_fwd_req.gr_pages[i].pa = + ipa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)iap; + svc_pin_store(page, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + iap); + src = "ipin"; + goto note_done; + } + } if (!svc_ledger_lookup(page, &lpa, &lap)) { u64 rpa = 0; u32 rap = 0; @@ -14867,13 +18638,45 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) */ svc_pin_store(page, lpa & ~(u64)(SEMA_PAGE_SIZE - 1), (int)lap); - return; + pa = lpa; + ap = lap; + src = "ledger"; + goto note_done; } } if (svc_resolve_va(vgpu, page, &pa, &ap)) return; svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); + /* + * r370: store the landing table for EVERY launch page, not just the + * data-fill spans. An indirect/bank operand (the kernel's output + * buffer, qmd_word NONE) was resolved HERE, the engine wrote it via + * this hpa, the copy-out wrote it back to this hpa - and the client's + * D2H readback then resolved the same VA through the ledger/pin to a + * DIFFERENT page (an older landing), read stale zeros and the kernel + * looked wrong (victory_reduce sum=0 with the correct sum proven in + * the bounce page by OUT-PAGE-WIT). Recording this resolution makes + * the readback read exactly the page the engine wrote. + */ + svc_ledger_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)svc_ap_of(ap)); + svc_pin_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)svc_ap_of(ap)); + src = "walk"; +note_done: + /* + * r372 diagnostic: the exact (pa, ap) and branch the launch page + * resolved through - the value the engine's hpa and the copy-out use. + * A D2H readback that lands elsewhere (the sum=0 class) names its + * disagreement against these lines. + */ + if (svc_gr_pagenote_log < 512) { + svc_gr_pagenote_log++; + pr_info("sema-gr: page-note va=0x%llx pa=0x%llx ap=%u src=%s qmd=%u\n", + page, svc_fwd_req.gr_pages[i].pa, + svc_fwd_req.gr_pages[i].ap, src, qmd_word); + } } /* Does any captured inline write land in this page? */ @@ -14935,7 +18738,12 @@ static bool svc_gr_page_inline_covers(u64 page) * reach the engine. The legacy value is the pre-B-2 window below the mail * page (GSP_PHYS_RMCALL_GR_MAIL_SLOT = 206). */ -#define SVC_GR_DATA_BOUNCE_MAX GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES +/* + * fail4: each launch may relocate up to half of the grown region - the + * other half is the alternate base of a concurrently in-flight launch + * (see the per-capture base flip in svc_gr_data_fill). + */ +#define SVC_GR_DATA_BOUNCE_MAX (GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / 2u) #define SVC_GR_DATA_BOUNCE_MAX_OLD 205u static u32 svc_gr_data_log; /* m142: bounded per-page attribution log */ @@ -15154,6 +18962,127 @@ int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out) } /* + * r378 write guard, GR backend half (see gsp_sema_service.h). The GR submit + * resolves every operand page once and the copy-out/release writes run + * seconds later (the bounded poll, the retire wait, the outcome retries); a + * guest buffer freed and re-homed inside that window would receive a full + * page of another buffer's content through the stale HPA - the guest + * boot-corruption class (slab freelist clobbered by a foreign page write). + * + * Authority order mirrors the fill's own choice (r372/r376 pin-wins): a + * FRESH injected pin is the guest's newest statement, so the guard compares + * the write target against the pin's page first; only a pin-less VA falls + * back to the trusted resolver chain. The landing table is deliberately NOT + * the authority here - our own launches feed it, so it would confirm the + * target by construction. A miss on every source keeps the historical + * behaviour (the recorded landing is the write authority), so the guard can + * only remove provably stale writes. Bounded diagnostic budget. + */ +static u32 svc_wguard_log; /* r378 WGUARD-SKIP budget (256/boot) */ +bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 ipa = 0, cur_hpa = 0; + int iap = SVC_APERTURE_SYS; + u32 off = 0; + int ret; + + if (!va || !hpa) + return true; + if (!svc_ipin_lookup(page, &ipa, &iap)) { + u32 kind = svc_ap_of(iap) == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + + ret = svc_fwd_operand_hpa(svc_fwd_req.vgpu, kind, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + 0, SEMA_PAGE_SIZE, &cur_hpa, &off); + if (ret || off) + return true; /* pin page unresolvable: keep */ + } else { + struct gsp_sema_gr_reloc r; + + if (gsp_sema_gr_reloc_resolve(va, &r)) + return true; /* no current answer: keep */ + cur_hpa = r.hpa; + } + if ((cur_hpa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (hpa & ~(u64)(SEMA_PAGE_SIZE - 1))) + return true; + if (svc_wguard_log < 256) { + svc_wguard_log++; + pr_warn("vgsp-sema: WGUARD-SKIP path=%s va=0x%llx used=0x%llx now=0x%llx (stale write target)\n", + path, va, hpa, cur_hpa); + } + return false; +} + +/* + * r378 write guard, service half (see gsp_sema_service.h). A fresh injected + * pin is the guest's newest statement about a VA; a pin that disagrees with + * the recorded write target means the page was freed/re-homed (the r372/r376 + * pin-wins rule applied at the write instead of only at the resolution). A + * missing pin keeps the historical behaviour - the recorded landing is then + * the write authority - so no green path loses its write. + */ +bool gsp_sema_write_guard(u64 va, u64 pa, u32 ap, const char *path) +{ + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!va || !pa || va < SVC_VAS_FLOOR) + return true; + if (svc_ipin_lookup(va, &ipa, &iap)) + return true; /* no fresh pin: nothing newer to obey */ + if ((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + svc_ap_of(iap) == svc_ap_of((int)ap)) + return true; + if (svc_wguard_log < 256) { + svc_wguard_log++; + pr_warn("vgsp-sema: WGUARD-SKIP path=%s va=0x%llx used=0x%llx/%u pin=0x%llx/%d (stale write target)\n", + path, va, pa, ap, ipa, iap); + } + return false; +} + +/* + * r391 diagnostic (sum=0 class): the landing-table resolution of a GR data VA + * plus the first 8 bytes read through it - the exact page the D2H reader's + * ledger-first branch will use. Printed by the GR copy-out next to the write + * target, so a launch/readback disagreement is visible in one line. + */ +bool gsp_sema_gr_va_snap(u64 va, u64 *pa, u32 *ap, u64 *val8) +{ + u64 lpa = 0; + u32 lap = 0; + + if (svc_ledger_lookup(va, &lpa, &lap)) + return false; + *pa = lpa & ~(u64)(SEMA_PAGE_SIZE - 1); + *ap = lap; + *val8 = 0; + if (!svc_fwd_req.vgpu) + return true; + if (lap == SVC_APERTURE_FB) { + u8 *base = svc_fwd_req.vgpu->bars[1].vaddr; + size_t size = svc_fb_shadow_size(svc_fwd_req.vgpu); + + if (base && *pa + 8 <= size) + memcpy(val8, base + *pa, 8); + } else { + svc_read(svc_fwd_req.vgpu, *pa, val8, 8); + } + return true; +} + +/* r391 diagnostic: the shadow base this service resolves through (compare + * against the CE copy path's own vgpu->bars[1].vaddr in the same run). */ +u8 *gsp_sema_gr_shadow_base(void) +{ + return svc_fwd_req.vgpu ? svc_fwd_req.vgpu->bars[1].vaddr : NULL; +} + +/* * m144 (second-client launch-state divergence): collect device-VA-shaped * candidate pointers from the bytes the ENGINE will actually execute. * @@ -15258,13 +19187,68 @@ static bool svc_gr_hold_bound_expired(struct svc_channel *ch, u64 launch_va) launch_va = 1; /* anonymous launch: still absolutely bound */ if (!cap_ms) cap_ms = 3000; - if (ch->gr_hold_dead_va != launch_va) { + /* + * r368: mirror of the opdef fix - an ACTIVE bound is never re-armed + * by a different launch VA. The pump can hold several launches whose + * push VAs differ; arming on every VA change moved the deadline + * forever (the same identity bug class as the opdef=955/13/0 + * streams2 wedge). The first hold arms; others ride it; a fresh + * bound arms once the previous one has passed. + * + * r509: the bound must actually REPORT expiry. The historical arming + * rule re-armed the deadline on the very call that detected expiry + * and then compared against the NEW deadline - so the function could + * never return true, and a data-stalled launch whose arguments never + * resolved held its entry forever (r508 witness: matmul plus ten + * sibling channels in "GR hold ... data args unresolved" with zero + * GR-STALL-hold-bound lines for 10+ minutes; the designed + * retry-forward + consume never ran). The expiry is now one-shot: + * push the deadline, latch gr_hold_bound_hit for the r351 block of + * the same pass, return true. A fresh arm clears the latch (and the + * once-per-launch retry log). + */ + if (!ch->gr_hold_dead_va || + time_after_eq(jiffies, ch->gr_hold_deadline)) { ch->gr_hold_dead_va = launch_va; ch->gr_hold_dead_since = jiffies; ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); ch->gr_hold_exp_logged = false; + ch->gr_hold_bound_hit = false; + return false; } - return time_after_eq(jiffies, ch->gr_hold_deadline); + /* + * r420: pause the per-launch consumption bound while the backend + * deliberately holds the in-flight launch in its bounded WEDGE-QUIET + * window. At the bound the pump falls through to the historical + * consume, which authors the capture's QMD completion release in + * software (GR-STALL-hold) - but the backend is still going to + * submit/accept this very launch after the hold and copy its data + * out; authoring now releases the client onto stale data (the r419 + * mid-boot escape). The quiet hold is a bounded msleep (the + * backend's own caps), and the deferral here is bounded by + * SVC_GR_QUIET_PAUSE_MAX_MS from the bound's arm time as + * defence-in-depth. Rate-limited announcement. + */ + if (ch->gr_hold_dead_va && gsp_phys_gr_quiet_hold_active() && + time_before(jiffies, ch->gr_hold_dead_since + + msecs_to_jiffies(SVC_GR_QUIET_PAUSE_MAX_MS))) { + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + pr_info_ratelimited("vgsp-gr: GR-HOLD-BOUND-PAUSE launch=0x%llx (backend wedge-quiet hold active; consumption bound deferred)\n", + launch_va); + return false; + } + if (time_after_eq(jiffies, ch->gr_hold_deadline)) { + /* + * r509: one-shot expiry. Push the deadline so the hold window + * restarts for the caller's retry, latch the verdict for the + * second call site of this pass (the r351 retry-forward + * block), and report expired. + */ + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + ch->gr_hold_bound_hit = true; + return true; + } + return false; } static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, @@ -15282,8 +19266,32 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, * window below the mail page for A/B trialing. */ bool region_on = gsp_phys_gr_data_region_value() != 0; - u64 reloc_base = region_on ? SVC_GR_DATA_RELOC_BASE : - SVC_GR_DATA_RELOC_BASE_OLD; + /* + * fail4: two concurrent forwarded launches (streams2's vadd pair) + * both relocated their operands from the same base, so the second + * launch's copy-in overwrote the first's bounce windows and each + * engine read the other's inputs (c1 came out as a2+b2). Alternate + * the per-launch base across the grown region's two halves (32 MB + * each, above any single launch's span), keyed to the capture push + * so a retried launch keeps its half, so concurrent launches + * cannot collide. + */ + static u64 svc_gr_reloc_half_key; + static u8 svc_gr_reloc_half; + u64 reloc_off = 0; + + if (region_on) { + if (svc_gr_cand.push_va != svc_gr_reloc_half_key) { + svc_gr_reloc_half_key = svc_gr_cand.push_va; + svc_gr_reloc_half ^= 1; + } + if (svc_gr_reloc_half) + reloc_off = (u64)(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / + 2u) * SEMA_PAGE_SIZE; + } + u64 reloc_base = region_on ? + SVC_GR_DATA_RELOC_BASE + reloc_off : + SVC_GR_DATA_RELOC_BASE_OLD; u32 per_ptr = region_on ? SVC_GR_DATA_PAGES_PER_PTR : SVC_GR_DATA_PAGES_PER_PTR_OLD; u32 bounce_max = region_on ? SVC_GR_DATA_BOUNCE_MAX : @@ -15639,6 +19647,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, wins[w], ((u64)p + 1) << 12, (u64)p << 12); + /* + * r377 pin-demand: name every + * unresolved operand page as a + * stalled VA. The userspace pin + * stream watches dmesg for this + * exact line and pushes the VA + * to the in-guest scanner's + * demand file, so the page gets + * pinned and the bounded launch + * hold (gsp_phys_gr_fill_hold) + * resumes on real data instead + * of waiting out its bound (the + * r345 wedge: held forever + * because nobody demanded the + * page). + */ + svc_log_stalled_va(va); if (dropped_pages < 4) { drop_va[dropped_pages] = va; drop_kind[dropped_pages] = @@ -15662,8 +19687,34 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (va >= SVC_VAS_FLOOR) { u64 lpa = 0; u32 lap = 0; + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; - if (!svc_ledger_lookup(va, &lpa, &lap) && + /* + * r372 pin-wins: when a live pin exists it is the + * newest truth; the landing table may name a page + * the guest has since freed (VA reuse across + * clients). Keep the pin and repair the table. + * Only a pin-less VA still follows the table. + */ + if (!svc_ipin_lookup(va, &ipa, &iap)) { + pa = ipa; + ap = (u32)iap; + if (!svc_ledger_lookup(va, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (ipa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)iap)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + va, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, ipa, iap); + svc_ledger_store(va, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + } + } else if (!svc_ledger_lookup(va, &lpa, &lap) && ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || (u32)lap != (u32)ap)) { @@ -15699,12 +19750,27 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, rec->ap = (u8)svc_ap_of(ap); svc_fwd_req.gr_n_data++; /* + * r349: record the launch's own resolution as the + * ledger's landing authority for this VA. The read + * path's m171 read-after-write rule follows the + * ledger when it disagrees with the live translation, + * and the engine writeback below writes exactly this + * (pa, ap). Without the store the ledger kept a stale + * page for engine-written outputs (victory_reduce: + * the launch wrote FB 0x7c00000 while the ledger + * said SYS 0x7c00000; the D2H read followed the + * ledger and returned zeros while the correct sum + * sat on the written page). + */ + if (va >= SVC_VAS_FLOOR) + svc_ledger_store(va, rec->pa, rec->ap); + /* * m142: bounded per-page attribution log. A wrong data * page is invisible in the aggregate counters; these * lines let a post-mortem compare the chosen PA against * the guest's own mapping. */ - if (svc_gr_data_log < 16) { + if (svc_gr_data_log < 192) { svc_gr_data_log++; pr_info("vgsp-sema: svc-gr data page va=0x%llx -> pa=0x%llx ap=%u\n", va, rec->pa, rec->ap); @@ -15759,7 +19825,7 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, wrc = svc_gmmu_walk(vgpu, svc.vas[vi].pdb, svc.vas[vi].entries, base, svc.vas[vi].ap, &wa, &wap, - false); + gsp_sema_walk_trace != 0); o += snprintf(vb + o, sizeof(vb) - o, " vas[%d] pdb=0x%llx ent=%u ap=%d rc=%d pa=0x%llx", vi, svc.vas[vi].pdb, @@ -16047,8 +20113,23 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * The candidate is per-channel: a capture left unconsumed by an * earlier pass (e.g. the entry was held before its request) must never * be queued against a different channel's handle by that channel's - * own end-of-pass request call. + * own end-of-pass request call. fail4: with several reserved ring + * slots, select THIS channel's oldest unconsumed candidate first. */ + if (gsp_phys_gr_forward_enabled()) { + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % SVC_GR_CAND_RING; + + if (svc_gr_cand_ring[rp].valid && + !svc_gr_cand_ring[rp].offered && + svc_gr_cand_ring[rp].h_object == ch->h_object) { + svc_gr_cand_pos = rp; + break; + } + } + } if (!gsp_phys_gr_forward_enabled() || !svc_gr_cand.valid || svc_gr_cand.h_object != ch->h_object) return false; @@ -16161,6 +20242,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, mch->gr_hold_dead_since)); } svc_gr_data_stalled = false; /* proceed: forward */ + if (svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-gr: GRFILL-PROCEED launch=0x%llx (bound expired; forwarding best-effort)\n", + lva); + } } } if (svc_gr_data_stalled) { @@ -16548,7 +20634,7 @@ static bool svc_push_read_displaced(struct nv_vgpu_device *vgpu, u32 h, static void svc_entry_outcome(const char *what, const struct svc_channel *ch, u64 idx, u64 va, u32 len) { - if (!gsp_sema_dbg || !ch->engine_type || svc.gpfifo_entry_log >= 512) + if (!gsp_sema_dbg || svc.gpfifo_entry_log >= 16384) return; svc.gpfifo_entry_log++; pr_info("sema-entry: %s h=0x%x idx=%llu va=0x%llx len=%u want=%u exp=%u unread=%d\n", @@ -16638,121 +20724,52 @@ static void svc_userd_sweep(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * SVC_GR_STALE_WEDGE_MS (the (last_get, put) pair unchanged across the * window): * - * 1. stale epoch (host caught up, last_get == put): the ring still - * carries a session's residue (put > 1). The host resets the stale - * publish state - zero the host-owned completion word and - * counter_last (m181's resync; the residual put is NEVER published, - * see the r207 far-jump note in svc_counter_publish) - and, when the - * whole consumed range delivered nothing (counter_last == 0 and - * submissions >= 8), additionally rewinds last_get to slot 0 - * (reason=stale-dead-range) so the fresh session's own entries, which - * start at slot 0, are walked. The forced-replay marker - * (wd_nocap_put) makes that re-walk decode releases only (idempotent, - * forward-only) and never re-capture/re-forward a dead launch. - * 2. stale replay: last_get pinned behind put while put == get (the + * 1. stale-epoch: a GR channel that still carries a residual put + * (put > 1 at this pass, publish withheld) and whose ring the host + * believes is fully consumed (last_get == put) - the fresh session + * never proved its switch with a put drop. The host applies the + * m181 recovery on a TIMEOUT instead: arm the counter publish and + * void the withheld publish state (no last_get rewind - a timeout is + * not proof the ring is the dead session's; the proven m181 rewind + * stays reserved for its put-drop signature). This is the recovery + * the killed-client rerun needs. + * + * 2. stale-replay: last_get pinned behind put while put == get (the * producer has nothing pending) AND the loop consumed entries during - * the window (submissions moved). The cursor is clamped to put once - * per put epoch (bookkeeping only). The pin-batch rewind is NEVER - * touched: r235 proved disabling it kills the next client's - * context-init (its pins land after the first decode; the rewind is - * the retry path). A clamp that a rewind undoes is harmless. + * the window (submissions moved) - i.e. the rewind is re-opening an + * already-consumed window faster than the loop closes it. The window + * is dropped (last_get = put) and the pin-batch rewind is put on a + * cooldown (wd_rewind_off_until) so the futile retry cannot pin the + * pump; the rewind budget returns later and a translation that lands + * can still retry (bounded duty cycle). * - * Nothing here can fire on a green channel: a green channel either moves - * put or consumes its first entries within milliseconds (submissions > 0), - * so the no-progress window never forms; the stale-replay leg - * additionally requires entries consumed during a stalled window, which - * only the rewind re-opening of a consumed window produces. + * Nothing here can fire on a green channel: a fresh channel has put <= 1 + * (signature 1) or consumes its first entries within milliseconds + * (submissions > 0); signature 2 additionally requires BOTH a stalled + * (last_get, put) pair for 3 s with no pending producer work AND consumed + * entries during that window, which only the rewind re-opening of a + * consumed window produces. */ #define SVC_GR_STALE_WEDGE_MS 3000u -/* - * STALLFREE r235: the replay/clamp leg needs a MUCH longer stalled window - * than the epoch legs. A fresh client's context-init legitimately shows a - * 3 s position oscillation (rewind replays a just-consumed entry while its - * pins land); treating that as a livelock killed the second client (r235: - * 8 channels classified at waited_ms=3491). Only a position stalled for - * ~10 s with consumption happening is a livelock worth clamping. - */ -#define SVC_GR_STALE_REPLAY_MS 10000u -/* - * STALLFREE r229/r230: freshness window after a genuinely new channel - * binding. Within it, a GR entry whose page is unmapped or whose decode - * yields nothing is NOT silently consumed: the pump requests an immediate - * rebuild/pin pass and leaves the entry pending, so the first launch of a - * fresh client executes on its FIRST attempt instead of being dropped - * before the page translations land (r229/r231: attempt 1 stalls, attempt - * 2 works because the rebuild caught up). Stale-free cost: at most this - * window of extra latency on first contact. - */ -#define SVC_GR_NEWBIND_MS 3000u - static void svc_gr_stale_wedge_check(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u32 put, u32 get) { - bool catchup, residual, dead_reset, replay_cand, stalled_cand; - bool fire_replay, fire_reset, fire_stall; + bool residual, replay; const char *reason = ""; unsigned long waited; if (ch->engine_type || !ch->gpfifo_entries) return; - /* - * New producer work (put advanced) clears the per-put-epoch latches - * and the replay cooldown: the next stale region gets its own - * recovery. - */ - if (ch->wd_put != put) - ch->wd_epoch_done = false; - /* The forced-replay re-walk is done once the walk catches up. */ - if (ch->wd_nocap_put && ch->last_get == put) - ch->wd_nocap_put = 0; - + /* New work (put advanced) clears the replay cooldown immediately. */ + if (ch->wd_rewind_off_until && ch->wd_put != put) + ch->wd_rewind_off_until = 0; if (ch->wd_since && (ch->wd_put != put || ch->wd_last_get != ch->last_get)) ch->wd_since = 0; /* progress (or a new position): re-arm */ - - catchup = (ch->last_get == put); - replay_cand = (!catchup && put == get && !ch->wd_epoch_done); - /* - * Stale epoch: the host caught up (last_get == put) but the ring - * still carries a session's residue: put > 1 and the epoch reset - * latch is unset. Two additive recoveries: - * - residual: the host published its own value for this channel at - * some point (counter_last != 0), so reset the stale publish - * state (zero the word, m181 semantics). - * - dead-reset: nothing was ever published for this channel - * (counter_last == 0) and the consumed range delivered nothing - * while being non-trivial (submissions >= 8): the whole walked - * range was the dead session's ring. Rewind last_get to 0 so the - * fresh session's own entries (which start at slot 0) get walked; - * the re-walk forces replay so a dead launch is never re-captured - * or re-forwarded. - */ - residual = (catchup && put > 1 && !ch->wd_epoch_done && - !ch->gr_pub_armed && ch->counter_last != 0); - dead_reset = (catchup && put > 1 && !ch->wd_epoch_done && - !ch->gr_pub_armed && ch->counter_last == 0 && - ch->submissions >= 8); - /* - * STALLFREE r228: stalled PENDING entry. The producer has submitted - * work the host has not consumed (put > get) and the entry loop has - * made no progress - position AND submissions static across the - * window - while no bounded owner is active for the stuck entry: no - * unreadable-push grace (defer_since), no no-capture window, no - * opdef operand slot, no m150 hold window. That combination is a - * genuinely unexplained stall (r228: the opdef deadline kept being - * reset by rewind replays, fixed in svc_opdef_drop_stale; this leg - * is the safety net for any future variant that slips the bounded - * owners). A green channel always advances submissions (consume) or - * sets one of the bounded owners on a pending entry, so the leg - * cannot fire there. - */ - stalled_cand = (!catchup && put != get && !ch->defer_since && - !ch->gr_nocap_since && - !(ch->gr_hold_idx == (u32)ch->last_get && - time_before(jiffies, ch->gr_hold_until)) && - !svc_opdef_find(vgpu, ch->h_object, - (u32)ch->last_get)); - if (!residual && !dead_reset && !replay_cand && !stalled_cand) { + residual = (!ch->gr_pub_armed && put > 1 && ch->last_get == put); + replay = (ch->last_get != put && put == get && + ch->wd_since && ch->submissions != ch->wd_sub); + if (!residual && !replay) { ch->wd_since = 0; ch->wd_put = put; ch->wd_last_get = ch->last_get; @@ -16769,98 +20786,86 @@ static void svc_gr_stale_wedge_check(struct nv_vgpu_device *vgpu, msecs_to_jiffies(SVC_GR_STALE_WEDGE_MS))) return; waited = jiffies_to_msecs(jiffies - ch->wd_since); - fire_reset = dead_reset; - /* - * The replay leg only fires with proof of a rewind livelock: entries - * were consumed during the window while the position did not move - * (the rewind re-opened an already-consumed window). Without that - * proof this is a producer-pending channel on an unreadable entry, - * which the existing grace paths own. - */ - fire_replay = replay_cand && ch->submissions != ch->wd_sub && - waited >= SVC_GR_STALE_REPLAY_MS; - /* - * The stalled-entry leg fires only with proof of no progress at all: - * nothing was consumed during the window (submissions static), so - * the entry is genuinely parked with no bounded owner. - */ - fire_stall = stalled_cand && ch->submissions == ch->wd_sub; - if (!residual && !fire_reset && !fire_replay && !fire_stall) { - /* - * Keep an accumulating replay window alive (its evidence needs - * SVC_GR_STALE_REPLAY_MS); anything else restarts. - */ - if (!replay_cand) - ch->wd_since = 0; - return; - } - if (residual || fire_reset) { + if (residual) { /* - * Reset the stale publish state. The residual put is NEVER - * published: writing a far-forward value into the fresh - * client's completion word is the r207/mM2 far-jump wedge - * (UVM_GPU_SEMAPHORE_MAX_JUMP), which is exactly what - * gr_pub_armed exists to prevent. Only a word the host itself - * published this session is zeroed (m181 semantics) so a - * channel the host never wrote (counter_last == 0) is left - * untouched; the arm stays reserved for the proven put-drop - * (m181) and the pristine put <= 1 discovery. + * m181 recovery on timeout: the fresh session never dropped + * the residual put, so prove the epoch switch here. Arm the + * publish (the withheld completion counter was the plausible + * ctx-init wait - r225's client polled service-interrupts 1/s + * with no host edge) and void any publish state the stale + * pass advanced, exactly like m181; the guest word itself is + * only zeroed when the host authored it. No last_get rewind: + * a timeout is not proof that the ring is the dead session's + * (re-walking it could re-capture a stale launch), so the + * proven m181 rewind stays reserved for its put-drop + * signature; the client's own first put write will trigger it. */ + ch->gr_pub_armed = true; if (ch->counter_last) { u64 upa = ch->release_pa ? ch->release_pa : ch->counter_pa; u32 zero = 0; - if (upa) - (void)svc_write_ap(vgpu, upa, SVC_APERTURE_SYS, - &zero, sizeof(zero)); + /* + * r311: zero only a word authored by a DEAD + * generation. Zeroing a word this live session + * authored makes the client's UVM observe a + * downward jump (its tracker then never completes) + * while the publish re-advances it - the r311 + * vector_add wedge (pub-adv/zero cycles, client + * polling cur=0 forever). + */ + if (ch->counter_gen == svc_sess_gen) { + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: stale-recover-zero-skip h=0x%x upa=0x%llx last=%u (authored by live gen)\n", + ch->h_object, upa, + ch->counter_last); + } + ch->counter_last = 0; + } else if (upa) { + int zrc; + + /* + * r378: the zero is authored from a PA + * learned earlier (release_va -> release_pa); + * re-validate it before the 4-byte write + * lands so a freed/re-homed slot is never + * zeroed through the stale page. + */ + if (!gsp_sema_write_guard(ch->release_va, + upa, + SVC_APERTURE_SYS, + "stale-recover-zero")) + zrc = -ESTALE; + else + zrc = svc_write_ap(vgpu, upa, + SVC_APERTURE_SYS, + &zero, sizeof(zero)); + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: stale-recover-zero h=0x%x upa=0x%llx last=%u zrc=%d\n", + ch->h_object, upa, + ch->counter_last, zrc); + } + } ch->counter_last = 0; } - } - if (fire_reset || fire_stall) { - /* - * The canonical unstuck: rewind to slot 0 so the fresh - * session's own entries (which start there) get walked, with - * forced replay (wd_nocap_put) so the re-walk decodes releases - * only and never re-captures a launch; once per put epoch. - */ - ch->wd_epoch_done = true; - ch->last_get = 0; - ch->wd_nocap_put = put; - } - if (fire_replay) { - /* - * STALLFREE r235: clamp the stale software cursor to the - * producer position ONCE per put epoch. The pin-batch rewind - * is deliberately NOT touched: disabling it killed every - * second client (r235-2: 8 channels got a 24 s rewind - * cooldown at ctx-init while their pins were landing, so - * their init pushes were never retried). The clamp is - * bookkeeping only and rewind retries stay available. - */ + reason = "stale-epoch"; + } else { + /* Drop the rewind-reopened consumed window; cool the rewind. */ ch->last_get = put; - ch->wd_epoch_done = true; + ch->wd_rewind_off_until = jiffies + + msecs_to_jiffies(8u * SVC_GR_STALE_WEDGE_MS); + reason = "stale-replay"; } - if (residual || fire_reset || fire_replay) - ch->wd_epoch_done = true; ch->wd_since = 0; ch->wd_put = put; ch->wd_last_get = ch->last_get; svc.gr_stale_wedge++; - if (fire_replay) - reason = "stale-replay"; - else if (fire_reset) - reason = "stale-dead-range"; - else if (fire_stall) - reason = "stale-entry"; - else - reason = "stale-epoch"; - pr_warn_ratelimited("vgsp-gr: GR-STALL-stale-wedge h=0x%x va=0x%llx last=%u put=%u get=%u sub=%llu waited_ms=%u action=recover reason=%s reset=%d clamp=%d entry=%d cpa=0x%llx rpa=0x%llx\n", + pr_warn_ratelimited("vgsp-gr: GR-STALL-stale-wedge h=0x%x va=0x%llx last=%u put=%u get=%u sub=%llu waited_ms=%u action=recover reason=%s\n", ch->h_object, ch->gpfifo_va, (u32)ch->last_get, - put, get, ch->submissions, waited, reason, - (fire_reset || fire_stall) ? 1 : 0, - fire_replay ? 1 : 0, fire_stall ? 1 : 0, - ch->counter_pa, ch->release_pa); + put, get, ch->submissions, waited, reason); } static void svc_pump_channels(struct nv_vgpu_device *vgpu) @@ -16878,8 +20883,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) bool stalled = false; spin_lock_irqsave(&svc.chan_lock, flags); + /* + * r507o stream-order: walk the CE channels BEFORE the GR channels. + * The client's stream order places its H2D copies before the kernel + * that consumes them, but the channels register in handle order (the + * RM's GR channel early, the copy-ce channels later), and a pass that + * decoded the launch before the CE channel's entries ran let the + * forward workqueue execute the launch's copy-in against pre-copy + * data (r480-sep: copy-in 115 ms before its stream's H2D soft + * copies; r481 proved no host semaphore links the two, so the + * emulation must enforce stream order itself). The CE copies execute + * on this same service thread, so consuming the CE entries first + * fixes the inversion directly - no cross-channel hold, no latency + * tax. Stable partition; relative order within each class preserved. + */ for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].engine_type) + local[n++] = svc.chan[i]; + } + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + !svc.chan[i].engine_type) local[n++] = svc.chan[i]; } spin_unlock_irqrestore(&svc.chan_lock, flags); @@ -17238,12 +21263,17 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } } bitmap_zero(ch->slot_done_epoch, SVC_GPFIFO_MAX_BITS); + memset(ch->slot_pend_tries, 0, + sizeof(ch->slot_pend_tries)); /* GRFIX FIX1b: a recycled slot in the next put epoch * gets a fresh absolute-bound record; without this a * wrapped slot would inherit a stale expired launch * record. */ ch->gr_hold_dead_va = 0; ch->gr_hold_exp_logged = false; + /* r361: same for the opdef absolute bound. */ + ch->opdef_hold_dead_va = 0; + ch->opdef_hold_exp_logged = false; } put_prev_wit = ch->put_prev; ch->put_prev = put; @@ -17257,47 +21287,6 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc_gr_stale_wedge_check(vgpu, ch, put, get); /* - * STALLFREE r229: first-use reconciliation after a genuinely - * new binding (see svc_chan_registration). The rebinding's - * ring cursor starts at 0 (fresh sequence); drop the dead - * context's opdef slots under this handle (their waiters are - * gone and the sets must not ride the reused channel, the - * r228 opdef-hold class), and when the ring already carries - * content (put > 1) log the reconcile so the transition is - * visible. Captures stay enabled: the fresh client's own - * entries may already be at slot 0.., and suppressing them - * would lose exactly the first launch this must protect; a - * stale launch re-walk is bounded by the submit budget and - * the m181 put-drop recovery (same precedent). - */ - if (ch->wd_chanswitch) { - ch->wd_chanswitch = false; - svc_opdef_drop_channel(vgpu, ch->h_object, jiffies); - ch->wd_newbind_until = jiffies + - msecs_to_jiffies(SVC_GR_NEWBIND_MS); - /* - * Bounded proof (r234 asks): the window was armed for - * THIS binding; if a later stall shows no such line, - * its binding never reached the pump (pre-registration - * class) and this mechanism cannot apply. - */ - if (svc.gr_newbind_log < 16) { - svc.gr_newbind_log++; - pr_info("vgsp-gr: GR-STALL-newbind h=0x%x put=%u get=%u action=window reason=stale-entry-newbind\n", - ch->h_object, put, get); - } - if (put > 1) { - svc.gr_stale_wedge++; - pr_warn_ratelimited("vgsp-gr: GR-STALL-stale-wedge h=0x%x va=0x%llx last=%u put=%u get=%u sub=%llu waited_ms=0 action=recover reason=stale-entry-chanswitch reset=0 rewind=0 entry=0 cpa=0x%llx rpa=0x%llx\n", - ch->h_object, ch->gpfifo_va, - (u32)ch->last_get, put, get, - ch->submissions, - ch->counter_pa, - ch->release_pa); - } - } - - /* * m136 (SA-STALL M2): liveness watchdog for a held CE entry. * When this channel's entry loop is not re-entered (early * continue paths above, or no fresh pushes), a deferred entry @@ -17378,7 +21367,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * (forward-only, so it cannot double-apply) and drop * it before this entry is examined. */ - svc_opdef_drop_stale(vgpu, ch, (u32)idx); + svc_opdef_drop_stale(vgpu, ch, (u32)idx, put); /* * A pin-batch rewind replays recently consumed @@ -17448,10 +21437,17 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ { u64 ident = (u64)e0 | ((u64)e1 << 32); - bool same = ident && + /* + * r505b: the per-slot op bookkeeping arrays are + * SVC_GPFIFO_MAX_BITS (1024) entries; the X/GL + * channel's ring is 8192 slots. Bound every slot + * index here or a large ring writes out of + * bounds. + */ + bool same = ident && idx < SVC_GPFIFO_MAX_BITS && ch->slot_ops_ident[idx] == ident; - entry_ident = ident; + entry_ident = (idx < SVC_GPFIFO_MAX_BITS) ? ident : 0; /* * A complete prior pass suppresses every op. * A partial pass (an operand's targeted scan @@ -17459,22 +21455,23 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * leaves the per-op done bitmap to decide: * work that already ran must not run again, * the pending op must get its retry. + * + * r354: the content witness decides (the hash + * override sits below, after the push bytes + * are read): entry words alone cannot tell a + * true rewind replay (identical bytes - must + * not run twice) from a REWRITTEN push that + * reuses the same push-buffer VA and length + * with new methods (the driver's repeated + * page-table/fill pushes at the same ring + * slots - suppressing those left the guest's + * GPU page tables stale, the GMMU walk + * missing operand pages, and the first CUDA + * client of every fresh boot failing with + * dropped operands). */ entry_replay = same && ch->slot_ops_complete[idx]; - /* - * STALLFREE r227: while a stale-epoch recovery - * re-walks a dead session's consumed range - * (wd_nocap_put set by the watchdog), force - * replay for the dead slots: their releases - * are re-decoded/applied forward-only - * (idempotent) but a dead launch is never - * re-captured or re-forwarded. Cleared by the - * watchdog when the walk catches up. - */ - if (!entry_replay && ch->wd_nocap_put && - (u32)idx < ch->wd_nocap_put) - entry_replay = true; if (same) entry_done = ch->slot_ops_done[idx]; } @@ -17563,7 +21560,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) injected = (svc_ipin_lookup(va, &ipa, &iap) == 0); - if (!svc_method_stream_plausible(pb, len)) { + /* + * r496e: an all-zero push on a + * content-scanned-ring channel is the + * driver's NOP sanity push (one zero + * dword - NVA06F_DMA_NOP), not a bad + * translation: accept it and let the + * decode find no ops. + */ + bool zero_nop = ch->gr_ram_ring && + len >= 4 && + !memchr_inv(pb, 0, len); + + if (!svc_method_stream_plausible(pb, len) && + !zero_nop) { svc.false_trans++; if (injected) { u8 *fb = kmalloc(len, @@ -17754,6 +21764,39 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ bool cap_trust = injected || rehold; + /* + * r354: the content witness for + * the once-only op gate. Entry + * words alone cannot tell a true + * rewind replay (identical bytes) + * from a rewritten push that + * reuses the same push VA and + * length with new methods (the + * driver's repeated fill / + * page-table pushes at the same + * ring slots). Suppressing the + * rewritten push left the GPU + * page tables stale and the first + * client's operands unresolved. + * slot_ops_hash has been stored + * since r62 for this comparison. + */ + if (entry_replay && + ch->slot_ops_hash[idx] != + svc_push_hash(pb, len)) { + entry_replay = false; + if (svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-sema: rewritten push runs (idx=%llu va=0x%llx len=%u)\n", + (unsigned long long)idx, va, len); + } + } + if (rehold && !injected && + svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-gr: GR rehold capture idx=%llu va=0x%llx\n", + (unsigned long long)idx, va); + } if (!cap_trust && !entry_replay && !ch->engine_type && gsp_phys_gr_forward_enabled() && @@ -17781,41 +21824,6 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (rehold) ch->gr_rehold_idx = ~0u; /* - * STALLFREE r230: a fresh - * binding's entry that decoded - * NOTHING (no release, no QMD, - * no ops) is the r229 attempt-1 - * class: the page content read - * before the client's real - * writes/translations landed, so - * consuming it would drop the - * launch. Hold it for the short - * new-binding window and ask for - * an immediate rebuild; after the - * window it consumes as before, so - * the cost is bounded. - */ - if (!ch->engine_type && - !entry_replay && - svc_rel_count == rel_n_iter && - svc_gr_qmd_seen == 0 && - ch->wd_newbind_until && - time_before(jiffies, - ch->wd_newbind_until)) { - if (!ch->defer_since) - ch->defer_since = - jiffies; - svc.rebuild_now = true; - if (svc.gr_pending_map < 16) { - svc.gr_pending_map++; - pr_warn("vgsp-gr: GR-STALL-pending-map h=0x%x va=0x%llx entry=%u action=rebuild reason=stale-entry-newbind\n", - ch->h_object, va, - (u32)idx); - } - kfree(pb); - break; - } - /* * Record what ran for this entry * identity before any hold path can * break out: a held entry is decoded @@ -17862,8 +21870,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ if (svc.op_tr_pending && svc_opdef_hold(vgpu, ch, - (u32)idx)) { + (u32)idx, + va)) { svc.gpfifo_nomap++; + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-pend: hold h=0x%x idx=%u va=0x%llx\n", + ch->h_object, (u32)idx, va); + } /* * P6b wave 1.5: * releases recorded @@ -17887,7 +21901,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(pb); break; } - if (svc.op_tr_pending) + if (svc.op_tr_pending) { /* * Deadline hit: the * operand never @@ -17899,6 +21913,34 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc_opdef_flush(vgpu, ch, (u32)idx); + /* + * r505b: mark the slot's + * ops complete. The op was + * permanently skipped, but + * the decode above stored + * complete=0 (op_tr_pending + * was set), so every pin- + * batch rewind replay of + * this slot re-decoded it, + * re-set op_tr_pending and + * re-held the entry in a + * livelock (multikernel on + * r505: opdef held=18359, + * expired=61, the client + * waiting forever while the + * pump looped on slot 1). + * Nothing is owed after the + * flush, so a replay must + * skip the ops. + */ + if ((u32)idx < + SVC_GPFIFO_MAX_BITS) { + ch->slot_ops_complete[idx] = + 1; + ch->slot_ops_done[idx] = + entry_done; + } + } rel_seen = (svc_rel_count != rel_n_before); { bool fwd_guest_queued; @@ -17990,7 +22032,22 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc_gr_launch_va(ch)) && (ch->gr_hold_idx != (u32)idx || time_before(jiffies, - ch->gr_hold_until))) { + ch->gr_hold_until) || + /* + * r420: the backend's + * bounded WEDGE-QUIET + * hold outlives this + * one-shot window; keep + * re-holding (never + * consume-with-sw-release) + * while it is active, so + * the launch is still + * forwarded and copied out + * after the hold instead of + * being released onto stale + * data. + */ + gsp_phys_gr_quiet_hold_active())) { u32 hold_ms = gsp_sema_gr_grace_ms ? gsp_sema_gr_grace_ms : 8000; @@ -18119,15 +22176,49 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!fwd_guest_queued && !ch->engine_type && svc_gr_data_stalled && - svc_gr_hold_bound_expired(ch, - svc_gr_launch_va(ch)) && + ch->gr_hold_bound_hit && !ch->gr_hold_exp_logged) { ch->gr_hold_exp_logged = true; + /* r509: the latch is this pass's only + * report of the expiry; clear it so the + * NEXT expiry cycle (gr_hold_exp_logged + * now set) falls through to the + * historical consume. */ + ch->gr_hold_bound_hit = false; svc.fwd_gr_hold_exp++; - pr_warn("vgsp-gr: GR-STALL-hold-bound va=0x%llx waited_ms=%u action=proceed (data-stall hold bound expired; entry consumed, QMD release authored)\n", + /* + * r351: the bound firing used to + * fall through to the historical + * consume (entry consumed, QMD + * release authored) WITHOUT ever + * forwarding the launch - the + * client's kernel never ran and + * the client hung (victory_reduce + * with gsp_phys_gr_fill_hold=1: + * bound fired every 3 s, zero GR + * LAUNCH lines, 110 s+ hang). + * Instead, force ONE more fresh + * capture (rehold) with the stall + * verdict cleared: the request's + * own bound is expired too, so it + * proceeds and forwards the launch + * best-effort with whatever + * resolved. If that retry cannot + * run, the next pass (flag set) + * falls through to the historical + * consume as before. + */ + pr_warn("vgsp-gr: GR-STALL-hold-bound va=0x%llx waited_ms=%u action=retry-forward (bound expired; one best-effort forward pass)\n", svc_gr_launch_va(ch), jiffies_to_msecs(jiffies - ch->gr_hold_dead_since)); + svc_gr_data_stalled = false; + ch->gr_rehold_idx = (u32)idx; + ch->gr_hold_idx = (u32)idx; + ch->gr_hold_until = jiffies + + msecs_to_jiffies(200); + kfree(pb); + break; } /* * P6b wave 1.5: when @@ -18241,10 +22332,81 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * through to the historical * consume so the channel can * never livelock. + * + * r470: an ALL-ZERO push page is + * the same "not the client's work + * yet" condition - the ring entry + * is written (the previous + * session's or the client's own + * slot reuse) but the push + * CONTENT has not landed, so the + * decode sees no QMD and the + * entry would be consumed as a + * no-op; the end-of-pass counter + * publish then advances the + * guest's completion word while + * the real push was never read, + * and the client drains a + * staging buffer that still holds + * the PREVIOUS copy's bytes + * (streams_events: h_c[2] = c0 + * because the c2 D2H's + * completion was published 4.5 s + * before its copy ran, the ring + * slots having been decoded as + * all-zero pushes). Deliberately + * NARROWER than the CE-shaped + * plausibility verdict: compute + * pushes legitimately fail that + * matcher, so only an entirely + * zero page counts as unwritten. */ + { + bool all_zero = false; + + if (!ch->engine_type && + va && len) { + u32 zi; + + all_zero = true; + for (zi = 0; zi < len; + zi++) + if (pb[zi]) { + all_zero = + false; + break; + } + } + /* + * r471: publish clamp bookkeeping. + * A zero-content GR entry that is + * held or consumed must never let + * the end-of-pass counter publish + * advance the guest's completion + * word past it (the client would + * satisfy its readback wait before + * the real push/copy ran - the + * staging buffer still holds the + * previous copy's bytes). Record + * the lowest zero slot; a decode + * that reads non-zero content + * clears the clamp. + */ + if (!ch->engine_type) { + if (all_zero) { + if (!ch->gr_bad_put || + (u32)idx + 1 < + ch->gr_bad_put) + ch->gr_bad_put = + (u32)idx + 1; + } else if (va && len) { + ch->gr_bad_put = 0; + } + } if (!ch->engine_type && va && len && gsp_phys_gr_forward_enabled() && - svc_gr_qmd_seen > 0 && + (svc_gr_qmd_seen > 0 || + all_zero) && !entry_replay && !(svc_gr_cand.valid && svc_gr_cand.h_object == @@ -18263,6 +22425,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) jiffies; } if (gsp_sema_gr_nocap_hold_ms && + !ch->gr_ram_ring && time_before(jiffies, ch->gr_nocap_since + msecs_to_jiffies( @@ -18296,6 +22459,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->h_object, (u32)idx, va, len); } + } /* * A CE push that should carry this * channel's next tracking release but @@ -18872,30 +23036,6 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) !ch->defer_since) ch->defer_since = jiffies; - /* - * STALLFREE r230: a fresh - * binding's unmapped entry is - * the host-readiness race - - * request the immediate rebuild - * so the pages resolve inside - * this grace instead of the - * 10 s cadence (r229/r231: - * attempt 1 lost exactly - * here, attempt 2 won after - * the rebuild caught up). - */ - if (!ch->engine_type && - ch->wd_newbind_until && - time_before(jiffies, - ch->wd_newbind_until)) { - svc.rebuild_now = true; - if (svc.gr_pending_map < 16) { - svc.gr_pending_map++; - pr_warn("vgsp-gr: GR-STALL-pending-map h=0x%x va=0x%llx entry=%u action=rebuild reason=stale-entry-newbind\n", - ch->h_object, va, - (u32)idx); - } - } svc.gpfifo_nomap++; kfree(pb); break; @@ -19029,6 +23169,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].userd_pa = ch->userd_pa; svc.chan[k].userd_as = ch->userd_as; svc.chan[k].userd_ok = ch->userd_ok; + /* r496b: persist the RAM-scan cursor and the + * one-shot USERD-scan latch so they survive + * the per-pass snapshot copy. */ + svc.chan[k].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[k].gr_ram_passes = ch->gr_ram_passes; + svc.chan[k].gr_ram_logged = ch->gr_ram_logged; + svc.chan[k].gr_ram_ring = ch->gr_ram_ring; + svc.chan[k].userd_scans = ch->userd_scans; svc.chan[k].logged = ch->logged; svc.chan[k].logged_expected = ch->logged_expected; svc.chan[k].notified = ch->notified; @@ -19055,6 +23203,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].push_hold_until = ch->push_hold_until; svc.chan[k].defer_since = ch->defer_since; svc.chan[k].put_prev = ch->put_prev; + /* r471: the publish clamp watermark is set + * and cleared on the pump thread; persist it + * so the next pass still clamps. */ + svc.chan[k].gr_bad_put = ch->gr_bad_put; /* * Persist the cheap-wait window and the content * watermark: the pump works on a snapshot, so @@ -19102,6 +23254,21 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gr_hold_deadline; svc.chan[k].gr_hold_exp_logged = ch->gr_hold_exp_logged; + /* r509: the one-shot expiry latch is snapshot + * state too. */ + svc.chan[k].gr_hold_bound_hit = + ch->gr_hold_bound_hit; + /* r361: the opdef absolute-bound record is + * snapshot state too (the GRFIX lesson) - + * without the write-back the deadline would + * be zeroed every pass and the bound could + * never fire. */ + svc.chan[k].opdef_hold_dead_va = + ch->opdef_hold_dead_va; + svc.chan[k].opdef_hold_deadline = + ch->opdef_hold_deadline; + svc.chan[k].opdef_hold_exp_logged = + ch->opdef_hold_exp_logged; /* m164: the CLASS-2 no-capture window is snapshot * state too - without the write-back every pass * would re-arm it and rehold the entry forever @@ -19121,10 +23288,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].wd_put = ch->wd_put; svc.chan[k].wd_sub = ch->wd_sub; svc.chan[k].wd_since = ch->wd_since; - svc.chan[k].wd_epoch_done = ch->wd_epoch_done; - svc.chan[k].wd_nocap_put = ch->wd_nocap_put; - svc.chan[k].wd_chanswitch = ch->wd_chanswitch; - svc.chan[k].wd_newbind_until = ch->wd_newbind_until; + /* + * r312: gr_pub_armed / counter_gen live only on + * the snapshot copy if not persisted. Losing the + * armed bit re-arms the residual watchdog every + * pass; its recovery then keeps zeroing the + * completion word and re-publishing - the r311/ + * r312 vector_add wedge loop (66 stale-wedge + * rows with the client polling forever). + */ + svc.chan[k].gr_pub_armed = ch->gr_pub_armed; + svc.chan[k].counter_gen = ch->counter_gen; + svc.chan[k].wd_rewind_off_until = + ch->wd_rewind_off_until; svc.chan[k].pushpg_va = ch->pushpg_va; svc.chan[k].pushpg_n = ch->pushpg_n; svc.chan[k].content_max = ch->content_max; @@ -19150,6 +23326,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) memcpy(svc.chan[k].slot_ops_ident, ch->slot_ops_ident, sizeof(ch->slot_ops_ident)); + memcpy(svc.chan[k].slot_pend_tries, + ch->slot_pend_tries, + sizeof(ch->slot_pend_tries)); memcpy(svc.chan[k].slot_ops_done, ch->slot_ops_done, sizeof(ch->slot_ops_done)); @@ -19552,6 +23731,39 @@ static void svc_handle_map_add(void) } svc_ipin_store(va, req.pa + off, ap); svc_lat_store(va, req.pa + off, ap); + /* + * r372 (stale-ledger corruption class): a fresh pin is the + * guest's newest statement of where this VA's content + * lives. The landing table (ledger) otherwise only moves + * on host writes, so a VA re-pinned to a NEW page (the + * guest freed the old buffer and reallocated it - CUDA + * reuses VA ranges across processes) kept resolving to + * the OLD, freed page: every later write through that + * ledger entry landed on memory the guest kernel had + * since reused (proven class: 8 zero bytes into an + * nv_free_system_pages page array -> guest oops at + * client teardown). Repair the entry only when it + * disagrees with the pin (a no-op otherwise, so the + * table is not flooded with pin pages). + */ + { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(va, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + ((req.pa + off) & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)ap)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-REPAIR va=0x%llx old=0x%llx ap=%u -> new=0x%llx ap=%u\n", + va, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, + req.pa + off, ap); + svc_ledger_store(va, req.pa + off, ap); + } + } } /* * m149: bounded per-session evidence that the CE push-buffer @@ -19594,6 +23806,20 @@ static void svc_handle_map_add(void) if (!ch->valid || !ch->gpfifo_entries) continue; /* + * STALLFREE r225: a channel under the stale-replay + * watchdog's rewind cooldown is NOT rewound. Without + * this the rewind re-opened the same consumed window + * every batch while the producer had nothing pending, + * pinning last_get and re-walking stale entries + * forever (r225: slots 93..100 of 0x5c000059 for + * 42+ s). The cooldown expires on its own (bounded + * duty cycle) and is cleared by new work, so a + * translation landing later can still retry. + */ + if (ch->wd_rewind_off_until && + time_before(jiffies, ch->wd_rewind_off_until)) + continue; + /* * Clamp the rewind at 0: wrapping below zero (get=0 -> * entries - N) makes the pump read ring slots past the * producer that hold stale entries, so replay only @@ -19997,27 +24223,7 @@ static int svc_thread(void *arg) * svc_rebuild() publishes (svc.last_build). */ sess = svc_guest_get(vgpu, true); - /* - * STALLFREE r230: a pending rebuild request from the - * pump (a fresh binding's entry hit an unmapped page) - * forces the rebuild now instead of waiting for the - * 10 s cadence, so the first attempt's pages resolve. - */ - if (svc.rebuild_now) { - svc.rebuild_now = false; - /* - * Bounded proof (r234 asks): a pump-requested - * rebuild was actually consumed by the service - * thread here (the only consumer). - */ - if (svc.gr_rebuild_log < 16) { - svc.gr_rebuild_log++; - pr_info("vgsp-gr: GR-STALL-rebuild consumed=1 by=svc_thread action=rebuild reason=stale-entry-newbind\n"); - } - svc_rebuild(vgpu); - if (sess) - sess->last_build = svc.last_build; - } else if (svc_sess_flush || !svc.direct_base || + if (svc_sess_flush || !svc.direct_base || !sess || time_after(jiffies, (unsigned long)(sess->last_build + @@ -20545,6 +24751,6 @@ int gsp_sema_service_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, * after the write still receives the completion event. */ if (!rc) - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return rc; } diff --git a/kernel/gsp_sema_service.h b/kernel/gsp_sema_service.h index 55d323c..ee5c48a 100644 --- a/kernel/gsp_sema_service.h +++ b/kernel/gsp_sema_service.h @@ -81,6 +81,14 @@ struct sema_vaspace_desc { void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc); void gsp_sema_service_vaspace_del(u32 h_vaspace); +/* + * r496j: park a page-directory candidate (from COPY_SERVER_PDES) until a + * content-validated walk proves it belongs to a live client; only then is + * it registered (a blind registration let stale kernel directories answer + * other clients' VAs in the walker's first-success order). + */ +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem); /* m177: guest RM reload = new guest boot; wipe boot-scoped discovery state * (direct-map base, content-scanned tables) on the svc thread. Client * session flushes keep that state - see svc_session_flush. */ @@ -284,6 +292,13 @@ extern unsigned gsp_phys_gr_hold_max_ms; * hold and its absolute bound apply. */ extern unsigned gsp_phys_gr_fill_hold; +extern unsigned gsp_sema_walk_trace; + +/* + * r365: execute fill-shaped CE transfers to VIDMEM-window VAs (the guest's + * page-table writes). Off by default; see the definition in the .c. + */ +extern bool gsp_sema_winfill; /* * m164 CLASS-2: hold window (ms) for a GR launch push whose decode saw QMD @@ -302,6 +317,23 @@ extern bool gsp_sema_pool_hold_fresh; * walk/RAM scan until a pin confirms (default on). */ extern bool gsp_sema_copy_dst_pin_guard; +extern bool gsp_sema_wait_gate; +/* r401: bounded acquire-wait hold (fail-open) in ms; must exceed the + * emulated launch completion latency for copy-after-launch ordering. */ +extern unsigned gsp_sema_wait_hold_ms; +/* r481: verbatim method-stream dump for the first N large GR pushes + * (default 0 off). Diagnostic only; no behaviour change. */ +extern unsigned gsp_sema_push_raw; + +/* + * r384 (guest boot-corruption lottery): fill-destination guards (default + * on). Large virtual fills whose only authority is the walk/scan resolver + * are held through a short bounded window while their VA is demanded + * through the pin channel (fail-open after gsp_sema_fill_guard_ms); zero + * fills targeting a page-table-shaped guest page are refused loudly. + */ +extern bool gsp_sema_fill_guard; +extern unsigned gsp_sema_fill_guard_ms; /* * r231 READ-ORDER guard (default 0: witness only). Defer a device-src copy @@ -363,4 +395,42 @@ struct gsp_sema_gr_reloc { int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out); +/* + * r378 write guards. Both re-validate a guest-memory write target that was + * resolved earlier (ledger/pin/walk/counter registration, or the GR submit's + * per-page resolution) against the CURRENT resolution before the bytes land, + * and return false when a positive disagreement proves the target stale: + * + * gsp_sema_gr_write_guard(va, hpa, path): GR backend pages. A fresh + * injected pin is checked first (the r372/r376 pin-wins rule - the guest's + * newest statement), then the trusted resolver chain for a pin-less VA. + * The submit-to-copy-out window is seconds wide (the GR poll), and a guest + * buffer freed/re-homed in it would otherwise receive a full page of + * another buffer's content (the boot-corruption class). The landing table + * is deliberately not the authority: our own launches feed it, so it would + * confirm the target by construction. + * + * gsp_sema_write_guard(va, pa, ap, path): service-side SYS/FB targets. A + * fresh injected pin is the guest's newest statement about the VA; a pin + * that disagrees with the recorded target means the page moved (the + * r372/r376 pin-wins rule), so the stale write is skipped. + * + * Both fail OPEN when no current answer exists (the recorded landing is then + * the write authority) and log every skip (bounded, 256 lines) with the path + * and the VA/target/current values. + */ +bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path); +bool gsp_sema_write_guard(u64 va, u64 pa, u32 ap, const char *path); + +/* + * r391 diagnostic (sum=0 class): resolve a GR data VA through the landing + * table (the exact source the D2H reader's ledger-first branch uses) and + * return the resolution plus the first 8 bytes read through it. The GR + * copy-out prints it next to the write target it used, so a launch/readback + * disagreement (the write went to another page/aperture than the reader + * resolves) is one line instead of a forensics session. + */ +bool gsp_sema_gr_va_snap(u64 va, u64 *pa, u32 *ap, u64 *val8); +u8 *gsp_sema_gr_shadow_base(void); + #endif /* _GSP_SEMA_SERVICE_H_ */ diff --git a/kernel/main.c b/kernel/main.c index c150a1a..3ee5c23 100644 --- a/kernel/main.c +++ b/kernel/main.c @@ -2788,6 +2788,10 @@ MODULE_PARM_DESC(gsp_phys_gr_hold_max_ms, "Absolute GR fill hold bound in ms, ke */ module_param_named(gsp_phys_gr_fill_hold, gsp_phys_gr_fill_hold, uint, 0644); MODULE_PARM_DESC(gsp_phys_gr_fill_hold, "GR fill hold on unresolved operand pages (default 0: log-only, never hold)"); +module_param_named(gsp_sema_walk_trace, gsp_sema_walk_trace, uint, 0644); +MODULE_PARM_DESC(gsp_sema_walk_trace, "Trace GMMU walks for unresolved high-VA operands (default 0: off)"); +module_param_named(gsp_sema_winfill, gsp_sema_winfill, bool, 0644); +MODULE_PARM_DESC(gsp_sema_winfill, "Execute fill-shaped CE transfers to VIDMEM-window VAs (default 0: off)"); /* * F1-b: while a forwarded GR launch is in flight, defer (do not apply) a @@ -2824,9 +2828,27 @@ MODULE_PARM_DESC(sema_pool_hold_fresh, "apply pool releases at fresh-pin time in * the m163 retry path re-resolve with pins. */ module_param_named(sema_copy_dst_pin_guard, gsp_sema_copy_dst_pin_guard, bool, 0644); +module_param_named(sema_wait_gate, gsp_sema_wait_gate, bool, 0644); +MODULE_PARM_DESC(sema_wait_gate, "honor C56F host-semaphore acquires before running a ring entry's work (default 1 since r408; 0 restores pass-through)"); +module_param_named(sema_wait_hold_ms, gsp_sema_wait_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_wait_hold_ms, "bounded acquire-wait hold before fail-open, ms (r401, default 4000; must exceed the emulated launch latency)"); +module_param_named(sema_push_raw, gsp_sema_push_raw, uint, 0644); +MODULE_PARM_DESC(sema_push_raw, "r481 diagnostic: dump the decoded method stream of the first N large GR pushes (default 0 off)"); MODULE_PARM_DESC(sema_copy_dst_pin_guard, "hold device-dst copies without a pin-resolved translation (default 1)"); /* + * r384 (guest boot-corruption lottery): fill-destination guards. A large + * virtual fill whose only authority is the walk/scan resolver is held for + * gsp_sema_fill_guard_ms while its VA is demanded through the pin channel + * (fail-open, mirrors sema_copy_dst_pin_guard); a zero fill targeting a + * page-table-shaped guest page is refused loudly (FILL-PT-REFUSED). + */ +module_param_named(sema_fill_guard, gsp_sema_fill_guard, bool, 0644); +MODULE_PARM_DESC(sema_fill_guard, "guard virtual fill destinations (pin-demand hold + page-table-shape refusal, default 1)"); +module_param_named(sema_fill_guard_ms, gsp_sema_fill_guard_ms, uint, 0644); +MODULE_PARM_DESC(sema_fill_guard_ms, "fill pin-demand hold window in ms (default 800)"); + +/* * r231 READ-ORDER guard: defer a device-src copy whose source VA has a * pending device-dst write hold so the read re-decodes after the write * resolves (fixes the ctaid_probe RAWA upload/readback inversion). diff --git a/tools/cuda-ab/frame_producer.cu b/tools/cuda-ab/frame_producer.cu new file mode 100644 index 0000000..609114c --- /dev/null +++ b/tools/cuda-ab/frame_producer.cu @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * frame_producer.cu - render an animated frame on the guest GPU and publish + * it into the ivshmem (kvmfr) BAR so the host can see a real GPU-rendered + * frame from the guest. + * + * Path: CUDA kernel -> device buffer -> host buffer -> ivshmem BAR + * (/sys/bus/pci/devices/0000:00:03.0/resource2, the 128 MiB kvmfr + * device QEMU backs with /dev/shm/kvmfr-<uuid>). + * + * Layout in the BAR (offset 0x100000, clear of the kvmfr protocol header): + * struct frame_hdr { magic, version, width, height, stride, format, seq } + * then width*height*4 bytes of RGBA8. + * + * Usage: sudo ./frame_producer [frames] [width] [height] [fps] + * Prints: FRAME seq=<n> ms=<render+copy ms> (one line per frame, first 5 and + * every 30th) and FRAME_DONE frames=<n>. + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <ctime> +#include <fcntl.h> +#include <sys/mman.h> +#include <unistd.h> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + return 1; \ + } \ + } while (0) + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" little-endian-ish */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +__global__ void render_julia(uchar4 *fb, int w, int h, float t) +{ + int x = blockIdx.x * blockDim.x + threadIdx.x; + int y = blockIdx.y * blockDim.y + threadIdx.y; + + if (x >= w || y >= h) + return; + + float fx = (x - w * 0.5f) / (h * 0.45f); + float fy = (y - h * 0.5f) / (h * 0.45f); + float cr = 0.7885f * cosf(t); + float ci = 0.7885f * sinf(t); + float zr = fx, zi = fy; + int it = 0; + const int maxit = 96; + + while (zr * zr + zi * zi < 4.0f && it < maxit) { + float nzr = zr * zr - zi * zi + cr; + float nzi = 2.0f * zr * zi + ci; + + zr = nzr; + zi = nzi; + it++; + } + { + float v = (float)it / (float)maxit; + unsigned char r = (unsigned char)(255.0f * v); + unsigned char g = (unsigned char)(255.0f * (1.0f - v) * v * 4.0f); + unsigned char b = (unsigned char)(255.0f * (1.0f - v)); + unsigned char a = 255; + + fb[y * w + x] = make_uchar4(r, g, b, a); + } +} + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +int main(int argc, char **argv) +{ + int frames = argc > 1 ? atoi(argv[1]) : 60; + int w = argc > 2 ? atoi(argv[2]) : 800; + int h = argc > 3 ? atoi(argv[3]) : 600; + int fps = argc > 4 ? atoi(argv[4]) : 10; + const char *bar_path = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + size_t fb_bytes = (size_t)w * h * 4; + uchar4 *d_fb = NULL, *h_fb = NULL; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + int fd, i; + + setvbuf(stdout, NULL, _IONBF, 0); + + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + printf("OPEN_FAIL %s\n", bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (bar == MAP_FAILED) { + printf("MMAP_FAIL\n"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + CHECK(cudaMalloc(&d_fb, fb_bytes)); + h_fb = (uchar4 *)malloc(fb_bytes); + if (!h_fb) { + printf("HOST_ALLOC_FAIL\n"); + return 1; + } + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + + for (i = 0; i < frames; i++) { + uint64_t t0 = now_ns(); + float phase = (float)i * 0.15f; + dim3 block(16, 16); + dim3 grid((w + 15) / 16, (h + 15) / 16); + + render_julia<<<grid, block>>>(d_fb, w, h, phase); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_fb, d_fb, fb_bytes, + cudaMemcpyDeviceToHost)); + memcpy(bar + FRAME_OFFSET + sizeof(struct frame_hdr), h_fb, + fb_bytes); + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + if (i < 5 || (i % 30) == 29) + printf("FRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d %dx%d\n", frames, w, h); + CHECK(cudaFree(d_fb)); + free(h_fb); + munmap(bar, 128UL << 20); + close(fd); + return 0; +} diff --git a/tools/cuda-ab/inplace_scale.cu b/tools/cuda-ab/inplace_scale.cu new file mode 100644 index 0000000..e83a039 --- /dev/null +++ b/tools/cuda-ab/inplace_scale.cu @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * inplace_scale.cu - minimal in-place scale repro for the streams2 + * "scale wrote 1 page" class. + * + * One H2D, one in-place scale launch (c[i] *= 2 over 128 KB = 32 pages), + * one D2H, per-page diff. Run it repeatedly: if the launch writes only + * part of the buffer the per-page report names how many 4 KB blocks + * changed and the host log carries the launch's retire wait. + * + * Build: nvcc -O2 --cudart=static -o inplace_scale inplace_scale.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +int main(void) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *h, *d; + int i, bad = 0, first = -1, last = -1; + int nb = (n + 1023) / 1024, blk_bad = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + h = (float *)malloc(bytes); + if (!h) + return 1; + for (i = 0; i < n; i++) + h[i] = (float)(i % 1000) * 0.5f; + + CHECK(cudaMalloc(&d, bytes)); + CHECK(cudaMemcpy(d, h, bytes, cudaMemcpyHostToDevice)); + printf("PTR d=%p\n", (void *)d); + + scale<<<(n + 255) / 256, 256>>>(d, 2.0f, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + + CHECK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost)); + for (i = 0; i < n; i++) { + float exp = (float)(i % 1000) * 0.5f * 2.0f; + + if (h[i] != exp) { + bad++; + if (first < 0) + first = i; + last = i; + } + } + for (i = 0; i < nb; i++) { + int j, hit = 0; + + for (j = i * 1024; j < (i + 1) * 1024 && j < n; j++) + if (h[j] != (float)(j % 1000) * 0.5f * 2.0f) { + hit = 1; + break; + } + if (hit) + blk_bad++; + } + printf("SCALE n=%d bad=%d first=%d last=%d blocks=%d/%d\n", + n, bad, first, last, blk_bad, nb); + printf("STATUS %s\n", bad ? "FAIL" : "OK"); + return bad ? 1 : 0; +} diff --git a/tools/cuda-ab/ip_probe.cu b/tools/cuda-ab/ip_probe.cu new file mode 100644 index 0000000..366fbc8 --- /dev/null +++ b/tools/cuda-ab/ip_probe.cu @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe.cu - how many CTAs actually executed for an in-place kernel? + * + * scale_mark marks progress[blockIdx.x]=0xCAFEBABE after the c store; D2H of + * progress counts executed CTAs exactly. c (the in-place buffer) is read back + * and verified. seed (argv[1]) makes the run content-unique. + * + * Build: nvcc -O2 --cudart=static -o ip_probe ip_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_mark(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g grid=%d\n", prop.name, (double)k, grid); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + /* in-place scale with CTA markers */ + scale_mark<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0, first_missing = -1; + + for (i = 0; i < grid; i++) { + if (h_prog[i] == 0xCAFEBABEu) + marked++; + else if (first_missing < 0) + first_missing = i; + } + printf("CTA-MARK marked=%d/%d first_missing=%d\n", marked, grid, + first_missing); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + if (first >= 0) + printf("SCALE firstbad got=%g want=%g\n", hc[first], + (ha[first] + hb[first]) * 2.0f); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe2.cu b/tools/cuda-ab/ip_probe2.cu new file mode 100644 index 0000000..a5fa2ec --- /dev/null +++ b/tools/cuda-ab/ip_probe2.cu @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe2.cu - why do in-place c stores vanish while prog stores land? + * + * After a vadd that fills c, three variants run on the SAME c: + * 1. scale_if : if (i<n) c[i] *= s; (control, like ip_probe) + * 2. scale_u : c[i] *= s; (unconditional, grid exact) + * 3. fill_c : c[i] = 7.0f; (store only, no load) + * 4. add_c : c[i] = c[i] + 1.0f; (load + store) + * Each stage reads c back and compares against the expected value at that + * stage. A separate prog buffer marks CTA execution. + * + * Build: nvcc -O2 --cudart=static -o ip_probe2 ip_probe2.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_if(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +__global__ void scale_u(float *c, float s, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xDEADBEEFu; +} + +__global__ void fill_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = 7.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xFEEDFACu; +} + +__global__ void add_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] + 1.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0x0DDF00Du; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i, stage; + float mult; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + mult = 2.0f; + for (stage = 0; stage < 4; stage++) { + unsigned mark = 0; + + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + switch (stage) { + case 0: + scale_if<<<grid, 256>>>(dc, 2.0f, n, d_prog); + mult = 2.0f; + mark = 0xCAFEBABEu; + break; + case 1: + scale_u<<<grid, 256>>>(dc, 2.0f, d_prog); + mult = 4.0f; + mark = 0xDEADBEEFu; + break; + case 2: + fill_c<<<grid, 256>>>(dc, d_prog); + mark = 0xFEEDFACu; + break; + default: + add_c<<<grid, 256>>>(dc, d_prog); + mult = -1.0f; + mark = 0x0DDF00Du; + break; + } + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0; + + for (i = 0; i < grid; i++) + if (h_prog[i] == mark) + marked++; + printf("S%d CTA-MARK marked=%d/%d\n", stage + 1, + marked, grid); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + if (stage < 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) { + b++; + if (first < 0) + first = i; + } + printf("S%d readback bad=%d/%d first=%d\n", + stage + 1, b, n, first); + } else if (stage == 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 7.0f) { + b++; + if (first < 0) + first = i; + } + printf("S3 readback bad=%d/%d first=%d (fill 7.0)\n", + b, n, first); + } else { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 8.0f) { + b++; + if (first < 0) + first = i; + } + printf("S4 readback bad=%d/%d first=%d (add 1.0 to fill)\n", + b, n, first); + } + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe3.cu b/tools/cuda-ab/ip_probe3.cu new file mode 100644 index 0000000..a50df0e --- /dev/null +++ b/tools/cuda-ab/ip_probe3.cu @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe3.cu - what does the SM actually READ for the scale kernel? + * + * scale_report writes, per CTA: prog[2b] = bits of the float param s, + * prog[2b+1] = bits of c[b*256] (the load). + * The D2H of prog then shows exactly what the SM saw, independent of any + * store-visibility path for c. + * + * Build: nvcc -O2 --cudart=static -o ip_probe3 ip_probe3.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_report(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) { + prog[2 * blockIdx.x] = __float_as_uint(s); + prog[2 * blockIdx.x + 1] = __float_as_uint(c[blockIdx.x * 256]); + } +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[256]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 256 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 256 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + scale_report<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 256 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int s_ok = 0, s_1 = 0, c_ok = 0, n_seen = 0; + unsigned first_s = 0, first_c = 0; + + for (i = 0; i < grid; i++) { + unsigned sb = h_prog[2 * i]; + unsigned cb = h_prog[2 * i + 1]; + + if (i == 0) { + first_s = sb; + first_c = cb; + } + if (sb == 0x40000000u) + s_ok++; + if (sb == 0x3F800000u) + s_1++; + { + float want = ha[i * 256] + hb[i * 256]; + + if (cb == *(unsigned *)&want) + c_ok++; + } + n_seen++; + } + printf("REPORT ctas=%d s_eq_2.0=%d s_eq_1.0=%d c_load_ok=%d first_s=0x%08x first_c=0x%08x\n", + n_seen, s_ok, s_1, c_ok, first_s, first_c); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/launch_count.cu b/tools/cuda-ab/launch_count.cu new file mode 100644 index 0000000..02db1d6 --- /dev/null +++ b/tools/cuda-ab/launch_count.cu @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: BSD-3-Clause +// launch_count.cu - single-launch execution-count repro (r342). +// +// One launch, one atomic increment. The engine-written page witness showed the +// increment applied TWICE (1 -> 2; the 100-increment variant -> 200): every +// forwarded launch executes its engine work twice (one submission, one ring +// publish, completion OK). This is the root cause of the multi-pass content +// class (victory_reduce exactly 2x, scale/saxpy/smiley/multikernel wrong). +// Run in the guest; expected LAUNCH_COUNT=1. +// for a single launch, the launch path executes work more than once. +#include <cstdio> +#include <cuda_runtime.h> +__global__ void inc(unsigned long long *c) { if (threadIdx.x == 0 && blockIdx.x == 0) atomicAdd(c, 1ULL); } +int main(void) { + unsigned long long *d; unsigned long long h = 0; + if (cudaMalloc(&d, 8) != cudaSuccess) { printf("CUDA_ERROR malloc\nSTATUS FAIL\n"); return 1; } + if (cudaMemset(d, 0, 8) != cudaSuccess) { printf("CUDA_ERROR memset\nSTATUS FAIL\n"); return 1; } + inc<<<1, 32>>>(d); + if (cudaGetLastError() != cudaSuccess) { printf("CUDA_ERROR launch\nSTATUS FAIL\n"); return 1; } + if (cudaDeviceSynchronize() != cudaSuccess) { printf("CUDA_ERROR sync\nSTATUS FAIL\n"); return 1; } + if (cudaMemcpy(&h, d, 8, cudaMemcpyDeviceToHost) != cudaSuccess) { printf("CUDA_ERROR d2h\nSTATUS FAIL\n"); return 1; } + printf("LAUNCH_COUNT=%llu (expect 1)\n", h); + printf("STATUS %s\n", h == 1 ? "OK" : "FAIL"); + return h == 1 ? 0 : 1; +} diff --git a/tools/cuda-ab/memset2d_probe.cu b/tools/cuda-ab/memset2d_probe.cu new file mode 100644 index 0000000..acf43d4 --- /dev/null +++ b/tools/cuda-ab/memset2d_probe.cu @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset2d_probe.cu - per-phase diagnostics for victory_memset2d. + * + * victory_memset2d reports one aggregate bad count (~12.5M) that mixes all + * three phases. This probe runs the SAME three phases and reports each + * phase's bad count, the first mismatching byte (offset, got, want), and a + * per-phase 1MB-chunk histogram so the failing phase and the failing region + * are attributable in one run. + * + * Phases: + * P1 memset : cudaMemset(d_img, 0xA5, pitch*h); D2H 2D readback + * P2 roundtrip: H2D 2D pattern; D2H 2D readback + * P3 memset2d : cudaMemset2D quadrant 0x3C; D2H 2D readback + * + * Output lines: + * DEVICE ... + * PHASE p=1 bad=<n> first_off=<i> got=<g> want=<w> + * PHIST p=1 mb=<m> ok=<n> bad=<n> + * MEMSET2D2 bad_total=<n> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define IMG_W 4096 +#define IMG_H 1024 +#define FILL_BYTE 0xA5 +#define QUAD_BYTE 0x3C + +static int phase_check(const char *tag, const unsigned char *back, + const unsigned char *want, size_t n, int p) +{ + size_t i, first = (size_t)-1; + int bad = 0, got = 0, wantv = 0; + size_t mb; + + for (i = 0; i < n; i++) { + if (back[i] != want[i]) { + if (first == (size_t)-1) { + first = i; + got = back[i]; + wantv = want[i]; + } + bad++; + } + } + printf("PHASE p=%d bad=%d first_off=%zu got=0x%02x want=0x%02x\n", + p, bad, first, got, wantv); + if (bad) { + size_t nb = (n + (1u << 20) - 1) >> 20; + + for (mb = 0; mb < nb && mb < 8; mb++) { + size_t lo = mb << 20, hi = lo + (1u << 20); + size_t j; + int bok = 0, bbad = 0; + + if (hi > n) + hi = n; + for (j = lo; j < hi; j++) { + if (back[j] == want[j]) + bok++; + else + bbad++; + } + printf("PHIST p=%d mb=%zu ok=%d bad=%d\n", + p, mb, bok, bbad); + } + } + (void)tag; + return bad; +} + +int main(void) +{ + cudaDeviceProp prop; + const size_t w = IMG_W, h = IMG_H; + unsigned char *h_img, *h_back, *h_want; + unsigned char *d_img; + size_t pitch = 0; + int bad = 0, row, col; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_img = (unsigned char *)malloc(w * h); + h_back = (unsigned char *)malloc(w * h); + h_want = (unsigned char *)malloc(w * h); + if (!h_img || !h_back || !h_want) + return 1; + + CHECK(cudaMallocPitch(&d_img, &pitch, w, h)); + printf("PITCH pitch=%zu\n", pitch); + printf("DPTRS h_img=%p h_back=%p h_want=%p d_img=%p\n", (void *)h_img, + (void *)h_back, (void *)h_want, (void *)d_img); + + /* P1: full cudaMemset over the pitched allocation, 2D readback. */ + CHECK(cudaMemset(d_img, FILL_BYTE, pitch * h)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + memset(h_want, FILL_BYTE, w * h); + bad += phase_check("P1", h_back, h_want, w * h, 1); + + /* P2: pitched H2D pattern -> D2H round trip. */ + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) + h_img[row * w + col] = (unsigned char)((row + col) & 0xFF); + CHECK(cudaMemcpy2D(d_img, pitch, h_img, w, w, h, + cudaMemcpyHostToDevice)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P2", h_back, h_img, w * h, 2); + + /* P3: cudaMemset2D of the upper-left quadrant only. */ + CHECK(cudaMemset2D(d_img, pitch, QUAD_BYTE, w / 2, h / 2)); + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) { + unsigned char expect = h_img[row * w + col]; + + if (row < (int)(h / 2) && col < (int)(w / 2)) + expect = QUAD_BYTE; + h_want[row * w + col] = expect; + } + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P3", h_back, h_want, w * h, 3); + + printf("MEMSET2D2 bad_total=%d\n", bad); + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/memset_probe.cu b/tools/cuda-ab/memset_probe.cu new file mode 100644 index 0000000..a6a12e0 --- /dev/null +++ b/tools/cuda-ab/memset_probe.cu @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset_probe.cu - isolate the memcpy_roundtrip memset-phase failure. + * + * cudaMemset(8 MB, 0x5A) -> D2H -> byte verify, with a run-length report: + * the first bad offset, and a coverage map at 4 KB granularity (correct / + * wrong / untouched-malloc-prefill) so "fill under-delivered" is + * distinguishable from "readback delivered wrong pages". + * + * Build: nvcc -O2 --cudart=static -o memset_probe memset_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define PAGE 4096 +#define MemsetByte 0x5A + +int main(void) +{ + const size_t bytes = 8u << 20; + unsigned char *h; + unsigned char *d; + size_t i; + int bad = 0; + size_t first_bad = (size_t)-1; + size_t n_pages, p; + + setvbuf(stdout, NULL, _IONBF, 0); + h = (unsigned char *)malloc(bytes); + if (!h) + return 1; + memset(h, 0xEE, bytes); + + CHECK(cudaMalloc(&d, bytes)); + printf("DPTR d=%p bytes=%zu\n", (void *)d, bytes); + + CHECK(cudaMemset(d, MemsetByte, bytes)); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < bytes; i++) { + if (h[i] != (unsigned char)MemsetByte) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("FIRST_BAD off=%zu (0x%zx) bad=%d total=%zu\n", + first_bad, first_bad, bad, bytes); + n_pages = bytes / PAGE; + for (p = 0; p < n_pages; p++) { + int good = 1, prefill = 1; + + for (i = 0; i < PAGE; i++) { + if (h[p * PAGE + i] != (unsigned char)MemsetByte) + good = 0; + if (h[p * PAGE + i] != 0xEE) + prefill = 0; + if (!good && !prefill) + break; + } + if (!good) { + printf("PAGE %zu %s b0=%02x b1=%02x b2=%02x b3=%02x b4095=%02x\n", + p, prefill ? "PREFILL" : "OTHER", + h[p * PAGE], h[p * PAGE + 1], + h[p * PAGE + 2], h[p * PAGE + 3], + h[p * PAGE + 4095]); + if (p > 24) + break; + } + } + printf("STATUS %s\n", bad == 0 ? "OK" : "FAIL"); + return 0; +} diff --git a/tools/cuda-ab/rt_probe.cu b/tools/cuda-ab/rt_probe.cu new file mode 100644 index 0000000..025301d --- /dev/null +++ b/tools/cuda-ab/rt_probe.cu @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * rt_probe.cu - instrumented memcpy_roundtrip variant isolating the + * memset-phase D2H readback failure. + * + * Phases (argv[1]): + * "full" : H2D -> kernel -> D2H -> memset -> D2H (the original order) + * "sync" : same but cudaDeviceSynchronize() before the second D2H + * "memset" : memset -> D2H only (memset_probe shape, 3-buffer alloc) + * "rev" : memset -> D2H first, then H2D -> kernel -> D2H + * + * Forensics on the memset readback: first bad offset, byte histogram + * (0x5A / 0x00 / other), per-1MB chunk histogram, and the first 32 bytes. + * + * Build: nvcc -O2 --cudart=static -o rt_probe rt_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define MemsetByte 0x5A + +__global__ void bump(const float *in, float *out, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + out[i] = in[i] + 0.5f; +} + +static void memset_forensics(const unsigned char *h, size_t bytes, + const char *tag) +{ + size_t i, first_bad = (size_t)-1; + int bad = 0, n5a = 0, n00 = 0, nother = 0; + size_t chunk; + + for (i = 0; i < bytes; i++) { + if (h[i] == 0x5A) + n5a++; + else if (h[i] == 0x00) + n00++; + else + nother++; + if (h[i] != 0x5A) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("MEMSET-READ %s bad=%d first_bad=%zu n5a=%d n00=%d nother=%d\n", + tag, bad, first_bad, n5a, n00, nother); + printf("MEMSET-HEAD %s", tag); + for (i = 0; i < 32; i++) + printf(" %02x", h[i]); + printf("\n"); + if (first_bad != (size_t)-1) { + printf("MEMSET-ATBAD %s off=%zu:", tag, first_bad); + for (i = first_bad; i < first_bad + 16 && i < bytes; i++) + printf(" %02x", h[i]); + printf("\n"); + } + for (chunk = 0; chunk < bytes / (1u << 20); chunk++) { + size_t lo = chunk << 20, hi = lo + (1u << 20); + int c5a = 0, c00 = 0, cother = 0; + + for (i = lo; i < hi; i++) { + if (h[i] == 0x5A) + c5a++; + else if (h[i] == 0x00) + c00++; + else + cother++; + } + printf("MEMSET-MB %s mb=%zu 5a=%d 00=%d other=%d\n", + tag, chunk, c5a, c00, cother); + } +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "full"; + cudaDeviceProp prop; + const size_t bytes = 8u << 20; + const int n = (int)(bytes / sizeof(float)); + unsigned char *h_m; + float *h_in, *h_out; + float *d_in, *d_out; + unsigned char *d_m; + double csum = 0.0; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu mode=%s\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem, mode); + + h_in = (float *)malloc(bytes); + h_out = (float *)malloc(bytes); + h_m = (unsigned char *)malloc(bytes); + if (!h_in || !h_out || !h_m) + return 1; + + for (i = 0; i < n; i++) + h_in[i] = (float)(i % 1024) * 0.25f; + + CHECK(cudaMalloc(&d_in, bytes)); + CHECK(cudaMalloc(&d_out, bytes)); + CHECK(cudaMalloc(&d_m, bytes)); + printf("DPTRS d_in=%p d_out=%p d_m=%p\n", (void *)d_in, + (void *)d_out, (void *)d_m); + + if (!strcmp(mode, "memset") || !strcmp(mode, "rev")) { + /* memset phase first */ + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "early"); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync") || + !strcmp(mode, "rev")) { + CHECK(cudaMemcpy(d_in, h_in, bytes, cudaMemcpyHostToDevice)); + bump<<<grid, 256>>>(d_in, d_out, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < n; i++) { + if (h_out[i] != h_in[i] + 0.5f) + bad++; + csum += (double)h_out[i]; + } + printf("BUMP bad=%d csum=%.6f\n", bad, csum); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync")) { + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "late"); + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/run/egl_frame b/tools/cuda-ab/run/egl_frame Binary files differnew file mode 100755 index 0000000..130f032 --- /dev/null +++ b/tools/cuda-ab/run/egl_frame diff --git a/tools/cuda-ab/run/frame_producer b/tools/cuda-ab/run/frame_producer Binary files differnew file mode 100755 index 0000000..a202ac7 --- /dev/null +++ b/tools/cuda-ab/run/frame_producer diff --git a/tools/cuda-ab/run/inplace_scale b/tools/cuda-ab/run/inplace_scale Binary files differnew file mode 100755 index 0000000..bb47f42 --- /dev/null +++ b/tools/cuda-ab/run/inplace_scale diff --git a/tools/cuda-ab/run/ip4 b/tools/cuda-ab/run/ip4 Binary files differnew file mode 100755 index 0000000..54937e6 --- /dev/null +++ b/tools/cuda-ab/run/ip4 diff --git a/tools/cuda-ab/run/ip5 b/tools/cuda-ab/run/ip5 Binary files differnew file mode 100755 index 0000000..c4b4a39 --- /dev/null +++ b/tools/cuda-ab/run/ip5 diff --git a/tools/cuda-ab/run/ip6 b/tools/cuda-ab/run/ip6 Binary files differnew file mode 100755 index 0000000..56815c2 --- /dev/null +++ b/tools/cuda-ab/run/ip6 diff --git a/tools/cuda-ab/run/ip_probe b/tools/cuda-ab/run/ip_probe Binary files differnew file mode 100755 index 0000000..08fb86e --- /dev/null +++ b/tools/cuda-ab/run/ip_probe diff --git a/tools/cuda-ab/run/ip_probe2 b/tools/cuda-ab/run/ip_probe2 Binary files differnew file mode 100755 index 0000000..dfe6ab9 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe2 diff --git a/tools/cuda-ab/run/ip_probe3 b/tools/cuda-ab/run/ip_probe3 Binary files differnew file mode 100755 index 0000000..8a34c05 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe3 diff --git a/tools/cuda-ab/run/launch_count b/tools/cuda-ab/run/launch_count Binary files differnew file mode 100755 index 0000000..0c67f45 --- /dev/null +++ b/tools/cuda-ab/run/launch_count diff --git a/tools/cuda-ab/run/memset2d_probe b/tools/cuda-ab/run/memset2d_probe Binary files differnew file mode 100755 index 0000000..f5bd071 --- /dev/null +++ b/tools/cuda-ab/run/memset2d_probe diff --git a/tools/cuda-ab/run/memset_probe b/tools/cuda-ab/run/memset_probe Binary files differnew file mode 100755 index 0000000..bd9df5e --- /dev/null +++ b/tools/cuda-ab/run/memset_probe diff --git a/tools/cuda-ab/run/rt_probe b/tools/cuda-ab/run/rt_probe Binary files differnew file mode 100755 index 0000000..48d866f --- /dev/null +++ b/tools/cuda-ab/run/rt_probe diff --git a/tools/cuda-ab/run/scale_probe b/tools/cuda-ab/run/scale_probe Binary files differnew file mode 100755 index 0000000..ca3fc20 --- /dev/null +++ b/tools/cuda-ab/run/scale_probe diff --git a/tools/cuda-ab/run/streams2_probe b/tools/cuda-ab/run/streams2_probe Binary files differnew file mode 100755 index 0000000..206d6d6 --- /dev/null +++ b/tools/cuda-ab/run/streams2_probe diff --git a/tools/cuda-ab/run/streams2d b/tools/cuda-ab/run/streams2d Binary files differnew file mode 100755 index 0000000..90b6df8 --- /dev/null +++ b/tools/cuda-ab/run/streams2d diff --git a/tools/cuda-ab/run/streams_events_probe b/tools/cuda-ab/run/streams_events_probe Binary files differnew file mode 100755 index 0000000..00f3539 --- /dev/null +++ b/tools/cuda-ab/run/streams_events_probe diff --git a/tools/cuda-ab/run/tgemm_probe b/tools/cuda-ab/run/tgemm_probe Binary files differnew file mode 100755 index 0000000..d9c9aa3 --- /dev/null +++ b/tools/cuda-ab/run/tgemm_probe diff --git a/tools/cuda-ab/run/transformer b/tools/cuda-ab/run/transformer Binary files differnew file mode 100755 index 0000000..bcde39e --- /dev/null +++ b/tools/cuda-ab/run/transformer diff --git a/tools/cuda-ab/run/transformer_async b/tools/cuda-ab/run/transformer_async Binary files differnew file mode 100755 index 0000000..2911b5c --- /dev/null +++ b/tools/cuda-ab/run/transformer_async diff --git a/tools/cuda-ab/run/transformer_stages b/tools/cuda-ab/run/transformer_stages Binary files differnew file mode 100755 index 0000000..1461a3e --- /dev/null +++ b/tools/cuda-ab/run/transformer_stages diff --git a/tools/cuda-ab/run/wedge_ab b/tools/cuda-ab/run/wedge_ab Binary files differnew file mode 100755 index 0000000..23a90aa --- /dev/null +++ b/tools/cuda-ab/run/wedge_ab diff --git a/tools/cuda-ab/run/wedge_bisect b/tools/cuda-ab/run/wedge_bisect Binary files differnew file mode 100755 index 0000000..5898fcf --- /dev/null +++ b/tools/cuda-ab/run/wedge_bisect diff --git a/tools/cuda-ab/run/wedge_recover b/tools/cuda-ab/run/wedge_recover Binary files differnew file mode 100755 index 0000000..3d9351c --- /dev/null +++ b/tools/cuda-ab/run/wedge_recover diff --git a/tools/cuda-ab/scale_probe.cu b/tools/cuda-ab/scale_probe.cu new file mode 100644 index 0000000..8a0df59 --- /dev/null +++ b/tools/cuda-ab/scale_probe.cu @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * scale_probe.cu - localize the deterministic "in-place scale writes only the + * first 4 KB page" failure. + * + * Modes (argv[1]): + * V : vadd(a,b->c); sync; D2H+verify; vadd again; D2H+verify + * I : vadd(a,b->c); sync; D2H+verify; scale(c,*2 in place); sync; + * D2H+verify (with page map); scale again; sync; D2H+verify; D2H again + * O : vadd(a,b->c); sync; D2H+verify; scale_out(c->d,*2); sync; + * D2H+verify (with page map) + * + * Every stage also runs a GPU-side verify kernel that counts matching + * elements reading the device buffers (engine's own view) and writes the + * counts to a small buffer, D2H'd afterwards: separates "device pages never + * written" (GPU count low) from "D2H readback stale" (GPU count high, host + * count low). + * + * Build: nvcc -O2 --cudart=static -o scale_probe scale_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +__global__ void scale_out(const float *c, float *d, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + d[i] = c[i] * s; +} + +/* verify kernel: out[0]=count(c==a+b), out[1]=count(c==2*(a+b)) */ +__global__ void vdigest(const float *a, const float *b, const float *c, int n, + unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c[i] == a[i] + b[i]) + atomicAdd(&out[0], 1ull); + if (c[i] == (a[i] + b[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "V"; + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hd; + float *da, *db, *dc, *dd; + unsigned long long *d_out, h_out[2]; + int i; + float k = argc > 2 ? atof(argv[2]) : 0.0f; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + { + FILE *f = fopen("/proc/uptime", "r"); + double up = -1.0; + + if (f) { if (fscanf(f, "%lf", &up) != 1) up = -1.0; fclose(f); } + printf("UPTIME %.3f\n", up); + } + printf("DEVICE %s mode=%s seed=%g\n", prop.name, mode, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hd = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + memset(hd, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&dd, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da=%p db=%p dc=%p dd=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)dd); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S1 vadd readback bad=%d/%d\n", b, n); + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S1 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", h_out[0], + h_out[1]); + } + + if (!strcmp(mode, "V")) { + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S2 vadd-again readback bad=%d/%d\n", b, n); + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "I")) { + int stage; + + for (stage = 0; stage < 2; stage++) { + float mult = stage == 0 ? 2.0f : 4.0f; + + scale<<<grid, 256>>>(dc, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) + b++; + printf("S%d inplace-scale readback bad=%d/%d (mult=%g)\n", + 2 + stage, b, n, mult); + if (stage == 0) + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hc[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * mult) + pb++; + if (pb) + printf("S%d PAGE %d bad=%d/256 first=%g want=%g\n", + 2 + stage, page, + pb, hc[page * 256], + (ha[page * 256] + + hb[page * 256]) * mult); + } + } + if (stage == 0) { + CHECK(cudaMemset(d_out, 0, + 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", + h_out[0], h_out[1]); + } + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "O")) { + scale_out<<<grid, 256>>>(dc, dd, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hd, dd, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hd[i] != (ha[i] + hb[i]) * 2.0f) + b++; + printf("S2 outofplace-scale readback bad=%d/%d\n", b, n); + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hd[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * 2.0f) + pb++; + if (pb) + printf("S2 PAGE %d bad=%d/256\n", page, + pb); + } + } + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dd, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST d_eq_ab=%llu d_eq_2ab=%llu\n", h_out[0], + h_out[1]); + printf("STATUS DONE\n"); + return 0; + } + return 1; +} diff --git a/tools/cuda-ab/soak.sh b/tools/cuda-ab/soak.sh index f870fda..4f1430f 100755 --- a/tools/cuda-ab/soak.sh +++ b/tools/cuda-ab/soak.sh @@ -135,6 +135,19 @@ run_one() { r="$(gssh "cat $out 2>/dev/null")" || true [ -n "$r" ] && text="$r" printf '%s' "$text" | grep -q 'EXIT=' && break + # The guest wrapper's EXIT= marker can be lost when the guest's + # process reaping stalls (observed: the client reached STATUS OK + # but its zombie was never reaped and `timeout` never wrote the + # marker). A terminal STATUS line is proof of completion by + # itself - accept it after a short grace so the EXIT= marker + # still wins when it does arrive. + if printf '%s' "$text" | grep -qE 'STATUS (OK|FAIL)'; then + sleep 10 + r="$(gssh "cat $out 2>/dev/null")" || true + [ -n "$r" ] && text="$r" + printf '%s' "$text" | grep -q 'EXIT=' && break + break + fi tnow=$(date +%s) if [ $((tnow - t0)) -gt $((TMO + 120)) ]; then if client_alive "$bin"; then @@ -179,7 +192,7 @@ log "SOAK START mode=$MODE seconds=$SECS guest_dir=$GUEST_DIR timeout=${TMO}s lo # cannot silently shrink the battery. found="" for i in 1 2 3; do - found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done")" || found="" + found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done; true")" || found="" [ -n "$found" ] && break sleep 3 done diff --git a/tools/cuda-ab/streams2_probe.cu b/tools/cuda-ab/streams2_probe.cu new file mode 100644 index 0000000..5bebab4 --- /dev/null +++ b/tools/cuda-ab/streams2_probe.cu @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2_probe.cu - instrumented streams2 to localize the deterministic + * failure: per-array mismatch counts, first-bad indices with values, and a + * GPU-side verify-kernel digest of the device buffers (to separate "device + * pages never written" from "D2H readback wrong"). + * + * Modes (argv[1]): + * A : exactly streams2's shape (control) + * B : like A but the two D2H copy-backs use synchronous cudaMemcpy + * C : like A but no events cross the streams; both streams are joined + * with cudaStreamSynchronize before any copy-back + * D : single stream (everything on s1) + * + * Build: nvcc -O2 --cudart=static -o streams2_probe streams2_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +/* GPU-side digest: counts of matches for c1==a1+b1 and c2==(a2+b2)*2 */ +__global__ void digest(const float *a1, const float *b1, const float *c1, + const float *a2, const float *b2, const float *c2, + int n, unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c1[i] == a1[i] + b1[i]) + atomicAdd(&out[0], 1ull); + if (c2[i] == (a2[i] + b2[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +static int report(const char *tag, const float *h_c, const float *h_a, + const float *h_b, const float *h_a2, const float *h_b2, + int n, float scale2) +{ + int bad = 0, i, shown = 0; + + for (i = 0; i < n; i++) { + float want = h_a[i] + h_b[i]; + + if (scale2 != 1.0f) + want = (h_a2[i] + h_b2[i]) * scale2; + if (h_c[i] != want) { + bad++; + if (shown < 4) { + printf("FIRSTBAD %s i=%d got=%g want=%g\n", + tag, i, h_c[i], want); + shown++; + } + } + } + printf("BAD %s n=%d bad=%d\n", tag, n, bad); + return bad; +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "A"; + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + unsigned long long *d_out, h_out[2]; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s mode=%s\n", prop.name, mode); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + memset(h_c1, 0xCD, bytes); + memset(h_c2, 0xCD, bytes); + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da1=%p db1=%p dc1=%p da2=%p db2=%p dc2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, (void *)d_a2, + (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + if (!strcmp(mode, "D")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s1)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s1>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + scale<<<grid, 256, 0, s1>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + goto verify; + } + + if (!strcmp(mode, "S")) { + /* staged verification: check device content after every step */ + float *hb1 = (float *)malloc(bytes); + float *hb2 = (float *)malloc(bytes); + float *hb3 = (float *)malloc(bytes); + + CHECK(cudaMemcpy(d_a1, h_a1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b1, h_b1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_a2, h_a2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b2, h_b2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(hb1, d_a1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a1[i]) b++; + printf("STAGE-IN da1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b1[i]) b++; + printf("STAGE-IN db1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_a2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a2[i]) b++; + printf("STAGE-IN da2 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b2[i]) b++; + printf("STAGE-IN db2 bad=%d\n", b); } + + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; + if (first < 0) { first = i; + printf("S1-FIRSTBAD i=%d got=%g want=%g a2b2=%g\n", + i, hb1[i], h_a1[i]+h_b1[i], h_a2[i]+h_b2[i]); } } + printf("STAGE vadd1->c1 readback bad=%d first=%d\n", b, first); } + + CHECK(cudaDeviceSynchronize()); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb2, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb2[i] != h_a2[i] + h_b2[i]) { b++; if (first < 0) first = i; } + printf("STAGE vadd2->c2 readback bad=%d first=%d\n", b, first); } + + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb3, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb3[i] != (h_a2[i] + h_b2[i]) * 2.0f) { b++; if (first < 0) first = i; } + printf("STAGE scale->c2 readback bad=%d first=%d\n", b, first); } + /* second readback of c1 AFTER everything: does it change? */ + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; if (first < 0) first = i; } + printf("STAGE c1-late readback bad=%d first=%d\n", b, first); } + printf("STATUS OK\n"); + return 0; + } + + if (!strcmp(mode, "C")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + goto verify; + } + + /* A and B: original shape */ + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + if (!strcmp(mode, "B")) { + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + } else { + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + } + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + +verify: + { + int bad1, bad2; + + bad1 = report("C1", h_c1, h_a1, h_b1, h_a2, h_b2, n, 1.0f); + bad2 = report("C2", h_c2, h_a2, h_b2, h_a2, h_b2, n, s2); + printf("STREAMS n=%d bad=%d\n", n, bad1 + bad2); + printf("TIMING ms=%.3f\n", ms); + + /* GPU-side digest of the device buffers */ + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + digest<<<grid, 256>>>(d_a1, d_b1, d_c1, d_a2, d_b2, d_c2, n, + d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("GPU-DIGEST c1_ok=%llu c2_ok=%llu\n", h_out[0], + h_out[1]); + printf("STATUS %s\n", (bad1 | bad2) ? "FAIL" : "OK"); + } + return 0; +} diff --git a/tools/cuda-ab/streams2d.cu b/tools/cuda-ab/streams2d.cu new file mode 100644 index 0000000..04a0866 --- /dev/null +++ b/tools/cuda-ab/streams2d.cu @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2d.cu - streams2 with per-array / per-range diagnostic reporting. + * + * Same workload as streams2.cu (two streams, cross-stream event ordering) + * but instead of one "bad" count it reports, per array and per 4 KB block, + * how many elements mismatched, and prints the device pointers so a host + * log can be correlated with what the engine read/wrote. + * + * Output lines: + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * PTR <name>=0x<hex> (d_a1 d_b1 d_c1 d_a2 d_b2 d_c2) + * DIFF c1 total=<n> blocks=<first block index with mismatches> + * DIFF c2 total=<n> blocks=<...> + * STREAMS n=<count> bad=<mismatches> + * TIMING ms=<elapsed> + * STATUS OK|FAIL + * + * Build: nvcc -O2 --cudart=static -o streams2d streams2d.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +static void report_diff(const char *name, const float *got, + const float *exp, int n) +{ + int i, bad = 0, first = -1, last = -1; + int blk_hits = 0, blk_first = -1; + int nb = (n + 1023) / 1024; + int b; + + for (i = 0; i < n; i++) { + if (got[i] != exp[i]) { + bad++; + if (first < 0) + first = i; + last = i; + } + } + for (b = 0; b < nb; b++) { + int j, hit = 0; + + for (j = b * 1024; j < (b + 1) * 1024 && j < n; j++) + if (got[j] != exp[j]) { + hit = 1; + break; + } + if (hit) { + blk_hits++; + if (blk_first < 0) + blk_first = b; + } + } + printf("DIFF %s total=%d first=%d last=%d blocks=%d/%d blk_first=%d\n", + name, bad, first, last, blk_hits, nb, blk_first); + if (first >= 0) + printf("DIFF %s sample got[%d]=%a exp[%d]=%a got[%d]=%a exp[%d]=%a\n", + name, first, got[first], first, exp[first], last, + got[last], last, exp[last]); +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + if (!h_a1 || !h_b1 || !h_c1 || !h_a2 || !h_b2 || !h_c2) + return 1; + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + printf("PTR d_a1=%p d_b1=%p d_c1=%p d_a2=%p d_b2=%p d_c2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, + (void *)d_a2, (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + report_diff("c1", h_c1, h_a1, n); + { + float *exp2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) + exp2[i] = (h_a2[i] + h_b2[i]) * s2; + report_diff("c2", h_c2, exp2, n); + for (i = 0; i < n; i++) { + if (h_c1[i] != h_a1[i] + h_b1[i]) + bad++; + if (h_c2[i] != exp2[i]) + bad++; + } + free(exp2); + } + + printf("STREAMS n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/streams_events_probe.cu b/tools/cuda-ab/streams_events_probe.cu new file mode 100644 index 0000000..94aef69 --- /dev/null +++ b/tools/cuda-ab/streams_events_probe.cu @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams_events_probe.cu - instrumented victory_streams_events. + * + * Same 3-stream event chain (c0 = a0+b0; c1 = (a1+b1)+c0; c2 = (a2+b2)+c1) + * but with per-buffer diagnostics: device pointers, per-buffer bad counts, + * and the first 8 expected/actual values of every output so a stale-input + * vs stale-readback failure is distinguishable in one run. + * + * Output lines: + * DEVICE ... + * DPTRS da0..dc2 (device), ha0..hc2 (host) + * BUF s=<n> bad=<n> + * FIRST s=<n> exp=<v0..v7> got=<v0..v7> + * STREAMS3 n=<count> bad=<mismatches> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define NSTREAMS 3 + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vadd_into(float *c, const float *s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] + s[i]; +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 16; + size_t bytes = (size_t)n * sizeof(float); + float *h_a[NSTREAMS], *h_b[NSTREAMS], *h_c[NSTREAMS]; + float *d_a[NSTREAMS], *d_b[NSTREAMS], *d_c[NSTREAMS]; + cudaStream_t st[NSTREAMS]; + cudaEvent_t ev[NSTREAMS], t0, t1; + float ms = 0.0f; + int bad = 0, i, s; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + for (s = 0; s < NSTREAMS; s++) { + h_a[s] = (float *)malloc(bytes); + h_b[s] = (float *)malloc(bytes); + h_c[s] = (float *)malloc(bytes); + if (!h_a[s] || !h_b[s] || !h_c[s]) + return 1; + for (i = 0; i < n; i++) { + h_a[s][i] = (float)((i + s) % 1000) * 0.5f; + h_b[s][i] = (float)((i + 2 * s) % 997) * 0.25f; + } + CHECK(cudaMalloc(&d_a[s], bytes)); + CHECK(cudaMalloc(&d_b[s], bytes)); + CHECK(cudaMalloc(&d_c[s], bytes)); + CHECK(cudaStreamCreate(&st[s])); + CHECK(cudaEventCreate(&ev[s])); + } + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + printf("DPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" da%d=%p db%d=%p dc%d=%p", s, (void *)d_a[s], s, + (void *)d_b[s], s, (void *)d_c[s]); + printf("\n"); + printf("HPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" ha%d=%p hb%d=%p hc%d=%p", s, (void *)h_a[s], s, + (void *)h_b[s], s, (void *)h_c[s]); + printf("\n"); + + CHECK(cudaEventRecord(t0)); + + for (s = 0; s < NSTREAMS; s++) { + CHECK(cudaMemcpyAsync(d_a[s], h_a[s], bytes, + cudaMemcpyHostToDevice, st[s])); + CHECK(cudaMemcpyAsync(d_b[s], h_b[s], bytes, + cudaMemcpyHostToDevice, st[s])); + vadd<<<grid, 256, 0, st[s]>>>(d_a[s], d_b[s], d_c[s], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[s], st[s])); + } + + CHECK(cudaStreamWaitEvent(st[1], ev[0], 0)); + vadd_into<<<grid, 256, 0, st[1]>>>(d_c[1], d_c[0], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[1], st[1])); + + CHECK(cudaStreamWaitEvent(st[2], ev[1], 0)); + vadd_into<<<grid, 256, 0, st[2]>>>(d_c[2], d_c[1], n); + CHECK(cudaGetLastError()); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaMemcpyAsync(h_c[s], d_c[s], bytes, + cudaMemcpyDeviceToHost, st[s])); + CHECK(cudaEventRecord(t1, st[2])); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaStreamSynchronize(st[s])); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + for (s = 0; s < NSTREAMS; s++) { + int b = 0; + float exp0[8], got0[8]; + + for (i = 0; i < n; i++) { + float c0 = h_a[0][i] + h_b[0][i]; + float c1 = h_a[1][i] + h_b[1][i] + c0; + float c2 = h_a[2][i] + h_b[2][i] + c1; + float want = (s == 0) ? c0 : (s == 1) ? c1 : c2; + + if (i < 8) { + exp0[i] = want; + got0[i] = h_c[s][i]; + } + if (h_c[s][i] != want) + b++; + } + bad += b; + printf("BUF s=%d bad=%d\n", s, b); + printf("FIRST s=%d exp=", s); + for (i = 0; i < 8; i++) + printf("%.4f ", exp0[i]); + printf(" got="); + for (i = 0; i < 8; i++) + printf("%.4f ", got0[i]); + printf("\n"); + } + + printf("STREAMS3 n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/tgemm_probe.cu b/tools/cuda-ab/tgemm_probe.cu new file mode 100644 index 0000000..f0a1084 --- /dev/null +++ b/tools/cuda-ab/tgemm_probe.cu @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * tgemm_probe.cu - minimal isolation for the transformer K/V divergence. + * + * Reproduces exactly the transformer's Q/K/V GEMM shape (X[64x256] x + * W[256x256], tile 16) for three weight matrices uploaded back-to-back + * (wq, wk, wv), then verifies each against a CPU reference and prints + * per-GEMM bad counts plus a bad-position histogram (first bad index per + * 16-row band and per 16-col band) so a copy-side hole is distinguishable + * from a kernel-side error. + * + * Output: + * DEVICE ... + * GEMM name=<q|k|v> bad=<n> first=(row,col) got=.. ref=.. + * BADPOS name=.. rows=<r0,r1,..> cols=<c0,c1,..> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define SEQ 64 +#define DH 256 +#define TILE 16 + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static float xval(int i, int j) +{ + return (float)(((i * 7 + j * 3 + 1) % 9) - 4) * 0.25f; +} + +int main(void) +{ + cudaDeviceProp prop; + float *h_x, *h_w, *h_c, *h_ref; + float *d_x, *d_w[3], *d_c[3]; + const int seqdh = SEQ * DH, dh2 = DH * DH; + int g, i, j, k; + + setvbuf(stdout, NULL, _IONBF, 0); + if (cudaGetDeviceProperties(&prop, 0) != cudaSuccess) { + printf("STATUS FAIL\n"); + return 1; + } + printf("DEVICE %s cc=%d.%d\n", prop.name, prop.major, prop.minor); + + h_x = (float *)malloc(seqdh * 4); + h_w = (float *)malloc(dh2 * 4); + h_c = (float *)malloc(seqdh * 4); + h_ref = (float *)malloc(seqdh * 4); + + if (cudaMalloc(&d_x, seqdh * 4) != cudaSuccess) + return 1; + for (g = 0; g < 3; g++) { + if (cudaMalloc(&d_w[g], dh2 * 4) != cudaSuccess) + return 1; + if (cudaMalloc(&d_c[g], seqdh * 4) != cudaSuccess) + return 1; + } + + for (i = 0; i < seqdh; i++) + h_x[i] = xval(i / DH, i % DH); + + /* Upload X once (the transformer's shared input). */ + if (cudaMemcpy(d_x, h_x, seqdh * 4, cudaMemcpyHostToDevice) != + cudaSuccess) + return 1; + /* Verify the X upload itself (first row + full buffer). */ + { + float *h_xb = (float *)malloc(seqdh * 4); + int xbad = 0, xr0 = 0; + + if (!h_xb) + return 1; + if (cudaMemcpy(h_xb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (h_xb[i] != h_x[i]) { + xbad++; + if (i < DH) + xr0++; + } + printf("XVERIFY bad=%d row0_bad=%d\n", xbad, xr0); + free(h_xb); + } + + for (g = 0; g < 3; g++) { + int bad = 0, first_r = -1, first_c = -1; + float got = 0, ref = 0; + int rows[64] = {0}, cols[64] = {0}; + int nr = 0, nc = 0; + int pass; + + for (i = 0; i < dh2; i++) + h_w[i] = wval(i, g + 1); + if (cudaMemcpy(d_w[g], h_w, dh2 * 4, + cudaMemcpyHostToDevice) != cudaSuccess) + return 1; + + dim3 blk(TILE, TILE); + dim3 grd(DH / TILE, SEQ / TILE); + + for (pass = 0; pass < 3; pass++) { + float *hc = h_c, *hxb = h_ref; + int xbad = 0; + + gemm_nn<<<grd, blk>>>(d_x, d_w[g], d_c[g], SEQ, DH, DH); + if (cudaDeviceSynchronize() != cudaSuccess) { + printf("SYNC FAIL g=%d\n", g); + return 1; + } + if (cudaMemcpy(hc, d_c[g], seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + /* re-verify X after the launch */ + if (cudaMemcpy(hxb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (hxb[i] != h_x[i]) { + printf("XDIFF pass=%d i=%d got=%.6f want=%.6f\n", + pass, i, hxb[i], h_x[i]); + xbad++; + } + bad = 0; + nr = 0; + nc = 0; + for (i = 0; i < SEQ; i++) + for (j = 0; j < DH; j++) { + double acc = 0.0; + float r, a, diff; + + for (k = 0; k < DH; k++) + acc += (double)h_x[i * DH + k] * + (double)h_w[k * DH + j]; + r = (float)acc; + a = hc[i * DH + j]; + diff = a - r; + if (diff < 0) + diff = -diff; + if (diff > 1e-4f) { + int rr = i, cc = j; + + if (bad == 0) { + first_r = rr; + first_c = cc; + got = a; + ref = r; + } + bad++; + if (nr < 64 && (bad == 1 || + rr != rows[nr - 1])) + rows[nr++] = rr; + if (nc < 64 && (bad == 1 || + cc != cols[nc - 1])) + cols[nc++] = cc; + } + } + printf("PASS name=%c pass=%d bad=%d xbad=%d first=(%d,%d) got=%.6f ref=%.6f\n", + "qkv"[g], pass, bad, xbad, first_r, first_c, + got, ref); + if (bad) { + printf("ROW0 name=%c got=", "qkv"[g]); + for (i = 0; i < 8; i++) + printf("%.6f ", hc[i]); + printf(" ref="); + for (i = 0; i < 8; i++) { + double acc = 0.0; + + for (k = 0; k < DH; k++) + acc += (double)h_x[k] * + (double)h_w[k * DH + i]; + printf("%.6f ", (float)acc); + } + printf("\n"); + } + } + (void)got; + (void)ref; + (void)first_r; + (void)first_c; + (void)nr; + (void)nc; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/transformer.cu b/tools/cuda-ab/transformer.cu new file mode 100644 index 0000000..3044522 --- /dev/null +++ b/tools/cuda-ab/transformer.cu @@ -0,0 +1,472 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward ---- */ + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/transformer_async.cu b/tools/cuda-ab/transformer_async.cu new file mode 100644 index 0000000..213363a --- /dev/null +++ b/tools/cuda-ab/transformer_async.cu @@ -0,0 +1,522 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward (async chain, stage readbacks) ---- */ + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + { + float *hs = (float *)malloc(b_seqdh); + float *hs2 = (float *)malloc(b_seq2); + double *rw = (double *)malloc((size_t)dh2 * sizeof(double)); + double *rq = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rk = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rv = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rs = (double *)malloc((size_t)seq2 * sizeof(double)); + double *ra = (double *)malloc((size_t)seq2 * sizeof(double)); + double *rav = (double *)malloc((size_t)seqdh * sizeof(double)); + double *ry = (double *)malloc((size_t)seqdh * sizeof(double)); + int st; + + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 1); + ref_gemm(r_x, rw, rq, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 2); + ref_gemm(r_x, rw, rk, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 3); + ref_gemm(r_x, rw, rv, SEQ, DH, DH); + ref_gemm_nt(rq, rk, rs, SEQ, SEQ, DH); + ref_softmax(rs, ra, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(ra, rv, rav, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 4); + ref_gemm(rav, rw, ry, SEQ, DH, DH); + + CHECK(cudaMemcpy(hs, d_q, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rq, seqdh); + printf("ASYNC Q bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_k, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rk, seqdh); + printf("ASYNC K bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_v, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rv, seqdh); + printf("ASYNC V bad=%d\n", st); + CHECK(cudaMemcpy(hs2, d_s, b_seq2, cudaMemcpyDeviceToHost)); + st = count_bad(hs2, rs, seq2); + printf("ASYNC S bad=%d\n", st); + CHECK(cudaMemcpy(hs2, d_a, b_seq2, cudaMemcpyDeviceToHost)); + st = count_bad(hs2, ra, seq2); + printf("ASYNC A bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_av, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rav, seqdh); + printf("ASYNC AV bad=%d\n", st); + st = count_bad(h_y, ry, seqdh); + printf("ASYNC Y bad=%d\n", st); + } + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/transformer_stages.cu b/tools/cuda-ab/transformer_stages.cu new file mode 100644 index 0000000..d7f67c0 --- /dev/null +++ b/tools/cuda-ab/transformer_stages.cu @@ -0,0 +1,553 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +static void dump_bad(const char *tag, const float *gpu, const double *ref, + int n) +{ + int i, shown = 0; + + for (i = 0; i < n && shown < 8; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) { + printf("BAD %s i=%d gpu=%.6f ref=%.6f (tile %d,%d row %d col %d)\n", + tag, i, g, r, i / 16, (i % 256) / 16, + i / 256, i % 256); + shown++; + } + } +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward, stage-by-stage verify ---- */ + { + float *h_stage = (float *)malloc(b_seqdh); + float *h_s2 = (float *)malloc(b_seq2); + double *r_stage = (double *)malloc((size_t)seqdh * sizeof(double)); + double *r_s2 = (double *)malloc((size_t)seq2 * sizeof(double)); + double *r_w2 = (double *)malloc((size_t)dh2 * sizeof(double)); + int st; + + if (!h_stage || !h_s2 || !r_stage || !r_s2 || !r_w2) + return 1; + + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_q, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE Q bad=%d\n", st); + dump_bad("Q", h_stage, r_stage, seqdh); + + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_k, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE K bad=%d\n", st); + dump_bad("K", h_stage, r_stage, seqdh); + + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_v, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE V bad=%d\n", st); + dump_bad("V", h_stage, r_stage, seqdh); + + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_s2, d_s, b_seq2, cudaMemcpyDeviceToHost)); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + st = count_bad(h_s2, r_s, seq2); + printf("STAGE S bad=%d\n", st); + dump_bad("S", h_s2, r_s, seq2); + + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_s2, d_a, b_seq2, cudaMemcpyDeviceToHost)); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + st = count_bad(h_s2, r_a, seq2); + printf("STAGE A bad=%d\n", st); + dump_bad("A", h_s2, r_a, seq2); + + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_av, b_seqdh, cudaMemcpyDeviceToHost)); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + st = count_bad(h_stage, r_av, seqdh); + printf("STAGE AV bad=%d\n", st); + + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w2, r_y, SEQ, DH, DH); + st = count_bad(h_y, r_y, seqdh); + printf("STAGE Y bad=%d\n", st); + } + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/wedge_ab.cu b/tools/cuda-ab/wedge_ab.cu new file mode 100644 index 0000000..2d88f92 --- /dev/null +++ b/tools/cuda-ab/wedge_ab.cu @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_ab.cu - discriminate the wedge trigger: is it the second launch of + * the SAME program, or the second launch of the context regardless of + * program? + * + * Sequence (each launch verified against mutated inputs): + * 1. vadd (program A) + * 2. vsub (program B, different code) + * 3. vadd (program A again) + * 4. vsub (program B again) + * + * If launch 2 (B) executes -> the trigger is same-program reuse. + * If launch 2 (B) wedges -> the trigger is the second launch in the context. + * + * Build: nvcc -O2 --cudart=static -o wedge_ab wedge_ab.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vsub(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] - b[i]; +} + +int main(void) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc; + float *da, *db, *dc; + int i, bad, step; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + if (!ha || !hb || !hc) + return 1; + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + + for (step = 0; step < 4; step++) { + const char *which = (step & 1) ? "vsub" : "vadd"; + int is_sub = step & 1; + + if (step > 0) { + ha[step] += 1.0f; + hb[step * 7 + 3] += 2.0f; + } + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + if (is_sub) + vsub<<<(n + 255) / 256, 256>>>(da, db, dc, n); + else + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + bad = 0; + for (i = 0; i < n; i++) { + float expect = is_sub ? ha[i] - hb[i] : ha[i] + hb[i]; + + if (hc[i] != expect) + bad++; + } + printf("STEP %d %s bad=%d/%d\n", step, which, bad, n); + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/wedge_bisect.cu b/tools/cuda-ab/wedge_bisect.cu new file mode 100644 index 0000000..e062a2b --- /dev/null +++ b/tools/cuda-ab/wedge_bisect.cu @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_bisect.cu - bisect the wedge trigger: which inter-launch activity + * makes the SECOND vadd launch no-execute? + * + * Every iteration is: mutate host inputs -> H2D da/db -> launch -> sync -> + * D2H dc -> verify (the wedge_recover shape). Modes add activity between + * the verify and the next launch: + * + * mode A: none (pure second-launch class) + * mode B: an extra H2D of the same buffers (CE write before the launch) + * mode C: an extra D2H readback (CE read before the launch) + * mode D: a 500 ms sleep (no CE traffic) + * + * A no-execute launch is reported (bad>0) not masked: the inputs mutate every + * iteration. + * + * Build: nvcc -O2 --cudart=static -o wedge_bisect wedge_bisect.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> +#include <unistd.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +int main(int argc, char **argv) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc; + float *da, *db, *dc; + const char *mode = argc > 1 ? argv[1] : "A"; + int iters = argc > 2 ? atoi(argv[2]) : 4; + int i, bad = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + if (!ha || !hb || !hc) + return 1; + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + for (i = 0; i < iters; i++) { + int j, itbad = 0; + + if (i > 0) { + ha[i % n] += 1.0f; + hb[(i * 7 + 3) % n] += 2.0f; + } + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + for (j = 0; j < n; j++) + if (hc[j] != ha[j] + hb[j]) + itbad++; + printf("MODE %s iter=%d bad=%d/%d\n", mode, i, itbad, n); + bad += itbad; + if (i + 1 < iters) { + if (mode[0] == 'B') { + CHECK(cudaMemcpy(da, ha, bytes, + cudaMemcpyHostToDevice)); + } else if (mode[0] == 'C') { + CHECK(cudaMemcpy(hc, dc, bytes, + cudaMemcpyDeviceToHost)); + } else if (mode[0] == 'D') { + usleep(500000); + } + } + } + printf("STATUS %s\n", bad ? "FAIL" : "OK"); + return bad ? 1 : 0; +} diff --git a/tools/cuda-ab/wedge_recover.cu b/tools/cuda-ab/wedge_recover.cu new file mode 100644 index 0000000..fd8f3f9 --- /dev/null +++ b/tools/cuda-ab/wedge_recover.cu @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_recover.cu - reproduce the per-launch no-execute (wedge) and test + * whether an idle gap heals a wedged context. + * + * v2 (session 10, r416 finding): the v1 loop reused one deterministic + * kernel+buffers, so a no-write launch was MASKED - the stale buffer held + * exactly the previous iteration's (identical) values and the verify passed. + * r416 ran 60/60 "OK" while 58 consecutive launches no-wrote underneath. + * + * v2 mutates the inputs every iteration (H2D of fresh ha/hb), so a stale + * readback cannot match the new expected values: a no-write launch must + * report BAD. On BAD the same iteration is retried after a sleep, and the + * readback is classified: STALE-PREV (byte-identical to the previous + * iteration's output = launch never executed) vs PARTIAL/OTHER (engine wrote + * something). Latency and per-iteration outcome are printed. + * + * Build: nvcc -O2 --cudart=static -o wedge_recover wedge_recover.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> +#include <unistd.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +int main(int argc, char **argv) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hc_prev; + float *da, *db, *dc; + int iters = argc > 1 ? atoi(argv[1]) : 40; + int sleep_s = argc > 2 ? atoi(argv[2]) : 20; + int fails = 0, heals = 0, i; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hc_prev = (float *)malloc(bytes); + if (!ha || !hb || !hc || !hc_prev) + return 1; + /* iteration 0 baseline; mutated each iteration below (v2) */ + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + memset(hc_prev, 0, bytes); + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + for (i = 0; i < iters; i++) { + int attempt = 0; + long long ms[2] = { 0, 0 }; + + /* + * v2: fresh inputs every iteration so a stale (never + * executed) launch cannot match. Mutate after iteration 0. + */ + if (i > 0) { + ha[i % n] += 1.0f; + hb[(i * 7 + 3) % n] += 2.0f; + CHECK(cudaMemcpy(da, ha, bytes, + cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, + cudaMemcpyHostToDevice)); + } + + for (;;) { + int bad = 0, stale_prev = 1, j; + cudaEvent_t e0, e1; + float ems = 0; + + cudaEventCreate(&e0); + cudaEventCreate(&e1); + cudaEventRecord(e0); + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + cudaEventRecord(e1); + CHECK(cudaDeviceSynchronize()); + cudaEventElapsedTime(&ems, e0, e1); + cudaEventDestroy(e0); + cudaEventDestroy(e1); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + for (j = 0; j < n; j++) { + float expect = ha[j] + hb[j]; + + if (hc[j] != expect) { + bad++; + if (hc[j] != hc_prev[j]) + stale_prev = 0; + } + } + ms[attempt & 1] = (long long)(ems * 1000.0f); + if (bad == 0) { + if (attempt > 0) + heals++; + printf("ITER %d OK attempt=%d launch_us=%lld\n", + i, attempt, ms[attempt & 1]); + memcpy(hc_prev, hc, bytes); + break; + } + fails++; + printf("ITER %d BAD bad=%d/%d attempt=%d launch_us=%lld class=%s\n", + i, bad, n, attempt, ms[attempt & 1], + stale_prev ? "STALE-PREV(NOWRITE)" + : "OTHER/PARTIAL"); + if (attempt >= 3) { + printf("ITER %d GIVEUP\n", i); + break; + } + attempt++; + printf("ITER %d SLEEP %ds before retry\n", i, sleep_s); + sleep(sleep_s); + } + } + printf("SUMMARY iters=%d fails=%d heals=%d sleep_s=%d\n", + iters, fails, heals, sleep_s); + printf("STATUS %s\n", fails == 0 ? "OK" : "FAIL"); + return fails == 0 ? 0 : 1; +} diff --git a/tools/guest-diag/egl_frame.c b/tools/guest-diag/egl_frame.c new file mode 100644 index 0000000..ed10e39 --- /dev/null +++ b/tools/guest-diag/egl_frame.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * egl_frame.c - minimal EGL/GLES2 offscreen render test for the vGPU guest. + * + * Tries, in order: the EGL_EXT_platform_device platform with each enumerated + * EGLDeviceEXT, EGL_MESA_platform_surfaceless, then EGL_DEFAULT_DISPLAY. + * For the first display that initializes it creates an offscreen pbuffer + * context, renders colour bands into an FBO, reads them back and writes a + * PPM. One greppable line per stage. + * + * Build (guest): gcc -O2 -o egl_frame egl_frame.c -lEGL -lGLESv2 + */ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <EGL/egl.h> +#include <EGL/eglext.h> +#include <GLES2/gl2.h> + +#define W 320 +#define H 240 + +static EGLDisplay try_display(const char *name, EGLDisplay dpy, + const char *dev_name) +{ + EGLint major = 0, minor = 0; + + if (dpy == EGL_NO_DISPLAY) { + printf("EGL_PLATFORM %s -> no display\n", name); + return EGL_NO_DISPLAY; + } + if (!eglInitialize(dpy, &major, &minor)) { + printf("EGL_PLATFORM %s -> init fail 0x%x\n", name, + eglGetError()); + return EGL_NO_DISPLAY; + } + printf("EGL_PLATFORM %s ok %d.%d vendor=%s%s%s\n", name, major, minor, + eglQueryString(dpy, EGL_VENDOR), + dev_name ? " device=" : "", dev_name ? dev_name : ""); + return dpy; +} + +static EGLDisplay get_display(void) +{ + PFNEGLGETPLATFORMDISPLAYEXTPROC gp = + (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress( + "eglGetPlatformDisplayEXT"); + PFNEGLQUERYDEVICESEXTPROC qd = + (PFNEGLQUERYDEVICESEXTPROC)eglGetProcAddress( + "eglQueryDevicesEXT"); + EGLDisplay dpy; + + if (gp && qd) { + EGLDeviceEXT devs[8]; + EGLint n = 0; + int i; + + if (qd(8, devs, &n) && n > 0) { + printf("EGL_DEVICES n=%d\n", n); + for (i = 0; i < n; i++) { + EGLDisplay d = gp(EGL_PLATFORM_DEVICE_EXT, + devs[i], NULL); + + dpy = try_display("device", d, "enum"); + if (dpy != EGL_NO_DISPLAY) + return dpy; + } + } else { + printf("EGL_DEVICES n=0\n"); + } + if (gp) { + dpy = try_display("surfaceless", + gp(EGL_PLATFORM_SURFACELESS_MESA, + NULL, NULL), NULL); + if (dpy != EGL_NO_DISPLAY) + return dpy; + } + } + return try_display("default", eglGetDisplay(EGL_DEFAULT_DISPLAY), + NULL); +} + +static int render(EGLDisplay dpy) +{ + EGLConfig cfg; + EGLint ncfg = 0; + EGLSurface surf; + EGLContext ctx; + GLuint fbo = 0, tex = 0; + unsigned char *px; + FILE *f; + int i; + EGLint attrs[] = { + EGL_SURFACE_TYPE, EGL_PBUFFER_BIT, + EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT, + EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, + EGL_BLUE_SIZE, 8, EGL_ALPHA_SIZE, 8, + EGL_NONE + }; + EGLint pattrs[] = { EGL_WIDTH, W, EGL_HEIGHT, H, EGL_NONE }; + EGLint cattrs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; + + if (!eglChooseConfig(dpy, attrs, &cfg, 1, &ncfg) || ncfg < 1) { + printf("EGL_CONFIG_FAIL err=0x%x\n", eglGetError()); + return 1; + } + surf = eglCreatePbufferSurface(dpy, cfg, pattrs); + if (surf == EGL_NO_SURFACE) { + printf("EGL_PBUFFER_FAIL err=0x%x\n", eglGetError()); + return 1; + } + ctx = eglCreateContext(dpy, cfg, EGL_NO_CONTEXT, cattrs); + if (ctx == EGL_NO_CONTEXT) { + printf("EGL_CONTEXT_FAIL err=0x%x\n", eglGetError()); + return 1; + } + if (!eglMakeCurrent(dpy, surf, surf, ctx)) { + printf("EGL_CURRENT_FAIL err=0x%x\n", eglGetError()); + return 1; + } + printf("GL_RENDERER %s\n", glGetString(GL_RENDERER)); + printf("GL_VERSION %s\n", glGetString(GL_VERSION)); + + glGenTextures(1, &tex); + glBindTexture(GL_TEXTURE_2D, tex); + glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, W, H, 0, GL_RGBA, + GL_UNSIGNED_BYTE, NULL); + glGenFramebuffers(1, &fbo); + glBindFramebuffer(GL_FRAMEBUFFER, fbo); + glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, + GL_TEXTURE_2D, tex, 0); + if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != + GL_FRAMEBUFFER_COMPLETE) { + printf("GL_FBO_FAIL\n"); + return 1; + } + for (i = 0; i < 16; i++) { + glEnable(GL_SCISSOR_TEST); + glScissor(0, i * H / 16, W, H / 16); + glClearColor((float)i / 16.0f, 1.0f - (float)i / 16.0f, + 0.5f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT); + } + glFinish(); + printf("GL_ERR 0x%x\n", glGetError()); + px = (unsigned char *)malloc(W * H * 4); + glReadPixels(0, 0, W, H, GL_RGBA, GL_UNSIGNED_BYTE, px); + { + const char *out = getenv("EGL_FRAME_OUT"); + + if (!out) + out = "/tmp/egl_frame.ppm"; + f = fopen(out, "wb"); + if (!f) { + printf("PPM_OPEN_FAIL\n"); + return 1; + } + fprintf(f, "P6\n%d %d\n255\n", W, H); + for (i = 0; i < W * H; i++) + fwrite(px + i * 4, 1, 3, f); + fclose(f); + printf("EGL_FRAME_WRITTEN %s\n", out); + } + printf("EGL_FRAME_OK\n"); + return 0; +} + +int main(void) +{ + EGLDisplay dpy; + const char *ext; + + setvbuf(stdout, NULL, _IONBF, 0); + ext = eglQueryString(EGL_NO_DISPLAY, EGL_EXTENSIONS); + printf("EGL_CLIENT_EXT %s\n", ext ? ext : "(none)"); + dpy = get_display(); + if (dpy == EGL_NO_DISPLAY) { + printf("EGL_ALL_PLATFORMS_FAILED\n"); + return 1; + } + return render(dpy); +} diff --git a/tools/guest-diag/pin_gr_stream.py b/tools/guest-diag/pin_gr_stream.py index 6ba51ee..6eb5be7 100644 --- a/tools/guest-diag/pin_gr_stream.py +++ b/tools/guest-diag/pin_gr_stream.py @@ -211,7 +211,7 @@ def _demand_loop(): except Exception: continue now = time.time() - for m in re.finditer(r"(?:stalled VA #\d+|pending release starved va)=(0x[0-9a-f]+)", + for m in re.finditer(r"(?:stalled VA #\d+[:=]\s*|pending release starved va=)(0x[0-9a-f]+)", (r.stdout or "")[-200000:]): va = int(m.group(1), 16) track.setdefault(va, [now, 0.0]) diff --git a/tools/lg/frame_reader.py b/tools/lg/frame_reader.py new file mode 100644 index 0000000..f8bab40 --- /dev/null +++ b/tools/lg/frame_reader.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# frame_reader.py - read the GPU-rendered frames the guest publishes into the +# kvmfr ivshmem BAR (see tools/cuda-ab/frame_producer.cu) and save them on the +# host as PPM/PNG. This is the host half of the guest frame-producer path: +# the QEMU ivshmem-plain device is backed by /dev/shm/kvmfr-<uuid>, which the +# guest maps through /sys/bus/pci/devices/0000:00:03.0/resource2. +# +# Usage: frame_reader.py [--shm /dev/shm/kvmfr-<uuid>] [--frames N] [--out DIR] +import argparse +import mmap +import os +import struct +import sys +import time + +MAGIC = 0x4B564D46 +FRAME_OFFSET = 0x100000 +# magic, version, width, height, stride, format, seq, reserved, timestamp_ns +HDR = struct.Struct("<IIIIIIIIQ") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--shm", default=None) + ap.add_argument("--frames", type=int, default=5) + ap.add_argument("--out", default="/tmp/opencode/frames") + ap.add_argument("--timeout", type=float, default=120.0) + args = ap.parse_args() + + if not args.shm: + import glob + cands = [p for p in glob.glob("/dev/shm/kvmfr-*") + if os.path.getsize(p) > 0] + if not cands: + print("NO_SHM_FILE", file=sys.stderr) + return 1 + args.shm = max(cands, key=os.path.getsize) + os.makedirs(args.out, exist_ok=True) + print("SHM %s" % args.shm) + + fd = os.open(args.shm, os.O_RDWR) + size = os.fstat(fd).st_size + mm = mmap.mmap(fd, size, mmap.MAP_SHARED, + mmap.PROT_READ | mmap.PROT_WRITE) + + deadline = time.time() + args.timeout + saved = 0 + last_seq = 0 + while saved < args.frames and time.time() < deadline: + mm.seek(FRAME_OFFSET) + magic, ver, w, h, stride, fmt, seq, _res, ts = HDR.unpack( + mm.read(HDR.size)) + if magic != MAGIC or seq == 0 or seq == last_seq or w == 0: + time.sleep(0.05) + continue + fb = mm.read(w * h * 4) + last_seq = seq + path = os.path.join(args.out, "frame-%04d.ppm" % seq) + with open(path, "wb") as f: + f.write(b"P6\n%d %d\n255\n" % (w, h)) + # RGBA -> RGB + rgb = bytearray(w * h * 3) + rgb[0::3] = fb[0::4] + rgb[1::3] = fb[1::4] + rgb[2::3] = fb[2::4] + f.write(bytes(rgb)) + saved += 1 + print("SAVED seq=%d %dx%d fmt=%d -> %s" % (seq, w, h, fmt, path)) + mm.close() + os.close(fd) + print("READER_DONE saved=%d last_seq=%d" % (saved, last_seq)) + return 0 if saved else 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/lg/frame_sequence.sh b/tools/lg/frame_sequence.sh new file mode 100755 index 0000000..958a34a --- /dev/null +++ b/tools/lg/frame_sequence.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# frame_sequence.sh - capture an animated GPU-rendered frame sequence end to +# end: run the guest's CUDA frame producer into the kvmfr ivshmem BAR and +# save every published frame on the host as PPM/PNG (the proven +# CUDA -> ivshmem -> host path; see docs/05-operations/06). +# +# Usage: tools/lg/frame_sequence.sh [frames] [width] [height] [fps] [outdir] +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +FRAMES="${1:-40}" +W="${2:-800}" +H="${3:-600}" +FPS="${4:-10}" +OUT="${5:-/tmp/opencode/frames-seq}" + +mkdir -p "$OUT" +rm -f "$OUT"/frame-*.ppm "$OUT"/frame-*.png + +# Start the host reader first so no published frame is missed. +python3 "$REPO/tools/lg/frame_reader.py" --frames "$FRAMES" --out "$OUT" \ + --timeout 240 > "$OUT/reader.log" 2>&1 & +READER=$! + +sleep 1 +"$REPO/scripts/ssh_guest.sh" \ + "sudo /tmp/frame_producer $FRAMES $W $H $FPS" 2>&1 | grep -v "Permanently added" \ + | tail -5 + +wait "$READER" +echo "reader rc=$?" +ls -1 "$OUT"/frame-*.ppm 2>/dev/null | wc -l diff --git a/tools/site/pages/Main-Page.wiki b/tools/site/pages/Main-Page.wiki index 9a80009..f91fc17 100644 --- a/tools/site/pages/Main-Page.wiki +++ b/tools/site/pages/Main-Page.wiki @@ -18,7 +18,7 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i * '''[[Documentation]]''' — the full index of this wiki. * '''[[FAQ]]''' — the questions we get asked most. -== Project status (2026-10-12) == +== Project status (2026-10-20) == {| class="wikitable" ! Area !! State |- @@ -30,9 +30,9 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i |- | CUDA userspace init and the driver-API smoke (ALL OK, 64-byte payload round-trips) || Working |- -| Guest CUDA kernel execution on real hardware || In progress — a forwarded launch reaches the engine queue; engine execution is the frontier +| Guest CUDA kernel execution on real hardware || Working — vector-add, matrix-multiply and transformer kernels run green on fresh boots; hardening and breadth in progress |- -| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet +| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet — multiple mediated devices (different emulated VRAM sizes) and multiple guests already coexist; per-guest execution isolation is in progress |} The full story, with evidence and the current wall: [[Project Status]] and [[The Execution Frontier]]. diff --git a/wiki/.publish-manifest.json b/wiki/.publish-manifest.json index 20e9722..4fe571b 100644 --- a/wiki/.publish-manifest.json +++ b/wiki/.publish-manifest.json @@ -19,11 +19,11 @@ }, "CUDA Userspace Driver": { "file": "CUDA-Userspace-Driver.wiki", - "new_revid": 23963, + "new_revid": 23998, "old_revid": 23963, - "source_sha": "31d77a3a672b2943", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:06Z", + "source_sha": "d467dbcd9f323768", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:03Z", "title": "CUDA Userspace Driver" }, "Community": { @@ -163,11 +163,11 @@ }, "Main Page": { "file": "Main-Page.wiki", - "new_revid": 23972, - "old_revid": 23960, - "source_sha": "1b53ef37afca940e", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:23Z", + "new_revid": 24000, + "old_revid": 23972, + "source_sha": "2e8171cab56c394c", + "summary": "Status table refresh 2026-10-20: guest CUDA kernel execution working", + "timestamp": "2026-10-20T09:19:09Z", "title": "Main Page" }, "MediaWiki:Sidebar": { @@ -199,20 +199,20 @@ }, "Physical GPU Backend and Guest Wiring": { "file": "Physical-GPU-Backend.wiki", - "new_revid": 23985, - "old_revid": 23951, - "source_sha": "4bff4dbfa8b4644e", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:59Z", + "new_revid": 23999, + "old_revid": 23985, + "source_sha": "d762abd50cef636d", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:05Z", "title": "Physical GPU Backend and Guest Wiring" }, "Project Status": { "file": "Project-Status.wiki", - "new_revid": 23986, - "old_revid": 23952, - "source_sha": "6d07f2a0c6f524a9", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:00Z", + "new_revid": 23997, + "old_revid": 23986, + "source_sha": "cb714adeebcfdc33", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:01Z", "title": "Project Status" }, "RPC Message Format": { @@ -262,11 +262,11 @@ }, "The Execution Frontier": { "file": "Execution-Frontier.wiki", - "new_revid": 23966, + "new_revid": 23996, "old_revid": 23966, - "source_sha": "b26ce9bb6fe63f6a", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:07Z", + "source_sha": "8c0007ec287387b9", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:00Z", "title": "The Execution Frontier" }, "The Guest Stack and CUDASW": { diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md index 9a041ef..6975a1d 100644 --- a/wiki/CUDA-Userspace-Driver.md +++ b/wiki/CUDA-Userspace-Driver.md @@ -1,16 +1,20 @@ # CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive -> **Status (2026-10-12, honest).** The analysis covers the whole library for +> **Status (2026-10-20, honest).** The analysis covers the whole library for > driver 570.144 and the tooling is version-agnostic. The *control plane* gets > the stock guest stack through `cuInit`, device enumeration, `cuCtxCreate` > and the full driver-API smoke test — allocation, copies, free and destroy > complete end to end (`ALL OK`; the 64-byte payload round-trips exactly). > The *data plane* decodes guest push buffers and executes the memory and > semaphore work it understands; CUDA kernel launches are forwarded to the -> physical execution path, where a launch reaches the engine queue -> (front-end deposit and scheduling are observed) but an engine-authored -> release is not yet observed. Green runs are not yet reproducible on every -> boot — an open issue, tracked in the repository. Canonical detail: +> physical execution path and now execute on the physical GPU — the +> engine-authored completions relay to the guest and its synchronisation +> waits resolve. The vector-add (`VADD n=32768 bad=0`), matrix-multiply +> (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, +> `FFN4 layers=4 bad=0`) clients verify green on fresh boots. The +> large-buffer staged-transfer class (multi-megabyte transfers between guest +> memory and device memory) is under repair, and reproducibility across fresh +> boots is being re-measured after the boot-poisoning fix. Canonical detail: > [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) > through `08`; reading order > [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); @@ -172,19 +176,19 @@ the data plane (`docs/08`). emulation and object/control replies are implemented; the stock guest stack completes `cuInit`, device enumeration, `cuCtxCreate` and the driver-API smoke test (allocation, copies, free and destroy). -* **Current frontier:** a guest CUDA kernel launch is forwarded and reaches - the physical engine queue — the front end deposits the launch and the - scheduling path accepts it — but the engine does not yet execute the - forwarded launch descriptor: an engine-authored release is the only proof - of execution and it is not observed. The per-boot green/stall variance on - the emulation path is an open issue, tracked in the repository. The chain, - the evidence and the live state are on +* **Current frontier:** guest CUDA kernel launches execute on the physical + GPU with engine-authored completions relayed to the guest; the verified + clients are green on fresh boots. The leading edge is the large-buffer + staged-transfer class (some large multi-chunk clients still mis-verify), + re-measurement of reproducibility across fresh boots after the + boot-poisoning fix, and per-guest execution isolation. The chain, the + evidence and the live state are on [`Execution-Frontier.md`](Execution-Frontier.md). * **Data plane today:** the host module decodes guest push buffers as method streams and executes the memory and semaphore work it understands (memsets, copies, semaphore releases), and forwards engine-class work to - the physical backend; kernel launches are forwarded but not yet executed - by the engine. + the physical backend; kernel launches now execute on the engine through + the forwarded path, with completions relayed back to the guest. ## 5. The crash-path lesson: replies are consumed data diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md index fe21fda..2d9d80c 100644 --- a/wiki/Execution-Frontier.md +++ b/wiki/Execution-Frontier.md @@ -1,15 +1,16 @@ # The Execution Frontier -> **What this page is.** The current leading edge of guest-facing execution: -> the chain that must fire for a stock guest CUDA kernel to run on the -> physical GPU, what has been proven about it, where it stops, how we know, -> and the open lines of investigation and theories. It is evidence-led: every -> statement carries a live observation, a decompilation finding, or a clearly -> labelled theory. Condensed from +> **What this page is.** The state of guest-facing execution: the chain that +> must fire for a stock guest CUDA kernel to run on the physical GPU, what has +> been proven about it, how we know, and where the leading edge is now. The +> chain fires end to end: verified guest CUDA clients execute on the physical +> GPU and wake up correctly; the leading edge is now breadth and hardening. It +> is evidence-led: every statement carries a live observation, a +> decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. -Status: 2026-10-12 (source page dated 2026-10-06), tree `master @ 2f066727`. +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. @@ -45,46 +46,62 @@ guest: builds launch + method stream | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed | | 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) | | 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised) | `[wall]` not observed | -| 7 | **Engine reads the QMD** | `[wall]` not observed | +| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed | +| 7 | **Engine reads the QMD** | `[live]` observed | | 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in | -| 9 | GR dispatches the grid to the SMs | never observed on this stack | -| 10 | Engine authors release + channel semaphore | absent for forwarded launches | -| 11 | Host relays completion; guest wakes | the guest waits and times out | +| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve | -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +**One-line summary:** the whole chain is live; the leading edge is now breadth +and hardening — the large-buffer staged-transfer class, reproducibility across +boots, and per-guest execution isolation. ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire the ring entry is consumed and the front-end -methods run — but the launch descriptor is never materialised, the engine -never reads a QMD, no engine-authored completion appears, and the guest blocks -in its synchronisation wait. - -This is **not** a context-validity wall (ruled out at dispatch) and **not** a -window-mapping wall for the descriptor destination (that install is accepted). -The wall is the **form of the launch method stream**: what the front-end -requires in order to install the inline descriptor, and what the engine -requires in order to read and dispatch it. +A stock, unmodified guest CUDA program executes its kernels on the physical +GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. Engine-authored completions relay to the guest and its +synchronisation waits resolve. + +The leading edge has moved from the launch chain itself — it fires — to +breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise and the first + CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report). + Re-measurement across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. ## 3. How we know (observables and their limits) | Observable | What it proves | What it does not prove | |---|---|---| -| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | nothing about the engine | -| Method execution on the ladder channel | the front-end can execute the method vocabulary | **not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten with a detectable sentinel | whether the descriptor was materialised after submission | a read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | the inline-install method did not run | it does not say why | -| Engine-authored release word absent; poll times out | the engine did not complete the launch | it does not say whether the engine started | +| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results | +| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches | +| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient | +| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress | | No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction | -| Channel instance context flag set | the channel is context-valid at dispatch | nothing about the descriptor path | -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +**Consequence:** acceptance is verified guest results plus the engine-authored +release; a green run counts as progress only when it reproduces across fresh +boots. ## 4. What is exonerated (do not relitigate) @@ -98,68 +115,64 @@ signals, which we have already learned to distinguish. already initialised in the same cycle; not the dispatch delta. * Parenting/TSG for the fetch — both variants fetch identically. * The copy-engine path — CE-class work, releases and fault relay are live - (default off); the GR path is a separate problem. + (default off); the GR path is separate. +* The descriptor-gate line of investigation (the old T1–T4 theories) — the + inline-descriptor install, the engine read, the dispatch, the + engine-authored release and the guest wake-up are all observed; closed by + observation. +* The old boot-poisoning defect — root-caused to the status-queue consumption + cursor and fixed. ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** The forwarded stream includes a verbatim copy of the guest's -prelude whose inline-data writes target addresses not present in the engine's -view of the channel's page tables, and the referenced data is not relocated. -The shorter, proven stream shape fires its tail (channel semaphore) but still -does not schedule the descriptor. +**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged +transfers between guest memory and device memory are under repair: some large +multi-chunk clients still mis-verify. -**Issue B — the descriptor-virtualization control method is absent.** The -method that arms the inline-descriptor install path is not present in the -forwarded stream. Fixing it should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. +**Issue B — reproducibility across boots (re-measurement).** The +boot-poisoning defect (wrong status-queue consumption cursor under the +driver's status-queue mode; the reply ring could desynchronise under load, the +first client would fail) was root-caused and fixed. Re-measurement across +fresh boots is in progress; no "every boot" claim is made until it completes. -**Issue C — diagnostic hygiene.** The sentinel placed at word 0 overlaps real -descriptor fields; it must be moved to an unused word so diagnostics do not -perturb what they measure. +**Issue C — per-guest execution isolation.** Multiple mediated devices with +different emulated VRAM sizes and multiple guests coexist at the device level; +per-guest execution isolation for simultaneous CUDA guests is in progress. ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline and **re-add the - virtualization control method** — one variable at a time, with the install - sentinel and the engine-read signal as the readout. -2. **Relocate or drop the verbatim prelude** so inline-data writes target - addresses present in the engine's view, or replace it with the host-side - staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference** — our own ladder dispatching a - descriptor built from the same source material, or a captured working - launch of the stock stack on the same GPU — converting the remaining work - from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware debugger before spending a supervised cycle - ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)). - -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the control method enables the - install path. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Its refused writes leave - the launch path unable to schedule the descriptor. -* **T4 — relocation is the missing transform.** The guest's descriptor is - valid for the guest's view of memory; the forwarding path must relocate - referenced data into the channel's view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +1. **Repair the large-buffer staged-transfer class** — the three green clients + stay the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility. +3. **Per-guest execution isolation** — build from the device-level coexistence + to isolation for simultaneous CUDA guests. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel; each entry is accepted only on its own verification + output (`bad=0` + `STATUS OK`). + +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed, so +the launch chain they questioned is proven end to end. The remaining work is +engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest +isolation), each with an acceptance signal named before the work. ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end; the guest's own verification (`bad=0` + `STATUS OK`) + is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred. * **Emulator verdicts are instruction/memory-level.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** * **A wall is closed only by an acceptance signal**, named before the experiment. diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md index 1b83203..bbca084 100644 --- a/wiki/Physical-GPU-Backend.md +++ b/wiki/Physical-GPU-Backend.md @@ -5,7 +5,7 @@ > guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) > and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partially built; `[wall]` current blocker; `[target]` planned end state. @@ -64,13 +64,15 @@ independently verified: 5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]` 6. **Work path** — decoded guest pushes become real push streams; resolved host pages - map through module-built scratch page tables; the firmware schedules the work. `[gated]` + map through module-built scratch page tables; the firmware schedules the work. + `[live]` observed end to end for the verified guest kernel clients. 7. **Completion relay** — hardware-completed semaphores land in guest-polled memory; - events and faults relay through the existing MSI-X injection path. `[gated]` + events and faults relay through the existing MSI-X injection path. + `[live]` engine-authored completions relay to the guest and its waits resolve. ## 5. The two planes and the wiring -The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software. +The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software. ```text GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU @@ -85,8 +87,9 @@ INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live] RM battery, UVM registration [live] WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell host pump decodes; eligible CE copies forwarded via module-built scratch - page tables; GR: launch reaches the front-end, descriptor not seen [wall] -DONE engine authors the release in guest memory; host relays via MSI-X [gated] + page tables; GR: launches execute on the engine, completions relay + to the guest [live] +DONE engine authors the release in guest memory; host relays via MSI-X [live] TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown -> protected region down -> distribution driver rebound [live] FAIL a failed mutating operation parks the device in a locked fail-safe; @@ -104,17 +107,20 @@ FAIL a failed mutating operation parks the device in a locked fail-safe; | Power domain held active across the ownership window | verified across supervised cycles `[live]` | | Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | | Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | -| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` | -| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` | +| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` | +| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest | ## 7. The execution frontier The RM client ladder reaches the compute engine (a GR0 channel plus a -compute-class object). A forwarded guest CUDA launch now -reaches the front-end (ring entry fetched, methods executed) but the inline -launch-descriptor install has not been observed and no grid has dispatched; -context validity at dispatch is ruled out. The wall is the form of the -forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md). +compute-class object). Forwarded guest CUDA launches now execute on the +engine: the inline launch descriptor installs and is read, the grid +dispatches, the engine authors the completion, the host relays it, and the +guest's synchronisation wait resolves — verified on fresh boots with the +vector-add, matrix-multiply and transformer clients. The leading edge is the +large-buffer staged-transfer class (under repair), reproducibility across +fresh boots, and per-guest execution isolation — see +[`Execution-Frontier.md`](Execution-Frontier.md). A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step. @@ -147,6 +153,9 @@ fail-safe that is never auto-retried; recovery is a supervised step. access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). +* Multiple mediated devices with different emulated VRAM sizes and multiple + guests already coexist at the device level; per-guest execution isolation + for simultaneous CUDA guests is in progress. ## 10. Open questions @@ -159,7 +168,7 @@ fail-safe that is never auto-retried; recovery is a supervised step. * [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up. * [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design. -* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane. +* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane. * [`Host-Module.md`](Host-Module.md) — the module that owns both planes. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings. * [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture. diff --git a/wiki/Project-Status.md b/wiki/Project-Status.md index d6e60d1..e63a198 100644 --- a/wiki/Project-Status.md +++ b/wiki/Project-Status.md @@ -10,7 +10,7 @@ > (the execution frontier). When this page and the docs disagree, the docs tree > wins. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or behind supervised gates; `[wip]` partially implemented; `[wall]` current blocker; `[target]` named success state; `[planned]` designed, not built. @@ -24,9 +24,9 @@ named success state; `[planned]` designed, not built. | GSP-RM message queues and RPC transport | `[live]` working | | Stock guest driver probe and `nvidia-smi` (TU104) | `[live]` working | | CUDA userspace init and the driver-API smoke (`cuInit`, device enumeration, `cuCtxCreate`, allocation, copies, free, destroy) | `[live]` working — the smoke prints `ALL OK` and the 64-byte payload round-trips exactly | -| Guest CUDA kernel execution on real hardware | `[wall]` a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue | -| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above | -| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` designed; the baseline service-state ownership work is in progress | +| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. Remaining: the large-buffer staged-transfer class is under repair (some large multi-chunk clients still mis-verify); fresh-boot re-measurement is in progress after the boot-poisoning fix; per-guest execution isolation for simultaneous CUDA guests is in progress | +| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend | +| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | | OpenGL / Vulkan (3D), video | `[planned]` | | Windows guests | `[planned]` (target) | | Xen / Argo / OpenXT / Qubes Air transports | `[planned]` (the emulation core is hypervisor-independent by construction) | @@ -43,23 +43,32 @@ The project's definition of done for CUDA requires all of the following 2. Context creation and the full driver-API surface (`cuMemAlloc`, copies, free, destroy) — done. 3. A guest kernel launch that **executes on real hardware** and is proven by an - engine-authored completion — the current frontier. + engine-authored completion — demonstrated: the verified clients execute on + the physical engine and the engine-authored completions relay to the guest. 4. Memory the guest's work refers to resolving correctly end to end (the VA↔PA model) — partially, for the paths exercised so far. -5. Correct completion, semaphore and fault semantics back to the guest — CE - class work has the relay; GR is open. -6. Reproducibility: every boot, not a lucky one — an open issue. -7. No ad-hoc default-off parameters required for the canonical path. +5. Correct completion, semaphore and fault semantics back to the guest — the + GR completion path now relays (engine-authored releases wake the guest's + synchronisation waits); fault semantics remain partial. +6. Reproducibility: every boot, not a lucky one — root-caused and fixed (the + boot-poisoning defect: the wrong status-queue consumption cursor could + desynchronise the reply ring under load and poison the boot); + re-measurement across fresh boots is in progress. +7. No ad-hoc default-off parameters required for the canonical path — still + open. -Until every gate holds, the honest formulation is: **initialization and the -driver-API path work; engine execution is in progress.** +Until every gate holds, the honest formulation is: **initialization, the +driver-API path and kernel execution on the physical GPU work and verify; +hardening (large-buffer transfers, boot-to-boot reproducibility) and +per-guest isolation remain.** ## 3. How to see it yourself The reference setup takes about half an hour from a fresh host ([`Install-and-First-Run.md`](Install-and-First-Run.md)); the current expected outcome is exactly the ladder above — driver probe, `nvidia-smi`, the full -driver-API smoke, and the execution frontier where it stands. The +driver-API smoke, and guest CUDA kernels executing and verifying on the +physical GPU. The risc-v-gsp-rm-debugger workbench ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) is the host-side tooling used to study the firmware side of that frontier. @@ -68,7 +77,7 @@ tooling used to study the firmware side of that frontier. Every claim above carries a date and a tag, maps to a maintained document in the repository, and is refreshed as milestones land. The execution story, with -its evidence and its open theories, is on +its evidence, its resolved theories and its open work, is on [`Execution-Frontier.md`](Execution-Frontier.md). ## 5. Related pages |