diff options
80 files changed, 23412 insertions, 1199 deletions
@@ -153,3 +153,15 @@ userspace/sdks/ # Recorded debugger sessions (bridge checks, probe dumps) - never track /dumps/ + +# Editor/backup leftovers (never track). +*.bak +*.bak-* +*.orig +*.rej + +# Demo/release videos (task: "0.9.9 release preview" lives in the repo root, +# never tracked - large binary; the file is produced locally from the host +# KVMFR capture). +/*.mp4 +/open-iov-*.mp4 diff --git a/BIGPATH-REPORT.md b/BIGPATH-REPORT.md new file mode 100644 index 0000000..75b0273 --- /dev/null +++ b/BIGPATH-REPORT.md @@ -0,0 +1,197 @@ +# BIGPATH-REPORT — B-2 grown data-bounce region (offline implementation) + +* Branch: `bigpath`, based on `ecfb1278` (m155). Worktree: + `/home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath`. +* Scope: implement `AGENT-chunked-path-design.md` §3 (B-2) — make GR launches + with arbitrarily large operands (vector_add 3×4 MB, victory_math 320 pages, + memcpy_roundtrip 8 MB) execute correctly by replacing the 205-page / + 68-pages-per-pointer data window with a boot-planted 16384-page (64 MiB) + region, with fail-closed cap semantics. +* Constraints honoured: offline only — no module load/unload, no QEMU/mdev, + no guest, no dmesg writes, no reboot. Build: `kernel && make -j8`. + Commit local on `bigpath` only; no remote push. + +## 1. Build result + +Clean build `make -j8` from a `make clean` state: **exit 0**. +Warning delta vs the HEAD baseline (same toolchain): + +| Baseline (ecfb1278) | After B-2 | +|---|---| +| `gsp_phys_booter.c` frame 1120 B > 1024 (`gsp_phys_backend_guest_gr_submit`) | frame **1104 B** (same pre-existing warning, shrunk by the static scratch) | +| `gsp_sema_service.c:12518` unused variable `n_win` | removed (fixed in passing) | +| `main.c:402` frame 1304 B; `%x` vs `unsigned long` in `gsp_phys_rmcall_stage3` | unchanged (pre-existing) | + +No new warnings; the touched lines are `-Wextra` clean. + +## 2. Site → change table + +| # | Spec item | Site (post-change) | Change | +|---|---|---|---| +| 1 | Constants | `kernel/gsp_phys_booter.h:366-369` | `GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0` 0x0A000000, `_SLOT0` 80, `_PT_PAGES` 32, `_PAGES` 32*512=16384. Shared header constant: the SVC derives its reloc base and cap from here (no drift, design R5c). | +| 2 | Boot chunk fields | `kernel/gsp_phys_booter.c:2325-2330` | `struct gsp_phys_boot_chunk rmcall_gr_data_bounce;` (16384 pages) + `rmcall_gr_data_bounce_pt` (32 PT pages) added to `boot_stage`. | +| 3 | Stage-3 plant | `kernel/gsp_phys_booter.c:9770-9840` | One-shot (`!pages`) plant next to the 208-page bounce plant: allocate the data chunk (`gsp_phys_chunk_alloc`, one call, contiguous + per-page fallback) and the PT chunk, zero both, write PD0 dual entries 80..111 (same shape as ext pool 32..79: 0 at `slot*16`, `gsp_phys_rmcall_pde(dma)` at `slot*16+8`), then for every page i write `gsp_phys_rmcall_pte(page_dma(i))` at `PT[i/512] + (i%512)*8`; `dma_wmb()`; log `data-bounce region planted: …`. All at stage 3, before any engine fetch. Gated by `gsp_phys_gr_data_region`. | +| 4 | Old window untouched | `gsp_phys_booter.c:9727-9769` (unchanged), mail `9756`/marker `9765`, mail injection mva `17492-17493` | The 208-page window, mail slot 206, marker 207 and the bank-4 mail injection stay byte-identical; only its comment records the B-2 decommissioning as a data carrier. | +| 5 | `bi` helper | `gsp_phys_booter.c:16933-16953` (`gsp_phys_gr_data_chunk`), `16961-16982` (`gsp_phys_gr_data_bi`) | One helper returns `(reloc_va − VA0)>>12` bounded `< 16384` for the grown region (`GSP_PHYS_GR_DATA_BI_NONE` outside); with the parameter off it serves the legacy window bounded by `MAIL_SLOT`. A chunk selector names the active data chunk. | +| 6 | Five sites replaced | retire wait `:17025`, fill_bounce `:17091`, copy-in `:17972`, copy-out `:18653`, window diag `:18698` | All five inline `(reloc_va − 0x110000)>>12` + `MAIL_SLOT` bounds replaced by the helper + chunk selector. No other copy-path semantics changed. | +| 7 | Checksum/scratch arrays | `bounce_csum_in` `:17156` → 16384 entries (static); `checked[]` retire wait `:17015` → static 16384 entries | 64 KB each, off the kernel stack (design risk #4). Single-inflight caller documented in place. | +| 8 | SVC reloc base | `gsp_sema_service.c:12371-12372`, used `:12768` | `SVC_GR_DATA_RELOC_BASE` = shared grown constant; `_OLD` 0x110000 for the parameter-off path. | +| 9 | SVC per-ptr cap | `gsp_sema_service.c:12381-12382`, used `:12703` | 4096 (runaway guard); old 68 kept for legacy. Stop-on-miss still bounds real spans. | +| 10 | SVC total cap, fail-closed | `gsp_sema_service.c:12391-12392`, check `:12749-12767` | Cap = `GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES` (16384); on hit: `svc_gr_data_stalled = true` + `pr_err(… CAP-HIT: region full, launch held fail-closed …)` + return — no silent truncation, the existing bounded hold/retry path takes over. Old 205 kept for legacy. | +| 11 | SVC array headroom | `gsp_sema_service.c:3931` | `SVC_GR_DATA_MAX` 16384 → 32768 (gr_data[] and the backend hand-off array). | +| 12 | R3 hygiene | `gsp_sema_service.c:12671-12700` | Skip the launch's own param VA and the qmd_word 32..47 gr_pages (bank/param) as data candidates, next to the existing QMD-VA skip. Critical for B-2: without it the param-page candidate would consume up to a 4096-page span of resolvable guest memory. | +| 13 | Static asserts | `gsp_phys_booter.c:2365-2410` | Build-time disjointness/geometry contract: 32 PT pages × 512 = 16384 pages; VA0 == slot 80 × 2 MB; root/PD2/PD1 index 0 and PD0 80..111 ≤ 128; end ≤ VAS vaSize 0x10000000; region above ext pool end 0x0A000000; region disjoint from old bounce 0x110000..0x1dffff, win0 0x1e0000.., guest scratch 0x200000... | +| 14 | Module param | `kernel/gsp_phys_gpu.c:1080-1097`, decl `kernel/gsp_phys_gpu.h:357` | `gsp_phys_gr_data_region` (int, 0644, default 1). Read by the stage-3 plant, the SVC fill and the copy path. Boot-fixed (set at insmod). | +| 15 | Teardown | `gsp_phys_booter.c:19611-19613` | Free the two new chunks in `gsp_phys_rmcall_fini` (and, as a one-line in-passing fix, the old 208-page bounce chunk that was leaked there). | + +## 3. Deviations from the design + +1. **Shared header constants instead of per-file duplicates.** The four + region constants live in `gsp_phys_booter.h` and `gsp_sema_service.c` + derives `SVC_GR_DATA_RELOC_BASE`/`SVC_GR_DATA_BOUNCE_MAX` from them. The + design (§3.2, R5c) explicitly wanted the relocation base "comment-bound" + or shared to prevent drift; a shared constant is the stronger form. +2. **`bi` helper covers both modes.** The design's helper description assumes + the grown region only. Because the module parameter must flip the whole + data path back (spec item 7), the helper also serves the legacy window + (bounded by `MAIL_SLOT`), and a small `gsp_phys_gr_data_chunk()` selector + picks the chunk for the copy path. Behaviour with the parameter on is + exactly the design's. +3. **`checked[]` made static, not a bitmap.** The design allowed either; the + static array mirrors the existing single-inflight static scratch pattern + and keeps the wait logic byte-identical. The dedup loop remains O(n²) as + at HEAD; worst case (16384 records) is ~134M u32 compares once per + launch — accepted for bring-up (see residual risks). +4. **Old bounce chunk now freed in `rmcall_fini`.** Pre-existing leak spotted + next to the new frees; freeing it is safe (`gsp_sema_service_work_quiesce` + runs before `gsp_phys_boot_staging_free`). Reported here because it is a + teardown behaviour change beyond the B-2 allocation lifecycle. +5. **Memory-leak observation not fixed:** `rmcall_gr_shadow_win0` and + `rmcall_gr_mail` are also never freed in `rmcall_fini` (pre-existing, out + of scope). Likewise `rmcall_ctx_pt[]` last slot can collide with the ext + pool PD0[0] if the server MAIN context ever reaches the 32 MB maximum + (pre-existing latent; the live MAIN is ~1.2 MB). Neither is touched here. +6. **No F1-style permanent refusal on a persistent cap hit.** Per the spec, + the cap hit marks the launch stalled for the existing bounded hold. Note + honestly: that hold is one-shot per GPFIFO entry; if the region is + genuinely too small (negative control, > 16384 unique operand pages), the + entry can be consumed after the hold window expires, exactly like every + other stalled launch at this revision. The CAP-HIT `pr_err` makes it + visible, but the permanent fail-closed guarantee is `AGENT-bigcopy` + F1/§4 territory and was explicitly kept out of this change (design risk + #6). + +## 4. Risks + +**Design risk #1 — engine walker reach for PD0 80..111 (LOW).** The plant +shape is byte-for-byte the ext pool pattern (PD0 dual entry small half → +fresh 4 KB PT; leaves = `pte(dma)`), which is proven engine-readable at PD0 +32..79; entries 80..111 sit in the same PD0 page and same root/PD2/PD1 chain +(indices all zero, asserted at compile time). The only unproven ingredient is +the slot range itself. Fallback if the first big launch faults/times out with +`changed=0` and the data chunk otherwise healthy: move the region to the +reserved win1 data windows (`GSP_PHYS_RMCALL_VA_GR_DATA_BASE` 0x100000000, +prewin install) — the copy-path changes carry over unchanged. + +**Design risk #3 — 64 MB contiguous `dma_alloc_coherent` fails → per-page +fallback (MEDIUM).** `gsp_phys_chunk_alloc` warns and falls back to 16384 +per-page coherent allocations; boot-time stage-3 cost grows (seconds) and +memory is fragmented across pages (harmless to correctness: each PTE uses the +page's own DMA). Watch the boot log for `contiguous 16384-page coherent +allocation failed` and stage-3 wall time. Mitigation if needed: split the +chunk into a few smaller contiguous allocations — deferred until measured. + +**Other risks carried from the design:** +* *Per-pointer span overshoot.* 4096 pages is a runaway guard; stop-on-miss + still bounds a real operand. If a client's operands are laid out + contiguously with resolvable neighbours, one candidate can consume up to + 4096 pages and the cap can be reached before all operands are in. The + fail-closed stall + CAP-HIT log would expose this on the first big cycle; + the cap value (design open question #4) is the tuning knob. +* *Copy cost.* Up to 64 MB in + 64 MB out per launch (tens of ms), plus up to + 300 × 5 ms retire-wait polls over 16384 pages when nothing changes — all + bounded by the existing wait exits; correctness first (design risk #8). +* *Param flip consistency.* `gsp_phys_gr_data_region` is boot-fixed; a + runtime flip after the stage-3 plant would desync SVC vs plant. Documented + in the param description; the default cannot desync. + +## 5. Grep audit — remaining uses of the old bounce VA0 / 0x110000 + +`grep -rn "RMCALL_GR_BOUNCE_VA0|SVC_GR_DATA_RELOC_BASE_OLD|0x110000" +kernel/*.c kernel/*.h` (excluding unrelated hits `bar_emulation.c:62` LTC, +`gsp_emulation.h:40` firmware block): + +| Site | What it is | Indexes data pages? | +|---|---|---| +| `gsp_phys_booter.c:2338` | the legacy constant definition | no | +| `:2349-2351` | win0-vs-bounce overlap `static_assert` (layout) | no | +| `:2399` | B-2 disjointness assert | no | +| `:9744-9745` | old bounce plant `pr_info` | no | +| `:1011, 2306, 2320, 2371, 16952` | comments/history | no | +| `:16975, 16977` | `gsp_phys_gr_data_bi` legacy branch — reachable only when `gsp_phys_gr_data_region=0`, where operand records really are in that window; bound `MAIL_SLOT` excludes mail/marker | only in legacy mode, by design | +| `:17492-17493` | mail-injection `mva = VA0 + (MAIL_SLOT<<12)` — the mail message page, never a data record | no | +| `gsp_sema_service.c:12372` | the legacy reloc base constant (param-off path) | only in legacy mode | +| `gsp_sema_service.c:12367, 12478, 12646` | comments | no | + +The five data-page index derivations all now go through +`gsp_phys_gr_data_bi()`; with the grown region on (default), no code path can +index a data page through the old window. The mail injection and the old +window's own plant/diagnostics are the only remaining consumers, exactly as +the design requires. + +## 6. Sanity math (checked by computation) + +* `32 × 512 × 4096 = 67 108 864 B = 64 MiB` ✔ +* VA range `0x0A000000 .. 0x0DFFFFFF` (end-exclusive `0x0E000000`) `< + 0x10000000` (VAS vaSize) with 32 MB slack ✔ +* PD0 slot = `0x0A000000 >> 21` = 80; slots 80..111 ≤ 128 (non-shared VAS + ceiling) ✔ +* Disjointness: ext pool `[0x04000000, 0x0A000000)` meets the region exactly + at VA0; old bounce ends `0x1E0000`, win0 ends `0x1F0000`, guest scratch + ends `0x2200000` — all far below 0x0A000000 ✔ (all compile-time asserted) + +## 7. Live validation commands for the lead (supervised; NOT run here) + +```sh +# 1. Bring the branch build into the campaign tree (lead decides merge vs +# manual copy of vgsp.ko) and confirm the param exists: +git -C /home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath log --oneline -1 +cd /home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath/kernel && make -j8 +modinfo vgsp.ko | grep gsp_phys_gr_data_region + +# 2. One full supervised cycle with the B-2 default (grown region ON; +# module args unchanged, gsp_phys_gr_data_region defaults to 1): +cd /home/rmcore/Git/integdev_gpu_drv/open-iov +./backups/evidence-20261002-victory/victory_grfwd_cycle.sh b2-r1 + +# 3. Boot gate (host dmesg) — expect exactly one plant line, no alloc error: +# vgsp-boot: rm-call: data-bounce region planted: 16384 pages at VA +# 0xa000000..0xdffffff (PD0 slots 80..111, 32 PT pages, before first fetch) +# and no "data-bounce region allocation failed" / no +# "contiguous 16384-page coherent allocation failed" (or accept the +# fallback warning and time stage 3). + +# 4. Guest clients: push the battery and run the three big clients plus the +# regression set (the cycle already runs the runner; for manual runs): +./tools/cuda-ab/push_battery.sh +# in the guest: run /tmp/victory_math /tmp/vector_add /tmp/memcpy_roundtrip +# plus regressions (victory_matmul smiley mall victory_mail victory_saxpy +# victory_strided victory_reduce victory_atomics victory_memset2d +# victory_streams_events vecadd_loop). +# Expected host log: "svc-gr data fill: … pages resolved (reloc 0xa000000+)" +# with NO "CAP-HIT"; copy-out "changed by engine" > 0 naming output pages; +# guest: MATH badf=0 badd=0 badi=0 / RESULT bad=0 / ROUNDTRIP bad=0 csum=269221888. + +# 5. Rollback / A-B trial (legacy 205-page window, one flip at insmod): +./backups/evidence-20261002-victory/victory_grfwd_cycle.sh b2-legacy-r1 gsp_phys_gr_data_region=0 + +# 6. Negative control (over-region, > 16384 unique operand pages, e.g. +# vector_add n=2^23): expect the CAP-HIT pr_err + hold, no silent success. +``` + +## 8. Rollback parameter + +`gsp_phys_gr_data_region=0` at insmod restores the legacy 205-page / +68-pages-per-pointer path (reloc base 0x110000): the stage-3 plant is skipped, +the SVC keeps the old caps, and the copy path indexes the old bounce chunk. +Boot-fixed; do not flip at runtime. diff --git a/DUAL-REPORT.md b/DUAL-REPORT.md new file mode 100644 index 0000000..334a772 --- /dev/null +++ b/DUAL-REPORT.md @@ -0,0 +1,360 @@ +# DUAL-REPORT.md — Stage B+C (minimal) per-mdev separation for dual-guest CUDA + +**Date:** 2026-10-15 +**Worktree:** `/home/rmcore/Git/integdev_gpu_drv/open-iov-dual` (branch `dual`, base `ecfb1278`) +**Commits:** `81c11ca0` (stage B+C separation), `5769d5c4` (CUDA-bridge owner gating) +**Build:** `cd kernel && make -j8` — clean success, `vgsp.ko` linked. No module +was loaded, no device touched, no guest run (offline implementation only). + +Implementation of stages B+C (minimal) of +`analysis-2026-10-15/AGENT-dual-guest-inventory.md` §3.1/§3.2 (items G1–G8), +against the user scope: G1, G2, G3, G5-minimal, G6, G7, G8(a). +Explicitly **not** implemented: G4 remainder (per-guest learned tables), +G10 per-mdev debugfs views, parallel forwarding, boot_stage split. + +--- + +## 1. Per-item status table + +| # | Item | Status | Change site (file:line @ `5769d5c4`) | Verified by | +|---|------|--------|--------------------------------------|-------------| +| G1 | Sema-service thread services ALL open mdevs each pass | DONE | `kernel/gsp_sema_service.c:16034` (`svc_thread`): collects every open mdev into `vgpus[]` (:16060–:16075) and runs the pass body per guest; `svc_handle_map_add()` drains once per pass; the scans/hint loop is per guest (:16174–:16226). Per-guest rebuild freshness (`sess->last_build`, :16105–:16119). | build only (`make -j8`) | +| G2 | Channel table keyed by `(vgpu, h_object)`; per-guest rank; `vgpu` field in `svc_channel` | DONE | struct field `kernel/gsp_sema_service.c:165`; match in `channel_add` :1154 + `same` :1168; rank loops :1204–:1221; `channel_del` :1328; pump snapshot :13653; pump writeback :15201; counter paths :5198/:5271/:5360/:5551/:5600/:5626/:5686; `svc_pb_discover` :7821; `svc_gr_content_discover` :9103; `svc_gr_ring_autopin` :9445; chan dump :15516. Registration carries the owner: `kernel/gsp_rm_emulation.c:2360`; free passes it: :3248; desc field `kernel/gsp_sema_service.h:30`. | build only | +| G3 | Per-vgpu session active/flush flags; departing guest never flushes a live guest | DONE (staged semantics — see §3.1) | session records + helpers `kernel/gsp_sema_service.c:468–:520` (struct :473, `svc_guest_get` :489); `channel_add` marks active :1242; `channel_del` deactivates per guest and requests the flush only when **no** guest has a live channel :1356–:1378; `gsp_sema_service_vgpu_cleanup` :16364 (session end on mdev remove). | build only | +| G4 (remainder) | Per-guest learned tables (map/pin/vas/pending/…) | NOT IN SCOPE (documented risk §5.1) | — | — | +| G5 (minimal) | Forward request/hold carry the owning vgpu; single global in-flight kept | DONE | req already carried `vgpu` (verified, set at :12972 GR `svc_gr_req_fill`, :13304 CE); hold abort keyed `(vgpu, handle)` :11091; dead-owner guards on both resolution paths :11928 (`svc_vgpu_open`), :11956 (`gone_finish`), :12019 (`stop_flush`); operand translation through the request's vgpu :10922. Follow-on: CUDA-bridge fallback queue now carries its owner (`kernel/gsp_cuda_bridge.c:143`, :624, :985, :1353). | build only | +| G6 | Per-mdev guest-RAM backing {file, path, ranges} + per-mdev selector | DONE | records `kernel/gsp_phys_gpu.c:1264–:1373`; `gsp_phys_guest_ram_configure_vgpu` :1494; `gsp_phys_guest_ram_vgpu_clear` :1586; clear-all :1564; active :1604; state (:1646, legacy format byte-identical); debugfs writer `kernel/gsp_phys_booter.c:20487` (4-token legacy or 5-token `<uuid>` selector); selector resolve :20542 via `gsp_phys_open_mdev_by_name` (`gsp_phys_gpu.c:1356`). | build only | +| G7 | `gpa_to_hpa` takes the requesting vgpu; all call sites plumbed | DONE | `gsp_phys_gpa_to_hpa_vgpu` `kernel/gsp_phys_gpu.c:1737`; wrapper `gsp_phys_gpa_to_hpa` :1827. Call sites: `gsp_sema_service.c:10922` (forwarding operands — passes the request's vgpu); `gsp_phys_booter.c:9738`, :19125, :19212 (single-guest guest-GPA pin — explicit `NULL` = first open mdev, see §3.2); no others remain (`grep`). | build only | +| G8(a) | Guest-GPA pin explicitly single-guest; refusal + log | DONE | `kernel/gsp_phys_booter.c:20430–:20436`: pin refused `-EBUSY` with a log when `gsp_phys_open_mdev_count() > 1` (`gsp_phys_gpu.c:1333`); existing different-GPA refusal kept. Boot-ladder trigger call sites documented as first-open. | build only | +| G10 | Per-mdev debugfs views | NOT IN SCOPE | — | — | + +Diff stats (`git diff --stat ecfb1278..HEAD`): + +``` + kernel/gsp_cuda_bridge.c | 92 ++++++- + kernel/gsp_phys_booter.c | 115 +++++++-- + kernel/gsp_phys_gpu.c | 432 ++++++++++++++++++++++++++----- + kernel/gsp_phys_gpu.h | 35 ++- + kernel/gsp_rm_emulation.c | 3 +- + kernel/gsp_sema_service.c | 644 +++++++++++++++++++++++++++++++++----------- + kernel/gsp_sema_service.h | 20 +- + kernel/main.c | 10 + + 8 files changed, 1098 insertions(+), 253 deletions(-) +``` + +### 1.1 One-mdev equivalence argument (explicit fast paths) + +With exactly ONE open mdev every changed path degenerates to the old one: + +1. **Thread (G1):** `vgpus[]` has one entry; the per-guest loop body is the + historical pass in the historical order (probe → walk_probe → chan_dump → + scrub → rebuild → pump → bridge → pending → sweep, then the scan loop). + `svc_handle_map_add()` runs once per pass (`gsp_sema_service.c:16085`), + exactly as before. +2. **Channels (G2):** `vgpu` is constant, so `(vgpu, h_object)` matches the + same slots as `h_object` alone; rank loops count the same set; the pump + snapshot/writeback select the same channels. +3. **Session/flush (G3):** the single guest's last-channel close empties the + global table → `svc_sess_active = 0; svc_sess_flush = 1` at the same + transition as before. +4. **Guest RAM (G6/G7):** a 4-token `gsp-phys-guest-ram` write configures the + legacy record with byte-identical log lines; `gpa_to_hpa_vgpu(vgpu…)` + resolves the same first-open mdev and the same legacy record; the + `gsp-phys-state` `guest_ram:` field renders the byte-identical + `active=1 path=… ranges=… gpa=…` text (and `active=0` when unconfigured). +5. **Forwarding (G5):** `svc_fwd_req`/`svc_fwd_hold` set and read the same + vgpu the single pump always used; the dead-owner guards never trigger while + the guest is open; the CUDA bridge's `head_owner == vgpu` is always true. +6. **Log lines relied on by the campaign** are unchanged: `sema-svc:`, + `vgsp-gr:`, `GR LAUNCH submitted`, `GUEST GR LAUNCH OK`, `REAL GR SEM OK`, + `backend start: joined the live module-owned claim (READY)`, `guest RAM + configured:` / `guest RAM released:`. Debugfs file names are unchanged + (the selector is an extra token on the same file). Parameter defaults are + untouched. + +--- + +## 2. What changed, by item + +### G6 — per-mdev guest-RAM backing (`gsp_phys_gpu.c`, `gsp_phys_booter.c`) + +* `guest_ram_file`/`guest_ram_path`/`guest_ram_ranges` globals replaced by + `guest_ram_legacy` (slot for the historical top-level write) + up to 4 + `guest_ram_mdevs[]` records (vgpu-pointer keyed), all under the existing + `guest_ram_lock`. +* New API: `gsp_phys_guest_ram_configure_vgpu()`, + `gsp_phys_guest_ram_vgpu_clear()`, `gsp_phys_open_mdev_count()`, + `gsp_phys_open_mdev_by_name()` (exports + prototypes in `gsp_phys_gpu.h`). +* debugfs `gsp-phys-guest-ram` accepts both + `"<path> <gpa_base> <off> <size>"` (legacy: first/only mdev) and + `"<mdev-uuid> <path> <gpa_base> <off> <size>"` (per-mdev). Unknown selector + → `-ENODEV` with a log; `"-"` clears every record. Token count (4 vs 5) + discriminates; paths/UUIDs contain no spaces. +* mdev removal releases the record (`main.c:1750` + `gsp_phys_guest_ram_vgpu_clear(vgpu)`); physical release clears all records + (`gsp_phys_guest_ram_clear()`). + +### G7 — vgpu-aware GPA→HPA (`gsp_phys_gpu.c` + call sites) + +* `gsp_phys_gpa_to_hpa_vgpu(vgpu, gpa, *hpa)`: + * `vgpu == NULL` → first open mdev (historical); + * named vgpu whose session is closed → `-ENODEV` (never retargeted); + * translation uses the vgpu's own record, else the legacy record when the + vgpu is the first open mdev. +* `gsp_phys_gpa_to_hpa()` kept as a thin `NULL` wrapper. +* Forwarding operands (`svc_fwd_operand_hpa`) pass the request's vgpu; the + boot-ladder guest-GPA pin paths pass `NULL` on purpose (§3.2). + +### G1 — sema-service thread (`gsp_sema_service.c`) + +* Per pass: bounded list walk collects all open mdevs (≤ `SVC_MAX_GUESTS`=8, + the driver's per-type ceiling) and runs the full per-guest body for each. +* Per-guest `svc_rebuild` freshness so the first guest's 10 s tick cannot + starve the second guest of rebuilds. +* `pumped` is an aggregate: while any guest is progressing, the pass loops + tightly and skips the heavy scans, as before. + +### G2 — channel identity (`gsp_sema_service.c`, `gsp_rm_emulation.c`) + +* `svc_channel.vgpu`; add/del match `(vgpu, h_object)`; `same` (re-register) + requires the same owner; ranks count only same-owner channels; counter, + scrubber-adjacent, opdef-drop, push-buffer discovery, GR content discovery, + GR ring autopin max-CE scan, pump snapshot and pump write-back are all + filtered by the owning vgpu. +* Scrubber channel state (`svc_scrub` → `svc_scrubs[]`) made per guest: the + authored finish/USERD words are written through the owning guest's address + space, never through whichever guest opened first. + +### G3 — sessions (`gsp_sema_service.c`) + +* `svc_guests[]` records: `active` + `last_build`, pointer-keyed, freed on + mdev cleanup. `channel_add` marks active; `channel_del` deactivates the + owner when its last channel closes and requests the global flush only when + **no guest** has a live channel (staged semantics, §3.1). +* New `gsp_sema_service_vgpu_cleanup(vgpu)` (called from + `main.c:1749`): quiesces forwarding, drops the guest's hold (pure clear — + never a resolution through a vanishing device), removes its channels, + drops its opdef release slots, frees its session and scrubber records. + +### G5 (minimal) — forwarding identity (`gsp_sema_service.c`, `gsp_cuda_bridge.c`) + +* The single-inflight request keeps ONE global slot (serialization kept) but + carries its owner: CE requests set `svc_fwd_req.vgpu` (:13304), GR requests + set it in `svc_gr_req_fill` (:12972), and the operand resolver translates + through it. +* Hold abort keyed by `(vgpu, handle)`; both hold resolution paths + (`svc_fwd_hold_gone_finish`, `svc_fwd_hold_stop_flush`) and the pump's + cross-vgpu resolution check `svc_vgpu_open()` before ever dereferencing the + stored device and drop the hold when its owner is gone. +* CUDA bridge (the GR fallback under the single-inflight slot): request and + pending records now carry the owning mdev, the owner is part of the dedup + identity, only the owner's pass marshals/finishes, and stale-owner records + are dropped. This was not itemised in the inventory, but without it the + expected dual steady state (guest B's forward refused while A's hold is + active) would marshal guest B's fallback job through guest A. + +### G8(a) — guest-GPA pin + +* Refused while more than one mdev is open, with an explicit log; the pin + remains a single-guest test aid translating through the first open mdev. + +--- + +## 3. Deviations from the inventory plan (deliberate, minimal) + +### 3.1 G3 flush is deferred, not per-guest-selective + +The inventory asks to "flush only the departing vgpu's learned state". The +learned tables (`svc.map`, `svc.pin`, `svc.ipin`, `svc.lat`, `svc.vas*`, +`svc_pool_pa`, `svc_pending`, `svc_retry_q`, `svc_pb[]`, rank bitmaps) are +still shared — that is the explicitly out-of-scope G4 remainder. With shared +tables a selective flush is impossible; the only safe minimal semantic is: +**deactivate the departing guest's session and defer the flush while any other +guest still has live channels.** One guest: identical to today (last-channel +close is the global empty transition). Consequence for T5: guest B's +close→reopen while A is live does not wipe B's stale learned entries (they +must survive for A), so B's *second* session may hit the ISSUE-0002 +contamination class for its own VAs. Closing that hole is the G4 minimum +(per-entry owner tags + per-guest lookup filter). + +### 3.2 Boot-ladder guest-GPA call sites pass `NULL` + +`gsp_phys_rmcall_pdb_build` (booter :9738) and triggers 6/7 (:19125, :19212) +have no request context; the only user of that GPA is the single-guest +test-only pin, which is now refused while >1 mdev is open (G8a). `NULL` is +the documented first-open resolution — identical behaviour for every +single-guest run, and explicitly not per-mdev. + +### 3.3 Debugfs selector: extra token, not a subdir/per-uuid file + +The hard requirement says "same debugfs file names". The recommended subdir +per UUID / `gsp-phys-guest-ram-<uuid>` file would add files; the optional +first-token UUID keeps the tree and the historical 4-token write untouched +while giving per-mdev targeting. The per-mdev form is additive: 5 tokens. + +### 3.4 Scrubber registration binds to the first open mdev + +`sema_scrubbuf_add` (debug tooling written by the guest scanner) has no +selector; it binds to the first open mdev's scrubber record (single-guest +convention, like the scanner itself) and is refused with no open mdev. + +### 3.5 CUDA bridge owner gating (follow-on commit) + +See §2/G5. Default-on (`cuda_bridge=1`); the change is inert for one guest. + +--- + +## 4. Self-check: module-global session/pump state audit + +Every read of module-global session/pump state after this change, classified: + +| # | Global state | Sites | Status | +|---|--------------|-------|--------| +| 1 | channel table `svc.chan[]` | add :1154, del :1338, ranks :1204–:1221, counters :5198/:5271/:5360/:5551/:5600/:5626/:5686, pb says :7821, gr-content :9103, gr autopin :9445, pump :13653/:15201, chan dump :15516 | **fixed** per `(vgpu, h)` | +| 2 | session flags `svc_sess_active/flush` | :468–:520, :1356–:1378, :16364 | **fixed** (per-guest records drive the global flush) | +| 3 | scrubber `svc_scrub` | :535–:580, :1272–:1295, :15973, :15912 | **fixed** per guest | +| 4 | hold `svc_fwd_hold` | :11010–:12100 (:11956/:12019 guards) | **fixed**: vgpu-keyed abort + dead-owner guards | +| 5 | request `svc_fwd_req` | :10648, :12972/:13304, :13140–:13170 | **fixed**: owner carried; single-inflight kept | +| 6 | bridge req/pend | `gsp_cuda_bridge.c` (:143, :624, :985, :1353) | **fixed**: owner carried, owner-only marshal/finish | +| 7 | opdef slots | :11185–:11500 (drop_gone :11468) | pre-existing per `(vgpu,h)`; `drop_gone` filter **fixed** | +| 8 | learned map `svc.map` / `svc.map_n` / `svc.map_hash` | lookup :3300+, rebuild :3366 | **shared (G4)** — risk §5.1 | +| 9 | pins `svc.pin[]/ipin[]/svc_ipin_alt` | :2023–:2235, flush :7176 | **shared (G4)** — risk §5.1 | +| 10 | `svc.lat[]`, `svc.vas[]`, VA shadow, `svc_hot` | :1818+, :1878+, :2011 | **shared (G4)** — risk §5.1 | +| 11 | `svc_pending[]`, `svc_retry_q[]`, `svc_pool_pa*`, `svc_cnt_der` | :7081+, :885+, :2586+, :5335 | **shared (G4)** — risk §5.1 (pending is the top cross-guest write vector) | +| 12 | rank latches `svc_ring_done`, `svc_pb_done`, `svc_gr_ring_done`, `svc_pb[]`, gr-content state | :355, :341–:347, :501–:511 | **shared latches (G4)** — re-arm is content-validated/idempotent; extra re-scans possible | +| 13 | `svc_userd_sweep_at/n` (ce_index) | :13782+ | **shared (G4)** — equal ranks across guests share sweep slots | +| 14 | per-pass capture `svc_fwd_cand`, `svc_gr_cand`, `svc_fwd_rel` | :3896+, :2587+, :7233 | serially reused by the single service thread; safe with one in-flight, listed for completeness | +| 15 | debug surfaces reading the first open mdev | `add_map` rewind :15693, `dump_vidmem` :15783, `find_vidmem` :15819, `counter_add` :15868, scrubbuf :15912 | **debug tooling, single-guest by design** — do not use in dual runs | +| 16 | `gsp_phys_gpa_to_hpa(NULL,…)` (boot-ladder pin) | booter :9738/:19125/:19212 | **by design** first-open (§3.2, G8a refuses dual) | +| 17 | last-build freshness `svc.last_build` | :3570, :16105–:16114 | **fixed**: per-guest gate; `svc.last_build` kept as the published stamp | +| 18 | bridge debugfs FIFO views (job/in/out) | `gsp_cuda_bridge.c` | one global daemon FIFO; single-guest tooling surface | + +No call site that carries a requesting/owning vgpu still matches session state +by handle alone in the paths in scope. + +--- + +## 5. Risks (ranked) + +1. **Shared learned tables (G4 remainder) — highest.** Cross-guest resolution + is still possible: a lookup for guest B's VA can hit guest A's pin/map + entry with the same VA; `svc_pending[]`/`svc_retry_q[]` entries carry no + owner, so a queued release decoded for B can be applied through A (or + vice-versa). Expected symptoms under T2/T4: wrong results, stalls, + `rel-apply` writes to the wrong guest. This is the next planned change + (G4 minimum: owner tag on pin/map/lat/pending + per-guest lookup filter). +2. **mdev-removal lifetime race.** The pump can be mid-pass for a vgpu whose + fd just closed; the vgpu struct is freed after `vfio_put_device()` in the + remove path. Pre-existing class (single-guest too), reduced by + `gsp_sema_service_vgpu_cleanup` (channels/hold/records cleared before the + free), not fully closed (the pump pass itself is not synchronised). +3. **G3 staged semantics (T5).** Guest B reopen while A is live inherits B's + previous learned entries (see §3.1). A's next run is protected (no flush + while A is live) — that is the T5 guard — but B's own reopen may hit the + ISSUE-0002 wedge until G4-minimum. +4. **Shared rank latches/pairing (`svc_pb[]`, done bitmaps, sweep arrays).** + Content validation makes re-arms idempotent, but equal ranks across guests + can pair the wrong push-buffer base records and share USERD sweep slots; + possible slowdowns/mis-pairs under concurrent CE load. +5. **Legacy guest-RAM write semantics.** With two guests, the 4-token write + configures the legacy record served by the first open mdev; a script that + writes it expecting "the new guest" will bind guest A. Use the 5-token + UUID form for the second guest. +6. **Reducer scope:** no boot_stage split, no per-mdev debugfs views, no + parallel forwarding — per plan. The single physical GR channel stays safe + under `gsp_phys_rm_lock` (inventory §4), serialized at launch granularity. + +--- + +## 6. Live-validation plan (offline work ended here) + +All of the following is for the LEAD under the supervised main-job rules +(no unattended cycles; stock guests; evidence per run). The module was NOT +loaded by this task. + +### 6.0 Build/load (from the dual worktree) + +```bash +cd ~/Git/integdev_gpu_drv/open-iov-dual/kernel && make -j8 +# note: modpost needs +# ~/Git/integdev_gpu_drv/open-iov-dual/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers +# (gitignored; copied into this worktree from the master tree — keep it present +# or the final link fails with "No such file or directory") +``` + +Load (after a clean/unloaded host; parameters per inventory §5): + +```bash +sudo insmod ~/Git/integdev_gpu_drv/open-iov-dual/kernel/vgsp.ko \ + gsp_phys_gpu=1 gsp_phys_mdev_owned=1 gsp_phys_guest_mem=1 \ + gsp_phys_work_forward=1 gsp_phys_gr_forward=1 +``` + +### 6.1 Setup — two turing mdevs, two QEMUs + +```bash +GPU=0000:01:00.0 # substitute the campaign's GPU BDF (10de:…) +T=nvidia-nvidia-vgpu-turing +UUID_A=$(uuidgen); UUID_B=$(uuidgen) +echo "$UUID_A" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create +echo "$UUID_B" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create +# available_instances must read 2 - 2 = 0 afterwards +``` + +Launch QEMU A and B, each bound to its own mdev +(`-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID_A` resp. `$UUID_B`), +each with `memory-backend-file,share=yes` pointing at a **different** file +(e.g. `/dev/shm/vgpu-ram-a`, `/dev/shm/vgpu-ram-b`). + +Register each guest's RAM window per mdev (5-token selector form): + +```bash +echo "$UUID_A /dev/shm/vgpu-ram-a 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram +echo "$UUID_B /dev/shm/vgpu-ram-b 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram +# guest A may equally be configured by the legacy campaign scripts (4-token form); +# it serves the first open mdev. A second write of a different path to the same +# record is refused -EBUSY until that mdev is removed. +``` + +In each guest: stock 570.144 driver + `tools/cuda-ab/run/vecadd_loop` (or the +guest-side build from `tools/cuda-ab/`). + +### 6.2 Test matrix (inventory §5, T1–T5) + +| Test | Action | Pass criteria | +|------|--------|---------------| +| T1 | `nvidia-smi -L` in both guests simultaneously | no host error/stall (`dmesg -T | grep -i vgsp`) | +| T2 | start `./vecadd_loop` in A and B within 1 s | each prints `LOOP launches=100 bad=0`, exit 0 within 600 s | +| T3 | A vecadd, B started 10 s later | both green | +| T4 | `smiley` in A + `vecadd_loop` in B (CE vs GR cross) | both green | +| T5 | stop guest B → its mdev removed → recreate B (+ its RAM record) → rerun A and B | A green across B's cycle (B itself: see risk §5.3) | + +Evidence capture: per-guest client logs + host dmesg to +`backups/evidence-*/dual-guest-<run>/`; require ≥200 lines each of +`GUEST GR LAUNCH OK` and zero `fwd_gr_fault`/timeouts in the steady state. +Per-mdev attribution in the log: `guest RAM configured for mdev <uuid>`, +`channel 0x… registered` around each guest's `GR LAUNCH submitted` stream. + +Useful live checks: + +```bash +dmesg -T | grep -E "sema-svc|vgsp-gr|GUEST GR LAUNCH OK|cuda-bridge" | tail -200 +cat /sys/kernel/debug/vgsp/gsp-phys-state | grep -E "guest_ram|work_guest|backend" +sudo cat /sys/kernel/debug/vgsp/gsp-phys-state | grep guest_ram # both mdev records visible +``` + +Baseline (pre-fix) for the regression control: T2 with the old module should +show the second guest stalling or wrong results (inventory §5). + +### 6.3 Rollback + +Worktree-local: `git checkout ecfb1278 -- kernel/ && make -j8` in +`open-iov-dual`; the master worktree is untouched. + +--- + +## 7. Artifacts + +* Branch `dual` commits `81c11ca0`, `5769d5c4` (local only, not pushed). +* `kernel/vgsp.ko` built at `5769d5c4` (offline; never loaded). +* This report. No sessions/dumps generated (no device access). diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index c16b774..27295df 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,8 +1,13 @@ # Open-IOV Current State -> **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-07 ~01:15; debugging-options -> catalogue added 2026-10-05).** This is a dated public +> **Snapshot: 2026-10-20 ~20:00 (two more fixes landed: the SWAP_RX cursor +> and the pin-wins ledger divergence that cured a guest-kernel-corruption +> panic; guest CUDA kernels green on fresh boots - `victory_mail`, +> `victory_matmul` (first-run flake), transformer, `vector_add` (1M, +> `bad=0`), `vecadd_loop` (100 launches); two guests with two emulated VRAM +> sizes (512M + 1GiB) coexist live; frontier = the multi-pass content class +> and the streams2 launch-order wait).** +> This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: @@ -146,6 +151,218 @@ isolation, performance, portability). aperture. Ranked next: read the server-FBMEM fault buffer via a copy-engine transfer, per-VEID/compute-context execution requirements, and the scheduler CWD field semantics. Host clean, GPU green, no reboots. + **Update 2026-10-14 ~13:30 (major - guest compute runs on the physical + engine):** the physical-backed guest campaign now runs a multi-client + CUDA battery against the real TU104 with the host CUDA bridge disabled + (`cuda_bridge=0`, GR forwarding on). In one cycle (r137) the first client + (`victory_mail`: `MAIL` + `VADD bad=0` + `STATUS OK`) and the second + client (`smiley`: `STATUS OK`) are both green on the physical backend, + and the guest's compute launches execute on the physical GR engine (the + window-attribution defects where kernels wrote into the wrong window are + fixed; first-client guest VADD results verify). The 2026-10-08..12 + second-client campaign closed its chain (CE USERD location, CE + push-buffer pin delivery, a channel-anchored CE pool resolver + `ade77d9a`, session-flush pin semantics, data-argument resolution + order). Wall closure since 2026-10-07: (a) the third-client + (`victory_matmul`) PMA-scrub freeze is root-caused live - the + boot-session scrubber channel's finish word is a guest-RAM vmap that + collides with a client VA - and fixed by authoring the scrubber + channel's progress (`624438bb` + guest scanner `80af379d`), so + `victory_matmul` gets past `cudaMalloc`; (b) the intermittent + GR-forward launch timeout is root-caused: the host publishes forwarded + launches modulo an 8-entry ladder ring while the user-lane channel was + declared with 1024 GPFIFO entries, so after the first host-ring wrap the + front-end's GET/PUT became inconsistent and it stopped fetching + (`074f493c` fixes the lane depth to 8; the earlier two-client acceptance + never wrapped and had masked the regression). The r139 cycle validated + the ring fix: the 100-launch soak crossed the host-ring wrap cleanly + (`LOOP launches=100 bad=0 STATUS OK`; the previous cycle died at launch + 8), c1 `victory_mail` is green (60 s), and `smiley` green on a later + retry. The residual is an intermittent cold-context/fetched-not-executed + launch race (18 failed launches in the cycle: 6 never fetched by the + front-end, 12 fetched but not completed by the engine; some later + clients intermittently return all-zero results), for which a bounded + in-submit retry with a fresh ring entry plus a cold pre-warm is being + implemented; the "OK with 0 changed" change-counter is not a reliable + execution witness - the client's own `STATUS` is the acceptance signal. + Host clean all session, no Xid in the campaign logs, pstore empty, no + reboots (in-session PCI function-level reset recovery where needed). + ISSUE-0002 (second client in one boot): the sequential case can pass + (first two clients green in one cycle; smiley green on retry in the + next) but is not yet deterministic; the issue remains open pending + repeat-run plus concurrent/simultaneous validation. Active front: + the multi-client battery on the physical backend plus launch-latency + robustness. + **Update 2026-10-14 ~15:40 (afternoon wave, r140-r143):** the full fix + set is committed and pushed (HEAD `17393a74`: ring depth `074f493c`, + scrubber `624438bb`/`80af379d`, bounded fresh-entry launch retry + `6b090f70`, 208-page budgeted data window `fce5ab8d`/`4a23c196`). r142: + c1 `victory_mail` and c2 `smiley` GREEN; c3 `victory_matmul` FAIL + (all-wrong operands - the three 256 KB operands resolved only ~32 of the + needed 64+ pages each); launch counters `ok=86 failed=10 retries=4` + (the retry recovers a subset of the cold-context timeouts silently); + then `victory_saxpy`/`memcpy_roundtrip` stalled and the guest SSH wedged + - the intermittent stall/wedge class survives. r143 (live at this + check, same fix set): c1 green within ~2 min; c3 matmul launched and + improved to `bad=8192` of 65536 with a non-zero checksum (operand + resolution now 68/64/64 pages) - STATUS FAIL. **Root cause found + (sub-agent, evidence-grounded): the win0 QMD shadow pool base was still + `0x190000` (main-PT slots 0x190..0x197), which lies INSIDE the enlarged + 208-page bounce window (`0x110000..0x1df000`); the win0 plant runs on + every `pdb_build()` while the bounce plant is one-shot, so win0 stole the + PTEs for bounce pages 128..135 = C rows 0..31 = exactly `bad=8192`. Fix + landed (unvalidated live at pause): `GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE` + moved to `0x1e0000` (slots 0x1e0..0x1e7, above the mail/marker pages, + below the 0x200000 guest window) plus a `static_assert` invariant and + comment/log corrections.** Ranked next blockers: (1) the 256 KB operand + data-path placement; (2) the intermittent client stall/wedge class + (the top reliability blocker; needs the stalled-client capture); + (3) the cold-context launch race (cold GR-sem pre-warm designed); + (4) then multi-mdev, two guests, VRAM sizes and the concurrent/full + soak. Host clean, GPU green, no reboots. + **Update 2026-10-15 ~18:15 (resume + parallel workstreams; pause snapshot):** + the host rebooted (the OS moved the kernel to 6.8.0-147; the module was rebuilt + for it). The r155 cycle on the physical backend: ladder green; `victory_mail` + (`VADD bad=0` + `STATUS OK`) and `victory_matmul` (`MATMUL n=256 bad=0` + + `STATUS OK`) both green end-to-end, and a 100-launch `vecadd_loop` passed; the + sequential battery then hit the known acked-no-write class on `streams2` + (all-zero output despite `qmd_rel=0x6` completion acks -> `STATUS FAIL`). Three + offline workstreams landed as pushed branches: (a) `bigpath` - a 64 MiB + boot-fixed data-bounce region (stage-3-planted PTEs) with fail-closed cap + semantics, unblocking large operands (victory_math 512 KB, vector_add 4 MB, + memcpy_roundtrip 8 MB); (b) `dual` - per-mdev separation (per-guest channel + keys, session records, guest-RAM records, forward-request ownership) for two + simultaneous guests; (c) `lottery` - analysis proving the completion ack is not + an execution witness, rejecting naive fail-closed (idempotent-client false + positives), and ranking the next experiments (win0 bank rotation, qmdwin + double-buffer, warm-sem retry, executed-vs-wedged discriminator). Host clean, + GPU green, VM left live; user-requested poweroff at pause. + **Update 2026-10-16 ~15:30 (integration wave; five fixes landed; no reboots):** + bigpath + dual merged to master; then, pushed through `7d123630`: qmdwin + double-buffering (E3); a warm-sem-push discriminator on zero-write retries (E5 - + proved the engine is ALIVE during the acked-no-execute class, so it is a + launch-form/dispatch race, not an engine wedge); R1/m157-m159 - busy-refused GR + captures now wait (bounded) and retry inline, so multi-launch clients' kernels all + execute (streams2's three kernels ran serially on the physical engine); and a + guest-side demand-pin offset fix (CE-window mask `&0xFFFFF` folded every 1 MB rank + to page 0) that removed the repeat-client stall class. Verified live (r161/r162, + 512M mdev): the exact user repro (`victory_mail` then `victory_matmul`) green + back-to-back with `bad=0` + `STATUS OK`, and a four-consecutive-client green boot + (`victory_mail` x2, `victory_matmul`, `smiley`). Remaining single-guest gaps: the + intermittent acked-no-execute launch lottery (one `smiley` pass in r162) and a + streams2 second-channel capture miss (entry consumed before its push page was + readable); both characterized with fix directions in the private handoff. + **1.0 multi-tenant:** two mdevs of different VRAM sizes (512M + 2 GiB) and two + QEMU guests coexist; guest B reports `memory.total = 2048 MiB` with `nvidia-smi` + green. Guest-B CUDA is blocked by the shared host pin table (cross-guest + contamination demonstrated live) and needs the per-guest ownership rework + (owner-tagged pin/map/pending lookups) before any dual-CUDA claim. No reboot; + host GPU green; both guests left running. + **Update 2026-10-16 ~22:35, extended ~23:05 (reliability wave 2, m163-m170; cycles r169-r177, no host reboots):** + the stall classes that made clients freeze at DEVICE are largely closed. Fixed and + verified across boots r172/r174/r175/r177: the session flush can no longer wipe a live + session (a deferred flush executed with the client's channels valid, dropped its + queued launch release, and froze it in the tracker wait); GR entries whose decode + sees QMD words but builds no launch candidate are held and re-captured instead of + consumed silently; a failed device-destination copy holds its entry instead of + silently dropping the data; pool releases apply at fresh-pin time instead of + registration+5 s; the timeout ladder fails fast when the engine is wedged; argument + spans whose first page resolves nothing hold the launch (bounded) instead of + forwarding stale operands; device-destination copies answered only by a page-table + walk wait (bounded 3 s) for a pin. `victory_mail`, `victory_matmul` and `smiley` + are green on every boot of the wave, including the user's exact repro and mail as + the eighth client of a used boot; r177 ran mail, matmul, smiley, matmul (repeat) + and mail GREEN back-to-back as its first five clients. streams2/memset2d/saxpy no + longer wedge - they + now fail fast (~35-60 s) on the remaining data class: an H2D into a device buffer + that does not land where the launch's data fill later reads the operand (the + vidmem write path vs the fill's read path divergence; analysis and the ranked + ledger fix are in the private handoff, cont. 23). + **Update 2026-10-17 ~01:00 (stall pinpointed on r179; diagnostics only, no tree change yet):** + a minimal repro (`cudaEventCreate` + `cudaEventRecord` on the default stream + + `cudaEventSynchronize`) sits forever after the record - the first CUDA call pinned down + as a hard hang. A 3-stream workload using event record/wait/elapsed time stalls even as + the very first client of a fresh boot (intrinsic, not position- or poison-dependent; + unclean client kills were tested and do not break the next client). On a used boot the + user's `victory_mail`/`victory_matmul` hit the same wait: the client spins in driver + userspace while a background worker polls the interrupt-service RM control once per + second. The host's event-registration table and post path exist (register on NV01_EVENT + alloc, retire on free, per-channel/all post helpers) and the stall windows still show + frequent post failures with EINVAL; the host sema side is verified healthy at the stall + (client pushes decode, pool releases apply to the correct page, then the machine idles). + Cross-boot matrix: mail/matmul/smiley green early; strided/atomics/memset2d/saxpy fail + fast on the remaining data class at any position; event-sync stalls always. Next: fix + the completion post so the recorded event wakes (details in the private handoff, cont. 24). + **Update 2026-10-16 ~18:40 (reliability wave m160-m162 + harness fixes; verified battery, no reboots):** + five cycles (r163-r167) landed and verified the bounds of the earlier "STATUS FAIL / + freeze" repro. Fixed and live-verified across boots: `victory_mail` and + `victory_matmul` (repeats), `smiley`, `vector_add`, `victory_math`, `victory_strided` + and `vecadd_loop` (100 launches) all green on the physical engine; the acked-no-execute + class is recovered by outcome retries (fresh release-word ack, fresh inline-QMD bank + + rebuilt method block: a launch with zero engine writes recovered on retry 2 with its + full 1024-page output). A hardware fetch witness (QMD sentinel) now distinguishes + fetched-vs-not in every retry log. The post-teardown DEVICE-freeze cascade was rooted in + the pin-feeder tooling: blocked guest-trace reads over the guest agent stopped the + pool/tracker pin flow after ~5 client cycles and every later client froze in + `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the + demand-pin content resolver searches a learned per-allocation offset. Remaining + characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch + class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out + mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, + retest pending). + **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):** + the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was + reproduced and root-caused: under the driver's status-queue SWAP_RX mode the + emulation's status free-space check read the status queue's mirrored RX header + (which the emulation publishes itself) instead of the peer consumption cursor in + the command queue's RX header. With the guest mid-drain the status ring was + declared full, replies were written anyway, unread records were overwritten, and + the guest RM read a record one full ring lap ahead + (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed + and the first client's `cudaMalloc` returned "device busy or unavailable" - + poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the + cursor is read from the command queue's RX header per the driver's swap-mode + semantics; per-client channel unregister and session flush work again (one + unregister in the poisoned boot; 79 with one flush per client after). Verified on + fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul` + (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`, + `FFN4 layers=4 bad=0` + `STATUS OK`) all green. + Remaining large-buffer class (current frontier): a `vector_add` cold-run content + gap (a staging read/write race between the emulated CE copy and the guest's + staging ping-pong: ~0.68 MB of the third MB reads the previous staging content), + `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop` + hang class correlated with a boot where the injected-pin stream never re-targeted + the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class + (RM object fields overwritten with client data; stale system-memory destination + resolution on freed staging pages) is characterized but open. Next: pin-stream + re-targeting, then the staging-race and stale-destination fixes; then the + boot-lottery re-measure over fresh boots, the full battery + soak, two guests, + multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live + with the three green kernels. + **Update 2026-10-20 ~20:00 (pin-wins corruption fix; multi-tenant live; battery frontier):** + a second, independent fix landed after the SWAP_RX cursor: the emulation's + write-side ledger could redirect a copy to the page an earlier write landed + on even when the guest had already remapped or freed that VA (4448 such + divergences in the failing run), so guest-kernel memory could receive client + copy data - observed as an RM object's pointer field holding client float + data, a free-path traversal faulting, and a guest panic + ("Attempted to kill init") on the `vector_add` client. With a pin-backed + translation (guest pagemap truth) now winning the divergence and the ledger + re-homing after the write (`720d3f5f`): `vector_add` cold AND warm run + `bad=0` + `STATUS OK` and the guest survives; the cold-run content gap + class is cured with it. Battery on fresh boots: `victory_mail`, + `vecadd_loop` (100 launches), `victory_matmul` (a first-run all-bad flake, + green on immediate retry), `victory_strided`, `victory_math`, + `vector_add`, and a transformer kernel (`dh=256`/`FFN4` `bad=0`) are green; + streams2's ordering class is root-caused (core CUDA starts compute-channel + pushes with a C56F host-semaphore acquire that the decoder ignored, so a + D2H readback ran before the scale launch it waits for) and the multi-pass + content class is the remaining frontier (multi-launch/accumulating kernels + wrong; `victory_reduce` is exactly 2x - the engine never wrote on five + launch-shaped pushes of a one-launch client). Multi-tenant: two mediated + devices (512M + 1GiB) and two guests coexist; guest B reports its 1GiB + VRAM with `nvidia-smi` green; guest-B CUDA awaits its own pin stream. Host + clean, no reboots. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -265,6 +482,12 @@ isolation, performance, portability). the full FWCLIENT client-managed construct (client ctx buffers + VAS mapping + reference promote set), then the server commit. See ADDENDUM 53/54. + **Update 2026-10-14:** this wall moved decisively - the physical GR + engine now executes forwarded guest compute launches, and the first two + battery clients are green end-to-end on the physical backend (host CUDA + bridge disabled). The remaining items are launch-latency robustness (a + first launch after an idle can time out and is auto-retried to success) + and the third-client chain; see the status update above. 2. **Hardening the accepted no-claim path.** The first green guest CUDA acceptance has landed and is holding; the residual classes are (a) corruption via a late first decode behind pump backlog (~3-5% of runs; the experiment @@ -463,6 +686,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM | | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | @@ -499,6 +723,29 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX + cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20 + ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`, + `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM + logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed + (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable") + onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr + lives in the command queue's RX header (the driver's swap-mode semantics), + while the status queue's own RX header is the mirror the emulation publishes + (its own command-queue read position); the status free-space check read the + mirror, so with the guest mid-drain the ring was declared full and replies + were written over unread records. Fix (`c72f8948`): read the peer + consumption cursor from the command queue's RX header. Verified on fresh + boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256 + bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`) + all green with `STATUS OK`; per-client channel unregister + session flush + restored (1 unregister in the poisoned boot vs 79 after). Remaining + frontier: the large-buffer class (`vector_add` cold-run staging race, + `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with + an un-targeted injected-pin stream), then boot-lottery re-measure, full + battery + soak, two guests, multi-mdev, VRAM sizes. Evidence: + `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in + tmux session 0 (SSH to the guest, three green kernels). - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns @@ -1583,7 +1830,7 @@ progress.** ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. -- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages +- Host OS: Ubuntu, kernel 6.8.0-146; the distribution NVIDIA driver manages the physical GPU; `vgsp.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index 733da13..3d97d07 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -568,6 +568,39 @@ forwards (local evidence and the prepared harness: forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. +### 3.5a Copy-out change detection: fingerprint rule + +A forwarded launch stages every referenced data page into the data-bounce +chunk, and after the retire wait the copy-out ships back only the pages the +engine actually modified. That decision used to be a 32-bit additive +checksum of the page's words, compared against a baseline captured at +copy-in. The additive form has a structural blind spot: doubling every word +of a 4 KB page raises each float exponent by one (+2^23 per word), and 1024 +words contribute `1024 * 2^23 = 2^33`, which is exactly `0 (mod 2^32)` — an +in-place power-of-two scale of a whole page (`c[i] *= 2.0f`) left the +checksum bit-identical, the page was classified "engine did not touch", +the write-back was skipped, and the client read the pre-scale values. The +same blindness applies to `*= 0.5f`, `* 4`, `* 8`, etc. Since `r452` +(commit `19399f4f`) every change decision in the copy path — the retire +wait's settle scan, the tracked and gap copy-outs, the window witness, the +late-store witness and the idempotent-launch fingerprint — uses a 64-bit +FNV-1a hash over the page instead; the additive sum survives only as the +all-zero filter. Policy rule for the plane: **a change-detection +fingerprint must not be an algebraic invariant of any plausible kernel +operation** — use a hash, never a sum. + +The reader side of the same decision is the landing table: a D2H copy of a +device VA resolves through the last recorded write of that VA. Since `r453` +the forwarded QMD's release0 is redirected to the module semaphore by +default (`gsp_phys_rmcall_gr_rel_mod=1`): the guest-visible release word is +authored by the host only on the completion edge, after the copy-out has +shipped the engine's stores, so a client's stream-ordered readback can no +longer run ahead of the write-back (the historical shape: the engine's own +release satisfied the guest while the copy-out was still pending). The +engine-completion observable remains the module semaphore, and the +per-outcome authoring path guarantees the guest word is applied on every +outcome (timeout, wedge, no-write). + ### 3.6 Citation map for this section | Claim | Authority | @@ -1038,6 +1071,53 @@ technical barrier between the project's architecture and its promise. --- +## m91-class I-fetch faults: capture and icache coherence (2026-10-18) + +The physical submit's remaining transient failure class ("m91 class" after the +trap-decode campaign that first named the SM instruction-fetch signature) has +two observable faces. In one, the first attempt of a mid-sequence launch +raises SM warp exceptions ("Illegal Instruction Encoding", "Out Of Range +Register") on garbage instruction bytes while the engine-authored release +stays at its failure value, and a retry of the *identical* QMD and shadow +content succeeds. In the other, the launch reports fetch success but writes +nothing (dispatch loss). The submit retry that recovers the first face is the +fast re-arm: the same method block is re-published on a fresh GPFIFO ring +entry and the mode-35 lane-enable tail plus channel/TSG schedules and the +doorbell are re-issued — no shadow re-copy, no QMD re-read, no content +change. Identical content flipping outcome across attempts is the signature +of stale engine-side fetch state, not a content or relocation error. + +Two code changes follow from that: + +* **FAULT-CAP (diagnostic).** When a forwarded launch's fault epoch advanced + across its window, the submit emits one `vgsp-m91: FAULT-CAP` line per + launch (capped at 20 per module-load cycle, then one suppression notice): + fault-epoch delta, the forwarded QMD address and first 64 B, the live + program pair (word 48/49 read) with its resolved engine-visible VA/PA, the + first 64 B at that code address, the retry decision (fast re-arm / + resched/fresh-entry / none) and outcome (ok vs timeout), and the newest + relay-decoded fault note (exceptType, chid, errString). It is inert on a + fault-free launch: one epoch read and one compare. +* **icache coherence (fix, default on).** The window-0 shadow pool alternates + two 8-page banks by submit epoch, so the bytes at a program VA change every + two launches, while the captured guest QMDs carry the QMD v2.3 + cache-control word (word 5) with the + `INVALIDATE_INSTRUCTION_CACHE` bit (bit 30) clear — the field the client + driver arms only under its forced-invalidate mode. The submit now sets + that bit in the forwarded QMD when the QMD carries the driver's standard + cache-control signature (word 5 bits 26..29 set), forcing an instruction + fetch at dispatch and removing the stale-line window; an unknown QMD + revision is never touched, and `gsp_phys_gr_icache_inv=0` restores the + exact pre-change QMD for a one-flip A/B. Setting the bit cannot alter + instruction bytes or data — it only invalidates cached fetch state — so + green launches keep their semantics. + +The next physical cycle's evidence is the FAULT-CAP lines (or their absence) +against the four-client class: zero FAULT-CAP lines with the four clients +green confirms the stale-fetch mechanism; FAULT-CAP content policies the +remaining candidates (shadow-VA reuse distance, cross-cycle context state +left by killed clients). + ## See also * [`01-system-overview.md`](01-system-overview.md) — principles and layer map @@ -1057,3 +1137,4 @@ technical barrier between the project's architecture and its promise. |---|---| | 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. | | 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (`svc_thread`, `svc_gr_request`, `svc_pending_add`, H3 guard, readiness gate); GR completion §3.4/§6.1 reconciled with FORENSICS-grv5 (wake-up fires; the counter publication is the leading reading); discovery cadence wording corrected (2-second paced passes, not "milliseconds"); mode-matrix source note corrected. | +| 2026-10-18 | m91-class I-fetch section: FAULT-CAP first-fault capture (`vgsp-m91: FAULT-CAP`, 20/cycle) and the default-on forwarded-QMD instruction-cache invalidate (`gsp_phys_gr_icache_inv`) with the retry-state analysis (identical content, fresh ring entry + lane-enable replay; stale engine-side fetch state at reused shadow VAs). | diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md index f5546ca..a8e03ee 100644 --- a/docs/04-implementation/03-execution-frontier.md +++ b/docs/04-implementation/03-execution-frontier.md @@ -1,11 +1,17 @@ # The Execution Frontier: The GR Compute-Launch Chain -> **Status: 2026-10-06.** This page describes the current leading edge of +> **Status: 2026-10-20.** This page describes the current state of > guest-facing execution: the chain that must fire for a stock guest CUDA -> kernel to run on the physical GPU, what we have proven about it, where it -> currently stops, how we know, and the open lines of investigation and -> theories. It is evidence-led: every statement carries either a live -> observation, a decompilation finding, or a clearly labelled theory. +> kernel to run on the physical GPU, what we have proven about it, how we +> know, and where the leading edge is now. The chain fires end to end: +> verified guest CUDA clients (vector-add, matrix-multiply and a transformer +> kernel) execute on the physical GPU, their results verify, and the +> engine-authored completions relay back to the guest. The leading edge has +> moved from "can a forwarded launch execute at all?" to breadth and +> hardening — the large-buffer staged-transfer class, reproducibility across +> boots, and per-guest execution isolation. It is evidence-led: every +> statement carries either a live observation, a decompilation finding, or a +> clearly labelled theory. > > Evidence sources for this page: supervised physical cycles (module-owned > GSP-RM on the reference TU104), offline firmware emulation/probing, the host @@ -40,40 +46,59 @@ flowchart TB J["Host polls release, relays to guest;<br/>CUDA wakes"] ``` -| # | Stage | State (2026-10-06) | +| # | Stage | State (2026-10-20) | |---|---|---| | 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) | | 2 | GPFIFO entry + doorbell | ✅ observed | | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) | | 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) | -| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** | -| 7 | **Engine reads the QMD** | ✗ **not observed** | -| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 | -| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack | -| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches | -| 11 | Host relays completion; guest wakes | ✗ guest waits and times out | - -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) | +| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed | +| 7 | Engine reads the QMD | ✅ observed | +| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 | +| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve | + +**One-line summary:** the whole chain is live — verified guest CUDA clients +execute on the physical GPU and wake up correctly. The leading edge is no +longer the chain itself but its breadth and hardening: the large-buffer +staged-transfer class, reproducibility across boots, and per-guest execution +isolation. --- ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire we observe the ring entry consumed and the -front-end methods run — but the launch descriptor is never materialised, the -engine never reads a QMD, no engine-authored completion appears, and the guest -blocks in its synchronisation wait. - -This is **not** a context-validity wall (that was the previous frontier and is -now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the -descriptor destination (that install is accepted). The wall is the **form of -the launch method stream**: what the front-end requires in order to install the -inline descriptor, and what the engine requires in order to read and dispatch -it. +A stock, unmodified guest CUDA program now executes its kernels on the +physical GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. The engine authors the completions, the host relays them to the +guest, and the guest's synchronisation waits resolve. + +The leading edge has therefore moved from the launch chain itself — it fires — +to breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise, the first + CUDA client would fail, and the rest of that boot was poisoned (the + user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement + across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. --- @@ -81,17 +106,17 @@ it. | Observable | What it proves | What it does **not** prove | |---|---|---| -| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine | -| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) | -| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started | -| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction | -| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path | - -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results | +| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches | +| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored | +| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress | +| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction | + +**Consequence:** acceptance is now measured by verified guest results plus the +engine-authored release; front-end signals remain necessary but not +sufficient. A green run counts as progress only when the same acceptance +reproduces across fresh boots. --- @@ -111,90 +136,94 @@ signals, which we have already learned to distinguish. * **Parenting/TSG for the fetch** — both parenting variants fetch identically. * **The copy-engine path** — CE-class work, its releases and its fault relay are live (default off); the GR path is a separate problem. +* **The descriptor-gate line of investigation (the old T1–T4 theories).** The + inline-descriptor install, the engine's QMD read, the grid dispatch, the + engine-authored release and the guest wake-up are all now observed; the + gate they questioned is closed by observation (§1, §7). +* **The old boot-poisoning defect.** Root-caused to the status-queue + consumption cursor and fixed; it is no longer the explanation for a failing + first client (§5, issue B). --- ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** -The forwarded stream includes a verbatim copy of the guest's prelude whose -inline-data writes target addresses that are not present in the engine's view -of the channel's page tables, and the referenced data is not relocated. The -shorter, proven stream shape fires its tail (channel semaphore) but still does -not schedule the descriptor. -*How we know:* the prelude stalls in the forwarded form; the shorter shape's -tail fires; the descriptor-install sentinel never changes in either case. -*If fixed:* the install path is reached and we can observe stage 6 directly. - -**Issue B — the descriptor-virtualization control method is absent.** -The method that arms the inline-descriptor install path (the QMD -virtualization control) is not present in the forwarded stream. -*How we know:* the method does not appear in the submitted stream, and the -install sentinel never changes. -*If fixed:* the front-end should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. - -**Issue C — diagnostic hygiene.** -The descriptor sentinel placed at word 0 overlaps real descriptor fields; it -must be moved to an unused word so diagnostics do not perturb what they -measure. +**Issue A — the large-buffer staged-transfer class.** +Multi-megabyte staged transfers between guest memory and device memory are +under repair: some large multi-chunk clients still mis-verify. +*How we know:* the small and medium clients (vector-add, matrix-multiply, the +transformer kernel) verify green while large multi-chunk transfers mis-compare +or stall in the same configurations. +*If fixed:* the client battery should accept the large-buffer class with the +same verification discipline as the green clients. + +**Issue B — reproducibility across boots (re-measurement).** +A boot-poisoning defect was root-caused and fixed: under the driver's +status-queue mode the emulation read the wrong queue cursor, so under load the +reply ring could desynchronise, the first CUDA client would fail, and the rest +of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report). +*How we know:* the failure was reproduced, the cursor rule was corrected, and +per-client unregister and session flush work again. +*If fixed:* green on every fresh boot — re-measurement is in progress, and no +"every boot" claim is made until it completes. + +**Issue C — per-guest execution isolation.** +Multiple mediated devices with different emulated VRAM sizes and multiple +guests already coexist at the device level; per-guest execution isolation for +multiple simultaneous CUDA guests is in progress. +*How we know:* the device-level coexistence is live; execution isolation is +not complete. +*If fixed:* multiple CUDA guests execute simultaneously without interference. --- ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline (the shorter stream that - fires its tail) and **re-add the virtualization control method** — one - variable at a time, with the install sentinel and the engine-read signal as - the readout. -2. **Relocate or drop the verbatim prelude** so the inline-data writes target - addresses present in the engine's view, or replace it with the equivalent - host-side staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference.** Two ways to obtain one: (a) make - our own ladder dispatch a descriptor built from the same source material - (in-house reference), or (b) capture a working launch of the stock stack on - the same GPU (real method stream + descriptor + engine-visible channel - state) and compare field by field. Either reference converts the remaining - work from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware emulator/probe tooling before spending a - supervised cycle. +1. **Repair the large-buffer staged-transfer class** — the multi-megabyte + guest↔device transfers that still mis-verify. The three green clients stay + the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility, and + keep the re-measured battery as the regression gate. +3. **Per-guest execution isolation** — build from the device-level + coexistence of multiple mediated devices and guests to per-guest execution + isolation. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel run as a battery on every re-measurement; each entry is + accepted only on its own verification output (`bad=0` + `STATUS OK`). --- -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the virtualization-control method - enables the install path; its absence alone explains the inert descriptor. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape - works, the prelude's refused writes leave the launch path in a state that - prevents the descriptor from being scheduled. -* **T4 — relocation is the missing transform.** The guest's inline descriptor - is valid, but it is built for the guest's view of memory; the forwarding path - must relocate referenced data (and any embedded pointers) into the channel's - view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed (§1), +so the launch chain they questioned is now proven end to end. + +The remaining leading-edge work is engineering rather than theory: the +large-buffer transfer class, boot-to-boot reproducibility, and per-guest +execution isolation. Each has an acceptance signal named in §5 before the +work. --- ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals (ring cursor, method execution, semaphores that the front-end can - author) are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end for the verified clients; the guest's own verification + (`bad=0` + `STATUS OK`) is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred; the current GR path runs without an error-reporting object, which is itself a known observability gap. * **Emulator verdicts are L0/L1.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** The + boot-poisoning defect is the worked example: one green boot is not + reproducibility. * **A wall is closed only by an acceptance signal**, and the acceptance signal for this frontier is named before the experiment. @@ -218,3 +247,4 @@ signal, and the engine-authored release, in that order. | Date | Change | |---|---| | 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. | +| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. | diff --git a/docs/05-operations/05-gpubreach-audit.md b/docs/05-operations/05-gpubreach-audit.md new file mode 100644 index 0000000..3c60385 --- /dev/null +++ b/docs/05-operations/05-gpubreach-audit.md @@ -0,0 +1,95 @@ +# GPUBreach-class audit and threat model (vGPU host module) + +Status: audit performed 2026-10-21 against the host module at `229bec2c` +(`vgsp.ko`, `kernel/`), with the write-path guards landed in the same window +(`7d4919dd`, `7a1f4805`). Scope: every host-side write whose target address +derives from guest-supplied data. Companion reading: +[`01-performance-correctness-security.md`](01-performance-correctness-security.md). + +## 1. What the GPUBreach threat is + +GPUBreach (IEEE S&P 2026, U. Toronto) is a GPU Rowhammer chain: the attacker +flips bits in the physical GPU's page tables (PTEs), which turns the GPU's +MMU into an arbitrary *device-memory* read/write primitive; the paper then +escalates to CPU root by exploiting memory-safety bugs in the *driver* code +that consumes those corrupted translations. It works with the IOMMU on, +because the corruption happens inside the GPU, after translation. + +The relevant question for this project is therefore not "can a guest read +device memory" (device memory is the guest's own resource here) but: + +1. can guest-influenced data reach a *host* write without validation, and +2. what bounds a write that lands on a *wrong* target. + +## 2. Structural mitigations (why the host is not on that path) + +* **Guest GPU page tables are host-emulated.** The guest's GMMU page tables + live in the emulated vidmem shadow (BAR1 backing) or in guest RAM; the + host resolves guest GPU VAs through its own walk of the guest's published + page directories (`SET_PAGE_DIRECTORY`) or through content-verified pins. + There is no physical GPU MMU walk of guest-authored tables. +* **The physical GPU's PTEs are host-owned.** Backend jobs are submitted by + the host module on host-authored PTE lines; the guest never supplies a + descriptor that the real hardware MMU consumes for host memory. +* **Every write is aperture-bounded.** The one write primitive, + `svc_write_ap()`, sends SYS writes through `vfio_dma_rw()` on the mdev's + own VFIO container - the IOMMU/vfio layer only maps the guest's RAM - and + FB writes are bounds-checked against the emulated shadow size. A + guest-influenced target can therefore only ever land inside the guest's + own RAM or the vidmem shadow; a wrong target is a correctness bug, not a + host-memory write. +* **No guest DMA to host buffers.** The guest's drivers talk to the + emulated device; they cannot address host kernel buffers. IOMMU is on + (vfio), and the mdev exposes only guest RAM plus the emulated windows. +* **Guest descriptors are not trusted by construction.** Every path that + turns a guest VA into a host PA goes through the resolution chain + (guest-verified injected pins -> write/read landing ledger -> published + page-directory walk -> targeted record scan), and the chain's outputs are + re-validated per page (see §3). + +## 3. Write-path inventory and validation state + +| Write path | Target derivation | Validation today | +|---|---|---| +| CE fill / memset (`svc_fill`) | guest VA or physical FB/GPA from the push | per-page destination resolution (pin -> ledger -> walk), page-boundary clamped; a page with no resolution truncates loudly and demands the VA through the pin channel (`FILL-TRUNC`, `stalled VA`); zero fills onto page-table-shaped guest pages are refused (`FILL-PT-REFUSED`); below-floor window fills that resolve outside vidmem are refused (`LOWFILL-REFUSED`) | +| CE copy (`svc_copy`) | dst/src VAs from the push | per-page resolution with pin-agreement (a fresh pin beats a stale ledger entry and repairs it); device-dst copies without a pin-resolved translation are held (`sema_copy_dst_pin_guard`) | +| GRCE inline writes (`LOAD_INLINE_DATA`) | destination VA from the push | once-only identity gate keyed by (push VA, push content hash, dst, bytes), session-scoped and 8192 slots wide | +| GR launch operand staging (`svc_gr_data_fill`) | argument pointers scanned from the QMD | per-page pin/ledger resolution; the resolution is stored in the ledger as the landing authority; dropped pages are named (`GRFILL-DROP`) and demanded | +| Launch copy-out (engine results back to the shadow/guest page) | the launch's own landing records | per-page destination resolution; every write is witnessed (`OUT-PAGE-WIT`/`OUT-PAGE-POST`); staged-identity rule (only the page staged into a slot may receive the slot's content) | +| Release/counter writes (semaphores, completion counters) | guest semaphore VAs from pushes | pool-range writes go through a pin/record-verified cache (`pool-pa`), re-validated against the current pin before the 4-byte write; exact-slot fallback for the client's tracker; forward-only apply | +| GPFIFO/USERD writes (put/get bookkeeping) | channel registration data | the channel's own USERD/instance blocks from the alloc params, aperture-tagged | +| Window/PTE installs (forward PDE install) | the launch's windows | installs are refused loudly when the backend rejects the PDE; the regular path leaves guest VAs in place (no host-side table mutation) | + +## 4. Residual risks and findings + +* **Wrong-but-valid writes remain possible** when every resolver agrees on a + stale page (e.g. a guest freed and reallocated a VA). The r391/r398/r401 + fixes removed the known instances (contiguous fill past the first page, + replayed inline writes, capture-window eviction), but the class is bounded, + not eliminated: the guards convert silent corruption into loud + truncation/refusal where a page is unresolved, and the ledger is + session-scoped and pin-repaired where a fresh pin exists. +* **The boot-corruption lottery** (a zero fill landing on guest page-table + pages) is structurally mitigated by the page-table-shape refusal and the + per-page clamping; no instance has been observed since r391. +* **Diagnostics stay on**: `DST-DIVERGE` (copy-dst ledger vs probe), + `FILL-TRUNC`, `GRFILL-DROP`, `NOWRITE-*` and the stalled-VA demand rows + exist so a future divergence is visible in the boot log rather than + silent. +* **What this audit does not claim**: no isolation or side-channel claim. + GPU Rowhammer itself is a *physical* fault-injection threat that this + software emulation cannot mitigate; the mitigation argument here is only + that the host write paths cannot be steered outside the guest's own + memory, and that wrong targets are bounded by IOMMU-mapped guest RAM or + the emulated shadow. + +## 5. Recommended follow-ups + +1. Keep the copy-destination ledger under the `DST-DIVERGE` witness and + promote it to a bounded hold only if a failing run names it. +2. Extend the pin-demand channel to client device VAs (today the in-guest + scanner resolves only its own push/table windows), so an unresolved + operand page becomes a wait on real data instead of a truncated fill. +3. Re-run this audit when the write-path guards change (any new + `svc_write_ap()` caller with a guest-derived target is in scope by + default). diff --git a/docs/05-operations/06-guest-frame-path-and-looking-glass.md b/docs/05-operations/06-guest-frame-path-and-looking-glass.md new file mode 100644 index 0000000..bc5447a --- /dev/null +++ b/docs/05-operations/06-guest-frame-path-and-looking-glass.md @@ -0,0 +1,102 @@ +# Guest frame path and Looking Glass status (2026-10-21) + +This page records the state of the "show the guest's GPU output on the host" +goal: what is proven, what is built, and exactly which pieces remain. It is +written from live evidence on the vGPU-IO stack (guest CUDA green, ivshmem +devices present, kvmfr module loaded on the host). + +## 1. What is proven + +* **Guest GPU rendering works through CUDA.** `tools/cuda-ab/frame_producer.cu` + renders an animated Julia set on the guest GPU (800x600 RGBA, ~10 fps), + copies the frame to host memory and publishes it into the ivshmem BAR + (`/sys/bus/pci/devices/0000:00:03.0/resource2`, the 128 MiB kvmfr device + QEMU backs with `/dev/shm/kvmfr-<uuid>`), with a small header + (`magic/version/width/height/stride/format/seq/timestamp`) at offset 1 MiB. +* **The host can read those frames.** `tools/lg/frame_reader.py` maps the same + shm file and writes each new sequence number out as a PPM, i.e. a real + GPU-rendered guest frame lands on the host. This is the full transport path + (guest GPU -> guest sysmem -> ivshmem -> host), independent of any display + protocol. +* **The ivshmem devices are visible in the guest** (`lspci`: two + "Inter-VM shared memory" functions; `resource2` of `0000:00:03.0` is the + 128 MiB window, `0000:00:04.0` the 2 MiB cursor window), and the guest can + mmap them (root required for the sysfs resource files). +* **EGL enumerates the NVIDIA device in the guest.** With + `EGL_EXT_platform_device`/`EGL_EXT_device_enumeration` present, + `eglQueryDevicesEXT` returns 3 devices and `eglInitialize` on the NVIDIA + device succeeds (`vendor=NVIDIA version=1.5`). + +## 2. What is blocked, and why + +* **Guest GL contexts do not allocate.** `eglMakeCurrent` on an offscreen + pbuffer fails with `EGL_BAD_ALLOC` (0x3003) on every platform tried + (device, surfaceless, default). Xorg's own failure is the same class: + `(EE) NVIDIA(GPU-0): Failed to allocate VASpace object` then + `Failing initialization of X screen`. The driver cannot allocate GPU virtual + address space for a GL context/surface through the vGPU, so both the X + display path and the EGL offscreen path stop at the same wall. This is the + single remaining blocker for guest OpenGL (and therefore for GZDOOM / + doomsday rendering in the guest). +* **The Looking Glass client (B4) cannot consume our frames directly.** + `looking-glass-client` (installed at `/usr/local/bin/looking-glass-client`, + runs under Xvfb with the `libbfd` shim in `/tmp/opencode/lgcompat`) supports + two transports, LGMP and SPICE. Its `app:shmFile` option still expects an + **LGMP session** (the `KVMFR` session header plus the shared-memory message + queues), not a raw framebuffer. Upstream's frame *producer* (`host/`) is the + Windows DXGI/IDD path. So a Linux guest-side producer must either implement + the LGMP session protocol or we keep the frame reader as the host-side + substitute (which is what `frame_reader.py` is). +* **GZDOOM is not packaged for Ubuntu noble** (`apt-cache policy gzdoom` has + no candidate); `doomsday` + `freedoom` are installed as the fallback, but + doomsday cannot start a renderer without a working GL context (blocker + above). + +## 3. Exact remaining pieces (in order of leverage) + +1. **Fix the guest GL VASpace allocation.** This is an RM/vGPU-side gap: the + driver's `nv_vgpu` VASpace allocation for a GL context fails. Fixing it + unlocks Xorg, EGL offscreen GL, doomsday/GZDOOM, and the standard Looking + Glass guest path (the guest's GL capture driver would then work as + upstream expects). +2. **Implement the LGMP session producer** (or a small Linux capture client) + that writes frames into the ivshmem region with the KVMFR session header + and the frame queue protocol the client expects. The transport half is + already proven; this is protocol work in the guest (or a host-side + substitute that speaks LGMP). +3. **Packaging**: GZDOOM would need a source build or a flatpak/snap; the + doomsday fallback is installed and only blocked by (1). + +## 4. Reproduce + +```bash +# guest (root, because the sysfs resource is root-only): +sudo /tmp/frame_producer 60 800 600 10 # 60 frames, 800x600, 10 fps + +# host: +python3 tools/lg/frame_reader.py --frames 5 --out /tmp/opencode/frames +# -> SAVED seq=N 800x600 fmt=0 -> /tmp/opencode/frames/frame-NNNN.ppm + +# guest EGL probe (shows the enumeration success and the MakeCurrent wall): +gcc -O2 -o egl_frame tools/guest-diag/egl_frame.c -lEGL -lGLESv2 +./egl_frame +``` + +## 5. Session-13 update (2026-10-21, r432→r441) + +* The single-frame path is unchanged and still proven: guest `frame_producer` + (CUDA Julia) -> kvmfr ivshmem BAR -> host `frame_reader.py` -> + `/tmp/opencode/frames/frame-0001.png` (320x240, 1431 colours). +* `tools/lg/frame_sequence.sh` captures an **animated** sequence end to end + (reader first, then N frames from the guest). It is blocked by the + GR no-execute wedge: when the lane is wedged the render launches write zero + pages and the producer republishes stale identical frames (observed: 6/6 + identical MD5s on r438; r439 published 5 identical plus one frame left in + the BAR by the previous boot). A multi-frame animation needs a healthy + lane; the frame pipeline itself is not the blocker. +* The GL wall is untouched: `Xorg :0` -> `(EE) NVIDIA(GPU-0): Failed to + allocate VASpace object`, guest dmesg `NVRM: vaspaceapiConstruct_IMPL: + Could not construct VA space. Status ffff`, i.e. `vmmCreateVaspace` inside + the guest's own RM. Next step: reproduce with the guest RM's + `ResmanDebugLevel` raised (a guest module reload, the service's m177 path) + to capture which inner call returns the generic error. diff --git a/docs/README.md b/docs/README.md index 13f0ae5..27e04e9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -183,6 +183,7 @@ Read in order if you are new. Jump directly if you know what you need. | 23 | [GSP in a RISC-V VM (risc-v-gsp-rm-debugger, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | +| 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min | | 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | | 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | | 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | diff --git a/kernel/gsp_cuda_bridge.c b/kernel/gsp_cuda_bridge.c index 2fac2b2..7adc8e3 100644 --- a/kernel/gsp_cuda_bridge.c +++ b/kernel/gsp_cuda_bridge.c @@ -141,6 +141,9 @@ MODULE_PARM_DESC(cuda_bridge, /* ------------------------------------------------------------------ */ struct cubb_req { + struct nv_vgpu_device *vgpu; /* owning mdev (guest session); the + * queue is FIFO, so only the owner's + * service pass may marshal this job */ u32 h_object; u64 qmd_dev_va; u64 push_va; @@ -178,6 +181,9 @@ static u64 cubb_dedup_hits; struct cubb_pend { bool valid; + struct nv_vgpu_device *vgpu; /* owning mdev: the completion write-back + * must run through this guest's + * address space, never another's */ u64 seq; u32 h_object; u32 opcode; @@ -576,12 +582,15 @@ void gsp_cuda_bridge_launch(struct nv_vgpu_device *vgpu, u32 h_object, if (!gsp_cuda_bridge_enabled()) return; - (void)vgpu; for (i = 0; i < 64; i++) { hash ^= qmd[i]; hash *= 1099511628211ULL; } + /* The owner is part of the identity: two guests' identical launches + * must never dedup against each other. */ + hash ^= (u64)(unsigned long)vgpu; + hash *= 1099511628211ULL; hash ^= (u64)h_object << 32; hash *= 1099511628211ULL; hash ^= qmd_dev_va; @@ -612,6 +621,7 @@ void gsp_cuda_bridge_launch(struct nv_vgpu_device *vgpu, u32 h_object, struct cubb_req *r = &cubb_req[cubb_req_head % CUBB2_REQ_MAX]; + r->vgpu = vgpu; r->h_object = h_object; r->qmd_dev_va = qmd_dev_va; r->push_va = push_va; @@ -972,6 +982,7 @@ static int cubb_marshal_one(struct nv_vgpu_device *vgpu) } job.seq = ++cubb_seq; cubb_pend[slot].valid = true; + cubb_pend[slot].vgpu = r.vgpu; cubb_pend[slot].seq = job.seq; cubb_pend[slot].h_object = r.h_object; cubb_pend[slot].opcode = opcode; @@ -1303,12 +1314,48 @@ static bool cubb_finish_try(struct nv_vgpu_device *vgpu, struct cubb_pend *p, } /* + * Is this mdev still on the driver list with an open VFIO device? Guards the + * bridge's per-request/per-pend owner pointers (dual-guest goal 1.0): a + * removed mdev must never be dereferenced and its queued work can never be + * observed by a live guest. Pointer comparisons only - safe to call with a + * possibly-stale pointer. + */ +static bool cubb_vgpu_open(const struct nv_vgpu_device *vgpu) +{ + struct nv_vgpu_device *d; + bool open = false; + + if (!vgpu) + return false; + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (d == vgpu && atomic_read(&d->vfio_dev_open_count) > 0) { + open = true; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return open; +} + +/* * Service-thread hook: marshal at most one queued launch into a daemon-visible * job, stage any returned done records, then drive staged completions. Called * every service pass; a no-op when no launch has been seen. + * + * Dual-guest (goal 1.0): request and pend records carry their owning mdev + * (the bridge falls back here when forwarding was refused under the + * single-inflight slot). Only the owning guest's pass marshals its job (the + * marshal resolves VAs and reads guest memory through that guest) and only + * the owning pass writes a completion back; a head/record whose owner is + * gone is dropped instead of ever being dereferenced. */ void gsp_cuda_bridge_service(struct nv_vgpu_device *vgpu) { + struct nv_vgpu_device *head_owner = NULL; + unsigned long flags; + u32 head_h = 0; + bool have_head = false; struct cubb2_done d; int guard = 0; int i; @@ -1316,7 +1363,29 @@ void gsp_cuda_bridge_service(struct nv_vgpu_device *vgpu) if (!gsp_cuda_bridge_enabled()) return; - (void)cubb_marshal_one(vgpu); + spin_lock_irqsave(&cubb_req_lock, flags); + if (cubb_req_head != cubb_req_tail) { + struct cubb_req *head = + &cubb_req[cubb_req_tail % CUBB2_REQ_MAX]; + + head_owner = head->vgpu; + head_h = head->h_object; + have_head = true; + } + spin_unlock_irqrestore(&cubb_req_lock, flags); + if (have_head) { + if (head_owner == vgpu) { + (void)cubb_marshal_one(vgpu); + } else if (!cubb_vgpu_open(head_owner)) { + /* The owner's guest is gone: nobody can observe it. */ + cubb_req_drop_head(); + cubb_dropped++; + pr_info_ratelimited("cuda-bridge: dropped a queued launch whose mdev is gone (h=0x%x)\n", + head_h); + } + /* else: another live guest's head; its own pass marshals it. */ + } + while (guard++ < CUBB2_DONE_RING && cubb_done_pop(&d)) { if (d.magic != CUBB2_DONE_MAGIC || d.version != CUBB2_VERSION) { @@ -1330,8 +1399,23 @@ void gsp_cuda_bridge_service(struct nv_vgpu_device *vgpu) mutex_lock(&cubb_lock); for (i = 0; i < CUBB2_PEND_MAX; i++) { - if (cubb_pend[i].valid && cubb_pend[i].have_done && - cubb_finish_try(vgpu, &cubb_pend[i], i)) + struct cubb_pend *p = &cubb_pend[i]; + + if (!p->valid || !p->have_done) + continue; + if (p->vgpu != vgpu) { + /* + * Another guest's staged completion: its own pass + * drives the write-back. A removed owner's record is + * released here so the slot frees. + */ + if (!cubb_vgpu_open(p->vgpu)) { + cubb_pend_release(p, i); + i = -1; /* compaction shifted the entries */ + } + continue; + } + if (cubb_finish_try(vgpu, p, i)) i = -1; /* compaction shifted the entries; rescan */ } mutex_unlock(&cubb_lock); diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 72cda10..734ebf2 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -39,6 +39,7 @@ #include <linux/seq_file.h> #include <linux/uaccess.h> #include <linux/jiffies.h> +#include <linux/ktime.h> #include <linux/timekeeping.h> #include <linux/byteorder/generic.h> #include <linux/pci_regs.h> @@ -73,6 +74,140 @@ module_param_named(gsp_phys_nocat, gsp_phys_nocat_param, bool, 0644); MODULE_PARM_DESC(gsp_phys_nocat, "Log every GSP-RM NOCAT journal record (status-queue function 0x1020) individually during the stage-3e RM-init poll (default 0; the unique-record summary is always accumulated and dumped when the poll ends)"); +/* + * GRFIX FIX3 (relocation coverage, default 0): the QMD/bank rewrite scan + * counts 8-byte words that pass the rewrite's own device-VA shape test + * (>= 0x10000000, < 2^47, 256-byte aligned) but match NO tracked data page + * (req->data[x].va). Such a word is left as an unrelocated guest VA for the + * engine to walk, which is the multi-pointer QMD failure signature. With the + * parameter clear the scan only logs the per-page count (RELOC-GAP, + * rate-limited); with it set a launch carrying any such word is held + * fail-closed before publish (same backend-refusal path as the copy-in hold; + * the sema hold applies the software releases - bounded, never a hang). + * The hold is opt-in because the shape heuristic can false-positive on + * legitimate non-pointer data words; the RELOC-GAP count measures that rate. + */ +static bool gsp_phys_gr_strict_reloc_param; +module_param_named(gsp_phys_gr_strict_reloc, gsp_phys_gr_strict_reloc_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_strict_reloc, + "Hold a GR launch fail-closed when the QMD rewrite finds device-VA-shaped words that match no tracked data page (default 0: RELOC-GAP log only; 1: hold the launch)"); + +/* + * relfix (agent-A FIX A, default 1): the GR QMD rewrite's device-VA-shaped + * words that match NO tracked data page (the RELOC-GAP class - the raw guest + * pointers the real engine then faults on; r216: MMU FAULT_PDE at a gap VA, + * SM warp exceptions) are re-resolved through the sema service's trusted + * resolver (gsp_sema_gr_reloc_resolve). A word that resolves is authored + * into the bounce window exactly like a late-tracked data page - it and the + * contiguous span the resolver also answers are copied in before the + * doorbell and copied back after the retire wait, the same m142 ordering as + * every tracked page - and the word is rewritten to the page's reloc VA plus + * its in-page offset, the exact encoding the matched-word relocation uses. + * Unresolved words keep the historical RELOC-GAP behaviour (counted and left + * as guest VAs; never guessed). 0 = exact pre-relfix behaviour for one-flip + * A/B. The rewrite runs on the single-inflight GR work item with + * gsp_phys_rm_lock held; the resolver is read-only w.r.t. the service tables + * but may sleep in the same vfio walks the fill's resolver uses. + */ +static unsigned int gsp_phys_gr_reloc_resolve_param = 1; +module_param_named(gsp_phys_gr_reloc_resolve, gsp_phys_gr_reloc_resolve_param, + uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_reloc_resolve, + "Resolve QMD words that match no tracked data page and author them as late data pages (default 1: RELOC-FIXED; 0: RELOC-GAP only)"); + +/* + * COPYWIT (2026-10-18 fail4 discrimination tooling, default 1): witness the + * copy-out of every tracked page the engine changed - device VA, guest HPA + * the results were written back to, and the first/last 8 bytes actually + * shipped. Correlating these lines with the sema-bigcopy D2H resolution lines + * (same VA -> pa) and the client's own check discriminates "engine result + * never produced" from "produced but landed on a different guest page than + * the client's read resolves" - the two classes the all-bad client results + * cannot separate from the aggregate counters alone. Bounded per launch + * (GSP_PHYS_GR_COPYWIT_MAX) and rate-limited; 0 disables both this and the + * per-launch RELOC-CENSUS line. + */ +#define GSP_PHYS_GR_COPYWIT_MAX 12u +static bool gsp_phys_gr_copyout_dump_param = true; +module_param_named(gsp_phys_gr_copyout_dump, gsp_phys_gr_copyout_dump_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_copyout_dump, + "Log per-changed-page copy-out witnesses (VA, HPA, first/last 8 bytes) and the per-launch relocation census (default 1)"); + +static inline bool gsp_phys_gr_copywit_enabled(void) +{ + return READ_ONCE(gsp_phys_gr_copyout_dump_param); +} + + +/* + * r405 late-write probes (the r405 "partial write" class: the scale launch + * of streams2 wrote 1 of its 32 expected in-place pages and every other + * launch settled with far fewer changed pages than the kernel should + * produce). Two knobs, both runtime-settable: + * + * - gsp_phys_gr_late_probe (default 0): after the retire wait concludes, + * sleep +500 ms and re-scan the tracked pages. Run-once diagnostic: + * "ENGINE STILL WRITING" proves the copy-out shipped before the stores + * landed (the fix is a longer settle window); "stable" proves the engine + * genuinely stopped, which needs a different treatment. + * - gsp_phys_gr_settle_ms (default 300): the content-stable window the + * partial-changed rule requires before it declares the stores settled. + * Raise it to give a still-writing engine its time (A/B in one cycle). + */ +static bool gsp_phys_gr_late_probe_param; +module_param_named(gsp_phys_gr_late_probe, gsp_phys_gr_late_probe_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_gr_late_probe, + "Diagnostic: after the GR retire wait, re-scan the tracked pages +500 ms later and log whether the engine is still writing (default 0)"); + +static int gsp_phys_gr_settle_ms_param = 300; +module_param_named(gsp_phys_gr_settle_ms, gsp_phys_gr_settle_ms_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_settle_ms, + "GR retire wait: content-stable window (ms) required before a partial changed-page count counts as settled (default 300)"); + +static inline u32 gsp_phys_gr_settle_iters(void) +{ + int ms = READ_ONCE(gsp_phys_gr_settle_ms_param); + + if (ms < 60) + ms = 60; + if (ms > 1400) + ms = 1400; + return (u32)(ms / 5); +} + +/* + * COPYWIT fail4 r224 follow-up: the r224 OUT-PAGE-WIT dumps proved the + * engine's outputs are structurally corrupted (row bands >= 16 replicate a + * single tile per buffer) while the copy-out write/read pairs agree + * byte-for-byte. Two witnesses make the next cycle name the source: + * + * - OUT-ALIAS-WIT: during copy-out, every changed page's first 8 bytes are + * matched against the earlier changed pages of the SAME launch; a hit + * logs the (va, bi) pair, i.e. the replicated-page structure and the rec + * partners - the direct evidence for "one tile replicated across the + * band" and against "pass-through of pre-launch content" (the pre/post + * csums in OUT-PAGE-WIT prove the engine changed the page). + * - IN-ALIAS-WIT: during the copy-in fill, consecutive recs with identical + * content (equal per-page csum) are logged - a replicated INPUT buffer + * (e.g. an X or weight region whose H2D/copy-in served one page many + * times) is caught here before any launch runs. + * Bounded boot-wide and per launch; same single-inflight static discipline + * as the other submit scratch. + */ +#define GSP_PHYS_GR_OUTWIT_SLOTS 16u +static u64 gsp_phys_gr_outwit_first8[GSP_PHYS_GR_OUTWIT_SLOTS]; +static u64 gsp_phys_gr_outwit_va[GSP_PHYS_GR_OUTWIT_SLOTS]; +static u32 gsp_phys_gr_outwit_bi[GSP_PHYS_GR_OUTWIT_SLOTS]; +static u32 gsp_phys_gr_outwit_n; +static u32 gsp_phys_gr_alias_log; /* boot-wide alias-line budget */ +static u32 gsp_phys_gr_recdump_budget; /* r233 REC-DUMP boot budget */ +static u32 gsp_phys_gr_rdw_budget; /* r234 RDW arming-line budget */ +static u32 gsp_phys_gr_atrace_budget; /* r234 A-TRACE boot budget */ + bool gsp_phys_boot_enabled(void) { return gsp_phys_boot_param; @@ -999,16 +1134,34 @@ bool gsp_phys_boot_requested(void) #define GSP_PHYS_RMCALL_VA_DST (GSP_PHYS_RMCALL_VA_BASE + 3 * GSP_PHYS_PAGE_SIZE) #define GSP_PHYS_RMCALL_VA_SEM (GSP_PHYS_RMCALL_VA_BASE + 4 * GSP_PHYS_PAGE_SIZE) /* - * m69: window-0 shadow pool - 8 module pages at VAs 0x108000..0x10f000 - * (small-PT slots 0x108..0x10f, the same 128 B PTE fetch line as the - * GPFIFO/PB/QMDWIN slots, planted before the engine's first fetch). The - * Tier-2 copies of the QMD's program/param pages are placed here so the GR - * engine never has to read through the window-1 scratch VA range whose PT - * chain (DMA_UPDATE_PDE_2 pdeIndex=1) has never been proven engine-readable. + * m69: window-0 shadow pool - 16 module pages holding the Tier-2 copies of + * the QMD's program/param pages, so the GR engine never has to read through + * the window-1 scratch VA range whose PT chain (DMA_UPDATE_PDE_2 + * pdeIndex=1) has never been proven engine-readable. The pool is split + * into two 8-page banks; the forwarded launch alternates banks per launch + * (m151) so one launch's copy-in can never clobber the pages the previous + * launch's engine fetches may still be reading (late-fetch safety). + * + * The base MUST stay clear of the m114/m150 GR data bounce window + * (GSP_PHYS_RMCALL_GR_BOUNCE_VA0 .. +PAGES, main-PT slots 0x110..0x1df): + * this pool is re-planted on every gsp_phys_rmcall_pdb_build() while the + * bounce PTEs are a one-shot plant, so any overlap lets the later win0 + * plant steal the bounce slots and the engine writes results into the + * shadow pages, losing them. 0x1e0000 (slots 0x1e0..0x1ef) sits directly + * above the last bounce/mail/marker page (0x1df000) and below the + * mail-injection slot 0x1f0 and the guest scratch window (0x200000). Keep + * it that way (see the static_assert next to the bounce constants). */ #define GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE \ - (GSP_PHYS_RMCALL_VA_BASE + 0x90000ULL) -#define GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS 8u + (GSP_PHYS_RMCALL_VA_BASE + 0xE0000ULL) +#define GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS 16u +/* + * m151: one bank = one launch's worth of QMD shadow pages. A forwarded + * launch consumes at most eight pages (param via qmd_word 32, program 48, + * bank 0/1/2/3/5/7 at qmd_word 34..46 even); the per-launch bounds check + * uses this so a launch can never spill into the other bank. + */ +#define GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS 8u /* * m99: engine-tree branches for identity-mapping the forwarded kernel's * guest data pages. The walker uses our root[0] -> PD2 -> PD1 -> PD0 dual -> @@ -1043,6 +1196,16 @@ bool gsp_phys_boot_requested(void) * engine's first fetch of the line - and never at launch time. */ #define GSP_PHYS_RMCALL_VA_GR_QMDWIN (GSP_PHYS_RMCALL_VA_BASE + 0x7000) +/* + * m156 (E3): double-buffer the inline-QMD destination. The page is + * prewritten every launch; when a single VA is reused the engine's QMD-unit + * look-ahead reads of the previous launch may still be in flight against the + * rewrite (the rationale the m151 win0 banks give). Slot 0x108 sits in the + * same already-proven 128 B PTE fetch line as 0x107 and is otherwise unused; + * launches alternate 0x107000/0x108000 by submission epoch. Both PTEs are + * planted once at stage 3 (no invalidate path exists). + */ +#define GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS 2u /* * P6a guest-memory windows (gsp_phys_guest_mem=1). Two more slots in the same @@ -1100,6 +1263,33 @@ bool gsp_phys_boot_requested(void) /* Full 128 B fetch lines per 4 KB small-PT page: (4096/8)/16 = 32. */ #define GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE \ ((GSP_PHYS_PAGE_SIZE / 8u) / GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES) +/* + * m152: extended scratch-PT pool. The original 16 x 2 MB PD0 windows + * (512 never-reused 128 B fetch lines per boot) exhausted under a normal + * launch mix (each distinct guest page consumed a line per launch) and the + * exhaustion was permanent for the boot: every later launch that needed a + * new mapping was refused ("launch refused ... no Tier-1 mapping and no + * captured inline data"), which is the deterministic once-broken-forever + * failure the longevity runs kept hitting. Two changes fix the class: + * (a) GR alias cache (below): a consumed fetch line is keyed by the HPA + * it maps, so a repeat launch of the same buffers (or the same chunk + * page / semaphore page) reuses the identical never-rewritten line and + * consumes nothing; + * (b) a 48-window extension pool right above the ladder's free gap + * (window VA 0x04000000.., PD0 indices 32..79) backed by one 48-page + * PT chunk, raising capacity to 64 windows x 32 lines = 2048. + * lines/boot. Both are pure capacity mechanics: no PTE and no mapping any + * engine may have fetched is ever rewritten. + */ +#define GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE 0x04000000ULL +#define GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES 48u +#define GSP_PHYS_RMCALL_GUEST_PT_TOTAL_PAGES \ + (GSP_PHYS_RMCALL_GUEST_PT_PAGES + GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES) +#define GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES \ + (GSP_PHYS_RMCALL_GUEST_PT_TOTAL_PAGES * \ + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) +/* HPA -> shadow-VA reuse cache; hit = no new fetch line is consumed. */ +#define GSP_PHYS_RMCALL_GR_ALIAS_CACHE 768u /* * GR forwarding (gsp_phys_gr_forward=1; default off) scratch resources, @@ -1253,8 +1443,21 @@ bool gsp_phys_boot_requested(void) #define GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE \ (u32)(GSP_PHYS_RMCALL_CHANNEL_FLAG_PRIVILEGED | (4u << 8) | \ (1u << 21)) -/* Captured user-lane GPFIFO depth (#825: 1024 entries). */ -#define GSP_PHYS_RMCALL_USER_LANE_GPFIFO_ENTRIES 1024u +/* + * User-lane GPFIFO depth. MUST equal the host ring's push modulus + * (GSP_PHYS_RMCALL_GPFIFO_ENTRIES, 8): the host publishes into the + * one-page ladder ring modulo 8, so if the channel declares a different + * count the engine's GET/PUT wrap desyncs at the host's first wrap + * (GP_PUT 7 -> 0 while the engine's GET keeps counting to 8) and every + * later launch on the boot is never fetched (r138: launches 1..7 green, + * the 8th and all after it silent with `qmd_rel=5 ch_rel=0`). The captured + * #825 value (1024) describes the real client's own much larger ring, not + * this emulated one; the campaign's 2026-10-06 "fidelity" change to 1024 + * was a wrap-only difference that broke exactly here (M1's two-launch + * acceptance never wrapped). Keep the two ends equal. + */ +#define GSP_PHYS_RMCALL_USER_LANE_GPFIFO_ENTRIES \ + GSP_PHYS_RMCALL_GPFIFO_ENTRIES #define GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE 0x1du #define GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE 0x2000u #define GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE 0xd100u @@ -1379,6 +1582,58 @@ enum gsp_phys_rmcall_buf { GSP_PHYS_RMCALL_BUF_GUEST_PT13, GSP_PHYS_RMCALL_BUF_GUEST_PT14, GSP_PHYS_RMCALL_BUF_GUEST_PT15, + /* m152: extension pool PT pages: 48 separate single-page chunks + * (GUEST_PT_EXT0..47). The server only accepts single-page memdescs + * for client buffers (a 48-page chunk was refused with rpc_result + * 0x1f + a firmware ASSERT in stage 2). */ + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT0, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT1, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT2, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT3, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT4, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT5, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT6, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT7, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT8, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT9, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT10, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT11, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT12, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT13, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT14, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT15, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT16, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT17, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT18, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT19, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT20, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT21, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT22, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT23, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT24, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT25, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT26, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT27, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT28, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT29, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT30, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT31, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT32, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT33, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT34, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT35, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT36, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT37, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT38, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT39, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT40, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT41, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT42, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT43, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT44, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT45, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT46, + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT47, /* * Golden9g: the three client page-table levels the real golden VAS * hands to the server with VASPACE_COPY_SERVER_RESERVED_PDES @@ -1409,6 +1664,27 @@ enum gsp_phys_rmcall_buf { * generous and progress is logged every ~500 ms. */ #define GSP_PHYS_RMCALL_GR_POLL_MS 5000u +/* + * Bounded retry of a forwarded GR launch whose QMD did not run. A green + * launch completes in ~170-195 ms; when the engine drops it (cold context + * save/unload, or an entry the PBDMA never fetched) `qmd_rel` never reaches + * the wanted value and the client would fall back to the software release + * and read stale/zero results. Because the timeout path never copies the + * result out to the guest, a launch that timed out did not reach the + * client: re-publishing a FRESH ring entry (same PB/QMD) and re-issuing the + * schedule+doorbell gives the front-end another chance and the guest still + * sees only the success path. Bounded so the service thread is never held + * past the existing hold budget. + */ +#define GSP_PHYS_RMCALL_GR_RETRY_MS 2000u +/* + * F2: the short re-poll after the first-timeout fast re-arm (extended + * re-arm + fresh ring entry). The engine ack for a recovered launch is + * observed within tens of ms (r198); 1500 ms covers the ctxsw path without + * burning another full window before the historical ladder continues. + */ +#define GSP_PHYS_RMCALL_GR_FAST_RETRY_MS 1500u +#define GSP_PHYS_RMCALL_GR_ATTEMPTS 3u /* P6a guest-memory CE copies: same bounded poll cap as the P5 shadow path. */ #define GSP_PHYS_RMCALL_GUEST_POLL_MS 200u @@ -2136,17 +2412,31 @@ static struct gsp_phys_boot_stage { bool rmcall_gr_exec_ok; /* a real GR submission proved out */ u32 rmcall_gr_submit_epoch; /* per-submission mapper epoch */ u32 rmcall_gr_pt_used; /* Tier-1 window pool slots used */ - u32 rmcall_guest_pt_win_mask; /* Tier-2 scratch windows installed */ + u64 rmcall_guest_pt_win_mask; /* Tier-2 scratch windows installed */ + /* + * m152: shadow alias reuse cache: hpa -> never-rewritten shadow VA. + * Populated by gsp_phys_rmcall_gr_map_shadow_locked; a hit returns the + * existing line so repeat launches of the same physical pages consume + * no new fetch line. Fill-order/LRU-free by design: entries are dead + * weight once a client exits, but a boot has 2048 lines to give and + * the cache only ever returns identical PTE content. + */ + u64 rmcall_gr_alias_hpa[GSP_PHYS_RMCALL_GR_ALIAS_CACHE]; + u64 rmcall_gr_alias_va[GSP_PHYS_RMCALL_GR_ALIAS_CACHE]; + u32 rmcall_gr_alias_count; struct gsp_phys_boot_chunk rmcall_gr_pt[GSP_PHYS_RMCALL_GR_PT_PAGES]; struct gsp_phys_boot_chunk rmcall_gr_shadow; /* Tier-2 pages */ /* * m69: window-0 shadow pool (gsp_phys_rmcall_gr_shadow_win0=1). - * Eight module pages whose PTEs are planted in the proven window-0 - * PT line at slots 0x108..0x10f (VAs 0x108000..0x10f000) by - * gsp_phys_rmcall_pdb_build, before the engine's first fetch (same - * pattern as the qmdwin slot). The launch copies the QMD's - * program/param page content into them and rewrites the QMD pointers - * there, so the GR engine never reads the window-1 scratch range. + * Sixteen module pages (two 8-page banks; m151 alternates the bank + * per launch so a launch's copy-in cannot clobber pages the previous + * launch's engine fetches may still be reading) whose PTEs are + * planted in the proven window-0 PT (slots derive from + * GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE) by gsp_phys_rmcall_pdb_build, + * before the engine's first fetch (same pattern as the qmdwin slot). + * The launch copies the QMD's program/param page content into them + * and rewrites the QMD pointers there, so the GR engine never reads + * the window-1 scratch range. */ struct gsp_phys_boot_chunk rmcall_gr_shadow_win0; u32 rmcall_gr_shadow_win0_used; @@ -2161,19 +2451,114 @@ static struct gsp_phys_boot_stage { /* m104: module page holding the injected mail_kernel message. */ struct gsp_phys_boot_chunk rmcall_gr_mail; /* - * m114: pre-commit bounce window. 112 pages planted into the main - * small-PT at slots 0x110..0x17f during stage 3 (the proven win0 - * pattern: the SM walker resolves the commit-time tree only). The - * launch copies the kernel's data pages in, rewrites the pointers to - * these VAs (reloc base 0x110000), and copies the results back after - * the release; page 100 (VA 0x174000) carries the mail message. + * m114: pre-commit bounce window, planted into the main small-PT at + * slots 0x110..0x1df during stage 3 (the proven win0 pattern: the SM + * walker resolves the commit-time tree only). The launch copies the + * kernel's data pages in, rewrites the pointers to these VAs (reloc + * base 0x110000), and copies the results back after the release. + * B-2: with the grown data region on (gsp_phys_gr_data_region=1, the + * default) this window no longer carries operand data - only the mail + * slot and marker - but it stays allocated so the mail injection and + * the copy diagnostics are unchanged, and it remains the whole data + * region with the parameter off. + * + * The window is 208 pages so the data region below the mail page can + * carry the operand pages of kernels whose buffers exceed the old + * 128 KB (victory_matmul's 256 KB buffers need 64 pages each; three + * of them plus argument/bank candidates exceed the old 100-page / + * 33-pages-per-pointer caps and the engine then ran on truncated + * windows). The mail page is the LAST data-free page (slot 206) and + * the m132 marker sits at 207; the main PT page deliberately covers + * the whole 0x110000..0x1df000 range with nothing else planted there. */ struct gsp_phys_boot_chunk rmcall_gr_bounce; u32 rmcall_gr_bounce_used; -#define GSP_PHYS_RMCALL_GR_BOUNCE_PAGES 112u + /* + * B-2: grown boot-fixed data-bounce region (gsp_phys_gr_data_region=1, + * the default; see the constants in gsp_phys_booter.h). One 16384-page + * (64 MiB) data chunk plus GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES 4 KB + * PT pages (one per 2 MB window), all planted once at stage 3 before + * any engine fetch. The old 208-page window above keeps its mail slot + * 206 and marker 207 exactly as-is (the mail injection and copy + * diagnostics rely on them); when the grown region is on it no longer + * carries operand data. + */ + struct gsp_phys_boot_chunk rmcall_gr_data_bounce; + struct gsp_phys_boot_chunk rmcall_gr_data_bounce_pt; +#define GSP_PHYS_RMCALL_GR_BOUNCE_PAGES 208u #define GSP_PHYS_RMCALL_GR_BOUNCE_SLOT0 0x110u #define GSP_PHYS_RMCALL_GR_BOUNCE_VA0 0x110000ULL -#define GSP_PHYS_RMCALL_GR_MAIL_SLOT 100u +#define GSP_PHYS_RMCALL_GR_MAIL_SLOT 206u +/* + * m150b: the win0 QMD shadow pool must never overlap the bounce window. + * The win0 plant runs on every pdb_build() (unconditional) and the bounce + * plant is one-shot, so an overlap silently hijacks bounce PTEs and the + * engine's results land in the shadow pages (victory_matmul lost C rows + * 0..31 exactly this way before 0x1e0000 moved the pool clear). + */ +static_assert(GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS * GSP_PHYS_PAGE_SIZE <= + GSP_PHYS_RMCALL_GR_BOUNCE_VA0 || + GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE >= + GSP_PHYS_RMCALL_GR_BOUNCE_VA0 + + (u64)GSP_PHYS_RMCALL_GR_BOUNCE_PAGES * GSP_PHYS_PAGE_SIZE, + "win0 shadow pool must not overlap the GR data bounce window"); +/* + * m151: and the two-bank pool must end below the reserved mail-injection + * slot 0x1f0 (VA 0x1f0000) so the extra slots never steal that PTE. + */ +static_assert(GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS * GSP_PHYS_PAGE_SIZE <= + GSP_PHYS_RMCALL_VA_BASE + 0xF0000ULL, + "win0 shadow pool must end below the reserved mail-injection slot"); +static_assert(GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS == + 2u * GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS, + "win0 shadow pool must be exactly two banks"); +/* + * B-2: the grown data-bounce region is a hard VA-layout contract. One fresh + * PT page per 2 MB window; PD0 slots immediately above the m152 extension + * pool's 32..79 under the same root/PD2/PD1 chain as the ladder; the whole + * region below the non-shared VAS vaSize (a non-shared VAS addresses PD0 + * entries 0..127 = 256 MB, so the region's 64 MB is a 32 MB-slack fit); and + * disjoint from every other planted window in the PDB (old bounce 0x110000.. + * 0x1dffff, win0 pool 0x1e0000.., guest scratch windows 0x200000..). Any + * future move of VA0/SLOT0/size fails at compile time. + */ +static_assert(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES == + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES * 512u, + "the grown data-bounce region must be exactly one PT page per 2 MB window"); +static_assert(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 == + (u64)GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0 * (2ULL << 20), + "the grown data-bounce VA0 must equal its PD0 slot base"); +static_assert(((GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >> 47) & 0x3) == 0 && + ((GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >> 38) & 0x1ff) == 0 && + ((GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >> 29) & 0x1ff) == 0 && + ((GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >> 21) & 0xff) == + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0 && + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0 + + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES <= 128, + "the grown data-bounce region must use PD0 entries 80..111 under the ladder's root/PD2/PD1 chain"); +static_assert(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + (u64)GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES * + GSP_PHYS_PAGE_SIZE <= + GSP_PHYS_RMCALL_VAS_SIZE, + "the grown data-bounce region must stay below the VAS vaSize limit"); +static_assert(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >= + GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE + + (u64)GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES * (2ULL << 20), + "the grown data-bounce region must start above the m152 extension scratch pool"); +static_assert(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >= + GSP_PHYS_RMCALL_GR_BOUNCE_VA0 + + (u64)GSP_PHYS_RMCALL_GR_BOUNCE_PAGES * + GSP_PHYS_PAGE_SIZE && + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >= + GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS * + GSP_PHYS_PAGE_SIZE && + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 >= + GSP_PHYS_RMCALL_VA_GUEST_PT_BASE + + (u64)GSP_PHYS_RMCALL_GUEST_PT_PAGES * (2ULL << 20), + "the grown data-bounce region must be disjoint from the old bounce window, the win0 pool and the guest scratch window"); /* m99: guest data identity-map table branches (see the PD*_MAX block). */ struct gsp_phys_boot_chunk rmcall_gr_data_pd1[GSP_PHYS_RMCALL_GR_DATA_PD1_MAX]; @@ -2813,6 +3198,7 @@ static void gsp_phys_rmcall_reset(void) boot_stage.rmcall_gr_submit_epoch = 0; boot_stage.rmcall_gr_pt_used = 0; boot_stage.rmcall_guest_pt_win_mask = 0; + boot_stage.rmcall_gr_alias_count = 0; memset(boot_stage.rmcall_gr_pt, 0, sizeof(boot_stage.rmcall_gr_pt)); memset(&boot_stage.rmcall_gr_shadow, 0, sizeof(boot_stage.rmcall_gr_shadow)); @@ -6442,6 +6828,35 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); /* + * Mode-35 KGR static-data snapshots. The user-lane query set + * (gsp_phys_rmcall_user_lane_kgr_queries) is answered by the LIVE GSP with + * the device's true GR static data; the replies are kept here so the + * guest-side emulation can serve the identical bytes instead of fabricating + * GR values. The concrete consumer is the X driver's 3D-engine selection: + * nv3dAllocDevice maps GR_GET_INFO index SM_VERSION (0x0c) through a table + * and fails with "Failed to select a 3D engine" when it reads back as + * SM_VERSION_NONE (0) - the fabricated fill never set it, while the live + * GSP's KGR_GET_INFO reply carries the real value (0x75 for TU104) along + * with every other GR static field (warps, caps, topology, ctx buffers). + * + * One entry per captured cmd; the generation key ties the cache to the + * backend session exactly like gsp_phys_backend_static above, and the + * getter refuses entries once the session has moved on. + */ +#define GSP_PHYS_KGR_SNAP_ENTRIES 10 +#define GSP_PHYS_KGR_SNAP_MAX_SIZE 3712u /* KGR_GET_INFO: 8 engines x 58 x 8 */ + +struct gsp_phys_kgr_snap { + u32 cmd; + u32 size; + bool valid; + u32 generation; + u8 data[GSP_PHYS_KGR_SNAP_MAX_SIZE]; +}; + +static struct gsp_phys_kgr_snap gsp_phys_kgr_snaps[GSP_PHYS_KGR_SNAP_ENTRIES]; + +/* * Drop the cached static-config snapshot. Called when the backend session * leaves READY (or a new claim starts): the cache generation key is bumped * past the current backend generation so a concurrent getter cannot serve the @@ -6450,9 +6865,13 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); */ void gsp_phys_backend_static_invalidate(void) { + unsigned int i; + mutex_lock(&gsp_phys_rm_lock); gsp_phys_backend_static.valid = false; gsp_phys_backend_static.generation = gsp_phys_backend_generation() + 1; + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) + gsp_phys_kgr_snaps[i].valid = false; mutex_unlock(&gsp_phys_rm_lock); } @@ -6506,6 +6925,71 @@ int gsp_phys_backend_static_get(struct gsp_phys_backend_static_config *out) } EXPORT_SYMBOL_GPL(gsp_phys_backend_static_get); +/* + * Copy the captured reply for cmd into dst. Returns 0 on success (with + * *out_len set), -ENODATA when nothing was captured for cmd (or the entry's + * generation no longer matches the backend session), -ENOSPC when the + * snapshot does not fit dst_len, -EAGAIN while the backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len) +{ + unsigned int i; + + if (!dst || !out_len) + return -EINVAL; + + if (gsp_phys_backend_state() != GSP_PHYS_BACKEND_READY) + return -EAGAIN; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + struct gsp_phys_kgr_snap *s = &gsp_phys_kgr_snaps[i]; + + if (s->cmd != cmd) + continue; + if (!s->valid || + s->generation != gsp_phys_backend_generation()) + return -ENODATA; + if (s->size > dst_len) + return -ENOSPC; + memcpy(dst, s->data, s->size); + *out_len = s->size; + return 0; + } + return -ENODATA; +} +EXPORT_SYMBOL_GPL(gsp_phys_kgr_snap_get); + +/* + * Record one mode-35 KGR query reply. Called under the boot thread only + * (same thread as the query itself), so no lock is needed beyond the + * generation key. + */ +static void gsp_phys_kgr_snap_store(u32 cmd, const void *data, u32 size) +{ + unsigned int i; + struct gsp_phys_kgr_snap *slot = NULL; + + if (size > GSP_PHYS_KGR_SNAP_MAX_SIZE) + return; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + if (gsp_phys_kgr_snaps[i].cmd == cmd) { + slot = &gsp_phys_kgr_snaps[i]; + break; + } + if (!slot && !gsp_phys_kgr_snaps[i].valid) + slot = &gsp_phys_kgr_snaps[i]; + } + if (!slot) + return; + + slot->cmd = cmd; + slot->size = size; + slot->generation = gsp_phys_backend_generation(); + memcpy(slot->data, data, size); + slot->valid = true; +} + /* ============================================================ * Host RM client ladder (R1): gsp-phys-rm-call * ============================================================ */ @@ -7215,6 +7699,65 @@ static void gsp_phys_rmcall_ctx_diag_dump(const char *what, static void gsp_phys_rmcall_ctx_diag_decode(const char *tag); /* + * m152: scratch-PT pool geometry helpers. Pool local page P (0-based across + * the original 16-window region then the 48-window extension) maps to a PD0 + * window VA and to a page of either the per-window single-page PT chunks + * (P < 16) or the multi-page extension chunk (P >= 16). Keep every + * consumer on these two helpers so the two regions can never drift. + */ +static u64 gsp_phys_rmcall_guest_pt_win_va(u32 page) +{ + if (page < GSP_PHYS_RMCALL_GUEST_PT_PAGES) + return boot_stage.rmcall_va_guest_pt + ((u64)page << 21); + return GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE + + ((u64)(page - GSP_PHYS_RMCALL_GUEST_PT_PAGES) << 21); +} + +static u8 *gsp_phys_rmcall_guest_pt_page_va(u32 page) +{ + if (page < GSP_PHYS_RMCALL_GUEST_PT_PAGES) + return gsp_phys_chunk_page_va( + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GUEST_PT0 + + page], 0); + if (page < GSP_PHYS_RMCALL_GUEST_PT_TOTAL_PAGES) + return gsp_phys_chunk_page_va( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT0 + + (page - GSP_PHYS_RMCALL_GUEST_PT_PAGES)], 0); + return NULL; +} + +static u64 gsp_phys_rmcall_guest_pt_page_dma(u32 page) +{ + if (page < GSP_PHYS_RMCALL_GUEST_PT_PAGES) + return (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GUEST_PT0 + + page], 0); + return (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT0 + + (page - GSP_PHYS_RMCALL_GUEST_PT_PAGES)], 0); +} + +/* + * PD0 index -> pool local page, or 0xffffffffu when the index is not a + * scratch-pool window. Consults both pool regions. + */ +static u32 gsp_phys_rmcall_guest_pt_page_of_pde(u32 pde_index) +{ + u32 base = (u32)(boot_stage.rmcall_va_guest_pt >> 21); + + if (pde_index >= base && + pde_index < base + GSP_PHYS_RMCALL_GUEST_PT_PAGES) + return pde_index - base; + base = (u32)(GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE >> 21); + if (pde_index >= base && + pde_index < base + GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES) + return pde_index - base + GSP_PHYS_RMCALL_GUEST_PT_PAGES; + return 0xffffffffu; +} + +/* * Shared-management VAS: install the ladder's page-table window through the * server's own MMU walker. NV0080_CTRL_CMD_DMA_UPDATE_PDE_2 (0x80180f, * gpu_vaspace.c gvaspaceUpdatePde2_IMPL) takes the system-physical base of a @@ -9182,14 +9725,68 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0) * (swref/published/pascal/gp100/dev_mmu.h:97-119,157; * uvm_pascal_mmu.c:64-141,235-303). */ -static u64 gsp_phys_rmcall_pde(u64 pa) +/* + * r431/r436 COHERENCY ROOT-CAUSE CANDIDATE: the leaf-PTE aperture for every + * host page this backend maps (the ladder windows, the win0 shadow pool, the + * bounce/data regions, the release/scratch shadows) was hardcoded to + * aperture 3 = SYSTEM_NON_COHERENT_MEMORY. The reference RM derives the + * pair from the allocation's CPU cache attribute (fabric_vaspace.c:1123-1134 + * for the aperture, mem_desc.c memdescGetVolatility for VOL: NV_MEMORY_CACHED + * -> SYSTEM_COHERENT (2) + volatile FALSE, NV_MEMORY_UNCACHED -> NON_COHERENT + * (3) + volatile TRUE). The r431 experiment used aperture 2 with VOL still 1 + * (0xd) - a pair the reference RM never produces - and it made the engine's + * truncated writes visible sooner but worsened the wedge on a fresh boot. + * r436 turns the knob into the full 2x2 matrix so each consistent encoding + * can be A/B'd: + * 0 = aperture 3 + VOL 1 (0xf) - historical, NV_MEMORY_UNCACHED + * 1 = aperture 2 + VOL 1 (0xd) - r431 experiment (inconsistent pair) + * 2 = aperture 2 + VOL 0 (0x5) - NV_MEMORY_CACHED (coherent, non-volatile) + * 3 = aperture 3 + VOL 0 (0x7) - non-coherent, non-volatile + * The PTEs are planted before the engine's first fetch, so the flag is + * boot-time only. + */ +static int gsp_phys_rmcall_pte_coh_param; +module_param_named(gsp_phys_rmcall_pte_coh, gsp_phys_rmcall_pte_coh_param, + int, 0444); +MODULE_PARM_DESC(gsp_phys_rmcall_pte_coh, + "r436: sysmem leaf-PTE aperture/VOL mode: 0=0xf (aperture3+VOL, historical), 1=0xd (aperture2+VOL, r431), 2=0x5 (aperture2, non-volatile), 3=0x7 (aperture3, non-volatile)"); + +static u64 gsp_phys_rmcall_sysmem_base_low(void) { - return 0xeull | ((pa >> 12) << 8); + switch (gsp_phys_rmcall_pte_coh_param) { + case 1: + return 0xdull; /* aperture 2, VOL 1 */ + case 2: + return 0x5ull; /* aperture 2, VOL 0 */ + case 3: + return 0x7ull; /* aperture 3, VOL 0 */ + default: + return 0xfull; /* aperture 3, VOL 1 (historical) */ + } } static u64 gsp_phys_rmcall_pte(u64 pa) { - return 0xfull | ((pa >> 12) << 8); + /* + * Low nibble: VALID (1) | APERTURE<<1 | VOL (8). Aperture 2 = + * SYSTEM_COHERENT_MEMORY -> 1|4|8 = 0xd; aperture 3 = NON_COHERENT + * -> 1|6|8 = 0xf. r436: see gsp_phys_rmcall_pte_coh_param. + */ + u64 base = gsp_phys_rmcall_sysmem_base_low(); + + return base | ((pa >> 12) << 8); +} + +/* + * r436: the single PDE uses the same aperture/VOL pair as the leaf PTEs but + * with IS_PDE=0 in bit 0 (the low bit of the PTE base is VALID there), so the + * PDE base is the PTE base with bit 0 cleared: 0xe/0xc/0x4/0x6. + */ +static u64 gsp_phys_rmcall_pde(u64 pa) +{ + u64 base = gsp_phys_rmcall_sysmem_base_low() & ~1ull; + + return base | ((pa >> 12) << 8); } /* @@ -9336,35 +9933,39 @@ static void gsp_phys_rmcall_pdb_build(void) &boot_stage.rmcall_gr_qmdwin; u32 qw_idx = (u32)((GSP_PHYS_RMCALL_VA_GR_QMDWIN >> 12) & 0x1ffu); + u32 qw_s; if (!qw->pages) { if (!boot_stage.pdev) { pr_warn("vgsp-boot: rm-call: qmdwin plant skipped: no claimed pci_dev; launches keep the guest QMD destination\n"); - } else if (!gsp_phys_chunk_alloc(boot_stage.pdev, qw, 1u)) { + } else if (!gsp_phys_chunk_alloc(boot_stage.pdev, qw, + GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS)) { gsp_phys_chunk_zero(qw); } else { pr_warn("vgsp-boot: rm-call: qmdwin page allocation failed; launches keep the guest QMD destination\n"); } } - if (qw->pages) { - put_unaligned_le64(gsp_phys_rmcall_pte( - (u64)gsp_phys_chunk_page_dma(qw, 0)), - (u8 *)gsp_phys_chunk_page_va(pt, 0) + - (size_t)qw_idx * 8); + if (qw->pages >= GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS) { + for (qw_s = 0; qw_s < GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS; + qw_s++) + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma(qw, + qw_s)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)(qw_idx + qw_s) * 8); if (!boot_stage.rmcall_gr_qmdwin_va) - pr_info("vgsp-boot: rm-call: qmdwin module page planted VA=0x%llx ptSlot=0x%x pte=0x%016llx dma=0x%llx (before the engine's first fetch)\n", - GSP_PHYS_RMCALL_VA_GR_QMDWIN, qw_idx, - get_unaligned_le64( - (u8 *)gsp_phys_chunk_page_va( - pt, 0) + - (size_t)qw_idx * 8), - (u64)gsp_phys_chunk_page_dma(qw, 0)); + pr_info("vgsp-boot: rm-call: qmdwin module pages planted VA=0x%llx..0x%llx ptSlots=0x%x/0x%x (double-buffer, before the engine's first fetch)\n", + GSP_PHYS_RMCALL_VA_GR_QMDWIN, + GSP_PHYS_RMCALL_VA_GR_QMDWIN + + (GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS - 1u) * + GSP_PHYS_PAGE_SIZE, + qw_idx, qw_idx + 1u); boot_stage.rmcall_gr_qmdwin_va = GSP_PHYS_RMCALL_VA_GR_QMDWIN; } } /* - * m69: plant the window-0 shadow pool PTEs (slots 0x108..0x10f) in the + * m69: plant the window-0 shadow pool PTEs (current win0 slot range) in the * same proven small-PT line, before the engine's first fetch. The * launch copies the QMD's program/param page content into these pages * and rewrites the QMD pointers to these VAs, so the GR engine never @@ -9410,12 +10011,18 @@ static void gsp_phys_rmcall_pdb_build(void) (size_t)idx * 8); } dma_wmb(); - pr_info("vgsp-boot: rm-call: shadow-win0 pool planted %u pages at VA 0x%llx..0x%llx (small-PT slots 0x108..0x10f, before first fetch)\n", + pr_info("vgsp-boot: rm-call: shadow-win0 pool planted %u pages at VA 0x%llx..0x%llx (small-PT slots 0x%x..0x%x, before first fetch)\n", GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS, GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE, GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + (GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS - 1) * - GSP_PHYS_PAGE_SIZE); + GSP_PHYS_PAGE_SIZE, + (u32)((GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE >> 12) & + 0x1ffu), + (u32)(((GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS - 1) * + GSP_PHYS_PAGE_SIZE) >> 12) & + 0x1ffu)); } } /* @@ -9468,6 +10075,77 @@ static void gsp_phys_rmcall_pdb_build(void) pr_err("vgsp-boot: rm-call: bounce window allocation failed\n"); } } + /* + * B-2: grown data-bounce region (gsp_phys_gr_data_region=1, default). + * One 16384-page (64 MiB) data chunk plus one fresh 4 KB PT page per + * 2 MB window; PD0 dual entries 80..111 in the client PDB (immediately + * above the m152 extension pool's 32..79), every leaf PTE pre-written + * with the chunk page's DMA address. The whole map is final at stage 3 + * - before the engine's first fetch - so the 128 B PTE fetch-line + * granule is fully populated at plant time (the GMMU has no invalidate + * path; a runtime PTE write would be invisible). One-shot like the + * bounce plant above: no later pdb_build() call re-zeroes the PD0 page, + * and the region's other writers (ext pool, ctx, win0) target disjoint + * slots. Planted only when the parameter is on; with 0 the SVC and the + * copy path stay on the legacy 205-page window. + */ + if (gsp_phys_gr_data_region_value() && + (!boot_stage.rmcall_gr_data_bounce.pages || + !boot_stage.rmcall_gr_data_bounce_pt.pages)) { + struct gsp_phys_boot_chunk *ptd = + &boot_stage.rmcall_gr_data_bounce_pt; + u32 i; + + if (!gsp_phys_chunk_alloc(boot_stage.pdev, + &boot_stage.rmcall_gr_data_bounce, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES)) + gsp_phys_chunk_zero(&boot_stage.rmcall_gr_data_bounce); + if (!gsp_phys_chunk_alloc(boot_stage.pdev, ptd, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES)) + gsp_phys_chunk_zero(ptd); + if (!boot_stage.rmcall_gr_data_bounce.pages || !ptd->pages) { + pr_err("vgsp-boot: rm-call: data-bounce region allocation failed (data=%u pt=%u pages); launches fail closed on the grown region\n", + boot_stage.rmcall_gr_data_bounce.pages, + ptd->pages); + } else { + for (i = 0; i < GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES; i++) { + u32 pd0_idx = + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0 + i; + u64 pde = gsp_phys_rmcall_pde( + (u64)gsp_phys_chunk_page_dma(ptd, i)); + + put_unaligned_le64(0, + (u8 *)gsp_phys_chunk_page_va(pd0, 0) + + (size_t)pd0_idx * 16); + put_unaligned_le64(pde, + (u8 *)gsp_phys_chunk_page_va(pd0, 0) + + (size_t)pd0_idx * 16 + 8); + } + for (i = 0; i < GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES; i++) { + u32 win = i / 512u; + u32 slot = i % 512u; + + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_data_bounce, + i)), + (u8 *)gsp_phys_chunk_page_va(ptd, win) + + (size_t)slot * 8); + } + dma_wmb(); + pr_info("vgsp-boot: rm-call: data-bounce region planted: %u pages at VA 0x%llx..0x%llx (PD0 slots %u..%u, %u PT pages, before first fetch)\n", + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + (GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES - + 1) * GSP_PHYS_PAGE_SIZE, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0 + + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES - + 1, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES); + } + } /* * P6a guest windows: start from explicit zero (invalid PTE) for both @@ -9491,8 +10169,16 @@ static void gsp_phys_rmcall_pdb_build(void) u64 guest_pte; int gpa_ret; - gpa_ret = gsp_phys_gpa_to_hpa(boot_stage.rmcall_guest_gpa, - &guest_hpa); + /* + * The pinned GPA is a single-guest test aid (the pin write is + * refused while more than one mdev is open; see + * gsp_phys_guest_gpa_write): resolve through the first open + * mdev exactly as before the per-mdev records, via the NULL + * selector (a per-mdev pin is not modelled). + */ + gpa_ret = gsp_phys_gpa_to_hpa_vgpu(NULL, + boot_stage.rmcall_guest_gpa, + &guest_hpa); if (!gpa_ret) { guest_pte = gsp_phys_rmcall_pte(guest_hpa); put_unaligned_le64(guest_pte, @@ -9532,6 +10218,30 @@ static void gsp_phys_rmcall_pdb_build(void) (size_t)(1 + i) * 16 + 8); } /* + * m152: extension scratch pool - PD0 dual entries 32..79 back the + * GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES pages of the multi-page extension + * PT chunk at GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE (0x04000000; free + * space above the GR-context windows, which end well below 0x03000000). + * Same shape and same once-before-any-engine-access rule as the 16-window + * loop above. + */ + for (i = 0; i < GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES; i++) { + u64 guest_pt_pde = gsp_phys_rmcall_pde( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_GUEST_PT_EXT0 + i], + 0)); + u32 pd0_idx = (u32)(GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE >> 21) + + i; + + put_unaligned_le64(0, + (u8 *)gsp_phys_chunk_page_va(pd0, 0) + + (size_t)pd0_idx * 16); + put_unaligned_le64(guest_pt_pde, + (u8 *)gsp_phys_chunk_page_va(pd0, 0) + + (size_t)pd0_idx * 16 + 8); + } + /* * H1 GR context window: when the GR ladder prepared the promoted MAIN * context buffer (gsp_phys_rmcall_gr_ctx_prepare ran before this * builder), append its page tables as fresh PD0 dual entries 17.. (the @@ -9698,11 +10408,12 @@ static void gsp_phys_rmcall_pdb_build(void) src_idx, pt_src, (u64)gsp_phys_chunk_page_dma(src, 0), dst_idx, pt_dst, (u64)gsp_phys_chunk_page_dma(dst, 0), sem_idx, pt_sem, (u64)gsp_phys_chunk_page_dma(sem, 0)); - pr_info("vgsp-boot: rm-call: guest scratch PT built: pages=%u vaBase=0x%llx capacity=%u\n", + pr_info("vgsp-boot: rm-call: guest scratch PT built: pages=%u (+%u ext) vaBase=0x%llx extBase=0x%llx capacity=%u\n", GSP_PHYS_RMCALL_GUEST_PT_PAGES, + GSP_PHYS_RMCALL_GUEST_PT_EXT_PAGES, GSP_PHYS_RMCALL_VA_GUEST_PT_BASE, - GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE); + GSP_PHYS_RMCALL_VA_GUEST_PT_EXT_BASE, + GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES); } /* @@ -12993,6 +13704,12 @@ static int gsp_phys_rmcall_user_lane_userd(void __iomem *bar0) * (CONTEXT_BUFFERS_INFO in particular) may be prepared by the query. All are * zero-in best-effort getters at the capture's payload sizes; a failure * warns and the lane continues. + * + * Every successful reply is retained via gsp_phys_kgr_snap_store(): the + * guest-side emulation serves these exact bytes for the same commands, so + * the guest's GL/3D stack reads the live GSP's true GR static values + * (SM_VERSION in particular - see the snapshot comment near + * gsp_phys_backend_static_get). */ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) { @@ -13031,6 +13748,8 @@ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) q[i].name); pr_info("vgsp-boot: rm-call: mode 35 KGR query %s: size=%u status=0x%x rpc_result=0x%x ret=%d\n", q[i].name, q[i].size, status, rpc_result, ret); + if (ret == 0 && rpc_result == 0 && status == 0) + gsp_phys_kgr_snap_store(q[i].cmd, buf, q[i].size); kvfree(buf); } } @@ -14536,6 +15255,39 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, * semaphore. On timeout the existing status-queue drain runs and the helper * returns -ETIMEDOUT; it never retries. */ +static int gsp_phys_rmcall_gr_resched_locked(void __iomem *bar0); +static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0); + +/* + * r445 wedge hard re-arm: the outcome-retry's warm semaphore probe proves the + * engine refused the channel's proven GR block (the sticky post-dispatch + * no-store wedge). One bounded recovery attempt instead of an immediate + * break: force a context save/unload with a bWait TSG preempt, re-run the + * mode-35 lane-enable tail + schedules (the m155 extended re-arm), then + * repeat the warm probe. Caller holds gsp_phys_rm_lock. Returns 0 when the + * engine executed the probe after the re-arm. + */ +static int gsp_phys_gr_wedge_hard_rearm_locked(void __iomem *bar0) +{ + u32 pre[2] = { 1u, 0u }; /* bWait=1, bManualTimeout=0 */ + u32 rpc_result = 0, status = 0; + int ret; + + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_CTRL_TSG_PREEMPT, + pre, sizeof(pre), sizeof(pre), NULL, 0, + &rpc_result, &status, + "TSG PREEMPT bWait=1 (r445 wedge hard re-arm)"); + pr_warn("vgsp-gr: WEDGE-REARM preempt status=0x%x rpc=0x%x ret=%d\n", + status, rpc_result, ret); + ret = gsp_phys_rmcall_gr_resched_locked(bar0); + pr_warn("vgsp-gr: WEDGE-REARM resched ret=%d\n", ret); + ret = gsp_phys_rmcall_gr_sem_push_locked(bar0); + pr_warn("vgsp-gr: WEDGE-REARM probe rc=%d (engine %s after re-arm)\n", + ret, ret ? "still wedged" : "ALIVE"); + return ret; +} + static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) { struct gsp_phys_boot_chunk *userd_c = @@ -15162,7 +15914,6 @@ static int gsp_phys_rmcall_guest_map_locked(u64 src_hpa, u64 dst_hpa, u64 sem_hpa, u64 *src_va, u64 *dst_va, u64 *sem_va) { - struct gsp_phys_boot_chunk *guest_pt_chunk; u32 line = boot_stage.rmcall_guest_pt_next_line; u32 page, line_in_page; u64 va; @@ -15170,19 +15921,14 @@ static int gsp_phys_rmcall_guest_map_locked(u64 src_hpa, u64 dst_hpa, if (!boot_stage.rmcall_va_guest_pt) return -EIO; - if (line >= GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + if (line >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) return -ENOSPC; page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; line_in_page = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; - guest_pt_chunk = &boot_stage.rmcall_chunks[ - GSP_PHYS_RMCALL_BUF_GUEST_PT0 + page]; - if (!guest_pt_chunk->pages) - return -EIO; /* 2 MB PD0 slot stride; a full line is 16 PTEs = 0x10000 of VA. */ - va = boot_stage.rmcall_va_guest_pt + ((u64)page << 21) + + va = gsp_phys_rmcall_guest_pt_win_va(page) + ((u64)line_in_page << 16); - pt = gsp_phys_chunk_page_va(guest_pt_chunk, 0); + pt = gsp_phys_rmcall_guest_pt_page_va(page); if (!pt) return -EIO; /* @@ -15403,15 +16149,13 @@ static int gsp_phys_guest_ce_copy_common(u64 src_hpa, u32 src_off, u64 dst_hpa, if ((gsp_phys_relay_pool_exhausted & 63) == 1) pr_err("vgsp-work: GUEST CE RELAY FAILED: scratch-PT pool exhausted (no line reuse) pool_exhausted=%llu capacity=%u; falling back to the software release\n", gsp_phys_relay_pool_exhausted, - GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE); + GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES); } else { gsp_phys_guest_pool_exhausted++; if ((gsp_phys_guest_pool_exhausted & 63) == 1) pr_err("vgsp-work: GUEST CE COPY FAILED: scratch-PT pool exhausted (no line reuse) pool_exhausted=%llu capacity=%u; falling back to the ladder path\n", gsp_phys_guest_pool_exhausted, - GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE); + GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES); } } else { if (relay) @@ -15730,6 +16474,19 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_guest_ce_copy); #define GSP_PHYS_RMCALL_GR_CWD_REFCOUNT 1536u #define GSP_PHYS_RMCALL_GR_CWD_LEVELS 64u #define GSP_PHYS_RMCALL_GR_MAX_SM_COUNT 24u +/* + * r434 CWD-slot-count path: the falcon PRI write (CUDA's + * turingFalconPRIWriteMasked: flag/data/mask through SET_MME_SHADOW_SCRATCH + * 0/1/2, then SET_FALCON04 = register offset) and the gated method. Used only + * when gsp_phys_rmcall_gr_cwd_slot >= 0. + */ +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH0 0x3400u +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH1 0x3404u +#define GSP_PHYS_RMCALL_GR_MME_SCRATCH2 0x3408u +#define GSP_PHYS_RMCALL_GR_SET_FALCON04 0x0510u +#define GSP_PHYS_RMCALL_GR_SET_CWD_SLOT_COUNT 0x02b0u +#define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1 0x00407040u +#define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT 0x800u /* Method block bound: init(152) + inline run(67) + PCAS(4) + REPORT_SEM(8) + * the optional QMD-virt pair(2). */ #define GSP_PHYS_RMCALL_GR_METHOD_WORDS 512u @@ -15738,7 +16495,109 @@ static u64 gsp_phys_gr_submitted; static u64 gsp_phys_gr_ok; static u64 gsp_phys_gr_refused; static u64 gsp_phys_gr_timed_out; +static u64 gsp_phys_gr_retries; /* fresh-entry launch retries published */ static u64 gsp_phys_gr_pool_exhausted; +/* m156 (E5): zero-write retries where the warm sem push proved the engine + * wedged (the persistent-lottery class); further retries are skipped. */ +static u64 gsp_phys_gr_sem_wedged; +/* + * STALLFREE: bounded-completion telemetry. stall_wait_max counts launches + * that crossed gsp_phys_gr_wait_max_ms and failed out of the submit ladder + * (the sema service then authors the QMD release in software); stall_retry + * counts launches whose retirement retry loop stopped early on the same + * budget (the launch proceeds with its current outcome). Both are zero on a + * green run. + */ +static u64 gsp_phys_gr_stall_wait_max; +static u64 gsp_phys_gr_stall_retry; +/* + * r410 wedge quiet window state. nw_streak counts consecutive accepted + * zero-write launches with tracked data (reset by any launch that wrote); + * the sticky wedge is the state where this grows without bound while the + * lane is kept busy. quiet counts the quiet holds issued; heals counts the + * quiet retries after which the engine wrote. Single-inflight submit path, + * so plain statics. + */ +static u32 gsp_phys_gr_nw_streak; +static u64 gsp_phys_gr_wedge_quiet; +static u64 gsp_phys_gr_wedge_heals; +/* + * r411: once a quiet heal has succeeded on this boot, the lane is in a + * wedge-prone episode. r418 showed the wedge can alternate one-no-write / + * one-heal, and the healed launch resets nw_streak, so a streak-only trigger + * quieted only every second wedged launch and the other was accepted. The + * heal count makes the episode sticky for the boot: any later zero-write + * launch gets the quiet path on its first retry, so every wedged launch + * either heals (engine writes, client data correct) or is loudly classified. + */ +static bool gsp_phys_gr_wedge_episode; +/* + * r420: true exactly while a WEDGE-QUIET hold sleeps with NO submission + * outstanding. The hold is a deliberate, bounded wait for the sticky wedge to + * clear (r410/r411); during it the in-flight forwarded request is NOT done. + * The sema service's completion deadlines - the 12 s GR release hold + * (SVC_FWD_GR_HOLD_TIMEOUT_MS) and the 3 s per-launch consumption bound + * (gsp_phys_gr_hold_max_ms) - are shorter than this hold, so without the + * publish they fire UNDER it and author the launch's release word before the + * post-hold copy-out: the client then verifies stale data (r419 soak: + * `victory_matmul bad=65536` while its launch was held; the authoring paths + * are GR-STALL-fwd-hold / the busy-defer inline software release). The + * accessor lets both deadlines pause instead; the flag is bounded by the + * msleep itself and by the pause caps on the sema side. Single-inflight + * submit path, so a plain static. + */ +static bool gsp_phys_gr_quiet_hold_on; + +bool gsp_phys_gr_quiet_hold_active(void) +{ + return READ_ONCE(gsp_phys_gr_quiet_hold_on); +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_quiet_hold_active); + +/* + * r425 experiment (the second-launch wedge): the stock guest driver emits the + * per-channel compute init block ONCE at channel creation (turing_init.c), + * long before any launch, while this backend re-emits it as the preamble of + * every forwarded launch. The wedge strikes the SECOND launch of a channel, so + * test whether the re-emission is the trigger: with gsp_phys_rmcall_gr_init_once + * set, the init preamble is emitted only on the first forwarded launch after + * the module load (the engine's compute-class/CWD state persists in the + * context; every launch still carries its QMD + report semaphore). 0 keeps the + * historical per-launch emission (default; runtime-writable A/B). + */ +static bool gsp_phys_rmcall_gr_init_once_param; +module_param_named(gsp_phys_rmcall_gr_init_once, gsp_phys_rmcall_gr_init_once_param, + bool, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_init_once, + "r425: emit the GR compute init preamble only on the first forwarded launch (0 = every launch, default)"); +static bool gsp_phys_rmcall_gr_init_emitted; + +/* + * r425 stale-QMD witness: the first engine-changed page of the LAST completed + * launch (bounce index + post-copy-out checksum). At a zero-write accept the + * page is re-scanned; a change proves a late store (a stuck/queued grid or a + * stale-QMD execution) landed after its launch was shipped. + */ +static u32 gsp_phys_gr_prev_out_bi; +static u64 gsp_phys_gr_prev_out_csum; +static bool gsp_phys_gr_prev_out_valid; + +/* r425: SKED/FECS/CWD register witness (one line; the healthy-vs-wedged + * comparison baseline lives in the same log). */ +static void gsp_phys_gr_sked_wit(void __iomem *bar0, const char *tag) +{ + if (!bar0) + return; + pr_info("vgsp-gr: SKED-WIT %s cwd0=%08x cwd1=%08x cwd2=%08x cwd3=%08x info1=%08x act=%08x grfifo=%08x fecs=%08x new=%08x ctxsw1=%08x qmdvirt0=%08x tpcm0=%08x\n", + tag, + readl(bar0 + 0x00407060u), readl(bar0 + 0x00407064u), + readl(bar0 + 0x00407068u), readl(bar0 + 0x0040706cu), + readl(bar0 + 0x00407028u), readl(bar0 + 0x00407054u), + readl(bar0 + 0x00400504u), readl(bar0 + 0x00409b00u), + readl(bar0 + 0x00409b04u), readl(bar0 + 0x00409400u), + readl(bar0 + 0x0040a500u), readl(bar0 + 0x0040a680u)); +} + /* m86: program-page override blob (see gsp-phys-gr-override). */ static u8 gsp_phys_gr_override[GSP_PHYS_PAGE_SIZE]; static u32 gsp_phys_gr_override_len; @@ -15848,6 +16707,29 @@ static u32 *gsp_phys_rmcall_build_gr_init(u32 *p) GSP_PHYS_RMCALL_GR_MAX_SM_COUNT); p = gsp_phys_rmcall_push_gr(p, GSP_PHYS_RMCALL_GR_INIT_SM_DISP_CTRL, 0u); + /* + * r434 CWD-slot-count path (gsp_phys_rmcall_gr_cwd_slot >= 0): masked + * PRI write of SKED_DEBUG_1.ENABLE_CWD_SLOT_COUNT (bit 11) through the + * falcon scratch registers, then the gated SET_CWD_SLOT_COUNT method - + * the CUDA driver's own turingLaunchSetCwdSlotCount sequence. -1 (the + * default) emits nothing, so the historical preamble is unchanged. + */ + if (gsp_phys_rmcall_gr_cwd_slot_value() >= 0) { + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH0, 0u); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH1, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_MME_SCRATCH2, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_SET_FALCON04, + GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1); + p = gsp_phys_rmcall_push_gr( + p, GSP_PHYS_RMCALL_GR_SET_CWD_SLOT_COUNT, + (u32)gsp_phys_rmcall_gr_cwd_slot_value()); + } return p; } @@ -15868,7 +16750,17 @@ static u32 gsp_phys_rmcall_build_gr_launch(u32 *pb, int qmdvirt = gsp_phys_rmcall_gr_qmdvirt_value(); int two_step = gsp_phys_rmcall_gr_pcas_two_step_value(); - p = gsp_phys_rmcall_build_gr_init(p); + /* + * r425 A/B: with gsp_phys_rmcall_gr_init_once the compute init preamble + * is emitted only on the first forwarded launch (the stock driver's + * per-channel shape); subsequent launches start at the inline-QMD + * window. + */ + if (!READ_ONCE(gsp_phys_rmcall_gr_init_once_param) || + !READ_ONCE(gsp_phys_rmcall_gr_init_emitted)) { + p = gsp_phys_rmcall_build_gr_init(p); + WRITE_ONCE(gsp_phys_rmcall_gr_init_emitted, true); + } if (qmdvirt) p = gsp_phys_rmcall_push_gr(p, GSP_PHYS_RMCALL_GR_SET_QMD_VIRT_CONTROL, @@ -16077,26 +16969,26 @@ static void __iomem *gsp_phys_gr_scratch_bar0; static int gsp_phys_rmcall_gr_scratch_win_install_locked(u32 page) { struct gsp_phys_rmcall_update_pde2_params p; - struct gsp_phys_boot_chunk *pt; u32 rpc_result = 0, status = 0, pde_index; + u64 pt_dma; int ret; - if (page >= GSP_PHYS_RMCALL_GUEST_PT_PAGES) + if (page >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_PAGES) return -EINVAL; if (!gsp_phys_rmcall_vas_shared_value()) return 0; - if (boot_stage.rmcall_guest_pt_win_mask & BIT(page)) + if (boot_stage.rmcall_guest_pt_win_mask & BIT_ULL(page)) return 0; - pt = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GUEST_PT0 + page]; - if (!pt->pages) + pt_dma = gsp_phys_rmcall_guest_pt_page_dma(page); + if (!pt_dma) return -EIO; if (!boot_stage.rmcall_va_guest_pt) return -EIO; - pde_index = (u32)((boot_stage.rmcall_va_guest_pt >> 21) + page); + pde_index = (u32)(gsp_phys_rmcall_guest_pt_win_va(page) >> 21); memset(&p, 0, sizeof(p)); p.pde_index = cpu_to_le32(pde_index); p.flags = cpu_to_le32(GSP_PHYS_RMCALL_PDE2_FLAG_FLUSH); - p.pt[0].phys_addr = cpu_to_le64((u64)gsp_phys_chunk_page_dma(pt, 0)); + p.pt[0].phys_addr = cpu_to_le64(pt_dma); p.pt[0].aperture = cpu_to_le32(GSP_PHYS_RMCALL_PDE2_APERTURE_SYS_COH); p.pt[1].aperture = cpu_to_le32(GSP_PHYS_RMCALL_PDE2_APERTURE_INVALID); p.h_va_space = cpu_to_le32(boot_stage.rmcall_vas); @@ -16113,10 +17005,9 @@ static int gsp_phys_rmcall_gr_scratch_win_install_locked(u32 page) ret); return ret; } - boot_stage.rmcall_guest_pt_win_mask |= BIT(page); + boot_stage.rmcall_guest_pt_win_mask |= BIT_ULL(page); pr_info("vgsp-gr: scratch window installed pdeIndex=%u (VA 0x%llx) ptPa=0x%llx status=0x%x (Tier-2 shadow legs now reachable through the server walker)\n", - pde_index, (u64)pde_index << 21, - (u64)gsp_phys_chunk_page_dma(pt, 0), status); + pde_index, (u64)pde_index << 21, pt_dma, status); return 0; } @@ -16129,13 +17020,11 @@ static int gsp_phys_rmcall_gr_scratch_win_install_locked(u32 page) static bool gsp_phys_rmcall_gr_win_is_installed_locked(u64 va) { u32 pde_index = (u32)(va >> 21); - u32 base = (u32)(boot_stage.rmcall_va_guest_pt >> 21); + u32 page = gsp_phys_rmcall_guest_pt_page_of_pde(pde_index); u32 i; - if (pde_index >= base && - pde_index < base + GSP_PHYS_RMCALL_GUEST_PT_PAGES) - return !!(boot_stage.rmcall_guest_pt_win_mask & - BIT(pde_index - base)); + if (page != 0xffffffffu) + return !!(boot_stage.rmcall_guest_pt_win_mask & BIT_ULL(page)); for (i = 0; i < boot_stage.rmcall_gr_pt_used; i++) if (boot_stage.rmcall_gr_win[i].used && boot_stage.rmcall_gr_win[i].pde_index == pde_index) @@ -16154,13 +17043,11 @@ static bool gsp_phys_rmcall_gr_win_is_installed_locked(u64 va) static int gsp_phys_rmcall_gr_win_ensure_locked(void __iomem *bar0, u64 va) { u32 pde_index = (u32)(va >> 21); - u32 base = (u32)(boot_stage.rmcall_va_guest_pt >> 21); + u32 page = gsp_phys_rmcall_guest_pt_page_of_pde(pde_index); u32 slot; - if (pde_index >= base && - pde_index < base + GSP_PHYS_RMCALL_GUEST_PT_PAGES) - return gsp_phys_rmcall_gr_scratch_win_install_locked( - pde_index - base); + if (page != 0xffffffffu) + return gsp_phys_rmcall_gr_scratch_win_install_locked(page); return gsp_phys_rmcall_gr_win_slot_locked(bar0, pde_index, &slot); } @@ -16171,16 +17058,35 @@ static int gsp_phys_rmcall_gr_win_ensure_locked(void __iomem *bar0, u64 va) * supplies the backing page. Called with gsp_phys_rm_lock held. */ static int gsp_phys_rmcall_gr_map_shadow_locked(u32 slot, u64 hpa, - u64 *va_out) + u64 *va_out, u32 *line_out) { struct gsp_phys_boot_chunk *sh = &boot_stage.rmcall_gr_shadow; - struct gsp_phys_boot_chunk *pt; u32 line = boot_stage.rmcall_guest_pt_next_line; - u32 page, line_in_page; + u32 page, line_in_page, i; u64 va; - u8 *pt_va; + u8 *pt; int ret; + /* + * m152: reuse before consume. A shadow VA is a never-rewritten PTE: + * once hpa X maps at VA V, every later request for X can be served by + * V with byte-identical engine-visible content, so the pool only + * needs one line per distinct HPA per boot instead of one per launch. + * This is safe against the GMMU's no-invalidate fetch-line cache by + * construction: the PTE value never differs from what the engine may + * have already fetched. A hit reports line_out = ~0 so a caller that + * records contiguity bookkeeping never contig-extends an old line + * (a new PTE in a line the engine may have already fetched would be + * invisible). + */ + for (i = 0; i < boot_stage.rmcall_gr_alias_count; i++) { + if (boot_stage.rmcall_gr_alias_hpa[i] == hpa) { + *va_out = boot_stage.rmcall_gr_alias_va[i]; + if (line_out) + *line_out = 0xffffffffu; + return 0; + } + } if (!sh->pages) { ret = gsp_phys_chunk_alloc(boot_stage.pdev, sh, GSP_PHYS_RMCALL_GR_SHADOW_PAGES); @@ -16192,28 +17098,34 @@ static int gsp_phys_rmcall_gr_map_shadow_locked(u32 slot, u64 hpa, return -ENOSPC; if (!boot_stage.rmcall_va_guest_pt) return -EIO; - if (line >= GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + if (line >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) return -ENOSPC; page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; line_in_page = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; ret = gsp_phys_rmcall_gr_scratch_win_install_locked(page); if (ret) return ret; - pt = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GUEST_PT0 + page]; - if (!pt->pages) - return -EIO; - pt_va = gsp_phys_chunk_page_va(pt, 0); - if (!pt_va) + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) return -EIO; put_unaligned_le64(gsp_phys_rmcall_pte(hpa), - pt_va + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * line_in_page) * 8); dma_wmb(); boot_stage.rmcall_guest_pt_next_line = line + 1; - va = boot_stage.rmcall_va_guest_pt + ((u64)page << 21) + + va = gsp_phys_rmcall_guest_pt_win_va(page) + ((u64)line_in_page << 16); *va_out = va; + if (line_out) + *line_out = line; + if (boot_stage.rmcall_gr_alias_count < + GSP_PHYS_RMCALL_GR_ALIAS_CACHE) { + u32 ci = boot_stage.rmcall_gr_alias_count; + + boot_stage.rmcall_gr_alias_hpa[ci] = hpa; + boot_stage.rmcall_gr_alias_va[ci] = va; + boot_stage.rmcall_gr_alias_count = ci + 1u; + } return 0; } @@ -16227,32 +17139,27 @@ static int gsp_phys_rmcall_gr_map_shadow_locked(u32 slot, u64 hpa, static int gsp_phys_rmcall_gr_map_shadow_contig_locked(u32 line, u32 pte, u64 hpa, u64 *va_out) { - struct gsp_phys_boot_chunk *pt; u32 page, line_in_page; - u8 *pt_va; + u8 *pt; int ret; if (pte >= GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES) return -EINVAL; - if (line >= GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + if (line >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) return -ENOSPC; page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; line_in_page = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; ret = gsp_phys_rmcall_gr_scratch_win_install_locked(page); if (ret) return ret; - pt = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GUEST_PT0 + page]; - if (!pt->pages) - return -EIO; - pt_va = gsp_phys_chunk_page_va(pt, 0); - if (!pt_va) + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) return -EIO; put_unaligned_le64(gsp_phys_rmcall_pte(hpa), - pt_va + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * line_in_page + pte) * 8); dma_wmb(); - *va_out = boot_stage.rmcall_va_guest_pt + ((u64)page << 21) + + *va_out = gsp_phys_rmcall_guest_pt_win_va(page) + ((u64)line_in_page << 16) + (u64)pte * GSP_PHYS_PAGE_SIZE; return 0; } @@ -16315,14 +17222,23 @@ static void gsp_phys_rmcall_gr_gpc_mmu_init(void __iomem *bar0) * the affine-relocated server-managed VA (the engine cannot walk the * client-managed guest VAs: UPDATE_PDE_2 refuses them with 0x1f). */ -static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, - const struct gsp_phys_gr_submit *req, - u32 bank, u32 *mirror) +/* + * relfix (agent-A FIX A): resolve-and-author one unmatched device-VA word. + * Defined after gsp_phys_gr_data_bi (the bounce-index helpers it needs); + * returns true when the word was rewritten to a reloc VA (and the pages + * queued for copy-in/out), false to leave it to the RELOC-GAP accounting. + */ +static bool gsp_phys_gr_reloc_fix_word(const struct gsp_phys_gr_submit *req, + u32 *w, u32 k, u32 *mirror); +static u32 gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, + const struct gsp_phys_gr_submit *req, + u32 bank, u32 *mirror, u64 *gap_first, + bool fix_enabled) { - u32 rew = 0, k; + u32 rew = 0, skipped = 0, gap = 0, k; if (!req->n_data) - return; + return 0; /* * m117: 8-byte stride with a 256-byte alignment guard. The m103 * 4-byte stride mis-matched arbitrary adjacent 4-byte words (the @@ -16335,19 +17251,6 @@ static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, u64 wb; u32 x; - /* - * m135: never relocate below offset 0x100. The param-page - * header (grid/block dims and driver bookkeeping at 0x0..0xff) - * can contain server-region VAs (e.g. 0x100000100) that are - * NOT kernel parameters; relocating them corrupted the - * vadd launch's blockDim word (read as 0x170100 instead of - * 0x100, so every CTA past 0 computed out-of-range indices - * and exited: VADD bad=32512 exactly one block correct). - * Kernel parameters live at >= 0x140 (JIT sm_75 layout: - * a/b/c at 0x160/0x168/0x170, n at 0x178). - */ - if (k * 4u < 0x100u) - continue; if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 0xff)) continue; wb = v & ~0xfffULL; @@ -16355,6 +17258,34 @@ static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, if (req->data[x].va != wb || !req->data[x].reloc_va) continue; + /* + * m135: never relocate below offset 0x100. The + * param-page header (grid/block dims and driver + * bookkeeping at 0x0..0xff) can contain + * server-region VAs (e.g. 0x100000100) that are NOT + * kernel parameters; relocating them corrupted the + * vadd launch's blockDim word (read as 0x170100 + * instead of 0x100, so every CTA past 0 computed + * out-of-range indices and exited: VADD bad=32512 + * exactly one block correct). Kernel parameters live + * at >= 0x140 (JIT sm_75 layout: a/b/c at + * 0x160/0x168/0x170, n at 0x178). + * + * m181 diagnostic: a below-0x100 value whose page is + * an operand VA of this launch is still logged (the + * observed class is the driver control block at + * bank offset 0, present in green clients too); the + * guard itself is unchanged until a dereferenced + * pointer in that range is proven. + */ + if (k * 4u < 0x100u) { + if (skipped < 16) { + skipped++; + pr_info("vgsp-gr: bank %u RELOC-SKIP @0x%x: 0x%llx is operand page 0x%llx (below 0x100, left guest-VA)\n", + bank, k * 4, v, wb); + } + break; + } /* m120: log the hit site so coverage is provable. */ if (rew < 16) pr_info("vgsp-gr: bank %u hit @0x%x: 0x%llx -> 0x%llx\n", @@ -16388,10 +17319,30 @@ static void gsp_phys_gr_rewrite_data_vars(u32 *w, u32 n_words, rew++; break; } + /* + * GRFIX FIX3: the inner scan ran out (x reached n_data) + * without matching req->data[x].va - the word passes the + * exact shape heuristic the match test above uses, so it + * will be left as an unrelocated guest VA in this page. + * relfix: first give the sema service's trusted resolver a + * chance to translate it; a word that resolves is authored + * as a late data page and rewritten to its reloc VA (the + * exact matched-word encoding). Only an unresolved word is + * counted as a gap for the caller's RELOC-GAP log/hold. + */ + if (x == req->n_data) { + if (fix_enabled && gsp_phys_gr_reloc_resolve_param && + gsp_phys_gr_reloc_fix_word(req, w, k, mirror)) + continue; + gap++; + if (gap_first && !*gap_first) + *gap_first = v; + } } if (rew) pr_info("vgsp-gr: bank %u: %u data pointers relocated\n", bank, rew); + return gap; } /* @@ -16471,6 +17422,24 @@ static int gsp_phys_rmcall_gr_resched_locked(void __iomem *bar0) u32 runlist_id, ch_id, doorbell; int ret; + /* + * m155 (HF-3 extended re-arm): after the channel has been idle long + * enough for the FECS to save/unload the context, re-issuing only the + * two GPFIFO_SCHEDULE verbs does not revive dispatch - r154: matmul's + * post-smiley launches acked (qmd_rel/ch_rel) but SKED stayed idle + * (cwd0=0x2086/fecs_cur=0) for every attempt including outcome + * retries, while the same channel executed mail/smiley earlier in the + * boot. Replay the mode-35 lane-enable tail (CILP preempt mode, + * timeslice, device sched) that stage 4 ran when the lane first went + * live - it is proven accepted and is the design's extended re-arm. + * The poison variants (explicit PREEMPT/tail, BIND, PROMOTE) stay out. + */ + ret = gsp_phys_rmcall_user_lane_lane_enable(bar0); + if (ret) + pr_info("vgsp-gr: rearm lane-enable rc=%d (continuing to schedules)\n", + ret); + rpc_result = 0; + status = 0; ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_channel, GSP_PHYS_RMCALL_CTRL_GPFIFO_SCHEDULE, sched_params, sizeof(sched_params), 16, @@ -16495,6 +17464,1581 @@ static int gsp_phys_rmcall_gr_resched_locked(void __iomem *bar0) return 0; } +/* + * m151: FNV-1a 64-bit over one page. Used to compare the win0 param page + * and the guest param page before and after the poll, so a launch that + * wrote (or clobbered) the page the engine read is provable from the log. + */ +static inline u64 gsp_phys_gr_fnv1a64(const u8 *p, size_t len) +{ + u64 h = 0xcbf29ce484222325ULL; + size_t i; + + for (i = 0; i < len; i++) { + h ^= p[i]; + h *= 0x100000001b3ULL; + } + return h; +} + +/* + * B-2: the data chunk the active operand region lives in. With + * gsp_phys_gr_data_region=1 (default) that is the grown 16384-page region; + * with 0 it is the legacy 208-page bounce window (whose data pages sit below + * the mail slot). The copy path always asks this instead of naming a chunk so + * the two paths share one index space per mode. + */ +static struct gsp_phys_boot_chunk *gsp_phys_gr_data_chunk(void) +{ + if (gsp_phys_gr_data_region_value()) + return &boot_stage.rmcall_gr_data_bounce; + return &boot_stage.rmcall_gr_bounce; +} + +/* + * r452: scan the whole active GR data-bounce chunk for an 8-byte pattern + * (write "0x<pattern>" to /sys/module/vgsp/parameters/gsp_phys_gr_scan). + * Verifies by content where a lost engine store actually landed: the chunk + * is the only bulk memory the operand window PTEs map, so a store that + * executed but is absent from the writer's tracked pages must be findable + * here (a stale-PTE landing) or nowhere (a dropped store). + */ +static int gsp_phys_gr_scan_set(const char *val, + const struct kernel_param *kp) +{ + struct gsp_phys_boot_chunk *c = gsp_phys_gr_data_chunk(); + u64 pattern = 0; + u32 i, hits = 0; + + if (kstrtoull(val, 0, &pattern)) + return -EINVAL; + if (!c->pages) + return -ENODEV; + for (i = 0; i < c->pages; i++) { + const u8 *p = gsp_phys_chunk_page_va(c, i); + u32 off; + + if (!p) + continue; + for (off = 0; off + 8 <= GSP_PHYS_PAGE_SIZE; off += 8) { + if (get_unaligned_le64(p + off) == pattern) { + if (hits < 32) + pr_info("vgsp-gr: CHUNK-SCAN found 0x%llx at bi=%u off=0x%x (va=0x%llx)\n", + pattern, i, off, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + ((u64)i << 12) + off); + hits++; + } + } + } + pr_info("vgsp-gr: CHUNK-SCAN 0x%llx done: %u hit(s) over %u pages\n", + pattern, hits, c->pages); + return 0; +} + +static const struct kernel_param_ops gsp_phys_gr_scan_ops = { + .set = gsp_phys_gr_scan_set, +}; +module_param_cb(gsp_phys_gr_scan, &gsp_phys_gr_scan_ops, NULL, 0200); +MODULE_PARM_DESC(gsp_phys_gr_scan, + "Scan the GR data-bounce chunk for an 8-byte pattern (hex)"); + +/* + * B-2: page index of a relocated operand VA inside the engine-visible data + * region, replacing the five inlined (reloc_va - 0x110000) >> 12 derivations + * and their MAIL_SLOT bounds. Grown region: VAs from + * GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0, bound + * GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES. Legacy window: VAs from + * GSP_PHYS_RMCALL_GR_BOUNCE_VA0, bound GSP_PHYS_RMCALL_GR_MAIL_SLOT so the + * mail (206) and marker (207) pages are never treated as data. Returns + * GSP_PHYS_GR_DATA_BI_NONE when the VA is outside the active region (the copy + * sites skip such records exactly as they skipped the old mail-slot bound). + */ +#define GSP_PHYS_GR_DATA_BI_NONE 0xffffffffu +static u32 gsp_phys_gr_data_bi(u64 reloc_va) +{ + u32 bi; + + if (gsp_phys_gr_data_region_value()) { + if (reloc_va < GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0) + return GSP_PHYS_GR_DATA_BI_NONE; + bi = (u32)((reloc_va - GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0) >> + 12); + if (bi >= GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES) + return GSP_PHYS_GR_DATA_BI_NONE; + return bi; + } + if (reloc_va < GSP_PHYS_RMCALL_GR_BOUNCE_VA0) + return GSP_PHYS_GR_DATA_BI_NONE; + bi = (u32)((reloc_va - GSP_PHYS_RMCALL_GR_BOUNCE_VA0) >> 12); + if (bi >= GSP_PHYS_RMCALL_GR_MAIL_SLOT) + return GSP_PHYS_GR_DATA_BI_NONE; + return bi; +} + +/* + * relfix: late-resolved data pages. A QMD word that matches no tracked data + * page yet resolves at rewrite time becomes a real operand page of this + * launch: it gets the next bounce-region index after the fill's tracked set, + * is copied in before the doorbell and copied back after the retire wait + * (the same m142 ordering as every tracked page), and the word is rewritten + * to its reloc VA - so the engine stops walking an untranslatable guest VA + * (r216 MMU FAULT_PDE class). Bounds are per-launch and fail-open: at the + * caps, or when no source answers, the word simply stays a RELOC-GAP (the + * pre-relfix behaviour). Static scratch: the single-inflight ordered + * gsp-phys-work item is the only caller, reset per submission exactly like + * bounce_csum_in. + */ +#define GSP_PHYS_GR_GAP_MAX 4096u /* total late pages per launch */ +/* + * relfix PART 3 (r217 span containment): the late path authors ONLY the + * pages actually needed by the fixed word. r217 showed two fixed words + * (the launch's own param/program descriptor fields, which the fill skips + * as launch-internal) walking ~1242 fully-mapped neighbour pages each - + * 2484 late pages, ~10 MB copied in and out per launch, for zero engine + * writes - and the clients converted from fast-fail (~60 s verdict) to + * killed at 250 s. The span is now a small fetch line, not a region walk: + * the multi-page operand class is owned by the fill's own span pipeline + * (which the PART 3 candidate superset now feeds), and this path is a + * last-resort proxy for words the fill could not own. + */ +#define GSP_PHYS_GR_GAP_SPAN_MAX 16u /* contiguous pages per word max */ +#define GSP_PHYS_GR_GAP_WORD_MAX 32u /* distinct fixed words per launch */ +#define GSP_PHYS_GR_GAP_LOG_MAX 16u /* distinct gap VAs stage-logged */ +struct gsp_phys_gr_gap_page { + u64 va; /* device VA of the page (unique-proxy key) */ + u64 hpa; /* guest page copied in (results copied back to) */ + u64 reloc_va; /* engine-visible bounce-region VA of that page */ + u32 bi; /* bounce page index (n_data + late index) */ +}; +static struct gsp_phys_gr_gap_page + gsp_phys_gr_gap[GSP_PHYS_GR_GAP_MAX]; +static u32 gsp_phys_gr_gap_n; +struct gsp_phys_gr_gap_word { + u64 va; /* device VA of the fixed word (page base) */ + u64 reloc_va; /* reloc VA of its first page */ + u64 pa; /* resolved guest PA (logging) */ +}; +static struct gsp_phys_gr_gap_word + gsp_phys_gr_gap_word[GSP_PHYS_GR_GAP_WORD_MAX]; +static u32 gsp_phys_gr_gap_word_n; +static u64 gsp_phys_gr_reloc_fixed; /* words rewritten this launch */ +static u64 gsp_phys_gr_reloc_fixed_first; /* first fixed device VA */ +static u64 gsp_phys_gr_gap_logged[GSP_PHYS_GR_GAP_LOG_MAX]; +static u32 gsp_phys_gr_gap_logged_n; + +/* + * relfix: reloc VA of the i-th bounce-region index, region mode exactly as + * gsp_phys_gr_data_bi; the legacy window's small bound fails open. + */ +static bool gsp_phys_gr_data_reloc_va(u32 idx, u64 *va_out) +{ + if (gsp_phys_gr_data_region_value()) { + if (idx >= GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES) + return false; + *va_out = GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + ((u64)idx << 12); + return true; + } + if (idx >= GSP_PHYS_RMCALL_GR_MAIL_SLOT) + return false; + *va_out = GSP_PHYS_RMCALL_GR_BOUNCE_VA0 + + ((u64)idx << 12); + return true; +} + +/* + * relfix: append one late page at the next free bounce index (n_data + + * gap_n, the slot right after the fill's tracked set). Fail-open on the + * caps or an absent bounce page. + */ +static bool gsp_phys_gr_gap_add(u32 n_data, u64 va, u64 hpa, u64 *reloc_out) +{ + u32 gi = gsp_phys_gr_gap_n; + u32 idx = n_data + gi; + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u64 reloc_va; + + if (gi >= GSP_PHYS_GR_GAP_MAX) + return false; + if (!gsp_phys_gr_data_reloc_va(idx, &reloc_va)) + return false; + if (!dchunk->pages || !gsp_phys_chunk_page_va(dchunk, idx)) + return false; + gsp_phys_gr_gap[gi].va = va; + gsp_phys_gr_gap[gi].hpa = hpa; + gsp_phys_gr_gap[gi].reloc_va = reloc_va; + gsp_phys_gr_gap[gi].bi = idx; + gsp_phys_gr_gap_n = gi + 1u; + *reloc_out = reloc_va; + return true; +} + +/* + * relfix: is this device page already covered by a proxy this launch? + * Either the fill's tracked set (req->data, each page exactly once - the + * fill dedupes) or a late page queued by this fix. A second proxy for one + * guest page is unsound: the engine may author results through either + * slot, and the two copy-outs would race to write back to the same guest + * page, so one would clobber the other's results. The fix therefore never + * creates a second proxy (see the span rollback in + * gsp_phys_gr_reloc_fix_word). + */ +static bool gsp_phys_gr_page_tracked(const struct gsp_phys_gr_submit *req, + u64 va) +{ + u32 i; + + for (i = 0; i < req->n_data; i++) + if (req->data[i].va == va) + return true; + for (i = 0; i < gsp_phys_gr_gap_n; i++) + if (gsp_phys_gr_gap[i].va == va) + return true; + return false; +} + +/* + * relfix PART 3: one rate-limited log per distinct gap VA per launch + * (the dedup helper both gap diagnostics below share). + */ +static bool gsp_phys_gr_gap_log_once(u64 va) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_gap_logged_n; i++) + if (gsp_phys_gr_gap_logged[i] == va) + return false; + if (gsp_phys_gr_gap_logged_n < ARRAY_SIZE(gsp_phys_gr_gap_logged)) + gsp_phys_gr_gap_logged[gsp_phys_gr_gap_logged_n++] = va; + return true; +} + +/* + * relfix PART 3/4: one rate-limited RELOC-GAP-STAGE line per distinct gap VA + * per launch. `stage` is the resolver source chain (r.stage, e.g. + * "ipin:-2,pin:-2,lat:-2,map:-2,vas:-2,disc:-2,ledger:-2," - the resolving + * source ends in ":0,") for a word no source answered, or a short authoring + * refusal token ("span-overlap", "region-full", "wordmax-full", + * "base-tracked") for a word that resolved but was deliberately not + * authored. A FULL chain of -2 is classified as RELOC-GAP-UNMAPPED instead + * (see gsp_phys_gr_gap_unmapped_log): the sources are exhausted, so the word + * cannot be authored this launch; note the map stage's population is + * range-gated to [SEMA_VA_LO 4 GiB, SEMA_VA_HI 128 GiB) so its -2 is + * uninformative for VAs above that window - the vas/disc WALKS (49-bit + * full-range) are the authoritative absence signal. + */ +static void gsp_phys_gr_gap_stage_log(u64 va, const char *stage, int rc) +{ + if (!gsp_phys_gr_gap_log_once(va)) + return; + pr_info_ratelimited("vgsp-gr: RELOC-GAP-STAGE va=0x%llx stage=%s rc=%d\n", + va, stage, rc); +} + +/* + * relfix PART 4: definitive classification for a shape-passing word that NO + * resolver source answers (chain fully -2, r218: 0x760275000000, + * 0x760273000000, 0x3000000000, 0x17400000000, 0x17c00000000 - every stage + * -ENOENT). The walks cover the full 49-bit range, so a walk miss + no + * pin/cache/ledger entry means no mapping we can see exists anywhere; + * r218 additionally shows these words produce NO engine MMU fault (unlike + * the dereferenced argument words the late path now fixes), i.e. the engine + * does not consume them: they are descriptor/metadata values that are merely + * VA-shaped ("junk" as pointers). Any remaining engine fault is therefore + * NOT from this class - it redirects diagnosis to the code-pointer/entry + * word and page-content paths. + */ +static void gsp_phys_gr_gap_unmapped_log(u64 va) +{ + if (!gsp_phys_gr_gap_log_once(va)) + return; + pr_info_ratelimited("vgsp-gr: RELOC-GAP-UNMAPPED va=0x%llx (all stages -2; VA-shape junk)\n", + va); +} + +/* + * relfix PART 4: bounded content bound for a late-resolved word whose page is + * one of the launch's OWN pages (req->pages[], the only class with ground + * truth). The late copy-in itself is a direct copy of the resolved source + * page (gsp_phys_gr_fill_gap_bounce: src = __va(r.hpa)), so a dst-vs-src + * compare is vacuous; the meaningful question is whether the RESOLVER named + * the same page the launch pipeline uses. Compare the resolved source page + * against req->pages[i].hpa content and log the mission's + * LATE-CONTENT-MISMATCH line once per launch when they differ: that is the + * only way the engine could have been handed wrong bytes through a late + * proxy. Words whose page is not launch-internal have no ground truth and + * are skipped (their resolution is validated by the same walk/pin sources as + * every tracked data page). + */ +static void gsp_phys_gr_late_content_check(const struct gsp_phys_gr_submit *req, + u64 wb, u64 resolved_hpa, u32 off) +{ + u32 pi; + + for (pi = 0; pi < req->n_pages; pi++) { + const u8 *gt, *src; + + if ((req->pages[pi].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != wb) + continue; + if (!req->pages[pi].hpa) + break; + gt = (const u8 *)__va(req->pages[pi].hpa); + src = (const u8 *)__va(resolved_hpa); + if (!gt || !src) + break; + if (memcmp(gt, src, GSP_PHYS_PAGE_SIZE) && + gsp_phys_gr_gap_log_once(wb)) + pr_info_ratelimited("vgsp-gr: LATE-CONTENT-MISMATCH word_va=0x%llx page=0x%llx off=%u src=0x%llx\n", + wb, (u64)req->pages[pi].hpa, off, + resolved_hpa); + break; + } +} + +/* + * relfix: resolve and author one unmatched word (see gsp_phys_gr_gap_page). + * The base VA resolves through the sema service's trusted resolver; on a hit + * the page is queued and the word rewritten to <reloc VA> | offset - the + * exact encoding the matched-word relocation uses - and up to + * GSP_PHYS_GR_GAP_SPAN_MAX contiguous pages the resolver also answers are + * queued behind it so a small operand's base+offset reads stay contiguous. + * PART 3 (r217): that span is a small fetch line (16 pages), NOT a region + * walk - r217's 1242-page spans turned fast fails into 250 s kills; the + * multi-page operand class belongs to the fill's own span pipeline, which + * the PART 3 candidate superset now feeds. A repeated word (same base VA + * authored in another QMD page) reuses its recorded reloc VA. NEVER guesses: + * only a successful resolve rewrites, and a page that already has a proxy + * (the fill's or an earlier fix's) stops the fix - the queued pages of THIS + * word are rolled back and the word stays a RELOC-GAP - so one guest page + * can never gain a second copy-out writer (the clobber class + * gsp_phys_gr_page_tracked describes). Every word that stays a gap is + * stage-logged once (RELOC-GAP-STAGE) with the resolver chain or the + * authoring refusal token. + */ +static bool gsp_phys_gr_reloc_fix_word(const struct gsp_phys_gr_submit *req, + u32 *w, u32 k, u32 *mirror) +{ + u64 v = (u64)w[k] | ((u64)w[k + 1] << 32); + u64 wb = v & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + struct gsp_sema_gr_reloc r; + struct gsp_phys_gr_gap_word *gw = NULL; + u32 gap_start = gsp_phys_gr_gap_n; + u64 reloc_va = 0; + u32 wi, i; + + for (wi = 0; wi < gsp_phys_gr_gap_word_n; wi++) { + if (gsp_phys_gr_gap_word[wi].va == wb) { + gw = &gsp_phys_gr_gap_word[wi]; + break; + } + } + if (!gw) { + bool overlap = false; + u64 base_pa; + + if (gsp_phys_gr_gap_word_n >= GSP_PHYS_GR_GAP_WORD_MAX) { + gsp_phys_gr_gap_stage_log(wb, "wordmax-full", + -ENOSPC); + return false; + } + if (gsp_phys_gr_page_tracked(req, wb)) { + gsp_phys_gr_gap_stage_log(wb, "base-tracked", -EEXIST); + return false; + } + if (gsp_sema_gr_reloc_resolve(wb, &r)) { + /* + * A FULL chain of misses is the definitive + * no-mapping-anywhere class (PART 4); anything else + * (hpa conversion failure) keeps the stage chain. + */ + if (r.stage_rc == -ENOENT) + gsp_phys_gr_gap_unmapped_log(wb); + else + gsp_phys_gr_gap_stage_log(wb, r.stage, + r.stage_rc); + return false; + } + base_pa = r.pa; + /* + * PART 4: the copy-in is a direct copy of this source page; + * for a launch-internal word, bound it against the launch's + * own page content (the LATE-CONTENT-MISMATCH witness). + */ + gsp_phys_gr_late_content_check(req, wb, r.hpa, + (u32)(v & (u64)(GSP_PHYS_PAGE_SIZE - 1))); + if (!gsp_phys_gr_gap_add(req->n_data, wb, r.hpa, &reloc_va)) { + gsp_phys_gr_gap_stage_log(wb, "region-full", -ENOSPC); + return false; + } + { + u32 span_done = 1; /* pages authored this word */ + bool span_capped = false; + + for (i = 1; i < GSP_PHYS_GR_GAP_SPAN_MAX; i++) { + u64 va = wb + ((u64)i << 12); + u64 ignored = 0; + + /* + * A page with an existing proxy cannot get a + * second one (two copy-out writers for one + * guest page would clobber each other); stop + * the span before it. A merely unresolved + * page (the common end-of-buffer case) + * truncates the span like the fill's own span + * walk does. + */ + if (gsp_phys_gr_page_tracked(req, va)) { + overlap = true; + break; + } + if (gsp_sema_gr_reloc_resolve(va, &r)) + break; + if (!gsp_phys_gr_gap_add(req->n_data, va, r.hpa, + &ignored)) { + /* capacity cut: tail cannot be + * authored though more mapping may + * exist - reported below. */ + span_capped = true; + break; + } + span_done++; + } + /* + * COPYAUDIT completeness probe (2026-10-18): the + * span cap is intentional (r217 containment), but a + * word cut at the cap with a still-resolvable next + * page means the engine's base+offset reads past + * the authored run into later operands' bounce + * pages - silent partial data. One probe page past + * the cut proves it and makes the truncation loud. + */ + if (!overlap && span_done == GSP_PHYS_GR_GAP_SPAN_MAX) { + struct gsp_sema_gr_reloc r2; + u64 next_va = wb + + ((u64)GSP_PHYS_GR_GAP_SPAN_MAX << 12); + + if (!gsp_sema_gr_reloc_resolve(next_va, &r2)) + span_capped = true; + } + if (span_capped) + pr_info_ratelimited("vgsp: COPY-TRUNC va=0x%llx want>=0x%llx done=0x%llx path=reloc-fix-span\n", + wb, + ((u64)span_done + 1) << 12, + (u64)span_done << 12); + } + if (overlap) { + /* + * Roll the whole word back (base included) and + * leave it a RELOC-GAP: a partial run would make + * the engine's next base+offset read land on a + * later word's page. + */ + gsp_phys_gr_gap_stage_log(wb, "span-overlap", -EEXIST); + gsp_phys_gr_gap_n = gap_start; + return false; + } + gw = &gsp_phys_gr_gap_word[gsp_phys_gr_gap_word_n]; + gw->va = wb; + gw->reloc_va = gsp_phys_gr_gap[gap_start].reloc_va; + gw->pa = base_pa; + gsp_phys_gr_gap_word_n++; + } + + v = gw->reloc_va | (v & (u64)(GSP_PHYS_PAGE_SIZE - 1)); + w[k] = (u32)v; + w[k + 1] = (u32)(v >> 32); + if (mirror) { + mirror[k] = (u32)v; + mirror[k + 1] = (u32)(v >> 32); + } + gsp_phys_gr_reloc_fixed++; + if (!gsp_phys_gr_reloc_fixed_first) + gsp_phys_gr_reloc_fixed_first = wb; + pr_info_ratelimited("vgsp-gr: RELOC-FIXED va=0x%llx pa=0x%llx off=%u\n", + wb, gw->pa, (u32)(v & (u64)(GSP_PHYS_PAGE_SIZE - 1))); + return true; +} + +/* + * m151: output retirement wait. qmd_rel is a schedule/fetch ack (the front + * end latched and fetched the QMD), not a data-path retirement: the SM's + * stores into the bounce pages can still be in flight when the ack lands. + * Copying out - and the next launch's copy-in - before those stores land + * loses them: the guest observes zeros for a completed launch, or + * stale/mixed content when the next copy-in races the previous engine's + * write tail. + * + * The wait covers EVERY resolved data page (footprint-by-inference was + * tried twice and is unreliable: the fill's candidate list ends with the + * launch's own bank/param pages, so the max-reloc run is 3-4 pages, not + * the output operand). Adaptive exits make a whole-window count cheap: + * - all resolved pages changed -> done (normal case, one poll) + * - changed>0 and stable for 3 polls -> stores settled (masked or + * repeated-identical kernels) + * - no writes at all for 600 ms -> no output for this launch + * - hard cap 1.5 s -> genuinely late remainder + * noinline so this rarely-taken wait does not grow the submit frame, which + * is already near the warning bound. + */ +/* + * m180: scan the request's unique tracked bounce pages and return how many + * differ from the copy-in checksum baseline. `hash_out` (optional) receives + * an FNV-1a fold of the per-page checksums - the content fingerprint the + * settle rule needs so a store ramp with a constant changed-page COUNT can + * never read as settled again. Static scratch: the only caller is the + * single-inflight ordered gsp-phys-work item (same discipline as the other + * submit-scope statics). + */ +static noinline u32 +gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, + const u64 *csum_in, u32 *n_tracked, u64 *hash_out) +{ + static u32 checked[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u32 n_all = 0, di, j, changed = 0; + u64 hash = 1469598103934665603ULL; + + for (di = 0; di < req->n_data; di++) { + u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); + u32 k; + bool dup = false; + + if (bi == GSP_PHYS_GR_DATA_BI_NONE) + continue; + for (k = 0; k < n_all; k++) + if (checked[k] == bi) { + dup = true; + break; + } + if (!dup) + checked[n_all++] = bi; + } + /* + * relfix: late-resolved pages are tracked exactly like fill-tracked + * data pages, so the retire wait, the late-check and the NOWRITE + * classification see stores the engine authors into them. + */ + for (di = 0; di < gsp_phys_gr_gap_n; di++) { + u32 bi = gsp_phys_gr_gap[di].bi; + u32 k; + bool dup = false; + + if (bi >= GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES) + continue; + for (k = 0; k < n_all; k++) + if (checked[k] == bi) { + dup = true; + break; + } + if (!dup) + checked[n_all++] = bi; + } + if (n_tracked) + *n_tracked = n_all; + for (j = 0; j < n_all; j++) { + const u8 *src = gsp_phys_chunk_page_va(dchunk, checked[j]); + u64 fp; + + if (!src) + continue; + /* + * r452: FNV-1a per page, not a 32-bit additive sum. The + * add-sum was PROVABLY blind to a power-of-two in-place + * scale: doubling every word of a 4 KB page shifts each + * exponent by +1, i.e. +2^23 per word, and 1024 words give + * 2^33 == 0 (mod 2^32) - so c[i] *= 2.0f left the checksum + * bit-identical, the page classified "unchanged", and the + * client read the pre-scale values (the deterministic + * streams2/scale class). FNV-1a has no such structure. + */ + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp != csum_in[checked[j]]) + changed++; + hash ^= fp; + hash *= 1099511628211ULL; + } + if (hash_out) + *hash_out = hash; + return changed; +} + +/* + * r434: post-launch deposit dump (hardware-only words). The r430 QMD64 dump + * runs at submission, before the engine touches the deposit, so the + * hardware-only fields (w50 INNER_GET, w51 INNER_PUT, w52 span-list head, + * w53 next-QMD) are always zero there. Reading the same page AFTER the + * retire wait shows what the SKED actually consumed: w50/w51 are the CTA + * (INNER) get/put counters, so a wedged launch whose grid "completed" with + * zero stores is distinguishable from a genuinely dispatched grid by + * whether INNER_PUT advanced to the CTA count. One line per launch, gated + * by the skedrec param. + */ +static void gsp_phys_gr_qmd64_post(const u8 *deposit, const char *tag) +{ + char qb[64 * 9 + 1]; + int qo = 0; + u32 i; + + if (!deposit || !gsp_phys_rmcall_gr_skedrec_value()) + return; + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS && qo < (int)sizeof(qb) - 10; + i++) + qo += scnprintf(qb + qo, sizeof(qb) - qo, "%08x ", + get_unaligned_le32(deposit + i * 4)); + pr_info("vgsp-gr: QMD64-POST %s %s\n", tag, qb); +} + +static noinline u32 +gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, + const u64 *csum_in, u32 *n_tracked) +{ + u32 n_all = 0, iter; + u32 changed = 0; + u64 prev_hash = 0; + const char *why = " (TIMEOUT)"; + unsigned long wait_start = jiffies; + + /* + * m179: report how many pages were actually trackable. "wrote == 0" + * from a launch with n_tracked == 0 (no data entry resolves to a + * bounce page) is not a no-execution signature and must not trigger + * the no-write retry/accept path. + */ + if (n_tracked) + *n_tracked = 0; + if (!req->n_data) + return 0; + /* + * r343: scan_pages RETURNS the changed count and reports the tracked + * total through the out-param. The old call put the return (changed) + * into n_all while the tracker total went to *n_tracked, so the + * loop's `changed >= n_all` was `changed >= changed` - always true + * after the first scan (observed: "1/1 pages changed after 4 ms"), + * and the `!n_all` early-return also skipped the no-write settle + * when zero pages changed. Net effect: the copy-out shipped the + * bounce mid-ramp, so a cold engine (slowest store ramp) produced + * all-bad first runs (the matmul cold flake) and late stores could + * be dropped. Fix: tracked total into n_all; the content-stable + * rule below becomes the real settle bound. + */ + (void)gsp_phys_gr_scan_pages(req, csum_in, &n_all, NULL); + /* + * r405: the r343 fix moved the tracked total from the out-param to + * the local n_all but FORGOT to write it back, so *n_tracked stayed + * 0 for every launch and the whole no-write classification chain in + * the caller (`wrote == 0 && n_tracked`: first-attempt witness, + * outcome retries, NOWRITE accept + release restore, late-store + * check) has been dead since ed91540a/r345 - the r404 forensics + * show "RETIRE-WIT n_data=968 n_tracked=0" while the same wait + * reports n_all=1032 in the "0/1032 pages changed (no writes)" + * line. Propagate the tracked total so the classifier can fire. + */ + if (n_tracked) + *n_tracked = n_all; + /* + * RETIRE-WIT (r227): the r227 capture shows n_tracked far below the + * copy-in/copy-out rec counts for the same launch (4 tracked vs 1482 + * written back) - either the rec array genuinely aliases reloc VAs + * mid-launch or the observed kernel differs from this source. One + * line per launch proves which: n_data, the unique-bi count, and the + * first/last recs' va/reloc_va/bi must be sequential and in-region + * for a healthy launch. + */ + if (gsp_phys_gr_copywit_enabled()) + pr_info_ratelimited("vgsp-gr: RETIRE-WIT n_data=%u n_tracked=%u first_va=0x%llx first_rel=0x%llx first_bi=%u last_va=0x%llx last_rel=0x%llx last_bi=%u\n", + req->n_data, n_tracked ? *n_tracked : 0, + req->data[0].va, req->data[0].reloc_va, + gsp_phys_gr_data_bi(req->data[0].reloc_va), + req->data[req->n_data - 1].va, + req->data[req->n_data - 1].reloc_va, + gsp_phys_gr_data_bi(req->data[req->n_data - 1].reloc_va)); + if (!n_all) + return 0; + for (iter = 0; iter < 300u; iter++) { + u64 hash = 0; + + changed = gsp_phys_gr_scan_pages(req, csum_in, NULL, &hash); + if (changed >= n_all) { + why = ""; + break; + } + /* + * m180 (H3): a partial count must be CONTENT-stable for + * ~300 ms before it counts as settled. The m162 count-only + * rule accepted a continuing store ramp whose page count + * happened to be constant (the always-failing class's + * "17/32 settled" copy-outs), and accepted an + * un-dispatched grid as settled. A genuinely masked/short + * kernel costs at most 300 ms extra; a ramping engine gets + * the time its stores need. + */ + if (changed > 0 && hash == prev_hash && + iter >= gsp_phys_gr_settle_iters()) { + why = " (settled)"; + break; + } + if (changed == 0 && iter >= 120u) { + why = " (no writes)"; + break; + } + prev_hash = hash; + usleep_range(4000, 6000); + } + pr_info("vgsp-gr: output retirement wait: %u/%u pages changed after %u ms%s\n", + changed, n_all, jiffies_to_msecs(jiffies - wait_start), why); + /* + * r405: run-once late-write probe (see the param comment above). + */ + if (READ_ONCE(gsp_phys_gr_late_probe_param) && changed < n_all) { + u32 late_n = 0, late; + unsigned long lp_start = jiffies; + + msleep(500); + late = gsp_phys_gr_scan_pages(req, csum_in, &late_n, NULL); + pr_info("vgsp-gr: retire LATE-PROBE %u -> %u of %u after +%u ms%s\n", + changed, late, late_n, + jiffies_to_msecs(jiffies - lp_start), + late != changed ? + " (ENGINE STILL WRITING - stores landed late)" : + " (stable - writes are final)"); + } + return changed; +} + +/* + * m161: hardware fetch witness. The host writes GSP_PHYS_GR_QMD_SENTINEL + * into the inline-QMD page before every deposit; the engine's QMD fetch is + * the only actor that overwrites it (the post-mortem has used this as its + * ran/not-run marker since m110). Computing it as a standalone check lets + * the outcome-retry path classify a zero-write result beyond the release + * word, which the sema service's pending-release sweep can satisfy in + * software. + */ +static bool gsp_phys_gr_qmd_fetched(const u8 *qmd_mem) +{ + if (!qmd_mem) + return false; + return get_unaligned_le32(qmd_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4) != + GSP_PHYS_GR_QMD_SENTINEL; +} + +/* + * r427 idempotent fast-accept (CONT.86 §6.1). A zero-write launch is either + * the no-execute wedge OR a legitimate idempotent re-launch (the same QMD + * content with the same staged inputs, whose result is already in place) - + * the host cannot tell them apart, and both burn the 40 s + 80 s quiet holds. + * The launch identity below (QMD minus the per-launch release0 address and + * payload, folded with the staged content checksums of every tracked page) is + * recorded for every launch that WROTE; a zero-write launch whose identity + * matches an entry accepts immediately, with no retries and no quiet window. + * Safe by construction: identical code + identical staged content + a prior + * execution that produced this launch's own footprint means the correct + * result is already in the buffers; a side-effecting kernel's re-launch is + * equally wrong under the current accept (the client sees the stale result + * either way) - this only removes the stall. + */ +#define GSP_PHYS_GR_IDEM_SLOTS 64u +static u64 gsp_phys_gr_idem_fp[GSP_PHYS_GR_IDEM_SLOTS]; +static u32 gsp_phys_gr_idem_n; + +static u64 gsp_phys_gr_launch_fp(const u32 *qmd, + const struct gsp_phys_gr_submit *req, + const u64 *csum_in) +{ + u64 h = 1469598103934665603ULL; + u32 i; + + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS; i++) { + /* Skip release0's address (w23/w24) and payload (w25): the + * driver allocates a fresh release slot per launch, so they + * differ even for an identical re-launch. */ + u32 w; + + if (i == 23u || i == 24u || i == 25u) + continue; + w = qmd[i]; + h ^= w; + h *= 1099511628211ULL; + } + for (i = 0; i < req->n_data; i++) { + u32 bi = gsp_phys_gr_data_bi(req->data[i].reloc_va); + + if (bi == GSP_PHYS_GR_DATA_BI_NONE) + continue; + h ^= csum_in[bi]; + h *= 1099511628211ULL; + } + return h; +} + +static bool gsp_phys_gr_idem_match(u64 fp) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_idem_n; i++) + if (gsp_phys_gr_idem_fp[i] == fp) + return true; + return false; +} + +static void gsp_phys_gr_idem_record(u64 fp) +{ + u32 i; + + for (i = 0; i < gsp_phys_gr_idem_n; i++) + if (gsp_phys_gr_idem_fp[i] == fp) + return; + if (gsp_phys_gr_idem_n < GSP_PHYS_GR_IDEM_SLOTS) { + gsp_phys_gr_idem_fp[gsp_phys_gr_idem_n++] = fp; + } else { + /* ring: drop the oldest */ + memmove(&gsp_phys_gr_idem_fp[0], &gsp_phys_gr_idem_fp[1], + (GSP_PHYS_GR_IDEM_SLOTS - 1u) * sizeof(u64)); + gsp_phys_gr_idem_fp[GSP_PHYS_GR_IDEM_SLOTS - 1u] = fp; + } +} + +/* + * COPYAUDIT (2026-10-18 copy-completeness audit): per-copy sighting of the + * GR operand staging copies. Every staged page is spot-checked (16 bytes + * at the start and end of the page, source vs destination, right after the + * memcpy) and any difference logs COPY-MISMATCH; the callers log the + * per-launch COPYAUDIT aggregate when the staged set exceeds 32 KB, and + * COPY-TRUNC whenever fewer pages were staged than the launch declared + * (want>done). Purely diagnostic: no control flow, locking or timing + * change; the memcmp reads memory the memcpy just touched. + */ +#define GSP_PHYS_GR_AUDIT_MIN (32u << 10) /* >32 KB staged sets */ + +static void gsp_phys_gr_page_spot_check(const u8 *dst, const u8 *src, + u64 va, const char *path) +{ + if (memcmp(dst, src, 16) || + memcmp(dst + GSP_PHYS_PAGE_SIZE - 16, + src + GSP_PHYS_PAGE_SIZE - 16, 16)) + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x%llx path=%s\n", + va, 0ull, path); +} + +/* + * r452: the legacy 32-bit additive page sum, kept ONLY for the all-zero + * filter. It is never a change-detection fingerprint any more: doubling every + * word of a 4 KB page leaves the sum bit-identical (1024 x 2^23 == 2^33 == 0 + * mod 2^32), which silently classified an in-place x2 scale as "no engine + * store" and shipped stale values (the deterministic streams2 class). + */ +static inline u32 gsp_phys_gr_page_sum32(const u8 *p) +{ + u32 ci, csum = 0; + + for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) + csum += get_unaligned_le32(p + ci); + return csum; +} + +/* + * COPYWIT r226 zero filter: an all-zero page is never a meaningful alias + * partner. A fresh client's fill stages long runs of zero pages + * (uninitialized cudaMalloc buffers), and consecutive zero pages produced + * the IN-ALIAS-WIT false positives of the r226a green run (csum=0x00000000 + * sequences). A page counts as zero only when its csum is zero AND every + * word is zero (csum==0 alone can hide cancelling contents). + */ +static inline bool gsp_phys_gr_page_all_zero(u32 csum, const u8 *p) +{ + u32 i; + + if (csum) + return false; + for (i = 0; i < GSP_PHYS_PAGE_SIZE; i += 4) + if (get_unaligned_le32(p + i)) + return false; + return true; +} + +/* + * COPYWIT r22x INPUT-ALIAS-WIT: per-launch census of the copy-in recs (the + * exact VA -> HPA -> content triples the engine will read for this launch, + * inputs included). Each new non-zero rec is compared against the last + * GSP_PHYS_GR_IN_CENSUS recs: + * - distinct VA resolving to the SAME host page (hpa collision) is the + * resolver-aliasing smoking gun: several guest pages served by one host + * page would replicate those rows through every consumer (the observed + * band-replication signature); + * - distinct HPA with identical content is the upstream-write-clone class + * (an H2D that landed on one page, or guest pages that genuinely hold + * the same bytes). + * Zero pages are excluded on both sides (fresh-client zero runs are not + * aliasing evidence). Bounded boot-wide; same single-inflight discipline. + */ +#define GSP_PHYS_GR_IN_CENSUS 64u +static u32 gsp_phys_gr_in_census_log; + +/* + * m153: fill the bounce window from the request's guest pages and capture a + * per-page checksum baseline. Used before the first publish and again before + * every outcome retry (a mis-executing first attempt can leave stale or + * wrong-window content behind, so a retry must start from the guest inputs + * again). Returns the number of pages staged. + */ +static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, + u64 *csum_in, u64 *staged_hpa) +{ + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + struct { + u64 va; + u64 hpa; + u32 csum; + } census[GSP_PHYS_GR_IN_CENSUS]; + u32 n_census = 0; + u32 di, copied = 0; + + for (di = 0; di < req->n_data; di++) { + u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); + const u8 *src; + u8 *dst; + u32 ci, csum = 0; + bool zero; + + if (bi == GSP_PHYS_GR_DATA_BI_NONE || !req->data[di].hpa) + continue; + src = (const u8 *)__va(req->data[di].hpa); + dst = gsp_phys_chunk_page_va(dchunk, bi); + if (!dst) + continue; + memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + /* + * r378: record which guest page's content now occupies this + * bounce slot. The copy-out may only write the slot back to + * that page - a rec whose reloc VA aliases another rec's slot + * would otherwise ship the other page's content into this + * guest page (the multi-writer clobber class). + */ + staged_hpa[bi] = req->data[di].hpa; + gsp_phys_gr_page_spot_check(dst, src, req->data[di].va, + "gr-bounce-copy-in"); + for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) + csum += get_unaligned_le32(dst + ci); + /* r452: the change baseline is the FNV-1a fingerprint (the + * add-sum above survives only as the all-zero test). */ + csum_in[bi] = gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); + zero = gsp_phys_gr_page_all_zero(csum, dst); + if (!zero && gsp_phys_gr_copywit_enabled()) { + u32 j; + + for (j = 0; j < n_census; j++) { + if (census[j].hpa == req->data[di].hpa && + census[j].va != req->data[di].va) { + if (gsp_phys_gr_in_census_log < 24) { + gsp_phys_gr_in_census_log++; + pr_info("vgsp-gr: INPUT-ALIAS-WIT kind=hpa-collision va=0x%llx hpa=0x%llx prev_va=0x%llx (two guest pages served by one host page)\n", + req->data[di].va, + req->data[di].hpa, + census[j].va); + } + } else if (census[j].csum == csum && + census[j].hpa != req->data[di].hpa) { + if (gsp_phys_gr_in_census_log < 24) { + gsp_phys_gr_in_census_log++; + pr_info("vgsp-gr: INPUT-ALIAS-WIT kind=content-clone va=0x%llx hpa=0x%llx prev_va=0x%llx prev_hpa=0x%llx csum=0x%08x\n", + req->data[di].va, + req->data[di].hpa, + census[j].va, + census[j].hpa, + csum); + } + } + } + if (n_census < GSP_PHYS_GR_IN_CENSUS) { + census[n_census].va = req->data[di].va; + census[n_census].hpa = req->data[di].hpa; + census[n_census].csum = csum; + n_census++; + } + } + copied++; + } + dma_wmb(); + return copied; +} + +/* + * relfix: copy the late-resolved pages' guest content into their bounce + * pages and capture the copy-in checksum baseline, exactly as + * gsp_phys_gr_fill_bounce does for the fill's tracked data pages. Called + * before the doorbell (initial copy-in) and again before every outcome + * retry, so a retried launch starts from the guest inputs in both sets. + * Returns the number of pages staged. + */ +static u32 gsp_phys_gr_fill_gap_bounce(u64 *csum_in, u64 *staged_hpa) +{ + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u32 gi, copied = 0; + + for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { + const u8 *src = (const u8 *)__va(gsp_phys_gr_gap[gi].hpa); + u8 *dst = gsp_phys_chunk_page_va(dchunk, gsp_phys_gr_gap[gi].bi); + + if (!src || !dst) + continue; + memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + /* r378: the slot now holds THIS page's content (see the + * tracked-page fill above). */ + staged_hpa[gsp_phys_gr_gap[gi].bi] = gsp_phys_gr_gap[gi].hpa; + gsp_phys_gr_page_spot_check(dst, src, gsp_phys_gr_gap[gi].va, + "gr-bounce-copy-in"); + /* r452: FNV-1a baseline (see gsp_phys_gr_scan_pages). */ + csum_in[gsp_phys_gr_gap[gi].bi] = + gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); + copied++; + } + dma_wmb(); + return copied; +} + +/* + * F3 retry hygiene: an outcome retry re-publishes the same QMD. If the QMD's + * pointer pairs name the win0 shadow bank the previous attempt used, move the + * already-prepared param/program/bank pages into the alternate bank and + * rewrite the pairs (and the program prefetch) to the fresh VAs. The copies + * come from the module shadow pages - after relocation, inline replay and + * pointer fix-ups - so the engine-visible content is byte-identical to the + * first attempt and only the VA changes; the engine therefore cannot serve + * the retry from a cached fetch line of the previous attempt's bank. With + * gsp_phys_rmcall_gr_ptr_guest the pairs hold guest VAs, match nothing here, + * and the engine keeps reading the guest VAs exactly as before. Returns true + * when at least one pair moved (the caller must rebuild the method block). + */ +static bool gsp_phys_gr_win0_retry_rearm(u32 *qmd, + const struct gsp_phys_gr_submit *req, + u32 old_bank_base, u32 new_bank_base, + u32 new_bank_limit) +{ + struct gsp_phys_boot_chunk *pool = &boot_stage.rmcall_gr_shadow_win0; + u64 old_lo = GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)old_bank_base * GSP_PHYS_PAGE_SIZE; + u64 old_hi = old_lo + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS * + GSP_PHYS_PAGE_SIZE; + u32 next_slot = new_bank_base; + u32 i; + bool moved = false; + + if (!pool->pages) + return false; + for (i = 0; i < req->n_pages; i++) { + const struct gsp_phys_gr_page *pg = &req->pages[i]; + u32 w = pg->qmd_word; + u64 pair_va, new_va; + u32 off, old_slot; + u8 *src, *dst; + + if (w != 32u && w != 48u && + !(w >= 34u && w <= 46u && !(w & 1u))) + continue; + pair_va = ((u64)(qmd[w + 1] & 0x1ffffu) << 32) | qmd[w]; + if (pair_va < old_lo || pair_va >= old_hi) + continue; + if (next_slot >= new_bank_limit) + break; + off = (u32)(pair_va & (GSP_PHYS_PAGE_SIZE - 1)); + old_slot = (u32)((pair_va - + GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE) / + GSP_PHYS_PAGE_SIZE); + src = gsp_phys_chunk_page_va(pool, old_slot); + dst = gsp_phys_chunk_page_va(pool, next_slot); + if (!src || !dst) + break; + memcpy(dst, src, GSP_PHYS_PAGE_SIZE); + new_va = GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)next_slot * GSP_PHYS_PAGE_SIZE + off; + qmd[w] = (u32)new_va; + qmd[w + 1] = (qmd[w + 1] & ~0x1ffffu) | + ((u32)(new_va >> 32) & 0x1ffffu); + if (w == 48u) { + u64 pref = ((u64)(qmd[31] & 0x1ffu) << 40) | + ((u64)qmd[8] << 8); + + if ((pref & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == + (pair_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) { + u64 np = new_va | + (pref & (GSP_PHYS_PAGE_SIZE - 1)); + + qmd[8] = (u32)(np >> 8); + qmd[31] = (qmd[31] & ~0x1ffu) | + ((u32)(np >> 40) & 0x1ffu); + } + } + next_slot++; + moved = true; + } + if (moved) + dma_wmb(); + return moved; +} + +/* + * F4-a: the first poll break, rendered from a noinline helper so its + * addresses/counter scratch never grow the (already warning-bound) submit + * frame. The USERD GP_GET/GP_PUT at the break discriminate an engine ack + * (GP_GET advanced, break latency long enough for a real retirement) from a + * host-authored word (break within a few ms of the doorbell, GP_GET frozen). + */ +static noinline void +gsp_phys_gr_poll_break_log(u32 w0, u32 want_qmd, u32 w1, u32 since_us, + const struct gsp_phys_rmcall_ce_submit *sub) +{ + struct gsp_phys_boot_chunk *ud_c = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; + u8 *ud = ud_c->pages ? gsp_phys_chunk_page_va(ud_c, 0) : NULL; + u32 usr_gpget = ud ? + get_unaligned_le32(ud + gsp_phys_rmcall_userd_ctl_off() + + GSP_PHYS_RMCALL_USERD_GPGET_OFF) : 0; + + pr_info("vgsp-gr: poll break w0=0x%x want=0x%x ch=0x%x after %u us usr_gpget=0x%x gpget=0x%x gpput=0x%x\n", + w0, want_qmd, w1, + (u32)ktime_to_us(ktime_get()) - since_us, + usr_gpget, sub->gpget, sub->gpput); + /* r425: the healthy-launch SKED/FECS baseline for the wedge compare. */ + gsp_phys_gr_sked_wit(gsp_phys_gpu_bar0(), "post-poll"); +} + +/* + * F4-b: one-shot deposit readback at the first timeout (noinline for the same + * frame reason as above). Reads the qmdwin/shadow destination (or the guest + * QMD page) with the booter's existing unaligned host readback; w63 is the + * engine fetch sentinel. + */ +static noinline void +gsp_phys_gr_deposit_rb_log(const struct gsp_phys_gr_submit *req, + u8 *qmd_dst_mem) +{ + u8 *qp = qmd_dst_mem ? qmd_dst_mem : + (req->qmd_hpa ? (u8 *)__va(req->qmd_hpa) : NULL); + + if (!qp) + return; + pr_info("vgsp-gr: deposit readback w0=0x%08x w4=0x%08x w12=0x%08x w13=0x%08x w18=0x%08x w19=0x%08x w23=0x%08x w32=0x%08x w33=0x%08x w48=0x%08x w49=0x%08x w63=0x%08x\n", + get_unaligned_le32(qp), + get_unaligned_le32(qp + 16), + get_unaligned_le32(qp + 48), + get_unaligned_le32(qp + 52), + get_unaligned_le32(qp + 72), + get_unaligned_le32(qp + 76), + get_unaligned_le32(qp + 92), + get_unaligned_le32(qp + 128), + get_unaligned_le32(qp + 132), + get_unaligned_le32(qp + 192), + get_unaligned_le32(qp + 196), + get_unaligned_le32(qp + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4)); +} + +/* + * CODEHASH (2026-10-18): code-fetch content verification. At author time, + * sample the host-readable bytes the engine is about to fetch for this + * launch's program (the QMD word 48/49 pair): the first 16 B at the entry + * offset plus 16 B at PA+0 / PA+mid / PA+end of the entry page, and the + * same three windows on up to three further consecutive tracked pages of + * the code run. One bounded line per launch, emitted when the content is + * suspicious (all-zero samples, or the first four entry words identical - + * a page that never landed or a repeated filler word) or for the first few + * healthy launches per boot (so the healthy signature is visible without + * flooding). A missing host-readable landing (hpa 0 on the entry page) + * logs CODEHASH-UNREADABLE. sh0 is the engine-visible shadow page's entry + * word when the Tier-2 copy exists (NULL on the Tier-1/guest path). + * noinline: keeps the (warning-bound) submit frame unchanged. + */ +static bool gsp_phys_gr_codehash_done; +static noinline void +gsp_phys_gr_codehash(const struct gsp_phys_gr_submit *req, + const struct gsp_phys_gr_page *pg, const u8 *sh_page) +{ + static u32 ch_checked, ch_logged, ch_unread; + u64 va, h0 = 0, ent = 0, sh0 = 0; + u32 off, n = 0, w[4] = { 0, 0, 0, 0 }; + bool zero3 = false, suspect = false; + const u8 *p = NULL; + u32 i; + + if (gsp_phys_gr_codehash_done) + return; + gsp_phys_gr_codehash_done = true; + ch_checked++; + off = (u32)(req->qmd[48] & (GSP_PHYS_PAGE_SIZE - 1)); + va = (u64)req->qmd[48] | ((u64)(req->qmd[49] & 0x1ffffu) << 32); + if (!pg->hpa) { + if (ch_unread < 32) { + ch_unread++; + pr_info("vgsp-gr: CODEHASH-UNREADABLE va=0x%llx off=0x%x (entry page has no host-readable landing)\n", + va, off); + } + return; + } + p = (const u8 *)__va(pg->hpa); + n = 1; + ent = get_unaligned_le64(p + off); + h0 = get_unaligned_le64(p); + for (i = 0; i < 4; i++) + w[i] = get_unaligned_le32(p + off + i * 4); + zero3 = !(get_unaligned_le64(p) | get_unaligned_le64(p + 8) | + get_unaligned_le64(p + 0x800) | + get_unaligned_le64(p + 0x808) | + get_unaligned_le64(p + 0xff0) | + get_unaligned_le64(p + 0xff8)); + for (i = 1; i < 4; i++) { + u64 tva = pg->guest_va + ((u64)i << 12); + u32 j; + + for (j = 0; j < req->n_pages; j++) + if (req->pages[j].guest_va == tva && + req->pages[j].hpa) + break; + if (j == req->n_pages) + break; + { + const u8 *q = (const u8 *)__va(req->pages[j].hpa); + + n++; + if (get_unaligned_le64(q) || get_unaligned_le64(q + 8) || + get_unaligned_le64(q + 0x800) || + get_unaligned_le64(q + 0x808) || + get_unaligned_le64(q + 0xff0) || + get_unaligned_le64(q + 0xff8)) + zero3 = false; + } + } + if (sh_page) + sh0 = get_unaligned_le64(sh_page + off); + /* + * Conservative plausibility test: all three sampled windows zero; or + * the first two entry words zero; or the first four entry words + * identical (an all-zero page also lands here). + */ + suspect = zero3 || (!w[0] && !w[1]) || + (w[0] == w[1] && w[1] == w[2] && w[2] == w[3]); + if (!suspect && ch_logged >= 3) + return; + if (ch_logged >= 64) + return; + ch_logged++; + pr_info("vgsp-gr: CODEHASH va=0x%llx pa=0x%llx n=%u h0=0x%016llx ent=0x%016llx off=0x%x w=%08x %08x %08x %08x sh0=0x%016llx %s (first 16B)\n", + va, pg->hpa, n, h0, ent, off, w[0], w[1], w[2], w[3], sh0, + suspect ? "SUSPECT" : "ok"); +} + +/* + * M91 FAULT-CAP (2026-10-18): first-fault capture for the transient SM + * instruction-fetch class ("m91 class": the first attempt of a mid-sequence + * launch faults on SM garbage - "Illegal Instruction Encoding" / "Out Of + * Range Register" - while a retry of the identical QMD and identical + * shadow content succeeds). The submit path calls this once per launch at + * the end of the launch window (poll break or timeout) with the retry + * levers that fired; it emits one vgsp-m91: FAULT-CAP line per faulting + * launch carrying: the fault-epoch delta since the launch started, the + * forwarded QMD's address and first 64 B, the live program pair (w48/49) + * with its resolved engine-visible VA/PA, the first 64 B at that code + * address (the bytes the SM will fetch), the retry decision and outcome, + * and the newest relay-decoded 0x1006 note (exceptType/chid/errString). + * Statically inert on a fault-free launch: one READ_ONCE of the fault + * epoch and one compare, no reads of content or engine state. Full dumps + * are capped at GSP_PHYS_M91_CAP_MAX per module-load cycle, followed by a + * single suppression notice; the existing vgsp-fault lines keep decoding + * every fault either way. Static state follows the other per-launch submit + * scratch: the single-inflight ordered work item is the only caller. + */ +#define GSP_PHYS_M91_CAP_MAX 20u +static u32 gsp_phys_m91_fault_start; +static u32 gsp_phys_m91_cap_used; +static bool gsp_phys_m91_cap_suppressed; +static bool gsp_phys_m91_icache_log; +static bool gsp_phys_r449_ccache_log; +static bool gsp_phys_gr_rel_membar_log; +static noinline void +gsp_phys_gr_m91_fault_cap(const struct gsp_phys_gr_submit *req, + const u32 *qmd, u64 qmd_dst, const u8 *qmd_dst_mem, + u32 attempts, bool rearmed, bool fast, bool ok) +{ + u32 faults = READ_ONCE(gsp_phys_fault_epoch) - gsp_phys_m91_fault_start; + const struct gsp_phys_fault_info *fn; + const u8 *code = NULL; + const char *src = "unresolved"; + u64 pva, ppa = 0; + u32 off = 0, cn = 0, i; + u8 qbuf[64], cbuf[64]; + char qhex[2 * 64 + 1], chex[2 * 64 + 1]; + + /* INERT until a fault: fault-free launches take one compare and return. */ + if (!faults || !req || !qmd) + return; + if (gsp_phys_m91_cap_used >= GSP_PHYS_M91_CAP_MAX) { + if (!gsp_phys_m91_cap_suppressed) { + gsp_phys_m91_cap_suppressed = true; + pr_info("vgsp-m91: FAULT-CAP suppressed: %u dumps already emitted this cycle (the existing vgsp-fault lines still decode every fault)\n", + GSP_PHYS_M91_CAP_MAX); + } + return; + } + gsp_phys_m91_cap_used++; + + /* First 64 B of the forwarded QMD (qmdwin copy preferred). */ + memset(qbuf, 0, sizeof(qbuf)); + if (qmd_dst_mem) { + memcpy(qbuf, qmd_dst_mem, sizeof(qbuf)); + } else if (req->qmd_hpa) { + memcpy(qbuf, (const u8 *)__va(req->qmd_hpa), sizeof(qbuf)); + } + bin2hex(qhex, qbuf, sizeof(qbuf)); + qhex[sizeof(qhex) - 1] = '\0'; + + /* Program pair (QMD w48/49) and its engine-visible resolution. */ + pva = ((u64)(qmd[49] & 0x1ffffu) << 32) | qmd[48]; + off = (u32)(pva & (GSP_PHYS_PAGE_SIZE - 1)); + if (boot_stage.rmcall_gr_shadow_win0.pages && + pva >= GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE && + pva < GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE + + (u64)GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS * GSP_PHYS_PAGE_SIZE) { + u32 slot = (u32)((pva - GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE) / + GSP_PHYS_PAGE_SIZE); + + if (slot < boot_stage.rmcall_gr_shadow_win0.pages) { + code = gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_shadow_win0, slot); + ppa = (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_shadow_win0, slot); + src = "win0"; + } + } else { + u64 base = pva & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + for (i = 0; i < boot_stage.rmcall_gr_alias_count; i++) { + if (boot_stage.rmcall_gr_alias_va[i] == base) { + ppa = boot_stage.rmcall_gr_alias_hpa[i]; + code = (const u8 *)__va(ppa); + src = "shadow-line"; + break; + } + } + for (i = 0; req->pages && !code && i < req->n_pages; i++) { + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != base || + !req->pages[i].hpa) + continue; + ppa = req->pages[i].hpa; + code = (const u8 *)__va(ppa); + src = "guest-hpa"; + } + } + memset(cbuf, 0, sizeof(cbuf)); + memset(chex, 0, sizeof(chex)); + if (code) { + cn = min_t(u32, sizeof(cbuf), GSP_PHYS_PAGE_SIZE - off); + memcpy(cbuf, code + off, cn); + bin2hex(chex, cbuf, cn); + chex[sizeof(chex) - 1] = '\0'; + } + + /* Newest relay-decoded 0x1006 (valid whenever the epoch advanced). */ + fn = &gsp_phys_fault_notes[(gsp_phys_fault_note_next + + GSP_PHYS_FAULT_NOTE_MAX - 1u) % + GSP_PHYS_FAULT_NOTE_MAX]; + pr_info("vgsp-m91: FAULT-CAP ok=%u faults=%u retry=%s attempts=%u qmd=0x%llx qmd64=%s prog_w48=0x%08x w49=0x%08x prog_va=0x%llx prog_pa=0x%llx code_src=%s off=0x%x code64=%s fault_et=0x%x fault_chid=%u fault_err=\"%.48s\"\n", + ok ? 1u : 0u, faults, + fast ? "fast-rearm" : (rearmed ? "resched/fresh-entry" : "none"), + attempts, qmd_dst, qhex, qmd[48], qmd[49], pva, ppa, src, off, + cn ? chex : "unresolved", + fn->valid ? fn->except_type : 0u, + fn->valid ? fn->chid : 0u, + fn->valid ? fn->err : "no relayed 0x1006 note"); +} + +/* + * CODESRC (2026-10-18, r221 follow-up): conservative refusal of a + * non-plausible Tier-2 program-pair source. The program page (QMD w48/49) + * is copied verbatim into the shadow the engine fetches from (win0 or the + * Tier-2 line); a source that is empty, PTE-shaped or one of the engine + * page-tree's own module pages would execute PTE/filler bytes as SASS. + * Detection is deliberately narrow so it is statically inert on every + * observed launch (greens included): + * (a) empty: the entry 16 B AND all three sampled windows are zero + * (a page that never landed anywhere the engine starts); + * (b) PTE page: >=12 of the 16 8-byte entries at the entry offset are + * identically-flagged valid PTEs (dense table page), or the run is + * >=12 zero entries with only PTE-signed entries (sparse table); + * (c) module page: the source HPA is a page of one of our own page-tree + * chunks (main PT, guest PT pool, Tier-2 shadow pools, qmdwin) - a + * guest page can never legitimately resolve there. + * A refused source is re-resolved through the late resolver + * (gsp_sema_gr_reloc_resolve: pins/map/walks before the ledger, the + * guest-walk-first counterpart of the fill's ledger-first order). Only a + * different, plausibly-code page is adopted (mode ALT; the engine-visible + * copy is taken from it); anything else leaves the QMD pair at the guest + * VA (mode GUEST_VA, the ptr_guest behaviour for this pair). One + * rate-limited CODESRC-SUSPECT line per detection. + */ +#define GSP_PHYS_GR_CODESRC_OK 0 +#define GSP_PHYS_GR_CODESRC_ALT 1 +#define GSP_PHYS_GR_CODESRC_GUEST_VA 2 + +static bool +gsp_phys_gr_codesrc_chunk_has(const struct gsp_phys_boot_chunk *c, u64 hpa) +{ + u32 i, n; + + if (!c || !c->pages) + return false; + n = c->pages < 32u ? c->pages : 32u; + for (i = 0; i < n; i++) { + u64 d = (u64)gsp_phys_chunk_page_dma(c, i); + + if (d && (d & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == hpa) + return true; + } + return false; +} + +static bool gsp_phys_gr_codesrc_modpt(u64 hpa) +{ + u32 i; + + if (gsp_phys_gr_codesrc_chunk_has( + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PT], hpa)) + return true; + for (i = 0; i < GSP_PHYS_RMCALL_GR_PT_PAGES; i++) + if (gsp_phys_gr_codesrc_chunk_has(&boot_stage.rmcall_gr_pt[i], + hpa)) + return true; + return gsp_phys_gr_codesrc_chunk_has(&boot_stage.rmcall_gr_shadow, + hpa) || + gsp_phys_gr_codesrc_chunk_has( + &boot_stage.rmcall_gr_shadow_win0, hpa) || + gsp_phys_gr_codesrc_chunk_has(&boot_stage.rmcall_gr_qmdwin, hpa); +} + +/* Entry sampling window: 16 8-byte entries starting at the entry offset. */ +static bool gsp_phys_gr_codesrc_ptelike(const u8 *p, u32 off) +{ + u32 start = off & ~7u, i, hits = 0, zeros = 0, nz = 0, flag = 0; + bool flag_set = false, sig_ok = true; + + if (start + 128u > GSP_PHYS_PAGE_SIZE) + start = GSP_PHYS_PAGE_SIZE - 128u; + for (i = 0; i < 16u; i++) { + u64 v = get_unaligned_le64(p + start + i * 8); + u32 lo = (u32)(v & 0xffu); + /* VALID, aperture not PEER(1): the PTE shape family. */ + bool pte = (v & 1u) && (((v >> 1) & 0x3u) != 1u); + + if (!v) { + zeros++; + continue; + } + nz++; + if (!pte) { + sig_ok = false; + continue; + } + if (!flag_set) { + flag = lo; + flag_set = true; + } + if (lo == flag) + hits++; + } + if (hits >= 12u) /* dense table page: a uniform valid-PTE run */ + return true; + /* sparse table page: mostly zero, only PTE-signed entries */ + return sig_ok && nz >= 1u && nz <= 4u && zeros >= 12u; +} + +static bool gsp_phys_gr_codesrc_empty(const u8 *p, u32 off) +{ + const u8 *e = p + off; + + return !(get_unaligned_le64(e) | get_unaligned_le64(e + 8)) && + !(get_unaligned_le64(p) | get_unaligned_le64(p + 8) | + get_unaligned_le64(p + 0x800) | get_unaligned_le64(p + 0x808) | + get_unaligned_le64(p + 0xff0) | get_unaligned_le64(p + 0xff8)); +} + +static noinline int +gsp_phys_gr_code_src(const struct gsp_phys_gr_submit *req, + const struct gsp_phys_gr_page *pg, u64 *alt_hpa) +{ + u64 page = pg->guest_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u32 off = (u32)(req->qmd[48] & (GSP_PHYS_PAGE_SIZE - 1)); + const u8 *p = (const u8 *)__va(pg->hpa); + const char *why = NULL, *used; + struct gsp_sema_gr_reloc r; + int mode = GSP_PHYS_GR_CODESRC_GUEST_VA; + + *alt_hpa = 0; + if (off > GSP_PHYS_PAGE_SIZE - 16u) + off = GSP_PHYS_PAGE_SIZE - 16u; /* keep 16 B in-page */ + if (gsp_phys_gr_codesrc_empty(p, off)) + why = "empty-page"; + else if (gsp_phys_gr_codesrc_ptelike(p, off)) + why = "pte-page"; + else if (gsp_phys_gr_codesrc_modpt(pg->hpa)) + why = "module-pt-page"; + if (!why) + return GSP_PHYS_GR_CODESRC_OK; + /* + * Alternative resolution order; only a different page that itself + * passes the content checks is adopted. + */ + if (!gsp_sema_gr_reloc_resolve(page, &r) && r.hpa && + (r.hpa & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != + (pg->hpa & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) && + !gsp_phys_gr_codesrc_empty((const u8 *)__va(r.hpa), off) && + !gsp_phys_gr_codesrc_ptelike((const u8 *)__va(r.hpa), off)) { + *alt_hpa = r.hpa; + mode = GSP_PHYS_GR_CODESRC_ALT; + used = "alt"; + } else { + used = "guest-va"; + } + pr_info_ratelimited("vgsp-gr: CODESRC-SUSPECT va=0x%llx pa=0x%llx alt=%s used=%s why=%s\n", + page, pg->hpa, + mode == GSP_PHYS_GR_CODESRC_ALT ? r.stage : "none", + used, why); + return mode; +} + +/* + * STALLFREE: the absolute per-launch wait budget. Every ladder phase is + * individually bounded, but a launch can chain them (fast re-arm + fresh + * entries + retirement retries), so the sum - and with it the time the + * single-inflight work item holds the client's completion - is not. This + * predicate bounds the whole submit: once it is true, no further re-arm, + * re-publish or re-execution runs. Green launches complete in ~0.2-11 s and + * never approach the default 60 s. + */ +static bool gsp_phys_gr_wait_budget_over(unsigned long start) +{ + unsigned int budget = gsp_phys_gr_wait_max_ms_value(); + + return budget && time_after_eq(jiffies, + start + msecs_to_jiffies(budget)); +} + +/* + * STALLFREE: one distinguishable bounded-completion line per stall episode. + * `action=fail` means the submit gives up (the sema service's failure edge + * authors the QMD completion release in software, so the guest proceeds); + * `action=proceed` means the phase gave up but the launch continues with its + * current outcome. + */ +static void gsp_phys_gr_stall_log(const char *reason, u64 push_va, + unsigned long start, const char *action) +{ + pr_warn("vgsp-gr: GR-STALL-%s va=0x%llx waited_ms=%u action=%s\n", + reason, push_va, jiffies_to_msecs(jiffies - start), action); +} + int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, struct gsp_phys_backend_copy_result *out) { @@ -16504,50 +19048,135 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; void __iomem *bar0; struct gsp_phys_rmcall_ce_submit sub; - struct gsp_phys_fault_info fault = { }; /* * Static scratch: the only caller is the single-inflight sema work * item (ordered gsp-phys-work queue), so no concurrent submission can - * touch these. Keeping the 96-dword method block and the 64-word QMD - * copy off the stack keeps the frame under the warning bound. + * touch these. Keeping the 96-dword method block, the 64-word QMD + * copy and the ~370-byte fault record off the stack keeps the frame + * under the warning bound. */ static u32 methods[GSP_PHYS_RMCALL_GR_METHOD_WORDS]; static u32 qmd[GSP_PHYS_GR_QMD_WORDS]; + static struct gsp_phys_fault_info fault; unsigned long start = jiffies; unsigned long deadline; u8 *sem_va; u32 epoch, words, i, n_shadow = 0; + u32 win0_bank_base = 0, shadow_slot_limit = 0; + const u8 *guest_param_pg; u32 want_qmd = 0, want_ch = 0; bool qmd_rel = false, ch_rel = false; + bool qmd_rel_redirected = false; /* r420b: release0 -> module sem */ u64 qmd_rel_hpa = 0, ch_rel_hpa = 0; u64 ch_sem_va = GSP_PHYS_RMCALL_VA_SEM + 4; u64 qmd_dst = GSP_PHYS_RMCALL_VA_SRC; u8 *qmd_dst_mem = NULL; u32 qmd_rel_off = 0, ch_rel_off = 0; u32 poll = 0, w0 = 0, w1 = 0, grpoll_n = 0; + bool sem_gated = false; static u32 sked_prev[11]; static u32 sked_sample[11]; static u32 sked_tick; static bool sked_first; /* - * m142: per-bounce-page checksums captured at copy-in. The copy-out + * m142: per-data-page checksums captured at copy-in. The copy-out * compares them to report how many pages the engine actually modified; * an all-unchanged window after a reported completion is the * "front-end ran, grid never dispatched / read zeros" signature that - * previously masqueraded as a successful launch. Safe as static: the - * only caller is the single-inflight ordered gsp-phys-work item. + * previously masqueraded as a successful launch. B-2: one entry per + * page of the grown region (64 KB, static - must not sit on the kernel + * stack). Safe as static: the only caller is the single-inflight + * ordered gsp-phys-work item. + */ + static u64 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + /* + * r378: which guest page's content each bounce slot holds this + * launch (set by the copy-in fills, cleared per submission). The + * copy-out writes a slot back only to that exact page, so a rec + * aliasing another rec's slot - or a slot left over from a previous + * launch - can never ship foreign content into a guest page. + * Static like bounce_csum_in: the single-inflight ordered + * gsp-phys-work item is the only caller. */ - static u32 bounce_csum_in[GSP_PHYS_RMCALL_GR_MAIL_SLOT]; + static u64 bounce_staged_hpa[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; sked_tick = 0; sked_first = true; + guest_param_pg = NULL; u32 qmd_win_pde = 0, qmd_win_installed = 0; int qmd_win_ret = 0; bool qmd_win_diag = false; bool qmdwin_on = false; + u32 qw_bank = 0; char qmdwin_note[32] = ""; bool rearmed = false; + u32 attempts = 0; + /* + * F2/F4 state: the fast first-timeout re-arm runs once per launch; the + * deposit readback is one-shot per launch (the first timeout is the + * evidence window - the old r198 post-mortem only ran after attempt 3, + * which recovered launches never reached); poll_ms is the window in + * force for the current poll, so the timeout log reports it accurately. + * Static like the other submit scratch: the single-inflight ordered + * work item is the only caller, and statics keep the frame at its + * warning-bound size. + */ + static bool fast_rearmed; + static bool deposit_rb_done; + static u16 poll_ms; + static u32 poll_start_us; + /* + * GRFIX FIX3: per-launch relocation coverage. Static like the other + * submit scratch: the single-inflight ordered work item is the only + * caller, and statics keep the frame at its warning-bound size. + */ + static u32 reloc_gap; + static u64 reloc_gap_first; int ret = 0; + fast_rearmed = false; + deposit_rb_done = false; + gsp_phys_gr_codehash_done = false; /* CODEHASH: once per launch */ + /* + * M91 FAULT-CAP: snapshot the fault epoch per launch, so the capture + * at the launch's end attributes exactly the faults consumed inside + * this window (and a refusal path that never publishes compares equal + * and stays inert). + */ + gsp_phys_m91_fault_start = READ_ONCE(gsp_phys_fault_epoch); + reloc_gap = 0; + reloc_gap_first = 0; + /* + * relfix: the late-resolved page list is per-launch scratch exactly + * like bounce_csum_in; the single-inflight work item is the only + * caller. + */ + gsp_phys_gr_gap_n = 0; + gsp_phys_gr_gap_word_n = 0; + gsp_phys_gr_reloc_fixed = 0; + gsp_phys_gr_reloc_fixed_first = 0; + gsp_phys_gr_outwit_n = 0; /* COPYWIT: per-launch alias table */ + gsp_phys_gr_gap_logged_n = 0; + /* + * The fault record is static scratch (frame bound): reset it per + * submission so a previous launch's fault can never leak into this + * one's result - the old on-stack `= { }` semantics. + */ + memset(&fault, 0, sizeof(fault)); + /* + * r378: the bounce staging identity is per launch. A slot left over + * from a previous launch (or never staged this one) must not be + * treated as this launch's page even when the HPA matches. + */ + memset(bounce_staged_hpa, 0, sizeof(bounce_staged_hpa)); + /* + * F2: the shortened first window is part of the fast path; with + * gsp_phys_gr_retry_fast=0 the launch keeps today's exact behavior + * (the historical 5 s first window and ladder). + */ + poll_ms = gsp_phys_gr_retry_fast_value() ? + (u16)gsp_phys_gr_first_poll_ms_value() : + (u16)GSP_PHYS_RMCALL_GR_POLL_MS; + if (out) memset(out, 0, sizeof(*out)); if (!req || !req->qmd || @@ -16588,8 +19217,13 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * the program page (silent corruption; no launch can survive it). * The win0 PTEs are planted once at stage 3; reusing the same VAs * with fresh content is the proven pattern. + * + * m151: the cursor now selects one of two 8-page banks, alternating + * with the submission epoch (set below). The engine's index/data + * fetches of the previous launch's QMD shadow pages can still be in + * flight when this launch runs, so the bank alternation protects + * them from this launch's copy-in (late-fetch safety). */ - boot_stage.rmcall_gr_shadow_win0_used = 0; mutex_lock(&gsp_phys_rm_lock); if (!gsp_phys_backend_gr_ready()) { mutex_unlock(&gsp_phys_rm_lock); @@ -16600,8 +19234,84 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, epoch = ++boot_stage.rmcall_gr_submit_epoch; if (!epoch) epoch = ++boot_stage.rmcall_gr_submit_epoch; + boot_stage.rmcall_gr_shadow_win0_used = + (epoch & 1u) * GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; + win0_bank_base = boot_stage.rmcall_gr_shadow_win0_used; + shadow_slot_limit = win0_bank_base + + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; memcpy(qmd, req->qmd, sizeof(qmd)); /* + * M91 icache coherence (gsp_phys_gr_icache_inv, default 1): the + * window-0 shadow pool reuses code VAs across launches (two 8-page + * banks alternate per submit epoch), so the bytes at a program VA + * change between launches, while every captured guest QMD carries + * the QMDV02_03 cache-control block with INVALIDATE_INSTRUCTION_CACHE + * (word 5 bit 30) clear - the same field the CUDA driver arms only + * under its forceIcacheInvalidate knob. An SM instruction-fetch line + * fetched for launch N can then survive into launch N+2 and execute + * the previous kernel's bytes at the same VA, the observed + * "first attempt faults, identical retry succeeds" class. Set the + * bit only when the QMD carries the driver's standard cache-control + * signature (word 5 bits 26..29) so an unknown QMD revision is never + * touched; 0 restores the exact pre-change QMD for a one-flip A/B. + */ + if (gsp_phys_gr_icache_inv_value() && + (qmd[5] & 0x3c000000u) == 0x3c000000u && + !(qmd[5] & (1u << 30))) { + qmd[5] |= 1u << 30; + if (!gsp_phys_m91_icache_log) { + gsp_phys_m91_icache_log = true; + pr_info("vgsp-gr: M91 icache-invalidate bit set in the forwarded QMD (w5=0x%08x; QMDV02_03 word5 bit30, the guest driver's forceIcacheInvalidate field)\n", + qmd[5]); + } + } + /* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default + * 0 = off): the same cache-control word's bit 31 (the V01_06-layout + * SHADER_CONSTANT_CACHE position) is never set by the captured guest + * QMDs. The forwarded param page is rewritten in place between launches + * (the two-bank win0 pool), so a stale constant-cache line can serve + * the PREVIOUS launch's parameters and the current launch's output + * pages stay untouched - the post-dispatch no-store signature. Mode 1 + * sets bit 31; mode 2 forces the whole invalidate set (bits 24..31). + * Only when the QMD carries the driver's standard cache-control + * signature (bits 26..29) so an unknown QMD revision is never touched. + */ + if (gsp_phys_gr_ccache_inv_value() && + (qmd[5] & 0x3c000000u) == 0x3c000000u) { + u32 mode = gsp_phys_gr_ccache_inv_value(); + + if (mode == 2) + qmd[5] |= 0xff000000u; + else + qmd[5] |= 1u << 31; + if (!gsp_phys_r449_ccache_log) { + gsp_phys_r449_ccache_log = true; + pr_info("vgsp-gr: r449 constant-cache invalidate mode=%u applied (w5=0x%08x)\n", + mode, qmd[5]); + } + } + /* + * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar): force the + * forwarded QMD's RELEASE_MEMBAR_TYPE to FE_SYSMEMBAR (QMDV00_06 + * word 11 bit 14) so the front-end flushes its pending system-memory + * stores before writing the release semaphore. The captured guest + * QMDs carry FE_NONE, so a grid that completes with its data still + * in the GPU's L2 releases the client while no store is host-visible + * - the post-dispatch store-loss signature. The release ordering then + * guarantees the host's retire scan and copy-out can never observe a + * completion whose data has not landed. 0 (default) keeps the + * guest's captured value. + */ + if (gsp_phys_gr_rel_membar_value() && !(qmd[11] & (1u << 14))) { + qmd[11] |= 1u << 14; + if (!gsp_phys_gr_rel_membar_log) { + gsp_phys_gr_rel_membar_log = true; + pr_info("vgsp-gr: r433 RELEASE_MEMBAR_TYPE forced to FE_SYSMEMBAR in the forwarded QMD (w11=0x%08x)\n", + qmd[11]); + } + } + /* * Two-step PCAS (gsp_phys_rmcall_gr_pcas_two_step): set the QMD's * REQUIRE_SCHEDULING_PCAS (w4 bit12) so the copy PCAS does not * schedule and the trailing schedule-only PCAS does. @@ -16628,7 +19338,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, ret = gsp_phys_rmcall_gr_map_shadow_locked(n_shadow, req->rel_hpa, - &sh_va); + &sh_va, NULL); if (!ret) { n_shadow++; qmd_rel_hpa = req->rel_hpa; @@ -16660,6 +19370,75 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, */ qmd_rel_hpa = 0; qmd_rel_off = 0; + /* + * r420b: the ENGINE will not write the guest's QMD + * release word this launch (rel_mod, or no targetable + * guest release page). Tell the caller so it authors + * the word in software on every outcome - otherwise a + * launch without deferred release records is never + * completed on the guest word (the r414 saxpy hang). + */ + qmd_rel_redirected = true; + } + } + + /* + * GRID-WIT (r226): the engine reads CTA geometry from this verbatim + * QMD copy, and the copy-out witnesses show the corruption of the + * failing clients starts exactly at a CTA-band boundary (rows >= 16 + * replicate one tile). Dump the geometry-relevant words once per + * launch so the next capture proves what raster/thread dimensions the + * engine actually sees and lets them be compared against the client's + * own launch config: w12 = CTA_RASTER_WIDTH (gridDim.x), + * w13.lo = CTA_RASTER_HEIGHT (gridDim.y), w13.hi = DEPTH (z), + * w14/w15 = raster resume W/H, w18.hi = CTA_THREAD_DIMENSION0 + * (blockDim.x), w19.lo/hi = DIM1/DIM2 (blockDim.y/z), w18.lo = + * QMD_VERSION/MAJOR nibbles. w4/w5/w8/w17: PCAS/cache-control, + * prefetch pair low word, shared-memory size. + */ + if (gsp_phys_gr_copywit_enabled()) + pr_info("vgsp-gr: GRID-WIT w4=%08x w5=%08x w8=%08x w12=%08x w13=%08x w14=%08x w15=%08x w17=%08x w18=%08x w19=%08x\n", + qmd[4], qmd[5], qmd[8], qmd[12], qmd[13], qmd[14], + qmd[15], qmd[17], qmd[18], qmd[19]); + + /* + * r234 RDW arming witness: one line per launch with the rec count, + * so a capture missing REC-DUMP lines can never be confused with + * "the witness did not run" - probe-class launches must show + * nrecs>0 here. + */ + if (req->n_data && gsp_phys_gr_rdw_budget < 256u) { + gsp_phys_gr_rdw_budget++; + pr_info("vgsp-gr: RDW nrecs=%u launch_va=0x%llx\n", + req->n_data, req->push_va); + } + + /* + * REC-DUMP (r229): the probe-class launches (small data sets) get + * their full rec table logged - every (va, hpa, reloc_va) the engine + * will read/write - so the next capture can compare the resolved + * HPAs against the buffer the client actually uploaded (distinct + * VAs collapsing onto one HPA, or HPAs in a foreign range, are + * visible directly). Big launches log the first/last 8 recs only. + * r233: plain pr_info with a boot-wide line budget - the earlier + * pr_info_ratelimited version was suppressed wholesale during the + * probe's post-stall logging burst, so the probe-class recs never + * reached the ring. + */ + if (gsp_phys_gr_copywit_enabled() && req->n_data && + gsp_phys_gr_recdump_budget < 512u) { + u32 d; + + for (d = 0; d < req->n_data; d++) { + if (req->n_data > 64u && d >= 8u && + d < req->n_data - 8u) + continue; + if (gsp_phys_gr_recdump_budget >= 512u) + break; + gsp_phys_gr_recdump_budget++; + pr_info("vgsp-gr: REC-DUMP[%u/%u] va=0x%llx hpa=0x%llx reloc=0x%llx\n", + d, req->n_data, req->data[d].va, + req->data[d].hpa, req->data[d].reloc_va); } } @@ -16700,9 +19479,16 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, if (pg->hpa) { ret = gsp_phys_rmcall_gr_map_va_locked(bar0, pg_va, pg->hpa, epoch); - if (!ret) + if (!ret) { mapped = true; - else if (ret == -ENOSPC) + /* + * CODEHASH: on the Tier-1 identity path the + * engine reads the guest page itself; sample + * the entry before the QMD pair is authored. + */ + if (pg->qmd_word == 48u) + gsp_phys_gr_codehash(req, pg, NULL); + } else if (ret == -ENOSPC) gsp_phys_gr_pool_exhausted++; } /* @@ -16719,6 +19505,8 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, for (k = 0; k < n_sh_rec; k++) { u64 va = 0; + if (sh_rec[k].line == 0xffffffffu) + continue; /* cached line: never extended */ if (pg_va != sh_rec[k].guest_va + (u64)sh_rec[k].npte * GSP_PHYS_PAGE_SIZE) @@ -16745,11 +19533,11 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, */ if (!mapped && pg->hpa && pg->qmd_word == GSP_PHYS_GR_QMD_WORD_NONE) { - u32 sh_line = boot_stage.rmcall_guest_pt_next_line; + u32 sh_line = 0xffffffffu; if (n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES && !gsp_phys_rmcall_gr_map_shadow_locked( - n_shadow, pg->hpa, &sh_va)) { + n_shadow, pg->hpa, &sh_va, &sh_line)) { u32 k, slot = n_shadow; mapped = true; @@ -16801,6 +19589,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, &boot_stage.rmcall_gr_shadow; u8 *sh_page; u32 j, copied = 0; + bool code_keep_guest = false; if (n_shadow >= GSP_PHYS_RMCALL_GR_SHADOW_PAGES) { gsp_phys_gr_pool_exhausted++; @@ -16826,8 +19615,32 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * domain, copy the full guest page into the shadow and * rewrite the QMD pointer pair, so no inline coverage * is needed for pages the engine only reads. + * + * CODESRC: the program page (qmd_word 48) is the page + * the engine's I-fetch reads; screen its source + * before publishing it. A refused source is either + * replaced by the alternative resolver's page or the + * pair is left at the guest VA (no copy, no rewrite, + * no mapping - the page is fully withdrawn from the + * submission). */ - if (pg->hpa) { + if (pg->hpa && pg->qmd_word == 48u) { + u64 alt_hpa = 0; + int cs = gsp_phys_gr_code_src(req, pg, &alt_hpa); + + if (cs == GSP_PHYS_GR_CODESRC_GUEST_VA) { + code_keep_guest = true; + mapped = true; + } else { + const u8 *csrc = __va(cs == + GSP_PHYS_GR_CODESRC_ALT ? + alt_hpa : pg->hpa); + + memcpy(sh_page, csrc, + GSP_PHYS_PAGE_SIZE); + copied++; + } + } else if (pg->hpa) { memcpy(sh_page, (u8 *)__va(pg->hpa), GSP_PHYS_PAGE_SIZE); copied++; @@ -16845,10 +19658,11 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, } if (pg->hpa && pg->qmd_word >= 32 && pg->qmd_word <= 47 && !(pg->qmd_word & 1)) - gsp_phys_gr_rewrite_data_vars((u32 *)sh_page, + (void)gsp_phys_gr_rewrite_data_vars( + (u32 *)sh_page, GSP_PHYS_PAGE_SIZE / 4, req, (pg->qmd_word - 32) / 2, - (u32 *)__va(pg->hpa)); + (u32 *)__va(pg->hpa), NULL, false); if (pg->hpa && pg->qmd_word == 40u && gsp_phys_rmcall_gr_mail_patch_value()) { u32 *w = (u32 *)sh_page; @@ -16884,7 +19698,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * data-access instruction and the guest VA arguments it * fetches can be read from the log. */ - if (pg->hpa && pg->qmd_word == 48u) { + if (pg->hpa && pg->qmd_word == 48u && !code_keep_guest) { u32 po = (qmd[48] & (GSP_PHYS_PAGE_SIZE - 1)) / 4; const u32 *w = (const u32 *)sh_page + po; @@ -16899,6 +19713,12 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, w[27], w[28], w[29], w[30], w[31], w[32], w[33], w[34], w[35], w[36], w[37], w[38], w[39]); + /* + * CODEHASH: the engine-visible code copy (sh_page) + * is authored here - sample source + shadow + * before the QMD pair points at this page. + */ + gsp_phys_gr_codehash(req, pg, sh_page); } if (pg->hpa && pg->qmd_word == 32u) { const u32 *w = (const u32 *)sh_page; @@ -16959,11 +19779,31 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * relocated values no longer match a guest data VA. */ if (pg->hpa && pg->qmd_word >= 32 && pg->qmd_word <= 47 && - !(pg->qmd_word & 1)) - gsp_phys_gr_rewrite_data_vars((u32 *)sh_page, + !(pg->qmd_word & 1)) { + u64 gf = 0; + u32 gg; + + /* + * GRFIX FIX3: this is the final authoring + * of the engine-visible page (after the + * inline replay), so its gap count is the + * launch's relocation-coverage measure. + * Log it (rate-limited) and accumulate it + * for the strict hold below. + */ + gg = gsp_phys_gr_rewrite_data_vars( + (u32 *)sh_page, GSP_PHYS_PAGE_SIZE / 4, req, (pg->qmd_word - 32) / 2, - (u32 *)__va(pg->hpa)); + (u32 *)__va(pg->hpa), &gf, true); + if (gg) { + reloc_gap += gg; + if (!reloc_gap_first) + reloc_gap_first = gf; + pr_info_ratelimited("vgsp-gr: RELOC-GAP words=%u first=0x%llx\n", + gg, gf); + } + } if (copied) { u32 sh_line = boot_stage.rmcall_guest_pt_next_line; @@ -16975,7 +19815,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, gsp_phys_rmcall_gr_shadow_win0_value() && boot_stage.rmcall_gr_shadow_win0.pages && boot_stage.rmcall_gr_shadow_win0_used < - GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS; + shadow_slot_limit; /* * m86: SASS override. When gsp-phys-gr-override @@ -17036,7 +19876,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, n_shadow, (u64)gsp_phys_chunk_page_dma(sh, n_shadow), - &sh_va); + &sh_va, NULL); } if (!ret) { bool keep_guest = @@ -17147,6 +19987,40 @@ gr_page_unmapped: } /* + * GRFIX FIX3 strict hold: with gsp_phys_gr_strict_reloc=1, a launch + * whose rewritten pages carry device-VA-shaped words that match no + * tracked data page is refused before publish - the word stays what + * it was (an unrelocated guest VA), but the engine never sees the + * QMD, so it can never walk an untranslatable pointer. The refusal + * rides the existing backend-failure path (caller bridge fallback + + * the sema service's recorded-release hold; bounded, never a hang). + * The hold is opt-in because the shape heuristic can false-positive + * on non-pointer data words; the RELOC-GAP line (emitted with the + * parameter off as well) is the false-positive evidence. + */ + if (reloc_gap && gsp_phys_gr_strict_reloc_param) { + mutex_unlock(&gsp_phys_rm_lock); + gsp_phys_gr_refused++; + pr_err("vgsp-gr: RELOC-GAP-STRICT words=%u first=0x%llx; launch held fail-closed (unrelocated device-VA words never reach the engine)\n", + reloc_gap, reloc_gap_first); + ret = -EAGAIN; + goto out; + } + /* + * COPYWIT fail4: one per-launch census of the rewrite outcome so the + * resolved/unresolved split of every bank pass is provable from a + * single line (gaps left as guest VAs vs late-resolved words authored + * via RELOC-FIXED). Rate-limited like the per-word lines. + */ + if (gsp_phys_gr_copywit_enabled() && + (reloc_gap || gsp_phys_gr_reloc_fixed)) + pr_info_ratelimited("vgsp-gr: RELOC-CENSUS gaps=%u first_gap=0x%llx fixed=%llu first_fixed=0x%llx late_pages=%u\n", + reloc_gap, reloc_gap_first, + gsp_phys_gr_reloc_fixed, + gsp_phys_gr_reloc_fixed_first, + gsp_phys_gr_gap_n); + + /* * Rank-2 qmdwin (gsp_phys_rmcall_gr_qmdwin=1): relocate the inline-QMD * destination to the dedicated module page planted in the ladder's * window-0 small-PT line (VA 0x107000). The PTE was written by @@ -17163,11 +20037,16 @@ gr_page_unmapped: NULL; if (boot_stage.rmcall_gr_qmdwin_va && qw) { + qw_bank = (epoch & 1u) % + GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS; + qmdwin_on = true; - qmd_dst = boot_stage.rmcall_gr_qmdwin_va; - qmd_dst_mem = qw; - pr_info("vgsp-gr: qmdwin: relocating inline-QMD dest 0x%llx -> 0x%llx (module ring chunk page)\n", - req->qmd_va, qmd_dst); + qmd_dst = boot_stage.rmcall_gr_qmdwin_va + + (u64)qw_bank * GSP_PHYS_PAGE_SIZE; + qmd_dst_mem = gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_qmdwin, qw_bank); + pr_info("vgsp-gr: qmdwin: relocating inline-QMD dest 0x%llx -> 0x%llx (module ring chunk page, bank %u)\n", + req->qmd_va, qmd_dst, qw_bank); } else { pr_warn("vgsp-gr: qmdwin=1 but no planted module page (stage-3 plant skipped or failed); using the guest QMD destination\n"); } @@ -17197,7 +20076,7 @@ gr_page_unmapped: if (n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES && !gsp_phys_rmcall_gr_map_shadow_locked( - n_shadow, req->qmd_hpa, &sh_va)) { + n_shadow, req->qmd_hpa, &sh_va, NULL)) { qmd_dst_mem = (u8 *)gsp_phys_chunk_page_va( &boot_stage.rmcall_gr_shadow, n_shadow) + qmd_off; @@ -17240,7 +20119,8 @@ gr_page_unmapped: u64 sh_va = 0; ret = gsp_phys_rmcall_gr_map_shadow_locked(n_shadow, - req->sem_hpa, &sh_va); + req->sem_hpa, &sh_va, + NULL); if (!ret) { n_shadow++; ch_rel_hpa = req->sem_hpa; @@ -17258,6 +20138,27 @@ gr_page_unmapped: /* A stale release from a previous submission must not be observed. */ WRITE_ONCE(*(u32 *)sem_va, 0); WRITE_ONCE(*(u32 *)(sem_va + 4), 0); + /* + * r441: zero the GUEST-VISIBLE release words for the first attempt + * too, not only for outcome retries. The r439 evidence shows a first + * attempt whose poll broke on a STALE value (qmd_rel=0x00000005 from + * the previous launch's engine write, want=6) while the engine never + * fetched this launch's deposit (sentinel intact, "engine fetch + * NONE"): the host then shipped a completion for a launch that did + * not run. Zeroing forces the poll to wait for THIS launch's own + * engine write; when the engine does not write it, the timeout ladder + * and the software-authored completion take over (the honest path), + * and the wedge classification stops being satisfiable by a stale + * value. The write guard re-validates the target like the retry path. + */ + if (qmd_rel_hpa && qmd_rel_off < GSP_PHYS_PAGE_SIZE && + gsp_sema_gr_write_guard(req->rel_va, qmd_rel_hpa, + "gr-rel-zero-first")) + WRITE_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + qmd_rel_off), 0); + if (ch_rel_hpa && ch_rel_off < GSP_PHYS_PAGE_SIZE && + gsp_sema_gr_write_guard(req->rel_va, ch_rel_hpa, + "gr-chrel-zero-first")) + WRITE_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + ch_rel_off), 0); wmb(); /* @@ -17284,6 +20185,20 @@ gr_page_unmapped: pr_info("vgsp-gr: qmd w0..3=%08x %08x %08x %08x w4..7=%08x %08x %08x %08x w23..25=%08x %08x %08x w32/33=%08x %08x w48/49=%08x %08x\n", qmd[0], qmd[1], qmd[2], qmd[3], qmd[4], qmd[5], qmd[6], qmd[7], qmd[23], qmd[24], qmd[25], qmd[32], qmd[33], qmd[48], qmd[49]); + /* + * r430: full 64-word deposit dump for the wedge's healthy-vs-wedged + * QMD diff (hardware-only fields w50/w51 INNER_GET/PUT, w52 span-list, + * w53 next-QMD pointer are not in the partial line above). Gated by + * the skedrec param (default on); one long line per launch. + */ + if (gsp_phys_rmcall_gr_skedrec_value()) { + char qb[64 * 9 + 1]; + int qo = 0; + + for (i = 0; i < GSP_PHYS_GR_QMD_WORDS && qo < (int)sizeof(qb) - 10; i++) + qo += scnprintf(qb + qo, sizeof(qb) - qo, "%08x ", qmd[i]); + pr_info("vgsp-gr: QMD64 %s\n", qb); + } /* m120: dump the main PT page slots 0x100..0x11f so the PTE content * the walker would read (win0 + bounce window) is visible in the log. */ { @@ -17332,38 +20247,125 @@ gr_page_unmapped: * removes the race entirely; per-page checksums are kept so the * copy-out can prove whether the engine modified the window at all. */ - if (req->n_data && boot_stage.rmcall_gr_bounce.pages) { + if (req->n_data && gsp_phys_gr_data_chunk()->pages) { + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); u32 dcopied = 0, dfailed = 0, di; + /* + * GRFIX FIX2 (copy-in fail-closed): classify every data page + * BEFORE the bounce copy. An entry with no resolved HPA, no + * bounce index or no backing bounce page cannot be freshly + * filled for this launch; publishing it would hand the engine + * a QMD data pointer into a bounce page still holding a + * previous launch's bytes (the multi-page stale-output + * class). Hold on the existing backend-refusal path: the + * caller's bridge fallback and the sema service's recorded- + * release hold apply the software releases - bounded, never + * a hang - and the engine never sees the QMD. + */ for (di = 0; di < req->n_data; di++) { - u32 bi = (u32)((req->data[di].reloc_va - - GSP_PHYS_RMCALL_GR_BOUNCE_VA0) >> - 12); - const u8 *src; - u8 *dst; - u32 ci = 0, csum = 0; + u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); - if (bi >= GSP_PHYS_RMCALL_GR_MAIL_SLOT || - !req->data[di].hpa) { + if (bi == GSP_PHYS_GR_DATA_BI_NONE || + !req->data[di].hpa || + !gsp_phys_chunk_page_va(dchunk, bi)) dfailed++; - continue; - } - src = (const u8 *)__va(req->data[di].hpa); - dst = gsp_phys_chunk_page_va( - &boot_stage.rmcall_gr_bounce, bi); - memcpy(dst, src, GSP_PHYS_PAGE_SIZE); - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(dst + ci); - bounce_csum_in[bi] = csum; - dcopied++; } - dma_wmb(); + if (dfailed) { + mutex_unlock(&gsp_phys_rm_lock); + gsp_phys_gr_refused++; + pr_err("vgsp-gr: copy-in hold: %u/%u data pages unresolved at copy-in (no fresh bounce content); launch held fail-closed, no publish\n", + dfailed, req->n_data); + ret = -EAGAIN; + goto out; + } + dcopied = gsp_phys_gr_fill_bounce(req, bounce_csum_in, + bounce_staged_hpa); pr_info("vgsp-gr: bounce copy-in: %u pages, %u skipped (pre-publish)\n", dcopied, dfailed); - for (di = 0; di < req->n_data && di < 3; di++) - pr_info("vgsp-gr: data[%u] va=0x%llx hpa=0x%llx reloc=0x%llx\n", + /* + * r234 A-TRACE pt=P3 (engine-read point): small launches get + * the source-page content of their first recs - the exact + * bytes the copy-in stages and the engine will read. The probe + * A buffer is 16 pages of ints 0..16383 (w0=0 w1=1 w2=2 ...); + * any point in the pipeline where that pattern breaks names + * the bug site. + */ + if (req->n_data <= 1024u && gsp_phys_gr_atrace_budget < 128u) { + u32 d; + + for (d = 0; d < req->n_data && d < 16u; d++) { + const u32 *pp = + (const u32 *)__va(req->data[d].hpa); + + if (!pp || gsp_phys_gr_atrace_budget >= 128u) + break; + gsp_phys_gr_atrace_budget++; + pr_info("vgsp: A-TRACE pt=P3 va=0x%llx hpa=0x%llx w0..15=%08x %08x %08x %08x %08x %08x %08x %08x | %08x %08x %08x %08x %08x %08x %08x %08x\n", + req->data[d].va, req->data[d].hpa, + pp[0], pp[1], pp[2], pp[3], pp[4], + pp[5], pp[6], pp[7], pp[8], pp[9], + pp[10], pp[11], pp[12], pp[13], + pp[14], pp[15]); + } + } + if ((u64)req->n_data << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-bounce-copy-in va=0x%llx len=0x%llx done=0x%llx\n", + req->data[0].va, (u64)req->n_data << 12, + (u64)dcopied << 12); + if (dcopied < req->n_data) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-bounce-copy-in\n", + req->data[0].va, + (u64)req->n_data << 12, + (u64)dcopied << 12); + for (di = 0; di < req->n_data && di < 3; di++) { + u32 w0 = 0, w1 = 0; + + /* + * fail4 definitive witness: the bytes the engine will + * read for the first recs (the rec hpa's content) + * beside the rec identity - names a +2 MB rec-page + * phase directly (client a[0..3] = 00000000 + * 0000003f 0000003f... would appear as exp[2M] + * values if the hpa points 512 pages late). + */ + if (req->data[di].hpa) { + const u32 *p = (const u32 *)__va( + req->data[di].hpa); + + w0 = p[0]; + w1 = p[1]; + } + pr_info("vgsp-gr: data[%u] va=0x%llx hpa=0x%llx reloc=0x%llx first8=%08x %08x\n", di, req->data[di].va, req->data[di].hpa, - req->data[di].reloc_va); + req->data[di].reloc_va, w0, w1); + } + } + + /* + * relfix: the late-resolved pages copy in exactly like the tracked + * data pages (same m142 pre-doorbell ordering). The word rewrites + * already point the engine at these bounce pages; filling them here + * is what makes those pointers real. Still under gsp_phys_rm_lock, + * like the block above. + */ + if (gsp_phys_gr_gap_n && gsp_phys_gr_data_chunk()->pages) { + u32 gcopied = gsp_phys_gr_fill_gap_bounce(bounce_csum_in, + bounce_staged_hpa); + + pr_info("vgsp-gr: gap copy-in: %u late-resolved pages, %llu word(s) fixed first=0x%llx\n", + gsp_phys_gr_gap_n, gsp_phys_gr_reloc_fixed, + gsp_phys_gr_reloc_fixed_first); + if ((u64)gsp_phys_gr_gap_n << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-gap-copy-in va=0x%llx len=0x%llx done=0x%llx\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gcopied << 12); + if (gcopied < gsp_phys_gr_gap_n) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-gap-copy-in\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gcopied << 12); } ret = gsp_phys_rmcall_publish_locked(bar0, boot_stage.rmcall_va_pb, @@ -17445,24 +20447,37 @@ gr_page_unmapped: gsp_phys_rmcall_gr_gpc_mmu_init(bar0); /* * m143 DIAGNOSTIC (second-client launch-state divergence): dump the - * exact param page the FE/SM will read (win0 slot 0, the QMD bank-0 - * target) and the guest param page at the three JIT parameter slots, - * right before the doorbell. The JIT reads a@0x160/b@0x168/c@0x170 - * and n@0x178 (verified with ptxas -arch=sm_75). If the win0 copy is - * correct here but the engine still stores to the wrong relocated - * window, the divergence is engine-side (a stale mapping), not the - * relaunched page content. + * exact param page the FE/SM will read (this launch's win0 bank base + * slot, the QMD bank-0 target) and the guest param page at the three + * JIT parameter slots, right before the doorbell. The JIT reads + * a@0x160/b@0x168/c@0x170 and n@0x178 (verified with ptxas + * -arch=sm_75). If the win0 copy is correct here but the engine still + * stores to the wrong relocated window, the divergence is engine-side + * (a stale mapping), not the relaunched page content. */ if (boot_stage.rmcall_gr_shadow_win0.pages) { const u8 *wp = gsp_phys_chunk_page_va( - &boot_stage.rmcall_gr_shadow_win0, 0); + &boot_stage.rmcall_gr_shadow_win0, win0_bank_base); + + if (wp) { + u32 off; - if (wp) - pr_info("vgsp-gr: DIAG win0[0] 160=%llx 168=%llx 170=%llx 178=%llx\n", + pr_info("vgsp-gr: DIAG win0[%u] 160=%llx 168=%llx 170=%llx 178=%llx\n", + win0_bank_base, (u64)get_unaligned_le64(wp + 0x160), (u64)get_unaligned_le64(wp + 0x168), (u64)get_unaligned_le64(wp + 0x170), (u64)get_unaligned_le64(wp + 0x178)); + pr_info("vgsp-gr: win0 param hash pre=0x%016llx\n", + gsp_phys_gr_fnv1a64(wp, GSP_PHYS_PAGE_SIZE)); + for (off = 0x100u; off < 0x200u; off += 0x20u) + pr_info("vgsp-gr: win0 page +0x%03x: %016llx %016llx %016llx %016llx\n", + off, + (u64)get_unaligned_le64(wp + off), + (u64)get_unaligned_le64(wp + off + 8), + (u64)get_unaligned_le64(wp + off + 16), + (u64)get_unaligned_le64(wp + off + 24)); + } } { u32 pi; @@ -17474,12 +20489,15 @@ gr_page_unmapped: !req->pages[pi].hpa) continue; gp = (const u8 *)__va(req->pages[pi].hpa); + guest_param_pg = gp; pr_info("vgsp-gr: DIAG guest-param pa=0x%llx 160=%llx 168=%llx 170=%llx 178=%llx\n", req->pages[pi].hpa, (u64)get_unaligned_le64(gp + 0x160), (u64)get_unaligned_le64(gp + 0x168), (u64)get_unaligned_le64(gp + 0x170), (u64)get_unaligned_le64(gp + 0x178)); + pr_info("vgsp-gr: guest-param hash pre=0x%016llx\n", + gsp_phys_gr_fnv1a64(gp, GSP_PHYS_PAGE_SIZE)); break; } } @@ -17489,13 +20507,15 @@ gr_page_unmapped: gsp_phys_rmcall_gr_prelude_value(), gsp_phys_rmcall_gr_qmdvirt_value(), boot_stage.rmcall_token, - qmd_rel, want_ch, GSP_PHYS_RMCALL_GR_POLL_MS, qmdwin_note); + qmd_rel, want_ch, poll_ms, qmdwin_note); if (qmd_win_diag) - pr_info("vgsp-gr: qmd dest va=0x%llx pdeIndex=%u installed=%u win_mask=0x%x ensure_ret=%d\n", + pr_info("vgsp-gr: qmd dest va=0x%llx pdeIndex=%u installed=%u win_mask=0x%llx ensure_ret=%d\n", qmd_dst, qmd_win_pde, qmd_win_installed, - boot_stage.rmcall_guest_pt_win_mask, qmd_win_ret); + (unsigned long long)boot_stage.rmcall_guest_pt_win_mask, + qmd_win_ret); - deadline = jiffies + msecs_to_jiffies(GSP_PHYS_RMCALL_GR_POLL_MS); + poll_start_us = (u32)ktime_to_us(ktime_get()); + deadline = jiffies + msecs_to_jiffies(poll_ms); for (;;) { /* * m72: SKED transition recorder - sample the CWD/HWW registers @@ -17503,8 +20523,14 @@ gr_page_unmapped: * cancel or dispatch sequence of the launch window is on the * record even when the poll times out. */ - if (gsp_phys_rmcall_gr_skedrec_value() && bar0 && - (++sked_tick & 7u) == 0u) { + if (gsp_phys_rmcall_gr_skedrec_value() && bar0) { + /* + * r430: sample EVERY poll iteration (not every 8th): + * the wedge's release word lands ~7 ms after the + * doorbell, so the old 8-iteration gate saw at most one + * sample of the dispatch window. Changes only. + */ + sked_tick++; sked_sample[0] = ioread32(bar0 + 0x00407060u); sked_sample[1] = ioread32(bar0 + 0x00407064u); sked_sample[2] = ioread32(bar0 + 0x00407068u); @@ -17520,7 +20546,7 @@ gr_page_unmapped: memcmp(sked_sample, sked_prev, sizeof(sked_sample))) { pr_info("vgsp-gr: sked rec +%ums cwd0=%08x cwd1=%08x cwd2=%08x info1=%08x info2=%08x act=%08x esr=%08x grfifo=%08x fecs_cur=%08x fecs_new=%08x ctxsw1=%08x\n", - (sked_tick / 8u) * + sked_tick * GSP_PHYS_RMCALL_POLL_MS, sked_sample[0], sked_sample[1], sked_sample[2], sked_sample[3], @@ -17545,17 +20571,99 @@ gr_page_unmapped: ch_rel_off)); else w1 = READ_ONCE(*(u32 *)(sem_va + 4)); - if ((!qmd_rel || w0 == want_qmd) && w1 == want_ch) + if ((!qmd_rel || w0 == want_qmd) && w1 == want_ch) { + /* F4-a: first poll break + USERD witnesses. */ + gsp_phys_gr_poll_break_log(w0, want_qmd, w1, + poll_start_us, &sub); break; + } + /* + * STALLFREE: the whole-launch budget. Checked after the release + * acks (a completed launch always wins) and before any further + * ladder work; on expiry the submit fails out now and the sema + * service's failure edge authors the QMD completion release in + * software, so the guest client is never left waiting on a + * launch this host will not finish. + */ + if (gsp_phys_gr_wait_budget_over(start)) { + gsp_phys_gr_stall_wait_max++; + gsp_phys_gr_stall_log("wait-max", req->push_va, start, + "fail"); + pr_err("vgsp-gr: GR LAUNCH FAILED: wait budget crossed (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x want=0x%08x) h=0x%x push=0x%llx; ladder stopped\n", + w0, want_qmd, w1, want_ch, req->h_object, + req->push_va); + ret = -ETIMEDOUT; + goto out; + } if ((++grpoll_n % 100) == 0) pr_info("vgsp-gr: poll +%ums qmd_rel=0x%08x ch_rel=0x%08x\n", grpoll_n * GSP_PHYS_RMCALL_POLL_MS, w0, w1); if (time_after_eq(jiffies, deadline)) { gsp_phys_gr_timed_out++; pr_err("vgsp-gr: GR LAUNCH FAILED: timeout after %u ms (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x want=0x%08x) h=0x%x push=0x%llx\n", - GSP_PHYS_RMCALL_GR_POLL_MS, w0, want_qmd, w1, + poll_ms, w0, want_qmd, w1, want_ch, req->h_object, req->push_va); /* + * F4-b: one-shot deposit readback at the FIRST timeout + * (the old post-mortem only ran after attempt 3, which + * a recovered launch never reached). Reading the + * qmdwin/shadow destination here proves whether THIS + * attempt's inline-QMD copy landed and what the engine + * would fetch (w63 is the fetch sentinel). Bound: one + * line per launch. + */ + if (!deposit_rb_done) { + deposit_rb_done = true; + gsp_phys_gr_deposit_rb_log(req, qmd_dst_mem); + } + /* + * F2: fast re-arm at the first timeout. The m155 + * extended re-arm (lane-enable replay + GPFIFO/TSG + * schedule + doorbell) plus a FRESH ring entry recovers + * the r198 class in ~11 s only because it runs after two + * full 5 s windows. Do it now and re-poll briefly; if it + * still fails, fall through to the historical ladder + * (the fast entry counts against the same attempt + * budget, so the total number of host publishes and the + * ring wrap exposure stay as before). + */ + if (gsp_phys_gr_retry_fast_value() && !fast_rearmed) { + int fr; + + fast_rearmed = true; + attempts++; + mutex_lock(&gsp_phys_rm_lock); + fr = gsp_phys_rmcall_publish_locked( + bar0, boot_stage.rmcall_va_pb, words, + &sub); + if (!fr) + fr = gsp_phys_rmcall_gr_resched_locked( + bar0); + mutex_unlock(&gsp_phys_rm_lock); + if (!fr) { + gsp_phys_gr_retries++; + poll_ms = + GSP_PHYS_RMCALL_GR_FAST_RETRY_MS; + pr_info("vgsp-gr: fast re-arm after first timeout: extended re-arm + fresh ring entry slot=%u published; re-polling %u ms (attempts %u/%u)\n", + sub.slot, poll_ms, attempts, + GSP_PHYS_RMCALL_GR_ATTEMPTS); + deadline = jiffies + + msecs_to_jiffies(poll_ms); + /* + * The immediate m142 re-arm would + * only duplicate the resched that just + * ran; the ladder below still owns the + * warm-sem gate and the remaining + * fresh entries. + */ + rearmed = true; + grpoll_n = 0; + continue; + } + pr_info("vgsp-gr: fast re-arm failed rc=%d; falling through to the timeout ladder\n", + fr); + } + /* * m142: one bounded re-arm before declaring the launch * dead. Re-issue the stage-4 schedule controls and re-ring * the doorbell, then poll again; if the channel was merely @@ -17570,17 +20678,81 @@ gr_page_unmapped: rr = gsp_phys_rmcall_gr_resched_locked(bar0); mutex_unlock(&gsp_phys_rm_lock); if (!rr) { + poll_ms = GSP_PHYS_RMCALL_GR_POLL_MS; pr_info("vgsp-gr: rearm after timeout: schedule re-issued + doorbell re-rung; re-polling %u ms\n", - GSP_PHYS_RMCALL_GR_POLL_MS); + poll_ms); deadline = jiffies + - msecs_to_jiffies( - GSP_PHYS_RMCALL_GR_POLL_MS); + msecs_to_jiffies(poll_ms); grpoll_n = 0; continue; } pr_info("vgsp-gr: rearm after timeout failed rc=%d\n", rr); } + /* + * m166: liveness gate before spending the fresh-entry + * retries. The m156 warm sem push (the stage-4 GR sem + * block) is the only proven "engine alive" probe; if + * it times out, the engine is wedged for the boot and + * every further retry is futile - fail fast instead of + * burning ~20 s with the single-inflight slot held + * (r167 memset2d: 8 internal ladder timeouts over 31 s + * for an entry the front-end never fetched). The probe + * itself is bounded by the sem push's own poll. + */ + if (!sem_gated) { + int sp_ret; + + sem_gated = true; + mutex_lock(&gsp_phys_rm_lock); + sp_ret = gsp_phys_rmcall_gr_sem_push_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + if (sp_ret) { + gsp_phys_gr_sem_wedged++; + pr_info("vgsp-gr: timeout ladder: warm sem WEDGED (rc=%d) - failing fast instead of fresh-entry retries\n", + sp_ret); + break; + } + pr_info("vgsp-gr: timeout ladder: warm sem ALIVE - continuing fresh-entry retries\n"); + } + /* + * The re-arm above only replays the schedule verbs; if the + * entry was already fetched or the front-end never latched + * it, that cannot make the QMD run. Re-publish a FRESH ring + * entry (same PB content and QMD) with a re-issued schedule + * + doorbell, bounded. The timeout path never copied the + * result out to the guest, so a timed-out attempt did not + * reach the client; the guest therefore still sees only the + * successful attempt. + */ + if (attempts < GSP_PHYS_RMCALL_GR_ATTEMPTS) { + int pr_ret; + + attempts++; + mutex_lock(&gsp_phys_rm_lock); + pr_ret = gsp_phys_rmcall_publish_locked( + bar0, boot_stage.rmcall_va_pb, words, + &sub); + if (!pr_ret) + pr_ret = gsp_phys_rmcall_gr_resched_locked( + bar0); + mutex_unlock(&gsp_phys_rm_lock); + if (!pr_ret) { + gsp_phys_gr_retries++; + poll_ms = GSP_PHYS_RMCALL_GR_RETRY_MS; + pr_info("vgsp-gr: retry %u/%u: fresh ring entry slot=%u published + rescheduled; re-polling %u ms\n", + attempts, + GSP_PHYS_RMCALL_GR_ATTEMPTS, + sub.slot, poll_ms); + deadline = jiffies + + msecs_to_jiffies(poll_ms); + rearmed = false; + grpoll_n = 0; + continue; + } + pr_info("vgsp-gr: retry %u publish failed rc=%d\n", + attempts, pr_ret); + } if (gsp_phys_gr_fault_active()) { struct gsp_phys_fault_info fi; u32 fep = 0; @@ -17871,39 +21043,764 @@ gr_page_unmapped: gsp_phys_gr_ok++; /* + * m151: retire the output footprint before the copy-out (and before + * any later launch's copy-in). qmd_rel above is a schedule/fetch ack, + * not a data-path retirement, so the engine's stores into the bounce + * pages can land after it and be lost by an early copy-out; the next + * launch's copy-in would also race that write tail. Success path + * only - the timeout/fault paths never copied out. See + * gsp_phys_gr_output_retire_wait(). + * + * m153: outcome-verifying retry. The intermittent per-boot + * no-execution class (qmd_rel/ch_rel satisfied but the engine wrote + * nothing at all) cannot be caught by the completion poll; the + * retirement wait is the only execution witness. On zero writes, + * refill the bounce window from the guest inputs, re-publish the + * same method block on a fresh ring entry, re-issue the schedule and + * re-poll, then re-wait. + * + * m179: the budget is now DEDICATED (gsp_phys_gr_nowrite_retries, + * default 3) instead of sharing the timeout ladder's attempt counter - + * a launch that had already spent attempts on the ladder used to reach + * the accept path with no re-execution at all (the r200-r202 + * memcpy_roundtrip / multikernel all-wrong class). The trigger is + * specific: only a launch with n_tracked > 0 tracked data pages can be + * a no-execution, so a launch whose data entries resolve nothing + * legitimately reports zero writes without entering this path. After + * the budget the launch proceeds to copy-out with whatever the engine + * produced, and the accept is logged loudly (pr_warn) and classified + * by the QMD fetch sentinel (CONFIRMED lost vs AMBIGUOUS): a + * genuinely read-only kernel is indistinguishable from a + * no-execution here and simply pays the bounded extra polls (safe to + * re-execute by construction - a kernel that writes nothing has no + * side effects to duplicate). A + * wrong-place write detector was tried and removed: the highest-reloc + * data entry is often a UVM-internal buffer, not the output, so it + * false-fired on correct launches and would re-execute + * side-effecting kernels (atomics) up to 3x. + */ + if (req->n_data && gsp_phys_gr_data_chunk()->pages) { + u32 n_tracked = 0; + u32 nw_attempts = 0; + bool idem = false; + u32 wrote = gsp_phys_gr_output_retire_wait(req, bounce_csum_in, + &n_tracked); + + /* + * r434: post-launch hardware-only QMD dump (w50/w51 INNER + * get/put = the SKED's CTA dispatch counters) so the + * healthy-vs-wedged diff has the fields the submission-time + * dump cannot show. + */ + gsp_phys_gr_qmd64_post(qmd_dst_mem, "retire"); + + /* + * m179: the zero-write test is specific to launches with a + * non-empty TRACKED data set. A launch whose data entries do + * not resolve to any bounce page legitimately reports zero + * writes; treating that as a lost launch would re-execute a + * kernel for no reason (a false positive). Only + * n_tracked > 0 makes "wrote == 0" a possible no-execution + * signature. + */ + if (wrote == 0 && n_tracked) { + bool ff = gsp_phys_gr_qmd_fetched(qmd_dst_mem); + + pr_info("vgsp-gr: first-attempt witness: engine fetch %s (qmd sentinel %s) tracked=%u\n", + ff ? "OK" : "NONE", + ff ? "overwritten" : "intact", n_tracked); + } + /* + * r427 idempotent fast-accept: a zero-write launch whose identity + * (QMD minus release0 + staged content checksums) matches an + * earlier WRITING launch of this boot is a legitimate idempotent + * re-launch - the result is already in place. Accept immediately: + * no retries, no quiet holds. (See gsp_phys_gr_launch_fp.) + */ + if (wrote == 0 && n_tracked) { + u64 fp = gsp_phys_gr_launch_fp(qmd, req, bounce_csum_in); + + if (gsp_phys_gr_idem_match(fp)) { + pr_warn("vgsp-gr: IDEMPOTENT-ACCEPT h=0x%x push=0x%llx fp=0x%llx tracked=%u (same QMD+content as an earlier writing launch; no retries, no hold)\n", + req->h_object, req->push_va, fp, + n_tracked); + idem = true; + } + } + if (wrote == 0 && n_tracked && !idem && + gsp_phys_rmcall_gr_nowrite_retry_value()) { + u32 nw_retries = + (u32)gsp_phys_gr_nowrite_retries_value(); + /* + * r410: the sticky per-lane wedge (previous launch + * also no-wrote while the lane stays busy) is only + * cleared by idleness (r409 ~37 s, r416 ~52 s), so a + * back-to-back retry is futile and keeps it alive. + * When the streak says the lane is wedged, hold a + * quiet window with NO submissions before the retry + * (base on the first retry, doubled on the second), + * then let the normal fresh-bank + resched retry run. + * The quiet path relaxes the whole-launch budget + * (its own hold is the bound: 2 holds max) - the + * alternative is accepting every launch silently. + */ + u32 quiet_used = 0; + + while (wrote == 0 && nw_attempts < nw_retries) { + u32 quiet_ms = 0; + + if ((READ_ONCE(gsp_phys_gr_nw_streak) > 0 || + READ_ONCE(gsp_phys_gr_wedge_episode)) && + nw_attempts <= 1) { + quiet_ms = + gsp_phys_gr_wedge_quiet_ms_value(); + if (quiet_ms && nw_attempts) + quiet_ms <<= 1; + if (quiet_ms > 240000u) + quiet_ms = 240000u; + } + /* + * STALLFREE: the same whole-launch budget caps + * the retirement retry chain. When it is crossed + * the loop stops and the launch PROCEEDS with + * its current outcome (the zero-write accept + * below restores the release words and logs + * loudly), so a chain of slow dud retries can + * never hold the client past the budget. The + * r410 quiet path is exempt: it is entered only + * for a launch whose previous launch also + * no-wrote, and its own holds are bounded. + */ + if (!quiet_ms && + gsp_phys_gr_wait_budget_over(start)) { + gsp_phys_gr_stall_retry++; + gsp_phys_gr_stall_log("retry", + req->push_va, start, + "proceed"); + break; + } + nw_attempts++; + pr_info("vgsp-gr: outcome retry %u/%u (wrote=%u tracked=%u): refill + re-publish\n", + nw_attempts, nw_retries, + wrote, n_tracked); + /* + * r445 wedge hard re-arm (gsp_phys_gr_wedge_rearm, + * default off): the r445 evidence shows every + * post-dispatch no-store episode leaves the warm + * ladder sem probe ALIVE (the ladder block executes) + * while the client launch keeps writing nothing, so + * the plain refill+re-publish retries are futile. + * On the FINAL retry attempt, force one context + * save/unload (TSG PREEMPT bWait=1) and re-run the + * mode-35 lane-enable tail + schedules before the + * re-publish, so the retried launch meets a freshly + * restored context. Bounded: one preempt per launch + * (2 s RPC timeout), only on the last retry. + */ + if (nw_attempts == nw_retries && + gsp_phys_gr_wedge_rearm_value()) { + mutex_lock(&gsp_phys_rm_lock); + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + } + if (quiet_ms) { + gsp_phys_gr_wedge_quiet++; + pr_warn("vgsp-gr: WEDGE-QUIET hold %u ms before outcome retry %u (push=0x%llx; %u consecutive zero-write launch(es); no submissions in between)\n", + quiet_ms, nw_attempts, + req->push_va, + READ_ONCE(gsp_phys_gr_nw_streak)); + /* + * r420: publish the deliberate hold so + * the sema service's completion + * deadlines (the 12 s GR release hold + * and the 3 s per-launch consumption + * bound) pause instead of authoring + * this launch's release word under it - + * the client must not be released + * before the post-hold copy-out + * (gsp_phys_gr_quiet_hold_active). + */ + WRITE_ONCE(gsp_phys_gr_quiet_hold_on, + true); + msleep(quiet_ms); + WRITE_ONCE(gsp_phys_gr_quiet_hold_on, + false); + quiet_used = quiet_ms; + pr_warn("vgsp-gr: WEDGE-QUIET hold done (push=0x%llx); lanes re-armed by the retry below\n", + req->push_va); + } + /* + * m156 (E5): on the first zero-write retry, push the + * proven stage-4 GR semaphore block before re-publishing. + * It is the discriminator the lottery report asked for: + * if the channel can still dispatch the ladder's own + * 10-dword sem block, the engine is alive and the failed + * launch is a launch-form race (retry can recover); if + * the sem push times out, the engine/channel is wedged + * for the boot and every further retry is futile - break + * immediately instead of burning ~20 s of futile polls + * (r155: 3 outcome retries + rearm all dead after the + * wedge). Both outcomes are logged distinctly. + */ + if (nw_attempts == 1) { + int sp_ret; + + mutex_lock(&gsp_phys_rm_lock); + sp_ret = gsp_phys_rmcall_gr_sem_push_locked( + bar0); + mutex_unlock(&gsp_phys_rm_lock); + pr_info("vgsp-gr: outcome retry %u: warm sem push rc=%d (engine %s)\n", + nw_attempts, sp_ret, + sp_ret ? "WEDGED - skipping further retries" : + "ALIVE"); + if (sp_ret) { + gsp_phys_gr_sem_wedged++; + break; + } + } + /* + * m160 (fresh-ack + fresh-deposit retry): the m153 + * retry re-published the same PB and then polled + * the release words WITHOUT clearing them - the + * previous attempt's ack (the identical payload) + * satisfied the poll instantly, so every retry + * burned only the retirement wait and concluded + * "engine wrote 0" without ever having waited for + * the engine to consume the re-published entry + * (r162: 3 dud retries, ~870 ms each, then + * accepted). Clear the guest release words (and + * the module semaphore when the QMD won't reach a + * guest page) so the poll below waits for a FRESH + * ack, and move the inline-QMD deposit to the + * other qmdwin bank (fresh epoch) and rebuild the + * method block for it, so the engine cannot have + * a cached fetch line or a stale deposit for the + * retry's entry. The mechanical sequence is + * exactly the one r162's ok=7 saga used when it + * recovered the no-execute launch (fresh ring + * entry + resched + fresh deposit). + */ + /* + * r378: re-validate the guest release page + * before zeroing it - the HPA was resolved at + * submission and the retry runs seconds later. + */ + if (qmd_rel_hpa && + gsp_sema_gr_write_guard(req->rel_va, + qmd_rel_hpa, + "gr-rel-zero")) + WRITE_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + + qmd_rel_off), 0); + if (ch_rel_hpa) + WRITE_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + + ch_rel_off), 0); + WRITE_ONCE(*(u32 *)sem_va, 0); + WRITE_ONCE(*(u32 *)(sem_va + 4), 0); + wmb(); + /* + * F3 retry hygiene: recompute the win0 bank + * and slot limit for THIS attempt (they were + * fixed at the first submission's epoch) and + * re-copy the prepared param/program/bank + * shadow pages into the alternate bank before + * the re-publish. The qmdwin bank swap (below) + * is unchanged; when both are active they share + * the same fresh epoch. + */ + { + bool win0_on = + gsp_phys_rmcall_gr_shadow_win0_value() && + boot_stage.rmcall_gr_shadow_win0.pages; + bool win0_moved = false; + bool qmdwin_swap = qmdwin_on && + boot_stage.rmcall_gr_qmdwin.pages; + + if (qmdwin_swap || win0_on) { + epoch = ++boot_stage.rmcall_gr_submit_epoch; + if (!epoch) + epoch = ++boot_stage.rmcall_gr_submit_epoch; + boot_stage.rmcall_gr_shadow_win0_used = + (epoch & 1u) * + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; + if (win0_on) + win0_moved = + gsp_phys_gr_win0_retry_rearm( + qmd, req, + win0_bank_base, + boot_stage.rmcall_gr_shadow_win0_used, + boot_stage.rmcall_gr_shadow_win0_used + + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS); + win0_bank_base = + boot_stage.rmcall_gr_shadow_win0_used; + shadow_slot_limit = + win0_bank_base + + GSP_PHYS_RMCALL_GR_SHADOW0_BANK_SLOTS; + } + if (qmdwin_swap) { + qw_bank = (epoch & 1u) % + GSP_PHYS_RMCALL_GR_QMDWIN_SLOTS; + qmd_dst = + boot_stage.rmcall_gr_qmdwin_va + + (u64)qw_bank * + GSP_PHYS_PAGE_SIZE; + qmd_dst_mem = + gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_qmdwin, + qw_bank); + if (qmd_dst_mem) { + memcpy(qmd_dst_mem, qmd, + GSP_PHYS_GR_QMD_WORDS * 4); + put_unaligned_le32( + GSP_PHYS_GR_QMD_SENTINEL, + qmd_dst_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4); + /* + * m179: order the fresh + * deposit before the + * re-publish/doorbell + * below (the first-attempt + * path has the same + * barrier; the publish tail + * also carries dma_wmb + + * wmb, this keeps the + * deposit->kick order + * explicit on the retry + * path). + */ + wmb(); + } + } + if (qmdwin_swap || win0_moved) { + words = gsp_phys_rmcall_build_gr_launch( + methods, req, qmd, + qmd_dst, ch_sem_va, + want_ch); + memcpy(gsp_phys_chunk_page_va(pb, 0), + methods, words * 4); + } + if (qmdwin_swap) + pr_info("vgsp-gr: outcome retry %u: release words zeroed + fresh qmdwin bank %u dest 0x%llx + win0 bank %u%s + PB rebuilt\n", + nw_attempts, qw_bank, + qmd_dst, win0_bank_base, + win0_moved ? + " (shadow pages re-copied)" : + ""); + else if (win0_moved) + pr_info("vgsp-gr: outcome retry %u: release words zeroed + fresh win0 bank %u (shadow pages re-copied) + PB rebuilt\n", + nw_attempts, win0_bank_base); + else + pr_info("vgsp-gr: outcome retry %u: release words zeroed (win0/qmdwin off)\n", + nw_attempts); + } + gsp_phys_gr_fill_bounce(req, bounce_csum_in, + bounce_staged_hpa); + gsp_phys_gr_fill_gap_bounce(bounce_csum_in, + bounce_staged_hpa); + mutex_lock(&gsp_phys_rm_lock); + ret = gsp_phys_rmcall_publish_locked( + bar0, boot_stage.rmcall_va_pb, + words, &sub); + if (!ret) + ret = gsp_phys_rmcall_gr_resched_locked( + bar0); + mutex_unlock(&gsp_phys_rm_lock); + if (ret) { + pr_info("vgsp-gr: outcome retry %u publish/resched failed rc=%d (accepting current outcome)\n", + nw_attempts, ret); + ret = 0; + break; + } + { + unsigned long dl = jiffies + + msecs_to_jiffies( + GSP_PHYS_RMCALL_GR_POLL_MS); + unsigned long ack_start = jiffies; + u32 rw0 = 0, rw1 = 0; + bool acked = false; + + for (;;) { + if (qmd_rel_hpa) + rw0 = READ_ONCE( + *(u32 *)((u8 *)__va( + qmd_rel_hpa) + + qmd_rel_off)); + else if (qmd_rel) + rw0 = READ_ONCE( + *(u32 *)sem_va); + if (ch_rel_hpa) + rw1 = READ_ONCE( + *(u32 *)((u8 *)__va( + ch_rel_hpa) + + ch_rel_off)); + else + rw1 = READ_ONCE( + *(u32 *)(sem_va + 4)); + if ((!qmd_rel || + rw0 == want_qmd) && + rw1 == want_ch) { + acked = true; + break; + } + if (time_after_eq(jiffies, dl)) + break; + msleep(GSP_PHYS_RMCALL_POLL_MS); + } + pr_info("vgsp-gr: outcome retry %u: fresh ack %s (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x after %u ms)\n", + nw_attempts, + acked ? "OK" : "TIMEOUT", + rw0, want_qmd, rw1, + jiffies_to_msecs(jiffies - + ack_start)); + } + /* + * m161: hardware fetch witness. The fresh-ack + * poll cannot distinguish the engine's release + * from the sema service's software release (its + * pending-release sweep re-applies the payload + * as soon as the word reads below it). The QMD + * page's sentinel word is written only by the + * engine's QMD fetch, so it is the one side + * effect the sweep cannot fake: intact after the + * wait = this deposit never reached the engine. + */ + if (qmd_dst_mem) { + unsigned long wdl = jiffies + + msecs_to_jiffies(1500); + bool fetched = false; + + while (time_before(jiffies, wdl)) { + if (gsp_phys_gr_qmd_fetched( + qmd_dst_mem)) { + fetched = true; + break; + } + msleep(5); + } + pr_info("vgsp-gr: outcome retry %u: engine fetch witness %s (qmd sentinel %s)\n", + nw_attempts, + fetched ? "OK" : "NONE", + fetched ? "overwritten" : "intact"); + } + wrote = gsp_phys_gr_output_retire_wait( + req, bounce_csum_in, &n_tracked); + gsp_phys_gr_qmd64_post(qmd_dst_mem, + "retry-retire"); + pr_info("vgsp-gr: outcome retry %u done: engine wrote %u page(s)\n", + nw_attempts, wrote); + /* r410: the quiet-window verdict. */ + if (quiet_used) { + if (wrote) { + gsp_phys_gr_wedge_heals++; + WRITE_ONCE(gsp_phys_gr_wedge_episode, + true); + pr_warn("vgsp-gr: WEDGE-QUIET healed: engine wrote %u page(s) after the %u ms hold (streak was %u)\n", + wrote, quiet_used, + READ_ONCE( + gsp_phys_gr_nw_streak)); + } else { + pr_warn("vgsp-gr: WEDGE-QUIET no heal: %u ms quiet and retry %u still wrote 0 page(s)\n", + quiet_used, nw_attempts); + } + quiet_used = 0; + } + } + } + /* + * m160/m179: if every fresh-ack retry still produced no engine + * writes, the release words may be left zeroed from the last + * retry. Restore the expected payloads so the guest client + * observes its completion and proceeds to its own result check + * (a genuinely lost launch then surfaces as the client's own + * FAIL, never as a hang). This accept runs even when the retry + * loop is disabled (gsp_phys_rmcall_gr_nowrite_retry=0) so a + * zero-write outcome is never silent. + */ + if (wrote == 0 && n_tracked && !idem) { + bool restored = false; + bool fetched = gsp_phys_gr_qmd_fetched(qmd_dst_mem); + + /* + * m179: the no-write accept is the last resort after + * the dedicated re-execution budget + * (gsp_phys_gr_nowrite_retries). It stays + * unconditional - the alternative is stalling the + * client forever - but it is now LOUD and classified + * by the hardware fetch witness: an intact sentinel + * means the deposit never reached the engine (a + * CONFIRMED lost launch, the sweep/hold-faked-ack + * class); an overwritten sentinel means the engine + * consumed the deposit, so the zero tracked writes + * may also be a legitimately read-only kernel + * (AMBIGUOUS). + */ + pr_warn("vgsp-gr: NOWRITE-%s h=0x%x push=0x%llx n_data=%u tracked=%u wrote=0 retries=%u qmd_rel=0x%x (engine fetch %s) consec=%u quiet=%llu heals=%llu\n", + fetched ? "AMBIGUOUS" : "CONFIRMED", + req->h_object, req->push_va, req->n_data, + n_tracked, nw_attempts, want_qmd, + fetched ? "OK" : "NONE", + READ_ONCE(gsp_phys_gr_nw_streak) + 1u, + gsp_phys_gr_wedge_quiet, + gsp_phys_gr_wedge_heals); + + /* + * m180 (lottery discriminator, agent-order H1/H2). + * Read BOTH qmdwin banks' sentinel words. Facts only: + * the other bank is normally the previous launch's + * deposit (legitimately consumed), so only THIS + * launch's sentinel classifies the event: intact = + * this deposit never reached the engine; overwritten + * = the engine consumed it (or something zeroed the + * word), so the zero stores are an engine-side + * non-execution of a fetched deposit, not a lost + * deposit. + */ + if (qmdwin_on && boot_stage.rmcall_gr_qmdwin.pages && + qmd_dst_mem) { + u8 *om = gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_qmdwin, + qw_bank ^ 1u); + + pr_warn("vgsp-gr: zero-write deposit stamp: bank=%u sentinel=0x%08x (%s) other-bank=%u %s\n", + qw_bank, + get_unaligned_le32( + qmd_dst_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4), + fetched ? "this deposit fetched" : + "this deposit NOT fetched", + qw_bank ^ 1u, + om ? (gsp_phys_gr_qmd_fetched(om) ? + "sentinel overwritten (previous deposit)" : + "sentinel intact") : "n/a"); + } + /* + * (b) Deposit readback at the accept (m181): the same + * QMD-word dump the first-timeout path logs, so a + * zero-write launch's deposit content (w0/w4/w12/.../ + * w63) shows whether the engine consumed the header + * words or the page is untouched, without needing a + * timeout to have happened first. + */ + gsp_phys_gr_deposit_rb_log(req, qmd_dst_mem); + /* + * GRFIX FIX4 (agent-A decisive experiment): bounded + * real-GR register dump at the zero-write accept, one + * line per classified launch, using the same readl + * accessor and the same offsets as the read-only + * debugfs probe (gsp_phys_reg_probe_show): FE_HWW_ESR + * 0x404000, SKED_HWW_ESR 0x407020, PGRAPH_TRAPPED_ADDR + * 0x400704 + TRAPPED_DATA_LO/HI 0x400708/70c, + * FECS_IRQSTAT 0x409008, FECS_CUR/NEW_CTX 0x409b00/04. + * No new locks or sleeps: the submit context already + * holds gsp_phys_rm_lock exactly as the surrounding + * post-mortem reads do. Zero is a legitimate value for + * several of these (no context resident), so the line + * is classified with the sentinel/faultbuf evidence, + * not on its own. + */ + if (bar0) + pr_warn("vgsp-gr: NOWRITE-REGS fe_esr=0x%08x sked_esr=0x%08x trap=0x%08x/%08x/%08x fecs_irq=0x%08x ctx=0x%08x->0x%08x\n", + readl(bar0 + 0x00404000u), + readl(bar0 + 0x00407020u), + readl(bar0 + 0x00400704u), + readl(bar0 + 0x00400708u), + readl(bar0 + 0x0040070cu), + readl(bar0 + 0x00409008u), + readl(bar0 + 0x00409b00u), + readl(bar0 + 0x00409b04u)); + /* r425: the wedged-launch SKED/FECS baseline. */ + gsp_phys_gr_sked_wit(bar0, "accept"); + /* + * r425 stale-store witness: re-scan the last completed + * launch's first engine-changed page. A change since its + * copy-out proves a late store landed after the launch + * was shipped (a stuck/queued grid or a stale-QMD + * execution), the mechanism behind the "1 page" heals. + */ + if (gsp_phys_gr_prev_out_valid && + bounce_staged_hpa[gsp_phys_gr_prev_out_bi] == 0 && + gsp_phys_gr_data_chunk()->pages) { + const u8 *pp = gsp_phys_chunk_page_va( + gsp_phys_gr_data_chunk(), + gsp_phys_gr_prev_out_bi); + u64 pc; + + if (pp) { + pc = gsp_phys_gr_fnv1a64(pp, GSP_PHYS_PAGE_SIZE); + if (pc != gsp_phys_gr_prev_out_csum) + pr_warn("vgsp-gr: PREV-OUT-CHANGED bi=%u fp %016llx -> %016llx (late store after its launch shipped)\n", + gsp_phys_gr_prev_out_bi, + gsp_phys_gr_prev_out_csum, + pc); + } + } + /* + * (c) Late-store check: a store ramp that retires after + * the host's window is indistinguishable from a lost + * dispatch at the accept moment. Re-scan once at +1 s + * and report whether the tracked pages changed in the + * meantime; the copy-out below picks up any stores + * that arrived late, so this both classifies and heals + * the late sub-case. + */ + { + u32 late_tracked = 0; + u32 late; + + msleep(1000); + late = gsp_phys_gr_scan_pages(req, bounce_csum_in, + &late_tracked, NULL); + pr_warn("vgsp-gr: zero-write late-check: %u/%u pages changed after +1000 ms (%s)\n", + late, late_tracked, + late ? "LATE-STORES - copy-out will ship them" : + "NO-STORES - dispatch lost or read-only kernel"); + } + + if (qmd_rel_hpa) { + u8 *rp = (u8 *)__va(qmd_rel_hpa) + qmd_rel_off; + + /* + * r378: same target re-validation as the + * retry zero above. + */ + if (READ_ONCE(*(u32 *)rp) != want_qmd && + gsp_sema_gr_write_guard(req->rel_va, + qmd_rel_hpa, + "gr-rel-restore")) { + WRITE_ONCE(*(u32 *)rp, want_qmd); + restored = true; + } + } + if (ch_rel_hpa) { + u8 *rp = (u8 *)__va(ch_rel_hpa) + ch_rel_off; + + if (READ_ONCE(*(u32 *)rp) != want_ch) { + WRITE_ONCE(*(u32 *)rp, want_ch); + restored = true; + } + } + if (restored) + pr_warn("vgsp-gr: no-write accept: release words restored (qmd=0x%08x ch=0x%08x); client's own check decides\n", + want_qmd, want_ch); + } + /* + * r427: record the launch identity of every launch that WROTE, + * so a later idempotent re-launch (zero writes, same identity) + * can accept immediately (see gsp_phys_gr_launch_fp). + */ + if (wrote && n_tracked) + gsp_phys_gr_idem_record( + gsp_phys_gr_launch_fp(qmd, req, bounce_csum_in)); + /* + * r410: streak bookkeeping for the quiet-window trigger. A + * launch with tracked data that wrote nothing extends the + * zero-write streak; any launch that wrote resets it. Runs + * even when the retry loop is disabled so the streak always + * reflects the accepted-outcome history. + */ + if (n_tracked) { + WRITE_ONCE(gsp_phys_gr_nw_streak, + wrote ? 0u : + READ_ONCE(gsp_phys_gr_nw_streak) + 1u); + /* r411: arm the wedge episode on any accepted + * zero-write launch, so the next zero-write launch + * quiets even after a heal reset the streak. */ + if (!wrote) + WRITE_ONCE(gsp_phys_gr_wedge_episode, true); + } + } + /* * m114: the kernel ran against the pre-commit bounce window; copy the * results back to the guest's real pages (identity HPAs in the * request). All mapped pages are written back; inputs are unchanged * by the kernels, so copying them back is a no-op. */ - if (req->n_data && boot_stage.rmcall_gr_bounce.pages) { - u32 wbi, wrote = 0, changed = 0; + if (req->n_data && gsp_phys_gr_data_chunk()->pages) { + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u32 wbi, wrote = 0, changed = 0, eligible = 0; + u32 aliased = 0, unchanged = 0, skipped = 0; u32 chg[8] = { }; + u32 outwit = 0; /* COPYWIT: bounded per-launch witness lines */ for (wbi = 0; wbi < req->n_data; wbi++) { - u32 bi = (u32)((req->data[wbi].reloc_va - - GSP_PHYS_RMCALL_GR_BOUNCE_VA0) >> - 12); + u32 bi = gsp_phys_gr_data_bi(req->data[wbi].reloc_va); const u8 *src; - u32 ci, csum = 0; + u64 fp; - if (bi >= GSP_PHYS_RMCALL_GR_MAIL_SLOT || + if (bi == GSP_PHYS_GR_DATA_BI_NONE || !req->data[wbi].hpa) continue; - src = gsp_phys_chunk_page_va( - &boot_stage.rmcall_gr_bounce, bi); - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != bounce_csum_in[bi]) { + src = gsp_phys_chunk_page_va(dchunk, bi); + if (!src) + continue; + eligible++; + /* + * r378: only the page whose content was staged into + * this slot this launch may be written back. A rec + * aliasing another rec's reloc slot would otherwise + * ship the other page's content into this guest page + * (the multi-writer clobber class), and a slot left + * over from a previous launch likewise. + */ + if (bounce_staged_hpa[bi] != req->data[wbi].hpa) { + aliased++; + continue; + } + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { + /* + * r378: the engine did not touch this page, + * so the guest page still holds exactly this + * content (the slot was copied in from it). + * Writing it back is a no-op at best and + * stale-content corruption at worst - skip. + */ + unchanged++; /* - * m142: which pages the engine touched. The - * indexes name the relocated VA (bi page of - * 0x110000); a c-range write is bi >= 0x40. - * Also dump the first words of the first two - * changed pages: float sums prove the kernel - * wrote them (the c-pointer landed on this - * buffer); pointers/QMD words prove something - * else wrote. + * r452: witness the UNCHANGED page's chunk + * content (first 4 words) and the guest + * page's, bounded per launch. This is the + * only place the two can be compared for a + * page the csum rule classified as "engine + * did not touch" - it separates "the store + * never landed in the chunk" from "the + * store landed but the csum rule missed + * it" (or "the reader resolves a different + * page than the writer wrote"). + */ + if (gsp_phys_gr_copywit_enabled() && + outwit < GSP_PHYS_GR_COPYWIT_MAX) { + const u8 *gp = req->data[wbi].hpa ? + (const u8 *)__va(req->data[wbi].hpa) : + NULL; + + outwit++; + pr_info("vgsp-gr: OUT-UNCH bi=%u va=0x%llx fp=%016llx pre=%016llx chunk=%08x %08x %08x %08x guest=%08x %08x %08x %08x\n", + bi, req->data[wbi].va, fp, + bounce_csum_in[bi], + get_unaligned_le32(src), + get_unaligned_le32(src + 4), + get_unaligned_le32(src + 8), + get_unaligned_le32(src + 12), + gp ? get_unaligned_le32(gp) : 0, + gp ? get_unaligned_le32(gp + 4) : 0, + gp ? get_unaligned_le32(gp + 8) : 0, + gp ? get_unaligned_le32(gp + 12) : 0); + } + continue; + } + { + u64 f8 = (u64)get_unaligned_le64(src); + + /* + * m142: which pages the engine touched. bi is + * the page index inside the active data region + * (grown: VA 0x0a000000 + bi*4096; legacy: + * 0x110000 + bi*4096). Also dump the first + * words of the first two changed pages: float + * sums prove the kernel wrote them (the + * c-pointer landed on this buffer); + * pointers/QMD words prove something else + * wrote. */ if (changed < 2) pr_info("vgsp-gr: changed page bi=%u w0..3=%08x %08x %08x %08x\n", @@ -17912,34 +21809,173 @@ gr_page_unmapped: get_unaligned_le32(src + 4), get_unaligned_le32(src + 8), get_unaligned_le32(src + 12)); + /* + * COPYWIT fail4: per-changed-page copy-out + * witness - the exact (device VA -> guest + * HPA) pair the results were written back to + * plus the first/last 8 bytes shipped and the + * pre/post csum pair (proof the engine changed + * the page vs the copy-in baseline). The D2H + * side of the same VA is named by the + * sema-bigcopy resolution lines. + */ + if (gsp_phys_gr_copywit_enabled() && + outwit < GSP_PHYS_GR_COPYWIT_MAX) { + /* + * r391 diagnostic (sum=0 class): the + * landing-table resolution of this VA + * (the page the D2H reader will use) + * and its first 8 bytes beside the + * write target - a launch/readback + * disagreement in one line. + */ + u64 rpa = 0, rval = 0; + u32 rap = 0; + bool rhave = gsp_sema_gr_va_snap( + req->data[wbi].va, &rpa, + &rap, &rval); + + outwit++; + pr_info("vgsp-gr: OUT-PAGE-WIT va=0x%llx hpa=0x%llx bi=%u first8=%016llx last8=%016llx pre=%016llx post=%016llx res=%s pa=0x%llx ap=%u val8=%016llx vbase=%px\n", + req->data[wbi].va, + req->data[wbi].hpa, bi, f8, + (u64)get_unaligned_le64( + src + GSP_PHYS_PAGE_SIZE - + 8), + bounce_csum_in[bi], fp, + rhave ? "ledger" : "none", + rpa, rap, rval, + (void *)gsp_sema_gr_shadow_base()); + } + /* + * OUT-ALIAS-WIT: identical first 8 bytes as + * an earlier changed page of this launch = + * the replicated-tile structure, with both + * recs named. r226: all-zero pages are + * excluded on both sides (the fresh-client + * fill stages zero runs; they are not alias + * evidence). + */ + if (gsp_phys_gr_copywit_enabled() && + !gsp_phys_gr_page_all_zero( + gsp_phys_gr_page_sum32(src), src)) { + u32 wi; + + for (wi = 0; wi < gsp_phys_gr_outwit_n; + wi++) { + if (gsp_phys_gr_outwit_first8[wi] != + f8) + continue; + if (gsp_phys_gr_alias_log < 12) { + gsp_phys_gr_alias_log++; + pr_info("vgsp-gr: OUT-ALIAS-WIT va=0x%llx bi=%u partner_va=0x%llx partner_bi=%u first8=%016llx\n", + req->data[wbi].va, + bi, + gsp_phys_gr_outwit_va[wi], + gsp_phys_gr_outwit_bi[wi], + f8); + } + break; + } + if (gsp_phys_gr_outwit_n < + GSP_PHYS_GR_OUTWIT_SLOTS) { + gsp_phys_gr_outwit_first8[gsp_phys_gr_outwit_n] = f8; + gsp_phys_gr_outwit_va[gsp_phys_gr_outwit_n] = req->data[wbi].va; + gsp_phys_gr_outwit_bi[gsp_phys_gr_outwit_n] = bi; + gsp_phys_gr_outwit_n++; + } + } if (changed < 8) chg[changed] = bi; + /* + * r425: remember the first engine-changed page + * (post-copy-out content) for the next zero-write + * accept's late-store witness. + */ + if (changed == 0) { + gsp_phys_gr_prev_out_bi = bi; + gsp_phys_gr_prev_out_csum = fp; + gsp_phys_gr_prev_out_valid = true; + } changed++; } + /* + * r378: the target HPA was resolved at submission; + * the poll/retire/retry window is seconds wide and a + * guest buffer freed and re-homed in it would receive + * this full page through the stale target. Re-resolve + * the VA and skip (bounded log) on a positive + * disagreement. + */ + if (!gsp_sema_gr_write_guard(req->data[wbi].va, + req->data[wbi].hpa, + "gr-bounce-copy-out")) { + skipped++; + continue; + } memcpy((u8 *)__va(req->data[wbi].hpa), src, GSP_PHYS_PAGE_SIZE); + gsp_phys_gr_page_spot_check( + (const u8 *)__va(req->data[wbi].hpa), src, + req->data[wbi].va, "gr-bounce-copy-out"); + /* + * r391d post-write witness (sum=0 class): read the VA's + * landing page right after the write through the same + * path the D2H reader uses. If val8 is not first8 the + * write target and the reader's page are different + * (aperture/staleness disagreement), not a lost write. + */ + if (outwit < GSP_PHYS_GR_COPYWIT_MAX) { + u64 rpa2 = 0, rval2 = 0; + u32 rap2 = 0; + + if (gsp_sema_gr_va_snap(req->data[wbi].va, + &rpa2, &rap2, &rval2)) { + outwit++; + pr_info("vgsp-gr: OUT-PAGE-POST va=0x%llx hpa=0x%llx res_pa=0x%llx ap=%u val8=%016llx vbase=%px\n", + req->data[wbi].va, + req->data[wbi].hpa, rpa2, + rap2, rval2, + (void *)gsp_sema_gr_shadow_base()); + } + } wrote++; } + if ((u64)req->n_data << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-bounce-copy-out va=0x%llx len=0x%llx done=0x%llx\n", + req->data[0].va, (u64)req->n_data << 12, + (u64)wrote << 12); + /* + * r378: the copy-out is now selective (staged + changed + + * re-validated pages only), so completeness is an accounting + * check over the eligible recs: every one is either written, + * skipped as unchanged, skipped as aliased, or refused by the + * write guard (logged as WGUARD-SKIP). This can only fire if + * a rec vanished between the filters. + */ + if (wrote + unchanged + aliased + skipped < eligible) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-bounce-copy-out\n", + req->data[0].va, + (u64)eligible << 12, + (u64)(wrote + unchanged + aliased + + skipped) << 12); /* m143b: unconditional first words of the first page of each - * 32-page relocation window (a=0x110000, b=0x130000, c=0x150000) - * after the run, so the engine's read/write footprint per window - * is provable without relying on the changed-page counter. */ + * 32-page relocation window after the run, so the engine's + * read/write footprint per window is provable without relying + * on the changed-page counter. */ { u32 wi; for (wi = 0; wi + 1 < req->n_data; wi += 32) { - u32 bi = (u32)((req->data[wi].reloc_va - - GSP_PHYS_RMCALL_GR_BOUNCE_VA0) >> - 12); + u32 bi = gsp_phys_gr_data_bi( + req->data[wi].reloc_va); const u8 *wp; u32 wc = 0, k; - if (bi >= GSP_PHYS_RMCALL_GR_MAIL_SLOT) + if (bi == GSP_PHYS_GR_DATA_BI_NONE) continue; - wp = gsp_phys_chunk_page_va( - &boot_stage.rmcall_gr_bounce, bi); - for (k = 0; k < GSP_PHYS_PAGE_SIZE; k += 4) - wc += get_unaligned_le32(wp + k); + wp = gsp_phys_chunk_page_va(dchunk, bi); + (void)k; (void)wc; pr_info("vgsp-gr: window[%u] reloc=0x%llx first=%08x %08x %08x %08x changed=%u\n", bi, req->data[wi].reloc_va, @@ -17947,13 +21983,84 @@ gr_page_unmapped: get_unaligned_le32(wp + 4), get_unaligned_le32(wp + 8), get_unaligned_le32(wp + 12), - wc != bounce_csum_in[bi]); + gsp_phys_gr_fnv1a64( + wp, GSP_PHYS_PAGE_SIZE) != + bounce_csum_in[bi]); } } - pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine (first bi: %u %u %u %u %u %u %u %u)\n", - wrote, changed, chg[0], chg[1], chg[2], chg[3], chg[4], + pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u alias-skipped, %u stale-skipped (first bi: %u %u %u %u %u %u %u %u)\n", + wrote, changed, unchanged, aliased, skipped, + chg[0], chg[1], chg[2], chg[3], chg[4], chg[5], chg[6], chg[7]); } + /* + * relfix: copy the late-resolved pages back to the guest pages the + * resolver named - the same identity the word rewrite handed the + * engine - after the retire wait, exactly like the tracked data + * pages above. + */ + if (gsp_phys_gr_gap_n && gsp_phys_gr_data_chunk()->pages) { + struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); + u32 gi, gwrote = 0, gchanged = 0; + u32 gunchanged = 0, gskipped = 0; + u32 gchg[4] = { }; + + for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { + u32 bi = gsp_phys_gr_gap[gi].bi; + const u8 *src = gsp_phys_chunk_page_va(dchunk, bi); + u64 fp; + + if (!src) + continue; + /* + * r378: same staging-identity rule as the tracked + * copy-out above - only the page staged into this + * slot may receive the slot's content. + */ + if (bounce_staged_hpa[bi] != gsp_phys_gr_gap[gi].hpa) { + gskipped++; + continue; + } + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { + /* Engine did not touch it: nothing to ship. */ + gunchanged++; + continue; + } + if (gchanged < 4) + gchg[gchanged] = bi; + gchanged++; + if (!gsp_sema_gr_write_guard(gsp_phys_gr_gap[gi].va, + gsp_phys_gr_gap[gi].hpa, + "gr-gap-copy-out")) { + gskipped++; + continue; + } + memcpy((u8 *)__va(gsp_phys_gr_gap[gi].hpa), src, + GSP_PHYS_PAGE_SIZE); + gsp_phys_gr_page_spot_check( + (const u8 *)__va(gsp_phys_gr_gap[gi].hpa), src, + gsp_phys_gr_gap[gi].va, "gr-gap-copy-out"); + gwrote++; + } + pr_info("vgsp-gr: gap copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u stale-skipped (first bi: %u %u %u %u)\n", + gwrote, gchanged, gunchanged, gskipped, + gchg[0], gchg[1], gchg[2], gchg[3]); + if ((u64)gsp_phys_gr_gap_n << 12 >= GSP_PHYS_GR_AUDIT_MIN) + pr_info("vgsp: COPYAUDIT path=gr-gap-copy-out va=0x%llx len=0x%llx done=0x%llx\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gwrote << 12); + /* + * r378: same accounting rule as the tracked copy-out above - + * the intentional unchanged/stale skips are not truncation. + */ + if (gwrote + gunchanged + gskipped < gsp_phys_gr_gap_n) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=gr-gap-copy-out\n", + gsp_phys_gr_gap[0].va, + (u64)gsp_phys_gr_gap_n << 12, + (u64)gwrote << 12); + } poll = w1; if (bar0) pr_info("vgsp-gr: SKED readback (ok): HWW_ESR=0x%08x INFO2=0x%08x CWD_T0=0x%08x CWD_T1=0x%08x CWD_T2=0x%08x\n", @@ -17962,13 +22069,42 @@ gr_page_unmapped: ioread32(bar0 + 0x00407060u), ioread32(bar0 + 0x00407064u), ioread32(bar0 + 0x00407068u)); + /* m151: post-poll page hashes; compare with the pre-doorbell ones. */ + { + const u8 *wp = boot_stage.rmcall_gr_shadow_win0.pages ? + gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_shadow_win0, + win0_bank_base) : NULL; + + if (wp || guest_param_pg) + pr_info("vgsp-gr: win0 param hash post=0x%016llx guest-param hash post=0x%016llx\n", + wp ? gsp_phys_gr_fnv1a64(wp, + GSP_PHYS_PAGE_SIZE) : + 0ULL, + guest_param_pg ? + gsp_phys_gr_fnv1a64(guest_param_pg, + GSP_PHYS_PAGE_SIZE) : + 0ULL); + } pr_info("vgsp-gr: GUEST GR LAUNCH OK h=0x%x push=0x%llx qmd=0x%llx prog/pages=%u qmd_rel=0x%08x ch_rel=0x%08x elapsed_ms=%u ok=%llu\n", req->h_object, req->push_va, req->qmd_va, req->n_pages, w0, w1, jiffies_to_msecs(jiffies - start), gsp_phys_gr_ok); /* m128: post-success qmdwin words so a probe that stores a value into * the qmdwin page (e.g. p6's constant-read dump at w48) is readable - * from the success log (the post-mortem only prints on timeout). */ - if (boot_stage.rmcall_gr_qmdwin.pages) { + * from the success log (the post-mortem only prints on timeout). + * r434: read the launch's OWN bank (qmd_dst_mem) when available - + * the old bank-0-only read showed a stale page for odd epochs. */ + if (qmd_dst_mem) { + const u32 *qw = (const u32 *)qmd_dst_mem; + + pr_info("vgsp-gr: post-OK qmdwin w48..59=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + get_unaligned_le32(qw + 48), get_unaligned_le32(qw + 49), + get_unaligned_le32(qw + 50), get_unaligned_le32(qw + 51), + get_unaligned_le32(qw + 52), get_unaligned_le32(qw + 53), + get_unaligned_le32(qw + 54), get_unaligned_le32(qw + 55), + get_unaligned_le32(qw + 56), get_unaligned_le32(qw + 57), + get_unaligned_le32(qw + 58), get_unaligned_le32(qw + 59)); + } else if (boot_stage.rmcall_gr_qmdwin.pages) { const u32 *qw = (const u32 *)gsp_phys_chunk_page_va( &boot_stage.rmcall_gr_qmdwin, 0); @@ -17980,11 +22116,19 @@ gr_page_unmapped: ret = 0; out: + /* + * M91 FAULT-CAP: one line per faulting launch, emitted at the end of + * the launch window (fault epoch compared against the snapshot taken + * at submission; inert - one compare - when no fault was seen). + */ + gsp_phys_gr_m91_fault_cap(req, qmd, qmd_dst, qmd_dst_mem, attempts, + rearmed, fast_rearmed, ret == 0); if (out) { out->sem_value = poll; out->elapsed_ms = jiffies_to_msecs(jiffies - start); out->rc = ret; out->fault = fault; + out->qmd_rel_redirected = qmd_rel_redirected; } return ret; } @@ -18511,7 +22655,11 @@ static int gsp_phys_rmcall_guest_g2h(void __iomem *bar0) return -EIO; } - ret = gsp_phys_gpa_to_hpa(gpa, &hpa); + /* + * Single-guest pin (see gsp_phys_guest_gpa_write): NULL selects the + * first open mdev, exactly as before the per-mdev records. + */ + ret = gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, &hpa); if (ret) { pr_err("vgsp-boot: rm-call: GUEST G2H COPY FAILED: hpa=0x%llx pte=0x%016llx: GPA 0x%llx -> HPA translation failed (%d: no active mdev/VFIO domain, GPA unmapped, or not a RAM PFN)\n", hpa, pte, gpa, ret); @@ -18594,7 +22742,11 @@ static int gsp_phys_rmcall_guest_w2g(void __iomem *bar0) return -EIO; } - ret = gsp_phys_gpa_to_hpa(gpa, &hpa); + /* + * Single-guest pin (see gsp_phys_guest_gpa_write): NULL selects the + * first open mdev, exactly as before the per-mdev records. + */ + ret = gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, &hpa); if (ret) { pr_err("vgsp-boot: rm-call: GUEST W2G COPY FAILED: hpa=0x%llx pte=0x%016llx: GPA 0x%llx -> HPA translation failed (%d: no active mdev/VFIO domain, GPA unmapped, or not a RAM PFN)\n", hpa, pte, gpa, ret); @@ -18821,6 +22973,14 @@ static void gsp_phys_rmcall_fini(struct pci_dev *pdev) gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_pt[i]); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_shadow); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_qmdwin); + /* + * B-2: the old 208-page bounce window (previously leaked here) and the + * grown data-bounce region chunks. The region's PT chunk and data chunk + * are freed only when allocated. + */ + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_bounce); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_data_bounce); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_data_bounce_pt); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_patch); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_pm); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_fecs); @@ -18992,8 +23152,7 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) fwd_cand_none, fwd_cand_sys, fwd_cand_mixed, fwd_cand_vid, fwd_resolve_fail, boot_stage.rmcall_guest_pt_next_line, - GSP_PHYS_RMCALL_GUEST_PT_PAGES * - GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE); + GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES); } { /* @@ -19073,11 +23232,18 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) gsp_phys_backend_gr_counters(&b_sub, &b_ok, &b_refused, &b_timeout, &b_pool); seq_printf(m, - "work_gr: req=%llu fallback=%llu ok=%llu refused=%llu timeout=%llu fault=%llu submitted=%llu backend_ok=%llu backend_refused=%llu backend_timeout=%llu pool_exhausted=%llu forward_param=%u exec_ok=%u\n", + "work_gr: req=%llu fallback=%llu ok=%llu refused=%llu timeout=%llu fault=%llu submitted=%llu backend_ok=%llu backend_refused=%llu backend_timeout=%llu pool_exhausted=%llu forward_param=%u exec_ok=%u sem_wedged=%llu stall_wait_max=%llu stall_retry=%llu wait_max_ms=%u nw_streak=%u wedge_quiet=%llu wedge_heals=%llu\n", gr_req, gr_fallback, gr_ok, gr_refused, gr_timeout, gr_fault, b_sub, b_ok, b_refused, b_timeout, b_pool, gsp_phys_gr_forward_enabled(), - boot_stage.rmcall_gr_exec_ok); + boot_stage.rmcall_gr_exec_ok, + READ_ONCE(gsp_phys_gr_sem_wedged), + READ_ONCE(gsp_phys_gr_stall_wait_max), + READ_ONCE(gsp_phys_gr_stall_retry), + gsp_phys_gr_wait_max_ms_value(), + READ_ONCE(gsp_phys_gr_nw_streak), + READ_ONCE(gsp_phys_gr_wedge_quiet), + READ_ONCE(gsp_phys_gr_wedge_heals)); } { /* @@ -19806,6 +23972,18 @@ static ssize_t gsp_phys_guest_gpa_write(struct file *file, gpa); return -EINVAL; } + /* + * Dual-guest guard (goal 1.0): the pin is a single-guest test aid. The + * triggers translate it through the first open mdev (see + * gsp_phys_gpa_to_hpa / boot_stage.rmcall_guest_gpa), so with more than + * one open mdev the pin cannot name its owner and is refused instead of + * silently resolving through whichever guest opened first. + */ + if (gsp_phys_open_mdev_count() > 1) { + pr_err("vgsp-boot: gsp-phys-guest-gpa: refused: %d mdevs are open; the guest-GPA pin is a single-guest test aid (translates through the first open mdev). Close all but one mdev, or use the per-mdev guest/GPA paths\n", + gsp_phys_open_mdev_count()); + return -EBUSY; + } if (boot_stage.rmcall_guest_gpa_valid && boot_stage.rmcall_guest_gpa != gpa) { pr_err("vgsp-boot: gsp-phys-guest-gpa: refused: this boot already pinned GPA 0x%llx; one GPA per boot (stale GMMU TLB safety). Reuse that GPA, or release/reboot before changing it\n", @@ -19828,18 +24006,26 @@ static const struct file_operations gsp_phys_guest_gpa_fops = { }; /* - * gsp-phys-guest-ram (write-only, mode 0200): configure the shared guest RAM - * backing file that gsp_phys_gpa_to_hpa() translates through (QEMU + * gsp-phys-guest-ram (write-only, mode 0200): configure a guest RAM backing + * file that gsp_phys_gpa_to_hpa_vgpu() translates through (QEMU * memory-backend-file,...,share=yes). One write is * * "<path> <gpa_base_hex> <file_offset_hex> <size_hex>" * * (path without spaces; all three numbers hex with 0x, e.g. - * "/dev/shm/vgpu-ram 0x100000000 0x0 0x40000000"). The first write opens the - * file and fixes the path for this module cycle; later writes may add ranges - * (up to four) or replace a range with the same gpa_base. The ranges are - * only consulted by gsp_phys_gpa_to_hpa(), and only while a live mdev - * session exists. A single "-" clears the configuration and closes the file. + * "/dev/shm/vgpu-ram 0x100000000 0x0 0x40000000"). This historical form + * configures the legacy record and serves the FIRST/ONLY open mdev exactly + * as before the per-mdev records existed. With two live guests each writes + * its own record by naming its mdev UUID first: + * + * "<mdev-uuid> <path> <gpa_base_hex> <file_offset_hex> <size_hex>" + * + * Paths and UUIDs contain no spaces, so the token count (4 vs 5) decides. + * The first write for a record opens the file and fixes the path for that + * record; later writes may add ranges (up to four) or replace a range with + * the same gpa_base. The ranges are only consulted by the translators, and + * only while a live mdev session exists. A single "-" clears ALL records + * (legacy and per-mdev) and closes the files. */ static char *gsp_phys_guest_ram_token(char **cursor) { @@ -19854,9 +24040,11 @@ static ssize_t gsp_phys_guest_ram_write(struct file *file, const char __user *ubuf, size_t len, loff_t *ppos) { - char buf[384]; - char *p, *tok, *path; + char buf[448]; + char *t[5]; + char *path; u64 gpa_base, file_offset, size; + int nt = 0, off; int ret; if (len == 0 || len >= sizeof(buf)) @@ -19865,30 +24053,63 @@ static ssize_t gsp_phys_guest_ram_write(struct file *file, return -EFAULT; buf[len] = '\0'; - p = strim(buf); - if (!strcmp(p, "-")) { + if (!strcmp(strim(buf), "-")) { gsp_phys_guest_ram_clear(); pr_info("vgsp-boot: gsp-phys-guest-ram: cleared the guest RAM file configuration\n"); return len; } - path = gsp_phys_guest_ram_token(&p); - tok = gsp_phys_guest_ram_token(&p); - if (!path || !tok || kstrtoull(tok, 0, &gpa_base)) { + { + char *p = buf; + + while (nt < 5 && (t[nt] = gsp_phys_guest_ram_token(&p))) + nt++; + } + if (nt != 4 && nt != 5) { + pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected \"<path> <gpa_base_hex> <file_offset_hex> <size_hex>\", \"<mdev-uuid> <path> <gpa_base_hex> <file_offset_hex> <size_hex>\" or \"-\"\n"); + return -EINVAL; + } + off = nt - 4; /* 0 = legacy form, 1 = per-mdev form */ + path = t[off]; + if (!path || kstrtoull(t[off + 1], 0, &gpa_base)) { pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected \"<path> <gpa_base_hex> <file_offset_hex> <size_hex>\" or \"-\"\n"); return -EINVAL; } - tok = gsp_phys_guest_ram_token(&p); - if (!tok || kstrtoull(tok, 0, &file_offset)) { + if (kstrtoull(t[off + 2], 0, &file_offset)) { pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected a hex file offset after the GPA\n"); return -EINVAL; } - tok = gsp_phys_guest_ram_token(&p); - if (!tok || kstrtoull(tok, 0, &size)) { + if (kstrtoull(t[off + 3], 0, &size)) { pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected a hex size after the file offset\n"); return -EINVAL; } + if (off) { + /* + * Per-mdev form: the selector must name an OPEN mdev (the mdev + * UUID is its device name). Refusing an unknown selector keeps + * a typo from silently configuring the legacy record. + */ + struct nv_vgpu_device *mdev = + gsp_phys_open_mdev_by_name(t[0]); + + if (!mdev) { + pr_err("vgsp-boot: gsp-phys-guest-ram: no open mdev named %s; write refused\n", + t[0]); + return -ENODEV; + } + ret = gsp_phys_guest_ram_configure_vgpu(mdev, path, gpa_base, + file_offset, size); + if (ret) { + pr_err("vgsp-boot: gsp-phys-guest-ram: configure failed: mdev=%s path=%s gpa=0x%llx off=0x%llx size=0x%llx ret=%d\n", + t[0], path, gpa_base, file_offset, size, ret); + return ret; + } + pr_info("vgsp-boot: gsp-phys-guest-ram: configured mdev=%s path=%s gpa=0x%llx off=0x%llx size=0x%llx\n", + t[0], path, gpa_base, file_offset, size); + return len; + } + ret = gsp_phys_guest_ram_configure(path, gpa_base, file_offset, size); if (ret) { pr_err("vgsp-boot: gsp-phys-guest-ram: configure failed: path=%s gpa=0x%llx off=0x%llx size=0x%llx ret=%d\n", @@ -20001,6 +24222,30 @@ static int gsp_phys_reg_probe_show(struct seq_file *m, void *unused) sked_cwd1 = gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_1", 0x00407064u); + /* r425: the wedge diagnostics - the rest of the SKED CWD table, the + * per-8-group GR TPC masks (a zero compute mask would dispatch no + * TPCs), and the FE QMD-virtualization control for VEID 0. */ + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_2", + 0x00407068u); + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_CWD_TABLE_3", + 0x0040706cu); + (void)gsp_phys_reg_probe_line(m, bar0, "SKED_DEBUG_INFO_1", + 0x00407028u); + { + u32 gi; + + for (gi = 0; gi < 8u; gi++) { + gsp_phys_reg_probe_line(m, bar0, + "FE_TPCM_GR_TPC_MASK", + 0x0040a680u + gi * 4u); + gsp_phys_reg_probe_line(m, bar0, + "FE_LAST_GR_TPC_MASK", + 0x0040a6a0u + gi * 4u); + gsp_phys_reg_probe_line(m, bar0, + "FE_CWD_OWN_TPC_MASK", + 0x0040a6c0u + gi * 4u); + } + } fecs_irq = gsp_phys_reg_probe_line(m, bar0, "FECS_IRQSTAT", 0x00409008u); fecs_cpuctl = gsp_phys_reg_probe_line(m, bar0, "FECS_CPUCTL", @@ -20625,8 +24870,8 @@ static_assert(offsetof(struct gsp_phys_rmcall_channel_params, process_id) == 296 "NV_CHANNEL_ALLOC_PARAMS.ProcessID must be at offset 296"); static_assert(offsetof(struct gsp_phys_rmcall_channel_params, tpc_config_id) == 360, "NV_CHANNEL_ALLOC_PARAMS.tpcConfigID must be at offset 360"); -static_assert(GSP_PHYS_RMCALL_BUF_COUNT == 33, - "the rm-call ladder registers exactly thirty-three DMA buffers (14 ladder + 16 single-page guest scratch PTs + 3 golden9g VAS levels)"); +static_assert(GSP_PHYS_RMCALL_BUF_COUNT == 81, + "the rm-call ladder registers exactly eighty-one DMA buffers (14 ladder + 16 single-page guest scratch PTs + 48 single-page extension scratch PTs + 3 golden9g VAS levels)"); static_assert((GSP_PHYS_RMCALL_HANDLE_MEM_BASE + (GSP_PHYS_RMCALL_BUF_COUNT - 1) * GSP_PHYS_RMCALL_HANDLE_MEM_STRIDE) < 0xcaf80000u, diff --git a/kernel/gsp_phys_booter.h b/kernel/gsp_phys_booter.h index 896a436..af74f7c 100644 --- a/kernel/gsp_phys_booter.h +++ b/kernel/gsp_phys_booter.h @@ -164,6 +164,16 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, void gsp_phys_backend_static_invalidate(void); /* + * Mode-35 KGR static-data snapshots (see gsp_phys_booter.c): the live GSP's + * replies to the real client's KGR query set, captured at backend boot and + * served byte-identically by the guest-side emulation. Returns 0 with + * *out_len set, -ENODATA when cmd was not captured for the current backend + * generation, -ENOSPC when the snapshot exceeds dst_len, -EAGAIN when the + * backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len); + +/* * Run one fn1+fn65 probe on a READY backend to fill the static snapshot * cache. Takes gsp_phys_rm_lock inside (via gsp_phys_rm_probe), so it must * run on a work item, never inline under a caller-held rm lock or on the mdev @@ -275,6 +285,17 @@ struct gsp_phys_backend_copy_result { u32 elapsed_ms; /* return-to-caller elapsed milliseconds */ int rc; /* duplicated function return code */ struct gsp_phys_fault_info fault; /* decoded engine fault, if any */ + /* + * r420b: the forwarded launch's QMD release0 was redirected to the + * module semaphore (gsp_phys_rmcall_gr_rel_mod=1, or an unresolved + * guest release page), so the ENGINE will never write the guest's own + * QMD release word. The caller must author it in software on every + * outcome (the same write the hold's success edge performs); without + * this a rel_mod=1 launch with no deferred release records is never + * completed on the guest word and the client hangs (the r414 saxpy + * f0 cur=5 stall). + */ + bool qmd_rel_redirected; }; int gsp_phys_backend_guest_ce_copy_ex(u64 src_hpa, u32 src_off, u64 dst_hpa, @@ -349,6 +370,25 @@ void gsp_phys_backend_relay_counters(u64 *requested, u64 *ok, u64 *verify_fail, #define GSP_PHYS_GR_QMD_WORD_NONE 0xffffffffu #define GSP_PHYS_GR_RAW_WORDS 256u +/* + * B-2 (large-operand data path): the grown, boot-fixed GR data-bounce region. + * One PD0 dual entry per 2 MB window (PD0 slots 80..111, immediately above + * the m152 extension scratch pool's 32..79) backed by one fresh 4 KB small-PT + * page each, every leaf PTE pointing at one of the 16384 pages (64 MiB) of a + * single data chunk. The whole map is planted once at stage 3, before any + * engine fetch, so the GMMU's no-invalidate 128 B fetch-line rule is + * satisfied by construction (no PTE is ever rewritten at runtime). The SVC + * assigns operand reloc VAs from VA0 upward; gsp_sema_service.c derives its + * reloc base and page cap from these constants so the two sides cannot drift + * (the pre-B-2 205-page window below the mail page stays allocated for the + * mail injection and diagnostics, it just no longer carries operand data when + * gsp_phys_gr_data_region=1, the default). + */ +#define GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 0x0A000000ULL +#define GSP_PHYS_RMCALL_GR_DATA_BOUNCE_SLOT0 80u +#define GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PT_PAGES 32u +#define GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES (32u * 512u) /* 16384 */ + struct gsp_phys_gr_page { u64 guest_va; /* guest VA (page aligned) */ u64 hpa; /* host PA (page aligned; 0 = unresolved) */ @@ -402,6 +442,9 @@ struct gsp_phys_gr_submit { u64 rel_hpa; /* guest-visible QMD release0 page, or 0 */ u32 rel_off; u32 rel_payload; + u64 rel_va; /* r378: device VA rel_hpa was resolved from, + * for write-target re-validation before the + * release word is zeroed/restored */ u64 sem_hpa; /* guest-visible channel release page, or 0 */ u32 sem_off; u32 sem_payload; @@ -414,6 +457,16 @@ struct gsp_phys_gr_submit { * whose boot submission proved out (rmcall_gr_exec_ok). */ bool gsp_phys_backend_gr_ready(void); +/* + * r420: true exactly while the GR submit path sleeps in a WEDGE-QUIET hold + * (a deliberate, bounded wait for the sticky wedge to clear). The in-flight + * forwarded request is NOT done during it, so the sema service's completion + * deadlines (the 12 s GR release hold and the 3 s per-launch consumption + * bound) pause while this returns true instead of authoring the launch's + * release word under the hold (the r419 mid-boot stale-data escape). + */ +bool gsp_phys_gr_quiet_hold_active(void); + int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, struct gsp_phys_backend_copy_result *out); diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c index c5419a5..74062dd 100644 --- a/kernel/gsp_phys_gpu.c +++ b/kernel/gsp_phys_gpu.c @@ -638,13 +638,20 @@ int gsp_phys_rmcall_gr_qmdvirt_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdvirt_value); /* - * Explicit-PCAS gate (default 1). The rebuilt launch carries the vendor's - * turingPushPcas(CU_PCAS_TYPE_COPY) pair (SEND_PCAS_A = QMD>>8 followed by - * SEND_SIGNALING_PCAS_B = INVALIDATE|SCHEDULE). The stock guest push does - * not emit these (it relies on the IQ2M window's implied copy PCAS); setting - * this to 0 emits the pure vendor window-only form for the A/B test. - */ -static int gsp_phys_rmcall_gr_pcas_param = 1; + * Explicit-PCAS gate (default 0 since r343 - the double-execution fix). + * The rebuilt launch used to carry the vendor's turingPushPcas + * (CU_PCAS_TYPE_COPY) pair (SEND_PCAS_A = QMD>>8 followed by + * SEND_SIGNALING_PCAS_B = INVALIDATE|SCHEDULE) ADDED to the inline-QMD run + * that the stock guest push already carries. The QMD's + * REQUIRE_SCHEDULING_PCAS bit is clear, so the inline run schedules by + * itself - the explicit pair was a SECOND scheduling edge and the same QMD + * was dispatched twice (launch_count: one atomicAdd +100 landed as +200; + * victory_reduce exactly 2x; scale/saxpy/memset2d doubled). The stock guest + * push has NO PCAS pair (captured push dumps); default 0 restores that + * single-edge form. 1 = re-add the pair (A/B debug only - with it on, the + * grid executes twice). + */ +static int gsp_phys_rmcall_gr_pcas_param; module_param_named(gsp_phys_rmcall_gr_pcas, gsp_phys_rmcall_gr_pcas_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_pcas, @@ -657,6 +664,29 @@ int gsp_phys_rmcall_gr_pcas_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_pcas_value); /* + * r434 CWD-slot-count path (gsp_phys_rmcall_gr_cwd_slot; default -1 = off). + * On Volta+ the SET_CWD_SLOT_COUNT method is gated by the falcon PRI bit + * NV_PGRAPH_PRI_SKED_DEBUG_1.ENABLE_CWD_SLOT_COUNT (the CUDA driver's + * turingLaunchSetCwdSlotCount: a masked PRI write via SET_MME_SHADOW_SCRATCH + * 0/1/2 + SET_FALCON04, then the method itself). When >= 0 the init preamble + * emits that PRI write (ENABLE_CWD_SLOT_COUNT=1) followed by + * SET_CWD_SLOT_COUNT with the given value, so the wedge's "grid truncated to + * ~4 CTAs" signature can be A/B'd against the CWD slot-count watermark. + * Read per launch; -1 restores the historical preamble byte-for-byte. + */ +static int gsp_phys_rmcall_gr_cwd_slot_param = -1; +module_param_named(gsp_phys_rmcall_gr_cwd_slot, + gsp_phys_rmcall_gr_cwd_slot_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_cwd_slot, + "r434: enable the CWD slot count PRI (SKED_DEBUG_1 bit11) and emit SET_CWD_SLOT_COUNT with this value in the GR init preamble (-1 = off, default)"); + +int gsp_phys_rmcall_gr_cwd_slot_value(void) +{ + return gsp_phys_rmcall_gr_cwd_slot_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_cwd_slot_value); + +/* * Two-step PCAS gate (default 0). With the explicit PCAS pair on, setting * this marks the QMD's REQUIRE_SCHEDULING_PCAS (w4 bit12) and emits the copy * PCAS (INVALIDATE only) followed by a schedule-only PCAS (SCHEDULE), the @@ -704,20 +734,30 @@ int gsp_phys_rmcall_gr_qmdwin_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdwin_value); /* - * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod; default 0). - * When set, a forwarded GR launch ignores the resolved guest release page and - * points QMD release0 at the module window-0 semaphore VA - * (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host VA. - * Discriminant for "engine completed but the shadow release write was lost" - * vs "engine never completed": with the module release, a completed QMD is - * always observable. Also makes the existing module-release fallback - * observable (it previously set no poll source, so it could never complete). - */ -static int gsp_phys_rmcall_gr_rel_mod_param; + * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod). + * + * DEFAULT 1 since r453. When set, a forwarded GR launch ignores the resolved + * guest release page and points QMD release0 at the module window-0 semaphore + * VA (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host + * VA; the GUEST-VISIBLE release word is authored by the host only on the + * completion edge - after the retire wait and after the bounce copy-out has + * shipped the engine's stores to the guest pages. Evidence (r453, the FNV + * change-detection fix on top): + * - with 0: the guest's release could be satisfied by the engine's own + * early write while the host copy-out was still pending, so a client's + * stream-ordered readback (streams2's c2, memcpy_roundtrip's memset + * phase, streams_events' chain) read PRE-launch content: 11/15 battery. + * - with 1: 13/15 battery (streams_events and memcpy_roundtrip flipped + * green), no stalls across the full battery, mail+saxpy clean. + * The r414 stall hole (a launch that never reached the hold edge left the + * guest word unauthored) is closed by the r420b per-outcome authoring path; + * one-line rollback: echo 0 > .../parameters/gsp_phys_rmcall_gr_rel_mod. + */ +static int gsp_phys_rmcall_gr_rel_mod_param = 1; module_param_named(gsp_phys_rmcall_gr_rel_mod, gsp_phys_rmcall_gr_rel_mod_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_rel_mod, - "Force the forwarded GR QMD release0 to the module semaphore (default 0; 1 = m64 engine-completion discriminant)"); + "Force the forwarded GR QMD release0 to the module semaphore; guest release authored only on the post-copy-out completion edge (default 1 since r453; 0 = historical engine-authored guest release)"); int gsp_phys_rmcall_gr_rel_mod_value(void) { @@ -949,6 +989,286 @@ int gsp_phys_rmcall_gr_skedrec_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_skedrec_value); /* + * m153 no-write/wrong-place launch retry (gsp_phys_rmcall_gr_nowrite_retry; + * default 1). After a forwarded launch reports completion, the output + * retirement wait measures how many resolved data pages the engine actually + * changed. Two mis-execution signatures are known to be intermittent per + * boot: the engine writes nothing at all, or it writes into a lower operand + * window while the highest (output) operand stays untouched (a stale + * constant-source state). Both re-execute correctly on a fresh ring entry + * often enough to be worth a bounded retry: refill the bounce window from + * the guest inputs, re-publish the same method block, re-issue the schedule + * and re-poll, then re-wait. Set to 0 to disable the retry and accept the + * first outcome unconditionally. + */ +static int gsp_phys_rmcall_gr_nowrite_retry_param = 1; +module_param_named(gsp_phys_rmcall_gr_nowrite_retry, + gsp_phys_rmcall_gr_nowrite_retry_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_nowrite_retry, + "Re-execute a forwarded GR launch whose retirement wait saw zero/wrong-place engine writes (default 1)"); + +int gsp_phys_rmcall_gr_nowrite_retry_value(void) +{ + return gsp_phys_rmcall_gr_nowrite_retry_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_nowrite_retry_value); + +/* + * F2 fast re-arm (gsp_phys_gr_retry_fast; default 1). r198: the first + * forwarded-launch timeout costs 2x5 s before the fresh-entry ladder runs, + * yet the m155 extended re-arm (lane-enable replay + GPFIFO/TSG schedule + + * doorbell) plus one fresh ring entry recovers most of these launches. With + * this on, the first timeout immediately runs that re-arm + a fresh entry + * and re-polls with a short window; only if that still fails does the + * historical timeout ladder continue. Set to 0 for today's exact behavior. + */ +static int gsp_phys_gr_retry_fast_param = 1; +module_param_named(gsp_phys_gr_retry_fast, + gsp_phys_gr_retry_fast_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_retry_fast, + "On the first GR launch timeout, run the extended re-arm + a fresh ring entry immediately and re-poll briefly (default 1; 0 = legacy ladder timing)"); + +int gsp_phys_gr_retry_fast_value(void) +{ + return gsp_phys_gr_retry_fast_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_retry_fast_value); + +/* + * F2 first-poll window (gsp_phys_gr_first_poll_ms; default 2500, clamped to + * [1000, 30000]). With gsp_phys_gr_retry_fast=1 the FIRST poll window of a + * forwarded GR launch (before any re-arm) uses this instead of the fixed 5 s, + * so the fast re-arm path above starts earlier; with retry_fast=0 the launch + * keeps today's exact behavior and this knob is not read. The later ladder + * windows keep their own constants. + */ +static unsigned int gsp_phys_gr_first_poll_ms_param = 2500u; +module_param_named(gsp_phys_gr_first_poll_ms, + gsp_phys_gr_first_poll_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_first_poll_ms, + "First GR launch poll window in ms, clamped [1000,30000], read when gsp_phys_gr_retry_fast=1 (default 2500)"); + +unsigned int gsp_phys_gr_first_poll_ms_value(void) +{ + unsigned int v = gsp_phys_gr_first_poll_ms_param; + + if (v < 1000u) + v = 1000u; + if (v > 30000u) + v = 30000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_first_poll_ms_value); + +/* + * M91 icache coherence (gsp_phys_gr_icache_inv; default 1). The m91 class is + * the transient SM instruction-fetch lottery: the first attempt of a + * mid-sequence launch faults on SM garbage ("Illegal Instruction Encoding" / + * "Out Of Range Register") while a retry of the identical QMD and identical + * shadow content succeeds. The window-0 shadow pool reuses code VAs across + * launches (two 8-page banks alternate by submit epoch), so the bytes at a + * program VA change between launches, and every captured guest QMD carries + * the QMDV02_03 cache-control block with INVALIDATE_INSTRUCTION_CACHE (word 5 + * bit 30) CLEAR - the field the CUDA driver arms only under its + * forceIcacheInvalidate knob. An SM fetch line cached for launch N can then + * execute the previous kernel's bytes at the same VA in launch N+2. With this + * on, the forwarded QMD sets that bit (only when the QMD carries the driver's + * standard cache-control signature, bits 26..29 of word 5, so an unknown QMD + * revision is never touched), forcing a refetch at dispatch. 0 restores the + * exact pre-change QMD for a one-flip A/B. + */ +static int gsp_phys_gr_icache_inv_param = 1; +module_param_named(gsp_phys_gr_icache_inv, gsp_phys_gr_icache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_icache_inv, + "Set QMDV02_03 INVALIDATE_INSTRUCTION_CACHE (word5 bit30) on forwarded GR launches whose QMD has the driver cache-control signature (default 1; 0 = pre-m91 QMD words)"); + +int gsp_phys_gr_icache_inv_value(void) +{ + return gsp_phys_gr_icache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_icache_inv_value); + +/* + * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar; default 0). The + * captured guest QMDs carry RELEASE_MEMBAR_TYPE=FE_NONE (QMDV00_06 word 11 + * bit 14 clear, w11=0x44010000), so when the front-end completes a grid it + * writes the QMD release semaphore without first flushing its pending + * system-memory stores. The post-dispatch store-loss signature (the engine + * processes the QMD, writes the release, the grid completes and ZERO data + * pages are host-visible) is exactly what an unflushed write-back L2 looks + * like from the host: the release is host-visible while the data sits in the + * GPU's L2. Forcing RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR makes the release + * ordering guarantee the stores are in memory before the semaphore write, so + * the host's retire scan (and the copy-out) can never observe a completion + * whose data has not landed. 0 keeps the guest's captured value (the + * historical behaviour) for a one-flip A/B; read per launch. + */ +static int gsp_phys_gr_rel_membar_param; +module_param_named(gsp_phys_gr_rel_membar, gsp_phys_gr_rel_membar_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_rel_membar, + "r433: force QMD RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR (word11 bit14) on forwarded GR launches (default 0 = guest's captured FE_NONE)"); + +int gsp_phys_gr_rel_membar_value(void) +{ + return gsp_phys_gr_rel_membar_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_rel_membar_value); + +/* + * m179 no-write retry budget (gsp_phys_gr_nowrite_retries; default 3, clamped + * to [0, 4]). The m153 outcome retry loop re-executes a data-bearing launch + * whose retirement wait saw zero engine writes; it shared the timeout ladder's + * attempt counter, so a launch that had already spent attempts on the ladder + * (fast re-arm + fresh ring entries) reached the accept path with NO + * re-execution at all - the r200/r201/r202 memcpy_roundtrip / multikernel + * class (a lost launch silently accepted, client output all wrong). This + * budget is dedicated to the no-write loop and independent of the ladder; it + * is read once per submission. 0 keeps the accept-without-retry behavior + * (still logged loudly). The clamp keeps the worst-case host publishes per + * launch (initial + fast re-arm + 2 fresh entries + budget) within the + * 8-entry GPFIFO ring. + */ +static int gsp_phys_gr_nowrite_retries_param = 3; +module_param_named(gsp_phys_gr_nowrite_retries, + gsp_phys_gr_nowrite_retries_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_nowrite_retries, + "Dedicated re-execution budget for a data-bearing GR launch that wrote nothing tracked, clamped [0,4] (default 3)"); + +/* + * STALLFREE: absolute per-launch wall-clock budget (gsp_phys_gr_wait_max_ms; + * default 60000, clamped [10000, 600000]; 0 disables the extra cap and the + * built-in ladder bounds stand alone). The GR submit ladder and its retirement + * (no-write) retries are individually bounded, but their SUM grows with every + * re-arm/re-publish: a launch whose engine ack never arrives could hold the + * single-inflight work item for tens of seconds while the guest client waits + * on its QMD completion word. When this budget is crossed the submit stops its + * ladder immediately (no further re-arm/publish/retry), logs + * 'vgsp-gr: GR-STALL-wait-max ... action=fail' and returns -ETIMEDOUT; the + * sema service's failure edge then authors the QMD completion release in + * software, so the client always completes. A green launch (~0.2-11 s end to + * end, including the fast-re-arm recovery) never approaches the default, so + * the default does not change green behavior. + */ +static unsigned int gsp_phys_gr_wait_max_ms_param = 60000u; +module_param_named(gsp_phys_gr_wait_max_ms, + gsp_phys_gr_wait_max_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wait_max_ms, + "Absolute per-launch GR submit budget in ms, clamped [10000,600000]; 0 = only the built-in per-ladder bounds (default 60000)"); + +unsigned int gsp_phys_gr_wait_max_ms_value(void) +{ + unsigned int v = gsp_phys_gr_wait_max_ms_param; + + if (!v) + return 0; + if (v < 10000u) + v = 10000u; + if (v > 600000u) + v = 600000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wait_max_ms_value); + +int gsp_phys_gr_nowrite_retries_value(void) +{ + int v = gsp_phys_gr_nowrite_retries_param; + + if (v < 0) + v = 0; + if (v > 4) + v = 4; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_nowrite_retries_value); + +/* + * r445 wedge hard re-arm (gsp_phys_gr_wedge_rearm; default 0 = off). When the + * outcome-retry's warm GR semaphore probe proves the engine wedged for the + * context (the proven-sticky post-dispatch no-store state), this knob makes + * the submit attempt ONE bounded recovery instead of breaking immediately: + * a TSG PREEMPT(bWait=1) forces the FECS to save/unload the channel's + * context, then the mode-35 lane-enable tail + GPFIFO schedules are re-issued + * (the m155 extended re-arm) and the semaphore probe is repeated. A pass + * means the lane revived for this launch and the normal retry ladder + * continues; a fail leaves the historical WEDGED break. Read once per wedge + * episode. Off by default for A/B. + */ +static int gsp_phys_gr_wedge_rearm_param; +module_param_named(gsp_phys_gr_wedge_rearm, gsp_phys_gr_wedge_rearm_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wedge_rearm, + "r445: on the proven engine-wedged sem probe, attempt a bounded preempt+resched hard re-arm before giving up (default 0=off)"); + +int gsp_phys_gr_wedge_rearm_value(void) +{ + return gsp_phys_gr_wedge_rearm_param ? 1 : 0; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_rearm_value); + +/* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default 0). + * The forwarded QMD's cache-control block (QMDV02_03 word 5) carries the + * per-cache invalidate bits; the captured guest QMDs always set bits 26..29 + * (texture header/sampler/data + shader DATA) and the M91 fix adds bit 30 + * (INSTRUCTION), but bit 31 (SHADER CONSTANT - where a kernel's launch + * parameters live) is never set. The forwarded launch's param page is + * rewritten in place between launches (the two-bank win0 pool), so a stale + * constant-cache line can serve the PREVIOUS launch's parameters - the + * kernel then computes with the previous launch's pointers/sizes and the + * current launch's tracked output pages stay untouched ("no stores"). + * Mode: 0 = off, 1 = set bit 31 only, 2 = force the full 0xff000000 + * invalidate set. Read at submission time, so it can be A/B'd at runtime. + */ +static int gsp_phys_gr_ccache_inv_param; +module_param_named(gsp_phys_gr_ccache_inv, gsp_phys_gr_ccache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_ccache_inv, + "r449: forwarded-QMD shader-constant-cache invalidate: 0=off, 1=bit31, 2=all invalidate bits (default 0)"); + +unsigned int gsp_phys_gr_ccache_inv_value(void) +{ + if (gsp_phys_gr_ccache_inv_param < 0) + return 0; + if (gsp_phys_gr_ccache_inv_param > 2) + return 2; + return (unsigned int)gsp_phys_gr_ccache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_ccache_inv_value); + +/* + * r410 wedge quiet window (gsp_phys_gr_wedge_quiet_ms; default 40000, clamped + * [0,120000]; 0 disables). The sticky per-lane no-execute wedge (100% of + * launches no-write while kept busy) is proven to CLEAR when the lane stays + * quiet for tens of seconds (r409: a ~37 s client gap healed a wedged run; + * r416: 58 consecutive no-write launches during a 126 s client run, first + * launches green again after the lane sat ~52 s idle; r417: a 20 s host hold + * did not heal, a 40 s hold did - "WEDGE-QUIET healed: engine wrote 1 + * page(s) after the 40000 ms hold"). Immediate retries and + * re-arms do not heal it (r409/r154). So when a zero-write outcome follows a + * previous accepted zero-write launch, the retry loop first holds this many ms + * with NO submissions to the engine, then re-arms (lane enable + schedules) + * and re-publishes once. The value doubles per further quiet retry. The + * quiet window is recovery-only: a launch that wrote anything never enters it. + */ +static unsigned int gsp_phys_gr_wedge_quiet_ms_param = 40000u; +module_param_named(gsp_phys_gr_wedge_quiet_ms, + gsp_phys_gr_wedge_quiet_ms_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_wedge_quiet_ms, + "Quiet-window ms before retrying a launch on a lane whose previous launch also no-wrote (0=disable, default 40000; r417: a 20 s hold did not heal, 40 s healed)"); + +unsigned int gsp_phys_gr_wedge_quiet_ms_value(void) +{ + unsigned int v = gsp_phys_gr_wedge_quiet_ms_param; + + if (v > 120000u) + v = 120000u; + return v; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_quiet_ms_value); + +/* * m69 window-0 shadow pool (gsp_phys_rmcall_gr_shadow_win0; default 0). When * set, the forwarded GR launch copies the QMD's program/param pages into the * window-0 shadow pool (module pages at VAs 0x108000..0x10f000, PTEs planted @@ -1050,6 +1370,28 @@ int gsp_phys_rmcall_gr_data_prewin_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_data_prewin_value); /* + * B-2: grown boot-fixed data-bounce region (gsp_phys_gr_data_region; default + * 1). When set, gsp_phys_rmcall_pdb_build plants the 16384-page (64 MiB) + * region at GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 (PD0 80..111) and the SVC + * relocates operand pages into it; the booter's copy-in/copy-out index the + * grown chunk. When 0, everything falls back to the legacy 205-page / + * 68-pages-per-pointer window below the mail page, for one-flip A/B + * trialing. Boot-fixed: it is read by the stage-3 plant, so set it at + * insmod, not at runtime. + */ +static int gsp_phys_gr_data_region_param = 1; +module_param_named(gsp_phys_gr_data_region, + gsp_phys_gr_data_region_param, int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_data_region, + "Use the grown 64 MiB data-bounce region for GR operands (default 1; 0 = legacy 205-page window)"); + +int gsp_phys_gr_data_region_value(void) +{ + return gsp_phys_gr_data_region_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_data_region_value); + +/* * m113: after planting the launch's data mappings, re-register the VAS page * directory (UNSET+SET) to force the server walker to re-snapshot the tree; * all evidence says the SM walker resolves only the commit-time tree, and @@ -1231,16 +1573,28 @@ EXPORT_SYMBOL_GPL(gsp_phys_gpu_fw_crc32); * of a shared file. A GPA is translated by reading the file page that holds * it (read_cache_page() pulls it into the page cache and pins it for the * read) and converting that page to a host PA with page_to_phys(). The - * ranges map guest GPA windows to file offsets; up to four windows of one - * file per module cycle. + * ranges map guest GPA windows to file offsets; up to four windows per file. + * + * Per-mdev records (dual-guest goal 1.0): a guest RAM backing file belongs + * to ONE guest session, so the record moves with the mdev. Slot 0 is the + * legacy record (vgpu == NULL): it is what the historical top-level + * gsp-phys-guest-ram debugfs write configures and it serves the first/only + * open mdev exactly as the pre-dual-guest code did (one guest: byte-identical + * behaviour). A per-mdev record is configured with the mdev UUID as an extra + * first token on the same debugfs file and serves only that mdev; the + * record is released when the mdev is removed (gsp_phys_guest_ram_vgpu_clear) + * or when the physical cycle ends (gsp_phys_guest_ram_clear). * * Locking: guest_ram_lock serialises configure/clear against each other and * against the translators; it is held across read_cache_page() (which may * sleep) so a clear cannot fput the file while a translation is using it. - * One file per module cycle: a second path is refused -EBUSY because a fresh - * cycle reloads the module; more ranges of the same file may be added. + * Within one record the historical rule is unchanged: the first configure + * opens the file, a different path is refused -EBUSY, more ranges of the + * same file may be added. */ #define GSP_PHYS_GUEST_RAM_RANGES 4 +/* Per-mdev guest-RAM records (a handful of guests; 4 is far beyond 1.0). */ +#define GSP_PHYS_GUEST_RAM_MDEVS 4 struct gsp_phys_guest_ram_range { u64 gpa_base; @@ -1248,35 +1602,121 @@ struct gsp_phys_guest_ram_range { u64 size; }; -static struct file *guest_ram_file; -static char guest_ram_path[128]; -static struct gsp_phys_guest_ram_range guest_ram_ranges[GSP_PHYS_GUEST_RAM_RANGES]; -static unsigned int guest_ram_ranges_n; +struct gsp_phys_guest_ram_rec { + struct nv_vgpu_device *vgpu; /* NULL = legacy (first/only open mdev) */ + struct file *file; + char path[128]; + struct gsp_phys_guest_ram_range ranges[GSP_PHYS_GUEST_RAM_RANGES]; + unsigned int ranges_n; +}; + +static struct gsp_phys_guest_ram_rec guest_ram_legacy; +static struct gsp_phys_guest_ram_rec guest_ram_mdevs[GSP_PHYS_GUEST_RAM_MDEVS]; static DEFINE_MUTEX(guest_ram_lock); +/* Caller holds guest_ram_lock. NULL when this vgpu has no bound record. */ +static struct gsp_phys_guest_ram_rec * +gsp_phys_guest_ram_rec_find_locked(const struct nv_vgpu_device *vgpu) +{ + unsigned int i; + + if (!vgpu) + return NULL; + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) + if (guest_ram_mdevs[i].vgpu == vgpu) + return &guest_ram_mdevs[i]; + return NULL; +} + +/* Caller holds guest_ram_lock. NULL when every per-mdev slot is taken. */ +static struct gsp_phys_guest_ram_rec * +gsp_phys_guest_ram_rec_alloc_locked(struct nv_vgpu_device *vgpu) +{ + unsigned int i; + + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) + if (!guest_ram_mdevs[i].vgpu) { + guest_ram_mdevs[i].vgpu = vgpu; + return &guest_ram_mdevs[i]; + } + return NULL; +} + /* - * Configure the guest RAM backing file and add (or replace, keyed by - * gpa_base) one GPA window. The first call opens the file read-only and - * fixes the path for this module cycle; later calls may only add ranges of - * that same file. The file must be a regular file and every range must fit - * inside its current size, so a translation can never index a page beyond - * EOF. Returns 0, -EBUSY for a different path, -ENOSPC with all four ranges - * in use, or the file/argument error. + * Number of mdevs whose VFIO device is currently open. Shared by the + * per-mdev debugfs selector (which mdev does this UUID name?) and the + * single-guest guard on the legacy test-only paths. */ -int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base, - u64 file_offset, u64 size) +int gsp_phys_open_mdev_count(void) +{ + struct nv_vgpu_device *d; + int n = 0; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) + n++; + } + mutex_unlock(&nv_vgpu_driver.lock); + return n; +} +EXPORT_SYMBOL_GPL(gsp_phys_open_mdev_count); + +/* + * Resolve an OPEN mdev by its device name (the mdev UUID, e.g. + * "b6acdc5e-..."). Returns the device or NULL. The caller must use the + * result only while that mdev's VFIO fd is known to be open (the same + * lifetime rule as the first-open resolution below): the driver list is + * walked under nv_vgpu_driver.lock, but the returned pointer is not + * reference-counted. + */ +struct nv_vgpu_device *gsp_phys_open_mdev_by_name(const char *name) +{ + struct nv_vgpu_device *d, *found = NULL; + + if (!name || !*name) + return NULL; + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0 && d->mdev && + !strcmp(dev_name(&d->mdev->dev), name)) { + found = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return found; +} +EXPORT_SYMBOL_GPL(gsp_phys_open_mdev_by_name); + +/* + * Shared configure body for one record: open the record's file (first call) + * and add (or replace, keyed by gpa_base) one GPA window. The first call + * opens the file read-only and fixes the path for the record's lifetime; + * later calls may only add ranges of that same file. The file must be a + * regular file and every range must fit inside its current size, so a + * translation can never index a page beyond EOF. Returns 0, -EBUSY for a + * different path, -ENOSPC with all four ranges in use, or the + * file/argument error. `who` is "" for the legacy record and + * " for mdev <uuid>" for a per-mdev record; the legacy strings stay + * byte-identical. + */ +static int gsp_phys_guest_ram_rec_configure(struct gsp_phys_guest_ram_rec *rec, + const char *path, u64 gpa_base, + u64 file_offset, u64 size, + const char *who) { struct gsp_phys_guest_ram_range *range; struct file *f = NULL; unsigned int i; int ret = 0; - if (!path || !*path) + if (!rec || !path || !*path) return -EINVAL; mutex_lock(&guest_ram_lock); - if (!guest_ram_file) { + if (!rec->file) { /* * O_LARGEFILE is required: the guest RAM file is multi-GiB and * generic_file_open() (shmem's open) refuses a file beyond @@ -1306,35 +1746,38 @@ int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base, ret = -ERANGE; goto out_fput; } - guest_ram_file = f; - strscpy(guest_ram_path, path, sizeof(guest_ram_path)); - } else if (strcmp(guest_ram_path, path)) { - pr_err("vgsp: guest RAM already configured from %s; one file per module cycle (reload to switch); refusing %s\n", - guest_ram_path, path); + rec->file = f; + strscpy(rec->path, path, sizeof(rec->path)); + } else if (strcmp(rec->path, path)) { + if (who[0]) + pr_err("vgsp: guest RAM already configured from %s; one file per guest session (reload or mdev remove to switch); refusing %s\n", + rec->path, path); + else + pr_err("vgsp: guest RAM already configured from %s; one file per module cycle (reload to switch); refusing %s\n", + rec->path, path); ret = -EBUSY; goto out; } - for (i = 0, range = NULL; i < guest_ram_ranges_n; i++) { - if (guest_ram_ranges[i].gpa_base == gpa_base) { - range = &guest_ram_ranges[i]; + for (i = 0, range = NULL; i < rec->ranges_n; i++) { + if (rec->ranges[i].gpa_base == gpa_base) { + range = &rec->ranges[i]; break; } } if (!range) { - if (guest_ram_ranges_n >= GSP_PHYS_GUEST_RAM_RANGES) { + if (rec->ranges_n >= GSP_PHYS_GUEST_RAM_RANGES) { ret = -ENOSPC; goto out; } - range = &guest_ram_ranges[guest_ram_ranges_n++]; + range = &rec->ranges[rec->ranges_n++]; range->gpa_base = gpa_base; } range->file_offset = file_offset; range->size = size; - pr_info("vgsp: guest RAM configured: path=%s gpa=0x%llx off=0x%llx size=0x%llx (ranges=%u)\n", - guest_ram_path, gpa_base, file_offset, size, - guest_ram_ranges_n); + pr_info("vgsp: guest RAM configured%s: path=%s gpa=0x%llx off=0x%llx size=0x%llx (ranges=%u)\n", + who, rec->path, gpa_base, file_offset, size, rec->ranges_n); out: mutex_unlock(&guest_ram_lock); @@ -1344,35 +1787,148 @@ out_fput: fput(f); goto out; } + +/* + * Configure the legacy guest RAM record (the historical top-level + * gsp-phys-guest-ram write). Serves the first/only open mdev exactly as + * before the per-mdev records existed. + */ +int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base, + u64 file_offset, u64 size) +{ + return gsp_phys_guest_ram_rec_configure(&guest_ram_legacy, path, + gpa_base, file_offset, size, + ""); +} EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_configure); /* - * Close the configured guest RAM file (if any) and drop every range. Called - * from the physical-GPU release path so module unload and a backend release - * end the file reference; a fresh cycle reconfigures from debugfs. + * Configure the guest RAM record bound to one mdev (per-mdev selector on the + * debugfs write). A record for this vgpu is created on first use and closed + * by gsp_phys_guest_ram_vgpu_clear() when the mdev is removed. A failed + * first configure leaves no bound record behind. + */ +int gsp_phys_guest_ram_configure_vgpu(struct nv_vgpu_device *vgpu, + const char *path, u64 gpa_base, + u64 file_offset, u64 size) +{ + struct gsp_phys_guest_ram_rec *rec; + char who[80]; + const char *name = (vgpu && vgpu->mdev) ? + dev_name(&vgpu->mdev->dev) : "?"; + bool fresh = false; + int ret; + + if (!vgpu) + return -EINVAL; + + mutex_lock(&guest_ram_lock); + rec = gsp_phys_guest_ram_rec_find_locked(vgpu); + if (!rec) { + rec = gsp_phys_guest_ram_rec_alloc_locked(vgpu); + if (!rec) { + mutex_unlock(&guest_ram_lock); + pr_err("vgsp: guest RAM: no free per-mdev record slot (max %u); refusing mdev %s\n", + GSP_PHYS_GUEST_RAM_MDEVS, name); + return -ENOSPC; + } + fresh = true; + } + mutex_unlock(&guest_ram_lock); + + scnprintf(who, sizeof(who), " for mdev %s", name); + ret = gsp_phys_guest_ram_rec_configure(rec, path, gpa_base, + file_offset, size, who); + + /* + * A fresh record whose first configure failed must not hold the slot + * (the vgpu pointer would otherwise be reserved with no file and the + * next attempt could not create a clean record). + */ + if (ret && fresh) { + mutex_lock(&guest_ram_lock); + if (rec->file) + ret = 0; /* raced with another configure; keep it */ + else + rec->vgpu = NULL; + mutex_unlock(&guest_ram_lock); + } + return ret; +} +EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_configure_vgpu); + +/* Lock held: close one record's file and drop its ranges. */ +static void gsp_phys_guest_ram_rec_close(struct gsp_phys_guest_ram_rec *rec, + const char *release_log) +{ + if (rec->file) { + pr_info("%s: path=%s ranges=%u\n", release_log, + rec->path, rec->ranges_n); + fput(rec->file); + rec->file = NULL; + } + rec->path[0] = '\0'; + memset(rec->ranges, 0, sizeof(rec->ranges)); + rec->ranges_n = 0; +} + +/* + * Close every configured guest RAM record (legacy and per-mdev) and drop + * every range. Called from the physical-GPU release path so module unload + * and a backend release end the file references; a fresh cycle reconfigures + * from debugfs. The legacy release log line is unchanged. */ void gsp_phys_guest_ram_clear(void) { + unsigned int i; + mutex_lock(&guest_ram_lock); - if (guest_ram_file) { - pr_info("vgsp: guest RAM released: path=%s ranges=%u\n", - guest_ram_path, guest_ram_ranges_n); - fput(guest_ram_file); - guest_ram_file = NULL; - } - guest_ram_path[0] = '\0'; - memset(guest_ram_ranges, 0, sizeof(guest_ram_ranges)); - guest_ram_ranges_n = 0; + gsp_phys_guest_ram_rec_close(&guest_ram_legacy, + "vgsp: guest RAM released"); + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) { + gsp_phys_guest_ram_rec_close(&guest_ram_mdevs[i], + "vgsp: guest RAM released for mdev"); + guest_ram_mdevs[i].vgpu = NULL; + } mutex_unlock(&guest_ram_lock); } EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_clear); +/* + * Release the guest RAM record bound to one mdev. Called from the mdev + * cleanup path so the file reference and the vgpu pointer never outlive the + * session (a removed mdev's translations must stop, and the record slot must + * free up for a restarted guest). + */ +void gsp_phys_guest_ram_vgpu_clear(struct nv_vgpu_device *vgpu) +{ + unsigned int i; + + if (!vgpu) + return; + mutex_lock(&guest_ram_lock); + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) { + if (guest_ram_mdevs[i].vgpu == vgpu) { + gsp_phys_guest_ram_rec_close(&guest_ram_mdevs[i], + "vgsp: guest RAM released for mdev"); + guest_ram_mdevs[i].vgpu = NULL; + } + } + mutex_unlock(&guest_ram_lock); +} +EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_vgpu_clear); + bool gsp_phys_guest_ram_active(void) { - bool active; + bool active = false; + unsigned int i; mutex_lock(&guest_ram_lock); - active = guest_ram_file != NULL; + if (guest_ram_legacy.file) + active = true; + for (i = 0; !active && i < GSP_PHYS_GUEST_RAM_MDEVS; i++) + if (guest_ram_mdevs[i].file) + active = true; mutex_unlock(&guest_ram_lock); return active; } @@ -1380,33 +1936,78 @@ EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_active); /* * Lock-safe one-line snapshot for the gsp-phys-state dump: active flag, - * path, range count and every range (gpa/offset/size). The caller supplies - * the buffer; nothing is written when the file is not configured beyond the - * active=0 marker. + * path, range count and every range (gpa/offset/size). With only the legacy + * record configured (the one-mdev campaign) the output is byte-identical to + * the pre-dual-guest format; a per-mdev record adds its mdev UUID so a + * dual-guest operator can tell the records apart. */ +static void gsp_phys_guest_ram_rec_state(char *buf, size_t len, + unsigned int *n, + const struct gsp_phys_guest_ram_rec *rec, + const char *label) +{ + unsigned int i; + + if (*n >= len) + return; + *n += scnprintf(buf + *n, len - *n, "%s path=%s ranges=%u", + label, rec->path, rec->ranges_n); + for (i = 0; i < rec->ranges_n && *n < len; i++) + *n += scnprintf(buf + *n, len - *n, + " gpa=0x%llx off=0x%llx size=0x%llx", + rec->ranges[i].gpa_base, + rec->ranges[i].file_offset, + rec->ranges[i].size); +} + void gsp_phys_guest_ram_state(char *buf, size_t len) { unsigned int i, n = 0; + bool any_mdev = false; if (!buf || !len) return; buf[0] = '\0'; mutex_lock(&guest_ram_lock); - if (!guest_ram_file) { - scnprintf(buf, len, "active=0"); + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) + if (guest_ram_mdevs[i].file) { + any_mdev = true; + break; + } + + if (!any_mdev) { + /* + * Legacy-only (or nothing): keep the historical shape - the + * campaign's parser (if any) sees the same text as before. + */ + if (!guest_ram_legacy.file) { + scnprintf(buf, len, "active=0"); + mutex_unlock(&guest_ram_lock); + return; + } + n += scnprintf(buf + n, len - n, "active=1"); + gsp_phys_guest_ram_rec_state(buf, len, &n, + &guest_ram_legacy, ""); mutex_unlock(&guest_ram_lock); return; } - n += scnprintf(buf + n, len - n, "active=1 path=%s ranges=%u", - guest_ram_path, guest_ram_ranges_n); - for (i = 0; i < guest_ram_ranges_n && n < len; i++) - n += scnprintf(buf + n, len - n, - " gpa=0x%llx off=0x%llx size=0x%llx", - guest_ram_ranges[i].gpa_base, - guest_ram_ranges[i].file_offset, - guest_ram_ranges[i].size); + n += scnprintf(buf + n, len - n, "active=1"); + if (guest_ram_legacy.file) + gsp_phys_guest_ram_rec_state(buf, len, &n, &guest_ram_legacy, + " legacy"); + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS && n < len; i++) { + char label[64]; + + if (!guest_ram_mdevs[i].file) + continue; + scnprintf(label, sizeof(label), " mdev=%s", + (guest_ram_mdevs[i].vgpu && guest_ram_mdevs[i].vgpu->mdev) ? + dev_name(&guest_ram_mdevs[i].vgpu->mdev->dev) : "?"); + gsp_phys_guest_ram_rec_state(buf, len, &n, + &guest_ram_mdevs[i], label); + } mutex_unlock(&guest_ram_lock); } EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_state); @@ -1427,12 +2028,20 @@ EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_state); * an unmapped hole or an MMIO frame can never be handed to the engine as * guest RAM. * - * Guards kept from the original path: the active-mdev walk + * Guards kept from the original path: the mdev walk * (nv_vgpu_driver.lock for the bounded list scan) gates every translation - - * the engine path must only run with a live guest session; no such session - * returns -ENODEV. The IOMMU-domain query remains as a documented fallback - * for a setup whose mdev has a real domain (it is inert with the - * emulated-IOMMU mdev used here). + * the engine path must only run with a live guest session; a vgpu given by + * the caller must itself be open, and no open session at all returns + * -ENODEV. The IOMMU-domain query remains as a documented fallback for a + * setup whose mdev has a real domain (it is inert with the emulated-IOMMU + * mdev used here). + * + * Per-mdev resolution (dual-guest goal 1.0): the caller names the owning + * vgpu (a captured forwarding request, a held release, ...). The vgpu's own + * record resolves the translation; the legacy record (configured by the + * historical top-level debugfs write) still serves the FIRST open mdev so + * single-guest scripts are unchanged. vgpu == NULL keeps the historical + * resolution: the first open mdev. * * Locking: guest_ram_lock is held across the range walk and the * read_cache_page() so a concurrent configure/clear cannot change the file @@ -1442,13 +2051,15 @@ EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_state); * own lifetime guarantees for an open fd. No guest or rm lock may be held by * the caller. */ -int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out) +int gsp_phys_gpa_to_hpa_vgpu(struct nv_vgpu_device *vgpu, u64 gpa, u64 *hpa_out) { - struct nv_vgpu_device *vgpu = NULL; struct nv_vgpu_device *d; + struct nv_vgpu_device *first = NULL; + struct gsp_phys_guest_ram_rec *rec = NULL; struct iommu_domain *dom; phys_addr_t p; unsigned int i; + bool vgpu_gone = false; if (!hpa_out) return -EINVAL; @@ -1456,19 +2067,28 @@ int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out) mutex_lock(&nv_vgpu_driver.lock); list_for_each_entry(d, &nv_vgpu_driver.devices, list) { if (atomic_read(&d->vfio_dev_open_count) > 0) { - vgpu = d; + first = d; break; } } + if (vgpu && atomic_read(&vgpu->vfio_dev_open_count) <= 0) + vgpu_gone = true; /* named session is gone: never retarget */ mutex_unlock(&nv_vgpu_driver.lock); + if (vgpu_gone) + return -ENODEV; + if (!vgpu) + vgpu = first; /* historical resolution: first open mdev */ if (!vgpu) return -ENODEV; mutex_lock(&guest_ram_lock); - if (guest_ram_file) { - for (i = 0; i < guest_ram_ranges_n; i++) { + rec = gsp_phys_guest_ram_rec_find_locked(vgpu); + if (!rec && vgpu == first) + rec = &guest_ram_legacy; /* legacy file: first/only mdev */ + if (rec && rec->file) { + for (i = 0; i < rec->ranges_n; i++) { const struct gsp_phys_guest_ram_range *r = - &guest_ram_ranges[i]; + &rec->ranges[i]; struct page *page; u64 file_off; @@ -1477,7 +2097,7 @@ int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out) continue; file_off = r->file_offset + (gpa - r->gpa_base); - page = read_cache_page(guest_ram_file->f_mapping, + page = read_cache_page(rec->file->f_mapping, file_off >> PAGE_SHIFT, NULL, NULL); mutex_unlock(&guest_ram_lock); @@ -1513,6 +2133,18 @@ int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out) *hpa_out = (u64)p; return 0; } +EXPORT_SYMBOL_GPL(gsp_phys_gpa_to_hpa_vgpu); + +/* + * Historical entry point: resolve through the first open mdev (and that + * mdev's own record, else the legacy record). Used by the single-guest + * boot-ladder guest-GPA pin (test-only; see gsp_phys_guest_gpa_write) and + * kept as the compatibility wrapper for callers without a request context. + */ +int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out) +{ + return gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, hpa_out); +} EXPORT_SYMBOL_GPL(gsp_phys_gpa_to_hpa); /* diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h index f93603c..36c2220 100644 --- a/kernel/gsp_phys_gpu.h +++ b/kernel/gsp_phys_gpu.h @@ -3,6 +3,8 @@ #include <linux/types.h> +struct nv_vgpu_device; + /* * Physical GPU ownership (real-GSP execution backend, bring-up rung 1). * Claim is disabled unless the gsp_phys_gpu module parameter is set. @@ -352,6 +354,10 @@ int gsp_phys_rmcall_gr_mail_patch_value(void); /* m106: pre-install the data PT window pool in stage 3. */ int gsp_phys_rmcall_gr_data_prewin_value(void); +/* B-2: use the grown 64 MiB boot-fixed data-bounce region (default 1; 0 = + * legacy 205-page window). Read by the stage-3 plant and the SVC fill. */ +int gsp_phys_gr_data_region_value(void); + /* m113: re-register the VAS page directory after planting data. */ int gsp_phys_rmcall_gr_recommit_value(void); @@ -365,6 +371,56 @@ unsigned int gsp_phys_rmcall_gr_lmem_tpc_size_value(void); /* m72: log SKED CWD/HWW register transitions during the launch poll. */ int gsp_phys_rmcall_gr_skedrec_value(void); +/* m153: re-execute a forwarded launch whose retirement wait saw zero or + * wrong-place engine writes (default 1 = enabled). */ +int gsp_phys_rmcall_gr_nowrite_retry_value(void); + +/* F2: on the first GR launch timeout, run the extended re-arm + a fresh ring + * entry immediately and re-poll briefly (default 1; 0 = legacy ladder). */ +int gsp_phys_gr_retry_fast_value(void); + +/* F2: first GR launch poll window in ms, clamped [1000,30000], read while + * gsp_phys_gr_retry_fast=1 (default 2500). */ +unsigned int gsp_phys_gr_first_poll_ms_value(void); + +/* M91: set QMDV02_03 INVALIDATE_INSTRUCTION_CACHE (word5 bit30) on forwarded + * GR launches whose QMD carries the driver cache-control signature + * (default 1; 0 = pre-m91 QMD words). */ +int gsp_phys_gr_icache_inv_value(void); + +/* r433: force QMD RELEASE_MEMBAR_TYPE=FE_SYSMEMBAR (QMDV00_06 word11 bit14) + * on forwarded GR launches so the engine flushes pending sysmem stores + * before writing the release semaphore (default 0 = guest's FE_NONE). */ +int gsp_phys_gr_rel_membar_value(void); + +/* r434: enable the CWD slot count PRI (SKED_DEBUG_1 bit11) and emit + * SET_CWD_SLOT_COUNT with this value in the GR init preamble + * (-1 = off, default). */ +int gsp_phys_rmcall_gr_cwd_slot_value(void); + +/* m179: dedicated re-execution budget for a data-bearing GR launch whose + * retirement wait saw zero engine writes, independent of the timeout ladder's + * attempt counter (default 3, clamped [0,4]). */ +int gsp_phys_gr_nowrite_retries_value(void); +int gsp_phys_gr_wedge_rearm_value(void); +unsigned int gsp_phys_gr_ccache_inv_value(void); + +/* STALLFREE: absolute per-launch GR submit wall-clock budget in ms, clamped + * [10000,600000]; 0 = only the built-in per-ladder bounds (default 60000). + * On expiry the submit fails out immediately with a GR-STALL-wait-max line + * and the sema service authors the QMD completion release in software. */ +unsigned int gsp_phys_gr_wait_max_ms_value(void); + +/* r410 wedge quiet window: when a launch's zero-write outcome follows a + * previous accepted zero-write launch (the sticky per-lane no-execute wedge), + * hold the retry for this many ms with NO submissions to the engine before + * re-publishing. The wedge is proven to clear when the lane stays quiet for + * tens of seconds (r409 ~37 s, r416 ~52 s); immediate retries keep it alive. + * Base value for the first quiet retry, doubled per further attempt, clamped + * [0,120000]; 0 disables the quiet path (retries run back-to-back as before). + * Default 20000. */ +unsigned int gsp_phys_gr_wedge_quiet_ms_value(void); + /* m73: use host-visible sysmem chunks for V7 instance/RAMFC (m62 shape). */ int gsp_phys_rmcall_gr_inst_sys_value(void); @@ -425,21 +481,50 @@ bool gsp_phys_rmcall_vas_shared_value(void); int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out); /* + * Per-mdev variant: translate the GPA for the named owning vgpu (a captured + * forwarding request, a held release). The vgpu's own guest-RAM record is + * authoritative; the legacy record still serves the FIRST open mdev so + * single-guest callers are unchanged. NULL keeps the historical resolution + * (first open mdev). A named vgpu whose session is closed returns -ENODEV + * rather than being retargeted at another guest. + */ +int gsp_phys_gpa_to_hpa_vgpu(struct nv_vgpu_device *vgpu, u64 gpa, + u64 *hpa_out); + +/* * Guest RAM backing-file configuration (debugfs gsp-phys-guest-ram). One - * shared file per module cycle, up to four GPA->file-offset ranges keyed by + * record per guest session, up to four GPA->file-offset ranges keyed by * gpa_base. The first configure opens the file; the same path adds/replaces * ranges, a different path is refused -EBUSY, and the ranges are dropped and - * the file closed by gsp_phys_guest_ram_clear() on physical-GPU release. The + * the file closed by gsp_phys_guest_ram_clear() on physical-GPU release (or + * gsp_phys_guest_ram_vgpu_clear() when one mdev is removed). The + * gsp_phys_guest_ram_configure() entry point configures the legacy record + * (the historical top-level debugfs write: serves the first/only open mdev); + * gsp_phys_guest_ram_configure_vgpu() binds the record to one mdev. The * state accessor is lock-safe and writes "active=0" or the active path, * range count and per-range gpa/off/size into the caller's buffer. */ int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base, u64 file_offset, u64 size); +int gsp_phys_guest_ram_configure_vgpu(struct nv_vgpu_device *vgpu, + const char *path, u64 gpa_base, + u64 file_offset, u64 size); void gsp_phys_guest_ram_clear(void); +void gsp_phys_guest_ram_vgpu_clear(struct nv_vgpu_device *vgpu); bool gsp_phys_guest_ram_active(void); void gsp_phys_guest_ram_state(char *buf, size_t len); /* + * Open-mdev helpers for the per-mdev debugfs selector and the single-guest + * guards: count the mdevs whose VFIO device is open, or resolve one open + * mdev by its device name (the mdev UUID). The returned device is not + * reference-counted; the caller must only use it while that mdev's fd is + * known to be open. + */ +int gsp_phys_open_mdev_count(void); +struct nv_vgpu_device *gsp_phys_open_mdev_by_name(const char *name); + +/* * Domain shape of the physical (claimed) device, as the P6a engine path * needs it: the module programs raw host PAs into the CE page tables, so the * engine may only run when the physical device's DMA addresses ARE host PAs. diff --git a/kernel/gsp_rm_emulation.c b/kernel/gsp_rm_emulation.c index 8318f56..90556f2 100644 --- a/kernel/gsp_rm_emulation.c +++ b/kernel/gsp_rm_emulation.c @@ -452,8 +452,26 @@ static u32 gsp_rm_build_record(struct gsp_rm_emu *rm, u8 *buf, u32 buf_size, /* * Write a status queue record (1+ elements) and advance the writePtr. */ -static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, - u32 sequence, u32 result, const void *data, u32 data_len) +/* + * m171: the event burst budget is a rate over a short time window, not a + * monotonic session counter. Without the time reset a flood left the counter + * pinned at the cap and silently dropped the completion post of a client + * that only started waiting later - a minimal cudaEventSynchronize repro sat + * forever while its registrations were valid and 'completion hint rc=-22' + * repeated. Channel-completion posts get headroom over hint re-posts: a hint + * can be retried a moment later, a completion wake-up cannot. + */ +#define GSPRM_EVENT_BURST_CRIT_MAX 48 +#define GSPRM_EVENT_BURST_WINDOW (HZ) +/* Strictly shorter than the service's 0.5 s hint cadence: an equal window + * races the boundary and every later hint skips forever (observed r181: + * 'completion hint rc=-2' forever after one success and the stream client + * still stalled - the late-waiter wake-up never fired again). */ +#define GSPRM_EVENT_HINT_DEDUP (HZ / 4) + +static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, + u32 sequence, u32 result, const void *data, + u32 data_len, bool critical) { struct gsp_rm_emu *rm = ctx->rm_emu; u8 *buf; @@ -489,13 +507,22 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, * newer record and the client aborts with "Bad sequence number". */ if (function == GSPRM_RPC_EVENT_POST_EVENT) { - if (rm->event_burst >= GSPRM_EVENT_BURST_MAX) { + u32 cap = critical ? GSPRM_EVENT_BURST_CRIT_MAX + : GSPRM_EVENT_BURST_MAX; + + /* m171: expire the budget by wall time so a long-ago + * flood can never starve a fresh completion post. */ + if (time_after(jiffies, rm->event_burst_at + + GSPRM_EVENT_BURST_WINDOW)) + rm->event_burst = 0; + rm->event_burst_at = jiffies; + if (rm->event_burst >= cap) { rm->events_dropped++; if (rm->events_dropped <= 8 || (rm->events_dropped % 1000) == 0) nv_vgpu_info(ctx->vgpu, - "vgsp-rm: event burst capped (%u), dropped total=%llu", - GSPRM_EVENT_BURST_MAX, + "vgsp-rm: event burst capped (%u/%u), dropped total=%llu", + rm->event_burst, cap, rm->events_dropped); mutex_unlock(&rm->stat_lock); kfree(buf); @@ -504,6 +531,7 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, rm->event_burst++; } else { rm->event_burst = 0; + rm->event_burst_at = jiffies; } record_size = gsp_rm_build_record(rm, buf, buf_size, function, @@ -524,6 +552,106 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, return -EOVERFLOW; } + /* + * fail4: free-space discipline. The guest's consumption cursor for + * the status queue (msgqRxHeader.readPtr, at stat+rxHdrOff under the + * queue's SWAP_RX layout - same location msgqTxGetFreeSpace polls) + * gates this write. Overwriting unread records mixes entries in the + * guest's RPC stream: under a completion burst the guest then runs + * garbage RM callbacks (rm_run_rc_callback+0x0) and dies. Events are + * wake-ups and drop freely; a reply waits briefly for the ring to + * drain and is dropped (never overwritten) if it stays full. + */ + { + u32 peer_read = 0; + bool have_peer = false; + int w; + + for (w = 0; w < 51; w++) { + u32 used, space, rxoff = 0; + u8 shdr[32]; + + /* + * r332 SWAP_RX cursor fix: the guest's consumption + * cursor for the STATUS queue lives in the COMMAND + * queue's RX header, not in the status queue's own + * rxHdrOff. Under MSGQ_FLAGS_SWAP_RX (both sides set + * it: flags=1 in both headers) the driver's + * msgqRxMarkConsumed writes its status readPtr to + * pOurRxHdr->readPtr, and the driver's pOurRxHdr is + * the COMMAND-queue buffer's RX header (msgq.c + * msgqRxLink swap branch: pReadOutgoing = + * &pOurRxHdr->readPtr with pOurRxHdr from + * msgqTxCreate's pBackingStore = pCommandQueue). The + * status queue's own +rxHdrOff is the mirror we + * publish: OUR command-queue read position + * (written below after every consumed command + * record). Reading that mirror here made the + * free-space math track our own writes: the ring was + * declared "full" with the guest mid-drain, replies + * were written into it anyway, unread records were + * overwritten, and the guest RM read a newer record + * ("Bad sequence number. Expected N got N+63") - the + * first-client cudaMalloc then fails with "device + * busy or unavailable" (the boot lottery) and every + * later RPC fails until a fresh boot. + */ + if (gsp_rm_read_buf(ctx, rm->cmd_off, shdr, sizeof(shdr))) + break; + memcpy(&rxoff, shdr + 24, 4); + if (!rxoff || rxoff >= 0x1000) + break; + if (gsp_rm_read_buf(ctx, rm->cmd_off + rxoff, &peer_read, 4) || + peer_read >= rm->stat_msgcount) + break; /* no usable cursor: blind write */ + have_peer = true; + used = (rm->stat_write + rm->stat_msgcount - + peer_read) % rm->stat_msgcount; + space = rm->stat_msgcount - used - 1u; + if (elem_count <= space) + break; + /* + * Only event wake-ups drop freely; a reply waits + * (bounded) and then proceeds - dropping replies + * breaks the guest's RM init handshake (observed: + * two fn=0x1 replies dropped at boot -> the device + * never comes up). + */ + if (function == GSPRM_RPC_EVENT_POST_EVENT || w == 50) + break; + mutex_unlock(&rm->stat_lock); + msleep(1); + mutex_lock(&rm->stat_lock); + } + if (have_peer) { + u32 used = (rm->stat_write + rm->stat_msgcount - + peer_read) % rm->stat_msgcount; + u32 space = rm->stat_msgcount - used - 1u; + + if (elem_count > space && + function == GSPRM_RPC_EVENT_POST_EVENT) { + rm->events_dropped++; + if (rm->events_dropped <= 8 || + (rm->events_dropped % 1000) == 0) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: status ring full (free=%u need=%u peer=%u write=%u mc=%u); event dropped, total=%llu", + space, elem_count, + peer_read, rm->stat_write, + rm->stat_msgcount, + rm->events_dropped); + kfree(buf); + mutex_unlock(&rm->stat_lock); + return -EAGAIN; + } + if (elem_count > space) + nv_vgpu_warn(ctx->vgpu, + "vgsp-rm: status ring full for reply fn=0x%x (free=%u need=%u peer=%u write=%u mc=%u); writing anyway after wait", + function, space, elem_count, + peer_read, rm->stat_write, + rm->stat_msgcount); + } + } + /* Write all elements to consecutive queue slots. Page-aware: the * queue buffer is non-contiguous, so translate per page. */ slot = rm->stat_write % rm->stat_msgcount; @@ -589,6 +717,14 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, return 0; } +/* Non-event sends: the historical entry point, budget untouched. */ +static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, + u32 sequence, u32 result, const void *data, u32 data_len) +{ + return gsp_rm_send_flags(ctx, function, sequence, result, data, + data_len, false); +} + /* * Post the GPU event that tells the guest's CPU-side RM a channel's * submitted work is complete. A real GPU raises an interrupt, the host GSP @@ -627,9 +763,10 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) memcpy(payload + 8, &rm->events[i].notify_index, 4); memcpy(payload + 12, &rm->events[i].data, 4); /* info16@16, status@20, eventDataSize@24, bNotifyList@28 = 0 */ - if (gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload))) + if (gsp_rm_send_flags(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, + payload, sizeof(payload), true)) continue; + rm->events[i].last_post = jiffies; posted++; } @@ -648,9 +785,11 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) memcpy(payload + 4, &rm->events[i].h_event, 4); memcpy(payload + 8, &rm->events[i].notify_index, 4); memcpy(payload + 12, &rm->events[i].data, 4); - if (gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload))) + if (gsp_rm_send_flags(ctx, GSPRM_RPC_EVENT_POST_EVENT, + 0, 0, payload, + sizeof(payload), true)) continue; + rm->events[i].last_post = jiffies; posted++; } } @@ -675,6 +814,7 @@ static int gsp_rm_post_event_idx(struct gsp_emu_context *ctx, u32 idx) { struct gsp_rm_emu *rm = ctx->rm_emu; u8 payload[32]; + int rc; if (idx >= rm->events_n || !rm->events[idx].valid) return -EINVAL; @@ -685,8 +825,11 @@ static int gsp_rm_post_event_idx(struct gsp_emu_context *ctx, u32 idx) memcpy(payload + 8, &rm->events[idx].notify_index, 4); memcpy(payload + 12, &rm->events[idx].data, 4); - return gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload)); + rc = gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, + payload, sizeof(payload)); + if (!rc) + rm->events[idx].last_post = jiffies; + return rc; } int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx) @@ -709,8 +852,14 @@ int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx) } for (i = 0; i < rm->events_n; i++) { - int r = gsp_rm_post_event_idx(ctx, i); + int r; + /* m171: retired slots are not errors - skipping them + * keeps the hint's rc meaningful (a valid table must not + * report EINVAL because the last allocated slot is dead). */ + if (!rm->events[i].valid) + continue; + r = gsp_rm_post_event_idx(ctx, i); if (r) last = r; else @@ -725,10 +874,53 @@ int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx) * guest allocates carry no source object, so a per-channel association is * not available in the ABI). Used by the service's completion hint so a * client that starts waiting after the event fired still gets a wake-up. + * + * m171: this is a HINT, and hints repeat every 0.5 s while the window is + * open. Re-posting an event that a real completion post already woke moments + * ago burns the burst budget and floods the status ring without adding a + * wake-up, so skip events posted within GSPRM_EVENT_HINT_DEDUP. A waiter + * that arrives after the edge still gets a re-post within half a second. */ int gsp_rm_post_all_events(struct nv_vgpu_device *vgpu) { - return gsp_rm_force_post(vgpu, 0xffffffffu); + struct gsp_emu_context *ctx = vgpu ? vgpu->gsp_comm_ctx : NULL; + struct gsp_rm_emu *rm; + u8 payload[32]; + u32 i; + int n = 0, skipped = 0; + + if (!ctx) + return -ENODEV; + rm = ctx->rm_emu; + if (!rm) + return -ENODEV; + + for (i = 0; i < rm->events_n; i++) { + if (!rm->events[i].valid) + continue; + if (time_before(jiffies, rm->events[i].last_post + + GSPRM_EVENT_HINT_DEDUP)) { + skipped++; + continue; + } + + memset(payload, 0, sizeof(payload)); + memcpy(payload + 0, &rm->events[i].h_client, 4); + memcpy(payload + 4, &rm->events[i].h_event, 4); + memcpy(payload + 8, &rm->events[i].notify_index, 4); + memcpy(payload + 12, &rm->events[i].data, 4); + if (gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, + payload, sizeof(payload))) + continue; + rm->events[i].last_post = jiffies; + n++; + } + + if (n) + return 0; + /* -EAGAIN: valid events existed but every one was dedup-skipped + * (normal during an active post burst); -ENOENT: nothing to post. */ + return skipped ? -EAGAIN : -ENOENT; } /* @@ -977,6 +1169,7 @@ static const char *gsp_rm_rpc_name(u32 function) case GSPRM_RPC_FUNC_GSP_RM_CONTROL: return "GSP_RM_CONTROL"; case GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER: return "UNLOADING_GUEST_DRIVER"; case GSPRM_RPC_FUNC_UPDATE_BAR_PDE: return "UPDATE_BAR_PDE"; + case GSPRM_RPC_FUNC_DUP_OBJECT: return "DUP_OBJECT"; default: return "UNKNOWN"; } } @@ -2264,6 +2457,7 @@ static void gsp_rm_handle_bus_get_info_v2(struct gsp_emu_context *ctx, #define GSPRM_COPY_PDES_MAX_LEVELS 6 static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, + u32 h_object, const u8 *params, u32 params_size) { struct gsp_rm_emu *rm = ctx->rm_emu; @@ -2314,6 +2508,46 @@ static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, "vgsp-rm: pt level[%u] phys=0x%llx size=0x%llx aperture=%u pageShift=%u", i, phys, size, aperture, page_shift); } + + /* + * r496j: this control IS the kernel RM publishing the page-table + * chain of its internal VAS (the level[0..] instances with their + * phys addresses and page shifts). The chain can name the X/GL + * channel's pushbuffer pages and progress semaphore, for which no + * other translation source exists (the fn-54 SET_PAGE_DIRECTORY + * path is external-VAS/UVM only). It is NOT registered directly: + * a blast registration of every client's directory let a stale + * kernel directory answer another client's VA in the walker's + * first-success order (the r459 stale-directory class; observed + * as a CUDA client hanging after DEVICE with r496g). Instead the + * root is offered as a CANDIDATE; the progress service registers + * it only after it has been VERIFIED against a content-validated + * pair of that very client (the ring found by the RAM content + * scan). + */ + if (num_levels >= 1 && rm->pt.levels[0].phys && + rm->pt.levels[0].size >= 16 && + rm->pt.levels[0].size <= 4096 * 8) { + /* + * The level phys values are guest PAs on this platform + * (observed 4+ GiB, above any FB shadow); aperture 3 in the + * message does not match the walker's FB/SYS encoding, so + * classify by address: values inside the emulated VRAM are + * FB, everything else sysmem. The walker also retries the + * other aperture, so a misclassification only costs a walk. + */ + bool vidmem = rm->pt.levels[0].phys < + (u64)ctx->vgpu->bars[1].size; + + gsp_sema_service_vaspace_candidate_add( + h_object, rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), vidmem); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: VAS 0x%x page directory offered as candidate pdb=0x%llx entries=%u (%s; registered only after a content-verified ring walk)", + h_object, (u64)rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), + vidmem ? "vidmem" : "sysmem"); + } } static bool gsp_rm_alloc_interesting(u32 class_id); @@ -2357,10 +2591,24 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, return; memset(&desc, 0, sizeof(desc)); + desc.vgpu = ctx->vgpu; desc.h_object = h_object; memcpy(&desc.gpfifo_va, params + GSPRM_CHAN_GPFIFO_OFFSET, 8); memcpy(&desc.gpfifo_entries, params + GSPRM_CHAN_GPFIFO_ENTRIES, 4); memcpy(&desc.engine_type, params + GSPRM_CHAN_ENGINE_TYPE, 4); + /* + * The wire carries NV2080_ENGINE_TYPE_*: NULL = 0, GR0 (== GRAPHICS) + * = 1, COPY0 = 9, NVDEC0 = 0x13, ... The progress service's + * convention is "engine_type == 0 means the GR channel" (CUDA's + * kernel-RM GR channels arrive with engineType NULL), so an X/GL 3D + * channel - which carries the real NV2080_ENGINE_TYPE_GRAPHICS = 1 - + * was misclassified as a CE and its push never executed ("Failed to + * initialize DMA" in the X driver's pushbuffer NOP test). Normalize + * GR0..GR7 (1..8) to 0: the emulated device has a single GR and every + * other engine id is >= COPY0 (9), so this cannot capture a CE. + */ + if (desc.engine_type >= 1 && desc.engine_type <= 8) + desc.engine_type = 0; memcpy(&desc.cid, params + GSPRM_CHAN_CID, 4); memcpy(&desc.instance_pa, params + GSPRM_CHAN_INSTANCE_MEM, 8); memcpy(&desc.userd_pa, params + GSPRM_CHAN_USERD_MEM, 8); @@ -2443,51 +2691,80 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, * notifyIndex, data}). The driver allocates one per channel and waits * for the GPU to post the event before it considers submitted work * complete - remember them so completed work can be signalled back. + * + * The table is append+retire: FREE retires records (valid=false) but + * never shrinks events_n, so long-lived boots with many client + * create/destroy cycles would otherwise exhaust the 128 slots and + * SILENTLY drop new registrations - after which the completion + * wake-ups (post_channel_event / post_all_events) fail with EINVAL + * and a later client waits forever for a completion that can no + * longer be notified (observed r162: `completion hint rc=-22` + * repeating from the first post-teardown client onward). Reclaim the + * first retired slot for a new registration, and log loudly when the + * table is genuinely full of live entries. */ - if (class_id == 0x79 && params_size >= 24 && rm->events_n < GSPRM_MAX_EVENTS) { + if (class_id == 0x79 && params_size >= 24) { u32 h_src = 0, notify_index = 0, hclass = 0; u64 data64 = 0; int hc_off = -1, i; + u32 slot = rm->events_n; - /* - * Locate hClass (0x79) inside the params instead of trusting a - * fixed offset: NV0005_ALLOC_PARAMETERS is - * {hParentClient, hSrcResource, hClass, notifyIndex, data}, - * with hSrcResource immediately before hClass. - */ - for (i = 0; i + 4 <= (int)params_size; i += 4) { - u32 v; - - memcpy(&v, params + i, 4); - if (v == 0x79) { - hc_off = i; + for (i = 0; i < (int)rm->events_n; i++) { + if (!rm->events[i].valid) { + slot = (u32)i; break; } } - if (hc_off >= 4) { - memcpy(&h_src, params + hc_off - 4, 4); - memcpy(&hclass, params + hc_off, 4); - if (hc_off + 8 <= (int)params_size) - memcpy(¬ify_index, params + hc_off + 4, 4); - if (hc_off + 16 <= (int)params_size) - memcpy(&data64, params + hc_off + 8, 8); - } else { - memcpy(&h_src, params + 4, 4); - memcpy(¬ify_index, params + 12, 4); - memcpy(&data64, params + 16, 8); + if (slot >= GSPRM_MAX_EVENTS) + nv_vgpu_err(ctx->vgpu, + "vgsp-rm: event table full (%u live records); dropping registration hObject=0x%x - completion wake-ups will not reach this client", + rm->events_n, ho); + else { + /* + * Locate hClass (0x79) inside the params instead of + * trusting a fixed offset: NV0005_ALLOC_PARAMETERS is + * {hParentClient, hSrcResource, hClass, notifyIndex, + * data}, with hSrcResource immediately before hClass. + */ + for (i = 0; i + 4 <= (int)params_size; i += 4) { + u32 v; + + memcpy(&v, params + i, 4); + if (v == 0x79) { + hc_off = i; + break; + } + } + if (hc_off >= 4) { + memcpy(&h_src, params + hc_off - 4, 4); + memcpy(&hclass, params + hc_off, 4); + if (hc_off + 8 <= (int)params_size) + memcpy(¬ify_index, + params + hc_off + 4, 4); + if (hc_off + 16 <= (int)params_size) + memcpy(&data64, + params + hc_off + 8, 8); + } else { + memcpy(&h_src, params + 4, 4); + memcpy(¬ify_index, params + 12, 4); + memcpy(&data64, params + 16, 8); + } + rm->events[slot].h_src = h_src; + rm->events[slot].h_event = ho; + rm->events[slot].h_client = hc; + rm->events[slot].notify_index = notify_index; + rm->events[slot].data = (u32)data64; + rm->events[slot].valid = true; + if (slot == rm->events_n) + rm->events_n++; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: event registered hEvent=0x%x hSrc=0x%x notifyIndex=%u (off %d, slot %u) params=%*phN", + ho, h_src, notify_index, hc_off, + slot, + (int)min(params_size, (u32)24), + params); } - rm->events[rm->events_n].h_src = h_src; - rm->events[rm->events_n].h_event = ho; - rm->events[rm->events_n].h_client = hc; - rm->events[rm->events_n].notify_index = notify_index; - rm->events[rm->events_n].data = (u32)data64; - rm->events[rm->events_n].valid = true; - rm->events_n++; - if (gsp_sema_dbg) - nv_vgpu_info(ctx->vgpu, - "vgsp-rm: event registered hEvent=0x%x hSrc=0x%x notifyIndex=%u (off %d) params=%*phN", - ho, h_src, notify_index, hc_off, - (int)min(params_size, (u32)24), params); } /* Echo the fixed alloc header + as many param bytes as are contiguous @@ -2679,6 +2956,115 @@ gsp_rm_handle_promote_fault_method_buffers(struct gsp_emu_context *ctx, } /* + * r519 experiment switch, independent of the P3 whitelist: forward ONLY the + * BINAPI reserved-interface query 0x20810107 to the live GSP session. The P3 + * whitelist may be off (its mixed real/emulated device answers perturb the + * boot); this one command is what the GL surface init needs answered. + */ +static bool gsp_rm_binapi07_fwd_param; +module_param_named(gsp_rm_binapi07_fwd, gsp_rm_binapi07_fwd_param, bool, 0644); +MODULE_PARM_DESC(gsp_rm_binapi07_fwd, + "Forward only the guest GL core's BINAPI 0x20810107 query to the live GSP session (default 0; diagnostic/experiment)"); + +/* + * The live session's oracle client/object for the 0x20810107 experiment, + * created on demand; 0 until the first query builds it. One device per module + * load, so file-statics are enough; reset by unload. + */ +static u32 gsp_rm_binapi07_be_client; +static u32 gsp_rm_binapi07_be_object; + +/* + * One GSP_RM_ALLOC (fn 103) against the live backend, built exactly like the + * rm-call ladder's helper (rpc_gsp_rm_alloc_v03_00: hClient, hParent, + * hObject, hClass, status, paramsSize, flags, reserved[4] + params). + * object_is_out selects the reply read: a ROOT alloc returns the + * server-chosen client handle in params[0] (payload +32), anything else + * echoes the client-chosen hObject at +8; status is at +16 in both cases. + */ +static int gsp_rm_binapi07_backend_alloc(u32 h_client, u32 h_parent, + u32 h_object, u32 h_class, + const void *params, u32 params_len, + bool object_is_out, + u32 *out_object, u32 *out_status) +{ + u8 *areq; + void *arsp = NULL; + u32 arsp_len = 0, st = 0xffffffff, ho = 0; + int ret; + + areq = kzalloc(32 + params_len, GFP_KERNEL); + if (!areq) + return -ENOMEM; + memcpy(areq + 0, &h_client, 4); + memcpy(areq + 4, &h_parent, 4); + memcpy(areq + 8, &h_object, 4); + memcpy(areq + 12, &h_class, 4); + memcpy(areq + 20, ¶ms_len, 4); + if (params_len) + memcpy(areq + 32, params, params_len); + + ret = gsp_phys_backend_rpc(GSPRM_RPC_FUNC_GSP_RM_ALLOC, areq, + 32 + params_len, &arsp, &arsp_len, 500); + kfree(areq); + if (!ret && arsp) { + if (arsp_len >= 20) + memcpy(&st, (u8 *)arsp + 16, 4); + if (object_is_out) { + if (arsp_len >= 36) + memcpy(&ho, (u8 *)arsp + 32, 4); + } else if (arsp_len >= 12) { + memcpy(&ho, (u8 *)arsp + 8, 4); + } + } + kfree(arsp); + if (out_object) + *out_object = ho; + if (out_status) + *out_status = st; + return ret; +} + +/* + * Build the oracle object on the live session: a class-0x2081 child of the + * session's internal subdevice, since the server dispatches the method from + * the OBJECT's class (the subdevice attempt alone answered + * NV_ERR_NOT_SUPPORTED, 0x56, and a fresh root client at runtime answered + * NV_ERR_INVALID_CLIENT, 0x23 - the ladder "adopts" the server's own + * internal client instead of creating one). A small candidate list of + * likely-free handles in that client's namespace is tried; the first + * success wins. Returns 0 with *out_object set. + */ +static int gsp_rm_binapi07_backend_setup(struct gsp_emu_context *ctx, + struct gsp_phys_backend_static_config *snap, + u32 *out_object) +{ + static const u32 cand[] = { 0xabcd2081u, 0xabcd2001u, 0xabcd2a2au, + 0xabcd5f5fu, 0xabcd1234u }; + u8 p4[4] = { 0, 0, 0, 0 }; + u32 st = 0xffffffff, obj = 0; + int i, ret; + + for (i = 0; i < (int)ARRAY_SIZE(cand); i++) { + u32 want = cand[i]; + + obj = want; + ret = gsp_rm_binapi07_backend_alloc(snap->h_internal_client, + snap->h_internal_subdevice, + want, 0x2081, p4, 4, + false, &obj, &st); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT oracle BINAPI alloc attempt %d: hObject=0x%x ret=%d status=0x%x", + i, want, ret, st); + if (!ret && st == 0 && obj) { + *out_object = obj; + return 0; + } + } + return -EIO; +} + +/* * Bounded logging for the P3 forward counters: the first four fallbacks (so a * session that cannot forward is visible) and every 64th event. Never called * when the parameter is off. @@ -2727,12 +3113,37 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, u32 out_params_size; int ret; + bool guest_handles = false; + /* Wave 1 whitelist: compile-time side-effect-free queries only. */ switch (req->cmd) { case TU104_CTRL_BUS_GET_PCI_BAR_INFO: case TU104_CTRL_BUS_GET_INFO_V2: case TU104_CTRL_FB_GET_INFO_V2: case TU104_CTRL_INTERNAL_GMMU_GET_STATIC_INFO: + if (!gsp_phys_ctrl_forward_enabled()) + return false; + break; + case 0x20810107: /* NV2081_BINAPI reserved-interface method 7 */ + /* + * EXPERIMENT (r519): the guest GL core allocates a BINAPI + * object (class 0x2081) and sends this 2-byte query during + * 3D-surface init; our {0,0} generic reply leaves the + * driver's feature flag clear and the EGL surface init tears + * down (EGL_BAD_ALLOC at eglMakeCurrent). The guest-side + * handles (hClient 0xc1d..., a 0xbeef object) have no backend + * counterpart: allocate a real BINAPI object (class 0x2081) on + * the live session once, then send the query to it. The first + * attempt (subdevice translation) came back NV_ERR_NOT_SUPPORTED + * (0x56) - the server dispatches the method by the OBJECT's + * class, so a subdevice cannot answer a BINAPI method. The + * reply is logged either way; an error falls back to the + * emulated generic success until the semantics are known. + * Gated by its own parameter so the P3 whitelist can stay off. + */ + if (!gsp_rm_binapi07_fwd_param) + return false; + guest_handles = true; break; default: return false; @@ -2744,24 +3155,44 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, if (req->paramsSize > GSPRM_MAX_SINGLE_ELEM_DATA - GSP_RM_CTRL_HDR_SIZE) goto fallback; + /* The experiment is scoped to the observed shape: 2-byte in/out. */ + if (guest_handles && req->paramsSize != 2) + goto fallback; + /* A valid snapshot implies a READY session and the adopted handles. */ if (gsp_phys_backend_static_get(&snap)) goto fallback; + if (guest_handles && !gsp_rm_binapi07_be_object) { + u32 obj = 0; + + if (gsp_rm_binapi07_backend_setup(ctx, &snap, &obj)) + goto fallback; + /* + * The object lives in the internal client's namespace; the + * control is sent as that client (snap.h_internal_client). + */ + gsp_rm_binapi07_be_client = snap.h_internal_client; + gsp_rm_binapi07_be_object = obj; + } + /* Translate the guest's emulated internal handles; any other handle * domain stays emulated. */ - if (req->hClient != TU104_H_INTERNAL_CLIENT || - (req->hObject != TU104_H_INTERNAL_DEVICE && - req->hObject != TU104_H_INTERNAL_SUBDEVICE)) + if (!guest_handles && + (req->hClient != TU104_H_INTERNAL_CLIENT || + (req->hObject != TU104_H_INTERNAL_DEVICE && + req->hObject != TU104_H_INTERNAL_SUBDEVICE))) goto fallback; fwd_req = kmalloc(GSP_RM_CTRL_HDR_SIZE + req->paramsSize, GFP_KERNEL); if (!fwd_req) goto fallback; memset(&fwd_hdr, 0, sizeof(fwd_hdr)); - fwd_hdr.hClient = snap.h_internal_client; - fwd_hdr.hObject = req->hObject == TU104_H_INTERNAL_DEVICE ? - snap.h_internal_device : snap.h_internal_subdevice; + fwd_hdr.hClient = guest_handles ? gsp_rm_binapi07_be_client : + snap.h_internal_client; + fwd_hdr.hObject = guest_handles ? gsp_rm_binapi07_be_object : + (req->hObject == TU104_H_INTERNAL_DEVICE ? + snap.h_internal_device : snap.h_internal_subdevice); fwd_hdr.cmd = req->cmd; fwd_hdr.paramsSize = req->paramsSize; fwd_hdr.flags = req->flags; @@ -2781,6 +3212,27 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, } /* + * r519 experiment evidence: log the live GSP's own answer to the + * BINAPI query (status + first params bytes) before deciding. + */ + if (req->cmd == 0x20810107) { + u32 st = 0xffffffff, ps = 0xffffffff; + u8 b[8] = { 0 }; + + if (rsp && rsp_len >= GSP_RM_CTRL_HDR_SIZE) { + memcpy(&st, (u8 *)rsp + 12, 4); + memcpy(&ps, (u8 *)rsp + 16, 4); + if (ps <= 8 && ps <= rsp_len - GSP_RM_CTRL_HDR_SIZE) + memcpy(b, (u8 *)rsp + GSP_RM_CTRL_HDR_SIZE, + ps); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT 0x20810107 forwarded to live GSP: rsp_len=%u status=0x%x outSize=%u bytes=%02x %02x %02x %02x %02x %02x %02x %02x", + rsp_len, st, ps, b[0], b[1], b[2], b[3], b[4], + b[5], b[6], b[7]); + } + + /* * The reply is rpc_gsp_rm_control_v03_00: status at +12, paramsSize * at +16, echoed params at +24 - the same 24-byte prefix the emulated * handler builds below. The RPC-level rpc_result was already checked @@ -2826,6 +3278,101 @@ fallback: return false; } +/* + * r489: a GSP_RM_CONTROL reply whose rpc record exceeds the driver's + * pRpc->maxRpcSize must be sent with the GSP's continuation-record protocol + * (nv gsp_rpc.c _rpcWriteContinuationEntries): the FIRST record carries + * exactly maxRpcSize bytes of [rpc header + ctrl header + params]; each + * following NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD record carries up to + * maxRpcSize - sizeof(hdr) bytes of the remainder. The driver's + * _issueRpcLarge consumes the first frame as min(bufSize, maxRpcSize) and + * then requires one continuation record per remaining chunk - without them + * it times out ("rpcRecvPoll timedout for fn 71 continuation record") and + * the control fails (NV_ERR_INVALID_PARAMETER), which is how the GL VASpace + * construction died on cmd 0x2080a0a4 (paramsSize 67392 > maxRpcSize). + * + * maxRpcSize is the guest driver's GSP_MSG_QUEUE_RPC_SIZE_MAX; the value is + * pinned by the driver's own arithmetic: for the 0x2080a0a4 reply + * (full_param_size = 32 + 24 + 67392 = 67448) it reported remainingSize + * 0x940 = 2368 after the first frame, i.e. maxRpcSize = 67448 - 2368 = + * 65080. A build whose constant differs shows up immediately as a + * mismatched continuation count in the guest log; adjust here. + */ +#define GSPRM_CRM_CONT_MAXRPC 65080u + +static int gsp_rm_send_rm_control(struct gsp_emu_context *ctx, u32 sequence, + const void *data, u32 data_len) +{ + u32 total = sizeof(struct gsprm_rpc_header) + data_len; + const u8 *p = data; + u32 off = 0; + int ret; + + if (total <= GSPRM_CRM_CONT_MAXRPC) + return gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, + sequence, 0, data, data_len); + + /* Frame 1: rpc.length = maxRpcSize. */ + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, + p, GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)); + if (ret) + return ret; + off = GSPRM_CRM_CONT_MAXRPC - sizeof(struct gsprm_rpc_header); + + /* Continuations: rpc.length = hdr + chunk. */ + while (off < data_len) { + u32 chunk = data_len - off; + + if (chunk > GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)) + chunk = GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header); + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_CONTINUATION_RECORD, + sequence, 0, p + off, chunk); + if (ret) + return ret; + off += chunk; + } + return 0; +} + +/* + * Real-GSP KGR static-data snapshots (mode-35 boot capture; see + * gsp_phys_kgr_snap_get in gsp_phys_booter.c). The physical backend asks the + * live GSP the real client's KGR query set at boot and retains the replies; + * serve the identical bytes for the same commands so the guest's GL/3D stack + * reads the device's true GR static values. Concrete consumer: the X + * driver's 3D-engine selection maps GR_GET_INFO index SM_VERSION (0x0c) + * through a table and fails with "Failed to select a 3D engine" when it + * reads back as SM_VERSION_NONE (0); the live GSP's KGR_GET_INFO reply + * carries the real value (0x75 on TU104). Returns true when the reply was + * served from the snapshot (out_params_size set); false falls through to the + * emulated handler unchanged. + */ +static bool gsp_rm_kgr_snap_serve(struct gsp_emu_context *ctx, u32 cmd, + u32 req_params_size, + u8 *out_params, u32 out_len, + u32 *out_params_size) +{ + u32 n = 0; + int ret; + + ret = gsp_phys_kgr_snap_get(cmd, out_params, out_len, &n); + if (ret || !n) + return false; + + /* Never answer with more bytes than the guest asked for. */ + if (n > req_params_size) + return false; + + *out_params_size = n; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: KGR cmd=0x%x served from the real-GSP snapshot (%u bytes)", + cmd, n); + return true; +} + static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, const u8 *elem, u32 sequence) { @@ -2860,13 +3407,26 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, * return means nothing was sent and the emulated switch below runs * byte-identically to the parameter-off path. */ - if (gsp_phys_ctrl_forward_enabled() && + if ((gsp_phys_ctrl_forward_enabled() || gsp_rm_binapi07_fwd_param) && gsp_rm_ctrl_forward(ctx, &req, req_data + GSP_RM_CTRL_HDR_SIZE, reply, reply_data_len, sequence)) { kfree(reply); return; } + /* + * KGR static-data snapshots (mode-35 boot capture): the live GSP's own + * replies to the real client's KGR query set are served byte-identically + * for the same commands, so the guest's GL/3D stack sees the device's + * true GR static values. Falls through to the emulated switch when no + * snapshot exists (backend not READY, other boot mode, cmd not captured). + */ + if (gsp_rm_kgr_snap_serve(ctx, req.cmd, req.paramsSize, + reply + GSP_RM_CTRL_HDR_SIZE, + reply_data_len - GSP_RM_CTRL_HDR_SIZE, + &out_params_size)) + goto rm_control_reply; + switch (req.cmd) { case TU104_CTRL_FIFO_GET_DEVICE_INFO_TABLE: gsp_rm_handle_fifo_device_table(ctx, sequence, &req, @@ -2876,7 +3436,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; case 0x90f10106: /* NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES */ - gsp_rm_handle_copy_server_pdes(ctx, req_data + GSP_RM_CTRL_HDR_SIZE, + gsp_rm_handle_copy_server_pdes(ctx, req.hObject, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize); out_status = 0; out_params_size = 0; @@ -3187,6 +3748,27 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + /* + * r487/r489 THE GL/VULKAN VASPACE WALL (Xorg/EGL/GZDOOM, every + * session to date): the reply must follow the GSP's own large-reply + * framing. r487 clamped the echo to fit one 64 KB record; the guest + * then refused the record ("Incorrect message length 65528") and, + * when it fit, waited for CONTINUATION records that never came + * ("rpcRecvPoll timedout for fn 71 continuation record + * (remainingSize=0x940)") - rpc.c's _issueRpcLarge consumes the + * first frame up to pRpc->maxRpcSize (65080) and then requires + * NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD frames for the rest + * (the GSP's own _rpcWriteContinuationEntries). The size is left + * unclamped here; gsp_rm_send_rm_control() frames it per that + * protocol. Root evidence: cmd 0x2080a0a4 (PERF_VPSTATES_INFO, + * paramsSize 67392, FINN-serialized) sent during Xorg VASpace + * construction; its failure made vmmCreateVaspace return ffff and + * killed GL - and, after the length was fixed, the 3D-engine + * selection ("Failed to select a 3D engine") until the framing was + * correct. + */ + +rm_control_reply: /* Build the reply header */ memcpy(reply + 0, &req.hClient, 4); memcpy(reply + 4, &req.hObject, 4); @@ -3206,8 +3788,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, out_status, dt_us); } - if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, - reply, GSP_RM_CTRL_HDR_SIZE + out_params_size)) + if (gsp_rm_send_rm_control(ctx, sequence, reply, + GSP_RM_CTRL_HDR_SIZE + out_params_size)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: RM_CONTROL reply send failed cmd=0x%x", req.cmd); else @@ -3217,6 +3799,61 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, } /* + * r510gl: DUP_OBJECT (function 21) - NVOS55_PARAMETERS echo with status=NV_OK. + * + * The guest driver dups an object into a second client at EGL/GLX + * MakeCurrent time (memory and VASpace objects: the GL context's client + * shares the device's allocations), and the historical generic-SUCCESS + * default replied with an EMPTY payload - the driver's reply must carry the + * params struct back (rpc_dup_object_v03_00 = NVOS55_PARAMETERS + * {hClient, hParent, hObject, hClientSrc, hObjectSrc, flags, status}), so + * an empty reply failed the call and the driver tore the whole EGL session + * down (guest witness: 'RPC UNKNOWN(21) seq=0 len=60 -> generic SUCCESS', + * immediately followed by the session's FREE cascade and + * eglMakeCurrent -> EGL_BAD_ALLOC (0x3003), 2026-10-23 r510). + * + * The emulation does not model per-object state for these classes (controls + * and allocs are answered from the request), so the echo with NV_OK is the + * complete answer; the dup'd handle is used by later calls exactly like the + * source handle. + */ +static void gsp_rm_handle_dup_object(struct gsp_emu_context *ctx, + const u8 *elem, u32 sequence) +{ + struct gsp_rm_emu *rm = ctx->rm_emu; + const struct gsprm_rpc_header *hdr = + (const struct gsprm_rpc_header *)(elem + GSPRM_ELEM_HDR_SIZE); + const u8 *req = elem + GSPRM_ELEM_HDR_SIZE + sizeof(*hdr); + u8 params[28]; /* NVOS55_PARAMETERS_v03_00 */ + u32 hDstClient = 0, hDstObject = 0, hSrcClient = 0, hSrcObject = 0; + + if (hdr->length >= sizeof(*hdr) + sizeof(params)) + memcpy(params, req, sizeof(params)); + else + memset(params, 0, sizeof(params)); + + memcpy(&hDstClient, params + 0, 4); + memcpy(&hDstObject, params + 8, 4); + memcpy(&hSrcClient, params + 12, 4); + memcpy(&hSrcObject, params + 16, 4); + memset(params + 24, 0, 4); /* status = NV_OK */ + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DUP_OBJECT hDstClient=0x%x hDstObject=0x%x hSrcClient=0x%x hSrcObject=0x%x seq=%u -> NV_OK echo", + hDstClient, hDstObject, hSrcClient, hSrcObject, sequence); + + /* Small delay before the reply: same reply-matching race mitigation as + * FREE/UNLOADING. */ + udelay(2000); + + if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_DUP_OBJECT, sequence, 0, + params, sizeof(params))) + nv_vgpu_err(ctx->vgpu, "vgsp-rm: DUP_OBJECT reply send failed"); + else + rm->rpcs_replied++; +} + +/* * GSP_RM_FREE (function 10) - NVOS00_PARAMETERS echo with status=NV_OK. * * rpc_gsp_free: { hRoot, hObjectParent, hObjectOld, status }. Echo the @@ -3244,7 +3881,7 @@ static void gsp_rm_handle_free(struct gsp_emu_context *ctx, const u8 *elem, memcpy(&hObjOld, params + 8, 4); memset(params + 12, 0, 4); /* status = NV_OK */ - gsp_sema_service_channel_del(hObjOld); + gsp_sema_service_channel_del(ctx->vgpu, hObjOld); /* * Retire event records that refer to the object being freed. Posting an @@ -3253,11 +3890,19 @@ static void gsp_rm_handle_free(struct gsp_emu_context *ctx, const u8 *elem, * client's notifier events (hSrcResource 0) linger across runs and the * fallback wake-up nudge would hit them. Also retire a client's events * when its root object is freed (hObjOld == hRoot). + * + * m176: the hEvent match must also match the owning client. Handle + * values are recycled across runs, so a late FREE of an old client's + * hEvent must never retire a new client's record with the same numeric + * handle - the new client's completion wake-up would be lost and it + * would wait forever (late-client stall class). hRoot is the freeing + * client's own handle in NVOS00_PARAMETERS. */ for (i = 0; i < rm->events_n; i++) { if (!rm->events[i].valid) continue; - if (rm->events[i].h_event == hObjOld || + if ((rm->events[i].h_event == hObjOld && + (!hRoot || rm->events[i].h_client == hRoot)) || (hObjOld && hObjOld == hRoot && rm->events[i].h_client == hRoot)) { rm->events[i].valid = false; @@ -3455,14 +4100,18 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) return; case GSPRM_RPC_FUNC_CONTINUATION_RECORD: - /* CONTINUATION_RECORD: driver expects a reply. Send SUCCESS. */ - nv_vgpu_info(ctx->vgpu, "vgsp-rm: async RPC %s(%u) seq=%u len=%u -> SUCCESS reply", - gsp_rm_rpc_name(function), function, sequence, - hdr->length); - if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) - nv_vgpu_err(ctx->vgpu, "vgsp-rm: reply send failed fn=%u", function); - else - rm->rpcs_replied++; + /* + * r489: a CONTINUATION_RECORD from the driver is ONE-WAY data + * - the tail of a multi-frame request (its _issueRpcLarge + * sends frame 1 then continuation records). The old reply + * here was a guess ("driver expects a reply"); it injected an + * extra fn-71 record into the reply stream that the driver + * drained as an event ("Unexpected RPC event from GPU0: 0x47") + * and it corrupted the framing of our own large replies + * (which use fn-71 continuations the other way). Consume + * silently: the driver waits only for the reply to frame 1. + */ + rm->rpcs_replied++; return; case GSPRM_RPC_FUNC_SET_GUEST_SYSTEM_INFO: @@ -3485,6 +4134,10 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) gsp_rm_handle_free(ctx, elem, sequence); return; + case GSPRM_RPC_FUNC_DUP_OBJECT: + gsp_rm_handle_dup_object(ctx, elem, sequence); + return; + case GSPRM_RPC_FUNC_SET_PAGE_DIRECTORY: gsp_rm_handle_set_page_directory(ctx, elem, sequence, hdr->length); return; @@ -3657,8 +4310,40 @@ void gsp_rm_emu_fini(struct gsp_emu_context *ctx) void gsp_rm_emu_reset(struct gsp_emu_context *ctx) { struct gsp_rm_emu *rm = ctx->rm_emu; + u32 i, stale; - if (!rm || !rm->parsed) + if (!rm) + return; + + /* + * m176: a reset means the emulated GSP is gone and every RM client + * with it (probe/unload, engine reset). Event records left valid from + * the previous session would be posted to with stale + * (hClient, hEvent) pairs - the guest RM asserts in CliGetEventInfo + * and drops the record - and a recycled hEvent can absorb a wake-up + * meant for the new client. Drop them regardless of parse state. + */ + stale = 0; + for (i = 0; i < GSPRM_MAX_EVENTS; i++) { + if (rm->events[i].valid) { + rm->events[i].valid = false; + stale++; + } + } + if (stale) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: reset dropped %u stale event record(s)", + stale); + + /* + * m177: RM reload means a new guest boot: the guest's direct-map base + * and every content-scanned translation may have changed. Raise the + * boot-scoped wipe (executed on the sema svc thread; client session + * flushes intentionally keep that state - see svc_session_flush). + */ + gsp_sema_service_guest_reset(); + + if (!rm->parsed) return; nv_vgpu_info(ctx->vgpu, "vgsp-rm: reset (rpcs=%llu replied=%llu unknown=%llu)", diff --git a/kernel/gsp_rm_emulation.h b/kernel/gsp_rm_emulation.h index e464809..45fcc64 100644 --- a/kernel/gsp_rm_emulation.h +++ b/kernel/gsp_rm_emulation.h @@ -45,6 +45,7 @@ struct gsp_emu_context; #define GSPRM_RPC_FUNC_GSP_RM_CONTROL 76 #define GSPRM_RPC_FUNC_GSP_RM_ALLOC 103 #define GSPRM_RPC_FUNC_FREE 10 +#define GSPRM_RPC_FUNC_DUP_OBJECT 21 #define GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER 47 #define GSPRM_RPC_FUNC_UPDATE_BAR_PDE 70 #define GSPRM_RPC_EVENT_POST_EVENT 0x1003 @@ -197,11 +198,13 @@ struct gsp_rm_emu { u32 notify_index; u32 data; bool valid; + unsigned long last_post; /* m171: jiffies of the last posted record */ } events[GSPRM_MAX_EVENTS]; u32 events_n; u64 events_posted; u64 events_dropped; - u32 event_burst; /* event records written since the last reply */ + u32 event_burst; /* event records written in the current window */ + unsigned long event_burst_at; /* m171: burst window start (time-based reset) */ }; int gsp_rm_emu_init(struct nv_vgpu_device *vgpu, struct gsp_emu_context *ctx); diff --git a/kernel/gsp_rm_tu104.h b/kernel/gsp_rm_tu104.h index b5ee4d7..3382939 100644 --- a/kernel/gsp_rm_tu104.h +++ b/kernel/gsp_rm_tu104.h @@ -754,7 +754,18 @@ static const struct tu104_internal_device_info tu104_internal_device_table[] = { TU104_CE_CAPS0_SYSMEM | \ TU104_CE_CAPS0_P2P | \ TU104_CE_CAPS0_BL_GT_64K) -#define TU104_CE_CAPS1 0x00 +/* + * Caps byte 1: the 2D/3D copy path requires a CE that supports the + * blocklinear access modes (hardware bug 200288104): the X driver's + * nvCEChooseEngine skips every CE without CE_SUPPORTS_NONPIPELINED_BL + * (byte1 0x01; and PIPELINED_BL 0x02 for the preferred pass) and then + * fails with "Failed to find a suitable copy object" - the observed wall + * after the DMA pushbuffer test started passing. Turing's CEs are fully + * capable (pipelined + non-pipelined BL + pitchlinear): byte1 = 0x03. + * (Reference: ctrl2080ce.h NV2080_CTRL_CE_CAPS_CE_SUPPORTS_*_BL 1:0x01/ + * 1:0x02.) + */ +#define TU104_CE_CAPS1 0x03 /* ENGINE_TYPE_COPY0 = 0x9, COPY(i) = COPY0 + i (cl2080_notification.h) */ #define TU104_ENGINE_TYPE_COPY0 0x00000009 diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index 20dfc54..42e8d76 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -160,8 +160,27 @@ struct sema_map_entry { #define SVC_USERD_GP_GET 0x88 #define SVC_USERD_GP_PUT 0x8c +/* + * r496b: guest-RAM ring content scan (see gr_ram_cursor in struct + * svc_channel) and the VRAM USERD shape scan (svc_chan_userd). One scan + * slice per service pass so the service thread stays responsive. + */ +#define SVC_GR_RAM_SLICE (512ULL << 20) /* guest RAM per pass */ +#define SVC_GR_RAM_PASS_MAX 16 /* slices before giving up */ +#define SVC_GR_RAM_WIN (2ULL << 20) /* allocation window pinned */ + struct svc_channel { bool valid; + u32 session_gen; /* m176: session generation this entry was + * created in; learned PA state (counter_pa/ + * release_pa) is only inherited on a re- + * registration within the same generation, so + * a recycled handle from a previous client can + * never keep the dead session's write targets */ + struct nv_vgpu_device *vgpu; /* owning mdev (guest session); the + * table is keyed by (vgpu, h_object) + * because two guests' stock allocators + * choose identical handles */ u32 h_object; u32 engine_type; u32 cid; /* RM channel id (doorbell token vector) */ @@ -184,6 +203,28 @@ struct svc_channel { * reset (PUT dropping below put_prev) resyncs * last_get to slot 0 so the live first push is * processed instead of skipped. */ + bool gr_pub_armed; /* mm2 (r207/r208 second-client hang): a GR + * channel's completion word may only receive a + * GP_PUT publish after the channel has been + * proven to belong to the CURRENT session. A + * GR ring/USERD surviving in its fixed VRAM + * slot from the previous session reads a + * residual put at discovery (e.g. 109/112); + * publishing that residual put into the + * channel's own word (8c634144 made the target + * release_pa - the client's waited word) while + * the stale ring is consumed poisons the fresh + * client's UVM tracking semaphore: its + * completed_value observes a far-forward + * payload, the later reset/release sequence is + * a >MAX_JUMP (2*entries) downward observation, + * and the fresh session's first waits wedge + * (matmul: one GR push per channel then 242 s + * silent). Armed when a GR channel is seen with + * put <= 1 (pristine ring) or when the fresh- + * put-drop rewind proves a session switch; the + * rewind also voids any publish state the + * stale pass advanced. */ bool logged; /* one state log per channel */ bool pool_rel_seen; /* ISSUE-0002 r70: a pool release word for * this channel has been read through a @@ -228,6 +269,10 @@ struct svc_channel { * window (observed: one 8 s grace per push, * serializing the runtime init) */ DECLARE_BITMAP(slot_done_epoch, SVC_GPFIFO_MAX_BITS); + u8 slot_pend_tries[SVC_GPFIFO_MAX_BITS]; /* fail4 r308: bounded + * re-holds of a denied device-dst copy slot */ + u32 counter_gen; /* r311: session generation that authored the + * completion word (guards the residue zero) */ /* GPFIFO slots already consumed in the current * put epoch (the client's put value since it * last wrapped). A pin-batch rewind replays @@ -298,6 +343,10 @@ struct svc_channel { * publish targets counter_pa instead, so both * families of the counter page get written * (SA-COUNTER FORENSICS-grv7-counter F1) */ + u64 release_va; /* r378: the release VA release_pa was learned + * from, kept so a late write (stale-recover + * zero) can re-validate the target through + * gsp_sema_write_guard before it lands */ u32 counter_last; /* last value published to counter_pa */ u64 pushpg_va; /* last push page logged as pin-vs-window * divergent (rate limit: one line per page) */ @@ -343,7 +392,107 @@ struct svc_channel { u32 gr_hold_idx; /* GPFIFO slot the window was armed for (identity: * an entry holds at most once, then is consumed * even if still unresolved - bounded zeros, never - * a hang) */ + * a hang); m156: re-held while the window is + * live so a busy forward gets its real retry */ + /* + * r509: one-shot verdict of the absolute bound. The expiry call + * pushes the deadline (so the hold window restarts) and latches this + * flag; the r351 retry-forward block consumes it within the same + * pass. Without the latch the two call sites in one pass could not + * both observe the expiry (the first push of the deadline made the + * second call see a fresh window - r213-r215's arming bug in + * reverse). + */ + bool gr_hold_bound_hit; + unsigned long gr_hold_deadline; /* GRFIX FIX1b: hard expiry of the data- + * stalled hold for gr_hold_dead_va. + * Fixed once on the first hold of that + * launch push VA and NEVER moved while + * the launch stays held; when it passes + * the launch proceeds (pre-8b058884 + * behaviour), so the guest can never + * wait unboundedly. Keyed by the launch + * VA, not the GPFIFO slot: pin-batch + * rewinds re-decode one launch at + * different slot indices (r213: + * entry 107 -> 107 -> 99), which + * re-armed a slot-keyed deadline on + * every pass. */ + u64 gr_hold_dead_va; /* launch push VA the bound belongs to */ + unsigned long gr_hold_dead_since; /* first-hold stamp (waited_ms) */ + bool gr_hold_exp_logged; /* one GRFILL-EXPIRED line per launch */ + u64 opdef_hold_dead_va; /* r361: launch push VA the opdef bound + * belongs to (slot-keyed `since` is + * defeated by rewind replays) */ + unsigned long opdef_hold_deadline; /* absolute opdef hold expiry */ + bool opdef_hold_exp_logged; /* one OPDEF-EXPIRED line per launch */ + u32 gr_rehold_idx; /* m156: slot whose next decode must force a + * fresh launch capture (a held entry is + * re-decoded as a replay, which suppresses + * the capture; forcing it lets a deferred + * launch be forwarded once the engine's + * in-flight work completes) */ + u32 gr_nocap_idx; /* m164 CLASS-2: slot in the no-capture hold + * (a readable launch push whose decode saw + * QMD words but built no candidate) */ + u32 gr_nocap_replay;/* 1 = window opened on a replay decode */ + u64 gr_nocap_va; /* push VA the window was opened for */ + unsigned long gr_nocap_since; /* absolute first-seen stamp */ + /* + * r496b (X/GL kernel-RM sysmem channel): ring discovery by content in + * guest RAM. The CPU RM maps sysmem objects locally (no + * SET_PAGE_DIRECTORY, no rm_mem record carrying the ring VA), so + * svc_lookup_trusted cannot name the ring page; the driver's kickoff + * GPFIFO entries {pushbuffer VA, len} identify it instead. The scan + * is sliced (SVC_GR_RAM_SLICE per pass) and cursor-resumed so the + * service thread stays responsive. + */ + u64 gr_ram_cursor; /* resume point of the content scan */ + u8 gr_ram_passes; /* slices walked so far */ + bool gr_ram_logged; /* one discovery line per channel */ + /* + * r496e: true when the ring was found by the RAM content scan (a + * kernel-RM sysmem channel - the X/GL 3D channel). Such a channel's + * pushes are 2D/3D method streams the emulation does not execute; + * the correct model is consume-and-apply-releases: no QMD capture, + * no launch forward, no no-capture hold. Also allows an all-zero + * NOP push (the driver's pushbuffer sanity test writes exactly one + * zero dword) to be consumed as the no-op it is. + */ + bool gr_ram_ring; + /* + * r496b: the FB-scan USERD (see svc_userd_fb_scan). Latched when the + * registered/ring-derived candidates all fail; the scan validates the + * hit against the ring entries the put claims, so a foreign channel's + * USERD cannot be adopted. Bounded retries: the driver kicks off a + * few passes after channel registration, so a scan before the first + * put must not retire the discovery. + */ + u8 userd_scans; + /* + * r471: the completion-counter publish must never advance the + * guest's word past a push whose content was still unwritten when + * the entry was consumed (the streams_events staging-reuse + * corruption: the publish satisfied the client's D2H wait before + * its copy ran, and the readback drained a staging buffer that + * still held the PREVIOUS copy's bytes). Lowest ring slot consumed + * with all-zero content; 0 = none. Cleared when a later decode of + * a GR push reads non-zero content (the content landed - the rewinds + * re-read entries constantly, so this heals within a pass or two). + */ + u32 gr_bad_put; + /* + * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. A + * channel that cannot make forward progress (last_get pinned behind + * put, or a residual session put that never drops) is detected here + * and recovered instead of spinning. All fields MUST be in the pump + * snapshot write-back list (GRFIX NOTE). + */ + u64 wd_last_get; /* last_get at the window start */ + u64 wd_put; /* put at the window start */ + u64 wd_sub; /* submissions at the window start */ + unsigned long wd_since; /* window start (0 = no window) */ + unsigned long wd_rewind_off_until; /* pin-batch rewind cooldown */ }; /* @@ -399,6 +548,12 @@ struct svc_vaspace { u64 pdb; u32 entries; /* root entry count */ int ap; /* SVC_APERTURE_* of the directory itself */ + unsigned long stamp; /* r459: last registration/use; LRU evict + * when the table is full (a stale entry from + * a dead client otherwise blocks every new + * client after the 8th registration: the + * mid-boot walk failure that made the 10th+ + * client's copies fail rc=-2) */ }; struct svc_lat { @@ -449,11 +604,136 @@ static u32 svc_ring_n; * client spins). One clean boot always works, so make the next session look * like one: when the channel set empties, request a flush and let the next * rebuild start from the same empty state as a fresh boot. + * + * Dual-guest staging (goal 1.0): the discovery tables are still shared + * between guests (per-guest learned tables are the staged G4 remainder), so + * the flush is requested only when the channel set is globally empty. The + * active decision is made per guest (svc_guests below): one guest closing + * its last channel never flushes while another guest still has live + * channels - otherwise the departing guest would wipe the live guest's + * pins/maps and stall it. Single-guest behaviour is unchanged: the one + * guest's last-channel close is also the global empty transition. */ static u8 svc_sess_flush; +static u8 svc_guest_reset_req; /* m177: guest RM reload; wiped on svc thread */ static u8 svc_sess_active; /* a channel of a session has registered (until flush) */ +static u32 svc_sess_gen; /* m176: bumped by every executed session flush; + * stamps channels/scrubbers so per-session learned + * state is never inherited by a recycled handle */ static struct completion svc_sess_flush_done; /* a pending flush has executed */ static unsigned long svc_ring_done; /* CE ranks whose ring is pinned (session state) */ + +/* Per-guest session records (one per open mdev; see SVC_MAX_CHANNELS for the channel cap). */ +#define SVC_MAX_GUESTS 8 +struct svc_guest_sess { + struct nv_vgpu_device *vgpu; /* NULL = free slot */ + bool active; /* a channel of this guest is + * registered until this guest's + * last channel closes */ + u64 last_build; /* per-guest svc_rebuild freshness */ +}; +static struct svc_guest_sess svc_guests[SVC_MAX_GUESTS]; +static DEFINE_SPINLOCK(svc_guest_lock); + +/* + * Find (or, with create, allocate) the session record for one mdev. The + * records are pointer-keyed; a removed mdev's record is freed by + * gsp_sema_service_vgpu_cleanup() so the pointer can never match a newly + * allocated device. + */ +static struct svc_guest_sess *svc_guest_get(struct nv_vgpu_device *vgpu, + bool create) +{ + struct svc_guest_sess *sess = NULL; + unsigned long flags; + int i; + + if (!vgpu) + return NULL; + spin_lock_irqsave(&svc_guest_lock, flags); + for (i = 0; i < SVC_MAX_GUESTS; i++) { + if (svc_guests[i].vgpu == vgpu) { + sess = &svc_guests[i]; + break; + } + if (create && !sess && !svc_guests[i].vgpu) + sess = &svc_guests[i]; + } + if (sess && create && !sess->vgpu) { + memset(sess, 0, sizeof(*sess)); + sess->vgpu = vgpu; + } + spin_unlock_irqrestore(&svc_guest_lock, flags); + return sess; +} + +/* + * CeUtils scrubber channel (the guest PMA's internal CE channel created at + * driver init). When the PMA must hand out scrubbed pages it waits on the + * channel's finish payload (channel buffer offset finishPayloadOffset = + * channelPbSize + GPFIFO_SIZE + 4; for the regular scrubber = 0x6c004), + * which the physical CE would advance as the scrub work completes. In this + * emulation nothing authors it, so the third+ CUDA session's cudaMalloc + * freezes in scrubCheckAndWaitForSize. + * + * The scrubber channel is identified by its GPFIFO entry count + * (NUM_COPY_BLOCKS = 4096; client channels use 1024); its USERD PA is known + * from the channel ALLOC registration. The channel buffer's PA cannot be + * resolved through the global VA map (the kernel VA space collides + * numerically with the client's CE push-buffer window), so the guest + * scanner (vmallocinfo + kcore page-table walk) registers it through + * sema_scrubbuf_add. The service then authors, from the channel's own + * GP_PUT: USERD GP_GET = put, host semaphore = put (so the guest's + * channelWaitForFreeEntry never blocks), and finish payload = submitted + * block count (so the scrub wait completes). Idempotent, forward-only. + */ +struct svc_scrub { + struct nv_vgpu_device *vgpu; /* owning mdev (NULL = free slot) */ + bool userd_ok; /* a 4096-entry channel was registered */ + u32 h_object; /* handle of the registered scrubber channel */ + u64 userd_pa; + u32 userd_as; + u64 gpfifo_va; + u32 entries; + bool pa_ok; /* scanner registered fin/hsem PAs */ + u64 fin_pa; /* finish payload word (u32) */ + u64 hsem_pa; /* channel host semaphore word (u32) */ + bool seeded; + u32 put_last; + u64 submitted; /* blocks submitted since seeding */ + u64 authored; /* last authored finish value */ + u32 pa_gen; /* scanner registration generation (logged) */ + u32 session_gen; /* m176: session generation of the last registration */ + u32 log_budget; +}; +/* + * One scrubber record per guest: every guest's PMA has its own 4096-entry + * scrubber channel, and the authored words must be written through the + * owning guest's address space (writing guest A's PA through guest B's VFIO + * device would corrupt B). Keyed by vgpu pointer, freed by + * gsp_sema_service_vgpu_cleanup(). + */ +static struct svc_scrub svc_scrubs[SVC_MAX_GUESTS]; + +static struct svc_scrub *svc_scrub_get(struct nv_vgpu_device *vgpu, bool create) +{ + struct svc_scrub *s = NULL; + int i; + + if (!vgpu) + return NULL; + for (i = 0; i < SVC_MAX_GUESTS; i++) { + if (svc_scrubs[i].vgpu == vgpu) + return &svc_scrubs[i]; + if (create && !s && !svc_scrubs[i].vgpu) + s = &svc_scrubs[i]; + } + if (s) { + memset(s, 0, sizeof(*s)); + s->vgpu = vgpu; + } + return s; +} static u8 svc_pb_passes; static u64 svc_pb_last; /* @@ -592,6 +872,10 @@ struct sema_svc { u64 op_tr_log; /* operand targeted-resolve budget */ u32 op_tr_pending; /* an operand's targeted scan is in progress */ u64 copy_watch_log; /* sema_copy_watch log budget */ + u64 copy_dst_hold; /* m169: device-dst copies held for + * want of a pin (stale-table guard) */ + u64 copy_dst_relaxed; /* m169b: guards released without + * a pin after the bounded wait */ u64 copy_pat_log; /* sema_copy_pat log budget */ u64 copy_dst_log; /* sema_copy_dst log budget */ u32 op_tr_src; /* provenance of last svc_op_translate */ @@ -599,6 +883,20 @@ struct sema_svc { u64 push_pat_log; /* push-content pattern budget */ u64 dma_skip_log; /* unrecognized-DMA budget */ u64 xfer_raw_log; /* operandless transfer raw-dump budget */ + u64 fill_ram_log; /* r361 physical-RAM fill build budget */ + u64 pushtrace_all; /* fail4: de-windowed push decode + * trace (every push, bounded) */ + u64 opadd_log; /* fail4: every op appended */ + u64 op2d_raw_log; /* r458: raw 2D-shaped copy pushes */ + u64 dmatrace_log; /* fail4: denied-transfer trace */ + u64 pend_dbg_log; /* fail4 r305: pending/hold witness */ + u64 snap_gate_log; /* fail4 r311: capture-gate witness */ + u64 copyent_log; /* fail4: svc_copy entry trace */ + u64 bigrun_log; /* fail4: big method-data run headers */ + u64 lid_hdr_log; /* fail4: LOAD_INLINE_DATA headers */ + u64 skiprun_log; /* fail4: silently skipped data runs */ + u64 mystery_log; /* fail4: moderate op=1 runs on big pushes */ + u64 next_hdr_log; /* fail4: header after each big LID run */ u64 grce_log; /* GRCE inline-write budget */ u32 cur_ch; /* channel of the op being run */ u64 entry_log; /* per-push summary budget */ @@ -644,10 +942,16 @@ struct sema_svc { * expectation advanced, write queued */ u64 gpfifo_held; /* passes an entry was left pending * for a translation/grace window */ + u64 gr_nocap_held; /* m164: GR holds opened for a + * launch push whose decode saw QMD + * words but built no candidate */ + u64 gr_nocap_consumed; /* m164: no-capture windows that + * expired and consumed the entry */ u64 gpfifo_starved; /* entries consumed without a * deliverable exact slot */ u64 counter_rel_applied; /* releases to registered counters */ u64 counter_rel_dbg; /* rel-apply diagnostic budget */ + u64 crel_entry_dbg; /* C1: rel-apply entry budget, 512/session */ u64 counter_noslot_dbg; /* "no slot registered" budget, * separate so it cannot hide the * written/skip lines */ @@ -669,9 +973,13 @@ struct sema_svc { u64 opdef_held; u64 opdef_applied; u64 opdef_expired; + u64 opdef_delivered_late; /* stall fix: set delivered at consume */ + u64 opdef_drop_lost; /* withheld releases dropped unapplied */ u64 vidmem_writes; /* semaphores written in vidmem */ u64 gpfifo_log; /* channel-state log budget */ u64 gpfifo_entry_log; /* per-entry log budget */ + u64 walk_dbg; /* C4: GR put/walk witness budget */ + u64 gr_rel_auth_log; /* F1: GR-REL-AUTHOR budget, 256/session */ u64 doorbell_count; /* NV_VIRTUAL_FUNCTION_DOORBELL pokes */ u32 doorbell_token; /* last token written */ u32 doorbell_log; /* token log budget */ @@ -783,6 +1091,22 @@ struct sema_svc { * abort, or a release0 page the backend could not observe). */ u64 fwd_gr_sw_rel; + /* + * STALLFREE: failed forwarded launches the pass-top outcome check + * completed in software because no hold owned their release (the + * bounded-completion guarantee; zero on a green run). + */ + u64 fwd_gr_stall_swrel; + /* + * STALLFREE r225: GR channels recovered by the stale-epoch / + * stale-replay wedge watchdog (zero on a green run). + */ + u64 gr_stale_wedge; + u64 fwd_gr_hold_exp; /* GRFIX FIX1: data-stalled holds + * that reached the hard bound and + * were consumed with the software + * release (proof the bound fires + * instead of a stall) */ /* Distinct stalled VA tracking for rate-limited logging. */ #define SVC_STALLED_MAX 256 @@ -950,6 +1274,13 @@ static bool svc_gr_content_rearm; static u32 svc_cepin_log; /* + * m154: injected pins dropped because they collide with a live channel's + * GPFIFO ring range (see svc_handle_map_add). Counted + logged so the + * defensive drop is visible in the boot log. + */ +static u32 svc_ring_pin_skips; + +/* * m148 (second-client wedge): re-arm the per-rank one-shot ring/push-buffer * latches for a (re-)registered channel. The guest driver rebuilds the GSP * session for every client context and RM reuses the same object handles and @@ -995,6 +1326,7 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) if (!desc->engine_type) { WRITE_ONCE(svc_gr_content_rearm, true); svc_cepin_log = 0; + svc_ring_pin_skips = 0; } /* @@ -1019,7 +1351,8 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) } spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].h_object == desc->h_object) { + if (svc.chan[i].valid && svc.chan[i].vgpu == desc->vgpu && + svc.chan[i].h_object == desc->h_object) { ch = &svc.chan[i]; break; } @@ -1030,9 +1363,21 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) int k; u64 keep_counter_pa = 0; u64 keep_release_pa = 0; + u64 keep_release_va = 0; u8 keep_gr_index = 0, keep_ce_index = 0; + bool keep_pub_armed = false; + u32 keep_counter_gen = 0; DECLARE_BITMAP(keep_rel_done, SVC_GPFIFO_MAX_BITS); - bool same = ch->valid && ch->h_object == desc->h_object; + bool same = ch->valid && ch->vgpu == desc->vgpu && + ch->h_object == desc->h_object; + /* m176: only a same-generation re-registration may inherit the + * learned write targets. After a session flush the same numeric + * handle belongs to a fresh client whose counter/release slots + * live in a brand-new page; inheriting the dead session's PAs + * made every release land on the old page (and be skipped + * against its stale contents) while the new client polled its + * own slot forever. */ + bool same_gen = same && ch->session_gen == svc_sess_gen; /* * A re-registration of the same channel must not lose the @@ -1041,32 +1386,60 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * counter layout does not change). Same for the increment * release marks: losing them would re-apply pushes. */ - if (same) { + if (same_gen) { keep_counter_pa = ch->counter_pa; keep_release_pa = ch->release_pa; + keep_release_va = ch->release_va; keep_gr_index = ch->gr_index; keep_ce_index = ch->ce_index; + /* + * r311: a same-session re-registration must not + * forget that this session already proved itself + * (gr_pub_armed) - losing it re-arms the residual + * watchdog, and its recovery zeroes a completion + * word this live session authored, which the + * client's UVM reads as a downward jump and never + * completes on (r311 vector_add: pub-adv/zero + * 3 s cycles with the client polling forever). + */ + keep_pub_armed = ch->gr_pub_armed; + keep_counter_gen = ch->counter_gen; bitmap_copy(keep_rel_done, ch->rel_done, SVC_GPFIFO_MAX_BITS); } memset(ch, 0, sizeof(*ch)); - if (same) + ch->gr_hold_idx = ~0u; /* m156: no hold armed yet */ + ch->gr_hold_dead_va = 0; /* GRFIX: no launch bound armed */ + ch->gr_hold_exp_logged = false; + ch->opdef_hold_dead_va = 0; /* r361: no opdef bound armed */ + ch->opdef_hold_exp_logged = false; + ch->gr_rehold_idx = ~0u; + ch->gr_nocap_idx = ~0u; /* m164: no no-capture window armed */ + ch->session_gen = svc_sess_gen; /* m176 */ + if (same_gen) { bitmap_copy(ch->rel_done, keep_rel_done, SVC_GPFIFO_MAX_BITS); + ch->gr_pub_armed = keep_pub_armed; + ch->counter_gen = keep_counter_gen; + } /* - * Rank among the live CE channels. The RM hands out object - * handles in creation order and the guest's UVM allocates one - * push buffer, ring and tracking semaphore per channel in that - * same order, so the rank is what ties a channel to its ring - * offset and its push-buffer base (see svc_ring_autopin / - * svc_pb_pair). + * Rank among the live CE channels OF THIS GUEST. The RM hands + * out object handles in creation order and the guest's UVM + * allocates one push buffer, ring and tracking semaphore per + * channel in that same order, so the rank is what ties a + * channel to its ring offset and its push-buffer base (see + * svc_ring_autopin / svc_pb_pair). The rank must not count the + * other guest's channels: their handles are independent and + * their allocators interleave. */ if (desc->engine_type) { u8 rank = 0; for (k = 0; k < SVC_MAX_CHANNELS; k++) - if (svc.chan[k].valid && svc.chan[k].engine_type && + if (svc.chan[k].valid && + svc.chan[k].vgpu == desc->vgpu && + svc.chan[k].engine_type && svc.chan[k].h_object < desc->h_object) rank++; ch->ce_index = rank; @@ -1074,13 +1447,16 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) u8 rank = 0; for (k = 0; k < SVC_MAX_CHANNELS; k++) - if (svc.chan[k].valid && !svc.chan[k].engine_type && + if (svc.chan[k].valid && + svc.chan[k].vgpu == desc->vgpu && + !svc.chan[k].engine_type && svc.chan[k].h_object < desc->h_object) rank++; ch->gr_index = rank; } ch->valid = true; svc_sess_active = 1; + ch->vgpu = desc->vgpu; ch->h_object = desc->h_object; ch->engine_type = desc->engine_type; ch->cid = desc->cid; @@ -1095,9 +1471,18 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) if (same) { ch->counter_pa = keep_counter_pa; ch->release_pa = keep_release_pa; + ch->release_va = keep_release_va; ch->gr_index = keep_gr_index; ch->ce_index = keep_ce_index; } + /* Per-guest session flag: this guest has a live channel. */ + { + struct svc_guest_sess *sess = + svc_guest_get(desc->vgpu, true); + + if (sess) + sess->active = true; + } } spin_unlock_irqrestore(&svc.chan_lock, flags); @@ -1110,6 +1495,48 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) */ svc_chan_rearm_rank(ch); svc.last_chan_reg = jiffies; + /* + * CeUtils scrubber channel: NUM_COPY_BLOCKS = 4096 GPFIFO + * entries (client channels use 1024). Stash its USERD and ring + * geometry for the scrub pass; the channel itself is left + * entirely alone (no rank change, no pump touch changes) - the + * pass authors the finish payload from the USERD alone. A + * colliding numeric-handle FREE may still unregister the + * channel; the stashed geometry survives, so the scrub path + * does not depend on the channel-table identity at all. + */ + if (desc->gpfifo_entries >= 4096 && desc->gpfifo_va && + desc->userd_pa) { + struct svc_scrub *s = svc_scrub_get(desc->vgpu, true); + + if (!s) + goto scrub_done; /* table full: max guests */ + /* Re-seed the counting on a fresh scrubber object or a + * new client session (m176: the counter/submitted state + * is per-session; a recycled scrubber handle after a + * flush must start from the new channel's GPPut, not + * carry the dead session's counts). */ + if (!s->userd_ok || + s->session_gen != svc_sess_gen || + s->h_object != desc->h_object || + s->gpfifo_va != desc->gpfifo_va) { + s->seeded = false; + s->submitted = 0; + } + s->session_gen = svc_sess_gen; + s->userd_ok = true; + s->h_object = desc->h_object; + s->userd_pa = desc->userd_pa; + s->userd_as = desc->userd_as ? desc->userd_as + : SVC_APERTURE_SYS; + s->gpfifo_va = desc->gpfifo_va; + s->entries = desc->gpfifo_entries; + pr_info("sema-svc: scrubber channel 0x%x detected (gpfifo=0x%llx entries=%u userd=0x%llx as=%u)\n", + desc->h_object, desc->gpfifo_va, + desc->gpfifo_entries, desc->userd_pa, + s->userd_as); + } +scrub_done: pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx\n", desc->h_object, desc->gpfifo_va, desc->gpfifo_entries, desc->userd_pa, desc->engine_type, desc->cid, @@ -1121,7 +1548,8 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) } /* P6b wave-1.5 hold state is defined with the forwarding section below. */ -static void svc_fwd_hold_abort(u32 h_object); +static void svc_fwd_hold_abort(struct nv_vgpu_device *vgpu, u32 h_object); +static void svc_fwd_hold_clear(void); /* m176: session flush drops a stale hold */ /* * P6b wave-2 inline software fallback for a withheld (skipped) candidate; * defined with the forwarding section below. Service-thread only. @@ -1139,8 +1567,23 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, static void svc_opdef_drop(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u32 entry_idx); static void svc_opdef_reset_all(void); - -void gsp_sema_service_channel_del(u32 h_object) +static void svc_opdef_finish_ch(struct svc_channel *ch, u32 idx); +static u32 svc_opdef_log; /* m176: forward decl for the session flush */ +/* + * One-cycle GAP-1/GAP-2 instrumentation budgets (agent-datapath plan C1-C10): + * small, reset per session where they are service-thread state, so a later + * client's strand is never blacked out by the first client's spend. + */ +static u32 svc_relx_log; /* C2: deferred-launch apply entry witness */ +static u32 svc_opdef_clr_log; /* C3: opdef-clear caller witness */ +static u32 svc_busy_defer_log; /* C8: GR busy-defer timeout witness */ +static u32 svc_hold_fb_log; /* C7: hold failure/timeout fallback witness */ +static u32 svc_cand_ovw_log; /* C9: candidate overwrite witness */ +static u32 svc_inline_recs_log; /* C10: inline-recs digest witness */ +static u32 svc_gr_await_log; /* r351: launch-held-for-trust witness */ +static u32 svc_gr_retry_log; /* r351: retry-forward/proceed witness */ + +void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) { unsigned long flags; int i; @@ -1150,7 +1593,8 @@ void gsp_sema_service_channel_del(u32 h_object) spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].h_object == h_object) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == h_object) { svc.chan[i].valid = false; pr_info("sema-svc: channel 0x%x unregistered (submissions=%llu)\n", h_object, svc.chan[i].submissions); @@ -1160,19 +1604,27 @@ void gsp_sema_service_channel_del(u32 h_object) /* * ISSUE-0002: when the last channel closes, the next client is a new * session in a new UVM VA space; its discovery state must not inherit - * this session's map/pins/page-directory latch. Request the flush - * here; it executes at the next rebuild (the only writer of that - * state) just like a fresh boot would start empty. Only the transition - * from "a session existed" to "empty" may request it: redundant - * deletes for handles that never registered would otherwise re-flush - * (and wipe) a session that started after the real one ended. + * this session's map/pins/page-directory latch. The active decision is + * per guest (svc_guests) and the flush request is raised only when NO + * guest has a live channel left: the discovery tables are still shared + * between guests (staged G4), so flushing while another guest is live + * would wipe that guest's pins and stall it. One guest: the last-channel + * close IS the global empty transition, so behaviour is unchanged. */ if (svc_sess_active) { + struct svc_guest_sess *sess = svc_guest_get(vgpu, false); + bool any = false, own = false; + for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid) - break; + if (!svc.chan[i].valid) + continue; + any = true; + if (svc.chan[i].vgpu == vgpu) + own = true; } - if (i == SVC_MAX_CHANNELS) { + if (sess && !own) + sess->active = false; + if (!any) { svc_sess_active = 0; svc_sess_flush = 1; } @@ -1187,9 +1639,10 @@ void gsp_sema_service_channel_del(u32 h_object) * P6b wave 1.5: a deleted channel's held releases must not wait out * the timeout; mark the hold aborted so the pump applies them (or * drops them if the snapshot already lost the channel) on its next - * pass. + * pass. The hold is matched per (vgpu, handle): the other guest's + * identical handle must not abort this guest's hold. */ - svc_fwd_hold_abort(h_object); + svc_fwd_hold_abort(vgpu, h_object); } /* @@ -1197,6 +1650,90 @@ void gsp_sema_service_channel_del(u32 h_object) * the page tables itself and hands their base address to the GPU). Store it so * the service can translate that space's VAs directly. */ +/* + * r496j: unverified page-directory candidates from COPY_SERVER_PDES. A + * kernel-RM internal VAS publishes its chain there, but registering every + * client's directory directly lets a stale kernel directory answer another + * client's VA in the walker's first-success order (the r459 stale-directory + * class; observed as a CUDA client hanging after DEVICE). Candidates are + * parked here and registered only after a walk of the client's own + * content-validated ring VA through the candidate reproduces that ring's PA + * (svc_gr_ram_ring_scan's latch calls svc_vas_candidates_verify). + */ +#define SVC_VAS_CAND_MAX 16 +static struct { + bool valid; + u32 h_vaspace; + u64 pdb; + u32 entries; + bool vidmem; +} svc_vas_cand[SVC_VAS_CAND_MAX]; + +static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, + u64 va, int tap, u64 *addr, int *ap, bool trace); + +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem) +{ + int i; + + if (!pdb || entries < 2 || entries > 4096) + return; + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + if (svc_vas_cand[i].valid && + svc_vas_cand[i].h_vaspace == h_vaspace && + svc_vas_cand[i].pdb == pdb) + return; /* already parked */ + if (!svc_vas_cand[i].valid) { + svc_vas_cand[i].valid = true; + svc_vas_cand[i].h_vaspace = h_vaspace; + svc_vas_cand[i].pdb = pdb; + svc_vas_cand[i].entries = entries; + svc_vas_cand[i].vidmem = vidmem; + return; + } + } +} + +/* + * Verify each parked candidate against one content-validated translation + * (the ring VA -> PA the RAM scan just proved): walk the ring page through + * the candidate and register only the directory that reproduces it. Runs on + * the service thread from the ring-scan latch; the walk itself is cheap (a + * few table reads). + */ +static void svc_vas_candidates_verify(struct nv_vgpu_device *vgpu, u64 va, + u64 pa) +{ + int i; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + u64 wpa = 0; + int wap = 0; + int ap = svc_vas_cand[i].vidmem ? SVC_APERTURE_FB + : SVC_APERTURE_SYS; + struct sema_vaspace_desc d; + + if (!svc_vas_cand[i].valid) + continue; + if (svc_gmmu_walk(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, va & ~0xfffULL, + ap, &wpa, &wap, false) || + (wpa & ~0xfffULL) != (pa & ~0xfffULL)) + continue; + memset(&d, 0, sizeof(d)); + d.h_vaspace = svc_vas_cand[i].h_vaspace; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + pr_info("sema-svc: VAS 0x%x page directory registered after verified ring walk (pdb=0x%llx entries=%u ap=%d)\n", + d.h_vaspace, d.pdb, d.num_entries, ap); + break; + } +} + void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) { struct svc_vaspace *vs = NULL; @@ -1215,12 +1752,40 @@ void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) if (!vs && !svc.vas[i].valid) vs = &svc.vas[i]; } + /* + * r459: the table is 8 slots and the guest driver does not always + * send a teardown for a client's VA space (observed: 23 distinct + * registrations, 0 drops in one boot). Once all slots hold a dead + * client's directory, a NEW client's registration found no slot and + * every walk of its VAs failed (copies rc=-2, launches reading + * stale pages - the 10th-client mid-boot degradation). Evict the + * least-recently-registered entry instead of refusing the new one; + * the old client is gone and its directory pointer is dead guest + * memory, so the eviction can only remove stale authority. + */ + if (!vs) { + unsigned long oldest = 0; + int oi = 0; + + for (i = 0; i < SVC_MAX_VASPACES; i++) { + if (!svc.vas[i].valid) + continue; + if (!oldest || time_before(svc.vas[i].stamp, oldest)) { + oldest = svc.vas[i].stamp; + oi = i; + } + } + vs = &svc.vas[oi]; + pr_info("sema-svc: VA space table full; evicting 0x%x (pdb=0x%llx) for 0x%x\n", + vs->h_vaspace, vs->pdb, desc->h_vaspace); + } if (vs) { vs->valid = true; vs->h_vaspace = desc->h_vaspace; vs->pdb = desc->pdb; vs->entries = desc->num_entries; vs->ap = desc->b_vidmem ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + vs->stamp = jiffies; /* A new directory invalidates what we learned from the old one. */ memset(svc.lat, 0, sizeof(svc.lat)); } @@ -1444,8 +2009,18 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, return -ENOENT; } - /* PD0: one dual entry per index (big half then small half). */ + /* + * PD0: one dual entry per index (big half then small half). + * Geometry per the driver's own GMMU format (gp10x family): + * PD3 [48:47], PD2 [46:38], PD1 [37:29], PD0 [28:21], PT + * [20:12]. PD0 is 8 bits - an earlier 0x1ff experiment here was + * reverted; the observed +2 MB client-page phase is NOT a PD0 + * index error. + */ idx = (va >> 21) & 0xff; + if (trace) + pr_info("sema-walk: PD0 table=0x%llx idx=%u va=0x%llx tap=%d\n", + table, idx, va, tap); if (svc_read_ap(vgpu, table + 16 * idx, tap == SVC_APERTURE_FB ? SVC_APERTURE_FB : SVC_APERTURE_SYS, @@ -1455,6 +2030,9 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, : SVC_APERTURE_SYS, &small, sizeof(small))) return -ENOENT; + if (trace) + pr_info("sema-walk: PD0 big=0x%016llx small=0x%016llx\n", + big, small); /* Small half: 4 KB page table under it. */ sap = svc_pde_aperture(((small >> 1) & 0x3)); @@ -1466,6 +2044,12 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, *addr += va & 0xfff; return 0; } + if (trace) + pr_info("sema-walk: small miss sap=%d t=0x%llx\n", + sap, t); + } else if (trace) { + pr_info("sema-walk: small skip sap=%d small&1=%llu\n", + sap, small & 1); } /* Big half: a 2 MB page here, or a large (64 KB) page table. */ @@ -1498,6 +2082,9 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, return 0; } } + if (trace) + pr_info("sema-walk: no leaf va=0x%llx big=0x%llx small=0x%llx\n", + va, big, small); return -ENOENT; } @@ -1526,6 +2113,26 @@ static int svc_walk_vaspaces(struct nv_vgpu_device *vgpu, u64 va, } spin_unlock_irqrestore(&svc.vas_lock, flags); + /* + * r459: try the NEWEST registration first. A dead client's stale + * directory (the table outlives clients; only 8 slots) can still + * resolve a VA - through guest memory the driver has since reused - + * to a wrong page, and the first success wins. Ordering by the + * registration stamp makes the live client's directory the + * authority and only falls through to older ones when it cannot + * answer. + */ + for (i = 1; i < n; i++) { + struct svc_vaspace k = local[i]; + int j = i - 1; + + while (j >= 0 && time_before(local[j].stamp, k.stamp)) { + local[j + 1] = local[j]; + j--; + } + local[j + 1] = k; + } + for (i = 0; i < n; i++) { int alt; @@ -1651,8 +2258,23 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) } if (n < SVC_PD_MIN_HITS) { n = 0; - if (svc.map_n < SVC_PD_SAMPLES) - return; /* no evidence yet: retry when pairs appear */ + /* + * r510: a LARGE map is a mixed set of VA spaces cached over + * the whole guest boot (the comment above already names the + * consequence: no single directory reproduces a mixed set), so + * scanning with its stride subsample burns the attempts budget + * on evidence that cannot succeed - and by the time the + * push-buffer pairs (the strongest samples) exist, the budget + * is spent (r508/r509 witness: 'not found (base=0x0 ap=ram + * cands=39621 best=0/8)' during the first client, eight RAM + * walks consumed, every later client's high argument pointers + * unresolvable - the terminal class). Only a SMALL map, which + * is one session's own pairs, still falls back; otherwise wait + * for the push-buffer pairs and keep the attempts. + */ + if (svc.map_n > (u32)(SVC_PD_SAMPLES * 2) || + svc.map_n < SVC_PD_SAMPLES) + return; /* no coherent evidence yet: retry later */ step = svc.map_n / SVC_PD_SAMPLES; for (i = 0; i < SVC_PD_SAMPLES; i++) { sva[i] = svc.map[i * step].gpu_page; @@ -1725,13 +2347,133 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) svc_pd_cands, svc_pd_best, n); } +/* + * m177c: a SECOND discovered-directory slot for the UVM SYS VA space (the + * 0x2xxxxxxx family: GR rings/pushes, QMD windows, counter pages). The + * primary slot (above) is validated against rm_mem-content pairs - the + * 0x1xxxxxxx pool family - and cannot walk the UVM space, so without pins + * the counter page (0x20400fXXX) never resolved: the guest's KMD/UVM + * epoch waited on completions that could never be written and re-ran its + * ~103 s retry loop forever (r194/r195: 'rel-apply ... slot pending' x8, + * then silence). Samples for this slot are content-VERIFIED + * (push_va -> push_pa) pairs from the GR content discovery and the CE + * push pairing: the push bytes themselves are the proof, and they live in + * exactly the space this directory maps. + */ +#define SVC_PD_RING_SAMPLES 8 +static u64 svc_pd_r_v[SVC_PD_RING_SAMPLES]; +static u64 svc_pd_r_pa[SVC_PD_RING_SAMPLES]; +static u32 svc_pd_r_n; +static u64 svc_pd_r_base; +static int svc_pd_r_ap; +static u8 svc_pd_r_found; +static u8 svc_pd_r_attempts; +static u64 svc_pd_r_ev_hash; + +static void svc_pd_ring_note(u64 va, u64 pa) +{ + u32 i; + + if (!va || !pa) + return; + va &= ~0xfffULL; + pa &= ~0xfffULL; + for (i = 0; i < svc_pd_r_n; i++) + if (svc_pd_r_v[i] == va) + return; + if (svc_pd_r_n < SVC_PD_RING_SAMPLES) { + svc_pd_r_v[svc_pd_r_n] = va; + svc_pd_r_pa[svc_pd_r_n] = pa; + svc_pd_r_n++; + } +} + +static void svc_pd_ring_discover(struct nv_vgpu_device *vgpu) +{ + u64 chunk = 64 * 1024; + u8 *buf; + u64 gpa; + u64 ev = svc_pd_r_n; + u32 i; + + if (svc_pd_r_found || svc_pd_r_attempts >= SVC_PD_ATTEMPTS_MAX) + return; + if (svc_pd_r_n < SVC_PD_MIN_HITS) + return; + for (i = 0; i < svc_pd_r_n; i++) + ev = (ev + svc_pd_r_v[i] + (svc_pd_r_pa[i] << 12)) * 1000003; + if (svc_pd_r_attempts && ev == svc_pd_r_ev_hash) + return; + svc_pd_r_ev_hash = ev; + svc_pd_r_attempts++; + + buf = kmalloc(chunk, GFP_KERNEL); + if (!buf) + return; + for (gpa = 0; gpa < SEMA_PA_MAX && !svc_pd_r_found; gpa += chunk) { + u32 len = (u32)min_t(u64, chunk, SEMA_PA_MAX - gpa); + u64 off; + + if (svc_read(vgpu, gpa, buf, len)) + continue; + for (off = 0; off + 8 <= len && !svc_pd_r_found; + off += SEMA_PAGE_SIZE) { + u64 e; + u64 a = 0; + int eap = 0; + u32 hits = 1; + + memcpy(&e, buf + off, sizeof(e)); + if ((e & 1) || !((e >> 1) & 0x3)) + continue; + /* First sample rejects almost every candidate. */ + if (svc_gmmu_walk(vgpu, gpa + off, 4, svc_pd_r_v[0], + SVC_APERTURE_SYS, &a, &eap, false) || + (a & ~0xfffULL) != svc_pd_r_pa[0]) + continue; + for (i = 1; i < svc_pd_r_n; i++) { + if (!svc_gmmu_walk(vgpu, gpa + off, 4, + svc_pd_r_v[i], + SVC_APERTURE_SYS, &a, &eap, + false) && + (a & ~0xfffULL) == svc_pd_r_pa[i]) + hits++; + } + if (hits >= SVC_PD_MIN_HITS) { + svc_pd_r_base = gpa + off; + svc_pd_r_ap = SVC_APERTURE_SYS; + svc_pd_r_found = 1; + pr_info("sema-svc: discovered UVM ring-space page directory 0x%llx (ram), %u/%u content-verified samples\n", + svc_pd_r_base, hits, svc_pd_r_n); + } + } + } + kfree(buf); +} + static int svc_walk_discovered(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, int *ap) { - if (!svc_pd_found) - return -ENOENT; - return svc_gmmu_walk(vgpu, svc_pd_base, svc_pd_entries, va, - svc_pd_ap, addr, ap, false); + int rc = -ENOENT; + + if (svc_pd_found) + rc = svc_gmmu_walk(vgpu, svc_pd_base, svc_pd_entries, va, + svc_pd_ap, addr, ap, false); + if (rc && svc_pd_r_found) + rc = svc_gmmu_walk(vgpu, svc_pd_r_base, 4, va, + svc_pd_r_ap, addr, ap, false); + /* + * m177c: the ring-space directory discovery runs on new samples + * only (fingerprint + attempt cap inside). A first call with + * enough verified ring/push samples performs one bounded scan; + * after that this is a flag check. + */ + if (rc && !svc_pd_r_found && svc_pd_r_n >= SVC_PD_MIN_HITS) + svc_pd_ring_discover(vgpu); + if (rc && svc_pd_r_found) + rc = svc_gmmu_walk(vgpu, svc_pd_r_base, 4, va, + svc_pd_r_ap, addr, ap, false); + return rc; } /* Learned translations: a walk costs a handful of guest reads, so cache the @@ -2495,6 +3237,29 @@ static inline bool svc_pool_hold_active(void) } /* + * m165 (D-F2a): the blanket pool hold exists to stop a release decoded + * before the guest's own injection lands from being applied through a + * stale translation. Once a host-verified injected pin for THIS VA is + * present, that premise is gone: the pin source is the resolver's first + * choice (read-back proven) and the ipin table is wiped at session flush, + * so it is this session's ground truth. r156 measured every pool release + * waiting the full registration+5.00 s with the pin ready all along. + */ +static bool svc_pool_hold_active_va(u64 va) +{ + u64 ipa = 0; + int iap = 0; + + if (!svc_pool_hold_active()) + return false; + if (!gsp_sema_pool_hold_fresh) + return true; + if (svc_ipin_lookup(va, &ipa, &iap) == 0) + return false; + return true; +} + +/* * Verified pool-semaphore translations, keyed by page VA. An entry is only * created from an injected pin or from a targeted rm_mem resolve whose page * was read back successfully; the learned cache and the discovered map are @@ -2518,6 +3283,16 @@ static u32 svc_pool_pa_n; static u32 svc_pool_pa_next; static u64 svc_pool_pa_logged[SVC_POOL_PA_LOG_MAX]; static u32 svc_pool_pa_logged_n; +/* + * fail4: the completion-hint window was 10 s. It outlived short-lived + * clients: after a client freed its RM events the window kept re-posting + * hint events every 0.5 s, and a post already queued in the status ring + * could be read after the guest's own free processing - the guest ISR + * then ran a freed RC callback and the guest panicked in interrupt + * (rm_run_rc_callback+0x0). 1.5 s still covers a waiter that arrives + * after the edge while shrinking the post-free exposure ~7x. + */ +#define SVC_POST_HINT_MS 1500 static u64 svc_pool_pa_miss_log; static u32 svc_pool_pa_hit_log; static u32 svc_pool_wr_log; @@ -2566,6 +3341,14 @@ static void svc_pool_wr_note(u32 h, u64 va, u64 pa, const char *src, } } +/* r372: landing-table entries repaired from a fresh pin (stale-ledger class). */ +static u64 svc_ledger_pin_repairs; +/* r372: pool alt fan-outs skipped because the primary pin moved on. */ +static u64 svc_pool_alt_skipped; +/* r372: bounded page-note resolution diagnostics (va/pa/ap/branch). */ +static u32 svc_gr_pagenote_log; +static u32 svc_fbhpa_log; + /* * Fan a pool release out to the last displaced translation of the same page * (see svc_ipin_alt): the primary target may be the stale one when two host @@ -2589,6 +3372,27 @@ static void svc_pool_alt_apply(struct nv_vgpu_device *vgpu, u32 h, u64 va, return; if ((pa & ~(u64)(SEMA_PAGE_SIZE - 1)) == page) return; /* never write an identity translation */ + /* + * r372 (stale-ledger corruption class): fan out ONLY while the + * current primary pin for this VA still names the displaced page. + * Once a newer pin has displaced it, the old page belongs to a + * buffer the guest has freed - writing it lands on whatever the + * guest kernel reused the page for. A missing current pin is + * equally unprovable: skip. The primary release write already + * succeeded either way, so this is a lost best-effort, not a + * failure. + */ + { + u64 cpa = 0; + int cap = SVC_APERTURE_SYS; + + if (svc_ipin_lookup(va, &cpa, &cap) || + ((cpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + svc_pool_alt_skipped++; + return; + } + } if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &cur, sizeof(cur))) return; if (cur >= val) @@ -2621,6 +3425,12 @@ static inline bool svc_pool_pa_identity(u64 va, u64 pa) (va & ~(u64)(SEMA_PAGE_SIZE - 1)); } +static void svc_pool_boot_store(u64 va, u64 pa, int ap); /* m168 */ +static bool svc_dst_guard_hold(u64 va); /* m169b */ +static bool svc_dst_guard_pending(u64 va); /* r231 */ +static u32 svc_read_order_log; /* r231 budget */ +static u32 svc_atrace_log; /* r234 A-TRACE budget */ + static int svc_pool_pa_lookup(u64 va, u64 *pa, int *ap) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); @@ -2651,6 +3461,8 @@ static void svc_pool_pa_store(u64 va, u64 pa, int ap) /* Never cache an identity translation (see svc_pool_pa_identity). */ if (svc_pool_pa_identity(page, pa)) return; + /* m168: every proven resolution also feeds the boot-scoped cache. */ + svc_pool_boot_store(va, pa, ap); for (i = 0; i < svc_pool_pa_n; i++) { if (svc_pool_pa[i].va_page == page) { svc_pool_pa[i].pa_page = @@ -2680,6 +3492,97 @@ static void svc_pool_pa_forget(u64 va) } /* + * m168 (D-F2c): boot-scoped verified pool-page cache. Within one guest boot + * the UVM semaphore pool (gpu->semaphore_pool, a 4 KB sysmem rm_mem at + * 0x121000000 whose 1024 words are the per-channel tracker slots) belongs + * to the shared GPU object; clients of the same boot reuse the pages while + * their per-process vaspaces come and go. The session flush wipes + * svc_pool_pa deliberately (a VA may name a new allocation for the next + * client), but wiping the POOL page forces every new client through the + * slow resolver: r170's clients 4-6 starved 2+ minutes on + * "pending release starved ... pool/waiter page" while the scanner learned + * the live tracker sems. This cache survives the flush; it is filled ONLY + * from read-back-proven resolutions (pin/cache/walk/vaspaces/targeted) and + * every use re-proves the exact offset readable, so a page that moved + * fails the proof and is dropped. A fresh injected pin always overwrites + * it (pins are the resolver's first source and run before this cache). + */ +struct svc_pool_boot_pa { + u64 va_page; + u64 pa_page; + int ap; +}; + +static struct svc_pool_boot_pa svc_pool_boot[SVC_POOL_PA_MAX]; +static u32 svc_pool_boot_n; +static u32 svc_pool_boot_next; +static u32 svc_pool_boot_conflicts; +static u32 svc_pool_boot_hits; + +static int svc_pool_boot_lookup(u64 va, u64 *pa, int *ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_pool_boot_n; i++) { + if (svc_pool_boot[i].va_page == page) { + if (svc_pool_pa_identity(page, + svc_pool_boot[i].pa_page)) + return -ENOENT; + *pa = svc_pool_boot[i].pa_page | + (va & (SEMA_PAGE_SIZE - 1)); + *ap = svc_pool_boot[i].ap; + return 0; + } + } + return -ENOENT; +} + +static void svc_pool_boot_store(u64 va, u64 pa, int ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + if (svc_pool_pa_identity(page, pa)) + return; + for (i = 0; i < svc_pool_boot_n; i++) { + if (svc_pool_boot[i].va_page == page) { + if (svc_pool_boot[i].pa_page != pa_page) { + /* The page moved (or a source lied): a + * fresh proof wins, but the conflict is on + * record. */ + svc_pool_boot_conflicts++; + pr_info("sema-svc: pool-boot conflict va=0x%llx cached=0x%llx new=0x%llx (replacing)\n", + page, svc_pool_boot[i].pa_page, + pa_page); + } + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; + return; + } + } + if (svc_pool_boot_n < SVC_POOL_PA_MAX) { + i = svc_pool_boot_n++; + } else { + i = svc_pool_boot_next++ % SVC_POOL_PA_MAX; + } + svc_pool_boot[i].va_page = page; + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; +} + +static void svc_pool_boot_forget(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_pool_boot_n; i++) + if (svc_pool_boot[i].va_page == page) + svc_pool_boot[i].va_page = 0; +} + +/* * Resolve one pool-semaphore target the way the successful deferred apply did: * injected pin (host-verified ground truth), the verified cache, then the * targeted rm_mem record scan, whose result is read back before it is cached. @@ -2721,10 +3624,34 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, if (src) *src = "pin"; svc_pool_pa_note(h, va, pa, "pin"); + svc_pool_boot_store(va, pa, a); + *addr = pa; + *ap = a; + return 0; + } + } + /* + * m168: boot-scoped page, survives the session flush. Every use + * re-proves the exact offset readable: a page that moved fails the + * proof and is dropped (the fresh pin path above always wins when a + * pin exists). + */ + if (!svc_pool_boot_lookup(va, &pa, &a)) { + u32 bprobe = 0; + + if (!svc_read_ap(vgpu, pa, svc_ap_of(a), &bprobe, + sizeof(bprobe))) { + svc_tr_clear_failure(page); + svc_pool_boot_hits++; + if (src) + *src = "bootcache"; *addr = pa; *ap = a; return 0; } + pr_info("sema-svc: pool-boot stale va=0x%llx pa=0x%llx (read-back failed; dropping)\n", + va, pa); + svc_pool_boot_forget(va); } if (!svc_pool_pa_lookup(va, &pa, &a)) { svc_tr_clear_failure(page); @@ -2900,6 +3827,17 @@ static int svc_sem_read(struct nv_vgpu_device *vgpu, u64 va, u32 *val) } static void svc_pending_add(u64 va, u32 payload, u32 op); +/* r34x: defined with the pending ring / opdef slots (later in the file); + * the acquire gate drains/appraises withheld releases before it holds. */ +static void svc_apply_pending(struct nv_vgpu_device *vgpu); +static bool svc_wait_pend_release(u64 va); + +/* + * F1: GR release-word authorship checks. Defined next to the svc_fwd_req + * slot; forward-declared because the sweep writer runs earlier in the file. + */ +static bool svc_gr_rel_match(u64 va, u32 *payload); +static bool svc_gr_rel_inflight(u64 va, u32 *payload); static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) { @@ -3285,6 +4223,133 @@ static int svc_map_lookup_range(u64 gpu_va, u32 from, u32 to, u64 *pa) static void svc_session_flush(void); /* defined next to the pending queue */ +/* + * Ledger sizing + counters hoisted above svc_rebuild so the periodic diag + * can report the table's fill state: a full table degrades the write/read + * authority silently (home replacement evicts another VA's landing), which + * is exactly the D2H "reads back prior content" window. + */ +#define SVC_LEDGER_MAX (1u << 14) +#define SVC_LEDGER_MASK (SVC_LEDGER_MAX - 1u) +#define SVC_LEDGER_PROBE 16u +/* + * STALLFREE: eviction burst threshold for the cap-and-degrade reset. A + * cluster keeps 16 same-home VA pages; EXIT_RESET evictions inside one session + * means the table is thrashing (pin churn / leftover crashed clients), and + * the half-evicted table silently strands the evicted pages' landing + * authority. At the threshold the cache is cleared and rebuilt from live + * writes (uniform resolver fallback instead of silent partial authority). + */ +#define SVC_LEDGER_EVICT_RESET 256u +static u32 svc_ledger_n; +static u64 svc_ledger_diverges; +static u64 svc_ledger_pinwins; /* r336: pin-backed divergence wins */ +static u32 svc_d2hmap_log; /* D2H-MAP per-page destination budget */ +/* + * r336/r34x: host-semaphore acquire (C56F SEM_EXECUTE op 0/2/3/4/5) wait + * state. The decoder captures EVERY acquire of the ring entry currently + * being decoded (svc_waits[], reset per entry at the top of + * svc_decode_stream), and svc_run_ops honors them before running any of + * the entry's work, so a copy cannot execute ahead of the dependency its + * push waits on (the streams2 readback-before-scale class). Per-entry + * capture is essential: the wait state used to leak across entries (a + * later launch's self-release wait, e.g. need=85 while the held push + * releases 84, gated an earlier entry that carries no acquire at all - + * the r340 over-hold). The re-decode re-parses the entry, so the state + * is refreshed on every pass. + * + * Semantics per op (class contract, see svc_wait_satisfied): + * 0 ACQUIRE = (v == p); 2 ACQ_STRICT_GEQ = v >= p; 3 ACQ_CIRC_GEQ = + * circular (s32)(v - p) >= 0; 4 ACQ_AND = (v & p) != 0; + * 5 ACQ_NOR = ~(v | p) != 0. + */ +#define SVC_WAIT_MAX 8 +/* + * r401: bounded acquire wait (fail-open), now a module parameter. The + * original 100 ms bound was shorter than a single GR launch's completion + * latency in this emulation (observed 0.5-1.6 s per launch), so a copy + * that waited on a launch's tracker (streams2: the D2H readback's + * ACQ_CIRC_GEQ on the scale channel's progress counter) gave up and ran + * BEFORE the launch wrote its output - the host read pre-scale data + * (bad=32767, c2 all wrong). 4 s covers several launch latencies; the + * bound still guarantees the entry is never parked forever (a genuinely + * unsatisfiable acquire fails open late, after the dependency's own work + * would have landed - late-but-correct instead of early-and-wrong). + */ +unsigned gsp_sema_wait_hold_ms = 4000; +struct svc_wait_op { + u64 va; + u32 payload; + u8 op; /* C56F SEM_EXECUTE operation */ + u32 off; /* method offset in the entry (order) */ +}; +static struct svc_wait_op svc_waits[SVC_WAIT_MAX]; +static u32 svc_wait_n; /* acquires captured this entry */ +/* + * r484 THE STAGING-DRAIN COMPLETION EDGE: when the acquire gate holds an + * entry, the op can still have executed on an earlier decode whose gate did + * not trigger (svc_wait_n was 0 on that pass), and the once-only done + * bitmap then suppresses it on the first pass whose gate finally passes - + * the pass at which the dependency just became installed and the op's + * inputs changed. The release then applies and the guest drains a staging + * buffer holding the PRE-dependency bytes (streams_events s=2: the D2H + * copy read dc2 before the vadd_into wrote it, ran early on gate-less + * passes, was once-only-skipped on the passing pass, and the client + * drained a2+b2 instead of a2+b2+c1). Remember the last held (channel, + * entry) and force exactly ONE re-run of its ops on the first passing + * pass. + */ +static u32 svc_wait_hold_h; +static u32 svc_wait_hold_idx = 0xffffffffu; +static bool svc_wait_repass; +static u32 svc_wait_ovf; /* acquires dropped past the array */ +static u64 svc_wait_held; +static u64 svc_wait_pass; +static u64 svc_wait_giveup; +static u32 svc_wait_log; /* per-session held-row print budget */ +static u32 svc_wait_pass_log; /* per-session resolved-row budget */ +static u32 svc_remapconst_log; /* r343 FILL: SET_REMAP_CONST_A rows */ +static u32 svc_pitch_log; /* r402: 2D (pitched) op builds */ +static u32 svc_fill_raw_log; /* r365 FILL-RAW: raw dropped-fill dumps */ +static u32 svc_lowfill_log; /* r365b LOWFILL-WIT: below-floor fills */ +static u32 svc_tabcopy_log; /* r367 TABLE-COPY: table-page copies */ +static u32 svc_fill_noremap; /* r343 FILL: fail-closed holds */ +static u64 svc_wait_rel_va; /* this entry's own release (mirror) */ +static u32 svc_wait_rel_payload; +static u32 svc_wait_rel_off; /* its method offset (order check) */ +static bool svc_wait_rel_have; +/* + * Per-channel first-hold clock for the bounded wait: a gate evaluation + * that cannot satisfy the acquire arms a 100 ms budget per hold episode; + * expiry fails open (ops run) with a loud 'give-up' row. The table is + * tiny (a handful of channels), service-thread only, cleared per pass + * resolution / session flush. + */ +struct svc_wait_hold { + bool active; + u32 h; + unsigned long since; +}; +#define SVC_WAIT_HOLD_MAX 16 +static struct svc_wait_hold svc_wait_holds[SVC_WAIT_HOLD_MAX]; +static u32 svc_dst_div_log; /* r402: copy-dst ledger/probe divergence */ +static u64 svc_ledger_stores; +static u64 svc_ledger_evictions; +static u64 svc_ledger_evict_since_reset; +static u64 svc_ledger_evict_resets; +static u64 svc_ledger_rewrites; +static u64 svc_ledger_reloc_diverges; +static u32 svc_ledger_store_log; +/* + * STALLFREE: pending-ring overflow telemetry (the negative queue silently + * overwrote its oldest entry when full). Hoisted here so svc_rebuild's diag + * can print it; the ring itself is defined with the pending queue below. + */ +static u64 svc_pending_ovf; +static u32 svc_pending_ovf_log; +/* STALLFREE: rescue scans that hit their wall-clock cap (bounded service). */ +static u64 svc_rescue_scan_capped; + static void svc_rebuild(struct nv_vgpu_device *vgpu) { static u8 *buf; @@ -3387,18 +4452,23 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) svc.false_trans, svc.gpfifo_skipped, svc.gpfifo_rescued, svc.releases_faked, svc.releases_recovered, svc.counter_rel_applied); - pr_info("sema-svc: drain held=%llu starved=%llu forwarded=%llu faked=%llu rec=%llu ops_once=%llu reident=%llu opdef=%llu/%llu/%llu\n", + pr_info("sema-svc: drain held=%llu starved=%llu forwarded=%llu faked=%llu rec=%llu ops_once=%llu reident=%llu opdef=%llu/%llu/%llu gr_nocap=%llu/%llu gr_hold_exp=%llu gr_swrel=%llu gr_wedge=%llu pend_ovf=%llu\n", svc.gpfifo_held, svc.gpfifo_starved, svc.releases_forwarded, svc.releases_faked, svc.releases_recovered, svc.ops_once_skipped, svc.ops_reident, - svc.opdef_held, svc.opdef_applied, svc.opdef_expired); - pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu altw=%llu residue=%llu\n", + svc.opdef_held, svc.opdef_applied, svc.opdef_expired, + svc.gr_nocap_held, svc.gr_nocap_consumed, + svc.fwd_gr_hold_exp, svc.fwd_gr_stall_swrel, + svc.gr_stale_wedge, svc_pending_ovf); + pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu altw=%llu residue=%llu wait=%llu/%llu giveup=%llu ovf=%u\n", svc.counter_pub_calls, svc.gr_rel_applied, svc.sem_write_reresolve, svc.pushpg_fallback, - svc.pool_alt_applied, svc.pool_residue); - if (svc.resc_run) - pr_info("sema-svc: rescue_run=%llu (multi-page content pins)\n", - svc.resc_run); + svc.pool_alt_applied, svc.pool_residue, + svc_wait_held, svc_wait_pass, svc_wait_giveup, + svc_wait_ovf); + if (svc.resc_run || svc_rescue_scan_capped) + pr_info("sema-svc: rescue_run=%llu scan_capped=%llu (multi-page content pins)\n", + svc.resc_run, svc_rescue_scan_capped); pr_info("sema-svc: gpfifo done=%llu nomap=%llu vidmem_writes=%llu ring_auto=%u pb=%u/%u pins=%u/%llu ipins=%u/%llu\n", svc.gpfifo_entries_done, svc.gpfifo_nomap, svc.vidmem_writes, svc_ring_n, svc_pb_n, svc_pb_passes, @@ -3407,6 +4477,19 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) svc_vas_n, SVC_VAS_SLOTS, svc.vas_writes, svc.vas_reads, svc.vas_shadowed); /* + * D2H-source authority visibility (r220): a full ledger degrades to + * home-slot replacement, which silently evicts another VA's landing + * and lets readers fall back to the resolver - the "reads back prior + * content" window. n approaching SVC_LEDGER_MAX or evict > 0 in a + * client's window is the tell. + */ + pr_info("sema-svc: ledger n=%u/%u stores=%llu evict=%llu resets=%llu diverges=%llu pinwins=%llu reloc_div=%llu rewrites=%llu pinrep=%llu alt_skip=%llu\n", + svc_ledger_n, SVC_LEDGER_MAX, svc_ledger_stores, + svc_ledger_evictions, svc_ledger_evict_resets, + svc_ledger_diverges, svc_ledger_pinwins, + svc_ledger_reloc_diverges, svc_ledger_rewrites, + svc_ledger_pin_repairs, svc_pool_alt_skipped); + /* * P6b wave 1: guest-forwarding summary. Printed whenever any counter * moved (not only on requests) so a run that captured no candidate is * still diagnosable from the log. @@ -3464,6 +4547,9 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) #define CE_M_OFFSET_OUT_UPPER 0x408 #define CE_M_OFFSET_OUT_LOWER 0x40c #define CE_M_LINE_LENGTH_IN 0x418 +#define CE_M_LINE_COUNT 0x41c +#define CE_M_PITCH_IN 0x410 +#define CE_M_PITCH_OUT 0x414 #define CE_M_SET_REMAP_CONST_A 0x700 #define CE_M_SET_REMAP_CONST_B 0x704 #define CE_M_SET_REMAP_COMPONENTS 0x708 @@ -3538,6 +4624,19 @@ struct svc_memset_op { u64 bytes; /* copy length */ u32 src_phys; u32 src_aperture; + /* + * r402: 2D (pitched) transfers. rows > 1 selects row-wise mode: + * the op runs `rows` times with the destination advancing by + * dpitch (and the source by spitch) per row, each row carrying + * `bytes` (copy) or elem*count (fill) bytes. cudaMemset2D and + * cudaMemcpy2D issue exactly this shape (LINE_COUNT rows, + * PITCH_OUT/PITCH_IN in bytes); without it the fill/copy ran flat + * and the partial-width rows corrupted the neighbouring columns + * (victory_memset2d bad>12M, memcpy_roundtrip pitched class). + */ + u32 rows; + u32 dpitch; + u32 spitch; }; /* Decoder state carried across the methods of one push. */ @@ -3549,9 +4648,19 @@ struct svc_push_state { u32 have_in, have_out; u32 line_length; u32 gr_line_count; /* GRCE LINE_COUNT */ + /* + * r402: 2D transfer geometry (CE class): PITCH_IN/PITCH_OUT in + * bytes and LINE_COUNT rows. A pitched fill/copy whose pitch + * exceeds the row length must run row-wise; a flat transfer leaves + * pitch == line_length (or no pitch method at all). + */ + u32 pitch_in, pitch_out; + u32 have_pitch_in, have_pitch_out; + u32 line_count; u32 sem_va_hi, sem_va_lo, sem_payload; u32 have_sem; u32 have_remap; + u32 have_remap_a, have_remap_b, have_remap_c; u32 release_dma; u32 sem_inc; u32 rs_upper, rs_lower, rs_payload; @@ -3682,7 +4791,18 @@ static int svc_op_translate(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, * withholding the release corrupted the next session (r59: the late * memset overwrote refilled data after the client had passed the * release), so the two must move together. + * + * r383 (streams2 big-copy class): name the missed VA through the + * same stalled-VA demand channel the GR fill uses. The userspace pin + * stream watches the "stalled VA" row and pushes the VA to the + * in-guest scanner, so the pin lands on the next scanner pass + * instead of waiting out a full scan cadence. The hold itself is + * unchanged; this only shortens the window in which the entry sits + * op-pending (r377: a held H2D copy executed ~1.5 s late and read a + * staging page the client had since refilled with the next copy's + * bytes - the neighbor-data class). */ + svc_log_stalled_va(va & ~(u64)(SEMA_PAGE_SIZE - 1)); svc.op_tr_pending = 1; return -ENOENT; } @@ -3707,19 +4827,281 @@ static bool svc_dst_watched(u64 dst) } /* + * COPYAUDIT shared budget (see the svc_copy comment for the contract): + * every copy at/above SVC_COPY_AUDIT_MIN is logged once with want/done and + * gets a first/last-16-bytes destination readback vs the written bytes + * (COPY-MISMATCH). One budget covers every copy path in this file so a + * whole cycle stays visible. + */ +#define SVC_COPY_AUDIT_MIN (32u << 10) /* >32 KB copies audited */ +#define SVC_COPY_AUDIT_MAX 2048u /* per-boot line budget */ +#define SVC_CEAUDIT_MAX 4096u /* per-boot CEAUDIT budget */ +static u64 svc_copy_audit_n; +static u64 svc_ceaudit_n; + +/* COPYAUDIT for a finished fill: log want/done and check the first 16 + * destination bytes against the repeated pattern. The readback uses the + * accessor matching the aperture the fill wrote through. */ +static void svc_fill_audit(struct nv_vgpu_device *vgpu, u64 va0, u64 addr, + u32 aperture, size_t total, const u8 *pat, u32 elem) +{ + u8 exp[16], got[16] = {0}; + size_t i; + bool ok = false; + + if (total < SVC_COPY_AUDIT_MIN || svc_copy_audit_n >= SVC_COPY_AUDIT_MAX) + return; + /* + * The fill writes the 8-byte pattern group repeatedly (pat is the + * element zero-padded to 8 bytes), so the expectation mirrors that: + * byte i of the destination equals pat[i % 8]. + */ + for (i = 0; i < sizeof(exp); i++) + exp[i] = pat[i % 8]; + if (aperture == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + + if (base && addr + sizeof(got) <= svc_fb_shadow_size(vgpu)) { + memcpy(got, base + addr, sizeof(got)); + ok = true; + } + } else if (!svc_read(vgpu, addr, got, sizeof(got))) { + ok = true; + } + svc_copy_audit_n++; + pr_info("vgsp: COPYAUDIT path=sema-fill va=0x%llx len=0x%llx done=0x%llx\n", + va0, (u64)total, (u64)total); + if (ok && memcmp(exp, got, sizeof(exp))) + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x0 path=sema-fill want=%*phN got=%*phN\n", + va0, 16, exp, 16, got); +} + +/* * Write `count` elements of `value` (element size `elem`) starting at * `addr`, in vidmem or guest RAM depending on the aperture DST_TYPE * selected. Returns 0 on success. */ +static void svc_ledger_store(u64 va, u64 pa, u32 ap); /* R2 ledger, defined below */ +static int svc_ledger_lookup(u64 va, u64 *pa, u32 *ap); /* R2 ledger, defined below */ +static void svc_ledger_clear(void); /* STALLFREE: cap-and-degrade reset */ + +/* + * r384 (the guest boot-corruption lottery, root-caused 2026-10-20): + * page-table pages of the guest kernel's direct map were found ZEROED + * (two adjacent 4 KiB PTE pages at gpa 0x12d3c2000/0x3000 => two 2 MiB + * holes in the guest physmap, kernel objects inside them unreachable, + * multi-oops panic). The only emulation writers of page-sized ZERO runs + * are the virtual FILL executor (svc_fill / CE memsets), whose large + * >= SVC_VAS_FLOOR destinations resolve through the walk/scan when no + * pin and no ledger answer exists - and that resolver is demonstrably + * non-deterministic (four executions of the same 8 KiB engine-window + * clear resolved to four different guest pages, all refused by the + * below-floor guard). + * + * Two nets, mirroring the m169/r378 copy guards: + * 1. svc_fill_guard_hold(): a LARGE (>= 64 KiB) virtual fill whose only + * authority is the walk/scan (no pin, no ledger) is held through a + * short bounded window and its VA pushed to the pin-demand channel + * (the userspace pin stream injects the guest-pagemap truth), so the + * retried decode resolves through a real pin. Fail-open after the + * bound with the historical behaviour, like the copy guard. + * 2. svc_fill_target_is_pt(): a ZERO fill about to land in guest RAM on + * a page whose first 64 qwords are all PTE-shaped (present, NX, + * physical address in range) is REFUSED loudly. A page table page is + * never a legitimate zero-fill destination; this directly blocks the + * observed corruption shape even when some other resolver approved + * the target. Pin-backed resolutions are exempt (the guest's own + * pagemap is truth - if it says the VA maps the page, obey it). + */ +unsigned gsp_sema_fill_guard_ms = 800; +bool gsp_sema_fill_guard = true; +#define SVC_FILL_GUARD_SLOTS 16 +static struct { + u64 va_page; + unsigned long first; +} svc_fill_guard[SVC_FILL_GUARD_SLOTS]; +static u32 svc_fill_guard_next; +static u32 svc_fill_held; +static u32 svc_fill_pt_refused; +static u32 svc_fill_exec_log; /* r384 FILL-EXEC witness budget */ + +/* true = the guard may hold this fill; false = fail-open, run it. */ +static bool svc_fill_guard_hold(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i, slot = SVC_FILL_GUARD_SLOTS; + + for (i = 0; i < SVC_FILL_GUARD_SLOTS; i++) { + if (svc_fill_guard[i].va_page == page) { + if (time_before(now, svc_fill_guard[i].first + + msecs_to_jiffies( + gsp_sema_fill_guard_ms))) + return true; + if (svc_fill_guard[i].first) { + svc_fill_guard[i].first = 0; + pr_info("sema-svc: fill guard released without a pin va=0x%llx (using the walk translation)\n", + page); + } + return false; + } + if (!svc_fill_guard[i].va_page) + slot = i; + } + if (slot == SVC_FILL_GUARD_SLOTS) + slot = svc_fill_guard_next++ % SVC_FILL_GUARD_SLOTS; + svc_fill_guard[slot].va_page = page; + svc_fill_guard[slot].first = now; + return true; +} + +/* + * A guest-kernel page table page: the first SVC_PT_SCAN qwords are all + * present PTEs (bit 0), non-executable (bit 63 - kernel data mappings), + * with a physical address inside the 34-bit guest range. Random payload + * qwords pass all three conditions with probability far below the 75% + * threshold at SVC_PT_SCAN=64; a genuine table passes at 100%. + */ +#define SVC_PT_SCAN 64u +#define SVC_PT_MIN 48u +static bool svc_fill_target_is_pt(struct nv_vgpu_device *vgpu, u64 addr) +{ + u64 ents[SVC_PT_SCAN]; + u32 i, hits = 0; + + if (addr & (SEMA_PAGE_SIZE - 1)) + return false; /* only a page-aligned table shape */ + if (svc_read(vgpu, addr, ents, sizeof(ents))) + return false; + for (i = 0; i < SVC_PT_SCAN; i++) { + u64 e = ents[i]; + + if (!(e & 1ull)) + continue; + if (!(e & (1ull << 63))) + continue; + if ((e & 0x000ffffffffff000ull) == 0) + continue; + if ((e & 0x000ffffffffff000ull) >= (1ull << 34)) + continue; + hits++; + } + return hits >= SVC_PT_MIN; +} + +/* + * r391 per-page destination resolution for a VIRTUAL fill (the r390 + * guest-panic class). A device VA's pages are independent guest allocations + * - the pin stream shows split runs (va=0x7c36c7400000 -> 29 pages at + * 0x131b63000, the next run at 0x131a80000) - so resolving only the fill's + * first page and writing `total` bytes contiguously past it writes over + * whatever memory follows the first run. On r390 a 383620-byte zero fill + * did exactly that: the contiguous write covered the guest kernel's + * page-table pages (0x131ba1000/0x131ba2000/0x131bab000), the direct map + * grew holes, and the guest died with "Fatal exception in interrupt" + * (unmap_page_range / cpuacct / psi faults). svc_copy has always translated + * virtual operands per 4 KB page; the fill must do the same. + * + * Authority order mirrors the top-level resolution (and svc_copy's per-page + * dst): a fresh pin (guest pagemap truth) wins and repairs the landing + * table, then the landing table (the page a previous write of this VA + * landed on), then the resolver chain. A page with no resolution returns + * -ENOENT: the caller fails the fill loudly instead of writing through a + * contiguous guess. + */ +static int svc_fill_dst_page_pa(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out, u32 *prov_out) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0, lpa = 0; + u32 aper = 0, lap = 0; + + if (!svc_pin_lookup_any(page, &pa, &aper)) { + *prov_out = 1; + if (!svc_ledger_lookup(page, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != page) || + (u32)lap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill-page va=0x%llx ledger=0x%llx/%u pin=0x%llx/%u\n", + page, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, pa, aper); + svc_ledger_store(page, pa, aper); + } + *pa_out = pa; + *ap_out = aper; + return 0; + } + if (!svc_ledger_lookup(page, &pa, &aper)) { + *prov_out = 2; + *pa_out = pa; + *ap_out = aper; + return 0; + } + if (!svc_op_translate(vgpu, page, &pa, &aper)) { + *prov_out = 3; + *pa_out = pa; + *ap_out = aper; + return 0; + } + return -ENOENT; +} + +static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, + u32 elem, u32 count, u32 phys, u32 aperture, + u64 *done_out); +static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, + size_t total, u32 dst_phys, u32 dst_aperture, + u32 src_phys, u32 src_aperture, u64 *done_out, + u32 owner_ch, u32 owner_idx, u64 owner_hash); + +/* + * r402 row-wise wrapper: a 2D (pitched) fill runs the 1D fill once per + * row with the destination advancing by dpitch. Each row keeps the whole + * per-page resolution, ledger-store and guard machinery of the 1D path + * (no new resolution semantics), and a failed row's retry simply re-runs + * the already-written rows with the same pattern - idempotent for fills. + */ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, - u32 count, u32 phys, u32 aperture) + u32 count, u32 phys, u32 aperture, u64 *done_out, + u32 rows, u32 dpitch) +{ + if (rows > 1 && dpitch) { + u64 done = 0; + int rc = 0; + u32 r; + + for (r = 0; r < rows; r++) { + u64 row_done = 0; + + rc = svc_fill_1d(vgpu, addr + (u64)r * dpitch, + value, elem, count, phys, aperture, + &row_done); + done += row_done; + if (rc) + break; + } + if (done_out) + *done_out = done; + return rc; + } + return svc_fill_1d(vgpu, addr, value, elem, count, phys, aperture, + done_out); +} + +static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, + u32 count, u32 phys, u32 aperture, u64 *done_out) { u8 pat[8]; u8 bounce[256]; size_t total = (size_t)elem * count; size_t o; bool shadow = false; + u64 va0 = addr; /* original destination VA (fills, C2) */ + if (done_out) + *done_out = 0; if (elem != 1 && elem != 2 && elem != 4 && elem != 8) return -EINVAL; if (!count || total > (256ULL << 20)) @@ -3730,21 +5112,151 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, if (!phys) { /* Virtual destination: the VA shadow first (its pages stay - * authoritative), then the pinned map and the VA->PA map. */ + * authoritative), then a live pin (the guest's newest + * statement - r372: a re-pinned VA must not be written + * through a stale landing), then the unified landing table + * (the page a previous write of this VA landed on - always + * the authoritative location of the content), then the + * pinned map and the VA->PA map. */ u64 pa; u32 aper; + u32 prov = 0; /* r384: 1 pin, 2 ledger, 3 walk/scan */ if (svc_vas_has(addr)) { shadow = true; + } else if (!svc_pin_lookup_any(addr, &pa, &aper)) { + u64 lpa = 0; + u32 lap = 0; + + prov = 1; + /* + * r372 pin-wins: repair the landing table when it + * still names the pre-re-pin page, so every later + * reader follows the pin too. + */ + if (!svc_ledger_lookup(addr, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill-dst va=0x%llx ledger=0x%llx/%u pin=0x%llx/%u\n", + addr, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, pa, aper); + svc_ledger_store(addr, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper); + } + addr = pa; + aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (!svc_ledger_lookup(addr, &pa, &aper)) { + prov = 2; + addr = pa; + aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; } else if (svc_op_translate(vgpu, addr, &pa, &aper)) { if (!svc_vas_can_shadow(addr)) return -ENOENT; shadow = true; svc.vas_shadowed++; } else { + prov = 3; addr = pa; aperture = aper; } + /* + * r365b LOWFILL-WIT: a below-floor virtual fill (the guest's + * vidmem-window page-table writes) is a corruption risk: its + * resolution must be proven before the op is allowed to run + * (r365: 16-byte and 32 KB zero fills landed in guest RAM and + * killed the guest kernel). Log every resolution bounded. + */ + if (va0 < SVC_VAS_FLOOR && svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill: va=0x%llx shadow=%d resolved=0x%llx ap=%u elem=%u count=%u value=0x%llx\n", + va0, shadow, addr, aperture, elem, count, + value); + } + /* + * r369: a below-floor fill may ONLY land in vidmem. The + * 0x204020000 engine-window clear resolved to a DIFFERENT + * guest-RAM page on every execution (9 runs, 9 pages) and + * the guest died with its task_struct corrupted (gmain NULL + * deref in __fdget) - each wrong resolution zeroed 8 KB of + * guest kernel memory. Vidmem resolutions (the page-table + * region, ap=VID) are the emulation's own shadow and stay + * allowed; a SYS resolution for a window VA is refused and + * logged loudly. + */ + if (va0 < SVC_VAS_FLOOR && !shadow && + aperture != CE_APERTURE_VID) { + pr_err_ratelimited("vgsp-sema: LOWFILL-REFUSED va=0x%llx pa=0x%llx ap=%u elem=%u count=%u (below-floor fill resolved outside vidmem)\n", + va0, addr, aperture, elem, count); + return -ENOENT; + } + /* + * r384 net 1 (pin-demand hold for unproven large fills): a + * >= 64 KiB virtual fill whose only authority is the + * walk/scan resolver (no pin, no landing-table entry) AND + * which would land in GUEST RAM (SYS aperture) is the class + * that zeroed the guest's page-table pages (see the guard + * commentary above svc_fill_guard_hold). Demand the VA + * through the pin channel and hold the op through a short + * bounded window so the retried decode resolves through the + * guest's own pagemap; fail open with the historical + * behaviour after the bound. Vidmem-shadow resolutions are + * the emulation's own memory and stay immediate. + */ + if (gsp_sema_fill_guard && va0 >= SVC_VAS_FLOOR && !shadow && + total >= (64u << 10) && prov >= 3 && + aperture == CE_APERTURE_SYS) { + svc_log_stalled_va(va0); + if (svc_fill_guard_hold(va0)) { + svc_fill_held++; + if (svc_fill_held <= 32) + pr_info("sema-svc: FILL-HOLD va=0x%llx resolved=0x%llx ap=%u total=%zu value=0x%llx (unproven large fill; pin demanded)\n", + va0, addr, aperture, total, + value); + return -ENOENT; + } + } + /* + * r384 net 2 (page-table-shape refusal): a ZERO fill about + * to land in guest RAM on a page whose first 64 qwords are + * all present+NX+in-range PTEs is refused loudly - a page + * table page is never a legitimate zero-fill destination. + * This blocks the observed corruption shape (two adjacent + * PTE pages zeroed => two 2 MiB holes in the guest physmap) + * even when some other resolver approved the target, and + * irrespective of how wrong the resolution is. Pin-backed + * resolutions are exempt: the guest's own pagemap is truth + * there. + */ + if (gsp_sema_fill_guard && va0 >= SVC_VAS_FLOOR && !shadow && + value == 0 && total >= SEMA_PAGE_SIZE && + aperture == CE_APERTURE_SYS && prov != 1 && + svc_fill_target_is_pt(vgpu, addr)) { + svc_fill_pt_refused++; + if (svc_fill_pt_refused <= 32) + pr_err("vgsp-sema: FILL-PT-REFUSED va=0x%llx pa=0x%llx prov=%u total=%zu (zero fill target is a page table)\n", + va0, addr, prov, total); + svc_log_stalled_va(va0); + return -ENOENT; + } + /* + * r384 witness: where every >= 4 KiB virtual fill actually + * lands (va, pa, aperture, resolution provenance, pattern) - + * the decisive line for the boot-corruption forensics (the + * large zero fills were invisible until now). Bounded. + */ + if (va0 >= SVC_VAS_FLOOR && !shadow && + total >= SEMA_PAGE_SIZE && svc_fill_exec_log < 64) { + svc_fill_exec_log++; + pr_info("sema-svc: FILL-EXEC va=0x%llx pa=0x%llx ap=%u prov=%u total=%zu value=0x%llx\n", + va0, addr, aperture, prov, total, value); + } } if (gsp_sema_dbg && !shadow && svc_dst_watched(addr) && @@ -3759,10 +5271,40 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, svc.op_tr_src, elem, pb, svc.cur_ch); } - if (shadow) - return svc_vas_fill(addr, pat, elem, total); + if (shadow) { + int rc = svc_vas_fill(addr, pat, elem, total); - if (aperture == CE_APERTURE_VID) { + if (!rc && done_out) + *done_out = (u64)total; + return rc; + } + + /* + * C2 (data class) + m180 unification: a large virtual fill is a + * write/read pair with the memset's readers like any H2D copy - + * record its landing pages so the engine fill and the D2H read of + * this VA use exactly the page the fill wrote. Only the VID + * (shadow) case is contiguous; a SYS landing is recorded per page + * in the loop below (r391: the contiguous `addr + po` store also + * poisoned the table with pages the fill never wrote). + */ + if (!phys && !shadow && va0 >= SVC_VAS_FLOOR && + total >= (64u << 10) && aperture == CE_APERTURE_VID) { + u64 po; + + for (po = 0; po < total; po += SEMA_PAGE_SIZE) + svc_ledger_store(va0 + po, addr + po, + SVC_APERTURE_FB); + } + + /* + * r391: a virtual device destination is translated per 4 KB page + * (see svc_fill_dst_page_pa) - never written through the first + * page's PA contiguously. The whole-range shadow write below only + * applies to a resolution that is contiguous by construction. + */ + if (aperture == CE_APERTURE_VID && + !(va0 >= SVC_VAS_FLOOR && !phys && !shadow)) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); @@ -3772,6 +5314,9 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, memcpy(base + addr + o, pat, 8); if (o < total) memcpy(base + addr + o, pat, total - o); + if (done_out) + *done_out = (u64)total; + svc_fill_audit(vgpu, va0, addr, aperture, total, pat, elem); return 0; } @@ -3779,17 +5324,107 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, for (o = 0; o < total; ) { size_t chunk = total - o; size_t i; + u64 wpa = addr; + u32 wap = aperture; + bool per_page = !phys && !shadow && va0 >= SVC_VAS_FLOOR; if (chunk > sizeof(bounce)) chunk = sizeof(bounce); + /* + * r391: resolve THIS page's own landing (see + * svc_fill_dst_page_pa) and clamp the chunk to the page + * boundary - a virtual VA's pages are independent guest + * allocations and must never be written through the first + * page's PA contiguously. + */ + if (per_page) { + u32 off = (u32)((va0 + o) & (SEMA_PAGE_SIZE - 1)); + u64 ppa = 0; + u32 pap = 0, prov = 0; + int prc; + + chunk = min_t(size_t, chunk, SEMA_PAGE_SIZE - off); + prc = svc_fill_dst_page_pa(vgpu, va0 + o, &ppa, + &pap, &prov); + if (prc) { + if (done_out) + *done_out = (u64)o; + /* + * r391: demand the exact page through the pin + * channel (the userspace stream pushes this VA + * to the in-guest scanner) and hold the fill + * for a bounded retry - never write through a + * contiguous guess past a missing page. + */ + svc_log_stalled_va((va0 + o) & + ~(u64)(SEMA_PAGE_SIZE - 1)); + pr_err_ratelimited("vgsp: FILL-TRUNC va=0x%llx want=0x%llx done=0x%llx path=sema-fill (page 0x%llx unresolved)\n", + va0, (u64)total, (u64)o, + (va0 + o) & + ~(u64)(SEMA_PAGE_SIZE - 1)); + return -ENOENT; + } + wap = (pap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wpa = (ppa & ~(u64)(SEMA_PAGE_SIZE - 1)) + off; + /* + * r391 net: the r384 page-table-shape refusal + * applied to EVERY page of a virtual zero fill. + * The r390 fill's first page was the client's + * buffer, the later pages were the guest's PTE + * pages - the first-page-only check could not see + * them. A page-table-shaped guest-RAM page is never + * a legitimate zero-fill target: skip it loudly + * instead of zeroing it. Pin-backed resolutions + * are exempt (the guest's pagemap is truth there). + */ + if (wap == CE_APERTURE_SYS && value == 0 && prov != 1 && + svc_fill_target_is_pt(vgpu, + wpa & ~(u64)(SEMA_PAGE_SIZE - 1))) { + svc_fill_pt_refused++; + if (svc_fill_pt_refused <= 32) + pr_err("vgsp-sema: FILL-PT-REFUSED va=0x%llx pa=0x%llx prov=%u total=%zu (zero fill page is a page table; page skipped)\n", + va0 + o, wpa, prov, total); + o += chunk; + continue; + } + } for (i = 0; i + 8 <= chunk; i += 8) memcpy(bounce + i, pat, 8); if (i < chunk) memcpy(bounce + i, pat, chunk - i); - if (svc_write(vgpu, addr + o, bounce, chunk)) + if (wap == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = svc_fb_shadow_size(vgpu); + + if (!base || wpa + chunk > size) { + if (done_out) + *done_out = (u64)o; + return -EINVAL; + } + memcpy(base + wpa, bounce, chunk); + } else if (svc_write(vgpu, wpa, bounce, chunk)) { + if (done_out) + *done_out = (u64)o; + if (total >= SVC_COPY_AUDIT_MIN) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=sema-fill rc=%d\n", + va0, (u64)total, (u64)o, -EIO); return -EIO; + } + /* + * r391: record this page's landing (the readers follow the + * table) - SYS and VID alike, exactly like the copy path. + */ + if (per_page) + svc_ledger_store(va0 + o, + wpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (wap == CE_APERTURE_VID) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS); o += chunk; } + if (done_out) + *done_out = (u64)total; + svc_fill_audit(vgpu, va0, addr, aperture, total, pat, elem); return 0; } @@ -3844,13 +5479,23 @@ static struct svc_fwd_cand svc_fwd_cand; #define SVC_GR_POINTER_MAX 8 #define SVC_GR_PAGES_MAX 16 /* + * relfix PART 3 (fill candidate superset): maximum distinct candidate pages + * per collection pass. Raised from 16 because the full-page scans added + * below (aligned with the rewrite's own whole-page scan) can legitimately + * carry more pointer-shaped words than the argument region alone; the old + * cap silently dropped the extra words, which then resurfaced as RELOC-GAP + * at rewrite time. It remains a hard bound with the historical silent-drop + * semantics beyond it (n_cand is logged in the fill summary). + */ +#define SVC_GR_CAND_MAX 24u +/* * m99: guest data identity-map capacity. The forwarded kernel's global data * pages (discovered by scanning the param/const-0 page for device VAs) are - * resolved on the service thread and identity-mapped into the engine tree. - * 16K pages = 64 MB of window coverage, comfortably above the demo buffers - * (2 x 16 MB observed) with headroom. + * resolved on the service thread and relocated into the engine-visible + * region. B-2: 32K records = 2x the grown region's 16384 pages of headroom, + * so the hand-off array is never the binding limit. */ -#define SVC_GR_DATA_MAX 16384 +#define SVC_GR_DATA_MAX 32768 #define SVC_GR_PUSH_WORDS 256 #define SVC_GR_QMD_WORD_NONE 0xffffffffu /* svc_gr_cand.rel_flags bits (decoded from qmd[24] and qmd[4]). */ @@ -3894,9 +5539,18 @@ struct svc_gr_data_win { u64 reloc_base; /* relocated window base */ }; -struct svc_gr_cand { +struct svc_gr_cand_rec { bool valid; bool offered; /* consumed by a request or a fallback */ + /* + * STALLFREE: exactly-once latch for the candidate's QMD completion + * release when the launch is consumed without ever being forwarded + * (a data-stall/busy/nocap hold fell through). Zeroed with the rest + * of the candidate at capture. + */ + bool rel_done; + unsigned long captured_jiffies; /* STALLFREE: capture stamp for the + * GR-STALL waited_ms accounting */ u32 h_object; u64 push_va; u32 push_len; @@ -3921,7 +5575,105 @@ struct svc_gr_cand { u32 pointer_word[SVC_GR_POINTER_MAX]; /* QMD word of the pair */ }; -static struct svc_gr_cand svc_gr_cand; +/* + * fail4: the capture was a single slot; two concurrent launches (streams2's + * vadd pair on two channels) overwrote each other's candidate while the + * first was kept for a retry (the candidate-overwrite witness) and the + * engines ran crossed parameters (c1 == a2+b2). Keep up to four candidates + * side by side in a ring; the accessor macro addresses the current slot, + * so every existing use keeps working on the selected candidate, and + * capture/consume select the slot instead of clobbering it. + */ +#define SVC_GR_CAND_RING 4 +static struct svc_gr_cand_rec svc_gr_cand_ring[SVC_GR_CAND_RING]; +static u32 svc_gr_cand_pos; +#define svc_gr_cand (svc_gr_cand_ring[svc_gr_cand_pos]) +/* + * r383: push-VA history of recent captures, beyond the (four-slot) + * candidate ring's lifetime. A replayed decode of a complete QMD asks + * "has this launch ever been captured?"; the ring alone can have rotated + * past it, and a wrong "no" would re-capture (and re-execute) an already + * forwarded launch. Bounded, service thread only. + */ +#define SVC_GR_CAP_HIST 8 +static u64 svc_gr_cap_hist[SVC_GR_CAP_HIST]; +static u32 svc_gr_cap_hist_pos; + +/* + * r401: robust session-scoped "already captured" memory for the r383 + * replay-capture path. The 4-slot candidate ring + 8-VA history evicted a + * real capture within ~12 later captures, so a true replay of an + * already-forwarded launch (identical push VA and content) was + * re-captured and re-executed: the r399 soak log shows the same nine + * launch pushes re-submitted every ~10-11 s for minutes (idempotent + * kernels survive it; accumulators would not). The open-addressed table + * keeps the (push_va, push_hash) identity of every capture of the + * session, so a true replay always classifies as "seen" (LAUNCH-SKIP, + * exactly once) while a genuinely new push - even at the same VA with + * the same entry words - is captured through the normal + * `trusted && !replay` path and never consults this table. Cleared by + * svc_session_flush (a new client's identical-payload pushes restart + * their release counters, so a boot-scoped table would wrongly suppress + * them). Service-thread only. + */ +#define SVC_GR_SEEN_SLOTS 2048 +#define SVC_GR_SEEN_PROBE 8 +static struct { + u64 push_va; + u64 hash; +} svc_gr_seen[SVC_GR_SEEN_SLOTS]; +static u64 svc_gr_seen_stale; /* probe-chain evictions (telemetry) */ + +static u32 svc_gr_seen_mix(u64 push_va, u64 hash) +{ + u64 k = push_va * 0x9E3779B97F4A7C15ULL; + + k ^= hash + 0x9E3779B97F4A7C15ULL + (k << 6) + (k >> 2); + return (u32)(k % SVC_GR_SEEN_SLOTS); +} + +static bool svc_gr_seen_lookup(u64 push_va, u64 hash) +{ + u32 i, base = svc_gr_seen_mix(push_va, hash); + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_SEEN_PROBE; i++) { + u32 k = (base + i) % SVC_GR_SEEN_SLOTS; + + if (!svc_gr_seen[k].push_va) + return false; + if (svc_gr_seen[k].push_va == push_va && + svc_gr_seen[k].hash == hash) + return true; + } + return false; +} + +static void svc_gr_seen_store(u64 push_va, u64 hash) +{ + u32 i, base = svc_gr_seen_mix(push_va, hash); + + if (!push_va) + return; + for (i = 0; i < SVC_GR_SEEN_PROBE; i++) { + u32 k = (base + i) % SVC_GR_SEEN_SLOTS; + + if (!svc_gr_seen[k].push_va || + (svc_gr_seen[k].push_va == push_va && + svc_gr_seen[k].hash == hash)) { + svc_gr_seen[k].push_va = push_va; + svc_gr_seen[k].hash = hash; + return; + } + } + /* Full probe chain: replace the tail (oldest-ish) entry. */ + svc_gr_seen_stale++; + svc_gr_seen[(base + SVC_GR_SEEN_PROBE - 1) % SVC_GR_SEEN_SLOTS].push_va = + push_va; + svc_gr_seen[(base + SVC_GR_SEEN_PROBE - 1) % SVC_GR_SEEN_SLOTS].hash = + hash; +} static_assert(SVC_GR_QMD_WORDS == GSP_PHYS_GR_QMD_WORDS, "the sema GR QMD word count must match the backend ABI"); @@ -3957,7 +5709,7 @@ static_assert(sizeof(struct svc_gr_page) == 24, static u64 svc_push_hash(const u8 *p, u32 len); /* QMD field decode: the exact shape the host CUDA bridge uses. */ -static void svc_gr_qmd_decode(struct svc_gr_cand *c) +static void svc_gr_qmd_decode(struct svc_gr_cand_rec *c) { const u32 *q = c->qmd; u32 i; @@ -3996,13 +5748,50 @@ static void svc_gr_cand_capture(struct svc_channel *ch, { u32 i, n; + /* + * C9 (GAP-2 A4): a capture over a still-valid candidate (never + * offered) destroys an unforwarded launch - the single-slot + * overwrite hazard. Bounded, unconditional (gsp_sema_dbg is defined + * later in the file and this site may not reference it). + */ + /* + * fail4: never overwrite a DIFFERENT launch's unconsumed candidate - + * move to a free ring slot instead (same-push re-captures still + * refresh in place). + */ + if (svc_gr_cand.valid && !svc_gr_cand.offered && + svc_gr_cand.push_va != push_va) { + u32 ri; + + for (ri = 1; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % SVC_GR_CAND_RING; + + if (!svc_gr_cand_ring[rp].valid || + svc_gr_cand_ring[rp].offered) { + svc_gr_cand_pos = rp; + break; + } + } + } + if (svc_gr_cand.valid && svc_cand_ovw_log < 64) { + svc_cand_ovw_log++; + pr_info("sema-gr: CAND-OVERWRITE old_h=0x%x old_push=0x%llx new_push=0x%llx\n", + svc_gr_cand.h_object, svc_gr_cand.push_va, push_va); + } memset(&svc_gr_cand, 0, sizeof(svc_gr_cand)); svc_gr_cand.valid = true; + svc_gr_cand.captured_jiffies = jiffies; svc_gr_cand.h_object = ch->h_object; svc_gr_cand.push_va = push_va; svc_gr_cand.push_len = push_len; + /* r383: remember the identity beyond this ring slot's lifetime. */ + svc_gr_cap_hist[svc_gr_cap_hist_pos % SVC_GR_CAP_HIST] = push_va; + svc_gr_cap_hist_pos++; if (push && push_len) svc_gr_cand.push_hash = svc_push_hash(push, push_len); + /* r401: the session-wide exactly-once memory (survives ring rotation; + * see svc_gr_seen_lookup). */ + svc_gr_seen_store(push_va, svc_gr_cand.push_hash); if (push && push_len >= 4) { u32 total = push_len / 4; u32 n = total < SVC_GR_PUSH_WORDS ? total : SVC_GR_PUSH_WORDS; @@ -4095,8 +5884,14 @@ static bool svc_fwd_defer_arm(void) { if (!gsp_phys_guest_mem_enabled()) return false; - if (svc_gr_cand.valid && gsp_phys_gr_forward_enabled()) - return true; + if (gsp_phys_gr_forward_enabled()) { + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered) + return true; + } return svc_fwd_cand.valid && gsp_phys_work_defer_release_enabled(); } @@ -4155,8 +5950,434 @@ static bool svc_fwd_defer_armed; #define SVC_COPY_FORWARDED 1 /* Byte copy between two CE addresses (src/dst decoding mirrors svc_fill). */ +/* + * m171 (AGENT-F R2): write/read consistency ledger. A large H2D write to a + * device VA records {va_page -> pa, aperture} for every page it wrote; the + * later read of the same VA (the launch data fill, or a D2H copy) prefers + * the recorded page whenever its own resolution disagrees. The pipeline + * then reads exactly what was written regardless of which of the two + * resolutions is stale - the r174/r175 first-page divergence (the write + * landed on pa X, the fill read a stale page Y, and the engine computed on + * old bytes) can no longer split a write/read pair. Session-scoped: cleared + * with the flush; zero cost while empty. Apertures are stored in the SVC_ + * encoding, the canonical space for the svc accessors. + */ +struct svc_ledger_ent { + u64 va_page; + u64 pa_page; + u32 ap; +}; +/* + * Unified VA->landing-page authority ("R2 ledger"). + * + * m172 grew the original 256-entry ring to 2048, but a ring still evicts + * unrelated VAs wholesale: one 4 MiB H2D plus one launch's data-page + * record set is 4-6 K inserts, so an 8 MiB machine operand's early pages + * were evicted before the launch fill looked them up, and the fill fell + * back to its own (possibly stale) resolution - the exact "write path and + * read path pick different pages" class. + * + * m180: keyed hash table instead of a ring. One entry per VA page, O(1) + * update/lookup, no cross-VA eviction, and the insert of a page already + * present rewrites it in place (a write is a new landing and is always + * the truth). 16384 slots cover a full multi-buffer client's working set + * (8 MB in + 8 MB out + engine records ~ 12 K pages); hash collisions + * only occur between VA pages 64 MiB apart or across clients whose pages + * do not co-reside in time. + * + * COPYAUDIT r220: SVC_LEDGER_MAX/MASK/PROBE and the counters live above + * svc_rebuild (hoisted for the diag line); only the table and the home + * helper stay here. + */ +static struct svc_ledger_ent svc_ledger[SVC_LEDGER_MAX]; + +static inline u32 svc_ledger_home(u64 page) +{ + return (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & + SVC_LEDGER_MASK; +} + +static void svc_ledger_store(u64 va, u64 pa, u32 ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 home = svc_ledger_home(page); + u32 p; + + for (p = 0; p < SVC_LEDGER_PROBE; p++) { + struct svc_ledger_ent *e = &svc_ledger[(home + p) & + SVC_LEDGER_MASK]; + + if (e->va_page == page) { + /* + * A write path moving an existing page: its landing + * is the new truth. Logged (bounded) because a move + * is the write-side half of the DIVERGE class. + */ + if (e->pa_page != pa_page) { + svc_ledger_rewrites++; + if (svc_ledger_rewrites <= 16) + pr_info("sema-ledger: rewrite va=0x%llx old=0x%llx new=0x%llx ap=%u\n", + page, e->pa_page, pa_page, + ap); + } + e->pa_page = pa_page; + e->ap = ap; + return; + } + if (!e->va_page) { + e->va_page = page; + e->pa_page = pa_page; + e->ap = ap; + svc_ledger_n++; + svc_ledger_stores++; + if (svc_ledger_store_log < 64) { + svc_ledger_store_log++; + pr_info("sema-ledger: store va=0x%llx pa=0x%llx ap=%u n=%u\n", + page, pa_page, ap, svc_ledger_n); + } + return; + } + } + /* + * Cluster full (16 same-home VA pages co-resident): replace the home + * slot. Bounded, rare, and only affects VAs whose pages hash to the + * identical home slot. r220: this is a SILENT loss of the evicted + * VA's landing authority (its later readers fall back to the + * resolver, possibly stale), so the first occurrences are logged and + * counted into the diag line. + * + * STALLFREE cap-and-degrade: a sustained eviction burst means the + * table is being thrashed (a pin storm / leftover crashed client + * churning VA pages faster than the working set), and a half-evicted + * table silently strands authority for exactly the evicted pages. + * Past SVC_LEDGER_EVICT_RESET evictions in one session the cache is + * cleared wholesale and rebuilt from live writes: every reader + * uniformly falls back to the resolver until its page is rewritten, + * which removes the stranded-authority class at a known, logged + * cost. The clear is a cache operation only (no control flow, no + * wait); the current store then lands in the empty table. + */ + svc_ledger_evictions++; + svc_ledger_evict_since_reset++; + if (svc_ledger_evict_since_reset >= SVC_LEDGER_EVICT_RESET) { + svc_ledger_evict_since_reset = 0; + svc_ledger_evict_resets++; + pr_warn("sema-ledger: LEDGER-RESET evict=%llu resets=%llu (cluster thrash under churn; cache cleared, readers fall back to the resolver until rewritten)\n", + svc_ledger_evictions, svc_ledger_evict_resets); + svc_ledger_clear(); + } + if (svc_ledger_evictions <= 32) + pr_warn("sema-ledger: EVICT home=%u old_va=0x%llx new_va=0x%llx (cluster full; old landing authority lost until rewritten)\n", + home, (u64)svc_ledger[home].va_page, page); + svc_ledger[home].va_page = page; + svc_ledger[home].pa_page = pa_page; + svc_ledger[home].ap = ap; + svc_ledger_stores++; +} + +static int svc_ledger_lookup(u64 va, u64 *pa, u32 *ap) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 home = svc_ledger_home(page); + u32 p; + + for (p = 0; p < SVC_LEDGER_PROBE; p++) { + const struct svc_ledger_ent *e = + &svc_ledger[(home + p) & SVC_LEDGER_MASK]; + + if (!e->va_page) + return -ENOENT; + if (e->va_page == page) { + *pa = e->pa_page | (va & (SEMA_PAGE_SIZE - 1)); + *ap = e->ap; + return 0; + } + } + return -ENOENT; +} + +static void svc_ledger_clear(void) +{ + memset(svc_ledger, 0, sizeof(svc_ledger)); + svc_ledger_n = 0; + svc_ledger_rewrites = 0; + svc_ledger_reloc_diverges = 0; + svc_ledger_evict_since_reset = 0; +} + +/* + * COPYAUDIT (2026-10-18 copy-completeness audit): every copy at/above + * SVC_COPY_AUDIT_MIN is logged once with its want/done byte counts, and + * its first/last 16 destination bytes are read back immediately and + * compared against the bytes just written (COPY-MISMATCH on difference). + * COPY-TRUNC marks a copy that failed after partially writing (want>done), + * so a partial transfer can never be silent. Pure diagnostics: no control + * flow, locking or timing change; the readback uses the same accessor the + * write just used. Definitions live with svc_fill (first user). + */ +static void svc_copy_audit_chunk(struct nv_vgpu_device *vgpu, u64 dst_va, + u64 o, const u8 *bounce, size_t chunk, + bool shadow, u64 pa_eff, u32 aper, bool tail) +{ + size_t n = chunk < 16 ? chunk : 16; + size_t so = tail ? chunk - n : 0; /* true end sample */ + u8 rb[16] = {0}; + + if (shadow) { + if (svc_vas_read(dst_va + o + so, rb, n)) + return; + } else if (aper == CE_APERTURE_VID) { + u8 *base = vgpu->bars[1].vaddr; + + if (!base || pa_eff + so + n > svc_fb_shadow_size(vgpu)) + return; + memcpy(rb, base + pa_eff + so, n); + } else if (svc_read(vgpu, pa_eff + so, rb, n)) { + return; + } + if (memcmp(rb, bounce + so, n)) { + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x%llx path=ce-soft-copy want=%*phN got=%*phN\n", + dst_va + o + so, o + so, (int)n, + bounce + so, (int)n, rb); + pr_info_ratelimited("vgsp: CEAUDIT-MISMATCH op=copy va=0x%llx off=0x%llx path=ce-soft-copy want=%*phN got=%*phN\n", + dst_va + o + so, o + so, (int)n, + bounce + so, (int)n, rb); + } +} + +#define SVC_COPY_TRUNC_RET(rc_) \ + do { \ + if (done_out) \ + *done_out = (u64)o; \ + if (total >= SVC_COPY_AUDIT_MIN) \ + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=ce-soft-copy rc=%d\n", \ + dst_orig, (u64)total, (u64)o, (rc_)); \ + return (rc_); \ + } while (0) + +/* + * fail4: pipeline-order source snapshot for held copies. The m169 + * device-dst guard defers a staged H2D (host->device) for up to 3 s; + * the driver's ping-pong staging buffers are overwritten by the next + * chunk meanwhile, so the deferred read returns the content of a later + * chunk (vector_add: device a[i] pages held a[i+2 MB] - exactly the + * 2x1 MB staging stride). A held copy snapshots its source at hold + * time and the retried execution copies from the snapshot, matching + * the hardware's in-order read of the staging. + */ +#define SVC_COPY_SNAP_N 16 +#define SVC_COPY_SNAP_MAX (4u << 20) +#define SVC_COPY_SNAP_TTL_MS 30000u +struct svc_copy_snap { + u64 dst, src; + size_t total; + /* + * Owner identity (r444): the channel, ring entry and push-content + * hash of the decode pass that captured this snapshot. A snapshot + * exists to serve ONE entry's retried copy (the held staging read); + * keying it by (dst,src,total) alone let a snapshot captured for an + * earlier entry poison a later copy with the same operand triple - + * the wedge_ab "bad = mutated-element count" class: step N's H2D + * executed from step N-1's snapshot because the re-decode of the + * completed step N-1 entry re-captured its (unchanged) source after + * its copy had already run, and step N's copy then consumed it. + * The pass hash changes whenever the guest rewrites the push, so a + * ring slot reused by a fresh push can never match a stale owner. + */ + u64 owner_hash; + u32 owner_ch; + u32 owner_idx; + u8 *buf; + unsigned long stamp; +}; +static struct svc_copy_snap svc_copy_snaps[SVC_COPY_SNAP_N]; + +static u8 *svc_copy_snap_lookup(u64 dst, u64 src, size_t total, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].dst == dst && + svc_copy_snaps[i].src == src && + svc_copy_snaps[i].total == total && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash == owner_hash) + return svc_copy_snaps[i].buf; + return NULL; +} + +static void svc_copy_snap_free(u64 dst, u64 src, size_t total, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].dst == dst && + svc_copy_snaps[i].src == src && + svc_copy_snaps[i].total == total && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash == owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + return; + } +} + +/* + * r456/r457: the snapshot lifecycle is capture-once -> free on the copy's + * own successful consumption (svc_copy) -> free on ring-slot reuse with a + * new push hash (svc_copy_snap_capture) -> free when a re-decode skips an + * already-done op (svc_run_ops) -> TTL. The old per-entry owner free was + * removed from every opdef_clear caller: an entry whose release was + * delivered can still be REPLAYED (the pin-batch rewind) with its copy + * still owed, and freeing at that point let the replay re-capture the + * driver's refilled ping-pong staging (the r457 mode-C b1 H2D wrote b2's + * bytes into db1). A stale snapshot can only ever serve its own + * (ch, idx, hash) owner, so the r444 cross-entry poisoning stays closed. + */ +static void __maybe_unused svc_copy_snap_free_owner(u32 owner_ch, u32 owner_idx) +{ + u32 i; + + for (i = 0; i < SVC_COPY_SNAP_N; i++) + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } +} + +/* Capture the source of a virtual-operand copy about to be held. */ +static void svc_copy_snap_capture(struct nv_vgpu_device *vgpu, u64 dst, + u64 src, size_t total, u32 owner_ch, + u32 owner_idx, u64 owner_hash) +{ + struct svc_copy_snap *s = NULL; + u8 *buf; + size_t o; + u32 i; + + if (!total || total > SVC_COPY_SNAP_MAX) + return; + if (svc_copy_snap_lookup(dst, src, total, owner_ch, owner_idx, + owner_hash)) + return; /* already captured */ + /* + * r456: evict any snapshot left by the SAME ring slot under a + * DIFFERENT push hash (the slot was reused by a fresh push). The + * surviving-across-retries snapshot (svc_opdef_clear APPLY no + * longer frees it) must never be readable by the next occupant of + * the slot; the pass hash changes whenever the guest rewrites the + * push, so a same-slot different-hash snapshot is by definition + * stale for the new push. + */ + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash != owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } + } + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (!svc_copy_snaps[i].buf) { + s = &svc_copy_snaps[i]; + break; + } + if (time_after(jiffies, svc_copy_snaps[i].stamp + + msecs_to_jiffies(SVC_COPY_SNAP_TTL_MS))) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + s = &svc_copy_snaps[i]; + break; + } + } + if (!s) + return; /* all slots live: keep the historical behaviour */ + buf = kmalloc(total, GFP_KERNEL); + if (!buf) + return; + for (o = 0; o < total; ) { + size_t chunk = min_t(size_t, 0x1000 - ((src + o) & 0xfff), + total - o); + u64 pa = 0; + u32 ap = 0; + + /* The source is a virtual host VA; resolve per page first. */ + if (svc_op_translate(vgpu, src + o, &pa, &ap) || + svc_read(vgpu, pa, buf + o, chunk)) { + kfree(buf); + return; + } + o += chunk; + } + s->dst = dst; + s->src = src; + s->total = total; + s->owner_ch = owner_ch; + s->owner_idx = owner_idx; + s->owner_hash = owner_hash; + s->stamp = jiffies; + s->buf = buf; + pr_info_ratelimited("vgsp-sema: copy snapshot captured dst=0x%llx src=0x%llx total=%zu owner=0x%x/%u/%016llx first4=%08x %08x %08x %08x\n", + dst, src, total, owner_ch, owner_idx, + owner_hash, + total >= 4 ? *(const u32 *)buf : 0, + total >= 8 ? *(const u32 *)(buf + 4) : 0, + total >= 12 ? *(const u32 *)(buf + 8) : 0, + total >= 16 ? *(const u32 *)(buf + 12) : 0); +} + +static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind, + u64 pa, u32 off, u32 bytes, u64 *hpa, + u32 *out_off); + +/* r402 row-wise wrapper: see svc_fill. The source advances by spitch. */ +/* r402: read-only resolver probe (defined with the GR fill diagnostics). */ +static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out); static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, - u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture) + u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture, + u64 *done_out, u32 rows, u32 dpitch, u32 spitch, + u32 owner_ch, u32 owner_idx, u64 owner_hash) +{ + if (rows > 1 && dpitch) { + u64 done = 0; + int rc = 0; + u32 r; + + for (r = 0; r < rows; r++) { + u64 row_done = 0; + + rc = svc_copy_1d(vgpu, dst + (u64)r * dpitch, + src + (u64)r * spitch, total, + dst_phys, dst_aperture, src_phys, + src_aperture, &row_done, + owner_ch, owner_idx, owner_hash); + done += row_done; + if (rc) + break; + } + if (done_out) + *done_out = done; + return rc; + } + return svc_copy_1d(vgpu, dst, src, total, dst_phys, dst_aperture, + src_phys, src_aperture, done_out, + owner_ch, owner_idx, owner_hash); +} + +static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, + u32 dst_phys, u32 dst_aperture, u32 src_phys, u32 src_aperture, + u64 *done_out, u32 owner_ch, u32 owner_idx, u64 owner_hash) { u8 bounce[256]; size_t o; @@ -4166,10 +6387,29 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u64 dst_orig = dst, src_orig = src; u32 dst_prov = 0, src_prov = 0; bool bigcopy_this = false; + bool audit = total >= SVC_COPY_AUDIT_MIN; + u8 *snap = NULL; static u64 bigcopy_log; + bool trace64 = (total == 65536u); + const char *dir = "?"; + if (done_out) + *done_out = 0; /* 0 until the first byte lands */ if (!total || total > (256ULL << 20)) return -EINVAL; + /* + * fail4: name every copy the executor is asked to run (pre- + * resolution), so a copy that fails or is held before any witness + * still has an entry record. + */ + if (gsp_sema_dbg && svc.copyent_log < 128) { + svc.copyent_log++; + pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu dphys=%u dap=%u sphys=%u sap=%u\n", + svc.cur_ch, dst, src, total, dst_phys, dst_aperture, + src_phys, src_aperture); + } + snap = svc_copy_snap_lookup(dst_orig, src_orig, total, owner_ch, + owner_idx, owner_hash); if (!dst_phys) { u64 pa; @@ -4177,17 +6417,96 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, if (svc_vas_has(dst)) { dst_shadow = true; + } else if (!svc_ledger_lookup(dst, &pa, &aper)) { + /* + * r402 diagnostic (bounded): name the case where the + * ledger's landing page for a copy DESTINATION + * disagrees with the live resolver (pin/walk). The + * D2H staging writes failed on the class this + * witness detects: the writer follows the ledger + * while the guest's reader holds a different page, + * so the client reads stale data (streams2 c1 + * zeros / smiley 3-of-4 passes). Read-only probe + * (svc_gr_src_probe stores nothing). + */ + if (gsp_sema_dbg && svc_dst_div_log < 64 && + dst >= SVC_VAS_FLOOR) { + u64 wpa = 0; + u32 wap = 0; + const char *src = + svc_gr_src_probe(vgpu, dst, &wpa, &wap); + + if (wpa && + (((wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + ((wap == SVC_APERTURE_FB) != + (aper == SVC_APERTURE_FB)))) { + svc_dst_div_log++; + pr_err_ratelimited("vgsp-sema: DST-DIVERGE va=0x%llx ledger=0x%llx/%u probe=%s 0x%llx/%u (write follows ledger)\n", + dst, pa, aper, src, wpa, wap); + } + } + /* + * m180 unification: a previous write of this VA is + * the authority for where the content lives (and so + * for where a new write lands); it also spares the + * m169 stale-walk hold below, which exists precisely + * because no pin was known. + * + * r376 pin-agreement: a FRESH pin that disagrees is + * newer than the recorded landing - the guest freed + * or moved the buffer - so the pin wins and the + * entry is repaired (the same rule the fills and the + * launch paths apply; without it a stale entry + * writes a freed page - the guest-corruption class). + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 64) + pr_info("sema-ledger: PIN-WINS copy-dst va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + dst, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper, ipa, iap); + svc_ledger_store(dst, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + pa = ipa; + aper = (u32)iap; + } + } + dst = pa; + dst_aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + dst_prov = 1; + dst_virt = true; } else if (svc_op_translate(vgpu, dst, &pa, &aper)) { - if (!svc_vas_can_shadow(dst)) { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(dst, &lpa, &lap)) { + dst = lpa; + dst_aperture = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + dst_prov = 1; + dst_virt = true; + } else if (!svc_vas_can_shadow(dst)) { if (gsp_sema_dbg && svc.op_tr_log < 64) { svc.op_tr_log++; pr_info("sema-tr-miss: DST va=0x%llx h=0x%x\n", dst, svc.cur_ch); } return -ENOENT; + } else { + dst_shadow = true; + svc.vas_shadowed++; } - dst_shadow = true; - svc.vas_shadowed++; } else { dst = pa; dst_aperture = aper; @@ -4195,24 +6514,177 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, dst_virt = true; } } + /* + * m169 (AGENT-F R1): a device-lane destination translated only by a + * page-directory walk (prov 3/4) or the RAM scan (5) can be a stale + * PTE from the previous session - the table CONTENT is not wiped by + * the session flush - and a wrong-but-valid vidmem write is silent. + * r172 streams2_diag: the stream-1 a H2D ran rc=0 through a + * walk-resolved destination while the launch (pin-resolved) read the + * true page still holding the previous client's h_a; the engine + * computed from stale inputs and the client verified 32768 bad. + * Since the resolver tries pins FIRST, prov >= 3 means no pin existed + * at resolve time: hold the copy (-ENOENT feeds the m163 bounded + * hold), which re-decodes after the client's pins land (seconds) and + * is loud on expiry - never silent corruption. Device destinations + * only; sysmem copies keep the historical sources. + */ + if (dst_virt && dst_aperture == CE_APERTURE_VID && + dst_prov >= 3 && dst_orig >= SVC_VAS_FLOOR && + gsp_sema_copy_dst_pin_guard && + svc_dst_guard_hold(dst_orig)) { + svc.copy_dst_hold++; + if (svc.copy_dst_hold <= 16) + pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u (no pin yet; stale-table guard)\n", + dst_orig, dst_prov); + /* + * fail4: the staging source is valid NOW; snapshot it so + * the deferred execution copies the pipeline-order bytes. + */ + if (!src_phys && src_orig >= SVC_VAS_FLOOR) + svc_copy_snap_capture(vgpu, dst_orig, src_orig, + total, owner_ch, owner_idx, + owner_hash); + return -ENOENT; + } + /* + * r378: the same stale-table class for SYS destinations resolved by + * the walk/scan (prov >= 3): a FRESH injected pin that disagrees is + * the guest's newest statement, so writing through the resolved page + * lands on a page the guest has freed/re-homed (the guest-corruption + * class). Hold through the same bounded re-decode path instead of + * writing; a missing pin keeps the historical behaviour (the + * resolution is then the write authority). + */ + if (dst_virt && dst_aperture == CE_APERTURE_SYS && + dst_prov >= 3 && dst_orig >= SVC_VAS_FLOOR && + gsp_sema_copy_dst_pin_guard) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_orig, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dst & ~(u64)(SEMA_PAGE_SIZE - 1))) || + svc_ap_of(iap) != SVC_APERTURE_SYS) && + svc_dst_guard_hold(dst_orig)) { + svc.copy_dst_hold++; + if (svc.copy_dst_hold <= 16) + pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u pin=0x%llx (stale SYS target)\n", + dst_orig, dst_prov, ipa); + if (!src_phys && src_orig >= SVC_VAS_FLOOR) + svc_copy_snap_capture(vgpu, dst_orig, + src_orig, total, + owner_ch, owner_idx, + owner_hash); + return -ENOENT; + } + } + /* + * r231 READ-ORDER guard/witness: a device-src copy whose source VA + * has a PENDING device-dst write hold reads the PRE-write content - + * the writer's bytes land only when the guard resolves while the + * reader executes immediately (the ctaid_probe RAWA round trip is + * the reproducer). Witnessed always (bounded); deferred only with + * sema_read_order_guard, through the same bounded-hold re-decode. + */ + if (src_orig >= SVC_VAS_FLOOR && svc_dst_guard_pending(src_orig)) { + if (svc_read_order_log < 16) { + svc_read_order_log++; + pr_info("sema-svc: READ-ORDER-WIT device-src copy va=0x%llx reads while a device-dst write to the same VA is still held%s\n", + src_orig, gsp_sema_read_order_guard ? + " (deferring)" : " (guard off)"); + } + if (gsp_sema_read_order_guard) + return -ENOENT; + } if (!src_phys) { u64 pa; u32 aper; if (svc_vas_has(src)) { src_shadow = true; + } else if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src, &pa, &aper)) { + /* + * r375 read-after-write authority: the landing table + * records where every write of this VA landed (writes + * store it, fresh pins repair it). A reader MUST read + * exactly that page or the engine's output is invisible + * (the sum=0 class). The walk/pin sources can answer + * differently - e.g. the same numeric pa in a different + * aperture - so the table wins for device VAs; only a + * VA with no recorded landing falls through to the + * resolver below. + * + * r376 pin-agreement: unless a FRESH pin disagrees - + * then the pin is newer (the guest re-pinned the VA) + * and wins, and the entry is repaired. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(src, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)aper)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 64) + pr_info("sema-ledger: PIN-WINS copy-src va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + src, + pa & ~(u64)(SEMA_PAGE_SIZE - 1), + aper, ipa, iap); + svc_ledger_store(src, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + pa = ipa; + aper = (u32)iap; + } + } + src = pa; + src_aperture = (aper == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + src_prov = 1; + src_virt = true; } else if (svc_op_translate(vgpu, src, &pa, &aper)) { - if (gsp_sema_dbg && svc.op_tr_log < 64) { - svc.op_tr_log++; - pr_info("sema-tr-miss: SRC va=0x%llx h=0x%x\n", - src, svc.cur_ch); + u64 lpa = 0; + u32 lap = 0; + + /* + * m180 unification: read the page the last host write + * of this VA landed on when the resolver can no longer + * answer (same rule as the per-page loop below), instead + * of failing the whole copy. + */ + if (!svc_ledger_lookup(src, &lpa, &lap)) { + src = lpa; + src_aperture = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + src_prov = 1; + src_virt = true; + } else { + if (gsp_sema_dbg && svc.op_tr_log < 64) { + svc.op_tr_log++; + pr_info("sema-tr-miss: SRC va=0x%llx h=0x%x\n", + src, svc.cur_ch); + } + return -ENOENT; } - return -ENOENT; } else { src = pa; src_aperture = aper; src_prov = svc.op_tr_src; src_virt = true; + /* + * r375b: do NOT record a resolver answer into the + * landing table here. The resolver can return a stale + * PTE page (the m169 class) and the table is the WRITE + * authority - a poisoned entry makes every later write + * of this VA land on the stale page (r375 boot: slab + * GPF in kmem_cache_alloc at guest t=77s, before any + * client). The table is fed by writes and fresh pins + * only. + */ } } @@ -4222,8 +6694,35 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, * 16 bytes on each side before the copy. The vadd data path is exactly * these copies; this separates a wrong resolution from a wrong copy. */ - if (total >= (1u << 20) && bigcopy_log < 24) { + if ((total >= SVC_COPY_AUDIT_MIN || + (dst_virt && dst_orig >= SVC_VAS_FLOOR)) && + (bigcopy_log < gsp_sema_bigcopy_log_max || total == 65536u)) { u8 sb[16] = {0}, db[16] = {0}; + /* + * r235: 64 KB copies are exempt from the bigcopy budget - the + * probe's A upload/readbacks must ALWAYS appear (the budget + * belongs to the multi-MB traffic and had absorbed every 64 KB + * probe copy in the r235 capture). + */ + bool trace64 = (total == 65536u); + const char *dir = + (dst_aperture == CE_APERTURE_VID && + src_aperture == CE_APERTURE_SYS) ? "H2D" : + (src_aperture == CE_APERTURE_VID && + dst_aperture == CE_APERTURE_SYS) ? "D2H" : + src_shadow ? "shadow->" : "phys"; + + /* + * r235: 64 KB copies are exempt from the bigcopy budget - the + * probe's A upload/readbacks must ALWAYS appear (the budget + * belongs to the multi-MB traffic and had absorbed every 64 KB + * probe copy in the r235 capture). + */ + dir = (dst_aperture == CE_APERTURE_VID && + src_aperture == CE_APERTURE_SYS) ? "H2D" : + (src_aperture == CE_APERTURE_VID && + dst_aperture == CE_APERTURE_SYS) ? "D2H" : + src_shadow ? "shadow->" : "phys"; bigcopy_log++; bigcopy_this = true; @@ -4247,12 +6746,55 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, } else { svc_read(vgpu, dst, db, sizeof(db)); } - pr_info("sema-bigcopy: dst_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | src_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | bytes=%zu\n", + pr_info("sema-bigcopy: dst_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | src_va=0x%llx -> pa=0x%llx ap=%u prov=%u shadow=%u | bytes=%zu vbase=%px\n", dst_orig, dst, dst_aperture, dst_prov, dst_shadow, src_orig, src, src_aperture, src_prov, src_shadow, - total); + total, (void *)vgpu->bars[1].vaddr); + /* + * r348 diagnostic: for a high-VA SYS source name the exact + * host PA this read will touch (gpa_to_hpa) - the direct + * comparison against the svc-fwd-hpa writeback line. + */ + if (src_orig >= SVC_VAS_FLOOR && + src_aperture == CE_APERTURE_SYS) { + u64 shpa = 0; + u32 soff = 0; + + if (!svc_fwd_operand_hpa(vgpu, SVC_APERTURE_SYS, + src, 0, SEMA_PAGE_SIZE, + &shpa, &soff) && !soff) + pr_info("sema-bigcopy: src-hpa va=0x%llx gpa=0x%llx -> hpa=0x%llx\n", + src_orig, src, shpa); + else + pr_info("sema-bigcopy: src-hpa va=0x%llx gpa=0x%llx -> UNMAPPED\n", + src_orig, src); + } pr_info("sema-bigcopy: src16=%16phN dst16_before=%16phN\n", sb, db); + /* + * r234 A-TRACE: one labeled line per 64 KB copy so the probe A + * upload/readback is greppable end-to-end. pt=P1 = H2D source + * read (the uploaded bytes as ints), pt=P4 = D2H source read + * (the engine-authored page as ints). The expected content is + * ints 0,1,2,3,... for the probe A buffer. + */ + if (trace64 && svc_atrace_log < 128u) { + u32 i0, i1, i2, i3, j0 = 0, j1 = 0, j2 = 0, j3 = 0; + + svc_atrace_log++; + memcpy(&i0, sb, 4); + memcpy(&i1, sb + 4, 4); + memcpy(&i2, sb + 8, 4); + memcpy(&i3, sb + 12, 4); + memcpy(&j0, db, 4); + memcpy(&j1, db + 4, 4); + memcpy(&j2, db + 8, 4); + memcpy(&j3, db + 12, 4); + pr_info("vgsp: A-TRACE pt=%s dir=%s dst_va=0x%llx dst_pa=0x%llx dst_prov=%u | src_va=0x%llx src_pa=0x%llx src_prov=%u | src_int4=%d %d %d %d dst_int4_before=%d %d %d %d\n", + !strcmp(dir, "H2D") ? "P1" : "P4", dir, + dst_orig, dst, dst_prov, src_orig, src, + src_prov, i0, i1, i2, i3, j0, j1, j2, j3); + } } /* @@ -4301,6 +6843,17 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, svc_fwd_cand.bytes = (u32)total; /* + * m180 unification: record the destination landing + * even when the copy itself is deferred to the + * forwarded-request path (the software copy below + * never runs then, so the in-loop store would be + * skipped and the page would have no authority for + * the fill/D2H readers). + */ + if (dst_orig >= SVC_VAS_FLOOR) + svc_ledger_store(dst_orig, dst_page, dst_kind); + + /* * P6b wave 2: with the hardware-primary path armed * (parameter on and all three prerequisites on, so the * request can queue as a guest copy and its releases @@ -4434,64 +6987,267 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, chunk = min_t(size_t, chunk, 0x1000 - ((dst_orig + o) & 0xfff)); - if (src_shadow) { + if (snap) { + /* + * r314: consume the decode-time staging snapshot. + * The capture machinery existed since cb50282b but + * the execution never read it (lookup at svc_copy's + * top, free on success, NO consumer) - every copy + * read the live staging, which the driver had + * refilled once the push's fence fired (the + * deterministic 786335-bad vector_add output, + * identical across boots). Reading from the + * snapshot also needs no live source translation, + * so the copies execute on their first allowed + * pass instead of storming 60+ rc=-2 retries. + */ + memcpy(bounce, snap + o, chunk); + } else if (src_shadow) { if (svc_vas_read(src_orig + o, bounce, chunk)) - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); } else if (src_virt) { u64 pa = 0; u32 aper = 0; + u64 lpa = 0; + u32 lap = 0; - if (svc_op_translate(vgpu, src_orig + o, &pa, &aper)) - return -ENOENT; + /* + * r375: the landing table is the read-after-write + * authority for device VAs (see the top-level src + * resolve). Per page, the table wins whenever it has + * an entry; the resolver below only answers a VA the + * table has never seen. r376: a fresh disagreeing pin + * is newer than the table and wins (repaired). + */ + if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src_orig + o, &lpa, &lap)) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(src_orig + o, &ipa, &iap) && + (((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (lpa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)iap != (u32)lap)) { + svc_ledger_pin_repairs++; + svc_ledger_store(src_orig + o, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + lpa = ipa & ~(u64)(SEMA_PAGE_SIZE - 1); + lap = (u32)iap; + } + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (svc_op_translate(vgpu, src_orig + o, &pa, &aper)) { + /* + * m180 unification: a per-page resolution miss + * must not fail the whole copy when the unified + * table knows where this VA's content landed. + */ + if (svc_ledger_lookup(src_orig + o, &lpa, &lap)) + SVC_COPY_TRUNC_RET(-ENOENT); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } else if (src_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(src_orig + o, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (lap == SVC_APERTURE_FB) != + (aper == CE_APERTURE_VID))) { + /* + * m171 (AGENT-F R2): read-after-write - when + * a large H2D wrote this device VA page + * through a different translation, read the + * page the write landed on. + */ + svc_ledger_diverges++; + if (svc_ledger_diverges <= 32) + pr_info("sema-ledger: DIVERGE read va=0x%llx wrote=0x%llx read=0x%llx\n", + src_orig + o, lpa, pa); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + } if (aper == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; if (!base || pa + chunk > svc_fb_shadow_size(vgpu)) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(bounce, base + pa, chunk); } else if (svc_read(vgpu, pa, bounce, chunk)) { - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); } } else if (src_aperture == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); if (!base || src + o + chunk > size) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(bounce, base + src + o, chunk); } else { if (svc_read(vgpu, src + o, bounce, chunk)) - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); + } + + /* + * r367 TABLE-COPY witness: a page-table page/entry copy is + * the guest's authoritative table content; dump the carried + * bytes so a table that later walks as ZERO can be traced to + * the copy that built it (or proven the source was already + * zero). Bounded; table-shaped sizes only. + */ + if (!o && (total == 4096 || total == 16) && + svc_tabcopy_log < 64) { + svc_tabcopy_log++; + pr_info("sema-tabcopy: dst_orig=0x%llx src_orig=0x%llx total=%zu dphys=%u dvirt=%d svirt=%d r_dst=0x%llx r_src=0x%llx first16=%16phN\n", + dst_orig, src_orig, total, dst_phys, dst_virt, + src_virt, dst, src, bounce); } if (dst_shadow) { if (svc_vas_write(dst_orig + o, bounce, chunk)) - return -ENOMEM; + SVC_COPY_TRUNC_RET(-ENOMEM); + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, true, 0, 0, + (o != 0)); } else if (dst_virt) { u64 pa = 0; u32 aper = 0; + u64 lpa = 0; + u32 lap = 0; + u32 wprov = 0; - if (svc_op_translate(vgpu, dst_orig + o, &pa, &aper)) - return -ENOENT; + if (svc_op_translate(vgpu, dst_orig + o, &pa, &aper)) { + /* + * m180 unification: a per-page resolution miss + * lands on the known page instead of failing + * the whole copy (readers already use it). + */ + if (svc_ledger_lookup(dst_orig + o, &lpa, &lap)) + SVC_COPY_TRUNC_RET(-ENOENT); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wprov = 0xF1; + } else if (dst_orig >= SVC_VAS_FLOOR && + !svc_ledger_lookup(dst_orig + o, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (lap == SVC_APERTURE_FB) != + (aper == CE_APERTURE_VID))) { + svc_ledger_diverges++; + if (svc.op_tr_src == 1) { + /* + * r336 PIN-WINS (the r335 vector_add + * guest-panic class): a pin-backed + * translation (guest pagemap truth, + * op_tr_src == 1) that disagrees + * with the ledger means the guest + * remapped or reused this VA. The + * ledger page is where an earlier + * write landed; the guest has since + * freed or moved it, so writing it + * can land on memory the guest + * kernel now owns (observed: RM + * object fields overwritten with + * client c-data, slab freelist + * corruption, guest panic). Trust + * the pin for the write and re-home + * the ledger (the store after the + * write records the fresh page); + * readers then converge on the same + * page too. + */ + svc_ledger_pinwins++; + if (svc_ledger_diverges <= 512) + pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx (pin wins)\n", + dst_orig + o, lpa, pa); + wprov = svc.op_tr_src; + } else { + /* + * m180 unification: keep ONE + * authoritative page per VA. A new + * write goes to the page the + * previous content landed on, and + * the fill/D2H readers (which + * prefer the table) stay consistent + * with it. Non-pin translations + * (2..5) can carry stale PTEs + * (m169), so the write history + * stays the authority there. + */ + if (svc_ledger_diverges <= 512) + pr_info("sema-ledger: DIVERGE write va=0x%llx prev=0x%llx cur=0x%llx\n", + dst_orig + o, lpa, pa); + pa = lpa; + aper = (lap == SVC_APERTURE_FB) ? + CE_APERTURE_VID : CE_APERTURE_SYS; + wprov = 0xF0; + } + } else { + wprov = svc.op_tr_src; + } + /* + * D2H-MAP (diagnostic): per-page destination + * resolution of large host-dest copies - the direct + * witness for which guest pages a readback lands on. + * Bounded; one line per 4 KB page of copies >= 1 MB. + */ + if (total >= (1u << 20) && !(o & (SEMA_PAGE_SIZE - 1)) && + svc_d2hmap_log < 4096u) { + svc_d2hmap_log++; + pr_info("sema-d2hmap: va=0x%llx o=%zu pa=0x%llx ap=%u prov=%u\n", + dst_orig + o, o, pa, aper, wprov); + } if (aper == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; if (!base || pa + chunk > svc_fb_shadow_size(vgpu)) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(base + pa, bounce, chunk); } else if (svc_write(vgpu, pa, bounce, chunk)) { - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); } + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, false, pa, aper, + (o != 0)); + /* + * m171/m180: a large device-dest write records its + * landing page - SYS and VID alike, because the + * sysmem-backed UVM buffers are the dominant + * device-visible case and were the half the table + * never saw. The later kernel fill / D2H read of the + * same VA then uses exactly this page even if its own + * resolution is stale. + */ + if (dst_orig >= SVC_VAS_FLOOR && total >= (64u << 10)) + svc_ledger_store(dst_orig + o, pa, + (aper == CE_APERTURE_VID) ? + SVC_APERTURE_FB : + SVC_APERTURE_SYS); } else if (dst_aperture == CE_APERTURE_VID) { u8 *base = vgpu->bars[1].vaddr; size_t size = svc_fb_shadow_size(vgpu); if (!base || dst + o + chunk > size) - return -EINVAL; + SVC_COPY_TRUNC_RET(-EINVAL); memcpy(base + dst + o, bounce, chunk); + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, false, dst + o, + CE_APERTURE_VID, + (o != 0)); } else { if (svc_write(vgpu, dst + o, bounce, chunk)) - return -EIO; + SVC_COPY_TRUNC_RET(-EIO); + if (audit && (!o || o + chunk == total)) + svc_copy_audit_chunk(vgpu, dst_orig, o, bounce, + chunk, false, dst + o, + CE_APERTURE_SYS, + (o != 0)); } (void)i; o += chunk; @@ -4517,6 +7273,23 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, } pr_info("sema-bigcopy: dst16_after=%16phN (dst_va=0x%llx pa=0x%llx)\n", da, dst_orig, dst); + /* + * r234 A-TRACE pt=P2/P4R: the destination content right after + * the copy - for H2D this is the landed upload (P2), for D2H + * the host-side readback result (P4R). + */ + if (trace64 && svc_atrace_log < 128u) { + u32 i0, i1, i2, i3; + + svc_atrace_log++; + memcpy(&i0, da, 4); + memcpy(&i1, da + 4, 4); + memcpy(&i2, da + 8, 4); + memcpy(&i3, da + 12, 4); + pr_info("vgsp: A-TRACE pt=%s dir=%s dst_va=0x%llx dst_pa=0x%llx dst_int4_after=%d %d %d %d\n", + !strcmp(dir, "H2D") ? "P2" : "P4R", dir, + dst_orig, dst, i0, i1, i2, i3); + } } if (gsp_sema_dbg && !src_shadow && !dst_shadow && @@ -4553,6 +7326,34 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, pr_info("sema-copy-dst: after dst=0x%llx dst16=%*phN\n", dst, n, da); } + if (audit && svc_copy_audit_n < SVC_COPY_AUDIT_MAX) { + svc_copy_audit_n++; + pr_info("vgsp: COPYAUDIT path=ce-soft-copy va=0x%llx len=0x%llx done=0x%llx\n", + dst_orig, (u64)total, (u64)o); + } + /* + * m180 parity with svc_fill: a large virtual-destination copy is a + * write/read pair with the launch rec-build readers - record its + * landing pages so the GR fill follows the page the copy actually + * wrote. Without this, a rec table built before the pins landed + * kept the stale pre-pin pages while the copy landed on the pinned + * page (streams2 r254: the b1 rec range carried the a2 pages and + * the vadd read a2's pattern as its second input). + */ + if (!dst_phys && dst_virt && dst_orig >= SVC_VAS_FLOOR && + total >= (64u << 10)) { + u32 cap = (dst_aperture == CE_APERTURE_VID) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + size_t po; + + for (po = 0; po < total; po += SEMA_PAGE_SIZE) + svc_ledger_store(dst_orig + po, dst + po, cap); + } + if (snap) + svc_copy_snap_free(dst_orig, src_orig, total, owner_ch, + owner_idx, owner_hash); + if (done_out) + *done_out = (u64)o; /* o == total on the success path */ return 0; } @@ -4583,6 +7384,23 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, svc.memsets_done++; return 0; } + /* + * m180 unification: a page with a known landing (previous H2D/fill or + * engine write-back) is authoritative - write the inline data there + * so the engine fill / D2H readers that follow the table see it. + */ + { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(dst_va, &lpa, &lap)) { + pa = lpa & ~0xfffULL; + ap = (lap == SVC_APERTURE_FB) ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + svc_pin_store(dst_va & ~0xfffULL, pa, ap); + goto write_landing; + } + } if (svc_lookup_trusted(vgpu, dst_va & ~0xfffULL, &pa, &ap)) { if (!svc_vas_write(dst_va, data, bytes)) { svc.vas_shadowed++; @@ -4602,6 +7420,7 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, return -ENOENT; } svc_pin_store(dst_va & ~0xfffULL, pa, ap); +write_landing: pa += dst_va & 0xfff; if (gsp_sema_dbg && svc.grce_log < (gsp_sema_ops_log > 64 ? gsp_sema_ops_log : 64)) { @@ -4624,24 +7443,343 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, for (o = 0; o < bytes; o += 4096) { u32 chunk = bytes - o > 4096 ? 4096 : bytes - o; - if (svc_write(vgpu, pa + o, (void *)(data + o), chunk)) + if (svc_write(vgpu, pa + o, (void *)(data + o), chunk)) { + if (bytes >= SVC_COPY_AUDIT_MIN) + pr_err_ratelimited("vgsp: COPY-TRUNC va=0x%llx want=0x%llx done=0x%llx path=grce-inline rc=%d\n", + dst_va, (u64)bytes, + (u64)o, -EIO); return -EIO; + } + } + } + /* + * COPYAUDIT (see the svc_copy comment): log the big inline write and + * read the first 16 destination bytes back through the aperture it + * was written to, comparing against the source bytes. + */ + if (bytes >= SVC_COPY_AUDIT_MIN && svc_copy_audit_n < SVC_COPY_AUDIT_MAX) { + u8 rb[16] = {0}; + bool ok = false; + + if (ap == SVC_APERTURE_FB) { + u8 *base = vgpu->bars[1].vaddr; + + if (base && pa + sizeof(rb) <= svc_fb_shadow_size(vgpu)) { + memcpy(rb, base + pa, sizeof(rb)); + ok = true; + } + } else if (!svc_read(vgpu, pa, rb, sizeof(rb))) { + ok = true; } + svc_copy_audit_n++; + pr_info("vgsp: COPYAUDIT path=grce-inline va=0x%llx len=0x%llx done=0x%llx\n", + dst_va, (u64)bytes, (u64)bytes); + if (ok && memcmp(rb, data, sizeof(rb))) + pr_info_ratelimited("vgsp: COPY-MISMATCH va=0x%llx off=0x0 path=grce-inline want=%*phN got=%*phN\n", + dst_va, 16, data, 16, rb); } + /* + * m180: record the landing so the fill/D2H readers of this VA follow + * the bytes the engine's inline path wrote, not their own resolution. + */ + if (dst_va >= SVC_VAS_FLOOR) + svc_ledger_store(dst_va, pa, (u32)ap); svc.memsets_done++; return 0; } +/* defined below (svc_handle_map_add); the CE decode path drains the queued + * scanner injections before resolving copy operands, exactly like the GR + * path's m143 drain. */ +static void svc_handle_map_add(void); + /* Execute a decoded operation list (fills and copies). `ops_done` (may be * NULL) is the per-entry bitmap of ops that already ran for the entry's * identity; `ops_base` is the sequence number of ops[0] within the entry. */ + +/* + * r34x: evaluate a C56F SEM_EXECUTE acquire against the semaphore word. + * The class defines the opcodes (clc56f.h NVC56F_SEM_EXECUTE_OPERATION: + * 0 ACQUIRE, 1 RELEASE, 2 ACQ_STRICT_GEQ, 3 ACQ_CIRC_GEQ, 4 ACQ_AND, + * 5 ACQ_NOR, 6 REDUCTION) and the CUDA driver documents the wait + * conditions each op must implement (cuda_device.h CUIsemaphoreAquire- + * Flags_enum: GEQ default = circular >=, EQUAL = ==, AND = (v & p) != 0, + * NOR = ~(v | p) != 0, STRICT_GEQ = >= with no wrap; ampere_semaphore.c + * maps those flags onto the opcodes: EQUAL->0, STRICT_GEQ->2, GEQ->3, + * AND->4, NOR->5). Note op 0 is EQUALITY, not a >=: only op 2/3 are + * ordering compares. 32-bit payloads: every observed CUDA acquire is + * PAYLOAD_SIZE 32BIT (SEM_PAYLOAD_HI == 0); a 64-bit acquire is captured + * and judged on the low word only (its high word is not modelled). + */ +static bool svc_wait_satisfied(u32 cur, u8 op, u32 need) +{ + switch (op) { + case 0: /* ACQUIRE: wait for v == p */ + return cur == need; + case 2: /* ACQ_STRICT_GEQ: v >= p, no wrap */ + return cur >= need; + case 3: /* ACQ_CIRC_GEQ: v in [p, p + 2^31 - 1] (wrap allowed) */ + return (s32)(cur - need) >= 0; + case 4: /* ACQ_AND: (v & p) != 0 */ + return (cur & need) != 0; + case 5: /* ACQ_NOR: ~(v | p) != 0 */ + return ~(cur | need) != 0; + default: + return true; + } +} + +/* r34x: the per-channel first-hold clock for the bounded wait. */ +static struct svc_wait_hold *svc_wait_hold_get(u32 h, bool create) +{ + struct svc_wait_hold *free_slot = NULL; + int i; + + for (i = 0; i < SVC_WAIT_HOLD_MAX; i++) { + struct svc_wait_hold *e = &svc_wait_holds[i]; + + if (e->active && e->h == h) + return e; + if (!e->active && !free_slot) + free_slot = e; + } + if (!create || !free_slot) + return NULL; + memset(free_slot, 0, sizeof(*free_slot)); + free_slot->active = true; + free_slot->h = h; + free_slot->since = jiffies; + return free_slot; +} + +/* + * r506w/r507a: bounded per-VA failure latch for virtual ops. A virtual op + * that keeps failing -ENOENT (an unresolvable VA) re-runs on every decode + * pass and sets svc.op_tr_pending each time, so its push holds for the full + * SVC_OPDEF_HOLD_MS opdef bound - a client whose stream carries such an op + * (r505/r506 witness: the RM's channel-progress zeroing at 0x2000c0xxx + * after the first client, dst never translated) ground its whole stream + * at ~10 s per push ("hang after DEVICE" for matmul/multikernel). + * + * The window must still cover the legitimate pin-late retry (observed + * pin latency 0.5-2 s: the H2D copy's operands land a moment after the + * push, r507 matmul's 1 MB UVA copy at 0x7136... needed >8 fast passes), + * so the latch is TIME-based: the FIRST failure of a VA arms a + * SVC_OPDEF_HOLD_MS window during which the op holds its entry exactly as + * before; after a full bound with no success the op still attempts the + * write but no longer forces the hold - the entry consumes promptly and + * its withheld release applies, exactly the outcome the expired bound + * already sanctioned (op stays skipped, release delivered), minus the + * 10 s tax per subsequent pass. A success clears the record (a later + * failure gets a fresh window). Service-thread only, like every other op + * path. + */ +#define SVC_OPFAIL_MAX 32 +/* + * r507a: the op-failure window rides the opdef hold bound (10000 ms, + * SVC_OPDEF_HOLD_MS defined further down). Keep the literal here so the + * latch (used by svc_run_ops) precedes that define. + */ +#define SVC_OPFAIL_WINDOW_MS 10000u +static struct { + u64 va; + unsigned long since; +} svc_opfail[SVC_OPFAIL_MAX]; +static u32 svc_opfail_log; + +static void svc_opfail_clear(u64 va) +{ + u32 i; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) + if (svc_opfail[i].va == va) + svc_opfail[i].va = 0; +} + +static bool svc_opfail_latched(u64 va) +{ + u32 i; + u32 free_i = SVC_OPFAIL_MAX; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) { + if (svc_opfail[i].va == va) { + if (time_after_eq(jiffies, + svc_opfail[i].since + + msecs_to_jiffies(SVC_OPFAIL_WINDOW_MS))) { + if (svc_opfail_log < 32) { + svc_opfail_log++; + pr_info("sema-svc: OPFAIL-LATCH va=0x%llx (>=%u ms of failures; op no longer holds its entry)\n", + va, SVC_OPFAIL_WINDOW_MS); + } + return true; + } + return false; + } + if (!svc_opfail[i].va && free_i == SVC_OPFAIL_MAX) + free_i = i; + } + if (free_i < SVC_OPFAIL_MAX) { + svc_opfail[free_i].va = va; + svc_opfail[free_i].since = jiffies; + } + return false; +} + static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, - u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base) + u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base, + u32 entry_idx, u64 pass_hash) { u32 i; static u64 bigop_log; svc.cur_ch = ch_h; + /* r484: one-shot per call; set only by the hold->pass transition. */ + svc_wait_repass = false; + /* + * m163: apply every queued scanner injection before resolving this + * pass's copy operands. Without this, a CE copy decoded in the window + * between the client's push and its pins being processed resolves + * through the previous client's stale translation (or fails), while + * its release is still applied - the client's cudaMemcpy "completes" + * with no data and the next GR launch computes from the stale vidmem + * page (streams2 r163: a1 held the previous client's h_a; the vadd + * output was a_stale + b_fresh). + */ + svc_handle_map_add(); + /* + * r336/r34x: honor every host-semaphore acquire (C56F SEM_EXECUTE + * op 0/2/3/4/5) captured for THIS ring entry before any of its work + * runs, so a copy cannot execute ahead of the dependency its push + * waits on (the streams2 readback-before-scale class). Decision + * ladder per wait: + * 1. saved value satisfies the op's condition -> pass; + * 2. the entry's own release, when its method offset precedes the + * acquire's (release-then-acquire, channel.c crossEngineSync), + * counts as the candidate value - the release method has + * already executed by the time the acquire is evaluated; + * an acquire that precedes its own trailing tracking release + * does NOT count it (nothing has installed that value yet); + * 3. drain the pending-translation release queue (releases OUR + * machinery is merely late about) and re-read once; + * 4. still unsatisfied -> hold the entry (release withheld through + * the op-pending machinery) for at most gsp_sema_wait_hold_ms + * per episode, then fail open loudly ('give-up': the value + * cannot be installed by firmware alone - it sits behind an + * upstream opdef/operand hold - so the old pass-through + * behaviour is safer than parking the channel forever). + * Unreadable semaphore: fail-open (the pre-r336 behaviour). + * The pass/hold/give-up decisions are logged as 'sema-wait' rows + * (op, payload, observed value, decision). + */ + if (svc_wait_n && gsp_sema_wait_gate) { + struct svc_wait_hold *gh; + bool hold = false; + u32 wi; + /* + * Kill switch: a wait that never satisfies (a counter our + * emulation does not advance the way the driver expects) + * would otherwise park entries until the sweep expires each + * one. After 2048 holds in a boot the gate fails open + * permanently (the pre-r336 behaviour) and says so once. + */ + if (svc_wait_held > 2048) { + if (svc_wait_log < 4096) { + svc_wait_log++; + pr_info("sema-wait: gate disabled after %llu holds (fail-open)\n", + svc_wait_held); + } + svc_wait_pass++; + goto svc_wait_done; + } + gh = svc_wait_hold_get(ch_h, true); + for (wi = 0; wi < svc_wait_n; wi++) { + const struct svc_wait_op *w = &svc_waits[wi]; + u32 cur = 0, waited_ms = 0; + bool ok = true, same_rel = false, pend = false; + + if (gh) + waited_ms = jiffies_to_msecs(jiffies - gh->since); + if (svc_sem_read(vgpu, w->va, &cur)) + continue; /* unreadable: fail-open */ + ok = svc_wait_satisfied(cur, w->op, w->payload); + if (!ok && svc_wait_rel_have && + svc_wait_rel_va == w->va && + svc_wait_rel_off < w->off) { + same_rel = true; + ok = svc_wait_satisfied(svc_wait_rel_payload, + w->op, w->payload); + } + if (!ok) { + svc_apply_pending(vgpu); + if (!svc_sem_read(vgpu, w->va, &cur)) + ok = svc_wait_satisfied(cur, w->op, + w->payload); + } + if (ok) { + if (svc_wait_pass_log < 256) { + svc_wait_pass_log++; + pr_info("sema-wait: pass ch=0x%x va=0x%llx op=%u pay=%u cur=%u same_rel=%u waited_ms=%u -> ops run this pass\n", + ch_h, w->va, w->op, w->payload, + cur, same_rel, waited_ms); + } + continue; + } + pend = svc_wait_pend_release(w->va); + if (waited_ms >= gsp_sema_wait_hold_ms) { + /* + * Bounded episode expired: fail open for the + * WHOLE entry (one row, then the ops run). A + * per-wait continue here would ping-pong + * with a second unsatisfied wait: the first + * gave up and reset the shared clock before + * the second was judged, and the pair would + * hold forever 100 ms at a time. + */ + svc_wait_giveup++; + if (svc_wait_log < 2048) { + svc_wait_log++; + pr_info("sema-wait: give-up ch=0x%x va=0x%llx op=%u pay=%u cur=%u pend=%u waited_ms=%u -> fail-open (cannot be satisfied by firmware alone; ops run)\n", + ch_h, w->va, w->op, w->payload, + cur, pend, waited_ms); + } + if (gh) + gh->active = false; + hold = false; + svc_wait_pass++; + goto svc_wait_done; + } + hold = true; + if (svc_wait_log < 512) { + svc_wait_log++; + pr_info("sema-wait: hold ch=0x%x va=0x%llx op=%u pay=%u cur=%u same_rel=%u pend=%u waited_ms=%u -> entry held (acquire not satisfied)\n", + ch_h, w->va, w->op, w->payload, cur, + same_rel, pend, waited_ms); + } + break; /* one unsatisfied wait holds the entry */ + } + if (hold) { + svc_wait_held++; + svc_wait_hold_h = ch_h; + svc_wait_hold_idx = entry_idx; + svc.op_tr_pending = 1; + return; + } + /* + * r484: the gate just PASSED for an entry it previously held - + * the dependency is installed now, so this pass's ops must + * re-run even if the once-only bitmap says they already did + * (they ran against the pre-dependency state). + */ + if (ch_h == svc_wait_hold_h && + entry_idx == svc_wait_hold_idx) { + svc_wait_repass = true; + svc_wait_hold_idx = 0xffffffffu; + } + svc_wait_pass++; + if (gh) + gh->active = false; + } +svc_wait_done: for (i = 0; i < n_ops; i++) { struct svc_memset_op *m = &ops[i]; u32 seq = ops_base + i; @@ -4654,18 +7792,70 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * ran; re-running a CE copy would read a staging buffer the * client has since refilled and overwrite finished data. */ - if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) { + if (!svc_wait_repass && ops_done && seq < 64 && + ((*ops_done >> seq) & 1)) { svc.ops_once_skipped++; + /* + * r456: this op already ran in an earlier pass. Any + * staging snapshot a later re-decode pass re-captured + * for it (the capture sites run on every decode) is + * a leak that the r444 APPLY-free used to sweep; with + * that free removed (r456 keeps the snapshot alive + * across retries) drop it here so it can never serve + * a re-push of the ring slot. + */ + if (m->type == 1 && m->bytes <= SVC_COPY_SNAP_MAX) + svc_copy_snap_free(m->dst, m->src, m->bytes, + ch_h, entry_idx, pass_hash); continue; } - if (m->type == 1) - rc = svc_copy(vgpu, m->dst, m->src, m->bytes, - m->phys, m->aperture, - m->src_phys, m->src_aperture); - else - rc = svc_fill(vgpu, m->dst, m->value, m->elem, - m->count, m->phys, m->aperture); + { + u64 ce_len = (m->type == 1) ? m->bytes : + (u64)m->elem * m->count; + u64 ce_done = 0; + + if (m->type == 1) + rc = svc_copy(vgpu, m->dst, m->src, m->bytes, + m->phys, m->aperture, + m->src_phys, m->src_aperture, + &ce_done, m->rows, m->dpitch, + m->spitch, ch_h, entry_idx, + pass_hash); + else + rc = svc_fill(vgpu, m->dst, m->value, m->elem, + m->count, m->phys, m->aperture, + &ce_done, m->rows, m->dpitch); + /* + * CEAUDIT (2026-10-18 r220): per-op audit of the CE + * data path - the software executor of every guest + * cudaMemcpy - at/above SVC_COPY_AUDIT_MIN, including + * the rc != 0 attempts the copy-level COPYAUDIT cannot + * see (the pre-loop resolution misses return before + * any byte moves: done=0x0). src/dst are the guest + * VAs of the op; done is the byte count that landed. + * Bounded per boot. + */ + if (ce_len >= SVC_COPY_AUDIT_MIN && + svc_ceaudit_n < SVC_CEAUDIT_MAX) { + svc_ceaudit_n++; + /* + * A failing op is retried by the hold and would + * spam: throttle the rc != 0 repeats; successful + * big ops log unconditionally (budget-bounded). + */ + if (rc) + pr_info_ratelimited("vgsp: CEAUDIT op=%s src=0x%llx dst=0x%llx len=0x%llx done=0x%llx rc=%d\n", + m->type == 1 ? "copy" : "fill", + m->type == 1 ? m->src : 0ull, + m->dst, ce_len, ce_done, rc); + else + pr_info("vgsp: CEAUDIT op=%s src=0x%llx dst=0x%llx len=0x%llx done=0x%llx rc=%d\n", + m->type == 1 ? "copy" : "fill", + m->type == 1 ? m->src : 0ull, + m->dst, ce_len, ce_done, rc); + } + } if ((rc == 0 || rc == SVC_COPY_FORWARDED) && ops_done && seq < 64) *ops_done |= 1ULL << seq; @@ -4692,9 +7882,11 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, */ if (rc == SVC_COPY_FORWARDED) continue; - if (rc == 0) + if (rc == 0) { svc.memsets_done++; - else if (rc == -ENOENT) + /* r507a: a success re-arms the failure window. */ + svc_opfail_clear(m->dst); + } else if (rc == -ENOENT) svc.memsets_nomap++; else svc.memsets_fail++; @@ -4713,9 +7905,20 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * (observed r72 c2) must not withhold a release that gates a GR * launch with a 5 s poll deadline. On expiry the release is * flushed and the op stays skipped (the pre-r62 fallback). + * m163 extends the hold to copy ops whose destination is a + * device buffer: a lost small H2D that silently "completes" + * corrupts the next launch's input (streams2 r163: the 128 KB + * a1 copy's release was applied with no copy op ever run; the + * vadd computed a_stale + b and the client read the wrong c). + * A small failing device copy now costs a bounded hold and the + * retried decode runs it once the pins land - never silent + * corruption. Small failing memsets keep the old trade. */ if (rc != 0 && rc != SVC_COPY_FORWARDED && - m->bytes >= (256u << 10)) + (m->bytes >= (256u << 10) || + (m->type == 1 && m->dst >= SVC_VAS_FLOOR) || + (m->type == 0 && !m->phys && m->dst >= SVC_VAS_FLOOR)) && + !svc_opfail_latched(m->dst)) svc.op_tr_pending = 1; if (gsp_sema_dbg && m->bytes == 256 && svc.sem256_log < 64) { svc.sem256_log++; @@ -4726,7 +7929,7 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, } if (gsp_sema_dbg && ((rc == 0 && svc.memsets_done <= gsp_sema_ops_log) || - (rc != 0 && svc.memsets_fail + svc.memsets_nomap <= 64))) + (rc != 0 && svc.memsets_fail + svc.memsets_nomap <= 2048))) pr_info("sema-svc: %s dst=0x%llx value=0x%llx elem=%u count=%u bytes=%llu phys=%u ap=%u src=0x%llx srcphys=%u srcap=%u rc=%d h=0x%x\n", m->type == 1 ? "memcopy" : "memset", m->dst, m->value, m->elem, m->count, @@ -4744,7 +7947,7 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, * read-back signature: always keep room for it past the * general trace budget so the DtoH's src/dst/rc is on record. */ - if ((svc.op_trace_log < 160 && + if ((svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && (time_before(jiffies, svc.op_trace_until) || m->dst >= SVC_VAS_FLOOR || (m->type == 1 && !m->src_phys && m->src >= SVC_VAS_FLOOR))) || @@ -4973,6 +8176,8 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, bool skip; int i; u32 dbg = 0; + u64 rel_pa = 0; + u64 cnt_pa = 0; if (!h || !payload) { /* @@ -5000,7 +8205,8 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].h_object == h) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == h) { /* * The release targets the slot the push itself named * (release_pa learned from the release VA); the @@ -5009,12 +8215,23 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, * FORENSICS-grv7-counter F1). Fall back to counter_pa * when the release slot could not be resolved. */ - pa = svc.chan[i].release_pa ? - svc.chan[i].release_pa : svc.chan[i].counter_pa; + rel_pa = svc.chan[i].release_pa; + cnt_pa = svc.chan[i].counter_pa; + pa = rel_pa ? rel_pa : cnt_pa; break; } } spin_unlock_irqrestore(&svc.chan_lock, flags); + /* + * C1 (GAP-1 H3/H4): was the apply entered, and with which slot? + * Budget 512/session (reset in svc_session_flush); the per-branch + * prints below stay on their own counter_rel_dbg budget. + */ + if (gsp_sema_dbg && svc.crel_entry_dbg < 512) { + svc.crel_entry_dbg++; + pr_info("sema-diag: crel-entry h=0x%x va=0x%llx pay=%u inc=%d rel_pa=0x%llx cnt_pa=0x%llx\n", + h, sem_va, payload, inc ? 1 : 0, rel_pa, cnt_pa); + } if (!pa) { /* * Pool (UVM tracking) semaphores have no counter slot. The @@ -5051,7 +8268,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, wrc = svc_sem_write(vgpu, h, sem_va, payload); if (!wrc) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-apply h=0x%x va=0x%llx pay=%u cur=%u -> pool rescue\n", @@ -5072,6 +8289,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) { if (svc.chan[i].valid && + svc.chan[i].vgpu == vgpu && svc.chan[i].h_object == h) { if (payload > svc.chan[i].pending_rel_pay) { @@ -5124,6 +8342,15 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, return; } nv = inc ? cur + payload : payload; + /* + * r378: the slot PA was learned from the release VA (or registered + * from the client's pagemap) earlier; re-validate it against the + * current pin before the 4-byte release lands, so a slot the guest + * has since freed/re-homed is never written through the stale page. + */ + if (sem_va && !gsp_sema_write_guard(sem_va, pa, SVC_APERTURE_SYS, + "counter-rel")) + return; if (svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, &nv, sizeof(nv))) { if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; @@ -5135,7 +8362,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, svc.releases_applied++; svc.counter_rel_applied++; svc.gr_rel_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-apply h=0x%x pa=0x%llx pay=%u cur=%u -> written\n", @@ -5151,7 +8378,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h, * harmless for an absolute release (its guard drops it), but an increment * would advance the slot twice. */ -static void svc_counter_pending_clear(u32 h) +static void svc_counter_pending_clear(struct nv_vgpu_device *vgpu, u32 h) { unsigned long flags; int i; @@ -5160,7 +8387,8 @@ static void svc_counter_pending_clear(u32 h) return; spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].h_object == h) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == h) { svc.chan[i].pending_rel_pay = 0; svc.chan[i].pending_rel_inc = 0; break; @@ -5194,13 +8422,13 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu, if (payload > 0x100000) { ch->pending_rel_pay = 0; ch->pending_rel_inc = 0; - svc_counter_pending_clear(ch->h_object); + svc_counter_pending_clear(ch->vgpu, ch->h_object); return; } } else if (payload <= cur || payload - cur > 0x100000) { ch->pending_rel_pay = 0; ch->pending_rel_inc = 0; - svc_counter_pending_clear(ch->h_object); + svc_counter_pending_clear(ch->vgpu, ch->h_object); if (cur > ch->counter_last) ch->counter_last = cur; return; @@ -5211,13 +8439,13 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu, return; ch->pending_rel_pay = 0; ch->pending_rel_inc = 0; - svc_counter_pending_clear(ch->h_object); + svc_counter_pending_clear(ch->vgpu, ch->h_object); if (nv > ch->counter_last) ch->counter_last = nv; svc.releases_applied++; svc.counter_rel_applied++; svc.gr_rel_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg && svc.counter_rel_dbg < 256) { svc.counter_rel_dbg++; pr_info("sema-chan: rel-flush h=0x%x pa=0x%llx pay=%u cur=%u -> written\n", @@ -5259,13 +8487,13 @@ static struct { u64 pa_page; } svc_cnt_der[SVC_CNT_DER_MAX]; static u32 svc_cnt_der_n; +static u32 svc_cnt_der_log; static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u64 off = va & (SEMA_PAGE_SIZE - 1); u64 pa = 0; - int ap = SVC_APERTURE_SYS; u32 probe = 0; u32 i; bool fresh = true; @@ -5278,13 +8506,57 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) } } if (fresh) { - if (svc_walk_vaspaces(vgpu, page, &pa, &ap) && - svc_walk_discovered(vgpu, page, &pa, &ap) && - svc_targeted_resolve(vgpu, page, &pa)) - return -ENOENT; - if (ap != SVC_APERTURE_SYS) + u64 p = 0; + int a = SVC_APERTURE_SYS; + bool ok = false; + const char *src = "none"; + + /* + * m177b: first READABLE source wins. The previous chain kept + * the first source that merely PRODUCED a translation and then + * failed -EIO on an unreadable page without trying the rest: + * the published directory can yield a stale/incorrect chain + * for a counter VA while the rm_mem-content scan would resolve + * it (r194: the counter page 0x20400f000 never resolved; its + * walk hit probed unreadable and targeted was never reached). + */ + if (!svc_walk_vaspaces(vgpu, page, &p, &a) && + a == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "vaspaces"; + } + if (!ok && !svc_walk_discovered(vgpu, page, &p, &a) && + a == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "discovered"; + } + if (!ok && !svc_targeted_resolve(vgpu, page, &p) && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "targeted"; + } + if (!ok) { + if (svc_cnt_der_log < 16) { + svc_cnt_der_log++; + pr_info("sema-chan: counter page 0x%llx unreadable via vaspaces/discovered/targeted; slot stays pending\n", + page); + } return -ENOENT; - pa = (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + } + pa = (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + if (svc_cnt_der_log < 16) { + svc_cnt_der_log++; + pr_info("sema-chan: counter page 0x%llx resolved via %s (pa=0x%llx)\n", + page, src, pa); + } } /* Read the slot before trusting the mapping: an unreadable page is * not registered, so the caller keeps the "slot pending" fallback. */ @@ -5302,14 +8574,22 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) /* * Auto-register a GR channel's completion-counter slot from the observed - * layout, without waiting for a readable push: the counter records form a - * 16-byte-strided array in the client's counter page, allocated in channel - * order from the low end (SA-COUNTER FORENSICS-grv7-counter: the ascending - * slot fec0 + r*0x10 is the polled completion word; the push's REPORT_SEM - * release writes a DIFFERENT (descending) family and is tracked separately - * in release_pa). The slot PA comes from a host-verified pin or from the - * in-kernel structural derivation (svc_counter_derive), so the write target - * is a page the client actually polls. + * layout, without waiting for a readable push: the client zips its GR + * channels in ascending handle order onto DESCENDING 16-byte slots, so the + * rank-r channel's own completion word - the slot its pushes' REPORT_SEM + * release names, verified for all 20 ranks - is + * 0x20400fec0 + (SVC_GR_COUNTER_TOTAL-1-r)*0x10. The ascending formula + * (F1/eea5ea93) registered the completion word of the PAIRED rank (19-r) + * instead, so svc_counter_publish wrote each channel's ring PUT into + * another channel's waited word: the busy ctx-init channel rank 16 + * (h=0x5c000059, PUT~94) overwrote rank 3's completion word 0x20400ffc0 + * from 1 to 94 (witness `pub-adv h=0x5c000059 pa=...fc0 val=94 cur=1`), + * after which the rank-3 stream's own release (payload 1) was + * forward-only-skipped and its sync never completed (evtest3 + * cudaStreamSynchronize(st[2]); victory_streams_events). The slot PA comes + * from a host-verified pin or from the in-kernel structural derivation + * (svc_counter_derive), so the write target is a page the client actually + * polls. */ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, struct svc_channel *ch, int n_gr) @@ -5322,7 +8602,8 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, return; if (ch->gr_index >= n_gr) return; - va = 0x20400fec0ULL + (u64)ch->gr_index * 0x10ULL; + va = 0x20400fec0ULL + + (u64)(SVC_GR_COUNTER_TOTAL - 1 - ch->gr_index) * 0x10ULL; if (ch->counter_pa) { /* * FORENSICS-v2 anomaly 6: a slot derived in an earlier pass @@ -5350,6 +8631,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, ch->counter_pa = ipa & ~0x3ULL; for (j = 0; j < SVC_MAX_CHANNELS; j++) { if (svc.chan[j].valid && + svc.chan[j].vgpu == vgpu && svc.chan[j].h_object == ch->h_object) { svc.chan[j].counter_pa = ch->counter_pa; @@ -5398,6 +8680,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, spin_lock_irqsave(&svc.chan_lock, flags); for (j = 0; j < SVC_MAX_CHANNELS; j++) { if (svc.chan[j].valid && + svc.chan[j].vgpu == vgpu && svc.chan[j].h_object != ch->h_object && svc.chan[j].counter_pa == pa) { collision = true; @@ -5423,6 +8706,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, ch->counter_pa = pa; for (j = 0; j < SVC_MAX_CHANNELS; j++) { if (svc.chan[j].valid && + svc.chan[j].vgpu == vgpu && svc.chan[j].h_object == ch->h_object) { svc.chan[j].counter_pa = pa; break; @@ -5448,12 +8732,14 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, /* * Learn a GR channel's REPORT_SEM release slot from a release VA the push * carried (0x20400fXX observed). This is the slot the release write - * targets (release_pa); it is deliberately NOT the completion-counter slot - * (counter_pa), which the publish advances and which the client polls - * (SA-COUNTER FORENSICS-grv7-counter F1: green wrote both families of the - * counter page). An injected pin is the authoritative source; on a miss the - * slot is derived in-kernel (svc_counter_derive) so the release is authored - * immediately rather than waiting for the external pin watcher. + * targets (release_pa) and - per the descending layout the autoreg above + * documents - the channel's own polled completion word, the same slot the + * autoreg registers as counter_pa; keeping it in its own field preserves + * the push-named truth even when the registration derive fails, and + * svc_counter_publish prefers it. An injected pin is the authoritative + * source; on a miss the slot is derived in-kernel (svc_counter_derive) so + * the release is authored immediately rather than waiting for the external + * pin watcher. */ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va) { @@ -5482,7 +8768,8 @@ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va) spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].h_object == h) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == h) { if (svc.chan[i].release_pa != (pa & ~0x3ULL)) { svc.chan[i].release_pa = pa & ~0x3ULL; svc.chan[i].relearn_n++; @@ -5497,6 +8784,13 @@ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va) pr_info("sema-chan: release learned h=0x%x pa=0x%llx (from release va=0x%llx)\n", h, pa & ~0x3ULL, sem_va); } + /* + * r378: keep the VA the slot was learned from so a + * late write to release_pa (the stale-recover zero) + * can re-validate the target through + * gsp_sema_write_guard before it lands. + */ + svc.chan[i].release_va = sem_va; break; } } @@ -5529,7 +8823,7 @@ static bool svc_rel_apply_906f(struct nv_vgpu_device *vgpu, cur = (op == HOST_SEM_OP_REDUCTION) ? cur + payload : payload; if (!svc_sem_write(vgpu, ch->h_object, va, cur)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } } @@ -5557,7 +8851,7 @@ static bool svc_rel_apply_reportsem(struct nv_vgpu_device *vgpu, svc.pushes_pending++; if (!svc_sem_write(vgpu, ch->h_object, va, payload)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } return false; @@ -5568,7 +8862,33 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, u32 payload, u32 cur, bool pending, bool inc, bool inc_apply) { - if (pending || inc_apply) { + /* + * P4 (GAP-1 H3): the exact-slot counter delivery runs before every + * path that can early-return, so a GR completion-counter release can + * never be skipped by the generic write's outcome. The generic + * write/learn tail below is skipped entirely for counter VAs: an + * increment release is not idempotent, so it must apply exactly once + * (the early return here is that exactly-once edge). + */ + if (ch->engine_type == 0 && svc_counter_va(sem_va)) { + if (pending || inc_apply) + svc.pushes_pending++; + svc_counter_learn(vgpu, ch->h_object, sem_va); + svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, + inc && inc_apply); + return false; + } + /* + * r506w: generic absolute releases are forward-only by VALUE, whichever + * pass discovers the gap. The recorded pending bit came from the + * decode's read of the release word, and that read can be blinded by a + * stale/crossed landing (r506 witness: a withheld tracker release went + * to the flush with pending=0 while the fresh read still showed 0, and + * the release was then dropped by the counter fallback below - the + * client's tracker never advanced and its stream ground at the 10 s + * bound per push). Increments keep the one-shot claim gate (`inc_apply`). + */ + if (pending || inc_apply || (!inc && payload > cur)) { svc.pushes_pending++; if (inc) cur += payload; @@ -5579,19 +8899,41 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, * generic translation can be a stale content-scanned page; * those releases go through the injected-pin-backed exact slot * (svc_counter_learn / svc_counter_release_apply) below. + * Counter VAs are handled by the early block above; the check + * stays as belt-and-braces for the generic write. */ if (ch->engine_type == 0 && svc_counter_va(sem_va)) { /* handled by the exact-slot path */ } else if (!svc_sem_write(vgpu, ch->h_object, sem_va, cur)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return true; } } - if (ch->engine_type == 0) - svc_counter_learn(vgpu, ch->h_object, sem_va); - svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, - inc && inc_apply); + /* + * r506w: the exact-slot counter fallback is for COUNTER VAs only. Its + * `release_pa ?: counter_pa` fallback writes (and forward-only-skips + * against) the channel's REGISTERED slot, which belongs to a different + * word - the r506 witness: releases to the tracker VA 0x20480fef0 were + * skipped against the polluted PUT counter (cur=96) and lost. For a + * non-counter VA the generic write above is the delivery path. + */ + if (svc_counter_va(sem_va)) { + if (ch->engine_type == 0) + svc_counter_learn(vgpu, ch->h_object, sem_va); + svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, + inc && inc_apply); + } + /* + * Stall-class fix (R2): when the decode said this absolute release is + * genuinely pending and every write attempt above failed without + * queueing, the release must not vanish - retry it through the pending + * pass. Increments stay excluded (their one-shot claim makes + * re-queueing unsafe until a diag run proves otherwise). + */ + if (pending && !inc && !(ch->engine_type == 0 && + svc_counter_va(sem_va))) + svc_pending_add(sem_va, payload, 0); return false; } @@ -5695,6 +9037,93 @@ static void svc_fwd_rel_direct_note(u32 h, u64 va, u32 payload) * mapped to its register offset. Each completed QMD resets the accumulator, * so a push may carry more than one launch. */ +/* + * m164: per-decode QMD witness for the CLASS-2 no-capture hold. Set to the + * max number of QMD words any accumulator saw during the current + * svc_decode_stream() call; reset at its top and read by the entry loop's + * no-capture check. Service-thread only. + */ +static u32 svc_gr_qmd_seen; +static u64 svc_gr_nocap_log; + +/* + * r383: has a launch with this push VA ever been captured? Scans the live + * candidate ring, the recent-capture history and the r401 session-wide + * (push_va, hash) seen table. Used by the replayed-QMD dispatch below: a + * replay whose push was never captured is a launch that would otherwise be + * silently dropped (the streams2 second-vadd class); a replay of a + * captured/offered push is the exactly-once no-op. The hash is required + * for the seen table so a later, DIFFERENT push at the same VA (a loop + * iteration: same entry words, new content) is never misclassified as + * seen - the caller only consults this on a true replay anyway. + */ +static bool svc_gr_cand_push_seen(u64 push_va, u64 hash) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + svc_gr_cand_ring[i].push_va == push_va) + return true; + for (i = 0; i < SVC_GR_CAP_HIST; i++) + if (svc_gr_cap_hist[i] == push_va) + return true; + return svc_gr_seen_lookup(push_va, hash); +} + +/* + * r391f: once-only identity for GRCE inline writes. A push's + * LOAD_INLINE_DATA runs author bytes into the vidmem shadow; a re-decode of + * the same push (replay/rescue/rehold) must not re-run them - the target + * page may have been superseded in between (victory_reduce: the client's + * cudaMemset inline write re-zeroed the accumulator after the launch's + * copy-out, the sum=0 class). Keyed by (push_va, push content hash, dst, + * bytes): the release payload inside every push makes the hash unique per + * push, so a new client's identical-looking write never collides. Session + * scoped (cleared by svc_session_flush). Service-thread only. + * + * Sizing (r398): the ring must outlive the gap between a push's first + * execution and its next re-decode. The observed failing case had 76 + * inline writes between the two, which evicted a 64-slot table and let + * the memset re-run; 8192 slots (256 KB) gives a ~100x margin over the + * busiest observed burst. + */ +#define SVC_INLINE_ONCE_SLOTS 8192 +static struct { + u64 push_va; + u64 hash; + u64 dst; + u32 bytes; +} svc_inline_once[SVC_INLINE_ONCE_SLOTS]; +static u32 svc_inline_once_next; + +static bool svc_inline_once_seen(u64 push_va, u64 hash, u64 dst, u32 bytes) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_INLINE_ONCE_SLOTS; i++) + if (svc_inline_once[i].push_va == push_va && + svc_inline_once[i].hash == hash && + svc_inline_once[i].dst == dst && + svc_inline_once[i].bytes == bytes) + return true; + return false; +} + +static void svc_inline_once_store(u64 push_va, u64 hash, u64 dst, u32 bytes) +{ + u32 i = svc_inline_once_next++ % SVC_INLINE_ONCE_SLOTS; + + svc_inline_once[i].push_va = push_va; + svc_inline_once[i].hash = hash; + svc_inline_once[i].dst = dst; + svc_inline_once[i].bytes = bytes; +} + static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, struct svc_push_state *st, u32 addr, u32 count, const u8 *data, const u8 *push, u32 push_len, @@ -5724,6 +9153,8 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } } } + if (st->qmd_n > svc_gr_qmd_seen) + svc_gr_qmd_seen = st->qmd_n; if (st->qmd_have == 3) st->qmd_dev_va = (((u64)st->qmd_hi << 32) | st->qmd_lo) << 8; if (st->qmd_n == SVC_QMD_WORDS) { @@ -5757,13 +9188,70 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, memcpy(recs[n].data, rec->data, rec->len); n++; } - if (gsp_phys_gr_forward_enabled() && trusted && !replay) - svc_gr_cand_capture(ch, st, recs, n, push, push_len, - st->push_va); - else if (!replay || !gsp_phys_gr_forward_enabled()) + /* + * C10 (GAP-2 A5): duplicate destinations in the newest-first + * inline capture decide which record wins the backend's + * overlay; bounded and unconditional (gsp_sema_dbg is defined + * later in the file and this site may not reference it). + */ + if (svc_inline_recs_log < 64) { + u32 dup = 0, a2, b2; + + for (a2 = 0; a2 < n; a2++) + for (b2 = a2 + 1; b2 < n; b2++) + if (recs[a2].dst_va == + recs[b2].dst_va) + dup++; + svc_inline_recs_log++; + pr_info("sema-gr: inline recs n=%u dup_dst=%u\n", + n, dup); + } + if (gsp_phys_gr_forward_enabled()) { + u64 push_hash = (push && push_len) ? + svc_push_hash(push, push_len) : 0; + + if (trusted && !replay) { + svc_gr_cand_capture(ch, st, recs, n, push, + push_len, st->push_va); + } else if (trusted && replay && + !svc_gr_cand_push_seen(st->push_va, + push_hash)) { + /* + * r383 (streams2 second-vadd class): a + * complete QMD first seen on a replay pass + * fell through BOTH arms below - the + * forward path skips replays and the + * bridge arm needs !replay - so the + * launch was silently dropped (zero + * engine writes, client read stale/zero + * outputs). The push VA was never + * captured, so capture it now (trusted + * content only, as the first-sight path); + * a replay of an already captured/offered + * launch stays the exactly-once no-op. + */ + pr_info("sema-gr: REPLAY-CAPTURE h=0x%x push=0x%llx (complete QMD first seen on a replay pass; no capture on record)\n", + ch->h_object, st->push_va); + svc_gr_cand_capture(ch, st, recs, n, push, + push_len, st->push_va); + } else if (replay && svc_gr_nocap_log < 64) { + svc_gr_nocap_log++; + pr_info("sema-gr: LAUNCH-SKIP h=0x%x push=0x%llx (replay; already captured=%d trusted=%d)\n", + ch->h_object, st->push_va, + svc_gr_cand_push_seen(st->push_va, + push_hash), + trusted); + } else if (!replay) { + gsp_cuda_bridge_launch(vgpu, ch->h_object, + st->qmd, + st->qmd_dev_va, + st->push_va, recs, n); + } + } else if (!replay) { gsp_cuda_bridge_launch(vgpu, ch->h_object, st->qmd, st->qmd_dev_va, st->push_va, recs, n); + } st->qmd_n = 0; st->qmd_bits = 0; st->qmd_have = 0; @@ -5772,6 +9260,89 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } } +static u64 svc_pass_hash; /* r391f: this decode pass's push content hash */ + +/* + * r481 RISK-REDUCTION diagnostic (default 0): dump the full decoded method + * stream of the first few large GR-class pushes (len >= 1024) so the launch + * push's acquire/release method groups (or their absence) are visible + * verbatim. The streams_events class evidence shows the st2 vadd's copy-in + * running ~115 ms BEFORE its stream's H2D soft copies; the ordering the + * driver relied on is only provable by reading the push's own methods. + * Bounded: gsp_sema_push_raw dumps, at most once per push VA. + */ +unsigned gsp_sema_push_raw; +static u64 svc_pushraw_last_va; +static u32 svc_pushraw_n; + +static void svc_push_raw_dump(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, gi = 0; + + if (!gsp_sema_push_raw || svc_pushraw_n >= gsp_sema_push_raw) + return; + if (len < 512 || len > 8192) + return; + if (push_va == svc_pushraw_last_va) + return; + svc_pushraw_last_va = push_va; + svc_pushraw_n++; + pr_info("sema-pushraw: BEGIN h=0x%x va=0x%llx len=%u n=%u\n", + h_object, push_va, len, svc_pushraw_n); + while (off + 4 <= len && gi < 160) { + u32 hdr, op, cnt, subch, addr, v[4] = {0,0,0,0}, k, n; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 0x7; + cnt = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + addr = (hdr & 0x1fff) << 2; + n = cnt > 4 ? 4 : cnt; + for (k = 0; k < n && off + 8 + 4 * k <= len; k++) + memcpy(&v[k], buf + off + 4 + 4 * k, 4); + pr_info("sema-pushraw: [%u] off=0x%x op=%u sub=%u addr=0x%x cnt=%u v0=%08x v1=%08x v2=%08x v3=%08x\n", + gi, off, op, subch, addr, cnt, + v[0], v[1], v[2], v[3]); + if (op == 3 && cnt) { + off += 4 + cnt * 4; + gi++; + continue; + } + if (!cnt) { + gi++; + off += 4; + continue; + } + if (op == 1) { + off += 4 + cnt * 4; + gi++; + continue; + } + { + u32 p; + bool found = false; + + for (p = off + 4; p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 1 && + ((h2 >> 16) & 0xfff)) { + off = p; + found = true; + break; + } + } + if (!found) + off = len; + gi++; + } + } + pr_info("sema-pushraw: END h=0x%x va=0x%llx consumed=0x%x/%u\n", + h_object, push_va, off, len); +} + static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 len, u64 page_pa, bool allow_ops, struct svc_channel *ch, u32 entry_idx, @@ -5779,6 +9350,35 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u64 *ops_done) { struct svc_push_state st; + + /* r391f: the inline-write once-only key needs the push's identity. */ + svc_pass_hash = svc_push_hash(buf, len); + + /* + * r316: pending releases record the channel they were decoded for; + * the verified pool resolver is channel-keyed (see svc_pending_add + * and the matmul r322 starved/hung note). + */ + svc.cur_ch = ch->h_object; + /* + * r34x: the acquire-wait capture is PER RING ENTRY. It used to be + * reset only at SET_OBJECT, so a decode pass over a later entry (or + * a set of entries, since a held entry is re-decoded after the + * sweep walks ahead) leaked its acquire into an earlier entry's + * release-site gate: r340 held a D2H copy entry that carries no + * acquire at all on a later launch's 'wait for the next tracking + * value' (need=85 while the held entry releases 84) - a circular + * wait only our own withheld release could ever satisfy, so every + * hold expired via the 10 s fallback (wait=4542 etc.). Reset here, + * per entry; SET_OBJECT still resets at the push boundary, and the + * re-decode re-captures the same waits, so held entries re-evaluate + * identically. + */ + svc_wait_n = 0; + svc_wait_rel_va = 0; + svc_wait_rel_payload = 0; + svc_wait_rel_off = 0; + svc_wait_rel_have = false; /* * Static for the same reason as svc_handle_chan_dump: a 32-entry ops * array is a ~2 KB frame, and this is called from the single service @@ -5790,6 +9390,9 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 ops_seq = 0; /* ops appended this pass (sequence numbers) */ u32 off = 0; + /* m164: fresh CLASS-2 QMD witness for this decode pass. */ + svc_gr_qmd_seen = 0; + /* * A false VA->PA pair (the discovered map is content-scanned) can * point a ring entry's push VA at an unrelated page - observed: @@ -5804,6 +9407,13 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, allow_ops = false; } + /* + * r481: verbatim method-stream dump for large GR-class pushes + * (default 0 = off). See svc_push_raw_dump. + */ + if (gsp_sema_push_raw && ch && !ch->engine_type && len >= 512) + svc_push_raw_dump(buf, len, push_va, ch->h_object); + memset(&st, 0, sizeof(st)); st.push_va = push_va; /* @@ -5829,14 +9439,34 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc_fwd_defer_armed = false; svc.pages_decoded++; /* Diagnostic: name every push decoded in the post-completion window. */ - if (svc.op_trace_log < 160 && + if (svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && time_before(jiffies, svc.op_trace_until)) { svc.op_trace_log++; pr_info("sema-pushtrace: h=0x%x eng=%u va=0x%llx len=%u allow=%d trusted=%d\n", ch->h_object, ch->engine_type, push_va, len, allow_ops, trusted); } - if (gsp_sema_dbg && ch->engine_type && svc_dbg_streams < 96) { + /* + * fail4: de-windowed decode trace - every push the service looks at, + * bounded per session. The windowed/identified-channel diagnostics + * above can miss a whole client's push stream (unidentified channel, + * no bridge completion to arm the window); this record cannot. + */ + /* + * fail4 r302: with sema_dbg on the boot-time driver init spends the + * 512-line budget long before the run under diagnosis, leaving the + * interesting push stream (a later client's copies) invisible. The + * debug flag raises the bound so one boot covers init + run. + */ + if (svc.pushtrace_all < (gsp_sema_dbg ? 32768 : 512)) { + svc.pushtrace_all++; + pr_info("sema-pushall: h=0x%x eng=%u va=0x%llx pa=0x%llx len=%u allow=%d trusted=%d first8=%08x %08x\n", + ch->h_object, ch->engine_type, push_va, page_pa, len, + allow_ops, trusted, + len >= 4 ? *(const u32 *)(buf + 0) : 0, + len >= 8 ? *(const u32 *)(buf + 4) : 0); + } + if (gsp_sema_dbg && svc_dbg_streams < 96) { svc_dbg_streams++; pr_info("sema-dbg: stream h=0x%x eng=%u va=0x%llx len=%u allow=%d w=%08x %08x %08x %08x %08x %08x %08x %08x\n", ch->h_object, ch->engine_type, push_va, len, allow_ops, @@ -5884,28 +9514,195 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, addr = (hdr & 0x1fff) << 2; /* + * fail4: name every big method-data run and every run whose + * address is the LOAD_INLINE_DATA method (any op/subch - the + * encoding of the 64 KB H2D upload is what we are pinning). + */ + if (gsp_sema_dbg && len >= 8192 && count >= 1024 && + svc.bigrun_log < 128) { + svc.bigrun_log++; + pr_info("sema-bigrun: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x have_out=%u out=0x%llx line=%u lcnt=%u w=%08x %08x\n", + ch->h_object, push_va, off, hdr, op, count, subch, + addr, st.have_out, st.offset_out, st.line_length, + st.gr_line_count, + off + 8 <= len ? *(const u32 *)(buf + off + 4) : 0, + off + 12 <= len ? *(const u32 *)(buf + off + 8) : 0); + } + if (gsp_sema_dbg && len >= 8192 && + (addr == GRCE_M_LOAD_INLINE_DATA || addr == 0x1b8) && + svc.lid_hdr_log < 64) { + svc.lid_hdr_log++; + pr_info("sema-lid-hdr: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x trusted=%d allow=%d\n", + ch->h_object, push_va, off, hdr, op, count, subch, + addr, trusted, allow_ops); + } + + /* * GRCE LOAD_INLINE_DATA: a data run (opcode 3) that carries * the DMA source bytes in the push itself. Run the write to * the destination the preceding methods selected. */ if (op == 3 && subch == 1 && addr == GRCE_M_LOAD_INLINE_DATA && count && off + 4 + count * 4 <= len) { - if ((trusted || allow_ops) && !gsp_sema_skip_ops && - st.have_out && st.line_length) { - u32 bytes = st.line_length * - (st.gr_line_count ? st.gr_line_count : 1); - u32 have = count * 4; + /* + * fail4 slot rule: in the UVM upload pushes each + * LOAD_INLINE_DATA run fills a whole 32 KB slot - its + * data runs from the header to the next run header + * (or to the next 0x8000 boundary when this is the + * last run), regardless of the count field. A compact + * push (no room for a following slot) keeps the + * count*4 reading (the small parameter banks). The + * old count*4 reading truncated the big runs to half + * their slot and left the tail to be misparsed as + * methods (the split-upload bug). + */ + bool slot_run = (off + 0x8000 < len); + u32 slot_end = off + 0x8000; + u32 slot_bytes = 0; + /* + * fail4: the bridge's argument-block search needs the + * run's START as its destination; the cursor advance + * below must not leak into the rec (an end-of-run + * dst_va makes the QMD param patch miss and the + * engine never consumes the launch). + */ + u64 run_dst = st.offset_out; + + if (slot_run) { + u32 p; + + for (p = off + 4 + count * 4; + p + 4 <= slot_end && p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 3 && + ((h2 >> 13) & 0x7) == 1 && + ((h2 & 0x1fff) << 2) == + GRCE_M_LOAD_INLINE_DATA && + ((h2 >> 16) & 0xfff)) { + slot_end = p; + break; + } + } + slot_bytes = slot_end - (off + 4); + if (slot_bytes > 0x8000 - 4) + slot_bytes = 0x8000 - 4; + } + /* + * r391f once-only (the sum=0 class): the same + * inline write must not run twice for one push + * identity. The ops are gated by allow_ops + * (entry_replay clears it), but the inline-write + * gate `trusted || allow_ops` re-ran the write on + * every re-decode (a replay is trusted by + * construction): victory_reduce's + * cudaMemset(d_sum,0,8) is exactly such an inline + * write (observed: `inline write + * dst=0x7845ef000000 -> pa=0x7c00000 bytes=8 + * data8=0` firing 86 us before the D2H readback, + * zeroing the accumulator the launch had just + * written - the client then read zeros). A coarse + * !replay gate is wrong (a push whose first pass + * was untrusted must still author its bytes on the + * later trusted pass); key the write by + * (push_va, push content hash, dst, bytes) - a new + * push's release payload makes the hash unique - + * and skip only a genuine duplicate. The table is + * session-scoped (a new client starts clean). + */ + { + u32 bytes = slot_run ? slot_bytes : + st.line_length * + (st.gr_line_count ? + st.gr_line_count : 1); + u32 have = slot_run ? slot_bytes : count * 4; if (bytes > have) bytes = have; - svc_grce_inline_write(vgpu, st.offset_out, - buf + off + 4, bytes, - ch->h_object); - } else if (gsp_sema_dbg && svc.grce_log < 16) { - svc.grce_log++; - pr_info("sema-grce: inline data not run (trusted=%d allow=%d have_out=%d line=%u) h=0x%x\n", - trusted, allow_ops, st.have_out, - st.line_length, ch->h_object); + /* + * r391f once-only (the sum=0 class): the same + * inline write must not run twice for one push + * identity. The ops are gated by allow_ops + * (entry_replay clears it), but the + * inline-write gate `trusted || allow_ops` + * re-ran the write on every re-decode (a replay + * is trusted by construction): victory_reduce's + * cudaMemset(d_sum,0,8) is exactly such an + * inline write (observed: `inline write + * dst=0x7845ef000000 -> pa=0x7c00000 bytes=8 + * data8=0` firing 86 us before the D2H + * readback, zeroing the accumulator the launch + * had just written - the client then read + * zeros). A coarse !replay gate is wrong (a push + * whose first pass was untrusted must still + * author its bytes on the later trusted pass); + * key the write by (push_va, push content hash, + * dst, bytes) - the release payload inside every + * push makes the hash unique per push - and skip + * only a genuine duplicate. Session-scoped: a new + * client starts clean. + */ + if ((trusted || allow_ops) && !gsp_sema_skip_ops && + st.have_out && st.line_length && bytes && + !svc_inline_once_seen(push_va, svc_pass_hash, + st.offset_out, + bytes)) { + svc_inline_once_store(push_va, + svc_pass_hash, + st.offset_out, + bytes); + svc_grce_inline_write(vgpu, st.offset_out, + buf + off + 4, + bytes, + ch->h_object); + /* + * fail4: name the header that follows + * each big LID run - the split 64 KB + * upload's other payload carriers sit + * exactly there. + */ + if (gsp_sema_dbg && len >= 4096 && + svc.next_hdr_log < 24) { + u32 noff = off + 4 + count * 4; + u32 nh = 0, nw = 0; + + if (noff + 4 <= len) + memcpy(&nh, buf + noff, 4); + if (noff + 8 <= len) + memcpy(&nw, buf + noff + 4, + 4); + svc.next_hdr_log++; + pr_info("sema-next-hdr: h=0x%x push=0x%llx after=0x%x cnt=%u nxt_hdr=0x%08x nxt_w0=%08x cursor=0x%llx\n", + ch->h_object, push_va, + off, count, nh, nw, + st.offset_out); + } + } else if (gsp_sema_dbg && svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: inline data not run (trusted=%d allow=%d have_out=%d line=%u) h=0x%x\n", + trusted, allow_ops, st.have_out, + st.line_length, ch->h_object); + } + /* + * fail4 (r526): hardware semantics - a + * LOAD_INLINE_DATA run advances the destination + * cursor by the bytes it carries whether or not + * the once-only gate let the write run. Without + * this, the consecutive runs of a split upload + * all land at the first destination (the old + * RAWA bug), and worse: on a re-decode pass where + * the gate suppresses the FIRST run, the cursor + * stays at the first run's destination and the + * SECOND run executes there - its data (the + * buffer's bytes 8..16) overwriting offset 0 + * (the observed tgemm XDIFF: X[0..1] <- + * X[2..3], deterministic, every pass). The gate + * suppresses the write, not the method stream's + * positional semantics. + */ + if (bytes) + st.offset_out += bytes; } /* * Record the destination and the payload bytes for @@ -5915,21 +9712,43 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * kernel arguments in the QMD parameter bank, and the * destination VA may not be readable later. */ - if (st.have_out && st.offset_out) { + if (st.have_out && run_dst) { struct svc_grce_rec *rec = &svc_grce_dst_ring[svc_grce_dst_ring_next % SVC_GRCE_DST_MAX]; - rec->dst_va = st.offset_out; - rec->len = min_t(u32, count * 4, + rec->dst_va = run_dst; + rec->len = min_t(u32, + slot_run ? slot_bytes : count * 4, GSP_BRIDGE_INLINE_BYTES); memcpy(rec->data, buf + off + 4, rec->len); svc_grce_dst_ring_next++; } - off += 4 + count * 4; + /* + * fail4 slot rule: hop to the next slot's run header + * instead of walking the slot's raw tail as methods. + */ + off += slot_run ? (slot_end - off) : 4 + count * 4; continue; } + /* + * fail4: name the moderate op=1 method runs on big pushes - + * the split 64 KB upload's payload carriers beyond the LID + * class land here (their addr selects the write port). + */ + if (gsp_sema_dbg && len >= 4096 && op == 1 && + count >= 16 && count <= 1023 && svc.mystery_log < 48) { + svc.mystery_log++; + pr_info("sema-mystery: h=0x%x push=0x%llx off=0x%x hdr=0x%08x cnt=%u subch=%u addr=0x%x have_out=%u out=0x%llx line=%u lcnt=%u w=%08x %08x %08x\n", + ch->h_object, push_va, off, hdr, count, subch, + addr, st.have_out, st.offset_out, + st.line_length, st.gr_line_count, + off + 8 <= len ? *(const u32 *)(buf + off + 4) : 0, + off + 12 <= len ? *(const u32 *)(buf + off + 8) : 0, + off + 16 <= len ? *(const u32 *)(buf + off + 12) : 0); + } + if (op == 4 || op == 7) { /* Header-only methods (immediate data in the header * for op 4; end-of-push segment for op 7). */ @@ -5994,6 +9813,13 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * held INC-shaped words queued a bogus LAUNCH_DMA). * No release is decoded from these encodings. */ + if (gsp_sema_dbg && len >= 8192 && count >= 1024 && + svc.skiprun_log < 64) { + svc.skiprun_log++; + pr_info("sema-skiprun: h=0x%x push=0x%llx off=0x%x hdr=0x%08x op=%u cnt=%u subch=%u addr=0x%x trusted=%d allow=%d\n", + ch->h_object, push_va, off, hdr, op, + count, subch, addr, trusted, allow_ops); + } off += 4 + count * 4; continue; } @@ -6060,6 +9886,12 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, /* SET_OBJECT on the channel subchannel: a push starts */ n_ops = 0; memset(&st, 0, sizeof(st)); + /* r336/r34x: new push, new acquire set + release mirror */ + svc_wait_n = 0; + svc_wait_rel_va = 0; + svc_wait_rel_payload = 0; + svc_wait_rel_off = 0; + svc_wait_rel_have = false; off += 4 + count * 4; continue; } @@ -6078,6 +9910,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc.pushes_seen++; st.rel_va = va; st.rel_payload = payload; + svc_wait_rel_va = va; + svc_wait_rel_payload = payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; svc_rel_va_seen = va; svc_rel_count++; @@ -6085,7 +9921,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * REPORT_SEM comment below. */ if (allow_ops && n_ops && !gsp_sema_skip_ops) { svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -6130,7 +9967,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, or.va = va; or.payload = payload; or.entry_idx = entry_idx; - if (svc_fwd_defer_arm()) { + if (svc_fwd_defer_arm() && + !svc.op_tr_pending) { svc_opdef_drop(vgpu, ch, entry_idx); if (svc_fwd_rel_n < SVC_FWD_REL_MAX) { @@ -6175,6 +10013,42 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, va, payload, op); } } + } else if (op != HOST_SEM_OP_RELEASE && + va >= SEMA_VA_LO && va < SEMA_VA_HI) { + /* + * r336/r34x: C56F SEM_EXECUTE acquire ops + * (0 = ACQUIRE/equal, 2 = strict GEQ, + * 3 = circular GEQ, 4 = AND, 5 = NOR). + * Core CUDA starts its compute-channel + * pushes with one of these to wait for a + * progress counter (observed: + * [LO=0x0400ffd0][HI=2][pay=4][0][0x1003] on + * the D2H readback push - ACQ_CIRC_GEQ on + * the channel counter 0x20400ffd0). + * svc_run_ops honors every captured acquire + * of this entry before running its work, so + * a copy cannot execute ahead of the + * dependency it waits on (the streams2 + * readback-before-scale class). Capturing + * into the per-entry list (instead of a + * single last-wait global) is what stops a + * later entry's acquire from gating an + * earlier entry (the r340 over-hold). + * PAYLOAD_SIZE 64BIT (execute bit 24) is + * captured on the low word only; every + * observed CUDA acquire is 32-bit. + */ + if (svc_wait_n < SVC_WAIT_MAX) { + struct svc_wait_op *w = + &svc_waits[svc_wait_n++]; + + w->va = va; + w->payload = payload; + w->op = (u8)op; + w->off = off; + } else { + svc_wait_ovf++; + } } off += 4 + count * 4; continue; @@ -6224,6 +10098,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc.pushes_seen++; st.rel_va = va; st.rel_payload = st.rs_payload; + svc_wait_rel_va = va; + svc_wait_rel_payload = st.rs_payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; svc_rel_va_seen = va; svc_rel_count++; @@ -6244,7 +10122,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, */ if (allow_ops && n_ops && !gsp_sema_skip_ops) { svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -6359,7 +10238,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * operand state, so one run covers both the * C5B5 and the GRCE copy shapes. */ - if (ch->h_object && gsp_sema_dbg && + if (gsp_sema_dbg && svc.xfer_raw_log < 48) { u32 dl = len < 192 ? len : 192; @@ -6401,11 +10280,61 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.have_sem = 1; break; case CE_M_SET_REMAP_CONST_A: + /* + * r343 FILL instrument: a fill-shaped LAUNCH_DMA + * (REMAP_ENABLE + destination PITCH, OFFSET_OUT and + * no OFFSET_IN) needs this constant to build its op. + * The 8MB-memset class (memcpy_roundtrip all-bad, + * victory_memset2d) shows fill-shaped launches with + * have_remap=0 that decode nothing and are consumed + * silently. Log every decode (bounded) so one run + * captures the real run shape even when the count + * guard below cannot use it. + */ + if (svc_remapconst_log < 256) { + svc_remapconst_log++; + pr_info("sema-remapconst: h=0x%x push=0x%llx off=0x%x count=%u v0=0x%x v1=0x%x v2=0x%x trusted=%d allow=%u\n", + ch->h_object, push_va, off, count, + v0, count >= 2 ? v1 : 0u, + count >= 3 ? v2 : 0u, trusted, allow_ops); + } if (count >= 3) { st.remap_a = v0; st.remap_b = v1; st.remap_comp = v2; st.have_remap = 1; + st.have_remap_a = 1; + st.have_remap_b = 1; + st.have_remap_c = 1; + } else if (count >= 1) { + /* + * r365: the driver also pushes the fill constants + * as SEPARATE count=1 methods (the CE + * page-table fills carry only SET_REMAP_CONST_A + * with A=0). Capture the individual parts but do + * NOT set have_remap: r365b proved that letting + * these pass the fill guard executes below-floor + * window fills (0x2000c0xxx, 0x204020000) whose + * svc_fill resolution landed in GUEST RAM and + * zeroed slab pages (guest kernel GPF in + * kmem_cache_alloc -> panic at ~75 s). The + * window fills only run behind gsp_sema_winfill + * with the correct resolution proven first. + */ + st.remap_a = v0; + st.have_remap_a = 1; + } + break; + case CE_M_SET_REMAP_CONST_B: + if (count >= 1) { + st.remap_b = v0; + st.have_remap_b = 1; + } + break; + case CE_M_SET_REMAP_COMPONENTS: + if (count >= 1) { + st.remap_comp = v0; + st.have_remap_c = 1; } break; case CE_M_SET_SRC_PHYS_MODE: @@ -6418,12 +10347,42 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, break; case CE_M_LINE_LENGTH_IN: st.line_length = v0; + /* + * r458 THE memset2d root cause: the driver encodes + * LINE_LENGTH_IN (0x418) and the adjacent LINE_COUNT + * (0x41c) as ONE count=2 method group - observed raw + * push bytes `20028106 00001000 00000100` = header + * (subch 4, count 2, addr 0x418), line length 4096, + * line count 0x100 = 256. Reading only v0 left + * line_count = 0, so the 2D-copy builder below took + * the single-row path: one 4 KB op per push, and a + * 4 MB cudaMemcpy2D delivered exactly one page per + * 1 MB block (memset2d probe: per-MB ok=4096, + * bad=12.5M/12.58M). The GRCE variant below has + * always decoded the count=2 form; the CE path did + * not. + */ + if (count >= 2) + st.line_count = v1; + break; + /* r402: 2D geometry (cudaMemset2D / cudaMemcpy2D). */ + case CE_M_LINE_COUNT: + st.line_count = v0; + break; + case CE_M_PITCH_IN: + st.pitch_in = v0 & 0xffffff; + st.have_pitch_in = 1; + break; + case CE_M_PITCH_OUT: + st.pitch_out = v0 & 0xffffff; + st.have_pitch_out = 1; break; case CE_M_LAUNCH_DMA: { u32 dma = v0; u32 transfer = DMA_DATA_TRANSFER_TYPE(dma); + u32 n_ops_before = n_ops; - if (gsp_sema_dbg && ch->h_object && + if (gsp_sema_dbg && svc.lda_log < (gsp_sema_ops_log > 48 ? gsp_sema_ops_log : 48)) { svc.lda_log++; pr_info("sema-lda: h=0x%x eng=%u push=0x%llx inj=%u allow=%u dma=0x%08x xfer=%u remap=%u dpitch=%u dstphys=%u srcphys=%u sem=%u out=0x%llx in=0x%llx line=%u have_out=%u have_in=%u have_remap=%u have_sem=%u ops=%u\n", @@ -6440,7 +10399,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* Diagnostic: full operand state in the post-completion * window (or for high device VAs), bounded. */ - if (svc.op_trace_log < 160 && ch->h_object && + if (svc.op_trace_log < (gsp_sema_dbg ? 32768 : 2048) && ch->h_object && (time_before(jiffies, svc.op_trace_until) || st.offset_out >= SVC_VAS_FLOOR || st.offset_in >= SVC_VAS_FLOOR)) { @@ -6464,7 +10423,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * bounded number of such launches so the operand * encoding (or its absence) is on record. */ - if (transfer != 0 && ch->h_object && gsp_sema_dbg && + if (transfer != 0 && gsp_sema_dbg && svc.xfer_raw_log < 48 && (!st.have_out || !st.line_length)) { u32 dl = len < 192 ? len : 192; @@ -6479,25 +10438,182 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, if (allow_ops && transfer != 0 && st.line_length && st.have_out) { - if (DMA_REMAP_ENABLE(dma) && st.have_remap && + /* + * r361: a fill-shaped transfer whose + * destination is PHYSICAL guest RAM (the + * driver zeroes freshly-allocated PTE / + * page-directory pages with exactly this + * push shape) needs no translation: the + * destination already IS the GPA. The old + * `offset_out >= SVC_VAS_FLOOR` guard + * compared the GPA against the VA floor + * and blocked every one of these, so the + * PD0 pages the GMMU walk later reads + * stayed stale and the first CUDA client + * of a fresh boot dropped operand pages + * (GRFILL-DROP). SEMA_PA_MAX bounds the + * range so a ramfill can only touch guest + * RAM; virtual engine-window fills + * (0x2040xxxxx) stay excluded. + */ + bool ramfill = DMA_DST_TYPE_PHYSICAL(dma) && + st.have_dst_mode && + st.dst_phys_mode == CE_APERTURE_SYS && + st.offset_out + (u64)st.line_length <= + SEMA_PA_MAX; + /* r365: vidmem-window fill (see gsp_sema_winfill). */ + bool winfill = gsp_sema_winfill && + st.offset_out >= SVC_VIDMEM_VA_LO && + st.offset_out < SVC_VIDMEM_VA_BASE + + svc_fb_shadow_size(vgpu); + + if (DMA_REMAP_ENABLE(dma) && + (st.have_remap || + (st.offset_out >= SVC_VAS_FLOOR) || + ramfill || winfill) && DMA_DST_MEMORY_LAYOUT_PITCH(dma) && n_ops < SVC_MAX_MEMSETS) { - u32 csz = (st.remap_comp >> 16) & 0x3; - u32 ndst = (st.remap_comp >> 24) & 0x3; + u32 csz = st.have_remap ? + ((st.remap_comp >> 16) & 0x3) : 0; + u32 ndst = st.have_remap ? + ((st.remap_comp >> 24) & 0x3) : 0; + u32 dst_x = st.have_remap ? + (st.remap_comp & 0x7) : 4; struct svc_memset_op *m = &ops[n_ops]; m->type = 0; m->dst = st.offset_out; - m->value = ndst >= 1 ? - (((u64)st.remap_b << 32) | - st.remap_a) : st.remap_b; - m->elem = (csz + 1) * (ndst + 1); - m->count = st.line_length; + /* + * r367: the driver's own memset push + * (channel_utils.c) proves the + * encoding: LINE_LENGTH_IN is the + * fill SIZE IN BYTES, and + * SET_REMAP_COMPONENTS selects + * COMPONENT_SIZE ([17:16], 0..3 -> + * 1/2/3/4 bytes) and + * NUM_DST_COMPONENTS ([25:24], n+1 + * components). elem = + * (csz+1)*(ndst+1) is the pattern + * width; count must therefore be + * line_length/elem - the old + * count=line_length wrote + * elem x too much (r365: a 383620- + * byte memset wrote 1534480 bytes, + * clobbering the bytes after it). + * The value comes from the mapped + * constants: X<-CONST_A (4) and + * Y<-CONST_B (5); with two + * components the pattern is + * B:A 64-bit. + */ + if (st.have_remap) + m->value = ndst >= 1 ? + (((u64)st.remap_b << 32) | + st.remap_a) : + (dst_x == 5 ? st.remap_b : + st.remap_a); + else + /* + * r357: a fill-shaped transfer + * (REMAP_ENABLE + DST PITCH, + * no source) to a CLIENT DEVICE + * VA whose value constant did + * not decode is a ZERO fill: + * the driver's page-table + * clears and cudaMemset(0) + * issue exactly this shape with + * no SET_REMAP_CONST method. + * Building no op left the + * guest's GPU page tables + * stale (the GMMU walk then + * missed freshly-mapped + * operand pages and the first + * CUDA client of every fresh + * boot ran with dropped + * operands). The >= floor + * guard keeps engine-window + * fills (0x2040xxxxx init + * clears) OUT of the executor: + * executing those clobbered + * emulated engine state and + * hung client init (r356). + */ + m->value = 0; + m->elem = st.have_remap ? + (csz + 1) * (ndst + 1) : 1; + if (m->elem > 1 && + st.line_length % m->elem) { + /* not pattern-divisible: exact + * byte count wins */ + m->elem = 1; + m->count = st.line_length; + } else { + m->count = st.line_length / + m->elem; + } + /* + * r402: a 2D fill (cudaMemset2D): + * LINE_COUNT rows of + * LINE_LENGTH_IN bytes at + * PITCH_OUT stride. A pitch WIDER + * than the row selects row mode; a + * flat fill carries pitch == row + * length (or no pitch method) and + * keeps the historical single-run + * path. + * + * r420c: LINE_COUNT is + * authoritative for the flat case + * too - cudaMemset2D with pitch == + * width (victory_memset2d, + * pitch=4096/w=4096) is CONTIGUOUS + * rows, so the single run must + * cover rows*line bytes. The + * historical path filled one row + * and the readback lost the rest + * (bad ~= 12.5M of 12.58M). + */ + if (st.line_count > 1) { + u32 fpitch = st.have_pitch_out ? + st.pitch_out : + st.line_length; + + if (fpitch > st.line_length) { + m->rows = + st.line_count; + m->dpitch = fpitch; + } else if (st.line_length % + m->elem) { + m->elem = 1; + m->count = + st.line_length * + st.line_count; + } else { + m->count = + (st.line_length / + m->elem) * + st.line_count; + } + } + if (m->rows > 1 && svc_pitch_log < 32) { + svc_pitch_log++; + pr_info("sema-2d: fill rows=%u dpitch=%u line=%u dst=0x%llx h=0x%x push=0x%llx\n", + m->rows, m->dpitch, + st.line_length, m->dst, + ch->h_object, push_va); + } m->phys = DMA_DST_TYPE_PHYSICAL(dma); m->aperture = st.have_dst_mode ? st.dst_phys_mode : CE_APERTURE_VID; n_ops++; ops_seq++; + if (ramfill && svc.fill_ram_log < 24) { + svc.fill_ram_log++; + pr_info("vgsp-sema: RAMFILL-EXEC push=0x%llx dst=0x%llx line=%u ap=%u value=0x%llx (physical guest-RAM fill built)\n", + push_va, st.offset_out, + st.line_length, + m->aperture, m->value); + } } else if (!DMA_REMAP_ENABLE(dma) && st.have_in && n_ops < SVC_MAX_MEMSETS) { @@ -6514,6 +10630,80 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, DMA_SRC_TYPE_PHYSICAL(dma); m->src_aperture = st.have_src_mode ? st.src_phys_mode : CE_APERTURE_VID; + /* + * r402: a 2D copy (cudaMemcpy2D): + * LINE_COUNT rows of + * LINE_LENGTH_IN bytes with + * PITCH_IN/PITCH_OUT strides. + * Row mode when a pitch is + * wider than the row. + * + * r420c: pitch == row length + * (or no pitch method) makes the + * rows CONTIGUOUS, so the single + * flat run must cover + * rows*line bytes - the + * historical path copied one row + * (victory_memset2d's pitched + * D2H/H2D at pitch 4096 = width + * 4096 lost all but the first + * row). + */ + if (st.line_count > 1) { + u32 dpitch = st.have_pitch_out ? + st.pitch_out : + st.line_length; + u32 spitch = st.have_pitch_in ? + (st.pitch_in ? + st.pitch_in : + dpitch) : + dpitch; + + if (dpitch == st.line_length && + spitch == st.line_length) { + m->bytes = + st.line_length * + st.line_count; + } else { + m->rows = + st.line_count; + m->dpitch = dpitch; + m->spitch = spitch; + } + } + /* + * r458 memset2d forensics: the + * driver's cudaMemcpy2D pushes decode + * LINE_LENGTH=4096 with PITCH_IN/OUT + * present but LINE_COUNT absent + * (lc=0), so only one 4 KB row per + * push executes and a 4 MB copy + * delivers 4 pages. Dump the raw push + * (bounded) for every pitch-shaped + * copy so the method stream that + * carries the row count is on record. + */ + if (gsp_sema_dbg && + (st.have_pitch_out || + st.have_pitch_in) && + svc.op2d_raw_log < 12) { + u32 dl = len < 220 ? len : 220; + + svc.op2d_raw_log++; + pr_info("sema-op2d-raw: h=0x%x push=0x%llx len=%u off=0x%x line=%u lc=%u po=%u/%u pi=%u/%u out=0x%llx in=0x%llx bytes=%u dma=0x%08x\n", + ch->h_object, push_va, + len, off, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in, + m->dst, m->src, + (u32)m->bytes, dma); + pr_info("sema-op2d-raw: raw %*phN\n", + dl, buf); + } n_ops++; ops_seq++; } else if (gsp_sema_dbg && svc.dma_skip_log < 32) { @@ -6542,6 +10732,317 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* + * r365 FILL-RAW: the fill class that still decodes no + * op (window VA, no remap const decoded) - dump the + * raw push and decode its method stream so the true + * value/component encoding is on record. Bounded. + */ + if (allow_ops && transfer != 0 && + DMA_REMAP_ENABLE(dma) && + DMA_DST_MEMORY_LAYOUT_PITCH(dma) && + st.have_out && !st.have_in && !st.have_remap && + svc_fill_raw_log < 48) { + u32 dl = len < 160 ? len : 160; + u32 mo = 0, mn = 0; + + svc_fill_raw_log++; + pr_info("sema-fill-raw: h=0x%x push=0x%llx out=0x%llx line=%u dma=0x%08x dmode=%u have_dstmode=%u trusted=%d allow=%d\n", + ch->h_object, push_va, st.offset_out, + st.line_length, dma, st.dst_phys_mode, + st.have_dst_mode, trusted, allow_ops); + pr_info("sema-fill-raw: raw len=%u %*phN\n", + len, dl, buf); + /* decode the first method pairs */ + while (mo + 8 <= len && mn < 14) { + u32 h2, c2, a2; + + memcpy(&h2, buf + mo, 4); + c2 = (h2 >> 16) & 0xfff; + a2 = (h2 & 0x1fff) << 2; + if ((h2 >> 29) != 1 || !c2 || + mo + 4 + c2 * 4 > len) + break; + if (c2 >= 2) + pr_info("sema-fill-raw: m[%u] addr=0x%x cnt=%u v0=0x%08x v1=0x%08x\n", + mn, a2, c2, + *(const u32 *)(buf + mo + 4), + *(const u32 *)(buf + mo + 8)); + else + pr_info("sema-fill-raw: m[%u] addr=0x%x cnt=%u v0=0x%08x\n", + mn, a2, c2, + *(const u32 *)(buf + mo + 4)); + mo += 4 + c2 * 4; + mn++; + } + } + + /* + * r343 FILL diagnostic (LOG-ONLY, no hold): a + * fill-shaped transfer (REMAP_ENABLE + destination + * PITCH, OFFSET_OUT and no OFFSET_IN) whose value + * constant did not decode currently decodes no op + * and is consumed silently - the guest's cudaMemset + * then reports success with zero bytes written + * (r341 memcpy_roundtrip: 8MB memset readback + * all-bad; victory_memset2d). An op_tr_pending hold + * here WEDGED client init (r343: first cudaMalloc + * retried forever - driver init issues fill-shaped + * CE pushes, the hold re-armed every decode pass + * and never expired). So: log loudly only; the + * sema-remapconst rows above capture the real + * decode shape to build the proper fill decoder. + */ + if (allow_ops && transfer != 0 && + st.line_length >= 1024 && st.have_out && + !st.have_in && DMA_REMAP_ENABLE(dma) && + DMA_DST_MEMORY_LAYOUT_PITCH(dma) && + !st.have_remap && n_ops == n_ops_before && + svc_fill_noremap < 8) { + svc_fill_noremap++; + pr_err("vgsp: FILL-NOREMAP h=0x%x push=0x%llx dma=0x%08x line=%u out=0x%llx phys=%u dstmode=%u: fill-shaped launch, value source not decoded (log-only diagnostic)\n", + ch->h_object, push_va, dma, + st.line_length, st.offset_out, + DMA_DST_TYPE_PHYSICAL(dma), + st.have_dst_mode ? st.dst_phys_mode : 99); + } + + /* + * fail4: name a transfer-shaped launch that decoded NO + * op because the pass was denied (allow_ops=0) - the + * op-chain above (and its dma-skip dump) is skipped + * wholesale in that case, so this class was invisible. + */ + if (gsp_sema_dbg && transfer != 0 && !allow_ops && + svc.dmatrace_log < (gsp_sema_dbg ? 4096 : 64)) { + svc.dmatrace_log++; + pr_info("sema-dmatrace: denied h=0x%x push=0x%llx dma=0x%08x line=%u in=0x%llx out=0x%llx have_in=%u have_out=%u have_remap=%u trusted=%d replay=%d eng=%u pend_would=%d\n", + ch->h_object, push_va, dma, + st.line_length, st.offset_in, + st.offset_out, st.have_in, st.have_out, + st.have_remap, trusted, replay, + ch->engine_type, + (!allow_ops && trusted && + !ch->engine_type && transfer != 0 && + st.line_length && st.have_out && + st.have_in && + !DMA_REMAP_ENABLE(dma) && + st.offset_out >= SVC_VAS_FLOOR && + entry_idx != SVC_ENTRY_NONE && + entry_idx < SVC_GPFIFO_MAX_BITS && + ch->slot_pend_tries[entry_idx] < 4) ? 1 : 0); + } + /* + * fail4 r302: a denied device-destination copy that + * WOULD have built a type=1 op (the exact branch shape + * above: no remap, both offsets, device dst) must not + * be consumed silently. This is the sw-copy client + * mode: the GR-classified GRCE channel's ops run only + * once the pin/trust stream (injected) reaches the + * entry, and the first pass over a chunk push lands + * before that - the copy then never ran and the guest + * computed on stale VRAM (vector_add r302: chunk + * copies cc..d1 skipped; only the tail entries were + * re-decoded by a rewind and ran). Mark the pass + * op-pending so the pump holds the entry through the + * m163/opdef machinery and the retry runs the copy as + * soon as the trust (or the operand translation) + * arrives; the bounded SVC_OPDEF_HOLD_MS expiry keeps + * the pre-existing skip as the last resort. + */ + if (!allow_ops && trusted && + !ch->engine_type && transfer != 0 && + st.line_length && st.have_out && st.have_in && + !DMA_REMAP_ENABLE(dma) && + st.offset_out >= SVC_VAS_FLOOR) { + /* + * r314 (CONT.57 lead 2): capture the staging + * source on the DENIED pass too. This is the + * pipeline-order moment: the push was just + * decoded, the driver has not seen this + * push's fence (it applies later in this same + * pass at the earliest), so the staging still + * holds this chunk's bytes. The old capture + * gate lived in the op-build block, which + * only runs on ALLOWED passes - by then the + * fence had long fired and the staging was + * refilled (r302-r313: captures 0-3/8, the + * late executions read refilled staging, the + * deterministic 786335-bad output). Same + * translation-probe guard as the build-time + * gate: only an SYS-resolving source is + * host staging. + */ + if (st.line_length <= SVC_COPY_SNAP_MAX) { + u64 spa = 0; + u32 sap = 0; + + if (!svc_op_translate(vgpu, + st.offset_in, + &spa, &sap) && + sap == CE_APERTURE_SYS) + svc_copy_snap_capture(vgpu, + st.offset_out, + st.offset_in, + st.line_length, + ch->h_object, + entry_idx, + svc_pass_hash); + } + /* + * r308: the `!replay` gate above kept this + * dead on the sw-copy channel: after the first + * consumed pass every later pass classifies as + * a replay, so the denied chunk copies were + * never retried (vector_add: 6 of 8 chunks + * skipped). A replay pass that marks the slot + * pending un-completes it (slot_ops_complete + * is written from svc.op_tr_pending below), so + * the next pass is treated fresh and the copy + * runs once trust/translation arrives. The + * per-slot counter caps the re-holds so a slot + * whose trust never arrives cannot wedge the + * channel in 10 s hold cycles. + */ + if (entry_idx != SVC_ENTRY_NONE && + entry_idx < SVC_GPFIFO_MAX_BITS && + ch->slot_pend_tries[entry_idx] < 4) { + ch->slot_pend_tries[entry_idx]++; + svc.op_tr_pending = 1; + if (gsp_sema_dbg && + svc.pend_dbg_log < 512) { + svc.pend_dbg_log++; + pr_info("sema-pend: h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u replay=%d trusted=%d tries=%u -> op_tr_pending\n", + ch->h_object, push_va, + entry_idx, st.offset_out, + st.offset_in, + st.line_length, + replay, trusted, + ch->slot_pend_tries[entry_idx]); + } + } + } + /* + * fail4: name every op appended to this pass (copy or + * fill) with its resolved operand fields, so the + * executor's population is on record even when the + * 64 KB copy witness below never fires for it. + */ + if (n_ops != n_ops_before) { + struct svc_memset_op *lm = &ops[n_ops - 1]; + + /* + * fail4 CONT.53: H2D-shaped copies (virtual + * device destination, virtual host source) + * read the driver's ping-pong staging. The + * driver paces restaging on the copy's + * completion fence; our execution may run + * later than the push's processing position, + * so capture the source NOW (bounded pool) + * and let svc_copy execute from the capture - + * the hardware's in-order staging read. + */ + /* + * H2D shape ONLY: device destination (VID + * aperture), host source (SYS). A D2H readback + * (host dst) captured here would freeze + * pre-kernel zeros (observed in the r300 + * captures) and break readbacks. + */ + if (lm->type == 1 && !lm->phys && + lm->dst >= SVC_VAS_FLOOR && + lm->aperture == CE_APERTURE_VID && + !lm->src_phys && + lm->src >= SVC_VAS_FLOOR && + lm->bytes <= SVC_COPY_SNAP_MAX) { + /* + * r311: the src_aperture equality + * gate never matched the sw-copy + * client (the driver sends no + * SET_SRC_PHYS_MODE, so src_ap + * defaults to VID=0 while the + * source is host staging; r302-r310 + * logged zero captures and the + * late executions read refilled + * staging). The translation is the + * truth: capture only when the + * first source page resolves + * through the SYS aperture. + */ + u64 spa = 0; + u32 sap = 0; + + if (!svc_op_translate(vgpu, lm->src, + &spa, &sap) && + sap == CE_APERTURE_SYS) + svc_copy_snap_capture(vgpu, + lm->dst, + lm->src, + lm->bytes, + ch->h_object, + entry_idx, + svc_pass_hash); + if (gsp_sema_dbg && + svc.snap_gate_log < 256) { + svc.snap_gate_log++; + pr_info("sema-snapgate: dst=0x%llx src=0x%llx trc=%d sap=%u\n", + lm->dst, lm->src, + svc_op_translate(vgpu, lm->src, &spa, &sap), + sap); + } + } + + /* + * fail4 CE linear auto-advance, BOTH sides: the + * driver re-sets only ONE offset per op and the + * copy engine advances the other by the transfer + * size. H2D ops show an explicit advancing DST + * with a FIXED SRC base (every 1 MB chunk read + * the source base again: vector_add's device a + * held h_a[0..1MB) x4 -> 75% mismatch; the + * readbacks are the mirror image). Advance both + * registers; an explicit re-set simply overrides. + */ + { + u64 adv = (lm->type == 1) ? lm->bytes : + (u64)lm->elem * lm->count; + + st.offset_out += adv; + if (lm->type == 1) + st.offset_in += adv; + } + if (svc.opadd_log < 2048) { + svc.opadd_log++; + pr_info("sema-opadd: h=0x%x type=%u dst=0x%llx src=0x%llx bytes=%llu elem=%u count=%u ap=%u src_ap=%u phys=%u src_phys=%u\n", + ch->h_object, lm->type, + lm->dst, lm->src, lm->bytes, + lm->elem, lm->count, + lm->aperture, lm->src_aperture, + lm->phys, lm->src_phys); + /* + * r420d: 2D geometry of every copy op + * (the memset2d readback forensics - + * the driver decomposes cudaMemcpy2D + * into per-push ops whose LINE_COUNT/ + * PITCH decode must be visible to + * tell "one row per op" from "N rows + * in one op"). Bounded. + */ + if (lm->type == 1) + pr_info("sema-op2d: h=0x%x dst=0x%llx src=0x%llx bytes=%u rows=%u dpitch=%u spitch=%u line=%u lc=%u have_po=%u po=%u have_pi=%u pi=%u\n", + ch->h_object, lm->dst, + lm->src, lm->bytes, + lm->rows, lm->dpitch, + lm->spitch, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in); + } + } + + /* * Both release encodings write the payload word first * (NvGpuSemaphore.data[0] for the four-word format, the * single word for the one-word format), and the client's @@ -6567,6 +11068,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.sem_va_lo; st.rel_va = sem_va; st.rel_payload = st.sem_payload; + svc_wait_rel_va = sem_va; + svc_wait_rel_payload = st.sem_payload; + svc_wait_rel_off = off; + svc_wait_rel_have = true; st.have_rel = 1; if (st.have_sem && st.sem_payload && @@ -6664,6 +11169,22 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, ch->entry_want = st.sem_payload; have_cur = true; pending = 1; + /* + * fail4: apply this seed ABSOLUTE. + * The generic applier still sees the + * increment flag (inc), so leaving + * the residue in cur makes it write + * cur+payload (observed 134+1=135) + * - a completed value far past the + * guest's queued count, which + * trips the guest UVM assert + * (uvm_channel.c:204 completed <= + * queued) and kills the device. + * cur=0 makes inc-apply land the + * payload itself, exactly the + * re-seed the client polls. + */ + cur = 0; svc.pool_residue++; if (svc.pool_residue_log < 32) { svc.pool_residue_log++; @@ -6780,7 +11301,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, gsp_sema_skip_ops); } svc_run_ops(vgpu, ops, n_ops, ch->h_object, - ops_done, ops_seq - n_ops); + ops_done, ops_seq - n_ops, + entry_idx, svc_pass_hash); svc.ring_ops++; ops_runs++; } @@ -6876,7 +11398,44 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, */ svc.fwd_rel_overflow++; svc_fwd_soft_copy_inline(vgpu); - if (svc_rel_apply_launch(vgpu, ch, + /* + * fail4 CONT.53: never clear a + * push's fence before its ops + * read their source. The pass's + * ops run at the pass end; the + * driver's ping-pong staging + * restages as soon as the fence + * clears, and a later op read + * then returns chunk+N bytes + * (vector_add +2 MB). Queue the + * release through the pending + * pass instead. + */ + if (n_ops > 0 || + svc.op_tr_pending) { + /* + * r316 (CONT.60): + * op_tr_pending covers + * the denied pass - the + * pass built no op + * (allow=0) but a copy is + * still owed; applying + * the fence here (the old + * n_ops==0 path) let the + * driver refill the + * ping-pong staging before + * the copy ever ran - the + * cold-run capture gap + * (vector_add run 1 + * bad=157696/193536 vs + * bad=0 warm). + */ + svc_pending_add(sem_va, + st.sem_payload, + inc ? 1u : 0u); + svc_rel_va_seen = sem_va; + svc_rel_count++; + } else if (svc_rel_apply_launch(vgpu, ch, sem_va, st.sem_payload, cur, pending, @@ -6910,7 +11469,20 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc_fwd_rel_direct_note(ch->h_object, sem_va, st.sem_payload); - if (svc_rel_apply_launch(vgpu, ch, sem_va, + /* + * fail4 CONT.53: same ops-order + * guard as above - a pass with + * pending copy ops must run + * them before its fence clears. + */ + if (n_ops > 0) { + svc_pending_add(sem_va, + st.sem_payload, + inc ? 1u : 0u); + svc_rel_va_seen = sem_va; + svc_rel_count++; + } else if (svc_rel_apply_launch(vgpu, ch, + sem_va, st.sem_payload, cur, pending, inc, inc_apply)) { @@ -6930,7 +11502,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, off += 4 + count * 4; } - if (gsp_sema_dbg && ch->h_object && svc.entry_log < 512) { + if (gsp_sema_dbg && ch->h_object && + svc.entry_log < 16384) { svc.entry_log++; { u8 tail[24]; @@ -6996,6 +11569,12 @@ struct svc_pending_release { u64 va; u32 payload; u32 op; /* 0 = RELEASE, 1 = REDUCTION (add payload) */ + u32 h; /* r316: channel the release was decoded for; the + * verified pool resolver is channel-keyed and h=0 + * never matches its per-channel entries (matmul + * r322: pool-rel cand resolved at decode (rrc=0, + * h=channel) while svc_apply_pending with h=0 + * starved forever and hung the client). */ /* * Resolution attempts that found no translation for this VA. * Telemetry for ISSUE-0001: a pending release that keeps failing to @@ -7012,25 +11591,67 @@ static struct svc_pending_release svc_pending[SVC_PENDING_MAX]; static u32 svc_pending_n; static u32 svc_pending_next; +static u64 svc_pending_rescued; /* stall-class: entries the old >= dedup would have dropped */ +static u32 svc_pending_rescued_log; +/* + * STALLFREE: the pending ring silently overwrote its oldest entry when full + * (an evicted release is a client wait that never clears). The counters are + * hoisted next to the ledger counters so the periodic diag (above) can print + * them; the ring is a bounded queue, so under churn this is the back-pressure + * signal, not an error path. + */ + static void svc_pending_add(u64 va, u32 payload, u32 op) { u32 i; if (!payload) return; - for (i = 0; i < svc_pending_n; i++) - if (svc_pending[i].va == va && - svc_pending[i].payload >= payload) + for (i = 0; i < svc_pending_n; i++) { + if (svc_pending[i].va != va) + continue; + /* + * Stall-class fix (R1): only an EXACT duplicate may be + * dropped. The old rule dropped every release behind an + * already-queued higher payload; in a burst the queue could + * end up holding only the last value (observed: payloads 7..34 + * decoded, queue held {7,34}, the guest word stopped short of + * the waiter's value and the client spun forever with + * starved=0). Distinct releases must always queue - the apply + * pass already stale-drops values the word has passed. + */ + if (svc_pending[i].payload == payload && + svc_pending[i].op == op) return; + if (svc_pending[i].payload >= payload) { + svc_pending_rescued++; + if (svc_pending_rescued_log < 32) { + svc_pending_rescued_log++; + pr_info("sema-svc: pending rescue va=0x%llx pay=%u op=%u (queued; ahead entry pay=%u op=%u)\n", + va, payload, op, + svc_pending[i].payload, + svc_pending[i].op); + } + } + } if (svc_pending_n < SVC_PENDING_MAX) { i = svc_pending_n++; } else { i = svc_pending_next++ % SVC_PENDING_MAX; + svc_pending_ovf++; + if (svc_pending_ovf_log < 8) { + svc_pending_ovf_log++; + pr_warn("sema-svc: PENDING-OVF ring full (%u); overwriting oldest entry va=0x%llx pay=%u op=%u (n=%llu)\n", + SVC_PENDING_MAX, svc_pending[i].va, + svc_pending[i].payload, svc_pending[i].op, + svc_pending_ovf); + } } svc_pending[i].va = va; svc_pending[i].payload = payload; svc_pending[i].op = op; svc_pending[i].tries = 0; + svc_pending[i].h = svc.cur_ch; } /* @@ -7049,6 +11670,35 @@ static void svc_session_flush(void) unsigned long flags; u32 i; + /* + * m167: never wipe a LIVE session. The flush is requested when a + * channel close leaves the table empty, but that transition fires + * during ordinary UVM channel cycling and the deferred execution can + * land after the same client (or the next client, when its chan_add + * raced the requested flush) has registered channels again - r169: + * the flush executed 2.5 s after smiley's launch with its channels + * live (registered 7 s earlier), wiped the queued completion releases, + * and the client spun forever in uvm_channel_is_value_completed + * (release never applied; pool word frozen). A live channel's session + * owns the tables - skipping is always safe because a genuine session + * end re-arms the flush when its own channels close. + */ + { + bool live = false; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) + if (svc.chan[i].valid) { + live = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (live) { + pr_info("sema-svc: session flush skipped: live channel(s) present (deferred flush raced a new session registration)\n"); + return; + } + } + svc_gr_diag_reset(); /* m150: per-session diagnostic budgets */ svc.map_n = 0; svc_pb_n = 0; @@ -7057,7 +11707,26 @@ static void svc_session_flush(void) svc_pd_best = 0; svc_pd_cands = 0; svc_pd_ev_hash = 0; - svc.direct_base = 0; + /* + * m177c: the UVM ring-space directory is per-client guest memory + * (and its samples come from this session's rings): reset with the + * session like the primary slot. + */ + svc_pd_r_found = 0; + svc_pd_r_attempts = 0; + svc_pd_r_base = 0; + svc_pd_r_ev_hash = 0; + svc_pd_r_n = 0; + /* + * m177: svc.direct_base is NOT cleared here. It is the guest kernel's + * direct-map base - a per-BOOT constant, identical for every client + * session of the same guest boot - and it is the gate and input of + * the pin-free resolution paths (svc_targeted_resolve). Clearing it + * per session forced every client to wait for a rebuild scan while + * its first cudaMalloc polled, and made the module depend on the + * host pin tools entirely. It is wiped on a guest reboot by + * svc_guest_reset_wipe() (gsp_sema_service_guest_reset()). + */ svc_pending_n = 0; svc_pending_next = 0; /* @@ -7068,6 +11737,18 @@ static void svc_session_flush(void) svc_opdef_reset_all(); svc_pool_pa_n = 0; svc_pool_pa_next = 0; + /* + * m168 addendum (r171): the UVM semaphore pool is per-CLIENT, not + * per-boot. The boot cache's conflict line proved it: the same VA + * resolved to a different PA for the next client + * (cached=0x1247b5000 new=0x1321fa000). A cache surviving the flush + * therefore served the previous client's dead page: the release was + * applied (read-back passes on a dead-but-readable page), dropped + * from pending, and the live client spun forever. Session-scope the + * cache; the scanner's per-client resolution is the source of truth. + */ + svc_pool_boot_n = 0; + svc_pool_boot_next = 0; svc_retry_next = 0; svc_fwd_rel_n = 0; svc_grce_dst_ring_next = 0; @@ -7122,9 +11803,32 @@ static void svc_session_flush(void) * because the counters were already exhausted by the first client). */ svc.counter_rel_dbg = 0; + svc.crel_entry_dbg = 0; + svc.walk_dbg = 0; + svc.gr_rel_auth_log = 0; svc.counter_noslot_dbg = 0; svc.sem256_log = 0; svc.xfer_raw_log = 0; + svc.pushtrace_all = 0; + svc.opadd_log = 0; + svc.dmatrace_log = 0; + svc.copyent_log = 0; + svc_wait_log = 0; + svc_wait_pass_log = 0; + svc_remapconst_log = 0; + svc_pitch_log = 0; + svc_fill_raw_log = 0; + svc_lowfill_log = 0; + svc_tabcopy_log = 0; + svc_fill_noremap = 0; + svc_wait_n = 0; + svc_wait_rel_have = false; + memset(svc_wait_holds, 0, sizeof(svc_wait_holds)); + svc.bigrun_log = 0; + svc.lid_hdr_log = 0; + svc.skiprun_log = 0; + svc.mystery_log = 0; + svc.next_hdr_log = 0; /* * The push decode budget (`sema-push` lines) is per-session too: the * first client of a boot otherwise spends it and the next session's @@ -7135,6 +11839,14 @@ static void svc_session_flush(void) svc.entry_log = 0; svc_pushalt_log = 0; /* + * m169: the op-trace budget is session state too. Left un-reset, the + * first client of a boot spends it and the failing client's copy ops + * are invisible in exactly the runs that need them (r172: streams2's + * H2D evidence was blacked out because the shared budget was spent at + * 29704, before the client ran). + */ + svc.op_trace_log = 0; + /* * Per-entry/stream decode diagnostics are per-session: the first * client otherwise spends them and the second client's CE->GR * transition is invisible (r24-r30: c2's follow-up pushes never @@ -7153,6 +11865,8 @@ static void svc_session_flush(void) memset(svc_retry_q, 0, sizeof(svc_retry_q)); memset(svc_pool_pa, 0, sizeof(svc_pool_pa)); memset(svc_fwd_rel, 0, sizeof(svc_fwd_rel)); + memset(svc_inline_once, 0, sizeof(svc_inline_once)); + svc_inline_once_next = 0; memset(svc_grce_dst_ring, 0, sizeof(svc_grce_dst_ring)); for (i = 0; i < SVC_VAS_SLOTS; i++) { if (svc_vas[i].data) @@ -7162,11 +11876,117 @@ static void svc_session_flush(void) } svc_vas_n = 0; memset(svc.lat, 0, sizeof(svc.lat)); + /* m171 (AGENT-F R2): the write/read ledger is per-session. */ + svc_ledger_clear(); spin_lock_irqsave(&svc.vas_lock, flags); for (i = 0; i < SVC_MAX_VASPACES; i++) svc.vas[i].valid = false; spin_unlock_irqrestore(&svc.vas_lock, flags); - pr_info("sema-svc: session flush (all channels closed): discovery state reset for the next client\n"); + /* + * m176: carryover state that previously survived the flush and could + * poison the next client: + * - user-verified scan failures (tr_failed) held the pool VA page for + * 60 s even after the new client's page was reachable by targeted + * scan - clear them so the next session starts unblocked; + * - a forwarding hold whose channel is gone is resolved against a + * recycled handle with stale entry indices - drop it (no channels + * are live by definition of this flush); + * - the captured GR candidate belongs to the dead session's QMD/push; + * offering it to a recycled channel would replay a dead launch. + */ + memset(tr_failed, 0, sizeof(tr_failed)); + tr_fail_next = 0; + svc_fwd_hold_clear(); + memset(&svc_gr_cand, 0, sizeof(svc_gr_cand)); + /* m176: re-arm the opdef diagnostic budget (it was never per-session, + * so a late client's strand could be silent). */ + svc_opdef_log = 0; + /* One-cycle diag budgets (C2/C3/C7/C8/C9/C10) are per-session too. */ + svc_relx_log = 0; + svc_opdef_clr_log = 0; + svc_busy_defer_log = 0; + svc_hold_fb_log = 0; + svc_cand_ovw_log = 0; + svc_inline_recs_log = 0; + /* + * r401: the exactly-once capture memory is session-scoped. A new + * client's pushes restart their release counters, so their content + * (and hence hash) can collide with the previous client's - keeping + * the table would wrongly classify a fresh launch of the new + * session as already captured (LAUNCH-SKIP, silent drop). + */ + memset(svc_gr_seen, 0, sizeof(svc_gr_seen)); + svc_gr_seen_stale = 0; + /* m176: new session generation - channels and scrubbers re-stamp on + * registration and never inherit per-session learned state. */ + svc_sess_gen++; + pr_info("sema-svc: session flush (all channels closed): discovery state reset for the next client (gen %u)\n", + svc_sess_gen); +} + +/* + * m177: boot-scoped wipe for a GUEST boot (RM reload). Unlike the + * per-client session flush this clears the state that is valid for one + * guest boot only: the direct-map base and every content-scanned table. + * Runs on the svc thread (svc_guest_reset_wipe is invoked from the thread + * loop below), the only writer of this state. + */ +static void svc_guest_reset_wipe(void) +{ + svc.direct_base = 0; + svc.map_n = 0; + svc_pd_found = 0; + svc_pd_attempts = 0; + svc_pd_best = 0; + svc_pd_cands = 0; + svc_pd_ev_hash = 0; + svc_pd_r_found = 0; + svc_pd_r_attempts = 0; + svc_pd_r_base = 0; + svc_pd_r_ev_hash = 0; + svc_pd_r_n = 0; + svc_pb_n = 0; + svc_pool_boot_n = 0; + svc_pool_boot_next = 0; + svc_pool_pa_n = 0; + svc_pool_pa_next = 0; + memset(svc_pool_pa, 0, sizeof(svc_pool_pa)); + memset(tr_failed, 0, sizeof(tr_failed)); + tr_fail_next = 0; + memset(svc.lat, 0, sizeof(svc.lat)); + pr_info("sema-svc: guest reset: boot-scoped discovery state wiped (direct_base, map, caches)\n"); +} + +/* + * m177: called by gsp_rm_emu_reset() when the guest's GSP-RM is (re)loaded + * - a new guest boot, where every VA->PA relation can have changed. The + * request is latched and executed on the svc thread, which owns the + * discovery state (a cross-thread wipe here would race svc_rebuild). + */ +void gsp_sema_service_guest_reset(void) +{ + WRITE_ONCE(svc_guest_reset_req, 1); + if (svc.task) + wake_up_process(svc.task); +} + +/* + * r317: queue a pass's recorded forward releases through the pending + * machinery instead of applying them at a hold break / fallback while an + * op of that pass is still owed. Applying the fence early let the driver + * refill the ping-pong staging before the copy read it (the cold-run + * capture gap: vector_add run 1 of every cold boot, last ~2 chunks). The + * pending store applies each release once the word resolves (the copy has + * run by then, since the op execution is what clears the pending state on + * the retry). + */ +static void svc_fwd_rel_queue_pending(struct svc_fwd_rel *rel, u8 n) +{ + u8 i; + + for (i = 0; i < n; i++) + svc_pending_add(rel[i].va, rel[i].payload, + rel[i].inc_claimed ? 1u : 0u); } /* @@ -7200,8 +12020,11 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) * resolver: the learned map can answer with a stale * page whose value makes this release look already * delivered and drops it (see svc_sem_write). + * r316: resolve with the channel the release was + * decoded for (h=0 never matches the per-channel + * pool entries - matmul r322 starved/hung). */ - if (svc_pool_pa_resolve(vgpu, 0, p->va, &ppa, &pap, + if (svc_pool_pa_resolve(vgpu, p->h, p->va, &ppa, &pap, NULL)) { /* not translatable yet: keep (ISSUE-0001) */ p->tries++; @@ -7229,7 +12052,7 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) * a pool VA the base may not have landed: applying while * the word is still 0 is absorbed, so hold and retry. */ - if (svc_pool_sem_va(p->va) && svc_pool_hold_active()) { + if (svc_pool_sem_va(p->va) && svc_pool_hold_active_va(p->va)) { p->tries++; svc_pending_note_starved(p); i++; @@ -7255,7 +12078,7 @@ static void svc_apply_pending(struct nv_vgpu_device *vgpu) if (!svc_sem_write(vgpu, 0, p->va, p->op ? cur + p->payload : p->payload)) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); if (gsp_sema_dbg) pr_info("sema-svc: deferred release va=0x%llx payload=%u (was cur=%u)\n", p->va, p->payload, cur); @@ -7387,6 +12210,25 @@ static u32 __maybe_unused __svc_apply_page_data_disabled(struct nv_vgpu_device * } /* + * F1-b: while the forwarded GR launch this release belongs to + * is still in flight, the sweep must not author its QMD + * completion word - the backend poll would break on a + * host-authored ack (r198: qmd_rel satisfied ~6 ms after the + * doorbell, zero engine writes). Defer: nothing is written or + * consumed here; the release stays in the guest page and later + * sweep passes re-see it, while the hold/fallback path owns a + * failed hardware copy. + */ + if (gsp_sema_gr_swrel_gate && svc_gr_rel_inflight(va, NULL)) { + if (svc.gr_rel_auth_log < 256) { + svc.gr_rel_auth_log++; + pr_info("sema-diag: GR-REL-AUTHOR src=sweep va=0x%llx pay=%u h=0x%x DEFER\n", + va, payload, 0u); + } + continue; + } + + /* * A release at page offset 0x10 with payload 1 in front of a * channel's first push identifies the page as that channel's * push-buffer base (see svc_pb_add). @@ -7395,9 +12237,21 @@ static u32 __maybe_unused __svc_apply_page_data_disabled(struct nv_vgpu_device * (lo & 0xfff) < 0x20 && svc_pb_first_push(buf)) svc_pb_add(pa_page, lo & 0xfff); + /* + * F1-a: tag software authorship of the forwarded launch's + * release word (gate off, or the request already resolved and + * the forward progress rule still allows the write). + */ + if (svc_gr_rel_match(va, NULL) && + svc.gr_rel_auth_log < 256) { + svc.gr_rel_auth_log++; + pr_info("sema-diag: GR-REL-AUTHOR src=sweep va=0x%llx pay=%u h=0x%x\n", + va, payload, 0u); + } + if (svc_sem_write(vgpu, 0, va, payload) == 0) { svc.releases_applied++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); applied++; if (payload != cur + 1) { svc.releases_jumped++; @@ -7616,7 +12470,8 @@ static void svc_pb_discover(struct nv_vgpu_device *vgpu) u32 slot, k; bool have = false; - if (!svc.chan[i].valid || !svc.chan[i].engine_type) + if (!svc.chan[i].valid || svc.chan[i].vgpu != vgpu || + !svc.chan[i].engine_type) continue; ce = true; if (svc.chan[i].ce_index >= 32) @@ -8397,6 +13252,119 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, return 0; } } + + /* + * r496b: no candidate matched. For a GR channel whose ring is known + * (translated and pinned - the sysmem content scan or the VRAM + * conventions), the USERD may still be elsewhere: the X/GL driver + * allocates its USERD in VRAM and the CPU RM maps it through a BAR1 + * window, so the driver's GP_PUT write lands at a BAR1 offset that is + * neither the registered memdesc base nor ring-derived. Scan the + * emulated VRAM for the USERD shape and validate each hit against the + * ring: the put it claims must correspond to entries whose pushbuffer + * VAs all sit in the ring's allocation window. A foreign channel's + * USERD cannot pass that test. One scan per channel (the result + * persists through the pump's write-back), logged on latch. + */ + if (!ch->engine_type && ch->userd_scans < 8 && ring_pa) { + u64 off, lim; + u8 *base = vgpu->bars[1].vaddr; + u64 best = 0, best_dist = ~0ULL; + u32 best_put = 0, best_get = 0; + bool best_pending = false; + + ch->userd_scans++; + lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + if (base && lim) { + for (off = 0; off + 0xa0 <= lim; off += 0x200) { + u32 hp, hg; + bool ok = true; + u32 k; + u64 dist; + bool pending; + + memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); + memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); + if (!hp || hp > ch->gpfifo_entries || hg > hp) + continue; + /* Validate: every claimed entry decodes to a + * pushbuffer VA in the ring's window. */ + for (k = 0; k < hp && k < 64; k++) { + u8 e[8]; + u64 eva; + u32 elen, a0, a1; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)k * 8, + e, sizeof(e))) { + ok = false; + break; + } + memcpy(&a0, e, 4); + memcpy(&a1, e + 4, 4); + eva = ((u64)(a1 & 0xff) << 32) | + (u64)(a0 & 0xfffffffc); + elen = ((a1 >> 10) & 0x1fffff) << 2; + /* + * Every entry the put claims must name a + * VA in the ring's allocation window - + * pushbuffers below the ring, the + * progress tracker above it. + */ + if (!eva || + eva + SVC_GR_RAM_WIN / 2 < + ch->gpfifo_va || + eva >= ch->gpfifo_va + + SVC_GR_RAM_WIN / 2 || + elen < 4 || elen > 0x100000) { + ok = false; + break; + } + } + if (!ok) + continue; + /* + * Several blocks can validate: a previous + * session leaves its old USERD holding the same + * put/entry values while the new channel's block + * sits elsewhere (observed r504: the re- + * registered channel re-scanned, adopted the + * stale block, and its DMA test timed out). + * Prefer a block with work PENDING (get < put); + * among equals prefer the one nearest the + * registered memdesc offset (the driver's own + * hint). The pump's GP_GET write-back goes to + * the adopted block, so a stale adoption strands + * the client's poll. + */ + dist = off > ch->userd_pa ? off - ch->userd_pa + : ch->userd_pa - off; + pending = hg < hp; + if (best && + (best_pending > pending || + (best_pending == pending && dist >= best_dist))) + continue; + best = off; + best_dist = dist; + best_put = hp; + best_get = hg; + best_pending = pending; + } + if (best) { + ch->userd_pa = best; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + *put = best_put; + *get = best_get; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by VRAM shape scan at fb 0x%llx (put=%u get=%u%s, ring-validated)\n", + ch->h_object, ch->gr_index, best, + best_put, best_get, + best_pending ? "" : ", idle"); + return 0; + } + } + } return -ENOENT; } @@ -8494,7 +13462,7 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, /* Keep the completion visible to clients that start waiting * after the event fired. */ - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); } } @@ -8791,6 +13759,9 @@ static bool svc_gr_content_pin(struct nv_vgpu_device *vgpu, u32 rank, svc_pin_store(push_va, push_pa, SVC_APERTURE_SYS); svc_lat_store(push_va, push_pa, SVC_APERTURE_SYS); svc_mark_hot(push_pa); + /* m177c: a content-verified UVM-space pair - feed the ring-space + * directory discovery (counter pages resolve through it). */ + svc_pd_ring_note(push_va, push_pa); if (ring_ap == SVC_APERTURE_FB) pr_info("sema-chan: content-discovered GR ring rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u; push pa=0x%llx; userd rc=%d put=%u get=%u)\n", rank, ring_va, ring_pa, push_va, len, push_pa, prc, @@ -8897,7 +13868,8 @@ static void svc_gr_content_discover(struct nv_vgpu_device *vgpu) int pap = 0; u32 r = c->gr_index; - if (!c->valid || c->engine_type || r >= SVC_GR_CONTENT_RANKS) + if (!c->valid || c->vgpu != vgpu || c->engine_type || + r >= SVC_GR_CONTENT_RANKS) continue; if (test_bit(r, &svc_gr_ring_done)) continue; @@ -9126,6 +14098,202 @@ bool gsp_sema_counter_derive = true; unsigned gsp_sema_gr_grace_ms = 8000; /* + * GRFIX FIX1b (r213): absolute per-LAUNCH bound (ms) for the data-stalled + * hold of svc_gr_data_fill. The identity is the launch push VA, not the + * GPFIFO slot: pin-batch rewinds/replays re-decode the same launch at + * different slot indices (r213: entry 107 -> 107 -> 99 -> ... on one + * channel), so an idx-keyed deadline re-armed on every pass and never + * fired - 410+ holds over 216 s, gr_hold_exp=0, client killed at its 240 s + * timeout. The deadline is fixed at the first hold of a launch; when it + * passes, svc_gr_request proceeds with the launch exactly as the + * pre-8b058884 path did (forward best-effort - never block the guest). + */ +unsigned gsp_phys_gr_hold_max_ms = 3000; + +/* + * GRFIX FIX1e (r215): master gate for the GR fill hold on dropped operand + * pages. Default 0 = LOG-ONLY: svc_gr_data_fill never holds on dropped + * pages - the drops are logged (GRFILL-DROP) and the launch takes the + * historical forward path immediately (exactly the pre-8b058884 behaviour); + * the GRFILL-HOLD/GRFILL-EXPIRED machinery and the per-launch absolute + * bound are skipped entirely (the pump's own m150/m156 hold and its + * absolute bound are NOT - they run regardless of this gate; the record + * fields are now in the snapshot write-back list, see the GRFIX NOTE). + * When set to 1 the page-count hold + absolute bound apply on the fill + * path too. + */ +unsigned gsp_phys_gr_fill_hold; /* default 0: log-only, never hold */ + +/* + * r346: trace GMMU walks for unresolved high-VA operands (the arg-miss + * diagnostic re-walks each published vaspace for a failing base). When + * nonzero, those walks print the per-level entries (L0..L2, PD0 halves) + * so a failing VA shows exactly where the walk dies. + */ +unsigned gsp_sema_walk_trace; + +/* + * F1-b: GR release-word authorship gate (default 1). While a forwarded GR + * launch is in flight, the RAM-content sweep must not author the very QMD + * completion release the backend poll is waiting for: a sweep write could + * satisfy the poll before the engine retires the QMD (false completion, + * silent corruption). With the gate on, such a write is deferred - the + * release stays in RAM and later sweep passes re-see it; when the hardware + * copy resolves, the hold/fallback path applies it. 0 restores the + * unconditional sweep apply for an A/B. + */ +bool gsp_sema_gr_swrel_gate = true; + +/* + * m164 CLASS-2: hold window (ms) for a readable GR launch push whose decode + * saw QMD words but built no launch candidate for its channel (a replay- + * suppressed capture, a partial/split QMD, or a push page the pin stream has + * not reached yet). The entry is left unconsumed for this window so the next + * pass re-decodes it with a forced fresh capture (gr_rehold_idx); on expiry + * the historical consume applies. Without this, such entries were consumed + * silently and the client's launch never ran (streams2 second vadd: 56 + * consumes / 2 captures on one channel; the client read zeros). + */ +unsigned gsp_sema_gr_nocap_hold_ms = 6000; + +/* + * m165 (D-F2a): when set (default), a pool release is only deferred by the + * post-registration hold while NO host-verified injected pin exists for its + * VA; a fresh pin is this session's ground truth and lets the apply land at + * pin time (~0.5-2 s) instead of registration+5 s. Set 0 to restore the + * legacy blanket 5 s hold. + */ +bool gsp_sema_pool_hold_fresh = 1; + +/* + * m169 (AGENT-F R1): hold device-destination copies whose only + * translation is a page-directory walk or RAM scan (no pin). Default on; + * 0 restores the historical immediate-write behavior. + */ +bool gsp_sema_copy_dst_pin_guard = 1; +/* + * r336/r34x wait gate: honor the C56F host-semaphore acquire ops captured + * for a ring entry before running its work. DEFAULT ON since r408: the + * r408 streams2d forensics reproduced the class the gate was built for + * with the gate off - the c2 readback (C56F ACQ_CIRC_GEQ pay=4 on the + * scale channel's progress counter 0x20400ffd0) ran at 106617.279 with + * cur=2, i.e. BEFORE the c2-add launch was even submitted (106617.411), + * read the not-yet-produced buffer (zeros) and the client failed with + * 'c2 all zeros'. With the gate on the same readback holds (sema-wait: + * hold ... cur=2 waited_ms=353) until the counter reaches 4 (pass ... + * cur=4 at 106812.259) and the ordering is correct. The semantics are + * the class contract's: op 0 = equal, 2 = strict GEQ, 3 = circular GEQ, + * 4 = AND, 5 = NOR; the wait set is captured PER ENTRY (the old + * single-last-wait global leaked a later entry's acquire into an earlier + * entry - the r340 over-hold); the entry's own release counts only when + * its method order precedes the acquire; pending releases our machinery + * is merely late about are drained first; and every hold is bounded at + * gsp_sema_wait_hold_ms (default 4000 ms, r401) then fails open loudly + * ('give-up') so a value that can only be installed behind an upstream + * opdef hold cannot park a channel. Diagnostics: 'sema-wait: + * pass/hold/give-up' rows carry op, payload, observed value and decision. + * sema_wait_gate=0 restores the pre-r336 pass-through for an A/B. + */ +bool gsp_sema_wait_gate = 1; + +/* + * m169b: the guard above must never deadlock the boot's first client. The + * first device-destination H2D of a client runs before any launch, and the + * pins for device VAs are (re)discovered by the launch/fill paths - so at + * that instant NO pin can exist. r173: mail's H2D was held forever + * (opdef_held ~6000 passes, six expiry cycles, the entry re-held on every + * re-decode because the retry keeps resolving prov>=3). Bound the wait per + * destination page: after this many ms the guard releases and the copy runs + * with the walk translation (the historical behavior; the walk of a + * just-built client table is fresh), logged loudly. + */ +unsigned gsp_sema_copy_dst_guard_ms = 3000; + +/* + * r231 READ-ORDER guard (default 0: witness only). A device-src copy whose + * source VA has a PENDING device-dst write hold (the m169 guard window is + * still open) reads the PRE-write content: the writer's bytes land only + * when the guard resolves, while the reader can execute immediately. The + * ctaid_probe RAWA round trip is the reproducer - upload a fresh 64 KB + * buffer, read it straight back with no kernel - and the readback came + * back as foreign deterministic content. With this on, such a read is + * deferred through the existing bounded-hold machinery (-ENOENT) so it + * re-decodes after the write resolves; with it off the condition is + * logged (READ-ORDER-WIT) so the next capture proves the coincidence. + * Opt-in until an A/B proves no green regression. + */ +bool gsp_sema_read_order_guard; + +/* + * COPYWIT fail4: per-boot budget for the sema-bigcopy resolution witnesses + * (svc_copy's large-copy proof: both operands' VA -> pa/aperture/provenance + * plus the first 16 bytes each side). The shipped default kept the boot's + * head only; a failing-client run happens minutes later, so the budget is a + * module parameter now - raise it for a discrimination cycle and correlate + * each D2H's resolved source PA with the booter's OUT-PAGE-WIT copy-out PA + * for the same device VA. 0 disables the lines. + */ +unsigned gsp_sema_bigcopy_log_max = 24; +#define SVC_DST_GUARD_SLOTS 16 +struct svc_dst_guard { + u64 va_page; + unsigned long first; +}; +static struct svc_dst_guard svc_dst_guard[SVC_DST_GUARD_SLOTS]; +static u32 svc_dst_guard_next; + +/* true = the guard may hold this destination; false = release it. */ +static bool svc_dst_guard_hold(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i, slot = SVC_DST_GUARD_SLOTS; + + for (i = 0; i < SVC_DST_GUARD_SLOTS; i++) { + if (svc_dst_guard[i].va_page == page) { + if (time_before(now, svc_dst_guard[i].first + + msecs_to_jiffies( + gsp_sema_copy_dst_guard_ms))) + return true; + if (svc_dst_guard[i].first) { + svc_dst_guard[i].first = 0; + svc.copy_dst_relaxed++; + pr_info("sema-svc: device-dst guard released without a pin va=0x%llx (using the walk translation)\n", + page); + } + return false; + } + if (!svc_dst_guard[i].va_page) + slot = i; + } + if (slot == SVC_DST_GUARD_SLOTS) + slot = svc_dst_guard_next++ % SVC_DST_GUARD_SLOTS; + svc_dst_guard[slot].va_page = page; + svc_dst_guard[slot].first = now; + return true; +} + +/* + * r231 READ-ORDER guard: true while a held device-dst write for this VA + * page is still inside its bounded window (the writer's bytes have not + * landed yet). See the gsp_sema_read_order_guard declaration. + */ +static bool svc_dst_guard_pending(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + unsigned long now = jiffies; + u32 i; + + for (i = 0; i < SVC_DST_GUARD_SLOTS; i++) + if (svc_dst_guard[i].va_page == page && + svc_dst_guard[i].first && + time_before(now, svc_dst_guard[i].first + + msecs_to_jiffies(gsp_sema_copy_dst_guard_ms))) + return true; + return false; +} + +/* * CE release fallback: when a push carrying a tracking-semaphore release * cannot be read (and the content rescue failed), advance the channel's * tracking semaphore to the expected payload directly. The push's data @@ -9156,6 +14324,24 @@ bool gsp_sema_rel_recover; bool gsp_sema_dbg; /* + * r365: execute fill-shaped CE transfers whose destination is a VIDMEM + * WINDOW VA (>= SVC_VIDMEM_VA_LO, below the 1 TiB VA floor). The guest's + * page-table writes arrive exactly this way (dma=0x586, REMAP+PITCH, a + * count=1 SET_REMAP_CONST_A, line=4, out=0x2000c0xxx) and previously + * decoded no op, so the shadow never saw them and the first client of a + * fresh boot dropped operand pages (GRFILL-DROP -> launch held/fail-open + * with 8 dropped pages -> wrong results or a client wedge). + * + * The op is built with the VA unchanged (phys=0): svc_fill's virtual path + * resolves it through the guest's own published page tables + * (svc_op_translate) and only falls back to the shadow for floor-range + * VAs - never through the window convention (r362 proved a forced + * `addr - SVC_VIDMEM_VA_BASE` writes the wrong page and corrupts results: + * reduce sum=0 + mail hang). Default off for A/B. + */ +bool gsp_sema_winfill; + +/* * Skip executing a push's collected work (CE memsets/copies into the vidmem * shadow) while still applying its semaphore release. The client only ever * observes the release; the shadow writes exist so the emulation can walk @@ -9214,6 +14400,139 @@ unsigned gsp_sema_push_hold_ms = 2500; static unsigned long svc_gr_ring_tried; /* ranks that got a diagnostic line */ +static bool svc_gr_ram_ring_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out) +{ + static u8 *buf; + u64 start = ch->gr_ram_cursor; + u64 end = start + SVC_GR_RAM_SLICE; + u64 gpa; + int i; + unsigned long flags; + + if (end > SEMA_PA_MAX) + end = SEMA_PA_MAX; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + for (gpa = start; gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + 16 <= chunk; o += SEMA_PAGE_SIZE) { + u32 e0, e1, f0, f1; + u64 va0, va1; + u32 len0, len1; + + memcpy(&e0, buf + o, 4); + memcpy(&e1, buf + o + 4, 4); + va0 = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len0 = ((e1 >> 10) & 0x1fffff) << 2; + /* entry0: main pushbuffer, just below the ring */ + if (va0 >= ch->gpfifo_va || + va0 + SVC_GR_RAM_WIN / 2 < ch->gpfifo_va) + continue; + if (va0 < SEMA_VA_LO || va0 >= SEMA_VA_HI) + continue; + if (len0 < 4 || len0 > 0x100000) + continue; + memcpy(&f0, buf + o + 8, 4); + memcpy(&f1, buf + o + 12, 4); + va1 = ((u64)(f1 & 0xff) << 32) | + (u64)(f0 & 0xfffffffc); + len1 = ((f1 >> 10) & 0x1fffff) << 2; + /* entry1: progress tracker, above entry0 */ + if (va1 <= va0 || + va1 > ch->gpfifo_va + SVC_GR_RAM_WIN / 2) + continue; + if (len1 < 4 || len1 > 0x100000) + continue; + *pa_out = gpa + o; + return true; + } + } + ch->gr_ram_cursor = (end >= SEMA_PA_MAX) ? 0 : end; + ch->gr_ram_passes++; + if (ch->gr_ram_cursor == 0 && !ch->gr_ram_logged) { + ch->gr_ram_logged = true; + pr_info("sema-chan: GR ring h=0x%x rank=%u RAM content scan found no candidate in %u slice(s) (va=0x%llx)\n", + ch->h_object, ch->gr_index, ch->gr_ram_passes, + ch->gpfifo_va); + } + /* + * Persist the cursor immediately: the pump's stalled-userd path + * `continue`s past the end-of-pass write-back, so without this the + * scan would restart from slice 0 every pass and never reach the + * ring's page. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[i].gr_ram_passes = ch->gr_ram_passes; + svc.chan[i].gr_ram_logged = ch->gr_ram_logged; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +/* + * r496f: discover a kernel-RM client VAS's page directory in the emulated + * VRAM by verification. The CPU RM builds internal-VAS page tables in + * vidmem and never publishes them (SET_PAGE_DIRECTORY is an externally + * owned / UVM-only RPC), so the pushbuffer pages and the progress + * semaphore of an X/GL channel cannot be translated from any existing + * source. Try each 4 KB page as the walk root and accept the first whose + * walk reproduces the ring VA -> ring PA pair we already validated by + * content; registering it with the VAS table makes every other VA of the + * same client resolvable through the existing walker. Bounded to the + * vidmem shadow; verified, so a wrong candidate cannot be registered. + */ +static void svc_xvas_pdb_discover(struct nv_vgpu_device *vgpu, u64 ring_va, + u64 ring_pa) +{ + static const u32 ent_tab[] = { 4, 8, 16, 32, 64, 2, 128, 256, 512 }; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + unsigned int ei; + + if (!vgpu->bars[1].vaddr || !lim) + return; + for (off = 0; off + 0x1000 <= lim; off += 0x1000) { + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) { + u64 pa = 0; + int ap = 0; + + if (!svc_gmmu_walk(vgpu, off, ent_tab[ei], + ring_va & ~0xfffULL, + SVC_APERTURE_FB, &pa, &ap, false) && + (pa & ~0xfffULL) == (ring_pa & ~0xfffULL)) { + struct sema_vaspace_desc d; + + memset(&d, 0, sizeof(d)); + d.h_vaspace = 0x5a5a0000u | + (u32)(ring_va >> 20); + d.pdb = off; + d.num_entries = ent_tab[ei]; + d.b_vidmem = true; + gsp_sema_service_vaspace_add(&d); + pr_info("sema-chan: client VAS page directory discovered pdb=0x%llx entries=%u (ring VA 0x%llx -> pa 0x%llx verified; the client's other VAs are now walkable)\n", + off, ent_tab[ei], ring_va, ring_pa); + return; + } + } + } +} + static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -9238,7 +14557,8 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, */ spin_lock_irqsave(&svc.chan_lock, flags); for (k = 0; k < SVC_MAX_CHANNELS; k++) - if (svc.chan[k].valid && svc.chan[k].engine_type && + if (svc.chan[k].valid && svc.chan[k].vgpu == vgpu && + svc.chan[k].engine_type && svc.chan[k].ce_index + 1 > max_ce) max_ce = svc.chan[k].ce_index + 1; spin_unlock_irqrestore(&svc.chan_lock, flags); @@ -9489,6 +14809,124 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, pr_info("sema-chan: fbs scan done h=0x%x hits=%d (lim 0x%llx)\n", ch->h_object, hits, lim); } + + /* + * r495 diagnostic (X/GL channel): the X driver's GR channel + * registers its USERD at a memdesc base that may be an FB + * offset or a BAR1 window. The driver's kickoff writes GP_PUT + * through its CPU mapping of that object; if the write landed + * in the emulated FB shadow, scanning for the USERD shape + * (GP_PUT in [1..entries] at +0x8c, GP_GET <= GP_PUT at +0x90, + * both nonzero somewhere in the 0x200 block) names the real + * offset. One-shot per channel, diagnostic only. + */ + if (gsp_sema_dbg && vgpu->bars[1].vaddr && + svc_fb_shadow_size(vgpu) && ch->gpfifo_entries <= 0x20000) { + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + int hits = 0; + + for (off = 0; off + 0xa0 <= lim && hits < 8; + off += 0x200) { + u32 hp, hg; + + memcpy(&hp, vgpu->bars[1].vaddr + off + 0x8c, 4); + memcpy(&hg, vgpu->bars[1].vaddr + off + 0x90, 4); + if (!hp || hp > ch->gpfifo_entries || hg > hp) + continue; + pr_info("sema-chan: userd scan hit off=0x%llx put=%u get=%u (blk %08x %08x %08x %08x)\n", + off, hp, hg, + *(u32 *)(vgpu->bars[1].vaddr + off + 0x80), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x84), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x88), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x94)); + hits++; + } + pr_info("sema-chan: userd scan done h=0x%x hits=%d (lim 0x%llx)\n", + ch->h_object, hits, lim); + } + } + + /* + * r496b: guest-RAM content scan for the ring (see the gr_ram_cursor + * field comment). The driver's kickoff writes GPFIFO entries whose + * pushbuffer VAs sit just BELOW the ring VA in the same allocation + * (main pushbuffer first, GPFIFO after - nvidia-push-priv.h): entry0 + * in (ring_va - 0x100000, ring_va), entry1 above entry0 and up to + * ring_va + 0x100000 (the progress tracker). A page whose first two + * 8-byte words both name such VAs with sane lengths is the ring; pin + * it (validated) and its allocation window (guesses) so the ring + * entry and pushbuffer reads resolve. Sliced so the service thread + * stays responsive; gives up per channel after + * SVC_GR_RAM_PASS_MAX slices. + */ + if (ch->gr_ram_passes < SVC_GR_RAM_PASS_MAX && + !test_bit(ch->gr_index, &svc_gr_ring_done)) { + u64 hit_pa = 0; + + if (svc_gr_ram_ring_scan(vgpu, ch, &hit_pa)) { + /* + * Pin ONLY the content-validated ring page (validated, + * not a window guess). The pushbuffer pages and the + * tracker push of this client resolve through the VAS + * page directory registered from COPY_SERVER_PDES + * (r496g) - the surface is NONCONTIGUOUS, so a linear + * window guess is wrong for them and, being a + * pin_lookup_any hit, would shadow the correct walk in + * svc_lookup_trusted (observed r502: the tracker push + * read zeros through the wrong window page and the + * driver's DMA poll never completed). + */ + svc_pin_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + ch->gr_ram_logged = true; + ch->gr_ram_ring = true; + set_bit(ch->gr_index, &svc_gr_ring_done); + svc_ring_n++; + /* + * r496j: with the ring content-validated, verify the + * parked COPY_SERVER_PDES candidates against it and + * register the directory that reproduces the ring's + * PA; its walk then resolves the client's other VAs + * (pushbuffer pages, the progress semaphore the + * driver polls). r496f's blind vidmem sweep stays as + * a fallback for directories the control never + * published. + */ + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + hit_pa); + svc_xvas_pdb_discover(vgpu, ch->gpfifo_va, hit_pa); + /* + * Persist the latch immediately: a later stalled + * path in this pass may `continue` past the + * end-of-pass write-back, and the next pass's + * snapshot must carry gr_ram_ring (the zero-NOP + * exception and the no-capture-hold skip key on + * it). + */ + { + unsigned long flags2; + int i2; + + spin_lock_irqsave(&svc.chan_lock, flags2); + for (i2 = 0; i2 < SVC_MAX_CHANNELS; i2++) + if (svc.chan[i2].valid && + svc.chan[i2].vgpu == vgpu && + svc.chan[i2].h_object == + ch->h_object) { + svc.chan[i2].gr_ram_ring = true; + svc.chan[i2].gr_ram_logged = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags2); + } + pr_info("sema-chan: GR ring h=0x%x rank=%u found by RAM content scan va=0x%llx -> pa=0x%llx (allocation window pinned; entry/push reads resolve through it)\n", + ch->h_object, ch->gr_index, ch->gpfifo_va, + hit_pa); + return; + } } } @@ -9496,8 +14934,7 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { u64 pa = 0, va = 0; - int ap = SVC_APERTURE_SYS; - u8 e[SVC_GPFIFO_ENTRY_SIZE]; + int ap = SVC_APERTURE_SYS; u8 e[SVC_GPFIFO_ENTRY_SIZE]; u32 e0, e1, len; if (!ch->engine_type) { @@ -9580,6 +15017,9 @@ static void svc_pb_pair(struct nv_vgpu_device *vgpu, struct svc_channel *ch) } svc_pb[hit_i].va = va; set_bit(ch->ce_index, &svc_pb_done); + /* m177c: the base page was content-read (svc_chan_entry0) - a + * verified UVM-space sample for the ring-space directory. */ + svc_pd_ring_note(va, hit_pa); /* * The pinned pairs are verified samples of UVM's own page directory; @@ -9946,6 +15386,19 @@ static int svc_rescue_run_try(struct nv_vgpu_device *vgpu, } /* + * STALLFREE: wall-clock cap for one content-rescue scan range. The scan is + * bounded by the guest-RAM range, not by time: the full fallback walks up to + * 6 GB in 4 KB vfio reads, and under a pin storm / leftover crashed client's + * churn that walk can run for tens of seconds on the service thread, + * delaying every channel (GR launches included). Green scans answer from the + * hot set or the near window in milliseconds, so a 3 s cap never fires on a + * healthy boot; when it does fire the entry falls through its existing grace + * path (hold, then consume with the software release) instead of pinning the + * whole pump. + */ +#define SVC_RESCUE_SCAN_MAX_MS 3000u + +/* * Scan a guest-RAM range descending for a page that validates as this entry's * push (method stream + expected release payload), pinning it and returning 0 * on the first hit. `hi` and `lo` are inclusive chunk start addresses. @@ -9958,10 +15411,20 @@ static int svc_rescue_scan_range(struct nv_vgpu_device *vgpu, { u64 gpa = hi; unsigned long t0 = jiffies; + unsigned long deadline = + t0 + msecs_to_jiffies(SVC_RESCUE_SCAN_MAX_MS); u32 total = 0; u64 bytes = 0; for (;;) { + if (time_after_eq(jiffies, deadline)) { + svc_rescue_scan_capped++; + if (svc_rescue_scan_capped <= 8) + pr_warn("sema-svc: RESCUE-SCAN-CAP h=0x%x va=0x%llx [0x%llx..0x%llx] %llu MB in %u ms (capped; entry keeps its grace path)\n", + ch->h_object, entry_va, hi, lo, + bytes >> 20, SVC_RESCUE_SCAN_MAX_MS); + break; + } if (!svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) { u32 ncand, n; @@ -10311,20 +15774,96 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, { u32 val; u32 cur = 0; + /* + * STALL FIX (rank-3 completion-word pollution): the publish must + * target the channel's OWN completion word - the descending slot its + * pushes release, which the autoreg above registers and which the + * decoded releases write (release_pa). Writing the ring PUT into the + * ascending rank slot (F1/eea5ea93) made every channel's publish land + * in the paired rank (19-r)'s waited word: the busy ctx-init channel + * rank 16 (h=0x5c000059, PUT~94) overwrote rank 3's word 0x20400ffc0 + * from 1 to 94 (`pub-adv h=0x5c000059 pa=...fc0 val=94 cur=1`), the + * rank-3 stream's own release (payload 1) was then forward-only- + * skipped, and its exact-value sync wait never completed (evtest3 + * cudaStreamSynchronize(st[2]); victory_streams_events). release_pa + * is the push-named truth and wins when known; counter_pa stays the + * fallback for a channel whose pushes are unreadable. + */ + u64 pa = ch->release_pa ? ch->release_pa : ch->counter_pa; - if (!ch->counter_pa) + if (!pa) + return; + /* + * COMPLETION GATE (r205 mail/matmul regression): GP_PUT is the + * client's SUBMISSION position; the pump's last_get is its next + * unconsumed entry. While any submitted entry is outstanding + * (last_get != put) the pushes up to put have NOT all completed, so + * the put must never be published into the completion word: doing so + * short-circuits every client wait (r205 mail: rank-0's word jumped + * to the submitted 108 at last_get=104 - `pub-adv h=0x5c000019 + * pa=...ff0 val=108 cur=104` - ~1 s before the engine ran + * `output retirement wait: 1/968 ... settled`, so the host's + * sub-put release payloads were then forward-only-skipped and the + * client read a stale buffer and failed; matmul's stage ordering + * broke the same way). Only the release applies deliver completions; + * this publish is the catch-up for entries the pump consumed WITHOUT + * a decoded release, so it may run solely when nothing is + * outstanding - the same equality the entry loop consumes to + * (wrap-safe for the 1024-entry rings). + */ + if (ch->last_get != put) return; svc.counter_pub_calls++; if (gsp_sema_dbg && !ch->engine_type && svc.counter_pub_dbg < 96) { svc.counter_pub_dbg++; pr_info("sema-chan: pub-info h=0x%x rank=%u pa=0x%llx put=%u last=%u\n", - ch->h_object, ch->gr_index, ch->counter_pa, put, + ch->h_object, ch->gr_index, pa, put, ch->counter_last); } svc_counter_flush_pending(vgpu, ch); + /* + * mm2 (r207/r208 second-client hang): suppress the GR publish until + * the channel is proven to belong to the current session. During a + * fresh session's first pass the pump consumes the PREVIOUS + * session's residual ring; the completion gate above passes exactly + * then (the stale consumption satisfies last_get == put) and the + * residual put would land in the client's own completion word + * (release_pa). The fresh client's UVM then sees a far-forward + * payload followed by its real release sequence - a downward + * observation larger than UVM_GPU_SEMAPHORE_MAX_JUMP that corrupts + * the tracking counter and wedges the session's first waits (r207 + * victory_matmul: 1 GR push per channel, zero forwards, 242 s). + * gr_pub_armed is set at the fresh-put-drop rewind (session switch + * proven) or when a GR channel is seen with put <= 1 (pristine + * ring, nothing stale to mispublish); a live mid-stream discovery + * (put high, never dropping) loses only this catch-up publish, + * which the release applies already cover. + */ + if (!ch->engine_type && !ch->gr_pub_armed) + return; + val = put; + /* + * r471: never advance the guest's completion word past a push whose + * content was still unwritten when its entry was consumed. The + * clamp position is the lowest all-zero GR slot (gr_bad_put = slot + * + 1; 0 = none). Without it the publish can satisfy a D2H wait + * before its copy ran and the client drains a reused staging buffer + * holding the previous copy's bytes (streams_events h_c[2] = c0). + * The clamp heals when the rewinds re-read the slot with real + * content; the real completions still arrive through the decoded + * releases. + */ + if (!ch->engine_type && ch->gr_bad_put && val >= ch->gr_bad_put) { + val = ch->gr_bad_put - 1; + if (gsp_sema_dbg && svc.counter_pub_dbg < 96) { + svc.counter_pub_dbg++; + pr_info("sema-diag: pub-clamp h=0x%x pa=0x%llx val=%u (bad=%u) cur=%u\n", + ch->h_object, pa, val, ch->gr_bad_put, cur); + } + } if (val == ch->counter_last) return; /* @@ -10332,22 +15871,40 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, * advanced the slot past the put position the client last submitted; * the client polls this u32, so never write it backwards. */ - if (svc_read_ap(vgpu, ch->counter_pa, SVC_APERTURE_SYS, &cur, - sizeof(cur))) + if (svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, &cur, sizeof(cur))) return; if (val <= cur) { if (cur > ch->counter_last) ch->counter_last = cur; return; } - if (svc_write_ap(vgpu, ch->counter_pa, SVC_APERTURE_SYS, - &val, sizeof(val))) - return; + /* + * C5 (GAP-1 H4 secondary): the put-based publish is about to advance + * the same word the decoded releases target; a pre-advance here can + * make a later absolute release a silent skip. Witness every advance. + */ + if (gsp_sema_dbg) + pr_info("sema-diag: pub-adv h=0x%x pa=0x%llx val=%u cur=%u last=%u\n", + ch->h_object, pa, val, cur, + ch->counter_last); + { + int wrc = svc_write_ap(vgpu, pa, SVC_APERTURE_SYS, + &val, sizeof(val)); + if (wrc) { + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: pub-adv-wrc h=0x%x pa=0x%llx val=%u wrc=%d (write did not land)\n", + ch->h_object, pa, val, wrc); + } + return; + } + } ch->counter_last = val; + ch->counter_gen = svc_sess_gen; if (gsp_sema_dbg && svc.counter_pub_dbg < 48) { svc.counter_pub_dbg++; pr_info("sema-chan: counter publish h=0x%x pa=0x%llx val=%u (was %u)\n", - ch->h_object, ch->counter_pa, val, cur); + ch->h_object, pa, val, cur); } } @@ -10441,7 +15998,7 @@ static bool svc_fake_release(struct nv_vgpu_device *vgpu, struct svc_channel *ch svc.releases_applied++; if (!gsp_sema_fake_rel) svc.releases_recovered++; - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); } } } @@ -10495,7 +16052,33 @@ static bool svc_entry_want_reached(struct nv_vgpu_device *vgpu, /* Mark a consumed GPFIFO slot for the current put epoch. */ static void svc_replay_note(struct svc_channel *ch, u32 idx, u32 put) { + static u32 consume_log; + + /* + * fail4 r303: name every consumed slot (all channels). Paired with + * the sema-entry decode rows this shows exactly which entries were + * consumed without ops and when (the vector_add chunk-drop chase: + * a consumed slot whose decode row carried a device-dst copy but + * whose op never ran is the silent skip under diagnosis). + */ + if (gsp_sema_dbg && consume_log < 16384) { + consume_log++; + pr_info("sema-consume: h=0x%x idx=%u put=%u\n", + ch->h_object, idx, put); + } (void)put; + /* + * STALL FIX (entry=82/pay=84 class, r186): the entry leaving the + * decode path must never strand a release set that svc_opdef_release() + * withheld for it while an operand was pending. svc_opdef_flush() is + * only reachable from the entry decode path; once the entry is + * consumed (e.g. via the entry_want shortcut) the withheld set sat + * until a reset/drop cleared it WITHOUT applying, and the guest's + * tracker word stopped short of the waiter's value (evtest3 + * cudaStreamSynchronize(st[2]) hung with starved=0). Deliver the set + * here, forward-only, before the slot is marked done. + */ + svc_opdef_finish_ch(ch, idx); if (idx < SVC_GPFIFO_MAX_BITS) set_bit(idx, ch->slot_done_epoch); } @@ -10622,7 +16205,29 @@ struct svc_fwd_req { bool gr; bool gr_bridge_done; bool gr_page_overflow; + /* + * STALLFREE: bounded-completion state for the forwarded launch. The + * request's backend outcome is the only host edge that can complete a + * launch whose release records engage no hold (svc_fwd_rel_n == 0); + * gr_swrel_done is the exactly-once latch for the software authorship + * of the QMD completion word, and gr_fill_jiffies stamps the request + * so the GR-STALL line reports the real waited time. + */ + bool gr_swrel_done; + /* + * STALLFREE: set by the work item when release0 could not target the + * guest page (unresolved release PA / HPA), so a successful launch's + * engine completion lands only on the module semaphore and the guest + * QMD word stays behind. The pump's bounded outcome check then + * authors it in software (the same write the hold's success edge + * performs), so the client always completes. + */ + bool gr_rel_unobserved; + unsigned long gr_fill_jiffies; u32 gr_h_object; + u32 gr_dropped; /* GRFIX: pages the fill's last pass could not + * resolve (0 = launch fully resolved); read + * by the absolute-bound expiry log */ u64 gr_push_va; u32 gr_push_len; u32 gr_push_n; @@ -10670,6 +16275,8 @@ static struct workqueue_struct *svc_fwd_wq; static unsigned long svc_fwd_state; static u64 svc_fwd_requests; static u64 svc_fwd_dropped; +static u32 svc_grvaudit_launches; /* GRVAUDIT: per-launch audit counter */ +static u32 svc_readdiv_log; /* r233 READ-DIVERGE-WIT budget */ /* * Pause gate shared by the pump's request path and the backend release path: * quiesce takes this lock, sets paused and waits for the in-flight work item; @@ -10681,6 +16288,34 @@ static DEFINE_MUTEX(svc_fwd_lock); static bool svc_fwd_paused; /* + * F1 helpers: does a software release write target the release word of the + * forwarded GR launch the request slot holds? The slot is written only by the + * service thread (this code also runs there) and is never reused while the + * hold is active, so no lock is needed - same convention as the existing + * svc_fwd_hold_gr_release()/svc_fwd_hold_expired() reads. svc_gr_rel_match() + * covers the just-resolved hold (done may already be true); the inflight + * variant additionally requires the SVC_FWD_INFLIGHT bit and an unresolved + * work item, i.e. the hardware copy the poll is waiting on is still running. + */ +static bool svc_gr_rel_match(u64 va, u32 *payload) +{ + if (!va || !svc_fwd_req.gr || svc_fwd_req.gr_rel_va != va) + return false; + if (payload) + *payload = svc_fwd_req.gr_rel_payload; + return true; +} + +static bool svc_gr_rel_inflight(u64 va, u32 *payload) +{ + if (!svc_gr_rel_match(va, payload)) + return false; + if (!test_bit(SVC_FWD_INFLIGHT, &svc_fwd_state)) + return false; + return !smp_load_acquire(&svc_fwd_req.done); +} + +/* * Resolve one captured operand to an HPA. SYS (guest RAM) operands resolve * the page-aligned GPA through the guest-RAM backing file; VID operands * resolve the shadow byte offset through the vmalloc backing of BAR1 (the @@ -10710,11 +16345,29 @@ static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind, return -EFAULT; *hpa = (u64)page_to_phys(pg); *out_off = page_off; + /* + * r372 two-path identity check: the copy-out writes through + * __va(hpa) while every reader reads base+pa - if those are + * not the same memory the engine's output silently vanishes + * (the sum=0 class). Prove it empirically, bounded. + */ + if (svc_fbhpa_log < 16) { + u64 at_base = 0, at_va = 0; + + svc_fbhpa_log++; + memcpy(&at_base, base + page, sizeof(at_base)); + memcpy(&at_va, (u8 *)__va(*hpa), sizeof(at_va)); + pr_info("sema-fbhpa: pa=0x%llx base=%px pg=%px hpa=0x%llx shadow=0x%llx at_base=%016llx at___va=%016llx %s\n", + page, base, pg, *hpa, + (unsigned long long)svc_fb_shadow_size(vgpu), + at_base, at_va, + at_base == at_va ? "SAME" : "DIFFER"); + } return 0; } if (kind != SVC_APERTURE_SYS) return -EINVAL; - ret = gsp_phys_gpa_to_hpa(page, hpa); + ret = gsp_phys_gpa_to_hpa_vgpu(vgpu, page, hpa); if (ret) return ret; *out_off = page_off; @@ -10829,6 +16482,19 @@ static void svc_fwd_soft_copy_inline(struct nv_vgpu_device *vgpu) * hardware has not produced yet. */ #define SVC_FWD_GR_HOLD_TIMEOUT_MS 12000u +/* + * r420: hard cap on the WEDGE-QUIET deferral of the GR hold deadline. The + * backend's quiet hold is a bounded msleep (its own caps: <=240 s per hold, + * two holds per launch), so the pause can only last a few minutes; the cap is + * a defence-in-depth bound so a stuck quiet-hold flag can never defer the + * client's completion indefinitely. + */ +#define SVC_FWD_GR_HOLD_PAUSE_MAX_MS 600000u +/* + * r420: same defence for the per-launch consumption bound pause in + * svc_gr_hold_bound_expired(). + */ +#define SVC_GR_QUIET_PAUSE_MAX_MS 600000u struct svc_fwd_hold { bool active; @@ -10864,9 +16530,9 @@ struct svc_fwd_hold { /* * P6c completion relay: the request selected record relay_index and * the hardware copy was asked to author it into the guest-visible - * semaphore page. On the success edge the hold re-reads the guest - * word and, when it carries the payload, skips that one record's - * software apply. Cleared with the hold. + * semaphore page. On the success edge the hold re-reads the guest word + * and, when it carries the payload, skips that one record's software + * apply. Cleared with the hold. */ bool relay_armed; u8 relay_index; @@ -10879,13 +16545,26 @@ struct svc_fwd_hold { struct svc_fwd_rel rel[SVC_FWD_REL_MAX]; u64 last_rel_va; unsigned long since; + /* + * r420: WEDGE-QUIET deferral state. While the backend deliberately + * holds the in-flight launch in its bounded quiet window + * (gsp_phys_gr_quiet_hold_active), the GR deadline is pushed forward + * (paused_until) so the hold cannot author the launch's release word + * under the hold - the client would then verify data the post-hold + * copy-out has not written yet (the r419 mid-boot escape). The + * deferral is itself bounded by SVC_FWD_GR_HOLD_PAUSE_MAX_MS from + * `since` and is announced once per hold. + */ + unsigned long paused_until; + bool pause_logged; }; static struct svc_fwd_hold svc_fwd_hold; -static void svc_fwd_hold_abort(u32 h_object) +static void svc_fwd_hold_abort(struct nv_vgpu_device *vgpu, u32 h_object) { if (READ_ONCE(svc_fwd_hold.active) && + svc_fwd_hold.vgpu == vgpu && svc_fwd_hold.ch_handle == h_object) WRITE_ONCE(svc_fwd_hold.aborted, true); } @@ -10929,8 +16608,19 @@ static void svc_fwd_rel_apply_launch_deferred(struct nv_vgpu_device *vgpu, bool pending = r->op != 0; bool inc = DMA_SEM_REDUCTION_EN(r->dma) && DMA_SEM_REDUCTION(r->dma) == SEM_REDUCTION_INC; + int rd = svc_sem_read(vgpu, r->va, &cur); - if (svc_sem_read(vgpu, r->va, &cur)) { + /* + * C2 (GAP-1 H2/H3): prove the opdef apply handed the record to the + * helper and what the software read of the release word returned. + */ + if (gsp_sema_dbg && svc_relx_log < 128) { + svc_relx_log++; + pr_info("sema-diag: relx kind=%u op=%u inc_claim=%u va=0x%llx pay=%u rd=%d\n", + (unsigned)r->kind, (unsigned)r->op, + (unsigned)r->inc_claimed, r->va, r->payload, rd); + } + if (rd) { svc_pending_add(r->va, r->payload, inc ? 1 : 0); svc_rel_apply_launch(vgpu, ch, r->va, r->payload, 0, pending, inc, r->inc_claimed); @@ -11067,8 +16757,54 @@ static struct svc_opdef_ent *svc_opdef_alloc(struct nv_vgpu_device *vgpu, return NULL; } -static void svc_opdef_clear(struct svc_opdef_ent *e) +/* C3 caller tags for the opdef-clear witness (H2: which path cleared). */ +#define SVC_OPDEF_CLR_APPLY 0u +#define SVC_OPDEF_CLR_FINISH 1u +#define SVC_OPDEF_CLR_FLUSH 2u +#define SVC_OPDEF_CLR_DROP 3u +#define SVC_OPDEF_CLR_STALE 4u +#define SVC_OPDEF_CLR_COUNTED 5u +#define SVC_OPDEF_CLR_SWEEP 6u +#define SVC_OPDEF_CLR_STOP 7u + +static void svc_opdef_clear(struct svc_opdef_ent *e, u32 caller) { + /* + * C3 (GAP-1 H2): every clear is witnessed with its caller and the + * slot contents, so a "silent" clear (n=0, or a kind the apply + * switch ignores) is distinguishable from a delivery. + */ + if (gsp_sema_dbg && e->active && svc_opdef_clr_log < 128) { + svc_opdef_clr_log++; + pr_info("sema-diag: opdef-clear caller=%u h=0x%x entry=%u n=%u kind0=%u va0=0x%llx pay0=%u\n", + caller, e->h_object, e->entry_idx, (unsigned)e->n, + e->n ? (unsigned)e->rel[0].kind : 0u, + e->n ? e->rel[0].va : 0ull, + e->n ? e->rel[0].payload : 0u); + } + /* + * r456/r457: the snapshot lifecycle is now "capture once, consume + * on the copy's success, evict on ring-slot reuse, TTL". Every + * opdef_clear caller can still re-decode the entry: + * - APPLY (0): the pass's own op execution (which consumes the + * snapshot on success) follows the apply; + * - FINISH (1): the entry can be REPLAYED after a consume (the + * pin-batch rewind replays entries while mappings stream in), + * and an owed copy still needs the pipeline-order bytes; + * - FLUSH (2): the op stays skipped this pass but a later replay + * may run it. + * The r457 mode-C evidence: the b1 H2D's snapshot was captured at + * 191607.816388 with b1's bytes (first4=00000000 3e800000 ...), + * FINISH freed it on the entry's consume, the rewind replay + * re-captured at 191608.798838 from the REFILLED staging + * (first4=00000000 3d800000 ... = b2) and the deferred execution + * wrote b2's bytes into db1 (dst16_after=000000000000803d...). + * The r444 leak the old free guarded against is already covered + * by the owner key (a stale snapshot can only ever serve its own + * (ch, idx, hash)), the slot-reuse eviction in + * svc_copy_snap_capture, the ops_done skip free in svc_run_ops + * and the 30 s TTL. + */ memset(e, 0, sizeof(*e)); } @@ -11100,13 +16836,78 @@ static void svc_opdef_apply(struct nv_vgpu_device *vgpu, if (e->n) { svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); svc.opdef_applied++; - if (gsp_sema_dbg && svc_opdef_log < 32) { + if (gsp_sema_dbg && svc_opdef_log < (gsp_sema_dbg ? 512 : 32)) { svc_opdef_log++; pr_info("sema-opdef: apply h=0x%x entry=%u n=%u (ops complete)\n", ch->h_object, entry_idx, e->n); } } - svc_opdef_clear(e); + svc_opdef_clear(e, SVC_OPDEF_CLR_APPLY); +} + +/* + * r34x: is a release to this VA currently withheld by our own machinery? + * True when the pending-translation ring holds one (translation late) or + * an opdef slot has one recorded (operand hold). The acquire gate calls + * this to classify an unsatisfied wait in its diagnostics: pend=1 names + * the 'value sits behind our own withhold' case - the bounded 100 ms + * give-up is exactly for it (waiting longer cannot install the value + * until the upstream entry's ops complete). Never forces an opdef release + * here: those releases are withheld because their ops have NOT completed, + * and delivering early re-opens the work-before-fence hole they close. + */ +static bool svc_wait_pend_release(u64 va) +{ + u32 i; + + for (i = 0; i < svc_pending_n; i++) + if (svc_pending[i].va == va) + return true; + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + u8 k; + + if (!e->active) + continue; + for (k = 0; k < e->n; k++) + if (e->rel[k].va == va) + return true; + } + return false; +} + +/* + * STALL FIX (r186 entry=82/pay=84): called from svc_replay_note() on every + * entry-consume path. If a release set was withheld for this entry while an + * operand was pending and the entry is now leaving the decode path (so the + * 10 s flush can never run for it), deliver the set forward-only and clear + * the slot. A drop without delivery is the stall class: the guest word + * stops short of the value its tracker waits for. + */ +static void svc_opdef_finish_ch(struct svc_channel *ch, u32 idx) +{ + u32 i; + + if (!ch || idx == SVC_ENTRY_NONE) + return; + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + + if (!e->active || e->h_object != ch->h_object || + e->vgpu != ch->vgpu || e->entry_idx != idx) + continue; + if (e->n) { + svc_fwd_rel_apply_all(e->vgpu, ch, e->rel, e->n); + svc.opdef_delivered_late++; + if (svc_opdef_log < 64) { + svc_opdef_log++; + pr_info("sema-opdef: LATE-FLUSH h=0x%x entry=%u n=%u (entry consumed before deadline; releases delivered)\n", + ch->h_object, idx, e->n); + } + } + svc_opdef_clear(e, SVC_OPDEF_CLR_FINISH); + return; + } } /* @@ -11143,7 +16944,7 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, if (!svc_opdef_contains(e, r)) { if (r->payload && e->n < SVC_FWD_REL_MAX) { e->rel[e->n++] = *r; - if (gsp_sema_dbg && svc_opdef_log < 32) { + if (gsp_sema_dbg && svc_opdef_log < (gsp_sema_dbg ? 512 : 32)) { svc_opdef_log++; pr_info("sema-opdef: hold h=0x%x entry=%u kind=%u va=0x%llx pay=%u (operand pending)\n", ch->h_object, entry_idx, @@ -11185,7 +16986,7 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, * flushes the slot and consumes the entry. */ static bool svc_opdef_hold(struct nv_vgpu_device *vgpu, struct svc_channel *ch, - u32 entry_idx) + u32 entry_idx, u64 launch_va) { struct svc_opdef_ent *e = svc_opdef_find(vgpu, ch->h_object, entry_idx); @@ -11194,11 +16995,46 @@ static bool svc_opdef_hold(struct nv_vgpu_device *vgpu, struct svc_channel *ch, if (!e) return false; } + /* + * r361: absolute per-launch bound keyed to the LAUNCH PUSH VA + * (the GRFIX lesson, r213). The slot's `since` restarts whenever + * a pin-batch rewind replays the entry at a different GPFIFO + * index, so a slot-keyed-only bound can be re-armed forever - + * observed as opdef=3301/32/0 (held climbing, expired 0) with the + * guest wedged. The push VA never changes across replays, so the + * first hold of a launch arms an ABSOLUTE deadline nothing moves. + */ + if (!launch_va) + launch_va = 1; /* anonymous launch: still absolutely bound */ + /* + * r368: an ACTIVE bound is never re-armed. The pump walks up to + * SVC_OPDEF_SLOTS held entries, each with its OWN push VA; the r361 + * arm-on-VA-change rule restarted the deadline on every entry + * change, so with several held launches the deadline moved forever + * (streams2 wedge: opdef=955/13/0, held climbing, expired 0). The + * first hold arms the absolute deadline; entries held while it is + * active ride it and flush together when it fires. A fresh bound is + * armed once the previous one has passed. + */ + if (!ch->opdef_hold_dead_va || + time_after_eq(jiffies, ch->opdef_hold_deadline)) { + ch->opdef_hold_dead_va = launch_va; + ch->opdef_hold_deadline = jiffies + + msecs_to_jiffies(SVC_OPDEF_HOLD_MS); + ch->opdef_hold_exp_logged = false; + } if (time_before(jiffies, - e->since + msecs_to_jiffies(SVC_OPDEF_HOLD_MS))) { + e->since + msecs_to_jiffies(SVC_OPDEF_HOLD_MS)) && + time_before(jiffies, ch->opdef_hold_deadline)) { svc.opdef_held++; return true; } + if (!ch->opdef_hold_exp_logged) { + ch->opdef_hold_exp_logged = true; + pr_err_ratelimited("vgsp-sema: OPDEF-EXPIRED h=0x%x idx=%u va=0x%llx (absolute per-launch bound %u ms; rewind replays no longer re-arm)\n", + ch->h_object, entry_idx, launch_va, + SVC_OPDEF_HOLD_MS); + } return false; } @@ -11216,7 +17052,7 @@ static void svc_opdef_flush(struct nv_vgpu_device *vgpu, struct svc_channel *ch, ch->h_object, entry_idx, e->n); } svc.opdef_expired++; - svc_opdef_clear(e); + svc_opdef_clear(e, SVC_OPDEF_CLR_FLUSH); } /* @@ -11230,17 +17066,67 @@ static void svc_opdef_drop(struct nv_vgpu_device *vgpu, struct svc_channel *ch, { struct svc_opdef_ent *e = svc_opdef_find(vgpu, ch->h_object, entry_idx); - if (e) - svc_opdef_clear(e); + if (!e) + return; + /* + * STALL FIX (r186/r187 entry=82, va=0x20400fef0 pay=84): when the + * forwarding machinery takes ownership of an entry, the withheld set + * may NOT ride the forwarded candidate's rel[] (svc_opdef_release() + * returned before the release was appended), so a blind clear strands + * the guest's tracker word with NO LATE-FLUSH/LOSS/EXPIRED ever + * logged (crel freezes right after the launches; evtest3 + * cudaStreamSynchronize(st[2]) hangs). Counter releases are + * monotonic completion signals: apply them forward-only here - a + * record at/ahead of the word is a no-op, so the fwd edge cannot + * double-apply and an additive reduction is skipped. + */ + { + u8 i; + u32 delivered = 0; + + for (i = 0; i < e->n; i++) { + if (!svc_counter_va(e->rel[i].va)) + continue; + if (e->rel[i].kind == SVC_FWD_REL_906F && + e->rel[i].op == HOST_SEM_OP_REDUCTION) + continue; /* additive: not idempotent */ + svc_fwd_rel_apply_all(vgpu, ch, &e->rel[i], 1); + svc.opdef_delivered_late++; + delivered++; + pr_info_ratelimited("sema-opdef: DROP-APPLY h=0x%x entry=%u va=0x%llx pay=%u (counter release delivered at fwd-ownership handoff)\n", + ch->h_object, entry_idx, + e->rel[i].va, e->rel[i].payload); + } + if (delivered < e->n) { + svc.opdef_drop_lost += (u32)(e->n - delivered); + pr_info_ratelimited("sema-opdef: LOSS h=0x%x entry=%u %u of %u withheld release(s) dropped at fwd-ownership handoff\n", + ch->h_object, entry_idx, + (u32)(e->n - delivered), e->n); + } + } + svc_opdef_clear(e, SVC_OPDEF_CLR_DROP); } /* * The ring advanced past a slot's entry (a slot for another entry of the * same channel is being processed): its set belongs to a consumed entry. * Deliver late (forward-only, so it cannot double-apply) and drop it. + * + * r406: an entry is only "passed" if the ring actually CONSUMED it + * (slot_done_epoch). Without that proof the pin-batch rewind breaks the + * invariant: it re-opens lower indices (e.g. 83) while a held entry + * (e.g. 91) sits AHEAD in the same epoch, and the old unconditional drop + * treated the held entry as consumed and delivered its withheld releases + * early. Observed (r406 inplace_scale): the H2D copy's operand was + * unresolved (rc=-2), the entry held with its release withheld; a rewind + * pass at idx=83 fired drop-stale caller=4 for idx=91 and the release + * went out at 104955.751 - while the copy only executed at 104957.418. + * The client raced past its blocking memcpy, launched, the launch read + * the stale buffer (previous run's data), and the copy landed after the + * launch's copy-out: wrong results with 32/1032 "changed". */ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu, - struct svc_channel *ch, u32 keep_idx) + struct svc_channel *ch, u32 keep_idx, u32 put) { int i; @@ -11250,36 +17136,115 @@ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu, if (!e->active || e->vgpu != vgpu || e->h_object != ch->h_object || e->entry_idx == keep_idx) continue; + if (!svc_replay_seen(ch, e->entry_idx, put)) + continue; /* never consumed: not stale (rewind) */ if (e->n) svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); - svc_opdef_clear(e); + svc_opdef_clear(e, SVC_OPDEF_CLR_STALE); + } +} + +/* A slot of THIS pump's guest whose channel is gone from the pass snapshot: + * drop it. Slots of other guests are never touched (their channels are not in + * this snapshot by construction). */ +static void svc_opdef_drop_counted(struct svc_opdef_ent *e) +{ + if (e->n) { + svc.opdef_drop_lost += e->n; + pr_info_ratelimited("sema-opdef: LOSS entry h=0x%x idx=%u dropped with %u undelivered release(s)\n", + e->h_object, e->entry_idx, e->n); } + svc_opdef_clear(e, SVC_OPDEF_CLR_COUNTED); } -/* A slot whose channel is gone from this pass's snapshot: drop it. */ -static void svc_opdef_drop_gone(const struct svc_channel *local, int n) +static void svc_opdef_drop_gone(struct nv_vgpu_device *vgpu, + const struct svc_channel *local, int n) { int i, k; for (i = 0; i < SVC_OPDEF_SLOTS; i++) { struct svc_opdef_ent *e = &svc_opdef[i]; - if (!e->active) + if (!e->active || e->vgpu != vgpu) continue; for (k = 0; k < n; k++) - if (local[k].h_object == e->h_object) + if (local[k].vgpu == e->vgpu && + local[k].h_object == e->h_object) break; if (k == n) - svc_opdef_clear(e); + svc_opdef_drop_counted(e); } } static void svc_opdef_reset_all(void) { + int i; + + /* STALL FIX: a non-applying clear is a silent release loss - count it. */ + for (i = 0; i < SVC_OPDEF_SLOTS; i++) + if (svc_opdef[i].active && svc_opdef[i].n) + svc_opdef_drop_counted(&svc_opdef[i]); memset(svc_opdef, 0, sizeof(svc_opdef)); } /* + * STALL FIX (r189 entry=82/83 va=0x20400fef0 pay=84/85): the 10 s deadline + * flush (svc_opdef_flush) is only reachable while the pump keeps RE-DECODING + * the entry. When a withheld entry stops being revisited (ring advanced, the + * channel went quiet, the entry was parked while the pump moved on), the + * deadline never runs and the withheld counter release strands forever - + * the guest's tracker word stops short (evtest3 cudaStreamSynchronize(st[2]) + * hangs; opdef=531/8/0 static, no EXPIRED/LOSS lines). Sweep expired slots + * once per service iteration: deliver forward-only to the still-live owner + * channel, count a LOSS when the owner is gone. + */ +static void svc_opdef_expire_sweep(void) +{ + /* + * struct svc_channel is >25 KB (per-slot op bitmaps); a stack copy + * overflows the 16 KB kernel stack. Only the single service thread + * calls this, so a static scratch is safe (same pattern as + * svc_handle_chan_dump / svc_pump_channels). + */ + static struct svc_channel ch_scratch; + int i; + + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + struct svc_channel *local = NULL; + unsigned long flags; + int c; + + if (!e->active || !e->n) + continue; + if (time_before(jiffies, e->since + + msecs_to_jiffies(SVC_OPDEF_HOLD_MS))) + continue; + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) + if (svc.chan[c].valid && + svc.chan[c].vgpu == e->vgpu && + svc.chan[c].h_object == e->h_object) { + ch_scratch = svc.chan[c]; + local = &ch_scratch; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (local) { + svc_fwd_rel_apply_all(e->vgpu, local, e->rel, e->n); + svc.opdef_expired++; + pr_info_ratelimited("sema-opdef: SWEEP-EXPIRED h=0x%x entry=%u n=%u applied forward-only (entry parked past the hold window)\n", + e->h_object, e->entry_idx, e->n); + } else { + svc.opdef_drop_lost += e->n; + pr_info_ratelimited("sema-opdef: SWEEP-LOSS h=0x%x entry=%u n=%u owner channel gone (releases dropped)\n", + e->h_object, e->entry_idx, e->n); + } + svc_opdef_clear(e, SVC_OPDEF_CLR_SWEEP); + } +} + +/* * Proof line for a held record set resolved without a successful forwarded * copy (failure/timeout/abort): the releases below are applied directly, so * none is lost to a hardware copy that never ran. Called with svc_fwd_hold @@ -11316,6 +17281,16 @@ static void svc_gr_sw_release(struct nv_vgpu_device *vgpu, u32 h, u64 va, return; if (!svc_sem_read(vgpu, va, &cur) && cur >= payload) return; + /* + * F1-a: this is the hold/fallback authorship of the forwarded launch's + * release word (src="hold"), budgeted with the sweep tag so one + * 256-line session window covers both authors. + */ + if (svc_gr_rel_match(va, NULL) && svc.gr_rel_auth_log < 256) { + svc.gr_rel_auth_log++; + pr_info("sema-diag: GR-REL-AUTHOR src=hold va=0x%llx pay=%u h=0x%x\n", + va, payload, h); + } if (!svc_sem_write(vgpu, h, va, payload)) { svc.fwd_gr_sw_rel++; pr_info("vgsp-work: GR launch release software-applied h=0x%x va=0x%llx pay=0x%x (no successful hardware copy; forward-only)\n", @@ -11338,11 +17313,107 @@ static void svc_fwd_hold_gr_release(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_rel_payload); } +/* + * STALLFREE: bounded completion for a failed forwarded GR launch that no hold + * completes. A GR push whose decode recorded no push-level release records + * (svc_fwd_rel_n == 0) engages no release hold, so when the backend fails + * (ladder timeout, fault abort, refusal) nothing authors the QMD completion + * word the guest client polls after the kick: the client's launch wait then + * has no host edge and lasts until the guest's own timeout (the r219/r222 + * stall class, qmd_rel frozen at the previous launch's value). + * + * This runs on the service thread at the top of every pump pass, before any + * request fill of that pass (svc_fwd_request requires SVC_FWD_INFLIGHT free, + * which the work item only releases after publishing done, and the fill's + * first write is done=false), so a pass that sees done also sees this + * request's fields, and every finished request is observed before its slot + * can be reused. Authors the release once (forward-only, full resolver - the + * same write the hold's failure edge performs), latches gr_swrel_done, and + * logs the distinguishable GR-STALL line. + * + * While a hold is active the hold owns the completion (its failure edge + * applies the recorded releases AND the QMD word, bounded by the existing + * hold timeout), so this check stands down for it. + */ +static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) +{ + bool done = smp_load_acquire(&svc_fwd_req.done); + unsigned long waited; + + if (!svc_fwd_req.gr || svc_fwd_req.vgpu != vgpu || + svc_fwd_req.gr_swrel_done || svc_fwd_hold.active || !done) + return; + /* + * Complete when the backend failed (the never-completing r219/r222 + * class) OR when a successful launch's release0 could not target the + * guest page, so the engine's completion landed only on the module + * semaphore and the client's QMD word is still behind (the same edge + * the hold's success path covers for held requests). + */ + if (svc_fwd_req.hw_rc == 0 && !svc_fwd_req.gr_rel_unobserved) + return; + /* + * The outcome is final; latch it first so no later pass, hold edge or + * consume tail accounts it again (the release write itself is + * forward-only, so even a duplicate could not corrupt the word). + */ + svc_fwd_req.gr_swrel_done = true; + waited = jiffies_to_msecs(jiffies - svc_fwd_req.gr_fill_jiffies); + if (!svc_fwd_req.gr_rel_va || !svc_fwd_req.gr_rel_payload) { + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (launch carries no release record; no word to author)\n", + svc_fwd_req.gr_push_va, waited); + return; + } + svc_gr_sw_release(vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_rel_va, + svc_fwd_req.gr_rel_payload); + svc.fwd_gr_stall_swrel++; + if (svc_fwd_req.hw_rc != 0) + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (failed forwarded launch, rc=%d; QMD release authored in software)\n", + svc_fwd_req.gr_push_va, waited, svc_fwd_req.hw_rc); + else + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (successful launch whose release0 could not target the guest page; QMD release authored in software)\n", + svc_fwd_req.gr_push_va, waited); + pr_info("vgsp-gr: GR-COMPLETE-SW h=0x%x push=0x%llx rel=0x%llx pay=0x%x waited_ms=%lu n=%llu (backend outcome without a hold)\n", + svc_fwd_req.gr_h_object, svc_fwd_req.gr_push_va, + svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload, waited, + svc.fwd_gr_stall_swrel); +} + static bool svc_fwd_hold_expired(void) { - if (svc_fwd_req.gr && !smp_load_acquire(&svc_fwd_req.done)) + if (svc_fwd_req.gr && !smp_load_acquire(&svc_fwd_req.done)) { + /* + * r420: while the backend deliberately holds the in-flight + * launch in its bounded WEDGE-QUIET window, the GR deadline + * must not fire - expiring here authors the launch's release + * word (svc_fwd_hold_gr_release) and applies the recorded + * releases before the post-hold copy-out, so the client + * verifies stale data (the r419 soak's mid-boot + * "matmul fails after time"). Pause instead, bounded by + * SVC_FWD_GR_HOLD_PAUSE_MAX_MS from the hold's start; the + * pause extends the deadline a full GR window past the last + * quiet-hold pass, so the backend's retry/accept can finish + * before the fallback would author. Announced once per hold. + */ + if (gsp_phys_gr_quiet_hold_active() && + time_before(jiffies, svc_fwd_hold.since + + msecs_to_jiffies(SVC_FWD_GR_HOLD_PAUSE_MAX_MS))) { + if (!svc_fwd_hold.pause_logged) { + svc_fwd_hold.pause_logged = true; + pr_warn("vgsp-gr: GR-HOLD-PAUSE va=0x%llx (backend wedge-quiet hold active; release authorship deferred under the bounded hold)\n", + svc_fwd_req.gr_push_va); + } + svc_fwd_hold.paused_until = jiffies + + msecs_to_jiffies(SVC_FWD_GR_HOLD_TIMEOUT_MS); + return false; + } + if (svc_fwd_hold.paused_until && + time_before(jiffies, svc_fwd_hold.paused_until)) + return false; return time_after(jiffies, svc_fwd_hold.since + msecs_to_jiffies(SVC_FWD_GR_HOLD_TIMEOUT_MS)); + } return time_after(jiffies, svc_fwd_hold.since + msecs_to_jiffies(SVC_FWD_HOLD_TIMEOUT_MS)); } @@ -11376,6 +17447,9 @@ static void svc_fwd_hold_engage(struct nv_vgpu_device *vgpu, svc_fwd_hold.relay_armed = svc_fwd_req.relay; svc_fwd_hold.relay_index = svc_fwd_req.relay_index; svc_fwd_hold.fault_delivered = false; + /* r420: a fresh hold starts with no WEDGE-QUIET deferral. */ + svc_fwd_hold.paused_until = 0; + svc_fwd_hold.pause_logged = false; svc.fwd_hold_engaged++; svc.fwd_rel_pending = n; } @@ -11402,6 +17476,8 @@ static void svc_fwd_hold_clear(void) svc_fwd_hold.relay_armed = false; svc_fwd_hold.relay_index = 0; svc_fwd_hold.fault_delivered = false; + svc_fwd_hold.paused_until = 0; + svc_fwd_hold.pause_logged = false; svc.fwd_rel_pending = 0; } @@ -11555,6 +17631,15 @@ static void svc_fwd_hold_expire_check(struct nv_vgpu_device *vgpu, svc_fwd_hold.relay_armed) svc_fwd_fault_deliver(hv, ch); /* + * STALLFREE: the unpumpable-channel expiry edge (the entry loop never + * ran for this hold). The releases and the QMD completion word are + * software-applied below; exactly one line per hold. + */ + if (svc_fwd_req.gr) + pr_warn("vgsp-gr: GR-STALL-fwd-hold va=0x%llx waited_ms=%u action=proceed (hold expired on an unpumpable channel; releases + QMD word software-applied)\n", + svc_fwd_req.gr_push_va, + jiffies_to_msecs(jiffies - svc_fwd_hold.since)); + /* * P6b wave 2: a skipped candidate whose hardware copy did not complete * successfully is authored by one software copy here, BEFORE the * recorded releases below. A successful hardware copy needs none. @@ -11665,9 +17750,36 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, * done && hw_rc == -ETIMEDOUT; abort/cancel paths never write * guest memory). */ + /* + * C7 (GAP-2 A1): the false-completion edge - a hold whose + * hardware copy failed/timed out still software-applies the + * release set here. + */ + if (gsp_sema_dbg && svc_hold_fb_log < 64) { + svc_hold_fb_log++; + pr_info("sema-diag: hold-fallback h=0x%x gr=%d done=%d hw_rc=%d nrel=%u rel0=0x%llx/%u -> sw apply\n", + ch->h_object, ch->engine_type == 0 ? 1 : 0, + done ? 1 : 0, svc_fwd_req.hw_rc, + (unsigned)svc_fwd_hold.nrel, + svc_fwd_hold.nrel ? + svc_fwd_hold.rel[0].va : 0ull, + svc_fwd_hold.nrel ? + svc_fwd_hold.rel[0].payload : 0u); + } if (done && svc_fwd_req.hw_rc == -ETIMEDOUT && svc_fwd_hold.relay_armed) svc_fwd_fault_deliver(hv, ch); + /* + * STALLFREE: the bounded failure edge of a held forwarded GR + * launch - the releases AND the QMD completion word are + * software-applied below, so the client always completes. + * Exactly one line per hold (the hold is cleared at the end). + */ + if (svc_fwd_req.gr) + pr_warn("vgsp-gr: GR-STALL-fwd-fail va=0x%llx waited_ms=%u action=proceed (hold resolved on backend failure/expiry; releases + QMD word software-applied)\n", + svc_fwd_req.gr_push_va, + jiffies_to_msecs(jiffies - + svc_fwd_hold.since)); svc_fwd_hold_soft_copy(hv); if (svc_fwd_hold.relay_armed) svc.fwd_relay_fallback++; @@ -11693,6 +17805,14 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, ch->last_get = (idx + 1) % ch->gpfifo_entries; ch->rescue_fail = 0; ch->defer_since = 0; + /* + * GRFIX FIX1c (r214): do NOT reset the per-launch bound record + * here. This consume can belong to a different entry than the held + * launch, and resetting on every consume re-armed the bound before + * it could pass (r214: 420 re-holds, GRFILL-EXPIRED=0 over 220 s). + * The record is keyed by the launch push VA; it resets only on + * channel (re)registration or the session diag reset. + */ ch->submissions++; svc.gpfifo_entries_done++; if (ch->engine_type && ch->expected_sem_payload) @@ -11709,6 +17829,31 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, } /* + * Is this mdev still on the driver list with an open VFIO device? Decides + * whether a hold that names it can still be resolved through it: a removed + * device must never be dereferenced (its struct can be freed before the next + * pump pass would see the hold). Pointer comparisons only - safe to call + * with a possibly-stale pointer. + */ +static bool svc_vgpu_open(const struct nv_vgpu_device *vgpu) +{ + struct nv_vgpu_device *d; + bool open = false; + + if (!vgpu) + return false; + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (d == vgpu && atomic_read(&d->vfio_dev_open_count) > 0) { + open = true; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return open; +} + +/* * Channel vanished while a hold was active. Any resolved hold (completed, * aborted, failed or expired) applies the recorded releases through a shim * channel - holds exist only for CE channels and the vanished channel object @@ -11728,6 +17873,18 @@ static void svc_fwd_hold_gone_finish(struct nv_vgpu_device *vgpu) bool done = smp_load_acquire(&svc_fwd_req.done); bool aborted = READ_ONCE(svc_fwd_hold.aborted); + if (svc_fwd_hold.vgpu && !svc_vgpu_open(svc_fwd_hold.vgpu)) { + /* + * The owning mdev was removed mid-hold (the mdev cleanup path + * already released its session state): its guest can never + * observe the releases and dereferencing the device would be a + * use-after-free. Drop the hold; the request slot frees for the + * next guest. + */ + svc.fwd_hold_canceled++; + svc_fwd_hold_clear(); + return; + } if (!done && !aborted && !svc_fwd_hold_expired()) return; /* @@ -11780,6 +17937,15 @@ static void svc_fwd_hold_stop_flush(struct nv_vgpu_device *vgpu) if (!svc_fwd_hold.active) return; + /* + * A hold whose owning mdev is already gone must never be dereferenced: + * drop it (the guest cannot observe the releases anyway). + */ + if (svc_fwd_hold.vgpu && !svc_vgpu_open(svc_fwd_hold.vgpu)) { + svc.fwd_hold_canceled++; + svc_fwd_hold_clear(); + return; + } done = smp_load_acquire(&svc_fwd_req.done); /* * P6b wave 2: the backend is going away; a skipped candidate whose @@ -11836,6 +18002,10 @@ static void svc_fwd_gr_work(void) struct gsp_phys_backend_copy_result cp = { 0 }; u64 rel_hpa = 0; u32 off = 0, i; + /* GRVAUDIT: aggregated from the data[]-build loop below (first + * problem page in ga_va/ga_used/ga_led; 0 while clean). */ + u32 ga_unlanded = 0, ga_ptebad = 0; + u64 ga_va = 0, ga_used = 0, ga_led = 0; int hw_rc; memset(&gs, 0, sizeof(gs)); @@ -11868,6 +18038,7 @@ static void svc_fwd_gr_work(void) (SEMA_PAGE_SIZE - 1)); u8 rel_kind = (svc_fwd_req.gr_rel_ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + bool rel_unobserved = false; if (svc_fwd_req.gr_rel_pa && (svc_fwd_operand_hpa(svc_fwd_req.vgpu, rel_kind, @@ -11880,11 +18051,24 @@ static void svc_fwd_gr_work(void) * means the engine would author nothing this launch; point * release0 at the module semaphore instead so the host still * observes completion. + * + * STALLFREE: when the guest page could not be targeted at all + * (unresolved release PA or HPA), the engine's completion will + * land only on the module semaphore and the guest QMD word + * stays behind - a launch whose client would otherwise wait + * forever even on a SUCCESSFUL submit. Record it so the pump's + * bounded outcome check authors the word in software (the + * same write the hold's success edge performs). A resolved + * page whose word is already ahead needs nothing. */ - if (rel_hpa && - READ_ONCE(*(u32 *)((u8 *)__va(rel_hpa) + roff)) >= + if (rel_hpa) { + if (READ_ONCE(*(u32 *)((u8 *)__va(rel_hpa) + roff)) >= svc_fwd_req.gr_rel_payload) - rel_hpa = 0; + rel_hpa = 0; + } else { + rel_unobserved = true; + } + svc_fwd_req.gr_rel_unobserved = rel_unobserved; } gs.qmd = svc_fwd_req.gr_qmd; @@ -11920,6 +18104,96 @@ static void svc_fwd_gr_work(void) data[gs.n_data].va = dg->va; data[gs.n_data].hpa = dhpa; data[gs.n_data].reloc_va = dg->reloc_va; + /* + * r348 diagnostic: the last 32 high-VA data pages (the output + * operands live there) with the exact (gpa, ap) -> hpa the + * writeback will use - comparable against the D2H read's + * resolution line (sema-bigcopy) for the same VA. + */ + if (dg->va >= SVC_VAS_FLOOR && + i + 32 >= svc_fwd_req.gr_n_data) + pr_info("svc-fwd-hpa: va=0x%llx gpa=0x%llx ap=%u hpa=0x%llx bi=%u\n", + dg->va, dg->pa, dg->ap, dhpa, i); + /* + * GRVAUDIT (diagnostic only): audit the m180 landing + * authority against the PA this submit will put on the wire + * for every tracked data page - the data[]/n_data set the + * backend's NOWRITE analysis later enumerates. The lookup + * runs BEFORE the store underneath re-records this launch's + * own landing, so it reads the pre-existing authority: + * unlanded = no landing recorded for this VA at submit; + * pte_bad = a landing exists on a different page/aperture + * than the PA the submit path computed/used. Pages below + * SVC_VAS_FLOOR are outside the ledger's domain (all store + * sites gate on the floor) and are not classified. + */ + if (dg->va >= SVC_VAS_FLOOR) { + u64 lpa = 0; + u32 lap = 0; + + if (svc_ledger_lookup(dg->va, &lpa, &lap)) { + ga_unlanded++; + if (!ga_va) { + ga_va = dg->va; + ga_used = dg->pa; + ga_led = 0; + } + } else if ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dg->pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != (u32)dkind) { + ga_ptebad++; + if (!ga_va) { + ga_va = dg->va; + ga_used = dg->pa; + ga_led = lpa; + } + } + } + /* + * r233 READ-DIVERGE-WIT: the engine's read of this VA uses + * exactly what the fill resolved (dg->pa: the page whose + * content was copied into the bounce). The copy paths (H2D + * writes, D2H reads) resolve independently through + * svc_op_translate/ledger. For small launches (the probe + * class) compare the three for the first recs and flag any + * page-level split - the direct witness for a read/write + * permutation. Bounded: first 8 recs, 4 lines per launch, + * boot budget 128. + */ + if (svc_fwd_req.gr_n_data <= 1024u && i < 8u && + svc_readdiv_log < 128u) { + u64 opa = 0, lpa2 = 0; + u32 oap = 0, lap2 = 0; + bool op_miss, have_led; + + op_miss = svc_op_translate(svc_fwd_req.vgpu, dg->va, + &opa, &oap) != 0; + have_led = !svc_ledger_lookup(dg->va, &lpa2, &lap2); + if (op_miss || (!op_miss && + (opa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dg->pa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (have_led && + (lpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (dg->pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + svc_readdiv_log++; + pr_info("vgsp-gr: READ-DIVERGE-WIT va=0x%llx fill_pa=0x%llx op_pa=0x%llx op_src=%u op_rc=%d ledger_pa=0x%llx%s\n", + dg->va, dg->pa, opa, + op_miss ? 0xffffffffu : svc.op_tr_src, + op_miss ? -ENOENT : 0, + have_led ? lpa2 : 0, + op_miss ? " (op-miss)" : ""); + } + } + /* + * C3 (data class): after the run the backend copies this + * page's results back to dg->pa (gsp_phys_booter.c copy-out). + * Record that page as the truth for this VA so the guest's + * D2H read of an engine-authored output follows the page the + * results actually landed on, whatever a later resolver says + * (atomics: engine ran, readback read a different page). + */ + if (dg->va >= SVC_VAS_FLOOR) + svc_ledger_store(dg->va, dg->pa, dkind); gs.n_data++; } gs.data = data; @@ -11939,6 +18213,12 @@ static void svc_fwd_gr_work(void) gs.rel_hpa = rel_hpa; gs.rel_off = (u32)(svc_fwd_req.gr_rel_va & (SEMA_PAGE_SIZE - 1)); gs.rel_payload = svc_fwd_req.gr_rel_payload; + /* + * r378: carry the release VA so the backend can re-validate the + * release-word write targets (zero-on-retry, restore-on-accept) + * through gsp_sema_gr_write_guard before they land. + */ + gs.rel_va = svc_fwd_req.gr_rel_va; gs.sem_hpa = 0; gs.sem_off = 0; gs.sem_payload = 0; @@ -11950,7 +18230,29 @@ static void svc_fwd_gr_work(void) gs.raw_qmd_end = svc_fwd_req.gr_push_qmd_end; gs.h_object = svc_fwd_req.gr_h_object; + /* + * GRVAUDIT: one line per launch (n_data matches the backend + * NOWRITE line's req->n_data); nonzero counts always log, an + * all-clean audit logs once per 50 launches. + */ + svc_grvaudit_launches++; + if (ga_unlanded || ga_ptebad || !(svc_grvaudit_launches % 50u)) + pr_info("vgsp-gr: GRVAUDIT n_data=%u unlanded=%u pte_bad=%u sample_va=0x%llx pa_used=0x%llx ledger_pa=0x%llx\n", + gs.n_data, ga_unlanded, ga_ptebad, + ga_va, ga_used, ga_led); + hw_rc = gsp_phys_backend_guest_gr_submit(&gs, &cp); + /* + * r420b: the backend redirected the launch's QMD release0 to the + * module semaphore (rel_mod=1 or no targetable guest release page), so + * the ENGINE never writes the guest's own QMD release word. Treat the + * word as unobserved so the bounded outcome check (and the hold's + * edges) author it in software on every outcome; without this a + * rel_mod=1 launch with no deferred release records is never completed + * on the guest word and the client hangs (the r414 saxpy stall). + */ + if (cp.qmd_rel_redirected) + svc_fwd_req.gr_rel_unobserved = true; if (!hw_rc) { svc.fwd_gr_ok++; } else { @@ -12221,12 +18523,16 @@ static void svc_fwd_relay_select(struct svc_fwd_req *req) * svc_gr_req_fill from its QMD pointer association and the captured inline * bytes. Overflow is a failure (fail-closed), not a silent drop. */ +static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa_out, u32 *ap_out); static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u64 pa = 0; u32 ap = 0; u32 i; + /* r372 diagnostic: which branch resolved this launch page. */ + const char *src = "?"; if (!page) return; @@ -12245,10 +18551,132 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) i = svc_fwd_req.gr_n_pages++; svc_fwd_req.gr_pages[i].va = page; svc_fwd_req.gr_pages[i].qmd_word = qmd_word; + /* + * m181 (relocation/ledger unification): the unified landing table is + * the authority for a page a host write already landed on. The CE + * inline writes that author these launch pages (param, program, + * constant banks, inline destinations) go through + * svc_grce_inline_write(), which records its landing. The backend's + * QMD relocation reads the page content through pg->pa (win0 shadow + * copy) and the m129 mirror patches that same page, so resolving + * independently here could hand the engine a stale page whose + * pointer values are not this launch's VAs: the relocation then + * matches nothing, the engine executes against unrelocated guest VAs + * it cannot walk, and the grid aborts silently (zero stores, ack + * still delivered) - the cross-session zero-write class - or runs on + * the previous client's parameters (the multi-page wrongness). + * Resolve once, from the table, and record the independent answer as + * stale when it differs. + */ + { + u64 lpa = 0; + u32 lap = 0; + + /* + * r372 pin-wins: a live injected pin is the guest's newest + * statement about this VA. If it disagrees with the landing + * table, the pin wins and the table is repaired (see the + * PIN-REPAIR note in svc_handle_map_add): the old landing was + * a freed page, and resolving the launch there wrote engine + * output onto memory the guest kernel had reused. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(page, &ipa, &iap)) { + u64 lpa2 = 0; + u32 lap2 = 0; + + if (svc_ledger_lookup(page, &lpa2, &lap2) || + ((lpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (ipa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap2 != (u32)iap) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS note va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + page, lpa2, lap2, ipa, iap); + svc_ledger_store(page, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + } + svc_fwd_req.gr_pages[i].pa = + ipa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)iap; + svc_pin_store(page, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + iap); + src = "ipin"; + goto note_done; + } + } + if (!svc_ledger_lookup(page, &lpa, &lap)) { + u64 rpa = 0; + u32 rap = 0; + const char *rsrc = svc_gr_src_probe(vgpu, page, &rpa, + &rap); + + if (rpa && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (rpa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != (u32)rap)) { + svc_ledger_reloc_diverges++; + if (svc_ledger_reloc_diverges <= 16) + pr_info("sema-ledger: RELOC-DIVERGE va=0x%llx ledger_pa=0x%llx %s_pa=0x%llx (ledger wins)\n", + page, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + rsrc, rpa & + ~(u64)(SEMA_PAGE_SIZE - 1)); + } + svc_fwd_req.gr_pages[i].pa = + lpa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)lap; + /* + * Keep the pin table consistent with the authority so + * the win0 copy path and later resolves agree (and no + * stale self-pin is planted for this VA). + */ + svc_pin_store(page, lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)lap); + pa = lpa; + ap = lap; + src = "ledger"; + goto note_done; + } + } if (svc_resolve_va(vgpu, page, &pa, &ap)) return; svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); + /* + * r370: store the landing table for EVERY launch page, not just the + * data-fill spans. An indirect/bank operand (the kernel's output + * buffer, qmd_word NONE) was resolved HERE, the engine wrote it via + * this hpa, the copy-out wrote it back to this hpa - and the client's + * D2H readback then resolved the same VA through the ledger/pin to a + * DIFFERENT page (an older landing), read stale zeros and the kernel + * looked wrong (victory_reduce sum=0 with the correct sum proven in + * the bounce page by OUT-PAGE-WIT). Recording this resolution makes + * the readback read exactly the page the engine wrote. + */ + svc_ledger_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)svc_ap_of(ap)); + svc_pin_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)svc_ap_of(ap)); + src = "walk"; +note_done: + /* + * r372 diagnostic: the exact (pa, ap) and branch the launch page + * resolved through - the value the engine's hpa and the copy-out use. + * A D2H readback that lands elsewhere (the sum=0 class) names its + * disagreement against these lines. + */ + if (svc_gr_pagenote_log < 512) { + svc_gr_pagenote_log++; + pr_info("sema-gr: page-note va=0x%llx pa=0x%llx ap=%u src=%s qmd=%u\n", + page, svc_fwd_req.gr_pages[i].pa, + svc_fwd_req.gr_pages[i].ap, src, qmd_word); + } } /* Does any captured inline write land in this page? */ @@ -12282,35 +18710,65 @@ static bool svc_gr_page_inline_covers(u64 page) * Resolution only happens on the service thread; the work item turns the * guest PAs into HPAs and the backend plants the identity PTEs. */ -#define SVC_GR_DATA_RELOC_BASE 0x110000ULL +/* + * B-2 (large-operand data path): the SVC assigns each resolved operand page + * a reloc VA in the grown boot-planted region (the layout constants live in + * gsp_phys_booter.h so the two sides cannot drift; gsp_phys_gr_data_region + * selects grown vs legacy, default grown). The legacy 0x110000 window below + * the mail page (205 data pages max, 68 per pointer) remains selectable with + * gsp_phys_gr_data_region=0 for A/B trialing. + */ +#define SVC_GR_DATA_RELOC_BASE GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 +#define SVC_GR_DATA_RELOC_BASE_OLD 0x110000ULL /* * Per candidate pointer, map at most this many pages from the pointer's - * page onward (the engine-visible main-PT window has ~220 free 4 KB slots; - * the demo buffers are small on the native path). - * m120: lowered 64 -> 32 so every pointer page fits under the 100-page - * bounce cap (mail 2x32=64, vadd 3x32=96); at 64/ptr the second/third - * candidate run was truncated by the cap and its pointer word never - * rewritten (m119: first LDG base c[0x0][0x168] possibly untouched). - * m140: 32 -> 33. A cudaMalloc'd 128 KB buffer whose pointer is 0x400 - * into its page spans 33 pages (32 full + the 1 KB tail), and the kernel - * accesses the tail: with 32 pages the tail read the NEXT candidate's - * relocated page (wrong values; vadd tails). 3x33=99 <= the 100-page cap. - */ -#define SVC_GR_DATA_PAGES_PER_PTR 33u -/* m114: total pages the pre-commit bounce window can carry (slot 100 is - * the mail message). */ -#define SVC_GR_DATA_BOUNCE_MAX 100u + * page onward. With the grown region this is a runaway guard, not a budget: + * a cold page still stops the run (stop-on-miss below), so a real operand's + * span is bounded by its extent. History: 64 -> 32 (m120) -> 33 (m140) under + * the old 100-page cap -> 68 (m150, 256 KB operands under the 205-page + * window); the old value is kept for the legacy path. + */ +#define SVC_GR_DATA_PAGES_PER_PTR 4096u +#define SVC_GR_DATA_PAGES_PER_PTR_OLD 68u +/* + * Total operand pages the engine-visible data region can carry; derived from + * the booter's shared constant (no drift). A cap hit is FAIL-CLOSED: the + * fill marks the launch stalled for the hold/retry path and returns instead + * of silently truncating later pointers - a truncated operand must never + * reach the engine. The legacy value is the pre-B-2 window below the mail + * page (GSP_PHYS_RMCALL_GR_MAIL_SLOT = 206). + */ +/* + * fail4: each launch may relocate up to half of the grown region - the + * other half is the alternate base of a concurrently in-flight launch + * (see the per-capture base flip in svc_gr_data_fill). + */ +#define SVC_GR_DATA_BOUNCE_MAX (GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / 2u) +#define SVC_GR_DATA_BOUNCE_MAX_OLD 205u static u32 svc_gr_data_log; /* m142: bounded per-page attribution log */ static u32 svc_gr_src_log; /* m150: bounded per-candidate source log */ static bool svc_gr_data_stalled; /* m150: high data args unresolved */ +/* + * m156 (R1): the GR request was refused because a forward was already + * in flight (or a release hold was active) - the launch COULD run later, so + * the entry must be deferred, not consumed with a fabricated software + * release. r157 streams2: kernels 2 and 3 were refused-while-busy (kernel 1 + * in flight) and soft-released 36 us later; the client read all zeros. + */ +static bool svc_gr_busy_defer; static u64 svc_gr_hold_log; /* m150: bounded GR-hold proof lines */ +static u64 svc_gr_drop_log_va; /* GRFIX: launch whose dropped-page detail + * (GRFILL-DROP) was already logged - the + * detail prints once per launch episode, + * not once per ~100 ms re-decode */ /* m150: re-arm both fill diagnostics for each client session. */ static void svc_gr_diag_reset(void) { svc_gr_data_log = 0; svc_gr_src_log = 0; + svc_gr_drop_log_va = 0; } /* @@ -12376,6 +18834,255 @@ static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, } /* + * relfix PART 3: append one attempt to the resolver's diagnostic chain + * ("name:rc,"). Bounded; the tail is simply dropped once full. + */ +static void svc_gr_reloc_stage_note(struct gsp_sema_gr_reloc *out, + const char *name, int rc) +{ + size_t n = strlen(out->stage); + + if (n + 16 >= sizeof(out->stage)) + return; + scnprintf(out->stage + n, sizeof(out->stage) - n, "%s:%d,", name, rc); +} + +/* + * relfix (agent-A FIX A): rewrite-time GR relocation resolver (see + * gsp_sema_service.h). Mirrors the per-page source order of the fill + * (svc_gr_data_fill: svc_ipin_lookup -> svc_resolve_va's trusted chain -> + * ledger) and of the read-only attribution probe svc_gr_src_probe above, + * then applies the m180 ledger in the fill's two roles: fallback when + * nothing above answered (the C1 landing path), and the m171 landing + * preference for VAs at/above the ledger floor when the ledger disagrees + * with the resolver (the skip-ahead class where a walk result is stale but + * the landing is authoritative). + * + * Deliberately excluded versus svc_resolve_va: pin_store/lat_store writes, + * svc_pd_discover and the sliced targeted RAM scan - those mutate + * service-thread state or can block for seconds, and this runs on the GR + * work item (process context; the page-table walks take svc.vas_lock only + * for their publication snapshot and may sleep in vfio_dma_rw, exactly the + * discipline the walk comment above svc_walk_vaspaces mandates). + * + * Every attempt is recorded in out->stage ("name:rc," chain; the resolving + * source ends in ":0,") for the booter's RELOC-GAP-STAGE diagnostic. + * + * Returns 0 with out filled on a successful translation, -ENOENT on a miss + * (caller must leave the word untouched), or a conversion errno. + */ +int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out) +{ + struct nv_vgpu_device *vgpu = svc_fwd_req.vgpu; + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0, lpa = 0, hpa = 0; + int ap = SVC_APERTURE_SYS; + u32 lap = 0, off = 0; + bool resolved = false; + int rc, ret; + + if (!vgpu || !out) + return -EINVAL; + + out->stage[0] = '\0'; + out->stage_rc = 0; + + rc = svc_ipin_lookup(page, &pa, &ap); + svc_gr_reloc_stage_note(out, "ipin", rc); + if (!rc) + resolved = true; + if (!resolved) { + rc = svc_pin_lookup(page, &pa, &ap); + svc_gr_reloc_stage_note(out, "pin", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_lat_lookup(page, &pa, &ap); + svc_gr_reloc_stage_note(out, "lat", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_map_lookup(page, &pa); + svc_gr_reloc_stage_note(out, "map", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_walk_vaspaces(vgpu, page, &pa, &ap); + svc_gr_reloc_stage_note(out, "vas", rc); + if (!rc) + resolved = true; + } + if (!resolved) { + rc = svc_walk_discovered(vgpu, page, &pa, &ap); + svc_gr_reloc_stage_note(out, "disc", rc); + if (!rc) + resolved = true; + } + + if (!resolved) { + /* C1 fallback: a landing is a page the client already backed + * with content even when every resolver missed (r174). */ + rc = svc_ledger_lookup(page, &lpa, &lap); + svc_gr_reloc_stage_note(out, "ledger", rc); + if (rc) { + out->stage_rc = -ENOENT; + return -ENOENT; + } + pa = lpa; + ap = (int)lap; + } else if (page >= SVC_VAS_FLOOR) { + /* m171: the landing table wins over a disagreeing resolver. */ + if (!svc_ledger_lookup(page, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != svc_ap_of(ap))) { + pa = lpa; + ap = (int)lap; + } + } + + ret = svc_fwd_operand_hpa(vgpu, + svc_ap_of(ap) == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + pa, 0, SEMA_PAGE_SIZE, &hpa, &off); + if (ret || off) { + svc_gr_reloc_stage_note(out, "hpa", ret ? ret : -EFAULT); + out->stage_rc = ret ? ret : -EFAULT; + return ret ? ret : -EFAULT; + } + + out->pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + out->ap = svc_ap_of(ap); + out->hpa = hpa; + out->stage_rc = 0; + return 0; +} + +/* + * r378 write guard, GR backend half (see gsp_sema_service.h). The GR submit + * resolves every operand page once and the copy-out/release writes run + * seconds later (the bounded poll, the retire wait, the outcome retries); a + * guest buffer freed and re-homed inside that window would receive a full + * page of another buffer's content through the stale HPA - the guest + * boot-corruption class (slab freelist clobbered by a foreign page write). + * + * Authority order mirrors the fill's own choice (r372/r376 pin-wins): a + * FRESH injected pin is the guest's newest statement, so the guard compares + * the write target against the pin's page first; only a pin-less VA falls + * back to the trusted resolver chain. The landing table is deliberately NOT + * the authority here - our own launches feed it, so it would confirm the + * target by construction. A miss on every source keeps the historical + * behaviour (the recorded landing is the write authority), so the guard can + * only remove provably stale writes. Bounded diagnostic budget. + */ +static u32 svc_wguard_log; /* r378 WGUARD-SKIP budget (256/boot) */ +bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 ipa = 0, cur_hpa = 0; + int iap = SVC_APERTURE_SYS; + u32 off = 0; + int ret; + + if (!va || !hpa) + return true; + if (!svc_ipin_lookup(page, &ipa, &iap)) { + u32 kind = svc_ap_of(iap) == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS; + + ret = svc_fwd_operand_hpa(svc_fwd_req.vgpu, kind, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + 0, SEMA_PAGE_SIZE, &cur_hpa, &off); + if (ret || off) + return true; /* pin page unresolvable: keep */ + } else { + struct gsp_sema_gr_reloc r; + + if (gsp_sema_gr_reloc_resolve(va, &r)) + return true; /* no current answer: keep */ + cur_hpa = r.hpa; + } + if ((cur_hpa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (hpa & ~(u64)(SEMA_PAGE_SIZE - 1))) + return true; + if (svc_wguard_log < 256) { + svc_wguard_log++; + pr_warn("vgsp-sema: WGUARD-SKIP path=%s va=0x%llx used=0x%llx now=0x%llx (stale write target)\n", + path, va, hpa, cur_hpa); + } + return false; +} + +/* + * r378 write guard, service half (see gsp_sema_service.h). A fresh injected + * pin is the guest's newest statement about a VA; a pin that disagrees with + * the recorded write target means the page was freed/re-homed (the r372/r376 + * pin-wins rule applied at the write instead of only at the resolution). A + * missing pin keeps the historical behaviour - the recorded landing is then + * the write authority - so no green path loses its write. + */ +bool gsp_sema_write_guard(u64 va, u64 pa, u32 ap, const char *path) +{ + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!va || !pa || va < SVC_VAS_FLOOR) + return true; + if (svc_ipin_lookup(va, &ipa, &iap)) + return true; /* no fresh pin: nothing newer to obey */ + if ((ipa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + svc_ap_of(iap) == svc_ap_of((int)ap)) + return true; + if (svc_wguard_log < 256) { + svc_wguard_log++; + pr_warn("vgsp-sema: WGUARD-SKIP path=%s va=0x%llx used=0x%llx/%u pin=0x%llx/%d (stale write target)\n", + path, va, pa, ap, ipa, iap); + } + return false; +} + +/* + * r391 diagnostic (sum=0 class): the landing-table resolution of a GR data VA + * plus the first 8 bytes read through it - the exact page the D2H reader's + * ledger-first branch will use. Printed by the GR copy-out next to the write + * target, so a launch/readback disagreement is visible in one line. + */ +bool gsp_sema_gr_va_snap(u64 va, u64 *pa, u32 *ap, u64 *val8) +{ + u64 lpa = 0; + u32 lap = 0; + + if (svc_ledger_lookup(va, &lpa, &lap)) + return false; + *pa = lpa & ~(u64)(SEMA_PAGE_SIZE - 1); + *ap = lap; + *val8 = 0; + if (!svc_fwd_req.vgpu) + return true; + if (lap == SVC_APERTURE_FB) { + u8 *base = svc_fwd_req.vgpu->bars[1].vaddr; + size_t size = svc_fb_shadow_size(svc_fwd_req.vgpu); + + if (base && *pa + 8 <= size) + memcpy(val8, base + *pa, 8); + } else { + svc_read(svc_fwd_req.vgpu, *pa, val8, 8); + } + return true; +} + +/* r391 diagnostic: the shadow base this service resolves through (compare + * against the CE copy path's own vgpu->bars[1].vaddr in the same run). */ +u8 *gsp_sema_gr_shadow_base(void) +{ + return svc_fwd_req.vgpu ? svc_fwd_req.vgpu->bars[1].vaddr : NULL; +} + +/* * m144 (second-client launch-state divergence): collect device-VA-shaped * candidate pointers from the bytes the ENGINE will actually execute. * @@ -12407,7 +19114,7 @@ static void svc_gr_scan_cand(u64 *wins, u32 *n_cand, const u8 *data, u32 bytes) for (w = 0; w < *n_cand; w++) if (wins[w] == wv) break; - if (w == *n_cand && *n_cand < 16u) + if (w == *n_cand && *n_cand < SVC_GR_CAND_MAX) wins[(*n_cand)++] = wv; } } @@ -12425,14 +19132,171 @@ static void svc_gr_scan_inline(u64 *wins, u32 *n_cand, u64 page_va) } } -static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) +/* + * STALLFREE: identity of the launch currently held/retried on this channel - + * the capture's push VA when it belongs to this channel (the busy-refusal + * path never reaches the fill, so the request slot can still hold a previous + * launch's VA), else the request slot's when it names this channel. The + * absolute hold bound is keyed to this identity so a pin-batch rewind that + * replays the entry at a different slot index can never re-arm it. + */ +static u64 svc_gr_launch_va(const struct svc_channel *ch) +{ + if (svc_gr_cand.valid && svc_gr_cand.h_object == ch->h_object) + return svc_gr_cand.push_va; + if (svc_fwd_req.gr && svc_fwd_req.gr_h_object == ch->h_object) + return svc_fwd_req.gr_push_va; + return 0; +} + +/* + * GRFIX FIX1b (r213): absolute per-launch hold bound. The m150/m156 data + * hold had no working bound: the 8b058884 deadline was keyed by the GPFIFO + * slot index, and a pin-batch rewind/replay re-decodes the same launch push + * at different slots (r213: entry 107 -> 107 -> 99 on one channel, other + * passes at yet other indices), so "first arm" re-armed on every pass; the + * launch held for 216+ s (410+ GRFILL-HOLD lines) until the client's 240 s + * timeout killed it (gr_hold_exp=0). The identity is now the launch push VA: + * the first hold of a VA arms an ABSOLUTE deadline that nothing moves while + * the launch is held. Returns true when the bound has passed (caller must + * let the launch proceed); false leaves the record armed. + */ +static bool svc_gr_hold_bound_expired(struct svc_channel *ch, u64 launch_va) +{ + u32 cap_ms = gsp_phys_gr_hold_max_ms; + + /* + * STALLFREE (r213-r215 follow-up): the absolute per-launch bound is + * now effective on the default path. Before the write-back fix the + * bound's record fields were excluded from the pump snapshot and + * zeroed every pass, so this returned false forever and the legacy + * m150/m156 hold window (gr_hold_until) was the only expiry - and + * that window is keyed to the GPFIFO slot index, which a pin-batch + * rewind changes on every replay (the r213 identity bug). Under pin + * churn the same launch was therefore re-held indefinitely: r220 + * held one launch from 155221 to 155417 (196 s, 1631 re-fills of + * entry 106) until the guest's 240 s kill. The bound is keyed to the + * LAUNCH PUSH VA and nothing re-arms it, so a replayed slot no + * longer extends the hold; at the cap the pump falls through to the + * historical consume, where STALLFREE authors the QMD completion + * release and logs GR-STALL-hold action=proceed. Default cap 3000 ms + * (gsp_phys_gr_hold_max_ms) sits above the observed 0.5-2 s pin + * latency, so the legitimate pin-late retry still wins. + */ + if (!launch_va) + launch_va = 1; /* anonymous launch: still absolutely bound */ + if (!cap_ms) + cap_ms = 3000; + /* + * r368: mirror of the opdef fix - an ACTIVE bound is never re-armed + * by a different launch VA. The pump can hold several launches whose + * push VAs differ; arming on every VA change moved the deadline + * forever (the same identity bug class as the opdef=955/13/0 + * streams2 wedge). The first hold arms; others ride it; a fresh + * bound arms once the previous one has passed. + * + * r509: the bound must actually REPORT expiry. The historical arming + * rule re-armed the deadline on the very call that detected expiry + * and then compared against the NEW deadline - so the function could + * never return true, and a data-stalled launch whose arguments never + * resolved held its entry forever (r508 witness: matmul plus ten + * sibling channels in "GR hold ... data args unresolved" with zero + * GR-STALL-hold-bound lines for 10+ minutes; the designed + * retry-forward + consume never ran). The expiry is now one-shot: + * push the deadline, latch gr_hold_bound_hit for the r351 block of + * the same pass, return true. A fresh arm clears the latch (and the + * once-per-launch retry log). + */ + if (!ch->gr_hold_dead_va || + time_after_eq(jiffies, ch->gr_hold_deadline)) { + ch->gr_hold_dead_va = launch_va; + ch->gr_hold_dead_since = jiffies; + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + ch->gr_hold_exp_logged = false; + ch->gr_hold_bound_hit = false; + return false; + } + /* + * r420: pause the per-launch consumption bound while the backend + * deliberately holds the in-flight launch in its bounded WEDGE-QUIET + * window. At the bound the pump falls through to the historical + * consume, which authors the capture's QMD completion release in + * software (GR-STALL-hold) - but the backend is still going to + * submit/accept this very launch after the hold and copy its data + * out; authoring now releases the client onto stale data (the r419 + * mid-boot escape). The quiet hold is a bounded msleep (the + * backend's own caps), and the deferral here is bounded by + * SVC_GR_QUIET_PAUSE_MAX_MS from the bound's arm time as + * defence-in-depth. Rate-limited announcement. + */ + if (ch->gr_hold_dead_va && gsp_phys_gr_quiet_hold_active() && + time_before(jiffies, ch->gr_hold_dead_since + + msecs_to_jiffies(SVC_GR_QUIET_PAUSE_MAX_MS))) { + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + pr_info_ratelimited("vgsp-gr: GR-HOLD-BOUND-PAUSE launch=0x%llx (backend wedge-quiet hold active; consumption bound deferred)\n", + launch_va); + return false; + } + if (time_after_eq(jiffies, ch->gr_hold_deadline)) { + /* + * r509: one-shot expiry. Push the deadline so the hold window + * restarts for the caller's retry, latch the verdict for the + * second call site of this pass (the r351 retry-forward + * block), and report expired. + */ + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + ch->gr_hold_bound_hit = true; + return true; + } + return false; +} + +static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) { - u64 wins[16]; - bool is_arg[16]; /* m150: candidate came from the argument scan */ - u32 n_win = 0, n_cand; + u64 wins[SVC_GR_CAND_MAX]; + bool is_arg[SVC_GR_CAND_MAX]; /* m150: candidate came from the arg scan */ + u32 n_cand; u32 off = 0, i, w; u8 kind; /* + * B-2: active data-region geometry. gsp_phys_gr_data_region=1 (the + * default) selects the grown 16384-page region planted by the booter + * at GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0; 0 keeps the legacy 205-page + * window below the mail page for A/B trialing. + */ + bool region_on = gsp_phys_gr_data_region_value() != 0; + /* + * fail4: two concurrent forwarded launches (streams2's vadd pair) + * both relocated their operands from the same base, so the second + * launch's copy-in overwrote the first's bounce windows and each + * engine read the other's inputs (c1 came out as a2+b2). Alternate + * the per-launch base across the grown region's two halves (32 MB + * each, above any single launch's span), keyed to the capture push + * so a retried launch keeps its half, so concurrent launches + * cannot collide. + */ + static u64 svc_gr_reloc_half_key; + static u8 svc_gr_reloc_half; + u64 reloc_off = 0; + + if (region_on) { + if (svc_gr_cand.push_va != svc_gr_reloc_half_key) { + svc_gr_reloc_half_key = svc_gr_cand.push_va; + svc_gr_reloc_half ^= 1; + } + if (svc_gr_reloc_half) + reloc_off = (u64)(GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / + 2u) * SEMA_PAGE_SIZE; + } + u64 reloc_base = region_on ? + SVC_GR_DATA_RELOC_BASE + reloc_off : + SVC_GR_DATA_RELOC_BASE_OLD; + u32 per_ptr = region_on ? SVC_GR_DATA_PAGES_PER_PTR : + SVC_GR_DATA_PAGES_PER_PTR_OLD; + u32 bounce_max = region_on ? SVC_GR_DATA_BOUNCE_MAX : + SVC_GR_DATA_BOUNCE_MAX_OLD; + /* * m150: high (user/device-space) argument candidates and how many of * them produced at least one resolved page. None resolving is the * second-client zeros signature: the engine would run the kernel @@ -12440,8 +19304,35 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * retry instead of forwarded (svc_gr_data_stalled). */ u32 n_arg_hi = 0, n_arg_hi_res = 0; + /* + * GRFIX FIX1 (full-page fail-closed): every page of every tracked + * span whose resolution chain (ipin > resolve > ledger) misses is + * counted here, first page or mid-span alike. Any drop holds the + * launch through the same bounded m150/m156 retry path as the + * first-page guard below, so a QMD is never forwarded with a data + * pointer left as an unresolvable guest VA (the zero-store class) + * and the bounce window is never served stale for this launch. + * FIX1b: the hold carries an absolute per-launch bound + * (gsp_phys_gr_hold_max_ms) - when it passes the launch proceeds. + */ + u32 dropped_pages = 0; + bool bound_hit = false; /* GRFIX FIX1c: the absolute bound fired this + * pass - the launch proceeds, no hold */ + /* + * GRFIX FIX1c (r213): detail for the first (up to 4) dropped pages, + * so the cycle records WHICH VAs and which candidate class dropped. + * There is no read/write direction anywhere in the candidate or + * data[] structures (both are va/pa/reloc_va only), so kind is the + * truthful candidate origin: 'a' = argument page (c[0x0] param / + * inline), 'b' = const-bank page; idx is the candidate index (the + * same index the per-candidate probe lines use). + */ + u64 drop_va[4]; + char drop_kind[4]; + u32 drop_idx[4]; svc_fwd_req.gr_n_data = 0; + svc_fwd_req.gr_dropped = 0; svc_fwd_req.gr_n_data_wins = 0; svc_gr_data_stalled = false; memset(is_arg, 0, sizeof(is_arg)); @@ -12502,6 +19393,52 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * actually dereferences. */ for (w = 0; w < n_cand; w++) is_arg[w] = true; + /* + * relfix PART 3 (fill candidate superset): the rewrite scans the WHOLE + * authored param page (0x0..0xfff) and re-resolves every + * device-VA-shaped word it finds; any word this fill does not collect + * resurfaces as a RELOC-GAP (r217: the two words the late path fixed + * were param/program descriptor fields below 0x150 / past 0x800 that + * the argument scan never looked at). Collect the page remainder here, + * AFTER the argument marking, so these page words keep is_arg=false - + * the per-argument hold and diagnostic semantics stay exactly as + * before. Budget priority is also unchanged: the argument candidates + * above are resolved first. Launch-internal pages remain filtered at + * resolution time (R3 hygiene below). + */ + for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { + const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; + u64 hpa = 0; + const u8 *page; + u32 k; + + if (pp->qmd_word != 32u || !pp->pa) + continue; + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + for (k = 0; k < SEMA_PAGE_SIZE / 8; k++) { + u64 v, wv; + + if (k >= 0x150u / 8u && k < 256u) + continue; /* arg region: scanned above */ + v = get_unaligned_le64(page + k * 8); + if (v < 0x10000000ULL || v >= (1ULL << 47) || + (v & 3)) + continue; + wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); + for (w = 0; w < n_cand; w++) + if (wins[w] == wv) + break; + if (w == n_cand && n_cand < ARRAY_SIZE(wins)) + wins[n_cand++] = wv; + } + } for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; @@ -12519,7 +19456,12 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) page = (const u8 *)__va(hpa); if (!page) continue; - for (k = 0; k < 256; k++) { + /* + * relfix PART 3: whole page, matching the rewrite's scan + * range (it walks 0x0..0xfff qword-pairs); the old 0..0x7ff + * bound left the second half as permanent RELOC-GAPs. + */ + for (k = 0; k < SEMA_PAGE_SIZE / 8; k++) { u64 v = get_unaligned_le64(page + k * 8); u64 wv; @@ -12537,17 +19479,69 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) * inline-captured bytes as well (see svc_gr_scan_inline). */ svc_gr_scan_inline(wins, &n_cand, pp->va); } - /* m110: one contiguous run of pages per candidate pointer, relocated - * into the engine-walkable main-PT window (0x110000 onward, fresh - * PTE lines; the commit-time snapshot only preserves that window). */ + /* m110/B-2: one contiguous run of pages per candidate pointer, + * relocated into the active engine-visible data region (grown region + * 0x0a000000+ with gsp_phys_gr_data_region=1 - the default - or the + * legacy 0x110000 window; both are boot-planted, never-rewritten PTE + * ranges the commit-time snapshot preserves). */ for (w = 0; w < n_cand; w++) { u64 base = wins[w]; u32 p; u32 before = svc_fwd_req.gr_n_data; + bool first_unresolved = false; /* m170: span's first page failed */ u64 pba = 0; u32 pba_ap = 0; + u32 span; const char *pba_src = "?"; + /* + * Window budget discipline (large buffers): only real device + * operands (high, cudaMalloc-shaped VAs) get the full per-pointer + * span. Low-VA candidates (the launch's own QMD/release/reloc + * pointers) need a single page at most, and the launch's QMD VA + * is not a kernel operand at all - relocating a full span for it + * starved the operands that follow and blew the old window cap + * (r141: a 256 KB operand resolved 0 pages because the QMD and + * param candidates consumed the window). + */ + if ((base & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (svc_fwd_req.gr_qmd_dev_va & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + /* + * R3 hygiene (large-buffer budget discipline): the launch's own + * param/bank pages (the qmd_word 32..47 gr_pages entries) and + * the param VA are driver bookkeeping, not kernel operands - the + * observed fills resolved a full 68-page span for the param page + * in every launch and starved the real operands under the old + * window. Skip them as data candidates; they are + * shadowed/rewritten through gr_pages/Tier 2, not this path. + * P6: the QMD program pair (word 48/49) is launch-internal in + * exactly the same way and is skipped with them. + */ + if (svc_fwd_req.gr_param_va && + (base & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (svc_fwd_req.gr_param_va & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + { + bool launch_internal = false; + + for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { + const struct svc_gr_page *pp = + &svc_fwd_req.gr_pages[i]; + + if (pp->qmd_word >= 32u && + pp->qmd_word <= 49u && + (base & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (pp->va & ~(u64)(SEMA_PAGE_SIZE - 1))) { + launch_internal = true; + break; + } + } + if (launch_internal) + continue; + } + span = (base >= 0x40000000000ULL) ? per_ptr : 1u; + if (is_arg[w] && base >= 0x40000000000ULL) n_arg_hi++; @@ -12562,7 +19556,7 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) svc_pd_found, svc.ipin_n); } - for (p = 0; p < SVC_GR_DATA_PAGES_PER_PTR; p++) { + for (p = 0; p < span; p++) { u64 va = wins[w] + (u64)p * SEMA_PAGE_SIZE; u64 pa = 0; u32 ap = 0, x; @@ -12592,29 +19586,191 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) pa = ipa; ap = (u32)iap; } else if (svc_resolve_va(vgpu, va, &pa, &ap)) { - break; + /* + * C1 (stall/data class): the page may + * still be readable - the ledger holds + * the landing page of the H2D that + * wrote it even when every resolver + * misses (r174 cand[] probe=none, 0 + * pages, bad=65536). Use it before + * declaring the page dead; only a page + * with no ledger entry can truncate + * the span. + */ + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(va, &lpa, &lap)) { + pa = lpa; + ap = lap; + } else { + /* + * m170 (R4a): the very first + * page of this span failed to + * resolve - the fail-closed + * trigger below distinguishes + * it from the benign "0 new + * pages" of a span whose + * pages were already mapped. + * + * GRFIX FIX1: every failed + * page counts (first or mid- + * span) for the launch-level + * hold; the span still stops + * here exactly as before. + * FIX1c: capture the first 4 + * (VA + candidate origin + + * candidate index) for the + * GRFILL-DROP detail. + */ + if (p == 0 && + svc_fwd_req.gr_n_data == before) + first_unresolved = true; + /* + * COPYAUDIT (2026-10-18): a + * span cut after at least one + * authored page is the + * completeness hazard - the + * rewritten base pointer is + * still handed to the engine, + * whose base+offset reads then + * run past the authored pages + * into later operands' bounce + * pages. Make the truncation + * loud (want is a lower bound: + * the guest asked for >= p+1 + * pages; done is the authored + * page count before the cut). + */ + if (p > 0) + pr_info_ratelimited("vgsp: COPY-TRUNC va=0x%llx want>=0x%llx done=0x%llx path=gr-fill-span\n", + wins[w], + ((u64)p + 1) << 12, + (u64)p << 12); + /* + * r377 pin-demand: name every + * unresolved operand page as a + * stalled VA. The userspace pin + * stream watches dmesg for this + * exact line and pushes the VA + * to the in-guest scanner's + * demand file, so the page gets + * pinned and the bounded launch + * hold (gsp_phys_gr_fill_hold) + * resumes on real data instead + * of waiting out its bound (the + * r345 wedge: held forever + * because nobody demanded the + * page). + */ + svc_log_stalled_va(va); + if (dropped_pages < 4) { + drop_va[dropped_pages] = va; + drop_kind[dropped_pages] = + is_arg[w] ? 'a' : 'b'; + drop_idx[dropped_pages] = w; + } + dropped_pages++; + break; + } } } - if (svc_fwd_req.gr_n_data >= SVC_GR_DATA_BOUNCE_MAX) { - pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved (reloc 0x%llx+) CAP-HIT\n", + /* + * m171 (AGENT-F R2): prefer the write/read ledger's + * landing page for this operand VA when it disagrees + * with the resolution - the kernel then reads exactly + * the page a large H2D wrote (r174: write landed on + * pa X while the fill read a stale page Y). Compare + * aperture-aware (a SYS page and an FB page with the + * same numeric address are not the same page). + */ + if (va >= SVC_VAS_FLOOR) { + u64 lpa = 0; + u32 lap = 0; + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + /* + * r372 pin-wins: when a live pin exists it is the + * newest truth; the landing table may name a page + * the guest has since freed (VA reuse across + * clients). Keep the pin and repair the table. + * Only a pin-less VA still follows the table. + */ + if (!svc_ipin_lookup(va, &ipa, &iap)) { + pa = ipa; + ap = (u32)iap; + if (!svc_ledger_lookup(va, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (ipa & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)iap)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-WINS fill va=0x%llx ledger=0x%llx/%u pin=0x%llx/%d\n", + va, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, ipa, iap); + svc_ledger_store(va, + ipa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)iap); + } + } else if (!svc_ledger_lookup(va, &lpa, &lap) && + ((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) || + (u32)lap != (u32)ap)) { + svc_ledger_diverges++; + if (svc_ledger_diverges <= 32) + pr_info("sema-ledger: DIVERGE fill va=0x%llx wrote=0x%llx read=0x%llx\n", + va, lpa, pa); + pa = lpa; + ap = lap; + } + } + if (svc_fwd_req.gr_n_data >= bounce_max) { + /* + * B-2 fail-closed cap: the active region is + * full and later pointers would be silently + * dropped (the old CAP-HIT return forwarded + * a truncated operand set). Mark the launch + * stalled so svc_gr_request holds/retries it + * via the existing bounded hold instead of + * forwarding incomplete data. + */ + svc_gr_data_stalled = true; + pr_err("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved (reloc 0x%llx+) CAP-HIT: region full, launch held fail-closed (no silent truncation)\n", n_cand, svc_fwd_req.gr_n_data, - (u64)SVC_GR_DATA_RELOC_BASE); + reloc_base); return; } rec = &svc_fwd_req.gr_data[svc_fwd_req.gr_n_data]; rec->va = va; rec->pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); - rec->reloc_va = SVC_GR_DATA_RELOC_BASE + + rec->reloc_va = reloc_base + ((u64)svc_fwd_req.gr_n_data << 12); rec->ap = (u8)svc_ap_of(ap); svc_fwd_req.gr_n_data++; /* + * r349: record the launch's own resolution as the + * ledger's landing authority for this VA. The read + * path's m171 read-after-write rule follows the + * ledger when it disagrees with the live translation, + * and the engine writeback below writes exactly this + * (pa, ap). Without the store the ledger kept a stale + * page for engine-written outputs (victory_reduce: + * the launch wrote FB 0x7c00000 while the ledger + * said SYS 0x7c00000; the D2H read followed the + * ledger and returned zeros while the correct sum + * sat on the written page). + */ + if (va >= SVC_VAS_FLOOR) + svc_ledger_store(va, rec->pa, rec->ap); + /* * m142: bounded per-page attribution log. A wrong data * page is invisible in the aggregate counters; these * lines let a post-mortem compare the chosen PA against * the guest's own mapping. */ - if (svc_gr_data_log < 16) { + if (svc_gr_data_log < 192) { svc_gr_data_log++; pr_info("vgsp-sema: svc-gr data page va=0x%llx -> pa=0x%llx ap=%u\n", va, rec->pa, rec->ap); @@ -12627,6 +19783,24 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) pr_info("vgsp-sema: svc-gr candidate[%u] base=0x%llx -> %u new pages (total %u)\n", w, base, svc_fwd_req.gr_n_data - before, svc_fwd_req.gr_n_data); + if (is_arg[w] && svc_fwd_req.gr_n_data == before && + first_unresolved) { + /* + * m170 (AGENT-F R4a): a high argument pointer whose + * FIRST page resolved nothing at fill time is the + * partial-operand signature - the launch would forward + * with stale/zero data for that operand (r174: matmul + * cand[2]/[3] probe=none, 0 pages, bad=65536; the + * other operands resolved via pins). Fail closed: mark + * the launch stalled so the m150 hold keeps the entry + * and retries the fill while the pins land (bounded, + * then forwards anyway - never a hang, never silent + * truncation). + */ + svc_gr_data_stalled = true; + pr_err_ratelimited("vgsp-sema: svc-gr data fill: arg base=0x%llx resolved 0 pages; launch held fail-closed\n", + base); + } /* * m150: an argument pointer that resolves zero pages is the * second-client zeros signature; record the walk result per @@ -12651,7 +19825,7 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) wrc = svc_gmmu_walk(vgpu, svc.vas[vi].pdb, svc.vas[vi].entries, base, svc.vas[vi].ap, &wa, &wap, - false); + gsp_sema_walk_trace != 0); o += snprintf(vb + o, sizeof(vb) - o, " vas[%d] pdb=0x%llx ent=%u ap=%d rc=%d pa=0x%llx", vi, svc.vas[vi].pdb, @@ -12663,14 +19837,104 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu) svc.ipin_n, svc_pd_base, svc_pd_ap, vb); } } - svc_gr_data_stalled = (n_arg_hi > 0 && n_arg_hi_res == 0); + /* + * m170 revert: the per-argument fail-closed signal is kept as + * DIAGNOSTIC only (the held fail-closed line above identifies the + * exact unresolved operand); the historical aggregate rule held only + * a launch with NO resolvable high argument, because a fresh client's + * launch stream carries BENIGN 0-page candidates (junk pointer words + * that never translate) and persisting the per-arg flag held mail's + * first launch forever (r176). + * + * GRFIX FIX1: the stall is now page-count based - ANY page in a + * tracked span that failed resolution (or any tracked data VA whose + * resolution returned nothing) holds the launch, so a QMD can never + * be forwarded with a data pointer left as an unresolvable guest VA + * and no bounce page is ever served stale for this launch. The + * response is the unchanged bounded m150/m156 hold (leave the GPFIFO + * entry unconsumed, retry the fill while the pins land); the pump + * side now carries a hard wall-clock bound (gr_hold_deadline) so a + * word that never resolves can no longer re-arm the window forever: + * at the bound the entry is consumed with the historical software + * release - bounded, never a hang, never a silent stale-operand + * submission. + */ + svc_gr_data_stalled = (dropped_pages > 0) || + (n_arg_hi > 0 && n_arg_hi_res == 0); + svc_fwd_req.gr_dropped = dropped_pages; + /* + * GRFIX FIX1e (r215): gsp_phys_gr_fill_hold is the master gate. + * Default 0 = LOG-ONLY: the fill NEVER holds on dropped pages - the + * stalled verdict keeps only the historical pre-8b058884 aggregate + * rule (all high argument pointers unresolved) and the HOLD/EXPIRED + * machinery below is skipped entirely; the launch takes the forward + * path immediately. When 1 the page-count hold applies, bounded by + * the per-launch absolute deadline below. + * + * GRFIX FIX1c/FIX1d (live only with fill_hold=1): the decision point + * is self-enforcing - the absolute per-launch bound is consulted at + * the only place the stalled verdict is produced, before the + * GRFILL-HOLD line can be emitted. When the bound passes the launch + * PROCEEDS via the historical forward path. The benign class + * (probe=none / walk rc=-2) is indistinguishable at fill time from a + * page whose pin has not landed yet, so a "never hold" predicate + * would also disable the pin-late retry; the bound is the only + * guard. NOTE (r213-r215): the bound never fired in the live cycles + * because these per-channel record fields are not in the + * svc_pump_channels snapshot write-back list (see the GRFIX NOTE + * there) and were zeroed on every pass - which is why the gate now + * defaults to 0. + */ + if (!gsp_phys_gr_fill_hold) { + svc_gr_data_stalled = (n_arg_hi > 0 && n_arg_hi_res == 0); + } else if (svc_gr_data_stalled && ch && + svc_gr_hold_bound_expired(ch, svc_fwd_req.gr_push_va)) { + svc.fwd_gr_hold_exp++; + bound_hit = true; + if (!ch->gr_hold_exp_logged) { + ch->gr_hold_exp_logged = true; + pr_info("vgsp-gr: GRFILL-EXPIRED launch=0x%llx dropped=%u waited_ms=%u\n", + svc_fwd_req.gr_push_va, dropped_pages, + jiffies_to_msecs(jiffies - + ch->gr_hold_dead_since)); + } + svc_gr_data_stalled = false; /* proceed: forward */ + } + if (dropped_pages > 0) { + if (gsp_phys_gr_fill_hold && !bound_hit) + pr_info_ratelimited("vgsp-gr: GRFILL-HOLD launch=0x%llx dropped=%u kept=%u\n", + svc_fwd_req.gr_push_va, dropped_pages, + svc_fwd_req.gr_n_data); + /* + * GRFIX FIX1c: name the dropped pages once per launch + * episode (the fill re-runs every pass for a held launch; + * the marker makes the detail a first-pass-only record). + * kind 'a' = argument-page candidate, 'b' = bank candidate; + * idx = candidate index (same indexing as the probe lines). + */ + if (svc_gr_drop_log_va != svc_fwd_req.gr_push_va) { + u32 dl; + + svc_gr_drop_log_va = svc_fwd_req.gr_push_va; + for (dl = 0; dl < 4 && dl < dropped_pages; dl++) + pr_info("vgsp-gr: GRFILL-DROP va=0x%llx kind=%c idx=%u\n", + drop_va[dl], drop_kind[dl], + drop_idx[dl]); + } + } if (svc_gr_data_stalled) - pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u high argument ptr(s), 0 resolved -> launch held for a bounded retry\n", - n_cand, svc_fwd_req.gr_n_data, n_arg_hi); + pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u high argument ptr(s), %u resolved; %u dropped page(s) -> launch held for a bounded retry\n", + n_cand, svc_fwd_req.gr_n_data, n_arg_hi, n_arg_hi_res, + dropped_pages); + else if (bound_hit) + pr_info_ratelimited("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u dropped page(s); hold bound expired -> launch proceeds\n", + n_cand, svc_fwd_req.gr_n_data, dropped_pages); + else if (dropped_pages > 0 && !gsp_phys_gr_fill_hold) + pr_info_ratelimited("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved; %u dropped page(s); fill hold disabled (log-only) -> launch proceeds\n", + n_cand, svc_fwd_req.gr_n_data, dropped_pages); else pr_info("vgsp-sema: svc-gr data fill: %u candidate ptrs, %u pages resolved (reloc 0x%llx+)\n", - n_cand, svc_fwd_req.gr_n_data, - (u64)SVC_GR_DATA_RELOC_BASE); + n_cand, svc_fwd_req.gr_n_data, reloc_base); } static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, @@ -12684,6 +19948,11 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.guest = false; svc_fwd_req.gr = true; svc_fwd_req.gr_bridge_done = false; + /* STALLFREE: fresh request - re-arm the software-completion latch and + * stamp the fill time for the GR-STALL waited_ms accounting. */ + svc_fwd_req.gr_swrel_done = false; + svc_fwd_req.gr_rel_unobserved = false; + svc_fwd_req.gr_fill_jiffies = jiffies; svc_fwd_req.defer = true; svc_fwd_req.relay = false; svc_fwd_req.relay_index = 0; @@ -12786,7 +20055,14 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, if (!svc_gr_page_inline_covers(pg->va)) return -ENOENT; } - svc_gr_data_fill(vgpu); + /* + * GRFIX FIX1c: the fill hosts the self-enforcing decision point and + * therefore needs the channel's per-launch bound record. The const + * view exists for the read-only gates; the channel is the pump's + * mutable live bookkeeping object and this runs on the same + * service thread. + */ + svc_gr_data_fill(vgpu, (struct svc_channel *)ch); return 0; } @@ -12830,12 +20106,30 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, { int rc; + /* m156: never let a previous call's busy verdict leak into an early + * (candidate-mismatch) return; only the busy paths below set it. */ + svc_gr_busy_defer = false; /* * The candidate is per-channel: a capture left unconsumed by an * earlier pass (e.g. the entry was held before its request) must never * be queued against a different channel's handle by that channel's - * own end-of-pass request call. + * own end-of-pass request call. fail4: with several reserved ring + * slots, select THIS channel's oldest unconsumed candidate first. */ + if (gsp_phys_gr_forward_enabled()) { + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % SVC_GR_CAND_RING; + + if (svc_gr_cand_ring[rp].valid && + !svc_gr_cand_ring[rp].offered && + svc_gr_cand_ring[rp].h_object == ch->h_object) { + svc_gr_cand_pos = rp; + break; + } + } + } if (!gsp_phys_gr_forward_enabled() || !svc_gr_cand.valid || svc_gr_cand.h_object != ch->h_object) return false; @@ -12854,6 +20148,17 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, if (svc_fwd_hold.active) { svc.fwd_gr_refused++; svc.fwd_gr_fallback++; + /* + * m159: with the bridge disabled the capture is not consumed: + * the candidate survives for the caller's bounded + * wait+retry (the launch can still run once the active + * forward resolves). With the bridge enabled the fallback + * owns the work and consumes the candidate as before. + */ + if (!gsp_cuda_bridge_enabled()) { + svc_gr_busy_defer = true; + return false; + } svc_gr_bridge_fallback(vgpu); return false; } @@ -12863,6 +20168,10 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, mutex_unlock(&svc_fwd_lock); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; + if (!gsp_cuda_bridge_enabled()) { + svc_gr_busy_defer = true; + return false; + } svc_gr_bridge_fallback(vgpu); return false; } @@ -12898,6 +20207,48 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * the channel's gr_hold window at the call site. Once the window * expires the entry is consumed exactly as before. */ + if (svc_gr_data_stalled && gsp_phys_gr_fill_hold) { + u64 lva = svc_fwd_req.gr_push_va; + /* + * The channel is the pump's mutable live bookkeeping object + * (the const view exists for the read-only gates); these + * record fields are the same ones the pump's hold writes, + * on this same service thread. + */ + struct svc_channel *mch = (struct svc_channel *)ch; + + /* + * GRFIX FIX1b (r213): the ABSOLUTE per-launch bound. The + * first hold of this push VA arms a fixed deadline + * (svc_gr_hold_bound_expired) that nothing re-arms; once it + * passes, the launch PROCEEDS exactly as the pre-8b058884 + * path - the candidate is consumed below and forwarded + * best-effort - so the guest is never blocked by a page that + * never resolves. r213: a slot-keyed deadline was re-armed + * on every re-decode (the same launch is replayed at + * different GPFIFO slots after rewinds) and held one launch + * for 216+ s until the client's 240 s timeout killed it. + * GRFIX FIX1e: gated by gsp_phys_gr_fill_hold (default 0) - + * with the gate off the fill never stalls on drops, so this + * block only ever sees the historical aggregate stall. + */ + if (svc_gr_hold_bound_expired(mch, lva)) { + svc.fwd_gr_hold_exp++; + if (!mch->gr_hold_exp_logged) { + mch->gr_hold_exp_logged = true; + pr_info("vgsp-gr: GRFILL-EXPIRED launch=0x%llx dropped=%u waited_ms=%u\n", + lva, svc_fwd_req.gr_dropped, + jiffies_to_msecs(jiffies - + mch->gr_hold_dead_since)); + } + svc_gr_data_stalled = false; /* proceed: forward */ + if (svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-gr: GRFILL-PROCEED launch=0x%llx (bound expired; forwarding best-effort)\n", + lva); + } + } + } if (svc_gr_data_stalled) { clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); mutex_unlock(&svc_fwd_lock); @@ -13283,7 +20634,7 @@ static bool svc_push_read_displaced(struct nv_vgpu_device *vgpu, u32 h, static void svc_entry_outcome(const char *what, const struct svc_channel *ch, u64 idx, u64 va, u32 len) { - if (!gsp_sema_dbg || !ch->engine_type || svc.gpfifo_entry_log >= 512) + if (!gsp_sema_dbg || svc.gpfifo_entry_log >= 16384) return; svc.gpfifo_entry_log++; pr_info("sema-entry: %s h=0x%x idx=%llu va=0x%llx len=%u want=%u exp=%u unread=%d\n", @@ -13351,6 +20702,172 @@ static void svc_userd_sweep(struct nv_vgpu_device *vgpu, struct svc_channel *ch, ch->h_object, hi, lo); } +/* + * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. + * + * r225 evidence (witness-r225b.log): after a client was killed mid-run, a + * fresh client's GR channel (0x5c000059) was registered with a RESIDUAL + * USERD put=101 left by the dead session. Its fresh session wrote no put + * (it was stuck at ctx-init, polling MC_SERVICE_INTERRUPTS once a second), + * so the m181 fresh-put-drop recovery - which requires put < put_prev - + * never fired, and the host-side completion-counter publish stayed + * withheld (gr_pub_armed false). Meanwhile the pin-batch rewind + * (svc_handle_map_add, SVC_PIN_REWIND=8) kept yanking last_get from 101 + * back to 93 on every injection batch, so the entry loop re-walked slots + * 93..100 forever: 8 stale entries per batch, all decoding to nothing + * (rel=0, no capture) and consuming with zero delivery. The whole stall + * window shows zero captures, zero GR requests and therefore zero + * GR-STALL lines - the launch never reached the bounded submit path + * because the session epoch itself was wedged. + * + * Two signatures are detected, both keyed to NO FORWARD PROGRESS for + * SVC_GR_STALE_WEDGE_MS (the (last_get, put) pair unchanged across the + * window): + * + * 1. stale-epoch: a GR channel that still carries a residual put + * (put > 1 at this pass, publish withheld) and whose ring the host + * believes is fully consumed (last_get == put) - the fresh session + * never proved its switch with a put drop. The host applies the + * m181 recovery on a TIMEOUT instead: arm the counter publish and + * void the withheld publish state (no last_get rewind - a timeout is + * not proof the ring is the dead session's; the proven m181 rewind + * stays reserved for its put-drop signature). This is the recovery + * the killed-client rerun needs. + * + * 2. stale-replay: last_get pinned behind put while put == get (the + * producer has nothing pending) AND the loop consumed entries during + * the window (submissions moved) - i.e. the rewind is re-opening an + * already-consumed window faster than the loop closes it. The window + * is dropped (last_get = put) and the pin-batch rewind is put on a + * cooldown (wd_rewind_off_until) so the futile retry cannot pin the + * pump; the rewind budget returns later and a translation that lands + * can still retry (bounded duty cycle). + * + * Nothing here can fire on a green channel: a fresh channel has put <= 1 + * (signature 1) or consumes its first entries within milliseconds + * (submissions > 0); signature 2 additionally requires BOTH a stalled + * (last_get, put) pair for 3 s with no pending producer work AND consumed + * entries during that window, which only the rewind re-opening of a + * consumed window produces. + */ +#define SVC_GR_STALE_WEDGE_MS 3000u +static void svc_gr_stale_wedge_check(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 put, u32 get) +{ + bool residual, replay; + const char *reason = ""; + unsigned long waited; + + if (ch->engine_type || !ch->gpfifo_entries) + return; + /* New work (put advanced) clears the replay cooldown immediately. */ + if (ch->wd_rewind_off_until && ch->wd_put != put) + ch->wd_rewind_off_until = 0; + if (ch->wd_since && + (ch->wd_put != put || ch->wd_last_get != ch->last_get)) + ch->wd_since = 0; /* progress (or a new position): re-arm */ + residual = (!ch->gr_pub_armed && put > 1 && ch->last_get == put); + replay = (ch->last_get != put && put == get && + ch->wd_since && ch->submissions != ch->wd_sub); + if (!residual && !replay) { + ch->wd_since = 0; + ch->wd_put = put; + ch->wd_last_get = ch->last_get; + return; + } + if (!ch->wd_since) { + ch->wd_since = jiffies; + ch->wd_put = put; + ch->wd_last_get = ch->last_get; + ch->wd_sub = ch->submissions; + return; + } + if (time_before(jiffies, ch->wd_since + + msecs_to_jiffies(SVC_GR_STALE_WEDGE_MS))) + return; + waited = jiffies_to_msecs(jiffies - ch->wd_since); + if (residual) { + /* + * m181 recovery on timeout: the fresh session never dropped + * the residual put, so prove the epoch switch here. Arm the + * publish (the withheld completion counter was the plausible + * ctx-init wait - r225's client polled service-interrupts 1/s + * with no host edge) and void any publish state the stale + * pass advanced, exactly like m181; the guest word itself is + * only zeroed when the host authored it. No last_get rewind: + * a timeout is not proof that the ring is the dead session's + * (re-walking it could re-capture a stale launch), so the + * proven m181 rewind stays reserved for its put-drop + * signature; the client's own first put write will trigger it. + */ + ch->gr_pub_armed = true; + if (ch->counter_last) { + u64 upa = ch->release_pa ? ch->release_pa : + ch->counter_pa; + u32 zero = 0; + + /* + * r311: zero only a word authored by a DEAD + * generation. Zeroing a word this live session + * authored makes the client's UVM observe a + * downward jump (its tracker then never completes) + * while the publish re-advances it - the r311 + * vector_add wedge (pub-adv/zero cycles, client + * polling cur=0 forever). + */ + if (ch->counter_gen == svc_sess_gen) { + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: stale-recover-zero-skip h=0x%x upa=0x%llx last=%u (authored by live gen)\n", + ch->h_object, upa, + ch->counter_last); + } + ch->counter_last = 0; + } else if (upa) { + int zrc; + + /* + * r378: the zero is authored from a PA + * learned earlier (release_va -> release_pa); + * re-validate it before the 4-byte write + * lands so a freed/re-homed slot is never + * zeroed through the stale page. + */ + if (!gsp_sema_write_guard(ch->release_va, + upa, + SVC_APERTURE_SYS, + "stale-recover-zero")) + zrc = -ESTALE; + else + zrc = svc_write_ap(vgpu, upa, + SVC_APERTURE_SYS, + &zero, sizeof(zero)); + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-diag: stale-recover-zero h=0x%x upa=0x%llx last=%u zrc=%d\n", + ch->h_object, upa, + ch->counter_last, zrc); + } + } + ch->counter_last = 0; + } + reason = "stale-epoch"; + } else { + /* Drop the rewind-reopened consumed window; cool the rewind. */ + ch->last_get = put; + ch->wd_rewind_off_until = jiffies + + msecs_to_jiffies(8u * SVC_GR_STALE_WEDGE_MS); + reason = "stale-replay"; + } + ch->wd_since = 0; + ch->wd_put = put; + ch->wd_last_get = ch->last_get; + svc.gr_stale_wedge++; + pr_warn_ratelimited("vgsp-gr: GR-STALL-stale-wedge h=0x%x va=0x%llx last=%u put=%u get=%u sub=%llu waited_ms=%u action=recover reason=%s\n", + ch->h_object, ch->gpfifo_va, (u32)ch->last_get, + put, get, ch->submissions, waited, reason); +} + static void svc_pump_channels(struct nv_vgpu_device *vgpu) { /* Only ever called from the service thread. */ @@ -13366,8 +20883,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) bool stalled = false; spin_lock_irqsave(&svc.chan_lock, flags); + /* + * r507o stream-order: walk the CE channels BEFORE the GR channels. + * The client's stream order places its H2D copies before the kernel + * that consumes them, but the channels register in handle order (the + * RM's GR channel early, the copy-ce channels later), and a pass that + * decoded the launch before the CE channel's entries ran let the + * forward workqueue execute the launch's copy-in against pre-copy + * data (r480-sep: copy-in 115 ms before its stream's H2D soft + * copies; r481 proved no host semaphore links the two, so the + * emulation must enforce stream order itself). The CE copies execute + * on this same service thread, so consuming the CE entries first + * fixes the inversion directly - no cross-channel hold, no latency + * tax. Stable partition; relative order within each class preserved. + */ + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].engine_type) + local[n++] = svc.chan[i]; + } for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + !svc.chan[i].engine_type) local[n++] = svc.chan[i]; } spin_unlock_irqrestore(&svc.chan_lock, flags); @@ -13375,6 +20912,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) n_gr = SVC_GR_COUNTER_TOTAL; /* + * STALLFREE: complete a failed forwarded GR launch no hold owns (see + * svc_fwd_outcome_check). Runs before this pass's channel loop, so a + * finished request is always observed before any fill can reuse its + * slot. + */ + svc_fwd_outcome_check(vgpu); + + /* * H1 last resort: one self-gated content-discovery pass for the GR * ctx-init rings before this pass's channel loop. The pins it * installs (ring VA + push VA pages, both validated from content and @@ -13403,7 +20948,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * pass's snapshot belongs to a vanished channel; its waiter is gone, * so drop the set instead of resolving it later. */ - svc_opdef_drop_gone(local, n); + svc_opdef_drop_gone(vgpu, local, n); for (i = 0; i < n; i++) { struct svc_channel *ch = &local[i]; @@ -13412,6 +20957,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) int rrc; u32 put = 0, get = 0, guard = 0; u32 get_before; + u32 put_prev_wit; u64 applied_before; bool held = false; @@ -13458,7 +21004,8 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * svc_counter_autoreg). Skip while a relay hold owns * the channel, exactly like the end-of-pass call. */ - if (ch->counter_pa && ch->userd_pa && + if ((ch->release_pa || ch->counter_pa) && + ch->userd_pa && !(svc_fwd_hold.active && svc_fwd_hold.vgpu == vgpu && svc_fwd_hold.ch_handle == ch->h_object) && @@ -13576,6 +21123,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->logged = true; } + /* + * mm2: a GR channel whose USERD already reads PUT <= 1 cannot + * hold a stale consumed prefix, so its publishes (the catch-up + * for releases missing from consumed entries) are safe to + * enable immediately - this is the pristine first-boot ring. + * Channels discovered with a residual put stay unarmed until + * the fresh-put-drop rewind proves the session switch, so the + * residual put can never reach the fresh client's completion + * word (see gr_pub_armed and svc_counter_publish). + */ + if (!ch->engine_type && !ch->gr_pub_armed && put <= 1 && + !ch->userd_wait_fresh) + ch->gr_pub_armed = true; + /* Initialize expected semaphore payload from the pool (if mapped). */ svc_init_expected_payload(vgpu, ch); @@ -13629,12 +21190,103 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->h_object, ch->ce_index, put, (u32)ch->last_get); ch->last_get = put; + } else if (!ch->engine_type && ch->gpfifo_entries && + ch->last_get > put && + ch->submissions < ch->gpfifo_entries / 2) { + /* + * m181 (r203 GR ctx-init stall): a GR channel's + * ring/USERD live in fixed VRAM slots + * (fb 0x140000 + rank*0x3000) that keep the + * PREVIOUS session's final state across the + * session flush (m176 reset learned counter + * state but not the ring-consumption + * bookkeeping). A new session's first pump + * pass therefore sees the residual put (e.g. + * 107 = the last session's final GP_PUT) and + * consumes the stale ring up to it, advancing + * last_get before the client has written its + * own entries. The client's fresh put then + * drops below put_prev, and without this + * rewind last_get stays ahead of the producer: + * the live ctx-init entries are never decoded, + * their REPORT_SEM releases are never applied + * (the completion word freezes at the first + * stale-decoded payload), and the client's + * ctx-init wait polls forever - r203 rank 0 + * hung exactly so in multikernel / streams2 / + * victory_mail / smiley whenever the prior + * session had reached a high put (the sessions + * after a hung one, whose final put is 1, + * proceeded - a perfect alternation). Rewind + * to slot 0 like the r48 resync: a fresh + * session restarts its push sequence at slot + * 0, and re-decoding slot 0 is idempotent + * (the stale pass could only have authored the + * same payload, and the forward-only apply + * drops a duplicate). A genuine wrap cannot be + * this fresh (submissions guard), mirroring + * the r73 CE rewind. + */ + pr_info_ratelimited("sema-chan: fresh put drop (GR) h=0x%x rank=%u put=%u last=%u -> rewind to 0\n", + ch->h_object, ch->gr_index, put, + (u32)ch->last_get); + ch->last_get = 0; + /* + * mm2: the session switch is now proven - arm + * the publish for the live entries that follow, + * and void any publish state the stale pass + * advanced. Everything consumed to reach + * last_get was the dead session's ring; a + * publish issued on it wrote the residual put + * into the very word the fresh client waits on + * (its tracking semaphore; pub target is + * release_pa since 8c634144) and the + * forward-only apply then skips the real + * release payloads that follow. Re-arm the + * word to 0 and reset the catch-up bookkeeping + * so the re-decoded live releases re-advance + * it from the client's own sequence (1, 2, + * ...). Idempotent when no stale publish + * happened (counter_last == 0). + */ + ch->gr_pub_armed = true; + if (ch->counter_last) { + u64 upa = ch->release_pa ? + ch->release_pa : ch->counter_pa; + u32 zero = 0; + + if (upa) + (void)svc_write_ap(vgpu, upa, + SVC_APERTURE_SYS, + &zero, sizeof(zero)); + ch->counter_last = 0; + } } bitmap_zero(ch->slot_done_epoch, SVC_GPFIFO_MAX_BITS); + memset(ch->slot_pend_tries, 0, + sizeof(ch->slot_pend_tries)); + /* GRFIX FIX1b: a recycled slot in the next put epoch + * gets a fresh absolute-bound record; without this a + * wrapped slot would inherit a stale expired launch + * record. */ + ch->gr_hold_dead_va = 0; + ch->gr_hold_exp_logged = false; + /* r361: same for the opdef absolute bound. */ + ch->opdef_hold_dead_va = 0; + ch->opdef_hold_exp_logged = false; } + put_prev_wit = ch->put_prev; ch->put_prev = put; /* + * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. + * Runs before the entry loop so a wedged session epoch is + * recovered (and logged) instead of spinning: see + * svc_gr_stale_wedge_check. + */ + svc_gr_stale_wedge_check(vgpu, ch, put, get); + + /* * m136 (SA-STALL M2): liveness watchdog for a held CE entry. * When this channel's entry loop is not re-entered (early * continue paths above, or no fresh pushes), a deferred entry @@ -13669,6 +21321,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) get_before = ch->last_get; applied_before = svc.releases_applied; + /* + * C4 (GAP-1 H1): the GR walk witness - does put advance, and + * does last_get reach it? Budget 512/session; fires on a put + * change or while last_get is in the stalled 80..100 band. + */ + if (gsp_sema_dbg && svc.walk_dbg < 512 && + (put != put_prev_wit || + ((u32)ch->last_get >= 80 && (u32)ch->last_get <= 100))) { + svc.walk_dbg++; + pr_info("sema-diag: walk h=0x%x put=%u get=%u last=%u prev=%u\n", + ch->h_object, put, get, (u32)ch->last_get, + put_prev_wit); + } + while (ch->last_get != put && guard++ < ch->gpfifo_entries) { u8 entry[SVC_GPFIFO_ENTRY_SIZE]; u64 va, idx = ch->last_get; @@ -13701,7 +21367,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * (forward-only, so it cannot double-apply) and drop * it before this entry is examined. */ - svc_opdef_drop_stale(vgpu, ch, (u32)idx); + svc_opdef_drop_stale(vgpu, ch, (u32)idx, put); /* * A pin-batch rewind replays recently consumed @@ -13771,10 +21437,17 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ { u64 ident = (u64)e0 | ((u64)e1 << 32); - bool same = ident && + /* + * r505b: the per-slot op bookkeeping arrays are + * SVC_GPFIFO_MAX_BITS (1024) entries; the X/GL + * channel's ring is 8192 slots. Bound every slot + * index here or a large ring writes out of + * bounds. + */ + bool same = ident && idx < SVC_GPFIFO_MAX_BITS && ch->slot_ops_ident[idx] == ident; - entry_ident = ident; + entry_ident = (idx < SVC_GPFIFO_MAX_BITS) ? ident : 0; /* * A complete prior pass suppresses every op. * A partial pass (an operand's targeted scan @@ -13782,6 +21455,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * leaves the per-op done bitmap to decide: * work that already ran must not run again, * the pending op must get its retry. + * + * r354: the content witness decides (the hash + * override sits below, after the push bytes + * are read): entry words alone cannot tell a + * true rewind replay (identical bytes - must + * not run twice) from a REWRITTEN push that + * reuses the same push-buffer VA and length + * with new methods (the driver's repeated + * page-table/fill pushes at the same ring + * slots - suppressing those left the guest's + * GPU page tables stale, the GMMU walk + * missing operand pages, and the first CUDA + * client of every fresh boot failing with + * dropped operands). */ entry_replay = same && ch->slot_ops_complete[idx]; @@ -13873,7 +21560,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) injected = (svc_ipin_lookup(va, &ipa, &iap) == 0); - if (!svc_method_stream_plausible(pb, len)) { + /* + * r496e: an all-zero push on a + * content-scanned-ring channel is the + * driver's NOP sanity push (one zero + * dword - NVA06F_DMA_NOP), not a bad + * translation: accept it and let the + * decode find no ops. + */ + bool zero_nop = ch->gr_ram_ring && + len >= 4 && + !memchr_inv(pb, 0, len); + + if (!svc_method_stream_plausible(pb, len) && + !zero_nop) { svc.false_trans++; if (injected) { u8 *fb = kmalloc(len, @@ -14024,6 +21724,86 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) { u32 rel_n_before = svc_rel_count; bool rel_seen; + /* + * m156: this entry is inside a live + * GR hold window and its decode is a + * replay that would suppress the + * launch capture. Force exactly one + * fresh capture pass (ops stay + * suppressed: they already ran; only + * the candidate build is re-done) so + * the deferred launch is forwarded + * for real once the engine's + * in-flight work completes. + */ + bool rehold = + !ch->engine_type && + ch->gr_rehold_idx == + (u32)idx; + /* + * m160: the capture gate used to + * require a host-injected pin. A + * fresh ring entry whose push read + * succeeded and whose bytes parse + * as a plausible method stream is + * the client's actual work - the + * same evidence the untrusted ops + * gate already accepts - and a GR + * capture built from it is exactly + * what the m156 rehold forces for + * deferred launches. Without this, + * a channel whose pages the pin + * stream had not reached yet (the + * streams2 second-channel case: + * entry consumed, no capture, the + * client read zeros) never gets + * its launch forwarded at all. + * Replays stay suppressed: a + * consumed entry must not run its + * work twice. + */ + bool cap_trust = injected || rehold; + + /* + * r354: the content witness for + * the once-only op gate. Entry + * words alone cannot tell a true + * rewind replay (identical bytes) + * from a rewritten push that + * reuses the same push VA and + * length with new methods (the + * driver's repeated fill / + * page-table pushes at the same + * ring slots). Suppressing the + * rewritten push left the GPU + * page tables stale and the first + * client's operands unresolved. + * slot_ops_hash has been stored + * since r62 for this comparison. + */ + if (entry_replay && + ch->slot_ops_hash[idx] != + svc_push_hash(pb, len)) { + entry_replay = false; + if (svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-sema: rewritten push runs (idx=%llu va=0x%llx len=%u)\n", + (unsigned long long)idx, va, len); + } + } + if (rehold && !injected && + svc_gr_retry_log < 64) { + svc_gr_retry_log++; + pr_info("vgsp-gr: GR rehold capture idx=%llu va=0x%llx\n", + (unsigned long long)idx, va); + } + if (!cap_trust && !entry_replay && + !ch->engine_type && + gsp_phys_gr_forward_enabled() && + !unreadable && + svc_method_stream_plausible(pb, + len)) + cap_trust = true; svc_entry_outcome("decode", ch, idx, va, len); @@ -14031,15 +21811,18 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc_decode_stream(vgpu, pb, len, 0, (allow_ops || injected) && - !entry_replay, + !entry_replay && + !rehold, ch, idx, va, - injected, - entry_replay, - entry_replay ? + cap_trust, + entry_replay && !rehold, + (entry_replay && !rehold) ? NULL : &entry_done); + if (rehold) + ch->gr_rehold_idx = ~0u; /* * Record what ran for this entry * identity before any hold path can @@ -14087,8 +21870,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ if (svc.op_tr_pending && svc_opdef_hold(vgpu, ch, - (u32)idx)) { + (u32)idx, + va)) { svc.gpfifo_nomap++; + if (gsp_sema_dbg && svc.pend_dbg_log < 1024) { + svc.pend_dbg_log++; + pr_info("sema-pend: hold h=0x%x idx=%u va=0x%llx\n", + ch->h_object, (u32)idx, va); + } /* * P6b wave 1.5: * releases recorded @@ -14112,7 +21901,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(pb); break; } - if (svc.op_tr_pending) + if (svc.op_tr_pending) { /* * Deadline hit: the * operand never @@ -14124,6 +21913,34 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc_opdef_flush(vgpu, ch, (u32)idx); + /* + * r505b: mark the slot's + * ops complete. The op was + * permanently skipped, but + * the decode above stored + * complete=0 (op_tr_pending + * was set), so every pin- + * batch rewind replay of + * this slot re-decoded it, + * re-set op_tr_pending and + * re-held the entry in a + * livelock (multikernel on + * r505: opdef held=18359, + * expired=61, the client + * waiting forever while the + * pump looped on slot 1). + * Nothing is owed after the + * flush, so a replay must + * skip the ops. + */ + if ((u32)idx < + SVC_GPFIFO_MAX_BITS) { + ch->slot_ops_complete[idx] = + 1; + ch->slot_ops_done[idx] = + entry_done; + } + } rel_seen = (svc_rel_count != rel_n_before); { bool fwd_guest_queued; @@ -14152,42 +21969,258 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) !ch->engine_type) ch->gr_hold_idx = ~0u; /* + * m159: a busy refusal (forward + * in flight / release hold + * active) means this launch CAN + * still run - only later. The + * candidate survives the refusal + * (the busy paths do not consume + * it), so wait bounded for the + * slot to free - resolving the + * active hold through its own + * channel, since this pump thread + * is the hold resolver - then + * retry the request inline. This + * serializes multi-launch clients + * (streams2: kernels 2/3 queued + * while kernel 1 is in flight) + * without losing their releases. + * On timeout the historical + * consume + soft-release path + * stands (bounded, never a hang). + */ + if (!fwd_guest_queued && + !ch->engine_type && + svc_gr_busy_defer) { + unsigned long wend = jiffies + + msecs_to_jiffies(4000); + + while (time_before(jiffies, + wend)) { + if (svc_fwd_hold.active && + svc_fwd_hold.vgpu == + vgpu) { + int hk; + + for (hk = 0; hk < n; hk++) { + if (local[hk].h_object != + svc_fwd_hold.ch_handle) + continue; + svc_fwd_hold_poll(vgpu, + &local[hk], + 0); + break; + } + } + if (!svc_fwd_hold.active && + !test_bit(SVC_FWD_INFLIGHT, + &svc_fwd_state) && + smp_load_acquire( + &svc_fwd_req.done)) + break; + msleep(5); + } + if (!svc_fwd_hold.active && + !test_bit(SVC_FWD_INFLIGHT, + &svc_fwd_state)) + fwd_guest_queued = + svc_fwd_request(vgpu, + ch); + if (fwd_guest_queued) { + svc.fwd_gr_defer++; + } else if (!svc_gr_hold_bound_expired(ch, + svc_gr_launch_va(ch)) && + (ch->gr_hold_idx != (u32)idx || + time_before(jiffies, + ch->gr_hold_until) || + /* + * r420: the backend's + * bounded WEDGE-QUIET + * hold outlives this + * one-shot window; keep + * re-holding (never + * consume-with-sw-release) + * while it is active, so + * the launch is still + * forwarded and copied out + * after the hold instead of + * being released onto stale + * data. + */ + gsp_phys_gr_quiet_hold_active())) { + u32 hold_ms = gsp_sema_gr_grace_ms ? + gsp_sema_gr_grace_ms : 8000; + + /* + * P2 (GAP-2 A3): the wait + * expired without queueing. + * Do NOT fall through to the + * software-release consume - + * the launch was never + * submitted, so authoring + * its release would be a + * false completion. Hold + * exactly like the + * data-stalled path + * (m150/m156); the one-shot + * window lets the historical + * consume resume after it + * expires. + */ + ch->gr_rehold_idx = (u32)idx; + ch->gr_hold_idx = (u32)idx; + ch->gr_hold_until = jiffies + + msecs_to_jiffies(hold_ms); + svc.fwd_gr_defer++; + if (svc_busy_defer_log < 64) { + svc_busy_defer_log++; + pr_info("vgsp-work: GR busy-defer timeout h=0x%x idx=%llu -> hold %u ms\n", + ch->h_object, + (unsigned long long)idx, + hold_ms); + } + kfree(pb); + break; + } else if (svc_busy_defer_log < 64) { + /* + * C8 (GAP-2 A3): the bounded + * hold window is over; the + * historical consume with the + * software release resumes. + */ + svc_busy_defer_log++; + pr_info("vgsp-work: GR busy-defer timeout h=0x%x idx=%llu -> consume w/ sw release\n", + ch->h_object, + (unsigned long long)idx); + } + } + /* * m150: the GR fill held this - * launch because its high - * argument pointers do not - * translate yet. Leave the - * entry unconsumed (do not - * write GP_GET) so the pump - * re-captures and retries it; - * one bounded window per entry, - * then fall through and consume - * exactly as before. The slot's - * identity makes the hold - * one-shot per entry: after - * expiry the same entry is - * never re-held. + * launch because it carries + * data pages that do not + * resolve yet. Leave the entry + * unconsumed (do not write + * GP_GET) so the pump + * re-captures and retries it. + * GRFIX FIX1b (r213): the hold + * is bounded ABSOLUTELY per + * launch push VA + * (svc_gr_hold_bound_expired + * fixes the deadline on the + * first hold and nothing + * re-arms it; svc_gr_request + * enforces it and forwards the + * launch once it passes). The + * extra check here is the + * defence for a stale stalled + * verdict when the request + * path did not run this pass - + * past the bound the entry + * falls through to the + * historical consume. */ if (!fwd_guest_queued && !ch->engine_type && svc_gr_data_stalled && - ch->gr_hold_idx != (u32)idx) { + !svc_gr_hold_bound_expired(ch, + svc_gr_launch_va(ch)) && + (ch->gr_hold_idx != (u32)idx || + time_before(jiffies, + ch->gr_hold_until))) { u32 hold_ms = gsp_sema_gr_grace_ms ? gsp_sema_gr_grace_ms : 8000; + /* + * m156: arm the forced + * re-capture so the next + * decode of this entry + * rebuilds the GR candidate + * (a replay decode would + * suppress it) and the + * deferred launch is + * forwarded for real once + * the in-flight work has + * completed. + */ + ch->gr_rehold_idx = + (u32)idx; ch->gr_hold_idx = (u32)idx; ch->gr_hold_until = jiffies + msecs_to_jiffies(hold_ms); svc.fwd_gr_defer++; if (svc_gr_hold_log < 16) { svc_gr_hold_log++; - pr_info("sema-svc: GR hold h=0x%x entry=%u (data args unresolved; retry window %u ms)\n", + pr_info("sema-svc: GR hold h=0x%x entry=%u (%s; retry window %u ms)\n", ch->h_object, - (u32)idx, hold_ms); + (u32)idx, + svc_gr_data_stalled ? + "data args unresolved" : + "forward busy (m156)", + hold_ms); } kfree(pb); break; } /* + * STALLFREE: the absolute per-launch + * hold bound just fired - the entry + * falls through to the historical + * consume, where the tail authors the + * QMD completion release and logs + * GR-STALL-hold. This line pins the + * bound expiry itself (the r220 hold + * storm's fix: one line per launch, + * nothing re-arms it across rewind + * replays). + */ + if (!fwd_guest_queued && + !ch->engine_type && + svc_gr_data_stalled && + ch->gr_hold_bound_hit && + !ch->gr_hold_exp_logged) { + ch->gr_hold_exp_logged = true; + /* r509: the latch is this pass's only + * report of the expiry; clear it so the + * NEXT expiry cycle (gr_hold_exp_logged + * now set) falls through to the + * historical consume. */ + ch->gr_hold_bound_hit = false; + svc.fwd_gr_hold_exp++; + /* + * r351: the bound firing used to + * fall through to the historical + * consume (entry consumed, QMD + * release authored) WITHOUT ever + * forwarding the launch - the + * client's kernel never ran and + * the client hung (victory_reduce + * with gsp_phys_gr_fill_hold=1: + * bound fired every 3 s, zero GR + * LAUNCH lines, 110 s+ hang). + * Instead, force ONE more fresh + * capture (rehold) with the stall + * verdict cleared: the request's + * own bound is expired too, so it + * proceeds and forwards the launch + * best-effort with whatever + * resolved. If that retry cannot + * run, the next pass (flag set) + * falls through to the historical + * consume as before. + */ + pr_warn("vgsp-gr: GR-STALL-hold-bound va=0x%llx waited_ms=%u action=retry-forward (bound expired; one best-effort forward pass)\n", + svc_gr_launch_va(ch), + jiffies_to_msecs(jiffies - + ch->gr_hold_dead_since)); + svc_gr_data_stalled = false; + ch->gr_rehold_idx = (u32)idx; + ch->gr_hold_idx = (u32)idx; + ch->gr_hold_until = jiffies + + msecs_to_jiffies(200); + kfree(pb); + break; + } + /* * P6b wave 1.5: when * this pass recorded * releases and the @@ -14263,14 +22296,171 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * bridge job owns the release. */ if (!ch->engine_type && - !gsp_cuda_bridge_enabled()) + !gsp_cuda_bridge_enabled()) { svc_gr_sw_release(vgpu, ch->h_object, svc_gr_cand.rel_va, svc_gr_cand.rel_payload); + /* STALLFREE: latch so the + * consume tail cannot author + * the same release twice. */ + svc_gr_cand.rel_done = true; + } } } /* + * m164 CLASS-2 no-capture hold: a + * readable GR launch push whose + * decode saw QMD words but built + * no launch candidate for this + * channel (replay-suppressed + * capture, partial/split QMD, or + * a page the pin stream has not + * reached) must not be consumed + * at the tail - consuming drops + * the client's launch and only + * the RAM sweep's release reaches + * it (r161/r167 streams2 second + * vadd: 56 consumes / 2 captures + * on channel 0x5c000023). Hold + * the entry and force a fresh + * capture on the next pass + * (gr_rehold_idx makes the replay + * decode rebuild the candidate); + * one bounded window per + * (slot, push VA), then fall + * through to the historical + * consume so the channel can + * never livelock. + * + * r470: an ALL-ZERO push page is + * the same "not the client's work + * yet" condition - the ring entry + * is written (the previous + * session's or the client's own + * slot reuse) but the push + * CONTENT has not landed, so the + * decode sees no QMD and the + * entry would be consumed as a + * no-op; the end-of-pass counter + * publish then advances the + * guest's completion word while + * the real push was never read, + * and the client drains a + * staging buffer that still holds + * the PREVIOUS copy's bytes + * (streams_events: h_c[2] = c0 + * because the c2 D2H's + * completion was published 4.5 s + * before its copy ran, the ring + * slots having been decoded as + * all-zero pushes). Deliberately + * NARROWER than the CE-shaped + * plausibility verdict: compute + * pushes legitimately fail that + * matcher, so only an entirely + * zero page counts as unwritten. + */ + { + bool all_zero = false; + + if (!ch->engine_type && + va && len) { + u32 zi; + + all_zero = true; + for (zi = 0; zi < len; + zi++) + if (pb[zi]) { + all_zero = + false; + break; + } + } + /* + * r471: publish clamp bookkeeping. + * A zero-content GR entry that is + * held or consumed must never let + * the end-of-pass counter publish + * advance the guest's completion + * word past it (the client would + * satisfy its readback wait before + * the real push/copy ran - the + * staging buffer still holds the + * previous copy's bytes). Record + * the lowest zero slot; a decode + * that reads non-zero content + * clears the clamp. + */ + if (!ch->engine_type) { + if (all_zero) { + if (!ch->gr_bad_put || + (u32)idx + 1 < + ch->gr_bad_put) + ch->gr_bad_put = + (u32)idx + 1; + } else if (va && len) { + ch->gr_bad_put = 0; + } + } + if (!ch->engine_type && va && len && + gsp_phys_gr_forward_enabled() && + (svc_gr_qmd_seen > 0 || + all_zero) && + !entry_replay && + !(svc_gr_cand.valid && + svc_gr_cand.h_object == + ch->h_object && + svc_gr_cand.push_va == va)) { + if (ch->gr_nocap_idx != + (u32)idx || + ch->gr_nocap_va != va) { + ch->gr_nocap_idx = + (u32)idx; + ch->gr_nocap_va = va; + ch->gr_nocap_replay = + entry_replay ? + 1 : 0; + ch->gr_nocap_since = + jiffies; + } + if (gsp_sema_gr_nocap_hold_ms && + !ch->gr_ram_ring && + time_before(jiffies, + ch->gr_nocap_since + + msecs_to_jiffies( + gsp_sema_gr_nocap_hold_ms))) { + /* Force a fresh capture on + * the retry: a replay decode + * would suppress it. */ + ch->gr_rehold_idx = + (u32)idx; + svc.gpfifo_held++; + svc.gr_nocap_held++; + if (svc_gr_nocap_log < 16) { + svc_gr_nocap_log++; + pr_info("sema-chan: GR no-capture hold h=0x%x entry=%u va=0x%llx len=%u replay=%d qmd_seen=%u (forcing capture; window %u ms)\n", + ch->h_object, + (u32)idx, va, + len, + ch->gr_nocap_replay, + svc_gr_qmd_seen, + gsp_sema_gr_nocap_hold_ms); + } + kfree(pb); + break; + } + /* Window expired: consume + * exactly as the pre-change + * tail does, with a proof + * line. */ + svc.gr_nocap_consumed++; + pr_info_ratelimited("sema-chan: GR no-capture hold expired h=0x%x entry=%u va=0x%llx len=%u (consuming without a launch)\n", + ch->h_object, + (u32)idx, va, len); + } + } + /* * A CE push that should carry this * channel's next tracking release but * decoded none read from a @@ -14806,10 +22996,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since = 0; ch->unread_va = 0; ch->unread_until = 0; - if (gsp_sema_fake_rel || - gsp_sema_rel_recover) - svc_replay_note(ch, idx, - put); + /* + * m177: deliver any + * withheld set even + * on the GR stale-skip + * path. GR channels + * have no rescue and + * their consume-time + * delivery gate is + * below; without this + * the slot strands + * until the sweep. + */ + svc_replay_note(ch, idx, + put); ch->last_get = (idx + 1) % ch->gpfifo_entries; svc.gpfifo_skipped++; @@ -14845,13 +23045,61 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } } - if (ch->engine_type && svc_rel_count != rel_n_iter) + /* + * m177: GR channels (engine_type == 0) have no rescue and + * no CE-shaped delivery branches above; deliver any + * withheld set for this entry whenever the entry is + * consumed so a held release can never strand until the + * 10 s sweep (evtest3's st[2] sync hang). + */ + if (svc_rel_count != rel_n_iter || !ch->engine_type) svc_replay_note(ch, idx, put); + /* + * STALLFREE: bounded completion for a GR capture that was + * consumed without ever being forwarded - a data-stall, + * busy-defer or no-capture hold window expired and the + * entry fell through to the historical consume. No + * hardware copy is coming and no release hold owns the + * launch, so without this the QMD completion word the + * client polls is never authored by the host and the + * launch wait runs to the guest's own timeout. Author + * it now, forward-only, latched once per capture. + * svc_gr_cand.push_va is the entry's push VA, so a + * candidate captured for a later entry of the same + * channel can never be completed by this entry's + * consume. + */ + if (!ch->engine_type && svc_gr_cand.valid && + svc_gr_cand.h_object == ch->h_object && + !svc_gr_cand.offered && !svc_gr_cand.rel_done && + svc_gr_cand.rel_va && + svc_gr_cand.push_va == va) { + svc_gr_cand.rel_done = true; + svc_gr_sw_release(vgpu, ch->h_object, + svc_gr_cand.rel_va, + svc_gr_cand.rel_payload); + svc.fwd_gr_stall_swrel++; + pr_warn("vgsp-gr: GR-STALL-hold va=0x%llx waited_ms=%u action=proceed (unforwarded GR capture consumed; QMD release authored in software)\n", + va, + jiffies_to_msecs(jiffies - + svc_gr_cand.captured_jiffies)); + } ch->last_get = (idx + 1) % ch->gpfifo_entries; ch->rescue_fail = 0; ch->defer_since = 0; + /* m164: the entry was consumed (or fell through the + * no-capture window); clear the hold identity so a + * future push on this slot opens a fresh window. */ + ch->gr_nocap_idx = ~0u; + ch->gr_nocap_since = 0; + ch->gr_nocap_va = 0; ch->unread_va = 0; ch->unread_until = 0; + /* GRFIX FIX1c (r214): do NOT reset the per-launch + * bound record here - a consume of an unrelated entry + * would re-arm the held launch's bound every pass + * (see svc_fwd_hold_poll). Reset happens on channel + * (re)registration and the session diag reset. */ /* Publish progress the way the GPU would. */ g = ch->last_get; @@ -14916,11 +23164,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) spin_lock_irqsave(&svc.chan_lock, flags); for (k = 0; k < SVC_MAX_CHANNELS; k++) { - if (svc.chan[k].valid && + if (svc.chan[k].valid && svc.chan[k].vgpu == vgpu && svc.chan[k].h_object == ch->h_object) { svc.chan[k].userd_pa = ch->userd_pa; svc.chan[k].userd_as = ch->userd_as; svc.chan[k].userd_ok = ch->userd_ok; + /* r496b: persist the RAM-scan cursor and the + * one-shot USERD-scan latch so they survive + * the per-pass snapshot copy. */ + svc.chan[k].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[k].gr_ram_passes = ch->gr_ram_passes; + svc.chan[k].gr_ram_logged = ch->gr_ram_logged; + svc.chan[k].gr_ram_ring = ch->gr_ram_ring; + svc.chan[k].userd_scans = ch->userd_scans; svc.chan[k].logged = ch->logged; svc.chan[k].logged_expected = ch->logged_expected; svc.chan[k].notified = ch->notified; @@ -14947,6 +23203,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].push_hold_until = ch->push_hold_until; svc.chan[k].defer_since = ch->defer_since; svc.chan[k].put_prev = ch->put_prev; + /* r471: the publish clamp watermark is set + * and cleared on the pump thread; persist it + * so the next pass still clamps. */ + svc.chan[k].gr_bad_put = ch->gr_bad_put; /* * Persist the cheap-wait window and the content * watermark: the pump works on a snapshot, so @@ -14967,6 +23227,80 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc.chan[k].gr_hold_until = ch->gr_hold_until; svc.chan[k].gr_hold_idx = ch->gr_hold_idx; + /* + * GRFIX NOTE (r213-r215): ANY new per-channel + * record field MUST be added to this snapshot + * write-back list or it is zeroed on every pass + * (the pump works on the local[] snapshot copied + * at the top of svc_pump_channels). + * + * STALLFREE (r213-r215 follow-up): the GRFIX + * absolute-bound fields were the proven record + * destroyer - they were intentionally left out, + * which is why gsp_phys_gr_fill_hold defaulted + * to 0 (the bound could never fire). They are now + * written back, so the per-launch bound survives + * the snapshot and is effective on the default + * path: a data-stalled launch is held for at most + * gsp_phys_gr_hold_max_ms (default 3000) and then + * is consumed with a software completion, instead + * of being re-held forever by a rewind replay at + * a different slot index (the r220 196 s hold). + */ + svc.chan[k].gr_hold_dead_va = ch->gr_hold_dead_va; + svc.chan[k].gr_hold_dead_since = + ch->gr_hold_dead_since; + svc.chan[k].gr_hold_deadline = + ch->gr_hold_deadline; + svc.chan[k].gr_hold_exp_logged = + ch->gr_hold_exp_logged; + /* r509: the one-shot expiry latch is snapshot + * state too. */ + svc.chan[k].gr_hold_bound_hit = + ch->gr_hold_bound_hit; + /* r361: the opdef absolute-bound record is + * snapshot state too (the GRFIX lesson) - + * without the write-back the deadline would + * be zeroed every pass and the bound could + * never fire. */ + svc.chan[k].opdef_hold_dead_va = + ch->opdef_hold_dead_va; + svc.chan[k].opdef_hold_deadline = + ch->opdef_hold_deadline; + svc.chan[k].opdef_hold_exp_logged = + ch->opdef_hold_exp_logged; + /* m164: the CLASS-2 no-capture window is snapshot + * state too - without the write-back every pass + * would re-arm it and rehold the entry forever + * (the m150 lesson). */ + svc.chan[k].gr_nocap_idx = ch->gr_nocap_idx; + svc.chan[k].gr_nocap_replay = + ch->gr_nocap_replay; + svc.chan[k].gr_nocap_va = ch->gr_nocap_va; + svc.chan[k].gr_nocap_since = ch->gr_nocap_since; + /* + * STALLFREE r225: the stale-wedge watchdog + * state is snapshot state too - without the + * write-back the window would restart every + * pass and never expire. + */ + svc.chan[k].wd_last_get = ch->wd_last_get; + svc.chan[k].wd_put = ch->wd_put; + svc.chan[k].wd_sub = ch->wd_sub; + svc.chan[k].wd_since = ch->wd_since; + /* + * r312: gr_pub_armed / counter_gen live only on + * the snapshot copy if not persisted. Losing the + * armed bit re-arms the residual watchdog every + * pass; its recovery then keeps zeroing the + * completion word and re-publishing - the r311/ + * r312 vector_add wedge loop (66 stale-wedge + * rows with the client polling forever). + */ + svc.chan[k].gr_pub_armed = ch->gr_pub_armed; + svc.chan[k].counter_gen = ch->counter_gen; + svc.chan[k].wd_rewind_off_until = + ch->wd_rewind_off_until; svc.chan[k].pushpg_va = ch->pushpg_va; svc.chan[k].pushpg_n = ch->pushpg_n; svc.chan[k].content_max = ch->content_max; @@ -14992,6 +23326,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) memcpy(svc.chan[k].slot_ops_ident, ch->slot_ops_ident, sizeof(ch->slot_ops_ident)); + memcpy(svc.chan[k].slot_pend_tries, + ch->slot_pend_tries, + sizeof(ch->slot_pend_tries)); memcpy(svc.chan[k].slot_ops_done, ch->slot_ops_done, sizeof(ch->slot_ops_done)); @@ -15230,7 +23567,7 @@ static void svc_handle_chan_dump(struct nv_vgpu_device *vgpu) spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) - if (svc.chan[i].valid) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu) local[n++] = svc.chan[i]; spin_unlock_irqrestore(&svc.chan_lock, flags); @@ -15308,6 +23645,41 @@ static u32 svc_map_add_head; /* producer */ static u32 svc_map_add_tail; /* consumer (service thread) */ static DEFINE_SPINLOCK(svc_map_add_lock); +/* + * m154: does this injected-pin VA fall inside any live channel's GPFIFO ring + * range? The ring VA translation must come from the channel's own discovery + * path (channel object / push-buffer scan); a scanner run that happens to + * span the ring VA is enumerating a different mapping and its translation is + * always wrong for the ring (r153: an all-zero read consumed a ring entry + * and wedged the client for the boot). Called on the service thread; takes + * chan_lock like the rewind below. + */ +static bool svc_va_hits_chan_ring(u64 va) +{ + unsigned long flags; + u32 i; + bool hit = false; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *ch = &svc.chan[i]; + u64 lo, hi; + + if (!ch->valid || !ch->gpfifo_va || !ch->gpfifo_entries) + continue; + lo = ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1); + hi = (ch->gpfifo_va + + (u64)ch->gpfifo_entries * 8 + SEMA_PAGE_SIZE - 1) & + ~(u64)(SEMA_PAGE_SIZE - 1); + if (va >= lo && va < hi) { + hit = true; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return hit; +} + static void svc_handle_map_add(void) { unsigned long flags; @@ -15332,8 +23704,66 @@ static void svc_handle_map_add(void) ap = req.ap ? req.ap : SVC_APERTURE_SYS; for (off = 0; off < (req.len ? req.len : 1); off += SEMA_PAGE_SIZE) { - svc_ipin_store(req.va + off, req.pa + off, ap); - svc_lat_store(req.va + off, req.pa + off, ap); + u64 va = req.va + off; + + /* + * m154: never let an injected pin shadow a live + * channel's GPFIFO ring. r153: the scanner's merged + * CE-window run (va=0x121000000 len=0x40000, the + * length varies per session) covered a CE channel's + * ring VA 0x121010000; the injected translation took + * precedence in svc_ipin_lookup/read, the ring then + * read the guest page at 0x102e02000 - an all-zero + * entry was silently consumed, the pool release was + * never authored, and the client spun in userspace + * for the rest of the boot (pushes frozen). The ring + * pages already have their own discovery path (the + * channel object / push-buffer scan); a colliding + * injection is always the wrong translation, so drop + * it and leave the ring to that path. + */ + if (svc_va_hits_chan_ring(va)) { + svc_ring_pin_skips++; + if (svc_ring_pin_skips <= 8) + pr_info("sema-svc: injected pin va=0x%llx DROPPED (live channel ring range; ring reads stay on the channel's own translation)\n", + va); + continue; + } + svc_ipin_store(va, req.pa + off, ap); + svc_lat_store(va, req.pa + off, ap); + /* + * r372 (stale-ledger corruption class): a fresh pin is the + * guest's newest statement of where this VA's content + * lives. The landing table (ledger) otherwise only moves + * on host writes, so a VA re-pinned to a NEW page (the + * guest freed the old buffer and reallocated it - CUDA + * reuses VA ranges across processes) kept resolving to + * the OLD, freed page: every later write through that + * ledger entry landed on memory the guest kernel had + * since reused (proven class: 8 zero bytes into an + * nv_free_system_pages page array -> guest oops at + * client teardown). Repair the entry only when it + * disagrees with the pin (a no-op otherwise, so the + * table is not flooded with pin pages). + */ + { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(va, &lpa, &lap) && + (((lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + ((req.pa + off) & ~(u64)(SEMA_PAGE_SIZE - 1))) || + (u32)lap != (u32)ap)) { + svc_ledger_pin_repairs++; + if (svc_ledger_pin_repairs <= 32) + pr_info("sema-ledger: PIN-REPAIR va=0x%llx old=0x%llx ap=%u -> new=0x%llx ap=%u\n", + va, + lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + lap, + req.pa + off, ap); + svc_ledger_store(va, req.pa + off, ap); + } + } } /* * m149: bounded per-session evidence that the CE push-buffer @@ -15376,6 +23806,20 @@ static void svc_handle_map_add(void) if (!ch->valid || !ch->gpfifo_entries) continue; /* + * STALLFREE r225: a channel under the stale-replay + * watchdog's rewind cooldown is NOT rewound. Without + * this the rewind re-opened the same consumed window + * every batch while the producer had nothing pending, + * pinning last_get and re-walking stale entries + * forever (r225: slots 93..100 of 0x5c000059 for + * 42+ s). The cooldown expires on its own (bounded + * duty cycle) and is cleared by new work, so a + * translation landing later can still retry. + */ + if (ch->wd_rewind_off_until && + time_before(jiffies, ch->wd_rewind_off_until)) + continue; + /* * Clamp the rewind at 0: wrapping below zero (get=0 -> * entries - N) makes the pump read ring slots past the * producer that hold stale entries, so replay only @@ -15555,9 +23999,145 @@ int gsp_sema_service_counter_add(u32 h_object, u64 pa) return ret; } +/* + * Register the guest PMA scrubber's channel-buffer words. Called from the + * sema_scrubbuf_add debug parameter with the PAs the guest scanner resolved + * (vmallocinfo vmap area + kcore page-table walk). fin_pa is the finish + * payload the guest's scrub wait polls; hsem_pa is the channel's host + * semaphore (channelWaitForFreeEntry polls it). Plain state update. + * + * Dual-guest: the words are written through a specific guest's address space + * (see svc_scrub_pass), so the registration binds to the FIRST open mdev - + * the same single-guest convention the scanner tooling follows (it tracks + * one client). Refused with no open mdev rather than parked globally. + */ +int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa) +{ + struct nv_vgpu_device *vgpu = NULL; + struct nv_vgpu_device *d; + struct svc_scrub *s; + + if (!svc.task) + return -ENODEV; + /* Both words are u32: require 4-byte alignment and a RAM-shaped PA. */ + if (!fin_pa || (fin_pa & 0x3) || fin_pa >= SEMA_PA_MAX) + return -EINVAL; + if (!hsem_pa || (hsem_pa & 0x3) || hsem_pa >= SEMA_PA_MAX) + return -EINVAL; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (!vgpu) { + pr_warn_ratelimited("sema-svc: scrubber buffer registration ignored: no open mdev to bind it to\n"); + return -ENODEV; + } + s = svc_scrub_get(vgpu, true); + if (!s) + return -ENOSPC; + + s->fin_pa = fin_pa; + s->hsem_pa = hsem_pa; + s->pa_ok = true; + s->pa_gen++; + pr_info("sema-svc: scrubber buffer registered fin_pa=0x%llx hsem_pa=0x%llx (gen %u)\n", + fin_pa, hsem_pa, s->pa_gen); + return 0; +} + +/* + * Author the CeUtils scrubber channel's progress once per service pass. + * + * The physical CE would advance, per completed scrub block, the channel's + * finish payload (channel buffer + 0x6c004) and write the channel's host + * semaphore as it consumes GPFIFO entries. Nothing in this emulation does, + * so the guest PMA's scrub wait (scrubCheckAndWaitForSize) freezes from the + * third CUDA session on. This pass closes that loop from the channel's own + * USERD GPPut (which the guest driver writes before every submit): + * + * submitted = number of scrub blocks submitted (GPPut is end-exclusive + * and counts submissions until the ring wraps; track the wrap ourselves), + * then author USERD GP_GET = put (consumed), host sema = put and finish + * payload = submitted. Forward-only, idempotent, retried every pass so a + * late scanner registration (or a post-flush reset) heals by itself. + * + * The scrub work itself is not executed (no shadow memsets): the guest + * only needs the progress to proceed. Executing the memsets requires + * decoding the channel's pushes, which the scanner can later provide pins + * for; correctness impact is limited to stale data in pages the guest + * believes scrubbed. + */ +static void svc_scrub_pass(struct nv_vgpu_device *vgpu) +{ + struct svc_scrub *s = svc_scrub_get(vgpu, false); + u32 put = 0; + u64 delta; + + if (!s || !s->userd_ok || !s->pa_ok || !s->entries) + return; + if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as, + &put, sizeof(put))) { + /* Retry through the alternate aperture (as registration did). */ + if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, + s->userd_as == SVC_APERTURE_FB ? + SVC_APERTURE_SYS : SVC_APERTURE_FB, + &put, sizeof(put))) + return; + } + if (put >= s->entries) + return; /* junk translation; retry next pass */ + + if (!s->seeded) { + s->seeded = true; + s->put_last = put; + /* GPPut = (lastSubmittedEntry + 1) % entries: the submission + * count until the ring wraps. Seeding from it adopts any + * pre-host scrub work (the boot session's scrubbers). */ + s->submitted = put; + } else if (put != s->put_last) { + delta = (put + s->entries - s->put_last) % s->entries; + if (!delta) + delta = s->entries; /* exact wrap */ + s->submitted += delta; + s->put_last = put; + } else if (s->submitted == s->authored) { + return; /* nothing new; no guest writes */ + } + + if (s->log_budget < 64) { + s->log_budget++; + pr_info("sema-svc: scrub author put=%u submitted=%llu fin_pa=0x%llx\n", + put, s->submitted, s->fin_pa); + } + + /* + * Author (forward-only). The USERD GP_GET and host-sema writes mark + * the submissions consumed; the finish write releases the scrub + * wait. Written only when GP_PUT advanced (or on the seeding pass), + * so the per-pass cost is zero while the scrubber idles. + */ + svc_write_ap(vgpu, s->userd_pa + SVC_USERD_GP_GET, s->userd_as, + &put, sizeof(put)); + svc_write_ap(vgpu, s->hsem_pa, SVC_APERTURE_SYS, &put, sizeof(put)); + { + u32 fin = (u32)s->submitted; + + svc_write_ap(vgpu, s->fin_pa, SVC_APERTURE_SYS, &fin, + sizeof(fin)); + } + s->authored = s->submitted; +} + static int svc_thread(void *arg) { + struct nv_vgpu_device *vgpus[SVC_MAX_GUESTS]; struct nv_vgpu_device *vgpu; + int n_vgpus, g; pr_info("sema-svc: thread started\n"); @@ -15577,129 +24157,192 @@ static int svc_thread(void *arg) complete_all(&svc_sess_flush_done); } - vgpu = NULL; + /* + * m177: a guest RM (re)load means a new guest boot: wipe the + * boot-scoped discovery state here, on the thread that owns + * it. Raised by gsp_sema_service_guest_reset(). + */ + if (READ_ONCE(svc_guest_reset_req)) { + svc_guest_reset_req = 0; + svc_guest_reset_wipe(); + } + + /* + * STALL FIX (r189): expire withheld release sets whose entry + * stopped being re-decoded (the deadline flush is only + * reachable from the entry path). Forward-only delivery to a + * live owner; counted LOSS otherwise. Cheap: 8 slots. + */ + svc_opdef_expire_sweep(); + + /* + * Service EVERY open mdev each pass, in list order. With one + * mdev this is exactly the historical single-vgpu pass; with + * two, both guests get a pump turn per iteration (hardware + * forwarding stays serialized by the single-inflight slot; see + * the forwarding section). The list walk is the historical + * bounded scan under nv_vgpu_driver.lock. + */ + n_vgpus = 0; mutex_lock(&nv_vgpu_driver.lock); { struct nv_vgpu_device *d; list_for_each_entry(d, &nv_vgpu_driver.devices, list) { - if (atomic_read(&d->vfio_dev_open_count) > 0) { - vgpu = d; - break; - } + if (atomic_read(&d->vfio_dev_open_count) > 0 && + n_vgpus < SVC_MAX_GUESTS) + vgpus[n_vgpus++] = d; } } mutex_unlock(&nv_vgpu_driver.lock); - if (!vgpu) { + if (!n_vgpus) { msleep(500); continue; } - svc_handle_probe(vgpu); - svc_handle_walk_probe(vgpu); + /* Host-injection queue drain: once per pass, not per guest. */ svc_handle_map_add(); - svc_handle_chan_dump(vgpu); - if (svc_sess_flush || !svc.direct_base || - time_after(jiffies, - (unsigned long)(svc.last_build + 10 * HZ))) - svc_rebuild(vgpu); + for (g = 0; g < n_vgpus; g++) { + struct svc_guest_sess *sess; + u64 done_before; - /* - * Channel-driven execution: notice submissions as they are made - * and run the pushed work immediately, so the guest's UVM - * progress waits complete in milliseconds rather than in the - * time it takes the fallback scan to find the push. A doorbell - * token means a channel was just scheduled; run everything now. - */ - { - u64 done_before = svc.gpfifo_entries_done; + vgpu = vgpus[g]; + svc_handle_probe(vgpu); + svc_handle_walk_probe(vgpu); + svc_handle_chan_dump(vgpu); + svc_scrub_pass(vgpu); + /* + * Per-guest rebuild freshness: the discovery map is + * still shared between guests (staged G4), but each + * guest's RAM must be scanned on its own clock, or the + * first guest's 10 s tick would starve the second + * guest of rebuilds. The freshness stamp is the one + * svc_rebuild() publishes (svc.last_build). + */ + sess = svc_guest_get(vgpu, true); + if (svc_sess_flush || !svc.direct_base || + !sess || + time_after(jiffies, + (unsigned long)(sess->last_build + + 10 * HZ))) { + svc_rebuild(vgpu); + if (sess) + sess->last_build = svc.last_build; + } + + /* + * Channel-driven execution: notice submissions as they + * are made and run the pushed work immediately, so the + * guest's UVM progress waits complete in milliseconds + * rather than in the time it takes the fallback scan + * to find the push. A doorbell token means a channel + * was just scheduled; run everything now. + */ + done_before = svc.gpfifo_entries_done; svc.doorbell_pending = false; svc_pump_channels(vgpu); - pumped = (svc.gpfifo_entries_done != done_before); - } + if (svc.gpfifo_entries_done != done_before) + pumped = true; - /* - * Host CUDA bridge: marshal one queued launch into a daemon - * job and resolve completions. Inert until a launch is seen. - */ - gsp_cuda_bridge_service(vgpu); + /* + * Host CUDA bridge: marshal one queued launch into a + * daemon job and resolve completions. Inert until a + * launch is seen. + */ + gsp_cuda_bridge_service(vgpu); - /* - * Releases whose target VA was not translatable when the scan - * saw them: retry every pass, so a pin or a learned pair applies - * them immediately instead of on the next full sweep. - */ - svc_apply_pending(vgpu); + /* + * Releases whose target VA was not translatable when + * the scan saw them: retry every pass, so a pin or a + * learned pair applies them immediately instead of on + * the next full sweep. + */ + svc_apply_pending(vgpu); - /* - * A fresh VA->PA injection changes which releases are - * applicable: run the fast sliced sweep while the forced window - * is open (the pump's stall path also uses it). - */ - if (time_before(jiffies, (unsigned long)svc.fast_force_until)) - svc_apply_sweep_slice(vgpu); + /* + * A fresh VA->PA injection changes which releases are + * applicable: run the fast sliced sweep while the + * forced window is open (the pump's stall path also + * uses it). + */ + if (time_before(jiffies, + (unsigned long)svc.fast_force_until)) + svc_apply_sweep_slice(vgpu); + } /* * While the pump is making progress the client is streaming * pushes and waiting on their releases: loop tightly and skip - * the discovery/sweep passes, which otherwise dominate the pass - * period (the pump cadence is what the client's context init - * deadline sees). Scans resume the first pass without progress. + * the discovery/sweep passes, which otherwise dominate the + * pass period (the pump cadence is what the client's context + * init deadline sees). Scans resume the first pass without + * progress. (With two guests, any guest's progress keeps the + * loop tight; the next iteration pumps both.) */ if (pumped) { msleep(1); continue; } - /* - * Push-buffer base discovery: a chunked walk of guest RAM - * finds the pages whose first push names a channel, so a - * channel that appears with work already waiting is pinned and - * pumped without waiting for the rolling sweep. - */ - svc_pb_discover(vgpu); + for (g = 0; g < n_vgpus; g++) { + vgpu = vgpus[g]; - /* - * Completion hint: re-post the events for a few seconds after a - * completion, so a client that starts waiting after the - * edge-triggered wake-up still observes it. - */ - if (time_before(jiffies, (unsigned long)svc.post_hint_until) && - time_after(jiffies, (unsigned long)svc.last_hint + HZ / 2)) { - int rc; + /* + * Push-buffer base discovery: a chunked walk of guest + * RAM finds the pages whose first push names a + * channel, so a channel that appears with work + * already waiting is pinned and pumped without + * waiting for the rolling sweep. + */ + svc_pb_discover(vgpu); - svc.last_hint = jiffies; - rc = gsp_rm_post_all_events(vgpu); - if (gsp_sema_dbg) - pr_info_ratelimited("sema-svc: completion hint rc=%d\n", - rc); - } + /* + * Completion hint: re-post the events for a few + * seconds after a completion, so a client that starts + * waiting after the edge-triggered wake-up still + * observes it. + */ + if (time_before(jiffies, + (unsigned long)svc.post_hint_until) && + time_after(jiffies, + (unsigned long)svc.last_hint + HZ / 2)) { + int rc; + + svc.last_hint = jiffies; + rc = gsp_rm_post_all_events(vgpu); + if (gsp_sema_dbg) + pr_info_ratelimited("sema-svc: completion hint rc=%d\n", + rc); + } - /* - * Fast path for the pushes themselves: every mapped page is a - * candidate pushbuffer page, and there are far fewer of those - * than there are pages of guest RAM. - */ - if (svc.direct_base && svc.map_n) - svc_apply_mapped(vgpu); + /* + * Fast path for the pushes themselves: every mapped + * page is a candidate pushbuffer page, and there are + * far fewer of those than there are pages of guest + * RAM. + */ + if (svc.direct_base && svc.map_n) + svc_apply_mapped(vgpu); - /* - * Rolling RAM scan: the pushbuffers of some clients are not - * covered by the VA map, so sweep guest RAM in bounded windows - * in parallel with the targeted pass. Work per pass is fixed, - * which keeps the emulation responsive while a full sweep - * completes every few seconds. - */ - svc_apply_pushes(vgpu); + /* + * Rolling RAM scan: the pushbuffers of some clients + * are not covered by the VA map, so sweep guest RAM in + * bounded windows in parallel with the targeted pass. + * Work per pass is fixed, which keeps the emulation + * responsive while a full sweep completes every few + * seconds. + */ + svc_apply_pushes(vgpu); - /* - * Deferred readability probes for faked CE pushes (observation - * only; see svc_retry_poll). - */ - svc_retry_poll(vgpu); + /* + * Deferred readability probes for faked CE pushes + * (observation only; see svc_retry_poll). + */ + svc_retry_poll(vgpu); + } msleep(10); } @@ -15822,6 +24465,97 @@ void gsp_sema_service_stop(void) svc_vas_free_all(); } +/* + * Release every piece of service state owned by one mdev (dual-guest goal + * 1.0). Called from the mdev cleanup path after that mdev's emulation thread + * stopped and before its struct can be freed: + * - quiesce forwarding so no in-flight work item can still be reading the + * request's vgpu, and abort a release hold that names it; + * - drop any hold that survives (a request that never queued leaves no work + * item to clear it) BEFORE the device can go away - resolution would + * dereference it; + * - drop the guest's channels; request the global session flush only when no + * guest has live channels left (the discovery tables are still shared; + * staged G4); + * - drop the guest's held-operand release slots (its waiters are gone); + * - free the per-guest session and scrubber records so the vgpu pointer can + * never match a newly allocated device. + */ +void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu) +{ + unsigned long flags; + int i; + + if (!vgpu) + return; + + /* + * The forward work item runs on its own kthread and may be using the + * request's vgpu; quiesce waits for it (bounded by the submit's + * 200 ms poll cap) and aborts an active hold. Resume afterwards so the + * other guests keep forwarding. + */ + gsp_sema_service_work_quiesce(); + if (READ_ONCE(svc_fwd_hold.active) && svc_fwd_hold.vgpu == vgpu) + WRITE_ONCE(svc_fwd_hold.aborted, true); + gsp_sema_service_work_resume(); + + if (READ_ONCE(svc_fwd_hold.active) && svc_fwd_hold.vgpu == vgpu) { + /* + * Pure state clear (never a resolution through the vanishing + * device): the releases belong to a guest that is gone. + */ + svc.fwd_hold_canceled++; + svc_fwd_hold_clear(); + } + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *ch = &svc.chan[i]; + + if (!ch->valid || ch->vgpu != vgpu) + continue; + pr_info("sema-svc: channel 0x%x dropped (mdev cleanup; submissions=%llu)\n", + ch->h_object, ch->submissions); + memset(ch, 0, sizeof(*ch)); + } + if (svc_sess_active) { + struct svc_guest_sess *sess = svc_guest_get(vgpu, false); + bool any = false; + + for (i = 0; i < SVC_MAX_CHANNELS; i++) + if (svc.chan[i].valid) { + any = true; + break; + } + if (sess) + sess->active = false; + if (!any) { + svc_sess_active = 0; + svc_sess_flush = 1; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + + /* Held-operand release slots of this guest cannot be delivered. */ + for (i = 0; i < SVC_OPDEF_SLOTS; i++) + if (svc_opdef[i].active && svc_opdef[i].vgpu == vgpu) + svc_opdef_clear(&svc_opdef[i], SVC_OPDEF_CLR_STOP); + + /* Free the per-guest session and scrubber records. */ + spin_lock_irqsave(&svc_guest_lock, flags); + for (i = 0; i < SVC_MAX_GUESTS; i++) + if (svc_guests[i].vgpu == vgpu) + memset(&svc_guests[i], 0, sizeof(svc_guests[i])); + spin_unlock_irqrestore(&svc_guest_lock, flags); + for (i = 0; i < SVC_MAX_GUESTS; i++) + if (svc_scrubs[i].vgpu == vgpu) + memset(&svc_scrubs[i], 0, sizeof(svc_scrubs[i])); + + if (svc_sess_flush && svc.task) + wake_up_process(svc.task); +} + /* ------------------------------------------------------------------ */ /* host CUDA bridge exports (gsp_cuda_bridge.c) */ /* ------------------------------------------------------------------ */ @@ -16017,6 +24751,6 @@ int gsp_sema_service_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, * after the write still receives the completion event. */ if (!rc) - svc.post_hint_until = jiffies + 10 * HZ; + svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return rc; } diff --git a/kernel/gsp_sema_service.h b/kernel/gsp_sema_service.h index 4b04892..ee5c48a 100644 --- a/kernel/gsp_sema_service.h +++ b/kernel/gsp_sema_service.h @@ -18,8 +18,16 @@ * * All addresses are guest-physical except gpfifo_va, which is the GPU VA of * the channel's GPFIFO ring (translated through the discovered VA->PA map). + * + * vgpu names the owning mdev (guest session). Two guests' stock allocators + * choose the same object handles, so the channel table is keyed by + * (vgpu, h_object): a registration can never overwrite another guest's slot, + * and ranks/session bookkeeping are computed per guest. */ +struct nv_vgpu_device; + struct sema_channel_desc { + struct nv_vgpu_device *vgpu; /* owning mdev (guest session) */ u32 h_object; u32 engine_type; u32 cid; /* RM channel id (the doorbell token vector) */ @@ -33,7 +41,27 @@ struct sema_channel_desc { }; void gsp_sema_service_channel_add(const struct sema_channel_desc *desc); -void gsp_sema_service_channel_del(u32 h_object); +void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object); + +/* + * Release every piece of service state that belongs to one mdev before its + * guest session disappears: drop the mdev's channels (per-vgpu session end), + * clear a release hold that names it (never dereference a removed device), + * and free its per-guest session record. Called from the mdev cleanup path + * after the device's emulation thread stopped and before the mdev struct can + * be freed. No-op for a vgpu that never registered a channel. + */ +void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu); + +/* + * Register the guest PMA scrubber's channel-buffer words (the CeUtils + * finish payload the guest's scrub wait polls, and the channel's host + * semaphore). The PAs come from the guest scanner (vmallocinfo + kcore): + * the guest RM kernel channel lives in a GPU VA space that collides + * numerically with the client's CE window in the global translation map, + * so it cannot be resolved through that map. + */ +int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa); /* * A GPU VA space whose top-level page directory the driver has published (the @@ -53,6 +81,18 @@ struct sema_vaspace_desc { void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc); void gsp_sema_service_vaspace_del(u32 h_vaspace); +/* + * r496j: park a page-directory candidate (from COPY_SERVER_PDES) until a + * content-validated walk proves it belongs to a live client; only then is + * it registered (a blind registration let stale kernel directories answer + * other clients' VAs in the walker's first-success order). + */ +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem); +/* m177: guest RM reload = new guest boot; wipe boot-scoped discovery state + * (direct-map base, content-scanned tables) on the svc thread. Client + * session flushes keep that state - see svc_session_flush. */ +void gsp_sema_service_guest_reset(void); int gsp_sema_service_start(void); void gsp_sema_service_stop(void); @@ -235,4 +275,162 @@ extern unsigned gsp_sema_push_hold_ms; */ extern unsigned gsp_sema_gr_grace_ms; +/* + * GRFIX FIX1b (r213): absolute per-launch bound (ms) for the GR + * data-stalled hold. Keyed by the launch push VA (slot indices are not + * stable across pin-batch rewinds); first hold fixes the deadline, nothing + * re-arms it, and when it passes the launch proceeds (the pre-8b058884 + * behaviour) so the guest can never wait unboundedly. + */ +extern unsigned gsp_phys_gr_hold_max_ms; + +/* + * GRFIX FIX1e (r215): master gate for the GR fill hold (default 0 = + * log-only, never hold). At 0 the fill logs dropped operand pages + * (GRFILL-DROP) and takes the historical forward path immediately; the + * GRFILL-HOLD/GRFILL-EXPIRED machinery is skipped. At 1 the page-count + * hold and its absolute bound apply. + */ +extern unsigned gsp_phys_gr_fill_hold; +extern unsigned gsp_sema_walk_trace; + +/* + * r365: execute fill-shaped CE transfers to VIDMEM-window VAs (the guest's + * page-table writes). Off by default; see the definition in the .c. + */ +extern bool gsp_sema_winfill; + +/* + * m164 CLASS-2: hold window (ms) for a GR launch push whose decode saw QMD + * words but built no launch candidate; 0 disables the hold. + */ +extern unsigned gsp_sema_gr_nocap_hold_ms; + +/* + * m165 (D-F2a): skip the post-registration pool hold when a verified + * injected pin exists for the VA (default on); 0 restores the blanket 5 s. + */ +extern bool gsp_sema_pool_hold_fresh; + +/* + * m169 (AGENT-F R1): hold device-destination copies translated only by a + * walk/RAM scan until a pin confirms (default on). + */ +extern bool gsp_sema_copy_dst_pin_guard; +extern bool gsp_sema_wait_gate; +/* r401: bounded acquire-wait hold (fail-open) in ms; must exceed the + * emulated launch completion latency for copy-after-launch ordering. */ +extern unsigned gsp_sema_wait_hold_ms; +/* r481: verbatim method-stream dump for the first N large GR pushes + * (default 0 off). Diagnostic only; no behaviour change. */ +extern unsigned gsp_sema_push_raw; + +/* + * r384 (guest boot-corruption lottery): fill-destination guards (default + * on). Large virtual fills whose only authority is the walk/scan resolver + * are held through a short bounded window while their VA is demanded + * through the pin channel (fail-open after gsp_sema_fill_guard_ms); zero + * fills targeting a page-table-shaped guest page are refused loudly. + */ +extern bool gsp_sema_fill_guard; +extern unsigned gsp_sema_fill_guard_ms; + +/* + * r231 READ-ORDER guard (default 0: witness only). Defer a device-src copy + * whose source VA has a pending device-dst write hold (READ-ORDER-WIT) so + * the read re-decodes after the write resolves. Opt-in pending an A/B. + */ +extern bool gsp_sema_read_order_guard; + +/* + * COPYWIT fail4: per-boot line budget for the sema-bigcopy resolution + * witnesses (svc_copy's large-copy VA -> pa proof). Raise it for a + * discrimination cycle so the failing clients' H2D/D2H resolutions are + * visible; 0 disables the lines. + */ +extern unsigned gsp_sema_bigcopy_log_max; + +/* + * F1-b: authorship gate for the GR completion release the forwarded launch's + * poll waits on. When on (default), a RAM-sweep release whose VA equals the + * in-flight forward request's gr_rel_va is deferred (skipped, not applied) + * while the hardware copy is still in flight, so the poll can only break on + * an engine-authored word; the deferred release is re-seen by later sweep + * passes and the hold/fallback paths apply it if the hardware copy fails. + * 0 restores the unconditional sweep apply. + */ +extern bool gsp_sema_gr_swrel_gate; + +/* + * relfix (agent-A FIX A): rewrite-time GR relocation resolver. Resolves one + * device VA through the same trusted source chain the GR fill's per-page + * resolution uses (ipin -> self pin -> learned cache -> discovered map -> + * published vaspaces -> discovered directory), then the m180 ledger in the + * fill's two roles (fallback when nothing else answered; m171 landing + * preference above the ledger floor when it disagrees). The result is + * converted to a host page the caller can copy/map. Deliberately read-only + * w.r.t. the resolver tables - no pin or cache stores, no directory + * discovery, no targeted RAM scan: the caller is the GR work item, not the + * service thread, and must not write service state or block on a full-RAM + * scan. The page-table walks may sleep in vfio_dma_rw exactly like the + * fill's walks; callers run in process context (the GR work item, holding + * gsp_phys_rm_lock) with no spinlocks held. A live forwarded GR request + * (svc_fwd_req.vgpu) is required. A successful return (0) means the VA is a + * translated page of that request's client - the caller may act on it; a + * miss returns -ENOENT and the caller must change nothing. + * + * `stage` carries the per-source diagnostic chain of THIS call + * ("ipin:-2,pin:-2,...,ledger:-2," with the resolving source ending in + * ":0,") and `stage_rc` the overall result (0 on success, else the miss + * errno). relfix PART 3: the booter's RELOC-GAP-STAGE line prints them for + * words that stayed unrelocated, so the failing source is on the record. + */ +struct gsp_sema_gr_reloc { + u64 pa; /* resolved guest physical page base */ + u32 ap; /* aperture: 1 = SYS (guest RAM), 2 = FB (vidmem) */ + u64 hpa; /* host physical address of that guest page */ + char stage[96]; /* per-attempt "name:rc," source chain (diagnostic) */ + int stage_rc; /* 0 on success; else the failure errno */ +}; + +int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out); + +/* + * r378 write guards. Both re-validate a guest-memory write target that was + * resolved earlier (ledger/pin/walk/counter registration, or the GR submit's + * per-page resolution) against the CURRENT resolution before the bytes land, + * and return false when a positive disagreement proves the target stale: + * + * gsp_sema_gr_write_guard(va, hpa, path): GR backend pages. A fresh + * injected pin is checked first (the r372/r376 pin-wins rule - the guest's + * newest statement), then the trusted resolver chain for a pin-less VA. + * The submit-to-copy-out window is seconds wide (the GR poll), and a guest + * buffer freed/re-homed in it would otherwise receive a full page of + * another buffer's content (the boot-corruption class). The landing table + * is deliberately not the authority: our own launches feed it, so it would + * confirm the target by construction. + * + * gsp_sema_write_guard(va, pa, ap, path): service-side SYS/FB targets. A + * fresh injected pin is the guest's newest statement about the VA; a pin + * that disagrees with the recorded target means the page moved (the + * r372/r376 pin-wins rule), so the stale write is skipped. + * + * Both fail OPEN when no current answer exists (the recorded landing is then + * the write authority) and log every skip (bounded, 256 lines) with the path + * and the VA/target/current values. + */ +bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path); +bool gsp_sema_write_guard(u64 va, u64 pa, u32 ap, const char *path); + +/* + * r391 diagnostic (sum=0 class): resolve a GR data VA through the landing + * table (the exact source the D2H reader's ledger-first branch uses) and + * return the resolution plus the first 8 bytes read through it. The GR + * copy-out prints it next to the write target it used, so a launch/readback + * disagreement (the write went to another page/aperture than the reader + * resolves) is one line instead of a forensics session. + */ +bool gsp_sema_gr_va_snap(u64 va, u64 *pa, u32 *ap, u64 *val8); +u8 *gsp_sema_gr_shadow_base(void); + #endif /* _GSP_SEMA_SERVICE_H_ */ diff --git a/kernel/main.c b/kernel/main.c index ac970f1..3ee5c23 100644 --- a/kernel/main.c +++ b/kernel/main.c @@ -1740,6 +1740,16 @@ static void nv_vgpu_device_cleanup_work(struct work_struct *work) gsp_emu_cleanup(vgpu); /* + * Dual-guest (goal 1.0): release everything the sema service still holds + * for this mdev (channel entries, session/scrubber records, a release + * hold that names it - quiescing any in-flight forwarding work) and drop + * its per-mdev guest-RAM record, BEFORE the vgpu struct can be freed: a + * stale pointer in the service's tables must never match a later device. + */ + gsp_sema_service_vgpu_cleanup(vgpu); + gsp_phys_guest_ram_vgpu_clear(vgpu); + + /* * Last-reference backend teardown: gsp_emu_cleanup() above has stopped * this mdev's emulation thread, so no forward can be in flight when the * claim is released. The backend session stays up until the last mdev @@ -2544,6 +2554,37 @@ module_param_cb(sema_probe, &nv_vgpu_sema_probe_ops, NULL, 0200); MODULE_PARM_DESC(sema_probe, "Dump sema-service map + slot values for a GPU VA"); /* + * Register the guest PMA scrubber channel buffer words (finish payload PA, + * host-semaphore PA) the guest scanner resolved: + * echo 0x11a8ff004,0x11a8ff000 > .../parameters/sema_scrubbuf_add + */ +static int nv_vgpu_sema_scrubbuf_set(const char *val, + const struct kernel_param *kp) +{ + char buf[64]; + char *comma; + u64 fin_pa = 0, hsem_pa = 0; + + strscpy(buf, val, sizeof(buf)); + comma = strchr(buf, ','); + if (!comma) + return -EINVAL; + *comma = '\0'; + if (kstrtoull(buf, 0, &fin_pa) || kstrtoull(comma + 1, 0, &hsem_pa)) + return -EINVAL; + + return gsp_sema_service_scrubbuf_add(fin_pa, hsem_pa); +} + +static const struct kernel_param_ops nv_vgpu_sema_scrubbuf_ops = { + .set = nv_vgpu_sema_scrubbuf_set, +}; + +module_param_cb(sema_scrubbuf_add, &nv_vgpu_sema_scrubbuf_ops, NULL, 0200); +MODULE_PARM_DESC(sema_scrubbuf_add, + "Register the PMA scrubber finish/host-sema PAs (hex fin,hex hsem)"); + +/* * Debug: run the page-table walks for a GPU VA with per-level tracing, * bypassing the pin/lat/pair map (see gsp_sema_service_walk_probe): * echo 0x20400f000 > .../parameters/walk_probe @@ -2730,6 +2771,104 @@ module_param_named(sema_gr_grace_ms, gsp_sema_gr_grace_ms, uint, 0644); MODULE_PARM_DESC(sema_gr_grace_ms, "GR pending-entry grace window in ms (default 8000)"); /* + * GRFIX FIX1b (r213): absolute per-launch bound in ms for the GR fill + * data-stalled hold (the FIX1 page-drop hold). The deadline is keyed by the + * launch push VA, fixed on the first hold and never re-armed; when it passes + * the launch proceeds (forwarded best-effort) so a never-resolving page can + * never block the guest. 0 keeps the 3000 ms default. + */ +module_param_named(gsp_phys_gr_hold_max_ms, gsp_phys_gr_hold_max_ms, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_hold_max_ms, "Absolute GR fill hold bound in ms, keyed by launch push VA (default 3000; launch proceeds when it passes)"); + +/* + * GRFIX FIX1e (r215): master gate for the GR fill hold. At 0 (default) the + * fill logs dropped operand pages and takes the historical forward path + * immediately - it can never hold or delay a launch. At 1 the page-count + * hold and the absolute bound apply. + */ +module_param_named(gsp_phys_gr_fill_hold, gsp_phys_gr_fill_hold, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_fill_hold, "GR fill hold on unresolved operand pages (default 0: log-only, never hold)"); +module_param_named(gsp_sema_walk_trace, gsp_sema_walk_trace, uint, 0644); +MODULE_PARM_DESC(gsp_sema_walk_trace, "Trace GMMU walks for unresolved high-VA operands (default 0: off)"); +module_param_named(gsp_sema_winfill, gsp_sema_winfill, bool, 0644); +MODULE_PARM_DESC(gsp_sema_winfill, "Execute fill-shaped CE transfers to VIDMEM-window VAs (default 0: off)"); + +/* + * F1-b: while a forwarded GR launch is in flight, defer (do not apply) a + * RAM-sweep release whose VA is that launch's QMD completion word, so the + * backend poll can only break on an engine-authored ack. Deferred releases + * are re-seen by later sweep passes; the hold/fallback path applies them when + * the hardware copy fails. 0 restores the unconditional sweep apply. + */ +module_param_named(gsp_sema_gr_swrel_gate, gsp_sema_gr_swrel_gate, bool, 0644); +MODULE_PARM_DESC(gsp_sema_gr_swrel_gate, "Defer a sweep-authored in-flight GR release word (default 1; 0 = unconditional sweep apply)"); + +/* + * m164 CLASS-2: hold window (ms) for a readable GR launch push whose decode + * saw QMD words but built no launch candidate (replay-suppressed capture, + * partial/split QMD, or an unreadable push page). The entry stays unconsumed + * for this window and the next pass forces a fresh capture; on expiry the + * historical consume applies. 0 disables. + */ +module_param_named(sema_gr_nocap_hold_ms, gsp_sema_gr_nocap_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_gr_nocap_hold_ms, "GR no-capture entry hold in ms (default 6000, 0 disables)"); + +/* + * m165 (D-F2a): skip the post-registration pool-release hold when a + * host-verified injected pin for the VA exists (default on). 0 restores the + * legacy blanket 5 s hold. + */ +module_param_named(sema_pool_hold_fresh, gsp_sema_pool_hold_fresh, bool, 0644); +MODULE_PARM_DESC(sema_pool_hold_fresh, "apply pool releases at fresh-pin time instead of registration+5s (default 1)"); + +/* + * m169 (AGENT-F R1): hold device-destination copies translated only by a + * page-directory walk until a pin confirms (default on). A wrong-but-valid + * stale table entry silently writes the wrong vidmem page; the hold lets + * the m163 retry path re-resolve with pins. + */ +module_param_named(sema_copy_dst_pin_guard, gsp_sema_copy_dst_pin_guard, bool, 0644); +module_param_named(sema_wait_gate, gsp_sema_wait_gate, bool, 0644); +MODULE_PARM_DESC(sema_wait_gate, "honor C56F host-semaphore acquires before running a ring entry's work (default 1 since r408; 0 restores pass-through)"); +module_param_named(sema_wait_hold_ms, gsp_sema_wait_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_wait_hold_ms, "bounded acquire-wait hold before fail-open, ms (r401, default 4000; must exceed the emulated launch latency)"); +module_param_named(sema_push_raw, gsp_sema_push_raw, uint, 0644); +MODULE_PARM_DESC(sema_push_raw, "r481 diagnostic: dump the decoded method stream of the first N large GR pushes (default 0 off)"); +MODULE_PARM_DESC(sema_copy_dst_pin_guard, "hold device-dst copies without a pin-resolved translation (default 1)"); + +/* + * r384 (guest boot-corruption lottery): fill-destination guards. A large + * virtual fill whose only authority is the walk/scan resolver is held for + * gsp_sema_fill_guard_ms while its VA is demanded through the pin channel + * (fail-open, mirrors sema_copy_dst_pin_guard); a zero fill targeting a + * page-table-shaped guest page is refused loudly (FILL-PT-REFUSED). + */ +module_param_named(sema_fill_guard, gsp_sema_fill_guard, bool, 0644); +MODULE_PARM_DESC(sema_fill_guard, "guard virtual fill destinations (pin-demand hold + page-table-shape refusal, default 1)"); +module_param_named(sema_fill_guard_ms, gsp_sema_fill_guard_ms, uint, 0644); +MODULE_PARM_DESC(sema_fill_guard_ms, "fill pin-demand hold window in ms (default 800)"); + +/* + * r231 READ-ORDER guard: defer a device-src copy whose source VA has a + * pending device-dst write hold so the read re-decodes after the write + * resolves (fixes the ctaid_probe RAWA upload/readback inversion). + * Default 0: witness only (READ-ORDER-WIT). Opt-in pending an A/B proving + * no green regression. + */ +module_param_named(sema_read_order_guard, gsp_sema_read_order_guard, bool, 0644); +MODULE_PARM_DESC(sema_read_order_guard, "defer device-src reads of a VA with a pending held device-dst write (default 0: witness only)"); + +/* + * COPYWIT fail4 (2026-10-18): per-boot budget for the sema-bigcopy resolution + * witnesses. svc_copy's large-copy proof logs both operands' resolved + * VA -> pa/aperture/provenance, which is the read side of the client data + * path; raise this for a failing-client discrimination cycle so the lines + * survive past the boot's head. 0 disables. + */ +module_param_named(sema_bigcopy_log, gsp_sema_bigcopy_log_max, uint, 0644); +MODULE_PARM_DESC(sema_bigcopy_log, "sema-bigcopy resolution witness budget per boot (default 24; 0 disables)"); + +/* * Debug: inject a verified {GPU VA -> guest PA} translation into the * semaphore service (used when the client's mapping cannot be discovered * from guest structures): diff --git a/tools/cuda-ab/frame_producer.cu b/tools/cuda-ab/frame_producer.cu new file mode 100644 index 0000000..609114c --- /dev/null +++ b/tools/cuda-ab/frame_producer.cu @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * frame_producer.cu - render an animated frame on the guest GPU and publish + * it into the ivshmem (kvmfr) BAR so the host can see a real GPU-rendered + * frame from the guest. + * + * Path: CUDA kernel -> device buffer -> host buffer -> ivshmem BAR + * (/sys/bus/pci/devices/0000:00:03.0/resource2, the 128 MiB kvmfr + * device QEMU backs with /dev/shm/kvmfr-<uuid>). + * + * Layout in the BAR (offset 0x100000, clear of the kvmfr protocol header): + * struct frame_hdr { magic, version, width, height, stride, format, seq } + * then width*height*4 bytes of RGBA8. + * + * Usage: sudo ./frame_producer [frames] [width] [height] [fps] + * Prints: FRAME seq=<n> ms=<render+copy ms> (one line per frame, first 5 and + * every 30th) and FRAME_DONE frames=<n>. + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <ctime> +#include <fcntl.h> +#include <sys/mman.h> +#include <unistd.h> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + return 1; \ + } \ + } while (0) + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" little-endian-ish */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +__global__ void render_julia(uchar4 *fb, int w, int h, float t) +{ + int x = blockIdx.x * blockDim.x + threadIdx.x; + int y = blockIdx.y * blockDim.y + threadIdx.y; + + if (x >= w || y >= h) + return; + + float fx = (x - w * 0.5f) / (h * 0.45f); + float fy = (y - h * 0.5f) / (h * 0.45f); + float cr = 0.7885f * cosf(t); + float ci = 0.7885f * sinf(t); + float zr = fx, zi = fy; + int it = 0; + const int maxit = 96; + + while (zr * zr + zi * zi < 4.0f && it < maxit) { + float nzr = zr * zr - zi * zi + cr; + float nzi = 2.0f * zr * zi + ci; + + zr = nzr; + zi = nzi; + it++; + } + { + float v = (float)it / (float)maxit; + unsigned char r = (unsigned char)(255.0f * v); + unsigned char g = (unsigned char)(255.0f * (1.0f - v) * v * 4.0f); + unsigned char b = (unsigned char)(255.0f * (1.0f - v)); + unsigned char a = 255; + + fb[y * w + x] = make_uchar4(r, g, b, a); + } +} + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +int main(int argc, char **argv) +{ + int frames = argc > 1 ? atoi(argv[1]) : 60; + int w = argc > 2 ? atoi(argv[2]) : 800; + int h = argc > 3 ? atoi(argv[3]) : 600; + int fps = argc > 4 ? atoi(argv[4]) : 10; + const char *bar_path = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + size_t fb_bytes = (size_t)w * h * 4; + uchar4 *d_fb = NULL, *h_fb = NULL; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + int fd, i; + + setvbuf(stdout, NULL, _IONBF, 0); + + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + printf("OPEN_FAIL %s\n", bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (bar == MAP_FAILED) { + printf("MMAP_FAIL\n"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + CHECK(cudaMalloc(&d_fb, fb_bytes)); + h_fb = (uchar4 *)malloc(fb_bytes); + if (!h_fb) { + printf("HOST_ALLOC_FAIL\n"); + return 1; + } + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + + for (i = 0; i < frames; i++) { + uint64_t t0 = now_ns(); + float phase = (float)i * 0.15f; + dim3 block(16, 16); + dim3 grid((w + 15) / 16, (h + 15) / 16); + + render_julia<<<grid, block>>>(d_fb, w, h, phase); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_fb, d_fb, fb_bytes, + cudaMemcpyDeviceToHost)); + memcpy(bar + FRAME_OFFSET + sizeof(struct frame_hdr), h_fb, + fb_bytes); + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + if (i < 5 || (i % 30) == 29) + printf("FRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d %dx%d\n", frames, w, h); + CHECK(cudaFree(d_fb)); + free(h_fb); + munmap(bar, 128UL << 20); + close(fd); + return 0; +} diff --git a/tools/cuda-ab/inplace_scale.cu b/tools/cuda-ab/inplace_scale.cu new file mode 100644 index 0000000..e83a039 --- /dev/null +++ b/tools/cuda-ab/inplace_scale.cu @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * inplace_scale.cu - minimal in-place scale repro for the streams2 + * "scale wrote 1 page" class. + * + * One H2D, one in-place scale launch (c[i] *= 2 over 128 KB = 32 pages), + * one D2H, per-page diff. Run it repeatedly: if the launch writes only + * part of the buffer the per-page report names how many 4 KB blocks + * changed and the host log carries the launch's retire wait. + * + * Build: nvcc -O2 --cudart=static -o inplace_scale inplace_scale.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +int main(void) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *h, *d; + int i, bad = 0, first = -1, last = -1; + int nb = (n + 1023) / 1024, blk_bad = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + h = (float *)malloc(bytes); + if (!h) + return 1; + for (i = 0; i < n; i++) + h[i] = (float)(i % 1000) * 0.5f; + + CHECK(cudaMalloc(&d, bytes)); + CHECK(cudaMemcpy(d, h, bytes, cudaMemcpyHostToDevice)); + printf("PTR d=%p\n", (void *)d); + + scale<<<(n + 255) / 256, 256>>>(d, 2.0f, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + + CHECK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost)); + for (i = 0; i < n; i++) { + float exp = (float)(i % 1000) * 0.5f * 2.0f; + + if (h[i] != exp) { + bad++; + if (first < 0) + first = i; + last = i; + } + } + for (i = 0; i < nb; i++) { + int j, hit = 0; + + for (j = i * 1024; j < (i + 1) * 1024 && j < n; j++) + if (h[j] != (float)(j % 1000) * 0.5f * 2.0f) { + hit = 1; + break; + } + if (hit) + blk_bad++; + } + printf("SCALE n=%d bad=%d first=%d last=%d blocks=%d/%d\n", + n, bad, first, last, blk_bad, nb); + printf("STATUS %s\n", bad ? "FAIL" : "OK"); + return bad ? 1 : 0; +} diff --git a/tools/cuda-ab/ip_probe.cu b/tools/cuda-ab/ip_probe.cu new file mode 100644 index 0000000..366fbc8 --- /dev/null +++ b/tools/cuda-ab/ip_probe.cu @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe.cu - how many CTAs actually executed for an in-place kernel? + * + * scale_mark marks progress[blockIdx.x]=0xCAFEBABE after the c store; D2H of + * progress counts executed CTAs exactly. c (the in-place buffer) is read back + * and verified. seed (argv[1]) makes the run content-unique. + * + * Build: nvcc -O2 --cudart=static -o ip_probe ip_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_mark(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g grid=%d\n", prop.name, (double)k, grid); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + /* in-place scale with CTA markers */ + scale_mark<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0, first_missing = -1; + + for (i = 0; i < grid; i++) { + if (h_prog[i] == 0xCAFEBABEu) + marked++; + else if (first_missing < 0) + first_missing = i; + } + printf("CTA-MARK marked=%d/%d first_missing=%d\n", marked, grid, + first_missing); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + if (first >= 0) + printf("SCALE firstbad got=%g want=%g\n", hc[first], + (ha[first] + hb[first]) * 2.0f); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe2.cu b/tools/cuda-ab/ip_probe2.cu new file mode 100644 index 0000000..a5fa2ec --- /dev/null +++ b/tools/cuda-ab/ip_probe2.cu @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe2.cu - why do in-place c stores vanish while prog stores land? + * + * After a vadd that fills c, three variants run on the SAME c: + * 1. scale_if : if (i<n) c[i] *= s; (control, like ip_probe) + * 2. scale_u : c[i] *= s; (unconditional, grid exact) + * 3. fill_c : c[i] = 7.0f; (store only, no load) + * 4. add_c : c[i] = c[i] + 1.0f; (load + store) + * Each stage reads c back and compares against the expected value at that + * stage. A separate prog buffer marks CTA execution. + * + * Build: nvcc -O2 --cudart=static -o ip_probe2 ip_probe2.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_if(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +__global__ void scale_u(float *c, float s, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xDEADBEEFu; +} + +__global__ void fill_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = 7.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xFEEDFACu; +} + +__global__ void add_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] + 1.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0x0DDF00Du; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i, stage; + float mult; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + mult = 2.0f; + for (stage = 0; stage < 4; stage++) { + unsigned mark = 0; + + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + switch (stage) { + case 0: + scale_if<<<grid, 256>>>(dc, 2.0f, n, d_prog); + mult = 2.0f; + mark = 0xCAFEBABEu; + break; + case 1: + scale_u<<<grid, 256>>>(dc, 2.0f, d_prog); + mult = 4.0f; + mark = 0xDEADBEEFu; + break; + case 2: + fill_c<<<grid, 256>>>(dc, d_prog); + mark = 0xFEEDFACu; + break; + default: + add_c<<<grid, 256>>>(dc, d_prog); + mult = -1.0f; + mark = 0x0DDF00Du; + break; + } + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0; + + for (i = 0; i < grid; i++) + if (h_prog[i] == mark) + marked++; + printf("S%d CTA-MARK marked=%d/%d\n", stage + 1, + marked, grid); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + if (stage < 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) { + b++; + if (first < 0) + first = i; + } + printf("S%d readback bad=%d/%d first=%d\n", + stage + 1, b, n, first); + } else if (stage == 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 7.0f) { + b++; + if (first < 0) + first = i; + } + printf("S3 readback bad=%d/%d first=%d (fill 7.0)\n", + b, n, first); + } else { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 8.0f) { + b++; + if (first < 0) + first = i; + } + printf("S4 readback bad=%d/%d first=%d (add 1.0 to fill)\n", + b, n, first); + } + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe3.cu b/tools/cuda-ab/ip_probe3.cu new file mode 100644 index 0000000..a50df0e --- /dev/null +++ b/tools/cuda-ab/ip_probe3.cu @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe3.cu - what does the SM actually READ for the scale kernel? + * + * scale_report writes, per CTA: prog[2b] = bits of the float param s, + * prog[2b+1] = bits of c[b*256] (the load). + * The D2H of prog then shows exactly what the SM saw, independent of any + * store-visibility path for c. + * + * Build: nvcc -O2 --cudart=static -o ip_probe3 ip_probe3.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_report(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) { + prog[2 * blockIdx.x] = __float_as_uint(s); + prog[2 * blockIdx.x + 1] = __float_as_uint(c[blockIdx.x * 256]); + } +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[256]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 256 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 256 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + scale_report<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 256 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int s_ok = 0, s_1 = 0, c_ok = 0, n_seen = 0; + unsigned first_s = 0, first_c = 0; + + for (i = 0; i < grid; i++) { + unsigned sb = h_prog[2 * i]; + unsigned cb = h_prog[2 * i + 1]; + + if (i == 0) { + first_s = sb; + first_c = cb; + } + if (sb == 0x40000000u) + s_ok++; + if (sb == 0x3F800000u) + s_1++; + { + float want = ha[i * 256] + hb[i * 256]; + + if (cb == *(unsigned *)&want) + c_ok++; + } + n_seen++; + } + printf("REPORT ctas=%d s_eq_2.0=%d s_eq_1.0=%d c_load_ok=%d first_s=0x%08x first_c=0x%08x\n", + n_seen, s_ok, s_1, c_ok, first_s, first_c); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/launch_count.cu b/tools/cuda-ab/launch_count.cu new file mode 100644 index 0000000..02db1d6 --- /dev/null +++ b/tools/cuda-ab/launch_count.cu @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: BSD-3-Clause +// launch_count.cu - single-launch execution-count repro (r342). +// +// One launch, one atomic increment. The engine-written page witness showed the +// increment applied TWICE (1 -> 2; the 100-increment variant -> 200): every +// forwarded launch executes its engine work twice (one submission, one ring +// publish, completion OK). This is the root cause of the multi-pass content +// class (victory_reduce exactly 2x, scale/saxpy/smiley/multikernel wrong). +// Run in the guest; expected LAUNCH_COUNT=1. +// for a single launch, the launch path executes work more than once. +#include <cstdio> +#include <cuda_runtime.h> +__global__ void inc(unsigned long long *c) { if (threadIdx.x == 0 && blockIdx.x == 0) atomicAdd(c, 1ULL); } +int main(void) { + unsigned long long *d; unsigned long long h = 0; + if (cudaMalloc(&d, 8) != cudaSuccess) { printf("CUDA_ERROR malloc\nSTATUS FAIL\n"); return 1; } + if (cudaMemset(d, 0, 8) != cudaSuccess) { printf("CUDA_ERROR memset\nSTATUS FAIL\n"); return 1; } + inc<<<1, 32>>>(d); + if (cudaGetLastError() != cudaSuccess) { printf("CUDA_ERROR launch\nSTATUS FAIL\n"); return 1; } + if (cudaDeviceSynchronize() != cudaSuccess) { printf("CUDA_ERROR sync\nSTATUS FAIL\n"); return 1; } + if (cudaMemcpy(&h, d, 8, cudaMemcpyDeviceToHost) != cudaSuccess) { printf("CUDA_ERROR d2h\nSTATUS FAIL\n"); return 1; } + printf("LAUNCH_COUNT=%llu (expect 1)\n", h); + printf("STATUS %s\n", h == 1 ? "OK" : "FAIL"); + return h == 1 ? 0 : 1; +} diff --git a/tools/cuda-ab/memset2d_probe.cu b/tools/cuda-ab/memset2d_probe.cu new file mode 100644 index 0000000..acf43d4 --- /dev/null +++ b/tools/cuda-ab/memset2d_probe.cu @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset2d_probe.cu - per-phase diagnostics for victory_memset2d. + * + * victory_memset2d reports one aggregate bad count (~12.5M) that mixes all + * three phases. This probe runs the SAME three phases and reports each + * phase's bad count, the first mismatching byte (offset, got, want), and a + * per-phase 1MB-chunk histogram so the failing phase and the failing region + * are attributable in one run. + * + * Phases: + * P1 memset : cudaMemset(d_img, 0xA5, pitch*h); D2H 2D readback + * P2 roundtrip: H2D 2D pattern; D2H 2D readback + * P3 memset2d : cudaMemset2D quadrant 0x3C; D2H 2D readback + * + * Output lines: + * DEVICE ... + * PHASE p=1 bad=<n> first_off=<i> got=<g> want=<w> + * PHIST p=1 mb=<m> ok=<n> bad=<n> + * MEMSET2D2 bad_total=<n> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define IMG_W 4096 +#define IMG_H 1024 +#define FILL_BYTE 0xA5 +#define QUAD_BYTE 0x3C + +static int phase_check(const char *tag, const unsigned char *back, + const unsigned char *want, size_t n, int p) +{ + size_t i, first = (size_t)-1; + int bad = 0, got = 0, wantv = 0; + size_t mb; + + for (i = 0; i < n; i++) { + if (back[i] != want[i]) { + if (first == (size_t)-1) { + first = i; + got = back[i]; + wantv = want[i]; + } + bad++; + } + } + printf("PHASE p=%d bad=%d first_off=%zu got=0x%02x want=0x%02x\n", + p, bad, first, got, wantv); + if (bad) { + size_t nb = (n + (1u << 20) - 1) >> 20; + + for (mb = 0; mb < nb && mb < 8; mb++) { + size_t lo = mb << 20, hi = lo + (1u << 20); + size_t j; + int bok = 0, bbad = 0; + + if (hi > n) + hi = n; + for (j = lo; j < hi; j++) { + if (back[j] == want[j]) + bok++; + else + bbad++; + } + printf("PHIST p=%d mb=%zu ok=%d bad=%d\n", + p, mb, bok, bbad); + } + } + (void)tag; + return bad; +} + +int main(void) +{ + cudaDeviceProp prop; + const size_t w = IMG_W, h = IMG_H; + unsigned char *h_img, *h_back, *h_want; + unsigned char *d_img; + size_t pitch = 0; + int bad = 0, row, col; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_img = (unsigned char *)malloc(w * h); + h_back = (unsigned char *)malloc(w * h); + h_want = (unsigned char *)malloc(w * h); + if (!h_img || !h_back || !h_want) + return 1; + + CHECK(cudaMallocPitch(&d_img, &pitch, w, h)); + printf("PITCH pitch=%zu\n", pitch); + printf("DPTRS h_img=%p h_back=%p h_want=%p d_img=%p\n", (void *)h_img, + (void *)h_back, (void *)h_want, (void *)d_img); + + /* P1: full cudaMemset over the pitched allocation, 2D readback. */ + CHECK(cudaMemset(d_img, FILL_BYTE, pitch * h)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + memset(h_want, FILL_BYTE, w * h); + bad += phase_check("P1", h_back, h_want, w * h, 1); + + /* P2: pitched H2D pattern -> D2H round trip. */ + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) + h_img[row * w + col] = (unsigned char)((row + col) & 0xFF); + CHECK(cudaMemcpy2D(d_img, pitch, h_img, w, w, h, + cudaMemcpyHostToDevice)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P2", h_back, h_img, w * h, 2); + + /* P3: cudaMemset2D of the upper-left quadrant only. */ + CHECK(cudaMemset2D(d_img, pitch, QUAD_BYTE, w / 2, h / 2)); + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) { + unsigned char expect = h_img[row * w + col]; + + if (row < (int)(h / 2) && col < (int)(w / 2)) + expect = QUAD_BYTE; + h_want[row * w + col] = expect; + } + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P3", h_back, h_want, w * h, 3); + + printf("MEMSET2D2 bad_total=%d\n", bad); + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/memset_probe.cu b/tools/cuda-ab/memset_probe.cu new file mode 100644 index 0000000..a6a12e0 --- /dev/null +++ b/tools/cuda-ab/memset_probe.cu @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset_probe.cu - isolate the memcpy_roundtrip memset-phase failure. + * + * cudaMemset(8 MB, 0x5A) -> D2H -> byte verify, with a run-length report: + * the first bad offset, and a coverage map at 4 KB granularity (correct / + * wrong / untouched-malloc-prefill) so "fill under-delivered" is + * distinguishable from "readback delivered wrong pages". + * + * Build: nvcc -O2 --cudart=static -o memset_probe memset_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define PAGE 4096 +#define MemsetByte 0x5A + +int main(void) +{ + const size_t bytes = 8u << 20; + unsigned char *h; + unsigned char *d; + size_t i; + int bad = 0; + size_t first_bad = (size_t)-1; + size_t n_pages, p; + + setvbuf(stdout, NULL, _IONBF, 0); + h = (unsigned char *)malloc(bytes); + if (!h) + return 1; + memset(h, 0xEE, bytes); + + CHECK(cudaMalloc(&d, bytes)); + printf("DPTR d=%p bytes=%zu\n", (void *)d, bytes); + + CHECK(cudaMemset(d, MemsetByte, bytes)); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < bytes; i++) { + if (h[i] != (unsigned char)MemsetByte) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("FIRST_BAD off=%zu (0x%zx) bad=%d total=%zu\n", + first_bad, first_bad, bad, bytes); + n_pages = bytes / PAGE; + for (p = 0; p < n_pages; p++) { + int good = 1, prefill = 1; + + for (i = 0; i < PAGE; i++) { + if (h[p * PAGE + i] != (unsigned char)MemsetByte) + good = 0; + if (h[p * PAGE + i] != 0xEE) + prefill = 0; + if (!good && !prefill) + break; + } + if (!good) { + printf("PAGE %zu %s b0=%02x b1=%02x b2=%02x b3=%02x b4095=%02x\n", + p, prefill ? "PREFILL" : "OTHER", + h[p * PAGE], h[p * PAGE + 1], + h[p * PAGE + 2], h[p * PAGE + 3], + h[p * PAGE + 4095]); + if (p > 24) + break; + } + } + printf("STATUS %s\n", bad == 0 ? "OK" : "FAIL"); + return 0; +} diff --git a/tools/cuda-ab/push_battery.sh b/tools/cuda-ab/push_battery.sh new file mode 100755 index 0000000..644beba --- /dev/null +++ b/tools/cuda-ab/push_battery.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# push_battery.sh - push every CUDA battery binary from tools/cuda-ab/run/ +# to the running guest via scp over the SSH hostfwd (port 2222). +# Usage: push_battery.sh [guest_dir] (default /tmp) +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" +RUN="$HERE/run" +GUEST_DIR="${1:-/tmp}" +CLIENTS="victory_mail smiley vecadd_loop streams2 memcpy_roundtrip multikernel victory_saxpy victory_reduce victory_matmul victory_strided victory_memset2d victory_atomics victory_streams_events victory_math vector_add" +export SSHPASS="${VGPU_GUEST_PASS:-ubuntu}" +FILES="" +for c in $CLIENTS; do [ -x "$RUN/$c" ] && FILES="$FILES $RUN/$c"; done +[ -n "$FILES" ] || { echo "no binaries in $RUN"; exit 1; } +# shellcheck disable=SC2086 +sshpass -e scp -P "${VGPU_GUEST_PORT:-2222}" \ + -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o PubkeyAuthentication=no -o PreferredAuthentications=password \ + -o ConnectTimeout=15 $FILES "ubuntu@127.0.0.1:$GUEST_DIR/" && \ +sshpass -e ssh -p "${VGPU_GUEST_PORT:-2222}" \ + -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + -o PubkeyAuthentication=no -o PreferredAuthentications=password \ + "ubuntu@127.0.0.1" "chmod +x $GUEST_DIR/* 2>/dev/null; ls $GUEST_DIR | wc -l; echo BATTERY_PUSHED" diff --git a/tools/cuda-ab/rt_probe.cu b/tools/cuda-ab/rt_probe.cu new file mode 100644 index 0000000..025301d --- /dev/null +++ b/tools/cuda-ab/rt_probe.cu @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * rt_probe.cu - instrumented memcpy_roundtrip variant isolating the + * memset-phase D2H readback failure. + * + * Phases (argv[1]): + * "full" : H2D -> kernel -> D2H -> memset -> D2H (the original order) + * "sync" : same but cudaDeviceSynchronize() before the second D2H + * "memset" : memset -> D2H only (memset_probe shape, 3-buffer alloc) + * "rev" : memset -> D2H first, then H2D -> kernel -> D2H + * + * Forensics on the memset readback: first bad offset, byte histogram + * (0x5A / 0x00 / other), per-1MB chunk histogram, and the first 32 bytes. + * + * Build: nvcc -O2 --cudart=static -o rt_probe rt_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define MemsetByte 0x5A + +__global__ void bump(const float *in, float *out, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + out[i] = in[i] + 0.5f; +} + +static void memset_forensics(const unsigned char *h, size_t bytes, + const char *tag) +{ + size_t i, first_bad = (size_t)-1; + int bad = 0, n5a = 0, n00 = 0, nother = 0; + size_t chunk; + + for (i = 0; i < bytes; i++) { + if (h[i] == 0x5A) + n5a++; + else if (h[i] == 0x00) + n00++; + else + nother++; + if (h[i] != 0x5A) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("MEMSET-READ %s bad=%d first_bad=%zu n5a=%d n00=%d nother=%d\n", + tag, bad, first_bad, n5a, n00, nother); + printf("MEMSET-HEAD %s", tag); + for (i = 0; i < 32; i++) + printf(" %02x", h[i]); + printf("\n"); + if (first_bad != (size_t)-1) { + printf("MEMSET-ATBAD %s off=%zu:", tag, first_bad); + for (i = first_bad; i < first_bad + 16 && i < bytes; i++) + printf(" %02x", h[i]); + printf("\n"); + } + for (chunk = 0; chunk < bytes / (1u << 20); chunk++) { + size_t lo = chunk << 20, hi = lo + (1u << 20); + int c5a = 0, c00 = 0, cother = 0; + + for (i = lo; i < hi; i++) { + if (h[i] == 0x5A) + c5a++; + else if (h[i] == 0x00) + c00++; + else + cother++; + } + printf("MEMSET-MB %s mb=%zu 5a=%d 00=%d other=%d\n", + tag, chunk, c5a, c00, cother); + } +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "full"; + cudaDeviceProp prop; + const size_t bytes = 8u << 20; + const int n = (int)(bytes / sizeof(float)); + unsigned char *h_m; + float *h_in, *h_out; + float *d_in, *d_out; + unsigned char *d_m; + double csum = 0.0; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu mode=%s\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem, mode); + + h_in = (float *)malloc(bytes); + h_out = (float *)malloc(bytes); + h_m = (unsigned char *)malloc(bytes); + if (!h_in || !h_out || !h_m) + return 1; + + for (i = 0; i < n; i++) + h_in[i] = (float)(i % 1024) * 0.25f; + + CHECK(cudaMalloc(&d_in, bytes)); + CHECK(cudaMalloc(&d_out, bytes)); + CHECK(cudaMalloc(&d_m, bytes)); + printf("DPTRS d_in=%p d_out=%p d_m=%p\n", (void *)d_in, + (void *)d_out, (void *)d_m); + + if (!strcmp(mode, "memset") || !strcmp(mode, "rev")) { + /* memset phase first */ + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "early"); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync") || + !strcmp(mode, "rev")) { + CHECK(cudaMemcpy(d_in, h_in, bytes, cudaMemcpyHostToDevice)); + bump<<<grid, 256>>>(d_in, d_out, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < n; i++) { + if (h_out[i] != h_in[i] + 0.5f) + bad++; + csum += (double)h_out[i]; + } + printf("BUMP bad=%d csum=%.6f\n", bad, csum); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync")) { + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "late"); + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/run/egl_frame b/tools/cuda-ab/run/egl_frame Binary files differnew file mode 100755 index 0000000..130f032 --- /dev/null +++ b/tools/cuda-ab/run/egl_frame diff --git a/tools/cuda-ab/run/frame_producer b/tools/cuda-ab/run/frame_producer Binary files differnew file mode 100755 index 0000000..a202ac7 --- /dev/null +++ b/tools/cuda-ab/run/frame_producer diff --git a/tools/cuda-ab/run/inplace_scale b/tools/cuda-ab/run/inplace_scale Binary files differnew file mode 100755 index 0000000..bb47f42 --- /dev/null +++ b/tools/cuda-ab/run/inplace_scale diff --git a/tools/cuda-ab/run/ip4 b/tools/cuda-ab/run/ip4 Binary files differnew file mode 100755 index 0000000..54937e6 --- /dev/null +++ b/tools/cuda-ab/run/ip4 diff --git a/tools/cuda-ab/run/ip5 b/tools/cuda-ab/run/ip5 Binary files differnew file mode 100755 index 0000000..c4b4a39 --- /dev/null +++ b/tools/cuda-ab/run/ip5 diff --git a/tools/cuda-ab/run/ip6 b/tools/cuda-ab/run/ip6 Binary files differnew file mode 100755 index 0000000..56815c2 --- /dev/null +++ b/tools/cuda-ab/run/ip6 diff --git a/tools/cuda-ab/run/ip_probe b/tools/cuda-ab/run/ip_probe Binary files differnew file mode 100755 index 0000000..08fb86e --- /dev/null +++ b/tools/cuda-ab/run/ip_probe diff --git a/tools/cuda-ab/run/ip_probe2 b/tools/cuda-ab/run/ip_probe2 Binary files differnew file mode 100755 index 0000000..dfe6ab9 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe2 diff --git a/tools/cuda-ab/run/ip_probe3 b/tools/cuda-ab/run/ip_probe3 Binary files differnew file mode 100755 index 0000000..8a34c05 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe3 diff --git a/tools/cuda-ab/run/launch_count b/tools/cuda-ab/run/launch_count Binary files differnew file mode 100755 index 0000000..0c67f45 --- /dev/null +++ b/tools/cuda-ab/run/launch_count diff --git a/tools/cuda-ab/run/memset2d_probe b/tools/cuda-ab/run/memset2d_probe Binary files differnew file mode 100755 index 0000000..f5bd071 --- /dev/null +++ b/tools/cuda-ab/run/memset2d_probe diff --git a/tools/cuda-ab/run/memset_probe b/tools/cuda-ab/run/memset_probe Binary files differnew file mode 100755 index 0000000..bd9df5e --- /dev/null +++ b/tools/cuda-ab/run/memset_probe diff --git a/tools/cuda-ab/run/rt_probe b/tools/cuda-ab/run/rt_probe Binary files differnew file mode 100755 index 0000000..48d866f --- /dev/null +++ b/tools/cuda-ab/run/rt_probe diff --git a/tools/cuda-ab/run/scale_probe b/tools/cuda-ab/run/scale_probe Binary files differnew file mode 100755 index 0000000..ca3fc20 --- /dev/null +++ b/tools/cuda-ab/run/scale_probe diff --git a/tools/cuda-ab/run/streams2_probe b/tools/cuda-ab/run/streams2_probe Binary files differnew file mode 100755 index 0000000..206d6d6 --- /dev/null +++ b/tools/cuda-ab/run/streams2_probe diff --git a/tools/cuda-ab/run/streams2d b/tools/cuda-ab/run/streams2d Binary files differnew file mode 100755 index 0000000..90b6df8 --- /dev/null +++ b/tools/cuda-ab/run/streams2d diff --git a/tools/cuda-ab/run/streams_events_probe b/tools/cuda-ab/run/streams_events_probe Binary files differnew file mode 100755 index 0000000..00f3539 --- /dev/null +++ b/tools/cuda-ab/run/streams_events_probe diff --git a/tools/cuda-ab/run/tgemm_probe b/tools/cuda-ab/run/tgemm_probe Binary files differnew file mode 100755 index 0000000..d9c9aa3 --- /dev/null +++ b/tools/cuda-ab/run/tgemm_probe diff --git a/tools/cuda-ab/run/transformer b/tools/cuda-ab/run/transformer Binary files differnew file mode 100755 index 0000000..bcde39e --- /dev/null +++ b/tools/cuda-ab/run/transformer diff --git a/tools/cuda-ab/run/transformer_async b/tools/cuda-ab/run/transformer_async Binary files differnew file mode 100755 index 0000000..2911b5c --- /dev/null +++ b/tools/cuda-ab/run/transformer_async diff --git a/tools/cuda-ab/run/transformer_stages b/tools/cuda-ab/run/transformer_stages Binary files differnew file mode 100755 index 0000000..1461a3e --- /dev/null +++ b/tools/cuda-ab/run/transformer_stages diff --git a/tools/cuda-ab/run/wedge_ab b/tools/cuda-ab/run/wedge_ab Binary files differnew file mode 100755 index 0000000..23a90aa --- /dev/null +++ b/tools/cuda-ab/run/wedge_ab diff --git a/tools/cuda-ab/run/wedge_bisect b/tools/cuda-ab/run/wedge_bisect Binary files differnew file mode 100755 index 0000000..5898fcf --- /dev/null +++ b/tools/cuda-ab/run/wedge_bisect diff --git a/tools/cuda-ab/run/wedge_recover b/tools/cuda-ab/run/wedge_recover Binary files differnew file mode 100755 index 0000000..3d9351c --- /dev/null +++ b/tools/cuda-ab/run/wedge_recover diff --git a/tools/cuda-ab/scale_probe.cu b/tools/cuda-ab/scale_probe.cu new file mode 100644 index 0000000..8a0df59 --- /dev/null +++ b/tools/cuda-ab/scale_probe.cu @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * scale_probe.cu - localize the deterministic "in-place scale writes only the + * first 4 KB page" failure. + * + * Modes (argv[1]): + * V : vadd(a,b->c); sync; D2H+verify; vadd again; D2H+verify + * I : vadd(a,b->c); sync; D2H+verify; scale(c,*2 in place); sync; + * D2H+verify (with page map); scale again; sync; D2H+verify; D2H again + * O : vadd(a,b->c); sync; D2H+verify; scale_out(c->d,*2); sync; + * D2H+verify (with page map) + * + * Every stage also runs a GPU-side verify kernel that counts matching + * elements reading the device buffers (engine's own view) and writes the + * counts to a small buffer, D2H'd afterwards: separates "device pages never + * written" (GPU count low) from "D2H readback stale" (GPU count high, host + * count low). + * + * Build: nvcc -O2 --cudart=static -o scale_probe scale_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +__global__ void scale_out(const float *c, float *d, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + d[i] = c[i] * s; +} + +/* verify kernel: out[0]=count(c==a+b), out[1]=count(c==2*(a+b)) */ +__global__ void vdigest(const float *a, const float *b, const float *c, int n, + unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c[i] == a[i] + b[i]) + atomicAdd(&out[0], 1ull); + if (c[i] == (a[i] + b[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "V"; + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hd; + float *da, *db, *dc, *dd; + unsigned long long *d_out, h_out[2]; + int i; + float k = argc > 2 ? atof(argv[2]) : 0.0f; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + { + FILE *f = fopen("/proc/uptime", "r"); + double up = -1.0; + + if (f) { if (fscanf(f, "%lf", &up) != 1) up = -1.0; fclose(f); } + printf("UPTIME %.3f\n", up); + } + printf("DEVICE %s mode=%s seed=%g\n", prop.name, mode, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hd = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + memset(hd, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&dd, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da=%p db=%p dc=%p dd=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)dd); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S1 vadd readback bad=%d/%d\n", b, n); + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S1 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", h_out[0], + h_out[1]); + } + + if (!strcmp(mode, "V")) { + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S2 vadd-again readback bad=%d/%d\n", b, n); + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "I")) { + int stage; + + for (stage = 0; stage < 2; stage++) { + float mult = stage == 0 ? 2.0f : 4.0f; + + scale<<<grid, 256>>>(dc, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) + b++; + printf("S%d inplace-scale readback bad=%d/%d (mult=%g)\n", + 2 + stage, b, n, mult); + if (stage == 0) + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hc[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * mult) + pb++; + if (pb) + printf("S%d PAGE %d bad=%d/256 first=%g want=%g\n", + 2 + stage, page, + pb, hc[page * 256], + (ha[page * 256] + + hb[page * 256]) * mult); + } + } + if (stage == 0) { + CHECK(cudaMemset(d_out, 0, + 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", + h_out[0], h_out[1]); + } + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "O")) { + scale_out<<<grid, 256>>>(dc, dd, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hd, dd, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hd[i] != (ha[i] + hb[i]) * 2.0f) + b++; + printf("S2 outofplace-scale readback bad=%d/%d\n", b, n); + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hd[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * 2.0f) + pb++; + if (pb) + printf("S2 PAGE %d bad=%d/256\n", page, + pb); + } + } + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dd, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST d_eq_ab=%llu d_eq_2ab=%llu\n", h_out[0], + h_out[1]); + printf("STATUS DONE\n"); + return 0; + } + return 1; +} diff --git a/tools/cuda-ab/soak.sh b/tools/cuda-ab/soak.sh index f870fda..4f1430f 100755 --- a/tools/cuda-ab/soak.sh +++ b/tools/cuda-ab/soak.sh @@ -135,6 +135,19 @@ run_one() { r="$(gssh "cat $out 2>/dev/null")" || true [ -n "$r" ] && text="$r" printf '%s' "$text" | grep -q 'EXIT=' && break + # The guest wrapper's EXIT= marker can be lost when the guest's + # process reaping stalls (observed: the client reached STATUS OK + # but its zombie was never reaped and `timeout` never wrote the + # marker). A terminal STATUS line is proof of completion by + # itself - accept it after a short grace so the EXIT= marker + # still wins when it does arrive. + if printf '%s' "$text" | grep -qE 'STATUS (OK|FAIL)'; then + sleep 10 + r="$(gssh "cat $out 2>/dev/null")" || true + [ -n "$r" ] && text="$r" + printf '%s' "$text" | grep -q 'EXIT=' && break + break + fi tnow=$(date +%s) if [ $((tnow - t0)) -gt $((TMO + 120)) ]; then if client_alive "$bin"; then @@ -179,7 +192,7 @@ log "SOAK START mode=$MODE seconds=$SECS guest_dir=$GUEST_DIR timeout=${TMO}s lo # cannot silently shrink the battery. found="" for i in 1 2 3; do - found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done")" || found="" + found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done; true")" || found="" [ -n "$found" ] && break sleep 3 done diff --git a/tools/cuda-ab/streams2_probe.cu b/tools/cuda-ab/streams2_probe.cu new file mode 100644 index 0000000..5bebab4 --- /dev/null +++ b/tools/cuda-ab/streams2_probe.cu @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2_probe.cu - instrumented streams2 to localize the deterministic + * failure: per-array mismatch counts, first-bad indices with values, and a + * GPU-side verify-kernel digest of the device buffers (to separate "device + * pages never written" from "D2H readback wrong"). + * + * Modes (argv[1]): + * A : exactly streams2's shape (control) + * B : like A but the two D2H copy-backs use synchronous cudaMemcpy + * C : like A but no events cross the streams; both streams are joined + * with cudaStreamSynchronize before any copy-back + * D : single stream (everything on s1) + * + * Build: nvcc -O2 --cudart=static -o streams2_probe streams2_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +/* GPU-side digest: counts of matches for c1==a1+b1 and c2==(a2+b2)*2 */ +__global__ void digest(const float *a1, const float *b1, const float *c1, + const float *a2, const float *b2, const float *c2, + int n, unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c1[i] == a1[i] + b1[i]) + atomicAdd(&out[0], 1ull); + if (c2[i] == (a2[i] + b2[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +static int report(const char *tag, const float *h_c, const float *h_a, + const float *h_b, const float *h_a2, const float *h_b2, + int n, float scale2) +{ + int bad = 0, i, shown = 0; + + for (i = 0; i < n; i++) { + float want = h_a[i] + h_b[i]; + + if (scale2 != 1.0f) + want = (h_a2[i] + h_b2[i]) * scale2; + if (h_c[i] != want) { + bad++; + if (shown < 4) { + printf("FIRSTBAD %s i=%d got=%g want=%g\n", + tag, i, h_c[i], want); + shown++; + } + } + } + printf("BAD %s n=%d bad=%d\n", tag, n, bad); + return bad; +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "A"; + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + unsigned long long *d_out, h_out[2]; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s mode=%s\n", prop.name, mode); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + memset(h_c1, 0xCD, bytes); + memset(h_c2, 0xCD, bytes); + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da1=%p db1=%p dc1=%p da2=%p db2=%p dc2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, (void *)d_a2, + (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + if (!strcmp(mode, "D")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s1)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s1>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + scale<<<grid, 256, 0, s1>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + goto verify; + } + + if (!strcmp(mode, "S")) { + /* staged verification: check device content after every step */ + float *hb1 = (float *)malloc(bytes); + float *hb2 = (float *)malloc(bytes); + float *hb3 = (float *)malloc(bytes); + + CHECK(cudaMemcpy(d_a1, h_a1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b1, h_b1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_a2, h_a2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b2, h_b2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(hb1, d_a1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a1[i]) b++; + printf("STAGE-IN da1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b1[i]) b++; + printf("STAGE-IN db1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_a2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a2[i]) b++; + printf("STAGE-IN da2 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b2[i]) b++; + printf("STAGE-IN db2 bad=%d\n", b); } + + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; + if (first < 0) { first = i; + printf("S1-FIRSTBAD i=%d got=%g want=%g a2b2=%g\n", + i, hb1[i], h_a1[i]+h_b1[i], h_a2[i]+h_b2[i]); } } + printf("STAGE vadd1->c1 readback bad=%d first=%d\n", b, first); } + + CHECK(cudaDeviceSynchronize()); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb2, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb2[i] != h_a2[i] + h_b2[i]) { b++; if (first < 0) first = i; } + printf("STAGE vadd2->c2 readback bad=%d first=%d\n", b, first); } + + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb3, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb3[i] != (h_a2[i] + h_b2[i]) * 2.0f) { b++; if (first < 0) first = i; } + printf("STAGE scale->c2 readback bad=%d first=%d\n", b, first); } + /* second readback of c1 AFTER everything: does it change? */ + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; if (first < 0) first = i; } + printf("STAGE c1-late readback bad=%d first=%d\n", b, first); } + printf("STATUS OK\n"); + return 0; + } + + if (!strcmp(mode, "C")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + goto verify; + } + + /* A and B: original shape */ + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + if (!strcmp(mode, "B")) { + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + } else { + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + } + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + +verify: + { + int bad1, bad2; + + bad1 = report("C1", h_c1, h_a1, h_b1, h_a2, h_b2, n, 1.0f); + bad2 = report("C2", h_c2, h_a2, h_b2, h_a2, h_b2, n, s2); + printf("STREAMS n=%d bad=%d\n", n, bad1 + bad2); + printf("TIMING ms=%.3f\n", ms); + + /* GPU-side digest of the device buffers */ + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + digest<<<grid, 256>>>(d_a1, d_b1, d_c1, d_a2, d_b2, d_c2, n, + d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("GPU-DIGEST c1_ok=%llu c2_ok=%llu\n", h_out[0], + h_out[1]); + printf("STATUS %s\n", (bad1 | bad2) ? "FAIL" : "OK"); + } + return 0; +} diff --git a/tools/cuda-ab/streams2d.cu b/tools/cuda-ab/streams2d.cu new file mode 100644 index 0000000..04a0866 --- /dev/null +++ b/tools/cuda-ab/streams2d.cu @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2d.cu - streams2 with per-array / per-range diagnostic reporting. + * + * Same workload as streams2.cu (two streams, cross-stream event ordering) + * but instead of one "bad" count it reports, per array and per 4 KB block, + * how many elements mismatched, and prints the device pointers so a host + * log can be correlated with what the engine read/wrote. + * + * Output lines: + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * PTR <name>=0x<hex> (d_a1 d_b1 d_c1 d_a2 d_b2 d_c2) + * DIFF c1 total=<n> blocks=<first block index with mismatches> + * DIFF c2 total=<n> blocks=<...> + * STREAMS n=<count> bad=<mismatches> + * TIMING ms=<elapsed> + * STATUS OK|FAIL + * + * Build: nvcc -O2 --cudart=static -o streams2d streams2d.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +static void report_diff(const char *name, const float *got, + const float *exp, int n) +{ + int i, bad = 0, first = -1, last = -1; + int blk_hits = 0, blk_first = -1; + int nb = (n + 1023) / 1024; + int b; + + for (i = 0; i < n; i++) { + if (got[i] != exp[i]) { + bad++; + if (first < 0) + first = i; + last = i; + } + } + for (b = 0; b < nb; b++) { + int j, hit = 0; + + for (j = b * 1024; j < (b + 1) * 1024 && j < n; j++) + if (got[j] != exp[j]) { + hit = 1; + break; + } + if (hit) { + blk_hits++; + if (blk_first < 0) + blk_first = b; + } + } + printf("DIFF %s total=%d first=%d last=%d blocks=%d/%d blk_first=%d\n", + name, bad, first, last, blk_hits, nb, blk_first); + if (first >= 0) + printf("DIFF %s sample got[%d]=%a exp[%d]=%a got[%d]=%a exp[%d]=%a\n", + name, first, got[first], first, exp[first], last, + got[last], last, exp[last]); +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + if (!h_a1 || !h_b1 || !h_c1 || !h_a2 || !h_b2 || !h_c2) + return 1; + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + printf("PTR d_a1=%p d_b1=%p d_c1=%p d_a2=%p d_b2=%p d_c2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, + (void *)d_a2, (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + report_diff("c1", h_c1, h_a1, n); + { + float *exp2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) + exp2[i] = (h_a2[i] + h_b2[i]) * s2; + report_diff("c2", h_c2, exp2, n); + for (i = 0; i < n; i++) { + if (h_c1[i] != h_a1[i] + h_b1[i]) + bad++; + if (h_c2[i] != exp2[i]) + bad++; + } + free(exp2); + } + + printf("STREAMS n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/streams_events_probe.cu b/tools/cuda-ab/streams_events_probe.cu new file mode 100644 index 0000000..94aef69 --- /dev/null +++ b/tools/cuda-ab/streams_events_probe.cu @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams_events_probe.cu - instrumented victory_streams_events. + * + * Same 3-stream event chain (c0 = a0+b0; c1 = (a1+b1)+c0; c2 = (a2+b2)+c1) + * but with per-buffer diagnostics: device pointers, per-buffer bad counts, + * and the first 8 expected/actual values of every output so a stale-input + * vs stale-readback failure is distinguishable in one run. + * + * Output lines: + * DEVICE ... + * DPTRS da0..dc2 (device), ha0..hc2 (host) + * BUF s=<n> bad=<n> + * FIRST s=<n> exp=<v0..v7> got=<v0..v7> + * STREAMS3 n=<count> bad=<mismatches> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define NSTREAMS 3 + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vadd_into(float *c, const float *s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] + s[i]; +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 16; + size_t bytes = (size_t)n * sizeof(float); + float *h_a[NSTREAMS], *h_b[NSTREAMS], *h_c[NSTREAMS]; + float *d_a[NSTREAMS], *d_b[NSTREAMS], *d_c[NSTREAMS]; + cudaStream_t st[NSTREAMS]; + cudaEvent_t ev[NSTREAMS], t0, t1; + float ms = 0.0f; + int bad = 0, i, s; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + for (s = 0; s < NSTREAMS; s++) { + h_a[s] = (float *)malloc(bytes); + h_b[s] = (float *)malloc(bytes); + h_c[s] = (float *)malloc(bytes); + if (!h_a[s] || !h_b[s] || !h_c[s]) + return 1; + for (i = 0; i < n; i++) { + h_a[s][i] = (float)((i + s) % 1000) * 0.5f; + h_b[s][i] = (float)((i + 2 * s) % 997) * 0.25f; + } + CHECK(cudaMalloc(&d_a[s], bytes)); + CHECK(cudaMalloc(&d_b[s], bytes)); + CHECK(cudaMalloc(&d_c[s], bytes)); + CHECK(cudaStreamCreate(&st[s])); + CHECK(cudaEventCreate(&ev[s])); + } + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + printf("DPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" da%d=%p db%d=%p dc%d=%p", s, (void *)d_a[s], s, + (void *)d_b[s], s, (void *)d_c[s]); + printf("\n"); + printf("HPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" ha%d=%p hb%d=%p hc%d=%p", s, (void *)h_a[s], s, + (void *)h_b[s], s, (void *)h_c[s]); + printf("\n"); + + CHECK(cudaEventRecord(t0)); + + for (s = 0; s < NSTREAMS; s++) { + CHECK(cudaMemcpyAsync(d_a[s], h_a[s], bytes, + cudaMemcpyHostToDevice, st[s])); + CHECK(cudaMemcpyAsync(d_b[s], h_b[s], bytes, + cudaMemcpyHostToDevice, st[s])); + vadd<<<grid, 256, 0, st[s]>>>(d_a[s], d_b[s], d_c[s], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[s], st[s])); + } + + CHECK(cudaStreamWaitEvent(st[1], ev[0], 0)); + vadd_into<<<grid, 256, 0, st[1]>>>(d_c[1], d_c[0], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[1], st[1])); + + CHECK(cudaStreamWaitEvent(st[2], ev[1], 0)); + vadd_into<<<grid, 256, 0, st[2]>>>(d_c[2], d_c[1], n); + CHECK(cudaGetLastError()); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaMemcpyAsync(h_c[s], d_c[s], bytes, + cudaMemcpyDeviceToHost, st[s])); + CHECK(cudaEventRecord(t1, st[2])); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaStreamSynchronize(st[s])); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + for (s = 0; s < NSTREAMS; s++) { + int b = 0; + float exp0[8], got0[8]; + + for (i = 0; i < n; i++) { + float c0 = h_a[0][i] + h_b[0][i]; + float c1 = h_a[1][i] + h_b[1][i] + c0; + float c2 = h_a[2][i] + h_b[2][i] + c1; + float want = (s == 0) ? c0 : (s == 1) ? c1 : c2; + + if (i < 8) { + exp0[i] = want; + got0[i] = h_c[s][i]; + } + if (h_c[s][i] != want) + b++; + } + bad += b; + printf("BUF s=%d bad=%d\n", s, b); + printf("FIRST s=%d exp=", s); + for (i = 0; i < 8; i++) + printf("%.4f ", exp0[i]); + printf(" got="); + for (i = 0; i < 8; i++) + printf("%.4f ", got0[i]); + printf("\n"); + } + + printf("STREAMS3 n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/tgemm_probe.cu b/tools/cuda-ab/tgemm_probe.cu new file mode 100644 index 0000000..f0a1084 --- /dev/null +++ b/tools/cuda-ab/tgemm_probe.cu @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * tgemm_probe.cu - minimal isolation for the transformer K/V divergence. + * + * Reproduces exactly the transformer's Q/K/V GEMM shape (X[64x256] x + * W[256x256], tile 16) for three weight matrices uploaded back-to-back + * (wq, wk, wv), then verifies each against a CPU reference and prints + * per-GEMM bad counts plus a bad-position histogram (first bad index per + * 16-row band and per 16-col band) so a copy-side hole is distinguishable + * from a kernel-side error. + * + * Output: + * DEVICE ... + * GEMM name=<q|k|v> bad=<n> first=(row,col) got=.. ref=.. + * BADPOS name=.. rows=<r0,r1,..> cols=<c0,c1,..> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define SEQ 64 +#define DH 256 +#define TILE 16 + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static float xval(int i, int j) +{ + return (float)(((i * 7 + j * 3 + 1) % 9) - 4) * 0.25f; +} + +int main(void) +{ + cudaDeviceProp prop; + float *h_x, *h_w, *h_c, *h_ref; + float *d_x, *d_w[3], *d_c[3]; + const int seqdh = SEQ * DH, dh2 = DH * DH; + int g, i, j, k; + + setvbuf(stdout, NULL, _IONBF, 0); + if (cudaGetDeviceProperties(&prop, 0) != cudaSuccess) { + printf("STATUS FAIL\n"); + return 1; + } + printf("DEVICE %s cc=%d.%d\n", prop.name, prop.major, prop.minor); + + h_x = (float *)malloc(seqdh * 4); + h_w = (float *)malloc(dh2 * 4); + h_c = (float *)malloc(seqdh * 4); + h_ref = (float *)malloc(seqdh * 4); + + if (cudaMalloc(&d_x, seqdh * 4) != cudaSuccess) + return 1; + for (g = 0; g < 3; g++) { + if (cudaMalloc(&d_w[g], dh2 * 4) != cudaSuccess) + return 1; + if (cudaMalloc(&d_c[g], seqdh * 4) != cudaSuccess) + return 1; + } + + for (i = 0; i < seqdh; i++) + h_x[i] = xval(i / DH, i % DH); + + /* Upload X once (the transformer's shared input). */ + if (cudaMemcpy(d_x, h_x, seqdh * 4, cudaMemcpyHostToDevice) != + cudaSuccess) + return 1; + /* Verify the X upload itself (first row + full buffer). */ + { + float *h_xb = (float *)malloc(seqdh * 4); + int xbad = 0, xr0 = 0; + + if (!h_xb) + return 1; + if (cudaMemcpy(h_xb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (h_xb[i] != h_x[i]) { + xbad++; + if (i < DH) + xr0++; + } + printf("XVERIFY bad=%d row0_bad=%d\n", xbad, xr0); + free(h_xb); + } + + for (g = 0; g < 3; g++) { + int bad = 0, first_r = -1, first_c = -1; + float got = 0, ref = 0; + int rows[64] = {0}, cols[64] = {0}; + int nr = 0, nc = 0; + int pass; + + for (i = 0; i < dh2; i++) + h_w[i] = wval(i, g + 1); + if (cudaMemcpy(d_w[g], h_w, dh2 * 4, + cudaMemcpyHostToDevice) != cudaSuccess) + return 1; + + dim3 blk(TILE, TILE); + dim3 grd(DH / TILE, SEQ / TILE); + + for (pass = 0; pass < 3; pass++) { + float *hc = h_c, *hxb = h_ref; + int xbad = 0; + + gemm_nn<<<grd, blk>>>(d_x, d_w[g], d_c[g], SEQ, DH, DH); + if (cudaDeviceSynchronize() != cudaSuccess) { + printf("SYNC FAIL g=%d\n", g); + return 1; + } + if (cudaMemcpy(hc, d_c[g], seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + /* re-verify X after the launch */ + if (cudaMemcpy(hxb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (hxb[i] != h_x[i]) { + printf("XDIFF pass=%d i=%d got=%.6f want=%.6f\n", + pass, i, hxb[i], h_x[i]); + xbad++; + } + bad = 0; + nr = 0; + nc = 0; + for (i = 0; i < SEQ; i++) + for (j = 0; j < DH; j++) { + double acc = 0.0; + float r, a, diff; + + for (k = 0; k < DH; k++) + acc += (double)h_x[i * DH + k] * + (double)h_w[k * DH + j]; + r = (float)acc; + a = hc[i * DH + j]; + diff = a - r; + if (diff < 0) + diff = -diff; + if (diff > 1e-4f) { + int rr = i, cc = j; + + if (bad == 0) { + first_r = rr; + first_c = cc; + got = a; + ref = r; + } + bad++; + if (nr < 64 && (bad == 1 || + rr != rows[nr - 1])) + rows[nr++] = rr; + if (nc < 64 && (bad == 1 || + cc != cols[nc - 1])) + cols[nc++] = cc; + } + } + printf("PASS name=%c pass=%d bad=%d xbad=%d first=(%d,%d) got=%.6f ref=%.6f\n", + "qkv"[g], pass, bad, xbad, first_r, first_c, + got, ref); + if (bad) { + printf("ROW0 name=%c got=", "qkv"[g]); + for (i = 0; i < 8; i++) + printf("%.6f ", hc[i]); + printf(" ref="); + for (i = 0; i < 8; i++) { + double acc = 0.0; + + for (k = 0; k < DH; k++) + acc += (double)h_x[k] * + (double)h_w[k * DH + i]; + printf("%.6f ", (float)acc); + } + printf("\n"); + } + } + (void)got; + (void)ref; + (void)first_r; + (void)first_c; + (void)nr; + (void)nc; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/transformer.cu b/tools/cuda-ab/transformer.cu new file mode 100644 index 0000000..3044522 --- /dev/null +++ b/tools/cuda-ab/transformer.cu @@ -0,0 +1,472 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward ---- */ + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/transformer_async.cu b/tools/cuda-ab/transformer_async.cu new file mode 100644 index 0000000..213363a --- /dev/null +++ b/tools/cuda-ab/transformer_async.cu @@ -0,0 +1,522 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward (async chain, stage readbacks) ---- */ + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + { + float *hs = (float *)malloc(b_seqdh); + float *hs2 = (float *)malloc(b_seq2); + double *rw = (double *)malloc((size_t)dh2 * sizeof(double)); + double *rq = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rk = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rv = (double *)malloc((size_t)seqdh * sizeof(double)); + double *rs = (double *)malloc((size_t)seq2 * sizeof(double)); + double *ra = (double *)malloc((size_t)seq2 * sizeof(double)); + double *rav = (double *)malloc((size_t)seqdh * sizeof(double)); + double *ry = (double *)malloc((size_t)seqdh * sizeof(double)); + int st; + + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 1); + ref_gemm(r_x, rw, rq, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 2); + ref_gemm(r_x, rw, rk, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 3); + ref_gemm(r_x, rw, rv, SEQ, DH, DH); + ref_gemm_nt(rq, rk, rs, SEQ, SEQ, DH); + ref_softmax(rs, ra, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(ra, rv, rav, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + rw[i] = (double)wval(i, 4); + ref_gemm(rav, rw, ry, SEQ, DH, DH); + + CHECK(cudaMemcpy(hs, d_q, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rq, seqdh); + printf("ASYNC Q bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_k, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rk, seqdh); + printf("ASYNC K bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_v, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rv, seqdh); + printf("ASYNC V bad=%d\n", st); + CHECK(cudaMemcpy(hs2, d_s, b_seq2, cudaMemcpyDeviceToHost)); + st = count_bad(hs2, rs, seq2); + printf("ASYNC S bad=%d\n", st); + CHECK(cudaMemcpy(hs2, d_a, b_seq2, cudaMemcpyDeviceToHost)); + st = count_bad(hs2, ra, seq2); + printf("ASYNC A bad=%d\n", st); + CHECK(cudaMemcpy(hs, d_av, b_seqdh, cudaMemcpyDeviceToHost)); + st = count_bad(hs, rav, seqdh); + printf("ASYNC AV bad=%d\n", st); + st = count_bad(h_y, ry, seqdh); + printf("ASYNC Y bad=%d\n", st); + } + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/transformer_stages.cu b/tools/cuda-ab/transformer_stages.cu new file mode 100644 index 0000000..d7f67c0 --- /dev/null +++ b/tools/cuda-ab/transformer_stages.cu @@ -0,0 +1,553 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * transformer.cu - single-head attention forward + 4-layer FFN mini-transformer. + * + * Two independent checks in one binary, both verified against an internal + * double-precision CPU reference (tolerance 1e-3 relative, 1e-6 absolute floor): + * + * 1. Attention: X[64x256], Wq/Wk/Wv/Wo[256x256] (head dim 256) + * Q = X*Wq, K = X*Wk, V = X*Wv + * S = Q*K^T / 16 (scale = 1/sqrt(head_dim)) + * A = softmax(S) (row-wise) + * Y = (A*V)*Wo + * 2. Mini-transformer FFN head: h <- ReLU(h*Wf[l]) for 4 layers, then + * logits = h*Wh, P = softmax(logits) (row-wise over 256) + * + * Inputs and weights are multiples of 0.25/0.125, so every GEMM is exact in + * fp32; only the softmax exponentials differ from the CPU reference, hence + * the relative tolerance. + * + * Output lines are stable and greppable: + * + * DEVICE <name> cc=<major>.<minor> sm=<count> mem=<bytes> + * TRANSFORMER dh=256 seq=64 bad=<mismatches> ysum=<sum> + * FFN4 layers=4 bad=<mismatches> psum=<sum> + * STATUS OK|FAIL + * + * Build with a static CUDA runtime (--cudart=static) so the guest only needs + * the driver's libcuda.so, not a CUDA toolkit installation: + * + * nvcc -O2 --cudart=static -arch=sm_75 -o transformer transformer.cu + * + * Then, from the guest: /tmp/transformer + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cmath> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define SEQ 64 +#define DH 256 +#define TILE 16 +#define TOL 1e-3f + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + /* C[i][j] = sum_k A[i][k] * B[j][k] (B is [N][K], row-major). */ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(blockIdx.x * TILE + threadIdx.y) * K + + t * TILE + threadIdx.x]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[threadIdx.x][k]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +__global__ void softmax_rows(const float *S, float *P, int rows, int cols, + float scale) +{ + extern __shared__ float sm[]; + int row = blockIdx.x; + int t = threadIdx.x; + float v = S[row * cols + t] * scale; + int stride; + + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] = fmaxf(sm[t], sm[t + stride]); + __syncthreads(); + } + v = expf(v - sm[0]); + sm[t] = v; + __syncthreads(); + for (stride = blockDim.x >> 1; stride > 0; stride >>= 1) { + if (t < stride) + sm[t] += sm[t + stride]; + __syncthreads(); + } + P[row * cols + t] = v / sm[0]; +} + +__global__ void relu_inplace(float *h, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n && h[i] < 0.0f) + h[i] = 0.0f; +} + +/* ------------------------------------------------------------------ host */ + +static void launch_gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nn<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_gemm_nt(const float *A, const float *B, float *C, + int M, int N, int K) +{ + gemm_nt<<<dim3(N / TILE, M / TILE), dim3(TILE, TILE)>>>(A, B, C, M, N, K); +} + +static void launch_softmax(const float *S, float *P, int rows, int cols, + float scale) +{ + softmax_rows<<<rows, cols, (size_t)cols * sizeof(float)>>>(S, P, rows, + cols, + scale); +} + +/* Deterministic inputs: multiples of 0.25 (X) and 0.125 (weights). */ +static float xval(int i) +{ + return (float)(((i % 7) - 3)) * 0.25f; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static void ref_gemm(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[k * N + j]; + C[i * N + j] = acc; + } +} + +static void ref_gemm_nt(const double *A, const double *B, double *C, + int M, int N, int K) +{ + for (int i = 0; i < M; i++) + for (int j = 0; j < N; j++) { + double acc = 0.0; + + for (int k = 0; k < K; k++) + acc += A[i * K + k] * B[j * K + k]; + C[i * N + j] = acc; + } +} + +static void ref_softmax(const double *S, double *P, int rows, int cols, + double scale) +{ + for (int i = 0; i < rows; i++) { + const double *s = S + i * cols; + double *p = P + i * cols; + double m = s[0] * scale, sum = 0.0; + + for (int j = 1; j < cols; j++) + if (s[j] * scale > m) + m = s[j] * scale; + for (int j = 0; j < cols; j++) { + p[j] = exp(s[j] * scale - m); + sum += p[j]; + } + for (int j = 0; j < cols; j++) + p[j] /= sum; + } +} + +static void ref_relu(double *h, int n) +{ + for (int i = 0; i < n; i++) + if (h[i] < 0.0) + h[i] = 0.0; +} + +static int count_bad(const float *gpu, const double *ref, int n) +{ + int bad = 0; + + for (int i = 0; i < n; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) + bad++; + } + return bad; +} + +static double dsum(const float *h, int n) +{ + double s = 0.0; + + for (int i = 0; i < n; i++) + s += (double)h[i]; + return s; +} + +static void dump_bad(const char *tag, const float *gpu, const double *ref, + int n) +{ + int i, shown = 0; + + for (i = 0; i < n && shown < 8; i++) { + double g = (double)gpu[i]; + double r = ref[i]; + + if (fabs(g - r) > (double)TOL * (fabs(r) + 1e-6)) { + printf("BAD %s i=%d gpu=%.6f ref=%.6f (tile %d,%d row %d col %d)\n", + tag, i, g, r, i / 16, (i % 256) / 16, + i / 256, i % 256); + shown++; + } + } +} + +#define NF 4 /* FFN layers */ + +int main(void) +{ + cudaDeviceProp prop; + const int seqdh = SEQ * DH, dh2 = DH * DH, seq2 = SEQ * SEQ; + const size_t b_seqdh = (size_t)seqdh * sizeof(float); + const size_t b_dh2 = (size_t)dh2 * sizeof(float); + const size_t b_seq2 = (size_t)seq2 * sizeof(float); + float *h_x, *h_y, *h_p; + double *r_x, *r_q, *r_k, *r_v, *r_s, *r_a, *r_av, *r_y; + double *r_f0, *r_f1, *r_logits, *r_p; + float *d_x, *d_q, *d_k, *d_v, *d_s, *d_a, *d_av, *d_y; + float *d_wq, *d_wk, *d_wv, *d_wo; + float *d_wf[NF], *d_wh, *d_f0, *d_f1, *d_logits, *d_p; + float *d_ping, *d_pong, *tmpf; + double *r_ping, *r_pong, *tmpd; + int bad_y, bad_p, i, l; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + /* host inputs + CPU reference workspace */ + h_x = (float *)malloc(b_seqdh); + h_y = (float *)malloc(b_seqdh); + h_p = (float *)malloc(b_seqdh); + r_x = (double *)malloc((size_t)seqdh * sizeof(double)); + r_q = (double *)malloc((size_t)seqdh * sizeof(double)); + r_k = (double *)malloc((size_t)seqdh * sizeof(double)); + r_v = (double *)malloc((size_t)seqdh * sizeof(double)); + r_s = (double *)malloc((size_t)seq2 * sizeof(double)); + r_a = (double *)malloc((size_t)seq2 * sizeof(double)); + r_av = (double *)malloc((size_t)seqdh * sizeof(double)); + r_y = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f0 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_f1 = (double *)malloc((size_t)seqdh * sizeof(double)); + r_logits = (double *)malloc((size_t)seqdh * sizeof(double)); + r_p = (double *)malloc((size_t)seqdh * sizeof(double)); + if (!h_x || !h_y || !h_p || !r_x || !r_q || !r_k || !r_v || !r_s || + !r_a || !r_av || !r_y || !r_f0 || !r_f1 || !r_logits || !r_p) + return 1; + + for (i = 0; i < seqdh; i++) { + h_x[i] = xval(i); + r_x[i] = (double)xval(i); + } + + /* device buffers */ + CHECK(cudaMalloc(&d_x, b_seqdh)); + CHECK(cudaMalloc(&d_q, b_seqdh)); + CHECK(cudaMalloc(&d_k, b_seqdh)); + CHECK(cudaMalloc(&d_v, b_seqdh)); + CHECK(cudaMalloc(&d_s, b_seq2)); + CHECK(cudaMalloc(&d_a, b_seq2)); + CHECK(cudaMalloc(&d_av, b_seqdh)); + CHECK(cudaMalloc(&d_y, b_seqdh)); + CHECK(cudaMalloc(&d_wq, b_dh2)); + CHECK(cudaMalloc(&d_wk, b_dh2)); + CHECK(cudaMalloc(&d_wv, b_dh2)); + CHECK(cudaMalloc(&d_wo, b_dh2)); + CHECK(cudaMalloc(&d_wh, b_dh2)); + CHECK(cudaMalloc(&d_f0, b_seqdh)); + CHECK(cudaMalloc(&d_f1, b_seqdh)); + CHECK(cudaMalloc(&d_logits, b_seqdh)); + CHECK(cudaMalloc(&d_p, b_seqdh)); + for (l = 0; l < NF; l++) + CHECK(cudaMalloc(&d_wf[l], b_dh2)); + + /* weights: deterministic, uploaded once */ + { + float *w = (float *)malloc(b_dh2); + + if (!w) + return 1; + for (i = 0; i < dh2; i++) + w[i] = wval(i, 1); + CHECK(cudaMemcpy(d_wq, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 2); + CHECK(cudaMemcpy(d_wk, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 3); + CHECK(cudaMemcpy(d_wv, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 4); + CHECK(cudaMemcpy(d_wo, w, b_dh2, cudaMemcpyHostToDevice)); + for (i = 0; i < dh2; i++) + w[i] = wval(i, 20); + CHECK(cudaMemcpy(d_wh, w, b_dh2, cudaMemcpyHostToDevice)); + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + w[i] = wval(i, 10 + l); + CHECK(cudaMemcpy(d_wf[l], w, b_dh2, + cudaMemcpyHostToDevice)); + } + free(w); + } + CHECK(cudaMemcpy(d_x, h_x, b_seqdh, cudaMemcpyHostToDevice)); + + /* ---- check 1: attention forward, stage-by-stage verify ---- */ + { + float *h_stage = (float *)malloc(b_seqdh); + float *h_s2 = (float *)malloc(b_seq2); + double *r_stage = (double *)malloc((size_t)seqdh * sizeof(double)); + double *r_s2 = (double *)malloc((size_t)seq2 * sizeof(double)); + double *r_w2 = (double *)malloc((size_t)dh2 * sizeof(double)); + int st; + + if (!h_stage || !h_s2 || !r_stage || !r_s2 || !r_w2) + return 1; + + launch_gemm_nn(d_x, d_wq, d_q, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_q, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE Q bad=%d\n", st); + dump_bad("Q", h_stage, r_stage, seqdh); + + launch_gemm_nn(d_x, d_wk, d_k, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_k, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE K bad=%d\n", st); + dump_bad("K", h_stage, r_stage, seqdh); + + launch_gemm_nn(d_x, d_wv, d_v, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_v, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w2, r_stage, SEQ, DH, DH); + st = count_bad(h_stage, r_stage, seqdh); + printf("STAGE V bad=%d\n", st); + dump_bad("V", h_stage, r_stage, seqdh); + + launch_gemm_nt(d_q, d_k, d_s, SEQ, SEQ, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_s2, d_s, b_seq2, cudaMemcpyDeviceToHost)); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + st = count_bad(h_s2, r_s, seq2); + printf("STAGE S bad=%d\n", st); + dump_bad("S", h_s2, r_s, seq2); + + launch_softmax(d_s, d_a, SEQ, SEQ, 1.0f / 16.0f); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_s2, d_a, b_seq2, cudaMemcpyDeviceToHost)); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + st = count_bad(h_s2, r_a, seq2); + printf("STAGE A bad=%d\n", st); + dump_bad("A", h_s2, r_a, seq2); + + launch_gemm_nn(d_a, d_v, d_av, SEQ, DH, SEQ); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_stage, d_av, b_seqdh, cudaMemcpyDeviceToHost)); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + st = count_bad(h_stage, r_av, seqdh); + printf("STAGE AV bad=%d\n", st); + + launch_gemm_nn(d_av, d_wo, d_y, SEQ, DH, DH); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_y, d_y, b_seqdh, cudaMemcpyDeviceToHost)); + for (i = 0; i < dh2; i++) + r_w2[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w2, r_y, SEQ, DH, DH); + st = count_bad(h_y, r_y, seqdh); + printf("STAGE Y bad=%d\n", st); + } + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 1); + ref_gemm(r_x, r_w, r_q, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 2); + ref_gemm(r_x, r_w, r_k, SEQ, DH, DH); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 3); + ref_gemm(r_x, r_w, r_v, SEQ, DH, DH); + ref_gemm_nt(r_q, r_k, r_s, SEQ, SEQ, DH); + ref_softmax(r_s, r_a, SEQ, SEQ, 1.0 / 16.0); + ref_gemm(r_a, r_v, r_av, SEQ, DH, SEQ); + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 4); + ref_gemm(r_av, r_w, r_y, SEQ, DH, DH); + free(r_w); + } + + bad_y = count_bad(h_y, r_y, seqdh); + printf("TRANSFORMER dh=256 seq=64 bad=%d ysum=%.6f\n", bad_y, + dsum(h_y, seqdh)); + + /* ---- check 2: 4-layer FFN stack + softmax head ---- */ + CHECK(cudaMemcpy(d_f0, h_x, b_seqdh, cudaMemcpyHostToDevice)); + d_ping = d_f0; + d_pong = d_f1; + for (l = 0; l < NF; l++) { + launch_gemm_nn(d_ping, d_wf[l], d_pong, SEQ, DH, DH); + relu_inplace<<<(seqdh + 255) / 256, 256>>>(d_pong, seqdh); + tmpf = d_ping; + d_ping = d_pong; + d_pong = tmpf; + } + launch_gemm_nn(d_ping, d_wh, d_logits, SEQ, DH, DH); + launch_softmax(d_logits, d_p, SEQ, DH, 1.0f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_p, d_p, b_seqdh, cudaMemcpyDeviceToHost)); + + { + double *r_w = (double *)malloc((size_t)dh2 * sizeof(double)); + + if (!r_w) + return 1; + memcpy(r_f0, r_x, (size_t)seqdh * sizeof(double)); + r_ping = r_f0; + r_pong = r_f1; + for (l = 0; l < NF; l++) { + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 10 + l); + ref_gemm(r_ping, r_w, r_pong, SEQ, DH, DH); + ref_relu(r_pong, seqdh); + tmpd = r_ping; + r_ping = r_pong; + r_pong = tmpd; + } + for (i = 0; i < dh2; i++) + r_w[i] = (double)wval(i, 20); + ref_gemm(r_ping, r_w, r_logits, SEQ, DH, DH); + ref_softmax(r_logits, r_p, SEQ, DH, 1.0); + free(r_w); + } + + bad_p = count_bad(h_p, r_p, seqdh); + printf("FFN4 layers=4 bad=%d psum=%.6f\n", bad_p, dsum(h_p, seqdh)); + + if (bad_y != 0 || bad_p != 0) { + printf("STATUS FAIL\n"); + return 1; + } + + printf("STATUS OK\n"); + CHECK(cudaFree(d_x)); + CHECK(cudaFree(d_q)); + CHECK(cudaFree(d_k)); + CHECK(cudaFree(d_v)); + CHECK(cudaFree(d_s)); + CHECK(cudaFree(d_a)); + CHECK(cudaFree(d_av)); + CHECK(cudaFree(d_y)); + CHECK(cudaFree(d_wq)); + CHECK(cudaFree(d_wk)); + CHECK(cudaFree(d_wv)); + CHECK(cudaFree(d_wo)); + CHECK(cudaFree(d_wh)); + for (l = 0; l < NF; l++) + CHECK(cudaFree(d_wf[l])); + CHECK(cudaFree(d_f0)); + CHECK(cudaFree(d_f1)); + CHECK(cudaFree(d_logits)); + CHECK(cudaFree(d_p)); + free(h_x); + free(h_y); + free(h_p); + free(r_x); + free(r_q); + free(r_k); + free(r_v); + free(r_s); + free(r_a); + free(r_av); + free(r_y); + free(r_f0); + free(r_f1); + free(r_logits); + free(r_p); + return 0; +} diff --git a/tools/cuda-ab/wedge_ab.cu b/tools/cuda-ab/wedge_ab.cu new file mode 100644 index 0000000..2d88f92 --- /dev/null +++ b/tools/cuda-ab/wedge_ab.cu @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_ab.cu - discriminate the wedge trigger: is it the second launch of + * the SAME program, or the second launch of the context regardless of + * program? + * + * Sequence (each launch verified against mutated inputs): + * 1. vadd (program A) + * 2. vsub (program B, different code) + * 3. vadd (program A again) + * 4. vsub (program B again) + * + * If launch 2 (B) executes -> the trigger is same-program reuse. + * If launch 2 (B) wedges -> the trigger is the second launch in the context. + * + * Build: nvcc -O2 --cudart=static -o wedge_ab wedge_ab.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vsub(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] - b[i]; +} + +int main(void) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc; + float *da, *db, *dc; + int i, bad, step; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + if (!ha || !hb || !hc) + return 1; + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + + for (step = 0; step < 4; step++) { + const char *which = (step & 1) ? "vsub" : "vadd"; + int is_sub = step & 1; + + if (step > 0) { + ha[step] += 1.0f; + hb[step * 7 + 3] += 2.0f; + } + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + if (is_sub) + vsub<<<(n + 255) / 256, 256>>>(da, db, dc, n); + else + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + bad = 0; + for (i = 0; i < n; i++) { + float expect = is_sub ? ha[i] - hb[i] : ha[i] + hb[i]; + + if (hc[i] != expect) + bad++; + } + printf("STEP %d %s bad=%d/%d\n", step, which, bad, n); + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/wedge_bisect.cu b/tools/cuda-ab/wedge_bisect.cu new file mode 100644 index 0000000..e062a2b --- /dev/null +++ b/tools/cuda-ab/wedge_bisect.cu @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_bisect.cu - bisect the wedge trigger: which inter-launch activity + * makes the SECOND vadd launch no-execute? + * + * Every iteration is: mutate host inputs -> H2D da/db -> launch -> sync -> + * D2H dc -> verify (the wedge_recover shape). Modes add activity between + * the verify and the next launch: + * + * mode A: none (pure second-launch class) + * mode B: an extra H2D of the same buffers (CE write before the launch) + * mode C: an extra D2H readback (CE read before the launch) + * mode D: a 500 ms sleep (no CE traffic) + * + * A no-execute launch is reported (bad>0) not masked: the inputs mutate every + * iteration. + * + * Build: nvcc -O2 --cudart=static -o wedge_bisect wedge_bisect.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> +#include <unistd.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +int main(int argc, char **argv) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc; + float *da, *db, *dc; + const char *mode = argc > 1 ? argv[1] : "A"; + int iters = argc > 2 ? atoi(argv[2]) : 4; + int i, bad = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + if (!ha || !hb || !hc) + return 1; + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + for (i = 0; i < iters; i++) { + int j, itbad = 0; + + if (i > 0) { + ha[i % n] += 1.0f; + hb[(i * 7 + 3) % n] += 2.0f; + } + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + for (j = 0; j < n; j++) + if (hc[j] != ha[j] + hb[j]) + itbad++; + printf("MODE %s iter=%d bad=%d/%d\n", mode, i, itbad, n); + bad += itbad; + if (i + 1 < iters) { + if (mode[0] == 'B') { + CHECK(cudaMemcpy(da, ha, bytes, + cudaMemcpyHostToDevice)); + } else if (mode[0] == 'C') { + CHECK(cudaMemcpy(hc, dc, bytes, + cudaMemcpyDeviceToHost)); + } else if (mode[0] == 'D') { + usleep(500000); + } + } + } + printf("STATUS %s\n", bad ? "FAIL" : "OK"); + return bad ? 1 : 0; +} diff --git a/tools/cuda-ab/wedge_recover.cu b/tools/cuda-ab/wedge_recover.cu new file mode 100644 index 0000000..fd8f3f9 --- /dev/null +++ b/tools/cuda-ab/wedge_recover.cu @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * wedge_recover.cu - reproduce the per-launch no-execute (wedge) and test + * whether an idle gap heals a wedged context. + * + * v2 (session 10, r416 finding): the v1 loop reused one deterministic + * kernel+buffers, so a no-write launch was MASKED - the stale buffer held + * exactly the previous iteration's (identical) values and the verify passed. + * r416 ran 60/60 "OK" while 58 consecutive launches no-wrote underneath. + * + * v2 mutates the inputs every iteration (H2D of fresh ha/hb), so a stale + * readback cannot match the new expected values: a no-write launch must + * report BAD. On BAD the same iteration is retried after a sleep, and the + * readback is classified: STALE-PREV (byte-identical to the previous + * iteration's output = launch never executed) vs PARTIAL/OTHER (engine wrote + * something). Latency and per-iteration outcome are printed. + * + * Build: nvcc -O2 --cudart=static -o wedge_recover wedge_recover.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cuda_runtime.h> +#include <unistd.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +int main(int argc, char **argv) +{ + const int n = 1 << 15; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hc_prev; + float *da, *db, *dc; + int iters = argc > 1 ? atoi(argv[1]) : 40; + int sleep_s = argc > 2 ? atoi(argv[2]) : 20; + int fails = 0, heals = 0, i; + + setvbuf(stdout, NULL, _IONBF, 0); + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hc_prev = (float *)malloc(bytes); + if (!ha || !hb || !hc || !hc_prev) + return 1; + /* iteration 0 baseline; mutated each iteration below (v2) */ + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 977) * 0.25f; + hb[i] = (float)(i % 251) * 0.5f; + } + memset(hc_prev, 0, bytes); + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + for (i = 0; i < iters; i++) { + int attempt = 0; + long long ms[2] = { 0, 0 }; + + /* + * v2: fresh inputs every iteration so a stale (never + * executed) launch cannot match. Mutate after iteration 0. + */ + if (i > 0) { + ha[i % n] += 1.0f; + hb[(i * 7 + 3) % n] += 2.0f; + CHECK(cudaMemcpy(da, ha, bytes, + cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, + cudaMemcpyHostToDevice)); + } + + for (;;) { + int bad = 0, stale_prev = 1, j; + cudaEvent_t e0, e1; + float ems = 0; + + cudaEventCreate(&e0); + cudaEventCreate(&e1); + cudaEventRecord(e0); + vadd<<<(n + 255) / 256, 256>>>(da, db, dc, n); + CHECK(cudaGetLastError()); + cudaEventRecord(e1); + CHECK(cudaDeviceSynchronize()); + cudaEventElapsedTime(&ems, e0, e1); + cudaEventDestroy(e0); + cudaEventDestroy(e1); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + for (j = 0; j < n; j++) { + float expect = ha[j] + hb[j]; + + if (hc[j] != expect) { + bad++; + if (hc[j] != hc_prev[j]) + stale_prev = 0; + } + } + ms[attempt & 1] = (long long)(ems * 1000.0f); + if (bad == 0) { + if (attempt > 0) + heals++; + printf("ITER %d OK attempt=%d launch_us=%lld\n", + i, attempt, ms[attempt & 1]); + memcpy(hc_prev, hc, bytes); + break; + } + fails++; + printf("ITER %d BAD bad=%d/%d attempt=%d launch_us=%lld class=%s\n", + i, bad, n, attempt, ms[attempt & 1], + stale_prev ? "STALE-PREV(NOWRITE)" + : "OTHER/PARTIAL"); + if (attempt >= 3) { + printf("ITER %d GIVEUP\n", i); + break; + } + attempt++; + printf("ITER %d SLEEP %ds before retry\n", i, sleep_s); + sleep(sleep_s); + } + } + printf("SUMMARY iters=%d fails=%d heals=%d sleep_s=%d\n", + iters, fails, heals, sleep_s); + printf("STATUS %s\n", fails == 0 ? "OK" : "FAIL"); + return fails == 0 ? 0 : 1; +} diff --git a/tools/guest-diag/egl_frame.c b/tools/guest-diag/egl_frame.c new file mode 100644 index 0000000..ed10e39 --- /dev/null +++ b/tools/guest-diag/egl_frame.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * egl_frame.c - minimal EGL/GLES2 offscreen render test for the vGPU guest. + * + * Tries, in order: the EGL_EXT_platform_device platform with each enumerated + * EGLDeviceEXT, EGL_MESA_platform_surfaceless, then EGL_DEFAULT_DISPLAY. + * For the first display that initializes it creates an offscreen pbuffer + * context, renders colour bands into an FBO, reads them back and writes a + * PPM. One greppable line per stage. + * + * Build (guest): gcc -O2 -o egl_frame egl_frame.c -lEGL -lGLESv2 + */ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <EGL/egl.h> +#include <EGL/eglext.h> +#include <GLES2/gl2.h> + +#define W 320 +#define H 240 + +static EGLDisplay try_display(const char *name, EGLDisplay dpy, + const char *dev_name) +{ + EGLint major = 0, minor = 0; + + if (dpy == EGL_NO_DISPLAY) { + printf("EGL_PLATFORM %s -> no display\n", name); + return EGL_NO_DISPLAY; + } + if (!eglInitialize(dpy, &major, &minor)) { + printf("EGL_PLATFORM %s -> init fail 0x%x\n", name, + eglGetError()); + return EGL_NO_DISPLAY; + } + printf("EGL_PLATFORM %s ok %d.%d vendor=%s%s%s\n", name, major, minor, + eglQueryString(dpy, EGL_VENDOR), + dev_name ? " device=" : "", dev_name ? dev_name : ""); + return dpy; +} + +static EGLDisplay get_display(void) +{ + PFNEGLGETPLATFORMDISPLAYEXTPROC gp = + (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress( + "eglGetPlatformDisplayEXT"); + PFNEGLQUERYDEVICESEXTPROC qd = + (PFNEGLQUERYDEVICESEXTPROC)eglGetProcAddress( + "eglQueryDevicesEXT"); + EGLDisplay dpy; + + if (gp && qd) { + EGLDeviceEXT devs[8]; + EGLint n = 0; + int i; + + if (qd(8, devs, &n) && n > 0) { + printf("EGL_DEVICES n=%d\n", n); + for (i = 0; i < n; i++) { + EGLDisplay d = gp(EGL_PLATFORM_DEVICE_EXT, + devs[i], NULL); + + dpy = try_display("device", d, "enum"); + if (dpy != EGL_NO_DISPLAY) + return dpy; + } + } else { + printf("EGL_DEVICES n=0\n"); + } + if (gp) { + dpy = try_display("surfaceless", + gp(EGL_PLATFORM_SURFACELESS_MESA, + NULL, NULL), NULL); + if (dpy != EGL_NO_DISPLAY) + return dpy; + } + } + return try_display("default", eglGetDisplay(EGL_DEFAULT_DISPLAY), + NULL); +} + +static int render(EGLDisplay dpy) +{ + EGLConfig cfg; + EGLint ncfg = 0; + EGLSurface surf; + EGLContext ctx; + GLuint fbo = 0, tex = 0; + unsigned char *px; + FILE *f; + int i; + EGLint attrs[] = { + EGL_SURFACE_TYPE, EGL_PBUFFER_BIT, + EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT, + EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, + EGL_BLUE_SIZE, 8, EGL_ALPHA_SIZE, 8, + EGL_NONE + }; + EGLint pattrs[] = { EGL_WIDTH, W, EGL_HEIGHT, H, EGL_NONE }; + EGLint cattrs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; + + if (!eglChooseConfig(dpy, attrs, &cfg, 1, &ncfg) || ncfg < 1) { + printf("EGL_CONFIG_FAIL err=0x%x\n", eglGetError()); + return 1; + } + surf = eglCreatePbufferSurface(dpy, cfg, pattrs); + if (surf == EGL_NO_SURFACE) { + printf("EGL_PBUFFER_FAIL err=0x%x\n", eglGetError()); + return 1; + } + ctx = eglCreateContext(dpy, cfg, EGL_NO_CONTEXT, cattrs); + if (ctx == EGL_NO_CONTEXT) { + printf("EGL_CONTEXT_FAIL err=0x%x\n", eglGetError()); + return 1; + } + if (!eglMakeCurrent(dpy, surf, surf, ctx)) { + printf("EGL_CURRENT_FAIL err=0x%x\n", eglGetError()); + return 1; + } + printf("GL_RENDERER %s\n", glGetString(GL_RENDERER)); + printf("GL_VERSION %s\n", glGetString(GL_VERSION)); + + glGenTextures(1, &tex); + glBindTexture(GL_TEXTURE_2D, tex); + glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, W, H, 0, GL_RGBA, + GL_UNSIGNED_BYTE, NULL); + glGenFramebuffers(1, &fbo); + glBindFramebuffer(GL_FRAMEBUFFER, fbo); + glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, + GL_TEXTURE_2D, tex, 0); + if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != + GL_FRAMEBUFFER_COMPLETE) { + printf("GL_FBO_FAIL\n"); + return 1; + } + for (i = 0; i < 16; i++) { + glEnable(GL_SCISSOR_TEST); + glScissor(0, i * H / 16, W, H / 16); + glClearColor((float)i / 16.0f, 1.0f - (float)i / 16.0f, + 0.5f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT); + } + glFinish(); + printf("GL_ERR 0x%x\n", glGetError()); + px = (unsigned char *)malloc(W * H * 4); + glReadPixels(0, 0, W, H, GL_RGBA, GL_UNSIGNED_BYTE, px); + { + const char *out = getenv("EGL_FRAME_OUT"); + + if (!out) + out = "/tmp/egl_frame.ppm"; + f = fopen(out, "wb"); + if (!f) { + printf("PPM_OPEN_FAIL\n"); + return 1; + } + fprintf(f, "P6\n%d %d\n255\n", W, H); + for (i = 0; i < W * H; i++) + fwrite(px + i * 4, 1, 3, f); + fclose(f); + printf("EGL_FRAME_WRITTEN %s\n", out); + } + printf("EGL_FRAME_OK\n"); + return 0; +} + +int main(void) +{ + EGLDisplay dpy; + const char *ext; + + setvbuf(stdout, NULL, _IONBF, 0); + ext = eglQueryString(EGL_NO_DISPLAY, EGL_EXTENSIONS); + printf("EGL_CLIENT_EXT %s\n", ext ? ext : "(none)"); + dpy = get_display(); + if (dpy == EGL_NO_DISPLAY) { + printf("EGL_ALL_PLATFORMS_FAILED\n"); + return 1; + } + return render(dpy); +} diff --git a/tools/guest-diag/guest_pin_stream.py b/tools/guest-diag/guest_pin_stream.py index 97c68f2..bb253a2 100644 --- a/tools/guest-diag/guest_pin_stream.py +++ b/tools/guest-diag/guest_pin_stream.py @@ -266,6 +266,103 @@ def _vmalloc_pages_for(kc, va): return 0 +def _nv_vmap_areas(kc): + """nv_vm_map_pages vmalloc areas: [(lo, hi, pages)] (5 s cache). + + Every CE push-buffer rm_mem is a vmap created by nv_vm_map_pages; the + demand pinner walks these areas to find the page holding a stalled + push VA by content, without needing any kprobe event or object chain. + """ + now = time.time() + if getattr(_nv_vmap_areas, "ts", 0) > now - 5.0: + return _nv_vmap_areas.areas + areas = [] + try: + for line in open("/proc/vmallocinfo"): + if "nv_vm_map_pages" not in line: + continue + p = line.split() + if len(p) < 2: + continue + try: + lo, hi = p[0].split("-") + lo = int(lo, 16) + hi = int(hi, 16) + except ValueError: + continue + pages = (hi - lo) >> 12 + if pages > 0: + areas.append((lo, hi, pages)) + except OSError: + pass + _nv_vmap_areas.areas = areas + _nv_vmap_areas.ts = now + return areas + + +# Learned page-offset per nv_vm_map_pages area for the demand content match +# (2026-10-16b): the first successful resolution records which page index the +# VA maps to, and later pushes in the same allocation retry that offset first +# instead of walking the whole +/-16 neighbourhood. +_area_off = {} + + +def resolve_va_by_content(kc, va, out): + """Demand pin: find the page carrying GPU VA 'va' by content match. + + The host reports the CE push-buffer VAs whose GPFIFO entries it cannot + read (dmesg 'stalled VA #N: 0x...'). The guest walks every + nv_vm_map_pages vmap area and tests the candidate leaf page at the VA's + in-window offset (plus a neighbourhood, for the per-mem header-page + offset ambiguity) against the host's method-stream prefix check. On a + match it emits the exact pair, so the host can translate and consume the + entry - no kprobe event, no object-chain discovery, no resolution + lottery. Only the requested VA is pinned, so a wrong candidate cannot + poison other VAs. + + 2026-10-16b: the offset search window was +/-1 page. Pushes of a client + whose pushbuffer vmap carries a larger per-allocation header (observed + with multi-MB CE staging: pushes at 0x120000f58+ never resolved and the + demand loop re-sent the same VA for minutes while the client crawled) + fall outside it. Widen to +/-16 pages and LEARN the winning offset per + vmap area, because consecutive pushes advance sequentially inside the + same allocation and share the offset. + """ + if not (PB_CE_LO <= va < PB_CE_HI): + return False + off = (va - PB_CE_LO) & 0xFFFFFF + k = off >> 12 + areas = _nv_vmap_areas(kc) + for (lo, _hi, pages) in areas: + learned = _area_off.get(lo) + if learned is not None: + adjs = [learned] + else: + adjs = [0, 1, -1] + adjs += [d for m in (2, 4, 8, 16) for d in (m, -m)] + for adj in adjs: + idx = k + adj + if idx < 0 or idx >= pages: + continue + cpu = lo + idx * 0x1000 + d = kc.read(cpu, 64) + if not d or not any(d[:32]): + continue + if not _push_prefix_ok(d): + continue + pa = kc.va_to_pa(cpu) + if pa is None: + continue + _area_off[lo] = adj + out.append("0x%x,0x%x,1\n" % (va & ~0xFFF, pa)) + dbg("demand-va: 0x%x -> pa=0x%x (area off=%d adj=%+d)" + % (va & ~0xFFF, pa, idx, adj)) + return True + dbg("demand-va: 0x%x not found by content (%d areas)" + % (va & ~0xFFF, len(areas))) + return False + + def resolve_mem_pins(kc, mem, out, default_pages=16, va_lo=0, va_hi=0, page_off=0): """uvm_rm_mem kptr -> GPU-VA/GPA runs (direct map or page-table walk). @@ -400,6 +497,121 @@ def mem_page_off(kc, mem): return 0 +def _ce_mem_pages(kc, mem): + """CE mem walk parameters: (gpus, cpu, off, npages) or None. + + Walks the same page set resolve_mem_pins used: the whole vmalloc area + (minus the header offset) when known, else size at +0x128, else the + default. The area governs: the live pushes are written sequentially and + may run past the size field, so a size-derived prefix stopped one page + short of the write frontier (r144: last page 0x120030000 emitted, the + live page 0x120031000 never in any pin set). all CE-window GPU VAs share + the CPU mapping. + """ + vas = [kc.read64(mem + 0x10 + i * 8) for i in range(16)] + cpu = next((v for v in vas if is_kern_ptr(v)), None) + gpus = [v for v in vas if v and PB_CE_LO <= v < PB_CE_HI] + if not cpu or not gpus: + return None + if _direct_map_pa(kc, cpu) is not None: + return None # never a vmap: not the push-buffer shape + off = mem_page_off(kc, mem) + area = _vmalloc_pages_for(kc, cpu) + size = _mem_size(kc, mem) + size_pages = max(1, min(size >> 12, 4098)) if size else 0 + area_pages = area - off if area and area > off else 0 + npages = min(max(size_pages, area_pages) or 64, 4098) + return (gpus, cpu, off, npages) + + +def _emit_ce_mem_delta(kc, st, out, fresh_map=False): + """Emit the CE mem's live {va,pa} pairs as merged runs. + + Two emission modes share one merge step: + + * full re-assert (KC_FULLREM_S cadence, and every fresh_map pass): every + page that translates is emitted, content-independent. This is the + fix for the host's session flush: at a client switch the host wipes + its whole sema map when the previous client's last channel closes + ("session flush (all channels closed)"), and a pin batch that raced + in for the new client is lost. The old content-gated delta never + re-emitted those unchanged pairs, so the new client's CE pushes were + permanently unreadable ("NO ENTRY" in sema-probe) and every GPFIFO + entry was skipped after the 8 s grace (r144/r145 second-client stall, + both mail and matmul). Re-asserting the full pair set every couple of + seconds heals any number of flush wipes; merged contiguous runs keep + the traffic to a few lines per pass. Zero-content pages are pinned + deliberately: the pin carries the translation, the host reads the + live content, and the client writes it later. + + * delta (between full passes): content-gated, emits pages whose PA is + new or changed, so a remap or a freshly-faulted page is picked up + without waiting for the next full pass. Never drops a previously + emitted pair (the host's last-store-wins pin sees the truth, and a + failed burst never clears the whole map). + """ + gpus, cpu, off, npages = st["gpus"], st["cpu"], st["page_off"], st["npages"] + last = st["last"] + nowm = time.monotonic() + full = nowm >= st.get("full_next", 0.0) + if full: + age = nowm - st.get("resolved_at", nowm) + st["full_next"] = nowm + (KC_FULLREM_S if age < KC_FULLREM_FAST_S + else KC_FULLREM_SLOW_S) + if fresh_map: + for k in range(npages): + kc._pa_cache.pop(cpu + (k + off) * 0x1000, None) + changed = [] + if full: + # Re-walk pages that did not translate on an earlier pass: the + # vmap's leaf sizes are huge (2 MB/1 GB), so most pages translate + # immediately; a straggler (the vmap populated lazily) is retried + # with its negative PA-cache entry popped, bounded per pass so the + # sweep stays cheap. + miss = st.setdefault("miss", set()) + retry = 0 + for k in range(npages): + page = cpu + (k + off) * 0x1000 + pa = kc.va_to_pa(page) + if pa is None: + if k in miss and retry < 512: + kc._pa_cache.pop(page, None) + retry += 1 + else: + miss.add(k) + continue + miss.discard(k) + for gv in gpus: + changed.append((gv + k * 0x1000, pa)) + else: + for k in range(npages): + d = kc.read(cpu + (k + off) * 0x1000, 32) + if not d or not any(d): + continue # zero/unreadable: nothing to pin yet + pa = kc.va_to_pa(cpu + (k + off) * 0x1000) + if pa is None: + continue + for gv in gpus: + if last.get(gv + k * 0x1000) != pa: + changed.append((gv + k * 0x1000, pa)) + changed.sort() + merged = [] + for va, pa in changed: + if merged and va == merged[-1][0] + merged[-1][2] * 0x1000 and \ + pa == merged[-1][1] + merged[-1][2] * 0x1000: + merged[-1][2] += 1 + else: + merged.append([va, pa, 1]) + for va, pa, n in merged: + out.append("0x%x,0x%x,%d\n" % (va, pa, n)) + for va, pa in changed: + last[va] = pa + st["emit_total"] = st.get("emit_total", 0) + len(merged) + if full: + st["full_total"] = st.get("full_total", 0) + 1 + return len(merged) + + def _emit_ring_pins(kc, gpu_va, entries_cpu, entries, out): """Pin the GPFIFO ring pages (and the pages after it, where the GPPUT buffer usually sits) from the ring's CPU mapping.""" @@ -522,6 +734,49 @@ def resolve_sem_pins(kc, sem, out): return resolve_mem_pins(kc, mem, out, default_pages=16) +def resolve_pool_va(kc, va, out, sems): + """Pool/waiter VA -> PA via tracker-sem -> page -> mem; NO content match. + + The word at POOL base + 4*idx is a slot in a UVM semaphore-pool page: a + sysmem uvm_rm_mem whose PA is only reachable through its kernel CPU + alias (the client pagemap has no present entry for it). Same kcore walk + the CE pushbuffer uses; the chain is shorter (sem+0 = page, page+0 = + mem). The page sits 16 MB above the CE push buffer base, outside + resolve_sem_pins' default_pages window, which is why the host saw + 'pending release starved' for it on every client's first seconds. + """ + page = va & ~0xFFF + if not (POOL_SEM_LO <= page < POOL_SEM_HI): + return False + for sem in list(sems): + if not is_kern_ptr(sem): + continue + try: + pg = kc.read64(sem) + if not is_kern_ptr(pg): + continue + mem = kc.read64(pg) + if not is_kern_ptr(mem): + continue + got = [] + resolve_mem_pins(kc, mem, got, default_pages=1, + va_lo=page, va_hi=page + 0x1000) + except Exception: + continue + for line in got: + try: + lva = int(line.split(',')[0], 16) + except Exception: + continue + if lva == page: + out.append(line) + dbg('pool-va: 0x%x -> %s (mem=0x%x)' + % (page, line.strip(), mem)) + return True + dbg('pool-va: 0x%x unresolved (%d known sems)' % (page, len(sems))) + return False + + def resolve_pb_pins(kc, pb, out): mem = kc.read64(pb + 8) if not mem: @@ -555,12 +810,35 @@ def resolve_pb_pins(kc, pb, out): # hold the ring-entry pushes is chosen (content anchor, _mem_content_ok). PB_CE_LO = 0x120000000 PB_CE_HI = 0x121000000 +# m165 (D-F1a): UVM semaphore-pool page range. The pool is a 4 KB sysmem +# uvm_rm_mem placed by the UVM arena right after the 16 MB CE push buffer; +# the client's tracker words (0x121000000 + 4*idx) live in its first page. +# The client pagemap has no present entry for it, so only the kcore +# tracker-sem -> page -> mem walk can translate it. +POOL_SEM_LO = 0x121000000 +POOL_SEM_HI = 0x122000000 +KC_POOL_REEMIT_S = 2.0 # re-assert the pool page pair this often too KC_CHAIN_BACK = 0x2000 KC_CHAIN_SPAN = 0xA000 KC_CHAIN_MAX = 400 RESOLVED_KC_PB = set() -KC_PB_STATE = {} # sem kptr -> resolved-mem fingerprint (freshness) +KC_PB_STATE = {} # sem kptr -> resolved-mem state (fingerprint + pins) KC_REFRESH_S = 2.0 # re-validate a resolved CE mem at most this often +KC_REEMIT_S = 2.0 # sweep the CE mem's live pages this often +KC_REMAP_S = 30.0 # full re-translation sweep (catches remaps) +# Full re-assert cadence: every pass at least this far apart re-emits the +# whole translatable pair set regardless of the delta, healing host-side +# session-flush wipes (see _emit_ce_mem_delta). Must stay well under the +# host's gsp_sema_ce_grace_ms (8 s) so a wiped-and-needed page is restored +# inside the hold window. The fast cadence applies to a mem's first +# KC_FULLREM_FAST_S seconds (the flush race lives at the client switch); +# after that the re-assert backs off to KC_FULLREM_SLOW_S - a fully +# fragmented pair set (descending PAs) emits one page per line, and 4 k +# lines every 2 s saturated a guest under multi-client load (r148: load 19, +# three scanners from an install race, clients wedged). +KC_FULLREM_S = float(os.environ.get("STREAM_FULLREM_S", "1.5")) +KC_FULLREM_FAST_S = float(os.environ.get("STREAM_FULLREM_FAST_S", "45")) +KC_FULLREM_SLOW_S = float(os.environ.get("STREAM_FULLREM_SLOW_S", "10.0")) def _chain_mems_at(kc, w): @@ -639,6 +917,16 @@ def _kc_pb_fresh(kc, sem): return False if not st.get("page_va"): return True + if not st.get("page_pa"): + # Unknown PA at store time: the manual page-table walk could not + # translate the vmap page (it may have been read before the kernel + # page tables were walkable). Treat the record as fresh instead of + # failing every check - a permanently-failing freshness test made + # the scanner drop the latch and re-resolve in a tight loop, and + # each failed re-resolution left the host with no pins at all + # (observed r131: "kc-chain stale: re-resolving" every second for + # five sems while smiley sat in cudaMalloc). + return True pa = kc.va_to_pa(st["page_va"]) if pa is None or pa != st.get("page_pa"): return False @@ -669,6 +957,42 @@ def resolve_pb_chain(kc, sem, out): return False words = struct.unpack_from("<%dQ" % (len(data) // 8), data, 0) cands = list(dict.fromkeys(w for w in words if is_kern_ptr(w))) + # Channel-anchored expansion (r134 live dump): the tracking sem is a + # member of a uvm_channel_t table; the channel records sit in the sem's + # own window and carry their CE ring GPU VA ([0x121010000, 0x121400000)) + # plus back-to-back 32-bit object handles. The pool/manager/pushbuffer + # pointers hang off those records' targets, which the bare sem-window + # scan never reaches - the second-client stall is exactly this: the only + # live uvmt event is the spinning sem, and no mem candidate is found + # around it. For each ring-VA word, scan the surrounding record for + # kptrs, take those targets, and try the chain shape on both. Additive + # and bounded; content scoring still decides, so a wrong candidate + # cannot shadow the live pages. + extra = [] + for i, w in enumerate(words): + if not w or not (0x121010000 <= w < 0x121400000): + continue + lo = max(0, i - 0x60) + hi = min(len(words), i + 0x60) + for c in words[lo:hi]: + if not is_kern_ptr(c): + continue + extra.append(c) + tgt = kc.read(c, 0x100) + if tgt: + tw = struct.unpack_from("<%dQ" % (len(tgt) // 8), + tgt, 0) + for w2 in tw: + if is_kern_ptr(w2): + extra.append(w2) + if len(extra) >= 512: + break + if extra: + head = [] + for w in extra: + if w not in head and w not in cands: + head.append(w) + cands = head + cands mems = [] for w in cands[:KC_CHAIN_MAX]: for mem in _chain_mems_at(kc, w): @@ -697,18 +1021,24 @@ def resolve_pb_chain(kc, sem, out): if mem in RESOLVED_KC_PB: return True RESOLVED_KC_PB.add(mem) - off = mem_page_off(kc, mem) - n = resolve_mem_pins(kc, mem, out, default_pages=64, - va_lo=PB_CE_LO, va_hi=PB_CE_HI, - page_off=off) - st = {"mem": mem, "gpu": gpu, "cpu": cpu, - "page_va": cpu, "page_pa": kc.va_to_pa(cpu) or 0, "page_off": 0} + params = _ce_mem_pages(kc, mem) + if not params: + return found + gpus, cpu, off, npages = params + st = {"mem": mem, "gpu": gpu, "cpu": cpu, "gpus": gpus, + "page_va": cpu, "page_pa": kc.va_to_pa(cpu) or 0, + "page_off": off, "npages": npages, "last": {}, + "resolved_at": time.monotonic()} + # Fresh translation for the first sweep: the Kcore PA cache may hold + # pairs from before this mem's vmap area was (re)populated. + kc._pa_cache.clear() + n = _emit_ce_mem_delta(kc, st, out) KC_PB_STATE[sem] = st try: with open('/tmp/guest_pin_stream.log', 'a') as f: f.write('%s kc-chain CE push-buffer mem=0x%x sem=0x%x ' - 'pin-runs=%d off=%d size=%d score=%d\n' - % (time.strftime('%T'), mem, sem, n, off, + 'pin-runs=%d off=%d npages=%d size=%d score=%d\n' + % (time.strftime('%T'), mem, sem, n, off, npages, _mem_size(kc, mem), _score)) except OSError: pass @@ -975,6 +1305,75 @@ def new_sems(seen): return found +# --------------------------------------------------------------- scrubber +# PMA scrubber channel discovery (third-client scrub freeze). The guest's +# PMA scrubber is an internal CeUtils CE channel (4096 GPFIFO entries, +# pbGpuVA 0x120000000, ring at pbGpuVA+0x64000, finish payload at +# pbGpuVA+0x6c004). It lives in a kernel vmap area, not in any user pagemap, +# and its GPU VAs collide numerically with the client CE push-buffer window +# in the host's global translation map - so the host cannot resolve it and +# must be told the finish/host-semaphore word PAs directly. Resolve them +# here (vmallocinfo + manual kernel page-table walk) and emit one +# #scrub <fin_pa>,<hsem_pa> +# line; pin_gr_stream.py registers them through sema_scrubbuf_add. +SCRUB_AREA_SIZES = (0x6e000, 0x6d000, 0x6c000, 0x6b000, 0x6f000, + 0x70000, 0x71000, 0x72000) +_scrub_sent = {} + + +def _scrub_ring_ok(kc, ring_pa): + """A 906F GPFIFO ring page: all-zero (fresh) or entries whose GET names a + push inside the channel's own 0x120000000 window (encoded as + push_va - 0x100000000, 4-byte aligned).""" + buf = kc.read(kc.page_offset_base + ring_pa, 0x1000) + if not buf: + return False + for off in range(0, 0x100, 8): + e0, _e1 = struct.unpack_from('<II', buf, off) + if e0 == 0: + continue + if not (0x20000000 <= e0 < 0x20080000) or (e0 & 3): + return False + return True + + +def scrub_scan(kc, lines): + """Emit the scrubber buffer words' PAs once (re-emitted if they move).""" + try: + f = open('/proc/vmallocinfo') + except OSError: + return + with f: + for line in f: + if 'nvidia' not in line or 'ioremap' in line or 'vmap' not in line: + continue + parts = line.split() + try: + a, b = parts[0].split('-') + lo, hi = int(a, 16), int(b, 16) + except (ValueError, IndexError): + continue + if (hi - lo) not in SCRUB_AREA_SIZES: + continue + fin_pa = kc.va_to_pa(lo + 0x6c000) + ring_pa = kc.va_to_pa(lo + 0x64000) + if fin_pa is None or ring_pa is None: + continue + if not _scrub_ring_ok(kc, ring_pa): + dbg('scrub candidate 0x%x rejected (ring shape)' % lo) + continue + fin = fin_pa + 4 + hsem = fin_pa + key = (fin, hsem) + if _scrub_sent.get('key') == key: + return + _scrub_sent['key'] = key + lines.append('#scrub 0x%x,0x%x\n' % (fin, hsem)) + dbg('scrub buffer emitted lo=0x%x fin=0x%x hsem=0x%x' + % (lo, fin, hsem)) + return + + def nvidia_fds(pid): """CUDA device fds this pid holds: 'uvm', 'gpu', or None. @@ -999,6 +1398,34 @@ def nvidia_fds(pid): return kind +def pid_starttime(pid): + """Starttime (clock ticks, field 22) of /proc/<pid>, or None. + + Part of the client identity (pid, starttime): pids are recycled, so a + per-client reset keyed on the pid alone can mis-fire on a recycled pid. + Parsed after the last ')' so comm names with spaces/parens are safe.""" + try: + with open('/proc/%d/stat' % pid) as f: + st = f.read() + rp = st.rfind(')') + if rp < 0: + return None + toks = st[rp + 2:].split() + if len(toks) < 20: # starttime = field 22 overall = index 19 + return None + return int(toks[19]) + except (OSError, ValueError): + return None + + +def pid_line(tag, pid, start): + """'#<tag> <pid> [starttime]' - the optional starttime keeps the line + backward compatible with older hosts (they parse the pid only).""" + if start is None: + return "#%s %d\n" % (tag, pid) + return "#%s %d %d\n" % (tag, pid, start) + + def cdt_pid(): prefixes = tuple(p for p in os.environ.get("CLIENT_PREFIXES", "cdt,va,victory").split(",") if p) @@ -1206,6 +1633,8 @@ def main(): resolved_kc_pb = set() retry_at = {} kc_refresh_at = {} + kc_reemit_at = {} + kc_remap_at = {} tracer = TraceReader() kc = Kcore() kc_ok = kc.load() @@ -1216,13 +1645,34 @@ def main(): except OSError: pass last_pid = None + last_start = None last_log = 0.0 + scrub_next = 0.0 + demand_next = 0.0 + # m165 (D-F1a): pool page VAs learned from demand, re-asserted every + # KC_POOL_REEMIT_S. Boot-stable (UVM frees pool pages only at GPU + # deinit), so the set deliberately persists across client switches. + pool_pages = {} + pool_reemit_next = 0.0 + _want_state = {"mtime": None, "done": set()} loop = 0 while True: pid = cdt_pid() loop += 1 if pid is None: + if last_pid is not None and sock is not None: + # In-band client-death signal: the host tools purge the dead + # client's per-client state on this line, so the NEXT client + # works without a VM cycle (the kill cascade). Best-effort - + # if the socket is gone the host still has its own probe. + try: + sock.sendall(pid_line("nopid", last_pid, + last_start).encode()) + except OSError: + sock.close() + sock = None last_pid = None + last_start = None seen = {} seen_sems = set() seen_pbs = set() @@ -1231,6 +1681,7 @@ def main(): continue if pid != last_pid: last_pid = pid + last_start = pid_starttime(pid) seen = {} seen_sems = set() seen_pbs = set() @@ -1242,6 +1693,8 @@ def main(): resolved_kc_pb = set() retry_at = {} kc_refresh_at = {} + kc_reemit_at = {} + kc_remap_at = {} # Client switch: the old client's channel/put pointers are freed # memory. Clearing the #put map stops stale reads from being # emitted as if they were the new client's counters. @@ -1259,7 +1712,7 @@ def main(): dbg("switch seed: +%d event kptr(s)" % _seeded) if sock is not None: try: - sock.sendall(("#pid %d\n" % pid).encode()) + sock.sendall(pid_line("pid", pid, last_start).encode()) except OSError: sock.close() sock = None @@ -1293,6 +1746,84 @@ def main(): new = [(va, pa) for va, pa in cur.items() if seen.get(va) != pa] pending_cur = cur lines = [] + # Demand pins (host -> this scanner): the host appends the CE + # push-buffer VAs of GPFIFO entries it cannot read to + # /tmp/guest_want_vas.txt (written via the guest agent by + # pin_gr_stream); resolve each by content against the + # nv_vm_map_pages areas and emit the pair. This is the + # event-independent hole filler for the stall class where the + # one-shot #sem kprobe event was lost and the object-chain + # resolver never learns the client. + if kc_ok and time.time() >= demand_next: + demand_next = time.time() + 1.0 + try: + st_d = os.stat("/tmp/guest_want_vas.txt") + if st_d.st_mtime != _want_state.get("mtime"): + with open("/tmp/guest_want_vas.txt") as fw: + wvas = [int(t, 16) for t in fw.read().split() + if t.startswith("0x")] + _want_state["mtime"] = st_d.st_mtime + for wv in wvas[-64:]: + if wv in _want_state["done"]: + continue + got = [] + if POOL_SEM_LO <= (wv & ~0xFFF) < POOL_SEM_HI: + # m165 (D-F1a): pool/waiter VAs have no method + # stream content and the CE offset math does not + # apply; route them to the structural tracker-sem + # walk instead. m168: the tracker sems are + # long-lived objects (allocated once) whose page + # pointer follows the pool across clients, so try + # every sem we have ever resolved a chain for - + # seen_sems alone can miss the live generation. + _psems = set(seen_sems) | set(KC_PB_STATE.keys()) \ + | set(resolved_kc_pb) + try: + ok = resolve_pool_va(kc, wv, got, + _psems) + except Exception: + ok = False + got = [] + dbg('demand-pool-va: 0x%x %s' + % (wv, 'ok' if ok else 'pending')) + if ok and got: + pool_pages.setdefault(wv & ~0xFFF, True) + else: + try: + ok = resolve_va_by_content(kc, wv, got) + except Exception: + ok = False + got = [] + if ok and got: + lines.extend(got) + _want_state["done"].add(wv) + except OSError: + pass + # m165 (D-F1a): re-assert the pool page pair every KC_POOL_REEMIT_S. + # The pool page is boot-stable (UVM frees it only at GPU deinit) but + # the host pin table is wiped at every session flush, so a one-shot + # pin leaves later clients starving until the slow structural walks + # answer; the periodic re-assert heals the wipe within seconds (the + # same argument as the CE mem sweep above). + if kc_ok and pool_pages and time.time() >= pool_reemit_next: + pool_reemit_next = time.time() + KC_POOL_REEMIT_S + for pv in list(pool_pages): + got = [] + try: + ok = resolve_pool_va(kc, pv, got, seen_sems) + except Exception: + ok = False + got = [] + if ok and got: + lines.extend(got) + # PMA scrubber buffer discovery (throttled): resolve the finish/host- + # semaphore word PAs and stream them for host registration. + if kc_ok and time.time() >= scrub_next: + scrub_next = time.time() + 5.0 + try: + scrub_scan(kc, lines) + except Exception as e: + dbg('scrub scan failed: %s' % e) for ctl in tracer.drain(): parts = ctl.split() if ctl.startswith("#sem") and len(parts) >= 2: @@ -1333,12 +1864,68 @@ def main(): except Exception: _fresh = True if not _fresh: - _st = KC_PB_STATE.pop(sem, None) + # Re-resolve, but never lose the latch on a failed + # attempt: a dropped latch means the host keeps no + # pins for this channel at all, which is worse than + # a possibly-stale pin (the host content-checks the + # first page and now rejects all-FF continuation + # chunks, so stale pages are bounded). On failure + # keep the old state and defer the next attempt. + _st = KC_PB_STATE.get(sem) if _st and _st.get("mem"): RESOLVED_KC_PB.discard(_st["mem"]) - resolved_kc_pb.discard(sem) - retry_at.pop(("pk", sem), None) - dbg("kc-chain stale: re-resolving sem=0x%x" % sem) + got = [] + try: + _found2 = resolve_pb_chain(kc, sem, got) + except Exception: + _found2 = False + got = [] + if _found2 and got: + _new = KC_PB_STATE.get(sem) + _newmem = _new.get("mem") if _new else None + dbg("kc-chain stale: re-resolved sem=0x%x " + "new-mem=%s runs=%d" + % (sem, "yes" if _newmem != + (_st.get("mem") if _st else None) + else "no", len(got))) + lines.extend(got) + added += len(got) + resolved_kc_pb.add(sem) + else: + if _st and _st.get("mem"): + RESOLVED_KC_PB.add(_st["mem"]) + if sem not in KC_PB_STATE: + KC_PB_STATE[sem] = _st + kc_refresh_at[sem] = now_r + 5.0 + dbg("kc-chain stale: keep old latch sem=0x%x" + % sem) + else: + # CE mem delta sweep (second-client pins): the one-shot + # emission pinned the mem once; pages the client writes + # later (or whose vmap pairing moved) were never + # re-emitted, so the host kept zeros for the other CE + # ranks' pushes. Sweep every KC_REEMIT_S, emitting only + # new/changed nonzero pairs; burst the Kcore PA cache + # every KC_REMAP_S so a remap is not shadowed. + _st = KC_PB_STATE.get(sem) + if _st is not None and \ + now_r >= kc_reemit_at.get(sem, 0): + _fresh_map = now_r >= kc_remap_at.get(sem, 0) + if _fresh_map: + kc_remap_at[sem] = now_r + KC_REMAP_S + kc_reemit_at[sem] = now_r + KC_REEMIT_S + got = [] + try: + _n2 = _emit_ce_mem_delta(kc, _st, got, + fresh_map=_fresh_map) + except Exception: + _n2 = 0 + got = [] + if got: + lines.extend(got) + added += len(got) + dbg("ce-delta sem=0x%x runs=%d mem=0x%x" + % (sem, _n2, _st.get("mem", 0))) # CE push-buffer chain (second-client stall fix): resolve # pool -> manager -> pushbuffer -> rm_mem directly, so the # push-buffer pages never depend on the racy one-shot uvbp @@ -1480,7 +2067,8 @@ def main(): sock.connect((CID, PORT)) sock.settimeout(None) if last_pid is not None: - sock.sendall(("#pid %d\n" % last_pid).encode()) + sock.sendall(pid_line("pid", last_pid, + last_start).encode()) except OSError: if sock: sock.close() diff --git a/tools/guest-diag/pin_gr_stream.py b/tools/guest-diag/pin_gr_stream.py index b864ab5..6eb5be7 100644 --- a/tools/guest-diag/pin_gr_stream.py +++ b/tools/guest-diag/pin_gr_stream.py @@ -16,6 +16,7 @@ import re import socket import subprocess import sys +import threading import time HERE = os.path.dirname(os.path.abspath(__file__)) @@ -31,12 +32,44 @@ LO = os.environ.get("PINWIN_LO", "0x200000000") HI = os.environ.get("PINWIN_HI", "0x206000000") PAMAX = os.environ.get("PINWIN_PA_MAX", "0x180000000") SEMSTREAM = os.environ.get("SEM_STREAM_FILE", "/tmp/opencode/sem_stream.txt") +# Liveness file checked by pin_supervisor.sh (mtime must stay < 75 s; the +# supervisor grants a 120 s startup grace on top). +HEARTBEAT_FILE = os.environ.get("PIN_GR_HEARTBEAT", "/tmp/opencode/pin_gr_heartbeat") +# Inter-tool re-assert request (written by semfix_uvmt.py's stall watchdog): +# when its mtime moves, re-assert every pair this tool has ever pinned. +REASSERT_REQ = os.environ.get("PIN_REASSERT_FILE", + "/tmp/opencode/pin_reassert_request") +# Stall watchdog: a known client with no pin/data activity for this long gets +# an automatic recovery (full re-assert + demand of the fixed counter/pool +# VAs). Pre-capture (no pin yet for this client) uses the tighter limit. +STALL_PRE_S = float(os.environ.get("PIN_STALL_PRE", "60")) +STALL_POST_S = float(os.environ.get("PIN_STALL_POST", "120")) +STALL_RETRY_S = float(os.environ.get("PIN_STALL_RETRY", "30")) +STALL_ESCALATE_S = float(os.environ.get("PIN_STALL_ESCALATE", "180")) def log(msg): print("[%s] %s" % (time.strftime("%H:%M:%S"), msg), flush=True) +def _heartbeat(): + try: + with open(HEARTBEAT_FILE, "w") as f: + f.write("%.3f\n" % time.time()) + except OSError: + pass + + +def _hb_loop(): + """Refresh the heartbeat on a fixed cadence, independent of data-path + activity: recv can be legally idle for minutes between client phases and + the supervisor used to read that silence as a hung tool (r198 restarts at + 34 s staleness while injection bursts were running).""" + while True: + _heartbeat() + time.sleep(5) + + def _mirror(line): """Append guest control lines (#pid/#sem) to the shared host file so the single-QMP watcher can consume them without guest-agent round trips.""" @@ -56,18 +89,50 @@ def vexec(cmd, timeout=40): return "" +def _scanner_alive(): + """True when the in-guest scanner process still exists. + + install_and_start() pkill's and restarts the scanner, which resets every + in-guest latch; probing first avoids killing a healthy scanner that is + merely silent because no client exists yet (r198 pre-client churn).""" + out = vexec("sudo sh -c 'kill -0 $(cat /tmp/guest_pin_stream.pid) " + "2>/dev/null && echo ALIVE'") + return "ALIVE" in (out or "") + + +def _demand_send(vas): + """Append VA hints to the guest scanner's demand file (best-effort). + + Same channel pin_gr_stream._demand_loop uses for host-dmesg stalls; the + stall watchdog uses it to ask the scanner to re-resolve the fixed + pre-capture VAs (counter page / UVM pool) by content/pagemap.""" + lines = "".join("0x%x\n" % va for va in vas) + try: + vexec("printf '%%s' '%s' >> /tmp/guest_want_vas.txt" % lines, 30) + except Exception: + pass + + def _start_cmd(): with open(GUEST_SCRIPT) as f: b64 = subprocess.run(["base64", "-w0"], stdin=f, - capture_output=True, text=True).stdout + capture_output=True, text=True, + timeout=10).stdout prefixes = os.environ.get("CLIENT_PREFIXES", "cdt,va,victory") vma_every = os.environ.get("STREAM_VMA_EVERY", "1") - return ("echo '%s' | base64 -d > /tmp/guest_pin_stream.py; " - "if [ -f /tmp/guest_pin_stream.pid ]; then " - "p=$(cat /tmp/guest_pin_stream.pid 2>/dev/null); " - "[ -n \"$p\" ] && grep -q guest_pin_stream " - "/proc/$p/cmdline 2>/dev/null && sudo kill $p 2>/dev/null; " - "fi; " + # Kill EVERY existing scanner, not just the pid-file one: a retry after + # a partially-failed first attempt (pid file not written yet) used to + # leave the first scanner alive, and multiple scanners each re-assert + # the whole CE pair set - r148 ran three and drove the guest to load 19 + # (clients wedged in userspace spins). The pattern is anchored + # (^python3) and assembled at runtime via printf: an unanchored pattern + # or a literal one would also match this command's own shell (its + # cmdline contains the script path), and pkill would kill the shell + # before it could start the new scanner - every start attempt then + # failed silently (r149). + return ("P=$(printf '^python3 /tmp/guest_pin_%%s.py' stream); " + "sudo pkill -f \"$P\" 2>/dev/null; sleep 1; " + "echo '%s' | base64 -d > /tmp/guest_pin_stream.py; " "sudo rm -f /tmp/guest_pin_stream.pid; " "sudo setsid nohup env CLIENT_PREFIXES='%s' " "STREAM_VMA_EVERY='%s' " @@ -82,6 +147,7 @@ def install_and_start(): # or a stale qga client); retry a few times instead of giving up, because # a boot without the scanner streams no pins and the client then hangs. for attempt in range(5): + _heartbeat() out = vexec(_start_cmd(), 60) if any(ch.isdigit() for ch in out): return True @@ -93,78 +159,297 @@ def install_and_start(): def inject(va, pa, ln): cmd = "printf '0x%x,0x%x,0x%x,1' > %s/sema_map_add" % (va, pa, ln, PARAM) for _ in range(60): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=10) + except subprocess.TimeoutExpired: + time.sleep(0.05) + continue + if r.returncode == 0: + return True + time.sleep(0.05) + return False + + +def scrub_register(fin_pa, hsem_pa): + """Register the PMA scrubber buffer words (guest-resolved PAs) with the + service so it can author the scrub progress (third-client scrub freeze).""" + cmd = ("printf '0x%x,0x%x' > %s/sema_scrubbuf_add" + % (fin_pa, hsem_pa, PARAM)) + for _ in range(60): + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=10) + except subprocess.TimeoutExpired: + time.sleep(0.05) + continue if r.returncode == 0: + log("scrub registered fin=0x%x hsem=0x%x" % (fin_pa, hsem_pa)) return True time.sleep(0.05) + log("scrub register FAILED fin=0x%x hsem=0x%x" % (fin_pa, hsem_pa)) return False +def _demand_loop(): + """Push the host's unreadable CE push VAs to the guest scanner. + + The service logs each distinct stalled VA once per boot ('stalled VA + #N: 0x...'). Each sighting is tracked and re-sent every 20 s for five + minutes: the guest resolves the VA by content against the + nv_vm_map_pages areas (event-independent, no object chain), so a page + that was not yet written when first asked is picked up on a retry. + This closes the stall class where the one-shot #sem kprobe event was + lost and the chain resolver never learns the client. + """ + track = {} # va -> [first_seen, last_sent] + while True: + time.sleep(3) + try: + r = subprocess.run(["sudo", "-n", "dmesg"], + capture_output=True, text=True, timeout=20) + except Exception: + continue + now = time.time() + for m in re.finditer(r"(?:stalled VA #\d+[:=]\s*|pending release starved va=)(0x[0-9a-f]+)", + (r.stdout or "")[-200000:]): + va = int(m.group(1), 16) + track.setdefault(va, [now, 0.0]) + for va in list(track): + first, last = track[va] + if now - first > 300.0: + del track[va] + todo = [va for va, (f, l) in track.items() if now - l > 20.0] + if not todo: + continue + lines = "".join("0x%x\n" % va for va in todo) + try: + vexec("printf '%%s' '%s' >> /tmp/guest_want_vas.txt" % lines, 30) + except Exception: + continue + for va in todo: + track[va][1] = now + log("demand: sent %d stalled VA(s) to guest" % len(todo)) + + +_DEMAND_STARTED = False +_HB_STARTED = False +_REASSERT_REQ_SEEN = 0.0 + + def main(): + global _DEMAND_STARTED, _HB_STARTED + _heartbeat() + if not _HB_STARTED: + _HB_STARTED = True + try: + threading.Thread(target=_hb_loop, daemon=True).start() + except Exception as e: + log("WARN: heartbeat thread failed to start: %s" % e) if not install_and_start(): log("WARN: could not start guest scanner (will retry via vsock wait)") + if not _DEMAND_STARTED: + _DEMAND_STARTED = True + try: + threading.Thread(target=_demand_loop, daemon=True).start() + except Exception as e: + log("WARN: demand thread failed to start: %s" % e) srv = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM) srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) srv.bind((socket.VMADDR_CID_ANY, PORT)) + srv.settimeout(10) srv.listen(4) log("listening on vsock port %d (window %s..%s)" % (PORT, LO, HI)) seen = {} pin_total = 0 - buf = b"" + last_conn = time.time() + last_reinstall = 0.0 + # Stall-watchdog state for the connected client. client_pid is the + # scanner-reported pid (None until a client appears); last_activity marks + # the last pin/data/control event; pins_since_client counts pages pinned + # for the current client (0 = pre-capture). + client_pid = None + client_start = None + last_activity = time.time() + pins_since_client = 0 + stall_since = None + recover_next = 0.0 + escalated = False + no_conn_warn_next = 0.0 + + def _activity(): + nonlocal last_activity, stall_since, escalated + last_activity = time.time() + stall_since = None + escalated = False + while True: try: conn, _ = srv.accept() + except socket.timeout: + now = time.time() + # No guest scanner connection for 30 s: it died or never started + # (a boot without it streams no pins and every client then hangs). + # Probe the scanner first instead of reinstalling blindly: + # install_and_start() pkill's a live scanner and resets all + # in-guest latches, and pre-client silence is normal (r198 churn). + if now - last_conn > 30.0 and now - last_reinstall >= 60.0: + last_reinstall = now + if _scanner_alive(): + log("no guest connection for %.0fs but scanner alive; " + "not reinstalling" % (now - last_conn)) + if now - last_conn > 180.0 and now >= no_conn_warn_next: + no_conn_warn_next = now + 60.0 + log("STALL WATCHDOG: no guest scanner connection for " + "%.0fs (scanner process alive) - pins/demand may " + "be stranded" % (now - last_conn)) + continue + log("reinstall attempt: no guest connection for %.0fs" + % (now - last_conn)) + try: + install_and_start() + except Exception as e: + log("reinstall failed: %s" % e) + continue except OSError as e: log("accept failed: %s" % e) time.sleep(1) continue + last_conn = time.time() + _heartbeat() conn.settimeout(5) log("guest connected") buf = b"" runs = [] - while True: - try: - chunk = conn.recv(1 << 20) - except socket.timeout: + try: + while True: + try: + chunk = conn.recv(1 << 20) + except socket.timeout: + if runs: + novel = _novel_pages(runs, seen) + pages = _flush(runs, seen) + pin_total += pages + pins_since_client += pages + if novel: + _activity() + runs = [] + _maybe_reassert_request(seen) + now = time.time() + # Stall watchdog: a known client with no pin/data + # activity gets a bounded, idempotent recovery (re-assert + # everything + demand the fixed counter/pool VAs), then a + # loud escalation if the stall persists. This is the + # "first cudaMalloc / first GR capture must not stall + # silently" guard. recover_next rate-limits attempts (the + # loop wakes every 5 s on the recv timeout). + if client_pid is not None: + limit = STALL_PRE_S if pins_since_client == 0 \ + else STALL_POST_S + idle = now - last_activity + if idle >= limit and now >= recover_next: + recover_next = now + STALL_RETRY_S + if stall_since is None: + stall_since = now + log("STALL WATCHDOG: client pid %d - no pin " + "activity for %.0fs (pre-capture=%s) - " + "re-asserting all known pairs + demanding " + "counter/pool VAs" + % (client_pid, idle, pins_since_client == 0)) + if seen: + _flush([], seen, full=True) + _demand_send([0x20400f000, 0x121000000]) + if not escalated and stall_since is not None \ + and now - stall_since >= STALL_ESCALATE_S: + escalated = True + log("STALL WATCHDOG: recovery FAILED for pid %d " + "after %.0fs - stall persists, needs manual " + "attention" % (client_pid, now - stall_since)) + continue + except OSError: + break + if not chunk: + break + _heartbeat() + buf += chunk + while b"\n" in buf: + line, buf = buf.split(b"\n", 1) + text = line.decode(errors="replace").strip() + if not text: + continue + if text.startswith("#pid"): + m = re.match(r"#pid\s+(\d+)(?:\s+(\d+))?", text) + seen = {} + client_pid = int(m.group(1)) if m else None + client_start = (int(m.group(2)) if m and m.group(2) + else None) + pins_since_client = 0 + _activity() + log("guest reports cdt pid %s%s" + % (client_pid if client_pid is not None else "?", + (" start=%s" % client_start + if client_start is not None else ""))) + _mirror(text) + continue + if text.startswith("#nopid"): + # In-band client death: the guest scanner tells us the + # client is gone (with its identity). Drop the per-pair + # dedupe map so the next client's pairs are injected + # unconditionally; semfix_uvmt consumes the mirrored + # line and purges its per-client state too. + client_pid = None + client_start = None + seen = {} + pins_since_client = 0 + _activity() + log("guest reports client gone (%s)" % text) + _mirror(text) + continue + if text.startswith("#sem"): + _mirror(text) + _activity() + continue + if text.startswith("#pb"): + _mirror(text) + _activity() + continue + if text.startswith("#put"): + log(text) + _mirror(text) + _activity() + continue + if text.startswith("#scrub"): + m = re.search(r"0x([0-9a-fA-F]+),0x([0-9a-fA-F]+)", + text) + _mirror(text) + if m: + scrub_register(int(m.group(1), 16), + int(m.group(2), 16)) + continue + m = re.match(r"^(0x[0-9a-f]+),(0x[0-9a-f]+)(?:,(\d+))?$", + text) + if m: + runs.append((int(m.group(1), 16), + int(m.group(2), 16), + int(m.group(3)) if m.group(3) else 1)) if runs: - _flush(runs, seen) + novel = _novel_pages(runs, seen) + pages = _flush(runs, seen) + pin_total += pages + pins_since_client += pages + if novel: + _activity() runs = [] - continue - except OSError: - break - if not chunk: - break - buf += chunk - while b"\n" in buf: - line, buf = buf.split(b"\n", 1) - text = line.decode(errors="replace").strip() - if not text: - continue - if text.startswith("#pid"): - m = re.search(r"(\d+)", text) - seen = {} - log("guest reports cdt pid %s" % (m.group(1) if m else "?")) - _mirror(text) - continue - if text.startswith("#sem"): - _mirror(text) - continue - if text.startswith("#pb"): - _mirror(text) - continue - if text.startswith("#put"): - log(text) - _mirror(text) - continue - m = re.match(r"^(0x[0-9a-f]+),(0x[0-9a-f]+)(?:,(\d+))?$", text) - if m: - runs.append((int(m.group(1), 16), int(m.group(2), 16), - int(m.group(3)) if m.group(3) else 1)) - if runs: - pin_total += _flush(runs, seen) - runs = [] + # Poll the semfix re-assert request on every recv cycle, not + # only on recv-timeouts: sustained scanner traffic starves the + # timeout branch, so the request fired post-mortem in r203 + # (119 s late, `seen` already cleared by #nopid). + _maybe_reassert_request(seen) + except Exception as e: + # A transient error on one connection must not kill the process: + # log it, drop the connection and go back to accept(). + log("connection handler error: %s (continuing)" % e) log("guest disconnected (pins this session: %d)" % pin_total) try: conn.close() @@ -172,19 +457,82 @@ def main(): pass -def _flush(runs, seen): +def _maybe_reassert_request(seen): + """Honour a full-re-assert request written by semfix_uvmt's watchdog.""" + global _REASSERT_REQ_SEEN + try: + m = os.stat(REASSERT_REQ).st_mtime + except OSError: + return + if m > _REASSERT_REQ_SEEN: + _REASSERT_REQ_SEEN = m + if seen: + log("semfix requested a full pin re-assert - re-asserting %d " + "pair(s)" % len(seen)) + _flush([], seen, full=True) + + +def _coalesce_seen(seen): + """Coalesce the seen {va: pa} per-page map into (va, pa, npages) runs.""" + items = sorted(seen.items()) + runs = [] + i = 0 + while i < len(items): + va, pa = items[i] + n = 1 + while (i + n < len(items) + and items[i + n][0] == va + n * 0x1000 + and items[i + n][1] == pa + n * 0x1000): + n += 1 + runs.append((va, pa, n)) + i += n + return runs + + +def _novel_pages(runs, seen): + """Count pages in runs whose translation is new or changed. + + Progress accounting for the stall watchdog: the guest re-streams the CE + window and UVM pool every couple of seconds even when nothing moves, so a + pure repeat of an already-known pair must not count as pin activity + (r203: those repeats masked the watchdog through every hang).""" + return sum(1 for va, pa, n in runs for i in range(n) + if seen.get(va + i * 0x1000) != pa + i * 0x1000) + + +def _flush(runs, seen, full=False): """Inject run lines (one batched sudo pass), return number of pages pinned. The old implementation spawned one sudo process per run; with ~400 runs that cost ~4 s, which landed the GR pins after the client's context-init deadline. The batch writes all runs in a single shell, retrying only the ones the kernel queue rejected (ENOSPC) individually. + + full=True re-asserts every pair ever pinned (coalesced), not just this + batch's runs: the kernel session flush wipes the pin table when a client's + channels close, and the stall watchdog / semfix re-assert request use this + to heal it without touching the guest. Writes are idempotent. """ - todo = [] + todo = {} for va, pa, n in runs: - if seen.get(va) == pa and n == 1: + # The CE window (0x120000000..0x121000000) and the UVM semaphore + # pool page range (0x121000000..0x122000000) are re-asserted by the + # guest every couple of seconds on purpose: the host's session flush + # wipes the whole pin table when the previous client's last channel + # closes, and a pin batch that raced in for the new client must be + # allowed to land again even when the pair is unchanged. Skipping + # repeats there stranded the second client's CE pushes (the r144/ + # r145 stall: sema-probe "NO ENTRY" for the live push pages) and the + # pool/tracker page (pending release starved, every client's first + # seconds). + ce_win = 0x120000000 <= va < 0x122000000 + if not full and not ce_win and seen.get(va) == pa and n == 1: continue - todo.append((va, pa, n)) + todo[(va, pa, n)] = True + if full: + for va, pa, n in _coalesce_seen(seen): + todo[(va, pa, n)] = True + todo = list(todo) if not todo: return 0 @@ -202,6 +550,7 @@ def _flush(runs, seen): CHUNK = 200 for start in range(0, len(script), CHUNK): batch = script[start:start + CHUNK] + _heartbeat() try: r = subprocess.run(["sudo", "-n", "sh", "-c", "\n".join(batch)], capture_output=True, text=True, timeout=60) @@ -235,4 +584,12 @@ def _flush(runs, seen): if __name__ == "__main__": - main() + # main() loops forever; only an unexpected exception exits it. Restart + # instead of dying silently - a dead listener means no pins and every + # later CUDA client hangs at first cudaMalloc. + while True: + try: + main() + except Exception as e: + log("FATAL: main() exited: %r - restarting in 2s" % (e,)) + time.sleep(2) diff --git a/tools/guest-diag/pin_supervisor.sh b/tools/guest-diag/pin_supervisor.sh new file mode 100755 index 0000000..b87d5c1 --- /dev/null +++ b/tools/guest-diag/pin_supervisor.sh @@ -0,0 +1,169 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2026 Open-IOV contributors +# pin_supervisor.sh - supervise the host-side pin tools (pin_gr_stream.py and +# semfix_uvmt.py) so a silent death cannot strand every later CUDA client at +# first cudaMalloc. +# +# Both tools write a heartbeat file; this script checks, every 5 s, that each +# process is alive (exact script path) and that its heartbeat mtime is < 75 s. +# A startup grace (120 s, skipped early once the tool log shows it has done +# first useful work) keeps a slow cold start from being restarted. On failure +# it kills the tool (exact path match) and restarts it, appending RESTART +# lines to the supervisor log. After 10 restarts in one hour for a tool it +# logs GIVEUP and exits 1. Only one semfix_uvmt instance is ever started +# (QEMU's QMP socket accepts a single client). +# +# Liveness policy (do not tighten without re-checking the soak evidence): +# * the tools refresh their heartbeats from dedicated loops/threads even +# while idle (pin_gr_stream: 5 s thread; semfix: per loop + around every +# blocking guest-agent/dmesg call), so 75 s never trips on a healthy +# client between phases; +# * a DEAD process is restarted on the next 5 s tick regardless of grace; +# * dead-pid/state recovery (the kill cascade) is handled INSIDE the tools +# (in-band #nopid + idle liveness probe + stall watchdog), NOT by this +# supervisor - a log-based restart here could fire mid-run of a healthy +# client (the m178 lesson). +# +# Usage: pin_supervisor.sh LOGDIR TAG +# Logs: LOGDIR/12-pin-supervisor-TAG.log (supervisor) +# LOGDIR/12-pin_gr_stream-TAG.log, LOGDIR/12-semfix-TAG.log (tools) +set -uo pipefail + +LOGDIR="${1:?usage: pin_supervisor.sh LOGDIR TAG}" +TAG="${2:?usage: pin_supervisor.sh LOGDIR TAG}" + +HERE=$(cd "$(dirname "$0")" && pwd) +PIN="$HERE/pin_gr_stream.py" +SEMFIX="$HERE/semfix_uvmt.py" +PIN_HB="${PIN_GR_HEARTBEAT:-/tmp/opencode/pin_gr_heartbeat}" +SEMFIX_HB="${SEMFIX_HEARTBEAT_FILE:-/tmp/opencode/semfix_heartbeat}" +SUPLOG="$LOGDIR/12-pin-supervisor-$TAG.log" +PIN_LOG="$LOGDIR/12-pin_gr_stream-$TAG.log" +SEMFIX_LOG="$LOGDIR/12-semfix-$TAG.log" +WINDOW=3600 +CAP=10 +STALE=75 +GRACE=120 +mkdir -p "$LOGDIR" /tmp/opencode + +log() { echo "[$(date +%H:%M:%S)] $*" | tee -a "$SUPLOG"; } + +tool_alive() { pgrep -f "python3 $1" >/dev/null 2>&1; } + +hb_age() { # age in seconds of the heartbeat file, 9999 if missing + local f="$1" m + m=$(stat -c %Y "$f" 2>/dev/null) || { echo 9999; return; } + echo $(( $(date +%s) - m )) +} + +kill_tool() { + pkill -f "python3 $1" 2>/dev/null + sleep 1 + if pgrep -f "python3 $1" >/dev/null 2>&1; then + pkill -9 -f "python3 $1" 2>/dev/null + sleep 1 + fi + return 0 +} + +start_pin() { + if tool_alive "$PIN"; then return 0; fi + touch "$PIN_HB" 2>/dev/null || true + ( setsid nohup env SEMFIX_COUNTER_RETRY=10 \ + python3 "$PIN" >> "$PIN_LOG" 2>&1 & ) + pin_started=$(date +%s) + log "START pin_gr_stream" +} + +start_semfix() { + # pgrep exact-count check before start: if an instance already exists + # (ours or a leftover), never start a second one. + if tool_alive "$SEMFIX"; then + log "semfix already running - not starting another" + return 0 + fi + touch "$SEMFIX_HB" 2>/dev/null || true + ( setsid nohup env SEMFIX_SKIP_PB=0 SEMFIX_COUNTER_RETRY=10 \ + python3 "$SEMFIX" 0.5 >> "$SEMFIX_LOG" 2>&1 & ) + semfix_started=$(date +%s) + log "START semfix_uvmt" +} + +prune() { # $1 now, $2 list -> list with entries older than WINDOW dropped + local now="$1" t out="" + for t in $2; do + [ $(( now - t )) -lt "$WINDOW" ] && out="$out $t" + done + printf '%s' "$out" +} + +count_list() { # $1 list -> element count + local n=0 t + for t in $1; do n=$(( n + 1 )); done + echo "$n" +} + +restarts_pin="" +restarts_semfix="" +pin_started=$(date +%s) +semfix_started=$(date +%s) + +restart() { # $1 label, $2 script path, $3 start function + local now list n + now=$(date +%s) + if [ "$1" = pin_gr_stream ]; then + restarts_pin=$(prune "$now" "$restarts_pin") + n=$(count_list "$restarts_pin") + else + restarts_semfix=$(prune "$now" "$restarts_semfix") + n=$(count_list "$restarts_semfix") + fi + if [ "$n" -ge "$CAP" ]; then + log "GIVEUP $1: $n restarts within ${WINDOW}s" + exit 1 + fi + kill_tool "$2" + if [ "$1" = pin_gr_stream ]; then + restarts_pin="$restarts_pin $now" + else + restarts_semfix="$restarts_semfix $now" + fi + log "RESTART $(( n + 1 )) $1 $(date +%Y-%m-%dT%H:%M:%S)" + "$3" +} + +check_tool() { # $1 label, $2 path, $3 heartbeat file, $4 start function, + # $5 tool start TS, $6 tool log file, $7 first-work pattern + local age runtime + age=$(hb_age "$3") + if tool_alive "$2" && [ "$age" -le "$STALE" ]; then + return 0 + fi + if tool_alive "$2"; then + # Startup grace: a cold tool can spend its first ~2 min installing the + # guest scanner or connecting QMP with a lagging heartbeat; only + # enforce staleness once the grace has passed or the log shows the + # tool has already done successful work. + runtime=$(( $(date +%s) - $5 )) + if [ "$runtime" -lt "$GRACE" ] && \ + ! grep -q -- "$7" "$6" 2>/dev/null; then + return 0 + fi + log "$1 heartbeat stale (${age}s) - restarting" + else + log "$1 not running (heartbeat age ${age}s) - restarting" + fi + restart "$1" "$2" "$4" +} + +log "pin_supervisor up (LOGDIR=$LOGDIR TAG=$TAG)" +start_pin +start_semfix +while true; do + sleep 5 + check_tool pin_gr_stream "$PIN" "$PIN_HB" start_pin "$pin_started" \ + "$PIN_LOG" "listening on vsock" + check_tool semfix_uvmt "$SEMFIX" "$SEMFIX_HB" start_semfix \ + "$semfix_started" "$SEMFIX_LOG" "semfix_uvmt: watching" +done diff --git a/tools/guest-diag/semfix_uvmt.py b/tools/guest-diag/semfix_uvmt.py index cf54445..11c191c 100755 --- a/tools/guest-diag/semfix_uvmt.py +++ b/tools/guest-diag/semfix_uvmt.py @@ -53,14 +53,47 @@ SKIP_PB = os.environ.get("SEMFIX_SKIP_PB", "0") == "1" # Seconds between counter-page pin attempts (the guest-agent exec blocks for # its full timeout when the client saturates the guest). COUNTER_RETRY_S = float(os.environ.get("SEMFIX_COUNTER_RETRY", "10")) +# Liveness file checked by pin_supervisor.sh (mtime must stay < 75 s; the +# supervisor grants a 120 s startup grace on top). +SEMFIX_HB_FILE = os.environ.get("SEMFIX_HEARTBEAT_FILE", + "/tmp/opencode/semfix_heartbeat") +# Dead-pid/stall watchdog tuning (seconds). The tracked client is probed in +# the guest only when it has been idle this long, on a cadence, so live +# clients are never probed and a killed client is detected within ~1 min. +LIVENESS_IDLE_S = float(os.environ.get("SEMFIX_LIVENESS_IDLE", "15")) +LIVENESS_EVERY_S = float(os.environ.get("SEMFIX_LIVENESS_EVERY", "25")) +# Pre-capture stall: client alive, counter page not pinned, no progress for +# this long -> recovery attempt; escalate loudly if it persists. +STALL_PRECAP_S = float(os.environ.get("SEMFIX_STALL_PRECAP", "60")) +STALL_RETRY_S = float(os.environ.get("SEMFIX_STALL_RETRY", "30")) +STALL_ESCALATE_S = float(os.environ.get("SEMFIX_STALL_ESCALATE", "180")) +# Inter-tool re-assert request: written here, consumed by pin_gr_stream.py +# (full ping re-assert of every pair it has ever pinned). +REASSERT_REQ = os.environ.get("PIN_REASSERT_FILE", + "/tmp/opencode/pin_reassert_request") + + +def _hb(): + try: + with open(SEMFIX_HB_FILE, "w") as f: + f.write("%.3f\n" % time.time()) + except OSError: + pass class Stream: """Tail the guest scanner's control lines mirrored by pin_gr_stream.py. - The guest emits "#pid <n>" and "#sem 0x<k>" over vsock; the listener - appends them here. This replaces per-loop guest-agent trace reads, which - were too slow to catch short-lived tracking semaphores. + The guest emits "#pid <n> [starttime]" and "#sem 0x<k>" over vsock; the + listener appends them here. This replaces per-loop guest-agent trace + reads, which were too slow to catch short-lived tracking semaphores. + + Replay safety (the m178 regression): the file persists across tool + restarts, so a fresh process re-reads the whole history. Lines written + before this process started (byte offset < init_size) are absorbed as + HISTORY - they seed the dedupe/identity state and the sem/pb lists but + can never trigger a per-client reset. Only lines appended afterwards + are live. This makes a supervisor restart mid-run harmless. """ def __init__(self, path): @@ -70,6 +103,15 @@ class Stream: self.pids = [] self.pbs = [] self.new_pbs = [] + self.new_nopids = [] + self.history_pids = [] # (pid, start) identities from history + self.pid_keys = set() # (pid, start) dedupe, per generation + try: + # Everything already in the file when this process starts is + # history (a restart replays it); never reset on it. + self._init_size = os.path.getsize(path) + except OSError: + self._init_size = 0 def reset(self): """Forget everything learned about the previous client. A new client @@ -80,6 +122,16 @@ class Stream: self.pbs = [] self.pids = [] self.new_pbs = [] + self.new_nopids = [] + self.pid_keys = set() + + @staticmethod + def _ident(line): + """Parse '#pid 1234' or '#pid 1234 5678900' -> (pid, start|None).""" + nums = re.findall(r"\d+", line) + if not nums: + return None, None + return int(nums[0]), (int(nums[1]) if len(nums) > 1 else None) def poll(self): try: @@ -90,12 +142,19 @@ class Stream: self.off = 0 f.seek(self.off) data = f.read() + base = self.off self.off = f.tell() except OSError: - return [], [] + return [], [], [] new_sems, new_pids = [], [] self.new_pbs = [] + self.new_nopids = [] + pos = base for line in data.splitlines(): + # A line is live when it ends past the startup boundary; history + # lines are absorbed (identity/dedupe) but never emitted as live. + live = pos + len(line) + 1 > self._init_size + pos += len(line) + 1 if line.startswith("#sem"): m = re.search(r"0x([0-9a-fA-F]+)", line) if m: @@ -111,13 +170,116 @@ class Stream: self.pbs.append(v) self.new_pbs.append(v) elif line.startswith("#pid"): - m = re.search(r"(\d+)", line) - if m: - v = int(m.group(1)) - if v not in self.pids: - self.pids.append(v) - new_pids.append(v) - return new_sems, new_pids + pid, start = self._ident(line) + if pid is not None and (pid, start) not in self.pid_keys: + self.pid_keys.add((pid, start)) + self.pids.append(pid) + if live: + new_pids.append((pid, start)) + else: + self.history_pids.append((pid, start)) + elif line.startswith("#nopid"): + # In-band client-death signal (new guest scanner; older ones + # simply never send it and the host probe covers the case). + pid, start = self._ident(line) + if live and pid is not None: + self.new_nopids.append((pid, start)) + return new_sems, new_pids, self.new_nopids + + +class ClientTracker: + """Monotonic, replay-proof client-identity tracking. + + A client's identity is (pid, starttime). Rules, in priority order: + * an identity adopted once is never reset a second time (duplicate + lines, trace/stream replays, supervisor restarts); + * a backwards pid, or the same pid with an older starttime, is a + replay - never reset; + * a pid recycled with a NEWER starttime is a real new client - reset + immediately (the previous holder is definitionally gone); + * a newer unseen identity only resets the per-client state once the + CURRENT client is CONFIRMED gone - a reset fired while the previous + client was still active is the m178 third-client hang regression. + + Pure logic: the caller feeds guest liveness results in and performs the + actual state purge when an action says so. + """ + + def __init__(self): + self.seen = set() + self.current = None + self.pending = None + self.dead = set() + + @staticmethod + def _start_match(a, b): + # Unknown starttimes match conservatively (legacy scanner lines). + return a[1] == b[1] or a[1] is None or b[1] is None + + def observe_history(self, pid, start): + """Absorb an identity found in pre-restart history: the process has + no per-client state yet, so history only seeds identity tracking and + the freshest identity becomes current - no reset is ever requested + for history lines.""" + key = (pid, start) + if key in self.seen: + return + self.seen.add(key) + self.current = key + + def note_dead(self, pid, start): + """A #nopid or a successful dead probe: identity is gone.""" + key = (pid, start) + self.dead.add(key) + if self.current and self.current[0] == pid \ + and self._start_match(self.current, key): + self.current = None + if self.pending and self.pending[0] == pid \ + and self._start_match(self.pending, key): + self.pending = None + + def is_current(self, pid, start): + """True when (pid, start) is the currently tracked identity (used to + decide whether a death signal may purge - a stale #nopid for a + recycled pid must never purge the new holder).""" + return self.current is not None and self.current[0] == pid \ + and self._start_match(self.current, (pid, start)) + + def decide(self, pid, start): + """Classify a live #pid line: adopt | skip | skip_backwards | + maybe_reset | reset.""" + key = (pid, start) + if key in self.seen: + return "skip" + self.seen.add(key) + cur = self.current + if cur is None: + self.current = key + return "adopt" + if pid == cur[0] and self._start_match(cur, key): + return "skip" + if pid == cur[0] and start is not None and cur[1] is not None \ + and start > cur[1]: + # Recycled pid, newer process: safe to reset now. + self.current = key + return "reset" + if pid < cur[0] or (pid == cur[0] and start is not None + and cur[1] is not None and start < cur[1]): + return "skip_backwards" + self.pending = key + return "maybe_reset" + + def resolve(self, old_dead): + """Caller learned whether the CURRENT client is dead. Returns + 'reset' (purge + adopt the pending client) or 'defer'.""" + if self.pending is None: + return None + if old_dead: + key = self.pending + self.pending = None + self.current = key + return "reset" + return "defer" def log(msg): @@ -196,7 +358,9 @@ def dmesg_releases(seen): Only lines after the last "sema-svc: thread started" marker count: dmesg keeps the previous module load's lines and the old client's releases. + Returns the number of newly printed lines (used as watchdog progress). """ + n = 0 try: r = subprocess.run( ["sudo", "-n", "sh", "-c", @@ -205,13 +369,15 @@ def dmesg_releases(seen): "| grep 'sema-push' | tail -300"], capture_output=True, text=True, timeout=10) except Exception: - return + return 0 for line in r.stdout.splitlines(): line = line.strip() if not line or line in seen: continue seen.add(line) + n += 1 log("push: %s" % line.replace("vgsp: ", "")) + return n def cdt_pid(): @@ -224,6 +390,34 @@ def cdt_pid(): return int(pids[-1]) if pids else None +def _pid_alive_cmd(pid): + """Guest shell that reports ALIVE / DEAD for a pid. Zombie-aware: a + killed (SIGKILLed) client lingers as a defunct /proc entry, and a plain + `test -d /proc/<pid>` would latch it as alive forever (the m178 + dead-pid latch).""" + return ("f=/proc/%d/stat; if [ -r \"$f\" ]; then " + "s=$(sed -e 's/^[^)]*) //' -e 's/ .*//' \"$f\"); " + "if [ \"$s\" = Z ] || [ -z \"$s\" ]; then echo DEAD; " + "else echo ALIVE; fi; else echo DEAD; fi" % pid) + + +def pid_alive(pid): + """Guest liveness of a client pid: True alive, False dead/zombie/gone, + None unknown (agent error). Callers must treat None as alive (never + purge a client on an inconclusive probe).""" + try: + r = subprocess.run([VMEXEC, "exec", _pid_alive_cmd(pid)], + capture_output=True, text=True, timeout=20) + except Exception: + return None + out = r.stdout or "" + if "ALIVE" in out: + return True + if "DEAD" in out: + return False + return None + + def pin_counter_page(pid, pinned_pids): """Pin the client's completion-counter page from its own pagemap. @@ -268,10 +462,87 @@ def main(): dseen = set() counter_pids = set() counter_next_try = {} + # Current client pid/starttime, mirrored from the tracker (starttime + # disambiguates recycled pids). last_pid = None + last_start = None + # Stale-sem tracking: consecutive unreadable reads, first-unreadable time, + # and the per-sem read skip deadline once a sem has failed 3 times. + unread_count = {} + first_unreadable_t = {} + skip_until = {} + # Failed pool-pin tracking: retry with a 15 s backoff after 2 failures. + sem_pin_fail = {} + sem_pin_skip_until = {} + qmp_next_try = 0.0 stream = Stream(SEMSTREAM) + tracker = ClientTracker() + # Dead-pid/backwards-transition proofing and the stall watchdog. + history_absorbed = False + adopted_from_history = False + probe_next = 0.0 # next allowed idle-liveness probe + recover_next = 0.0 # next watchdog recovery attempt + stall_since = None # first observation of the pre-capture stall + escalated = False + progress_t = time.time() # last host-side progress for the client loop = 0 + def _clear_client_state(): + """Purge every per-client latch. ONLY valid when the previous client + is CONFIRMED dead (or at fresh process start): the m178 third-client + hang came from wiping the state of an ACTIVE client on a replayed + (#pid) identity.""" + stream.reset() + known_sems.clear() + seen_sem_pin.clear() + last_state.clear() + last_seen_t.clear() + pinned_ranges.clear() + seen_ch.clear() + seen_pb.clear() + counter_pids.clear() + counter_next_try.clear() + sem_pin_fail.clear() + sem_pin_skip_until.clear() + unread_count.clear() + first_unreadable_t.clear() + skip_until.clear() + + def _adopt(pid, start, why): + nonlocal last_pid, last_start, progress_t, adopted_from_history + nonlocal stall_since, escalated + last_pid, last_start = pid, start + progress_t = time.time() + adopted_from_history = False + stall_since = None + escalated = False + log("semfix: tracking client pid %d%s (%s)" + % (pid, " start=%s" % start if start is not None else "", why)) + + def _watchdog_reset(): + nonlocal stall_since, escalated + stall_since = None + escalated = False + + def _watchdog_recover(): + """Idempotent pre-capture stall recovery: re-arm every sem latch so + the next loop re-resolves and re-pins, force an immediate counter + retry, and request a full pin re-assert from pin_gr_stream.""" + for sem in set(known_sems) | set(stream.sems): + skip_until.pop(sem, None) + sem_pin_skip_until.pop(sem, None) + sem_pin_fail.pop(sem, None) + unread_count.pop(sem, None) + first_unreadable_t.pop(sem, None) + seen_sem_pin.discard(sem) + if last_pid is not None: + counter_next_try[last_pid] = 0.0 + try: + with open(REASSERT_REQ, "w") as f: + f.write("%.3f\n" % time.time()) + except OSError: + pass + def pin_range_once(gpu_vas, cpu, gpa, size=0): key = tuple(gpu_vas), cpu, size st = pinned_ranges.get(key) @@ -293,40 +564,104 @@ def main(): while True: loop += 1 + _hb() sems = set() pbs = set() chans = set() - new_sems, new_pids = stream.poll() + new_sems, new_pids, new_nopids = stream.poll() for s in new_sems: log("stream: new uvmt sem 0x%x" % s) - for p in new_pids: - log("stream: client pid %d" % p) - if last_pid is not None and p != last_pid: - stream.reset() - known_sems.clear() - seen_sem_pin.clear() - last_state.clear() - last_seen_t.clear() - pinned_ranges.clear() - seen_ch.clear() - seen_pb.clear() - counter_pids.clear() - log("semfix: new client pid %d - reset per-client pin state" % p) - last_pid = p + if new_sems and last_pid is not None: + progress_t = time.time() + + # In-band client-death signals (new guest scanner): purge the dead + # client's per-client state immediately so the NEXT client works + # without a VM cycle (the kill cascade). + for p, st in new_nopids: + log("stream: client gone (%d%s)" + % (p, " start=%s" % st if st is not None else "")) + was_current = tracker.is_current(p, st) + tracker.note_dead(p, st) + if was_current and last_pid == p: + _clear_client_state() + last_pid = last_start = None + _watchdog_reset() + log("semfix: purged per-client state of exited pid %d" % p) + # The old client is gone: adopt any client whose reset was deferred + # while it was still (assumed) alive. + if last_pid is None and tracker.pending is not None \ + and tracker.current is None: + if tracker.resolve(old_dead=True) == "reset": + p, st = tracker.current + _clear_client_state() + _adopt(p, st, "deferred reset") + # Pre-restart history (a supervisor restart re-reads the file): absorb + # identities WITHOUT any reset - the fresh process already has empty + # state; the freshest identity is the client that was running. + if stream.history_pids and not history_absorbed: + for p, st in stream.history_pids: + tracker.observe_history(p, st) + history_absorbed = True + if tracker.current is not None and last_pid is None: + p, st = tracker.current + _adopt(p, st, "resumed from pre-restart history") + adopted_from_history = True + + for p, st in new_pids: + cur_before = tracker.current + action = tracker.decide(p, st) + tag = " start=%s" % st if st is not None else "" + if action == "skip": + log("stream: client pid %d%s seen before - no reset" % (p, tag)) + elif action == "skip_backwards": + log("semfix: client pid %d%s backwards/replay - no reset" + % (p, tag)) + elif action == "adopt": + log("stream: client pid %d%s" % (p, tag)) + _adopt(p, st, "new client") + elif action == "reset": + # Recycled pid, newer starttime: the previous holder of this + # pid is definitionally gone, safe to reset now. + log("semfix: pid %d recycled (start=%s) - reset per-client " + "pin state" % (p, st)) + _clear_client_state() + _adopt(p, st, "pid recycled") + elif action == "maybe_reset": + old = cur_before + alive = pid_alive(old[0]) if old else False + if alive is False: + log("semfix: new client pid %d%s (pid %d gone) - reset " + "per-client pin state" + % (p, tag, old[0] if old else -1)) + _clear_client_state() + tracker.resolve(old_dead=True) + _adopt(p, st, "client switch") + else: + # NEVER clear the active client's state on the account of + # a newer pid: the m178 hang. Deferred until the old pid + # is confirmed gone (periodic idle probe below). + log("semfix: client pid %d%s appeared while pid %s is " + "alive%s - reset deferred" + % (p, tag, old[0] if old else "?", + "" if alive else " (probe unknown)")) for pb in getattr(stream, "new_pbs", []): log("stream: new pushbuffer 0x%x" % pb) pbs.add(pb) + if getattr(stream, "new_pbs", []) and last_pid is not None: + progress_t = time.time() sems.update(stream.sems) # The guest-agent trace read is slow under client load; use it only # as a fallback for channel/pushbuffer discovery and to recover sems # the scanner may have missed. if loop % TRACE_EVERY == 0: + _hb() try: t_sems, t_pbs, t_chans = guest_trace() except Exception: t_sems, t_pbs, t_chans = set(), set(), set() + _hb() sems.update(t_sems) pbs.update(t_pbs) chans.update(t_chans) @@ -336,16 +671,54 @@ def main(): for sem in sorted(sems - known_sems): known_sems.add(sem) - for sem in sorted(known_sems): + for hb_i, sem in enumerate(sorted(known_sems)): + if hb_i % 16 == 0: + # A full pass can run 60-90 s (thousands of QMP reads); keep + # the heartbeat fresh inside it (r203: 79 s gap -> supervisor + # restarted the tool mid-pass). + _hb() + now = time.time() + if now < skip_until.get(sem, 0): + continue try: st = uvmt_state(q, sem) except Exception: st = None + if st is None: + # Stale-sem hygiene: the trace ring replays dead clients' + # kernel pointers forever; without eviction every loop burns + # QMP reads on them and the log floods with "unreadable". + cnt = unread_count.get(sem, 0) + 1 + unread_count[sem] = cnt + first_unreadable_t.setdefault(sem, now) + if cnt >= 3: + skip_until[sem] = now + 30.0 + if cnt == 3: + log("semfix: sem=0x%x unreadable x3 - retrying every " + "30s" % sem) + age = now - first_unreadable_t[sem] + if age >= 60.0 and sem not in stream.sems: + known_sems.discard(sem) + unread_count.pop(sem, None) + first_unreadable_t.pop(sem, None) + skip_until.pop(sem, None) + last_state.pop(sem, None) + last_seen_t.pop(sem, None) + seen_sem_pin.discard(sem) + sem_pin_fail.pop(sem, None) + sem_pin_skip_until.pop(sem, None) + log("semfix: evicting stale sem=0x%x (unreadable %.0fs, " + "absent from stream)" % (sem, age)) + continue + else: + unread_count.pop(sem, None) + first_unreadable_t.pop(sem, None) + skip_until.pop(sem, None) key = state_key(st) - now = time.time() if key != last_state.get(sem): last_state[sem] = key last_seen_t[sem] = now + progress_t = time.time() log("uvmt %s" % state_str(sem, st)) elif now - last_seen_t.get(sem, 0) >= HEARTBEAT: last_seen_t[sem] = now @@ -354,18 +727,34 @@ def main(): # Pin the pool page once per (sem, word) resolution. The payload # word is what the releasing push writes; pinning the page makes # the host able to apply that release. - if st and sem not in seen_sem_pin: + if (st and sem not in seen_sem_pin + and now >= sem_pin_skip_until.get(sem, 0)): try: pairs = resolve_sem_pool(q, sem) except Exception as e: pairs = [] log("semfix: sem=0x%x resolve failed: %s" % (sem, e)) if pairs: - seen_sem_pin.add(sem) + ok_any = False for gv, gpa in pairs: ok = pin(gv, gpa) + ok_any = ok_any or ok log("semfix: sem=0x%x pool va=0x%x -> gpa=0x%x pin=%s" % (sem, gv, gpa, "ok" if ok else "FAILED")) + if not ok_any: + # Mark pinned only after a successful pin(): a failed + # pin used to be recorded as done and never retried. + fails = sem_pin_fail.get(sem, 0) + 1 + sem_pin_fail[sem] = fails + if fails >= 2: + sem_pin_skip_until[sem] = now + 15.0 + log("semfix: sem=0x%x pool pin failed x%d - retry " + "in 15s" % (sem, fails)) + continue + seen_sem_pin.add(sem) + sem_pin_fail.pop(sem, None) + sem_pin_skip_until.pop(sem, None) + progress_t = time.time() if SKIP_PB: continue d = {} @@ -385,6 +774,7 @@ def main(): d["exception"] = str(e)[:40] if res: pin_range_once(*res) + progress_t = time.time() log("semfix: sem=0x%x pushbuf -> %s" % (sem, diag_str(d))) else: @@ -393,7 +783,22 @@ def main(): else: log("semfix: sem=0x%x pool unresolved (retry)" % sem) + if getattr(q, "_fails", 0) >= 5: + # QMP went away (QEMU restart, monitor died): all reads have been + # empty for 5 consecutive replies. Recreate the client loudly + # instead of reading empty forever ("uvmt(hb) ... unreadable"). + now = time.time() + if now >= qmp_next_try: + log("semfix: QMP unreadable (%d consecutive empty replies) - " + "recreating connection" % q._fails) + try: + q = Qmp(QMP_SOCK) + except Exception as e: + log("semfix: QMP recreate failed: %s" % e) + qmp_next_try = now + 5.0 + for ch in sorted(chans - seen_ch): + _hb() # a resolve can block for tens of seconds d = {} try: res = resolve_from_channel(q, ch, d) @@ -403,10 +808,12 @@ def main(): if res: seen_ch.add(ch) pin_range_once(*res) + progress_t = time.time() log("semfix: ch=0x%x -> %s" % (ch, diag_str(d))) else: log("semfix: ch=0x%x pushbuf miss [%s]" % (ch, diag_str(d))) for pb in sorted(pbs - seen_pb): + _hb() # a resolve can block for tens of seconds d = {} try: res = resolve_from_pushbuffer(q, pb, d) @@ -416,12 +823,15 @@ def main(): if res: seen_pb.add(pb) pin_range_once(*res) + progress_t = time.time() log("semfix: pb=0x%x -> %s" % (pb, diag_str(d))) else: log("semfix: pb=0x%x pushbuf miss [%s]" % (pb, diag_str(d))) if DLOG: - dmesg_releases(dseen) + if dmesg_releases(dseen): + progress_t = time.time() + _hb() # Retry incomplete push-buffer pins: gva2gpa of the kernel vmalloc # alias fails while the sampled vCPU runs in user mode (PTI), so a @@ -430,24 +840,89 @@ def main(): if loop % 8 == 0: for st in list(pinned_ranges.values()): if not st["done"] and st["tries"] > 0: + _hb() # pin_range_once can walk thousands of QMP reads pin_range_once(*st["args"]) + # Dead-pid detection and the pre-capture stall watchdog. The guest is + # probed ONLY when the tracked client has been quiet for a while (a + # live, pushing client is never probed), so the agent is not burdened + # during healthy runs. A confirmed-dead client is purged immediately: + # the NEXT client then starts with clean state - no VM cycle needed + # (the kill cascade). + now = time.time() + if adopted_from_history and last_pid is not None: + # The client resumed from pre-restart history may have died while + # this process was down; do not latch a dead pid into the counter + # retries. + alive = pid_alive(last_pid) + adopted_from_history = False + if alive is False: + log("semfix: resumed pid %d already gone - dropping" % last_pid) + tracker.note_dead(last_pid, last_start) + last_pid = last_start = None + _watchdog_reset() + if last_pid is not None and now >= probe_next \ + and now - progress_t >= LIVENESS_IDLE_S: + probe_next = now + LIVENESS_EVERY_S + alive = pid_alive(last_pid) + if alive is False: + log("semfix: client pid %d gone (probe) - purging per-client " + "state" % last_pid) + tracker.note_dead(last_pid, last_start) + _clear_client_state() + last_pid = last_start = None + _watchdog_reset() + if tracker.pending is not None and tracker.current is None \ + and tracker.resolve(old_dead=True) == "reset": + p, st = tracker.current + log("semfix: adopting deferred client pid %d" % p) + _adopt(p, st, "deferred reset") + elif alive is True and last_pid not in counter_pids: + # Pre-capture stall: the client is alive but has no pinned + # counter page and nothing progressed. Recover (idempotent), + # then escalate loudly if the stall persists - the goal is + # that a first cudaMalloc / first GR capture never stalls + # silently. + idle = now - progress_t + if idle >= STALL_PRECAP_S: + if stall_since is None: + stall_since = now + if now >= recover_next: + recover_next = now + STALL_RETRY_S + log("STALL WATCHDOG: client pid %d alive, no " + "pre-capture progress for %.0fs - recovery " + "attempt (re-arm sems, re-pin counter, request " + "pin re-assert)" % (last_pid, idle)) + _watchdog_recover() + if not escalated and now - stall_since >= STALL_ESCALATE_S: + escalated = True + log("STALL WATCHDOG: recovery FAILED for pid %d " + "after %.0fs (alive, counter page not pinned) - " + "pre-capture hang persists, needs manual " + "attention" % (last_pid, now - stall_since)) + # Pin the completion-counter page of a newly started client, before its # first GR pushes decode their releases. The pid comes from the guest # scanner; the guest-agent fallback is only a safety net. The exec can # block for its whole timeout under client load, so retry on a slow # cadence: hammering the agent here starves it and wedges the run. if last_pid is None and loop % TRACE_EVERY == 0: - last_pid = cdt_pid() - if last_pid: - log("counter: client pid %d (agent)" % last_pid) + p = cdt_pid() + _hb() + if p: + if tracker.current is None and (p, None) not in tracker.seen: + tracker.observe_history(p, None) + last_pid, last_start = p, None + log("counter: client pid %d (agent)" % p) if last_pid and last_pid not in counter_pids: now = time.time() if now >= counter_next_try.get(last_pid, 0): try: - pin_counter_page(last_pid, counter_pids) + if pin_counter_page(last_pid, counter_pids): + progress_t = time.time() except Exception as e: log("counter: pin failed: %s" % e) + _hb() counter_next_try[last_pid] = now + COUNTER_RETRY_S time.sleep(interval) diff --git a/tools/guest-diag/semfix_watch.py b/tools/guest-diag/semfix_watch.py index 5124805..4b60fd6 100644 --- a/tools/guest-diag/semfix_watch.py +++ b/tools/guest-diag/semfix_watch.py @@ -58,12 +58,47 @@ CHAN_MAX_CANDS = int(os.environ.get("CHAN_MAX_CANDS", "256")) class Qmp: def __init__(self, path): + self.path = path + self._cache = {} + self._fails = 0 + self._last_reconnect = 0.0 + self._reconnecting = False self.s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) self.s.connect(path) self.s.settimeout(15) self._rd() self.cmd("qmp_capabilities") - self._cache = {} + + def reconnect(self): + """Re-establish the QMP connection after the socket died. + + QEMU's QMP socket accepts a single client and closes when the guest + reboots; cmd() used to swallow every exception and return {}, so all + reads stayed empty forever ("uvmt(hb) sem=... unreadable") and + pinning stopped. Rate-limited to one attempt per second so a failing + monitor does not turn every read into a connect storm.""" + if self._reconnecting or time.time() - self._last_reconnect < 1.0: + return + self._reconnecting = True + self._last_reconnect = time.time() + try: + try: + self.s.close() + except Exception: + pass + self.s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self.s.connect(self.path) + self.s.settimeout(15) + self._rd() + self.cmd("qmp_capabilities") + self._cache = {} + self._ncpu = 0 + self._cpu = 0 + self._fails = 0 + except Exception: + pass + finally: + self._reconnecting = False def _rd(self): """Read one QMP reply (JSON object ends with a newline). Poll with a @@ -91,8 +126,17 @@ class Qmp: def cmd(self, name, **args): try: self.s.sendall(json.dumps({"execute": name, "arguments": args}).encode() + b"\n") - return json.loads(self._rd() or "{}") + reply = self._rd() + if not reply: + raise OSError("empty QMP reply") + self._fails = 0 + return json.loads(reply) except Exception: + # Broken socket (QEMU restart, monitor hiccup): try exactly one + # reconnect, rate-limited inside reconnect(). + self._fails = getattr(self, "_fails", 0) + 1 + if not getattr(self, "_reconnecting", False): + self.reconnect() return {} def hmp(self, line): @@ -143,8 +187,14 @@ class Qmp: break m = re.search(r":(.*)$", out, re.S) if not m: + if not out.strip(): + # Empty monitor output: the socket is broken, not the address. + self.reconnect() return [] vals = [int(x, 16) for x in re.findall(r"0x([0-9a-fA-F]+)", m.group(1))][:n] + if not vals: + self.reconnect() + return [] if cache: self._cache[key] = vals return vals @@ -189,18 +239,46 @@ def guest_trace(): Read *before* clearing: the create-time probes fire once per client and would be wiped by a clear that races them. + + 2026-10-16 robustness fix (the cascade root): this used to run a bare + `tail` over the guest agent. A tracefs read that blocks under a busy + guest left *uninterruptible* tail processes behind (observed dozens in + D state), the guest-agent exec then blocked for its full 90 s timeout + every iteration, and the pool/tracker pins this tool supplies stopped + flowing - after ~5 clients every new client froze at DEVICE waiting on + its UVM tracker with `pending release starved ... pool/waiter page`. + Now the read is bounded remotely (`timeout -s KILL 12`) so no tail can + outlive its call, and SSH (which stays healthy when the guest agent + wedges) is preferred with the agent as fallback. """ - inner = ("tail -c 131072 /sys/kernel/tracing/trace " + inner = ("timeout -s KILL 12 tail -c 131072 /sys/kernel/tracing/trace " "| grep -E \"uvmt|uvch|uvbp|uvpbf\" " "| tail -400") if not os.environ.get("KEEP_TRACE"): inner += "; echo > /sys/kernel/tracing/trace" - cmd = "sudo sh -c '%s'" % inner + outer = "sudo sh -c '%s'" % inner + out = "" + ssh = os.environ.get("SEMFIX_SSH_PORT", "2222") + ssh_cmd = [ + "sshpass", "-p", os.environ.get("VGPU_GUEST_PASS", "ubuntu"), + "ssh", "-o", "StrictHostKeyChecking=no", + "-o", "UserKnownHostsFile=/dev/null", + "-o", "PubkeyAuthentication=no", "-o", "ConnectTimeout=6", + "-p", ssh, "ubuntu@127.0.0.1", outer, + ] try: - out = subprocess.run([VMEXEC, "exec", cmd], capture_output=True, - text=True, timeout=90).stdout + r = subprocess.run(ssh_cmd, capture_output=True, text=True, + timeout=25) + out = r.stdout or "" except Exception: - return set(), set(), set() + out = "" + if not out: + try: + out = subprocess.run([VMEXEC, "exec", outer], + capture_output=True, text=True, + timeout=25).stdout or "" + except Exception: + return set(), set(), set() sems = {int(m, 16) for m in re.findall(r"sem=0x([0-9a-fA-F]+)", out)} pbs = {int(m, 16) for m in re.findall(r"pb=0x([0-9a-fA-F]+)", out)} chans = {int(m, 16) for m in re.findall(r"ch=0x([0-9a-fA-F]+)", out)} @@ -369,8 +447,11 @@ def resolve_pushbuffer_wide(q, sem, d): def pin(va, pa): cmd = "printf '0x%x,0x%x' > %s/sema_map_add" % (va, pa, PARAM) for _ in range(40): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=5) + except subprocess.TimeoutExpired: + continue if r.returncode == 0 and "timed out" not in (r.stderr or ""): return True time.sleep(0.05) # async queue full: drain and retry @@ -380,8 +461,11 @@ def pin(va, pa): def pin_bulk(va, pa, nbytes): cmd = "printf '0x%x,0x%x,0x%x' > %s/sema_map_add" % (va, pa, nbytes, PARAM) for _ in range(40): - r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], - capture_output=True, text=True) + try: + r = subprocess.run(["sudo", "-n", "sh", "-c", cmd], + capture_output=True, text=True, timeout=5) + except subprocess.TimeoutExpired: + continue if r.returncode == 0 and "timed out" not in (r.stderr or ""): return True time.sleep(0.05) diff --git a/tools/lg/frame_reader.py b/tools/lg/frame_reader.py new file mode 100644 index 0000000..f8bab40 --- /dev/null +++ b/tools/lg/frame_reader.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# frame_reader.py - read the GPU-rendered frames the guest publishes into the +# kvmfr ivshmem BAR (see tools/cuda-ab/frame_producer.cu) and save them on the +# host as PPM/PNG. This is the host half of the guest frame-producer path: +# the QEMU ivshmem-plain device is backed by /dev/shm/kvmfr-<uuid>, which the +# guest maps through /sys/bus/pci/devices/0000:00:03.0/resource2. +# +# Usage: frame_reader.py [--shm /dev/shm/kvmfr-<uuid>] [--frames N] [--out DIR] +import argparse +import mmap +import os +import struct +import sys +import time + +MAGIC = 0x4B564D46 +FRAME_OFFSET = 0x100000 +# magic, version, width, height, stride, format, seq, reserved, timestamp_ns +HDR = struct.Struct("<IIIIIIIIQ") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--shm", default=None) + ap.add_argument("--frames", type=int, default=5) + ap.add_argument("--out", default="/tmp/opencode/frames") + ap.add_argument("--timeout", type=float, default=120.0) + args = ap.parse_args() + + if not args.shm: + import glob + cands = [p for p in glob.glob("/dev/shm/kvmfr-*") + if os.path.getsize(p) > 0] + if not cands: + print("NO_SHM_FILE", file=sys.stderr) + return 1 + args.shm = max(cands, key=os.path.getsize) + os.makedirs(args.out, exist_ok=True) + print("SHM %s" % args.shm) + + fd = os.open(args.shm, os.O_RDWR) + size = os.fstat(fd).st_size + mm = mmap.mmap(fd, size, mmap.MAP_SHARED, + mmap.PROT_READ | mmap.PROT_WRITE) + + deadline = time.time() + args.timeout + saved = 0 + last_seq = 0 + while saved < args.frames and time.time() < deadline: + mm.seek(FRAME_OFFSET) + magic, ver, w, h, stride, fmt, seq, _res, ts = HDR.unpack( + mm.read(HDR.size)) + if magic != MAGIC or seq == 0 or seq == last_seq or w == 0: + time.sleep(0.05) + continue + fb = mm.read(w * h * 4) + last_seq = seq + path = os.path.join(args.out, "frame-%04d.ppm" % seq) + with open(path, "wb") as f: + f.write(b"P6\n%d %d\n255\n" % (w, h)) + # RGBA -> RGB + rgb = bytearray(w * h * 3) + rgb[0::3] = fb[0::4] + rgb[1::3] = fb[1::4] + rgb[2::3] = fb[2::4] + f.write(bytes(rgb)) + saved += 1 + print("SAVED seq=%d %dx%d fmt=%d -> %s" % (seq, w, h, fmt, path)) + mm.close() + os.close(fd) + print("READER_DONE saved=%d last_seq=%d" % (saved, last_seq)) + return 0 if saved else 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/lg/frame_sequence.sh b/tools/lg/frame_sequence.sh new file mode 100755 index 0000000..958a34a --- /dev/null +++ b/tools/lg/frame_sequence.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# frame_sequence.sh - capture an animated GPU-rendered frame sequence end to +# end: run the guest's CUDA frame producer into the kvmfr ivshmem BAR and +# save every published frame on the host as PPM/PNG (the proven +# CUDA -> ivshmem -> host path; see docs/05-operations/06). +# +# Usage: tools/lg/frame_sequence.sh [frames] [width] [height] [fps] [outdir] +set -u +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +FRAMES="${1:-40}" +W="${2:-800}" +H="${3:-600}" +FPS="${4:-10}" +OUT="${5:-/tmp/opencode/frames-seq}" + +mkdir -p "$OUT" +rm -f "$OUT"/frame-*.ppm "$OUT"/frame-*.png + +# Start the host reader first so no published frame is missed. +python3 "$REPO/tools/lg/frame_reader.py" --frames "$FRAMES" --out "$OUT" \ + --timeout 240 > "$OUT/reader.log" 2>&1 & +READER=$! + +sleep 1 +"$REPO/scripts/ssh_guest.sh" \ + "sudo /tmp/frame_producer $FRAMES $W $H $FPS" 2>&1 | grep -v "Permanently added" \ + | tail -5 + +wait "$READER" +echo "reader rc=$?" +ls -1 "$OUT"/frame-*.ppm 2>/dev/null | wc -l diff --git a/tools/site/pages/Main-Page.wiki b/tools/site/pages/Main-Page.wiki index 9a80009..f91fc17 100644 --- a/tools/site/pages/Main-Page.wiki +++ b/tools/site/pages/Main-Page.wiki @@ -18,7 +18,7 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i * '''[[Documentation]]''' — the full index of this wiki. * '''[[FAQ]]''' — the questions we get asked most. -== Project status (2026-10-12) == +== Project status (2026-10-20) == {| class="wikitable" ! Area !! State |- @@ -30,9 +30,9 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i |- | CUDA userspace init and the driver-API smoke (ALL OK, 64-byte payload round-trips) || Working |- -| Guest CUDA kernel execution on real hardware || In progress — a forwarded launch reaches the engine queue; engine execution is the frontier +| Guest CUDA kernel execution on real hardware || Working — vector-add, matrix-multiply and transformer kernels run green on fresh boots; hardening and breadth in progress |- -| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet +| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet — multiple mediated devices (different emulated VRAM sizes) and multiple guests already coexist; per-guest execution isolation is in progress |} The full story, with evidence and the current wall: [[Project Status]] and [[The Execution Frontier]]. diff --git a/wiki/.publish-manifest.json b/wiki/.publish-manifest.json index 20e9722..4fe571b 100644 --- a/wiki/.publish-manifest.json +++ b/wiki/.publish-manifest.json @@ -19,11 +19,11 @@ }, "CUDA Userspace Driver": { "file": "CUDA-Userspace-Driver.wiki", - "new_revid": 23963, + "new_revid": 23998, "old_revid": 23963, - "source_sha": "31d77a3a672b2943", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:06Z", + "source_sha": "d467dbcd9f323768", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:03Z", "title": "CUDA Userspace Driver" }, "Community": { @@ -163,11 +163,11 @@ }, "Main Page": { "file": "Main-Page.wiki", - "new_revid": 23972, - "old_revid": 23960, - "source_sha": "1b53ef37afca940e", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:23Z", + "new_revid": 24000, + "old_revid": 23972, + "source_sha": "2e8171cab56c394c", + "summary": "Status table refresh 2026-10-20: guest CUDA kernel execution working", + "timestamp": "2026-10-20T09:19:09Z", "title": "Main Page" }, "MediaWiki:Sidebar": { @@ -199,20 +199,20 @@ }, "Physical GPU Backend and Guest Wiring": { "file": "Physical-GPU-Backend.wiki", - "new_revid": 23985, - "old_revid": 23951, - "source_sha": "4bff4dbfa8b4644e", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:59Z", + "new_revid": 23999, + "old_revid": 23985, + "source_sha": "d762abd50cef636d", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:05Z", "title": "Physical GPU Backend and Guest Wiring" }, "Project Status": { "file": "Project-Status.wiki", - "new_revid": 23986, - "old_revid": 23952, - "source_sha": "6d07f2a0c6f524a9", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:00Z", + "new_revid": 23997, + "old_revid": 23986, + "source_sha": "cb714adeebcfdc33", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:01Z", "title": "Project Status" }, "RPC Message Format": { @@ -262,11 +262,11 @@ }, "The Execution Frontier": { "file": "Execution-Frontier.wiki", - "new_revid": 23966, + "new_revid": 23996, "old_revid": 23966, - "source_sha": "b26ce9bb6fe63f6a", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:07Z", + "source_sha": "8c0007ec287387b9", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:00Z", "title": "The Execution Frontier" }, "The Guest Stack and CUDASW": { diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md index 9a041ef..6975a1d 100644 --- a/wiki/CUDA-Userspace-Driver.md +++ b/wiki/CUDA-Userspace-Driver.md @@ -1,16 +1,20 @@ # CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive -> **Status (2026-10-12, honest).** The analysis covers the whole library for +> **Status (2026-10-20, honest).** The analysis covers the whole library for > driver 570.144 and the tooling is version-agnostic. The *control plane* gets > the stock guest stack through `cuInit`, device enumeration, `cuCtxCreate` > and the full driver-API smoke test — allocation, copies, free and destroy > complete end to end (`ALL OK`; the 64-byte payload round-trips exactly). > The *data plane* decodes guest push buffers and executes the memory and > semaphore work it understands; CUDA kernel launches are forwarded to the -> physical execution path, where a launch reaches the engine queue -> (front-end deposit and scheduling are observed) but an engine-authored -> release is not yet observed. Green runs are not yet reproducible on every -> boot — an open issue, tracked in the repository. Canonical detail: +> physical execution path and now execute on the physical GPU — the +> engine-authored completions relay to the guest and its synchronisation +> waits resolve. The vector-add (`VADD n=32768 bad=0`), matrix-multiply +> (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, +> `FFN4 layers=4 bad=0`) clients verify green on fresh boots. The +> large-buffer staged-transfer class (multi-megabyte transfers between guest +> memory and device memory) is under repair, and reproducibility across fresh +> boots is being re-measured after the boot-poisoning fix. Canonical detail: > [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) > through `08`; reading order > [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); @@ -172,19 +176,19 @@ the data plane (`docs/08`). emulation and object/control replies are implemented; the stock guest stack completes `cuInit`, device enumeration, `cuCtxCreate` and the driver-API smoke test (allocation, copies, free and destroy). -* **Current frontier:** a guest CUDA kernel launch is forwarded and reaches - the physical engine queue — the front end deposits the launch and the - scheduling path accepts it — but the engine does not yet execute the - forwarded launch descriptor: an engine-authored release is the only proof - of execution and it is not observed. The per-boot green/stall variance on - the emulation path is an open issue, tracked in the repository. The chain, - the evidence and the live state are on +* **Current frontier:** guest CUDA kernel launches execute on the physical + GPU with engine-authored completions relayed to the guest; the verified + clients are green on fresh boots. The leading edge is the large-buffer + staged-transfer class (some large multi-chunk clients still mis-verify), + re-measurement of reproducibility across fresh boots after the + boot-poisoning fix, and per-guest execution isolation. The chain, the + evidence and the live state are on [`Execution-Frontier.md`](Execution-Frontier.md). * **Data plane today:** the host module decodes guest push buffers as method streams and executes the memory and semaphore work it understands (memsets, copies, semaphore releases), and forwards engine-class work to - the physical backend; kernel launches are forwarded but not yet executed - by the engine. + the physical backend; kernel launches now execute on the engine through + the forwarded path, with completions relayed back to the guest. ## 5. The crash-path lesson: replies are consumed data diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md index fe21fda..2d9d80c 100644 --- a/wiki/Execution-Frontier.md +++ b/wiki/Execution-Frontier.md @@ -1,15 +1,16 @@ # The Execution Frontier -> **What this page is.** The current leading edge of guest-facing execution: -> the chain that must fire for a stock guest CUDA kernel to run on the -> physical GPU, what has been proven about it, where it stops, how we know, -> and the open lines of investigation and theories. It is evidence-led: every -> statement carries a live observation, a decompilation finding, or a clearly -> labelled theory. Condensed from +> **What this page is.** The state of guest-facing execution: the chain that +> must fire for a stock guest CUDA kernel to run on the physical GPU, what has +> been proven about it, how we know, and where the leading edge is now. The +> chain fires end to end: verified guest CUDA clients execute on the physical +> GPU and wake up correctly; the leading edge is now breadth and hardening. It +> is evidence-led: every statement carries a live observation, a +> decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. -Status: 2026-10-12 (source page dated 2026-10-06), tree `master @ 2f066727`. +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. @@ -45,46 +46,62 @@ guest: builds launch + method stream | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed | | 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) | | 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised) | `[wall]` not observed | -| 7 | **Engine reads the QMD** | `[wall]` not observed | +| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed | +| 7 | **Engine reads the QMD** | `[live]` observed | | 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in | -| 9 | GR dispatches the grid to the SMs | never observed on this stack | -| 10 | Engine authors release + channel semaphore | absent for forwarded launches | -| 11 | Host relays completion; guest wakes | the guest waits and times out | +| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve | -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +**One-line summary:** the whole chain is live; the leading edge is now breadth +and hardening — the large-buffer staged-transfer class, reproducibility across +boots, and per-guest execution isolation. ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire the ring entry is consumed and the front-end -methods run — but the launch descriptor is never materialised, the engine -never reads a QMD, no engine-authored completion appears, and the guest blocks -in its synchronisation wait. - -This is **not** a context-validity wall (ruled out at dispatch) and **not** a -window-mapping wall for the descriptor destination (that install is accepted). -The wall is the **form of the launch method stream**: what the front-end -requires in order to install the inline descriptor, and what the engine -requires in order to read and dispatch it. +A stock, unmodified guest CUDA program executes its kernels on the physical +GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. Engine-authored completions relay to the guest and its +synchronisation waits resolve. + +The leading edge has moved from the launch chain itself — it fires — to +breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise and the first + CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report). + Re-measurement across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. ## 3. How we know (observables and their limits) | Observable | What it proves | What it does not prove | |---|---|---| -| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | nothing about the engine | -| Method execution on the ladder channel | the front-end can execute the method vocabulary | **not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten with a detectable sentinel | whether the descriptor was materialised after submission | a read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | the inline-install method did not run | it does not say why | -| Engine-authored release word absent; poll times out | the engine did not complete the launch | it does not say whether the engine started | +| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results | +| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches | +| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient | +| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress | | No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction | -| Channel instance context flag set | the channel is context-valid at dispatch | nothing about the descriptor path | -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +**Consequence:** acceptance is verified guest results plus the engine-authored +release; a green run counts as progress only when it reproduces across fresh +boots. ## 4. What is exonerated (do not relitigate) @@ -98,68 +115,64 @@ signals, which we have already learned to distinguish. already initialised in the same cycle; not the dispatch delta. * Parenting/TSG for the fetch — both variants fetch identically. * The copy-engine path — CE-class work, releases and fault relay are live - (default off); the GR path is a separate problem. + (default off); the GR path is separate. +* The descriptor-gate line of investigation (the old T1–T4 theories) — the + inline-descriptor install, the engine read, the dispatch, the + engine-authored release and the guest wake-up are all observed; closed by + observation. +* The old boot-poisoning defect — root-caused to the status-queue consumption + cursor and fixed. ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** The forwarded stream includes a verbatim copy of the guest's -prelude whose inline-data writes target addresses not present in the engine's -view of the channel's page tables, and the referenced data is not relocated. -The shorter, proven stream shape fires its tail (channel semaphore) but still -does not schedule the descriptor. +**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged +transfers between guest memory and device memory are under repair: some large +multi-chunk clients still mis-verify. -**Issue B — the descriptor-virtualization control method is absent.** The -method that arms the inline-descriptor install path is not present in the -forwarded stream. Fixing it should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. +**Issue B — reproducibility across boots (re-measurement).** The +boot-poisoning defect (wrong status-queue consumption cursor under the +driver's status-queue mode; the reply ring could desynchronise under load, the +first client would fail) was root-caused and fixed. Re-measurement across +fresh boots is in progress; no "every boot" claim is made until it completes. -**Issue C — diagnostic hygiene.** The sentinel placed at word 0 overlaps real -descriptor fields; it must be moved to an unused word so diagnostics do not -perturb what they measure. +**Issue C — per-guest execution isolation.** Multiple mediated devices with +different emulated VRAM sizes and multiple guests coexist at the device level; +per-guest execution isolation for simultaneous CUDA guests is in progress. ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline and **re-add the - virtualization control method** — one variable at a time, with the install - sentinel and the engine-read signal as the readout. -2. **Relocate or drop the verbatim prelude** so inline-data writes target - addresses present in the engine's view, or replace it with the host-side - staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference** — our own ladder dispatching a - descriptor built from the same source material, or a captured working - launch of the stock stack on the same GPU — converting the remaining work - from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware debugger before spending a supervised cycle - ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)). - -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the control method enables the - install path. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Its refused writes leave - the launch path unable to schedule the descriptor. -* **T4 — relocation is the missing transform.** The guest's descriptor is - valid for the guest's view of memory; the forwarding path must relocate - referenced data into the channel's view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +1. **Repair the large-buffer staged-transfer class** — the three green clients + stay the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility. +3. **Per-guest execution isolation** — build from the device-level coexistence + to isolation for simultaneous CUDA guests. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel; each entry is accepted only on its own verification + output (`bad=0` + `STATUS OK`). + +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed, so +the launch chain they questioned is proven end to end. The remaining work is +engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest +isolation), each with an acceptance signal named before the work. ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end; the guest's own verification (`bad=0` + `STATUS OK`) + is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred. * **Emulator verdicts are instruction/memory-level.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** * **A wall is closed only by an acceptance signal**, named before the experiment. diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md index 1b83203..bbca084 100644 --- a/wiki/Physical-GPU-Backend.md +++ b/wiki/Physical-GPU-Backend.md @@ -5,7 +5,7 @@ > guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) > and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partially built; `[wall]` current blocker; `[target]` planned end state. @@ -64,13 +64,15 @@ independently verified: 5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]` 6. **Work path** — decoded guest pushes become real push streams; resolved host pages - map through module-built scratch page tables; the firmware schedules the work. `[gated]` + map through module-built scratch page tables; the firmware schedules the work. + `[live]` observed end to end for the verified guest kernel clients. 7. **Completion relay** — hardware-completed semaphores land in guest-polled memory; - events and faults relay through the existing MSI-X injection path. `[gated]` + events and faults relay through the existing MSI-X injection path. + `[live]` engine-authored completions relay to the guest and its waits resolve. ## 5. The two planes and the wiring -The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software. +The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software. ```text GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU @@ -85,8 +87,9 @@ INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live] RM battery, UVM registration [live] WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell host pump decodes; eligible CE copies forwarded via module-built scratch - page tables; GR: launch reaches the front-end, descriptor not seen [wall] -DONE engine authors the release in guest memory; host relays via MSI-X [gated] + page tables; GR: launches execute on the engine, completions relay + to the guest [live] +DONE engine authors the release in guest memory; host relays via MSI-X [live] TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown -> protected region down -> distribution driver rebound [live] FAIL a failed mutating operation parks the device in a locked fail-safe; @@ -104,17 +107,20 @@ FAIL a failed mutating operation parks the device in a locked fail-safe; | Power domain held active across the ownership window | verified across supervised cycles `[live]` | | Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | | Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | -| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` | -| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` | +| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` | +| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest | ## 7. The execution frontier The RM client ladder reaches the compute engine (a GR0 channel plus a -compute-class object). A forwarded guest CUDA launch now -reaches the front-end (ring entry fetched, methods executed) but the inline -launch-descriptor install has not been observed and no grid has dispatched; -context validity at dispatch is ruled out. The wall is the form of the -forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md). +compute-class object). Forwarded guest CUDA launches now execute on the +engine: the inline launch descriptor installs and is read, the grid +dispatches, the engine authors the completion, the host relays it, and the +guest's synchronisation wait resolves — verified on fresh boots with the +vector-add, matrix-multiply and transformer clients. The leading edge is the +large-buffer staged-transfer class (under repair), reproducibility across +fresh boots, and per-guest execution isolation — see +[`Execution-Frontier.md`](Execution-Frontier.md). A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step. @@ -147,6 +153,9 @@ fail-safe that is never auto-retried; recovery is a supervised step. access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). +* Multiple mediated devices with different emulated VRAM sizes and multiple + guests already coexist at the device level; per-guest execution isolation + for simultaneous CUDA guests is in progress. ## 10. Open questions @@ -159,7 +168,7 @@ fail-safe that is never auto-retried; recovery is a supervised step. * [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up. * [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design. -* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane. +* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane. * [`Host-Module.md`](Host-Module.md) — the module that owns both planes. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings. * [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture. diff --git a/wiki/Project-Status.md b/wiki/Project-Status.md index d6e60d1..e63a198 100644 --- a/wiki/Project-Status.md +++ b/wiki/Project-Status.md @@ -10,7 +10,7 @@ > (the execution frontier). When this page and the docs disagree, the docs tree > wins. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or behind supervised gates; `[wip]` partially implemented; `[wall]` current blocker; `[target]` named success state; `[planned]` designed, not built. @@ -24,9 +24,9 @@ named success state; `[planned]` designed, not built. | GSP-RM message queues and RPC transport | `[live]` working | | Stock guest driver probe and `nvidia-smi` (TU104) | `[live]` working | | CUDA userspace init and the driver-API smoke (`cuInit`, device enumeration, `cuCtxCreate`, allocation, copies, free, destroy) | `[live]` working — the smoke prints `ALL OK` and the 64-byte payload round-trips exactly | -| Guest CUDA kernel execution on real hardware | `[wall]` a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue | -| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above | -| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` designed; the baseline service-state ownership work is in progress | +| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. Remaining: the large-buffer staged-transfer class is under repair (some large multi-chunk clients still mis-verify); fresh-boot re-measurement is in progress after the boot-poisoning fix; per-guest execution isolation for simultaneous CUDA guests is in progress | +| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend | +| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | | OpenGL / Vulkan (3D), video | `[planned]` | | Windows guests | `[planned]` (target) | | Xen / Argo / OpenXT / Qubes Air transports | `[planned]` (the emulation core is hypervisor-independent by construction) | @@ -43,23 +43,32 @@ The project's definition of done for CUDA requires all of the following 2. Context creation and the full driver-API surface (`cuMemAlloc`, copies, free, destroy) — done. 3. A guest kernel launch that **executes on real hardware** and is proven by an - engine-authored completion — the current frontier. + engine-authored completion — demonstrated: the verified clients execute on + the physical engine and the engine-authored completions relay to the guest. 4. Memory the guest's work refers to resolving correctly end to end (the VA↔PA model) — partially, for the paths exercised so far. -5. Correct completion, semaphore and fault semantics back to the guest — CE - class work has the relay; GR is open. -6. Reproducibility: every boot, not a lucky one — an open issue. -7. No ad-hoc default-off parameters required for the canonical path. +5. Correct completion, semaphore and fault semantics back to the guest — the + GR completion path now relays (engine-authored releases wake the guest's + synchronisation waits); fault semantics remain partial. +6. Reproducibility: every boot, not a lucky one — root-caused and fixed (the + boot-poisoning defect: the wrong status-queue consumption cursor could + desynchronise the reply ring under load and poison the boot); + re-measurement across fresh boots is in progress. +7. No ad-hoc default-off parameters required for the canonical path — still + open. -Until every gate holds, the honest formulation is: **initialization and the -driver-API path work; engine execution is in progress.** +Until every gate holds, the honest formulation is: **initialization, the +driver-API path and kernel execution on the physical GPU work and verify; +hardening (large-buffer transfers, boot-to-boot reproducibility) and +per-guest isolation remain.** ## 3. How to see it yourself The reference setup takes about half an hour from a fresh host ([`Install-and-First-Run.md`](Install-and-First-Run.md)); the current expected outcome is exactly the ladder above — driver probe, `nvidia-smi`, the full -driver-API smoke, and the execution frontier where it stands. The +driver-API smoke, and guest CUDA kernels executing and verifying on the +physical GPU. The risc-v-gsp-rm-debugger workbench ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) is the host-side tooling used to study the firmware side of that frontier. @@ -68,7 +77,7 @@ tooling used to study the firmware side of that frontier. Every claim above carries a date and a tag, maps to a maintained document in the repository, and is refreshed as milestones land. The execution story, with -its evidence and its open theories, is on +its evidence, its resolved theories and its open work, is on [`Execution-Frontier.md`](Execution-Frontier.md). ## 5. Related pages |