From f8edfb21e0880935fbe5aa62996fcd9af64c32ff Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 8 Oct 2026 11:07:42 -0400 Subject: debug/issue: issue register + ISSUE-0001 (reproducibility) + ISSUE-0002 (multiple kernels per VM) - debug/issue/README.md: when to file (nasty bugs only; default is no issues), template, numbering, lifecycle incl. solved/ move. - issue-number-1.md: guest CUDA is not reproducibly green (top priority); evidence discriminators, root cause (starved pool release translation), the three fixes landed, 10/10-boot validation plan. - issue-number-2.md: multiple kernels in one VM (sequential/concurrent/ simultaneous) must work; second-client wedge root-cause chain + plan. - tools/cuda-ab/smiley.cu: 4-pass vectorAdd that renders a 64x32 ASCII smiley from GPU-computed distance fields; ab.sh gains a generic 'push' mode so any WORKLOAD binary can be delivered to the guest. --- debug/issue/issue-number-1.md | 128 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 debug/issue/issue-number-1.md (limited to 'debug/issue/issue-number-1.md') diff --git a/debug/issue/issue-number-1.md b/debug/issue/issue-number-1.md new file mode 100644 index 0000000..f23721f --- /dev/null +++ b/debug/issue/issue-number-1.md @@ -0,0 +1,128 @@ +# ISSUE-0001: Guest CUDA is not reproducibly green — the client stalls at DEVICE (GR context init never submitted) + +* Status: FIX IMPLEMENTED (VALIDATION PENDING) +* Filed: 2026-10-08, session `smiley-m1`/deep-debug | Priority: **top (blocks 1.0)** | Component: kernel (`vgsp.ko` sema service + RM emulation) with a userspace-watcher race as the variance source + +## Symptom + +`victory_mail` (and every other stock-CUDA client) run in the guest does one +of two things on a cold boot: + +* **Green**: prints `DEVICE …`, then `MAIL …`, `VADD n=32768 bad=0`, `STATUS OK` + within ~15–190 ms of engine work; host forwards CE and GR pushes + (`work_gr req>0`, `GUEST GR LAUNCH OK`). +* **Stalled**: prints `DEVICE …` and then nothing. The guest RM spins in + `NV_ESC_RM_CONTROL` (`0x2a`, `NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS` + `0x20801702`) at ~1 Hz forever. The GR rings are **never written by the + guest**; the host sees `work_gr req=0`; FB scans find only CE ring entries. + +The split is per-boot. Historically (m104, m141-r3, vmkeep6/7/14) roughly +30–50 % of boots were green on a *fresh* host; after ~8 h of host uptime and +~20 module claim/release cycles the green rate goes to ~0 % and the failure +can degrade further (the guest agent never comes up; smiley r1–r3). + +**This is not a regression against any code change: m141-r3 (the accepted +victory, commit `895ab8b8`) was itself a lucky boot of the same intermittent +class; the same recipe/commit stalls on other boots.** + +## Impact + +* Blocks the 1.0 acceptance: "stock guest CUDA works flawlessly, every time". +* Forces `GRFWD_RETRIES`/`GRFWD_STALL_SECS` lottery runs (minutes per boot) + and hides new regressions under the stall noise. +* Blocks everything behind it: smiley/multi-kernel validation, P2 kernels, + performance numbers, multi-mdev, two guests. + +## Evidence + +One-line discriminators (host dmesg counters, `sema-svc:` periodic stats): + +| Counter | Green (m141-r3) | Stalled (p13-r4) | Meaning | +|---|---|---|---| +| `pushes` / `applied` | 655 / 655 | **90 / 89, then frozen** | one completion never lands | +| `pushes_nomap` | 0 | **1** | one release page had no translation | +| `gr_rel` / `pub_calls` | 219 / 2801 | **0 / 0** | guest never starts GR | +| `ring_auto` | 26 | **6** (CE only) | GR rings never created/pinned | +| `work_gr req` | 2+ | **0** | no GR submission reaches the host | +| `injected mapping` lines | 10 / 40 | **0** | watcher variance | + +Other confirmed facts: + +* `tools/guest-diag/guest_kprobes.sh` armed all 6 kprobes (`ok`×6) and the + semfix watcher translated CE sem words in stalled boots (p13-r4) — so the + stall is **not** the earlier "kprobes never armed" class (`6eee9ac3` fix + was real but is not this issue). +* The guest's own wait is a tracker completion: wedged stacks end in + `uvm_tracker_wait_for_entry ← uvm_push_end_and_wait ← channel_pool_add ← + uvm_channel_manager_create` (repeat-client instance) — i.e. the guest is + waiting for a CE push completion to become visible in a semaphore/waiter + page. +* `svc_apply_pending()` keeps a release that cannot translate ("not + translatable yet: keep") and retries every pass; nothing else ever moves it. +* Related (separate correctness bug, fixed under this issue): the + `PROMOTE_CTX` (0x2080012b) reply. See "Fixes". + +## Root cause + +The completion (release) of the client's last CE push in the boot-critical +window cannot be written back because its semaphore/waiter page has no +VA→PA translation yet. The release is deferred into `svc_pending[]` and waits +for a translation that arrives only from the userspace watcher pipeline +(`pin_gr_stream`/`semfix_uvmt`) or from the kernel's *partial* fallback set. +The watcher pipeline is racy across boots (kprobe arming vs the client's +first push, scanner vs pool allocation, kcore scans vs tracker windows), so +on some boots every fallback misses; the guest's tracker wait never wakes; +the client never creates its GR channels; nothing ever reaches the GR rings. + +Kernel fallback coverage was incomplete exactly for the pool/waiter page: the +pool resolver (`svc_pool_pa_resolve`) had no walk of the **driver-published +VAS page directories** (`svc_walk_vaspaces`) — the deterministic, watcher-free +source the general resolver (`svc_lookup_trusted`) has always used. + +Confidence: high for the mechanism (the deferral + the missing walk + the +observed freeze at `applied=89/90`). The exact *reason the walk itself misses +on some boots* (directory registered later than the first push, or the +watcher path succeeding first on greens) remains boot-timing dependent; the +fix below removes the dependence rather than chasing the timing. + +## Fix + +1. **`kernel/gsp_sema_service.c` — `svc_pool_pa_resolve`:** add + `svc_walk_vaspaces()` (driver-published directories, read-back proven) + between the self-discovered walk and the targeted rm_mem scan. Kernel-native + translation of the deferred release page no longer depends on the watcher. +2. **`kernel/gsp_sema_service.c` — `svc_pending` starvation telemetry:** + per-entry `tries` counter + bounded logs at 4/64/1024 attempts naming + `va`, `payload` and the consequence. A future starvation is visible in + dmesg instead of silent. +3. **`kernel/gsp_rm_emulation.c` — PROMOTE_CTX reply (supersedes + `fa9296af`):** the reply must keep `bInitialize` set. Per + `kernel_graphics_object.c` `kgrobjPromoteContext_IMPL`: + `if (!params.promoteEntry[i].bInitialize) continue; + kgrctxMarkCtxBufferInitialized(...)` — the flag is the server's + "initialized" ACK; clearing it (fa9296af) made every ctx buffer re-promote + forever. This was a correctness bug in the stall's vicinity, not the stall + itself (the stall predates and outlives it). + +Durable direction (if a stall variant survives the fix): fold the watcher's +job into the kernel entirely (complete `svc_pb_discover`/`svc_pd` coverage of +the CE push/pool windows) so the watchers are accelerators only. + +## Validation plan + +1. Fresh host boot (reboot approved 2026-10-08): accepted recipe, plain + runner — expect `victory_mail` green. +2. **10/10 consecutive cold boots** green within seconds; zero stall + captures; `pending release starved` never logs; `pushes_nomap` returns 0 + on every green. +3. Re-run the smiley workload (`tools/cuda-ab/smiley.cu`) per green VM. +4. Rollback: revert commits 1–3 independently; rebuild; the previous + behaviour (lottery) returns without new failure modes. + +## History + +* 2026-10-02..07: stall hunted through m1xx series; greens m104, m141-r3; + kprobe arming fixed (`6eee9ac3`); plan to kernel-native discovery recorded. +* 2026-10-08: campaigns p1–p13 + smiley r1–r3 all stall; evidence shows the + missing completion + missing pool-page translation; `fa9296af` PROMOTE_CTX + change tested and disproven (and is itself corrected here). -- cgit v1.2.3