1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
|
# ISSUE-0001: Guest CUDA is not reproducibly green — the client stalls at DEVICE (GR context init never submitted)
* Status: FIX IMPLEMENTED (VALIDATION PENDING)
* Filed: 2026-10-08, session `smiley-m1`/deep-debug | Priority: **top (blocks 1.0)** | Component: kernel (`vgsp.ko` sema service + RM emulation) with a userspace-watcher race as the variance source
## Symptom
`victory_mail` (and every other stock-CUDA client) run in the guest does one
of two things on a cold boot:
* **Green**: prints `DEVICE …`, then `MAIL …`, `VADD n=32768 bad=0`, `STATUS OK`
within ~15–190 ms of engine work; host forwards CE and GR pushes
(`work_gr req>0`, `GUEST GR LAUNCH OK`).
* **Stalled**: prints `DEVICE …` and then nothing. The guest RM spins in
`NV_ESC_RM_CONTROL` (`0x2a`, `NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS`
`0x20801702`) at ~1 Hz forever. The GR rings are **never written by the
guest**; the host sees `work_gr req=0`; FB scans find only CE ring entries.
The split is per-boot. Historically (m104, m141-r3, vmkeep6/7/14) roughly
30–50 % of boots were green on a *fresh* host; after ~8 h of host uptime and
~20 module claim/release cycles the green rate goes to ~0 % and the failure
can degrade further (the guest agent never comes up; smiley r1–r3).
**This is not a regression against any code change: m141-r3 (the accepted
victory, commit `895ab8b8`) was itself a lucky boot of the same intermittent
class; the same recipe/commit stalls on other boots.**
## Impact
* Blocks the 1.0 acceptance: "stock guest CUDA works flawlessly, every time".
* Forces `GRFWD_RETRIES`/`GRFWD_STALL_SECS` lottery runs (minutes per boot)
and hides new regressions under the stall noise.
* Blocks everything behind it: smiley/multi-kernel validation, P2 kernels,
performance numbers, multi-mdev, two guests.
## Evidence
One-line discriminators (host dmesg counters, `sema-svc:` periodic stats):
| Counter | Green (m141-r3) | Stalled (p13-r4) | Meaning |
|---|---|---|---|
| `pushes` / `applied` | 655 / 655 | **90 / 89, then frozen** | one completion never lands |
| `pushes_nomap` | 0 | **1** | one release page had no translation |
| `gr_rel` / `pub_calls` | 219 / 2801 | **0 / 0** | guest never starts GR |
| `ring_auto` | 26 | **6** (CE only) | GR rings never created/pinned |
| `work_gr req` | 2+ | **0** | no GR submission reaches the host |
| `injected mapping` lines | 10 / 40 | **0** | watcher variance |
Other confirmed facts:
* `tools/guest-diag/guest_kprobes.sh` armed all 6 kprobes (`ok`×6) and the
semfix watcher translated CE sem words in stalled boots (p13-r4) — so the
stall is **not** the earlier "kprobes never armed" class (`6eee9ac3` fix
was real but is not this issue).
* The guest's own wait is a tracker completion: wedged stacks end in
`uvm_tracker_wait_for_entry ← uvm_push_end_and_wait ← channel_pool_add ←
uvm_channel_manager_create` (repeat-client instance) — i.e. the guest is
waiting for a CE push completion to become visible in a semaphore/waiter
page.
* `svc_apply_pending()` keeps a release that cannot translate ("not
translatable yet: keep") and retries every pass; nothing else ever moves it.
* Related (separate correctness bug, fixed under this issue): the
`PROMOTE_CTX` (0x2080012b) reply. See "Fixes".
## Root cause
The completion (release) of the client's last CE push in the boot-critical
window cannot be written back because its semaphore/waiter page has no
VA→PA translation yet. The release is deferred into `svc_pending[]` and waits
for a translation that arrives only from the userspace watcher pipeline
(`pin_gr_stream`/`semfix_uvmt`) or from the kernel's *partial* fallback set.
The watcher pipeline is racy across boots (kprobe arming vs the client's
first push, scanner vs pool allocation, kcore scans vs tracker windows), so
on some boots every fallback misses; the guest's tracker wait never wakes;
the client never creates its GR channels; nothing ever reaches the GR rings.
Kernel fallback coverage was incomplete exactly for the pool/waiter page: the
pool resolver (`svc_pool_pa_resolve`) had no walk of the **driver-published
VAS page directories** (`svc_walk_vaspaces`) — the deterministic, watcher-free
source the general resolver (`svc_lookup_trusted`) has always used.
Confidence: high for the mechanism (the deferral + the missing walk + the
observed freeze at `applied=89/90`). The exact *reason the walk itself misses
on some boots* (directory registered later than the first push, or the
watcher path succeeding first on greens) remains boot-timing dependent; the
fix below removes the dependence rather than chasing the timing.
## Fix
1. **`kernel/gsp_sema_service.c` — `svc_pool_pa_resolve`:** add
`svc_walk_vaspaces()` (driver-published directories, read-back proven)
between the self-discovered walk and the targeted rm_mem scan. Kernel-native
translation of the deferred release page no longer depends on the watcher.
2. **`kernel/gsp_sema_service.c` — `svc_pending` starvation telemetry:**
per-entry `tries` counter + bounded logs at 4/64/1024 attempts naming
`va`, `payload` and the consequence. A future starvation is visible in
dmesg instead of silent.
3. **`kernel/gsp_rm_emulation.c` — PROMOTE_CTX reply (supersedes
`fa9296af`):** the reply must keep `bInitialize` set. Per
`kernel_graphics_object.c` `kgrobjPromoteContext_IMPL`:
`if (!params.promoteEntry[i].bInitialize) continue;
kgrctxMarkCtxBufferInitialized(...)` — the flag is the server's
"initialized" ACK; clearing it (fa9296af) made every ctx buffer re-promote
forever. This was a correctness bug in the stall's vicinity, not the stall
itself (the stall predates and outlives it).
Durable direction (if a stall variant survives the fix): fold the watcher's
job into the kernel entirely (complete `svc_pb_discover`/`svc_pd` coverage of
the CE push/pool windows) so the watchers are accelerators only.
## Validation plan
1. Fresh host boot (reboot approved 2026-10-08): accepted recipe, plain
runner — expect `victory_mail` green.
2. **10/10 consecutive cold boots** green within seconds; zero stall
captures; `pending release starved` never logs; `pushes_nomap` returns 0
on every green.
3. Re-run the smiley workload (`tools/cuda-ab/smiley.cu`) per green VM.
4. Rollback: revert commits 1–3 independently; rebuild; the previous
behaviour (lottery) returns without new failure modes.
## History
* 2026-10-02..07: stall hunted through m1xx series; greens m104, m141-r3;
kprobe arming fixed (`6eee9ac3`); plan to kernel-native discovery recorded.
* 2026-10-08: campaigns p1–p13 + smiley r1–r3 all stall; evidence shows the
missing completion + missing pool-page translation; `fa9296af` PROMOTE_CTX
change tested and disproven (and is itself corrected here).
|
