1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
# ISSUE-0001: Guest CUDA is not reproducibly green — the client stalls at DEVICE (GR context init never submitted)
* Status: FIX IMPLEMENTED (VALIDATION PENDING)
* Filed: 2026-10-08, session `smiley-m1`/deep-debug | Priority: **top (blocks 1.0)** | Component: kernel (`vgsp.ko` sema service + RM emulation) with a userspace-watcher race as the variance source
## Symptom
`victory_mail` (and every other stock-CUDA client) run in the guest does one
of two things on a cold boot:
* **Green**: prints `DEVICE …`, then `MAIL …`, `VADD n=32768 bad=0`, `STATUS OK`
within ~15–190 ms of engine work; host forwards CE and GR pushes
(`work_gr req>0`, `GUEST GR LAUNCH OK`).
* **Stalled**: prints `DEVICE …` and then nothing. The guest RM spins in
`NV_ESC_RM_CONTROL` (`0x2a`, `NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS`
`0x20801702`) at ~1 Hz forever. The GR rings are **never written by the
guest**; the host sees `work_gr req=0`; FB scans find only CE ring entries.
The split is per-boot. Historically (m104, m141-r3, vmkeep6/7/14) roughly
30–50 % of boots were green on a *fresh* host; after ~8 h of host uptime and
~20 module claim/release cycles the green rate goes to ~0 % and the failure
can degrade further (the guest agent never comes up; smiley r1–r3).
**This is not a regression against any code change: m141-r3 (the accepted
victory, commit `895ab8b8`) was itself a lucky boot of the same intermittent
class; the same recipe/commit stalls on other boots.**
## Impact
* Blocks the 1.0 acceptance: "stock guest CUDA works flawlessly, every time".
* Forces `GRFWD_RETRIES`/`GRFWD_STALL_SECS` lottery runs (minutes per boot)
and hides new regressions under the stall noise.
* Blocks everything behind it: smiley/multi-kernel validation, P2 kernels,
performance numbers, multi-mdev, two guests.
## Evidence
One-line discriminators (host dmesg counters, `sema-svc:` periodic stats):
| Counter | Green (m141-r3) | Stalled (p13-r4) | Meaning |
|---|---|---|---|
| `pushes` / `applied` | 655 / 655 | **90 / 89, then frozen** | one completion never lands |
| `pushes_nomap` | 0 | **1** | one release page had no translation |
| `gr_rel` / `pub_calls` | 219 / 2801 | **0 / 0** | guest never starts GR |
| `ring_auto` | 26 | **6** (CE only) | GR rings never created/pinned |
| `work_gr req` | 2+ | **0** | no GR submission reaches the host |
| `injected mapping` lines | 10 / 40 | **0** | watcher variance |
Other confirmed facts:
* `tools/guest-diag/guest_kprobes.sh` armed all 6 kprobes (`ok`×6) and the
semfix watcher translated CE sem words in stalled boots (p13-r4) — so the
stall is **not** the earlier "kprobes never armed" class (`6eee9ac3` fix
was real but is not this issue).
* The guest's own wait is a tracker completion: wedged stacks end in
`uvm_tracker_wait_for_entry ← uvm_push_end_and_wait ← channel_pool_add ←
uvm_channel_manager_create` (repeat-client instance) — i.e. the guest is
waiting for a CE push completion to become visible in a semaphore/waiter
page.
* `svc_apply_pending()` keeps a release that cannot translate ("not
translatable yet: keep") and retries every pass; nothing else ever moves it.
* Related (separate correctness bug, fixed under this issue): the
`PROMOTE_CTX` (0x2080012b) reply. See "Fixes".
## Root cause
The completion (release) of the client's last CE push in the boot-critical
window cannot be written back because its semaphore/waiter page has no
VA→PA translation yet. The release is deferred into `svc_pending[]` and waits
for a translation that arrives only from the userspace watcher pipeline
(`pin_gr_stream`/`semfix_uvmt`) or from the kernel's *partial* fallback set.
The watcher pipeline is racy across boots (kprobe arming vs the client's
first push, scanner vs pool allocation, kcore scans vs tracker windows), so
on some boots every fallback misses; the guest's tracker wait never wakes;
the client never creates its GR channels; nothing ever reaches the GR rings.
Kernel fallback coverage was incomplete exactly for the pool/waiter page: the
pool resolver (`svc_pool_pa_resolve`) had no walk of the **driver-published
VAS page directories** (`svc_walk_vaspaces`) — the deterministic, watcher-free
source the general resolver (`svc_lookup_trusted`) has always used.
Confidence: high for the mechanism (the deferral + the missing walk + the
observed freeze at `applied=89/90`). The exact *reason the walk itself misses
on some boots* (directory registered later than the first push, or the
watcher path succeeding first on greens) remains boot-timing dependent; the
fix below removes the dependence rather than chasing the timing.
## Fix
1. **`kernel/gsp_sema_service.c` — `svc_pool_pa_resolve`:** add
`svc_walk_vaspaces()` (driver-published directories, read-back proven)
between the self-discovered walk and the targeted rm_mem scan. Kernel-native
translation of the deferred release page no longer depends on the watcher.
2. **`kernel/gsp_sema_service.c` — `svc_pending` starvation telemetry:**
per-entry `tries` counter + bounded logs at 4/64/1024 attempts naming
`va`, `payload` and the consequence. A future starvation is visible in
dmesg instead of silent.
3. **`kernel/gsp_rm_emulation.c` — PROMOTE_CTX reply (supersedes
`fa9296af`):** the reply must keep `bInitialize` set. Per
`kernel_graphics_object.c` `kgrobjPromoteContext_IMPL`:
`if (!params.promoteEntry[i].bInitialize) continue;
kgrctxMarkCtxBufferInitialized(...)` — the flag is the server's
"initialized" ACK; clearing it (fa9296af) made every ctx buffer re-promote
forever. This was a correctness bug in the stall's vicinity, not the stall
itself (the stall predates and outlives it).
Durable direction (if a stall variant survives the fix): fold the watcher's
job into the kernel entirely (complete `svc_pb_discover`/`svc_pd` coverage of
the CE push/pool windows) so the watchers are accelerators only.
## Validation plan
1. Fresh host boot (reboot approved 2026-10-08): accepted recipe, plain
runner — expect `victory_mail` green.
2. **10/10 consecutive cold boots** green within seconds; zero stall
captures; `pending release starved` never logs; `pushes_nomap` returns 0
on every green.
3. Re-run the smiley workload (`tools/cuda-ab/smiley.cu`) per green VM.
4. Rollback: revert commits 1–3 independently; rebuild; the previous
behaviour (lottery) returns without new failure modes.
## History
* 2026-10-02..07: stall hunted through m1xx series; greens m104, m141-r3;
kprobe arming fixed (`6eee9ac3`); plan to kernel-native discovery recorded.
* 2026-10-08: campaigns p1–p13 + smiley r1–r3 all stall; evidence shows the
missing completion + missing pool-page translation; `fa9296af` PROMOTE_CTX
change tested and disproven (and is itself corrected here).
## Resolution update (2026-10-08 evening) — root cause found, gate in progress
**Root cause of the reproducible stall (fixed):** commit `1c5864ad` (m143,
"BAR1 size = fb_emul_size") made the guest see a 512 MB BAR1 while the
physical aperture is 256 MB. Every build at/after m143 stalls the CUDA
client (spin inside its first `cudaMalloc`); the pre-m143 m140 reference
reports 256 MB and is green. `26263e1d` restores the physical exposure
(forward-fix under design for the multi-size types: see
`/tmp/opencode/BAR1-512-ANALYSIS.md`). Merged with two companion reverts
(`8791acc3` on master): the walk-pin fix A revert (`2e1c9d2c`, it gated
guest GR pushes) and the end-of-pass GR counter publish restore
(`ed6d458d`, H3's early publish read put=0).
**Interim gate result:** 9/10 green, then a wedge on run #10 (`rep10-r10`,
2026-10-08 17:25 EDT): GSP-RM ladder completed host-side, but the guest's
first UVM push was never serviced (`ring_auto=1`, `gpfifo done=1`, zero
pushes decoded), the client produced no output, and the guest hard-died
~10 min later. Guest logs (extracted to `/tmp/opencode/r10-guest-logs/`,
analysis `/tmp/opencode/R10-WEDGE-ANALYSIS.md`): the driver sequence is
identical to green runs until 14 ms after the 16 MB UVM pushbuffer system-
page allocation, where it goes silent — before the GPU-VA window mappings
every green run performs next. Leading hypothesis: ring discovery missed
the client channel's rings on that boot (host `ring_auto=1`), so the first
completion wait never resolved. Watchers/kprobes were armed normally.
Discriminating check: `uvtw`/`uvsv` kprobe traces during a stall (a uvtw
loop with value 0 and no `uvbp/uvch` submit = the push never left the
guest; submission present = the ring decode/emulation missed it).
Wedge rate seen so far: 1 in 10 monitored cold boots. A clean 10/10 gate
re-run is in progress; the wedge stays open under this issue until a
10/10 consecutive-green streak is recorded.
|
