1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
|
# ISSUE-0002: Multiple CUDA kernels/workloads inside one VM — sequential, concurrent and simultaneous must all work
* Status: OPEN (root cause established for the sequential case; concurrent/simultaneous untested)
* Filed: 2026-10-08 | Priority: **p1 (blocks 1.0 "multiple kernels", follows ISSUE-0001)** | Component: kernel per-client/session state + guest UVM/RM interaction
## Symptom
* **Sequential (known):** a *second* CUDA process in the same boot wedges:
spins at ~100 % CPU with no output (not even `DEVICE`), host traffic for it
is zero. The first client in the same boot is green (10–34 s). Until fixed,
operations are "one client per boot, reboot for a rerun".
* **Concurrent:** not yet tested (blocked by the sequential wedge).
* **Simultaneous (one process, multiple streams/kernels at once):** not yet
tested; expected to hit the same per-channel/per-client state class.
## Impact
* Blocks 1.0 acceptance "arbitrary CUDA kernels flawlessly" beyond a single
sequential run, all realistic workloads (frameworks spawn many processes),
and the P2 kernel suite (`tools/cuda-ab/p2/run_guest.sh`) which runs
multiple programs per boot by design.
## Evidence
* Wedged guest stack (vmkeep10 run C, pid 1447):
`uvm_ioctl → uvm_api_register_gpu → uvm_va_space_register_gpu →
uvm_gpu_retain_by_uuid+0x44 [nvidia_uvm]` (main thread), second thread in
`do_poll`.
* Source (guest UVM, `kernel-open/nvidia-uvm/uvm_gpu.c:3129-3163`):
`uvm_gpu_retain_by_uuid` takes `g_uvm_global.global_lock`;
`uvm_gpu_release_locked` (last-release path) holds the same lock while
calling `nvUvmInterfaceUnregisterGpu` — an RM call to the host. If that
exchange never completes host-side, the lock is held forever and every
later client wedges at registration.
* Host view (vmkeep10): run A full RM traffic (channel `0xcaf000d4`
registered), then idle; run B issues **zero** RM/sema traffic; each client
run re-runs the GSP boot handshake (`GSP_INIT_DONE` per run start), so the
host must fully re-arm per session. Around run A's exit:
`sema-svc: completion hint rc=-22` twice on the fresh channel.
* Host counters freeze for the second run (`counter pub_calls=2790
gr_rel=216`, `drain held=2`, `faked=0`).
* Evidence pack: `backups/evidence-20261002-victory/analysis-2026-10-08-repeat-stall/`
(`run1/2/3.log`, `dmesg-w-vmkeep10.log`, semfix/pin logs).
* An m146/m147 experiment (re-arm per-rank one-shot latches on channel
registration; derive CE rank from ring VA; re-arm on GSP session reset via
`gsp_sema_service_session_reset()` from `gsp_rm_emu_reset`) was implemented
and **reverted** — it did not fix run B and temporarily stopped first-run
greens from reproducing. WIP diff:
`…/analysis-2026-10-08-repeat-stall/m146-m147-session-reset-WIP.patch`.
## Root cause
Established for the sequential case: per-client state in the host sema/GR
service is not reset when a new guest client context appears (the guest
re-runs the GSP boot handshake per client). The old client's channel/rank/
ring/pool state makes the host's completion path go quiet for the new client
(`faked=0`, `drain held=2`, zero new applies), so the new client's UVM
registration RM exchange never completes — and because the guest UVM holds
`global_lock` across that exchange (`uvm_gpu_release_locked` →
`nvUvmInterfaceUnregisterGpu`), the whole guest UVM wedges for every later
client too. Concurrent/simultaneous cases are expected to share this class
(per-channel one-shot latches, single per-vgpu sema context).
## Fix / fix plan
1. Reset/rebind host per-client/per-channel sema+GR state when a new client
context appears (new GSP session / new channel registration): re-arm
one-shot rank latches, clear stale ring/userd publications, bound any
drain hold so a miss degrades instead of freezing (the `drain held` path).
2. Per-vgpu sema context + P0 isolation work: ordered hunks in
`backups/evidence-20261002-victory/analysis-2026-10-07/sa-p2-patchnotes.md`
(P0-1..P0-7, stages A1–A6, gates G1/G3, `users/idle` anti-UAF drain).
3. Graduation criteria: sequential 3+ clients in one boot green; two
concurrent processes (different streams) green; one process with N
simultaneous streams/kernels green; then the P2 suite in a single boot.
## Validation plan
* Sequential: 3 consecutive `victory_mail` runs in one VM, all green.
* Concurrent: two `victory_mail` instances launched together, both green.
* Simultaneous: multi-stream kernel test (P2 suite addition) green,
oracle-matched against bare metal.
* Rollback: patch reverts independently (see WIP diff for the reverted
experiment).
## History
* 2026-10-08 00:05: first-client greens became deterministic (`6eee9ac3`);
second-client wedge discovered the same night (vmkeep9–14).
* 2026-10-08 01:00: root-cause chain assembled; m146/m147 experiment
reverted as ineffective; kernel-native per-client reset identified as the
durable path.
|
