# ISSUE-0002: Multiple CUDA kernels/workloads inside one VM — sequential, concurrent and simultaneous must all work

* Status: OPEN (root cause established for the sequential case; concurrent/simultaneous untested)
* Filed: 2026-10-08 | Priority: **p1 (blocks 1.0 "multiple kernels", follows ISSUE-0001)** | Component: kernel per-client/session state + guest UVM/RM interaction

## Symptom

* **Sequential (known):** a *second* CUDA process in the same boot wedges:
  spins at ~100 % CPU with no output (not even `DEVICE`), host traffic for it
  is zero. The first client in the same boot is green (10–34 s). Until fixed,
  operations are "one client per boot, reboot for a rerun".
* **Concurrent:** not yet tested (blocked by the sequential wedge).
* **Simultaneous (one process, multiple streams/kernels at once):** not yet
  tested; expected to hit the same per-channel/per-client state class.

## Impact

* Blocks 1.0 acceptance "arbitrary CUDA kernels flawlessly" beyond a single
  sequential run, all realistic workloads (frameworks spawn many processes),
  and the P2 kernel suite (`tools/cuda-ab/p2/run_guest.sh`) which runs
  multiple programs per boot by design.

## Evidence

* Wedged guest stack (vmkeep10 run C, pid 1447):
  `uvm_ioctl → uvm_api_register_gpu → uvm_va_space_register_gpu →
  uvm_gpu_retain_by_uuid+0x44 [nvidia_uvm]` (main thread), second thread in
  `do_poll`.
* Source (guest UVM, `kernel-open/nvidia-uvm/uvm_gpu.c:3129-3163`):
  `uvm_gpu_retain_by_uuid` takes `g_uvm_global.global_lock`;
  `uvm_gpu_release_locked` (last-release path) holds the same lock while
  calling `nvUvmInterfaceUnregisterGpu` — an RM call to the host. If that
  exchange never completes host-side, the lock is held forever and every
  later client wedges at registration.
* Host view (vmkeep10): run A full RM traffic (channel `0xcaf000d4`
  registered), then idle; run B issues **zero** RM/sema traffic; each client
  run re-runs the GSP boot handshake (`GSP_INIT_DONE` per run start), so the
  host must fully re-arm per session. Around run A's exit:
  `sema-svc: completion hint rc=-22` twice on the fresh channel.
* Host counters freeze for the second run (`counter pub_calls=2790
  gr_rel=216`, `drain held=2`, `faked=0`).
* Evidence pack: `backups/evidence-20261002-victory/analysis-2026-10-08-repeat-stall/`
  (`run1/2/3.log`, `dmesg-w-vmkeep10.log`, semfix/pin logs).
* An m146/m147 experiment (re-arm per-rank one-shot latches on channel
  registration; derive CE rank from ring VA; re-arm on GSP session reset via
  `gsp_sema_service_session_reset()` from `gsp_rm_emu_reset`) was implemented
  and **reverted** — it did not fix run B and temporarily stopped first-run
  greens from reproducing. WIP diff:
  `…/analysis-2026-10-08-repeat-stall/m146-m147-session-reset-WIP.patch`.

## Root cause

Established for the sequential case: per-client state in the host sema/GR
service is not reset when a new guest client context appears (the guest
re-runs the GSP boot handshake per client). The old client's channel/rank/
ring/pool state makes the host's completion path go quiet for the new client
(`faked=0`, `drain held=2`, zero new applies), so the new client's UVM
registration RM exchange never completes — and because the guest UVM holds
`global_lock` across that exchange (`uvm_gpu_release_locked` →
`nvUvmInterfaceUnregisterGpu`), the whole guest UVM wedges for every later
client too. Concurrent/simultaneous cases are expected to share this class
(per-channel one-shot latches, single per-vgpu sema context).

## Fix / fix plan

1. Reset/rebind host per-client/per-channel sema+GR state when a new client
   context appears (new GSP session / new channel registration): re-arm
   one-shot rank latches, clear stale ring/userd publications, bound any
   drain hold so a miss degrades instead of freezing (the `drain held` path).
2. Per-vgpu sema context + P0 isolation work: ordered hunks in
   `backups/evidence-20261002-victory/analysis-2026-10-07/sa-p2-patchnotes.md`
   (P0-1..P0-7, stages A1–A6, gates G1/G3, `users/idle` anti-UAF drain).
3. Graduation criteria: sequential 3+ clients in one boot green; two
   concurrent processes (different streams) green; one process with N
   simultaneous streams/kernels green; then the P2 suite in a single boot.

## Validation plan

* Sequential: 3 consecutive `victory_mail` runs in one VM, all green.
* Concurrent: two `victory_mail` instances launched together, both green.
* Simultaneous: multi-stream kernel test (P2 suite addition) green,
  oracle-matched against bare metal.
* Rollback: patch reverts independently (see WIP diff for the reverted
  experiment).

## History

* 2026-10-08 00:05: first-client greens became deterministic (`6eee9ac3`);
  second-client wedge discovered the same night (vmkeep9–14).
* 2026-10-08 01:00: root-cause chain assembled; m146/m147 experiment
  reverted as ineffective; kernel-native per-client reset identified as the
  durable path.