debug/issue: issue register + ISSUE-0001 (reproducibility) + ISSUE-0002 (multiple kernels per VM)

- debug/issue/README.md: when to file (nasty bugs only; default is no
  issues), template, numbering, lifecycle incl. solved/ move.
- issue-number-1.md: guest CUDA is not reproducibly green (top priority);
  evidence discriminators, root cause (starved pool release translation),
  the three fixes landed, 10/10-boot validation plan.
- issue-number-2.md: multiple kernels in one VM (sequential/concurrent/
  simultaneous) must work; second-client wedge root-cause chain + plan.
- tools/cuda-ab/smiley.cu: 4-pass vectorAdd that renders a 64x32 ASCII
  smiley from GPU-computed distance fields; ab.sh gains a generic 'push'
  mode so any WORKLOAD binary can be delivered to the guest.

1 files changed, 92 insertions, 0 deletions

diff --git a/debug/issue/issue-number-2.md b/debug/issue/issue-number-2.md
new file mode 100644
index 0000000..e04e08f
--- /dev/null
+++ b/debug/issue/issue-number-2.md
@@ -0,0 +1,92 @@
+# ISSUE-0002: Multiple CUDA kernels/workloads inside one VM — sequential, concurrent and simultaneous must all work
+
+* Status: OPEN (root cause established for the sequential case; concurrent/simultaneous untested)
+* Filed: 2026-10-08 | Priority: **p1 (blocks 1.0 "multiple kernels", follows ISSUE-0001)** | Component: kernel per-client/session state + guest UVM/RM interaction
+
+## Symptom
+
+* **Sequential (known):** a *second* CUDA process in the same boot wedges:
+  spins at ~100 % CPU with no output (not even `DEVICE`), host traffic for it
+  is zero. The first client in the same boot is green (10–34 s). Until fixed,
+  operations are "one client per boot, reboot for a rerun".
+* **Concurrent:** not yet tested (blocked by the sequential wedge).
+* **Simultaneous (one process, multiple streams/kernels at once):** not yet
+  tested; expected to hit the same per-channel/per-client state class.
+
+## Impact
+
+* Blocks 1.0 acceptance "arbitrary CUDA kernels flawlessly" beyond a single
+  sequential run, all realistic workloads (frameworks spawn many processes),
+  and the P2 kernel suite (`tools/cuda-ab/p2/run_guest.sh`) which runs
+  multiple programs per boot by design.
+
+## Evidence
+
+* Wedged guest stack (vmkeep10 run C, pid 1447):
+  `uvm_ioctl → uvm_api_register_gpu → uvm_va_space_register_gpu →
+  uvm_gpu_retain_by_uuid+0x44 [nvidia_uvm]` (main thread), second thread in
+  `do_poll`.
+* Source (guest UVM, `kernel-open/nvidia-uvm/uvm_gpu.c:3129-3163`):
+  `uvm_gpu_retain_by_uuid` takes `g_uvm_global.global_lock`;
+  `uvm_gpu_release_locked` (last-release path) holds the same lock while
+  calling `nvUvmInterfaceUnregisterGpu` — an RM call to the host. If that
+  exchange never completes host-side, the lock is held forever and every
+  later client wedges at registration.
+* Host view (vmkeep10): run A full RM traffic (channel `0xcaf000d4`
+  registered), then idle; run B issues **zero** RM/sema traffic; each client
+  run re-runs the GSP boot handshake (`GSP_INIT_DONE` per run start), so the
+  host must fully re-arm per session. Around run A's exit:
+  `sema-svc: completion hint rc=-22` twice on the fresh channel.
+* Host counters freeze for the second run (`counter pub_calls=2790
+  gr_rel=216`, `drain held=2`, `faked=0`).
+* Evidence pack: `backups/evidence-20261002-victory/analysis-2026-10-08-repeat-stall/`
+  (`run1/2/3.log`, `dmesg-w-vmkeep10.log`, semfix/pin logs).
+* An m146/m147 experiment (re-arm per-rank one-shot latches on channel
+  registration; derive CE rank from ring VA; re-arm on GSP session reset via
+  `gsp_sema_service_session_reset()` from `gsp_rm_emu_reset`) was implemented
+  and **reverted** — it did not fix run B and temporarily stopped first-run
+  greens from reproducing. WIP diff:
+  `…/analysis-2026-10-08-repeat-stall/m146-m147-session-reset-WIP.patch`.
+
+## Root cause
+
+Established for the sequential case: per-client state in the host sema/GR
+service is not reset when a new guest client context appears (the guest
+re-runs the GSP boot handshake per client). The old client's channel/rank/
+ring/pool state makes the host's completion path go quiet for the new client
+(`faked=0`, `drain held=2`, zero new applies), so the new client's UVM
+registration RM exchange never completes — and because the guest UVM holds
+`global_lock` across that exchange (`uvm_gpu_release_locked` →
+`nvUvmInterfaceUnregisterGpu`), the whole guest UVM wedges for every later
+client too. Concurrent/simultaneous cases are expected to share this class
+(per-channel one-shot latches, single per-vgpu sema context).
+
+## Fix / fix plan
+
+1. Reset/rebind host per-client/per-channel sema+GR state when a new client
+   context appears (new GSP session / new channel registration): re-arm
+   one-shot rank latches, clear stale ring/userd publications, bound any
+   drain hold so a miss degrades instead of freezing (the `drain held` path).
+2. Per-vgpu sema context + P0 isolation work: ordered hunks in
+   `backups/evidence-20261002-victory/analysis-2026-10-07/sa-p2-patchnotes.md`
+   (P0-1..P0-7, stages A1–A6, gates G1/G3, `users/idle` anti-UAF drain).
+3. Graduation criteria: sequential 3+ clients in one boot green; two
+   concurrent processes (different streams) green; one process with N
+   simultaneous streams/kernels green; then the P2 suite in a single boot.
+
+## Validation plan
+
+* Sequential: 3 consecutive `victory_mail` runs in one VM, all green.
+* Concurrent: two `victory_mail` instances launched together, both green.
+* Simultaneous: multi-stream kernel test (P2 suite addition) green,
+  oracle-matched against bare metal.
+* Rollback: patch reverts independently (see WIP diff for the reverted
+  experiment).
+
+## History
+
+* 2026-10-08 00:05: first-client greens became deterministic (`6eee9ac3`);
+  second-client wedge discovered the same night (vmkeep9–14).
+* 2026-10-08 01:00: root-cause chain assembled; m146/m147 experiment
+  reverted as ineffective; kernel-native per-client reset identified as the
+  durable path.