From f8edfb21e0880935fbe5aa62996fcd9af64c32ff Mon Sep 17 00:00:00 2001
From: Arthur Hanson Rasmusson <arthur@vgpu.io>
Date: Thu, 8 Oct 2026 11:07:42 -0400
Subject: debug/issue: issue register + ISSUE-0001 (reproducibility) +
 ISSUE-0002 (multiple kernels per VM)

- debug/issue/README.md: when to file (nasty bugs only; default is no
  issues), template, numbering, lifecycle incl. solved/ move.
- issue-number-1.md: guest CUDA is not reproducibly green (top priority);
  evidence discriminators, root cause (starved pool release translation),
  the three fixes landed, 10/10-boot validation plan.
- issue-number-2.md: multiple kernels in one VM (sequential/concurrent/
  simultaneous) must work; second-client wedge root-cause chain + plan.
- tools/cuda-ab/smiley.cu: 4-pass vectorAdd that renders a 64x32 ASCII
  smiley from GPU-computed distance fields; ab.sh gains a generic 'push'
  mode so any WORKLOAD binary can be delivered to the guest.
---
 debug/issue/README.md         |  74 ++++++++++++++++++++++++
 debug/issue/issue-number-1.md | 128 ++++++++++++++++++++++++++++++++++++++++++
 debug/issue/issue-number-2.md |  92 ++++++++++++++++++++++++++++++
 3 files changed, 294 insertions(+)
 create mode 100644 debug/issue/README.md
 create mode 100644 debug/issue/issue-number-1.md
 create mode 100644 debug/issue/issue-number-2.md

(limited to 'debug/issue')

diff --git a/debug/issue/README.md b/debug/issue/README.md
new file mode 100644
index 0000000..2c08afc
--- /dev/null
+++ b/debug/issue/README.md
@@ -0,0 +1,74 @@
+# Issue register — process
+
+**Policy: do NOT create issues by default.** An issue here is a *nasty bug*:
+something that blocks mission-critical behaviour, is non-deterministic or not
+reproducible on demand, corrupts state, or spans subsystems such that a
+single-session fix is unrealistic. Ordinary fixes, TODOs and feature work do
+not get an issue — they live in the handoff and the code comments.
+
+File an issue when **any** of these holds:
+
+1. Reproducibility / liveness is variable (works sometimes, stalls others)
+   and the variance itself is the bug.
+2. The failure blocks a 1.0 acceptance criterion (guest CUDA, multi-VM,
+   performance target, ...).
+3. Cross-subsystem state (host module + guest driver + physical GPU +
+   userspace tools) is involved and the ownership of the fix is unclear.
+4. The investigation has already burned more than one session without a
+   confirmed root cause.
+
+## How to file
+
+* Directory: `open-iov/debug/issue/`; solved ones move to
+  `open-iov/debug/issue/solved/` (never delete, never renumber).
+* Filename: `issue-number-N.md` with the next free integer `N`
+  (`issue-number-1.md`, `issue-number-2.md`, ...). Create
+  `issue-number-N+1.md` as a copy of the template below — do not create
+  placeholder issues.
+* Keep it **public-safe** (this directory is tracked): describe components,
+  behaviour and evidence (same rules as every tracked document).
+* Reference evidence by artifact: session/log file names, counter lines,
+  commit hashes. Every claim needs its evidence pointer.
+
+## Template
+
+```markdown
+# ISSUE-00NN: <one-line title>
+* Status: OPEN | FIX IMPLEMENTED (VALIDATION PENDING) | SOLVED
+* Filed: <date>, session <label> | Priority: <top/p1/p2> | Component: <kernel/guest/...>
+
+## Symptom
+What is observed, by whom, how often. Include the exact observable signature.
+
+## Impact
+Which acceptance criterion is blocked; what work is blocked behind it.
+
+## Evidence
+Logs/counters/commits that establish the facts. Include the one-line
+discriminators between healthy and failing runs.
+
+## Root cause
+The causal chain established so far, with confidence and what is still
+hypothesis (label uncertainty explicitly).
+
+## Fix / fix plan
+Code changes with commit hashes; for open items, the ordered plan and the
+files/functions to touch.
+
+## Validation plan
+Acceptance criteria (numbers of consecutive runs, counters that must stay
+zero, suites to run) and how the fix is rolled back if it regresses.
+
+## History
+Dated one-liners: what was tried, what it taught, what was rejected.
+```
+
+## Lifecycle
+
+* OPEN: filed, root cause partial.
+* FIX IMPLEMENTED (VALIDATION PENDING): code landed (committed) but the
+  acceptance runs have not completed on a live stack.
+* SOLVED: acceptance criteria met with evidence; move the file to `solved/`
+  and add a `## Resolution` section (what fixed it, the validating runs,
+  residual risks). If the same failure returns, file a **new** number and
+  cross-reference — do not reopen the moved file.
diff --git a/debug/issue/issue-number-1.md b/debug/issue/issue-number-1.md
new file mode 100644
index 0000000..f23721f
--- /dev/null
+++ b/debug/issue/issue-number-1.md
@@ -0,0 +1,128 @@
+# ISSUE-0001: Guest CUDA is not reproducibly green — the client stalls at DEVICE (GR context init never submitted)
+
+* Status: FIX IMPLEMENTED (VALIDATION PENDING)
+* Filed: 2026-10-08, session `smiley-m1`/deep-debug | Priority: **top (blocks 1.0)** | Component: kernel (`vgsp.ko` sema service + RM emulation) with a userspace-watcher race as the variance source
+
+## Symptom
+
+`victory_mail` (and every other stock-CUDA client) run in the guest does one
+of two things on a cold boot:
+
+* **Green**: prints `DEVICE …`, then `MAIL …`, `VADD n=32768 bad=0`, `STATUS OK`
+  within ~15–190 ms of engine work; host forwards CE and GR pushes
+  (`work_gr req>0`, `GUEST GR LAUNCH OK`).
+* **Stalled**: prints `DEVICE …` and then nothing. The guest RM spins in
+  `NV_ESC_RM_CONTROL` (`0x2a`, `NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS`
+  `0x20801702`) at ~1 Hz forever. The GR rings are **never written by the
+  guest**; the host sees `work_gr req=0`; FB scans find only CE ring entries.
+
+The split is per-boot. Historically (m104, m141-r3, vmkeep6/7/14) roughly
+30–50 % of boots were green on a *fresh* host; after ~8 h of host uptime and
+~20 module claim/release cycles the green rate goes to ~0 % and the failure
+can degrade further (the guest agent never comes up; smiley r1–r3).
+
+**This is not a regression against any code change: m141-r3 (the accepted
+victory, commit `895ab8b8`) was itself a lucky boot of the same intermittent
+class; the same recipe/commit stalls on other boots.**
+
+## Impact
+
+* Blocks the 1.0 acceptance: "stock guest CUDA works flawlessly, every time".
+* Forces `GRFWD_RETRIES`/`GRFWD_STALL_SECS` lottery runs (minutes per boot)
+  and hides new regressions under the stall noise.
+* Blocks everything behind it: smiley/multi-kernel validation, P2 kernels,
+  performance numbers, multi-mdev, two guests.
+
+## Evidence
+
+One-line discriminators (host dmesg counters, `sema-svc:` periodic stats):
+
+| Counter | Green (m141-r3) | Stalled (p13-r4) | Meaning |
+|---|---|---|---|
+| `pushes` / `applied` | 655 / 655 | **90 / 89, then frozen** | one completion never lands |
+| `pushes_nomap` | 0 | **1** | one release page had no translation |
+| `gr_rel` / `pub_calls` | 219 / 2801 | **0 / 0** | guest never starts GR |
+| `ring_auto` | 26 | **6** (CE only) | GR rings never created/pinned |
+| `work_gr req` | 2+ | **0** | no GR submission reaches the host |
+| `injected mapping` lines | 10 / 40 | **0** | watcher variance |
+
+Other confirmed facts:
+
+* `tools/guest-diag/guest_kprobes.sh` armed all 6 kprobes (`ok`×6) and the
+  semfix watcher translated CE sem words in stalled boots (p13-r4) — so the
+  stall is **not** the earlier "kprobes never armed" class (`6eee9ac3` fix
+  was real but is not this issue).
+* The guest's own wait is a tracker completion: wedged stacks end in
+  `uvm_tracker_wait_for_entry ← uvm_push_end_and_wait ← channel_pool_add ←
+  uvm_channel_manager_create` (repeat-client instance) — i.e. the guest is
+  waiting for a CE push completion to become visible in a semaphore/waiter
+  page.
+* `svc_apply_pending()` keeps a release that cannot translate ("not
+  translatable yet: keep") and retries every pass; nothing else ever moves it.
+* Related (separate correctness bug, fixed under this issue): the
+  `PROMOTE_CTX` (0x2080012b) reply. See "Fixes".
+
+## Root cause
+
+The completion (release) of the client's last CE push in the boot-critical
+window cannot be written back because its semaphore/waiter page has no
+VA→PA translation yet. The release is deferred into `svc_pending[]` and waits
+for a translation that arrives only from the userspace watcher pipeline
+(`pin_gr_stream`/`semfix_uvmt`) or from the kernel's *partial* fallback set.
+The watcher pipeline is racy across boots (kprobe arming vs the client's
+first push, scanner vs pool allocation, kcore scans vs tracker windows), so
+on some boots every fallback misses; the guest's tracker wait never wakes;
+the client never creates its GR channels; nothing ever reaches the GR rings.
+
+Kernel fallback coverage was incomplete exactly for the pool/waiter page: the
+pool resolver (`svc_pool_pa_resolve`) had no walk of the **driver-published
+VAS page directories** (`svc_walk_vaspaces`) — the deterministic, watcher-free
+source the general resolver (`svc_lookup_trusted`) has always used.
+
+Confidence: high for the mechanism (the deferral + the missing walk + the
+observed freeze at `applied=89/90`). The exact *reason the walk itself misses
+on some boots* (directory registered later than the first push, or the
+watcher path succeeding first on greens) remains boot-timing dependent; the
+fix below removes the dependence rather than chasing the timing.
+
+## Fix
+
+1. **`kernel/gsp_sema_service.c` — `svc_pool_pa_resolve`:** add
+   `svc_walk_vaspaces()` (driver-published directories, read-back proven)
+   between the self-discovered walk and the targeted rm_mem scan. Kernel-native
+   translation of the deferred release page no longer depends on the watcher.
+2. **`kernel/gsp_sema_service.c` — `svc_pending` starvation telemetry:**
+   per-entry `tries` counter + bounded logs at 4/64/1024 attempts naming
+   `va`, `payload` and the consequence. A future starvation is visible in
+   dmesg instead of silent.
+3. **`kernel/gsp_rm_emulation.c` — PROMOTE_CTX reply (supersedes
+   `fa9296af`):** the reply must keep `bInitialize` set. Per
+   `kernel_graphics_object.c` `kgrobjPromoteContext_IMPL`:
+   `if (!params.promoteEntry[i].bInitialize) continue;
+   kgrctxMarkCtxBufferInitialized(...)` — the flag is the server's
+   "initialized" ACK; clearing it (fa9296af) made every ctx buffer re-promote
+   forever. This was a correctness bug in the stall's vicinity, not the stall
+   itself (the stall predates and outlives it).
+
+Durable direction (if a stall variant survives the fix): fold the watcher's
+job into the kernel entirely (complete `svc_pb_discover`/`svc_pd` coverage of
+the CE push/pool windows) so the watchers are accelerators only.
+
+## Validation plan
+
+1. Fresh host boot (reboot approved 2026-10-08): accepted recipe, plain
+   runner — expect `victory_mail` green.
+2. **10/10 consecutive cold boots** green within seconds; zero stall
+   captures; `pending release starved` never logs; `pushes_nomap` returns 0
+   on every green.
+3. Re-run the smiley workload (`tools/cuda-ab/smiley.cu`) per green VM.
+4. Rollback: revert commits 1–3 independently; rebuild; the previous
+   behaviour (lottery) returns without new failure modes.
+
+## History
+
+* 2026-10-02..07: stall hunted through m1xx series; greens m104, m141-r3;
+  kprobe arming fixed (`6eee9ac3`); plan to kernel-native discovery recorded.
+* 2026-10-08: campaigns p1–p13 + smiley r1–r3 all stall; evidence shows the
+  missing completion + missing pool-page translation; `fa9296af` PROMOTE_CTX
+  change tested and disproven (and is itself corrected here).
diff --git a/debug/issue/issue-number-2.md b/debug/issue/issue-number-2.md
new file mode 100644
index 0000000..e04e08f
--- /dev/null
+++ b/debug/issue/issue-number-2.md
@@ -0,0 +1,92 @@
+# ISSUE-0002: Multiple CUDA kernels/workloads inside one VM — sequential, concurrent and simultaneous must all work
+
+* Status: OPEN (root cause established for the sequential case; concurrent/simultaneous untested)
+* Filed: 2026-10-08 | Priority: **p1 (blocks 1.0 "multiple kernels", follows ISSUE-0001)** | Component: kernel per-client/session state + guest UVM/RM interaction
+
+## Symptom
+
+* **Sequential (known):** a *second* CUDA process in the same boot wedges:
+  spins at ~100 % CPU with no output (not even `DEVICE`), host traffic for it
+  is zero. The first client in the same boot is green (10–34 s). Until fixed,
+  operations are "one client per boot, reboot for a rerun".
+* **Concurrent:** not yet tested (blocked by the sequential wedge).
+* **Simultaneous (one process, multiple streams/kernels at once):** not yet
+  tested; expected to hit the same per-channel/per-client state class.
+
+## Impact
+
+* Blocks 1.0 acceptance "arbitrary CUDA kernels flawlessly" beyond a single
+  sequential run, all realistic workloads (frameworks spawn many processes),
+  and the P2 kernel suite (`tools/cuda-ab/p2/run_guest.sh`) which runs
+  multiple programs per boot by design.
+
+## Evidence
+
+* Wedged guest stack (vmkeep10 run C, pid 1447):
+  `uvm_ioctl → uvm_api_register_gpu → uvm_va_space_register_gpu →
+  uvm_gpu_retain_by_uuid+0x44 [nvidia_uvm]` (main thread), second thread in
+  `do_poll`.
+* Source (guest UVM, `kernel-open/nvidia-uvm/uvm_gpu.c:3129-3163`):
+  `uvm_gpu_retain_by_uuid` takes `g_uvm_global.global_lock`;
+  `uvm_gpu_release_locked` (last-release path) holds the same lock while
+  calling `nvUvmInterfaceUnregisterGpu` — an RM call to the host. If that
+  exchange never completes host-side, the lock is held forever and every
+  later client wedges at registration.
+* Host view (vmkeep10): run A full RM traffic (channel `0xcaf000d4`
+  registered), then idle; run B issues **zero** RM/sema traffic; each client
+  run re-runs the GSP boot handshake (`GSP_INIT_DONE` per run start), so the
+  host must fully re-arm per session. Around run A's exit:
+  `sema-svc: completion hint rc=-22` twice on the fresh channel.
+* Host counters freeze for the second run (`counter pub_calls=2790
+  gr_rel=216`, `drain held=2`, `faked=0`).
+* Evidence pack: `backups/evidence-20261002-victory/analysis-2026-10-08-repeat-stall/`
+  (`run1/2/3.log`, `dmesg-w-vmkeep10.log`, semfix/pin logs).
+* An m146/m147 experiment (re-arm per-rank one-shot latches on channel
+  registration; derive CE rank from ring VA; re-arm on GSP session reset via
+  `gsp_sema_service_session_reset()` from `gsp_rm_emu_reset`) was implemented
+  and **reverted** — it did not fix run B and temporarily stopped first-run
+  greens from reproducing. WIP diff:
+  `…/analysis-2026-10-08-repeat-stall/m146-m147-session-reset-WIP.patch`.
+
+## Root cause
+
+Established for the sequential case: per-client state in the host sema/GR
+service is not reset when a new guest client context appears (the guest
+re-runs the GSP boot handshake per client). The old client's channel/rank/
+ring/pool state makes the host's completion path go quiet for the new client
+(`faked=0`, `drain held=2`, zero new applies), so the new client's UVM
+registration RM exchange never completes — and because the guest UVM holds
+`global_lock` across that exchange (`uvm_gpu_release_locked` →
+`nvUvmInterfaceUnregisterGpu`), the whole guest UVM wedges for every later
+client too. Concurrent/simultaneous cases are expected to share this class
+(per-channel one-shot latches, single per-vgpu sema context).
+
+## Fix / fix plan
+
+1. Reset/rebind host per-client/per-channel sema+GR state when a new client
+   context appears (new GSP session / new channel registration): re-arm
+   one-shot rank latches, clear stale ring/userd publications, bound any
+   drain hold so a miss degrades instead of freezing (the `drain held` path).
+2. Per-vgpu sema context + P0 isolation work: ordered hunks in
+   `backups/evidence-20261002-victory/analysis-2026-10-07/sa-p2-patchnotes.md`
+   (P0-1..P0-7, stages A1–A6, gates G1/G3, `users/idle` anti-UAF drain).
+3. Graduation criteria: sequential 3+ clients in one boot green; two
+   concurrent processes (different streams) green; one process with N
+   simultaneous streams/kernels green; then the P2 suite in a single boot.
+
+## Validation plan
+
+* Sequential: 3 consecutive `victory_mail` runs in one VM, all green.
+* Concurrent: two `victory_mail` instances launched together, both green.
+* Simultaneous: multi-stream kernel test (P2 suite addition) green,
+  oracle-matched against bare metal.
+* Rollback: patch reverts independently (see WIP diff for the reverted
+  experiment).
+
+## History
+
+* 2026-10-08 00:05: first-client greens became deterministic (`6eee9ac3`);
+  second-client wedge discovered the same night (vmkeep9–14).
+* 2026-10-08 01:00: root-cause chain assembled; m146/m147 experiment
+  reverted as ineffective; kernel-native per-client reset identified as the
+  durable path.
-- 
cgit v1.2.3