1 files changed, 128 insertions, 0 deletions
diff --git a/debug/issue/issue-number-1.md b/debug/issue/issue-number-1.md
new file mode 100644
index 0000000..f23721f
--- /dev/null
+++ b/debug/issue/issue-number-1.md
@@ -0,0 +1,128 @@
+# ISSUE-0001: Guest CUDA is not reproducibly green — the client stalls at DEVICE (GR context init never submitted)
+
+* Status: FIX IMPLEMENTED (VALIDATION PENDING)
+* Filed: 2026-10-08, session `smiley-m1`/deep-debug | Priority: **top (blocks 1.0)** | Component: kernel (`vgsp.ko` sema service + RM emulation) with a userspace-watcher race as the variance source
+
+## Symptom
+
+`victory_mail` (and every other stock-CUDA client) run in the guest does one
+of two things on a cold boot:
+
+* **Green**: prints `DEVICE …`, then `MAIL …`, `VADD n=32768 bad=0`, `STATUS OK`
+  within ~15–190 ms of engine work; host forwards CE and GR pushes
+  (`work_gr req>0`, `GUEST GR LAUNCH OK`).
+* **Stalled**: prints `DEVICE …` and then nothing. The guest RM spins in
+  `NV_ESC_RM_CONTROL` (`0x2a`, `NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS`
+  `0x20801702`) at ~1 Hz forever. The GR rings are **never written by the
+  guest**; the host sees `work_gr req=0`; FB scans find only CE ring entries.
+
+The split is per-boot. Historically (m104, m141-r3, vmkeep6/7/14) roughly
+30–50 % of boots were green on a *fresh* host; after ~8 h of host uptime and
+~20 module claim/release cycles the green rate goes to ~0 % and the failure
+can degrade further (the guest agent never comes up; smiley r1–r3).
+
+**This is not a regression against any code change: m141-r3 (the accepted
+victory, commit `895ab8b8`) was itself a lucky boot of the same intermittent
+class; the same recipe/commit stalls on other boots.**
+
+## Impact
+
+* Blocks the 1.0 acceptance: "stock guest CUDA works flawlessly, every time".
+* Forces `GRFWD_RETRIES`/`GRFWD_STALL_SECS` lottery runs (minutes per boot)
+  and hides new regressions under the stall noise.
+* Blocks everything behind it: smiley/multi-kernel validation, P2 kernels,
+  performance numbers, multi-mdev, two guests.
+
+## Evidence
+
+One-line discriminators (host dmesg counters, `sema-svc:` periodic stats):
+
+| Counter | Green (m141-r3) | Stalled (p13-r4) | Meaning |
+|---|---|---|---|
+| `pushes` / `applied` | 655 / 655 | **90 / 89, then frozen** | one completion never lands |
+| `pushes_nomap` | 0 | **1** | one release page had no translation |
+| `gr_rel` / `pub_calls` | 219 / 2801 | **0 / 0** | guest never starts GR |
+| `ring_auto` | 26 | **6** (CE only) | GR rings never created/pinned |
+| `work_gr req` | 2+ | **0** | no GR submission reaches the host |
+| `injected mapping` lines | 10 / 40 | **0** | watcher variance |
+
+Other confirmed facts:
+
+* `tools/guest-diag/guest_kprobes.sh` armed all 6 kprobes (`ok`×6) and the
+  semfix watcher translated CE sem words in stalled boots (p13-r4) — so the
+  stall is **not** the earlier "kprobes never armed" class (`6eee9ac3` fix
+  was real but is not this issue).
+* The guest's own wait is a tracker completion: wedged stacks end in
+  `uvm_tracker_wait_for_entry ← uvm_push_end_and_wait ← channel_pool_add ←
+  uvm_channel_manager_create` (repeat-client instance) — i.e. the guest is
+  waiting for a CE push completion to become visible in a semaphore/waiter
+  page.
+* `svc_apply_pending()` keeps a release that cannot translate ("not
+  translatable yet: keep") and retries every pass; nothing else ever moves it.
+* Related (separate correctness bug, fixed under this issue): the
+  `PROMOTE_CTX` (0x2080012b) reply. See "Fixes".
+
+## Root cause
+
+The completion (release) of the client's last CE push in the boot-critical
+window cannot be written back because its semaphore/waiter page has no
+VA→PA translation yet. The release is deferred into `svc_pending[]` and waits
+for a translation that arrives only from the userspace watcher pipeline
+(`pin_gr_stream`/`semfix_uvmt`) or from the kernel's *partial* fallback set.
+The watcher pipeline is racy across boots (kprobe arming vs the client's
+first push, scanner vs pool allocation, kcore scans vs tracker windows), so
+on some boots every fallback misses; the guest's tracker wait never wakes;
+the client never creates its GR channels; nothing ever reaches the GR rings.
+
+Kernel fallback coverage was incomplete exactly for the pool/waiter page: the
+pool resolver (`svc_pool_pa_resolve`) had no walk of the **driver-published
+VAS page directories** (`svc_walk_vaspaces`) — the deterministic, watcher-free
+source the general resolver (`svc_lookup_trusted`) has always used.
+
+Confidence: high for the mechanism (the deferral + the missing walk + the
+observed freeze at `applied=89/90`). The exact *reason the walk itself misses
+on some boots* (directory registered later than the first push, or the
+watcher path succeeding first on greens) remains boot-timing dependent; the
+fix below removes the dependence rather than chasing the timing.
+
+## Fix
+
+1. **`kernel/gsp_sema_service.c` — `svc_pool_pa_resolve`:** add
+   `svc_walk_vaspaces()` (driver-published directories, read-back proven)
+   between the self-discovered walk and the targeted rm_mem scan. Kernel-native
+   translation of the deferred release page no longer depends on the watcher.
+2. **`kernel/gsp_sema_service.c` — `svc_pending` starvation telemetry:**
+   per-entry `tries` counter + bounded logs at 4/64/1024 attempts naming
+   `va`, `payload` and the consequence. A future starvation is visible in
+   dmesg instead of silent.
+3. **`kernel/gsp_rm_emulation.c` — PROMOTE_CTX reply (supersedes
+   `fa9296af`):** the reply must keep `bInitialize` set. Per
+   `kernel_graphics_object.c` `kgrobjPromoteContext_IMPL`:
+   `if (!params.promoteEntry[i].bInitialize) continue;
+   kgrctxMarkCtxBufferInitialized(...)` — the flag is the server's
+   "initialized" ACK; clearing it (fa9296af) made every ctx buffer re-promote
+   forever. This was a correctness bug in the stall's vicinity, not the stall
+   itself (the stall predates and outlives it).
+
+Durable direction (if a stall variant survives the fix): fold the watcher's
+job into the kernel entirely (complete `svc_pb_discover`/`svc_pd` coverage of
+the CE push/pool windows) so the watchers are accelerators only.
+
+## Validation plan
+
+1. Fresh host boot (reboot approved 2026-10-08): accepted recipe, plain
+   runner — expect `victory_mail` green.
+2. **10/10 consecutive cold boots** green within seconds; zero stall
+   captures; `pending release starved` never logs; `pushes_nomap` returns 0
+   on every green.
+3. Re-run the smiley workload (`tools/cuda-ab/smiley.cu`) per green VM.
+4. Rollback: revert commits 1–3 independently; rebuild; the previous
+   behaviour (lottery) returns without new failure modes.
+
+## History
+
+* 2026-10-02..07: stall hunted through m1xx series; greens m104, m141-r3;
+  kprobe arming fixed (`6eee9ac3`); plan to kernel-native discovery recorded.
+* 2026-10-08: campaigns p1–p13 + smiley r1–r3 all stall; evidence shows the
+  missing completion + missing pool-page translation; `fa9296af` PROMOTE_CTX
+  change tested and disproven (and is itself corrected here).