STATE-CURRENT: 2026-10-20 snapshot - boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end (mail/matmul/transformer); large-buffer class frontier

1 files changed, 60 insertions, 5 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 5322e98..a335ac9 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,7 +1,9 @@
 # Open-IOV Current State
 
-> **Snapshot: 2026-10-16 ~15:30 (integration wave: bigpath+dual merged, five fixes
-> landed through `7d123630`; live r161/r162 cycles on kernel 6.8.0-147).**
+> **Snapshot: 2026-10-20 ~05:30 (boot lottery root-caused and fixed - the
+> status-queue SWAP_RX consumption cursor; guest CUDA kernels green end-to-end
+> on fresh boots: `victory_mail` / `victory_matmul` / transformer; demo VM live
+> on kernel 6.8.0-147).**
 > This is a dated public
 > status page, not live state; it is refreshed on every progress check. Live
 > working notes are maintained outside this repository; the public
@@ -302,9 +304,38 @@ isolation, performance, portability).
        `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the
        demand-pin content resolver searches a learned per-allocation offset. Remaining
        characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch
-       class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out
-       mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed,
-       retest pending).
+        class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out
+        mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed,
+        retest pending).
+       **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):**
+        the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was
+        reproduced and root-caused: under the driver's status-queue SWAP_RX mode the
+        emulation's status free-space check read the status queue's mirrored RX header
+        (which the emulation publishes itself) instead of the peer consumption cursor in
+        the command queue's RX header. With the guest mid-drain the status ring was
+        declared full, replies were written anyway, unread records were overwritten, and
+        the guest RM read a record one full ring lap ahead
+        (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed
+        and the first client's `cudaMalloc` returned "device busy or unavailable" -
+        poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the
+        cursor is read from the command queue's RX header per the driver's swap-mode
+        semantics; per-client channel unregister and session flush work again (one
+        unregister in the poisoned boot; 79 with one flush per client after). Verified on
+        fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul`
+        (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`,
+        `FFN4 layers=4 bad=0` + `STATUS OK`) all green.
+        Remaining large-buffer class (current frontier): a `vector_add` cold-run content
+        gap (a staging read/write race between the emulated CE copy and the guest's
+        staging ping-pong: ~0.68 MB of the third MB reads the previous staging content),
+        `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop`
+        hang class correlated with a boot where the injected-pin stream never re-targeted
+        the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class
+        (RM object fields overwritten with client data; stale system-memory destination
+        resolution on freed staging pages) is characterized but open. Next: pin-stream
+        re-targeting, then the staging-race and stale-destination fixes; then the
+        boot-lottery re-measure over fresh boots, the full battery + soak, two guests,
+        multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live
+        with the three green kernels.
 - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING
   (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only
   ("no-claim") campaign runs an unmodified guest CUDA program to completion
@@ -628,6 +659,7 @@ chain succeeds.
 
 | ID | Milestone | When | State |
 |---|---|---|---|
+| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM |
 | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident |
 | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) |
 | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) |
@@ -664,6 +696,29 @@ chain succeeds.
 
 ## Milestone log - current frontier and recent milestones (granular IDs in each entry)
 
+- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX
+  cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20
+  ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`,
+  `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM
+  logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed
+  (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable")
+  onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr
+  lives in the command queue's RX header (the driver's swap-mode semantics),
+  while the status queue's own RX header is the mirror the emulation publishes
+  (its own command-queue read position); the status free-space check read the
+  mirror, so with the guest mid-drain the ring was declared full and replies
+  were written over unread records. Fix (`c72f8948`): read the peer
+  consumption cursor from the command queue's RX header. Verified on fresh
+  boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256
+  bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`)
+  all green with `STATUS OK`; per-client channel unregister + session flush
+  restored (1 unregister in the poisoned boot vs 79 after). Remaining
+  frontier: the large-buffer class (`vector_add` cold-run staging race,
+  `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with
+  an un-targeted injected-pin stream), then boot-lottery re-measure, full
+  battery + soak, two guests, multi-mdev, VRAM sizes. Evidence:
+  `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in
+  tmux session 0 (SSH to the guest, three green kernels).
 - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI);
   golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23
   hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns