diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 65 |
1 files changed, 60 insertions, 5 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 5322e98..a335ac9 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,9 @@ # Open-IOV Current State -> **Snapshot: 2026-10-16 ~15:30 (integration wave: bigpath+dual merged, five fixes -> landed through `7d123630`; live r161/r162 cycles on kernel 6.8.0-147).** +> **Snapshot: 2026-10-20 ~05:30 (boot lottery root-caused and fixed - the +> status-queue SWAP_RX consumption cursor; guest CUDA kernels green end-to-end +> on fresh boots: `victory_mail` / `victory_matmul` / transformer; demo VM live +> on kernel 6.8.0-147).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -302,9 +304,38 @@ isolation, performance, portability). `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the demand-pin content resolver searches a learned per-allocation offset. Remaining characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch - class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out - mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, - retest pending). + class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out + mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, + retest pending). + **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):** + the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was + reproduced and root-caused: under the driver's status-queue SWAP_RX mode the + emulation's status free-space check read the status queue's mirrored RX header + (which the emulation publishes itself) instead of the peer consumption cursor in + the command queue's RX header. With the guest mid-drain the status ring was + declared full, replies were written anyway, unread records were overwritten, and + the guest RM read a record one full ring lap ahead + (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed + and the first client's `cudaMalloc` returned "device busy or unavailable" - + poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the + cursor is read from the command queue's RX header per the driver's swap-mode + semantics; per-client channel unregister and session flush work again (one + unregister in the poisoned boot; 79 with one flush per client after). Verified on + fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul` + (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`, + `FFN4 layers=4 bad=0` + `STATUS OK`) all green. + Remaining large-buffer class (current frontier): a `vector_add` cold-run content + gap (a staging read/write race between the emulated CE copy and the guest's + staging ping-pong: ~0.68 MB of the third MB reads the previous staging content), + `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop` + hang class correlated with a boot where the injected-pin stream never re-targeted + the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class + (RM object fields overwritten with client data; stale system-memory destination + resolution on freed staging pages) is characterized but open. Next: pin-stream + re-targeting, then the staging-race and stale-destination fixes; then the + boot-lottery re-measure over fresh boots, the full battery + soak, two guests, + multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live + with the three green kernels. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -628,6 +659,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM | | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | @@ -664,6 +696,29 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX + cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20 + ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`, + `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM + logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed + (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable") + onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr + lives in the command queue's RX header (the driver's swap-mode semantics), + while the status queue's own RX header is the mirror the emulation publishes + (its own command-queue read position); the status free-space check read the + mirror, so with the guest mid-drain the ring was declared full and replies + were written over unread records. Fix (`c72f8948`): read the peer + consumption cursor from the command queue's RX header. Verified on fresh + boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256 + bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`) + all green with `STATUS OK`; per-client channel unregister + session flush + restored (1 unregister in the poisoned boot vs 79 after). Remaining + frontier: the large-buffer class (`vector_add` cold-run staging race, + `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with + an un-targeted injected-pin stream), then boot-lottery re-measure, full + battery + soak, two guests, multi-mdev, VRAM sizes. Evidence: + `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in + tmux session 0 (SSH to the guest, three green kernels). - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns |