diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 255 |
1 files changed, 251 insertions, 4 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index c16b774..27295df 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,8 +1,13 @@ # Open-IOV Current State -> **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-07 ~01:15; debugging-options -> catalogue added 2026-10-05).** This is a dated public +> **Snapshot: 2026-10-20 ~20:00 (two more fixes landed: the SWAP_RX cursor +> and the pin-wins ledger divergence that cured a guest-kernel-corruption +> panic; guest CUDA kernels green on fresh boots - `victory_mail`, +> `victory_matmul` (first-run flake), transformer, `vector_add` (1M, +> `bad=0`), `vecadd_loop` (100 launches); two guests with two emulated VRAM +> sizes (512M + 1GiB) coexist live; frontier = the multi-pass content class +> and the streams2 launch-order wait).** +> This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: @@ -146,6 +151,218 @@ isolation, performance, portability). aperture. Ranked next: read the server-FBMEM fault buffer via a copy-engine transfer, per-VEID/compute-context execution requirements, and the scheduler CWD field semantics. Host clean, GPU green, no reboots. + **Update 2026-10-14 ~13:30 (major - guest compute runs on the physical + engine):** the physical-backed guest campaign now runs a multi-client + CUDA battery against the real TU104 with the host CUDA bridge disabled + (`cuda_bridge=0`, GR forwarding on). In one cycle (r137) the first client + (`victory_mail`: `MAIL` + `VADD bad=0` + `STATUS OK`) and the second + client (`smiley`: `STATUS OK`) are both green on the physical backend, + and the guest's compute launches execute on the physical GR engine (the + window-attribution defects where kernels wrote into the wrong window are + fixed; first-client guest VADD results verify). The 2026-10-08..12 + second-client campaign closed its chain (CE USERD location, CE + push-buffer pin delivery, a channel-anchored CE pool resolver + `ade77d9a`, session-flush pin semantics, data-argument resolution + order). Wall closure since 2026-10-07: (a) the third-client + (`victory_matmul`) PMA-scrub freeze is root-caused live - the + boot-session scrubber channel's finish word is a guest-RAM vmap that + collides with a client VA - and fixed by authoring the scrubber + channel's progress (`624438bb` + guest scanner `80af379d`), so + `victory_matmul` gets past `cudaMalloc`; (b) the intermittent + GR-forward launch timeout is root-caused: the host publishes forwarded + launches modulo an 8-entry ladder ring while the user-lane channel was + declared with 1024 GPFIFO entries, so after the first host-ring wrap the + front-end's GET/PUT became inconsistent and it stopped fetching + (`074f493c` fixes the lane depth to 8; the earlier two-client acceptance + never wrapped and had masked the regression). The r139 cycle validated + the ring fix: the 100-launch soak crossed the host-ring wrap cleanly + (`LOOP launches=100 bad=0 STATUS OK`; the previous cycle died at launch + 8), c1 `victory_mail` is green (60 s), and `smiley` green on a later + retry. The residual is an intermittent cold-context/fetched-not-executed + launch race (18 failed launches in the cycle: 6 never fetched by the + front-end, 12 fetched but not completed by the engine; some later + clients intermittently return all-zero results), for which a bounded + in-submit retry with a fresh ring entry plus a cold pre-warm is being + implemented; the "OK with 0 changed" change-counter is not a reliable + execution witness - the client's own `STATUS` is the acceptance signal. + Host clean all session, no Xid in the campaign logs, pstore empty, no + reboots (in-session PCI function-level reset recovery where needed). + ISSUE-0002 (second client in one boot): the sequential case can pass + (first two clients green in one cycle; smiley green on retry in the + next) but is not yet deterministic; the issue remains open pending + repeat-run plus concurrent/simultaneous validation. Active front: + the multi-client battery on the physical backend plus launch-latency + robustness. + **Update 2026-10-14 ~15:40 (afternoon wave, r140-r143):** the full fix + set is committed and pushed (HEAD `17393a74`: ring depth `074f493c`, + scrubber `624438bb`/`80af379d`, bounded fresh-entry launch retry + `6b090f70`, 208-page budgeted data window `fce5ab8d`/`4a23c196`). r142: + c1 `victory_mail` and c2 `smiley` GREEN; c3 `victory_matmul` FAIL + (all-wrong operands - the three 256 KB operands resolved only ~32 of the + needed 64+ pages each); launch counters `ok=86 failed=10 retries=4` + (the retry recovers a subset of the cold-context timeouts silently); + then `victory_saxpy`/`memcpy_roundtrip` stalled and the guest SSH wedged + - the intermittent stall/wedge class survives. r143 (live at this + check, same fix set): c1 green within ~2 min; c3 matmul launched and + improved to `bad=8192` of 65536 with a non-zero checksum (operand + resolution now 68/64/64 pages) - STATUS FAIL. **Root cause found + (sub-agent, evidence-grounded): the win0 QMD shadow pool base was still + `0x190000` (main-PT slots 0x190..0x197), which lies INSIDE the enlarged + 208-page bounce window (`0x110000..0x1df000`); the win0 plant runs on + every `pdb_build()` while the bounce plant is one-shot, so win0 stole the + PTEs for bounce pages 128..135 = C rows 0..31 = exactly `bad=8192`. Fix + landed (unvalidated live at pause): `GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE` + moved to `0x1e0000` (slots 0x1e0..0x1e7, above the mail/marker pages, + below the 0x200000 guest window) plus a `static_assert` invariant and + comment/log corrections.** Ranked next blockers: (1) the 256 KB operand + data-path placement; (2) the intermittent client stall/wedge class + (the top reliability blocker; needs the stalled-client capture); + (3) the cold-context launch race (cold GR-sem pre-warm designed); + (4) then multi-mdev, two guests, VRAM sizes and the concurrent/full + soak. Host clean, GPU green, no reboots. + **Update 2026-10-15 ~18:15 (resume + parallel workstreams; pause snapshot):** + the host rebooted (the OS moved the kernel to 6.8.0-147; the module was rebuilt + for it). The r155 cycle on the physical backend: ladder green; `victory_mail` + (`VADD bad=0` + `STATUS OK`) and `victory_matmul` (`MATMUL n=256 bad=0` + + `STATUS OK`) both green end-to-end, and a 100-launch `vecadd_loop` passed; the + sequential battery then hit the known acked-no-write class on `streams2` + (all-zero output despite `qmd_rel=0x6` completion acks -> `STATUS FAIL`). Three + offline workstreams landed as pushed branches: (a) `bigpath` - a 64 MiB + boot-fixed data-bounce region (stage-3-planted PTEs) with fail-closed cap + semantics, unblocking large operands (victory_math 512 KB, vector_add 4 MB, + memcpy_roundtrip 8 MB); (b) `dual` - per-mdev separation (per-guest channel + keys, session records, guest-RAM records, forward-request ownership) for two + simultaneous guests; (c) `lottery` - analysis proving the completion ack is not + an execution witness, rejecting naive fail-closed (idempotent-client false + positives), and ranking the next experiments (win0 bank rotation, qmdwin + double-buffer, warm-sem retry, executed-vs-wedged discriminator). Host clean, + GPU green, VM left live; user-requested poweroff at pause. + **Update 2026-10-16 ~15:30 (integration wave; five fixes landed; no reboots):** + bigpath + dual merged to master; then, pushed through `7d123630`: qmdwin + double-buffering (E3); a warm-sem-push discriminator on zero-write retries (E5 - + proved the engine is ALIVE during the acked-no-execute class, so it is a + launch-form/dispatch race, not an engine wedge); R1/m157-m159 - busy-refused GR + captures now wait (bounded) and retry inline, so multi-launch clients' kernels all + execute (streams2's three kernels ran serially on the physical engine); and a + guest-side demand-pin offset fix (CE-window mask `&0xFFFFF` folded every 1 MB rank + to page 0) that removed the repeat-client stall class. Verified live (r161/r162, + 512M mdev): the exact user repro (`victory_mail` then `victory_matmul`) green + back-to-back with `bad=0` + `STATUS OK`, and a four-consecutive-client green boot + (`victory_mail` x2, `victory_matmul`, `smiley`). Remaining single-guest gaps: the + intermittent acked-no-execute launch lottery (one `smiley` pass in r162) and a + streams2 second-channel capture miss (entry consumed before its push page was + readable); both characterized with fix directions in the private handoff. + **1.0 multi-tenant:** two mdevs of different VRAM sizes (512M + 2 GiB) and two + QEMU guests coexist; guest B reports `memory.total = 2048 MiB` with `nvidia-smi` + green. Guest-B CUDA is blocked by the shared host pin table (cross-guest + contamination demonstrated live) and needs the per-guest ownership rework + (owner-tagged pin/map/pending lookups) before any dual-CUDA claim. No reboot; + host GPU green; both guests left running. + **Update 2026-10-16 ~22:35, extended ~23:05 (reliability wave 2, m163-m170; cycles r169-r177, no host reboots):** + the stall classes that made clients freeze at DEVICE are largely closed. Fixed and + verified across boots r172/r174/r175/r177: the session flush can no longer wipe a live + session (a deferred flush executed with the client's channels valid, dropped its + queued launch release, and froze it in the tracker wait); GR entries whose decode + sees QMD words but builds no launch candidate are held and re-captured instead of + consumed silently; a failed device-destination copy holds its entry instead of + silently dropping the data; pool releases apply at fresh-pin time instead of + registration+5 s; the timeout ladder fails fast when the engine is wedged; argument + spans whose first page resolves nothing hold the launch (bounded) instead of + forwarding stale operands; device-destination copies answered only by a page-table + walk wait (bounded 3 s) for a pin. `victory_mail`, `victory_matmul` and `smiley` + are green on every boot of the wave, including the user's exact repro and mail as + the eighth client of a used boot; r177 ran mail, matmul, smiley, matmul (repeat) + and mail GREEN back-to-back as its first five clients. streams2/memset2d/saxpy no + longer wedge - they + now fail fast (~35-60 s) on the remaining data class: an H2D into a device buffer + that does not land where the launch's data fill later reads the operand (the + vidmem write path vs the fill's read path divergence; analysis and the ranked + ledger fix are in the private handoff, cont. 23). + **Update 2026-10-17 ~01:00 (stall pinpointed on r179; diagnostics only, no tree change yet):** + a minimal repro (`cudaEventCreate` + `cudaEventRecord` on the default stream + + `cudaEventSynchronize`) sits forever after the record - the first CUDA call pinned down + as a hard hang. A 3-stream workload using event record/wait/elapsed time stalls even as + the very first client of a fresh boot (intrinsic, not position- or poison-dependent; + unclean client kills were tested and do not break the next client). On a used boot the + user's `victory_mail`/`victory_matmul` hit the same wait: the client spins in driver + userspace while a background worker polls the interrupt-service RM control once per + second. The host's event-registration table and post path exist (register on NV01_EVENT + alloc, retire on free, per-channel/all post helpers) and the stall windows still show + frequent post failures with EINVAL; the host sema side is verified healthy at the stall + (client pushes decode, pool releases apply to the correct page, then the machine idles). + Cross-boot matrix: mail/matmul/smiley green early; strided/atomics/memset2d/saxpy fail + fast on the remaining data class at any position; event-sync stalls always. Next: fix + the completion post so the recorded event wakes (details in the private handoff, cont. 24). + **Update 2026-10-16 ~18:40 (reliability wave m160-m162 + harness fixes; verified battery, no reboots):** + five cycles (r163-r167) landed and verified the bounds of the earlier "STATUS FAIL / + freeze" repro. Fixed and live-verified across boots: `victory_mail` and + `victory_matmul` (repeats), `smiley`, `vector_add`, `victory_math`, `victory_strided` + and `vecadd_loop` (100 launches) all green on the physical engine; the acked-no-execute + class is recovered by outcome retries (fresh release-word ack, fresh inline-QMD bank + + rebuilt method block: a launch with zero engine writes recovered on retry 2 with its + full 1024-page output). A hardware fetch witness (QMD sentinel) now distinguishes + fetched-vs-not in every retry log. The post-teardown DEVICE-freeze cascade was rooted in + the pin-feeder tooling: blocked guest-trace reads over the guest agent stopped the + pool/tracker pin flow after ~5 client cycles and every later client froze in + `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the + demand-pin content resolver searches a learned per-allocation offset. Remaining + characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch + class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out + mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, + retest pending). + **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):** + the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was + reproduced and root-caused: under the driver's status-queue SWAP_RX mode the + emulation's status free-space check read the status queue's mirrored RX header + (which the emulation publishes itself) instead of the peer consumption cursor in + the command queue's RX header. With the guest mid-drain the status ring was + declared full, replies were written anyway, unread records were overwritten, and + the guest RM read a record one full ring lap ahead + (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed + and the first client's `cudaMalloc` returned "device busy or unavailable" - + poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the + cursor is read from the command queue's RX header per the driver's swap-mode + semantics; per-client channel unregister and session flush work again (one + unregister in the poisoned boot; 79 with one flush per client after). Verified on + fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul` + (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`, + `FFN4 layers=4 bad=0` + `STATUS OK`) all green. + Remaining large-buffer class (current frontier): a `vector_add` cold-run content + gap (a staging read/write race between the emulated CE copy and the guest's + staging ping-pong: ~0.68 MB of the third MB reads the previous staging content), + `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop` + hang class correlated with a boot where the injected-pin stream never re-targeted + the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class + (RM object fields overwritten with client data; stale system-memory destination + resolution on freed staging pages) is characterized but open. Next: pin-stream + re-targeting, then the staging-race and stale-destination fixes; then the + boot-lottery re-measure over fresh boots, the full battery + soak, two guests, + multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live + with the three green kernels. + **Update 2026-10-20 ~20:00 (pin-wins corruption fix; multi-tenant live; battery frontier):** + a second, independent fix landed after the SWAP_RX cursor: the emulation's + write-side ledger could redirect a copy to the page an earlier write landed + on even when the guest had already remapped or freed that VA (4448 such + divergences in the failing run), so guest-kernel memory could receive client + copy data - observed as an RM object's pointer field holding client float + data, a free-path traversal faulting, and a guest panic + ("Attempted to kill init") on the `vector_add` client. With a pin-backed + translation (guest pagemap truth) now winning the divergence and the ledger + re-homing after the write (`720d3f5f`): `vector_add` cold AND warm run + `bad=0` + `STATUS OK` and the guest survives; the cold-run content gap + class is cured with it. Battery on fresh boots: `victory_mail`, + `vecadd_loop` (100 launches), `victory_matmul` (a first-run all-bad flake, + green on immediate retry), `victory_strided`, `victory_math`, + `vector_add`, and a transformer kernel (`dh=256`/`FFN4` `bad=0`) are green; + streams2's ordering class is root-caused (core CUDA starts compute-channel + pushes with a C56F host-semaphore acquire that the decoder ignored, so a + D2H readback ran before the scale launch it waits for) and the multi-pass + content class is the remaining frontier (multi-launch/accumulating kernels + wrong; `victory_reduce` is exactly 2x - the engine never wrote on five + launch-shaped pushes of a one-launch client). Multi-tenant: two mediated + devices (512M + 1GiB) and two guests coexist; guest B reports its 1GiB + VRAM with `nvidia-smi` green; guest-B CUDA awaits its own pin stream. Host + clean, no reboots. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -265,6 +482,12 @@ isolation, performance, portability). the full FWCLIENT client-managed construct (client ctx buffers + VAS mapping + reference promote set), then the server commit. See ADDENDUM 53/54. + **Update 2026-10-14:** this wall moved decisively - the physical GR + engine now executes forwarded guest compute launches, and the first two + battery clients are green end-to-end on the physical backend (host CUDA + bridge disabled). The remaining items are launch-latency robustness (a + first launch after an idle can time out and is auto-retried to success) + and the third-client chain; see the status update above. 2. **Hardening the accepted no-claim path.** The first green guest CUDA acceptance has landed and is holding; the residual classes are (a) corruption via a late first decode behind pump backlog (~3-5% of runs; the experiment @@ -463,6 +686,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM | | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | @@ -499,6 +723,29 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX + cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20 + ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`, + `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM + logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed + (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable") + onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr + lives in the command queue's RX header (the driver's swap-mode semantics), + while the status queue's own RX header is the mirror the emulation publishes + (its own command-queue read position); the status free-space check read the + mirror, so with the guest mid-drain the ring was declared full and replies + were written over unread records. Fix (`c72f8948`): read the peer + consumption cursor from the command queue's RX header. Verified on fresh + boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256 + bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`) + all green with `STATUS OK`; per-client channel unregister + session flush + restored (1 unregister in the poisoned boot vs 79 after). Remaining + frontier: the large-buffer class (`vector_add` cold-run staging race, + `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with + an un-targeted injected-pin stream), then boot-lottery re-measure, full + battery + soak, two guests, multi-mdev, VRAM sizes. Evidence: + `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in + tmux session 0 (SSH to the guest, three green kernels). - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns @@ -1583,7 +1830,7 @@ progress.** ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. -- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages +- Host OS: Ubuntu, kernel 6.8.0-146; the distribution NVIDIA driver manages the physical GPU; `vgsp.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. |