diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 244 |
1 files changed, 242 insertions, 2 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 9bfbff5..441ea85 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,118 @@ # Open-IOV Current State -> **Snapshot: 2026-10-16 ~15:30 (integration wave: bigpath+dual merged, five fixes -> landed through `7d123630`; live r161/r162 cycles on kernel 6.8.0-147).** +> **Snapshot: 2026-10-30 ~12:20 (SESSION 153, r1013 live; HEAD `a64169db`).** +> The arc since the SESSION 143 snapshot split in two. Guest CUDA on the +> physical engine: the post-reboot stall is root-caused and fixed - the GR +> ring-autopin fallback and the CE first-submission capture each guarded +> block ownership only within their own engine class, so a channel could +> latch a live foreign FB block (`41343c63`: cross-class block claims, +> pre-latch CE reservations, own-block latch semantics) - and the fixed +> build is green repeatedly (`victory_mail` `VADD bad=0` + `STATUS OK`; +> `vecadd_loop` 100 launches PASS). The fresh-boot battery's remaining red +> class is D2H-stage (release-before-work, a ledger-vs-writer page +> divergence and a memset readback/writeback race are all proven; +> release-discipline fixes are staged but uncommitted; the opdef-deadline +> slot-complete path is the next fix). X/GL on the emulated vGPU: the wall +> that stood since the early sessions is closed - `fffae7b1` gets Xorg `:0` +> through the nvidia-push DMA sanity push and ScreenInit (`glxinfo` = +> OpenGL 4.6.0 NVIDIA 570.144, direct rendering, GZDoom launches), with the +> RC watchdog channel registered/serviced (`8898bd00`+`42c9626b`) and +> per-client VAS resolution (`38ba1e2d`: a channel's VAs resolve through +> its own VAS first). The app-side 10 s/frame waits are the client's own +> give-up manager; the `sema_xw_mirror` pair completes presents (glxgears +> latches 128-146 FPS, 200+ in the later shim runs) but produces no +> rendered output - the GL draw stream is marshalled, not executed. The +> engine now executes the real GL method stream in two forms: the C597 +> section capture + synthesized compute launch retires its QMD on real +> sm_75 SASS (`0ee574a3`..`595e023a`), and the verbatim 3D method-stream +> replay lane retires real 3D methods with no QMD (r932); the hard limit +> is that rasterised GL pixels still need the real 3D pipeline. SESSION +> 153's headline: the direct GL client's device-init gate is root-caused to +> the driver-shared memfd slot word (bit 0 set = glcore skips its +> class-0xc46f channel creation); cleared, the client allocates its +> channels (0xbeef0100/0101/0102 plus 0x902d/0xc5b5/0xc597/0x9072/0xa140/ +> 0x90f1) and `glReadPixels` returns with GL error 0 - fixed field-side by +> the kernel keeper `sema_shared_hold_pa` (`a64169db`, holds bit0 clear +> deterministically) plus host hold/toggle automation (the MakeCurrent +> handshake needs the 2<->3 transition). Current walls (ranked): (a) the +> fresh-boot Xorg DMA-init failure in r1013 (a 10 s `Failed to initialize +> DMA` -> no X server, so no client; a different wait from the member-spin +> class); (b) the client PUT-visibility wall - the direct client's channels +> never submit (its process writes no GP_PUT anywhere, the BAR0/BAR1-shadow +> premise was falsified by direct measurement, and the submitting DDX-side +> channel's USERD is guest RAM already tracked by the RAM scan); (c) the +> pixel wall - no render content reaches any ring (X-side pushes are 32-64 +> B housekeeping BEGINs; every capture/readback is black). Landmine: +> killing/exiting a direct client can trip a 4 GB `nv_alloc_system_pages` +> guest-kernel oops - never kill clients; re-cycle instead. Delivered on +> the emulated vGPU: the R18 CUDA-animation video (`70134652`; 60/60 +> frames zero faults, frames exact vs CPU math, single-page kernels only - +> the window-0 code shadow carries one 4 KB page, so kernels >4 KB stay +> hostile until it grows) and two two-VM GZDoom videos (`375d627a` tools; +> the GPU-accelerated one runs on virtio-gpu/virgl on the host iGPU, not +> the emulated vGPU, which remains the core frontier). Host clean, pstore +> empty; live: r1013 mdev `95446a6c-d5da-4e74-920b-5ac0d7ccc303`, QEMU +> 2335256, guest no Xorg/no GL client, pin/sem helpers up. Also open: the +> SESSION 143 Vulkan-client completion regression (fence timeout with the +> push re-completed) - not re-measured since r816, priority moved to the +> GL path. Commits this arc: `41343c63`, `fffae7b1`, `8898bd00`, +> `42c9626b`, `38ba1e2d`, `6f2e6193`, `0ee574a3`..`595e023a`, `70134652`, +> `375d627a`, `1fd8d50a`, `f500779b`..`a64169db` (SESSION 153 ladder). +> +> **Snapshot: 2026-10-26 (SESSION 143, post-reboot).** The r800 walk guard +> holds and the `gsp_phys_gr_min_data=0` bypass reaches the real engine: the +> guest Vulkan compute launch is now *submitted* (no oops, no degenerate-set +> refusal), the engine fetches the host-deposited QMD and the firmware reports +> a real `MMU Fault ... @ 0xdf_d8feb000, FAULT_PDE` at the guest program VA - +> the channel/context are alive, the program page is empty and unmapped. The +> util-sweep channel-visibility work (per-channel begin/skip/read=none lines) +> proves the Vulkan copy-class channels (0xbeef0101/0xbeef0102, engines 11/12) +> are registered and swept but never pushed, and that no channel carries +> program bytes or a `LAUNCH_DMA`: the code page (0x62b000 / rotating 0xc8b000) +> never fills. The frontier is therefore (1) the driver's staging-ring -> +> code-heap upload copy, which is not among the decoded pushes, and (2) the +> engine page-table mapping for the program VA. Separately, the refusal path no +> longer reaches the earlier three-black-frame baseline (`SYNTH-FAULT` vs the +> software completion/fence ordering), which must be restored before further +> observation. Docs: `docs/04-implementation/04-real-render-walls.md` sections +> 20-21. Commits: `34e611f4`, `c5510744`, `28def33e`, `0ca8ff9f`. +> +> **Snapshot: 2026-11-06 ~01:00 (the guest's REAL Vulkan pipeline now +> renders on the emulated vGPU: the first `VKFRAME seq=1..3` / +> `VKFRAME_DONE frames=3 640x480 saved=3` / `VKFRAME_IDLE ok` runs +> end-to-end — instance, device, allocation, image/view/buffer/shader/ +> descriptor/pipeline/command-buffer/fence, queue submit, fence release +> and frame readback all complete with a zero-touch service pipeline. +> The chain of fixes behind it: the push-stream parser now accepts the +> driver's real shapes (the `0x0001fff0` SET_SUBDEVICE_MASK bracket and +> zero NOP gaps — the class that silently skipped every large push), the +> fence fold (`sema_gr_ahead`) sweeps the pre-built GPFIFO continuation +> beyond GP_PUT and applies the submit's INC-5 RELEASE groups +> forward-only, the host-triggered VAS page-directory anchoring verifies +> the parked `COPY_SERVER_PDES` candidate against a content-located ring +> pair so the walk resolves every push VA before its first read, the +> COMPR+2MB VidHeap class is downgraded at the ioctl so +> `vkAllocateMemory` succeeds, the SEC2 scrubber's finish payload is +> authored by the service, the `0x9096` VidHeap enumeration and the +> `FIFO_GET_ENGINE_CONTEXT_PROPERTIES` ZCULL geometry are answered, and +> the C5C0-section -> vendor-launch translator found the real dispatch +> (a `SET_INLINE_QMD_ADDRESS_B` + 64-word run building QMD `0x4b85f00`, +> grid 40x30, plus the C5B5 copy-out). The frames are still BLACK: the +> driver delivers SASS/descriptors through its MME program + inline-to- +> memory stream (decoded in full; the payload rides the next GPFIFO +> entry as a source-view call), and the utility/copy-channel +> `LOAD_INLINE_DATA` arm that must materialize those writes is the +> current frontier (arbitration + `sema-utildec` witness landed). CUDA +> meanwhile: the best fresh-boot battery is 8 pass / 3 fail / 1 hang +> (the Vulkan work does not regress CUDA; always battery-first on a +> fresh boot); GL runs on the guest GPU (next wall = the FBO class); the +> Looking Glass frame path is proven end-to-end (guest render -> KVMFR +> ivshmem -> host PNG frame); two guests with 512M + 1GiB emulated VRAM +> coexist at boot level; the 0.9.9 release-preview demo video v1 is +> synced to files.vgpu.io (the updated video with GZDOOM working is the +> next publish). Frontier = non-black frames, then Xorg -> GZDOOM +> (GL/Vulkan) -> KVMFR/LG frames -> CUDA 15/15 + soak -> the perfect +> 1.0.0 video -> Windows guests).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -253,6 +364,111 @@ isolation, performance, portability). contamination demonstrated live) and needs the per-guest ownership rework (owner-tagged pin/map/pending lookups) before any dual-CUDA claim. No reboot; host GPU green; both guests left running. + **Update 2026-10-16 ~22:35, extended ~23:05 (reliability wave 2, m163-m170; cycles r169-r177, no host reboots):** + the stall classes that made clients freeze at DEVICE are largely closed. Fixed and + verified across boots r172/r174/r175/r177: the session flush can no longer wipe a live + session (a deferred flush executed with the client's channels valid, dropped its + queued launch release, and froze it in the tracker wait); GR entries whose decode + sees QMD words but builds no launch candidate are held and re-captured instead of + consumed silently; a failed device-destination copy holds its entry instead of + silently dropping the data; pool releases apply at fresh-pin time instead of + registration+5 s; the timeout ladder fails fast when the engine is wedged; argument + spans whose first page resolves nothing hold the launch (bounded) instead of + forwarding stale operands; device-destination copies answered only by a page-table + walk wait (bounded 3 s) for a pin. `victory_mail`, `victory_matmul` and `smiley` + are green on every boot of the wave, including the user's exact repro and mail as + the eighth client of a used boot; r177 ran mail, matmul, smiley, matmul (repeat) + and mail GREEN back-to-back as its first five clients. streams2/memset2d/saxpy no + longer wedge - they + now fail fast (~35-60 s) on the remaining data class: an H2D into a device buffer + that does not land where the launch's data fill later reads the operand (the + vidmem write path vs the fill's read path divergence; analysis and the ranked + ledger fix are in the private handoff, cont. 23). + **Update 2026-10-17 ~01:00 (stall pinpointed on r179; diagnostics only, no tree change yet):** + a minimal repro (`cudaEventCreate` + `cudaEventRecord` on the default stream + + `cudaEventSynchronize`) sits forever after the record - the first CUDA call pinned down + as a hard hang. A 3-stream workload using event record/wait/elapsed time stalls even as + the very first client of a fresh boot (intrinsic, not position- or poison-dependent; + unclean client kills were tested and do not break the next client). On a used boot the + user's `victory_mail`/`victory_matmul` hit the same wait: the client spins in driver + userspace while a background worker polls the interrupt-service RM control once per + second. The host's event-registration table and post path exist (register on NV01_EVENT + alloc, retire on free, per-channel/all post helpers) and the stall windows still show + frequent post failures with EINVAL; the host sema side is verified healthy at the stall + (client pushes decode, pool releases apply to the correct page, then the machine idles). + Cross-boot matrix: mail/matmul/smiley green early; strided/atomics/memset2d/saxpy fail + fast on the remaining data class at any position; event-sync stalls always. Next: fix + the completion post so the recorded event wakes (details in the private handoff, cont. 24). + **Update 2026-10-16 ~18:40 (reliability wave m160-m162 + harness fixes; verified battery, no reboots):** + five cycles (r163-r167) landed and verified the bounds of the earlier "STATUS FAIL / + freeze" repro. Fixed and live-verified across boots: `victory_mail` and + `victory_matmul` (repeats), `smiley`, `vector_add`, `victory_math`, `victory_strided` + and `vecadd_loop` (100 launches) all green on the physical engine; the acked-no-execute + class is recovered by outcome retries (fresh release-word ack, fresh inline-QMD bank + + rebuilt method block: a launch with zero engine writes recovered on retry 2 with its + full 1024-page output). A hardware fetch witness (QMD sentinel) now distinguishes + fetched-vs-not in every retry log. The post-teardown DEVICE-freeze cascade was rooted in + the pin-feeder tooling: blocked guest-trace reads over the guest agent stopped the + pool/tracker pin flow after ~5 client cycles and every later client froze in + `uvm_tracker_wait`; the trace reads are now remotely timed and SSH-preferred, and the + demand-pin content resolver searches a learned per-allocation offset. Remaining + characterized gaps (evidence in the private handoff): a fetched-but-not-executed launch + class, a no-ack front-end stall class, the multi-buffer streams2 reloc/copy-out + mismatch, and the 8 MB CE roundtrip demand-resolution crawl (scanner fix landed, + retest pending). + **Update 2026-10-20 ~05:30 (boot lottery root-caused and fixed; guest CUDA kernels green end-to-end; transformer verified):** + the user's reported repro (`victory_mail` `STATUS FAIL`, `victory_matmul` freeze) was + reproduced and root-caused: under the driver's status-queue SWAP_RX mode the + emulation's status free-space check read the status queue's mirrored RX header + (which the emulation publishes itself) instead of the peer consumption cursor in + the command queue's RX header. With the guest mid-drain the status ring was + declared full, replies were written anyway, unread records were overwritten, and + the guest RM read a record one full ring lap ahead + (`GspMsgQueueReceiveStatus: Bad sequence number`), after which every RPC failed + and the first client's `cudaMalloc` returned "device busy or unavailable" - + poisoning the whole boot (the boot lottery). Fixed (`c72f8948`, pushed): the + cursor is read from the command queue's RX header per the driver's swap-mode + semantics; per-client channel unregister and session flush work again (one + unregister in the poisoned boot; 79 with one flush per client after). Verified on + fresh boots: `victory_mail` (`VADD n=32768 bad=0` + `STATUS OK`), `victory_matmul` + (`n=256 bad=0` + `STATUS OK`) and a transformer kernel (`dh=256 seq=64 bad=0`, + `FFN4 layers=4 bad=0` + `STATUS OK`) all green. + Remaining large-buffer class (current frontier): a `vector_add` cold-run content + gap (a staging read/write race between the emulated CE copy and the guest's + staging ping-pong: ~0.68 MB of the third MB reads the previous staging content), + `smiley` all-bad, `streams2` `bad=65503`, and a `memcpy_roundtrip`/`vecadd_loop` + hang class correlated with a boot where the injected-pin stream never re-targeted + the live client (ipins=0 vs ~15k on green boots). A guest-kernel corruption class + (RM object fields overwritten with client data; stale system-memory destination + resolution on freed staging pages) is characterized but open. Next: pin-stream + re-targeting, then the staging-race and stale-destination fixes; then the + boot-lottery re-measure over fresh boots, the full battery + soak, two guests, + multi-mdev and the VRAM sizes. Host clean, GPU green, no reboots; demo VM live + with the three green kernels. + **Update 2026-10-20 ~20:00 (pin-wins corruption fix; multi-tenant live; battery frontier):** + a second, independent fix landed after the SWAP_RX cursor: the emulation's + write-side ledger could redirect a copy to the page an earlier write landed + on even when the guest had already remapped or freed that VA (4448 such + divergences in the failing run), so guest-kernel memory could receive client + copy data - observed as an RM object's pointer field holding client float + data, a free-path traversal faulting, and a guest panic + ("Attempted to kill init") on the `vector_add` client. With a pin-backed + translation (guest pagemap truth) now winning the divergence and the ledger + re-homing after the write (`720d3f5f`): `vector_add` cold AND warm run + `bad=0` + `STATUS OK` and the guest survives; the cold-run content gap + class is cured with it. Battery on fresh boots: `victory_mail`, + `vecadd_loop` (100 launches), `victory_matmul` (a first-run all-bad flake, + green on immediate retry), `victory_strided`, `victory_math`, + `vector_add`, and a transformer kernel (`dh=256`/`FFN4` `bad=0`) are green; + streams2's ordering class is root-caused (core CUDA starts compute-channel + pushes with a C56F host-semaphore acquire that the decoder ignored, so a + D2H readback ran before the scale launch it waits for) and the multi-pass + content class is the remaining frontier (multi-launch/accumulating kernels + wrong; `victory_reduce` is exactly 2x - the engine never wrote on five + launch-shaped pushes of a one-launch client). Multi-tenant: two mediated + devices (512M + 1GiB) and two guests coexist; guest B reports its 1GiB + VRAM with `nvidia-smi` green; guest-B CUDA awaits its own pin stream. Host + clean, no reboots. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -576,6 +792,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261020-01 | Boot lottery root-caused and fixed (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end | 2026-10-20 ~03:00-05:30 | `c72f8948`; mail `VADD n=32768 bad=0` + matmul `n=256 bad=0` + transformer `dh=256 bad=0`/`FFN4 4 layers bad=0` all `STATUS OK` on fresh boots; per-client unregister/flush restored; remaining: large-buffer class (staging race, smiley/streams2, hang vs un-targeted pin stream), then battery/soak/two-guest/multi-mdev/VRAM | | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | @@ -612,6 +829,29 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261020-01` **Boot lottery root-caused and fixed (status-queue SWAP_RX + cursor); guest CUDA kernels green end-to-end on fresh boots (2026-10-20 + ~03:00-05:30).** The user's exact repro (`victory_mail` `STATUS FAIL`, + `victory_matmul` freeze) was reproduced as a poisoned boot: the guest RM + logged `GspMsgQueueReceiveStatus: Bad sequence number` and every RPC failed + (`0x25`) from the first client's `cudaMalloc` ("device busy or unavailable") + onward. Root cause: under `MSGQ_FLAGS_SWAP_RX` the driver's status readPtr + lives in the command queue's RX header (the driver's swap-mode semantics), + while the status queue's own RX header is the mirror the emulation publishes + (its own command-queue read position); the status free-space check read the + mirror, so with the guest mid-drain the ring was declared full and replies + were written over unread records. Fix (`c72f8948`): read the peer + consumption cursor from the command queue's RX header. Verified on fresh + boots: `victory_mail` (`VADD n=32768 bad=0`), `victory_matmul` (`n=256 + bad=0`) and a transformer kernel (`dh=256 bad=0`, `FFN4 layers=4 bad=0`) + all green with `STATUS OK`; per-client channel unregister + session flush + restored (1 unregister in the poisoned boot vs 79 after). Remaining + frontier: the large-buffer class (`vector_add` cold-run staging race, + `smiley`, `streams2`, `memcpy_roundtrip`/`vecadd_loop` hang correlated with + an un-targeted injected-pin stream), then boot-lottery re-measure, full + battery + soak, two guests, multi-mdev, VRAM sizes. Evidence: + `backups/evidence-20261002-victory/` cycles r330-r335; demo VM live in + tmux session 0 (SSH to the guest, three green kernels). - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns |