From 3ae7f57fd16b4c1850ad0a7dfffdb781f9bac14d Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Mon, 5 Oct 2026 12:12:01 -0400 Subject: STATE-CURRENT: hourly check-back snapshot (2026-10-05 12:12:01) --- STATE-CURRENT.md | 77 ++++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 13 deletions(-) diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 5bd6cad..71da3bc 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,7 @@ # Open-IOV Current State > **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-05 ~10:38; debugging-options +> physical/guest campaign data as of 2026-10-05 ~12:10; debugging-options > catalogue added 2026-10-05).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -30,8 +30,16 @@ isolation, performance, portability). semaphore released, destination equals source). Guest CE copies are hardware-authored with engine-authored releases via the P6b/P6c forwarding and relay paths (all parameters default off; one relayed copy's data - mismatch is under triage), and the host GR ladder allocates a GR0 channel + - compute object. + mismatch is under triage), and the host GR ladder allocates a GR0 channel + + compute object. **Update 2026-10-05 ~12:00:** the promote wall is crossed on + the shared-management VAS - with the srvctx registry profile + (`RMSetClientRMAllocatedCtxBuffer=0`, `RMSplitVasMgmtServerClientRm=0`; both + default off), the V5 channel and the client `MAP_MEMORY_DMA` path, the + `TURING_COMPUTE_A` compute-object alloc succeeds, the promote/commit passes + and the ladder completes stage 3 (`hGr`, clean releases, no server wedge, + no reboot needed) - the first time. The new wall is stage 4: the submitted + GR semaphore push is not consumed by the engine (`GPGet=0`, no release, + `-110`). - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -54,7 +62,15 @@ isolation, performance, portability). Residual classes: late-decode corruption (~3-5%), an early UVM channel-manager stall (~5-10%), and a guest init panic (~1/10, retried). The vadd executes via the host CUDA bridge daemon; forwarding guest engine - work to the physical GPU remains the open backend wall. + work to the physical GPU remains the open backend wall. **Update 2026-10-05 + ~12:10:** the validation campaign was green through `242` when a later, + uncommitted follow-on experiment (refusing pool writes) **panicked the host + kernel at 11:18** - a sleeping VFIO mutex taken from the sema service's GMMU + walk in atomic context, then a vmalloc fatal exception in the bridge marshal + path; the experiment was saved as a patch, reverted, and a reboot restored + the host. Per the 2026-10-05 directive, priority is now **native guest + execution without the host CUDA bridge**, so this front is a stability + exercise rather than the destination. - **Hardware incident + recovery (2026-10-03).** The first emulation-only (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the host driver owned the device and triggered an uncorrectable ECC Xid, after @@ -82,10 +98,20 @@ isolation, performance, portability). the V6 soft-0x40 is the known rec-677 message emitted through the promote error machinery. The rec-677 format text lives in runtime-inflated metadata (not in the shipped image), so naming it needs a live-GSP metadata dump. - Next routes: name the soft-0x40 record from live metadata, root-cause the V5 - first failure (the log call site before the assert is now identified), or - root-cause the shared-management `TURING_COMPUTE_A` wedge; physical cycles - only with a named hypothesis and supervision. + Next routes: name the soft-0x40 record from live metadata, root-cause the V5 + first failure (the log call site before the assert is now identified), or + root-cause the shared-management `TURING_COMPUTE_A` wedge; physical cycles + only with a named hypothesis and supervision. **Update 2026-10-05 ~12:00: + this wall moved.** The untested combination (shared-management VAS + + srvctx registry + V5 channel + no client promote) is confirmed with **no + wedge**: `TURING_COMPUTE_A` succeeds, the promote/commit passes and the + ladder completes stage 3 with clean releases, once the srvctx registry + also disables split VAS management (`RMSplitVasMgmtServerClientRm=0`) and + the client sends `MAP_MEMORY_DMA` for the ladder buffers - on a shared VAS + the server owns the lower page tables, so raw client PTEs are not + authoritative. New wall: stage 4's GR push is submitted but not + fetched/executed by the engine (`GPGet=0`, `-110`). Next: root-cause the + stage-4 non-consumption (GPFIFO fetch, server PDB, doorbell). 2. **Hardening the accepted no-claim path.** The first green guest CUDA acceptance has landed and is holding; the residual classes are (a) corruption via a late first decode behind pump backlog (~3-5% of runs; the experiment @@ -97,8 +123,12 @@ isolation, performance, portability). panic class (~1/10, retried; host unaffected). Remaining work: settle or fail-loud the residual classes, trim/re-run with the bounded diagnostics, and extend acceptance to a second workload. Guest-truth kernel probes - (installed per guest boot) are load-bearing for the GPU-only pool/push-buffer - translations; without them the host falls back to content heuristics. + (installed per guest boot) are load-bearing for the GPU-only pool/push-buffer + translations; without them the host falls back to content heuristics. The + 2026-10-05 continuation experiment (refusing pool writes) panicked the host + and is reverted; the directive now prioritizes bridge-free guest GR + execution, so this front continues as a stability exercise on the + committed hold-all-unreadable build (`4115f1dd`). 3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side data compare; handled fail-closed, parameter default-off). 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect @@ -107,9 +137,13 @@ isolation, performance, portability). exactly 5 power-domain locks per release); and the **no-claim BAR2 exposure** (real GPU MMIO mapped to the guest in emulation-only mode can drive the host-owned device to an ECC Xid; shadowed BAR2 prevents it - - committed and holding, no Xid since). Recurring and separate: guest-kernel boot - panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup - - guest-side only, host stays healthy. + committed and holding, no Xid since). Recurring and separate: guest-kernel boot + panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup - + guest-side only, host stays healthy. A new host panic class was logged + 2026-10-05 11:18 with the (saved, reverted) pool-write-refusal experiment: a + sleeping VFIO mutex taken in atomic context in the sema GMMU walk plus a + vmalloc fatal exception in the bridge marshal path; reboot recovered the + host, no Xid. ## Options to accelerate debugging toward end-to-end CUDA (candidate techniques; not directives) @@ -276,6 +310,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261005-02 | GR promote wall crossed on the shared-VAS+srvctx path; new wall = stage-4 push not consumed | 2026-10-05 ~12:00 | srvctx registry (`RMSetClientRMAllocatedCtxBuffer=0`, `RMSplitVasMgmtServerClientRm=0`) + V5 + mode 17 + client `MAP_MEMORY_DMA`; stage 3 completes (`hGr`, clean release, no wedge); stage 4 `GPGet=0`/`-110` | | MS-20261005-01 | Hardening the accepted no-claim path (hazard classes closed; rates up) | 2026-10-05 overnight | per-op once bitmap + epoch/content-hash gate + grace 15 s (`b645cf6b`, `3b094c2d`, `c3cd4631`); first-attempt 86% -> ~94-95%; fakes/run 3.39 -> 0.61; hold-all-unreadable/25 s experiment committed (`4115f1dd`, module `e7ff3bf7`), campaign `239..262` validating (`239-242` green; `241a`/`243a` infra retries; `243b` live) | | MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign | | MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet | @@ -308,6 +343,22 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261005-02` **GR promote wall crossed on the shared-management path + (2026-10-05 ~12:00).** The untested GR-ladder combination - shared-management + VAS, the srvctx pre-boot registry (`RMSetClientRMAllocatedCtxBuffer=0`), + a V5 (non-skip) channel and no client promote - gets past the promote wall + with **no wedge**: the `TURING_COMPUTE_A` compute-object alloc succeeds (the + earlier shared-VAS assert storm does not recur), BIND/schedule/token are + accepted, and the ladder completes stage 3 for the first time (`hGr` + allocated; clean releases; WPR2 down; device rebound). Two further + requirements were source-confirmed and implemented: on a shared VAS the + server owns the lower page tables, so the client must send + `MAP_MEMORY_DMA` for the ladder buffers, and GSP-RM only serves that RPC + when the srvctx registry also sets `RMSplitVasMgmtServerClientRm=0`. New + wall: stage 4's GR semaphore push is submitted but not fetched/executed by + the engine (`GPGet=0`, no release, `-110`). All ladder cycles released the + device cleanly (no reboot, no Xid, pstore empty); the fixes are uncommitted + in the working tree. - `MS-20261005-01` **Hardening the accepted no-claim path (2026-10-05 overnight).** Three kernel-side fixes closed the replay/fake hazard classes after the first acceptance: (1) a held-entry retry must not re-run completed -- cgit v1.2.3