From d350fb6eb56c67fd8a435f4f41f7b81d7c62c7cb Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Mon, 5 Oct 2026 10:39:31 -0400 Subject: STATE-CURRENT: 10:38 check-back snapshot (campaign 239-242 green; 243b live; committed experiment e7ff3bf7) --- STATE-CURRENT.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 7a52b80..5bd6cad 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,7 @@ # Open-IOV Current State > **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-05 ~10:12; debugging-options +> physical/guest campaign data as of 2026-10-05 ~10:38; debugging-options > catalogue added 2026-10-05).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -48,8 +48,9 @@ isolation, performance, portability). and fakes/run fell 3.39 -> 0.61. A further experiment (hold every unreadable CE entry for the bounded grace, delete the silent pre-rescue fake, grace 25 s) is now committed (`4115f1dd`, module `e7ff3bf7`) and under live - validation: campaign `239..262` is running with `237-240` green on the - deterministic acceptance and only a harness push failure (`241a`, retried). + validation: campaign `239..262` is running with `239-242` green on the + deterministic acceptance (`241a`/`243a` harness push failures retried) and + `243b` in progress at the 10:38 check. Residual classes: late-decode corruption (~3-5%), an early UVM channel-manager stall (~5-10%), and a guest init panic (~1/10, retried). The vadd executes via the host CUDA bridge daemon; forwarding guest engine @@ -59,8 +60,9 @@ isolation, performance, portability). host driver owned the device and triggered an uncorrectable ECC Xid, after which `RmInitAdapter` failed persistently even after a full driver reload. The host recovered after a reboot (~20:11); `nvidia-smi -L` is green and the - device is healthy. The fix (shadow BAR2 in emulation-only mode) is in flight - and gates any further no-claim run. + device is healthy. The fix (shadow BAR2 in emulation-only mode) is committed + (in `cf6ff542`) and has held across the acceptance campaigns since - no Xid + observed. - **Simulator workstream:** see the section below (separate, development-only). **Blockers (current walls).** @@ -102,11 +104,10 @@ isolation, performance, portability). 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect (`staging not visible before kick`; fail-open, quarantined off the critical path); the D0/D3cold power-gating freeze hazard (default-latched locks, - exactly 5 power-domain locks per release); and the newly found **no-claim - BAR2 exposure** (real GPU MMIO mapped to the guest in emulation-only mode - can drive the host-owned device to an ECC Xid; shadow BAR2 before any - further no-claim run); the shadow-BAR2 fix is now in the working tree and in - the running module, pending commit. Recurring and separate: guest-kernel boot + exactly 5 power-domain locks per release); and the **no-claim BAR2 + exposure** (real GPU MMIO mapped to the guest in emulation-only mode + can drive the host-owned device to an ECC Xid; shadowed BAR2 prevents it - + committed and holding, no Xid since). Recurring and separate: guest-kernel boot panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup - guest-side only, host stays healthy. @@ -275,7 +276,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| -| MS-20261005-01 | Hardening the accepted no-claim path (hazard classes closed; rates up) | 2026-10-05 overnight | per-op once bitmap + epoch/content-hash gate + grace 15 s (`b645cf6b`, `3b094c2d`, `c3cd4631`); first-attempt 86% -> ~94-95%; fakes/run 3.39 -> 0.61; hold-all-unreadable/25 s experiment committed (`4115f1dd`, module `e7ff3bf7`), campaign `239..262` validating (`237-240` green) | +| MS-20261005-01 | Hardening the accepted no-claim path (hazard classes closed; rates up) | 2026-10-05 overnight | per-op once bitmap + epoch/content-hash gate + grace 15 s (`b645cf6b`, `3b094c2d`, `c3cd4631`); first-attempt 86% -> ~94-95%; fakes/run 3.39 -> 0.61; hold-all-unreadable/25 s experiment committed (`4115f1dd`, module `e7ff3bf7`), campaign `239..262` validating (`239-242` green; `241a`/`243a` infra retries; `243b` live) | | MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign | | MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet | | MS-20261004-02 | Latent bridge pending-slot oops found, fixed; reboot | 2026-10-04 ~14:45-15:00 | `cubb_pend_release` slot reuse -> NULL deref; fixed `85539844...`; approved reboot restored the GPU | @@ -321,7 +322,8 @@ chain succeeds. defer-the-release branch missed the late-first-decode path); a third (hold every unreadable CE entry, delete the silent pre-rescue fake, 25 s grace) is now committed (`4115f1dd`, module `e7ff3bf7`) and under live validation - (campaign `239..262`; `237-240` green, only an infra push failure at `241a`). + (campaign `239..262`; `239-242` green, the only failures being harness pushes + at `241a`/`243a`, retried; `243b` live at the 10:38 check). Residual classes: late-decode corruption (~3-5%), early UVM channel-manager stall (~5-10%), guest init panic (~1/10, retried). - `MS-20261004-04` **First green end-to-end guest CUDA acceptance (2026-10-04 -- cgit v1.2.3