diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 82 |
1 files changed, 67 insertions, 15 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index d0e44d8..ed0b0fa 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,7 @@ # Open-IOV Current State > **Snapshot: 2026-10-03 (reorganized; simulator status current 2026-10-03, -> physical campaign data as of 2026-10-02 ~22:12).** This is a dated public +> physical campaign data as of 2026-10-03 ~21:12).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: @@ -31,10 +31,23 @@ isolation, performance, portability). and relay paths (all parameters default off; one relayed copy's data mismatch is under triage), and the host GR ladder allocates a GR0 channel + compute object. -- **Guest frontier (2026-10-02 ~22:01).** The first real guest CUDA program - against the backend enumerated device properties (`cc=7.5 sm=48 - mem=134152192`) and then stalled in its first device allocation / UVM - channel path before any kernel executed; teardown was clean. +- **Guest frontier (2026-10-03 ~06:10).** The first real guest CUDA program + against the backend enumerates device properties (`cc=7.5 sm=48 + mem=134152192`); the emulation pipeline is now live for it (101+ GR pushes + decoded, 219 GR releases, ~165k completion-counter publishes, 20 GR rings + auto-pinned, 88 forwarded works verified), and it reaches its first + `cudaMalloc`. It then hangs on the UVM page-table tracker: the pool + semaphore lives at a GPU-only address (`0x121000000`) with no CPU pagemap, so + its host physical address is resolved nondeterministically and the guest + never observes the releases. Authoritative GPU-only VA->PA translation is + the immediate blocker. +- **Hardware incident + recovery (2026-10-03).** The first emulation-only + (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the + host driver owned the device and triggered an uncorrectable ECC Xid, after + which `RmInitAdapter` failed persistently even after a full driver reload. + The host recovered after a reboot (~20:11); `nvidia-smi -L` is green and the + device is healthy. The fix (shadow BAR2 in emulation-only mode) is in flight + and gates any further no-claim run. - **Simulator workstream:** see the section below (separate, development-only). **Blockers (current walls).** @@ -48,14 +61,21 @@ isolation, performance, portability). `NV_ERR_INVALID_STATE` (0x40) each. The V5 commit assert `0x52a8b04` is a secondary failure. Next routes: name the soft-0x40 check, root-cause the V5 first failure, or rehost the promote flow in risc-v-gsp-rm-debugger. -2. **First device allocation / UVM channel path in the guest** (the frontier - program's stall) - the first green guest execution is the immediate goal. +2. **Authoritative GPU-only VA->PA translation.** The guest's first + `cudaMalloc` hangs in the UVM page-table tracker because the pool semaphore + (`0x121000000`) has no CPU pagemap and is resolved nondeterministically; + derive it from the guest's GPU page tables, the module's shadow page tables, + or a verified debugfs candidate. This is the immediate goal for the first + green guest execution. 3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side data compare; handled fail-closed, parameter default-off). 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect (`staging not visible before kick`; fail-open, quarantined off the critical - path) and the D0/D3cold power-gating freeze hazard (default-latched locks, - exactly 5 power-domain locks per release). + path); the D0/D3cold power-gating freeze hazard (default-latched locks, + exactly 5 power-domain locks per release); and the newly found **no-claim + BAR2 exposure** (real GPU MMIO mapped to the guest in emulation-only mode + can drive the host-owned device to an ECC Xid; shadow BAR2 before any + further no-claim run). ## Simulator workstream (risc-v-gsp-rm-debugger) @@ -115,6 +135,9 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261003-01 | Guest reaches first `cudaMalloc`; emulation pipeline live | 2026-10-03 ~06:10 | 101+ GR pushes, 219 GR releases, ~165k publishes, 88 forwarded works; hang in UVM tracker pool-sem VA->PA | +| MS-20261003-02 | Hardware incident (Xid 140) + reboot recovery | 2026-10-03 13:47 / ~20:11 | no-claim BAR2 exposure faulted the GPU; reboot restored it; fix in flight | +| MS-20261003-03 | Large-push parser wall fixed; UVM fully engaged | 2026-10-03 ~00:10 | `NON_INC` count>64 / zero-padding parser fix; `verify_ok=62` forwarded guest copies | | MS-20261002-01 | First real guest CUDA program vs the physical backend | 2026-10-02 ~22:01 | device props enumerated; stalls in first alloc/UVM; clean teardown | | MS-20261002-02 | GR promote-side wall root-caused; current wall shared-mgmt GR alloc | 2026-10-02 ~17:35 | default-off change; server wedge on `TURING_COMPUTE_A`; fail-safe held | | MS-20261002-03 | Guest-submission relay + fault relay (P6c) | 2026-10-02 07:43-08:07 | landed default-off; `relays=64`; one data-mismatch under triage | @@ -139,6 +162,31 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261003-01` **The emulation pipeline is live and the guest reaches its first `cudaMalloc` + (2026-10-03 ~00:10-06:10).** The method-stream parser that could never + validate UVM's large page-table pushes (`NON_INC` methods with count > 64, + pushes with leading zero padding) was fixed; UVM is now fully engaged + (11,867 pushes in ~10 s, `walks=124/178`, 62 guest copies forwarded to the + physical GPU with `verify_ok=62`). The victory wave then decoded 101+ GR + pushes, 219 GR releases and ~165k completion-counter publishes, auto-pinned + all 20 GR rings, and verified 88 forwarded works. The first `cudaMalloc` + now proceeds further but hangs on the UVM page-table tracker: the pool + semaphore at GPU-only VA `0x121000000` has no CPU pagemap, so the host + resolves a different physical address across runs and the guest never + observes the releases. Immediate blocker: authoritative GPU-only VA->PA. +- `MS-20261003-02` **Hardware incident and recovery (2026-10-03).** The first + emulation-only (no-claim) acceptance run had the module map real GPU BAR2 + PFNs to the guest while the host driver owned the device; this triggered an + uncorrectable ECC Xid and persistent `RmInitAdapter` failure that survived a + full driver reload. The host was recovered by a reboot (~20:11): + `nvidia-smi -L` green, device bound and healthy, no Xid this boot. The + no-claim BAR2 shadow fix is in flight and gates further no-claim runs. +- `MS-20261003-03` **Large-push parser wall fixed (2026-10-03 ~00:10).** The + parser rejected `NON_INC` method streams over 64 methods and pushes with + leading zero padding, so no rescue could validate UVM's page-table pushes; + after the fix UVM's page-table pushes execute and the guest advances past + the previous registration stall. This is the enabling fix for + `MS-20261003-01`. - `MS-20261002-01` **First real guest CUDA program exercised against the physical backend (2026-10-02 ~22:01).** A fresh supervised module-owned cycle (claim + full GSP-RM boot + guest RAM + hardware-primary work forwarding) booted the real @@ -705,12 +753,16 @@ chain succeeds. ## Evidence pointers -- **Module sha256 (latest runs):** `e5318ee0...` (Addendum 30, 2026-10-03, - large-push parser fix); earlier: P6b `5fb5bfbe...`/`9d444908...`, P6a - `e1bce651...`, P3 `30dfd311...`, P2 `69eac176...`, G0 `dde36d92...`, P5 - wave 1 `83952506...`; boot `12cf3a41-...` unchanged across Addenda 28-30. -- **Campaign detail:** `Handoff-vGPU-IO.md` Part 93 Addenda 26-30 (the - authoritative live narrative; this page is the dated public snapshot). +- **Module sha256 (latest runs):** no-claim incident run `30a13ab4...` + (2026-10-03, emulation-only acceptance); earlier `e5318ee0...` (large-push + parser fix), P6b `5fb5bfbe...`/`9d444908...`, P6a `e1bce651...`, P3 + `30dfd311...`, P2 `69eac176...`, G0 `dde36d92...`, P5 wave 1 `83952506...`. + The host was rebooted 2026-10-03 ~20:11 (previous boot `12cf3a41-...`); the + module was rebuilt 21:10:57 and reloaded 21:12 with the post-incident + no-claim fixes. +- **Campaign detail:** the private live campaign narrative (handoff/logs, + maintained outside this repository) is authoritative; this page is the dated + public snapshot. - **Local evidence archive:** `open-iov/backups/evidence-20261002-victory/` (`*-victory8/9`, `14-guest-dmesg-fail-*`, `20-gsp-phys-state-*`); earlier cycle 11-25 runs with per-wall reports (local archive, untracked). |