diff options
| -rw-r--r-- | STATE-CURRENT.md | 72 |
1 files changed, 53 insertions, 19 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index ed0b0fa..7f7e335 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,7 @@ # Open-IOV Current State -> **Snapshot: 2026-10-03 (reorganized; simulator status current 2026-10-03, -> physical campaign data as of 2026-10-03 ~21:12).** This is a dated public +> **Snapshot: 2026-10-04 (reorganized; simulator status current 2026-10-04, +> physical/guest campaign data as of 2026-10-04 ~15:12).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: @@ -31,16 +31,19 @@ isolation, performance, portability). and relay paths (all parameters default off; one relayed copy's data mismatch is under triage), and the host GR ladder allocates a GR0 channel + compute object. -- **Guest frontier (2026-10-03 ~06:10).** The first real guest CUDA program - against the backend enumerates device properties (`cc=7.5 sm=48 - mem=134152192`); the emulation pipeline is now live for it (101+ GR pushes - decoded, 219 GR releases, ~165k completion-counter publishes, 20 GR rings - auto-pinned, 88 forwarded works verified), and it reaches its first - `cudaMalloc`. It then hangs on the UVM page-table tracker: the pool - semaphore lives at a GPU-only address (`0x121000000`) with no CPU pagemap, so - its host physical address is resolved nondeterministically and the guest - never observes the releases. Authoritative GPU-only VA->PA translation is - the immediate blocker. +- **Guest frontier (2026-10-04 ~15:03).** The emulation-only ("no-claim") + campaign advanced the guest CUDA program past its first `cudaMalloc` to the + `vectorAdd` launch. Guest-truth kernel probes now back the pool/push-buffer + translations (pushes decode with no fakes), the device-to-host mail read-back + succeeds (the host daemon completes the marshalled job, `CUBB2 status=ok`), + and raising the emulated FB from 256 MiB to 512 MiB unblocked `cudaMalloc`, so + the guest reaches `launch ... op=vadd grid=4096,1,1 ... n=1048576`. A latent + bridge pending-slot defect (a freed slot reused with a stale done-length) was + found and fixed after it oopsed the service thread and forced an approved + reboot (~15:00). The first **green** guest `vectorAdd` (`STATUS OK`, `bad=0`) + has **not** yet been captured: the only post-reboot acceptance run was blocked + by a guest-boot kernel panic before the client started. The host physical/GR + wall is unchanged. - **Hardware incident + recovery (2026-10-03).** The first emulation-only (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the host driver owned the device and triggered an uncorrectable ECC Xid, after @@ -61,12 +64,12 @@ isolation, performance, portability). `NV_ERR_INVALID_STATE` (0x40) each. The V5 commit assert `0x52a8b04` is a secondary failure. Next routes: name the soft-0x40 check, root-cause the V5 first failure, or rehost the promote flow in risc-v-gsp-rm-debugger. -2. **Authoritative GPU-only VA->PA translation.** The guest's first - `cudaMalloc` hangs in the UVM page-table tracker because the pool semaphore - (`0x121000000`) has no CPU pagemap and is resolved nondeterministically; - derive it from the guest's GPU page tables, the module's shadow page tables, - or a verified debugfs candidate. This is the immediate goal for the first - green guest execution. +2. **First green guest acceptance (no-claim path).** The no-claim bridge now + reaches the `vectorAdd` launch, but no `STATUS OK` has been captured because a + guest-boot kernel panic blocked the only post-reboot run; a clean guest boot on + the fixed module is the immediate step. Guest-truth kernel probes (installed + per guest boot) are load-bearing for the GPU-only pool/push-buffer + translations; without them the host falls back to content heuristics. 3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side data compare; handled fail-closed, parameter default-off). 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect @@ -75,7 +78,10 @@ isolation, performance, portability). exactly 5 power-domain locks per release); and the newly found **no-claim BAR2 exposure** (real GPU MMIO mapped to the guest in emulation-only mode can drive the host-owned device to an ECC Xid; shadow BAR2 before any - further no-claim run). + further no-claim run); the shadow-BAR2 fix is now in the working tree and in + the running module, pending commit. Recurring and separate: guest-kernel boot + panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup - + guest-side only, host stays healthy. ## Simulator workstream (risc-v-gsp-rm-debugger) @@ -135,6 +141,9 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet | +| MS-20261004-02 | Latent bridge pending-slot oops found, fixed; reboot | 2026-10-04 ~14:45-15:00 | `cubb_pend_release` slot reuse -> NULL deref; fixed `85539844...`; approved reboot restored the GPU | +| MS-20261004-03 | Post-reboot acceptance run blocked by guest boot panic | 2026-10-04 15:03 | `noclaim32` `accepted: 0`, clean teardown; fixed module's deep path unverified | | MS-20261003-01 | Guest reaches first `cudaMalloc`; emulation pipeline live | 2026-10-03 ~06:10 | 101+ GR pushes, 219 GR releases, ~165k publishes, 88 forwarded works; hang in UVM tracker pool-sem VA->PA | | MS-20261003-02 | Hardware incident (Xid 140) + reboot recovery | 2026-10-03 13:47 / ~20:11 | no-claim BAR2 exposure faulted the GPU; reboot restored it; fix in flight | | MS-20261003-03 | Large-push parser wall fixed; UVM fully engaged | 2026-10-03 ~00:10 | `NON_INC` count>64 / zero-padding parser fix; `verify_ok=62` forwarded guest copies | @@ -162,6 +171,31 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261004-01` **The no-claim bridge reaches the guest `vectorAdd` launch + (2026-10-04 ~14:45).** The emulation-only campaign fixed three defects in one + window: the guest-truth UVM kernel probes were never installed, so the + GPU-only pool/push-buffer translations fell back to content heuristics (with + the probes the pushes decode with no fakes and the bridge completes); the QMD + accumulator's address window swallowed the copy-class DtoH operands, so the + read-back never ran; and at 256 MiB the guest had too little free FB. After + the fixes the guest prints its full acceptance string, the DtoH mail read-back + works, `cudaMalloc` is unblocked by a 512 MiB emulated FB, and the client + reaches the `vadd` launch (`grid=4096,1,1 ... n=1048576`). Not yet green: no + `STATUS OK`/`bad=0` captured. +- `MS-20261004-02` **Latent bridge pending-slot defect found, fixed, and reboot + (2026-10-04 ~14:45-15:00).** A freed bridge pending slot was reused with a + stale `have_done`/`done_out_len`, so the next pass copied from a NULL blob and + dereferenced NULL in the sema service; the fix clears the done flags on release + and on staging plus a defensive guard. The oops had killed the service thread + holding its lock, so `rmmod` failed and an approved reboot (~15:00) restored the + host; the GPU is healthy and `nvidia-smi -L` is green. +- `MS-20261004-03` **Post-reboot acceptance run blocked by a guest boot panic + (2026-10-04 15:03).** The first post-reboot no-claim run (`noclaim32`) loaded + the fixed module, created an mdev and booted the guest, but the guest kernel + panicked ~11 s into boot (a NULL-in-interrupt/swap class), so the client could + not be pushed (`accepted: 0`). Teardown was clean (`rmmod rc=0`, nvidia + rebound, pstore empty); the fixed module's deep launch/vadd path is still + unexercised. - `MS-20261003-01` **The emulation pipeline is live and the guest reaches its first `cudaMalloc` (2026-10-03 ~00:10-06:10).** The method-stream parser that could never validate UVM's large page-table pushes (`NON_INC` methods with count > 64, |