diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 68 |
1 files changed, 46 insertions, 22 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 7f7e335..93ae38f 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,7 @@ # Open-IOV Current State > **Snapshot: 2026-10-04 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-04 ~15:12).** This is a dated public +> physical/guest campaign data as of 2026-10-04 ~18:45).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: @@ -31,19 +31,22 @@ isolation, performance, portability). and relay paths (all parameters default off; one relayed copy's data mismatch is under triage), and the host GR ladder allocates a GR0 channel + compute object. -- **Guest frontier (2026-10-04 ~15:03).** The emulation-only ("no-claim") - campaign advanced the guest CUDA program past its first `cudaMalloc` to the - `vectorAdd` launch. Guest-truth kernel probes now back the pool/push-buffer - translations (pushes decode with no fakes), the device-to-host mail read-back - succeeds (the host daemon completes the marshalled job, `CUBB2 status=ok`), - and raising the emulated FB from 256 MiB to 512 MiB unblocked `cudaMalloc`, so - the guest reaches `launch ... op=vadd grid=4096,1,1 ... n=1048576`. A latent - bridge pending-slot defect (a freed slot reused with a stale done-length) was - found and fixed after it oopsed the service thread and forced an approved - reboot (~15:00). The first **green** guest `vectorAdd` (`STATUS OK`, `bad=0`) - has **not** yet been captured: the only post-reboot acceptance run was blocked - by a guest-boot kernel panic before the client started. The host physical/GR - wall is unchanged. +- **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA (2026-10-04 ~16:20).** + The emulation-only ("no-claim") campaign now runs an unmodified guest CUDA + program to completion with verified results: `MAIL the little gpu man says + you've got mail`, `VADD n=1048576 bad=0 checksum=392344090.500000`, + `STATUS OK`. Three consecutive supervised cycles (`noclaim38/39/40`) + accepted, deterministic (identical IN/OUT hashes; `noclaim39/40` on the same + module `582bf1da...`), pixel-verified by the host bridge daemon + (`CUBB2 seq=2 op=2 ... status=ok bad=0`), and committed (`cf6ff542`). Two + data-path defects were root-caused to get there: a replayed GPFIFO entry + re-executed its CE copies (fixed with per-slot raw-entry identity - replays + re-apply releases but never re-run operations) and a multi-page virtual + operand was copied linearly from its first page (fixed with page-granular + re-resolution). A repeat-run stability campaign since is at 14 of 15 green + (`noclaim42`-`55` OK; `noclaim41` failed on the guest-side flake); `noclaim56` + live at check. The vadd executes via the host CUDA bridge daemon; forwarding + guest engine work to the physical GPU remains the open backend wall. - **Hardware incident + recovery (2026-10-03).** The first emulation-only (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the host driver owned the device and triggered an uncorrectable ECC Xid, after @@ -64,12 +67,14 @@ isolation, performance, portability). `NV_ERR_INVALID_STATE` (0x40) each. The V5 commit assert `0x52a8b04` is a secondary failure. Next routes: name the soft-0x40 check, root-cause the V5 first failure, or rehost the promote flow in risc-v-gsp-rm-debugger. -2. **First green guest acceptance (no-claim path).** The no-claim bridge now - reaches the `vectorAdd` launch, but no `STATUS OK` has been captured because a - guest-boot kernel panic blocked the only post-reboot run; a clean guest boot on - the fixed module is the immediate step. Guest-truth kernel probes (installed - per guest boot) are load-bearing for the GPU-only pool/push-buffer - translations; without them the host falls back to content heuristics. +2. **Hardening the accepted no-claim path.** The first green guest CUDA + acceptance has landed and is holding across repeats; what remains is the + intermittent guest-side corruption flake (intermittent guest boot + double-faults/UVM teardown oopses; `noclaim32`/`41`; host unaffected), + trimming/re-running with the bounded diagnostics, and extending acceptance + to a second workload. Guest-truth kernel probes (installed per guest boot) + are load-bearing for the GPU-only pool/push-buffer translations; without + them the host falls back to content heuristics. 3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side data compare; handled fail-closed, parameter default-off). 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect @@ -126,8 +131,8 @@ meant to execute on the physical GPU. | CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | | `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | -| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | -| CUDA kernel execution (`vectorAdd`) | 🟡 **In progress - the physical backend is live** (full boot + RM objects + a real CE copy + guest CE forwarding); the current wall is the GR context/execution path and the guest's first device allocation. The first green guest CUDA execution is the immediate goal; the chronological detail is in the milestone log below and the pre-2026-10-02 mega-summary is preserved in the archive. | +| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime initializes end-to-end with **real releases** and the launch now executes through the host bridge (see next row); a full `nvidia-smi` (per-process/utilization) remains unverified | +| CUDA kernel execution (`vectorAdd`) | ✅ **First green end-to-end guest CUDA (2026-10-04 ~16:20, no-claim host bridge):** `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, three consecutive deterministic supervised cycles, committed `cf6ff542`. The vadd runs through the host CUDA bridge daemon; forwarding guest engine work to the real GPU is the remaining backend path. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -141,6 +146,7 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign | | MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet | | MS-20261004-02 | Latent bridge pending-slot oops found, fixed; reboot | 2026-10-04 ~14:45-15:00 | `cubb_pend_release` slot reuse -> NULL deref; fixed `85539844...`; approved reboot restored the GPU | | MS-20261004-03 | Post-reboot acceptance run blocked by guest boot panic | 2026-10-04 15:03 | `noclaim32` `accepted: 0`, clean teardown; fixed module's deep path unverified | @@ -171,6 +177,24 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261004-04` **First green end-to-end guest CUDA acceptance (2026-10-04 + ~16:20).** After `noclaim32` (guest double-fault) and `noclaim33` (vadd data + corrupt), two data-path defects were root-caused with bounded + instrumentation and fixed: (1) a replayed GPFIFO entry re-executed its CE + copies - the pump rewinds `last_get` on each injected-translation batch and + the replay suppression missed some, so a replayed multi-MB HtoD re-copied a + staging buffer the client had already refilled; fixed with per-slot + raw-entry identity so a matching replay re-applies releases but never + re-runs operations; (2) `svc_copy` copied a multi-page virtual operand + linearly from its first page while client staging ranges are split; fixed + with page-granular re-resolution for virtual operands. Then `noclaim38` + reported `INCHECK/OUTCHECK bad=0` and the guest printed + `VADD n=1048576 bad=0 checksum=392344090.500000` -> `STATUS OK`; + `noclaim39/40` repeated green on the same module (`582bf1da...`) with + identical hashes. A repeat-run stability campaign since is at 14 of 15 green + (one guest-side flake failure). The vadd executes through the host CUDA + bridge daemon; forwarding guest engine work to the physical GPU remains the + backend wall. - `MS-20261004-01` **The no-claim bridge reaches the guest `vectorAdd` launch (2026-10-04 ~14:45).** The emulation-only campaign fixed three defects in one window: the guest-truth UVM kernel probes were never installed, so the |