summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-04 18:42:32 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-04 18:42:32 -0400
commit8e05785634f546f8738672d067e9aae1c8ad060a (patch)
treea2ba8e137bf43b4ad0bc9c286db7adb1c43dbce1
parent31594b263cc4fbe30ebbe412a4a788f726f03a02 (diff)
downloadopen-iov-8e05785634f546f8738672d067e9aae1c8ad060a.tar.gz
open-iov-8e05785634f546f8738672d067e9aae1c8ad060a.zip
STATE-CURRENT: 18:45 check-back snapshot (first green end-to-end guest CUDA acceptance; 14/15 repeat cycles green)
Refresh the public status page for the 2026-10-04 acceptance: noclaim38/39/40 report MAIL + VADD n=1048576 bad=0 checksum=392344090.500000 + STATUS OK with identical hashes (committed 31594b26); the repeat campaign shows 14 of 15 green with the remaining failure the known intermittent guest-side flake. Update the milestone ladder (CUDA kernel execution now green via the no-claim host bridge; guest engine forwarding to the physical GPU still the open backend wall) and the milestone index/log.
-rw-r--r--STATE-CURRENT.md68
1 files changed, 46 insertions, 22 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 7f7e335..93ae38f 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,7 +1,7 @@
# Open-IOV Current State
> **Snapshot: 2026-10-04 (reorganized; simulator status current 2026-10-04,
-> physical/guest campaign data as of 2026-10-04 ~15:12).** This is a dated public
+> physical/guest campaign data as of 2026-10-04 ~18:45).** This is a dated public
> status page, not live state; it is refreshed on every progress check. Live
> working notes are maintained outside this repository; the public
> known-knowns/unknowns document is authoritative here:
@@ -31,19 +31,22 @@ isolation, performance, portability).
and relay paths (all parameters default off; one relayed copy's data
mismatch is under triage), and the host GR ladder allocates a GR0 channel +
compute object.
-- **Guest frontier (2026-10-04 ~15:03).** The emulation-only ("no-claim")
- campaign advanced the guest CUDA program past its first `cudaMalloc` to the
- `vectorAdd` launch. Guest-truth kernel probes now back the pool/push-buffer
- translations (pushes decode with no fakes), the device-to-host mail read-back
- succeeds (the host daemon completes the marshalled job, `CUBB2 status=ok`),
- and raising the emulated FB from 256 MiB to 512 MiB unblocked `cudaMalloc`, so
- the guest reaches `launch ... op=vadd grid=4096,1,1 ... n=1048576`. A latent
- bridge pending-slot defect (a freed slot reused with a stale done-length) was
- found and fixed after it oopsed the service thread and forced an approved
- reboot (~15:00). The first **green** guest `vectorAdd` (`STATUS OK`, `bad=0`)
- has **not** yet been captured: the only post-reboot acceptance run was blocked
- by a guest-boot kernel panic before the client started. The host physical/GR
- wall is unchanged.
+- **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA (2026-10-04 ~16:20).**
+ The emulation-only ("no-claim") campaign now runs an unmodified guest CUDA
+ program to completion with verified results: `MAIL the little gpu man says
+ you've got mail`, `VADD n=1048576 bad=0 checksum=392344090.500000`,
+ `STATUS OK`. Three consecutive supervised cycles (`noclaim38/39/40`)
+ accepted, deterministic (identical IN/OUT hashes; `noclaim39/40` on the same
+ module `582bf1da...`), pixel-verified by the host bridge daemon
+ (`CUBB2 seq=2 op=2 ... status=ok bad=0`), and committed (`cf6ff542`). Two
+ data-path defects were root-caused to get there: a replayed GPFIFO entry
+ re-executed its CE copies (fixed with per-slot raw-entry identity - replays
+ re-apply releases but never re-run operations) and a multi-page virtual
+ operand was copied linearly from its first page (fixed with page-granular
+ re-resolution). A repeat-run stability campaign since is at 14 of 15 green
+ (`noclaim42`-`55` OK; `noclaim41` failed on the guest-side flake); `noclaim56`
+ live at check. The vadd executes via the host CUDA bridge daemon; forwarding
+ guest engine work to the physical GPU remains the open backend wall.
- **Hardware incident + recovery (2026-10-03).** The first emulation-only
(no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the
host driver owned the device and triggered an uncorrectable ECC Xid, after
@@ -64,12 +67,14 @@ isolation, performance, portability).
`NV_ERR_INVALID_STATE` (0x40) each. The V5 commit assert `0x52a8b04` is a
secondary failure. Next routes: name the soft-0x40 check, root-cause the
V5 first failure, or rehost the promote flow in risc-v-gsp-rm-debugger.
-2. **First green guest acceptance (no-claim path).** The no-claim bridge now
- reaches the `vectorAdd` launch, but no `STATUS OK` has been captured because a
- guest-boot kernel panic blocked the only post-reboot run; a clean guest boot on
- the fixed module is the immediate step. Guest-truth kernel probes (installed
- per guest boot) are load-bearing for the GPU-only pool/push-buffer
- translations; without them the host falls back to content heuristics.
+2. **Hardening the accepted no-claim path.** The first green guest CUDA
+ acceptance has landed and is holding across repeats; what remains is the
+ intermittent guest-side corruption flake (intermittent guest boot
+ double-faults/UVM teardown oopses; `noclaim32`/`41`; host unaffected),
+ trimming/re-running with the bounded diagnostics, and extending acceptance
+ to a second workload. Guest-truth kernel probes (installed per guest boot)
+ are load-bearing for the GPU-only pool/push-buffer translations; without
+ them the host falls back to content heuristics.
3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side
data compare; handled fail-closed, parameter default-off).
4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect
@@ -126,8 +131,8 @@ meant to execute on the physical GPU.
| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest |
-| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend |
-| CUDA kernel execution (`vectorAdd`) | 🟡 **In progress - the physical backend is live** (full boot + RM objects + a real CE copy + guest CE forwarding); the current wall is the GR context/execution path and the guest's first device allocation. The first green guest CUDA execution is the immediate goal; the chronological detail is in the milestone log below and the pre-2026-10-02 mega-summary is preserved in the archive. |
+| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime initializes end-to-end with **real releases** and the launch now executes through the host bridge (see next row); a full `nvidia-smi` (per-process/utilization) remains unverified |
+| CUDA kernel execution (`vectorAdd`) | ✅ **First green end-to-end guest CUDA (2026-10-04 ~16:20, no-claim host bridge):** `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, three consecutive deterministic supervised cycles, committed `cf6ff542`. The vadd runs through the host CUDA bridge daemon; forwarding guest engine work to the real GPU is the remaining backend path. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
@@ -141,6 +146,7 @@ chain succeeds.
| ID | Milestone | When | State |
|---|---|---|---|
+| MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign |
| MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet |
| MS-20261004-02 | Latent bridge pending-slot oops found, fixed; reboot | 2026-10-04 ~14:45-15:00 | `cubb_pend_release` slot reuse -> NULL deref; fixed `85539844...`; approved reboot restored the GPU |
| MS-20261004-03 | Post-reboot acceptance run blocked by guest boot panic | 2026-10-04 15:03 | `noclaim32` `accepted: 0`, clean teardown; fixed module's deep path unverified |
@@ -171,6 +177,24 @@ chain succeeds.
## Milestone log - current frontier and recent milestones (granular IDs in each entry)
+- `MS-20261004-04` **First green end-to-end guest CUDA acceptance (2026-10-04
+ ~16:20).** After `noclaim32` (guest double-fault) and `noclaim33` (vadd data
+ corrupt), two data-path defects were root-caused with bounded
+ instrumentation and fixed: (1) a replayed GPFIFO entry re-executed its CE
+ copies - the pump rewinds `last_get` on each injected-translation batch and
+ the replay suppression missed some, so a replayed multi-MB HtoD re-copied a
+ staging buffer the client had already refilled; fixed with per-slot
+ raw-entry identity so a matching replay re-applies releases but never
+ re-runs operations; (2) `svc_copy` copied a multi-page virtual operand
+ linearly from its first page while client staging ranges are split; fixed
+ with page-granular re-resolution for virtual operands. Then `noclaim38`
+ reported `INCHECK/OUTCHECK bad=0` and the guest printed
+ `VADD n=1048576 bad=0 checksum=392344090.500000` -> `STATUS OK`;
+ `noclaim39/40` repeated green on the same module (`582bf1da...`) with
+ identical hashes. A repeat-run stability campaign since is at 14 of 15 green
+ (one guest-side flake failure). The vadd executes through the host CUDA
+ bridge daemon; forwarding guest engine work to the physical GPU remains the
+ backend wall.
- `MS-20261004-01` **The no-claim bridge reaches the guest `vectorAdd` launch
(2026-10-04 ~14:45).** The emulation-only campaign fixed three defects in one
window: the guest-truth UVM kernel probes were never installed, so the