diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 101 |
1 files changed, 97 insertions, 4 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index c16b774..d26ffec 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,8 +1,9 @@ # Open-IOV Current State -> **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-07 ~01:15; debugging-options -> catalogue added 2026-10-05).** This is a dated public +> **Snapshot: 2026-10-14 ~13:30 (physical/guest campaign data current to the +> live r139 cycle at snapshot time; simulator workstream status 2026-10-03; +> page reorganized and debugging-options catalogue added 2026-10-05).** +> This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: @@ -146,6 +147,92 @@ isolation, performance, portability). aperture. Ranked next: read the server-FBMEM fault buffer via a copy-engine transfer, per-VEID/compute-context execution requirements, and the scheduler CWD field semantics. Host clean, GPU green, no reboots. + **Update 2026-10-14 ~13:30 (major - guest compute runs on the physical + engine):** the physical-backed guest campaign now runs a multi-client + CUDA battery against the real TU104 with the host CUDA bridge disabled + (`cuda_bridge=0`, GR forwarding on). In one cycle (r137) the first client + (`victory_mail`: `MAIL` + `VADD bad=0` + `STATUS OK`) and the second + client (`smiley`: `STATUS OK`) are both green on the physical backend, + and the guest's compute launches execute on the physical GR engine (the + window-attribution defects where kernels wrote into the wrong window are + fixed; first-client guest VADD results verify). The 2026-10-08..12 + second-client campaign closed its chain (CE USERD location, CE + push-buffer pin delivery, a channel-anchored CE pool resolver + `ade77d9a`, session-flush pin semantics, data-argument resolution + order). Wall closure since 2026-10-07: (a) the third-client + (`victory_matmul`) PMA-scrub freeze is root-caused live - the + boot-session scrubber channel's finish word is a guest-RAM vmap that + collides with a client VA - and fixed by authoring the scrubber + channel's progress (`624438bb` + guest scanner `80af379d`), so + `victory_matmul` gets past `cudaMalloc`; (b) the intermittent + GR-forward launch timeout is root-caused: the host publishes forwarded + launches modulo an 8-entry ladder ring while the user-lane channel was + declared with 1024 GPFIFO entries, so after the first host-ring wrap the + front-end's GET/PUT became inconsistent and it stopped fetching + (`074f493c` fixes the lane depth to 8; the earlier two-client acceptance + never wrapped and had masked the regression). The r139 cycle validated + the ring fix: the 100-launch soak crossed the host-ring wrap cleanly + (`LOOP launches=100 bad=0 STATUS OK`; the previous cycle died at launch + 8), c1 `victory_mail` is green (60 s), and `smiley` green on a later + retry. The residual is an intermittent cold-context/fetched-not-executed + launch race (18 failed launches in the cycle: 6 never fetched by the + front-end, 12 fetched but not completed by the engine; some later + clients intermittently return all-zero results), for which a bounded + in-submit retry with a fresh ring entry plus a cold pre-warm is being + implemented; the "OK with 0 changed" change-counter is not a reliable + execution witness - the client's own `STATUS` is the acceptance signal. + Host clean all session, no Xid in the campaign logs, pstore empty, no + reboots (in-session PCI function-level reset recovery where needed). + ISSUE-0002 (second client in one boot): the sequential case can pass + (first two clients green in one cycle; smiley green on retry in the + next) but is not yet deterministic; the issue remains open pending + repeat-run plus concurrent/simultaneous validation. Active front: + the multi-client battery on the physical backend plus launch-latency + robustness. + **Update 2026-10-14 ~15:40 (afternoon wave, r140-r143):** the full fix + set is committed and pushed (HEAD `17393a74`: ring depth `074f493c`, + scrubber `624438bb`/`80af379d`, bounded fresh-entry launch retry + `6b090f70`, 208-page budgeted data window `fce5ab8d`/`4a23c196`). r142: + c1 `victory_mail` and c2 `smiley` GREEN; c3 `victory_matmul` FAIL + (all-wrong operands - the three 256 KB operands resolved only ~32 of the + needed 64+ pages each); launch counters `ok=86 failed=10 retries=4` + (the retry recovers a subset of the cold-context timeouts silently); + then `victory_saxpy`/`memcpy_roundtrip` stalled and the guest SSH wedged + - the intermittent stall/wedge class survives. r143 (live at this + check, same fix set): c1 green within ~2 min; c3 matmul launched and + improved to `bad=8192` of 65536 with a non-zero checksum (operand + resolution now 68/64/64 pages) - STATUS FAIL. **Root cause found + (sub-agent, evidence-grounded): the win0 QMD shadow pool base was still + `0x190000` (main-PT slots 0x190..0x197), which lies INSIDE the enlarged + 208-page bounce window (`0x110000..0x1df000`); the win0 plant runs on + every `pdb_build()` while the bounce plant is one-shot, so win0 stole the + PTEs for bounce pages 128..135 = C rows 0..31 = exactly `bad=8192`. Fix + landed (unvalidated live at pause): `GSP_PHYS_RMCALL_VA_GR_SHADOW0_BASE` + moved to `0x1e0000` (slots 0x1e0..0x1e7, above the mail/marker pages, + below the 0x200000 guest window) plus a `static_assert` invariant and + comment/log corrections.** Ranked next blockers: (1) the 256 KB operand + data-path placement; (2) the intermittent client stall/wedge class + (the top reliability blocker; needs the stalled-client capture); + (3) the cold-context launch race (cold GR-sem pre-warm designed); + (4) then multi-mdev, two guests, VRAM sizes and the concurrent/full + soak. Host clean, GPU green, no reboots. + **Update 2026-10-15 ~18:15 (resume + parallel workstreams; pause snapshot):** + the host rebooted (the OS moved the kernel to 6.8.0-147; the module was rebuilt + for it). The r155 cycle on the physical backend: ladder green; `victory_mail` + (`VADD bad=0` + `STATUS OK`) and `victory_matmul` (`MATMUL n=256 bad=0` + + `STATUS OK`) both green end-to-end, and a 100-launch `vecadd_loop` passed; the + sequential battery then hit the known acked-no-write class on `streams2` + (all-zero output despite `qmd_rel=0x6` completion acks -> `STATUS FAIL`). Three + offline workstreams landed as pushed branches: (a) `bigpath` - a 64 MiB + boot-fixed data-bounce region (stage-3-planted PTEs) with fail-closed cap + semantics, unblocking large operands (victory_math 512 KB, vector_add 4 MB, + memcpy_roundtrip 8 MB); (b) `dual` - per-mdev separation (per-guest channel + keys, session records, guest-RAM records, forward-request ownership) for two + simultaneous guests; (c) `lottery` - analysis proving the completion ack is not + an execution witness, rejecting naive fail-closed (idempotent-client false + positives), and ranking the next experiments (win0 bank rotation, qmdwin + double-buffer, warm-sem retry, executed-vs-wedged discriminator). Host clean, + GPU green, VM left live; user-requested poweroff at pause. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -265,6 +352,12 @@ isolation, performance, portability). the full FWCLIENT client-managed construct (client ctx buffers + VAS mapping + reference promote set), then the server commit. See ADDENDUM 53/54. + **Update 2026-10-14:** this wall moved decisively - the physical GR + engine now executes forwarded guest compute launches, and the first two + battery clients are green end-to-end on the physical backend (host CUDA + bridge disabled). The remaining items are launch-latency robustness (a + first launch after an idle can time out and is auto-retried to success) + and the third-client chain; see the status update above. 2. **Hardening the accepted no-claim path.** The first green guest CUDA acceptance has landed and is holding; the residual classes are (a) corruption via a late first decode behind pump backlog (~3-5% of runs; the experiment @@ -1583,7 +1676,7 @@ progress.** ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. -- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages +- Host OS: Ubuntu, kernel 6.8.0-146; the distribution NVIDIA driver manages the physical GPU; `vgsp.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. |