diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 114 |
1 files changed, 109 insertions, 5 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 17aab60..7f21f85 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,7 +1,7 @@ # Open-IOV Current State > **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, -> physical/guest campaign data as of 2026-10-05 ~12:16; debugging-options +> physical/guest campaign data as of 2026-10-05 ~15:05; debugging-options > catalogue added 2026-10-05).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -42,7 +42,24 @@ isolation, performance, portability). plus a cache flush/PT refill, the GR push is now fetched (`GPGet=1`), but the GR engine faults during context switch (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), so no release lands (`-110`); 10 - clean cycles, no wedge, no Xid. + clean cycles, no wedge, no Xid. **Update 2026-10-05 ~14:50:** the rung-2 + campaign decompiled the commit chain end-to-end (`PROMOTE_CTX` handler + `0x526a530` -> engine-class commit `0x5350078` -> per-entry `0x5290a48` -> + per-buffer commit `0x5b00d2c` -> engine-ctx memdesc install `0x514cab8` -> + record/instance/RAMFC/engine HAL); the fixed `GR_GET_CTX_BUFFER_INFO` ABI + (80-byte entries, params 5136) returns the server's 8 buffers with + MAIN/PATCH geometry visible; the server-managed engine-ctx `vaList` is + pre-booked, so the UVM-shape VA-only promote is structurally blocked + (`vaListSetManaged(NV_FALSE)` -> `NV_ERR_INVALID_STATE` 0x40), `EVICT_CTX` + does not release it, and PA-only entries are accepted but never reach the + commit; V6 (`SKIP_CTXBUFFER_ALLOC`) is a dead end (the `TURING_COMPUTE_A` + construct gets no context memdesc -> -5). The one remaining shape is the + `hVirtMemory` no-entry promote with a physical FBMEM client memdesc + (mode 23), which bypasses the booked `vaList` and hands the memdesc to + `0x5b00d2c`; it was built (`3bf3e2b2...`, uncommitted) and run + (`grv23`/`grv23v6`): `PROMOTE_CTX hVirtMemory ... status=0x1f`, a soft-stop + before BIND, clean release, no wedge. Next: mode 24 (`GPU_INITIALIZE_CTX` + with the same FBMEM handle, then the mode-23 promote). - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion @@ -73,7 +90,15 @@ isolation, performance, portability). path; the experiment was saved as a patch, reverted, and a reboot restored the host. Per the 2026-10-05 directive, priority is now **native guest execution without the host CUDA bridge**, so this front is a stability - exercise rather than the destination. + exercise rather than the destination. **Update 2026-10-05 ~13:10:** the + per-type emulated-VRAM fix (`c527727a`) is validated live + (`nvidia-vgpu-turing` = `framebuffer=512M`/`max_instance=2`, + `nvidia-vgpu-turing-1g` = `framebuffer=1024M`; the guest boots to + `DEVICE ...` with ~347 MiB free of the 512 MiB FB); the follow-up + acceptance run (`reb1`) hit the known stall class (`held=4300 starved=1 + ring_auto=6 nomap=4300 faked=0`, the client stuck after `DEVICE`), not a + VRAM regression - zero Xid, clean teardown. The campaign stays demoted per + the 10:50 directive. - **Hardware incident + recovery (2026-10-03).** The first emulation-only (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the host driver owned the device and triggered an uncorrectable ECC Xid, after @@ -81,7 +106,13 @@ isolation, performance, portability). The host recovered after a reboot (~20:11); `nvidia-smi -L` is green and the device is healthy. The fix (shadow BAR2 in emulation-only mode) is committed (in `cf6ff542`) and has held across the acceptance campaigns since - no Xid - observed. + observed. **2026-10-05 midday:** a second Xid 140 fired ~12:17 during the + first no-claim acceptance run of that boot (after 13 clean GR cycles; the + BAR2 shadow held, so the trigger is under review), leaving + `RmInitAdapter` failed; a manual reboot recovered the device. A second + reboot was needed after the 13:40 external-VAS VA-promote wedge (GSP-RM + stopped replying, WPR2 up); both reboots were operator-executed and fully + recovered (`nvidia-smi -L` green, no Xid since). - **Simulator workstream:** see the section below (separate, development-only). **Blockers (current walls).** @@ -118,6 +149,21 @@ isolation, performance, portability). context switch (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), so no release lands (`-110`). Next: root-cause the ctxsw fault (engine-context buffer init, subcontext header, or a VA+PA promote); 10 clean cycles, no wedge. + **Update 2026-10-05 ~14:50:** the wall is now one promote/commit shape. + The rung-2 campaign decompiled the full commit chain (`0x526a530` -> + `0x5350078` -> `0x5290a48` -> `0x5b00d2c` -> `0x514cab8` -> record/ + instance/RAMFC/engine HAL) and fixed `GR_GET_CTX_BUFFER_INFO` (80-byte + entry, params 5136; the server's 8 buffers now visible). On `srvctx=1` + the server pre-books MAIN in the engine-ctx `vaList`, so the UVM-shape + VA-only promote returns 0x40 at `vaListSetManaged(NV_FALSE)` and + `EVICT_CTX` does not release it; PA-only is accepted but never commits; + V6 is a dead end; an external-VAS VA promote wedged GSP-RM (recovered by + reboot). Mode 23 (the `hVirtMemory` no-entry promote with a physical + FBMEM client memdesc - the one shape that bypasses the booked `vaList`) + was built and run (`grv23`/`grv23v6`): soft-stop at `PROMOTE_CTX + hVirtMemory ... status=0x1f`, clean release, no wedge. Next: mode 24 = + `GPU_INITIALIZE_CTX` with the same FBMEM handle, then the mode-23 + promote. 2. **Hardening the accepted no-claim path.** The first green guest CUDA acceptance has landed and is holding; the residual classes are (a) corruption via a late first decode behind pump backlog (~3-5% of runs; the experiment @@ -316,6 +362,8 @@ chain succeeds. | ID | Milestone | When | State | |---|---|---|---| +| MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | +| MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | | MS-20261005-02 | GR promote wall crossed on the shared-VAS+srvctx path; new wall = GR ctxsw fault after push fetch | 2026-10-05 ~12:16 | srvctx registry (`RMSetClientRMAllocatedCtxBuffer=0`, `RMSplitVasMgmtServerClientRm=0`) + V5 + mode 17 + client page-table install/cache flush; stage 3 completes (`hGr`, clean release, no wedge); stage 4 fetches (`GPGet=1`) but GR faults during ctxsw (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), no release, `-110`; 10 clean cycles | | MS-20261005-01 | Hardening the accepted no-claim path (hazard classes closed; rates up) | 2026-10-05 overnight | per-op once bitmap + epoch/content-hash gate + grace 15 s (`b645cf6b`, `3b094c2d`, `c3cd4631`); first-attempt 86% -> ~94-95%; fakes/run 3.39 -> 0.61; hold-all-unreadable/25 s experiment committed (`4115f1dd`, module `e7ff3bf7`), campaign `239..262` validating (`239-242` green; `241a`/`243a` infra retries; `243b` live) | | MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign | @@ -349,6 +397,51 @@ chain succeeds. ## Milestone log - current frontier and recent milestones (granular IDs in each entry) +- `MS-20261005-04` **GR rung 2: the commit chain is decompiled and the promote + shape space is closed to one remaining form (2026-10-05 13:00-14:48).** After + stage 3 first completed on the shared-VAS+srvctx path, the campaign ran six + supervised cycles (`grctx1-6`) that (1) fixed the `GR_GET_CTX_BUFFER_INFO` + ABI - the entry is 80 B (uuid at 60, flags 56), `paramSize=5136`; the query + now returns the server's 8 context buffers (MAIN `0x133000` FBMEM, PATCH + `0x5000`, plus type 3/4/5/6/10/11) with status 0; (2) decompiled the real + promote commit chain (`PROMOTE_CTX` handler `0x526a530` -> engine-class + commit `0x5350078` -> per-entry `0x5290a48` -> per-buffer commit + `0x5b00d2c` -> engine-ctx memdesc install `0x514cab8` -> record/instance/ + RAMFC/engine HAL, the `channelCommitEngineContext_HAL` analogue); (3) + proved the UVM-shape VA-only promote returns 0x40 at + `vaListSetManaged(NV_FALSE)` (`0x52915e8`) because the server pre-books + MAIN in the engine-ctx `vaList` when it manages the ctx buffers, and + `EVICT_CTX` does not release that booking; PA-only entries are accepted + (status 0) but never take the commit branch, so RAMFC stays zero and no + engine-context pointer appears. The external-VAS VA-promote variant + wedged GSP-RM (`-110`, WPR2 up) and needed an operator reboot (the exact + shape must not be repeated). Post-reboot: V6 (`SKIP_CTXBUFFER_ALLOC`) is a + dead end (the `TURING_COMPUTE_A` construct fails -5 without a ctx + memdesc), and the registry semantics were resolved from source (default = + client-managed; `srvctx=1` = server-managed). The one remaining shape is + the `hVirtMemory` no-entry promote with a physical FBMEM client memdesc + (mode 23), which bypasses the booked `vaList`; it was implemented + (uncommitted, module `3bf3e2b2...`) and run twice (`grv23` V5+srvctx and + `grv23v6`): `PROMOTE_CTX hVirtMemory ... status=0x1f` at stage 3, a + soft-stop before BIND, clean release each time, no wedge, `nvidia-smi` + green. Next: mode 24 = `GPU_INITIALIZE_CTX` with the same FBMEM handle, + then the mode-23 promote. Evidence: + `backups/evidence-20261005-grctx/` (`REPORT-GR-PROMOTE-COMMIT.md` + + Addendum A, `REPORT-GR-CTX-SOURCE.md`, `REPORT-GR-EVIDENCE-DECODE.md`, + `REPORT-GR-FORWARDING-DESIGN.md`), `backups/evidence-20261002-gr/ + gr-run-grctx{1..6}.log`, `gr-run-grv23.log`, `gr-run-grv23v6.log`. +- `MS-20261005-03` **Per-type emulated VRAM (1.0) validated live (2026-10-05 + ~13:05).** The per-mdev emulated-VRAM fix (`c527727a`) resolves the earlier + multi-size regression (a `sysfs_name` prefix strip plus a non-canonical FB + encoder): `nvidia-vgpu-turing` reports `framebuffer=512M`, + `max_instance=2`, and the new `nvidia-vgpu-turing-1g` + `framebuffer=1024M`; a guest on the 512 MiB type boots to `DEVICE NVIDIA + GeForce RTX 2080 SUPER cc=7.5 sm=48` with ~347 MiB free FB. The follow-up + no-claim acceptance run (`reb1`) then hit the known emulation-front stall + class (`held=4300 starved=1 ring_auto=6 nomap=4300 faked=0`, the client + stuck after `DEVICE`), explicitly not a VRAM regression: zero Xid, clean + teardown, `nvidia-smi` green. The no-claim campaign stays demoted per the + 2026-10-05 directive; multi-mdev (two available instances) is parked. - `MS-20261005-02` **GR promote wall crossed on the shared-management path (2026-10-05 ~12:00).** The untested GR-ladder combination - shared-management VAS, the srvctx pre-boot registry (`RMSetClientRMAllocatedCtxBuffer=0`), @@ -1025,7 +1118,18 @@ chain succeeds. `30dfd311...`, P2 `69eac176...`, G0 `dde36d92...`, P5 wave 1 `83952506...`. The host was rebooted 2026-10-03 ~20:11 (previous boot `12cf3a41-...`); the module was rebuilt 21:10:57 and reloaded 21:12 with the post-incident - no-claim fixes. + no-claim fixes. **2026-10-05 onward:** acceptance builds `e7ff3bf7` (hold + all unreadable) and `cb1724d3` (per-type VRAM); GR rung-2 builds + `19d1e0fc`, `1783dade`, `c998abef`, `b6a9d4eb`, `9075bf57`, `69500d6e` + (grctx1-6), `c5c3f396` (grv6a), and `3bf3e2b2...` (mode 23, uncommitted). + GR reports: `backups/evidence-20261005-grctx/REPORT-GR-PROMOTE-COMMIT.md` + (+ Addendum A), `REPORT-GR-CTX-SOURCE.md`, `REPORT-GR-EVIDENCE-DECODE.md`, + `REPORT-GR-FORWARDING-DESIGN.md`, `NEXT-BOOT-GR-RUNBOOK.md`; run logs + `backups/evidence-20261002-gr/gr-run-grctx{1..6}.log`, `gr-run-grv6a.log`, + `gr-run-grv23.log`, `gr-run-grv23v6.log` with the `06/20/21/30/31/32-*` + capture sets. HEAD `449c53bd` is in sync with `vgpu/master`; the working + tree carries the uncommitted mode-23 change (`gsp_phys_booter.c`, + `gsp_phys_gpu.c`). - **Campaign detail:** the private live campaign narrative (handoff/logs, maintained outside this repository) is authoritative; this page is the dated public snapshot. |