From 0db92fe87b814af0ad60a6929c8210640a9fe6b9 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 14:12:01 -0400 Subject: STATE-CURRENT: hourly check-back snapshot (2026-10-02 14:12:01) --- STATE-CURRENT.md | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 69f5574..87fadf6 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -13,7 +13,7 @@ Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality (CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without proprietary GRID licenses - via host-side GSP emulation in -`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side +`vgsp.ko`. The guest stays stock: no guest patches, no guest-side Open-IOV code. Architecture stance: *emulate the interface, pass the work through to real @@ -81,25 +81,25 @@ chain succeeds. ## Right now (frontier) -- **Host is parked in the designed locked fail-safe; reboot pending approval - (2026-10-02 ~10:07, user hold ~10:25).** The GR context-promote retry - (default-off follow-up to the H1 host GR ladder) went green through - boot/READY, the GR context queries, a zeroed 307-page context buffer with - its page-directory mapping and the GR0 channel + compute object, but the - server-side `GPU_PROMOTE_CTX` request **wedged the resource manager** (an - internal assert `0x52a8b04`, no reply, no engine fault), so the ladder - aborted by design. Teardown could not clear the protected region (the - booter-unload gate needs a handoff bit the wedged server never sets; the - host-side unlock is fuse-locked on this part), so the module refused the - hand-back and kept the power-domain locks: module unloaded, GPU unbound, - protected region still up, `nvidia-smi` unavailable. This is the fail-safe - working as designed - host stable, no freeze, pstore empty, no rail - cycles. The acting session deliberately paused here; the reboot request is - awaiting approval because other work is running, and no further live GPU - cycles are planned this boot. The promote-assert bisection plan is ready - for the next boot. The promote is gated by `gsp_phys_rmcall_gr_promote` - (default 0), and the parameter documentation warns that it can wedge the - device. +- **Post-reboot H1 safe half is GREEN (2026-10-02 ~13:55).** After the + approved reboot (boot 13:39), the module rebuilt byte-identical + (`e4f5d7cc...`) and one supervised module-owned cycle with the promote gate + disabled (`gsp_phys_rmcall_engine=1`, `gsp_phys_rmcall_gr_promote=0`) ran + the whole GR pre-promote half on the live GSP-RM: geometry (`raw=0xf2d00`, + `maxSubctx=64`), a zeroed 307-page contiguous context buffer + (`pa=0x5b0000000 va=0x2200000 total=0x133000`), its PDB mapping, the V5 GR0 + channel and the `TURING_COMPUTE_A` compute object; `PROMOTE_CTX` was never + sent and the ladder stopped before BIND with `-EOPNOTSUPP` + (`rm_call_ctx: prepared=1 valid=0`). Release was latched and clean + (distribution driver rebound, protected region down, exactly 5 + power-domain locks, 0 rail cycles, pstore empty, same boot). The p6h1b + assert `0x52a8b04` was decoded statically to the server's + `grctxPromoteAPI` path, and the 570.144 client confirms the canonical + external-VAS shape (PA-only `bInitialize=1 bNonmapped=1` init entry, then + VA-only `bInitialize=0 bNonmapped=0` promote entry) that the next + bisection variant implements. Next: promote variant 2 (two-phase), then 3, + then the FBMEM/aperture fallback if sysmem is rejected; commit only after + `REAL GR SEM OK`. Evidence: `backups/evidence-20261002-gr/` (private). - **Guest-submission relay and fault relay landed, default off (P6c C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical engine authors the guest CE copy and the release semaphore: the @@ -616,7 +616,7 @@ chain succeeds. ## First real GSP boot + live GSP-RM session (2026-10-01) Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`), -`open-iov-vfio.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order +`vgsp.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order observed on the physical TU104: 1. **FWSEC** — WPR2 came up; FRTS error code 0; `WPR2_LO=0x01ffe000`, @@ -667,7 +667,7 @@ over-strict host-side cap on the request's own buffer size. The message was read raw from host memory, the cap corrected (build `87309e3a…`), and the sequencer then ran live in cycles 2-3 (see the frontier section). -Ending a claimed session: `sudo rmmod open_iov_vfio`, then `echo auto | sudo +Ending a claimed session: `sudo rmmod vgsp`, then `echo auto | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the same for `0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. Since cycles 2-3 this passes the handoff gate (the server's shutdown sets the last handoff @@ -684,7 +684,7 @@ Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, HOST SIDE GUEST (stock driver, unmodified) ========= ================================ - [done] build + load open-iov-vfio.ko + [done] build + load vgsp.ko | (module first: exports the symbols the NVIDIA open modules link against) v [done] mdev create ------------------------------------> / @@ -778,7 +778,7 @@ Recent chain of walls removed (2026-09-28 to 2026-09-30): ## Tools, code paths and fixes behind this progress ``` - KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked) + KERNEL MODULE (vgsp.ko) GUEST-SIDE TOOLS (tracked) ================================ ========================== gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh @@ -877,7 +877,7 @@ progress.** - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. - Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages - the physical GPU; `open-iov-vfio.ko` loads as an independent module. + the physical GPU; `vgsp.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. - Firmware: TU104 GSP 570.144 (public release). @@ -888,7 +888,7 @@ Build: host module first (it generates the symbol versions the NVIDIA open kernel modules link against), then the NVIDIA open kernel modules, then userspace. -Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and +Load: VFIO prerequisites, then `vgsp.ko`, then create the mdev and attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the device. -- cgit v1.2.3