STATE-CURRENT: 16:00 snapshot (GR golden replay, mode 24, FLR recovery)

1 files changed, 71 insertions, 12 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 7f21f85..c4566dd 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,7 +1,7 @@
 # Open-IOV Current State
 
 > **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04,
-> physical/guest campaign data as of 2026-10-05 ~15:05; debugging-options
+> physical/guest campaign data as of 2026-10-05 ~15:55; debugging-options
 > catalogue added 2026-10-05).** This is a dated public
 > status page, not live state; it is refreshed on every progress check. Live
 > working notes are maintained outside this repository; the public
@@ -59,7 +59,22 @@ isolation, performance, portability).
    `0x5b00d2c`; it was built (`3bf3e2b2...`, uncommitted) and run
    (`grv23`/`grv23v6`): `PROMOTE_CTX hVirtMemory ... status=0x1f`, a soft-stop
    before BIND, clean release, no wedge. Next: mode 24 (`GPU_INITIALIZE_CTX`
-   with the same FBMEM handle, then the mode-23 promote).
+   with the same FBMEM handle, then the mode-23 promote). **Update 2026-10-05
+   ~15:55:** mode 23 was run on both registries - V5+srvctx soft-fails `0x1f`
+   (the hVirt branch needs the KMD/V6 channel), while V6+client-managed
+   reaches the per-buffer commit `0x5b00d2c` and dies at its hard
+   `pEngCtx->pMemDesc == NULL` gate (`0x5b01210`): the server never installed
+   a MAIN memdesc on the skip-ctxbuffer channel. Root cause (570.144 source):
+   the GR-instance/golden-image initialization
+   (`kgraphicsCreateGoldenImageChannel`) runs only in the host client
+   (`PF_KERNEL_ONLY`), never in GSP-RM, and no control/RPC triggers it. First
+   golden-replay iteration (`grgold22`/`grgold23`): the golden VAS alloc is
+   green, the PB sysmem step is client-local (`rpc_result=0x22`), and a bare
+   `TURING_A` (0xc597) class alloc on the ladder channel does not initialize
+   the instance (NOCAT byte-identical to grs22). Next named step: the full
+   FWCLIENT client-managed construct (client MAIN/PATCH/PM/global allocs +
+   channel-VAS mapping + the reference promote set). Two clean cycles, GPU
+   green.
 - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING
   (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only
   ("no-claim") campaign runs an unmodified guest CUDA program to completion
@@ -112,7 +127,13 @@ isolation, performance, portability).
   `RmInitAdapter` failed; a manual reboot recovered the device. A second
   reboot was needed after the 13:40 external-VAS VA-promote wedge (GSP-RM
   stopped replying, WPR2 up); both reboots were operator-executed and fully
-  recovered (`nvidia-smi -L` green, no Xid since).
+  recovered (`nvidia-smi -L` green, no Xid since). **2026-10-05 ~15:20
+  (`reb2`):** a no-claim acceptance run left the GPU in `RmInitAdapter
+  failed! (0x62:0x65:2522)` with no Xid and pstore empty; a driver reload did
+  not recover it, but an NVIDIA-unbound PCI function-level reset (FLR)
+  restored the device in-session - **no reboot needed**. The no-claim
+  campaign has now damaged the GPU twice (Xid 140, this RmInitAdapter latch):
+  treat it as GPU-risky and keep GR work first.
 - **Simulator workstream:** see the section below (separate, development-only).
 
 **Blockers (current walls).**
@@ -160,10 +181,19 @@ isolation, performance, portability).
     V6 is a dead end; an external-VAS VA promote wedged GSP-RM (recovered by
     reboot). Mode 23 (the `hVirtMemory` no-entry promote with a physical
     FBMEM client memdesc - the one shape that bypasses the booked `vaList`)
-    was built and run (`grv23`/`grv23v6`): soft-stop at `PROMOTE_CTX
-    hVirtMemory ... status=0x1f`, clean release, no wedge. Next: mode 24 =
-    `GPU_INITIALIZE_CTX` with the same FBMEM handle, then the mode-23
-    promote.
+     was built and run (`grv23`/`grv23v6`): soft-stop at `PROMOTE_CTX
+     hVirtMemory ... status=0x1f`, clean release, no wedge. Next: mode 24 =
+     `GPU_INITIALIZE_CTX` with the same FBMEM handle, then the mode-23
+     promote. **Update 2026-10-05 ~15:55:** V5+srvctx `0x1f` confirmed as the
+     hVirt branch's KMD/V6 requirement; V6+client-managed reached `0x5b00d2c`
+     and died at the `pEngCtx->pMemDesc == NULL` gate (`0x5b01210`). The
+     missing provider is the client-side GR-instance/golden init
+     (`kgraphicsCreateGoldenImageChannel`, host-client-only) - hypothesis
+     H-GI. First replay iteration (golden VAS green; PB sysmem is client-local
+     `0x22`; bare `TURING_A` alloc insufficient) leads to the named next step:
+     the full FWCLIENT client-managed construct (client ctx buffers + VAS
+     mapping + reference promote set), then the server commit. See ADDENDUM
+     53/54.
 2. **Hardening the accepted no-claim path.** The first green guest CUDA
    acceptance has landed and is holding; the residual classes are (a) corruption
    via a late first decode behind pump backlog (~3-5% of runs; the experiment
@@ -362,6 +392,7 @@ chain succeeds.
 
 | ID | Milestone | When | State |
 |---|---|---|---|
+| MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident |
 | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) |
 | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) |
 | MS-20261005-02 | GR promote wall crossed on the shared-VAS+srvctx path; new wall = GR ctxsw fault after push fetch | 2026-10-05 ~12:16 | srvctx registry (`RMSetClientRMAllocatedCtxBuffer=0`, `RMSplitVasMgmtServerClientRm=0`) + V5 + mode 17 + client page-table install/cache flush; stage 3 completes (`hGr`, clean release, no wedge); stage 4 fetches (`GPGet=1`) but GR faults during ctxsw (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), no release, `-110`; 10 clean cycles |
@@ -397,6 +428,30 @@ chain succeeds.
 
 ## Milestone log - current frontier and recent milestones (granular IDs in each entry)
 
+- `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI);
+  golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23
+  hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns
+  `0x1f` because the hVirt branch requires the KMD/V6 channel; `grv23v6`
+  (V6+external+client-managed) returns `0x40` but **reaches the per-buffer
+  commit `0x5b00d2c`** and dies at its hard `pEngCtx->pMemDesc == NULL` gate
+  (`0x5b01210` from `0x5b00dd8`) because the server never installed a MAIN
+  memdesc on the skip-ctxbuffer channel (modes 10/11/14/15/16/18/22 instead
+  die earlier at `0x52a94f8`, gated by graphics-context `+0x135`). Source
+  root cause: the GR-instance/golden-image initialization
+  (`kgraphicsCreateGoldenImageChannel`) runs only in the host client
+  (`PF_KERNEL_ONLY`, `kfifoStatePostLoad_GM107`); GSP-RM does not run it and
+  no CTRL/RPC triggers it. First replay iteration: `grgold22` (golden VAS
+  alloc green; PB `NV01_MEMORY_SYSTEM` physical rejected `rpc_result=0x22` -
+  client-local on GSP) and `grgold23` (best-effort replay; `TURING_A` 0xc597
+  alloc on the ladder channel status 0, but NOCAT byte-identical to grs22 -
+  a bare class alloc does not initialize the instance). Named next step: the
+  full FWCLIENT client-managed construct (client MAIN/PATCH/PM/global allocs
+  + channel-VAS mapping + the reference promote set). Both cycles clean, GPU
+  green. Also in this window: the `reb2` no-claim run left `RmInitAdapter
+  failed! (0x62:0x65:2522)` (no Xid); an NVIDIA-unbound PCI FLR recovered it
+  in-session - **no reboot needed** - a new banked recovery capability.
+  Evidence: `backups/evidence-20261002-gr/*-grgold22/23*`,
+  `backups/evidence-20261002-victory/run-noclaim-reb2.log`.
 - `MS-20261005-04` **GR rung 2: the commit chain is decompiled and the promote
   shape space is closed to one remaining form (2026-10-05 13:00-14:48).** After
   stage 3 first completed on the shared-VAS+srvctx path, the campaign ran six
@@ -1121,15 +1176,19 @@ chain succeeds.
   no-claim fixes. **2026-10-05 onward:** acceptance builds `e7ff3bf7` (hold
   all unreadable) and `cb1724d3` (per-type VRAM); GR rung-2 builds
   `19d1e0fc`, `1783dade`, `c998abef`, `b6a9d4eb`, `9075bf57`, `69500d6e`
-  (grctx1-6), `c5c3f396` (grv6a), and `3bf3e2b2...` (mode 23, uncommitted).
+  (grctx1-6), `c5c3f396` (grv6a), `3bf3e2b2...` (mode 23; committed `4212cdb8`),
+  `c905e9da` (grgold22), and `f17b35d6...` (grgold23, golden replay +
+  on-ladder 3D; uncommitted in `gsp_phys_booter.c`, `gsp_phys_gpu.[ch]`).
   GR reports: `backups/evidence-20261005-grctx/REPORT-GR-PROMOTE-COMMIT.md`
   (+ Addendum A), `REPORT-GR-CTX-SOURCE.md`, `REPORT-GR-EVIDENCE-DECODE.md`,
   `REPORT-GR-FORWARDING-DESIGN.md`, `NEXT-BOOT-GR-RUNBOOK.md`; run logs
   `backups/evidence-20261002-gr/gr-run-grctx{1..6}.log`, `gr-run-grv6a.log`,
-  `gr-run-grv23.log`, `gr-run-grv23v6.log` with the `06/20/21/30/31/32-*`
-  capture sets. HEAD `449c53bd` is in sync with `vgpu/master`; the working
-  tree carries the uncommitted mode-23 change (`gsp_phys_booter.c`,
-  `gsp_phys_gpu.c`).
+  `gr-run-grv23.log`, `gr-run-grv23v6.log`, `gr-run-grgold22.log`,
+  `gr-run-grgold23.log` with the `06/20/21/30/31/32-*` capture sets, plus
+  `backups/evidence-20261002-victory/run-noclaim-reb2.log`. HEAD `072b13ba`
+  (check-back commits; `master` in sync with `vgpu/master`); the working tree
+  carries the uncommitted golden-replay edits (`gsp_phys_booter.c`,
+  `gsp_phys_gpu.c`, `gsp_phys_gpu.h`).
 - **Campaign detail:** the private live campaign narrative (handoff/logs,
   maintained outside this repository) is authoritative; this page is the dated
   public snapshot.