# Open-IOV Current State > **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04, > physical/guest campaign data as of 2026-10-06 ~10:55; debugging-options > catalogue added 2026-10-05).** This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public > known-knowns/unknowns document is authoritative here: > [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md). > The former 2026-09-21 snapshot (`STATE.md`) is retired; its content remains > in the git history. ## Objective, status and blockers (this snapshot) **Objective.** Land the first green execution of an unmodified guest CUDA program against the physical TU104 backend (a `vectorAdd`-class kernel whose results verify), then the remaining open gates (multi-VM, 3D, Windows, isolation, performance, portability). **Status.** - **Emulation front: GREEN.** The driver-API smoke completes end-to-end (`ALL OK`, payload 64/64); the runtime client initializes with `rc=0` and queues a kernel launch, whose execution requires the backend. - **Physical backend: live on the real TU104.** Supervised full boot lifecycle (FWSEC -> Booter Load -> RISC-V active -> RM init-done `0x1001`, `rpc_result=0`), static-config RPC served, RM objects + a real channel allocated, and a real copy-engine 4 KiB copy executed (`REAL COPY OK`, semaphore released, destination equals source). Guest CE copies are hardware-authored with engine-authored releases via the P6b/P6c forwarding and relay paths (all parameters default off; one relayed copy's data mismatch is under triage), and the host GR ladder allocates a GR0 channel + compute object. **Update 2026-10-05 ~12:00:** the promote wall is crossed on the shared-management VAS - with the srvctx registry profile (`RMSetClientRMAllocatedCtxBuffer=0`, `RMSplitVasMgmtServerClientRm=0`; both default off), the V5 channel and the client `MAP_MEMORY_DMA` path, the `TURING_COMPUTE_A` compute-object alloc succeeds, the promote/commit passes and the ladder completes stage 3 (`hGr`, clean releases, no server wedge, no reboot needed) - the first time. **Update 2026-10-05 ~12:16:** the stage-4 frontier moved - with a client page-table install (`UPDATE_PDE_2`) plus a cache flush/PT refill, the GR push is now fetched (`GPGet=1`), but the GR engine faults during context switch (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), so no release lands (`-110`); 10 clean cycles, no wedge, no Xid. **Update 2026-10-05 ~14:50:** the rung-2 campaign decompiled the commit chain end-to-end (`PROMOTE_CTX` handler `0x526a530` -> engine-class commit `0x5350078` -> per-entry `0x5290a48` -> per-buffer commit `0x5b00d2c` -> engine-ctx memdesc install `0x514cab8` -> record/instance/RAMFC/engine HAL); the fixed `GR_GET_CTX_BUFFER_INFO` ABI (80-byte entries, params 5136) returns the server's 8 buffers with MAIN/PATCH geometry visible; the server-managed engine-ctx `vaList` is pre-booked, so the UVM-shape VA-only promote is structurally blocked (`vaListSetManaged(NV_FALSE)` -> `NV_ERR_INVALID_STATE` 0x40), `EVICT_CTX` does not release it, and PA-only entries are accepted but never reach the commit; V6 (`SKIP_CTXBUFFER_ALLOC`) is a dead end (the `TURING_COMPUTE_A` construct gets no context memdesc -> -5). The one remaining shape is the `hVirtMemory` no-entry promote with a physical FBMEM client memdesc (mode 23), which bypasses the booked `vaList` and hands the memdesc to `0x5b00d2c`; it was built (`3bf3e2b2...`, uncommitted) and run (`grv23`/`grv23v6`): `PROMOTE_CTX hVirtMemory ... status=0x1f`, a soft-stop before BIND, clean release, no wedge. Next: mode 24 (`GPU_INITIALIZE_CTX` with the same FBMEM handle, then the mode-23 promote). **Update 2026-10-05 ~15:55:** mode 23 was run on both registries - V5+srvctx soft-fails `0x1f` (the hVirt branch needs the KMD/V6 channel), while V6+client-managed reaches the per-buffer commit `0x5b00d2c` and dies at its hard `pEngCtx->pMemDesc == NULL` gate (`0x5b01210`): the server never installed a MAIN memdesc on the skip-ctxbuffer channel. Root cause (570.144 source): the GR-instance/golden-image initialization (`kgraphicsCreateGoldenImageChannel`) runs only in the host client (`PF_KERNEL_ONLY`), never in GSP-RM, and no control/RPC triggers it. First golden-replay iteration (`grgold22`/`grgold23`): the golden VAS alloc is green, the PB sysmem step is client-local (`rpc_result=0x22`), and a bare `TURING_A` (0xc597) class alloc on the ladder channel does not initialize the instance (NOCAT byte-identical to grs22). Next named step: the full FWCLIENT client-managed construct (client MAIN/PATCH/PM/global allocs + channel-VAS mapping + the reference promote set). Two clean cycles, GPU green. **Update 2026-10-06 ~10:55:** the golden-replay ladder and the client-managed FWCLIENT construct were exercised through their full elimination matrix (all supervised, clean teardowns; in-session PCI function-level reset where an RM task hung - no reboots). Results: (a) individual-entry bisection shows the resource manager accepts PM, PATCH, PRIV_ACCESS_MAP and UNRESTRICTED_PRIV_ACCESS_MAP context buffers one at a time and refuses FECS_EVENT (`0x1f`), so the context can be partially completed but not completed; (b) the "golden execution lane" is walled - an RM-managed golden channel tolerates only its exact captured server-owned shape, and attaching client submission wiring (GPFIFO ring or system-memory USERD) makes the next GR object construct hang; (c) the full captured 208-dword guest launch push is decoded and byte-validated and the forward maps every page, yet across the whole hypothesis matrix (verbatim replay, QMD prewrite + explicit PCAS, INITIALIZE_CTX, per-channel compute init, no-SET_OBJECT, sized FBMEM aux, server-managed context) the physical GR engine still never processes the forwarded launch: the inline-QMD destination stays zero, no release fires, no engine fault is recorded. Correction to earlier notes: GPFIFO GET progress and the channel report semaphore are authored by the front-end (PBDMA), not by the GR engine, so they do not prove engine execution. A full individual acceptance-sweep cycle was live at snapshot time. The intermittent guest `cudaMalloc` stall class now has concrete (unvalidated) fix candidates: pool-VA resolution ordering before fallback, and verification of an unreadable entry before it is consumed. Host clean; pstore empty. - **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only ("no-claim") campaign runs an unmodified guest CUDA program to completion with verified results: `MAIL the little gpu man says you've got mail`, `VADD n=1048576 bad=0 checksum=392344090.500000`, `STATUS OK`, deterministic (identical IN/OUT hashes), pixel-verified by the host bridge daemon, and committed (`cf6ff542`). Three data-path/hazard classes were then closed: replayed GPFIFO entries never re-run work (per-slot entry identity, then a per-op executed bitmap); a slot consumed by a grace fake is epoch-marked so its work can never run later against a refilled staging buffer (plus an FNV content witness in the once gate); and the CE pending-entry grace was widened 8 s -> 15 s so a copy's first run precedes its fallback release. Measured effect: first-attempt acceptance moved from 86% to ~94-95% across campaigns and fakes/run fell 3.39 -> 0.61. A further experiment (hold every unreadable CE entry for the bounded grace, delete the silent pre-rescue fake, grace 25 s) is now committed (`4115f1dd`, module `e7ff3bf7`) and under live validation: campaign `239..262` is running with `239-242` green on the deterministic acceptance (`241a`/`243a` harness push failures retried) and `243b` in progress at the 10:38 check. Residual classes: late-decode corruption (~3-5%), an early UVM channel-manager stall (~5-10%), and a guest init panic (~1/10, retried). The vadd executes via the host CUDA bridge daemon; forwarding guest engine work to the physical GPU remains the open backend wall. **Update 2026-10-05 ~12:10:** the validation campaign was green through `242` when a later, uncommitted follow-on experiment (refusing pool writes) **panicked the host kernel at 11:18** - a sleeping VFIO mutex taken from the sema service's GMMU walk in atomic context, then a vmalloc fatal exception in the bridge marshal path; the experiment was saved as a patch, reverted, and a reboot restored the host. Per the 2026-10-05 directive, priority is now **native guest execution without the host CUDA bridge**, so this front is a stability exercise rather than the destination. **Update 2026-10-05 ~13:10:** the per-type emulated-VRAM fix (`c527727a`) is validated live (`nvidia-vgpu-turing` = `framebuffer=512M`/`max_instance=2`, `nvidia-vgpu-turing-1g` = `framebuffer=1024M`; the guest boots to `DEVICE ...` with ~347 MiB free of the 512 MiB FB); the follow-up acceptance run (`reb1`) hit the known stall class (`held=4300 starved=1 ring_auto=6 nomap=4300 faked=0`, the client stuck after `DEVICE`), not a VRAM regression - zero Xid, clean teardown. The campaign stays demoted per the 10:50 directive. - **Hardware incident + recovery (2026-10-03).** The first emulation-only (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the host driver owned the device and triggered an uncorrectable ECC Xid, after which `RmInitAdapter` failed persistently even after a full driver reload. The host recovered after a reboot (~20:11); `nvidia-smi -L` is green and the device is healthy. The fix (shadow BAR2 in emulation-only mode) is committed (in `cf6ff542`) and has held across the acceptance campaigns since - no Xid observed. **2026-10-05 midday:** a second Xid 140 fired ~12:17 during the first no-claim acceptance run of that boot (after 13 clean GR cycles; the BAR2 shadow held, so the trigger is under review), leaving `RmInitAdapter` failed; a manual reboot recovered the device. A second reboot was needed after the 13:40 external-VAS VA-promote wedge (GSP-RM stopped replying, WPR2 up); both reboots were operator-executed and fully recovered (`nvidia-smi -L` green, no Xid since). **2026-10-05 ~15:20 (`reb2`):** a no-claim acceptance run left the GPU in `RmInitAdapter failed! (0x62:0x65:2522)` with no Xid and pstore empty; a driver reload did not recover it, but an NVIDIA-unbound PCI function-level reset (FLR) restored the device in-session - **no reboot needed**. The no-claim campaign has now damaged the GPU twice (Xid 140, this RmInitAdapter latch): treat it as GPU-risky and keep GR work first. - **Simulator workstream:** see the section below (separate, development-only). **Blockers (current walls).** 1. **GR context commit/execution.** The promote-side wall is root-caused (the ladder VA space was externally owned, so the engine-context commit was skipped and a promote VA was never booked). A default-off shared-management address space gets past it, but the next RM call (`TURING_COMPUTE_A` compute-object alloc) wedges the server; three further cycles closed the client-promote shape space on the default external VAS with a soft `NV_ERR_INVALID_STATE` (0x40) each. The V5 commit assert `0x52a8b04` is a secondary failure. Offline (2026-10-05): the promote control handler and its commit chain are decompiled; the external-VAS branch is confirmed in `kchannelMapEngineCtxBuf_IMPL` (a externally-owned VAS requires the engine context VA to be booked in the server's `vaList`, which a raw-PTE client map never creates, while a non-external VAS is mapped and booked by the server); the V6 soft-0x40 is the known rec-677 message emitted through the promote error machinery. The rec-677 format text lives in runtime-inflated metadata (not in the shipped image), so naming it needs a live-GSP metadata dump. Next routes: name the soft-0x40 record from live metadata, root-cause the V5 first failure (the log call site before the assert is now identified), or root-cause the shared-management `TURING_COMPUTE_A` wedge; physical cycles only with a named hypothesis and supervision. **Update 2026-10-05 ~12:00: this wall moved.** The untested combination (shared-management VAS + srvctx registry + V5 channel + no client promote) is confirmed with **no wedge**: `TURING_COMPUTE_A` succeeds, the promote/commit passes and the ladder completes stage 3 with clean releases, once the srvctx registry also disables split VAS management (`RMSplitVasMgmtServerClientRm=0`) and the client sends `MAP_MEMORY_DMA` for the ladder buffers - on a shared VAS the server owns the lower page tables, so raw client PTEs are not authoritative. **Update 2026-10-05 ~12:16:** the stage-4 frontier moved - a client page-table install plus a cache flush/PT refill makes the GR PBDMA fetch the submitted push (`GPGet=1`), but the GR engine faults during context switch (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), so no release lands (`-110`). Next: root-cause the ctxsw fault (engine-context buffer init, subcontext header, or a VA+PA promote); 10 clean cycles, no wedge. **Update 2026-10-05 ~14:50:** the wall is now one promote/commit shape. The rung-2 campaign decompiled the full commit chain (`0x526a530` -> `0x5350078` -> `0x5290a48` -> `0x5b00d2c` -> `0x514cab8` -> record/ instance/RAMFC/engine HAL) and fixed `GR_GET_CTX_BUFFER_INFO` (80-byte entry, params 5136; the server's 8 buffers now visible). On `srvctx=1` the server pre-books MAIN in the engine-ctx `vaList`, so the UVM-shape VA-only promote returns 0x40 at `vaListSetManaged(NV_FALSE)` and `EVICT_CTX` does not release it; PA-only is accepted but never commits; V6 is a dead end; an external-VAS VA promote wedged GSP-RM (recovered by reboot). Mode 23 (the `hVirtMemory` no-entry promote with a physical FBMEM client memdesc - the one shape that bypasses the booked `vaList`) was built and run (`grv23`/`grv23v6`): soft-stop at `PROMOTE_CTX hVirtMemory ... status=0x1f`, clean release, no wedge. Next: mode 24 = `GPU_INITIALIZE_CTX` with the same FBMEM handle, then the mode-23 promote. **Update 2026-10-05 ~15:55:** V5+srvctx `0x1f` confirmed as the hVirt branch's KMD/V6 requirement; V6+client-managed reached `0x5b00d2c` and died at the `pEngCtx->pMemDesc == NULL` gate (`0x5b01210`). The missing provider is the client-side GR-instance/golden init (`kgraphicsCreateGoldenImageChannel`, host-client-only) - hypothesis H-GI. First replay iteration (golden VAS green; PB sysmem is client-local `0x22`; bare `TURING_A` alloc insufficient) leads to the named next step: the full FWCLIENT client-managed construct (client ctx buffers + VAS mapping + reference promote set), then the server commit. See ADDENDUM 53/54. 2. **Hardening the accepted no-claim path.** The first green guest CUDA acceptance has landed and is holding; the residual classes are (a) corruption via a late first decode behind pump backlog (~3-5% of runs; the experiment deleting the silent pre-rescue fake and holding every unreadable CE entry for a 25 s bounded grace is committed and under live validation), (b) an early stall before any push reaches the service (client waits in `uvm_channel_manager_create`/`channel_pool_add`; `ring_auto=6` vs 26 is the discriminator; ~5-10%, retried by the campaign), and (c) the guest init panic class (~1/10, retried; host unaffected). Remaining work: settle or fail-loud the residual classes, trim/re-run with the bounded diagnostics, and extend acceptance to a second workload. Guest-truth kernel probes (installed per guest boot) are load-bearing for the GPU-only pool/push-buffer translations; without them the host falls back to content heuristics. The 2026-10-05 continuation experiment (refusing pool writes) panicked the host and is reverted; the directive now prioritizes bridge-free guest GR execution, so this front continues as a stability exercise on the committed hold-all-unreadable build (`4115f1dd`). 3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side data compare; handled fail-closed, parameter default-off). 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect (`staging not visible before kick`; fail-open, quarantined off the critical path); the D0/D3cold power-gating freeze hazard (default-latched locks, exactly 5 power-domain locks per release); and the **no-claim BAR2 exposure** (real GPU MMIO mapped to the guest in emulation-only mode can drive the host-owned device to an ECC Xid; shadowed BAR2 prevents it - committed and holding, no Xid since). Recurring and separate: guest-kernel boot panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup - guest-side only, host stays healthy. A new host panic class was logged 2026-10-05 11:18 with the (saved, reverted) pool-write-refusal experiment: a sleeping VFIO mutex taken in atomic context in the sema GMMU walk plus a vmalloc fatal exception in the bridge marshal path; reboot recovered the host, no Xid. ## Options to accelerate debugging toward end-to-end CUDA (candidate techniques; not directives) > Added 2026-10-05. These are options the project can pick up when it fits the > current campaign; they do not redirect the work in progress and are not > commitments. Each is a tool/workflow introduction and would need a > `docs/eng-ops.md` admission record (scenario + measurable outcome + baseline + > evaluation threshold + failure signal) before adoption. The present issues they target: (a) the residual no-claim failure classes - a late-decode corruption (~3-5% of runs: a queued copy's first run can land after the client has refilled its staging buffer), an early stall before pushes reach the service (~5-10%; the client waits in the UVM channel-manager path), and a guest-kernel init panic (~1/10, retried); (b) the physical GR engine-context commit wall (external-VAS booking; the shared-management `TURING_COMPUTE_A` wedge; the rec-677 soft-0x40 message); (c) the cost of a supervised cycle (a wedge currently ends in a reboot). The end-to-end objective is an unmodified guest CUDA workload executing on the physical GPU; the nearer objective is a settled, repeatable acceptance on the no-claim bridge. 1. **Deterministic fault injection for the service pump.** The pump's hostile interleavings (translation latency, replay/rewind, grace fakes, staging refill) are reachable today only by waiting for a low-probability schedule. Debug knobs that force each one would turn the 3-5% classes into 100% reproductions and let a hostile-interleaving suite run every build. *Helps now:* residual corruption/stall fixes become fast and testable instead of probabilistic. *End-to-end:* removes the replay/fake hazard class from the acceptance path and gives a deterministic regression gate for every future workload. *Status:* not in use (threshold parameters exist; no forced-interleaving surface). 2. **Offline replay and an invariant model of the pump.** The decision logic (hold/retry, once-gate, epoch marks, fakes) is instrumented and mostly isolated; replaying captured push bytes plus translation/release timings through a userspace mirror, and property-checking the invariants (exactly-once work per slot/push generation; no work after the client has observed a release; fakes only after bounded retries and marked consumed), would find holes in minutes instead of supervised cycles. *Helps now:* the residual classes are exactly invariant violations. *End-to-end:* a checked model of the bridge's safety contract is reusable for GR forwarding and future workloads. *Status:* not in use; the simulator workstream is firmware-focused and this would be its host-side sibling. 3. **Structured host tracing instead of printk floods.** Pump decisions are traced as text today; runs have produced hundreds of MB of kernel log, slow to collect and hard to order. ftrace trace events (or `trace_printk`) with a binary ring and a per-run timeline, plus eBPF/bpftrace kprobes on the service functions for latency histograms and stack traces without rebuilds, would give ordered evidence at low overhead. *Helps now:* the residual bugs are ordering bugs. *End-to-end:* a permanent low-overhead tracing surface for every future performance/porting question. *Status:* not in use on the host (guest kprobes are in use). 4. **Guest-memory forensics for the panic/corruption class.** The init panic has no identified writer. The QEMU gdbstub is already listening, so hardware watchpoints on the guest address mapping the corrupted page plus a `dump-guest-memory` at panic would name a CPU writer; a KVM dirty-log view would attribute writes per guest frame; a diagnostic guest kernel with KASAN/KFENCE/slub_debug (product guests stay stock) would catch a guest-software wild write; and because device DMA bypasses CPU watchpoints, host-side checks on every module/bridge write to guest page-table frames are the complement. *Helps now:* converts an unexplained ~1/10 flake into a located bug. *End-to-end:* guest-memory integrity is a prerequisite for multi-tenancy/isolation and for trusting any physical-backend result. *Status:* named as a next step but not executed. 5. **Physical GR wall: runtime metadata, a golden reference, static-analysis throughput.** Three independent moves: (a) snapshot the runtime-inflated logging metadata from a live boot to name the rec-677 soft-0x40 message (the format text is not in the shipped image); (b) use a host RM client as a golden reference - the userspace oracle showed the engine-context controls are not reachable from userspace, so a kernel-side RM client is the plausible path to run the real promote/allocate sequence and diff shapes against our ladder; (c) restore Ghidra headless scripting (batch decompile and xrefs) or use an alternate static-analysis engine for targeted "which check returns 0x40" queries - the promote path has multi-KB functions that are impractical to read by hand. *Helps now:* each shortens the GR commit investigation. *End-to-end:* GR execution is the remaining gate between the host-bridge acceptance and real guest compute on the GPU. *Status:* (a) identified, not done; (b) oracle exists, userspace route proven blocked; (c) headless scripting currently unavailable. 6. **Cut the supervised-cycle cost.** A wedged GSP boot currently ends in a supervised reboot, which caps experiment throughput. The tree already has a WPR2-clear utility (BAR0 MMIO) and a designed fail-safe; extending the supervised recovery path to use them, or restarting only the GSP boot ladder instead of the host, would multiply cycles per hour. *Helps now:* every residual-class fix and every GR hypothesis gets more attempts. *End-to-end:* shortens the critical path to physical GR proof and to multi-version/ multi-device porting. *Status:* utilities exist; not part of the routine recovery flow. 7. **Concurrency and robustness checks on the host module.** KCSAN (data races), lockdep (the stuck-daemon class seen after the service thread oops), KFENCE (the slot-reuse use-after-free class), and sparse/smatch over the large service file would catch hazard classes before they surface as oopses in supervised runs. *Helps now:* the module has had both a use-after-free and a lock-hold failure. *End-to-end:* hardening needed for long-running multi-VM operation. *Status:* not in use. 8. **Wire the existing analysis tools into the loop.** The failure classifier and guest-init monitor exist; running them automatically per cycle and tracking acceptance rates and hazard-branch coverage per commit would turn the campaign into a regression gate and make sure the once-gate/fake/epoch paths are exercised deliberately rather than by luck. *Helps now:* rates are currently hand-tallied. *End-to-end:* the acceptance suite becomes CI for future workloads and versions. *Status:* tools exist; not automated. ## Simulator workstream (risc-v-gsp-rm-debugger) Development-only tooling for simulating and inspecting GSP firmware runtime memory; never shipped, never a guest dependency, no GPU/mdev/kernel-module touch. Status 2026-10-03: - **S1 device model + boot ladder:** the real 570.144 image boots natively to rung **`B2`** (relocation copy; 14,682 instructions, 0 unknown MMIO, 2-run determinism), two rungs past the QEMU first fault; `B3/B4` (message loop) are the targets. Docs: `debug/risc-v-gsp-rm-vm/docs/35` (design, Phase S/G) and `36` (device model, `B0`-`B2`). - **Evaluation harness:** `harness/sim_qemu_line.py` runs the simulator-vs-QEMU A/B on frozen thresholds; verdict **`above`** on the S1 path (docs/37). - **One-command triage:** `./risc-v-gsp-rm-debugger.sh triage` (or `python3 harness/triage.py --firmware --out dumps/triage`) chains identity/probe/native-boot/A-B/version-diff into one report (docs/40). - **Tests:** **290 collected on 2026-10-03** (`make test` in the tool directory; audit baseline 155; docs/00 §Status, docs/27). - **Docs index:** `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` + README document map (docs 30-40, one line each). ## Mission Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality (CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without proprietary GRID licenses - via host-side GSP emulation in `vgsp.ko`. The guest stays stock: no guest patches, no guest-side Open-IOV code. Architecture stance: *emulate the interface, pass the work through to real hardware, never patch anyone.* The control plane is emulated; the work is meant to execute on the physical GPU. ## Milestone ladder (this snapshot) | Milestone | Status | |---|---| | GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ | | Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ | | `nvidia-smi -L` enumerates the emulated TU104 | ✅ | | CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | | `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟡 the static runtime initializes end-to-end with **real releases** and the launch now executes through the host bridge (see next row); a full `nvidia-smi` (per-process/utilization) remains unverified | | CUDA kernel execution (`vectorAdd`) | ✅ **First green end-to-end guest CUDA (2026-10-04 ~16:20, no-claim host bridge):** `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, three consecutive deterministic supervised cycles, committed `cf6ff542`. The vadd runs through the host CUDA bridge daemon; forwarding guest engine work to the real GPU is the remaining backend path. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | Test status is tracked with the CUDA driver-API smoke test ([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises `cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/ DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole chain succeeds. ## Milestone index (granular IDs; full detail in the milestone log below) | ID | Milestone | When | State | |---|---|---|---| | MS-20261005-05 | Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration | 2026-10-05 15:35-15:55 | V5+srvctx 0x1f (hVirt needs KMD/V6); V6+client-managed reaches `0x5b00d2c` -> 0x40 at `pEngCtx->pMemDesc == NULL`; goldens create only in the host client (`PF_KERNEL_ONLY`); `grgold22/23` clean, PB step client-local (0x22), bare `TURING_A` alloc insufficient; next: full FWCLIENT construct; FLR recovery banked after the reb2 RmInitAdapter incident | | MS-20261005-04 | GR rung 2: ctx-query ABI fixed, commit chain decompiled, mode 23 soft-stop | 2026-10-05 13:00-14:48 | `GR_GET_CTX_BUFFER_INFO` fixed (80 B entry, params 5136; 8 buffers); VA-only promote blocked by the server-booked `vaList` (`vaListSetManaged` -> 0x40), `EVICT_CTX` no-op, external-VAS promote wedge (reboot), V6 dead end (`TURING_COMPUTE_A` -5); mode 23 (`hVirtMemory` + FBMEM memdesc) built (`3bf3e2b2...`, uncommitted) and run twice: `PROMOTE_CTX hVirtMemory status=0x1f`, clean release, no wedge; next mode 24 (`GPU_INITIALIZE_CTX` + mode 23) | | MS-20261005-03 | Per-type emulated VRAM (1.0) validated live | 2026-10-05 ~13:05 | `c527727a`; `nvidia-vgpu-turing` = 512M/max 2, `nvidia-vgpu-turing-1g` = 1024M; guest sees ~347 MiB free of the 512 MiB FB; `reb1` acceptance failed on the known UVM-channel stall class (no Xid, clean teardown) | | MS-20261005-02 | GR promote wall crossed on the shared-VAS+srvctx path; new wall = GR ctxsw fault after push fetch | 2026-10-05 ~12:16 | srvctx registry (`RMSetClientRMAllocatedCtxBuffer=0`, `RMSplitVasMgmtServerClientRm=0`) + V5 + mode 17 + client page-table install/cache flush; stage 3 completes (`hGr`, clean release, no wedge); stage 4 fetches (`GPGet=1`) but GR faults during ctxsw (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), no release, `-110`; 10 clean cycles | | MS-20261005-01 | Hardening the accepted no-claim path (hazard classes closed; rates up) | 2026-10-05 overnight | per-op once bitmap + epoch/content-hash gate + grace 15 s (`b645cf6b`, `3b094c2d`, `c3cd4631`); first-attempt 86% -> ~94-95%; fakes/run 3.39 -> 0.61; hold-all-unreadable/25 s experiment committed (`4115f1dd`, module `e7ff3bf7`), campaign `239..262` validating (`239-242` green; `241a`/`243a` infra retries; `243b` live) | | MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign | | MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet | | MS-20261004-02 | Latent bridge pending-slot oops found, fixed; reboot | 2026-10-04 ~14:45-15:00 | `cubb_pend_release` slot reuse -> NULL deref; fixed `85539844...`; approved reboot restored the GPU | | MS-20261004-03 | Post-reboot acceptance run blocked by guest boot panic | 2026-10-04 15:03 | `noclaim32` `accepted: 0`, clean teardown; fixed module's deep path unverified | | MS-20261003-01 | Guest reaches first `cudaMalloc`; emulation pipeline live | 2026-10-03 ~06:10 | 101+ GR pushes, 219 GR releases, ~165k publishes, 88 forwarded works; hang in UVM tracker pool-sem VA->PA | | MS-20261003-02 | Hardware incident (Xid 140) + reboot recovery | 2026-10-03 13:47 / ~20:11 | no-claim BAR2 exposure faulted the GPU; reboot restored it; fix in flight | | MS-20261003-03 | Large-push parser wall fixed; UVM fully engaged | 2026-10-03 ~00:10 | `NON_INC` count>64 / zero-padding parser fix; `verify_ok=62` forwarded guest copies | | MS-20261002-01 | First real guest CUDA program vs the physical backend | 2026-10-02 ~22:01 | device props enumerated; stalls in first alloc/UVM; clean teardown | | MS-20261002-02 | GR promote-side wall root-caused; current wall shared-mgmt GR alloc | 2026-10-02 ~17:35 | default-off change; server wedge on `TURING_COMPUTE_A`; fail-safe held | | MS-20261002-03 | Guest-submission relay + fault relay (P6c) | 2026-10-02 07:43-08:07 | landed default-off; `relays=64`; one data-mismatch under triage | | MS-20261002-04 | Host GR ladder (H1) opened | 2026-10-02 08:29 | GR0 channel + compute object; first submit `CTXNOTVALID` | | MS-20261002-05 | risc-v-gsp-rm-debugger status (tool) | 2026-10-02 08:36 | superseded 2026-10-03: see the Simulator workstream section | | MS-20261002-06 | P6a trigger-path fix | 2026-10-02 07:10 | guest G2H/W2G copy green; PTE self-test green | | MS-20261002-07 | P6b wave 2 (hardware-primary copies) | 2026-10-02 | `copy_skipped=119`; queued guest copies 100/100 verified | | MS-20261002-08 | P6b wave 1 (guest CE forwarding) | 2026-10-02 | `req=42 ok=42 verify_ok=42`; 512-pair scratch-pool rule | | MS-20261002-09 | P6a guest-page DMA proven on the physical engine | 2026-10-02 | G2H + W2G 4 KiB with semaphore; guest observed the write | | MS-20261002-10 | P6 guest-memory first attempts | 2026-10-02 ~02:42 | resolved by P6a | | MS-20261002-11 | P5 wave 1 guest-submission shadow execution | 2026-10-02 ~02:00 | 514 real CE copies; 194 coalesced drops | | MS-20261002-12 | P3 read-only control forwarding | 2026-10-02 01:33 | `ok=22 fallback=2 timeout=0`; smoke green | | MS-20261002-13 | P2 guest-facing identity overlay | 2026-10-02 01:20 | live Product Name A/B (overlay 4x/0) | | MS-20261002-14 | G0 freeze-fix bundle validated live | 2026-10-02 ~00:57 | 2/2 green cycles; committed `56766b2` | | MS-20261002-15 | P1 mdev-owned lifecycle (fail-open) | 2026-10-02 ~00:58 | retested after G0; no freeze | | MS-20261001-01 | Guest-wiring rung opened (P1) | 2026-10-01 22:00 | backend-session API inert; baseline re-verified | | MS-20261001-02 | Submission rung: real CE copy on the physical engine | 2026-10-01 17:24-21:05 | `REAL COPY OK`; clean release each cycle | | MS-20261001-03 | GSP-RM serves the static-config RPC | 2026-10-01 16:45 | `rpc_result=0`; 1656-byte config; graceful unload | | MS-20261001-04 | Driver-API baseline re-verified green | 2026-10-01 14:20 | `ALL OK` 64/64; `faked=0 miss=0 stale=0` | | MS-20261001-05 | First real GSP boot + live GSP-RM session | 2026-10-01 | WPR2 up; RISC-V active; first event `0x1002` | | MS-LEGACY-01..06 | Emulation-campaign milestones (pre-2026-10-01) | 2026-09-26..09-30 | details in the milestone log/archive below | ## Milestone log - current frontier and recent milestones (granular IDs in each entry) - `MS-20261005-05` **Mode-23 root-caused to the missing GR-instance init (H-GI); golden replay first iteration (2026-10-05 15:35-15:55).** The mode-23 hVirtMemory+FBMEM runs were dissected: `grv23` (V5+shared+srvctx) returns `0x1f` because the hVirt branch requires the KMD/V6 channel; `grv23v6` (V6+external+client-managed) returns `0x40` but **reaches the per-buffer commit `0x5b00d2c`** and dies at its hard `pEngCtx->pMemDesc == NULL` gate (`0x5b01210` from `0x5b00dd8`) because the server never installed a MAIN memdesc on the skip-ctxbuffer channel (modes 10/11/14/15/16/18/22 instead die earlier at `0x52a94f8`, gated by graphics-context `+0x135`). Source root cause: the GR-instance/golden-image initialization (`kgraphicsCreateGoldenImageChannel`) runs only in the host client (`PF_KERNEL_ONLY`, `kfifoStatePostLoad_GM107`); GSP-RM does not run it and no CTRL/RPC triggers it. First replay iteration: `grgold22` (golden VAS alloc green; PB `NV01_MEMORY_SYSTEM` physical rejected `rpc_result=0x22` - client-local on GSP) and `grgold23` (best-effort replay; `TURING_A` 0xc597 alloc on the ladder channel status 0, but NOCAT byte-identical to grs22 - a bare class alloc does not initialize the instance). Named next step: the full FWCLIENT client-managed construct (client MAIN/PATCH/PM/global allocs + channel-VAS mapping + the reference promote set). Both cycles clean, GPU green. Also in this window: the `reb2` no-claim run left `RmInitAdapter failed! (0x62:0x65:2522)` (no Xid); an NVIDIA-unbound PCI FLR recovered it in-session - **no reboot needed** - a new banked recovery capability. Evidence: `backups/evidence-20261002-gr/*-grgold22/23*`, `backups/evidence-20261002-victory/run-noclaim-reb2.log`. - `MS-20261005-04` **GR rung 2: the commit chain is decompiled and the promote shape space is closed to one remaining form (2026-10-05 13:00-14:48).** After stage 3 first completed on the shared-VAS+srvctx path, the campaign ran six supervised cycles (`grctx1-6`) that (1) fixed the `GR_GET_CTX_BUFFER_INFO` ABI - the entry is 80 B (uuid at 60, flags 56), `paramSize=5136`; the query now returns the server's 8 context buffers (MAIN `0x133000` FBMEM, PATCH `0x5000`, plus type 3/4/5/6/10/11) with status 0; (2) decompiled the real promote commit chain (`PROMOTE_CTX` handler `0x526a530` -> engine-class commit `0x5350078` -> per-entry `0x5290a48` -> per-buffer commit `0x5b00d2c` -> engine-ctx memdesc install `0x514cab8` -> record/instance/ RAMFC/engine HAL, the `channelCommitEngineContext_HAL` analogue); (3) proved the UVM-shape VA-only promote returns 0x40 at `vaListSetManaged(NV_FALSE)` (`0x52915e8`) because the server pre-books MAIN in the engine-ctx `vaList` when it manages the ctx buffers, and `EVICT_CTX` does not release that booking; PA-only entries are accepted (status 0) but never take the commit branch, so RAMFC stays zero and no engine-context pointer appears. The external-VAS VA-promote variant wedged GSP-RM (`-110`, WPR2 up) and needed an operator reboot (the exact shape must not be repeated). Post-reboot: V6 (`SKIP_CTXBUFFER_ALLOC`) is a dead end (the `TURING_COMPUTE_A` construct fails -5 without a ctx memdesc), and the registry semantics were resolved from source (default = client-managed; `srvctx=1` = server-managed). The one remaining shape is the `hVirtMemory` no-entry promote with a physical FBMEM client memdesc (mode 23), which bypasses the booked `vaList`; it was implemented (uncommitted, module `3bf3e2b2...`) and run twice (`grv23` V5+srvctx and `grv23v6`): `PROMOTE_CTX hVirtMemory ... status=0x1f` at stage 3, a soft-stop before BIND, clean release each time, no wedge, `nvidia-smi` green. Next: mode 24 = `GPU_INITIALIZE_CTX` with the same FBMEM handle, then the mode-23 promote. Evidence: `backups/evidence-20261005-grctx/` (`REPORT-GR-PROMOTE-COMMIT.md` + Addendum A, `REPORT-GR-CTX-SOURCE.md`, `REPORT-GR-EVIDENCE-DECODE.md`, `REPORT-GR-FORWARDING-DESIGN.md`), `backups/evidence-20261002-gr/ gr-run-grctx{1..6}.log`, `gr-run-grv23.log`, `gr-run-grv23v6.log`. - `MS-20261005-03` **Per-type emulated VRAM (1.0) validated live (2026-10-05 ~13:05).** The per-mdev emulated-VRAM fix (`c527727a`) resolves the earlier multi-size regression (a `sysfs_name` prefix strip plus a non-canonical FB encoder): `nvidia-vgpu-turing` reports `framebuffer=512M`, `max_instance=2`, and the new `nvidia-vgpu-turing-1g` `framebuffer=1024M`; a guest on the 512 MiB type boots to `DEVICE NVIDIA GeForce RTX 2080 SUPER cc=7.5 sm=48` with ~347 MiB free FB. The follow-up no-claim acceptance run (`reb1`) then hit the known emulation-front stall class (`held=4300 starved=1 ring_auto=6 nomap=4300 faked=0`, the client stuck after `DEVICE`), explicitly not a VRAM regression: zero Xid, clean teardown, `nvidia-smi` green. The no-claim campaign stays demoted per the 2026-10-05 directive; multi-mdev (two available instances) is parked. - `MS-20261005-02` **GR promote wall crossed on the shared-management path (2026-10-05 ~12:00).** The untested GR-ladder combination - shared-management VAS, the srvctx pre-boot registry (`RMSetClientRMAllocatedCtxBuffer=0`), a V5 (non-skip) channel and no client promote - gets past the promote wall with **no wedge**: the `TURING_COMPUTE_A` compute-object alloc succeeds (the earlier shared-VAS assert storm does not recur), BIND/schedule/token are accepted, and the ladder completes stage 3 for the first time (`hGr` allocated; clean releases; WPR2 down; device rebound). Two further requirements were source-confirmed and implemented: on a shared VAS the server owns the lower page tables, so the client must send `MAP_MEMORY_DMA` for the ladder buffers, and GSP-RM only serves that RPC when the srvctx registry also sets `RMSplitVasMgmtServerClientRm=0`. **Update ~12:16:** the stage-4 frontier moved - with a client page-table install (`UPDATE_PDE_2`) plus a cache flush/PT refill, the GR PBDMA now fetches the submitted push (`GPGet=1`), but the GR engine faults during context switch (`ROBUST_CHANNEL_GR_FAULT_DURING_CTXSW`), so no release lands (`-110`). All ladder cycles released the device cleanly (no reboot, no Xid, pstore empty); the fixes are now committed (`79d5b65e`). - `MS-20261005-01` **Hardening the accepted no-claim path (2026-10-05 overnight).** Three kernel-side fixes closed the replay/fake hazard classes after the first acceptance: (1) a held-entry retry must not re-run completed ops - per-slot executed-op bitmap plus a complete flag, persisted in the snapshot (`b645cf6b`); (2) a slot consumed by a grace fake is epoch-marked so its work can never run later against a refilled staging buffer, and the once gate gained an FNV content witness so a rewritten push is treated as new (`3b094c2d`); (3) the CE pending-entry grace widened 8 s -> 15 s so a copy's first run precedes its fallback release (`c3cd4631`). Measured: first-attempt acceptance 86% -> ~94-95% across campaigns, fakes/run 3.39 -> 0.61. Two further experiments were tested and reverted (a defer-the-release branch missed the late-first-decode path); a third (hold every unreadable CE entry, delete the silent pre-rescue fake, 25 s grace) is now committed (`4115f1dd`, module `e7ff3bf7`) and under live validation (campaign `239..262`; `239-242` green, the only failures being harness pushes at `241a`/`243a`, retried; `243b` live at the 10:38 check). Residual classes: late-decode corruption (~3-5%), early UVM channel-manager stall (~5-10%), guest init panic (~1/10, retried). - `MS-20261004-04` **First green end-to-end guest CUDA acceptance (2026-10-04 ~16:20).** After `noclaim32` (guest double-fault) and `noclaim33` (vadd data corrupt), two data-path defects were root-caused with bounded instrumentation and fixed: (1) a replayed GPFIFO entry re-executed its CE copies - the pump rewinds `last_get` on each injected-translation batch and the replay suppression missed some, so a replayed multi-MB HtoD re-copied a staging buffer the client had already refilled; fixed with per-slot raw-entry identity so a matching replay re-applies releases but never re-runs operations; (2) `svc_copy` copied a multi-page virtual operand linearly from its first page while client staging ranges are split; fixed with page-granular re-resolution for virtual operands. Then `noclaim38` reported `INCHECK/OUTCHECK bad=0` and the guest printed `VADD n=1048576 bad=0 checksum=392344090.500000` -> `STATUS OK`; `noclaim39/40` repeated green on the same module (`582bf1da...`) with identical hashes. A repeat-run stability campaign since is at 14 of 15 green (one guest-side flake failure). The vadd executes through the host CUDA bridge daemon; forwarding guest engine work to the physical GPU remains the backend wall. - `MS-20261004-01` **The no-claim bridge reaches the guest `vectorAdd` launch (2026-10-04 ~14:45).** The emulation-only campaign fixed three defects in one window: the guest-truth UVM kernel probes were never installed, so the GPU-only pool/push-buffer translations fell back to content heuristics (with the probes the pushes decode with no fakes and the bridge completes); the QMD accumulator's address window swallowed the copy-class DtoH operands, so the read-back never ran; and at 256 MiB the guest had too little free FB. After the fixes the guest prints its full acceptance string, the DtoH mail read-back works, `cudaMalloc` is unblocked by a 512 MiB emulated FB, and the client reaches the `vadd` launch (`grid=4096,1,1 ... n=1048576`). Not yet green: no `STATUS OK`/`bad=0` captured. - `MS-20261004-02` **Latent bridge pending-slot defect found, fixed, and reboot (2026-10-04 ~14:45-15:00).** A freed bridge pending slot was reused with a stale `have_done`/`done_out_len`, so the next pass copied from a NULL blob and dereferenced NULL in the sema service; the fix clears the done flags on release and on staging plus a defensive guard. The oops had killed the service thread holding its lock, so `rmmod` failed and an approved reboot (~15:00) restored the host; the GPU is healthy and `nvidia-smi -L` is green. - `MS-20261004-03` **Post-reboot acceptance run blocked by a guest boot panic (2026-10-04 15:03).** The first post-reboot no-claim run (`noclaim32`) loaded the fixed module, created an mdev and booted the guest, but the guest kernel panicked ~11 s into boot (a NULL-in-interrupt/swap class), so the client could not be pushed (`accepted: 0`). Teardown was clean (`rmmod rc=0`, nvidia rebound, pstore empty); the fixed module's deep launch/vadd path is still unexercised. - `MS-20261003-01` **The emulation pipeline is live and the guest reaches its first `cudaMalloc` (2026-10-03 ~00:10-06:10).** The method-stream parser that could never validate UVM's large page-table pushes (`NON_INC` methods with count > 64, pushes with leading zero padding) was fixed; UVM is now fully engaged (11,867 pushes in ~10 s, `walks=124/178`, 62 guest copies forwarded to the physical GPU with `verify_ok=62`). The victory wave then decoded 101+ GR pushes, 219 GR releases and ~165k completion-counter publishes, auto-pinned all 20 GR rings, and verified 88 forwarded works. The first `cudaMalloc` now proceeds further but hangs on the UVM page-table tracker: the pool semaphore at GPU-only VA `0x121000000` has no CPU pagemap, so the host resolves a different physical address across runs and the guest never observes the releases. Immediate blocker: authoritative GPU-only VA->PA. - `MS-20261003-02` **Hardware incident and recovery (2026-10-03).** The first emulation-only (no-claim) acceptance run had the module map real GPU BAR2 PFNs to the guest while the host driver owned the device; this triggered an uncorrectable ECC Xid and persistent `RmInitAdapter` failure that survived a full driver reload. The host was recovered by a reboot (~20:11): `nvidia-smi -L` green, device bound and healthy, no Xid this boot. The no-claim BAR2 shadow fix is in flight and gates further no-claim runs. - `MS-20261003-03` **Large-push parser wall fixed (2026-10-03 ~00:10).** The parser rejected `NON_INC` method streams over 64 methods and pushes with leading zero padding, so no rescue could validate UVM's page-table pushes; after the fix UVM's page-table pushes execute and the guest advances past the previous registration stall. This is the enabling fix for `MS-20261003-01`. - `MS-20261002-01` **First real guest CUDA program exercised against the physical backend (2026-10-02 ~22:01).** A fresh supervised module-owned cycle (claim + full GSP-RM boot + guest RAM + hardware-primary work forwarding) booted the real server (`stage 3e complete`, init event `0x1001`, `rpc_result=0` in ~952 ms), the host ladder ran its `REAL COPY OK`, and the guest enumerated `NVIDIA GeForce RTX 2080 SUPER`. The static-runtime victory program (`tools/cuda-ab/victory_mail.cu`) was pushed and started: it printed `DEVICE ... cc=7.5 sm=48 mem=134152192`, so device-property enumeration now returns in the guest, then stalled in its first device allocation / UVM channel path before any kernel executed (only a handful of forwarded ops reached the host, `work_guest req=3 ok=3`). Teardown was clean (QEMU stopped, mdev removed, protected region down, host driver healthy, pstore empty). This is the first time a real CUDA program ran that far in the guest against the real backend; it is not yet a green execution. - `MS-20261002-02` **GR context campaign: the promote-side wall is root-caused; the current wall is the shared-management GR object alloc (2026-10-02 ~17:35).** The long-running context-promote wall was the ladder's virtual address space being allocated as *externally owned* (the client owns page tables, VA bookings and engine-context commitment), so the server skipped the engine-context commit and asserted when a promote VA had never been booked. Switching the ladder's address space to *shared management* with a UVM-sized (128 TB) range is a default-off change now in the working tree; it gets past the promote wall entirely - the shared space is accepted and a kernel-managed channel allocates (both firsts on this path) - but the next resource-manager call, the `TURING_COMPUTE_A` compute-object alloc, then stops replying and wedges the server (no reply; the fail-safe holds the protected region). The same alloc wedges on both the kernel-managed and the normal channel variants, so the gate is the server-internal context/global-buffer mapping under shared management, not the channel kind or the promote shape. Since then three more supervised cycles (modes 20, 21 and 22, module `1c69c03b…`) closed out the client-promote shape space on the default external VAS: every kernel-managed-channel variant - single MAIN, VA-only, bare VA+size, `hVirtMemory`, and `GPU_INITIALIZE_CTX` followed by `hVirtMemory` - returns the same soft `NV_ERR_INVALID_STATE` (0x40) with a clean release each time; adding the initialize step does not move it. The five-element engine-class table decode and the Ghidra headless helper scripts are recorded in the handoff (Addendum 25); the V5 path's commit assert `0x52a8b04` is a secondary failure after an earlier non-zero commit status. Current wall: commit the GR engine context and make the engine execute the submitted compute block. Next routes: name the soft-0x40 check in the commit helper, root-cause the V5 assert's first failure, or rehost the promote flow in risc-v-gsp-rm-debugger. All parameters remain default off; the software path is unchanged with them off. The guest-runnable victory client (`tools/cuda-ab/victory_mail.cu`, host-oracle green) has now run under the full physical-backed stack (front bullet above): the guest enumerates and reads device properties, then stalls in its first device allocation / UVM channel path before any kernel executes - the first green guest execution remains the immediate goal. - `MS-20261002-03` **Guest-submission relay and fault relay landed, default off (P6c C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical engine authors the guest CE copy and the release semaphore: the engine-authored release was observed in the guest page (`relays=64`), releases are delivered exactly once (deferred hold), and the relay-on run stayed green (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64). A forced-unresolvable control proved the software fallback recovers every copy exactly once. One relayed copy of 68 released its semaphore but failed the host-side data compare; it was handled fail-closed (software apply) and the parameter stays default-off pending triage, with hardening now in place: retry-once, snapshot capture and relay counters. `gsp_phys_fault_relay=1` relays forwarded-engine faults to the guest channel notifier/event path; it delivered a real CE0 fault live (default off), and the auxiliary event post reported `no_event`. The software path is unchanged with all parameters off. - `MS-20261002-04` **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02 08:29).** The host resource-manager ladder is now selectable to the compute engine (`gsp_phys_rmcall_engine=1`, default off) and brings up a GR0 channel with a `TURING_COMPUTE_A` compute object on the live GSP-RM (`engine=GR0 hGr=0xcaf0002c`), but the first host GR semaphore push faults: the submit timed out (`-110`) with no release and the drained fault records read `CTXNOTVALID` - the external-VAS context was never committed. The copy-engine control run stayed green (`REAL COPY OK`), and both runs released latched and clean (protected region down, distribution driver rebound, exactly 5 power-domain locks, 0 rail cycles, same boot). The follow-up context-promote binding attempt wedged GSP-RM (host assert, fuse-locked WPR2) - see the top bullet. Current wall: commit the GR context and make the engine execute the submitted compute block. Next: host `vectorAdd`, then guest GR forwarding. - `MS-20261002-05` **Experimental tooling: risc-v-gsp-rm-debugger (2026-10-02 08:36, in development).** `debug/risc-v-gsp-rm-vm/` boots GSP/uproc firmware in a RISC-V QEMU VM with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin API and a Ghidra bridge for GSP-RM runtime-memory analysis and emulation-accuracy measurement. It is debug tooling only (no product code changes); the in-development notices were removed 2026-10-02 after the tool audit and value validation; see [`docs/05-operations/03-risc-v-gsp-rm-vm.md`](docs/05-operations/03-risc-v-gsp-rm-vm.md). - `MS-20261002-06` **P6a trigger-path fix landed (2026-10-02 07:10).** The debug copy triggers now resolve the guest page host-physical address at trigger time through the fresh-scratch-line path, retiring the stale-PTE data-check failure (live: `GUEST G2H COPY OK`, `GUEST W2G COPY OK verify_ok=1`, PTE self-test and ladder `REAL COPY OK`, clean release). Forwarding was never affected (HPAs are resolved at submit time). - `MS-20261002-07` **Guest CE copies are hardware-authored with verified data, with deferred exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d; superseded as a frontier by the P6c relay and H1 bullets above).** Wave 1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus `gsp_phys_work_forward=1 gsp_phys_guest_mem=1`): a forwarded guest push's releases are recorded and its GPFIFO entry is held until the hardware copy completes, then the service thread delivers the releases; the software copy still runs. Live: `engaged=75 applied=75 timeouts=0 canceled=0 overflow=0 pending=0`, the CUDA driver-API smoke stayed green (`ALL OK` 64/64), releases exactly once, clean release. Wave 2, with `gsp_phys_work_hw_primary=1` added: the software copy is skipped for forwarded operations, so the physical engine authors the data and the guest reads it back correctly. Live: `copy_skipped=119`, queued guest copies `req=100 ok=100 verify_ok=100 verify_fail=0 fallback=0 timeouts=0`, and the full CUDA smoke stayed green (`matched=64/64`, `ALL OK`). The 19 remaining skipped copies were software-authored inline fallbacks when the single forward slot was busy (`copy_soft=19 copy_fail=0`), the designed fail-closed path; a separate run with the guest-RAM ranges unregistered (p6w2c) proved the fallback again (`copy_skipped=121`, recovered once each by the software copy, `copy_fail=0`, smoke still green). The software fallback therefore remains for any unqueued or failed copy. Residual: the P6a debug-trigger path pre-builds its page-table slots once during ladder stage 3, and a guest page's host physical address can change between translations (observed `0x13a591000` then `0x3b8318000` for the same GPA); a runtime PTE update into an already-fetched 128-byte line is invisible, so the later write-direction trigger failed its data check. Forwarding is unaffected (HPAs are resolved at submit time); the follow-up is fresh scratch lines for the debug triggers (the forwarding path) or pinning the translated page. All parameters default off and the software path is unchanged with them off; release stayed latched in every run (distribution driver rebound, protected region down, exactly 5 power-domain locks held, 0 rail cycles, no reboot). Module sha256 `5fb5bfbe...`. **Next: P6c** (completion/fault relay, real guest page tables/channels, `vectorAdd`) plus the trigger-page pinning follow-up; the wave-1 forwarding bullet below is subsumed by this result. The trigger-path fix, the P6c relay/fault relay and the host GR ladder have since landed (bullets above); the relay data-mismatch triage and the GR execution wall are the open items. - `MS-20261002-08` **Guest CE copies run on the physical engine with verified data (P6b wave 1, 2026-10-02, run p6run13; superseded by the P6b wave-2 result above).** With `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` during a full CUDA driver-API smoke, the sema service captures the first eligible copy per push from the software path's own resolved operands - both real push shapes (a physical emulated-VRAM destination plus a virtual source, and vice versa; a mix of sysmem guest RAM and emulated-VRAM shadow pages) - resolves each side to a host physical address (guest RAM through the QEMU shared-file page cache; shadow pages through the vmalloc-to-physical mapping of the emulated VRAM shadow), maps both sides onto fresh module-built GMMU scratch slots and submits a real copy-engine copy on the module-owned channel. Any miss falls back to the existing ladder-buffer shadow copy; the software pump stays authoritative. **PTE fetch-line rule (learned live):** the engine caches PTE fetches at 128-byte granularity and there is no invalidate path, so a scratch copy pair consumes one whole 128-byte line (16 PTEs) from a pool of 16 zeroed page-table pages built before the first engine access: capacity 512 pairs per boot, and a line is never reused within a boot. **Live:** `req=42 ok=42 verify_ok=42 verify_fail=0 fallback=0 resolve_fail=0 pool_exhausted=0 refused=0` (503 ladder shadow copies also ran); the guest smoke stayed green (`matched=64/64`, `ALL OK`), the P6a guest-page read/write proof and the informational PTE self-test stayed green, and release stayed latched (distribution driver rebound, protected region down, exactly 5 power-domain locks held, 0 rail cycles, no reboot). Shadow-mode limits: the software copy already applied the same bytes, so the hard evidence is acceptance of the translated-HPA submission, the semaphore release and the host-side data-consistency verification (`verify_ok`); a late hardware write can theoretically race guest reuse within the submit window. Module sha256 `9d444908...`. **Next: P6b wave 2** (hardware-primary release relay), then **P6c** (real guest page tables/channels). - `MS-20261002-09` **Guest-page DMA proven on the physical engine (P6a, 2026-10-02; superseded by the P6b wave-1 result above).** The IOMMU/DMA reachability known-unknown is closed on the reference host: the physical GPU's IOMMU group is an identity domain (all-RAM identity) and the mediated device registers as an emulated-IOMMU VFIO device, so its type-1 container keeps guest DMA mappings privately and no kernel `iommu_domain` exists for guest GPAs - `iommu_iova_to_phys` cannot translate them. Guest RAM is instead a QEMU file-backed shared-memory region (`memory-backend-file,share=yes,prealloc=on` under `/dev/shm`); the module opens that file and translates GPA -> HPA through its page cache (`read_cache_page` + `page_to_phys` + `pfn_valid`, behind the active-mdev gate), under the new default-off `gsp_phys_guest_mem` parameter and write-only `gsp-phys-guest-ram` / `gsp-phys-guest-gpa` debugfs controls (the ladder trigger gained host->guest write, guest->host read and a fresh-slot PTE self-test). On the reference q35 4 GiB guest, GPA equals file offset below 4 GiB and offset = GPA - 0x80000000 above it (the 2 GiB 32-bit PCI hole). Channel page-table slots for the guest window are pre-built during ladder stage 3 before any engine access: a PTE written into a page-table line the GMMU had already walked was accepted but never completed, and no TLB-invalidate path exists yet. Live: the physical copy engine copied 4 KiB from a real guest RAM page into a module buffer (G2H) and back into the same page (W2G) with the semaphore released; the guest process observed the write in its own page; the driver-API smoke stayed GREEN `ALL OK` 64/64 and the informational PTE self-test passed. Release stayed latched as designed: distribution driver rebound, protected region down, exactly 5 power-domain locks intentionally held, 0 in-window rail cycles, no reboot. Module sha256 `e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory shadowing, real guest channels, push forwarding) - landed afterwards (superseded by the P6b/P6c bullets above). - `MS-20261002-10` **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved by the P6a result above).** The rung puts guest memory into the real execution path: guest RAM is now a QEMU file-backed `pc.ram` region (`/dev/shm/vgpu-guest-ram`, 4096M, `share=yes`, `prealloc=on`) that the host module can map, under a new `gsp_phys_guest_mem` parameter plus guest-GPA and guest-RAM debugfs interfaces, with a small guest pagemap probe (`tools/test/gpage_test.c`) that publishes its own guest-physical address for a host-side guest-to-host / host-to-guest copy check. The full boot + RM ladder (`REAL COPY OK`) and the latched clean release remain green on every attempt, but the data path is not yet green: the first attempts exposed two concrete host-side defects - the guest physical address resolved to no host translation through the active mediated device's VFIO domain, and the guest-RAM file setup failed with EOVERFLOW because the host-side file open omitted the large-file flag on the 4 GiB shared-memory file (the latter was root-caused in-run). The P6 code was uncommitted at the time of that snapshot; the follow-up attempt resolved both defects (P6a result above). - `MS-20261002-11` **Guest-submission shadow execution (P5 wave 1) landed and verified live (2026-10-02 ~02:00-02:20).** Under the new `gsp_phys_work_forward` parameter (bool, default off) the sema pump requests ONE shadow copy for each eligible CE push through a dedicated ordered workqueue (`gsp-phys-work`), coalescing/dropping requests while one is in flight; the work item runs the ladder channel's module-owned 4 KiB copy via the shared refactored `gsp_phys_backend_ce_copy_submit()` (rm lock only for clear+methods+GPFIFO+ GP_PUT+doorbell, then a lock-free semaphore poll <=200 ms). The software release path remains authoritative and unchanged; refusals/timeouts/drops are counted; no guest memory or guest VA is touched in wave 1. Full guest smoke stayed GREEN `ALL OK` 64/64 while the shadow ran **514 real CE copies** on the physical engine (bytes=4096, elapsed 7-9 ms), with **194 requests dropped while one was in flight, 0 submit refusals, 0 copy timeouts**. Release ran latched as designed: rebound `nvidia` (rc=1), exactly 5 `power domain left LOCKED`, WPR2 down, 0 in-window rail cycles, host stable, no reboot. Module sha256 `83952506...`; code HEAD `82fc7a8` (`56766b2` carries the G0+P2/P3/P5-wave-1 code, `82fc7a8` the STATE-CURRENT/design-doc update). Ladder regression order: trigger 5 (adopt handles) -> trigger 3 (VAS/TSG/ channel, variant V5 engine COPY0) -> trigger 4 (`REAL COPY OK`); trigger 4 alone on a fresh session fails at stage 1 with `NV01_ROOT 0x23`, which is pre-existing expected behavior. **Next: P6** (guest page tables/memory into the real MMU so guest data itself executes on hardware; includes the IOMMU/DMA reachability known-unknown), with optional **P5 wave 2** (hardware-primary release relay with a non-applying decode mode; **subsumed** by P6b wave 2 + the P6c relay above); **P4** remains deferred. - `MS-20261002-12` **Read-only control forwarding (P3) landed and verified live (2026-10-02 01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of side-effect-free queries (`BUS_GET_PCI_BAR_INFO`, `BUS_GET_INFO_V2`, `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) from the live GSP-RM under the new `gsp_phys_ctrl_forward` parameter (bool, default off). The guest's emulated internal handles are translated to the snapshot's adopted server handles before the forward, and the server's echoed params become the guest reply; everything else falls back to the existing emulated handler (fail-open), counted as forwarded/fallback/timeout. A/B on the full guest smoke: forwarding ON stayed GREEN `ALL OK` 64/64 with counters ok=22 fallback=2 timeout=0, zero reply-send failures and `nvidia-smi -q` working; OFF showed zero forward counters and a healthy guest. Release ran latched as designed: WPR2 down, exactly 5 power locks intentionally held, 0 in-window rail cycles, host stable, no reboot. Module sha256 `30dfd311...`. **P5 wave 1 has since landed** (guest submissions shadow-executed on the physical copy engine; bullet above). **P4 is deliberately deferred**: the P2-P5 plan itself states P5 wave 1 does not require it, and adopting real handles would change guest-visible handles and risk the currently green guest path. The P2/P3/P5-wave-1 bundles and the G0 freeze fix are now committed (`56766b2`, `82fc7a8`; HEAD `82fc7a8`). - `MS-20261002-13` **Guest-facing identity wiring (P2) landed and verified live (2026-10-02 01:20-01:45).** On a module-owned live GSP-RM session the emulation overlays the server's real static configuration identity into the guest-facing `GET_GSP_STATIC_INFO` reply under the new `gsp_phys_static_live` parameter (bool, default off): a bounded 2 s wait on the READY-time snapshot, fail-open; the snapshot cache is generation-keyed and invalidated on session changes. A/B on the full guest smoke: with the overlay ON the guest's `nvidia-smi -q` Product Name is `NVIDIA GeForce RTX 2080 Super with Max-Q Design` (live; overlay applied 4x) and with it OFF it is `NVIDIA GeForce RTX 2080 SUPER` (overlay applied 0); the driver-API smoke stays GREEN `ALL OK` 64/64. FB geometry, masks, engine caps, PDE bases and handles remain emulated in P2. One overlay-OFF full run hit a pre-existing emulated-path flake (the guest wedged in `cuInit` with `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`; no host abort); a subsequent OFF quick run was green, so it is not a P2 regression and is being watched. Release ran latched as designed: WPR2 down, distribution driver rebound, exactly 5 power locks intentionally left held, 0 in-window rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3 read-only control forwarding has since landed and verified live (bullet above). At that point guest work was not yet forwarded to the physical engine (the P6b/P6c forwarding results are above). The P2 bundle and the G0 freeze fix have since been committed (`56766b2`). - `MS-20261002-14` **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** The first P1 mdev-owned acceptance cycle ended in the known hard freeze (journal stops, pstore empty): the P1 release path ran outside the supervised D0 pre-lock window and the teardown restored `power/control=auto`, re-arming D3cold; the resume ladder ran 22:32:11-22:32:44 and the host froze at 22:32:53. The same cycle's first async boot failed at FWSEC - the staging DMA never landed after an RTD3 rail cycle - yet the backend was marked READY. The G0 bundle: a new `gsp_phys_repower` parameter (default 0) makes the power unlock latch the D0/D3cold locks instead of releasing them; release gains a 10 s post-rebind readiness gate; claim gains a 2 s D0 settle/verify before the unbind; READY is granted only when the boot proves a live RM session; FWSEC reads back the IMEM/DMEM staging before the kick; and the claim harness leaves the board latched by default (`--restore-power` opts out). Implemented, rebuilt clean, two clean no-param insmod/rmmod smoke cycles at 00:55, then **validated live** (2026-10-02 ~00:57-00:58): two module-owned supervised claim/boot cycles were 2/2 green - claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms), WPR2 down, `released 0000:01:00.0; rebound to nvidia (rc=1)`, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host survived, pstore empty; module sha256 `dde36d92...`. Committed in `56766b2`. - `MS-20261002-15` **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0 (2026-10-02 ~00:58).** A module parameter selects a lifecycle where the physical-GPU claim and boot are deferred to the first mediated device create and released on the last remove, run asynchronously and **fail-open**: if the backend boot fails the device keeps serving the emulation path. The first acceptance cycle verified the deferred claim, the fail-open behaviour (the boot failed at the secure verification stage, yet the smoke test still passed `ALL OK` payload 64/64 while the backend stayed off), and the release on remove; the release/rebind was followed by a host power-management freeze of the known kind (the GPU power-gating hazard), which needed an unplanned reboot; the release path now latches the D0/D3cold locks by default (the new `gsp_phys_repower=0` default in the G0 bundle above), keeping the board powered through the driver resume. Retested with the fixed module, the exact path that hard-froze now **fails open safely**: the first mdev create schedules the async claim, the new READY gate finds no live GSP-RM session (`rm_init_done=0 unload_submitted=0`) and releases the claim latched (5 `power domain left LOCKED`, `released 0000:01:00.0; rebound to nvidia (rc=1)`); the host survives, no freeze. The async FWSEC boot itself still fails (`staging not visible before kick` - DMEM0/IMEM all zero, the falcon is not accepting DMA - aborting before STARTCPU -> `FWSEC/FRTS execution failed: -5`), surfaced in milliseconds by the new pre-kick readback instead of the old 5 s halt timeout; it is an open, fail-open-safe defect off the critical path (an isolation experiment ruled out the obvious hypotheses - stopping `nvidia-persistenced` and setting `power/control=on` plus waiting runtime active before mdev create gives the identical signature, and the module-owned path succeeds with the very same preconditions in the same boot). Guest wiring proceeds on the module-owned live session (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays fail-open. At that point no guest work was forwarded to the physical engine yet (the P6b/P6c forwarding results are above). - `MS-20261001-01` **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session API code is in the tree, inert.** A skeleton that exposes the live physical session to the device layer (session request entry points plus a static-info snapshot helper) was added behind the existing gates and does not yet alter behaviour. In the same window the driver-API baseline was **re-verified green on the current build**: a fresh VM cycle ran the smoke test end-to-end (`ALL OK`, payload 64/64) with real releases (`faked=0 miss=0 stale=0`), so the emulation front is intact ahead of the wiring work. The phased plan for connecting the live session to the guest-facing path is [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md). Post-single-VM-CUDA research notes (concurrent multi-VM virtual functions, 3D acceleration, and a native-vs-guest performance program; and a host-side acceleration feasibility study) were produced and are kept in the local evidence archive (untracked). - `MS-20261001-02` **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes on the physical engine.** The host resource-manager client allocated a virtual-address-space object with a committed page directory, a channel-group object and a real copy-engine channel, bound and scheduled it, obtained a work-submit token, and filed a copy work item in the channel's GPFIFO. The engine's first fetches faulted; draining and decoding the status queue then exposed the wrong virtual address in the fetch, and with that fixed the physical copy engine executed the block: `REAL COPY OK sem=0xc0ffee01 src=0x1170d5000 dst=0x11ccbd000 bytes=4096 GPGet=1 GPPut=1 words=26 token=0x00000001` - a 4 KiB copy with the semaphore released and destination equal to source. Every cycle ran under the supervised harness and released the device cleanly (server shutdown, protected region down, distribution driver rebound); no reboot was needed and the host stayed healthy. The next rung is wiring this live session into the guest-facing mdev/vfio path so guest submissions execute on the physical engine (phased plan: [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). Evidence: the local evidence archive (cycle 18 through cycle 25 runs, with per-wall reports). - `MS-20261001-03` **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM completes initialization on the real TU104 and serves the static-config RPC.** Three supervised cycles, no reboot, clean release each time: 1. the resource-manager log ring is now registered correctly (its libos init-argument id was malformed), which made the server's previously invisible first error readable: a device-name lookup failing because the server's chip device-id was zero (cycle 11); 2. the pre-boot system-information payload now carries the raw PCI config-space identity dwords (`device<<16|vendor`) instead of unpacked 16-bit fields, and the static-config RPC now declares the full reply payload; the server then reports init-done with `rpc_result=0` (cycle 12); 3. the static-config RPC returns the full 1656-byte configuration - GPU name `NVIDIA GeForce RTX 2080 Super with Max-Q Design`, short `TU104-B`, 8 GiB FB, internal client/device/subdevice handles, BAR PDE bases, engine caps - and shutdown runs through the unload RPC (GSP-RM processor suspended in ~220 ms, handoff bits complete) with WPR2 down and the distribution driver rebound (cycle 13). The physical backend is now a live, queryable GSP-RM: the host can boot it, send RM RPCs and receive full replies, and shut it down gracefully. The next rung is to drive the guest-facing emulation from this backend. Evidence: the local evidence archive (cycles 11-13). - `MS-20261001-04` **Driver-API baseline re-verified green on the current build** (2026-10-01 14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause campaign (cycles 4-10) did not regress it. - `MS-20261001-05` **Physical-backend milestone (2026-10-01)**: the module-owned GPU path reached the first successful secure-verification (FWSEC) carve of the protected region on the real TU104 under the verified D0 power-domain harness. The teardown of a FWSEC-only carve is not host-recoverable on this fused part (both the secure-scratch and MMU privilege masks are fuse-locked, so the bootloader handoff cannot be seeded and WPR2 cannot be unlocked from the host), so the fail-safe deliberately held the device and an **approved reboot restored it** (distribution driver rebound, `nvidia-smi -L` green, host healthy, no crash record). The one-way stage is now retired: the only device-write path is a single supervised full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization). That lifecycle is now implemented in the module sources (bootloader runner + generated verification/boot image data) and has run live under the supervised harness: the first real boot succeeded (see the section below), and a second live cycle on the rung-4 build repeated the boot and reached the server's run-sequencer request; the executor's size cap rejected the real request value (16354), which was diagnosed from a raw host-memory read of the message, fixed, and rebuilt. The post-reboot recovery is now verified and the zero-timeout REG_POLL default fix (commit `98b861d`, build `87309e3a…`) was confirmed live. Cycles 2-3 (2026-10-01 11:40 and 11:46) then reached **stage 3e on real hardware**: the real GSP-RM executed the full 165-command/490-dword sequencer and reported RM init failure `0x59` (`NV_ERR_OPERATING_SYSTEM`) on both firmware pairings (the default 570.211.01 image and a verified matching 570.144 image - the image-pairing hypothesis is disproven). Each failed cycle now **releases cleanly back to the distribution driver without a reboot**: the server's own shutdown sets the remaining secure-scratch handoff bit, so all four handoff bits are set and a single `rmmod` clears the protected region, rebinds the distribution driver and releases the power-domain locks. The error-journal capture has since run on ten supervised cycles (each one attempt, each released cleanly with no reboot): the journal was read, and the failure was root-caused to two pre-boot initialization messages the matching driver queues before the kernel kick (a system-information message and a packed registry table) - our boot queue was empty. That handshake is now implemented and **verified live**: the boot consumes a true system-information message and a real registry profile, the registry transport and packing are proven (a display-disable key took effect in the server's journal), the journal collapsed from 29 unique/851 total records to 4 unique/5, the pre-kick GPU timer is programmed (the live timer line was observed, and the generation-correct zero timer offset was confirmed), and the chipset/bridge block of the system information is populated from live values. Init now reaches the server's **static-config collection** and fails there (`0x59`); the device-id override key was exercised with no change, and a display-disable experiment showed display is not the first error. The collector's failing step is not visible in the server journal (its log ring is enabled only after the collector, and the protected region blocks host reads). A source-level review had **closed the host-side input campaign** and named a server-PMU dependency as the one hard requirement. **That verdict was superseded by cycles 11-13**: the log ring was in fact never mapped (its init-argument id was malformed), and once it was registered the failure was a zero server-side device-id caused by our unpacked identity payload - the PMU path itself was fine. The raw config-space identity fix completed initialization (see the milestone bullet above). - `MS-LEGACY-01` **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload roundtripping exactly (64/64 verified in the guest). - `MS-LEGACY-02` **The last two walls**: the GPU page-table walk's address decode (child pointers and leaves were 16x too low; fixed with the real entry encoding - walks now succeed at scale) and the device-to-host copy (a copy-engine class transfer bound on a compute channel whose work was denied by the engine-type gate; trusted host-injected pushes may now execute copy-engine work as well). - `MS-LEGACY-03` **Baseline green and quiet**: the payload roundtrip is back to 64/64 (`ALL OK`), the staging buffer's translation stays resolvable via pins streamed from the guest, and high-frequency diagnostics are now gated behind a default-off debug switch - a normal boot+run produces a ~520-line log instead of a flood (verified twice). The page-directory entry writes have since been captured live and validated (the earlier all-zero boots were operation-skipped/pre-pin boots); the remaining question is the device-address walk gap for the runtime's own buffers. - Recap of the fixes behind this point: all 20/20 compute-channel releases land (three decode paths routed to the exact counter slot); registration delivers the incrementing tracking release for every consumed submission; the copy-engine push buffer is pinned from guest truth on every boot; and the copy-engine decoder consults pinned translations before the scanned map, so every push-data operation executes (`memset=19473`, `ringops=10551` live). - `MS-LEGACY-04` **The runtime client now initializes and reaches kernel launch**: with a static-runtime `vectorAdd` client, `cudaGetDeviceCount`, `cudaSetDevice`, `cudaFree`, three `cudaMalloc`, the HtoD copies and `cudaGetLastError` all return success. The client then parks in `cudaDeviceSynchronize` - the launch is queued but never **executed**, because real execution requires the physical-GPU backend (the next project). - `MS-LEGACY-05` **What fixed it**: the failing call was never a kernel call - it was the clock-domain probe inside libcuda (two device attributes) whose three legacy clock controls were answered with zeros; they are now populated with exactly the fields the helper checks. - `MS-LEGACY-06` **Real-release mode now reaches the same rc=0 chain**: the earlier per-push 8 s stall was two pump bookkeeping bugs (release detection keyed on an unchanged semaphore VA; rewind replays re-entering the CE grace path) - both fixed, and the runtime init completes in ~1 min with real releases. The content rescue also learned the real method encoding (NON_INC/IMMD/ ONE_INC/END_PB headers). - Active work: the owed driver-API smoke-test rerun is now **verified 64/64 (`ALL OK`)** on the current commits, so the recent pump/parser changes did not regress the baseline. The device-claim bring-up verified ownership, firmware staging and the boot layout, but unbinding the distribution driver can power-gate this laptop's dGPU and hard-freeze the host (two occurrences, now root-caused), so that path is disabled behind a two-parameter hazard gate. A power-domain lock that keeps the board in D0 across the handoff is implemented and **verified across three supervised claim/release cycles on 2026-10-01** (zero power-domain cycles, zero failed resumes, distribution driver rebound cleanly each time, host healthy); the verified launch harness holds the GPU and root port in D0 before the distribution driver is stopped, and the root-port hold is the load-bearing invariant. The real GSP boot has since run live on this path (see the 2026-10-01 section below). - Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an operation log are now module parameters, and the test's output survives guest-agent stalls via the serial console. - Guest health note: agent/SSH timeouts during long runs are the guest-side trace reader flooding the ring; they recover on their own, so a first timeout is not a reason to restart the VM. Separately, one no-watcher boot crashed the guest kernel (a NULL callback in interrupt context) before any emulation channel existed - guest-side and untriaged; treat no-watcher runs as a known risk. The host stays stable throughout. - Execution backend (chosen; ownership hazard mitigated and verified): the module owns the physical GPU and will run its real GSP firmware as the resource-manager server, so real work executes with an unmodified host driver and no host daemon; see [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). The ownership path was verified live (release the distribution driver from the device, bind the module, map the BARs, sample the falcon state, then release and rebind the distribution driver); with the device owned, the matching GSP image is fetched and validated (ELF64 RISC-V, loadable image/version/signature sections identified) and the protected-region video-memory/boot layout is computed and logged - all without device writes. The unbind/rebind hazard is mitigated by the D0 power-domain lock, verified over three supervised cycles; the ownership handoff is now also hardened against the distribution driver's udev bind-event power-policy write, and a reusable supervised claim-cycle harness exists (`tools/guest-diag/phys_claim_cycle.sh`). The real-GSP-boot design is complete: exact secure-verification/booter/RISC-V/RM-init sequence and boot artifacts (booter/FMC/SEC2 BL bindata; VBIOS BIT FWSEC blob). The real FB size (8 GiB, decoded from the local-memory-range register - the BAR1-aperture approximation is retired) and the protected-region boot layout have been verified live under the supervised claim harness (three green claim/release cycles; the root-port D0 hold is the load-bearing invariant), and the secure-verification stage (VBIOS security-blob extraction, falcon reset and staging) was validated offline and in a read-only dry-run, then run **live under the supervised claim harness**: it raised the protected region (WPR2 up) for the first time on real hardware. Its release teardown, however, could not clear WPR2, and the fail-safe deliberately held the device (distribution driver unbound, power-domain locks held) rather than hand it back; a rescue path is being debugged (or, failing that, an approved reboot). A distribution-driver bus-master quirk on the claim path was also found and fixed in this run. The bootloader/RISC-V/resource-manager-init stages are now implemented in the module sources (bootloader runner plus generated boot and verification image data) and have now run live: the boot completed through an active RISC-V core and the module received the first real GSP-RM message (see the section below); a clean release is pending the server shutdown RPC. - Host stability: the host hang class observed on 2026-09-28 (kernel stack overflow in a channel snapshot path) is fixed and frame-audited; repeated VM load/attach/teardown cycles run without panics or pstore records. A second, distinct freeze class was root-caused on 2026-09-30: the dGPU board's power rail is gated by an ACPI power resource behind its root port, released through an embedded-controller command. Unbinding the distribution driver (as the physical-GPU claim path does) lets the device runtime-suspend, the rail is cut, and the re-bind intermittently fails to power it back (`not ready ... after resume`), with the host hard-freezing about a minute later (no crash record; journal stops, pstore empty; both occurrences followed an unbind/rebind, and prior log history has no other instance). The claim path refuses to run unless both `gsp_phys_gpu=1` and `gsp_phys_hazard_ack=1` are passed, and the power-domain lock (hold the GPU and its board functions plus the root port in D0 across the handoff, aborting and holding the locks on any power-loss signature) is now **implemented and verified across three supervised claim/release cycles** (2026-10-01: zero power-domain cycles, zero failed resumes, clean rebind, host healthy). The verified launch harness holds the GPU and root port in D0 before the distribution driver is stopped; the root-port hold is the load-bearing invariant. The path must still not run unattended and every cycle is supervised. The device has since been restored by the approved reboot (distribution driver rebound, GPU healthy); the secure-verification stage is now folded into a single full boot lifecycle, and normal module/VM work does not touch the driver binding and is unaffected. See [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). --- ## Evidence pointers - **Module sha256 (latest runs):** no-claim incident run `30a13ab4...` (2026-10-03, emulation-only acceptance); earlier `e5318ee0...` (large-push parser fix), P6b `5fb5bfbe...`/`9d444908...`, P6a `e1bce651...`, P3 `30dfd311...`, P2 `69eac176...`, G0 `dde36d92...`, P5 wave 1 `83952506...`. The host was rebooted 2026-10-03 ~20:11 (previous boot `12cf3a41-...`); the module was rebuilt 21:10:57 and reloaded 21:12 with the post-incident no-claim fixes. **2026-10-05 onward:** acceptance builds `e7ff3bf7` (hold all unreadable) and `cb1724d3` (per-type VRAM); GR rung-2 builds `19d1e0fc`, `1783dade`, `c998abef`, `b6a9d4eb`, `9075bf57`, `69500d6e` (grctx1-6), `c5c3f396` (grv6a), `3bf3e2b2...` (mode 23; committed `4212cdb8`), `c905e9da` (grgold22), and `f17b35d6...` (grgold23, golden replay + on-ladder 3D; uncommitted in `gsp_phys_booter.c`, `gsp_phys_gpu.[ch]`). GR reports: `backups/evidence-20261005-grctx/REPORT-GR-PROMOTE-COMMIT.md` (+ Addendum A), `REPORT-GR-CTX-SOURCE.md`, `REPORT-GR-EVIDENCE-DECODE.md`, `REPORT-GR-FORWARDING-DESIGN.md`, `NEXT-BOOT-GR-RUNBOOK.md`; run logs `backups/evidence-20261002-gr/gr-run-grctx{1..6}.log`, `gr-run-grv6a.log`, `gr-run-grv23.log`, `gr-run-grv23v6.log`, `gr-run-grgold22.log`, `gr-run-grgold23.log` with the `06/20/21/30/31/32-*` capture sets, plus `backups/evidence-20261002-victory/run-noclaim-reb2.log`. HEAD `072b13ba` (check-back commits; `master` in sync with `vgpu/master`); the working tree carries the uncommitted golden-replay edits (`gsp_phys_booter.c`, `gsp_phys_gpu.c`, `gsp_phys_gpu.h`). - **Campaign detail:** the private live campaign narrative (handoff/logs, maintained outside this repository) is authoritative; this page is the dated public snapshot. - **Local evidence archive:** `open-iov/backups/evidence-20261002-victory/` (`*-victory8/9`, `14-guest-dmesg-fail-*`, `20-gsp-phys-state-*`); earlier cycle 11-25 runs with per-wall reports (local archive, untracked). - **Simulator sessions:** `debug/risc-v-gsp-rm-vm/dumps/gsp-boot-b1-run1/2`, `b2-run1/2`, and `debug/risc-v-gsp-rm-vm/harness/qemu_line/` (docs/36 §4, docs/37 §4.3). - **Definition of done:** `docs/04-implementation/01-known-knowns-and-unknowns.md` §4 + the open-gates section (archived below). ## Archive - historical narrative (pre-2026-10-02) The sections from here down are the dated historical record kept for evidence; the current state is in the top sections of this page. The 2026-10-02 milestone-ladder mega-summary is preserved verbatim below. ### Preserved mega-summary (2026-10-02 milestone-ladder cell, verbatim) > | CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles; the 2026-10-01 P1 mdev-owned release path initially ran outside that pre-lock window and re-armed D3cold, freezing the host at 22:32:53 - the G0 fix bundle now latches the D0/D3cold locks by default; see the frontier section), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle > completed a **clean release without a reboot**. Cycles 4-10 then captured > the resource-manager's own error journal and root-caused the failure: the > boot queue was missing the two pre-boot initialization messages the matching > driver queues before the kernel kick (a system-information message and a > packed registry table). That handshake is now implemented and **verified > live**: the registry transport and packing are proven (a display-disable key > took effect in the server's journal), the GPU timer is programmed before > the kick, and the chipset/bridge block of the system information is > populated from live values. Init now reaches the server's **static-config > collection** and fails there (`0x59`); the device-id override key was > exercised with no change, and a display-disable experiment showed display is > not the first error. A source-level review of the failing collector has now > closed the host side: it reads none of the remaining host-supplied fields > (the one registry key on its call graph can only clear an error), so the > ACPI/workspace/zeroed-field leads are retired. The one hard dependency left > runtime is enabled only after the collector and the protected region blocks > host reads). **Cycles 11-13 resolved this wall**: the resource-manager log > ring had never actually been mapped (its libos init-argument id was > malformed), which is why the first error was invisible; with the ring live > the failure traced to the server's chip device-id reading as zero because > the pre-boot identity fields were sent unpacked instead of as raw PCI > config-space dwords. Fixing the payload made the server report init-done > with a zero result and serve the full static-config reply (GPU name, 8 GiB > FB, internal handles, BAR PDE bases, engine caps), and graceful shutdown > works through the unload RPC. Each of cycles 4-13 released the device > cleanly with no reboot. Since 2026-10-01 17:24 the next rung has both code > and live results: a host resource-manager client creates memory and > channel-group objects on the booted server, allocates and programs the > page-directory base of a virtual-address-space object, and **allocates a real > channel** on the live server. In cycles 18-25 it went all the way to > execution: the channel was bound and scheduled, a work-submit token was > obtained, a copy work item was placed in its GPFIFO and the doorbell rung, > and after the engine's fetch and method-path faults were diagnosed and fixed > the physical copy engine ran it - a 4 KiB copy with the semaphore released > and destination equal to source. Every cycle released the device cleanly > with no reboot; the next rung is wiring the live physical session into the > guest-facing mdev/vfio path > ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the fail-safe held the device until an approved reboot restored it (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | ## First real GSP boot + live GSP-RM session (2026-10-01) Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`), `vgsp.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order observed on the physical TU104: 1. **FWSEC** — WPR2 came up; FRTS error code 0; `WPR2_LO=0x01ffe000`, `WPR2_HI=0x01ffee00`. 2. **Booter Load** on SEC2 — halted with `M0=0x0` after 146 ms. 3. **RISC-V active** — `RISCV_STATUS=0x00000035` (active bit 0), `FALCON_OS=0`; secure scratch14 raw `0x11000000` (booter-load handoff bit 24 = 1, booter binary version 1). 4. **First real GSP-RM message** — the staged status queue linked and the server's first event was `function=0x1002` (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, `rpc_result=0`), carrying a command buffer the host must execute; `GSP_INIT_DONE` (0x1001) follows only after that sequencer runs and was not observed in the 60 s poll. This was the first host↔real-GSP-RM exchange on the physical path. At that point a clean release was not yet possible: the Booter Unload gate requires all four secure scratch14 handoff bits (24–27); a host-driven boot set only bit 24, and bit 27 was expected from the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`, not yet implemented then). The host-side seed/unlock alternative is fuse-locked on this part. The session was therefore intentionally left claimed (`--keep`): the GPU was bound to `open-iov-gsp-phys` and runtime-active; `nvidia` stayed unbound (host headless; `nvidia-smi` fails by design); host healthy, pstore empty. A fresh live cycle needed a power cycle at that point. **Status update (cycles 2-3, same day):** this limitation is retired - the server's shutdown set the remaining handoff bit and a single `rmmod` released the device without a reboot (see the frontier section). **Next steps:** (a) execute the 0x1002 sequencer command buffer in the module and log 0x1001 as `stage 3e complete`; (b) send `UNLOADING_GUEST_DRIVER` on the command queue so the server shuts down, sets the remaining handoff bit and teardown can hand the GPU back without a reboot; (c) prove a request/reply round trip with the server. All three were implemented, built and have now run live: cycles 2-3 executed the sequencer and reached init-done, and the server's own shutdown supplied the missing handoff bit, so each failed cycle released without a reboot; a request/reply round trip remains to be exercised. The module sources contain the sequencer executor (BAR0 register-write/modify/poll, delay, store and core-resume opcodes), a generalized msgq producer (multi-element records, checksum/sequence/free-space rules, doorbell) with the unload RPC, a reply-matching status-queue reader, and a debugfs probe that requests the server's static configuration after init-done; the rung 4a/4b/4c code passed an adversarial wire-format review plus its fixes. **Prepared for the next cycle (now history):** an early build (`f61308bd…`) ran the boot green but the server's run-sequencer request was rejected by an over-strict host-side cap on the request's own buffer size. The message was read raw from host memory, the cap corrected (build `87309e3a…`), and the sequencer then ran live in cycles 2-3 (see the frontier section). Ending a claimed session: `sudo rmmod vgsp`, then `echo auto | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the same for `0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. Since cycles 2-3 this passes the handoff gate (the server's shutdown sets the last handoff bit) and completes without a power cycle; if the protected region survives teardown the fail-safe keeps the locks and a power cycle is still required, with approval. ## Initialization, work and teardown path Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, `[not yet]`. ``` HOST SIDE GUEST (stock driver, unmodified) ========= ================================ [done] build + load vgsp.ko | (module first: exports the symbols the NVIDIA open modules link against) v [done] mdev create ------------------------------------> / | BAR/PCI/MSI-X emulation; GSP emulation thread v [done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap | v [done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter | v [done] GSP-RM message queues / RPC / controls ---------> object + client setup | v [done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB | (two-GRCE fix; async copy engine) v [done] ------------------------------------------------> nvidia-smi -L enumerates TU104 | v [done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet | v [done] ------------------------------------------------> cuCtxCreate | UVM register GPU; channel manager | GR + CE channels; push buffers | semaphore pool; completion counters v [done] ------------------------------------------------> cuMemAlloc + cuMemcpy roundtrip | (passes end-to-end; payload 64/64 correct) [partial] ---------------------------------------------> full nvidia-smi output / kernel launch | (runtime init rc=0; launch queued; | execution awaits the backend) ``` Work execution (host service vs client synchronization): ``` [done] host decodes guest pushes (method streams): memsets, copies, releases - CE pushes execute at scale; the compute-class inline-DMA HtoD and the copy-engine-on-compute DtoH both execute; payload 64/64 - the runtime client now initializes (rc=0) and queues a kernel launch; the queued compute pushes decode but are not executed [partial] GPU VA -> guest PA translation: page-table walk fixed and succeeding; server-owned page-table population still in progress [done] real work executes on the physical GPU: the host RM client's copy engine block runs on the physical CE - 4 KiB copied, semaphore released, destination equals source [partial] guest-submitted work relayed to the physical engine: eligible guest CE copies execute on hardware with hardware-authored data, deferred exactly-once release, engine-authored release semaphores and forwarded-fault relay (all default off; one relay data-mismatch under triage; software fail-closed fallback remains); a host GR ladder allocates a GR channel + compute object but the first submit faulted `CTXNOTVALID` and the promote fix wedged the server (fail-safe held the device until an approved reboot restored it) - real guest page tables/channels and GR execution remain [not yet] correct results (vectorAdd) ``` Teardown and recovery: ``` [done] happy-path teardown exercised end-to-end (cuCtxDestroy OK) [done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached [done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited ``` Recent chain of walls removed (2026-09-28 to 2026-09-30): ``` host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild) -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20) -> stale semaphore-pool translation -> fake-only proven insufficient -> push buffer found and validated -> per-page pins -> authoritative pin table -> deterministic chain solved -> pin-drop and release-push fixes -> copy-engine data path executes -> compute-window pinning -> registration release delivery -> 20/20 counter slots -> GR-info clock fix -> cuCtxCreate OK; payload found (compute-class inline DMA) -> GMMU walk decode fixed (child/leaf encoding) -> inline HtoD lands -> DtoH copy-engine-on-compute allowed -> payload roundtrip 64/64 -> runtime client initializes rc=0 (legacy clock-domain fix) -> kernel launch queued -> real-release chain rc=0 (pump bookkeeping fixes) -> execution backend bring-up -> real GSP-RM session -> host RM client -> real CE copy on the physical engine -> guest CE copies hardware-authored with engine-authored releases -> forwarded-fault relay -> host GR ladder -> GR ctx CTXNOTVALID fault -> promote-ctx server assert + designed locked fail-safe (cleared by an approved reboot) -> GR execution is the next wall ``` ## Tools, code paths and fixes behind this progress ``` KERNEL MODULE (vgsp.ko) GUEST-SIDE TOOLS (tracked) ================================ ========================== gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py tools/guest-diag/guest_kprobes.sh DATA STRUCTURES DEBUG PLANES =============== ============ channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset / notifier, ring) ringops / nomap / skip / faked / VA->PA map (merge rebuild, hot pages, cache) pins / ipins pin table + authoritative injected pin table debug params: sema_probe, sema_counter, completion counter table (per channel) sema_map_add, chan_dump, shadow_dump semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc//pagemap client trackers / RM error record chain marker-based /dev/kmsg parsing ``` Experimental developer tooling (not part of the product path): **risc-v-gsp-rm-debugger** (`debug/risc-v-gsp-rm-vm/`) boots GSP/uproc firmware in a RISC-V QEMU VM with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin API and a Ghidra bridge for GSP-RM runtime-memory analysis and emulation-accuracy measurement. Status is tracked by the tool audit (`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`); the in-development notices were removed 2026-10-02 after the then-194-test suite and end-to-end value demonstrations (`debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md`); the 2026-10-03 Phase S wave (device model `B0`-`B2`, A/B line, triage; 290 tests) is summarised in the "Simulator workstream" section above (docs 35-40). See [`docs/05-operations/03-risc-v-gsp-rm-vm.md`](docs/05-operations/03-risc-v-gsp-rm-vm.md). | Wall | Fix | |---|---| | Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build | | Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels | | VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace | | Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits | | Four-word semaphore releases ignored | decode both release encodings | | Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed | | UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot | | Acknowledging releases without executing work | proven insufficient; pushes must be read and executed | | Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table | | Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe | | Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins | | Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot | ## What "CUDA works" means (open gates) Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) section 4, a defensible claim needs **all** of: 1. every guest-to-device message for the CUDA path answered or forwarded, with coherent replies for consumed fields; 2. every work VA translatable and server-owned page-table ranges populated; 3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g. `vectorAdd`'s `<<>>`) actually executed on the physical GPU's compute engine - not merely acknowledged - and copies executing on real hardware with correct results; 4. completion (semaphores, events, faults) behaving as the guest expects; 5. at least two driver families passing; 6. tenant isolation; 7. measured performance within an agreed factor of native. Until then the honest statement is: **initialization works; execution is in progress.** ## Deferred goals - Host NVIDIA driver (`nvidia.ko`) over a virtual function — host OS acceleration / host display from the open-iov device model — is a **post-1.0 stretch goal we are still only considering**. We may work on it at some point; we will probably take a crack at it because it is a great feature, but **do not block on it, pursue it, or design around it**. It is strictly lower priority than every baseline goal (guest CUDA execution, multi-VM, 3D, Windows, isolation, performance, portability). It is **never pursued automatically**; it requires a human decision once we are certain there is no risk — which we are not certain of now — and it may be scrapped on architectural grounds. Treat it until further notice as **do not implement**. - First design (`13-host-vf-acceleration-design.md`) was written and then **reverted on 2026-10-02** after independent reviews falsified its load-bearing mechanisms; all associated doc edits were reverted. - The corrected record, candidate approaches, experimental gating policy and revisit criteria are in the post-1.0 document [`docs/10-post-1.0/experimental-host-acceleration.md`](docs/10-post-1.0/experimental-host-acceleration.md) (intentionally not linked from the docs front door and not part of the active reading order; don't index it and don't design toward it). - Earlier verdict and options: [`docs/03-architecture/12-host-vf-acceleration-future-goal.md`](docs/03-architecture/12-host-vf-acceleration-future-goal.md). ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. - Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages the physical GPU; `vgsp.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. - Firmware: TU104 GSP 570.144 (public release). ## Build and load order Build: host module first (it generates the symbol versions the NVIDIA open kernel modules link against), then the NVIDIA open kernel modules, then userspace. Load: VFIO prerequisites, then `vgsp.ko`, then create the mdev and attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the device. ## See also - [`docs/README.md`](docs/README.md) - documentation front door - [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done - [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point - [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session) - [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags - [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail - [`docs/05-operations/03-risc-v-gsp-rm-vm.md`](docs/05-operations/03-risc-v-gsp-rm-vm.md) - risc-v-gsp-rm-debugger GSP RISC-V VM runtime analysis (status: docs/27 tool audit, docs/25 value validation) --- *Refresh this page on every progress check (snapshot time in the header); prefer moving detail into the `docs/` tree and keeping this page short.*