gsp-phys: P6b waves 1.5 and 2 - deferred release and hardware-authored guest copies

- new default-off gsp_phys_work_defer_release: a forwarded guest push's
  releases are recorded and its GPFIFO entry is held (no consume, no
  GP_GET write, no counter publish/stall) until the work item publishes
  completion; the service thread then delivers the releases, with a
  fallback on failure/timeout (500 ms cap enforced every pump pass)/
  pause/quiesce/channel-delete
- new default-off gsp_phys_work_hw_primary: the software copy is skipped
  for forwarded ops, so the physical engine authors the data; a once-only
  software copy runs before the releases on any hardware failure or when
  the request cannot be queued
- release-apply helpers extracted; exactly-once latches; channel-delete
  and teardown handling; counters and a work_guard state line
- live: deferred-release run engaged=75 applied=75 timeouts=0 with the
  full smoke green; hardware-primary run req=100 ok=100 verify_ok=100
  fallback=0 copy_skipped=119 copy_fail=0, smoke ALL OK 64/64 (guest read
  data the engine authored), with 19 fail-closed inline software
  fallbacks under single-inflight contention; a range-unregistered run
  proved the fallback (121 skipped copies, all recovered, smoke green)
- defaults off; the software path and all existing guards unchanged

1 files changed, 43 insertions, 7 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index c3c307d..b53145b 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -69,7 +69,7 @@ meant to execute on the physical GPU.
    and destination equal to source. Every cycle released the device cleanly
    with no reboot; the next rung is wiring the live physical session into the
    guest-facing mdev/vfio path
-   ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware with host-side data consistency verified, the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c) remain, **P6b wave 2** (hardware-primary release relay) is next, and P4 (alloc/free handle adoption) remains deliberately deferred. |
+   ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. |
 | Graphics (3D), video | ❌ |
 | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
 
@@ -81,11 +81,46 @@ chain succeeds.
 
 ## Right now (frontier)
 
+- **Guest CE copies are hardware-authored with verified data, with deferred
+  exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d).** Wave
+  1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus
+  `gsp_phys_work_forward=1 gsp_phys_guest_mem=1`): a forwarded guest push's
+  releases are recorded and its GPFIFO entry is held until the hardware copy
+  completes, then the service thread delivers the releases; the software copy
+  still runs. Live: `engaged=75 applied=75 timeouts=0 canceled=0 overflow=0
+  pending=0`, the CUDA driver-API smoke stayed green (`ALL OK` 64/64),
+  releases exactly once, clean release. Wave 2, with
+  `gsp_phys_work_hw_primary=1` added: the software copy is skipped for
+  forwarded operations, so the physical engine authors the data and the guest
+  reads it back correctly. Live: `copy_skipped=119`, queued
+  guest copies `req=100 ok=100 verify_ok=100 verify_fail=0 fallback=0
+  timeouts=0`, and the full CUDA smoke stayed green (`matched=64/64`,
+  `ALL OK`). The 19 remaining skipped copies were software-authored inline
+  fallbacks when the single forward slot was busy (`copy_soft=19
+  copy_fail=0`), the designed fail-closed path; a separate run with the
+  guest-RAM ranges unregistered (p6w2c) proved the fallback again
+  (`copy_skipped=121`, recovered once each by the software copy,
+  `copy_fail=0`, smoke still green). The software fallback therefore remains
+  for any unqueued or failed copy. Residual: the P6a
+  debug-trigger path pre-builds its page-table slots once during ladder stage
+  3, and a guest page's host physical address can change between translations
+  (observed `0x13a591000` then `0x3b8318000` for the same GPA); a runtime PTE
+  update into an already-fetched 128-byte line is invisible, so the later
+  write-direction trigger failed its data check. Forwarding is unaffected
+  (HPAs are resolved at submit time); the follow-up is fresh scratch lines for
+  the debug triggers (the forwarding path) or pinning the translated page. All
+  parameters default off and the software path is unchanged with them off;
+  release stayed latched in every run (distribution driver rebound, protected
+  region down, exactly 5 power-domain locks held, 0 rail cycles, no reboot).
+  Module sha256 `5fb5bfbe...`. **Next: P6c** (completion/fault relay, real
+  guest page tables/channels, `vectorAdd`) plus the trigger-page pinning
+  follow-up; the wave-1 forwarding bullet below is subsumed by this result.
 - **Guest CE copies run on the physical engine with verified data (P6b wave 1,
-  2026-10-02, run p6run13).** With `gsp_phys_work_forward=1
-  gsp_phys_guest_mem=1` during a full CUDA driver-API smoke, the sema service
-  captures the first eligible copy per push from the software path's own
-  resolved operands - both real push shapes (a physical emulated-VRAM
+  2026-10-02, run p6run13; superseded by the P6b wave-2 result above).**
+  With `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` during a full CUDA
+  driver-API smoke, the sema service captures the first eligible copy per push
+  from the software path's own resolved operands - both real push shapes (a
+  physical emulated-VRAM
   destination plus a virtual source, and vice versa; a mix of sysmem guest RAM
   and emulated-VRAM shadow pages) - resolves each side to a host physical
   address (guest RAM through the QEMU shared-file page cache; shadow pages
@@ -631,8 +666,9 @@ Work execution (host service vs client synchronization):
            released, destination equals source
  [partial] guest-submitted work relayed to the physical engine: eligible guest
            CE copies execute on hardware through module-built scratch mappings
-           with host-side data verification (shadow mode; the software release
-           stays authoritative); real guest page tables/channels (P6c) remain
+           with hardware-authored data, host-side verification and deferred,
+           exactly-once release (software fail-closed fallback remains); real
+           guest page tables/channels (P6c) remain
  [not yet] correct results (vectorAdd)
 ```