gsp-phys: P6b waves 1.5 and 2 - deferred release and hardware-authored guest copies

- new default-off gsp_phys_work_defer_release: a forwarded guest push's
  releases are recorded and its GPFIFO entry is held (no consume, no
  GP_GET write, no counter publish/stall) until the work item publishes
  completion; the service thread then delivers the releases, with a
  fallback on failure/timeout (500 ms cap enforced every pump pass)/
  pause/quiesce/channel-delete
- new default-off gsp_phys_work_hw_primary: the software copy is skipped
  for forwarded ops, so the physical engine authors the data; a once-only
  software copy runs before the releases on any hardware failure or when
  the request cannot be queued
- release-apply helpers extracted; exactly-once latches; channel-delete
  and teardown handling; counters and a work_guard state line
- live: deferred-release run engaged=75 applied=75 timeouts=0 with the
  full smoke green; hardware-primary run req=100 ok=100 verify_ok=100
  fallback=0 copy_skipped=119 copy_fail=0, smoke ALL OK 64/64 (guest read
  data the engine authored), with 19 fail-closed inline software
  fallbacks under single-inflight contention; a range-unregistered run
  proved the fallback (121 skipped copies, all recovered, smoke green)
- defaults off; the software path and all existing guards unchanged

1 files changed, 27 insertions, 1 deletions

diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h
index 1d1db64..70b98e4 100644
--- a/kernel/gsp_phys_gpu.h
+++ b/kernel/gsp_phys_gpu.h
@@ -84,7 +84,9 @@ bool gsp_phys_ctrl_forward_enabled(void);
  * guest's CE pushes stay software-emulated exactly as before). When set, the
  * sema service shadows each decoded CE push as one copy on the debugfs ladder's
  * real channel using only module-owned ladder buffers; the software release
- * remains authoritative and is applied unchanged either way.
+ * remains authoritative and is applied unchanged either way, unless
+ * gsp_phys_work_defer_release moves a forwarded push's release after its
+ * hardware copy (see below).
  */
 bool gsp_phys_work_forward_enabled(void);
 
@@ -98,6 +100,30 @@ bool gsp_phys_work_forward_enabled(void);
 bool gsp_phys_guest_mem_enabled(void);
 
 /*
+ * True when the gsp_phys_work_defer_release module parameter is set (default
+ * 0: a forwarded guest CE push's release is applied inline with the software
+ * copy, byte-identical to wave 1). When set with gsp_phys_work_forward and
+ * gsp_phys_guest_mem, the release(s) a forwarded push decoded are held on the
+ * service thread until the hardware copy completes, its bounded timeout
+ * expires, or the forwarding is aborted; they are then applied from the
+ * service thread. One-line rollback: gsp_phys_work_defer_release=0.
+ */
+bool gsp_phys_work_defer_release_enabled(void);
+
+/*
+ * True when the gsp_phys_work_hw_primary module parameter is set (default
+ * 0: byte-identical to wave 1.5, the software copy always runs inline). When
+ * set with gsp_phys_work_forward, gsp_phys_guest_mem and
+ * gsp_phys_work_defer_release, a forwarded guest CE copy skips the software
+ * copy (the hardware authors the data) and the release still lands only
+ * after hardware completion through the wave-1.5 hold. Any hardware failure,
+ * timeout, abort or failed guest queue falls back to exactly one software
+ * copy in the service thread before the releases are applied, so the guest
+ * always reads correct data. One-line rollback: gsp_phys_work_hw_primary=0.
+ */
+bool gsp_phys_work_hw_primary_enabled(void);
+
+/*
  * Translate a guest GPA to the pinned host PA (gsp_phys_guest_mem ladder
  * support). The mdev is registered with vfio_register_emulated_iommu_dev():
  * its VFIO type-1 container keeps the guest DMA mappings privately, so