vgsp: guest GR forwarding backend and capture (gsp_phys_gr_forward, default off)

Add a default-off guest GR (compute) forwarding path so a guest CUDA
launch submitted on its GR channel can be executed by the physical GR
engine on the ladder channel, with fail-closed fallback to the host CUDA
bridge and the unchanged software release path.

- New module parameter gsp_phys_gr_forward (default 0) and accessor.
- New GR execution gate: boot_stage.rmcall_gr_exec_ok, latched only after
  a real GR submission logs REAL GR SEM OK; the backend refuses while it
  is unset.
- Sema capture: a completed 64-word inline QMD on a trusted, non-replay
  GR decode becomes an svc_gr_cand (push identity, decoded release0,
  parameter/program/pointer fields, up to 8 captured inline-write
  records); the CUDA bridge stays authoritative while the parameter is
  off and is reached in order forward -> bridge -> software release.
- Backend: gsp_phys_backend_guest_gr_submit() maps pages at their own
  guest VAs through fresh Tier-1 small-PT pages/PDE installs or relocates
  inline-captured pages to ladder shadow pages (Tier 2), builds
  SET_OBJECT + the inline-QMD method run + REPORT_SEMAPHORE on the
  compute subchannel, reuses the shared GPFIFO/USERD/doorbell publish
  tail (extracted from the CE submit path, CE bytes unchanged) and polls
  the QMD release and/or module semaphore, draining the epoch-gated
  fault queue on timeout.
- Sema wiring: GR requests reuse the INFLIGHT/hold/deferral machinery so
  the guest channel release applies only on the submit's success/failure
  edge; hold shims preserve the channel engine type.
- Counters and a work_gr state line plus a stats accessor.
- Bounds are fixed: 16 pages, 8 inline records, 64 QMD words; every
  pointer/size checked, pool exhaustion and stale mappings fail closed.
- New ABI layouts pinned with static_asserts.

Compile-only change: no device test performed.

(cherry picked from commit ce40bd872ea05776156a68afcc6d4571a0bbcad2)

1 files changed, 18 insertions, 0 deletions

diff --git a/kernel/main.c b/kernel/main.c
index f662b94..258de41 100644
--- a/kernel/main.c
+++ b/kernel/main.c
@@ -174,6 +174,7 @@ static rm_vgpu_vfio_ops_t g_rm_vgpu_vfio_ops;
 static struct mdev_parent *g_mdev_parent = NULL;
 
 static bool gsp_phys_gpu_param;
+static bool gsp_phys_gr_forward_param;
 
 /* VFIO ops implementation */
 static NV_STATUS nv_vgpu_vfio_probe(struct pci_dev *pdev, NvU32 vgpu_type_id, NvU32 *vgpu_id);
@@ -2596,6 +2597,23 @@ bool gsp_phys_gpu_enabled(void)
 }
 
 /*
+ * GR (compute) forwarding: when set, a guest GR channel's decoded inline-QMD
+ * launch is captured by the sema service and submitted as real work on the
+ * ladder's physical GR channel instead of (or before) the host CUDA bridge.
+ * Default off: every path is inert and the bridge/software emulation stays
+ * byte-identical. The backend additionally refuses until a real GR
+ * submission has proved out at boot (boot_stage.rmcall_gr_exec_ok), so the
+ * parameter alone can never make the module pretend GR works.
+ */
+module_param_named(gsp_phys_gr_forward, gsp_phys_gr_forward_param, bool, 0644);
+MODULE_PARM_DESC(gsp_phys_gr_forward, "Forward guest GR (compute) launches to the physical GR engine on the debugfs ladder channel (default 0; requires gsp_phys_gpu + a proven GR ladder submission)");
+
+bool gsp_phys_gr_forward_enabled(void)
+{
+	return gsp_phys_gr_forward_param;
+}
+
+/*
  * GR ring discovery: pin the compute channels' rings/USERDs, which follow the
  * CE rings in the channel video-memory allocation. Default on; set 0 to leave
  * GR channels untranslated when debugging push-execution side effects.