vgsp: fail4 - route pending-pass releases through opdef withholding (cold-run gap)

The fwd-defer arm took priority over svc_opdef_release, so a pass whose op
was still pending recorded its fence release in svc_fwd_rel (applied at the
hold break / fallback) instead of the opdef slot - the fence fired while
the copy was still owed, the driver refilled the ping-pong staging, and the
late capture/read got refilled bytes (cold-run vector_add gap: one to four
chunks per cold boot). svc_fwd_defer_arm() is now skipped when
svc.op_tr_pending, so those releases go through svc_opdef_release and apply
only when the retried pass has no pending op (the op has run by then).

Measured cold-run vector_add bad across this iteration: 193536 -> 100352
(r323) -> 479232 (r324 wrong queue-pending experiment, reverted) -> 66560
(r325, ~1 chunk). All other acceptance items green on r325 (mail OK, smi OK,
matmul OK, transformer OK).

1 files changed, 21 insertions, 1 deletions

diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c
index f55502b..4c063db 100644
--- a/kernel/gsp_sema_service.c
+++ b/kernel/gsp_sema_service.c
@@ -8050,7 +8050,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf,
 					or.va = va;
 					or.payload = payload;
 					or.entry_idx = entry_idx;
-					if (svc_fwd_defer_arm()) {
+					if (svc_fwd_defer_arm() &&
+					    !svc.op_tr_pending) {
 						svc_opdef_drop(vgpu, ch,
 							       entry_idx);
 						if (svc_fwd_rel_n < SVC_FWD_REL_MAX) {
@@ -9573,6 +9574,25 @@ void gsp_sema_service_guest_reset(void)
 }
 
 /*
+ * r317: queue a pass's recorded forward releases through the pending
+ * machinery instead of applying them at a hold break / fallback while an
+ * op of that pass is still owed. Applying the fence early let the driver
+ * refill the ping-pong staging before the copy read it (the cold-run
+ * capture gap: vector_add run 1 of every cold boot, last ~2 chunks). The
+ * pending store applies each release once the word resolves (the copy has
+ * run by then, since the op execution is what clears the pending state on
+ * the retry).
+ */
+static void svc_fwd_rel_queue_pending(struct svc_fwd_rel *rel, u8 n)
+{
+	u8 i;
+
+	for (i = 0; i < n; i++)
+		svc_pending_add(rel[i].va, rel[i].payload,
+				rel[i].inc_claimed ? 1u : 0u);
+}
+
+/*
  * Loud, bounded telemetry for a starved deferred release (ISSUE-0001).
  * Called from the two "no translation yet, keep" paths of
  * svc_apply_pending() with an incremented try count; logs at 4/64/1024