r456 keep the copy staging snapshot alive across opdef retries (the r455 operand-shift root cause)

The fail4 pipeline-order snapshot (captured when a device-dst H2D copy is
held for its operand pins) was freed by svc_opdef_clear on the APPLY path
on every re-decode pass of the held entry. The retried copy therefore
re-captured from the driver's REFILLED ping-pong staging: the a1 H2D of
the r455 streams2_probe D run executed ~3 s after its push carrying a2's
bytes (snapshot first4 flipped 00000000 3f000000.. -> 00000000 3e000000..
across retries; sema-bigcopy dst16_after=000000000000003e0000803e0000c03e
proves the a2 pattern landed in da1's page; the vadd computed a2+b1 and
the readback faithfully delivered it - there is no separate read-side
landing bug for this class).

- svc_opdef_clear: keep the snapshot on the APPLY path (the pass's own
  op execution, which consumes the snapshot on success, follows the
  apply); every genuine consume path (FINISH/FLUSH/DROP/STALE/COUNTED/
  SWEEP/STOP) and the TTL still free it.
- svc_copy_snap_capture: evict same-ring-slot snapshots with a different
  push hash (slot reuse) so the surviving snapshot can never serve a
  fresh push.
- svc_run_ops: an op skipped via ops_done drops any snapshot a later
  re-decode re-captured for it (the leak the r444 APPLY-free used to
  sweep).

1 files changed, 52 insertions, 3 deletions

diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c
index b90c948..bcd8ac3 100644
--- a/kernel/gsp_sema_service.c
+++ b/kernel/gsp_sema_service.c
@@ -6032,6 +6032,24 @@ static void svc_copy_snap_capture(struct nv_vgpu_device *vgpu, u64 dst,
 	if (svc_copy_snap_lookup(dst, src, total, owner_ch, owner_idx,
 				 owner_hash))
 		return;	/* already captured */
+	/*
+	 * r456: evict any snapshot left by the SAME ring slot under a
+	 * DIFFERENT push hash (the slot was reused by a fresh push). The
+	 * surviving-across-retries snapshot (svc_opdef_clear APPLY no
+	 * longer frees it) must never be readable by the next occupant of
+	 * the slot; the pass hash changes whenever the guest rewrites the
+	 * push, so a same-slot different-hash snapshot is by definition
+	 * stale for the new push.
+	 */
+	for (i = 0; i < SVC_COPY_SNAP_N; i++) {
+		if (svc_copy_snaps[i].buf &&
+		    svc_copy_snaps[i].owner_ch == owner_ch &&
+		    svc_copy_snaps[i].owner_idx == owner_idx &&
+		    svc_copy_snaps[i].owner_hash != owner_hash) {
+			kfree(svc_copy_snaps[i].buf);
+			svc_copy_snaps[i].buf = NULL;
+		}
+	}
 	for (i = 0; i < SVC_COPY_SNAP_N; i++) {
 		if (!svc_copy_snaps[i].buf) {
 			s = &svc_copy_snaps[i];
@@ -7452,6 +7470,18 @@ svc_wait_done:
 		 */
 		if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) {
 			svc.ops_once_skipped++;
+			/*
+			 * r456: this op already ran in an earlier pass. Any
+			 * staging snapshot a later re-decode pass re-captured
+			 * for it (the capture sites run on every decode) is
+			 * a leak that the r444 APPLY-free used to sweep; with
+			 * that free removed (r456 keeps the snapshot alive
+			 * across retries) drop it here so it can never serve
+			 * a re-push of the ring slot.
+			 */
+			if (m->type == 1 && m->bytes <= SVC_COPY_SNAP_MAX)
+				svc_copy_snap_free(m->dst, m->src, m->bytes,
+						   ch_h, entry_idx, pass_hash);
 			continue;
 		}
 
@@ -15876,9 +15906,28 @@ static void svc_opdef_clear(struct svc_opdef_ent *e, u32 caller)
 			e->n ? e->rel[0].va : 0ull,
 			e->n ? e->rel[0].payload : 0u);
 	}
-	/* r444: the entry is being consumed; no retry can need its
-	 * staging snapshot any more (see svc_copy_snap_free_owner). */
-	if (e->active)
+	/*
+	 * r444: the entry is being consumed; no retry can need its
+	 * staging snapshot any more (see svc_copy_snap_free_owner).
+	 *
+	 * r456: EXCEPT on the APPLY path. The apply runs at the head of a
+	 * re-decode pass of an entry whose copy is still held for its
+	 * operand pins; the pass's own op execution (which consumes the
+	 * snapshot on success) follows the apply. Freeing here destroyed
+	 * the pipeline-order snapshot before it could ever be read and
+	 * the next pass's guard re-capture then read the driver's
+	 * REFILLED ping-pong staging - the r455 operand-shift: the a1 H2D
+	 * executed ~3 s after its push with a2's bytes (owner
+	 * 0x5c00005c/0/6076682fa76b8792, snapshot first4 flipped from
+	 * 00000000 3f000000... to 00000000 3e000000... across retries;
+	 * dst16_after=000000000000003e0000803e0000c03e proves the a2
+	 * pattern landed in da1's page). The snapshot now survives the
+	 * retries and is consumed (and freed) by the copy that finally
+	 * executes; svc_copy_snap_free_owner still runs on every genuine
+	 * consume path (FINISH/FLUSH/DROP/STALE/COUNTED/SWEEP/STOP) and
+	 * the TTL covers the rest.
+	 */
+	if (e->active && caller != SVC_OPDEF_CLR_APPLY)
 		svc_copy_snap_free_owner(e->h_object, e->entry_idx);
 	memset(e, 0, sizeof(*e));
 }