vgsp-sema: mm2 - arm GR counter publish only after a proven session switch (fix r207/r208 victory_matmul second-client hang)

Regression window: r202 (m178) passed mail->matmul on the same host;
r207/r208 (8c634144 + 5de33b14 + 8ceea88d) hang the SECOND client of a
boot: victory_matmul stops after one GR push per channel and 44/51 CE
pushes (no pushes, no forwards, no launches for 242 s; the guest RM
thread idles on 1 Hz MC_SERVICE_INTERRUPTS). r208's bisect attempt did
not actually change the kernel (identical Module srcversion) and
reproduced the hang, so the rewind cannot be exonerated by it.

Mechanism: 8c634144 made svc_counter_publish target release_pa - the
channel's own completion word, i.e. the UVM tracking semaphore the
client polls. In a fresh session's first pass the pump consumes the
PREVIOUS session's residual ring and then publishes the residual GP_PUT
(109/112, also 96 on rank 16) into that word. The 8ceea88d completion
gate does not suppress it: the stale consumption satisfies last_get ==
put, which is exactly the gate's pass condition. The guest UVM then
observes a far-forward payload before the real release sequence; the
later downward observation is a >2*entries jump (UVM_GPU_SEMAPHORE_MAX_
JUMP), corrupting the 64-bit tracking counter and wedging the session's
first waits. The 5de33b14 rewind corrects the ring position but leaves
the polluted completion word and counter_last in place, so the live
ctx-init releases are skipped (rank 16's own pay=1/2 die against the
published 96).

Fix (worktree mm2-fix, tree unchanged otherwise):
- new per-channel gr_pub_armed: a GR channel's word may only receive a
  GP_PUT publish once the channel is proven to belong to the current
  session. Armed when the USERD already reads put <= 1 (pristine ring,
  first client of a boot) or when the fresh-put-drop rewind proves the
  session switch.
- svc_counter_publish returns before any write for an unarmed GR
  channel (after the pending-release flush, so retry writes still run):
  a residual put can no longer reach a fresh client's waited word, and
  the pre-rewind race window disappears entirely.
- the GR rewind additionally voids the stale pass's publish state:
  counter_last is reset and the word is re-armed to 0 so the re-decoded
  live releases re-advance it from the client's own 1,2,... sequence
  (idempotent when no stale publish happened).

Why the first client worked: a fresh boot's rings are pristine
(put=0/1), nothing stale to publish. Why post-hang clients worked: the
hung session's final put is 1, so the residual publish value equals the
fresh client's own first payload (harmless).

Build: make kernel -j green (only kernel/gsp_sema_service.c touched,
+87 lines; vgsp.ko srcversion 40EF888D2E67B00CF9F43E6, differs from the
running r207/r208 module 1418681BF898EE0C1071019).

1 files changed, 87 insertions, 0 deletions

diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c
index 4a86235..aa56c5e 100644
--- a/kernel/gsp_sema_service.c
+++ b/kernel/gsp_sema_service.c
@@ -194,6 +194,28 @@ struct svc_channel {
 				 * reset (PUT dropping below put_prev) resyncs
 				 * last_get to slot 0 so the live first push is
 				 * processed instead of skipped. */
+	bool	gr_pub_armed;	/* mm2 (r207/r208 second-client hang): a GR
+				 * channel's completion word may only receive a
+				 * GP_PUT publish after the channel has been
+				 * proven to belong to the CURRENT session. A
+				 * GR ring/USERD surviving in its fixed VRAM
+				 * slot from the previous session reads a
+				 * residual put at discovery (e.g. 109/112);
+				 * publishing that residual put into the
+				 * channel's own word (8c634144 made the target
+				 * release_pa - the client's waited word) while
+				 * the stale ring is consumed poisons the fresh
+				 * client's UVM tracking semaphore: its
+				 * completed_value observes a far-forward
+				 * payload, the later reset/release sequence is
+				 * a >MAX_JUMP (2*entries) downward observation,
+				 * and the fresh session's first waits wedge
+				 * (matmul: one GR push per channel then 242 s
+				 * silent). Armed when a GR channel is seen with
+				 * put <= 1 (pristine ring) or when the fresh-
+				 * put-drop rewind proves a session switch; the
+				 * rewind also voids any publish state the
+				 * stale pass advanced. */
 	bool	logged;		/* one state log per channel */
 	bool	pool_rel_seen;	/* ISSUE-0002 r70: a pool release word for
 				 * this channel has been read through a
@@ -11528,6 +11550,27 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu,
 	}
 	svc_counter_flush_pending(vgpu, ch);
 
+	/*
+	 * mm2 (r207/r208 second-client hang): suppress the GR publish until
+	 * the channel is proven to belong to the current session. During a
+	 * fresh session's first pass the pump consumes the PREVIOUS
+	 * session's residual ring; the completion gate above passes exactly
+	 * then (the stale consumption satisfies last_get == put) and the
+	 * residual put would land in the client's own completion word
+	 * (release_pa). The fresh client's UVM then sees a far-forward
+	 * payload followed by its real release sequence - a downward
+	 * observation larger than UVM_GPU_SEMAPHORE_MAX_JUMP that corrupts
+	 * the tracking counter and wedges the session's first waits (r207
+	 * victory_matmul: 1 GR push per channel, zero forwards, 242 s).
+	 * gr_pub_armed is set at the fresh-put-drop rewind (session switch
+	 * proven) or when a GR channel is seen with put <= 1 (pristine
+	 * ring, nothing stale to mispublish); a live mid-stream discovery
+	 * (put high, never dropping) loses only this catch-up publish,
+	 * which the release applies already cover.
+	 */
+	if (!ch->engine_type && !ch->gr_pub_armed)
+		return;
+
 	val = put;
 	if (val == ch->counter_last)
 		return;
@@ -15290,6 +15333,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu)
 			ch->logged = true;
 		}
 
+		/*
+		 * mm2: a GR channel whose USERD already reads PUT <= 1 cannot
+		 * hold a stale consumed prefix, so its publishes (the catch-up
+		 * for releases missing from consumed entries) are safe to
+		 * enable immediately - this is the pristine first-boot ring.
+		 * Channels discovered with a residual put stay unarmed until
+		 * the fresh-put-drop rewind proves the session switch, so the
+		 * residual put can never reach the fresh client's completion
+		 * word (see gr_pub_armed and svc_counter_publish).
+		 */
+		if (!ch->engine_type && !ch->gr_pub_armed && put <= 1 &&
+		    !ch->userd_wait_fresh)
+			ch->gr_pub_armed = true;
+
 		/* Initialize expected semaphore payload from the pool (if mapped). */
 		svc_init_expected_payload(vgpu, ch);
 
@@ -15384,6 +15441,36 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu)
 					ch->h_object, ch->gr_index, put,
 					(u32)ch->last_get);
 				ch->last_get = 0;
+				/*
+				 * mm2: the session switch is now proven - arm
+				 * the publish for the live entries that follow,
+				 * and void any publish state the stale pass
+				 * advanced. Everything consumed to reach
+				 * last_get was the dead session's ring; a
+				 * publish issued on it wrote the residual put
+				 * into the very word the fresh client waits on
+				 * (its tracking semaphore; pub target is
+				 * release_pa since 8c634144) and the
+				 * forward-only apply then skips the real
+				 * release payloads that follow. Re-arm the
+				 * word to 0 and reset the catch-up bookkeeping
+				 * so the re-decoded live releases re-advance
+				 * it from the client's own sequence (1, 2,
+				 * ...). Idempotent when no stale publish
+				 * happened (counter_last == 0).
+				 */
+				ch->gr_pub_armed = true;
+				if (ch->counter_last) {
+					u64 upa = ch->release_pa ?
+						ch->release_pa : ch->counter_pa;
+					u32 zero = 0;
+
+					if (upa)
+						(void)svc_write_ap(vgpu, upa,
+							SVC_APERTURE_SYS,
+							&zero, sizeof(zero));
+					ch->counter_last = 0;
+				}
 			}
 			bitmap_zero(ch->slot_done_epoch, SVC_GPFIFO_MAX_BITS);
 		}