From 5de33b14586b8669ebba9150b54c7ea182859214 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Sun, 18 Oct 2026 00:50:35 -0400 Subject: vgsp-sema: GR fresh-put-drop rewind - stop consuming the previous session's residual ring r203 battery forensics (rank 0, sema_dbg=1): hang and progress alternate perfectly with the first USERD GP_PUT read of a fresh session. A session whose first read is residual (106/107/108 = the previous session's final put, GET==PUT) consumes the stale ring in its first pump pass, then never decodes the new client's live entries: its ctx-init completion word (the channel's descending counter slot) freezes at the first stale-decoded payload (1), publishes stop (its USERD put reads never advance past 1), and the client polls forever - multikernel (gen3), streams2 (gen5), victory_mail (gen7) and smiley (gen9) hung; memcpy (gen4), streams_events (gen6), victory_matmul (gen8) - each following a session whose final put was 1 - proceeded. The GR ring/USERD live in fixed VRAM slots (fb 0x140000 + rank*0x3000) that keep the dead session's state across svc_session_flush; m176 reset learned counter state but not the ring bookkeeping. m178/m179's r48 resync (last_get=0 on the first put drop) applies only when userd_wait_fresh is set, and the r73 rewind is CE-only (engine_type != 0), so a GR channel whose fresh put drops below put_prev keeps last_get ahead of the producer forever. Extend the r73 rewind shape to GR: rewind last_get to slot 0 (the fresh session restarts its push sequence at slot 0, and re-decoding slot 0 is idempotent - the forward-only apply drops the duplicate payload), gated by the same submissions < gpfifo_entries/2 freshness guard so a genuine wrap is unaffected. Live evidence (smiley, killed at 127105): counter page word 0x20400fff0 stuck at 1; ring put=107 get=107 at discovery; pub-adv 107->1; repeated rel-apply pay=1 -> skip once per pass. Worktree build: make kernel green (vgsp.ko linked). --- kernel/gsp_sema_service.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) (limited to 'kernel/gsp_sema_service.c') diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index dd7a5e1..6a1595e 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -15323,6 +15323,47 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->h_object, ch->ce_index, put, (u32)ch->last_get); ch->last_get = put; + } else if (!ch->engine_type && ch->gpfifo_entries && + ch->last_get > put && + ch->submissions < ch->gpfifo_entries / 2) { + /* + * m181 (r203 GR ctx-init stall): a GR channel's + * ring/USERD live in fixed VRAM slots + * (fb 0x140000 + rank*0x3000) that keep the + * PREVIOUS session's final state across the + * session flush (m176 reset learned counter + * state but not the ring-consumption + * bookkeeping). A new session's first pump + * pass therefore sees the residual put (e.g. + * 107 = the last session's final GP_PUT) and + * consumes the stale ring up to it, advancing + * last_get before the client has written its + * own entries. The client's fresh put then + * drops below put_prev, and without this + * rewind last_get stays ahead of the producer: + * the live ctx-init entries are never decoded, + * their REPORT_SEM releases are never applied + * (the completion word freezes at the first + * stale-decoded payload), and the client's + * ctx-init wait polls forever - r203 rank 0 + * hung exactly so in multikernel / streams2 / + * victory_mail / smiley whenever the prior + * session had reached a high put (the sessions + * after a hung one, whose final put is 1, + * proceeded - a perfect alternation). Rewind + * to slot 0 like the r48 resync: a fresh + * session restarts its push sequence at slot + * 0, and re-decoding slot 0 is idempotent + * (the stale pass could only have authored the + * same payload, and the forward-only apply + * drops a duplicate). A genuine wrap cannot be + * this fresh (submissions guard), mirroring + * the r73 CE rewind. + */ + pr_info_ratelimited("sema-chan: fresh put drop (GR) h=0x%x rank=%u put=%u last=%u -> rewind to 0\n", + ch->h_object, ch->gr_index, put, + (u32)ch->last_get); + ch->last_get = 0; } bitmap_zero(ch->slot_done_epoch, SVC_GPFIFO_MAX_BITS); } -- cgit v1.2.3