vgsp: rescue the direct-GL draw-state pushes from the LOW fb band, admit them to the 3D replay lane, and let the forced golden MME overlay satisfy the prelude maturity gate (eng28)HEADmaster

The direct-GL client's draw-state pushes (476/624/728/324 B; invariant lead
90110087 = IMMD NVC597_INVALIDATE_SHADER_CACHES 0x21C<-0x1011) are
intermittently unreadable while byte-identical copies park in the LOW
framebuffer; the copies do not share the VA's in-page offset, so the
page-stepped rescue scan could never land on them. Add a word-granular
LOW-band sweep gated on the two recognisable push images (state lead via the
method-stream window gate; 256-B VTG SPH upload by exact signature) and pin
hits exactly like the existing rescue.

The same pushes carry the frame's real state (SET_COLOR_CLEAR_VALUE 0xD80,
NVC597_CLEAR_SURFACE 0x19D0, MME macro calls, vertex state) but no BEGIN/RT
opener, so they never opened a C597 section and the state stream never
reached the replay lane; admit the 0x21C lead as an opener on beef channels
and exempt state-led windows from the eng25 colour-target guard (the guard's
data-blob rejects continue unchanged - only the invariant state lead passes).

Where the client's 0x114 IRAM upload is not captured (code words stay <= 4
on every boot observed), the forced golden overlay (sema_gl_mme_golden +
_force; golden-only, fail-closed) can serve every called macro slot; add
gsp_phys_gr_golden_ready() and skip the IMMATURE-SKIP software completion
while it holds (see the golden-slot/pack machinery).

Live (c348/c350/c351, r1624 params): the LOW-band rescue fired (VTG SPH
256-B push -> fb 0x2010200, pinned); the 324-B state push opened a section,
captured (recs=4 nprog=1) and was marked for the verbatim 3D lane; with the
bypass the lane submitted it to the engine for the first time (GR LAUNCH
submitted + GUEST GR LAUNCH OK, MME-PRELUDE pack 1701 IR words, MME-GOLDEN
slot 0 accepted). The emitted 51-word window still omits the 0xD80/0x19D0
clear runs (marked window 81), so pixel stores remain 0 and frames black;
receipts under /tmp/opencode/eng28-*.

Refs: CONT.490.

3 files changed, 206 insertions, 4 deletions

diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c
index e1563dd..8756d83 100644
--- a/kernel/gsp_phys_booter.c
+++ b/kernel/gsp_phys_booter.c
@@ -18437,6 +18437,26 @@ static bool gsp_phys_gr_slot_golden_blank(u32 j)
 }
 
 /*
+ * eng28 (CONT.490): golden READY. With the golden overlay forced (mode 2,
+ * sema_gl_mme_golden + sema_gl_mme_golden_force), the pack fills every
+ * called slot from the driver-emitted golden program whether or not the
+ * client's 0x114 IRAM upload was captured, and golden-only blanks every
+ * body the library cannot serve (fail-closed). In that state the captured
+ * macro program is no longer a PREREQUISITE for a safe first launch, so
+ * the service's immature-skip gate (see svc_gr_request) must not
+ * software-complete beef 3D windows that the golden program can serve.
+ * The client's own state pushes (clear colour + CLEAR_SURFACE + MME macro
+ * calls) then reach the engine - without this the whole draw-state stream
+ * was consumed with zero engine contact on boots whose setup push never
+ * became readable.
+ */
+bool gsp_phys_gr_golden_ready(void)
+{
+	return READ_ONCE(gsp_phys_rmcall_gr_mme_golden) &&
+	       READ_ONCE(gsp_phys_rmcall_gr_mme_golden_force);
+}
+
+/*
  * SESSION 153-CONTINUED-79: per-launch method-block word cap + prelude drain.
  * The r1172-r1175 A/B: every beef launch whose emitted block exceeded ~2000
  * words (3083-word full-SA prelude + window, 3315-3484 words) NEVER STARTED
diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h
index d84b080..d2faa1e 100644
--- a/kernel/gsp_phys_gpu.h
+++ b/kernel/gsp_phys_gpu.h
@@ -350,6 +350,16 @@ bool gsp_phys_rmcall_gr_sph_bind_value(void);
 u32 gsp_phys_rmcall_gr_sph_bind_mask_value(void);
 
 /*
+ * eng28 (CONT.490): the golden MME overlay is forced (sema_gl_mme_golden +
+ * sema_gl_mme_golden_force), so every called macro slot is served from the
+ * driver-emitted golden program (golden-only, fail-closed) and the client's
+ * captured 0x114 upload is not a prerequisite for a safe launch. The
+ * service's immature-skip gate consults this so beef draw-state windows are
+ * not software-completed while the golden program can serve them.
+ */
+bool gsp_phys_gr_golden_ready(void);
+
+/*
  * SESSION 155 CONT 11: bind a synthesized header-only GEOMETRY SPH to
  * stage 4 with ENABLE=0 (sema_gl_sph_gs_img, default 0). The sema service
  * uses this value to author the 128-byte image at family+0x780.
diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c
index 4a64a4f..5071314 100644
--- a/kernel/gsp_sema_service.c
+++ b/kernel/gsp_sema_service.c
@@ -80,6 +80,14 @@ static inline u64 svc_fb_shadow_size(const struct nv_vgpu_device *vgpu)
 #define SEMA_PAGE_SIZE		4096
 #define SEMA_PAGE_SHIFT		12
 /*
+ * eng28 (CONT.490): the invariant lead word of the direct-GL client's
+ * draw-state pushes (IMMD NVC597_INVALIDATE_SHADER_CACHES 0x21C <- 0x1011,
+ * word 0x90110087). Defined early because both the FB-push rescue (the LOW
+ * fb copies are content-gated on it) and the 3D replay admission guard
+ * (svc_gl_window_drawwin_ok) key on it; see the long clause at the rescue.
+ */
+#define SVC_FB_STATE_LEAD	0x90110087u
+/*
  * Highest guest-physical address the service scans / accepts translations for.
  * A 4 GB guest puts its RAM low plus a remapped window above 4 GB (the PCI
  * hole), so the top of RAM is around 6 GB, not 4 GB. Anything below this is
@@ -11829,6 +11837,39 @@ static bool svc_gl_window_rt_valid(const u8 *buf, u32 len, u32 start_off,
 	return false;
 }
 
+/*
+ * eng28 (CONT.490): the draw-state push window.
+ *
+ * svc_gl_window_rt_valid guards against DATA BLOBS whose lattice runs
+ * numerically span the C597 colour-target block (eng25: a blob's last
+ * A/B/W/H words are unmapped VAs; replay writes nothing). The client's
+ * real STATE pushes are method streams that legitimately carry no RT bind
+ * of their own - the target is bound by the surrounding carriers - while
+ * they DO carry the frame's clear (SET_COLOR_CLEAR_VALUE 0xD80 +
+ * NVC597_CLEAR_SURFACE 0x19D0) and the MME macro invocations. Their
+ * invariant lead word (0x90110087) distinguishes them from the blob
+ * class, so a window led by it is admitted to the verbatim replay lane
+ * without the blob guard (the guard's own reject witness stays for every
+ * other 'd' window).
+ */
+static bool svc_gl_window_state_push(const u8 *buf, u32 len, u32 start_off)
+{
+	u32 w0;
+
+	if (!buf || start_off + 4u > len)
+		return false;
+	memcpy(&w0, buf + start_off, 4);
+	return w0 == SVC_FB_STATE_LEAD;
+}
+
+static bool svc_gl_window_drawwin_ok(const u8 *buf, u32 len, u32 start_off,
+				     u32 at_off)
+{
+	if (svc_gl_window_state_push(buf, len, start_off))
+		return true;
+	return svc_gl_window_rt_valid(buf, len, start_off, at_off);
+}
+
 static void svc_gl_flush(struct nv_vgpu_device *vgpu,
 			 struct svc_channel *ch,
 			 const struct svc_push_state *st, const u8 *buf,
@@ -12910,7 +12951,8 @@ static void svc_gl_flush(struct nv_vgpu_device *vgpu,
 	if ((op_src[0] == 's' || op_src[0] == 'h' ||
 	     (op_src[0] == 'd' &&
 	      (ch->h_object & 0xfffff000u) == 0xbeef0000u &&
-	      svc_gl_window_rt_valid(buf, len, svc_gl.start_off, at_off))) &&
+	      svc_gl_window_drawwin_ok(buf, len, svc_gl.start_off,
+				       at_off))) &&
 	    svc_gr_cand.valid &&
 	    len == svc_gr_cand.push_n * 4 &&
 	    svc_gl.start_off + 8u <= at_off &&
@@ -12937,8 +12979,8 @@ static void svc_gl_flush(struct nv_vgpu_device *vgpu,
 		 * bound to size).
 		 */
 		if (op_src[0] == 'd' &&
-		    !svc_gl_window_rt_valid(buf, len, svc_gl.start_off,
-					    at_off)) {
+		    !svc_gl_window_drawwin_ok(buf, len, svc_gl.start_off,
+					      at_off)) {
 			/* eng25: bounded witness when a beef draw-synth
 			 * window is refused - its last colour-target bind
 			 * is not a usable C597 RT (see the clause above). */
@@ -16044,6 +16086,62 @@ static bool svc_fb_push_window_ok(const u8 *buf, u32 len)
 	return inc >= 2 && nseen >= 8;
 }
 
+/*
+ * eng28 (CONT.490): the draw-state push LEAD.
+ *
+ * The direct-GL client's real state pushes (476/624/728/324 B) all begin
+ * with the invariant word 90110087 = IMMD NVC597_INVALIDATE_SHADER_CACHES
+ * (0x21C) <- 0x1011, followed by their real method stream: SET_COLOR_
+ * CLEAR_VALUE(0..3) (0xD80), NVC597_CLEAR_SURFACE <- 0x3C (0x19D0, the
+ * clear TRIGGER), the MME macro invocations (0x3800/0x39C8/0x3810), the
+ * vertex attribute/stream-format block (0x1160/0x1C00) and the frame's
+ * SET_REPORT_SEMAPHORE. These pushes are intermittently unreadable through
+ * every primary translation while a byte-identical copy parks in the LOW
+ * framebuffer (measured: the 476-B push's full bytes at fb 0x2010300; the
+ * fb copies do NOT share the VA's in-page offset, so the page-stepped
+ * rescue scan could never land on them). The lead word is the content
+ * signature that makes those copies identifiable without trusting the
+ * page the failed read answered with.
+ */
+#define SVC_FB_SPH_VTG_LEAD	0x000a0881u
+#define SVC_FB_SPH_VTG_SECOND	0x06000000u
+/*
+ * eng28: the LOW framebuffer band the draw-state copies park in (measured
+ * 0x2010200/0x2010300 on the c346 boot; the band is scanned at word
+ * granularity because the copy's in-band offset has no relation to the
+ * push VA's page offset).
+ */
+#define SVC_FB_LOW_LO		0x2000000ull
+#define SVC_FB_LOW_HI		0x2800000ull
+
+/*
+ * eng28: does the window at [off, off+len) carry a recognised push image?
+ * Either the draw-state lead (with the full method-stream gate) or the
+ * 256-B TURING vertex-shader SPH upload (lead 000a0881 + second word
+ * 06000000 - the PS/VS image leads 000a1482/020a0481 are deliberately not
+ * accepted: they are data, not pushes).
+ */
+static bool svc_fb_push_image_ok(const u8 *p, u32 len)
+{
+	u32 w0, w1 = 0;
+
+	if (len < 8u)
+		return false;
+	memcpy(&w0, p, 4);
+	memcpy(&w1, p + 4, 4);
+	if (w0 == SVC_FB_STATE_LEAD)
+		return svc_fb_push_window_ok(p, len);
+	/*
+	 * The 256-B VTG upload is an SPH image, not a method stream (its
+	 * words do not parse with the run grammar), so its gate is the exact
+	 * two-word image signature instead of svc_fb_push_window_ok.
+	 */
+	if (w0 == SVC_FB_SPH_VTG_LEAD && w1 == SVC_FB_SPH_VTG_SECOND &&
+	    len == 256u)
+		return true;
+	return false;
+}
+
 static bool svc_fb_push_rescue(struct nv_vgpu_device *vgpu, u32 h_object,
 			       u64 va, u32 len, u8 *out, u64 *off_out)
 {
@@ -16111,6 +16209,51 @@ static bool svc_fb_push_rescue(struct nv_vgpu_device *vgpu, u32 h_object,
 		}
 		return true;
 	}
+	/*
+	 * eng28 (CONT.490): word-granular LOW-band sweep for the draw-state
+	 * push images. The page-stepped scan above only visits offsets whose
+	 * in-page position equals the VA's (poff); the LOW fb copies do not
+	 * share it (measured: the 476-B state push at fb 0x2010300 while the
+	 * VA's page offset is 0x79c, so the scan could never land on it and
+	 * the push stayed unreadable for the whole boot). Sweep the LOW band
+	 * the pushes park in at word granularity and accept the first window
+	 * carrying a recognised push image (draw-state lead or VTG SPH
+	 * upload; see svc_fb_push_image_ok). A wrong candidate costs one
+	 * failed image gate; the band is a few MiB.
+	 */
+	{
+		u64 lo = SVC_FB_LOW_LO, hi = min_t(u64, SVC_FB_LOW_HI, lim);
+		u64 o;
+
+		if (hi > lo + len) {
+			for (o = lo; o + len <= hi; o += 4) {
+				u32 lead;
+
+				memcpy(&lead, base + o, 4);
+				if (lead != SVC_FB_STATE_LEAD &&
+				    lead != SVC_FB_SPH_VTG_LEAD)
+					continue;
+				if (!svc_fb_push_image_ok(base + o, len))
+					continue;
+				memcpy(out, base + o, len);
+				*off_out = o;
+				for (i = 0; i < ARRAY_SIZE(hits); i++) {
+					if (!hits[i].n || hits[i].len == len) {
+						hits[i].n = 1;
+						hits[i].len = len;
+						hits[i].off = o;
+						break;
+					}
+				}
+				if (log < 16) {
+					log++;
+					pr_info("sema-chan: FB-PUSH rescue h=0x%x va=0x%llx len=%u -> fb 0x%llx (LOW-band push image; pinned for later reads)\n",
+						h_object, va, len, o);
+				}
+				return true;
+			}
+		}
+	}
 	if (log < 16) {
 		log++;
 		pr_info("sema-chan: FB-PUSH rescue h=0x%x va=0x%llx len=%u: no content-gated window in 0x%llx shadow\n",
@@ -22292,6 +22435,23 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf,
 		 */
 		if (gsp_sema_gl_xlate && !ch->engine_type &&
 		    ((op == 4 && addr == 0x1618) ||
+		     /*
+		      * eng28 (CONT.490): the draw-state push lead. Every real
+		      * client state push (476/624/728/324 B) begins with
+		      * IMMD NVC597_INVALIDATE_SHADER_CACHES (0x21C) <- 0x1011
+		      * (word 0x90110087) and then carries the frame's REAL
+		      * state methods: SET_COLOR_CLEAR_VALUE (0xD80),
+		      * NVC597_CLEAR_SURFACE (0x19D0 = the clear trigger), the
+		      * MME macro invocations (0x3800/0x39C8/0x3810) and the
+		      * vertex attribute/stream-format block (0x1160/0x1C00).
+		      * None of the old openers (0x1618 BEGIN / 0x840 RT) is
+		      * present - the draws ride MME macros - so these pushes
+		      * never opened a section and their clear/state stream
+		      * never reached the replay lane. Admit the lead as an
+		      * opener on attached beef channels like the others.
+		      */
+		     (op == 4 && addr == 0x21c &&
+		      (ch->h_object & 0xfffff000u) == 0xbeef0000u) ||
 		     (op == 1 && addr == 0x840 && count >= 4) ||
 		     (op == 1 && count &&
 		      ((addr <= 0x840u &&
@@ -42275,11 +42435,23 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu,
 	 * engine poisons on the first CALL_MME_MACRO without a program.
 	 * Consume it with the software release instead; the client advances,
 	 * and the lane survives until the setup push matures the capture.
+	 *
+	 * eng28 (CONT.490): ... unless the GOLDEN overlay is forced
+	 * (gsp_phys_gr_golden_ready: sema_gl_mme_golden + _force). The
+	 * launch builder then packs the driver-emitted golden program for
+	 * every called slot (golden-only blanks whatever the library cannot
+	 * serve, fail-closed), so the first launch is safe with an empty
+	 * capture and the client's draw-state windows (clear colour +
+	 * NVC597_CLEAR_SURFACE + the MME macro calls) must reach the engine
+	 * instead of being consumed with zero engine contact. On the c348
+	 * boot the marked 324-B state push was software-skipped here while
+	 * the golden overlay covered all 55 started slots.
 	 */
 	if (READ_ONCE(gsp_sema_mme_immature_skip) &&
 	    (ch->h_object & 0xfffff000u) == 0xbeef0000u &&
 	    READ_ONCE(svc_mme_prelude_flag) &&
-	    !gsp_sema_mme_prelude_mature()) {
+	    !gsp_sema_mme_prelude_mature() &&
+	    !gsp_phys_gr_golden_ready()) {
 		unsigned hold_ms =
 			min_t(unsigned, READ_ONCE(gsp_sema_gr_setup_hold_ms),
 			      2000u);