vgsp: m91 - first-fault capture + QMD instruction-cache invalidate for the SM I-fetch lottery

m91-class recap: the first attempt of a mid-sequence forwarded GR launch
raises SM warp exceptions ("Illegal Instruction Encoding", "Out Of Range
Register") on garbage instruction bytes while a retry of the IDENTICAL QMD
and identical shadow content succeeds (r219: fault at +2.5 s, fast re-arm
recovered 3 ms later; r223 same pattern). The host retry path only replays
the mode-35 lane-enable tail + channel/TSG schedules + doorbell and
re-publishes the same method block on a fresh ring entry: no shadow re-copy,
no QMD re-read, no cache/TLB invalidate exists anywhere in the ladder. Plain
re-submission of byte-identical content flipping the outcome names stale
engine-side fetch state at first dispatch, not a content or programming
error: the window-0 shadow pool alternates two 8-page banks by submit
epoch, so a program VA's bytes change every two launches, and every
captured guest QMD carries the QMD v2.3 cache-control word (word 5) with
INVALIDATE_INSTRUCTION_CACHE (bit 30) clear - the client driver arms that
bit only under its forced-invalidate mode.

Two changes:

1. FAULT-CAP (diagnostic, inert until a fault): when a forwarded launch's
   fault epoch advanced across its window, emit one "vgsp-m91: FAULT-CAP"
   line per launch (capped at 20 per module-load cycle, then one
   suppression notice) with: fault-epoch delta, QMD address + first 64 B,
   live program pair (w48/w49) + resolved engine-visible VA/PA, the first
   64 B at that code address, retry decision (fast re-arm /
   resched/fresh-entry / none) + outcome (ok vs timeout), and the newest
   relay-decoded 0x1006 note (exceptType/chid/errString). A fault-free
   launch costs one epoch read and one compare (greens unaffected
   statically: no prints, no content reads, no engine access).

2. QMD icache invalidate (fix, param-gated default-on
   gsp_phys_gr_icache_inv=1): set QMD v2.3 word 5 bit 30 in the forwarded
   QMD (forcing an instruction refetch at dispatch) only when the QMD
   carries the client driver's standard cache-control signature (word 5
   bits 26..29 set) so an unknown QMD revision is never touched; 0 restores
   the exact pre-change QMD for a one-flip A/B. The bit cannot alter
   instruction bytes or data - it only invalidates cached fetch state (the
   same field the client driver sets under forced invalidate) - so green
   launches keep their semantics; the only cost is a per-launch refetch.

Expected next-cycle evidence: zero FAULT-CAP lines with the multi-page
client set green confirms the stale-fetch mechanism; any FAULT-CAP content
policies the remaining candidates (shadow-VA reuse distance, cross-cycle
context state left by killed clients). Doc: 03-architecture/13-vexecution-
plane.md m91-class section.

(cherry picked from commit 682a5559b1dbbd482d8fe6bdabbc46220ce0d5de)

1 files changed, 29 insertions, 0 deletions

diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c
index 81c68b4..9568d22 100644
--- a/kernel/gsp_phys_gpu.c
+++ b/kernel/gsp_phys_gpu.c
@@ -1021,6 +1021,35 @@ unsigned int gsp_phys_gr_first_poll_ms_value(void)
 EXPORT_SYMBOL_GPL(gsp_phys_gr_first_poll_ms_value);
 
 /*
+ * M91 icache coherence (gsp_phys_gr_icache_inv; default 1). The m91 class is
+ * the transient SM instruction-fetch lottery: the first attempt of a
+ * mid-sequence launch faults on SM garbage ("Illegal Instruction Encoding" /
+ * "Out Of Range Register") while a retry of the identical QMD and identical
+ * shadow content succeeds. The window-0 shadow pool reuses code VAs across
+ * launches (two 8-page banks alternate by submit epoch), so the bytes at a
+ * program VA change between launches, and every captured guest QMD carries
+ * the QMDV02_03 cache-control block with INVALIDATE_INSTRUCTION_CACHE (word 5
+ * bit 30) CLEAR - the field the CUDA driver arms only under its
+ * forceIcacheInvalidate knob. An SM fetch line cached for launch N can then
+ * execute the previous kernel's bytes at the same VA in launch N+2. With this
+ * on, the forwarded QMD sets that bit (only when the QMD carries the driver's
+ * standard cache-control signature, bits 26..29 of word 5, so an unknown QMD
+ * revision is never touched), forcing a refetch at dispatch. 0 restores the
+ * exact pre-change QMD for a one-flip A/B.
+ */
+static int gsp_phys_gr_icache_inv_param = 1;
+module_param_named(gsp_phys_gr_icache_inv, gsp_phys_gr_icache_inv_param,
+		   int, 0644);
+MODULE_PARM_DESC(gsp_phys_gr_icache_inv,
+		 "Set QMDV02_03 INVALIDATE_INSTRUCTION_CACHE (word5 bit30) on forwarded GR launches whose QMD has the driver cache-control signature (default 1; 0 = pre-m91 QMD words)");
+
+int gsp_phys_gr_icache_inv_value(void)
+{
+	return gsp_phys_gr_icache_inv_param;
+}
+EXPORT_SYMBOL_GPL(gsp_phys_gr_icache_inv_value);
+
+/*
  * m179 no-write retry budget (gsp_phys_gr_nowrite_retries; default 3, clamped
  * to [0, 4]). The m153 outcome retry loop re-executes a data-bearing launch
  * whose retirement wait saw zero engine writes; it shared the timeout ladder's