From 7fc3c3ca9f0c2e25c5a932e3fa955dcb0a261ec6 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Sun, 18 Oct 2026 12:09:37 -0400 Subject: vgsp: m91 - first-fault capture + QMD instruction-cache invalidate for the SM I-fetch lottery m91-class recap: the first attempt of a mid-sequence forwarded GR launch raises SM warp exceptions ("Illegal Instruction Encoding", "Out Of Range Register") on garbage instruction bytes while a retry of the IDENTICAL QMD and identical shadow content succeeds (r219: fault at +2.5 s, fast re-arm recovered 3 ms later; r223 same pattern). The host retry path only replays the mode-35 lane-enable tail + channel/TSG schedules + doorbell and re-publishes the same method block on a fresh ring entry: no shadow re-copy, no QMD re-read, no cache/TLB invalidate exists anywhere in the ladder. Plain re-submission of byte-identical content flipping the outcome names stale engine-side fetch state at first dispatch, not a content or programming error: the window-0 shadow pool alternates two 8-page banks by submit epoch, so a program VA's bytes change every two launches, and every captured guest QMD carries the QMD v2.3 cache-control word (word 5) with INVALIDATE_INSTRUCTION_CACHE (bit 30) clear - the client driver arms that bit only under its forced-invalidate mode. Two changes: 1. FAULT-CAP (diagnostic, inert until a fault): when a forwarded launch's fault epoch advanced across its window, emit one "vgsp-m91: FAULT-CAP" line per launch (capped at 20 per module-load cycle, then one suppression notice) with: fault-epoch delta, QMD address + first 64 B, live program pair (w48/w49) + resolved engine-visible VA/PA, the first 64 B at that code address, retry decision (fast re-arm / resched/fresh-entry / none) + outcome (ok vs timeout), and the newest relay-decoded 0x1006 note (exceptType/chid/errString). A fault-free launch costs one epoch read and one compare (greens unaffected statically: no prints, no content reads, no engine access). 2. QMD icache invalidate (fix, param-gated default-on gsp_phys_gr_icache_inv=1): set QMD v2.3 word 5 bit 30 in the forwarded QMD (forcing an instruction refetch at dispatch) only when the QMD carries the client driver's standard cache-control signature (word 5 bits 26..29 set) so an unknown QMD revision is never touched; 0 restores the exact pre-change QMD for a one-flip A/B. The bit cannot alter instruction bytes or data - it only invalidates cached fetch state (the same field the client driver sets under forced invalidate) - so green launches keep their semantics; the only cost is a per-launch refetch. Expected next-cycle evidence: zero FAULT-CAP lines with the multi-page client set green confirms the stale-fetch mechanism; any FAULT-CAP content policies the remaining candidates (shadow-VA reuse distance, cross-cycle context state left by killed clients). Doc: 03-architecture/13-vexecution- plane.md m91-class section. (cherry picked from commit 682a5559b1dbbd482d8fe6bdabbc46220ce0d5de) --- kernel/gsp_phys_gpu.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) (limited to 'kernel/gsp_phys_gpu.c') diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c index 81c68b4..9568d22 100644 --- a/kernel/gsp_phys_gpu.c +++ b/kernel/gsp_phys_gpu.c @@ -1020,6 +1020,35 @@ unsigned int gsp_phys_gr_first_poll_ms_value(void) } EXPORT_SYMBOL_GPL(gsp_phys_gr_first_poll_ms_value); +/* + * M91 icache coherence (gsp_phys_gr_icache_inv; default 1). The m91 class is + * the transient SM instruction-fetch lottery: the first attempt of a + * mid-sequence launch faults on SM garbage ("Illegal Instruction Encoding" / + * "Out Of Range Register") while a retry of the identical QMD and identical + * shadow content succeeds. The window-0 shadow pool reuses code VAs across + * launches (two 8-page banks alternate by submit epoch), so the bytes at a + * program VA change between launches, and every captured guest QMD carries + * the QMDV02_03 cache-control block with INVALIDATE_INSTRUCTION_CACHE (word 5 + * bit 30) CLEAR - the field the CUDA driver arms only under its + * forceIcacheInvalidate knob. An SM fetch line cached for launch N can then + * execute the previous kernel's bytes at the same VA in launch N+2. With this + * on, the forwarded QMD sets that bit (only when the QMD carries the driver's + * standard cache-control signature, bits 26..29 of word 5, so an unknown QMD + * revision is never touched), forcing a refetch at dispatch. 0 restores the + * exact pre-change QMD for a one-flip A/B. + */ +static int gsp_phys_gr_icache_inv_param = 1; +module_param_named(gsp_phys_gr_icache_inv, gsp_phys_gr_icache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_icache_inv, + "Set QMDV02_03 INVALIDATE_INSTRUCTION_CACHE (word5 bit30) on forwarded GR launches whose QMD has the driver cache-control signature (default 1; 0 = pre-m91 QMD words)"); + +int gsp_phys_gr_icache_inv_value(void) +{ + return gsp_phys_gr_icache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_icache_inv_value); + /* * m179 no-write retry budget (gsp_phys_gr_nowrite_retries; default 3, clamped * to [0, 4]). The m153 outcome retry loop re-executes a data-bearing launch -- cgit v1.2.3