summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-06 20:15:45 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-06 20:16:31 -0400
commit0c1e58b9ceeeff42931222ab14e36a18530a6b7b (patch)
tree9151c671806564508f0c4974c090a42f1443f4f0
parent23d8683f3012ac579db424735d1418814bb7ff01 (diff)
downloadopen-iov-0c1e58b9ceeeff42931222ab14e36a18530a6b7b.tar.gz
open-iov-0c1e58b9ceeeff42931222ab14e36a18530a6b7b.zip
vgsp: mode 35 - real-client user lane (TSG fault buffers, share, V7 channel, 3-entry PA-only promote, V7 execution swap; compile-verified, untested)
Replays the decoded real 570.144 GSP client compute-context sequence on the ladder, additive to modes 0-34: - TSG fault-method buffers: one 0xa06c010a on the ladder TSG with two 0x2000 {base,size,alignment=1,addressSpace,cache} descriptors + zero bar2Addr words + numValidEntries=2; server FBMEM first, registered host sysmem fallback (capture order TSG -> faults -> share -> channel). - FERMI_CONTEXT_SHARE_A (0x9067) reuse of the mode-34 helper (ladder TSG, SHARED_MANAGEMENT VAS, flags SYNC -> VEID 0). - V7 channel 0xcaf000a0: class 0xc46f, TSG parent, hVASpace=0, hContextShare, captured flags 0x00201320 and internalFlags 0x1d, engineType 0, sysmem hObjectError/errorNotifierMem, FBMEM instance/ramfc, sysmem USERD. - One atomic 3-entry GPU_PROMOTE_CTX on V7 (MAIN/PM/PATCH, PA-only, physAttr 4, bInitialize=1, bNonmapped=1) before any 0xc5c0 construct. - TURING_COMPUTE_A (0xc5c0) on V7 after the accepted promote; execution channel swapped to V7 for stage 4 BIND/SCHEDULE/token/GR-sem and forwarding (V6 allocated but unscheduled; its promote suppressed). - Mode-35-only golden9 entry refresh: FECS_EVENT attr=5 with the ctx-window VA in the physAddr field, PRIV_ACCESS_MAP va=0/bNonmapped=1. - gr_promote clamp/param doc extended to 0..35.
-rw-r--r--kernel/gsp_phys_booter.c738
-rw-r--r--kernel/gsp_phys_gpu.c6
2 files changed, 710 insertions, 34 deletions
diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c
index 90b960e..ca58c28 100644
--- a/kernel/gsp_phys_booter.c
+++ b/kernel/gsp_phys_booter.c
@@ -254,6 +254,14 @@ bool gsp_phys_boot_requested(void)
#define GSP_PHYS_RMCALL_CTRL_GPFIFO_SCHEDULE 0xa06f0103u
#define GSP_PHYS_RMCALL_CTRL_BIND_CHANNEL 0xa06f0104u
#define GSP_PHYS_RMCALL_CTRL_TSG_GPFIFO_SCHEDULE 0xa06c0101u
+/*
+ * NVA06C_CTRL_CMD_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS (0xa06c010a,
+ * ctrla06c.h:354-364): set the per-runqueue fault method buffer descriptors
+ * on the channel group (TSG). Mode 35 sends it immediately after the ladder
+ * TSG exists, before the share/channel batch, exactly where the real
+ * 570.144 GSP client sends it (capture #821).
+ */
+#define GSP_PHYS_RMCALL_CTRL_TSG_PROMOTE_FAULT_BUFFERS 0xa06c010au
#define GSP_PHYS_RMCALL_CTRL_DMA_SET_PAGE_DIRECTORY 0x801813u
#define GSP_PHYS_RMCALL_CTRL_DMA_UPDATE_PDE_2 0x80180fu
#define GSP_PHYS_RMCALL_ENGINE_COPY0 0x9u
@@ -704,6 +712,52 @@ bool gsp_phys_boot_requested(void)
*/
#define GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE 34u
/*
+ * Mode 35 (2026-10-06, experiment): the real-client user lane. The 570.144
+ * GSP client's captured compute-context sequence is replayed on the ladder,
+ * in the capture's order, after the existing golden replay (D) has built the
+ * golden image:
+ *
+ * 1. TSG (ladder TSG, existing) - the capture's #819;
+ * 2. TSG fault-method-buffer promote 0xa06c010a (#821): two 0x2000
+ * descriptors {base, size, alignment=1, addressSpace, cache} + two zero
+ * bar2Addr words + numValidEntries=2; server FBMEM first, registered
+ * host sysmem fallback;
+ * 3. FERMI_CONTEXT_SHARE_A 0x9067 (#823), parent = ladder TSG, VAS = the
+ * ladder SHARED_MANAGEMENT VAS, flags SYNC (VEID 0) - the mode-34
+ * helper, allocated before the replay like mode 34;
+ * 4. V7 (0xcaf000a0, #825): class 0xc46f, parent = TSG, hVASpace=0,
+ * hContextShare = the share, flags 0x00201320 (PRIVILEGED |
+ * USERD_INDEX=3 | USERD_PAGE_FIXED, no SKIP bits), internalFlags 0x1d
+ * (ADMIN + error-notifier MEMORY), engineType 0 (the TSG carries GR0),
+ * hObjectError = a registered sysmem notifier object, errorNotifierMem =
+ * its sysmem 0x1000 memdesc, instance/ramfc FBMEM memdescs, the proven
+ * sysmem USERD chunk with hUserdMemory[0]=0 (the captured handle is
+ * client-local; deviation logged). Recorded deviations from #825:
+ * gpFifoEntries 8 (the ladder's ring; captured 1024) and the USERD
+ * aperture (captured memory object/FB).
+ * 5. lane promote (#827): ONE atomic GPU_PROMOTE_CTX on V7 with exactly 3
+ * entries - MAIN (the GR ctx FBMEM MAIN), PM (a fresh FBMEM buffer,
+ * captured raw size 0xd100, allocation page-rounded), PATCH (the GR
+ * FBMEM aux PATCH) - all PA-only, physAttr=4 (VIDMEM), bInitialize=1,
+ * bNonmapped=1 - sent BEFORE any 0xc5c0 construct on V7;
+ * 6. TURING_COMPUTE_A (0xc5c0) on V7 after the accepted promote (#829);
+ * 7. execution swap: stage 4 BIND/SCHEDULE/token/GR-sem and every
+ * forwarded launch run on V7 (rmcall_channel = V7, server_mem=false).
+ * V6 is still allocated (share child, unscheduled) as in mode 33/34;
+ * its FWCLIENT promote and GR-object constructs are suppressed.
+ *
+ * The golden replay D runs the mode-25/31 shape (device-parented golden
+ * channel, 9-entry promote, TURING_A) and, for this mode only, with the
+ * capture-refreshed FECS_EVENT (attr=5, VA in the physAddr field) and
+ * PRIV_ACCESS_MAP (va=0, bNonmapped=1) entries from the current #471
+ * decode; every other golden mode keeps its historical entries.
+ *
+ * Requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and
+ * gsp_phys_rmcall_vas_shared=1 (the share and the V7 GPFIFO VA ride the
+ * ladder SHARED_MANAGEMENT VAS). Any failure aborts stage 3 with no retry.
+ */
+#define GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE 35u
+/*
* Mode-28 golden execution channel shape bits (gsp_phys_rmcall_golden_exec_shape,
* default 7). Every combination keeps the captured flags 0x00200320 and the
* FBMEM instance/RAMFC/mthd slices; the bits only choose the client-visible
@@ -817,7 +871,7 @@ bool gsp_phys_boot_requested(void)
*/
#define GSP_PHYS_RMCALL_FWCLIENT_MASK_FECS_SYSMEM (1u << 16)
#define GSP_PHYS_RMCALL_GR_PROMOTE_MAX \
- (GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE)
+ (GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE)
/*
* Work-submit token notifier slot. The default index is
@@ -1108,6 +1162,41 @@ bool gsp_phys_boot_requested(void)
#define GSP_PHYS_RMCALL_HANDLE_GOLDEN_CTXSHARE_GR_OBJ 0xcaf00090u
/*
+ * Mode 35 (real-client user lane) handles. 0xcaf00094..0xcaf0009f are free
+ * between the mode-34 compute object and 0xcaf000a0; the registered sysmem
+ * method-buffer objects (0xcaf000c0/0xc4) sit below the 0xcaf01000
+ * registered-buffer base used by the shared chunk table.
+ */
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE 0xcaf000a0u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FB 0xcaf000a4u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ 0xcaf000a8u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER 0xcaf000acu
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB0 0xcaf000b0u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB1 0xcaf000b4u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_PM_FB 0xcaf000b8u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM0 0xcaf000c0u
+#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM1 0xcaf000c4u
+
+/*
+ * Mode-35 V7 channel flags/privilege and fault/PM geometry, decoded from the
+ * current 570.144 client capture:
+ * flags 0x00201320 = PRIVILEGED_CHANNEL (bit 5) | USERD_INDEX_VALUE=3
+ * (bits 10:8) | USERD_INDEX_PAGE_FIXED (bit 21, page 1 in bits 20:12);
+ * no SKIP_SCRUBBER (23), no SKIP_CTXBUFFER_ALLOC (31);
+ * internalFlags 0x1d = privilege ADMIN (bits 1:0=1) | error-notifier
+ * MEMORY (bits 3:2=3) | ECC-notifier type 1 (bits 5:4=1);
+ * TSG fault method buffers: two descriptors of 0x2000 bytes each;
+ * PM ctxsw buffer: the captured raw size 0xd100 (the allocation is page
+ * rounded; the promote entry keeps the captured raw size).
+ */
+#define GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE \
+ (u32)(GSP_PHYS_RMCALL_CHANNEL_FLAG_PRIVILEGED | (3u << 8) | \
+ (1u << 21))
+#define GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE 0x1du
+#define GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE 0x2000u
+#define GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE 0xd100u
+
+/*
* Golden-image replay sizes (kernel_graphics.c:2058-2060, 2273-2276): the
* GPFIFO pushbuffer is 32 entries x NVA06F_GP_ENTRY__SIZE (8 B,
* class/clc46f.h:264) = 256 B, and the client USERD is one 4 KB page (the
@@ -1905,6 +1994,14 @@ static struct gsp_phys_boot_stage {
u32 rmcall_priv_size;
u32 rmcall_unr_size;
/*
+ * Mode 35 (real-client user lane): the TSG fault-method-buffer sysmem
+ * fallback chunks (two 0x2000 pages, registered as method-buffer
+ * objects) and the sysmem error-notifier page registered as V7's
+ * hObjectError memory object. All freed by gsp_phys_rmcall_fini().
+ */
+ struct gsp_phys_boot_chunk rmcall_user_lane_fault[2];
+ struct gsp_phys_boot_chunk rmcall_user_lane_notifier;
+ /*
* GR context-promote bisection snapshot (stage 3, GR0 only): the 0..3
* gsp_phys_rmcall_gr_promote value, the number of accepted promote
* controls (0 = none, 1 = PA/init accepted, 2 = VA accepted; mode 2
@@ -2627,6 +2724,11 @@ static void gsp_phys_rmcall_reset(void)
sizeof(boot_stage.rmcall_chunks[i]));
}
memset(&boot_stage.rmcall_ctx, 0, sizeof(boot_stage.rmcall_ctx));
+ for (i = 0; i < 2; i++)
+ memset(&boot_stage.rmcall_user_lane_fault[i], 0,
+ sizeof(boot_stage.rmcall_user_lane_fault[i]));
+ memset(&boot_stage.rmcall_user_lane_notifier, 0,
+ sizeof(boot_stage.rmcall_user_lane_notifier));
for (i = 0; i < GSP_PHYS_RMCALL_GR_CTX_PT_MAX; i++)
memset(&boot_stage.rmcall_ctx_pt[i], 0,
sizeof(boot_stage.rmcall_ctx_pt[i]));
@@ -7738,6 +7840,8 @@ static const char *gsp_phys_rmcall_gr_promote_name(void)
return "mode 33: mode 31 with the TSG sibling A wired as the object-carrying execution channel (ring/sysmem USERD/TSG parent, TURING_COMPUTE_A on A, execution swap to A)";
case GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE:
return "mode 34: golden channel A (0xcaf00054) TSG-parented through FERMI_CONTEXT_SHARE_A (0x9067, VEID 0, ladder VAS), captured unwired shape + 9-entry promote + TURING_A + TURING_COMPUTE_A; V6 (0xcaf00034) executes on the same share with no object of its own and its promote suppressed";
+ case GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE:
+ return "mode 35: real-client user lane (TSG fault-buffer promote + share 0x9067 + V7 0xcaf000a0 captured channel flags 0x00201320/internalFlags 0x1d + one atomic 3-entry MAIN/PM/PATCH PA-only promote on V7 + TURING_COMPUTE_A on V7; execution switched to V7, V6 unscheduled)";
default:
return "off (PROMOTE_CTX never sent)";
}
@@ -7932,7 +8036,9 @@ static int gsp_phys_rmcall_alloc_channel(void __iomem *bar0)
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE)) {
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE)) {
vars = &gsp_phys_rmcall_channel_variant_skipctx;
attempts = 1;
}
@@ -8144,6 +8250,19 @@ static void gsp_phys_rmcall_golden_3d_on_ladder(void __iomem *bar0)
GSP_PHYS_RMCALL_HANDLE_GOLDEN_CHANNEL);
return;
}
+ /*
+ * Mode 35: the execution channel is V7, which gets its own
+ * TURING_COMPUTE_A after the lane promote; V6 stays unscheduled and
+ * must not attach the ladder's extra 3D object (the capture never
+ * allocates 0xc597 on the user-lane channel).
+ */
+ if (boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
+ pr_info("vgsp-boot: rm-call: mode 35: no GOLDEN TURING_A on V6 0x%x; the execution channel is V7 0x%x and the capture carries no 0xc597 on the user lane\n",
+ boot_stage.rmcall_channel,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE);
+ return;
+ }
ret = gsp_phys_rmcall_alloc_gr_object_class(bar0,
GSP_PHYS_RMCALL_CLASS_GR_3D,
GSP_PHYS_RMCALL_HANDLE_GOLDEN_GR3D_LADDER,
@@ -8413,7 +8532,9 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0)
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) &&
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) &&
gsp_phys_rmcall_engine_is_gr()) {
if (gsp_phys_rmcall_fwclient_mask_value() &
GSP_PHYS_RMCALL_GOLDEN9_MASK_LADDER_VAS) {
@@ -8529,7 +8650,9 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0)
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) &&
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) &&
gsp_phys_rmcall_engine_is_gr()) {
ch.flags = cpu_to_le32(GSP_PHYS_RMCALL_CHANNEL_FLAG_GOLDEN9);
ch.internal_flags = cpu_to_le32(
@@ -8760,7 +8883,9 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0)
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) &&
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) &&
gsp_phys_rmcall_engine_is_gr()) {
ret = gsp_phys_rmcall_golden9_promote(bar0);
if (ret)
@@ -9704,9 +9829,11 @@ static int gsp_phys_rmcall_gr_ctx_prepare(void __iomem *bar0)
* allocation (VIDMEM aperture, the shape the KMD promote path
* expects); mode 23 uses the same FBMEM object as the hVirtMemory
* promote target; mode 25 and mode 31 use the FBMEM MAIN that made
- * the golden first-init construct succeed (m25c); every other mode
- * keeps the host sysmem chunk. The PT pages are always host sysmem
- * pages behind the sysmem-aperture PD0.
+ * the golden first-init construct succeed (m25c); mode 34 and mode 35
+ * use it so the golden replay's D promote and the mode-35 lane
+ * promote carry a VIDMEM MAIN; every other mode keeps the host
+ * sysmem chunk. The PT pages are always host sysmem pages behind the
+ * sysmem-aperture PD0.
*/
if (boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_OLDIFACE_FB ||
@@ -9721,7 +9848,9 @@ static int gsp_phys_rmcall_gr_ctx_prepare(void __iomem *bar0)
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) {
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
u64 fb_base = 0;
ret = gsp_phys_rmcall_alloc_fb_mem(bar0,
@@ -9776,14 +9905,18 @@ static int gsp_phys_rmcall_gr_ctx_prepare(void __iomem *bar0)
/*
* Mode-25 golden9e and mode-31 (whose D first-init replay uses the
* same golden9 promote): prepare the FBMEM PATCH/FECS/PRIV/UNR
- * buffers now, before the PDB build maps their promoted VAs.
+ * buffers now, before the PDB build maps their promoted VAs. Mode 35
+ * uses the same FBMEM aux set for its D replay promote and for the
+ * V7 lane promote's PATCH entry.
*/
if (boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN9 ||
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC) {
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
ret = gsp_phys_rmcall_golden9_fb_aux_prepare(bar0);
if (ret) {
pr_err("vgsp-boot: rm-call: GOLDEN9E FBMEM aux prepare failed (%d); aborting the ladder with no retry\n",
@@ -10132,16 +10265,46 @@ static int gsp_phys_rmcall_golden9_promote_on(void __iomem *bar0,
* promote failure. A only carries the TSG group commit, which the
* MAIN/PATCH entries drive.
*/
- if (!skip_fecs)
+ if (!skip_fecs) {
+ /*
+ * Mode 35 fidelity refresh: the current capture's FECS_EVENT
+ * form is attr 0x5 with the ctx-window VA in the physAddr field
+ * (the captured pa field carried a VA, not a PA); every other
+ * golden mode keeps the historical combined PA+VA form. The
+ * VA used is the FECS buffer's mapped ctx-window VA.
+ */
+ if (boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE)
+ gsp_phys_rmcall_promote_entry_fill(p, entry_count++,
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_FECS_EVENT,
+ GSP_PHYS_RMCALL_VA_GOLDEN9_FECS,
+ GSP_PHYS_RMCALL_VA_GOLDEN9_FECS,
+ boot_stage.rmcall_fecs_size, 0x5u, 1, 0);
+ else
+ gsp_phys_rmcall_promote_entry_fill(p, entry_count++,
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_FECS_EVENT,
+ boot_stage.rmcall_fecs_pa,
+ GSP_PHYS_RMCALL_VA_GOLDEN9_FECS,
+ boot_stage.rmcall_fecs_size, fattr, 1, 0);
+ }
+ /*
+ * Mode 35 fidelity refresh: the current capture's PRIV_ACCESS_MAP
+ * entry is PA-only (va=0) with bNonmapped=1; every other golden mode
+ * keeps the historical combined PA+VA form.
+ */
+ if (boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE)
gsp_phys_rmcall_promote_entry_fill(p, entry_count++,
- GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_FECS_EVENT,
- boot_stage.rmcall_fecs_pa,
- GSP_PHYS_RMCALL_VA_GOLDEN9_FECS,
- boot_stage.rmcall_fecs_size, fattr, 1, 0);
- gsp_phys_rmcall_promote_entry_fill(p, entry_count++,
- GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PRIV_MAP,
- boot_stage.rmcall_priv_pa, GSP_PHYS_RMCALL_VA_GOLDEN9_PRIV,
- boot_stage.rmcall_priv_size, fattr, 1, 0);
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PRIV_MAP,
+ boot_stage.rmcall_priv_pa, 0,
+ boot_stage.rmcall_priv_size,
+ GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1);
+ else
+ gsp_phys_rmcall_promote_entry_fill(p, entry_count++,
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PRIV_MAP,
+ boot_stage.rmcall_priv_pa,
+ GSP_PHYS_RMCALL_VA_GOLDEN9_PRIV,
+ boot_stage.rmcall_priv_size, fattr, 1, 0);
gsp_phys_rmcall_promote_entry_fill(p, entry_count++,
GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_UNR_PRIV_MAP,
boot_stage.rmcall_unrestricted_pa, GSP_PHYS_RMCALL_VA_GOLDEN9_UNR,
@@ -10526,26 +10689,30 @@ static int gsp_phys_rmcall_gr_ctx_promote(void __iomem *bar0)
mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING ||
mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT ||
mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
- mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) &&
+ mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) &&
(!gsp_phys_rmcall_golden_enabled() || !gsp_phys_gpu_enabled())) {
pr_err("vgsp-boot: rm-call: mode %u requires gsp_phys_rmcall_golden=1 and gsp_phys_gpu=1 (the golden replay did not run); no PROMOTE_CTX is sent and the ladder aborts with no retry\n",
mode);
return -EINVAL;
}
/*
- * Mode 30/31/33/34: the sibling A's 9-entry promote carries the group
- * context commit; nothing is sent on the V6 channel. A multi-entry
- * promote containing MAIN is refused for a SKIP-ctx channel
- * (graphics_context.c: bKMDVirtualContext -> NV_ERR_INVALID_ARGUMENT)
- * and the single-MAIN exemption needs the pCtxPhysMemDesc the FWCLIENT
- * shapes never establish, so a V6 promote would abort stage 3 on 0x1f.
- * Mode 34's promote ran on the context-share golden channel A inside
- * the replay (skip_fecs=false), before V6 existed.
+ * Mode 30/31/33/34/35: the sibling A's 9-entry promote carries the
+ * group context commit; nothing is sent on the V6 channel. A
+ * multi-entry promote containing MAIN is refused for a SKIP-ctx
+ * channel (graphics_context.c: bKMDVirtualContext ->
+ * NV_ERR_INVALID_ARGUMENT) and the single-MAIN exemption needs the
+ * pCtxPhysMemDesc the FWCLIENT shapes never establish, so a V6
+ * promote would abort stage 3 on 0x1f. Mode 34's promote ran on the
+ * context-share golden channel A inside the replay (skip_fecs=false),
+ * before V6 existed; mode 35's 3-entry lane promote ran on V7 after
+ * V6's alloc (none is sent on V6).
*/
if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING ||
mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT ||
mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
- mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) {
+ mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING)
pr_info("vgsp-boot: rm-call: mode 30: no PROMOTE_CTX on V6; the sibling A carries the group context commit (config_group ppEngCtxDesc), V6 inherits via the TSG commit/channel setup\n");
else if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT)
@@ -10558,6 +10725,13 @@ static int gsp_phys_rmcall_gr_ctx_promote(void __iomem *bar0)
GSP_PHYS_RMCALL_HANDLE_GOLDEN_GR_OBJ,
GSP_PHYS_RMCALL_HANDLE_GOLDEN_CTXSHARE_GR_OBJ,
boot_stage.rmcall_channel);
+ else if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE)
+ pr_info("vgsp-boot: rm-call: mode 35: no PROMOTE_CTX on V6 0x%x; the one atomic 3-entry MAIN/PM/PATCH PA-only lane promote already ran on V7 0x%x (hContextShare=0x%x, TSG 0x%x) and V7 carries TURING_COMPUTE_A 0x%x - stage 4 and forwarding execute on V7\n",
+ boot_stage.rmcall_channel,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE,
+ boot_stage.rmcall_ctxshare,
+ boot_stage.rmcall_tsg,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ);
else
pr_info("vgsp-boot: rm-call: mode 33: no PROMOTE_CTX on V6; wired sibling A's 9-entry promote ran (TSG commit) and A carries both the class object and the execution (hChannel=0x%x, hGr=0x%x)\n",
GSP_PHYS_RMCALL_HANDLE_GOLDEN_SIBLING,
@@ -12243,6 +12417,388 @@ static int gsp_phys_rmcall_alloc_ctxshare(void __iomem *bar0)
return 0;
}
+/*
+ * Mode 35: the TSG fault-method-buffer descriptor pair
+ * (NVA06C_CTRL_CMD_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS 0xa06c010a). The
+ * real client's capture carries two 32-byte
+ * NV2080_CTRL_INTERNAL_MEMDESC_INFO entries {base, size=0x2000, alignment=1,
+ * addressSpace, cpuCacheAttrib}, two zero bar2Addr words and
+ * numValidEntries=2 (one per runqueue). Mode 35 sends the identical shape on
+ * the ladder TSG immediately after the TSG exists, before the share and the
+ * V7 channel - the capture's order (#819 -> #821 -> #823 -> #825).
+ */
+struct gsp_phys_rmcall_fault_memdesc {
+ __le64 base;
+ __le64 size;
+ __le64 alignment;
+ __le32 address_space;
+ __le32 cpu_cache_attrib;
+};
+
+struct gsp_phys_rmcall_fault_buffers_params {
+ struct gsp_phys_rmcall_fault_memdesc memdesc[2];
+ __le64 bar2_addr[2];
+ __le32 num_valid_entries;
+ __le32 pad;
+};
+
+/*
+ * Mode 35: register one arbitrary sysmem chunk as an
+ * NV01_MEMORY_LIST_SYSTEM object. This is the gsp_phys_rmcall_register_mem
+ * body generalised so mode 35 can register its fault-buffer/notifier
+ * fallback chunks without adding entries to the shared ladder chunk table;
+ * every existing mode keeps its exact stage-2 RPC sequence.
+ */
+static int gsp_phys_rmcall_register_chunk(void __iomem *bar0,
+ struct gsp_phys_boot_chunk *c,
+ u32 handle, const char *what)
+{
+ struct gsp_phys_rmcall_alloc_memory_req *req;
+ struct gsp_phys_rx_record rec;
+ u32 pages = c->pages;
+ size_t req_len = offsetof(struct gsp_phys_rmcall_alloc_memory_req, pte) +
+ (size_t)pages * sizeof(__le64);
+ u32 i;
+ int ret;
+
+ req = kvzalloc(req_len, GFP_KERNEL);
+ if (!req)
+ return -ENOMEM;
+ req->h_client = cpu_to_le32(boot_stage.rmcall_client);
+ req->h_device = cpu_to_le32(boot_stage.rmcall_device);
+ req->h_memory = cpu_to_le32(handle);
+ req->h_class = cpu_to_le32(GSP_PHYS_RMCALL_CLASS_MEMLIST_SYSTEM);
+ req->flags = cpu_to_le32(c->page_dma ?
+ GSP_PHYS_RMCALL_OS02_FLAG_NONCONTIGUOUS : 0);
+ req->pte_adjust = 0;
+ req->format = 0;
+ req->length = cpu_to_le64((u64)pages * GSP_PHYS_PAGE_SIZE);
+ req->page_count = cpu_to_le32(pages);
+ req->pte_desc.idr_len = cpu_to_le32((pages & 0xffffu) << 16);
+ for (i = 0; i < pages; i++)
+ req->pte[i] = cpu_to_le64(gsp_phys_chunk_page_dma(c, i) >> 12);
+
+ ret = gsp_phys_rmcall_rpc(bar0, GSP_PHYS_RMCALL_FUNC_ALLOC_MEMORY,
+ req, (u32)req_len, 0, &rec, true, NULL);
+ kvfree(req);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: mode 35 %s ALLOC_MEMORY (hMemory=0x%x dma=0x%llx) failed: %d\n",
+ what, handle, (u64)gsp_phys_chunk_page_dma(c, 0), ret);
+ gsp_phys_rx_record_free(&rec);
+ return ret;
+ }
+ gsp_phys_rx_record_free(&rec);
+ pr_info("vgsp-boot: rm-call: mode 35 %s registered: hMemory=0x%x class=0x%x %u page(s) dma=0x%llx\n",
+ what, handle, GSP_PHYS_RMCALL_CLASS_MEMLIST_SYSTEM, pages,
+ (u64)gsp_phys_chunk_page_dma(c, 0));
+ return 0;
+}
+
+/*
+ * Mode 35 step 2: promote the TSG fault-method buffers. Each of the two
+ * runqueue buffers is a server FBMEM allocation first (alloc_fb_mem - the
+ * shape the other ladder context buffers use) with a registered host sysmem
+ * chunk as fallback; the capture itself carried addressSpace=1 (sysmem), so
+ * the FBMEM aperture is the one logged deviation here. The two base PAs and
+ * the rest of the fields are filled exactly as the capture does.
+ */
+static int gsp_phys_rmcall_tsg_fault_buffers(void __iomem *bar0)
+{
+ static const u32 fb_handles[2] = {
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB0,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB1,
+ };
+ static const u32 mem_handles[2] = {
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM0,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM1,
+ };
+ struct gsp_phys_rmcall_fault_buffers_params p;
+ u32 rpc_result = 0, status = 0;
+ u32 i;
+ int ret;
+
+ memset(&p, 0, sizeof(p));
+ for (i = 0; i < 2; i++) {
+ u64 fb = 0;
+
+ ret = gsp_phys_rmcall_alloc_fb_mem(bar0, fb_handles[i],
+ GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE, &fb,
+ "TSG fault buffer");
+ if (!ret) {
+ p.memdesc[i].base = cpu_to_le64(fb);
+ p.memdesc[i].address_space =
+ cpu_to_le32(GSP_PHYS_RMCALL_ADDR_FBMEM);
+ pr_info("vgsp-boot: rm-call: mode 35 TSG fault buffer %u: server FBMEM base=0x%llx size=0x%x\n",
+ i, fb, GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE);
+ } else {
+ struct gsp_phys_boot_chunk *c =
+ &boot_stage.rmcall_user_lane_fault[i];
+
+ if (!boot_stage.pdev)
+ return -EINVAL;
+ if (!c->pages) {
+ ret = gsp_phys_chunk_alloc(boot_stage.pdev, c,
+ GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE /
+ GSP_PHYS_PAGE_SIZE);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: mode 35 TSG fault buffer %u sysmem fallback allocation failed (%d); no fault-buffer promote is sent and the ladder aborts with no retry\n",
+ i, ret);
+ return ret;
+ }
+ }
+ gsp_phys_chunk_zero(c);
+ ret = gsp_phys_rmcall_register_chunk(bar0, c,
+ mem_handles[i],
+ "TSG fault buffer (sysmem fallback)");
+ if (ret)
+ return ret;
+ p.memdesc[i].base = cpu_to_le64(
+ (u64)gsp_phys_chunk_page_dma(c, 0));
+ p.memdesc[i].address_space =
+ cpu_to_le32(GSP_PHYS_RMCALL_ADDR_SYSMEM);
+ pr_info("vgsp-boot: rm-call: mode 35 TSG fault buffer %u: sysmem fallback pa=0x%llx size=0x%x (capture used as=1; FBMEM was tried first)\n",
+ i, (u64)gsp_phys_chunk_page_dma(c, 0),
+ GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE);
+ }
+ p.memdesc[i].size =
+ cpu_to_le64(GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE);
+ p.memdesc[i].alignment = cpu_to_le64(1);
+ p.memdesc[i].cpu_cache_attrib = cpu_to_le32(0);
+ p.bar2_addr[i] = cpu_to_le64(0);
+ }
+ p.num_valid_entries = cpu_to_le32(2);
+
+ ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_tsg,
+ GSP_PHYS_RMCALL_CTRL_TSG_PROMOTE_FAULT_BUFFERS,
+ &p, sizeof(p), 16, NULL, 0, &rpc_result, &status,
+ "INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS");
+ pr_info("vgsp-boot: rm-call: mode 35 TSG fault-method-buffer promote: hTsg=0x%x numValid=2 md0={base=0x%llx size=0x%llx as=%u align=0x%llx} md1={base=0x%llx size=0x%llx as=%u} bar2=0 status=0x%x rpc_result=0x%x ret=%d\n",
+ boot_stage.rmcall_tsg,
+ le64_to_cpu(p.memdesc[0].base),
+ le64_to_cpu(p.memdesc[0].size),
+ le32_to_cpu(p.memdesc[0].address_space),
+ le64_to_cpu(p.memdesc[0].alignment),
+ le64_to_cpu(p.memdesc[1].base),
+ le64_to_cpu(p.memdesc[1].size),
+ le32_to_cpu(p.memdesc[1].address_space),
+ status, rpc_result, ret);
+ return ret;
+}
+
+/*
+ * Mode 35 step 4a: the V7 error-notifier page. The capture's hObjectError
+ * 0x5c000018 is a client-local memory object holding the 0x1000-byte
+ * notifier page (it is referenced by #825 but never appears as an alloc in
+ * the capture); the ladder's closest equivalent is a registered host sysmem
+ * page (NV01_MEMORY_LIST_SYSTEM, addressSpace=1 like the captured
+ * errorNotifierMem) whose handle becomes hObjectError.
+ */
+static int gsp_phys_rmcall_user_lane_notifier(void __iomem *bar0)
+{
+ struct gsp_phys_boot_chunk *c = &boot_stage.rmcall_user_lane_notifier;
+ int ret;
+
+ if (!boot_stage.pdev)
+ return -EINVAL;
+ if (!c->pages) {
+ ret = gsp_phys_chunk_alloc(boot_stage.pdev, c, 1u);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: mode 35 user-lane error-notifier page allocation failed (%d); no V7 channel is allocated and the ladder aborts with no retry\n",
+ ret);
+ return ret;
+ }
+ }
+ gsp_phys_chunk_zero(c);
+ ret = gsp_phys_rmcall_register_chunk(bar0, c,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER,
+ "user-lane error-notifier (sysmem)");
+ if (ret)
+ return ret;
+ pr_info("vgsp-boot: rm-call: mode 35 user-lane error notifier: hObject=0x%x sysmem pa=0x%llx size=0x%x (closest equivalent of the capture's hObjectError 0x5c000018 object, which the capture never allocated; deviation logged)\n",
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER,
+ (u64)gsp_phys_chunk_page_dma(c, 0),
+ GSP_PHYS_RMCALL_NOTIFIER_SIZE);
+ return 0;
+}
+
+/*
+ * Mode 35 step 4b: the V7 real-client user-lane channel (0xcaf000a0). Every
+ * captured #825 field is carried: class 0xc46f, hParent = ladder TSG,
+ * hVASpace 0, hContextShare (the mode-34 share), flags 0x00201320,
+ * internalFlags 0x1d, engineType 0, hObjectError = the registered notifier
+ * object, errorNotifierMem = its sysmem memdesc, instance/ramfc FBMEM
+ * memdescs, mthdbuf sysmem. Recorded deviations: gpFifoEntries 8 (the
+ * ladder ring; captured 1024, a wrap-size difference only), gpFifoOffset at
+ * the ladder VA (captured a client VA), hUserdMemory[0]=0 with the proven
+ * sysmem USERD chunk (captured a client USERD memory object/FB aperture),
+ * and ProcessID 0 (captured 695446, cosmetic). The server validates and
+ * returns; a rejection aborts stage 3 with no retry.
+ */
+static int gsp_phys_rmcall_alloc_user_lane_channel(void __iomem *bar0)
+{
+ struct gsp_phys_rmcall_channel_params ch;
+ u64 fb = 0;
+ u32 rpc = 0, status = 0;
+ int ret;
+
+ ret = gsp_phys_rmcall_alloc_fb_mem(bar0,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_FB, 0x1000, &fb,
+ "mode 35 user-lane instance/RAMFC");
+ if (ret)
+ return ret;
+ ret = gsp_phys_rmcall_user_lane_notifier(bar0);
+ if (ret)
+ return ret;
+
+ memset(&ch, 0, sizeof(ch));
+ ch.h_object_error =
+ cpu_to_le32(GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER);
+ ch.gp_fifo_offset = cpu_to_le64(GSP_PHYS_RMCALL_VA_GPFIFO);
+ ch.gp_fifo_entries = cpu_to_le32(GSP_PHYS_RMCALL_GPFIFO_ENTRIES);
+ ch.flags = cpu_to_le32(GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE);
+ ch.h_context_share = cpu_to_le32(boot_stage.rmcall_ctxshare);
+ ch.h_va_space = cpu_to_le32(0);
+ ch.engine_type = cpu_to_le32(0); /* the TSG carries GR0 */
+ ch.cid = cpu_to_le32(0);
+ ch.sub_device_id = cpu_to_le32(0);
+ ch.internal_flags = cpu_to_le32(
+ GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE);
+ ch.instance_mem = gsp_phys_rmcall_mem_desc_fb(fb, 0x1000);
+ ch.ramfc_mem = gsp_phys_rmcall_mem_desc_fb(fb, 0x200);
+ ch.userd_mem = gsp_phys_rmcall_mem_desc(GSP_PHYS_RMCALL_BUF_USERD);
+ ch.mthdbuf_mem = gsp_phys_rmcall_mem_desc(GSP_PHYS_RMCALL_BUF_MTHDBUF);
+ ch.error_notifier_mem.base = cpu_to_le64(
+ (u64)gsp_phys_chunk_page_dma(
+ &boot_stage.rmcall_user_lane_notifier, 0));
+ ch.error_notifier_mem.size =
+ cpu_to_le64(GSP_PHYS_RMCALL_NOTIFIER_SIZE);
+ ch.error_notifier_mem.address_space =
+ cpu_to_le32(GSP_PHYS_RMCALL_ADDR_SYSMEM);
+ ch.error_notifier_mem.cache_attrib =
+ cpu_to_le32(GSP_PHYS_RMCALL_CACHE_UNCACHED);
+
+ ret = gsp_phys_rmcall_alloc(bar0, boot_stage.rmcall_client,
+ boot_stage.rmcall_tsg,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE,
+ GSP_PHYS_RMCALL_CLASS_CHANNEL, &ch, sizeof(ch),
+ false, NULL,
+ "mode 35 V7 real-client user lane channel", true,
+ &rpc, &status);
+ pr_info("vgsp-boot: rm-call: mode 35 V7 channel alloc: hObject=0x%x hParent=hTsg=0x%x flags=0x%x internalFlags=0x%x engineType=0 hContextShare=0x%x hVASpace=0 gpFifoOffset=0x%llx gpFifoEntries=%u hUserdMemory[0]=0 userdMem=sysmem instance=0x%llx/0x1000 FBMEM ramfc=0x%llx/0x200 FBMEM hObjectError=0x%x errorNotifier=sysmem status=0x%x rpc_result=0x%x ret=%d (deviations: entries=%u vs captured 1024; sysmem USERD vs captured hUserdMemory object)\n",
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE, boot_stage.rmcall_tsg,
+ GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE,
+ GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE,
+ le32_to_cpu(ch.h_context_share),
+ le64_to_cpu(ch.gp_fifo_offset),
+ le32_to_cpu(ch.gp_fifo_entries), fb, fb,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER,
+ status, rpc, ret, GSP_PHYS_RMCALL_GPFIFO_ENTRIES);
+ return ret;
+}
+
+/*
+ * Mode 35 step 5: the lane promote. ONE atomic GPU_PROMOTE_CTX (0x2080012b)
+ * on V7 with exactly 3 entries - MAIN, PM, PATCH in the captured bufferId
+ * order - all PA-only (va=0), physAttr 4 (VIDMEM), bInitialize=1,
+ * bNonmapped=1, matching the capture's #827. The PM buffer is a fresh
+ * server FBMEM allocation sized ALIGN(0xd100, 4 KB) while the promote entry
+ * carries the captured raw 0xd100. Sent BEFORE any 0xc5c0 construct on V7.
+ */
+static int gsp_phys_rmcall_user_lane_promote(void __iomem *bar0)
+{
+ struct gsp_phys_rmcall_promote_params *p;
+ u32 rpc_result = 0, status = 0;
+ u32 alloc_size;
+ unsigned int i;
+ int ret;
+
+ if (!boot_stage.rmcall_ctx_fbmem || !boot_stage.rmcall_golden_fb_aux ||
+ !boot_stage.rmcall_patch_pa) {
+ pr_err("vgsp-boot: rm-call: mode 35 lane promote aborted: the MAIN/PATCH buffers are not server FBMEM (ctx_fbmem=%u fb_aux=%u patch=0x%llx); no promote is sent and the ladder aborts with no retry\n",
+ boot_stage.rmcall_ctx_fbmem,
+ boot_stage.rmcall_golden_fb_aux,
+ boot_stage.rmcall_patch_pa);
+ return -EINVAL;
+ }
+ if (!boot_stage.rmcall_pm_prepared) {
+ u64 fb = 0;
+
+ alloc_size = ALIGN(GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE,
+ GSP_PHYS_PAGE_SIZE);
+ ret = gsp_phys_rmcall_alloc_fb_mem(bar0,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_PM_FB,
+ alloc_size, &fb, "mode 35 PM ctxsw");
+ if (ret)
+ return ret;
+ boot_stage.rmcall_pm_pa = fb;
+ boot_stage.rmcall_pm_size =
+ GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE;
+ boot_stage.rmcall_pm_prepared = true;
+ pr_info("vgsp-boot: rm-call: mode 35 PM ctxsw FBMEM: base=0x%llx allocation=0x%x (page-rounded) promote size=0x%x (captured raw)\n",
+ fb, alloc_size, boot_stage.rmcall_pm_size);
+ }
+
+ if (!boot_stage.rmcall_gr_promote_warned) {
+ boot_stage.rmcall_gr_promote_warned = true;
+ pr_warn("vgsp-boot: rm-call: WARNING: mode 35 sends GPU_PROMOTE_CTX on V7 (one atomic 3-entry MAIN/PM/PATCH PA-only): a failed GR context promote can assert GSP-RM (NOCAT recType=5 errorCode=0x52a8b04) and wedge the device with WPR2 left up and teardown unable to clear it; recovery requires a reboot with human approval - run only when reboot readiness is confirmed\n");
+ }
+
+ p = kvzalloc(sizeof(*p), GFP_KERNEL);
+ if (!p)
+ return -ENOMEM;
+ p->engine_type = cpu_to_le32(GSP_PHYS_RMCALL_ENGINE_GR0);
+ p->h_client = cpu_to_le32(0);
+ p->ch_id = cpu_to_le32(0);
+ p->h_chan_client = cpu_to_le32(boot_stage.rmcall_client);
+ p->h_object = cpu_to_le32(GSP_PHYS_RMCALL_HANDLE_USER_LANE);
+ p->h_virt_memory = cpu_to_le32(0);
+ p->virt_address = cpu_to_le64(0);
+ p->size = cpu_to_le64(0);
+
+ gsp_phys_rmcall_promote_entry_fill(p, 0,
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_MAIN,
+ boot_stage.rmcall_ctx_pa, 0, boot_stage.rmcall_ctx_size,
+ GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1);
+ gsp_phys_rmcall_promote_entry_fill(p, 1,
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PM,
+ boot_stage.rmcall_pm_pa, 0, boot_stage.rmcall_pm_size,
+ GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1);
+ gsp_phys_rmcall_promote_entry_fill(p, 2,
+ GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PATCH,
+ boot_stage.rmcall_patch_pa, 0, boot_stage.rmcall_patch_size,
+ GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1);
+ p->entry_count = cpu_to_le32(3);
+
+ for (i = 0; i < 3; i++)
+ pr_info("vgsp-boot: rm-call: mode 35 lane promote entry[%u]: bufferId=%u pa=0x%llx va=0 size=0x%llx physAttr=0x%x bInitialize=%u bNonmapped=%u\n",
+ i, le16_to_cpu(p->entry[i].buffer_id),
+ le64_to_cpu(p->entry[i].gpu_phys_addr),
+ le64_to_cpu(p->entry[i].size),
+ le32_to_cpu(p->entry[i].phys_attr),
+ p->entry[i].b_initialize, p->entry[i].b_nonmapped);
+
+ ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_subdevice,
+ GSP_PHYS_RMCALL_CTRL_GPU_PROMOTE_CTX,
+ p, sizeof(*p), 24, NULL, 0, &rpc_result, &status,
+ "GPU_PROMOTE_CTX (mode 35 lane, atomic 3-entry)");
+ pr_info("vgsp-boot: rm-call: PROMOTE_CTX mode 35 lane: entryCount=3 status=0x%x rpc_result=0x%x (hObject=V7 0x%x hChanClient=0x%x engineType=0x%x MAIN pa=0x%llx size=0x%x PM pa=0x%llx size=0x%x PATCH pa=0x%llx size=0x%x)\n",
+ status, rpc_result, GSP_PHYS_RMCALL_HANDLE_USER_LANE,
+ boot_stage.rmcall_client, GSP_PHYS_RMCALL_ENGINE_GR0,
+ boot_stage.rmcall_ctx_pa, boot_stage.rmcall_ctx_size,
+ boot_stage.rmcall_pm_pa, boot_stage.rmcall_pm_size,
+ boot_stage.rmcall_patch_pa, boot_stage.rmcall_patch_size);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: PROMOTE_CTX mode 35 lane failed (status=0x%x rpc_result=0x%x, %s, ret=%d); the V7 context is not committed, no 0xc5c0 is constructed and the ladder aborts with no retry\n",
+ status, rpc_result, gsp_phys_rmcall_status_str(status),
+ ret);
+ kvfree(p);
+ return ret;
+ }
+ boot_stage.rmcall_gr_promote_step = 1;
+ kvfree(p);
+ return 0;
+}
+
static int gsp_phys_rmcall_stage3(void __iomem *bar0)
{
struct gsp_phys_rmcall_vas_params vas;
@@ -12378,6 +12934,22 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0)
boot_stage.rmcall_tsg = GSP_PHYS_RMCALL_HANDLE_TSG;
/*
+ * Mode 35 step 2 (capture order): promote the two TSG fault-method
+ * buffers (0xa06c010a) on the just-created TSG, before the share and
+ * the channel batch. A rejection aborts stage 3 with no retry.
+ */
+ if (gsp_phys_rmcall_engine_is_gr() &&
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
+ ret = gsp_phys_rmcall_tsg_fault_buffers(bar0);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 TSG fault-method-buffer promote failed (%d); the lane does not start and the ladder aborts with no retry\n",
+ ret);
+ return ret;
+ }
+ }
+
+ /*
* Mode 34: allocate FERMI_CONTEXT_SHARE_A before the golden replay, so
* the replay's golden channel A can attach to it (hContextShare) and
* the ladder's V6 channel can attach to the same share later. The share
@@ -12403,6 +12975,29 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0)
}
/*
+ * Mode 35 step 3: the same FERMI_CONTEXT_SHARE_A helper for the
+ * real-client user lane - one share per TSG, VEID 0, ladder
+ * SHARED_MANAGEMENT VAS - allocated before the golden replay so V7
+ * and V6 attach to it (mode 34's ordering; the capture allocates it
+ * after the fault buffers and before the channel, which the two
+ * ladder steps preserve).
+ */
+ if (gsp_phys_rmcall_engine_is_gr() &&
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
+ if (!gsp_phys_rmcall_vas_shared_value()) {
+ pr_err("vgsp-boot: rm-call: mode 35 needs the ladder SHARED_MANAGEMENT VAS (gsp_phys_rmcall_vas_shared=1): the share and the V7 GPFIFO VA ride the ladder VAS page tables; aborting with no retry\n");
+ return -EINVAL;
+ }
+ ret = gsp_phys_rmcall_alloc_ctxshare(bar0);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 FERMI_CONTEXT_SHARE_A alloc failed (%d); V7 has no share and the ladder aborts with no retry\n",
+ ret);
+ return ret;
+ }
+ }
+
+ /*
* Golden-image replay (gsp_phys_rmcall_golden=1 + gsp_phys_gpu=1):
* replay the stock client's post-load golden channel allocations after
* the ladder's VAS/TSG exist and before the ladder's own channel/GR
@@ -12429,7 +13024,9 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0)
boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC ||
boot_stage.rmcall_gr_promote ==
- GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) &&
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE ||
+ boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) &&
gsp_phys_rmcall_engine_is_gr()) {
pr_err("vgsp-boot: rm-call: stage 3 aborted: mode-%u golden replay failed (%d); the golden construct is atomic and is not retried\n",
boot_stage.rmcall_gr_promote, ret);
@@ -12567,6 +13164,63 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0)
GSP_PHYS_RMCALL_HANDLE_GOLDEN_GR_OBJ,
GSP_PHYS_RMCALL_HANDLE_GOLDEN_CTXSHARE_GR_OBJ,
boot_stage.rmcall_gr_object);
+ } else if (boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) {
+ /*
+ * Mode 35: build the real-client user lane on V7 after
+ * V6's alloc and before the promote suppression below:
+ * allocate V7 (captured #825 shape), run the one
+ * atomic 3-entry MAIN/PM/PATCH PA-only lane promote
+ * (#827) BEFORE any construct, construct the
+ * TURING_COMPUTE_A (0xc5c0) on V7 (#829), and swap the
+ * execution channel to V7 so stage 4 BIND/SCHEDULE/
+ * token/GR-sem and every forwarded launch run on it.
+ * Any failure aborts stage 3 with no retry.
+ */
+ u32 ul_rpc = 0, ul_status = 0;
+
+ if (!gsp_phys_rmcall_golden_enabled() ||
+ !gsp_phys_gpu_enabled()) {
+ pr_err("vgsp-boot: rm-call: mode 35 requires gsp_phys_rmcall_golden=1 and gsp_phys_gpu=1 (the golden replay did not run); the user lane is not built and the ladder aborts with no retry\n");
+ return -EINVAL;
+ }
+ ret = gsp_phys_rmcall_alloc_user_lane_channel(bar0);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 V7 channel alloc failed (%d); the user lane is not built and the ladder aborts with no retry\n",
+ ret);
+ return ret;
+ }
+ ret = gsp_phys_rmcall_user_lane_promote(bar0);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 lane promote failed (%d); the V7 context is not committed, no 0xc5c0 is constructed and the ladder aborts with no retry\n",
+ ret);
+ return ret;
+ }
+ ret = gsp_phys_rmcall_alloc_gr_object_class(bar0,
+ GSP_PHYS_RMCALL_CLASS_COMPUTE,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE,
+ "mode 35 TURING_COMPUTE_A on V7 (after promote)",
+ &ul_rpc, &ul_status);
+ if (ret) {
+ pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 TURING_COMPUTE_A on V7 failed (status=0x%x rpc_result=0x%x ret=%d); the V7 context has no class object and the ladder aborts with no retry\n",
+ ul_status, ul_rpc, ret);
+ return ret;
+ }
+ boot_stage.rmcall_gr_object =
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ;
+ pr_info("vgsp-boot: rm-call: mode 35: execution channel switched from V6 0x%x to V7 0x%x (captured flags=0x%x internalFlags=0x%x hContextShare=0x%x hVASpace=0, gpFifoVa=0x%llx entries=%u, sysmem USERD, FBMEM instance/ramfc, TURING_COMPUTE_A 0x%x); V6 stays allocated and unscheduled, stage 4 + forwarding run on V7\n",
+ boot_stage.rmcall_channel,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE,
+ GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE,
+ GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE,
+ boot_stage.rmcall_ctxshare,
+ GSP_PHYS_RMCALL_VA_GPFIFO,
+ GSP_PHYS_RMCALL_GPFIFO_ENTRIES,
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ);
+ boot_stage.rmcall_channel =
+ GSP_PHYS_RMCALL_HANDLE_USER_LANE;
+ boot_stage.rmcall_channel_server_mem = false;
} else {
ret = gsp_phys_rmcall_alloc_gr_object(bar0);
if (ret) {
@@ -12834,9 +13488,16 @@ struct gsp_phys_rmcall_ce_submit {
* offset 0: m14 proved the server applies the sysmem userd_mem base
* directly and ignores the captured USERD_INDEX_VALUE, and the PBDMA
* consumed the entry at +0x0 (GP_GET advanced) while +0x600 stayed dead.
+ * Mode 35's V7 carries the captured flags 0x00201320 (USERD_INDEX_VALUE=3)
+ * but the proven sysmem USERD chunk with hUserdMemory[0]=0, so the host
+ * control block is written at offset 0 like the other sysmem-USERD shapes.
*/
static u32 gsp_phys_rmcall_userd_ctl_off(void)
{
+ if (boot_stage.rmcall_gr_promote ==
+ GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE &&
+ gsp_phys_rmcall_engine_is_gr())
+ return 0;
if ((boot_stage.rmcall_gr_promote ==
GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_EXEC ||
boot_stage.rmcall_gr_promote ==
@@ -16082,6 +16743,9 @@ static void gsp_phys_rmcall_fini(struct pci_dev *pdev)
gsp_phys_chunk_free(pdev, &boot_stage.rmcall_fecs);
gsp_phys_chunk_free(pdev, &boot_stage.rmcall_priv);
gsp_phys_chunk_free(pdev, &boot_stage.rmcall_unrestricted);
+ gsp_phys_chunk_free(pdev, &boot_stage.rmcall_user_lane_fault[0]);
+ gsp_phys_chunk_free(pdev, &boot_stage.rmcall_user_lane_fault[1]);
+ gsp_phys_chunk_free(pdev, &boot_stage.rmcall_user_lane_notifier);
gsp_phys_rmcall_reset();
}
@@ -17792,6 +18456,18 @@ static_assert(sizeof(struct gsp_phys_rmcall_ctx_buffer_info_params) == 5136,
static_assert(offsetof(struct gsp_phys_rmcall_ctx_buffer_info_params,
ctx_buffer_info) == 16,
"NV2080_CTRL_GR_GET_CTX_BUFFER_INFO_PARAMS.ctxBufferInfo must be at offset 16 (8-byte aligned)");
+static_assert(sizeof(struct gsp_phys_rmcall_fault_memdesc) == 32,
+ "NV2080_CTRL_INTERNAL_MEMDESC_INFO must be 32 bytes");
+static_assert(offsetof(struct gsp_phys_rmcall_fault_memdesc, alignment) == 16,
+ "NV2080_CTRL_INTERNAL_MEMDESC_INFO.alignment must be at offset 16");
+static_assert(offsetof(struct gsp_phys_rmcall_fault_memdesc, address_space) == 24,
+ "NV2080_CTRL_INTERNAL_MEMDESC_INFO.addressSpace must be at offset 24");
+static_assert(sizeof(struct gsp_phys_rmcall_fault_buffers_params) == 88,
+ "NVA06C_CTRL_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS_PARAMS must be 88 bytes for two runqueues");
+static_assert(offsetof(struct gsp_phys_rmcall_fault_buffers_params, bar2_addr) == 64,
+ "PROMOTE_FAULT_METHOD_BUFFERS.bar2Addr must be at offset 64");
+static_assert(offsetof(struct gsp_phys_rmcall_fault_buffers_params, num_valid_entries) == 80,
+ "PROMOTE_FAULT_METHOD_BUFFERS.numValidEntries must be at offset 80");
static_assert(sizeof(struct gsp_phys_rmcall_channel_params) == 368,
"NV_CHANNEL_ALLOC_PARAMS must be 368 bytes (reference ABI)");
static_assert(offsetof(struct gsp_phys_rmcall_channel_params, gp_fifo_offset) == 8,
diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c
index 339aa06..e2fffc9 100644
--- a/kernel/gsp_phys_gpu.c
+++ b/kernel/gsp_phys_gpu.c
@@ -551,8 +551,8 @@ static int gsp_phys_rmcall_gr_promote_set(const char *val,
return ret;
if (gsp_phys_rmcall_gr_promote_param < 0 ||
- gsp_phys_rmcall_gr_promote_param > 34) {
- pr_warn("vgsp: gsp_phys_rmcall_gr_promote=%d is outside 0..34; clamping to 0 (no PROMOTE_CTX)\n",
+ gsp_phys_rmcall_gr_promote_param > 35) {
+ pr_warn("vgsp: gsp_phys_rmcall_gr_promote=%d is outside 0..35; clamping to 0 (no PROMOTE_CTX)\n",
gsp_phys_rmcall_gr_promote_param);
gsp_phys_rmcall_gr_promote_param = 0;
}
@@ -571,7 +571,7 @@ module_param_cb(gsp_phys_rmcall_gr_promote,
&gsp_phys_rmcall_gr_promote_ops,
&gsp_phys_rmcall_gr_promote_param, 0644);
MODULE_PARM_DESC(gsp_phys_rmcall_gr_promote,
- "GR context-promote bisection (default 0: never send PROMOTE_CTX; 1: single entry init+nonmapped=0 size=raw+subctx, the p6h1b shape; 2: two-phase PA-only nonmapped then VA-only with the same fields; 3: single entry size=raw; 4: two-phase with a zero-field VA-only step; 5: batched entryCount=2 PA-only init + zero-field VA-only; 6: no-entry hVirtMemory, NV01_MEMORY_VIRTUAL (class 0x70) on the VAS at the context window + PROMOTE_CTX entryCount=0; 7: no-entry bare virtAddress+size, entryCount=0; 8: single MAIN entry like 1 on the KMD-managed channel (V6 SKIP_CTXBUFFER_ALLOC), server creates the virtual context memdesc itself; 9: same V6 channel with the stock client's zero-field VA-only MAIN entry, bInitialize=0; 10: same V6 channel, GPU_INITIALIZE_CTX (0x2080012d) installs the physical context memdesc then PROMOTE_CTX entryCount=0 bare virtAddress+size; 11: mode 10 with a server FBMEM context buffer, VIDMEM physAttr 0x4; 12: V5 UVM/vGPU-faithful PA-only MAIN entry (bInitialize=1 bNonmapped=1, no VA) and continue to BIND/SCHEDULE/submit; 13: mode 12 plus a PATCH PA-only promote first, sized from GET_ENGINE_CONTEXT_PROPERTIES engineId 0x10; 14: KMD channel (V6) + PATCH PA-only then the single-entry MAIN init+VA (the flow that books the context VA and commits ENG_CTX_VALID); 15: mode 14 with a server FBMEM MAIN context buffer and VIDMEM physAttr 0x4; 16: mode 14 plus a PM ctxsw PA-only promote first; 17: no client PROMOTE_CTX, paired with gsp_phys_srv_ctxbuf=1 so the GSP-RM server-manages and commits the context; 18: mode 8 shape plus a client FERMI_CONTEXT_SHARE_A on the TSG (hContextShare) whose construct allocates the GraphicsSubcontext; 19: the stock guest's exact 9-entry promote set (MAIN 2 MB PA+VA, PATCH, global buffers, FECS event, private access maps) on V5; 20: mode 6's no-entry hVirtMemory promote on the KMD-managed V6 channel (SKIP_CTXBUFFER_ALLOC), untested - the combination the V5 refusal (0x1f) points at; 21: mode 7's no-entry bare (virtAddress, size) promote on the V6 KMD-managed channel; 22: GPU_INITIALIZE_CTX then mode 20's no-entry hVirtMemory promote; 23: the hVirtMemory no-entry promote backed by the server-owned FBMEM context object (GSP_RM_ALLOC class 0x40) on the server-managed shared V5 channel - a physical memdesc reaches the per-buffer commit without the ADDR_VIRTUAL vaList gate; 24: the reference FWCLIENT non-3D set {MAIN, PM, PATCH, FECS_EVENT, PRIV_ACCESS_MAP, UNRESTRICTED_PRIV_ACCESS_MAP} as one control on the V6 KMD-managed channel, combined PA+VA entries where the ladder has a VA (MAIN PDB-mapped; PATCH/FECS/UNRESTRICTED booking-only), PM and PRIV_ACCESS_MAP init-only (no VA source), unsourceable buffers omitted and logged); 25: the real golden-client construct on the golden replay handles (captured channel flags 0x00200320 + single atomic 9-entry PROMOTE_CTX on the golden channel), then the V6 FWCLIENT follow-on promote on the golden-initialized GR-instance state; requires gsp_phys_rmcall_golden=1 and gsp_phys_rmcall_engine=1; 26/27: golden9 promote on the V6 execution channel (dead ends); 28: the golden execution lane - the golden channel gets a client GPFIFO ring (VA_GPFIFO), a sysmem host-writable USERD and the ladder TSG as parent, keeps the 9-entry promote, and the ladder switches execution to it (V6 stays unscheduled); 29: mode 28 minus the TURING_COMPUTE_A (0xc5c0) object construct on the client-wired golden channel (every GR object construct there deadlocks RM), i.e. the construct-last golden execution lane; 30: the TSG-sibling split: a clean unwired golden sibling A (captured shape, TSG-parented) runs the 9-entry promote and its own TURING_A first-init while V6 stays the execution channel (no swap) with its FWCLIENT promote suppressed; 31: mode-25's device-parented golden first-init (9-entry promote + TURING_A) followed by a clean TSG-parented sibling A whose 9-entry promote runs after V6's own constructs, so the TSG-wide server commit sets V6's instance pointer + RAMFC while V6 stays the execution channel and its own promote is suppressed; 33: mode 31 with the TSG sibling A wired as the object-carrying execution channel (mode-29 golden_exec_shape wiring: ring at VA_GPFIFO, 8 GPFIFO entries, ladder sysmem USERD, TSG parent, host USERD control offset 0 via shape bit7), A's 9-entry promote (skip_fecs), a TURING_COMPUTE_A (0xc5c0) object constructed on A after the accepted promote, and boot_stage.rmcall_channel swapped to A for stage 4 and all forwarded launches; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_golden_exec_shape=0xE7; 34: the context-share golden lane - a FERMI_CONTEXT_SHARE_A (0x9067, TSG-parented, ladder SHARED_MANAGEMENT VAS, flags SYNC -> VEID 0) is allocated before the replay; the golden replay's channel A (0xcaf00054) joins the ladder TSG through hContextShare with hVASpace=0 while keeping the captured unwired shape, runs the 9-entry promote and carries TURING_A (0xc597) + TURING_COMPUTE_A (0xc5c0, handle 0xcaf00090); V6 (0xcaf00034) attaches to the same share, constructs nothing (a later construct would overwrite the shared VEID's subcontext header) and its promote is suppressed, so stage 4 BIND/SCHEDULE/token/GR-sem and every forwarded launch run on V6 inside A's shared VEID/subcontext; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_vas_shared=1; values outside 0..34 clamp to 0 with a warning");
+ "GR context-promote bisection (default 0: never send PROMOTE_CTX; 1: single entry init+nonmapped=0 size=raw+subctx, the p6h1b shape; 2: two-phase PA-only nonmapped then VA-only with the same fields; 3: single entry size=raw; 4: two-phase with a zero-field VA-only step; 5: batched entryCount=2 PA-only init + zero-field VA-only; 6: no-entry hVirtMemory, NV01_MEMORY_VIRTUAL (class 0x70) on the VAS at the context window + PROMOTE_CTX entryCount=0; 7: no-entry bare virtAddress+size, entryCount=0; 8: single MAIN entry like 1 on the KMD-managed channel (V6 SKIP_CTXBUFFER_ALLOC), server creates the virtual context memdesc itself; 9: same V6 channel with the stock client's zero-field VA-only MAIN entry, bInitialize=0; 10: same V6 channel, GPU_INITIALIZE_CTX (0x2080012d) installs the physical context memdesc then PROMOTE_CTX entryCount=0 bare virtAddress+size; 11: mode 10 with a server FBMEM context buffer, VIDMEM physAttr 0x4; 12: V5 UVM/vGPU-faithful PA-only MAIN entry (bInitialize=1 bNonmapped=1, no VA) and continue to BIND/SCHEDULE/submit; 13: mode 12 plus a PATCH PA-only promote first, sized from GET_ENGINE_CONTEXT_PROPERTIES engineId 0x10; 14: KMD channel (V6) + PATCH PA-only then the single-entry MAIN init+VA (the flow that books the context VA and commits ENG_CTX_VALID); 15: mode 14 with a server FBMEM MAIN context buffer and VIDMEM physAttr 0x4; 16: mode 14 plus a PM ctxsw PA-only promote first; 17: no client PROMOTE_CTX, paired with gsp_phys_srv_ctxbuf=1 so the GSP-RM server-manages and commits the context; 18: mode 8 shape plus a client FERMI_CONTEXT_SHARE_A on the TSG (hContextShare) whose construct allocates the GraphicsSubcontext; 19: the stock guest's exact 9-entry promote set (MAIN 2 MB PA+VA, PATCH, global buffers, FECS event, private access maps) on V5; 20: mode 6's no-entry hVirtMemory promote on the KMD-managed V6 channel (SKIP_CTXBUFFER_ALLOC), untested - the combination the V5 refusal (0x1f) points at; 21: mode 7's no-entry bare (virtAddress, size) promote on the V6 KMD-managed channel; 22: GPU_INITIALIZE_CTX then mode 20's no-entry hVirtMemory promote; 23: the hVirtMemory no-entry promote backed by the server-owned FBMEM context object (GSP_RM_ALLOC class 0x40) on the server-managed shared V5 channel - a physical memdesc reaches the per-buffer commit without the ADDR_VIRTUAL vaList gate; 24: the reference FWCLIENT non-3D set {MAIN, PM, PATCH, FECS_EVENT, PRIV_ACCESS_MAP, UNRESTRICTED_PRIV_ACCESS_MAP} as one control on the V6 KMD-managed channel, combined PA+VA entries where the ladder has a VA (MAIN PDB-mapped; PATCH/FECS/UNRESTRICTED booking-only), PM and PRIV_ACCESS_MAP init-only (no VA source), unsourceable buffers omitted and logged); 25: the real golden-client construct on the golden replay handles (captured channel flags 0x00200320 + single atomic 9-entry PROMOTE_CTX on the golden channel), then the V6 FWCLIENT follow-on promote on the golden-initialized GR-instance state; requires gsp_phys_rmcall_golden=1 and gsp_phys_rmcall_engine=1; 26/27: golden9 promote on the V6 execution channel (dead ends); 28: the golden execution lane - the golden channel gets a client GPFIFO ring (VA_GPFIFO), a sysmem host-writable USERD and the ladder TSG as parent, keeps the 9-entry promote, and the ladder switches execution to it (V6 stays unscheduled); 29: mode 28 minus the TURING_COMPUTE_A (0xc5c0) object construct on the client-wired golden channel (every GR object construct there deadlocks RM), i.e. the construct-last golden execution lane; 30: the TSG-sibling split: a clean unwired golden sibling A (captured shape, TSG-parented) runs the 9-entry promote and its own TURING_A first-init while V6 stays the execution channel (no swap) with its FWCLIENT promote suppressed; 31: mode-25's device-parented golden first-init (9-entry promote + TURING_A) followed by a clean TSG-parented sibling A whose 9-entry promote runs after V6's own constructs, so the TSG-wide server commit sets V6's instance pointer + RAMFC while V6 stays the execution channel and its own promote is suppressed; 33: mode 31 with the TSG sibling A wired as the object-carrying execution channel (mode-29 golden_exec_shape wiring: ring at VA_GPFIFO, 8 GPFIFO entries, ladder sysmem USERD, TSG parent, host USERD control offset 0 via shape bit7), A's 9-entry promote (skip_fecs), a TURING_COMPUTE_A (0xc5c0) object constructed on A after the accepted promote, and boot_stage.rmcall_channel swapped to A for stage 4 and all forwarded launches; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_golden_exec_shape=0xE7; 34: the context-share golden lane - a FERMI_CONTEXT_SHARE_A (0x9067, TSG-parented, ladder SHARED_MANAGEMENT VAS, flags SYNC -> VEID 0) is allocated before the replay; the golden replay's channel A (0xcaf00054) joins the ladder TSG through hContextShare with hVASpace=0 while keeping the captured unwired shape, runs the 9-entry promote and carries TURING_A (0xc597) + TURING_COMPUTE_A (0xc5c0, handle 0xcaf00090); V6 (0xcaf00034) attaches to the same share, constructs nothing (a later construct would overwrite the shared VEID's subcontext header) and its promote is suppressed, so stage 4 BIND/SCHEDULE/token/GR-sem and every forwarded launch run on V6 inside A's shared VEID/subcontext; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_vas_shared=1; values outside 0..34 clamp to 0 with a warning; 35: the real-client user lane - after the golden replay (mode-25/31 shape with the capture-refreshed FECS_EVENT/PRIV_ACCESS_MAP entries) the ladder promotes the two TSG fault-method buffers (0xa06c010a), allocates the mode-34 FERMI_CONTEXT_SHARE_A, builds V7 (0xcaf000a0: class 0xc46f, TSG parent, hVASpace=0, hContextShare, captured flags 0x00201320, internalFlags 0x1d, hObjectError/errorNotifierMem sysmem, FBMEM instance/ramfc, gpFifoEntries 8, sysmem USERD), sends the one atomic 3-entry MAIN/PM/PATCH PA-only nonmapped promote on V7 before any construct, constructs TURING_COMPUTE_A (0xc5c0) on V7 and swaps stage 4 + forwarding to V7; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_vas_shared=1; values outside 0..35 clamp to 0 with a warning");
int gsp_phys_rmcall_gr_promote_value(void)
{