diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-06 20:15:45 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-06 20:16:31 -0400 |
| commit | 0c1e58b9ceeeff42931222ab14e36a18530a6b7b (patch) | |
| tree | 9151c671806564508f0c4974c090a42f1443f4f0 | |
| parent | 23d8683f3012ac579db424735d1418814bb7ff01 (diff) | |
| download | open-iov-0c1e58b9ceeeff42931222ab14e36a18530a6b7b.tar.gz open-iov-0c1e58b9ceeeff42931222ab14e36a18530a6b7b.zip | |
vgsp: mode 35 - real-client user lane (TSG fault buffers, share, V7 channel, 3-entry PA-only promote, V7 execution swap; compile-verified, untested)
Replays the decoded real 570.144 GSP client compute-context sequence on the
ladder, additive to modes 0-34:
- TSG fault-method buffers: one 0xa06c010a on the ladder TSG with two
0x2000 {base,size,alignment=1,addressSpace,cache} descriptors + zero
bar2Addr words + numValidEntries=2; server FBMEM first, registered host
sysmem fallback (capture order TSG -> faults -> share -> channel).
- FERMI_CONTEXT_SHARE_A (0x9067) reuse of the mode-34 helper (ladder TSG,
SHARED_MANAGEMENT VAS, flags SYNC -> VEID 0).
- V7 channel 0xcaf000a0: class 0xc46f, TSG parent, hVASpace=0, hContextShare,
captured flags 0x00201320 and internalFlags 0x1d, engineType 0, sysmem
hObjectError/errorNotifierMem, FBMEM instance/ramfc, sysmem USERD.
- One atomic 3-entry GPU_PROMOTE_CTX on V7 (MAIN/PM/PATCH, PA-only,
physAttr 4, bInitialize=1, bNonmapped=1) before any 0xc5c0 construct.
- TURING_COMPUTE_A (0xc5c0) on V7 after the accepted promote; execution
channel swapped to V7 for stage 4 BIND/SCHEDULE/token/GR-sem and
forwarding (V6 allocated but unscheduled; its promote suppressed).
- Mode-35-only golden9 entry refresh: FECS_EVENT attr=5 with the ctx-window
VA in the physAddr field, PRIV_ACCESS_MAP va=0/bNonmapped=1.
- gr_promote clamp/param doc extended to 0..35.
| -rw-r--r-- | kernel/gsp_phys_booter.c | 738 | ||||
| -rw-r--r-- | kernel/gsp_phys_gpu.c | 6 |
2 files changed, 710 insertions, 34 deletions
diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 90b960e..ca58c28 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -254,6 +254,14 @@ bool gsp_phys_boot_requested(void) #define GSP_PHYS_RMCALL_CTRL_GPFIFO_SCHEDULE 0xa06f0103u #define GSP_PHYS_RMCALL_CTRL_BIND_CHANNEL 0xa06f0104u #define GSP_PHYS_RMCALL_CTRL_TSG_GPFIFO_SCHEDULE 0xa06c0101u +/* + * NVA06C_CTRL_CMD_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS (0xa06c010a, + * ctrla06c.h:354-364): set the per-runqueue fault method buffer descriptors + * on the channel group (TSG). Mode 35 sends it immediately after the ladder + * TSG exists, before the share/channel batch, exactly where the real + * 570.144 GSP client sends it (capture #821). + */ +#define GSP_PHYS_RMCALL_CTRL_TSG_PROMOTE_FAULT_BUFFERS 0xa06c010au #define GSP_PHYS_RMCALL_CTRL_DMA_SET_PAGE_DIRECTORY 0x801813u #define GSP_PHYS_RMCALL_CTRL_DMA_UPDATE_PDE_2 0x80180fu #define GSP_PHYS_RMCALL_ENGINE_COPY0 0x9u @@ -704,6 +712,52 @@ bool gsp_phys_boot_requested(void) */ #define GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE 34u /* + * Mode 35 (2026-10-06, experiment): the real-client user lane. The 570.144 + * GSP client's captured compute-context sequence is replayed on the ladder, + * in the capture's order, after the existing golden replay (D) has built the + * golden image: + * + * 1. TSG (ladder TSG, existing) - the capture's #819; + * 2. TSG fault-method-buffer promote 0xa06c010a (#821): two 0x2000 + * descriptors {base, size, alignment=1, addressSpace, cache} + two zero + * bar2Addr words + numValidEntries=2; server FBMEM first, registered + * host sysmem fallback; + * 3. FERMI_CONTEXT_SHARE_A 0x9067 (#823), parent = ladder TSG, VAS = the + * ladder SHARED_MANAGEMENT VAS, flags SYNC (VEID 0) - the mode-34 + * helper, allocated before the replay like mode 34; + * 4. V7 (0xcaf000a0, #825): class 0xc46f, parent = TSG, hVASpace=0, + * hContextShare = the share, flags 0x00201320 (PRIVILEGED | + * USERD_INDEX=3 | USERD_PAGE_FIXED, no SKIP bits), internalFlags 0x1d + * (ADMIN + error-notifier MEMORY), engineType 0 (the TSG carries GR0), + * hObjectError = a registered sysmem notifier object, errorNotifierMem = + * its sysmem 0x1000 memdesc, instance/ramfc FBMEM memdescs, the proven + * sysmem USERD chunk with hUserdMemory[0]=0 (the captured handle is + * client-local; deviation logged). Recorded deviations from #825: + * gpFifoEntries 8 (the ladder's ring; captured 1024) and the USERD + * aperture (captured memory object/FB). + * 5. lane promote (#827): ONE atomic GPU_PROMOTE_CTX on V7 with exactly 3 + * entries - MAIN (the GR ctx FBMEM MAIN), PM (a fresh FBMEM buffer, + * captured raw size 0xd100, allocation page-rounded), PATCH (the GR + * FBMEM aux PATCH) - all PA-only, physAttr=4 (VIDMEM), bInitialize=1, + * bNonmapped=1 - sent BEFORE any 0xc5c0 construct on V7; + * 6. TURING_COMPUTE_A (0xc5c0) on V7 after the accepted promote (#829); + * 7. execution swap: stage 4 BIND/SCHEDULE/token/GR-sem and every + * forwarded launch run on V7 (rmcall_channel = V7, server_mem=false). + * V6 is still allocated (share child, unscheduled) as in mode 33/34; + * its FWCLIENT promote and GR-object constructs are suppressed. + * + * The golden replay D runs the mode-25/31 shape (device-parented golden + * channel, 9-entry promote, TURING_A) and, for this mode only, with the + * capture-refreshed FECS_EVENT (attr=5, VA in the physAddr field) and + * PRIV_ACCESS_MAP (va=0, bNonmapped=1) entries from the current #471 + * decode; every other golden mode keeps its historical entries. + * + * Requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and + * gsp_phys_rmcall_vas_shared=1 (the share and the V7 GPFIFO VA ride the + * ladder SHARED_MANAGEMENT VAS). Any failure aborts stage 3 with no retry. + */ +#define GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE 35u +/* * Mode-28 golden execution channel shape bits (gsp_phys_rmcall_golden_exec_shape, * default 7). Every combination keeps the captured flags 0x00200320 and the * FBMEM instance/RAMFC/mthd slices; the bits only choose the client-visible @@ -817,7 +871,7 @@ bool gsp_phys_boot_requested(void) */ #define GSP_PHYS_RMCALL_FWCLIENT_MASK_FECS_SYSMEM (1u << 16) #define GSP_PHYS_RMCALL_GR_PROMOTE_MAX \ - (GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) + (GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) /* * Work-submit token notifier slot. The default index is @@ -1108,6 +1162,41 @@ bool gsp_phys_boot_requested(void) #define GSP_PHYS_RMCALL_HANDLE_GOLDEN_CTXSHARE_GR_OBJ 0xcaf00090u /* + * Mode 35 (real-client user lane) handles. 0xcaf00094..0xcaf0009f are free + * between the mode-34 compute object and 0xcaf000a0; the registered sysmem + * method-buffer objects (0xcaf000c0/0xc4) sit below the 0xcaf01000 + * registered-buffer base used by the shared chunk table. + */ +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE 0xcaf000a0u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FB 0xcaf000a4u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ 0xcaf000a8u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER 0xcaf000acu +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB0 0xcaf000b0u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB1 0xcaf000b4u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_PM_FB 0xcaf000b8u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM0 0xcaf000c0u +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM1 0xcaf000c4u + +/* + * Mode-35 V7 channel flags/privilege and fault/PM geometry, decoded from the + * current 570.144 client capture: + * flags 0x00201320 = PRIVILEGED_CHANNEL (bit 5) | USERD_INDEX_VALUE=3 + * (bits 10:8) | USERD_INDEX_PAGE_FIXED (bit 21, page 1 in bits 20:12); + * no SKIP_SCRUBBER (23), no SKIP_CTXBUFFER_ALLOC (31); + * internalFlags 0x1d = privilege ADMIN (bits 1:0=1) | error-notifier + * MEMORY (bits 3:2=3) | ECC-notifier type 1 (bits 5:4=1); + * TSG fault method buffers: two descriptors of 0x2000 bytes each; + * PM ctxsw buffer: the captured raw size 0xd100 (the allocation is page + * rounded; the promote entry keeps the captured raw size). + */ +#define GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE \ + (u32)(GSP_PHYS_RMCALL_CHANNEL_FLAG_PRIVILEGED | (3u << 8) | \ + (1u << 21)) +#define GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE 0x1du +#define GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE 0x2000u +#define GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE 0xd100u + +/* * Golden-image replay sizes (kernel_graphics.c:2058-2060, 2273-2276): the * GPFIFO pushbuffer is 32 entries x NVA06F_GP_ENTRY__SIZE (8 B, * class/clc46f.h:264) = 256 B, and the client USERD is one 4 KB page (the @@ -1905,6 +1994,14 @@ static struct gsp_phys_boot_stage { u32 rmcall_priv_size; u32 rmcall_unr_size; /* + * Mode 35 (real-client user lane): the TSG fault-method-buffer sysmem + * fallback chunks (two 0x2000 pages, registered as method-buffer + * objects) and the sysmem error-notifier page registered as V7's + * hObjectError memory object. All freed by gsp_phys_rmcall_fini(). + */ + struct gsp_phys_boot_chunk rmcall_user_lane_fault[2]; + struct gsp_phys_boot_chunk rmcall_user_lane_notifier; + /* * GR context-promote bisection snapshot (stage 3, GR0 only): the 0..3 * gsp_phys_rmcall_gr_promote value, the number of accepted promote * controls (0 = none, 1 = PA/init accepted, 2 = VA accepted; mode 2 @@ -2627,6 +2724,11 @@ static void gsp_phys_rmcall_reset(void) sizeof(boot_stage.rmcall_chunks[i])); } memset(&boot_stage.rmcall_ctx, 0, sizeof(boot_stage.rmcall_ctx)); + for (i = 0; i < 2; i++) + memset(&boot_stage.rmcall_user_lane_fault[i], 0, + sizeof(boot_stage.rmcall_user_lane_fault[i])); + memset(&boot_stage.rmcall_user_lane_notifier, 0, + sizeof(boot_stage.rmcall_user_lane_notifier)); for (i = 0; i < GSP_PHYS_RMCALL_GR_CTX_PT_MAX; i++) memset(&boot_stage.rmcall_ctx_pt[i], 0, sizeof(boot_stage.rmcall_ctx_pt[i])); @@ -7738,6 +7840,8 @@ static const char *gsp_phys_rmcall_gr_promote_name(void) return "mode 33: mode 31 with the TSG sibling A wired as the object-carrying execution channel (ring/sysmem USERD/TSG parent, TURING_COMPUTE_A on A, execution swap to A)"; case GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE: return "mode 34: golden channel A (0xcaf00054) TSG-parented through FERMI_CONTEXT_SHARE_A (0x9067, VEID 0, ladder VAS), captured unwired shape + 9-entry promote + TURING_A + TURING_COMPUTE_A; V6 (0xcaf00034) executes on the same share with no object of its own and its promote suppressed"; + case GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE: + return "mode 35: real-client user lane (TSG fault-buffer promote + share 0x9067 + V7 0xcaf000a0 captured channel flags 0x00201320/internalFlags 0x1d + one atomic 3-entry MAIN/PM/PATCH PA-only promote on V7 + TURING_COMPUTE_A on V7; execution switched to V7, V6 unscheduled)"; default: return "off (PROMOTE_CTX never sent)"; } @@ -7932,7 +8036,9 @@ static int gsp_phys_rmcall_alloc_channel(void __iomem *bar0) boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE)) { + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE)) { vars = &gsp_phys_rmcall_channel_variant_skipctx; attempts = 1; } @@ -8144,6 +8250,19 @@ static void gsp_phys_rmcall_golden_3d_on_ladder(void __iomem *bar0) GSP_PHYS_RMCALL_HANDLE_GOLDEN_CHANNEL); return; } + /* + * Mode 35: the execution channel is V7, which gets its own + * TURING_COMPUTE_A after the lane promote; V6 stays unscheduled and + * must not attach the ladder's extra 3D object (the capture never + * allocates 0xc597 on the user-lane channel). + */ + if (boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { + pr_info("vgsp-boot: rm-call: mode 35: no GOLDEN TURING_A on V6 0x%x; the execution channel is V7 0x%x and the capture carries no 0xc597 on the user lane\n", + boot_stage.rmcall_channel, + GSP_PHYS_RMCALL_HANDLE_USER_LANE); + return; + } ret = gsp_phys_rmcall_alloc_gr_object_class(bar0, GSP_PHYS_RMCALL_CLASS_GR_3D, GSP_PHYS_RMCALL_HANDLE_GOLDEN_GR3D_LADDER, @@ -8413,7 +8532,9 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0) boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) && + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) && gsp_phys_rmcall_engine_is_gr()) { if (gsp_phys_rmcall_fwclient_mask_value() & GSP_PHYS_RMCALL_GOLDEN9_MASK_LADDER_VAS) { @@ -8529,7 +8650,9 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0) boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) && + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) && gsp_phys_rmcall_engine_is_gr()) { ch.flags = cpu_to_le32(GSP_PHYS_RMCALL_CHANNEL_FLAG_GOLDEN9); ch.internal_flags = cpu_to_le32( @@ -8760,7 +8883,9 @@ static int gsp_phys_rmcall_golden_replay(void __iomem *bar0) boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) && + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) && gsp_phys_rmcall_engine_is_gr()) { ret = gsp_phys_rmcall_golden9_promote(bar0); if (ret) @@ -9704,9 +9829,11 @@ static int gsp_phys_rmcall_gr_ctx_prepare(void __iomem *bar0) * allocation (VIDMEM aperture, the shape the KMD promote path * expects); mode 23 uses the same FBMEM object as the hVirtMemory * promote target; mode 25 and mode 31 use the FBMEM MAIN that made - * the golden first-init construct succeed (m25c); every other mode - * keeps the host sysmem chunk. The PT pages are always host sysmem - * pages behind the sysmem-aperture PD0. + * the golden first-init construct succeed (m25c); mode 34 and mode 35 + * use it so the golden replay's D promote and the mode-35 lane + * promote carry a VIDMEM MAIN; every other mode keeps the host + * sysmem chunk. The PT pages are always host sysmem pages behind the + * sysmem-aperture PD0. */ if (boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_OLDIFACE_FB || @@ -9721,7 +9848,9 @@ static int gsp_phys_rmcall_gr_ctx_prepare(void __iomem *bar0) boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) { + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { u64 fb_base = 0; ret = gsp_phys_rmcall_alloc_fb_mem(bar0, @@ -9776,14 +9905,18 @@ static int gsp_phys_rmcall_gr_ctx_prepare(void __iomem *bar0) /* * Mode-25 golden9e and mode-31 (whose D first-init replay uses the * same golden9 promote): prepare the FBMEM PATCH/FECS/PRIV/UNR - * buffers now, before the PDB build maps their promoted VAs. + * buffers now, before the PDB build maps their promoted VAs. Mode 35 + * uses the same FBMEM aux set for its D replay promote and for the + * V7 lane promote's PATCH entry. */ if (boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN9 || boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC) { + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { ret = gsp_phys_rmcall_golden9_fb_aux_prepare(bar0); if (ret) { pr_err("vgsp-boot: rm-call: GOLDEN9E FBMEM aux prepare failed (%d); aborting the ladder with no retry\n", @@ -10132,16 +10265,46 @@ static int gsp_phys_rmcall_golden9_promote_on(void __iomem *bar0, * promote failure. A only carries the TSG group commit, which the * MAIN/PATCH entries drive. */ - if (!skip_fecs) + if (!skip_fecs) { + /* + * Mode 35 fidelity refresh: the current capture's FECS_EVENT + * form is attr 0x5 with the ctx-window VA in the physAddr field + * (the captured pa field carried a VA, not a PA); every other + * golden mode keeps the historical combined PA+VA form. The + * VA used is the FECS buffer's mapped ctx-window VA. + */ + if (boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) + gsp_phys_rmcall_promote_entry_fill(p, entry_count++, + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_FECS_EVENT, + GSP_PHYS_RMCALL_VA_GOLDEN9_FECS, + GSP_PHYS_RMCALL_VA_GOLDEN9_FECS, + boot_stage.rmcall_fecs_size, 0x5u, 1, 0); + else + gsp_phys_rmcall_promote_entry_fill(p, entry_count++, + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_FECS_EVENT, + boot_stage.rmcall_fecs_pa, + GSP_PHYS_RMCALL_VA_GOLDEN9_FECS, + boot_stage.rmcall_fecs_size, fattr, 1, 0); + } + /* + * Mode 35 fidelity refresh: the current capture's PRIV_ACCESS_MAP + * entry is PA-only (va=0) with bNonmapped=1; every other golden mode + * keeps the historical combined PA+VA form. + */ + if (boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) gsp_phys_rmcall_promote_entry_fill(p, entry_count++, - GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_FECS_EVENT, - boot_stage.rmcall_fecs_pa, - GSP_PHYS_RMCALL_VA_GOLDEN9_FECS, - boot_stage.rmcall_fecs_size, fattr, 1, 0); - gsp_phys_rmcall_promote_entry_fill(p, entry_count++, - GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PRIV_MAP, - boot_stage.rmcall_priv_pa, GSP_PHYS_RMCALL_VA_GOLDEN9_PRIV, - boot_stage.rmcall_priv_size, fattr, 1, 0); + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PRIV_MAP, + boot_stage.rmcall_priv_pa, 0, + boot_stage.rmcall_priv_size, + GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1); + else + gsp_phys_rmcall_promote_entry_fill(p, entry_count++, + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PRIV_MAP, + boot_stage.rmcall_priv_pa, + GSP_PHYS_RMCALL_VA_GOLDEN9_PRIV, + boot_stage.rmcall_priv_size, fattr, 1, 0); gsp_phys_rmcall_promote_entry_fill(p, entry_count++, GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_UNR_PRIV_MAP, boot_stage.rmcall_unrestricted_pa, GSP_PHYS_RMCALL_VA_GOLDEN9_UNR, @@ -10526,26 +10689,30 @@ static int gsp_phys_rmcall_gr_ctx_promote(void __iomem *bar0) mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING || mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT || mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || - mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) && + mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) && (!gsp_phys_rmcall_golden_enabled() || !gsp_phys_gpu_enabled())) { pr_err("vgsp-boot: rm-call: mode %u requires gsp_phys_rmcall_golden=1 and gsp_phys_gpu=1 (the golden replay did not run); no PROMOTE_CTX is sent and the ladder aborts with no retry\n", mode); return -EINVAL; } /* - * Mode 30/31/33/34: the sibling A's 9-entry promote carries the group - * context commit; nothing is sent on the V6 channel. A multi-entry - * promote containing MAIN is refused for a SKIP-ctx channel - * (graphics_context.c: bKMDVirtualContext -> NV_ERR_INVALID_ARGUMENT) - * and the single-MAIN exemption needs the pCtxPhysMemDesc the FWCLIENT - * shapes never establish, so a V6 promote would abort stage 3 on 0x1f. - * Mode 34's promote ran on the context-share golden channel A inside - * the replay (skip_fecs=false), before V6 existed. + * Mode 30/31/33/34/35: the sibling A's 9-entry promote carries the + * group context commit; nothing is sent on the V6 channel. A + * multi-entry promote containing MAIN is refused for a SKIP-ctx + * channel (graphics_context.c: bKMDVirtualContext -> + * NV_ERR_INVALID_ARGUMENT) and the single-MAIN exemption needs the + * pCtxPhysMemDesc the FWCLIENT shapes never establish, so a V6 + * promote would abort stage 3 on 0x1f. Mode 34's promote ran on the + * context-share golden channel A inside the replay (skip_fecs=false), + * before V6 existed; mode 35's 3-entry lane promote ran on V7 after + * V6's alloc (none is sent on V6). */ if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING || mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT || mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || - mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) { + mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING) pr_info("vgsp-boot: rm-call: mode 30: no PROMOTE_CTX on V6; the sibling A carries the group context commit (config_group ppEngCtxDesc), V6 inherits via the TSG commit/channel setup\n"); else if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_COMMIT) @@ -10558,6 +10725,13 @@ static int gsp_phys_rmcall_gr_ctx_promote(void __iomem *bar0) GSP_PHYS_RMCALL_HANDLE_GOLDEN_GR_OBJ, GSP_PHYS_RMCALL_HANDLE_GOLDEN_CTXSHARE_GR_OBJ, boot_stage.rmcall_channel); + else if (mode == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) + pr_info("vgsp-boot: rm-call: mode 35: no PROMOTE_CTX on V6 0x%x; the one atomic 3-entry MAIN/PM/PATCH PA-only lane promote already ran on V7 0x%x (hContextShare=0x%x, TSG 0x%x) and V7 carries TURING_COMPUTE_A 0x%x - stage 4 and forwarding execute on V7\n", + boot_stage.rmcall_channel, + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + boot_stage.rmcall_ctxshare, + boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ); else pr_info("vgsp-boot: rm-call: mode 33: no PROMOTE_CTX on V6; wired sibling A's 9-entry promote ran (TSG commit) and A carries both the class object and the execution (hChannel=0x%x, hGr=0x%x)\n", GSP_PHYS_RMCALL_HANDLE_GOLDEN_SIBLING, @@ -12243,6 +12417,388 @@ static int gsp_phys_rmcall_alloc_ctxshare(void __iomem *bar0) return 0; } +/* + * Mode 35: the TSG fault-method-buffer descriptor pair + * (NVA06C_CTRL_CMD_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS 0xa06c010a). The + * real client's capture carries two 32-byte + * NV2080_CTRL_INTERNAL_MEMDESC_INFO entries {base, size=0x2000, alignment=1, + * addressSpace, cpuCacheAttrib}, two zero bar2Addr words and + * numValidEntries=2 (one per runqueue). Mode 35 sends the identical shape on + * the ladder TSG immediately after the TSG exists, before the share and the + * V7 channel - the capture's order (#819 -> #821 -> #823 -> #825). + */ +struct gsp_phys_rmcall_fault_memdesc { + __le64 base; + __le64 size; + __le64 alignment; + __le32 address_space; + __le32 cpu_cache_attrib; +}; + +struct gsp_phys_rmcall_fault_buffers_params { + struct gsp_phys_rmcall_fault_memdesc memdesc[2]; + __le64 bar2_addr[2]; + __le32 num_valid_entries; + __le32 pad; +}; + +/* + * Mode 35: register one arbitrary sysmem chunk as an + * NV01_MEMORY_LIST_SYSTEM object. This is the gsp_phys_rmcall_register_mem + * body generalised so mode 35 can register its fault-buffer/notifier + * fallback chunks without adding entries to the shared ladder chunk table; + * every existing mode keeps its exact stage-2 RPC sequence. + */ +static int gsp_phys_rmcall_register_chunk(void __iomem *bar0, + struct gsp_phys_boot_chunk *c, + u32 handle, const char *what) +{ + struct gsp_phys_rmcall_alloc_memory_req *req; + struct gsp_phys_rx_record rec; + u32 pages = c->pages; + size_t req_len = offsetof(struct gsp_phys_rmcall_alloc_memory_req, pte) + + (size_t)pages * sizeof(__le64); + u32 i; + int ret; + + req = kvzalloc(req_len, GFP_KERNEL); + if (!req) + return -ENOMEM; + req->h_client = cpu_to_le32(boot_stage.rmcall_client); + req->h_device = cpu_to_le32(boot_stage.rmcall_device); + req->h_memory = cpu_to_le32(handle); + req->h_class = cpu_to_le32(GSP_PHYS_RMCALL_CLASS_MEMLIST_SYSTEM); + req->flags = cpu_to_le32(c->page_dma ? + GSP_PHYS_RMCALL_OS02_FLAG_NONCONTIGUOUS : 0); + req->pte_adjust = 0; + req->format = 0; + req->length = cpu_to_le64((u64)pages * GSP_PHYS_PAGE_SIZE); + req->page_count = cpu_to_le32(pages); + req->pte_desc.idr_len = cpu_to_le32((pages & 0xffffu) << 16); + for (i = 0; i < pages; i++) + req->pte[i] = cpu_to_le64(gsp_phys_chunk_page_dma(c, i) >> 12); + + ret = gsp_phys_rmcall_rpc(bar0, GSP_PHYS_RMCALL_FUNC_ALLOC_MEMORY, + req, (u32)req_len, 0, &rec, true, NULL); + kvfree(req); + if (ret) { + pr_err("vgsp-boot: rm-call: mode 35 %s ALLOC_MEMORY (hMemory=0x%x dma=0x%llx) failed: %d\n", + what, handle, (u64)gsp_phys_chunk_page_dma(c, 0), ret); + gsp_phys_rx_record_free(&rec); + return ret; + } + gsp_phys_rx_record_free(&rec); + pr_info("vgsp-boot: rm-call: mode 35 %s registered: hMemory=0x%x class=0x%x %u page(s) dma=0x%llx\n", + what, handle, GSP_PHYS_RMCALL_CLASS_MEMLIST_SYSTEM, pages, + (u64)gsp_phys_chunk_page_dma(c, 0)); + return 0; +} + +/* + * Mode 35 step 2: promote the TSG fault-method buffers. Each of the two + * runqueue buffers is a server FBMEM allocation first (alloc_fb_mem - the + * shape the other ladder context buffers use) with a registered host sysmem + * chunk as fallback; the capture itself carried addressSpace=1 (sysmem), so + * the FBMEM aperture is the one logged deviation here. The two base PAs and + * the rest of the fields are filled exactly as the capture does. + */ +static int gsp_phys_rmcall_tsg_fault_buffers(void __iomem *bar0) +{ + static const u32 fb_handles[2] = { + GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB0, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_FB1, + }; + static const u32 mem_handles[2] = { + GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM0, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM1, + }; + struct gsp_phys_rmcall_fault_buffers_params p; + u32 rpc_result = 0, status = 0; + u32 i; + int ret; + + memset(&p, 0, sizeof(p)); + for (i = 0; i < 2; i++) { + u64 fb = 0; + + ret = gsp_phys_rmcall_alloc_fb_mem(bar0, fb_handles[i], + GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE, &fb, + "TSG fault buffer"); + if (!ret) { + p.memdesc[i].base = cpu_to_le64(fb); + p.memdesc[i].address_space = + cpu_to_le32(GSP_PHYS_RMCALL_ADDR_FBMEM); + pr_info("vgsp-boot: rm-call: mode 35 TSG fault buffer %u: server FBMEM base=0x%llx size=0x%x\n", + i, fb, GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE); + } else { + struct gsp_phys_boot_chunk *c = + &boot_stage.rmcall_user_lane_fault[i]; + + if (!boot_stage.pdev) + return -EINVAL; + if (!c->pages) { + ret = gsp_phys_chunk_alloc(boot_stage.pdev, c, + GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE / + GSP_PHYS_PAGE_SIZE); + if (ret) { + pr_err("vgsp-boot: rm-call: mode 35 TSG fault buffer %u sysmem fallback allocation failed (%d); no fault-buffer promote is sent and the ladder aborts with no retry\n", + i, ret); + return ret; + } + } + gsp_phys_chunk_zero(c); + ret = gsp_phys_rmcall_register_chunk(bar0, c, + mem_handles[i], + "TSG fault buffer (sysmem fallback)"); + if (ret) + return ret; + p.memdesc[i].base = cpu_to_le64( + (u64)gsp_phys_chunk_page_dma(c, 0)); + p.memdesc[i].address_space = + cpu_to_le32(GSP_PHYS_RMCALL_ADDR_SYSMEM); + pr_info("vgsp-boot: rm-call: mode 35 TSG fault buffer %u: sysmem fallback pa=0x%llx size=0x%x (capture used as=1; FBMEM was tried first)\n", + i, (u64)gsp_phys_chunk_page_dma(c, 0), + GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE); + } + p.memdesc[i].size = + cpu_to_le64(GSP_PHYS_RMCALL_TSG_FAULT_BUF_SIZE); + p.memdesc[i].alignment = cpu_to_le64(1); + p.memdesc[i].cpu_cache_attrib = cpu_to_le32(0); + p.bar2_addr[i] = cpu_to_le64(0); + } + p.num_valid_entries = cpu_to_le32(2); + + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_CTRL_TSG_PROMOTE_FAULT_BUFFERS, + &p, sizeof(p), 16, NULL, 0, &rpc_result, &status, + "INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS"); + pr_info("vgsp-boot: rm-call: mode 35 TSG fault-method-buffer promote: hTsg=0x%x numValid=2 md0={base=0x%llx size=0x%llx as=%u align=0x%llx} md1={base=0x%llx size=0x%llx as=%u} bar2=0 status=0x%x rpc_result=0x%x ret=%d\n", + boot_stage.rmcall_tsg, + le64_to_cpu(p.memdesc[0].base), + le64_to_cpu(p.memdesc[0].size), + le32_to_cpu(p.memdesc[0].address_space), + le64_to_cpu(p.memdesc[0].alignment), + le64_to_cpu(p.memdesc[1].base), + le64_to_cpu(p.memdesc[1].size), + le32_to_cpu(p.memdesc[1].address_space), + status, rpc_result, ret); + return ret; +} + +/* + * Mode 35 step 4a: the V7 error-notifier page. The capture's hObjectError + * 0x5c000018 is a client-local memory object holding the 0x1000-byte + * notifier page (it is referenced by #825 but never appears as an alloc in + * the capture); the ladder's closest equivalent is a registered host sysmem + * page (NV01_MEMORY_LIST_SYSTEM, addressSpace=1 like the captured + * errorNotifierMem) whose handle becomes hObjectError. + */ +static int gsp_phys_rmcall_user_lane_notifier(void __iomem *bar0) +{ + struct gsp_phys_boot_chunk *c = &boot_stage.rmcall_user_lane_notifier; + int ret; + + if (!boot_stage.pdev) + return -EINVAL; + if (!c->pages) { + ret = gsp_phys_chunk_alloc(boot_stage.pdev, c, 1u); + if (ret) { + pr_err("vgsp-boot: rm-call: mode 35 user-lane error-notifier page allocation failed (%d); no V7 channel is allocated and the ladder aborts with no retry\n", + ret); + return ret; + } + } + gsp_phys_chunk_zero(c); + ret = gsp_phys_rmcall_register_chunk(bar0, c, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER, + "user-lane error-notifier (sysmem)"); + if (ret) + return ret; + pr_info("vgsp-boot: rm-call: mode 35 user-lane error notifier: hObject=0x%x sysmem pa=0x%llx size=0x%x (closest equivalent of the capture's hObjectError 0x5c000018 object, which the capture never allocated; deviation logged)\n", + GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER, + (u64)gsp_phys_chunk_page_dma(c, 0), + GSP_PHYS_RMCALL_NOTIFIER_SIZE); + return 0; +} + +/* + * Mode 35 step 4b: the V7 real-client user-lane channel (0xcaf000a0). Every + * captured #825 field is carried: class 0xc46f, hParent = ladder TSG, + * hVASpace 0, hContextShare (the mode-34 share), flags 0x00201320, + * internalFlags 0x1d, engineType 0, hObjectError = the registered notifier + * object, errorNotifierMem = its sysmem memdesc, instance/ramfc FBMEM + * memdescs, mthdbuf sysmem. Recorded deviations: gpFifoEntries 8 (the + * ladder ring; captured 1024, a wrap-size difference only), gpFifoOffset at + * the ladder VA (captured a client VA), hUserdMemory[0]=0 with the proven + * sysmem USERD chunk (captured a client USERD memory object/FB aperture), + * and ProcessID 0 (captured 695446, cosmetic). The server validates and + * returns; a rejection aborts stage 3 with no retry. + */ +static int gsp_phys_rmcall_alloc_user_lane_channel(void __iomem *bar0) +{ + struct gsp_phys_rmcall_channel_params ch; + u64 fb = 0; + u32 rpc = 0, status = 0; + int ret; + + ret = gsp_phys_rmcall_alloc_fb_mem(bar0, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_FB, 0x1000, &fb, + "mode 35 user-lane instance/RAMFC"); + if (ret) + return ret; + ret = gsp_phys_rmcall_user_lane_notifier(bar0); + if (ret) + return ret; + + memset(&ch, 0, sizeof(ch)); + ch.h_object_error = + cpu_to_le32(GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER); + ch.gp_fifo_offset = cpu_to_le64(GSP_PHYS_RMCALL_VA_GPFIFO); + ch.gp_fifo_entries = cpu_to_le32(GSP_PHYS_RMCALL_GPFIFO_ENTRIES); + ch.flags = cpu_to_le32(GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE); + ch.h_context_share = cpu_to_le32(boot_stage.rmcall_ctxshare); + ch.h_va_space = cpu_to_le32(0); + ch.engine_type = cpu_to_le32(0); /* the TSG carries GR0 */ + ch.cid = cpu_to_le32(0); + ch.sub_device_id = cpu_to_le32(0); + ch.internal_flags = cpu_to_le32( + GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE); + ch.instance_mem = gsp_phys_rmcall_mem_desc_fb(fb, 0x1000); + ch.ramfc_mem = gsp_phys_rmcall_mem_desc_fb(fb, 0x200); + ch.userd_mem = gsp_phys_rmcall_mem_desc(GSP_PHYS_RMCALL_BUF_USERD); + ch.mthdbuf_mem = gsp_phys_rmcall_mem_desc(GSP_PHYS_RMCALL_BUF_MTHDBUF); + ch.error_notifier_mem.base = cpu_to_le64( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_user_lane_notifier, 0)); + ch.error_notifier_mem.size = + cpu_to_le64(GSP_PHYS_RMCALL_NOTIFIER_SIZE); + ch.error_notifier_mem.address_space = + cpu_to_le32(GSP_PHYS_RMCALL_ADDR_SYSMEM); + ch.error_notifier_mem.cache_attrib = + cpu_to_le32(GSP_PHYS_RMCALL_CACHE_UNCACHED); + + ret = gsp_phys_rmcall_alloc(bar0, boot_stage.rmcall_client, + boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + GSP_PHYS_RMCALL_CLASS_CHANNEL, &ch, sizeof(ch), + false, NULL, + "mode 35 V7 real-client user lane channel", true, + &rpc, &status); + pr_info("vgsp-boot: rm-call: mode 35 V7 channel alloc: hObject=0x%x hParent=hTsg=0x%x flags=0x%x internalFlags=0x%x engineType=0 hContextShare=0x%x hVASpace=0 gpFifoOffset=0x%llx gpFifoEntries=%u hUserdMemory[0]=0 userdMem=sysmem instance=0x%llx/0x1000 FBMEM ramfc=0x%llx/0x200 FBMEM hObjectError=0x%x errorNotifier=sysmem status=0x%x rpc_result=0x%x ret=%d (deviations: entries=%u vs captured 1024; sysmem USERD vs captured hUserdMemory object)\n", + GSP_PHYS_RMCALL_HANDLE_USER_LANE, boot_stage.rmcall_tsg, + GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE, + GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE, + le32_to_cpu(ch.h_context_share), + le64_to_cpu(ch.gp_fifo_offset), + le32_to_cpu(ch.gp_fifo_entries), fb, fb, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_NOTIFIER, + status, rpc, ret, GSP_PHYS_RMCALL_GPFIFO_ENTRIES); + return ret; +} + +/* + * Mode 35 step 5: the lane promote. ONE atomic GPU_PROMOTE_CTX (0x2080012b) + * on V7 with exactly 3 entries - MAIN, PM, PATCH in the captured bufferId + * order - all PA-only (va=0), physAttr 4 (VIDMEM), bInitialize=1, + * bNonmapped=1, matching the capture's #827. The PM buffer is a fresh + * server FBMEM allocation sized ALIGN(0xd100, 4 KB) while the promote entry + * carries the captured raw 0xd100. Sent BEFORE any 0xc5c0 construct on V7. + */ +static int gsp_phys_rmcall_user_lane_promote(void __iomem *bar0) +{ + struct gsp_phys_rmcall_promote_params *p; + u32 rpc_result = 0, status = 0; + u32 alloc_size; + unsigned int i; + int ret; + + if (!boot_stage.rmcall_ctx_fbmem || !boot_stage.rmcall_golden_fb_aux || + !boot_stage.rmcall_patch_pa) { + pr_err("vgsp-boot: rm-call: mode 35 lane promote aborted: the MAIN/PATCH buffers are not server FBMEM (ctx_fbmem=%u fb_aux=%u patch=0x%llx); no promote is sent and the ladder aborts with no retry\n", + boot_stage.rmcall_ctx_fbmem, + boot_stage.rmcall_golden_fb_aux, + boot_stage.rmcall_patch_pa); + return -EINVAL; + } + if (!boot_stage.rmcall_pm_prepared) { + u64 fb = 0; + + alloc_size = ALIGN(GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE, + GSP_PHYS_PAGE_SIZE); + ret = gsp_phys_rmcall_alloc_fb_mem(bar0, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_PM_FB, + alloc_size, &fb, "mode 35 PM ctxsw"); + if (ret) + return ret; + boot_stage.rmcall_pm_pa = fb; + boot_stage.rmcall_pm_size = + GSP_PHYS_RMCALL_USER_LANE_PM_RAW_SIZE; + boot_stage.rmcall_pm_prepared = true; + pr_info("vgsp-boot: rm-call: mode 35 PM ctxsw FBMEM: base=0x%llx allocation=0x%x (page-rounded) promote size=0x%x (captured raw)\n", + fb, alloc_size, boot_stage.rmcall_pm_size); + } + + if (!boot_stage.rmcall_gr_promote_warned) { + boot_stage.rmcall_gr_promote_warned = true; + pr_warn("vgsp-boot: rm-call: WARNING: mode 35 sends GPU_PROMOTE_CTX on V7 (one atomic 3-entry MAIN/PM/PATCH PA-only): a failed GR context promote can assert GSP-RM (NOCAT recType=5 errorCode=0x52a8b04) and wedge the device with WPR2 left up and teardown unable to clear it; recovery requires a reboot with human approval - run only when reboot readiness is confirmed\n"); + } + + p = kvzalloc(sizeof(*p), GFP_KERNEL); + if (!p) + return -ENOMEM; + p->engine_type = cpu_to_le32(GSP_PHYS_RMCALL_ENGINE_GR0); + p->h_client = cpu_to_le32(0); + p->ch_id = cpu_to_le32(0); + p->h_chan_client = cpu_to_le32(boot_stage.rmcall_client); + p->h_object = cpu_to_le32(GSP_PHYS_RMCALL_HANDLE_USER_LANE); + p->h_virt_memory = cpu_to_le32(0); + p->virt_address = cpu_to_le64(0); + p->size = cpu_to_le64(0); + + gsp_phys_rmcall_promote_entry_fill(p, 0, + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_MAIN, + boot_stage.rmcall_ctx_pa, 0, boot_stage.rmcall_ctx_size, + GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1); + gsp_phys_rmcall_promote_entry_fill(p, 1, + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PM, + boot_stage.rmcall_pm_pa, 0, boot_stage.rmcall_pm_size, + GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1); + gsp_phys_rmcall_promote_entry_fill(p, 2, + GSP_PHYS_RMCALL_GR_CTX_BUFFER_ID_PATCH, + boot_stage.rmcall_patch_pa, 0, boot_stage.rmcall_patch_size, + GSP_PHYS_RMCALL_GR_CTX_PHYS_ATTR_FB, 1, 1); + p->entry_count = cpu_to_le32(3); + + for (i = 0; i < 3; i++) + pr_info("vgsp-boot: rm-call: mode 35 lane promote entry[%u]: bufferId=%u pa=0x%llx va=0 size=0x%llx physAttr=0x%x bInitialize=%u bNonmapped=%u\n", + i, le16_to_cpu(p->entry[i].buffer_id), + le64_to_cpu(p->entry[i].gpu_phys_addr), + le64_to_cpu(p->entry[i].size), + le32_to_cpu(p->entry[i].phys_attr), + p->entry[i].b_initialize, p->entry[i].b_nonmapped); + + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_subdevice, + GSP_PHYS_RMCALL_CTRL_GPU_PROMOTE_CTX, + p, sizeof(*p), 24, NULL, 0, &rpc_result, &status, + "GPU_PROMOTE_CTX (mode 35 lane, atomic 3-entry)"); + pr_info("vgsp-boot: rm-call: PROMOTE_CTX mode 35 lane: entryCount=3 status=0x%x rpc_result=0x%x (hObject=V7 0x%x hChanClient=0x%x engineType=0x%x MAIN pa=0x%llx size=0x%x PM pa=0x%llx size=0x%x PATCH pa=0x%llx size=0x%x)\n", + status, rpc_result, GSP_PHYS_RMCALL_HANDLE_USER_LANE, + boot_stage.rmcall_client, GSP_PHYS_RMCALL_ENGINE_GR0, + boot_stage.rmcall_ctx_pa, boot_stage.rmcall_ctx_size, + boot_stage.rmcall_pm_pa, boot_stage.rmcall_pm_size, + boot_stage.rmcall_patch_pa, boot_stage.rmcall_patch_size); + if (ret) { + pr_err("vgsp-boot: rm-call: PROMOTE_CTX mode 35 lane failed (status=0x%x rpc_result=0x%x, %s, ret=%d); the V7 context is not committed, no 0xc5c0 is constructed and the ladder aborts with no retry\n", + status, rpc_result, gsp_phys_rmcall_status_str(status), + ret); + kvfree(p); + return ret; + } + boot_stage.rmcall_gr_promote_step = 1; + kvfree(p); + return 0; +} + static int gsp_phys_rmcall_stage3(void __iomem *bar0) { struct gsp_phys_rmcall_vas_params vas; @@ -12378,6 +12934,22 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0) boot_stage.rmcall_tsg = GSP_PHYS_RMCALL_HANDLE_TSG; /* + * Mode 35 step 2 (capture order): promote the two TSG fault-method + * buffers (0xa06c010a) on the just-created TSG, before the share and + * the channel batch. A rejection aborts stage 3 with no retry. + */ + if (gsp_phys_rmcall_engine_is_gr() && + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { + ret = gsp_phys_rmcall_tsg_fault_buffers(bar0); + if (ret) { + pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 TSG fault-method-buffer promote failed (%d); the lane does not start and the ladder aborts with no retry\n", + ret); + return ret; + } + } + + /* * Mode 34: allocate FERMI_CONTEXT_SHARE_A before the golden replay, so * the replay's golden channel A can attach to it (hContextShare) and * the ladder's V6 channel can attach to the same share later. The share @@ -12403,6 +12975,29 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0) } /* + * Mode 35 step 3: the same FERMI_CONTEXT_SHARE_A helper for the + * real-client user lane - one share per TSG, VEID 0, ladder + * SHARED_MANAGEMENT VAS - allocated before the golden replay so V7 + * and V6 attach to it (mode 34's ordering; the capture allocates it + * after the fault buffers and before the channel, which the two + * ladder steps preserve). + */ + if (gsp_phys_rmcall_engine_is_gr() && + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { + if (!gsp_phys_rmcall_vas_shared_value()) { + pr_err("vgsp-boot: rm-call: mode 35 needs the ladder SHARED_MANAGEMENT VAS (gsp_phys_rmcall_vas_shared=1): the share and the V7 GPFIFO VA ride the ladder VAS page tables; aborting with no retry\n"); + return -EINVAL; + } + ret = gsp_phys_rmcall_alloc_ctxshare(bar0); + if (ret) { + pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 FERMI_CONTEXT_SHARE_A alloc failed (%d); V7 has no share and the ladder aborts with no retry\n", + ret); + return ret; + } + } + + /* * Golden-image replay (gsp_phys_rmcall_golden=1 + gsp_phys_gpu=1): * replay the stock client's post-load golden channel allocations after * the ladder's VAS/TSG exist and before the ladder's own channel/GR @@ -12429,7 +13024,9 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0) boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_SIBLING_EXEC || boot_stage.rmcall_gr_promote == - GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE) && + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_CTXSHARE || + boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) && gsp_phys_rmcall_engine_is_gr()) { pr_err("vgsp-boot: rm-call: stage 3 aborted: mode-%u golden replay failed (%d); the golden construct is atomic and is not retried\n", boot_stage.rmcall_gr_promote, ret); @@ -12567,6 +13164,63 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0) GSP_PHYS_RMCALL_HANDLE_GOLDEN_GR_OBJ, GSP_PHYS_RMCALL_HANDLE_GOLDEN_CTXSHARE_GR_OBJ, boot_stage.rmcall_gr_object); + } else if (boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE) { + /* + * Mode 35: build the real-client user lane on V7 after + * V6's alloc and before the promote suppression below: + * allocate V7 (captured #825 shape), run the one + * atomic 3-entry MAIN/PM/PATCH PA-only lane promote + * (#827) BEFORE any construct, construct the + * TURING_COMPUTE_A (0xc5c0) on V7 (#829), and swap the + * execution channel to V7 so stage 4 BIND/SCHEDULE/ + * token/GR-sem and every forwarded launch run on it. + * Any failure aborts stage 3 with no retry. + */ + u32 ul_rpc = 0, ul_status = 0; + + if (!gsp_phys_rmcall_golden_enabled() || + !gsp_phys_gpu_enabled()) { + pr_err("vgsp-boot: rm-call: mode 35 requires gsp_phys_rmcall_golden=1 and gsp_phys_gpu=1 (the golden replay did not run); the user lane is not built and the ladder aborts with no retry\n"); + return -EINVAL; + } + ret = gsp_phys_rmcall_alloc_user_lane_channel(bar0); + if (ret) { + pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 V7 channel alloc failed (%d); the user lane is not built and the ladder aborts with no retry\n", + ret); + return ret; + } + ret = gsp_phys_rmcall_user_lane_promote(bar0); + if (ret) { + pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 lane promote failed (%d); the V7 context is not committed, no 0xc5c0 is constructed and the ladder aborts with no retry\n", + ret); + return ret; + } + ret = gsp_phys_rmcall_alloc_gr_object_class(bar0, + GSP_PHYS_RMCALL_CLASS_COMPUTE, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ, + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + "mode 35 TURING_COMPUTE_A on V7 (after promote)", + &ul_rpc, &ul_status); + if (ret) { + pr_err("vgsp-boot: rm-call: stage 3 aborted: mode 35 TURING_COMPUTE_A on V7 failed (status=0x%x rpc_result=0x%x ret=%d); the V7 context has no class object and the ladder aborts with no retry\n", + ul_status, ul_rpc, ret); + return ret; + } + boot_stage.rmcall_gr_object = + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ; + pr_info("vgsp-boot: rm-call: mode 35: execution channel switched from V6 0x%x to V7 0x%x (captured flags=0x%x internalFlags=0x%x hContextShare=0x%x hVASpace=0, gpFifoVa=0x%llx entries=%u, sysmem USERD, FBMEM instance/ramfc, TURING_COMPUTE_A 0x%x); V6 stays allocated and unscheduled, stage 4 + forwarding run on V7\n", + boot_stage.rmcall_channel, + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + GSP_PHYS_RMCALL_CHANNEL_FLAG_USER_LANE, + GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_USER_LANE, + boot_stage.rmcall_ctxshare, + GSP_PHYS_RMCALL_VA_GPFIFO, + GSP_PHYS_RMCALL_GPFIFO_ENTRIES, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ); + boot_stage.rmcall_channel = + GSP_PHYS_RMCALL_HANDLE_USER_LANE; + boot_stage.rmcall_channel_server_mem = false; } else { ret = gsp_phys_rmcall_alloc_gr_object(bar0); if (ret) { @@ -12834,9 +13488,16 @@ struct gsp_phys_rmcall_ce_submit { * offset 0: m14 proved the server applies the sysmem userd_mem base * directly and ignores the captured USERD_INDEX_VALUE, and the PBDMA * consumed the entry at +0x0 (GP_GET advanced) while +0x600 stayed dead. + * Mode 35's V7 carries the captured flags 0x00201320 (USERD_INDEX_VALUE=3) + * but the proven sysmem USERD chunk with hUserdMemory[0]=0, so the host + * control block is written at offset 0 like the other sysmem-USERD shapes. */ static u32 gsp_phys_rmcall_userd_ctl_off(void) { + if (boot_stage.rmcall_gr_promote == + GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_USER_LANE && + gsp_phys_rmcall_engine_is_gr()) + return 0; if ((boot_stage.rmcall_gr_promote == GSP_PHYS_RMCALL_GR_PROMOTE_GOLDEN_EXEC || boot_stage.rmcall_gr_promote == @@ -16082,6 +16743,9 @@ static void gsp_phys_rmcall_fini(struct pci_dev *pdev) gsp_phys_chunk_free(pdev, &boot_stage.rmcall_fecs); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_priv); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_unrestricted); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_user_lane_fault[0]); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_user_lane_fault[1]); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_user_lane_notifier); gsp_phys_rmcall_reset(); } @@ -17792,6 +18456,18 @@ static_assert(sizeof(struct gsp_phys_rmcall_ctx_buffer_info_params) == 5136, static_assert(offsetof(struct gsp_phys_rmcall_ctx_buffer_info_params, ctx_buffer_info) == 16, "NV2080_CTRL_GR_GET_CTX_BUFFER_INFO_PARAMS.ctxBufferInfo must be at offset 16 (8-byte aligned)"); +static_assert(sizeof(struct gsp_phys_rmcall_fault_memdesc) == 32, + "NV2080_CTRL_INTERNAL_MEMDESC_INFO must be 32 bytes"); +static_assert(offsetof(struct gsp_phys_rmcall_fault_memdesc, alignment) == 16, + "NV2080_CTRL_INTERNAL_MEMDESC_INFO.alignment must be at offset 16"); +static_assert(offsetof(struct gsp_phys_rmcall_fault_memdesc, address_space) == 24, + "NV2080_CTRL_INTERNAL_MEMDESC_INFO.addressSpace must be at offset 24"); +static_assert(sizeof(struct gsp_phys_rmcall_fault_buffers_params) == 88, + "NVA06C_CTRL_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS_PARAMS must be 88 bytes for two runqueues"); +static_assert(offsetof(struct gsp_phys_rmcall_fault_buffers_params, bar2_addr) == 64, + "PROMOTE_FAULT_METHOD_BUFFERS.bar2Addr must be at offset 64"); +static_assert(offsetof(struct gsp_phys_rmcall_fault_buffers_params, num_valid_entries) == 80, + "PROMOTE_FAULT_METHOD_BUFFERS.numValidEntries must be at offset 80"); static_assert(sizeof(struct gsp_phys_rmcall_channel_params) == 368, "NV_CHANNEL_ALLOC_PARAMS must be 368 bytes (reference ABI)"); static_assert(offsetof(struct gsp_phys_rmcall_channel_params, gp_fifo_offset) == 8, diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c index 339aa06..e2fffc9 100644 --- a/kernel/gsp_phys_gpu.c +++ b/kernel/gsp_phys_gpu.c @@ -551,8 +551,8 @@ static int gsp_phys_rmcall_gr_promote_set(const char *val, return ret; if (gsp_phys_rmcall_gr_promote_param < 0 || - gsp_phys_rmcall_gr_promote_param > 34) { - pr_warn("vgsp: gsp_phys_rmcall_gr_promote=%d is outside 0..34; clamping to 0 (no PROMOTE_CTX)\n", + gsp_phys_rmcall_gr_promote_param > 35) { + pr_warn("vgsp: gsp_phys_rmcall_gr_promote=%d is outside 0..35; clamping to 0 (no PROMOTE_CTX)\n", gsp_phys_rmcall_gr_promote_param); gsp_phys_rmcall_gr_promote_param = 0; } @@ -571,7 +571,7 @@ module_param_cb(gsp_phys_rmcall_gr_promote, &gsp_phys_rmcall_gr_promote_ops, &gsp_phys_rmcall_gr_promote_param, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_promote, - "GR context-promote bisection (default 0: never send PROMOTE_CTX; 1: single entry init+nonmapped=0 size=raw+subctx, the p6h1b shape; 2: two-phase PA-only nonmapped then VA-only with the same fields; 3: single entry size=raw; 4: two-phase with a zero-field VA-only step; 5: batched entryCount=2 PA-only init + zero-field VA-only; 6: no-entry hVirtMemory, NV01_MEMORY_VIRTUAL (class 0x70) on the VAS at the context window + PROMOTE_CTX entryCount=0; 7: no-entry bare virtAddress+size, entryCount=0; 8: single MAIN entry like 1 on the KMD-managed channel (V6 SKIP_CTXBUFFER_ALLOC), server creates the virtual context memdesc itself; 9: same V6 channel with the stock client's zero-field VA-only MAIN entry, bInitialize=0; 10: same V6 channel, GPU_INITIALIZE_CTX (0x2080012d) installs the physical context memdesc then PROMOTE_CTX entryCount=0 bare virtAddress+size; 11: mode 10 with a server FBMEM context buffer, VIDMEM physAttr 0x4; 12: V5 UVM/vGPU-faithful PA-only MAIN entry (bInitialize=1 bNonmapped=1, no VA) and continue to BIND/SCHEDULE/submit; 13: mode 12 plus a PATCH PA-only promote first, sized from GET_ENGINE_CONTEXT_PROPERTIES engineId 0x10; 14: KMD channel (V6) + PATCH PA-only then the single-entry MAIN init+VA (the flow that books the context VA and commits ENG_CTX_VALID); 15: mode 14 with a server FBMEM MAIN context buffer and VIDMEM physAttr 0x4; 16: mode 14 plus a PM ctxsw PA-only promote first; 17: no client PROMOTE_CTX, paired with gsp_phys_srv_ctxbuf=1 so the GSP-RM server-manages and commits the context; 18: mode 8 shape plus a client FERMI_CONTEXT_SHARE_A on the TSG (hContextShare) whose construct allocates the GraphicsSubcontext; 19: the stock guest's exact 9-entry promote set (MAIN 2 MB PA+VA, PATCH, global buffers, FECS event, private access maps) on V5; 20: mode 6's no-entry hVirtMemory promote on the KMD-managed V6 channel (SKIP_CTXBUFFER_ALLOC), untested - the combination the V5 refusal (0x1f) points at; 21: mode 7's no-entry bare (virtAddress, size) promote on the V6 KMD-managed channel; 22: GPU_INITIALIZE_CTX then mode 20's no-entry hVirtMemory promote; 23: the hVirtMemory no-entry promote backed by the server-owned FBMEM context object (GSP_RM_ALLOC class 0x40) on the server-managed shared V5 channel - a physical memdesc reaches the per-buffer commit without the ADDR_VIRTUAL vaList gate; 24: the reference FWCLIENT non-3D set {MAIN, PM, PATCH, FECS_EVENT, PRIV_ACCESS_MAP, UNRESTRICTED_PRIV_ACCESS_MAP} as one control on the V6 KMD-managed channel, combined PA+VA entries where the ladder has a VA (MAIN PDB-mapped; PATCH/FECS/UNRESTRICTED booking-only), PM and PRIV_ACCESS_MAP init-only (no VA source), unsourceable buffers omitted and logged); 25: the real golden-client construct on the golden replay handles (captured channel flags 0x00200320 + single atomic 9-entry PROMOTE_CTX on the golden channel), then the V6 FWCLIENT follow-on promote on the golden-initialized GR-instance state; requires gsp_phys_rmcall_golden=1 and gsp_phys_rmcall_engine=1; 26/27: golden9 promote on the V6 execution channel (dead ends); 28: the golden execution lane - the golden channel gets a client GPFIFO ring (VA_GPFIFO), a sysmem host-writable USERD and the ladder TSG as parent, keeps the 9-entry promote, and the ladder switches execution to it (V6 stays unscheduled); 29: mode 28 minus the TURING_COMPUTE_A (0xc5c0) object construct on the client-wired golden channel (every GR object construct there deadlocks RM), i.e. the construct-last golden execution lane; 30: the TSG-sibling split: a clean unwired golden sibling A (captured shape, TSG-parented) runs the 9-entry promote and its own TURING_A first-init while V6 stays the execution channel (no swap) with its FWCLIENT promote suppressed; 31: mode-25's device-parented golden first-init (9-entry promote + TURING_A) followed by a clean TSG-parented sibling A whose 9-entry promote runs after V6's own constructs, so the TSG-wide server commit sets V6's instance pointer + RAMFC while V6 stays the execution channel and its own promote is suppressed; 33: mode 31 with the TSG sibling A wired as the object-carrying execution channel (mode-29 golden_exec_shape wiring: ring at VA_GPFIFO, 8 GPFIFO entries, ladder sysmem USERD, TSG parent, host USERD control offset 0 via shape bit7), A's 9-entry promote (skip_fecs), a TURING_COMPUTE_A (0xc5c0) object constructed on A after the accepted promote, and boot_stage.rmcall_channel swapped to A for stage 4 and all forwarded launches; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_golden_exec_shape=0xE7; 34: the context-share golden lane - a FERMI_CONTEXT_SHARE_A (0x9067, TSG-parented, ladder SHARED_MANAGEMENT VAS, flags SYNC -> VEID 0) is allocated before the replay; the golden replay's channel A (0xcaf00054) joins the ladder TSG through hContextShare with hVASpace=0 while keeping the captured unwired shape, runs the 9-entry promote and carries TURING_A (0xc597) + TURING_COMPUTE_A (0xc5c0, handle 0xcaf00090); V6 (0xcaf00034) attaches to the same share, constructs nothing (a later construct would overwrite the shared VEID's subcontext header) and its promote is suppressed, so stage 4 BIND/SCHEDULE/token/GR-sem and every forwarded launch run on V6 inside A's shared VEID/subcontext; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_vas_shared=1; values outside 0..34 clamp to 0 with a warning"); + "GR context-promote bisection (default 0: never send PROMOTE_CTX; 1: single entry init+nonmapped=0 size=raw+subctx, the p6h1b shape; 2: two-phase PA-only nonmapped then VA-only with the same fields; 3: single entry size=raw; 4: two-phase with a zero-field VA-only step; 5: batched entryCount=2 PA-only init + zero-field VA-only; 6: no-entry hVirtMemory, NV01_MEMORY_VIRTUAL (class 0x70) on the VAS at the context window + PROMOTE_CTX entryCount=0; 7: no-entry bare virtAddress+size, entryCount=0; 8: single MAIN entry like 1 on the KMD-managed channel (V6 SKIP_CTXBUFFER_ALLOC), server creates the virtual context memdesc itself; 9: same V6 channel with the stock client's zero-field VA-only MAIN entry, bInitialize=0; 10: same V6 channel, GPU_INITIALIZE_CTX (0x2080012d) installs the physical context memdesc then PROMOTE_CTX entryCount=0 bare virtAddress+size; 11: mode 10 with a server FBMEM context buffer, VIDMEM physAttr 0x4; 12: V5 UVM/vGPU-faithful PA-only MAIN entry (bInitialize=1 bNonmapped=1, no VA) and continue to BIND/SCHEDULE/submit; 13: mode 12 plus a PATCH PA-only promote first, sized from GET_ENGINE_CONTEXT_PROPERTIES engineId 0x10; 14: KMD channel (V6) + PATCH PA-only then the single-entry MAIN init+VA (the flow that books the context VA and commits ENG_CTX_VALID); 15: mode 14 with a server FBMEM MAIN context buffer and VIDMEM physAttr 0x4; 16: mode 14 plus a PM ctxsw PA-only promote first; 17: no client PROMOTE_CTX, paired with gsp_phys_srv_ctxbuf=1 so the GSP-RM server-manages and commits the context; 18: mode 8 shape plus a client FERMI_CONTEXT_SHARE_A on the TSG (hContextShare) whose construct allocates the GraphicsSubcontext; 19: the stock guest's exact 9-entry promote set (MAIN 2 MB PA+VA, PATCH, global buffers, FECS event, private access maps) on V5; 20: mode 6's no-entry hVirtMemory promote on the KMD-managed V6 channel (SKIP_CTXBUFFER_ALLOC), untested - the combination the V5 refusal (0x1f) points at; 21: mode 7's no-entry bare (virtAddress, size) promote on the V6 KMD-managed channel; 22: GPU_INITIALIZE_CTX then mode 20's no-entry hVirtMemory promote; 23: the hVirtMemory no-entry promote backed by the server-owned FBMEM context object (GSP_RM_ALLOC class 0x40) on the server-managed shared V5 channel - a physical memdesc reaches the per-buffer commit without the ADDR_VIRTUAL vaList gate; 24: the reference FWCLIENT non-3D set {MAIN, PM, PATCH, FECS_EVENT, PRIV_ACCESS_MAP, UNRESTRICTED_PRIV_ACCESS_MAP} as one control on the V6 KMD-managed channel, combined PA+VA entries where the ladder has a VA (MAIN PDB-mapped; PATCH/FECS/UNRESTRICTED booking-only), PM and PRIV_ACCESS_MAP init-only (no VA source), unsourceable buffers omitted and logged); 25: the real golden-client construct on the golden replay handles (captured channel flags 0x00200320 + single atomic 9-entry PROMOTE_CTX on the golden channel), then the V6 FWCLIENT follow-on promote on the golden-initialized GR-instance state; requires gsp_phys_rmcall_golden=1 and gsp_phys_rmcall_engine=1; 26/27: golden9 promote on the V6 execution channel (dead ends); 28: the golden execution lane - the golden channel gets a client GPFIFO ring (VA_GPFIFO), a sysmem host-writable USERD and the ladder TSG as parent, keeps the 9-entry promote, and the ladder switches execution to it (V6 stays unscheduled); 29: mode 28 minus the TURING_COMPUTE_A (0xc5c0) object construct on the client-wired golden channel (every GR object construct there deadlocks RM), i.e. the construct-last golden execution lane; 30: the TSG-sibling split: a clean unwired golden sibling A (captured shape, TSG-parented) runs the 9-entry promote and its own TURING_A first-init while V6 stays the execution channel (no swap) with its FWCLIENT promote suppressed; 31: mode-25's device-parented golden first-init (9-entry promote + TURING_A) followed by a clean TSG-parented sibling A whose 9-entry promote runs after V6's own constructs, so the TSG-wide server commit sets V6's instance pointer + RAMFC while V6 stays the execution channel and its own promote is suppressed; 33: mode 31 with the TSG sibling A wired as the object-carrying execution channel (mode-29 golden_exec_shape wiring: ring at VA_GPFIFO, 8 GPFIFO entries, ladder sysmem USERD, TSG parent, host USERD control offset 0 via shape bit7), A's 9-entry promote (skip_fecs), a TURING_COMPUTE_A (0xc5c0) object constructed on A after the accepted promote, and boot_stage.rmcall_channel swapped to A for stage 4 and all forwarded launches; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_golden_exec_shape=0xE7; 34: the context-share golden lane - a FERMI_CONTEXT_SHARE_A (0x9067, TSG-parented, ladder SHARED_MANAGEMENT VAS, flags SYNC -> VEID 0) is allocated before the replay; the golden replay's channel A (0xcaf00054) joins the ladder TSG through hContextShare with hVASpace=0 while keeping the captured unwired shape, runs the 9-entry promote and carries TURING_A (0xc597) + TURING_COMPUTE_A (0xc5c0, handle 0xcaf00090); V6 (0xcaf00034) attaches to the same share, constructs nothing (a later construct would overwrite the shared VEID's subcontext header) and its promote is suppressed, so stage 4 BIND/SCHEDULE/token/GR-sem and every forwarded launch run on V6 inside A's shared VEID/subcontext; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_vas_shared=1; values outside 0..34 clamp to 0 with a warning; 35: the real-client user lane - after the golden replay (mode-25/31 shape with the capture-refreshed FECS_EVENT/PRIV_ACCESS_MAP entries) the ladder promotes the two TSG fault-method buffers (0xa06c010a), allocates the mode-34 FERMI_CONTEXT_SHARE_A, builds V7 (0xcaf000a0: class 0xc46f, TSG parent, hVASpace=0, hContextShare, captured flags 0x00201320, internalFlags 0x1d, hObjectError/errorNotifierMem sysmem, FBMEM instance/ramfc, gpFifoEntries 8, sysmem USERD), sends the one atomic 3-entry MAIN/PM/PATCH PA-only nonmapped promote on V7 before any construct, constructs TURING_COMPUTE_A (0xc5c0) on V7 and swaps stage 4 + forwarding to V7; requires gsp_phys_rmcall_golden=1, gsp_phys_gpu=1 and gsp_phys_rmcall_vas_shared=1; values outside 0..35 clamp to 0 with a warning"); int gsp_phys_rmcall_gr_promote_value(void) { |
