r507o walk the CE channels before the GR channels: enforce the client's stream order in the pump itself

The streams order their H2D copies before the kernel that consumes them,
but the channels register in handle order (the RM's GR channel early, the
copy-CE channels later) and the pump walked them in that order; a pass that
decoded the GR launch before the CE channel's entries ran let the forward
workqueue execute the launch's copy-in against pre-copy data (r480-sep:
copy-in 115 ms before its stream's H2D soft copies; r481 proved no host
semaphore links the two). The CE copies execute on the same service
thread, so consuming the CE entries first fixes the inversion without any
cross-channel hold. Stable partition of the pass snapshot; relative order
within each class is preserved.

1 files changed, 21 insertions, 1 deletions

diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c
index 4c46bf3..a3783dc 100644
--- a/kernel/gsp_sema_service.c
+++ b/kernel/gsp_sema_service.c
@@ -20825,8 +20825,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu)
 	bool stalled = false;
 
 	spin_lock_irqsave(&svc.chan_lock, flags);
+	/*
+	 * r507o stream-order: walk the CE channels BEFORE the GR channels.
+	 * The client's stream order places its H2D copies before the kernel
+	 * that consumes them, but the channels register in handle order (the
+	 * RM's GR channel early, the copy-ce channels later), and a pass that
+	 * decoded the launch before the CE channel's entries ran let the
+	 * forward workqueue execute the launch's copy-in against pre-copy
+	 * data (r480-sep: copy-in 115 ms before its stream's H2D soft
+	 * copies; r481 proved no host semaphore links the two, so the
+	 * emulation must enforce stream order itself). The CE copies execute
+	 * on this same service thread, so consuming the CE entries first
+	 * fixes the inversion directly - no cross-channel hold, no latency
+	 * tax. Stable partition; relative order within each class preserved.
+	 */
 	for (i = 0; i < SVC_MAX_CHANNELS; i++) {
-		if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu)
+		if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu &&
+		    svc.chan[i].engine_type)
+			local[n++] = svc.chan[i];
+	}
+	for (i = 0; i < SVC_MAX_CHANNELS; i++) {
+		if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu &&
+		    !svc.chan[i].engine_type)
 			local[n++] = svc.chan[i];
 	}
 	spin_unlock_irqrestore(&svc.chan_lock, flags);