diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-19 18:28:36 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-19 19:23:21 -0400 |
| commit | 03740ac333b76a79a8fd481c20748f31bb21fdb7 (patch) | |
| tree | 539e61eef0cba0da39ba54dc6db8ad2fb073a07c /scripts/test_cuda_workloads.sh | |
| parent | 4e863b8ed36c5b7bbf64520491f5f1b9d2929684 (diff) | |
| download | open-iov-03740ac333b76a79a8fd481c20748f31bb21fdb7.tar.gz open-iov-03740ac333b76a79a8fd481c20748f31bb21fdb7.zip | |
vgsp: fail4 - denied device-dst copies hold+retry (fix the chunk drop)
Root cause of the vector_add chunk drop (CONT.55 chase): the sw-copy
client's GR-classified GRCE channel (eng=0, h=0x5c000059) decodes its
1MB H2D chunk copies as CE transfers, but ops on that channel run only
once the pin/trust stream (injected) reaches each push page. The first
pass over a chunk push lands before that; with allow_ops=0 the copy was
never built and the entry was consumed silently. All later passes
classify as replays (slot_ops_complete=1), so the entry was never
retried: 6 of 8 chunks skipped, kernel computed on stale VRAM
(vector_add r302-r307: 2-5 of 8 chunk ops ever existed, the rest never
ran; the +2MB 'phase' signature is stale residue).
Fix: a denied launch that WOULD have built a type=1 device-dst copy
(no remap, both offsets, out >= SVC_VAS_FLOOR) with a trusted push now
marks the pass op-pending. op_tr_pending un-completes the slot, so the
next pass is treated fresh and the copy runs as soon as trust (injected
with allow=1) or the operand translation arrives - the existing
m163/opdef hold+retry machinery. Replay passes are deliberately
eligible (that is what makes the retry possible); a per-slot
slot_pend_tries cap (4 per put epoch) prevents a slot whose trust never
arrives from wedging the channel in 10s hold cycles.
Measured (r308, sema_dbg): all 8 consecutive chunk dsts build ops
(0x...cc..d3) and retry to success - 28 CEAUDIT rc=0 soft copies, 38
rc=0 vs 20 rc=-2 executions; previously max 5 dsts and only the last
2-3 ran. Regression checks on r309 with the fix: probe RAWA2+PROBE2B
bad=0, victory_mail OK, victory_matmul OK (one transient first-run
fail, clean re-run), transformer OK. vector_add now advances through
the full H2D set but still does not complete (retry storms stretch the
copy phase; the run later wedges in the GR stale-wedge recovery loop -
named as the next lead).
Diagnostics kept (dbg-gated, bounded): sema-pend set/hold witness,
sema-consume consumed-slot witness, sema-entry de-gated for eng=0
channels, dmatrace extended with replay/eng/pend_would, budgets raised
under sema_dbg so boot init cannot spend the trace before the run.
Diffstat (limited to 'scripts/test_cuda_workloads.sh')
0 files changed, 0 insertions, 0 deletions
