summaryrefslogtreecommitdiffstats
path: root/tools/cuda-ab/run/wedge_recover
Commit message (Collapse)AuthorAge
* gr: r410 wedge quiet window - bounded idle hold before retrying a lane whose ↵Arthur Hanson Rasmusson9 hours
| | | | | | | | | | | | | | | | | | | | | previous launch also no-wrote (r416: 58-launch sticky wedge clears after ~50 s quiet); wedge counters in state; wedge_recover v2 unmask The r416 boot reproduced the sticky no-execute wedge inside 60 s: 58 consecutive launches accepted zero writes while the lane stayed busy (wedge_recover v1's deterministic verify masked every one of them), and the first launches were green again only after the lane sat idle ~52 s. Both r409 (~37 s idle heal) and r416 therefore show the wedge clears on idleness and persists under back-to-back retries (r154/r409), so the retry loop now holds a quiet window (gsp_phys_gr_wedge_quiet_ms, default 20000, doubling on the second quiet retry, 0 disables) before the normal fresh-bank + resched retry when the previous launch also no-wrote. The whole-launch stall budget is relaxed only on this bounded recovery path. NOWRITE lines now carry the consecutive streak and quiet/heal counters, and the state file shows nw_streak/wedge_quiet/wedge_heals. wedge_recover v2 mutates the inputs every iteration so a stale (never executed) launch can no longer match the expected values, and classifies the readback STALE-PREV(NOWRITE) vs OTHER/PARTIAL.
* tools/cuda-ab: memset_probe + wedge_recover forensics clients (session 10)Arthur Hanson Rasmusson10 hours
memset_probe: linear 8 MB cudaMemset + D2H with first-bad offset and a 4 KB page map (correct / other / malloc-prefill) - separates 'fill under-delivered' from 'readback delivered wrong pages'; standalone green, bisects the memcpy_roundtrip memset phase. wedge_recover: launch+verify loop that on failure sleeps N seconds and retries the same iteration (same context), reporting per-iteration attempts and launch times - tests whether an idle gap heals the per-boot no-execute wedge (r409 log shows a ~37 s client-side gap healed a wedged run). Also ships the missing inplace_scale run binary (its source landed in 7685d1b3).