summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* sema-svc: run CE work before its release, grace unreadable CE entries, ↵Arthur Hanson Rasmusson2026-09-30
| | | | | | | | | | | | | | | | | | | payload-pattern audit - CE LAUNCH_DMA releases are now applied after the push's collected work runs: the client reads copied data as soon as it observes the release, so release-first let cuMemcpyDtoH observe a copy that had not executed. - Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms, default 8s) before the release fallback consumes them, so a translation that lands within a service pass lets the push be decoded and its work executed instead of being dropped. - sema_copy_pat matches an 8-byte value against every copy's src/dst first word and scans decoded pushes for it, naming the submission that carries a known payload regardless of size/encoding. - sema-dma-skip logs transfer-shaped launches that collect no op; op log lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG. - A real release application now records the release VA, so decoded pushes no longer trigger a spurious fake release.
* guest-diag: cdt CDT_WAIT park-at-end (live memory inspection); run_cycle ↵Arthur Hanson Rasmusson2026-09-30
| | | | CDT_WAIT passthrough
* guest-diag: pushbuffer pin survives PTI holes (full-range scan + bounded ↵Arthur Hanson Rasmusson2026-09-30
| | | | retries until complete)
* sema-svc: sema_copy_watch (size-keyed src/dst byte audit for the CE data path)Arthur Hanson Rasmusson2026-09-30
|
* sema-svc: sema_ops_log param (default 8) to audit executed CE operands; ↵Arthur Hanson Rasmusson2026-09-30
| | | | run_cycle OPS_LOG knob
* guest-diag: run_cycle enables semfix pushbuffer chain walk by default (CE ↵Arthur Hanson Rasmusson2026-09-30
| | | | window has no CPU alias; the vsock stream cannot supply it)
* guest-diag: run_cycle SKIP_OPS knob + cdt output on serial; cdt richer ↵Arthur Hanson Rasmusson2026-09-30
| | | | roundtrip diagnostics
* guest-diag: bpz21 - dump the FUN_00337e60 info array from FUN_00550ff0Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz20 - watchpoint on ds+0xe94 to catch the zeroing writeArthur Hanson Rasmusson2026-09-30
|
* guest-diag: push bpz18Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: run_cycle gdb-launch mode + bpz18 early path traceArthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz19 - attach-mode path trace through FUN_0042aec0Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz16 - live-probe chip-info vtable + 0xc48/0xc4c at the failing ↵Arthur Hanson Rasmusson2026-09-30
| | | | call
* guest-diag: bpz14 - dump guest TPC/SM count fields at the failing clock ↵Arthur Hanson Rasmusson2026-09-30
| | | | validation
* guest-diag: bpz13 - dump FUN_003e35f0 clock-arg validation at the failing callArthur Hanson Rasmusson2026-09-30
|
* guest-diag: run_cycle - verify live gdb before declaring arm failureArthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz12 - trace call returns in FUN_00386b90 (failing section ↵Arthur Hanson Rasmusson2026-09-30
| | | | callback)
* guest-diag: bpz11 - trace every call return inside FUN_00391ee0Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz10 - trace every call return inside FUN_003e90c0Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz9 - split the failing ucode image create (FUN_003e99d0)Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: run_cycle - retry gdb arming (transient agent timeouts)Arthur Hanson Rasmusson2026-09-30
|
* guest-diag: bpz8 - localize the ucode-load failure inside FUN_00e21850Arthur Hanson Rasmusson2026-09-30
| | | | | | | Run #7 proved the membar-WAR descriptor lookup succeeds in the guest (remap (7,5)->(7,0)); FUN_00e21850 then returns 1. bpz8 breaks after its two inner calls (FUN_003e99d0 image create, FUN_003dfae0 image load) to name which one fails.
* guest-diag: bpz7 - split membar hook failure into lookup vs ucode loadArthur Hanson Rasmusson2026-09-30
| | | | | | | | Host oracle (real TU104, 570.211): FUN_0035a4d0(0x13,7,5) succeeds because the 3-entry remap table at 0x3dc5b80 maps (7,5)->(7,0); the guest has the same table. bpz7 breaks before/after the lookup and the FUN_00e21850 ucode load in FUN_005d7e70 to see which one fails in the guest. Also fix invd.py's lookup address (0x25a4d0, not 0x35a4d0).
* guest-diag: bpz6/invd - read the live arch/impl and test the WAR-ucode lookupArthur Hanson Rasmusson2026-09-30
| | | | | | | | | Run #5 named the failing hook: idx=3 FUN_005d7e70 ('membar_war_bug2022401') returned 999; its only source is FUN_0035a4d0(0x13, dev+0xc50, dev+0xc54) returning NULL or the ucode load failing. bpz6 reads the live device state arch/impl at the failure and evaluates the lookup in-process, scanning (arch,rev) pairs to show which WAR-ucode descriptors exist; invd.py is the standalone same-probe for a parked cdt.
* guest-diag: bpz5 - name the failing engine-setup hook (slot+0x4580)Arthur Hanson Rasmusson2026-09-30
| | | | | | | Run #4 identified the +0x4580 target as file 0x4d1fa0: it runs four descriptors 0x440f780/7c0/800/840 whose fns are 0x4d9350, 0x4c8a70, 0x4ad980, 0x4d7e70 and returns the first non-zero status. bpz5 breaks on the hook call/return and prints the failing index, target and status.
* guest-diag: bpz4 - identify the vtable slot+0x4580 that returns 999Arthur Hanson Rasmusson2026-09-30
| | | | | | | Run #3 named the failing store: file 0x275cec, the indirect call at 0x275ce6 (slot +0x4580 of the table at [ctx+0x40], called with (ctx, obj+0xb8)) returned 0x3e7. bpz4 prints the runtime target pointer so it maps to the architecture vtable implementation that failed.
* guest-diag: bpz3 - probe every r14d store in the ctx bring-up (85 sites)Arthur Hanson Rasmusson2026-09-30
| | | | | | | Run #2: all post-pump call sites up to 0x2755bf returned 0, but FUN_00374240 returned r14d=0x3e7 (999) 554 ms later. bpz3 arms every mov %eax,%r14d store in FUN_00374240 plus the alternate epilogue and the single direct 999 store, printing the first non-zero status with a backtrace.
* guest-diag: post-pump ctx-init status probes (bpz/bpz2 gdb scripts)Arthur Hanson Rasmusson2026-09-30
| | | | | | | | bpz.py probes the four known rc=999 stores on the cuCtxCreate path plus the scanner force sites; the live run eliminated all of them and showed the engine event pump (FUN_0031a3b0) returning success. bpz2.py probes every post-pump status store in the ctx bring-up (FUN_00374240) and the function's final return, printing the first non-zero status with a backtrace.
* guest-diag: UVM API kretprobe installer (retval probes for the ↵Arthur Hanson Rasmusson2026-09-30
| | | | failure-instant trace)
* sema-svc: route 906f/report-semaphore releases to the exact GR counter slot ↵Arthur Hanson Rasmusson2026-09-30
| | | | (all 20 channel releases now land), queue payloads that arrive before the slot pin
* gsprm: reply-latency logging + param dump cap; guest-diag: CDT_PAUSE, ↵Arthur Hanson Rasmusson2026-09-30
| | | | watcher reaping, wider CE stream window
* guest-diag: one-shot gdb/state samplers for timed post-control capturesArthur Hanson Rasmusson2026-09-29
| | | | | | | gdb_now.sh attaches once at a host-chosen moment (final RM control) and dumps all threads; oneshot_gdb.sh triggers on user-running state; state_log.sh records state/syscall/wchan so kernel busy-waits can be told apart from real userspace spins.
* guest-diag: semfix counter-pin retry backoff + shorter exec timeout (stops ↵Arthur Hanson Rasmusson2026-09-29
| | | | guest-agent starvation)
* guest-diag: single-read counter sampler (kills phantom 4-of-20 view); ↵Arthur Hanson Rasmusson2026-09-29
| | | | vmrestart reaps stale watchers
* guest-diag: enlarge and clear the trace ring at probe installArthur Hanson Rasmusson2026-09-29
| | | | | | The client's spin floods the default ring with uvmt/uvsv events; a one-shot uvbp event was overwritten before the pin scanner read it (push-buffer pin landed ~50 s after the client began pushing). 64 MB ring + clear.
* guest-diag: stream uvbp pushbuffer kptrs for immediate pinningArthur Hanson Rasmusson2026-09-29
| | | | | | | | | | The first CE channel's push-buffer was only pinned ~1 minute after the client's failure: uvbp fired at t+4s but semfix's trace read (guest agent, every 5 loops) picked it up at t+70s, so the tracker wait for value 3 spun for 16 s. The guest scanner now emits '#pb 0x<kptr>' from the uvbp trace lines as soon as they appear; the host mirrors them into the sem stream and semfix resolves/pins the push-buffer through the UVM chain on the next loop (sub-second).
* guest-diag: probe the tracker completion check and its resultArthur Hanson Rasmusson2026-09-29
| | | | | | uvsv/uvsvr capture (sem, value) and the boolean return of uvm_gpu_tracking_semaphore_is_value_completed, so a wait that fails despite an advanced payload is visible as a false return.
* guest-diag: pin the CE push-buffer window tooArthur Hanson Rasmusson2026-09-29
| | | | | | | | | | The failing wait (uvtw) is uvm_tracker_wait_for_entry for value 3 on a CE channel whose tracking sem stayed at payload 1 for 14 s: its pushes' release values 2..3 were never applied because the CE push-buffer pages (0x120000000 + 0x100000*rank) are outside the GR window and were only found by the content sweep, late and often wrong (false=3065). Scan both windows in the guest so the host gets guest-truth translations for the CE pushes as well.
* guest-diag: probe uvm_tracker_wait_for_entry for the exact (channel, value) waitArthur Hanson Rasmusson2026-09-29
| | | | | | The failing wait is uvm_tracker_wait_for_entry inside channel_pool_add (UVM_GPU_REGISTER): capture the tracker entry's channel pointer and value so the host can tell which release value is still missing.
* guest-diag: per-thread kernel stack sampler for the CUDA clientArthur Hanson Rasmusson2026-09-29
| | | | | | The failing wait can live in a libcuda worker thread (strace without -f misses it); sample every /proc/<pid>/task/*/stack + wchan while the client runs.
* guest-diag: non-blocking trace tail + 100ms scan loop + raw counter-page dumpArthur Hanson Rasmusson2026-09-29
| | | | | | | | | | | - the trace tail (subprocess + 256KB read) ran inside the scan loop and blocked it for seconds exactly when the client's GR window appeared; moved to a background thread whose queue the loop drains. - scan interval 250ms -> 100ms. - track_waits sampler also dumps 512 raw bytes at page+0xe00 each sample: the per-slot reads disagreed with the bulk dump within one sample because host writes land between the two, and the raw dump is the only trustworthy view of the slot layout.
* sema-svc: bounded diagnostics for GR counter writes; scanner guarantees ↵Arthur Hanson Rasmusson2026-09-29
| | | | | | | | | | | counter-page delta - rel-apply/publish now log the first 32/48 outcomes (written, skipped with current value, no slot, read failure) so the live counter page can be matched against host writes. - guest pin scanner logs the counter page's pagemap state each loop and adds it to the delta set explicitly, so the page is pinned the moment it is mapped even if the window scan misses it.
* sema-svc: persist GR counter slot per channel; stable slot layout; bulk pin ↵Arthur Hanson Rasmusson2026-09-29
| | | | | | | | | | | | | | | | queue 4k - counter_pa autoreg result is written back to the channel table so later passes cannot re-target a different slot when the live GR count changes during teardown; re-registration of the same channel preserves the slot. - counter slot layout uses the max GR count ever seen in the generation instead of the shrinking live count. - sema_map_add queue 64 -> 4096 entries so a full-window pin burst drains without ENOSPC retries. - pin_gr_stream: one batched sudo pass per flush (was one process per run, ~4 s for 400 runs), retrying only queue-rejected runs. - semfix_uvmt: SEMFIX_SKIP_PB=1 skips the blocking push-buffer chain walk when the vsock window stream already supplies those pages.
* guest-diag: scan 512KB per 2MB region and start window at 0x200000000 (cover ↵Arthur Hanson Rasmusson2026-09-29
| | | | GR GPFIFO rings)
* sema-svc: apply releases before push memsets; poller: bypass read cacheArthur Hanson Rasmusson2026-09-29
|
* guest-diag: stream uvmt sems over vsock (no guest-agent race); dump channel ↵Arthur Hanson Rasmusson2026-09-29
| | | | last release VA; larger push log budget
* sema-svc: never apply swept-page releases with no live channel (guest boot ↵Arthur Hanson Rasmusson2026-09-29
| | | | corruption); tooling guards
* sema-svc: apply decoded releases to registered counter slots; single-QMP ↵Arthur Hanson Rasmusson2026-09-29
| | | | semfix+uvmt watcher
* guest-diag: live tracking-sem poller (payload VA + raw fields)Arthur Hanson Rasmusson2026-09-29
|
* guest-diag: print tracking-sem state (index, queued, payload) on changeArthur Hanson Rasmusson2026-09-29
| | | | | | | The final cuCtxCreate stall is a UVM tracking semaphore whose completed value stays 0; the pool payload is the ground truth. semfix now prints idx/+0x10/+0x28/payload for every tracking sem it sees, whenever a field changes, so the last stall before a failure is visible in its log.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-14 16:17:46 +0000