summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* guest-diag: reset per-client pin state when a new client pid appearsArthur Hanson Rasmusson12 days
| | | | | | | A new CUDA client allocates fresh channel pools at the same GPU VAs; the watcher kept the previous client's sem kptrs and seen-pin marks, so it pinned the previous (freed) pool pages and the new client's release writes went to dead memory. Clear the learned sems/marks on each pid change.
* guest-diag: pin scanner matches va* clients too (static-cudart vectorAdd)Arthur Hanson Rasmusson12 days
|
* guest-diag: cycle monitor CPUs on kernel-VA translation failureArthur Hanson Rasmusson12 days
| | | | | | | | | HMP x/gva2gpa translate with the selected vCPU's CR3; while that CPU runs user code under PTI the kernel half is hidden, so kernel-VA reads (the tracking-sem chain and the CE pushbuffer CPU alias) fail and the pin pass aborts. Try the other vCPUs before giving up: during an active CUDA client at least one CPU is in kernel mode, so the chain/pin work no longer require the client to be parked in sleep().
* guest-diag: cdt2 mapping scan is opt-in (CDT_SCAN=1); it can wedge the clientArthur Hanson Rasmusson12 days
|
* guest-diag: cdt2 payload scan limited to anon mappings (device-mapping reads ↵Arthur Hanson Rasmusson12 days
| | | | can block)
* guest-diag: cdt2 payload scan only touches anon/device mappings (avoid ↵Arthur Hanson Rasmusson12 days
| | | | SIGBUS on truncated file maps)
* guest-diag: cdt2 scans its own mappings for the payload after HtoD (DtoH ↵Arthur Hanson Rasmusson12 days
| | | | mechanism probe)
* guest-diag: pin scanner matches cdt* so the copy-path test binary is trackedArthur Hanson Rasmusson12 days
| | | | | | | The GR-window pin scanner matched comm=="cdt" exactly, so runs using the cuda_drv_test2 binary (comm "cdt2") never streamed their mappings: the GR push pages stayed unresolvable and the client stalled in context creation. Match the cdt prefix instead.
* sema-svc: decode GRCE inline-data DMA (small HtoD path)Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA submits it on the compute channel as a compute-class DMA whose source is the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>, LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run (opcode 3) and no GR work ran at all. - Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT, OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run. - svc_grce_inline_write resolves the destination VA through trusted sources (pins, map, published page directories, targeted rm_mem resolve), pins the page and writes into vidmem or guest RAM; budgeted sema-grce logs name the target or report UNRESOLVED. - cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection of the copy path; pushed by push_guest_tools.sh.
* sema-svc: run CE work before its release, grace unreadable CE entries, ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | payload-pattern audit - CE LAUNCH_DMA releases are now applied after the push's collected work runs: the client reads copied data as soon as it observes the release, so release-first let cuMemcpyDtoH observe a copy that had not executed. - Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms, default 8s) before the release fallback consumes them, so a translation that lands within a service pass lets the push be decoded and its work executed instead of being dropped. - sema_copy_pat matches an 8-byte value against every copy's src/dst first word and scans decoded pushes for it, naming the submission that carries a known payload regardless of size/encoding. - sema-dma-skip logs transfer-shaped launches that collect no op; op log lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG. - A real release application now records the release VA, so decoded pushes no longer trigger a spurious fake release.
* guest-diag: cdt CDT_WAIT park-at-end (live memory inspection); run_cycle ↵Arthur Hanson Rasmusson12 days
| | | | CDT_WAIT passthrough
* guest-diag: pushbuffer pin survives PTI holes (full-range scan + bounded ↵Arthur Hanson Rasmusson12 days
| | | | retries until complete)
* sema-svc: sema_copy_watch (size-keyed src/dst byte audit for the CE data path)Arthur Hanson Rasmusson12 days
|
* sema-svc: sema_ops_log param (default 8) to audit executed CE operands; ↵Arthur Hanson Rasmusson12 days
| | | | run_cycle OPS_LOG knob
* guest-diag: run_cycle enables semfix pushbuffer chain walk by default (CE ↵Arthur Hanson Rasmusson12 days
| | | | window has no CPU alias; the vsock stream cannot supply it)
* guest-diag: run_cycle SKIP_OPS knob + cdt output on serial; cdt richer ↵Arthur Hanson Rasmusson12 days
| | | | roundtrip diagnostics
* guest-diag: bpz21 - dump the FUN_00337e60 info array from FUN_00550ff0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz20 - watchpoint on ds+0xe94 to catch the zeroing writeArthur Hanson Rasmusson12 days
|
* guest-diag: push bpz18Arthur Hanson Rasmusson12 days
|
* guest-diag: run_cycle gdb-launch mode + bpz18 early path traceArthur Hanson Rasmusson12 days
|
* guest-diag: bpz19 - attach-mode path trace through FUN_0042aec0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz16 - live-probe chip-info vtable + 0xc48/0xc4c at the failing ↵Arthur Hanson Rasmusson12 days
| | | | call
* guest-diag: bpz14 - dump guest TPC/SM count fields at the failing clock ↵Arthur Hanson Rasmusson12 days
| | | | validation
* guest-diag: bpz13 - dump FUN_003e35f0 clock-arg validation at the failing callArthur Hanson Rasmusson12 days
|
* guest-diag: run_cycle - verify live gdb before declaring arm failureArthur Hanson Rasmusson12 days
|
* guest-diag: bpz12 - trace call returns in FUN_00386b90 (failing section ↵Arthur Hanson Rasmusson12 days
| | | | callback)
* guest-diag: bpz11 - trace every call return inside FUN_00391ee0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz10 - trace every call return inside FUN_003e90c0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz9 - split the failing ucode image create (FUN_003e99d0)Arthur Hanson Rasmusson12 days
|
* guest-diag: run_cycle - retry gdb arming (transient agent timeouts)Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz8 - localize the ucode-load failure inside FUN_00e21850Arthur Hanson Rasmusson12 days
| | | | | | | Run #7 proved the membar-WAR descriptor lookup succeeds in the guest (remap (7,5)->(7,0)); FUN_00e21850 then returns 1. bpz8 breaks after its two inner calls (FUN_003e99d0 image create, FUN_003dfae0 image load) to name which one fails.
* guest-diag: bpz7 - split membar hook failure into lookup vs ucode loadArthur Hanson Rasmusson12 days
| | | | | | | | Host oracle (real TU104, 570.211): FUN_0035a4d0(0x13,7,5) succeeds because the 3-entry remap table at 0x3dc5b80 maps (7,5)->(7,0); the guest has the same table. bpz7 breaks before/after the lookup and the FUN_00e21850 ucode load in FUN_005d7e70 to see which one fails in the guest. Also fix invd.py's lookup address (0x25a4d0, not 0x35a4d0).
* guest-diag: bpz6/invd - read the live arch/impl and test the WAR-ucode lookupArthur Hanson Rasmusson12 days
| | | | | | | | | Run #5 named the failing hook: idx=3 FUN_005d7e70 ('membar_war_bug2022401') returned 999; its only source is FUN_0035a4d0(0x13, dev+0xc50, dev+0xc54) returning NULL or the ucode load failing. bpz6 reads the live device state arch/impl at the failure and evaluates the lookup in-process, scanning (arch,rev) pairs to show which WAR-ucode descriptors exist; invd.py is the standalone same-probe for a parked cdt.
* guest-diag: bpz5 - name the failing engine-setup hook (slot+0x4580)Arthur Hanson Rasmusson12 days
| | | | | | | Run #4 identified the +0x4580 target as file 0x4d1fa0: it runs four descriptors 0x440f780/7c0/800/840 whose fns are 0x4d9350, 0x4c8a70, 0x4ad980, 0x4d7e70 and returns the first non-zero status. bpz5 breaks on the hook call/return and prints the failing index, target and status.
* guest-diag: bpz4 - identify the vtable slot+0x4580 that returns 999Arthur Hanson Rasmusson12 days
| | | | | | | Run #3 named the failing store: file 0x275cec, the indirect call at 0x275ce6 (slot +0x4580 of the table at [ctx+0x40], called with (ctx, obj+0xb8)) returned 0x3e7. bpz4 prints the runtime target pointer so it maps to the architecture vtable implementation that failed.
* guest-diag: bpz3 - probe every r14d store in the ctx bring-up (85 sites)Arthur Hanson Rasmusson12 days
| | | | | | | Run #2: all post-pump call sites up to 0x2755bf returned 0, but FUN_00374240 returned r14d=0x3e7 (999) 554 ms later. bpz3 arms every mov %eax,%r14d store in FUN_00374240 plus the alternate epilogue and the single direct 999 store, printing the first non-zero status with a backtrace.
* guest-diag: post-pump ctx-init status probes (bpz/bpz2 gdb scripts)Arthur Hanson Rasmusson12 days
| | | | | | | | bpz.py probes the four known rc=999 stores on the cuCtxCreate path plus the scanner force sites; the live run eliminated all of them and showed the engine event pump (FUN_0031a3b0) returning success. bpz2.py probes every post-pump status store in the ctx bring-up (FUN_00374240) and the function's final return, printing the first non-zero status with a backtrace.
* guest-diag: UVM API kretprobe installer (retval probes for the ↵Arthur Hanson Rasmusson12 days
| | | | failure-instant trace)
* sema-svc: route 906f/report-semaphore releases to the exact GR counter slot ↵Arthur Hanson Rasmusson12 days
| | | | (all 20 channel releases now land), queue payloads that arrive before the slot pin
* gsprm: reply-latency logging + param dump cap; guest-diag: CDT_PAUSE, ↵Arthur Hanson Rasmusson12 days
| | | | watcher reaping, wider CE stream window
* guest-diag: one-shot gdb/state samplers for timed post-control capturesArthur Hanson Rasmusson13 days
| | | | | | | gdb_now.sh attaches once at a host-chosen moment (final RM control) and dumps all threads; oneshot_gdb.sh triggers on user-running state; state_log.sh records state/syscall/wchan so kernel busy-waits can be told apart from real userspace spins.
* guest-diag: semfix counter-pin retry backoff + shorter exec timeout (stops ↵Arthur Hanson Rasmusson13 days
| | | | guest-agent starvation)
* guest-diag: single-read counter sampler (kills phantom 4-of-20 view); ↵Arthur Hanson Rasmusson13 days
| | | | vmrestart reaps stale watchers
* guest-diag: enlarge and clear the trace ring at probe installArthur Hanson Rasmusson13 days
| | | | | | The client's spin floods the default ring with uvmt/uvsv events; a one-shot uvbp event was overwritten before the pin scanner read it (push-buffer pin landed ~50 s after the client began pushing). 64 MB ring + clear.
* guest-diag: stream uvbp pushbuffer kptrs for immediate pinningArthur Hanson Rasmusson13 days
| | | | | | | | | | The first CE channel's push-buffer was only pinned ~1 minute after the client's failure: uvbp fired at t+4s but semfix's trace read (guest agent, every 5 loops) picked it up at t+70s, so the tracker wait for value 3 spun for 16 s. The guest scanner now emits '#pb 0x<kptr>' from the uvbp trace lines as soon as they appear; the host mirrors them into the sem stream and semfix resolves/pins the push-buffer through the UVM chain on the next loop (sub-second).
* guest-diag: probe the tracker completion check and its resultArthur Hanson Rasmusson13 days
| | | | | | uvsv/uvsvr capture (sem, value) and the boolean return of uvm_gpu_tracking_semaphore_is_value_completed, so a wait that fails despite an advanced payload is visible as a false return.
* guest-diag: pin the CE push-buffer window tooArthur Hanson Rasmusson13 days
| | | | | | | | | | The failing wait (uvtw) is uvm_tracker_wait_for_entry for value 3 on a CE channel whose tracking sem stayed at payload 1 for 14 s: its pushes' release values 2..3 were never applied because the CE push-buffer pages (0x120000000 + 0x100000*rank) are outside the GR window and were only found by the content sweep, late and often wrong (false=3065). Scan both windows in the guest so the host gets guest-truth translations for the CE pushes as well.
* guest-diag: probe uvm_tracker_wait_for_entry for the exact (channel, value) waitArthur Hanson Rasmusson13 days
| | | | | | The failing wait is uvm_tracker_wait_for_entry inside channel_pool_add (UVM_GPU_REGISTER): capture the tracker entry's channel pointer and value so the host can tell which release value is still missing.
* guest-diag: per-thread kernel stack sampler for the CUDA clientArthur Hanson Rasmusson13 days
| | | | | | The failing wait can live in a libcuda worker thread (strace without -f misses it); sample every /proc/<pid>/task/*/stack + wchan while the client runs.
* guest-diag: non-blocking trace tail + 100ms scan loop + raw counter-page dumpArthur Hanson Rasmusson13 days
| | | | | | | | | | | - the trace tail (subprocess + 256KB read) ran inside the scan loop and blocked it for seconds exactly when the client's GR window appeared; moved to a background thread whose queue the loop drains. - scan interval 250ms -> 100ms. - track_waits sampler also dumps 512 raw bytes at page+0xe00 each sample: the per-slot reads disagreed with the bulk dump within one sample because host writes land between the two, and the raw dump is the only trustworthy view of the slot layout.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-12 13:48:57 +0000