summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* guest-diag: retry the guest pin-scanner startup (a transient agent exec ↵Arthur Hanson Rasmusson12 days
| | | | failure left a boot without pins and the client hung)
* guest-diag: mount tracefs before installing kprobesArthur Hanson Rasmusson12 days
| | | | | | A fresh boot can leave /sys/kernel/tracing as an unmounted mount point; the probe adds then fail silently (probe count 0) and the whole sem/kptr pipeline goes blind.
* guest-diag: reset per-client pin state when a new client pid appearsArthur Hanson Rasmusson12 days
| | | | | | | A new CUDA client allocates fresh channel pools at the same GPU VAs; the watcher kept the previous client's sem kptrs and seen-pin marks, so it pinned the previous (freed) pool pages and the new client's release writes went to dead memory. Clear the learned sems/marks on each pid change.
* guest-diag: pin scanner matches va* clients too (static-cudart vectorAdd)Arthur Hanson Rasmusson12 days
|
* guest-diag: cycle monitor CPUs on kernel-VA translation failureArthur Hanson Rasmusson12 days
| | | | | | | | | HMP x/gva2gpa translate with the selected vCPU's CR3; while that CPU runs user code under PTI the kernel half is hidden, so kernel-VA reads (the tracking-sem chain and the CE pushbuffer CPU alias) fail and the pin pass aborts. Try the other vCPUs before giving up: during an active CUDA client at least one CPU is in kernel mode, so the chain/pin work no longer require the client to be parked in sleep().
* guest-diag: cdt2 mapping scan is opt-in (CDT_SCAN=1); it can wedge the clientArthur Hanson Rasmusson12 days
|
* guest-diag: cdt2 payload scan limited to anon mappings (device-mapping reads ↵Arthur Hanson Rasmusson12 days
| | | | can block)
* guest-diag: cdt2 payload scan only touches anon/device mappings (avoid ↵Arthur Hanson Rasmusson12 days
| | | | SIGBUS on truncated file maps)
* guest-diag: cdt2 scans its own mappings for the payload after HtoD (DtoH ↵Arthur Hanson Rasmusson12 days
| | | | mechanism probe)
* guest-diag: pin scanner matches cdt* so the copy-path test binary is trackedArthur Hanson Rasmusson12 days
| | | | | | | The GR-window pin scanner matched comm=="cdt" exactly, so runs using the cuda_drv_test2 binary (comm "cdt2") never streamed their mappings: the GR push pages stayed unresolvable and the client stalled in context creation. Match the cdt prefix instead.
* sema-svc: decode GRCE inline-data DMA (small HtoD path)Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA submits it on the compute channel as a compute-class DMA whose source is the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>, LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run (opcode 3) and no GR work ran at all. - Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT, OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run. - svc_grce_inline_write resolves the destination VA through trusted sources (pins, map, published page directories, targeted rm_mem resolve), pins the page and writes into vidmem or guest RAM; budgeted sema-grce logs name the target or report UNRESOLVED. - cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection of the copy path; pushed by push_guest_tools.sh.
* sema-svc: run CE work before its release, grace unreadable CE entries, ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | payload-pattern audit - CE LAUNCH_DMA releases are now applied after the push's collected work runs: the client reads copied data as soon as it observes the release, so release-first let cuMemcpyDtoH observe a copy that had not executed. - Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms, default 8s) before the release fallback consumes them, so a translation that lands within a service pass lets the push be decoded and its work executed instead of being dropped. - sema_copy_pat matches an 8-byte value against every copy's src/dst first word and scans decoded pushes for it, naming the submission that carries a known payload regardless of size/encoding. - sema-dma-skip logs transfer-shaped launches that collect no op; op log lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG. - A real release application now records the release VA, so decoded pushes no longer trigger a spurious fake release.
* guest-diag: cdt CDT_WAIT park-at-end (live memory inspection); run_cycle ↵Arthur Hanson Rasmusson12 days
| | | | CDT_WAIT passthrough
* guest-diag: pushbuffer pin survives PTI holes (full-range scan + bounded ↵Arthur Hanson Rasmusson12 days
| | | | retries until complete)
* sema-svc: sema_copy_watch (size-keyed src/dst byte audit for the CE data path)Arthur Hanson Rasmusson12 days
|
* sema-svc: sema_ops_log param (default 8) to audit executed CE operands; ↵Arthur Hanson Rasmusson12 days
| | | | run_cycle OPS_LOG knob
* guest-diag: run_cycle enables semfix pushbuffer chain walk by default (CE ↵Arthur Hanson Rasmusson12 days
| | | | window has no CPU alias; the vsock stream cannot supply it)
* guest-diag: run_cycle SKIP_OPS knob + cdt output on serial; cdt richer ↵Arthur Hanson Rasmusson12 days
| | | | roundtrip diagnostics
* guest-diag: bpz21 - dump the FUN_00337e60 info array from FUN_00550ff0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz20 - watchpoint on ds+0xe94 to catch the zeroing writeArthur Hanson Rasmusson12 days
|
* guest-diag: push bpz18Arthur Hanson Rasmusson12 days
|
* guest-diag: run_cycle gdb-launch mode + bpz18 early path traceArthur Hanson Rasmusson12 days
|
* guest-diag: bpz19 - attach-mode path trace through FUN_0042aec0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz16 - live-probe chip-info vtable + 0xc48/0xc4c at the failing ↵Arthur Hanson Rasmusson12 days
| | | | call
* guest-diag: bpz14 - dump guest TPC/SM count fields at the failing clock ↵Arthur Hanson Rasmusson12 days
| | | | validation
* guest-diag: bpz13 - dump FUN_003e35f0 clock-arg validation at the failing callArthur Hanson Rasmusson12 days
|
* guest-diag: run_cycle - verify live gdb before declaring arm failureArthur Hanson Rasmusson12 days
|
* guest-diag: bpz12 - trace call returns in FUN_00386b90 (failing section ↵Arthur Hanson Rasmusson12 days
| | | | callback)
* guest-diag: bpz11 - trace every call return inside FUN_00391ee0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz10 - trace every call return inside FUN_003e90c0Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz9 - split the failing ucode image create (FUN_003e99d0)Arthur Hanson Rasmusson12 days
|
* guest-diag: run_cycle - retry gdb arming (transient agent timeouts)Arthur Hanson Rasmusson12 days
|
* guest-diag: bpz8 - localize the ucode-load failure inside FUN_00e21850Arthur Hanson Rasmusson12 days
| | | | | | | Run #7 proved the membar-WAR descriptor lookup succeeds in the guest (remap (7,5)->(7,0)); FUN_00e21850 then returns 1. bpz8 breaks after its two inner calls (FUN_003e99d0 image create, FUN_003dfae0 image load) to name which one fails.
* guest-diag: bpz7 - split membar hook failure into lookup vs ucode loadArthur Hanson Rasmusson12 days
| | | | | | | | Host oracle (real TU104, 570.211): FUN_0035a4d0(0x13,7,5) succeeds because the 3-entry remap table at 0x3dc5b80 maps (7,5)->(7,0); the guest has the same table. bpz7 breaks before/after the lookup and the FUN_00e21850 ucode load in FUN_005d7e70 to see which one fails in the guest. Also fix invd.py's lookup address (0x25a4d0, not 0x35a4d0).
* guest-diag: bpz6/invd - read the live arch/impl and test the WAR-ucode lookupArthur Hanson Rasmusson12 days
| | | | | | | | | Run #5 named the failing hook: idx=3 FUN_005d7e70 ('membar_war_bug2022401') returned 999; its only source is FUN_0035a4d0(0x13, dev+0xc50, dev+0xc54) returning NULL or the ucode load failing. bpz6 reads the live device state arch/impl at the failure and evaluates the lookup in-process, scanning (arch,rev) pairs to show which WAR-ucode descriptors exist; invd.py is the standalone same-probe for a parked cdt.
* guest-diag: bpz5 - name the failing engine-setup hook (slot+0x4580)Arthur Hanson Rasmusson12 days
| | | | | | | Run #4 identified the +0x4580 target as file 0x4d1fa0: it runs four descriptors 0x440f780/7c0/800/840 whose fns are 0x4d9350, 0x4c8a70, 0x4ad980, 0x4d7e70 and returns the first non-zero status. bpz5 breaks on the hook call/return and prints the failing index, target and status.
* guest-diag: bpz4 - identify the vtable slot+0x4580 that returns 999Arthur Hanson Rasmusson12 days
| | | | | | | Run #3 named the failing store: file 0x275cec, the indirect call at 0x275ce6 (slot +0x4580 of the table at [ctx+0x40], called with (ctx, obj+0xb8)) returned 0x3e7. bpz4 prints the runtime target pointer so it maps to the architecture vtable implementation that failed.
* guest-diag: bpz3 - probe every r14d store in the ctx bring-up (85 sites)Arthur Hanson Rasmusson12 days
| | | | | | | Run #2: all post-pump call sites up to 0x2755bf returned 0, but FUN_00374240 returned r14d=0x3e7 (999) 554 ms later. bpz3 arms every mov %eax,%r14d store in FUN_00374240 plus the alternate epilogue and the single direct 999 store, printing the first non-zero status with a backtrace.
* guest-diag: post-pump ctx-init status probes (bpz/bpz2 gdb scripts)Arthur Hanson Rasmusson12 days
| | | | | | | | bpz.py probes the four known rc=999 stores on the cuCtxCreate path plus the scanner force sites; the live run eliminated all of them and showed the engine event pump (FUN_0031a3b0) returning success. bpz2.py probes every post-pump status store in the ctx bring-up (FUN_00374240) and the function's final return, printing the first non-zero status with a backtrace.
* guest-diag: UVM API kretprobe installer (retval probes for the ↵Arthur Hanson Rasmusson12 days
| | | | failure-instant trace)
* sema-svc: route 906f/report-semaphore releases to the exact GR counter slot ↵Arthur Hanson Rasmusson12 days
| | | | (all 20 channel releases now land), queue payloads that arrive before the slot pin
* gsprm: reply-latency logging + param dump cap; guest-diag: CDT_PAUSE, ↵Arthur Hanson Rasmusson12 days
| | | | watcher reaping, wider CE stream window
* guest-diag: one-shot gdb/state samplers for timed post-control capturesArthur Hanson Rasmusson12 days
| | | | | | | gdb_now.sh attaches once at a host-chosen moment (final RM control) and dumps all threads; oneshot_gdb.sh triggers on user-running state; state_log.sh records state/syscall/wchan so kernel busy-waits can be told apart from real userspace spins.
* guest-diag: semfix counter-pin retry backoff + shorter exec timeout (stops ↵Arthur Hanson Rasmusson13 days
| | | | guest-agent starvation)
* guest-diag: single-read counter sampler (kills phantom 4-of-20 view); ↵Arthur Hanson Rasmusson13 days
| | | | vmrestart reaps stale watchers
* guest-diag: enlarge and clear the trace ring at probe installArthur Hanson Rasmusson13 days
| | | | | | The client's spin floods the default ring with uvmt/uvsv events; a one-shot uvbp event was overwritten before the pin scanner read it (push-buffer pin landed ~50 s after the client began pushing). 64 MB ring + clear.
* guest-diag: stream uvbp pushbuffer kptrs for immediate pinningArthur Hanson Rasmusson13 days
| | | | | | | | | | The first CE channel's push-buffer was only pinned ~1 minute after the client's failure: uvbp fired at t+4s but semfix's trace read (guest agent, every 5 loops) picked it up at t+70s, so the tracker wait for value 3 spun for 16 s. The guest scanner now emits '#pb 0x<kptr>' from the uvbp trace lines as soon as they appear; the host mirrors them into the sem stream and semfix resolves/pins the push-buffer through the UVM chain on the next loop (sub-second).
* guest-diag: probe the tracker completion check and its resultArthur Hanson Rasmusson13 days
| | | | | | uvsv/uvsvr capture (sem, value) and the boolean return of uvm_gpu_tracking_semaphore_is_value_completed, so a wait that fails despite an advanced payload is visible as a false return.
* guest-diag: pin the CE push-buffer window tooArthur Hanson Rasmusson13 days
| | | | | | | | | | The failing wait (uvtw) is uvm_tracker_wait_for_entry for value 3 on a CE channel whose tracking sem stayed at payload 1 for 14 s: its pushes' release values 2..3 were never applied because the CE push-buffer pages (0x120000000 + 0x100000*rank) are outside the GR window and were only found by the content sweep, late and often wrong (false=3065). Scan both windows in the guest so the host gets guest-truth translations for the CE pushes as well.
* guest-diag: probe uvm_tracker_wait_for_entry for the exact (channel, value) waitArthur Hanson Rasmusson13 days
| | | | | | The failing wait is uvm_tracker_wait_for_entry inside channel_pool_add (UVM_GPU_REGISTER): capture the tracker entry's channel pointer and value so the host can tell which release value is still missing.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-12 10:58:37 +0000