open-iov.git/tools/guest-diag/guest_kprobes.sh, branch master

spdx: add BSD-3-Clause identifiers to userspace/tools/scripts sources

2026-10-08T15:53:20+00:00

Sweep of the 155 tracked C/Python/shell files under userspace/ (excluding the MIT-licensed vendored userspace/extern/), tools/ and scripts/ that had no per-file license identifier. Each gains the minimal two-line header in the area's existing form: 'SPDX-License-Identifier: BSD-3-Clause' plus 'Copyright (c) 2026 Open-IOV contributors'; shebang and coding lines stay first. No other content touched. kernel/** is deliberately out of scope (recorded as a follow-up in CODESTD-REPORT-2026-10-08.md): its SPDX coverage and MODULE_LICENSE normalization need their own GPL-2.0 pass.

vgsp-rm: harden the boot-time queue link against the 4 s GspStatusQueueInit window

2026-10-08T13:38:39+00:00

The guest's first RmInitAdapter intermittently fails: GspStatusQueueInit gives the status-queue link only 4 s (message_queue_cpu.c: timeoutUs = 4000000) and the first boot attempt can burn it, after which the RM tears down, leaks the device and retries; the surviving retry leaves the GR/ctx state degraded, which is the persistent cudaMalloc stall class. Changes: - gsp_rm_emu_poll: parse-retry cadence 250 ms -> 50 ms (the 250 ms tick consumed a fifth of the guest's budget per attempt). - gsp_rm_emu_reset: clear last_parse_retry so a new boot gets a fresh retry budget; take stat_lock while clearing queue cursors so a sema event post cannot race the reset. - gsp_rm_emu_on_booter_done: on GSP_INIT_DONE send failure re-arm the retry path (parsed/active were already set, so the poll loop would never retry). - gsp_rm_parse_queue_chain: free the previous ptes before re-alloc so retries and post-reset re-parses do not leak. - gsp_rm_post_channel_event: never post events across a queue reset (parsed gate). guest_kprobes.sh: add rpcs/rpcp transport probes (the direct RPC-wait symbol is notrace in 570.144) and ccset entry/return probes for rmapiControlCacheSet so a stall names the control whose reply breaks the guest's control-cache bookkeeping.

tools: arm UVM kprobes deterministically (modprobe+wait+verify) and retry pushbuffer pins

2026-10-08T04:01:54+00:00

Root cause of the cudaMalloc/UVM-init stall on our bring-up path: guest_kprobes.sh ran while nvidia_uvm was not yet loaded (it loads lazily with the first CUDA client), so every probe add failed silently (symbols not in kallsyms) and the host never got the #sem -> pushbuffer-pin pipeline. The CE push page (0x120021db8) then never translated and the guest spun forever at DEVICE. - guest_kprobes.sh: modprobe nvidia_uvm, wait (bounded 30 s) for the uvmt symbol, install, and print KP_ARMED/KP_FAIL for bring-up checks. - semfix_watch.pin_pushbuffer: retry transient/partial gva2gpa misses in-call (bounded passes with a short sleep) instead of aborting on the first all-miss block; report unresolved page counts. Validated: 2/2 cold boots (vmkeep6/7) arm the probes and run victory_mail green in 15-20 s (MAIL + VADD bad=0 checksum=12204516 + STATUS OK, oracle-matched); pre-fix boots with unarmed probes stalled 4/4.

guest-diag: mount tracefs before installing kprobes

2026-09-30T14:14:09+00:00

A fresh boot can leave /sys/kernel/tracing as an unmounted mount point; the probe adds then fail silently (probe count 0) and the whole sem/kptr pipeline goes blind.

guest-diag: enlarge and clear the trace ring at probe install

2026-09-29T21:31:55+00:00

The client's spin floods the default ring with uvmt/uvsv events; a one-shot uvbp event was overwritten before the pin scanner read it (push-buffer pin landed ~50 s after the client began pushing). 64 MB ring + clear.

guest-diag: probe the tracker completion check and its result

2026-09-29T21:17:42+00:00

uvsv/uvsvr capture (sem, value) and the boolean return of uvm_gpu_tracking_semaphore_is_value_completed, so a wait that fails despite an advanced payload is visible as a false return.

guest-diag: probe uvm_tracker_wait_for_entry for the exact (channel, value) wait

2026-09-29T21:02:05+00:00

The failing wait is uvm_tracker_wait_for_entry inside channel_pool_add (UVM_GPU_REGISTER): capture the tracker entry's channel pointer and value so the host can tell which release value is still missing.

kernel: authoritative host-injected VA->PA pins (ipin table) + wider pin table

2026-09-29T17:52:49+00:00

The CE push-buffer page 0x120000000 resolved to a zero-filled page even after sema_map_add injected the correct translation: the shared pin table silently dropped inserts once a hash cluster exceeded the 32-probe window (the page hashes to slot 0, a hot cluster), and lookups fell through to the content-scanned map and answered with a stale page - so every CE push read as implausible and memset stayed 0. Injected translations now live in a separate authoritative table checked before every other source and are never written by self-discovery; the general pin table is 4x larger with a 4x probe window and counts drops (pin_drop/ipin_drop) in the stats line. guest_kprobes.sh disables the kprobe group before removal (EBUSY) and matches p:kprobes/; the guest wait sampler can run counters-only with GDB=0.

guest-diag: deterministic channel->pool->manager->pushbuffer chain + chunk-read diagnostics

2026-09-29T17:32:46+00:00

semfix_watch.py now resolves the CE pushbuffer from guest truth instead of scanning a wide raw window: it walks uvm_channel_t +0=pool, pool+0=manager, manager+8=pushbuffer, pushbuffer+8=rm_mem, rm_mem+0x10=vas[] and prefers the rm_mem whose GPU VA is in the 0x120000000 CE pushbuffer range. Three entry points: uvch probe on __uvm_push_begin_acquire_on_channel_with_info, uvbp probe on uvm_pushbuffer_begin_push, and the uvmt tracking-sem kptr (channel scan back from sem). Misses now print structured hop counters so a failed chunk read is distinguishable from a failed chain. guest_kprobes.sh (re)installs uvmt/uvch/uvbp/uvpbf after every guest boot (plain, module: and bare spellings).