open-iov.git/tools/guest-diag/pin_gr_stream.py, branch stallfree

tools: prompt re-assert after recv chunks, heartbeat inside long QMP loops, novel-only activity accounting (tools-fix a4840952)

2026-10-18T04:56:45+00:00

tools: kill-cascade purge + stall watchdogs + monotonic client identity (tools-fix c8fb5377)

2026-10-18T04:26:02+00:00

Dead-pid purge via in-band #nopid (zombie-aware probe), monotonic (pid,starttime) client identity so trace-ring replays can never reset an active client (fixes the m178 mid-run reset regression avenue), pre-capture and post-capture stall watchdogs that re-assert pins and escalate loudly, heartbeat threads for both tools, supervisor STALE 75s/GRACE 120s.

tools: revert m178 latching changes (client-#3 hang regression); kernel m178 stays

2026-10-18T00:35:21+00:00

The m178 tool latching changes caused a reproducible pre-capture hang of the third CUDA client after two green clients (r199: smiley #3; r200: multikernel #3; both recovered by reverting tools only, r201). Prime suspect: semfix per-client reset re-firing mid-run (same pid reset twice 2 min apart while the client was active). Tools restored to the m177-proven state until the reset logic is fixed with a monotonic per-pid guard.

vgsp: m178 GR launch reliability + diagnostics; tools: pin latching robustness

2026-10-17T23:18:50+00:00

Kernel: authorship instrumentation + swrel gate for in-flight GR release VAs; first-poll shortening + fast extended re-arm (params gr_retry_fast=1, gr_first_poll_ms=2500); per-attempt win0 bank recompute + shadow recopy in the outcome-retry loop; poll-break/deposit-readback diagnostics; C1-C10 sema diagnostics (crel-entry, relx, opdef-clear, walk, pub-adv, hold-fallback, busy-defer, CAND-OVERWRITE, inline recs); busy-defer hold instead of software-release consume; counter fast path; QMD word 48/49 skip. Tools: heartbeat thread, scanner-alive probe on reinstall, full periodic re-assert, #nopid handling, per-client reset fix (debounce), dead-pid drop, supervisor grace+threshold.

vgsp: m177 pin-free self-sufficiency wave + tool hardening

2026-10-17T21:51:19+00:00

(1) svc_session_flush keeps svc.direct_base (per-boot constant; the pin-free targeted scanner's gate/input) and a new gsp_sema_service_guest_reset() hook wipes boot-scoped discovery state on guest RM reload; (2) counter-page derive now takes the first READABLE source (vaspaces -> discovered -> targeted) instead of failing -EIO on a walked-but-garbage page, with one-shot diagnostics; (3) new m177c UVM ring-space directory discovery: content-verified (push_va -> push_pa) samples from the GR content discovery and CE push pairing feed a second discovered-directory slot (walks the 0x2xxxxxxx family), consulted by svc_walk_discovered; (4) GR channels now deliver withheld opdef sets at entry consume (normal tail and the stale-skip path) instead of waiting for the 10 s sweep; (5) host tools hardened: semfix_uvmt QMP reconnect + pid-generation dedupe + stale-sem eviction + post-success pin latch + heartbeats; pin_gr_stream accept timeout + guest-scanner reinstall backoff + restart loop + subprocess timeouts; new pin_supervisor.sh (heartbeat watchdog, capped restarts) wired into start_vm_keep.sh. Evidence: r194-r196 cycles; pool resolves src=trusted pin-free; ring dir 0x10801c000 found 4/4; row: 'rel-apply ... written' with tools up.

vgsp: m163 lost-H2D copy integrity (drain injected pins before CE op resolution; hold failed device-copy ops) + m164 CLASS-2 GR no-capture entry hold + m165 pool release applies at fresh-pin time + m166 timeout-ladder warm-sem fail-fast; scanner: first-class pool VA resolution + periodic re-assert + demand routing

2026-10-17T00:54:49+00:00

guest-diag: demand pinning for stalled CE VAs (event-independent hole filler)

2026-10-15T06:06:03+00:00

The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).

guest-diag: single-instance scanner start + full-re-assert backoff

2026-10-15T05:19:12+00:00

r148 ran THREE guest scanners (pin_gr_stream install_and_start retried while a partially-failed first attempt had already started one, and the pid-file kill missed it). Each scanner re-asserted the whole CE pair set every 2 s; with a fully fragmented pair set (descending PAs, one page per line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19 with clients wedged spinning in userspace. - pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern assembled at runtime with printf so the command's own cmdline cannot self-match), then start exactly one. - guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s) as a safety net. Configurable via env.

guest-diag: heal session-flush pin wipes with periodic full CE re-assert

2026-10-15T03:41:59+00:00

The host wipes its whole sema map when the previous client's last channel closes ('session flush (all channels closed)'). A pin batch that raced in for the new client (observed: the 1-page 0x120000000 pin emitted in the same second as the flush) was lost, and the scanner's content-gated delta never re-emitted the unchanged pairs - so the second client's CE pushes were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'), every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls), and the guest spun in uvm_tracker_wait_for_entry. Fix (scanner, guest side): - _ce_mem_pages: walk the whole vmalloc area (minus header off), not the size-derived prefix that stopped one page short of the write frontier. - _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus every fresh_map pass; emits every translatable pair content-independent, merged into contiguous runs. Zero-content pages are pinned on purpose: the pin carries the translation, the host reads live content. - missing pages retried with the negative PA cache popped (bounded 512/pass). Fix (host tool): - pin_gr_stream._flush: never dedupe single-page repeats of the CE window (0x120000000..0x121000000) so the re-asserts land. Live-validated in r145 (same-boot, no rebuild): mail green as 4th client, matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9), both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same sequences stalled. Headless smiley all-wrong (bad=131072) reproduced independently and is the next target (engine-side constant class).

guest-diag: resolve and stream the PMA scrubber buffer words (third-client scrub freeze)

2026-10-14T15:44:59+00:00

guest_pin_stream: find the scrubber channel buffer among the nvidia vmap areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when non-empty), walk its pages with kcore and emit '#scrub ,' (the finish payload at +0x6c004 and the host semaphore at +0x6c000). pin_gr_stream: handle '#scrub' lines and register the PAs through the sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can author the guest's scrub progress.