summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* r586: demanded staging VAs resolve via the client pagemap; fail-closed at ↵Arthur Hanson Rasmusson16 hours
| | | | | | | | | | | | | | | | | | | | | | | the r543 argument defer cap Two halves of the staging-pin coverage class (r584/r585 boots): 1. guest_pin_stream demand handler: resolve_va_by_content rejects any VA outside the CE pushbuffer window (PB_CE_LO..PB_CE_HI), so the demands for the GRFILL dropped staging pages (high client VAs like 0x703304e00000) were silent no-ops - no pin ever landed and the fills kept dropping them. r586 falls back to the client's own pagemap (scan_range on the exact page), the same walk the window scans use. 2. vgsp-work: the r543 argument-defer cap expiry now takes the r583 fail-closed path instead of forwarding: the r585 boot's saxpy was forwarded with n_data=969 dropped=7 partial=1 at the cap and read stale operands (bad=1047527 csum=130523290); the r584 matmul (n_data=2 dropped=18) reached the r564 degenerate refusal. Refuse, author the QMD release in software (rel_done latched), consume, witness. The dropped pages were demanded first (r585), so the retry window had its chance; a launch that would run with unresolved operands is never silently forwarded.
* tools: transport-proof demand writes + tolerant demand parsing (the pool-pin ↵Arthur Hanson Rasmusson17 hours
| | | | | | | | | | | | | | | | | | | latency root cause) The r581 boot's guest scanner died mid-boot with ValueError in main(): "invalid literal for int() with base 16: '0x20400f000\\n0x121000000\\n0x12000ab4c'". A guest-agent transport-escaped append merged demand VAs onto one line with literal backslash-n, and the old parser crashed on it. The crash stopped ALL new pins; the host only re-asserted known pairs, so the UVM pool VA 0x121000000 never arrived while 185 other injections flowed - the pool-pin latency that blocked every validation boot. * pin_gr_stream._demand_send/_demand_loop now write one "echo <va> >>" per VA (no newlines, no backslashes in the transported command; immune to transport escaping). * guest_pin_stream's demand parser extracts every 0x token with re and skips malformed ones instead of raising out of main().
* vgsp-sema r577: never let a foreign release VA claim a CE channel's tracking ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | | | | | | | | | | | | | slot; event-record A/B knobs; GR-DEC dump deepened r575 starved-release wedge (live): an unreadable UVM CE entry's rescued stale page decoded a release to a foreign VA (0x12006c004, pay=60); the rescue paths adopted it as the channel's tracking slot (last_rel_va) and every later recovery release targeted it, while the client's real pool (0x121000000) stopped at 25 - the client spun in uvm_tracker_wait for 34 (exact-slot poke delivered it and the client completed green). For CE channels only a pool-shaped release target (0x121000000..0x122000000) may become last_rel_va; a foreign rescued release is refused (witnessed, bounded) and the pool step is delivered through the exact-slot fallback / deferred-release machinery instead. The fake's read-before-write keeps the delivery monotonic even when a legit push carries multiple releases. Also: gsp_rm_ev_{seq,nlist,cl} runtime knobs for the guest-RM event drop A/B (the r576 experiment: guest consumes every POST_EVENT record and never enters nv_post_event), and the GR-DEC launch dump now covers 256 data pages (the mail VADD fill carries n_data=969). New: tools/cuda-ab/evhold.cu - deterministic CUDA event holder/wait probe (context + cudaEventSynchronize + a hold window) for the event experiments.
* tools/cuda-ab: push_battery pushes the full client+probe set (inplace_scale ↵Arthur Hanson Rasmusson28 hours
| | | | was never pushed - the battery misreported it as a fail; probes/egl_frame/frame_producer now ride along)
* guest-pin-stream: skip non-readable VMAs in the high-VA scanArthur Hanson Rasmusson30 hours
| | | | | | | | | | | | A CUDA device buffer is a PROT_NONE reservation (often hundreds of MB). Its pages are never present in the pagemap, so scanning it yields no pairs - but it consumed the VMA_MAX_PAGES budget, and the scan never reached the client's present staging buffers that sit ABOVE the device reservation. Observed: a client whose device range started below its staging buffer got no high-VA pins at all; its H2D copies failed rc=-2 forever and the client read back stale data (soak run: 9/9 green then streams2 bad=32737). Skipping '---' VMAs restores the pins (9 fresh high-VA injections in the first run; streams2 bad=0).
* Revert "vgsp-sema: keep the learned VA map across the session flush ↵Arthur Hanson Rasmusson31 hours
| | | | | | (re-compact, not wipe)" This reverts commit 1a49c7eb69c03f58bc51939828de222a2ac42fea.
* vgsp-sema: keep the learned VA map across the session flush (re-compact, not ↵Arthur Hanson Rasmusson31 hours
| | | | | | | | | | | | | | | | | | wipe) The flush fires when the channel table momentarily empties - a live client's UVM channel cycling triggers it - and wiping the map left the client's already-resolved data VAs unresolvable until a rebuild scan re-found them, which it may not. Evidence: a mid-battery boot where multikernel's operands went NO ENTRY 40+ s after a flush (the rebuild restored 14-31 K entries without them); the pump stalled for minutes and the next clients read back whole-page garbage. The map is a cache with a merge-on-rebuild contract, so re-compact it (dedup + sort) and keep it as the merge base; reused-VA correctness stays with the ledger landing, the pin-wins rule and the fresh scan. Also: push_battery.sh now pushes inplace_scale (the battery list had it, the push list did not).
* tools/cuda-ab/run/tgemm_probe: rebuild from tgemm_probe.cu (the committed ↵Arthur Hanson Rasmusson37 hours
| | | | | | | | | | | binary was a copy of streams_events_probe) The run/tgemm_probe artifact had the same md5 as streams_events_probe, so every 'tgemm probe' invocation so far ran the streams probe instead. The rebuilt binary reproduces the transformer class on the r518 boot: 'PASS name=k pass=2 bad=256 xbad=2 first=(0,0)' plus the XDIFF rows that show the input's first two floats shifted by one pair (X[0..1] read as X[2..3]) - the deterministic first-page corruption.
* tools/cuda-ab/run: the egl_frame guest GL probe binary (committed for the GL ↵Arthur Hanson Rasmusson46 hours
| | | | work: EGL enumeration/MakeCurrent wall reproduction)
* r481c sema_push_raw: lower the call-site threshold to 512 bytes so launch ↵Arthur Hanson Rasmusson2 days
| | | | pushes (872/888B) dump; runtime-enable before the client run to avoid boot noise
* tools/cuda-ab/run: the r458-r459 probe binaries (memset2d_probe, ↵Arthur Hanson Rasmusson2 days
| | | | streams_events_probe, tgemm_probe)
* tools/cuda-ab: isolation probes from the transformer/streams_events forensicsArthur Hanson Rasmusson2 days
| | | | | | | | | | | | | | | - tgemm_probe: minimal X[64x256]xW[256x256] GEMM isolation. Found the deterministic first-page corruption: after a forwarded launch (and sometimes already after the H2D upload) the FIRST 8 bytes of the first data-window page are replaced by that page's bytes at offset 8 (X[0..1] <- X[2..3]; vidmem dump confirms; OUT-UNCH shows chunk==guest), which corrupts GEMM row 0 for all 256 columns (bad=256/pass, identical across 3 passes). Not rel_mod-dependent (A/B at runtime). - streams_events_probe: per-buffer DPTRS + per-buffer bad + first-8 expected/actual for the 3-stream event chain (green on a clean boot, bad=131072 after several clients - the client-switch resolution class). - memset2d_probe: per-phase bad/PHIST for victory_memset2d (verified the r458 LINE_COUNT fix: bad_total=0, all three phases).
* pin_gr_stream: the demand-loop regex never matched the kernel's stalled-VA ↵Arthur Hanson Rasmusson2 days
| | | | | | line (it expected 'stalled VA #N=0x...' but the kernel logs 'stalled VA #N: 0x...'), so host->guest demand pins never fired for stalled VAs and a client whose push/operand pages the resolver missed stayed stalled for the rest of the boot. Accept ':' or '=' after the entry number. Evidence: r463/r464 stalled clients log 'stalled VA #16: 0x755813400000' etc. while the host demand loop only ever emitted pool VAs (0x12100000x) to /tmp/guest_want_vas.txt.
* tools/cuda-ab/run: add the r451-r453 probe binaries (ip_probe*, ip4/5/6, ↵Arthur Hanson Rasmusson2 days
| | | | streams2_probe, scale_probe, rt_probe) - the suite that cracked the additive-csum collision and now serves as the regression detector for the reader-side resolution class
* tools/cuda-ab: the r451-r453 probe suite that cracked the deterministic ↵Arthur Hanson Rasmusson2 days
| | | | class - ip_probe (CTA-marker in-place scale), ip_probe2 (scale_if/scale_u/fill_c/add_c staged matrix), ip_probe3 (what the SM actually reads), streams2_probe (4 stream-ordering modes), scale_probe (in-place vs out-of-place), rt_probe (memcpy_roundtrip staged forensics); run them with setsid+poll, never timeout-kill mid-flight
* tools/soak.sh: fix the present-clients probe (the for-loop's exit status was ↵Arthur Hanson Rasmusson3 days
| | | | 1 whenever the last listed client was absent, so a partial battery always failed with 'no-client-binaries-in-/tmp'); append true inside the probe
* r441 zero the guest-visible release words on the FIRST attempt too (not only ↵Arthur Hanson Rasmusson3 days
| | | | outcome retries): the r439 evidence shows a first attempt breaking its poll on a stale value (qmd_rel=0x5 from the previous launch's engine write, want=6) while the engine never fetched the deposit (sentinel intact) - the host then shipped a completion for a launch that did not run; zeroing forces the poll to wait for this launch's own engine write, so the timeout ladder and the software-authored completion are the honest path and the wedge classification can no longer be satisfied by a stale value
* r436 full sysmem PTE aperture/VOL matrix: gsp_phys_rmcall_pte_coh becomes a ↵Arthur Hanson Rasmusson3 days
| | | | 4-mode selector (0=0xf aperture3+VOL historical, 1=0xd aperture2+VOL r431, 2=0x5 aperture2 non-volatile, 3=0x7 aperture3 non-volatile) applied consistently to the leaf PTE and the single PDE (bit0 = IS_PDE there), so the r431 finding (aperture 2 with VOL still 1 is a pair the reference RM never emits) can be A/B'd against the consistent encodings
* tools: frame_reader picks the largest kvmfr shm (a stray empty 'kvmfr-*' ↵Arthur Hanson Rasmusson3 days
| | | | file existed); wedge_ab same-program vs second-launch discriminator
* tools/lg + docs: guest frame producer path (CUDA -> ivshmem -> host) and the ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | | | | | | | | | Looking Glass status - tools/cuda-ab/frame_producer.cu: renders an animated Julia frame on the guest GPU and publishes it into the ivshmem BAR (the kvmfr 128 MiB window) with a small header + RGBA payload. - tools/lg/frame_reader.py: host-side reader of /dev/shm/kvmfr-<uuid> that saves each published frame as a PPM (the transport half of the frame path). - tools/guest-diag/egl_frame.c: EGL probe (device/surfaceless/default platforms) - live result: eglQueryDevicesEXT sees the NVIDIA device and eglInitialize succeeds, but eglMakeCurrent fails EGL_BAD_ALLOC (0x3003), the same VASpace-allocation wall as Xorg's 'Failed to allocate VASpace object'. - docs/05-operations/06: the state, the blockers and the exact remaining pieces (guest GL VASpace allocation; LGMP session producer).
* tools/cuda-ab: wedge_bisect - bisect the no-execute wedge trigger by ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | inter-launch activity Modes A/B/C/D add none / an extra H2D / an extra D2H / a 500 ms sleep between the verify and the next launch of the wedge_recover shape. The inputs mutate every iteration, so a no-execute launch is reported (bad>0), never masked by deterministic content.
* gr: r420b author the guest QMD release on every outcome when release0 is ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | | | | | | | | | | | redirected With gsp_phys_rmcall_gr_rel_mod=1 (release0 -> module semaphore, the ordering fix for the engine-release-before-copy-out race) the engine never writes the guest's own QMD release word, so the host must author it on EVERY backend outcome. The hold's success/fail edges do; a launch with no deferred release records (no hold engaged) fell through and its client hung on a word nothing would ever write - the r414 saxpy f0 cur=5 stall. The backend now reports qmd_rel_redirected in the copy result whenever it points release0 at the module semaphore (rel_mod, or no targetable guest release page). svc_fwd_gr_work folds that into the existing gr_rel_unobserved latch, which makes the bounded outcome check author the word on done (success or failure) - the same forward-only write the hold edges perform. rel_mod=0 paths are unchanged (the engine still authors the word and the latch is only set on an unresolved release page).
* gr: r410 wedge quiet window - bounded idle hold before retrying a lane whose ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | | | | | | | | | | | | | previous launch also no-wrote (r416: 58-launch sticky wedge clears after ~50 s quiet); wedge counters in state; wedge_recover v2 unmask The r416 boot reproduced the sticky no-execute wedge inside 60 s: 58 consecutive launches accepted zero writes while the lane stayed busy (wedge_recover v1's deterministic verify masked every one of them), and the first launches were green again only after the lane sat idle ~52 s. Both r409 (~37 s idle heal) and r416 therefore show the wedge clears on idleness and persists under back-to-back retries (r154/r409), so the retry loop now holds a quiet window (gsp_phys_gr_wedge_quiet_ms, default 20000, doubling on the second quiet retry, 0 disables) before the normal fresh-bank + resched retry when the previous launch also no-wrote. The whole-launch stall budget is relaxed only on this bounded recovery path. NOWRITE lines now carry the consecutive streak and quiet/heal counters, and the state file shows nw_streak/wedge_quiet/wedge_heals. wedge_recover v2 mutates the inputs every iteration so a stale (never executed) launch can no longer match the expected values, and classifies the readback STALE-PREV(NOWRITE) vs OTHER/PARTIAL.
* tools/cuda-ab: memset_probe + wedge_recover forensics clients (session 10)Arthur Hanson Rasmusson3 days
| | | | | | | | | | | | | | | | memset_probe: linear 8 MB cudaMemset + D2H with first-bad offset and a 4 KB page map (correct / other / malloc-prefill) - separates 'fill under-delivered' from 'readback delivered wrong pages'; standalone green, bisects the memcpy_roundtrip memset phase. wedge_recover: launch+verify loop that on failure sleeps N seconds and retries the same iteration (same context), reporting per-iteration attempts and launch times - tests whether an idle gap heals the per-boot no-execute wedge (r409 log shows a ~37 s client-side gap healed a wedged run). Also ships the missing inplace_scale run binary (its source landed in 7685d1b3).
* tools/cuda-ab: inplace_scale - minimal in-place scale repro for the r407 ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | drop-stale release race One H2D, one in-place scale launch (128 KB = 32 pages), one D2H, per-page diff. The client made the r406/r407 stale-release race reproducible in isolation: with the drop-stale fix the same binary runs 6/6 + 4/4 green across two fresh boots (was intermittent 2/3 before).
* sema: r401 acquire-wait hold bound + session-wide exactly-once capture memoryArthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two fixes for the streams2 / multi-stream ordering class, both verified on fresh boot r401 with the acquire gate enabled: 1. sema_wait_hold_ms (new module param, default 4000 ms) replaces the hardcoded 100 ms acquire-wait bound in svc_run_ops. The old bound is shorter than one emulated GR launch completion (0.5-1.6 s observed), so a D2H readback's ACQ_CIRC_GEQ on the producer channel's progress counter (op=3 pay=4 on 0x20400ffd0) gave up and ran BEFORE the scale launch wrote d_c2; the client then verified pre-scale data (bad=32767). r401 log: the same entry now holds ~1.2 s across cur=1->4 and runs after GUEST GR LAUNCH OK (copy-back optrace 92315.594824 with dst16_after = the post-scale bytes). Fail-open is preserved: a genuinely unsatisfiable acquire still expires, just late enough that the dependency's work has landed (late-but-correct over early-and-wrong). 2. svc_gr_seen: a session-scoped open-addressed (push_va, hash) table behind svc_gr_cand_push_seen(). The 4-slot candidate ring + 8-VA history evicted a real capture within ~12 later captures, so a true replay of an already-forwarded launch was re-captured and re-executed (r399: the same nine launch pushes re-submitted every ~10-11 s for minutes). Cleared in svc_session_flush because a new client's identical-content pushes restart their release counters - a boot-scoped table would silently drop them (LAUNCH-SKIP). Also adds tools/cuda-ab/streams2d.cu: streams2 with per-array/per-4KB-block diff reporting plus the device pointer dump, which localised the remaining failure to the D2H staging write path (our staging-page resolution vs the guest's view).
* cuda-ab/soak: accept a terminal STATUS line as completion - the guest ↵Arthur Hanson Rasmusson4 days
| | | | wrapper's EXIT= marker is lost when the guest's process reaping stalls (observed: victory_mail reached STATUS OK but the zombie was never reaped; soak reported a false still-running failure)
* cuda-ab: add launch_count guest binary (single-launch execution-count repro, ↵Arthur Hanson Rasmusson4 days
| | | | expect LAUNCH_COUNT=1)
* cuda-ab: launch_count - single-launch execution-count repro (each forwarded ↵Arthur Hanson Rasmusson4 days
| | | | launch runs its engine work twice; the multi-pass class root cause)
* site: 2026-10-20 status refresh - guest CUDA kernel execution workingArthur Hanson Rasmusson4 days
|
* tools: prompt re-assert after recv chunks, heartbeat inside long QMP loops, ↵Arthur Hanson Rasmusson7 days
| | | | novel-only activity accounting (tools-fix a4840952)
* tools: kill-cascade purge + stall watchdogs + monotonic client identity ↵Arthur Hanson Rasmusson7 days
| | | | | | | | | | (tools-fix c8fb5377) Dead-pid purge via in-band #nopid (zombie-aware probe), monotonic (pid,starttime) client identity so trace-ring replays can never reset an active client (fixes the m178 mid-run reset regression avenue), pre-capture and post-capture stall watchdogs that re-assert pins and escalate loudly, heartbeat threads for both tools, supervisor STALE 75s/GRACE 120s.
* tools: revert m178 latching changes (client-#3 hang regression); kernel m178 ↵Arthur Hanson Rasmusson7 days
| | | | | | | | | | | stays The m178 tool latching changes caused a reproducible pre-capture hang of the third CUDA client after two green clients (r199: smiley #3; r200: multikernel #3; both recovered by reverting tools only, r201). Prime suspect: semfix per-client reset re-firing mid-run (same pid reset twice 2 min apart while the client was active). Tools restored to the m177-proven state until the reset logic is fixed with a monotonic per-pid guard.
* vgsp: m178 GR launch reliability + diagnostics; tools: pin latching robustnessArthur Hanson Rasmusson7 days
| | | | | | | | | | | | | Kernel: authorship instrumentation + swrel gate for in-flight GR release VAs; first-poll shortening + fast extended re-arm (params gr_retry_fast=1, gr_first_poll_ms=2500); per-attempt win0 bank recompute + shadow recopy in the outcome-retry loop; poll-break/deposit-readback diagnostics; C1-C10 sema diagnostics (crel-entry, relx, opdef-clear, walk, pub-adv, hold-fallback, busy-defer, CAND-OVERWRITE, inline recs); busy-defer hold instead of software-release consume; counter fast path; QMD word 48/49 skip. Tools: heartbeat thread, scanner-alive probe on reinstall, full periodic re-assert, #nopid handling, per-client reset fix (debounce), dead-pid drop, supervisor grace+threshold.
* vgsp: m177 pin-free self-sufficiency wave + tool hardeningArthur Hanson Rasmusson7 days
| | | | | | | | | | | | | | | | | | | | | | (1) svc_session_flush keeps svc.direct_base (per-boot constant; the pin-free targeted scanner's gate/input) and a new gsp_sema_service_guest_reset() hook wipes boot-scoped discovery state on guest RM reload; (2) counter-page derive now takes the first READABLE source (vaspaces -> discovered -> targeted) instead of failing -EIO on a walked-but-garbage page, with one-shot diagnostics; (3) new m177c UVM ring-space directory discovery: content-verified (push_va -> push_pa) samples from the GR content discovery and CE push pairing feed a second discovered-directory slot (walks the 0x2xxxxxxx family), consulted by svc_walk_discovered; (4) GR channels now deliver withheld opdef sets at entry consume (normal tail and the stale-skip path) instead of waiting for the 10 s sweep; (5) host tools hardened: semfix_uvmt QMP reconnect + pid-generation dedupe + stale-sem eviction + post-success pin latch + heartbeats; pin_gr_stream accept timeout + guest-scanner reinstall backoff + restart loop + subprocess timeouts; new pin_supervisor.sh (heartbeat watchdog, capped restarts) wired into start_vm_keep.sh. Evidence: r194-r196 cycles; pool resolves src=trusted pin-free; ring dir 0x10801c000 found 4/4; row: 'rel-apply ... written' with tools up.
* vgsp: m168 revert to session scope - the pool page is per-CLIENT (conflict ↵Arthur Hanson Rasmusson8 days
| | | | line r171: cached=0x1247b5000 new=0x1321fa000); a cross-session cache served the previous client's dead page and silently absorbed releases. scanner: pool demand resolves against every known tracker sem (seen + resolved chains)
* vgsp: m163 lost-H2D copy integrity (drain injected pins before CE op ↵Arthur Hanson Rasmusson8 days
| | | | resolution; hold failed device-copy ops) + m164 CLASS-2 GR no-capture entry hold + m165 pool release applies at fresh-pin time + m166 timeout-ladder warm-sem fail-fast; scanner: first-class pool VA resolution + periodic re-assert + demand routing
* guest-diag: demand-pin content resolver searches +/-16 pages and learns the ↵Arthur Hanson Rasmusson8 days
| | | | winning per-area offset (multi-MB CE staging pushes never resolved at +/-1)
* guest-diag: semfix trace reads are remotely timed and SSH-preferred (stuck ↵Arthur Hanson Rasmusson8 days
| | | | QGA tails stopped pool/tracker pins and cascaded every client after ~5 to a DEVICE freeze)
* guest-diag: fix CE-window demand-pin offset mask (& 0xFFFFF folded every 1 ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | | MB rank to offset 0) Demand pins for 0x120700000/0x120800000/0x120a00000 all resolved to page 0 of the first nv_vm_map_pages area (the same wrong PA), so the host read garbage for every rank but the first and repeat clients stalled at DEVICE. Use the full 16 MB window offset (& 0xFFFFFF); the per-area page bound rejects areas too small. Live: r161 ran victory_mail green as clients 1 AND 2 in one boot, then matmul and smiley - the repeat-client stall class is gone for these.
* guest-diag: demand pinning for stalled CE VAs (event-independent hole filler)Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | | | | | | | The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).
* guest-diag: single-instance scanner start + full-re-assert backoffArthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | r148 ran THREE guest scanners (pin_gr_stream install_and_start retried while a partially-failed first attempt had already started one, and the pid-file kill missed it). Each scanner re-asserted the whole CE pair set every 2 s; with a fully fragmented pair set (descending PAs, one page per line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19 with clients wedged spinning in userspace. - pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern assembled at runtime with printf so the command's own cmdline cannot self-match), then start exactly one. - guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s) as a safety net. Configurable via env.
* guest-diag: heal session-flush pin wipes with periodic full CE re-assertArthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The host wipes its whole sema map when the previous client's last channel closes ('session flush (all channels closed)'). A pin batch that raced in for the new client (observed: the 1-page 0x120000000 pin emitted in the same second as the flush) was lost, and the scanner's content-gated delta never re-emitted the unchanged pairs - so the second client's CE pushes were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'), every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls), and the guest spun in uvm_tracker_wait_for_entry. Fix (scanner, guest side): - _ce_mem_pages: walk the whole vmalloc area (minus header off), not the size-derived prefix that stopped one page short of the write frontier. - _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus every fresh_map pass; emits every translatable pair content-independent, merged into contiguous runs. Zero-content pages are pinned on purpose: the pin carries the translation, the host reads live content. - missing pages retried with the negative PA cache popped (bounded 512/pass). Fix (host tool): - pin_gr_stream._flush: never dedupe single-page repeats of the CE window (0x120000000..0x121000000) so the re-asserts land. Live-validated in r145 (same-boot, no rebuild): mail green as 4th client, matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9), both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same sequences stalled. Headless smiley all-wrong (bad=131072) reproduced independently and is the next target (engine-side constant class).
* guest-diag: resolve and stream the PMA scrubber buffer words (third-client ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | scrub freeze) guest_pin_stream: find the scrubber channel buffer among the nvidia vmap areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when non-empty), walk its pages with kcore and emit '#scrub <fin_pa>,<hsem_pa>' (the finish payload at +0x6c004 and the host semaphore at +0x6c000). pin_gr_stream: handle '#scrub' lines and register the PAs through the sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can author the guest's scrub progress.
* vgsp-sema: author the PMA scrubber channel's progress (third-client scrub ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | | | | | | | freeze) The guest PMA scrubber (in-guest CeUtils CE channel, 4096 GPFIFO entries) waits on its channel buffer's finish payload (offset 0x6c004) after the third CUDA session's pages must be scrubbed; nothing in the emulation authored it, so cudaMalloc froze in scrubCheckAndWaitForSize. - channel_add: detect the 4096-entry scrubber channel and stash its USERD PA/ring geometry (the channel itself is left untouched: no rank change, no pump change - the colliding numeric-handle FREE may still unregister it, the geometry survives). - sema_scrubbuf_add debug param: register the finish/host-semaphore word PAs the guest scanner resolves (vmallocinfo + kcore walk), because the RM kernel VA space collides numerically with the client CE window in the global VA map and cannot be resolved through it. - svc_scrub_pass (service thread, per pass): from the channel's own USERD GPPut, track submitted scrub blocks across wraps, then author USERD GP_GET = put, host semaphore = put and finish payload = submitted. Forward-only, idempotent, zero writes while the scrubber idles. Live-validated mechanism (r137 VM): authoring the finish word at the kcore- resolved PA let the frozen victory_matmul leave scrubCheckAndWaitForSize and proceed to its launch.
* guest-diag: content-validated delta emission for the CE push-buffer memArthur Hanson Rasmusson12 days
| | | | | | | | The one-shot CE mem emission pinned the whole 16 MB window including pages whose content was zero at the time; the host then read zeros for the other CE ranks' pushes. Sweep the mem every 2 s, skip zero pages, emit only new/changed {va,pa} pairs, and burst the Kcore PA cache every 30 s so a remap is re-emitted instead of shadowed by the cache.
* guest-diag: channel-anchored CE pool resolution for the spinning ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | tracking-sem case The second-client stall leaves exactly one live uvmt event: a tracking semaphore spinning with val=0 (the other channel/one-shot probes are inlined in this driver build and never fire). The sem is a member of a uvm_channel_t whose records sit in the sem's own window and carry the CE ring GPU VA (0x121010000+) plus back-to-back object handles; the channel's pool pointer is the first field at channel+0 (channel = sem-0x80), and the pool chain already satisfies _chain_mems_at once it is tried. The resolver now expands the candidate set from every ring-VA word's surrounding records and their targets before scoring. Verified live: the r134 spinning sem resolves to the CE push-buffer rm_mem with 413 pin runs, where the bare sem-window scan had produced none.
* guest-diag: keep CE push-buffer latches on stale refresh; never fail ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | freshness on an unknown page PA Two scanner defects seen in r131: (1) a resolved CE mem whose first vmap page could not be translated at store time stored page_pa=0, and the freshness check then failed on every probe, dropping the latch and re-resolving in a tight loop - and each failed re-resolution left the host with no pins for the channel at all; (2) the stale path discarded the latch before knowing the re-resolution would produce anything. Unknown PA now counts as fresh (the {cpu,gpu} object check is the evidence), and a stale refresh keeps the previous latch unless a new resolution actually emits runs.
* guest-diag: read the trace snapshot non-blocking (no stuck tail processes)Arthur Hanson Rasmusson12 days
|
* guest-diag: seed channel-event sets from the trace ring on client switchArthur Hanson Rasmusson12 days
| | | | | | | | | | clear_trace() on a pid switch dropped exactly the one-shot uvmt/uvch events of the new client when they landed in the switch instant; the pipe reader then never saw them and the whole session ran with an empty sem set (r126: vecadd_loop sem=0 for minutes while its CE entries paid the 8 s grace). Scan the ring tail once per switch and seed the sets instead - stale kptrs that are also re-learned resolve to no content and are deferred by the candidate scoring.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-24 18:58:46 +0000