summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* wall-2 session 74: the client's 11s stalls are userspace completion-slot ↵Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | | | | | | | | | | spins; the next wall is the polled RM control 0x2080a097 Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield + clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings. Guest-side filling of the slot flags lets the client proceed immediately, but vkCreateDevice still fails -4: with the slots filled the client settles into an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation answers with the generic all-zero reply, so the poll never completes. Also captured: sema_push_raw dumps proving the GR channel-init push contains NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then zeros), and the negative result that neither channelFillGpFifo nor nvWriteGpEntry runs for this channel's kick. New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and vmap_watch2.py (vmap/kcore watchers), README-session74.md.
* wall-2 forensics tools: GR channel-init push localization (session 73)Arthur Hanson Rasmusson13 hours
| | | | | | | | | | | | | | | | | Tools and findings from the live wall-2 campaign (r674/r675): the vk_frame client's GR channel-init push (va=0x4480000 len=1432) cannot be translated by the sema service, so no release is applied and vkCreateDevice fails -4. Captured: the RM window-record/memdesc layout (ring memdesc PA == the service's validated ring pa; push memdesc PA injected live and read by the service without a false-translation line), the CE-only shape of the service rescue matcher, the notrace status of nvidia/modeset kprobes (QEMU gdb hbreak works), and the ring-entry encoding from channelFillGpFifo/nvWriteGpEntry (neither runs for this channel's kick). New tools: wall2_resolver{2..5}.py, wall2_livecap{1..3}.py, RAM structure scanners (scan_r673*.py, trace_push674.py), gdb675*.gdb probes, plus a README with the ranked next steps.
* docs + nvkmstrace: the GL/Vulkan wall fully bisected (SESSION 65) - ↵Arthur Hanson Rasmusson18 hours
| | | | ALLOC_DEVICE reply status 5 (CORE_CHANNEL_ALLOC_FAILED) decoded by the LD_PRELOAD nvKms logger with a widened reply scan; the empty display capability notifier is the first missing piece (a live-gdb fill of SYS_CAP/SYS_CAPB flips the driver to the window path, proving numWindows>0 unblocks it); the residual wall is the vGPU-guest display-class rejection: nvRmApiAlloc(0xc57e) returns NV_ERR_NOT_SUPPORTED inside the guest kernel and never reaches the emulated GSP - the guest runs in vGPU mode where display/modeset classes are host-service-provided by design
* tools/vk + nvkmstrace: real-Vulkan probe stack for the emulated vGPU - ↵Arthur Hanson Rasmusson20 hours
| | | | vk_probe/vk_probe2 (ctypes loader probes: enumerate + create device/queue), vk_frame (headless Vulkan compute renderer: loads the NVIDIA ICD, paints a plasma into an RGBA8 storage image, reads it back to PPM and publishes frames into the kvmfr ivshmem BAR), vk_plasma.comp -> SPIR-V; nvkmstrace.c (LD_PRELOAD NVKMS ioctl logger that decoded the vkCreateDevice failure to ALLOC_DEVICE cmd=0). Measured: the NVIDIA device enumerates and its Vulkan ICD loads (2 ICDs: NVIDIA + llvmpipe); vkCreateDevice fails VK_ERROR_INITIALIZATION_FAILED because nvKmsAllocDevice returns non-success after the display shim
* tools/guest-diag: GL/Vulkan probe + X capture publisher for the real-render ↵Arthur Hanson Rasmusson21 hours
| | | | path - egl_retry (explicit EGL make-current retry probe: the glsi engine-init returns busy once and eglcore never retries; prints the per-try EGL error), xshm_pub (X11/XShm screen capture into the kvmfr ivshmem BAR in the frame_producer layout so a GL/Vulkan-rendered guest frame reaches the host), xorg-vgpu.conf (headless nvidia Xorg server layout for the emulated vGPU)
* tools r625: stuck_client_capture.sh - one-shot capture for a stuck client ↵Arthur Hanson Rasmusson28 hours
| | | | (pause the trace-consuming scanner, read the retained uvtw/uvsv/uvmt tail, walk the tracker + pool page, resume); wraps the manual sequence that unlocked the first-wait evidence
* tools r624: tracker_walk.py + pool_page.py - guest-side (kcore) walks for ↵Arthur Hanson Rasmusson29 hours
| | | | the first-wait class: read a UVM tracker's semaphore object and expected payload, and dump the semaphore-pool page's applied values; together with the host pool-wr/sema-sw logs they compare what the driver waits for against what the host wrote
* tools r621: read_proc_va.py - read a running client's mapped words via ↵Arthur Hanson Rasmusson29 hours
| | | | /proc/<pid>/mem (the guest-side half of the sem-write divergence comparison; pairs with the r619/r620 sema-sw witness)
* tools r617: battery_resume.sh - zombie-tolerant battery resume ↵Arthur Hanson Rasmusson31 hours
| | | | (verdict-in-log completion check; the plain pgrep -x poll falsely hangs on a client that exited after printing its verdict but left an unreaped zombie comm)
* tools r615: redboot-triage.sh - one-pass evidence extraction for the ↵Arthur Hanson Rasmusson32 hours
| | | | unresolved-arg wedge class (sema-shadow-dst entry point, GRFILL-DROP/held/refused chain, untruncated arg-miss walk lists, cand probes, EVPOST drops)
* r586: demanded staging VAs resolve via the client pagemap; fail-closed at ↵Arthur Hanson Rasmusson43 hours
| | | | | | | | | | | | | | | | | | | | | | | the r543 argument defer cap Two halves of the staging-pin coverage class (r584/r585 boots): 1. guest_pin_stream demand handler: resolve_va_by_content rejects any VA outside the CE pushbuffer window (PB_CE_LO..PB_CE_HI), so the demands for the GRFILL dropped staging pages (high client VAs like 0x703304e00000) were silent no-ops - no pin ever landed and the fills kept dropping them. r586 falls back to the client's own pagemap (scan_range on the exact page), the same walk the window scans use. 2. vgsp-work: the r543 argument-defer cap expiry now takes the r583 fail-closed path instead of forwarding: the r585 boot's saxpy was forwarded with n_data=969 dropped=7 partial=1 at the cap and read stale operands (bad=1047527 csum=130523290); the r584 matmul (n_data=2 dropped=18) reached the r564 degenerate refusal. Refuse, author the QMD release in software (rel_done latched), consume, witness. The dropped pages were demanded first (r585), so the retry window had its chance; a launch that would run with unresolved operands is never silently forwarded.
* tools: transport-proof demand writes + tolerant demand parsing (the pool-pin ↵Arthur Hanson Rasmusson44 hours
| | | | | | | | | | | | | | | | | | | latency root cause) The r581 boot's guest scanner died mid-boot with ValueError in main(): "invalid literal for int() with base 16: '0x20400f000\\n0x121000000\\n0x12000ab4c'". A guest-agent transport-escaped append merged demand VAs onto one line with literal backslash-n, and the old parser crashed on it. The crash stopped ALL new pins; the host only re-asserted known pairs, so the UVM pool VA 0x121000000 never arrived while 185 other injections flowed - the pool-pin latency that blocked every validation boot. * pin_gr_stream._demand_send/_demand_loop now write one "echo <va> >>" per VA (no newlines, no backslashes in the transported command; immune to transport escaping). * guest_pin_stream's demand parser extracts every 0x token with re and skips malformed ones instead of raising out of main().
* vgsp-sema r577: never let a foreign release VA claim a CE channel's tracking ↵Arthur Hanson Rasmusson46 hours
| | | | | | | | | | | | | | | | | | | | | | | | slot; event-record A/B knobs; GR-DEC dump deepened r575 starved-release wedge (live): an unreadable UVM CE entry's rescued stale page decoded a release to a foreign VA (0x12006c004, pay=60); the rescue paths adopted it as the channel's tracking slot (last_rel_va) and every later recovery release targeted it, while the client's real pool (0x121000000) stopped at 25 - the client spun in uvm_tracker_wait for 34 (exact-slot poke delivered it and the client completed green). For CE channels only a pool-shaped release target (0x121000000..0x122000000) may become last_rel_va; a foreign rescued release is refused (witnessed, bounded) and the pool step is delivered through the exact-slot fallback / deferred-release machinery instead. The fake's read-before-write keeps the delivery monotonic even when a legit push carries multiple releases. Also: gsp_rm_ev_{seq,nlist,cl} runtime knobs for the guest-RM event drop A/B (the r576 experiment: guest consumes every POST_EVENT record and never enters nv_post_event), and the GR-DEC launch dump now covers 256 data pages (the mail VADD fill carries n_data=969). New: tools/cuda-ab/evhold.cu - deterministic CUDA event holder/wait probe (context + cudaEventSynchronize + a hold window) for the event experiments.
* tools/cuda-ab: push_battery pushes the full client+probe set (inplace_scale ↵Arthur Hanson Rasmusson2 days
| | | | was never pushed - the battery misreported it as a fail; probes/egl_frame/frame_producer now ride along)
* guest-pin-stream: skip non-readable VMAs in the high-VA scanArthur Hanson Rasmusson2 days
| | | | | | | | | | | | A CUDA device buffer is a PROT_NONE reservation (often hundreds of MB). Its pages are never present in the pagemap, so scanning it yields no pairs - but it consumed the VMA_MAX_PAGES budget, and the scan never reached the client's present staging buffers that sit ABOVE the device reservation. Observed: a client whose device range started below its staging buffer got no high-VA pins at all; its H2D copies failed rc=-2 forever and the client read back stale data (soak run: 9/9 green then streams2 bad=32737). Skipping '---' VMAs restores the pins (9 fresh high-VA injections in the first run; streams2 bad=0).
* Revert "vgsp-sema: keep the learned VA map across the session flush ↵Arthur Hanson Rasmusson2 days
| | | | | | (re-compact, not wipe)" This reverts commit 1a49c7eb69c03f58bc51939828de222a2ac42fea.
* vgsp-sema: keep the learned VA map across the session flush (re-compact, not ↵Arthur Hanson Rasmusson2 days
| | | | | | | | | | | | | | | | | | wipe) The flush fires when the channel table momentarily empties - a live client's UVM channel cycling triggers it - and wiping the map left the client's already-resolved data VAs unresolvable until a rebuild scan re-found them, which it may not. Evidence: a mid-battery boot where multikernel's operands went NO ENTRY 40+ s after a flush (the rebuild restored 14-31 K entries without them); the pump stalled for minutes and the next clients read back whole-page garbage. The map is a cache with a merge-on-rebuild contract, so re-compact it (dedup + sort) and keep it as the merge base; reused-VA correctness stays with the ledger landing, the pin-wins rule and the fresh scan. Also: push_battery.sh now pushes inplace_scale (the battery list had it, the push list did not).
* tools/cuda-ab/run/tgemm_probe: rebuild from tgemm_probe.cu (the committed ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | | | binary was a copy of streams_events_probe) The run/tgemm_probe artifact had the same md5 as streams_events_probe, so every 'tgemm probe' invocation so far ran the streams probe instead. The rebuilt binary reproduces the transformer class on the r518 boot: 'PASS name=k pass=2 bad=256 xbad=2 first=(0,0)' plus the XDIFF rows that show the input's first two floats shifted by one pair (X[0..1] read as X[2..3]) - the deterministic first-page corruption.
* tools/cuda-ab/run: the egl_frame guest GL probe binary (committed for the GL ↵Arthur Hanson Rasmusson3 days
| | | | work: EGL enumeration/MakeCurrent wall reproduction)
* r481c sema_push_raw: lower the call-site threshold to 512 bytes so launch ↵Arthur Hanson Rasmusson3 days
| | | | pushes (872/888B) dump; runtime-enable before the client run to avoid boot noise
* tools/cuda-ab/run: the r458-r459 probe binaries (memset2d_probe, ↵Arthur Hanson Rasmusson3 days
| | | | streams_events_probe, tgemm_probe)
* tools/cuda-ab: isolation probes from the transformer/streams_events forensicsArthur Hanson Rasmusson3 days
| | | | | | | | | | | | | | | - tgemm_probe: minimal X[64x256]xW[256x256] GEMM isolation. Found the deterministic first-page corruption: after a forwarded launch (and sometimes already after the H2D upload) the FIRST 8 bytes of the first data-window page are replaced by that page's bytes at offset 8 (X[0..1] <- X[2..3]; vidmem dump confirms; OUT-UNCH shows chunk==guest), which corrupts GEMM row 0 for all 256 columns (bad=256/pass, identical across 3 passes). Not rel_mod-dependent (A/B at runtime). - streams_events_probe: per-buffer DPTRS + per-buffer bad + first-8 expected/actual for the 3-stream event chain (green on a clean boot, bad=131072 after several clients - the client-switch resolution class). - memset2d_probe: per-phase bad/PHIST for victory_memset2d (verified the r458 LINE_COUNT fix: bad_total=0, all three phases).
* pin_gr_stream: the demand-loop regex never matched the kernel's stalled-VA ↵Arthur Hanson Rasmusson3 days
| | | | | | line (it expected 'stalled VA #N=0x...' but the kernel logs 'stalled VA #N: 0x...'), so host->guest demand pins never fired for stalled VAs and a client whose push/operand pages the resolver missed stayed stalled for the rest of the boot. Accept ':' or '=' after the entry number. Evidence: r463/r464 stalled clients log 'stalled VA #16: 0x755813400000' etc. while the host demand loop only ever emitted pool VAs (0x12100000x) to /tmp/guest_want_vas.txt.
* tools/cuda-ab/run: add the r451-r453 probe binaries (ip_probe*, ip4/5/6, ↵Arthur Hanson Rasmusson4 days
| | | | streams2_probe, scale_probe, rt_probe) - the suite that cracked the additive-csum collision and now serves as the regression detector for the reader-side resolution class
* tools/cuda-ab: the r451-r453 probe suite that cracked the deterministic ↵Arthur Hanson Rasmusson4 days
| | | | class - ip_probe (CTA-marker in-place scale), ip_probe2 (scale_if/scale_u/fill_c/add_c staged matrix), ip_probe3 (what the SM actually reads), streams2_probe (4 stream-ordering modes), scale_probe (in-place vs out-of-place), rt_probe (memcpy_roundtrip staged forensics); run them with setsid+poll, never timeout-kill mid-flight
* tools/soak.sh: fix the present-clients probe (the for-loop's exit status was ↵Arthur Hanson Rasmusson4 days
| | | | 1 whenever the last listed client was absent, so a partial battery always failed with 'no-client-binaries-in-/tmp'); append true inside the probe
* r441 zero the guest-visible release words on the FIRST attempt too (not only ↵Arthur Hanson Rasmusson4 days
| | | | outcome retries): the r439 evidence shows a first attempt breaking its poll on a stale value (qmd_rel=0x5 from the previous launch's engine write, want=6) while the engine never fetched the deposit (sentinel intact) - the host then shipped a completion for a launch that did not run; zeroing forces the poll to wait for this launch's own engine write, so the timeout ladder and the software-authored completion are the honest path and the wedge classification can no longer be satisfied by a stale value
* r436 full sysmem PTE aperture/VOL matrix: gsp_phys_rmcall_pte_coh becomes a ↵Arthur Hanson Rasmusson4 days
| | | | 4-mode selector (0=0xf aperture3+VOL historical, 1=0xd aperture2+VOL r431, 2=0x5 aperture2 non-volatile, 3=0x7 aperture3 non-volatile) applied consistently to the leaf PTE and the single PDE (bit0 = IS_PDE there), so the r431 finding (aperture 2 with VOL still 1 is a pair the reference RM never emits) can be A/B'd against the consistent encodings
* tools: frame_reader picks the largest kvmfr shm (a stray empty 'kvmfr-*' ↵Arthur Hanson Rasmusson4 days
| | | | file existed); wedge_ab same-program vs second-launch discriminator
* tools/lg + docs: guest frame producer path (CUDA -> ivshmem -> host) and the ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | Looking Glass status - tools/cuda-ab/frame_producer.cu: renders an animated Julia frame on the guest GPU and publishes it into the ivshmem BAR (the kvmfr 128 MiB window) with a small header + RGBA payload. - tools/lg/frame_reader.py: host-side reader of /dev/shm/kvmfr-<uuid> that saves each published frame as a PPM (the transport half of the frame path). - tools/guest-diag/egl_frame.c: EGL probe (device/surfaceless/default platforms) - live result: eglQueryDevicesEXT sees the NVIDIA device and eglInitialize succeeds, but eglMakeCurrent fails EGL_BAD_ALLOC (0x3003), the same VASpace-allocation wall as Xorg's 'Failed to allocate VASpace object'. - docs/05-operations/06: the state, the blockers and the exact remaining pieces (guest GL VASpace allocation; LGMP session producer).
* tools/cuda-ab: wedge_bisect - bisect the no-execute wedge trigger by ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | inter-launch activity Modes A/B/C/D add none / an extra H2D / an extra D2H / a 500 ms sleep between the verify and the next launch of the wedge_recover shape. The inputs mutate every iteration, so a no-execute launch is reported (bad>0), never masked by deterministic content.
* gr: r420b author the guest QMD release on every outcome when release0 is ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | | | redirected With gsp_phys_rmcall_gr_rel_mod=1 (release0 -> module semaphore, the ordering fix for the engine-release-before-copy-out race) the engine never writes the guest's own QMD release word, so the host must author it on EVERY backend outcome. The hold's success/fail edges do; a launch with no deferred release records (no hold engaged) fell through and its client hung on a word nothing would ever write - the r414 saxpy f0 cur=5 stall. The backend now reports qmd_rel_redirected in the copy result whenever it points release0 at the module semaphore (rel_mod, or no targetable guest release page). svc_fwd_gr_work folds that into the existing gr_rel_unobserved latch, which makes the bounded outcome check author the word on done (success or failure) - the same forward-only write the hold edges perform. rel_mod=0 paths are unchanged (the engine still authors the word and the latch is only set on an unresolved release page).
* gr: r410 wedge quiet window - bounded idle hold before retrying a lane whose ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | | | | | previous launch also no-wrote (r416: 58-launch sticky wedge clears after ~50 s quiet); wedge counters in state; wedge_recover v2 unmask The r416 boot reproduced the sticky no-execute wedge inside 60 s: 58 consecutive launches accepted zero writes while the lane stayed busy (wedge_recover v1's deterministic verify masked every one of them), and the first launches were green again only after the lane sat idle ~52 s. Both r409 (~37 s idle heal) and r416 therefore show the wedge clears on idleness and persists under back-to-back retries (r154/r409), so the retry loop now holds a quiet window (gsp_phys_gr_wedge_quiet_ms, default 20000, doubling on the second quiet retry, 0 disables) before the normal fresh-bank + resched retry when the previous launch also no-wrote. The whole-launch stall budget is relaxed only on this bounded recovery path. NOWRITE lines now carry the consecutive streak and quiet/heal counters, and the state file shows nw_streak/wedge_quiet/wedge_heals. wedge_recover v2 mutates the inputs every iteration so a stale (never executed) launch can no longer match the expected values, and classifies the readback STALE-PREV(NOWRITE) vs OTHER/PARTIAL.
* tools/cuda-ab: memset_probe + wedge_recover forensics clients (session 10)Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | memset_probe: linear 8 MB cudaMemset + D2H with first-bad offset and a 4 KB page map (correct / other / malloc-prefill) - separates 'fill under-delivered' from 'readback delivered wrong pages'; standalone green, bisects the memcpy_roundtrip memset phase. wedge_recover: launch+verify loop that on failure sleeps N seconds and retries the same iteration (same context), reporting per-iteration attempts and launch times - tests whether an idle gap heals the per-boot no-execute wedge (r409 log shows a ~37 s client-side gap healed a wedged run). Also ships the missing inplace_scale run binary (its source landed in 7685d1b3).
* tools/cuda-ab: inplace_scale - minimal in-place scale repro for the r407 ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | drop-stale release race One H2D, one in-place scale launch (128 KB = 32 pages), one D2H, per-page diff. The client made the r406/r407 stale-release race reproducible in isolation: with the drop-stale fix the same binary runs 6/6 + 4/4 green across two fresh boots (was intermittent 2/3 before).
* sema: r401 acquire-wait hold bound + session-wide exactly-once capture memoryArthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two fixes for the streams2 / multi-stream ordering class, both verified on fresh boot r401 with the acquire gate enabled: 1. sema_wait_hold_ms (new module param, default 4000 ms) replaces the hardcoded 100 ms acquire-wait bound in svc_run_ops. The old bound is shorter than one emulated GR launch completion (0.5-1.6 s observed), so a D2H readback's ACQ_CIRC_GEQ on the producer channel's progress counter (op=3 pay=4 on 0x20400ffd0) gave up and ran BEFORE the scale launch wrote d_c2; the client then verified pre-scale data (bad=32767). r401 log: the same entry now holds ~1.2 s across cur=1->4 and runs after GUEST GR LAUNCH OK (copy-back optrace 92315.594824 with dst16_after = the post-scale bytes). Fail-open is preserved: a genuinely unsatisfiable acquire still expires, just late enough that the dependency's work has landed (late-but-correct over early-and-wrong). 2. svc_gr_seen: a session-scoped open-addressed (push_va, hash) table behind svc_gr_cand_push_seen(). The 4-slot candidate ring + 8-VA history evicted a real capture within ~12 later captures, so a true replay of an already-forwarded launch was re-captured and re-executed (r399: the same nine launch pushes re-submitted every ~10-11 s for minutes). Cleared in svc_session_flush because a new client's identical-content pushes restart their release counters - a boot-scoped table would silently drop them (LAUNCH-SKIP). Also adds tools/cuda-ab/streams2d.cu: streams2 with per-array/per-4KB-block diff reporting plus the device pointer dump, which localised the remaining failure to the D2H staging write path (our staging-page resolution vs the guest's view).
* cuda-ab/soak: accept a terminal STATUS line as completion - the guest ↵Arthur Hanson Rasmusson5 days
| | | | wrapper's EXIT= marker is lost when the guest's process reaping stalls (observed: victory_mail reached STATUS OK but the zombie was never reaped; soak reported a false still-running failure)
* cuda-ab: add launch_count guest binary (single-launch execution-count repro, ↵Arthur Hanson Rasmusson5 days
| | | | expect LAUNCH_COUNT=1)
* cuda-ab: launch_count - single-launch execution-count repro (each forwarded ↵Arthur Hanson Rasmusson5 days
| | | | launch runs its engine work twice; the multi-pass class root cause)
* site: 2026-10-20 status refresh - guest CUDA kernel execution workingArthur Hanson Rasmusson6 days
|
* tools: prompt re-assert after recv chunks, heartbeat inside long QMP loops, ↵Arthur Hanson Rasmusson8 days
| | | | novel-only activity accounting (tools-fix a4840952)
* tools: kill-cascade purge + stall watchdogs + monotonic client identity ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | (tools-fix c8fb5377) Dead-pid purge via in-band #nopid (zombie-aware probe), monotonic (pid,starttime) client identity so trace-ring replays can never reset an active client (fixes the m178 mid-run reset regression avenue), pre-capture and post-capture stall watchdogs that re-assert pins and escalate loudly, heartbeat threads for both tools, supervisor STALE 75s/GRACE 120s.
* tools: revert m178 latching changes (client-#3 hang regression); kernel m178 ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | | stays The m178 tool latching changes caused a reproducible pre-capture hang of the third CUDA client after two green clients (r199: smiley #3; r200: multikernel #3; both recovered by reverting tools only, r201). Prime suspect: semfix per-client reset re-firing mid-run (same pid reset twice 2 min apart while the client was active). Tools restored to the m177-proven state until the reset logic is fixed with a monotonic per-pid guard.
* vgsp: m178 GR launch reliability + diagnostics; tools: pin latching robustnessArthur Hanson Rasmusson8 days
| | | | | | | | | | | | | Kernel: authorship instrumentation + swrel gate for in-flight GR release VAs; first-poll shortening + fast extended re-arm (params gr_retry_fast=1, gr_first_poll_ms=2500); per-attempt win0 bank recompute + shadow recopy in the outcome-retry loop; poll-break/deposit-readback diagnostics; C1-C10 sema diagnostics (crel-entry, relx, opdef-clear, walk, pub-adv, hold-fallback, busy-defer, CAND-OVERWRITE, inline recs); busy-defer hold instead of software-release consume; counter fast path; QMD word 48/49 skip. Tools: heartbeat thread, scanner-alive probe on reinstall, full periodic re-assert, #nopid handling, per-client reset fix (debounce), dead-pid drop, supervisor grace+threshold.
* vgsp: m177 pin-free self-sufficiency wave + tool hardeningArthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | (1) svc_session_flush keeps svc.direct_base (per-boot constant; the pin-free targeted scanner's gate/input) and a new gsp_sema_service_guest_reset() hook wipes boot-scoped discovery state on guest RM reload; (2) counter-page derive now takes the first READABLE source (vaspaces -> discovered -> targeted) instead of failing -EIO on a walked-but-garbage page, with one-shot diagnostics; (3) new m177c UVM ring-space directory discovery: content-verified (push_va -> push_pa) samples from the GR content discovery and CE push pairing feed a second discovered-directory slot (walks the 0x2xxxxxxx family), consulted by svc_walk_discovered; (4) GR channels now deliver withheld opdef sets at entry consume (normal tail and the stale-skip path) instead of waiting for the 10 s sweep; (5) host tools hardened: semfix_uvmt QMP reconnect + pid-generation dedupe + stale-sem eviction + post-success pin latch + heartbeats; pin_gr_stream accept timeout + guest-scanner reinstall backoff + restart loop + subprocess timeouts; new pin_supervisor.sh (heartbeat watchdog, capped restarts) wired into start_vm_keep.sh. Evidence: r194-r196 cycles; pool resolves src=trusted pin-free; ring dir 0x10801c000 found 4/4; row: 'rel-apply ... written' with tools up.
* vgsp: m168 revert to session scope - the pool page is per-CLIENT (conflict ↵Arthur Hanson Rasmusson9 days
| | | | line r171: cached=0x1247b5000 new=0x1321fa000); a cross-session cache served the previous client's dead page and silently absorbed releases. scanner: pool demand resolves against every known tracker sem (seen + resolved chains)
* vgsp: m163 lost-H2D copy integrity (drain injected pins before CE op ↵Arthur Hanson Rasmusson9 days
| | | | resolution; hold failed device-copy ops) + m164 CLASS-2 GR no-capture entry hold + m165 pool release applies at fresh-pin time + m166 timeout-ladder warm-sem fail-fast; scanner: first-class pool VA resolution + periodic re-assert + demand routing
* guest-diag: demand-pin content resolver searches +/-16 pages and learns the ↵Arthur Hanson Rasmusson9 days
| | | | winning per-area offset (multi-MB CE staging pushes never resolved at +/-1)
* guest-diag: semfix trace reads are remotely timed and SSH-preferred (stuck ↵Arthur Hanson Rasmusson9 days
| | | | QGA tails stopped pool/tracker pins and cascaded every client after ~5 to a DEVICE freeze)
* guest-diag: fix CE-window demand-pin offset mask (& 0xFFFFF folded every 1 ↵Arthur Hanson Rasmusson9 days
| | | | | | | | | | | MB rank to offset 0) Demand pins for 0x120700000/0x120800000/0x120a00000 all resolved to page 0 of the first nv_vm_map_pages area (the same wrong PA), so the host read garbage for every rank but the first and repeat clients stalled at DEVICE. Use the full 16 MB window offset (& 0xFFFFFF); the per-area page bound rejects areas too small. Live: r161 ran victory_mail green as clients 1 AND 2 in one boot, then matmul and smiley - the repeat-client stall class is gone for these.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-25 21:55:45 +0000