summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* tools/soak.sh: fix the present-clients probe (the for-loop's exit status was ↵Arthur Hanson Rasmusson14 hours
| | | | 1 whenever the last listed client was absent, so a partial battery always failed with 'no-client-binaries-in-/tmp'); append true inside the probe
* r441 zero the guest-visible release words on the FIRST attempt too (not only ↵Arthur Hanson Rasmusson16 hours
| | | | outcome retries): the r439 evidence shows a first attempt breaking its poll on a stale value (qmd_rel=0x5 from the previous launch's engine write, want=6) while the engine never fetched the deposit (sentinel intact) - the host then shipped a completion for a launch that did not run; zeroing forces the poll to wait for this launch's own engine write, so the timeout ladder and the software-authored completion are the honest path and the wedge classification can no longer be satisfied by a stale value
* r436 full sysmem PTE aperture/VOL matrix: gsp_phys_rmcall_pte_coh becomes a ↵Arthur Hanson Rasmusson18 hours
| | | | 4-mode selector (0=0xf aperture3+VOL historical, 1=0xd aperture2+VOL r431, 2=0x5 aperture2 non-volatile, 3=0x7 aperture3 non-volatile) applied consistently to the leaf PTE and the single PDE (bit0 = IS_PDE there), so the r431 finding (aperture 2 with VOL still 1 is a pair the reference RM never emits) can be A/B'd against the consistent encodings
* tools: frame_reader picks the largest kvmfr shm (a stray empty 'kvmfr-*' ↵Arthur Hanson Rasmusson23 hours
| | | | file existed); wedge_ab same-program vs second-launch discriminator
* tools/lg + docs: guest frame producer path (CUDA -> ivshmem -> host) and the ↵Arthur Hanson Rasmusson24 hours
| | | | | | | | | | | | | | | | | Looking Glass status - tools/cuda-ab/frame_producer.cu: renders an animated Julia frame on the guest GPU and publishes it into the ivshmem BAR (the kvmfr 128 MiB window) with a small header + RGBA payload. - tools/lg/frame_reader.py: host-side reader of /dev/shm/kvmfr-<uuid> that saves each published frame as a PPM (the transport half of the frame path). - tools/guest-diag/egl_frame.c: EGL probe (device/surfaceless/default platforms) - live result: eglQueryDevicesEXT sees the NVIDIA device and eglInitialize succeeds, but eglMakeCurrent fails EGL_BAD_ALLOC (0x3003), the same VASpace-allocation wall as Xorg's 'Failed to allocate VASpace object'. - docs/05-operations/06: the state, the blockers and the exact remaining pieces (guest GL VASpace allocation; LGMP session producer).
* tools/cuda-ab: wedge_bisect - bisect the no-execute wedge trigger by ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | inter-launch activity Modes A/B/C/D add none / an extra H2D / an extra D2H / a 500 ms sleep between the verify and the next launch of the wedge_recover shape. The inputs mutate every iteration, so a no-execute launch is reported (bad>0), never masked by deterministic content.
* gr: r420b author the guest QMD release on every outcome when release0 is ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | | | | | | | | | | | redirected With gsp_phys_rmcall_gr_rel_mod=1 (release0 -> module semaphore, the ordering fix for the engine-release-before-copy-out race) the engine never writes the guest's own QMD release word, so the host must author it on EVERY backend outcome. The hold's success/fail edges do; a launch with no deferred release records (no hold engaged) fell through and its client hung on a word nothing would ever write - the r414 saxpy f0 cur=5 stall. The backend now reports qmd_rel_redirected in the copy result whenever it points release0 at the module semaphore (rel_mod, or no targetable guest release page). svc_fwd_gr_work folds that into the existing gr_rel_unobserved latch, which makes the bounded outcome check author the word on done (success or failure) - the same forward-only write the hold edges perform. rel_mod=0 paths are unchanged (the engine still authors the word and the latch is only set on an unresolved release page).
* gr: r410 wedge quiet window - bounded idle hold before retrying a lane whose ↵Arthur Hanson Rasmusson27 hours
| | | | | | | | | | | | | | | | | | | | | previous launch also no-wrote (r416: 58-launch sticky wedge clears after ~50 s quiet); wedge counters in state; wedge_recover v2 unmask The r416 boot reproduced the sticky no-execute wedge inside 60 s: 58 consecutive launches accepted zero writes while the lane stayed busy (wedge_recover v1's deterministic verify masked every one of them), and the first launches were green again only after the lane sat idle ~52 s. Both r409 (~37 s idle heal) and r416 therefore show the wedge clears on idleness and persists under back-to-back retries (r154/r409), so the retry loop now holds a quiet window (gsp_phys_gr_wedge_quiet_ms, default 20000, doubling on the second quiet retry, 0 disables) before the normal fresh-bank + resched retry when the previous launch also no-wrote. The whole-launch stall budget is relaxed only on this bounded recovery path. NOWRITE lines now carry the consecutive streak and quiet/heal counters, and the state file shows nw_streak/wedge_quiet/wedge_heals. wedge_recover v2 mutates the inputs every iteration so a stale (never executed) launch can no longer match the expected values, and classifies the readback STALE-PREV(NOWRITE) vs OTHER/PARTIAL.
* tools/cuda-ab: memset_probe + wedge_recover forensics clients (session 10)Arthur Hanson Rasmusson28 hours
| | | | | | | | | | | | | | | | memset_probe: linear 8 MB cudaMemset + D2H with first-bad offset and a 4 KB page map (correct / other / malloc-prefill) - separates 'fill under-delivered' from 'readback delivered wrong pages'; standalone green, bisects the memcpy_roundtrip memset phase. wedge_recover: launch+verify loop that on failure sleeps N seconds and retries the same iteration (same context), reporting per-iteration attempts and launch times - tests whether an idle gap heals the per-boot no-execute wedge (r409 log shows a ~37 s client-side gap healed a wedged run). Also ships the missing inplace_scale run binary (its source landed in 7685d1b3).
* tools/cuda-ab: inplace_scale - minimal in-place scale repro for the r407 ↵Arthur Hanson Rasmusson32 hours
| | | | | | | | | drop-stale release race One H2D, one in-place scale launch (128 KB = 32 pages), one D2H, per-page diff. The client made the r406/r407 stale-release race reproducible in isolation: with the drop-stale fix the same binary runs 6/6 + 4/4 green across two fresh boots (was intermittent 2/3 before).
* sema: r401 acquire-wait hold bound + session-wide exactly-once capture memoryArthur Hanson Rasmusson36 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two fixes for the streams2 / multi-stream ordering class, both verified on fresh boot r401 with the acquire gate enabled: 1. sema_wait_hold_ms (new module param, default 4000 ms) replaces the hardcoded 100 ms acquire-wait bound in svc_run_ops. The old bound is shorter than one emulated GR launch completion (0.5-1.6 s observed), so a D2H readback's ACQ_CIRC_GEQ on the producer channel's progress counter (op=3 pay=4 on 0x20400ffd0) gave up and ran BEFORE the scale launch wrote d_c2; the client then verified pre-scale data (bad=32767). r401 log: the same entry now holds ~1.2 s across cur=1->4 and runs after GUEST GR LAUNCH OK (copy-back optrace 92315.594824 with dst16_after = the post-scale bytes). Fail-open is preserved: a genuinely unsatisfiable acquire still expires, just late enough that the dependency's work has landed (late-but-correct over early-and-wrong). 2. svc_gr_seen: a session-scoped open-addressed (push_va, hash) table behind svc_gr_cand_push_seen(). The 4-slot candidate ring + 8-VA history evicted a real capture within ~12 later captures, so a true replay of an already-forwarded launch was re-captured and re-executed (r399: the same nine launch pushes re-submitted every ~10-11 s for minutes). Cleared in svc_session_flush because a new client's identical-content pushes restart their release counters - a boot-scoped table would silently drop them (LAUNCH-SKIP). Also adds tools/cuda-ab/streams2d.cu: streams2 with per-array/per-4KB-block diff reporting plus the device pointer dump, which localised the remaining failure to the D2H staging write path (our staging-page resolution vs the guest's view).
* cuda-ab/soak: accept a terminal STATUS line as completion - the guest ↵Arthur Hanson Rasmusson47 hours
| | | | wrapper's EXIT= marker is lost when the guest's process reaping stalls (observed: victory_mail reached STATUS OK but the zombie was never reaped; soak reported a false still-running failure)
* cuda-ab: add launch_count guest binary (single-launch execution-count repro, ↵Arthur Hanson Rasmusson2 days
| | | | expect LAUNCH_COUNT=1)
* cuda-ab: launch_count - single-launch execution-count repro (each forwarded ↵Arthur Hanson Rasmusson2 days
| | | | launch runs its engine work twice; the multi-pass class root cause)
* site: 2026-10-20 status refresh - guest CUDA kernel execution workingArthur Hanson Rasmusson2 days
|
* tools: prompt re-assert after recv chunks, heartbeat inside long QMP loops, ↵Arthur Hanson Rasmusson5 days
| | | | novel-only activity accounting (tools-fix a4840952)
* tools: kill-cascade purge + stall watchdogs + monotonic client identity ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | (tools-fix c8fb5377) Dead-pid purge via in-band #nopid (zombie-aware probe), monotonic (pid,starttime) client identity so trace-ring replays can never reset an active client (fixes the m178 mid-run reset regression avenue), pre-capture and post-capture stall watchdogs that re-assert pins and escalate loudly, heartbeat threads for both tools, supervisor STALE 75s/GRACE 120s.
* tools: revert m178 latching changes (client-#3 hang regression); kernel m178 ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | stays The m178 tool latching changes caused a reproducible pre-capture hang of the third CUDA client after two green clients (r199: smiley #3; r200: multikernel #3; both recovered by reverting tools only, r201). Prime suspect: semfix per-client reset re-firing mid-run (same pid reset twice 2 min apart while the client was active). Tools restored to the m177-proven state until the reset logic is fixed with a monotonic per-pid guard.
* vgsp: m178 GR launch reliability + diagnostics; tools: pin latching robustnessArthur Hanson Rasmusson5 days
| | | | | | | | | | | | | Kernel: authorship instrumentation + swrel gate for in-flight GR release VAs; first-poll shortening + fast extended re-arm (params gr_retry_fast=1, gr_first_poll_ms=2500); per-attempt win0 bank recompute + shadow recopy in the outcome-retry loop; poll-break/deposit-readback diagnostics; C1-C10 sema diagnostics (crel-entry, relx, opdef-clear, walk, pub-adv, hold-fallback, busy-defer, CAND-OVERWRITE, inline recs); busy-defer hold instead of software-release consume; counter fast path; QMD word 48/49 skip. Tools: heartbeat thread, scanner-alive probe on reinstall, full periodic re-assert, #nopid handling, per-client reset fix (debounce), dead-pid drop, supervisor grace+threshold.
* vgsp: m177 pin-free self-sufficiency wave + tool hardeningArthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | (1) svc_session_flush keeps svc.direct_base (per-boot constant; the pin-free targeted scanner's gate/input) and a new gsp_sema_service_guest_reset() hook wipes boot-scoped discovery state on guest RM reload; (2) counter-page derive now takes the first READABLE source (vaspaces -> discovered -> targeted) instead of failing -EIO on a walked-but-garbage page, with one-shot diagnostics; (3) new m177c UVM ring-space directory discovery: content-verified (push_va -> push_pa) samples from the GR content discovery and CE push pairing feed a second discovered-directory slot (walks the 0x2xxxxxxx family), consulted by svc_walk_discovered; (4) GR channels now deliver withheld opdef sets at entry consume (normal tail and the stale-skip path) instead of waiting for the 10 s sweep; (5) host tools hardened: semfix_uvmt QMP reconnect + pid-generation dedupe + stale-sem eviction + post-success pin latch + heartbeats; pin_gr_stream accept timeout + guest-scanner reinstall backoff + restart loop + subprocess timeouts; new pin_supervisor.sh (heartbeat watchdog, capped restarts) wired into start_vm_keep.sh. Evidence: r194-r196 cycles; pool resolves src=trusted pin-free; ring dir 0x10801c000 found 4/4; row: 'rel-apply ... written' with tools up.
* vgsp: m168 revert to session scope - the pool page is per-CLIENT (conflict ↵Arthur Hanson Rasmusson6 days
| | | | line r171: cached=0x1247b5000 new=0x1321fa000); a cross-session cache served the previous client's dead page and silently absorbed releases. scanner: pool demand resolves against every known tracker sem (seen + resolved chains)
* vgsp: m163 lost-H2D copy integrity (drain injected pins before CE op ↵Arthur Hanson Rasmusson6 days
| | | | resolution; hold failed device-copy ops) + m164 CLASS-2 GR no-capture entry hold + m165 pool release applies at fresh-pin time + m166 timeout-ladder warm-sem fail-fast; scanner: first-class pool VA resolution + periodic re-assert + demand routing
* guest-diag: demand-pin content resolver searches +/-16 pages and learns the ↵Arthur Hanson Rasmusson6 days
| | | | winning per-area offset (multi-MB CE staging pushes never resolved at +/-1)
* guest-diag: semfix trace reads are remotely timed and SSH-preferred (stuck ↵Arthur Hanson Rasmusson6 days
| | | | QGA tails stopped pool/tracker pins and cascaded every client after ~5 to a DEVICE freeze)
* guest-diag: fix CE-window demand-pin offset mask (& 0xFFFFF folded every 1 ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | | | MB rank to offset 0) Demand pins for 0x120700000/0x120800000/0x120a00000 all resolved to page 0 of the first nv_vm_map_pages area (the same wrong PA), so the host read garbage for every rank but the first and repeat clients stalled at DEVICE. Use the full 16 MB window offset (& 0xFFFFFF); the per-area page bound rejects areas too small. Live: r161 ran victory_mail green as clients 1 AND 2 in one boot, then matmul and smiley - the repeat-client stall class is gone for these.
* guest-diag: demand pinning for stalled CE VAs (event-independent hole filler)Arthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | | | | | The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).
* guest-diag: single-instance scanner start + full-re-assert backoffArthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | r148 ran THREE guest scanners (pin_gr_stream install_and_start retried while a partially-failed first attempt had already started one, and the pid-file kill missed it). Each scanner re-asserted the whole CE pair set every 2 s; with a fully fragmented pair set (descending PAs, one page per line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19 with clients wedged spinning in userspace. - pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern assembled at runtime with printf so the command's own cmdline cannot self-match), then start exactly one. - guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s) as a safety net. Configurable via env.
* guest-diag: heal session-flush pin wipes with periodic full CE re-assertArthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The host wipes its whole sema map when the previous client's last channel closes ('session flush (all channels closed)'). A pin batch that raced in for the new client (observed: the 1-page 0x120000000 pin emitted in the same second as the flush) was lost, and the scanner's content-gated delta never re-emitted the unchanged pairs - so the second client's CE pushes were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'), every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls), and the guest spun in uvm_tracker_wait_for_entry. Fix (scanner, guest side): - _ce_mem_pages: walk the whole vmalloc area (minus header off), not the size-derived prefix that stopped one page short of the write frontier. - _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus every fresh_map pass; emits every translatable pair content-independent, merged into contiguous runs. Zero-content pages are pinned on purpose: the pin carries the translation, the host reads live content. - missing pages retried with the negative PA cache popped (bounded 512/pass). Fix (host tool): - pin_gr_stream._flush: never dedupe single-page repeats of the CE window (0x120000000..0x121000000) so the re-asserts land. Live-validated in r145 (same-boot, no rebuild): mail green as 4th client, matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9), both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same sequences stalled. Headless smiley all-wrong (bad=131072) reproduced independently and is the next target (engine-side constant class).
* guest-diag: resolve and stream the PMA scrubber buffer words (third-client ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | | | | scrub freeze) guest_pin_stream: find the scrubber channel buffer among the nvidia vmap areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when non-empty), walk its pages with kcore and emit '#scrub <fin_pa>,<hsem_pa>' (the finish payload at +0x6c004 and the host semaphore at +0x6c000). pin_gr_stream: handle '#scrub' lines and register the PAs through the sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can author the guest's scrub progress.
* vgsp-sema: author the PMA scrubber channel's progress (third-client scrub ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | | | | | freeze) The guest PMA scrubber (in-guest CeUtils CE channel, 4096 GPFIFO entries) waits on its channel buffer's finish payload (offset 0x6c004) after the third CUDA session's pages must be scrubbed; nothing in the emulation authored it, so cudaMalloc froze in scrubCheckAndWaitForSize. - channel_add: detect the 4096-entry scrubber channel and stash its USERD PA/ring geometry (the channel itself is left untouched: no rank change, no pump change - the colliding numeric-handle FREE may still unregister it, the geometry survives). - sema_scrubbuf_add debug param: register the finish/host-semaphore word PAs the guest scanner resolves (vmallocinfo + kcore walk), because the RM kernel VA space collides numerically with the client CE window in the global VA map and cannot be resolved through it. - svc_scrub_pass (service thread, per pass): from the channel's own USERD GPPut, track submitted scrub blocks across wraps, then author USERD GP_GET = put, host semaphore = put and finish payload = submitted. Forward-only, idempotent, zero writes while the scrubber idles. Live-validated mechanism (r137 VM): authoring the finish word at the kcore- resolved PA let the frozen victory_matmul leave scrubCheckAndWaitForSize and proceed to its launch.
* guest-diag: content-validated delta emission for the CE push-buffer memArthur Hanson Rasmusson10 days
| | | | | | | | The one-shot CE mem emission pinned the whole 16 MB window including pages whose content was zero at the time; the host then read zeros for the other CE ranks' pushes. Sweep the mem every 2 s, skip zero pages, emit only new/changed {va,pa} pairs, and burst the Kcore PA cache every 30 s so a remap is re-emitted instead of shadowed by the cache.
* guest-diag: channel-anchored CE pool resolution for the spinning ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | tracking-sem case The second-client stall leaves exactly one live uvmt event: a tracking semaphore spinning with val=0 (the other channel/one-shot probes are inlined in this driver build and never fire). The sem is a member of a uvm_channel_t whose records sit in the sem's own window and carry the CE ring GPU VA (0x121010000+) plus back-to-back object handles; the channel's pool pointer is the first field at channel+0 (channel = sem-0x80), and the pool chain already satisfies _chain_mems_at once it is tried. The resolver now expands the candidate set from every ring-VA word's surrounding records and their targets before scoring. Verified live: the r134 spinning sem resolves to the CE push-buffer rm_mem with 413 pin runs, where the bare sem-window scan had produced none.
* guest-diag: keep CE push-buffer latches on stale refresh; never fail ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | freshness on an unknown page PA Two scanner defects seen in r131: (1) a resolved CE mem whose first vmap page could not be translated at store time stored page_pa=0, and the freshness check then failed on every probe, dropping the latch and re-resolving in a tight loop - and each failed re-resolution left the host with no pins for the channel at all; (2) the stale path discarded the latch before knowing the re-resolution would produce anything. Unknown PA now counts as fresh (the {cpu,gpu} object check is the evidence), and a stale refresh keeps the previous latch unless a new resolution actually emits runs.
* guest-diag: read the trace snapshot non-blocking (no stuck tail processes)Arthur Hanson Rasmusson10 days
|
* guest-diag: seed channel-event sets from the trace ring on client switchArthur Hanson Rasmusson10 days
| | | | | | | | | | clear_trace() on a pid switch dropped exactly the one-shot uvmt/uvch events of the new client when they landed in the switch instant; the pipe reader then never saw them and the whole session ran with an empty sem set (r126: vecadd_loop sem=0 for minutes while its CE entries paid the 8 s grace). Scan the ring tail once per switch and seed the sets instead - stale kptrs that are also re-learned resolve to no content and are deferred by the candidate scoring.
* guest-diag: content-score CE mem candidates and refresh stale pin setsArthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | The kptr-keyed one-shot latches assumed object identity, but UVM frees and re-creates CE push-buffer rm_mems and channels mid-session and the slab reuses the same addresses; a resolution taken before such a move pinned pages that read all-zero when the host used them, so every CE ring entry stalled in the 8 s grace (r125: the live pushes sat at new vmap pages of the same rm_mem kptr). Candidates are now scored by their window content (a live push buffer is full of method streams, a freed range reads zeros) and the best is pinned - all-zero candidate sets emit nothing instead of poisoning the host map. A resolved mem is re-validated every ~2 s against its own {cpu,gpu} fields and first-page PA; when they move, the latches drop and the current truth is re-emitted.
* vgsp-sema: hold GR launches whose device argument pointers do not translate; ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | derive the CE vmap pairing per allocation Two second-client zeros/stall countermeasures: - svc_gr_data_fill now reports when a launch carries high (device-space) argument pointers and none of them resolves through any trusted source. svc_gr_request refuses to forward such a launch and leaves the capture unconsumed; the pump holds the ring entry for one bounded window per entry (gsp_sema_gr_grace_ms) and retries it, so a translation that appears a pass later (scanner pin, published directory, learned pair) is used instead of executing the kernel against unresolved pointers. On window expiry the entry is consumed exactly as before (bounded zeros, never a hang). Bounded per-session fill diagnostics name the source that serves each candidate (ipin/pin/lat/map/vas/disc/win) and dump the published vaspace walk results for misses. - The CE push-buffer kcore chain resolver derives its vmap page offset per allocation instead of using a fixed shift: the rm_mem size is read from its real offset (570.144 layout) and paired with the vmallocinfo area page count, so a leading header page is only skipped when it actually exists. With an unreadable size the two candidate offsets are content-probed with the host's method-stream plausibility test and an ambiguous result keeps the status-quo pairing. This removes the off-by-one that fed wrong push pages to the pump (content-rescue crawl).
* guest-diag: drop the fixed +1 vmap pairing (allocation-dependent)Arthur Hanson Rasmusson10 days
| | | | | | | | | | | The +1 vmap pairing was content-validated on one boot's CE push-buffer allocation (4097-page vmap: five rescue anchors at k+1), but r116/r117 show the leading header page is not universal: with the shift applied client #1 stalls mid-run (rescues resume) where it was green without it. Reverting the shift restores the previous behaviour; the kernel-side repair from 16124929 stays (a content-validated rescue replaces the injected pair, so a wrong pin is corrected once per page instead of being re-rescued on every pass).
* vgsp-sema: pair CE push-buffer vmap pages with the right offset; validated ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | rescues repair injections The CE push-buffer uvm_rm_mem is vmalloc'd as a 4097-page area: one leading page plus the 4096-page (16 MB) GPU data window, so GPU data page k lives at vmap page k+1. The kcore chain resolver paired page k with vmap page k, pinning the adjacent push for every page; the pump then read the wrong push, decode failed, and the host fell into the content rescue which crawled one entry per ~2 s (r115: rescue_run=4202, entry_va=0x12001b18c repeating). Verified against five independent content-validated rescue anchors (k=20..28, all delta +1), and the vmap area boundary confirms the extra page (nv_vm_map_pages 4097 pages). Five fixes: - resolve_mem_pins() takes page_off; the CE chain and uvbp push-buffer paths pass mem_page_off() (1 when the vmap is exactly 16 MB + 1 page; 0 for direct-mapped mems like the semaphore pool). - a content-validated rescue page now also replaces the injected-pin entry (svc_rescue_try / svc_rescue_run_try), so a stale or wrong injection can no longer shadow the validated page and re-run the rescue on every pass. Evidence: r115 rescue anchors va 0x120014000->pa 0x12a2e2000 .. va 0x12001c000->pa 0x12a2de000 all at vmap offset k+1; the candidate pages decode as the adjacent push (wrong semaphore payload) before the fix.
* guest-diag: stream the newest client and seed the CE chain from uvchArthur Hanson Rasmusson10 days
| | | | | | | | | | | Two robustness fixes for the sequential client battery. cdt_pid() now picks the highest-pid CUDA candidate: preferring a name-prefix match kept the scanner pinned to a previous client that was still tearing down while the new client had already started pushing, so its first pushes were unreadable, the release expectation drifted past the real payloads and the stall followed. The CE push-buffer chain resolver now also runs for uvch channel kptrs, so the pins do not depend on the uvmt semaphore events alone.
* vgsp-sema: resolve the CE push buffer from the kernel chain (second-client ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | | | | | stall) The second CUDA client's GPFIFO entries in the CE push-buffer window (0x120000000) had no injected pin: that window is never mapped into the client's pagemap, so the scanner's window scan cannot see it, and its only supplier - the one-shot uvbp kprobe event - is raced by two trace clears (the scanner's pid-switch clear and semfix's trace-tail clear). When the event loses the race the entry reads fall back to the ~250 ms content rescue, which is the second-client stall and the same resolution-miss class as the ISSUE-0001 lottery. The guest scanner now walks channel->pool->manager->pushbuffer->rm_mem in kernel memory with kcore and pins the push-buffer pages directly (no trace event to lose), and the trace reader's dedup marks reset on a client switch so a reused kptr cannot suppress the new client's #pb. Boot evidence (r112): rescue reject ipin=0 for va_page=0x120023000, rescue scans 175 ms - 1 s each, want incrementing to 39. Smoke test of this fix on the stalled VM: 478 pin runs (~7955 pages) streamed within seconds, first va=0x120000000 -> pa=0x108090000. A bounded kernel log now prints the first CE push-buffer pins of each session.
* vgsp-sema: re-arm ring/push-buffer discovery latches on channel ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | registration; scanner streams anonymous high-VA buffers Second-client wedge fix: a new GSP session re-registers the same channel handles/ranks, but the per-rank one-shot ring/push-buffer discovery latches stayed set from the previous session, so the new session's ring was never re-autopinned (ring_auto=0 pb=0/0, nomap growing) and the guest spun in its channel-creation wait. svc_chan_rearm_rank() clears the latches on every channel registration; all post-fix cycles launch the second client. Scanner: guest_pin_stream.py now streams the client's anonymous high-VA mappings (cudaMalloc buffers were skipped by the old len(parts)<6 check), so the host can learn their pages. Diagnostics: PT-window lines, per-window first words, post-OK qmdwin w48..59. Remaining: the fill-time svc_ipin_lookup miss for the second client's a/b/c pins (they exist in the pin log but are not seen at fill time) - next session.
* site: repository links use cgit tree URLs; external repo-link auditArthur Hanson Rasmusson11 days
| | | | | | | | | - titles.json: map relative doc links to https://git.vgpu.io/open-iov.git/tree/... (the host runs cgit; /blob/ URLs do not exist) - check_site.py: verify every git.vgpu.io URL on every page resolves (HTTP status check, cached, with backoff) - site pages: repository home links point at the .git/ URL - .publish-manifest.json: revised revision records
* site: open-iov.org 2026 documentation updateArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | - tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link mapping, local-marker scan, About-message insert), WikiBot publisher with diff/check modes and publish manifest, live site auditor, image uploader, site-only page sources (landing page, About, index, sidebar, banner templates, About-message template). - wiki: new public deep-dive sources (project status, install, FAQ, system overview, GSP and GSP-RM, execution frontier, host module, guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger, hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace status; corrected cross-document links; README rewritten for the new set. - wiki/.publish-manifest.json records each published revision for rollback.
* cuda-ab: soak.sh client list env-overridable (SOAK_CLIENTS); smiley last by ↵Arthur Hanson Rasmusson11 days
| | | | | | | default smiley (cudaMallocManaged-class, known-failing until the param-offset fix) moved last so a baseline pass exercises the other 13 first.
* cuda-ab: victory-series coverage battery (8 new clients) + soak runner listArthur Hanson Rasmusson11 days
| | | | | | | | victory_saxpy, victory_reduce (shared+atomics), victory_matmul (tiled shared), victory_strided, victory_memset2d (pitched), victory_atomics, victory_streams_events (3 streams + events), victory_math (mixed int/float/double). Each verifies via readback and prints STATUS OK|FAIL. soak.sh now runs the full 14-client set (seq or sustained).
* guest-diag: semfix_watch read64 uncached by default (stale-cache pool-pin ↵Arthur Hanson Rasmusson11 days
| | | | | | | | | | | failure) The watcher's resolve_sem_pool read sem->page->mem->vas through a process-lifetime cache: a transient zero captured while UVM was still initialising the structures was served forever, the pool pin never landed, and the second-session client hung in cudaMallocManaged (r88 evidence: 'pool unresolved (retry)' from the watcher while the same walk succeeded from a fresh QMP client). Cache is now opt-in.
* guest-diag: scanner EOF guard + exclude non-client nvidia fd holdersArthur Hanson Rasmusson11 days
| | | | | | | | | Two failure modes found when the scanner detects clients by device fd: a transient boot nvidia-smi was picked and its exited pagemap read back EOF, and scan_window advanced by got=0 forever (busy spin, no stream); and non-client nvidia fd holders must never be committed to. Break on got<=0 in scan_window (scan_range already did), skip nvidia-smi/ persistenced/Xorg, and prefer /dev/nvidia-uvm holders.
* guest-diag: scanner detects arbitrary CUDA clients by nvidia fdsArthur Hanson Rasmusson11 days
| | | | | | | | | | The guest pin scanner only tracked processes whose comm matched CLIENT_PREFIXES (cdt,va,victory). An arbitrary CUDA binary (e.g. /tmp/smiley) was invisible: no pages streamed, no VA->PA pins injected, the service could not read its pushes and the client hung in cudaMalloc/uvm_tracker_wait. Detect compute clients by the CUDA device they hold open (/dev/nvidia-uvm preferred, then any /dev/nvidia*), with the prefix match kept as the first-choice path.
* tools/guest-diag: single user netdev for QEMU (fix SSH pre-auth stall)Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | The launch line carried both an explicit rtl8139 on netdev net0 and a stray '-net user,hostfwd=tcp::2222-:22 -net nic' pair, creating a SECOND slirp network with a second NIC and a second 2222 forward. The SSH connection landed on one slirp instance while the guest's replies left through the other interface, stalling the handshake before auth. The explicit rtl8139 now owns the only user netdev with the 2222 forward. The generated launch_qemu_rootport.run.sh is patched in place by the harness from the same line. Verified: sshpass -p ubuntu ssh -p 2222 ubuntu@127.0.0.1 -> nvidia-smi green. Guest-side companion fix (persisted in the qcow2 image): the cloud image's /etc/ssh/sshd_config.d/60-cloudimg-settings.conf set PasswordAuthentication no; it is now yes.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-22 18:59:23 +0000