open-iov.git/tools, branch lottery-m156-gate-attempt

guest-diag: demand pinning for stalled CE VAs (event-independent hole filler)

2026-10-15T06:06:03+00:00

The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).

guest-diag: single-instance scanner start + full-re-assert backoff

2026-10-15T05:19:12+00:00

r148 ran THREE guest scanners (pin_gr_stream install_and_start retried while a partially-failed first attempt had already started one, and the pid-file kill missed it). Each scanner re-asserted the whole CE pair set every 2 s; with a fully fragmented pair set (descending PAs, one page per line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19 with clients wedged spinning in userspace. - pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern assembled at runtime with printf so the command's own cmdline cannot self-match), then start exactly one. - guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s) as a safety net. Configurable via env.

guest-diag: heal session-flush pin wipes with periodic full CE re-assert

2026-10-15T03:41:59+00:00

The host wipes its whole sema map when the previous client's last channel closes ('session flush (all channels closed)'). A pin batch that raced in for the new client (observed: the 1-page 0x120000000 pin emitted in the same second as the flush) was lost, and the scanner's content-gated delta never re-emitted the unchanged pairs - so the second client's CE pushes were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'), every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls), and the guest spun in uvm_tracker_wait_for_entry. Fix (scanner, guest side): - _ce_mem_pages: walk the whole vmalloc area (minus header off), not the size-derived prefix that stopped one page short of the write frontier. - _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus every fresh_map pass; emits every translatable pair content-independent, merged into contiguous runs. Zero-content pages are pinned on purpose: the pin carries the translation, the host reads live content. - missing pages retried with the negative PA cache popped (bounded 512/pass). Fix (host tool): - pin_gr_stream._flush: never dedupe single-page repeats of the CE window (0x120000000..0x121000000) so the re-asserts land. Live-validated in r145 (same-boot, no rebuild): mail green as 4th client, matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9), both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same sequences stalled. Headless smiley all-wrong (bad=131072) reproduced independently and is the next target (engine-side constant class).

guest-diag: resolve and stream the PMA scrubber buffer words (third-client scrub freeze)

2026-10-14T15:44:59+00:00

guest_pin_stream: find the scrubber channel buffer among the nvidia vmap areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when non-empty), walk its pages with kcore and emit '#scrub ,' (the finish payload at +0x6c004 and the host semaphore at +0x6c000). pin_gr_stream: handle '#scrub' lines and register the PAs through the sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can author the guest's scrub progress.

vgsp-sema: author the PMA scrubber channel's progress (third-client scrub freeze)

2026-10-14T15:39:30+00:00

The guest PMA scrubber (in-guest CeUtils CE channel, 4096 GPFIFO entries) waits on its channel buffer's finish payload (offset 0x6c004) after the third CUDA session's pages must be scrubbed; nothing in the emulation authored it, so cudaMalloc froze in scrubCheckAndWaitForSize. - channel_add: detect the 4096-entry scrubber channel and stash its USERD PA/ring geometry (the channel itself is left untouched: no rank change, no pump change - the colliding numeric-handle FREE may still unregister it, the geometry survives). - sema_scrubbuf_add debug param: register the finish/host-semaphore word PAs the guest scanner resolves (vmallocinfo + kcore walk), because the RM kernel VA space collides numerically with the client CE window in the global VA map and cannot be resolved through it. - svc_scrub_pass (service thread, per pass): from the channel's own USERD GPPut, track submitted scrub blocks across wraps, then author USERD GP_GET = put, host semaphore = put and finish payload = submitted. Forward-only, idempotent, zero writes while the scrubber idles. Live-validated mechanism (r137 VM): authoring the finish word at the kcore- resolved PA let the frozen victory_matmul leave scrubCheckAndWaitForSize and proceed to its launch.

guest-diag: content-validated delta emission for the CE push-buffer mem

2026-10-12T21:35:21+00:00

The one-shot CE mem emission pinned the whole 16 MB window including pages whose content was zero at the time; the host then read zeros for the other CE ranks' pushes. Sweep the mem every 2 s, skip zero pages, emit only new/changed {va,pa} pairs, and burst the Kcore PA cache every 30 s so a remap is re-emitted instead of shadowed by the cache.

guest-diag: channel-anchored CE pool resolution for the spinning tracking-sem case

2026-10-12T21:26:11+00:00

The second-client stall leaves exactly one live uvmt event: a tracking semaphore spinning with val=0 (the other channel/one-shot probes are inlined in this driver build and never fire). The sem is a member of a uvm_channel_t whose records sit in the sem's own window and carry the CE ring GPU VA (0x121010000+) plus back-to-back object handles; the channel's pool pointer is the first field at channel+0 (channel = sem-0x80), and the pool chain already satisfies _chain_mems_at once it is tried. The resolver now expands the candidate set from every ring-VA word's surrounding records and their targets before scoring. Verified live: the r134 spinning sem resolves to the CE push-buffer rm_mem with 413 pin runs, where the bare sem-window scan had produced none.

guest-diag: keep CE push-buffer latches on stale refresh; never fail freshness on an unknown page PA

2026-10-12T20:47:30+00:00

Two scanner defects seen in r131: (1) a resolved CE mem whose first vmap page could not be translated at store time stored page_pa=0, and the freshness check then failed on every probe, dropping the latch and re-resolving in a tight loop - and each failed re-resolution left the host with no pins for the channel at all; (2) the stale path discarded the latch before knowing the re-resolution would produce anything. Unknown PA now counts as fresh (the {cpu,gpu} object check is the evidence), and a stale refresh keeps the previous latch unless a new resolution actually emits runs.

guest-diag: read the trace snapshot non-blocking (no stuck tail processes)

2026-10-12T19:22:00+00:00

guest-diag: seed channel-event sets from the trace ring on client switch

2026-10-12T18:56:11+00:00

clear_trace() on a pid switch dropped exactly the one-shot uvmt/uvch events of the new client when they landed in the switch instant; the pipe reader then never saw them and the whole session ran with an empty sem set (r126: vecadd_loop sem=0 for minutes while its CE entries paid the 8 s grace). Scan the ring tail once per switch and seed the sets instead - stale kptrs that are also re-learned resolve to no content and are deferred by the candidate scoring.