open-iov.git/tools/guest-diag/pin_gr_stream.py, branch lottery

guest-diag: demand pinning for stalled CE VAs (event-independent hole filler)

2026-10-15T06:06:03+00:00

The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).

guest-diag: single-instance scanner start + full-re-assert backoff

2026-10-15T05:19:12+00:00

r148 ran THREE guest scanners (pin_gr_stream install_and_start retried while a partially-failed first attempt had already started one, and the pid-file kill missed it). Each scanner re-asserted the whole CE pair set every 2 s; with a fully fragmented pair set (descending PAs, one page per line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19 with clients wedged spinning in userspace. - pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern assembled at runtime with printf so the command's own cmdline cannot self-match), then start exactly one. - guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s) as a safety net. Configurable via env.

guest-diag: heal session-flush pin wipes with periodic full CE re-assert

2026-10-15T03:41:59+00:00

The host wipes its whole sema map when the previous client's last channel closes ('session flush (all channels closed)'). A pin batch that raced in for the new client (observed: the 1-page 0x120000000 pin emitted in the same second as the flush) was lost, and the scanner's content-gated delta never re-emitted the unchanged pairs - so the second client's CE pushes were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'), every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls), and the guest spun in uvm_tracker_wait_for_entry. Fix (scanner, guest side): - _ce_mem_pages: walk the whole vmalloc area (minus header off), not the size-derived prefix that stopped one page short of the write frontier. - _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus every fresh_map pass; emits every translatable pair content-independent, merged into contiguous runs. Zero-content pages are pinned on purpose: the pin carries the translation, the host reads live content. - missing pages retried with the negative PA cache popped (bounded 512/pass). Fix (host tool): - pin_gr_stream._flush: never dedupe single-page repeats of the CE window (0x120000000..0x121000000) so the re-asserts land. Live-validated in r145 (same-boot, no rebuild): mail green as 4th client, matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9), both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same sequences stalled. Headless smiley all-wrong (bad=131072) reproduced independently and is the next target (engine-side constant class).

guest-diag: resolve and stream the PMA scrubber buffer words (third-client scrub freeze)

2026-10-14T15:44:59+00:00

guest_pin_stream: find the scrubber channel buffer among the nvidia vmap areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when non-empty), walk its pages with kcore and emit '#scrub ,' (the finish payload at +0x6c004 and the host semaphore at +0x6c000). pin_gr_stream: handle '#scrub' lines and register the PAs through the sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can author the guest's scrub progress.

guest-diag: stream channel CPU-side GPUT on change (#put lines, ISSUE-0002 diag)

2026-10-09T18:07:11+00:00

spdx: add BSD-3-Clause identifiers to userspace/tools/scripts sources

2026-10-08T15:53:20+00:00

Sweep of the 155 tracked C/Python/shell files under userspace/ (excluding the MIT-licensed vendored userspace/extern/), tools/ and scripts/ that had no per-file license identifier. Each gains the minimal two-line header in the area's existing form: 'SPDX-License-Identifier: BSD-3-Clause' plus 'Copyright (c) 2026 Open-IOV contributors'; shebang and coding lines stay first. No other content touched. kernel/** is deliberately out of scope (recorded as a follow-up in CODESTD-REPORT-2026-10-08.md): its SPDX coverage and MODULE_LICENSE normalization need their own GPL-2.0 pass.

vgsp: host CUDA bridge + sema release recovery; greeneye simulator wave

2026-10-03T18:05:38+00:00

- kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean - tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests - tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode - debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding - STATE-CURRENT/docs updates

kernel: GR context-promote bisection modes 4-17 and server-managed-ctxbuf registry variant; complete vgsp rename; docs/tooling refresh

2026-10-02T19:13:06+00:00

- gsp_phys_booter: GR promote bisection modes 4-17 (two-phase zero-field VA, batched entries, hVirtMemory, bare VA, KMD SKIP_CTXBUFFER_ALLOC channel, old-interface GPU_INITIALIZE_CTX, FBMEM context buffer, UVM-faithful PA-only, PATCH/PM auxiliary promotes, server-managed no-promote path); PATCH/PM geometry queries and buffers; pre-boot SET_REGISTRY profile selection - gsp_phys_gpu: gsp_phys_rmcall_gr_promote clamp 0..17 and gsp_phys_srv_ctxbuf parameter; registry image variant with RMSetClientRMAllocatedCtxBuffer=0 - vgsp rename across kernel, scripts, tools and docs (module vgsp.ko, /sys/kernel/debug/vgsp/, vgsp-: log prefixes) - risc-v-gsp-rm-debugger user-facing rename and docs/CI refresh; guest-diag tool updates; README/SBOM/install/Makefile sync Live status: CE ladder and P6 guest-copy forwarding green; GR engine context commit still open (all promote shapes accepted or cleanly refused; the channel stays CTXNOTVALID) - see Handoff-vGPU-IO.md Addendum 21 (local).

guest-diag: retry the guest pin-scanner startup (a transient agent exec failure left a boot without pins and the client hung)

2026-09-30T14:52:12+00:00

guest-diag: stream uvbp pushbuffer kptrs for immediate pinning

2026-09-29T21:27:48+00:00

The first CE channel's push-buffer was only pinned ~1 minute after the client's failure: uvbp fired at t+4s but semfix's trace read (guest agent, every 5 loops) picked it up at t+70s, so the tracker wait for value 3 spun for 16 s. The guest scanner now emits '#pb 0x' from the uvbp trace lines as soon as they appear; the host mirrors them into the sem stream and semfix resolves/pins the push-buffer through the UVM chain on the next loop (sub-second).