| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The one-shot #sem kprobe event that drives the CE push-buffer chain
resolution can be lost (trace rotation, scanner restart, slab kptr reuse),
and the chain resolver then never learns a client - every GPFIFO entry of
its CE channels is grace-skipped and the client spins on its tracker
semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy
in r148; smiley as a later client in r147 took ~90 s to recover).
New demand channel, both ends here:
- pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N:
0x...' (the service logs each distinct unreadable push VA once), tracks
each sighting, and re-sends every 20 s for up to 5 min to
/tmp/guest_want_vas.txt in the guest (via the guest agent).
- guest_pin_stream: the main loop reads new VAs from that file and resolves
each by CONTENT against the nv_vm_map_pages vmap areas
(resolve_va_by_content: candidate leaf page at the VA's in-window offset,
+1/-1 neighbours for the header-page offset ambiguity, host-plausible
method-stream prefix). On a match the exact pair is emitted; only the
requested VA is pinned, so a wrong candidate cannot poison other VAs.
Also fix the scanner start: the kill-first pattern must be anchored
(^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy
shell's own cmdline and every start attempt killed its own shell before
launching the scanner (r149: no scanner for the whole boot).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
r148 ran THREE guest scanners (pin_gr_stream install_and_start retried
while a partially-failed first attempt had already started one, and the
pid-file kill missed it). Each scanner re-asserted the whole CE pair set
every 2 s; with a fully fragmented pair set (descending PAs, one page per
line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19
with clients wedged spinning in userspace.
- pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern
assembled at runtime with printf so the command's own cmdline cannot
self-match), then start exactly one.
- guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s
cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the
client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s)
as a safety net. Configurable via env.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The host wipes its whole sema map when the previous client's last channel
closes ('session flush (all channels closed)'). A pin batch that raced in
for the new client (observed: the 1-page 0x120000000 pin emitted in the
same second as the flush) was lost, and the scanner's content-gated delta
never re-emitted the unchanged pairs - so the second client's CE pushes
were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'),
every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls),
and the guest spun in uvm_tracker_wait_for_entry.
Fix (scanner, guest side):
- _ce_mem_pages: walk the whole vmalloc area (minus header off), not the
size-derived prefix that stopped one page short of the write frontier.
- _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus
every fresh_map pass; emits every translatable pair content-independent,
merged into contiguous runs. Zero-content pages are pinned on purpose:
the pin carries the translation, the host reads live content.
- missing pages retried with the negative PA cache popped (bounded 512/pass).
Fix (host tool):
- pin_gr_stream._flush: never dedupe single-page repeats of the CE window
(0x120000000..0x121000000) so the re-asserts land.
Live-validated in r145 (same-boot, no rebuild): mail green as 4th client,
matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9),
both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same
sequences stalled. Headless smiley all-wrong (bad=131072) reproduced
independently and is the next target (engine-side constant class).
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
scrub freeze)
guest_pin_stream: find the scrubber channel buffer among the nvidia vmap
areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when
non-empty), walk its pages with kcore and emit '#scrub <fin_pa>,<hsem_pa>'
(the finish payload at +0x6c004 and the host semaphore at +0x6c000).
pin_gr_stream: handle '#scrub' lines and register the PAs through the
sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can
author the guest's scrub progress.
|
| |
|
|
|
|
|
|
| |
The one-shot CE mem emission pinned the whole 16 MB window including
pages whose content was zero at the time; the host then read zeros for
the other CE ranks' pushes. Sweep the mem every 2 s, skip zero pages,
emit only new/changed {va,pa} pairs, and burst the Kcore PA cache every
30 s so a remap is re-emitted instead of shadowed by the cache.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
tracking-sem case
The second-client stall leaves exactly one live uvmt event: a tracking
semaphore spinning with val=0 (the other channel/one-shot probes are inlined
in this driver build and never fire). The sem is a member of a uvm_channel_t
whose records sit in the sem's own window and carry the CE ring GPU VA
(0x121010000+) plus back-to-back object handles; the channel's pool pointer is
the first field at channel+0 (channel = sem-0x80), and the pool chain already
satisfies _chain_mems_at once it is tried. The resolver now expands the
candidate set from every ring-VA word's surrounding records and their targets
before scoring. Verified live: the r134 spinning sem resolves to the CE
push-buffer rm_mem with 413 pin runs, where the bare sem-window scan had
produced none.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
freshness on an unknown page PA
Two scanner defects seen in r131: (1) a resolved CE mem whose first vmap
page could not be translated at store time stored page_pa=0, and the
freshness check then failed on every probe, dropping the latch and
re-resolving in a tight loop - and each failed re-resolution left the host
with no pins for the channel at all; (2) the stale path discarded the latch
before knowing the re-resolution would produce anything. Unknown PA now
counts as fresh (the {cpu,gpu} object check is the evidence), and a stale
refresh keeps the previous latch unless a new resolution actually emits
runs.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
clear_trace() on a pid switch dropped exactly the one-shot uvmt/uvch
events of the new client when they landed in the switch instant; the
pipe reader then never saw them and the whole session ran with an empty
sem set (r126: vecadd_loop sem=0 for minutes while its CE entries paid
the 8 s grace). Scan the ring tail once per switch and seed the sets
instead - stale kptrs that are also re-learned resolve to no content and
are deferred by the candidate scoring.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The kptr-keyed one-shot latches assumed object identity, but UVM frees and
re-creates CE push-buffer rm_mems and channels mid-session and the slab
reuses the same addresses; a resolution taken before such a move pinned
pages that read all-zero when the host used them, so every CE ring entry
stalled in the 8 s grace (r125: the live pushes sat at new vmap pages of
the same rm_mem kptr). Candidates are now scored by their window content
(a live push buffer is full of method streams, a freed range reads zeros)
and the best is pinned - all-zero candidate sets emit nothing instead of
poisoning the host map. A resolved mem is re-validated every ~2 s against
its own {cpu,gpu} fields and first-page PA; when they move, the latches
drop and the current truth is re-emitted.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
derive the CE vmap pairing per allocation
Two second-client zeros/stall countermeasures:
- svc_gr_data_fill now reports when a launch carries high (device-space)
argument pointers and none of them resolves through any trusted source.
svc_gr_request refuses to forward such a launch and leaves the capture
unconsumed; the pump holds the ring entry for one bounded window per
entry (gsp_sema_gr_grace_ms) and retries it, so a translation that
appears a pass later (scanner pin, published directory, learned pair)
is used instead of executing the kernel against unresolved pointers.
On window expiry the entry is consumed exactly as before (bounded
zeros, never a hang). Bounded per-session fill diagnostics name the
source that serves each candidate (ipin/pin/lat/map/vas/disc/win) and
dump the published vaspace walk results for misses.
- The CE push-buffer kcore chain resolver derives its vmap page offset
per allocation instead of using a fixed shift: the rm_mem size is read
from its real offset (570.144 layout) and paired with the vmallocinfo
area page count, so a leading header page is only skipped when it
actually exists. With an unreadable size the two candidate offsets are
content-probed with the host's method-stream plausibility test and an
ambiguous result keeps the status-quo pairing. This removes the
off-by-one that fed wrong push pages to the pump (content-rescue
crawl).
|
| |
|
|
|
|
|
|
|
|
|
| |
The +1 vmap pairing was content-validated on one boot's CE push-buffer
allocation (4097-page vmap: five rescue anchors at k+1), but r116/r117
show the leading header page is not universal: with the shift applied
client #1 stalls mid-run (rescues resume) where it was green without it.
Reverting the shift restores the previous behaviour; the kernel-side
repair from 16124929 stays (a content-validated rescue replaces the
injected pair, so a wrong pin is corrected once per page instead of
being re-rescued on every pass).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rescues repair injections
The CE push-buffer uvm_rm_mem is vmalloc'd as a 4097-page area: one
leading page plus the 4096-page (16 MB) GPU data window, so GPU data
page k lives at vmap page k+1. The kcore chain resolver paired page k
with vmap page k, pinning the adjacent push for every page; the pump
then read the wrong push, decode failed, and the host fell into the
content rescue which crawled one entry per ~2 s (r115: rescue_run=4202,
entry_va=0x12001b18c repeating). Verified against five independent
content-validated rescue anchors (k=20..28, all delta +1), and the vmap
area boundary confirms the extra page (nv_vm_map_pages 4097 pages).
Five fixes:
- resolve_mem_pins() takes page_off; the CE chain and uvbp push-buffer
paths pass mem_page_off() (1 when the vmap is exactly 16 MB + 1 page;
0 for direct-mapped mems like the semaphore pool).
- a content-validated rescue page now also replaces the injected-pin
entry (svc_rescue_try / svc_rescue_run_try), so a stale or wrong
injection can no longer shadow the validated page and re-run the
rescue on every pass.
Evidence: r115 rescue anchors va 0x120014000->pa 0x12a2e2000 .. va
0x12001c000->pa 0x12a2de000 all at vmap offset k+1; the candidate pages
decode as the adjacent push (wrong semaphore payload) before the fix.
|
| |
|
|
|
|
|
|
|
|
|
| |
Two robustness fixes for the sequential client battery. cdt_pid() now
picks the highest-pid CUDA candidate: preferring a name-prefix match
kept the scanner pinned to a previous client that was still tearing
down while the new client had already started pushing, so its first
pushes were unreadable, the release expectation drifted past the real
payloads and the stall followed. The CE push-buffer chain resolver now
also runs for uvch channel kptrs, so the pins do not depend on the uvmt
semaphore events alone.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
stall)
The second CUDA client's GPFIFO entries in the CE push-buffer window
(0x120000000) had no injected pin: that window is never mapped into the
client's pagemap, so the scanner's window scan cannot see it, and its
only supplier - the one-shot uvbp kprobe event - is raced by two trace
clears (the scanner's pid-switch clear and semfix's trace-tail clear).
When the event loses the race the entry reads fall back to the ~250 ms
content rescue, which is the second-client stall and the same
resolution-miss class as the ISSUE-0001 lottery.
The guest scanner now walks channel->pool->manager->pushbuffer->rm_mem
in kernel memory with kcore and pins the push-buffer pages directly (no
trace event to lose), and the trace reader's dedup marks reset on a
client switch so a reused kptr cannot suppress the new client's #pb.
Boot evidence (r112): rescue reject ipin=0 for va_page=0x120023000,
rescue scans 175 ms - 1 s each, want incrementing to 39. Smoke test of
this fix on the stalled VM: 478 pin runs (~7955 pages) streamed within
seconds, first va=0x120000000 -> pa=0x108090000. A bounded kernel log
now prints the first CE push-buffer pins of each session.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
registration; scanner streams anonymous high-VA buffers
Second-client wedge fix: a new GSP session re-registers the same channel
handles/ranks, but the per-rank one-shot ring/push-buffer discovery latches
stayed set from the previous session, so the new session's ring was never
re-autopinned (ring_auto=0 pb=0/0, nomap growing) and the guest spun in its
channel-creation wait. svc_chan_rearm_rank() clears the latches on every
channel registration; all post-fix cycles launch the second client.
Scanner: guest_pin_stream.py now streams the client's anonymous high-VA
mappings (cudaMalloc buffers were skipped by the old len(parts)<6 check),
so the host can learn their pages.
Diagnostics: PT-window lines, per-window first words, post-OK qmdwin
w48..59. Remaining: the fill-time svc_ipin_lookup miss for the second
client's a/b/c pins (they exist in the pin log but are not seen at fill
time) - next session.
|
| |
|
|
|
|
|
|
|
| |
Two failure modes found when the scanner detects clients by device fd:
a transient boot nvidia-smi was picked and its exited pagemap read back
EOF, and scan_window advanced by got=0 forever (busy spin, no stream);
and non-client nvidia fd holders must never be committed to. Break on
got<=0 in scan_window (scan_range already did), skip nvidia-smi/
persistenced/Xorg, and prefer /dev/nvidia-uvm holders.
|
| |
|
|
|
|
|
|
|
|
| |
The guest pin scanner only tracked processes whose comm matched
CLIENT_PREFIXES (cdt,va,victory). An arbitrary CUDA binary (e.g.
/tmp/smiley) was invisible: no pages streamed, no VA->PA pins injected,
the service could not read its pushes and the client hung in
cudaMalloc/uvm_tracker_wait. Detect compute clients by the CUDA device
they hold open (/dev/nvidia-uvm preferred, then any /dev/nvidia*), with
the prefix match kept as the first-choice path.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(ISSUE-0002)
svc_chan_userd now prefers the layout-derived CE USERD
(ring_pa + 0x10000 in the ring aperture) and accepts it while idle, so
a second session cannot latch onto the registered pool-relative offset
when that alias holds a DIFFERENT rank's stale USERD: r40 c2 rank0 read
rank3's stale 1/1 at FB 0xf0000 and the pump never followed rank0's own
GP_PUT (guest GPUT mapping for gpFifoGpuVa proves ring+0x10000).
guest_pin_stream.va_to_pa drops translations outside the registered
guest-RAM windows so BAR1-phys addresses (0x70000xxxxx) are no longer
injected as SYS pins; r40 rank2's ring resolved to the bogus
pa=0x70000c0000 ap=SYS and read 66/66.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
#put (ISSUE-0002 diag)
CE channels register gpfifo=0x121010000+0x30000*rank, outside the
GR-only window [0x200200000,0x200400000) the #put resolver searched, so
no #put line ever fired for the CE channels the second session stalls
on. Resolve channel_info from both windows with layout validation, keep
re-resolving the channel after the pool mem has resolved, reset the
#put map on client switch, and add bounded resolver diagnostics to the
guest log.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Sweep of the 155 tracked C/Python/shell files under userspace/ (excluding
the MIT-licensed vendored userspace/extern/), tools/ and scripts/ that had
no per-file license identifier. Each gains the minimal two-line header in
the area's existing form: 'SPDX-License-Identifier: BSD-3-Clause' plus
'Copyright (c) 2026 Open-IOV contributors'; shebang and coding lines stay
first. No other content touched. kernel/** is deliberately out of scope
(recorded as a follow-up in CODESTD-REPORT-2026-10-08.md): its SPDX
coverage and MODULE_LICENSE normalization need their own GPL-2.0 pass.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ring+userd) + GR ring userd-derived candidates
guest_pin_stream.py now resolves translations entirely from guest truth via
/proc/kcore + a manual page-table walk (no QMP races): the CE semaphore pool
and pushbuffer memory from the UVM rm_mem structures, and each channel's
GPFIFO ring + GPPUT from the tracking semaphore's embedding uvm_channel_t
(channel_info located by its gpFifoGpuVa value). This removes the watcher
pipeline's flakiness on the cudaMalloc stall class (missing pool/pushbuffer
pins).
gsp_sema_service.c: svc_gr_ring_autopin tries the guest-registered USERD
memdesc base as a ring candidate (the fixed FB convention only matches when
the guest's VRAM allocator happens to reproduce the old layout), plus an
optional (gsp_sema_dbg) one-shot FB scan + raw-word diagnostics when every
candidate fails.
tools/cuda-ab/p2: self-checking arbitrary-kernel acceptance suite (10
programs: sequential launches, shapes, params, smem/GEMM, streams, malloc
churn, printf+error, FMA benchmark, mixed, managed probe) with host oracle
expectations and a guest runner.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
second-client stall race)
The guest scanner's TraceReader tailed the last 256 KB of the trace ring
every 0.3 s. A one-shot uvmt sem= event could age out of that window
before the next poll (the client's spin floods the ring), so a second
client in the same VM was never discovered: semfix re-pinned the previous
client's freed sem while the new client's push page stayed untranslated
(host nomap, drain held, guest spins forever).
- guest_pin_stream.py: consume /sys/kernel/tracing/trace_pipe instead
(each event delivered exactly once; no ring-window race), with
reconnect on read error. The old ring-tail read is kept only as
semfix's fallback.
- clear the trace ring on every client switch so the fallback cannot
re-learn the previous client's stale events.
|
| |
|
|
|
|
|
|
| |
- kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean
- tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests
- tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode
- debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding
- STATE-CURRENT/docs updates
|
| |
|
|
|
|
|
|
|
| |
The DtoH staging buffer lives in the client's high-VA /dev/zero|nvidia
mappings (e.g. devptr+0x3c00100), outside the scanner's fixed CE/GR
windows; only the host's pagemap read can see it. The guest scanner
now walks those VMAs and streams their present pages so the module
pins them. run_cycle.sh learns the sema_dbg master switch (auto-on
when any watch knob is set).
|
| | |
|
| |
|
|
|
|
|
| |
The GR-window pin scanner matched comm=="cdt" exactly, so runs using the
cuda_drv_test2 binary (comm "cdt2") never streamed their mappings: the
GR push pages stayed unresolvable and the client stalled in context
creation. Match the cdt prefix instead.
|
| |
|
|
| |
watcher reaping, wider CE stream window
|
| |
|
|
|
|
|
|
|
|
| |
The first CE channel's push-buffer was only pinned ~1 minute after the
client's failure: uvbp fired at t+4s but semfix's trace read (guest agent,
every 5 loops) picked it up at t+70s, so the tracker wait for value 3 spun
for 16 s. The guest scanner now emits '#pb 0x<kptr>' from the uvbp trace
lines as soon as they appear; the host mirrors them into the sem stream and
semfix resolves/pins the push-buffer through the UVM chain on the next loop
(sub-second).
|
| |
|
|
|
|
|
|
|
|
| |
The failing wait (uvtw) is uvm_tracker_wait_for_entry for value 3 on a CE
channel whose tracking sem stayed at payload 1 for 14 s: its pushes'
release values 2..3 were never applied because the CE push-buffer pages
(0x120000000 + 0x100000*rank) are outside the GR window and were only
found by the content sweep, late and often wrong (false=3065). Scan both
windows in the guest so the host gets guest-truth translations for the CE
pushes as well.
|
| |
|
|
|
|
|
|
|
|
|
| |
- the trace tail (subprocess + 256KB read) ran inside the scan loop and
blocked it for seconds exactly when the client's GR window appeared;
moved to a background thread whose queue the loop drains.
- scan interval 250ms -> 100ms.
- track_waits sampler also dumps 512 raw bytes at page+0xe00 each sample:
the per-slot reads disagreed with the bulk dump within one sample
because host writes land between the two, and the raw dump is the only
trustworthy view of the slot layout.
|
| |
|
|
|
|
|
|
|
|
|
| |
counter-page delta
- rel-apply/publish now log the first 32/48 outcomes (written, skipped with
current value, no slot, read failure) so the live counter page can be
matched against host writes.
- guest pin scanner logs the counter page's pagemap state each loop and adds
it to the delta set explicitly, so the page is pinned the moment it is
mapped even if the window scan misses it.
|
| |
|
|
| |
GR GPFIFO rings)
|
| |
|
|
| |
last release VA; larger push log budget
|
| |
|
|
|
|
|
|
|
| |
The first deltas of a 15k-page window took tens of seconds to arrive while
the client was mapping under load, so the pins still landed after the
client's cuCtxCreate deadline. The scanner now reads only the first 64 KB
of every 2 MB region - the pushes sit at the per-channel allocation heads -
which cuts the per-loop page-table work ~50x, and it logs scan timing so
the latency stays visible.
|
|
|
The guest-agent exec path is too slow under driver-init load (a 15k-page
window scan took 17-35 s to complete), so GR window pins landed after the
channels' entries had been consumed and after the client's cuCtxCreate
timeout. The guest now scans /proc/<pid>/pagemap locally and streams only
delta runs over vsock (no agent on the hot path); the host listener pins
them as they arrive. The scanner writes a pidfile so restarts do not kill
the new instance.
|
