| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
large mixed-space map no longer burns them
r508/r509 witness: 'page directory discovery: not found (base=0x0 ap=ram
cands=39621 best=0/8)' during the very first client - eight RAM walks
spent on a stride subsample of the boot-scoped map, which holds pairs from
several VA spaces and (per the code's own comment) no single directory
reproduces. By the time the push-buffer pairs - the strongest samples -
formed, the attempts budget was gone; every later client's high argument
pointers stayed unresolvable and the boot ended in the held-launch class.
Only a small map (one session's own pairs) still falls back; a larger one
waits for the pairs and keeps the attempts.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
the r351 retry-forward consumes it
svc_gr_hold_bound_expired() re-armed the deadline on the very call that
detected expiry and then compared jiffies against the NEW deadline - the
function could never return true. A data-stalled launch whose arguments
never resolve therefore held its entry forever (r508 witness: matmul plus
ten sibling channels in 'GR hold ... data args unresolved' with zero
GR-STALL-hold-bound lines for 10+ minutes; the designed one retry-forward
plus consume never ran). The expiry is now one-shot: push the deadline,
latch gr_hold_bound_hit (snapshot-persisted) for the second call site of
the same pass, return true; the r351 retry-forward block consumes the
latch and forces its single best-effort forward, and the NEXT expiry cycle
(gr_hold_exp_logged now set) falls through to the historical consume with
the software completion - a never-resolving launch now costs ~6 s and the
client sees a stale-data FAIL instead of a hang.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
stream order in the pump itself
The streams order their H2D copies before the kernel that consumes them,
but the channels register in handle order (the RM's GR channel early, the
copy-CE channels later) and the pump walked them in that order; a pass that
decoded the GR launch before the CE channel's entries ran let the forward
workqueue execute the launch's copy-in against pre-copy data (r480-sep:
copy-in 115 ms before its stream's H2D soft copies; r481 proved no host
semaphore links the two). The CE copies execute on the same service
thread, so consuming the CE entries first fixes the inversion without any
cross-channel hold. Stable partition of the pass snapshot; relative order
within each class is preserved.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
holding (r507 matmul lost a pin-late 1 MB UVA copy and verified FAIL)
r506w's attempt-count latch tripped after 8 fast passes (~100 ms) while
the legitimate pin-late retry needs 0.5-2 s (r507 witness: memcopy
dst=0x71365ee02200 src=0x71365e500000 bytes=1048576 rc=-2 skipped; matmul
completed with STATUS FAIL instead of hang). The first failure of a VA now
arms a full 10000 ms window during which the op holds its entry exactly as
before; only after a full bound with no success does the op stop forcing
the hold. A success clears the record (fresh window on a later failure).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
release routing
Two independent defects turned one unresolvable virtual op into a
client-wide hang (r505 multikernel, r506 matmul: 'client at DEVICE
forever' while the pump ground its stream at ~10 s per push):
- an op that keeps failing -ENOENT re-runs on every decode pass and
re-set svc.op_tr_pending each time, so its push held for the full
SVC_OPDEF_HOLD_MS opdef bound every pass (r506 witness: the RM's
channel-progress zeroing at 0x2000c0xxx, 1913 failed attempts, 65 ring
entries in ~90 s). New per-VA failure latch: after 8 failures the op
still attempts its write but stops forcing the hold; the entry consumes
and its withheld release applies - exactly the outcome the expired
bound already sanctioned, minus the 10 s tax.
- svc_rel_apply_launch routed EVERY release through the exact-slot
counter fallback (release_pa ?: counter_pa), which writes (and
forward-only-skips against) the channel's registered completion slot -
a different word. The tracker releases to 0x20480fef0 were skipped
against the polluted PUT counter (cur=96) and lost: rel-apply
h=0x5c000059 pa=0x10809def0 pay=1 cur=96 -> skip. The fallback is now
restricted to counter VAs; generic releases are forward-only by value
(payload > fresh cur) so a decode blinded by a stale landing can no
longer drop them.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two fixes for the r505 multikernel livelock (opdef held=18359, expired=61,
client at DEVICE forever while the pump looped on ring slot 1):
- after the bounded opdef hold expires and svc_opdef_flush delivers the
withheld releases, the slot's ops are permanently skipped - but the
decode-time bookkeeping had stored slot_ops_complete=0 (op_tr_pending was
set when it ran), so every pin-batch rewind replay of the slot re-decoded
it, re-set op_tr_pending and re-held the entry forever. Mark the slot
complete on flush so a replay suppresses the ops (entry_replay) and the
entry consumes normally.
- the per-slot op arrays are SVC_GPFIFO_MAX_BITS (1024) but the X/GL
channel's ring is 8192 slots: bound every slot-index write/read
(entry_ident is zeroed past the array size, so the bookkeeping skips).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(+3 fixes)
The r496g blast registration of every COPY_SERVER_PDES directory let a
stale kernel directory answer another client's VA in the walker's
first-success order (the r459 stale-directory class) - measured as a CUDA
client hanging after DEVICE. Candidates are now parked and registered only
when a walk of the client's own content-validated ring VA through them
reproduces that ring's PA; the r496f vidmem sweep remains as a fallback.
Also in this batch:
- svc_chan_userd's VRAM shape scan now prefers a block with work PENDING
(get < put) and then the one nearest the registered memdesc offset: on a
channel re-registration the stale previous USERD still validates (it
holds the same put/entry values) and the old first-match adoption made
the pump write GP_GET into the dead block (r504 second DMA init).
- The USERD/FB scan and the CE caps byte-1 BL bits (r496i) are unchanged;
TU104_CE_CAPS1 = 0x03 needs the CUDA battery re-verification.
|
| |
|
|
|
|
|
|
|
|
|
| |
resolves the rest
The registered COPY_SERVER_PDES directory walks the ring VA correctly
(trace: L0 -> L1 -> L2 idx 9 -> PD0), but the linear window pin installed
around the ring shadows it in svc_lookup_trusted (pin_lookup_any returns
the win-flagged entry first), so the pushbuffer/tracker reads kept using
the wrong noncontiguous guess. Pin only the content-validated ring page;
every other VA of the client now resolves through the walk.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The X/GL channel's pushbuffer pages and progress semaphore live in a
kernel-RM internal VAS whose page directory is never published (fn 54 is
UVM/external-VAS only), so no translation source can resolve them: the
tracker push reads zeros/garbage and the driver's DMA sanity poll (which
waits on the progress-tracker semaphore the GPU writes when it consumes the
tracker push) never completes.
With the ring content-validated, try each 4 KB page of the vidmem shadow as
a walk root and register the first whose walk reproduces the ring
VA -> ring PA translation. The walker then resolves every other VA of the
same client - pushbuffer pages, the tracker push, the progress semaphore.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
capture holds for their entries
The driver's __nvPushTestPushBuffer writes one zero dword (NVA06F_DMA_NOP)
and waits for GP_GET to follow GP_PUT. The method-stream plausibility gate
rejects an all-zero buffer (it parses as zero methods), so the entry fell
into the unreadable/rescue path and the GR no-capture hold - the DMA sanity
test could never pass. For channels whose ring was found by the RAM content
scan (kernel-RM sysmem 3D channels), model them as consume-and-apply-
releases: accept an all-zero NOP push, and skip the no-capture hold (these
pushes are not compute launches; holding them 6 s per entry cannot help).
|
| |
|
|
|
|
|
|
|
| |
ring
The ring-ownership check rejected every entry VA >= ring_va, but the driver
writes TWO GPFIFO entries per kickoff - the main pushbuffer BELOW the ring
and the progress tracker ABOVE it (nvidia-push-priv.h). Accept VAs anywhere
in the 2 MB allocation window.
|
| |
|
|
|
|
|
| |
The pump's !userd_ok path continues past the end-of-pass write-back, so the
GR RAM ring scan restarted from slice 0 every pass and never reached the
ring page beyond the first 512 MB. Persist gr_ram_cursor/gr_ram_passes (and
the one-shot log flag) directly in the scan.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
USERD shape scan
With the GR classification fixed the pump reaches the X driver's 3D channel
for the first time, but both of its structures are outside every existing
translation source: the CPU RM maps the pushbuffer's NV01_MEMORY_SYSTEM
surface locally (no SET_PAGE_DIRECTORY, no rm_mem record carrying the ring
VA - the targeted resolve scans all of guest RAM and finds nothing), and the
driver's USERD GP_PUT write lands in the emulated VRAM at a BAR1-window
offset (observed: fb 0x80000) that is neither the registered memdesc base
(0x10000, pool-relative) nor ring-derived.
Two content scans close the gap, both validated so a wrong hit costs a
failed probe, not a bad pin:
- svc_gr_ram_ring_scan: the driver's kickoff writes GPFIFO entries whose
pushbuffer VAs sit just BELOW the ring VA in the same allocation (main
pushbuffer first, GPFIFO after - nvidia-push-priv.h). A page whose first
two 8-byte entries both name such VAs with sane lengths is the ring; the
ring page is pinned validated and the 2 MB allocation window as guesses
(the ring entry and pushbuffer reads resolve through it). Sliced 512 MB
per pass, cursor-resumed, 16 slices max.
- svc_userd_fb_scan (svc_chan_userd GR fallback): scan the VRAM for the
USERD shape (GP_PUT at +0x8c in [1..entries], GP_GET at +0x88 <= put) and
validate each hit against the ring: every entry the put claims must
decode to a pushbuffer VA in the ring's allocation window. A foreign
channel's USERD cannot pass that test. 8 bounded retries (the kickoff
lands a few passes after registration).
|
| |
|
|
| |
(its op may have already run on an earlier gate-less decode), force exactly one op re-run on the first pass whose gate finally passes - the pass at which the dependency just became installed. Without it the once-only done bitmap suppressed the op and the release applied while the staging still held the pre-dependency bytes (streams_events s=2: the D2H read dc2 before the vadd_into wrote it, ran early, was skipped when the wait passed, and the guest drained a2+b2 instead of a2+b2+c1)
|
| |
|
|
| |
pushes (872/888B) dump; runtime-enable before the client run to avoid boot noise
|
| |
|
|
| |
and bound to launch-sized pushes (512..8192 bytes) so the launch method stream dumps cleanly
|
| |
|
|
| |
large GR-class pushes (default 0 off, no behaviour change) so the launch push's acquire/release method groups are visible verbatim; needed to prove which ordering mechanism the driver relies on for async-stream H2D -> kernel dependencies (streams_events st2 vadd copy-in ran ~115 ms before its stream's H2D soft copies)
|
| |
|
|
| |
past a push whose content was still unwritten when its entry was consumed. The r477 failing probe run showed the reuse hazard directly: the channel registered with a residual USERD put (4) and ring slots 1-3 all-zero; the pump consumed them (the r470 hold only delays) and the end-of-pass publish then wrote val=2 while the client's own counter was still 1 - the D2H wait was satisfied before the copy ran and the readback drained a reused staging buffer holding the PREVIOUS copy's bytes (h_c[2] = c0's data). New per-channel gr_bad_put (lowest all-zero GR slot + 1, persisted in the pump write-back): set when a zero-content GR entry is held/consumed, cleared when a decode reads non-zero content (the rewinds re-read entries constantly, so it heals within passes); svc_counter_publish clamps val to it with a bounded pub-clamp witness.
|
| |
|
|
| |
entry written - previous session's or the client's own slot reuse - but the push content not yet landed) is held by the m164 GR no-capture window instead of being consumed as a no-op, so the end-of-pass counter publish cannot advance the guest's completion word before the real push is read (streams_events: h_c[2] = c0 because the c2 D2H completion was published 4.5 s before its copy ran). Deliberately narrower than the reverted r469 plausibility verdict: compute-class GR pushes legitimately fail the CE-shaped method matcher, so only an entirely zero page counts as unwritten (r469's broad hold delayed GR config pushes and triggered the engine-fetch-NONE wedge on the first client).
|
| |
|
|
|
|
| |
entry whose push content is implausible (unwritten / previous-epoch page, no QMD word seen) was consumed as a no-op and the end-of-pass counter publish then advanced the guest's completion word while the real push had never been read - the client drained a staging buffer that still held the PREVIOUS copy's bytes (probe run: BUF s=2 got=c0's 0,0.75,1.5.. instead of c2's 3.0,5.25,7.5..; log: 'pub-adv h=0x5c000062 val=4 cur=1' at 210741.947, 4.5 s before the c2 D2H copy executed at 210746.371, with the ring slots decoded as zero-content pushes at 210736). The implausible-content read now feeds the m164 GR no-capture hold (bounded gsp_sema_gr_nocap_hold_ms window, rehold forces a fresh capture): the entry stays unconsumed, the publish gate (last_get != put) suppresses the early completion, and the retry reads the client's real push once it lands. Plausible fallback reads (validated pin / displaced pair) clear the flag."
This reverts commit 349ee341539a0c1e9e3be11563251589eb42f8a8.
|
| |
|
|
| |
whose push content is implausible (unwritten / previous-epoch page, no QMD word seen) was consumed as a no-op and the end-of-pass counter publish then advanced the guest's completion word while the real push had never been read - the client drained a staging buffer that still held the PREVIOUS copy's bytes (probe run: BUF s=2 got=c0's 0,0.75,1.5.. instead of c2's 3.0,5.25,7.5..; log: 'pub-adv h=0x5c000062 val=4 cur=1' at 210741.947, 4.5 s before the c2 D2H copy executed at 210746.371, with the ring slots decoded as zero-content pushes at 210736). The implausible-content read now feeds the m164 GR no-capture hold (bounded gsp_sema_gr_nocap_hold_ms window, rehold forces a fresh capture): the entry stays unconsumed, the publish gate (last_get != put) suppresses the early completion, and the retry reads the client's real push once it lands. Plausible fallback reads (validated pin / displaced pair) clear the flag.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
with dead clients' registrations and every new client after the 8th got no slot
Evidence (r466 boot, one battery): 23 distinct 'VA space 0x... page directory
... entries=4' registrations and ZERO 'VA space ... dropped' lines. The
guest driver does not always tear a client's VA space down (client exit,
crash, or a switch the RM emulation does not forward), so after eight
registrations svc.vas[] had no free slot: gsp_sema_service_vaspace_add
silently refused the new directory (the 'if (vs)' guard), svc_walk_vaspaces
kept walking only the dead clients' directories, and every walk of the new
client's VAs failed -> its CE copies returned rc=-2 forever ('sema-optrace
... rc=-2' x8151 in the r466 log; the 10th+ client of the boot: streams2
bad=32737, streams_events no output, smiley hung, and the same class on
r463/r465). This is the 'no launch stalls / no mid-boot degradation'
blocker.
Fix:
- svc_vaspace_add evicts the least-recently-registered entry when the
table is full instead of refusing (logged), and stamps every
registration;
- svc_walk_vaspaces orders the snapshot by registration stamp, newest
first, so a live client's directory is the authority and a stale one
(whose guest memory the driver has since reused) can only answer when
the live directory cannot.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
and ignored the LINE_COUNT value the driver encodes in the SAME count=2 method group at 0x418
Raw push evidence (sema-op2d-raw, new bounded diagnostic):
raw 17000520 7c7f0204 ... 20018104 00001000 20018105 00001000
20028106 00001000 00000100 20048100 7d7e 40c00000
The group at 0x418 carries count=2: v0 = LINE_LENGTH_IN (4096),
v1 = LINE_COUNT (0x100 = 256 rows). The CE decode stored only v0, so
st.line_count stayed 0, the 2D-copy builder took the single-row path
(m->bytes = 4096) and a 4 MB cudaMemcpy2D delivered exactly ONE page
per 1 MB block. memset2d_probe (new per-phase client) shows the exact
shape: per-MB ok=4096 / bad=1044480 in all three phases, total
12,505,200 of 12,582,912 - deterministic across boots (r453/r454/r459
identical). The GRCE variant has always decoded the count=2 form
(gr_line_count = v1); the CE path did not.
Fix: CE_M_LINE_LENGTH_IN stores v1 into st.line_count when count >= 2,
so the existing r420c contiguous-run logic (pitch == line => flat run
of rows*line) finally sees the row count. The same decode feeds the
fill path (cudaMemset2D).
Also adds the bounded sema-op2d-raw witness (raw method stream of every
pitch-shaped copy push) that produced the evidence, and a per-phase
memset2d_probe client under tools/cuda-ab/.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(not on opdef_clear)
The r456 change kept the snapshot alive across opdef APPLY, but the r457
mode-C run proved a second freer: svc_opdef_clear's FINISH path (entry
consume via svc_replay_note) freed the b1 H2D's snapshot while the copy
was still owed. The pin-batch rewind then REPLAYED the entry, the
re-capture read the driver's REFILLED staging and the deferred execution
wrote b2's bytes into db1.
Evidence (r457, mode C, owner 0x5c00005c/1/0627e693d05aab3f):
191607.816388 copy snapshot captured dst=0x72e318c20000
src=0x72e31c400000 first4=00000000 3e800000 3f000000 3f400000 (b1)
191608.798838 copy snapshot captured <same key>
first4=00000000 3d800000 3e000000 3e400000 (b2 - refilled staging)
191611.404931 sema-bigcopy dst_va=0x72e318c20000 src_va=0x72e31c400000
dst16_after=000000000000803d0000003e0000403e (b2 landed in db1)
The op never succeeded (rc=-2 every pass), so the only free that could
have dropped the first snapshot is the opdef_clear owner free.
New lifecycle: capture once -> free on the copy's own successful
consumption -> free on ring-slot reuse with a new push hash -> free when
a re-decode skips an already-done op -> 30 s TTL. svc_copy_snap_free_owner
keeps its rationale but is no longer called (the r444 cross-entry
poisoning stays closed by the owner key + eviction).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
operand-shift root cause)
The fail4 pipeline-order snapshot (captured when a device-dst H2D copy is
held for its operand pins) was freed by svc_opdef_clear on the APPLY path
on every re-decode pass of the held entry. The retried copy therefore
re-captured from the driver's REFILLED ping-pong staging: the a1 H2D of
the r455 streams2_probe D run executed ~3 s after its push carrying a2's
bytes (snapshot first4 flipped 00000000 3f000000.. -> 00000000 3e000000..
across retries; sema-bigcopy dst16_after=000000000000003e0000803e0000c03e
proves the a2 pattern landed in da1's page; the vadd computed a2+b1 and
the readback faithfully delivered it - there is no separate read-side
landing bug for this class).
- svc_opdef_clear: keep the snapshot on the APPLY path (the pass's own
op execution, which consumes the snapshot on success, follows the
apply); every genuine consume path (FINISH/FLUSH/DROP/STALE/COUNTED/
SWEEP/STOP) and the TTL still free it.
- svc_copy_snap_capture: evict same-ring-slot snapshots with a different
push hash (slot reuse) so the surviving snapshot can never serve a
fresh push.
- svc_run_ops: an op skipped via ops_done drops any snapshot a later
re-decode re-captured for it (the leak the r444 APPLY-free used to
sweep).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(channel, ring entry, push-content hash), not (dst,src,total) alone
The wedge_ab deterministic reproducer (bad = mutated-element count) root
cause: a held CE copy's source snapshot was captured for one ring entry,
left behind when that entry's copy executed through a path that never
consumed it (hardware-primary forwarded copy, or a post-execution
re-decode whose copy was re-held), and then consumed by a LATER entry's
copy that happens to share the (dst,src,total) triple - the client's
cudaMemcpy released early with the PREVIOUS step's bytes in the device
buffer, and the next GR launch computed on stale inputs (r441/r442/r443
evidence: step N read step N-1 content; sema-bigcopy src16 shows the fresh
live source at execution time while dst16_after shows the old snapshot's
bytes).
Owner identity is (ch->h_object, entry_idx, svc_pass_hash); the pass hash
separates a ring slot reused by a fresh push from the entry that captured
the snapshot. svc_opdef_clear() additionally frees any snapshot still
owned by a consumed entry, so no snapshot can outlive its entry.
|
| |
|
|
|
|
|
| |
Every copy op logs dst/src/bytes/rows/dpitch/spitch beside the decoded
line_length/line_count/pitches, so the memset2d readback forensics can tell
'one row per push' from 'N rows in one op' and whether the pitch methods
arrived. Bounded by the existing opadd budget.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
victory_memset2d failed deterministically (bad ~= 12.5M of 12.58M) because
its pitched operations carry PITCH == LINE_LENGTH (cudaMallocPitch returned
pitch 4096 for width 4096) and the r402 2D decode only entered row mode
when a pitch was WIDER than the row. The flat path then ran ONE row
(LINE_LENGTH bytes) per op, so every cudaMemcpy2D readback and H2D upload
delivered only its first row and every cudaMemset2D-sized region beyond it
kept stale content - the 'deterministic readback loss' (45K of 12.58M
correct ~= the rows the single runs happened to cover).
LINE_COUNT is now authoritative for the flat case too: pitch == row length
(or no pitch method) means CONTIGUOUS rows, so the single run covers
rows*line bytes (element-exact when the line divides evenly, else byte
exact). A pitch wider than the row keeps the r402 row-wise mode; 1D ops are
unchanged. Both the fill (cudaMemset2D/fill) and the copy (cudaMemcpy2D)
decode are fixed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
redirected
With gsp_phys_rmcall_gr_rel_mod=1 (release0 -> module semaphore, the
ordering fix for the engine-release-before-copy-out race) the engine never
writes the guest's own QMD release word, so the host must author it on
EVERY backend outcome. The hold's success/fail edges do; a launch with no
deferred release records (no hold engaged) fell through and its client hung
on a word nothing would ever write - the r414 saxpy f0 cur=5 stall.
The backend now reports qmd_rel_redirected in the copy result whenever it
points release0 at the module semaphore (rel_mod, or no targetable guest
release page). svc_fwd_gr_work folds that into the existing
gr_rel_unobserved latch, which makes the bounded outcome check author the
word on done (success or failure) - the same forward-only write the hold
edges perform. rel_mod=0 paths are unchanged (the engine still authors the
word and the latch is only set on an unresolved release page).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The r419 soak reproduced the user's mid-boot degradation with the wedge fix
in tree: victory_matmul printed bad=65536 while its launch was held by the
booter's WEDGE-QUIET window - the client was released ~115 s before the
post-hold accept. Root cause: two sema-service deadlines are shorter than
the deliberate quiet hold and fire under it, authoring the launch's
completion release (and applying its recorded releases) while the engine has
not run and the copy-out has not happened:
- the 12 s GR release hold (SVC_FWD_GR_HOLD_TIMEOUT_MS) expiry
(GR-STALL-fwd-hold ... releases + QMD word software-applied), and
- the 3 s per-launch consumption bound (gsp_phys_gr_hold_max_ms) firing
the busy-defer consume-with-software-release path, whose inline apply
authors the capture's QMD release (GR-REL-AUTHOR src=hold).
Evidence (r419 live): push be94 captured 131333.687, WEDGE-QUIET holds
40 s + 80 s until 131345.598, accept 131346.552 - while the sema hold
expired at 131230.978 (GR-STALL-fwd-hold waited_ms=12032) and the inline
apply authored va=0x204027f7c pay=6 at 131341.734 (h=0x5c00001f). The
client's verify completed on the authored word and read stale data.
Fix: the booter publishes gsp_phys_gr_quiet_hold_active() exactly around
the quiet msleep (bounded by construction). The sema service pauses under
it: svc_fwd_hold_expired() pushes the GR deadline (bounded by a 600 s
defence-in-depth cap, announced once per hold as GR-HOLD-PAUSE);
svc_gr_hold_bound_expired() pushes the per-launch bound
(GR-HOLD-BOUND-PAUSE); and the busy-defer one-shot window re-holds while
the flag is set so the consume-with-sw-release path cannot run during a
quiet hold. The held launch is then forwarded and copied out after the
hold, and its completion is authored only on the hold's done edge - after
the data write-back.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
+ explicit param desc
With the gate off (the mission cycle recipe omitted sema_wait_gate=1),
the c2 readback's C56F ACQ_CIRC_GEQ pay=4 ran at 106617.279 with cur=2 -
before the c2-add launch was even submitted (106617.411) - and read the
not-yet-produced buffer (all zeros). With the gate on the same readback
holds until the scale channel's progress counter reaches 4 (sema-wait:
hold ... cur=2 waited_ms=353 -> pass ... cur=4) and the ordering is
correct. The holds are bounded (sema_wait_hold_ms, default 4000) and
fail open loudly, so the pre-r336 pass-through remains available with
sema_wait_gate=0 for an A/B.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(only consumed entries are stale)
The r406 inplace_scale repro root cause: the H2D copy's operand
translation was unresolved (memcopy rc=-2) and the entry was held with
its release withheld (correct); a pin-batch rewind then re-opened lower
indices (83) while the held entry (91) sat ahead, and the unconditional
svc_opdef_drop_stale treated 91 as an already-consumed entry and applied
its withheld release at 104955.751. The copy itself only executed at
104957.418 - the client raced past its blocking cudaMemcpy, launched,
the launch's copy-in read the stale buffer (previous run's data) and
the copy landed after the launch's copy-out: wrong results with a
healthy-looking 32/1032 changed and no classifier firing.
An entry is only 'passed' if the ring consumed it (slot_done_epoch);
gate the stale drop on svc_replay_seen so a rewind can never deliver a
still-held entry's release early.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two additions for the memset2d/memcpy_roundtrip and readback classes:
1. Pitched transfers. The CE decoder now captures LINE_COUNT (0x41c),
PITCH_IN (0x410) and PITCH_OUT (0x414); a fill/copy whose LINE_COUNT is
> 1 and whose destination pitch exceeds the row length builds a row-wise
op (svc_fill/svc_copy are thin wrappers around the existing 1D bodies,
advancing dst by dpitch and src by spitch per row). Each row keeps the
whole per-page resolution/ledger/guard machinery, and a failed row's
retry re-runs already-written rows with the same pattern (idempotent).
A flat transfer carries pitch == row length (or no pitch method) and
keeps the historical single-run path unchanged. Bounded 'sema-2d' row
names every 2D op build.
Note: the current victory_memset2d traffic reaches the engine as GR
kernels plus per-row 4 KB CE copies, so this capability does not yet
trigger for that client - it is the correct decoder for the CE-class 2D
shape (driver channel_utils.c memset2d emits LINE_COUNT/PITCH).
2. DST-DIVERGE witness: when a copy destination resolves from the write
ledger, a read-only resolver probe cross-checks the answer and logs
(bounded, gsp_sema_dbg) when the writer's page differs from the live
resolver's - the 'writer follows the ledger while the reader holds
another page' shape behind the failed D2H staging class (streams2 c1
zeros, smiley pass failures). Zero rows fired on r403, so the ledger is
not the active divergence for that boot; the witness stays for the next
failing capture.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two fixes for the streams2 / multi-stream ordering class, both verified on
fresh boot r401 with the acquire gate enabled:
1. sema_wait_hold_ms (new module param, default 4000 ms) replaces the
hardcoded 100 ms acquire-wait bound in svc_run_ops. The old bound is
shorter than one emulated GR launch completion (0.5-1.6 s observed), so
a D2H readback's ACQ_CIRC_GEQ on the producer channel's progress
counter (op=3 pay=4 on 0x20400ffd0) gave up and ran BEFORE the scale
launch wrote d_c2; the client then verified pre-scale data
(bad=32767). r401 log: the same entry now holds ~1.2 s across
cur=1->4 and runs after GUEST GR LAUNCH OK (copy-back optrace
92315.594824 with dst16_after = the post-scale bytes). Fail-open is
preserved: a genuinely unsatisfiable acquire still expires, just late
enough that the dependency's work has landed (late-but-correct over
early-and-wrong).
2. svc_gr_seen: a session-scoped open-addressed (push_va, hash) table
behind svc_gr_cand_push_seen(). The 4-slot candidate ring + 8-VA
history evicted a real capture within ~12 later captures, so a true
replay of an already-forwarded launch was re-captured and re-executed
(r399: the same nine launch pushes re-submitted every ~10-11 s for
minutes). Cleared in svc_session_flush because a new client's
identical-content pushes restart their release counters - a
boot-scoped table would silently drop them (LAUNCH-SKIP).
Also adds tools/cuda-ab/streams2d.cu: streams2 with per-array/per-4KB-block
diff reporting plus the device pointer dump, which localised the remaining
failure to the D2H staging write path (our staging-page resolution vs the
guest's view).
|
| |
|
|
|
|
|
|
| |
r398 evidence: the memset's inline write executed twice for the same push
(85634.101649 first, 85642.940808 replay 70 us before the D2H readback,
zeroing the accumulator). 76 inline writes happened between the two - the
64-slot table had evicted the key. 8192 slots give a ~100x margin over the
busiest observed burst.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
hash, dst, bytes)
The first take gated on !replay, which broke pushes whose first pass was
untrusted (their bytes are only authored on the later trusted pass) and
hung the second client of the boot. Key the inline write by
(push_va, push content hash, dst, bytes): a replay of the same push is a
genuine duplicate and is skipped, a new push (its release payload makes the
hash unique) always runs. Table is session-scoped (cleared by
svc_session_flush), 64 slots.
Root cause retained: victory_reduce's cudaMemset(d_sum,0,8) is a GRCE
LOAD_INLINE_DATA write; every re-decode of that push re-ran it and
re-zeroed the accumulator after the launch's copy-out (the sum=0 class).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A REPLAYED ring entry must not re-run its LOAD_INLINE_DATA writes. The
fill/copy ops were already gated by allow_ops (cleared for entry_replay),
but the inline-write gate was 'trusted || allow_ops' and a replay is
trusted by construction, so every re-decode of an already-processed push
re-wrote its inline bytes into the vidmem shadow.
victory_reduce's cudaMemset(d_sum, 0, 8) is exactly such an inline write.
Observed on r396: 'sema-grce: inline write dst=0x7845ef000000 ->
pa=0x7c00000 ap=vidmem bytes=8 data8=0000000000000000' fired 86 us before
the D2H readback of the same VA - zeroing the accumulator the launch had
just written (OUT-PAGE-POST val8=0000008000080000 = the correct sum) so
the client read zeros (sum=0, bad=1). The same replay rewrote the page
repeatedly (482 inline writes/boot). The r383 replay-capture path only
needs to CAPTURE the launch; the inline bytes were authored on the first
(non-replay) pass.
|
| |
|
|
|
|
|
|
|
|
| |
witnesses
Rule in/out a stale or duplicate vgpu->bars[1] shadow between the launch
copy-out (svc_fwd_req.vgpu) and the D2H reader (svc_copy's vgpu): the sum=0
runs show the write landing (OUT-PAGE-POST val8 = the sum) and the reader
of the same ledger page reading zeros 8 ms later with no logged writer in
between. Diagnostic only.
|
| |
|
|
|
|
|
|
|
|
| |
The GR copy-out now prints, per changed page, the landing-table resolution
of the VA (the exact source the D2H reader's ledger-first branch uses) and
its first 8 bytes beside the write target hpa - a launch/readback
disagreement (write landed on another page/aperture than the reader
resolves) becomes one line instead of a forensics session.
Diagnostic only (bounded by the existing copywit budget).
|
| |
|
|
|
|
|
|
|
| |
svc_op_translate returns 0 on success; the new svc_fill_dst_page_pa tested
it without negation, so every page answerable only by the walk/scan
returned -ENOENT and the fill truncated (r391 boot: FILL-TRUNC for the
page-table writes at va=0x...a00200 -> the client stalled at init).
Also demand the unresolved page through the pin channel before failing
(stalled VA row -> the userspace stream pushes it to the guest scanner).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
guest panic root cause
The 383620-byte context-init zero fill (FILL-NOREMAP class) resolved only
its FIRST page through the pin (va=0x7c36c7400000 -> pa=0x131b63000, the
pin stream's first run = 29 pages) and then wrote the whole 375 KiB
contiguously past it. The pin stream shows the VA's pages are independent
guest allocations with split runs (0x131b63000 / 0x131a80000 / 0x128340000),
so the contiguous write covered unrelated memory - on r390 exactly the
guest kernel's page-table pages 0x131ba1000/0x131ba2000/0x131bab000. The
direct map grew three 4 KiB holes and the guest died with 'Fatal exception
in interrupt' (unmap_page_range / cpuacct_account_field / psi_task_change
faults on physmap addresses).
svc_copy has always translated virtual operands per 4 KB page ('a linear PA
walk past the first page reads the wrong memory'); svc_fill never did.
New svc_fill_dst_page_pa() resolves every page of a virtual fill through
the same authority order (fresh pin wins and repairs the landing table,
then the landing table, then the resolver chain); the SYS write loop clamps
each chunk to the page boundary and writes at that page's own landing; a
page with no resolution fails the fill loudly (FILL-TRUNC) instead of
writing through a contiguous guess. The >= 64 KiB landing-table pre-store
now records only the contiguous VID (shadow) case; SYS landings are stored
per page as they are written.
Also extends the r384 page-table-shape refusal to every page of a virtual
zero fill (the r390 fill's first page was the client's buffer, so the
first-page-only check could not see the PTE pages it later covered).
Evidence: /tmp/opencode/r390-live.log (FILL-EXEC + sema-opadd lines),
/tmp/opencode/r390-guestram.raw (three zeroed PTE pages, CR3 walk),
pin log 12-pin_gr_stream-r390.log line 789 (the split runs).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
FILL-EXEC witness
- The r384 boot held a large fill whose walk resolution was VID (the
emulated vidmem shadow) - the hold was pointless overhead there and
the re-decode re-resolves (the resolver is non-deterministic), so the
hold now applies only when the fill would land in guest RAM (SYS
aperture), the class that zeroed the guest's page-table pages. Shadow
resolutions stay immediate.
- FILL-EXEC witness (bounded 64/boot): va, pa, aperture, provenance and
pattern of every >= 4 KiB virtual fill execution - the decisive line
for the boot-corruption forensics; the large zero fills were invisible
until now.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
fills + page-table-shape refusal for zero fills (the boot-corruption lottery root-caused to zero writes landing on guest page-table pages)
Forensics from the r383 boot (guest panic at t=216 during victory_strided):
the guest kernel's direct map had TWO 2 MiB holes (phys 0x105000000 and
0x131800000) because their PTE pages at gpa 0x12d3c2000/0x3000 - two
adjacent 4 KiB page-table pages - were found all-zero. Kernel objects
(the kprobe nodes UVM registers at module load) lived in those regions;
the first access after the zeroing faulted (Oops #1 get_kprobe from
uvm_init_entry), cascading to the t=230 panic. The zeroing is an 8 KiB
all-zero write, the exact shape of the guest's window page-table clear
(LINE_LENGTH=8192, value=0), and the emulation's virtual fill resolver
is demonstrably non-deterministic: four executions of that same 8 KiB
fill resolved to four different guest pages (0x126af8000, 0x12d8e4000,
0x12f20c000, 0x1187fb000 - every one refused by the below-floor guard).
The below-floor guard covers window VAs only; a >= 1 TiB VA fill that
resolves through the walk/scan had no guard at all.
Two nets in svc_fill, both default-on (sema_fill_guard=1) and both
fail-open/histed:
- net 1: a >= 64 KiB virtual fill whose only authority is the walk/scan
(no pin, no landing-table entry) is held through a bounded 800 ms
window (sema_fill_guard_ms) while its VA is demanded through the pin
channel (the userspace pin stream injects the guest-pagemap truth), so
the retried decode resolves through a real pin. Mirrors the m169/r378
copy guards; releases with the historical behaviour after the bound.
- net 2: a ZERO fill about to land in guest RAM on a page whose first 64
qwords are all present+NX+in-range PTEs is REFUSED loudly
(FILL-PT-REFUSED) - a page table page is never a legitimate zero-fill
destination. Pin-backed resolutions are exempt (guest pagemap truth).
New witnesses: FILL-HOLD (held fills) and FILL-PT-REFUSED (refused
page-table targets), both bounded.
|
| |
|
|
|
|
|
|
|
| |
surface) + write guards; REPLAY-CAPTURE closes the silent launch-drop hole; CE operand misses feed the pin-demand channel
- bounce copy-out now records staged_hpa[bi] per launch and skips: (a) recs whose hpa != staged slot content (alias/leftover), (b) pages the engine did not change (~955 of 968 per boot launch); the remaining changed pages re-validate via gsp_sema_gr_write_guard before the 4 KiB memcpy. Removes the blanket 968-page write-back that was the remaining boot-corruption lottery vector (guest slab freelist clobbered by a foreign page write).
- new bounded WGUARD-SKIP guards: gr_write_guard (pin-first re-resolution) on copy-out, release-word zero/restore (rel_va plumbed through the submit), stale-recover-zero, counter release; svc_write_guard pin-agreement on 4/8-byte SYS/FB service writes; svc_copy SYS-dst hold on a fresh-pin contradiction.
- svc_qmd_accum replay hole: a trusted replay of an unseen push now captures (REPLAY-CAPTURE) instead of silently dropping; known push -> LAUNCH-SKIP witness; capture history ring prevents double-capture.
- svc_op_translate miss path emits the stalled-VA demand signal for CE operands.
|
| |
|
|
| |
data-fill page is logged as a stalled VA (the userspace pin stream watches dmesg for that line and pushes the VA to the in-guest scanner's demand file), so the bounded launch hold can resume on real pins instead of waiting out its bound (the r345 wedge)
|
| |
|
|
| |
disagreeing pin beats a recorded landing on both dst (write) and src (read) at top level and per page, repairing the entry; closes the stale-entry write (guest-corruption) and read (wrong-result) classes
|
| |
|
|
| |
recorded as the write authority poisons every later write of the VA (r375 boot slab GPF in kmem_cache_alloc at guest t=77s before any client); the table is fed by writes and fresh pins only
|
| |
|
|
| |
copy sources (top-level and per page), and a resolver answer is recorded so launch/read paths converge; the engine output can no longer be read through a different page/aperture than it was written
|
| |
|
|
| |
the sum=0 class shows the engine output written through one path and read through the other
|
| |
|
|
| |
vs readback disagreement (sum=0 class); budget 512
|
