summaryrefslogtreecommitdiffstats
path: root/kernel
Commit message (Collapse)AuthorAge
* vgsp-sema: prefer the client's page tables over a disagreeing pool pin ↵Arthur Hanson Rasmusson10 hours
| | | | | | | | | | | | | | | | (moved-but-readable pool) The pool resolver trusted an injected pin after a read-back proof. That proof does not catch a pool page that MOVED: the old page is still readable, so the release write lands on the old page while the client polls the new one. Observed: 'pending release starved va=0x12100000x tries=64' for ~45 s with the stale pin (pa=0x120b30000) readable, until the fresh pin (pa=0x130e9d000) arrived; the client's tracker never woke and its init hung (multikernel battery timeout, copies stuck rc=-2). The client's own page tables are the GPU/UVM ground truth for where the tracker reads, so when a structural walk resolves a different page than the pin (and it is not a degenerate identity), the walk wins - both sources are read-back proven at the exact offset.
* vgsp-sema: chunk the discovery scan's guest reads (64 KB) - the rebuild was ↵Arthur Hanson Rasmusson11 hours
| | | | | | | | | | | | | starving the pump The full-RAM discovery scan read one 4 KB page per vfio_dma_rw call: 1.5 M guest reads over the 6 GB range, ~10-11 s per rebuild, and the rebuild runs every 10 s - so the pump got roughly one pass per rebuild. A client whose resolutions were dropped by a session flush then waited minutes for the scan to re-find its pages (the observed post-flush stall: data-VA copies returning -ENOENT with the pump counters frozen). Reading 64 KB chunks amortizes the iommu walk + pin over 16 pages; the per-page detection logic is unchanged.
* Revert "vgsp-sema: keep the learned VA map across the session flush ↵Arthur Hanson Rasmusson11 hours
| | | | | | (re-compact, not wipe)" This reverts commit 1a49c7eb69c03f58bc51939828de222a2ac42fea.
* vgsp-sema: keep the learned VA map across the session flush (re-compact, not ↵Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | | | | | | wipe) The flush fires when the channel table momentarily empties - a live client's UVM channel cycling triggers it - and wiping the map left the client's already-resolved data VAs unresolvable until a rebuild scan re-found them, which it may not. Evidence: a mid-battery boot where multikernel's operands went NO ENTRY 40+ s after a flush (the rebuild restored 14-31 K entries without them); the pump stalled for minutes and the next clients read back whole-page garbage. The map is a cache with a merge-on-rebuild contract, so re-compact it (dedup + sort) and keep it as the merge base; reused-VA correctness stays with the ledger landing, the pin-wins rule and the fresh scan. Also: push_battery.sh now pushes inplace_scale (the battery list had it, the push list did not).
* r526 THE XDIFF ROOT CAUSE: the LOAD_INLINE_DATA destination cursor must ↵Arthur Hanson Rasmusson15 hours
| | | | advance even when the r391f once-only gate suppresses the write. A re-decode pass that skipped the FIRST run of a split upload left the cursor at the first run's destination, so the SECOND run's data landed at offset 0 (the deterministic tgemm X[0..1]<-X[2..3] 8-byte overwrite, present in every pass); the gate suppresses the write, not the method stream's positional semantics
* r525 the BINAPI-07 oracle object under the adopted internal client: a ↵Arthur Hanson Rasmusson16 hours
| | | | runtime root alloc answers NV_ERR_INVALID_CLIENT (0x23) because the server's internal client is adopted, so allocate the class-0x2081 child directly under the internal subdevice with a candidate handle list
* r524 the BINAPI-07 oracle client: build the captured real-client sequence on ↵Arthur Hanson Rasmusson16 hours
| | | | the live session (root -> device 0xcaf00000 -> subdevice 0xcaf00004 -> class-0x2081 child 0xcaf00099) and send the query as that client, so the server dispatches from a genuine BINAPI object
* r523 the BINAPI-07 oracle reply parse: an alloc with paramsSize=0 echoes the ↵Arthur Hanson Rasmusson16 hours
| | | | 32-byte request header (hObject at +8, status at +16) - the backend alloc itself succeeded (rpc_result=0) in r522
* r522 the BINAPI-07 oracle parent fix: the guest allocates class 0x2081 under ↵Arthur Hanson Rasmusson16 hours
| | | | the SUBDEVICE (0xbeef0004 is the class-0x2080 child), so mirror that on the backend - the device parent answered NV_ERR_INVALID_OBJECT_PARENT (0x36)
* r521 the BINAPI-07 oracle step 2: allocate a real class-0x2081 object on the ↵Arthur Hanson Rasmusson16 hours
| | | | live session (GSP_RM_ALLOC as the internal client/device parent) and send the query to it - the subdevice attempt answered NV_ERR_NOT_SUPPORTED because the server dispatches by the object's class
* r519b the BINAPI-07 forward gets its own parameter (gsp_rm_binapi07_fwd): ↵Arthur Hanson Rasmusson16 hours
| | | | forwarding only 0x20810107 keeps the P3 whitelist off so the boot is perturbed by exactly the one control under study
* r519 the BINAPI 0x20810107 experiment: forward the guest GL core's ↵Arthur Hanson Rasmusson16 hours
| | | | reserved-interface query to the live GSP session (internal subdevice translation) and log the server's own status/bytes as evidence; scoped to the observed 2-byte shape, fail-open to the emulated generic reply
* r510gl implement DUP_OBJECT (fn 21) with the NVOS55 params echo: the EGL/GLX ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | | | | | MakeCurrent wall The guest driver dups memory/VASpace objects into the GL client at EGL MakeCurrent time; the generic-SUCCESS default replied with an EMPTY payload and the driver, expecting rpc_dup_object_v03_00 (NVOS55_PARAMETERS {hClient, hParent, hObject, hClientSrc, hObjectSrc, flags, status}) echoed back, failed the call and tore the whole EGL session down. r510 guest witness: 'RPC UNKNOWN(21) seq=0 len=60 -> generic SUCCESS' immediately followed by the session's FREE cascade and eglMakeCurrent -> EGL_BAD_ALLOC (0x3003); EGL device enumeration and eglInitialize were already green (n=3 devices, NVIDIA 1.5). The handler echoes the params with status=NV_OK (the emulation keeps no per-object state for these classes).
* r510 keep the page-directory discovery attempts for coherent evidence: a ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | | | large mixed-space map no longer burns them r508/r509 witness: 'page directory discovery: not found (base=0x0 ap=ram cands=39621 best=0/8)' during the very first client - eight RAM walks spent on a stride subsample of the boot-scoped map, which holds pairs from several VA spaces and (per the code's own comment) no single directory reproduces. By the time the push-buffer pairs - the strongest samples - formed, the attempts budget was gone; every later client's high argument pointers stayed unresolvable and the boot ended in the held-launch class. Only a small map (one session's own pairs) still falls back; a larger one waits for the pairs and keeps the attempts.
* r509 the GR data-hold bound can report expiry: one-shot verdict + latch, and ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | | | | | | | the r351 retry-forward consumes it svc_gr_hold_bound_expired() re-armed the deadline on the very call that detected expiry and then compared jiffies against the NEW deadline - the function could never return true. A data-stalled launch whose arguments never resolve therefore held its entry forever (r508 witness: matmul plus ten sibling channels in 'GR hold ... data args unresolved' with zero GR-STALL-hold-bound lines for 10+ minutes; the designed one retry-forward plus consume never ran). The expiry is now one-shot: push the deadline, latch gr_hold_bound_hit (snapshot-persisted) for the second call site of the same pass, return true; the r351 retry-forward block consumes the latch and forces its single best-effort forward, and the NEXT expiry cycle (gr_hold_exp_logged now set) falls through to the historical consume with the software completion - a never-resolving launch now costs ~6 s and the client sees a stale-data FAIL instead of a hang.
* r507o walk the CE channels before the GR channels: enforce the client's ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | | | | stream order in the pump itself The streams order their H2D copies before the kernel that consumes them, but the channels register in handle order (the RM's GR channel early, the copy-CE channels later) and the pump walked them in that order; a pass that decoded the GR launch before the CE channel's entries ran let the forward workqueue execute the launch's copy-in against pre-copy data (r480-sep: copy-in 115 ms before its stream's H2D soft copies; r481 proved no host semaphore links the two). The CE copies execute on the same service thread, so consuming the CE entries first fixes the inversion without any cross-channel hold. Stable partition of the pass snapshot; relative order within each class is preserved.
* r507a op-failure latch is time-based: a full opdef window before it stops ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | holding (r507 matmul lost a pin-late 1 MB UVA copy and verified FAIL) r506w's attempt-count latch tripped after 8 fast passes (~100 ms) while the legitimate pin-late retry needs 0.5-2 s (r507 witness: memcopy dst=0x71365ee02200 src=0x71365e500000 bytes=1048576 rc=-2 skipped; matmul completed with STATUS FAIL instead of hang). The first failure of a VA now arms a full 10000 ms window during which the op holds its entry exactly as before; only after a full bound with no success does the op stop forcing the hold. A success clears the record (fresh window on a later failure).
* r506w kill the r505/r506 post-DEVICE grind: bounded op-failure latch + exact ↵Arthur Hanson Rasmusson20 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | release routing Two independent defects turned one unresolvable virtual op into a client-wide hang (r505 multikernel, r506 matmul: 'client at DEVICE forever' while the pump ground its stream at ~10 s per push): - an op that keeps failing -ENOENT re-runs on every decode pass and re-set svc.op_tr_pending each time, so its push held for the full SVC_OPDEF_HOLD_MS opdef bound every pass (r506 witness: the RM's channel-progress zeroing at 0x2000c0xxx, 1913 failed attempts, 65 ring entries in ~90 s). New per-VA failure latch: after 8 failures the op still attempts its write but stops forcing the hold; the entry consumes and its withheld release applies - exactly the outcome the expired bound already sanctioned, minus the 10 s tax. - svc_rel_apply_launch routed EVERY release through the exact-slot counter fallback (release_pa ?: counter_pa), which writes (and forward-only-skips against) the channel's registered completion slot - a different word. The tracker releases to 0x20480fef0 were skipped against the polluted PUT counter (cur=96) and lost: rel-apply h=0x5c000059 pa=0x10809def0 pay=1 cur=96 -> skip. The fallback is now restricted to counter VAs; generic releases are forward-only by value (payload > fresh cur) so a decode blinded by a stale landing can no longer drop them.
* r505b opdef flush marks the slot complete; bound the slot arrays for large ringsArthur Hanson Rasmusson24 hours
| | | | | | | | | | | | | | | Two fixes for the r505 multikernel livelock (opdef held=18359, expired=61, client at DEVICE forever while the pump looped on ring slot 1): - after the bounded opdef hold expires and svc_opdef_flush delivers the withheld releases, the slot's ops are permanently skipped - but the decode-time bookkeeping had stored slot_ops_complete=0 (op_tr_pending was set when it ran), so every pin-batch rewind replay of the slot re-decoded it, re-set op_tr_pending and re-held the entry forever. Mark the slot complete on flush so a replay suppresses the ops (entry_replay) and the entry consumes normally. - the per-slot op arrays are SVC_GPFIFO_MAX_BITS (1024) but the X/GL channel's ring is 8192 slots: bound every slot-index write/read (entry_ident is zeroed past the array size, so the bookkeeping skips).
* r496j VAS candidates are registered only after a content-verified ring walk ↵Arthur Hanson Rasmusson24 hours
| | | | | | | | | | | | | | | | | | | | (+3 fixes) The r496g blast registration of every COPY_SERVER_PDES directory let a stale kernel directory answer another client's VA in the walker's first-success order (the r459 stale-directory class) - measured as a CUDA client hanging after DEVICE. Candidates are now parked and registered only when a walk of the client's own content-validated ring VA through them reproduces that ring's PA; the r496f vidmem sweep remains as a fallback. Also in this batch: - svc_chan_userd's VRAM shape scan now prefers a block with work PENDING (get < put) and then the one nearest the registered memdesc offset: on a channel re-registration the stale previous USERD still validates (it holds the same put/entry values) and the old first-match adoption made the pump write GP_GET into the dead block (r504 second DMA init). - The USERD/FB scan and the CE caps byte-1 BL bits (r496i) are unchanged; TU104_CE_CAPS1 = 0x03 needs the CUDA battery re-verification.
* r496i CE caps byte 1: the blocklinear support bits the 2D copy path requiresArthur Hanson Rasmusson25 hours
| | | | | | | | | nvCEChooseEngine skips every CE without CE_SUPPORTS_NONPIPELINED_BL (caps byte 1, 0x01) and, in the preferred pass, PIPELINED_BL (0x02). Our emulated CE caps left byte 1 zero, so every CE failed the filter and the X driver aborted with 'Failed to find a suitable copy object' - the wall after the DMA pushbuffer test started passing (r503). Turing CEs are fully capable (bug 200288104 case (a)): byte1 = 0x03.
* r496h do not pin a linear window for content-scanned rings; the VAS walk ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | | | resolves the rest The registered COPY_SERVER_PDES directory walks the ring VA correctly (trace: L0 -> L1 -> L2 idx 9 -> PD0), but the linear window pin installed around the ring shadows it in svc_lookup_trusted (pin_lookup_any returns the win-flagged entry first), so the pushbuffer/tracker reads kept using the wrong noncontiguous guess. Pin only the content-validated ring page; every other VA of the client now resolves through the walk.
* r496g register the internal VAS page directory from COPY_SERVER_RESERVED_PDESArthur Hanson Rasmusson25 hours
| | | | | | | | | | | | NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES (0x90f10106) is the kernel RM publishing its internal VAS page-table chain - level[0..] with phys addresses, sizes and page shifts (the handler already parsed and logged it, then discarded it). Register level[0] as the VAS root with the progress service so the walker translates every VA of that client, which is exactly what the X/GL channel needs: its pushbuffer pages and progress semaphore have no other translation source (fn-54 is external-VAS/UVM only, the rm_mem content scan finds nothing for these VAs, and the sysmem/linear window guesses are wrong for the noncontiguous surface).
* r496f discover the kernel-RM client VAS page directory by verificationArthur Hanson Rasmusson25 hours
| | | | | | | | | | | | | | The X/GL channel's pushbuffer pages and progress semaphore live in a kernel-RM internal VAS whose page directory is never published (fn 54 is UVM/external-VAS only), so no translation source can resolve them: the tracker push reads zeros/garbage and the driver's DMA sanity poll (which waits on the progress-tracker semaphore the GPU writes when it consumes the tracker push) never completes. With the ring content-validated, try each 4 KB page of the vidmem shadow as a walk root and register the first whose walk reproduces the ring VA -> ring PA translation. The walker then resolves every other VA of the same client - pushbuffer pages, the tracker push, the progress semaphore.
* r496e the X NOP push: all-zero accepted on content-scanned channels; no ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | | | | | capture holds for their entries The driver's __nvPushTestPushBuffer writes one zero dword (NVA06F_DMA_NOP) and waits for GP_GET to follow GP_PUT. The method-stream plausibility gate rejects an all-zero buffer (it parses as zero methods), so the entry fell into the unreadable/rescue path and the GR no-capture hold - the DMA sanity test could never pass. For channels whose ring was found by the RAM content scan (kernel-RM sysmem 3D channels), model them as consume-and-apply- releases: accept an all-zero NOP push, and skip the no-capture hold (these pushes are not compute launches; holding them 6 s per entry cannot help).
* r496d USERD shape-scan validation: the progress-tracker entry sits ABOVE the ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | ring The ring-ownership check rejected every entry VA >= ring_va, but the driver writes TWO GPFIFO entries per kickoff - the main pushbuffer BELOW the ring and the progress tracker ABOVE it (nvidia-push-priv.h). Accept VAs anywhere in the 2 MB allocation window.
* r496c RAM-scan cursor must persist through the stalled-userd continueArthur Hanson Rasmusson26 hours
| | | | | | | The pump's !userd_ok path continues past the end-of-pass write-back, so the GR RAM ring scan restarted from slice 0 every pass and never reached the ring page beyond the first 512 MB. Persist gr_ram_cursor/gr_ram_passes (and the one-shot log flag) directly in the scan.
* r496b the X/GL channel bring-up: sysmem ring discovery by content + VRAM ↵Arthur Hanson Rasmusson26 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | USERD shape scan With the GR classification fixed the pump reaches the X driver's 3D channel for the first time, but both of its structures are outside every existing translation source: the CPU RM maps the pushbuffer's NV01_MEMORY_SYSTEM surface locally (no SET_PAGE_DIRECTORY, no rm_mem record carrying the ring VA - the targeted resolve scans all of guest RAM and finds nothing), and the driver's USERD GP_PUT write lands in the emulated VRAM at a BAR1-window offset (observed: fb 0x80000) that is neither the registered memdesc base (0x10000, pool-relative) nor ring-derived. Two content scans close the gap, both validated so a wrong hit costs a failed probe, not a bad pin: - svc_gr_ram_ring_scan: the driver's kickoff writes GPFIFO entries whose pushbuffer VAs sit just BELOW the ring VA in the same allocation (main pushbuffer first, GPFIFO after - nvidia-push-priv.h). A page whose first two 8-byte entries both name such VAs with sane lengths is the ring; the ring page is pinned validated and the 2 MB allocation window as guesses (the ring entry and pushbuffer reads resolve through it). Sliced 512 MB per pass, cursor-resumed, 16 slices max. - svc_userd_fb_scan (svc_chan_userd GR fallback): scan the VRAM for the USERD shape (GP_PUT at +0x8c in [1..entries], GP_GET at +0x88 <= put) and validate each hit against the ring: every entry the put claims must decode to a pushbuffer VA in the ring's allocation window. A foreign channel's USERD cannot pass that test. 8 bounded retries (the kickoff lands a few passes after registration).
* r494 the GL 3D-channel classification: NV2080_ENGINE_TYPE_GRAPHICS (1) is a ↵Arthur Hanson Rasmusson26 hours
| | | | | | | | | | | | | | | | | | | | | GR channel With the real-GSP KGR snapshot in place the X driver's 3D-engine selection succeeds; the next wall was 'Failed to initialize DMA' / 'Failed to allocate push buffer' - the X driver's pushbuffer NOP test never saw its GPFIFO entry consumed. Root: the wire's NV_CHANNEL_ALLOC_PARAMS engineType carries NV2080_ENGINE_TYPE_* (NULL=0, GR0/GRAPHICS=1, COPY0=9); CUDA's kernel-RM GR channels arrive with NULL=0 and the progress service's convention is 'engine_type == 0 means GR', so the X/GL 3D channel's real GR0 value 1 was misclassified as a CE - the CE path then demanded a ring translation the channel does not have (rrc=-2 'norrng'), never probed the USERD, and the pump never executed a single entry (submissions=0 at the 10 s timeout). Normalize GR0..GR7 (1..8) to 0 at channel registration; every other engine id is >= COPY0 (9) so no CE can be captured. Also fixes the KGR snapshot max-size constant: KGR_GET_INFO is 3712 bytes (8 engines x 58 x 8), which the 1664-byte cap silently dropped - the one snapshot the engine selection actually needs.
* r493 THE 3D-ENGINE WALL: serve the live GSP's own KGR static data to the guestArthur Hanson Rasmusson26 hours
| | | | | | | | | | | | | | | | | | | | | The X driver's nvSelect3DClass -> nv3dAllocDevice maps GR_GET_INFO index SM_VERSION (0x0c) through an SM-version -> SPA-version table; the emulated KGR_GET_INFO fill never set index 0x0c, so SM_VERSION read back as SM_VERSION_NONE (0) and every Xorg/EGL session died at 'Failed to select a 3D engine' (the class list and the engine list were never the problem: the X log carries neither 'Failed to determine graphics capabilities' nor 'Failed to determine engines list'). The physical backend already asks the LIVE GSP the real client's whole KGR query set at mode-35 boot (gsp_phys_rmcall_user_lane_kgr_queries) and threw the replies away. Retain them (generation-keyed, invalidated with the backend static cache) and serve the identical bytes in the guest-side GSP_RM_CONTROL dispatch for the same commands, so the guest's GL/3D stack reads the device's true GR static values (SM_VERSION 0x75, warps, caps, topology, ctx-buffer tables) instead of fabricated approximations. Falls through to the emulated handlers unchanged when no capture exists (backend not READY, other boot mode, cmd not captured) and never answers with more bytes than the guest requested.
* r489b CONTINUATION_RECORD from the driver is one-way data (the tail of a ↵Arthur Hanson Rasmusson27 hours
| | | | multi-frame request): stop replying to it. The guess reply injected an extra fn-71 record the driver drained as an unexpected event (0x47) and corrupted our own large-reply continuation framing
* r489 the proper large-reply framing for GSP_RM_CONTROL: replies beyond the ↵Arthur Hanson Rasmusson27 hours
| | | | driver's pRpc->maxRpcSize (65080; pinned by the guest's own remainingSize 0x940 arithmetic for the 67448-byte PERF_VPSTATES_INFO reply) are now sent with the GSP continuation-record protocol - first frame length = maxRpcSize, then NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD frames of up to maxRpcSize-32 bytes each, exactly like the host GSP's _rpcWriteContinuationEntries. Replaces the r487 clamp (which produced a record the guest refused, and a size the guest then waited on fn-71 continuations for). Root: the GL VASpace construction and 3D engine selection die on this control's reply
* r487b tighten the RM_CONTROL reply clamp margin: the guest rejected ↵Arthur Hanson Rasmusson27 hours
| | | | rpc.length 65528 (its element header > 8 bytes); clamp to 16*4096 - 512 so any element-header size fits the bound
* r487 THE GL/VULKAN VASPACE WALL: clamp RM_CONTROL replies to the 64 KB wire ↵Arthur Hanson Rasmusson28 hours
| | | | record bound (16 x 4096; guest GspMsgQueueReceiveStatus rejects hdr + rpc.length beyond it - 'Incorrect message length 67448', rpcRecvPoll 0x3a, NV_ERR_INVALID_PARAMETER). The stock driver sends PERF_VPSTATES_INFO (0x2080a0a4, paramsSize 67392) during Xorg VASpace construction; the generic reply echoed the whole buffer as a 17-element record the guest refused, vmmCreateVaspace returned ffff and GL died with 'Failed to allocate VASpace object' / EGL_BAD_ALLOC on every prior session. A real GSP-RM cannot reply past the bound either; the echo is clamped with the tail zeroed
* r484 the staging-drain completion edge: when the acquire gate holds an entry ↵Arthur Hanson Rasmusson29 hours
| | | | (its op may have already run on an earlier gate-less decode), force exactly one op re-run on the first pass whose gate finally passes - the pass at which the dependency just became installed. Without it the once-only done bitmap suppressed the op and the release applied while the staging still held the pre-dependency bytes (streams_events s=2: the D2H read dc2 before the vadd_into wrote it, ran early, was skipped when the wait passed, and the guest drained a2+b2 instead of a2+b2+c1)
* r481c sema_push_raw: lower the call-site threshold to 512 bytes so launch ↵Arthur Hanson Rasmusson29 hours
| | | | pushes (872/888B) dump; runtime-enable before the client run to avoid boot noise
* r481b sema_push_raw walk fix: handle op!=1 method groups (data runs, scans) ↵Arthur Hanson Rasmusson29 hours
| | | | and bound to launch-sized pushes (512..8192 bytes) so the launch method stream dumps cleanly
* r481 sema_push_raw diagnostic: dump the decoded method stream of the first N ↵Arthur Hanson Rasmusson29 hours
| | | | large GR-class pushes (default 0 off, no behaviour change) so the launch push's acquire/release method groups are visible verbatim; needed to prove which ordering mechanism the driver relies on for async-stream H2D -> kernel dependencies (streams_events st2 vadd copy-in ran ~115 ms before its stream's H2D soft copies)
* r471 the counter-publish clamp: never advance the guest's completion word ↵Arthur Hanson Rasmusson30 hours
| | | | past a push whose content was still unwritten when its entry was consumed. The r477 failing probe run showed the reuse hazard directly: the channel registered with a residual USERD put (4) and ring slots 1-3 all-zero; the pump consumed them (the r470 hold only delays) and the end-of-pass publish then wrote val=2 while the client's own counter was still 1 - the D2H wait was satisfied before the copy ran and the readback drained a reused staging buffer holding the PREVIOUS copy's bytes (h_c[2] = c0's data). New per-channel gr_bad_put (lowest all-zero GR slot + 1, persisted in the pump write-back): set when a zero-content GR entry is held/consumed, cleared when a decode reads non-zero content (the rewinds re-read entries constantly, so it heals within passes); svc_counter_publish clamps val to it with a bounded pub-clamp witness.
* r470 the narrow streams_events stale-ring fix: an ALL-ZERO push page (ring ↵Arthur Hanson Rasmusson31 hours
| | | | entry written - previous session's or the client's own slot reuse - but the push content not yet landed) is held by the m164 GR no-capture window instead of being consumed as a no-op, so the end-of-pass counter publish cannot advance the guest's completion word before the real push is read (streams_events: h_c[2] = c0 because the c2 D2H completion was published 4.5 s before its copy ran). Deliberately narrower than the reverted r469 plausibility verdict: compute-class GR pushes legitimately fail the CE-shaped method matcher, so only an entirely zero page counts as unwritten (r469's broad hold delayed GR config pushes and triggered the engine-fetch-NONE wedge on the first client).
* Revert "r469 THE streams_events staging-reuse corruption root cause: a GR ↵Arthur Hanson Rasmusson31 hours
| | | | | | entry whose push content is implausible (unwritten / previous-epoch page, no QMD word seen) was consumed as a no-op and the end-of-pass counter publish then advanced the guest's completion word while the real push had never been read - the client drained a staging buffer that still held the PREVIOUS copy's bytes (probe run: BUF s=2 got=c0's 0,0.75,1.5.. instead of c2's 3.0,5.25,7.5..; log: 'pub-adv h=0x5c000062 val=4 cur=1' at 210741.947, 4.5 s before the c2 D2H copy executed at 210746.371, with the ring slots decoded as zero-content pushes at 210736). The implausible-content read now feeds the m164 GR no-capture hold (bounded gsp_sema_gr_nocap_hold_ms window, rehold forces a fresh capture): the entry stays unconsumed, the publish gate (last_get != put) suppresses the early completion, and the retry reads the client's real push once it lands. Plausible fallback reads (validated pin / displaced pair) clear the flag." This reverts commit 349ee341539a0c1e9e3be11563251589eb42f8a8.
* r469 THE streams_events staging-reuse corruption root cause: a GR entry ↵Arthur Hanson Rasmusson32 hours
| | | | whose push content is implausible (unwritten / previous-epoch page, no QMD word seen) was consumed as a no-op and the end-of-pass counter publish then advanced the guest's completion word while the real push had never been read - the client drained a staging buffer that still held the PREVIOUS copy's bytes (probe run: BUF s=2 got=c0's 0,0.75,1.5.. instead of c2's 3.0,5.25,7.5..; log: 'pub-adv h=0x5c000062 val=4 cur=1' at 210741.947, 4.5 s before the c2 D2H copy executed at 210746.371, with the ring slots decoded as zero-content pushes at 210736). The implausible-content read now feeds the m164 GR no-capture hold (bounded gsp_sema_gr_nocap_hold_ms window, rehold forces a fresh capture): the entry stays unconsumed, the publish gate (last_get != put) suppresses the early completion, and the retry reads the client's real push once it lands. Plausible fallback reads (validated pin / displaced pair) clear the flag.
* r459 THE mid-boot degradation root cause: the 8-slot VA-space table filled ↵Arthur Hanson Rasmusson33 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | with dead clients' registrations and every new client after the 8th got no slot Evidence (r466 boot, one battery): 23 distinct 'VA space 0x... page directory ... entries=4' registrations and ZERO 'VA space ... dropped' lines. The guest driver does not always tear a client's VA space down (client exit, crash, or a switch the RM emulation does not forward), so after eight registrations svc.vas[] had no free slot: gsp_sema_service_vaspace_add silently refused the new directory (the 'if (vs)' guard), svc_walk_vaspaces kept walking only the dead clients' directories, and every walk of the new client's VAs failed -> its CE copies returned rc=-2 forever ('sema-optrace ... rc=-2' x8151 in the r466 log; the 10th+ client of the boot: streams2 bad=32737, streams_events no output, smiley hung, and the same class on r463/r465). This is the 'no launch stalls / no mid-boot degradation' blocker. Fix: - svc_vaspace_add evicts the least-recently-registered entry when the table is full instead of refusing (logged), and stamps every registration; - svc_walk_vaspaces orders the snapshot by registration stamp, newest first, so a live client's directory is the authority and a stale one (whose guest memory the driver has since reused) can only answer when the live directory cannot.
* r458 THE memset2d root cause: the CE copy decode read only LINE_LENGTH_IN ↵Arthur Hanson Rasmusson35 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | and ignored the LINE_COUNT value the driver encodes in the SAME count=2 method group at 0x418 Raw push evidence (sema-op2d-raw, new bounded diagnostic): raw 17000520 7c7f0204 ... 20018104 00001000 20018105 00001000 20028106 00001000 00000100 20048100 7d7e 40c00000 The group at 0x418 carries count=2: v0 = LINE_LENGTH_IN (4096), v1 = LINE_COUNT (0x100 = 256 rows). The CE decode stored only v0, so st.line_count stayed 0, the 2D-copy builder took the single-row path (m->bytes = 4096) and a 4 MB cudaMemcpy2D delivered exactly ONE page per 1 MB block. memset2d_probe (new per-phase client) shows the exact shape: per-MB ok=4096 / bad=1044480 in all three phases, total 12,505,200 of 12,582,912 - deterministic across boots (r453/r454/r459 identical). The GRCE variant has always decoded the count=2 form (gr_line_count = v1); the CE path did not. Fix: CE_M_LINE_LENGTH_IN stores v1 into st.line_count when count >= 2, so the existing r420c contiguous-run logic (pitch == line => flat run of rows*line) finally sees the row count. The same decode feeds the fill path (cudaMemset2D). Also adds the bounded sema-op2d-raw witness (raw method stream of every pitch-shaped copy push) that produced the evidence, and a per-phase memset2d_probe client under tools/cuda-ab/.
* r457 free the copy staging snapshot only on consumption/slot-reuse/skip/TTL ↵Arthur Hanson Rasmusson37 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | (not on opdef_clear) The r456 change kept the snapshot alive across opdef APPLY, but the r457 mode-C run proved a second freer: svc_opdef_clear's FINISH path (entry consume via svc_replay_note) freed the b1 H2D's snapshot while the copy was still owed. The pin-batch rewind then REPLAYED the entry, the re-capture read the driver's REFILLED staging and the deferred execution wrote b2's bytes into db1. Evidence (r457, mode C, owner 0x5c00005c/1/0627e693d05aab3f): 191607.816388 copy snapshot captured dst=0x72e318c20000 src=0x72e31c400000 first4=00000000 3e800000 3f000000 3f400000 (b1) 191608.798838 copy snapshot captured <same key> first4=00000000 3d800000 3e000000 3e400000 (b2 - refilled staging) 191611.404931 sema-bigcopy dst_va=0x72e318c20000 src_va=0x72e31c400000 dst16_after=000000000000803d0000003e0000403e (b2 landed in db1) The op never succeeded (rc=-2 every pass), so the only free that could have dropped the first snapshot is the opdef_clear owner free. New lifecycle: capture once -> free on the copy's own successful consumption -> free on ring-slot reuse with a new push hash -> free when a re-decode skips an already-done op -> 30 s TTL. svc_copy_snap_free_owner keeps its rationale but is no longer called (the r444 cross-entry poisoning stays closed by the owner key + eviction).
* r456 keep the copy staging snapshot alive across opdef retries (the r455 ↵Arthur Hanson Rasmusson38 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | operand-shift root cause) The fail4 pipeline-order snapshot (captured when a device-dst H2D copy is held for its operand pins) was freed by svc_opdef_clear on the APPLY path on every re-decode pass of the held entry. The retried copy therefore re-captured from the driver's REFILLED ping-pong staging: the a1 H2D of the r455 streams2_probe D run executed ~3 s after its push carrying a2's bytes (snapshot first4 flipped 00000000 3f000000.. -> 00000000 3e000000.. across retries; sema-bigcopy dst16_after=000000000000003e0000803e0000c03e proves the a2 pattern landed in da1's page; the vadd computed a2+b1 and the readback faithfully delivered it - there is no separate read-side landing bug for this class). - svc_opdef_clear: keep the snapshot on the APPLY path (the pass's own op execution, which consumes the snapshot on success, follows the apply); every genuine consume path (FINISH/FLUSH/DROP/STALE/COUNTED/ SWEEP/STOP) and the TTL still free it. - svc_copy_snap_capture: evict same-ring-slot snapshots with a different push hash (slot reuse) so the surviving snapshot can never serve a fresh push. - svc_run_ops: an op skipped via ops_done drops any snapshot a later re-decode re-captured for it (the leak the r444 APPLY-free used to sweep).
* r453 rel_mod=1 default: route the forwarded QMD release0 to the module ↵Arthur Hanson Rasmusson39 hours
| | | | semaphore so the guest-visible release word is authored by the host only on the completion edge (after the retire wait and after the bounce copy-out shipped the engine's stores). With the historical engine-authored guest release the guest's stream-ordered readback could be satisfied while the host copy-out was still pending, so streams2's c2 / memcpy_roundtrip's memset phase / streams_events' chain read pre-launch content. Evidence on r453 (with the r452 FNV change-detection fix): rel_mod=0 -> 11/15 battery; rel_mod=1 -> 13/15 (victory_streams_events and memcpy_roundtrip flip green), no stalls across the full battery, victory_mail + victory_saxpy clean. The r414 stall hole is closed by the r420b per-outcome authoring; rollback is one echo to 0
* r452 change-detection fingerprint fix: the 32-bit ADDITIVE page checksum ↵Arthur Hanson Rasmusson40 hours
| | | | used for the copy-out 'engine changed this page' decision is provably blind to power-of-two in-place scales - doubling every word of a 4 KB page shifts each exponent by +1 (+2^23/word), and 1024 words give 1024*2^23 = 2^33 == 0 (mod 2^32), so c[i] *= 2.0f left csum(post) == csum(pre), the page classified 'unchanged', the copy-out was skipped and the client read the PRE-scale values (the deterministic streams2 / streams_events / in-place-scale class, reproduced by ip_probe2: the chunk holds 0x43010000 (129.0) while the guest page holds 0x42810000 (64.5) and csum post==pre). Replace every change-decision fingerprint (scan_pages, retire wait, copy-out, gap copy-out, window witness, prev-out late-store witness, idempotent launch fp) with FNV-1a 64 over the page; the additive sum survives only as the all-zero filter (gsp_phys_gr_page_sum32). New r452 diagnostics: OUT-UNCH witness (unchanged page: chunk vs guest first words, the line that cracked it) and gsp_phys_gr_scan (whole-chunk pattern scan)
* r449 shader-constant-cache invalidate experiment (gsp_phys_gr_ccache_inv, ↵Arthur Hanson Rasmusson41 hours
| | | | default off): the forwarded QMD's cache-control block (w5) always carries bits 26..29 and the M91 fix adds bit 30 (instruction), but the V01_06-layout SHADER_CONSTANT_CACHE position (bit 31) is never set - and the forwarded param page is rewritten in place between launches (two-bank win0 pool), so a stale constant-cache line can serve the PREVIOUS launch's parameters (stale pointers -> the current launch's output pages untouched = the post-dispatch no-store signature). Mode 1 sets bit 31, mode 2 forces bits 24..31; only applied when the standard signature is present
* r445b wedge hard re-arm trigger: the r445 battery shows every post-dispatch ↵Arthur Hanson Rasmusson43 hours
| | | | no-store episode leaves the warm ladder sem probe ALIVE (the ladder block runs; only the client launch no-writes), so the re-arm now fires on the FINAL no-write retry - TSG PREEMPT bWait=1 (forced context save/unload) + lane-enable/schedules re-run - before the last re-publish; default off
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-23 23:20:42 +0000