summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* gitignore: never track demo/release .mp4 videos (the 0.9.9 release preview ↵Arthur Hanson Rasmusson13 hours
| | | | will live in the repo root locally, produced from the host KVMFR capture)
* r489b CONTINUATION_RECORD from the driver is one-way data (the tail of a ↵Arthur Hanson Rasmusson13 hours
| | | | multi-frame request): stop replying to it. The guess reply injected an extra fn-71 record the driver drained as an unexpected event (0x47) and corrupted our own large-reply continuation framing
* r489 the proper large-reply framing for GSP_RM_CONTROL: replies beyond the ↵Arthur Hanson Rasmusson13 hours
| | | | driver's pRpc->maxRpcSize (65080; pinned by the guest's own remainingSize 0x940 arithmetic for the 67448-byte PERF_VPSTATES_INFO reply) are now sent with the GSP continuation-record protocol - first frame length = maxRpcSize, then NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD frames of up to maxRpcSize-32 bytes each, exactly like the host GSP's _rpcWriteContinuationEntries. Replaces the r487 clamp (which produced a record the guest refused, and a size the guest then waited on fn-71 continuations for). Root: the GL VASpace construction and 3D engine selection die on this control's reply
* r487b tighten the RM_CONTROL reply clamp margin: the guest rejected ↵Arthur Hanson Rasmusson14 hours
| | | | rpc.length 65528 (its element header > 8 bytes); clamp to 16*4096 - 512 so any element-header size fits the bound
* r487 THE GL/VULKAN VASPACE WALL: clamp RM_CONTROL replies to the 64 KB wire ↵Arthur Hanson Rasmusson14 hours
| | | | record bound (16 x 4096; guest GspMsgQueueReceiveStatus rejects hdr + rpc.length beyond it - 'Incorrect message length 67448', rpcRecvPoll 0x3a, NV_ERR_INVALID_PARAMETER). The stock driver sends PERF_VPSTATES_INFO (0x2080a0a4, paramsSize 67392) during Xorg VASpace construction; the generic reply echoed the whole buffer as a 17-element record the guest refused, vmmCreateVaspace returned ffff and GL died with 'Failed to allocate VASpace object' / EGL_BAD_ALLOC on every prior session. A real GSP-RM cannot reply past the bound either; the echo is clamped with the tail zeroed
* r484 the staging-drain completion edge: when the acquire gate holds an entry ↵Arthur Hanson Rasmusson15 hours
| | | | (its op may have already run on an earlier gate-less decode), force exactly one op re-run on the first pass whose gate finally passes - the pass at which the dependency just became installed. Without it the once-only done bitmap suppressed the op and the release applied while the staging still held the pre-dependency bytes (streams_events s=2: the D2H read dc2 before the vadd_into wrote it, ran early, was skipped when the wait passed, and the guest drained a2+b2 instead of a2+b2+c1)
* r481c sema_push_raw: lower the call-site threshold to 512 bytes so launch ↵Arthur Hanson Rasmusson15 hours
| | | | pushes (872/888B) dump; runtime-enable before the client run to avoid boot noise
* r481b sema_push_raw walk fix: handle op!=1 method groups (data runs, scans) ↵Arthur Hanson Rasmusson15 hours
| | | | and bound to launch-sized pushes (512..8192 bytes) so the launch method stream dumps cleanly
* r481 sema_push_raw diagnostic: dump the decoded method stream of the first N ↵Arthur Hanson Rasmusson15 hours
| | | | large GR-class pushes (default 0 off, no behaviour change) so the launch push's acquire/release method groups are visible verbatim; needed to prove which ordering mechanism the driver relies on for async-stream H2D -> kernel dependencies (streams_events st2 vadd copy-in ran ~115 ms before its stream's H2D soft copies)
* r471 the counter-publish clamp: never advance the guest's completion word ↵Arthur Hanson Rasmusson16 hours
| | | | past a push whose content was still unwritten when its entry was consumed. The r477 failing probe run showed the reuse hazard directly: the channel registered with a residual USERD put (4) and ring slots 1-3 all-zero; the pump consumed them (the r470 hold only delays) and the end-of-pass publish then wrote val=2 while the client's own counter was still 1 - the D2H wait was satisfied before the copy ran and the readback drained a reused staging buffer holding the PREVIOUS copy's bytes (h_c[2] = c0's data). New per-channel gr_bad_put (lowest all-zero GR slot + 1, persisted in the pump write-back): set when a zero-content GR entry is held/consumed, cleared when a decode reads non-zero content (the rewinds re-read entries constantly, so it heals within passes); svc_counter_publish clamps val to it with a bounded pub-clamp witness.
* r470 the narrow streams_events stale-ring fix: an ALL-ZERO push page (ring ↵Arthur Hanson Rasmusson17 hours
| | | | entry written - previous session's or the client's own slot reuse - but the push content not yet landed) is held by the m164 GR no-capture window instead of being consumed as a no-op, so the end-of-pass counter publish cannot advance the guest's completion word before the real push is read (streams_events: h_c[2] = c0 because the c2 D2H completion was published 4.5 s before its copy ran). Deliberately narrower than the reverted r469 plausibility verdict: compute-class GR pushes legitimately fail the CE-shaped method matcher, so only an entirely zero page counts as unwritten (r469's broad hold delayed GR config pushes and triggered the engine-fetch-NONE wedge on the first client).
* Revert "r469 THE streams_events staging-reuse corruption root cause: a GR ↵Arthur Hanson Rasmusson17 hours
| | | | | | entry whose push content is implausible (unwritten / previous-epoch page, no QMD word seen) was consumed as a no-op and the end-of-pass counter publish then advanced the guest's completion word while the real push had never been read - the client drained a staging buffer that still held the PREVIOUS copy's bytes (probe run: BUF s=2 got=c0's 0,0.75,1.5.. instead of c2's 3.0,5.25,7.5..; log: 'pub-adv h=0x5c000062 val=4 cur=1' at 210741.947, 4.5 s before the c2 D2H copy executed at 210746.371, with the ring slots decoded as zero-content pushes at 210736). The implausible-content read now feeds the m164 GR no-capture hold (bounded gsp_sema_gr_nocap_hold_ms window, rehold forces a fresh capture): the entry stays unconsumed, the publish gate (last_get != put) suppresses the early completion, and the retry reads the client's real push once it lands. Plausible fallback reads (validated pin / displaced pair) clear the flag." This reverts commit 349ee341539a0c1e9e3be11563251589eb42f8a8.
* r469 THE streams_events staging-reuse corruption root cause: a GR entry ↵Arthur Hanson Rasmusson18 hours
| | | | whose push content is implausible (unwritten / previous-epoch page, no QMD word seen) was consumed as a no-op and the end-of-pass counter publish then advanced the guest's completion word while the real push had never been read - the client drained a staging buffer that still held the PREVIOUS copy's bytes (probe run: BUF s=2 got=c0's 0,0.75,1.5.. instead of c2's 3.0,5.25,7.5..; log: 'pub-adv h=0x5c000062 val=4 cur=1' at 210741.947, 4.5 s before the c2 D2H copy executed at 210746.371, with the ring slots decoded as zero-content pushes at 210736). The implausible-content read now feeds the m164 GR no-capture hold (bounded gsp_sema_gr_nocap_hold_ms window, rehold forces a fresh capture): the entry stays unconsumed, the publish gate (last_get != put) suppresses the early completion, and the retry reads the client's real push once it lands. Plausible fallback reads (validated pin / displaced pair) clear the flag.
* tools/cuda-ab/run: the r458-r459 probe binaries (memset2d_probe, ↵Arthur Hanson Rasmusson19 hours
| | | | streams_events_probe, tgemm_probe)
* r459 THE mid-boot degradation root cause: the 8-slot VA-space table filled ↵Arthur Hanson Rasmusson19 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | with dead clients' registrations and every new client after the 8th got no slot Evidence (r466 boot, one battery): 23 distinct 'VA space 0x... page directory ... entries=4' registrations and ZERO 'VA space ... dropped' lines. The guest driver does not always tear a client's VA space down (client exit, crash, or a switch the RM emulation does not forward), so after eight registrations svc.vas[] had no free slot: gsp_sema_service_vaspace_add silently refused the new directory (the 'if (vs)' guard), svc_walk_vaspaces kept walking only the dead clients' directories, and every walk of the new client's VAs failed -> its CE copies returned rc=-2 forever ('sema-optrace ... rc=-2' x8151 in the r466 log; the 10th+ client of the boot: streams2 bad=32737, streams_events no output, smiley hung, and the same class on r463/r465). This is the 'no launch stalls / no mid-boot degradation' blocker. Fix: - svc_vaspace_add evicts the least-recently-registered entry when the table is full instead of refusing (logged), and stamps every registration; - svc_walk_vaspaces orders the snapshot by registration stamp, newest first, so a live client's directory is the authority and a stale one (whose guest memory the driver has since reused) can only answer when the live directory cannot.
* tools/cuda-ab: isolation probes from the transformer/streams_events forensicsArthur Hanson Rasmusson20 hours
| | | | | | | | | | | | | | | - tgemm_probe: minimal X[64x256]xW[256x256] GEMM isolation. Found the deterministic first-page corruption: after a forwarded launch (and sometimes already after the H2D upload) the FIRST 8 bytes of the first data-window page are replaced by that page's bytes at offset 8 (X[0..1] <- X[2..3]; vidmem dump confirms; OUT-UNCH shows chunk==guest), which corrupts GEMM row 0 for all 256 columns (bad=256/pass, identical across 3 passes). Not rel_mod-dependent (A/B at runtime). - streams_events_probe: per-buffer DPTRS + per-buffer bad + first-8 expected/actual for the 3-stream event chain (green on a clean boot, bad=131072 after several clients - the client-switch resolution class). - memset2d_probe: per-phase bad/PHIST for victory_memset2d (verified the r458 LINE_COUNT fix: bad_total=0, all three phases).
* pin_gr_stream: the demand-loop regex never matched the kernel's stalled-VA ↵Arthur Hanson Rasmusson20 hours
| | | | | | line (it expected 'stalled VA #N=0x...' but the kernel logs 'stalled VA #N: 0x...'), so host->guest demand pins never fired for stalled VAs and a client whose push/operand pages the resolver missed stayed stalled for the rest of the boot. Accept ':' or '=' after the entry number. Evidence: r463/r464 stalled clients log 'stalled VA #16: 0x755813400000' etc. while the host demand loop only ever emitted pool VAs (0x12100000x) to /tmp/guest_want_vas.txt.
* r458 THE memset2d root cause: the CE copy decode read only LINE_LENGTH_IN ↵Arthur Hanson Rasmusson21 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | and ignored the LINE_COUNT value the driver encodes in the SAME count=2 method group at 0x418 Raw push evidence (sema-op2d-raw, new bounded diagnostic): raw 17000520 7c7f0204 ... 20018104 00001000 20018105 00001000 20028106 00001000 00000100 20048100 7d7e 40c00000 The group at 0x418 carries count=2: v0 = LINE_LENGTH_IN (4096), v1 = LINE_COUNT (0x100 = 256 rows). The CE decode stored only v0, so st.line_count stayed 0, the 2D-copy builder took the single-row path (m->bytes = 4096) and a 4 MB cudaMemcpy2D delivered exactly ONE page per 1 MB block. memset2d_probe (new per-phase client) shows the exact shape: per-MB ok=4096 / bad=1044480 in all three phases, total 12,505,200 of 12,582,912 - deterministic across boots (r453/r454/r459 identical). The GRCE variant has always decoded the count=2 form (gr_line_count = v1); the CE path did not. Fix: CE_M_LINE_LENGTH_IN stores v1 into st.line_count when count >= 2, so the existing r420c contiguous-run logic (pitch == line => flat run of rows*line) finally sees the row count. The same decode feeds the fill path (cudaMemset2D). Also adds the bounded sema-op2d-raw witness (raw method stream of every pitch-shaped copy push) that produced the evidence, and a per-phase memset2d_probe client under tools/cuda-ab/.
* r457 free the copy staging snapshot only on consumption/slot-reuse/skip/TTL ↵Arthur Hanson Rasmusson23 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | (not on opdef_clear) The r456 change kept the snapshot alive across opdef APPLY, but the r457 mode-C run proved a second freer: svc_opdef_clear's FINISH path (entry consume via svc_replay_note) freed the b1 H2D's snapshot while the copy was still owed. The pin-batch rewind then REPLAYED the entry, the re-capture read the driver's REFILLED staging and the deferred execution wrote b2's bytes into db1. Evidence (r457, mode C, owner 0x5c00005c/1/0627e693d05aab3f): 191607.816388 copy snapshot captured dst=0x72e318c20000 src=0x72e31c400000 first4=00000000 3e800000 3f000000 3f400000 (b1) 191608.798838 copy snapshot captured <same key> first4=00000000 3d800000 3e000000 3e400000 (b2 - refilled staging) 191611.404931 sema-bigcopy dst_va=0x72e318c20000 src_va=0x72e31c400000 dst16_after=000000000000803d0000003e0000403e (b2 landed in db1) The op never succeeded (rc=-2 every pass), so the only free that could have dropped the first snapshot is the opdef_clear owner free. New lifecycle: capture once -> free on the copy's own successful consumption -> free on ring-slot reuse with a new push hash -> free when a re-decode skips an already-done op -> 30 s TTL. svc_copy_snap_free_owner keeps its rationale but is no longer called (the r444 cross-entry poisoning stays closed by the owner key + eviction).
* r456 keep the copy staging snapshot alive across opdef retries (the r455 ↵Arthur Hanson Rasmusson24 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | operand-shift root cause) The fail4 pipeline-order snapshot (captured when a device-dst H2D copy is held for its operand pins) was freed by svc_opdef_clear on the APPLY path on every re-decode pass of the held entry. The retried copy therefore re-captured from the driver's REFILLED ping-pong staging: the a1 H2D of the r455 streams2_probe D run executed ~3 s after its push carrying a2's bytes (snapshot first4 flipped 00000000 3f000000.. -> 00000000 3e000000.. across retries; sema-bigcopy dst16_after=000000000000003e0000803e0000c03e proves the a2 pattern landed in da1's page; the vadd computed a2+b1 and the readback faithfully delivered it - there is no separate read-side landing bug for this class). - svc_opdef_clear: keep the snapshot on the APPLY path (the pass's own op execution, which consumes the snapshot on success, follows the apply); every genuine consume path (FINISH/FLUSH/DROP/STALE/COUNTED/ SWEEP/STOP) and the TTL still free it. - svc_copy_snap_capture: evict same-ring-slot snapshots with a different push hash (slot reuse) so the surviving snapshot can never serve a fresh push. - svc_run_ops: an op skipped via ops_done drops any snapshot a later re-decode re-captured for it (the leak the r444 APPLY-free used to sweep).
* tools/cuda-ab/run: add the r451-r453 probe binaries (ip_probe*, ip4/5/6, ↵Arthur Hanson Rasmusson24 hours
| | | | streams2_probe, scale_probe, rt_probe) - the suite that cracked the additive-csum collision and now serves as the regression detector for the reader-side resolution class
* tools/cuda-ab: the r451-r453 probe suite that cracked the deterministic ↵Arthur Hanson Rasmusson25 hours
| | | | class - ip_probe (CTA-marker in-place scale), ip_probe2 (scale_if/scale_u/fill_c/add_c staged matrix), ip_probe3 (what the SM actually reads), streams2_probe (4 stream-ordering modes), scale_probe (in-place vs out-of-place), rt_probe (memcpy_roundtrip staged forensics); run them with setsid+poll, never timeout-kill mid-flight
* docs/vEP: record the r452 copy-out fingerprint rule (a change-detection ↵Arthur Hanson Rasmusson25 hours
| | | | fingerprint must not be an algebraic invariant of any plausible kernel operation - the 32-bit additive page checksum was blind to in-place power-of-two scales: 1024 words x 2^23 = 2^33 == 0 mod 2^32; FNV-1a 64 since r452) and the r453 rel_mod=1 default (guest release authored only on the post-copy-out completion edge)
* r453 rel_mod=1 default: route the forwarded QMD release0 to the module ↵Arthur Hanson Rasmusson25 hours
| | | | semaphore so the guest-visible release word is authored by the host only on the completion edge (after the retire wait and after the bounce copy-out shipped the engine's stores). With the historical engine-authored guest release the guest's stream-ordered readback could be satisfied while the host copy-out was still pending, so streams2's c2 / memcpy_roundtrip's memset phase / streams_events' chain read pre-launch content. Evidence on r453 (with the r452 FNV change-detection fix): rel_mod=0 -> 11/15 battery; rel_mod=1 -> 13/15 (victory_streams_events and memcpy_roundtrip flip green), no stalls across the full battery, victory_mail + victory_saxpy clean. The r414 stall hole is closed by the r420b per-outcome authoring; rollback is one echo to 0
* r452 change-detection fingerprint fix: the 32-bit ADDITIVE page checksum ↵Arthur Hanson Rasmusson26 hours
| | | | used for the copy-out 'engine changed this page' decision is provably blind to power-of-two in-place scales - doubling every word of a 4 KB page shifts each exponent by +1 (+2^23/word), and 1024 words give 1024*2^23 = 2^33 == 0 (mod 2^32), so c[i] *= 2.0f left csum(post) == csum(pre), the page classified 'unchanged', the copy-out was skipped and the client read the PRE-scale values (the deterministic streams2 / streams_events / in-place-scale class, reproduced by ip_probe2: the chunk holds 0x43010000 (129.0) while the guest page holds 0x42810000 (64.5) and csum post==pre). Replace every change-decision fingerprint (scan_pages, retire wait, copy-out, gap copy-out, window witness, prev-out late-store witness, idempotent launch fp) with FNV-1a 64 over the page; the additive sum survives only as the all-zero filter (gsp_phys_gr_page_sum32). New r452 diagnostics: OUT-UNCH witness (unchanged page: chunk vs guest first words, the line that cracked it) and gsp_phys_gr_scan (whole-chunk pattern scan)
* r449 shader-constant-cache invalidate experiment (gsp_phys_gr_ccache_inv, ↵Arthur Hanson Rasmusson27 hours
| | | | default off): the forwarded QMD's cache-control block (w5) always carries bits 26..29 and the M91 fix adds bit 30 (instruction), but the V01_06-layout SHADER_CONSTANT_CACHE position (bit 31) is never set - and the forwarded param page is rewritten in place between launches (two-bank win0 pool), so a stale constant-cache line can serve the PREVIOUS launch's parameters (stale pointers -> the current launch's output pages untouched = the post-dispatch no-store signature). Mode 1 sets bit 31, mode 2 forces bits 24..31; only applied when the standard signature is present
* tools/soak.sh: fix the present-clients probe (the for-loop's exit status was ↵Arthur Hanson Rasmusson28 hours
| | | | 1 whenever the last listed client was absent, so a partial battery always failed with 'no-client-binaries-in-/tmp'); append true inside the probe
* r445b wedge hard re-arm trigger: the r445 battery shows every post-dispatch ↵Arthur Hanson Rasmusson29 hours
| | | | no-store episode leaves the warm ladder sem probe ALIVE (the ladder block runs; only the client launch no-writes), so the re-arm now fires on the FINAL no-write retry - TSG PREEMPT bWait=1 (forced context save/unload) + lane-enable/schedules re-run - before the last re-publish; default off
* r445 wedge hard re-arm experiment (gsp_phys_gr_wedge_rearm, default off): on ↵Arthur Hanson Rasmusson29 hours
| | | | the proven engine-wedged warm-sem-probe failure, one bounded recovery attempt - TSG PREEMPT bWait=1 (forced context save/unload) + the m155 extended re-arm (mode-35 lane-enable tail + GPFIFO schedules + doorbell) + a repeated probe - before the historical WEDGED break
* r444 copy-snapshot ownership: key the fail4 held-copy staging snapshot by ↵Arthur Hanson Rasmusson30 hours
| | | | | | | | | | | | | | | | | | | | | (channel, ring entry, push-content hash), not (dst,src,total) alone The wedge_ab deterministic reproducer (bad = mutated-element count) root cause: a held CE copy's source snapshot was captured for one ring entry, left behind when that entry's copy executed through a path that never consumed it (hardware-primary forwarded copy, or a post-execution re-decode whose copy was re-held), and then consumed by a LATER entry's copy that happens to share the (dst,src,total) triple - the client's cudaMemcpy released early with the PREVIOUS step's bytes in the device buffer, and the next GR launch computed on stale inputs (r441/r442/r443 evidence: step N read step N-1 content; sema-bigcopy src16 shows the fresh live source at execution time while dst16_after shows the old snapshot's bytes). Owner identity is (ch->h_object, entry_idx, svc_pass_hash); the pass hash separates a ring slot reused by a fresh push from the entry that captured the snapshot. svc_opdef_clear() additionally frees any snapshot still owned by a consumed entry, so no snapshot can outlive its entry.
* docs: SESSION 13 handoff (transformer=wedge class, four negative wedge ↵Arthur Hanson Rasmusson31 hours
| | | | experiments, the r439 stale-release variant + r441 first-attempt zeroing, r437 battery 10/15, frame animation blocked by the wedge) + frame-sequence tool + LG doc update + transformer stage-bisect tools
* r441 zero the guest-visible release words on the FIRST attempt too (not only ↵Arthur Hanson Rasmusson31 hours
| | | | outcome retries): the r439 evidence shows a first attempt breaking its poll on a stale value (qmd_rel=0x5 from the previous launch's engine write, want=6) while the engine never fetched the deposit (sentinel intact) - the host then shipped a completion for a launch that did not run; zeroing forces the poll to wait for this launch's own engine write, so the timeout ladder and the software-authored completion are the honest path and the wedge classification can no longer be satisfied by a stale value
* r436 full sysmem PTE aperture/VOL matrix: gsp_phys_rmcall_pte_coh becomes a ↵Arthur Hanson Rasmusson33 hours
| | | | 4-mode selector (0=0xf aperture3+VOL historical, 1=0xd aperture2+VOL r431, 2=0x5 aperture2 non-volatile, 3=0x7 aperture3 non-volatile) applied consistently to the leaf PTE and the single PDE (bit0 = IS_PDE there), so the r431 finding (aperture 2 with VOL still 1 is a pair the reference RM never emits) can be A/B'd against the consistent encodings
* r434 wedge instrumentation + CWD-slot path: post-retire QMD64 dump ↵Arthur Hanson Rasmusson33 hours
| | | | (hardware-only w50/w51 INNER get/put = the SKED's CTA dispatch counters, which the submission-time dump cannot show) so the healthy-vs-wedged diff proves whether the grid was dispatched and truncated; post-OK qmdwin reads the launch's own bank; gsp_phys_rmcall_gr_cwd_slot >= 0 emits the CUDA driver's CWD-slot PRI write (SKED_DEBUG_1 ENABLE_CWD_SLOT_COUNT via falcon scratch) + SET_CWD_SLOT_COUNT in the init preamble for the mission's slot-count A/B (-1 = off)
* r433 RELEASE_MEMBAR_TYPE A/B: gsp_phys_gr_rel_membar forces QMD word11 bit14 ↵Arthur Hanson Rasmusson34 hours
| | | | (FE_SYSMEMBAR) on forwarded GR launches so the engine flushes pending sysmem stores before writing the release semaphore - the post-dispatch store-loss signature (release host-visible, data stuck in L2) is what an unflushed write-back L2 looks like from the host; 0 keeps the guest's captured FE_NONE for a one-flip A/B
* r431b revert the coherent-PTE default: the clean-boot A/B showed aperture 2 ↵Arthur Hanson Rasmusson34 hours
| | | | (SYSTEM_COHERENT) is worse on this platform (the first client of the boot wedged with 0/983 and the wedge persisted across the sweep), so the historical NON_COHERENT aperture stays the default; gsp_phys_rmcall_pte_coh=1 remains for experiments (it did make the engine's partial 1-page writes visible sooner)
* r431 sysmem leaf-PTE aperture fix (root-cause candidate): map every host ↵Arthur Hanson Rasmusson34 hours
| | | | dma_alloc_coherent page as SYSTEM_COHERENT (aperture 2) instead of SYSTEM_NON_COHERENT (3) - the reference RM picks the aperture from the allocation's CPU cache attribute, and non-coherent L2 let engine stores linger invisibly (the no-execute wedge) while re-staged copy-in content stayed invisible to the engine; gsp_phys_rmcall_pte_coh=0 restores the historical aperture for a boot-time A/B
* r430 wedge dispatch-window sampling: skedrec samples every poll iteration ↵Arthur Hanson Rasmusson35 hours
| | | | (the release lands ~7ms after the doorbell; the old 8-iteration gate saw at most one sample), full 64-word QMD64 dump per launch (hardware-only w50/w51/w52/w53 included) for the healthy-vs-wedged diff
* r427 idempotent fast-accept: a zero-write launch whose identity (QMD minus ↵Arthur Hanson Rasmusson36 hours
| | | | release0 + staged content checksums) matches an earlier writing launch accepts immediately (no retries, no quiet holds); recorded for every writing launch; wedge diagnostics retained
* r425 wedge diagnostics + init-once A/B: SKED/FECS/TPC/CWD register witnesses ↵Arthur Hanson Rasmusson36 hours
| | | | (healthy vs wedged baselines in-log), prev-out late-store witness, gsp_phys_rmcall_gr_init_once (emit the compute init preamble once per boot vs per launch)
* tools: frame_reader picks the largest kvmfr shm (a stray empty 'kvmfr-*' ↵Arthur Hanson Rasmusson38 hours
| | | | file existed); wedge_ab same-program vs second-launch discriminator
* tools/lg + docs: guest frame producer path (CUDA -> ivshmem -> host) and the ↵Arthur Hanson Rasmusson38 hours
| | | | | | | | | | | | | | | | | Looking Glass status - tools/cuda-ab/frame_producer.cu: renders an animated Julia frame on the guest GPU and publishes it into the ivshmem BAR (the kvmfr 128 MiB window) with a small header + RGBA payload. - tools/lg/frame_reader.py: host-side reader of /dev/shm/kvmfr-<uuid> that saves each published frame as a PPM (the transport half of the frame path). - tools/guest-diag/egl_frame.c: EGL probe (device/surfaceless/default platforms) - live result: eglQueryDevicesEXT sees the NVIDIA device and eglInitialize succeeds, but eglMakeCurrent fails EGL_BAD_ALLOC (0x3003), the same VASpace-allocation wall as Xorg's 'Failed to allocate VASpace object'. - docs/05-operations/06: the state, the blockers and the exact remaining pieces (guest GL VASpace allocation; LGMP session producer).
* sema: r420d bounded 2D-copy geometry witness (sema-op2d)Arthur Hanson Rasmusson38 hours
| | | | | | | Every copy op logs dst/src/bytes/rows/dpitch/spitch beside the decoded line_length/line_count/pitches, so the memset2d readback forensics can tell 'one row per push' from 'N rows in one op' and whether the pitch methods arrived. Bounded by the existing opadd budget.
* tools/cuda-ab: wedge_bisect - bisect the no-execute wedge trigger by ↵Arthur Hanson Rasmusson39 hours
| | | | | | | | | inter-launch activity Modes A/B/C/D add none / an extra H2D / an extra D2H / a 500 ms sleep between the verify and the next launch of the wedge_recover shape. The inputs mutate every iteration, so a no-execute launch is reported (bad>0), never masked by deterministic content.
* sema: r420c 2D fills/copies with pitch == row length cover all rowsArthur Hanson Rasmusson39 hours
| | | | | | | | | | | | | | | | | | victory_memset2d failed deterministically (bad ~= 12.5M of 12.58M) because its pitched operations carry PITCH == LINE_LENGTH (cudaMallocPitch returned pitch 4096 for width 4096) and the r402 2D decode only entered row mode when a pitch was WIDER than the row. The flat path then ran ONE row (LINE_LENGTH bytes) per op, so every cudaMemcpy2D readback and H2D upload delivered only its first row and every cudaMemset2D-sized region beyond it kept stale content - the 'deterministic readback loss' (45K of 12.58M correct ~= the rows the single runs happened to cover). LINE_COUNT is now authoritative for the flat case too: pitch == row length (or no pitch method) means CONTIGUOUS rows, so the single run covers rows*line bytes (element-exact when the line divides evenly, else byte exact). A pitch wider than the row keeps the r402 row-wise mode; 1D ops are unchanged. Both the fill (cudaMemset2D/fill) and the copy (cudaMemcpy2D) decode are fixed.
* gr: r420b author the guest QMD release on every outcome when release0 is ↵Arthur Hanson Rasmusson39 hours
| | | | | | | | | | | | | | | | | | | redirected With gsp_phys_rmcall_gr_rel_mod=1 (release0 -> module semaphore, the ordering fix for the engine-release-before-copy-out race) the engine never writes the guest's own QMD release word, so the host must author it on EVERY backend outcome. The hold's success/fail edges do; a launch with no deferred release records (no hold engaged) fell through and its client hung on a word nothing would ever write - the r414 saxpy f0 cur=5 stall. The backend now reports qmd_rel_redirected in the copy result whenever it points release0 at the module semaphore (rel_mod, or no targetable guest release page). svc_fwd_gr_work folds that into the existing gr_rel_unobserved latch, which makes the bounded outcome check author the word on done (success or failure) - the same forward-only write the hold edges perform. rel_mod=0 paths are unchanged (the engine still authors the word and the latch is only set on an unresolved release page).
* gr/sema: r420 pause completion deadlines under the backend wedge-quiet holdArthur Hanson Rasmusson40 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The r419 soak reproduced the user's mid-boot degradation with the wedge fix in tree: victory_matmul printed bad=65536 while its launch was held by the booter's WEDGE-QUIET window - the client was released ~115 s before the post-hold accept. Root cause: two sema-service deadlines are shorter than the deliberate quiet hold and fire under it, authoring the launch's completion release (and applying its recorded releases) while the engine has not run and the copy-out has not happened: - the 12 s GR release hold (SVC_FWD_GR_HOLD_TIMEOUT_MS) expiry (GR-STALL-fwd-hold ... releases + QMD word software-applied), and - the 3 s per-launch consumption bound (gsp_phys_gr_hold_max_ms) firing the busy-defer consume-with-software-release path, whose inline apply authors the capture's QMD release (GR-REL-AUTHOR src=hold). Evidence (r419 live): push be94 captured 131333.687, WEDGE-QUIET holds 40 s + 80 s until 131345.598, accept 131346.552 - while the sema hold expired at 131230.978 (GR-STALL-fwd-hold waited_ms=12032) and the inline apply authored va=0x204027f7c pay=6 at 131341.734 (h=0x5c00001f). The client's verify completed on the authored word and read stale data. Fix: the booter publishes gsp_phys_gr_quiet_hold_active() exactly around the quiet msleep (bounded by construction). The sema service pauses under it: svc_fwd_hold_expired() pushes the GR deadline (bounded by a 600 s defence-in-depth cap, announced once per hold as GR-HOLD-PAUSE); svc_gr_hold_bound_expired() pushes the per-launch bound (GR-HOLD-BOUND-PAUSE); and the busy-defer one-shot window re-holds while the flag is set so the consume-with-sw-release path cannot run during a quiet hold. The held launch is then forwarded and copied out after the hold, and its completion is authored only on the hold's done edge - after the data write-back.
* gr: r411 sticky wedge episode - quiet the first retry of any zero-write ↵Arthur Hanson Rasmusson41 hours
| | | | | | | | | | | | | | launch once the boot has seen a quiet heal (or a no-write accept) r418 with the r410 quiet path showed the wedge alternating one launch at a time: every second push no-wrote and was accepted (streak was reset by the healed launch in between, so the quiet trigger never armed), e.g. c584 NOWRITE / c92c healed / d424 NOWRITE / d7cc healed / ... 6 holds, 6 heals, 7 accepted no-writes. A zero-write accept (or a successful quiet heal) now arms a sticky wedge episode for the boot; any later zero-write launch takes the quiet path on its first retry, so in an episode every wedged launch either heals (engine writes, client data correct) or is classified loudly instead of riding through the quick-retry accept.
* gr: default wedge quiet window 40 s (r417 live: a 20 s host hold did not ↵Arthur Hanson Rasmusson41 hours
| | | | | | | | | | | | | | | heal, the next 40 s hold did - legit engine writes resumed) r417 reproduced the wedge deterministically with wedge_recover v2 (ITER 0 green, ITER 1 zero-write, both with the legacy win0 shadow path on and with the icache-invalidate bit forced off - the bit is not the trigger) and exercised the r410 quiet path for the first time: the 20 s hold left the outcome at 0 pages, the following 40 s hold logged 'WEDGE-QUIET healed: engine wrote 1 page(s) after the 40000 ms hold (streak was 1)'. A shadow-win0=0 A/B showed the engine executes but fetches the wrong program in that legacy shape (all-32768 wrong), so the win0 shadow path stays on. Default base raised to 40 s so the first hold heals instead of paying 20 s + 40 s.
* gr: r410 wedge quiet window - bounded idle hold before retrying a lane whose ↵Arthur Hanson Rasmusson42 hours
| | | | | | | | | | | | | | | | | | | | | previous launch also no-wrote (r416: 58-launch sticky wedge clears after ~50 s quiet); wedge counters in state; wedge_recover v2 unmask The r416 boot reproduced the sticky no-execute wedge inside 60 s: 58 consecutive launches accepted zero writes while the lane stayed busy (wedge_recover v1's deterministic verify masked every one of them), and the first launches were green again only after the lane sat idle ~52 s. Both r409 (~37 s idle heal) and r416 therefore show the wedge clears on idleness and persists under back-to-back retries (r154/r409), so the retry loop now holds a quiet window (gsp_phys_gr_wedge_quiet_ms, default 20000, doubling on the second quiet retry, 0 disables) before the normal fresh-bank + resched retry when the previous launch also no-wrote. The whole-launch stall budget is relaxed only on this bounded recovery path. NOWRITE lines now carry the consecutive streak and quiet/heal counters, and the state file shows nw_streak/wedge_quiet/wedge_heals. wedge_recover v2 mutates the inputs every iteration so a stale (never executed) launch can no longer match the expected values, and classifies the readback STALE-PREV(NOWRITE) vs OTHER/PARTIAL.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-23 09:23:40 +0000