summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAge
* wall2: session 88 - push page-5 injection fix (proven +30/sweep), ↵Arthur Hanson Rasmusson7 hours
| | | | host_resolve tool, RING GAP diagnosis (gpfifo done=0), gdb-SIGKILL RM-lock wedge lesson
* wall-2 session-87: window page-list hunt - the 2 GiB cycle's windows are ↵Arthur Hanson Rasmusson7 hours
| | | | scattered (later offsets at first_PA+off are zeros); the memdesc {size 0x20000, PA=0x107c47000, kptrs} page list is only reachable via the kernel pointers (kcore walk with fresh addresses needed); content search for the init-stream prefix found the window pages (0x103c06000+0x858, 0x11dbfa000+0x858, 0x126d69000+0x59c) but the VA<->page relation is non-linear; the client blocks in os_schedule at the allocation (the unread later pushes suspected)
* wall-2 session-86 tools: vk_memprops.py (standalone Vulkan memory-properties ↵Arthur Hanson Rasmusson7 hours
| | | | dump), vk_alloc_test.py (standalone allocation reproducer), t91.py (auto-clear + ICD memprops/alloc hooks), t90.py (A8 count auto-clear)
* wall-2 session-86: the DEVICE_LOCAL -2 wall CRACKED - the 512 MiB emulated ↵Arthur Hanson Rasmusson7 hours
| | | | VRAM was the gate (the ICD hooks captured ALLOC size=0x4000 typeIndex=1 valid; vk_memprops shows device-local heap[0]=512 MiB); the nvidia-nvidia-vgpu-turing-2g mdev type (2 GiB heap, VGPU_MDEV_TYPE env var honoured by start_vm_keep.sh) lets the same client with the same recipe proceed past vkAllocateMemory into a deeper wait (os_schedule, no -2); remaining: the later pushes at va=+0x52d4/+0x586c read pa=0x0 (window page list scattered; injections covered page 1 only) - extend the window injections; tools: vk_memprops.py, vk_alloc_test.py (hits -4 without the harness), t91.py
* wall-2 session-85 BREAKTHROUGH: the NvOgl wait CLEARED - the A8 loop's exit ↵Arthur Hanson Rasmusson8 hours
| | | | needs the session count *(u32*)(lib_base+0x2070b7c) to reach 0 (the engine sessions' completion counter); a direct gdb write of 0 cleared the wait and the client ADVANCED past the post-device phase for the first time (VKDEVICE ok then VKFAIL vkAllocateMemory -> -2 OUT_OF_DEVICE_MEMORY at vk_frame.c:204 - the new wall: the DEVICE_LOCAL heap/type reporting, no alloc RPC reached the emulation)
* wall-2 session-84: the a86038 want-scan mapped - all four entries PASS with ↵Arthur Hanson Rasmusson8 hours
| | | | the driven window value (wants {0,0,0,6} vs actuals {0,0,0,0xffffffff00000100}, no pending); a full-window write (0x0..0x1ff0) does not clear the wait; the residual gate is the A8 loop's internal callback/retry machinery (the a85f30 exit path decrements *(rbp) while firing the global callbacks 0x206f790/0x206f7a0 with args from rbp+0x30/0x38/0x40) and the slow yield-spin cadence (b284b0(...,0xb))
* wall-2 session-83: the A8 completion counter decoded to a client readback ↵Arthur Hanson Rasmusson8 hours
| | | | window - a82f30 -> *(obj+0x19b8)=eglcore+0xb250f0 -> b1bf00 (slow path obj+0x224==0) -> b27c90 reads *(u32*)(*(*(obj+0x1480)+0x58)+idx*16) (idx 0xbb.., min over n=*(u32*)(obj+0x22efc)); the window for handle 0xbeec0100 is VA 0x7827e6c12000 (/dev/nvidiactl) and the host CAN write it via the RAM file: writing 0x100 made the client read it (obj+0x1070=0x100, actual table entry 0xffffffff00000100) - first live proof the counter can be driven from the host; the wait persists in the loop's second scan (a85ec1/b29dc0) and the *(rsp+0x38) branch
* wall-2 session-82: counter-page PA hunt - the client's pagemap does not ↵Arthur Hanson Rasmusson8 hours
| | | | contain the counter pool (guest_pmap_win.py empty for 0x20400f000 and 0x121000000 ranges; semfix_uvmt's pin_counter_page never succeeded) and the vidmem-window hypothesis is not supported (poke_vidmem at the pool window offset 0x3F87000 + rank slots 0xfd0/0xfc0/0xfb0 with 6 then 0x40 changed the shadow but did not move the A8 wait); gdb attach hangs on the post-device spin (kill by PID + SIGCONT recovery)
* wall-2 session-81: counter-autoreg gap analysed - svc_counter_autoreg needs ↵Arthur Hanson Rasmusson8 hours
| | | | an ipin pin or svc_counter_derive for the rank slot (0x20400fec0+(19-r)*0x10); both miss for the counter page 0x20400f000 (the client does not map it - pagemap empty, no VMA; it is a UVM kernel object; the pin stream's sem resolutions are 0 despite 44 want-list entries); bypass: the sema_counter param registers a slot manually (echo <h_object>,<pa>); GL ranks 0,1,2 -> slots 0x20400ffd0/0xfc0/0xfb0
* wall-2 session-80: ALL low-aperture window records found via the 'liat'-tag ↵Arthur Hanson Rasmusson8 hours
| | | | fix (the record tag is the u32 0x7461696c whose on-disk bytes are 'liat', not 'tail'); wall2_recscan.py scans them; the GR init semaphore windows identified and injected (0x4b17000->0x12a76f000 verified by content: +0xbb0=3/+0xe30=1 = the release payloads; 0x4b19000->0x122f9b000, release applied there) - releases now land on the semaphore pages; the NvOgl wait still spins (the A8 loop's per-entry counter tables are the remaining completion condition)
* wall-2 session-79: resolver7 tool + the corrected pair-specific injection ↵Arthur Hanson Rasmusson9 hours
| | | | recipe (0x4480000->0x125692000, 0x4ca0000->0x131649000, 0x4b16000->0x12a76a000) and the window-record layout {tag,VA,SIZE,0,tail} stride 0x40; documents the resolver7 cross-pair bug, the decoded low-VA releases and the remaining semaphore-VA translations (pending=417)
* wall-2 session-78: post-device wait identified as the NvOgl (GL ucode) ↵Arthur Hanson Rasmusson9 hours
| | | | engine init (A8 retry loop work item named NvOgl_m0; D6 submit ORs 0x200000 into the pending word) and the push windows become READABLE via the resolver6 window-record/memdesc recipe (0x4480000 -> 0x120cfb000, 0x4ca0000 -> 0x12e51e000; the service decodes the GR init streams where sessions 73-74 saw unread) - remaining gate: have_rel=0 (CE-shaped matcher) and the GR forward not engaging (forwarded=0 gr_nocap=4 drain held=4)
* wall-2 session-77: vkCreateDevice PASSES - r682 0x7E event validated live ↵Arthur Hanson Rasmusson9 hours
| | | | (registered slot 13, EVPOST rc=0) and the completion word found/driven: the main-thread wait reads a u32 at +0x80 of a 16 KB rw-s /dev/nvidiactl notification mapping (should-wait = word != 1); a host write of 1 via the guest RAM file (PA from pagemap, file offset PA-0x80000000 above 4G) flips the wait and vkCreateDevice returns success (VKDEVICE ok family=0) - reproduced 3x (r683 manual, r684 nudger, r684d probe+single-word). Tools: t79.py (wait-chain resolver), guest_words.sh, nudge.sh; README-session77.md records the recipe, the residual guest event-dispatch gap (records consumed, notification word not written), and the -4/gdb-attach cautions.
* wall-2 session-76 addendum (r681): decode the main-thread vkCreateDevice ↵Arthur Hanson Rasmusson10 hours
| | | | wait - should-wait = (readiness word != 1), readiness = u32 at nvidiactl-mapped holder+0x80 (VM_IO page, 39/39 reads zero, all fd polls time out), give-up predicate dcee20 fires -4 only on RM error flags (b29d60 / +0x70cc); the word's writer is the guest kernel driver/RM, not host RAM
* vgsp-rm r680 + wall-2 session-76 forensics: 0x2080a097 reply record-area ↵Arthur Hanson Rasmusson10 hours
| | | | | | | | | | | | | | | | | | | | tunables (wall2_a097_count/rec0..rec7, default 0xffffffff = echo) and the evidence set that re-questions the a097 gate The real host RM 0x2080a097 handler (GSP-RM 570.144 control table entry, params 0x490) overwrites params+4..+0x23 with up to four {u32,u32} mode pairs (engine value + a x1000 clock/percent) and the client copies back the first 16 bytes; the tick at eglcore 0xd5e3e0 consumes the second word of the first pair as the per-engine progress (session +0xe104, compared against per-boot config threshold 95 at lib+0x2019320+0xffc with 10/20 bit-history cutoffs). New module params let one boot sweep the record matrix; the r680 sweep (flag/v488/v48c + count/rec matrix incl. progress 100..1000000) changed nothing observable, and hw watchpoints show E+0x150 stays 0x00200000 with no writer over 5+ minutes while the a097 poller is a worker thread and the main thread sits in a 10 s poll() loop on /dev/nvidia0 inside vkCreateDevice. Tools: t75g.py (in-guest gdb state/watch probes), diag76.py (thread bt + address resolution), slots_probe.sh, run75g.sh; README-session76.md records the firmware handler decode and the sweep.
* wall-2 session 74: the client's 11s stalls are userspace completion-slot ↵Arthur Hanson Rasmusson11 hours
| | | | | | | | | | | | | | | | | | | | | | spins; the next wall is the polled RM control 0x2080a097 Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield + clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings. Guest-side filling of the slot flags lets the client proceed immediately, but vkCreateDevice still fails -4: with the slots filled the client settles into an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation answers with the generic all-zero reply, so the poll never completes. Also captured: sema_push_raw dumps proving the GR channel-init push contains NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then zeros), and the negative result that neither channelFillGpFifo nor nvWriteGpEntry runs for this channel's kick. New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and vmap_watch2.py (vmap/kcore watchers), README-session74.md.
* wall-2 forensics tools: GR channel-init push localization (session 73)Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | | | | | Tools and findings from the live wall-2 campaign (r674/r675): the vk_frame client's GR channel-init push (va=0x4480000 len=1432) cannot be translated by the sema service, so no release is applied and vkCreateDevice fails -4. Captured: the RM window-record/memdesc layout (ring memdesc PA == the service's validated ring pa; push memdesc PA injected live and read by the service without a false-translation line), the CE-only shape of the service rescue matcher, the notrace status of nvidia/modeset kprobes (QEMU gdb hbreak works), and the ring-entry encoding from channelFillGpFifo/nvWriteGpEntry (neither runs for this channel's kick). New tools: wall2_resolver{2..5}.py, wall2_livecap{1..3}.py, RAM structure scanners (scan_r673*.py, trace_push674.py), gdb675*.gdb probes, plus a README with the ranked next steps.
* docs + nvkmstrace: the GL/Vulkan wall fully bisected (SESSION 65) - ↵Arthur Hanson Rasmusson17 hours
| | | | ALLOC_DEVICE reply status 5 (CORE_CHANNEL_ALLOC_FAILED) decoded by the LD_PRELOAD nvKms logger with a widened reply scan; the empty display capability notifier is the first missing piece (a live-gdb fill of SYS_CAP/SYS_CAPB flips the driver to the window path, proving numWindows>0 unblocks it); the residual wall is the vGPU-guest display-class rejection: nvRmApiAlloc(0xc57e) returns NV_ERR_NOT_SUPPORTED inside the guest kernel and never reaches the emulated GSP - the guest runs in vGPU mode where display/modeset classes are host-service-provided by design
* tools/vk + nvkmstrace: real-Vulkan probe stack for the emulated vGPU - ↵Arthur Hanson Rasmusson19 hours
| | | | vk_probe/vk_probe2 (ctypes loader probes: enumerate + create device/queue), vk_frame (headless Vulkan compute renderer: loads the NVIDIA ICD, paints a plasma into an RGBA8 storage image, reads it back to PPM and publishes frames into the kvmfr ivshmem BAR), vk_plasma.comp -> SPIR-V; nvkmstrace.c (LD_PRELOAD NVKMS ioctl logger that decoded the vkCreateDevice failure to ALLOC_DEVICE cmd=0). Measured: the NVIDIA device enumerates and its Vulkan ICD loads (2 ICDs: NVIDIA + llvmpipe); vkCreateDevice fails VK_ERROR_INITIALIZATION_FAILED because nvKmsAllocDevice returns non-success after the display shim
* tools/guest-diag: GL/Vulkan probe + X capture publisher for the real-render ↵Arthur Hanson Rasmusson20 hours
| | | | path - egl_retry (explicit EGL make-current retry probe: the glsi engine-init returns busy once and eglcore never retries; prints the per-try EGL error), xshm_pub (X11/XShm screen capture into the kvmfr ivshmem BAR in the frame_producer layout so a GL/Vulkan-rendered guest frame reaches the host), xorg-vgpu.conf (headless nvidia Xorg server layout for the emulated vGPU)
* tools r625: stuck_client_capture.sh - one-shot capture for a stuck client ↵Arthur Hanson Rasmusson28 hours
| | | | (pause the trace-consuming scanner, read the retained uvtw/uvsv/uvmt tail, walk the tracker + pool page, resume); wraps the manual sequence that unlocked the first-wait evidence
* tools r624: tracker_walk.py + pool_page.py - guest-side (kcore) walks for ↵Arthur Hanson Rasmusson28 hours
| | | | the first-wait class: read a UVM tracker's semaphore object and expected payload, and dump the semaphore-pool page's applied values; together with the host pool-wr/sema-sw logs they compare what the driver waits for against what the host wrote
* tools r621: read_proc_va.py - read a running client's mapped words via ↵Arthur Hanson Rasmusson28 hours
| | | | /proc/<pid>/mem (the guest-side half of the sem-write divergence comparison; pairs with the r619/r620 sema-sw witness)
* tools r617: battery_resume.sh - zombie-tolerant battery resume ↵Arthur Hanson Rasmusson30 hours
| | | | (verdict-in-log completion check; the plain pgrep -x poll falsely hangs on a client that exited after printing its verdict but left an unreaped zombie comm)
* tools r615: redboot-triage.sh - one-pass evidence extraction for the ↵Arthur Hanson Rasmusson31 hours
| | | | unresolved-arg wedge class (sema-shadow-dst entry point, GRFILL-DROP/held/refused chain, untruncated arg-miss walk lists, cand probes, EVPOST drops)
* r586: demanded staging VAs resolve via the client pagemap; fail-closed at ↵Arthur Hanson Rasmusson43 hours
| | | | | | | | | | | | | | | | | | | | | | | the r543 argument defer cap Two halves of the staging-pin coverage class (r584/r585 boots): 1. guest_pin_stream demand handler: resolve_va_by_content rejects any VA outside the CE pushbuffer window (PB_CE_LO..PB_CE_HI), so the demands for the GRFILL dropped staging pages (high client VAs like 0x703304e00000) were silent no-ops - no pin ever landed and the fills kept dropping them. r586 falls back to the client's own pagemap (scan_range on the exact page), the same walk the window scans use. 2. vgsp-work: the r543 argument-defer cap expiry now takes the r583 fail-closed path instead of forwarding: the r585 boot's saxpy was forwarded with n_data=969 dropped=7 partial=1 at the cap and read stale operands (bad=1047527 csum=130523290); the r584 matmul (n_data=2 dropped=18) reached the r564 degenerate refusal. Refuse, author the QMD release in software (rel_done latched), consume, witness. The dropped pages were demanded first (r585), so the retry window had its chance; a launch that would run with unresolved operands is never silently forwarded.
* tools: transport-proof demand writes + tolerant demand parsing (the pool-pin ↵Arthur Hanson Rasmusson44 hours
| | | | | | | | | | | | | | | | | | | latency root cause) The r581 boot's guest scanner died mid-boot with ValueError in main(): "invalid literal for int() with base 16: '0x20400f000\\n0x121000000\\n0x12000ab4c'". A guest-agent transport-escaped append merged demand VAs onto one line with literal backslash-n, and the old parser crashed on it. The crash stopped ALL new pins; the host only re-asserted known pairs, so the UVM pool VA 0x121000000 never arrived while 185 other injections flowed - the pool-pin latency that blocked every validation boot. * pin_gr_stream._demand_send/_demand_loop now write one "echo <va> >>" per VA (no newlines, no backslashes in the transported command; immune to transport escaping). * guest_pin_stream's demand parser extracts every 0x token with re and skips malformed ones instead of raising out of main().
* vgsp-sema r577: never let a foreign release VA claim a CE channel's tracking ↵Arthur Hanson Rasmusson45 hours
| | | | | | | | | | | | | | | | | | | | | | | | slot; event-record A/B knobs; GR-DEC dump deepened r575 starved-release wedge (live): an unreadable UVM CE entry's rescued stale page decoded a release to a foreign VA (0x12006c004, pay=60); the rescue paths adopted it as the channel's tracking slot (last_rel_va) and every later recovery release targeted it, while the client's real pool (0x121000000) stopped at 25 - the client spun in uvm_tracker_wait for 34 (exact-slot poke delivered it and the client completed green). For CE channels only a pool-shaped release target (0x121000000..0x122000000) may become last_rel_va; a foreign rescued release is refused (witnessed, bounded) and the pool step is delivered through the exact-slot fallback / deferred-release machinery instead. The fake's read-before-write keeps the delivery monotonic even when a legit push carries multiple releases. Also: gsp_rm_ev_{seq,nlist,cl} runtime knobs for the guest-RM event drop A/B (the r576 experiment: guest consumes every POST_EVENT record and never enters nv_post_event), and the GR-DEC launch dump now covers 256 data pages (the mail VADD fill carries n_data=969). New: tools/cuda-ab/evhold.cu - deterministic CUDA event holder/wait probe (context + cudaEventSynchronize + a hold window) for the event experiments.
* tools/cuda-ab: push_battery pushes the full client+probe set (inplace_scale ↵Arthur Hanson Rasmusson2 days
| | | | was never pushed - the battery misreported it as a fail; probes/egl_frame/frame_producer now ride along)
* guest-pin-stream: skip non-readable VMAs in the high-VA scanArthur Hanson Rasmusson2 days
| | | | | | | | | | | | A CUDA device buffer is a PROT_NONE reservation (often hundreds of MB). Its pages are never present in the pagemap, so scanning it yields no pairs - but it consumed the VMA_MAX_PAGES budget, and the scan never reached the client's present staging buffers that sit ABOVE the device reservation. Observed: a client whose device range started below its staging buffer got no high-VA pins at all; its H2D copies failed rc=-2 forever and the client read back stale data (soak run: 9/9 green then streams2 bad=32737). Skipping '---' VMAs restores the pins (9 fresh high-VA injections in the first run; streams2 bad=0).
* Revert "vgsp-sema: keep the learned VA map across the session flush ↵Arthur Hanson Rasmusson2 days
| | | | | | (re-compact, not wipe)" This reverts commit 1a49c7eb69c03f58bc51939828de222a2ac42fea.
* vgsp-sema: keep the learned VA map across the session flush (re-compact, not ↵Arthur Hanson Rasmusson2 days
| | | | | | | | | | | | | | | | | | wipe) The flush fires when the channel table momentarily empties - a live client's UVM channel cycling triggers it - and wiping the map left the client's already-resolved data VAs unresolvable until a rebuild scan re-found them, which it may not. Evidence: a mid-battery boot where multikernel's operands went NO ENTRY 40+ s after a flush (the rebuild restored 14-31 K entries without them); the pump stalled for minutes and the next clients read back whole-page garbage. The map is a cache with a merge-on-rebuild contract, so re-compact it (dedup + sort) and keep it as the merge base; reused-VA correctness stays with the ledger landing, the pin-wins rule and the fresh scan. Also: push_battery.sh now pushes inplace_scale (the battery list had it, the push list did not).
* tools/cuda-ab/run/tgemm_probe: rebuild from tgemm_probe.cu (the committed ↵Arthur Hanson Rasmusson3 days
| | | | | | | | | | | binary was a copy of streams_events_probe) The run/tgemm_probe artifact had the same md5 as streams_events_probe, so every 'tgemm probe' invocation so far ran the streams probe instead. The rebuilt binary reproduces the transformer class on the r518 boot: 'PASS name=k pass=2 bad=256 xbad=2 first=(0,0)' plus the XDIFF rows that show the input's first two floats shifted by one pair (X[0..1] read as X[2..3]) - the deterministic first-page corruption.
* tools/cuda-ab/run: the egl_frame guest GL probe binary (committed for the GL ↵Arthur Hanson Rasmusson3 days
| | | | work: EGL enumeration/MakeCurrent wall reproduction)
* r481c sema_push_raw: lower the call-site threshold to 512 bytes so launch ↵Arthur Hanson Rasmusson3 days
| | | | pushes (872/888B) dump; runtime-enable before the client run to avoid boot noise
* tools/cuda-ab/run: the r458-r459 probe binaries (memset2d_probe, ↵Arthur Hanson Rasmusson3 days
| | | | streams_events_probe, tgemm_probe)
* tools/cuda-ab: isolation probes from the transformer/streams_events forensicsArthur Hanson Rasmusson3 days
| | | | | | | | | | | | | | | - tgemm_probe: minimal X[64x256]xW[256x256] GEMM isolation. Found the deterministic first-page corruption: after a forwarded launch (and sometimes already after the H2D upload) the FIRST 8 bytes of the first data-window page are replaced by that page's bytes at offset 8 (X[0..1] <- X[2..3]; vidmem dump confirms; OUT-UNCH shows chunk==guest), which corrupts GEMM row 0 for all 256 columns (bad=256/pass, identical across 3 passes). Not rel_mod-dependent (A/B at runtime). - streams_events_probe: per-buffer DPTRS + per-buffer bad + first-8 expected/actual for the 3-stream event chain (green on a clean boot, bad=131072 after several clients - the client-switch resolution class). - memset2d_probe: per-phase bad/PHIST for victory_memset2d (verified the r458 LINE_COUNT fix: bad_total=0, all three phases).
* pin_gr_stream: the demand-loop regex never matched the kernel's stalled-VA ↵Arthur Hanson Rasmusson3 days
| | | | | | line (it expected 'stalled VA #N=0x...' but the kernel logs 'stalled VA #N: 0x...'), so host->guest demand pins never fired for stalled VAs and a client whose push/operand pages the resolver missed stayed stalled for the rest of the boot. Accept ':' or '=' after the entry number. Evidence: r463/r464 stalled clients log 'stalled VA #16: 0x755813400000' etc. while the host demand loop only ever emitted pool VAs (0x12100000x) to /tmp/guest_want_vas.txt.
* tools/cuda-ab/run: add the r451-r453 probe binaries (ip_probe*, ip4/5/6, ↵Arthur Hanson Rasmusson4 days
| | | | streams2_probe, scale_probe, rt_probe) - the suite that cracked the additive-csum collision and now serves as the regression detector for the reader-side resolution class
* tools/cuda-ab: the r451-r453 probe suite that cracked the deterministic ↵Arthur Hanson Rasmusson4 days
| | | | class - ip_probe (CTA-marker in-place scale), ip_probe2 (scale_if/scale_u/fill_c/add_c staged matrix), ip_probe3 (what the SM actually reads), streams2_probe (4 stream-ordering modes), scale_probe (in-place vs out-of-place), rt_probe (memcpy_roundtrip staged forensics); run them with setsid+poll, never timeout-kill mid-flight
* tools/soak.sh: fix the present-clients probe (the for-loop's exit status was ↵Arthur Hanson Rasmusson4 days
| | | | 1 whenever the last listed client was absent, so a partial battery always failed with 'no-client-binaries-in-/tmp'); append true inside the probe
* r441 zero the guest-visible release words on the FIRST attempt too (not only ↵Arthur Hanson Rasmusson4 days
| | | | outcome retries): the r439 evidence shows a first attempt breaking its poll on a stale value (qmd_rel=0x5 from the previous launch's engine write, want=6) while the engine never fetched the deposit (sentinel intact) - the host then shipped a completion for a launch that did not run; zeroing forces the poll to wait for this launch's own engine write, so the timeout ladder and the software-authored completion are the honest path and the wedge classification can no longer be satisfied by a stale value
* r436 full sysmem PTE aperture/VOL matrix: gsp_phys_rmcall_pte_coh becomes a ↵Arthur Hanson Rasmusson4 days
| | | | 4-mode selector (0=0xf aperture3+VOL historical, 1=0xd aperture2+VOL r431, 2=0x5 aperture2 non-volatile, 3=0x7 aperture3 non-volatile) applied consistently to the leaf PTE and the single PDE (bit0 = IS_PDE there), so the r431 finding (aperture 2 with VOL still 1 is a pair the reference RM never emits) can be A/B'd against the consistent encodings
* tools: frame_reader picks the largest kvmfr shm (a stray empty 'kvmfr-*' ↵Arthur Hanson Rasmusson4 days
| | | | file existed); wedge_ab same-program vs second-launch discriminator
* tools/lg + docs: guest frame producer path (CUDA -> ivshmem -> host) and the ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | Looking Glass status - tools/cuda-ab/frame_producer.cu: renders an animated Julia frame on the guest GPU and publishes it into the ivshmem BAR (the kvmfr 128 MiB window) with a small header + RGBA payload. - tools/lg/frame_reader.py: host-side reader of /dev/shm/kvmfr-<uuid> that saves each published frame as a PPM (the transport half of the frame path). - tools/guest-diag/egl_frame.c: EGL probe (device/surfaceless/default platforms) - live result: eglQueryDevicesEXT sees the NVIDIA device and eglInitialize succeeds, but eglMakeCurrent fails EGL_BAD_ALLOC (0x3003), the same VASpace-allocation wall as Xorg's 'Failed to allocate VASpace object'. - docs/05-operations/06: the state, the blockers and the exact remaining pieces (guest GL VASpace allocation; LGMP session producer).
* tools/cuda-ab: wedge_bisect - bisect the no-execute wedge trigger by ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | inter-launch activity Modes A/B/C/D add none / an extra H2D / an extra D2H / a 500 ms sleep between the verify and the next launch of the wedge_recover shape. The inputs mutate every iteration, so a no-execute launch is reported (bad>0), never masked by deterministic content.
* gr: r420b author the guest QMD release on every outcome when release0 is ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | | | redirected With gsp_phys_rmcall_gr_rel_mod=1 (release0 -> module semaphore, the ordering fix for the engine-release-before-copy-out race) the engine never writes the guest's own QMD release word, so the host must author it on EVERY backend outcome. The hold's success/fail edges do; a launch with no deferred release records (no hold engaged) fell through and its client hung on a word nothing would ever write - the r414 saxpy f0 cur=5 stall. The backend now reports qmd_rel_redirected in the copy result whenever it points release0 at the module semaphore (rel_mod, or no targetable guest release page). svc_fwd_gr_work folds that into the existing gr_rel_unobserved latch, which makes the bounded outcome check author the word on done (success or failure) - the same forward-only write the hold edges perform. rel_mod=0 paths are unchanged (the engine still authors the word and the latch is only set on an unresolved release page).
* gr: r410 wedge quiet window - bounded idle hold before retrying a lane whose ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | | | | | previous launch also no-wrote (r416: 58-launch sticky wedge clears after ~50 s quiet); wedge counters in state; wedge_recover v2 unmask The r416 boot reproduced the sticky no-execute wedge inside 60 s: 58 consecutive launches accepted zero writes while the lane stayed busy (wedge_recover v1's deterministic verify masked every one of them), and the first launches were green again only after the lane sat idle ~52 s. Both r409 (~37 s idle heal) and r416 therefore show the wedge clears on idleness and persists under back-to-back retries (r154/r409), so the retry loop now holds a quiet window (gsp_phys_gr_wedge_quiet_ms, default 20000, doubling on the second quiet retry, 0 disables) before the normal fresh-bank + resched retry when the previous launch also no-wrote. The whole-launch stall budget is relaxed only on this bounded recovery path. NOWRITE lines now carry the consecutive streak and quiet/heal counters, and the state file shows nw_streak/wedge_quiet/wedge_heals. wedge_recover v2 mutates the inputs every iteration so a stale (never executed) launch can no longer match the expected values, and classifies the readback STALE-PREV(NOWRITE) vs OTHER/PARTIAL.
* tools/cuda-ab: memset_probe + wedge_recover forensics clients (session 10)Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | memset_probe: linear 8 MB cudaMemset + D2H with first-bad offset and a 4 KB page map (correct / other / malloc-prefill) - separates 'fill under-delivered' from 'readback delivered wrong pages'; standalone green, bisects the memcpy_roundtrip memset phase. wedge_recover: launch+verify loop that on failure sleeps N seconds and retries the same iteration (same context), reporting per-iteration attempts and launch times - tests whether an idle gap heals the per-boot no-execute wedge (r409 log shows a ~37 s client-side gap healed a wedged run). Also ships the missing inplace_scale run binary (its source landed in 7685d1b3).
* tools/cuda-ab: inplace_scale - minimal in-place scale repro for the r407 ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | drop-stale release race One H2D, one in-place scale launch (128 KB = 32 pages), one D2H, per-page diff. The client made the r406/r407 stale-release race reproducible in isolation: with the drop-stale fix the same binary runs 6/6 + 4/4 green across two fresh boots (was intermittent 2/3 before).
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-25 21:19:32 +0000