summaryrefslogtreecommitdiffstats
path: root/tools/guest-diag
Commit message (Collapse)AuthorAge
* wall2: session 88 - push page-5 injection fix (proven +30/sweep), ↵Arthur Hanson Rasmusson9 hours
| | | | host_resolve tool, RING GAP diagnosis (gpfifo done=0), gdb-SIGKILL RM-lock wedge lesson
* wall-2 session-87: window page-list hunt - the 2 GiB cycle's windows are ↵Arthur Hanson Rasmusson9 hours
| | | | scattered (later offsets at first_PA+off are zeros); the memdesc {size 0x20000, PA=0x107c47000, kptrs} page list is only reachable via the kernel pointers (kcore walk with fresh addresses needed); content search for the init-stream prefix found the window pages (0x103c06000+0x858, 0x11dbfa000+0x858, 0x126d69000+0x59c) but the VA<->page relation is non-linear; the client blocks in os_schedule at the allocation (the unread later pushes suspected)
* wall-2 session-86 tools: vk_memprops.py (standalone Vulkan memory-properties ↵Arthur Hanson Rasmusson9 hours
| | | | dump), vk_alloc_test.py (standalone allocation reproducer), t91.py (auto-clear + ICD memprops/alloc hooks), t90.py (A8 count auto-clear)
* wall-2 session-86: the DEVICE_LOCAL -2 wall CRACKED - the 512 MiB emulated ↵Arthur Hanson Rasmusson9 hours
| | | | VRAM was the gate (the ICD hooks captured ALLOC size=0x4000 typeIndex=1 valid; vk_memprops shows device-local heap[0]=512 MiB); the nvidia-nvidia-vgpu-turing-2g mdev type (2 GiB heap, VGPU_MDEV_TYPE env var honoured by start_vm_keep.sh) lets the same client with the same recipe proceed past vkAllocateMemory into a deeper wait (os_schedule, no -2); remaining: the later pushes at va=+0x52d4/+0x586c read pa=0x0 (window page list scattered; injections covered page 1 only) - extend the window injections; tools: vk_memprops.py, vk_alloc_test.py (hits -4 without the harness), t91.py
* wall-2 session-85 BREAKTHROUGH: the NvOgl wait CLEARED - the A8 loop's exit ↵Arthur Hanson Rasmusson10 hours
| | | | needs the session count *(u32*)(lib_base+0x2070b7c) to reach 0 (the engine sessions' completion counter); a direct gdb write of 0 cleared the wait and the client ADVANCED past the post-device phase for the first time (VKDEVICE ok then VKFAIL vkAllocateMemory -> -2 OUT_OF_DEVICE_MEMORY at vk_frame.c:204 - the new wall: the DEVICE_LOCAL heap/type reporting, no alloc RPC reached the emulation)
* wall-2 session-84: the a86038 want-scan mapped - all four entries PASS with ↵Arthur Hanson Rasmusson10 hours
| | | | the driven window value (wants {0,0,0,6} vs actuals {0,0,0,0xffffffff00000100}, no pending); a full-window write (0x0..0x1ff0) does not clear the wait; the residual gate is the A8 loop's internal callback/retry machinery (the a85f30 exit path decrements *(rbp) while firing the global callbacks 0x206f790/0x206f7a0 with args from rbp+0x30/0x38/0x40) and the slow yield-spin cadence (b284b0(...,0xb))
* wall-2 session-83: the A8 completion counter decoded to a client readback ↵Arthur Hanson Rasmusson10 hours
| | | | window - a82f30 -> *(obj+0x19b8)=eglcore+0xb250f0 -> b1bf00 (slow path obj+0x224==0) -> b27c90 reads *(u32*)(*(*(obj+0x1480)+0x58)+idx*16) (idx 0xbb.., min over n=*(u32*)(obj+0x22efc)); the window for handle 0xbeec0100 is VA 0x7827e6c12000 (/dev/nvidiactl) and the host CAN write it via the RAM file: writing 0x100 made the client read it (obj+0x1070=0x100, actual table entry 0xffffffff00000100) - first live proof the counter can be driven from the host; the wait persists in the loop's second scan (a85ec1/b29dc0) and the *(rsp+0x38) branch
* wall-2 session-82: counter-page PA hunt - the client's pagemap does not ↵Arthur Hanson Rasmusson10 hours
| | | | contain the counter pool (guest_pmap_win.py empty for 0x20400f000 and 0x121000000 ranges; semfix_uvmt's pin_counter_page never succeeded) and the vidmem-window hypothesis is not supported (poke_vidmem at the pool window offset 0x3F87000 + rank slots 0xfd0/0xfc0/0xfb0 with 6 then 0x40 changed the shadow but did not move the A8 wait); gdb attach hangs on the post-device spin (kill by PID + SIGCONT recovery)
* wall-2 session-81: counter-autoreg gap analysed - svc_counter_autoreg needs ↵Arthur Hanson Rasmusson10 hours
| | | | an ipin pin or svc_counter_derive for the rank slot (0x20400fec0+(19-r)*0x10); both miss for the counter page 0x20400f000 (the client does not map it - pagemap empty, no VMA; it is a UVM kernel object; the pin stream's sem resolutions are 0 despite 44 want-list entries); bypass: the sema_counter param registers a slot manually (echo <h_object>,<pa>); GL ranks 0,1,2 -> slots 0x20400ffd0/0xfc0/0xfb0
* wall-2 session-80: ALL low-aperture window records found via the 'liat'-tag ↵Arthur Hanson Rasmusson11 hours
| | | | fix (the record tag is the u32 0x7461696c whose on-disk bytes are 'liat', not 'tail'); wall2_recscan.py scans them; the GR init semaphore windows identified and injected (0x4b17000->0x12a76f000 verified by content: +0xbb0=3/+0xe30=1 = the release payloads; 0x4b19000->0x122f9b000, release applied there) - releases now land on the semaphore pages; the NvOgl wait still spins (the A8 loop's per-entry counter tables are the remaining completion condition)
* wall-2 session-79: resolver7 tool + the corrected pair-specific injection ↵Arthur Hanson Rasmusson11 hours
| | | | recipe (0x4480000->0x125692000, 0x4ca0000->0x131649000, 0x4b16000->0x12a76a000) and the window-record layout {tag,VA,SIZE,0,tail} stride 0x40; documents the resolver7 cross-pair bug, the decoded low-VA releases and the remaining semaphore-VA translations (pending=417)
* wall-2 session-78: post-device wait identified as the NvOgl (GL ucode) ↵Arthur Hanson Rasmusson11 hours
| | | | engine init (A8 retry loop work item named NvOgl_m0; D6 submit ORs 0x200000 into the pending word) and the push windows become READABLE via the resolver6 window-record/memdesc recipe (0x4480000 -> 0x120cfb000, 0x4ca0000 -> 0x12e51e000; the service decodes the GR init streams where sessions 73-74 saw unread) - remaining gate: have_rel=0 (CE-shaped matcher) and the GR forward not engaging (forwarded=0 gr_nocap=4 drain held=4)
* wall-2 session-77: vkCreateDevice PASSES - r682 0x7E event validated live ↵Arthur Hanson Rasmusson11 hours
| | | | (registered slot 13, EVPOST rc=0) and the completion word found/driven: the main-thread wait reads a u32 at +0x80 of a 16 KB rw-s /dev/nvidiactl notification mapping (should-wait = word != 1); a host write of 1 via the guest RAM file (PA from pagemap, file offset PA-0x80000000 above 4G) flips the wait and vkCreateDevice returns success (VKDEVICE ok family=0) - reproduced 3x (r683 manual, r684 nudger, r684d probe+single-word). Tools: t79.py (wait-chain resolver), guest_words.sh, nudge.sh; README-session77.md records the recipe, the residual guest event-dispatch gap (records consumed, notification word not written), and the -4/gdb-attach cautions.
* wall-2 session-76 addendum (r681): decode the main-thread vkCreateDevice ↵Arthur Hanson Rasmusson12 hours
| | | | wait - should-wait = (readiness word != 1), readiness = u32 at nvidiactl-mapped holder+0x80 (VM_IO page, 39/39 reads zero, all fd polls time out), give-up predicate dcee20 fires -4 only on RM error flags (b29d60 / +0x70cc); the word's writer is the guest kernel driver/RM, not host RAM
* vgsp-rm r680 + wall-2 session-76 forensics: 0x2080a097 reply record-area ↵Arthur Hanson Rasmusson13 hours
| | | | | | | | | | | | | | | | | | | | tunables (wall2_a097_count/rec0..rec7, default 0xffffffff = echo) and the evidence set that re-questions the a097 gate The real host RM 0x2080a097 handler (GSP-RM 570.144 control table entry, params 0x490) overwrites params+4..+0x23 with up to four {u32,u32} mode pairs (engine value + a x1000 clock/percent) and the client copies back the first 16 bytes; the tick at eglcore 0xd5e3e0 consumes the second word of the first pair as the per-engine progress (session +0xe104, compared against per-boot config threshold 95 at lib+0x2019320+0xffc with 10/20 bit-history cutoffs). New module params let one boot sweep the record matrix; the r680 sweep (flag/v488/v48c + count/rec matrix incl. progress 100..1000000) changed nothing observable, and hw watchpoints show E+0x150 stays 0x00200000 with no writer over 5+ minutes while the a097 poller is a worker thread and the main thread sits in a 10 s poll() loop on /dev/nvidia0 inside vkCreateDevice. Tools: t75g.py (in-guest gdb state/watch probes), diag76.py (thread bt + address resolution), slots_probe.sh, run75g.sh; README-session76.md records the firmware handler decode and the sweep.
* wall-2 session 74: the client's 11s stalls are userspace completion-slot ↵Arthur Hanson Rasmusson14 hours
| | | | | | | | | | | | | | | | | | | | | | spins; the next wall is the polled RM control 0x2080a097 Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield + clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings. Guest-side filling of the slot flags lets the client proceed immediately, but vkCreateDevice still fails -4: with the slots filled the client settles into an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation answers with the generic all-zero reply, so the poll never completes. Also captured: sema_push_raw dumps proving the GR channel-init push contains NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then zeros), and the negative result that neither channelFillGpFifo nor nvWriteGpEntry runs for this channel's kick. New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and vmap_watch2.py (vmap/kcore watchers), README-session74.md.
* wall-2 forensics tools: GR channel-init push localization (session 73)Arthur Hanson Rasmusson14 hours
| | | | | | | | | | | | | | | | | Tools and findings from the live wall-2 campaign (r674/r675): the vk_frame client's GR channel-init push (va=0x4480000 len=1432) cannot be translated by the sema service, so no release is applied and vkCreateDevice fails -4. Captured: the RM window-record/memdesc layout (ring memdesc PA == the service's validated ring pa; push memdesc PA injected live and read by the service without a false-translation line), the CE-only shape of the service rescue matcher, the notrace status of nvidia/modeset kprobes (QEMU gdb hbreak works), and the ring-entry encoding from channelFillGpFifo/nvWriteGpEntry (neither runs for this channel's kick). New tools: wall2_resolver{2..5}.py, wall2_livecap{1..3}.py, RAM structure scanners (scan_r673*.py, trace_push674.py), gdb675*.gdb probes, plus a README with the ranked next steps.
* docs + nvkmstrace: the GL/Vulkan wall fully bisected (SESSION 65) - ↵Arthur Hanson Rasmusson19 hours
| | | | ALLOC_DEVICE reply status 5 (CORE_CHANNEL_ALLOC_FAILED) decoded by the LD_PRELOAD nvKms logger with a widened reply scan; the empty display capability notifier is the first missing piece (a live-gdb fill of SYS_CAP/SYS_CAPB flips the driver to the window path, proving numWindows>0 unblocks it); the residual wall is the vGPU-guest display-class rejection: nvRmApiAlloc(0xc57e) returns NV_ERR_NOT_SUPPORTED inside the guest kernel and never reaches the emulated GSP - the guest runs in vGPU mode where display/modeset classes are host-service-provided by design
* tools/vk + nvkmstrace: real-Vulkan probe stack for the emulated vGPU - ↵Arthur Hanson Rasmusson21 hours
| | | | vk_probe/vk_probe2 (ctypes loader probes: enumerate + create device/queue), vk_frame (headless Vulkan compute renderer: loads the NVIDIA ICD, paints a plasma into an RGBA8 storage image, reads it back to PPM and publishes frames into the kvmfr ivshmem BAR), vk_plasma.comp -> SPIR-V; nvkmstrace.c (LD_PRELOAD NVKMS ioctl logger that decoded the vkCreateDevice failure to ALLOC_DEVICE cmd=0). Measured: the NVIDIA device enumerates and its Vulkan ICD loads (2 ICDs: NVIDIA + llvmpipe); vkCreateDevice fails VK_ERROR_INITIALIZATION_FAILED because nvKmsAllocDevice returns non-success after the display shim
* tools/guest-diag: GL/Vulkan probe + X capture publisher for the real-render ↵Arthur Hanson Rasmusson22 hours
| | | | path - egl_retry (explicit EGL make-current retry probe: the glsi engine-init returns busy once and eglcore never retries; prints the per-try EGL error), xshm_pub (X11/XShm screen capture into the kvmfr ivshmem BAR in the frame_producer layout so a GL/Vulkan-rendered guest frame reaches the host), xorg-vgpu.conf (headless nvidia Xorg server layout for the emulated vGPU)
* tools r625: stuck_client_capture.sh - one-shot capture for a stuck client ↵Arthur Hanson Rasmusson30 hours
| | | | (pause the trace-consuming scanner, read the retained uvtw/uvsv/uvmt tail, walk the tracker + pool page, resume); wraps the manual sequence that unlocked the first-wait evidence
* tools r624: tracker_walk.py + pool_page.py - guest-side (kcore) walks for ↵Arthur Hanson Rasmusson30 hours
| | | | the first-wait class: read a UVM tracker's semaphore object and expected payload, and dump the semaphore-pool page's applied values; together with the host pool-wr/sema-sw logs they compare what the driver waits for against what the host wrote
* tools r621: read_proc_va.py - read a running client's mapped words via ↵Arthur Hanson Rasmusson31 hours
| | | | /proc/<pid>/mem (the guest-side half of the sem-write divergence comparison; pairs with the r619/r620 sema-sw witness)
* tools r615: redboot-triage.sh - one-pass evidence extraction for the ↵Arthur Hanson Rasmusson33 hours
| | | | unresolved-arg wedge class (sema-shadow-dst entry point, GRFILL-DROP/held/refused chain, untruncated arg-miss walk lists, cand probes, EVPOST drops)
* r586: demanded staging VAs resolve via the client pagemap; fail-closed at ↵Arthur Hanson Rasmusson45 hours
| | | | | | | | | | | | | | | | | | | | | | | the r543 argument defer cap Two halves of the staging-pin coverage class (r584/r585 boots): 1. guest_pin_stream demand handler: resolve_va_by_content rejects any VA outside the CE pushbuffer window (PB_CE_LO..PB_CE_HI), so the demands for the GRFILL dropped staging pages (high client VAs like 0x703304e00000) were silent no-ops - no pin ever landed and the fills kept dropping them. r586 falls back to the client's own pagemap (scan_range on the exact page), the same walk the window scans use. 2. vgsp-work: the r543 argument-defer cap expiry now takes the r583 fail-closed path instead of forwarding: the r585 boot's saxpy was forwarded with n_data=969 dropped=7 partial=1 at the cap and read stale operands (bad=1047527 csum=130523290); the r584 matmul (n_data=2 dropped=18) reached the r564 degenerate refusal. Refuse, author the QMD release in software (rel_done latched), consume, witness. The dropped pages were demanded first (r585), so the retry window had its chance; a launch that would run with unresolved operands is never silently forwarded.
* tools: transport-proof demand writes + tolerant demand parsing (the pool-pin ↵Arthur Hanson Rasmusson46 hours
| | | | | | | | | | | | | | | | | | | latency root cause) The r581 boot's guest scanner died mid-boot with ValueError in main(): "invalid literal for int() with base 16: '0x20400f000\\n0x121000000\\n0x12000ab4c'". A guest-agent transport-escaped append merged demand VAs onto one line with literal backslash-n, and the old parser crashed on it. The crash stopped ALL new pins; the host only re-asserted known pairs, so the UVM pool VA 0x121000000 never arrived while 185 other injections flowed - the pool-pin latency that blocked every validation boot. * pin_gr_stream._demand_send/_demand_loop now write one "echo <va> >>" per VA (no newlines, no backslashes in the transported command; immune to transport escaping). * guest_pin_stream's demand parser extracts every 0x token with re and skips malformed ones instead of raising out of main().
* guest-pin-stream: skip non-readable VMAs in the high-VA scanArthur Hanson Rasmusson2 days
| | | | | | | | | | | | A CUDA device buffer is a PROT_NONE reservation (often hundreds of MB). Its pages are never present in the pagemap, so scanning it yields no pairs - but it consumed the VMA_MAX_PAGES budget, and the scan never reached the client's present staging buffers that sit ABOVE the device reservation. Observed: a client whose device range started below its staging buffer got no high-VA pins at all; its H2D copies failed rc=-2 forever and the client read back stale data (soak run: 9/9 green then streams2 bad=32737). Skipping '---' VMAs restores the pins (9 fresh high-VA injections in the first run; streams2 bad=0).
* pin_gr_stream: the demand-loop regex never matched the kernel's stalled-VA ↵Arthur Hanson Rasmusson3 days
| | | | | | line (it expected 'stalled VA #N=0x...' but the kernel logs 'stalled VA #N: 0x...'), so host->guest demand pins never fired for stalled VAs and a client whose push/operand pages the resolver missed stayed stalled for the rest of the boot. Accept ':' or '=' after the entry number. Evidence: r463/r464 stalled clients log 'stalled VA #16: 0x755813400000' etc. while the host demand loop only ever emitted pool VAs (0x12100000x) to /tmp/guest_want_vas.txt.
* tools/lg + docs: guest frame producer path (CUDA -> ivshmem -> host) and the ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | Looking Glass status - tools/cuda-ab/frame_producer.cu: renders an animated Julia frame on the guest GPU and publishes it into the ivshmem BAR (the kvmfr 128 MiB window) with a small header + RGBA payload. - tools/lg/frame_reader.py: host-side reader of /dev/shm/kvmfr-<uuid> that saves each published frame as a PPM (the transport half of the frame path). - tools/guest-diag/egl_frame.c: EGL probe (device/surfaceless/default platforms) - live result: eglQueryDevicesEXT sees the NVIDIA device and eglInitialize succeeds, but eglMakeCurrent fails EGL_BAD_ALLOC (0x3003), the same VASpace-allocation wall as Xorg's 'Failed to allocate VASpace object'. - docs/05-operations/06: the state, the blockers and the exact remaining pieces (guest GL VASpace allocation; LGMP session producer).
* tools: prompt re-assert after recv chunks, heartbeat inside long QMP loops, ↵Arthur Hanson Rasmusson8 days
| | | | novel-only activity accounting (tools-fix a4840952)
* tools: kill-cascade purge + stall watchdogs + monotonic client identity ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | (tools-fix c8fb5377) Dead-pid purge via in-band #nopid (zombie-aware probe), monotonic (pid,starttime) client identity so trace-ring replays can never reset an active client (fixes the m178 mid-run reset regression avenue), pre-capture and post-capture stall watchdogs that re-assert pins and escalate loudly, heartbeat threads for both tools, supervisor STALE 75s/GRACE 120s.
* tools: revert m178 latching changes (client-#3 hang regression); kernel m178 ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | | stays The m178 tool latching changes caused a reproducible pre-capture hang of the third CUDA client after two green clients (r199: smiley #3; r200: multikernel #3; both recovered by reverting tools only, r201). Prime suspect: semfix per-client reset re-firing mid-run (same pid reset twice 2 min apart while the client was active). Tools restored to the m177-proven state until the reset logic is fixed with a monotonic per-pid guard.
* vgsp: m178 GR launch reliability + diagnostics; tools: pin latching robustnessArthur Hanson Rasmusson8 days
| | | | | | | | | | | | | Kernel: authorship instrumentation + swrel gate for in-flight GR release VAs; first-poll shortening + fast extended re-arm (params gr_retry_fast=1, gr_first_poll_ms=2500); per-attempt win0 bank recompute + shadow recopy in the outcome-retry loop; poll-break/deposit-readback diagnostics; C1-C10 sema diagnostics (crel-entry, relx, opdef-clear, walk, pub-adv, hold-fallback, busy-defer, CAND-OVERWRITE, inline recs); busy-defer hold instead of software-release consume; counter fast path; QMD word 48/49 skip. Tools: heartbeat thread, scanner-alive probe on reinstall, full periodic re-assert, #nopid handling, per-client reset fix (debounce), dead-pid drop, supervisor grace+threshold.
* vgsp: m177 pin-free self-sufficiency wave + tool hardeningArthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | (1) svc_session_flush keeps svc.direct_base (per-boot constant; the pin-free targeted scanner's gate/input) and a new gsp_sema_service_guest_reset() hook wipes boot-scoped discovery state on guest RM reload; (2) counter-page derive now takes the first READABLE source (vaspaces -> discovered -> targeted) instead of failing -EIO on a walked-but-garbage page, with one-shot diagnostics; (3) new m177c UVM ring-space directory discovery: content-verified (push_va -> push_pa) samples from the GR content discovery and CE push pairing feed a second discovered-directory slot (walks the 0x2xxxxxxx family), consulted by svc_walk_discovered; (4) GR channels now deliver withheld opdef sets at entry consume (normal tail and the stale-skip path) instead of waiting for the 10 s sweep; (5) host tools hardened: semfix_uvmt QMP reconnect + pid-generation dedupe + stale-sem eviction + post-success pin latch + heartbeats; pin_gr_stream accept timeout + guest-scanner reinstall backoff + restart loop + subprocess timeouts; new pin_supervisor.sh (heartbeat watchdog, capped restarts) wired into start_vm_keep.sh. Evidence: r194-r196 cycles; pool resolves src=trusted pin-free; ring dir 0x10801c000 found 4/4; row: 'rel-apply ... written' with tools up.
* vgsp: m168 revert to session scope - the pool page is per-CLIENT (conflict ↵Arthur Hanson Rasmusson9 days
| | | | line r171: cached=0x1247b5000 new=0x1321fa000); a cross-session cache served the previous client's dead page and silently absorbed releases. scanner: pool demand resolves against every known tracker sem (seen + resolved chains)
* vgsp: m163 lost-H2D copy integrity (drain injected pins before CE op ↵Arthur Hanson Rasmusson9 days
| | | | resolution; hold failed device-copy ops) + m164 CLASS-2 GR no-capture entry hold + m165 pool release applies at fresh-pin time + m166 timeout-ladder warm-sem fail-fast; scanner: first-class pool VA resolution + periodic re-assert + demand routing
* guest-diag: demand-pin content resolver searches +/-16 pages and learns the ↵Arthur Hanson Rasmusson9 days
| | | | winning per-area offset (multi-MB CE staging pushes never resolved at +/-1)
* guest-diag: semfix trace reads are remotely timed and SSH-preferred (stuck ↵Arthur Hanson Rasmusson9 days
| | | | QGA tails stopped pool/tracker pins and cascaded every client after ~5 to a DEVICE freeze)
* guest-diag: fix CE-window demand-pin offset mask (& 0xFFFFF folded every 1 ↵Arthur Hanson Rasmusson9 days
| | | | | | | | | | | MB rank to offset 0) Demand pins for 0x120700000/0x120800000/0x120a00000 all resolved to page 0 of the first nv_vm_map_pages area (the same wrong PA), so the host read garbage for every rank but the first and repeat clients stalled at DEVICE. Use the full 16 MB window offset (& 0xFFFFFF); the per-area page bound rejects areas too small. Live: r161 ran victory_mail green as clients 1 AND 2 in one boot, then matmul and smiley - the repeat-client stall class is gone for these.
* guest-diag: demand pinning for stalled CE VAs (event-independent hole filler)Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | | | | | The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).
* guest-diag: single-instance scanner start + full-re-assert backoffArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | r148 ran THREE guest scanners (pin_gr_stream install_and_start retried while a partially-failed first attempt had already started one, and the pid-file kill missed it). Each scanner re-asserted the whole CE pair set every 2 s; with a fully fragmented pair set (descending PAs, one page per line) that is ~2 k debugfs writes/s per scanner, and the guest hit load 19 with clients wedged spinning in userspace. - pin_gr_stream._start_cmd: kill EVERY existing scanner first (pattern assembled at runtime with printf so the command's own cmdline cannot self-match), then start exactly one. - guest_pin_stream._emit_ce_mem_delta: the full re-assert keeps its 1.5 s cadence for the first STREAM_FULLREM_FAST_S seconds of a mem's life (the client-switch flush race), then backs off to STREAM_FULLREM_SLOW_S (10 s) as a safety net. Configurable via env.
* guest-diag: heal session-flush pin wipes with periodic full CE re-assertArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The host wipes its whole sema map when the previous client's last channel closes ('session flush (all channels closed)'). A pin batch that raced in for the new client (observed: the 1-page 0x120000000 pin emitted in the same second as the flush) was lost, and the scanner's content-gated delta never re-emitted the unchanged pairs - so the second client's CE pushes were permanently unreadable (sema-probe 'NO ENTRY for page 0x120000000'), every GPFIFO entry was grace-skipped (r144/r145 matmul and mail stalls), and the guest spun in uvm_tracker_wait_for_entry. Fix (scanner, guest side): - _ce_mem_pages: walk the whole vmalloc area (minus header off), not the size-derived prefix that stopped one page short of the write frontier. - _emit_ce_mem_delta: full re-assert mode every KC_FULLREM_S (1.5 s) plus every fresh_map pass; emits every translatable pair content-independent, merged into contiguous runs. Zero-content pages are pinned on purpose: the pin carries the translation, the host reads live content. - missing pages retried with the negative PA cache popped (bounded 512/pass). Fix (host tool): - pin_gr_stream._flush: never dedupe single-page repeats of the CE window (0x120000000..0x121000000) so the re-asserts land. Live-validated in r145 (same-boot, no rebuild): mail green as 4th client, matmul green as 2nd client (bad=0, 64-changed acceptance of 857fe5f9), both with zero 'stalled VA'/'skipping stale entry' lines; pre-fix the same sequences stalled. Headless smiley all-wrong (bad=131072) reproduced independently and is the next target (engine-side constant class).
* guest-diag: resolve and stream the PMA scrubber buffer words (third-client ↵Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | scrub freeze) guest_pin_stream: find the scrubber channel buffer among the nvidia vmap areas (size in the CeUtils channel set, 906F ring shape at +0x64000 when non-empty), walk its pages with kcore and emit '#scrub <fin_pa>,<hsem_pa>' (the finish payload at +0x6c004 and the host semaphore at +0x6c000). pin_gr_stream: handle '#scrub' lines and register the PAs through the sema_scrubbuf_add module parameter, so the kernel's svc_scrub_pass can author the guest's scrub progress.
* guest-diag: content-validated delta emission for the CE push-buffer memArthur Hanson Rasmusson13 days
| | | | | | | | The one-shot CE mem emission pinned the whole 16 MB window including pages whose content was zero at the time; the host then read zeros for the other CE ranks' pushes. Sweep the mem every 2 s, skip zero pages, emit only new/changed {va,pa} pairs, and burst the Kcore PA cache every 30 s so a remap is re-emitted instead of shadowed by the cache.
* guest-diag: channel-anchored CE pool resolution for the spinning ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | tracking-sem case The second-client stall leaves exactly one live uvmt event: a tracking semaphore spinning with val=0 (the other channel/one-shot probes are inlined in this driver build and never fire). The sem is a member of a uvm_channel_t whose records sit in the sem's own window and carry the CE ring GPU VA (0x121010000+) plus back-to-back object handles; the channel's pool pointer is the first field at channel+0 (channel = sem-0x80), and the pool chain already satisfies _chain_mems_at once it is tried. The resolver now expands the candidate set from every ring-VA word's surrounding records and their targets before scoring. Verified live: the r134 spinning sem resolves to the CE push-buffer rm_mem with 413 pin runs, where the bare sem-window scan had produced none.
* guest-diag: keep CE push-buffer latches on stale refresh; never fail ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | freshness on an unknown page PA Two scanner defects seen in r131: (1) a resolved CE mem whose first vmap page could not be translated at store time stored page_pa=0, and the freshness check then failed on every probe, dropping the latch and re-resolving in a tight loop - and each failed re-resolution left the host with no pins for the channel at all; (2) the stale path discarded the latch before knowing the re-resolution would produce anything. Unknown PA now counts as fresh (the {cpu,gpu} object check is the evidence), and a stale refresh keeps the previous latch unless a new resolution actually emits runs.
* guest-diag: read the trace snapshot non-blocking (no stuck tail processes)Arthur Hanson Rasmusson13 days
|
* guest-diag: seed channel-event sets from the trace ring on client switchArthur Hanson Rasmusson13 days
| | | | | | | | | | clear_trace() on a pid switch dropped exactly the one-shot uvmt/uvch events of the new client when they landed in the switch instant; the pipe reader then never saw them and the whole session ran with an empty sem set (r126: vecadd_loop sem=0 for minutes while its CE entries paid the 8 s grace). Scan the ring tail once per switch and seed the sets instead - stale kptrs that are also re-learned resolve to no content and are deferred by the candidate scoring.
* guest-diag: content-score CE mem candidates and refresh stale pin setsArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | The kptr-keyed one-shot latches assumed object identity, but UVM frees and re-creates CE push-buffer rm_mems and channels mid-session and the slab reuses the same addresses; a resolution taken before such a move pinned pages that read all-zero when the host used them, so every CE ring entry stalled in the 8 s grace (r125: the live pushes sat at new vmap pages of the same rm_mem kptr). Candidates are now scored by their window content (a live push buffer is full of method streams, a freed range reads zeros) and the best is pinned - all-zero candidate sets emit nothing instead of poisoning the host map. A resolved mem is re-validated every ~2 s against its own {cpu,gpu} fields and first-page PA; when they move, the latches drop and the current truth is re-emitted.
* vgsp-sema: hold GR launches whose device argument pointers do not translate; ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | derive the CE vmap pairing per allocation Two second-client zeros/stall countermeasures: - svc_gr_data_fill now reports when a launch carries high (device-space) argument pointers and none of them resolves through any trusted source. svc_gr_request refuses to forward such a launch and leaves the capture unconsumed; the pump holds the ring entry for one bounded window per entry (gsp_sema_gr_grace_ms) and retries it, so a translation that appears a pass later (scanner pin, published directory, learned pair) is used instead of executing the kernel against unresolved pointers. On window expiry the entry is consumed exactly as before (bounded zeros, never a hang). Bounded per-session fill diagnostics name the source that serves each candidate (ipin/pin/lat/map/vas/disc/win) and dump the published vaspace walk results for misses. - The CE push-buffer kcore chain resolver derives its vmap page offset per allocation instead of using a fixed shift: the rm_mem size is read from its real offset (570.144 layout) and paired with the vmallocinfo area page count, so a leading header page is only skipped when it actually exists. With an unreadable size the two candidate offsets are content-probed with the host's method-stream plausibility test and an ambiguous result keeps the status-quo pairing. This removes the off-by-one that fed wrong push pages to the pump (content-rescue crawl).
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-25 23:30:31 +0000