summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* wall2 session 93: zero-ptrace pipeline (t101 instance clears self-disabling ↵Arthur Hanson Rasmusson6 hours
| | | | + unwait.py 16K/+0x80 word poller) reaches VKDEVICE on r693 with 5 auto-written words and no wedge risk; RM-lock wedge root-caused (gdb killed at a stop inside an RM critical section) with new attach rules + t102 safe diagnostic
* wall2 session 92: automated zero-touch pipeline reaches VKDEVICE on a fresh ↵Arthur Hanson Rasmusson6 hours
| | | | boot (t99 = marker-clear at +0xB33D30 + wait-word chase/write at +0xDCEBD9); live_walk PT injections alongside; attach-discipline lesson (wedged continue blocks later attaches); next = post-device wait identification
* wall2 session 91: vkCreateInstance wall BREACHED - the event-slot ↵Arthur Hanson Rasmusson7 hours
| | | | pending-marker (u16@+0xe=0xffff) cleared at each +0xB33D30 retry entry passes the instance phase for the first time; client then persists in vkCreateDevice's nvidia0 poll waiting for the event-fd signal (guest dispatch gap); tools t93/t94/t96b/t97
* wall2 session 90: current-boot wait site identified (eglcore+0xB2852D ↵Arthur Hanson Rasmusson7 hours
| | | | in-process word pair wait inside vkCreateInstance; old 0xa85d24/16K sites dead this boot); t83 sampler + widened nudge tools; multi-word nudge trips -4 (caution confirmed)
* wall2 session 89: counter-page record resolve proven live (slot learn + ↵Arthur Hanson Rasmusson8 hours
| | | | pub_calls 0->172); GR ctx-init release apply remains the single unmet wait
* wall2 session 89: GMMU PT walk replicated host-side + live inject ↵Arthur Hanson Rasmusson8 hours
| | | | (validated: ring entries and push method streams match); gpfifo done 0->6; root cause of -4 = release-apply gates (gr_forward defer-arm, op_tr_pending opdef withholding) + unresolvable counter page 0x20400f000 (slot stays pending); counter-page {pa,va} record tables found for host-side resolve (live only)
* wall2: session 88 - push page-5 injection fix (proven +30/sweep), ↵Arthur Hanson Rasmusson8 hours
| | | | host_resolve tool, RING GAP diagnosis (gpfifo done=0), gdb-SIGKILL RM-lock wedge lesson
* wall-2 session-87: window page-list hunt - the 2 GiB cycle's windows are ↵Arthur Hanson Rasmusson9 hours
| | | | scattered (later offsets at first_PA+off are zeros); the memdesc {size 0x20000, PA=0x107c47000, kptrs} page list is only reachable via the kernel pointers (kcore walk with fresh addresses needed); content search for the init-stream prefix found the window pages (0x103c06000+0x858, 0x11dbfa000+0x858, 0x126d69000+0x59c) but the VA<->page relation is non-linear; the client blocks in os_schedule at the allocation (the unread later pushes suspected)
* wall-2 session-86 tools: vk_memprops.py (standalone Vulkan memory-properties ↵Arthur Hanson Rasmusson9 hours
| | | | dump), vk_alloc_test.py (standalone allocation reproducer), t91.py (auto-clear + ICD memprops/alloc hooks), t90.py (A8 count auto-clear)
* wall-2 session-86: the DEVICE_LOCAL -2 wall CRACKED - the 512 MiB emulated ↵Arthur Hanson Rasmusson9 hours
| | | | VRAM was the gate (the ICD hooks captured ALLOC size=0x4000 typeIndex=1 valid; vk_memprops shows device-local heap[0]=512 MiB); the nvidia-nvidia-vgpu-turing-2g mdev type (2 GiB heap, VGPU_MDEV_TYPE env var honoured by start_vm_keep.sh) lets the same client with the same recipe proceed past vkAllocateMemory into a deeper wait (os_schedule, no -2); remaining: the later pushes at va=+0x52d4/+0x586c read pa=0x0 (window page list scattered; injections covered page 1 only) - extend the window injections; tools: vk_memprops.py, vk_alloc_test.py (hits -4 without the harness), t91.py
* wall-2 session-85 BREAKTHROUGH: the NvOgl wait CLEARED - the A8 loop's exit ↵Arthur Hanson Rasmusson10 hours
| | | | needs the session count *(u32*)(lib_base+0x2070b7c) to reach 0 (the engine sessions' completion counter); a direct gdb write of 0 cleared the wait and the client ADVANCED past the post-device phase for the first time (VKDEVICE ok then VKFAIL vkAllocateMemory -> -2 OUT_OF_DEVICE_MEMORY at vk_frame.c:204 - the new wall: the DEVICE_LOCAL heap/type reporting, no alloc RPC reached the emulation)
* wall-2 session-84: the a86038 want-scan mapped - all four entries PASS with ↵Arthur Hanson Rasmusson10 hours
| | | | the driven window value (wants {0,0,0,6} vs actuals {0,0,0,0xffffffff00000100}, no pending); a full-window write (0x0..0x1ff0) does not clear the wait; the residual gate is the A8 loop's internal callback/retry machinery (the a85f30 exit path decrements *(rbp) while firing the global callbacks 0x206f790/0x206f7a0 with args from rbp+0x30/0x38/0x40) and the slow yield-spin cadence (b284b0(...,0xb))
* wall-2 session-83: the A8 completion counter decoded to a client readback ↵Arthur Hanson Rasmusson10 hours
| | | | window - a82f30 -> *(obj+0x19b8)=eglcore+0xb250f0 -> b1bf00 (slow path obj+0x224==0) -> b27c90 reads *(u32*)(*(*(obj+0x1480)+0x58)+idx*16) (idx 0xbb.., min over n=*(u32*)(obj+0x22efc)); the window for handle 0xbeec0100 is VA 0x7827e6c12000 (/dev/nvidiactl) and the host CAN write it via the RAM file: writing 0x100 made the client read it (obj+0x1070=0x100, actual table entry 0xffffffff00000100) - first live proof the counter can be driven from the host; the wait persists in the loop's second scan (a85ec1/b29dc0) and the *(rsp+0x38) branch
* wall-2 session-82: counter-page PA hunt - the client's pagemap does not ↵Arthur Hanson Rasmusson10 hours
| | | | contain the counter pool (guest_pmap_win.py empty for 0x20400f000 and 0x121000000 ranges; semfix_uvmt's pin_counter_page never succeeded) and the vidmem-window hypothesis is not supported (poke_vidmem at the pool window offset 0x3F87000 + rank slots 0xfd0/0xfc0/0xfb0 with 6 then 0x40 changed the shadow but did not move the A8 wait); gdb attach hangs on the post-device spin (kill by PID + SIGCONT recovery)
* wall-2 session-81: counter-autoreg gap analysed - svc_counter_autoreg needs ↵Arthur Hanson Rasmusson10 hours
| | | | an ipin pin or svc_counter_derive for the rank slot (0x20400fec0+(19-r)*0x10); both miss for the counter page 0x20400f000 (the client does not map it - pagemap empty, no VMA; it is a UVM kernel object; the pin stream's sem resolutions are 0 despite 44 want-list entries); bypass: the sema_counter param registers a slot manually (echo <h_object>,<pa>); GL ranks 0,1,2 -> slots 0x20400ffd0/0xfc0/0xfb0
* wall-2 session-80: ALL low-aperture window records found via the 'liat'-tag ↵Arthur Hanson Rasmusson10 hours
| | | | fix (the record tag is the u32 0x7461696c whose on-disk bytes are 'liat', not 'tail'); wall2_recscan.py scans them; the GR init semaphore windows identified and injected (0x4b17000->0x12a76f000 verified by content: +0xbb0=3/+0xe30=1 = the release payloads; 0x4b19000->0x122f9b000, release applied there) - releases now land on the semaphore pages; the NvOgl wait still spins (the A8 loop's per-entry counter tables are the remaining completion condition)
* wall-2 session-79: resolver7 tool + the corrected pair-specific injection ↵Arthur Hanson Rasmusson10 hours
| | | | recipe (0x4480000->0x125692000, 0x4ca0000->0x131649000, 0x4b16000->0x12a76a000) and the window-record layout {tag,VA,SIZE,0,tail} stride 0x40; documents the resolver7 cross-pair bug, the decoded low-VA releases and the remaining semaphore-VA translations (pending=417)
* vgsp-sema r686: accept the low aperture (>=0x4000000) for GR/GL release ↵Arthur Hanson Rasmusson10 hours
| | | | | | | | | | | | | | | decode in both release forms (0x5c-form and SET_REPORT_SEMAPHORE) The GR/GL ucode init pushes carry their releases on the RM's low aperture (the same 0x4000000-0x4ffffff VA family as the push/ring windows): the live GL init push decodes as rel=0x4b19b10 pay=1 (REPORT_SEM group 200426c0 00000000 04b19b10 00000001 10000000) and rel=0x4b17bb0 pay=3 / 0x4b17e30 pay=1 (0x5c-form 2005c017 04923060 ...), all rejected by the sysmem-only SEMA_VA_LO bound - the NvOgl engine-init wait then spun forever. With the bound relaxed (and the push windows injected via the window-record/memdesc recipe) the releases decode live (have_rel=1, applied rising) and the un-translated ones queue in svc_pending (417 at the observed steady state) awaiting the semaphore-VA translations.
* wall-2 session-78: post-device wait identified as the NvOgl (GL ucode) ↵Arthur Hanson Rasmusson11 hours
| | | | engine init (A8 retry loop work item named NvOgl_m0; D6 submit ORs 0x200000 into the pending word) and the push windows become READABLE via the resolver6 window-record/memdesc recipe (0x4480000 -> 0x120cfb000, 0x4ca0000 -> 0x12e51e000; the service decodes the GR init streams where sessions 73-74 saw unread) - remaining gate: have_rel=0 (CE-shaped matcher) and the GR forward not engaging (forwarded=0 gr_nocap=4 drain held=4)
* wall-2 session-77: vkCreateDevice PASSES - r682 0x7E event validated live ↵Arthur Hanson Rasmusson11 hours
| | | | (registered slot 13, EVPOST rc=0) and the completion word found/driven: the main-thread wait reads a u32 at +0x80 of a 16 KB rw-s /dev/nvidiactl notification mapping (should-wait = word != 1); a host write of 1 via the guest RAM file (PA from pagemap, file offset PA-0x80000000 above 4G) flips the wait and vkCreateDevice returns success (VKDEVICE ok family=0) - reproduced 3x (r683 manual, r684 nudger, r684d probe+single-word). Tools: t79.py (wait-chain resolver), guest_words.sh, nudge.sh; README-session77.md records the recipe, the residual guest event-dispatch gap (records consumed, notification word not written), and the -4/gdb-attach cautions.
* vgsp-rm r682: register NV01_EVENT_KERNEL_CALLBACK (0x78) and ↵Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | | | | NV01_EVENT_KERNEL_CALLBACK_EX (0x7E) events alongside OS_EVENT (0x79) The driver's modern nvkms/Vulkan path allocates its per-channel completion events with class 0x7E (nvos.h NV01_EVENT_KERNEL_CALLBACK_EX; observed live as GSP_RM_ALLOC class=0x7e paramsSize=24 hParent=<subdevice> hObject=0xcaf00003 from the second RM client while its main thread sat in the vkCreateDevice wait loop, whose readiness word lives in a /dev/nvidiactl notification page that only an event completion can signal). The event registration scanned the params for hClass 0x79 only, so 0x7E allocations were never registered and their completion wake-ups (post_channel_event/post_all_events) could never reach them. Accept all three event classes; the NV0005_ALLOC_PARAMETERS layout is the same {hParentClient, hSrcResource, hClass, notifyIndex, data}.
* wall-2 session-76 addendum (r681): decode the main-thread vkCreateDevice ↵Arthur Hanson Rasmusson12 hours
| | | | wait - should-wait = (readiness word != 1), readiness = u32 at nvidiactl-mapped holder+0x80 (VM_IO page, 39/39 reads zero, all fd polls time out), give-up predicate dcee20 fires -4 only on RM error flags (b29d60 / +0x70cc); the word's writer is the guest kernel driver/RM, not host RAM
* vgsp-rm r680 + wall-2 session-76 forensics: 0x2080a097 reply record-area ↵Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | | | | | | | | tunables (wall2_a097_count/rec0..rec7, default 0xffffffff = echo) and the evidence set that re-questions the a097 gate The real host RM 0x2080a097 handler (GSP-RM 570.144 control table entry, params 0x490) overwrites params+4..+0x23 with up to four {u32,u32} mode pairs (engine value + a x1000 clock/percent) and the client copies back the first 16 bytes; the tick at eglcore 0xd5e3e0 consumes the second word of the first pair as the per-engine progress (session +0xe104, compared against per-boot config threshold 95 at lib+0x2019320+0xffc with 10/20 bit-history cutoffs). New module params let one boot sweep the record matrix; the r680 sweep (flag/v488/v48c + count/rec matrix incl. progress 100..1000000) changed nothing observable, and hw watchpoints show E+0x150 stays 0x00200000 with no writer over 5+ minutes while the a097 poller is a worker thread and the main thread sits in a 10 s poll() loop on /dev/nvidia0 inside vkCreateDevice. Tools: t75g.py (in-guest gdb state/watch probes), diag76.py (thread bt + address resolution), slots_probe.sh, run75g.sh; README-session76.md records the firmware handler decode and the sweep.
* r679: 0x2080a097 handler with runtime-tunable reply words (wall-2 session 75)Arthur Hanson Rasmusson13 hours
| | | | | | | | | | | | | | | | The guest Vulkan client (libnvidia-eglcore) issues RM control 0x2080a097 in a ~1.8 s retry loop while its engine-readiness check stays negative; the bare-metal oracle (same client on the host with the real driver) never issues it, so it is a vGPU-only path (engine type 0xe00101). The handler echoes the 1168-byte in/out record block and applies three module parameters - wall2_a097_flag/v488/v48c - so one boot can sweep reply variants without rebuilding (the client reads params +0x204, +0x488, +0x48c back; file offsets 0xecd45f..0xecd475 in libnvidia-eglcore.so.570.144). Sweep result so far: flag=1 with v488=0 or 1 does not stop the retry; the readiness wait lives in the client's engine loop (0xd6ef40..0xd6f0ab polling byte +0x1b038 of the engine object, recomputed by a91ea0 from engine-state fields), so the missing state is deeper than the reply words.
* wall-2 session 74: the client's 11s stalls are userspace completion-slot ↵Arthur Hanson Rasmusson13 hours
| | | | | | | | | | | | | | | | | | | | | | spins; the next wall is the polled RM control 0x2080a097 Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield + clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings. Guest-side filling of the slot flags lets the client proceed immediately, but vkCreateDevice still fails -4: with the slots filled the client settles into an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation answers with the generic all-zero reply, so the poll never completes. Also captured: sema_push_raw dumps proving the GR channel-init push contains NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then zeros), and the negative result that neither channelFillGpFifo nor nvWriteGpEntry runs for this channel's kick. New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and vmap_watch2.py (vmap/kcore watchers), README-session74.md.
* wall-2 forensics tools: GR channel-init push localization (session 73)Arthur Hanson Rasmusson14 hours
| | | | | | | | | | | | | | | | | Tools and findings from the live wall-2 campaign (r674/r675): the vk_frame client's GR channel-init push (va=0x4480000 len=1432) cannot be translated by the sema service, so no release is applied and vkCreateDevice fails -4. Captured: the RM window-record/memdesc layout (ring memdesc PA == the service's validated ring pa; push memdesc PA injected live and read by the service without a false-translation line), the CE-only shape of the service rescue matcher, the notrace status of nvidia/modeset kprobes (QEMU gdb hbreak works), and the ring-entry encoding from channelFillGpFifo/nvWriteGpEntry (neither runs for this channel's kick). New tools: wall2_resolver{2..5}.py, wall2_livecap{1..3}.py, RAM structure scanners (scan_r673*.py, trace_push674.py), gdb675*.gdb probes, plus a README with the ranked next steps.
* r655 diagnostics: dump unknown-RPC payloads and UPDATE_BAR_PDE payloads ↵Arthur Hanson Rasmusson15 hours
| | | | (wall-2 forensics aid - the r670 capture proved zero unknown RPCs and no mapping RPC during the client's vkCreateDevice attempts, so these dumps are bounded evidence tools for the next wall)
* r654 FIFO_GET_ENGINE_CONTEXT_PROPERTIES (0x801707): return the real ↵Arthur Hanson Rasmusson16 hours
| | | | per-engine context geometry captured from the GSP on the physical backend - the client queries ZCULL geometry (engineId=8) at the end of every vkCreateDevice attempt and the generic all-zero reply failed its ZCULL-context buffer allocation (VKFAIL -3); with the handler the client proceeds through GR_CTXSW_ZCULL_BIND, GR_SET_CTXSW_PREEMPTION_MODE, VRAM allocs and MAP_MEMORY_DMA to the GR channel kick
* r651-r653 display path: DISPLAY_GET_STATIC_INFO reports the ↵Arthur Hanson Rasmusson17 hours
| | | | one-head/8-window model (feHwSysCap head-0, windowPresentMask 0xff, numDispChannels 81 for the cursor channel numbers) - live gdb pinned the 0x56 to kdispGetChannelNum_v03_00 requiring windowPresentMask&NVBIT32(instance) for dispChnClass_Win and HEAD_EXISTS for Curs; + INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES (displayMask=1 numHeads=1; dispswConstruct rejected every head against numHeads=0); + NVC370_CTRL_CMD_GET_CHANNEL_INFO reports IDLE immediately (nvRMSyncEvoChannel polls 2s for IDLE after every kickoff)
* docs: SESSION 66 - the 0x56 pinned to the guest RS alloc path ↵Arthur Hanson Rasmusson18 hours
| | | | (rmapiAllocWithSecInfo enters with class 0xc57e, serverAllocResourceUnderLock returns 0x56 before the GSP-RM RPC at under-lock+0x447; _rpcAllocObjectPrologue exonerated); the guest's not-stripped nvidia.ko is a new live-debug instrument; gdb-stub hazards recorded (kill wedges the VM - detach-only recovery; stepi loops wedge the guest - prefer conditional breakpoints)
* docs + nvkmstrace: the GL/Vulkan wall fully bisected (SESSION 65) - ↵Arthur Hanson Rasmusson19 hours
| | | | ALLOC_DEVICE reply status 5 (CORE_CHANNEL_ALLOC_FAILED) decoded by the LD_PRELOAD nvKms logger with a widened reply scan; the empty display capability notifier is the first missing piece (a live-gdb fill of SYS_CAP/SYS_CAPB flips the driver to the window path, proving numWindows>0 unblocks it); the residual wall is the vGPU-guest display-class rejection: nvRmApiAlloc(0xc57e) returns NV_ERR_NOT_SUPPORTED inside the guest kernel and never reaches the emulated GSP - the guest runs in vGPU mode where display/modeset classes are host-service-provided by design
* docs: record the real-render walls measured this session (Vulkan ICD ↵Arthur Hanson Rasmusson20 hours
| | | | enumerates on the emulated vGPU; vkCreateDevice fails in nvidia-modeset's ALLOC_DEVICE past the new NV0073 head/connector shim, residual wall = the display core/modeset channel objects; Xorg ScreenInit fails in nvidia-push's 10 s channel test with submissions=0 and unresolved X-channel VAs - the guest scanner serves CUDA clients only - plus a rank-0 counter-slot collision with a live CUDA client; EGL/glsi unchanged) and link the page from the README
* tools/vk + nvkmstrace: real-Vulkan probe stack for the emulated vGPU - ↵Arthur Hanson Rasmusson21 hours
| | | | vk_probe/vk_probe2 (ctypes loader probes: enumerate + create device/queue), vk_frame (headless Vulkan compute renderer: loads the NVIDIA ICD, paints a plasma into an RGBA8 storage image, reads it back to PPM and publishes frames into the kvmfr ivshmem BAR), vk_plasma.comp -> SPIR-V; nvkmstrace.c (LD_PRELOAD NVKMS ioctl logger that decoded the vkCreateDevice failure to ALLOC_DEVICE cmd=0). Measured: the NVIDIA device enumerates and its Vulkan ICD loads (2 ICDs: NVIDIA + llvmpipe); vkCreateDevice fails VK_ERROR_INITIALIZATION_FAILED because nvKmsAllocDevice returns non-success after the display shim
* vgsp-rm r643/r644: NV0073 display-common shim for nvidia-modeset - report ↵Arthur Hanson Rasmusson21 hours
| | | | one head (GET_NUM_HEADS 0x730102, GET_ALL_HEAD_MASK 0x730287, flexible window assignment 0x7302ad) and one static TMDS/HDMI connector on display id 0 (GET_SUPPORTED 0x730107, OR_GET_INFO 0x73028b SOR+SOR_SINGLE_TMDS_A, GET_CONNECTOR_DATA 0x730250 HDMI_A, DFP_GET_INFO 0x731140); the generic all-zero replies made nvkms log 'No heads found on board!' and fail (killing Xorg ScreenInit and Vulkan vkCreateDevice), then fail NoValidDisplays() after the heads were added
* vgsp-sema r640: raise the wait-gate episode extension cap 25 s -> 45 s - the ↵Arthur Hanson Rasmusson21 hours
| | | | r639 streams2 transcript has the readback's acquire giving up at waited_ms=28378 while the launch whose release it waits for only forwarded ~9 s later (watermark defer + arg-partial defer + backend poll chained past the 25 s bound), so the readback ran out of order and the client read zeros (bad=65534); 45 s keeps >15 s of headroom under the clients' >= 60 s timeouts and only engages while a GR forward is unoffered (green boots never reach it)
* tools/guest-diag: GL/Vulkan probe + X capture publisher for the real-render ↵Arthur Hanson Rasmusson22 hours
| | | | path - egl_retry (explicit EGL make-current retry probe: the glsi engine-init returns busy once and eglcore never retries; prints the per-try EGL error), xshm_pub (X11/XShm screen capture into the kvmfr ivshmem BAR in the frame_producer layout so a GL/Vulkan-rendered guest frame reaches the host), xorg-vgpu.conf (headless nvidia Xorg server layout for the emulated vGPU)
* vgsp-sema r638: break the CE-watermark/wait-gate mutual citation - a channel ↵Arthur Hanson Rasmusson22 hours
| | | | whose unconsumed head entry is wait-gate held is a consumer-side wait, not a producer backlog; the watermark skips it while the candidate's operands are fully resolved and no decoded device write is pending, so the deferred launch forwards and its release installs (the r612/r636 streams chain: readback acquire op=3 pay=N sits behind the very launch whose release it wants, the two defer each other for the full 15 s cap and the copy runs ~20 s late); module param sema_mutual_skip (default 1, 0 = r636 behaviour)
* STATE-CURRENT: refresh snapshot - r635 closes the host-side ↵Arthur Hanson Rasmusson22 hours
| | | | operand-resolution gap (content-tracked shadow anchors + launch-level content anchor + bounded cross synth + table guard; the draft's self-feeding mint caught and gated, green-boot table 6144/8192 -> 0 minted), r636 fresh-boot battery best-equal 9 OK / 2 clean FAIL / 1 HANG with streams_events hang->clean CUDA_ERROR and the streams2 red chain measured (H2D dsts replay-denied -> GR forward deferred on the CE stream-order watermark -> degenerate refusal -> SYNTH-FAULT); remaining: streams2 CE-order chain, first-wait crawls, used-boot allocation exhaustion, then 15/15 + soak
* vgsp-sema r635: host-side operand resolution when the scanner misses - ↵Arthur Hanson Rasmusson23 hours
| | | | per-shadow-page REAL-content tracking (r->content set on every landed write) anchors a client's own H2D-shadowed pages and arms a launch-level content anchor, so an answerless output candidate (contiguous tail of a shadowed input, or a non-contiguous allocation) is served from a bounded synthetic page set (1024/candidate, 4096/launch, 3/4-table guard) instead of failing the launch closed; synthetic pages can never re-anchor (the first draft self-fed on its own minted pages and filled the shared table to the guard - 6144/8192 measured on a green boot - content gating restored 0 minted pages there and the historical green shape); trusted pin/walk/ledger answers keep the r633 anchored budgets and no longer arm the cross mint; fresh r636 boot: best-equal battery 9 OK / 2 clean FAIL / 1 HANG (streams2 bad=65534 - its H2D dsts are replay-denied, the GR forward defers on the CE stream-order watermark, degenerate refusal faults cleanly; streams_events hang->clean CUDA_ERROR via the auto-fault; smiley first-wait crawl)
* STATE-CURRENT: refresh snapshot - r632 mail hang root-caused+fixed (r633: ↵Arthur Hanson Rasmusson24 hours
| | | | held GR launches park their push-level release records and the eventual forward delivers them; mail green on two fresh boots), release-target question answered (rel_mod=1 default redirect, resolver works), synthetic shadow caps gated (anchor + per-episode; peak 2048/8192), best-ever battery r633b pass=9 fail=0 hang=1, r631 math red draw green; remaining top blocker = operand-resolution shortfall behind the wrongness class
* vgsp-sema r633b diag: log the hold-engage record set (nrel/va0/pay0/gr) and ↵Arthur Hanson Rasmusson25 hours
| | | | raise the non-pool device-VA release-write log budget 512 -> 4096/boot - the r633b battery's streams2 spin (userspace, launches completed, QMD releases applied) needs the held records and the release target PA visible; the 512-line budget was exhausted by mid-battery and hid every later application
* vgsp-sema r633: park a held GR launch's push-level release records and ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | deliver them on the eventual forward (the r632 mail hang), gate synthetic shadow pages on an anchored span with per-episode budgets (the r632 junk-candidate flood), and name the release0 redirect cause The r632 first boot's mail client spun forever in cudaDeviceSynchronize after its vadd launch: the push-level tracking release (0x20480fff0 pay=112) never landed while the client's word stayed at 111, even though the launch itself completed (GUEST GR LAUNCH OK) and its QMD release was software-applied. Root cause: the release records (svc_fwd_rel[]) are per-decode state (reset at the top of svc_decode_stream), and the two GR hold paths (busy-defer hold, m150 data-stalled hold) break out of the pump entry loop BEFORE the release block (hold-engage / inline apply) runs - so a held launch's records are dropped at the pass boundary. The eventual forward (queued from the capture ring at the defer cap, without a re-decode of the entry) then engaged the release hold with whatever records that pass happened to carry (often none, or another entry's). Live evidence: read_proc_va on the spinning client - 0x20480fff0 = 111 (payload 112 never applied), 0x204827f74 = 6 (QMD release delivered); host log - no sema-sw line for 0x20480fff0 val=0x70 anywhere. Fix (parked releases, keyed by vgpu/channel/push VA): both hold paths park the pass's records (svc_grpark_put) before their break; svc_gr_request restores the queued push's parked set into svc_fwd_rel[] at the consume point so the pump's hold-engage holds exactly those records; the two fail-closed REFUSED paths deliver the parked set inline; the session flush clears the table. Bounded log line "GR parked releases restored" (32/boot). Proven live: r633 first boot restored push=0x20043c1c8 n=1 va0=0x20480fff0 pay0=100; r633 mail now completes (VADD bad=0 checksum=12204516.000000, STATUS OK) on two consecutive fresh boots. Also r633: the r632 synthetic shadow caps were per-FILL-CALL and the fill re-runs on every retry pass, so one probe=none junk argument candidate could mint another 1024 synthetic pages per pass - the 8192-page shadow table filled within seconds of the r632 first launch (va-shadow 8192/8192, all synthetic, from the two 16 MB guessed spans of candidates no resolver ever answered), which then killed the lane for every later real page ("shadow table full ... write dropped"). Two changes: - ANCHOR GATE: a synthetic page is created only when the candidate's span already has >= 1 page answered by ipin/resolve/ledger. Existing H2D shadow content still serves unconditionally (the red-draw chain), and a real buffer's unresolved tail stays anchored by its resolved head. A candidate whose every page is answerless takes the historical drop path - its guessed span is not a buffer; - PER-EPISODE BUDGETS: the per-candidate (1024) and per-launch (4096) caps now persist per (push VA, span base) across the episode's passes, not per call. Measured r633: shadow table peak 2048/8192 (vs 8192/8192 full), 32 synthetic pages for the mail launch (vs 4041 junk-inflated n_data), the lane stays available for content serves. Diagnostics (bounded, one boot of evidence): "vgsp-work: gr-rel0 ..." names the release0 resolution chain (rel_pa/ap/hpa/word/pay/unobs) and "vgsp-gr: qmd-rel redirect: ..." names the backend redirect cause. The first live reading answers the release-target question: rel_hpa resolves (0x2fc495000), map_ret=0, unobs=0 - the redirect is the deliberate gsp_phys_rmcall_gr_rel_mod=1 default (m64 forced module-semaphore release), not a resolver failure. The r632 hang was the push-level release loss, fixed above. Build clean (kernel + r65535 warnings only).
* STATE-CURRENT: refresh snapshot - corruption panic class root-caused+fixed ↵Arthur Hanson Rasmusson26 hours
| | | | (r627), best battery 11/15 zero hangs, r632 shadow-lane operand serving, remaining: release-target resolver + wrongness class
* vgsp-sema r632: the GR data fill serves unresolved operand pages from the VA ↵Arthur Hanson Rasmusson26 hours
| | | | shadow lane (existing H2D content, or a synthetic page for an argument candidate under per-candidate/per-launch caps) instead of dropping them - the submit build hands the shadow page's host address to the bounce machinery, so copy-in stages the client's own bytes and copy-out writes the engine's results into the same page the D2H copies read (red-draw chain: walk=none -> H2D shadow -> fill fail-closes -> degenerate refusal -> SYNTH-FAULT); write guard keeps a shadowed VA's writeback; sema_no_shadow=1 keeps the historical drop path
* vgsp-sema r627: the copy-landing parity pass no longer clobbers per-page ↵Arthur Hanson Rasmusson29 hours
| | | | | | | | | | | | | | | | | | | | | | landings - the r623c wrong-write site The post-loop parity store (r254-era) recorded a large virtual-destination copy's whole range contiguously (dst + po) AFTER the per-page loop had already stored each page's true resolved PA (m180/r391). For a scattered UVM host buffer the contiguity guess overwrote the true entries; the next copy of the same VA then followed the poisoned ledger (the non-pin DIVERGE branch keeps the ledger) and wrote client c-data into guest pages the kernel had since re-used. r623c evidence chain (saxpy D2H readback of 0x70e64e700000): sema-ledger: rewrite va=0x70e64e77f000 old=0x12e700000 new=0x115800000 504x sema-ledger: DIVERGE write ... prev=0x1158xxxxx cur=0x12e7xxxxx guest panic: kchannelGetGfid_IMPL+0x12, garbage 0x44560800445590f0 (IEEE754 floats 854.25/856.125 = client c-data in nvidia channel structs) Fix: the parity pass skips pages the per-page loop already stored (lookup guard); it only seeds pages the loop did not reach. True per-page landings are never overwritten.
* vgsp-sema r626: sema_no_shadow - A/B knob to disable the VA shadow lane ↵Arthur Hanson Rasmusson29 hours
| | | | (copy-dst + fill fallbacks hold instead); isolates the lane as a variable for the fatal corruption class (nvidia channel structs overwritten with PTE-like patterns -> panic in kchannelGetGfid_IMPL via nvGpuOpsStopChannel; r613c/r616/r623c)
* tools r625: stuck_client_capture.sh - one-shot capture for a stuck client ↵Arthur Hanson Rasmusson29 hours
| | | | (pause the trace-consuming scanner, read the retained uvtw/uvsv/uvmt tail, walk the tracker + pool page, resume); wraps the manual sequence that unlocked the first-wait evidence
* tools r624: tracker_walk.py + pool_page.py - guest-side (kcore) walks for ↵Arthur Hanson Rasmusson30 hours
| | | | the first-wait class: read a UVM tracker's semaphore object and expected payload, and dump the semaphore-pool page's applied values; together with the host pool-wr/sema-sw logs they compare what the driver waits for against what the host wrote
* vgsp-rm r623: oserrlog - deliver a synthetic OS_ERROR_LOG (0x1006) ↵Arthur Hanson Rasmusson30 hours
| | | | engine-fault report to the guest's RM queue (debug param; payload mirrors the GSP ABI decode: exceptType/runlistId/chid/errString/prevXid) - the suspected missing recovery half after a synth-fault (post-fault clients stick pre-submission)
* tools r621: read_proc_va.py - read a running client's mapped words via ↵Arthur Hanson Rasmusson30 hours
| | | | /proc/<pid>/mem (the guest-side half of the sem-write divergence comparison; pairs with the r619/r620 sema-sw witness)
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-25 23:00:46 +0000