summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* vgsp-sema: on an implausible primary push read for a GR entry, retry via the ↵Arthur Hanson Rasmusson14 hours
| | | | DIRECT registered-VAS walk (svc_walk_vaspaces + svc_read_ap, the exact path walk_probe proves) instead of svc_lookup_trusted - whose pin/lat hits would answer with the same wrong page again; r711/r712 evidence: the user VAS registers and walk_probe rc=0, the rewind replays, but the retry/log never appears and the entries still skip after ~250 failed rescues (the sema-pushfail witness budget is per-boot and was exhausted earlier, so the read-failure detail was invisible this boot)
* vgsp-sema: every-pass VAS candidate verify with the last-good ring pair ↵Arthur Hanson Rasmusson14 hours
| | | | (r707e) - the r710 boot shows the host content-located trigger registers the user VAS and the walk resolves (walk_probe va=0x4ca0000: vas[3] h=0xfade0002 pdb=0x101d63000 -> rc=0 pa=0x11fa20000), while the pump's own svc_lookup pair only registered the three internal VASes (all rc=-2); the working order is: content-located trigger FIRST (registers the user VAS), then sema_chan_rewind so the replay re-reads the entries with the working VAS
* vgsp-sema (FOLD): verify the parked COPY_SERVER_PDES VAS candidates the ↵Arthur Hanson Rasmusson14 hours
| | | | moment any channel's ring resolves through the channel object, so the user-VAS page directory can register BEFORE the submit's first push reads (the host-triggered path could not win that window: a failed GR entry read is never retried with the fresh VAS); r709 evidence: three VASes registered (h=0xc/0xbaba0042/0xcaf00000, pdb=0x11b8db000/0x11b8e6000/0x11b8ef000) and walk_probe still rc=-2 for the client's push VAs - the registered directories do not cover the user VAS, so the content-pair verify against the ring must still land for that one
* vgsp-sema: bounded channel rewind (sema_chan_rewind: hex h_object or 'all' ↵Arthur Hanson Rasmusson15 hours
| | | | -> clears slot_done_epoch + last_get + the per-entry rescue state so entries skipped before the VAS pdb anchor re-decode) + bounded per-chunk push-read failure witness (sema-pushfail: va/chunk/page/last_rc/trusted_rc/ipin/alt/win per failed chunk); wall2: runbook.py (trigger the anchor then rewind). Session 118: the anchor+rewind path is wired; the automated rewind scheduler missed (dmesg churn) and the pushfail trace still needs one clean capture
* wall2: pdbtick.py loops (retry content-location + trigger every 1 s); ↵Arthur Hanson Rasmusson15 hours
| | | | session 117 evidence: the host-triggered pdb anchoring WORKS end-to-end (sema-pdb-disc -> 'VAS 0xfade0002 page directory registered after verified ring walk (pdb=0x109277000 entries=4 ap=1)'), the small GR pushes decode after the anchor (sema-push: h=0xbeec0100 va=0x4ca096c/0x4ca52c0), but the large ones (976/1432/17316 B) still skip after 210-275 rescues and the fence never releases - next: why the multi-page push reads/plausibility fail for the big pushes (the compute dispatch and the tracker), and give the pump a rewind/replay so already-skipped entries re-decode
* vgsp-sema: host-triggered VAS pdb discovery (sema_pdb_discover va,pa -> ↵Arthur Hanson Rasmusson15 hours
| | | | verify the parked COPY_SERVER_PDES candidates against a content-verified ring pair, then fall back to the vidmem walk-root sweep) + runtime-tunable candidate age-out (sema_vas_cand_ms; the default 15 s ages the client's VAS candidate out exactly when its channels first register, so the verified walk can never anchor it by itself)
* vgsp-sema/wall2: host-triggered VAS pdb discovery (sema_pdb_discover va,pa ↵Arthur Hanson Rasmusson15 hours
| | | | -> verify the parked COPY_SERVER_PDES candidates against a content-verified ring pair, then fall back to the vidmem walk-root sweep) and a runtime-tunable candidate age-out (sema_vas_cand_ms; the default 15 s ages the client's candidate out exactly when its channels first register); tools: fasttick.py (triggers on the GR0 ring resolution line) and pdbtick.py (content-locates the ring page and triggers) - r706/r707 evidence: three parked candidates (0x11cd3d000/0x11cd48000/0x11cd50000 entries=4), the ring pair appears only at channel registration (the same second as the age-out), ring resolution is racy across runs (r706 rc=0 / r707 rc=-2), the skip starts 145-194 rescues (~8-10 s) after the submit; next run: keep candidates parked + trigger within the window
* wall2 session 115: decompose the first-fence wall - the sema service cannot ↵Arthur Hanson Rasmusson15 hours
| | | | decode the submit's GR pushes because the client VAS page directory is undiscoverable (walk_probe: discovered pdb=0x0; the COPY_SERVER_PDES candidate is garbage; the r496f vidmem discovery never runs) and the pushbuffer is noncontiguous; the RAM ring scan gives up after 12 slices before the pushes exist; ring pages ARE content-locatable by their GPFIFO entries and injectable via sema_map_add (fence_assist.py) but the ring pair alone does not unlock the push decode; next = extract the pdb from the guest kernel's RM objects (PDB injection) or scan vidmem with the ring pair as verifier
* wall2 session 114: the COMPR+2MB VidHeap class (attr&0xF000==0x2000, owner ↵Arthur Hanson Rasmusson15 hours
| | | | 0x474C0000) is the last allocation gate - a live attr flip (COMPR_ANY->NONE at the ioctl) lets vkAllocateMemory through and the client runs the whole pipeline to the FIRST vkQueueSubmit, then times out on the FIRST fence (30 s, the new wall); the zero-touch create recipe is BC (eglcore+0xB33D30 marker clears) + BW (eglcore+0xDCEBD9 device wait-word chaser, the missing piece) + the a8clear count gate; the fence release (push-page readability/completion after the submit) is the only remaining gate to VKFRAME
* tools/vk: vk_frame gains an optional 7th arg (tiling: 0=optimal default, ↵Arthur Hanson Rasmusson16 hours
| | | | 1=linear) and a buffer-allocation type-retry loop; wall2: the QEMU-gdbstub kernel-breakpoint script (module VAs insert+patch but never fire - the stub's memory context does not resolve module VAs) and the kcore symbol dumper; session findings: the vkAllocateMemory -2 = the image's internal COMPR 2MB-page 64KB sub-allocation failing NV_ERR_NO_MEMORY after the image data alloc succeeds (both tilings)
* vgsp-rm: FB_GET_INFO_V2 COMPRESSION_SIZE reports a comptag pool ↵Arthur Hanson Rasmusson17 hours
| | | | (wall2_fb_compr_mb, default 128 MiB) - the guest driver attempts a COMPR-enabled 64 KB-aligned comptag allocation (vendor VidHeap alloc attr COMPR=2, r703 live: the last vidheap call returns NV_ERR_NO_MEMORY 0x51) and libcuda maps that to VK_ERROR_OUT_OF_DEVICE_MEMORY (the vkAllocateMemory -2 that gates the first Vulkan frame); 0 restores the historic no-compression reply for A/B
* vgsp-rm: answer the NV04_VID_HEAP_CONTROL enumeration pair ↵Arthur Hanson Rasmusson17 hours
| | | | 0x90960106/0x90960107 (legacy NVOS32 VidHeap API) - libnvidia-eglcore walks three heaps and treats an all-zero reply as 'no memory' (the flag byte at entry+0x30 == 0 skips every heap), so vkAllocateMemory failed -2 without any alloc reaching the emulation; replies are runtime-tunable (wall2_9096_lo/hi/ent_addr/ent_size/ent_type/ent_flag) for a one-boot sweep
* vgsp-sema: bind the scrub slot to the EXACT 4096-entry channel ↵Arthur Hanson Rasmusson17 hours
| | | | (NUM_COPY_BLOCKS) - the >=4096 test also matched every client channel (32768 entries in this driver version), so the pass authored for whichever client channel registered last while the real scrubber's finish payload stayed zero (the vkAllocateMemory -2 gate, r701 live evidence: slot held h=0xbeee0102/entries=32768 and the pass logged 'scrub author put=0')
* vgsp-sema: scrub-pass diagnostics - stash bind/MISS logs (4096+-entry ↵Arthur Hanson Rasmusson18 hours
| | | | channels that fail the userd/va condition were silent), pass idle/read-fail/junk-put logs; the r697 session proved the direct authoring works (fin/host sema advance unblocks scrubCheckAndWaitForSize permanently) but the in-service pass never fired
* wall2 session 111: FB FULLY disproved (whole-256MB fill_vidmem no flip -> ↵Arthur Hanson Rasmusson18 hours
| | | | memdesc not FB-backed); refined assist negative; SYS-fill of active neighborhoods wedged the guest (cycle next; never bulk-overwrite active-set-adjacent pages)
* wall2 session 110: the sec2 loop runs CONTINUOUSLY inside one long-lived ↵Arthur Hanson Rasmusson18 hours
| | | | ioctl that holds the RM write lock (closes the lock model: tid 1791 read-wait, new clients D-state); the 299-byte reader is a red herring (address never mapped, 5/s); PEBS unavailable; sema-address instruments exhausted short of blind assist
* wall2 session 109: zero-ioctl window vs 24% kernel profile = BURSTY retry ↵Arthur Hanson Rasmusson18 hours
| | | | rounds between 200ms heartbeats; the 299-byte reader named as a health-poll candidate; next = name the node and capture the burst with host-RPC correlation
* wall2 session 108: content-diff found 9 active driver pages (counter/staging ↵Arthur Hanson Rasmusson18 hours
| | | | semantics catalogued); the stuck loop's read is constant -> the sema is a static cell; next = bounded zero-cell sweep among the constant pages adjacent to the active set
* wall2 session 107: FB assist sweeps negative (320+1024 cells; perf still ↵Arthur Hanson Rasmusson18 hours
| | | | 23.13%) - memdesc read not FB-sourced; the byte-identical stream twins (0x1026ad000/0x12afba000) are the memdesc-backing candidates for the next assist sweep
* wall2 session 106: channel buffer is driver-internal (not in ↵Arthur Hanson Rasmusson18 hours
| | | | NV_CHANNEL_ALLOC_PARAMS; kprobe was the right path, notrace-blocked); finish-tail sweep + auth-tag hash scan negative; next = transfer-shadow properties or bounded FB assist
* wall2 session 105: kprobe blocked (channelReadChannelMemdesc is notrace; ↵Arthur Hanson Rasmusson18 hours
| | | | tracefs mounted, UVM kprobes pre-existing); BAR1-path analysis (pokes landed in the shadow yet no flip -> memdesc page != FB 0x660b10); channel-field offsets enumerated; next = CE channel alloc-params dump for the channel-buffer field
* wall2 session 104: sema VA 0x4b19b10 verified NOT the loop page (converged ↵Arthur Hanson Rasmusson18 hours
| | | | FB 0x660b10 pokes landed, no flip; perf still 24.52%); kprobe on channelReadChannelMemdesc names the real authTagBufSemaOffset (offset=%si) + the channel pointer -> memdesc
* wall2 session 103: scrub init stream content-located (pa=0x114e98000 + ↵Arthur Hanson Rasmusson18 hours
| | | | twins); release 0x4b17e30 delivered (cur=1); auth-tag sema 0x4b19b10 page ambiguous (FB 0x660b10 garbage did not flip; linear guesses zero); next = validate-by-value walk pick
* wall2 session 102: fd sampler found no transient (bursty reader); no nvidia ↵Arthur Hanson Rasmusson18 hours
| | | | debugfs; memdesc paths ranked (content-locate the channel buffer by scrub method shapes / scrubber sema-push decodes / narrower burst sampling)
* wall2 session 101: auth-tag wait loop decoded (sema at ↵Arthur Hanson Rasmusson18 hours
| | | | pChannelBufferMemdesc+finishPayloadOffset+4, exits when GetPtr passes nextPut); perf trace (no-ptrace) shows the 10ms futex retry + a transient open/read(299)/close polling reader; next = catch the transient fd path and write the sema
* wall2 session 100: CE userd hunt exhausted standard layouts (registered ↵Arthur Hanson Rasmusson19 hours
| | | | FB=zero, ring+0x10000 non-contiguous guesses=zero, PDB-walked gpfifo+0x10000 pages=all-zero); the SEC2 scrub flow is a circular wait (no consumption without the CE pump; no pump without a live userd; no userd without submits); next = write the auth-tag sema in the channel buffer (memdesc) page directly to free the slots
* wall2 session 99: scrubber gate connected to CE channels ok=0 (userd ↵Arthur Hanson Rasmusson19 hours
| | | | discovery fails for all CE-class channels; GR ok=1 and pumping); find_userd FB scan = zero; host shape scan too noisy; next = diff-watch to pinpoint the real CE userd page
* wall2 session 98: post-device spin NAMED via perf (no ptrace) - the SEC2 ↵Arthur Hanson Rasmusson19 hours
| | | | memory-scrubber channel waits on its auth-tag semaphore (channelReadChannelMemdesc 24.33%, _sec2utilsGetNextAuthTagSlot loop); fix direction = service-side completion of the scrubber's semaphore releases (authTagBufSemaOffset in the channel buffer); perf is the new no-ptrace spin locator
* wall2 session 97: deep submission state reached (gpfifo done=11576, ↵Arthur Hanson Rasmusson19 hours
| | | | pushes=10932); counter-slot collision (stale owner) + sema_counter bypass applied (pub_calls 1356->3739); post-device spin is the remaining gate; ioctl-arg sampling axis documented
* wall2 session 96: detach-first clearer t106 PROVEN (BC-DETACHED, TracerPid ↵Arthur Hanson Rasmusson19 hours
| | | | 0, VKDEVICE via 5 words on r696); orphan-tracer plague root-caused (interrupted gdb attempts) with strict discipline; next t107 = clearer+diagnostic in one self-contained gdb window
* wall2 session 95: gdb thread-stop interference freezes clients (workers ↵Arthur Hanson Rasmusson19 hours
| | | | stuck in ptrace_stop); SIGKILL nuance (tracees resumed when stopped at futex, survived); RM-lock holder dynamics; harness rules + service-side folding as the durable answer
* wall2 session 94: instance clears proven REQUIRED (zero-gdb run still -4'd); ↵Arthur Hanson Rasmusson19 hours
| | | | t104 self-exiting clearer + unwait reaches VKDEVICE again; next: detach-from-handler t104 so no gdb is ever held + t102 wait identification
* wall2 session 93: zero-ptrace pipeline (t101 instance clears self-disabling ↵Arthur Hanson Rasmusson20 hours
| | | | + unwait.py 16K/+0x80 word poller) reaches VKDEVICE on r693 with 5 auto-written words and no wedge risk; RM-lock wedge root-caused (gdb killed at a stop inside an RM critical section) with new attach rules + t102 safe diagnostic
* wall2 session 92: automated zero-touch pipeline reaches VKDEVICE on a fresh ↵Arthur Hanson Rasmusson20 hours
| | | | boot (t99 = marker-clear at +0xB33D30 + wait-word chase/write at +0xDCEBD9); live_walk PT injections alongside; attach-discipline lesson (wedged continue blocks later attaches); next = post-device wait identification
* wall2 session 91: vkCreateInstance wall BREACHED - the event-slot ↵Arthur Hanson Rasmusson21 hours
| | | | pending-marker (u16@+0xe=0xffff) cleared at each +0xB33D30 retry entry passes the instance phase for the first time; client then persists in vkCreateDevice's nvidia0 poll waiting for the event-fd signal (guest dispatch gap); tools t93/t94/t96b/t97
* wall2 session 90: current-boot wait site identified (eglcore+0xB2852D ↵Arthur Hanson Rasmusson21 hours
| | | | in-process word pair wait inside vkCreateInstance; old 0xa85d24/16K sites dead this boot); t83 sampler + widened nudge tools; multi-word nudge trips -4 (caution confirmed)
* wall2 session 89: counter-page record resolve proven live (slot learn + ↵Arthur Hanson Rasmusson21 hours
| | | | pub_calls 0->172); GR ctx-init release apply remains the single unmet wait
* wall2 session 89: GMMU PT walk replicated host-side + live inject ↵Arthur Hanson Rasmusson21 hours
| | | | (validated: ring entries and push method streams match); gpfifo done 0->6; root cause of -4 = release-apply gates (gr_forward defer-arm, op_tr_pending opdef withholding) + unresolvable counter page 0x20400f000 (slot stays pending); counter-page {pa,va} record tables found for host-side resolve (live only)
* wall2: session 88 - push page-5 injection fix (proven +30/sweep), ↵Arthur Hanson Rasmusson22 hours
| | | | host_resolve tool, RING GAP diagnosis (gpfifo done=0), gdb-SIGKILL RM-lock wedge lesson
* wall-2 session-87: window page-list hunt - the 2 GiB cycle's windows are ↵Arthur Hanson Rasmusson22 hours
| | | | scattered (later offsets at first_PA+off are zeros); the memdesc {size 0x20000, PA=0x107c47000, kptrs} page list is only reachable via the kernel pointers (kcore walk with fresh addresses needed); content search for the init-stream prefix found the window pages (0x103c06000+0x858, 0x11dbfa000+0x858, 0x126d69000+0x59c) but the VA<->page relation is non-linear; the client blocks in os_schedule at the allocation (the unread later pushes suspected)
* wall-2 session-86 tools: vk_memprops.py (standalone Vulkan memory-properties ↵Arthur Hanson Rasmusson23 hours
| | | | dump), vk_alloc_test.py (standalone allocation reproducer), t91.py (auto-clear + ICD memprops/alloc hooks), t90.py (A8 count auto-clear)
* wall-2 session-86: the DEVICE_LOCAL -2 wall CRACKED - the 512 MiB emulated ↵Arthur Hanson Rasmusson23 hours
| | | | VRAM was the gate (the ICD hooks captured ALLOC size=0x4000 typeIndex=1 valid; vk_memprops shows device-local heap[0]=512 MiB); the nvidia-nvidia-vgpu-turing-2g mdev type (2 GiB heap, VGPU_MDEV_TYPE env var honoured by start_vm_keep.sh) lets the same client with the same recipe proceed past vkAllocateMemory into a deeper wait (os_schedule, no -2); remaining: the later pushes at va=+0x52d4/+0x586c read pa=0x0 (window page list scattered; injections covered page 1 only) - extend the window injections; tools: vk_memprops.py, vk_alloc_test.py (hits -4 without the harness), t91.py
* wall-2 session-85 BREAKTHROUGH: the NvOgl wait CLEARED - the A8 loop's exit ↵Arthur Hanson Rasmusson23 hours
| | | | needs the session count *(u32*)(lib_base+0x2070b7c) to reach 0 (the engine sessions' completion counter); a direct gdb write of 0 cleared the wait and the client ADVANCED past the post-device phase for the first time (VKDEVICE ok then VKFAIL vkAllocateMemory -> -2 OUT_OF_DEVICE_MEMORY at vk_frame.c:204 - the new wall: the DEVICE_LOCAL heap/type reporting, no alloc RPC reached the emulation)
* wall-2 session-84: the a86038 want-scan mapped - all four entries PASS with ↵Arthur Hanson Rasmusson23 hours
| | | | the driven window value (wants {0,0,0,6} vs actuals {0,0,0,0xffffffff00000100}, no pending); a full-window write (0x0..0x1ff0) does not clear the wait; the residual gate is the A8 loop's internal callback/retry machinery (the a85f30 exit path decrements *(rbp) while firing the global callbacks 0x206f790/0x206f7a0 with args from rbp+0x30/0x38/0x40) and the slow yield-spin cadence (b284b0(...,0xb))
* wall-2 session-83: the A8 completion counter decoded to a client readback ↵Arthur Hanson Rasmusson23 hours
| | | | window - a82f30 -> *(obj+0x19b8)=eglcore+0xb250f0 -> b1bf00 (slow path obj+0x224==0) -> b27c90 reads *(u32*)(*(*(obj+0x1480)+0x58)+idx*16) (idx 0xbb.., min over n=*(u32*)(obj+0x22efc)); the window for handle 0xbeec0100 is VA 0x7827e6c12000 (/dev/nvidiactl) and the host CAN write it via the RAM file: writing 0x100 made the client read it (obj+0x1070=0x100, actual table entry 0xffffffff00000100) - first live proof the counter can be driven from the host; the wait persists in the loop's second scan (a85ec1/b29dc0) and the *(rsp+0x38) branch
* wall-2 session-82: counter-page PA hunt - the client's pagemap does not ↵Arthur Hanson Rasmusson24 hours
| | | | contain the counter pool (guest_pmap_win.py empty for 0x20400f000 and 0x121000000 ranges; semfix_uvmt's pin_counter_page never succeeded) and the vidmem-window hypothesis is not supported (poke_vidmem at the pool window offset 0x3F87000 + rank slots 0xfd0/0xfc0/0xfb0 with 6 then 0x40 changed the shadow but did not move the A8 wait); gdb attach hangs on the post-device spin (kill by PID + SIGCONT recovery)
* wall-2 session-81: counter-autoreg gap analysed - svc_counter_autoreg needs ↵Arthur Hanson Rasmusson24 hours
| | | | an ipin pin or svc_counter_derive for the rank slot (0x20400fec0+(19-r)*0x10); both miss for the counter page 0x20400f000 (the client does not map it - pagemap empty, no VMA; it is a UVM kernel object; the pin stream's sem resolutions are 0 despite 44 want-list entries); bypass: the sema_counter param registers a slot manually (echo <h_object>,<pa>); GL ranks 0,1,2 -> slots 0x20400ffd0/0xfc0/0xfb0
* wall-2 session-80: ALL low-aperture window records found via the 'liat'-tag ↵Arthur Hanson Rasmusson24 hours
| | | | fix (the record tag is the u32 0x7461696c whose on-disk bytes are 'liat', not 'tail'); wall2_recscan.py scans them; the GR init semaphore windows identified and injected (0x4b17000->0x12a76f000 verified by content: +0xbb0=3/+0xe30=1 = the release payloads; 0x4b19000->0x122f9b000, release applied there) - releases now land on the semaphore pages; the NvOgl wait still spins (the A8 loop's per-entry counter tables are the remaining completion condition)
* wall-2 session-79: resolver7 tool + the corrected pair-specific injection ↵Arthur Hanson Rasmusson24 hours
| | | | recipe (0x4480000->0x125692000, 0x4ca0000->0x131649000, 0x4b16000->0x12a76a000) and the window-record layout {tag,VA,SIZE,0,tail} stride 0x40; documents the resolver7 cross-pair bug, the decoded low-VA releases and the remaining semaphore-VA translations (pending=417)
* vgsp-sema r686: accept the low aperture (>=0x4000000) for GR/GL release ↵Arthur Hanson Rasmusson24 hours
| | | | | | | | | | | | | | | decode in both release forms (0x5c-form and SET_REPORT_SEMAPHORE) The GR/GL ucode init pushes carry their releases on the RM's low aperture (the same 0x4000000-0x4ffffff VA family as the push/ring windows): the live GL init push decodes as rel=0x4b19b10 pay=1 (REPORT_SEM group 200426c0 00000000 04b19b10 00000001 10000000) and rel=0x4b17bb0 pay=3 / 0x4b17e30 pay=1 (0x5c-form 2005c017 04923060 ...), all rejected by the sysmem-only SEMA_VA_LO bound - the NvOgl engine-init wait then spun forever. With the bound relaxed (and the push windows injected via the window-record/memdesc recipe) the releases decode live (have_rel=1, applied rising) and the un-translated ones queue in svc_pending (417 at the observed steady state) awaiting the semaphore-VA translations.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-26 12:39:13 +0000