summaryrefslogtreecommitdiffstats
path: root/tools/guest-diag/wall2/README-session77.md
Commit message (Collapse)AuthorAge
* wall2 session 115: decompose the first-fence wall - the sema service cannot ↵Arthur Hanson Rasmusson4 hours
| | | | decode the submit's GR pushes because the client VAS page directory is undiscoverable (walk_probe: discovered pdb=0x0; the COPY_SERVER_PDES candidate is garbage; the r496f vidmem discovery never runs) and the pushbuffer is noncontiguous; the RAM ring scan gives up after 12 slices before the pushes exist; ring pages ARE content-locatable by their GPFIFO entries and injectable via sema_map_add (fence_assist.py) but the ring pair alone does not unlock the push decode; next = extract the pdb from the guest kernel's RM objects (PDB injection) or scan vidmem with the ring pair as verifier
* wall2 session 114: the COMPR+2MB VidHeap class (attr&0xF000==0x2000, owner ↵Arthur Hanson Rasmusson4 hours
| | | | 0x474C0000) is the last allocation gate - a live attr flip (COMPR_ANY->NONE at the ioctl) lets vkAllocateMemory through and the client runs the whole pipeline to the FIRST vkQueueSubmit, then times out on the FIRST fence (30 s, the new wall); the zero-touch create recipe is BC (eglcore+0xB33D30 marker clears) + BW (eglcore+0xDCEBD9 device wait-word chaser, the missing piece) + the a8clear count gate; the fence release (push-page readability/completion after the submit) is the only remaining gate to VKFRAME
* wall2 session 111: FB FULLY disproved (whole-256MB fill_vidmem no flip -> ↵Arthur Hanson Rasmusson7 hours
| | | | memdesc not FB-backed); refined assist negative; SYS-fill of active neighborhoods wedged the guest (cycle next; never bulk-overwrite active-set-adjacent pages)
* wall2 session 110: the sec2 loop runs CONTINUOUSLY inside one long-lived ↵Arthur Hanson Rasmusson7 hours
| | | | ioctl that holds the RM write lock (closes the lock model: tid 1791 read-wait, new clients D-state); the 299-byte reader is a red herring (address never mapped, 5/s); PEBS unavailable; sema-address instruments exhausted short of blind assist
* wall2 session 109: zero-ioctl window vs 24% kernel profile = BURSTY retry ↵Arthur Hanson Rasmusson7 hours
| | | | rounds between 200ms heartbeats; the 299-byte reader named as a health-poll candidate; next = name the node and capture the burst with host-RPC correlation
* wall2 session 108: content-diff found 9 active driver pages (counter/staging ↵Arthur Hanson Rasmusson7 hours
| | | | semantics catalogued); the stuck loop's read is constant -> the sema is a static cell; next = bounded zero-cell sweep among the constant pages adjacent to the active set
* wall2 session 107: FB assist sweeps negative (320+1024 cells; perf still ↵Arthur Hanson Rasmusson7 hours
| | | | 23.13%) - memdesc read not FB-sourced; the byte-identical stream twins (0x1026ad000/0x12afba000) are the memdesc-backing candidates for the next assist sweep
* wall2 session 106: channel buffer is driver-internal (not in ↵Arthur Hanson Rasmusson7 hours
| | | | NV_CHANNEL_ALLOC_PARAMS; kprobe was the right path, notrace-blocked); finish-tail sweep + auth-tag hash scan negative; next = transfer-shadow properties or bounded FB assist
* wall2 session 105: kprobe blocked (channelReadChannelMemdesc is notrace; ↵Arthur Hanson Rasmusson7 hours
| | | | tracefs mounted, UVM kprobes pre-existing); BAR1-path analysis (pokes landed in the shadow yet no flip -> memdesc page != FB 0x660b10); channel-field offsets enumerated; next = CE channel alloc-params dump for the channel-buffer field
* wall2 session 104: sema VA 0x4b19b10 verified NOT the loop page (converged ↵Arthur Hanson Rasmusson7 hours
| | | | FB 0x660b10 pokes landed, no flip; perf still 24.52%); kprobe on channelReadChannelMemdesc names the real authTagBufSemaOffset (offset=%si) + the channel pointer -> memdesc
* wall2 session 103: scrub init stream content-located (pa=0x114e98000 + ↵Arthur Hanson Rasmusson7 hours
| | | | twins); release 0x4b17e30 delivered (cur=1); auth-tag sema 0x4b19b10 page ambiguous (FB 0x660b10 garbage did not flip; linear guesses zero); next = validate-by-value walk pick
* wall2 session 102: fd sampler found no transient (bursty reader); no nvidia ↵Arthur Hanson Rasmusson7 hours
| | | | debugfs; memdesc paths ranked (content-locate the channel buffer by scrub method shapes / scrubber sema-push decodes / narrower burst sampling)
* wall2 session 101: auth-tag wait loop decoded (sema at ↵Arthur Hanson Rasmusson7 hours
| | | | pChannelBufferMemdesc+finishPayloadOffset+4, exits when GetPtr passes nextPut); perf trace (no-ptrace) shows the 10ms futex retry + a transient open/read(299)/close polling reader; next = catch the transient fd path and write the sema
* wall2 session 100: CE userd hunt exhausted standard layouts (registered ↵Arthur Hanson Rasmusson7 hours
| | | | FB=zero, ring+0x10000 non-contiguous guesses=zero, PDB-walked gpfifo+0x10000 pages=all-zero); the SEC2 scrub flow is a circular wait (no consumption without the CE pump; no pump without a live userd; no userd without submits); next = write the auth-tag sema in the channel buffer (memdesc) page directly to free the slots
* wall2 session 99: scrubber gate connected to CE channels ok=0 (userd ↵Arthur Hanson Rasmusson7 hours
| | | | discovery fails for all CE-class channels; GR ok=1 and pumping); find_userd FB scan = zero; host shape scan too noisy; next = diff-watch to pinpoint the real CE userd page
* wall2 session 98: post-device spin NAMED via perf (no ptrace) - the SEC2 ↵Arthur Hanson Rasmusson7 hours
| | | | memory-scrubber channel waits on its auth-tag semaphore (channelReadChannelMemdesc 24.33%, _sec2utilsGetNextAuthTagSlot loop); fix direction = service-side completion of the scrubber's semaphore releases (authTagBufSemaOffset in the channel buffer); perf is the new no-ptrace spin locator
* wall2 session 97: deep submission state reached (gpfifo done=11576, ↵Arthur Hanson Rasmusson7 hours
| | | | pushes=10932); counter-slot collision (stale owner) + sema_counter bypass applied (pub_calls 1356->3739); post-device spin is the remaining gate; ioctl-arg sampling axis documented
* wall2 session 96: detach-first clearer t106 PROVEN (BC-DETACHED, TracerPid ↵Arthur Hanson Rasmusson7 hours
| | | | 0, VKDEVICE via 5 words on r696); orphan-tracer plague root-caused (interrupted gdb attempts) with strict discipline; next t107 = clearer+diagnostic in one self-contained gdb window
* wall2 session 95: gdb thread-stop interference freezes clients (workers ↵Arthur Hanson Rasmusson8 hours
| | | | stuck in ptrace_stop); SIGKILL nuance (tracees resumed when stopped at futex, survived); RM-lock holder dynamics; harness rules + service-side folding as the durable answer
* wall2 session 94: instance clears proven REQUIRED (zero-gdb run still -4'd); ↵Arthur Hanson Rasmusson8 hours
| | | | t104 self-exiting clearer + unwait reaches VKDEVICE again; next: detach-from-handler t104 so no gdb is ever held + t102 wait identification
* wall2 session 93: zero-ptrace pipeline (t101 instance clears self-disabling ↵Arthur Hanson Rasmusson8 hours
| | | | + unwait.py 16K/+0x80 word poller) reaches VKDEVICE on r693 with 5 auto-written words and no wedge risk; RM-lock wedge root-caused (gdb killed at a stop inside an RM critical section) with new attach rules + t102 safe diagnostic
* wall2 session 92: automated zero-touch pipeline reaches VKDEVICE on a fresh ↵Arthur Hanson Rasmusson9 hours
| | | | boot (t99 = marker-clear at +0xB33D30 + wait-word chase/write at +0xDCEBD9); live_walk PT injections alongside; attach-discipline lesson (wedged continue blocks later attaches); next = post-device wait identification
* wall2 session 91: vkCreateInstance wall BREACHED - the event-slot ↵Arthur Hanson Rasmusson9 hours
| | | | pending-marker (u16@+0xe=0xffff) cleared at each +0xB33D30 retry entry passes the instance phase for the first time; client then persists in vkCreateDevice's nvidia0 poll waiting for the event-fd signal (guest dispatch gap); tools t93/t94/t96b/t97
* wall2 session 90: current-boot wait site identified (eglcore+0xB2852D ↵Arthur Hanson Rasmusson10 hours
| | | | in-process word pair wait inside vkCreateInstance; old 0xa85d24/16K sites dead this boot); t83 sampler + widened nudge tools; multi-word nudge trips -4 (caution confirmed)
* wall2 session 89: counter-page record resolve proven live (slot learn + ↵Arthur Hanson Rasmusson10 hours
| | | | pub_calls 0->172); GR ctx-init release apply remains the single unmet wait
* wall2 session 89: GMMU PT walk replicated host-side + live inject ↵Arthur Hanson Rasmusson10 hours
| | | | (validated: ring entries and push method streams match); gpfifo done 0->6; root cause of -4 = release-apply gates (gr_forward defer-arm, op_tr_pending opdef withholding) + unresolvable counter page 0x20400f000 (slot stays pending); counter-page {pa,va} record tables found for host-side resolve (live only)
* wall2: session 88 - push page-5 injection fix (proven +30/sweep), ↵Arthur Hanson Rasmusson11 hours
| | | | host_resolve tool, RING GAP diagnosis (gpfifo done=0), gdb-SIGKILL RM-lock wedge lesson
* wall-2 session-87: window page-list hunt - the 2 GiB cycle's windows are ↵Arthur Hanson Rasmusson11 hours
| | | | scattered (later offsets at first_PA+off are zeros); the memdesc {size 0x20000, PA=0x107c47000, kptrs} page list is only reachable via the kernel pointers (kcore walk with fresh addresses needed); content search for the init-stream prefix found the window pages (0x103c06000+0x858, 0x11dbfa000+0x858, 0x126d69000+0x59c) but the VA<->page relation is non-linear; the client blocks in os_schedule at the allocation (the unread later pushes suspected)
* wall-2 session-86: the DEVICE_LOCAL -2 wall CRACKED - the 512 MiB emulated ↵Arthur Hanson Rasmusson11 hours
| | | | VRAM was the gate (the ICD hooks captured ALLOC size=0x4000 typeIndex=1 valid; vk_memprops shows device-local heap[0]=512 MiB); the nvidia-nvidia-vgpu-turing-2g mdev type (2 GiB heap, VGPU_MDEV_TYPE env var honoured by start_vm_keep.sh) lets the same client with the same recipe proceed past vkAllocateMemory into a deeper wait (os_schedule, no -2); remaining: the later pushes at va=+0x52d4/+0x586c read pa=0x0 (window page list scattered; injections covered page 1 only) - extend the window injections; tools: vk_memprops.py, vk_alloc_test.py (hits -4 without the harness), t91.py
* wall-2 session-85 BREAKTHROUGH: the NvOgl wait CLEARED - the A8 loop's exit ↵Arthur Hanson Rasmusson12 hours
| | | | needs the session count *(u32*)(lib_base+0x2070b7c) to reach 0 (the engine sessions' completion counter); a direct gdb write of 0 cleared the wait and the client ADVANCED past the post-device phase for the first time (VKDEVICE ok then VKFAIL vkAllocateMemory -> -2 OUT_OF_DEVICE_MEMORY at vk_frame.c:204 - the new wall: the DEVICE_LOCAL heap/type reporting, no alloc RPC reached the emulation)
* wall-2 session-84: the a86038 want-scan mapped - all four entries PASS with ↵Arthur Hanson Rasmusson12 hours
| | | | the driven window value (wants {0,0,0,6} vs actuals {0,0,0,0xffffffff00000100}, no pending); a full-window write (0x0..0x1ff0) does not clear the wait; the residual gate is the A8 loop's internal callback/retry machinery (the a85f30 exit path decrements *(rbp) while firing the global callbacks 0x206f790/0x206f7a0 with args from rbp+0x30/0x38/0x40) and the slow yield-spin cadence (b284b0(...,0xb))
* wall-2 session-83: the A8 completion counter decoded to a client readback ↵Arthur Hanson Rasmusson12 hours
| | | | window - a82f30 -> *(obj+0x19b8)=eglcore+0xb250f0 -> b1bf00 (slow path obj+0x224==0) -> b27c90 reads *(u32*)(*(*(obj+0x1480)+0x58)+idx*16) (idx 0xbb.., min over n=*(u32*)(obj+0x22efc)); the window for handle 0xbeec0100 is VA 0x7827e6c12000 (/dev/nvidiactl) and the host CAN write it via the RAM file: writing 0x100 made the client read it (obj+0x1070=0x100, actual table entry 0xffffffff00000100) - first live proof the counter can be driven from the host; the wait persists in the loop's second scan (a85ec1/b29dc0) and the *(rsp+0x38) branch
* wall-2 session-82: counter-page PA hunt - the client's pagemap does not ↵Arthur Hanson Rasmusson12 hours
| | | | contain the counter pool (guest_pmap_win.py empty for 0x20400f000 and 0x121000000 ranges; semfix_uvmt's pin_counter_page never succeeded) and the vidmem-window hypothesis is not supported (poke_vidmem at the pool window offset 0x3F87000 + rank slots 0xfd0/0xfc0/0xfb0 with 6 then 0x40 changed the shadow but did not move the A8 wait); gdb attach hangs on the post-device spin (kill by PID + SIGCONT recovery)
* wall-2 session-81: counter-autoreg gap analysed - svc_counter_autoreg needs ↵Arthur Hanson Rasmusson12 hours
| | | | an ipin pin or svc_counter_derive for the rank slot (0x20400fec0+(19-r)*0x10); both miss for the counter page 0x20400f000 (the client does not map it - pagemap empty, no VMA; it is a UVM kernel object; the pin stream's sem resolutions are 0 despite 44 want-list entries); bypass: the sema_counter param registers a slot manually (echo <h_object>,<pa>); GL ranks 0,1,2 -> slots 0x20400ffd0/0xfc0/0xfb0
* wall-2 session-80: ALL low-aperture window records found via the 'liat'-tag ↵Arthur Hanson Rasmusson12 hours
| | | | fix (the record tag is the u32 0x7461696c whose on-disk bytes are 'liat', not 'tail'); wall2_recscan.py scans them; the GR init semaphore windows identified and injected (0x4b17000->0x12a76f000 verified by content: +0xbb0=3/+0xe30=1 = the release payloads; 0x4b19000->0x122f9b000, release applied there) - releases now land on the semaphore pages; the NvOgl wait still spins (the A8 loop's per-entry counter tables are the remaining completion condition)
* wall-2 session-79: resolver7 tool + the corrected pair-specific injection ↵Arthur Hanson Rasmusson13 hours
| | | | recipe (0x4480000->0x125692000, 0x4ca0000->0x131649000, 0x4b16000->0x12a76a000) and the window-record layout {tag,VA,SIZE,0,tail} stride 0x40; documents the resolver7 cross-pair bug, the decoded low-VA releases and the remaining semaphore-VA translations (pending=417)
* wall-2 session-78: post-device wait identified as the NvOgl (GL ucode) ↵Arthur Hanson Rasmusson13 hours
| | | | engine init (A8 retry loop work item named NvOgl_m0; D6 submit ORs 0x200000 into the pending word) and the push windows become READABLE via the resolver6 window-record/memdesc recipe (0x4480000 -> 0x120cfb000, 0x4ca0000 -> 0x12e51e000; the service decodes the GR init streams where sessions 73-74 saw unread) - remaining gate: have_rel=0 (CE-shaped matcher) and the GR forward not engaging (forwarded=0 gr_nocap=4 drain held=4)
* wall-2 session-77: vkCreateDevice PASSES - r682 0x7E event validated live ↵Arthur Hanson Rasmusson13 hours
(registered slot 13, EVPOST rc=0) and the completion word found/driven: the main-thread wait reads a u32 at +0x80 of a 16 KB rw-s /dev/nvidiactl notification mapping (should-wait = word != 1); a host write of 1 via the guest RAM file (PA from pagemap, file offset PA-0x80000000 above 4G) flips the wait and vkCreateDevice returns success (VKDEVICE ok family=0) - reproduced 3x (r683 manual, r684 nudger, r684d probe+single-word). Tools: t79.py (wait-chain resolver), guest_words.sh, nudge.sh; README-session77.md records the recipe, the residual guest event-dispatch gap (records consumed, notification word not written), and the -4/gdb-attach cautions.
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-26 01:16:57 +0000