summaryrefslogtreecommitdiffstats
path: root/tools/guest-diag/wall2
Commit message (Collapse)AuthorAge
* wall-2 session-86 tools: vk_memprops.py (standalone Vulkan memory-properties ↵Arthur Hanson Rasmusson7 hours
| | | | dump), vk_alloc_test.py (standalone allocation reproducer), t91.py (auto-clear + ICD memprops/alloc hooks), t90.py (A8 count auto-clear)
* wall-2 session-86: the DEVICE_LOCAL -2 wall CRACKED - the 512 MiB emulated ↵Arthur Hanson Rasmusson7 hours
| | | | VRAM was the gate (the ICD hooks captured ALLOC size=0x4000 typeIndex=1 valid; vk_memprops shows device-local heap[0]=512 MiB); the nvidia-nvidia-vgpu-turing-2g mdev type (2 GiB heap, VGPU_MDEV_TYPE env var honoured by start_vm_keep.sh) lets the same client with the same recipe proceed past vkAllocateMemory into a deeper wait (os_schedule, no -2); remaining: the later pushes at va=+0x52d4/+0x586c read pa=0x0 (window page list scattered; injections covered page 1 only) - extend the window injections; tools: vk_memprops.py, vk_alloc_test.py (hits -4 without the harness), t91.py
* wall-2 session-85 BREAKTHROUGH: the NvOgl wait CLEARED - the A8 loop's exit ↵Arthur Hanson Rasmusson8 hours
| | | | needs the session count *(u32*)(lib_base+0x2070b7c) to reach 0 (the engine sessions' completion counter); a direct gdb write of 0 cleared the wait and the client ADVANCED past the post-device phase for the first time (VKDEVICE ok then VKFAIL vkAllocateMemory -> -2 OUT_OF_DEVICE_MEMORY at vk_frame.c:204 - the new wall: the DEVICE_LOCAL heap/type reporting, no alloc RPC reached the emulation)
* wall-2 session-84: the a86038 want-scan mapped - all four entries PASS with ↵Arthur Hanson Rasmusson8 hours
| | | | the driven window value (wants {0,0,0,6} vs actuals {0,0,0,0xffffffff00000100}, no pending); a full-window write (0x0..0x1ff0) does not clear the wait; the residual gate is the A8 loop's internal callback/retry machinery (the a85f30 exit path decrements *(rbp) while firing the global callbacks 0x206f790/0x206f7a0 with args from rbp+0x30/0x38/0x40) and the slow yield-spin cadence (b284b0(...,0xb))
* wall-2 session-83: the A8 completion counter decoded to a client readback ↵Arthur Hanson Rasmusson8 hours
| | | | window - a82f30 -> *(obj+0x19b8)=eglcore+0xb250f0 -> b1bf00 (slow path obj+0x224==0) -> b27c90 reads *(u32*)(*(*(obj+0x1480)+0x58)+idx*16) (idx 0xbb.., min over n=*(u32*)(obj+0x22efc)); the window for handle 0xbeec0100 is VA 0x7827e6c12000 (/dev/nvidiactl) and the host CAN write it via the RAM file: writing 0x100 made the client read it (obj+0x1070=0x100, actual table entry 0xffffffff00000100) - first live proof the counter can be driven from the host; the wait persists in the loop's second scan (a85ec1/b29dc0) and the *(rsp+0x38) branch
* wall-2 session-82: counter-page PA hunt - the client's pagemap does not ↵Arthur Hanson Rasmusson8 hours
| | | | contain the counter pool (guest_pmap_win.py empty for 0x20400f000 and 0x121000000 ranges; semfix_uvmt's pin_counter_page never succeeded) and the vidmem-window hypothesis is not supported (poke_vidmem at the pool window offset 0x3F87000 + rank slots 0xfd0/0xfc0/0xfb0 with 6 then 0x40 changed the shadow but did not move the A8 wait); gdb attach hangs on the post-device spin (kill by PID + SIGCONT recovery)
* wall-2 session-81: counter-autoreg gap analysed - svc_counter_autoreg needs ↵Arthur Hanson Rasmusson8 hours
| | | | an ipin pin or svc_counter_derive for the rank slot (0x20400fec0+(19-r)*0x10); both miss for the counter page 0x20400f000 (the client does not map it - pagemap empty, no VMA; it is a UVM kernel object; the pin stream's sem resolutions are 0 despite 44 want-list entries); bypass: the sema_counter param registers a slot manually (echo <h_object>,<pa>); GL ranks 0,1,2 -> slots 0x20400ffd0/0xfc0/0xfb0
* wall-2 session-80: ALL low-aperture window records found via the 'liat'-tag ↵Arthur Hanson Rasmusson8 hours
| | | | fix (the record tag is the u32 0x7461696c whose on-disk bytes are 'liat', not 'tail'); wall2_recscan.py scans them; the GR init semaphore windows identified and injected (0x4b17000->0x12a76f000 verified by content: +0xbb0=3/+0xe30=1 = the release payloads; 0x4b19000->0x122f9b000, release applied there) - releases now land on the semaphore pages; the NvOgl wait still spins (the A8 loop's per-entry counter tables are the remaining completion condition)
* wall-2 session-79: resolver7 tool + the corrected pair-specific injection ↵Arthur Hanson Rasmusson8 hours
| | | | recipe (0x4480000->0x125692000, 0x4ca0000->0x131649000, 0x4b16000->0x12a76a000) and the window-record layout {tag,VA,SIZE,0,tail} stride 0x40; documents the resolver7 cross-pair bug, the decoded low-VA releases and the remaining semaphore-VA translations (pending=417)
* wall-2 session-78: post-device wait identified as the NvOgl (GL ucode) ↵Arthur Hanson Rasmusson9 hours
| | | | engine init (A8 retry loop work item named NvOgl_m0; D6 submit ORs 0x200000 into the pending word) and the push windows become READABLE via the resolver6 window-record/memdesc recipe (0x4480000 -> 0x120cfb000, 0x4ca0000 -> 0x12e51e000; the service decodes the GR init streams where sessions 73-74 saw unread) - remaining gate: have_rel=0 (CE-shaped matcher) and the GR forward not engaging (forwarded=0 gr_nocap=4 drain held=4)
* wall-2 session-77: vkCreateDevice PASSES - r682 0x7E event validated live ↵Arthur Hanson Rasmusson9 hours
| | | | (registered slot 13, EVPOST rc=0) and the completion word found/driven: the main-thread wait reads a u32 at +0x80 of a 16 KB rw-s /dev/nvidiactl notification mapping (should-wait = word != 1); a host write of 1 via the guest RAM file (PA from pagemap, file offset PA-0x80000000 above 4G) flips the wait and vkCreateDevice returns success (VKDEVICE ok family=0) - reproduced 3x (r683 manual, r684 nudger, r684d probe+single-word). Tools: t79.py (wait-chain resolver), guest_words.sh, nudge.sh; README-session77.md records the recipe, the residual guest event-dispatch gap (records consumed, notification word not written), and the -4/gdb-attach cautions.
* wall-2 session-76 addendum (r681): decode the main-thread vkCreateDevice ↵Arthur Hanson Rasmusson10 hours
| | | | wait - should-wait = (readiness word != 1), readiness = u32 at nvidiactl-mapped holder+0x80 (VM_IO page, 39/39 reads zero, all fd polls time out), give-up predicate dcee20 fires -4 only on RM error flags (b29d60 / +0x70cc); the word's writer is the guest kernel driver/RM, not host RAM
* vgsp-rm r680 + wall-2 session-76 forensics: 0x2080a097 reply record-area ↵Arthur Hanson Rasmusson10 hours
| | | | | | | | | | | | | | | | | | | | tunables (wall2_a097_count/rec0..rec7, default 0xffffffff = echo) and the evidence set that re-questions the a097 gate The real host RM 0x2080a097 handler (GSP-RM 570.144 control table entry, params 0x490) overwrites params+4..+0x23 with up to four {u32,u32} mode pairs (engine value + a x1000 clock/percent) and the client copies back the first 16 bytes; the tick at eglcore 0xd5e3e0 consumes the second word of the first pair as the per-engine progress (session +0xe104, compared against per-boot config threshold 95 at lib+0x2019320+0xffc with 10/20 bit-history cutoffs). New module params let one boot sweep the record matrix; the r680 sweep (flag/v488/v48c + count/rec matrix incl. progress 100..1000000) changed nothing observable, and hw watchpoints show E+0x150 stays 0x00200000 with no writer over 5+ minutes while the a097 poller is a worker thread and the main thread sits in a 10 s poll() loop on /dev/nvidia0 inside vkCreateDevice. Tools: t75g.py (in-guest gdb state/watch probes), diag76.py (thread bt + address resolution), slots_probe.sh, run75g.sh; README-session76.md records the firmware handler decode and the sweep.
* wall-2 session 74: the client's 11s stalls are userspace completion-slot ↵Arthur Hanson Rasmusson11 hours
| | | | | | | | | | | | | | | | | | | | | | spins; the next wall is the polled RM control 0x2080a097 Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield + clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings. Guest-side filling of the slot flags lets the client proceed immediately, but vkCreateDevice still fails -4: with the slots filled the client settles into an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation answers with the generic all-zero reply, so the poll never completes. Also captured: sema_push_raw dumps proving the GR channel-init push contains NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then zeros), and the negative result that neither channelFillGpFifo nor nvWriteGpEntry runs for this channel's kick. New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and vmap_watch2.py (vmap/kcore watchers), README-session74.md.
* wall-2 forensics tools: GR channel-init push localization (session 73)Arthur Hanson Rasmusson12 hours
Tools and findings from the live wall-2 campaign (r674/r675): the vk_frame client's GR channel-init push (va=0x4480000 len=1432) cannot be translated by the sema service, so no release is applied and vkCreateDevice fails -4. Captured: the RM window-record/memdesc layout (ring memdesc PA == the service's validated ring pa; push memdesc PA injected live and read by the service without a false-translation line), the CE-only shape of the service rescue matcher, the notrace status of nvidia/modeset kprobes (QEMU gdb hbreak works), and the ring-entry encoding from channelFillGpFifo/nvWriteGpEntry (neither runs for this channel's kick). New tools: wall2_resolver{2..5}.py, wall2_livecap{1..3}.py, RAM structure scanners (scan_r673*.py, trace_push674.py), gdb675*.gdb probes, plus a README with the ranked next steps.