summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
Diffstat
-rw-r--r--tools/guest-diag/wall2/README.md85
-rw-r--r--tools/guest-diag/wall2/gdb675.gdb21
-rw-r--r--tools/guest-diag/wall2/gdb675c.gdb12
-rw-r--r--tools/guest-diag/wall2/gdb675d.gdb11
-rw-r--r--tools/guest-diag/wall2/scan_r673.py82
-rw-r--r--tools/guest-diag/wall2/scan_r673b.py53
-rw-r--r--tools/guest-diag/wall2/scan_r673c.py45
-rw-r--r--tools/guest-diag/wall2/scan_r673d.py69
-rw-r--r--tools/guest-diag/wall2/session73-notes.md51
-rw-r--r--tools/guest-diag/wall2/trace_push674.py71
-rw-r--r--tools/guest-diag/wall2/wall2_livecap.py112
-rw-r--r--tools/guest-diag/wall2/wall2_livecap2.py109
-rw-r--r--tools/guest-diag/wall2/wall2_livecap3.py106
-rw-r--r--tools/guest-diag/wall2/wall2_resolver2.py274
-rw-r--r--tools/guest-diag/wall2/wall2_resolver3.py136
-rw-r--r--tools/guest-diag/wall2/wall2_resolver4.py149
-rw-r--r--tools/guest-diag/wall2/wall2_resolver5.py102
17 files changed, 1488 insertions, 0 deletions
diff --git a/tools/guest-diag/wall2/README.md b/tools/guest-diag/wall2/README.md
new file mode 100644
index 0000000..1ef9558
--- /dev/null
+++ b/tools/guest-diag/wall2/README.md
@@ -0,0 +1,85 @@
+# WALL #2 (GR channel-init release) — session 73 findings + tools
+
+## Goal
+The vk_frame Vulkan client fails `vkCreateDevice -> VKFAIL -4` because the
+kernel RM kicks its GR channel (h=0xbeee0100, ring VA 0x44a0000, push VA
+0x4480000) and waits ~11 s per attempt for a completion the sema service
+cannot apply: it cannot translate the push VA ("false translation for push
+va=0x4480000 ... rescuing"), so no release is decoded and the wait times out.
+
+## Findings (r674/r675, 8+ client runs)
+1. The push is RM-built (nvidia.ko), NOT UVM: a kprobe on
+ `uvm_hal_turing_host_semaphore_release` never fired during a client run.
+2. nvidia.ko / nvidia-modeset.ko functions are `notrace` -> `kprobe_events`
+ refuses them ("Could not probe notrace function ... [nvidia]"). UVM module
+ probes DO work. QEMU gdbstub hardware breakpoints (hbreak) work fine.
+3. The service's rescue matcher is CE-shaped only (hdr 0x20038090, hi==1,
+ lo&0xfff==slot, w5/v5 shape) -> it can never match a GR push. Entries 0/1/2
+ are consumed as "skipping stale entry ... after ~150 failed rescues";
+ gr_wedge=1.
+4. Host-side content searches of /dev/shm/vgpu-guest-ram during the stall found
+ NO push data: neither the CE release group (0x20038090) nor the host-class
+ release header (0x20050017) appears outside driver code pages.
+5. Ring entry encoding (from `channelFillGpFifo` in nvidia.ko and
+ `nvWriteGpEntry` in nvidia-modeset.ko): [0] = va & ~3, [1] =
+ ((va>>32)&0xff) | ((len<<8)&0x7ffffc00) | 0x80000000 (flag bit seen on
+ entry[1]).
+6. Neither `channelFillGpFifo` (nvidia.ko) nor `nvWriteGpEntry`
+ (nvidia-modeset.ko) is called during the client run (gdb hbreaks, 0 hits
+ for the GR channel) -> the GR channel's ring entries are written by an
+ unidentified writer (possibly the emulation/GSP path); the push content
+ was never located in guest RAM.
+7. RM window records + memdescs (host-readable, stable layout):
+ - a page holds the client channel's window records
+ {VA=0x4480000, SIZE=0x20000} (push) and {VA=0x44a0000, SIZE=0x40000}
+ (ring), each inside head/tail list nodes;
+ - memdescs `{SIZE, 0x03008000, FIRST_PA, ptr}` live in the same page:
+ the ring memdesc's FIRST_PA == the service's validated ring pa
+ (verified twice: 0x1158ce000, 0x129415000) -> memdesc PA == the PA of the
+ window's first VA page;
+ - the push memdesc `{0x20000, 0x03008000, PA}` was injected as
+ 0x4480000 -> PA (resolver4, r675: PA 0x11595c000; "injected mapping
+ va=0x4480000 ... (sysmem)" in dmesg) and the service then read the page
+ WITHOUT the "false translation" line - but decoded no release (the page
+ content does not look like the method stream; later candidates read as
+ zero).
+8. The kernel-side stall shows no GSP RPCs and no ioctls; the wait function
+ `channelWaitForFinishPayload` (polls `channelReadChannelMemdesc(channel,
+ channel->0x50)`, which reads `*(channel+0xe8)+off`) was NEVER called during
+ a client run (gdb hbreak) -> the RM's wait path for this kick is elsewhere.
+
+## Tools (this directory)
+- wall2_resolver2.py — v2: CE release-group content search + tuple arrays
+ (pa&~0xfff, strides 0x20/0x40/0x60). No candidates live.
+- wall2_resolver3.py — v3: ring memdesc -> same-page push memdesc recipe.
+- wall2_resolver4.py — v4: window-record pair + ring memdesc cross-check
+ (this one injected successfully; rdesc[0] bug fixed in v5).
+- wall2_resolver5.py — v5: continuous scan (no ring-line gate), injects every
+ 0x20000 memdesc PA found in pages holding both window records.
+- wall2_livecap{,2,3}.py — live stall-time scanners for the CE group and the
+ host release header (both negative; useful for re-checks).
+- scan_r673*.py — RAM structure analysis (ring-PA refs, record pages, memdesc
+ scans).
+- trace_push674.py — VA-reference tracer with direct-map pointer following.
+- gdb675*.gdb — QEMU gdbstub scripts (channelFillGpFifo / nvWriteGpEntry
+ probes, channel struct dumps).
+
+## Next steps (ranked)
+1. Find the GR ring-entry writer (gdb watchpoint on the ring page's CPU VA;
+ get the VA from the channel struct; find the channel struct via the ring
+ memdesc page's head/tail pointers). The writer's caller holds the push's
+ CPU mapping -> PA -> inject -> service decodes the release.
+2. Or: gdb-break at the stall on the RM's wait (sample the sleeping task's
+ kernel stack; the current candidates never fired) and read the semaphore
+ value it polls; bump it host-side via the ring-page PA (the finish payload
+ may live at ring_cpu + channel->0x50).
+3. Or: fix the service's rescue matcher to accept GR-shaped releases once the
+ push content is captured (see item 1).
+
+## Ops
+- pkill -f self-matches: the bracket trick fails when the pattern text appears
+ elsewhere in the agent's own command line; kill by explicit PID from
+ `ps -eo pid,comm,args | awk '$2=="python3" && /<script>/ {print $1}'`.
+- r674 wedged after a gdb-interrupted run; fresh cycle r675 used (no host
+ reboot). gdb674d/gdb675d sessions left attached (never kill gdbs).
+- Never reload the guest nvidia modules; never kill clients mid-flight.
diff --git a/tools/guest-diag/wall2/gdb675.gdb b/tools/guest-diag/wall2/gdb675.gdb
new file mode 100644
index 0000000..e4b73ce
--- /dev/null
+++ b/tools/guest-diag/wall2/gdb675.gdb
@@ -0,0 +1,21 @@
+set pagination off
+set confirm off
+set print elements 0
+target remote :1234
+hbreak *0xffffffffc0d0b9f0
+condition 1 *(unsigned int*)($rdi+0x110) == 0xc46f
+commands 1
+silent
+printf "GRFILL ch=%p idx=%d len=%d\n", $rdi, $esi, $edx
+printf " +0x048=%p +0x0b8=%p +0x0e8=%p\n", *(unsigned long*)($rdi+0x48), *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0xe8)
+printf " +0x060=%p +0x0c0=%p +0x0c8=%p +0x0d0=%p +0x0d8=%p\n", *(unsigned long*)($rdi+0x60), *(unsigned long*)($rdi+0xc0), *(unsigned long*)($rdi+0xc8), *(unsigned long*)($rdi+0xd0), *(unsigned long*)($rdi+0xd8)
+printf " +0x0e0=%p +0x0f0=%p +0x0f8=%p +0x108=%p +0x140=%p +0x148=%p\n", *(unsigned long*)($rdi+0xe0), *(unsigned long*)($rdi+0xf0), *(unsigned long*)($rdi+0xf8), *(unsigned long*)($rdi+0x108), *(unsigned long*)($rdi+0x140), *(unsigned long*)($rdi+0x148)
+x/8gx $rdi+0x40
+x/8gx $rdi+0x80
+x/8gx $rdi+0xc0
+x/8gx $rdi+0xe0
+x/8gx $rdi+0x100
+x/8gx $rdi+0x140
+continue
+end
+continue
diff --git a/tools/guest-diag/wall2/gdb675c.gdb b/tools/guest-diag/wall2/gdb675c.gdb
new file mode 100644
index 0000000..25b5ca0
--- /dev/null
+++ b/tools/guest-diag/wall2/gdb675c.gdb
@@ -0,0 +1,12 @@
+set pagination off
+set confirm off
+set print elements 0
+target remote :1234
+hbreak *0xffffffffc0d0b9f0
+commands 1
+silent
+printf "FILL ch=%p cls100=0x%x b8=0x%x stride48=0x%x idx=%d len=%d\n", $rdi, *(unsigned int*)($rdi+0x100), *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0x48), $esi, $edx
+printf " d0=%p d8=%p e8=%p f8=%p 60=%p\n", *(unsigned long*)($rdi+0xd0), *(unsigned long*)($rdi+0xd8), *(unsigned long*)($rdi+0xe8), *(unsigned long*)($rdi+0xf8), *(unsigned long*)($rdi+0x60)
+continue
+end
+continue
diff --git a/tools/guest-diag/wall2/gdb675d.gdb b/tools/guest-diag/wall2/gdb675d.gdb
new file mode 100644
index 0000000..f9dc460
--- /dev/null
+++ b/tools/guest-diag/wall2/gdb675d.gdb
@@ -0,0 +1,11 @@
+set pagination off
+set confirm off
+set print elements 0
+target remote :1234
+hbreak *0xffffffffc1605fd0
+commands 1
+silent
+printf "NVWRITE ch=%p idx=%u len=%u b8=0x%x f68=%p f50=0x%x fb0=0x%x fc0=0x%x f198=%p\n", $rdi, $esi, $edx, *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0x68), *(unsigned int*)($rdi+0x50), *(unsigned int*)($rdi+0xb0), *(unsigned int*)($rdi+0xc0), *(unsigned long*)($rdi+0x198)
+continue
+end
+continue
diff --git a/tools/guest-diag/wall2/scan_r673.py b/tools/guest-diag/wall2/scan_r673.py
new file mode 100644
index 0000000..71a0128
--- /dev/null
+++ b/tools/guest-diag/wall2/scan_r673.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+# Analyze the live r673 guest RAM: locate every reference to the validated ring
+# PA (0x12a783000) including flag-bit variants, and characterize the record
+# arrays around each hit. Also dump the two pages named in the handoff.
+import numpy as np, struct, sys, time
+
+RAM="/dev/shm/vgpu-guest-ram"
+RING=0x12a783000
+MASK=~0xfff & ((1<<64)-1)
+
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000+(pa-0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000+(off-0x80000000)
+
+img=open(RAM,"rb")
+import os
+fsz=os.path.getsize(RAM)
+print("ram size 0x%x" % fsz)
+
+t0=time.time()
+hits=[]
+CH=64<<20
+for pos in range(0,fsz,CH):
+ img.seek(pos); d=img.read(min(CH,fsz-pos))
+ n=len(d)//8
+ a=np.frombuffer(d[:n*8],dtype=np.uint64)
+ idx=np.nonzero((a & np.uint64(MASK))==np.uint64(RING))[0]
+ for i in idx:
+ off=pos+int(i)*8
+ val=int(a[i])
+ hits.append((off,val))
+print("hits=%d scan=%.1fs" % (len(hits), time.time()-t0))
+for off,val in hits:
+ print(" off=0x%x gpa=0x%x val=0x%x (low=0x%x)" % (off,off2gpa(off),val,val&0xfff))
+
+# characterize each hit page
+def rd(pa,n):
+ off=gpa2off(pa)
+ if off is None or off+n>fsz: return None
+ img.seek(off); return img.read(n)
+
+def plausible_pa(v): return 0x100000000 <= (v & MASK) < 0x180000000
+def ptrish(v):
+ return (0xffff800000000000 <= v < 0xffffffffffffffff)
+
+for off,val in hits:
+ gpa=off2gpa(off)
+ page=gpa & MASK
+ rec_off=gpa-page
+ d=rd(page,0x1000)
+ print("=== hit gpa=0x%x page=0x%x rec_off=0x%x ===" % (gpa,page,rec_off))
+ for stride in (0x20,0x40,0x60,0x30,0x50,0x28,0x48):
+ # record start such that the hit qword falls inside the record
+ for rel in range(0,stride,8):
+ start=rec_off-rel
+ if start<0: continue
+ # walk records; count plausible PA fields anywhere in each record
+ n_pa=0; n_ptr=0; n_1000=0
+ for o in range(start,0x1000-stride,stride):
+ rec=d[o:o+stride]
+ qs=struct.unpack_from("<%dQ"%(stride//8),rec,0)
+ if any(plausible_pa(q) for q in qs): n_pa+=1
+ if any(ptrish(q) for q in qs): n_ptr+=1
+ if 0x1000 in qs: n_1000+=1
+ if n_pa>=8:
+ print(" stride=0x%x start=0x%x: PA-records=%d ptr-records=%d 0x1000=%d" % (stride,start,n_pa,n_ptr,n_1000))
+ # dump the first 0x100 bytes of the page and around the hit
+ print(" page head: %s" % d[:0x60].hex())
+ s=max(0,rec_off-0x40); e=min(0x1000,rec_off+0x80)
+ print(" around hit [0x%x..0x%x]: %s" % (s,e,d[s:e].hex()))
+
+# dump the two named pages
+for pa in (0x101499000,0x127cf2000):
+ d=rd(pa,0x1000)
+ print("=== page 0x%x ===" % pa)
+ if d is None: print(" unreadable"); continue
+ for o in range(0,0x120,0x20):
+ qs=struct.unpack_from("<4Q",d,o)
+ print(" +0x%03x: %016x %016x %016x %016x" % ((o,)+qs))
diff --git a/tools/guest-diag/wall2/scan_r673b.py b/tools/guest-diag/wall2/scan_r673b.py
new file mode 100644
index 0000000..11613b1
--- /dev/null
+++ b/tools/guest-diag/wall2/scan_r673b.py
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# r673: dump the full record page 0x101499000 and search RAM for the ring/push
+# VA structures to identify which record is the 16-page push (VA 0x4500000).
+import numpy as np, struct, os, time
+
+RAM="/dev/shm/vgpu-guest-ram"
+MASK=~0xfff & ((1<<64)-1)
+img=open(RAM,"rb")
+fsz=os.path.getsize(RAM)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000+(pa-0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000+(off-0x80000000)
+def rd(pa,n):
+ off=gpa2off(pa)
+ if off is None or off+n>fsz: return None
+ img.seek(off); return img.read(n)
+
+print("== full record page 0x101499000 ==")
+d=rd(0x101499000,0x1000)
+nrec=0
+for o in range(0,0x1000,32):
+ p,f,pa,sz=struct.unpack_from("<QQQQ",d,o)
+ if p==0 and f==0 and pa==0 and sz==0: continue
+ nrec+=1
+ print(" rec[%3d] vmap=%016x flags=%016x pa=%012x size=0x%x" % (o//32,p,f,pa,sz))
+print("nonempty records:",nrec)
+
+print()
+print("== search RAM for qword 0x4520000 (ring VA) ==")
+t0=time.time(); CH=64<<20
+for pat_name,pat in (("0x4520000",0x4520000),("0x4500000",0x4500000)):
+ print("-- pattern %s --" % pat_name)
+ hits=[]
+ for pos in range(0,fsz,CH):
+ img.seek(pos); dd=img.read(min(CH,fsz-pos))
+ n=len(dd)//8
+ a=np.frombuffer(dd[:n*8],dtype=np.uint64)
+ idx=np.nonzero(a==np.uint64(pat))[0]
+ for i in idx:
+ hits.append(pos+int(i)*8)
+ print(" hits=%d (%.1fs)" % (len(hits),time.time()-t0))
+ for off in hits[:40]:
+ gpa=off2gpa(off); page=gpa&MASK
+ # show 32 bytes before/after
+ dd=rd(page,0x1000)
+ if not dd: continue
+ ro=gpa-page
+ s=max(0,ro-32); e=min(0x1000,ro+64)
+ print(" gpa=0x%x page=0x%x off=0x%x ctx=%s" % (gpa,page,ro,dd[s:e].hex()))
+ if len(hits)>40: print(" ... %d more" % (len(hits)-40))
diff --git a/tools/guest-diag/wall2/scan_r673c.py b/tools/guest-diag/wall2/scan_r673c.py
new file mode 100644
index 0000000..db69b5d
--- /dev/null
+++ b/tools/guest-diag/wall2/scan_r673c.py
@@ -0,0 +1,45 @@
+#!/usr/bin/env python3
+# r673: find the push page. Patterns: ring entry raw qwords, the sem VA, and
+# method-stream signatures. Dump contexts.
+import numpy as np, struct, os, time
+
+RAM="/dev/shm/vgpu-guest-ram"
+img=open(RAM,"rb"); fsz=os.path.getsize(RAM)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000+(pa-0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000+(off-0x80000000)
+def rd(pa,n):
+ off=gpa2off(pa)
+ if off is None or off+n>fsz: return None
+ img.seek(off); return img.read(n)
+
+pats=[
+ ("entry0raw", 0x0005980004500000),
+ ("entry1raw", 0x0000040004500598),
+ ("entry2raw", 0x0003d0000450059c),
+ ("sem_va", 0x4500598),
+ ("sem_va2", 0x450059c),
+ ("m20038090", 0x20038090),
+ ("push_va", 0x4500000),
+]
+CH=64<<20
+for name,pat in pats:
+ hits=[]
+ t0=time.time()
+ for pos in range(0,fsz,CH):
+ img.seek(pos); d=img.read(min(CH,fsz-pos))
+ n=len(d)//8
+ a=np.frombuffer(d[:n*8],dtype=np.uint64)
+ idx=np.nonzero(a==np.uint64(pat))[0]
+ for i in idx: hits.append(pos+int(i)*8)
+ print("== %s pat=0x%x hits=%d (%.1fs)" % (name,pat,len(hits),time.time()-t0))
+ for off in hits[:12]:
+ gpa=off2gpa(off); page=gpa&~0xfff
+ d=rd(page,0x1000)
+ ro=gpa-page
+ s=max(0,ro-24); e=min(0x1000,ro+40)
+ print(" gpa=0x%x page=0x%x off=0x%x ctx=%s" % (gpa,page,ro,d[s:e].hex() if d else "?"))
+ if len(hits)>12: print(" ... %d more" % (len(hits)-12))
diff --git a/tools/guest-diag/wall2/scan_r673d.py b/tools/guest-diag/wall2/scan_r673d.py
new file mode 100644
index 0000000..1fa89ca
--- /dev/null
+++ b/tools/guest-diag/wall2/scan_r673d.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+# r673 deep-dive: all ring-PA refs with low bits, ring page dump, RM record
+# pages, and a RAM-wide scan for 0xa3bf20 VA-table records mentioning 0x4500000.
+import numpy as np, struct, os, time
+
+RAM="/dev/shm/vgpu-guest-ram"
+RING=0x12a783000; TAG=0xa3bf20
+img=open(RAM,"rb"); fsz=os.path.getsize(RAM)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000+(pa-0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000+(off-0x80000000)
+def rd(pa,n):
+ off=gpa2off(pa)
+ if off is None or off+n>fsz: return None
+ img.seek(off); return img.read(n)
+
+MASK=~0xfff & ((1<<64)-1)
+print("== all qwords (v & ~0xfff)==ring_pa ==")
+CH=64<<20
+for pos in range(0,fsz,CH):
+ img.seek(pos); d=img.read(min(CH,fsz-pos))
+ n=len(d)//8
+ a=np.frombuffer(d[:n*8],dtype=np.uint64)
+ idx=np.nonzero((a & np.uint64(MASK))==np.uint64(RING))[0]
+ for i in idx:
+ off=pos+int(i)*8
+ gpa=off2gpa(off); page=gpa&~0xfff
+ dd=rd(page,0x1000); ro=gpa-page
+ s=max(0,ro-32); e=min(0x1000,ro+48)
+ print(" gpa=0x%x low=0x%x ctx=%s" % (gpa,int(a[i])&0xfff,dd[s:e].hex() if dd else "?"))
+
+print()
+print("== ring page 0x%x first 0x120 ==" % RING)
+d=rd(RING,0x120)
+for o in range(0,0x120,0x20):
+ print(" +0x%03x: %s" % (o,d[o:o+0x20].hex()))
+
+for pa in (0x1013a6000,0x104acb000,0x114760000,0x1202cd000,0x13e2c1000,0x13e62e000):
+ print()
+ print("== page 0x%x first 0x140 ==" % pa)
+ d=rd(pa,0x140)
+ for o in range(0,0x140,0x20):
+ q=struct.unpack_from("<4Q",d,o)
+ print(" +0x%03x: %016x %016x %016x %016x" % ((o,)+q))
+
+print()
+print("== RAM-wide: 32-byte records {va, cpu, 0xa3bf20, pages} with va==0x4500000 ==")
+t0=time.time()
+found=0
+for pos in range(0,fsz,CH):
+ img.seek(pos); d=img.read(min(CH,fsz-pos))
+ n=len(d)//8
+ a=np.frombuffer(d[:n*8],dtype=np.uint64)
+ idx=np.nonzero(a==np.uint64(TAG))[0]
+ for i in idx:
+ off=pos+int(i)*8
+ gpa=off2gpa(off); page=gpa&~0xfff
+ dd=rd(page,0x1000)
+ ro=gpa-page
+ if ro>=16 and ro+8<=0x1000:
+ va,cpu=struct.unpack_from("<QQ",dd,ro-16)
+ pgs=struct.unpack_from("<Q",dd,ro+8)[0]
+ if va==0x4500000 or (0x4400000<=va<=0x4600000):
+ print(" rec gpa=0x%x va=0x%x cpu=0x%x pages=0x%x" % (gpa,va,cpu,pgs))
+ found+=1
+print("total", found, "(%.1fs)" % (time.time()-t0))
diff --git a/tools/guest-diag/wall2/session73-notes.md b/tools/guest-diag/wall2/session73-notes.md
new file mode 100644
index 0000000..d2079d3
--- /dev/null
+++ b/tools/guest-diag/wall2/session73-notes.md
@@ -0,0 +1,51 @@
+# SESSION 73 notes (wall #2 live validation, r674 + r675)
+
+## r674 cycle results (8 client runs, all VKFAIL -4)
+- Client vk_frame reproducibly fails at vkCreateDevice (-4) as before.
+- wall2_resolver2 (content-first CE group 0x20038090 + tuple arrays): no candidates found
+ during the stall; tuple arrays contain the ring PA but not the push.
+- livecap/livecap2/livecap3 (0x20038090 shape / 0x20050017 host-release shape): NO data
+ hits in guest RAM during the stall; only driver CODE pages match.
+- Conclusion: the GR channel-init push content is NOT findable in /dev/shm/vgpu-guest-ram
+ by these patterns, even live. (The ring page content IS there and matches the service's
+ decode.)
+
+## Structural findings
+- The push is RM-built (nvidia.ko), NOT UVM: kprobe on
+ uvm_hal_turing_host_semaphore_release never fired during the client run.
+- nvidia.ko functions are `notrace` -> kprobe_events refuses them
+ ("Could not probe notrace function ... [nvidia]"). UVM module probes DO work.
+ QEMU gdbstub hardware breakpoints work fine (hbreak *addr).
+- Service rescue matcher is CE-shaped only (hdr 0x20038090, hi==1, lo&0xfff==slot,
+ w5/v5 shape) -> can never match a GR push. Entries 0/1/2 are consumed as "stale"
+ after ~150 failed rescues each; gr_wedge=1.
+- channelFillGpFifo(channel, idx, len) disasm: rdi=channel; +0x178=gpu;
+ +0xe8=cached ring CPU VA; +0xb8=push GPU VA base; +0x48=stride;
+ +0x110=class (0xc5b5 seen for a CE channel; GR = 0xc46f).
+ Entry encoding: [0]=va&~3 (u32), [1]=(va>>32 &0xff)|((len<<8)&0x7ffffc00).
+- r674 ring PAs per run: 0x12a783000(?) no: run1 0x1050b6000, run2 0x10297d000,
+ later 0x102317000 (ring line format: "h=0x... gpfifo=0x44a0000 ent=32768 userd=0xffe00 as=2 put=3 get=0 ring rc=0 pa=0x... ap=1").
+- Push VA = 0x4480000, ring VA = 0x44a0000 (delta 0x20000); entry0 len=1432,
+ entry1 {va=0x4480598,len=4}, entry2 {va=0x448059c,len=976}.
+- The pushbuffer (UVM) = 0x20000 (32 pages) aligned; UVM_PUSHBUFFER_SIZE=0x20000
+ (UVM_MAX_PUSH_SIZE=0x400). UVM_RM_MEM_TYPE_SYS with CPU map.
+- RM memdesc records {size=0x20000, flags=0x03008000, first_PA, ptr} exist in RAM
+ (11 found post-run) but their PAs did not contain the release group.
+- RM window record {0x4480000, 0x20000} + head/tail list nodes exist in RAM.
+- The r673 VA-table record {0x10, 0x4500000, 0x786fd9598ab0, 0xa3bf20} has a
+ "cpu" field that may be a user VA of the client (0x64..-0x78.. range).
+
+## Ops
+- r674 wedged after the gdb-interrupted vk31 run (SSH + guest agent dead, vCPUs idle).
+ Fresh cycle r675 started. gdb674d still attached to the dead QEMU (never kill gdbs).
+- /dev/shm 53%.
+
+## r675 plan
+1. gdb675.gdb: hbreak channelFillGpFifo if class(+0x110)==0xc46f; on hit dump the
+ channel struct pointer fields (see gdb675.gdb) -> identify the push's CPU mapping
+ (direct-map or vmalloc) by content.
+2. Compute PA (direct-map: VA - 0xffff8916c0000000 style base) and read the push
+ content from the RAM file; find the release group.
+3. Inject 0x4480000 -> PA via sema_map_add BEFORE/DURING the stall; watch the service
+ apply the release and vkCreateDevice pass.
+4. Durability: integrate into pin_gr_stream.py demand path or a service hook.
diff --git a/tools/guest-diag/wall2/trace_push674.py b/tools/guest-diag/wall2/trace_push674.py
new file mode 100644
index 0000000..ced4168
--- /dev/null
+++ b/tools/guest-diag/wall2/trace_push674.py
@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+# r674: find every reference to the push VA 0x4480000 and follow RM structures
+# (direct-map pointers: PA = ptr - page_offset_base) toward a page list.
+import numpy as np, struct, os, time, sys
+
+RAM="/dev/shm/vgpu-guest-ram"
+VA=0x4480000
+BASE=0xffff8916c0000000 # r674 page_offset_base (from dmesg rebuild #15)
+img=open(RAM,"rb"); fsz=os.path.getsize(RAM)
+def gpa2off(pa):
+ if pa<0x80000000: return pa
+ if pa<0x180000000: return 0x80000000+(pa-0x100000000)
+ return None
+def off2gpa(off):
+ return off if off<0x80000000 else 0x100000000+(off-0x80000000)
+def rd(pa,n):
+ off=gpa2off(pa)
+ if off is None or off+n>fsz: return None
+ img.seek(off); return img.read(n)
+def ptr2pa(v):
+ if BASE<=v<BASE+0x180000000:
+ return v-BASE
+ return None
+
+CH=64<<20
+hits=[]
+t0=time.time()
+for pos in range(0,fsz,CH):
+ img.seek(pos); d=img.read(min(CH,fsz-pos))
+ n=len(d)//8
+ a=np.frombuffer(d[:n*8],dtype=np.uint64)
+ idx=np.nonzero(a==np.uint64(VA))[0]
+ for i in idx: hits.append(off2gpa(pos+int(i)*8))
+print("VA hits:", len(hits), "(%.1fs)" % (time.time()-t0))
+
+def tag(v):
+ bs=struct.pack("<Q",v)
+ return bs.decode() if all(32<=c<127 for c in bs) else ""
+
+for gpa in hits:
+ page=gpa&~0xfff
+ d=rd(page,0x1000)
+ if not d: continue
+ ro=gpa-page
+ print("== hit gpa=0x%x page=0x%x off=0x%x ==" % (gpa,page,ro))
+ # print 0x60 before, 0x80 after
+ s=max(0,ro-0x60); e=min(0x1000,ro+0x80)
+ for o in range(s,e,0x10):
+ a,b=struct.unpack_from("<QQ",d,o)
+ print(" +0x%03x: %016x %016x %s %s" % (o,a,b,tag(a),tag(b)))
+ # follow direct-map pointers in this page
+ for o in range(0,0x1000-8,8):
+ v=struct.unpack_from("<Q",d,o)[0]
+ pa=ptr2pa(v)
+ if pa is None: continue
+ dd=rd(pa&~0xfff,0x1000)
+ if not dd: continue
+ # search that page for PA runs (>=8 page-aligned RAM PAs ascending/descending)
+ runs=[]; cur=[]
+ for oo in range(0,0xff8,8):
+ vv=struct.unpack_from("<Q",dd,oo)[0]
+ if 0x100000000<=vv<0x180000000 and (vv&0xfff)==0:
+ cur.append((oo,vv))
+ else:
+ if len(cur)>=8: runs.append(cur)
+ cur=[]
+ if len(cur)>=8: runs.append(cur)
+ if runs:
+ print(" ptr +0x%03x -> pa=0x%x has %d PA-run(s):" % (o,pa&~0xfff,len(runs)))
+ for r in runs[:4]:
+ print(" run at +0x%x: %d pages first=0x%x last=0x%x" % (r[0][0],len(r),r[0][1],r[-1][1]))
diff --git a/tools/guest-diag/wall2/wall2_livecap.py b/tools/guest-diag/wall2/wall2_livecap.py
new file mode 100644
index 0000000..af68c03
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_livecap.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+# wall2_livecap.py - live capture of the GR channel-init push page.
+#
+# The release group in the push is {0x20038090, sem_va_hi, sem_va_lo, payload,
+# continuation}: sem_va = (hi<<32)|lo, and for the channel-init entry the sem
+# sits at push_va + entry_len (entry0: len=1432=0x598, sem at +0x598), with the
+# group itself at push_va + entry_len - 24 = +0x580.
+#
+# This watcher scans guest RAM for 0x20038090 groups during the client's stall,
+# classifies them, and injects push_va -> page_pa via sema_map_add when the
+# group's sem VA and offset identify the push page. Prints full evidence.
+import subprocess, struct, time, sys, re, os
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_livecap.log"
+MASK = ~0xfff & ((1 << 64) - 1)
+
+ENTRY_LEN = int(os.environ.get("W2_LEN", "1432"))
+WATCH_S = int(os.environ.get("W2_WATCH", "120"))
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+img = open(RAM, "rb", buffering=0)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None: return None
+ img.seek(off); return img.read(n)
+
+def cont_ok(v):
+ op = (v >> 29) & 7
+ cnt = (v >> 16) & 0xfff
+ return (op in (1, 3, 5) and cnt != 0) or op in (4, 7)
+
+def scan():
+ img.seek(0, 2); fsz = img.tell()
+ pat = b"\x90\x80\x03\x20"
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ img.seek(pos); d = img.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0: break
+ st = i + 1
+ if i + 24 > len(d): continue
+ g = d[i:i+24]
+ hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g)
+ gpa = off2gpa(pos + i)
+ page = gpa & MASK
+ off = gpa - page
+ out.append((page, off, hi, lo, pay, w5, v5, g))
+ pos += CH
+ return out
+
+def main():
+ log("=== livecap start len=%d watch=%ds ===" % (ENTRY_LEN, WATCH_S))
+ injected = {}
+ deadline = time.time() + WATCH_S
+ while time.time() < deadline:
+ hits = scan()
+ cands = []
+ for (page, off, hi, lo, pay, w5, v5, g) in hits:
+ sem = (hi << 32) | lo
+ # classify
+ pool = (hi == 1 and lo < 0x10000 and 0 < pay < (1 << 24)
+ and (w5 >> 29) == 1 and ((w5 >> 13) & 7) == 4
+ and ((w5 & 0x1fff) << 2) == 0x300 and (v5 & 3) == 0)
+ if pool:
+ continue # CE pool release family, not the push
+ if not (0x4000000 <= sem < 0x100000000):
+ continue
+ if not (0 < pay < (1 << 24)):
+ continue
+ if not cont_ok(w5):
+ continue
+ push_va = sem - ENTRY_LEN
+ cands.append((page, off, sem, pay, push_va, w5, v5, g))
+ if cands:
+ # prefer the canonical group offset 0x580 / sem-consistent push VA
+ cands.sort(key=lambda c: (c[1] != (ENTRY_LEN - 24), -c[3]))
+ page, off, sem, pay, push_va, w5, v5, g = cands[0]
+ d = rd(page, 0x100)
+ log("livecap: %d cand(s); page=0x%x off=0x%x sem=0x%x pay=%u push_va=0x%x w5=0x%08x v5=0x%08x"
+ % (len(cands), page, off, sem, pay, push_va, w5, v5))
+ log(" page head: %s" % d[:0x40].hex())
+ if (push_va & 0xfff) == 0 and push_va not in injected:
+ try:
+ with open(PARAM, "w") as f:
+ f.write("0x%x,0x%x,0x%x,1\n" % (push_va, page, 0x1000))
+ log(" INJECT 0x%x -> 0x%x (off=0x%x)" % (push_va, page, off))
+ injected[push_va] = page
+ except OSError as e:
+ log(" inject FAIL", e)
+ time.sleep(1)
+ log("=== livecap done: %d injection(s) ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/guest-diag/wall2/wall2_livecap2.py b/tools/guest-diag/wall2/wall2_livecap2.py
new file mode 100644
index 0000000..a33b25b
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_livecap2.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+# wall2_livecap2.py - capture the GR channel-init push page by its release
+# group {0x20038090, hi, lo, payload, continuation} at page offset 0x598.
+# The push VA comes from the service's own entry[0] decode in dmesg.
+import subprocess, struct, time, sys, re, os
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_livecap2.log"
+MASK = ~0xfff & ((1 << 64) - 1)
+WATCH_S = int(os.environ.get("W2_WATCH", "180"))
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+img = open(RAM, "rb", buffering=0)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None: return None
+ img.seek(off); return img.read(n)
+
+def prefix_ok(d):
+ off = 0
+ while off + 4 <= len(d):
+ hdr = struct.unpack_from("<I", d, off)[0]
+ if hdr: break
+ off += 4
+ if off + 4 > len(d): return False
+ op = (hdr >> 29) & 7
+ if op in (4, 7): return True
+ if op not in (1, 3, 5): return False
+ return ((hdr >> 16) & 0xfff) != 0
+
+def cont_ok(v):
+ op = (v >> 29) & 7
+ cnt = (v >> 16) & 0xfff
+ return (op in (1, 3, 5) and cnt != 0) or op in (4, 7)
+
+def scan():
+ img.seek(0, 2); fsz = img.tell()
+ pat = b"\x90\x80\x03\x20"
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ img.seek(pos); d = img.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0: break
+ st = i + 1
+ if i + 24 > len(d): continue
+ g = d[i:i+24]
+ hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g)
+ gpa = off2gpa(pos + i)
+ page = gpa & MASK
+ off = gpa - page
+ out.append((page, off, hi, lo, pay, w5, v5))
+ pos += CH
+ return out
+
+def main():
+ log("=== livecap2 start watch=%ds ===" % WATCH_S)
+ injected = set()
+ deadline = time.time() + WATCH_S
+ seen_pages = {}
+ while time.time() < deadline:
+ hits = scan()
+ for (page, off, hi, lo, pay, w5, v5) in hits:
+ if not (0 < pay < (1 << 24)):
+ continue
+ if not cont_ok(w5):
+ continue
+ d = rd(page, 0x1000)
+ if not d: continue
+ pok = prefix_ok(d[:0x100])
+ key = (page, off)
+ if key in seen_pages:
+ continue
+ seen_pages[key] = 1
+ log("hit page=0x%x off=0x%x sem=0x%x pay=%u w5=0x%08x v5=0x%08x prefix_ok=%s"
+ % (page, off, (hi << 32) | lo, pay, w5, v5, pok))
+ log(" head: %s" % d[:0x30].hex())
+ log(" @0x590: %s" % d[0x590:0x5c0].hex())
+ if pok and off in (0x598, 0x590, 0x580, 0x588):
+ va = 0x4480000
+ if va not in injected:
+ try:
+ with open(PARAM, "w") as f:
+ f.write("0x%x,0x%x,0x%x,1\n" % (va, page, 0x1000))
+ log(" INJECT 0x%x -> 0x%x" % (va, page))
+ injected.add(va)
+ except OSError as e:
+ log(" inject FAIL", e)
+ time.sleep(0.5)
+ log("=== livecap2 done: %d injections ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/guest-diag/wall2/wall2_livecap3.py b/tools/guest-diag/wall2/wall2_livecap3.py
new file mode 100644
index 0000000..db28bc2
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_livecap3.py
@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+# wall2_livecap3.py - capture the UVM GR channel-init push page via the host
+# (C46F) semaphore release group emitted by uvm_hal_turing_host_semaphore_release:
+# {0x20050017 (INC_METHOD count=5 addr=0x5c), sem_lo, sem_hi, payload, 0, exec}
+# The group sits at push_va + entry_len - 24 = push_va + 0x580 (entry0 len 1432),
+# with the header at +0x598 (entry1's 4-byte push). The push VA comes from the
+# service's own entry[0] decode in dmesg.
+import subprocess, struct, time, sys, re, os
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_livecap3.log"
+MASK = ~0xfff & ((1 << 64) - 1)
+WATCH_S = int(os.environ.get("W2_WATCH", "240"))
+PUSH_VA_DEFAULT = 0x4480000
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+img = open(RAM, "rb", buffering=0)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None: return None
+ img.seek(off); return img.read(n)
+
+def dmesg():
+ return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout
+
+def push_va_now():
+ # newest entry[0] line for the Vulkan client's channel
+ va = None
+ for ln in dmesg().splitlines():
+ m = re.search(r"entry\[0\] h=0x([0-9a-f]+) eng=0 raw=\S+ va=0x([0-9a-f]+) len=(\d+)", ln)
+ if m:
+ va = int(m.group(2), 16)
+ return va
+
+def scan():
+ img.seek(0, 2); fsz = img.tell()
+ pat = struct.pack("<I", 0x20050017)
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ img.seek(pos); d = img.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0: break
+ st = i + 1
+ if i + 24 > len(d): continue
+ lo, hi, pay, w4, w5 = struct.unpack_from("<5I", d, i + 4)
+ # data-word sanity: payload_hi==0, RELEASE op, sane payload
+ if w4 != 0: continue
+ if (w5 & 0x7) != 1: continue
+ if not (0 < pay < (1 << 24)): continue
+ sem = ((hi & 0xffffffff) << 32) | ((lo & 0x3fffffff) << 2)
+ gpa = off2gpa(pos + i)
+ page = gpa & MASK
+ off = gpa - page
+ out.append((page, off, sem, pay, w5))
+ pos += CH
+ return out
+
+def main():
+ log("=== livecap3 start watch=%ds ===" % WATCH_S)
+ injected = {}
+ seen = set()
+ deadline = time.time() + WATCH_S
+ while time.time() < deadline:
+ hits = scan()
+ if hits:
+ for (page, off, sem, pay, w5) in hits:
+ key = (page, off)
+ if key in seen: continue
+ seen.add(key)
+ d = rd(page, 0x1000)
+ head = d[:0x20].hex() if d else "?"
+ log("hit page=0x%x off=0x%x sem=0x%x pay=%u exec=0x%08x head=%s"
+ % (page, off, sem, pay, w5, head))
+ log(" @0x580: %s" % (d[0x580:0x5b0].hex() if d else "?"))
+ if off in (0x598, 0x580, 0x590):
+ pv = push_va_now() or PUSH_VA_DEFAULT
+ if pv not in injected:
+ try:
+ with open(PARAM, "w") as f:
+ f.write("0x%x,0x%x,0x%x,1\n" % (pv, page, 0x1000))
+ log(" INJECT 0x%x -> 0x%x" % (pv, page))
+ injected[pv] = page
+ except OSError as e:
+ log(" inject FAIL", e)
+ time.sleep(0.5)
+ log("=== livecap3 done: %d injections ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/guest-diag/wall2/wall2_resolver2.py b/tools/guest-diag/wall2/wall2_resolver2.py
new file mode 100644
index 0000000..785e5e2
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_resolver2.py
@@ -0,0 +1,274 @@
+#!/usr/bin/env python3
+# wall2_resolver2.py - HOST-SIDE wall#2 resolver v2 (r674+).
+#
+# Why: the vk_frame client's channel-init push (va=0x4500000 len=1432) is
+# kernel-built and kernel-only; the sema service cannot translate it
+# ("false translation for push ... rescuing"), the client times out (~11 s)
+# per attempt, retries, then VKFAIL -4. Guest RAM is host-readable, so the
+# push page can be found and injected via sema_map_add.
+#
+# v2 recipe (per the r672/r673 structure analysis):
+# 1. wait for the channel ring line in dmesg (validated RING_PA + handle)
+# 2. scan guest RAM for the push page:
+# a) CONTENT: the 24-byte release group the service's own rescue matches
+# (hdr=0x20038090, hi=1, payload sane, w5/v5 shape). The channel-init
+# entry (va=0x4500000 len=1432) puts that group at page offset 0x580.
+# b) TUPLES: pages of 32-byte records {vmap_ptr; flags; PA; size} (both
+# observed layouts) matched on (PA & ~0xfff) == RING_PA with strides
+# 0x20/0x40/0x60; entries below the ring entry in VA order map
+# 0x4500000.. (push VA = ring VA 0x4520000 - 0x20000 = 32 pages).
+# 3. inject (va -> pa, len) through /sys/module/vgsp/parameters/sema_map_add
+# 4. keep scanning/re-injecting for the watch window (retries allocate new
+# pages) and log everything.
+#
+# Run this FIRST (foreground), then launch the client in the guest.
+import subprocess, struct, time, sys, re, os, signal
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_resolver2.log"
+RING_VA = 0x4520000
+PUSH_VA = 0x4500000
+DELTA = (RING_VA - PUSH_VA) // 0x1000 # 32 pages
+REL_OFF = 0x580 # group offset in the push page
+MASK = ~0xfff & ((1 << 64) - 1)
+
+WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420"))
+WATCH_S = int(os.environ.get("W2_WATCH", "150"))
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+def dmesg():
+ return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout
+
+def inject(va, pa, ln=0x1000, ap=1):
+ try:
+ with open(PARAM, "w") as f:
+ f.write("0x%x,0x%x,0x%x,%d\n" % (va, pa, ln, ap))
+ log("INJECT va=0x%x -> pa=0x%x len=0x%x" % (va, pa, ln))
+ return True
+ except OSError as e:
+ log("inject FAIL", e)
+ return False
+
+def prefix_ok(d):
+ off = 0
+ while off + 4 <= len(d):
+ hdr = struct.unpack_from("<I", d, off)[0]
+ if hdr:
+ break
+ off += 4
+ if off + 4 > len(d):
+ return False
+ op = (hdr >> 29) & 7
+ if op in (4, 7):
+ return True
+ if op not in (1, 3, 5):
+ return False
+ return ((hdr >> 16) & 0xfff) != 0
+
+def rel_shape(g):
+ hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g)
+ return (hdr == 0x20038090 and hi == 1 and 0 < pay < (1 << 24)
+ and (w5 >> 29) == 1 and ((w5 >> 13) & 7) == 4
+ and ((w5 & 0x1fff) << 2) == 0x300 and (v5 & 3) == 0), lo & 0xfff, pay
+
+img = None
+def open_ram():
+ global img
+ if img is None:
+ img = open(RAM, "rb", buffering=0)
+ return img
+
+def gpa2off(pa):
+ if pa < 0x80000000:
+ return pa
+ if pa < 0x180000000:
+ return 0x80000000 + (pa - 0x100000000)
+ return None
+
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None:
+ return None
+ f = open_ram()
+ try:
+ f.seek(off)
+ return f.read(n)
+ except OSError:
+ return None
+
+def find_content_candidates():
+ """Pages containing the release group; returns [(page_pa, off, slot, pay)]."""
+ f = open_ram()
+ f.seek(0, 2)
+ fsz = f.tell()
+ pat = b"\x90\x80\x03\x20"
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ f.seek(pos)
+ d = f.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0:
+ break
+ st = i + 1
+ if i + 24 > len(d):
+ continue
+ ok, slot, pay = rel_shape(d[i:i + 24])
+ if not ok:
+ continue
+ gpa = off2gpa(pos + i)
+ page = gpa & MASK
+ off = gpa - page
+ out.append((page, off, slot, pay))
+ pos += CH
+ return out
+
+def find_tuple_arrays(ring_pa):
+ """Find record pages containing the ring PA; returns list of
+ (page_gpa, stride, ring_index, entries[list of pa])."""
+ f = open_ram()
+ f.seek(0, 2)
+ fsz = f.tell()
+ pat = struct.pack("<Q", ring_pa)
+ arrays = []
+ CH = 64 << 20
+ pos = 0
+ hits = []
+ while pos < fsz:
+ f.seek(pos)
+ d = f.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0:
+ break
+ hits.append(off2gpa(pos + i))
+ st = i + 1
+ pos += CH
+ seen = set()
+ for gpa in hits:
+ page = gpa & MASK
+ if page in seen:
+ continue
+ seen.add(page)
+ d = rd(page, 0x1000)
+ if not d:
+ continue
+ rec_off = gpa - page
+ for stride in (0x20, 0x40, 0x60):
+ for rel in range(0, min(stride, rec_off + 1), 8):
+ start = rec_off - rel
+ ents = []
+ for o in range(start, 0x1000 - stride + 1, stride):
+ # PA field anywhere in the record: try +0x10 first, then any qword
+ vals = struct.unpack_from("<%dQ" % (stride // 8), d, o)
+ pa = None
+ for cand_off in (0x10, 0x8, 0x18, 0x0, 0x20, 0x28):
+ if cand_off + 8 <= stride:
+ v = vals[cand_off // 8]
+ if (v & MASK) == ring_pa:
+ pa = v & MASK
+ break
+ if pa is None:
+ continue
+ ents.append((o, pa))
+ if len(ents) >= 4:
+ idx = [e[1] for e in ents].index(ring_pa)
+ arrays.append((page, stride, start, idx, [e[1] for e in ents]))
+ break
+ if arrays and arrays[-1][0] == page:
+ break
+ return arrays
+
+def main():
+ log("=== wall2_resolver2 start (wait_ring=%ss watch=%ss) ===" % (WAIT_RING_S, WATCH_S))
+ ring_pa = None
+ ring_h = None
+ deadline = time.time() + WAIT_RING_S
+ while time.time() < deadline:
+ for ln in dmesg().splitlines():
+ m = re.search(r"h=(0x[0-9a-f]+) .*ring rc=0 pa=0x([0-9a-f]+)", ln)
+ if m:
+ ring_h = m.group(1)
+ ring_pa = int(m.group(2), 16)
+ break
+ if ring_pa:
+ break
+ time.sleep(1)
+ if not ring_pa:
+ log("no ring line seen in %ss; exiting" % WAIT_RING_S)
+ return 1
+ log("RING %s PA=0x%x" % (ring_h, ring_pa))
+
+ injected = set()
+ watch_deadline = time.time() + WATCH_S
+ last_pass = 0
+ while time.time() < watch_deadline:
+ # ---- content pass
+ try:
+ cands = find_content_candidates()
+ except Exception as e:
+ log("content scan error:", e)
+ cands = []
+ # prefer the canonical group offset 0x580, then highest payload
+ cands.sort(key=lambda c: (c[1] != REL_OFF, -c[3]))
+ if cands:
+ page, off, slot, pay = cands[0]
+ d = rd(page, 0x1000)
+ pok = bool(d) and prefix_ok(d[:0x100])
+ log("content: %d candidate(s); best page=0x%x off=0x%x slot=0x%x pay=%u prefix_ok=%s"
+ % (len(cands), page, off, slot, pay, pok))
+ key = ("C", PUSH_VA, page)
+ if key not in injected and pok:
+ if inject(PUSH_VA, page, 0x1000):
+ injected.add(key)
+ # also cover the following page (entry2 tail) if non-contiguous unknown; skip
+ else:
+ # ---- tuple fallback (only every ~10 s to save time)
+ if time.time() - last_pass > 10:
+ last_pass = time.time()
+ try:
+ arrs = find_tuple_arrays(ring_pa)
+ except Exception as e:
+ log("tuple scan error:", e)
+ arrs = []
+ if arrs:
+ page, stride, start, idx, ents = arrs[0]
+ log("tuple: array page=0x%x stride=0x%x ring_index=%d entries=%d"
+ % (page, stride, idx, len(ents)))
+ for k in range(idx - DELTA, idx):
+ if k < 0 or k >= len(ents):
+ continue
+ va = PUSH_VA + (k - (idx - DELTA)) * 0x1000
+ pa = ents[k]
+ key = ("T", va, pa)
+ if key in injected:
+ continue
+ d = rd(pa, 0x40)
+ if d and any(d):
+ inject(va, pa, 0x1000)
+ injected.add(key)
+ else:
+ log("tuple: no array with ring PA found")
+ # stop early when the service confirms our mapping
+ if len(injected) and "injected mapping va=0x%x" % PUSH_VA in dmesg():
+ log("service confirmed the injected mapping; done")
+ break
+ time.sleep(1)
+ log("=== done: %d injection(s) ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/guest-diag/wall2/wall2_resolver3.py b/tools/guest-diag/wall2/wall2_resolver3.py
new file mode 100644
index 0000000..4b16686
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_resolver3.py
@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+# wall2_resolver3.py - HOST-SIDE wall#2 resolver v3: the RM window-memdesc recipe.
+#
+# Proven structure (r674/r675): the RM keeps per-window records
+# {..., VA, SIZE, ...} (head/tail list nodes)
+# and per-window memdescs in the same page:
+# {SIZE, 0x03008000, FIRST_PA, ptr}
+# where FIRST_PA is the PA of the window's FIRST VA page. Verified: the ring
+# window {VA=0x44a0000, SIZE=0x40000} -> memdesc PA == the service's validated
+# ring pa (0x1158ce000 r675). Therefore the push window
+# {VA=0x4480000, SIZE=0x20000} -> memdesc PA == PA of VA 0x4480000 (the push
+# page the sema service cannot translate).
+#
+# Recipe: wait for the ring line -> find the ring window memdesc
+# {0x40000, 0x03008000, RING_PA} -> in the same page find the push window
+# memdesc {0x20000, 0x03008000, PUSH_PA} -> inject 0x4480000 -> PUSH_PA.
+import subprocess, struct, time, sys, re, os
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_resolver3.log"
+PUSH_VA = 0x4480000
+WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420"))
+WATCH_S = int(os.environ.get("W2_WATCH", "150"))
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+img = open(RAM, "rb", buffering=0)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None: return None
+ img.seek(off); return img.read(n)
+
+def dmesg():
+ return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout
+
+def inject(va, pa, ln=0x1000, ap=1):
+ try:
+ with open(PARAM, "w") as f:
+ f.write("0x%x,0x%x,0x%x,%d\n" % (va, pa, ln, ap))
+ log("INJECT va=0x%x -> pa=0x%x len=0x%x" % (va, pa, ln))
+ return True
+ except OSError as e:
+ log("inject FAIL", e)
+ return False
+
+def find_memdescs(size):
+ """Find {size, 0x03008000, PA} patterns; returns [(page, off, pa)]."""
+ img.seek(0, 2); fsz = img.tell()
+ pat = struct.pack("<QQ", size, 0x03008000)
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ img.seek(pos); d = img.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0: break
+ st = i + 1
+ if i + 24 > len(d): continue
+ pa = struct.unpack_from("<Q", d, i + 16)[0]
+ gpa = off2gpa(pos + i)
+ if 0x100000000 <= pa < 0x180000000:
+ out.append((gpa & ~0xfff, gpa - (gpa & ~0xfff), pa))
+ pos += CH
+ return out
+
+def main():
+ log("=== resolver3 start ===")
+ # count pre-existing ring lines; wait for a NEW one (fresh client kick)
+ def ring_lines():
+ out = []
+ for ln in dmesg().splitlines():
+ m = re.search(r"ring rc=0 pa=0x([0-9a-f]+) ap=", ln)
+ if m:
+ out.append(int(m.group(1), 16))
+ return out
+ base = ring_lines()
+ log("pre-existing ring lines: %d" % len(base))
+ ring_pa = None
+ deadline = time.time() + WAIT_RING_S
+ while time.time() < deadline:
+ cur = ring_lines()
+ if len(cur) > len(base):
+ ring_pa = cur[-1]
+ break
+ time.sleep(1)
+ if not ring_pa:
+ log("no NEW ring line; exit"); return 1
+ log("NEW RING PA = 0x%x" % ring_pa)
+
+ injected = set()
+ watch = time.time() + WATCH_S
+ while time.time() < watch:
+ # ring window memdesc {0x40000, 0x03008000, ring_pa}
+ rd_hits = [x for x in find_memdescs(0x40000) if x[2] == ring_pa]
+ if not rd_hits:
+ time.sleep(0.5); continue
+ for (page, off, _pa) in rd_hits:
+ d = rd(page, 0x1000)
+ if not d: continue
+ # find sibling push memdesc in the same page
+ for o in range(0, 0x1000 - 24, 8):
+ sz, fl = struct.unpack_from("<QQ", d, o)
+ if sz == 0x20000 and fl == 0x03008000:
+ pa = struct.unpack_from("<Q", d, o + 16)[0]
+ if not (0x100000000 <= pa < 0x180000000):
+ continue
+ key = (page, pa)
+ if key in injected: continue
+ log("memdesc page=0x%x ring_desc@0x%x push_desc@0x%x push_pa=0x%x"
+ % (page, off, o, pa))
+ # content sanity: read the push page
+ pd = rd(pa, 0x40)
+ log(" push page head: %s" % (pd.hex() if pd else "?"))
+ if inject(PUSH_VA, pa, 0x1000):
+ injected.add(key)
+ if injected and ("injected mapping va=0x%x" % PUSH_VA) in dmesg():
+ log("service confirmed; keep watching")
+ time.sleep(0.5)
+ log("=== resolver3 done: %d injection(s) ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/guest-diag/wall2/wall2_resolver4.py b/tools/guest-diag/wall2/wall2_resolver4.py
new file mode 100644
index 0000000..140aaa7
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_resolver4.py
@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+# wall2_resolver4.py - HOST-SIDE wall#2 resolver v4 (final recipe).
+#
+# Association proven on r675: the RM keeps, in ONE page, the client's channel
+# window records and their memdescs:
+# push window record {VA=0x4480000, SIZE=0x20000}
+# ring window record {VA=0x44a0000, SIZE=0x40000}
+# push memdesc {0x20000, 0x03008000, FIRST_PA, ptr}
+# ring memdesc {0x40000, 0x03008000, FIRST_PA, ptr}
+# where FIRST_PA is the PA of the window's FIRST VA page. Verified: the ring
+# memdesc's PA == the service's validated ring pa. So the push memdesc's PA ==
+# the PA of VA 0x4480000 (the push page the service cannot translate).
+#
+# Recipe: wait for a NEW ring line -> find the page containing the ring window
+# record {0x44a0000, 0x40000} -> require the sibling push window record
+# {0x4480000, 0x20000} and the memdescs in the same page -> cross-check the
+# ring memdesc PA == ring_pa -> inject 0x4480000 -> push memdesc PA (sudo).
+import subprocess, struct, time, sys, re, os
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_resolver4.log"
+PUSH_VA = 0x4480000
+RING_VA = 0x44a0000
+WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420"))
+WATCH_S = int(os.environ.get("W2_WATCH", "180"))
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+img = open(RAM, "rb", buffering=0)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None: return None
+ img.seek(off); return img.read(n)
+
+def dmesg():
+ return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout
+
+def inject(va, pa, ln=0x1000, ap=1):
+ cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM)
+ r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True)
+ if r.returncode == 0:
+ log("INJECT va=0x%x -> pa=0x%x len=0x%x OK" % (va, pa, ln))
+ return True
+ log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip()))
+ return False
+
+def find_qpair(a, b):
+ """Find qword pairs {a,b}; returns [(page, off)]."""
+ img.seek(0, 2); fsz = img.tell()
+ pat = struct.pack("<QQ", a, b)
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ img.seek(pos); d = img.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0: break
+ st = i + 1
+ gpa = off2gpa(pos + i)
+ out.append((gpa & ~0xfff, gpa - (gpa & ~0xfff)))
+ pos += CH
+ return out
+
+def scan_page_descs(page):
+ d = rd(page, 0x1000)
+ if not d: return []
+ out = []
+ for o in range(0, 0x1000 - 24, 8):
+ sz, fl = struct.unpack_from("<QQ", d, o)
+ if fl != 0x03008000: continue
+ if sz not in (0x20000, 0x40000): continue
+ pa = struct.unpack_from("<Q", d, o + 16)[0]
+ if 0x100000000 <= pa < 0x180000000:
+ out.append((sz, o, pa))
+ return out
+
+def main():
+ log("=== resolver4 start ===")
+ def ring_lines():
+ return re.findall(r"ring rc=0 pa=0x([0-9a-f]+) ap=", dmesg())
+ base = ring_lines()
+ log("pre-existing ring lines: %d" % len(base))
+ ring_pa = None
+ deadline = time.time() + WAIT_RING_S
+ while time.time() < deadline:
+ cur = ring_lines()
+ if len(cur) > len(base):
+ ring_pa = int(cur[-1], 16)
+ break
+ time.sleep(1)
+ if not ring_pa:
+ log("no NEW ring line; exit"); return 1
+ log("NEW RING PA = 0x%x" % ring_pa)
+
+ injected = set()
+ LEARNED = [0x101e39000, 0x102a62000] # RM window-record pages (stable across runs)
+ def try_page(page):
+ descs = scan_page_descs(page)
+ rdesc = [x for x in descs if x[0] == 0x40000]
+ pdesc = [x for x in descs if x[0] == 0x20000]
+ if not rdesc or not pdesc:
+ return False
+ if rdesc[0][2] != ring_pa:
+ return False
+ ok = False
+ for (_sz, o, pa) in pdesc:
+ key = (page, o)
+ if key in injected: continue
+ pd = rd(pa, 0x40)
+ log("page=0x%x ring_desc@0x%x(pa=0x%x OK) push_desc@0x%x pa=0x%x head=%s"
+ % (page, rdesc[0][1], rdesc[0][2], o, pa,
+ pd.hex() if pd else "?"))
+ if inject(PUSH_VA, pa, 0x1000):
+ injected.add(key); ok = True
+ return ok
+ log("learned pages: %s" % [hex(p) for p in LEARNED])
+ watch = time.time() + WATCH_S
+ last_full = 0
+ while time.time() < watch:
+ hit = False
+ for page in LEARNED:
+ if try_page(page):
+ hit = True
+ if not hit and time.time() - last_full > 5:
+ last_full = time.time()
+ for (page, _off) in find_qpair(RING_VA, 0x40000):
+ if try_page(page):
+ if page not in LEARNED:
+ LEARNED.append(page)
+ hit = True
+ time.sleep(0.25)
+ log("=== resolver4 done: %d injection(s) ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/guest-diag/wall2/wall2_resolver5.py b/tools/guest-diag/wall2/wall2_resolver5.py
new file mode 100644
index 0000000..faec902
--- /dev/null
+++ b/tools/guest-diag/wall2/wall2_resolver5.py
@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+# wall2_resolver5.py - final: continuous scan, all-candidate injection.
+#
+# Structure (r675, verified): an RM page holds the client channel's window
+# records and memdescs:
+# {0x4480000, 0x20000} push window record
+# {0x44a0000, 0x40000} ring window record
+# {0x20000, 0x03008000, PUSH_PA} push window memdesc
+# {0x40000, 0x03008000, RING_PA} ring window memdesc (PA == validated ring pa)
+# The resolver scans CONTINUOUSLY (no ring-line gate), finds pages holding the
+# pair of window records, and injects 0x4480000 -> every 0x20000 memdesc PA in
+# the same page (last write wins in the service; content check arbitrates).
+import subprocess, struct, time, sys, os
+
+RAM = "/dev/shm/vgpu-guest-ram"
+PARAM = "/sys/module/vgsp/parameters/sema_map_add"
+LOG = "/tmp/opencode/wall2_resolver5.log"
+PUSH_VA = 0x4480000
+RING_VA = 0x44a0000
+WATCH_S = int(os.environ.get("W2_WATCH", "600"))
+
+logf = open(LOG, "a", buffering=1)
+def log(*a):
+ s = " ".join(str(x) for x in a)
+ print(s, flush=True)
+ logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s))
+
+img = open(RAM, "rb", buffering=0)
+def gpa2off(pa):
+ if pa < 0x80000000: return pa
+ if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000)
+ return None
+def off2gpa(off):
+ return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000)
+def rd(pa, n):
+ off = gpa2off(pa)
+ if off is None: return None
+ img.seek(off); return img.read(n)
+
+def inject(va, pa, ln=0x1000, ap=1):
+ cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM)
+ r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True)
+ if r.returncode == 0:
+ log("INJECT va=0x%x -> pa=0x%x" % (va, pa))
+ return True
+ log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip()))
+ return False
+
+def find_qpair(a, b):
+ img.seek(0, 2); fsz = img.tell()
+ pat = struct.pack("<QQ", a, b)
+ out = []
+ CH = 64 << 20
+ pos = 0
+ while pos < fsz:
+ img.seek(pos); d = img.read(min(CH, fsz - pos))
+ st = 0
+ while True:
+ i = d.find(pat, st)
+ if i < 0: break
+ st = i + 1
+ gpa = off2gpa(pos + i)
+ out.append(gpa & ~0xfff)
+ pos += CH
+ return out
+
+def main():
+ log("=== resolver5 start (watch=%ds) ===" % WATCH_S)
+ injected = set()
+ deadline = time.time() + WATCH_S
+ while time.time() < deadline:
+ try:
+ push_pages = set(find_qpair(PUSH_VA, 0x20000))
+ ring_pages = set(find_qpair(RING_VA, 0x40000))
+ except Exception as e:
+ log("scan error:", e); time.sleep(2); continue
+ for page in sorted(push_pages & ring_pages):
+ d = rd(page, 0x1000)
+ if not d: continue
+ # collect 0x20000 descs in this page
+ cands = []
+ for o in range(0, 0x1000 - 24, 8):
+ sz, fl = struct.unpack_from("<QQ", d, o)
+ if sz == 0x20000 and fl == 0x03008000:
+ pa = struct.unpack_from("<Q", d, o + 16)[0]
+ if 0x100000000 <= pa < 0x180000000:
+ cands.append((o, pa))
+ for (o, pa) in cands:
+ key = (page, o)
+ if key in injected: continue
+ pd = rd(pa, 0x40)
+ nz = sum(1 for b in pd if b) if pd else -1
+ log("page=0x%x desc@0x%x pa=0x%x live_nz=%d head=%s"
+ % (page, o, pa, nz, pd.hex()[:64] if pd else "?"))
+ if inject(PUSH_VA, pa):
+ injected.add(key)
+ time.sleep(1)
+ log("=== resolver5 done: %d injection(s) ===" % len(injected))
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(main())
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-25 13:29:41 +0000