1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
|
# WALL #2 (GR channel-init release) — session 73 findings + tools
## Goal
The vk_frame Vulkan client fails `vkCreateDevice -> VKFAIL -4` because the
kernel RM kicks its GR channel (h=0xbeee0100, ring VA 0x44a0000, push VA
0x4480000) and waits ~11 s per attempt for a completion the sema service
cannot apply: it cannot translate the push VA ("false translation for push
va=0x4480000 ... rescuing"), so no release is decoded and the wait times out.
## Findings (r674/r675, 8+ client runs)
1. The push is RM-built (nvidia.ko), NOT UVM: a kprobe on
`uvm_hal_turing_host_semaphore_release` never fired during a client run.
2. nvidia.ko / nvidia-modeset.ko functions are `notrace` -> `kprobe_events`
refuses them ("Could not probe notrace function ... [nvidia]"). UVM module
probes DO work. QEMU gdbstub hardware breakpoints (hbreak) work fine.
3. The service's rescue matcher is CE-shaped only (hdr 0x20038090, hi==1,
lo&0xfff==slot, w5/v5 shape) -> it can never match a GR push. Entries 0/1/2
are consumed as "skipping stale entry ... after ~150 failed rescues";
gr_wedge=1.
4. Host-side content searches of /dev/shm/vgpu-guest-ram during the stall found
NO push data: neither the CE release group (0x20038090) nor the host-class
release header (0x20050017) appears outside driver code pages.
5. Ring entry encoding (from `channelFillGpFifo` in nvidia.ko and
`nvWriteGpEntry` in nvidia-modeset.ko): [0] = va & ~3, [1] =
((va>>32)&0xff) | ((len<<8)&0x7ffffc00) | 0x80000000 (flag bit seen on
entry[1]).
6. Neither `channelFillGpFifo` (nvidia.ko) nor `nvWriteGpEntry`
(nvidia-modeset.ko) is called during the client run (gdb hbreaks, 0 hits
for the GR channel) -> the GR channel's ring entries are written by an
unidentified writer (possibly the emulation/GSP path); the push content
was never located in guest RAM.
7. RM window records + memdescs (host-readable, stable layout):
- a page holds the client channel's window records
{VA=0x4480000, SIZE=0x20000} (push) and {VA=0x44a0000, SIZE=0x40000}
(ring), each inside head/tail list nodes;
- memdescs `{SIZE, 0x03008000, FIRST_PA, ptr}` live in the same page:
the ring memdesc's FIRST_PA == the service's validated ring pa
(verified twice: 0x1158ce000, 0x129415000) -> memdesc PA == the PA of the
window's first VA page;
- the push memdesc `{0x20000, 0x03008000, PA}` was injected as
0x4480000 -> PA (resolver4, r675: PA 0x11595c000; "injected mapping
va=0x4480000 ... (sysmem)" in dmesg) and the service then read the page
WITHOUT the "false translation" line - but decoded no release (the page
content does not look like the method stream; later candidates read as
zero).
8. The kernel-side stall shows no GSP RPCs and no ioctls; the wait function
`channelWaitForFinishPayload` (polls `channelReadChannelMemdesc(channel,
channel->0x50)`, which reads `*(channel+0xe8)+off`) was NEVER called during
a client run (gdb hbreak) -> the RM's wait path for this kick is elsewhere.
## Tools (this directory)
- wall2_resolver2.py — v2: CE release-group content search + tuple arrays
(pa&~0xfff, strides 0x20/0x40/0x60). No candidates live.
- wall2_resolver3.py — v3: ring memdesc -> same-page push memdesc recipe.
- wall2_resolver4.py — v4: window-record pair + ring memdesc cross-check
(this one injected successfully; rdesc[0] bug fixed in v5).
- wall2_resolver5.py — v5: continuous scan (no ring-line gate), injects every
0x20000 memdesc PA found in pages holding both window records.
- wall2_livecap{,2,3}.py — live stall-time scanners for the CE group and the
host release header (both negative; useful for re-checks).
- scan_r673*.py — RAM structure analysis (ring-PA refs, record pages, memdesc
scans).
- trace_push674.py — VA-reference tracer with direct-map pointer following.
- gdb675*.gdb — QEMU gdbstub scripts (channelFillGpFifo / nvWriteGpEntry
probes, channel struct dumps).
## Next steps (ranked)
1. Find the GR ring-entry writer (gdb watchpoint on the ring page's CPU VA;
get the VA from the channel struct; find the channel struct via the ring
memdesc page's head/tail pointers). The writer's caller holds the push's
CPU mapping -> PA -> inject -> service decodes the release.
2. Or: gdb-break at the stall on the RM's wait (sample the sleeping task's
kernel stack; the current candidates never fired) and read the semaphore
value it polls; bump it host-side via the ring-page PA (the finish payload
may live at ring_cpu + channel->0x50).
3. Or: fix the service's rescue matcher to accept GR-shaped releases once the
push content is captured (see item 1).
## Ops
- pkill -f self-matches: the bracket trick fails when the pattern text appears
elsewhere in the agent's own command line; kill by explicit PID from
`ps -eo pid,comm,args | awk '$2=="python3" && /<script>/ {print $1}'`.
- r674 wedged after a gdb-interrupted run; fresh cycle r675 used (no host
reboot). gdb674d/gdb675d sessions left attached (never kill gdbs).
- Never reload the guest nvidia modules; never kill clients mid-flight.
|
