1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
|
# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion
## The two-part unblock
1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the
nvkms/Vulkan path allocates its completion event as
NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now
registers it alongside 0x79. Live proof (r683, sema_dbg=1):
GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011
hParent=0xcaf00001 hObject=0xcaf00003
event registered hEvent=0xcaf00003 hSrc=0x0
notifyIndex=1006632972 (off 8, slot 13)
params=1100e0c1000000007e0000000c00003c0000000000000000
EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0
(notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc =
NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records
(`guest_read == our_write` in the rmstate dump), but the guest's
event dispatch does NOT write the event's notification word — the
residual delivery gap.
2. **The completion word found and driven (the actual unblock)**. The
main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads
a u32 readiness word: `should-wait = (word != 1)`; the word lives at
`+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event
notification buffer; holder chain sub -> obj2 -> holder, resolved
live by t79.py). The host can write it directly in the guest RAM
file:
PA = pagemap PFN << 12 (+0x80); file offset = PA - 0x80000000
(above-4G RAM; below-4G would be PA itself per the cycle's
guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000)
Writing 1 flipped the wait and **vkCreateDevice returned success**
(`VKDEVICE ok family=0`) — reproduced three times:
* r683: manual write after a t79 probe;
* r684: nudger (all 16 KB mapping words) + slotfill restart;
* r684d: probe -> single-word write (the reliable recipe).
## The reliable recipe (per run)
1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional.
2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the
slots re-arm, so re-run slotfill if the client is stuck in the
sched_yield spin before the word-wait).
3. Wait for the main thread in `do_poll` (the word-wait).
4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA;
host write `01 00 00 00` at `PA-0x80000000`.
5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to
image/memory/pipeline setup; its next wait is NOT the same word
(the word stays 1) — next session: probe the follow-on state
(attach fails while the client spins in the post-device phase;
use the /proc-only checks + a longer observation window).
## Cautions learned
* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the
-4 give-up path (an RM error flag word gets written): restrict nudges
to the single probed word.
* Attaching gdb to the client while it is in the tight post-device spin
can hang the attach (leave a `timeout`-wrapped gdb, then kill that
gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a
dead tracer otherwise wedges it).
* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in
python, not zsh.
## Tools
* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA)
and prints the client's nvidiactl rw-s mapping sizes.
* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB
nvidiactl mapping (pagemap; run in the guest as root).
* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any
zero word (multi-page variant; prefer the single-word recipe).
## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6)
* The post-device spin is the **GL module bring-up**: the main thread's
A8-layer retry loop (eglcore 0xa85d24) works on a work item whose
embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6
submit layer (0xd624e0) ORs **0x200000** into its pending word (the
same engine-pending bit SESSION 75/76 watched on E+0x150), then the
A8 loop yields via the B284B0 spin helper (sched_yield) waiting for
the GL ucode init to complete on the GR engine.
* **The push windows ARE now readable** (first time): the resolver6
recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} ->
same-page 0x20000 memdesc PA -> `sema_map_add`) injected
`0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d
68000000` = the 2-method semaphore-like push) and `0x4ca0000 ->
0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5,
`0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d
70/68`). The service now decodes the streams (`sema-push ... tail=0dc0
012000000070...`) where SESSION 73-74 saw "unread".
CAUTION: the resolver injects every candidate; a zero-page candidate
injected last replaces the good mapping for that VA (re-inject the
good page after the scan).
* **Remaining gate**: the service still reports `have_rel=0` for every
GR push (the CE-shaped release matcher) and — decisively — the
GR-forward machinery is NOT engaging: `drain held=4 starved=149
forwarded=0 ... gr_nocap=4`. The GL ucode init work is never
forwarded/executed, so the completion (and the 0x200000 bit clear)
never arrives. Next: the GR forward/capture path for these channels
(operand capture + release decode for the GR shape), then the client
proceeds to VKFRAME.
## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe
* The GR/GL init releases decode only with the low-aperture acceptance
(`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms,
commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group
`200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0
pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ...
02000001`); `have_rel=1`, `applied` rising, un-translated releases
queue in svc_pending (417 observed).
* Corrected pair-specific push injections for r686:
`0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc),
`0x4ca0000 -> 0x131649000` (page 0x122f20000), plus
`0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window).
CAUTION: resolver7 injects each page's descs for BOTH pairs (union
loop) - fix it to keep per-pair pages or inject manually from the
per-pair scan. The window-record layout is {tag, VA, SIZE, 0,
"tail", ...} stride 0x40 (search the VA u64 with size at +8 and
"tail" at +0x10).
* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no
window records in the known record pages; the service cannot resolve
them (targeted rm_mem miss) so the releases stay pending and the
NvOgl wait spins. Find their PAs (client mapping pages / RM
VA-tracker / live poll-watch), inject, and the pending releases
should apply.
## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying
* **The record scan fix**: the RM window record is 0x40 bytes
{u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is
stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT
"tail" (the hex display `7461696c` reads as "tail" only as a number).
Searching for b'tail' finds nothing; `wall2_recscan.py` searches for
the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at
-0x20/-0x18/-0x10. It found **all 75 records** in one pass.
* **The GR init semaphore windows** (r686):
`va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content:
+0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 /
rel=0x4b17e30 pay=1 payloads) and
`va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the
release applied there: +0xb10 became 2). Injections:
`echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and
`echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`.
* **State after the injections**: the low-VA releases decode
(`have_rel=1`) and apply to the semaphore pages; `pending` hovers at
~418 (the client re-kicks continuously); the NvOgl wait still spins
(the A8 loop compares per-entry counter tables r14+0x5e8 vs the
wanted set - the completion is not just the semaphore value). Next:
the A8 counter tables' source (what fills r14+0x5e8 / the "wanted"
table - likely the GR channel's own progress counters) or the GR
forward/ucode execution (`forwarded=0 gr_nocap=7` still).
## ADDENDUM (r686, session 81) — the counter-autoreg gap analysed: the counter page 0x20400f000 is unresolvable; the manual `sema_counter` registration is the bypass
* `svc_counter_autoreg` (per-pass for eng==0 channels, n_gr=20) needs a
translation for the rank slot VA `0x20400fec0 + (19-r)*0x10`
(r = gr_index = rank by ascending handle among the guest's GR
channels): either an ipin pin (`svc_ipin_lookup`, 9275 entries live
from the guest pin stream) or `svc_counter_derive` (the
vaspaces/discovered/targeted walk chain, `sema_counter_derive=1`
default). Both miss for the counter page -> no "counter auto" log,
`pub_calls=0` -> the A8 loop's per-entry counter tables never fill and
the NvOgl wait spins.
* The client does NOT map the counter pool: `/proc/<pid>/pagemap` for
0x20400f000/0x20400ffd0/0x20400ffc0/0x20400ffb0 is all-zero (not
present) and no VMA covers it - it is a UVM kernel-side object, so the
pin stream cannot resolve it through the client's page tables. The
stream's log shows `sem=0` (zero semaphore resolutions) even though
the want-list (`/tmp/guest_want_vas.txt`, host-written) contains 44
`20400f` entries; the stream's "channel-anchored expansion" for the
UVM tracking sems (guest_pin_stream.py ~line 950-1010) is the
machinery that must find them (it is written for the CUDA second-client
stall - "the only live uvmt event is the spinning sem").
* Bypass available now: the `sema_counter` parameter registers a channel
slot manually - `echo <h_object>,<pa> > .../parameters/sema_counter`.
Once the counter page's PA is known (via the guest kcore/UVM walk),
this wires the publish path without a kernel rebuild.
* The rank slots for the GL client's channels (handles 0xbeec0100,
0xbeec0104, 0xbeee0100 -> ranks 0,1,2): 0x20400ffd0 / 0x20400ffc0 /
0x20400ffb0.
## ADDENDUM (r686, session 82) — the counter-page PA hunt: pagemap route confirmed dead, vidmem-window hypothesis tested and not supported
* The client's own pagemap does NOT contain the counter pool: the
semfix_uvmt tool's resolver (`guest_pmap_win.py <pid> 0x20400f000
0x204010000 64`) returns NOTHING, as does the UVM pool range
(`0x121000000 0x121100000`). The tool's `pin_counter_page` has
therefore never succeeded for these clients (the STALL WATCHDOG's
"re-pin counter" attempts across sessions produced no "counter:
pinned" line).
* VIDMEM-window hypothesis tested: the counter VA 0x20400f000 is in the
emulated vidmem window (>= SVC_VIDMEM_VA_BASE 0x200080000), so its
window offset is 0x3F87000 (0x20400f000 - 0x200080000). The shadow
there is reachable and all-zero (`sema_vidmem_dump 0x3f87000,64`);
poking the rank slots (+0xfd0/+0xfc0/+0xfb0 = ranks 0/1/2) with 6 and
then 0x40 via `poke_vidmem` changed the shadow (verified by dump) but
did NOT move the client's wait. The vidmem route is not (or not only)
the completion path; the A8 loop's tables have another source.
* gdb-attach hazard (again): attaches to the client in its post-device
spin hang with no output; two gdb chains had to be killed by PID
(never a pattern) and the client SIGCONT'd. Probe only while the
client is in do_poll.
## ADDENDUM (r687, session 83) — THE A8 COUNTER SOURCE DECODED: a client readback window the host can write (first live read-back confirmed)
* The A8 loop calls `a82f30(obj)` per entry; a82f30 calls the object's
method pointer at ***(obj+0x19b8)** = eglcore+0xb250f0, which returns
`(hi<<32)|count`. The count comes from **b1bf00**:
- fast path (obj+0x224 != 0): `*(u32*)(obj+0x1078) - 1`;
- slow path (observed: obj+0x224 == 0): **b27c90(obj, 0x800000bb/0xbc)**
which reads `*(u32*)(*(*(obj+0x1480)+0x58) + idx*16)` - a **client
READBACK WINDOW** (idx = 0xbb..0xbb+n, n = *(u32*)(obj+0x22efc));
the loop takes the MIN of those reads.
* Live (r687, object handle 0xbeec0100): the window = VA
0x7827e6c12000 (a /dev/nvidiactl mapping), all zeros; obj+0x224=0
(slow path active), obj+0x1078=7 (fast-path field unused).
* **The host CAN write it**: resolving the window's PA via the client's
pagemap and writing 0x100 into the window (offsets 0xbb0..0x1ff0, via
the RAM file) made the client read it - **obj+0x1070 = 0x100** (the
method stores the returned count there) and the A8 "actual" table
entry became `0xffffffff00000100`. FIRST LIVE PROOF that the client's
completion counter can be driven from the host.
* The wait still spins: the A8 loop's first scan (compare
a85e1d: actual vs want, want observed 0) passes with our value, but
the loop continues into a second scan (compare a85ec1, driven by
b29dc0) and the `*(rsp+0x38)` branch (a85e60). NEXT: decode those
(the second scan's actual source and the rsp+0x38 flag), and whether
the other objects (entries 0-2 of the A8 table) need the same window
writes.
* The other objects' windows were not observed (only 0xbeec0100 called
a82f30 in the sampled window); the nvidia0-side window (obj+0x1478)
is NOT present in the pagemap (VM_IO; host cannot write it via the
RAM file).
|
