1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
|
# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion
## The two-part unblock
1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the
nvkms/Vulkan path allocates its completion event as
NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now
registers it alongside 0x79. Live proof (r683, sema_dbg=1):
GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011
hParent=0xcaf00001 hObject=0xcaf00003
event registered hEvent=0xcaf00003 hSrc=0x0
notifyIndex=1006632972 (off 8, slot 13)
params=1100e0c1000000007e0000000c00003c0000000000000000
EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0
(notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc =
NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records
(`guest_read == our_write` in the rmstate dump), but the guest's
event dispatch does NOT write the event's notification word — the
residual delivery gap.
2. **The completion word found and driven (the actual unblock)**. The
main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads
a u32 readiness word: `should-wait = (word != 1)`; the word lives at
`+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event
notification buffer; holder chain sub -> obj2 -> holder, resolved
live by t79.py). The host can write it directly in the guest RAM
file:
PA = pagemap PFN << 12 (+0x80); file offset = PA - 0x80000000
(above-4G RAM; below-4G would be PA itself per the cycle's
guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000)
Writing 1 flipped the wait and **vkCreateDevice returned success**
(`VKDEVICE ok family=0`) — reproduced three times:
* r683: manual write after a t79 probe;
* r684: nudger (all 16 KB mapping words) + slotfill restart;
* r684d: probe -> single-word write (the reliable recipe).
## The reliable recipe (per run)
1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional.
2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the
slots re-arm, so re-run slotfill if the client is stuck in the
sched_yield spin before the word-wait).
3. Wait for the main thread in `do_poll` (the word-wait).
4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA;
host write `01 00 00 00` at `PA-0x80000000`.
5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to
image/memory/pipeline setup; its next wait is NOT the same word
(the word stays 1) — next session: probe the follow-on state
(attach fails while the client spins in the post-device phase;
use the /proc-only checks + a longer observation window).
## Cautions learned
* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the
-4 give-up path (an RM error flag word gets written): restrict nudges
to the single probed word.
* Attaching gdb to the client while it is in the tight post-device spin
can hang the attach (leave a `timeout`-wrapped gdb, then kill that
gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a
dead tracer otherwise wedges it).
* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in
python, not zsh.
## Tools
* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA)
and prints the client's nvidiactl rw-s mapping sizes.
* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB
nvidiactl mapping (pagemap; run in the guest as root).
* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any
zero word (multi-page variant; prefer the single-word recipe).
## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6)
* The post-device spin is the **GL module bring-up**: the main thread's
A8-layer retry loop (eglcore 0xa85d24) works on a work item whose
embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6
submit layer (0xd624e0) ORs **0x200000** into its pending word (the
same engine-pending bit SESSION 75/76 watched on E+0x150), then the
A8 loop yields via the B284B0 spin helper (sched_yield) waiting for
the GL ucode init to complete on the GR engine.
* **The push windows ARE now readable** (first time): the resolver6
recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} ->
same-page 0x20000 memdesc PA -> `sema_map_add`) injected
`0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d
68000000` = the 2-method semaphore-like push) and `0x4ca0000 ->
0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5,
`0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d
70/68`). The service now decodes the streams (`sema-push ... tail=0dc0
012000000070...`) where SESSION 73-74 saw "unread".
CAUTION: the resolver injects every candidate; a zero-page candidate
injected last replaces the good mapping for that VA (re-inject the
good page after the scan).
* **Remaining gate**: the service still reports `have_rel=0` for every
GR push (the CE-shaped release matcher) and — decisively — the
GR-forward machinery is NOT engaging: `drain held=4 starved=149
forwarded=0 ... gr_nocap=4`. The GL ucode init work is never
forwarded/executed, so the completion (and the 0x200000 bit clear)
never arrives. Next: the GR forward/capture path for these channels
(operand capture + release decode for the GR shape), then the client
proceeds to VKFRAME.
## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe
* The GR/GL init releases decode only with the low-aperture acceptance
(`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms,
commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group
`200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0
pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ...
02000001`); `have_rel=1`, `applied` rising, un-translated releases
queue in svc_pending (417 observed).
* Corrected pair-specific push injections for r686:
`0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc),
`0x4ca0000 -> 0x131649000` (page 0x122f20000), plus
`0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window).
CAUTION: resolver7 injects each page's descs for BOTH pairs (union
loop) - fix it to keep per-pair pages or inject manually from the
per-pair scan. The window-record layout is {tag, VA, SIZE, 0,
"tail", ...} stride 0x40 (search the VA u64 with size at +8 and
"tail" at +0x10).
* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no
window records in the known record pages; the service cannot resolve
them (targeted rm_mem miss) so the releases stay pending and the
NvOgl wait spins. Find their PAs (client mapping pages / RM
VA-tracker / live poll-watch), inject, and the pending releases
should apply.
## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying
* **The record scan fix**: the RM window record is 0x40 bytes
{u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is
stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT
"tail" (the hex display `7461696c` reads as "tail" only as a number).
Searching for b'tail' finds nothing; `wall2_recscan.py` searches for
the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at
-0x20/-0x18/-0x10. It found **all 75 records** in one pass.
* **The GR init semaphore windows** (r686):
`va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content:
+0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 /
rel=0x4b17e30 pay=1 payloads) and
`va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the
release applied there: +0xb10 became 2). Injections:
`echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and
`echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`.
* **State after the injections**: the low-VA releases decode
(`have_rel=1`) and apply to the semaphore pages; `pending` hovers at
~418 (the client re-kicks continuously); the NvOgl wait still spins
(the A8 loop compares per-entry counter tables r14+0x5e8 vs the
wanted set - the completion is not just the semaphore value). Next:
the A8 counter tables' source (what fills r14+0x5e8 / the "wanted"
table - likely the GR channel's own progress counters) or the GR
forward/ucode execution (`forwarded=0 gr_nocap=7` still).
|
