1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
|
# Driver Lifecycle Field Guide
> A walking tour of what actually happens between "guest boots" and "CUDA
> initializes", with the real message and control IDs we observe from an
> unmodified 570.144 guest. Use this to interpret logs and to place new
> observations.
## 0. How to use this document
Each stage lists: what the guest does, what the device/firmware must answer,
what our emulation does today, and the log signature to look for
(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest).
The IDs are for the 570.144 generation on the Turing (TU104) profile. Other
generations will have additions and version differences; the *stages* are
stable.
## 1. PCI probe and config space
**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs,
enables Memory Space and Bus Master, programs MSI-X.
**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB
BAR1, an MSI-X capability, and a link capability the driver accepts.
**Emulation:** config space is synthesized; BAR sizes are enforced; the
*hypervisor* is the usual source of surprises (a device attached to the root
bus can have its PCIe capability rewritten to "root complex integrated
endpoint", which removes link information the driver wants — attach behind a
root port).
**Log signature:** none from us; guest-side `lspci -vvv` is the reference.
## 2. GSP boot handshake
**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the
RISC-V core; polls a status mailbox for "GSP init done".
**Device must:** complete every stage in the order the driver polls, and leave
the mailbox in the state the driver expects.
**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and
auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls
terminate.
**Log signature:**
```
gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ...
```
## 3. Static configuration
**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo`
structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS
info.
**Device must:** provide self-consistent sizes, engine inventories, memory
geometry, interrupt topology, and a VBIOS blob the driver can parse.
**Emulation:** static structures are synthesized for TU104 (FB size, BAR
layout, engine list, vGPU-friendly VF values).
**Log signature:**
```
GSPRM: GET_GSP_STATIC_INFO → ...
```
## 4. Message queues
**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue
buffer description, and a page table (the queue buffer is
`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72),
`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic.
**Device must:** consume them silently (async) and start consuming the command
queue exactly in the order the guest writes elements.
**Emulation:** parses the queue chain, builds the queue geometry, links the
status queue, and consumes elements one by one. **Element-based, not
byte-stream-based**: each message is one element and may span pages.
**Log signature:**
```
gsp-resman-vfio: libos region[i] id8=... pa=... size=...
gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=...
```
## 5. Resource-manager initialization (the control battery)
**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes.
Of the ~160 observed control IDs, 23 are special-cased and answered with real
values; the rest are answered with a generic `SUCCESS` and zeroed params (the
fallback). Representative controls:
| Control | Meaning | What matters |
|---|---|---|
| `0x20800a40`-family | internal device info table | per-index values, entry count |
| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries |
| `0x20800a5c` | interrupt kernel table | vector topology the driver maps |
| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early |
| `0x20800a01` | display static info | display count (headless values are fine) |
| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations |
| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support |
| `0x20802a0a/0b` | all-CE caps | caps table for all engines |
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here |
| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently |
**Log signature:**
```
GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...
```
## 6. Object creation
**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a
CUDA process:
```
0x0 NV01_ROOT (client root, 120-byte params)
0x80 NV01_DEVICE_0 (56 bytes)
0x2080 NV20_SUBDEVICE_0 (4 bytes)
0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space
0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group
0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076 GP100_UVM_SW (0 bytes)
0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)
```
The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault
was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack
local while the kernel side treats the class params as 16 bytes and copies the
reply params back, so the emulation must echo the request params back in the
alloc reply instead of writing a reply the caller did not ask for.
Channel parameters decode into: `gpFifoOffset` (a GPU VA),
`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and
`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with
**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`).
**Emulation:** handle management and parameter shapes are validated against
the reference structures; values are logged in a form that makes the object
tree reconstructible from `dmesg`.
**Log signature:**
```
GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...
```
## 7. UVM registration (the wall we climbed)
**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation →
page-tree init.
**Device must:** let UVM's channels initialize (each channel does a first push
and waits on a semaphore), and let page-tree initialization proceed.
**Emulation (today):** we can decode the pushes UVM generates and execute
their memsets, copies and semaphore releases in software by translating the
semaphore VA to a guest physical address (see the UVM and host-module docs).
This is a bridge, not forwarding: no work reaches the physical GPU yet.
**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate`
returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks
the async copy engine (`COPY14`). The engine-record layout and the alloc
reply-fidelity checks are the first things to inspect
([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)).
**Log signatures (guest side):**
```
nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N
```
## 8. What comes after registration (roadmap)
```
UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)
```
Each step will need the same treatment: satisfy the interface, forward the
work, relay the completions. The known-unknowns document lists what we expect
to be hard.
## 9. Quick reference: message/control IDs seen from a stock 570.144 guest
| Function | Name | Frequency per CUDA run (approx.) |
|---|---|---|
| 1 | SET_GUEST_SYSTEM_INFO | once |
| 10 | FREE | ~14 |
| 47 | UNLOADING_GUEST_DRIVER | on unload |
| 65 | GET_GSP_STATIC_INFO | once |
| 70 | UPDATE_BAR_PDE | a few |
| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each |
| 76 | GSP_RM_CONTROL | ~160 |
| 103 | GSP_RM_ALLOC | ~60 |
If a new driver version produces **new function numbers**, do not guess: add a
handler that logs the raw payload, then decode it against the decompiled
study notes for that driver version.
|
