1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
|
# 03 — What the CUDA userspace driver asks of the kernel and the GPU
This document is the emulation contract: everything `libcuda` needs from
below itself. It is the checklist Open-IOV's host-side emulation has to
satisfy for unmodified CUDA to run, and the map used when a guest call fails.
Sources: live ioctl traces from the guest (ptrace tracer), host-side GSP-RM
emulation logs, and decompilation of the RM backend in `libcuda` 570.144. The
ioctl numbers below were observed on the wire; the object classes and control
IDs are quoted from the trace and from public CUDA/RM headers.
---
## 1. Device nodes and the ioctl vocabulary
| Node | Used for |
|---|---|
| `/dev/nvidiactl` | process-wide control: version negotiation, client creation, GPU attach, OS events, exported-object file descriptors |
| `/dev/nvidiaN` | per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation |
| `/dev/nvidia-uvm` | managed memory and page migration (UVM userspace API) |
| `/dev/nvidia-vgpuN` | vGPU plugin node (only inside NVIDIA's own vGPU stack; not used by stock CUDA paths) |
Control-device ioctls (`_IOWR('F', cmd, …)` with magic `'F'`):
| cmd | Name | Purpose |
|---|---|---|
| 200 (`0xc8`) | `NV_ESC_CARD_INFO` | enumerate GPUs and minors |
| 201 (`0xc9`) | `NV_ESC_REGISTER_FD` | bind a per-GPU fd to the control fd |
| 206/207 | `NV_ESC_ALLOC_OS_EVENT` / `FREE_OS_EVENT` | OS event objects (blocking waits, GPU events) |
| 209 (`0xd1`) | `NV_ESC_STATUS_CODE` | driver status on open failure |
| 210 (`0xd2`) | `NV_ESC_CHECK_VERSION_STR` | RM/driver version negotiation |
| 211 (`0xd3`) | `NV_ESC_IOCTL_XFER_CMD` | inter-GPU ioctl forwarding |
| 212 (`0xd4`) | `NV_ESC_ATTACH_GPUS_TO_FD` | attach GPUs to an fd (exported objects) |
| 213 (`0xd5`) | `NV_ESC_QUERY_DEVICE_INTR` | interrupt query |
| 214 (`0xd6`) | `NV_ESC_SYS_PARAMS` | system parameters |
RM escapes (same magic, used on control and per-GPU fds):
| cmd | Name | Purpose |
|---|---|---|
| 0x27 | `NV_ESC_RM_ALLOC_MEMORY` | allocate memory (older path; NVOS02) |
| 0x28 | `NV_ESC_RM_ALLOC_OBJECT` | allocate object (NVOS05) |
| 0x29 | `NV_ESC_RM_FREE` | free object (NVOS00) |
| 0x2a | `NV_ESC_RM_CONTROL` | issue an RM control (NVOS54) |
| 0x2b | `NV_ESC_RM_ALLOC` | allocate object (NVOS21/NVOS64) — **the main alloc path** |
| 0x34 | `NV_ESC_RM_DUP_OBJECT` | duplicate/import an object |
| 0x35 | `NV_ESC_RM_SHARE` | share object |
| 0x41 | `NV_ESC_RM_IDLE_CHANNELS` | idle channels |
| 0x4a | `NV_ESC_RM_VID_HEAP_CONTROL` | heap allocation (NVOS32) |
| 0x4e / 0x4f | `NV_ESC_RM_MAP_MEMORY` / `UNMAP_MEMORY` | map memory into the process (NVOS33/NVOS34) |
| 0x57 / 0x58 | `NV_ESC_RM_MAP_MEMORY_DMA` / `UNMAP_MEMORY_DMA` | map memory into a device VA space (NVOS46/NVOS47) |
| 0x5c / 0x5d | `NV_ESC_RM_EXPORT_OBJECT_TO_FD` / `IMPORT_OBJECT_FROM_FD` | cross-process object sharing |
| 0x52 | `NV_ESC_RM_GET_EVENT_DATA` | read event payload (NVOS41) |
| 0x32/0x33 | `NV_ESC_RM_CONFIG_GET/SET` | registry-style configuration |
In a trace, the ioctl request word encodes the escape and the size; e.g.
`_IOWR('F',0x2b,48)` = `NV_ESC_RM_ALLOC` with a 48-byte payload. Seeing this
request in a crash register dump (`rsi=0xc030462b`) identifies "driver was
allocating an RM object" — that is exactly what happens at the `cuCtxCreate`
crash (see `docs/05`).
## 2. The RM object model as CUDA uses it
Every RM call names a **class**. The classes CUDA allocates, in the order
observed for 570.144 on an emulated TU104:
| Class | Name (public RM header) | When |
|---|---|---|
| `0x0` | `NV01_ROOT` | process client (once, at `cuInit`) |
| `0x80` | `NV01_DEVICE_0` | per GPU, at attach |
| `0x2080` | `NV20_SUBDEVICE_0` | per GPU/subdevice |
| `0x40` | `NV01_EVENT` | events (several) |
| `0xa06c` | `KEPLER_CHANNEL_GROUP_A` | context channel group (TSG) |
| `0x9067` | channel-group/runlist object | context |
| `0x3e` | `NV01_MEMORY_*` | memory objects |
| `0xc46f` | `TURING_CHANNEL_GPFIFO_A` | the channel itself |
| `0xc5c0` | `TURING_COMPUTE_A` | compute engine object bound to the channel |
| `0xc5b5` | (compute class, second engine) | used by the host A/B run for a CE-style object |
| `0xa080` | `KEPLER_DEVICE_VGPU` | vGPU capability query (vGPU only) |
| `0x90f1`-family | VA space (`FERMI_VASPACE_A` etc.) | memory model |
`cuCtxCreate`'s observed ioctl sequence (guest, 570.144):
```
NV_ESC_RM_ALLOC class=0x0 (NV01_ROOT)
NV_ESC_RM_ALLOC class=0x80 (NV01_DEVICE_0)
NV_ESC_RM_ALLOC class=0x2080 (NV20_SUBDEVICE_0)
NV_ESC_RM_ALLOC class=0x40 (events)
NV_ESC_RM_ALLOC class=0xa06c (KEPLER_CHANNEL_GROUP_A)
NV_ESC_RM_ALLOC class=0x9067 (channel group object)
NV_ESC_RM_ALLOC class=0x3e (memory)
NV_ESC_RM_ALLOC class=0xc46f (TURING_CHANNEL_GPFIFO_A) <- channel
NV_ESC_RM_CONTROL cmd=0x80170d (NV0080_CTRL_FIFO_GET_CHANNELLIST)
NV_ESC_RM_ALLOC class=0xc5c0 (TURING_COMPUTE_A) <- last alloc
[engine registration attempt; pre-fix this segfaulted — now rc=999, see docs/05 §3.5-3.6]
```
The first two allocs (`NV01_ROOT`, `NV01_DEVICE_0`) actually happen at
`cuInit`/device attach; the trace window shows them repeated per client setup.
## 3. RM controls the CUDA path uses
Controls are `NV_ESC_RM_CONTROL` with a 32-bit command ID and a fixed-size
parameter block. Families relevant to CUDA bring-up (IDs in the low 32 bits
are `(class << 16) | cmd`; the list below is from traces and decompiled call
sites, not exhaustive):
* **Device/GPU info** — `NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`
(`0x00800089`), `NV2080_CTRL_CMD_GPU_GET_INFO`, `GPU_GET_ENGINES`,
`GPU_GET_CLASSLIST`, `GPU_GET_SIMULATION_INFO`, `FB_GET_INFO`,
`BUS_GET_INFO`, `BUS_GET_PCI_INFO`, `BUS_GET_PCI_BAR_INFO`,
`MC_GET_ARCH_INFO`, `GR_GET_INFO`, `GR_GET_GPC_MASK`, `GR_GET_TPC_MASK`,
`GR_GET_GLOBAL_SM_ORDER`, `FB_GET_LTC_INFO_FOR_FBP`, `CE_GET_CAPS`,
`GSP_GET_FEATURES`, `GPU_GET_SW_FEATURES`, `GPU_GET_MEMOP_ENABLE`,
`FB_GET_CAPS`, `GR_GET_CAPS_V2`.
* **FIFO/channels** — `NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` (`0x80170d`),
`NV0080_CTRL_CMD_FIFO_GET_CAPS`, `NVA06F/NVA06C_CTRL_CMD_GPFIFO_SCHEDULE`,
`SET_ERROR_NOTIFIER`, `NVA06C_CTRL_CMD_SET/GET_TIMESLICE`,
`NV2080_CTRL_CMD_FIFO_DISABLE_CHANNELS`, per-class
`GET_CLASS_ENGINEID` (e.g. `NV906F`, `NVA06F`-family),
`NVC36F_CTRL_CMD_GPFIFO_GET_WORK_SUBMIT_TOKEN`.
* **GR/context** — `GR_CTXSW_PM_MODE`, `GR_CTXSW_SMPC_MODE`,
`GR_SET_CTXSW_PREEMPTION_MODE`, `GR_PC_SAMPLING_MODE`,
`GR_GET_CTX_BUFFER_SIZE`, `FLCN_GET_CTX_BUFFER_SIZE`.
* **Memory** — `FB_GET_INFO` (BAR1 info), `NV0000_CTRL_CMD_OS_UNIX_*`
(flush user cache, exported objects), `NV0041_CTRL_CMD_GET_SURFACE_INFO`,
`NV0000_CTRL_CMD_CLIENT_GET_ADDR_SPACE_TYPE`.
* **Events/interrupts** — `NV90E6_CTRL_CMD_MASTER_GET_VIRTUAL_FUNCTION_ERROR_CONT_INTR_MASK`
(interrupt-tree setup), `NV83DE_CTRL_CMD_DEBUG_*` (SM error states, debug
mode), `RC_ENABLE/DISABLE/SOFT_DISABLE_WATCHDOG`, `RELEASE_WATCHDOG_REQUESTS`.
* **vGPU/licensing** — `NVA080_CTRL_CMD_VGPU_GET_CONFIG` (`0xa0800301`),
`NVA080_CTRL_CMD_VGPU_SET_LICENSE_INFO`,
`NV2080_CTRL_CMD_GPU_GET_LICENSABLE_FEATURES`,
`GPU_GET_GRID_UNLICENSED_STATE_MACHINE_INFO` (`0x2080018c`).
* **Perf/clocks** — `GPU_GET_COMPUTE_POLICY_CONFIG`, boost/sync-boost
controls, `NV2080_CTRL_CMD_GPU_ACQUIRE/RELEASE_COMPUTE_MODE_RESERVATION`.
Traces show the driver tolerating unknown-but-successful controls: it only
fails when a *specific* reply field is consumed and wrong. This is why the
emulation can start with generic success and then bisect the consumed fields
(documented in `docs/04` as the "reply-fidelity ladder").
## 4. Memory: allocation, mapping, VA
CUDA's memory path uses three layers:
1. **VA reservation** — `NV_ESC_RM_VID_HEAP_CONTROL` with a virtual
`NVOS32_FUNCTION_ALLOC_SIZE` request (or no reservation at all when UVM
owns the VA space).
2. **Backing allocation** — vidmem/sysmem `NVOS32` allocation with attributes:
location (`VIDMEM`/`PCI`/`SYSTEM`), page size, coherency, compression,
protection, fixed physical address (if requested), or pinning an existing
host address (`NV01_MEMORY_SYSTEM_OS_DESCRIPTOR`).
3. **Mapping** — `NV_ESC_RM_MAP_MEMORY` (`NVOS33`) to map into the process
(for host-visible/CPU access, flags include MAP_FIXED, RESERVE_ON_UNMAP,
DIRECT for incoherent sysmem) and `NV_ESC_RM_MAP_MEMORY_DMA` (`NVOS46`) to
map into a device VA space (flags include RW/RO, snoop, fixed DMA offset,
page size, peer flags).
Related classes: `FERMI_VASPACE_A` (VA space; one is shared with UVM, one
legacy, optionally one for fabric-local aperture), `NV01_MEMORY_FABRIC*` /
`NV_MEMORY_FABRIC*` (NVLink/fabric), `NV01_MEMORY_SYSTEM_OS_DESCRIPTOR`
(pinned host memory), `NV01_MEMORY_LOCAL_USER` (vidmem).
For Open-IOV the practical summary is: **all guest memory that CUDA uses is
either an RM allocation backed by emulated VRAM/BAR1, or pinned guest RAM that
the GPU must be able to DMA to.** The project's current model backs VRAM with
host RAM and executes push-buffer work in the host (see the parent handoff).
## 5. Channels, GPFIFO, USERD and doorbells
A CUDA channel is a GPFIFO (array of pushbuffer descriptors) plus one or more
engine objects:
* GPFIFO memory: allocated by CUDA (vidmem or sysmem), passed to the kernel
driver at channel creation with its offset and entry count, together with
an error notifier object.
* **USERD**: on Volta and later, CUDA allocates the user-mode doorbell/control
area itself and passes its handles to the channel allocation; older
generations map the kernel-provided USERD. On Hopper, a second BAR1-mapped
usermode region is used for doorbells.
* **Doorbell**: submission writes the GPFIFO put pointer into USERD and rings
a doorbell token (obtained via `NVC36F_CTRL_CMD_GPFIFO_GET_WORK_SUBMIT_TOKEN`
on newer ABIs, or a static token on older ones) at a token/offset the
channel allocation returned.
* **Pushbuffers**: command streams in vidmem or sysmem; a push is a sequence
of methods (`SET_OBJECT`, engine methods, semaphore releases) terminated by
a tracking-semaphore release and a new GPFIFO entry.
* **Context share (Volta+)**: subcontexts are created by allocating a
`FERMI_CONTEXT_SHARE_A` object with the VA space handle and subcontext id;
channels then reference it instead of the VA space directly.
Open-IOV relevance: the host emulation sees the GPFIFO and pushbuffer memory
the guest writes. The project already decodes push buffers as method streams
and executes memory ops (`kernel/gsp_sema_service.c`); this section defines
the contract those streams follow.
## 6. Synchronization objects below the API
* **Semaphores** — plain 32/64-bit memory values, allocated by CUDA in
GPU-visible memory. The GPU releases them (semaphore release methods or
inline memcpy of the payload), the CPU polls or waits. Tracking semaphores
count completed pushes; QMD semaphores count completed grids.
* **Notifiers** — 4 KB error-notifier memory per channel; the kernel driver
writes a notification record on channel errors, which CUDA reads when
synchronizing to map to `CUDA_ERROR_LAUNCH_*`.
* **OS events** — kernel objects (`NV_ESC_ALLOC_OS_EVENT`) used to wake
blocked threads; GPU events bind an engine event to an OS event so an
interrupt can signal user space.
* **Interrupts** — the kernel driver owns the GPU interrupt; CUDA's ISR
thread is woken through the OS event/monitored-fence mechanism and then
walks channel state, updates blocking-sync accounting and runs DPCs.
* **Trap handler** — for debuggers and `device-side assert`, CUDA installs a
trap handler (embedded device code) and reads a trace buffer in GPU memory.
## 7. UVM interface
When UVM is available, the userspace driver calls the UVM API (through
`/dev/nvidia-uvm`):
* set the expected kernel driver version (forward compatibility);
* initialize UVM (sharing mode for MPS, HMM/pageable-memory options);
* register each GPU and each VA space;
* register each channel with its platform parameters (the RM handles and
context-buffer VA/size the kernel driver provided);
* allocate/register managed allocations, prefetch/advice calls;
* semaphores: UVM semaphore acquire/release are pushed into the same streams.
If UVM is unavailable or the vGPU profile lacks UVM/fault capability, CUDA
disables kernel-mode UVM and degrades to zero-copy behavior.
## 8. GSP-RM in the picture
Modern drivers run the resource manager on the GSP (a RISC-V core on the
GPU). From the userspace driver's perspective this changes nothing: it still
issues RM ioctls. Inside the kernel driver, controls become GSP-RM RPCs; in
the Open-IOV emulation, the host answers those RPCs. The host log for the
`cuCtxCreate` window shows, in order:
```
CE_GET_FAULT_METHOD_BUFFER_SIZE -> 12288
GPU_PROMOTE_CTX (0x2080012b, 560 B) -> generic SUCCESS
TURING_CHANNEL_GPFIFO_A alloc, TURING_COMPUTE_A alloc
```
The userspace failure now surfaces after the engine registration attempt: with
the alloc reply fixed (`docs/05` §3.5) the walker reads a zeroed secondary
engine record and `cuCtxCreate` returns `rc=999` (`docs/05` §3.6). The
function-level path is in `docs/05`; the emulation-side question is what input
builds the driver's class-filtered engine list.
## 9. Emulation checklist (derived from the above)
For an unmodified CUDA 12.x userspace driver in a guest to reach a working
context, the host emulation must provide, at minimum:
1. **Control device semantics** — version negotiation, client creation, GPU
attach, OS events, per-GPU fds.
2. **Object allocation** for the classes in §2, with plausible sizes/handles,
and successful `NV_ESC_RM_FREE`.
3. **Device/engine topology** — a consistent engine list and class list
(`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`, static info) matching what the
caps/controls claim; per-engine records must be self-consistent (see the
engine-descriptor discussion in `docs/05`).
4. **Memory** — VA spaces, vidmem/sysmem allocations with the requested
attributes, device and host mappings, BAR1 info.
5. **Channels** — GPFIFO alloc with notifier and USERD handles, engine object
alloc, scheduling, channel ids, doorbell tokens/offsets.
6. **Interrupt/event plumbing** — interrupt tree or usermode region, OS
events, error notifiers.
7. **Reply fidelity for consumed controls** — the small set of controls whose
*values* (not just status) the driver reads: memory sizes, engine ids,
class engine ids, BAR1 info, channel ids, version info, UVM caps.
8. **Push-buffer execution** — the GPU work encoded by userspace must actually
happen (memsets, copies, semaphore releases, kernel grids) or the driver
will hang in synchronization rather than fail fast.
|
