1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
|
# Guest Driver Lifecycle: A Field Guide
> **Scope.** A walking tour of what happens between "guest boots" and "CUDA
> initializes", using message, function and control IDs observed from an
> unmodified 570.144 guest. Canonical source:
> [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md); if this page and the docs tree disagree, the docs tree wins.
>
> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival.
## 1. How to read this page
Each stage lists what the **guest** does, what the **device/firmware** must answer, what host emulation does today, and the log signature to look for (host-side `dmesg` with prefixes `vgsp:` / `vgsp-rm:`; guest-side `dmesg` in the VM). IDs are for 570.144 on the Turing (TU104) profile; other generations add IDs and change versioned values, but the *stages* are stable.
## 2. Stage map
```text
guest boots
│
├─ 1. PCI probe & config space ........ lspci is the reference
├─ 2. GSP boot handshake .............. WPR, FWSEC, booter, RISC-V core
├─ 3. Static configuration ............ GET_GSP_STATIC_INFO (65)
├─ 4. Message queues .................. SET_GUEST_SYSTEM_INFO (1) + init traffic
├─ 5. RM control battery .............. GSP_RM_CONTROL (76) × ~160
├─ 6. Object creation ................. GSP_RM_ALLOC (103) × ~60
└─ 7. UVM registration/context ........ cuInit → channel manager → execution frontier
```
## 3. PCI probe and config space
**Guest:** reads IDs, walks capabilities, sizes BARs, enables Memory Space and Bus Master, programs MSI-X. **Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB BAR1, an MSI-X capability and a link capability the driver accepts. **Emulation:** config space is synthesized and BAR sizes are enforced. The hypervisor is the usual source of surprises — a device on the root bus can have its PCIe capability rewritten to "root complex integrated endpoint", removing link information the driver wants; attach behind a root port. **Log:** guest-side `lspci -vvv` is the reference.
## 4. GSP boot handshake
**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the RISC-V core; polls a status mailbox for "GSP init done". **Device must:** complete every stage in the order the driver polls and leave the mailbox in the state the driver expects. **Emulation:** the host module models the stage state machine and auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls terminate.
```text
vgsp: GSP-BOOT: ... trigger auto-cleared ...
```
## 5. Static configuration
**Guest:** `GET_GSP_STATIC_INFO` (65) → a large `GspStaticConfigInfo` structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS info. **Device must:** provide self-consistent sizes, engine inventories, memory geometry, interrupt topology and a parseable VBIOS blob. **Emulation:** the static structures are synthesized for TU104 (FB size, BAR layout, engine list, vGPU-friendly VF values).
```text
vgsp-rm: GET_GSP_STATIC_INFO → ...
```
## 6. Message queues
**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue buffer description and a page table (the buffer is `NV_MEMORY_NONCONTIGUOUS`); also `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic. **Device must:** consume them silently (async) and consume the command queue exactly in the guest's write order. **Emulation:** parses the queue chain, builds the geometry, links the status queue and consumes elements one by one — **element-based, not byte-stream-based**: one message is one element and may span pages.
```text
vgsp: libos region[i] id8=... pa=... size=...
vgsp: queue args sharedPA=... ptes=... cmdOff=... statOff=...
```
## 7. The RM control battery
**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. Of the ~160 observed control IDs, 23 are special-cased with real values; the rest are answered with a generic `SUCCESS` and zeroed params (the fallback).
| Control | Meaning | What matters |
|---|---|---|
| `0x20800a40` family | internal device info table | per-index values, entry count |
| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries |
| `0x20800a5c` | interrupt kernel table | vector topology the driver maps |
| `0x20800a59` | GMMU static info | fault buffer sizes; a wrong value breaks UVM early |
| `0x20800a01` | display static info | display count; headless values are fine |
| `0x20801823/-a/-b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth math |
| `0x20802a07` | CE physical caps | per-engine bits; UVM refuses channels without `sysmem` support |
| `0x20802a0a/0b` | all-CE caps | caps table for all engines |
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | carries page-table physical addresses (see [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md)) |
| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory; the guest must see the full list, including async copy (`COPY14`) |
| `0x800201` | `GPU_GET_CLASSLIST` | class list; must agree with engine list and CE caps |
| `0x2080012b` | `GPU_PROMOTE_CTX` (560-byte params) | load-bearing GR context-validity operation; a wrong shape is a server wedge handled by the supervised fail-safe |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these often |
```text
vgsp-rm: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...
```
## 8. Object creation
**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a CUDA process:
```text
0x0 NV01_ROOT (client root, 120-byte params)
0x80 NV01_DEVICE_0 (56 bytes)
0x2080 NV20_SUBDEVICE_0 (4 bytes)
0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space
0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group
0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076 GP100_UVM_SW (0 bytes)
0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)
```
A hard-won rule from the `TURING_COMPUTE_A` (0xc5c0) alloc: the caller may issue it with `paramsSize=0` against a small stack local while the kernel side treats the class params as 16 bytes and copies reply params back. The emulation must **echo the request params back in the alloc reply**, or the user-mode client segfaults. Channel parameters decode into `gpFifoOffset` (a GPU VA), `gpFifoEntries`, `engineType` (e.g. `COPY0 = 0x9`), and `NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with **physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`).
```text
vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...
```
## 9. UVM registration and the execution frontier
**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. **Emulation today:** the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet.
**Progress:** context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted `CTXNOTVALID`, and a following `GPU_PROMOTE_CTX` wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [`Execution-Frontier.md`](Execution-Frontier.md). The roadmap from here is:
```text
UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)
```
Each remaining step needs the same treatment: satisfy the interface, forward the work, relay the completions.
```text
nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N
```
## 10. Quick reference: function IDs from a stock 570.144 guest
| Function | Name | Frequency per CUDA run (approx.) |
|---|---|---|
| 1 | SET_GUEST_SYSTEM_INFO | once |
| 10 | FREE | ~14 |
| 47 | UNLOADING_GUEST_DRIVER | on unload |
| 65 | GET_GSP_STATIC_INFO | once |
| 70 | UPDATE_BAR_PDE | a few |
| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each |
| 76 | GSP_RM_CONTROL | ~160 |
| 103 | GSP_RM_ALLOC | ~60 |
If a new driver version produces **new function numbers**, do not guess: add a handler that logs the raw payload, then decode it against that version's observed traffic.
## 11. Related documentation
* [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md) — canonical field guide.
* [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — GSP boot, message queues and RM architecture.
* [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — control/alloc emulation on the host side.
* [`RPC-Message-Format.md`](RPC-Message-Format.md) — wire format of the function 76/103 traffic.
* [`Address-Reference.md`](Address-Reference.md) and [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) — ID reference and the firmware they belong to.
* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`Execution-Frontier.md`](Execution-Frontier.md), [`Host-Module.md`](Host-Module.md), [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — registration, the frontier, the host stages and the forwarding backend.
|
