1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
|
# GSP and GSP-RM
> **What this page is.** The GPU System Processor hosts NVIDIA's Resource
> Manager in firmware; since Turing, the guest driver's management path is
> *talking to that firmware*, not to silicon. This page explains that
> architecture and the interface the project emulates. Condensed from
> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md);
> the docs tree wins on disagreement.
Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the
stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]`
designed end state.
## 1. What the GSP is
The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA
GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF)
that contains, among other things, a full port of NVIDIA's **Resource Manager
(RM)** — the component that owns GPU initialization, memory, engines,
interrupts and scheduling.
```text
┌──────────────────────────── GUEST ────────────────────────────┐
│ nvidia.ko │
│ RM client path ── RPC ──┐ │
│ GSP boot code │ │
└────────────────────────────┼──────────────────────────────────┘
│ shared memory queues + doorbells
┌────────────────────────────┼──────────────────────────────────┐
│ GSP (firmware, RISC-V) │ │
│ RM core (RMCore) ◄─────┘ │
│ RM API (object model, clients, handles) │
│ Resource server (resserv): allocation & lifetime │
│ memory manager, GMMU, FIFO/runlist, interrupts, PMU │
└───────────────────────────────────────────────────────────────┘
```
A driver as complex as NVIDIA's is not written to program silicon directly;
it is written to talk to an RM, and on modern GPUs a copy of the RM runs on
the GSP. The driver is therefore an **RM client**: it issues requests,
receives callbacks, and observes state through a well-defined protocol.
### What the GSP does on bare metal
* **It is the device's manager processor.** Since Turing, the driver does not
program most of the GPU. The GSP runs NVIDIA's signed firmware — a small
RTOS plus the RM server — and owns GPU-global state: initialisation, the
memory-management unit's deeper page-table levels, engine configuration,
context scheduling, interrupts and power/clock management (PMU). The
driver's kernel RM boots it, then becomes its client.
* **It boots from driver-provided memory.** At power-on the driver carves a
firmware-protected region of video memory (WPR2), copies the firmware image
there, has the device verify signatures (FWSEC), hands control to the boot
ROM ("booter"), which starts the RISC-V core; the firmware runs its RTOS and
RM initialisation and signals readiness. Everything the driver does
afterwards is a request to that server.
* **Why this matters for virtualization.** Because the management path is a
*firmware interface* — shared-memory queues plus a small RPC protocol,
not a pile of register lore — it is small, stable, and observable. That is
the interface Open-IOV emulates; it is the reason a virtual NVIDIA GPU is
practical at all.
## 2. The boot handshake we emulate
The guest driver boots the GSP exactly as it would on bare metal:
1. **WPR setup** — carve a firmware-protected region of VRAM and write
descriptors into BAR0 registers ([`WPR-Management.md`](WPR-Management.md)).
2. **FWSEC** — the driver verifies firmware signatures.
3. **Booter** — the boot ROM consumes the firmware image.
4. **RISCV start** — the RISC-V core begins executing the firmware.
5. **GSP init done** — the firmware signals readiness through a status
mailbox; the driver proceeds to initialization RPCs.
The emulation implements **stage completion** rather than the stages
themselves — legitimate because the only observable to the driver is the state
machine it polls. The same handshake has also been run against real firmware
on the physical backend: the firmware comes up, delivers its first status
events, serves the static configuration, and stays live until the unload RPC
releases it ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).
## 3. The transport: message queues and RPC
GSP-RM's transport is a shared-memory message queue pair:
```text
guest system memory (allocated by the guest driver, registered with us)
┌───────────────────────────────┐
│ command queue (guest -> GSP) │ element ring, 4 KB elements
├───────────────────────────────┤
│ status queue (GSP -> guest) │ element ring
└───────────────────────────────┘
+ a page table describing the queue pages (the buffer is
non-contiguous: each page is an independent allocation)
```
Each element begins with a transport header followed by an **RPC message**
whose 32-byte header carries header version, length, function, result and
sequence number. Observations worth knowing (all verified against the stock
guest):
* `length` covers the fixed part of the message, not variable payloads — do
not use it to bound a payload;
* function numbers are a flat enum shared by several protocols (`SET_GUEST_SYSTEM_INFO` 1,
`FREE` 10, `GET_GSP_STATIC_INFO` 65, `GSP_RM_CONTROL` 76, `GSP_RM_ALLOC` 103, ...);
* replies must be matched to request sequence numbers — the driver runs a
reply-matching state machine, and a slow reply is indistinguishable from a
hang.
## 4. RMCore, the RM API and the resource server
Three RM concepts make every log readable:
* **RMCore** — the object system. Everything is an object with a class (a
32-bit class ID), a handle in a per-client handle space, and a parent
object forming a tree rooted at a client. Traces of `GSP_RM_ALLOC` are
object creation; traces of `GSP_RM_CONTROL` are method calls on objects.
* **RM API** — how in-kernel components (and user space via ioctls) allocate
objects and call controls, with an implicit-handle and an explicit-handle
variant. For a GSP client, part of the API becomes RPCs while some calls
are handled locally; memory objects, for example, are allocated locally by
the client RM.
* **Resource server (resserv)** — the allocation authority: it owns handle
spaces, enforces privileges and tracks object lifetimes.
## 5. The client/server split (and why page tables matter)
Modern RM is client/server: the **client** (guest `nvidia.ko`) owns
per-process state and submits work; the **server** (GSP-RM) owns GPU-global
state — routing, page tables, scheduling domains, engine configuration. For
virtualized setups the client tells the server which VA ranges the server
owns, and the server manages the deeper page-table levels for those ranges.
The client hands over the **physical addresses of its page-table levels**
plus the VA ranges they cover; on real hardware the server then fills in the
lower levels for server-owned ranges and the client never writes them.
This is a central fact for the execution work: when the guest's UVM asks the
"GPU" to map memory, the mapping is expected to materialize in page tables
that only the server side can write — which is us
([`UVM-in-the-Guest.md`](UVM-in-the-Guest.md),
[`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)).
## 6. Event channels and notifications
Beyond request/reply, the guest needs asynchronous events: event objects
registered with a notification configuration (an MSI-X vector or a memory
"semaphore" the GPU can write), completion events when channels finish work or
errors occur, delivered as a strict bit/handle/state triple on the exact
vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM →
guest vector, with the pending-bit array emulated so a masked vector's
interrupt is not lost.
## 7. How Open-IOV deals with the GSP — two modes
The project handles the GSP in two ways, and they complement each other:
1. **Guest-facing GSP emulation** (`vgsp.ko` *is* the virtual GSP). For each
mediated device, the module presents a device whose GSP behaviour is
software: it implements the boot state machine the driver polls (stage
completion, not silicon), answers the WPR/BAR0 register protocol, serves
the message-queue transport and every RPC verb the driver uses during
probe, initialisation and runtime, and delivers events via emulated MSI-X.
The guest's RM believes it is talking to firmware; it is talking to the
emulation.
2. **The real GSP as the backend server.** When the module owns the physical
GPU, it loads the *matching real firmware image* into the device, runs the
actual boot (FWSEC → booter → RISC-V → RM init-done), and becomes the RM
client itself — "be the host driver's RM" becomes "be the RM client". The
same protocol the guest uses one layer up is used here one layer down, and
it is what the work path submits through
([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).
Keeping the two in sync is a project discipline: what the guest's RM is
promised (objects, controls, page tables, completions) must be answerable by
the emulation, and what executes on hardware must round-trip through the real
firmware session.
### What is emulated today
| Layer | State | Notes |
|---|---|---|
| GSP boot state machine | `[live]` | stages complete; the driver's poll loop is satisfied |
| Message queue transport | `[live]` | including the non-contiguous page table for the queue buffer |
| RPC request/reply | `[live]` | every function the driver uses during init is answered; unknown functions get a well-formed success reply |
| Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list |
| Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving |
| Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet |
| Page tables for server-owned ranges | `[wip]` | contract received and recorded; forwarded paths run on module-built scratch tables while full server-owned leaf population is the remaining execution work |
| Event/notification delivery | `[live]` | MSI-X works; completion counters, notifiers and channel events are serviced and relayed to the guest |
| Real GSP as the backend server | `[live]` | the module boots the real firmware, serves its RM client ladder, and executes forwarded work through it |
| Firmware behaviour (routing, scheduling, real execution) | `[live]` | forwarded guest CUDA launches and graphics method streams execute on the physical engines — see [`Execution-Frontier.md`](Execution-Frontier.md) |
## 8. Where the firmware knowledge lives
* [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) —
how the 28.5 MB image was decompiled and mapped.
* [`Address-Reference.md`](Address-Reference.md) — firmware-side address
tables.
* [`RPC-Message-Format.md`](RPC-Message-Format.md),
[`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — historical RPC analysis
(superseded; kept for the record).
* [`WPR-Management.md`](WPR-Management.md) — write-protected region semantics.
* [The execution plane](Execution-Plane.md) and
[the execution frontier](Execution-Frontier.md) — what turns this protocol
into real work on real hardware.
|
