1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
|
# GSP and GSP-RM
> **What this page is.** The GPU System Processor hosts NVIDIA's Resource
> Manager in firmware; since Turing, the guest driver's management path is
> *talking to that firmware*, not to silicon. This page explains that
> architecture and the interface the project emulates. Condensed from
> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md);
> the docs tree wins on disagreement.
Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]`
designed end state.
## 1. What the GSP is
The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA
GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF)
that contains, among other things, a full port of NVIDIA's **Resource Manager
(RM)** — the component that owns GPU initialization, memory, engines,
interrupts and scheduling.
```text
┌──────────────────────────── GUEST ────────────────────────────┐
│ nvidia.ko │
│ RM client path ── RPC ──┐ │
│ GSP boot code │ │
└────────────────────────────┼──────────────────────────────────┘
│ shared memory queues + doorbells
┌────────────────────────────┼──────────────────────────────────┐
│ GSP (firmware, RISC-V) │ │
│ RM core (RMCore) ◄─────┘ │
│ RM API (object model, clients, handles) │
│ Resource server (resserv): allocation & lifetime │
│ memory manager, GMMU, FIFO/runlist, interrupts, PMU │
└───────────────────────────────────────────────────────────────┘
```
A driver as complex as NVIDIA's is not written to program silicon directly;
it is written to talk to an RM, and on modern GPUs a copy of the RM runs on
the GSP. The driver is therefore an **RM client**: it issues requests,
receives callbacks, and observes state through a well-defined protocol.
## 2. The boot handshake we emulate
The guest driver boots the GSP exactly as it would on bare metal:
1. **WPR setup** — carve a firmware-protected region of VRAM and write
descriptors into BAR0 registers ([`WPR-Management.md`](WPR-Management.md)).
2. **FWSEC** — the driver verifies firmware signatures.
3. **Booter** — the boot ROM consumes the firmware image.
4. **RISCV start** — the RISC-V core begins executing the firmware.
5. **GSP init done** — the firmware signals readiness through a status
mailbox; the driver proceeds to initialization RPCs.
The emulation implements **stage completion** rather than the stages
themselves — legitimate because the only observable to the driver is the state
machine it polls. The same handshake has also been run against real firmware
on the physical backend: the firmware comes up, delivers its first status
events, serves the static configuration, and stays live until the unload RPC
releases it ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).
## 3. The transport: message queues and RPC
GSP-RM's transport is a shared-memory message queue pair:
```text
guest system memory (allocated by the guest driver, registered with us)
┌───────────────────────────────┐
│ command queue (guest -> GSP) │ element ring, 4 KB elements
├───────────────────────────────┤
│ status queue (GSP -> guest) │ element ring
└───────────────────────────────┘
+ a page table describing the queue pages (the buffer is
non-contiguous: each page is an independent allocation)
```
Each element begins with a transport header followed by an **RPC message**
whose 32-byte header carries header version, length, function, result and
sequence number. Observations worth knowing (all verified against the stock
guest):
* `length` covers the fixed part of the message, not variable payloads — do
not use it to bound a payload;
* function numbers are a flat enum shared by several protocols (`SET_GUEST_SYSTEM_INFO` 1,
`FREE` 10, `GET_GSP_STATIC_INFO` 65, `GSP_RM_CONTROL` 76, `GSP_RM_ALLOC` 103, ...);
* replies must be matched to request sequence numbers — the driver runs a
reply-matching state machine, and a slow reply is indistinguishable from a
hang.
## 4. RMCore, the RM API and the resource server
Three RM concepts make every log readable:
* **RMCore** — the object system. Everything is an object with a class (a
32-bit class ID), a handle in a per-client handle space, and a parent
object forming a tree rooted at a client. Traces of `GSP_RM_ALLOC` are
object creation; traces of `GSP_RM_CONTROL` are method calls on objects.
* **RM API** — how in-kernel components (and user space via ioctls) allocate
objects and call controls, with an implicit-handle and an explicit-handle
variant. For a GSP client, part of the API becomes RPCs while some calls
are handled locally; memory objects, for example, are allocated locally by
the client RM.
* **Resource server (resserv)** — the allocation authority: it owns handle
spaces, enforces privileges and tracks object lifetimes.
## 5. The client/server split (and why page tables matter)
Modern RM is client/server: the **client** (guest `nvidia.ko`) owns
per-process state and submits work; the **server** (GSP-RM) owns GPU-global
state — routing, page tables, scheduling domains, engine configuration. For
virtualized setups the client tells the server which VA ranges the server
owns, and the server manages the deeper page-table levels for those ranges.
The client hands over the **physical addresses of its page-table levels**
plus the VA ranges they cover; on real hardware the server then fills in the
lower levels for server-owned ranges and the client never writes them.
This is a central fact for the execution work: when the guest's UVM asks the
"GPU" to map memory, the mapping is expected to materialize in page tables
that only the server side can write — which is us
([`UVM-in-the-Guest.md`](UVM-in-the-Guest.md),
[`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)).
## 6. Event channels and notifications
Beyond request/reply, the guest needs asynchronous events: event objects
registered with a notification configuration (an MSI-X vector or a memory
"semaphore" the GPU can write), completion events when channels finish work or
errors occur, delivered as a strict bit/handle/state triple on the exact
vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM →
guest vector, with the pending-bit array emulated so a masked vector's
interrupt is not lost.
## 7. What we emulate today
| Layer | State | Notes |
|---|---|---|
| GSP boot state machine | `[live]` | stages complete; the driver's poll loop is satisfied |
| Message queue transport | `[live]` | including the non-contiguous page table for the queue buffer |
| RPC request/reply | `[live]` | every function the driver uses during init is answered; unknown functions get a well-formed success reply |
| Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list |
| Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving |
| Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet |
| Page tables for server-owned ranges | `[wip]` | contract received and recorded; leaf population is the execution work |
| Event/notification delivery | `[wip]` | MSI-X works; richer event semantics land with forwarding |
| Firmware behaviour (routing, scheduling, real execution) | `[wall]` | the execution frontier — see [`Execution-Frontier.md`](Execution-Frontier.md) |
## 8. Where the firmware knowledge lives
* [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) —
how the 28.5 MB image was decompiled and mapped.
* [`Address-Reference.md`](Address-Reference.md) — firmware-side address
tables.
* [`RPC-Message-Format.md`](RPC-Message-Format.md),
[`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — historical RPC analysis
(superseded; kept for the record).
* [`WPR-Management.md`](WPR-Management.md) — write-protected region semantics.
* [The execution plane](Execution-Plane.md) and
[the execution frontier](Execution-Frontier.md) — what turns this protocol
into real work on real hardware.
|
