# GSP and GSP-RM

> **What this page is.** The GPU System Processor hosts NVIDIA's Resource
> Manager in firmware; since Turing, the guest driver's management path is
> *talking to that firmware*, not to silicon. This page explains that
> architecture and the interface the project emulates. Condensed from
> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md);
> the docs tree wins on disagreement.

Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]`
designed end state.

## 1. What the GSP is

The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA
GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF)
that contains, among other things, a full port of NVIDIA's **Resource Manager
(RM)** — the component that owns GPU initialization, memory, engines,
interrupts and scheduling.

```text
   ┌──────────────────────────── GUEST ────────────────────────────┐
   │  nvidia.ko                                                   │
   │    RM client path ── RPC ──┐                                  │
   │    GSP boot code           │                                  │
   └────────────────────────────┼──────────────────────────────────┘
                                │  shared memory queues + doorbells
   ┌────────────────────────────┼──────────────────────────────────┐
   │  GSP (firmware, RISC-V)    │                                   │
   │    RM core (RMCore)  ◄─────┘                                   │
   │    RM API  (object model, clients, handles)                    │
   │    Resource server (resserv): allocation & lifetime            │
   │    memory manager, GMMU, FIFO/runlist, interrupts, PMU         │
   └───────────────────────────────────────────────────────────────┘
```

A driver as complex as NVIDIA's is not written to program silicon directly;
it is written to talk to an RM, and on modern GPUs a copy of the RM runs on
the GSP. The driver is therefore an **RM client**: it issues requests,
receives callbacks, and observes state through a well-defined protocol.

## 2. The boot handshake we emulate

The guest driver boots the GSP exactly as it would on bare metal:

1. **WPR setup** — carve a firmware-protected region of VRAM and write
   descriptors into BAR0 registers ([`WPR-Management.md`](WPR-Management.md)).
2. **FWSEC** — the driver verifies firmware signatures.
3. **Booter** — the boot ROM consumes the firmware image.
4. **RISCV start** — the RISC-V core begins executing the firmware.
5. **GSP init done** — the firmware signals readiness through a status
   mailbox; the driver proceeds to initialization RPCs.

The emulation implements **stage completion** rather than the stages
themselves — legitimate because the only observable to the driver is the state
machine it polls. The same handshake has also been run against real firmware
on the physical backend: the firmware comes up, delivers its first status
events, serves the static configuration, and stays live until the unload RPC
releases it ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).

## 3. The transport: message queues and RPC

GSP-RM's transport is a shared-memory message queue pair:

```text
  guest system memory (allocated by the guest driver, registered with us)
  ┌───────────────────────────────┐
  │ command queue (guest -> GSP)  │  element ring, 4 KB elements
  ├───────────────────────────────┤
  │ status queue  (GSP -> guest)  │  element ring
  └───────────────────────────────┘
     + a page table describing the queue pages (the buffer is
       non-contiguous: each page is an independent allocation)
```

Each element begins with a transport header followed by an **RPC message**
whose 32-byte header carries header version, length, function, result and
sequence number. Observations worth knowing (all verified against the stock
guest):

* `length` covers the fixed part of the message, not variable payloads — do
  not use it to bound a payload;
* function numbers are a flat enum shared by several protocols (`SET_GUEST_SYSTEM_INFO` 1,
  `FREE` 10, `GET_GSP_STATIC_INFO` 65, `GSP_RM_CONTROL` 76, `GSP_RM_ALLOC` 103, ...);
* replies must be matched to request sequence numbers — the driver runs a
  reply-matching state machine, and a slow reply is indistinguishable from a
  hang.

## 4. RMCore, the RM API and the resource server

Three RM concepts make every log readable:

* **RMCore** — the object system. Everything is an object with a class (a
  32-bit class ID), a handle in a per-client handle space, and a parent
  object forming a tree rooted at a client. Traces of `GSP_RM_ALLOC` are
  object creation; traces of `GSP_RM_CONTROL` are method calls on objects.
* **RM API** — how in-kernel components (and user space via ioctls) allocate
  objects and call controls, with an implicit-handle and an explicit-handle
  variant. For a GSP client, part of the API becomes RPCs while some calls
  are handled locally; memory objects, for example, are allocated locally by
  the client RM.
* **Resource server (resserv)** — the allocation authority: it owns handle
  spaces, enforces privileges and tracks object lifetimes.

## 5. The client/server split (and why page tables matter)

Modern RM is client/server: the **client** (guest `nvidia.ko`) owns
per-process state and submits work; the **server** (GSP-RM) owns GPU-global
state — routing, page tables, scheduling domains, engine configuration. For
virtualized setups the client tells the server which VA ranges the server
owns, and the server manages the deeper page-table levels for those ranges.
The client hands over the **physical addresses of its page-table levels**
plus the VA ranges they cover; on real hardware the server then fills in the
lower levels for server-owned ranges and the client never writes them.

This is a central fact for the execution work: when the guest's UVM asks the
"GPU" to map memory, the mapping is expected to materialize in page tables
that only the server side can write — which is us
([`UVM-in-the-Guest.md`](UVM-in-the-Guest.md),
[`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)).

## 6. Event channels and notifications

Beyond request/reply, the guest needs asynchronous events: event objects
registered with a notification configuration (an MSI-X vector or a memory
"semaphore" the GPU can write), completion events when channels finish work or
errors occur, delivered as a strict bit/handle/state triple on the exact
vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM →
guest vector, with the pending-bit array emulated so a masked vector's
interrupt is not lost.

## 7. What we emulate today

| Layer | State | Notes |
|---|---|---|
| GSP boot state machine | `[live]` | stages complete; the driver's poll loop is satisfied |
| Message queue transport | `[live]` | including the non-contiguous page table for the queue buffer |
| RPC request/reply | `[live]` | every function the driver uses during init is answered; unknown functions get a well-formed success reply |
| Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list |
| Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving |
| Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet |
| Page tables for server-owned ranges | `[wip]` | contract received and recorded; leaf population is the execution work |
| Event/notification delivery | `[wip]` | MSI-X works; richer event semantics land with forwarding |
| Firmware behaviour (routing, scheduling, real execution) | `[wall]` | the execution frontier — see [`Execution-Frontier.md`](Execution-Frontier.md) |

## 8. Where the firmware knowledge lives

* [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) —
  how the 28.5 MB image was decompiled and mapped.
* [`Address-Reference.md`](Address-Reference.md) — firmware-side address
  tables.
* [`RPC-Message-Format.md`](RPC-Message-Format.md),
  [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — historical RPC analysis
  (superseded; kept for the record).
* [`WPR-Management.md`](WPR-Management.md) — write-protected region semantics.
* [The execution plane](Execution-Plane.md) and
  [the execution frontier](Execution-Frontier.md) — what turns this protocol
  into real work on real hardware.