# GSP and GSP-RM

> **What this page is.** The GPU System Processor hosts NVIDIA's Resource
> Manager in firmware; since Turing, the guest driver's management path is
> *talking to that firmware*, not to silicon. This page explains that
> architecture and the interface the project emulates. Condensed from
> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md);
> the docs tree wins on disagreement.

Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the
stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]`
designed end state.

## 1. What the GSP is

The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA
GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF)
that contains, among other things, a full port of NVIDIA's **Resource Manager
(RM)** — the component that owns GPU initialization, memory, engines,
interrupts and scheduling.

```text
   ┌──────────────────────────── GUEST ────────────────────────────┐
   │  nvidia.ko                                                   │
   │    RM client path ── RPC ──┐                                  │
   │    GSP boot code           │                                  │
   └────────────────────────────┼──────────────────────────────────┘
                                │  shared memory queues + doorbells
   ┌────────────────────────────┼──────────────────────────────────┐
   │  GSP (firmware, RISC-V)    │                                   │
   │    RM core (RMCore)  ◄─────┘                                   │
   │    RM API  (object model, clients, handles)                    │
   │    Resource server (resserv): allocation & lifetime            │
   │    memory manager, GMMU, FIFO/runlist, interrupts, PMU         │
   └───────────────────────────────────────────────────────────────┘
```

A driver as complex as NVIDIA's is not written to program silicon directly;
it is written to talk to an RM, and on modern GPUs a copy of the RM runs on
the GSP. The driver is therefore an **RM client**: it issues requests,
receives callbacks, and observes state through a well-defined protocol.

### What the GSP does on bare metal

* **It is the device's manager processor.** Since Turing, the driver does not
  program most of the GPU. The GSP runs NVIDIA's signed firmware — a small
  RTOS plus the RM server — and owns GPU-global state: initialisation, the
  memory-management unit's deeper page-table levels, engine configuration,
  context scheduling, interrupts and power/clock management (PMU). The
  driver's kernel RM boots it, then becomes its client.
* **It boots from driver-provided memory.** At power-on the driver carves a
  firmware-protected region of video memory (WPR2), copies the firmware image
  there, has the device verify signatures (FWSEC), hands control to the boot
  ROM ("booter"), which starts the RISC-V core; the firmware runs its RTOS and
  RM initialisation and signals readiness. Everything the driver does
  afterwards is a request to that server.
* **Why this matters for virtualization.** Because the management path is a
  *firmware interface* — shared-memory queues plus a small RPC protocol,
  not a pile of register lore — it is small, stable, and observable. That is
  the interface Open-IOV emulates; it is the reason a virtual NVIDIA GPU is
  practical at all.

## 2. The boot handshake we emulate

The guest driver boots the GSP exactly as it would on bare metal:

1. **WPR setup** — carve a firmware-protected region of VRAM and write
   descriptors into BAR0 registers ([`WPR-Management.md`](WPR-Management.md)).
2. **FWSEC** — the driver verifies firmware signatures.
3. **Booter** — the boot ROM consumes the firmware image.
4. **RISCV start** — the RISC-V core begins executing the firmware.
5. **GSP init done** — the firmware signals readiness through a status
   mailbox; the driver proceeds to initialization RPCs.

The emulation implements **stage completion** rather than the stages
themselves — legitimate because the only observable to the driver is the state
machine it polls. The same handshake has also been run against real firmware
on the physical backend: the firmware comes up, delivers its first status
events, serves the static configuration, and stays live until the unload RPC
releases it ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).

## 3. The transport: message queues and RPC

GSP-RM's transport is a shared-memory message queue pair:

```text
  guest system memory (allocated by the guest driver, registered with us)
  ┌───────────────────────────────┐
  │ command queue (guest -> GSP)  │  element ring, 4 KB elements
  ├───────────────────────────────┤
  │ status queue  (GSP -> guest)  │  element ring
  └───────────────────────────────┘
     + a page table describing the queue pages (the buffer is
       non-contiguous: each page is an independent allocation)
```

Each element begins with a transport header followed by an **RPC message**
whose 32-byte header carries header version, length, function, result and
sequence number. Observations worth knowing (all verified against the stock
guest):

* `length` covers the fixed part of the message, not variable payloads — do
  not use it to bound a payload;
* function numbers are a flat enum shared by several protocols (`SET_GUEST_SYSTEM_INFO` 1,
  `FREE` 10, `GET_GSP_STATIC_INFO` 65, `GSP_RM_CONTROL` 76, `GSP_RM_ALLOC` 103, ...);
* replies must be matched to request sequence numbers — the driver runs a
  reply-matching state machine, and a slow reply is indistinguishable from a
  hang.

## 4. RMCore, the RM API and the resource server

Three RM concepts make every log readable:

* **RMCore** — the object system. Everything is an object with a class (a
  32-bit class ID), a handle in a per-client handle space, and a parent
  object forming a tree rooted at a client. Traces of `GSP_RM_ALLOC` are
  object creation; traces of `GSP_RM_CONTROL` are method calls on objects.
* **RM API** — how in-kernel components (and user space via ioctls) allocate
  objects and call controls, with an implicit-handle and an explicit-handle
  variant. For a GSP client, part of the API becomes RPCs while some calls
  are handled locally; memory objects, for example, are allocated locally by
  the client RM.
* **Resource server (resserv)** — the allocation authority: it owns handle
  spaces, enforces privileges and tracks object lifetimes.

## 5. The client/server split (and why page tables matter)

Modern RM is client/server: the **client** (guest `nvidia.ko`) owns
per-process state and submits work; the **server** (GSP-RM) owns GPU-global
state — routing, page tables, scheduling domains, engine configuration. For
virtualized setups the client tells the server which VA ranges the server
owns, and the server manages the deeper page-table levels for those ranges.
The client hands over the **physical addresses of its page-table levels**
plus the VA ranges they cover; on real hardware the server then fills in the
lower levels for server-owned ranges and the client never writes them.

This is a central fact for the execution work: when the guest's UVM asks the
"GPU" to map memory, the mapping is expected to materialize in page tables
that only the server side can write — which is us
([`UVM-in-the-Guest.md`](UVM-in-the-Guest.md),
[`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)).

## 6. Event channels and notifications

Beyond request/reply, the guest needs asynchronous events: event objects
registered with a notification configuration (an MSI-X vector or a memory
"semaphore" the GPU can write), completion events when channels finish work or
errors occur, delivered as a strict bit/handle/state triple on the exact
vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM →
guest vector, with the pending-bit array emulated so a masked vector's
interrupt is not lost.

## 7. How Open-IOV deals with the GSP — two modes

The project handles the GSP in two ways, and they complement each other:

1. **Guest-facing GSP emulation** (`vgsp.ko` *is* the virtual GSP). For each
   mediated device, the module presents a device whose GSP behaviour is
   software: it implements the boot state machine the driver polls (stage
   completion, not silicon), answers the WPR/BAR0 register protocol, serves
   the message-queue transport and every RPC verb the driver uses during
   probe, initialisation and runtime, and delivers events via emulated MSI-X.
   The guest's RM believes it is talking to firmware; it is talking to the
   emulation.
2. **The real GSP as the backend server.** When the module owns the physical
   GPU, it loads the *matching real firmware image* into the device, runs the
   actual boot (FWSEC → booter → RISC-V → RM init-done), and becomes the RM
   client itself — "be the host driver's RM" becomes "be the RM client". The
   same protocol the guest uses one layer up is used here one layer down, and
   it is what the work path submits through
   ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).

Keeping the two in sync is a project discipline: what the guest's RM is
promised (objects, controls, page tables, completions) must be answerable by
the emulation, and what executes on hardware must round-trip through the real
firmware session.

### What is emulated today

| Layer | State | Notes |
|---|---|---|
| GSP boot state machine | `[live]` | stages complete; the driver's poll loop is satisfied |
| Message queue transport | `[live]` | including the non-contiguous page table for the queue buffer |
| RPC request/reply | `[live]` | every function the driver uses during init is answered; unknown functions get a well-formed success reply |
| Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list |
| Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving |
| Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet |
| Page tables for server-owned ranges | `[wip]` | contract received and recorded; forwarded paths run on module-built scratch tables while full server-owned leaf population is the remaining execution work |
| Event/notification delivery | `[live]` | MSI-X works; completion counters, notifiers and channel events are serviced and relayed to the guest |
| Real GSP as the backend server | `[live]` | the module boots the real firmware, serves its RM client ladder, and executes forwarded work through it |
| Firmware behaviour (routing, scheduling, real execution) | `[live]` | forwarded guest CUDA launches and graphics method streams execute on the physical engines — see [`Execution-Frontier.md`](Execution-Frontier.md) |

## 8. Where the firmware knowledge lives

* [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) —
  how the 28.5 MB image was decompiled and mapped.
* [`Address-Reference.md`](Address-Reference.md) — firmware-side address
  tables.
* [`RPC-Message-Format.md`](RPC-Message-Format.md),
  [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — historical RPC analysis
  (superseded; kept for the record).
* [`WPR-Management.md`](WPR-Management.md) — write-protected region semantics.
* [The execution plane](Execution-Plane.md) and
  [the execution frontier](Execution-Frontier.md) — what turns this protocol
  into real work on real hardware.