# GSP and GSP-RM

> The GPU System Processor hosts NVIDIA's Resource Manager in firmware. Since
> Turing, the guest driver's management path is *talking to that firmware*, not
> to silicon. This document explains that architecture and the interface we
> emulate.

## 1. What the GSP is

The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs.
It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that
contains, among other things, a full port of NVIDIA's **Resource Manager (RM)**
— the component that owns GPU initialization, memory, engines, interrupts and
scheduling.

Why this matters: a GPU driver as complex as NVIDIA's is not written to
program silicon directly. It is written to talk to an **RM**, and on modern
GPUs a *copy* of the RM runs on the GSP. The driver is then an **RM client**:
it issues requests, receives callbacks, and observes state through a
well-defined protocol.

```
   ┌──────────────────────────── GUEST ────────────────────────────┐
   │  nvidia.ko                                                   │
   │    RM client path ── RPC ──┐                                  │
   │    GSP boot code           │                                  │
   └────────────────────────────┼──────────────────────────────────┘
                                │  shared memory queues + doorbells
   ┌────────────────────────────┼──────────────────────────────────┐
   │  GSP (firmware, RISC-V)    │                                   │
   │    RM core (RMCore)  ◄─────┘                                   │
   │    RM API  (object model, clients, handles)                    │
   │    Resource server (resserv): allocation & lifetime            │
   │    Memory manager, GMMU, FIFO/runlist, interrupts, PMU         │
   └───────────────────────────────────────────────────────────────┘
```

## 2. GSP boot: the handshake we emulate

The guest driver boots the GSP exactly as it would on bare metal:

1. **WPR (Write-Protected Region) setup** — carve a protected region of VRAM
   for firmware, write descriptors into BAR0 registers.
2. **FWSEC** — the driver verifies firmware signatures (our emulation answers
   the microcode/falcon handshake plausibly).
3. **Booter** — the boot ROM/booter is released and consumes the firmware
   image (we emulate completion).
4. **RISCV start** — the RISC-V core starts executing the firmware; on real
   hardware the firmware then initializes.
5. **GSP init done** — the firmware signals readiness through a status mailbox;
   the driver proceeds to initialization RPCs.

Our emulation implements stage completion rather than the stages themselves —
which is legitimate because the *only observable* to the driver is the state
machine it polls.

On the physical path (2026-10-01), the same handshake ran against real
firmware: FWSEC raised WPR2 with a zero FRTS error code, the SEC2 booter halted
with mailbox0 = 0, the RISC-V core reported active (`RISCV_STATUS=0x00000035`,
active bit 0) and `FALCON_OS` read 0. The server's first message was then a
status-queue event, function `0x1002` (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`,
`rpc_result=0`), carrying a command buffer the host must execute;
`GSP_INIT_DONE` (`0x1001`) follows only after that sequencer runs. The
host-side queue pair behind the exchange is one shared page table plus a 256 KB
command queue and a 256 KB status queue of 4 KB elements.

```mermaid
sequenceDiagram
  autonumber
  participant GD as Guest driver
  participant R0 as BAR0 shadow
  participant GE as GSP emulation thread
  participant MQ as Message queues in guest sysmem
  participant GRM as GSP-RM emulation

  GD->>R0: WPR descriptors and boot kicks
  GE-->>R0: stage status transitions
  GD->>R0: poll for boot complete
  R0-->>GD: firmware running
  GD->>MQ: set up queues and ring the doorbell
  GE->>MQ: read and dispatch elements
  GRM-->>MQ: write replies
  GE-->>GD: MSI-X completion
```

The cross-component detail (doorbells, queues, interrupts, latency classes) is
in [`09-full-system-picture.md`](09-full-system-picture.md) and
[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md).

## 3. The transport: message queues and RPC

GSP-RM's transport is a shared-memory message queue pair:

```
  guest system memory (allocated by the guest driver, registered with us)
  ┌───────────────────────────────┐
  │ command queue (guest→"GSP")   │  512 KB element ring, 4 KB elements
  ├───────────────────────────────┤
  │ status queue  ("GSP"→guest)   │  512 KB element ring
  └───────────────────────────────┘
     + a page table describing the queue pages (the buffer is
       NV_MEMORY_NONCONTIGUOUS: each 4 KB page is an independent allocation)
```

Each element begins with a transport header (`authTag`, `aad`, checksum,
sequence, element count) followed by an **RPC message**:

```
struct rpc_message_header_v {
    u32 header_version; u32 signature; u32 length; u32 function;
    u32 rpc_result; u32 rpc_result_private; u32 sequence;
    union { u32 spare; u32 cpuRmGfid; };
};   // 32 bytes; payload follows in a union of message structs
```

Observations worth knowing (all verified on live hardware):

* `length` is `sizeof(header) + sizeof(fixed message struct)` — for messages
  with variable payloads (e.g. `GSP_RM_ALLOC` with parameters) it does **not**
  include the parameters. Do not use it to bound a payload.
* Function numbers are a flat enum shared by several protocols
  (GSP-client RPCs, vGPU plugin RPCs). Examples observed from a stock 570.144
  guest: `SET_GUEST_SYSTEM_INFO` (1), `FREE` (10), `UNLOADING_GUEST_DRIVER`
  (47), `GET_GSP_STATIC_INFO` (65), `UPDATE_BAR_PDE` (70),
  `CONTINUATION_RECORD` (71), `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73),
  `GSP_RM_CONTROL` (76), `GSP_RM_ALLOC` (103).
* Timing matters: a reply must be matched to the request sequence number;
  the driver runs a reply-matching state machine, and a slow reply is
  indistinguishable from a hang.

## 4. RMCore, RM API and the resource server

Three concepts from the RM show up constantly in traces and in the guest's
driver. Understanding them makes every log readable.

### 4.1 RMCore

RMCore is the object system. Everything in the RM is an **object** with:

* a **class** (a 32-bit class ID, e.g. `0x2080` = `NV20_SUBDEVICE_0`,
  `0x906f`-family = channels, `0xc46f` = `TURING_CHANNEL_GPFIFO_A`),
* a **handle** allocated in a per-client handle space (e.g. `0xc1d00002`),
* a **parent** object (forming a tree rooted at a client).

Traces of `GSP_RM_ALLOC` are therefore traces of object creation, and traces
of `GSP_RM_CONTROL` are method calls on those objects.

### 4.2 RM API

The **RM API** is how in-kernel components (and user space via ioctls) allocate
objects and call controls. Two important variants exist in the client:

* `Alloc` — allocates an object with an implicit handle,
* `AllocWithHandle` — the caller chooses the handle.

For a GSP client, the *kernel-internal* API table is overridden so that some
calls become RPCs while others are handled locally. Knowing which is which is
the difference between "we see it in the trace" and "we must derive it another
way". Notably, memory objects are allocated **locally by the client RM** (the
pages are guest RAM), while *server-owned* ranges (see §5) are managed by the
GSP.

### 4.3 Resource server (`resserv`)

The resource server is the allocation authority: it owns the handle spaces,
enforces privileges, and tracks object lifetimes. Its directives include flags
that matter to us, for example `RMAPI_ALLOC_FLAGS_SERIALIZED` (the parameters
in an RPC are a serialized form that must be deserialized by the receiver) and
`RMAPI_ALLOC_FLAGS_SKIP_RPC` (the allocation is handled locally and never
reaches the server).

## 5. The client/server split (and why page tables are ours)

Modern RM has a **client/server** architecture:

* The **client** (guest `nvidia.ko`) owns per-process state and submits work.
* The **server** (GSP-RM on the GPU, under our emulation) owns GPU-global
  state: routing, page tables, scheduling domains, engine configuration.

For virtualized/partitioned setups, the client tells the server which VA
ranges it wants the *server* to own, and the server manages the deeper page
table levels for those ranges. We receive this contract explicitly in the
`NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` control, which carries the
**physical addresses of the client's page-table levels** plus the VA range and
page shifts they cover:

```
levels[0] phys=0x1297ac000  pageShift=47   (root page directory)
levels[1] phys=0x1297ad000  pageShift=38
levels[2] phys=0x1297ae000  pageShift=29
```

In a real system, the server then *fills in* the lower levels for
server-owned ranges; the client never writes them. This is a central fact for
the forwarding work: **when the guest's UVM asks the "GPU" to map memory, the
mapping is expected to materialize in page tables that only we can write.**

## 6. Event channels and notifications

Beyond request/reply, the guest needs asynchronous events:

* **Event objects** (`NV01_EVENT_*` classes) are created by the client and
  registered with a *notification* configuration: MSI-X vector, or a memory
  "semaphore" the GPU can write.
* **Completion events** are delivered when objects change state (channels
  finish work, errors occur).
* `cl2080_notification.h` documents the notification layout; the emulation
  must deliver exactly the bit/handle/state triple the guest expects, on the
  exact vector.

Our MSI-X path is: emulation thread → `eventfd` → KVM → guest interrupt
vector. The PBA (pending bit array) is emulated so a masked vector's interrupt
is not lost.

## 7. What we emulate today, precisely

| Layer | Emulated | Notes |
|---|---|---|
| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied |
| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer |
| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload |
| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is now complete (the async copy engine is included) and agrees with the class list |
| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected |
| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked |
| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work |
| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands |
| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs |

The engine-inventory row was the CUDA blocker at the time of writing:
`libcuda` filters the engine list it receives against the class list and
expects the async copy engine to be present, because the secondary (async)
engine record is otherwise zeroed. That inventory is now complete; the
context-creation wait is later, in the completion path. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the offsets and the consumed fields, and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
section 9 for the emulation checklist.

## 8. Where the firmware analysis lives

* `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was
  decompiled, RPC dispatchers located, jump tables mapped.
* `wiki/Address-Reference.md` — firmware addresses for structures, functions
  and tables.
* `wiki/RPC-Message-Format.md`, `wiki/GSP-RPC-Emulation.md` — protocol details.
* `wiki/WPR-Management.md` — write-protected region semantics during boot.
* `kernel/gsp_rpc_client.c` — the firmware-derived handlers expressed as C.