diff options
Diffstat (limited to 'wiki/GSP-and-GSP-RM.md')
| -rw-r--r-- | wiki/GSP-and-GSP-RM.md | 159 |
1 files changed, 159 insertions, 0 deletions
diff --git a/wiki/GSP-and-GSP-RM.md b/wiki/GSP-and-GSP-RM.md new file mode 100644 index 0000000..c714612 --- /dev/null +++ b/wiki/GSP-and-GSP-RM.md @@ -0,0 +1,159 @@ +# GSP and GSP-RM + +> **What this page is.** The GPU System Processor hosts NVIDIA's Resource +> Manager in firmware; since Turing, the guest driver's management path is +> *talking to that firmware*, not to silicon. This page explains that +> architecture and the interface the project emulates. Condensed from +> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md); +> the docs tree wins on disagreement. + +Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` +designed end state. + +## 1. What the GSP is + +The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA +GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) +that contains, among other things, a full port of NVIDIA's **Resource Manager +(RM)** — the component that owns GPU initialization, memory, engines, +interrupts and scheduling. + +```text + ┌──────────────────────────── GUEST ────────────────────────────┐ + │ nvidia.ko │ + │ RM client path ── RPC ──┐ │ + │ GSP boot code │ │ + └────────────────────────────┼──────────────────────────────────┘ + │ shared memory queues + doorbells + ┌────────────────────────────┼──────────────────────────────────┐ + │ GSP (firmware, RISC-V) │ │ + │ RM core (RMCore) ◄─────┘ │ + │ RM API (object model, clients, handles) │ + │ Resource server (resserv): allocation & lifetime │ + │ memory manager, GMMU, FIFO/runlist, interrupts, PMU │ + └───────────────────────────────────────────────────────────────┘ +``` + +A driver as complex as NVIDIA's is not written to program silicon directly; +it is written to talk to an RM, and on modern GPUs a copy of the RM runs on +the GSP. The driver is therefore an **RM client**: it issues requests, +receives callbacks, and observes state through a well-defined protocol. + +## 2. The boot handshake we emulate + +The guest driver boots the GSP exactly as it would on bare metal: + +1. **WPR setup** — carve a firmware-protected region of VRAM and write + descriptors into BAR0 registers ([`WPR-Management.md`](WPR-Management.md)). +2. **FWSEC** — the driver verifies firmware signatures. +3. **Booter** — the boot ROM consumes the firmware image. +4. **RISCV start** — the RISC-V core begins executing the firmware. +5. **GSP init done** — the firmware signals readiness through a status + mailbox; the driver proceeds to initialization RPCs. + +The emulation implements **stage completion** rather than the stages +themselves — legitimate because the only observable to the driver is the state +machine it polls. The same handshake has also been run against real firmware +on the physical backend: the firmware comes up, delivers its first status +events, serves the static configuration, and stays live until the unload RPC +releases it ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). + +## 3. The transport: message queues and RPC + +GSP-RM's transport is a shared-memory message queue pair: + +```text + guest system memory (allocated by the guest driver, registered with us) + ┌───────────────────────────────┐ + │ command queue (guest -> GSP) │ element ring, 4 KB elements + ├───────────────────────────────┤ + │ status queue (GSP -> guest) │ element ring + └───────────────────────────────┘ + + a page table describing the queue pages (the buffer is + non-contiguous: each page is an independent allocation) +``` + +Each element begins with a transport header followed by an **RPC message** +whose 32-byte header carries header version, length, function, result and +sequence number. Observations worth knowing (all verified against the stock +guest): + +* `length` covers the fixed part of the message, not variable payloads — do + not use it to bound a payload; +* function numbers are a flat enum shared by several protocols (`SET_GUEST_SYSTEM_INFO` 1, + `FREE` 10, `GET_GSP_STATIC_INFO` 65, `GSP_RM_CONTROL` 76, `GSP_RM_ALLOC` 103, ...); +* replies must be matched to request sequence numbers — the driver runs a + reply-matching state machine, and a slow reply is indistinguishable from a + hang. + +## 4. RMCore, the RM API and the resource server + +Three RM concepts make every log readable: + +* **RMCore** — the object system. Everything is an object with a class (a + 32-bit class ID), a handle in a per-client handle space, and a parent + object forming a tree rooted at a client. Traces of `GSP_RM_ALLOC` are + object creation; traces of `GSP_RM_CONTROL` are method calls on objects. +* **RM API** — how in-kernel components (and user space via ioctls) allocate + objects and call controls, with an implicit-handle and an explicit-handle + variant. For a GSP client, part of the API becomes RPCs while some calls + are handled locally; memory objects, for example, are allocated locally by + the client RM. +* **Resource server (resserv)** — the allocation authority: it owns handle + spaces, enforces privileges and tracks object lifetimes. + +## 5. The client/server split (and why page tables matter) + +Modern RM is client/server: the **client** (guest `nvidia.ko`) owns +per-process state and submits work; the **server** (GSP-RM) owns GPU-global +state — routing, page tables, scheduling domains, engine configuration. For +virtualized setups the client tells the server which VA ranges the server +owns, and the server manages the deeper page-table levels for those ranges. +The client hands over the **physical addresses of its page-table levels** +plus the VA ranges they cover; on real hardware the server then fills in the +lower levels for server-owned ranges and the client never writes them. + +This is a central fact for the execution work: when the guest's UVM asks the +"GPU" to map memory, the mapping is expected to materialize in page tables +that only the server side can write — which is us +([`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), +[`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)). + +## 6. Event channels and notifications + +Beyond request/reply, the guest needs asynchronous events: event objects +registered with a notification configuration (an MSI-X vector or a memory +"semaphore" the GPU can write), completion events when channels finish work or +errors occur, delivered as a strict bit/handle/state triple on the exact +vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM → +guest vector, with the pending-bit array emulated so a masked vector's +interrupt is not lost. + +## 7. What we emulate today + +| Layer | State | Notes | +|---|---|---| +| GSP boot state machine | `[live]` | stages complete; the driver's poll loop is satisfied | +| Message queue transport | `[live]` | including the non-contiguous page table for the queue buffer | +| RPC request/reply | `[live]` | every function the driver uses during init is answered; unknown functions get a well-formed success reply | +| Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list | +| Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving | +| Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet | +| Page tables for server-owned ranges | `[wip]` | contract received and recorded; leaf population is the execution work | +| Event/notification delivery | `[wip]` | MSI-X works; richer event semantics land with forwarding | +| Firmware behaviour (routing, scheduling, real execution) | `[wall]` | the execution frontier — see [`Execution-Frontier.md`](Execution-Frontier.md) | + +## 8. Where the firmware knowledge lives + +* [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) — + how the 28.5 MB image was decompiled and mapped. +* [`Address-Reference.md`](Address-Reference.md) — firmware-side address + tables. +* [`RPC-Message-Format.md`](RPC-Message-Format.md), + [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — historical RPC analysis + (superseded; kept for the record). +* [`WPR-Management.md`](WPR-Management.md) — write-protected region semantics. +* [The execution plane](Execution-Plane.md) and + [the execution frontier](Execution-Frontier.md) — what turns this protocol + into real work on real hardware. |