diff options
Diffstat (limited to 'docs/03-architecture/03-gsp-and-gsp-rm.md')
| -rw-r--r-- | docs/03-architecture/03-gsp-and-gsp-rm.md | 204 |
1 files changed, 204 insertions, 0 deletions
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md new file mode 100644 index 0000000..20d335b --- /dev/null +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -0,0 +1,204 @@ +# GSP and GSP-RM + +> The GPU System Processor hosts NVIDIA's Resource Manager in firmware. Since +> Turing, the guest driver's management path is *talking to that firmware*, not +> to silicon. This document explains that architecture and the interface we +> emulate. + +## 1. What the GSP is + +The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs. +It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that +contains, among other things, a full port of NVIDIA's **Resource Manager (RM)** +— the component that owns GPU initialization, memory, engines, interrupts and +scheduling. + +Why this matters: a GPU driver as complex as NVIDIA's is not written to +program silicon directly. It is written to talk to an **RM**, and on modern +GPUs a *copy* of the RM runs on the GSP. The driver is then an **RM client**: +it issues requests, receives callbacks, and observes state through a +well-defined protocol. + +``` + ┌──────────────────────────── GUEST ────────────────────────────┐ + │ nvidia.ko │ + │ RM client path ── RPC ──┐ │ + │ GSP boot code │ │ + └────────────────────────────┼──────────────────────────────────┘ + │ shared memory queues + doorbells + ┌────────────────────────────┼──────────────────────────────────┐ + │ GSP (firmware, RISC-V) │ │ + │ RM core (RMCore) ◄─────┘ │ + │ RM API (object model, clients, handles) │ + │ Resource server (resserv): allocation & lifetime │ + │ Memory manager, GMMU, FIFO/runlist, interrupts, PMU │ + └───────────────────────────────────────────────────────────────┘ +``` + +## 2. GSP boot: the handshake we emulate + +The guest driver boots the GSP exactly as it would on bare metal: + +1. **WPR (Write-Protected Region) setup** — carve a protected region of VRAM + for firmware, write descriptors into BAR0 registers. +2. **FWSEC** — the driver verifies firmware signatures (our emulation answers + the microcode/falcon handshake plausibly). +3. **Booter** — the boot ROM/booter is released and consumes the firmware + image (we emulate completion). +4. **RISCV start** — the RISC-V core starts executing the firmware; on real + hardware the firmware then initializes. +5. **GSP init done** — the firmware signals readiness through a status mailbox; + the driver proceeds to initialization RPCs. + +Our emulation implements stage completion rather than the stages themselves — +which is legitimate because the *only observable* to the driver is the state +machine it polls. + +## 3. The transport: message queues and RPC + +GSP-RM's transport is a shared-memory message queue pair: + +``` + guest system memory (allocated by the guest driver, registered with us) + ┌───────────────────────────────┐ + │ command queue (guest→"GSP") │ 512 KB element ring, 4 KB elements + ├───────────────────────────────┤ + │ status queue ("GSP"→guest) │ 512 KB element ring + └───────────────────────────────┘ + + a page table describing the queue pages (the buffer is + NV_MEMORY_NONCONTIGUOUS: each 4 KB page is an independent allocation) +``` + +Each element begins with a transport header (`authTag`, `aad`, checksum, +sequence, element count) followed by an **RPC message**: + +``` +struct rpc_message_header_v { + u32 header_version; u32 signature; u32 length; u32 function; + u32 rpc_result; u32 rpc_result_private; u32 sequence; + union { u32 spare; u32 cpuRmGfid; }; +}; // 32 bytes; payload follows in a union of message structs +``` + +Observations worth knowing (all verified on live hardware): + +* `length` is `sizeof(header) + sizeof(fixed message struct)` — for messages + with variable payloads (e.g. `GSP_RM_ALLOC` with parameters) it does **not** + include the parameters. Do not use it to bound a payload. +* Function numbers are a flat enum shared by several protocols + (GSP-client RPCs, vGPU plugin RPCs). Examples observed from a stock 570.144 + guest: `SET_GUEST_SYSTEM_INFO` (1), `FREE` (10), `UNLOADING_GUEST_DRIVER` + (47), `GET_GSP_STATIC_INFO` (65), `UPDATE_BAR_PDE` (70), + `CONTINUATION_RECORD` (71), `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73), + `GSP_RM_CONTROL` (76), `GSP_RM_ALLOC` (103). +* Timing matters: a reply must be matched to the request sequence number; + the driver runs a reply-matching state machine, and a slow reply is + indistinguishable from a hang. + +## 4. RMCore, RM API and the resource server + +Three concepts from the RM show up constantly in traces and in the guest's +driver. Understanding them makes every log readable. + +### 4.1 RMCore + +RMCore is the object system. Everything in the RM is an **object** with: + +* a **class** (a 32-bit class ID, e.g. `0x2080` = `NV20_SUBDEVICE_0`, + `0x906f`-family = channels, `0xc46f` = `TURING_CHANNEL_GPFIFO_A`), +* a **handle** allocated in a per-client handle space (e.g. `0xc1d00002`), +* a **parent** object (forming a tree rooted at a client). + +Traces of `GSP_RM_ALLOC` are therefore traces of object creation, and traces +of `GSP_RM_CONTROL` are method calls on those objects. + +### 4.2 RM API + +The **RM API** is how in-kernel components (and user space via ioctls) allocate +objects and call controls. Two important variants exist in the client: + +* `Alloc` — allocates an object with an implicit handle, +* `AllocWithHandle` — the caller chooses the handle. + +For a GSP client, the *kernel-internal* API table is overridden so that some +calls become RPCs while others are handled locally. Knowing which is which is +the difference between "we see it in the trace" and "we must derive it another +way". Notably, memory objects are allocated **locally by the client RM** (the +pages are guest RAM), while *server-owned* ranges (see §5) are managed by the +GSP. + +### 4.3 Resource server (`resserv`) + +The resource server is the allocation authority: it owns the handle spaces, +enforces privileges, and tracks object lifetimes. Its directives include flags +that matter to us, for example `RMAPI_ALLOC_FLAGS_SERIALIZED` (the parameters +in an RPC are a serialized form that must be deserialized by the receiver) and +`RMAPI_ALLOC_FLAGS_SKIP_RPC` (the allocation is handled locally and never +reaches the server). + +## 5. The client/server split (and why page tables are ours) + +Modern RM has a **client/server** architecture: + +* The **client** (guest `nvidia.ko`) owns per-process state and submits work. +* The **server** (GSP-RM on the GPU, under our emulation) owns GPU-global + state: routing, page tables, scheduling domains, engine configuration. + +For virtualized/partitioned setups, the client tells the server which VA +ranges it wants the *server* to own, and the server manages the deeper page +table levels for those ranges. We receive this contract explicitly in the +`NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` control, which carries the +**physical addresses of the client's page-table levels** plus the VA range and +page shifts they cover: + +``` +levels[0] phys=0x1297ac000 pageShift=47 (root page directory) +levels[1] phys=0x1297ad000 pageShift=38 +levels[2] phys=0x1297ae000 pageShift=29 +``` + +In a real system, the server then *fills in* the lower levels for +server-owned ranges; the client never writes them. This is a central fact for +the forwarding work: **when the guest's UVM asks the "GPU" to map memory, the +mapping is expected to materialize in page tables that only we can write.** + +## 6. Event channels and notifications + +Beyond request/reply, the guest needs asynchronous events: + +* **Event objects** (`NV01_EVENT_*` classes) are created by the client and + registered with a *notification* configuration: MSI-X vector, or a memory + "semaphore" the GPU can write. +* **Completion events** are delivered when objects change state (channels + finish work, errors occur). +* `cl2080_notification.h` documents the notification layout; the emulation + must deliver exactly the bit/handle/state triple the guest expects, on the + exact vector. + +Our MSI-X path is: emulation thread → `eventfd` → KVM → guest interrupt +vector. The PBA (pending bit array) is emulated so a masked vector's interrupt +is not lost. + +## 7. What we emulate today, precisely + +| Layer | Emulated | Notes | +|---|---|---| +| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | +| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | +| RPC request/reply | ✅ | all functions the driver uses during init are answered | +| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values | +| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values | +| Object creation | ✅ | correct handle/parameter shapes | +| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | +| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | +| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | + +## 8. Where the firmware analysis lives + +* `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was + decompiled, RPC dispatchers located, jump tables mapped. +* `wiki/Address-Reference.md` — firmware addresses for structures, functions + and tables. +* `wiki/RPC-Message-Format.md`, `wiki/GSP-RPC-Emulation.md` — protocol details. +* `wiki/WPR-Management.md` — write-protected region semantics during boot. +* `kernel/gsp_rpc_client.c` — the firmware-derived handlers expressed as C. |