diff options
Diffstat (limited to 'wiki/GSP-and-GSP-RM.md')
| -rw-r--r-- | wiki/GSP-and-GSP-RM.md | 56 |
1 files changed, 51 insertions, 5 deletions
diff --git a/wiki/GSP-and-GSP-RM.md b/wiki/GSP-and-GSP-RM.md index c714612..651259b 100644 --- a/wiki/GSP-and-GSP-RM.md +++ b/wiki/GSP-and-GSP-RM.md @@ -7,7 +7,7 @@ > [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md); > the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` designed end state. @@ -40,6 +40,26 @@ it is written to talk to an RM, and on modern GPUs a copy of the RM runs on the GSP. The driver is therefore an **RM client**: it issues requests, receives callbacks, and observes state through a well-defined protocol. +### What the GSP does on bare metal + +* **It is the device's manager processor.** Since Turing, the driver does not + program most of the GPU. The GSP runs NVIDIA's signed firmware — a small + RTOS plus the RM server — and owns GPU-global state: initialisation, the + memory-management unit's deeper page-table levels, engine configuration, + context scheduling, interrupts and power/clock management (PMU). The + driver's kernel RM boots it, then becomes its client. +* **It boots from driver-provided memory.** At power-on the driver carves a + firmware-protected region of video memory (WPR2), copies the firmware image + there, has the device verify signatures (FWSEC), hands control to the boot + ROM ("booter"), which starts the RISC-V core; the firmware runs its RTOS and + RM initialisation and signals readiness. Everything the driver does + afterwards is a request to that server. +* **Why this matters for virtualization.** Because the management path is a + *firmware interface* — shared-memory queues plus a small RPC protocol, + not a pile of register lore — it is small, stable, and observable. That is + the interface Open-IOV emulates; it is the reason a virtual NVIDIA GPU is + practical at all. + ## 2. The boot handshake we emulate The guest driver boots the GSP exactly as it would on bare metal: @@ -130,7 +150,32 @@ vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM → guest vector, with the pending-bit array emulated so a masked vector's interrupt is not lost. -## 7. What we emulate today +## 7. How Open-IOV deals with the GSP — two modes + +The project handles the GSP in two ways, and they complement each other: + +1. **Guest-facing GSP emulation** (`vgsp.ko` *is* the virtual GSP). For each + mediated device, the module presents a device whose GSP behaviour is + software: it implements the boot state machine the driver polls (stage + completion, not silicon), answers the WPR/BAR0 register protocol, serves + the message-queue transport and every RPC verb the driver uses during + probe, initialisation and runtime, and delivers events via emulated MSI-X. + The guest's RM believes it is talking to firmware; it is talking to the + emulation. +2. **The real GSP as the backend server.** When the module owns the physical + GPU, it loads the *matching real firmware image* into the device, runs the + actual boot (FWSEC → booter → RISC-V → RM init-done), and becomes the RM + client itself — "be the host driver's RM" becomes "be the RM client". The + same protocol the guest uses one layer up is used here one layer down, and + it is what the work path submits through + ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). + +Keeping the two in sync is a project discipline: what the guest's RM is +promised (objects, controls, page tables, completions) must be answerable by +the emulation, and what executes on hardware must round-trip through the real +firmware session. + +### What is emulated today | Layer | State | Notes | |---|---|---| @@ -140,9 +185,10 @@ interrupt is not lost. | Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list | | Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving | | Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet | -| Page tables for server-owned ranges | `[wip]` | contract received and recorded; leaf population is the execution work | -| Event/notification delivery | `[wip]` | MSI-X works; richer event semantics land with forwarding | -| Firmware behaviour (routing, scheduling, real execution) | `[wall]` | the execution frontier — see [`Execution-Frontier.md`](Execution-Frontier.md) | +| Page tables for server-owned ranges | `[wip]` | contract received and recorded; forwarded paths run on module-built scratch tables while full server-owned leaf population is the remaining execution work | +| Event/notification delivery | `[live]` | MSI-X works; completion counters, notifiers and channel events are serviced and relayed to the guest | +| Real GSP as the backend server | `[live]` | the module boots the real firmware, serves its RM client ladder, and executes forwarded work through it | +| Firmware behaviour (routing, scheduling, real execution) | `[live]` | forwarded guest CUDA launches and graphics method streams execute on the physical engines — see [`Execution-Frontier.md`](Execution-Frontier.md) | ## 8. Where the firmware knowledge lives |