diff options
Diffstat (limited to 'wiki/System-Overview.md')
| -rw-r--r-- | wiki/System-Overview.md | 152 |
1 files changed, 152 insertions, 0 deletions
diff --git a/wiki/System-Overview.md b/wiki/System-Overview.md new file mode 100644 index 0000000..3f2d3e2 --- /dev/null +++ b/wiki/System-Overview.md @@ -0,0 +1,152 @@ +# System Overview + +> **What this page is.** The one page to read if you want to understand *how +> Open-IOV works* before diving into any subsystem. Condensed from +> [`../docs/03-architecture/01-system-overview.md`](../docs/03-architecture/01-system-overview.md) +> (with [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md) +> and [`../docs/03-architecture/10-interaction-and-scheduling.md`](../docs/03-architecture/10-interaction-and-scheduling.md) +> for the full picture); the docs tree wins on disagreement. + +Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +stock guest driver; `[gated]` implemented default-off; `[wip]` partial; +`[wall]` current blocker; `[target]` designed end state. + +## 1. The core principle + +**Emulate the interface. Pass through the work. Never patch anyone.** + +A virtual GPU is a *mediated passthrough* device. The guest must believe it +owns a real NVIDIA GPU with a real GPU System Processor running firmware; the +host must arrange for the guest's work to execute on real hardware, because +emulating compute in software is orders of magnitude too slow to be useful. + +```text +EMULATE (software; the guest expects virtual hardware) + PCI configuration space, capability structures, power management + BAR0 register interface (shadow, not the physical GPU's BAR0) + BAR1 VRAM aperture (guest-visible video memory) + GSP firmware boot handshake (FWSEC -> booter -> RISCV -> boot complete) + GSP-RM message queues, RPC transport, resource-manager controls + event/notification channels; MSI-X injection (via eventfd) + +PASS THROUGH (real hardware; correctness and performance) + GPU memory: guest VRAM objects backed by real VRAM [target] + page tables / GMMU state used by the GPU [wip] + channel execution: guest push buffers on real channels [gated/wall] + compute (CUDA) and, later, 3D/video work [wip/planned] + DMA, completion and fault interrupts relayed to the guest +``` + +The line between the two is the **interface boundary**, and it moves as the +project matures: early work emulates everything the guest can observe; later +work forwards everything that must be fast. + +## 2. Layer map + +```text +GUEST + application ── CUDA runtime (libcudart) ─┐ + application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack + OpenGL/Vulkan ── libGLX/libvulkan ──────┘ + │ ioctls / mmaps + nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm + │ MMIO + DMA + MSI-X (a virtual PCIe device) +════════╪════════════════════════════════════════════════════════ +HYPERVISOR (KVM/QEMU, later Xen/...) VFIO mdev transport + IOMMU + │ +HOST MODULE vgsp.ko + ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X + ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) + ├─ memory model: guest RAM access, VA<->PA maps, VRAM backing + └─ work path: channel/push forwarding to the physical GPU + │ module-owned device + real GSP session +HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module + │ (the distribution nvidia.ko stays unmodified and unbound in + │ this mode) + v +REAL HARDWARE (engines, memory, interrupts) +``` + +The guest never contains Open-IOV code; the host driver is unmodified; the +hypervisor is unmodified. The only new component is `vgsp.ko`. + +## 3. How each side works + +* **Guest side.** The stock driver performs the GSP boot handshake, sets up + RPC message queues in guest memory, creates RM objects, programs channels + and submits work — exactly as on bare metal + ([`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md), + [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md), + [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)). +* **Host module.** `vgsp.ko` is a VFIO mediated-device driver: it registers a + parent on the physical GPU, exposes virtual GPU types, and each mdev is one + virtual GPU for one VM. Inside, it presents a plausible Turing-class device + (config space, BAR shadows, MSI-X), answers the GSP boot handshake and the + RPC protocol with well-formed replies, and maintains the memory model that + resolves guest physical addresses to host addresses + ([`Host-Module.md`](Host-Module.md)). +* **The interface that makes this possible.** Since Turing, the driver's + management path is not *silicon*, it is *firmware*: the GSP runs a version of + NVIDIA's Resource Manager and the driver talks to it over shared-memory + queues with a small RPC protocol. That protocol is small, stable and + observable — so the project emulates **the firmware's interface** instead of + emulating a GPU ([`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md), + [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md)). +* **Execution.** Guest work that must be fast (copies, kernels) is forwarded + to a real pipeline driven through the module's own GSP-RM session + ([`Execution-Plane.md`](Execution-Plane.md), + [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), + [`Execution-Frontier.md`](Execution-Frontier.md)). + +## 4. Data flows + +```text +MMIO from the guest: + guest nvidia.ko writes BAR0 offset X + -> KVM exit -> QEMU -> VFIO region write into the mdev + -> BAR0 shadow state machine (doorbells, VF registers, notify) + -> may wake the emulation thread + +RPC from the guest to "firmware": + guest writes a command-queue element into a guest-memory msgq + -> doorbell (BAR0) + -> emulation thread reads the queue through the hypervisor's guest-RAM + accessor, parses the element, dispatches the RPC + -> handler produces a reply element in the status queue + -> emulation injects an MSI-X interrupt to the guest +``` + +## 5. What runs today + +1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`. +2. The GSP boot handshake completes against the emulation. +3. Message queues are set up, versions negotiated, RM initialization + exchanges complete; object creation, capability queries, vaspaces, + channels and the engine inventory are answered with well-formed replies + (a small set of control IDs is special-cased with real values; reply + fidelity for consumed fields keeps improving). +4. `nvidia-smi -L` enumerates the device; CUDA userspace starts; context + creation completes; the full driver-API smoke passes (`ALL OK`, 64-byte + payload round-trips). +5. CE-class copies are forwarded to the physical copy engine (default off) + with engine-authored semaphore releases and a fault relay. For GR + (graphics/compute), a forwarded guest launch reaches the front-end but the + engine has not executed it yet — the current frontier + ([`Execution-Frontier.md`](Execution-Frontier.md)). + +## 6. Where the detail lives + +* [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the firmware/RM architecture and + the transport we emulate. +* [`Host-Module.md`](Host-Module.md) — the module's internals. +* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md), + [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the guest side. +* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md), + [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md) — memory and translation. +* [`Execution-Plane.md`](Execution-Plane.md), + [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), + [`Execution-Frontier.md`](Execution-Frontier.md) — work and execution. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — supported + combinations and portability. +* [`../docs/03-architecture/01-system-overview.md`](../docs/03-architecture/01-system-overview.md) + and the rest of the architecture corpus in the repository. |