From 68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- docs/03-architecture/01-system-overview.md | 183 +++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 docs/03-architecture/01-system-overview.md (limited to 'docs/03-architecture/01-system-overview.md') diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md new file mode 100644 index 0000000..ed9b24b --- /dev/null +++ b/docs/03-architecture/01-system-overview.md @@ -0,0 +1,183 @@ +# System Overview + +> The one document to read if you want to understand *how Open-IOV works* before +> diving into any subsystem. + +## 1. The core architectural principle + +> **Emulate the interface. Pass through the work.** + +A virtual GPU is a *mediated passthrough* device. The guest must believe it +owns a real NVIDIA GPU with a real GPU System Processor (GSP) running NVIDIA's +firmware. The host must arrange for the guest's work to execute on real +hardware, because emulating compute in software is 100–1000× too slow to be +useful. + +Everything therefore falls into two buckets: + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ EMULATE (software; guest expects virtual hardware) │ +│ • PCI configuration space, capability structures, power management │ +│ • BAR0 register interface (shadow, not the physical GPU's BAR0) │ +│ • BAR1 VRAM aperture (guest-visible video memory) │ +│ • Instance/PRAMIN window │ +│ • GSP firmware boot handshake (FWSEC → booter → RISCV → boot complete)│ +│ • GSP-RM message queues, RPC transport, resource-manager controls │ +│ • Event/notification channels │ +│ • MSI-X injection (via eventfd, from the hypervisor's perspective) │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────┐ +│ PASS THROUGH (real hardware; for correctness and performance) │ +│ • GPU memory: guest VRAM objects backed by real VRAM (goal) │ +│ • Page tables / GMMU state used by the GPU (goal) │ +│ • Channel execution: the guest's push buffers run on real channels │ +│ • Compute (CUDA) and, later, 3D/video work │ +│ • DMA, completion and fault interrupts relayed back to the guest │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +The line between the two is the **interface boundary**, and it moves as the +project matures: early work emulates more (everything the guest can observe), +later work forwards more (everything that must be fast). + +## 2. Layer map + +``` +GUEST + application ── CUDA runtime (libcudart) ─┐ + application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack + OpenGL/Vulkan ── libGLX/libvulkan ──────┘ + │ ioctls / mmaps + nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm + │ MMIO + DMA + MSI-X (a virtual PCIe device) +════════╪════════════════════════════════════════════════════════════════ +HYPERVISOR (KVM/Xen/...) VFIO mdev transport + IOMMU isolation + │ +HOST MODULE open-iov-vfio.ko + ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X + ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) + ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing + └─ work path: channel/push forwarding to the physical GPU [in progress] + │ +HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces + │ +PHYSICAL GPU +``` + +## 3. Host module architecture + +`open-iov-vfio.ko` is a **VFIO mediated-device (mdev) driver**. It registers a +parent on the physical GPU and exposes virtual GPU types; each `mdev` instance +is one virtual GPU for one VM. + +| Subsystem | Files (illustrative) | Responsibility | +|---|---|---| +| mdev/VFIO registration | `kernel/main.c` | parent/type registration, VFIO device, region and IRQ setup | +| Config space | `kernel/main.c`, `kernel/bar_init.c` | identify/command/status, BARs, PCIe caps, MSI-X caps, VF windows | +| BAR0 shadow | `kernel/bar_emulation.c` | 16 MB emulated register space; register reads/writes the guest driver depends on | +| VRAM aperture (BAR1) | `kernel/main.c`, `memory.c` | 256 MB guest-visible video memory (host RAM today; real VRAM later) | +| Instance memory (BAR3) | `kernel/bar_emulation.c` | instance/PRAMIN window behaviour | +| GSP emulation thread | `kernel/gsp_emulation.c` | boot stage completion, doorbell polling, MSI-X injection | +| GSP-RM emulation | `kernel/gsp_rm_emulation.c` | message queues, RPC dispatch, resource-manager controls, objects | +| RPC handlers (firmware-derived) | `kernel/gsp_rpc_client.c` | behaviours decompiled from the GSP firmware | +| Firmware analysis (runtime) | `kernel/firmware_analyzer.c` | CRC/SHA signatures, pattern matching, version identification | +| Version dispatch | `kernel/dynamic_rpc_dispatcher.c`, `kernel/version/` | version-aware handler selection | +| Hardware abstraction | `kernel/architecture/` | per-architecture constants and behaviours | +| Userspace library | `userspace/lib/libvgpu.c` | management API (`vgpu_*`), diagnostics | + +**Data flow for an MMIO access from the guest:** + +``` +guest nvidia.ko writes BAR0 offset X + → KVM exit → QEMU → VFIO region write into the mdev + → nv_vgpu_vfio_dev_write() (or directly into the BAR0 shadow if mmap'd) + → BAR0 shadow state machine (doorbells, VF registers, notify) + → may wake the GSP emulation thread +``` + +**Data flow for an RPC from the guest to "firmware":** + +``` +guest writes a command-queue element into a guest-memory msgq and rings the +GSP doorbell (BAR0) + → emulation thread reads the queue through the hypervisor's guest-RAM + accessor (vfio_dma_rw), parses the element, dispatches the RPC + → handler produces a reply element in the status queue + → emulation injects an MSI-X interrupt to the guest +``` + +## 4. Guest stack summary + +* `nvidia.ko` is the **unmodified** NVIDIA kernel driver. It contains the + resource manager (RM) core, the resource server (`resserv`), the memory + manager, the FIFO/runlist code, the GPU System Processor client path, and + UVM (the unified memory/virtual-memory module, `nvidia_uvm.ko`). +* User space consists of the CUDA driver (`libcuda.so`), the CUDA runtime + (`libcudart.so`), plus graphics/video libraries. The family of NVIDIA + *software* components that mediate between applications and the driver is + referred to in this project as **CUDASW**. +* The guest expects to program a real GPU: it writes registers, allocates + objects, creates channels, submits push buffers and waits on semaphores — + exactly as on bare metal. + +Detailed subsystem documents: + +* [`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md) +* [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) +* [`04-uvm.md`](04-uvm.md) +* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) +* [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) +* [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md) + +## 5. Why a GSP emulation is possible at all + +The guest driver never talks to GPU silicon directly for management: since the +Turing generation, NVIDIA GPUs boot a **RISC-V firmware (the GSP)** that hosts +the Resource Manager. The driver's normal path is to talk to that firmware over +shared memory queues using a small RPC protocol. + +That protocol is: + +* **small** (a few dozen message types at the levels we need), +* **stable** (message layouts are versioned and compiled into the driver), and +* **observable** (everything arrives in guest memory we can read). + +So instead of emulating a GPU, we emulate **the firmware's interface**. The +guest driver is satisfied because the protocol is satisfied; the guest's +*work* is then given to real hardware. + +The firmware itself we study directly: it is a 28 MB RV64IMAC ELF that we have +decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and +`wiki/Address-Reference.md`). + +## 6. What runs today + +1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`. +2. The driver performs the GSP boot handshake with our emulation and receives + plausible firmware identity/configuration. +3. The driver sets up the RPC message queues (guest sysmem), negotiates + versions, and performs the resource-manager initialization exchanges. +4. Object creation (clients, devices, subdevices), capability queries, + vaspaces, channels, CE caps, GMMU static info — every control the driver + probes — is answered with the correct shape and values. +5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and + page-tree initialization. + +## 7. What is next (and why) + +CUDA kernel execution requires the guest's *work* to run on the physical GPU. +That means the host module must: + +1. know the memory the guest's GPU work refers to (the VA↔PA model), +2. reproduce or forward the channel/page-table state that the guest expects the + GSP to own, and +3. submit the guest's compute work to real engine(s) and relay completions, + faults and semaphores back. + +The design for this, including what is known and what remains to be learned, is +in: +[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) +and +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md). -- cgit v1.2.3