summaryrefslogtreecommitdiffstats
path: root/wiki/System-Overview.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
commitcff1461ef459e98dea063abc72ae0819fea7b5ce (patch)
treedbceb906db4bb6c2e0e621eed2515a06e701236d /wiki/System-Overview.md
parent40a01810c77838b3876ecea8fed2f5d72e08a378 (diff)
downloadopen-iov-cff1461ef459e98dea063abc72ae0819fea7b5ce.tar.gz
open-iov-cff1461ef459e98dea063abc72ae0819fea7b5ce.zip
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link mapping, local-marker scan, About-message insert), WikiBot publisher with diff/check modes and publish manifest, live site auditor, image uploader, site-only page sources (landing page, About, index, sidebar, banner templates, About-message template). - wiki: new public deep-dive sources (project status, install, FAQ, system overview, GSP and GSP-RM, execution frontier, host module, guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger, hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace status; corrected cross-document links; README rewritten for the new set. - wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/System-Overview.md')
-rw-r--r--wiki/System-Overview.md152
1 files changed, 152 insertions, 0 deletions
diff --git a/wiki/System-Overview.md b/wiki/System-Overview.md
new file mode 100644
index 0000000..3f2d3e2
--- /dev/null
+++ b/wiki/System-Overview.md
@@ -0,0 +1,152 @@
+# System Overview
+
+> **What this page is.** The one page to read if you want to understand *how
+> Open-IOV works* before diving into any subsystem. Condensed from
+> [`../docs/03-architecture/01-system-overview.md`](../docs/03-architecture/01-system-overview.md)
+> (with [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md)
+> and [`../docs/03-architecture/10-interaction-and-scheduling.md`](../docs/03-architecture/10-interaction-and-scheduling.md)
+> for the full picture); the docs tree wins on disagreement.
+
+Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
+stock guest driver; `[gated]` implemented default-off; `[wip]` partial;
+`[wall]` current blocker; `[target]` designed end state.
+
+## 1. The core principle
+
+**Emulate the interface. Pass through the work. Never patch anyone.**
+
+A virtual GPU is a *mediated passthrough* device. The guest must believe it
+owns a real NVIDIA GPU with a real GPU System Processor running firmware; the
+host must arrange for the guest's work to execute on real hardware, because
+emulating compute in software is orders of magnitude too slow to be useful.
+
+```text
+EMULATE (software; the guest expects virtual hardware)
+ PCI configuration space, capability structures, power management
+ BAR0 register interface (shadow, not the physical GPU's BAR0)
+ BAR1 VRAM aperture (guest-visible video memory)
+ GSP firmware boot handshake (FWSEC -> booter -> RISCV -> boot complete)
+ GSP-RM message queues, RPC transport, resource-manager controls
+ event/notification channels; MSI-X injection (via eventfd)
+
+PASS THROUGH (real hardware; correctness and performance)
+ GPU memory: guest VRAM objects backed by real VRAM [target]
+ page tables / GMMU state used by the GPU [wip]
+ channel execution: guest push buffers on real channels [gated/wall]
+ compute (CUDA) and, later, 3D/video work [wip/planned]
+ DMA, completion and fault interrupts relayed to the guest
+```
+
+The line between the two is the **interface boundary**, and it moves as the
+project matures: early work emulates everything the guest can observe; later
+work forwards everything that must be fast.
+
+## 2. Layer map
+
+```text
+GUEST
+ application ── CUDA runtime (libcudart) ─┐
+ application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack
+ OpenGL/Vulkan ── libGLX/libvulkan ──────┘
+ │ ioctls / mmaps
+ nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm
+ │ MMIO + DMA + MSI-X (a virtual PCIe device)
+════════╪════════════════════════════════════════════════════════
+HYPERVISOR (KVM/QEMU, later Xen/...) VFIO mdev transport + IOMMU
+ │
+HOST MODULE vgsp.ko
+ ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
+ ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
+ ├─ memory model: guest RAM access, VA<->PA maps, VRAM backing
+ └─ work path: channel/push forwarding to the physical GPU
+ │ module-owned device + real GSP session
+HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
+ │ (the distribution nvidia.ko stays unmodified and unbound in
+ │ this mode)
+ v
+REAL HARDWARE (engines, memory, interrupts)
+```
+
+The guest never contains Open-IOV code; the host driver is unmodified; the
+hypervisor is unmodified. The only new component is `vgsp.ko`.
+
+## 3. How each side works
+
+* **Guest side.** The stock driver performs the GSP boot handshake, sets up
+ RPC message queues in guest memory, creates RM objects, programs channels
+ and submits work — exactly as on bare metal
+ ([`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md),
+ [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md),
+ [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)).
+* **Host module.** `vgsp.ko` is a VFIO mediated-device driver: it registers a
+ parent on the physical GPU, exposes virtual GPU types, and each mdev is one
+ virtual GPU for one VM. Inside, it presents a plausible Turing-class device
+ (config space, BAR shadows, MSI-X), answers the GSP boot handshake and the
+ RPC protocol with well-formed replies, and maintains the memory model that
+ resolves guest physical addresses to host addresses
+ ([`Host-Module.md`](Host-Module.md)).
+* **The interface that makes this possible.** Since Turing, the driver's
+ management path is not *silicon*, it is *firmware*: the GSP runs a version of
+ NVIDIA's Resource Manager and the driver talks to it over shared-memory
+ queues with a small RPC protocol. That protocol is small, stable and
+ observable — so the project emulates **the firmware's interface** instead of
+ emulating a GPU ([`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md),
+ [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md)).
+* **Execution.** Guest work that must be fast (copies, kernels) is forwarded
+ to a real pipeline driven through the module's own GSP-RM session
+ ([`Execution-Plane.md`](Execution-Plane.md),
+ [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md),
+ [`Execution-Frontier.md`](Execution-Frontier.md)).
+
+## 4. Data flows
+
+```text
+MMIO from the guest:
+ guest nvidia.ko writes BAR0 offset X
+ -> KVM exit -> QEMU -> VFIO region write into the mdev
+ -> BAR0 shadow state machine (doorbells, VF registers, notify)
+ -> may wake the emulation thread
+
+RPC from the guest to "firmware":
+ guest writes a command-queue element into a guest-memory msgq
+ -> doorbell (BAR0)
+ -> emulation thread reads the queue through the hypervisor's guest-RAM
+ accessor, parses the element, dispatches the RPC
+ -> handler produces a reply element in the status queue
+ -> emulation injects an MSI-X interrupt to the guest
+```
+
+## 5. What runs today
+
+1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`.
+2. The GSP boot handshake completes against the emulation.
+3. Message queues are set up, versions negotiated, RM initialization
+ exchanges complete; object creation, capability queries, vaspaces,
+ channels and the engine inventory are answered with well-formed replies
+ (a small set of control IDs is special-cased with real values; reply
+ fidelity for consumed fields keeps improving).
+4. `nvidia-smi -L` enumerates the device; CUDA userspace starts; context
+ creation completes; the full driver-API smoke passes (`ALL OK`, 64-byte
+ payload round-trips).
+5. CE-class copies are forwarded to the physical copy engine (default off)
+ with engine-authored semaphore releases and a fault relay. For GR
+ (graphics/compute), a forwarded guest launch reaches the front-end but the
+ engine has not executed it yet — the current frontier
+ ([`Execution-Frontier.md`](Execution-Frontier.md)).
+
+## 6. Where the detail lives
+
+* [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the firmware/RM architecture and
+ the transport we emulate.
+* [`Host-Module.md`](Host-Module.md) — the module's internals.
+* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md),
+ [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the guest side.
+* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md),
+ [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md) — memory and translation.
+* [`Execution-Plane.md`](Execution-Plane.md),
+ [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md),
+ [`Execution-Frontier.md`](Execution-Frontier.md) — work and execution.
+* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — supported
+ combinations and portability.
+* [`../docs/03-architecture/01-system-overview.md`](../docs/03-architecture/01-system-overview.md)
+ and the rest of the architecture corpus in the repository.