1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
|
# System Overview
> **What this page is.** The one page to read if you want to understand *how
> Open-IOV works* before diving into any subsystem. Condensed from
> [`../docs/03-architecture/01-system-overview.md`](../docs/03-architecture/01-system-overview.md)
> (with [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md)
> and [`../docs/03-architecture/10-interaction-and-scheduling.md`](../docs/03-architecture/10-interaction-and-scheduling.md)
> for the full picture); the docs tree wins on disagreement.
Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented default-off; `[wip]` partial;
`[wall]` current blocker; `[target]` designed end state.
## 1. The core principle
**Emulate the interface. Pass through the work. Never patch anyone.**
A virtual GPU is a *mediated passthrough* device. The guest must believe it
owns a real NVIDIA GPU with a real GPU System Processor running firmware; the
host must arrange for the guest's work to execute on real hardware, because
emulating compute in software is orders of magnitude too slow to be useful.
```text
EMULATE (software; the guest expects virtual hardware)
PCI configuration space, capability structures, power management
BAR0 register interface (shadow, not the physical GPU's BAR0)
BAR1 VRAM aperture (guest-visible video memory)
GSP firmware boot handshake (FWSEC -> booter -> RISCV -> boot complete)
GSP-RM message queues, RPC transport, resource-manager controls
event/notification channels; MSI-X injection (via eventfd)
PASS THROUGH (real hardware; correctness and performance)
GPU memory: guest VRAM objects backed by real VRAM [target]
page tables / GMMU state used by the GPU [wip]
channel execution: guest push buffers on real channels [gated/wall]
compute (CUDA) and, later, 3D/video work [wip/planned]
DMA, completion and fault interrupts relayed to the guest
```
The line between the two is the **interface boundary**, and it moves as the
project matures: early work emulates everything the guest can observe; later
work forwards everything that must be fast.
## 2. Layer map
```text
GUEST
application ── CUDA runtime (libcudart) ─┐
application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack
OpenGL/Vulkan ── libGLX/libvulkan ──────┘
│ ioctls / mmaps
nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm
│ MMIO + DMA + MSI-X (a virtual PCIe device)
════════╪════════════════════════════════════════════════════════
HYPERVISOR (KVM/QEMU, later Xen/...) VFIO mdev transport + IOMMU
│
HOST MODULE vgsp.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA<->PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU
│ module-owned device + real GSP session
HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
│ (the distribution nvidia.ko stays unmodified and unbound in
│ this mode)
v
REAL HARDWARE (engines, memory, interrupts)
```
The guest never contains Open-IOV code; the host driver is unmodified; the
hypervisor is unmodified. The only new component is `vgsp.ko`.
## 3. How each side works
* **Guest side.** The stock driver performs the GSP boot handshake, sets up
RPC message queues in guest memory, creates RM objects, programs channels
and submits work — exactly as on bare metal
([`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md),
[`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md),
[`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)).
* **Host module.** `vgsp.ko` is a VFIO mediated-device driver: it registers a
parent on the physical GPU, exposes virtual GPU types, and each mdev is one
virtual GPU for one VM. Inside, it presents a plausible Turing-class device
(config space, BAR shadows, MSI-X), answers the GSP boot handshake and the
RPC protocol with well-formed replies, and maintains the memory model that
resolves guest physical addresses to host addresses
([`Host-Module.md`](Host-Module.md)).
* **The interface that makes this possible.** Since Turing, the driver's
management path is not *silicon*, it is *firmware*: the GSP runs a version of
NVIDIA's Resource Manager and the driver talks to it over shared-memory
queues with a small RPC protocol. That protocol is small, stable and
observable — so the project emulates **the firmware's interface** instead of
emulating a GPU ([`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md),
[`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md)).
* **Execution.** Guest work that must be fast (copies, kernels) is forwarded
to a real pipeline driven through the module's own GSP-RM session
([`Execution-Plane.md`](Execution-Plane.md),
[`Physical-GPU-Backend.md`](Physical-GPU-Backend.md),
[`Execution-Frontier.md`](Execution-Frontier.md)).
## 4. Data flows
```text
MMIO from the guest:
guest nvidia.ko writes BAR0 offset X
-> KVM exit -> QEMU -> VFIO region write into the mdev
-> BAR0 shadow state machine (doorbells, VF registers, notify)
-> may wake the emulation thread
RPC from the guest to "firmware":
guest writes a command-queue element into a guest-memory msgq
-> doorbell (BAR0)
-> emulation thread reads the queue through the hypervisor's guest-RAM
accessor, parses the element, dispatches the RPC
-> handler produces a reply element in the status queue
-> emulation injects an MSI-X interrupt to the guest
```
## 5. What runs today
1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`.
2. The GSP boot handshake completes against the emulation.
3. Message queues are set up, versions negotiated, RM initialization
exchanges complete; object creation, capability queries, vaspaces,
channels and the engine inventory are answered with well-formed replies
(a small set of control IDs is special-cased with real values; reply
fidelity for consumed fields keeps improving).
4. `nvidia-smi -L` enumerates the device; CUDA userspace starts; context
creation completes; the full driver-API smoke passes (`ALL OK`, 64-byte
payload round-trips).
5. CE-class copies are forwarded to the physical copy engine (default off)
with engine-authored semaphore releases and a fault relay. For GR
(graphics/compute), a forwarded guest launch reaches the front-end but the
engine has not executed it yet — the current frontier
([`Execution-Frontier.md`](Execution-Frontier.md)).
## 6. Where the detail lives
* [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the firmware/RM architecture and
the transport we emulate.
* [`Host-Module.md`](Host-Module.md) — the module's internals.
* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md),
[`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the guest side.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md),
[`UVM-in-the-Guest.md`](UVM-in-the-Guest.md) — memory and translation.
* [`Execution-Plane.md`](Execution-Plane.md),
[`Physical-GPU-Backend.md`](Physical-GPU-Backend.md),
[`Execution-Frontier.md`](Execution-Frontier.md) — work and execution.
* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — supported
combinations and portability.
* [`../docs/03-architecture/01-system-overview.md`](../docs/03-architecture/01-system-overview.md)
and the rest of the architecture corpus in the repository.
|
