1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
|
# System Overview
> The one document to read if you want to understand *how Open-IOV works* before
> diving into any subsystem.
## 1. The core architectural principle
> **Emulate the interface. Pass through the work.**
A virtual GPU is a *mediated passthrough* device. The guest must believe it
owns a real NVIDIA GPU with a real GPU System Processor (GSP) running NVIDIA's
firmware. The host must arrange for the guest's work to execute on real
hardware, because emulating compute in software is 100–1000× too slow to be
useful.
Everything therefore falls into two buckets:
```
┌─────────────────────────────────────────────────────────────────────────┐
│ EMULATE (software; guest expects virtual hardware) │
│ • PCI configuration space, capability structures, power management │
│ • BAR0 register interface (shadow, not the physical GPU's BAR0) │
│ • BAR1 VRAM aperture (guest-visible video memory) │
│ • Instance/PRAMIN window │
│ • GSP firmware boot handshake (FWSEC → booter → RISCV → boot complete)│
│ • GSP-RM message queues, RPC transport, resource-manager controls │
│ • Event/notification channels │
│ • MSI-X injection (via eventfd, from the hypervisor's perspective) │
└─────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│ PASS THROUGH (real hardware; for correctness and performance) │
│ • GPU memory: guest VRAM objects backed by real VRAM (goal) │
│ • Page tables / GMMU state used by the GPU (goal) │
│ • Channel execution: guest push buffers on real GPU channels (goal; │
│ today the host software-executes decoded work — a bridge) │
│ • Compute (CUDA) and, later, 3D/video work │
│ • DMA, completion and fault interrupts relayed back to the guest │
└─────────────────────────────────────────────────────────────────────────┘
```
The line between the two is the **interface boundary**, and it moves as the
project matures: early work emulates more (everything the guest can observe),
later work forwards more (everything that must be fast).
## 2. Layer map
```
GUEST
application ── CUDA runtime (libcudart) ─┐
application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack
OpenGL/Vulkan ── libGLX/libvulkan ──────┘
│ ioctls / mmaps
nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm
│ MMIO + DMA + MSI-X (a virtual PCIe device)
════════╪════════════════════════════════════════════════════════════════
HYPERVISOR (KVM/Xen/...) VFIO mdev transport + IOMMU isolation
│
HOST MODULE open-iov-vfio.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
│ module-owned device + real GSP session
HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
│ (the distribution nvidia.ko stays unmodified and unbound in
│ this mode; see 08-physical-gpu-integration.md)
v
REAL HARDWARE (engines, memory, interrupts)
```
## 3. Host module architecture
`open-iov-vfio.ko` is a **VFIO mediated-device (mdev) driver**. It registers a
parent on the physical GPU and exposes virtual GPU types; each `mdev` instance
is one virtual GPU for one VM.
| Subsystem | Files (illustrative) | Responsibility |
|---|---|---|
| mdev/VFIO registration | `kernel/main.c` | parent/type registration, VFIO device, region and IRQ setup |
| Config space | `kernel/main.c`, `kernel/bar_init.c` | identify/command/status, BARs, PCIe caps, MSI-X caps, VF windows |
| BAR0 shadow | `kernel/bar_emulation.c` | 16 MB emulated register space; register reads/writes the guest driver depends on |
| VRAM aperture (BAR1) | `kernel/main.c`, `memory.c` | 256 MB guest-visible video memory (host RAM today; real VRAM later) |
| Instance memory (BAR3) | `kernel/bar_emulation.c` | instance/PRAMIN window behaviour |
| GSP emulation thread | `kernel/gsp_emulation.c` | boot stage completion, doorbell polling, MSI-X injection |
| GSP-RM emulation | `kernel/gsp_rm_emulation.c` | message queues, RPC dispatch, resource-manager controls, objects |
| RPC handlers (firmware-derived) | `kernel/gsp_rpc_client.c` | behaviours decompiled from the GSP firmware |
| Firmware analysis (runtime) | `kernel/firmware_analyzer.c` | CRC/SHA signatures, pattern matching, version identification |
| Version dispatch | `kernel/dynamic_rpc_dispatcher.c`, `kernel/version/` | version-aware handler selection |
| Hardware abstraction | `kernel/architecture/` | per-architecture constants and behaviours |
| Userspace library | `userspace/lib/libvgpu.c` | management API (`vgpu_*`), diagnostics |
**Data flow for an MMIO access from the guest:**
```
guest nvidia.ko writes BAR0 offset X
→ KVM exit → QEMU → VFIO region write into the mdev
→ nv_vgpu_vfio_dev_write() (or directly into the BAR0 shadow if mmap'd)
→ BAR0 shadow state machine (doorbells, VF registers, notify)
→ may wake the GSP emulation thread
```
**Data flow for an RPC from the guest to "firmware":**
```
guest writes a command-queue element into a guest-memory msgq and rings the
GSP doorbell (BAR0)
→ emulation thread reads the queue through the hypervisor's guest-RAM
accessor (vfio_dma_rw), parses the element, dispatches the RPC
→ handler produces a reply element in the status queue
→ emulation injects an MSI-X interrupt to the guest
```
## 4. Guest stack summary
* `nvidia.ko` is the **unmodified** NVIDIA kernel driver. It contains the
resource manager (RM) core, the resource server (`resserv`), the memory
manager, the FIFO/runlist code, the GPU System Processor client path, and
UVM (the unified memory/virtual-memory module, `nvidia_uvm.ko`).
* User space consists of the CUDA driver (`libcuda.so`), the CUDA runtime
(`libcudart.so`), plus graphics/video libraries. The family of NVIDIA
*software* components that mediate between applications and the driver is
referred to in this project as **CUDASW**.
* The guest expects to program a real GPU: it writes registers, allocates
objects, creates channels, submits push buffers and waits on semaphores —
exactly as on bare metal.
Detailed subsystem documents:
* [`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md)
* [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md)
* [`04-uvm.md`](04-uvm.md)
* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md)
* [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md)
* [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md)
## 5. Why a GSP emulation is possible at all
The guest driver never talks to GPU silicon directly for management: since the
Turing generation, NVIDIA GPUs boot a **RISC-V firmware (the GSP)** that hosts
the Resource Manager. The driver's normal path is to talk to that firmware over
shared memory queues using a small RPC protocol.
That protocol is:
* **small** (a few dozen message types at the levels we need),
* **stable** (message layouts are versioned and compiled into the driver), and
* **observable** (everything arrives in guest memory we can read).
So instead of emulating a GPU, we emulate **the firmware's interface**. The
guest driver is satisfied because the protocol is satisfied; the guest's
*work* is then given to real hardware.
The firmware itself we study directly: it is a 28 MB RV64IMAC ELF that we have
decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
`wiki/Address-Reference.md`).
## 6. What runs today
1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`.
2. The driver performs the GSP boot handshake with our emulation and receives
plausible firmware identity/configuration.
3. The driver sets up the RPC message queues (guest sysmem), negotiates
versions, and performs the resource-manager initialization exchanges.
4. Object creation (clients, devices, subdevices), capability queries,
vaspaces, channels, CE caps, GMMU static info — every control the driver
probes — is answered with a well-formed reply. Only a small set (23 control
IDs) is special-cased with real values; the rest fall back to generic
`SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params
without server-side object management. Where the driver consumes a reply
value, that value is still being corrected (the reply-fidelity ladder).
5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts
(`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches
`cuCtxCreate`, which currently ends in a forced `rc=999` on a completion
wait; the copy-engine method streams execute in software as a bridge, and
the execution backend is not built yet (see
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
The CUDA-side contract and the reply-fidelity rules are documented in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 7. What is next (and why)
CUDA kernel execution requires the guest's *work* to run on the physical GPU.
That means the host module must:
1. know the memory the guest's GPU work refers to (the VA↔PA model),
2. reproduce or forward the channel/page-table state that the guest expects the
GSP to own, and
3. submit the guest's compute work to real engine(s) and relay completions,
faults and semaphores back.
The design for this, including what is known and what remains to be learned, is
in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
and
[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md);
the chosen execution backend is specified in
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
|
