1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
|
# The Host Module and the Hybrid Model
> `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the
> memory model, and how work will be forwarded. This is the "where the code
> lives" document.
## 1. Role and position
```
guest VM
│ PCIe MMIO / DMA / MSI-X
┌────┴──────────────────────────────────────────────┐
│ hypervisor (KVM/QEMU or Xen) │
│ VFIO mdev transport, IOMMU (per-tenant memory) │
└────┬──────────────────────────────────────────────┘
│
┌────┴───────────────────────────────────────────────────────────────┐
│ open-iov-vfio.ko │
│ virtual device ── emulation core ── memory model ── work path │
└────┬─────────────────────────────────────┬─────────────────────────┘
│ host driver interfaces │ (future) vendor-neutral
┌────┴────────────────────┐ │
│ unmodified host nvidia.ko│ direct GPU interfaces / VF backends
└──────────────────────────┘
```
The module registers itself as an **mdev parent** on the physical GPU and
exposes one device type per supported profile. Each `mdev` instance is an
independent virtual GPU with its own state, its own BAR windows, its own
interrupts and its own emulation context.
## 2. Virtual device
### 2.1 PCI configuration space
The guest sees a plausible Turing-class GPU: vendor/device IDs, class code,
BARs, capability list (PCIe, MSI-X, power management), and the subsystem
identifiers the driver keys on. Several fields exist only because the driver
reads them for validation:
* **PCIe link capabilities** — the guest's driver derives bus width/speed and
availability. The emulation must report them; the *hypervisor* can
accidentally rewrite them (see the debugging doc's root-port note).
* **MSI-X table/PBA** — emulated, with per-vector masking and pending bits.
* **VF (virtual function) windows** — registers the SR-IOV-aware driver paths
read during initialization; the emulation fills them with values consistent
with a non-SR-IOV or lightly-SR-IOV configuration as appropriate.
### 2.2 BAR layout
| BAR | Size | Backing | Notes |
|---|---|---|---|
| BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS |
| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase |
| BAR3 | — | instance window | instance/PRAMIN behaviour |
MMIO reads and writes are served either by the module's VFIO region callbacks
or directly from the shadow when the hypervisor has mapped it into the guest.
### 2.3 VRAM backing
Today BAR1 is host RAM. This is deliberate: correctness first, performance and
real VRAM semantics later. Consequences and their mitigations:
* Guest reads/writes of "VRAM" are cheap and coherent. ✅
* The real GPU cannot read it. The forwarding phase must either migrate these
objects to real VRAM or expose a DMA-mapped system-memory region to the GPU
(see §5).
## 3. Emulation core
### 3.1 GSP emulation thread
A kernel thread drives the device's "firmware" behaviour:
1. Watches BAR0 doorbells (and VF registers) for guest actions.
2. Reads message-queue elements from guest memory.
3. Dispatches RPCs to handlers.
4. Writes replies into the status queue.
5. Injects MSI-X interrupts to the guest.
Guest memory is accessed through the hypervisor's DMA interface
(`vfio_dma_rw()`), which requires calls to originate from a kernel thread
context — a fact the debug tooling codifies.
### 3.2 RPC handling
Handlers are organized by function and, inside controls, by control ID. The
implementation philosophy:
* **Correct shape first.** A reply must have the exact structure the driver
expects, or the driver walks off into uninitialized memory.
* **Plausible values second.** Values must be self-consistent (e.g. FB size,
BAR sizes, engine counts, page sizes) or later code paths compute nonsense.
* **Never echo garbage.** Queries that pass uninitialized input must receive
zeroed/valid output; echoing the request's stack garbage has caused real
failures.
The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
"unknown control returns success with zeroed output" fallback.
### 3.3 Event channels and interrupts
Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor.
The emulation tracks vector masking and pending state so that an interrupt
raised while masked is delivered when unmasked, not lost.
## 4. Memory model (the heart of the project)
The guest's GPU work refers to *GPU virtual addresses*. To act on that work the
host module must translate them to memory it can touch.
### 4.1 What the client gives us
* **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical*
addresses: instance memory (VRAM offset), method buffer, userd, ramfc,
error notifiers. These are precise and trustworthy.
* **The page-table contract** for server-owned VA ranges
(`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page
shifts, VA range — but with *empty* leaf levels, because the server owns the
leaves.
* **No mapping RPCs for the client's own system-memory objects.** The client
allocates sysmem locally and never tells the server its physical pages.
### 4.2 What we derive instead
Because the client does not hand us its sysmem mappings, the module derives
them from the guest's own in-memory bookkeeping:
1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that
record, for every mapped object, both the GPU VA and the kernel CPU VA.
Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at
+0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`.
2. **CPU VAs are direct-map addresses**, so
`PA = cpu_va - page_offset_base`. The direct-map base can be recovered from
guest memory itself (structures that store their own address), from
`/proc/kallsyms` during development, or from a debug poke session.
3. Therefore a translation table can be built **without any cooperation from
the guest driver** — a crucial property for adversarial interoperability,
and a general technique applicable to future versions.
### 4.3 Acting on translations
The module can read and write guest physical memory (via the hypervisor's DMA
interface). That is enough to:
* complete semaphore releases (write the payload to the mapped slot),
* inspect push buffers and page tables,
* and later, create page tables for server-owned ranges.
## 5. The forwarding path (design)
```
guest push submitted
→ module detects submission (doorbell/GPPUT or push-buffer scan)
→ decode the push (method stream)
→ for each command decide:
(a) EMULATE: pure state/notification (semaphore release, membar, fence)
(b) FORWARD: real engine work (GR grid launch, CE copy, memop)
→ for (b): submit equivalent work on a real channel bound to the physical GPU
→ relay completions (semaphore/fault/interrupt) back into guest memory
```
Three forwarding backends are candidates, in increasing order of capability:
1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real
memory and create real channels, then replay guest work through them.
2. **Direct engine submission** — build our own channel/push streams to real
engines using the GMMU/HAL definitions, bypassing the host driver.
3. **Hybrid** — proxy for management (memory, channels) and direct submission
for hot-path work (grid launches, copies).
The decision between them is driven by the practical challenges document; the
short version is that (1) is safest for correctness, (2) is best for latency,
and both must be isolated to be acceptable.
## 6. Debug tooling in the module
The module exposes debug parameters (writes to
`/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the
emulated device inspectable without rebuilding:
| Parameter | Purpose |
|---|---|
| `scan_guest_ram=<v>` | find a 64-bit value in guest RAM (0–8 GB), report GPAs |
| `scan_vidmem=<v>` | same for the emulated VRAM shadow |
| `scan_seq=<v1,v2,...>` | find a 32-bit *sequence* (e.g. a method stream) |
| `scan_pt=<pa>` | find page-table entries pointing at a physical address |
| `scan_masked=<v>,<mask>` | find `(x & mask) == v` (e.g. a page-base VA) |
| `dump_vidmem=<off>,<len>` / `summary` | dump/scan the VRAM shadow |
| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory |
| `poke_fill=<gpa>,<v>,<n>` | fill a range |
| `fill_vidmem=<v>` | fill the VRAM shadow |
These are development-grade but production-useful: they are how the memory
model was reverse-engineered, and they remain the fastest way to diagnose a
mismatch. Their workflows are documented in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md).
## 7. Build and symbol contracts
* `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface
(`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for
hosts that want to pair our device with the NVIDIA OpenRM host modules.
* When NVIDIA's OpenRM modules are built from source, they must be built with
`KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the
Open-IOV module, or symbols will not resolve.
* On typical single-GPU developer systems the distribution's NVIDIA driver
manages the physical GPU and the Open-IOV module is used standalone; the
OpenRM submodule path is only needed for the native-vGPU-management
workflows.
## 8. What is implemented vs. designed
| Area | Status |
|---|---|
| mdev/VFIO device, config space, BARs, MSI-X | implemented |
| BAR0 register state machine, boot handshake | implemented |
| Message queues + RPC + all init controls observed from stock 570.144 | implemented |
| Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) |
| Push decoding + semaphore completion | demonstrated live; being productized |
| Page-table population for server-owned ranges | designed, not implemented |
| GPU work forwarding (GR/CE execution) | designed, not implemented |
| Real VRAM backing | designed, not implemented |
| Windows guest, Xen transport, aarch64 hosts | designed, not implemented |
|
