1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
|
# Open-IOV vGPU-IO Refactored Architecture
## System Design from First Principles
### 1. Problem Statement
**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with full GPU functionality (CUDA, graphics, video encode/decode, AI/ML) on Turing GPUs (TU102/TU104) without requiring:
- Proprietary GRID host drivers
- Modified guest drivers
- Paravirtualization interfaces
- Guest-side Open-IOV code
**Constraints**:
- Guest runs stock `nvidia.ko` + `libcuda.so` + `libnvidia-ml.so` + `libnvidia-opencl.so`
- Host runs OpenRM `nvidia.ko` (for physical GPU management) + Open-IOV hypervisor components
- No guest kernel modifications, no guest userspace Open-IOV libraries
- High performance: near-native GPU utilization for compute and graphics workloads
- Support both Linux and Windows guests
### 2. Core Design Principle: Host-Side GSP Emulation
The fundamental insight: **The GSP (GPU System Processor) is a RISC-V microcontroller running firmware that implements 12 RPC types. We can emulate this firmware entirely in the host kernel.**
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ FIRST PRINCIPLE: GSP IS JUST AN RPC SERVER │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Physical Reality: │
│ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │
│ │ Guest RM │ ─────────────────────────────► │ GSP Firmware │ │
│ │ (nvidia.ko)│ ◄───────────────────────────── │ (570.144W, RV64IMAC) │ │
│ └─────────────┘ Responses └─────────────────────────┘ │
│ │ │ │
│ │ 12 RPC Types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, │ │
│ │ SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG │ │
│ │ │ │
│ ▼ │ │
│ Virtualized Reality: │ │
│ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │
│ │ Guest RM │ ─────────────────────────────► │ Host Kernel │ │
│ │ (nvidia.ko)│ ◄───────────────────────────── │ (open-iov-vfio.ko) │ │
│ └─────────────┘ Responses │ gsp_rpc_client.c │ │
│ │ │ (12 emulated handlers)│ │
│ │ Guest sees identical └─────────────────────────┘ │
│ │ interface - cannot tell │
│ ▼ │
│ No guest-side code required. │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
### 3. System Architecture Overview
```
╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ HOST SYSTEM ║
║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║
║ ║ USERSPACE ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ QEMU/KVM + libvfio-user │ ║ ║
║ ║ │ • -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID> │ ║ ║
║ ║ │ • VFIO region mmap, ioctl, eventfd for IRQs │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ║ ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ mdev management CLI / mdev management daemon (Management Tools) │ ║ ║
║ ║ │ • Link against: libvgpu.so (host-side) │ ║ ║
║ ║ │ • VF lifecycle: create/destroy, resource allocation, migration orchestration │ ║ ║
║ ║ │ • Calls: vgpu_open(), vgpu_vf_create(), vgpu_mem_alloc(), vgpu_bar_config(), │ ║ ║
║ ║ │ vgpu_wpr_protect(), vgpu_schedule(), vgpu_migrate_start(), etc. │ ║ ║
║ ║ │ • Communicates via VFIO ioctls on /dev/vfio/<mdev> │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ║ ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ libvgpu.so / libiov.so (HOST-SIDE ONLY) │ ║ ║
║ ║ │ • Full vGPU management API: vgpu_*() functions │ ║ ║
║ ║ │ • Implements: VF create/destroy, memory ops, BAR config, WPR, scheduling, │ ║ ║
║ ║ │ notifications, migration │ ║ ║
║ ║ │ • rpc_emulator.c: Host-side emulation fallbacks for management operations │ ║ ║
║ ║ │ (rpc_emu_bar_config/save/restore, rpc_emu_wpr_*, rpc_emu_schedule/yield, │ ║ ║
║ ║ │ rpc_emu_notify_register/unregister/send, rpc_emu_migrate_*) │ ║ ║
║ ║ │ • NO GSP RPC emulation here - that lives in kernel │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║
║ ║
║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║
║ ║ KERNEL SPACE ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ nvidia.ko (Host RM - OpenRM 570.144) │ ║ ║
║ ║ │ • Boots physical GSP firmware (570.144W) │ ║ ║
║ ║ │ • Manages physical GPU resources (VRAM, engines, display) │ ║ ║
║ ║ │ • Exports vGPU VFIO symbols: │ ║ ║
║ ║ │ - nvidia_vgpu_vfio_get_ops() / nvidia_vgpu_vfio_set_ops() │ ║ ║
║ ║ │ - nv_vgpu_mdev_register_parent() / nv_vgpu_mdev_unregister_parent() │ ║ ║
║ ║ │ • GPC callbacks from open-iov-vfio.ko: vgpu_create, vgpu_delete, vgpu_bar_info, │ ║ ║
║ ║ │ vgpu_start, get_name, get_description, get_instances, get_sparse_mmap, update_request │ ║ ║
║ ║ │ • Owns physical MSI-X interrupt │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ║ ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ open-iov-vfio.ko (HYPERVISOR - Core Component) │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ mdev Parent Registration │ │ ║ ║
║ ║ │ │ • Registers with nvidia.ko via nv_vgpu_mdev_register_parent() │ │ ║ ║
║ ║ │ │ • Creates mediated device types: nvidia-vgpu-turing, nvidia-vgpu-ampere, etc. │ │ ║ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ PCI Config Space Emulation (256 bytes per mdev) │ │ ║ ║
║ ║ │ │ • Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU), Class=0x030000 │ │ ║ ║
║ ║ │ │ • BAR0=16MB (64-bit, prefetchable), BAR1=256MB (64-bit, prefetchable) │ │ ║ ║
║ ║ │ │ • BAR2=32MB, BAR3=32MB (64-bit) │ │ ║ ║
║ ║ │ │ • Capabilities: PM (0x40), MSI (0x50), MSI-X (0x70), PCIe (0x80) │ │ ║ ║
║ ║ │ │ • MSI-X Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ │ ║ ║
║ ║ │ │ • VFIO_CONFIG_REGION_INDEX mapped to this emulated config space │ │ ║ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ BAR Region Emulation (VFIO Region Indices) │ │ ║ ║
║ ║ │ │ • VFIO_PCI_BAR0_REGION_INDEX → 16MB emulated config space │ │ ║ ║
║ ║ │ │ • VFIO_PCI_BAR1_REGION_INDEX → 256MB emulated framebuffer (backed by host VRAM allocation) │ │ ║ ║
║ ║ │ │ • VFIO_PCI_BAR2_REGION_INDEX → 32MB reserved │ │ ║ ║
║ ║ │ │ • VFIO_PCI_BAR3_REGION_INDEX → 32MB reserved │ │ ║ ║
║ ║ │ │ • VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table (0x2000) + PBA (0x3000) in BAR0 │ │ ║ ║
║ ║ │ │ • BAR trap handling: 64 trapped regions covering all BAR0 registers (bar_emulation.c) │ │ ║ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ GSP EMULATION ENGINE (NEW - Core Innovation) │ │ ║ ║
║ ║ │ │ │ │ ║ ║
║ ║ │ │ Per-mdev GSP Emulation Thread: │ │ ║ ║
║ ║ │ │ ```c │ │ ║ ║
║ ║ │ │ static int gsp_emu_thread(void *data) { │ │ ║ ║
║ ║ │ │ struct nv_vgpu_device *vgpu = data; │ │ ║ ║
║ ║ │ │ while (!kthread_should_stop()) { │ │ ║ ║
║ ║ │ │ u32 doorbell = readl(vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); │ │ ║ ║
║ ║ │ │ if (doorbell) { │ │ ║ ║
║ ║ │ │ writel(0, vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); // ACK │ │ ║ ║
║ ║ │ │ gsp_emu_process_rpc(vgpu); // Process via gsp_rpc_client.c │ │ ║ ║
║ ║ │ │ } │ │ ║ ║
║ ║ │ │ usleep_range(100, 200); // 1-2μs polling interval │ │ ║ ║
║ ║ │ │ } │ │ ║ ║
║ ║ │ │ return 0; │ │ ║ ║
║ ║ │ │ } │ │ ║ ║
║ ║ │ │ ``` │ │ ║ ║
║ ║ │ │ │ │ ║ ║
║ ║ │ │ GSP Shared Memory Management: │ │ ║ ║
║ ║ │ │ • Allocates contiguous host memory for GSP mailbox (4KB aligned) │ │ ║ ║
�b ║ │ │ • Maps into guest BAR0 at GSP_SHARED_MEM_OFFSET (0x1000) │ │ ║ ║
║ ║ │ │ • WC (Write-Combining) mapping for performance │ │ ║ ║
║ ║ │ │ • Cache coherency: clflush on write, memory barriers │ │ ║ ║
║ ║ │ │ • Matches firmware mailbox layout exactly (from decompilation) │ │ ║ ║
║ ║ │ │ │ │ ║ ║
║ ║ │ │ RPC Handler Suite (gsp_rpc_client.c - ALL 12 TYPES): │ │ ║ ║
║ ║ │ │ ┌─────────────┬──────────────────────────────────────────────────────────────────────────────────┐ │ │ ║
║ ║ │ │ │ RPC Type │ Emulated Implementation │ │ │ ║
║ ║ │ │ ├─────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ │ ║
║ ║ │ │ │ ALLOC (0x01)│ VRAM allocation via host nvidia.ko, returns GPU VA + handle │ │ │ ║
║ ║ │ │ │ FREE (0x02) │ Free allocation, unmap pages │ │ │ ║
║ ║ │ │ │ MAP (0x03) │ Map allocation to GPU VA, update page tables │ │ │ ║
║ ║ │ │ │ UNMAP (0x04)│ Unmap GPU VA range │ │ │ ║
║ ║ │ │ │ PIN (0x05) │ Pin pages for DMA, return DMA address │ │ │ ║
║ ║ │ │ │ UNPIN (0x06)│ Unpin pages │ │ │ ║
║ ║ │ │ │ SCHEDULE(0x07)│ Submit work to GSP scheduler (local workqueue) │ │ │ ║
║ ║ │ │ │ YIELD (0x08)│ Timeslice yield (no-op success) │ │ │ ║
║ ║ │ │ │ NOTIFY(0x09)│ Async notification registration, eventfd signaling │ │ │ ║
║ ║ │ │ │ BAR_TRAP(0x0A)│ BAR register read/write emulation via shadow memory │ │ │ ║
║ ║ │ │ │ MIGRATE(0x0B)│ 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP) │ │ │ ║
║ ║ │ │ │ DEBUG (0x0C)│ Debug interface, firmware introspection │ │ │ ║
║ ║ │ │ └─────────────┴──────────────────────────────────────────────────────────────────────────────────┘ │ │ ║
║ ║ │ │ │ │ ║
║ ║ │ │ Firmware Address Mapping (from decompilation): │ │ ║
║ ║ │ │ • Primary Dispatcher: 0x057A1AC0 (32 sub-commands) │ │ ║
║ ║ │ │ • Secondary Dispatcher: 0x04ED39DC │ │ ║
║ ║ │ │ • BAR Trap Handler: 0x04CB8188 │ │ ║
║ ║ │ │ • WPR Handlers: 0x04F00B0C, 0x04F00C78 │ │ ║
║ ║ │ │ • ALLOC Main: 0x057E94EC │ │ ║
║ ║ │ │ • SCHEDULE Main: 0x051EFBF4 │ │ ║
║ ║ │ │ • Migration Tables: 0x04179ED8, 0x04179FBC (16 entries each) │ │ ║
║ ║ │ │ • RPC Jump Table: 0x059BFEB0 (12 handlers) │ │ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ MSI-X Interrupt Injection │ │ ║ ║
║ ║ │ │ • When emulated GSP completes RPC → write to MSI-X PBA → eventfd_signal() │ │ ║ ║
║ ║ │ │ • QEMU/KVM delivers MSI-X interrupt to guest nvidia.ko ISR │ │ ║ ║
║ ║ │ │ • Guest sees normal GSP completion interrupt │ │ ║ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ Memory Management (memory.c) │ │ ║ ║
║ ║ │ │ • DMA mapping via IOMMU (VFIO DMA API) │ │ ║ ║
║ ║ │ │ • Pin/unpin for device DMA │ │ ║ ║
║ ║ │ │ • VRAM allocation via host nvidia.ko │ │ ║ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ │ │ ║ ║
║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║
║ ║ │ │ Live Migration (migration.c) │ │ ║ ║
║ ║ │ │ • 4-phase: QUERY → PREPARE → TRANSFER → COMMIT → CLEANUP │ │ ║ ║
║ ║ │ │ • State save/restore: GSP context, page tables, VRAM contents, BAR state │ │ ║ ║
║ ║ │ │ • Dual jump tables from firmware (0x04179ED8, 0x04179FBC) │ │ ║ ║
║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║
║ ║
║ PHYSICAL GPU: TU102/TU104 (RTX 2080 Ti / T4 / RTX 2080 Super) ║
║ • REAL GSP Firmware 570.144W running on FALCON RISC-V ║
║ • Used by host nvidia.ko for physical GPU management ║
║ • NOT used for guest vGPU RPCs (those are emulated in host kernel) ║
║ ║
╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
│ VFIO MEDIATED DEVICE (mdev) │
│ /sys/bus/mdev/devices/<UUID>/ │
▼
╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ GUEST VM ║
║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║
║ ║ USERSPACE ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ Guest Applications │ ║ ║
║ ║ │ • CUDA Applications (cuBLAS, cuDNN, custom kernels) │ ║ ║
║ ║ │ • Graphics: OpenGL, Vulkan, DirectX (via WINE/Proton on Linux, native on Windows) │ ║ ║
║ ║ │ • Video: NVENC/NVDEC (FFmpeg, GStreamer, Media Foundation) │ ║ ║
║ ║ │ • AI/ML: TensorRT, PyTorch, TensorFlow │ ║ ║
║ ║ │ • NO Open-IOV libraries, NO modified code │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ║ ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ Standard NVIDIA Userspace Drivers (UNMODIFIED) │ ║ ║
║ ║ │ • libcuda.so / libcuda.so.1 (CUDA Driver API) │ ║ ║
║ ║ │ • libnvidia-ml.so / libnvidia-ml.so.1 (NVML) │ ║ ║
║ ║ │ • libnvidia-opencl.so / libnvidia-opencl.so.1 (OpenCL) │ ║ ║
║ ║ │ • libnvidia-vulkan.so / libGLX_nvidia.so (Vulkan/OpenGL) │ ║ ║
║ ║ │ • libnvidia-encode.so / libnvidia-decode.so (Video) │ ║ ║
║ ║ │ • nvidia-smi, nvidia-settings, nvidia-debugdump │ ║ ║
║ ║ │ • ALL communicate with guest nvidia.ko via /dev/nvidia* ioctls │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║
║ ║
║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║
║ ║ KERNEL SPACE ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ nvidia.ko (Guest RM - UNMODIFIED OpenRM 570.144) │ ║ ║
║ ║ │ • PCI Probe: detects 10de:1ed3 (TU104 vGPU) from emulated config space │ ║ ║
║ ║ │ • BAR Mapping: │ ║ ║
║ ║ │ - BAR0 (16MB): Config registers, GSP doorbell @ 0x110c00, shared mem @ 0x1000 │ ║ ║
║ ║ │ - BAR1 (256MB): Framebuffer (VRAM) │ ║ ║
║ ║ │ - BAR2 (32MB), BAR3 (32MB): Reserved │ ║ ║
║ ║ │ • MSI-X Setup: Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ ║ ║
║ ║ │ • GSP Client Initialization: │ ║ ║
║ ║ │ - Writes to doorbell (queue head) to submit RPCs │ ║ ║
║ ║ │ - Reads/writes shared memory for RPC messages │ ║ ║
║ ║ │ - Waits for MSI-X interrupt on completion │ ║ ║
║ ║ │ • RM Subsystems (all functional): │ ║ ║
║ ║ │ - Memory: alloc, free, map, unmap, pin, unpin via GSP RPC │ ║ ║
║ ║ │ - Scheduling: work submission, yield, priority via GSP RPC │ ║ ║
║ ║ │ - BAR: register access via BAR_TRAP RPC │ ║ ║
║ ║ │ - WPR: protect/unprotect via GSP RPC │ ║ ║
║ ║ │ - Migration: save/restore via MIGRATE RPC │ ║ ║
║ ║ │ - Notifications: async events via NOTIFY RPC │ ║ ║
║ ║ │ • NO Open-IOV code in guest kernel │ ║ ║
║ ║ │ • NO gsp-vgpu.ko, NO libvgpu.so in guest │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ║ ║ ║
║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║
║ ║ │ vfio-pci.ko (Guest VFIO Core) │ ║ ║
║ ║ │ • mmap of BAR regions → host VFIO regions │ ║ ║
║ ║ │ • MSI-X interrupt handling → eventfd from host │ ║ ║
║ ║ │ • Config space read/write → host emulated config space │ ║ ║
║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║
║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║
║ ║
║ VIRTUAL GPU (mdev as seen by Guest): ║
║ PCI Config (256B) │ BAR0(16MB) │ BAR1(256MB) │ BAR2(32MB) │ BAR3(32MB) │ MSI-X Table/PBA ║
║ Vendor=10DE │ Config │ Framebuffer │ Reserved │ Reserved │ Table@0x2000 ║
║ Device=1ED3 │ Doorbell │ (VRAM) │ │ │ PBA@0x3000 ║
║ Class=030000 │ Queue Heads│ │ │ │ ║
║ BARs: 64-bit │ Shared Mem │ │ │ │ ║
║ Caps: PM,MSI, │ @0x110c00 │ │ │ │ ║
║ MSI-X, PCIe │ @0x1000 │ │ │ │ ║
╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
```
### 4. Communication Pathways (Complete)
| # | Pathway | Direction | Mechanism | Latency Target |
|---|---------|-----------|-----------|----------------|
| 1 | **GSP RPC (Guest→Host)** | Guest nvidia.ko → Host open-iov-vfio.ko | Doorbell write (MMIO) → host polling thread → gsp_rpc_client.c | < 5μs |
| 2 | **GSP Response (Host→Guest)** | Host open-iov-vfio.ko → Guest nvidia.ko | Write shared memory → MSI-X interrupt injection (eventfd) | < 3μs |
| 3 | **BAR Register Access** | Guest nvidia.ko → Host open-iov-vfio.ko | MMIO page fault → bar_emulation.c shadow memory | < 2μs |
| 4 | **PCI Config Space** | Guest kernel → Host open-iov-vfio.ko | VFIO_CONFIG_REGION_INDEX ioctl → emulated 256B config | < 1μs |
| 5 | **VFIO DMA (VRAM)** | Guest nvidia.ko ↔ Physical VRAM | IOMMU DMA mapping via VFIO DMA API | Near-native |
| 6 | **MSI-X Interrupts** | Physical GPU → Guest nvidia.ko | Host nvidia.ko ISR → nv_vgpu_vfio_inject_interrupt → eventfd → QEMU → Guest MSI-X | < 10μs |
| 7 | **Management Ops** | mdev management CLI/mgr → Host open-iov-vfio.ko | VFIO ioctls on mdev device → libvgpu.so → rpc_emulator.c | < 100μs |
| 8 | **Migration** | Host ↔ Host | Migration protocol over network, state via open-iov-vfio.ko | Seconds |
### 5. GSP Emulation Engine - Detailed Design
#### 5.1 Mailbox Layout (from Firmware Decompilation)
```
GSP Shared Memory Mailbox (at BAR0 + 0x1000, 4KB):
┌─────────────────────────────────────────────────────────────────────────────┐
│ Offset │ Size │ Field │ Description │
├────────┼───────┼──────────────────────────┼─────────────────────────────────┤
│ 0x0000 │ 64B │ RPC Request Header │ type, subcmd, seq_id, payload │
│ 0x0040 │ 64B │ RPC Request Payload[6] │ u64 payload[6] per RPC type │
│ 0x0080 │ 64B │ RPC Response Header │ status, type, seq_id │
│ 0x00C0 │ 64B │ RPC Response Payload[6] │ u64 result[6] │
│ 0x0100 │ 3KB │ Reserved / Extended │ Future use, migration state │
└─────────────────────────────────────────────────────────────────────────────┘
```
#### 5.2 Doorbell Register (at BAR0 + 0x110C00)
```
Queue Head Register (0x110C00):
┌─────────────────────────────────────────────────────────────────────────────┐
│ Bits │ Field │ Description │
├───────┼────────────────┼───────────────────────────────────────────────────┤
│ 31:1 │ Reserved │ Must be zero │
│ 0 │ Doorbell Valid │ 1 = New RPC request in mailbox, 0 = Idle │
└─────────────────────────────────────────────────────────────────────────────┘
```
#### 5.3 Per-mdev GSP Emulation Thread
```c
// In open-iov-vfio.ko - new file: gsp_emulation.c
struct gsp_emu_context {
struct nv_vgpu_device *vgpu;
struct task_struct *thread;
void __iomem *bar0_vaddr;
void __iomem *doorbell_reg;
void __iomem *shared_mem;
struct gsp_client *rpc_client; // From gsp_rpc_client.c
spinlock_t lock;
bool running;
u64 processed_rpcs;
u64 failed_rpcs;
};
static int gsp_emu_thread_fn(void *data) {
struct gsp_emu_context *ctx = data;
while (!kthread_should_stop()) {
// Read doorbell (atomic, no lock needed for single reader)
u32 doorbell = readl(ctx->doorbell_reg);
if (doorbell & 0x1) {
// ACK immediately to prevent guest retry
writel(0, ctx->doorbell_reg);
// Process RPC - this is the hot path
int ret = gsp_emu_process_rpc(ctx);
if (ret)
ctx->failed_rpcs++;
else
ctx->processed_rpcs++;
}
// Adaptive polling: busy-wait for first 100μs, then sleep
if (ctx->processed_rpcs % 1000 == 0)
usleep_range(50, 100);
else
cpu_relax();
}
return 0;
}
static int gsp_emu_process_rpc(struct gsp_emu_context *ctx) {
// Copy request from shared memory (guest → host)
struct gsp_rpc_msg req;
memcpy_fromio(&req, ctx->shared_mem, sizeof(req));
// Validate sequence ID
if (req.seq_id == 0)
return -EINVAL;
// Dispatch to firmware-decompiled handler
struct gsp_rpc_response rsp = {0};
rsp.seq_id = req.seq_id;
rsp.type = req.type;
int ret = 0;
switch (req.type) {
case GSP_RPC_ALLOC:
ret = gsp_rpc_alloc(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2],
req.payload[3], req.payload[4], req.payload[5],
&rsp.result[0], &rsp.result[1], &rsp.result[2]);
break;
case GSP_RPC_FREE:
ret = gsp_rpc_free(ctx->rpc_client, req.payload[0]);
break;
case GSP_RPC_MAP:
ret = gsp_rpc_map(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2],
req.payload[3], req.payload[4], req.payload[5],
&rsp.result[0]);
break;
case GSP_RPC_UNMAP:
ret = gsp_rpc_unmap(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2]);
break;
case GSP_RPC_PIN:
ret = gsp_rpc_pin(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2],
req.payload[3], &rsp.result[0], &rsp.result[1]);
break;
case GSP_RPC_UNPIN:
ret = gsp_rpc_unpin(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2]);
break;
case GSP_RPC_SCHEDULE:
ret = gsp_rpc_schedule(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2],
req.payload[3], req.payload[4]);
break;
case GSP_RPC_YIELD:
ret = gsp_rpc_yield(ctx->rpc_client);
break;
case GSP_RPC_NOTIFY:
ret = gsp_rpc_notify_register(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2]);
break;
case GSP_RPC_BAR_TRAP:
ret = gsp_rpc_bar_trap(ctx->rpc_client,
req.payload[0], req.payload[1], req.payload[2],
req.payload[3], req.payload[4], &rsp.result[0]);
break;
case GSP_RPC_MIGRATE:
// Handle migration sub-commands
ret = gsp_emu_handle_migrate(ctx, &req, &rsp);
break;
case GSP_RPC_DEBUG:
ret = gsp_emu_handle_debug(ctx, &req, &rsp);
break;
default:
rsp.status = -ENOTTY;
ret = -ENOTTY;
}
// Write response to shared memory (host → guest)
rsp.status = ret ? -EIO : 0;
memcpy_toio(ctx->shared_mem + 0x80, &rsp, sizeof(rsp));
// Inject MSI-X completion interrupt
gsp_emu_inject_completion_interrupt(ctx, req.seq_id);
return ret;
}
```
#### 5.4 MSI-X Completion Interrupt Injection
```c
static void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id) {
// Write to MSI-X PBA (Pending Bit Array) at BAR0 + 0x3000
// Vector 0 = GSP completion interrupt
void __iomem *pba = ctx->bar0_vaddr + 0x3000;
writel(0x1, pba); // Set pending bit for vector 0
// Trigger eventfd for VFIO interrupt injection
if (ctx->vgpu->msix_trigger[0])
eventfd_signal(ctx->vgpu->msix_trigger[0], 1);
}
```
### 6. Performance Requirements
| Metric | Target | Measurement Method |
|--------|--------|-------------------|
| GSP RPC round-trip (emulated) | < 10μs | ktime_get_ns() in gsp_emu_thread |
| BAR register access | < 2μs | Guest rdtsc around MMIO |
| VRAM allocation (ALLOC) | < 50μs | Guest nvidia.ko tracing |
| VRAM map (MAP) | < 20μs | Guest nvidia.ko tracing |
| DMA pin (PIN) | < 30μs | Guest nvidia.ko tracing |
| CUDA kernel launch overhead | < 5% vs native | CUDA events, nvprof |
| Graphics frame time (Vulkan) | < 2ms @ 60fps | PresentMon, vk_perf |
| Migration downtime | < 100ms | Stopwatch, ping during migration |
| MSI-X injection latency | < 5μs | Guest ISR timestamp vs host |
### 7. Testing Strategy - Incremental Validation
#### Phase 0: Host Infrastructure (Week 1)
- [ ] `open-iov-vfio.ko` loads, registers mdev parent with `nvidia.ko`
- [ ] mdev creation via sysfs succeeds
- [ ] PCI config space presented correctly (lspci in guest shows 10de:1ed3)
- [ ] BAR regions mmap'able in guest
- [ ] MSI-X table/PBA accessible in guest BAR0
#### Phase 1: GSP Emulation Core (Week 2)
- [ ] GSP emulation thread starts per mdev
- [ ] Doorbell polling detects guest writes
- [ ] ALLOC RPC: guest allocates VRAM, gets valid GPU VA + handle
- [ ] FREE RPC: guest frees allocation, no leaks
- [ ] MAP/UNMAP RPC: guest maps/unmaps GPU VA ranges
- [ ] PIN/UNPIN RPC: guest pins pages, gets valid DMA addresses
#### Phase 2: Advanced RPCs (Week 3)
- [ ] SCHEDULE/YIELD: guest submits work, gets completion
- [ ] BAR_TRAP: guest reads/writes BAR registers, gets emulated values
- [ ] NOTIFY: guest registers for async events, receives them
- [ ] MIGRATE: 4-phase migration saves/restores state
#### Phase 3: Guest Driver Integration (Week 4)
- [ ] Guest `nvidia.ko` loads, probes vGPU successfully
- [ ] `nvidia-smi` shows vGPU (T4 / A100 equivalent)
- [ ] CUDA context creation succeeds
- [ ] Simple CUDA kernel (vectorAdd) executes correctly
- [ ] `nvidia-ml` (NVML) queries work (temperature, utilization, memory)
#### Phase 4: Compute Workloads (Week 5)
- [ ] cuBLAS SGEMM/DGEMM passes numerical verification
- [ ] cuDNN convolution forward/backward passes
- [ ] PyTorch/TensorFlow training step completes
- [ ] Multi-GPU (NCCL) across vGPUs works
- [ ] CUDA Graphs capture/replay works
#### Phase 5: Graphics Workloads (Week 6)
- [ ] Vulkan: `vkCreateDevice` succeeds, render triangle
- [ ] OpenGL: `glxinfo` shows renderer, `glxgears` runs
- [ ] Video encode: FFmpeg + NVENC H.264/HEVC
- [ ] Video decode: FFmpeg + NVDEC H.264/HEVC
- [ ] Windows guest: D3D11/D3D12 device creation, render loop
#### Phase 6: Stress & Reliability (Week 7)
- [ ] 24-hour CUDA stress test (memtest, compute)
- [ ] 24-hour graphics stress (Unigine, FurMark equivalent)
- [ ] Hot-plug VF create/destroy cycles (1000x)
- [ ] Migration under load (live migrate running CUDA kernel)
- [ ] Host reboot recovery (mdev persists, guest recovers)
#### Phase 7: Windows Guest (Week 8)
- [ ] Windows 10/11 guest installs NVIDIA driver
- [ ] Device Manager shows "NVIDIA T4" (or equivalent)
- [ ] CUDA-Z / GPU-Z detect vGPU correctly
- [ ] DirectX 11/12: D3D11CreateDevice, D3D12CreateDevice
- [ ] Vulkan on Windows: vkCreateInstance, render
- [ ] Video encode/decode via Media Foundation
### 8. Implementation File Map
```
open-iov/
├── kernel/
│ ├── gsp_emulation.c # NEW: GSP emulation thread + RPC processing
│ ├── gsp_emulation.h # NEW: Structures, prototypes
│ ├── gsp_rpc_client.c # EXISTING: Firmware-decompiled handlers
│ ├── gsp_rpc_integration.c # EXISTING: Integration layer
│ ├── bar_emulation.c # EXISTING: BAR trap handling
│ ├── memory.c # EXISTING: DMA, VRAM management
│ ├── irq.c # EXISTING: MSI-X injection
│ ├── migration.c # EXISTING: Live migration
│ ├── main.c # MODIFY: Add gsp_emu_context to nv_vgpu_device
│ └── Makefile # MODIFY: Add gsp_emulation.o
├── userspace/lib/
│ ├── libvgpu.c # EXISTING: Management API
│ ├── rpc_emulator.c # EXISTING: Host-side emulation fallbacks
│ └── Makefile # UNCHANGED
├── scripts/
│ ├── test_gsp_emu.sh # NEW: Automated GSP emulation tests
│ ├── test_guest_driver.sh # NEW: Guest driver validation
│ ├── test_cuda_workloads.sh # NEW: CUDA workload validation
│ ├── test_graphics_workloads.sh # NEW: Graphics validation
│ └── test_windows_guest.sh # NEW: Windows guest validation
└── docs/
├── refactored-architecture.md # THIS DOCUMENT
└── refactored-implementation-to-do.md # TODO LIST
```
### 9. Risk Mitigation
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| GSP mailbox layout mismatch | Medium | High | Validate against firmware decompilation; add runtime checksum |
| MSI-X injection race | Medium | High | Use atomic PBA updates; test with concurrent RPCs |
| Cache coherency (WC memory) | High | Medium | clflush + mfence on every shared mem write; test on AMD/Intel |
| Guest driver version skew | Medium | Medium | Test against multiple OpenRM versions (550, 560, 570) |
| Windows guest driver issues | Medium | High | Separate Windows test CI; use WHQL driver |
| Performance regression vs native | Low | High | Continuous benchmarking; profile hot paths |
| Migration state corruption | Low | High | Checksum all migrated state; dry-run validation |
---
*Document Version: 1.0 (Refactored Architecture)*
*Author: Open-IOV Team*
*Date: $(date)*
*Repository: $OPEN_IOV_ROOT*
*Git Commit: f9ed5d8 (restore point)*
|
