1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
# Open-IOV
**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU) technology. It provides:
- **open-iov-vfio.ko** - Kernel VFIO mediated device driver for vGPU
- **open-iov.so** - Userspace library for vGPU management (replaces libvgpu.so)
- **GSP Firmware Analysis** - Complete decompilation of NVIDIA GSP 570.144W (TU102/TU104)
- **RPC Interface** - 12 RPC types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Open-IOV Stack │
├─────────────────────────────────────────────────────────────────┤
│ Userspace │
│ ├── open-iov.so (libopen-iov) │
│ ├── examples/ (demo applications) │
│ └── tests/ (unit/integration tests) │
├─────────────────────────────────────────────────────────────────┤
│ Kernel │
│ ├── open-iov-vfio.ko (VFIO mdev driver) │
│ │ ├── mdev parent registration │
│ │ ├── VF lifecycle (create/destroy) │
│ │ ├── BAR emulation (BAR1, ForceBarAccess, Aperture) │
│ │ ├── GSP RPC communication (doorbell, rings) │
│ │ ├── WPR management (write/read protect) │
│ │ ├── Memory management (alloc, map, pin) │
│ │ └── Live migration (4-phase protocol) │
│ └── debugfs interface (/sys/kernel/debug/open-iov/) │
├─────────────────────────────────────────────────────────────────┤
│ Firmware (GSP 570.144W - TU102/TU104) │
│ ├── Primary Dispatcher 0x057A1AC0 (32 sub-commands) │
│ ├── Secondary Dispatcher 0x04ED39DC (141KB) │
│ ├── BAR Trap Handler 0x04CB8188 (40KB) │
│ ├── WPR Handlers 0x04F00B0C, 0x04F00C78 (62KB each) │
│ ├── Migration Tables 0x04179ED8, 0x04179FBC (dual 16-entry)│
│ └── Entry Point 0x04009000 │
└─────────────────────────────────────────────────────────────────┘
```
## Quick Start
### Build Everything
```bash
cd open-iov
make # Builds kernel module, userspace lib, examples
```
### Install Kernel Module
```bash
# Copy firmware
sudo cp firmware/570-GSP/gsp-570.144.fwimage.bin /lib/firmware/
# Load kernel module (emulation mode ON by default, auto-detects GSP capabilities)
sudo insmod kernel/open-iov-vfio.ko
# Or load guest driver with emulation
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin
# Verify
dmesg -T | grep -E "gsp-vgpu|open-iov"
ls -la /dev/gsp-vgpu /dev/open-iov
```
### Run Example
```bash
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu
```
## Emulation Mode
Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers (BAR_TRAP, WPR, etc.) in userspace and kernel.
### How It Works
| Component | Emulated In | RPC Types Emulated |
|-----------|-------------|-------------------|
| **Kernel** | `gsp-vgpu.ko` | BAR_TRAP (0x0A), WPR (MAP/UNMAP for WPR), ALLOC, PIN, MAP |
| **Userspace** | `libiov.so` | BAR config/force/save/restore, WPR protect/unprotect/timeout |
When emulation mode is enabled:
- BAR aperture configuration is handled locally (no GSP RPC)
- WPR regions are tracked in kernel/userspace memory
- Memory allocation returns synthetic handles/VAs
- Memory pinning returns synthetic DMA addresses
- BAR trap handling is simulated
### Auto-Detection (Default)
**Emulation mode is ON by default** and auto-detects GSP capabilities:
1. On driver load, sends a test `BAR_TRAP` RPC to GSP
2. If GSP responds successfully → vGPU handlers present → **disables emulation**
3. If GSP times out/errors → vGPU handlers missing → **enables emulation**
```
dmesg output:
gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
# or
gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode
```
### Override Flags
Force emulation mode behavior via module parameters:
| Parameter | Values | Description |
|-----------|--------|-------------|
| `emulation_mode` | `0`/`1` | Current state (read-only, shows auto-detected value) |
| `emulation_override` | `0`/`1` | **Override auto-detection**: `1`=force emulation ON, `0`=force emulation OFF |
```bash
# Force emulation ON (for testing with GRID GSP)
sudo insmod driver/gsp-vgpu.ko emulation_override=1
# Force emulation OFF (requires GRID host driver with vGPU GSP)
sudo insmod driver/gsp-vgpu.ko emulation_override=0
# Default: auto-detect (emulation_mode=1, emulation_override=0)
sudo insmod driver/gsp-vgpu.ko
```
### GSP Blob Handling Options
The vGPU RPC handlers (at 0x04CB8188, 0x04F00B0C, etc.) must be present in GSP firmware. Options:
| Option | Description | Pros | Cons |
|--------|-------------|------|------|
| **GRID Host Driver** | Use NVIDIA GRID vGPU KVM driver (535+/550+) to boot GSP with vGPU firmware | Official, complete vGPU support | Requires NVIDIA Enterprise license |
| **OpenRM GSP** | Use GSP firmware from NVIDIA open kernel modules (OpenRM) | Open source, no license | Consumer GSP lacks vGPU RPC handlers |
| **Custom GSP Blob** | Inject vGPU handlers into GSP firmware | Full control | Complex, requires firmware signing |
| **Userspace Emulation** | Emulate vGPU RPC handlers in userspace/kernel | Works with any GSP, no license | Performance, completeness |
**Recommended**: Use GRID host driver on host, Open-IOV guest driver in VM. For development, OpenRM GSP + emulation mode (auto-enabled).
## GPC (GPU Partitioning Controller) Interfaces
Open-IOV implements GPC-compatible interfaces for GPU partitioning:
- **VF Creation/Deletion** - `nv_vgpu_create_callback`, `nv_vgpu_delete_callback`
- **BAR Management** - `nv_vgpu_bar_info_callback`
- **Memory Operations** - Alloc, map, pin via GSP RPC
- **Migration** - 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP)
- **Scheduling** - Work submission, yield, priority
## Documentation
- [Firmware Analysis](wiki/GSP-570.144W-Firmware-Analysis.md)
- [RPC Message Format](wiki/RPC-Message-Format.md)
- [BAR Emulation](wiki/BAR-Emulation-Specification.md)
- [WPR Management](wiki/WPR-Management.md)
- [Migration Protocol](wiki/Migration-Protocol.md)
- [Integration Guide](wiki/vGPU-IO-Integration-Guide.md)
## Requirements
- Linux kernel 5.15+ (6.8+ tested)
- NVIDIA GPU: TU102/TU104 (RTX 2080/2080 Super)
- For full vGPU: GRID KVM driver 535+ or 550+
- Build: gcc, make, kernel headers
## License
GPL v2 (kernel), LGPL (userspace library)
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
