1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
|
# GSP vGPU RPC Emulation
> **Historical document — superseded.** This page describes an earlier
> emulation-mode design and is retained for reference only. The project does
> **not** use NVIDIA's proprietary vGPU stack. Canonical documentation:
> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md)
> and [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md).
## Overview
Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.
## Problem
The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:
| Handler | Address | Size | Purpose |
|---------|---------|------|---------|
| BAR Trap | 0x04CB8188 | 40KB | BAR1/ForceBarAccess/Aperture |
| WPR Handler 1 | 0x04F00B0C | 62KB | Write/Read Protect Regions |
| WPR Handler 2 | 0x04F00C78 | 62KB | Write/Read Protect Regions |
| Migration | 0x04179ED8/0x04179FBC | - | Live migration |
Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).
## Solution: Emulation Mode
Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:
```
┌─────────────────────────────────────────────────────────────────┐
│ Emulation Architecture │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Userspace App (libiov.so) │
│ │ │
│ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Kernel IOCTL │────▶│ Emulation Layer │ │
│ │ Handler │ │ (kernel) │ │
│ └─────────────────┘ └────────┬────────┘ │
│ │ │
│ ┌───────────────┼───────────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ BAR Emu │ │ WPR Emu │ │ Mem Emu │ │
│ │ (kernel) │ │ (kernel) │ │ (kernel) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────┐ │
│ │ Userspace Emulator (libiov.so) │ │
│ │ rpc_emu_bar_*(), rpc_emu_wpr_*() │ │
│ └──────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
## Auto-Detection (Default Behavior)
**Emulation mode is ON by default** and auto-detects at driver load:
1. Driver initializes RPC communication with GSP
2. Sends test `BAR_TRAP` RPC (non-destructive READ)
3. **If GSP responds** (ret=0) → vGPU handlers present → **emulation disabled**
4. **If GSP times out/errors** (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → **emulation enabled**
```
dmesg output:
gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
# or
gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode
```
## Module Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `emulation_mode` | bool | 1 (ON) | Current state (read-only, shows auto-detected value) |
| `emulation_override` | bool | 0 (auto) | Override: 1=force emulation ON, 0=force emulation OFF |
```bash
# Default: auto-detect
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin
# Force emulation ON (testing with GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1
# Force emulation OFF (requires GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0
```
## Emulated RPC Handlers
### Kernel Emulation (`driver/`)
| RPC Type | Handler | Emulation |
|----------|---------|-----------|
| `GSP_RPC_ALLOC` (0x01) | `gsp_vf_mem_alloc()` | Synthetic handles/VAs from local counter |
| `GSP_RPC_PIN` (0x05) | `gsp_vf_mem_pin()` | Synthetic DMA addresses from local counter |
| `GSP_RPC_MAP` (0x03) | `gsp_vf_mem_map()` | Identity map (return requested VA) |
| `GSP_RPC_BAR_TRAP` (0x0A) | `gsp_bar_trap_handler()` | Local aperture/force_access state |
| `GSP_RPC_MAP` (WPR) | `gsp_vf_wpr_protect()` | Local WPR region list |
| `GSP_RPC_UNMAP` (WPR) | `gsp_vf_wpr_unprotect()` | Local WPR region removal |
### Userspace Emulation (`userspace/lib/rpc_emulator.c`)
| Function | Purpose |
|----------|---------|
| `rpc_emu_bar_config()` | Local aperture offset/size + backing store |
| `rpc_emu_bar_force_access()` | Local force_access state |
| `rpc_emu_bar_save/restore()` | Migration state serialization |
| `rpc_emu_wpr_protect_write/read()` | Local WPR region tracking |
| `rpc_emu_wpr_unprotect()` | Local WPR region removal |
| `rpc_emu_wpr_set_timeout()` | Local timeout tracking |
| `rpc_emu_wpr_check_access()` | Access violation detection |
## Emulation Data Structures
### BAR State (per VF)
```c
struct gsp_bar_state {
uint64_t aperture_offset; // BAR1 aperture base
uint64_t aperture_size; // BAR1 aperture size (max 256KB)
bool force_access; // ForceBarAccess enabled
uint32_t force_vf; // VF holding force access
void *backing_store; // Emulated BAR memory
size_t backing_size; // Backing store size
};
```
### WPR Region List (per VF)
```c
struct gsp_wpr_region {
struct list_head list;
uint64_t base; // Region base address
uint64_t size; // Region size
uint32_t flags; // GSP_WPR_WRITE/READ/EXECUTE
uint32_t vf_id; // Owning VF
void (*completion_cb)(...); // Write/read complete callback
void (*timeout_cb)(...); // Timeout callback
uint64_t timeout_ns; // Timeout in nanoseconds
};
```
### Userspace Emulator State
```c
struct emu_bar_state {
uint64_t aperture_offset;
uint64_t aperture_size;
bool force_access;
uint32_t force_vf;
void *backing_store;
size_t backing_size;
pthread_mutex_t lock;
};
struct emu_wpr_state {
struct emu_wpr_region *regions; // Linked list
pthread_mutex_t lock;
};
```
## Fallback Logic
Userspace library (`libiov.so`) implements fallback:
```c
vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
// 1. Try kernel IOCTL (sends RPC to GSP)
ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
// 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
if (ret < 0 && dev->emulation_enabled &&
(errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
return rpc_emu_bar_config(vf->id, offset, size);
}
return map_errno(errno);
}
```
## Limitations
| Feature | Emulation Status | Notes |
|---------|-----------------|-------|
| BAR Aperture Config | ✅ Full | Local state + backing store |
| ForceBarAccess | ✅ Full | Per-VF exclusive access |
| BAR Save/Restore | ✅ Full | Migration support |
| WPR Write Protect | ✅ Full | Region tracking + access check |
| WPR Read Protect | ✅ Full | Region tracking + access check |
| WPR Timeout | ✅ Partial | Timeout stored, callbacks not wired |
| Memory Alloc | ✅ Basic | Synthetic handles/VAs |
| Memory Pin | ✅ Basic | Synthetic DMA addresses |
| Memory Map | ✅ Basic | Identity mapping |
| Migration | ⚠️ Partial | BAR/WPR state only |
| Scheduling | ❌ No | Returns success, no actual scheduling |
| Notifications | ❌ No | No async event delivery |
## Testing Emulation
```bash
# 1. Load driver (auto-enables emulation on consumer GSP)
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin
# 2. Verify emulation enabled in dmesg
dmesg -T | grep -i emulation
# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
# 3. Run example - should complete without timeouts
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu
# 4. Check debugfs for emulated state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions
```
## Future Work
- [ ] Wire WPR timeout callbacks to kernel timers
- [ ] Implement emulated scheduling with work queues
- [ ] Add async notification support via eventfd
- [ ] Complete migration state for WPR regions
- [ ] Performance optimization (batch RPC emulation)
- [ ] Conformance tests vs GRID GSP behavior
## Related Documentation
- [BAR Emulation Specification](BAR-Emulation-Specification.md)
- [WPR Management](WPR-Management.md)
- [RPC Message Format](RPC-Message-Format.md)
- [Integration Guide](vGPU-IO-Integration-Guide.md)
|
