1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
|
# Open-IOV GSP Emulation Engine — Architecture
> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`,
> `kernel/bar_emulation.c`, or debugging guest driver bring-up.
>
> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21
> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`;
> for the original refactor plan see `refactored-architecture.md`.
---
## 1. Purpose
An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects
to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC
messages to a mailbox in BAR0, rings a doorbell, and waits for a completion
interrupt plus a response written back into the mailbox.
Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the
host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the
host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes
RPCs, performs them in software (or declines them), writes responses back, and
injects MSI-X interrupts.
```
┌──────────────────────────── Guest VM ────────────────────────────┐
│ stock nvidia.ko (570.144) │
│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │
│ ▼ │
│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │
└──────────────────────────────────────────────────────────────────┘
│ (16 MB vzalloc'd shadow, uncached)
┌──────────────────────────── Host ────────────────────────────────┐
│ gsp_emu thread (open-iov-vfio.ko) │
│ poll shadow+0x110C00 (doorbell) │
│ → read RPC msg from shadow+0x1000 │
│ → dispatch (12 RPC types, emulation implementations) │
│ → write response to shadow+0x1080 │
│ → set PBA bit at shadow+0x3000 │
│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │
└──────────────────────────────────────────────────────────────────┘
```
---
## 2. The single source of truth: `bar0_shadow`
The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software
shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()`
(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler
(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`).
Everything the guest driver does with BAR0 lands in this shadow:
| BAR0 offset | Region | Guest writes go to | Emulation reads from |
|------------------|-----------------------------------|--------------------|----------------------|
| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow |
| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` |
| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads |
| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) |
| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads |
| `0x110C00` | **GSP doorbell** | shadow | thread polls this |
| rest | emulated register space | shadow | shadow |
> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled
> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real
> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer —
> it could never see guest traffic and even wrote PBA bits into the real GPU's
> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`.
> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so
> cleanup never `dma_free`s it.
### 2.1 Transport protocol (as emulated)
1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000`
(`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`).
2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`.
3. The emulation thread sees the doorbell, ACKs it (writes 0), validates
`seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`.
4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`,
`result[6]`, `seq_id`) written to `BAR0+0x1080`.
5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired
eventfd → QEMU delivers MSI-X to the guest.
6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and
accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the
public `gsp_emu_process_rpc()` entry does not bump counters).
---
## 3. RPC dispatch
`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls
the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation
implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by
`gsp_emu_init`).
| Type (val) | Handler path in emulation | Notes |
|----------------|--------------------------------------------------------|-------|
| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list |
| FREE (0x02) | `gsp_rpc_free_emul` | |
| MAP (0x03) | `gsp_rpc_map_emul` | |
| UNMAP (0x04) | `gsp_rpc_unmap_emul` | |
| PIN (0x05) | `gsp_rpc_pin_emul` | |
| UNPIN (0x06) | `gsp_rpc_unpin_emul` | |
| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission |
| YIELD (0x08) | no-op success | |
| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet |
| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO |
| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO |
| DEBUG (0xFF) | log-only in dispatch | |
| unknown | `-ENOTSUPP` | |
> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to
> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox
> and **rings the same doorbell the thread polls** — a self-triggered infinite
> RPC loop plus multi-second stalls. Emulation mode now fails these fast with
> `pr_warn_ratelimited`.
### Memory model caveat (design note)
Guest VRAM (BAR1) is mapped **directly to the host GPU's physical BAR1** (gpa).
`gsp_rpc_alloc_emul`, however, services ALLOC with *host system pages* DMA-mapped
to the physical device. These two models are not yet unified: a guest allocation
returning `gpu_va=0x100000000+` is not visible through the guest's BAR1 window.
This is a known design gap to resolve before CUDA workloads; it does not affect
driver probe/GSP boot, which is the current milestone.
---
## 4. Interrupt delivery (MSI-X)
1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`.
Our ioctl handler takes `eventfd_ctx_fdget()` references into
`vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs).
2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`:
sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger
from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after*
`gsp_emu_init`, so an init-time copy would be permanently NULL; this was a
real bug fixed 2026-09-21).
INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator
currently fires vector 0 for GSP completion.
---
## 5. Lifecycle
```
mdev create (nv_vgpu_mdev_create)
└─ schedule init_work
└─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow)
└─ nv_vgpu_irq_init / nv_vgpu_memory_init
└─ gsp_emu_init
├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases)
├─ gsp_client_init_emulation (RPC client, emulation_mode=true)
├─ gsp_rpc_emulation_set(true)
└─ kthread_run(gsp_emu_thread_fn)
mdev remove (nv_vgpu_mdev_remove)
└─ cancel init_work, wait for VFIO fds to close
└─ vfio_unregister_group_dev
└─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion)
→ memory → irq → BAR (frees bar0_shadow)
└─ list_del, pci_dev_put, vfio_put_device
```
---
## 6. Debugging the emulator
- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'`
- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on
the thread path only)
- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub
`tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation
checklist.
- Unit test: `kernel/tests/gsp_emulation_test.ko` drives
`gsp_emu_process_rpc()` with crafted mailbox messages (requires
`open-iov-vfio.ko` loaded; `depends:` is set accordingly).
---
## 7. Key files
| File | Role |
|------|------|
| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection |
| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs |
| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state |
| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space |
| `kernel/memory.c` | simple mapping/pin tracking (host management API) |
| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) |
|
