1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
|
# Open-IOV GSP Emulation Engine — Architecture
> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`,
> `kernel/bar_emulation.c`, or debugging guest driver bring-up.
>
> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21
> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`;
> for the original refactor plan see `refactored-architecture.md`.
---
## 1. Purpose
An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects
to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC
messages to a mailbox in BAR0, rings a doorbell, and waits for a completion
interrupt plus a response written back into the mailbox.
Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the
host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the
host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes
RPCs, performs them in software (or declines them), writes responses back, and
injects MSI-X interrupts.
```
┌──────────────────────────── Guest VM ────────────────────────────┐
│ stock nvidia.ko (570.144) │
│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │
│ ▼ │
│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │
└──────────────────────────────────────────────────────────────────┘
│ (16 MB vzalloc'd shadow, uncached)
┌──────────────────────────── Host ────────────────────────────────┐
│ gsp_emu thread (open-iov-vfio.ko) │
│ poll shadow+0x110C00 (doorbell) │
│ → read RPC msg from shadow+0x1000 │
│ → dispatch (12 RPC types, emulation implementations) │
│ → write response to shadow+0x1080 │
│ → set PBA bit at shadow+0x3000 │
│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │
└──────────────────────────────────────────────────────────────────┘
```
---
## 2. The single source of truth: `bar0_shadow`
The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software
shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()`
(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler
(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`).
Everything the guest driver does with BAR0 lands in this shadow:
| BAR0 offset | Region | Guest writes go to | Emulation reads from |
|------------------|-----------------------------------|--------------------|----------------------|
| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow |
| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` |
| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads |
| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) |
| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads |
| `0x110C00` | **GSP doorbell** | shadow | thread polls this |
| rest | emulated register space | shadow | shadow |
> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled
> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real
> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer —
> it could never see guest traffic and even wrote PBA bits into the real GPU's
> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`.
> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so
> cleanup never `dma_free`s it.
### 2.1 Transport protocol (as emulated)
1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000`
(`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`).
2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`.
3. The emulation thread sees the doorbell, ACKs it (writes 0), validates
`seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`.
4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`,
`result[6]`, `seq_id`) written to `BAR0+0x1080`.
5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired
eventfd → QEMU delivers MSI-X to the guest.
6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and
accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the
public `gsp_emu_process_rpc()` entry does not bump counters).
---
## 3. RPC dispatch
`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls
the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation
implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by
`gsp_emu_init`).
| Type (val) | Handler path in emulation | Notes |
|----------------|--------------------------------------------------------|-------|
| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list |
| FREE (0x02) | `gsp_rpc_free_emul` | |
| MAP (0x03) | `gsp_rpc_map_emul` | |
| UNMAP (0x04) | `gsp_rpc_unmap_emul` | |
| PIN (0x05) | `gsp_rpc_pin_emul` | |
| UNPIN (0x06) | `gsp_rpc_unpin_emul` | |
| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission |
| YIELD (0x08) | no-op success | |
| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet |
| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO |
| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO |
| DEBUG (0xFF) | log-only in dispatch | |
| unknown | `-ENOTSUPP` | |
> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to
> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox
> and **rings the same doorbell the thread polls** — a self-triggered infinite
> RPC loop plus multi-second stalls. Emulation mode now fails these fast with
> `pr_warn_ratelimited`.
### Memory model (corrected 2026-09-22)
Guest BAR1 (VRAM aperture) is backed by a **dedicated host-RAM vzalloc buffer**
(256 MB, `bar_init.c`), NOT the host GPU's physical BAR1. Guest VRAM accesses
hit our clean host-RAM backing. This is a placeholder — for real CUDA compute,
the memory model must evolve (see `refactored-architecture.md` Phase 2):
- **Current (Phase 1)**: Guest VRAM = host RAM (safe, no host GPU corruption).
Guest driver probes but cannot compute.
- **Phase 2**: Guest VRAM backed by DMA-coherent host memory or physical GPU
VRAM partition (required for CUDA).
- **Phase 3**: Guest channel/pushbuffer work forwarded to physical GPU
channels (required for compute).
Guest BAR3 (instance memory window) is also a **vzalloc'd shadow** (32 MB,
`bar_init.c`), NOT the physical GPU's BAR3. Guest BAR3 writes must not reach
the host GPU's instance memory (root cause of host GSP crash, fixed `85d0668`).
---
## 4. Interrupt delivery (MSI-X)
1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`.
Our ioctl handler takes `eventfd_ctx_fdget()` references into
`vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs).
2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`:
sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger
from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after*
`gsp_emu_init`, so an init-time copy would be permanently NULL; this was a
real bug fixed 2026-09-21).
INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator
currently fires vector 0 for GSP completion.
---
## 5. Lifecycle
```
mdev create (nv_vgpu_mdev_create)
└─ schedule init_work
└─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow)
└─ nv_vgpu_irq_init / nv_vgpu_memory_init
└─ gsp_emu_init
├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases)
├─ gsp_client_init_emulation (RPC client, emulation_mode=true)
├─ gsp_rpc_emulation_set(true)
└─ kthread_run(gsp_emu_thread_fn)
mdev remove (nv_vgpu_mdev_remove)
└─ cancel init_work, wait for VFIO fds to close
└─ vfio_unregister_group_dev
└─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion)
→ memory → irq → BAR (frees bar0_shadow)
└─ list_del, pci_dev_put, vfio_put_device
```
---
## 6. Debugging the emulator
- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'`
- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on
the thread path only)
- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub
`tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation
checklist.
- Unit test: `kernel/tests/gsp_emulation_test.ko` drives
`gsp_emu_process_rpc()` with crafted mailbox messages (requires
`open-iov-vfio.ko` loaded; `depends:` is set accordingly).
---
## 7. Key files
| File | Role |
|------|------|
| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection |
| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs |
| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state |
| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space |
| `kernel/memory.c` | simple mapping/pin tracking (host management API) |
| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) |
|
