# Open-IOV GSP Emulation Engine — Architecture > **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, > `kernel/bar_emulation.c`, or debugging guest driver bring-up. > > **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21 > fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`; > for the original refactor plan see `refactored-architecture.md`. --- ## 1. Purpose An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC messages to a mailbox in BAR0, rings a doorbell, and waits for a completion interrupt plus a response written back into the mailbox. Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes RPCs, performs them in software (or declines them), writes responses back, and injects MSI-X interrupts. ``` ┌──────────────────────────── Guest VM ────────────────────────────┐ │ stock nvidia.ko (570.144) │ │ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │ │ ▼ │ │ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │ └──────────────────────────────────────────────────────────────────┘ │ (16 MB vzalloc'd shadow, uncached) ┌──────────────────────────── Host ────────────────────────────────┐ │ gsp_emu thread (open-iov-vfio.ko) │ │ poll shadow+0x110C00 (doorbell) │ │ → read RPC msg from shadow+0x1000 │ │ → dispatch (12 RPC types, emulation implementations) │ │ → write response to shadow+0x1080 │ │ → set PBA bit at shadow+0x3000 │ │ → eventfd_signal(msix_trigger[0]) → guest MSI-X │ └──────────────────────────────────────────────────────────────────┘ ``` --- ## 2. The single source of truth: `bar0_shadow` The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()` (`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler (`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`). Everything the guest driver does with BAR0 lands in this shadow: | BAR0 offset | Region | Guest writes go to | Emulation reads from | |------------------|-----------------------------------|--------------------|----------------------| | `0x0000-0x0FFF` | (shadow register area) | shadow | shadow | | `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` | | `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads | | `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) | | `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads | | `0x110C00` | **GSP doorbell** | shadow | thread polls this | | rest | emulated register space | shadow | shadow | > **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled > the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real > GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer — > it could never see guest traffic and even wrote PBA bits into the real GPU's > BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`. > `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so > cleanup never `dma_free`s it. ### 2.1 Transport protocol (as emulated) 1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000` (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`). 2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`. 3. The emulation thread sees the doorbell, ACKs it (writes 0), validates `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`. 4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`, `result[6]`, `seq_id`) written to `BAR0+0x1080`. 5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired eventfd → QEMU delivers MSI-X to the guest. 6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the public `gsp_emu_process_rpc()` entry does not bump counters). --- ## 3. RPC dispatch `gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by `gsp_emu_init`). | Type (val) | Handler path in emulation | Notes | |----------------|--------------------------------------------------------|-------| | ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list | | FREE (0x02) | `gsp_rpc_free_emul` | | | MAP (0x03) | `gsp_rpc_map_emul` | | | UNMAP (0x04) | `gsp_rpc_unmap_emul` | | | PIN (0x05) | `gsp_rpc_pin_emul` | | | UNPIN (0x06) | `gsp_rpc_unpin_emul` | | | SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission | | YIELD (0x08) | no-op success | | | NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet | | BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO | | MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO | | DEBUG (0xFF) | log-only in dispatch | | | unknown | `-ENOTSUPP` | | > **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to > `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox > and **rings the same doorbell the thread polls** — a self-triggered infinite > RPC loop plus multi-second stalls. Emulation mode now fails these fast with > `pr_warn_ratelimited`. ### Memory model caveat (design note) Guest VRAM (BAR1) is mapped **directly to the host GPU's physical BAR1** (gpa). `gsp_rpc_alloc_emul`, however, services ALLOC with *host system pages* DMA-mapped to the physical device. These two models are not yet unified: a guest allocation returning `gpu_va=0x100000000+` is not visible through the guest's BAR1 window. This is a known design gap to resolve before CUDA workloads; it does not affect driver probe/GSP boot, which is the current milestone. --- ## 4. Interrupt delivery (MSI-X) 1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`. Our ioctl handler takes `eventfd_ctx_fdget()` references into `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs). 2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`: sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after* `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a real bug fixed 2026-09-21). INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator currently fires vector 0 for GSP completion. --- ## 5. Lifecycle ``` mdev create (nv_vgpu_mdev_create) └─ schedule init_work └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow) └─ nv_vgpu_irq_init / nv_vgpu_memory_init └─ gsp_emu_init ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases) ├─ gsp_client_init_emulation (RPC client, emulation_mode=true) ├─ gsp_rpc_emulation_set(true) └─ kthread_run(gsp_emu_thread_fn) mdev remove (nv_vgpu_mdev_remove) └─ cancel init_work, wait for VFIO fds to close └─ vfio_unregister_group_dev └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion) → memory → irq → BAR (frees bar0_shadow) └─ list_del, pci_dev_put, vfio_put_device ``` --- ## 6. Debugging the emulator - Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'` - Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on the thread path only) - Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation checklist. - Unit test: `kernel/tests/gsp_emulation_test.ko` drives `gsp_emu_process_rpc()` with crafted mailbox messages (requires `open-iov-vfio.ko` loaded; `depends:` is set accordingly). --- ## 7. Key files | File | Role | |------|------| | `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection | | `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs | | `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state | | `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space | | `kernel/memory.c` | simple mapping/pin tracking (host management API) | | `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) |