diff options
Diffstat (limited to 'docs/GSP_EMULATION_ARCHITECTURE.md')
| -rw-r--r-- | docs/GSP_EMULATION_ARCHITECTURE.md | 210 |
1 files changed, 0 insertions, 210 deletions
diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md deleted file mode 100644 index 70b4b76..0000000 --- a/docs/GSP_EMULATION_ARCHITECTURE.md +++ /dev/null @@ -1,210 +0,0 @@ -# Open-IOV GSP Emulation Engine — Architecture - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Architecture -> replacement: -> [03-architecture/01](03-architecture/01-system-overview.md) and -> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md); -> the transport section in particular is superseded by the msgq -> documentation in -> [03-architecture/03](03-architecture/03-gsp-and-gsp-rm.md). - -> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, -> `kernel/bar_emulation.c`, or debugging guest driver bring-up. -> -> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21 -> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`; -> for the original refactor plan see `refactored-architecture.md`. - ---- - -## 1. Purpose - -An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects -to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC -messages to a mailbox in BAR0, rings a doorbell, and waits for a completion -interrupt plus a response written back into the mailbox. - -Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the -host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the -host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes -RPCs, performs them in software (or declines them), writes responses back, and -injects MSI-X interrupts. - -``` -┌──────────────────────────── Guest VM ────────────────────────────┐ -│ stock nvidia.ko (570.144) │ -│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │ -│ ▼ │ -│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │ -└──────────────────────────────────────────────────────────────────┘ - │ (16 MB vzalloc'd shadow, uncached) -┌──────────────────────────── Host ────────────────────────────────┐ -│ gsp_emu thread (open-iov-vfio.ko) │ -│ poll shadow+0x110C00 (doorbell) │ -│ → read RPC msg from shadow+0x1000 │ -│ → dispatch (12 RPC types, emulation implementations) │ -│ → write response to shadow+0x1080 │ -│ → set PBA bit at shadow+0x3000 │ -│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │ -└──────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 2. The single source of truth: `bar0_shadow` - -The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software -shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()` -(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler -(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`). - -Everything the guest driver does with BAR0 lands in this shadow: - -| BAR0 offset | Region | Guest writes go to | Emulation reads from | -|------------------|-----------------------------------|--------------------|----------------------| -| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow | -| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` | -| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads | -| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) | -| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads | -| `0x110C00` | **GSP doorbell** | shadow | thread polls this | -| rest | emulated register space | shadow | shadow | - -> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled -> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real -> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer — -> it could never see guest traffic and even wrote PBA bits into the real GPU's -> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`. -> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so -> cleanup never `dma_free`s it. - -### 2.1 Transport protocol (as emulated) - -1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000` - (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`). -2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`. -3. The emulation thread sees the doorbell, ACKs it (writes 0), validates - `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`. -4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`, - `result[6]`, `seq_id`) written to `BAR0+0x1080`. -5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired - eventfd → QEMU delivers MSI-X to the guest. -6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and - accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the - public `gsp_emu_process_rpc()` entry does not bump counters). - ---- - -## 3. RPC dispatch - -`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls -the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation -implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by -`gsp_emu_init`). - -| Type (val) | Handler path in emulation | Notes | -|----------------|--------------------------------------------------------|-------| -| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list | -| FREE (0x02) | `gsp_rpc_free_emul` | | -| MAP (0x03) | `gsp_rpc_map_emul` | | -| UNMAP (0x04) | `gsp_rpc_unmap_emul` | | -| PIN (0x05) | `gsp_rpc_pin_emul` | | -| UNPIN (0x06) | `gsp_rpc_unpin_emul` | | -| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission | -| YIELD (0x08) | no-op success | | -| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet | -| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO | -| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO | -| DEBUG (0xFF) | log-only in dispatch | | -| unknown | `-ENOTSUPP` | | - -> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to -> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox -> and **rings the same doorbell the thread polls** — a self-triggered infinite -> RPC loop plus multi-second stalls. Emulation mode now fails these fast with -> `pr_warn_ratelimited`. - -### Memory model (corrected 2026-09-22) - -Guest BAR1 (VRAM aperture) is backed by a **dedicated host-RAM vzalloc buffer** -(256 MB, `bar_init.c`), NOT the host GPU's physical BAR1. Guest VRAM accesses -hit our clean host-RAM backing. This is a placeholder — for real CUDA compute, -the memory model must evolve (see `refactored-architecture.md` Phase 2): - -- **Current (Phase 1)**: Guest VRAM = host RAM (safe, no host GPU corruption). - Guest driver probes but cannot compute. -- **Phase 2**: Guest VRAM backed by DMA-coherent host memory or physical GPU - VRAM partition (required for CUDA). -- **Phase 3**: Guest channel/pushbuffer work forwarded to physical GPU - channels (required for compute). - -Guest BAR3 (instance memory window) is also a **vzalloc'd shadow** (32 MB, -`bar_init.c`), NOT the physical GPU's BAR3. Guest BAR3 writes must not reach -the host GPU's instance memory (root cause of host GSP crash, fixed `85d0668`). - ---- - -## 4. Interrupt delivery (MSI-X) - -1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`. - Our ioctl handler takes `eventfd_ctx_fdget()` references into - `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs). -2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`: - sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger - from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after* - `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a - real bug fixed 2026-09-21). - -INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator -currently fires vector 0 for GSP completion. - ---- - -## 5. Lifecycle - -``` -mdev create (nv_vgpu_mdev_create) - └─ schedule init_work - └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow) - └─ nv_vgpu_irq_init / nv_vgpu_memory_init - └─ gsp_emu_init - ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases) - ├─ gsp_client_init_emulation (RPC client, emulation_mode=true) - ├─ gsp_rpc_emulation_set(true) - └─ kthread_run(gsp_emu_thread_fn) - -mdev remove (nv_vgpu_mdev_remove) - └─ cancel init_work, wait for VFIO fds to close - └─ vfio_unregister_group_dev - └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion) - → memory → irq → BAR (frees bar0_shadow) - └─ list_del, pci_dev_put, vfio_put_device -``` - ---- - -## 6. Debugging the emulator - -- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'` -- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on - the thread path only) -- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub - `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation - checklist. -- Unit test: `kernel/tests/gsp_emulation_test.ko` drives - `gsp_emu_process_rpc()` with crafted mailbox messages (requires - `open-iov-vfio.ko` loaded; `depends:` is set accordingly). - ---- - -## 7. Key files - -| File | Role | -|------|------| -| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection | -| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs | -| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state | -| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space | -| `kernel/memory.c` | simple mapping/pin tracking (host management API) | -| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) | |