summaryrefslogtreecommitdiffstats
path: root/docs/GSP_EMULATION_ARCHITECTURE.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:07 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:07 -0400
commit681494a79e6c0e084354eec0229386a696a314c0 (patch)
treef0d69e9583cd6ed752da7e40c8167df24e228a9a /docs/GSP_EMULATION_ARCHITECTURE.md
parentbfcb856e09207d8edb714373d775537165d06b05 (diff)
downloadopen-iov-681494a79e6c0e084354eec0229386a696a314c0.tar.gz
open-iov-681494a79e6c0e084354eec0229386a696a314c0.zip
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful material in the structured tree: debugging workflows and the EPT-livelock case study (05-operations/02), performance microbenchmarks and A/B methodology (05-operations/01), the version-upgrade runbook (05-operations/02 section 4, with corrected host/guest framing), kernel-code pitfalls (07-developers) and the OpenRM build/load order (03-architecture/06 section 7). The remaining legacy files were verified redundant and removed. * docs/README now records where each migrated topic lives. * Portability: the config YAML files are called out as design targets that no code reads yet, with what adding support actually requires today. * Getting started: the OpenRM step is marked optional (native vGPU-management workflows only; the emulated path uses the distro driver). * tools docs: rmtool/rmtop marked as planned, pointing at the implemented tools/rmq subset. * wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the canonical userspace/cuda analysis) and index update. * scripts/check-docs-links.py plus a CI step so relative links cannot rot. * STATE.md references to removed documents repointed.
Diffstat (limited to 'docs/GSP_EMULATION_ARCHITECTURE.md')
-rw-r--r--docs/GSP_EMULATION_ARCHITECTURE.md210
1 files changed, 0 insertions, 210 deletions
diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md
deleted file mode 100644
index 70b4b76..0000000
--- a/docs/GSP_EMULATION_ARCHITECTURE.md
+++ /dev/null
@@ -1,210 +0,0 @@
-# Open-IOV GSP Emulation Engine — Architecture
-
-> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept
-> as a record. Current entry point: [docs/README.md](README.md). Architecture
-> replacement:
-> [03-architecture/01](03-architecture/01-system-overview.md) and
-> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md);
-> the transport section in particular is superseded by the msgq
-> documentation in
-> [03-architecture/03](03-architecture/03-gsp-and-gsp-rm.md).
-
-> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`,
-> `kernel/bar_emulation.c`, or debugging guest driver bring-up.
->
-> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21
-> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`;
-> for the original refactor plan see `refactored-architecture.md`.
-
----
-
-## 1. Purpose
-
-An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects
-to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC
-messages to a mailbox in BAR0, rings a doorbell, and waits for a completion
-interrupt plus a response written back into the mailbox.
-
-Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the
-host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the
-host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes
-RPCs, performs them in software (or declines them), writes responses back, and
-injects MSI-X interrupts.
-
-```
-┌──────────────────────────── Guest VM ────────────────────────────┐
-│ stock nvidia.ko (570.144) │
-│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │
-│ ▼ │
-│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │
-└──────────────────────────────────────────────────────────────────┘
- │ (16 MB vzalloc'd shadow, uncached)
-┌──────────────────────────── Host ────────────────────────────────┐
-│ gsp_emu thread (open-iov-vfio.ko) │
-│ poll shadow+0x110C00 (doorbell) │
-│ → read RPC msg from shadow+0x1000 │
-│ → dispatch (12 RPC types, emulation implementations) │
-│ → write response to shadow+0x1080 │
-│ → set PBA bit at shadow+0x3000 │
-│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │
-└──────────────────────────────────────────────────────────────────┘
-```
-
----
-
-## 2. The single source of truth: `bar0_shadow`
-
-The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software
-shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()`
-(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler
-(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`).
-
-Everything the guest driver does with BAR0 lands in this shadow:
-
-| BAR0 offset | Region | Guest writes go to | Emulation reads from |
-|------------------|-----------------------------------|--------------------|----------------------|
-| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow |
-| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` |
-| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads |
-| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) |
-| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads |
-| `0x110C00` | **GSP doorbell** | shadow | thread polls this |
-| rest | emulated register space | shadow | shadow |
-
-> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled
-> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real
-> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer —
-> it could never see guest traffic and even wrote PBA bits into the real GPU's
-> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`.
-> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so
-> cleanup never `dma_free`s it.
-
-### 2.1 Transport protocol (as emulated)
-
-1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000`
- (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`).
-2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`.
-3. The emulation thread sees the doorbell, ACKs it (writes 0), validates
- `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`.
-4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`,
- `result[6]`, `seq_id`) written to `BAR0+0x1080`.
-5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired
- eventfd → QEMU delivers MSI-X to the guest.
-6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and
- accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the
- public `gsp_emu_process_rpc()` entry does not bump counters).
-
----
-
-## 3. RPC dispatch
-
-`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls
-the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation
-implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by
-`gsp_emu_init`).
-
-| Type (val) | Handler path in emulation | Notes |
-|----------------|--------------------------------------------------------|-------|
-| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list |
-| FREE (0x02) | `gsp_rpc_free_emul` | |
-| MAP (0x03) | `gsp_rpc_map_emul` | |
-| UNMAP (0x04) | `gsp_rpc_unmap_emul` | |
-| PIN (0x05) | `gsp_rpc_pin_emul` | |
-| UNPIN (0x06) | `gsp_rpc_unpin_emul` | |
-| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission |
-| YIELD (0x08) | no-op success | |
-| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet |
-| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO |
-| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO |
-| DEBUG (0xFF) | log-only in dispatch | |
-| unknown | `-ENOTSUPP` | |
-
-> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to
-> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox
-> and **rings the same doorbell the thread polls** — a self-triggered infinite
-> RPC loop plus multi-second stalls. Emulation mode now fails these fast with
-> `pr_warn_ratelimited`.
-
-### Memory model (corrected 2026-09-22)
-
-Guest BAR1 (VRAM aperture) is backed by a **dedicated host-RAM vzalloc buffer**
-(256 MB, `bar_init.c`), NOT the host GPU's physical BAR1. Guest VRAM accesses
-hit our clean host-RAM backing. This is a placeholder — for real CUDA compute,
-the memory model must evolve (see `refactored-architecture.md` Phase 2):
-
-- **Current (Phase 1)**: Guest VRAM = host RAM (safe, no host GPU corruption).
- Guest driver probes but cannot compute.
-- **Phase 2**: Guest VRAM backed by DMA-coherent host memory or physical GPU
- VRAM partition (required for CUDA).
-- **Phase 3**: Guest channel/pushbuffer work forwarded to physical GPU
- channels (required for compute).
-
-Guest BAR3 (instance memory window) is also a **vzalloc'd shadow** (32 MB,
-`bar_init.c`), NOT the physical GPU's BAR3. Guest BAR3 writes must not reach
-the host GPU's instance memory (root cause of host GSP crash, fixed `85d0668`).
-
----
-
-## 4. Interrupt delivery (MSI-X)
-
-1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`.
- Our ioctl handler takes `eventfd_ctx_fdget()` references into
- `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs).
-2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`:
- sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger
- from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after*
- `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a
- real bug fixed 2026-09-21).
-
-INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator
-currently fires vector 0 for GSP completion.
-
----
-
-## 5. Lifecycle
-
-```
-mdev create (nv_vgpu_mdev_create)
- └─ schedule init_work
- └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow)
- └─ nv_vgpu_irq_init / nv_vgpu_memory_init
- └─ gsp_emu_init
- ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases)
- ├─ gsp_client_init_emulation (RPC client, emulation_mode=true)
- ├─ gsp_rpc_emulation_set(true)
- └─ kthread_run(gsp_emu_thread_fn)
-
-mdev remove (nv_vgpu_mdev_remove)
- └─ cancel init_work, wait for VFIO fds to close
- └─ vfio_unregister_group_dev
- └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion)
- → memory → irq → BAR (frees bar0_shadow)
- └─ list_del, pci_dev_put, vfio_put_device
-```
-
----
-
-## 6. Debugging the emulator
-
-- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'`
-- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on
- the thread path only)
-- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub
- `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation
- checklist.
-- Unit test: `kernel/tests/gsp_emulation_test.ko` drives
- `gsp_emu_process_rpc()` with crafted mailbox messages (requires
- `open-iov-vfio.ko` loaded; `depends:` is set accordingly).
-
----
-
-## 7. Key files
-
-| File | Role |
-|------|------|
-| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection |
-| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs |
-| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state |
-| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space |
-| `kernel/memory.c` | simple mapping/pin tracking (host management API) |
-| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) |