diff options
Diffstat (limited to 'docs/03-architecture/06-host-module-and-hybrid-model.md')
| -rw-r--r-- | docs/03-architecture/06-host-module-and-hybrid-model.md | 227 |
1 files changed, 227 insertions, 0 deletions
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md new file mode 100644 index 0000000..fc4bb23 --- /dev/null +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -0,0 +1,227 @@ +# The Host Module and the Hybrid Model + +> `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the +> memory model, and how work will be forwarded. This is the "where the code +> lives" document. + +## 1. Role and position + +``` + guest VM + │ PCIe MMIO / DMA / MSI-X + ┌────┴──────────────────────────────────────────────┐ + │ hypervisor (KVM/QEMU or Xen) │ + │ VFIO mdev transport, IOMMU (per-tenant memory) │ + └────┬──────────────────────────────────────────────┘ + │ + ┌────┴───────────────────────────────────────────────────────────────┐ + │ open-iov-vfio.ko │ + │ virtual device ── emulation core ── memory model ── work path │ + └────┬─────────────────────────────────────┬─────────────────────────┘ + │ host driver interfaces │ (future) vendor-neutral + ┌────┴────────────────────┐ │ + │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends + └──────────────────────────┘ +``` + +The module registers itself as an **mdev parent** on the physical GPU and +exposes one device type per supported profile. Each `mdev` instance is an +independent virtual GPU with its own state, its own BAR windows, its own +interrupts and its own emulation context. + +## 2. Virtual device + +### 2.1 PCI configuration space + +The guest sees a plausible Turing-class GPU: vendor/device IDs, class code, +BARs, capability list (PCIe, MSI-X, power management), and the subsystem +identifiers the driver keys on. Several fields exist only because the driver +reads them for validation: + +* **PCIe link capabilities** — the guest's driver derives bus width/speed and + availability. The emulation must report them; the *hypervisor* can + accidentally rewrite them (see the debugging doc's root-port note). +* **MSI-X table/PBA** — emulated, with per-vector masking and pending bits. +* **VF (virtual function) windows** — registers the SR-IOV-aware driver paths + read during initialization; the emulation fills them with values consistent + with a non-SR-IOV or lightly-SR-IOV configuration as appropriate. + +### 2.2 BAR layout + +| BAR | Size | Backing | Notes | +|---|---|---|---| +| BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS | +| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase | +| BAR3 | — | instance window | instance/PRAMIN behaviour | + +MMIO reads and writes are served either by the module's VFIO region callbacks +or directly from the shadow when the hypervisor has mapped it into the guest. + +### 2.3 VRAM backing + +Today BAR1 is host RAM. This is deliberate: correctness first, performance and +real VRAM semantics later. Consequences and their mitigations: + +* Guest reads/writes of "VRAM" are cheap and coherent. ✅ +* The real GPU cannot read it. The forwarding phase must either migrate these + objects to real VRAM or expose a DMA-mapped system-memory region to the GPU + (see §5). + +## 3. Emulation core + +### 3.1 GSP emulation thread + +A kernel thread drives the device's "firmware" behaviour: + +1. Watches BAR0 doorbells (and VF registers) for guest actions. +2. Reads message-queue elements from guest memory. +3. Dispatches RPCs to handlers. +4. Writes replies into the status queue. +5. Injects MSI-X interrupts to the guest. + +Guest memory is accessed through the hypervisor's DMA interface +(`vfio_dma_rw()`), which requires calls to originate from a kernel thread +context — a fact the debug tooling codifies. + +### 3.2 RPC handling + +Handlers are organized by function and, inside controls, by control ID. The +implementation philosophy: + +* **Correct shape first.** A reply must have the exact structure the driver + expects, or the driver walks off into uninitialized memory. +* **Plausible values second.** Values must be self-consistent (e.g. FB size, + BAR sizes, engine counts, page sizes) or later code paths compute nonsense. +* **Never echo garbage.** Queries that pass uninitialized input must receive + zeroed/valid output; echoing the request's stack garbage has caused real + failures. + +The handlers currently cover: the boot/static-info exchanges, the device/FB/ +BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, +vaspace reserved-PDE contracts, channel/CE object allocation, and the general +"unknown control returns success with zeroed output" fallback. + +### 3.3 Event channels and interrupts + +Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor. +The emulation tracks vector masking and pending state so that an interrupt +raised while masked is delivered when unmasked, not lost. + +## 4. Memory model (the heart of the project) + +The guest's GPU work refers to *GPU virtual addresses*. To act on that work the +host module must translate them to memory it can touch. + +### 4.1 What the client gives us + +* **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical* + addresses: instance memory (VRAM offset), method buffer, userd, ramfc, + error notifiers. These are precise and trustworthy. +* **The page-table contract** for server-owned VA ranges + (`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page + shifts, VA range — but with *empty* leaf levels, because the server owns the + leaves. +* **No mapping RPCs for the client's own system-memory objects.** The client + allocates sysmem locally and never tells the server its physical pages. + +### 4.2 What we derive instead + +Because the client does not hand us its sysmem mappings, the module derives +them from the guest's own in-memory bookkeeping: + +1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that + record, for every mapped object, both the GPU VA and the kernel CPU VA. + Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at + +0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`. +2. **CPU VAs are direct-map addresses**, so + `PA = cpu_va - page_offset_base`. The direct-map base can be recovered from + guest memory itself (structures that store their own address), from + `/proc/kallsyms` during development, or from a debug poke session. +3. Therefore a translation table can be built **without any cooperation from + the guest driver** — a crucial property for adversarial interoperability, + and a general technique applicable to future versions. + +### 4.3 Acting on translations + +The module can read and write guest physical memory (via the hypervisor's DMA +interface). That is enough to: + +* complete semaphore releases (write the payload to the mapped slot), +* inspect push buffers and page tables, +* and later, create page tables for server-owned ranges. + +## 5. The forwarding path (design) + +``` +guest push submitted + → module detects submission (doorbell/GPPUT or push-buffer scan) + → decode the push (method stream) + → for each command decide: + (a) EMULATE: pure state/notification (semaphore release, membar, fence) + (b) FORWARD: real engine work (GR grid launch, CE copy, memop) + → for (b): submit equivalent work on a real channel bound to the physical GPU + → relay completions (semaphore/fault/interrupt) back into guest memory +``` + +Three forwarding backends are candidates, in increasing order of capability: + +1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real + memory and create real channels, then replay guest work through them. +2. **Direct engine submission** — build our own channel/push streams to real + engines using the GMMU/HAL definitions, bypassing the host driver. +3. **Hybrid** — proxy for management (memory, channels) and direct submission + for hot-path work (grid launches, copies). + +The decision between them is driven by the practical challenges document; the +short version is that (1) is safest for correctness, (2) is best for latency, +and both must be isolated to be acceptable. + +## 6. Debug tooling in the module + +The module exposes debug parameters (writes to +`/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the +emulated device inspectable without rebuilding: + +| Parameter | Purpose | +|---|---| +| `scan_guest_ram=<v>` | find a 64-bit value in guest RAM (0–8 GB), report GPAs | +| `scan_vidmem=<v>` | same for the emulated VRAM shadow | +| `scan_seq=<v1,v2,...>` | find a 32-bit *sequence* (e.g. a method stream) | +| `scan_pt=<pa>` | find page-table entries pointing at a physical address | +| `scan_masked=<v>,<mask>` | find `(x & mask) == v` (e.g. a page-base VA) | +| `dump_vidmem=<off>,<len>` / `summary` | dump/scan the VRAM shadow | +| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory | +| `poke_fill=<gpa>,<v>,<n>` | fill a range | +| `fill_vidmem=<v>` | fill the VRAM shadow | + +These are development-grade but production-useful: they are how the memory +model was reverse-engineered, and they remain the fastest way to diagnose a +mismatch. Their workflows are documented in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 7. Build and symbol contracts + +* `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface + (`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for + hosts that want to pair our device with the NVIDIA OpenRM host modules. +* When NVIDIA's OpenRM modules are built from source, they must be built with + `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the + Open-IOV module, or symbols will not resolve. +* On typical single-GPU developer systems the distribution's NVIDIA driver + manages the physical GPU and the Open-IOV module is used standalone; the + OpenRM submodule path is only needed for the native-vGPU-management + workflows. + +## 8. What is implemented vs. designed + +| Area | Status | +|---|---| +| mdev/VFIO device, config space, BARs, MSI-X | implemented | +| BAR0 register state machine, boot handshake | implemented | +| Message queues + RPC + all init controls observed from stock 570.144 | implemented | +| Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | +| Push decoding + semaphore completion | demonstrated live; being productized | +| Page-table population for server-owned ranges | designed, not implemented | +| GPU work forwarding (GR/CE execution) | designed, not implemented | +| Real VRAM backing | designed, not implemented | +| Windows guest, Xen transport, aarch64 hosts | designed, not implemented | |