# The Host Module and the Hybrid Model > `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the > memory model, and how work will be forwarded. This is the "where the code > lives" document. ## 1. Role and position ``` guest VM │ PCIe MMIO / DMA / MSI-X ┌────┴──────────────────────────────────────────────┐ │ hypervisor (KVM/QEMU or Xen) │ │ VFIO mdev transport, IOMMU (per-tenant memory) │ └────┬──────────────────────────────────────────────┘ │ ┌────┴───────────────────────────────────────────────────────────────┐ │ open-iov-vfio.ko │ │ virtual device ── emulation core ── memory model ── work path │ └────┬─────────────────────────────────────┬─────────────────────────┘ │ host driver interfaces │ (future) vendor-neutral ┌────┴────────────────────┐ │ │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends └──────────────────────────┘ ``` The module registers itself as an **mdev parent** on the physical GPU and exposes one device type per supported profile. Each `mdev` instance is an independent virtual GPU with its own state, its own BAR windows, its own interrupts and its own emulation context. ## 2. Virtual device ### 2.1 PCI configuration space The guest sees a plausible Turing-class GPU: vendor/device IDs, class code, BARs, capability list (PCIe, MSI-X, power management), and the subsystem identifiers the driver keys on. Several fields exist only because the driver reads them for validation: * **PCIe link capabilities** — the guest's driver derives bus width/speed and availability. The emulation must report them; the *hypervisor* can accidentally rewrite them (see the debugging doc's root-port note). * **MSI-X table/PBA** — emulated, with per-vector masking and pending bits. * **VF (virtual function) windows** — registers the SR-IOV-aware driver paths read during initialization; the emulation fills them with values consistent with a non-SR-IOV or lightly-SR-IOV configuration as appropriate. ### 2.2 BAR layout | BAR | Size | Backing | Notes | |---|---|---|---| | BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS | | BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase | | BAR3 | — | instance window | instance/PRAMIN behaviour | MMIO reads and writes are served either by the module's VFIO region callbacks or directly from the shadow when the hypervisor has mapped it into the guest. ### 2.3 VRAM backing Today BAR1 is host RAM. This is deliberate: correctness first, performance and real VRAM semantics later. Consequences and their mitigations: * Guest reads/writes of "VRAM" are cheap and coherent. ✅ * The real GPU cannot read it. The forwarding phase must either migrate these objects to real VRAM or expose a DMA-mapped system-memory region to the GPU (see §5). ## 3. Emulation core ### 3.1 GSP emulation thread A kernel thread drives the device's "firmware" behaviour: 1. Watches BAR0 doorbells (and VF registers) for guest actions. 2. Reads message-queue elements from guest memory. 3. Dispatches RPCs to handlers. 4. Writes replies into the status queue. 5. Injects MSI-X interrupts to the guest. Guest memory is accessed through the hypervisor's DMA interface (`vfio_dma_rw()`), which requires calls to originate from a kernel thread context — a fact the debug tooling codifies. ### 3.2 RPC handling Handlers are organized by function and, inside controls, by control ID. The implementation philosophy: * **Correct shape first.** A reply must have the exact structure the driver expects, or the driver walks off into uninitialized memory. * **Plausible values second.** Values must be self-consistent (e.g. FB size, BAR sizes, engine counts, page sizes) or later code paths compute nonsense. * **Never echo garbage.** Queries that pass uninitialized input must receive zeroed/valid output; echoing the request's stack garbage has caused real failures. The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general "unknown control returns success with zeroed output" fallback. ### 3.3 Event channels and interrupts Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor. The emulation tracks vector masking and pending state so that an interrupt raised while masked is delivered when unmasked, not lost. ## 4. Memory model (the heart of the project) The guest's GPU work refers to *GPU virtual addresses*. To act on that work the host module must translate them to memory it can touch. ### 4.1 What the client gives us * **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical* addresses: instance memory (VRAM offset), method buffer, userd, ramfc, error notifiers. These are precise and trustworthy. * **The page-table contract** for server-owned VA ranges (`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page shifts, VA range — but with *empty* leaf levels, because the server owns the leaves. * **No mapping RPCs for the client's own system-memory objects.** The client allocates sysmem locally and never tells the server its physical pages. ### 4.2 What we derive instead Because the client does not hand us its sysmem mappings, the module derives them from the guest's own in-memory bookkeeping: 1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that record, for every mapped object, both the GPU VA and the kernel CPU VA. Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at +0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`. 2. **CPU VAs are direct-map addresses**, so `PA = cpu_va - page_offset_base`. The direct-map base can be recovered from guest memory itself (structures that store their own address), from `/proc/kallsyms` during development, or from a debug poke session. 3. Therefore a translation table can be built **without any cooperation from the guest driver** — a crucial property for adversarial interoperability, and a general technique applicable to future versions. ### 4.3 Acting on translations The module can read and write guest physical memory (via the hypervisor's DMA interface). That is enough to: * complete semaphore releases (write the payload to the mapped slot), * inspect push buffers and page tables, * and later, create page tables for server-owned ranges. ## 5. The forwarding path (design) ``` guest push submitted → module detects submission (doorbell/GPPUT or push-buffer scan) → decode the push (method stream) → for each command decide: (a) EMULATE: pure state/notification (semaphore release, membar, fence) (b) FORWARD: real engine work (GR grid launch, CE copy, memop) → for (b): submit equivalent work on a real channel bound to the physical GPU → relay completions (semaphore/fault/interrupt) back into guest memory ``` Three forwarding backends are candidates, in increasing order of capability: 1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real memory and create real channels, then replay guest work through them. 2. **Direct engine submission** — build our own channel/push streams to real engines using the GMMU/HAL definitions, bypassing the host driver. 3. **Hybrid** — proxy for management (memory, channels) and direct submission for hot-path work (grid launches, copies). The decision between them is driven by the practical challenges document; the short version is that (1) is safest for correctness, (2) is best for latency, and both must be isolated to be acceptable. ## 6. Debug tooling in the module The module exposes debug parameters (writes to `/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the emulated device inspectable without rebuilding: | Parameter | Purpose | |---|---| | `scan_guest_ram=` | find a 64-bit value in guest RAM (0–8 GB), report GPAs | | `scan_vidmem=` | same for the emulated VRAM shadow | | `scan_seq=` | find a 32-bit *sequence* (e.g. a method stream) | | `scan_pt=` | find page-table entries pointing at a physical address | | `scan_masked=,` | find `(x & mask) == v` (e.g. a page-base VA) | | `dump_vidmem=,` / `summary` | dump/scan the VRAM shadow | | `poke_guest=,` | write a 32-bit value to guest physical memory | | `poke_fill=,,` | fill a range | | `fill_vidmem=` | fill the VRAM shadow | These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a mismatch. Their workflows are documented in [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). ## 7. Build and symbol contracts * `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface (`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for hosts that want to pair our device with the NVIDIA OpenRM host modules. * When NVIDIA's OpenRM modules are built from source, they must be built with `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the Open-IOV module, or symbols will not resolve. * On typical single-GPU developer systems the distribution's NVIDIA driver manages the physical GPU and the Open-IOV module is used standalone; the OpenRM submodule path is only needed for the native-vGPU-management workflows. ## 8. What is implemented vs. designed | Area | Status | |---|---| | mdev/VFIO device, config space, BARs, MSI-X | implemented | | BAR0 register state machine, boot handshake | implemented | | Message queues + RPC + all init controls observed from stock 570.144 | implemented | | Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | | Push decoding + semaphore completion | demonstrated live; being productized | | Page-table population for server-owned ranges | designed, not implemented | | GPU work forwarding (GR/CE execution) | designed, not implemented | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented |