# The Host Module and the Hybrid Model > `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the > memory model, and how work will be forwarded. This is the "where the code > lives" document. ## 1. Role and position ``` guest VM │ PCIe MMIO / DMA / MSI-X ┌────┴──────────────────────────────────────────────┐ │ hypervisor (KVM/QEMU or Xen) │ │ VFIO mdev transport, IOMMU (per-tenant memory) │ └────┬──────────────────────────────────────────────┘ │ ┌────┴───────────────────────────────────────────────────────────────┐ │ open-iov-vfio.ko │ │ virtual device ── emulation core ── memory model ── work path │ └────┬─────────────────────────────────────┬─────────────────────────┘ │ module-owned device + real GSP │ (future) vendor-managed ┌────┴───────────────────────────────┐ │ sharing backends │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV │ driven by this module (the host │ (licensed / hardware-gated) │ driver stays unmodified, unbound) │ └────────────────────────────────────┘ ``` The module registers itself as an **mdev parent** on the physical GPU and exposes one device type per supported profile. Each `mdev` instance is an independent virtual GPU with its own state, its own BAR windows, its own interrupts and its own emulation context. ## 2. Virtual device ### 2.1 PCI configuration space The guest sees a plausible Turing-class GPU: vendor/device IDs, class code, BARs, capability list (PCIe, MSI-X, power management), and the subsystem identifiers the driver keys on. Several fields exist only because the driver reads them for validation: * **PCIe link capabilities** — the guest's driver derives bus width/speed and availability. The emulation must report them; the *hypervisor* can accidentally rewrite them (see the debugging doc's root-port note). * **MSI-X table/PBA** — emulated, with per-vector masking and pending bits. * **VF (virtual function) windows** — registers the SR-IOV-aware driver paths read during initialization; the emulation fills them with values consistent with a non-SR-IOV or lightly-SR-IOV configuration as appropriate. ### 2.2 BAR layout | BAR | Size | Backing | Notes | |---|---|---|---| | BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS | | BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase | | BAR3 | — | instance window | instance/PRAMIN behaviour | MMIO reads and writes are served either by the module's VFIO region callbacks or directly from the shadow when the hypervisor has mapped it into the guest. ### 2.3 VRAM backing Today BAR1 is host RAM. This is deliberate: correctness first, performance and real VRAM semantics later. Consequences and their mitigations: * Guest reads/writes of "VRAM" are cheap and coherent. ✅ * The real GPU cannot read it. The forwarding phase must either migrate these objects to real VRAM or expose a DMA-mapped system-memory region to the GPU (see §5). ## 3. Emulation core ### 3.1 GSP emulation thread A kernel thread drives the device's "firmware" behaviour: 1. Watches BAR0 doorbells (and VF registers) for guest actions. 2. Reads message-queue elements from guest memory. 3. Dispatches RPCs to handlers. 4. Writes replies into the status queue. 5. Injects MSI-X interrupts to the guest. Guest memory is accessed through the hypervisor's DMA interface (`vfio_dma_rw()`), which requires calls to originate from a kernel thread context — a fact the debug tooling codifies. ### 3.2 RPC handling Handlers are organized by function and, inside controls, by control ID. The implementation philosophy: * **Correct shape first.** A reply must have the exact structure the driver expects, or the driver walks off into uninitialized memory. * **Plausible values second.** Values must be self-consistent (e.g. FB size, BAR sizes, engine counts, page sizes) or later code paths compute nonsense. * **Never invent input.** Queries that pass uninitialized *input* must not read it; but in/out parameter blocks must be echoed back, because the driver copies the reply's class parameters over its own buffer at the size the kernel driver declares. Zeroing an in/out block is destructive: the `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and faulted `libcuda` (fixed by echoing the request parameters). The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general "unknown control/function returns success with zeroed output" fallback. The same fallback mattered for engine inventory: `libcuda` filters the engine list against the class list, and an incomplete inventory once left a record zeroed and blocked context creation (`cuCtxCreate` returned `999`); the emulated inventory is now complete (the async copy engine included). See [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the analysis. **Current work-execution bridge.** The module also decodes push buffers as method streams and executes the work it understands in software - memsets, copies and semaphore releases - and completes the corresponding semaphores. That is a correctness bridge for memory operations, not the data plane: no CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend (a module-owned GPU running its real GSP firmware as the resource-manager server) is in [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the trade-offs in [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) section 3. **Subsystem status note.** Some components are compiled but not currently wired into the live path: the physical-GSP load/boot path is skipped in favour of emulation, and the runtime firmware analyzer, version tables, hardware abstraction tables and dynamic RPC dispatcher are not yet called by the live flow. Treat their descriptions here as intended design, not active code. ### 3.3 Event channels and interrupts Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor. The emulation tracks vector masking and pending state so that an interrupt raised while masked is delivered when unmasked, not lost. ## 4. Memory model (the heart of the project) The guest's GPU work refers to *GPU virtual addresses*. To act on that work the host module must translate them to memory it can touch. ### 4.1 What the client gives us * **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical* addresses: instance memory (VRAM offset), method buffer, userd, ramfc, error notifiers. These are precise and trustworthy. * **The page-table contract** for server-owned VA ranges (`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page shifts, VA range — but with *empty* leaf levels, because the server owns the leaves. * **No mapping RPCs for the client's own system-memory objects.** The client allocates sysmem locally and never tells the server its physical pages. ### 4.2 What we derive instead Because the client does not hand us its sysmem mappings, the module derives them from the guest's own in-memory bookkeeping: 1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that record, for every mapped object, both the GPU VA and the kernel CPU VA. Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at +0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`. 2. **CPU VAs are direct-map addresses**, so `PA = cpu_va - page_offset_base`. The direct-map base can be recovered from guest memory itself (structures that store their own address), from `/proc/kallsyms` during development, or from a debug poke session. 3. Therefore a translation table can be built **without any cooperation from the guest driver** — a crucial property for adversarial interoperability, and a general technique applicable to future versions. ### 4.3 Acting on translations The module can read and write guest physical memory (via the hypervisor's DMA interface). That is enough to: * complete semaphore releases (write the payload to the mapped slot), * inspect push buffers and page tables, * and later, create page tables for server-owned ranges. Releases are recognised in the decoded method stream in both of the encodings the driver uses: the copy-engine `SET_SEMAPHORE_A/B` + `LAUNCH_DMA` group and the TURING+ `SET_REPORT_SEMAPHORE_A..D` group (`{offset_upper, offset_lower, payload, operation}`, operation 0 = release), which is how the host class ends a push. The learned translation cache is page-granular, so all semaphore slots of one page resolve from one entry. One class of pages is not derivable from the structures above: client mappings created by `mmap()` on the control device, where the kernel driver maps sysmem pages into the process and the *same* numeric address doubles as the GPU VA (observed live: a 2 MB shared mapping of the control device, and push releases whose semaphore address equals the mapped client VA). Those pages have no leaf PTEs in the GPU page tables (the client delegates them), and the kernel-side bookkeeping records physical pages without a GPU VA. Until a durable join is implemented, a session can feed verified pairs with the `sema_map_add` debug parameter. ## 5. The forwarding path (design) ``` guest push submitted → module detects submission (doorbell/GPPUT or push-buffer scan) → decode the push (method stream) → for each command decide: (a) EMULATE: pure state/notification (semaphore release, membar, fence) (b) FORWARD: real engine work (GR grid launch, CE copy, memop) → for (b): submit equivalent work on a real channel bound to the physical GPU → relay completions (semaphore/fault/interrupt) back into guest memory ``` Backend selection, including the interfaces that actually exist and why the distribution host driver cannot be called, is analyzed in [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short: proxying through the host driver has no callable entry point from kernel context, so the chosen mechanism is for the module to own the device and run its real GSP firmware as the resource-manager server, while the emulation path remains the control plane. Software execution of decoded memory/state operations stays a development bridge, not the data plane. ## 6. Debug tooling in the module The module exposes debug parameters (writes to `/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the emulated device inspectable without rebuilding: | Parameter | Purpose | |---|---| | `scan_guest_ram=` | find a 64-bit value in guest RAM (0–8 GB), report GPAs | | `scan_vidmem=` | same for the emulated VRAM shadow | | `scan_seq=` | find a 32-bit *sequence* (e.g. a method stream) | | `scan_pt=` | find page-table entries pointing at a physical address | | `scan_masked=,` | find `(x & mask) == v` (e.g. a page-base VA) | | `dump_vidmem=,` / `summary` | dump/scan the VRAM shadow | | `poke_guest=,` | write a 32-bit value to guest physical memory | | `poke_fill=,,` | fill a range | | `fill_vidmem=` | fill the VRAM shadow | | `sema_probe=[,count]` | probe a semaphore VA and nearby state in guest memory | | `sema_map_add=,` | teach the service one verified `{GPU VA -> guest PA}` pair | | `dump_guest=,` | dump guest physical memory (length capped) | These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a mismatch. Their workflows are documented in [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). ## 7. Build and symbol contracts * `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface (`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for hosts that want to pair our device with the NVIDIA OpenRM host modules. * When NVIDIA's OpenRM modules are built from source, they must be built with `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the Open-IOV module, or symbols will not resolve. * On typical single-GPU developer systems the distribution's NVIDIA driver manages the physical GPU and the Open-IOV module is used standalone; the OpenRM submodule path is only needed for the native-vGPU-management workflows (RM callbacks) and is unnecessary for the QEMU/emulated path. **Build and load order on the OpenRM path:** ```bash # 1. Kernel module first - generates kernel/Module.symvers. make kernel # or: ./scripts/stack_manager.sh build-kernel # 2. OpenRM modules with our symbols resolved. ./scripts/build_nvidia_openrm.sh # wraps KBUILD_EXTRA_SYMBOLS=/kernel/Module.symvers grep -i vgpu kernel/Module.symvers # symbols exported modinfo .../kernel-open/nvidia.ko | grep -i vgpu # symbols consumed # 3. Load order: Open-IOV first (exports and registers the mdev parent), # then the optional OpenRM modules, then firmware. ``` If a distribution driver already owns the physical GPU, skip step 2 and the OpenRM part of step 3: the emulated QEMU path runs with the distribution driver. Common failures on this path: `Unknown symbol nvidia_vgpu_vfio_*` (`KBUILD_EXTRA_SYMBOLS` missing or wrong) and symbols present in `Module.symvers` but absent from `nvidia.ko` (stale build cache — `make clean` in `kernel-open` first). ## 8. What is implemented vs. designed | Area | Status | |---|---| | mdev/VFIO device, config space, BARs, MSI-X | implemented | | BAR0 register state machine, boot handshake | implemented | | Message queues + RPC + all init controls observed from stock 570.144 | implemented | | Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | | Push decoding + semaphore completion | demonstrated live; being productized | | Page-table population for server-owned ranges | designed, not implemented | | GPU work forwarding (GR/CE execution) | designed, not implemented | | Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented |