path: root/docs/03-architecture/06-host-module-and-hybrid-model.md

# The Host Module and the Hybrid Model

> `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the
> memory model, and how work will be forwarded. This is the "where the code
> lives" document.

## 1. Role and position

```
      guest VM
        │  PCIe MMIO / DMA / MSI-X
   ┌────┴──────────────────────────────────────────────┐
   │ hypervisor (KVM/QEMU or Xen)                      │
   │   VFIO mdev transport, IOMMU (per-tenant memory)  │
   └────┬──────────────────────────────────────────────┘
        │
   ┌────┴───────────────────────────────────────────────────────────────┐
   │ open-iov-vfio.ko                                                   │
   │   virtual device ── emulation core ── memory model ── work path    │
   └────┬─────────────────────────────────────┬─────────────────────────┘
        │ module-owned device + real GSP      │ (future) vendor-managed
   ┌────┴───────────────────────────────┐      │  sharing backends
   │ physical GPU and its GSP firmware, │      vGPU host driver / SR-IOV
   │ driven by this module (the host    │      (licensed / hardware-gated)
   │ driver stays unmodified, unbound)  │
   └────────────────────────────────────┘
```

The module registers itself as an **mdev parent** on the physical GPU and
exposes one device type per supported profile. Each `mdev` instance is an
independent virtual GPU with its own state, its own BAR windows, its own
interrupts and its own emulation context.

```mermaid
sequenceDiagram
  autonumber
  participant ORCH as Orchestrator
  participant SYS as Module and mdev sysfs
  participant QEMU as QEMU with vfio-pci
  participant KVM as KVM
  participant GUEST as Guest

  ORCH->>SYS: create mdev for a VM
  QEMU->>SYS: open VFIO device, mmap regions, set IRQ eventfds
  QEMU->>KVM: map guest RAM, register irqfd
  GUEST->>SYS: MMIO and DMA through the emulated device
  SYS-->>GUEST: MSI-X via eventfd to irqfd
  ORCH->>SYS: destroy mdev after the VM stops
```

The full interaction and scheduling detail is in
[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
end-to-end maps are in
[`09-full-system-picture.md`](09-full-system-picture.md).

## 2. Virtual device

### 2.1 PCI configuration space

The guest sees a plausible Turing-class GPU: vendor/device IDs, class code,
BARs, capability list (PCIe, MSI-X, power management), and the subsystem
identifiers the driver keys on. Several fields exist only because the driver
reads them for validation:

* **PCIe link capabilities** — the guest's driver derives bus width/speed and
  availability. The emulation must report them; the *hypervisor* can
  accidentally rewrite them (see the debugging doc's root-port note).
* **MSI-X table/PBA** — emulated, with per-vector masking and pending bits.
* **VF (virtual function) windows** — registers the SR-IOV-aware driver paths
  read during initialization; the emulation fills them with values consistent
  with a non-SR-IOV or lightly-SR-IOV configuration as appropriate.

### 2.2 BAR layout

| BAR | Size | Backing | Notes |
|---|---|---|---|
| BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS |
| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase |
| BAR3 | — | instance window | instance/PRAMIN behaviour |

MMIO reads and writes are served either by the module's VFIO region callbacks
or directly from the shadow when the hypervisor has mapped it into the guest.

### 2.3 VRAM backing

Today BAR1 is host RAM. This is deliberate: correctness first, performance and
real VRAM semantics later. Consequences and their mitigations:

* Guest reads/writes of "VRAM" are cheap and coherent. ✅
* The real GPU cannot read it. The forwarding phase must either migrate these
  objects to real VRAM or expose a DMA-mapped system-memory region to the GPU
  (see §5).

## 3. Emulation core

### 3.1 GSP emulation thread

A kernel thread drives the device's "firmware" behaviour:

1. Watches BAR0 doorbells (and VF registers) for guest actions.
2. Reads message-queue elements from guest memory.
3. Dispatches RPCs to handlers.
4. Writes replies into the status queue.
5. Injects MSI-X interrupts to the guest.

Guest memory is accessed through the hypervisor's DMA interface
(`vfio_dma_rw()`), which requires calls to originate from a kernel thread
context — a fact the debug tooling codifies.

### 3.2 RPC handling

Handlers are organized by function and, inside controls, by control ID. The
implementation philosophy:

* **Correct shape first.** A reply must have the exact structure the driver
  expects, or the driver walks off into uninitialized memory.
* **Plausible values second.** Values must be self-consistent (e.g. FB size,
  BAR sizes, engine counts, page sizes) or later code paths compute nonsense.
* **Never invent input.** Queries that pass uninitialized *input* must not read
  it; but in/out parameter blocks must be echoed back, because the driver
  copies the reply's class parameters over its own buffer at the size the
  kernel driver declares. Zeroing an in/out block is destructive: the
  `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and
  faulted `libcuda` (fixed by echoing the request parameters).

The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
"unknown control/function returns success with zeroed output" fallback. The
same fallback mattered for engine inventory: `libcuda` filters the engine
list against the class list, and an incomplete inventory once left a record
zeroed and blocked context creation (`cuCtxCreate` returned `999`); the
emulated inventory is now complete (the async copy engine included). See
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the analysis.

**Current work-execution bridge.** The module also decodes push buffers as
method streams and executes the work it understands in software - memsets,
copies and semaphore releases - and completes the corresponding semaphores.
That is a correctness bridge for memory operations, not the data plane: no
CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend
(a module-owned GPU running its real GSP firmware as the resource-manager
server) is in
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the
trade-offs in
[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
section 3.

**Subsystem status note.** Some components are compiled but not currently
wired into the live path: the physical-GSP load/boot path is skipped in favour
of emulation, and the runtime firmware analyzer, version tables, hardware
abstraction tables and dynamic RPC dispatcher are not yet called by the live
flow. Treat their descriptions here as intended design, not active code.

### 3.3 Event channels and interrupts

Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor.
The emulation tracks vector masking and pending state so that an interrupt
raised while masked is delivered when unmasked, not lost.

## 4. Memory model (the heart of the project)

The guest's GPU work refers to *GPU virtual addresses*. To act on that work the
host module must translate them to memory it can touch.

### 4.1 What the client gives us

* **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical*
  addresses: instance memory (VRAM offset), method buffer, userd, ramfc,
  error notifiers. These are precise and trustworthy.
* **The page-table contract** for server-owned VA ranges
  (`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page
  shifts, VA range — but with *empty* leaf levels, because the server owns the
  leaves.
* **No mapping RPCs for the client's own system-memory objects.** The client
  allocates sysmem locally and never tells the server its physical pages.

### 4.2 What we derive instead

Because the client does not hand us its sysmem mappings, the module derives
them from the guest's own in-memory bookkeeping:

1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that
   record, for every mapped object, both the GPU VA and the kernel CPU VA.
   Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at
   +0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`.
2. **CPU VAs are direct-map addresses**, so
   `PA = cpu_va - page_offset_base`. The direct-map base can be recovered from
   guest memory itself (structures that store their own address), from
   `/proc/kallsyms` during development, or from a debug poke session.
3. Therefore a translation table can be built **without any cooperation from
   the guest driver** — a crucial property for adversarial interoperability,
   and a general technique applicable to future versions.

### 4.3 Acting on translations

The module can read and write guest physical memory (via the hypervisor's DMA
interface). That is enough to:

* complete semaphore releases (write the payload to the mapped slot),
* inspect push buffers and page tables,
* and later, create page tables for server-owned ranges.

Releases are recognised in the decoded method stream in both of the encodings
the driver uses: the copy-engine `SET_SEMAPHORE_A/B` + `LAUNCH_DMA` group and
the TURING+ `SET_REPORT_SEMAPHORE_A..D` group
(`{offset_upper, offset_lower, payload, operation}`, operation 0 = release),
which is how the host class ends a push. The learned translation cache is
page-granular, so all semaphore slots of one page resolve from one entry.

One class of pages is not derivable from the structures above: client
mappings created by `mmap()` on the control device, where the kernel driver
maps sysmem pages into the process and the *same* numeric address doubles as
the GPU VA (observed live: a 2 MB shared mapping of the control device, and
push releases whose semaphore address equals the mapped client VA). Those
pages have no leaf PTEs in the GPU page tables (the client delegates them),
and the kernel-side bookkeeping records physical pages without a GPU VA.
Until a durable join is implemented, a session can feed verified pairs with
the `sema_map_add` debug parameter.

## 5. The forwarding path (design)

```
guest push submitted
   → module detects submission (doorbell/GPPUT or push-buffer scan)
   → decode the push (method stream)
   → for each command decide:
        (a) EMULATE: pure state/notification (semaphore release, membar, fence)
        (b) FORWARD: real engine work (GR grid launch, CE copy, memop)
   → for (b): submit equivalent work on a real channel bound to the physical GPU
   → relay completions (semaphore/fault/interrupt) back into guest memory
```

Backend selection, including the interfaces that actually exist and why the
distribution host driver cannot be called, is analyzed in
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short:
proxying through the host driver has no callable entry point from kernel
context, so the chosen mechanism is for the module to own the device and run
its real GSP firmware as the resource-manager server, while the emulation path
remains the control plane. Software execution of decoded memory/state
operations stays a development bridge, not the data plane.

## 6. Debug tooling in the module

The module exposes debug parameters (writes to
`/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the
emulated device inspectable without rebuilding:

| Parameter | Purpose |
|---|---|
| `scan_guest_ram=<v>` | find a 64-bit value in guest RAM (0–8 GB), report GPAs |
| `scan_vidmem=<v>` | same for the emulated VRAM shadow |
| `scan_seq=<v1,v2,...>` | find a 32-bit *sequence* (e.g. a method stream) |
| `scan_pt=<pa>` | find page-table entries pointing at a physical address |
| `scan_masked=<v>,<mask>` | find `(x & mask) == v` (e.g. a page-base VA) |
| `dump_vidmem=<off>,<len>` / `summary` | dump/scan the VRAM shadow |
| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory |
| `poke_fill=<gpa>,<v>,<n>` | fill a range |
| `fill_vidmem=<v>` | fill the VRAM shadow |
| `sema_probe=<gpu_va>[,count]` | probe a semaphore VA and nearby state in guest memory |
| `sema_map_add=<va>,<pa>` | teach the service one verified `{GPU VA -> guest PA}` pair |
| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) |

These are development-grade but production-useful: they are how the memory
model was reverse-engineered, and they remain the fastest way to diagnose a
mismatch. Their workflows are documented in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md).

## 7. Build and symbol contracts

* `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface
  (`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for
  hosts that want to pair our device with the NVIDIA OpenRM host modules.
* When NVIDIA's OpenRM modules are built from source, they must be built with
  `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the
  Open-IOV module, or symbols will not resolve.
* On typical single-GPU developer systems the distribution's NVIDIA driver
  manages the physical GPU and the Open-IOV module is used standalone; the
  OpenRM submodule path is only needed for the native-vGPU-management
  workflows (RM callbacks) and is unnecessary for the QEMU/emulated path.

**Build and load order on the OpenRM path:**

```bash
# 1. Kernel module first - generates kernel/Module.symvers.
make kernel                      # or: ./scripts/stack_manager.sh build-kernel

# 2. OpenRM modules with our symbols resolved.
./scripts/build_nvidia_openrm.sh # wraps KBUILD_EXTRA_SYMBOLS=<repo>/kernel/Module.symvers
grep -i vgpu kernel/Module.symvers                       # symbols exported
modinfo .../kernel-open/nvidia.ko | grep -i vgpu         # symbols consumed

# 3. Load order: Open-IOV first (exports and registers the mdev parent),
#    then the optional OpenRM modules, then firmware.
```

If a distribution driver already owns the physical GPU, skip step 2 and the
OpenRM part of step 3: the emulated QEMU path runs with the distribution
driver. Common failures on this path: `Unknown symbol nvidia_vgpu_vfio_*`
(`KBUILD_EXTRA_SYMBOLS` missing or wrong) and symbols present in
`Module.symvers` but absent from `nvidia.ko` (stale build cache — `make clean`
in `kernel-open` first).

## 8. What is implemented vs. designed

| Area | Status |
|---|---|
| mdev/VFIO device, config space, BARs, MSI-X | implemented |
| BAR0 register state machine, boot handshake | implemented |
| Message queues + RPC + all init controls observed from stock 570.144 | implemented |
| Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) |
| Push decoding + semaphore completion | demonstrated live; being productized |
| Page-table population for server-owned ranges | designed, not implemented |
| GPU work forwarding (GR/CE execution) | designed, not implemented |
| Execution backend (module-owned GPU + direct real-GSP session) | mechanism chosen; ownership + firmware staging verified live; power-domain lock implemented, pending supervised verification; real GSP boot not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) |
| Real VRAM backing | designed, not implemented |
| Windows guest, Xen transport, aarch64 hosts | designed, not implemented |