# The Host Module (vgsp.ko)

> **Scope.** The host-side kernel module that presents one virtual
> Turing-class GPU per `mdev` instance: the virtual device, the emulation
> engine, the guest-memory model and the work-forwarding design. Canonical
> source: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md);
> the docs tree wins on disagreement with this page.
>
> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with
> the stock guest driver; `[wip]` demonstrated live, being productized;
> `[gated]` implemented default-off behind supervised gates; `[wall]` known
> blocking limit; `[target]` designed, not built; `[planned]` roadmap.

## 1. Role and position

```text
guest VM ── PCIe MMIO / DMA / MSI-X through the hypervisor
   (KVM/QEMU or Xen: VFIO mdev transport, IOMMU)
   │
vgsp.ko: virtual device · emulation core · memory model · work path
   │
physical GPU + GSP firmware, driven by this module (host driver unmodified)
```

The module registers as an **mdev parent** on the physical GPU, exposing one
device type per supported profile; each `mdev` is an independent virtual GPU
([`System-Overview.md`](System-Overview.md), [`Execution-Plane.md`](Execution-Plane.md)).

## 2. The virtual device

The guest sees a plausible Turing-class GPU (IDs, class code, BARs, capability
list). Some fields exist only for validation: PCIe link capabilities (the
hypervisor can rewrite them), MSI-X masking state, and non-SR-IOV VF windows.

| BAR | Size | Backing | Contents |
|---|---|---|---|
| BAR0 | 16 MB | host `vzalloc` shadow | boot, doorbells, VF registers, PRAMIN window, VBIOS |
| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture |
| BAR3 | — | instance window | instance/PRAMIN behaviour |

MMIO is served by VFIO region callbacks or directly from the shadow when
mapped into the guest. BAR1 is host RAM today — correctness first: the real
GPU cannot read it, so forwarding must later migrate objects to real VRAM or
expose DMA-mapped system memory [target] ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).

## 3. The emulation core

A kernel thread drives the device's "firmware" behaviour [live]:

```text
BAR0 doorbell / VF-register write → read msgq element from guest memory
   → dispatch RPC → write reply → inject MSI-X (eventfd → irqfd)
```

Guest memory is reached via `vfio_dma_rw()` from kernel-thread context. The
RPC philosophy: correct shape first (or the driver walks off into
uninitialized memory), plausible values second (values must be
self-consistent), never invent input (in/out parameter blocks are echoed;
zeroing the `GSP_RM_ALLOC` reply once faulted `libcuda`). Handlers cover
boot/static-info exchanges, device/FB/BUS/INTR tables, CE and GMMU queries,
reserved-PDE contracts, channel/CE allocation and an unknown-control fallback.
Push buffers are decoded as method streams and executed in software —
memsets, copies, semaphore releases — a **correctness bridge, not the data
plane** [wip]; emulation-mode CUDA is verified end to end (`vectorAdd` over
1M elements, `bad=0`) [live]. Physical-GSP boot is default-off [gated]; the
firmware analyzer, version tables, HAL tables and dynamic RPC dispatcher are
not yet on the live path [target].

## 4. The guest-memory model

**What the client gives us:** channel memory descriptors
(`NV_MEMORY_DESC_PARAMS`) with *physical* addresses (instance memory, method
buffer, userd, ramfc, error notifiers) and the page-table contract for
server-owned VA ranges (`COPY_SERVER_RESERVED_PDES` — table-level PAs, page
shifts, VA range, *empty* leaves). It gives us none of its own sysmem maps.

**What we derive instead** [live] — the module rebuilds the missing mappings
from the guest's own bookkeeping, with no cooperation from the guest driver:

1. **VA↔PA pairs live in guest memory**: UVM and RM record both the GPU VA and
   the kernel CPU VA per object (`uvm_rm_mem_t.vas[CPU]/vas[GPU]` +0x30/+0x38).
2. **CPU VAs are direct-map addresses**: `PA = cpu_va - page_offset_base`, the
   base recovered from guest memory or `/proc/kallsyms` during development.
3. **Exception:** control-device `mmap()` pages have no leaf PTEs; verified
   pairs arrive via `sema_map_add` until a durable join exists. The page-granular
   cache is fed by a trusted GMMU walk ([`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)).

**Load-bearing copy invariants:** (1) a copy runs at most once per GPFIFO
entry — replays are suppressed by entry-word identity plus an executed-operation
bitmap; (2) a virtual operand translates per 4 KB page (195 of 256 pages were
contiguous in one observed 1 MB range); (3) a consumed slot's work never runs
later — the per-put-epoch mark is authoritative until a put wrap.

## 5. The forwarding path (design)

```text
guest push submitted
   → detect (doorbell / GPPUT / push-buffer scan), decode the method stream
   → EMULATE state/notification (release, membar, fence) or
     FORWARD real engine work (GR launch, CE copy, memop)
   → FORWARD: submit equivalent work on a real channel bound to the physical GPU
   → relay completions (semaphore / fault / interrupt) back into guest memory
```

Proxying through the distribution host driver has **no callable entry point
from kernel context**, so the module **owns the device and runs its real GSP
firmware as the resource-manager server**, with emulation as the control
plane. CE copies, release and fault relay are implemented default-off and
verified live [gated]; GR execution is the open wall [wall] — a forwarded
launch reaches the front-end; the inline launch-descriptor install is unobserved
([`Execution-Frontier.md`](Execution-Frontier.md)).

## 6. Debug parameters

`/sys/module/vgsp/parameters/*` makes guest memory and the emulated device
inspectable without rebuilding:

| Parameter | Purpose |
|---|---|
| `scan_guest_ram=<v>` / `scan_vidmem=<v>` | find a 64-bit value in guest RAM (0–8 GB) or the VRAM shadow |
| `scan_seq=<v1,...>` / `scan_pt=<pa>` / `scan_masked=<v>,<mask>` | find a 32-bit sequence, entries pointing at a PA, or `(x & mask) == v` |
| `dump_vidmem=<off>,<len>` / `fill_vidmem=<v>` | dump/scan or fill the VRAM shadow |
| `poke_guest=<gpa>,<v>` / `poke_fill=<gpa>,<v>,<n>` | write a 32-bit value / fill a range in guest physical memory |
| `sema_probe=<gpu_va>[,count]` / `sema_map_add=<va>,<pa>` | probe a semaphore VA / teach one verified VA→PA pair |
| `dump_guest=<gpa>,<len>` | dump guest physical memory (capped) |

The physical-backend path adds a default-off set of backend controls
(documented in the repository; workflows in
[`../docs/05-operations/02-debugging-and-maintenance-workflows.md`](../docs/05-operations/02-debugging-and-maintenance-workflows.md)).

## 7. Build and symbol contracts

* `vgsp.ko` exports the `nvidia_vgpu_vfio_*` interface (`probe`/`remove`, ops
  get/set, mdev parent registration) to pair the device with OpenRM host
  modules.
* OpenRM modules built from source need
  `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers` and must build **after**
  this module (missing symbols and stale build caches are common otherwise).
* On single-GPU developer systems the distribution driver owns the physical
  GPU and the module runs standalone; the OpenRM path is only for native-vGPU
  management, not QEMU.

## 8. Implemented vs designed

| Area | Status |
|---|---|
| mdev/VFIO device, config space, BARs, MSI-X; BAR0 state machine; boot handshake [live] | implemented |
| Message queues + RPC + init controls (stock 570.144) [live] | implemented |
| Guest sysmem access + VA↔PA discovery [wip] | debug-grade, to be productized |
| Push decoding + semaphore completion [wip] | demonstrated live, being productized |
| Emulation-mode CUDA end-to-end [live] | repeatable 1M-element `vectorAdd`, `bad=0` |
| Guest-RAM physical path + server-owned page-table population [gated] | default-off; GPA→HPA via shared-file page cache verified live on the CE |
| GPU work forwarding: CE copy, release and fault relay [gated] | default-off, verified live; GR open [wall] |
| Execution backend (module-owned GPU, direct real-GSP session) [wall] | boot, static configuration and CE copies proven; GR execution open (failed mutating attempts park the device in a locked fail-safe, never auto-retried) |
| Real VRAM backing [target]; Windows, Xen, aarch64 [planned] | designed, not implemented |

## 9. Related documentation

* Canonical chapter: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md) — the source of this condensation.
* [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md), [`BAR-Emulation-Specification.md`](BAR-Emulation-Specification.md), [`WPR-Management.md`](WPR-Management.md); [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md), [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md).