diff options
Diffstat (limited to 'wiki/Host-Module.md')
| -rw-r--r-- | wiki/Host-Module.md | 160 |
1 files changed, 160 insertions, 0 deletions
diff --git a/wiki/Host-Module.md b/wiki/Host-Module.md new file mode 100644 index 0000000..acf98ed --- /dev/null +++ b/wiki/Host-Module.md @@ -0,0 +1,160 @@ +# The Host Module (vgsp.ko) + +> **Scope.** The host-side kernel module that presents one virtual +> Turing-class GPU per `mdev` instance: the virtual device, the emulation +> engine, the guest-memory model and the work-forwarding design. Canonical +> source: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md); +> the docs tree wins on disagreement with this page. +> +> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with +> the stock guest driver; `[wip]` demonstrated live, being productized; +> `[gated]` implemented default-off behind supervised gates; `[wall]` known +> blocking limit; `[target]` designed, not built; `[planned]` roadmap. + +## 1. Role and position + +```text +guest VM ── PCIe MMIO / DMA / MSI-X through the hypervisor + (KVM/QEMU or Xen: VFIO mdev transport, IOMMU) + │ +vgsp.ko: virtual device · emulation core · memory model · work path + │ +physical GPU + GSP firmware, driven by this module (host driver unmodified) +``` + +The module registers as an **mdev parent** on the physical GPU, exposing one +device type per supported profile; each `mdev` is an independent virtual GPU +([`System-Overview.md`](System-Overview.md), [`Execution-Plane.md`](Execution-Plane.md)). + +## 2. The virtual device + +The guest sees a plausible Turing-class GPU (IDs, class code, BARs, capability +list). Some fields exist only for validation: PCIe link capabilities (the +hypervisor can rewrite them), MSI-X masking state, and non-SR-IOV VF windows. + +| BAR | Size | Backing | Contents | +|---|---|---|---| +| BAR0 | 16 MB | host `vzalloc` shadow | boot, doorbells, VF registers, PRAMIN window, VBIOS | +| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture | +| BAR3 | — | instance window | instance/PRAMIN behaviour | + +MMIO is served by VFIO region callbacks or directly from the shadow when +mapped into the guest. BAR1 is host RAM today — correctness first: the real +GPU cannot read it, so forwarding must later migrate objects to real VRAM or +expose DMA-mapped system memory [target] ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). + +## 3. The emulation core + +A kernel thread drives the device's "firmware" behaviour [live]: + +```text +BAR0 doorbell / VF-register write → read msgq element from guest memory + → dispatch RPC → write reply → inject MSI-X (eventfd → irqfd) +``` + +Guest memory is reached via `vfio_dma_rw()` from kernel-thread context. The +RPC philosophy: correct shape first (or the driver walks off into +uninitialized memory), plausible values second (values must be +self-consistent), never invent input (in/out parameter blocks are echoed; +zeroing the `GSP_RM_ALLOC` reply once faulted `libcuda`). Handlers cover +boot/static-info exchanges, device/FB/BUS/INTR tables, CE and GMMU queries, +reserved-PDE contracts, channel/CE allocation and an unknown-control fallback. +Push buffers are decoded as method streams and executed in software — +memsets, copies, semaphore releases — a **correctness bridge, not the data +plane** [wip]; emulation-mode CUDA is verified end to end (`vectorAdd` over +1M elements, `bad=0`) [live]. Physical-GSP boot is default-off [gated]; the +firmware analyzer, version tables, HAL tables and dynamic RPC dispatcher are +not yet on the live path [target]. + +## 4. The guest-memory model + +**What the client gives us:** channel memory descriptors +(`NV_MEMORY_DESC_PARAMS`) with *physical* addresses (instance memory, method +buffer, userd, ramfc, error notifiers) and the page-table contract for +server-owned VA ranges (`COPY_SERVER_RESERVED_PDES` — table-level PAs, page +shifts, VA range, *empty* leaves). It gives us none of its own sysmem maps. + +**What we derive instead** [live] — the module rebuilds the missing mappings +from the guest's own bookkeeping, with no cooperation from the guest driver: + +1. **VA↔PA pairs live in guest memory**: UVM and RM record both the GPU VA and + the kernel CPU VA per object (`uvm_rm_mem_t.vas[CPU]/vas[GPU]` +0x30/+0x38). +2. **CPU VAs are direct-map addresses**: `PA = cpu_va - page_offset_base`, the + base recovered from guest memory or `/proc/kallsyms` during development. +3. **Exception:** control-device `mmap()` pages have no leaf PTEs; verified + pairs arrive via `sema_map_add` until a durable join exists. The page-granular + cache is fed by a trusted GMMU walk ([`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)). + +**Load-bearing copy invariants:** (1) a copy runs at most once per GPFIFO +entry — replays are suppressed by entry-word identity plus an executed-operation +bitmap; (2) a virtual operand translates per 4 KB page (195 of 256 pages were +contiguous in one observed 1 MB range); (3) a consumed slot's work never runs +later — the per-put-epoch mark is authoritative until a put wrap. + +## 5. The forwarding path (design) + +```text +guest push submitted + → detect (doorbell / GPPUT / push-buffer scan), decode the method stream + → EMULATE state/notification (release, membar, fence) or + FORWARD real engine work (GR launch, CE copy, memop) + → FORWARD: submit equivalent work on a real channel bound to the physical GPU + → relay completions (semaphore / fault / interrupt) back into guest memory +``` + +Proxying through the distribution host driver has **no callable entry point +from kernel context**, so the module **owns the device and runs its real GSP +firmware as the resource-manager server**, with emulation as the control +plane. CE copies, release and fault relay are implemented default-off and +verified live [gated]; GR execution is the open wall [wall] — a forwarded +launch reaches the front-end; the inline launch-descriptor install is unobserved +([`Execution-Frontier.md`](Execution-Frontier.md)). + +## 6. Debug parameters + +`/sys/module/vgsp/parameters/*` makes guest memory and the emulated device +inspectable without rebuilding: + +| Parameter | Purpose | +|---|---| +| `scan_guest_ram=<v>` / `scan_vidmem=<v>` | find a 64-bit value in guest RAM (0–8 GB) or the VRAM shadow | +| `scan_seq=<v1,...>` / `scan_pt=<pa>` / `scan_masked=<v>,<mask>` | find a 32-bit sequence, entries pointing at a PA, or `(x & mask) == v` | +| `dump_vidmem=<off>,<len>` / `fill_vidmem=<v>` | dump/scan or fill the VRAM shadow | +| `poke_guest=<gpa>,<v>` / `poke_fill=<gpa>,<v>,<n>` | write a 32-bit value / fill a range in guest physical memory | +| `sema_probe=<gpu_va>[,count]` / `sema_map_add=<va>,<pa>` | probe a semaphore VA / teach one verified VA→PA pair | +| `dump_guest=<gpa>,<len>` | dump guest physical memory (capped) | + +The physical-backend path adds a default-off set of backend controls +(documented in the repository; workflows in +[`../docs/05-operations/02-debugging-and-maintenance-workflows.md`](../docs/05-operations/02-debugging-and-maintenance-workflows.md)). + +## 7. Build and symbol contracts + +* `vgsp.ko` exports the `nvidia_vgpu_vfio_*` interface (`probe`/`remove`, ops + get/set, mdev parent registration) to pair the device with OpenRM host + modules. +* OpenRM modules built from source need + `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers` and must build **after** + this module (missing symbols and stale build caches are common otherwise). +* On single-GPU developer systems the distribution driver owns the physical + GPU and the module runs standalone; the OpenRM path is only for native-vGPU + management, not QEMU. + +## 8. Implemented vs designed + +| Area | Status | +|---|---| +| mdev/VFIO device, config space, BARs, MSI-X; BAR0 state machine; boot handshake [live] | implemented | +| Message queues + RPC + init controls (stock 570.144) [live] | implemented | +| Guest sysmem access + VA↔PA discovery [wip] | debug-grade, to be productized | +| Push decoding + semaphore completion [wip] | demonstrated live, being productized | +| Emulation-mode CUDA end-to-end [live] | repeatable 1M-element `vectorAdd`, `bad=0` | +| Guest-RAM physical path + server-owned page-table population [gated] | default-off; GPA→HPA via shared-file page cache verified live on the CE | +| GPU work forwarding: CE copy, release and fault relay [gated] | default-off, verified live; GR open [wall] | +| Execution backend (module-owned GPU, direct real-GSP session) [wall] | boot, static configuration and CE copies proven; GR execution open (failed mutating attempts park the device in a locked fail-safe, never auto-retried) | +| Real VRAM backing [target]; Windows, Xen, aarch64 [planned] | designed, not implemented | + +## 9. Related documentation + +* Canonical chapter: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md) — the source of this condensation. +* [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md), [`BAR-Emulation-Specification.md`](BAR-Emulation-Specification.md), [`WPR-Management.md`](WPR-Management.md); [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md), [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md). |