# Physical GPU Backend and Guest Wiring > **Scope.** The design that executes guest work on the real GPU: why the host > driver cannot be the path and how the live GSP-RM session is wired to the > guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) > and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partially built; `[wall]` current blocker; `[target]` planned end state. ## 1. The problem and its constraints The guest's real work (copies, memory operations, grid launches) must execute on the physical GPU. The constraints: one DKMS kernel module on the host, no daemon and no new host userspace; host and guest drivers unmodified, with no paravirtual enlightenment in the guest; all emulation and translation logic inside the module; and real execution — compute is not emulated in software. ## 2. Why the host driver is not the path | Interface | What it offers | Reachable from our module? | |---|---|---| | Host driver (distribution build) | resource management, scheduling, memory | No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface | | Host driver peer-memory exports | pin/map memory for peer devices | Partially — memory only, no submission | | Vendor vGPU host stack | mediated work submission and scheduling | Requires the licensed vGPU host branch and a vGPU-aware guest — excluded | | SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation | | The GPU's own GSP firmware | the resource manager server itself | Yes — once the module owns the device; this is the path used | With an unmodified host driver there is no callable submission interface into it, so the module must own the device and talk to the hardware itself. ## 3. Mechanism options and the decision ```text (A) Proxy through the host driver: module -> host driver's RM -> real channel -> GPU. Needs a modified host driver or a vendor interface — not implementable. (B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page tables -> GPU. Reproduces the scheduler, MMU ownership, context switching. (C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path. (D) Own the GPU; use its real GSP firmware as the server <- chosen guest push -> module -> real GSP-RM session -> real channel -> GPU The module becomes the host-side resource-manager client; scheduling, page tables and context management stay the firmware's job. (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope. ``` Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing. ## 4. The chosen mechanism: bring-up ladder The module binds the physical GPU as its driver while the distribution driver is unbound (dedicated-GPU deployment, §9). Each stage is reversible and independently verified: 1. **Ownership** — the board's power domain is held active across the window (the platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. `[live]` 2. **Firmware staging** — the matching GSP image is fetched and validated (ELF64 RISC-V; loadable section, version, hardware signature); no device state touched. `[live]` 3. **Boot** — protected region in video memory, secure-verification stage, booter, RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. `[live]` 4. **Host RM client** — the module speaks the already-mapped RPC protocol to the real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". `[live]` 5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]` 6. **Work path** — decoded guest pushes become real push streams; resolved host pages map through module-built scratch page tables; the firmware schedules the work. `[gated]` 7. **Completion relay** — hardware-completed semaphores land in guest-polled memory; events and faults relay through the existing MSI-X injection path. `[gated]` ## 5. The two planes and the wiring The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software. ```text GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU ----------------------------------------------------------------------------- INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live] claim: power domain held -> distribution driver released -> module binds the device -> liveness check [live] boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live] mdev + ladder: backend joins the live session; real objects, channels, first real CE copy [live] guest: nvidia.ko probes; emulation serves handshake, static config, RM battery, UVM registration [live] WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell host pump decodes; eligible CE copies forwarded via module-built scratch page tables; GR: launch reaches the front-end, descriptor not seen [wall] DONE engine authors the release in guest memory; host relays via MSI-X [gated] TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown -> protected region down -> distribution driver rebound [live] FAIL a failed mutating operation parks the device in a locked fail-safe; never auto-retried; recovery is supervised [live] ``` ## 6. Proven capabilities | Capability | Status | |---|---| | Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done `[live]` | | Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — 64/64 payload round-trip `[live]` | | Software bridge for decoded memory/state operations | in progress `[wip]` | | Device ownership, firmware staging, board-liveness checks | verified live `[live]` | | Power domain held active across the ownership window | verified across supervised cycles `[live]` | | Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | | Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | | Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` | | CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` | ## 7. The execution frontier The RM client ladder reaches the compute engine (a GR0 channel plus a compute-class object). A forwarded guest CUDA launch now reaches the front-end (ring entry fetched, methods executed) but the inline launch-descriptor install has not been observed and no grid has dispatched; context validity at dispatch is ruled out. The wall is the form of the forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md). A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step. ## 8. Safety and platform discipline * **Dedicated GPU deployment.** One GPU serves the vGPU service; the host uses another GPU or integrated graphics. The distribution driver stays unmodified and is unbound from that device only while the service is active. * **The board's power domain belongs to the platform.** Releasing the last driver can power-gate the board and the power-on sequence can fail; the ownership path holds the whole board in a held-active state across the window, verifies config-space liveness, and aborts on any power-loss signature without retry. * **Supervised cycles.** The device is driven by the module under supervised cycles with a written recovery plan. A failed mutating operation parks the device in a locked fail-safe and is never auto-retried; any reboot requires explicit human approval with a written reason and post-reboot plan first. * **Fail closed.** No forwarding seam programs guest-derived page tables into the real MMU until the population path is proven end to end; the current forwarded path submits engine work only through module-built scratch page tables mapping already-resolved host pages. ## 9. Sharing policy and isolation * **Dedicated GPU (chosen):** one GPU serves the vGPU service; a multi-GPU host gives the distribution driver its own GPU. * **Shared single GPU with the host driver bound:** no conforming mechanism exists on this generation without vendor interfaces; it would require relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon). * Isolation is one real GSP client per mdev with per-client page tables; host access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). ## 10. Open questions * Firmware/host driver ABI match for the supported family on the development GPU. * GPU unbind/reset lifecycle and recovery after guest faults. * Which guest allocations need real VRAM and which can stay in system memory. * Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU. ## 11. Related documentation * [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up. * [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design. * [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane. * [`Host-Module.md`](Host-Module.md) — the module that owns both planes. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings. * [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture.