# Physical GPU Integration: How Work Reaches Real Hardware

> **Status: chosen mechanism, not yet implemented.** The control plane works
> today (the guest's driver initializes and CUDA user-space reaches context
> creation); the guest's *work* is currently executed in software as a
> development bridge. This document records the interfaces that actually
> exist, the mechanism options, the chosen path, and what has to be built.
> It fills the gap left open by
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
> which named "the host driver's RM API" as an option without a concrete call
> path from kernel context.

## 1. The problem and its constraints

The guest's real work (copies, memory operations, grid launches) must execute
on the physical GPU. The project constraints are:

* one DKMS kernel module on the host; no daemon, coprocess or script is
  required for the guest to run;
* host and guest drivers unmodified; no paravirtual enlightenment in the
  guest — it believes it owns a physical GPU;
* all emulation and translation logic inside the module;
* real execution: compute must not be emulated in software.

Any mechanism must therefore be reachable from kernel context, with no new
host userspace, and without editing the host driver.

## 2. What interfaces actually exist

| Interface | What it offers | Reachable from our module? |
|---|---|---|
| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |

The conclusion: with an unmodified host driver there is **no callable
submission interface into it**. Real execution therefore requires the module
to own the device and talk to the hardware itself.

## 3. Mechanism options

```
 (A) Proxy through the host driver
     guest push -> our module -> host driver's RM -> real channel -> GPU
     Not implementable without a modified host driver or a vendor-managed
     interface (see section 2).

 (B) Own the GPU; own scheduling (no GSP)
     guest push -> our module -> our own channels/runlists/page tables -> GPU
     Possible in principle, but the module would reproduce the scheduler,
     MMU ownership and context switching; large and capability-limiting.

 (C) Hybrid
     Only meaningful once (A) and (B) both exist; not on the critical path.

 (D) Own the GPU; use its real GSP firmware as the server   <- chosen
     guest push -> our module -> real GSP-RM session -> real channel -> GPU
     The module becomes the host-side resource-manager client: the same
     protocol the guest driver speaks, used against the physical GPU's
     firmware. Scheduling, page tables and context management stay the
     firmware's job.

 (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
     Deployment-gated (licence, hardware generation); out of scope for
     consumer GPUs.
```

## 4. The chosen mechanism in detail

### 4.1 Ownership and lifecycle

The module binds the physical GPU as its driver. The distribution driver stays
unmodified and is unbound from that device while the vGPU service is active;
host display continues on another GPU (integrated graphics on the development
system). This is the dedicated-GPU deployment model; see section 5. An earlier
host-side GSP boot path existed in early versions and was disabled precisely
because it conflicted with the distribution driver owning the card — under
exclusive ownership that conflict disappears.

### 4.2 Device bring-up

PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
WPR2 setup, message queues in host system memory, doorbell, version exchange —
the same observable stages the guest emulation already models, with the module
now on the client side.

The bring-up is staged so each step is reversible and independently verified:

1. **Ownership** — the distribution driver is released from the device and the
   module binds it; BAR0/1/3 are mapped and the falcon state is sampled. On
   release the module unbinds and the distribution driver is rebound, so the
   host can always return to its normal driver.
2. **Firmware staging** — the matching GSP image (the same file the
   distribution driver would load) is fetched and validated (ELF64 RISC-V;
   the loadable image section, firmware version and hardware signature blob
   are identified and held for the boot stage). No device state is touched.
3. **Boot** — a protected region is laid out in video memory (firmware heap,
   image, boot binary, reserved areas), the secure-verification stage runs on
   the security falcon to authorize it, the booter is then run on the
   security falcon with the address of the boot metadata, and the GSP RISC-V
   core is started; success is observed as an active RISC-V core plus the
   firmware's resource manager completing initialization.
4. **Client** — the module speaks the already-mapped RPC protocol to the real
   firmware as the client (section 4.3).

Stages 1 and 2 are implemented in the module and verified live (a claim/release
cycle leaves the host driver working; the staged image parses as expected);
stage 3 is the current bring-up target.

### 4.3 Host-side resource-manager client

Using the wire protocol and RPC inventory already mapped for the supported
family, the module allocates real objects: device/subdevice, virtual address
spaces, channels, and memory mappings. This replaces "call the host driver's
RM" with "be the RM client".

### 4.4 Guest mediation

The guest-facing GSP-RM emulation stays as it is for everything the guest
observes. Objects that need real resources are translated into real GSP-RM
requests (handle-domain translation, allocation backing). The page-table
contract the guest already provides for server-owned ranges becomes
population of the real MMU instead of a shadow.

### 4.5 Work path

Decoded guest pushes are rebuilt into real push streams on real channels
(where shapes match, passed through); guest GPU virtual addresses are mapped
to the same physical pages in the real MMU; the real doorbell is rung and the
GSP firmware schedules the work.

### 4.6 Completion relay

Hardware-completed semaphores land in memory the guest already polls. Events
and faults that the guest expects as interrupts are relayed through the
existing MSI-X injection path.

### 4.7 Isolation

One real GSP client per mdev, per-client page tables; host access to guest
memory stays behind the hypervisor's DMA interface; the firmware provides
per-context scheduling. Tenant policy and fairness are a later layer on top.

## 5. Sharing policy

* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
  another GPU or integrated graphics.
* **Multi-GPU host:** one GPU for the distribution driver, one for the
  service.
* **Shared single GPU with the host driver bound:** no conforming mechanism
  exists on this generation without vendor interfaces; this would require
  relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
  daemon).

## 6. Status and milestones

| Step | Status |
|---|---|
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
| Device ownership + real GSP bring-up by the module | in progress — reversible claim/release and firmware staging verified; the real boot stage is next |
| Host-side resource-manager client for real objects/channels | not started |
| Work translation and real submission | not started |
| Completion relay through real hardware | not started |
| CUDA kernel execution (`vectorAdd`) | not started |

## 7. Open questions

* Firmware/host driver ABI match for the supported family on the development
  GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics when work runs on the real firmware.
* Per-tenant scheduling policy once multiple mdevs share the service GPU.

## 8. Decision record

* **Date:** 2026-09-29.
* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
  session as the execution backend. (A) is recorded as not implementable under
  the constraints; (B) remains a fallback only if (D) proves unworkable;
  (E) is deployment-gated and out of scope.
* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
  drivers, real execution, and a path to multi-tenant sharing.
* **Consequences:** the module grows a host-side resource-manager client and a
  device-ownership lifecycle; the emulation path remains the control plane and
  is reused.

## 9. Interaction sequence (end state)

```mermaid
sequenceDiagram
  autonumber
  participant GD as Guest driver
  participant EMU as GSP-RM emulation - control
  participant BE as Backend - real GSP client
  participant GSP as Real GSP firmware
  participant ENG as Engines

  GD->>EMU: objects, vaspaces, channel creation
  EMU->>BE: translate to real objects [planned]
  BE->>GSP: allocate real objects and channels
  GD->>EMU: work submissions
  EMU->>BE: decoded and translated push streams
  BE->>ENG: submit on real channels
  ENG-->>BE: completions and faults
  BE->>EMU: relay into guest memory
  EMU-->>GD: trackers and events advance
```

The full interaction and scheduling view (queues, interrupts, timing classes,
multi-tenant policy) is in
[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
system map is in [`09-full-system-picture.md`](09-full-system-picture.md).

## See also

- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view