path: root/docs/03-architecture/08-physical-gpu-integration.md

# Physical GPU Integration: How Work Reaches Real Hardware

> **Status: mechanism chosen; ownership + firmware staging implemented;
> power-domain lock verified; first real GSP boot and first live GSP-RM
> exchange complete (2026-10-01); host-side initialization follow-up and a
> clean release are in progress.** The control plane works today: the guest's
> driver initializes and the CUDA driver-API baseline passes end to end
> (context creation, allocation, copies, 64/64 payload round-trip), and the
> guest's decoded work is executed in software as a development bridge. This
> document records the interfaces that actually exist, the mechanism options,
> the chosen path, the platform power-domain hazard the ownership path
> exposes on the development host (§4.2), and what has to be built. It fills
> the gap left open by
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
> which named "the host driver's RM API" as an option without a concrete call
> path from kernel context.

## 1. The problem and its constraints

The guest's real work (copies, memory operations, grid launches) must execute
on the physical GPU. The project constraints are:

* one DKMS kernel module on the host; no daemon, coprocess or script is
  required for the guest to run;
* host and guest drivers unmodified; no paravirtual enlightenment in the
  guest — it believes it owns a physical GPU;
* all emulation and translation logic inside the module;
* real execution: compute must not be emulated in software.

Any mechanism must therefore be reachable from kernel context, with no new
host userspace, and without editing the host driver.

## 2. What interfaces actually exist

| Interface | What it offers | Reachable from our module? |
|---|---|---|
| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |

The conclusion: with an unmodified host driver there is **no callable
submission interface into it**. Real execution therefore requires the module
to own the device and talk to the hardware itself.

## 3. Mechanism options

```
 (A) Proxy through the host driver
     guest push -> our module -> host driver's RM -> real channel -> GPU
     Not implementable without a modified host driver or a vendor-managed
     interface (see section 2). Re-checked against the stock distribution
     build on the development host: it exports no resource-manager or vGPU
     symbols to other modules and installs no manager daemon, so there is no
     exported submission interface to use.

 (B) Own the GPU; own scheduling (no GSP)
     guest push -> our module -> our own channels/runlists/page tables -> GPU
     Possible in principle, but the module would reproduce the scheduler,
     MMU ownership and context switching; large and capability-limiting.

 (C) Hybrid
     Only meaningful once (A) and (B) both exist; not on the critical path.

 (D) Own the GPU; use its real GSP firmware as the server   <- chosen
     guest push -> our module -> real GSP-RM session -> real channel -> GPU
     The module becomes the host-side resource-manager client: the same
     protocol the guest driver speaks, used against the physical GPU's
     firmware. Scheduling, page tables and context management stay the
     firmware's job.

 (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
     Deployment-gated (licence, hardware generation); out of scope for
     consumer GPUs.
```

## 4. The chosen mechanism in detail

### 4.1 Ownership and lifecycle

The module binds the physical GPU as its driver. The distribution driver stays
unmodified and is unbound from that device while the vGPU service is active;
host display continues on another GPU (integrated graphics on the development
system). This is the dedicated-GPU deployment model; see section 5. An earlier
host-side GSP boot path existed in early versions and was disabled precisely
because it conflicted with the distribution driver owning the card — under
exclusive ownership that conflict disappears.

Releasing the distribution driver is not a neutral operation on every
platform, and this host proves it: the board's power domain belongs to the
platform, not to PCI (§4.2). The ownership sequence therefore holds an
explicit D0 lock over the whole board before the release and until after the
distribution driver has been rebound and probed. Without that lock, returning
to the normal driver is a dice roll: it has completed on clean cycles, but the
recovery is conditional and must never be assumed.

### 4.2 Board power-domain hazard and D0 lock

The GPU, its audio function, its USB host controller and its type-C connector
are one physical board behind a PCIe root port. That root port's firmware
declares the board's power resource, and the board rail itself is switched by
the platform's embedded controller. While the distribution driver is bound and
using the GPU, the board is held powered by the GPU's runtime activity. When
the last driver is released, the now-driverless GPU is allowed to suspend, the
root port autosuspends into D3, and the platform runs the power-resource *off*
sequence: the embedded controller is commanded to cut the board rail. The
companion USB controller witnesses this on its next resume by reporting that
its root hub lost power — that signature appears on every unbind, including
the clean cycles.

Re-binding must run the matching *on* sequence, and that sequence
intermittently fails on the development host. In the two observed failures the
GPU stopped answering config-space reads after the re-power (the PCI core's
readiness poll ran `not ready ... after resume` ladders), and the host
hard-froze roughly a minute later: no oops, no panic, no crash record; the
journal simply stops. The distribution driver's remove path itself requests no
power state (it only disables the device); the power-down is done by the PCI
core plus the platform firmware layer, so changing a sysfs unbind into an
in-kernel release changes nothing. Both freezes followed an unbind/rebind (one
an explicit sysfs cycle, one the module's claim/release path); the module
itself never logged an error in either event, and in the second event it had
already been unloaded when the freeze happened inside the distribution
driver's re-bind power-up path. Hold-time correlation exists in the small
sample (short holds failed, longer holds survived) but is not a proven
threshold and is not relied on.

**The lock.** Before releasing the distribution driver, the ownership path
must put the board's whole power domain into a held-D0 state:

* forbid runtime power management on the GPU, on each of its board companion
  functions, and on the root port — the platform must not be allowed to
  autosuspend any of them;
* disallow D3cold on the same set of devices, so the PCI core cannot route the
  bridge into the platform's D3 power-resource sequence even if runtime power
  management were re-enabled;
* verify the GPU's runtime status is active and its config space readable
  before the release proceeds.

Both controls live in the device's core state and are not reset by driver
unbind, so the lock survives the whole window: release → claim → hold →
re-bind. While the module owns the device it verifies that the GPU still
answers config space before proceeding and aborts on any power-loss signature
(the companion controller reporting a lost root hub, or a failed config read).
The distribution driver is then rebound with the lock still held, its probe is
allowed to complete, and only then is runtime power management restored to
automatic. Any power-loss signature aborts the sequence and forbids a retry in
that boot: an out-of-band platform power cut bypasses the lock and would
produce the identical signature, and retrying would be a second dice roll on a
host that has already lost the board.

The lock is implemented in the ownership path and verified across three
supervised claim/release cycles on 2026-10-01 (§6, §8); a claim is refused
unless both the claim parameter and the hazard acknowledgement are set (§8).

### 4.3 Device bring-up

PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
WPR2 setup, message queues in host system memory, doorbell, version exchange —
the same observable stages the guest emulation already models, with the module
now on the client side.

The bring-up is staged so each step is reversible and independently verified:

1. **Ownership** — the board power domain is locked in D0 and verified active
   (§4.2); the distribution driver is then released and the module binds the
   device, maps BAR0/1/3 and samples the falcon state, with the power-loss
   checks staying armed while the module owns the device. On release the
   module unbinds, the distribution driver is rebound with the lock still
   held, and runtime power management is restored only after its probe
   completes. Recovery to the normal driver has completed on every clean
   cycle but is conditional on the lock, never assumed.
2. **Firmware staging** — the matching GSP image (the same file the
   distribution driver would load) is fetched and validated (ELF64 RISC-V;
   the loadable image section, firmware version and hardware signature blob
   are identified and held for the boot stage). No device state is touched.
3. **Boot** — a protected region is laid out in video memory (firmware heap,
   image, boot binary, reserved areas), the secure-verification stage runs on
   the security falcon to authorize it, the booter is then run on the
   security falcon with the address of the boot metadata, and the GSP RISC-V
   core is started; success is observed as an active RISC-V core plus the
   firmware's resource manager completing initialization.
4. **Client** — the module speaks the already-mapped RPC protocol to the real
   firmware as the client (section 4.4).

Stages 1 and 2 are implemented in the module and verified live (a claim/release
cycle leaves the host driver working on the clean cycles; the staged image
parses as expected). The power-domain lock of §4.2 guards the claim window and
is verified across three supervised cycles. Stage 3 has now run live
(2026-10-01), in the observed order **FWSEC → Booter Load → RISC-V active →
first server event**: FWSEC raised WPR2 with a zero FRTS error code
(`WPR2_LO=0x01ffe000`, `WPR2_HI=0x01ffee00`), the SEC2 Booter Load halted with
`M0=0` after 146 ms, the RISC-V core reported active
(`RISCV_STATUS=0x00000035`) with `FALCON_OS=0`, and a host-driven boot sets
only the booter-load handoff bit (bit 24) in secure scratch14. Stage 4 has
begun: the staged command/status queues linked and the first real GSP-RM
message arrived (event 0x1002, run-CPU-sequencer, `rpc_result=0`). Before the
Booter Load kick the module also queues the two pre-boot asynchronous init
RPCs the production client sends in the same spot: `GSP_SET_SYSTEM_INFO` (72,
the host/GPU address and PCI environment description GSP-RM consumes before it
creates its GPU object) and `SET_REGISTRY` (73; a header-only empty table,
which the server explicitly accepts). The system-info payload also carries the
chipset description the production client syncs into the firmware before boot:
the core-logic property bitmask, the `Chipset` value and two host-bridge
BUSINFO blobs (device/vendor/subsystem ids and revision of the first host
bridge of the GPU's domain), built from the live PCI topology
(root-port/chipset class match, upstream link and LTR capabilities). The ACPI
method-data block is sent invalid/zero: the boot path deliberately makes no
ACPI/AML calls and every consumer tolerates the no-data encoding. The server
waits for the host to execute the sequencer command buffer carried in that
event before it raises `GSP_INIT_DONE` (0x1001). **A plain release is not yet
possible:** the Booter Unload gate requires all four secure scratch14 handoff
bits (24–27); a
host-driven boot sets only bit 24, and bit 27 is set by the server's shutdown
path (host RPC `UNLOADING_GUEST_DRIVER`; implemented in the module as the
rung-4b command-queue TX + shutdown path and pending its first live run),
while the
host-side seed/unlock alternative is fuse-locked on this part. Until that RPC
has run in a live session, a green session is left claimed with the harness
`--keep` flag and returning the GPU to the distribution driver takes a power
cycle. The supervised harness modes are `--boot` (the only write mode: full
boot lifecycle), `--boot-dryrun` (zero-write staging rehearsal) and `--keep`
(leave a verified claimed session running); `--fwsec` is retired.

### 4.4 Host-side resource-manager client

Using the wire protocol and RPC inventory already mapped for the supported
family, the module allocates real objects: device/subdevice, virtual address
spaces, channels, and memory mappings. This replaces "call the host driver's
RM" with "be the RM client".

### 4.5 Guest mediation

The guest-facing GSP-RM emulation stays as it is for everything the guest
observes. Objects that need real resources are translated into real GSP-RM
requests (handle-domain translation, allocation backing). The page-table
contract the guest already provides for server-owned ranges becomes
population of the real MMU instead of a shadow.

### 4.6 Work path

Decoded guest pushes are rebuilt into real push streams on real channels
(where shapes match, passed through); guest GPU virtual addresses are mapped
to the same physical pages in the real MMU; the real doorbell is rung and the
GSP firmware schedules the work.

### 4.7 Completion relay

Hardware-completed semaphores land in memory the guest already polls. Events
and faults that the guest expects as interrupts are relayed through the
existing MSI-X injection path.

### 4.8 Isolation

One real GSP client per mdev, per-client page tables; host access to guest
memory stays behind the hypervisor's DMA interface; the firmware provides
per-context scheduling. Tenant policy and fairness are a later layer on top.

## 5. Sharing policy

* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
  another GPU or integrated graphics.
* **Multi-GPU host:** one GPU for the distribution driver, one for the
  service.
* **Shared single GPU with the host driver bound:** no conforming mechanism
  exists on this generation without vendor interfaces; this would require
  relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
  daemon).

## 6. Status and milestones

| Step | Status |
|---|---|
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build |
| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
| Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) |
| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8) |
| Real GSP boot by the module | first live boot 2026-10-01 — FWSEC WPR2 up, Booter Load `M0=0`, RISC-V active; clean release pending the server shutdown RPC |
| Host-side resource-manager client for real objects/channels | transport alive 2026-10-01 — first real GSP-RM event (0x1002) received; init-done (0x1001) awaits host sequencer execution |
| Work translation and real submission | not started |
| Completion relay through real hardware | not started |
| CUDA kernel execution (`vectorAdd`) | not started |

## 7. Open questions

* Firmware/host driver ABI match for the supported family on the development
  GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics when work runs on the real firmware.
* Per-tenant scheduling policy once multiple mdevs share the service GPU.
* **Residual platform risk:** the D0 lock covers the PCI/runtime-PM route to
  the board's power resource. An out-of-band power cut — a platform or
  controller action that does not go through the root port's D3 path — would
  produce the identical power-loss signature; the claim must abort on that
  signature and must not be retried in the same boot.
* **Supervised operation of the lock:** the lock is implemented and verified
  across three supervised claim/release cycles (2026-10-01). Every claim
  remains supervised: explicit human go-ahead, other host work warned, a
  written recovery plan in place, a held-open window long enough to prove the
  board stayed powered, and pstore armed.

## 8. Decision record

* **Date:** 2026-09-29.
* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
  session as the execution backend. (A) is recorded as not implementable under
  the constraints; (B) remains a fallback only if (D) proves unworkable;
  (E) is deployment-gated and out of scope.
* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
  drivers, real execution, and a path to multi-tenant sharing.
* **Consequences:** the module grows a host-side resource-manager client and a
  device-ownership lifecycle; the emulation path remains the control plane and
  is reused.
* **Update (2026-10-01) — operating constraint:** never claim the physical GPU
  unless `gsp_phys_gpu=1` **and** `gsp_phys_hazard_ack=1`, the power-domain
  lock (§4.2) is active, and the test is supervised under a written recovery
  plan. The claim path refuses without the acknowledgement. At the first
  power-loss signature: abort and do not retry in that boot; a host reboot
  requires explicit human approval and a written recovery plan updated first.
* **Update (2026-10-01) — FWSEC-only retired:** stopping after the
  secure-verification carve alone is a one-way door on this fused part (the
  protected region is left up and is not host-recoverable). The supervised
  claim harness therefore refuses `--fwsec` before any module load. The only
  device-write mode is `--boot` (`gsp_phys_boot=1`, the full boot lifecycle
  through resource-manager init); the zero-write rehearsal is `--boot-dryrun`.
  Never stop after FWSEC alone on this host.
* **Update (2026-10-01) — first live boot; clean release gated:** the
  supervised `--boot --keep` cycle completed the first real boot (FWSEC →
  Booter Load → RISC-V active) and received the first real GSP-RM event
  (0x1002, run-CPU-sequencer); `GSP_INIT_DONE` (0x1001) awaits the host
  executing the sequencer buffer carried in that event. Clean release is not
  yet possible: the Booter Unload gate requires all four secure scratch14
  handoff bits (24–27), a host-driven boot sets only bit 24, bit 27 is set by
  the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`), and
  host-side seed/unlock is fuse-locked on this part. The live session is
  therefore left claimed (`--keep`) and a fresh cycle needs a power cycle.
  See §4.3.
* **Update (2026-10-01) — rung 4b implemented (build-only):** the module now
  produces real command-queue records to GSP-RM and attempts a clean shutdown
  on release. The producer writes the 48-byte transport element header plus
  the 32-byte rpc header (signature `0x43505256`, header version `0x03000000`,
  `rpc.length = 32 + payload`), an XOR-fold 64→32 checksum over the
  zero-padded record that makes a consumer's fold read zero, an incrementing
  sequence number, and publishes the wrap-around `writePtr` under a DMA write
  barrier before ringing the queue-head doorbell at `0x110c00`. The shutdown
  RPC is `UNLOADING_GUEST_DRIVER` (function 47) with `bInPMTransition=0`,
  `bGc6Entering=0`, `newLevel=0`; the wait criterion is the GSP falcon
  mailbox0 reading `0x80000000` (processor-suspended), bounded at 10 s. On a
  release the RPC runs once (only when the full boot stage ran) before the
  existing quiesce/FWSEC teardown; without a full boot the fail-safe
  behaviour is unchanged. A write-only debugfs trigger
  (`open-iov-vfio/gsp-phys-unload-rpc`, write "1") lets a live claimed
  session test the shutdown without touching WPR2. All of it is inert unless
  the full boot stage ran; the code has not yet run live, and this bullet
  must be updated with the live outcome.
* **Update (2026-10-01) — rung 4c slice 1 implemented (build-only):** for the
  next cycle a host resource-manager static-info probe is available: the
  write-only debugfs file `open-iov-vfio/gsp-phys-rm-probe` (write "1", only
  when GSP-RM init-done was observed) sends `GET_GSP_STATIC_INFO` over the
  real command queue and decodes the server's static configuration (internal
  handles, split-VA flag, BAR1/2 PDE bases, framebuffer geometry, engine
  caps), falling back once to `SET_GUEST_SYSTEM_INFO` plus one retry; the
  command-queue producer now splits records across up to 16 queue elements
  and the status-queue walk coalesces multi-element records. Not yet run
  live; the module was built only.

## 9. Interaction sequence (end state)

```mermaid
sequenceDiagram
  autonumber
  participant GD as Guest driver
  participant EMU as GSP-RM emulation - control
  participant BE as Backend - real GSP client
  participant GSP as Real GSP firmware
  participant ENG as Engines

  GD->>EMU: objects, vaspaces, channel creation
  EMU->>BE: translate to real objects [planned]
  BE->>GSP: allocate real objects and channels
  GD->>EMU: work submissions
  EMU->>BE: decoded and translated push streams
  BE->>ENG: submit on real channels
  ENG-->>BE: completions and faults
  BE->>EMU: relay into guest memory
  EMU-->>GD: trackers and events advance
```

The full interaction and scheduling view (queues, interrupts, timing classes,
multi-tenant policy) is in
[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
system map is in [`09-full-system-picture.md`](09-full-system-picture.md).

## See also

- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view