# Physical GPU Integration: How Work Reaches Real Hardware

> **Status: backend live; init-done, static configuration and clean releases
> proven; CE work forwarding with an engine-authored release and fault relay
> implemented default-off; GR execution is the open wall. The device is
> currently in the designed locked fail-safe awaiting a supervised reboot
> after a GR context-promote attempt wedged GSP-RM.** The control plane works
> today: the guest's driver initializes and the CUDA driver-API baseline
> passes end to end
> (context creation, allocation, copies, 64/64 payload round-trip), and the
> guest's decoded work is executed in software as a development bridge. This
> document records the interfaces that actually exist, the mechanism options,
> the chosen path, the platform power-domain hazard the ownership path
> exposes on the development host (§4.2), and what has to be built. It fills
> the gap left open by
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
> which named "the host driver's RM API" as an option without a concrete call
> path from kernel context.

## 1. The problem and its constraints

The guest's real work (copies, memory operations, grid launches) must execute
on the physical GPU. The project constraints are:

* one DKMS kernel module on the host; no daemon, coprocess or script is
  required for the guest to run;
* host and guest drivers unmodified; no paravirtual enlightenment in the
  guest — it believes it owns a physical GPU;
* all emulation and translation logic inside the module;
* real execution: compute must not be emulated in software.

Any mechanism must therefore be reachable from kernel context, with no new
host userspace, and without editing the host driver.

## 2. What interfaces actually exist

| Interface | What it offers | Reachable from our module? |
|---|---|---|
| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |

The conclusion: with an unmodified host driver there is **no callable
submission interface into it**. Real execution therefore requires the module
to own the device and talk to the hardware itself.

## 3. Mechanism options

```
 (A) Proxy through the host driver
     guest push -> our module -> host driver's RM -> real channel -> GPU
     Not implementable without a modified host driver or a vendor-managed
     interface (see section 2). Re-checked against the stock distribution
     build on the development host: it exports no resource-manager or vGPU
     symbols to other modules and installs no manager daemon, so there is no
     exported submission interface to use.

 (B) Own the GPU; own scheduling (no GSP)
     guest push -> our module -> our own channels/runlists/page tables -> GPU
     Possible in principle, but the module would reproduce the scheduler,
     MMU ownership and context switching; large and capability-limiting.

 (C) Hybrid
     Only meaningful once (A) and (B) both exist; not on the critical path.

 (D) Own the GPU; use its real GSP firmware as the server   <- chosen
     guest push -> our module -> real GSP-RM session -> real channel -> GPU
     The module becomes the host-side resource-manager client: the same
     protocol the guest driver speaks, used against the physical GPU's
     firmware. Scheduling, page tables and context management stay the
     firmware's job.

 (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
     Deployment-gated (licence, hardware generation); out of scope for
     consumer GPUs.
```

## 4. The chosen mechanism in detail

### 4.1 Ownership and lifecycle

The module binds the physical GPU as its driver. The distribution driver stays
unmodified and is unbound from that device while the vGPU service is active;
host display continues on another GPU (integrated graphics on the development
system). This is the dedicated-GPU deployment model; see section 5. An earlier
host-side GSP boot path existed in early versions and was disabled precisely
because it conflicted with the distribution driver owning the card — under
exclusive ownership that conflict disappears.

Releasing the distribution driver is not a neutral operation on every
platform, and this host proves it: the board's power domain belongs to the
platform, not to PCI (§4.2). The ownership sequence therefore holds an
explicit D0 lock over the whole board before the release and until after the
distribution driver has been rebound and probed. Without that lock, returning
to the normal driver is a dice roll: it has completed on clean cycles, but the
recovery is conditional and must never be assumed.

### 4.2 Board power-domain hazard and D0 lock

The GPU, its audio function, its USB host controller and its type-C connector
are one physical board behind a PCIe root port. That root port's firmware
declares the board's power resource, and the board rail itself is switched by
the platform's embedded controller. While the distribution driver is bound and
using the GPU, the board is held powered by the GPU's runtime activity. When
the last driver is released, the now-driverless GPU is allowed to suspend, the
root port autosuspends into D3, and the platform runs the power-resource *off*
sequence: the embedded controller is commanded to cut the board rail. The
companion USB controller witnesses this on its next resume by reporting that
its root hub lost power — that signature appears on every unbind, including
the clean cycles.

Re-binding must run the matching *on* sequence, and that sequence
intermittently fails on the development host. In the two observed failures the
GPU stopped answering config-space reads after the re-power (the PCI core's
readiness poll ran `not ready ... after resume` ladders), and the host
hard-froze roughly a minute later: no oops, no panic, no crash record; the
journal simply stops. The distribution driver's remove path itself requests no
power state (it only disables the device); the power-down is done by the PCI
core plus the platform firmware layer, so changing a sysfs unbind into an
in-kernel release changes nothing. Both freezes followed an unbind/rebind (one
an explicit sysfs cycle, one the module's claim/release path); the module
itself never logged an error in either event, and in the second event it had
already been unloaded when the freeze happened inside the distribution
driver's re-bind power-up path. Hold-time correlation exists in the small
sample (short holds failed, longer holds survived) but is not a proven
threshold and is not relied on.

**The lock.** Before releasing the distribution driver, the ownership path
must put the board's whole power domain into a held-D0 state:

* forbid runtime power management on the GPU, on each of its board companion
  functions, and on the root port — the platform must not be allowed to
  autosuspend any of them;
* disallow D3cold on the same set of devices, so the PCI core cannot route the
  bridge into the platform's D3 power-resource sequence even if runtime power
  management were re-enabled;
* verify the GPU's runtime status is active and its config space readable
  before the release proceeds.

Both controls live in the device's core state and are not reset by driver
unbind, so the lock survives the whole window: release → claim → hold →
re-bind. While the module owns the device it verifies that the GPU still
answers config space before proceeding and aborts on any power-loss signature
(the companion controller reporting a lost root hub, or a failed config read).
The distribution driver is then rebound with the lock still held, its probe is
allowed to complete, and the D0/D3cold locks stay latched by default
(`gsp_phys_repower=0`); restoring runtime power management is an explicit
opt-in through the power-restore path, taken only when the operator chooses
it. Any power-loss signature aborts the sequence and forbids a retry in
that boot: an out-of-band platform power cut bypasses the lock and would
produce the identical signature, and retrying would be a second dice roll on a
host that has already lost the board.

The lock is implemented in the ownership path and verified across three
supervised claim/release cycles on 2026-10-01 (§6, §8); a claim is refused
unless both the claim parameter and the hazard acknowledgement are set (§8).

### 4.3 Device bring-up

PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
WPR2 setup, message queues in host system memory, doorbell, version exchange —
the same observable stages the guest emulation already models, with the module
now on the client side.

The bring-up is staged so each step is reversible and independently verified:

1. **Ownership** — the board power domain is locked in D0 and verified active
   (§4.2); the distribution driver is then released and the module binds the
   device, maps BAR0/1/3 and samples the falcon state, with the power-loss
   checks staying armed while the module owns the device. On release the
   module unbinds, the distribution driver is rebound with the lock still
   held, and runtime power management is restored only after its probe
   completes. Recovery to the normal driver has completed on every clean
   cycle but is conditional on the lock, never assumed.
2. **Firmware staging** — the matching GSP image (the same file the
   distribution driver would load) is fetched and validated (ELF64 RISC-V;
   the loadable image section, firmware version and hardware signature blob
   are identified and held for the boot stage). No device state is touched.
3. **Boot** — a protected region is laid out in video memory (firmware heap,
   image, boot binary, reserved areas), the secure-verification stage runs on
   the security falcon to authorize it, the booter is then run on the
   security falcon with the address of the boot metadata, and the GSP RISC-V
   core is started; success is observed as an active RISC-V core plus the
   firmware's resource manager completing initialization.
4. **Client** — the module speaks the already-mapped RPC protocol to the real
   firmware as the client (section 4.4).

Stages 1 and 2 are implemented in the module and verified live (a claim/release
cycle leaves the host driver working on the clean cycles; the staged image
parses as expected). The power-domain lock of §4.2 guards the claim window and
is verified across three supervised cycles. Stage 3 has now run live
(2026-10-01), in the observed order **FWSEC → Booter Load → RISC-V active →
first server event**: FWSEC raised WPR2 with a zero FRTS error code
(`WPR2_LO=0x01ffe000`, `WPR2_HI=0x01ffee00`), the SEC2 Booter Load halted with
`M0=0` after 146 ms, the RISC-V core reported active
(`RISCV_STATUS=0x00000035`) with `FALCON_OS=0`, and a host-driven boot sets
only the booter-load handoff bit (bit 24) in secure scratch14. Stage 4 has
begun: the staged command/status queues linked and the first real GSP-RM
message arrived (event 0x1002, run-CPU-sequencer, `rpc_result=0`). Before the
Booter Load kick the module also queues the two pre-boot asynchronous init
RPCs the production client sends in the same spot: `GSP_SET_SYSTEM_INFO` (72,
the host/GPU address and PCI environment description GSP-RM consumes before it
creates its GPU object) and `SET_REGISTRY` (73; a header-only empty table,
which the server explicitly accepts). The system-info payload also carries the
chipset description the production client syncs into the firmware before boot:
the core-logic property bitmask, the `Chipset` value and two host-bridge
BUSINFO blobs (device/vendor/subsystem ids and revision of the first host
bridge of the GPU's domain), built from the live PCI topology
(root-port/chipset class match, upstream link and LTR capabilities). The ACPI
method-data block is sent invalid/zero: the boot path deliberately makes no
ACPI/AML calls and every consumer tolerates the no-data encoding. The server
waits for the host to execute the sequencer command buffer carried in that
event before it raises `GSP_INIT_DONE` (0x1001). **Clean release is
proven:** since 2026-10-01 the unload RPC path delivers a clean release with
no reboot on every cycle — the
Booter Unload gate requires all four secure scratch14 handoff bits (24–27); a
host-driven boot sets bit 24, and the server's shutdown path (host RPC
`UNLOADING_GUEST_DRIVER`, rung 4b, §8) sets the rest, while the host-side
seed/unlock alternative is fuse-locked on this part. The exception is a wedged
server: if the server never sets its bit, teardown cannot clear the protected
region and the fail-safe refuses the hand-back (no rebind, D0 locks held, a
reboot with approval required; §7). The supervised harness modes are `--boot`
(the only write mode: full boot lifecycle), `--boot-dryrun` (zero-write
staging rehearsal) and `--keep` (leave a verified claimed session running);
`--fwsec` is retired. The full parameter, debugfs and runbook inventory for
this path is maintained in the engineering-operations document
([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate
those tables.

### 4.4 Host-side resource-manager client

Using the wire protocol and RPC inventory already mapped for the supported
family, the module allocates real objects: device/subdevice, virtual address
spaces, channels, and memory mappings. This replaces "call the host driver's
RM" with "be the RM client".

### 4.5 Guest mediation

The guest-facing GSP-RM emulation stays as it is for everything the guest
observes. Objects that need real resources are translated into real GSP-RM
requests (handle-domain translation, allocation backing). The page-table
contract the guest already provides for server-owned ranges becomes
population of the real MMU instead of a shadow.

### 4.6 Work path

Decoded guest pushes are rebuilt into real push streams on real channels
(where shapes match, passed through); guest GPU virtual addresses are mapped
to the same physical pages in the real MMU; the real doorbell is rung and the
GSP firmware schedules the work.

### 4.7 Completion relay

Hardware-completed semaphores land in memory the guest already polls. Events
and faults that the guest expects as interrupts are relayed through the
existing MSI-X injection path.

### 4.8 Isolation

One real GSP client per mdev, per-client page tables; host access to guest
memory stays behind the hypervisor's DMA interface; the firmware provides
per-context scheduling. Tenant policy and fairness are a later layer on top.

## 5. Sharing policy

* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
  another GPU or integrated graphics.
* **Multi-GPU host:** one GPU for the distribution driver, one for the
  service.
* **Shared single GPU with the host driver bound:** no conforming mechanism
  exists on this generation without vendor interfaces; this would require
  relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
  daemon).

## 6. Status and milestones

| Step | Status |
|---|---|
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build |
| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
| Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) |
| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) |
| Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 |
| Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine |
| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; for GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed (see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) |
| Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage |
| CUDA kernel execution (`vectorAdd`) | open — context validity at dispatch is ruled out; the frontier is the launch descriptor install path (stage 6–7 of [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) |

## 7. Open questions

* Firmware/host driver ABI match for the supported family on the development
  GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics when work runs on the real firmware.
* Per-tenant scheduling policy once multiple mdevs share the service GPU.
* **Residual platform risk:** the D0 lock covers the PCI/runtime-PM route to
  the board's power resource. An out-of-band power cut — a platform or
  controller action that does not go through the root port's D3 path — would
  produce the identical power-loss signature; the claim must abort on that
  signature and must not be retried in the same boot.
* **Wedged server / protected-region fail-safe:** if a live session wedges the
  server so its shutdown path never completes, WPR2 cannot be cleared (the
  host-side unlock is fuse-locked on this part). The release then refuses to
  rebind and keeps the D0 locks; never retry in that boot, and return to the
  distribution driver only through a reboot governed by the project's
  reboot-approval policy (a written reason and post-reboot plan first).
* **Supervised operation of the lock:** the lock is implemented and verified
  across three supervised claim/release cycles (2026-10-01). Every claim
  remains supervised: explicit human go-ahead, other host work warned, a
  written recovery plan in place, a held-open window long enough to prove the
  board stayed powered, and pstore armed.

## 8. Decision record

* **Date:** 2026-09-29.
* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
  session as the execution backend. (A) is recorded as not implementable under
  the constraints; (B) remains a fallback only if (D) proves unworkable;
  (E) is deployment-gated and out of scope.
* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
  drivers, real execution, and a path to multi-tenant sharing.
* **Consequences:** the module grows a host-side resource-manager client and a
  device-ownership lifecycle; the emulation path remains the control plane and
  is reused.
* **Update (2026-10-01) — operating constraint:** never claim the physical GPU
  unless `gsp_phys_gpu=1` **and** `gsp_phys_hazard_ack=1`, the power-domain
  lock (§4.2) is active, and the test is supervised under a written recovery
  plan. The claim path refuses without the acknowledgement. At the first
  power-loss signature: abort and do not retry in that boot; a host reboot
  requires explicit human approval and a written recovery plan updated first.
* **Update (2026-10-01) — FWSEC-only retired:** stopping after the
  secure-verification carve alone is a one-way door on this fused part (the
  protected region is left up and is not host-recoverable). The supervised
  claim harness therefore refuses `--fwsec` before any module load. The only
  device-write mode is `--boot` (`gsp_phys_boot=1`, the full boot lifecycle
  through resource-manager init); the zero-write rehearsal is `--boot-dryrun`.
  Never stop after FWSEC alone on this host.
* **Update (2026-10-01) — first live boot; clean release gated:** the
  supervised `--boot --keep` cycle completed the first real boot (FWSEC →
  Booter Load → RISC-V active) and received the first real GSP-RM event
  (0x1002, run-CPU-sequencer); `GSP_INIT_DONE` (0x1001) awaits the host
  executing the sequencer buffer carried in that event. Clean release is not
  yet possible: the Booter Unload gate requires all four secure scratch14
  handoff bits (24–27), a host-driven boot sets only bit 24, bit 27 is set by
  the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`), and
  host-side seed/unlock is fuse-locked on this part. The live session is
  therefore left claimed (`--keep`) and a fresh cycle needs a power cycle.
  See §4.3. *(Historical: superseded by the clean-release result in the
  unload-RPC update below.)*
* **Update (2026-10-01) — rung 4b implemented (build-only):** the module now
  produces real command-queue records to GSP-RM and attempts a clean shutdown
  on release. The producer writes the 48-byte transport element header plus
  the 32-byte rpc header (signature `0x43505256`, header version `0x03000000`,
  `rpc.length = 32 + payload`), an XOR-fold 64→32 checksum over the
  zero-padded record that makes a consumer's fold read zero, an incrementing
  sequence number, and publishes the wrap-around `writePtr` under a DMA write
  barrier before ringing the queue-head doorbell at `0x110c00`. The shutdown
  RPC is `UNLOADING_GUEST_DRIVER` (function 47) with `bInPMTransition=0`,
  `bGc6Entering=0`, `newLevel=0`; the wait criterion is the GSP falcon
  mailbox0 reading `0x80000000` (processor-suspended), bounded at 10 s. On a
  release the RPC runs once (only when the full boot stage ran) before the
  existing quiesce/FWSEC teardown; without a full boot the fail-safe
  behaviour is unchanged. A write-only debugfs trigger
  (`vgsp/gsp-phys-unload-rpc`, write "1") lets a live claimed
  session test the shutdown without touching WPR2. All of it is inert unless
  the full boot stage ran. **Live-verified (2026-10-01):** the unload RPC
  shuts the server down (GSP-RM processor suspended in ~220 ms, handoff bits
  complete), so WPR2 goes down and the distribution driver is rebound with no
  reboot on every clean cycle.
* **Update (2026-10-01) — rung 4c slice 1 implemented (build-only):** for the
  next cycle a host resource-manager static-info probe is available: the
  write-only debugfs file `vgsp/gsp-phys-rm-probe` (write "1", only
  when GSP-RM init-done was observed) sends `GET_GSP_STATIC_INFO` over the
  real command queue and decodes the server's static configuration (internal
  handles, split-VA flag, BAR1/2 PDE bases, framebuffer geometry, engine
  caps), falling back once to `SET_GUEST_SYSTEM_INFO` plus one retry; the
  command-queue producer now splits records across up to 16 queue elements
  and the status-queue walk coalesces multi-element records. **Live-verified:**
  init-done with the full 1656-byte static-config reply was observed on the
  physical server, and the probe decodes its static configuration (internal
  handles, BAR PDE bases and engine caps); the multi-element producer and
  status-walk coalescing also run live.
* **Update (2026-10-02) — GR execution is the wall; device in locked
  fail-safe:** the resource-manager client ladder reaches the compute engine
  (GR0 channel plus a compute-class object), but the first GR semaphore push
  faulted with an uncommitted context (`CTXNOTVALID`), and the follow-up
  `GPU_PROMOTE_CTX` attempt wedged GSP-RM (internal assert `0x52a8b04`, no
  reply). Teardown could not clear WPR2, so the fail-safe held the device
  (module unloaded, GPU unbound, protected region up, D0 locks held). Recovery
  is a supervised reboot under the reboot-approval policy; the promote
  bisection resumes after that.

## 9. Interaction sequence (end state)

```mermaid
sequenceDiagram
  autonumber
  participant GD as Guest driver
  participant EMU as GSP-RM emulation - control
  participant BE as Backend - real GSP client
  participant GSP as Real GSP firmware
  participant ENG as Engines

  GD->>EMU: objects, vaspaces, channel creation
  EMU->>BE: translate to real objects [planned]
  BE->>GSP: allocate real objects and channels
  GD->>EMU: work submissions
  EMU->>BE: decoded and translated push streams
  BE->>ENG: submit on real channels
  ENG-->>BE: completions and faults
  BE->>EMU: relay into guest memory
  EMU-->>GD: trackers and events advance
```

The full interaction and scheduling view (queues, interrupts, timing classes,
multi-tenant policy) is in
[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
system map is in [`09-full-system-picture.md`](09-full-system-picture.md).

## See also

- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
- [`../eng-ops.md`](../eng-ops.md) - engineering operations; Part II carries the physical-backend parameter, debugfs and runbook inventory
- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view