1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
# Physical GPU Integration: How Work Reaches Real Hardware
> **Status: chosen mechanism, not yet implemented.** The control plane works
> today (the guest's driver initializes and CUDA user-space reaches context
> creation); the guest's *work* is currently executed in software as a
> development bridge. This document records the interfaces that actually
> exist, the mechanism options, the chosen path, and what has to be built.
> It fills the gap left open by
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
> which named "the host driver's RM API" as an option without a concrete call
> path from kernel context.
## 1. The problem and its constraints
The guest's real work (copies, memory operations, grid launches) must execute
on the physical GPU. The project constraints are:
* one DKMS kernel module on the host; no daemon, coprocess or script is
required for the guest to run;
* host and guest drivers unmodified; no paravirtual enlightenment in the
guest — it believes it owns a physical GPU;
* all emulation and translation logic inside the module;
* real execution: compute must not be emulated in software.
Any mechanism must therefore be reachable from kernel context, with no new
host userspace, and without editing the host driver.
## 2. What interfaces actually exist
| Interface | What it offers | Reachable from our module? |
|---|---|---|
| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |
The conclusion: with an unmodified host driver there is **no callable
submission interface into it**. Real execution therefore requires the module
to own the device and talk to the hardware itself.
## 3. Mechanism options
```
(A) Proxy through the host driver
guest push -> our module -> host driver's RM -> real channel -> GPU
Not implementable without a modified host driver or a vendor-managed
interface (see section 2).
(B) Own the GPU; own scheduling (no GSP)
guest push -> our module -> our own channels/runlists/page tables -> GPU
Possible in principle, but the module would reproduce the scheduler,
MMU ownership and context switching; large and capability-limiting.
(C) Hybrid
Only meaningful once (A) and (B) both exist; not on the critical path.
(D) Own the GPU; use its real GSP firmware as the server <- chosen
guest push -> our module -> real GSP-RM session -> real channel -> GPU
The module becomes the host-side resource-manager client: the same
protocol the guest driver speaks, used against the physical GPU's
firmware. Scheduling, page tables and context management stay the
firmware's job.
(E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
Deployment-gated (licence, hardware generation); out of scope for
consumer GPUs.
```
## 4. The chosen mechanism in detail
### 4.1 Ownership and lifecycle
The module binds the physical GPU as its driver. The distribution driver stays
unmodified and is unbound from that device while the vGPU service is active;
host display continues on another GPU (integrated graphics on the development
system). This is the dedicated-GPU deployment model; see section 5. An earlier
host-side GSP boot path existed in early versions and was disabled precisely
because it conflicted with the distribution driver owning the card — under
exclusive ownership that conflict disappears.
### 4.2 Device bring-up
PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
WPR2 setup, message queues in host system memory, doorbell, version exchange —
the same observable stages the guest emulation already models, with the module
now on the client side.
The bring-up is staged so each step is reversible and independently verified:
1. **Ownership** — the distribution driver is released from the device and the
module binds it; BAR0/1/3 are mapped and the falcon state is sampled. On
release the module unbinds and the distribution driver is rebound, so the
host can always return to its normal driver.
2. **Firmware staging** — the matching GSP image (the same file the
distribution driver would load) is fetched and validated (ELF64 RISC-V;
the loadable image section, firmware version and hardware signature blob
are identified and held for the boot stage). No device state is touched.
3. **Boot** — a protected region is laid out in video memory (firmware heap,
image, boot binary, reserved areas), the secure-verification stage runs on
the security falcon to authorize it, the booter is then run on the
security falcon with the address of the boot metadata, and the GSP RISC-V
core is started; success is observed as an active RISC-V core plus the
firmware's resource manager completing initialization.
4. **Client** — the module speaks the already-mapped RPC protocol to the real
firmware as the client (section 4.3).
Stages 1 and 2 are implemented in the module and verified live (a claim/release
cycle leaves the host driver working; the staged image parses as expected);
stage 3 is the current bring-up target.
### 4.3 Host-side resource-manager client
Using the wire protocol and RPC inventory already mapped for the supported
family, the module allocates real objects: device/subdevice, virtual address
spaces, channels, and memory mappings. This replaces "call the host driver's
RM" with "be the RM client".
### 4.4 Guest mediation
The guest-facing GSP-RM emulation stays as it is for everything the guest
observes. Objects that need real resources are translated into real GSP-RM
requests (handle-domain translation, allocation backing). The page-table
contract the guest already provides for server-owned ranges becomes
population of the real MMU instead of a shadow.
### 4.5 Work path
Decoded guest pushes are rebuilt into real push streams on real channels
(where shapes match, passed through); guest GPU virtual addresses are mapped
to the same physical pages in the real MMU; the real doorbell is rung and the
GSP firmware schedules the work.
### 4.6 Completion relay
Hardware-completed semaphores land in memory the guest already polls. Events
and faults that the guest expects as interrupts are relayed through the
existing MSI-X injection path.
### 4.7 Isolation
One real GSP client per mdev, per-client page tables; host access to guest
memory stays behind the hypervisor's DMA interface; the firmware provides
per-context scheduling. Tenant policy and fairness are a later layer on top.
## 5. Sharing policy
* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
another GPU or integrated graphics.
* **Multi-GPU host:** one GPU for the distribution driver, one for the
service.
* **Shared single GPU with the host driver bound:** no conforming mechanism
exists on this generation without vendor interfaces; this would require
relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
daemon).
## 6. Status and milestones
| Step | Status |
|---|---|
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
| Device ownership + real GSP bring-up by the module | in progress — reversible claim/release and firmware staging verified; the real boot stage is next |
| Host-side resource-manager client for real objects/channels | not started |
| Work translation and real submission | not started |
| Completion relay through real hardware | not started |
| CUDA kernel execution (`vectorAdd`) | not started |
## 7. Open questions
* Firmware/host driver ABI match for the supported family on the development
GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics when work runs on the real firmware.
* Per-tenant scheduling policy once multiple mdevs share the service GPU.
## 8. Decision record
* **Date:** 2026-09-29.
* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
session as the execution backend. (A) is recorded as not implementable under
the constraints; (B) remains a fallback only if (D) proves unworkable;
(E) is deployment-gated and out of scope.
* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
drivers, real execution, and a path to multi-tenant sharing.
* **Consequences:** the module grows a host-side resource-manager client and a
device-ownership lifecycle; the emulation path remains the control plane and
is reused.
## 9. Interaction sequence (end state)
```mermaid
sequenceDiagram
autonumber
participant GD as Guest driver
participant EMU as GSP-RM emulation - control
participant BE as Backend - real GSP client
participant GSP as Real GSP firmware
participant ENG as Engines
GD->>EMU: objects, vaspaces, channel creation
EMU->>BE: translate to real objects [planned]
BE->>GSP: allocate real objects and channels
GD->>EMU: work submissions
EMU->>BE: decoded and translated push streams
BE->>ENG: submit on real channels
ENG-->>BE: completions and faults
BE->>EMU: relay into guest memory
EMU-->>GD: trackers and events advance
```
The full interaction and scheduling view (queues, interrupts, timing classes,
multi-tenant policy) is in
[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
system map is in [`09-full-system-picture.md`](09-full-system-picture.md).
## See also
- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view
|
