1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
|
# Physical GPU Backend and Guest Wiring
> **Scope.** The design that executes guest work on the real GPU: why the host
> driver cannot be the path and how the live GSP-RM session is wired to the
> guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md)
> and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement.
Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented default-off; `[wip]` partially
built; `[wall]` current blocker; `[target]` planned end state.
## 1. The problem and its constraints
The guest's real work (copies, memory operations, grid launches) must execute
on the physical GPU. The constraints: one DKMS kernel module on the host, no
daemon and no new host userspace; host and guest drivers unmodified, with no
paravirtual enlightenment in the guest; all emulation and translation logic
inside the module; and real execution — compute is not emulated in software.
## 2. Why the host driver is not the path
| Interface | What it offers | Reachable from our module? |
|---|---|---|
| Host driver (distribution build) | resource management, scheduling, memory | No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface |
| Host driver peer-memory exports | pin/map memory for peer devices | Partially — memory only, no submission |
| Vendor vGPU host stack | mediated work submission and scheduling | Requires the licensed vGPU host branch and a vGPU-aware guest — excluded |
| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation |
| The GPU's own GSP firmware | the resource manager server itself | Yes — once the module owns the device; this is the path used |
With an unmodified host driver there is no callable submission interface into
it, so the module must own the device and talk to the hardware itself.
## 3. Mechanism options and the decision
```text
(A) Proxy through the host driver: module -> host driver's RM -> real channel
-> GPU. Needs a modified host driver or a vendor interface — not implementable.
(B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
(C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
(D) Own the GPU; use its real GSP firmware as the server <- chosen
guest push -> module -> real GSP-RM session -> real channel -> GPU
The module becomes the host-side resource-manager client; scheduling,
page tables and context management stay the firmware's job.
(E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.
```
Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.
## 4. The chosen mechanism: bring-up ladder
The module binds the physical GPU as its driver while the distribution driver
is unbound (dedicated-GPU deployment, §9). Each stage is reversible and
independently verified:
1. **Ownership** — the board's power domain is held active across the window (the
platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. `[live]`
2. **Firmware staging** — the matching GSP image is fetched and validated (ELF64
RISC-V; loadable section, version, hardware signature); no device state touched. `[live]`
3. **Boot** — protected region in video memory, secure-verification stage, booter,
RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. `[live]`
4. **Host RM client** — the module speaks the already-mapped RPC protocol to the
real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". `[live]`
5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource
objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]`
6. **Work path** — decoded guest pushes become real push streams; resolved host pages
map through module-built scratch page tables; the firmware schedules the work.
`[live]` observed end to end for the verified guest kernel clients.
7. **Completion relay** — hardware-completed semaphores land in guest-polled memory;
events and faults relay through the existing MSI-X injection path.
`[live]` engine-authored completions relay to the guest and its waits resolve.
## 5. The two planes and the wiring
The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software.
```text
GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
-----------------------------------------------------------------------------
INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
claim: power domain held -> distribution driver released -> module
binds the device -> liveness check [live]
boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live]
mdev + ladder: backend joins the live session; real objects,
channels, first real CE copy [live]
guest: nvidia.ko probes; emulation serves handshake, static config,
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
page tables; GR: launches execute on the engine, completions relay
to the guest [live]
DONE engine authors the release in guest memory; host relays via MSI-X [live]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
never auto-retried; recovery is supervised [live]
```
## 6. Proven capabilities
| Capability | Status |
|---|---|
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done `[live]` |
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — 64/64 payload round-trip `[live]` |
| Software bridge for decoded memory/state operations | in progress `[wip]` |
| Device ownership, firmware staging, board-liveness checks | verified live `[live]` |
| Power domain held active across the ownership window | verified across supervised cycles `[live]` |
| Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` |
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` |
| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` |
| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest |
## 7. The execution frontier
The RM client ladder reaches the compute engine (a GR0 channel plus a
compute-class object). Forwarded guest CUDA launches now execute on the
engine: the inline launch descriptor installs and is read, the grid
dispatches, the engine authors the completion, the host relays it, and the
guest's synchronisation wait resolves — verified on fresh boots with the
vector-add, matrix-multiply and transformer clients. The leading edge is the
large-buffer staged-transfer class (under repair), reproducibility across
fresh boots, and per-guest execution isolation — see
[`Execution-Frontier.md`](Execution-Frontier.md).
A failed mutating operation on the backend parks the device in a locked
fail-safe that is never auto-retried; recovery is a supervised step.
## 8. Safety and platform discipline
* **Dedicated GPU deployment.** One GPU serves the vGPU service; the host uses
another GPU or integrated graphics. The distribution driver stays unmodified
and is unbound from that device only while the service is active.
* **The board's power domain belongs to the platform.** Releasing the last driver
can power-gate the board and the power-on sequence can fail; the ownership path
holds the whole board in a held-active state across the window, verifies
config-space liveness, and aborts on any power-loss signature without retry.
* **Supervised cycles.** The device is driven by the module under supervised
cycles with a written recovery plan. A failed mutating operation parks the
device in a locked fail-safe and is never auto-retried; any reboot requires
explicit human approval with a written reason and post-reboot plan first.
* **Fail closed.** No forwarding seam programs guest-derived page tables into
the real MMU until the population path is proven end to end; the current
forwarded path submits engine work only through module-built scratch page
tables mapping already-resolved host pages.
## 9. Sharing policy and isolation
* **Dedicated GPU (chosen):** one GPU serves the vGPU service; a multi-GPU host
gives the distribution driver its own GPU.
* **Shared single GPU with the host driver bound:** no conforming mechanism
exists on this generation without vendor interfaces; it would require
relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
* Isolation is one real GSP client per mdev with per-client page tables; host
access to guest memory stays behind the hypervisor's per-device DMA
interface, and tenant policy is a later layer on top
([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)).
* Multiple mediated devices with different emulated VRAM sizes and multiple
guests already coexist at the device level; per-guest execution isolation
for simultaneous CUDA guests is in progress.
## 10. Open questions
* Firmware/host driver ABI match for the supported family on the development GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.
## 11. Related documentation
* [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up.
* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design.
* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane.
* [`Host-Module.md`](Host-Module.md) — the module that owns both planes.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings.
* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture.
|
