1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
|
# Physical GPU Backend and Guest Wiring
> **Scope.** The design that executes guest work on the real GPU: why the host
> driver cannot be the path and how the live GSP-RM session is wired to the
> guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md)
> and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement.
Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented default-off; `[wip]` partially
built; `[wall]` current blocker; `[target]` planned end state.
## 1. The problem and its constraints
The guest's real work (copies, memory operations, grid launches) must execute
on the physical GPU. The constraints: one DKMS kernel module on the host, no
daemon and no new host userspace; host and guest drivers unmodified, with no
paravirtual enlightenment in the guest; all emulation and translation logic
inside the module; and real execution — compute is not emulated in software.
## 2. Why the host driver is not the path
| Interface | What it offers | Reachable from our module? |
|---|---|---|
| Host driver (distribution build) | resource management, scheduling, memory | No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface |
| Host driver peer-memory exports | pin/map memory for peer devices | Partially — memory only, no submission |
| Vendor vGPU host stack | mediated work submission and scheduling | Requires the licensed vGPU host branch and a vGPU-aware guest — excluded |
| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation |
| The GPU's own GSP firmware | the resource manager server itself | Yes — once the module owns the device; this is the path used |
With an unmodified host driver there is no callable submission interface into
it, so the module must own the device and talk to the hardware itself.
## 3. Mechanism options and the decision
```text
(A) Proxy through the host driver: module -> host driver's RM -> real channel
-> GPU. Needs a modified host driver or a vendor interface — not implementable.
(B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
(C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
(D) Own the GPU; use its real GSP firmware as the server <- chosen
guest push -> module -> real GSP-RM session -> real channel -> GPU
The module becomes the host-side resource-manager client; scheduling,
page tables and context management stay the firmware's job.
(E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.
```
Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.
## 4. The chosen mechanism: bring-up ladder
The module binds the physical GPU as its driver while the distribution driver
is unbound (dedicated-GPU deployment, §9). Each stage is reversible and
independently verified:
1. **Ownership** — the board's power domain is held active across the window (the
platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. `[live]`
2. **Firmware staging** — the matching GSP image is fetched and validated (ELF64
RISC-V; loadable section, version, hardware signature); no device state touched. `[live]`
3. **Boot** — protected region in video memory, secure-verification stage, booter,
RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. `[live]`
4. **Host RM client** — the module speaks the already-mapped RPC protocol to the
real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". `[live]`
5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource
objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]`
6. **Work path** — decoded guest pushes become real push streams; resolved host pages
map through module-built scratch page tables; the firmware schedules the work. `[gated]`
7. **Completion relay** — hardware-completed semaphores land in guest-polled memory;
events and faults relay through the existing MSI-X injection path. `[gated]`
## 5. The two planes and the wiring
The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.
```text
GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
-----------------------------------------------------------------------------
INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
claim: power domain held -> distribution driver released -> module
binds the device -> liveness check [live]
boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live]
mdev + ladder: backend joins the live session; real objects,
channels, first real CE copy [live]
guest: nvidia.ko probes; emulation serves handshake, static config,
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
page tables; GR: launch reaches the front-end, descriptor not seen [wall]
DONE engine authors the release in guest memory; host relays via MSI-X [gated]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
never auto-retried; recovery is supervised [live]
```
## 6. Proven capabilities
| Capability | Status |
|---|---|
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done `[live]` |
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — 64/64 payload round-trip `[live]` |
| Software bridge for decoded memory/state operations | in progress `[wip]` |
| Device ownership, firmware staging, board-liveness checks | verified live `[live]` |
| Power domain held active across the ownership window | verified across supervised cycles `[live]` |
| Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` |
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` |
| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` |
| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` |
## 7. The execution frontier
The RM client ladder reaches the compute engine (a GR0 channel plus a
compute-class object). A forwarded guest CUDA launch now
reaches the front-end (ring entry fetched, methods executed) but the inline
launch-descriptor install has not been observed and no grid has dispatched;
context validity at dispatch is ruled out. The wall is the form of the
forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md).
A failed mutating operation on the backend parks the device in a locked
fail-safe that is never auto-retried; recovery is a supervised step.
## 8. Safety and platform discipline
* **Dedicated GPU deployment.** One GPU serves the vGPU service; the host uses
another GPU or integrated graphics. The distribution driver stays unmodified
and is unbound from that device only while the service is active.
* **The board's power domain belongs to the platform.** Releasing the last driver
can power-gate the board and the power-on sequence can fail; the ownership path
holds the whole board in a held-active state across the window, verifies
config-space liveness, and aborts on any power-loss signature without retry.
* **Supervised cycles.** The device is driven by the module under supervised
cycles with a written recovery plan. A failed mutating operation parks the
device in a locked fail-safe and is never auto-retried; any reboot requires
explicit human approval with a written reason and post-reboot plan first.
* **Fail closed.** No forwarding seam programs guest-derived page tables into
the real MMU until the population path is proven end to end; the current
forwarded path submits engine work only through module-built scratch page
tables mapping already-resolved host pages.
## 9. Sharing policy and isolation
* **Dedicated GPU (chosen):** one GPU serves the vGPU service; a multi-GPU host
gives the distribution driver its own GPU.
* **Shared single GPU with the host driver bound:** no conforming mechanism
exists on this generation without vendor interfaces; it would require
relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
* Isolation is one real GSP client per mdev with per-client page tables; host
access to guest memory stays behind the hypervisor's per-device DMA
interface, and tenant policy is a later layer on top
([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)).
## 10. Open questions
* Firmware/host driver ABI match for the supported family on the development GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.
## 11. Related documentation
* [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up.
* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design.
* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane.
* [`Host-Module.md`](Host-Module.md) — the module that owns both planes.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings.
* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture.
|
