1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
# Host VF Acceleration: A Deferred Future Goal
> **Status: deferred design goal (2026-10-02). Research and design only — nothing
> from this document is planned for the current roadmap, and no implementation
> is in the tree.** It records the goal, the feasibility verdict on the current
> platform, the options evaluated, and the conditions under which the work would
> be worth starting. It ranks below the active critical path: guest CUDA
> execution
> ([`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)),
> multi-VM, 3D and performance. The verdict is grounded in a read-only study on
> the development host (TU104 mobile, stock distribution driver, QEMU/KVM).
## 1. The goal
Give a process **running on the host OS** GPU acceleration through the open-iov
device model: bind the host's stock NVIDIA kernel driver to a virtual function
or virtual device presented by `vgsp.ko`, so that host userspace
(`libcuda`, `nvidia-smi`, CUDA applications) consumes the mediated device the
way a guest does, with no VM in between.
This is deliberately different from what the project does today:
* **Current path (sound, active):** a local VM or microVM is attached to the
mdev; the guest runs the stock driver and stock CUDA, and guest submissions
reach real hardware once the physical backend is wired in
([`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)).
The host orchestrates but does not own the CUDA context.
* **This goal:** the CUDA context lives in the host kernel, with no VM
boundary, no guest-memory pinning and no virtualized interrupt path.
## 2. Feasibility verdict on the current platform: not possible as-is
Four independent blockers were verified. Each alone is sufficient.
1. **Driver-model bus mismatch — decisive and structural.** A mediated device
is a device on the `mdev` bus. The stock NVIDIA driver registers only a PCI
driver, and the driver core offers a device only to drivers on that device's
own bus. An mdev has no PCI identity; the mdev bus has no auto-match at all
and binds only the parent's declared mdev driver at device-creation time.
Emulating PCI config space and BARs inside the mdev does not change the bus
the device lives on, so "bind the host NVIDIA driver to an mdev" is a
category error in the Linux driver model, not a missing feature.
2. **No SR-IOV silicon on this part.** The development GPU exposes no SR-IOV
extended capability: no PF SR-IOV sysfs attributes and no VF BARs. A module
cannot add a capability the silicon does not have, and the physical
function's config space belongs to the real hardware and to whichever driver
owns the PF. On parts that do expose VFs, the stock open driver still does
not run its resource manager on a VF: a normal build refuses the VF probe,
and a vGPU-aware build returns early after an IOMMU check and delegates to a
proprietary mediated-device driver. VFs are a VM-passthrough mechanism, not a
host-compute mechanism.
3. **A synthetic PCI shim is rejected.** Kernel PCI host-bridge APIs would in
principle let an out-of-tree module fabricate a root bus with custom config
space, but Linux has no generic virtual-PCI-function framework and no
supported way to trap the kernel's own MMIO accesses to a fabricated BAR.
The stock driver's probe also expects real BARs, DMA, interrupts and a full
GSP firmware boot. A shim would have to reimplement a hardware-equivalent GPU
in kernel context with no hypervisor trap layer and no isolation boundary:
any bug panics the host. It is strictly harder and less safe than the
existing VFIO/QEMU design, which already has both. Effort XL+, probability of
a maintainable, safe result very low.
4. **No host-side mdev support in the stock driver.** The stock driver
registers no mdev driver and no mdev parent operations; it has no code path
that runs RM against a mediated device. The vendor's own architecture uses
the physical function for host CUDA and the mediated device only between
hypervisor and guest. Verified rather than assumed.
**Today's sound stock-CUDA route therefore remains a local VM/microVM on the
mdev.** Everything a host-resident CUDA process would need from the virtual
device — a bindable host driver instance, host-memory DMA reachability, host
MSI-X delivery — does not exist and cannot be created without one of the
blockers above being overturned.
## 3. Options evaluated
| Option | Stock CUDA on host? | Effort | Probability | Verdict |
|---|---|---|---|---|
| Local VM / microVM on the mdev | In guest (stock) | S–M | High — control plane already works | The sound path today; current critical path |
| Module-owned host API (own RM client exposed to host userspace) | No — Open-IOV API only | L–XL | Medium | Viable as an Open-IOV interface, not `libcuda`; requires owning the GPU, so host PF acceleration is lost on a single-GPU host |
| SR-IOV VF model | No | XL | ~0 here; hardware/vendor-gated elsewhere | Blocked by silicon and vendor semantics |
| Synthetic PCI shim + host NVIDIA driver | Hypothetically | XL+ | Very low | Rejected (structurally unsound) |
| Status quo: host NVIDIA driver on the PF | Yes | None | Works | Not a mediated-device path; the baseline this goal would replace |
## 4. Why it is deferred
* The current critical path already delivers the user-visible capability — CUDA
running on the physical GPU through open-iov — via the guest path; this goal
adds a second, much harder path to the same capability.
* No sound implementation exists. Every non-VM option is either blocked by
silicon/vendor semantics or requires months of low-probability work with a
host-panic failure mode.
* On a single-GPU host, the project's real-execution model already owns the GPU
(unbinding the distribution driver), so host PF acceleration is unavailable
for the duration anyway; meaningful host acceleration from a mediated device
needs a multi-GPU or host-proxy design that does not exist yet.
* Priority order remains: guest CUDA execution, multi-VM, 3D, performance, then
host-resident acceleration.
## 5. Conditions that would make it worth starting
* **SR-IOV-capable hardware** where the PF stack can create VFs and there is a
sanctioned way for the stock driver to bind a VF for host compute — the only
option that keeps the NVIDIA driver inside its supported driver model.
* **Driver-model work**: a kernel-supported virtual-device framework, or a
sanctioned host-side mediated-device consumer in the stock driver. Without one
of these, no safe implementation exists regardless of effort.
* **A hard deployment requirement** that a local VM/microVM cannot satisfy: a
host-resident workload that cannot tolerate VM memory or I/O overhead, or a
constraint that mandates `libcuda` on the host rather than an Open-IOV API.
* **A multi-GPU or host-proxy execution design** so host acceleration and guest
execution can coexist on one machine.
## 6. First steps if it is picked up
1. **Re-check preconditions** (hardware and software) and write them down:
SR-IOV capability, PF provider support for VFs, stock driver VF handling,
host kernel device-model support. A negative answer anywhere ends the
project before code is written.
2. **Choose the device model** — VF-backed (preferred where possible) or a
host-bindable virtual device — and write a driver-model conformance note
before writing any code.
3. **Prototype only the identity and bind path**: present or configure the
device and confirm the stock driver probes it; stop before GSP boot, DMA and
interrupts, and record exactly what the probe requires.
4. **Design the memory and interrupt model next**: how host-process memory
becomes DMA-reachable, how MSI-X is delivered, and how faults surface, with
an isolation story for a host that has no hypervisor trap layer.
5. **Wire the RM path last**: serve the GSP-RM protocol for a host client,
starting from the module's live host-side GSP-RM session
([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)), with
state independent of the guest-facing emulation.
6. **Gate and supervise**: an explicit module parameter, a supervised hardware
harness, and a deployment model that states the single-GPU trade-off (host
acceleration is lost while the module owns the card).
## See also
* [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) — the chosen execution backend (module-owned GPU + real GSP-RM session)
* [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) — the active critical path this goal ranks below
* [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) — forwarding options and interoperability constraints
|
