diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-01 22:15:15 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-01 22:15:15 -0400 |
| commit | aa4cf303dd028b24ca29edbc0a5f3c237ab98e3f (patch) | |
| tree | 91412c3d372df8554205e10d5a2fc1e85d270c84 | |
| parent | 8c6bcb9a72c199368048d17e757814062b4765d1 (diff) | |
| download | open-iov-aa4cf303dd028b24ca29edbc0a5f3c237ab98e3f.tar.gz open-iov-aa4cf303dd028b24ca29edbc0a5f3c237ab98e3f.zip | |
docs: record host VF acceleration as a deferred future goal
| -rw-r--r-- | STATE-CURRENT.md | 9 | ||||
| -rw-r--r-- | docs/03-architecture/12-host-vf-acceleration-future-goal.md | 140 | ||||
| -rw-r--r-- | docs/README.md | 22 |
3 files changed, 163 insertions, 8 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 15c167d..261269e 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -534,6 +534,15 @@ section 4, a defensible claim needs **all** of: Until then the honest statement is: **initialization works; execution is in progress.** +## Deferred goals + +- Host NVIDIA driver (`nvidia.ko`) over a virtual function — host OS + acceleration from the open-iov device model — is a **deferred design goal**, + lower priority than guest CUDA execution, multi-VM, 3D and performance; the + verdict (not possible on this platform as-is) and a first design sketch are + in + [`docs/03-architecture/12-host-vf-acceleration-future-goal.md`](docs/03-architecture/12-host-vf-acceleration-future-goal.md). + ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. diff --git a/docs/03-architecture/12-host-vf-acceleration-future-goal.md b/docs/03-architecture/12-host-vf-acceleration-future-goal.md new file mode 100644 index 0000000..298b329 --- /dev/null +++ b/docs/03-architecture/12-host-vf-acceleration-future-goal.md @@ -0,0 +1,140 @@ +# Host VF Acceleration: A Deferred Future Goal + +> **Status: deferred design goal (2026-10-02). Research and design only — nothing +> from this document is planned for the current roadmap, and no implementation +> is in the tree.** It records the goal, the feasibility verdict on the current +> platform, the options evaluated, and the conditions under which the work would +> be worth starting. It ranks below the active critical path: guest CUDA +> execution +> ([`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)), +> multi-VM, 3D and performance. The verdict is grounded in a read-only study on +> the development host (TU104 mobile, stock distribution driver, QEMU/KVM). + +## 1. The goal + +Give a process **running on the host OS** GPU acceleration through the open-iov +device model: bind the host's stock NVIDIA kernel driver to a virtual function +or virtual device presented by `open-iov-vfio.ko`, so that host userspace +(`libcuda`, `nvidia-smi`, CUDA applications) consumes the mediated device the +way a guest does, with no VM in between. + +This is deliberately different from what the project does today: + +* **Current path (sound, active):** a local VM or microVM is attached to the + mdev; the guest runs the stock driver and stock CUDA, and guest submissions + reach real hardware once the physical backend is wired in + ([`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)). + The host orchestrates but does not own the CUDA context. +* **This goal:** the CUDA context lives in the host kernel, with no VM + boundary, no guest-memory pinning and no virtualized interrupt path. + +## 2. Feasibility verdict on the current platform: not possible as-is + +Four independent blockers were verified. Each alone is sufficient. + +1. **Driver-model bus mismatch — decisive and structural.** A mediated device + is a device on the `mdev` bus. The stock NVIDIA driver registers only a PCI + driver, and the driver core offers a device only to drivers on that device's + own bus. An mdev has no PCI identity; the mdev bus has no auto-match at all + and binds only the parent's declared mdev driver at device-creation time. + Emulating PCI config space and BARs inside the mdev does not change the bus + the device lives on, so "bind the host NVIDIA driver to an mdev" is a + category error in the Linux driver model, not a missing feature. +2. **No SR-IOV silicon on this part.** The development GPU exposes no SR-IOV + extended capability: no PF SR-IOV sysfs attributes and no VF BARs. A module + cannot add a capability the silicon does not have, and the physical + function's config space belongs to the real hardware and to whichever driver + owns the PF. On parts that do expose VFs, the stock open driver still does + not run its resource manager on a VF: a normal build refuses the VF probe, + and a vGPU-aware build returns early after an IOMMU check and delegates to a + proprietary mediated-device driver. VFs are a VM-passthrough mechanism, not a + host-compute mechanism. +3. **A synthetic PCI shim is rejected.** Kernel PCI host-bridge APIs would in + principle let an out-of-tree module fabricate a root bus with custom config + space, but Linux has no generic virtual-PCI-function framework and no + supported way to trap the kernel's own MMIO accesses to a fabricated BAR. + The stock driver's probe also expects real BARs, DMA, interrupts and a full + GSP firmware boot. A shim would have to reimplement a hardware-equivalent GPU + in kernel context with no hypervisor trap layer and no isolation boundary: + any bug panics the host. It is strictly harder and less safe than the + existing VFIO/QEMU design, which already has both. Effort XL+, probability of + a maintainable, safe result very low. +4. **No host-side mdev support in the stock driver.** The stock driver + registers no mdev driver and no mdev parent operations; it has no code path + that runs RM against a mediated device. The vendor's own architecture uses + the physical function for host CUDA and the mediated device only between + hypervisor and guest. Verified rather than assumed. + +**Today's sound stock-CUDA route therefore remains a local VM/microVM on the +mdev.** Everything a host-resident CUDA process would need from the virtual +device — a bindable host driver instance, host-memory DMA reachability, host +MSI-X delivery — does not exist and cannot be created without one of the +blockers above being overturned. + +## 3. Options evaluated + +| Option | Stock CUDA on host? | Effort | Probability | Verdict | +|---|---|---|---|---| +| Local VM / microVM on the mdev | In guest (stock) | S–M | High — control plane already works | The sound path today; current critical path | +| Module-owned host API (own RM client exposed to host userspace) | No — Open-IOV API only | L–XL | Medium | Viable as an Open-IOV interface, not `libcuda`; requires owning the GPU, so host PF acceleration is lost on a single-GPU host | +| SR-IOV VF model | No | XL | ~0 here; hardware/vendor-gated elsewhere | Blocked by silicon and vendor semantics | +| Synthetic PCI shim + host NVIDIA driver | Hypothetically | XL+ | Very low | Rejected (structurally unsound) | +| Status quo: host NVIDIA driver on the PF | Yes | None | Works | Not a mediated-device path; the baseline this goal would replace | + +## 4. Why it is deferred + +* The current critical path already delivers the user-visible capability — CUDA + running on the physical GPU through open-iov — via the guest path; this goal + adds a second, much harder path to the same capability. +* No sound implementation exists. Every non-VM option is either blocked by + silicon/vendor semantics or requires months of low-probability work with a + host-panic failure mode. +* On a single-GPU host, the project's real-execution model already owns the GPU + (unbinding the distribution driver), so host PF acceleration is unavailable + for the duration anyway; meaningful host acceleration from a mediated device + needs a multi-GPU or host-proxy design that does not exist yet. +* Priority order remains: guest CUDA execution, multi-VM, 3D, performance, then + host-resident acceleration. + +## 5. Conditions that would make it worth starting + +* **SR-IOV-capable hardware** where the PF stack can create VFs and there is a + sanctioned way for the stock driver to bind a VF for host compute — the only + option that keeps the NVIDIA driver inside its supported driver model. +* **Driver-model work**: a kernel-supported virtual-device framework, or a + sanctioned host-side mediated-device consumer in the stock driver. Without one + of these, no safe implementation exists regardless of effort. +* **A hard deployment requirement** that a local VM/microVM cannot satisfy: a + host-resident workload that cannot tolerate VM memory or I/O overhead, or a + constraint that mandates `libcuda` on the host rather than an Open-IOV API. +* **A multi-GPU or host-proxy execution design** so host acceleration and guest + execution can coexist on one machine. + +## 6. First steps if it is picked up + +1. **Re-check preconditions** (hardware and software) and write them down: + SR-IOV capability, PF provider support for VFs, stock driver VF handling, + host kernel device-model support. A negative answer anywhere ends the + project before code is written. +2. **Choose the device model** — VF-backed (preferred where possible) or a + host-bindable virtual device — and write a driver-model conformance note + before writing any code. +3. **Prototype only the identity and bind path**: present or configure the + device and confirm the stock driver probes it; stop before GSP boot, DMA and + interrupts, and record exactly what the probe requires. +4. **Design the memory and interrupt model next**: how host-process memory + becomes DMA-reachable, how MSI-X is delivered, and how faults surface, with + an isolation story for a host that has no hypervisor trap layer. +5. **Wire the RM path last**: serve the GSP-RM protocol for a host client, + starting from the module's live host-side GSP-RM session + ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)), with + state independent of the guest-facing emulation. +6. **Gate and supervise**: an explicit module parameter, a supervised hardware + harness, and a deployment model that states the single-GPU trade-off (host + acceleration is lost while the module owns the card). + +## See also + +* [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) — the chosen execution backend (module-owned GPU + real GSP-RM session) +* [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) — the active critical path this goal ranks below +* [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) — forwarding options and interoperability constraints diff --git a/docs/README.md b/docs/README.md index 5057c9f..c74c59a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -164,14 +164,20 @@ Read in order if you are new. Jump directly if you know what you need. | 12 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min | | 13 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min | | 14 | [Physical backend guest wiring](03-architecture/11-physical-backend-guest-wiring.md) | developers, architects | 25 min | -| 15 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | -| 16 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | -| 17 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | -| 18 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | -| 19 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | -| 20 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | -| 21 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | -| 22 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | +| 15 | [Host VF acceleration — future goal (deferred)](03-architecture/12-host-vf-acceleration-future-goal.md) | architects, planners | 10 min | +| 16 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | +| 17 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | +| 18 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | +| 19 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | +| 20 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | +| 21 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | +| 22 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | +| 23 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | + +Row 15 is a **deferred future goal**: host NVIDIA-driver acceleration over a +virtual function. The feasibility verdict and a first design sketch are +recorded there for planning; it is not on the active critical path (guest CUDA +execution, multi-VM, 3D, performance come first). Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware decompilation record, address reference, historical protocol notes). Wiki |
