summaryrefslogtreecommitdiffstats
path: root/docs/03-architecture/08-physical-gpu-integration.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
commit1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d (patch)
treecd16ecce1bbbf8e71c17091df843441965f871c1 /docs/03-architecture/08-physical-gpu-integration.md
parentf4803e303e1e4bfde9f241ec3b6226399d01825d (diff)
downloadopen-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.tar.gz
open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.zip
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'docs/03-architecture/08-physical-gpu-integration.md')
-rw-r--r--docs/03-architecture/08-physical-gpu-integration.md175
1 files changed, 175 insertions, 0 deletions
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md
new file mode 100644
index 0000000..f156429
--- /dev/null
+++ b/docs/03-architecture/08-physical-gpu-integration.md
@@ -0,0 +1,175 @@
+# Physical GPU Integration: How Work Reaches Real Hardware
+
+> **Status: chosen mechanism, not yet implemented.** The control plane works
+> today (the guest's driver initializes and CUDA user-space reaches context
+> creation); the guest's *work* is currently executed in software as a
+> development bridge. This document records the interfaces that actually
+> exist, the mechanism options, the chosen path, and what has to be built.
+> It fills the gap left open by
+> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
+> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
+> which named "the host driver's RM API" as an option without a concrete call
+> path from kernel context.
+
+## 1. The problem and its constraints
+
+The guest's real work (copies, memory operations, grid launches) must execute
+on the physical GPU. The project constraints are:
+
+* one DKMS kernel module on the host; no daemon, coprocess or script is
+ required for the guest to run;
+* host and guest drivers unmodified; no paravirtual enlightenment in the
+ guest — it believes it owns a physical GPU;
+* all emulation and translation logic inside the module;
+* real execution: compute must not be emulated in software.
+
+Any mechanism must therefore be reachable from kernel context, with no new
+host userspace, and without editing the host driver.
+
+## 2. What interfaces actually exist
+
+| Interface | What it offers | Reachable from our module? |
+|---|---|---|
+| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
+| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
+| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
+| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
+| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
+| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |
+
+The conclusion: with an unmodified host driver there is **no callable
+submission interface into it**. Real execution therefore requires the module
+to own the device and talk to the hardware itself.
+
+## 3. Mechanism options
+
+```
+ (A) Proxy through the host driver
+ guest push -> our module -> host driver's RM -> real channel -> GPU
+ Not implementable without a modified host driver or a vendor-managed
+ interface (see section 2).
+
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push -> our module -> our own channels/runlists/page tables -> GPU
+ Possible in principle, but the module would reproduce the scheduler,
+ MMU ownership and context switching; large and capability-limiting.
+
+ (C) Hybrid
+ Only meaningful once (A) and (B) both exist; not on the critical path.
+
+ (D) Own the GPU; use its real GSP firmware as the server <- chosen
+ guest push -> our module -> real GSP-RM session -> real channel -> GPU
+ The module becomes the host-side resource-manager client: the same
+ protocol the guest driver speaks, used against the physical GPU's
+ firmware. Scheduling, page tables and context management stay the
+ firmware's job.
+
+ (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
+ Deployment-gated (licence, hardware generation); out of scope for
+ consumer GPUs.
+```
+
+## 4. The chosen mechanism in detail
+
+### 4.1 Ownership and lifecycle
+
+The module binds the physical GPU as its driver. The distribution driver stays
+unmodified and is unbound from that device while the vGPU service is active;
+host display continues on another GPU (integrated graphics on the development
+system). This is the dedicated-GPU deployment model; see section 5. An earlier
+host-side GSP boot path existed in early versions and was disabled precisely
+because it conflicted with the distribution driver owning the card — under
+exclusive ownership that conflict disappears.
+
+### 4.2 Device bring-up
+
+PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
+WPR2 setup, message queues in host system memory, doorbell, version exchange —
+the same observable stages the guest emulation already models, with the module
+now on the client side.
+
+### 4.3 Host-side resource-manager client
+
+Using the wire protocol and RPC inventory already mapped for the supported
+family, the module allocates real objects: device/subdevice, virtual address
+spaces, channels, and memory mappings. This replaces "call the host driver's
+RM" with "be the RM client".
+
+### 4.4 Guest mediation
+
+The guest-facing GSP-RM emulation stays as it is for everything the guest
+observes. Objects that need real resources are translated into real GSP-RM
+requests (handle-domain translation, allocation backing). The page-table
+contract the guest already provides for server-owned ranges becomes
+population of the real MMU instead of a shadow.
+
+### 4.5 Work path
+
+Decoded guest pushes are rebuilt into real push streams on real channels
+(where shapes match, passed through); guest GPU virtual addresses are mapped
+to the same physical pages in the real MMU; the real doorbell is rung and the
+GSP firmware schedules the work.
+
+### 4.6 Completion relay
+
+Hardware-completed semaphores land in memory the guest already polls. Events
+and faults that the guest expects as interrupts are relayed through the
+existing MSI-X injection path.
+
+### 4.7 Isolation
+
+One real GSP client per mdev, per-client page tables; host access to guest
+memory stays behind the hypervisor's DMA interface; the firmware provides
+per-context scheduling. Tenant policy and fairness are a later layer on top.
+
+## 5. Sharing policy
+
+* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
+ another GPU or integrated graphics.
+* **Multi-GPU host:** one GPU for the distribution driver, one for the
+ service.
+* **Shared single GPU with the host driver bound:** no conforming mechanism
+ exists on this generation without vendor interfaces; this would require
+ relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
+ daemon).
+
+## 6. Status and milestones
+
+| Step | Status |
+|---|---|
+| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
+| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
+| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) |
+| Host-side resource-manager client for real objects/channels | not started |
+| Work translation and real submission | not started |
+| Completion relay through real hardware | not started |
+| CUDA kernel execution (`vectorAdd`) | not started |
+
+## 7. Open questions
+
+* Firmware/host driver ABI match for the supported family on the development
+ GPU.
+* GPU unbind/reset lifecycle and recovery after guest faults.
+* Which guest allocations need real VRAM and which can stay in system memory.
+* Fault semantics when work runs on the real firmware.
+* Per-tenant scheduling policy once multiple mdevs share the service GPU.
+
+## 8. Decision record
+
+* **Date:** 2026-09-29.
+* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
+ session as the execution backend. (A) is recorded as not implementable under
+ the constraints; (B) remains a fallback only if (D) proves unworkable;
+ (E) is deployment-gated and out of scope.
+* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
+ drivers, real execution, and a path to multi-tenant sharing.
+* **Consequences:** the module grows a host-side resource-manager client and a
+ device-ownership lifecycle; the emulation path remains the control plane and
+ is reused.
+
+## See also
+
+- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
+- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
+- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
+- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view