diff options
Diffstat (limited to 'docs/03-architecture/08-physical-gpu-integration.md')
| -rw-r--r-- | docs/03-architecture/08-physical-gpu-integration.md | 175 |
1 files changed, 175 insertions, 0 deletions
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md new file mode 100644 index 0000000..f156429 --- /dev/null +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -0,0 +1,175 @@ +# Physical GPU Integration: How Work Reaches Real Hardware + +> **Status: chosen mechanism, not yet implemented.** The control plane works +> today (the guest's driver initializes and CUDA user-space reaches context +> creation); the guest's *work* is currently executed in software as a +> development bridge. This document records the interfaces that actually +> exist, the mechanism options, the chosen path, and what has to be built. +> It fills the gap left open by +> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5 +> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3, +> which named "the host driver's RM API" as an option without a concrete call +> path from kernel context. + +## 1. The problem and its constraints + +The guest's real work (copies, memory operations, grid launches) must execute +on the physical GPU. The project constraints are: + +* one DKMS kernel module on the host; no daemon, coprocess or script is + required for the guest to run; +* host and guest drivers unmodified; no paravirtual enlightenment in the + guest — it believes it owns a physical GPU; +* all emulation and translation logic inside the module; +* real execution: compute must not be emulated in software. + +Any mechanism must therefore be reachable from kernel context, with no new +host userspace, and without editing the host driver. + +## 2. What interfaces actually exist + +| Interface | What it offers | Reachable from our module? | +|---|---|---| +| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). | +| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. | +| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. | +| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". | +| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. | +| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. | + +The conclusion: with an unmodified host driver there is **no callable +submission interface into it**. Real execution therefore requires the module +to own the device and talk to the hardware itself. + +## 3. Mechanism options + +``` + (A) Proxy through the host driver + guest push -> our module -> host driver's RM -> real channel -> GPU + Not implementable without a modified host driver or a vendor-managed + interface (see section 2). + + (B) Own the GPU; own scheduling (no GSP) + guest push -> our module -> our own channels/runlists/page tables -> GPU + Possible in principle, but the module would reproduce the scheduler, + MMU ownership and context switching; large and capability-limiting. + + (C) Hybrid + Only meaningful once (A) and (B) both exist; not on the critical path. + + (D) Own the GPU; use its real GSP firmware as the server <- chosen + guest push -> our module -> real GSP-RM session -> real channel -> GPU + The module becomes the host-side resource-manager client: the same + protocol the guest driver speaks, used against the physical GPU's + firmware. Scheduling, page tables and context management stay the + firmware's job. + + (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) + Deployment-gated (licence, hardware generation); out of scope for + consumer GPUs. +``` + +## 4. The chosen mechanism in detail + +### 4.1 Ownership and lifecycle + +The module binds the physical GPU as its driver. The distribution driver stays +unmodified and is unbound from that device while the vGPU service is active; +host display continues on another GPU (integrated graphics on the development +system). This is the dedicated-GPU deployment model; see section 5. An earlier +host-side GSP boot path existed in early versions and was disabled precisely +because it conflicted with the distribution driver owning the card — under +exclusive ownership that conflict disappears. + +### 4.2 Device bring-up + +PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP), +WPR2 setup, message queues in host system memory, doorbell, version exchange — +the same observable stages the guest emulation already models, with the module +now on the client side. + +### 4.3 Host-side resource-manager client + +Using the wire protocol and RPC inventory already mapped for the supported +family, the module allocates real objects: device/subdevice, virtual address +spaces, channels, and memory mappings. This replaces "call the host driver's +RM" with "be the RM client". + +### 4.4 Guest mediation + +The guest-facing GSP-RM emulation stays as it is for everything the guest +observes. Objects that need real resources are translated into real GSP-RM +requests (handle-domain translation, allocation backing). The page-table +contract the guest already provides for server-owned ranges becomes +population of the real MMU instead of a shadow. + +### 4.5 Work path + +Decoded guest pushes are rebuilt into real push streams on real channels +(where shapes match, passed through); guest GPU virtual addresses are mapped +to the same physical pages in the real MMU; the real doorbell is rung and the +GSP firmware schedules the work. + +### 4.6 Completion relay + +Hardware-completed semaphores land in memory the guest already polls. Events +and faults that the guest expects as interrupts are relayed through the +existing MSI-X injection path. + +### 4.7 Isolation + +One real GSP client per mdev, per-client page tables; host access to guest +memory stays behind the hypervisor's DMA interface; the firmware provides +per-context scheduling. Tenant policy and fairness are a later layer on top. + +## 5. Sharing policy + +* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses + another GPU or integrated graphics. +* **Multi-GPU host:** one GPU for the distribution driver, one for the + service. +* **Shared single GPU with the host driver bound:** no conforming mechanism + exists on this generation without vendor interfaces; this would require + relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host + daemon). + +## 6. Status and milestones + +| Step | Status | +|---|---| +| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done | +| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | +| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) | +| Host-side resource-manager client for real objects/channels | not started | +| Work translation and real submission | not started | +| Completion relay through real hardware | not started | +| CUDA kernel execution (`vectorAdd`) | not started | + +## 7. Open questions + +* Firmware/host driver ABI match for the supported family on the development + GPU. +* GPU unbind/reset lifecycle and recovery after guest faults. +* Which guest allocations need real VRAM and which can stay in system memory. +* Fault semantics when work runs on the real firmware. +* Per-tenant scheduling policy once multiple mdevs share the service GPU. + +## 8. Decision record + +* **Date:** 2026-09-29. +* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP + session as the execution backend. (A) is recorded as not implementable under + the constraints; (B) remains a fallback only if (D) proves unworkable; + (E) is deployment-gated and out of scope. +* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified + drivers, real execution, and a path to multi-tenant sharing. +* **Consequences:** the module grows a host-side resource-manager client and a + device-ownership lifecycle; the emulation path remains the control plane and + is reused. + +## See also + +- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts +- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans +- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs +- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view |