diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-12 00:32:00 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-12 00:32:00 -0400 |
| commit | 5df3147130c9254cd5bea34b330a2a42ee51d635 (patch) | |
| tree | 6a8fdf54bbb291eaccbeef8334e2cdd70af2caa8 /wiki/Physical-GPU-Backend.md | |
| parent | f1edf479a167898aaf70dc8f21e9cf18bccbe112 (diff) | |
| download | open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.tar.gz open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.zip | |
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link
mapping, local-marker scan, About-message insert), WikiBot publisher
with diff/check modes and publish manifest, live site auditor, image
uploader, site-only page sources (landing page, About, index, sidebar,
banner templates, About-message template).
- wiki: new public deep-dive sources (project status, install, FAQ,
system overview, GSP and GSP-RM, execution frontier, host module,
guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger,
hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace
status; corrected cross-document links; README rewritten for the new set.
- wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/Physical-GPU-Backend.md')
| -rw-r--r-- | wiki/Physical-GPU-Backend.md | 165 |
1 files changed, 165 insertions, 0 deletions
diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md new file mode 100644 index 0000000..1b83203 --- /dev/null +++ b/wiki/Physical-GPU-Backend.md @@ -0,0 +1,165 @@ +# Physical GPU Backend and Guest Wiring + +> **Scope.** The design that executes guest work on the real GPU: why the host +> driver cannot be the path and how the live GSP-RM session is wired to the +> guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) +> and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. + +Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +stock guest driver; `[gated]` implemented default-off; `[wip]` partially +built; `[wall]` current blocker; `[target]` planned end state. + +## 1. The problem and its constraints + +The guest's real work (copies, memory operations, grid launches) must execute +on the physical GPU. The constraints: one DKMS kernel module on the host, no +daemon and no new host userspace; host and guest drivers unmodified, with no +paravirtual enlightenment in the guest; all emulation and translation logic +inside the module; and real execution — compute is not emulated in software. + +## 2. Why the host driver is not the path + +| Interface | What it offers | Reachable from our module? | +|---|---|---| +| Host driver (distribution build) | resource management, scheduling, memory | No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface | +| Host driver peer-memory exports | pin/map memory for peer devices | Partially — memory only, no submission | +| Vendor vGPU host stack | mediated work submission and scheduling | Requires the licensed vGPU host branch and a vGPU-aware guest — excluded | +| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation | +| The GPU's own GSP firmware | the resource manager server itself | Yes — once the module owns the device; this is the path used | + +With an unmodified host driver there is no callable submission interface into +it, so the module must own the device and talk to the hardware itself. + +## 3. Mechanism options and the decision + +```text + (A) Proxy through the host driver: module -> host driver's RM -> real channel + -> GPU. Needs a modified host driver or a vendor interface — not implementable. + (B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page + tables -> GPU. Reproduces the scheduler, MMU ownership, context switching. + (C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path. + (D) Own the GPU; use its real GSP firmware as the server <- chosen + guest push -> module -> real GSP-RM session -> real channel -> GPU + The module becomes the host-side resource-manager client; scheduling, + page tables and context management stay the firmware's job. + (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope. +``` + +Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing. + +## 4. The chosen mechanism: bring-up ladder + +The module binds the physical GPU as its driver while the distribution driver +is unbound (dedicated-GPU deployment, §9). Each stage is reversible and +independently verified: + +1. **Ownership** — the board's power domain is held active across the window (the + platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. `[live]` +2. **Firmware staging** — the matching GSP image is fetched and validated (ELF64 + RISC-V; loadable section, version, hardware signature); no device state touched. `[live]` +3. **Boot** — protected region in video memory, secure-verification stage, booter, + RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. `[live]` +4. **Host RM client** — the module speaks the already-mapped RPC protocol to the + real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". `[live]` +5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource + objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]` +6. **Work path** — decoded guest pushes become real push streams; resolved host pages + map through module-built scratch page tables; the firmware schedules the work. `[gated]` +7. **Completion relay** — hardware-completed semaphores land in guest-polled memory; + events and faults relay through the existing MSI-X injection path. `[gated]` + +## 5. The two planes and the wiring + +The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software. + +```text +GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU +----------------------------------------------------------------------------- +INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live] + claim: power domain held -> distribution driver released -> module + binds the device -> liveness check [live] + boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live] + mdev + ladder: backend joins the live session; real objects, + channels, first real CE copy [live] + guest: nvidia.ko probes; emulation serves handshake, static config, + RM battery, UVM registration [live] +WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell + host pump decodes; eligible CE copies forwarded via module-built scratch + page tables; GR: launch reaches the front-end, descriptor not seen [wall] +DONE engine authors the release in guest memory; host relays via MSI-X [gated] +TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown + -> protected region down -> distribution driver rebound [live] +FAIL a failed mutating operation parks the device in a locked fail-safe; + never auto-retried; recovery is supervised [live] +``` + +## 6. Proven capabilities + +| Capability | Status | +|---|---| +| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done `[live]` | +| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — 64/64 payload round-trip `[live]` | +| Software bridge for decoded memory/state operations | in progress `[wip]` | +| Device ownership, firmware staging, board-liveness checks | verified live `[live]` | +| Power domain held active across the ownership window | verified across supervised cycles `[live]` | +| Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | +| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | +| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` | +| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` | + +## 7. The execution frontier + +The RM client ladder reaches the compute engine (a GR0 channel plus a +compute-class object). A forwarded guest CUDA launch now +reaches the front-end (ring entry fetched, methods executed) but the inline +launch-descriptor install has not been observed and no grid has dispatched; +context validity at dispatch is ruled out. The wall is the form of the +forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md). +A failed mutating operation on the backend parks the device in a locked +fail-safe that is never auto-retried; recovery is a supervised step. + +## 8. Safety and platform discipline + +* **Dedicated GPU deployment.** One GPU serves the vGPU service; the host uses + another GPU or integrated graphics. The distribution driver stays unmodified + and is unbound from that device only while the service is active. +* **The board's power domain belongs to the platform.** Releasing the last driver + can power-gate the board and the power-on sequence can fail; the ownership path + holds the whole board in a held-active state across the window, verifies + config-space liveness, and aborts on any power-loss signature without retry. +* **Supervised cycles.** The device is driven by the module under supervised + cycles with a written recovery plan. A failed mutating operation parks the + device in a locked fail-safe and is never auto-retried; any reboot requires + explicit human approval with a written reason and post-reboot plan first. +* **Fail closed.** No forwarding seam programs guest-derived page tables into + the real MMU until the population path is proven end to end; the current + forwarded path submits engine work only through module-built scratch page + tables mapping already-resolved host pages. + +## 9. Sharing policy and isolation + +* **Dedicated GPU (chosen):** one GPU serves the vGPU service; a multi-GPU host + gives the distribution driver its own GPU. +* **Shared single GPU with the host driver bound:** no conforming mechanism + exists on this generation without vendor interfaces; it would require + relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon). +* Isolation is one real GSP client per mdev with per-client page tables; host + access to guest memory stays behind the hypervisor's per-device DMA + interface, and tenant policy is a later layer on top + ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). + +## 10. Open questions + +* Firmware/host driver ABI match for the supported family on the development GPU. +* GPU unbind/reset lifecycle and recovery after guest faults. +* Which guest allocations need real VRAM and which can stay in system memory. +* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU. + +## 11. Related documentation + +* [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up. +* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design. +* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane. +* [`Host-Module.md`](Host-Module.md) — the module that owns both planes. +* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture. |
