summaryrefslogtreecommitdiffstats
path: root/wiki/Physical-GPU-Backend.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
commit5df3147130c9254cd5bea34b330a2a42ee51d635 (patch)
tree6a8fdf54bbb291eaccbeef8334e2cdd70af2caa8 /wiki/Physical-GPU-Backend.md
parentf1edf479a167898aaf70dc8f21e9cf18bccbe112 (diff)
downloadopen-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.tar.gz
open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.zip
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link mapping, local-marker scan, About-message insert), WikiBot publisher with diff/check modes and publish manifest, live site auditor, image uploader, site-only page sources (landing page, About, index, sidebar, banner templates, About-message template). - wiki: new public deep-dive sources (project status, install, FAQ, system overview, GSP and GSP-RM, execution frontier, host module, guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger, hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace status; corrected cross-document links; README rewritten for the new set. - wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/Physical-GPU-Backend.md')
-rw-r--r--wiki/Physical-GPU-Backend.md165
1 files changed, 165 insertions, 0 deletions
diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md
new file mode 100644
index 0000000..1b83203
--- /dev/null
+++ b/wiki/Physical-GPU-Backend.md
@@ -0,0 +1,165 @@
+# Physical GPU Backend and Guest Wiring
+
+> **Scope.** The design that executes guest work on the real GPU: why the host
+> driver cannot be the path and how the live GSP-RM session is wired to the
+> guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md)
+> and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement.
+
+Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
+stock guest driver; `[gated]` implemented default-off; `[wip]` partially
+built; `[wall]` current blocker; `[target]` planned end state.
+
+## 1. The problem and its constraints
+
+The guest's real work (copies, memory operations, grid launches) must execute
+on the physical GPU. The constraints: one DKMS kernel module on the host, no
+daemon and no new host userspace; host and guest drivers unmodified, with no
+paravirtual enlightenment in the guest; all emulation and translation logic
+inside the module; and real execution — compute is not emulated in software.
+
+## 2. Why the host driver is not the path
+
+| Interface | What it offers | Reachable from our module? |
+|---|---|---|
+| Host driver (distribution build) | resource management, scheduling, memory | No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface |
+| Host driver peer-memory exports | pin/map memory for peer devices | Partially — memory only, no submission |
+| Vendor vGPU host stack | mediated work submission and scheduling | Requires the licensed vGPU host branch and a vGPU-aware guest — excluded |
+| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation |
+| The GPU's own GSP firmware | the resource manager server itself | Yes — once the module owns the device; this is the path used |
+
+With an unmodified host driver there is no callable submission interface into
+it, so the module must own the device and talk to the hardware itself.
+
+## 3. Mechanism options and the decision
+
+```text
+ (A) Proxy through the host driver: module -> host driver's RM -> real channel
+ -> GPU. Needs a modified host driver or a vendor interface — not implementable.
+ (B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
+ tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
+ (C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
+ (D) Own the GPU; use its real GSP firmware as the server <- chosen
+ guest push -> module -> real GSP-RM session -> real channel -> GPU
+ The module becomes the host-side resource-manager client; scheduling,
+ page tables and context management stay the firmware's job.
+ (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.
+```
+
+Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.
+
+## 4. The chosen mechanism: bring-up ladder
+
+The module binds the physical GPU as its driver while the distribution driver
+is unbound (dedicated-GPU deployment, §9). Each stage is reversible and
+independently verified:
+
+1. **Ownership** — the board's power domain is held active across the window (the
+ platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. `[live]`
+2. **Firmware staging** — the matching GSP image is fetched and validated (ELF64
+ RISC-V; loadable section, version, hardware signature); no device state touched. `[live]`
+3. **Boot** — protected region in video memory, secure-verification stage, booter,
+ RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. `[live]`
+4. **Host RM client** — the module speaks the already-mapped RPC protocol to the
+ real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". `[live]`
+5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource
+ objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]`
+6. **Work path** — decoded guest pushes become real push streams; resolved host pages
+ map through module-built scratch page tables; the firmware schedules the work. `[gated]`
+7. **Completion relay** — hardware-completed semaphores land in guest-polled memory;
+ events and faults relay through the existing MSI-X injection path. `[gated]`
+
+## 5. The two planes and the wiring
+
+The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.
+
+```text
+GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
+-----------------------------------------------------------------------------
+INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
+ claim: power domain held -> distribution driver released -> module
+ binds the device -> liveness check [live]
+ boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live]
+ mdev + ladder: backend joins the live session; real objects,
+ channels, first real CE copy [live]
+ guest: nvidia.ko probes; emulation serves handshake, static config,
+ RM battery, UVM registration [live]
+WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
+ host pump decodes; eligible CE copies forwarded via module-built scratch
+ page tables; GR: launch reaches the front-end, descriptor not seen [wall]
+DONE engine authors the release in guest memory; host relays via MSI-X [gated]
+TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
+ -> protected region down -> distribution driver rebound [live]
+FAIL a failed mutating operation parks the device in a locked fail-safe;
+ never auto-retried; recovery is supervised [live]
+```
+
+## 6. Proven capabilities
+
+| Capability | Status |
+|---|---|
+| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done `[live]` |
+| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — 64/64 payload round-trip `[live]` |
+| Software bridge for decoded memory/state operations | in progress `[wip]` |
+| Device ownership, firmware staging, board-liveness checks | verified live `[live]` |
+| Power domain held active across the ownership window | verified across supervised cycles `[live]` |
+| Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` |
+| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` |
+| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` |
+| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` |
+
+## 7. The execution frontier
+
+The RM client ladder reaches the compute engine (a GR0 channel plus a
+compute-class object). A forwarded guest CUDA launch now
+reaches the front-end (ring entry fetched, methods executed) but the inline
+launch-descriptor install has not been observed and no grid has dispatched;
+context validity at dispatch is ruled out. The wall is the form of the
+forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md).
+A failed mutating operation on the backend parks the device in a locked
+fail-safe that is never auto-retried; recovery is a supervised step.
+
+## 8. Safety and platform discipline
+
+* **Dedicated GPU deployment.** One GPU serves the vGPU service; the host uses
+ another GPU or integrated graphics. The distribution driver stays unmodified
+ and is unbound from that device only while the service is active.
+* **The board's power domain belongs to the platform.** Releasing the last driver
+ can power-gate the board and the power-on sequence can fail; the ownership path
+ holds the whole board in a held-active state across the window, verifies
+ config-space liveness, and aborts on any power-loss signature without retry.
+* **Supervised cycles.** The device is driven by the module under supervised
+ cycles with a written recovery plan. A failed mutating operation parks the
+ device in a locked fail-safe and is never auto-retried; any reboot requires
+ explicit human approval with a written reason and post-reboot plan first.
+* **Fail closed.** No forwarding seam programs guest-derived page tables into
+ the real MMU until the population path is proven end to end; the current
+ forwarded path submits engine work only through module-built scratch page
+ tables mapping already-resolved host pages.
+
+## 9. Sharing policy and isolation
+
+* **Dedicated GPU (chosen):** one GPU serves the vGPU service; a multi-GPU host
+ gives the distribution driver its own GPU.
+* **Shared single GPU with the host driver bound:** no conforming mechanism
+ exists on this generation without vendor interfaces; it would require
+ relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
+* Isolation is one real GSP client per mdev with per-client page tables; host
+ access to guest memory stays behind the hypervisor's per-device DMA
+ interface, and tenant policy is a later layer on top
+ ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)).
+
+## 10. Open questions
+
+* Firmware/host driver ABI match for the supported family on the development GPU.
+* GPU unbind/reset lifecycle and recovery after guest faults.
+* Which guest allocations need real VRAM and which can stay in system memory.
+* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.
+
+## 11. Related documentation
+
+* [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up.
+* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design.
+* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane.
+* [`Host-Module.md`](Host-Module.md) — the module that owns both planes.
+* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings.
+* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture.