diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/03-architecture/08-physical-gpu-integration.md | 142 |
1 files changed, 121 insertions, 21 deletions
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index b1ade18..5c9e385 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -1,11 +1,15 @@ # Physical GPU Integration: How Work Reaches Real Hardware -> **Status: chosen mechanism, not yet implemented.** The control plane works -> today (the guest's driver initializes and CUDA user-space reaches context -> creation); the guest's *work* is currently executed in software as a -> development bridge. This document records the interfaces that actually -> exist, the mechanism options, the chosen path, and what has to be built. -> It fills the gap left open by +> **Status: mechanism chosen; ownership + firmware staging implemented; +> power-domain lock implemented, pending supervised verification; real GSP +> boot not started.** The control plane works today: the guest's driver +> initializes and the CUDA driver-API baseline passes end to end (context +> creation, allocation, copies, 64/64 payload round-trip), and the guest's +> decoded work is executed in software as a development bridge. This document +> records the interfaces that actually exist, the mechanism options, the +> chosen path, the platform power-domain hazard the ownership path exposes on +> the development host (§4.2), and what has to be built. It fills the gap +> left open by > [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5 > and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3, > which named "the host driver's RM API" as an option without a concrete call @@ -47,7 +51,10 @@ to own the device and talk to the hardware itself. (A) Proxy through the host driver guest push -> our module -> host driver's RM -> real channel -> GPU Not implementable without a modified host driver or a vendor-managed - interface (see section 2). + interface (see section 2). Re-checked against the stock distribution + build on the development host: it exports no resource-manager or vGPU + symbols to other modules and installs no manager daemon, so there is no + exported submission interface to use. (B) Own the GPU; own scheduling (no GSP) guest push -> our module -> our own channels/runlists/page tables -> GPU @@ -81,7 +88,76 @@ host-side GSP boot path existed in early versions and was disabled precisely because it conflicted with the distribution driver owning the card — under exclusive ownership that conflict disappears. -### 4.2 Device bring-up +Releasing the distribution driver is not a neutral operation on every +platform, and this host proves it: the board's power domain belongs to the +platform, not to PCI (§4.2). The ownership sequence therefore holds an +explicit D0 lock over the whole board before the release and until after the +distribution driver has been rebound and probed. Without that lock, returning +to the normal driver is a dice roll: it has completed on clean cycles, but the +recovery is conditional and must never be assumed. + +### 4.2 Board power-domain hazard and D0 lock + +The GPU, its audio function, its USB host controller and its type-C connector +are one physical board behind a PCIe root port. That root port's firmware +declares the board's power resource, and the board rail itself is switched by +the platform's embedded controller. While the distribution driver is bound and +using the GPU, the board is held powered by the GPU's runtime activity. When +the last driver is released, the now-driverless GPU is allowed to suspend, the +root port autosuspends into D3, and the platform runs the power-resource *off* +sequence: the embedded controller is commanded to cut the board rail. The +companion USB controller witnesses this on its next resume by reporting that +its root hub lost power — that signature appears on every unbind, including +the clean cycles. + +Re-binding must run the matching *on* sequence, and that sequence +intermittently fails on the development host. In the two observed failures the +GPU stopped answering config-space reads after the re-power (the PCI core's +readiness poll ran `not ready ... after resume` ladders), and the host +hard-froze roughly a minute later: no oops, no panic, no crash record; the +journal simply stops. The distribution driver's remove path itself requests no +power state (it only disables the device); the power-down is done by the PCI +core plus the platform firmware layer, so changing a sysfs unbind into an +in-kernel release changes nothing. Both freezes followed an unbind/rebind (one +an explicit sysfs cycle, one the module's claim/release path); the module +itself never logged an error in either event, and in the second event it had +already been unloaded when the freeze happened inside the distribution +driver's re-bind power-up path. Hold-time correlation exists in the small +sample (short holds failed, longer holds survived) but is not a proven +threshold and is not relied on. + +**The lock.** Before releasing the distribution driver, the ownership path +must put the board's whole power domain into a held-D0 state: + +* forbid runtime power management on the GPU, on each of its board companion + functions, and on the root port — the platform must not be allowed to + autosuspend any of them; +* disallow D3cold on the same set of devices, so the PCI core cannot route the + bridge into the platform's D3 power-resource sequence even if runtime power + management were re-enabled; +* verify the GPU's runtime status is active and its config space readable + before the release proceeds. + +Both controls live in the device's core state and are not reset by driver +unbind, so the lock survives the whole window: release → claim → hold → +re-bind. While the module owns the device it verifies that the GPU still +answers config space before proceeding and aborts on any power-loss signature +(the companion controller reporting a lost root hub, or a failed config read). +The distribution driver is then rebound with the lock still held, its probe is +allowed to complete, and only then is runtime power management restored to +automatic. Any power-loss signature aborts the sequence and forbids a retry in +that boot: an out-of-band platform power cut bypasses the lock and would +produce the identical signature, and retrying would be a second dice roll on a +host that has already lost the board. + +The lock is implemented in the ownership path and awaits supervised +verification (§6, §8); a claim is refused unless both the claim parameter and +the hazard acknowledgement are set (§8). The exact resource names, controller +commands, captured timelines and the mitigation's code-level design live in +the adjacent local analysis notes (`kernel/LOCAL-GSP-BOOT.md`); the +adjacent document is part of the local analysis. + +### 4.3 Device bring-up PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP), WPR2 setup, message queues in host system memory, doorbell, version exchange — @@ -90,10 +166,14 @@ now on the client side. The bring-up is staged so each step is reversible and independently verified: -1. **Ownership** — the distribution driver is released from the device and the - module binds it; BAR0/1/3 are mapped and the falcon state is sampled. On - release the module unbinds and the distribution driver is rebound, so the - host can always return to its normal driver. +1. **Ownership** — the board power domain is locked in D0 and verified active + (§4.2); the distribution driver is then released and the module binds the + device, maps BAR0/1/3 and samples the falcon state, with the power-loss + checks staying armed while the module owns the device. On release the + module unbinds, the distribution driver is rebound with the lock still + held, and runtime power management is restored only after its probe + completes. Recovery to the normal driver has completed on every clean + cycle but is conditional on the lock, never assumed. 2. **Firmware staging** — the matching GSP image (the same file the distribution driver would load) is fetched and validated (ELF64 RISC-V; the loadable image section, firmware version and hardware signature blob @@ -105,20 +185,21 @@ The bring-up is staged so each step is reversible and independently verified: core is started; success is observed as an active RISC-V core plus the firmware's resource manager completing initialization. 4. **Client** — the module speaks the already-mapped RPC protocol to the real - firmware as the client (section 4.3). + firmware as the client (section 4.4). Stages 1 and 2 are implemented in the module and verified live (a claim/release -cycle leaves the host driver working; the staged image parses as expected); -stage 3 is the current bring-up target. +cycle leaves the host driver working on the clean cycles; the staged image +parses as expected). The power-domain lock of §4.2 guards the claim window; +stage 3 — the real boot — has not been started. -### 4.3 Host-side resource-manager client +### 4.4 Host-side resource-manager client Using the wire protocol and RPC inventory already mapped for the supported family, the module allocates real objects: device/subdevice, virtual address spaces, channels, and memory mappings. This replaces "call the host driver's RM" with "be the RM client". -### 4.4 Guest mediation +### 4.5 Guest mediation The guest-facing GSP-RM emulation stays as it is for everything the guest observes. Objects that need real resources are translated into real GSP-RM @@ -126,20 +207,20 @@ requests (handle-domain translation, allocation backing). The page-table contract the guest already provides for server-owned ranges becomes population of the real MMU instead of a shadow. -### 4.5 Work path +### 4.6 Work path Decoded guest pushes are rebuilt into real push streams on real channels (where shapes match, passed through); guest GPU virtual addresses are mapped to the same physical pages in the real MMU; the real doorbell is rung and the GSP firmware schedules the work. -### 4.6 Completion relay +### 4.7 Completion relay Hardware-completed semaphores land in memory the guest already polls. Events and faults that the guest expects as interrupts are relayed through the existing MSI-X injection path. -### 4.7 Isolation +### 4.8 Isolation One real GSP client per mdev, per-client page tables; host access to guest memory stays behind the hypervisor's DMA interface; the firmware provides @@ -161,8 +242,11 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Step | Status | |---|---| | Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done | +| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build | | Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | -| Device ownership + real GSP bring-up by the module | in progress — reversible claim/release and firmware staging verified; the real boot stage is next | +| Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) | +| Power-domain lock across the claim window | implemented — pending supervised verification (§4.2, §8) | +| Real GSP boot by the module | not started | | Host-side resource-manager client for real objects/channels | not started | | Work translation and real submission | not started | | Completion relay through real hardware | not started | @@ -176,6 +260,16 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. * Which guest allocations need real VRAM and which can stay in system memory. * Fault semantics when work runs on the real firmware. * Per-tenant scheduling policy once multiple mdevs share the service GPU. +* **Residual platform risk:** the D0 lock covers the PCI/runtime-PM route to + the board's power resource. An out-of-band power cut — a platform or + controller action that does not go through the root port's D3 path — would + produce the identical power-loss signature; the claim must abort on that + signature and must not be retried in the same boot. +* **Supervised verification of the lock:** the lock is implemented but not yet + empirically verified. The first claim/release with it active must be + supervised: explicit human go-ahead, other host work warned, a written + recovery plan in place, a held-open window long enough to prove the board + stayed powered, and pstore armed. ## 8. Decision record @@ -189,6 +283,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. * **Consequences:** the module grows a host-side resource-manager client and a device-ownership lifecycle; the emulation path remains the control plane and is reused. +* **Update (2026-10-01) — operating constraint:** never claim the physical GPU + unless `gsp_phys_gpu=1` **and** `gsp_phys_hazard_ack=1`, the power-domain + lock (§4.2) is active, and the test is supervised under a written recovery + plan. The claim path refuses without the acknowledgement. At the first + power-loss signature: abort and do not retry in that boot; a host reboot + requires explicit human approval and a written recovery plan updated first. ## 9. Interaction sequence (end state) |