From 2ccd9f875e53f8c9a98bfea47dc2657f3d3a1d6a Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 00:30:21 -0400 Subject: docs: defer experimental host acceleration to post-1.0; record rationale The host-VF design is reverted. Independent adversarial, soundness and VM-support reviews falsified its load-bearing mechanisms: RAM-backed BARs cannot be ioremapped or claimed by the stock driver on x86_64 or aarch64; trap-free windows cannot observe edge-triggered writes or read side effects; multi-client GSP-RM is unproven; the MSI and bridge bring-up plans were incorrect; and the isolation and performance claims were overstated for the mechanism. Replace it with one self-contained post-1.0 document recording the findings, corrected candidate approaches and trade-offs, a minimal-risk future architecture, the compile-time + module-load experimental gating policy, a phased plan with abort gates, and the revisit criteria. No baseline design or code path is changed; all earlier doc edits are reverted. --- docs/10-post-1.0/experimental-host-acceleration.md | 389 +++++++++++++++++++++ 1 file changed, 389 insertions(+) create mode 100644 docs/10-post-1.0/experimental-host-acceleration.md diff --git a/docs/10-post-1.0/experimental-host-acceleration.md b/docs/10-post-1.0/experimental-host-acceleration.md new file mode 100644 index 0000000..c88bcaa --- /dev/null +++ b/docs/10-post-1.0/experimental-host-acceleration.md @@ -0,0 +1,389 @@ +# Experimental Host Acceleration and Host Display — Post-1.0 Goal + +> ## ⛔ DO NOT IMPLEMENT THIS DOCUMENT +> +> **This is not an implementation instruction.** No agent may act on anything in +> this document unless the user *explicitly* asks for this experimental feature +> to be implemented. It exists only so that the design is not lost, and so that +> the decision to revisit it is deliberate. It is deliberately **not** linked +> from the documentation front door and is **not** part of the active reading +> order until the revisit gate in §8 passes. +> +> **Status:** deferred / post-1.0 stretch goal. Reverted from the active tree on +> 2026-10-02 after independent adversarial, soundness, and VM-risk reviews +> falsified the first design. The review record is kept privately. +> +> **Experimental policy (binding when the time comes):** any host-acceleration +> work is experimental; it may only be built behind a compile-time flag and +> enabled by an explicit module-load flag; it must not alter, replace, or +> rearchitect any existing code path; where it needs different behaviour it +> must **replicate** the code it needs in its own files, gated and isolated. + +--- + +## 0. Why this document exists, and why the first design was reverted + +In October 2026 a full design (`13-host-vf-acceleration-design.md`) was written +for binding the stock host `nvidia.ko` to a synthetic PCI function whose BARs +were backed by RAM ("trap-free windows"), with GSP boot/RPC served by the +existing emulation and execution forwarded to the real backend. Three +independent reviews (adversarial design review, engineering-soundness/criteria +audit, and VM-support risk review) plus direct checks against the kernel tree +found load-bearing assumptions that do not hold. The design and the seven doc +edits that introduced it were reverted; this document replaces them with a +correct, narrower record and a future direction that does not compromise the +system we are building today. + +### 0.1 The findings that forced the revert + +| # | Finding | Evidence (verified) | Why it is fatal to the old design | +|---|---|---|---| +| 1 | `ioremap` refuses ordinary System RAM | `arch/x86/mm/ioremap.c:68-90`, `:214-218`; `arch/arm64/mm/ioremap.c:16` | A runtime RAM-backed BAR cannot be mapped by the stock driver; the "memremap fallback" fails because the driver itself calls `ioremap` | +| 2 | `request_mem_region` fails on busy System RAM | `kernel/resource.c` conflict path (`-EBUSY`); the driver requests BAR0 regions | The driver's probe fails before any emulation runs | +| 3 | Pure change detection cannot observe same-value writes or read side effects | by construction: no dirty bit on RAM, and reads are never seen | The "trap-free observable window" cannot be correct for edge-triggered or read-clear registers | +| 4 | Real GSP-RM multi-client support is unproven | the only second-client attempt failed `0x23`; the guest bridge still refuses a second mdev | Host + guest coexistence — the whole point — rests on an untested server capability | +| 5 | Wrong MSI mechanism | `platform_msi_write_msg` derefs `platform_data` the PCI path never allocates (`drivers/base/platform-msi.c:83-88`) | Interrupt delivery would NULL-deref; the correct API is `pci_msi_create_irq_domain` | +| 6 | Synthetic bridge bring-up is not routine | x86 derefs `bridge->bus->sysdata` when `parent == NULL` (`arch/x86/pci/acpi.c:591-604`); `pci_scan_root_bus_bridge` does not size/assign BARs or add devices (`drivers/pci/probe.c:3256-3294`) | Registration can oops, and BARs stay unset | +| 7 | The reused probe contract and device model were wrong | reused config model is VGA class `0x030000` (`kernel/main.c:1725-1726`), not 3D class; `request_mem_region`, RM identity checks, vgaarb/PM/DRM paths are all unaddressed | The stock probe would not complete | +| 8 | "Reuse the bridge unchanged" was false | the sema service and RM emulation are coupled to `vfio_dma_rw`/guest GPAs; the execution bridge is not in the tree | The host path needs a cross-client transport redesign, not a small additive seam | +| 9 | Ownership/lifecycle and isolation claims were overstated | release keyed to "last mdev"; shared DMA domain; software-only isolation | A live host client could be torn down with the backend; "full IOMMU protection" was not true as stated | +| 10 | Performance targets were unsupported | poll cadence, method rebuild, single serializing lock, and absent forwarding were not in the arithmetic | No honest claim of "within a small percentage of native" was possible | + +The deeper reason: the original constraints (zero Linux diff, no `nvidia.ko` +change) force a *trap-free observation* plane for a *prebuilt* driver, and the +kernel gives no supported way to make that plane correct. Any future design must +either (a) add a real trap/transport plane with an explicit, maintained policy +exception, or (b) not put the stock driver in the host kernel at all. + +### 0.2 What the revert preserves + +* The baseline system and its active critical path are untouched. +* The project's no-patch rule and component boundaries are intact. +* The review record is kept privately for the future revisit. +* The goal itself is not lost — it is captured here, with the corrected + understanding. + +--- + +## 1. Where this fits: baseline goals vs the stretch goal + +**Baseline (the only work that is scheduled; all are project goals):** guest +CUDA execution on the physical engines through the module-owned backend; the +mdev/VFIO guest path; multi-VM; 3D; Windows guests; isolation; performance; +hardware/software portability. Until these are complete, nothing in this +document may be started. + +**Stretch goal (this document):** on a single-GPU system, let the *host OS +itself* use the GPU at full capability — host desktop (X11/Wayland) on the +physical outputs (HDMI/DP), host CUDA/3D through the real stock driver, and +guests running concurrently in compartments, with scheduling and +compartmentalization enforced by `open-iov-vfio.ko`. + +This stretch goal is explicitly **one**: it is attempted only after every +baseline goal is met, and it must not jeopardize them. + +--- + +## 2. Requirements for the stretch goal (unchanged intent, corrected framing) + +| # | Requirement | +|---|---| +| S1 | The host's real, stock `nvidia.ko` provides host CUDA/3D and the host desktop | +| S2 | Full display output from the GPU's HDMI/DP, driving the host desktop | +| S3 | Guest frame buffers are shown on the host (Looking Glass / KVMFR / IVSHMEM), both sides accelerated | +| S4 | Host and guest work coexist on one GPU under `open-iov-vfio.ko` scheduling and compartmentalization | +| S5 | High performance: work-dominated workloads within a few percent of native; launch-bound latency bounded and published | +| S6 | No unnecessary memory copies; zero-copy workload path | +| S7 | Isolation: a guest can never reach host memory or host interrupts; fail closed | +| S8 | Portability across kernel, driver, GSP and VFIO/mdev revisions | +| S9 | The baseline system is never compromised: no rearchitecture, no altered core paths, experimental-only, dual-gated | +| S10 | The diff is as small and reviewable as possible; upstreamable mechanisms are preferred over private ones | + +**Non-goals:** Windows host; replacing the guest path; patching proprietary +components in place; shipping experimental code in default builds. + +--- + +## 3. Why the old approach is retired (with the corrected direction) + +| Retired detail | Corrected direction | +|---|---| +| RAM-backed BARs without a boot carve-out | If a synthetic function is ever used, its BARs must live in firmware/boot-reserved memory (e.g. `memmap=`) — a documented deployment requirement, proven on x86_64 **and** aarch64 — or the plane is not used at all | +| Pure change detection for triggers and reads | Edge-triggered and read-side-effect registers require a real trap plane: a small, generic kernel facility (accessor dispatch) or a paravirtual transport. There is no trap-free substitute | +| `platform_msi_*` for a PCI device | Use the PCI MSI domain (`pci_msi_create_irq_domain` + a custom chip); MSI-X table semantics verified first | +| Bridge registration without parent/sysdata | Follow the in-tree software-bridge pattern (valid sysdata/parent, explicit domain, bus-number allocation, resource sizing/assignment, device add) | +| "Reuse the bridge unchanged" | Provide a genuine cross-client transport abstraction; treat the execution bridge and multi-client support as prerequisites, not reused assets | +| "Last mdev removes the backend" | Ownership is per-client; release only when the last *client* (host or guest) is gone; D0/resume coordinated with persistenced/udev | +| "IOMMU protection" as a software statement | Isolation is enforced by revocable, domain-backed mappings plus per-client GPU page tables; identity/passthrough is never the default | +| Unmeasured performance claims | Measurement-first: no target is published until an A/B distribution exists | + +--- + +## 4. Candidate approaches and trade-offs + +| Approach | Exact? | Fast? | Kernel diff | Driver change | Host `nvidia.ko` | Robustness/portability | Standing | +|---|---|---|---|---|---|---|---| +| A. Paravirtual "virtual accelerator bus" in the open host modules | Yes | Yes | No (bus/driver code in our + open driver transport) | Yes (source, not artifact) | Yes | High; upstream-plausible (virtio/vDPA-like) | **Primary candidate** | +| B. Accessor-dispatch kernel patch + rebuilt open driver | Yes | Yes | Yes (narrow files, wide blast radius) | Rebuild only (artifact) | Yes | Medium; out-of-tree patch maintenance burden; upstream unlikely | **Alternative, only with a maintenance commitment** | +| C. GPU-as-a-compartment (mdev/VFIO micro-VM + Looking Glass) | Yes | Yes | No | No | Driver runs in the desktop compartment, not dom0 | High; uses existing project design and conventions | **Baseline-safe fallback; recommended ceiling until A/B are accepted** | +| D. Boot-reserved carve-out + synthetic PCI + a trap plane (T1/T2) | With trap | With trap | Only if T2 | Only if T2 | Yes | Low–medium; boot carve-out + wide patch surface | Retired as a standalone; only as a means inside B | +| E. Trap-free RAM-backed synthetic PCI (the reverted design) | No | No | No | No | Yes | Falsified | **Retired — do not revive** | +| F. kmmio debug trap | Approximate | No (µs/access, single-step) | No | No | Yes | Prototype-only | Test/validation tool only | +| G. Time-sliced physical-function ownership | Yes | Yes | No | No | PF-bound, not virtual; not concurrent | High | Baseline fallback for "host acceleration without guests" | +| H. mdev bound directly by `nvidia.ko` | — | — | — | — | No | Impossible (bus mismatch) | Rejected | +| I. SR-IOV VF / vendor vGPU | Yes | Yes | No | Vendor | Not on this silicon/vendor semantics | Hardware-gated | Revisit only if silicon appears | + +Key trade-off: the original constraints forbade B and A, which are the only two +approaches that are simultaneously exact and fast. The user has since stated +that a justified, maintained kernel patch is acceptable; that reopens B, and A +remains the most upstreamable and maintainable. Either way, the trap/transport +plane is the *whole* design problem — everything else (device model, forwarding, +display) exists or is already designed. + +--- + +## 5. Recommended future architecture (when the revisit gate passes) + +### 5.1 Principles + +1. **Baseline isolation first.** No existing function is modified to make room + for the experiment. If the experiment needs a different behaviour, that + behaviour is implemented in experimental files as a duplicate path selected + at build time, not by changing the shared function. +2. **Dual gating.** A compile-time flag builds the experimental objects; a + module-load flag (plus the existing hazard acknowledgement) activates them. + Default builds and default loads never contain or run the code. +3. **Prefer upstreamable mechanisms.** A generic, small, well-justified kernel + facility or a paravirtual transport beats a private accessor patch. +4. **Measurement-first.** No performance claim without an A/B distribution on + the project's harness. +5. **Fail closed and reversible.** Every experimental path has a kill switch, + cannot own hardware the baseline needs, and unloads cleanly. + +### 5.2 Primary candidate — virtual accelerator transport (A) + +* Add a small virtual bus (auxiliary/vDPA-like) owned by an experimental module. +* The open host kernel modules gain a transport backend that binds the bus and + presents the GPU device model through it. The driver's *source* is extended in + a clearly separated transport file; the proprietary artifact path is not used + for this experiment. +* No synthetic PCI config space, no fabricated BARs, no accessor patching: the + transport carries register windows, interrupts and DMA semantics explicitly — + the same model a paravirtual device uses, which is why it is exact and fast. +* Execution: the host client is one more client of the real GSP-RM session, with + its own root/objects/channels/page tables, subject to the same scheduler and + the module's policy layer. + +### 5.3 Alternative candidate — kernel trap plane (B) + +* A minimal, generic "virtual MMIO window" facility: a static-key-gated + dispatcher reached from the accessor layer, covering `readX/writeX`, relaxed + and raw variants, `memcpy_fromio`/`toio`, `memset_io`. +* Requires rebuilt open modules (source untouched, artifact rebuilt). +* Conditions: attempted upstream first with a generic justification; if not + accepted, carried as a maintained patch with a version matrix and CI; never + silently enabled. + +### 5.4 Baseline-safe fallback — GPU-as-a-compartment (C) + +* The requestor's desktop stack (compositor + CUDA/3D) runs in a lightweight + GPU compartment bound to an mdev exactly like a guest; Looking Glass presents + its frames (and guests' frames) to the physical display; the backend executes + real work. +* Delivers S1–S4 and S7 with zero kernel/driver changes and the conventions the + project already documents; the only difference is that the desktop's driver + lives in its compartment rather than in dom0. +* This is the recommended ceiling until A or B is accepted. + +--- + +## 6. Full scope of the experimental feature (design detail for the future) + +### 6.1 Device and transport plane + +* New experimental files only: bus/transport, device model, interrupt domain, + DMA/ownership, display forwarding. +* No reuse-by-rewrite of baseline internals: where baseline behaviour is needed, + the experimental path calls stable exported entry points or duplicates the + minimal logic with a documented contract. +* The device model (register windows, interrupts, DMA) is explicit in the + transport; no configuration-space emulation is needed for the stock driver if + the transport is paravirtual. + +### 6.2 Execution plane + +* Host client as a first-class GSP-RM client: distinct root, objects, channels, + page tables; per-client accounting. +* Fairness: host control bursts must not push guest RPC tail latency beyond the + guest's deadline budget; the shared serialization point must have an + acquisition bound or per-client queues. +* Completions: real semaphore observation, relay into the host client's + memory/interrupts; the existing guest path is untouched. + +### 6.3 Memory and isolation + +* Zero-copy payload: host pages mapped once into the host client's GPU page + tables; guest pages stay behind VFIO. +* Isolation: revocable, domain-backed mappings; per-client ownership tables; + cross-client addresses refused and counted; no identity default; negative + tests before any shared run. + +### 6.4 Display plane (stretch within the stretch) + +* Host-dom0 path: modeset/scanout programming forwarded to the real display + engine; scanout reads real VRAM; HDMI/DP drives the host desktop. +* Compartment path: guest or desktop compartment frames via KVMFR/IVSHMEM to + the Looking Glass client on the physical display. +* Display is phased last and independently gated; failure falls back to the + compartment display without touching the baseline. + +### 6.5 Lifecycle and ownership + +* Per-client refcounts; release only when the last client is gone; D0/resume + coordination with persistenced/udev; supervised claim discipline retained; + no second owner until the baseline lifecycle is demonstrably stable. + +### 6.6 Experimental isolation policy (binding) + +* **Compile-time flag** (e.g. `OPENIOV_EXPERIMENTAL_HOST_ACCEL`) builds the + experimental objects; default builds do not contain them. +* **Module-load flag** (e.g. `host_accel_experimental=1` plus hazard ack) + activates the built-in path; the code is inert without both gates. +* **No rearchitecture:** shared functions, structures, locking, and the guest + path are not edited to accommodate the experiment; needed behaviour is + duplicated behind the flag with an explicit contract. +* **A/B integrity:** with the flag off, binary and behaviour are identical to + baseline; with the flag on but the module flag off, nothing runs; both are + needed to compare and to isolate regressions. +* **Not shipped:** release/default artifacts never enable the feature; unload + and revert are one flag away. +* **Telemetry:** experimental counters are namespaced and read-only. + +--- + +## 7. Phased plan: prerequisites, testing, implementation + +Each phase has acceptance gates and abort criteria. No phase may modify a +baseline path. + +**P0 — Prerequisites (must be complete before any experimental code):** +multi-client GSP-RM proven (distinct client root + per-client page tables); +guest execution bridge (P3–P6 of the guest-wiring design) complete; baseline +lifecycle stable across repeated cycles; A/B harness working; profile plumbing +real (not aspirational). *Abort if multi-client cannot be shown.* + +**P1 — Bench prototype, no baseline code:** transport/trap prototype in a +scratch module and in a virtual machine; prove register semantics, interrupt +delivery, DMA/IOMMU reachability, and probe on the chosen candidate (A or B). +*Abort if the candidate cannot be exact on both x86_64 and aarch64.* + +**P2 — Experimental module skeleton behind the compile flag:** bus/transport, +device model, lifecycle hooks to the existing backend API (read-only use), +no execution. Gate: baseline rung tests green with the experiment built but +disabled; guest path byte-identical with the flag off. + +**P3 — Control plane:** host client boots and initializes against the emulated +control plane; `nvidia-smi` on the host client; no real work. Gate: guest +ladder green before/after; no backend claim yet. + +**P4 — Execution plane:** host client becomes a real GSP-RM client; first +correct host workload on hardware. Gate: guest CUDA unaffected; fairness and +isolation counters clean. + +**P5 — Display:** host desktop on the physical outputs, guest frames in a +window; fallback verified. Gate: no regression to guest display or baseline +headless operation. + +**P6 — Measurement and hardening:** A/B distributions for kernels, copies, +launch-bound, compositor; portability matrix; negative/fuzz tests for the +ownership and IRQ paths; unload/soak. + +**P7 — Decision:** keep experimental, upstream a mechanism, or retire. The +default is to retire unless the A/B and maintenance burden are acceptable. + +--- + +## 8. Revisit gate (when this may start) + +All of the following must hold, and the user must explicitly decide to start: + +1. Every baseline goal is complete and the rung ladder is green on the + reference hardware and guest matrix. +2. Multi-client GSP-RM is proven and documented. +3. The guest execution bridge is complete and stable; no open freeze classes. +4. The profile/configuration surface is real (code reads it), not aspirational. +5. An accepted candidate exists — A (paravirtual) or B (patch with a maintenance + commitment) — or the decision is to build C (compartment) instead. +6. The experimental isolation policy (§6.6) is written into the codebase and CI. + +Until then, this document is dormant. + +--- + +## 9. Known knowns + +* Trapping a prebuilt driver's inlined MMIO from a module is impossible; a + trap/transport plane is unavoidable for host-native stock-driver binding. +* RAM-backed BARs need boot-reserved memory to be ioremap-able/claimable, and + even then they are not observable. +* The stock driver's probe requirements and register/RPC inventory are + discoverable and testable (the project already owns tracing and analysis + tooling). +* The backend can execute real work; the missing piece is client multiplicity + and the bridge, not raw engine capability. +* Per-client GPU page tables are the isolation boundary that matters; platform + IOMMU alone cannot separate same-device clients. + +## 10. Known unknowns (each with its resolving experiment) + +* Whether GSP-RM supports a second concurrent client with independent roots + and page tables — experiment: allocate a distinct root/client on the live + session and schedule work from both. +* The exact stock-driver register/RPC inventory for the host version — derive + from traces and the open modules. +* Whether upstream would accept a generic virtual-MMIO facility, and in what + shape — write the RFC; measure reaction before committing to B. +* Display-engine ownership under a multi-client GSP-RM session — forward a + modeset on a test client and observe scanout. +* aarch64/UMA reachability semantics for host pages — prototype on GB10-class + hardware when available. + +## 11. Possible unknown unknowns (discovery methods) + +* Hidden driver behaviours outside the modeled register/RPC set — full-window + access discovery during bring-up, in a scratch VM, never on a production + system. +* Server-side limits (client counts, handle domains, scheduling granularity) + not visible in the trace — probe the server's own logs and error paths with a + supervised harness. +* Firmware/driver version interactions introduced by a new host driver family — + the version-keyed profile workflow (capture, diff, add) is the discovery + mechanism. +* Power/display interactions on laptops (MUX, panel ownership) — hardware + bring-up matrix with the D0 discipline. +* Undocumented vendor expectations encoded in the driver's identity checks — + A/B against the host oracle with the same device data. + +## 12. Limitations and the elegant workarounds + +| Limitation | Workaround that keeps the design simple | +|---|---| +| No upstream virtual-PCI framework | Use a paravirtual transport (A) or the compartment model (C); do not fabricate PCI | +| Boot carve-out is a deployment change | Make it a documented requirement only for the experimental path; default systems are unaffected | +| Only one physical GPU | One backend session, many logical clients; compartment model as the fallback | +| Display forwarding is complex | Phase it last; fall back to compartment presentation, which needs no display forwarding | +| Host driver version treadmill | Version-keyed profiles and discovery mode; the experiment is opt-in and can lag driver releases | +| Single serialization point | Bounded acquisition, per-client accounting, or per-client queues before enabling the experiment | + +## 13. Decision record + +* **2026-10-02 — first host-VF design reverted.** Load-bearing assumptions + falsified (RAM-backed BARs, trap-free observation, unproven multi-client + GSP-RM, wrong MSI mechanism, incomplete bridge reuse, overstated isolation and + performance). All tracked edits reverted; the active tree is unchanged. +* **2026-10-02 — goal retained here as a post-1.0 stretch goal.** Possible + approaches, the recommended future architecture, the experimental isolation + policy, and the revisit gate are recorded in this document. +* **Policy:** do not implement unless the user explicitly asks; when + implemented, it is experimental, compile-time gated, module-load gated, + duplicate-path, and reversible; the baseline system is never compromised. -- cgit v1.2.3