# Experimental Host Acceleration and Host Display — Post-1.0 Goal > ## ⛔ DO NOT IMPLEMENT THIS DOCUMENT > > **This is not an implementation instruction.** No agent may act on anything in > this document unless the user *explicitly* asks for this experimental feature > to be implemented. It exists only so that the design is not lost, and so that > the decision to revisit it is deliberate. It is deliberately **not** linked > from the documentation front door and is **not** part of the active reading > order until the revisit gate in §8 passes. > > **Status:** deferred / post-1.0 stretch goal. Reverted from the active tree on > 2026-10-02 after independent adversarial, soundness, and VM-risk reviews > falsified the first design. The review record is kept privately. > > **How to treat this goal (from the owner, 2026-10-02).** We are *considering* > doing this; we may do this at some point. We will probably take a crack at it > because it is a great feature — but **do not block on this, do not pursue it, > and do not design around it.** It will be important to try at some point, but > primary objectives and the other stretch goals come first. Only when we are > certain there is no risk — which we are not certain of now — will we make a > **human decision** to proceed. This is **never pursued automatically**. > Consider it, until further notice, **deferred**. > > **Do-not-implement rule (binding).** Treat this document and the goal it > describes as **"Do not implement"** until after *all* design goals are > reached — and even then the goal may be scrapped on architectural grounds. > No aspect of the software may be designed toward a forward-looking goal that > would support this experimental feature: no new abstractions, hooks, seams, > parameters, or "just in case" code for it, anywhere in the baseline system. > > **Experimental policy (binding when the time comes):** any host-acceleration > work is experimental; it may only be built behind a compile-time flag and > enabled by an explicit module-load flag; it must not alter, replace, or > rearchitect any existing code path; where it needs different behaviour it > must **replicate** the code it needs in its own files, gated and isolated. --- ## 0. Why this document exists, and why the first design was reverted In October 2026 a full design (`13-host-vf-acceleration-design.md`) was written for binding the stock host `nvidia.ko` to a synthetic PCI function whose BARs were backed by RAM ("trap-free windows"), with GSP boot/RPC served by the existing emulation and execution forwarded to the real backend. Three independent reviews (adversarial design review, engineering-soundness/criteria audit, and VM-support risk review) plus direct checks against the kernel tree found load-bearing assumptions that do not hold. The design and the seven doc edits that introduced it were reverted; this document replaces them with a correct, narrower record and a future direction that does not compromise the system we are building today. ### 0.1 The findings that forced the revert | # | Finding | Evidence (verified) | Why it is fatal to the old design | |---|---|---|---| | 1 | `ioremap` refuses ordinary System RAM | `arch/x86/mm/ioremap.c:68-90`, `:214-218`; `arch/arm64/mm/ioremap.c:16` | A runtime RAM-backed BAR cannot be mapped by the stock driver; the "memremap fallback" fails because the driver itself calls `ioremap` | | 2 | `request_mem_region` fails on busy System RAM | `kernel/resource.c` conflict path (`-EBUSY`); the driver requests BAR0 regions | The driver's probe fails before any emulation runs | | 3 | Pure change detection cannot observe same-value writes or read side effects | by construction: no dirty bit on RAM, and reads are never seen | The "trap-free observable window" cannot be correct for edge-triggered or read-clear registers | | 4 | Real GSP-RM multi-client support is unproven | the only second-client attempt failed `0x23`; the guest bridge still refuses a second mdev | Host + guest coexistence — the whole point — rests on an untested server capability | | 5 | Wrong MSI mechanism | `platform_msi_write_msg` derefs `platform_data` the PCI path never allocates (`drivers/base/platform-msi.c:83-88`) | Interrupt delivery would NULL-deref; the correct API is `pci_msi_create_irq_domain` | | 6 | Synthetic bridge bring-up is not routine | x86 derefs `bridge->bus->sysdata` when `parent == NULL` (`arch/x86/pci/acpi.c:591-604`); `pci_scan_root_bus_bridge` does not size/assign BARs or add devices (`drivers/pci/probe.c:3256-3294`) | Registration can oops, and BARs stay unset | | 7 | The reused probe contract and device model were wrong | reused config model is VGA class `0x030000` (`kernel/main.c:1725-1726`), not 3D class; `request_mem_region`, RM identity checks, vgaarb/PM/DRM paths are all unaddressed | The stock probe would not complete | | 8 | "Reuse the bridge unchanged" was false | the sema service and RM emulation are coupled to `vfio_dma_rw`/guest GPAs; the execution bridge is not in the tree | The host path needs a cross-client transport redesign, not a small additive seam | | 9 | Ownership/lifecycle and isolation claims were overstated | release keyed to "last mdev"; shared DMA domain; software-only isolation | A live host client could be torn down with the backend; "full IOMMU protection" was not true as stated | | 10 | Performance targets were unsupported | poll cadence, method rebuild, single serializing lock, and absent forwarding were not in the arithmetic | No honest claim of "within a small percentage of native" was possible | The deeper reason: the original constraints (zero Linux diff, no `nvidia.ko` change) force a *trap-free observation* plane for a *prebuilt* driver, and the kernel gives no supported way to make that plane correct. Any future design must either (a) add a real trap/transport plane with an explicit, maintained policy exception, or (b) not put the stock driver in the host kernel at all. ### 0.2 What the revert preserves * The baseline system and its active critical path are untouched. * The project's no-patch rule and component boundaries are intact. * The review record is kept privately for the future revisit. * The goal itself is not lost — it is captured here, with the corrected understanding. --- ## 1. Where this fits: baseline goals vs the stretch goal **Baseline (the only work that is scheduled; all are project goals):** guest CUDA execution on the physical engines through the module-owned backend; the mdev/VFIO guest path; multi-VM; 3D; Windows guests; isolation; performance; hardware/software portability. Until these are complete, nothing in this document may be started. **Stretch goal (this document):** on a single-GPU system, let the *host OS itself* use the GPU at full capability — host desktop (X11/Wayland) on the physical outputs (HDMI/DP), host CUDA/3D through the real stock driver, and guests running concurrently in compartments, with scheduling and compartmentalization enforced by `open-iov-vfio.ko`. This stretch goal is explicitly **one**: it is attempted only after every baseline goal is met, and it must not jeopardize them. --- ## 2. Requirements for the stretch goal (unchanged intent, corrected framing) | # | Requirement | |---|---| | S1 | The host's real, stock `nvidia.ko` provides host CUDA/3D and the host desktop | | S2 | Full display output from the GPU's HDMI/DP, driving the host desktop | | S3 | Guest frame buffers are shown on the host (Looking Glass / KVMFR / IVSHMEM), both sides accelerated | | S4 | Host and guest work coexist on one GPU under `open-iov-vfio.ko` scheduling and compartmentalization | | S5 | High performance: work-dominated workloads within a few percent of native; launch-bound latency bounded and published | | S6 | No unnecessary memory copies; zero-copy workload path | | S7 | Isolation: a guest can never reach host memory or host interrupts; fail closed | | S8 | Portability across kernel, driver, GSP and VFIO/mdev revisions | | S9 | The baseline system is never compromised: no rearchitecture, no altered core paths, experimental-only, dual-gated | | S10 | The diff is as small and reviewable as possible; upstreamable mechanisms are preferred over private ones | **Non-goals:** Windows host; replacing the guest path; patching proprietary components in place; shipping experimental code in default builds. --- ## 3. Why the old approach is retired (with the corrected direction) | Retired detail | Corrected direction | |---|---| | RAM-backed BARs without a boot carve-out | If a synthetic function is ever used, its BARs must live in firmware/boot-reserved memory (e.g. `memmap=`) — a documented deployment requirement, proven on x86_64 **and** aarch64 — or the plane is not used at all | | Pure change detection for triggers and reads | Edge-triggered and read-side-effect registers require a real trap plane: a small, generic kernel facility (accessor dispatch) or a paravirtual transport. There is no trap-free substitute | | `platform_msi_*` for a PCI device | Use the PCI MSI domain (`pci_msi_create_irq_domain` + a custom chip); MSI-X table semantics verified first | | Bridge registration without parent/sysdata | Follow the in-tree software-bridge pattern (valid sysdata/parent, explicit domain, bus-number allocation, resource sizing/assignment, device add) | | "Reuse the bridge unchanged" | Provide a genuine cross-client transport abstraction; treat the execution bridge and multi-client support as prerequisites, not reused assets | | "Last mdev removes the backend" | Ownership is per-client; release only when the last *client* (host or guest) is gone; D0/resume coordinated with persistenced/udev | | "IOMMU protection" as a software statement | Isolation is enforced by revocable, domain-backed mappings plus per-client GPU page tables; identity/passthrough is never the default | | Unmeasured performance claims | Measurement-first: no target is published until an A/B distribution exists | --- ## 4. Candidate approaches and trade-offs | Approach | Exact? | Fast? | Kernel diff | Driver change | Host `nvidia.ko` | Robustness/portability | Standing | |---|---|---|---|---|---|---|---| | A. Paravirtual "virtual accelerator bus" in the open host modules | Yes | Yes | No (bus/driver code in our + open driver transport) | Yes (source, not artifact) | Yes | High; upstream-plausible (virtio/vDPA-like) | **Primary candidate** | | B. Accessor-dispatch kernel patch + rebuilt open driver | Yes | Yes | Yes (narrow files, wide blast radius) | Rebuild only (artifact) | Yes | Medium; out-of-tree patch maintenance burden; upstream unlikely | **Alternative, only with a maintenance commitment** | | C. GPU-as-a-compartment (mdev/VFIO micro-VM + Looking Glass) | Yes | Yes | No | No | Driver runs in the desktop compartment, not dom0 | High; uses existing project design and conventions | **Baseline-safe fallback; recommended ceiling until A/B are accepted** | | D. Boot-reserved carve-out + synthetic PCI + a trap plane (T1/T2) | With trap | With trap | Only if T2 | Only if T2 | Yes | Low–medium; boot carve-out + wide patch surface | Retired as a standalone; only as a means inside B | | E. Trap-free RAM-backed synthetic PCI (the reverted design) | No | No | No | No | Yes | Falsified | **Retired — do not revive** | | F. kmmio debug trap | Approximate | No (µs/access, single-step) | No | No | Yes | Prototype-only | Test/validation tool only | | G. Time-sliced physical-function ownership | Yes | Yes | No | No | PF-bound, not virtual; not concurrent | High | Baseline fallback for "host acceleration without guests" | | H. mdev bound directly by `nvidia.ko` | — | — | — | — | No | Impossible (bus mismatch) | Rejected | | I. SR-IOV VF / vendor vGPU | Yes | Yes | No | Vendor | Not on this silicon/vendor semantics | Hardware-gated | Revisit only if silicon appears | Key trade-off: the original constraints forbade B and A, which are the only two approaches that are simultaneously exact and fast. The user has since stated that a justified, maintained kernel patch is acceptable; that reopens B, and A remains the most upstreamable and maintainable. Either way, the trap/transport plane is the *whole* design problem — everything else (device model, forwarding, display) exists or is already designed. --- ## 5. Recommended future architecture (when the revisit gate passes) ### 5.1 Principles 1. **Baseline isolation first.** No existing function is modified to make room for the experiment. If the experiment needs a different behaviour, that behaviour is implemented in experimental files as a duplicate path selected at build time, not by changing the shared function. 2. **Dual gating.** A compile-time flag builds the experimental objects; a module-load flag (plus the existing hazard acknowledgement) activates them. Default builds and default loads never contain or run the code. 3. **Prefer upstreamable mechanisms.** A generic, small, well-justified kernel facility or a paravirtual transport beats a private accessor patch. 4. **Measurement-first.** No performance claim without an A/B distribution on the project's harness. 5. **Fail closed and reversible.** Every experimental path has a kill switch, cannot own hardware the baseline needs, and unloads cleanly. ### 5.2 Primary candidate — virtual accelerator transport (A) * Add a small virtual bus (auxiliary/vDPA-like) owned by an experimental module. * The open host kernel modules gain a transport backend that binds the bus and presents the GPU device model through it. The driver's *source* is extended in a clearly separated transport file; the proprietary artifact path is not used for this experiment. * No synthetic PCI config space, no fabricated BARs, no accessor patching: the transport carries register windows, interrupts and DMA semantics explicitly — the same model a paravirtual device uses, which is why it is exact and fast. * Execution: the host client is one more client of the real GSP-RM session, with its own root/objects/channels/page tables, subject to the same scheduler and the module's policy layer. ### 5.3 Alternative candidate — kernel trap plane (B) * A minimal, generic "virtual MMIO window" facility: a static-key-gated dispatcher reached from the accessor layer, covering `readX/writeX`, relaxed and raw variants, `memcpy_fromio`/`toio`, `memset_io`. * Requires rebuilt open modules (source untouched, artifact rebuilt). * Conditions: attempted upstream first with a generic justification; if not accepted, carried as a maintained patch with a version matrix and CI; never silently enabled. ### 5.4 Baseline-safe fallback — GPU-as-a-compartment (C) * The requestor's desktop stack (compositor + CUDA/3D) runs in a lightweight GPU compartment bound to an mdev exactly like a guest; Looking Glass presents its frames (and guests' frames) to the physical display; the backend executes real work. * Delivers S1–S4 and S7 with zero kernel/driver changes and the conventions the project already documents; the only difference is that the desktop's driver lives in its compartment rather than in dom0. * This is the recommended ceiling until A or B is accepted. --- ## 6. Full scope of the experimental feature (design detail for the future) ### 6.1 Device and transport plane * New experimental files only: bus/transport, device model, interrupt domain, DMA/ownership, display forwarding. * No reuse-by-rewrite of baseline internals: where baseline behaviour is needed, the experimental path calls stable exported entry points or duplicates the minimal logic with a documented contract. * The device model (register windows, interrupts, DMA) is explicit in the transport; no configuration-space emulation is needed for the stock driver if the transport is paravirtual. ### 6.2 Execution plane * Host client as a first-class GSP-RM client: distinct root, objects, channels, page tables; per-client accounting. * Fairness: host control bursts must not push guest RPC tail latency beyond the guest's deadline budget; the shared serialization point must have an acquisition bound or per-client queues. * Completions: real semaphore observation, relay into the host client's memory/interrupts; the existing guest path is untouched. ### 6.3 Memory and isolation * Zero-copy payload: host pages mapped once into the host client's GPU page tables; guest pages stay behind VFIO. * Isolation: revocable, domain-backed mappings; per-client ownership tables; cross-client addresses refused and counted; no identity default; negative tests before any shared run. ### 6.4 Display plane (stretch within the stretch) * Host-dom0 path: modeset/scanout programming forwarded to the real display engine; scanout reads real VRAM; HDMI/DP drives the host desktop. * Compartment path: guest or desktop compartment frames via KVMFR/IVSHMEM to the Looking Glass client on the physical display. * Display is phased last and independently gated; failure falls back to the compartment display without touching the baseline. ### 6.5 Lifecycle and ownership * Per-client refcounts; release only when the last client is gone; D0/resume coordination with persistenced/udev; supervised claim discipline retained; no second owner until the baseline lifecycle is demonstrably stable. ### 6.6 Experimental isolation policy (binding) * **Compile-time flag** (e.g. `OPENIOV_EXPERIMENTAL_HOST_ACCEL`) builds the experimental objects; default builds do not contain them. * **Module-load flag** (e.g. `host_accel_experimental=1` plus hazard ack) activates the built-in path; the code is inert without both gates. * **No rearchitecture:** shared functions, structures, locking, and the guest path are not edited to accommodate the experiment; needed behaviour is duplicated behind the flag with an explicit contract. * **A/B integrity:** with the flag off, binary and behaviour are identical to baseline; with the flag on but the module flag off, nothing runs; both are needed to compare and to isolate regressions. * **Not shipped:** release/default artifacts never enable the feature; unload and revert are one flag away. * **Telemetry:** experimental counters are namespaced and read-only. --- ## 7. Phased plan: prerequisites, testing, implementation Each phase has acceptance gates and abort criteria. No phase may modify a baseline path. **P0 — Prerequisites (must be complete before any experimental code):** multi-client GSP-RM proven (distinct client root + per-client page tables); guest execution bridge (P3–P6 of the guest-wiring design) complete; baseline lifecycle stable across repeated cycles; A/B harness working; profile plumbing real (not aspirational). *Abort if multi-client cannot be shown.* **P1 — Bench prototype, no baseline code:** transport/trap prototype in a scratch module and in a virtual machine; prove register semantics, interrupt delivery, DMA/IOMMU reachability, and probe on the chosen candidate (A or B). *Abort if the candidate cannot be exact on both x86_64 and aarch64.* **P2 — Experimental module skeleton behind the compile flag:** bus/transport, device model, lifecycle hooks to the existing backend API (read-only use), no execution. Gate: baseline rung tests green with the experiment built but disabled; guest path byte-identical with the flag off. **P3 — Control plane:** host client boots and initializes against the emulated control plane; `nvidia-smi` on the host client; no real work. Gate: guest ladder green before/after; no backend claim yet. **P4 — Execution plane:** host client becomes a real GSP-RM client; first correct host workload on hardware. Gate: guest CUDA unaffected; fairness and isolation counters clean. **P5 — Display:** host desktop on the physical outputs, guest frames in a window; fallback verified. Gate: no regression to guest display or baseline headless operation. **P6 — Measurement and hardening:** A/B distributions for kernels, copies, launch-bound, compositor; portability matrix; negative/fuzz tests for the ownership and IRQ paths; unload/soak. **P7 — Decision:** keep experimental, upstream a mechanism, or retire. The default is to retire unless the A/B and maintenance burden are acceptable. --- ## 8. Revisit gate (when this may start) All of the following must hold, and the user must explicitly decide to start: 1. Every baseline goal is complete and the rung ladder is green on the reference hardware and guest matrix. 2. Multi-client GSP-RM is proven and documented. 3. The guest execution bridge is complete and stable; no open freeze classes. 4. The profile/configuration surface is real (code reads it), not aspirational. 5. An accepted candidate exists — A (paravirtual) or B (patch with a maintenance commitment) — or the decision is to build C (compartment) instead. 6. The experimental isolation policy (§6.6) is written into the codebase and CI. Until then, this document is dormant. --- ## 9. Known knowns * Trapping a prebuilt driver's inlined MMIO from a module is impossible; a trap/transport plane is unavoidable for host-native stock-driver binding. * RAM-backed BARs need boot-reserved memory to be ioremap-able/claimable, and even then they are not observable. * The stock driver's probe requirements and register/RPC inventory are discoverable and testable (the project already owns tracing and analysis tooling). * The backend can execute real work; the missing piece is client multiplicity and the bridge, not raw engine capability. * Per-client GPU page tables are the isolation boundary that matters; platform IOMMU alone cannot separate same-device clients. ## 10. Known unknowns (each with its resolving experiment) * Whether GSP-RM supports a second concurrent client with independent roots and page tables — experiment: allocate a distinct root/client on the live session and schedule work from both. * The exact stock-driver register/RPC inventory for the host version — derive from traces and the open modules. * Whether upstream would accept a generic virtual-MMIO facility, and in what shape — write the RFC; measure reaction before committing to B. * Display-engine ownership under a multi-client GSP-RM session — forward a modeset on a test client and observe scanout. * aarch64/UMA reachability semantics for host pages — prototype on GB10-class hardware when available. ## 11. Possible unknown unknowns (discovery methods) * Hidden driver behaviours outside the modeled register/RPC set — full-window access discovery during bring-up, in a scratch VM, never on a production system. * Server-side limits (client counts, handle domains, scheduling granularity) not visible in the trace — probe the server's own logs and error paths with a supervised harness. * Firmware/driver version interactions introduced by a new host driver family — the version-keyed profile workflow (capture, diff, add) is the discovery mechanism. * Power/display interactions on laptops (MUX, panel ownership) — hardware bring-up matrix with the D0 discipline. * Undocumented vendor expectations encoded in the driver's identity checks — A/B against the host oracle with the same device data. ## 12. Limitations and the elegant workarounds | Limitation | Workaround that keeps the design simple | |---|---| | No upstream virtual-PCI framework | Use a paravirtual transport (A) or the compartment model (C); do not fabricate PCI | | Boot carve-out is a deployment change | Make it a documented requirement only for the experimental path; default systems are unaffected | | Only one physical GPU | One backend session, many logical clients; compartment model as the fallback | | Display forwarding is complex | Phase it last; fall back to compartment presentation, which needs no display forwarding | | Host driver version treadmill | Version-keyed profiles and discovery mode; the experiment is opt-in and can lag driver releases | | Single serialization point | Bounded acquisition, per-client accounting, or per-client queues before enabling the experiment | ## 13. Decision record * **2026-10-02 — first host-VF design reverted.** Load-bearing assumptions falsified (RAM-backed BARs, trap-free observation, unproven multi-client GSP-RM, wrong MSI mechanism, incomplete bridge reuse, overstated isolation and performance). All tracked edits reverted; the active tree is unchanged. * **2026-10-02 — goal retained here as a post-1.0 stretch goal.** Possible approaches, the recommended future architecture, the experimental isolation policy, and the revisit gate are recorded in this document. * **Policy:** do not implement unless the user explicitly asks; when implemented, it is experimental, compile-time gated, module-load gated, duplicate-path, and reversible; the baseline system is never compromised. The goal is **never pursued automatically** — it requires an explicit human decision after every baseline goal is complete and we are certain there is no risk — and it may be scrapped on architectural grounds. Until then, no aspect of the baseline software may be designed toward it.