path: root/docs/10-post-1.0/experimental-host-acceleration.md

# Experimental Host Acceleration and Host Display — Post-1.0 Goal

> ## ⛔ DO NOT IMPLEMENT THIS DOCUMENT
>
> **This is not an implementation instruction.** No agent may act on anything in
> this document unless the user *explicitly* asks for this experimental feature
> to be implemented. It exists only so that the design is not lost, and so that
> the decision to revisit it is deliberate. It is deliberately **not** linked
> from the documentation front door and is **not** part of the active reading
> order until the revisit gate in §8 passes.
>
> **Status:** deferred / post-1.0 stretch goal. Reverted from the active tree on
> 2026-10-02 after independent adversarial, soundness, and VM-risk reviews
> falsified the first design. The review record is kept privately.
>
> **Experimental policy (binding when the time comes):** any host-acceleration
> work is experimental; it may only be built behind a compile-time flag and
> enabled by an explicit module-load flag; it must not alter, replace, or
> rearchitect any existing code path; where it needs different behaviour it
> must **replicate** the code it needs in its own files, gated and isolated.

---

## 0. Why this document exists, and why the first design was reverted

In October 2026 a full design (`13-host-vf-acceleration-design.md`) was written
for binding the stock host `nvidia.ko` to a synthetic PCI function whose BARs
were backed by RAM ("trap-free windows"), with GSP boot/RPC served by the
existing emulation and execution forwarded to the real backend. Three
independent reviews (adversarial design review, engineering-soundness/criteria
audit, and VM-support risk review) plus direct checks against the kernel tree
found load-bearing assumptions that do not hold. The design and the seven doc
edits that introduced it were reverted; this document replaces them with a
correct, narrower record and a future direction that does not compromise the
system we are building today.

### 0.1 The findings that forced the revert

| # | Finding | Evidence (verified) | Why it is fatal to the old design |
|---|---|---|---|
| 1 | `ioremap` refuses ordinary System RAM | `arch/x86/mm/ioremap.c:68-90`, `:214-218`; `arch/arm64/mm/ioremap.c:16` | A runtime RAM-backed BAR cannot be mapped by the stock driver; the "memremap fallback" fails because the driver itself calls `ioremap` |
| 2 | `request_mem_region` fails on busy System RAM | `kernel/resource.c` conflict path (`-EBUSY`); the driver requests BAR0 regions | The driver's probe fails before any emulation runs |
| 3 | Pure change detection cannot observe same-value writes or read side effects | by construction: no dirty bit on RAM, and reads are never seen | The "trap-free observable window" cannot be correct for edge-triggered or read-clear registers |
| 4 | Real GSP-RM multi-client support is unproven | the only second-client attempt failed `0x23`; the guest bridge still refuses a second mdev | Host + guest coexistence — the whole point — rests on an untested server capability |
| 5 | Wrong MSI mechanism | `platform_msi_write_msg` derefs `platform_data` the PCI path never allocates (`drivers/base/platform-msi.c:83-88`) | Interrupt delivery would NULL-deref; the correct API is `pci_msi_create_irq_domain` |
| 6 | Synthetic bridge bring-up is not routine | x86 derefs `bridge->bus->sysdata` when `parent == NULL` (`arch/x86/pci/acpi.c:591-604`); `pci_scan_root_bus_bridge` does not size/assign BARs or add devices (`drivers/pci/probe.c:3256-3294`) | Registration can oops, and BARs stay unset |
| 7 | The reused probe contract and device model were wrong | reused config model is VGA class `0x030000` (`kernel/main.c:1725-1726`), not 3D class; `request_mem_region`, RM identity checks, vgaarb/PM/DRM paths are all unaddressed | The stock probe would not complete |
| 8 | "Reuse the bridge unchanged" was false | the sema service and RM emulation are coupled to `vfio_dma_rw`/guest GPAs; the execution bridge is not in the tree | The host path needs a cross-client transport redesign, not a small additive seam |
| 9 | Ownership/lifecycle and isolation claims were overstated | release keyed to "last mdev"; shared DMA domain; software-only isolation | A live host client could be torn down with the backend; "full IOMMU protection" was not true as stated |
| 10 | Performance targets were unsupported | poll cadence, method rebuild, single serializing lock, and absent forwarding were not in the arithmetic | No honest claim of "within a small percentage of native" was possible |

The deeper reason: the original constraints (zero Linux diff, no `nvidia.ko`
change) force a *trap-free observation* plane for a *prebuilt* driver, and the
kernel gives no supported way to make that plane correct. Any future design must
either (a) add a real trap/transport plane with an explicit, maintained policy
exception, or (b) not put the stock driver in the host kernel at all.

### 0.2 What the revert preserves

* The baseline system and its active critical path are untouched.
* The project's no-patch rule and component boundaries are intact.
* The review record is kept privately for the future revisit.
* The goal itself is not lost — it is captured here, with the corrected
  understanding.

---

## 1. Where this fits: baseline goals vs the stretch goal

**Baseline (the only work that is scheduled; all are project goals):** guest
CUDA execution on the physical engines through the module-owned backend; the
mdev/VFIO guest path; multi-VM; 3D; Windows guests; isolation; performance;
hardware/software portability. Until these are complete, nothing in this
document may be started.

**Stretch goal (this document):** on a single-GPU system, let the *host OS
itself* use the GPU at full capability — host desktop (X11/Wayland) on the
physical outputs (HDMI/DP), host CUDA/3D through the real stock driver, and
guests running concurrently in compartments, with scheduling and
compartmentalization enforced by `open-iov-vfio.ko`.

This stretch goal is explicitly **one**: it is attempted only after every
baseline goal is met, and it must not jeopardize them.

---

## 2. Requirements for the stretch goal (unchanged intent, corrected framing)

| # | Requirement |
|---|---|
| S1 | The host's real, stock `nvidia.ko` provides host CUDA/3D and the host desktop |
| S2 | Full display output from the GPU's HDMI/DP, driving the host desktop |
| S3 | Guest frame buffers are shown on the host (Looking Glass / KVMFR / IVSHMEM), both sides accelerated |
| S4 | Host and guest work coexist on one GPU under `open-iov-vfio.ko` scheduling and compartmentalization |
| S5 | High performance: work-dominated workloads within a few percent of native; launch-bound latency bounded and published |
| S6 | No unnecessary memory copies; zero-copy workload path |
| S7 | Isolation: a guest can never reach host memory or host interrupts; fail closed |
| S8 | Portability across kernel, driver, GSP and VFIO/mdev revisions |
| S9 | The baseline system is never compromised: no rearchitecture, no altered core paths, experimental-only, dual-gated |
| S10 | The diff is as small and reviewable as possible; upstreamable mechanisms are preferred over private ones |

**Non-goals:** Windows host; replacing the guest path; patching proprietary
components in place; shipping experimental code in default builds.

---

## 3. Why the old approach is retired (with the corrected direction)

| Retired detail | Corrected direction |
|---|---|
| RAM-backed BARs without a boot carve-out | If a synthetic function is ever used, its BARs must live in firmware/boot-reserved memory (e.g. `memmap=`) — a documented deployment requirement, proven on x86_64 **and** aarch64 — or the plane is not used at all |
| Pure change detection for triggers and reads | Edge-triggered and read-side-effect registers require a real trap plane: a small, generic kernel facility (accessor dispatch) or a paravirtual transport. There is no trap-free substitute |
| `platform_msi_*` for a PCI device | Use the PCI MSI domain (`pci_msi_create_irq_domain` + a custom chip); MSI-X table semantics verified first |
| Bridge registration without parent/sysdata | Follow the in-tree software-bridge pattern (valid sysdata/parent, explicit domain, bus-number allocation, resource sizing/assignment, device add) |
| "Reuse the bridge unchanged" | Provide a genuine cross-client transport abstraction; treat the execution bridge and multi-client support as prerequisites, not reused assets |
| "Last mdev removes the backend" | Ownership is per-client; release only when the last *client* (host or guest) is gone; D0/resume coordinated with persistenced/udev |
| "IOMMU protection" as a software statement | Isolation is enforced by revocable, domain-backed mappings plus per-client GPU page tables; identity/passthrough is never the default |
| Unmeasured performance claims | Measurement-first: no target is published until an A/B distribution exists |

---

## 4. Candidate approaches and trade-offs

| Approach | Exact? | Fast? | Kernel diff | Driver change | Host `nvidia.ko` | Robustness/portability | Standing |
|---|---|---|---|---|---|---|---|
| A. Paravirtual "virtual accelerator bus" in the open host modules | Yes | Yes | No (bus/driver code in our + open driver transport) | Yes (source, not artifact) | Yes | High; upstream-plausible (virtio/vDPA-like) | **Primary candidate** |
| B. Accessor-dispatch kernel patch + rebuilt open driver | Yes | Yes | Yes (narrow files, wide blast radius) | Rebuild only (artifact) | Yes | Medium; out-of-tree patch maintenance burden; upstream unlikely | **Alternative, only with a maintenance commitment** |
| C. GPU-as-a-compartment (mdev/VFIO micro-VM + Looking Glass) | Yes | Yes | No | No | Driver runs in the desktop compartment, not dom0 | High; uses existing project design and conventions | **Baseline-safe fallback; recommended ceiling until A/B are accepted** |
| D. Boot-reserved carve-out + synthetic PCI + a trap plane (T1/T2) | With trap | With trap | Only if T2 | Only if T2 | Yes | Low–medium; boot carve-out + wide patch surface | Retired as a standalone; only as a means inside B |
| E. Trap-free RAM-backed synthetic PCI (the reverted design) | No | No | No | No | Yes | Falsified | **Retired — do not revive** |
| F. kmmio debug trap | Approximate | No (µs/access, single-step) | No | No | Yes | Prototype-only | Test/validation tool only |
| G. Time-sliced physical-function ownership | Yes | Yes | No | No | PF-bound, not virtual; not concurrent | High | Baseline fallback for "host acceleration without guests" |
| H. mdev bound directly by `nvidia.ko` | — | — | — | — | No | Impossible (bus mismatch) | Rejected |
| I. SR-IOV VF / vendor vGPU | Yes | Yes | No | Vendor | Not on this silicon/vendor semantics | Hardware-gated | Revisit only if silicon appears |

Key trade-off: the original constraints forbade B and A, which are the only two
approaches that are simultaneously exact and fast. The user has since stated
that a justified, maintained kernel patch is acceptable; that reopens B, and A
remains the most upstreamable and maintainable. Either way, the trap/transport
plane is the *whole* design problem — everything else (device model, forwarding,
display) exists or is already designed.

---

## 5. Recommended future architecture (when the revisit gate passes)

### 5.1 Principles

1. **Baseline isolation first.** No existing function is modified to make room
   for the experiment. If the experiment needs a different behaviour, that
   behaviour is implemented in experimental files as a duplicate path selected
   at build time, not by changing the shared function.
2. **Dual gating.** A compile-time flag builds the experimental objects; a
   module-load flag (plus the existing hazard acknowledgement) activates them.
   Default builds and default loads never contain or run the code.
3. **Prefer upstreamable mechanisms.** A generic, small, well-justified kernel
   facility or a paravirtual transport beats a private accessor patch.
4. **Measurement-first.** No performance claim without an A/B distribution on
   the project's harness.
5. **Fail closed and reversible.** Every experimental path has a kill switch,
   cannot own hardware the baseline needs, and unloads cleanly.

### 5.2 Primary candidate — virtual accelerator transport (A)

* Add a small virtual bus (auxiliary/vDPA-like) owned by an experimental module.
* The open host kernel modules gain a transport backend that binds the bus and
  presents the GPU device model through it. The driver's *source* is extended in
  a clearly separated transport file; the proprietary artifact path is not used
  for this experiment.
* No synthetic PCI config space, no fabricated BARs, no accessor patching: the
  transport carries register windows, interrupts and DMA semantics explicitly —
  the same model a paravirtual device uses, which is why it is exact and fast.
* Execution: the host client is one more client of the real GSP-RM session, with
  its own root/objects/channels/page tables, subject to the same scheduler and
  the module's policy layer.

### 5.3 Alternative candidate — kernel trap plane (B)

* A minimal, generic "virtual MMIO window" facility: a static-key-gated
  dispatcher reached from the accessor layer, covering `readX/writeX`, relaxed
  and raw variants, `memcpy_fromio`/`toio`, `memset_io`.
* Requires rebuilt open modules (source untouched, artifact rebuilt).
* Conditions: attempted upstream first with a generic justification; if not
  accepted, carried as a maintained patch with a version matrix and CI; never
  silently enabled.

### 5.4 Baseline-safe fallback — GPU-as-a-compartment (C)

* The requestor's desktop stack (compositor + CUDA/3D) runs in a lightweight
  GPU compartment bound to an mdev exactly like a guest; Looking Glass presents
  its frames (and guests' frames) to the physical display; the backend executes
  real work.
* Delivers S1–S4 and S7 with zero kernel/driver changes and the conventions the
  project already documents; the only difference is that the desktop's driver
  lives in its compartment rather than in dom0.
* This is the recommended ceiling until A or B is accepted.

---

## 6. Full scope of the experimental feature (design detail for the future)

### 6.1 Device and transport plane

* New experimental files only: bus/transport, device model, interrupt domain,
  DMA/ownership, display forwarding.
* No reuse-by-rewrite of baseline internals: where baseline behaviour is needed,
  the experimental path calls stable exported entry points or duplicates the
  minimal logic with a documented contract.
* The device model (register windows, interrupts, DMA) is explicit in the
  transport; no configuration-space emulation is needed for the stock driver if
  the transport is paravirtual.

### 6.2 Execution plane

* Host client as a first-class GSP-RM client: distinct root, objects, channels,
  page tables; per-client accounting.
* Fairness: host control bursts must not push guest RPC tail latency beyond the
  guest's deadline budget; the shared serialization point must have an
  acquisition bound or per-client queues.
* Completions: real semaphore observation, relay into the host client's
  memory/interrupts; the existing guest path is untouched.

### 6.3 Memory and isolation

* Zero-copy payload: host pages mapped once into the host client's GPU page
  tables; guest pages stay behind VFIO.
* Isolation: revocable, domain-backed mappings; per-client ownership tables;
  cross-client addresses refused and counted; no identity default; negative
  tests before any shared run.

### 6.4 Display plane (stretch within the stretch)

* Host-dom0 path: modeset/scanout programming forwarded to the real display
  engine; scanout reads real VRAM; HDMI/DP drives the host desktop.
* Compartment path: guest or desktop compartment frames via KVMFR/IVSHMEM to
  the Looking Glass client on the physical display.
* Display is phased last and independently gated; failure falls back to the
  compartment display without touching the baseline.

### 6.5 Lifecycle and ownership

* Per-client refcounts; release only when the last client is gone; D0/resume
  coordination with persistenced/udev; supervised claim discipline retained;
  no second owner until the baseline lifecycle is demonstrably stable.

### 6.6 Experimental isolation policy (binding)

* **Compile-time flag** (e.g. `OPENIOV_EXPERIMENTAL_HOST_ACCEL`) builds the
  experimental objects; default builds do not contain them.
* **Module-load flag** (e.g. `host_accel_experimental=1` plus hazard ack)
  activates the built-in path; the code is inert without both gates.
* **No rearchitecture:** shared functions, structures, locking, and the guest
  path are not edited to accommodate the experiment; needed behaviour is
  duplicated behind the flag with an explicit contract.
* **A/B integrity:** with the flag off, binary and behaviour are identical to
  baseline; with the flag on but the module flag off, nothing runs; both are
  needed to compare and to isolate regressions.
* **Not shipped:** release/default artifacts never enable the feature; unload
  and revert are one flag away.
* **Telemetry:** experimental counters are namespaced and read-only.

---

## 7. Phased plan: prerequisites, testing, implementation

Each phase has acceptance gates and abort criteria. No phase may modify a
baseline path.

**P0 — Prerequisites (must be complete before any experimental code):**
multi-client GSP-RM proven (distinct client root + per-client page tables);
guest execution bridge (P3–P6 of the guest-wiring design) complete; baseline
lifecycle stable across repeated cycles; A/B harness working; profile plumbing
real (not aspirational). *Abort if multi-client cannot be shown.*

**P1 — Bench prototype, no baseline code:** transport/trap prototype in a
scratch module and in a virtual machine; prove register semantics, interrupt
delivery, DMA/IOMMU reachability, and probe on the chosen candidate (A or B).
*Abort if the candidate cannot be exact on both x86_64 and aarch64.*

**P2 — Experimental module skeleton behind the compile flag:** bus/transport,
device model, lifecycle hooks to the existing backend API (read-only use),
no execution. Gate: baseline rung tests green with the experiment built but
disabled; guest path byte-identical with the flag off.

**P3 — Control plane:** host client boots and initializes against the emulated
control plane; `nvidia-smi` on the host client; no real work. Gate: guest
ladder green before/after; no backend claim yet.

**P4 — Execution plane:** host client becomes a real GSP-RM client; first
correct host workload on hardware. Gate: guest CUDA unaffected; fairness and
isolation counters clean.

**P5 — Display:** host desktop on the physical outputs, guest frames in a
window; fallback verified. Gate: no regression to guest display or baseline
headless operation.

**P6 — Measurement and hardening:** A/B distributions for kernels, copies,
launch-bound, compositor; portability matrix; negative/fuzz tests for the
ownership and IRQ paths; unload/soak.

**P7 — Decision:** keep experimental, upstream a mechanism, or retire. The
default is to retire unless the A/B and maintenance burden are acceptable.

---

## 8. Revisit gate (when this may start)

All of the following must hold, and the user must explicitly decide to start:

1. Every baseline goal is complete and the rung ladder is green on the
   reference hardware and guest matrix.
2. Multi-client GSP-RM is proven and documented.
3. The guest execution bridge is complete and stable; no open freeze classes.
4. The profile/configuration surface is real (code reads it), not aspirational.
5. An accepted candidate exists — A (paravirtual) or B (patch with a maintenance
   commitment) — or the decision is to build C (compartment) instead.
6. The experimental isolation policy (§6.6) is written into the codebase and CI.

Until then, this document is dormant.

---

## 9. Known knowns

* Trapping a prebuilt driver's inlined MMIO from a module is impossible; a
  trap/transport plane is unavoidable for host-native stock-driver binding.
* RAM-backed BARs need boot-reserved memory to be ioremap-able/claimable, and
  even then they are not observable.
* The stock driver's probe requirements and register/RPC inventory are
  discoverable and testable (the project already owns tracing and analysis
  tooling).
* The backend can execute real work; the missing piece is client multiplicity
  and the bridge, not raw engine capability.
* Per-client GPU page tables are the isolation boundary that matters; platform
  IOMMU alone cannot separate same-device clients.

## 10. Known unknowns (each with its resolving experiment)

* Whether GSP-RM supports a second concurrent client with independent roots
  and page tables — experiment: allocate a distinct root/client on the live
  session and schedule work from both.
* The exact stock-driver register/RPC inventory for the host version — derive
  from traces and the open modules.
* Whether upstream would accept a generic virtual-MMIO facility, and in what
  shape — write the RFC; measure reaction before committing to B.
* Display-engine ownership under a multi-client GSP-RM session — forward a
  modeset on a test client and observe scanout.
* aarch64/UMA reachability semantics for host pages — prototype on GB10-class
  hardware when available.

## 11. Possible unknown unknowns (discovery methods)

* Hidden driver behaviours outside the modeled register/RPC set — full-window
  access discovery during bring-up, in a scratch VM, never on a production
  system.
* Server-side limits (client counts, handle domains, scheduling granularity)
  not visible in the trace — probe the server's own logs and error paths with a
  supervised harness.
* Firmware/driver version interactions introduced by a new host driver family —
  the version-keyed profile workflow (capture, diff, add) is the discovery
  mechanism.
* Power/display interactions on laptops (MUX, panel ownership) — hardware
  bring-up matrix with the D0 discipline.
* Undocumented vendor expectations encoded in the driver's identity checks —
  A/B against the host oracle with the same device data.

## 12. Limitations and the elegant workarounds

| Limitation | Workaround that keeps the design simple |
|---|---|
| No upstream virtual-PCI framework | Use a paravirtual transport (A) or the compartment model (C); do not fabricate PCI |
| Boot carve-out is a deployment change | Make it a documented requirement only for the experimental path; default systems are unaffected |
| Only one physical GPU | One backend session, many logical clients; compartment model as the fallback |
| Display forwarding is complex | Phase it last; fall back to compartment presentation, which needs no display forwarding |
| Host driver version treadmill | Version-keyed profiles and discovery mode; the experiment is opt-in and can lag driver releases |
| Single serialization point | Bounded acquisition, per-client accounting, or per-client queues before enabling the experiment |

## 13. Decision record

* **2026-10-02 — first host-VF design reverted.** Load-bearing assumptions
  falsified (RAM-backed BARs, trap-free observation, unproven multi-client
  GSP-RM, wrong MSI mechanism, incomplete bridge reuse, overstated isolation and
  performance). All tracked edits reverted; the active tree is unchanged.
* **2026-10-02 — goal retained here as a post-1.0 stretch goal.** Possible
  approaches, the recommended future architecture, the experimental isolation
  policy, and the revisit gate are recorded in this document.
* **Policy:** do not implement unless the user explicitly asks; when
  implemented, it is experimental, compile-time gated, module-load gated,
  duplicate-path, and reversible; the baseline system is never compromised.