diff options
Diffstat (limited to 'docs/01-orientation')
| -rw-r--r-- | docs/01-orientation/01-purpose-and-scope.md | 157 | ||||
| -rw-r--r-- | docs/01-orientation/02-the-ai-edge-and-ecosystem.md | 173 |
2 files changed, 330 insertions, 0 deletions
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md new file mode 100644 index 0000000..eab63bc --- /dev/null +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -0,0 +1,157 @@ +# Purpose and Scope + +> Read this first. It defines what Open-IOV is for, what it is not for, and how +> we will judge whether we are succeeding. + +## 1. Mission + +**Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without +modifying NVIDIA software, firmware or hardware.** + +Concretely, Open-IOV builds an independent, maintained IP block that: + +* presents a virtual NVIDIA GPU to a guest operating system, +* lets the guest run the **stock NVIDIA driver** (Linux and Windows), +* supports **compute (CUDA)** first, then **3D graphics** and **video**, +* runs on ordinary NVIDIA GPUs already deployed in the field, +* isolates guests from each other and from the host, +* and does not require a license key, a proprietary host service, a patched + guest driver, or a specific hypervisor. + +## 2. What problem are we solving? + +Today there are four ways to use an NVIDIA GPU with virtual machines, and all +of them have a hard limitation: + +| Approach | Limitation | +|---|---| +| Discrete device assignment (VFIO passthrough) | One GPU per VM. Complex, fragile for laptops and consumer PCs. No sharing. | +| NVIDIA proprietary vGPU | Requires a license. Requires a vendor host stack. Driver-version locked. Windows/Linux only where the vendor ships builds. | +| Software emulation of a GPU (e.g. QEMU's virtual GPUs, virgl) | No real CUDA. No real 3D performance. Not usable for AI. | +| Cloud-only inference | Needs an external service. Unacceptable for secure edge, offline, or private data. | + +Open-IOV occupies the gap: **real GPU acceleration, multiple VMs, stock +drivers, no license.** + +## 3. Scope + +### In scope + +* **Host-side virtualization of NVIDIA GPUs** for Linux hosts via VFIO/mdev + (KVM now; Xen and others later), as a self-contained kernel module plus a + small userspace library. +* **Full guest-visible device emulation**: PCI configuration space, BARs + (MMIO register space, VRAM aperture, instance memory), MSI-X, doorbells, + power management stubs. +* **GSP and GSP-RM interface emulation**: firmware boot handshake, message + queues, RPC transport, resource-manager controls, object model, event + channels, and the pieces of the hosted resource manager the guest depends on. +* **A memory model** that connects guest GPU address spaces to real GPU memory + and to guest system memory, including page tables and fault handling. +* **Compute forwarding**: taking the work the guest submits (channels, push + buffers, runlists, compute commands) and executing it on the physical GPU in + a way that is correct, isolated and fast. +* **Graphics and video forwarding** (later phase, same architecture). +* **Guest support**: Linux (x86_64 and aarch64) and Windows (x86_64). +* **Host support**: Linux (x86_64 now, aarch64 later), KVM today, Xen/Argo/ + OpenXT/Qubes OS Air on the roadmap. +* **Hardware portability**: Turing (TU104) today; Ampere, Hopper, + Grace-Hopper, Grace-Blackwell and GB10 targets; a configuration-driven + adaptation layer for new GPUs. + +### Out of scope (non-goals) + +* **Replacing bare-metal inference serving.** vLLM's continuous batching on + bare-metal hosts is how the industry scales multi-tenant inference in + datacenters, and it is better at that than any virtualization layer will be. + Open-IOV is for the cases where compute must live *inside* a guest. +* **Forking or patching the NVIDIA driver**, in guest or host. +* **Forking or patching GPU firmware.** +* **Forking or patching hypervisors** (QEMU, KVM, Xen, Argo, OpenXT, ...) to + make the device work. +* **Anti-cheat / DRM circumvention**, cloud-gaming-specifically defeating + vendor licensing checks for their own products. We interoperate; we do not + counterfeit. (We do not need a license for *our* device, because it is our + device.) +* **Emulating GPU instruction execution in software.** Software emulation of + CUDA kernels is 100–1000× too slow; work must run on real silicon. + +## 4. Value + +For **users**: GPU acceleration in VMs on hardware you own, with no per-VM fee, +on Windows and Linux, across generations of GPUs. + +For **integrators and hypervisor projects**: a GPU component that behaves like +any other virtual device, enabling security models that were previously +impossible (see HAAEEP in +[`../08-ecosystem/haaeep-and-collaboration.md`](../08-ecosystem/haaeep-and-collaboration.md)). + +For **the wider ecosystem**: an independent, permissively licensed +implementation of a previously single-vendor interface, with public +documentation of the protocol, the firmware and the architecture — a public +good that outlives any one vendor's product decisions. + +## 5. Model of multi-tenancy + +Open-IOV is built for **multiple independent tenants on one GPU**: + +* Each VM gets its own mediated device instance (mdev) and its own + guest-visible GPU. +* Memory is isolated by the platform IOMMU: a tenant's device can only reach + *that tenant's* memory. +* Interrupts are delivered per instance; no shared vectors. +* The host module holds no tenant-secret state; everything is per-VM. +* Scheduling between tenants is explicit policy (time-sliced now; MIG-backed + partitioning later where hardware supports it). +* The security posture is *fail closed*: if the isolation assumptions cannot be + verified at run time, the device refuses to start. + +How this is enforced and audited is documented in +[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md). + +## 6. How we judge success + +We evaluate the project on five axes; each has a dedicated methodology +document: + +1. **Initialization** — the guest driver loads, probes, and initializes to the + same point it does on bare metal: `nvidia-smi`, CUDA runtime, `cuInit`, + context creation, module load. +2. **Runtime** — the guest *runs* real work: CUDA kernels with correct results, + OpenGL/Vulkan rendering, encode/decode; repeated, sustained, resilient. +3. **Interoperability** — no changes required anywhere: guest driver builds, + host driver builds, firmware, hypervisor, device model software. Both host + and guest firmware interfaces are honored. +4. **Stability** — crash-free under stress: start/stop cycles, VM kill, guest + driver reload, host driver reload, suspend/resume, migration. +5. **Performance and security** — within an agreed factor of native for the + supported cases, with isolation that survives adversarial review. + +## 7. Where we are, and where we need improvement + +* **Working today**: the virtual device, the GSP boot handshake, the RPC + transport, the resource-manager emulation surface the guest driver needs to + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the early UVM + initialization path. +* **Needs improvement**: GPU work forwarding (the reason CUDA kernels do not + yet run), the memory/page-table model, Windows guest support, non-KVM + hypervisors, and performance instrumentation. Each of these is tracked with + explicit "known-unknowns" in + [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). + +## 8. Guiding principles + +1. **Own the interface, not a snapshot of it.** Every emulated interface is + described by a version-agnostic model plus per-version configuration; new + versions are added by measurement, not by rewriting. +2. **Emulate the interface, pass through the work.** Anything that must be fast + runs on real hardware; everything the guest can observe is virtual. +3. **No patches anywhere.** If a change is needed in someone else's software, + firmware or hardware, our design is wrong. +4. **Make the invisible visible.** Instrument everything; keep + `dmesg`-readable telemetry for every decision the emulation makes. +5. **Fail closed, fail loudly.** Isolation failures stop the device; protocol + surprises are logged with enough context to reproduce them. +6. **Document for the next person.** A feature is not done until a newcomer can + understand and maintain it from these docs alone. +7. **Permissive and portable by design** (BSD-3-Clause): adoption is a feature. diff --git a/docs/01-orientation/02-the-ai-edge-and-ecosystem.md b/docs/01-orientation/02-the-ai-edge-and-ecosystem.md new file mode 100644 index 0000000..58b1cee --- /dev/null +++ b/docs/01-orientation/02-the-ai-edge-and-ecosystem.md @@ -0,0 +1,173 @@ +# The AI Edge, Multi-Tenancy, and the Ecosystem Open-IOV Serves + +> This document explains *why now* — the industry context that makes a +> permissively licensed GPU virtualization block necessary, the kinds of +> multi-tenancy we do and do not serve, and the projects Open-IOV is designed +> to plug into. + +## 1. First, what we are not: vLLM on bare metal stays the datacenter answer + +Multi-tenant inference at datacenter scale — the kind measured in tokens per +second per dollar across an NVIDIA SuperPOD — is best served by **vLLM-style +continuous batching running on bare-metal GPU hosts**. That architecture gives +one model server the whole GPU, batches requests dynamically across tenants at +the *request* layer, and achieves utilization that no virtualization scheme can +match. Open-IOV will not replace it, and is not intended to. + +The correct layering is: + +``` +Datacenter / centralized inference → bare-metal vLLM (unchanged) +Edge / sovereign / compartmentalized → virtualized GPUs, in guests (Open-IOV) +Everything in between → whichever constraint is binding +``` + +Open-IOV exists for the cases that bare-metal serving *cannot* satisfy: + +* **Windows guest compute.** CUDA in Windows VMs, for applications that are + not Linux containers. +* **3D graphics and video** in guests, on ordinary GPUs. +* **Secure edge inference** where data must not leave the device, and where + agents, models, and infrastructure must be isolated from each other. +* **Shared GPUs on consumer/edge hardware**, where discrete passthrough is + impossible or wasteful (laptops, small servers, single-GPU workstations). + +## 2. The three needs that converge + +### 2.1 Compute inside guests (Linux and Windows) + +AI engineering and interactive agent work increasingly happen on endpoints and +edge servers. Those environments are not Kubernetes clusters; they are +workstations, laptops, kiosks, and small virtualized servers. Users of these +systems need the *guest* to have a real CUDA runtime, real 3D, and real +video — without a vendor license, a vendor host stack, or a version-locked +driver. + +### 2.2 A secure agent runtime at the edge + +An autonomous agent (Claude Code, OpenCode, a harness of your choosing) is a +**capability-constrained, untrusted-by-default workload**. It has network +access, it runs arbitrary generated code, and it should never be able to reach +the model weights, the training data, or another tenant's GPU state. Running +the agent in its own VM with a *virtualized* GPU — rather than sharing a +physical GPU through a passthrough device or through a process-level +abstraction — gives the isolation boundary the security model needs. + +Because this need is concrete and urgent, and because no permissively licensed +implementation exists, **now is the right time to reboot Open-IOV** as a +maintained project rather than a research artifact. + +### 2.3 Durability across hardware, firmware and software generations + +Every GPU virtualization deployment eventually faces: a new driver version, a +new firmware image, a new GPU, a new kernel, a new hypervisor. A stack that +cannot be updated by its users is a dead end. Open-IOV is designed as a +**config-driven, version-aware IP block** so those updates are routine (see +[`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) +and [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)). + +## 3. HAAEEP: a High Assurance Agent Edge Environment Platform + +We propose a concrete, buildable target that ties all of the above together — +and we invite others to co-develop it. + +``` +┌───────────────────────────────────────────────────────────────────────┐ +│ EDGE DEVICE (single GPU, local inference only, fully offline) │ +│ │ +│ ┌──────────────────┐ capability-constrained, untrusted by default │ +│ │ Agent VM(s) │ (Claude Code / OpenCode / harness) │ +│ │ no GPU │ may hold credentials & code │ +│ └────────┬─────────┘ │ +│ │ API calls to model server (no external endpoint) │ +│ ┌────────┴─────────┐ model server VM: vLLM / SGLang / ... │ +│ │ Inference VM │ owns the GPU (via Open-IOV mdev or Xen fancy │ +│ │ GPU: virtual │ forwarding); isolated by the GPU hypervisor │ +│ └────────┬─────────┘ │ +│ │ │ +│ ┌────────┴─────────┐ infrastructure VMs: networking, storage, │ +│ │ Other VMs │ user desktop, ... — GPU-accelerated where │ +│ │ GPU: virtual │ needed, without touching the inference weights │ +│ └──────────────────┘ │ +│ │ +│ Hypervisor: KVM / Xen + Argo/OpenXT / Qubes OS Air — with │ +│ TrenchBoot measured boot and Flask-XSM/HMX mandatory access control │ +└───────────────────────────────────────────────────────────────────────┘ +``` + +Design points: + +1. **No external inference.** The model runs locally on the edge device inside + its own VM. No requests to an external OpenAI-style endpoint are required — + or permitted by the platform's network policy. +2. **GPU virtualization provides the separation.** A VM that owns the physical + GPU (e.g. a Xen domain with `xen-pci` passthrough) can serve GPU acceleration + to other domains through Open-IOV; an agent VM can never touch the GPU's + state, and the inference VM's weights are isolated from the agent VM. +3. **Layered isolation.** Agent VMs (capability-constrained) ≠ inference VM + (holds weights) ≠ adjacent functional VMs (desktop, network, storage) ≠ + hypervisor. Each boundary is enforced by a different mechanism (mandatory + access control, device virtualization, IOMMU). +4. **KV cache interoperability.** The inference VM's vLLM/SGLang deployment + keeps using its own plugins (including KV-cache management), unchanged. +5. **Attestable.** With TrenchBoot, the edge device can measure the + firmware → hypervisor → OS chain, so a remote party can verify the platform + before sending sensitive work to it. + +We call this the **High Assurance Agent Edge Environment Platform (HAAEEP)**. +It is a *use-case architecture*, not a product: Open-IOV provides the GPU +component, and each platform project provides the rest. See +[`../08-ecosystem/haaeep-and-collaboration.md`](../08-ecosystem/haaeep-and-collaboration.md) +for the full design, including the roadmap and how to help. + +## 4. The ecosystem Open-IOV plugs into + +Open-IOV is an *IP block*: it should be connectable to ecosystems without +those ecosystems having to modify themselves. + +| Project / platform | What it provides | What Open-IOV adds | +|---|---|---| +| **KVM + QEMU/VFIO** | Mainstream virtualization, VFIO/mdev transport | A GPU that works like any virtual device; multiple VMs per GPU | +| **Xen** | Strong isolation between domains, `xen-pci` forwarding | GPU virtualization *inside* domains; a domain can serve GPU to others | +| **Argo / OpenXT** | Hypervisor-level security, Hypervisor Mediated Exchange (HMX), Flask-XSM mandatory access control | A GPU path that respects MAC policy between VMs | +| **Qubes OS / Qubes OS Air** | Security-by-compartmentalization on modern hardware; administrative separation | GPU acceleration per compartment, especially AI workloads | +| **TrenchBoot** | Measured/attested boot (firmware → hypervisor → OS) | An attestable GPU service: the platform can prove what runs the model | +| **KVM/VFIO gaming setups** | Fast Windows VM + GPU passthrough | Solutions for users who need *multiple* VMs, or who cannot dedicate the whole GPU | +| **vLLM / SGLang / inference servers** | Continuous batching, KV-cache plugins, model serving | A way to run the server itself inside an isolated VM without losing GPU acceleration | +| **Windows driver ecosystems** | CUDA and 3D applications | Compute in Windows guests on consumer hardware | + +The unifying idea: **Open-IOV should be the component these projects reach for +when they need "an NVIDIA GPU, virtualized, on our terms."** + +## 5. Collaboration and IP models + +We explicitly invite *both* open source and commercial collaboration: + +* **Open source projects**: adopt BSD-3-Clause code directly; we will work with + you on interfaces (VFIO/mdev today, other transports on request) and accept + upstream requirements as design inputs. +* **Commercial vendors and integrators**: ship Open-IOV in products, add + proprietary value *on top*, fund specific hardware/driver-version support, + or co-develop with us under the same permissive terms. No license fees, no + contributor agreements that take rights away. +* **Hardware vendors**: nothing is required from you. Our goal is that new + hardware works through configuration and measurement, without firmware or + driver changes. If you want to help, test hardware and share traces. +* **Researchers and auditors**: the documentation in this tree — including the + protocol and firmware analysis — is written to be attackable. We want your + findings. + +The license split (BSD-3-Clause for the reusable project, GPL-2.0 for kernel +modules that link GPL-only kernel interfaces) is chosen to make all of the +above legally frictionless. See [`../09-reference/legal-and-licensing.md`](../09-reference/legal-and-licensing.md). + +## 6. What success looks like for this document + +* A security-focused platform project can integrate Open-IOV without asking us + to change the software. +* A user can run CUDA in a Windows guest on a laptop with one GPU, legally and + without a license server. +* An edge device can run an agent VM, an inference VM and a desktop VM on one + GPU, with the model never leaving the device. +* A new GPU generation is supported by a configuration update plus measured + verification — not a rewrite. |