1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
# The AI Edge, Multi-Tenancy, and the Ecosystem Open-IOV Serves
> This document explains *why now* — the industry context that makes a
> permissively licensed GPU virtualization block necessary, the kinds of
> multi-tenancy we do and do not serve, and the projects Open-IOV is designed
> to plug into.
## 1. First, what we are not: vLLM on bare metal stays the datacenter answer
Multi-tenant inference at datacenter scale — the kind measured in tokens per
second per dollar across an NVIDIA SuperPOD — is best served by **vLLM-style
continuous batching running on bare-metal GPU hosts**. That architecture gives
one model server the whole GPU, batches requests dynamically across tenants at
the *request* layer, and achieves utilization that no virtualization scheme can
match. Open-IOV will not replace it, and is not intended to.
The correct layering is:
```
Datacenter / centralized inference → bare-metal vLLM (unchanged)
Edge / sovereign / compartmentalized → virtualized GPUs, in guests (Open-IOV)
Everything in between → whichever constraint is binding
```
Open-IOV exists for the cases that bare-metal serving *cannot* satisfy:
* **Windows guest compute.** CUDA in Windows VMs, for applications that are
not Linux containers.
* **3D graphics and video** in guests, on ordinary GPUs.
* **Secure edge inference** where data must not leave the device, and where
agents, models, and infrastructure must be isolated from each other.
* **Shared GPUs on consumer/edge hardware**, where discrete passthrough is
impossible or wasteful (laptops, small servers, single-GPU workstations).
## 2. The three needs that converge
### 2.1 Compute inside guests (Linux and Windows)
AI engineering and interactive agent work increasingly happen on endpoints and
edge servers. Those environments are not Kubernetes clusters; they are
workstations, laptops, kiosks, and small virtualized servers. Users of these
systems need the *guest* to have a real CUDA runtime, real 3D, and real
video — without a vendor license, a vendor host stack, or a version-locked
driver.
### 2.2 A secure agent runtime at the edge
An autonomous agent (Claude Code, OpenCode, a harness of your choosing) is a
**capability-constrained, untrusted-by-default workload**. It has network
access, it runs arbitrary generated code, and it should never be able to reach
the model weights, the training data, or another tenant's GPU state. Running
the agent in its own VM with a *virtualized* GPU — rather than sharing a
physical GPU through a passthrough device or through a process-level
abstraction — gives the isolation boundary the security model needs.
Because this need is concrete and urgent, and because no permissively licensed
implementation exists, **now is the right time to reboot Open-IOV** as a
maintained project rather than a research artifact.
### 2.3 Durability across hardware, firmware and software generations
Every GPU virtualization deployment eventually faces: a new driver version, a
new firmware image, a new GPU, a new kernel, a new hypervisor. A stack that
cannot be updated by its users is a dead end. Open-IOV is designed as a
**config-driven, version-aware IP block** so those updates are routine (see
[`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md)
and [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)).
## 3. HAAEEP: a High Assurance Agent Edge Environment Platform
We propose a concrete, buildable target that ties all of the above together —
and we invite others to co-develop it.
```
┌───────────────────────────────────────────────────────────────────────┐
│ EDGE DEVICE (single GPU, local inference only, fully offline) │
│ │
│ ┌──────────────────┐ capability-constrained, untrusted by default │
│ │ Agent VM(s) │ (Claude Code / OpenCode / harness) │
│ │ no GPU │ may hold credentials & code │
│ └────────┬─────────┘ │
│ │ API calls to model server (no external endpoint) │
│ ┌────────┴─────────┐ model server VM: vLLM / SGLang / ... │
│ │ Inference VM │ owns the GPU (via Open-IOV mdev or Xen fancy │
│ │ GPU: virtual │ forwarding); isolated by the GPU hypervisor │
│ └────────┬─────────┘ │
│ │ │
│ ┌────────┴─────────┐ infrastructure VMs: networking, storage, │
│ │ Other VMs │ user desktop, ... — GPU-accelerated where │
│ │ GPU: virtual │ needed, without touching the inference weights │
│ └──────────────────┘ │
│ │
│ Hypervisor: KVM / Xen + Argo/OpenXT / Qubes OS Air — with │
│ TrenchBoot measured boot and Flask-XSM/HMX mandatory access control │
└───────────────────────────────────────────────────────────────────────┘
```
Design points:
1. **No external inference.** The model runs locally on the edge device inside
its own VM. No requests to an external OpenAI-style endpoint are required —
or permitted by the platform's network policy.
2. **GPU virtualization provides the separation.** A VM that owns the physical
GPU (e.g. a Xen domain with `xen-pci` passthrough) can serve GPU acceleration
to other domains through Open-IOV; an agent VM can never touch the GPU's
state, and the inference VM's weights are isolated from the agent VM.
3. **Layered isolation.** Agent VMs (capability-constrained) ≠ inference VM
(holds weights) ≠ adjacent functional VMs (desktop, network, storage) ≠
hypervisor. Each boundary is enforced by a different mechanism (mandatory
access control, device virtualization, IOMMU).
4. **KV cache interoperability.** The inference VM's vLLM/SGLang deployment
keeps using its own plugins (including KV-cache management), unchanged.
5. **Attestable.** With TrenchBoot, the edge device can measure the
firmware → hypervisor → OS chain, so a remote party can verify the platform
before sending sensitive work to it.
We call this the **High Assurance Agent Edge Environment Platform (HAAEEP)**.
It is a *use-case architecture*, not a product: Open-IOV provides the GPU
component, and each platform project provides the rest. See
[`../08-ecosystem/haaeep-and-collaboration.md`](../08-ecosystem/haaeep-and-collaboration.md)
for the full design, including the roadmap and how to help.
## 4. The ecosystem Open-IOV plugs into
Open-IOV is an *IP block*: it should be connectable to ecosystems without
those ecosystems having to modify themselves.
| Project / platform | What it provides | What Open-IOV adds |
|---|---|---|
| **KVM + QEMU/VFIO** | Mainstream virtualization, VFIO/mdev transport | A GPU that works like any virtual device; multiple VMs per GPU |
| **Xen** | Strong isolation between domains, `xen-pci` forwarding | GPU virtualization *inside* domains; a domain can serve GPU to others |
| **Argo / OpenXT** | Hypervisor-level security, Hypervisor Mediated Exchange (HMX), Flask-XSM mandatory access control | A GPU path that respects MAC policy between VMs |
| **Qubes OS / Qubes OS Air** | Security-by-compartmentalization on modern hardware; administrative separation | GPU acceleration per compartment, especially AI workloads |
| **TrenchBoot** | Measured/attested boot (firmware → hypervisor → OS) | An attestable GPU service: the platform can prove what runs the model |
| **KVM/VFIO gaming setups** | Fast Windows VM + GPU passthrough | Solutions for users who need *multiple* VMs, or who cannot dedicate the whole GPU |
| **vLLM / SGLang / inference servers** | Continuous batching, KV-cache plugins, model serving | A way to run the server itself inside an isolated VM without losing GPU acceleration |
| **Windows driver ecosystems** | CUDA and 3D applications | Compute in Windows guests on consumer hardware |
The unifying idea: **Open-IOV should be the component these projects reach for
when they need "an NVIDIA GPU, virtualized, on our terms."**
## 5. Collaboration and IP models
We explicitly invite *both* open source and commercial collaboration:
* **Open source projects**: adopt BSD-3-Clause code directly; we will work with
you on interfaces (VFIO/mdev today, other transports on request) and accept
upstream requirements as design inputs.
* **Commercial vendors and integrators**: ship Open-IOV in products, add
proprietary value *on top*, fund specific hardware/driver-version support,
or co-develop with us under the same permissive terms. No license fees, no
contributor agreements that take rights away.
* **Hardware vendors**: nothing is required from you. Our goal is that new
hardware works through configuration and measurement, without firmware or
driver changes. If you want to help, test hardware and share traces.
* **Researchers and auditors**: the documentation in this tree — including the
protocol and firmware analysis — is written to be attackable. We want your
findings.
The license split (BSD-3-Clause for the reusable project, GPL-2.0 for kernel
modules that link GPL-only kernel interfaces) is chosen to make all of the
above legally frictionless. See [`../09-reference/legal-and-licensing.md`](../09-reference/legal-and-licensing.md).
## 6. What success looks like for this document
* A security-focused platform project can integrate Open-IOV without asking us
to change the software.
* A user can run CUDA in a Windows guest on a laptop with one GPU, legally and
without a license server.
* An edge device can run an agent VM, an inference VM and a desktop VM on one
GPU, with the model never leaving the device.
* A new GPU generation is supported by a configuration update plus measured
verification — not a rewrite.
|
