1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
|
# Hardware and Software Matrix
> What Open-IOV runs on, what it targets, and exactly how support for a new
> hardware or software combination is added. Support here means *tested and
> documented*, not "should probably work".
## 1. Support ladder
| Level | Meaning |
|---|---|
| **Reference** | continuously tested; bugs block releases |
| **Supported** | tested per release; documented workarounds if any |
| **Experimental** | known to boot/test partially; may need configuration |
| **Target** | architecture designed for it; bring-up in progress |
| **Out of scope** | not planned |
## 2. Host platforms
| Platform | Status | Notes |
|---|---|---|
| x86_64 + NVIDIA Turing (TU104) | **Reference** | the current development hardware (RTX 2080-class) |
| x86_64 + Ampere / Ada consumer | **Target** | profile work; expected to be configuration + measurement |
| x86_64 + Hopper / datacenter | **Target** | requires MIG-aware policy for multi-tenancy |
| aarch64 + Grace-Hopper (GH200) | **Target** | host-architecture work in the memory model |
| aarch64 + Grace-Blackwell | **Target** | as above, newer firmware/driver family |
| GB10 (DGX Spark, Surface-class RTX laptops) | **Target** | same architecture; firmware/ROM quirks to be measured |
| Other NVIDIA generations | **Experimental** | bring-up path documented in §5 |
The defining property we are building toward: **each new host platform should
require configuration + verification, not new code.** That is the design
direction, not the current state — the profile files are not read by any code
yet and selection is compile-time (see §6).
## 3. Guest operating systems
| Guest | Status |
|---|---|
| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe and `nvidia-smi` work, but there is no continuous guest testing and context creation is currently blocked — see §6.1) |
| Linux aarch64 | **Target** |
| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** |
| Other Unix (FreeBSD, ...) | Out of scope until demand |
**Guest rule:** the guest never contains Open-IOV code. No guest agent, no
driver patch, no kernel parameters beyond what the stock driver needs.
## 4. Hypervisors and transports
| Hypervisor | Status | Transport |
|---|---|---|
| KVM + QEMU (VFIO mdev) | **Reference** | VFIO mdev + eventfd IRQ + DMA interface |
| libvirt (mdev attach) | **Supported** | same, via libvirt XML |
| Zen / KVM on other device models | **Experimental** | as above |
| Xen (incl. Argo/OpenXT) | **Target** | Xen device model + event channels |
| Qubes OS / Qubes OS Air | **Target** | compartmentalized VMs; GPU per compartment |
| Cloud hypervisor / crosvm | **Experimental** | VFIO-based; needs testing |
The transport boundary (how the device registers with the hypervisor, how
guest memory is accessed, how interrupts are injected) is deliberately narrow
so new transports are a small, isolated implementation.
## 5. Adding a new combination
### 5.1 New GPU / hardware profile
1. Create a hardware profile: PCI IDs, BAR layout and sizes, memory geometry
(FB size, BAR windows), engine inventory, interrupt topology, and the
register values the guest driver validates during probe.
2. Run the evaluation ladder (see
[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)).
3. Record the differences; prefer configuration over code.
4. Publish the profile, the evidence, and any workarounds.
### 5.2 New firmware version
See [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §4.1.
Firmware images are versioned under `firmware/<SERIES>-GSP/<VERSION>/` with
signatures and analysis artifacts.
### 5.3 New driver version
See the same document §4.2. The expected work per version is: run the trace,
diff the inventory, decode the differences against that version's reference
tree, add configuration/handlers, re-run the ladder.
## 6. Version matrix (maintained in the repository)
The version/hardware configuration lives in `config/`:
```
config/
hardware/<arch>.yaml # PCI IDs, BARs, memory geometry, engine lists
versions/<driver>.yaml # RPC/control inventories, capability tables
```
> **The `config/hardware/*.yaml` and `config/versions/*.yaml` files are
> design targets, not live configuration.** No code currently reads them:
> hardware/version selection is compile-time and the module is built for the
> TU104 profile. Treating the YAML surface as authoritative — or claiming
> "configuration-driven portability" — is aspirational until the profiles are
> wired into the module's load path. There is no `profiles/` directory today;
> `config/` is the only profile surface.
**What "adding support" requires today** (TU104/570.144 is the reference):
1. New GPU: edit the compile-time hardware constants/table for the target
(PCI IDs, BAR layout and sizes, memory geometry, engine inventory,
interrupt topology, probe-time register values) and rebuild the module;
the `config/hardware/*.yaml` file is documentation until the plumbing
exists.
2. New driver/firmware family: capture the init trace, diff the
RPC/control inventory, add version-keyed handlers, and register the
firmware signature for runtime identification; the `config/versions/*.yaml`
file is documentation until the plumbing exists.
3. Re-run the evaluation ladder and record the combination as
"hardware X + driver family Y + firmware Z, tested at rung R on date D."
Releases publish the table.
A profile is the *unit of support* — what is missing is the code that reads
the profile files, not the profile schema.
### 6.1 Userspace driver facts
* **Guest CUDA driver** (`libcuda`) under test: **570.144** (md5
`ed5b2fc0044df6052df1d5113754c499`).
* **Host oracle**: `libcuda` **570.211.01** with CUDA toolkit **12.0**, used
for A/B comparisons against the guest.
* **Versioned surface**: the RM ioctl / class / control interface the userspace
driver issues — not a stable public API — see
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md).
### 6.2 Engine-list portability
The engine inventory, and how it is filtered by the class list (classDB), are
per-architecture and per-driver data. Today's gap is concrete: the emulated
list lacks the async copy engine (`COPY14`), which blocks context creation
(`cuCtxCreate` returns `rc=999`). Engine inventory belongs in the hardware
profile, validated against the host oracle per combination. For the
re-analysis workflow see
[`../../userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 7. Architecture properties that enable portability
| Property | Why it matters |
|---|---|
| No architecture-specific assumptions in the memory model | x86_64 and aarch64 hosts share the same code |
| RPC/handler layer keyed by version | new driver families are additive |
| Hardware constants in profiles | new GPUs are data |
| Transport behind a narrow interface | KVM today, Xen/Argo/Qubes tomorrow |
| Guest-independent design | Windows and Linux guests share the same host code |
| Firmware analysis toolchain is version-generic | new firmware is analyzed with the same scripts |
These are design properties, not a description of the current build: today the
module is compiled for one profile (TU104) and the `config/` YAML is not on the
load path (see §6).
## 8. Known portability risks
| Risk | Mitigation |
|---|---|
| Page-size/cache-attribute differences on aarch64 | keep memory model abstract; test on Grace early |
| Windows driver ROM/VBIOS expectations | explicit ROM/VBIOS provisioning in the device profile; measured bring-up |
| Xen interrupt/memory interfaces differ | narrow transport interface + Xen backend plan |
| Vendor ROMs and laptop firmware variants (GB10/Surface) | profile system supports per-device overrides |
| Driver-family drift (580/610+) | version-keyed handlers; matrix-driven testing |
|
