1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
|
# Performance, Correctness, and Security Evaluation
> How we measure whether Open-IOV is any good — and how auditors can verify the
> claims. Numbers without methodology are not accepted here.
## 1. The evaluation ladder
Every platform (hardware + driver + hypervisor + guest) is evaluated through
the same five-rung ladder. Each rung is a gate: you do not claim the next one
until the previous one is green and reproducible.
```
Rung 1 INITIALIZATION guest driver probes; nvidia-smi; device enumeration
Rung 2 RUNTIME START cuInit; context; allocation; UVM registration
Rung 3 EXECUTION kernels produce correct results; copies verified
Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable
Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk
```
**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes,
`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes
(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation,
allocation, copies, free and destroy complete, and the copies are verified
byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical
copy engine (default off) with engine-authored releases and a fault relay; for
GR kernel execution a forwarded guest launch now reaches the front-end but the
inline launch-descriptor install has not been observed (no grid dispatched;
context validity at dispatch is ruled out —
[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)).
Before Rung 3
is claimed, the
**reply-fidelity gate** applies: a control whose reply fields the driver
consumes is not implemented until the reply is coherent (status-only →
coherent zeroed params → coherent values for consumed fields → exact values) —
see
[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md).
New to CUDA analysis here: start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 2. Correctness
### 2.1 Test corpus
| Layer | Tests |
|---|---|
| Device | PCI config validation, BAR sizing, MSI-X semantics, power-management stubs |
| Driver init | guest `nvidia.ko` probe, `nvidia-smi` full query, `deviceQuery` |
| CUDA | `vectorAdd`, reduction, shared-memory kernels, memcpy patterns, streams/events, error injection |
| UVM | oversubscribed memory (fault path), unified memory migration, concurrency |
| Graphics (later) | GLXInfo/VulkanInfo, rendering correctness screenshots |
| Stability | start/stop cycles ×100, VM kill during work, guest reboot in flight |
### 2.2 Verification methods
1. **Golden results**: kernel outputs compared against native execution on the
same GPU, byte-for-byte, for deterministic kernels.
2. **Guest self-checks**: the guest's own validation paths are the strongest
signal (UVM asserts, RM validation, CUDA runtime self-tests). Every guest
diagnostic that fires is a test failure for us, even if the workload
continues.
3. **Differential runs**: same workload on (a) native, (b) our device — diff
observables, not just outputs: semaphore value sequences, PUT/GET progress,
interrupt counts.
4. **Negative testing**: inject malformed device behaviour in a lab profile and
require the guest driver to recover exactly as it would on hardware with a
transient error.
### 2.3 Regression gates
`./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds
the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull
requests only — there is currently no scheduled guest suite and no
stock-driver contract test. A rung regression is a release blocker;
[`STATE-CURRENT.md`](../../STATE-CURRENT.md) and this docs tree carry the
current rung.
## 3. Performance
### 3.1 Metrics
| Metric | Definition | Why it matters |
|---|---|---|
| T_init | VM boot → `cuInit` returns | user-visible startup |
| RPC latency p50/p99 | request write → reply visible in status queue | init and management overhead |
| Push overhead | guest submission → engine start (forwarded) | steady-state latency |
| Semaphore completion latency | release written → tracker observes | per-operation sync cost |
| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality |
| Kernel throughput | kernels/second for a fixed workload | launch overhead |
| Multi-tenant fairness | per-tenant progress under contention | isolation quality |
### 3.2 Method
* **Same-guest baselines**: measure native GPU performance *in the same VM with
a passthrough device* where possible; otherwise on the same hardware with the
same driver. Comparisons across hosts are noise.
* **Warm and cold**: report both; report variance; report the number of runs.
* **Instrument everything**: the emulation timestamps requests and replies
(tracepoints, `trace_printk`, or counters) so overhead is attributable to a
component, not to a mystery.
* **Publish the harness**: the benchmark scripts live in `tools/`, and a
result is not comparable unless produced by them (or an equivalent,
documented harness).
* **Runbooks over methodology**: the live physical-backend runbooks, scenario
scripts, debugfs surfaces and module parameters are collected in
[`eng-ops.md` Part II](../eng-ops.md); repeatable procedures live there, and
this document stays the measurement contract.
### 3.3 Targets
* Initialization: within 2× of native, same guest.
* Synchronization-heavy operations: within an agreed constant factor, published
per operation class.
* Bulk copies and sustained compute: as close to native as the forwarding path
allows; the forwarding design (proxy vs direct) is decided by measurement.
### 3.4 Microbenchmarks (build these first)
1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop;
time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()`
gives `total_latency_ns` and counts — divide for per-RPC host cost. Add
`trace_printk` at dispatch entry/exit for a distribution; read via
`/sys/kernel/debug/tracing/trace`.
2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest
driver's interrupt handler. Compare with the host `eventfd_signal`
timestamp.
3. **BAR0 shadow read latency** — guest loops reading a shadow register;
expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs
means a trap where there should not be one.
4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the
adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and
react within one sleep quantum under load. Tune the poll quantum only with
data.
### 3.5 A/B methodology
Every measurement runs **twice on the same machine, same boot if possible**:
* **A (bare metal)**: the workload on the physical GPU via the host's stock
driver (or the OpenRM `nvidia.ko` directly on the hardware).
* **B (vGPU)**: the identical workload in the guest on the emulated vGPU.
Same binary, same input sizes, same environment (governor, clocks, hugepages).
Report **B/A ratio %** — never absolute numbers alone. Change one variable at a
time; if B regresses, you changed the stack, the workload, or the machine, and
finding which comes before optimizing.
The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B
pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the
guest test agent. Workload suites and their pass criteria:
| Suite | What | Pass criterion |
|---|---|---|
| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A |
| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A |
| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation |
| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A |
| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) |
| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A |
Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK`
on both sides — throttling skews everything), and the command lines. Record
results with the release evidence (§5) so regressions are visible across
commits.
### 3.6 Where the overhead lives (profile in this order)
1. **RPC dispatch** — the single biggest lever; everything the guest driver
does funnels through the emulation thread.
2. **Doorbell polling cadence** — too slow costs latency, too hot burns host
CPU.
3. **eventfd/MSI-X delivery** — measure, do not assume.
4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is
MMIO-correct but slow for large reads; if the driver bulk-reads shadows,
consider cache-safe sub-regions.
5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads
are not representative until the real backing path exists.
### 3.7 Optimizing safely
* Change one thing, re-run the A/B pair, keep the numbers with the change.
* Never optimize off bare-metal-only data: the goal is B/A, not A.
* A change that makes B faster but breaks the §2.1 corpus or the rung ladder
does not count.
* Record results (date, versions, B/A ratios) where the next release can see
them.
## 4. Security and isolation
### 4.1 Threat model
| Adversary | Capability | Boundary that must hold |
|---|---|---|
| Guest VM (tenant) | full control of its own driver, MMIO, DMA | cannot touch host or other tenants' memory/state |
| Malicious guest driver | arbitrary device programming | emulation must validate everything, fail closed |
| Host user (unprivileged) | can access `/dev/*` permissions given | no privilege escalation via our interfaces |
| Adjacent tenant | shares the GPU | cannot observe timing/state/interrupts beyond policy |
| Supply chain | modified driver/firmware | measured boot (TrenchBoot) verifies the platform |
### 4.2 Enforcement
* **Memory isolation is the platform IOMMU's job, and we never weaken it.**
The device performs DMA only through the hypervisor-provided DMA interface
for its own VM. No global mappings, no "identity for convenience".
* **Per-instance state**: every mdev has its own BAR shadows, queues,
interrupt state and emulation context. No shared mutable state between
tenants in the host module.
* **Fail closed**: if a required isolation property cannot be established
(IOMMU absent, DMA mapping failure, inconsistent state), the device refuses
to start and says why in `dmesg`.
* **Interrupts**: per-instance vectors; masked interrupts are queued in the
emulated PBA, not dropped into another tenant's vector.
* **Debug surfaces**: destructive debug parameters are opt-in, documented, and
excluded from hardened builds. Reads are side-effect-free.
* **No secrets in the module**: nothing about one tenant's workload is stored
where another tenant could reach it.
### 4.3 Auditing
* The security-relevant code paths are small and enumerated in this document
tree: device registration, DMA access, interrupt injection, memory
translation, and the debug surface. Auditors should be able to enumerate
them mechanically (`grep` for the gate points).
* Each release produces a checklist: threat model reviewed, fail-closed paths
exercised by tests, debug surface verified gated, dependency licenses
unchanged.
* We explicitly invite adversarial review — including of the emulation's
response to malformed guest input.
## 5. Reporting
Each release publishes:
1. **Rung achieved** per platform, with logs.
2. **Performance table** (metrics above) with methodology and variance.
3. **Known issues**, linked to the known-unknowns document.
4. **Security checklist** status.
This is the only accepted form of a claim in this project.
|
