# Performance, Correctness, and Security Evaluation

> How we measure whether Open-IOV is any good — and how auditors can verify the
> claims. Numbers without methodology are not accepted here.

## 1. The evaluation ladder

Every platform (hardware + driver + hypervisor + guest) is evaluated through
the same five-rung ladder. Each rung is a gate: you do not claim the next one
until the previous one is green and reproducible.

```
Rung 1  INITIALIZATION    guest driver probes; nvidia-smi; device enumeration
Rung 2  RUNTIME START     cuInit; context; allocation; UVM registration
Rung 3  EXECUTION         kernels produce correct results; copies verified
Rung 4  SUSTAINED         workloads run for hours; faults/errors recoverable
Rung 5  MULTI-TENANT      N VMs, isolation + fairness + no cross-talk
```

**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes,
`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes
(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation,
allocation, copies, free and destroy complete, and the copies are verified
byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical
copy engine (default off) with engine-authored releases and a fault relay, but
CUDA kernel execution is blocked on GR engine-context validity. Before Rung 3
is claimed, the
**reply-fidelity gate** applies: a control whose reply fields the driver
consumes is not implemented until the reply is coherent (status-only →
coherent zeroed params → coherent values for consumed fields → exact values) —
see
[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md).
New to CUDA analysis here: start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).

## 2. Correctness

### 2.1 Test corpus

| Layer | Tests |
|---|---|
| Device | PCI config validation, BAR sizing, MSI-X semantics, power-management stubs |
| Driver init | guest `nvidia.ko` probe, `nvidia-smi` full query, `deviceQuery` |
| CUDA | `vectorAdd`, reduction, shared-memory kernels, memcpy patterns, streams/events, error injection |
| UVM | oversubscribed memory (fault path), unified memory migration, concurrency |
| Graphics (later) | GLXInfo/VulkanInfo, rendering correctness screenshots |
| Stability | start/stop cycles ×100, VM kill during work, guest reboot in flight |

### 2.2 Verification methods

1. **Golden results**: kernel outputs compared against native execution on the
   same GPU, byte-for-byte, for deterministic kernels.
2. **Guest self-checks**: the guest's own validation paths are the strongest
   signal (UVM asserts, RM validation, CUDA runtime self-tests). Every guest
   diagnostic that fires is a test failure for us, even if the workload
   continues.
3. **Differential runs**: same workload on (a) native, (b) our device — diff
   observables, not just outputs: semaphore value sequences, PUT/GET progress,
   interrupt counts.
4. **Negative testing**: inject malformed device behaviour in a lab profile and
   require the guest driver to recover exactly as it would on hardware with a
   transient error.

### 2.3 Regression gates

`./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds
the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull
requests only — there is currently no scheduled guest suite and no
stock-driver contract test. A rung regression is a release blocker;
[`STATE-CURRENT.md`](../../STATE-CURRENT.md) and this docs tree carry the
current rung.

## 3. Performance

### 3.1 Metrics

| Metric | Definition | Why it matters |
|---|---|---|
| T_init | VM boot → `cuInit` returns | user-visible startup |
| RPC latency p50/p99 | request write → reply visible in status queue | init and management overhead |
| Push overhead | guest submission → engine start (forwarded) | steady-state latency |
| Semaphore completion latency | release written → tracker observes | per-operation sync cost |
| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality |
| Kernel throughput | kernels/second for a fixed workload | launch overhead |
| Multi-tenant fairness | per-tenant progress under contention | isolation quality |

### 3.2 Method

* **Same-guest baselines**: measure native GPU performance *in the same VM with
  a passthrough device* where possible; otherwise on the same hardware with the
  same driver. Comparisons across hosts are noise.
* **Warm and cold**: report both; report variance; report the number of runs.
* **Instrument everything**: the emulation timestamps requests and replies
  (tracepoints, `trace_printk`, or counters) so overhead is attributable to a
  component, not to a mystery.
* **Publish the harness**: the benchmark scripts live in `tools/`, and a
  result is not comparable unless produced by them (or an equivalent,
  documented harness).
* **Runbooks over methodology**: the live physical-backend runbooks, scenario
  scripts, debugfs surfaces and module parameters are collected in
  [`eng-ops.md` Part II](../eng-ops.md); repeatable procedures live there, and
  this document stays the measurement contract.

### 3.3 Targets

* Initialization: within 2× of native, same guest.
* Synchronization-heavy operations: within an agreed constant factor, published
  per operation class.
* Bulk copies and sustained compute: as close to native as the forwarding path
  allows; the forwarding design (proxy vs direct) is decided by measurement.

### 3.4 Microbenchmarks (build these first)

1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop;
   time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()`
   gives `total_latency_ns` and counts — divide for per-RPC host cost. Add
   `trace_printk` at dispatch entry/exit for a distribution; read via
   `/sys/kernel/debug/tracing/trace`.
2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest
   driver's interrupt handler. Compare with the host `eventfd_signal`
   timestamp.
3. **BAR0 shadow read latency** — guest loops reading a shadow register;
   expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs
   means a trap where there should not be one.
4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the
   adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and
   react within one sleep quantum under load. Tune the poll quantum only with
   data.

### 3.5 A/B methodology

Every measurement runs **twice on the same machine, same boot if possible**:

* **A (bare metal)**: the workload on the physical GPU via the host's stock
  driver (or the OpenRM `nvidia.ko` directly on the hardware).
* **B (vGPU)**: the identical workload in the guest on the emulated vGPU.

Same binary, same input sizes, same environment (governor, clocks, hugepages).
Report **B/A ratio %** — never absolute numbers alone. Change one variable at a
time; if B regresses, you changed the stack, the workload, or the machine, and
finding which comes before optimizing.

The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B
pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the
guest test agent. Workload suites and their pass criteria:

| Suite | What | Pass criterion |
|---|---|---|
| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A |
| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A |
| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation |
| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A |
| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) |
| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A |

Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK`
on both sides — throttling skews everything), and the command lines. Record
results with the release evidence (§5) so regressions are visible across
commits.

### 3.6 Where the overhead lives (profile in this order)

1. **RPC dispatch** — the single biggest lever; everything the guest driver
   does funnels through the emulation thread.
2. **Doorbell polling cadence** — too slow costs latency, too hot burns host
   CPU.
3. **eventfd/MSI-X delivery** — measure, do not assume.
4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is
   MMIO-correct but slow for large reads; if the driver bulk-reads shadows,
   consider cache-safe sub-regions.
5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads
   are not representative until the real backing path exists.

### 3.7 Optimizing safely

* Change one thing, re-run the A/B pair, keep the numbers with the change.
* Never optimize off bare-metal-only data: the goal is B/A, not A.
* A change that makes B faster but breaks the §2.1 corpus or the rung ladder
  does not count.
* Record results (date, versions, B/A ratios) where the next release can see
  them.

## 4. Security and isolation

### 4.1 Threat model

| Adversary | Capability | Boundary that must hold |
|---|---|---|
| Guest VM (tenant) | full control of its own driver, MMIO, DMA | cannot touch host or other tenants' memory/state |
| Malicious guest driver | arbitrary device programming | emulation must validate everything, fail closed |
| Host user (unprivileged) | can access `/dev/*` permissions given | no privilege escalation via our interfaces |
| Adjacent tenant | shares the GPU | cannot observe timing/state/interrupts beyond policy |
| Supply chain | modified driver/firmware | measured boot (TrenchBoot) verifies the platform |

### 4.2 Enforcement

* **Memory isolation is the platform IOMMU's job, and we never weaken it.**
  The device performs DMA only through the hypervisor-provided DMA interface
  for its own VM. No global mappings, no "identity for convenience".
* **Per-instance state**: every mdev has its own BAR shadows, queues,
  interrupt state and emulation context. No shared mutable state between
  tenants in the host module.
* **Fail closed**: if a required isolation property cannot be established
  (IOMMU absent, DMA mapping failure, inconsistent state), the device refuses
  to start and says why in `dmesg`.
* **Interrupts**: per-instance vectors; masked interrupts are queued in the
  emulated PBA, not dropped into another tenant's vector.
* **Debug surfaces**: destructive debug parameters are opt-in, documented, and
  excluded from hardened builds. Reads are side-effect-free.
* **No secrets in the module**: nothing about one tenant's workload is stored
  where another tenant could reach it.

### 4.3 Auditing

* The security-relevant code paths are small and enumerated in this document
  tree: device registration, DMA access, interrupt injection, memory
  translation, and the debug surface. Auditors should be able to enumerate
  them mechanically (`grep` for the gate points).
* Each release produces a checklist: threat model reviewed, fail-closed paths
  exercised by tests, debug surface verified gated, dependency licenses
  unchanged.
* We explicitly invite adversarial review — including of the emulation's
  response to malformed guest input.

## 5. Reporting

Each release publishes:

1. **Rung achieved** per platform, with logs.
2. **Performance table** (metrics above) with methodology and variance.
3. **Known issues**, linked to the known-unknowns document.
4. **Security checklist** status.

This is the only accepted form of a claim in this project.