# Performance, Correctness, and Security Evaluation

> How we measure whether Open-IOV is any good — and how auditors can verify the
> claims. Numbers without methodology are not accepted here.

## 1. The evaluation ladder

Every platform (hardware + driver + hypervisor + guest) is evaluated through
the same five-rung ladder. Each rung is a gate: you do not claim the next one
until the previous one is green and reproducible.

```
Rung 1  INITIALIZATION    guest driver probes; nvidia-smi; device enumeration
Rung 2  RUNTIME START     cuInit; context; allocation; UVM registration
Rung 3  EXECUTION         kernels produce correct results; copies verified
Rung 4  SUSTAINED         workloads run for hours; faults/errors recoverable
Rung 5  MULTI-TENANT      N VMs, isolation + fairness + no cross-talk
```

## 2. Correctness

### 2.1 Test corpus

| Layer | Tests |
|---|---|
| Device | PCI config validation, BAR sizing, MSI-X semantics, power-management stubs |
| Driver init | guest `nvidia.ko` probe, `nvidia-smi` full query, `deviceQuery` |
| CUDA | `vectorAdd`, reduction, shared-memory kernels, memcpy patterns, streams/events, error injection |
| UVM | oversubscribed memory (fault path), unified memory migration, concurrency |
| Graphics (later) | GLXInfo/VulkanInfo, rendering correctness screenshots |
| Stability | start/stop cycles ×100, VM kill during work, guest reboot in flight |

### 2.2 Verification methods

1. **Golden results**: kernel outputs compared against native execution on the
   same GPU, byte-for-byte, for deterministic kernels.
2. **Guest self-checks**: the guest's own validation paths are the strongest
   signal (UVM asserts, RM validation, CUDA runtime self-tests). Every guest
   diagnostic that fires is a test failure for us, even if the workload
   continues.
3. **Differential runs**: same workload on (a) native, (b) our device — diff
   observables, not just outputs: semaphore value sequences, PUT/GET progress,
   interrupt counts.
4. **Negative testing**: inject malformed device behaviour in a lab profile and
   require the guest driver to recover exactly as it would on hardware with a
   transient error.

### 2.3 Regression gates

`make test` runs the host suite. The CI ladder runs the guest suite on a
scheduled cadence and on every release. A rung regression is a release blocker;
the handoff documents carry the current rung.

## 3. Performance

### 3.1 Metrics

| Metric | Definition | Why it matters |
|---|---|---|
| T_init | VM boot → `cuInit` returns | user-visible startup |
| RPC latency p50/p99 | request write → reply visible in status queue | init and management overhead |
| Push overhead | guest submission → engine start (forwarded) | steady-state latency |
| Semaphore completion latency | release written → tracker observes | per-operation sync cost |
| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality |
| Kernel throughput | kernels/second for a fixed workload | launch overhead |
| Multi-tenant fairness | per-tenant progress under contention | isolation quality |

### 3.2 Method

* **Same-guest baselines**: measure native GPU performance *in the same VM with
  a passthrough device* where possible; otherwise on the same hardware with the
  same driver. Comparisons across hosts are noise.
* **Warm and cold**: report both; report variance; report the number of runs.
* **Instrument everything**: the emulation timestamps requests and replies
  (tracepoints, `trace_printk`, or counters) so overhead is attributable to a
  component, not to a mystery.
* **Publish the harness**: the benchmark scripts live in `tools/`, and a
  result is not comparable unless produced by them (or an equivalent,
  documented harness).

### 3.3 Targets

* Initialization: within 2× of native, same guest.
* Synchronization-heavy operations: within an agreed constant factor, published
  per operation class.
* Bulk copies and sustained compute: as close to native as the forwarding path
  allows; the forwarding design (proxy vs direct) is decided by measurement.

## 4. Security and isolation

### 4.1 Threat model

| Adversary | Capability | Boundary that must hold |
|---|---|---|
| Guest VM (tenant) | full control of its own driver, MMIO, DMA | cannot touch host or other tenants' memory/state |
| Malicious guest driver | arbitrary device programming | emulation must validate everything, fail closed |
| Host user (unprivileged) | can access `/dev/*` permissions given | no privilege escalation via our interfaces |
| Adjacent tenant | shares the GPU | cannot observe timing/state/interrupts beyond policy |
| Supply chain | modified driver/firmware | measured boot (TrenchBoot) verifies the platform |

### 4.2 Enforcement

* **Memory isolation is the platform IOMMU's job, and we never weaken it.**
  The device performs DMA only through the hypervisor-provided DMA interface
  for its own VM. No global mappings, no "identity for convenience".
* **Per-instance state**: every mdev has its own BAR shadows, queues,
  interrupt state and emulation context. No shared mutable state between
  tenants in the host module.
* **Fail closed**: if a required isolation property cannot be established
  (IOMMU absent, DMA mapping failure, inconsistent state), the device refuses
  to start and says why in `dmesg`.
* **Interrupts**: per-instance vectors; masked interrupts are queued in the
  emulated PBA, not dropped into another tenant's vector.
* **Debug surfaces**: destructive debug parameters are opt-in, documented, and
  excluded from hardened builds. Reads are side-effect-free.
* **No secrets in the module**: nothing about one tenant's workload is stored
  where another tenant could reach it.

### 4.3 Auditing

* The security-relevant code paths are small and enumerated in this document
  tree: device registration, DMA access, interrupt injection, memory
  translation, and the debug surface. Auditors should be able to enumerate
  them mechanically (`grep` for the gate points).
* Each release produces a checklist: threat model reviewed, fail-closed paths
  exercised by tests, debug surface verified gated, dependency licenses
  unchanged.
* We explicitly invite adversarial review — including of the emulation's
  response to malformed guest input.

## 5. Reporting

Each release publishes:

1. **Rung achieved** per platform, with logs.
2. **Performance table** (metrics above) with methodology and variance.
3. **Known issues**, linked to the known-unknowns document.
4. **Security checklist** status.

This is the only accepted form of a claim in this project.