summaryrefslogtreecommitdiffstats
path: root/docs/05-operations/01-performance-correctness-security.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:07 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:07 -0400
commitfade84f8b5b781d29daad40699d07928dedea550 (patch)
treee4156b3d0a6a1c794c90ef6834b949321cdfb5cb /docs/05-operations/01-performance-correctness-security.md
parentc587eac377c1c5b720142edadebaace5cb1b17c0 (diff)
downloadopen-iov-fade84f8b5b781d29daad40699d07928dedea550.tar.gz
open-iov-fade84f8b5b781d29daad40699d07928dedea550.zip
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful material in the structured tree: debugging workflows and the EPT-livelock case study (05-operations/02), performance microbenchmarks and A/B methodology (05-operations/01), the version-upgrade runbook (05-operations/02 section 4, with corrected host/guest framing), kernel-code pitfalls (07-developers) and the OpenRM build/load order (03-architecture/06 section 7). The remaining legacy files were verified redundant and removed. * docs/README now records where each migrated topic lives. * Portability: the config YAML files are called out as design targets that no code reads yet, with what adding support actually requires today. * Getting started: the OpenRM step is marked optional (native vGPU-management workflows only; the emulated path uses the distro driver). * tools docs: rmtool/rmtop marked as planned, pointing at the implemented tools/rmq subset. * wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the canonical userspace/cuda analysis) and index update. * scripts/check-docs-links.py plus a CI step so relative links cannot rot. * STATE.md references to removed documents repointed.
Diffstat (limited to 'docs/05-operations/01-performance-correctness-security.md')
-rw-r--r--docs/05-operations/01-performance-correctness-security.md71
1 files changed, 71 insertions, 0 deletions
diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md
index 0a27709..583d6c8 100644
--- a/docs/05-operations/01-performance-correctness-security.md
+++ b/docs/05-operations/01-performance-correctness-security.md
@@ -101,6 +101,77 @@ handoff documents carry the current rung.
* Bulk copies and sustained compute: as close to native as the forwarding path
allows; the forwarding design (proxy vs direct) is decided by measurement.
+### 3.4 Microbenchmarks (build these first)
+
+1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop;
+ time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()`
+ gives `total_latency_ns` and counts — divide for per-RPC host cost. Add
+ `trace_printk` at dispatch entry/exit for a distribution; read via
+ `/sys/kernel/debug/tracing/trace`.
+2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest
+ driver's interrupt handler. Compare with the host `eventfd_signal`
+ timestamp.
+3. **BAR0 shadow read latency** — guest loops reading a shadow register;
+ expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs
+ means a trap where there should not be one.
+4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the
+ adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and
+ react within one sleep quantum under load. Tune the poll quantum only with
+ data.
+
+### 3.5 A/B methodology
+
+Every measurement runs **twice on the same machine, same boot if possible**:
+
+* **A (bare metal)**: the workload on the physical GPU via the host's stock
+ driver (or the OpenRM `nvidia.ko` directly on the hardware).
+* **B (vGPU)**: the identical workload in the guest on the emulated vGPU.
+
+Same binary, same input sizes, same environment (governor, clocks, hugepages).
+Report **B/A ratio %** — never absolute numbers alone. Change one variable at a
+time; if B regresses, you changed the stack, the workload, or the machine, and
+finding which comes before optimizing.
+
+The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B
+pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the
+guest test agent. Workload suites and their pass criteria:
+
+| Suite | What | Pass criterion |
+|---|---|---|
+| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A |
+| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A |
+| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation |
+| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A |
+| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) |
+| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A |
+
+Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK`
+on both sides — throttling skews everything), and the command lines. Record
+results with the release evidence (§5) so regressions are visible across
+commits.
+
+### 3.6 Where the overhead lives (profile in this order)
+
+1. **RPC dispatch** — the single biggest lever; everything the guest driver
+ does funnels through the emulation thread.
+2. **Doorbell polling cadence** — too slow costs latency, too hot burns host
+ CPU.
+3. **eventfd/MSI-X delivery** — measure, do not assume.
+4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is
+ MMIO-correct but slow for large reads; if the driver bulk-reads shadows,
+ consider cache-safe sub-regions.
+5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads
+ are not representative until the real backing path exists.
+
+### 3.7 Optimizing safely
+
+* Change one thing, re-run the A/B pair, keep the numbers with the change.
+* Never optimize off bare-metal-only data: the goal is B/A, not A.
+* A change that makes B faster but breaks the §2.1 corpus or the rung ladder
+ does not count.
+* Record results (date, versions, B/A ratios) where the next release can see
+ them.
+
## 4. Security and isolation
### 4.1 Threat model