diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
| commit | fade84f8b5b781d29daad40699d07928dedea550 (patch) | |
| tree | e4156b3d0a6a1c794c90ef6834b949321cdfb5cb /docs/05-operations/01-performance-correctness-security.md | |
| parent | c587eac377c1c5b720142edadebaace5cb1b17c0 (diff) | |
| download | open-iov-fade84f8b5b781d29daad40699d07928dedea550.tar.gz open-iov-fade84f8b5b781d29daad40699d07928dedea550.zip | |
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful
material in the structured tree: debugging workflows and the EPT-livelock
case study (05-operations/02), performance microbenchmarks and A/B
methodology (05-operations/01), the version-upgrade runbook
(05-operations/02 section 4, with corrected host/guest framing), kernel-code
pitfalls (07-developers) and the OpenRM build/load order
(03-architecture/06 section 7). The remaining legacy files were verified
redundant and removed.
* docs/README now records where each migrated topic lives.
* Portability: the config YAML files are called out as design targets that no
code reads yet, with what adding support actually requires today.
* Getting started: the OpenRM step is marked optional (native vGPU-management
workflows only; the emulated path uses the distro driver).
* tools docs: rmtool/rmtop marked as planned, pointing at the implemented
tools/rmq subset.
* wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the
canonical userspace/cuda analysis) and index update.
* scripts/check-docs-links.py plus a CI step so relative links cannot rot.
* STATE.md references to removed documents repointed.
Diffstat (limited to 'docs/05-operations/01-performance-correctness-security.md')
| -rw-r--r-- | docs/05-operations/01-performance-correctness-security.md | 71 |
1 files changed, 71 insertions, 0 deletions
diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 0a27709..583d6c8 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -101,6 +101,77 @@ handoff documents carry the current rung. * Bulk copies and sustained compute: as close to native as the forwarding path allows; the forwarding design (proxy vs direct) is decided by measurement. +### 3.4 Microbenchmarks (build these first) + +1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop; + time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()` + gives `total_latency_ns` and counts — divide for per-RPC host cost. Add + `trace_printk` at dispatch entry/exit for a distribution; read via + `/sys/kernel/debug/tracing/trace`. +2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest + driver's interrupt handler. Compare with the host `eventfd_signal` + timestamp. +3. **BAR0 shadow read latency** — guest loops reading a shadow register; + expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs + means a trap where there should not be one. +4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the + adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and + react within one sleep quantum under load. Tune the poll quantum only with + data. + +### 3.5 A/B methodology + +Every measurement runs **twice on the same machine, same boot if possible**: + +* **A (bare metal)**: the workload on the physical GPU via the host's stock + driver (or the OpenRM `nvidia.ko` directly on the hardware). +* **B (vGPU)**: the identical workload in the guest on the emulated vGPU. + +Same binary, same input sizes, same environment (governor, clocks, hugepages). +Report **B/A ratio %** — never absolute numbers alone. Change one variable at a +time; if B regresses, you changed the stack, the workload, or the machine, and +finding which comes before optimizing. + +The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B +pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the +guest test agent. Workload suites and their pass criteria: + +| Suite | What | Pass criterion | +|---|---|---| +| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A | +| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A | +| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation | +| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A | +| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) | +| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A | + +Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK` +on both sides — throttling skews everything), and the command lines. Record +results with the release evidence (§5) so regressions are visible across +commits. + +### 3.6 Where the overhead lives (profile in this order) + +1. **RPC dispatch** — the single biggest lever; everything the guest driver + does funnels through the emulation thread. +2. **Doorbell polling cadence** — too slow costs latency, too hot burns host + CPU. +3. **eventfd/MSI-X delivery** — measure, do not assume. +4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is + MMIO-correct but slow for large reads; if the driver bulk-reads shadows, + consider cache-safe sub-regions. +5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads + are not representative until the real backing path exists. + +### 3.7 Optimizing safely + +* Change one thing, re-run the A/B pair, keep the numbers with the change. +* Never optimize off bare-metal-only data: the goal is B/A, not A. +* A change that makes B faster but breaks the §2.1 corpus or the rung ladder + does not count. +* Record results (date, versions, B/A ratios) where the next release can see + them. + ## 4. Security and isolation ### 4.1 Threat model |
