From fade84f8b5b781d29daad40699d07928dedea550 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Wed, 23 Sep 2026 13:48:07 -0400 Subject: docs: merge the legacy flat documents, add the CUDA deep-dive and link checks * Merge and remove the pre-reboot flat documents, preserving their still-useful material in the structured tree: debugging workflows and the EPT-livelock case study (05-operations/02), performance microbenchmarks and A/B methodology (05-operations/01), the version-upgrade runbook (05-operations/02 section 4, with corrected host/guest framing), kernel-code pitfalls (07-developers) and the OpenRM build/load order (03-architecture/06 section 7). The remaining legacy files were verified redundant and removed. * docs/README now records where each migrated topic lives. * Portability: the config YAML files are called out as design targets that no code reads yet, with what adding support actually requires today. * Getting started: the OpenRM step is marked optional (native vGPU-management workflows only; the emulated path uses the distro driver). * tools docs: rmtool/rmtop marked as planned, pointing at the implemented tools/rmq subset. * wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the canonical userspace/cuda analysis) and index update. * scripts/check-docs-links.py plus a CI step so relative links cannot rot. * STATE.md references to removed documents repointed. --- .../01-performance-correctness-security.md | 71 +++++++ .../02-debugging-and-maintenance-workflows.md | 206 ++++++++++++++++++++- 2 files changed, 267 insertions(+), 10 deletions(-) (limited to 'docs/05-operations') diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 0a27709..583d6c8 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -101,6 +101,77 @@ handoff documents carry the current rung. * Bulk copies and sustained compute: as close to native as the forwarding path allows; the forwarding design (proxy vs direct) is decided by measurement. +### 3.4 Microbenchmarks (build these first) + +1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop; + time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()` + gives `total_latency_ns` and counts — divide for per-RPC host cost. Add + `trace_printk` at dispatch entry/exit for a distribution; read via + `/sys/kernel/debug/tracing/trace`. +2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest + driver's interrupt handler. Compare with the host `eventfd_signal` + timestamp. +3. **BAR0 shadow read latency** — guest loops reading a shadow register; + expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs + means a trap where there should not be one. +4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the + adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and + react within one sleep quantum under load. Tune the poll quantum only with + data. + +### 3.5 A/B methodology + +Every measurement runs **twice on the same machine, same boot if possible**: + +* **A (bare metal)**: the workload on the physical GPU via the host's stock + driver (or the OpenRM `nvidia.ko` directly on the hardware). +* **B (vGPU)**: the identical workload in the guest on the emulated vGPU. + +Same binary, same input sizes, same environment (governor, clocks, hugepages). +Report **B/A ratio %** — never absolute numbers alone. Change one variable at a +time; if B regresses, you changed the stack, the workload, or the machine, and +finding which comes before optimizing. + +The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B +pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the +guest test agent. Workload suites and their pass criteria: + +| Suite | What | Pass criterion | +|---|---|---| +| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A | +| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A | +| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation | +| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A | +| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) | +| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A | + +Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK` +on both sides — throttling skews everything), and the command lines. Record +results with the release evidence (§5) so regressions are visible across +commits. + +### 3.6 Where the overhead lives (profile in this order) + +1. **RPC dispatch** — the single biggest lever; everything the guest driver + does funnels through the emulation thread. +2. **Doorbell polling cadence** — too slow costs latency, too hot burns host + CPU. +3. **eventfd/MSI-X delivery** — measure, do not assume. +4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is + MMIO-correct but slow for large reads; if the driver bulk-reads shadows, + consider cache-safe sub-regions. +5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads + are not representative until the real backing path exists. + +### 3.7 Optimizing safely + +* Change one thing, re-run the A/B pair, keep the numbers with the change. +* Never optimize off bare-metal-only data: the goal is B/A, not A. +* A change that makes B faster but breaks the §2.1 corpus or the rung ladder + does not count. +* Record results (date, versions, B/A ratios) where the next release can see + them. + ## 4. Security and isolation ### 4.1 Threat model diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index 331597a..52cee94 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -148,19 +148,166 @@ coherent. Start at and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). +### 3.7 QEMU gdbstub hang triage + +**The golden rule of failure triage: when the guest agent dies mid-test, the +guest is wedged in kernel mode. Stop retrying `exec` and go straight to the +gdbstub and host dmesg.** Triage order: + +1. `sudo dmesg -T | tail -50` — KVM warnings (e.g. `make_spte`, EPT), our + driver's last messages. +2. `tail -100 /tmp/vgpu-guest-serial.log` — the guest kernel's last words + (works even when guest userspace is dead). +3. gdbstub vCPU sampling (below) — *where* is each vCPU stuck? + +Attach without symbols for triage: + +```bash +gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ + -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ + -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' +``` + +Reading the result: + +| Observation | Meaning | +|-------------|---------| +| vCPU RIP in kernel text, disassembly shows `hlt` | CPU idle — fine | +| vCPU RIP in module range, **same RIP+RSP across samples** | hard spin in a kernel module (e.g. an `nvidia.ko` polling loop) — this is the hung driver | +| RIPs moving between samples | guest alive and progressing | + +Sample the stuck vCPU three times a few seconds apart. Identical `$rip` **and** +`$rsp` every time means the *same instruction* faults/retries (an +EPT-violation livelock) rather than a loop making progress. With guest symbols +(`vmlinux` plus the module `.ko` built with debug info) use `add-symbol-file` +to recover real function names. + +### 3.8 QMP NMI (soft alternative) + +```bash +(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ + echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ + | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock +``` + +Watch the serial log afterwards for backtraces. A truly hard-wedged vCPU +(spinning on a faulting MMIO instruction) may print nothing — use the gdbstub +instead. + ## 4. Workflows for new inputs -### 4.1 New firmware version +### 4.1 New firmware / driver version (upgrade runbook) + +**Version discipline first.** The anchor that must stay coherent is the +**guest driver ↔ emulation target** (the GSP firmware generation whose +protocol the host emulates): a guest driver whose protocol differs from what +the emulation implements produces failures indistinguishable from emulation +bugs. The host's physical-GPU driver is *not* part of that lock-step — the +reference host uses the distribution's driver, and the OpenRM host modules are +optional (native vGPU-management workflows only; see +[`../03-architecture/06-host-module-and-hybrid-model.md` §7](../03-architecture/06-host-module-and-hybrid-model.md)). +Today's emulation target is **570.144**; a version bump moves the guest driver +and the emulation, not the host distro driver. + +**1. Acquire the firmware and matching source reference.** + +* Extract the GSP firmware from the target driver runfile (or fetch the + versioned blob from `linux-firmware.git`), then extract the `.fwimage` + section (host boot image) and record its MD5: -1. Add the image under `firmware/-GSP//` (local study - material; firmware blobs are gitignored and not distributed in the - repository). -2. Compute signatures (CRC32/SHA256) with the firmware analyzer. -3. Add a firmware profile entry (addresses/signatures) so runtime analysis can - identify it. -4. Run the driver against a guest using that firmware's matching driver; diff - the RPC/control inventory against the baseline. -5. Add/adjust handlers only where the diff requires it; re-run the ladder. + ```bash + objcopy --only-section=.fwimage gsp-.fwimage.bin + md5sum gsp-.fwimage.bin + ``` + +* Note the matching NVIDIA OpenRM tag (e.g. `570.144`) — it is the + version-exact reference for headers and wire structures. + +**2. Lay down versioned directories (never overwrite a working version).** + +``` +firmware/-GSP// + gsp-.fwimage.bin + decompiled/ # Ghidra output lands here +config/versions/.yaml # design target; see §6 of the portability doc +``` + +Copy the analysis scripts from the previous version directory — they are +script-relative. + +**3. Reverse-engineer the new firmware (Ghidra headless).** + +```bash +cd tools/ghidra +python3 analyze_fwimage.py \ + ../../firmware/-GSP//gsp-.fwimage.bin \ + /tmp/ghidra_project_ "gsp_" +cd ../../firmware/-GSP/ +python3 analyze_firmware.py gsp-.fwimage.bin +``` + +Diff against the 570.144 record (`wiki/Address-Reference.md`). What to extract: + +| Artifact | Why it matters | +|---|---| +| Primary RPC dispatcher + jump table | RPC types/subcommands may shift | +| BAR trap / WPR handlers | BAR emulation behaviour | +| Entry/boot sequence | the boot RPC ordering the guest driver expects | +| Migration tables | migration protocol (if enabled) | +| RPC message layouts | struct changes break the emulator silently | +| Version string/signature | runtime firmware matching | + +Sanity checks: still RV64IMAC? still the same twelve top-level RPC types? new +subcommands? confidential-computing/TEE additions? Never carry 570.144 +addresses into a new configuration — re-derive per firmware. + +**4. Update the version surface (data first).** + +* Add a hardware/version profile entry (addresses, signatures, capability + toggles) so runtime analysis can identify the image; remember the YAML + files under `config/` are design targets today (see + [`../06-portability/hardware-software-matrix.md` §6](../06-portability/hardware-software-matrix.md)). +* Add the new entry to the runtime signature database in + `kernel/firmware_analyzer.c` (`known_signatures[]`: version string, hashes, + discovered addresses). This is what lets one `open-iov-vfio.ko` identify + multiple firmware versions safely. + +**5. Validate (same checklist every version).** + +1. Host: module load/unload ×3 and mdev create/destroy + (`./scripts/stack_manager.sh test`). +2. Guest: boot → `nvidia.ko` probe → `nvidia-smi` → teardown regression + (no leaked `gsp_emu` thread, mdev removable again). +3. Inventory diff: capture the full init trace and diff function/control IDs + against the previous version; for each difference decode against the + exact-version reference headers, then add a version-keyed handler or a + capability-table entry only where the diff requires it. +4. Then the evaluation ladder (rungs 1–2, then the CUDA corpus) and only then + performance comparisons + ([`01-performance-correctness-security.md`](01-performance-correctness-security.md)). + +**6. Rollback.** Keep the previous version's directory and configuration +untouched. Rollback = check out the old anchor's commit/tag and reinstall the +old guest driver. Because versions live in parallel directories, a forward +attempt never destroys the working setup. + +**Common pitfalls (learned the hard way).** + +* **Runfile version drift**: distribution metapackages can resolve to a newer + major than requested. Check the candidate version before installing; prefer + the exact-version runfile for the guest. +* **Firmware filename mismatch**: a wrong-version firmware file is a probe + failure that looks like an emulation bug. +* **Address drift per build**: kernel text addresses differ per firmware + build; re-derive Ghidra addresses, never reuse them. +* **Symbol renames across OpenRM tags**: renamed exports break the optional + OpenRM build with `Unknown symbol` — check the `kernel-open` diff when that + path is in use. + +**Multi-version testing matrix (target state).** For each supported anchor +version × each hardware family, run the step-5 bring-up sequence. When adding +a version, extend the matrix rather than replacing the old column — +regressions across versions are the cheapest bugs to catch. ### 4.2 New guest driver version @@ -188,6 +335,23 @@ and ## 5. Crash and hang recovery +### 5.1 Failure taxonomy (symptom → likely cause) + +| Symptom | Likely cause | First check | +|---------|--------------|-------------| +| `insmod open-iov-vfio.ko: Unknown symbol` | missing prerequisite module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | +| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | +| Guest: no PCI device | mdev not attached | QEMU command line for `sysfsdev=` | +| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | +| Guest: probe hangs, no KVM warning | doorbell/RPC never consumed | host dmesg for the emulation thread and doorbell writes | +| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | +| Guest agent dead, QEMU alive | guest kernel wedge | gdbstub vCPU sampling (§3.7) | +| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | +| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | +| `pr_warn_ratelimited(... not supported in emulation mode)` | guest used a BAR_TRAP/MIGRATE RPC | expected fail-fast; wire a real handler if hot | + +### 5.2 Recovery actions + | Situation | Action | |---|---| | Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) | @@ -195,6 +359,28 @@ and | Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first | | Kernel oops in the host module | capture the full oops + `dmesg` before reloading | +### 5.3 Case study: the EPT livelock + +Symptoms: guest `modprobe nvidia` hangs; the guest agent dies; QEMU stays +alive. + +1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` + with `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. +2. **gdbstub**: one vCPU frozen at a single address in module text, identical + RIP/RSP across samples; the other vCPUs in `hlt`. The faulting instruction + was `mov 0x4(%rax),%r13d` — a memory read that can never complete. +3. **Root cause**: the VFIO `mmap` handler used `virt_to_phys()` on the + `vzalloc`'d BAR0 shadow → garbage PFN in the EPT → every guest BAR0 read + faulted → KVM retried forever. +4. **Fix**: per-page `vmalloc_to_pfn()` mapping. + +**Lesson: a KVM `make_spte` reserved-bits warning plus one frozen vCPU means a +bad PFN from your VFIO mmap handler.** Always question `virt_to_phys()` on +anything that is not lowmem; the module rule is recorded in +[`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md) +and the shadow mapping is described in +[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md). + ## 6. Release engineering * Versions are tagged; each tag carries the rung/matrix table and the -- cgit v1.2.3