diff options
Diffstat (limited to 'docs/05-operations/02-debugging-and-maintenance-workflows.md')
| -rw-r--r-- | docs/05-operations/02-debugging-and-maintenance-workflows.md | 206 |
1 files changed, 196 insertions, 10 deletions
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index 331597a..52cee94 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -148,19 +148,166 @@ coherent. Start at and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). +### 3.7 QEMU gdbstub hang triage + +**The golden rule of failure triage: when the guest agent dies mid-test, the +guest is wedged in kernel mode. Stop retrying `exec` and go straight to the +gdbstub and host dmesg.** Triage order: + +1. `sudo dmesg -T | tail -50` — KVM warnings (e.g. `make_spte`, EPT), our + driver's last messages. +2. `tail -100 /tmp/vgpu-guest-serial.log` — the guest kernel's last words + (works even when guest userspace is dead). +3. gdbstub vCPU sampling (below) — *where* is each vCPU stuck? + +Attach without symbols for triage: + +```bash +gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ + -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ + -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' +``` + +Reading the result: + +| Observation | Meaning | +|-------------|---------| +| vCPU RIP in kernel text, disassembly shows `hlt` | CPU idle — fine | +| vCPU RIP in module range, **same RIP+RSP across samples** | hard spin in a kernel module (e.g. an `nvidia.ko` polling loop) — this is the hung driver | +| RIPs moving between samples | guest alive and progressing | + +Sample the stuck vCPU three times a few seconds apart. Identical `$rip` **and** +`$rsp` every time means the *same instruction* faults/retries (an +EPT-violation livelock) rather than a loop making progress. With guest symbols +(`vmlinux` plus the module `.ko` built with debug info) use `add-symbol-file` +to recover real function names. + +### 3.8 QMP NMI (soft alternative) + +```bash +(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ + echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ + | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock +``` + +Watch the serial log afterwards for backtraces. A truly hard-wedged vCPU +(spinning on a faulting MMIO instruction) may print nothing — use the gdbstub +instead. + ## 4. Workflows for new inputs -### 4.1 New firmware version +### 4.1 New firmware / driver version (upgrade runbook) + +**Version discipline first.** The anchor that must stay coherent is the +**guest driver ↔ emulation target** (the GSP firmware generation whose +protocol the host emulates): a guest driver whose protocol differs from what +the emulation implements produces failures indistinguishable from emulation +bugs. The host's physical-GPU driver is *not* part of that lock-step — the +reference host uses the distribution's driver, and the OpenRM host modules are +optional (native vGPU-management workflows only; see +[`../03-architecture/06-host-module-and-hybrid-model.md` §7](../03-architecture/06-host-module-and-hybrid-model.md)). +Today's emulation target is **570.144**; a version bump moves the guest driver +and the emulation, not the host distro driver. + +**1. Acquire the firmware and matching source reference.** + +* Extract the GSP firmware from the target driver runfile (or fetch the + versioned blob from `linux-firmware.git`), then extract the `.fwimage` + section (host boot image) and record its MD5: -1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study - material; firmware blobs are gitignored and not distributed in the - repository). -2. Compute signatures (CRC32/SHA256) with the firmware analyzer. -3. Add a firmware profile entry (addresses/signatures) so runtime analysis can - identify it. -4. Run the driver against a guest using that firmware's matching driver; diff - the RPC/control inventory against the baseline. -5. Add/adjust handlers only where the diff requires it; re-run the ladder. + ```bash + objcopy --only-section=.fwimage <extracted-blob> gsp-<VER>.fwimage.bin + md5sum gsp-<VER>.fwimage.bin + ``` + +* Note the matching NVIDIA OpenRM tag (e.g. `570.144`) — it is the + version-exact reference for headers and wire structures. + +**2. Lay down versioned directories (never overwrite a working version).** + +``` +firmware/<SERIES>-GSP/<VER>/ + gsp-<VER>.fwimage.bin + decompiled/ # Ghidra output lands here +config/versions/<MAJOR>.yaml # design target; see §6 of the portability doc +``` + +Copy the analysis scripts from the previous version directory — they are +script-relative. + +**3. Reverse-engineer the new firmware (Ghidra headless).** + +```bash +cd tools/ghidra +python3 analyze_fwimage.py \ + ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \ + /tmp/ghidra_project_<VER> "gsp_<VER>" +cd ../../firmware/<SERIES>-GSP/<VER> +python3 analyze_firmware.py gsp-<VER>.fwimage.bin +``` + +Diff against the 570.144 record (`wiki/Address-Reference.md`). What to extract: + +| Artifact | Why it matters | +|---|---| +| Primary RPC dispatcher + jump table | RPC types/subcommands may shift | +| BAR trap / WPR handlers | BAR emulation behaviour | +| Entry/boot sequence | the boot RPC ordering the guest driver expects | +| Migration tables | migration protocol (if enabled) | +| RPC message layouts | struct changes break the emulator silently | +| Version string/signature | runtime firmware matching | + +Sanity checks: still RV64IMAC? still the same twelve top-level RPC types? new +subcommands? confidential-computing/TEE additions? Never carry 570.144 +addresses into a new configuration — re-derive per firmware. + +**4. Update the version surface (data first).** + +* Add a hardware/version profile entry (addresses, signatures, capability + toggles) so runtime analysis can identify the image; remember the YAML + files under `config/` are design targets today (see + [`../06-portability/hardware-software-matrix.md` §6](../06-portability/hardware-software-matrix.md)). +* Add the new entry to the runtime signature database in + `kernel/firmware_analyzer.c` (`known_signatures[]`: version string, hashes, + discovered addresses). This is what lets one `open-iov-vfio.ko` identify + multiple firmware versions safely. + +**5. Validate (same checklist every version).** + +1. Host: module load/unload ×3 and mdev create/destroy + (`./scripts/stack_manager.sh test`). +2. Guest: boot → `nvidia.ko` probe → `nvidia-smi` → teardown regression + (no leaked `gsp_emu` thread, mdev removable again). +3. Inventory diff: capture the full init trace and diff function/control IDs + against the previous version; for each difference decode against the + exact-version reference headers, then add a version-keyed handler or a + capability-table entry only where the diff requires it. +4. Then the evaluation ladder (rungs 1–2, then the CUDA corpus) and only then + performance comparisons + ([`01-performance-correctness-security.md`](01-performance-correctness-security.md)). + +**6. Rollback.** Keep the previous version's directory and configuration +untouched. Rollback = check out the old anchor's commit/tag and reinstall the +old guest driver. Because versions live in parallel directories, a forward +attempt never destroys the working setup. + +**Common pitfalls (learned the hard way).** + +* **Runfile version drift**: distribution metapackages can resolve to a newer + major than requested. Check the candidate version before installing; prefer + the exact-version runfile for the guest. +* **Firmware filename mismatch**: a wrong-version firmware file is a probe + failure that looks like an emulation bug. +* **Address drift per build**: kernel text addresses differ per firmware + build; re-derive Ghidra addresses, never reuse them. +* **Symbol renames across OpenRM tags**: renamed exports break the optional + OpenRM build with `Unknown symbol` — check the `kernel-open` diff when that + path is in use. + +**Multi-version testing matrix (target state).** For each supported anchor +version × each hardware family, run the step-5 bring-up sequence. When adding +a version, extend the matrix rather than replacing the old column — +regressions across versions are the cheapest bugs to catch. ### 4.2 New guest driver version @@ -188,6 +335,23 @@ and ## 5. Crash and hang recovery +### 5.1 Failure taxonomy (symptom → likely cause) + +| Symptom | Likely cause | First check | +|---------|--------------|-------------| +| `insmod open-iov-vfio.ko: Unknown symbol` | missing prerequisite module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | +| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | +| Guest: no PCI device | mdev not attached | QEMU command line for `sysfsdev=` | +| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | +| Guest: probe hangs, no KVM warning | doorbell/RPC never consumed | host dmesg for the emulation thread and doorbell writes | +| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | +| Guest agent dead, QEMU alive | guest kernel wedge | gdbstub vCPU sampling (§3.7) | +| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | +| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | +| `pr_warn_ratelimited(... not supported in emulation mode)` | guest used a BAR_TRAP/MIGRATE RPC | expected fail-fast; wire a real handler if hot | + +### 5.2 Recovery actions + | Situation | Action | |---|---| | Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) | @@ -195,6 +359,28 @@ and | Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first | | Kernel oops in the host module | capture the full oops + `dmesg` before reloading | +### 5.3 Case study: the EPT livelock + +Symptoms: guest `modprobe nvidia` hangs; the guest agent dies; QEMU stays +alive. + +1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` + with `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. +2. **gdbstub**: one vCPU frozen at a single address in module text, identical + RIP/RSP across samples; the other vCPUs in `hlt`. The faulting instruction + was `mov 0x4(%rax),%r13d` — a memory read that can never complete. +3. **Root cause**: the VFIO `mmap` handler used `virt_to_phys()` on the + `vzalloc`'d BAR0 shadow → garbage PFN in the EPT → every guest BAR0 read + faulted → KVM retried forever. +4. **Fix**: per-page `vmalloc_to_pfn()` mapping. + +**Lesson: a KVM `make_spte` reserved-bits warning plus one frozen vCPU means a +bad PFN from your VFIO mmap handler.** Always question `virt_to_phys()` on +anything that is not lowmem; the module rule is recorded in +[`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md) +and the shadow mapping is described in +[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md). + ## 6. Release engineering * Versions are tagged; each tag carries the rung/matrix table and the |