diff options
Diffstat (limited to 'docs/DEBUGGING.md')
| -rw-r--r-- | docs/DEBUGGING.md | 196 |
1 files changed, 0 insertions, 196 deletions
diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md deleted file mode 100644 index e2d7fe0..0000000 --- a/docs/DEBUGGING.md +++ /dev/null @@ -1,196 +0,0 @@ -# Open-IOV Debugging Guide - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Debugging -> replacement: -> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md); -> the §3–§4 case studies here remain useful. - -> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest -> driver failures, and validating a fully working setup. Built from real -> debugging sessions (including the 2026-09-21 EPT-livelock hunt). - ---- - -## 1. Debug surfaces (know them all) - -| Surface | Where | What it shows | -|---------|-------|---------------| -| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns | -| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead | -| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** | -| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status | -| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** | -| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer | -| GDB on host kernel | (optional) | host-side driver state | - -The VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain: - -```yaml -serialLog: "/tmp/vgpu-guest-serial.log" -gdbPort: 1234 -gdbFreeze: false # true = halt at reset vector (boot debugging) -``` - ---- - -## 2. The golden rule of failure triage - -**When the guest agent dies mid-test, the guest is wedged in kernel mode. -Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.** - -Triage order: -1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages -2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words -3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck? - ---- - -## 3. QEMU GDB stub workflow - -Attach (no symbols needed for triage): - -```bash -gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ - -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ - -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' -``` - -Reading the result: - -| Observation | Meaning | -|-------------|---------| -| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine | -| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver | -| RIPs moving between samples | guest alive and progressing | - -For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical -`$rip` **and** `$rsp` every time = the *same instruction* faults/retries -(EPT-violation livelock) rather than a loop making progress. - -With guest symbols (`vmlinux` + module `.ko` with debug info), add -`add-symbol-file` and get real function names. - -### QMP NMI (soft alternative) - -```bash -(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ - echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ - | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock -``` - -Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU -(spinning with a faulting MMIO instruction) may not print anything — use the -GDB stub instead. - ---- - -## 4. Case study: the 2026-09-21 EPT livelock - -Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive. - -1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with - `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. -2. **GDB stub**: vCPU2 frozen at one address in module text - (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in - `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that - can never complete. -3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0 - shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM - retried forever. -4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see - `docs/GSP_EMULATION_ARCHITECTURE.md` §2. - -Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from -your VFIO mmap handler.** Always question `virt_to_phys()` on anything that -isn't lowmem. - ---- - -## 5. What a *successful* driver bring-up looks like (validation checklist) - -After `stack_manager.sh load-stack` on the host: - -- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers -- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1** -- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg - -After `vgpu_vm.py start vgpu-linux`: - -- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>` -- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started` -- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored) -- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped) -- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root` - -When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`): - -- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first) -- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment) -- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam -- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found") -- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done -- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144** - -Teardown regression: - -- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1** -- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread) -- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone - -Kernel unit test: - -- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED` -- [ ] `sudo rmmod gsp_emulation_test` → clean - ---- - -## 6. Failure taxonomy (symptom → likely cause) - -| Symptom | Likely cause | First check | -|---------|--------------|-------------| -| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | -| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | -| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` | -| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | -| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes | -| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | -| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) | -| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | -| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | -| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot | - ---- - -## 7. Host-guest version matrix discipline - -Probe failures caused by **version mismatch** look identical to emulation bugs. -Before deep debugging, verify the matrix: - -| Component | Must be | Check | -|-----------|---------|-------| -| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) | -| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) | -| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` | -| Emulation target | 570.144W | decompiled firmware basis | - -Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs -570.144 emulation cost a full session before the real mmap bug was found — -the mismatch masked the signal.) - ---- - -## 8. Quick reference - -```bash -# Stack -./scripts/stack_manager.sh status|load-stack|unload|dmesg -# VM (harness ONLY - never raw /qemu) -./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell -# Debug -tail -f /tmp/vgpu-guest-serial.log & -gdb -ex 'target remote localhost:1234' -sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov' -# Recover wedged VM -./scripts/vgpu_vm.py stop <uuid> # or: sudo kill -9 <qemu pid> -``` |