# Debugging and Maintenance Workflows

> The practical book: how to see what is happening, how to find the cause, and
> how to add a new firmware, driver, hypervisor or hardware version without
> breaking anything.

## 1. The observability model

Three layers, in order of usefulness:

1. **Guest diagnostics** — the guest driver is the best oracle. It prints
   precisely what it expected and what it got (e.g. UVM's "Tracker entry for
   value N … channel_init()"). Always start here.
2. **Host module trace** — every RPC and every emulation decision is logged
   with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
   lines that reconstruct the object tree and the control battery.
3. **Memory observation** — when the problem is in memory state (mappings,
   semaphores, pushes), the module's debug parameters read and write guest
   physical memory directly.

## 2. Standard captures for a bug report

```bash
# Host
sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
./scripts/stack_manager.sh status > status.log
# Guest
sudo dmesg > guest-dmesg.log                 # include full dmesg
sudo lspci -vvv -d 10de: > guest-lspci.log   # config space of the virtual GPU
cat /proc/cmdline > guest-cmdline.log
nvidia-smi -q > nvidia-smi-q.log
```

Plus: the emulated driver profile (GPU type, firmware image hash, driver
version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
collects some of this automatically.

## 3. Live debugging toolbox

### 3.1 Kernel logs

```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio'          # live emulation trace
sudo dmesg -T | grep gsp-resman-vfio | tail -100    # recent
# capture to a file when the ring buffer is small and traffic is heavy:
sudo stdbuf -oL dmesg -w > run.log
```

### 3.2 Module debug parameters

See the host-module doc for the full list. Canonical examples:

```bash
# Find where a value lives in guest RAM (e.g. a semaphore VA):
echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram

# Find a decoded method sequence in guest RAM:
echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq

# Write a value to a guest physical address (completing a semaphore release):
echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
```

**Caution:** all guest-RAM operations run in a kernel thread context by design;
calling them from a normal process context fails with `-EFAULT` (the
hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
all). This is not a bug.

### 3.3 QEMU gdbstub (guest virtual memory)

The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
read guest *virtual* memory through the vCPU's page tables and write to it —
the single most powerful trick we have when the guest's own kernel address
matters:

```bash
gdb -batch -ex 'target remote :1234' \
    -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
```

Workflow that made the memory model tractable:

1. Find the guest's `page_offset_base` (direct-map base) from
   `/proc/kallsyms` in the guest.
2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
3. Use `poke_guest` to write the physical address from the host module, or use
   the gdbstub to write the virtual address directly.

### 3.4 Guest-side tracing

* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw
  syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
  `PTRACE_GET_SYSCALL_INFO` does.
* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
  (messages appear as `nvidia-uvm:`).
* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.

### 3.5 Failure decision trees

When a failure is understood, write the decision tree into this document tree
— the tree is what makes the next failure cheap. The field guide's tables are
the result of that discipline so far.

### 3.6 Userspace driver (`libcuda`) crash workflow

When the guest segfaults inside the userspace driver, the fault is reported as
`lib[base+size] ip`. Resolve it and find the code before reading any
decompilation:

```bash
# ip/mapping-base decoder: prints the file vaddr, containing function start,
# and nearest export
userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip>

# decompile exactly one file vaddr (translates for the Ghidra image base)
userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c

# callers/callees of an address in the annotated corpus
userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN
```

* **Address convention**: public offsets are *file virtual addresses*
  (`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base
  `0x100000`**, so addresses in old decompilations produced without that
  translation are 0x100000 too low (the wrong functions); runtime
  observations from those sessions remain valid, but conclusions drawn from
  their decompiled code must be re-derived. The current tooling translates
  for you.
* The annotated corpus (layout in
  [`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md))
  is searchable by `libcuda+0x...`; names/indexes are our own work product.
* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's
  engine/class list, virtualization mode and CE caps; compare against the
  host oracle (the same query on the host driver) before blaming the
  emulation.
* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on
  bare metal and in the guest and diffs the stable result lines.

**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999`
(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async
copy engine `COPY14`; the guest kernel oopses in
`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and
control `0x2080012b` (560-byte params, issued immediately before context
creation) is answered with a generic SUCCESS whose consumed fields are not yet
coherent. Start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md),
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md).

### 3.7 QEMU gdbstub hang triage

**The golden rule of failure triage: when the guest agent dies mid-test, the
guest is wedged in kernel mode. Stop retrying `exec` and go straight to the
gdbstub and host dmesg.** Triage order:

1. `sudo dmesg -T | tail -50` — KVM warnings (e.g. `make_spte`, EPT), our
   driver's last messages.
2. `tail -100 /tmp/vgpu-guest-serial.log` — the guest kernel's last words
   (works even when guest userspace is dead).
3. gdbstub vCPU sampling (below) — *where* is each vCPU stuck?

Attach without symbols for triage:

```bash
gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \
    -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \
    -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit'
```

Reading the result:

| Observation | Meaning |
|-------------|---------|
| vCPU RIP in kernel text, disassembly shows `hlt` | CPU idle — fine |
| vCPU RIP in module range, **same RIP+RSP across samples** | hard spin in a kernel module (e.g. an `nvidia.ko` polling loop) — this is the hung driver |
| RIPs moving between samples | guest alive and progressing |

Sample the stuck vCPU three times a few seconds apart. Identical `$rip` **and**
`$rsp` every time means the *same instruction* faults/retries (an
EPT-violation livelock) rather than a loop making progress. With guest symbols
(`vmlinux` plus the module `.ko` built with debug info) use `add-symbol-file`
to recover real function names.

### 3.8 QMP NMI (soft alternative)

```bash
(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \
 echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \
  | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock
```

Watch the serial log afterwards for backtraces. A truly hard-wedged vCPU
(spinning on a faulting MMIO instruction) may print nothing — use the gdbstub
instead.

## 4. Workflows for new inputs

### 4.1 New firmware / driver version (upgrade runbook)

**Version discipline first.** The anchor that must stay coherent is the
**guest driver ↔ emulation target** (the GSP firmware generation whose
protocol the host emulates): a guest driver whose protocol differs from what
the emulation implements produces failures indistinguishable from emulation
bugs. The host's physical-GPU driver is *not* part of that lock-step — the
reference host uses the distribution's driver, and the OpenRM host modules are
optional (native vGPU-management workflows only; see
[`../03-architecture/06-host-module-and-hybrid-model.md` §7](../03-architecture/06-host-module-and-hybrid-model.md)).
Today's emulation target is **570.144**; a version bump moves the guest driver
and the emulation, not the host distro driver.

**1. Acquire the firmware and matching source reference.**

* Extract the GSP firmware from the target driver runfile (or fetch the
  versioned blob from `linux-firmware.git`), then extract the `.fwimage`
  section (host boot image) and record its MD5:

  ```bash
  objcopy --only-section=.fwimage <extracted-blob> gsp-<VER>.fwimage.bin
  md5sum gsp-<VER>.fwimage.bin
  ```

* Note the matching NVIDIA OpenRM tag (e.g. `570.144`) — it is the
  version-exact reference for headers and wire structures.

**2. Lay down versioned directories (never overwrite a working version).**

```
firmware/<SERIES>-GSP/<VER>/
    gsp-<VER>.fwimage.bin
    decompiled/                 # Ghidra output lands here
config/versions/<MAJOR>.yaml    # design target; see §6 of the portability doc
```

Copy the analysis scripts from the previous version directory — they are
script-relative.

**3. Reverse-engineer the new firmware (Ghidra headless).**

```bash
cd tools/ghidra
python3 analyze_fwimage.py \
    ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \
    /tmp/ghidra_project_<VER> "gsp_<VER>"
cd ../../firmware/<SERIES>-GSP/<VER>
python3 analyze_firmware.py gsp-<VER>.fwimage.bin
```

Diff against the 570.144 record (`wiki/Address-Reference.md`). What to extract:

| Artifact | Why it matters |
|---|---|
| Primary RPC dispatcher + jump table | RPC types/subcommands may shift |
| BAR trap / WPR handlers | BAR emulation behaviour |
| Entry/boot sequence | the boot RPC ordering the guest driver expects |
| Migration tables | migration protocol (if enabled) |
| RPC message layouts | struct changes break the emulator silently |
| Version string/signature | runtime firmware matching |

Sanity checks: still RV64IMAC? still the same twelve top-level RPC types? new
subcommands? confidential-computing/TEE additions? Never carry 570.144
addresses into a new configuration — re-derive per firmware.

**4. Update the version surface (data first).**

* Add a hardware/version profile entry (addresses, signatures, capability
  toggles) so runtime analysis can identify the image; remember the YAML
  files under `config/` are design targets today (see
  [`../06-portability/hardware-software-matrix.md` §6](../06-portability/hardware-software-matrix.md)).
* Add the new entry to the runtime signature database in
  `kernel/firmware_analyzer.c` (`known_signatures[]`: version string, hashes,
  discovered addresses). This is what lets one `open-iov-vfio.ko` identify
  multiple firmware versions safely.

**5. Validate (same checklist every version).**

1. Host: module load/unload ×3 and mdev create/destroy
   (`./scripts/stack_manager.sh test`).
2. Guest: boot → `nvidia.ko` probe → `nvidia-smi` → teardown regression
   (no leaked `gsp_emu` thread, mdev removable again).
3. Inventory diff: capture the full init trace and diff function/control IDs
   against the previous version; for each difference decode against the
   exact-version reference headers, then add a version-keyed handler or a
   capability-table entry only where the diff requires it.
4. Then the evaluation ladder (rungs 1–2, then the CUDA corpus) and only then
   performance comparisons
   ([`01-performance-correctness-security.md`](01-performance-correctness-security.md)).

**6. Rollback.** Keep the previous version's directory and configuration
untouched. Rollback = check out the old anchor's commit/tag and reinstall the
old guest driver. Because versions live in parallel directories, a forward
attempt never destroys the working setup.

**Common pitfalls (learned the hard way).**

* **Runfile version drift**: distribution metapackages can resolve to a newer
  major than requested. Check the candidate version before installing; prefer
  the exact-version runfile for the guest.
* **Firmware filename mismatch**: a wrong-version firmware file is a probe
  failure that looks like an emulation bug.
* **Address drift per build**: kernel text addresses differ per firmware
  build; re-derive Ghidra addresses, never reuse them.
* **Symbol renames across OpenRM tags**: renamed exports break the optional
  OpenRM build with `Unknown symbol` — check the `kernel-open` diff when that
  path is in use.

**Multi-version testing matrix (target state).** For each supported anchor
version × each hardware family, run the step-5 bring-up sequence. When adding
a version, extend the matrix rather than replacing the old column —
regressions across versions are the cheapest bugs to catch.

### 4.2 New guest driver version

1. Install the stock driver in a guest (no patches).
2. Capture the full init trace and diff function/control IDs against the
   previous version.
3. For each difference: decode against the exact-version reference headers.
4. Add a version-keyed handler or a capability table entry.
5. Re-run rungs 1–2; then the CUDA corpus.

### 4.3 New hardware generation

1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
2. Run the ladder on that hardware with the closest driver family.
3. Fix differences by configuration first; only add code when an interface
   genuinely differs.
4. Publish the profile and the test evidence.

### 4.4 New hypervisor

1. Implement the transport boundary (device registration, guest-memory access,
   interrupt injection) — the boundary is small by design.
2. Re-run the ladder; the emulation core must not need changes.
3. Document any hypervisor-specific quirk in the debugging docs.

## 5. Crash and hang recovery

### 5.1 Failure taxonomy (symptom → likely cause)

| Symptom | Likely cause | First check |
|---------|--------------|-------------|
| `insmod open-iov-vfio.ko: Unknown symbol` | missing prerequisite module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` |
| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` |
| Guest: no PCI device | mdev not attached | QEMU command line for `sysfsdev=` |
| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN |
| Guest: probe hangs, no KVM warning | doorbell/RPC never consumed | host dmesg for the emulation thread and doorbell writes |
| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL |
| Guest agent dead, QEMU alive | guest kernel wedge | gdbstub vCPU sampling (§3.7) |
| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? |
| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs |
| `pr_warn_ratelimited(... not supported in emulation mode)` | guest used a BAR_TRAP/MIGRATE RPC | expected fail-fast; wire a real handler if hot |

### 5.2 Recovery actions

| Situation | Action |
|---|---|
| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |

### 5.3 Case study: the EPT livelock

Symptoms: guest `modprobe nvidia` hangs; the guest agent dies; QEMU stays
alive.

1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte`
   with `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN.
2. **gdbstub**: one vCPU frozen at a single address in module text, identical
   RIP/RSP across samples; the other vCPUs in `hlt`. The faulting instruction
   was `mov 0x4(%rax),%r13d` — a memory read that can never complete.
3. **Root cause**: the VFIO `mmap` handler used `virt_to_phys()` on the
   `vzalloc`'d BAR0 shadow → garbage PFN in the EPT → every guest BAR0 read
   faulted → KVM retried forever.
4. **Fix**: per-page `vmalloc_to_pfn()` mapping.

**Lesson: a KVM `make_spte` reserved-bits warning plus one frozen vCPU means a
bad PFN from your VFIO mmap handler.** Always question `virt_to_phys()` on
anything that is not lowmem; the module rule is recorded in
[`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md)
and the shadow mapping is described in
[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md).

## 6. Release engineering

* Versions are tagged; each tag carries the rung/matrix table and the
  performance table for the platforms tested.
* Configuration profiles are versioned with the code; a profile bump is a
  reviewed change with evidence.
* The docs in this tree are part of the release: a release that changes
  behaviour without updating them is incomplete.