path: root/docs/05-operations/02-debugging-and-maintenance-workflows.md

# Debugging and Maintenance Workflows

> The practical book: how to see what is happening, how to find the cause, and
> how to add a new firmware, driver, hypervisor or hardware version without
> breaking anything.

## 1. The observability model

Three layers, in order of usefulness:

1. **Guest diagnostics** — the guest driver is the best oracle. It prints
   precisely what it expected and what it got (e.g. UVM's "Tracker entry for
   value N … channel_init()"). Always start here.
2. **Host module trace** — every RPC and every emulation decision is logged
   with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
   lines that reconstruct the object tree and the control battery.
3. **Memory observation** — when the problem is in memory state (mappings,
   semaphores, pushes), the module's debug parameters read and write guest
   physical memory directly.

## 2. Standard captures for a bug report

```bash
# Host
sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
./scripts/stack_manager.sh status > status.log
# Guest
sudo dmesg > guest-dmesg.log                 # include full dmesg
sudo lspci -vvv -d 10de: > guest-lspci.log   # config space of the virtual GPU
cat /proc/cmdline > guest-cmdline.log
nvidia-smi -q > nvidia-smi-q.log
```

Plus: the emulated driver profile (GPU type, firmware image hash, driver
version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
collects some of this automatically.

## 3. Live debugging toolbox

### 3.1 Kernel logs

```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio'          # live emulation trace
sudo dmesg -T | grep gsp-resman-vfio | tail -100    # recent
# capture to a file when the ring buffer is small and traffic is heavy:
sudo stdbuf -oL dmesg -w > run.log
```

### 3.2 Module debug parameters

See the host-module doc for the full list. Canonical examples:

```bash
# Find where a value lives in guest RAM (e.g. a semaphore VA):
echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram

# Find a decoded method sequence in guest RAM:
echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq

# Write a value to a guest physical address (completing a semaphore release):
echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
```

**Caution:** all guest-RAM operations run in a kernel thread context by design;
calling them from a normal process context fails with `-EFAULT` (the
hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
all). This is not a bug.

### 3.3 QEMU gdbstub (guest virtual memory)

The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
read guest *virtual* memory through the vCPU's page tables and write to it —
the single most powerful trick we have when the guest's own kernel address
matters:

```bash
gdb -batch -ex 'target remote :1234' \
    -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
```

Workflow that made the memory model tractable:

1. Find the guest's `page_offset_base` (direct-map base) from
   `/proc/kallsyms` in the guest.
2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
3. Use `poke_guest` to write the physical address from the host module, or use
   the gdbstub to write the virtual address directly.

### 3.4 Guest-side tracing

* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw
  syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
  `PTRACE_GET_SYSCALL_INFO` does.
* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
  (messages appear as `nvidia-uvm:`).
* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.

### 3.5 Failure decision trees

When a failure is understood, write the decision tree into this document tree
— the tree is what makes the next failure cheap. The field guide's tables are
the result of that discipline so far.

### 3.6 Userspace driver (`libcuda`) crash workflow

When the guest segfaults inside the userspace driver, the fault is reported as
`lib[base+size] ip`. Resolve it and find the code before reading any
decompilation:

```bash
# ip/mapping-base decoder: prints the file vaddr, containing function start,
# and nearest export
userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip>

# decompile exactly one file vaddr (translates for the Ghidra image base)
userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c

# callers/callees of an address in the annotated corpus
userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN
```

* **Address convention**: public offsets are *file virtual addresses*
  (`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base
  `0x100000`**, so addresses in old decompilations produced without that
  translation are 0x100000 too low (the wrong functions); runtime
  observations from those sessions remain valid, but conclusions drawn from
  their decompiled code must be re-derived. The current tooling translates
  for you.
* The annotated corpus (layout in
  [`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md))
  is searchable by `libcuda+0x...`; names/indexes are our own work product.
* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's
  engine/class list, virtualization mode and CE caps; compare against the
  host oracle (the same query on the host driver) before blaming the
  emulation.
* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on
  bare metal and in the guest and diffs the stable result lines.

**Current frontier (2026-10-12)**: context creation and the full driver-API
smoke pass (`ALL OK`, 64-byte payload round-trips); a guest CUDA kernel
launch is forwarded but the engine does not execute the forwarded launch
descriptor yet, and green runs are not reproducible on every boot (tracked as
an open issue in the repository). Start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md),
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md).

### 3.7 QEMU gdbstub hang triage

**The golden rule of failure triage: when the guest agent dies mid-test, the
guest is wedged in kernel mode. Stop retrying `exec` and go straight to the
gdbstub and host dmesg.** Triage order:

1. `sudo dmesg -T | tail -50` — KVM warnings (e.g. `make_spte`, EPT), our
   driver's last messages.
2. `tail -100 /tmp/vgpu-guest-serial.log` — the guest kernel's last words
   (works even when guest userspace is dead).
3. gdbstub vCPU sampling (below) — *where* is each vCPU stuck?

Attach without symbols for triage:

```bash
gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \
    -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \
    -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit'
```

Reading the result:

| Observation | Meaning |
|-------------|---------|
| vCPU RIP in kernel text, disassembly shows `hlt` | CPU idle — fine |
| vCPU RIP in module range, **same RIP+RSP across samples** | hard spin in a kernel module (e.g. an `nvidia.ko` polling loop) — this is the hung driver |
| RIPs moving between samples | guest alive and progressing |

Sample the stuck vCPU three times a few seconds apart. Identical `$rip` **and**
`$rsp` every time means the *same instruction* faults/retries (an
EPT-violation livelock) rather than a loop making progress. With guest symbols
(`vmlinux` plus the module `.ko` built with debug info) use `add-symbol-file`
to recover real function names.

### 3.8 QMP NMI (soft alternative)

```bash
(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \
 echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \
  | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock
```

Watch the serial log afterwards for backtraces. A truly hard-wedged vCPU
(spinning on a faulting MMIO instruction) may print nothing — use the gdbstub
instead.

### 3.9 Firmware runtime analysis — risc-v-gsp-rm-debugger (experimental)

> Status: implemented, documented and tested (audit: [docs/27](../../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); value demos: [docs/25](../../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)).

For firmware-side questions (what does the GSP/uproc server do at runtime,
what does it read/write, how do its state machines evolve), the experimental
risc-v-gsp-rm-debugger tooling runs uproc/GSP firmware in a `qemu-system-riscv64` VM with
hooks, watchpoints, memory dumps and session recording, without a physical
GPU, WPR2 risk or a reboot. See
[`03-risc-v-gsp-rm-vm.md`](03-risc-v-gsp-rm-vm.md) for purpose, status and
limits; tooling and full documentation live in
[`../../debug/risc-v-gsp-rm-vm/`](../../debug/risc-v-gsp-rm-vm/README.md).
It is development-only: never shipped, never in a guest, and not a
substitute for the physical backend when silicon truth is required.

## 4. Workflows for new inputs

### 4.1 New firmware / driver version (upgrade runbook)

**Version discipline first.** The anchor that must stay coherent is the
**guest driver ↔ emulation target** (the GSP firmware generation whose
protocol the host emulates): a guest driver whose protocol differs from what
the emulation implements produces failures indistinguishable from emulation
bugs. The host's physical-GPU driver is *not* part of that lock-step — the
reference host uses the distribution's driver, and the OpenRM host modules are
optional (native vGPU-management workflows only; see
[`../03-architecture/06-host-module-and-hybrid-model.md` §7](../03-architecture/06-host-module-and-hybrid-model.md)).
Today's emulation target is **570.144**; a version bump moves the guest driver
and the emulation, not the host distro driver.

**1. Acquire the firmware and matching source reference.**

* Extract the GSP firmware from the target driver runfile (or fetch the
  versioned blob from `linux-firmware.git`), then extract the `.fwimage`
  section (host boot image) and record its MD5:

  ```bash
  objcopy --only-section=.fwimage <extracted-blob> gsp-<VER>.fwimage.bin
  md5sum gsp-<VER>.fwimage.bin
  ```

* Note the matching NVIDIA OpenRM tag (e.g. `570.144`) — it is the
  version-exact reference for headers and wire structures.

**2. Lay down versioned directories (never overwrite a working version).**

```
firmware/<SERIES>-GSP/<VER>/
    gsp-<VER>.fwimage.bin
    decompiled/                 # Ghidra output lands here
config/versions/<MAJOR>.yaml    # design target; see §6 of the portability doc
```

Copy the analysis scripts from the previous version directory — they are
script-relative.

**3. Reverse-engineer the new firmware (Ghidra headless).**

```bash
cd tools/ghidra
python3 analyze_fwimage.py \
    ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \
    /tmp/ghidra_project_<VER> "gsp_<VER>"
cd ../../firmware/<SERIES>-GSP/<VER>
python3 analyze_firmware.py gsp-<VER>.fwimage.bin
```

Diff against the 570.144 record (`wiki/Address-Reference.md`). What to extract:

| Artifact | Why it matters |
|---|---|
| Primary RPC dispatcher + jump table | RPC types/subcommands may shift |
| BAR trap / WPR handlers | BAR emulation behaviour |
| Entry/boot sequence | the boot RPC ordering the guest driver expects |
| Migration tables | migration protocol (if enabled) |
| RPC message layouts | struct changes break the emulator silently |
| Version string/signature | runtime firmware matching |

Sanity checks: still RV64IMAC? still the same twelve top-level RPC types? new
subcommands? confidential-computing/TEE additions? Never carry 570.144
addresses into a new configuration — re-derive per firmware.

**4. Update the version surface (data first).**

* Add a hardware/version profile entry (addresses, signatures, capability
  toggles) so runtime analysis can identify the image; remember the YAML
  files under `config/` are design targets today (see
  [`../06-portability/hardware-software-matrix.md` §6](../06-portability/hardware-software-matrix.md)).
* Add the new entry to the runtime signature database in
  `kernel/firmware_analyzer.c` (`known_signatures[]`: version string, hashes,
  discovered addresses). This is what lets one `open-iov-vfio.ko` identify
  multiple firmware versions safely.

**5. Validate (same checklist every version).**

1. Host: module load/unload ×3 and mdev create/destroy
   (`./scripts/stack_manager.sh test`).
2. Guest: boot → `nvidia.ko` probe → `nvidia-smi` → teardown regression
   (no leaked `gsp_emu` thread, mdev removable again).
3. Inventory diff: capture the full init trace and diff function/control IDs
   against the previous version; for each difference decode against the
   exact-version reference headers, then add a version-keyed handler or a
   capability-table entry only where the diff requires it.
4. Then the evaluation ladder (rungs 1–2, then the CUDA corpus) and only then
   performance comparisons
   ([`01-performance-correctness-security.md`](01-performance-correctness-security.md)).

**6. Rollback.** Keep the previous version's directory and configuration
untouched. Rollback = check out the old anchor's commit/tag and reinstall the
old guest driver. Because versions live in parallel directories, a forward
attempt never destroys the working setup.

**Common pitfalls (learned the hard way).**

* **Runfile version drift**: distribution metapackages can resolve to a newer
  major than requested. Check the candidate version before installing; prefer
  the exact-version runfile for the guest.
* **Firmware filename mismatch**: a wrong-version firmware file is a probe
  failure that looks like an emulation bug.
* **Address drift per build**: kernel text addresses differ per firmware
  build; re-derive Ghidra addresses, never reuse them.
* **Symbol renames across OpenRM tags**: renamed exports break the optional
  OpenRM build with `Unknown symbol` — check the `kernel-open` diff when that
  path is in use.

**Multi-version testing matrix (target state).** For each supported anchor
version × each hardware family, run the step-5 bring-up sequence. When adding
a version, extend the matrix rather than replacing the old column —
regressions across versions are the cheapest bugs to catch.

### 4.1.1 Risc-V GSP-RM debugger-assisted upgrade/porting workflow (experimental)

<!-- change (2026-10-02): add the risc-v-gsp-rm-debugger probe -> symbols -> version diff ->
portability matrix -> decide pipeline to the upgrade runbook, with pointers to
the new RUNBOOK and risc-v-gsp-rm-debugger user guide; additive only. -->

For firmware-side questions the experimental risc-v-gsp-rm-debugger tooling (see
[`03-risc-v-gsp-rm-vm.md`](03-risc-v-gsp-rm-vm.md)) turns steps 3–5 above into
a repeatable pipeline that produces attachable evidence before any emulation
change. It never replaces the physical backend when silicon truth is required,
and it never touches a guest, a kernel module or the installed driver.

| Rung | Command (from `debug/risc-v-gsp-rm-vm/`) | Output |
|---|---|---|
| Probe | `./risc-v-gsp-rm-debugger.sh probe <new.fwimage> --symbols symbols-<ver>.json --record` | first-divergence report; first-touch map |
| Symbols | `./risc-v-gsp-rm-debugger.sh symbols export --index <export.tsv> --bias 0 -o symbols-<ver>.json` | runtime-address symbol DB with provenance |
| Version diff | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-<ver>` | shape/symbol/corpus/probe/behavior diff plus the M1–M6 skeleton |
| Portability matrix | `python3 harness/portability_matrix.py --target <old> --target <new> --out dumps/matrix-<ver>` | per-target sessions and per-pair `compare` reports |
| Decide | `./risc-v-gsp-rm-debugger.sh compare dumps/<old-run> dumps/<new-run> --symbols` | symbol-attributed deltas; a product change only after physical confirmation |

Decision discipline: verify the pairing first; behaviour that moved without a
code change is usually a state/ordering fix; changed code on a live path is
prioritised; a symbol never hit in either run is a coverage gap, not dead code.
The full incident procedures (symptoms, evidence, response order, rollback) are
in the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S1 and its OODA
ordering; the quickstart and tool chooser are in the
[risc-v-gsp-rm-debugger user guide](04-risc-v-gsp-rm-debugger-user-guide.md).

### 4.2 New guest driver version

1. Install the stock driver in a guest (no patches).
2. Capture the full init trace and diff function/control IDs against the
   previous version.
3. For each difference: decode against the exact-version reference headers.
4. Add a version-keyed handler or a capability table entry.
5. Re-run rungs 1–2; then the CUDA corpus.

### 4.3 New hardware generation

1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
2. Run the ladder on that hardware with the closest driver family.
3. Fix differences by configuration first; only add code when an interface
   genuinely differs.
4. Publish the profile and the test evidence.

### 4.4 New hypervisor

1. Implement the transport boundary (device registration, guest-memory access,
   interrupt injection) — the boundary is small by design.
2. Re-run the ladder; the emulation core must not need changes.
3. Document any hypervisor-specific quirk in the debugging docs.

## 5. Crash and hang recovery

### 5.1 Failure taxonomy (symptom → likely cause)

| Symptom | Likely cause | First check |
|---------|--------------|-------------|
| `insmod open-iov-vfio.ko: Unknown symbol` | missing prerequisite module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` |
| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` |
| Guest: no PCI device | mdev not attached | QEMU command line for `sysfsdev=` |
| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN |
| Guest: probe hangs, no KVM warning | doorbell/RPC never consumed | host dmesg for the emulation thread and doorbell writes |
| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL |
| Guest agent dead, QEMU alive | guest kernel wedge | gdbstub vCPU sampling (§3.7) |
| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? |
| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs |
| `pr_warn_ratelimited(... not supported in emulation mode)` | guest used a BAR_TRAP/MIGRATE RPC | expected fail-fast; wire a real handler if hot |

### 5.2 Recovery actions

| Situation | Action |
|---|---|
| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |

### 5.3 Case study: the EPT livelock

Symptoms: guest `modprobe nvidia` hangs; the guest agent dies; QEMU stays
alive.

1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte`
   with `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN.
2. **gdbstub**: one vCPU frozen at a single address in module text, identical
   RIP/RSP across samples; the other vCPUs in `hlt`. The faulting instruction
   was `mov 0x4(%rax),%r13d` — a memory read that can never complete.
3. **Root cause**: the VFIO `mmap` handler used `virt_to_phys()` on the
   `vzalloc`'d BAR0 shadow → garbage PFN in the EPT → every guest BAR0 read
   faulted → KVM retried forever.
4. **Fix**: per-page `vmalloc_to_pfn()` mapping.

**Lesson: a KVM `make_spte` reserved-bits warning plus one frozen vCPU means a
bad PFN from your VFIO mmap handler.** Always question `virt_to_phys()` on
anything that is not lowmem; the module rule is recorded in
[`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md)
and the shadow mapping is described in
[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md).

## 6. Release engineering

* Versions are tagged; each tag carries the rung/matrix table and the
  performance table for the platforms tested.
* Configuration profiles are versioned with the code; a profile bump is a
  reviewed change with evidence.
* The docs in this tree are part of the release: a release that changes
  behaviour without updating them is incomplete.