# Debugging and Maintenance Workflows

> The practical book: how to see what is happening, how to find the cause, and
> how to add a new firmware, driver, hypervisor or hardware version without
> breaking anything.

## 1. The observability model

Three layers, in order of usefulness:

1. **Guest diagnostics** — the guest driver is the best oracle. It prints
   precisely what it expected and what it got (e.g. UVM's "Tracker entry for
   value N … channel_init()"). Always start here.
2. **Host module trace** — every RPC and every emulation decision is logged
   with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
   lines that reconstruct the object tree and the control battery.
3. **Memory observation** — when the problem is in memory state (mappings,
   semaphores, pushes), the module's debug parameters read and write guest
   physical memory directly.

## 2. Standard captures for a bug report

```bash
# Host
sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
./scripts/stack_manager.sh status > status.log
# Guest
sudo dmesg > guest-dmesg.log                 # include full dmesg
sudo lspci -vvv -d 10de: > guest-lspci.log   # config space of the virtual GPU
cat /proc/cmdline > guest-cmdline.log
nvidia-smi -q > nvidia-smi-q.log
```

Plus: the emulated driver profile (GPU type, firmware image hash, driver
version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
collects some of this automatically.

## 3. Live debugging toolbox

### 3.1 Kernel logs

```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio'          # live emulation trace
sudo dmesg -T | grep gsp-resman-vfio | tail -100    # recent
# capture to a file when the ring buffer is small and traffic is heavy:
sudo stdbuf -oL dmesg -w > run.log
```

### 3.2 Module debug parameters

See the host-module doc for the full list. Canonical examples:

```bash
# Find where a value lives in guest RAM (e.g. a semaphore VA):
echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram

# Find a decoded method sequence in guest RAM:
echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq

# Write a value to a guest physical address (completing a semaphore release):
echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
```

**Caution:** all guest-RAM operations run in a kernel thread context by design;
calling them from a normal process context fails with `-EFAULT` (the
hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
all). This is not a bug.

### 3.3 QEMU gdbstub (guest virtual memory)

The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
read guest *virtual* memory through the vCPU's page tables and write to it —
the single most powerful trick we have when the guest's own kernel address
matters:

```bash
gdb -batch -ex 'target remote :1234' \
    -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
```

Workflow that made the memory model tractable:

1. Find the guest's `page_offset_base` (direct-map base) from
   `/proc/kallsyms` in the guest.
2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
3. Use `poke_guest` to write the physical address from the host module, or use
   the gdbstub to write the virtual address directly.

### 3.4 Guest-side tracing

* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw
  syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
  `PTRACE_GET_SYSCALL_INFO` does.
* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
  (messages appear as `nvidia-uvm:`).
* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.

### 3.5 Failure decision trees

When a failure is understood, write the decision tree into this document tree
— the tree is what makes the next failure cheap. The field guide's tables are
the result of that discipline so far.

## 4. Workflows for new inputs

### 4.1 New firmware version

1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS).
2. Compute signatures (CRC32/SHA256) with the firmware analyzer.
3. Add a firmware profile entry (addresses/signatures) so runtime analysis can
   identify it.
4. Run the driver against a guest using that firmware's matching driver; diff
   the RPC/control inventory against the baseline.
5. Add/adjust handlers only where the diff requires it; re-run the ladder.

### 4.2 New guest driver version

1. Install the stock driver in a guest (no patches).
2. Capture the full init trace and diff function/control IDs against the
   previous version.
3. For each difference: decode against our decompiled study notes for that version
   (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers).
4. Add a version-keyed handler or a capability table entry.
5. Re-run rungs 1–2; then the CUDA corpus.

### 4.3 New hardware generation

1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
2. Run the ladder on that hardware with the closest driver family.
3. Fix differences by configuration first; only add code when an interface
   genuinely differs.
4. Publish the profile and the test evidence.

### 4.4 New hypervisor

1. Implement the transport boundary (device registration, guest-memory access,
   interrupt injection) — the boundary is small by design.
2. Re-run the ladder; the emulation core must not need changes.
3. Document any hypervisor-specific quirk in the debugging docs.

## 5. Crash and hang recovery

| Situation | Action |
|---|---|
| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |

## 6. Release engineering

* Versions are tagged; each tag carries the rung/matrix table and the
  performance table for the platforms tested.
* Configuration profiles are versioned with the code; a profile bump is a
  reviewed change with evidence.
* The docs in this tree are part of the release: a release that changes
  behaviour without updating them is incomplete.