1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
# Debugging and Maintenance Workflows
> The practical book: how to see what is happening, how to find the cause, and
> how to add a new firmware, driver, hypervisor or hardware version without
> breaking anything.
## 1. The observability model
Three layers, in order of usefulness:
1. **Guest diagnostics** — the guest driver is the best oracle. It prints
precisely what it expected and what it got (e.g. UVM's "Tracker entry for
value N … channel_init()"). Always start here.
2. **Host module trace** — every RPC and every emulation decision is logged
with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
lines that reconstruct the object tree and the control battery.
3. **Memory observation** — when the problem is in memory state (mappings,
semaphores, pushes), the module's debug parameters read and write guest
physical memory directly.
## 2. Standard captures for a bug report
```bash
# Host
sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
./scripts/stack_manager.sh status > status.log
# Guest
sudo dmesg > guest-dmesg.log # include full dmesg
sudo lspci -vvv -d 10de: > guest-lspci.log # config space of the virtual GPU
cat /proc/cmdline > guest-cmdline.log
nvidia-smi -q > nvidia-smi-q.log
```
Plus: the emulated driver profile (GPU type, firmware image hash, driver
version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
collects some of this automatically.
## 3. Live debugging toolbox
### 3.1 Kernel logs
```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio' # live emulation trace
sudo dmesg -T | grep gsp-resman-vfio | tail -100 # recent
# capture to a file when the ring buffer is small and traffic is heavy:
sudo stdbuf -oL dmesg -w > run.log
```
### 3.2 Module debug parameters
See the host-module doc for the full list. Canonical examples:
```bash
# Find where a value lives in guest RAM (e.g. a semaphore VA):
echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram
# Find a decoded method sequence in guest RAM:
echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq
# Write a value to a guest physical address (completing a semaphore release):
echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
```
**Caution:** all guest-RAM operations run in a kernel thread context by design;
calling them from a normal process context fails with `-EFAULT` (the
hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
all). This is not a bug.
### 3.3 QEMU gdbstub (guest virtual memory)
The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
read guest *virtual* memory through the vCPU's page tables and write to it —
the single most powerful trick we have when the guest's own kernel address
matters:
```bash
gdb -batch -ex 'target remote :1234' \
-ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
```
Workflow that made the memory model tractable:
1. Find the guest's `page_offset_base` (direct-map base) from
`/proc/kallsyms` in the guest.
2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
3. Use `poke_guest` to write the physical address from the host module, or use
the gdbstub to write the virtual address directly.
### 3.4 Guest-side tracing
* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw
syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
`PTRACE_GET_SYSCALL_INFO` does.
* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
(messages appear as `nvidia-uvm:`).
* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.
### 3.5 Failure decision trees
When a failure is understood, write the decision tree into this document tree
— the tree is what makes the next failure cheap. The field guide's tables are
the result of that discipline so far.
## 4. Workflows for new inputs
### 4.1 New firmware version
1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS).
2. Compute signatures (CRC32/SHA256) with the firmware analyzer.
3. Add a firmware profile entry (addresses/signatures) so runtime analysis can
identify it.
4. Run the driver against a guest using that firmware's matching driver; diff
the RPC/control inventory against the baseline.
5. Add/adjust handlers only where the diff requires it; re-run the ladder.
### 4.2 New guest driver version
1. Install the stock driver in a guest (no patches).
2. Capture the full init trace and diff function/control IDs against the
previous version.
3. For each difference: decode against our decompiled study notes for that version
(`g_rpc-structures.h`, `rpc_global_enums.h`, control headers).
4. Add a version-keyed handler or a capability table entry.
5. Re-run rungs 1–2; then the CUDA corpus.
### 4.3 New hardware generation
1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
2. Run the ladder on that hardware with the closest driver family.
3. Fix differences by configuration first; only add code when an interface
genuinely differs.
4. Publish the profile and the test evidence.
### 4.4 New hypervisor
1. Implement the transport boundary (device registration, guest-memory access,
interrupt injection) — the boundary is small by design.
2. Re-run the ladder; the emulation core must not need changes.
3. Document any hypervisor-specific quirk in the debugging docs.
## 5. Crash and hang recovery
| Situation | Action |
|---|---|
| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |
## 6. Release engineering
* Versions are tagged; each tag carries the rung/matrix table and the
performance table for the platforms tested.
* Configuration profiles are versioned with the code; a profile bump is a
reviewed change with evidence.
* The docs in this tree are part of the release: a release that changes
behaviour without updating them is incomplete.
|
