1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
|
# Debugging and Maintenance Workflows
> The practical book: how to see what is happening, how to find the cause, and
> how to add a new firmware, driver, hypervisor or hardware version without
> breaking anything.
## 1. The observability model
Three layers, in order of usefulness:
1. **Guest diagnostics** — the guest driver is the best oracle. It prints
precisely what it expected and what it got (e.g. UVM's "Tracker entry for
value N … channel_init()"). Always start here.
2. **Host module trace** — every RPC and every emulation decision is logged
with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
lines that reconstruct the object tree and the control battery.
3. **Memory observation** — when the problem is in memory state (mappings,
semaphores, pushes), the module's debug parameters read and write guest
physical memory directly.
## 2. Standard captures for a bug report
```bash
# Host
sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
./scripts/stack_manager.sh status > status.log
# Guest
sudo dmesg > guest-dmesg.log # include full dmesg
sudo lspci -vvv -d 10de: > guest-lspci.log # config space of the virtual GPU
cat /proc/cmdline > guest-cmdline.log
nvidia-smi -q > nvidia-smi-q.log
```
Plus: the emulated driver profile (GPU type, firmware image hash, driver
version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
collects some of this automatically.
## 3. Live debugging toolbox
### 3.1 Kernel logs
```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio' # live emulation trace
sudo dmesg -T | grep gsp-resman-vfio | tail -100 # recent
# capture to a file when the ring buffer is small and traffic is heavy:
sudo stdbuf -oL dmesg -w > run.log
```
### 3.2 Module debug parameters
See the host-module doc for the full list. Canonical examples:
```bash
# Find where a value lives in guest RAM (e.g. a semaphore VA):
echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram
# Find a decoded method sequence in guest RAM:
echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq
# Write a value to a guest physical address (completing a semaphore release):
echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
```
**Caution:** all guest-RAM operations run in a kernel thread context by design;
calling them from a normal process context fails with `-EFAULT` (the
hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
all). This is not a bug.
### 3.3 QEMU gdbstub (guest virtual memory)
The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
read guest *virtual* memory through the vCPU's page tables and write to it —
the single most powerful trick we have when the guest's own kernel address
matters:
```bash
gdb -batch -ex 'target remote :1234' \
-ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
```
Workflow that made the memory model tractable:
1. Find the guest's `page_offset_base` (direct-map base) from
`/proc/kallsyms` in the guest.
2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
3. Use `poke_guest` to write the physical address from the host module, or use
the gdbstub to write the virtual address directly.
### 3.4 Guest-side tracing
* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw
syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
`PTRACE_GET_SYSCALL_INFO` does.
* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
(messages appear as `nvidia-uvm:`).
* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.
### 3.5 Failure decision trees
When a failure is understood, write the decision tree into this document tree
— the tree is what makes the next failure cheap. The field guide's tables are
the result of that discipline so far.
### 3.6 Userspace driver (`libcuda`) crash workflow
When the guest segfaults inside the userspace driver, the fault is reported as
`lib[base+size] ip`. Resolve it and find the code before reading any
decompilation:
```bash
# ip/mapping-base decoder: prints the file vaddr, containing function start,
# and nearest export
userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip>
# decompile exactly one file vaddr (translates for the Ghidra image base)
userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c
# callers/callees of an address in the annotated corpus
userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN
```
* **Address convention**: public offsets are *file virtual addresses*
(`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base
`0x100000`**, so addresses in old decompilations produced without that
translation are 0x100000 too low (the wrong functions); runtime
observations from those sessions remain valid, but conclusions drawn from
their decompiled code must be re-derived. The current tooling translates
for you.
* The annotated corpus (layout in
[`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md))
is searchable by `libcuda+0x...`; names/indexes are our own work product.
* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's
engine/class list, virtualization mode and CE caps; compare against the
host oracle (the same query on the host driver) before blaming the
emulation.
* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on
bare metal and in the guest and diffs the stable result lines.
**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999`
(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async
copy engine `COPY14`; the guest kernel oopses in
`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and
control `0x2080012b` (560-byte params, issued immediately before context
creation) is answered with a generic SUCCESS whose consumed fields are not yet
coherent. Start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md),
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md).
## 4. Workflows for new inputs
### 4.1 New firmware version
1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study
material; firmware blobs are gitignored and not distributed in the
repository).
2. Compute signatures (CRC32/SHA256) with the firmware analyzer.
3. Add a firmware profile entry (addresses/signatures) so runtime analysis can
identify it.
4. Run the driver against a guest using that firmware's matching driver; diff
the RPC/control inventory against the baseline.
5. Add/adjust handlers only where the diff requires it; re-run the ladder.
### 4.2 New guest driver version
1. Install the stock driver in a guest (no patches).
2. Capture the full init trace and diff function/control IDs against the
previous version.
3. For each difference: decode against the exact-version reference headers.
4. Add a version-keyed handler or a capability table entry.
5. Re-run rungs 1–2; then the CUDA corpus.
### 4.3 New hardware generation
1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
2. Run the ladder on that hardware with the closest driver family.
3. Fix differences by configuration first; only add code when an interface
genuinely differs.
4. Publish the profile and the test evidence.
### 4.4 New hypervisor
1. Implement the transport boundary (device registration, guest-memory access,
interrupt injection) — the boundary is small by design.
2. Re-run the ladder; the emulation core must not need changes.
3. Document any hypervisor-specific quirk in the debugging docs.
## 5. Crash and hang recovery
| Situation | Action |
|---|---|
| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |
## 6. Release engineering
* Versions are tagged; each tag carries the rung/matrix table and the
performance table for the platforms tested.
* Configuration profiles are versioned with the code; a profile bump is a
reviewed change with evidence.
* The docs in this tree are part of the release: a release that changes
behaviour without updating them is incomplete.
|
