diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:10 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:10 -0400 |
| commit | 692c095ae48f2829613873e60e4dbbf14b36323d (patch) | |
| tree | 6e48fc539c7b02b3f30654f7d1703beffa974277 /docs/05-operations/02-debugging-and-maintenance-workflows.md | |
| parent | f874d709e5a45d63c8ee9ab6b0300195aa5d20a5 (diff) | |
| download | open-iov-692c095ae48f2829613873e60e4dbbf14b36323d.tar.gz open-iov-692c095ae48f2829613873e60e4dbbf14b36323d.zip | |
docs: refresh status, supersede stale material, link the CUDA analysis
* Status now matches the live state: guest driver probe and nvidia-smi work;
cuInit/device queries work; cuCtxCreate returns 999 because the emulated
engine list is missing the async copy engine (COPY14). The userspace
libcuda segfault is recorded as solved, with its mechanism (an RM
allocation reply must echo request parameters; zeroing them corrupted the
caller's saved frame pointer).
* Correct the emulation claims: only a subset of RM controls are
special-cased; the rest return a success reply with zeroed parameters whose
consumed fields are still being corrected; allocation replies are coherent
but no server-side object model exists.
* Add the eight-point emulation checklist and reply-fidelity ladder as
explicit gates, documented in userspace/cuda and linked from the
architecture, implementation and operations docs.
* Fix operational errors: build/test commands, module load order, mdev type
names, LFS references, debugfs claims, in-repo firmware claims, CI claims,
portability matrix (guest status, config layout, userspace driver facts).
* Mark historical material: one superseded banner per legacy document with a
pointer to its replacement; STATE.md marked as a 2026-09-21 snapshot.
* Wiki: fix the index, add status banners and in-place factual corrections
(msgq transport vs legacy mailbox, BAR definitions, error statuses,
migration, glossary), and note the archived 2023 imports.
Diffstat (limited to 'docs/05-operations/02-debugging-and-maintenance-workflows.md')
| -rw-r--r-- | docs/05-operations/02-debugging-and-maintenance-workflows.md | 56 |
1 files changed, 52 insertions, 4 deletions
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index bc38a4e..331597a 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -88,7 +88,7 @@ Workflow that made the memory model tractable: ### 3.4 Guest-side tracing -* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw +* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with `PTRACE_GET_SYSCALL_INFO` does. * UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints` @@ -101,11 +101,60 @@ When a failure is understood, write the decision tree into this document tree — the tree is what makes the next failure cheap. The field guide's tables are the result of that discipline so far. +### 3.6 Userspace driver (`libcuda`) crash workflow + +When the guest segfaults inside the userspace driver, the fault is reported as +`lib[base+size] ip`. Resolve it and find the code before reading any +decompilation: + +```bash +# ip/mapping-base decoder: prints the file vaddr, containing function start, +# and nearest export +userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip> + +# decompile exactly one file vaddr (translates for the Ghidra image base) +userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c + +# callers/callees of an address in the annotated corpus +userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN +``` + +* **Address convention**: public offsets are *file virtual addresses* + (`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base + `0x100000`**, so addresses in old decompilations produced without that + translation are 0x100000 too low (the wrong functions); runtime + observations from those sessions remain valid, but conclusions drawn from + their decompiled code must be re-derived. The current tooling translates + for you. +* The annotated corpus (layout in + [`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md)) + is searchable by `libcuda+0x...`; names/indexes are our own work product. +* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's + engine/class list, virtualization mode and CE caps; compare against the + host oracle (the same query on the host driver) before blaming the + emulation. +* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on + bare metal and in the guest and diffs the stable result lines. + +**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async +copy engine `COPY14`; the guest kernel oopses in +`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and +control `0x2080012b` (560-byte params, issued immediately before context +creation) is answered with a generic SUCCESS whose consumed fields are not yet +coherent. Start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md), +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). + ## 4. Workflows for new inputs ### 4.1 New firmware version -1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS). +1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study + material; firmware blobs are gitignored and not distributed in the + repository). 2. Compute signatures (CRC32/SHA256) with the firmware analyzer. 3. Add a firmware profile entry (addresses/signatures) so runtime analysis can identify it. @@ -118,8 +167,7 @@ the result of that discipline so far. 1. Install the stock driver in a guest (no patches). 2. Capture the full init trace and diff function/control IDs against the previous version. -3. For each difference: decode against our decompiled study notes for that version - (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers). +3. For each difference: decode against the exact-version reference headers. 4. Add a version-keyed handler or a capability table entry. 5. Re-run rungs 1–2; then the CUDA corpus. |
