diff options
Diffstat (limited to 'docs/05-operations')
| -rw-r--r-- | docs/05-operations/01-performance-correctness-security.md | 21 | ||||
| -rw-r--r-- | docs/05-operations/02-debugging-and-maintenance-workflows.md | 56 |
2 files changed, 70 insertions, 7 deletions
diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index c4d8153..0a27709 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,6 +17,19 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` +**Current rung (2026-09-23):** Rung 1 is green; **Rung 2 is NOT green.** +`cuInit`/`cuDeviceGet` succeed, but `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy +engine (`COPY14`); `cuMemAlloc` is not reached and teardown of the failed +context still oopses in the guest kernel. Before Rung 3 is claimed, the +**reply-fidelity gate** applies: a control whose reply fields the driver +consumes is not implemented until the reply is coherent (status-only → +coherent zeroed params → coherent values for consumed fields → exact values) — +see +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). +New to CUDA analysis here: start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 2. Correctness ### 2.1 Test corpus @@ -47,9 +60,11 @@ Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ### 2.3 Regression gates -`make test` runs the host suite. The CI ladder runs the guest suite on a -scheduled cadence and on every release. A rung regression is a release blocker; -the handoff documents carry the current rung. +`./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds +the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull +requests only — there is currently no scheduled guest suite and no +stock-driver contract test. A rung regression is a release blocker; the +handoff documents carry the current rung. ## 3. Performance diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index bc38a4e..331597a 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -88,7 +88,7 @@ Workflow that made the memory model tractable: ### 3.4 Guest-side tracing -* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw +* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with `PTRACE_GET_SYSCALL_INFO` does. * UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints` @@ -101,11 +101,60 @@ When a failure is understood, write the decision tree into this document tree — the tree is what makes the next failure cheap. The field guide's tables are the result of that discipline so far. +### 3.6 Userspace driver (`libcuda`) crash workflow + +When the guest segfaults inside the userspace driver, the fault is reported as +`lib[base+size] ip`. Resolve it and find the code before reading any +decompilation: + +```bash +# ip/mapping-base decoder: prints the file vaddr, containing function start, +# and nearest export +userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip> + +# decompile exactly one file vaddr (translates for the Ghidra image base) +userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c + +# callers/callees of an address in the annotated corpus +userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN +``` + +* **Address convention**: public offsets are *file virtual addresses* + (`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base + `0x100000`**, so addresses in old decompilations produced without that + translation are 0x100000 too low (the wrong functions); runtime + observations from those sessions remain valid, but conclusions drawn from + their decompiled code must be re-derived. The current tooling translates + for you. +* The annotated corpus (layout in + [`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md)) + is searchable by `libcuda+0x...`; names/indexes are our own work product. +* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's + engine/class list, virtualization mode and CE caps; compare against the + host oracle (the same query on the host driver) before blaming the + emulation. +* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on + bare metal and in the guest and diffs the stable result lines. + +**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async +copy engine `COPY14`; the guest kernel oopses in +`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and +control `0x2080012b` (560-byte params, issued immediately before context +creation) is answered with a generic SUCCESS whose consumed fields are not yet +coherent. Start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md), +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). + ## 4. Workflows for new inputs ### 4.1 New firmware version -1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS). +1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study + material; firmware blobs are gitignored and not distributed in the + repository). 2. Compute signatures (CRC32/SHA256) with the firmware analyzer. 3. Add a firmware profile entry (addresses/signatures) so runtime analysis can identify it. @@ -118,8 +167,7 @@ the result of that discipline so far. 1. Install the stock driver in a guest (no patches). 2. Capture the full init trace and diff function/control IDs against the previous version. -3. For each difference: decode against our decompiled study notes for that version - (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers). +3. For each difference: decode against the exact-version reference headers. 4. Add a version-keyed handler or a capability table entry. 5. Re-run rungs 1–2; then the CUDA corpus. |