diff options
Diffstat (limited to 'userspace/cuda/READ-ME-FIRST.md')
| -rw-r--r-- | userspace/cuda/READ-ME-FIRST.md | 151 |
1 files changed, 151 insertions, 0 deletions
diff --git a/userspace/cuda/READ-ME-FIRST.md b/userspace/cuda/READ-ME-FIRST.md new file mode 100644 index 0000000..08f0b28 --- /dev/null +++ b/userspace/cuda/READ-ME-FIRST.md @@ -0,0 +1,151 @@ +# READ-ME-FIRST — CUDA analysis guide for agents + +You are being pointed here because you are working on CUDA bring-up, +debugging or emulation for Open-IOV and there is now a body of analysis that +will save you hours. This file tells you **what to read, in what order, and +why it matters**, plus the fastest recipes for the questions you are likely to +have. + +Everything referenced here lives under `open-iov/userspace/cuda/` and was +written by the CUDA-decompilation analysis (2026-09). Nothing in this area +modifies the guest, the kernel module or the emulation; it is analysis plus +tooling. + +--- + +## 0. Read these two things first (5 minutes, they change how you debug) + +1. **`docs/05-570.144-decompilation.md` §1.1 — the address convention.** + Ghidra loads these binaries at **image base `0x100000`**. The original + per-version helper passed file vaddrs straight to Ghidra, so the old + decompiled files in `570.144/run/` (`helper_453ef0.c`, `fn_0x481360.c`, …) + are **the wrong functions** (0x100000 too low). The runtime observations in + `570.144/docs/ANALYSIS-cuctxcreate-segv.md` (gdb traces, live table values, + ioctl traces) are still valid; any conclusion drawn from its decompiled + code must be re-derived. The new tooling translates addresses for you: + `tools/ghidra_decompile_at.sh` and the annotated corpus. +2. **`docs/05` §3 — the `cuCtxCreate` path: crash solved, engine topology + blocking.** The old fault at `libcuda+0x453f90` (helper at `+0x453ef0`) is + solved: the alloc reply must **echo the request's params**, because the + driver copies 16 bytes of class params back over the caller's 8-byte local + (the hook's saved `rbp`) — a zeroed reply is what made `rbp=0` (§3.5). The + current blocker is one step later: the class-filtered engine list is + missing the async copy engine `COPY14`, so `cuCtxCreate` returns `rc=999` + (§3.6). §3.3 gives the engine-record layout (`base+0x2fec`, + `+0x300c+(idx-1)*0x14`, `+0x350c`, `+0x351c+(idx-0x16)*0x14`; fields + +0/+8/+0xc) and §3.7 lists exactly what the emulation should check first. + +If you read nothing else, read those two. + +## 1. The five-document core (30–45 minutes total) + +Read in this order; each builds on the previous one. + +| # | Document | Time | What you gain | +|---|---|---|---| +| 1 | `docs/01-architecture.md` | 15 min | The mental model: layers (API → engine → driver-model → hardware), context/device/stream/channel/memory objects, the marker + tracking-semaphore sync model, launch/QMD flow, UVM integration, threading. This is the vocabulary the rest of the docs use. | +| 2 | `docs/03-driver-and-gpu-interface.md` | 15 min | The emulation contract: device nodes and every ioctl CUDA issues, RM object classes, control families, memory/VA/mapping, channels/GPFIFO/USERD/doorbells, semaphores/notifiers/interrupts, UVM, GSP-RM. Ends with an 8-point emulation checklist. | +| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and current engine-topology blocker, and how to look anything up in the corpus. | +| 4 | `docs/04-virtualization-and-emulation.md` | 15 min | Design doc: how NVIDIA's vGPU split works, how Open-IOV differs (emulate the interface, execute the work), and the anti-fragility/performance/portability/security principles that constrain any fix you make. | +| 5 | `docs/02-build-and-execution.md` | 10 min | How the library is built (generated API wrappers, entry tables, embedded device code) and how it executes in a process (init sequence, threads, interposition, `/dev` access, why LD_PRELOAD can't see its syscalls). | + +## 2. Reference material (use when you need it) + +* `docs/06-tooling.md` — every tool, its usage and its cost. The recipes below + are extracts. +* `docs/07-version-upgrade-playbook.md` — how to redo the analysis when the + driver/CUDA/GSP changes. Read §1 if you only need the cheap `delta` triage. +* `docs/08-virtualization-approaches.md` — one page on the plausible execution + approaches (software emulation vs host-RM proxy vs direct submission vs + hybrid), their trade-offs, and the honest current stage of the project. +* `570.144/annotations/README.md` + `names.tsv` + `subsystems.tsv` — our name + map (659 exports, 628 resolved wrappers, verified internals) and subsystem + index. The map is applied to the annotated corpus already. + +## 3. Fast recipes (the 90% cases) + +**"What does the driver do with the reply we just gave it?"** +→ Find the ioctl/control in the guest trace, then locate the consuming code: +`docs/03` maps controls to subsystems; `docs/05` §4 shows corpus searches. +The annotated corpus headers list callers/callees with names, so you can walk +the path. + +**"Where is `libcuda+0xNNNNNN`?"** +``` +tools/cudatool.py nearest 570.144/run/libcuda.so.570.144 0xNNNNNN +tools/ghidra_decompile_at.sh 570.144/run/ghidra-full cuda-full \ + libcuda.so.570.144 0xNNNNNN /tmp/fn.c +grep -n "libcuda+0xNNNNNN" 570.144/run/analysis/decompiled-annotated/buckets/*.c +``` + +**"A guest segfault gives me `lib[base+size] ip` — what is it?"** +``` +tools/cudatool.py crash 570.144/run/libcuda.so.570.144 <base> <ip> +``` +It prints the file vaddr, containing function start, and nearest export. + +**"Which functions call X / does X call?"** +``` +tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv callers 0xNNNNNN +tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv callees 0xNNNNNN 2 +tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv chain 0xA 0xB +``` + +**"What changed between driver versions?"** +``` +tools/cudatool.py delta <old libcuda> <new libcuda> +``` +Same-generation builds (e.g. 570.144 vs 570.211.01) show the same 659 exports +with small moves — the name map transfers; new/removed exports flag feature +changes. + +**"I need the whole picture of a subsystem."** +→ `docs/01` for the model, then `570.144/annotations/subsystems.tsv` for the +known addresses, then the annotated corpus. `docs/05` §5 lists what the +decompiler does *not* recover (types, indirect targets). + +## 4. What is where on disk + +``` +userspace/cuda/ +├── READ-ME-FIRST.md ← you are here +├── README.md index + honest status +├── WORKING-HERE.md coordination rules (run/ is shared scratch) +├── docs/01..07 public analysis (tracked) +├── tools/ reusable tooling (tracked) +├── LOCAL-SOURCES.md local source map (gitignored) +└── 570.144/ + ├── README.md, docs/, scripts/ version-specific helpers + old write-up + ├── annotations/ name map + subsystem index (tracked) + ├── LOCAL-ANALYSIS.md local version notes (gitignored) + └── run/ + ├── libcuda.so.570.144 the guest binary (md5 ed5b2fc0…) + ├── ghidra-full/ analyzed project (29,494 functions) + ├── analysis/elf-index/ functions/exports/callgraph/strings TSVs + ├── analysis/decompiled/ raw corpus (29,254 functions, 0 failures) + └── analysis/decompiled-annotated/ corpus + names + call context +``` + +## 5. Rules while you work in this area + +* **Do not modify anything outside `userspace/cuda/` from analysis work.** + Emulation fixes go through the kernel module and the main handoff. +* **Do not clean `run/`** — it holds hours of analysis. You may + `rm -rf 570.144/run/analysis/decompiled/wk` to reclaim ~2 GB (per-chunk + project copies; the batch decompiler recreates them). +* **Decompiled output is never committed.** It is vendor-derived; the + gitignores in `open-iov/` and `570.144/` cover the binary, projects and + corpus. Tracked artifacts are our own words, names, tables and tools. +* **Write the public description first.** Tracked docs describe components + and behaviour; keep implementation details in the code and its comments. +* **When you fix the emulation**, update `docs/03` (contract) and, if the + interface understanding changed, `docs/05`; record the delta for the + version. + +## 6. The shortest possible version + +1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash + solved; engine-topology blocker) (read now). +2. `docs/01` and `docs/03` — the model and the contract. +3. `docs/04` — the design principles your fix must respect. +4. `docs/06` — the recipes; then `tools/`. |