# 05 — 570.144 decompilation: map, key paths, and the `cuCtxCreate` bring-up This document is the version-specific result of the comprehensive decompilation of `libcuda.so.570.144` and the reference for anyone debugging the guest's CUDA bring-up. Addresses are **file virtual addresses** (`libcuda+0x...`). Artifacts behind it (gitignored, local): * `run/libcuda.so.570.144` — the guest's library, md5 `ed5b2fc0044df6052df1d5113754c499`; * `run/ghidra-full/` — the analyzed Ghidra project (29,494 functions); * `run/analysis/elf-index/` — `functions.tsv`, `exports.tsv`, `callgraph.tsv` (77,742 edges), `strings.tsv` (102,324 strings), `blocks.tsv`; * `run/analysis/decompiled/` — the full decompiled corpus (`buckets/*.c`, `index/*.tsv`). Reproduce with `tools/ghidra_full_analysis.sh`, `tools/ghidra_export_index.sh` and `tools/ghidra_batch_decompile.sh` (see `docs/06-tooling.md`). --- ## 1. Analysis result | Metric | Value | |---|---| | Functions (full auto-analysis) | 29,494 | | Functions with unwind info (FDEs) | 28,925 | | Exported symbols | 659 (plus 31 non-function exports) | | Call edges | 77,742 | | Defined strings | 102,324 | | Functions decompiled | 29,254 (all non-external functions in the executable range; 240 external imports) | | Decompilation failures | 0 | The analysis used Ghidra 12.1.2 with the two optional identifier analyzers disabled (parameter-ID and function-ID); everything else (disassembly, aggressive instruction finding, exception handlers, stack, reference analyzers) was on. This is the right trade-off for a stripped binary: it finishes in minutes and still finds essentially every function. ### 1.1 Address convention (read this before using any offset) Ghidra loads these shared objects at **image base `0x100000`**, so: ``` Ghidra program address = file virtual address + 0x100000 ``` The tools translate automatically. The earlier analysis in this tree did not: `run/helper_453ef0.c` and friends were produced by passing a file vaddr (`0x453ef0`) straight to Ghidra, which decompiled `libcuda+0x353ef0` — a *different function*. Every conclusion in the old write-up that depended on decompiled code was therefore about the wrong function. The runtime observations (gdb register traces, live table values, ioctl traces) remain valid and are used below. Correct addresses for the old write-up's landmarks: | What the old note called | File vaddr | Ghidra address | Correct identity | |---|---|---|---| | `FUN_00453ef0` (helper) | `0x453ef0` | `0x553ef0` | engine-context registration helper (per engine) | | crash site | `0x453f90` | `0x553f90` | continuation inside the helper after the hook call | | `FUN_0045a670` (walker) | `0x45a670` | `0x55a670` | device engine/context walker (calls the helper twice) | | `FUN_004676f0` (hook) | `0x4676f0` | `0x5676f0` | hook reached through `[table+0x40]+0x530` | | hook callee | `0x481360` | `0x581360` | engine context create/register (issues `NV_ESC_RM_ALLOC`) | | `FUN_003a9c20` | `0x3a9c20` | `0x4a9c20` | handle-pool allocation called before the hook | ## 2. The public API stubs Every exported `cu*` is a 24–30 byte stub: 1. compare the global initialization word against the "destroyed" sentinel `0x321CBA00`; if equal return `4` (`CUDA_ERROR_DEINITIALIZED`); 2. otherwise `jmp [slot]` — an indirect jump through the interposition table. Initial slot values (from relocations) give the generated wrapper addresses: | Export (file vaddr) | Wrapper target | |---|---| | `cuInit` `0x310060` | `0x31b7f0` | | `cuCtxCreate_v2` `0x310360` | `0x3206d0` | | `cuModuleLoadData` `0x3106e0` | `0x324ed0` | | `cuMemAlloc_v2` `0x310b20` | `0x329830` | | `cuStreamSynchronize` `0x36f3c0` | `0x380b40` | | `cuLaunchKernel` `0x36f420` | `0x3810f0` | This is the mechanism described in `docs/01` §2.2/§2.3: the wrapper does the tracing/tool callbacks and then calls the hand-written implementation. When chasing a call, follow the slot; when a tool interposes, the slot changes. ## 3. The `cuCtxCreate` path: crash solved, engine topology blocking The 2026-09 crash was deterministic: on a fresh guest `cuCtxCreate` segfaulted at `libcuda+0x453f90`. That is **solved** (§3.5). The call now returns `rc=999` (`CUDA_ERROR_UNKNOWN`), and the active blocker is engine topology — the emulated engine list is missing the async copy engine (§3.6). Fault facts from the crash period (live traces, corrected decompilation): ``` guest dmesg : segfault at ffffffffffffffc8 (NULL-0x38) ip in libcuda.so.570.144, mapping base 0x735cfcb79000 file vaddr = 0x453f90 gdb : rbp=0, rax=0, rsi=0xc030462b (_IOWR('F',0x2b,48) = NV_ESC_RM_ALLOC) ``` ### 3.1 The faulting instruction ``` libcuda+0x453ef0 push rbp; mov rbp,rsp; ... <- helper entry ... libcuda+0x453f5f call qword ptr [rax+0x530] <- engine hook libcuda+0x453f65 mov r12d,eax libcuda+0x453f68 test eax,eax libcuda+0x453f6a je 0x453f90 ... libcuda+0x453f90 mov rax,[rbp-0x38] <- FAULT (rbp==0) ``` The helper saved a table pointer at `[rbp-0x38]` in its prologue and reloads it after the hook returns. With `rbp==0` this reads `0xffffffffffffffc8`, exactly the fault address. ### 3.2 What the helper does (behaviour) `libcuda+0x453ef0(device, engine_desc)`: 1. `idx = engine_desc->[0]`, with `1` normalized to `0` (the GR engine); 2. read `table = *(*(*device + 8) + 0x40)` and save it; 3. allocate a handle from the global handle pool (`libcuda+0x3a9c20`, a bitmap allocator) → `engine_id`; 4. if the lookup succeeded, call hook `[*(table2+0x40)+0x530]` (`libcuda+0x4676f0`) with `(global, *(device+0x9488), engine_id, engine_desc)` — this is the engine create/register call that ultimately issues the `NV_ESC_RM_ALLOC` seen in `rsi` at the fault; 5. if the hook returns 0, reload the saved table, read `class = *(table+0xd40)`, and call a second hook `[*(table2+0x40)+0x250]` with `(global, *(device+0x9488), engine_id, class, device + idx*4 + 0x68)`; 6. on success record `device[idx*4 + 0x50] = engine_id` and `device[0x9488][idx*8 + 0x88] = engine_desc`; 7. on failure, clean up (`libcuda+0x453cc0`, `libcuda+0x3a2f50`). So the helper is "register one engine with the driver": look up the engine, create its context (hook 1), then bind/allocate the engine object (hook 2), then publish the result into per-engine arrays. ### 3.3 The engine walker `libcuda+0x45a670` walks the device's engine records and calls the helper at `libcuda+0x45afc1` and `libcuda+0x45b02b` (per engine, and once more for a secondary engine at `[device-table + 0x75c]`). The descriptor is selected from 20-byte (0x14) records: ``` engine index 0 -> table + 0x2fec engine index 1..0xa -> table + 0x300c + (index-1)*0x14 engine index 0xb -> table + 0x350c engine index >= 0x16 -> table + 0x351c + (index-0x16)*0x14 ``` before the helper, the walker calls the callback at `table+0x4e00` with `(table, desc->[0xc])` — a "does this engine/class exist" check — and only proceeds when it returns true. The helper then uses `desc->[0]` (engine kind), `desc->[8]` (instance) and `desc->[0xc]` (class) for the registration. ### 3.4 The engine create/register function `libcuda+0x481360` (called by the hook) is the function that actually asks RM for an engine context. Behaviour (condensed): * class `0x80` (device): find an existing context for the class, create one, then issue an RM alloc (`NV_ESC_RM_ALLOC`, request `0xc030462b`, 48-byte parameter block) to bind it; * classes `0x5`/`0x79`: use a per-class object from a global table, with a bounds-checked index; * class `0x2080` (subdevice): iterate a global array of 32 engine descriptors, matching `[context+0x28]`, and for a match build a 0x28-byte RM control (`0x202`) and alloc; * default: generic RM alloc via `libcuda+0x480d00`. Failure returns are small status codes (`0x1f`, `0x22`, `0x26`, `0x36`, `0x3d`, `0x56`, `0x59`, …) which the hook then routes into the driver's error reporting (`libcuda+0x387bf0`). ### 3.5 Resolution: the alloc-params copy-back The engine-registration hook reaches `libcuda+0x481360`, which for the `TURING_COMPUTE_A` class (`0xc5c0`) issues `NV_ESC_RM_ALLOC` (ioctl `0x2b`, NVOS64 form) with `pAllocParms` pointing at the caller's **own 8-byte stack local** and `paramsSize = 0`. The guest driver does not honour the caller's `paramsSize`: for class `0xc5c0` its resource table declares 16-byte class params (`NV_GR_ALLOCATION_PARAMETERS`), so it 1. reads 16 bytes starting at that 8-byte local — the upper 8 bytes are the hook's saved `rbp` — and sends them as the alloc parameters, and 2. on the reply, copies 16 bytes of the reply's params back over the same buffer. With the old generic (zeroed) alloc reply, step 2 zeroed the hook's saved `rbp`. The hook's `leave` then loaded `rbp = 0`, and `mov rax,[rbp-0x38]` at `libcuda+0x453f90` faulted at `0xffffffffffffffc8`. Live gdb intervals confirmed the writer: the hook's saved-`rbp` slot was intact at the pre-ioctl site (`libcuda+0x481448`) and zeroed at the post-ioctl site (`libcuda+0x48146a`), and hardware watchpoints never saw an in-process store — the store is the kernel's `copy_to_user` inside the ioctl. **Fix (host module, live-verified):** the alloc reply now echoes the request's params back, with only `status` forced to `NV_OK`. The driver's class-size copy-back then writes back exactly the bytes it read — including the caller's saved-`rbp` slot — and the `libcuda+0x453f90` fault disappears. This is also what the real GSP-RM does for these in/out params: the host runs the same libcuda code and survives for the same reason. It is the "coherent params" rung of the reply-fidelity ladder (`docs/04` §6). **Historical: why `rbp` was zero (solved).** Earlier revisions of this document listed three candidate writers of the hook's saved-`rbp` slot; the writer was the alloc reply's class-size copy-back described above. ### 3.6 Current blocker (2026-09-23): the missing async copy engine With the copy-back fixed, the walker completes but `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`); `cuMemAlloc` and later calls are not reached. Measured with the guest ioctl trace plus a host A/B oracle (the same libcuda path on the host's real driver): * libcuda asks the driver for the GPU's engines (`GET_ENGINES_V2`). The host oracle's filtered list has **7** engines: GR, CE0..CE4 and `COPY14`, the async copy engine. The guest list has only **6** — the same minus `COPY14`. * `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, client engine type `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's `{class, engDesc}` pair table pairs OBJCE instances `0..7` with class `0xcab5` and instances `0x0b..0x13` (including `0x0e` = 14) with class `0xc9b5`. * The driver filters the client engine table through its per-(class, instance) class database (`classDB`) before the engine walker (§3.3) sees it. With no classDB entry for descriptor `0x793ceb0e` the engine is silently skipped and libcuda never learns it exists. * The walker then registers a secondary engine from a record that is zeroed in the guest (host: `t=2 cls=0xc5b5`; guest: `t=0 cls=0`, dumped live with gdb). The zero class makes libcuda issue `NV_ESC_RM_ALLOC` for class `0`; the ioctl escape path rewrites root classes to `NV01_ROOT_CLIENT` (`0x41`), so the trace shows the single failing call `class=0x41 st=0x19` immediately after the compute-object alloc. * libcuda rolls back — it frees the channel and the compute object — and returns 999. The guest kernel still oopses in `uvm_pmm_devmem_page_to_va_space` while tearing down the failed context. A first emulator-side attempt (advertise the `COPY14` capability bit in the static info and add a 13th device-info entry with descriptor `0x793ceb0e`, instance 14, RM engine type `0x17`, FIFO tag 6, runlist 5) did not change the guest list, so the missing input is elsewhere: what builds the driver's classDB entry for descriptor `0x793ceb0e` (class list/static-info class data versus the device-info-table fields) is the open question. The record layout the walker reads is in §3.3; the per-engine fields it consumes are `+0` (kind), `+8` (instance) and `+0xc` (class). ### 3.7 What the emulation should check first * **Engine-list/classDB consistency — the active blocker (§3.6).** The engine records at the offsets in §3.3 must be present and self-consistent for every engine index the walker visits, the "does this engine exist" callback (`[table+0x4e00]`) must agree with them, and the whole set must agree with the class-filtered engine list the driver builds from the GSP static info (`GET_ENGINES_V2` + classDB). A missing entry is silent: the walker then reads a zeroed record. * **Consumed-control reply fidelity.** The controls answered generically (the host log lists `0x20803039`, `0x20802a12`, `0x20808162`, `0x2080182b`, `0x20800a70`, `0x20800a93`, …) should be checked for consumed fields, because the walker's engine table is built from data the driver collected earlier (device caps, engine list, class list). * The `NV_ESC_RM_ALLOC` parameter block built by `libcuda+0x481360` for class `0xc5c0` (`TURING_COMPUTE_A`) must be answered with a coherent object: echo the request's params back (§3.5) and return a coherent handle; the driver immediately consumes the reply's params and (for other classes) engine ids. ## 4. Navigating the decompiled corpus ``` # by address (file vaddr): grep -n "libcuda+0x45a670" 570.144/run/analysis/decompiled/buckets/*.c # by Ghidra name: awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv # callers of a function (program addresses): grep -P "^0x0055a670\t" 570.144/run/analysis/elf-index/callgraph.tsv # exports and their sizes: 570.144/run/analysis/elf-index/exports.tsv # strings around an address (error messages, table names): awk -F'\t' '$1>="0x..." && $1<="0x..."' 570.144/run/analysis/elf-index/strings.tsv ``` Every bucket header carries `addr=0x file_addr=libcuda+0x` so citations in docs and gdb sessions use the same numbers. ## 5. Known limits * Names are `FUN_` unless recovered; `570.144/annotations/names.tsv` holds our name map (exports are exact, internals grow over time). * The decompiler does not recover struct types; field offsets are named `[obj+0xNN]` and must be interpreted with the descriptions in `docs/01` and `docs/03`. * Indirect calls dominate the API dispatch; the callgraph contains direct edges only (77,742). Cross-referencing indirect targets requires reading the wrapper/stub relocations (as in §2) or live traces. * Embedded device code (cubins) is present in `.rodata` and is *not* host code; the corpus only covers `.text`.