# 05 — 570.144 decompilation: map, key paths, and the `cuCtxCreate` bring-up This document is the version-specific result of the comprehensive decompilation of `libcuda.so.570.144` and the reference for anyone debugging the guest's CUDA bring-up. Addresses are **file virtual addresses** (`libcuda+0x...`). Artifacts behind it (gitignored, local): * `run/libcuda.so.570.144` — the guest's library, md5 `ed5b2fc0044df6052df1d5113754c499`; * `run/ghidra-full/` — the analyzed Ghidra project (29,494 functions); * `run/analysis/elf-index/` — `functions.tsv`, `exports.tsv`, `callgraph.tsv` (77,742 edges), `strings.tsv` (102,324 strings), `blocks.tsv`; * `run/analysis/decompiled/` — the full decompiled corpus (`buckets/*.c`, `index/*.tsv`). Reproduce with `tools/ghidra_full_analysis.sh`, `tools/ghidra_export_index.sh` and `tools/ghidra_batch_decompile.sh` (see `docs/06-tooling.md`). --- ## 1. Analysis result | Metric | Value | |---|---| | Functions (full auto-analysis) | 29,494 | | Functions with unwind info (FDEs) | 28,925 | | Exported symbols | 659 (plus 31 non-function exports) | | Call edges | 77,742 | | Defined strings | 102,324 | | Functions decompiled | 29,254 (all non-external functions in the executable range; 240 external imports) | | Decompilation failures | 0 | The analysis used Ghidra 12.1.2 with the two optional identifier analyzers disabled (parameter-ID and function-ID); everything else (disassembly, aggressive instruction finding, exception handlers, stack, reference analyzers) was on. This is the right trade-off for a stripped binary: it finishes in minutes and still finds essentially every function. ### 1.1 Address convention (read this before using any offset) Ghidra loads these shared objects at **image base `0x100000`**, so: ``` Ghidra program address = file virtual address + 0x100000 ``` The tools translate automatically. The earlier analysis in this tree did not: `run/helper_453ef0.c` and friends were produced by passing a file vaddr (`0x453ef0`) straight to Ghidra, which decompiled `libcuda+0x353ef0` — a *different function*. Every conclusion in the old write-up that depended on decompiled code was therefore about the wrong function. The runtime observations (gdb register traces, live table values, ioctl traces) remain valid and are used below. Correct addresses for the old write-up's landmarks: | What the old note called | File vaddr | Ghidra address | Correct identity | |---|---|---|---| | `FUN_00453ef0` (helper) | `0x453ef0` | `0x553ef0` | engine-context registration helper (per engine) | | crash site | `0x453f90` | `0x553f90` | continuation inside the helper after the hook call | | `FUN_0045a670` (walker) | `0x45a670` | `0x55a670` | device engine/context walker (calls the helper twice) | | `FUN_004676f0` (hook) | `0x4676f0` | `0x5676f0` | hook reached through `[table+0x40]+0x530` | | hook callee | `0x481360` | `0x581360` | engine context create/register (issues `NV_ESC_RM_ALLOC`) | | `FUN_003a9c20` | `0x3a9c20` | `0x4a9c20` | handle-pool allocation called before the hook | ## 2. The public API stubs Every exported `cu*` is a 24–30 byte stub: 1. compare the global initialization word against the "destroyed" sentinel `0x321CBA00`; if equal return `4` (`CUDA_ERROR_DEINITIALIZED`); 2. otherwise `jmp [slot]` — an indirect jump through the interposition table. Initial slot values (from relocations) give the generated wrapper addresses: | Export (file vaddr) | Wrapper target | |---|---| | `cuInit` `0x310060` | `0x31b7f0` | | `cuCtxCreate_v2` `0x310360` | `0x3206d0` | | `cuModuleLoadData` `0x3106e0` | `0x324ed0` | | `cuMemAlloc_v2` `0x310b20` | `0x329830` | | `cuStreamSynchronize` `0x36f3c0` | `0x380b40` | | `cuLaunchKernel` `0x36f420` | `0x3810f0` | This is the mechanism described in `docs/01` §2.2/§2.3: the wrapper does the tracing/tool callbacks and then calls the hand-written implementation. When chasing a call, follow the slot; when a tool interposes, the slot changes. ## 3. The `cuCtxCreate` path: crash solved, engine topology blocking The 2026-09 crash was deterministic: on a fresh guest `cuCtxCreate` segfaulted at `libcuda+0x453f90`. That is **solved** (§3.5). The call now returns `rc=999` (`CUDA_ERROR_UNKNOWN`), and the active blocker is engine topology — the emulated engine list is missing the async copy engine (§3.6). Fault facts from the crash period (live traces, corrected decompilation): ``` guest dmesg : segfault at ffffffffffffffc8 (NULL-0x38) ip in libcuda.so.570.144, mapping base 0x735cfcb79000 file vaddr = 0x453f90 gdb : rbp=0, rax=0, rsi=0xc030462b (_IOWR('F',0x2b,48) = NV_ESC_RM_ALLOC) ``` ### 3.1 The faulting instruction ``` libcuda+0x453ef0 push rbp; mov rbp,rsp; ... <- helper entry ... libcuda+0x453f5f call qword ptr [rax+0x530] <- engine hook libcuda+0x453f65 mov r12d,eax libcuda+0x453f68 test eax,eax libcuda+0x453f6a je 0x453f90 ... libcuda+0x453f90 mov rax,[rbp-0x38] <- FAULT (rbp==0) ``` The helper saved a table pointer at `[rbp-0x38]` in its prologue and reloads it after the hook returns. With `rbp==0` this reads `0xffffffffffffffc8`, exactly the fault address. ### 3.2 What the helper does (behaviour) `libcuda+0x453ef0(device, engine_desc)`: 1. `idx = engine_desc->[0]`, with `1` normalized to `0` (the GR engine); 2. read `table = *(*(*device + 8) + 0x40)` and save it; 3. allocate a handle from the global handle pool (`libcuda+0x3a9c20`, a bitmap allocator) → `engine_id`; 4. if the lookup succeeded, call hook `[*(table2+0x40)+0x530]` (`libcuda+0x4676f0`) with `(global, *(device+0x9488), engine_id, engine_desc)` — this is the engine create/register call that ultimately issues the `NV_ESC_RM_ALLOC` seen in `rsi` at the fault; 5. if the hook returns 0, reload the saved table, read `class = *(table+0xd40)`, and call a second hook `[*(table2+0x40)+0x250]` with `(global, *(device+0x9488), engine_id, class, device + idx*4 + 0x68)`; 6. on success record `device[idx*4 + 0x50] = engine_id` and `device[0x9488][idx*8 + 0x88] = engine_desc`; 7. on failure, clean up (`libcuda+0x453cc0`, `libcuda+0x3a2f50`). So the helper is "register one engine with the driver": look up the engine, create its context (hook 1), then bind/allocate the engine object (hook 2), then publish the result into per-engine arrays. ### 3.3 The engine walker `libcuda+0x45a670` walks the device's engine records and calls the helper at `libcuda+0x45afc1` and `libcuda+0x45b02b` (per engine, and once more for a secondary engine at `[device-table + 0x75c]`). The descriptor is selected from 20-byte (0x14) records: ``` engine index 0 -> table + 0x2fec engine index 1..0xa -> table + 0x300c + (index-1)*0x14 engine index 0xb -> table + 0x350c engine index >= 0x16 -> table + 0x351c + (index-0x16)*0x14 ``` before the helper, the walker calls the callback at `table+0x4e00` with `(table, desc->[0xc])` — a "does this engine/class exist" check (a class-list membership test; the callback pointer is installed by the device ABI table setup at `libcuda+0x458040`/`libcuda+0x4494d0`, target `libcuda+0x44a170`) — and only proceeds when it returns true. The helper then uses `desc->[0]` (engine kind), `desc->[8]` (instance) and `desc->[0xc]` (class) for the registration. The records are built by `libcuda+0x450ff0` from the engine list that the `GET_ENGINES_V2` wrapper `libcuda+0x462770` returns (`{kind, engine type, instance}` triples at 0x10-byte stride). The record layout and the full field map are in §3.7. Note the walker's index ranges cover copy slots `1..0xa` and `0xc..0x15`, with `0xb` used for the SEC2 slot and `>=0x16` for decompression copy slots; the engine list is a different, dense array. ### 3.4 The engine create/register function `libcuda+0x481360` (called by the hook) is the function that actually asks RM for an engine context. Behaviour (condensed): * class `0x80` (device): find an existing context for the class, create one, then issue an RM alloc (`NV_ESC_RM_ALLOC`, request `0xc030462b`, 48-byte parameter block) to bind it; * classes `0x5`/`0x79`: use a per-class object from a global table, with a bounds-checked index; * class `0x2080` (subdevice): iterate a global array of 32 engine descriptors, matching `[context+0x28]`, and for a match build a 0x28-byte RM control (`0x202`) and alloc; * default: generic RM alloc via `libcuda+0x480d00`. Failure returns are small status codes (`0x1f`, `0x22`, `0x26`, `0x36`, `0x3d`, `0x56`, `0x59`, …) which the hook then routes into the driver's error reporting (`libcuda+0x387bf0`). ### 3.5 Resolution: the alloc-params copy-back The engine-registration hook reaches `libcuda+0x481360`, which for the `TURING_COMPUTE_A` class (`0xc5c0`) issues `NV_ESC_RM_ALLOC` (ioctl `0x2b`, NVOS64 form) with `pAllocParms` pointing at the caller's **own 8-byte stack local** and `paramsSize = 0`. The guest driver does not honour the caller's `paramsSize`: for class `0xc5c0` its resource table declares 16-byte class params (`NV_GR_ALLOCATION_PARAMETERS`), so it 1. reads 16 bytes starting at that 8-byte local — the upper 8 bytes are the hook's saved `rbp` — and sends them as the alloc parameters, and 2. on the reply, copies 16 bytes of the reply's params back over the same buffer. With the old generic (zeroed) alloc reply, step 2 zeroed the hook's saved `rbp`. The hook's `leave` then loaded `rbp = 0`, and `mov rax,[rbp-0x38]` at `libcuda+0x453f90` faulted at `0xffffffffffffffc8`. Live gdb intervals confirmed the writer: the hook's saved-`rbp` slot was intact at the pre-ioctl site (`libcuda+0x481448`) and zeroed at the post-ioctl site (`libcuda+0x48146a`), and hardware watchpoints never saw an in-process store — the store is the kernel's `copy_to_user` inside the ioctl. **Fix (host module, live-verified):** the alloc reply now echoes the request's params back, with only `status` forced to `NV_OK`. The driver's class-size copy-back then writes back exactly the bytes it read — including the caller's saved-`rbp` slot — and the `libcuda+0x453f90` fault disappears. This is also what the real GSP-RM does for these in/out params: the host runs the same libcuda code and survives for the same reason. It is the "coherent params" rung of the reply-fidelity ladder (`docs/04` §6). **Historical: why `rbp` was zero (solved).** Earlier revisions of this document listed three candidate writers of the hook's saved-`rbp` slot; the writer was the alloc reply's class-size copy-back described above. ### 3.6 Current blocker (2026-09-23): the missing async copy engine With the copy-back fixed, the walker completes but `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`); `cuMemAlloc` and later calls are not reached. Measured with the guest ioctl trace plus a host A/B oracle (the same libcuda path on the host's real driver): * libcuda asks the driver for the GPU's engines (`GET_ENGINES_V2`). The host oracle's filtered list has **7** engines: GR, CE0..CE4 and `COPY14`, the async copy engine. The guest list has only **6** — the same minus `COPY14`. * `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, engine type `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's `{class, engDesc}` pair table pairs OBJCE instances `0..7` with class `0xcab5` and instances `0x0b..0x13` (including `0x0e` = 14) with class `0xc9b5`. * The driver filters the client engine table through its per-(class, instance) class database (`classDB`) before the engine walker (§3.3) sees it. With no classDB entry for descriptor `0x793ceb0e` the engine is silently skipped and libcuda never learns it exists. * The walker then registers a secondary engine from a record that is zeroed in the guest (host: `t=2 cls=0xc5b5`; guest: `t=0 cls=0`, dumped live with gdb). The zero class makes libcuda issue `NV_ESC_RM_ALLOC` for class `0`; the ioctl escape path rewrites root classes to `NV01_ROOT_CLIENT` (`0x41`), so the trace shows the single failing call `class=0x41 st=0x19` immediately after the compute-object alloc. * libcuda rolls back — it frees the channel and the compute object — and returns 999. The guest kernel still oopses in `uvm_pmm_devmem_page_to_va_space` while tearing down the failed context. A first emulator-side attempt (advertise the `COPY14` capability bit in the static info and add a 13th device-info entry with descriptor `0x793ceb0e`, instance 14, RM engine type `0x17`, FIFO tag 6, runlist 5) did not change the guest list, so the missing input is elsewhere: what builds the driver's classDB entry for descriptor `0x793ceb0e` (class list/static-info class data versus the device-info-table fields) is the open question. ### 3.7 The engine descriptor record, and how it explains the blocker The device state holds a list of 0x10-byte engine entries plus a parallel set of 0x14-byte **per-engine descriptor records**; the walker (§3.3) reads the records, the helper and create hook consume them. The record was recovered from the record builder (`libcuda+0x450ff0`), the `GET_ENGINES_V2` wrapper (`libcuda+0x462770`), the create hook disassembly (`libcuda+0x4676f0`) and the live guest dump (`run/gdb32b.out`): | Offset | Field | Meaning | Evidence | |---|---|---|---| | `+0x00` | engine kind | `1` = graphics (GR), `3` = copy (CE), `4` = SEC2, `5` = decompression copy. The helper normalizes `1` to `0` for the per-context slot index. | builder branches on it; `GET_ENGINES_V2` wrapper classifies each engine type into it; live: GR `t=1`, CE `t=3` | | `+0x04` | engine type | raw `NV2080_ENGINE_TYPE` from the engine list (`1` = GR, `9..13` = COPY0..4, `0x38` = COPY14) | builder copies the first 8 bytes of the list entry into `+0/+4`; not read by the walker/helper | | `+0x08` | instance | engine instance (`COPY0..4` → `0..4`, `COPY14` → `14`) | helper + create hook read it; create hook copies it into the RM alloc parameters only for kinds 2/3/5; live dump | | `+0x0c` | class | RM allocation class for this engine: GR → the device's compute class (live `0xc5c0`), copy → the DMA copy class (live `0xc5b5`), SEC2/decomp copy → their classes | builder stores the per-kind class; the create hook passes `+0xc` straight to `libcuda+0x481360` as the `NV_ESC_RM_ALLOC` class; host capture shows `class=0xc5b5` | | `+0x10` | engine caps | for copy records: the CE capability bitmask; **bit 0 (`CE_GRCE`) marks the async/GRCE copy engine**. A copy record with bit 0 set is also copied to the *secondary record* slot and does not take a normal CE slot. | builder tests `+0x10 & 1` and copies to the secondary; live caps `0xbc`/`0xfc` (bit 0 clear) | Secondary record: the GR context registers a second engine from a dedicated 0x14-byte slot (same layout) that is filled from the GRCE-capable copy record and then normalized to kind `2` and the async copy class. GR's own record is larger (a 0x20-byte slot with an unrecovered tail at `+0x10..+0x1f`). **How this explains the blocker.** The builder derives the records from the engine list the driver returns for `GET_ENGINES_V2`; the guest list has 6 entries (GR + CE0..4) because the class-database filter drops `COPY14`. With no `COPY14` entry there is no copy record with the `CE_GRCE` capability bit set, so the secondary record slot is never filled and the walker sees class `0`. libcuda then allocates the engine object with class `0`, the escape path rewrites that to `NV01_ROOT_CLIENT` (`0x41`), the allocation fails with status `0x19`, the context rolls back and `cuCtxCreate` returns 999. In other words: the guest is not merely missing one entry in a list — it is missing the *async copy engine record that seeds the secondary record*. **What the emulation should check first (revised)**: * **`COPY14` in `GET_ENGINES_V2`** — the engine's classDB entry must exist so that `gpuUpdateEngineTable`'s class-list filter keeps OBJCE instance 14. The class/advertised-engine-table consistency is the root cause; the secondary-record zeroing is its symptom. * **CE capabilities.** The copy records' caps word must carry the `CE_GRCE` bit for the async CE. If `COPY14` is restored but its emulated CE caps still return bit 0 clear, the secondary record stays empty and the class-0 allocation returns. * **Consumed-control reply fidelity.** The controls answered generically (the host log lists `0x20803039`, `0x20802a12`, `0x20808162`, `0x2080182b`, `0x20800a70`, `0x20800a93`, …) should be checked for consumed fields, because the walker's engine table is built from data the driver collected earlier (device caps, engine list, class list, CE caps). * The `NV_ESC_RM_ALLOC` parameter block built by `libcuda+0x481360` for class `0xc5c0` (`TURING_COMPUTE_A`) must be answered with a coherent object: echo the request's params back (§3.5) and return a coherent handle; the driver immediately consumes the reply's params and (for other classes) engine ids. ## 4. Navigating the decompiled corpus ``` # by address (file vaddr): grep -n "libcuda+0x45a670" 570.144/run/analysis/decompiled/buckets/*.c # by Ghidra name: awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv # callers of a function (program addresses): grep -P "^0x0055a670\t" 570.144/run/analysis/elf-index/callgraph.tsv # exports and their sizes: 570.144/run/analysis/elf-index/exports.tsv # strings around an address (error messages, table names): awk -F'\t' '$1>="0x..." && $1<="0x..."' 570.144/run/analysis/elf-index/strings.tsv ``` Every bucket header carries `addr=0x file_addr=libcuda+0x` so citations in docs and gdb sessions use the same numbers. ## 5. Known limits * Names are `FUN_` unless recovered; `570.144/annotations/names.tsv` holds our name map (exports are exact, internals grow over time). * The decompiler does not recover struct types; field offsets are named `[obj+0xNN]` and must be interpreted with the descriptions in `docs/01` and `docs/03`. A targeted Decompiler Parameter ID pass (`tools/ghidra/AnalyzeFunctions.java`, see `docs/06` §2d) re-ran over 80 key functions (crash/engine path, resolved wrapper implementations, largest `.text`) and committed their signatures into a project copy (`run/ghidra-typed/`); the re-decompiled bodies are in `run/analysis/typed/` and the before/after report in `run/analysis/typed/paramid_report.tsv`. 16 of 80 bodies changed materially (parameter/type recovery), the rest now carry committed signatures that improve downstream decompilation. * Indirect calls dominate the API dispatch; the callgraph contains direct edges only (77,742). Cross-referencing indirect targets requires reading the wrapper/stub relocations (as in §2) or live traces. * Embedded device code (cubins) is present in `.rodata` and is *not* host code; the corpus only covers `.text`.