# CUDA userspace driver analysis (Open-IOV)

This directory contains the Open-IOV analysis of the NVIDIA CUDA userspace
driver (`libcuda.so.<version>`): what it is, how it is built, what it asks of
the kernel driver and the GPU, how that maps onto Open-IOV's emulation and
passthrough model, and the tooling to repeat the analysis on future driver
versions.

It exists to make virtualized CUDA **work correctly and fast** in Open-IOV
VMs and to keep working across driver/firmware/hardware changes. It is **not**
a CUDA reimplementation and not a way to produce an open-source CUDA: we
disassemble and decompile the vendor userspace driver to understand the
interface it expects, and we publish our own descriptions, tables and tools.
Decompiled output is never committed.

## Start here

> **New here (agent or human)? Read [`READ-ME-FIRST.md`](READ-ME-FIRST.md)
> first** — it is a reading guide: what to read in what order, what each
> document gives you, fast lookup recipes, and the two findings that change
> how you debug (Ghidra image base and the `cuCtxCreate` path — crash solved,
> engine topology blocking).

| Document | What it covers |
|---|---|
| [`docs/01-architecture.md`](docs/01-architecture.md) | the CUDA internal architecture: layers, object model, threads, memory, channels, launches, sync |
| [`docs/02-build-and-execution.md`](docs/02-build-and-execution.md) | how the driver is compiled and how it executes in a process (exports, generated wrappers, entry tables, embedded device code) |
| [`docs/03-driver-and-gpu-interface.md`](docs/03-driver-and-gpu-interface.md) | everything the driver touches below itself: device nodes, RM controls, memory, channels, doorbells, UVM, interrupts |
| [`docs/04-virtualization-and-emulation.md`](docs/04-virtualization-and-emulation.md) | NVIDIA's vGPU split, how Open-IOV differs, anti-fragility, performance, portability, security |
| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash, and the current engine-topology blocker |
| [`docs/06-tooling.md`](docs/06-tooling.md) | the reusable toolchain: ELF toolkit, Ghidra analysis/decompilation, name transfer, version deltas |
| [`docs/07-version-upgrade-playbook.md`](docs/07-version-upgrade-playbook.md) | step-by-step method to re-run the analysis on a new driver version |
| [`docs/08-virtualization-approaches.md`](docs/08-virtualization-approaches.md) | one-page comparison of execution-forwarding options (software emulation, host-RM proxy, direct submission, hybrid) and where the project stands |

Per-version material lives in version directories (`570.144/`), including the
crash-focused debug helpers (`570.144/scripts/`) and the original
`cuCtxCreate` write-up (`570.144/docs/ANALYSIS-cuctxcreate-segv.md`).

## Status (honest)

* **570.144 decompilation: complete for the whole library.** A full Ghidra
  auto-analysis produced 29,494 functions, 659 exported functions, 77,742 call edges and
  a complete decompiled C corpus; the corpus is machine-generated analysis
  output and is therefore kept out of git (see "Working with decompiled
  output" below).
* **Name recovery: partial.** Exports are exact; internal function names are
  recovered from call structure, string usage and cross-version matching. The
  name map is deliberately a separate, reviewable artifact
  (`570.144/annotations/names.tsv`) so it accumulates across versions.
* **Crash path (`cuCtxCreate`): solved.** The alloc reply must echo the
  request's params: the guest driver copies 16 bytes of class params back over
  the caller's 8-byte local, which zeroed the engine hook's saved `rbp` and
  faulted at `libcuda+0x453f90`. Echoing the request (only `status` forced to
  `NV_OK`) makes that copy-back idempotent; `cuCtxCreate` no longer segfaults
  (see `docs/05-...` §3.5).
* **Current blocker: engine topology.** `cuCtxCreate` now returns `rc=999`
  (`CUDA_ERROR_UNKNOWN`); the guest's class-filtered engine list has 6 engines
  (GR + CE0..CE4) where the oracle has 7 — the async copy engine `COPY14`
  (descriptor `0x793ceb0e`, instance 14, client type `0x38`) is missing, so the
  walker reads a zeroed secondary record, the alloc is rejected (class `0x41`,
  st `0x19`) and the context is rolled back (`docs/05` §3.6).
* **Open-IOV relevance: documented.** The emulation boundary, the controls the
  driver issues, and the anti-fragility strategy are in `docs/03`/`docs/04`.

## Layout

```
userspace/cuda/
├── README.md                 this file (index + status)
├── READ-ME-FIRST.md          reading guide for agents/debuggers
├── docs/                     public analysis documents
├── tools/                    reusable, version-agnostic tooling
│   ├── cudatool.py           ELF/export/FDE/wrapper/delta/fdematch toolkit
│   ├── callwalk.py           callgraph callee/caller/path walker
│   ├── name_candidates.py    propose impl names for wrapper callees
│   ├── subsystem_digest.py   per-subsystem call-tree digest
│   ├── diff_corpus.py        diff two decompiled corpus indexes
│   ├── annotate_corpus.py    annotate + rename the decompiled corpus
│   ├── ghidra_full_analysis.sh
│   ├── ghidra_export_index.sh
│   ├── ghidra_batch_decompile.sh
│   ├── ghidra_index_merge.sh
│   ├── ghidra_decompile_at.sh
│   └── ghidra/               Ghidra Java scripts (analysis, decompile, name map)
└── 570.144/
    ├── README.md             version-specific debug tooling (existing)
    ├── docs/                 version-specific write-ups
    ├── scripts/              version-specific helpers
    ├── ghidra/               version-specific Ghidra script
    ├── annotations/          our name map + subsystem index (tracked)
    └── run/                  analysis scratch: extracted binary, Ghidra projects,
                              decompiled corpus (all gitignored)
```

## Working with decompiled output

The decompiled corpus for 570.144 is under
`570.144/run/analysis/decompiled/` (gitignored):

* `buckets/bucket_<start>_<end>.c` — decompiled C for functions in that
  program-address range, in address order, each with a header comment giving
  the function address and `libcuda+0x...` file offset;
* `index/bucket_<start>_<end>.tsv` + merged `decompiled_index.tsv` —
  `addr, file_addr, size, name, file, line, status` for every function.

Search the corpus by address or by name:

```
grep -n "libcuda+0x453ef0" 570.144/run/analysis/decompiled/buckets/*.c
awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv
```

**Address convention (important).** Public docs cite *file virtual addresses*
(`libcuda+0x...`). Ghidra loads these binaries at image base `0x100000`, so
program addresses inside Ghidra/Ghidra output are `file vaddr + 0x100000`.
The tooling translates automatically; if you use raw Ghidra scripts, check
with `tools/ghidra/PrintProgramInfo.java`. An earlier analysis in this tree
did not account for this and decompiled addresses 0x100000 too low — see
`docs/05-570.144-decompilation.md`.

## Rules

* Tracked docs describe components and behaviour only.
* Decompiled/disassembled vendor output is never committed (see
  `.gitignore`); we cite `libcuda+0x...` offsets and describe behaviour.
* Every finding lands in `docs/` (structured) and, when deep, `wiki/`.