userspace/cuda: add CUDA userspace driver analysis, tooling and name maps

Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs:

* docs/01-08: architecture (layers, object model, threads, sync, memory,
  channels, launch), build and execution model, the driver/kernel/GPU
  interface contract, virtualization approaches, the 570.144 decompilation
  map with the corrected libcuda crash path and the current engine-topology
  blocker, tooling, and a version-upgrade playbook.
* tools/: version-agnostic toolchain - stdlib ELF/interface toolkit
  (exports, unwind-table function map, wrapper resolution, version delta,
  crash address decode), callgraph walker, corpus annotator, and Ghidra
  analysis/decompilation/name-map scripts with resumable parallel drivers.
* 570.144/annotations/: our name map (exports + resolved wrappers + verified
  internals) and subsystem index, tracked so naming work accumulates.
* READ-ME-FIRST.md: reading guide with the fast lookup recipes.

The full decompiled corpus and annotated corpus stay untracked (vendor-
derived, gitignored under the version run/ tree).

8 files changed, 1657 insertions, 0 deletions

diff --git a/userspace/cuda/docs/01-architecture.md b/userspace/cuda/docs/01-architecture.md
new file mode 100644
index 0000000..2cde42f
--- /dev/null
+++ b/userspace/cuda/docs/01-architecture.md
@@ -0,0 +1,302 @@
+# 01 — CUDA userspace driver architecture
+
+This document describes the *userspace* CUDA driver (`libcuda.so`), the layer
+that applications and language runtimes talk to. Kernel drivers, GSP firmware
+and the GPU are covered only as the interfaces this library consumes.
+
+Everything below is established from decompilation of shipped binaries,
+open-source study and live traces. Addresses are file virtual addresses
+(`libcuda+0x...`) for driver **570.144** (CUDA 12.8, build id
+`c0e1e77ea249944ac4fd43faebed9498a729606a`, md5
+`ed5b2fc0044df6052df1d5113754c499`).
+
+---
+
+## 1. What the library is
+
+`libcuda.so.1.1` is a single ~71 MB, fully stripped position-independent
+shared object. For 570.144:
+
+| Property | Value |
+|---|---|
+| Exported functions | 659 (public CUDA driver API plus internal/legacy entry points) |
+| Imported functions | 243 (libc/libpthread/libdl/librt only — no other NVIDIA userspace dependency for the CUDA path) |
+| `.text` | `0x179ec0 .. 0xe94c63` (13.7 MB) |
+| Functions found by full analysis | 29,494 (28,925 have unwind tables) |
+| Static data | `.rodata` 53 MB (error strings, tables, embedded device code), `.data`/`.bss` ~1.4 MB |
+| Build id | `c0e1e7…`; a matching `libcuda.so.1.1.debug` is referenced but not shipped |
+| CUDA version note | `.note.cuda.fwd_compatibility` says CUDA 12.8, supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` |
+
+Two entry surfaces coexist in the same library:
+
+1. **The public driver API** — `cu*` symbols (`cuInit libcuda+0x310060`,
+   `cuCtxCreate_v2 libcuda+0x310360`, `cuMemAlloc_v2 libcuda+0x310b20`,
+   `cuLaunchKernel libcuda+0x36f420`, …).
+2. **The entry-table API** — `cuGetExportTable` (`libcuda+0x350910`),
+   a UUID-keyed lookup that hands internal function tables to the CUDA
+   runtime (`libcudart`), tooling (profiler/debugger/memcheck) and
+   interposition libraries. These tables are versioned by a leading
+   `struct_size` field and are append-only.
+
+The library embeds device code (cubin images for memcpy/memset kernels,
+graph scheduler, syscall trampolines and preemption-handler assembly) in its
+read-only data; at initialization it uploads them through the normal module
+machinery.
+
+### 1.1 The CUDA runtime (`libcudart`) and the driver
+
+The CUDA *runtime* library (`libcudart`) is a separate userspace library that
+applications normally link against; it implements the `cuda*` API (as opposed
+to the driver's `cu*` API) and is statically linked in many test programs
+(including the project's `vector_add` A/B binary). There is no kernel
+interface for it: everything it does bottoms out in `libcuda`. Two facts
+matter for analysis and debugging:
+
+* **`libcudart` calls the driver through the entry tables as well as the
+  public API.** Fatbin loading, primary-context initialization and some
+  launch paths use the UUID tables (see §2.3) for speed and to pass internal
+  parameters the public API cannot express. So when a guest misbehaves in
+  `cuda*` calls, the relevant driver code may be reached through a different
+  door than the public `cu*` entry point.
+* **`libcudart` version and driver version are negotiated.** The driver
+  reports its CUDA version (`cuDriverGetVersion`) and exposes versioned
+  entry points; the runtime selects compatible paths. A guest crash can
+  therefore involve a runtime→driver version negotiation rather than a
+  single API call.
+
+In the Open-IOV guest the runtime is **stock and not our concern to modify**;
+it is, however, part of the A/B surface: the same statically linked binary
+run on the host (real GPU, apt driver) and in the guest isolates whether a
+difference is in the runtime or below it.
+
+## 2. Layer map
+
+The library is strictly layered. Each layer has one job, and the boundary
+between layers is where the interesting interfaces live:
+
+```
+application / libcudart
+        │  public cu* API  or  UUID entry tables
+        ▼
+public API wrappers            generated: per-thread/global checks, tracing,
+        │                      interposition hook table, errors → CUresult
+        ▼
+API layer                      hand-written per domain: TLS context stack,
+        │                      init/state validation, argument checks, one
+        │                      context lock, one call into the engine
+        ▼
+CUDA internal engine            the actual driver: contexts, devices, memory,
+        │                      streams, events, modules, graphs, sync
+        ├───────────────┬───────────────────────────┬───────────────┐
+        ▼               ▼                           ▼               ▼
+driver-model layer    hardware layer            UVM userspace    JIT/ELF
+(channels, allocs,    (pushbuffer/QMD/method    (managed memory, (PTX→SASS,
+ controls; RM client)  encoding per chip)        page migration)  cubin loading)
+        │                       │                     │
+        ▼                       ▼                     ▼
+/dev/nvidia* ioctls      GPU queues/doorbells    /dev/nvidia-uvm
+        │
+        ▼
+kernel driver + GSP-RM + GPU
+```
+
+The names of the layers are descriptive working names.
+
+### 2.1 API layer
+
+The `cu*` exports are **generated**, not hand-written. A master export list
+(name, platform flags, the CUDA version in which the symbol was introduced)
+is turned into:
+
+* the exported symbol table (`cuInit`, `cuCtxCreate`, `cuCtxCreate_v2`,
+  `_ptsz`/`_ptds` per-thread-stream variants, …);
+* a wrapper per function that performs the "driver destroyed" check, optional
+  API serialization, then dispatches through a **global interposition table**
+  so tools can replace any entry point before initialization;
+* a table of tool callbacks (enter/exit events keyed by stable numeric IDs)
+  consumed by the profiling/debugging entry tables;
+* the `cuGetProcAddress` lookup tables, which resolve
+  `(name, cudaVersion, flags)` to the right symbol variant.
+
+The hand-written API layer below the wrappers is uniform:
+
+1. read the per-thread context stack (the TLS "current context");
+2. verify the driver is initialized (`CUDA_ERROR_NOT_INITIALIZED` /
+   `_DEINITIALIZED`) and that the object handles are valid;
+3. validate arguments and version gates;
+4. take exactly one context lock (read or write; recursive RW lock);
+5. call one engine function;
+6. return a `CUresult`; sticky asynchronous errors are checked here too.
+
+This uniformity is what makes the API layer cheap to map: the semantic work
+always happens one layer down, in the engine.
+
+### 2.2 CUDA internal engine
+
+The engine owns all driver state. Its main object graph:
+
+* **Process globals** — initialization state, the single kernel-client handle
+  ("RM client") for the process, device list, handle pools, caches, per-thread
+  context stack.
+* **Device** — chip/arch identity, capability bitfields, limits, hardware
+  layer vtable, driver-model vtable, primary context. One per visible GPU.
+* **Context** — an application's CUDA context: its own memory manager, channel
+  manager, stream manager, semaphore pools, QMD pools, trap handler, ISR
+  thread, profiler state. A context is a *userspace* object; at the kernel
+  level it materializes as channel groups + channels + a VA space, not as a
+  distinct kernel object.
+* **Stream** — an ordered work queue. Each stream owns a small pool of QMDs
+  (launch descriptors) and a marker (see §2.5).
+* **Module/Function** — a loaded cubin image and its kernels; functions carry
+  the compiled metadata needed to build QMDs and constant banks.
+* **Memory objects** — a hierarchy: *memblock* (an allocation from the
+  kernel driver, possibly VA-reserved and mapped), *memobj* (a suballocation
+  of a memblock or a standalone object), plus the pointer/VA heaps that
+  resolve device pointers and unified addresses back to objects.
+* **Events, graphs, libraries** — user-visible objects layered on the same
+  marker/memory machinery.
+
+### 2.3 Driver-model layer
+
+Everything that depends on *which* kernel driver is present goes through a
+function table selected at initialization:
+
+* the **RM backend** (Liberty/vGPU, Linux and Windows datacenter) — the one
+  relevant to Open-IOV;
+* a SoC backend for Tegra-class `nvrm_gpu` stacks;
+* an Amodel backend (in-process simulator);
+* an MPS backend (multi-process service client/server);
+* a WDDM backend (Windows).
+
+The RM backend translates engine operations into *RM controls* and object
+allocations over `/dev/nvidiactl`/`/dev/nvidiaN`. It also owns the
+**driver-ABI adaptation** layer: the userspace driver can talk to kernel
+drivers from several ABI generations (570.144 advertises 470…570); one
+adaptation instance per ABI is compiled into the library and the matching one
+is selected at `cuInit` by brute-force version negotiation.
+
+### 2.4 Hardware layer
+
+Chip-specific behavior — method encodings, QMD layout, semaphore release
+encodings, memcpy engine commands, doorbell format — lives behind a
+per-architecture vtable selected from the device's compute capability. The
+userspace driver *builds* the GPU command streams itself; the kernel driver
+mainly allocates objects, maps memory, schedules channels and services
+interrupts.
+
+### 2.5 Synchronization model (the core mental model)
+
+Three primitives explain almost all CUDA synchronization:
+
+1. **Tracking semaphore** — each channel has a GPU-visible memory location
+   whose value counts completed pushes. The CPU tracks three counters:
+   issued-by-CPU, issued-to-GPU and finished-by-GPU. Channel completion is
+   "semaphore reached value N".
+2. **Marker** — a small set of entries (tracking-semaphore values, QMD launch
+   IDs, CPU semaphores, UVM semaphores) that together describe "all work
+   before this point". Every stream operation appends to / waits on markers;
+   stream sync, event record and stream-wait-event are marker operations.
+3. **QMD semaphore** — each launch descriptor (QMD) carries its own completion
+   semaphore, released by the GPU when the grid completes. Markers reference
+   `(QMD, launch id)` and the driver compares the QMD semaphore payload.
+
+Blocking waits are implemented with an ISR thread per context plus a DPC
+queue; a "blocking sync" fast path lets the driver park a thread on a GPU
+event rather than spin.
+
+### 2.6 Memory model
+
+* Device pointers come from a per-context address-space manager with four
+  views: 32-bit/40-bit device pointers, full device VA, host VA and unified
+  VA (UVA). Unified addresses are resolved by a process-global registry, so
+  any context can look up a pointer allocated by another.
+* Allocation is a two-step design: reserve VA, then back it. Backing comes
+  from kernel allocations (vidmem/sysmem), imported handles (IPC), host
+  registrations, or the UVM driver (managed memory).
+* Managed memory and pageable-memory access are delegated to the UVM
+  userspace library/driver, with CUDA maintaining a stream-dependency DAG so
+  UVM can migrate pages correctly (see §2.7).
+* Copies are dispatched by a policy engine: host↔host in the CPU, small
+  host→device inline in the compute pushbuffer, otherwise staged through
+  pinned bounce buffers or sent to a copy engine channel. The choice depends
+  on size, alignment, engine availability and whether UVM is active.
+
+### 2.7 UVM integration
+
+Managed memory is a collaboration: CUDA owns streams and knows the dependency
+order; UVM owns page placement and migration. CUDA registers each channel and
+each managed allocation with UVM, tracks stream dependencies in a DAG
+(`stream → node → event`), and converts user synchronizations
+(`cuStreamSynchronize`, event waits) into UVM DAG syncs. If UVM is
+unavailable or unlicensed (vGPU profiles without UVM), the driver falls back
+to zero-copy-style behavior.
+
+### 2.8 JIT and module loading
+
+`cuModuleLoad*` accepts fatbins, PTX, cubins and (newer) libraries. The path
+is: extract the right device image for the target ISA; JIT-compile PTX if
+needed (the embedded PTX compiler is loaded as a separate library); parse and
+relocate the ELF; register kernels; apply workarounds; make constant banks and
+texture/surface bindings; then upload through the channel machinery. Kernel
+launch subsequently builds a QMD referencing the loaded code and constant
+banks, and submits it with a PCAS ("prepare/switch") method stream on a
+compute channel.
+
+## 3. Execution model inside the process
+
+* **Threads.** Every API call runs on the caller thread. The ISR thread per
+  context services interrupts/DPCs; a per-context boost thread handles
+  drop-to-idle clock management. Lock ordering is globally defined and
+  enforced in debug builds, which is why deadlock analysis needs the lock
+  order list rather than guesswork.
+* **Context stack.** The "current context" is per thread, with a stack
+  (`cuCtxPushCurrent`/`PopCurrent`). Most calls implicitly use the top of the
+  stack; the primary context is retained on first use.
+* **Lazy initialization.** `cuInit` initializes globals and enumerates
+  devices but creates no context. Device enumeration itself performs RM
+  queries; the first context creates the bulk of the resources (channels,
+  semaphores, QMD pools, ISR thread, UVM registration).
+* **Async errors.** GPU faults surface asynchronously: the driver checks
+  channel error notifiers when synchronizing or when a marker reports error,
+  and returns sticky errors from subsequent API calls.
+
+## 4. The 570.144 binary in one screen
+
+| Object | Export | File vaddr |
+|---|---|---|
+| initialization | `cuInit` | `0x310060` |
+| device enumeration | `cuDeviceGet` / `cuDeviceGetCount` | `0x3100a0` / `0x3100c0` |
+| context | `cuCtxCreate_v2` / `_v3` | `0x310360` / `0x310380` |
+| module load | `cuModuleLoadData` / `Ex` | `0x3106e0` / `0x310700` |
+| library API | `cuLibraryLoadData` | `0x3108e0` |
+| memory | `cuMemAlloc_v2` / `cuMemFree_v2` | `0x310b20` / `0x310b60` |
+| VMM API | `cuMemAddressReserve` / `cuMemCreate` / `cuMemMap` / `cuMemSetAccess` | `0x330e10`…`0x330ef0` |
+| stream/event | `cuStreamCreate` / `cuStreamSynchronize` / `cuEventRecord` | `0x331370` / `0x36f3c0` / `0x36f3e0` |
+| launch | `cuLaunchKernel` / `_ptsz` | `0x36f420` / `0x331a50` |
+| copy | `cuMemcpyHtoD_v2` / `cuMemcpyDtoH_v2` | `0x36eda0` / `0x36edc0` |
+| async alloc | `cuMemAllocAsync` | `0x36fa80` |
+| dispatch | `cuGetProcAddress` / `cuGetExportTable` | `0x36fb20` / `0x350910` |
+
+The exports are only 24–30-byte stubs; the work happens in the internal
+functions reached from them. Full call chains for the core paths (init,
+device attach, context create, memory, module load, launch, sync, export
+tables) are in `docs/05-570.144-decompilation.md` and the machine-readable
+index `570.144/run/analysis/elf-index/callgraph.tsv`.
+
+## 5. Why this architecture is good news for Open-IOV
+
+1. **One narrow kernel interface.** Everything below the engine funnels
+   through the RM backend: object allocations, controls, memory mappings and
+   the `/dev/nvidia*` ioctls. That is the surface Open-IOV emulates — not the
+   whole API.
+2. **The driver builds its own command streams.** Kernel launches, copies and
+   syncs are encoded by *userspace* into pushbuffers, GPFIFO entries and QMDs.
+   The kernel driver does not generate commands that must be faithfully
+   emulated; it allocates, maps, schedules and services interrupts.
+3. **Stable, versioned side channels.** Entry tables, `cuGetProcAddress` and
+   the driver-ABI adaptation layer are explicitly versioned, so a version
+   change moves details, not the shape.
+4. **The object model is explicit and enumerable.** The exact kernel objects a
+   context needs are visible in the decompiled call order (see `docs/05`),
+   which is what makes a host-side emulation able to satisfy the driver
+   without touching it.
diff --git a/userspace/cuda/docs/02-build-and-execution.md b/userspace/cuda/docs/02-build-and-execution.md
new file mode 100644
index 0000000..9fc2f02
--- /dev/null
+++ b/userspace/cuda/docs/02-build-and-execution.md
@@ -0,0 +1,190 @@
+# 02 — How the CUDA userspace driver is built and how it executes
+
+This document explains the compilation model behind `libcuda.so` and the
+runtime model of the resulting binary: what is generated, what is compiled in,
+what the loader does, and what the driver does from process start to first
+kernel launch. It is written to be re-derivable on any driver version with the
+tooling in `tools/`.
+
+---
+
+## 1. Build model
+
+### 1.1 One library, many components
+
+The driver is one shared object assembled from several thousand translation
+units. In architecture terms the inputs are:
+
+| Input group | Nature | Notes |
+|---|---|---|
+| API wrappers | **generated** C | one exported function per API name/version variant; adds destroyed-state check, optional serialization, tracing/interposition dispatch |
+| API implementation | hand-written C | one module per API domain |
+| CUDA internal engine | hand-written C | the bulk of the driver |
+| Driver-model backends | hand-written C/C++ | RM/SoC/simulator/MPS/WDDM behind one function-table interface |
+| Hardware layer | hand-written C | per-architecture method/QMD encoders |
+| Device code | **assembly and CUDA C** compiled to cubins, embedded as byte arrays | memcpy/memset kernels, syscall trampolines, trap/exit handlers, preemption, graph scheduler |
+| Export lists | text | symbol name + platform flags + the CUDA version the symbol appeared in |
+| ABI adaptation | **one compiled instance per kernel-driver ABI** | same source compiled repeatedly with different ABI headers and symbol localization |
+| Generators | Perl/Python/Java | produce wrappers, proc-address tables, entry tables, interposition tables, error tables, abstraction-layer headers |
+
+Two consequences matter for analysis:
+
+* **The public ABI is data-driven.** Adding an API function is mostly an entry
+  in a list plus a hand-written implementation; the exported symbol,
+  the tracing callback ID, the `cuGetProcAddress` entry and the interposition
+  slot are generated from that list. A new driver version usually moves this
+  data, not the architecture.
+* **Versioning is explicit.** Symbols carry `_vN` suffixes and a
+  "since CUDA X.Y" attribute; per-thread default stream variants (`_ptsz`)
+  exist for many calls; `cuGetProcAddress` selects by requested version.
+
+### 1.2 Embedded device code
+
+The library carries its own GPU code. Device assembly sources are preprocessed,
+assembled per architecture, converted to C arrays and linked into the library;
+CUDA C device sources are compiled with the toolchain's device compiler. At
+runtime the driver treats them like user modules: it extracts the image for
+the current architecture and loads it through its own module machinery. This
+is why a stripped `libcuda.so` still contains many small ELF images and
+`.nv.*` section names in `.rodata` — those are cubins, not host code.
+
+### 1.3 Build flavors
+
+* `release` separates debug info into a `.debug` file (which NVIDIA does not
+  ship in the driver packages we analyse, hence the fully stripped binary).
+* `develop`/`debug` add logging and assertions; some entry tables (test
+  interfaces) are present only outside public-release builds.
+* A "safety" configuration removes the tooling dispatch entirely (exports call
+  the implementation directly).
+* The kernel-module build of the same sources exists but is not what runs in
+  a guest.
+
+### 1.4 What the produced binary looks like
+
+For 570.144 (`cudatool.py info`):
+
+```
+build id        : c0e1e77ea249944ac4fd43faebed9498a729606a
+exec segment base: 0x179000
+.text           : 0x179ec0 .. 0xe94c64 (13,741,476 bytes)
+exports         : 659        imports: 243 (libc/libpthread/libdl/librt)
+cuda note       : CUDA 12.8, driver ABI [470,535,550,560,565,570], device [1,2,7..14]
+debuglink       : libcuda.so.1.1.debug (not shipped)
+```
+
+Sections of interest:
+
+| Section | Why it matters |
+|---|---|
+| `.dynsym`/`.dynstr` | the 659 exports and 243 imports; no local symbols (stripped) |
+| `.text` | all host code |
+| `.rodata` (53 MB) | error strings, dispatch/version tables, **embedded cubins** |
+| `.data.rel.ro` (~900 KB) | relocated pointer tables (entry tables, vtables, function tables) |
+| `.eh_frame`/`.eh_frame_hdr` | unwind info; also the best stripped-binary function map (28,925 FDEs) |
+| `.note.cuda.fwd_compatibility` | CUDA version + supported kernel-driver ABI list |
+| `.gnu_debuglink` | name/CRC of the (absent) debug file |
+
+## 2. Execution model
+
+### 2.1 Load and initialization
+
+1. The dynamic loader maps the four PT_LOAD segments and relocates
+   `.rela.dyn`/`.rela.plt`. The library has no NVIDIA userspace dependencies
+   for the CUDA path; everything below it is a device file.
+2. C++-style constructors registered in `.init_array` run. They set up global
+   mutexes and install an `atexit`/destructor that marks the driver destroyed.
+3. The first `cuInit` runs the initialization state machine:
+   * initialize TLS and globals; select the driver-model backend;
+   * open the kernel driver's control device and create the process-wide
+     **kernel client** (one per process, not per context);
+   * enumerate and attach GPUs, build the device list (chip, caps, limits,
+     engine lists, memory sizes, BAR info);
+   * initialize the UVM userspace layer if the platform supports it;
+   * load embedded device code caches lazily.
+4. Subsequent API calls use the per-thread context stack; the first
+   `cuCtxCreate` builds the heavy per-context state.
+
+An `LD_PRELOAD` shim cannot observe the driver's kernel traffic: the RM client
+issues **raw `syscall(SYS_ioctl)`**, not libc `ioctl`. Instrumentation is done
+with `ptrace` (see `tools/guest-diag/ioctl_trace.c` in the parent tree) or
+with kernel-side tracing.
+
+### 2.2 Interposition and tooling dispatch
+
+Every exported call goes through a global function table before reaching the
+implementation. A tool library can register an interposition layer *before*
+`cuInit` and receive the original pointers to chain to. This is the sanctioned
+mechanism used by profilers/debuggers, and it is also a clean hook point for
+A/B instrumentation in the guest.
+
+### 2.3 Entry tables
+
+`cuGetExportTable(uuid)` returns pointers to internal tables for:
+
+* the CUDA runtime library (fatbin loading, primary-context initialization,
+  hidden module-load paths, kernel launch with internal parameters);
+* tooling (profiling, debugging, memory checking, launch tracking);
+* debugger support (context-local storage, trap handlers);
+* interop and special platforms (video, NVN, OpenCL bridges).
+
+Tables are identified by 128-bit UUIDs and start with a size field, so old
+clients and new drivers coexist. This is a key fact for emulation: the guest
+runtime/tools may call *through these tables* instead of the public API, and
+all of it still resolves to the same internal engine.
+
+### 2.4 Threading and process model
+
+* API calls are caller-threaded; each context has a recursive RW lock.
+* One ISR thread per context handles interrupts/DPC callbacks.
+* One boost thread per context handles drop-to-idle clock policy.
+* MPS (multi-process service) inserts a client/server IPC layer *below* the
+  normal API: clients share the server's channels; the wire format is a
+  private socket protocol, not the public API.
+* Inter-process memory sharing uses kernel-driver exported handles and
+  (optionally) POSIX file descriptors; CUDA's own IPC mem handles are an RM
+  object plus a key.
+
+### 2.5 What the driver touches at runtime (overview)
+
+| Resource | Access | Purpose |
+|---|---|---|
+| `/dev/nvidiactl` | ioctl | control device: version negotiation, client creation, GPU attach, OS events, exported-object fds |
+| `/dev/nvidiaN` | open/ioctl/mmap | per-GPU: object alloc/free, controls, memory alloc/map, interrupts |
+| `/dev/nvidia-uvm` | open/ioctl | managed memory, page migration, channel registration (via UVM userspace API) |
+| BAR regions | mmap | user-mode doorbells (USERD), interrupt tree/usermode region, framebuffer apertures |
+| sysmem | mmap/alloc | command buffers, semaphores, page tables (when the driver owns them) |
+| shared memory / sockets | IPC | MPS, IPC events/memory |
+
+The exact ioctl vocabulary observed from the 570.144 guest is enumerated in
+`docs/03-driver-and-gpu-interface.md`.
+
+## 3. Running in a guest (what Open-IOV sees)
+
+* The library is **unmodified** in the guest; it links against the guest's
+  kernel driver (`nvidia.ko` + `nvidia-uvm.ko`) exactly as on bare metal.
+* The kernel driver is also unmodified; it talks to the emulated PCI function
+  and the GSP-RM interface that Open-IOV emulates.
+* Therefore, from libcuda's point of view, the guest is a normal NVIDIA
+  machine whose kernel driver has certain capabilities and whose GPU answers
+  RM controls. The crash class we chase (`cuCtxCreate` segfault) is a
+  *userspace* consequence of *kernel/GSP* replies that differ from bare metal.
+* The library's own state is therefore the best available oracle: when the
+  driver misbehaves, comparing its state (or its control traffic) against a
+  bare-metal A/B run localizes the emulation gap. The A/B harness in
+  `tools/cuda-ab/` does this at the workload level; `docs/05` does it at the
+  function level.
+
+## 4. Reproducing the build/execution analysis for a new version
+
+```
+tools/cudatool.py info       <libcuda.so.NEW>     # size, exports, ABI note
+tools/cudatool.py exports    <libcuda.so.NEW>     # symbol map
+tools/cudatool.py functions  <libcuda.so.NEW>     # FDE function map
+tools/cudatool.py delta <old> <new>               # interface-level diff
+tools/ghidra_full_analysis.sh <libcuda.so.NEW>    # full auto-analysis
+tools/ghidra_export_index.sh ...                  # function/call/string index
+tools/ghidra_batch_decompile.sh ...               # decompiled corpus
+```
+
+See `docs/06-tooling.md` for details and `docs/07-version-upgrade-playbook.md`
+for the full procedure.
diff --git a/userspace/cuda/docs/03-driver-and-gpu-interface.md b/userspace/cuda/docs/03-driver-and-gpu-interface.md
new file mode 100644
index 0000000..4c4caa2
--- /dev/null
+++ b/userspace/cuda/docs/03-driver-and-gpu-interface.md
@@ -0,0 +1,276 @@
+# 03 — What the CUDA userspace driver asks of the kernel and the GPU
+
+This document is the emulation contract: everything `libcuda` needs from
+below itself. It is the checklist Open-IOV's host-side emulation has to
+satisfy for unmodified CUDA to run, and the map used when a guest call fails.
+
+Sources: live ioctl traces from the guest (ptrace tracer), host-side GSP-RM
+emulation logs, and decompilation of the RM backend in `libcuda` 570.144. The
+ioctl numbers below were observed on the wire; the object classes and control
+IDs are quoted from the trace and from public CUDA/RM headers.
+
+---
+
+## 1. Device nodes and the ioctl vocabulary
+
+| Node | Used for |
+|---|---|
+| `/dev/nvidiactl` | process-wide control: version negotiation, client creation, GPU attach, OS events, exported-object file descriptors |
+| `/dev/nvidiaN` | per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation |
+| `/dev/nvidia-uvm` | managed memory and page migration (UVM userspace API) |
+| `/dev/nvidia-vgpuN` | vGPU plugin node (only inside NVIDIA's own vGPU stack; not used by stock CUDA paths) |
+
+Control-device ioctls (`_IOWR('F', cmd, …)` with magic `'F'`):
+
+| cmd | Name | Purpose |
+|---|---|---|
+| 200 (`0xc8`) | `NV_ESC_CARD_INFO` | enumerate GPUs and minors |
+| 201 (`0xc9`) | `NV_ESC_REGISTER_FD` | bind a per-GPU fd to the control fd |
+| 206/207 | `NV_ESC_ALLOC_OS_EVENT` / `FREE_OS_EVENT` | OS event objects (blocking waits, GPU events) |
+| 209 (`0xd1`) | `NV_ESC_STATUS_CODE` | driver status on open failure |
+| 210 (`0xd2`) | `NV_ESC_CHECK_VERSION_STR` | RM/driver version negotiation |
+| 211 (`0xd3`) | `NV_ESC_IOCTL_XFER_CMD` | inter-GPU ioctl forwarding |
+| 212 (`0xd4`) | `NV_ESC_ATTACH_GPUS_TO_FD` | attach GPUs to an fd (exported objects) |
+| 213 (`0xd5`) | `NV_ESC_QUERY_DEVICE_INTR` | interrupt query |
+| 214 (`0xd6`) | `NV_ESC_SYS_PARAMS` | system parameters |
+
+RM escapes (same magic, used on control and per-GPU fds):
+
+| cmd | Name | Purpose |
+|---|---|---|
+| 0x27 | `NV_ESC_RM_ALLOC_MEMORY` | allocate memory (older path; NVOS02) |
+| 0x28 | `NV_ESC_RM_ALLOC_OBJECT` | allocate object (NVOS05) |
+| 0x29 | `NV_ESC_RM_FREE` | free object (NVOS00) |
+| 0x2a | `NV_ESC_RM_CONTROL` | issue an RM control (NVOS54) |
+| 0x2b | `NV_ESC_RM_ALLOC` | allocate object (NVOS21/NVOS64) — **the main alloc path** |
+| 0x34 | `NV_ESC_RM_DUP_OBJECT` | duplicate/import an object |
+| 0x35 | `NV_ESC_RM_SHARE` | share object |
+| 0x41 | `NV_ESC_RM_IDLE_CHANNELS` | idle channels |
+| 0x4a | `NV_ESC_RM_VID_HEAP_CONTROL` | heap allocation (NVOS32) |
+| 0x4e / 0x4f | `NV_ESC_RM_MAP_MEMORY` / `UNMAP_MEMORY` | map memory into the process (NVOS33/NVOS34) |
+| 0x57 / 0x58 | `NV_ESC_RM_MAP_MEMORY_DMA` / `UNMAP_MEMORY_DMA` | map memory into a device VA space (NVOS46/NVOS47) |
+| 0x5c / 0x5d | `NV_ESC_RM_EXPORT_OBJECT_TO_FD` / `IMPORT_OBJECT_FROM_FD` | cross-process object sharing |
+| 0x52 | `NV_ESC_RM_GET_EVENT_DATA` | read event payload (NVOS41) |
+| 0x32/0x33 | `NV_ESC_RM_CONFIG_GET/SET` | registry-style configuration |
+
+In a trace, the ioctl request word encodes the escape and the size; e.g.
+`_IOWR('F',0x2b,48)` = `NV_ESC_RM_ALLOC` with a 48-byte payload. Seeing this
+request in a crash register dump (`rsi=0xc030462b`) identifies "driver was
+allocating an RM object" — that is exactly what happens at the `cuCtxCreate`
+crash (see `docs/05`).
+
+## 2. The RM object model as CUDA uses it
+
+Every RM call names a **class**. The classes CUDA allocates, in the order
+observed for 570.144 on an emulated TU104:
+
+| Class | Name (public RM header) | When |
+|---|---|---|
+| `0x0` | `NV01_ROOT` | process client (once, at `cuInit`) |
+| `0x80` | `NV01_DEVICE_0` | per GPU, at attach |
+| `0x2080` | `NV20_SUBDEVICE_0` | per GPU/subdevice |
+| `0x40` | `NV01_EVENT` | events (several) |
+| `0xa06c` | `KEPLER_CHANNEL_GROUP_A` | context channel group (TSG) |
+| `0x9067` | channel-group/runlist object | context |
+| `0x3e` | `NV01_MEMORY_*` | memory objects |
+| `0xc46f` | `TURING_CHANNEL_GPFIFO_A` | the channel itself |
+| `0xc5c0` | `TURING_COMPUTE_A` | compute engine object bound to the channel |
+| `0xc5b5` | (compute class, second engine) | used by the host A/B run for a CE-style object |
+| `0xa080` | `KEPLER_DEVICE_VGPU` | vGPU capability query (vGPU only) |
+| `0x90f1`-family | VA space (`FERMI_VASPACE_A` etc.) | memory model |
+
+`cuCtxCreate`'s observed ioctl sequence (guest, 570.144):
+
+```
+NV_ESC_RM_ALLOC class=0x0      (NV01_ROOT)
+NV_ESC_RM_ALLOC class=0x80     (NV01_DEVICE_0)
+NV_ESC_RM_ALLOC class=0x2080   (NV20_SUBDEVICE_0)
+NV_ESC_RM_ALLOC class=0x40     (events)
+NV_ESC_RM_ALLOC class=0xa06c   (KEPLER_CHANNEL_GROUP_A)
+NV_ESC_RM_ALLOC class=0x9067   (channel group object)
+NV_ESC_RM_ALLOC class=0x3e     (memory)
+NV_ESC_RM_ALLOC class=0xc46f   (TURING_CHANNEL_GPFIFO_A)      <- channel
+NV_ESC_RM_CONTROL cmd=0x80170d (NV0080_CTRL_FIFO_GET_CHANNELLIST)
+NV_ESC_RM_ALLOC class=0xc5c0   (TURING_COMPUTE_A)             <- last alloc
+[engine registration attempt; pre-fix this segfaulted — now rc=999, see docs/05 §3.5-3.6]
+```
+
+The first two allocs (`NV01_ROOT`, `NV01_DEVICE_0`) actually happen at
+`cuInit`/device attach; the trace window shows them repeated per client setup.
+
+## 3. RM controls the CUDA path uses
+
+Controls are `NV_ESC_RM_CONTROL` with a 32-bit command ID and a fixed-size
+parameter block. Families relevant to CUDA bring-up (IDs in the low 32 bits
+are `(class << 16) | cmd`; the list below is from traces and decompiled call
+sites, not exhaustive):
+
+* **Device/GPU info** — `NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`
+  (`0x00800089`), `NV2080_CTRL_CMD_GPU_GET_INFO`, `GPU_GET_ENGINES`,
+  `GPU_GET_CLASSLIST`, `GPU_GET_SIMULATION_INFO`, `FB_GET_INFO`,
+  `BUS_GET_INFO`, `BUS_GET_PCI_INFO`, `BUS_GET_PCI_BAR_INFO`,
+  `MC_GET_ARCH_INFO`, `GR_GET_INFO`, `GR_GET_GPC_MASK`, `GR_GET_TPC_MASK`,
+  `GR_GET_GLOBAL_SM_ORDER`, `FB_GET_LTC_INFO_FOR_FBP`, `CE_GET_CAPS`,
+  `GSP_GET_FEATURES`, `GPU_GET_SW_FEATURES`, `GPU_GET_MEMOP_ENABLE`,
+  `FB_GET_CAPS`, `GR_GET_CAPS_V2`.
+* **FIFO/channels** — `NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` (`0x80170d`),
+  `NV0080_CTRL_CMD_FIFO_GET_CAPS`, `NVA06F/NVA06C_CTRL_CMD_GPFIFO_SCHEDULE`,
+  `SET_ERROR_NOTIFIER`, `NVA06C_CTRL_CMD_SET/GET_TIMESLICE`,
+  `NV2080_CTRL_CMD_FIFO_DISABLE_CHANNELS`, per-class
+  `GET_CLASS_ENGINEID` (e.g. `NV906F`, `NVA06F`-family),
+  `NVC36F_CTRL_CMD_GPFIFO_GET_WORK_SUBMIT_TOKEN`.
+* **GR/context** — `GR_CTXSW_PM_MODE`, `GR_CTXSW_SMPC_MODE`,
+  `GR_SET_CTXSW_PREEMPTION_MODE`, `GR_PC_SAMPLING_MODE`,
+  `GR_GET_CTX_BUFFER_SIZE`, `FLCN_GET_CTX_BUFFER_SIZE`.
+* **Memory** — `FB_GET_INFO` (BAR1 info), `NV0000_CTRL_CMD_OS_UNIX_*`
+  (flush user cache, exported objects), `NV0041_CTRL_CMD_GET_SURFACE_INFO`,
+  `NV0000_CTRL_CMD_CLIENT_GET_ADDR_SPACE_TYPE`.
+* **Events/interrupts** — `NV90E6_CTRL_CMD_MASTER_GET_VIRTUAL_FUNCTION_ERROR_CONT_INTR_MASK`
+  (interrupt-tree setup), `NV83DE_CTRL_CMD_DEBUG_*` (SM error states, debug
+  mode), `RC_ENABLE/DISABLE/SOFT_DISABLE_WATCHDOG`, `RELEASE_WATCHDOG_REQUESTS`.
+* **vGPU/licensing** — `NVA080_CTRL_CMD_VGPU_GET_CONFIG` (`0xa0800301`),
+  `NVA080_CTRL_CMD_VGPU_SET_LICENSE_INFO`,
+  `NV2080_CTRL_CMD_GPU_GET_LICENSABLE_FEATURES`,
+  `GPU_GET_GRID_UNLICENSED_STATE_MACHINE_INFO` (`0x2080018c`).
+* **Perf/clocks** — `GPU_GET_COMPUTE_POLICY_CONFIG`, boost/sync-boost
+  controls, `NV2080_CTRL_CMD_GPU_ACQUIRE/RELEASE_COMPUTE_MODE_RESERVATION`.
+
+Traces show the driver tolerating unknown-but-successful controls: it only
+fails when a *specific* reply field is consumed and wrong. This is why the
+emulation can start with generic success and then bisect the consumed fields
+(documented in `docs/04` as the "reply-fidelity ladder").
+
+## 4. Memory: allocation, mapping, VA
+
+CUDA's memory path uses three layers:
+
+1. **VA reservation** — `NV_ESC_RM_VID_HEAP_CONTROL` with a virtual
+   `NVOS32_FUNCTION_ALLOC_SIZE` request (or no reservation at all when UVM
+   owns the VA space).
+2. **Backing allocation** — vidmem/sysmem `NVOS32` allocation with attributes:
+   location (`VIDMEM`/`PCI`/`SYSTEM`), page size, coherency, compression,
+   protection, fixed physical address (if requested), or pinning an existing
+   host address (`NV01_MEMORY_SYSTEM_OS_DESCRIPTOR`).
+3. **Mapping** — `NV_ESC_RM_MAP_MEMORY` (`NVOS33`) to map into the process
+   (for host-visible/CPU access, flags include MAP_FIXED, RESERVE_ON_UNMAP,
+   DIRECT for incoherent sysmem) and `NV_ESC_RM_MAP_MEMORY_DMA` (`NVOS46`) to
+   map into a device VA space (flags include RW/RO, snoop, fixed DMA offset,
+   page size, peer flags).
+
+Related classes: `FERMI_VASPACE_A` (VA space; one is shared with UVM, one
+legacy, optionally one for fabric-local aperture), `NV01_MEMORY_FABRIC*` /
+`NV_MEMORY_FABRIC*` (NVLink/fabric), `NV01_MEMORY_SYSTEM_OS_DESCRIPTOR`
+(pinned host memory), `NV01_MEMORY_LOCAL_USER` (vidmem).
+
+For Open-IOV the practical summary is: **all guest memory that CUDA uses is
+either an RM allocation backed by emulated VRAM/BAR1, or pinned guest RAM that
+the GPU must be able to DMA to.** The project's current model backs VRAM with
+host RAM and executes push-buffer work in the host (see the parent handoff).
+
+## 5. Channels, GPFIFO, USERD and doorbells
+
+A CUDA channel is a GPFIFO (array of pushbuffer descriptors) plus one or more
+engine objects:
+
+* GPFIFO memory: allocated by CUDA (vidmem or sysmem), passed to the kernel
+  driver at channel creation with its offset and entry count, together with
+  an error notifier object.
+* **USERD**: on Volta and later, CUDA allocates the user-mode doorbell/control
+  area itself and passes its handles to the channel allocation; older
+  generations map the kernel-provided USERD. On Hopper, a second BAR1-mapped
+  usermode region is used for doorbells.
+* **Doorbell**: submission writes the GPFIFO put pointer into USERD and rings
+  a doorbell token (obtained via `NVC36F_CTRL_CMD_GPFIFO_GET_WORK_SUBMIT_TOKEN`
+  on newer ABIs, or a static token on older ones) at a token/offset the
+  channel allocation returned.
+* **Pushbuffers**: command streams in vidmem or sysmem; a push is a sequence
+  of methods (`SET_OBJECT`, engine methods, semaphore releases) terminated by
+  a tracking-semaphore release and a new GPFIFO entry.
+* **Context share (Volta+)**: subcontexts are created by allocating a
+  `FERMI_CONTEXT_SHARE_A` object with the VA space handle and subcontext id;
+  channels then reference it instead of the VA space directly.
+
+Open-IOV relevance: the host emulation sees the GPFIFO and pushbuffer memory
+the guest writes. The project already decodes push buffers as method streams
+and executes memory ops (`kernel/gsp_sema_service.c`); this section defines
+the contract those streams follow.
+
+## 6. Synchronization objects below the API
+
+* **Semaphores** — plain 32/64-bit memory values, allocated by CUDA in
+  GPU-visible memory. The GPU releases them (semaphore release methods or
+  inline memcpy of the payload), the CPU polls or waits. Tracking semaphores
+  count completed pushes; QMD semaphores count completed grids.
+* **Notifiers** — 4 KB error-notifier memory per channel; the kernel driver
+  writes a notification record on channel errors, which CUDA reads when
+  synchronizing to map to `CUDA_ERROR_LAUNCH_*`.
+* **OS events** — kernel objects (`NV_ESC_ALLOC_OS_EVENT`) used to wake
+  blocked threads; GPU events bind an engine event to an OS event so an
+  interrupt can signal user space.
+* **Interrupts** — the kernel driver owns the GPU interrupt; CUDA's ISR
+  thread is woken through the OS event/monitored-fence mechanism and then
+  walks channel state, updates blocking-sync accounting and runs DPCs.
+* **Trap handler** — for debuggers and `device-side assert`, CUDA installs a
+  trap handler (embedded device code) and reads a trace buffer in GPU memory.
+
+## 7. UVM interface
+
+When UVM is available, the userspace driver calls the UVM API (through
+`/dev/nvidia-uvm`):
+
+* set the expected kernel driver version (forward compatibility);
+* initialize UVM (sharing mode for MPS, HMM/pageable-memory options);
+* register each GPU and each VA space;
+* register each channel with its platform parameters (the RM handles and
+  context-buffer VA/size the kernel driver provided);
+* allocate/register managed allocations, prefetch/advice calls;
+* semaphores: UVM semaphore acquire/release are pushed into the same streams.
+
+If UVM is unavailable or the vGPU profile lacks UVM/fault capability, CUDA
+disables kernel-mode UVM and degrades to zero-copy behavior.
+
+## 8. GSP-RM in the picture
+
+Modern drivers run the resource manager on the GSP (a RISC-V core on the
+GPU). From the userspace driver's perspective this changes nothing: it still
+issues RM ioctls. Inside the kernel driver, controls become GSP-RM RPCs; in
+the Open-IOV emulation, the host answers those RPCs. The host log for the
+`cuCtxCreate` window shows, in order:
+
+```
+CE_GET_FAULT_METHOD_BUFFER_SIZE -> 12288
+GPU_PROMOTE_CTX (0x2080012b, 560 B) -> generic SUCCESS
+TURING_CHANNEL_GPFIFO_A alloc, TURING_COMPUTE_A alloc
+```
+
+The userspace failure now surfaces after the engine registration attempt: with
+the alloc reply fixed (`docs/05` §3.5) the walker reads a zeroed secondary
+engine record and `cuCtxCreate` returns `rc=999` (`docs/05` §3.6). The
+function-level path is in `docs/05`; the emulation-side question is what input
+builds the driver's class-filtered engine list.
+
+## 9. Emulation checklist (derived from the above)
+
+For an unmodified CUDA 12.x userspace driver in a guest to reach a working
+context, the host emulation must provide, at minimum:
+
+1. **Control device semantics** — version negotiation, client creation, GPU
+   attach, OS events, per-GPU fds.
+2. **Object allocation** for the classes in §2, with plausible sizes/handles,
+   and successful `NV_ESC_RM_FREE`.
+3. **Device/engine topology** — a consistent engine list and class list
+   (`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`, static info) matching what the
+   caps/controls claim; per-engine records must be self-consistent (see the
+   engine-descriptor discussion in `docs/05`).
+4. **Memory** — VA spaces, vidmem/sysmem allocations with the requested
+   attributes, device and host mappings, BAR1 info.
+5. **Channels** — GPFIFO alloc with notifier and USERD handles, engine object
+   alloc, scheduling, channel ids, doorbell tokens/offsets.
+6. **Interrupt/event plumbing** — interrupt tree or usermode region, OS
+   events, error notifiers.
+7. **Reply fidelity for consumed controls** — the small set of controls whose
+   *values* (not just status) the driver reads: memory sizes, engine ids,
+   class engine ids, BAR1 info, channel ids, version info, UVM caps.
+8. **Push-buffer execution** — the GPU work encoded by userspace must actually
+   happen (memsets, copies, semaphore releases, kernel grids) or the driver
+   will hang in synchronization rather than fail fast.
diff --git a/userspace/cuda/docs/04-virtualization-and-emulation.md b/userspace/cuda/docs/04-virtualization-and-emulation.md
new file mode 100644
index 0000000..89adc35
--- /dev/null
+++ b/userspace/cuda/docs/04-virtualization-and-emulation.md
@@ -0,0 +1,244 @@
+# 04 — Virtualizing CUDA: NVIDIA's vGPU model, Open-IOV's model, and how to stay unbreakable
+
+This is the design document for CUDA virtualization in Open-IOV. It answers
+four questions:
+
+1. How does NVIDIA virtualize CUDA in its own vGPU product?
+2. How does Open-IOV's model differ, and why?
+3. How do we stay working across CUDA/driver/GSP/firmware/hardware changes
+   (anti-fragility)?
+4. How do we get performance, portability, security and correctness?
+
+Companion documents: `docs/01-architecture.md` (what the driver is),
+`docs/03-driver-and-gpu-interface.md` (the interface we must satisfy).
+
+---
+
+## 1. NVIDIA's vGPU architecture (behavioural summary)
+
+NVIDIA's vGPU stack splits the driver across the VM boundary. The **guest is
+fully unmodified**: stock `libcuda`, stock `nvidia.ko`, stock `nvidia-uvm.ko`.
+Virtualization is hidden below the RM client interface, using a mediated PCI
+function and a host-side device model:
+
+```
+guest userspace      libcuda (unmodified)  ── /dev/nvidia* ioctls ──┐
+guest kernel         nvidia.ko, nvidia-uvm.ko (unmodified)         │
+                     │                                             │
+                     │  emulated PCI function: config space,       │
+                     │  BAR0/BAR1 MMIO, doorbells, MSI-X, VGA      │
+                     ▼                                             │
+hypervisor           mediated device (VFIO/mdev on KVM, VMBus on    │
+                     Hyper-V, device model on Xen/VMware)          │
+                     │                                             │
+host kernel          vGPU kernel module (VFIO side)                │
+                     │  ioctls: create/destroy, pin/unpin pages,   │
+                     │  reg access, mmio mappings, interrupt       │
+                     │  injection, migration state                 │
+                     ▼                                             │
+host userspace       vGPU manager daemon + plugin (VMIOP)  ◄───────┘
+                     │  NvRm API
+                     ▼
+host kernel + GPU    host nvidia.ko / GSP-RM / physical GPU
+```
+
+The pieces that make this work:
+
+* **A mediated PCI function.** The guest sees a GPU with a *vGPU profile*
+  (a slice of framebuffer, a set of engines, display heads, encoder capacity).
+  Config space, BAR sizes, MMIO registers, doorbells and MSI-X are emulated or
+  forwarded; the profile is what licenses and limits the VM.
+* **A guest-host wire protocol (VMIOP) between the guest kernel driver and the
+  host plugin.** The guest kernel driver's MMIO/register accesses arrive at
+  the host plugin as emulation callbacks (space, offset, width, data,
+  cacheability). The plugin either satisfies them from backing memory (with
+  per-register callback gating) or forwards them into host RM.
+* **Guest memory mapping and pinning.** Guest pages are pinned and mapped into
+  the plugin so the physical GPU can DMA into guest RAM; the guest framebuffer
+  is carved from physical VRAM and mapped into the guest's physical address
+  space.
+* **Interrupt injection.** Device interrupts are delivered to the guest as
+  MSI/MSI-X through the hypervisor.
+* **A capability negotiation on the CUDA side.** The CUDA userspace driver
+  detects vGPU (`GPU_GET_VIRTUALIZATION_MODE == VGX`) and queries a vGPU
+  capability block: is CUDA enabled for this profile, is UVM available, are
+  replayable faults available, is GPU-direct RDMA available, is TSG timeslice
+  override available. It then *restricts itself*: no UVM if the profile lacks
+  it, no debugger if disabled, no checkpoint/CRIU, CILP instead of MBP for
+  preemption, and it enforces the GRID license state (unlicensed VMs get
+  `CUDA_ERROR_DEVICE_NOT_LICENSED` or a sleep/timebomb behavior).
+* **Real objects on the host.** Channels, VA spaces, memory objects, engine
+  contexts and command execution are *real* host RM objects bound to real
+  channels; command streams are passed through and executed by the physical
+  GPU. What is virtualized is the *view* (profiles, MMIO, interrupts,
+  migration), not the execution.
+
+Two design consequences worth naming:
+
+* The guest driver is never patched; the virtualization is data (capabilities,
+  object parameters, emulated registers) plus a host device model.
+* The split is at the *kernel-driver ↔ hardware* boundary. Everything above
+  it (CUDA userspace, RM API semantics) is the stock stack.
+
+The exact plugin interfaces, capability bits, object classes and migration
+stages are vendor-internal and not needed to build an interoperable
+alternative.
+
+## 2. Open-IOV's model: emulate the interface, pass through the work
+
+Open-IOV makes the same fundamental choice as NVIDIA — **the guest stays
+stock** — but implements the host side differently:
+
+| Concern | NVIDIA vGPU | Open-IOV |
+|---|---|---|
+| Guest stack | stock | stock (identical constraint) |
+| Device presented | mediated PCI function with a vGPU profile, backed by host RM | emulated PCI function (mdev/VFIO) backed by the host module + GSP-RM emulation |
+| Where commands execute | physical GPU via host RM channels | host executes the work the guest encodes (e.g. memsets/copies/semaphore releases) or passes through, depending on the resource |
+| Host-side control plane | vGPU manager daemon + VMIOP plugin + host RM | `open-iov-vfio.ko` emulating the GSP-RM interface and device model directly in the kernel |
+| Licensing | GRID license server / vGPU licensing; CUDA gated on license state | no GRID licensing; the driver's vGPU detection must be answered in a way that does not gate CUDA (or the guest must not be told it is a licensed-vGPU-only environment) |
+| CUDA userspace | unmodified, capability-limited by vGPU profile | unmodified, no profile limits imposed |
+| Migration | vGPU-specific save/restore protocol with dirty-page tracking | 4-phase migration in the host module (project feature) |
+
+The key architectural difference: **Open-IOV does not need to virtualize the
+GPU's register interface at all if it can emulate the GSP-RM interface the
+kernel driver consumes.** The kernel driver asks the GSP (via RPCs) for
+objects, memory, channels, engines and caps; if the emulation answers those
+RPCs coherently, the driver builds its own command streams and the guest CUDA
+stack works. This moves the emulation boundary *up* from MMIO registers to
+RPC messages, which is far more stable: RPCs are versioned and typed, MMIO
+register maps are not.
+
+Where pass-through is possible (real GPU assigned, SR-IOV, or a future
+split-execution mode), the same userspace stack runs unchanged; the host-side
+work is the only difference. That is the portability goal: the analysis in
+`docs/01`–`docs/03` describes the *interface*, and both emulation and
+passthrough are implementations of it.
+
+## 3. Anti-fragility: how we survive version churn
+
+CUDA, the kernel driver, GSP firmware and GPU architectures all change on
+independent schedules. Our analysis and implementation are designed so that
+change moves *data*, not *architecture*:
+
+1. **Anchor on the stable interfaces.** The kernel-driver interface (RM
+   objects, controls, ioctls) and the userspace driver's internal layering
+   have been stable for many years; the driver-ABI adaptation layer inside
+   `libcuda` exists precisely because NVIDIA itself maintains that stability.
+   We track the interface, not the implementation.
+2. **Prefer behaviour over constants.** Every numeric ID we depend on
+   (classes, controls, capability bits) is versioned in config
+   (`config/versions/*.yaml`, `config/hardware/*.yaml`), with the analysis
+   tooling able to re-derive them from a new binary (`cudatool.py delta`,
+   `docs/07`).
+3. **Never require driver patches.** No guest modification, no userspace
+   shim, no `LD_PRELOAD`. Anything that would break on a driver update is
+   rejected by design. The only supported hook points are the ones NVIDIA
+   itself keeps stable: entry tables, `cuGetProcAddress` versioning, the RM
+   ABI.
+4. **Emulate replies, not side effects.** The host emulation answers RPCs with
+   *coherent data*; it does not fake hardware state that the driver might
+   later cross-check. Where a value is consumed (sizes, IDs, caps), we must
+   supply a consistent value — the crash analysis in `docs/05` is exactly a
+   case of inconsistent data reaching a userspace consumer.
+5. **Name transfer instead of re-analysis.** Function names and structure
+   knowledge are stored in a reviewable name map (`570.144/annotations/`) and
+   can be re-applied to a new version in seconds, then corrected where
+   signatures moved. Analysis effort accumulates across versions.
+6. **Decompose by failure mode.** The analysis is organized by what can break:
+   * *API layer* — new/changed entry points (`delta` detects them);
+   * *engine* — new object types, new control flows;
+   * *driver-model* — new controls/ABI versions;
+   * *hardware layer* — new chips and QMD/method encodings;
+   * *device code* — new syscalls/trampolines embedded in the library;
+   * *UVM* — page-migration and channel registration changes.
+7. **Keep a reference oracle.** The same CUDA workload runs on bare metal
+   (host A/B harness) and in the guest; divergence localizes the break to the
+   emulation, and the ioctl/control traces localize it further. This is the
+   fastest known way to find an emulation gap without understanding the whole
+   stack.
+
+## 4. Performance
+
+The performance budget for virtualized CUDA is dominated by three things, in
+order:
+
+1. **Push-buffer execution.** Kernel launches, copies and syncs are encoded by
+   guest userspace into memory. The host must execute that work efficiently.
+   The project's design decodes pushes as method streams and executes them
+   (memsets/copies/semaphore releases) rather than round-tripping each method.
+   Fast path: operate directly on the mapped memory; no per-method syscall.
+2. **Synchronization latency.** Every launch ends in a semaphore release; every
+   `cuStreamSynchronize` waits on a marker. Host-side completion must be
+   signaled promptly (ISR/MSI-X injection latency matters more than raw
+   throughput). The userspace driver's blocking-sync path can park threads on
+   OS events, so delivering the completion interrupt promptly avoids spinning.
+3. **Control-plane latency (setup, not steady state).** Context creation,
+   module load, first launch are control-heavy; these happen once. Optimizing
+   them is about correctness first (see `docs/05`), not microseconds.
+
+Design rules that follow:
+
+* **Fast path in the host kernel**: the push decoder and semaphore service run
+  in the kernel module, with no daemon in the loop for steady-state work.
+* **Batching**: process a whole push (or a whole page of pushes) in one pass,
+  keeping decode state across methods; avoid per-method wakeups.
+* **No copy where mapping suffices**: guest memory the GPU reads/writes is
+  pinned and mapped; the emulation should operate on it in place.
+* **Measure against the A/B oracle**: `tools/cuda-ab/` gives host-vs-guest
+  numbers for the same binary; a regression is visible immediately.
+* **Latency instrumentation**: RPC round-trip and interrupt-injection timing
+  are traceable (`trace_printk` in the module) so hot spots are found with
+  data.
+
+## 5. Portability
+
+The same analysis supports:
+
+* **Multiple driver versions** (550/560/570/580+, Windows and Linux): the
+  versioned interfaces are data-driven; the name map and config carry the
+  per-version facts.
+* **Multiple GPU architectures** (Turing → Blackwell): the hardware layer
+  (QMD/method encodings) is selected by compute capability inside the driver;
+  the host must only understand the *effects* (copy/memset/semaphore/grid),
+  not every encoding. Where a new architecture introduces a genuinely new
+  effect, the decoder learns it, not the whole stack.
+* **Multiple hypervisors**: the emulation boundary is VFIO/mdev today, but the
+  interface analysis is hypervisor-neutral (the same stock guest runs under
+  libvirt, / or raw QEMU).
+* **Emulation and passthrough**: as noted in §2, both are implementations of
+  the same interface; a passthrough deployment validates the guest stack and
+  narrows the emulation's responsibility to the control plane.
+
+## 6. Security and correctness
+
+* **Isolation.** The guest's CUDA objects must map to host objects with the
+  guest's identity and limits; no guest handle may name a host object it does
+  not own. Handles are per-client; the emulation must not let a guest forge
+  handles or reuse freed ones (handle generation).
+* **Memory safety.** Every guest-provided address/size (push buffers, GPFIFO
+  entries, semaphore addresses, VA ranges) is untrusted input to the host
+  decoder. Validate ranges against the mapped regions, bound loops, and never
+  index arrays with guest-controlled values.
+* **DMA containment.** With an IOMMU, guest memory used by the GPU is mapped
+  explicitly; the emulation must not grant broader DMA windows than the
+  allocation requires.
+* **Interrupt/event safety.** Event objects are per-client; signalling must be
+  routed to the right VM only.
+* **Correctness before speed.** A wrong reply is worse than a slow one: it
+  corrupts driver state (as in `docs/05`). The reply-fidelity ladder is:
+  status-only → coherent zeroed params → coherent values for consumed fields →
+  exact values. Every rung should be validated by the A/B harness before
+  moving on.
+* **Auditability.** All host-side emulation decisions are logged (control ID,
+  size, status, consuming context) so a guest failure can be reconstructed
+  from the host log alone.
+
+## 7. The practical path (where this analysis plugs in)
+
+1. Use `docs/03` as the contract checklist for the emulation.
+2. Use `docs/05` + the decompiled corpus to answer "what does the driver do
+   with the reply we just gave it" for any failure.
+3. Use `docs/06`/`docs/07` to re-run the analysis when the driver, CUDA
+   runtime, GSP firmware or hardware changes.
+4. Keep the guest stock; keep the interface emulated; keep the work executed.
+   Everything in this directory is organized around that sentence.
diff --git a/userspace/cuda/docs/05-570.144-decompilation.md b/userspace/cuda/docs/05-570.144-decompilation.md
new file mode 100644
index 0000000..84b377e
--- /dev/null
+++ b/userspace/cuda/docs/05-570.144-decompilation.md
@@ -0,0 +1,315 @@
+# 05 — 570.144 decompilation: map, key paths, and the `cuCtxCreate` bring-up
+
+This document is the version-specific result of the comprehensive
+decompilation of `libcuda.so.570.144` and the reference for anyone debugging
+the guest's CUDA bring-up. Addresses are **file virtual addresses**
+(`libcuda+0x...`).
+
+Artifacts behind it (gitignored, local):
+
+* `run/libcuda.so.570.144` — the guest's library, md5
+  `ed5b2fc0044df6052df1d5113754c499`;
+* `run/ghidra-full/` — the analyzed Ghidra project (29,494 functions);
+* `run/analysis/elf-index/` — `functions.tsv`, `exports.tsv`,
+  `callgraph.tsv` (77,742 edges), `strings.tsv` (102,324 strings),
+  `blocks.tsv`;
+* `run/analysis/decompiled/` — the full decompiled corpus
+  (`buckets/*.c`, `index/*.tsv`).
+
+Reproduce with `tools/ghidra_full_analysis.sh`, `tools/ghidra_export_index.sh`
+and `tools/ghidra_batch_decompile.sh` (see `docs/06-tooling.md`).
+
+---
+
+## 1. Analysis result
+
+| Metric | Value |
+|---|---|
+| Functions (full auto-analysis) | 29,494 |
+| Functions with unwind info (FDEs) | 28,925 |
+| Exported symbols | 659 (plus 31 non-function exports) |
+| Call edges | 77,742 |
+| Defined strings | 102,324 |
+| Functions decompiled | 29,254 (all non-external functions in the executable range; 240 external imports) |
+| Decompilation failures | 0 |
+
+The analysis used Ghidra 12.1.2 with the two optional identifier analyzers
+disabled (parameter-ID and function-ID); everything else (disassembly,
+aggressive instruction finding, exception handlers, stack, reference
+analyzers) was on. This is the right trade-off for a stripped binary: it
+finishes in minutes and still finds essentially every function.
+
+### 1.1 Address convention (read this before using any offset)
+
+Ghidra loads these shared objects at **image base `0x100000`**, so:
+
+```
+Ghidra program address = file virtual address + 0x100000
+```
+
+The tools translate automatically. The earlier analysis in this tree did not:
+`run/helper_453ef0.c` and friends were produced by passing a file vaddr
+(`0x453ef0`) straight to Ghidra, which decompiled `libcuda+0x353ef0` — a
+*different function*. Every conclusion in the old write-up that depended on
+decompiled code was therefore about the wrong function. The runtime
+observations (gdb register traces, live table values, ioctl traces) remain
+valid and are used below.
+
+Correct addresses for the old write-up's landmarks:
+
+| What the old note called | File vaddr | Ghidra address | Correct identity |
+|---|---|---|---|
+| `FUN_00453ef0` (helper) | `0x453ef0` | `0x553ef0` | engine-context registration helper (per engine) |
+| crash site | `0x453f90` | `0x553f90` | continuation inside the helper after the hook call |
+| `FUN_0045a670` (walker) | `0x45a670` | `0x55a670` | device engine/context walker (calls the helper twice) |
+| `FUN_004676f0` (hook) | `0x4676f0` | `0x5676f0` | hook reached through `[table+0x40]+0x530` |
+| hook callee | `0x481360` | `0x581360` | engine context create/register (issues `NV_ESC_RM_ALLOC`) |
+| `FUN_003a9c20` | `0x3a9c20` | `0x4a9c20` | handle-pool allocation called before the hook |
+
+## 2. The public API stubs
+
+Every exported `cu*` is a 24–30 byte stub:
+
+1. compare the global initialization word against the "destroyed" sentinel
+   `0x321CBA00`; if equal return `4` (`CUDA_ERROR_DEINITIALIZED`);
+2. otherwise `jmp [slot]` — an indirect jump through the interposition table.
+
+Initial slot values (from relocations) give the generated wrapper addresses:
+
+| Export (file vaddr) | Wrapper target |
+|---|---|
+| `cuInit` `0x310060` | `0x31b7f0` |
+| `cuCtxCreate_v2` `0x310360` | `0x3206d0` |
+| `cuModuleLoadData` `0x3106e0` | `0x324ed0` |
+| `cuMemAlloc_v2` `0x310b20` | `0x329830` |
+| `cuStreamSynchronize` `0x36f3c0` | `0x380b40` |
+| `cuLaunchKernel` `0x36f420` | `0x3810f0` |
+
+This is the mechanism described in `docs/01` §2.2/§2.3: the wrapper does the
+tracing/tool callbacks and then calls the hand-written implementation. When
+chasing a call, follow the slot; when a tool interposes, the slot changes.
+
+## 3. The `cuCtxCreate` path: crash solved, engine topology blocking
+
+The 2026-09 crash was deterministic: on a fresh guest `cuCtxCreate` segfaulted
+at `libcuda+0x453f90`. That is **solved** (§3.5). The call now returns
+`rc=999` (`CUDA_ERROR_UNKNOWN`), and the active blocker is engine topology —
+the emulated engine list is missing the async copy engine (§3.6).
+
+Fault facts from the crash period (live traces, corrected decompilation):
+
+```
+guest dmesg : segfault at ffffffffffffffc8  (NULL-0x38)
+              ip in libcuda.so.570.144, mapping base 0x735cfcb79000
+              file vaddr = 0x453f90
+gdb         : rbp=0, rax=0, rsi=0xc030462b (_IOWR('F',0x2b,48) = NV_ESC_RM_ALLOC)
+```
+
+### 3.1 The faulting instruction
+
+```
+libcuda+0x453ef0  push rbp; mov rbp,rsp; ...            <- helper entry
+...
+libcuda+0x453f5f  call qword ptr [rax+0x530]            <- engine hook
+libcuda+0x453f65  mov r12d,eax
+libcuda+0x453f68  test eax,eax
+libcuda+0x453f6a  je  0x453f90
+...
+libcuda+0x453f90  mov rax,[rbp-0x38]                    <- FAULT (rbp==0)
+```
+
+The helper saved a table pointer at `[rbp-0x38]` in its prologue and reloads
+it after the hook returns. With `rbp==0` this reads `0xffffffffffffffc8`,
+exactly the fault address.
+
+### 3.2 What the helper does (behaviour)
+
+`libcuda+0x453ef0(device, engine_desc)`:
+
+1. `idx = engine_desc->[0]`, with `1` normalized to `0` (the GR engine);
+2. read `table = *(*(*device + 8) + 0x40)` and save it;
+3. allocate a handle from the global handle pool (`libcuda+0x3a9c20`,
+   a bitmap allocator) → `engine_id`;
+4. if the lookup succeeded, call hook `[*(table2+0x40)+0x530]`
+   (`libcuda+0x4676f0`) with `(global, *(device+0x9488), engine_id,
+   engine_desc)` — this is the engine create/register call that ultimately
+   issues the `NV_ESC_RM_ALLOC` seen in `rsi` at the fault;
+5. if the hook returns 0, reload the saved table, read
+   `class = *(table+0xd40)`, and call a second hook
+   `[*(table2+0x40)+0x250]` with `(global, *(device+0x9488), engine_id,
+   class, device + idx*4 + 0x68)`;
+6. on success record `device[idx*4 + 0x50] = engine_id` and
+   `device[0x9488][idx*8 + 0x88] = engine_desc`;
+7. on failure, clean up (`libcuda+0x453cc0`, `libcuda+0x3a2f50`).
+
+So the helper is "register one engine with the driver": look up the engine,
+create its context (hook 1), then bind/allocate the engine object (hook 2),
+then publish the result into per-engine arrays.
+
+### 3.3 The engine walker
+
+`libcuda+0x45a670` walks the device's engine records and calls the helper at
+`libcuda+0x45afc1` and `libcuda+0x45b02b` (per engine, and once more for a
+secondary engine at `[device-table + 0x75c]`). The descriptor is selected
+from 20-byte (0x14) records:
+
+```
+engine index 0        -> table + 0x2fec
+engine index 1..0xa   -> table + 0x300c + (index-1)*0x14
+engine index 0xb      -> table + 0x350c
+engine index >= 0x16  -> table + 0x351c + (index-0x16)*0x14
+```
+
+before the helper, the walker calls the callback at `table+0x4e00`
+with `(table, desc->[0xc])` — a
+"does this engine/class exist" check — and only proceeds when it returns
+true. The helper then uses `desc->[0]` (engine kind), `desc->[8]` (instance)
+and `desc->[0xc]` (class) for the registration.
+
+### 3.4 The engine create/register function
+
+`libcuda+0x481360` (called by the hook) is the function that actually asks RM
+for an engine context. Behaviour (condensed):
+
+* class `0x80` (device): find an existing context for the class, create one,
+  then issue an RM alloc (`NV_ESC_RM_ALLOC`, request `0xc030462b`, 48-byte
+  parameter block) to bind it;
+* classes `0x5`/`0x79`: use a per-class object from a global table, with a
+  bounds-checked index;
+* class `0x2080` (subdevice): iterate a global array of 32 engine
+  descriptors, matching `[context+0x28]`, and for a match build a 0x28-byte
+  RM control (`0x202`) and alloc;
+* default: generic RM alloc via `libcuda+0x480d00`.
+
+Failure returns are small status codes (`0x1f`, `0x22`, `0x26`, `0x36`,
+`0x3d`, `0x56`, `0x59`, …) which the hook then routes into the driver's error
+reporting (`libcuda+0x387bf0`).
+
+### 3.5 Resolution: the alloc-params copy-back
+
+The engine-registration hook reaches `libcuda+0x481360`, which for the
+`TURING_COMPUTE_A` class (`0xc5c0`) issues `NV_ESC_RM_ALLOC` (ioctl `0x2b`,
+NVOS64 form) with `pAllocParms` pointing at the caller's **own 8-byte stack
+local** and `paramsSize = 0`.
+
+The guest driver does not honour the caller's `paramsSize`: for class `0xc5c0`
+its resource table declares 16-byte class params (`NV_GR_ALLOCATION_PARAMETERS`),
+so it
+
+1. reads 16 bytes starting at that 8-byte local — the upper 8 bytes are the
+   hook's saved `rbp` — and sends them as the alloc parameters, and
+2. on the reply, copies 16 bytes of the reply's params back over the same
+   buffer.
+
+With the old generic (zeroed) alloc reply, step 2 zeroed the hook's saved
+`rbp`. The hook's `leave` then loaded `rbp = 0`, and `mov rax,[rbp-0x38]` at
+`libcuda+0x453f90` faulted at `0xffffffffffffffc8`.
+
+Live gdb intervals confirmed the writer: the hook's saved-`rbp` slot was intact
+at the pre-ioctl site (`libcuda+0x481448`) and zeroed at the post-ioctl site
+(`libcuda+0x48146a`), and hardware watchpoints never saw an in-process store —
+the store is the kernel's `copy_to_user` inside the ioctl.
+
+**Fix (host module, live-verified):** the alloc reply now echoes the request's
+params back, with only `status` forced to `NV_OK`. The driver's class-size
+copy-back then writes back exactly the bytes it read — including the caller's
+saved-`rbp` slot — and the `libcuda+0x453f90` fault disappears. This is also
+what the real GSP-RM does for these in/out params: the host runs the same
+libcuda code and survives for the same reason. It is the "coherent params"
+rung of the reply-fidelity ladder (`docs/04` §6).
+
+**Historical: why `rbp` was zero (solved).** Earlier revisions of this document
+listed three candidate writers of the hook's saved-`rbp` slot; the writer was
+the alloc reply's class-size copy-back described above.
+
+### 3.6 Current blocker (2026-09-23): the missing async copy engine
+
+With the copy-back fixed, the walker completes but `cuCtxCreate` returns
+`rc=999` (`CUDA_ERROR_UNKNOWN`); `cuMemAlloc` and later calls are not reached.
+Measured with the guest ioctl trace plus a host A/B oracle (the same libcuda
+path on the host's real driver):
+
+* libcuda asks the driver for the GPU's engines (`GET_ENGINES_V2`). The host
+  oracle's filtered list has **7** engines: GR, CE0..CE4 and `COPY14`, the
+  async copy engine. The guest list has only **6** — the same minus `COPY14`.
+* `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, client engine
+  type `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's
+  `{class, engDesc}` pair table pairs OBJCE instances `0..7` with class
+  `0xcab5` and instances `0x0b..0x13` (including `0x0e` = 14) with class
+  `0xc9b5`.
+* The driver filters the client engine table through its per-(class, instance)
+  class database (`classDB`) before the engine walker (§3.3) sees it. With no
+  classDB entry for descriptor `0x793ceb0e` the engine is silently skipped and
+  libcuda never learns it exists.
+* The walker then registers a secondary engine from a record that is zeroed in
+  the guest (host: `t=2 cls=0xc5b5`; guest: `t=0 cls=0`, dumped live with gdb).
+  The zero class makes libcuda issue `NV_ESC_RM_ALLOC` for class `0`; the
+  ioctl escape path rewrites root classes to `NV01_ROOT_CLIENT` (`0x41`), so
+  the trace shows the single failing call `class=0x41 st=0x19` immediately
+  after the compute-object alloc.
+* libcuda rolls back — it frees the channel and the compute object — and
+  returns 999. The guest kernel still oopses in
+  `uvm_pmm_devmem_page_to_va_space` while tearing down the failed context.
+
+A first emulator-side attempt (advertise the `COPY14` capability bit in the
+static info and add a 13th device-info entry with descriptor `0x793ceb0e`,
+instance 14, RM engine type `0x17`, FIFO tag 6, runlist 5) did not change the
+guest list, so the missing input is elsewhere: what builds the driver's
+classDB entry for descriptor `0x793ceb0e` (class list/static-info class data
+versus the device-info-table fields) is the open question. The record layout
+the walker reads is in §3.3; the per-engine fields it consumes are
+`+0` (kind), `+8` (instance) and `+0xc` (class).
+
+### 3.7 What the emulation should check first
+
+* **Engine-list/classDB consistency — the active blocker (§3.6).** The engine
+  records at the offsets in §3.3 must be present and self-consistent for every
+  engine index the walker visits, the "does this engine exist" callback
+  (`[table+0x4e00]`) must agree with them, and the whole set must agree with
+  the class-filtered engine list the driver builds from the GSP static info
+  (`GET_ENGINES_V2` + classDB). A missing entry is silent: the walker then
+  reads a zeroed record.
+* **Consumed-control reply fidelity.** The controls answered generically (the
+  host log lists `0x20803039`, `0x20802a12`, `0x20808162`, `0x2080182b`,
+  `0x20800a70`, `0x20800a93`, …) should be checked for consumed fields, because
+  the walker's engine table is built from data the driver collected earlier
+  (device caps, engine list, class list).
+* The `NV_ESC_RM_ALLOC` parameter block built by `libcuda+0x481360` for class
+  `0xc5c0` (`TURING_COMPUTE_A`) must be answered with a coherent object: echo
+  the request's params back (§3.5) and return a coherent handle; the driver
+  immediately consumes the reply's params and (for other classes) engine ids.
+
+## 4. Navigating the decompiled corpus
+
+```
+# by address (file vaddr):
+grep -n "libcuda+0x45a670" 570.144/run/analysis/decompiled/buckets/*.c
+
+# by Ghidra name:
+awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv
+
+# callers of a function (program addresses):
+grep -P "^0x0055a670\t" 570.144/run/analysis/elf-index/callgraph.tsv
+
+# exports and their sizes:
+570.144/run/analysis/elf-index/exports.tsv
+
+# strings around an address (error messages, table names):
+awk -F'\t' '$1>="0x..." && $1<="0x..."' 570.144/run/analysis/elf-index/strings.tsv
+```
+
+Every bucket header carries `addr=0x<program> file_addr=libcuda+0x<file>` so
+citations in docs and gdb sessions use the same numbers.
+
+## 5. Known limits
+
+* Names are `FUN_<addr>` unless recovered; `570.144/annotations/names.tsv`
+  holds our name map (exports are exact, internals grow over time).
+* The decompiler does not recover struct types; field offsets are named
+  `[obj+0xNN]` and must be interpreted with the descriptions in `docs/01`
+  and `docs/03`.
+* Indirect calls dominate the API dispatch; the callgraph contains direct
+  edges only (77,742). Cross-referencing indirect targets requires reading the
+  wrapper/stub relocations (as in §2) or live traces.
+* Embedded device code (cubins) is present in `.rodata` and is *not* host
+  code; the corpus only covers `.text`.
diff --git a/userspace/cuda/docs/06-tooling.md b/userspace/cuda/docs/06-tooling.md
new file mode 100644
index 0000000..85d1c8c
--- /dev/null
+++ b/userspace/cuda/docs/06-tooling.md
@@ -0,0 +1,165 @@
+# 06 — Tooling: how to analyse a CUDA userspace driver
+
+The toolchain in `tools/` is version-agnostic. Point it at any
+`libcuda.so.<version>` and it produces the same artifacts as for 570.144. All
+tools are stdlib-only or Ghidra scripts; nothing needs network access.
+
+```
+userspace/cuda/tools/
+├── cudatool.py                     ELF/interface toolkit (python3, no deps)
+├── callwalk.py                     callee/caller trees and shortest call path
+├── annotate_corpus.py              annotated + name-substituted corpus
+├── ghidra_full_analysis.sh         import + full auto-analysis into a project
+├── ghidra_export_index.sh          functions/exports/callgraph/strings index
+├── ghidra_batch_decompile.sh       parallel, resumable full decompilation
+├── ghidra_decompile_at.sh          decompile one function by file vaddr
+├── ghidra_index_merge.sh           merge batch indexes
+└── ghidra/
+    ├── ConfigureAnalysis.java      analyzer tuning for big stripped binaries
+    ├── ExportFunctionIndex.java    index exporter
+    ├── BatchDecompile.java         chunk decompiler
+    ├── DecompileAt.java            single function (file-vaddr aware)
+    ├── ApplyNameMap.java           apply/transfer a name map
+    └── PrintProgramInfo.java       address convention + stats
+```
+
+## 1. Fast facts without Ghidra — `cudatool.py`
+
+```
+tools/cudatool.py info      <lib>          # build id, sections, exports, ABI note
+tools/cudatool.py exports   <lib>          # addr size name (file vaddrs)
+tools/cudatool.py functions <lib>          # function ranges from .eh_frame_hdr
+tools/cudatool.py nearest   <lib> <addr>   # function start + nearest export
+tools/cudatool.py crash     <lib> <map_base> <ip>   # dmesg -> file vaddr
+tools/cudatool.py strings   <lib> [--grep RE]
+tools/cudatool.py note      <lib>          # .note.cuda.fwd_compatibility JSON
+tools/cudatool.py delta     <libA> <libB>  # see §4
+```
+
+Useful for: checking what changed between two driver package versions before
+spending time on decompilation; converting a guest dmesg crash to a file vaddr
+without gdb; getting a function list in a second.
+
+### Address conventions
+
+* `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used
+  everywhere in this directory.
+* Ghidra may load the same binary at a non-zero image base. For 570.144 it is
+  `0x100000`; check with `PrintProgramInfo.java`. The Ghidra wrappers here
+  translate file vaddr → program address automatically; raw Ghidra scripts do
+  not.
+
+## 2. Full analysis — Ghidra
+
+```
+# 1. import + analyze (keep the project; can take minutes to hours)
+tools/ghidra_full_analysis.sh <lib> <project_dir> <project_name>
+
+# 2. export the index (fast, read-only)
+tools/ghidra_export_index.sh <project_dir> <project_name> <program> <outdir>
+
+# 3. decompile everything (parallel, resumable; safe to re-run)
+tools/ghidra_batch_decompile.sh <project_dir> <project_name> <program> <outdir> \
+    <start_hex> <end_hex> [chunk_hex] [jobs] [timeout_s]
+tools/ghidra_index_merge.sh <outdir>
+```
+
+Notes:
+
+* `ConfigureAnalysis.java` disables the two expensive identifier analyzers
+  (parameter-ID, function-ID) and keeps everything needed for a stripped
+  binary. Full analysis of 71 MB then takes minutes, not hours.
+* `ghidra_batch_decompile.sh` gives **every chunk its own project copy** under
+  `<outdir>/wk/` because Ghidra takes an exclusive project lock even in
+  read-only mode; completed chunks are skipped on re-run. Delete `wk/` to
+  reclaim disk.
+* The program name passed to Ghidra is the binary's file name
+  (`libcuda.so.570.144`).
+* Do **not** pass `-analysisTimeoutPerFile 0`: in this Ghidra build that means
+  "abort analysis immediately" (the original per-version helper in
+  `570.144/scripts/` did this and produced an analysed-looking project with no
+  analysis at all).
+
+## 2b. Annotating the corpus
+
+```
+tools/annotate_corpus.py <outdir> <indexdir> <names.tsv> <annotated_outdir>
+```
+
+Rewrites every function header with the recovered name, file address, size and
+direct callers/callees (named where the map has them), and substitutes
+recovered names for `FUN_<addr>` references in the bodies. Runs in seconds for
+30k functions; the output is gitignored like the raw corpus.
+
+## 2c. Walking the callgraph without Ghidra
+
+```
+tools/callwalk.py <indexdir>/callgraph.tsv callees 0x31b7f0 2 10
+tools/callwalk.py <indexdir>/callgraph.tsv callers 0x453ef0
+tools/callwalk.py <indexdir>/callgraph.tsv chain 0x31b7f0 0x2cb940
+```
+
+Addresses are file vaddrs; `names.tsv` next to the callgraph is used for
+labels.
+
+## 3. Looking things up
+
+```
+# one function by file vaddr
+tools/ghidra_decompile_at.sh <project_dir> <project_name> <program> 0x45a670 out.c
+
+# corpus search by file address (headers are annotated)
+grep -n "libcuda+0x45a670" <outdir>/buckets/*.c
+
+# index lookups
+awk -F'\t' '$2=="0x45a670"' <outdir>/decompiled_index.tsv
+grep -P "^0x0055a670\t" <indexdir>/callgraph.tsv     # direct callers/callees
+
+# exports/strings
+grep cuLaunchKernel <indexdir>/exports.tsv
+```
+
+## 4. Version deltas — `cudatool.py delta`
+
+```
+tools/cudatool.py delta <libA> <libB>
+```
+
+Reports, at the interface level: added/removed/moved/resized exports, import
+count changes, `.text` growth, build-id and the CUDA/ABI note. This is the
+first thing to run on a new driver release; it tells you immediately whether
+the update is a patch (nothing moved), a feature release (new exports) or a
+reshuffle (many moves).
+
+## 5. Name transfer — `ApplyNameMap.java`
+
+```
+tools/ghidra_decompile_at.sh-free invocation:
+  analyzeHeadless <project_dir> <project_name> -process <program> -noanalysis \
+      -scriptPath tools/ghidra -postScript ApplyNameMap.java <names.tsv> [--dry-run]
+```
+
+The map is `file_vaddr<TAB>name` per line (comments allowed). Name maps live
+in version directories (`570.144/annotations/names.tsv`) and are tracked, so
+naming work accumulates: apply the previous version's map to a new binary,
+then fix the misses. `--dry-run` reports how many names would land.
+
+## 6. Writing new Ghidra scripts
+
+Follow the existing pattern: a `GhidraScript` subclass in `tools/ghidra/`,
+args via `getScriptArgs()`, and file-vaddr ↔ program-address translation via
+`currentProgram.getImageBase()` (see `DecompileAt.java`). Keep outputs under
+the version's gitignored `run/` tree unless the artifact is our own work
+product (name maps, indexes of our findings), which may be tracked.
+
+## 7. Cost guide (measured on 570.144, 16 cores)
+
+| Step | Time | Disk |
+|---|---|---|
+| import alone | ~1 min | 324 MB project |
+| full analysis | ~10–15 min | same project |
+| index export | ~1–2 min | ~25 MB TSV |
+| full decompilation (5 workers) | ~20–30 min | ~3 GB C + 2 GB project copies |
+
+Notes for a rerun: set `GHIDRA_HEADLESS_MAXMEM` (default 12G for analysis,
+4G per batch job) and keep total batch jobs ≤ 5–6 on a 32 GB machine.
diff --git a/userspace/cuda/docs/07-version-upgrade-playbook.md b/userspace/cuda/docs/07-version-upgrade-playbook.md
new file mode 100644
index 0000000..9e53514
--- /dev/null
+++ b/userspace/cuda/docs/07-version-upgrade-playbook.md
@@ -0,0 +1,98 @@
+# 07 — Version-upgrade playbook: re-running the CUDA analysis
+
+When the guest driver, CUDA runtime, GSP firmware or GPU generation changes,
+this is the procedure that refreshes the analysis with minimal effort and
+keeps the name map and interface knowledge accumulating.
+
+## 0. Before you start
+
+* You need the new `libcuda.so.<version>` (from the guest with
+  `570.144/scripts/fetch_from_guest.sh`, or from the driver package with
+  `--extract-only`).
+* Work in a new version directory `userspace/cuda/<version>/` — never
+  overwrite an analysed version. Copy the structure from `570.144/`
+  (`scripts/`, `ghidra/`, `annotations/`) and create `run/`.
+* Update `WORKING-HERE.md` so parallel agents know which tree is being
+  analysed.
+
+## 1. Cheap triage (minutes)
+
+```
+tools/cudatool.py info    <lib>
+tools/cudatool.py delta   <previous_lib> <lib> > run/delta.txt
+tools/cudatool.py crash   <lib> <map_base> <ip>     # if a guest crash is known
+```
+
+Read `delta.txt`:
+
+* **no export moves, small text growth** → patch release: most names and
+  offsets from the previous version transfer; spot-check the subsystems
+  touched by the release notes.
+* **new exports** → feature release: identify the new API surface, map it to
+  engine functions via decompilation of the new stubs.
+* **many moves** → reshuffle (toolchain/linker change): plan a full re-analysis
+  and re-apply the name map, expecting to fix a fraction.
+
+## 2. Full analysis (tens of minutes)
+
+```
+tools/ghidra_full_analysis.sh <lib> <version_dir>/run/ghidra-full cuda-full
+tools/ghidra_export_index.sh  <version_dir>/run/ghidra-full cuda-full <program> <version_dir>/run/analysis/elf-index
+tools/ghidra_batch_decompile.sh <version_dir>/run/ghidra-full cuda-full <program> <version_dir>/run/analysis/decompiled \
+    <text_start> <text_end> 0x100000 5 120
+tools/ghidra_index_merge.sh <version_dir>/run/analysis/decompiled
+```
+
+Get `<text_start>`/`<text_end>` from `cudatool.py info`. Confirm the image base
+with `PrintProgramInfo.java` before citing addresses.
+
+## 3. Transfer names and knowledge
+
+```
+cp <previous>/annotations/names.tsv <new>/annotations/names.tsv
+# translate addresses: the map is file-vaddr keyed, so for unchanged
+# functions the offsets are usually identical; where the delta shows moves,
+# adjust from the decompiled corpus.
+analyzeHeadless <proj> cuda-full -process <program> -noanalysis \
+    -scriptPath tools/ghidra -postScript ApplyNameMap.java <new>/annotations/names.tsv --dry-run
+# then without --dry-run
+```
+
+Also transfer the *subsystem index* (`annotations/subsystems.tsv`: address,
+our name, subsystem, evidence) and re-validate the entries that the delta
+marked as moved.
+
+## 4. Validate against the guest
+
+1. Load the stack and run the guest smoke test (`tools/test/cuda_drv_test.c`
+   compiled as `/tmp/cdt`), with the ioctl tracer
+   (`tools/guest-diag/ioctl_trace.c`) attached.
+2. Compare the observed ioctl/control sequence with the previous version's
+   `docs/05` sequence; new or reordered controls are the interesting deltas.
+3. Run the host/guest A/B harness (`tools/cuda-ab/ab.sh all`) for a numerical
+   correctness check.
+4. If the guest fails, use the crash decoder + corpus lookup (§1, §2) to get
+   from a dmesg `ip` to the exact function and line in the decompiled corpus,
+   then to the control reply the driver consumed.
+
+## 5. Update the documents
+
+* `docs/05-<version>-decompilation.md` — new version-specific map and crash
+  notes (copy the structure from `docs/05-570.144-decompilation.md`).
+* `annotations/names.tsv` and `annotations/subsystems.tsv` — the deltas.
+* `docs/04-virtualization-and-emulation.md` — only if the *interface* changed
+  (new controls, new capability bits, new object classes).
+* `README.md` status table.
+
+## 6. What not to do
+
+* Do not patch the guest driver or add `LD_PRELOAD` shims to make things work;
+  that breaks the anti-fragility rule (`docs/04` §3). If a reply must change,
+  change the emulation.
+* Do not commit decompiled output, extracted binaries or Ghidra projects; they
+  are gitignored for a reason (vendor-derived, large).
+* Do not name non-public source locations in tracked docs; describe behaviour
+  and cite `libcuda+0x...` offsets.
+* Do not run a second analysis in the same Ghidra project while another
+  process has it open (exclusive lock, even read-only); use the per-chunk
+  project copies in `ghidra_batch_decompile.sh`.
diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md
new file mode 100644
index 0000000..3523ad1
--- /dev/null
+++ b/userspace/cuda/docs/08-virtualization-approaches.md
@@ -0,0 +1,67 @@
+# 08 — Virtualization approaches for CUDA: options, trade-offs, and where we are
+
+One-page decision summary for how a guest's CUDA work gets executed in
+Open-IOV. Companion to `docs/04-virtualization-and-emulation.md` (design
+principles) and to the project implementation docs (`docs/refactored-architecture.md`,
+`docs/04-implementation/02-practical-challenges-and-adversarial-interop.md`).
+
+## 1. The options
+
+| # | Approach | What runs where | Benefits | Drawbacks / what we must own |
+|---|---|---|---|---|
+| **0** | **Full software emulation** — execute every command in the host emulator (memsets, copies, semaphores, and in principle grids) | Everything in the host module | No physical GPU needed at all; perfectly isolated; simplest correctness story; portable | Orders of magnitude too slow for compute; only viable for state/memory operations; never the whole answer |
+| **A** | **Host RM/GSP proxy** — guest push → translate → host driver's RM API → real channel → GPU | Control and work both through the host driver | Reuses a mature driver's knowledge (page tables, scheduling, faults); fastest path to *correct*; no direct hardware ownership | Extra hop (latency/throughput cost); depends on host driver interfaces staying usable; sharing the host GPU with the host driver needs a policy |
+| **B** | **Direct engine submission** — guest push → translate → our own channel/push construction → GPU | Control emulated, work submitted by us | Lowest latency, no host-driver dependency, full control of scheduling/isolation | We must own page tables, channel/runlist setup, cache invalidation, completion, fault handling — the largest engineering burden |
+| **C** | **Hybrid** — management via A, hot path via B | Split by operation class | Pragmatic: correctness now, performance where it costs | Two data paths to keep consistent; the boundary must be an interface from day one |
+| **D** | **Hardware partitioning (SR-IOV/VF, MIG)** | Guest on its own partition/VF | Strong isolation, near-native performance | Not available on consumer GPUs; still needs a device model for the VF; complementary, not a replacement for A/B/C |
+| **E** | **Guest-assisted (patch driver / shim / API interception)** | Guest modified | Easy performance wins in the short term | Breaks the "stock guest" rule, the anti-fragility rules and the legal/clean-room stance — **rejected** |
+
+**Fast paths are orthogonal to the choice:** pure state operations (semaphores,
+fences, some memops) should be satisfied directly in the emulation regardless
+of A/B/C — they are microseconds in hardware and would lose by round-tripping.
+
+## 2. Design implications
+
+* Where the boundary sits decides what we must own: A keeps ownership in the
+  host driver, B moves it into Open-IOV. Either way the **interface** the
+  guest sees is identical — that is what makes the choice reversible.
+* Isolation and correctness do not depend on the path, but **the failure
+  modes do**: A fails as host-driver errors and sharing conflicts, B fails as
+  hardware faults and page-table/scheduling bugs we introduced.
+* Performance targets (launch overhead, copy bandwidth, completion latency)
+  can only be met by forwarding *real work*; software execution is for
+  state/memory operations and as a correctness bridge, not the data plane.
+* The work path must therefore be abstracted behind one interface now, so A
+  can be swapped for B (or C) without touching the control plane.
+
+## 3. What Open-IOV is pursuing
+
+**Stated architecture:** *emulate the interface, pass through the work, never
+patch anyone* — a mediated-passthrough vGPU where PCI config/BARs, the GSP boot
+handshake, the GSP-RM protocol and interrupt routing are emulated in
+`open-iov-vfio.ko`, while real work is meant to execute on the physical GPU.
+The project's own decision procedure is explicit: **prototype A first for
+correctness, measure, then move the hot path to B if the budgets demand it**
+(option C in practice), with the work path abstracted from day one.
+
+**Current reality (2026-09, honest):** the *control plane* follows the
+architecture — GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and
+object/control replies are implemented, and the guest's stock driver +
+`libcuda` now get through `cuInit`, `cuDeviceGetCount`/`cuDeviceGet` and into
+`cuCtxCreate`. The old `cuCtxCreate` segfault is solved (the alloc reply must
+echo the request's params — `docs/05` §3.5); context creation is blocked one
+step later by engine topology: the class-filtered engine list is missing the
+async copy engine, so the walker reads a zeroed secondary record and the call
+returns `rc=999` (`docs/05` §3.6). The *data plane* is at the bridge stage: the
+host module decodes the guest's push buffers as method streams and **executes
+the small work it understands itself** (memsets, copies, semaphore releases)
+instead of forwarding it to the physical GPU. No CUDA kernel has been
+forwarded yet. So: yes, we are pursuing the intended architecture, but we are
+in the "emulate + software-execute state/memory ops" phase of it, with the
+A→B/C forwarding decision still ahead.
+
+**Risks to guard now:** (1) software execution silently becoming the data
+plane instead of a bridge; (2) the work-path interface being skipped because
+the direct path "works for now"; (3) tuning the emulation to one driver
+version instead of the interface. The anti-fragility, performance and security
+rules in `docs/04` exist to prevent exactly this drift.