userspace/cuda: add CUDA userspace driver analysis, tooling and name maps

Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs:

* docs/01-08: architecture (layers, object model, threads, sync, memory,
  channels, launch), build and execution model, the driver/kernel/GPU
  interface contract, virtualization approaches, the 570.144 decompilation
  map with the corrected libcuda crash path and the current engine-topology
  blocker, tooling, and a version-upgrade playbook.
* tools/: version-agnostic toolchain - stdlib ELF/interface toolkit
  (exports, unwind-table function map, wrapper resolution, version delta,
  crash address decode), callgraph walker, corpus annotator, and Ghidra
  analysis/decompilation/name-map scripts with resumable parallel drivers.
* 570.144/annotations/: our name map (exports + resolved wrappers + verified
  internals) and subsystem index, tracked so naming work accumulates.
* READ-ME-FIRST.md: reading guide with the fast lookup recipes.

The full decompiled corpus and annotated corpus stay untracked (vendor-
derived, gitignored under the version run/ tree).

1 files changed, 244 insertions, 0 deletions

diff --git a/userspace/cuda/docs/04-virtualization-and-emulation.md b/userspace/cuda/docs/04-virtualization-and-emulation.md
new file mode 100644
index 0000000..89adc35
--- /dev/null
+++ b/userspace/cuda/docs/04-virtualization-and-emulation.md
@@ -0,0 +1,244 @@
+# 04 — Virtualizing CUDA: NVIDIA's vGPU model, Open-IOV's model, and how to stay unbreakable
+
+This is the design document for CUDA virtualization in Open-IOV. It answers
+four questions:
+
+1. How does NVIDIA virtualize CUDA in its own vGPU product?
+2. How does Open-IOV's model differ, and why?
+3. How do we stay working across CUDA/driver/GSP/firmware/hardware changes
+   (anti-fragility)?
+4. How do we get performance, portability, security and correctness?
+
+Companion documents: `docs/01-architecture.md` (what the driver is),
+`docs/03-driver-and-gpu-interface.md` (the interface we must satisfy).
+
+---
+
+## 1. NVIDIA's vGPU architecture (behavioural summary)
+
+NVIDIA's vGPU stack splits the driver across the VM boundary. The **guest is
+fully unmodified**: stock `libcuda`, stock `nvidia.ko`, stock `nvidia-uvm.ko`.
+Virtualization is hidden below the RM client interface, using a mediated PCI
+function and a host-side device model:
+
+```
+guest userspace      libcuda (unmodified)  ── /dev/nvidia* ioctls ──┐
+guest kernel         nvidia.ko, nvidia-uvm.ko (unmodified)         │
+                     │                                             │
+                     │  emulated PCI function: config space,       │
+                     │  BAR0/BAR1 MMIO, doorbells, MSI-X, VGA      │
+                     ▼                                             │
+hypervisor           mediated device (VFIO/mdev on KVM, VMBus on    │
+                     Hyper-V, device model on Xen/VMware)          │
+                     │                                             │
+host kernel          vGPU kernel module (VFIO side)                │
+                     │  ioctls: create/destroy, pin/unpin pages,   │
+                     │  reg access, mmio mappings, interrupt       │
+                     │  injection, migration state                 │
+                     ▼                                             │
+host userspace       vGPU manager daemon + plugin (VMIOP)  ◄───────┘
+                     │  NvRm API
+                     ▼
+host kernel + GPU    host nvidia.ko / GSP-RM / physical GPU
+```
+
+The pieces that make this work:
+
+* **A mediated PCI function.** The guest sees a GPU with a *vGPU profile*
+  (a slice of framebuffer, a set of engines, display heads, encoder capacity).
+  Config space, BAR sizes, MMIO registers, doorbells and MSI-X are emulated or
+  forwarded; the profile is what licenses and limits the VM.
+* **A guest-host wire protocol (VMIOP) between the guest kernel driver and the
+  host plugin.** The guest kernel driver's MMIO/register accesses arrive at
+  the host plugin as emulation callbacks (space, offset, width, data,
+  cacheability). The plugin either satisfies them from backing memory (with
+  per-register callback gating) or forwards them into host RM.
+* **Guest memory mapping and pinning.** Guest pages are pinned and mapped into
+  the plugin so the physical GPU can DMA into guest RAM; the guest framebuffer
+  is carved from physical VRAM and mapped into the guest's physical address
+  space.
+* **Interrupt injection.** Device interrupts are delivered to the guest as
+  MSI/MSI-X through the hypervisor.
+* **A capability negotiation on the CUDA side.** The CUDA userspace driver
+  detects vGPU (`GPU_GET_VIRTUALIZATION_MODE == VGX`) and queries a vGPU
+  capability block: is CUDA enabled for this profile, is UVM available, are
+  replayable faults available, is GPU-direct RDMA available, is TSG timeslice
+  override available. It then *restricts itself*: no UVM if the profile lacks
+  it, no debugger if disabled, no checkpoint/CRIU, CILP instead of MBP for
+  preemption, and it enforces the GRID license state (unlicensed VMs get
+  `CUDA_ERROR_DEVICE_NOT_LICENSED` or a sleep/timebomb behavior).
+* **Real objects on the host.** Channels, VA spaces, memory objects, engine
+  contexts and command execution are *real* host RM objects bound to real
+  channels; command streams are passed through and executed by the physical
+  GPU. What is virtualized is the *view* (profiles, MMIO, interrupts,
+  migration), not the execution.
+
+Two design consequences worth naming:
+
+* The guest driver is never patched; the virtualization is data (capabilities,
+  object parameters, emulated registers) plus a host device model.
+* The split is at the *kernel-driver ↔ hardware* boundary. Everything above
+  it (CUDA userspace, RM API semantics) is the stock stack.
+
+The exact plugin interfaces, capability bits, object classes and migration
+stages are vendor-internal and not needed to build an interoperable
+alternative.
+
+## 2. Open-IOV's model: emulate the interface, pass through the work
+
+Open-IOV makes the same fundamental choice as NVIDIA — **the guest stays
+stock** — but implements the host side differently:
+
+| Concern | NVIDIA vGPU | Open-IOV |
+|---|---|---|
+| Guest stack | stock | stock (identical constraint) |
+| Device presented | mediated PCI function with a vGPU profile, backed by host RM | emulated PCI function (mdev/VFIO) backed by the host module + GSP-RM emulation |
+| Where commands execute | physical GPU via host RM channels | host executes the work the guest encodes (e.g. memsets/copies/semaphore releases) or passes through, depending on the resource |
+| Host-side control plane | vGPU manager daemon + VMIOP plugin + host RM | `open-iov-vfio.ko` emulating the GSP-RM interface and device model directly in the kernel |
+| Licensing | GRID license server / vGPU licensing; CUDA gated on license state | no GRID licensing; the driver's vGPU detection must be answered in a way that does not gate CUDA (or the guest must not be told it is a licensed-vGPU-only environment) |
+| CUDA userspace | unmodified, capability-limited by vGPU profile | unmodified, no profile limits imposed |
+| Migration | vGPU-specific save/restore protocol with dirty-page tracking | 4-phase migration in the host module (project feature) |
+
+The key architectural difference: **Open-IOV does not need to virtualize the
+GPU's register interface at all if it can emulate the GSP-RM interface the
+kernel driver consumes.** The kernel driver asks the GSP (via RPCs) for
+objects, memory, channels, engines and caps; if the emulation answers those
+RPCs coherently, the driver builds its own command streams and the guest CUDA
+stack works. This moves the emulation boundary *up* from MMIO registers to
+RPC messages, which is far more stable: RPCs are versioned and typed, MMIO
+register maps are not.
+
+Where pass-through is possible (real GPU assigned, SR-IOV, or a future
+split-execution mode), the same userspace stack runs unchanged; the host-side
+work is the only difference. That is the portability goal: the analysis in
+`docs/01`–`docs/03` describes the *interface*, and both emulation and
+passthrough are implementations of it.
+
+## 3. Anti-fragility: how we survive version churn
+
+CUDA, the kernel driver, GSP firmware and GPU architectures all change on
+independent schedules. Our analysis and implementation are designed so that
+change moves *data*, not *architecture*:
+
+1. **Anchor on the stable interfaces.** The kernel-driver interface (RM
+   objects, controls, ioctls) and the userspace driver's internal layering
+   have been stable for many years; the driver-ABI adaptation layer inside
+   `libcuda` exists precisely because NVIDIA itself maintains that stability.
+   We track the interface, not the implementation.
+2. **Prefer behaviour over constants.** Every numeric ID we depend on
+   (classes, controls, capability bits) is versioned in config
+   (`config/versions/*.yaml`, `config/hardware/*.yaml`), with the analysis
+   tooling able to re-derive them from a new binary (`cudatool.py delta`,
+   `docs/07`).
+3. **Never require driver patches.** No guest modification, no userspace
+   shim, no `LD_PRELOAD`. Anything that would break on a driver update is
+   rejected by design. The only supported hook points are the ones NVIDIA
+   itself keeps stable: entry tables, `cuGetProcAddress` versioning, the RM
+   ABI.
+4. **Emulate replies, not side effects.** The host emulation answers RPCs with
+   *coherent data*; it does not fake hardware state that the driver might
+   later cross-check. Where a value is consumed (sizes, IDs, caps), we must
+   supply a consistent value — the crash analysis in `docs/05` is exactly a
+   case of inconsistent data reaching a userspace consumer.
+5. **Name transfer instead of re-analysis.** Function names and structure
+   knowledge are stored in a reviewable name map (`570.144/annotations/`) and
+   can be re-applied to a new version in seconds, then corrected where
+   signatures moved. Analysis effort accumulates across versions.
+6. **Decompose by failure mode.** The analysis is organized by what can break:
+   * *API layer* — new/changed entry points (`delta` detects them);
+   * *engine* — new object types, new control flows;
+   * *driver-model* — new controls/ABI versions;
+   * *hardware layer* — new chips and QMD/method encodings;
+   * *device code* — new syscalls/trampolines embedded in the library;
+   * *UVM* — page-migration and channel registration changes.
+7. **Keep a reference oracle.** The same CUDA workload runs on bare metal
+   (host A/B harness) and in the guest; divergence localizes the break to the
+   emulation, and the ioctl/control traces localize it further. This is the
+   fastest known way to find an emulation gap without understanding the whole
+   stack.
+
+## 4. Performance
+
+The performance budget for virtualized CUDA is dominated by three things, in
+order:
+
+1. **Push-buffer execution.** Kernel launches, copies and syncs are encoded by
+   guest userspace into memory. The host must execute that work efficiently.
+   The project's design decodes pushes as method streams and executes them
+   (memsets/copies/semaphore releases) rather than round-tripping each method.
+   Fast path: operate directly on the mapped memory; no per-method syscall.
+2. **Synchronization latency.** Every launch ends in a semaphore release; every
+   `cuStreamSynchronize` waits on a marker. Host-side completion must be
+   signaled promptly (ISR/MSI-X injection latency matters more than raw
+   throughput). The userspace driver's blocking-sync path can park threads on
+   OS events, so delivering the completion interrupt promptly avoids spinning.
+3. **Control-plane latency (setup, not steady state).** Context creation,
+   module load, first launch are control-heavy; these happen once. Optimizing
+   them is about correctness first (see `docs/05`), not microseconds.
+
+Design rules that follow:
+
+* **Fast path in the host kernel**: the push decoder and semaphore service run
+  in the kernel module, with no daemon in the loop for steady-state work.
+* **Batching**: process a whole push (or a whole page of pushes) in one pass,
+  keeping decode state across methods; avoid per-method wakeups.
+* **No copy where mapping suffices**: guest memory the GPU reads/writes is
+  pinned and mapped; the emulation should operate on it in place.
+* **Measure against the A/B oracle**: `tools/cuda-ab/` gives host-vs-guest
+  numbers for the same binary; a regression is visible immediately.
+* **Latency instrumentation**: RPC round-trip and interrupt-injection timing
+  are traceable (`trace_printk` in the module) so hot spots are found with
+  data.
+
+## 5. Portability
+
+The same analysis supports:
+
+* **Multiple driver versions** (550/560/570/580+, Windows and Linux): the
+  versioned interfaces are data-driven; the name map and config carry the
+  per-version facts.
+* **Multiple GPU architectures** (Turing → Blackwell): the hardware layer
+  (QMD/method encodings) is selected by compute capability inside the driver;
+  the host must only understand the *effects* (copy/memset/semaphore/grid),
+  not every encoding. Where a new architecture introduces a genuinely new
+  effect, the decoder learns it, not the whole stack.
+* **Multiple hypervisors**: the emulation boundary is VFIO/mdev today, but the
+  interface analysis is hypervisor-neutral (the same stock guest runs under
+  libvirt, / or raw QEMU).
+* **Emulation and passthrough**: as noted in §2, both are implementations of
+  the same interface; a passthrough deployment validates the guest stack and
+  narrows the emulation's responsibility to the control plane.
+
+## 6. Security and correctness
+
+* **Isolation.** The guest's CUDA objects must map to host objects with the
+  guest's identity and limits; no guest handle may name a host object it does
+  not own. Handles are per-client; the emulation must not let a guest forge
+  handles or reuse freed ones (handle generation).
+* **Memory safety.** Every guest-provided address/size (push buffers, GPFIFO
+  entries, semaphore addresses, VA ranges) is untrusted input to the host
+  decoder. Validate ranges against the mapped regions, bound loops, and never
+  index arrays with guest-controlled values.
+* **DMA containment.** With an IOMMU, guest memory used by the GPU is mapped
+  explicitly; the emulation must not grant broader DMA windows than the
+  allocation requires.
+* **Interrupt/event safety.** Event objects are per-client; signalling must be
+  routed to the right VM only.
+* **Correctness before speed.** A wrong reply is worse than a slow one: it
+  corrupts driver state (as in `docs/05`). The reply-fidelity ladder is:
+  status-only → coherent zeroed params → coherent values for consumed fields →
+  exact values. Every rung should be validated by the A/B harness before
+  moving on.
+* **Auditability.** All host-side emulation decisions are logged (control ID,
+  size, status, consuming context) so a guest failure can be reconstructed
+  from the host log alone.
+
+## 7. The practical path (where this analysis plugs in)
+
+1. Use `docs/03` as the contract checklist for the emulation.
+2. Use `docs/05` + the decompiled corpus to answer "what does the driver do
+   with the reply we just gave it" for any failure.
+3. Use `docs/06`/`docs/07` to re-run the analysis when the driver, CUDA
+   runtime, GSP firmware or hardware changes.
+4. Keep the guest stock; keep the interface emulated; keep the work executed.
+   Everything in this directory is organized around that sentence.