From 21b8ebc9db1d0ae649d6406087ea22b54e3b5775 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Wed, 23 Sep 2026 12:11:04 -0400 Subject: userspace/cuda: add CUDA userspace driver analysis, tooling and name maps Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs: * docs/01-08: architecture (layers, object model, threads, sync, memory, channels, launch), build and execution model, the driver/kernel/GPU interface contract, virtualization approaches, the 570.144 decompilation map with the corrected libcuda crash path and the current engine-topology blocker, tooling, and a version-upgrade playbook. * tools/: version-agnostic toolchain - stdlib ELF/interface toolkit (exports, unwind-table function map, wrapper resolution, version delta, crash address decode), callgraph walker, corpus annotator, and Ghidra analysis/decompilation/name-map scripts with resumable parallel drivers. * 570.144/annotations/: our name map (exports + resolved wrappers + verified internals) and subsystem index, tracked so naming work accumulates. * READ-ME-FIRST.md: reading guide with the fast lookup recipes. The full decompiled corpus and annotated corpus stay untracked (vendor- derived, gitignored under the version run/ tree). --- .../cuda/docs/08-virtualization-approaches.md | 67 ++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 userspace/cuda/docs/08-virtualization-approaches.md (limited to 'userspace/cuda/docs/08-virtualization-approaches.md') diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md new file mode 100644 index 0000000..3523ad1 --- /dev/null +++ b/userspace/cuda/docs/08-virtualization-approaches.md @@ -0,0 +1,67 @@ +# 08 — Virtualization approaches for CUDA: options, trade-offs, and where we are + +One-page decision summary for how a guest's CUDA work gets executed in +Open-IOV. Companion to `docs/04-virtualization-and-emulation.md` (design +principles) and to the project implementation docs (`docs/refactored-architecture.md`, +`docs/04-implementation/02-practical-challenges-and-adversarial-interop.md`). + +## 1. The options + +| # | Approach | What runs where | Benefits | Drawbacks / what we must own | +|---|---|---|---|---| +| **0** | **Full software emulation** — execute every command in the host emulator (memsets, copies, semaphores, and in principle grids) | Everything in the host module | No physical GPU needed at all; perfectly isolated; simplest correctness story; portable | Orders of magnitude too slow for compute; only viable for state/memory operations; never the whole answer | +| **A** | **Host RM/GSP proxy** — guest push → translate → host driver's RM API → real channel → GPU | Control and work both through the host driver | Reuses a mature driver's knowledge (page tables, scheduling, faults); fastest path to *correct*; no direct hardware ownership | Extra hop (latency/throughput cost); depends on host driver interfaces staying usable; sharing the host GPU with the host driver needs a policy | +| **B** | **Direct engine submission** — guest push → translate → our own channel/push construction → GPU | Control emulated, work submitted by us | Lowest latency, no host-driver dependency, full control of scheduling/isolation | We must own page tables, channel/runlist setup, cache invalidation, completion, fault handling — the largest engineering burden | +| **C** | **Hybrid** — management via A, hot path via B | Split by operation class | Pragmatic: correctness now, performance where it costs | Two data paths to keep consistent; the boundary must be an interface from day one | +| **D** | **Hardware partitioning (SR-IOV/VF, MIG)** | Guest on its own partition/VF | Strong isolation, near-native performance | Not available on consumer GPUs; still needs a device model for the VF; complementary, not a replacement for A/B/C | +| **E** | **Guest-assisted (patch driver / shim / API interception)** | Guest modified | Easy performance wins in the short term | Breaks the "stock guest" rule, the anti-fragility rules and the legal/clean-room stance — **rejected** | + +**Fast paths are orthogonal to the choice:** pure state operations (semaphores, +fences, some memops) should be satisfied directly in the emulation regardless +of A/B/C — they are microseconds in hardware and would lose by round-tripping. + +## 2. Design implications + +* Where the boundary sits decides what we must own: A keeps ownership in the + host driver, B moves it into Open-IOV. Either way the **interface** the + guest sees is identical — that is what makes the choice reversible. +* Isolation and correctness do not depend on the path, but **the failure + modes do**: A fails as host-driver errors and sharing conflicts, B fails as + hardware faults and page-table/scheduling bugs we introduced. +* Performance targets (launch overhead, copy bandwidth, completion latency) + can only be met by forwarding *real work*; software execution is for + state/memory operations and as a correctness bridge, not the data plane. +* The work path must therefore be abstracted behind one interface now, so A + can be swapped for B (or C) without touching the control plane. + +## 3. What Open-IOV is pursuing + +**Stated architecture:** *emulate the interface, pass through the work, never +patch anyone* — a mediated-passthrough vGPU where PCI config/BARs, the GSP boot +handshake, the GSP-RM protocol and interrupt routing are emulated in +`open-iov-vfio.ko`, while real work is meant to execute on the physical GPU. +The project's own decision procedure is explicit: **prototype A first for +correctness, measure, then move the hot path to B if the budgets demand it** +(option C in practice), with the work path abstracted from day one. + +**Current reality (2026-09, honest):** the *control plane* follows the +architecture — GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and +object/control replies are implemented, and the guest's stock driver + +`libcuda` now get through `cuInit`, `cuDeviceGetCount`/`cuDeviceGet` and into +`cuCtxCreate`. The old `cuCtxCreate` segfault is solved (the alloc reply must +echo the request's params — `docs/05` §3.5); context creation is blocked one +step later by engine topology: the class-filtered engine list is missing the +async copy engine, so the walker reads a zeroed secondary record and the call +returns `rc=999` (`docs/05` §3.6). The *data plane* is at the bridge stage: the +host module decodes the guest's push buffers as method streams and **executes +the small work it understands itself** (memsets, copies, semaphore releases) +instead of forwarding it to the physical GPU. No CUDA kernel has been +forwarded yet. So: yes, we are pursuing the intended architecture, but we are +in the "emulate + software-execute state/memory ops" phase of it, with the +A→B/C forwarding decision still ahead. + +**Risks to guard now:** (1) software execution silently becoming the data +plane instead of a bridge; (2) the work-path interface being skipped because +the direct path "works for now"; (3) tuning the emulation to one driver +version instead of the interface. The anti-fragility, performance and security +rules in `docs/04` exist to prevent exactly this drift. -- cgit v1.2.3