diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:04 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:04 -0400 |
| commit | f874d709e5a45d63c8ee9ab6b0300195aa5d20a5 (patch) | |
| tree | 2d07208f789e402686ac960e5ad7456b6fd21cfd /userspace/cuda/docs/08-virtualization-approaches.md | |
| parent | ae18e08d02500579f084616874d8f2ae57f8281b (diff) | |
| download | open-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.tar.gz open-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.zip | |
userspace/cuda: add CUDA userspace driver analysis, tooling and name maps
Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs:
* docs/01-08: architecture (layers, object model, threads, sync, memory,
channels, launch), build and execution model, the driver/kernel/GPU
interface contract, virtualization approaches, the 570.144 decompilation
map with the corrected libcuda crash path and the current engine-topology
blocker, tooling, and a version-upgrade playbook.
* tools/: version-agnostic toolchain - stdlib ELF/interface toolkit
(exports, unwind-table function map, wrapper resolution, version delta,
crash address decode), callgraph walker, corpus annotator, and Ghidra
analysis/decompilation/name-map scripts with resumable parallel drivers.
* 570.144/annotations/: our name map (exports + resolved wrappers + verified
internals) and subsystem index, tracked so naming work accumulates.
* READ-ME-FIRST.md: reading guide with the fast lookup recipes.
The full decompiled corpus and annotated corpus stay untracked (vendor-
derived, gitignored under the version run/ tree).
Diffstat (limited to 'userspace/cuda/docs/08-virtualization-approaches.md')
| -rw-r--r-- | userspace/cuda/docs/08-virtualization-approaches.md | 67 |
1 files changed, 67 insertions, 0 deletions
diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md new file mode 100644 index 0000000..3523ad1 --- /dev/null +++ b/userspace/cuda/docs/08-virtualization-approaches.md @@ -0,0 +1,67 @@ +# 08 — Virtualization approaches for CUDA: options, trade-offs, and where we are + +One-page decision summary for how a guest's CUDA work gets executed in +Open-IOV. Companion to `docs/04-virtualization-and-emulation.md` (design +principles) and to the project implementation docs (`docs/refactored-architecture.md`, +`docs/04-implementation/02-practical-challenges-and-adversarial-interop.md`). + +## 1. The options + +| # | Approach | What runs where | Benefits | Drawbacks / what we must own | +|---|---|---|---|---| +| **0** | **Full software emulation** — execute every command in the host emulator (memsets, copies, semaphores, and in principle grids) | Everything in the host module | No physical GPU needed at all; perfectly isolated; simplest correctness story; portable | Orders of magnitude too slow for compute; only viable for state/memory operations; never the whole answer | +| **A** | **Host RM/GSP proxy** — guest push → translate → host driver's RM API → real channel → GPU | Control and work both through the host driver | Reuses a mature driver's knowledge (page tables, scheduling, faults); fastest path to *correct*; no direct hardware ownership | Extra hop (latency/throughput cost); depends on host driver interfaces staying usable; sharing the host GPU with the host driver needs a policy | +| **B** | **Direct engine submission** — guest push → translate → our own channel/push construction → GPU | Control emulated, work submitted by us | Lowest latency, no host-driver dependency, full control of scheduling/isolation | We must own page tables, channel/runlist setup, cache invalidation, completion, fault handling — the largest engineering burden | +| **C** | **Hybrid** — management via A, hot path via B | Split by operation class | Pragmatic: correctness now, performance where it costs | Two data paths to keep consistent; the boundary must be an interface from day one | +| **D** | **Hardware partitioning (SR-IOV/VF, MIG)** | Guest on its own partition/VF | Strong isolation, near-native performance | Not available on consumer GPUs; still needs a device model for the VF; complementary, not a replacement for A/B/C | +| **E** | **Guest-assisted (patch driver / shim / API interception)** | Guest modified | Easy performance wins in the short term | Breaks the "stock guest" rule, the anti-fragility rules and the legal/clean-room stance — **rejected** | + +**Fast paths are orthogonal to the choice:** pure state operations (semaphores, +fences, some memops) should be satisfied directly in the emulation regardless +of A/B/C — they are microseconds in hardware and would lose by round-tripping. + +## 2. Design implications + +* Where the boundary sits decides what we must own: A keeps ownership in the + host driver, B moves it into Open-IOV. Either way the **interface** the + guest sees is identical — that is what makes the choice reversible. +* Isolation and correctness do not depend on the path, but **the failure + modes do**: A fails as host-driver errors and sharing conflicts, B fails as + hardware faults and page-table/scheduling bugs we introduced. +* Performance targets (launch overhead, copy bandwidth, completion latency) + can only be met by forwarding *real work*; software execution is for + state/memory operations and as a correctness bridge, not the data plane. +* The work path must therefore be abstracted behind one interface now, so A + can be swapped for B (or C) without touching the control plane. + +## 3. What Open-IOV is pursuing + +**Stated architecture:** *emulate the interface, pass through the work, never +patch anyone* — a mediated-passthrough vGPU where PCI config/BARs, the GSP boot +handshake, the GSP-RM protocol and interrupt routing are emulated in +`open-iov-vfio.ko`, while real work is meant to execute on the physical GPU. +The project's own decision procedure is explicit: **prototype A first for +correctness, measure, then move the hot path to B if the budgets demand it** +(option C in practice), with the work path abstracted from day one. + +**Current reality (2026-09, honest):** the *control plane* follows the +architecture — GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and +object/control replies are implemented, and the guest's stock driver + +`libcuda` now get through `cuInit`, `cuDeviceGetCount`/`cuDeviceGet` and into +`cuCtxCreate`. The old `cuCtxCreate` segfault is solved (the alloc reply must +echo the request's params — `docs/05` §3.5); context creation is blocked one +step later by engine topology: the class-filtered engine list is missing the +async copy engine, so the walker reads a zeroed secondary record and the call +returns `rc=999` (`docs/05` §3.6). The *data plane* is at the bridge stage: the +host module decodes the guest's push buffers as method streams and **executes +the small work it understands itself** (memsets, copies, semaphore releases) +instead of forwarding it to the physical GPU. No CUDA kernel has been +forwarded yet. So: yes, we are pursuing the intended architecture, but we are +in the "emulate + software-execute state/memory ops" phase of it, with the +A→B/C forwarding decision still ahead. + +**Risks to guard now:** (1) software execution silently becoming the data +plane instead of a bridge; (2) the work-path interface being skipped because +the direct path "works for now"; (3) tuning the emulation to one driver +version instead of the interface. The anti-fragility, performance and security +rules in `docs/04` exist to prevent exactly this drift. |
