diff options
Diffstat (limited to 'docs/03-architecture/02-guest-stack-and-cudasw.md')
| -rw-r--r-- | docs/03-architecture/02-guest-stack-and-cudasw.md | 139 |
1 files changed, 139 insertions, 0 deletions
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md new file mode 100644 index 0000000..4ec614b --- /dev/null +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -0,0 +1,139 @@ +# The Guest Stack and CUDASW + +> What runs inside the guest, how it is layered, and what it expects from the +> device. This is the "what must keep working, unmodified" document. + +## 1. Vocabulary: what "CUDASW" means here + +**CUDASW** is the family of NVIDIA *software* components that sit between an +application and the GPU driver. In this project the term is used deliberately +broadly, because our compatibility target is the whole stack, not one library: + +``` +application + │ CUDA API (runtime) +libcudart.so ──────────────────────────────┐ CUDA runtime: device/stream/ + │ CUDA driver API │ event management, kernel launch +libcuda.so ────────────────────────────────┤ CUDA driver: contexts, modules, + │ ioctls │ memory, channel submission +/dev/nvidia* ──────────────────────────────┘ +nvidia.ko (RM + UVM + modeset) +``` + +Adjacent components in the same family: + +* **libnvidia-ptxjitcompiler** — JIT compilation of PTX to SASS. (No GPU + interaction beyond driver interfaces; usually "just works" once the driver + initializes.) +* **libnvidia-fatbinaryloader / nvrtc** — fatbinary loading. +* **UVM** (`nvidia_uvm.ko`) — unified memory and GPU page-fault-driven memory + management; see [`04-uvm.md`](04-uvm.md). +* **Graphics stack** — `libGLX_nvidia`, `libnvidia-glcore`, `libvulkan` ICD, + plus the kernel-side `nvidia_modeset`/`nvidia_drm` (roadmap). +* **Video stack** — NVENC/NVDEC userspace + kernel interfaces (roadmap). + +## 2. Layering rules the emulation must respect + +1. **The kernel driver is the contract.** Everything above it (libcuda, + libcudart, libnvidia-*) talks to `nvidia.ko` through stable ioctls, memory + mappings and `/dev/nvidia*` nodes. If the kernel driver initializes and + behaves, user space follows. +2. **User space is opaque to us.** We never intercept user-space APIs. All our + hooks are in the device model (host side) or in the *hardware interface* the + kernel driver sees (guest side). +3. **Version skew is normal.** The driver version, the CUDA version and the + firmware version are three independent axes. Our compatibility target is + "the driver family that matches the emulated firmware profile", with + configuration-driven adaptation for others. + +## 3. Kernel driver anatomy (what the guest brings) + +`nvidia.ko` is a large, self-contained stack. The parts that matter to +virtualization: + +| Component | Role | Why it matters to us | +|---|---|---| +| **RM core** | object model, handles, locks, device lifecycle | drives the GSP boot and the entire RPC/control surface | +| **Resource server (`resserv`)** | allocates and tracks resources (objects, memory, channels) | the client-side half of the client/server RM split (see the GSP-RM doc) | +| **Memory manager (`memmgr`)** | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page tables for its VA spaces | +| **FIFO/runlist** | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted | +| **GSP client** | message queues + RPC transport + event handling | the interface we emulate | +| **UVM** | unified memory, page-fault handling, channel manager | starts early in CUDA initialization; extremely stateful | +| **`nvidia_modeset` / `nvidia_drm`** | display/DRM | needed later for graphics | +| **CE / GR HALs** | engine programming | needed to *execute* work (forwarded) | + +## 4. Driver initialization vs. runtime + +Understanding the split between these two phases is the key to debugging: + +**Initialization (boot/probe)** — deterministic, mostly one-shot: + +``` +PCI probe → identify GPU from config space and BAR0 registers + → boot the GSP (our emulated handshake) + → read static configuration (device info tables, FB/BUS info, VBIOS, clocks) + → create root client/device/subdevice objects + → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU) + → expose /dev/nvidia* and register with the kernel +``` + +**Runtime (per-process, per-context)** — driven by user-space workloads: + +``` +open /dev/nvidia* → cuInit → UVM registration (per process) + → cuCtxCreate → object/allocations → memory maps + → channel creation + push buffers + semaphores + → kernel launches (GR) / copies (CE) / memops + → faults, completions, events +``` + +The initialization path is where compatibility is won or lost; the runtime +path is where *execution* is won or lost. Open-IOV's current focus is the +runtime path (forwarding), having already reached the end of initialization. + +## 5. What the guest expects from the device, per phase + +| Phase | Guest action | Device/firmware expectation | +|---|---|---| +| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU | +| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" | +| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables | +| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values | +| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params | +| Runtime | `GSP_RM_ALLOC`/`GSP_RM_CONTROL` for maps, channels; pushes | executed work, semaphores, faults | + +The field guide [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md) +lists the actual message/control IDs observed at each step. + +## 6. Why user-space-only interception does not work + +Two hard-won facts from this project: + +1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim + around `ioctl()` sees nothing. If you need to observe CUDA's kernel calls, + use a `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as + our tools do. +2. **The CUDA driver talks to the *kernel driver*, not to the device.** All + device interaction happens in `nvidia.ko`. This is good news: it means the + emulation boundary is a small number of kernel-side interfaces (MMIO, DMA, + interrupts, and the GSP protocol) rather than a large user-space API + surface. + +## 7. Windows guests + +Windows brings its own `nvlddmkm.sys` driver, which uses the same underlying +device/firmware model (GSP) but different host interfaces (WDDM, MSI-X +semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the +firmware interface, forward the work. Windows support is a roadmap item (see +the portability matrix), and the design goal is that no Windows-specific code +lives in the guest. + +## 8. Compatibility notes for versions + +* The guest's driver and the emulated firmware profile must agree on the RPC + message versions. The guest negotiates these at start-up + (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); our emulation answers + with a profile that matches the driver family in use. +* CUDA versions ride on driver versions; the test matrix (see + [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)) + tracks (driver, CUDA, firmware) triples. |