diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-12 00:32:00 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-12 00:32:00 -0400 |
| commit | cff1461ef459e98dea063abc72ae0819fea7b5ce (patch) | |
| tree | dbceb906db4bb6c2e0e621eed2515a06e701236d /wiki/Guest-Stack-and-CUDASW.md | |
| parent | 40a01810c77838b3876ecea8fed2f5d72e08a378 (diff) | |
| download | open-iov-cff1461ef459e98dea063abc72ae0819fea7b5ce.tar.gz open-iov-cff1461ef459e98dea063abc72ae0819fea7b5ce.zip | |
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link
mapping, local-marker scan, About-message insert), WikiBot publisher
with diff/check modes and publish manifest, live site auditor, image
uploader, site-only page sources (landing page, About, index, sidebar,
banner templates, About-message template).
- wiki: new public deep-dive sources (project status, install, FAQ,
system overview, GSP and GSP-RM, execution frontier, host module,
guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger,
hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace
status; corrected cross-document links; README rewritten for the new set.
- wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/Guest-Stack-and-CUDASW.md')
| -rw-r--r-- | wiki/Guest-Stack-and-CUDASW.md | 156 |
1 files changed, 156 insertions, 0 deletions
diff --git a/wiki/Guest-Stack-and-CUDASW.md b/wiki/Guest-Stack-and-CUDASW.md new file mode 100644 index 0000000..1548f63 --- /dev/null +++ b/wiki/Guest-Stack-and-CUDASW.md @@ -0,0 +1,156 @@ +# The Guest Stack and CUDASW + +> **Scope.** What runs inside the guest, how the NVIDIA software layers sit on +> top of the device, and what the unmodified guest driver expects from the +> emulated GPU. Canonical source: +> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md); +> the docs tree wins on disagreement with this page. +> +> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with +> the stock guest driver; `[gated]` implemented default-off behind supervised +> gates; `[wall]` known blocking limit; `[target]` designed, not built; +> `[planned]` roadmap. + +## 1. Vocabulary: what "CUDASW" means here + +**CUDASW** is the family of NVIDIA *software* components that sit between an +application and the GPU driver. The project's compatibility target is the +whole stack, not one library: + +```text +application + │ CUDA API (runtime) +libcudart.so ─────────────────────────────┐ runtime: device/stream/event + │ CUDA driver API │ management, kernel launch +libcuda.so ───────────────────────────────┤ driver: contexts, modules, + │ ioctls │ memory, channel submission +/dev/nvidia* ─────────────────────────────┘ +nvidia.ko (RM + UVM + modeset) +``` + +Adjacent members of the family: **libnvidia-ptxjitcompiler** (PTX→SASS JIT — +usually "just works" once the driver initializes), **libnvidia-fatbinaryloader +/ nvrtc**, **UVM** (`nvidia_uvm.ko` — [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)), +the **graphics stack** (`libGLX_nvidia`, `libnvidia-glcore`, Vulkan ICD, +`nvidia_modeset`/`nvidia_drm`) and the **video stack** (NVENC/NVDEC) — both +roadmapped. + +## 2. Layering rules the emulation must respect + +1. **The kernel driver is the contract.** Everything above it talks to + `nvidia.ko` through stable ioctls, memory mappings and `/dev/nvidia*` + nodes. If the kernel driver initializes and behaves, user space follows. +2. **User space is opaque to us.** We never intercept user-space APIs. All + hooks are in the device model (host side) or in the *hardware interface* + the kernel driver sees (guest side). +3. **Version skew is normal.** Driver version, CUDA version and firmware + version are three independent axes. The compatibility target is "the + driver family that matches the emulated firmware profile", with + configuration-driven adaptation for others. + +## 3. Kernel-driver anatomy (what the guest brings) + +| Component | Role | Why it matters here | +|---|---|---| +| RM core | object model, handles, locks, device lifecycle | drives the GSP boot and the whole RPC/control surface | +| Resource server (`resserv`) | allocates/tracks objects, memory, channels | the client half of the client/server RM split | +| Memory manager (`memmgr`) | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page-table leaves for its VA spaces | +| FIFO/runlist | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted | +| GSP client | message queues, RPC transport, event handling | the interface we emulate | +| UVM | unified memory, page faults, channel manager | starts early in CUDA init, extremely stateful | +| `nvidia_modeset` / `nvidia_drm` | display / DRM | needed later for graphics | +| CE / GR HALs | engine programming | needed to *execute* forwarded work | + +## 4. Driver initialization vs runtime + +Understanding this split is the key to debugging. + +**Initialization (boot/probe)** — deterministic, mostly one-shot; where +compatibility is won or lost [live]: + +```text +PCI probe → identify GPU from config space and BAR0 registers + → boot the GSP (our emulated handshake) + → read static configuration (device info tables, FB/BUS info, VBIOS, clocks) + → create root client/device/subdevice objects + → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU) + → expose /dev/nvidia* and register with the kernel +``` + +The actual message/control IDs observed at each step are listed in +[`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md). + +**Runtime (per-process, per-context)** — driven by user-space workloads; +where *execution* is won or lost: + +```text +open /dev/nvidia* → cuInit → UVM registration (per process) + → cuCtxCreate → object/allocations → memory maps + → channel creation + push buffers + semaphores + → kernel launches (GR) / copies (CE) / memops + → faults, completions, events +``` + +Context creation and the full CUDA driver-API smoke now pass (`ALL OK`, +copies matched) [live]. The current focus is execution: the backend is a +**module-owned GPU running its real GSP firmware as the resource-manager +server** ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)), with +CE-class copies forwarded to the physical copy engine default-off [gated]. +For GR, a forwarded guest launch reaches the front-end but the inline +launch-descriptor install has not been observed — the open execution wall +([`Execution-Frontier.md`](Execution-Frontier.md)) [wall]. + +## 5. What the guest expects from the device, per phase + +| Phase | Guest action | Device/firmware expectation | +|---|---|---| +| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU | +| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" | +| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables | +| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values | +| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params | +| Runtime | allocs/controls for maps and channels; pushes | executed work, semaphores, faults | + +The RPC surface is described in [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) and +[`RPC-Message-Format.md`](RPC-Message-Format.md); the boot handshake in +[`WPR-Management.md`](WPR-Management.md). + +## 6. Why user-space-only interception does not work + +Two hard-won facts from this project: + +1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim + around `ioctl()` sees nothing. Observing CUDA's kernel calls requires a + `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as the + project's diagnostics do. +2. **The CUDA driver talks to the *kernel driver*, not to the device.** All + device interaction happens in `nvidia.ko`. This is good news: the + emulation boundary is a small set of kernel-side interfaces (MMIO, DMA, + interrupts, the GSP protocol) rather than a large user-space API surface. + +## 7. Windows guests + +Windows brings its own kernel-mode driver, `nvlddmkm.sys`, which uses the same +underlying GSP device/firmware model but different host interfaces (WDDM, +MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy +the firmware interface, forward the work. Windows support is a roadmap item +([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) [planned], and +the design goal is that no Windows-specific code lives in the guest. + +## 8. Compatibility notes for versions + +* The guest's driver and the emulated firmware profile must agree on the RPC + message versions. The guest negotiates these at start-up + (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); the emulation + answers with a profile that matches the driver family in use. +* CUDA versions ride on driver versions; the test matrix tracks + (driver, CUDA, firmware) triples + ([`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md)). + +## 9. Related documentation + +* [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) — canonical chapter. +* [`Host-Module.md`](Host-Module.md) — the other side of the boundary. +* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — user-space driver internals. +* [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the observed protocol reality. +* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md) — where guest work is headed. |
