summaryrefslogtreecommitdiffstats
path: root/wiki/Guest-Stack-and-CUDASW.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
commit5df3147130c9254cd5bea34b330a2a42ee51d635 (patch)
tree6a8fdf54bbb291eaccbeef8334e2cdd70af2caa8 /wiki/Guest-Stack-and-CUDASW.md
parentf1edf479a167898aaf70dc8f21e9cf18bccbe112 (diff)
downloadopen-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.tar.gz
open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.zip
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link mapping, local-marker scan, About-message insert), WikiBot publisher with diff/check modes and publish manifest, live site auditor, image uploader, site-only page sources (landing page, About, index, sidebar, banner templates, About-message template). - wiki: new public deep-dive sources (project status, install, FAQ, system overview, GSP and GSP-RM, execution frontier, host module, guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger, hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace status; corrected cross-document links; README rewritten for the new set. - wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/Guest-Stack-and-CUDASW.md')
-rw-r--r--wiki/Guest-Stack-and-CUDASW.md156
1 files changed, 156 insertions, 0 deletions
diff --git a/wiki/Guest-Stack-and-CUDASW.md b/wiki/Guest-Stack-and-CUDASW.md
new file mode 100644
index 0000000..1548f63
--- /dev/null
+++ b/wiki/Guest-Stack-and-CUDASW.md
@@ -0,0 +1,156 @@
+# The Guest Stack and CUDASW
+
+> **Scope.** What runs inside the guest, how the NVIDIA software layers sit on
+> top of the device, and what the unmodified guest driver expects from the
+> emulated GPU. Canonical source:
+> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md);
+> the docs tree wins on disagreement with this page.
+>
+> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with
+> the stock guest driver; `[gated]` implemented default-off behind supervised
+> gates; `[wall]` known blocking limit; `[target]` designed, not built;
+> `[planned]` roadmap.
+
+## 1. Vocabulary: what "CUDASW" means here
+
+**CUDASW** is the family of NVIDIA *software* components that sit between an
+application and the GPU driver. The project's compatibility target is the
+whole stack, not one library:
+
+```text
+application
+ │ CUDA API (runtime)
+libcudart.so ─────────────────────────────┐ runtime: device/stream/event
+ │ CUDA driver API │ management, kernel launch
+libcuda.so ───────────────────────────────┤ driver: contexts, modules,
+ │ ioctls │ memory, channel submission
+/dev/nvidia* ─────────────────────────────┘
+nvidia.ko (RM + UVM + modeset)
+```
+
+Adjacent members of the family: **libnvidia-ptxjitcompiler** (PTX→SASS JIT —
+usually "just works" once the driver initializes), **libnvidia-fatbinaryloader
+/ nvrtc**, **UVM** (`nvidia_uvm.ko` — [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)),
+the **graphics stack** (`libGLX_nvidia`, `libnvidia-glcore`, Vulkan ICD,
+`nvidia_modeset`/`nvidia_drm`) and the **video stack** (NVENC/NVDEC) — both
+roadmapped.
+
+## 2. Layering rules the emulation must respect
+
+1. **The kernel driver is the contract.** Everything above it talks to
+ `nvidia.ko` through stable ioctls, memory mappings and `/dev/nvidia*`
+ nodes. If the kernel driver initializes and behaves, user space follows.
+2. **User space is opaque to us.** We never intercept user-space APIs. All
+ hooks are in the device model (host side) or in the *hardware interface*
+ the kernel driver sees (guest side).
+3. **Version skew is normal.** Driver version, CUDA version and firmware
+ version are three independent axes. The compatibility target is "the
+ driver family that matches the emulated firmware profile", with
+ configuration-driven adaptation for others.
+
+## 3. Kernel-driver anatomy (what the guest brings)
+
+| Component | Role | Why it matters here |
+|---|---|---|
+| RM core | object model, handles, locks, device lifecycle | drives the GSP boot and the whole RPC/control surface |
+| Resource server (`resserv`) | allocates/tracks objects, memory, channels | the client half of the client/server RM split |
+| Memory manager (`memmgr`) | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page-table leaves for its VA spaces |
+| FIFO/runlist | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted |
+| GSP client | message queues, RPC transport, event handling | the interface we emulate |
+| UVM | unified memory, page faults, channel manager | starts early in CUDA init, extremely stateful |
+| `nvidia_modeset` / `nvidia_drm` | display / DRM | needed later for graphics |
+| CE / GR HALs | engine programming | needed to *execute* forwarded work |
+
+## 4. Driver initialization vs runtime
+
+Understanding this split is the key to debugging.
+
+**Initialization (boot/probe)** — deterministic, mostly one-shot; where
+compatibility is won or lost [live]:
+
+```text
+PCI probe → identify GPU from config space and BAR0 registers
+ → boot the GSP (our emulated handshake)
+ → read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
+ → create root client/device/subdevice objects
+ → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
+ → expose /dev/nvidia* and register with the kernel
+```
+
+The actual message/control IDs observed at each step are listed in
+[`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md).
+
+**Runtime (per-process, per-context)** — driven by user-space workloads;
+where *execution* is won or lost:
+
+```text
+open /dev/nvidia* → cuInit → UVM registration (per process)
+ → cuCtxCreate → object/allocations → memory maps
+ → channel creation + push buffers + semaphores
+ → kernel launches (GR) / copies (CE) / memops
+ → faults, completions, events
+```
+
+Context creation and the full CUDA driver-API smoke now pass (`ALL OK`,
+copies matched) [live]. The current focus is execution: the backend is a
+**module-owned GPU running its real GSP firmware as the resource-manager
+server** ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)), with
+CE-class copies forwarded to the physical copy engine default-off [gated].
+For GR, a forwarded guest launch reaches the front-end but the inline
+launch-descriptor install has not been observed — the open execution wall
+([`Execution-Frontier.md`](Execution-Frontier.md)) [wall].
+
+## 5. What the guest expects from the device, per phase
+
+| Phase | Guest action | Device/firmware expectation |
+|---|---|---|
+| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU |
+| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" |
+| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables |
+| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values |
+| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params |
+| Runtime | allocs/controls for maps and channels; pushes | executed work, semaphores, faults |
+
+The RPC surface is described in [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) and
+[`RPC-Message-Format.md`](RPC-Message-Format.md); the boot handshake in
+[`WPR-Management.md`](WPR-Management.md).
+
+## 6. Why user-space-only interception does not work
+
+Two hard-won facts from this project:
+
+1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim
+ around `ioctl()` sees nothing. Observing CUDA's kernel calls requires a
+ `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as the
+ project's diagnostics do.
+2. **The CUDA driver talks to the *kernel driver*, not to the device.** All
+ device interaction happens in `nvidia.ko`. This is good news: the
+ emulation boundary is a small set of kernel-side interfaces (MMIO, DMA,
+ interrupts, the GSP protocol) rather than a large user-space API surface.
+
+## 7. Windows guests
+
+Windows brings its own kernel-mode driver, `nvlddmkm.sys`, which uses the same
+underlying GSP device/firmware model but different host interfaces (WDDM,
+MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy
+the firmware interface, forward the work. Windows support is a roadmap item
+([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) [planned], and
+the design goal is that no Windows-specific code lives in the guest.
+
+## 8. Compatibility notes for versions
+
+* The guest's driver and the emulated firmware profile must agree on the RPC
+ message versions. The guest negotiates these at start-up
+ (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); the emulation
+ answers with a profile that matches the driver family in use.
+* CUDA versions ride on driver versions; the test matrix tracks
+ (driver, CUDA, firmware) triples
+ ([`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md)).
+
+## 9. Related documentation
+
+* [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) — canonical chapter.
+* [`Host-Module.md`](Host-Module.md) — the other side of the boundary.
+* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — user-space driver internals.
+* [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the observed protocol reality.
+* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md) — where guest work is headed.