# The Guest Stack and CUDASW

> **Scope.** What runs inside the guest, how the NVIDIA software layers sit on
> top of the device, and what the unmodified guest driver expects from the
> emulated GPU. Canonical source:
> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md);
> the docs tree wins on disagreement with this page.
>
> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with
> the stock guest driver; `[gated]` implemented default-off behind supervised
> gates; `[wall]` known blocking limit; `[target]` designed, not built;
> `[planned]` roadmap.

## 1. Vocabulary: what "CUDASW" means here

**CUDASW** is the family of NVIDIA *software* components that sit between an
application and the GPU driver. The project's compatibility target is the
whole stack, not one library:

```text
application
   │  CUDA API (runtime)
libcudart.so ─────────────────────────────┐  runtime: device/stream/event
   │  CUDA driver API                    │  management, kernel launch
libcuda.so ───────────────────────────────┤  driver: contexts, modules,
   │  ioctls                             │  memory, channel submission
/dev/nvidia* ─────────────────────────────┘
nvidia.ko (RM + UVM + modeset)
```

Adjacent members of the family: **libnvidia-ptxjitcompiler** (PTX→SASS JIT —
usually "just works" once the driver initializes), **libnvidia-fatbinaryloader
/ nvrtc**, **UVM** (`nvidia_uvm.ko` — [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)),
the **graphics stack** (`libGLX_nvidia`, `libnvidia-glcore`, Vulkan ICD,
`nvidia_modeset`/`nvidia_drm`) and the **video stack** (NVENC/NVDEC) — both
roadmapped.

## 2. Layering rules the emulation must respect

1. **The kernel driver is the contract.** Everything above it talks to
   `nvidia.ko` through stable ioctls, memory mappings and `/dev/nvidia*`
   nodes. If the kernel driver initializes and behaves, user space follows.
2. **User space is opaque to us.** We never intercept user-space APIs. All
   hooks are in the device model (host side) or in the *hardware interface*
   the kernel driver sees (guest side).
3. **Version skew is normal.** Driver version, CUDA version and firmware
   version are three independent axes. The compatibility target is "the
   driver family that matches the emulated firmware profile", with
   configuration-driven adaptation for others.

## 3. Kernel-driver anatomy (what the guest brings)

| Component | Role | Why it matters here |
|---|---|---|
| RM core | object model, handles, locks, device lifecycle | drives the GSP boot and the whole RPC/control surface |
| Resource server (`resserv`) | allocates/tracks objects, memory, channels | the client half of the client/server RM split |
| Memory manager (`memmgr`) | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page-table leaves for its VA spaces |
| FIFO/runlist | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted |
| GSP client | message queues, RPC transport, event handling | the interface we emulate |
| UVM | unified memory, page faults, channel manager | starts early in CUDA init, extremely stateful |
| `nvidia_modeset` / `nvidia_drm` | display / DRM | needed later for graphics |
| CE / GR HALs | engine programming | needed to *execute* forwarded work |

## 4. Driver initialization vs runtime

Understanding this split is the key to debugging.

**Initialization (boot/probe)** — deterministic, mostly one-shot; where
compatibility is won or lost [live]:

```text
PCI probe → identify GPU from config space and BAR0 registers
   → boot the GSP (our emulated handshake)
   → read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
   → create root client/device/subdevice objects
   → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
   → expose /dev/nvidia* and register with the kernel
```

The actual message/control IDs observed at each step are listed in
[`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md).

**Runtime (per-process, per-context)** — driven by user-space workloads;
where *execution* is won or lost:

```text
open /dev/nvidia* → cuInit → UVM registration (per process)
   → cuCtxCreate → object/allocations → memory maps
   → channel creation + push buffers + semaphores
   → kernel launches (GR) / copies (CE) / memops
   → faults, completions, events
```

Context creation and the full CUDA driver-API smoke now pass (`ALL OK`,
copies matched) [live]. The current focus is execution: the backend is a
**module-owned GPU running its real GSP firmware as the resource-manager
server** ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)), with
CE-class copies forwarded to the physical copy engine default-off [gated].
For GR, a forwarded guest launch reaches the front-end but the inline
launch-descriptor install has not been observed — the open execution wall
([`Execution-Frontier.md`](Execution-Frontier.md)) [wall].

## 5. What the guest expects from the device, per phase

| Phase | Guest action | Device/firmware expectation |
|---|---|---|
| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU |
| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" |
| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables |
| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values |
| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params |
| Runtime | allocs/controls for maps and channels; pushes | executed work, semaphores, faults |

The RPC surface is described in [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) and
[`RPC-Message-Format.md`](RPC-Message-Format.md); the boot handshake in
[`WPR-Management.md`](WPR-Management.md).

## 6. Why user-space-only interception does not work

Two hard-won facts from this project:

1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim
   around `ioctl()` sees nothing. Observing CUDA's kernel calls requires a
   `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as the
   project's diagnostics do.
2. **The CUDA driver talks to the *kernel driver*, not to the device.** All
   device interaction happens in `nvidia.ko`. This is good news: the
   emulation boundary is a small set of kernel-side interfaces (MMIO, DMA,
   interrupts, the GSP protocol) rather than a large user-space API surface.

## 7. Windows guests

Windows brings its own kernel-mode driver, `nvlddmkm.sys`, which uses the same
underlying GSP device/firmware model but different host interfaces (WDDM,
MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy
the firmware interface, forward the work. Windows support is a roadmap item
([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) [planned], and
the design goal is that no Windows-specific code lives in the guest.

## 8. Compatibility notes for versions

* The guest's driver and the emulated firmware profile must agree on the RPC
  message versions. The guest negotiates these at start-up
  (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); the emulation
  answers with a profile that matches the driver family in use.
* CUDA versions ride on driver versions; the test matrix tracks
  (driver, CUDA, firmware) triples
  ([`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md)).

## 9. Related documentation

* [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) — canonical chapter.
* [`Host-Module.md`](Host-Module.md) — the other side of the boundary.
* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — user-space driver internals.
* [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the observed protocol reality.
* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md) — where guest work is headed.