# The Guest Stack and CUDASW

> What runs inside the guest, how it is layered, and what it expects from the
> device. This is the "what must keep working, unmodified" document.

## 1. Vocabulary: what "CUDASW" means here

**CUDASW** is the family of NVIDIA *software* components that sit between an
application and the GPU driver. In this project the term is used deliberately
broadly, because our compatibility target is the whole stack, not one library:

```
application
   │  CUDA API (runtime)
libcudart.so ──────────────────────────────┐   CUDA runtime: device/stream/
   │  CUDA driver API                     │   event management, kernel launch
libcuda.so ────────────────────────────────┤   CUDA driver: contexts, modules,
   │  ioctls                             │   memory, channel submission
/dev/nvidia* ──────────────────────────────┘
nvidia.ko (RM + UVM + modeset)
```

Adjacent components in the same family:

* **libnvidia-ptxjitcompiler** — JIT compilation of PTX to SASS. (No GPU
  interaction beyond driver interfaces; usually "just works" once the driver
  initializes.)
* **libnvidia-fatbinaryloader / nvrtc** — fatbinary loading.
* **UVM** (`nvidia_uvm.ko`) — unified memory and GPU page-fault-driven memory
  management; see [`04-uvm.md`](04-uvm.md).
* **Graphics stack** — `libGLX_nvidia`, `libnvidia-glcore`, `libvulkan` ICD,
  plus the kernel-side `nvidia_modeset`/`nvidia_drm` (roadmap).
* **Video stack** — NVENC/NVDEC userspace + kernel interfaces (roadmap).

## 2. Layering rules the emulation must respect

1. **The kernel driver is the contract.** Everything above it (libcuda,
   libcudart, libnvidia-*) talks to `nvidia.ko` through stable ioctls, memory
   mappings and `/dev/nvidia*` nodes. If the kernel driver initializes and
   behaves, user space follows.
2. **User space is opaque to us.** We never intercept user-space APIs. All our
   hooks are in the device model (host side) or in the *hardware interface* the
   kernel driver sees (guest side).
3. **Version skew is normal.** The driver version, the CUDA version and the
   firmware version are three independent axes. Our compatibility target is
   "the driver family that matches the emulated firmware profile", with
   configuration-driven adaptation for others.

## 3. Kernel driver anatomy (what the guest brings)

`nvidia.ko` is a large, self-contained stack. The parts that matter to
virtualization:

| Component | Role | Why it matters to us |
|---|---|---|
| **RM core** | object model, handles, locks, device lifecycle | drives the GSP boot and the entire RPC/control surface |
| **Resource server (`resserv`)** | allocates and tracks resources (objects, memory, channels) | the client-side half of the client/server RM split (see the GSP-RM doc) |
| **Memory manager (`memmgr`)** | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page tables for its VA spaces |
| **FIFO/runlist** | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted |
| **GSP client** | message queues + RPC transport + event handling | the interface we emulate |
| **UVM** | unified memory, page-fault handling, channel manager | starts early in CUDA initialization; extremely stateful |
| **`nvidia_modeset` / `nvidia_drm`** | display/DRM | needed later for graphics |
| **CE / GR HALs** | engine programming | needed to *execute* work (forwarded) |

## 4. Driver initialization vs. runtime

Understanding the split between these two phases is the key to debugging:

**Initialization (boot/probe)** — deterministic, mostly one-shot:

```
PCI probe → identify GPU from config space and BAR0 registers
   → boot the GSP (our emulated handshake)
   → read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
   → create root client/device/subdevice objects
   → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
   → expose /dev/nvidia* and register with the kernel
```

**Runtime (per-process, per-context)** — driven by user-space workloads:

```
open /dev/nvidia* → cuInit → UVM registration (per process)
   → cuCtxCreate → object/allocations → memory maps
   → channel creation + push buffers + semaphores
   → kernel launches (GR) / copies (CE) / memops
   → faults, completions, events
```

The initialization path is where compatibility is won or lost; the runtime
path is where *execution* is won or lost. Context creation and the full CUDA
driver-API smoke now pass (`ALL OK`, copies matched), so the current focus is
execution: the forwarding decision is made and the backend is live — a
module-owned GPU running its real GSP firmware as the resource-manager server
(see
[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)
and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) — with
CE-class copies forwarded to the physical copy engine (default off); for GR, a
forwarded guest launch reaches the front-end but the inline launch-descriptor
install has not been observed (context validity at dispatch is ruled out; see
[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)).

For the CUDA userspace side of this stack — its architecture, how it is built
and executes, and the exact kernel/GPU interface contract we must satisfy —
see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md),
[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md)
and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md),
starting from
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).

## 5. What the guest expects from the device, per phase

| Phase | Guest action | Device/firmware expectation |
|---|---|---|
| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU |
| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" |
| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables |
| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values |
| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params |
| Runtime | `GSP_RM_ALLOC`/`GSP_RM_CONTROL` for maps, channels; pushes | executed work, semaphores, faults |

The field guide [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md)
lists the actual message/control IDs observed at each step.

## 6. Why user-space-only interception does not work

Two hard-won facts from this project:

1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim
   around `ioctl()` sees nothing. If you need to observe CUDA's kernel calls,
   use a `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as
   our tools do.
2. **The CUDA driver talks to the *kernel driver*, not to the device.** All
   device interaction happens in `nvidia.ko`. This is good news: it means the
   emulation boundary is a small number of kernel-side interfaces (MMIO, DMA,
   interrupts, and the GSP protocol) rather than a large user-space API
   surface.

## 7. Windows guests

Windows brings its own `nvlddmkm.sys` driver, which uses the same underlying
device/firmware model (GSP) but different host interfaces (WDDM, MSI-X
semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the
firmware interface, forward the work. Windows support is a roadmap item (see
the portability matrix), and the design goal is that no Windows-specific code
lives in the guest.

## 8. Compatibility notes for versions

* The guest's driver and the emulated firmware profile must agree on the RPC
  message versions. The guest negotiates these at start-up
  (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); our emulation answers
  with a profile that matches the driver family in use.
* CUDA versions ride on driver versions; the test matrix (see
  [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md))
  tracks (driver, CUDA, firmware) triples.