1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
|
# The Guest Stack and CUDASW
> **Scope.** What runs inside the guest, how the NVIDIA software layers sit on
> top of the device, and what the unmodified guest driver expects from the
> emulated GPU. Canonical source:
> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md);
> the docs tree wins on disagreement with this page.
>
> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with
> the stock guest driver; `[gated]` implemented default-off behind supervised
> gates; `[wall]` known blocking limit; `[target]` designed, not built;
> `[planned]` roadmap.
## 1. Vocabulary: what "CUDASW" means here
**CUDASW** is the family of NVIDIA *software* components that sit between an
application and the GPU driver. The project's compatibility target is the
whole stack, not one library:
```text
application
│ CUDA API (runtime)
libcudart.so ─────────────────────────────┐ runtime: device/stream/event
│ CUDA driver API │ management, kernel launch
libcuda.so ───────────────────────────────┤ driver: contexts, modules,
│ ioctls │ memory, channel submission
/dev/nvidia* ─────────────────────────────┘
nvidia.ko (RM + UVM + modeset)
```
Adjacent members of the family: **libnvidia-ptxjitcompiler** (PTX→SASS JIT —
usually "just works" once the driver initializes), **libnvidia-fatbinaryloader
/ nvrtc**, **UVM** (`nvidia_uvm.ko` — [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)),
the **graphics stack** (`libGLX_nvidia`, `libnvidia-glcore`, Vulkan ICD,
`nvidia_modeset`/`nvidia_drm`) and the **video stack** (NVENC/NVDEC) — both
roadmapped.
## 2. Layering rules the emulation must respect
1. **The kernel driver is the contract.** Everything above it talks to
`nvidia.ko` through stable ioctls, memory mappings and `/dev/nvidia*`
nodes. If the kernel driver initializes and behaves, user space follows.
2. **User space is opaque to us.** We never intercept user-space APIs. All
hooks are in the device model (host side) or in the *hardware interface*
the kernel driver sees (guest side).
3. **Version skew is normal.** Driver version, CUDA version and firmware
version are three independent axes. The compatibility target is "the
driver family that matches the emulated firmware profile", with
configuration-driven adaptation for others.
## 3. Kernel-driver anatomy (what the guest brings)
| Component | Role | Why it matters here |
|---|---|---|
| RM core | object model, handles, locks, device lifecycle | drives the GSP boot and the whole RPC/control surface |
| Resource server (`resserv`) | allocates/tracks objects, memory, channels | the client half of the client/server RM split |
| Memory manager (`memmgr`) | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page-table leaves for its VA spaces |
| FIFO/runlist | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted |
| GSP client | message queues, RPC transport, event handling | the interface we emulate |
| UVM | unified memory, page faults, channel manager | starts early in CUDA init, extremely stateful |
| `nvidia_modeset` / `nvidia_drm` | display / DRM | needed later for graphics |
| CE / GR HALs | engine programming | needed to *execute* forwarded work |
## 4. Driver initialization vs runtime
Understanding this split is the key to debugging.
**Initialization (boot/probe)** — deterministic, mostly one-shot; where
compatibility is won or lost [live]:
```text
PCI probe → identify GPU from config space and BAR0 registers
→ boot the GSP (our emulated handshake)
→ read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
→ create root client/device/subdevice objects
→ initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
→ expose /dev/nvidia* and register with the kernel
```
The actual message/control IDs observed at each step are listed in
[`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md).
**Runtime (per-process, per-context)** — driven by user-space workloads;
where *execution* is won or lost:
```text
open /dev/nvidia* → cuInit → UVM registration (per process)
→ cuCtxCreate → object/allocations → memory maps
→ channel creation + push buffers + semaphores
→ kernel launches (GR) / copies (CE) / memops
→ faults, completions, events
```
Context creation and the full CUDA driver-API smoke now pass (`ALL OK`,
copies matched) [live]. The current focus is execution: the backend is a
**module-owned GPU running its real GSP firmware as the resource-manager
server** ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)), with
CE-class copies forwarded to the physical copy engine default-off [gated].
For GR, a forwarded guest launch reaches the front-end but the inline
launch-descriptor install has not been observed — the open execution wall
([`Execution-Frontier.md`](Execution-Frontier.md)) [wall].
## 5. What the guest expects from the device, per phase
| Phase | Guest action | Device/firmware expectation |
|---|---|---|
| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU |
| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" |
| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables |
| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values |
| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params |
| Runtime | allocs/controls for maps and channels; pushes | executed work, semaphores, faults |
The RPC surface is described in [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) and
[`RPC-Message-Format.md`](RPC-Message-Format.md); the boot handshake in
[`WPR-Management.md`](WPR-Management.md).
## 6. Why user-space-only interception does not work
Two hard-won facts from this project:
1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim
around `ioctl()` sees nothing. Observing CUDA's kernel calls requires a
`ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as the
project's diagnostics do.
2. **The CUDA driver talks to the *kernel driver*, not to the device.** All
device interaction happens in `nvidia.ko`. This is good news: the
emulation boundary is a small set of kernel-side interfaces (MMIO, DMA,
interrupts, the GSP protocol) rather than a large user-space API surface.
## 7. Windows guests
Windows brings its own kernel-mode driver, `nvlddmkm.sys`, which uses the same
underlying GSP device/firmware model but different host interfaces (WDDM,
MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy
the firmware interface, forward the work. Windows support is a roadmap item
([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) [planned], and
the design goal is that no Windows-specific code lives in the guest.
## 8. Compatibility notes for versions
* The guest's driver and the emulated firmware profile must agree on the RPC
message versions. The guest negotiates these at start-up
(`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); the emulation
answers with a profile that matches the driver family in use.
* CUDA versions ride on driver versions; the test matrix tracks
(driver, CUDA, firmware) triples
([`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md)).
## 9. Related documentation
* [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) — canonical chapter.
* [`Host-Module.md`](Host-Module.md) — the other side of the boundary.
* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — user-space driver internals.
* [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the observed protocol reality.
* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md) — where guest work is headed.
|
