1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
|
# The Guest Stack and CUDASW
> What runs inside the guest, how it is layered, and what it expects from the
> device. This is the "what must keep working, unmodified" document.
## 1. Vocabulary: what "CUDASW" means here
**CUDASW** is the family of NVIDIA *software* components that sit between an
application and the GPU driver. In this project the term is used deliberately
broadly, because our compatibility target is the whole stack, not one library:
```
application
│ CUDA API (runtime)
libcudart.so ──────────────────────────────┐ CUDA runtime: device/stream/
│ CUDA driver API │ event management, kernel launch
libcuda.so ────────────────────────────────┤ CUDA driver: contexts, modules,
│ ioctls │ memory, channel submission
/dev/nvidia* ──────────────────────────────┘
nvidia.ko (RM + UVM + modeset)
```
Adjacent components in the same family:
* **libnvidia-ptxjitcompiler** — JIT compilation of PTX to SASS. (No GPU
interaction beyond driver interfaces; usually "just works" once the driver
initializes.)
* **libnvidia-fatbinaryloader / nvrtc** — fatbinary loading.
* **UVM** (`nvidia_uvm.ko`) — unified memory and GPU page-fault-driven memory
management; see [`04-uvm.md`](04-uvm.md).
* **Graphics stack** — `libGLX_nvidia`, `libnvidia-glcore`, `libvulkan` ICD,
plus the kernel-side `nvidia_modeset`/`nvidia_drm` (roadmap).
* **Video stack** — NVENC/NVDEC userspace + kernel interfaces (roadmap).
## 2. Layering rules the emulation must respect
1. **The kernel driver is the contract.** Everything above it (libcuda,
libcudart, libnvidia-*) talks to `nvidia.ko` through stable ioctls, memory
mappings and `/dev/nvidia*` nodes. If the kernel driver initializes and
behaves, user space follows.
2. **User space is opaque to us.** We never intercept user-space APIs. All our
hooks are in the device model (host side) or in the *hardware interface* the
kernel driver sees (guest side).
3. **Version skew is normal.** The driver version, the CUDA version and the
firmware version are three independent axes. Our compatibility target is
"the driver family that matches the emulated firmware profile", with
configuration-driven adaptation for others.
## 3. Kernel driver anatomy (what the guest brings)
`nvidia.ko` is a large, self-contained stack. The parts that matter to
virtualization:
| Component | Role | Why it matters to us |
|---|---|---|
| **RM core** | object model, handles, locks, device lifecycle | drives the GSP boot and the entire RPC/control surface |
| **Resource server (`resserv`)** | allocates and tracks resources (objects, memory, channels) | the client-side half of the client/server RM split (see the GSP-RM doc) |
| **Memory manager (`memmgr`)** | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page tables for its VA spaces |
| **FIFO/runlist** | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted |
| **GSP client** | message queues + RPC transport + event handling | the interface we emulate |
| **UVM** | unified memory, page-fault handling, channel manager | starts early in CUDA initialization; extremely stateful |
| **`nvidia_modeset` / `nvidia_drm`** | display/DRM | needed later for graphics |
| **CE / GR HALs** | engine programming | needed to *execute* work (forwarded) |
## 4. Driver initialization vs. runtime
Understanding the split between these two phases is the key to debugging:
**Initialization (boot/probe)** — deterministic, mostly one-shot:
```
PCI probe → identify GPU from config space and BAR0 registers
→ boot the GSP (our emulated handshake)
→ read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
→ create root client/device/subdevice objects
→ initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
→ expose /dev/nvidia* and register with the kernel
```
**Runtime (per-process, per-context)** — driven by user-space workloads:
```
open /dev/nvidia* → cuInit → UVM registration (per process)
→ cuCtxCreate → object/allocations → memory maps
→ channel creation + push buffers + semaphores
→ kernel launches (GR) / copies (CE) / memops
→ faults, completions, events
```
The initialization path is where compatibility is won or lost; the runtime
path is where *execution* is won or lost. Open-IOV's current focus is
pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context
creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
list lacks the async copy engine (`COPY14`); the forwarding decision
(host-RM proxy vs direct submission) is still ahead (see
[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)).
For the CUDA userspace side of this stack — its architecture, how it is built
and executes, and the exact kernel/GPU interface contract we must satisfy —
see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md),
[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md)
and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md),
starting from
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 5. What the guest expects from the device, per phase
| Phase | Guest action | Device/firmware expectation |
|---|---|---|
| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU |
| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" |
| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables |
| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values |
| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params |
| Runtime | `GSP_RM_ALLOC`/`GSP_RM_CONTROL` for maps, channels; pushes | executed work, semaphores, faults |
The field guide [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md)
lists the actual message/control IDs observed at each step.
## 6. Why user-space-only interception does not work
Two hard-won facts from this project:
1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim
around `ioctl()` sees nothing. If you need to observe CUDA's kernel calls,
use a `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as
our tools do.
2. **The CUDA driver talks to the *kernel driver*, not to the device.** All
device interaction happens in `nvidia.ko`. This is good news: it means the
emulation boundary is a small number of kernel-side interfaces (MMIO, DMA,
interrupts, and the GSP protocol) rather than a large user-space API
surface.
## 7. Windows guests
Windows brings its own `nvlddmkm.sys` driver, which uses the same underlying
device/firmware model (GSP) but different host interfaces (WDDM, MSI-X
semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the
firmware interface, forward the work. Windows support is a roadmap item (see
the portability matrix), and the design goal is that no Windows-specific code
lives in the guest.
## 8. Compatibility notes for versions
* The guest's driver and the emulated firmware profile must agree on the RPC
message versions. The guest negotiates these at start-up
(`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); our emulation answers
with a profile that matches the driver family in use.
* CUDA versions ride on driver versions; the test matrix (see
[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md))
tracks (driver, CUDA, firmware) triples.
|
