1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
|
# GMMU, AMAPlib, HALs, Runlists, and GR
> The engine side: how addresses translate, how work is scheduled, and what
> "GR" actually is. Read this when you need to know what the guest's work
> *means*.
## 1. GMMU — the GPU memory management unit
The GMMU translates the GPU virtual addresses (VAs) used by engines into
physical addresses (VRAM offsets or system-memory addresses).
### 1.1 Page-table hierarchy
NVIDIA's GMMU uses a multi-level page table whose shape depends on the AS
generation. Turing (the reference) uses a four-level hierarchy with entry
coverage:
| Level | Entries | VA bits consumed | Coverage per entry |
|---|---|---|---|
| Root page directory (PDB) | 4 | top bits | large |
| PDE level 1 | 512 | 9 bits | e.g. 2^38 |
| PDE level 2 | 512 | 9 bits | e.g. 2^29 |
| PTE level | 512+ | 9+ bits | 4 KiB / 64 KiB / 2 MiB pages |
Verified properties (from the live guest):
* **PTE encoding: `PA[47:12] | flags`**, where `flags = 0xe` for a valid
system-memory leaf entry in the format the client produces.
* Page sizes in use: 4 KiB (system memory objects) and larger (big pages) for
large allocations.
* **Apertures**: during enumeration the guest uses `ADDR_SYSMEM = 1` and
`ADDR_FBMEM = 2` to describe memory descriptors (e.g. channel instance
memory in VRAM at offset `0xffed000`, method buffers in system memory at a
physical address). The client hands the **physical** address in these
descriptors — a fact we rely on for the memory model.
* The page-table geometry arrives over
`NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` (level physical
addresses, page shifts, VA range) — see
[`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §5.
### 1.2 What "aperture" means for us
Two address spaces matter:
* **System memory (sysmem)**: guest RAM. For a virtual device, the "GPU's view"
of guest memory is established by the host (in real vGPU, by the vGPU manager
via the platform IOMMU; here, by our module).
* **Video memory (vidmem / FB)**: the GPU's local memory. In the current
emulation this is a host RAM buffer exposed as the BAR1 aperture; the
forwarding phase replaces it with real GPU memory.
## 2. AMAPlib and the HALs
NVIDIA's driver abstracts hardware in layers:
```
driver logic (memmgr, FIFO, engines)
│
├─ HAL layer: per-generation C functions that program registers and formats
│ e.g. gmmu HAL (page-table formats, invalidations),
│ engine HALs (CE/GR methods), host HAL (doorbells/PUT)
│
└─ AMAPlib: the "address mapping" library shared by RM and simulations
(DRAM swizzling, bank/channel mapping, physical-to-aperture rules)
```
Developers study two component families to understand these layers:
* **AMAPlib** — the shared "address mapping" component used by the RM and by
hardware simulations: DRAM swizzling, bank/channel mapping, aperture rules.
* **The HALs** — per-generation C code that programs registers and formats
page-table entries and engine commands.
Our knowledge of them comes from reverse engineering and decompilation of
driver and firmware behaviour, plus measurement of live devices. What is
published here is our observations, numeric IDs and layouts — which is what
another engineer needs.
Why we care:
* When we **populate page tables ourselves** (server-owned ranges), we must
produce entries the hardware and the guest both accept — AMAPlib/HAL code is
the specification.
* When we **forward work**, page tables we write must match what real engines
expect, including big-page encodings and invalidation semantics.
## 3. Channels, GPFIFO, USERD, doorbells
A **channel** is a hardware execution context. Its components:
| Piece | Where | Purpose |
|---|---|---|
| **GPFIFO** | GPU memory (sysmem or vidmem) | ring of entries; each points to a push buffer |
| **Push buffer** | GPU memory | the actual method stream the engine executes |
| **USERD** | vidmem (per channel) | channel control block: PUT/GET pointers, doorbell-related state |
| **Instance block** | vidmem or sysmem | the hardware context (engine state, page directory base, etc.) |
| **Runlist / TSG** | scheduled by the host | the list of channels currently eligible to run |
The submission flow (what a driver does, and what the guest therefore does):
```
write methods into a push buffer (CPU, via a mapping)
write a GPFIFO entry pointing at the push (CPU)
update the channel's PUT pointer (CPU → USERD or doorbell register)
... the scheduler eventually runs the channel; the engine fetches the push
... on completion the engine writes the GET pointer and/or a semaphore
```
Two details that repeatedly matter in practice:
* **The push buffer format** is a sequence of method/data pairs preceded by
*incrementing-method headers* that encode the method offset (`address >> 2`),
a subchannel and a count. A decoder for this format is the first tool you
need to read a trace (our `scan_seq` module parameter found UVM's pushes by
searching for the decoded data pattern).
* **Engine classes** are the vocabulary of pushes: `TURING_DMA_COPY_A`
(0xc5b5) for copy engines, `TURING_CHANNEL_GPFIFO_A` (0xc46f) for channel
objects, `GP100_UVM_SW` (0xc076) for UVM's software object, etc.
## 4. Runlist and scheduling
The GPU schedules channels through a **runlist**: a list of channel
descriptors the scheduler walks, giving each channel a time slice. A **TSG**
(Time-Slice Group) bundles channels that share a scheduling slot — this is the
mechanism for fair sharing and for interleaving contexts.
For virtualization this is a focal point:
* In real vGPU, the host vGPU manager owns the runlist and interleaves
guest channels with host channels.
* In our target architecture, *we* own the schedule: guest channels must be
mapped onto real GPU scheduling primitives with correct isolation (a guest
must never observe or influence another guest's channels).
* The guest driver observes scheduling only indirectly: PUT/GET progress,
semaphores, and fault timing. Faithful *semantics* (not exact timing) is the
requirement.
## 5. GR — the graphics/compute engine
**GR** is the engine that executes shaders: both graphics pipelines and CUDA
compute kernels. It is the reason a GPU is a GPU.
What GR needs to run work:
* **A context** (the "golden context" + per-context state): register state,
memory layout, page directory, shader code.
* **A channel** with a push buffer that issues *method* calls to GR:
loading shader code, binding surfaces, launching grids (`QMD`s in modern
architectures), and signaling completion.
* **Memory** containing the shader binary (SASS), kernel parameters,
descriptors and data.
* **Interrupts/semaphores** to signal completion and errors.
For CUDA specifically:
```
cuLaunchKernel
→ CUDA driver builds a GR push (grid config, QMD, parameters)
→ submits on the process's GR channel
→ GR executes the shader on real hardware
→ completion semaphore / event; UVM handles memory effects
```
**Consequence for Open-IOV:** CUDA kernel execution is *real hardware
execution*. No emulation of GR can be correct and fast enough; the only viable
path is forwarding the guest's GR work to real hardware (see
[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and
the challenges document). This is also why "CUDA works" is a much stronger
statement than "`nvidia-smi` works".
## 6. Copy engines (CE) and other engines
* **CE (copy engines)** — DMA between memory regions; also used for UVM's
internal work (page tables, memsets, semaphores). UVM cannot even initialize
without working CE channels, which is why our CE capability reporting was
the first big unblock (`CE_GET_PHYSICAL_CAPS`, `CE_GET_CE_PCE_MASK`).
* **SEC2** — secure copy engine used by UVM for protected operations.
* **NVENC/NVDEC/NVJPG/OFA** — video and image engines; roadmap for graphics
and video phases.
Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TABLE`)
and reported in engine/capability queries. A guest's channel is bound to one
engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).
## 7. Practical reading list inside this repository
| Topic | Where |
|---|---|
| Page-table formats and HAL concepts | this tree's architecture notes plus the decompiled analysis notes |
| Class/method definitions (classes, methods, notifications) | our decompilation notes and reformatted tables in `wiki/` |
| Channel allocation parameter structures | reformatted in the field guide and in the reference wiki pages |
| GMMU controls and static info | `wiki/Address-Reference.md`, `wiki/RPC-Message-Format.md` |
| Notification formats | the reference wiki pages |
| Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` |
**Rule:** tracked documentation describes components and behaviour, with our
own observations, IDs and layouts. Exact analysis sources live in the
maintainers' study notes.
|
