1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
# The Host Module (vgsp.ko)
> **Scope.** The host-side kernel module that presents one virtual
> Turing-class GPU per `mdev` instance: the virtual device, the emulation
> engine, the guest-memory model and the work-forwarding design. Canonical
> source: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md);
> the docs tree wins on disagreement with this page.
>
> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with
> the stock guest driver; `[wip]` demonstrated live, being productized;
> `[gated]` implemented default-off behind supervised gates; `[wall]` known
> blocking limit; `[target]` designed, not built; `[planned]` roadmap.
## 1. Role and position
```text
guest VM ── PCIe MMIO / DMA / MSI-X through the hypervisor
(KVM/QEMU or Xen: VFIO mdev transport, IOMMU)
│
vgsp.ko: virtual device · emulation core · memory model · work path
│
physical GPU + GSP firmware, driven by this module (host driver unmodified)
```
The module registers as an **mdev parent** on the physical GPU, exposing one
device type per supported profile; each `mdev` is an independent virtual GPU
([`System-Overview.md`](System-Overview.md), [`Execution-Plane.md`](Execution-Plane.md)).
## 2. The virtual device
The guest sees a plausible Turing-class GPU (IDs, class code, BARs, capability
list). Some fields exist only for validation: PCIe link capabilities (the
hypervisor can rewrite them), MSI-X masking state, and non-SR-IOV VF windows.
| BAR | Size | Backing | Contents |
|---|---|---|---|
| BAR0 | 16 MB | host `vzalloc` shadow | boot, doorbells, VF registers, PRAMIN window, VBIOS |
| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture |
| BAR3 | — | instance window | instance/PRAMIN behaviour |
MMIO is served by VFIO region callbacks or directly from the shadow when
mapped into the guest. BAR1 is host RAM today — correctness first: the real
GPU cannot read it, so forwarding must later migrate objects to real VRAM or
expose DMA-mapped system memory [target] ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)).
## 3. The emulation core
A kernel thread drives the device's "firmware" behaviour [live]:
```text
BAR0 doorbell / VF-register write → read msgq element from guest memory
→ dispatch RPC → write reply → inject MSI-X (eventfd → irqfd)
```
Guest memory is reached via `vfio_dma_rw()` from kernel-thread context. The
RPC philosophy: correct shape first (or the driver walks off into
uninitialized memory), plausible values second (values must be
self-consistent), never invent input (in/out parameter blocks are echoed;
zeroing the `GSP_RM_ALLOC` reply once faulted `libcuda`). Handlers cover
boot/static-info exchanges, device/FB/BUS/INTR tables, CE and GMMU queries,
reserved-PDE contracts, channel/CE allocation and an unknown-control fallback.
Push buffers are decoded as method streams and executed in software —
memsets, copies, semaphore releases — a **correctness bridge, not the data
plane** [wip]; emulation-mode CUDA is verified end to end (`vectorAdd` over
1M elements, `bad=0`) [live]. Physical-GSP boot is default-off [gated]; the
firmware analyzer, version tables, HAL tables and dynamic RPC dispatcher are
not yet on the live path [target].
## 4. The guest-memory model
**What the client gives us:** channel memory descriptors
(`NV_MEMORY_DESC_PARAMS`) with *physical* addresses (instance memory, method
buffer, userd, ramfc, error notifiers) and the page-table contract for
server-owned VA ranges (`COPY_SERVER_RESERVED_PDES` — table-level PAs, page
shifts, VA range, *empty* leaves). It gives us none of its own sysmem maps.
**What we derive instead** [live] — the module rebuilds the missing mappings
from the guest's own bookkeeping, with no cooperation from the guest driver:
1. **VA↔PA pairs live in guest memory**: UVM and RM record both the GPU VA and
the kernel CPU VA per object (`uvm_rm_mem_t.vas[CPU]/vas[GPU]` +0x30/+0x38).
2. **CPU VAs are direct-map addresses**: `PA = cpu_va - page_offset_base`, the
base recovered from guest memory or `/proc/kallsyms` during development.
3. **Exception:** control-device `mmap()` pages have no leaf PTEs; verified
pairs arrive via `sema_map_add` until a durable join exists. The page-granular
cache is fed by a trusted GMMU walk ([`Introduction-to-GMMU.md`](Introduction-to-GMMU.md)).
**Load-bearing copy invariants:** (1) a copy runs at most once per GPFIFO
entry — replays are suppressed by entry-word identity plus an executed-operation
bitmap; (2) a virtual operand translates per 4 KB page (195 of 256 pages were
contiguous in one observed 1 MB range); (3) a consumed slot's work never runs
later — the per-put-epoch mark is authoritative until a put wrap.
## 5. The forwarding path (design)
```text
guest push submitted
→ detect (doorbell / GPPUT / push-buffer scan), decode the method stream
→ EMULATE state/notification (release, membar, fence) or
FORWARD real engine work (GR launch, CE copy, memop)
→ FORWARD: submit equivalent work on a real channel bound to the physical GPU
→ relay completions (semaphore / fault / interrupt) back into guest memory
```
Proxying through the distribution host driver has **no callable entry point
from kernel context**, so the module **owns the device and runs its real GSP
firmware as the resource-manager server**, with emulation as the control
plane. CE copies, release and fault relay are implemented default-off and
verified live [gated]; GR execution is the open wall [wall] — a forwarded
launch reaches the front-end; the inline launch-descriptor install is unobserved
([`Execution-Frontier.md`](Execution-Frontier.md)).
## 6. Debug parameters
`/sys/module/vgsp/parameters/*` makes guest memory and the emulated device
inspectable without rebuilding:
| Parameter | Purpose |
|---|---|
| `scan_guest_ram=<v>` / `scan_vidmem=<v>` | find a 64-bit value in guest RAM (0–8 GB) or the VRAM shadow |
| `scan_seq=<v1,...>` / `scan_pt=<pa>` / `scan_masked=<v>,<mask>` | find a 32-bit sequence, entries pointing at a PA, or `(x & mask) == v` |
| `dump_vidmem=<off>,<len>` / `fill_vidmem=<v>` | dump/scan or fill the VRAM shadow |
| `poke_guest=<gpa>,<v>` / `poke_fill=<gpa>,<v>,<n>` | write a 32-bit value / fill a range in guest physical memory |
| `sema_probe=<gpu_va>[,count]` / `sema_map_add=<va>,<pa>` | probe a semaphore VA / teach one verified VA→PA pair |
| `dump_guest=<gpa>,<len>` | dump guest physical memory (capped) |
The physical-backend path adds a default-off set of backend controls
(documented in the repository; workflows in
[`../docs/05-operations/02-debugging-and-maintenance-workflows.md`](../docs/05-operations/02-debugging-and-maintenance-workflows.md)).
## 7. Build and symbol contracts
* `vgsp.ko` exports the `nvidia_vgpu_vfio_*` interface (`probe`/`remove`, ops
get/set, mdev parent registration) to pair the device with OpenRM host
modules.
* OpenRM modules built from source need
`KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers` and must build **after**
this module (missing symbols and stale build caches are common otherwise).
* On single-GPU developer systems the distribution driver owns the physical
GPU and the module runs standalone; the OpenRM path is only for native-vGPU
management, not QEMU.
## 8. Implemented vs designed
| Area | Status |
|---|---|
| mdev/VFIO device, config space, BARs, MSI-X; BAR0 state machine; boot handshake [live] | implemented |
| Message queues + RPC + init controls (stock 570.144) [live] | implemented |
| Guest sysmem access + VA↔PA discovery [wip] | debug-grade, to be productized |
| Push decoding + semaphore completion [wip] | demonstrated live, being productized |
| Emulation-mode CUDA end-to-end [live] | repeatable 1M-element `vectorAdd`, `bad=0` |
| Guest-RAM physical path + server-owned page-table population [gated] | default-off; GPA→HPA via shared-file page cache verified live on the CE |
| GPU work forwarding: CE copy, release and fault relay [gated] | default-off, verified live; GR open [wall] |
| Execution backend (module-owned GPU, direct real-GSP session) [wall] | boot, static configuration and CE copies proven; GR execution open (failed mutating attempts park the device in a locked fail-safe, never auto-retried) |
| Real VRAM backing [target]; Windows, Xen, aarch64 [planned] | designed, not implemented |
## 9. Related documentation
* Canonical chapter: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md) — the source of this condensation.
* [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md), [`BAR-Emulation-Specification.md`](BAR-Emulation-Specification.md), [`WPR-Management.md`](WPR-Management.md); [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md), [`Execution-Frontier.md`](Execution-Frontier.md), [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md).
|
