1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
|
# Open-IOV Developer Onboarding
> Welcome. This guide gets you from zero to a productive contributor in ~30
> minutes: what the project is, how the repo is laid out, how to build and
> test, and the standards we hold the code to.
---
## 1. What this project is (60 seconds)
**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver**
runs in a VM with full GPU functionality — no GRID license. The trick: the
host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the
firmware processor the driver talks to) in software, so the guest's stock
`nvidia.ko` can't tell it isn't running on real hardware.
Everything guest-facing is **host-side only**. There is no Open-IOV code in
the guest.
How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W
firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types
mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md`
for how the emulator works.
## 2. Repository tour
```
open-iov/
├── Makefile # top-level: all/kernel/userspace/examples/tests/clean
├── install.sh # fresh-machine reproduction (deps→nvidia→firmware→build→load)
├── README.md # project intro + quick start
├── STATE.md # live project state (works/limitations/bugs)
├── kernel/ # ★ the host driver (open-iov-vfio.ko)
│ ├── main.c # module init, mdev parent/driver, VFIO ops, PCI config
│ ├── gsp_emulation.c/h # GSP emulation engine (thread, dispatch, MSI-X)
│ ├── gsp_rpc_client.c/h # RPC API + emulation implementations + msg structs
│ ├── bar_emulation.c/h # BAR0 shadow + MSI-X shadow state
│ ├── bar_init.c # physical BAR mapping (host side)
│ ├── memory.c # simple mapping/pin tracking
│ ├── irq.c # IRQ/eventfd plumbing
│ ├── ioctl.c # host management ioctls (libiov consumers)
│ ├── migration.c # 4-phase live-migration skeleton
│ ├── firmware_analyzer.c/h # runtime firmware signature/pattern analysis
│ ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch
│ ├── architecture/ # hw abstraction (Turing/Ampere/Blackwell)
│ ├── version/ # driver-version adaptation (550/560/570/580)
│ ├── include/ # kernel headers (nvidia-vgpu-vfio.h is the main one)
│ └── tests/ # kernel unit tests (gsp_emulation_test)
├── userspace/
│ ├── lib/ # libiov.so (+ libvgpu.so compat) management API
│ ├── examples/ # example_vgpu test app
│ └── tests/
├── legacy-driver-guest/ # historical guest-side driver (reference only, NOT built)
├── firmware/57X-GSP/570.144/ # GSP blob + analysis scripts (per-version dirs)
├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko)
├── legacy-patches/nvidia/ # our delta vs upstream OpenRM tag
├── scripts/ # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers
├── tools/ # guest monitor, failure analyzer, benchmarks, ghidra
├── config/ # declarative YAML: hardware/, versions/
├── docs/ # ★ all developer documentation (start here)
└── wiki/ # firmware analysis pages (RPC format, addresses, ...)
```
## 3. Your first build & test
```bash
cd open-iov
./scripts/stack_manager.sh build-all # kernel → nvidia → userspace (ORDER MATTERS)
sudo ./scripts/stack_manager.sh load-stack
sudo ./scripts/stack_manager.sh mdev-create
./scripts/stack_manager.sh status
```
Rules that will save your week (also enforced in `stack_manager.sh`):
1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`.
nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`.
2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware.
3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`.
4. **Version matrix**: guest driver == host OpenRM == host GSP firmware ==
emulation target (all **570.144** today). Mismatches look exactly like
emulation bugs — check versions first (`docs/DEBUGGING.md` §7).
5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness).
Never raw `` or `qemu-system-*` commands.
Run the unit tests (host, needs the module loaded):
```bash
make tests
sudo insmod kernel/tests/gsp_emulation_test.ko # dmesg → ALL TESTS PASSED
sudo rmmod gsp_emulation_test
```
## 4. Coding standards
We follow the **Linux kernel coding style** for kernel code and common-sense
C for userspace. The practical checklist:
### Must
- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the
surrounding file.
- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate
`-ENOMEM` with clean unwinding (`goto err_...` in reverse init order).
- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use
`is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT`
(see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a
whole session — it is called out in code review.
- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32`
(IOMMU IOVAs can exceed 32 bits).
- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`,
`eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`,
module refs. Unbalanced gets become leaks that block unload.
- **Remove paths must fully unwind init paths** (we had a remove that forgot
`list_del` + kthread stop + VFIO unregister — leaked everything).
- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation
(it rings the doorbell our own thread polls → self-trigger livelock).
Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists.
- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant);
rate-limit anything a guest can spam.
- Comments explain **why**, not what — especially around hardware/firmware
behavior discovered by reverse engineering (cite the wiki/address).
- **Signed-off-by** style commit messages: short imperative summary line,
then the *why* (root cause), the *what*, and the user-visible effect.
Reference real dmesg/GDB evidence where relevant.
### Must not
- No guest-side Open-IOV code. Ever. The guest is stock.
- No reboots/system-affecting commands from scripts without explicit operator
action (see the project's hard rule).
- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone;
nvidia.ko imports *our* symbols, not vice versa.
- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`.
- Don't widen an API's hardware assumptions silently: if a function needs
`vgpu->pdev`, say so in the comment (unit tests can't call those paths).
### Kernel module specifics
- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`).
- Exported symbols: only what other components genuinely need
(`EXPORT_SYMBOL` right after the definition).
- New per-device resources belong in `struct nv_vgpu_device` with matching
init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain.
## 5. Documentation expectations
- New subsystem → new doc in `docs/` + a link from `README.md` and this file.
- Changed behavior → update the affected doc in the **same commit**.
- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table.
- Firmware addresses/structures → `wiki/Address-Reference.md` and friends.
## 6. Where to read next (in order)
1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works
2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist
3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design
4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end
5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal
6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference
7. `wiki/` — firmware reverse-engineering canon
## 7. Getting help / escalating
- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`.
- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first.
- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test.
- Found a new failure mode? Debug it per the guide, then **write it down** in
the taxonomy table so the next person doesn't rediscover it.
|
