1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
|
# Developer Onboarding, Standards, and Tools
> Everything a new contributor needs: what to read, how to build and test, how
> we write code, and how to land a change.
## 1. Your first day
### 1.1 Reading order (about 3 hours)
1. [`../README.md`](../README.md) — the project front door (20 min).
2. [`../01-orientation/01-purpose-and-scope.md`](../01-orientation/01-purpose-and-scope.md)
and [`../01-orientation/02-the-ai-edge-and-ecosystem.md`](../01-orientation/02-the-ai-edge-and-ecosystem.md)
— why the project exists (30 min).
3. [`../03-architecture/01-system-overview.md`](../03-architecture/01-system-overview.md)
— the architecture in one document (30 min).
4. [`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md)
— where the code lives (30 min).
5. [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
— what to work on and what is hard (30 min).
6. Working on CUDA: [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
— CUDA stack analysis, tooling, and the current blockers.
7. The subsystem doc for whatever you plan to touch.
### 1.2 Environment
```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps nvidia firmware build # full setup
./scripts/stack_manager.sh up # + load + create mdev
./scripts/stack_manager.sh test # host test suite
```
You will need: a supported NVIDIA GPU, IOMMU-enabled host, hugepages, kernel
headers, QEMU, and (for guest work) a Linux guest with the stock driver.
### 1.3 The daily loop
```bash
make kernel # kernel module (generates Module.symvers)
./scripts/stack_manager.sh build-kernel
./scripts/stack_manager.sh load-open-iov
./scripts/stack_manager.sh mdev-create
./scripts/vgpu_vm.py start vgpu-linux
# guest work ...
./scripts/stack_manager.sh dmesg # live trace
```
When you change kernel code, remember: reload requires removing the mdev and
the VM holding it. The stack manager enforces the order.
## 2. Repository map
```
open-iov/
├── kernel/ host module (GSP emulation, device model, memory model)
├── userspace/ management library + examples
├── userspace/cuda/ CUDA stack analysis + tooling (start at
│ userspace/cuda/READ-ME-FIRST.md)
├── tools/ debugging and analysis tools (guest-diag, rmq,
│ firmware analysis wrappers)
├── scripts/ build/load/test automation, VM harness
├── config/ hardware/version profiles (the portability surface)
├── firmware/ versioned firmware images + decompilation artifacts
├── (private) reverse-engineering study notes live untracked,
│ outside the published tree
├── docs/ THIS documentation tree (start at docs/README.md)
└── wiki/ long-form firmware/protocol deep dives
```
## 3. Coding standards
### 3.1 Kernel module (`kernel/`)
* Linux kernel C style (tabs, 80-column-ish, `checkpatch` clean where
practical).
* **No comments explaining *what* the code does — comments explain *why*,
especially protocol quirks.** Our code encodes observations from hardware;
the observation must be recorded next to the code that depends on it.
* Every handler logs its request/reply in a reconstructible form (the
trace-first policy). A handler that silently succeeds is an anti-pattern.
* Never trust guest input: validate sizes, handles, indices, and sequences.
A malformed element must not be able to steer a write outside guest memory.
* Memory: `kmalloc`/`vzalloc` per the use; DMA access through the hypervisor
interface only.
* Error paths: fail closed, log once with context, and give up the device
cleanly.
* **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate
`-ENOMEM` with clean unwinding (`goto err_...` in reverse init order).
The following rules each come from a real bug and are enforced in review:
* **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use
`is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys() >> PAGE_SHIFT`
(see `nv_vgpu_mmap_virt_to_pfn`). A bad PFN in a VFIO mmap puts a garbage
PFN in the guest's EPT and wedges the guest in an EPT-violation livelock —
case study in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §5.3.
* **`dma_addr_t` for every DMA address.** Never truncate it into `int`/`u32`:
IOMMU IOVAs can exceed 32 bits.
* **Every `get` needs a `put`**: `pci_dev_get`/`pci_dev_put`,
`eventfd_ctx_fdget`/`eventfd_ctx_put`, `vfio_alloc_device`/`vfio_put_device`,
module references. Unbalanced gets become leaks that block unload.
* **Remove paths must fully unwind init paths** (including `list_del` and
kthread stop); a remove that forgets one leaves the module unloadable.
* **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from the
emulation path — it writes into the same mailbox and rings the same
doorbell our own thread polls, a self-triggered RPC livelock. Return
`-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists.
* **Log through the NULL-tolerant `nv_vgpu_*` wrappers and rate-limit**
anything a guest can spam.
* `open-iov-vfio.ko` is **standalone**: no `MODULE_SOFTDEP`/dependency on
`nvidia` — `nvidia.ko` imports our symbols, not the other way around.
* VM access in tooling goes through `scripts/vgpu_vm.py`; do not invoke
QEMU directly.
* Scripts never reboot or otherwise change system state without explicit
operator action (project hard rule).
* Do not widen an API's hardware assumptions silently: if a function needs
`vgpu->pdev` (or similar), say so in the comment — unit tests cannot call
those paths.
* The guest is stock: **no Open-IOV code in the guest, ever** — no guest
agent, no driver patch, no extra kernel parameters.
### 3.2 Userspace and tools (`userspace/`, `tools/`, `scripts/`)
* C: same style as the kernel module where practical.
* Python/shell: `shellcheck` clean shell, `python3` with no exotic
dependencies; scripts must be re-runnable and idempotent.
* Tools must live **in-tree** (never in `/tmp`) so they survive reboots and
are reviewable.
### 3.3 Documentation
* Every behaviour change updates the doc that describes it. Docs are part of
the definition of done.
* Debugging discoveries go into the failure taxonomy in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)
so the next person does not rediscover them.
* Prefer a table or a decision tree over prose for anything an operator will
follow under pressure.
* Protocol facts (IDs, offsets, structures) must cite their source (a live
trace, or an analysis artifact). Userspace (`libcuda`) addresses are *file
virtual addresses* (`libcuda+0x...`); Ghidra loads that library at **image
base `0x100000`**, so old decompilations produced without translating the
offset are 0x100000 too low (the wrong functions) — re-derive any conclusion
drawn from them.
## 4. Tooling
| Tool | Purpose |
|---|---|
| `scripts/stack_manager.sh` | build/load/status/test/unload with correct ordering |
| `scripts/vgpu_vm.py` | VM lifecycle incl. guest-agent exec/shell |
| `tools/guest-diag/` | guest-side diagnostics (ioctl tracer, engines, probes, push helper) |
| `userspace/cuda/tools/` | CUDA binary analysis (crash decoder, callgraph walk, decompile-at, corpus annotation) |
| `tools/cuda-ab/` | host-vs-guest CUDA A/B harness (one deterministic workload, diffed results) |
| `tools/rmq/` | host RM-API oracle for ground-truth GPU/RM data |
| `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline |
| module debug parameters | guest-memory scanning/poking, VRAM inspection; physical-backend parameters and debugfs state/control files (see [`../eng-ops.md`](../eng-ops.md) Part II) |
| QEMU gdbstub workflow | guest virtual-memory inspection/writing |
| `debug/risc-v-gsp-rm-vm/` (risc-v-gsp-rm-debugger) | implemented, documented and tested ([audit](../../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) — RISC-V GSP/uproc VM runtime inspection, session recording and `gsp-rm-vfio` accuracy measurement (see [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md)) |
| `wiki/` documents | protocol and firmware deep dives |
## 5. Landing a change
1. **Pick from the known-unknowns list** or a bug with a reproduction.
2. **Reproduce** with the standard captures; write the decision tree if new.
3. **Implement** with tests: host-side unit tests where possible; a scripted
guest scenario where not.
4. **Evidence**: attach the rung-ladder result (see the evaluation doc) and
the before/after trace excerpt.
5. **Docs**: update the affected documents (including the known-unknowns entry
if you closed one).
6. **Review**: a reviewer must be able to reproduce your evidence from the
PR description alone.
## 6. Makefile targets
```bash
make kernel # open-iov-vfio.ko (first!)
./scripts/build_nvidia_openrm.sh # optional OpenRM modules, KBUILD_EXTRA_SYMBOLS from kernel/Module.symvers
make userspace # libiov.so, examples
make all # in the correct order
make tests # build the host test binaries only (does not run them)
./scripts/stack_manager.sh test # run the host test suite
make clean
```
## 7. Testing discipline
* Kernel changes: at minimum, load/unload cycle ×3, mdev create/destroy, and a
guest boot + `nvidia-smi`.
* Protocol changes: diff the RPC inventory before/after; attach both.
* Memory-model changes: re-run the VA↔PA discovery tooling and a guest CUDA
start-up; attach the trace.
* Live physical-backend cycles are **supervised only**: one cycle at a time,
never unattended, with the module sha256 recorded before and after and the
clean-release checklist walked every time. Do not retry after a
mutating-RPC failure or a wedge. A requested reboot needs a written reason
and a post-reboot plan before approval is asked (project hard rule). See
[`../eng-ops.md`](../eng-ops.md) Part II.
* Never land a change that makes a rung regress, even "temporarily".
## 8. Security hygiene for contributors
* Do not add code paths that map or copy memory outside the hypervisor-provided
DMA interface.
* Treat guest input as hostile; treat debug parameters as privileged.
* If you add a debug surface, document it in the debugging doc and gate it.
* If you discover a protocol fact that implies a security-relevant assumption
(e.g. "the guest can ask us to write anywhere"), stop and escalate: it is a
design issue, not a code issue.
## 9. Getting unstuck
* Start with the guest's own diagnostics (it is the oracle).
* Then the host trace; the answer is almost always in the last RPC before the
failure.
* Then memory observation (scan/poke) — but write down what you learn; the
next person needs it.
* Ask in the project channels with a *reproduction*, not a summary.
|
