# Developer Onboarding, Standards, and Tools

> Everything a new contributor needs: what to read, how to build and test, how
> we write code, and how to land a change.

## 1. Your first day

### 1.1 Reading order (about 3 hours)

1. [`../README.md`](../README.md) — the project front door (20 min).
2. [`../01-orientation/01-purpose-and-scope.md`](../01-orientation/01-purpose-and-scope.md)
   and [`../01-orientation/02-the-ai-edge-and-ecosystem.md`](../01-orientation/02-the-ai-edge-and-ecosystem.md)
   — why the project exists (30 min).
3. [`../03-architecture/01-system-overview.md`](../03-architecture/01-system-overview.md)
   — the architecture in one document (30 min).
4. [`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md)
   — where the code lives (30 min).
5. [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
   — what to work on and what is hard (30 min).
6. Working on CUDA: [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
   — CUDA stack analysis, tooling, and the current blockers.
7. The subsystem doc for whatever you plan to touch.

### 1.2 Environment

```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps nvidia firmware build   # full setup
./scripts/stack_manager.sh up             # + load + create mdev
./scripts/stack_manager.sh test           # host test suite
```

You will need: a supported NVIDIA GPU, IOMMU-enabled host, hugepages, kernel
headers, QEMU, and (for guest work) a Linux guest with the stock driver.

### 1.3 The daily loop

```bash
make kernel                    # kernel module (generates Module.symvers)
./scripts/stack_manager.sh build-kernel
./scripts/stack_manager.sh load-open-iov
./scripts/stack_manager.sh mdev-create
./scripts/vgpu_vm.py start vgpu-linux
# guest work ...
./scripts/stack_manager.sh dmesg        # live trace
```

When you change kernel code, remember: reload requires removing the mdev and
the VM holding it. The stack manager enforces the order.

## 2. Repository map

```
open-iov/
├── kernel/               host module (GSP emulation, device model, memory model)
├── userspace/            management library + examples
├── userspace/cuda/       CUDA stack analysis + tooling (start at
│                         userspace/cuda/READ-ME-FIRST.md)
├── tools/                debugging and analysis tools (guest-diag, rmq,
│                         firmware analysis wrappers)
├── scripts/              build/load/test automation, VM harness
├── config/               hardware/version profiles (the portability surface)
├── firmware/             versioned firmware images + decompilation artifacts
├── (private)             reverse-engineering study notes live untracked;
│                         see the maintainers' study index
├── docs/                 THIS documentation tree (start at docs/README.md)
└── wiki/                 long-form firmware/protocol deep dives
```

## 3. Coding standards

### 3.1 Kernel module (`kernel/`)

* Linux kernel C style (tabs, 80-column-ish, `checkpatch` clean where
  practical).
* **No comments explaining *what* the code does — comments explain *why*,
  especially protocol quirks.** Our code encodes observations from hardware;
  the observation must be recorded next to the code that depends on it.
* Every handler logs its request/reply in a reconstructible form (the
  trace-first policy). A handler that silently succeeds is an anti-pattern.
* Never trust guest input: validate sizes, handles, indices, and sequences.
  A malformed element must not be able to steer a write outside guest memory.
* Memory: `kmalloc`/`vzalloc` per the use; DMA access through the hypervisor
  interface only.
* Error paths: fail closed, log once with context, and give up the device
  cleanly.
* **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate
  `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order).

The following rules each come from a real bug and are enforced in review:

* **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use
  `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys() >> PAGE_SHIFT`
  (see `nv_vgpu_mmap_virt_to_pfn`). A bad PFN in a VFIO mmap puts a garbage
  PFN in the guest's EPT and wedges the guest in an EPT-violation livelock —
  case study in
  [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §5.3.
* **`dma_addr_t` for every DMA address.** Never truncate it into `int`/`u32`:
  IOMMU IOVAs can exceed 32 bits.
* **Every `get` needs a `put`**: `pci_dev_get`/`pci_dev_put`,
  `eventfd_ctx_fdget`/`eventfd_ctx_put`, `vfio_alloc_device`/`vfio_put_device`,
  module references. Unbalanced gets become leaks that block unload.
* **Remove paths must fully unwind init paths** (including `list_del` and
  kthread stop); a remove that forgets one leaves the module unloadable.
* **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from the
  emulation path — it writes into the same mailbox and rings the same
  doorbell our own thread polls, a self-triggered RPC livelock. Return
  `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists.
* **Log through the NULL-tolerant `nv_vgpu_*` wrappers and rate-limit**
  anything a guest can spam.
* `open-iov-vfio.ko` is **standalone**: no `MODULE_SOFTDEP`/dependency on
  `nvidia` — `nvidia.ko` imports our symbols, not the other way around.
* VM access in tooling goes through `scripts/vgpu_vm.py`; do not invoke
  QEMU directly.
* Scripts never reboot or otherwise change system state without explicit
  operator action (project hard rule).
* Do not widen an API's hardware assumptions silently: if a function needs
  `vgpu->pdev` (or similar), say so in the comment — unit tests cannot call
  those paths.
* The guest is stock: **no Open-IOV code in the guest, ever** — no guest
  agent, no driver patch, no extra kernel parameters.

### 3.2 Userspace and tools (`userspace/`, `tools/`, `scripts/`)

* C: same style as the kernel module where practical.
* Python/shell: `shellcheck` clean shell, `python3` with no exotic
  dependencies; scripts must be re-runnable and idempotent.
* Tools must live **in-tree** (never in `/tmp`) so they survive reboots and
  are reviewable.

### 3.3 Documentation

* Every behaviour change updates the doc that describes it. Docs are part of
  the definition of done.
* Debugging discoveries go into the failure taxonomy in
  [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)
  so the next person does not rediscover them.
* Prefer a table or a decision tree over prose for anything an operator will
  follow under pressure.
* Protocol facts (IDs, offsets, structures) must cite their source (a live
  trace, or an analysis artifact). Userspace (`libcuda`) addresses are *file
  virtual addresses* (`libcuda+0x...`); Ghidra loads that library at **image
  base `0x100000`**, so old decompilations produced without translating the
  offset are 0x100000 too low (the wrong functions) — re-derive any conclusion
  drawn from them.

## 4. Tooling

| Tool | Purpose |
|---|---|
| `scripts/stack_manager.sh` | build/load/status/test/unload with correct ordering |
| `scripts/vgpu_vm.py` | VM lifecycle incl. guest-agent exec/shell |
| `tools/guest-diag/` | guest-side diagnostics (ioctl tracer, engines, probes, push helper) |
| `userspace/cuda/tools/` | CUDA binary analysis (crash decoder, callgraph walk, decompile-at, corpus annotation) |
| `tools/cuda-ab/` | host-vs-guest CUDA A/B harness (one deterministic workload, diffed results) |
| `tools/rmq/` | host RM-API oracle for ground-truth GPU/RM data |
| `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline |
| module debug parameters | guest-memory scanning/poking, VRAM inspection |
| QEMU gdbstub workflow | guest virtual-memory inspection/writing |
| `wiki/` documents | protocol and firmware deep dives |

## 5. Landing a change

1. **Pick from the known-unknowns list** or a bug with a reproduction.
2. **Reproduce** with the standard captures; write the decision tree if new.
3. **Implement** with tests: host-side unit tests where possible; a scripted
   guest scenario where not.
4. **Evidence**: attach the rung-ladder result (see the evaluation doc) and
   the before/after trace excerpt.
5. **Docs**: update the affected documents (including the known-unknowns entry
   if you closed one).
6. **Review**: a reviewer must be able to reproduce your evidence from the
   PR description alone.

## 6. Makefile targets

```bash
make kernel        # open-iov-vfio.ko (first!)
./scripts/build_nvidia_openrm.sh   # optional OpenRM modules, KBUILD_EXTRA_SYMBOLS from kernel/Module.symvers
make userspace     # libiov.so, examples
make all           # in the correct order
make tests         # build the host test binaries only (does not run them)
./scripts/stack_manager.sh test    # run the host test suite
make clean
```

## 7. Testing discipline

* Kernel changes: at minimum, load/unload cycle ×3, mdev create/destroy, and a
  guest boot + `nvidia-smi`.
* Protocol changes: diff the RPC inventory before/after; attach both.
* Memory-model changes: re-run the VA↔PA discovery tooling and a guest CUDA
  start-up; attach the trace.
* Never land a change that makes a rung regress, even "temporarily".

## 8. Security hygiene for contributors

* Do not add code paths that map or copy memory outside the hypervisor-provided
  DMA interface.
* Treat guest input as hostile; treat debug parameters as privileged.
* If you add a debug surface, document it in the debugging doc and gate it.
* If you discover a protocol fact that implies a security-relevant assumption
  (e.g. "the guest can ask us to write anywhere"), stop and escalate: it is a
  design issue, not a code issue.

## 9. Getting unstuck

* Start with the guest's own diagnostics (it is the oracle).
* Then the host trace; the answer is almost always in the last RPC before the
  failure.
* Then memory observation (scan/poke) — but write down what you learn; the
  next person needs it.
* Ask in the project channels with a *reproduction*, not a summary.