# Open-IOV Developer Onboarding

> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept
> as a record. Current entry point: [docs/README.md](README.md). Onboarding
> replacement:
> [07-developers/onboarding-and-standards.md](07-developers/onboarding-and-standards.md).

> Welcome. This guide gets you from zero to a productive contributor in ~30
> minutes: what the project is, how the repo is laid out, how to build and
> test, and the standards we hold the code to.

---

## 1. What this project is (60 seconds)

**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver**
runs in a VM with full GPU functionality — no GRID license. The trick: the
host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the
firmware processor the driver talks to) in software, so the guest's stock
`nvidia.ko` can't tell it isn't running on real hardware.

Everything guest-facing is **host-side only**. There is no Open-IOV code in
the guest.

How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W
firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types
mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md`
for how the emulator works.

## 2. Repository tour

```
open-iov/
├── Makefile                 # top-level: all/kernel/userspace/examples/tests/clean
├── install.sh               # fresh-machine reproduction (deps→nvidia→firmware→build→load)
├── README.md                # project intro + quick start
├── STATE.md                 # live project state (works/limitations/bugs)
├── kernel/                  # ★ the host driver (open-iov-vfio.ko)
│   ├── main.c               #   module init, mdev parent/driver, VFIO ops, PCI config
│   ├── gsp_emulation.c/h    #   GSP emulation engine (thread, dispatch, MSI-X)
│   ├── gsp_rpc_client.c/h   #   RPC API + emulation implementations + msg structs
│   ├── bar_emulation.c/h    #   BAR0 shadow + MSI-X shadow state
│   ├── bar_init.c           #   physical BAR mapping (host side)
│   ├── memory.c             #   simple mapping/pin tracking
│   ├── irq.c                #   IRQ/eventfd plumbing
│   ├── ioctl.c              #   host management ioctls (libiov consumers)
│   ├── migration.c          #   4-phase live-migration skeleton
│   ├── firmware_analyzer.c/h    # runtime firmware signature/pattern analysis
│   ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch
│   ├── architecture/        #   hw abstraction (Turing/Ampere/Blackwell)
│   ├── version/             #   driver-version adaptation (550/560/570/580)
│   ├── include/             #   kernel headers (nvidia-vgpu-vfio.h is the main one)
│   └── tests/               #   kernel unit tests (gsp_emulation_test)
├── userspace/
│   ├── lib/                 # libiov.so (+ libvgpu.so compat) management API
│   ├── examples/            # example_vgpu test app
│   └── tests/
├── legacy-driver-guest/     # historical guest-side driver (reference only, NOT built)
├── firmware/57X-GSP/570.144/    # GSP blob + analysis scripts (per-version dirs)
├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko)
├── legacy-patches/nvidia/          # our delta vs upstream OpenRM tag
├── scripts/                 # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers
├── tools/                   # guest monitor, failure analyzer, benchmarks, ghidra
├── config/                  # declarative YAML: hardware/, versions/
├── docs/                    # ★ all developer documentation (start here)
└── wiki/                    # firmware analysis pages (RPC format, addresses, ...)
```

## 3. Your first build & test

```bash
cd open-iov
./scripts/stack_manager.sh build-all      # kernel → nvidia → userspace (ORDER MATTERS)
sudo ./scripts/stack_manager.sh load-stack
sudo ./scripts/stack_manager.sh mdev-create
./scripts/stack_manager.sh status
```

Rules that will save your week (also enforced in `stack_manager.sh`):

1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`.
   nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`.
2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware.
3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`.
4. **Version matrix**: guest driver == host OpenRM == host GSP firmware ==
   emulation target (all **570.144** today). Mismatches look exactly like
   emulation bugs — check versions first (`docs/DEBUGGING.md` §7).
5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness).
   Never raw `` or `qemu-system-*` commands.

Run the unit tests (host, needs the module loaded):

```bash
make tests
sudo insmod kernel/tests/gsp_emulation_test.ko   # dmesg → ALL TESTS PASSED
sudo rmmod gsp_emulation_test
```

## 4. Coding standards

We follow the **Linux kernel coding style** for kernel code and common-sense
C for userspace. The practical checklist:

### Must
- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the
  surrounding file.
- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate
  `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order).
- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use
  `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT`
  (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a
  whole session — it is called out in code review.
- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32`
  (IOMMU IOVAs can exceed 32 bits).
- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`,
  `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`,
  module refs. Unbalanced gets become leaks that block unload.
- **Remove paths must fully unwind init paths** (we had a remove that forgot
  `list_del` + kthread stop + VFIO unregister — leaked everything).
- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation
  (it rings the doorbell our own thread polls → self-trigger livelock).
  Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists.
- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant);
  rate-limit anything a guest can spam.
- Comments explain **why**, not what — especially around hardware/firmware
  behavior discovered by reverse engineering (cite the wiki/address).
- **Signed-off-by** style commit messages: short imperative summary line,
  then the *why* (root cause), the *what*, and the user-visible effect.
  Reference real dmesg/GDB evidence where relevant.

### Must not
- No guest-side Open-IOV code. Ever. The guest is stock.
- No reboots/system-affecting commands from scripts without explicit operator
  action (see the project's hard rule).
- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone;
  nvidia.ko imports *our* symbols, not vice versa.
- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`.
- Don't widen an API's hardware assumptions silently: if a function needs
  `vgpu->pdev`, say so in the comment (unit tests can't call those paths).

### Kernel module specifics
- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`).
- Exported symbols: only what other components genuinely need
  (`EXPORT_SYMBOL` right after the definition).
- New per-device resources belong in `struct nv_vgpu_device` with matching
  init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain.

## 5. Documentation expectations

- New subsystem → new doc in `docs/` + a link from `README.md` and this file.
- Changed behavior → update the affected doc in the **same commit**.
- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table.
- Firmware addresses/structures → `wiki/Address-Reference.md` and friends.

## 6. Where to read next (in order)

1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works
2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist
3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design
4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end
5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal
6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference
7. `wiki/` — firmware reverse-engineering canon

## 7. Getting help / escalating

- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`.
- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first.
- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test.
- Found a new failure mode? Debug it per the guide, then **write it down** in
  the taxonomy table so the next person doesn't rediscover it.