diff options
Diffstat (limited to 'docs/ONBOARDING.md')
| -rw-r--r-- | docs/ONBOARDING.md | 166 |
1 files changed, 166 insertions, 0 deletions
diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md new file mode 100644 index 0000000..1d0a4f7 --- /dev/null +++ b/docs/ONBOARDING.md @@ -0,0 +1,166 @@ +# Open-IOV Developer Onboarding + +> Welcome. This guide gets you from zero to a productive contributor in ~30 +> minutes: what the project is, how the repo is laid out, how to build and +> test, and the standards we hold the code to. + +--- + +## 1. What this project is (60 seconds) + +**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver** +runs in a VM with full GPU functionality — no GRID license. The trick: the +host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the +firmware processor the driver talks to) in software, so the guest's stock +`nvidia.ko` can't tell it isn't running on real hardware. + +Everything guest-facing is **host-side only**. There is no Open-IOV code in +the guest. + +How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W +firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types +mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md` +for how the emulator works. + +## 2. Repository tour + +``` +open-iov/ +├── Makefile # top-level: all/kernel/userspace/examples/tests/clean +├── install.sh # fresh-machine reproduction (deps→nvidia→firmware→build→load) +├── README.md # project intro + quick start +├── STATE.md # live project state (works/limitations/bugs) +├── kernel/ # ★ the host driver (open-iov-vfio.ko) +│ ├── main.c # module init, mdev parent/driver, VFIO ops, PCI config +│ ├── gsp_emulation.c/h # GSP emulation engine (thread, dispatch, MSI-X) +│ ├── gsp_rpc_client.c/h # RPC API + emulation implementations + msg structs +│ ├── bar_emulation.c/h # BAR0 shadow + MSI-X shadow state +│ ├── bar_init.c # physical BAR mapping (host side) +│ ├── memory.c # simple mapping/pin tracking +│ ├── irq.c # IRQ/eventfd plumbing +│ ├── ioctl.c # host management ioctls (libiov consumers) +│ ├── migration.c # 4-phase live-migration skeleton +│ ├── firmware_analyzer.c/h # runtime firmware signature/pattern analysis +│ ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch +│ ├── architecture/ # hw abstraction (Turing/Ampere/Blackwell) +│ ├── version/ # driver-version adaptation (550/560/570/580) +│ ├── include/ # kernel headers (nvidia-vgpu-vfio.h is the main one) +│ └── tests/ # kernel unit tests (gsp_emulation_test) +├── userspace/ +│ ├── lib/ # libiov.so (+ libvgpu.so compat) management API +│ ├── examples/ # example_vgpu test app +│ └── tests/ +├── legacy-driver-guest/ # historical guest-side driver (reference only, NOT built) +├── firmware/57X-GSP/570.144/ # GSP blob + analysis scripts (per-version dirs) +├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko) +├── patches/nvidia/ # our delta vs upstream OpenRM tag +├── scripts/ # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers +├── tools/ # guest monitor, failure analyzer, benchmarks, ghidra +├── config/ # declarative YAML: hardware/, versions/ +├── docs/ # ★ all developer documentation (start here) +└── wiki/ # firmware analysis pages (RPC format, addresses, ...) +``` + +## 3. Your first build & test + +```bash +cd open-iov +./scripts/stack_manager.sh build-all # kernel → nvidia → userspace (ORDER MATTERS) +sudo ./scripts/stack_manager.sh load-stack +sudo ./scripts/stack_manager.sh mdev-create +./scripts/stack_manager.sh status +``` + +Rules that will save your week (also enforced in `stack_manager.sh`): + +1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`. + nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`. +2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware. +3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`. +4. **Version matrix**: guest driver == host OpenRM == host GSP firmware == + emulation target (all **570.144** today). Mismatches look exactly like + emulation bugs — check versions first (`docs/DEBUGGING.md` §7). +5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness). + Never raw `` or `qemu-system-*` commands. + +Run the unit tests (host, needs the module loaded): + +```bash +make tests +sudo insmod kernel/tests/gsp_emulation_test.ko # dmesg → ALL TESTS PASSED +sudo rmmod gsp_emulation_test +``` + +## 4. Coding standards + +We follow the **Linux kernel coding style** for kernel code and common-sense +C for userspace. The practical checklist: + +### Must +- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the + surrounding file. +- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate + `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order). +- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use + `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT` + (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a + whole session — it is called out in code review. +- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32` + (IOMMU IOVAs can exceed 32 bits). +- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`, + `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`, + module refs. Unbalanced gets become leaks that block unload. +- **Remove paths must fully unwind init paths** (we had a remove that forgot + `list_del` + kthread stop + VFIO unregister — leaked everything). +- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation + (it rings the doorbell our own thread polls → self-trigger livelock). + Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists. +- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant); + rate-limit anything a guest can spam. +- Comments explain **why**, not what — especially around hardware/firmware + behavior discovered by reverse engineering (cite the wiki/address). +- **Signed-off-by** style commit messages: short imperative summary line, + then the *why* (root cause), the *what*, and the user-visible effect. + Reference real dmesg/GDB evidence where relevant. + +### Must not +- No guest-side Open-IOV code. Ever. The guest is stock. +- No reboots/system-affecting commands from scripts without explicit operator + action (see the project's hard rule). +- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone; + nvidia.ko imports *our* symbols, not vice versa. +- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`. +- Don't widen an API's hardware assumptions silently: if a function needs + `vgpu->pdev`, say so in the comment (unit tests can't call those paths). + +### Kernel module specifics +- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`). +- Exported symbols: only what other components genuinely need + (`EXPORT_SYMBOL` right after the definition). +- New per-device resources belong in `struct nv_vgpu_device` with matching + init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain. + +## 5. Documentation expectations + +- New subsystem → new doc in `docs/` + a link from `README.md` and this file. +- Changed behavior → update the affected doc in the **same commit**. +- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table. +- Firmware addresses/structures → `wiki/Address-Reference.md` and friends. + +## 6. Where to read next (in order) + +1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works +2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist +3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design +4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end +5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal +6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference +7. `wiki/` — firmware reverse-engineering canon + +## 7. Getting help / escalating + +- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`. +- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first. +- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test. +- Found a new failure mode? Debug it per the guide, then **write it down** in + the taxonomy table so the next person doesn't rediscover it. |