diff options
Diffstat (limited to 'docs/ONBOARDING.md')
| -rw-r--r-- | docs/ONBOARDING.md | 171 |
1 files changed, 0 insertions, 171 deletions
diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md deleted file mode 100644 index 615e016..0000000 --- a/docs/ONBOARDING.md +++ /dev/null @@ -1,171 +0,0 @@ -# Open-IOV Developer Onboarding - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Onboarding -> replacement: -> [07-developers/onboarding-and-standards.md](07-developers/onboarding-and-standards.md). - -> Welcome. This guide gets you from zero to a productive contributor in ~30 -> minutes: what the project is, how the repo is laid out, how to build and -> test, and the standards we hold the code to. - ---- - -## 1. What this project is (60 seconds) - -**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver** -runs in a VM with full GPU functionality — no GRID license. The trick: the -host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the -firmware processor the driver talks to) in software, so the guest's stock -`nvidia.ko` can't tell it isn't running on real hardware. - -Everything guest-facing is **host-side only**. There is no Open-IOV code in -the guest. - -How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W -firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types -mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md` -for how the emulator works. - -## 2. Repository tour - -``` -open-iov/ -├── Makefile # top-level: all/kernel/userspace/examples/tests/clean -├── install.sh # fresh-machine reproduction (deps→nvidia→firmware→build→load) -├── README.md # project intro + quick start -├── STATE.md # live project state (works/limitations/bugs) -├── kernel/ # ★ the host driver (open-iov-vfio.ko) -│ ├── main.c # module init, mdev parent/driver, VFIO ops, PCI config -│ ├── gsp_emulation.c/h # GSP emulation engine (thread, dispatch, MSI-X) -│ ├── gsp_rpc_client.c/h # RPC API + emulation implementations + msg structs -│ ├── bar_emulation.c/h # BAR0 shadow + MSI-X shadow state -│ ├── bar_init.c # physical BAR mapping (host side) -│ ├── memory.c # simple mapping/pin tracking -│ ├── irq.c # IRQ/eventfd plumbing -│ ├── ioctl.c # host management ioctls (libiov consumers) -│ ├── migration.c # 4-phase live-migration skeleton -│ ├── firmware_analyzer.c/h # runtime firmware signature/pattern analysis -│ ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch -│ ├── architecture/ # hw abstraction (Turing/Ampere/Blackwell) -│ ├── version/ # driver-version adaptation (550/560/570/580) -│ ├── include/ # kernel headers (nvidia-vgpu-vfio.h is the main one) -│ └── tests/ # kernel unit tests (gsp_emulation_test) -├── userspace/ -│ ├── lib/ # libiov.so (+ libvgpu.so compat) management API -│ ├── examples/ # example_vgpu test app -│ └── tests/ -├── legacy-driver-guest/ # historical guest-side driver (reference only, NOT built) -├── firmware/57X-GSP/570.144/ # GSP blob + analysis scripts (per-version dirs) -├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko) -├── legacy-patches/nvidia/ # our delta vs upstream OpenRM tag -├── scripts/ # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers -├── tools/ # guest monitor, failure analyzer, benchmarks, ghidra -├── config/ # declarative YAML: hardware/, versions/ -├── docs/ # ★ all developer documentation (start here) -└── wiki/ # firmware analysis pages (RPC format, addresses, ...) -``` - -## 3. Your first build & test - -```bash -cd open-iov -./scripts/stack_manager.sh build-all # kernel → nvidia → userspace (ORDER MATTERS) -sudo ./scripts/stack_manager.sh load-stack -sudo ./scripts/stack_manager.sh mdev-create -./scripts/stack_manager.sh status -``` - -Rules that will save your week (also enforced in `stack_manager.sh`): - -1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`. - nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`. -2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware. -3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`. -4. **Version matrix**: guest driver == host OpenRM == host GSP firmware == - emulation target (all **570.144** today). Mismatches look exactly like - emulation bugs — check versions first (`docs/DEBUGGING.md` §7). -5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness). - Never raw `` or `qemu-system-*` commands. - -Run the unit tests (host, needs the module loaded): - -```bash -make tests -sudo insmod kernel/tests/gsp_emulation_test.ko # dmesg → ALL TESTS PASSED -sudo rmmod gsp_emulation_test -``` - -## 4. Coding standards - -We follow the **Linux kernel coding style** for kernel code and common-sense -C for userspace. The practical checklist: - -### Must -- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the - surrounding file. -- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate - `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order). -- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use - `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT` - (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a - whole session — it is called out in code review. -- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32` - (IOMMU IOVAs can exceed 32 bits). -- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`, - `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`, - module refs. Unbalanced gets become leaks that block unload. -- **Remove paths must fully unwind init paths** (we had a remove that forgot - `list_del` + kthread stop + VFIO unregister — leaked everything). -- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation - (it rings the doorbell our own thread polls → self-trigger livelock). - Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists. -- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant); - rate-limit anything a guest can spam. -- Comments explain **why**, not what — especially around hardware/firmware - behavior discovered by reverse engineering (cite the wiki/address). -- **Signed-off-by** style commit messages: short imperative summary line, - then the *why* (root cause), the *what*, and the user-visible effect. - Reference real dmesg/GDB evidence where relevant. - -### Must not -- No guest-side Open-IOV code. Ever. The guest is stock. -- No reboots/system-affecting commands from scripts without explicit operator - action (see the project's hard rule). -- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone; - nvidia.ko imports *our* symbols, not vice versa. -- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`. -- Don't widen an API's hardware assumptions silently: if a function needs - `vgpu->pdev`, say so in the comment (unit tests can't call those paths). - -### Kernel module specifics -- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`). -- Exported symbols: only what other components genuinely need - (`EXPORT_SYMBOL` right after the definition). -- New per-device resources belong in `struct nv_vgpu_device` with matching - init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain. - -## 5. Documentation expectations - -- New subsystem → new doc in `docs/` + a link from `README.md` and this file. -- Changed behavior → update the affected doc in the **same commit**. -- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table. -- Firmware addresses/structures → `wiki/Address-Reference.md` and friends. - -## 6. Where to read next (in order) - -1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works -2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist -3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design -4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end -5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal -6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference -7. `wiki/` — firmware reverse-engineering canon - -## 7. Getting help / escalating - -- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`. -- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first. -- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test. -- Found a new failure mode? Debug it per the guide, then **write it down** in - the taxonomy table so the next person doesn't rediscover it. |