Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 166 insertions, 0 deletions

diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md
new file mode 100644
index 0000000..1d0a4f7
--- /dev/null
+++ b/docs/ONBOARDING.md
@@ -0,0 +1,166 @@
+# Open-IOV Developer Onboarding
+
+> Welcome. This guide gets you from zero to a productive contributor in ~30
+> minutes: what the project is, how the repo is laid out, how to build and
+> test, and the standards we hold the code to.
+
+---
+
+## 1. What this project is (60 seconds)
+
+**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver**
+runs in a VM with full GPU functionality — no GRID license. The trick: the
+host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the
+firmware processor the driver talks to) in software, so the guest's stock
+`nvidia.ko` can't tell it isn't running on real hardware.
+
+Everything guest-facing is **host-side only**. There is no Open-IOV code in
+the guest.
+
+How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W
+firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types
+mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md`
+for how the emulator works.
+
+## 2. Repository tour
+
+```
+open-iov/
+├── Makefile                 # top-level: all/kernel/userspace/examples/tests/clean
+├── install.sh               # fresh-machine reproduction (deps→nvidia→firmware→build→load)
+├── README.md                # project intro + quick start
+├── STATE.md                 # live project state (works/limitations/bugs)
+├── kernel/                  # ★ the host driver (open-iov-vfio.ko)
+│   ├── main.c               #   module init, mdev parent/driver, VFIO ops, PCI config
+│   ├── gsp_emulation.c/h    #   GSP emulation engine (thread, dispatch, MSI-X)
+│   ├── gsp_rpc_client.c/h   #   RPC API + emulation implementations + msg structs
+│   ├── bar_emulation.c/h    #   BAR0 shadow + MSI-X shadow state
+│   ├── bar_init.c           #   physical BAR mapping (host side)
+│   ├── memory.c             #   simple mapping/pin tracking
+│   ├── irq.c                #   IRQ/eventfd plumbing
+│   ├── ioctl.c              #   host management ioctls (libiov consumers)
+│   ├── migration.c          #   4-phase live-migration skeleton
+│   ├── firmware_analyzer.c/h    # runtime firmware signature/pattern analysis
+│   ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch
+│   ├── architecture/        #   hw abstraction (Turing/Ampere/Blackwell)
+│   ├── version/             #   driver-version adaptation (550/560/570/580)
+│   ├── include/             #   kernel headers (nvidia-vgpu-vfio.h is the main one)
+│   └── tests/               #   kernel unit tests (gsp_emulation_test)
+├── userspace/
+│   ├── lib/                 # libiov.so (+ libvgpu.so compat) management API
+│   ├── examples/            # example_vgpu test app
+│   └── tests/
+├── legacy-driver-guest/     # historical guest-side driver (reference only, NOT built)
+├── firmware/57X-GSP/570.144/    # GSP blob + analysis scripts (per-version dirs)
+├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko)
+├── patches/nvidia/          # our delta vs upstream OpenRM tag
+├── scripts/                 # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers
+├── tools/                   # guest monitor, failure analyzer, benchmarks, ghidra
+├── config/                  # declarative YAML: hardware/, versions/
+├── docs/                    # ★ all developer documentation (start here)
+└── wiki/                    # firmware analysis pages (RPC format, addresses, ...)
+```
+
+## 3. Your first build & test
+
+```bash
+cd open-iov
+./scripts/stack_manager.sh build-all      # kernel → nvidia → userspace (ORDER MATTERS)
+sudo ./scripts/stack_manager.sh load-stack
+sudo ./scripts/stack_manager.sh mdev-create
+./scripts/stack_manager.sh status
+```
+
+Rules that will save your week (also enforced in `stack_manager.sh`):
+
+1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`.
+   nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`.
+2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware.
+3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`.
+4. **Version matrix**: guest driver == host OpenRM == host GSP firmware ==
+   emulation target (all **570.144** today). Mismatches look exactly like
+   emulation bugs — check versions first (`docs/DEBUGGING.md` §7).
+5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness).
+   Never raw `` or `qemu-system-*` commands.
+
+Run the unit tests (host, needs the module loaded):
+
+```bash
+make tests
+sudo insmod kernel/tests/gsp_emulation_test.ko   # dmesg → ALL TESTS PASSED
+sudo rmmod gsp_emulation_test
+```
+
+## 4. Coding standards
+
+We follow the **Linux kernel coding style** for kernel code and common-sense
+C for userspace. The practical checklist:
+
+### Must
+- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the
+  surrounding file.
+- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate
+  `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order).
+- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use
+  `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT`
+  (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a
+  whole session — it is called out in code review.
+- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32`
+  (IOMMU IOVAs can exceed 32 bits).
+- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`,
+  `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`,
+  module refs. Unbalanced gets become leaks that block unload.
+- **Remove paths must fully unwind init paths** (we had a remove that forgot
+  `list_del` + kthread stop + VFIO unregister — leaked everything).
+- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation
+  (it rings the doorbell our own thread polls → self-trigger livelock).
+  Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists.
+- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant);
+  rate-limit anything a guest can spam.
+- Comments explain **why**, not what — especially around hardware/firmware
+  behavior discovered by reverse engineering (cite the wiki/address).
+- **Signed-off-by** style commit messages: short imperative summary line,
+  then the *why* (root cause), the *what*, and the user-visible effect.
+  Reference real dmesg/GDB evidence where relevant.
+
+### Must not
+- No guest-side Open-IOV code. Ever. The guest is stock.
+- No reboots/system-affecting commands from scripts without explicit operator
+  action (see the project's hard rule).
+- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone;
+  nvidia.ko imports *our* symbols, not vice versa.
+- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`.
+- Don't widen an API's hardware assumptions silently: if a function needs
+  `vgpu->pdev`, say so in the comment (unit tests can't call those paths).
+
+### Kernel module specifics
+- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`).
+- Exported symbols: only what other components genuinely need
+  (`EXPORT_SYMBOL` right after the definition).
+- New per-device resources belong in `struct nv_vgpu_device` with matching
+  init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain.
+
+## 5. Documentation expectations
+
+- New subsystem → new doc in `docs/` + a link from `README.md` and this file.
+- Changed behavior → update the affected doc in the **same commit**.
+- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table.
+- Firmware addresses/structures → `wiki/Address-Reference.md` and friends.
+
+## 6. Where to read next (in order)
+
+1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works
+2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist
+3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design
+4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end
+5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal
+6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference
+7. `wiki/` — firmware reverse-engineering canon
+
+## 7. Getting help / escalating
+
+- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`.
+- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first.
+- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test.
+- Found a new failure mode? Debug it per the guide, then **write it down** in
+  the taxonomy table so the next person doesn't rediscover it.