diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 240 |
1 files changed, 83 insertions, 157 deletions
@@ -1,193 +1,119 @@ # Open-IOV -**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU) technology. It provides: +**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU) +technology. It lets an **unmodified NVIDIA guest driver** run inside a VM with +full GPU functionality, without a proprietary GRID license, by emulating the +GSP (GPU System Processor) RPC interface on the host. -- **open-iov-vfio.ko** - Kernel VFIO mediated device driver for vGPU -- **open-iov.so** - Userspace library for vGPU management (replaces libvgpu.so) -- **GSP Firmware Analysis** - Complete decompilation of NVIDIA GSP 570.144W (TU102/TU104) -- **RPC Interface** - 12 RPC types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG +- **open-iov-vfio.ko** — host VFIO mediated-device (mdev) kernel driver with a + host-side GSP emulation engine (all 12 RPC types: ALLOC, FREE, MAP, UNMAP, + PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG) +- **libiov.so** — userspace management library (`libvgpu.so` compat symlinks) +- **GSP firmware analysis** — complete decompilation of NVIDIA GSP 570.144 + (RV64IMAC, TU102/TU104): RPC dispatchers, jump tables, WPR, BAR trap, + migration protocol, entry/boot sequence -## Architecture +The guest VM runs the **stock, unmodified** `nvidia.ko`. All Open-IOV code +lives on the host. -``` -┌─────────────────────────────────────────────────────────────────┐ -│ Open-IOV Stack │ -├─────────────────────────────────────────────────────────────────┤ -│ Userspace │ -│ ├── open-iov.so (libopen-iov) │ -│ ├── examples/ (demo applications) │ -│ └── tests/ (unit/integration tests) │ -├─────────────────────────────────────────────────────────────────┤ -│ Kernel │ -│ ├── open-iov-vfio.ko (VFIO mdev driver) │ -│ │ ├── mdev parent registration │ -│ │ ├── VF lifecycle (create/destroy) │ -│ │ ├── BAR emulation (BAR1, ForceBarAccess, Aperture) │ -│ │ ├── GSP RPC communication (doorbell, rings) │ -│ │ ├── WPR management (write/read protect) │ -│ │ ├── Memory management (alloc, map, pin) │ -│ │ └── Live migration (4-phase protocol) │ -│ └── debugfs interface (/sys/kernel/debug/open-iov/) │ -├─────────────────────────────────────────────────────────────────┤ -│ Firmware (GSP 570.144W - TU102/TU104) │ -│ ├── Primary Dispatcher 0x057A1AC0 (32 sub-commands) │ -│ ├── Secondary Dispatcher 0x04ED39DC (141KB) │ -│ ├── BAR Trap Handler 0x04CB8188 (40KB) │ -│ ├── WPR Handlers 0x04F00B0C, 0x04F00C78 (62KB each) │ -│ ├── Migration Tables 0x04179ED8, 0x04179FBC (dual 16-entry)│ -│ └── Entry Point 0x04009000 │ -└─────────────────────────────────────────────────────────────────┘ -``` +## Current state -## Quick Start +See **[STATE.md](STATE.md)** for the live project state: what works, current +limitations, recently fixed bugs, and known open issues. -### Build Everything +## Quick start (fresh Ubuntu 22.04/24.04 + supported NVIDIA GPU) ```bash +git clone https://git.vgpu.io/open-iov.git cd open-iov -make # Builds kernel module, userspace lib, examples -``` - -### Build NVIDIA OpenRM with vGPU VFIO Symbols - -**Critical**: The NVIDIA OpenRM `nvidia.ko` must be built with `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers` to resolve the vGPU VFIO interface symbols (`nvidia_vgpu_vfio_probe`, `nvidia_vgpu_vfio_remove`, etc.). - -```bash -# Build Open-IOV kernel module first (generates Module.symvers) -cd open-iov -make kernel - -# Build NVIDIA OpenRM with vGPU VFIO symbols (use helper script) -./scripts/build_nvidia_openrm.sh - -# Or manually: -cd nvidia-open-gpu-kernel-modules/kernel-open -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules +./install.sh ``` -### Install Kernel Module - -```bash -# Copy firmware -sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ +`install.sh` reproduces the entire setup: -# Load NVIDIA OpenRM host driver (exports vGPU VFIO symbols) -sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko +| Stage | What it does | +|-----------|--------------| +| `deps` | apt build/runtime dependencies | +| `nvidia` | clone NVIDIA open-gpu-kernel-modules @ `610.57.04`, apply `patches/nvidia/` | +| `firmware`| fetch GSP 570.144 from linux-firmware.git, extract `.fwimage`, verify MD5 | +| `build` | `open-iov-vfio.ko` → `nvidia.ko` (KBUILD_EXTRA_SYMBOLS) → userspace | +| `system` | VFIO/mdev module persistence, hugepages, IOMMU/blacklist checks | +| `load` | load the driver stack (via `scripts/stack_manager.sh`) | +| `mdev` | create a vGPU mdev device | -# Load Open-IOV host VFIO driver (with GSP emulation engine) -sudo insmod kernel/open-iov-vfio.ko +Run individual stages with `./install.sh deps nvidia`, or skip the root-requiring +tail with `./install.sh --no-load`. -# Or load guest driver with emulation (legacy path) -sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin - -# Verify -dmesg -T | grep -E "gsp-vgpu|open-iov|nvidia" -ls -la /dev/gsp-vgpu /dev/open-iov -``` - -### Run Example +### Day-to-day operations ```bash -sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu +scripts/stack_manager.sh up # build all + load stack + create mdev +scripts/stack_manager.sh status # modules, mdevs, vfio, firmware, dmesg +scripts/stack_manager.sh qemu # launch reference raw-QEMU guest +scripts/stack_manager.sh unload # correct-order teardown +scripts/stack_manager.sh test # test suite ``` -## Emulation Mode - -Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers (BAR_TRAP, WPR, etc.) in userspace and kernel. +(`stack_manager.sh` auto-detects its location; override with `OPEN_IOV_ROOT`.) -### How It Works +## Using your own VM stack (libvirt, , ...) -| Component | Emulated In | RPC Types Emulated | -|-----------|-------------|-------------------| -| **Kernel** | `gsp-vgpu.ko` | BAR_TRAP (0x0A), WPR (MAP/UNMAP for WPR), ALLOC, PIN, MAP | -| **Userspace** | `libiov.so` | BAR config/force/save/restore, WPR protect/unprotect/timeout | +Open-IOV is **hypervisor-management neutral**. It provides the vGPU itself +(an mdev device + VFIO group); how you attach it to a VM is up to you: -When emulation mode is enabled: -- BAR aperture configuration is handled locally (no GSP RPC) -- WPR regions are tracked in kernel/userspace memory -- Memory allocation returns synthetic handles/VAs -- Memory pinning returns synthetic DMA addresses -- BAR trap handling is simulated +- **raw QEMU**: `scripts/launch_qemu_vgpu.sh` (reference launcher), or + `-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>` +- **libvirt**: `<hostdev mode='subsystem' type='mdev' model='vfio-pci'>` + with the UUID printed by `scripts/create_mdev.sh` +- **anything else** that can pass a VFIO mdev to QEMU -### Auto-Detection (Default) +No Open-IOV component depends on a specific wrapper. -**Emulation mode is ON by default** and auto-detects GSP capabilities: - -1. On driver load, sends a test `BAR_TRAP` RPC to GSP -2. If GSP responds successfully → vGPU handlers present → **disables emulation** -3. If GSP times out/errors → vGPU handlers missing → **enables emulation** +## Repository layout ``` -dmesg output: - gsp-vgpu: Auto-detecting GSP vGPU RPC handler support... - gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode - # or - gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode +kernel/ open-iov-vfio.ko host driver (mdev, VFIO, GSP emulation, + firmware analyzer, dynamic RPC dispatcher, arch/version layers) +userspace/ libiov.so management library, examples, tests +driver/ legacy guest driver (being removed in the host-only refactor) +firmware/ GSP analysis: scripts, docs, address maps (blobs NOT in git - + fetched by install.sh from linux-firmware.git) +patches/nvidia/ patch series applied to NVIDIA open-gpu-kernel-modules + (base tag 610.57.04; re-brands version to 570.144 to match + the GSP firmware ABI) +config/ declarative hardware (turing/ampere/blackwell) and + driver-version (550-580) YAML configs +scripts/ stack_manager.sh + helpers (wrapper-neutral) +tools/ debugging/profiling tools, Ghidra analysis scripts +docs/ architecture and implementation docs +wiki/ technical wiki (GSP firmware analysis, RPC format, ...) ``` -### Override Flags - -Force emulation mode behavior via module parameters: +**Not tracked in git** (by design): GSP firmware blobs, decompiled firmware +output, the NVIDIA kernel tree (cloned+patched by `install.sh`), VM disk +images/ISOs, Ghidra project databases, and any local VM-wrapper stacks. -| Parameter | Values | Description | -|-----------|--------|-------------| -| `emulation_mode` | `0`/`1` | Current state (read-only, shows auto-detected value) | -| `emulation_override` | `0`/`1` | **Override auto-detection**: `1`=force emulation ON, `0`=force emulation OFF | - -```bash -# Force emulation ON (for testing with GRID GSP) -sudo insmod driver/gsp-vgpu.ko emulation_override=1 - -# Force emulation OFF (requires GRID host driver with vGPU GSP) -sudo insmod driver/gsp-vgpu.ko emulation_override=0 - -# Default: auto-detect (emulation_mode=1, emulation_override=0) -sudo insmod driver/gsp-vgpu.ko -``` - -### GSP Blob Handling Options - -The vGPU RPC handlers (at 0x04CB8188, 0x04F00B0C, etc.) must be present in GSP firmware. Options: - -| Option | Description | Pros | Cons | -|--------|-------------|------|------| -| **GRID Host Driver** | Use NVIDIA GRID vGPU KVM driver (535+/550+) to boot GSP with vGPU firmware | Official, complete vGPU support | Requires NVIDIA Enterprise license | -| **OpenRM GSP** | Use GSP firmware from NVIDIA open kernel modules (OpenRM) | Open source, no license | Consumer GSP lacks vGPU RPC handlers | -| **Custom GSP Blob** | Inject vGPU handlers into GSP firmware | Full control | Complex, requires firmware signing | -| **Userspace Emulation** | Emulate vGPU RPC handlers in userspace/kernel | Works with any GSP, no license | Performance, completeness | - -**Recommended**: Use GRID host driver on host, Open-IOV guest driver in VM. For development, OpenRM GSP + emulation mode (auto-enabled). - -## GPC (GPU Partitioning Controller) Interfaces - -Open-IOV implements GPC-compatible interfaces for GPU partitioning: +## Requirements -- **VF Creation/Deletion** - `nv_vgpu_create_callback`, `nv_vgpu_delete_callback` -- **BAR Management** - `nv_vgpu_bar_info_callback` -- **Memory Operations** - Alloc, map, pin via GSP RPC -- **Migration** - 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP) -- **Scheduling** - Work submission, yield, priority +- Linux kernel 6.8+ (tested on 6.8.0-139-generic), IOMMU enabled + (`intel_iommu=on` / `amd_iommu=on`) +- NVIDIA GPU: TU102/TU104 validated (RTX 2080/2080 Super); Ampere/Blackwell + config scaffolding in place +- No nvidia module blacklist on the kernel cmdline ## Documentation -- [Firmware Analysis](wiki/GSP-570.144W-Firmware-Analysis.md) -- [RPC Message Format](wiki/RPC-Message-Format.md) -- [BAR Emulation](wiki/BAR-Emulation-Specification.md) -- [WPR Management](wiki/WPR-Management.md) -- [Migration Protocol](wiki/Migration-Protocol.md) -- [Integration Guide](wiki/vGPU-IO-Integration-Guide.md) - -## Requirements - -- Linux kernel 5.15+ (6.8+ tested) -- NVIDIA GPU: TU102/TU104 (RTX 2080/2080 Super) -- For full vGPU: GRID KVM driver 535+ or 550+ -- Build: gcc, make, kernel headers +- [STATE.md](STATE.md) — current state, limitations, bug tracker +- [Portable architecture](docs/PORTABLE_ARCHITECTURE.md) +- [Host-only GSP emulation design](docs/refactored-architecture.md) +- [Implementation TODO](docs/refactored-implementation-to-do.md) +- [Firmware analysis](wiki/GSP-570.144W-Firmware-Analysis.md) — + [RPC format](wiki/RPC-Message-Format.md) — + [BAR emulation](wiki/BAR-Emulation-Specification.md) — + [Migration protocol](wiki/Migration-Protocol.md) — + [Address reference](wiki/Address-Reference.md) ## License -GPL v2 (kernel), LGPL (userspace library) - -## Contributing - -See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
\ No newline at end of file +GPL v2 (kernel components), LGPL (userspace library). +NVIDIA firmware remains property of NVIDIA and is not distributed here. |