# Open-IOV

**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU)
technology. It lets an **unmodified NVIDIA guest driver** run inside a VM with
full GPU functionality, without a proprietary GRID license, by emulating the
GSP (GPU System Processor) RPC interface on the host.

- **open-iov-vfio.ko** — host VFIO mediated-device (mdev) kernel driver with a
  host-side GSP emulation engine (all 12 RPC types: ALLOC, FREE, MAP, UNMAP,
  PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG)
- **libiov.so** — userspace management library (`libvgpu.so` compat symlinks)
- **GSP firmware analysis** — complete decompilation of NVIDIA GSP 570.144
  (RV64IMAC, TU102/TU104): RPC dispatchers, jump tables, WPR, BAR trap,
  migration protocol, entry/boot sequence

The guest VM runs the **stock, unmodified** `nvidia.ko`. All Open-IOV code
lives on the host.

## Current state

See **[STATE.md](STATE.md)** for the live project state: what works, current
limitations, recently fixed bugs, and known open issues.

## Quick start (fresh Ubuntu 22.04/24.04 + supported NVIDIA GPU)

```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh
```

`install.sh` reproduces the entire setup:

| Stage     | What it does |
|-----------|--------------|
| `deps`    | apt build/runtime dependencies |
| `nvidia`  | clone NVIDIA open-gpu-kernel-modules @ `610.57.04`, apply `patches/nvidia/` |
| `firmware`| fetch GSP 570.144 from linux-firmware.git, extract `.fwimage`, verify MD5 |
| `build`   | `open-iov-vfio.ko` → `nvidia.ko` (KBUILD_EXTRA_SYMBOLS) → userspace |
| `system`  | VFIO/mdev module persistence, hugepages, IOMMU/blacklist checks |
| `load`    | load the driver stack (via `scripts/stack_manager.sh`) |
| `mdev`    | create a vGPU mdev device |

Run individual stages with `./install.sh deps nvidia`, or skip the root-requiring
tail with `./install.sh --no-load`.

### Day-to-day operations

```bash
scripts/stack_manager.sh up        # build all + load stack + create mdev
scripts/stack_manager.sh status    # modules, mdevs, vfio, firmware, dmesg
scripts/stack_manager.sh qemu      # launch reference raw-QEMU guest
scripts/stack_manager.sh unload    # correct-order teardown
scripts/stack_manager.sh test      # test suite
```

(`stack_manager.sh` auto-detects its location; override with `OPEN_IOV_ROOT`.)

## Using your own VM stack (libvirt, , ...)

Open-IOV is **hypervisor-management neutral**. It provides the vGPU itself
(an mdev device + VFIO group); how you attach it to a VM is up to you:

- **raw QEMU**: `scripts/launch_qemu_vgpu.sh` (reference launcher), or
  `-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>`
- **libvirt**: `<hostdev mode='subsystem' type='mdev' model='vfio-pci'>`
  with the UUID printed by `scripts/create_mdev.sh`
- **anything else** that can pass a VFIO mdev to QEMU

No Open-IOV component depends on a specific wrapper.

## Repository layout

```
kernel/         open-iov-vfio.ko host driver (mdev, VFIO, GSP emulation,
                firmware analyzer, dynamic RPC dispatcher, arch/version layers)
userspace/      libiov.so management library, examples, tests
legacy-driver-guest/  historical guest-side driver, preserved for reference
                only - NOT built (see its README)
firmware/       GSP analysis: scripts, docs, address maps (blobs NOT in git -
                fetched by install.sh from linux-firmware.git)
patches/nvidia/ patch series applied to NVIDIA open-gpu-kernel-modules
                (base tag 610.57.04; re-brands version to 570.144 to match
                the GSP firmware ABI)
config/         declarative hardware (turing/ampere/blackwell) and
                driver-version (550-580) YAML configs
scripts/        stack_manager.sh + helpers (wrapper-neutral)
tools/          debugging/profiling tools, Ghidra analysis scripts
docs/           architecture and implementation docs
wiki/           technical wiki (GSP firmware analysis, RPC format, ...)
```

**Not tracked in git** (by design): GSP firmware blobs, decompiled firmware
output, the NVIDIA kernel tree (cloned+patched by `install.sh`), VM disk
images/ISOs, Ghidra project databases, and any local VM-wrapper stacks.

## Requirements

- Linux kernel 6.8+ (tested on 6.8.0-139-generic), IOMMU enabled
  (`intel_iommu=on` / `amd_iommu=on`)
- NVIDIA GPU: TU102/TU104 validated (RTX 2080/2080 Super); Ampere/Blackwell
  config scaffolding in place
- No nvidia module blacklist on the kernel cmdline

## Documentation

Start here: **[Developer onboarding](docs/ONBOARDING.md)** — repo tour,
coding standards, first build & test.

- [STATE.md](STATE.md) — current state, limitations, bug tracker
- [GSP emulation engine architecture](docs/GSP_EMULATION_ARCHITECTURE.md) —
  how the host-side emulator works (shadow BAR0, doorbell protocol, dispatch,
  MSI-X injection)
- [Debugging guide](docs/DEBUGGING.md) — QEMU GDB stub, serial console,
  failure taxonomy, bring-up validation checklist
- [Version upgrade guide](docs/VERSION_UPGRADE.md) — supporting a new NVIDIA
  driver/GSP version end-to-end (Ghidra workflow)
- [Performance testing](docs/PERFORMANCE.md) — bare-metal A/B methodology
- [Portable architecture](docs/PORTABLE_ARCHITECTURE.md) — multi-version,
  multi-hardware design
- [Host-only GSP emulation design](docs/refactored-architecture.md) —
  original refactor plan
- [Implementation TODO](docs/refactored-implementation-to-do.md)
- [Development workflow reference](docs/DEVELOPMENT_WORKFLOW.md) — exhaustive
  command reference
- Firmware canon: [analysis](wiki/GSP-570.144W-Firmware-Analysis.md) —
  [RPC format](wiki/RPC-Message-Format.md) —
  [BAR emulation](wiki/BAR-Emulation-Specification.md) —
  [migration protocol](wiki/Migration-Protocol.md) —
  [address reference](wiki/Address-Reference.md)

## License

GPL v2 (kernel components), LGPL (userspace library).
NVIDIA firmware remains property of NVIDIA and is not distributed here.