# Open-IOV

**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU) technology. It provides:

- **open-iov-vfio.ko** - Kernel VFIO mediated device driver for vGPU
- **open-iov.so** - Userspace library for vGPU management (replaces libvgpu.so)
- **GSP Firmware Analysis** - Complete decompilation of NVIDIA GSP 570.144W (TU102/TU104)
- **RPC Interface** - 12 RPC types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        Open-IOV Stack                            │
├─────────────────────────────────────────────────────────────────┤
│  Userspace                                                       │
│  ├── open-iov.so          (libopen-iov)                         │
│  ├── examples/            (demo applications)                   │
│  └── tests/               (unit/integration tests)              │
├─────────────────────────────────────────────────────────────────┤
│  Kernel                                                          │
│  ├── open-iov-vfio.ko     (VFIO mdev driver)                    │
│  │   ├── mdev parent registration                                │
│  │   ├── VF lifecycle (create/destroy)                          │
│  │   ├── BAR emulation (BAR1, ForceBarAccess, Aperture)         │
│  │   ├── GSP RPC communication (doorbell, rings)                │
│  │   ├── WPR management (write/read protect)                    │
│  │   ├── Memory management (alloc, map, pin)                   │
│  │   └── Live migration (4-phase protocol)                     │
│  └── debugfs interface    (/sys/kernel/debug/open-iov/)         │
├─────────────────────────────────────────────────────────────────┤
│  Firmware (GSP 570.144W - TU102/TU104)                          │
│  ├── Primary Dispatcher   0x057A1AC0 (32 sub-commands)          │
│  ├── Secondary Dispatcher 0x04ED39DC (141KB)                    │
│  ├── BAR Trap Handler     0x04CB8188 (40KB)                     │
│  ├── WPR Handlers         0x04F00B0C, 0x04F00C78 (62KB each)   │
│  ├── Migration Tables     0x04179ED8, 0x04179FBC (dual 16-entry)│
│  └── Entry Point          0x04009000                            │
└─────────────────────────────────────────────────────────────────┘
```

## Quick Start

### Build Everything

```bash
cd open-iov
make                    # Builds kernel module, userspace lib, examples
```

### Install Kernel Module

```bash
# Copy firmware
sudo cp firmware/570-GSP/gsp-570.144.fwimage.bin /lib/firmware/

# Load kernel module (emulation mode ON by default, auto-detects GSP capabilities)
sudo insmod kernel/open-iov-vfio.ko

# Or load guest driver with emulation
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# Verify
dmesg -T | grep -E "gsp-vgpu|open-iov"
ls -la /dev/gsp-vgpu /dev/open-iov
```

### Run Example

```bash
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu
```

## Emulation Mode

Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers (BAR_TRAP, WPR, etc.) in userspace and kernel.

### How It Works

| Component | Emulated In | RPC Types Emulated |
|-----------|-------------|-------------------|
| **Kernel** | `gsp-vgpu.ko` | BAR_TRAP (0x0A), WPR (MAP/UNMAP for WPR), ALLOC, PIN, MAP |
| **Userspace** | `libiov.so` | BAR config/force/save/restore, WPR protect/unprotect/timeout |

When emulation mode is enabled:
- BAR aperture configuration is handled locally (no GSP RPC)
- WPR regions are tracked in kernel/userspace memory
- Memory allocation returns synthetic handles/VAs
- Memory pinning returns synthetic DMA addresses
- BAR trap handling is simulated

### Auto-Detection (Default)

**Emulation mode is ON by default** and auto-detects GSP capabilities:

1. On driver load, sends a test `BAR_TRAP` RPC to GSP
2. If GSP responds successfully → vGPU handlers present → **disables emulation**
3. If GSP times out/errors → vGPU handlers missing → **enables emulation**

```
dmesg output:
  gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
  gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
  # or
  gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode
```

### Override Flags

Force emulation mode behavior via module parameters:

| Parameter | Values | Description |
|-----------|--------|-------------|
| `emulation_mode` | `0`/`1` | Current state (read-only, shows auto-detected value) |
| `emulation_override` | `0`/`1` | **Override auto-detection**: `1`=force emulation ON, `0`=force emulation OFF |

```bash
# Force emulation ON (for testing with GRID GSP)
sudo insmod driver/gsp-vgpu.ko emulation_override=1

# Force emulation OFF (requires GRID host driver with vGPU GSP)
sudo insmod driver/gsp-vgpu.ko emulation_override=0

# Default: auto-detect (emulation_mode=1, emulation_override=0)
sudo insmod driver/gsp-vgpu.ko
```

### GSP Blob Handling Options

The vGPU RPC handlers (at 0x04CB8188, 0x04F00B0C, etc.) must be present in GSP firmware. Options:

| Option | Description | Pros | Cons |
|--------|-------------|------|------|
| **GRID Host Driver** | Use NVIDIA GRID vGPU KVM driver (535+/550+) to boot GSP with vGPU firmware | Official, complete vGPU support | Requires NVIDIA Enterprise license |
| **OpenRM GSP** | Use GSP firmware from NVIDIA open kernel modules (OpenRM) | Open source, no license | Consumer GSP lacks vGPU RPC handlers |
| **Custom GSP Blob** | Inject vGPU handlers into GSP firmware | Full control | Complex, requires firmware signing |
| **Userspace Emulation** | Emulate vGPU RPC handlers in userspace/kernel | Works with any GSP, no license | Performance, completeness |

**Recommended**: Use GRID host driver on host, Open-IOV guest driver in VM. For development, OpenRM GSP + emulation mode (auto-enabled).

## GPC (GPU Partitioning Controller) Interfaces

Open-IOV implements GPC-compatible interfaces for GPU partitioning:

- **VF Creation/Deletion** - `nv_vgpu_create_callback`, `nv_vgpu_delete_callback`
- **BAR Management** - `nv_vgpu_bar_info_callback`
- **Memory Operations** - Alloc, map, pin via GSP RPC
- **Migration** - 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP)
- **Scheduling** - Work submission, yield, priority

## Documentation

- [Firmware Analysis](wiki/GSP-570.144W-Firmware-Analysis.md)
- [RPC Message Format](wiki/RPC-Message-Format.md)
- [BAR Emulation](wiki/BAR-Emulation-Specification.md)
- [WPR Management](wiki/WPR-Management.md)
- [Migration Protocol](wiki/Migration-Protocol.md)
- [Integration Guide](wiki/vGPU-IO-Integration-Guide.md)

## Requirements

- Linux kernel 5.15+ (6.8+ tested)
- NVIDIA GPU: TU102/TU104 (RTX 2080/2080 Super)
- For full vGPU: GRID KVM driver 535+ or 550+
- Build: gcc, make, kernel headers

## License

GPL v2 (kernel), LGPL (userspace library)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.