# GSP vGPU RPC Emulation

> **Historical document — superseded.** This page describes an earlier
> emulation-mode design and is retained for reference only. The project does
> **not** use NVIDIA's proprietary vGPU stack. Canonical documentation:
> [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md)
> and [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md).

## Overview

Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.

## Problem

The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:

| Handler | Address | Size | Purpose |
|---------|---------|------|---------|
| BAR Trap | 0x04CB8188 | 40KB | BAR1/ForceBarAccess/Aperture |
| WPR Handler 1 | 0x04F00B0C | 62KB | Write/Read Protect Regions |
| WPR Handler 2 | 0x04F00C78 | 62KB | Write/Read Protect Regions |
| Migration | 0x04179ED8/0x04179FBC | - | Live migration |

Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).

## Solution: Emulation Mode

Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:

```
┌─────────────────────────────────────────────────────────────────┐
│                    Emulation Architecture                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Userspace App (libiov.so)                                       │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────┐     ┌─────────────────┐                   │
│  │ Kernel IOCTL    │────▶│ Emulation Layer │                   │
│  │ Handler         │     │ (kernel)        │                   │
│  └─────────────────┘     └────────┬────────┘                   │
│                                    │                            │
│                    ┌───────────────┼───────────────┐           │
│                    ▼               ▼               ▼           │
│             ┌──────────┐    ┌──────────┐    ┌──────────┐      │
│             │ BAR Emu  │    │ WPR Emu  │    │ Mem Emu  │      │
│             │ (kernel) │    │ (kernel) │    │ (kernel) │      │
│             └──────────┘    └──────────┘    └──────────┘      │
│                    │               │               │           │
│                    ▼               ▼               ▼           │
│             ┌──────────────────────────────────────────┐       │
│             │         Userspace Emulator (libiov.so)   │       │
│             │  rpc_emu_bar_*(), rpc_emu_wpr_*()        │       │
│             └──────────────────────────────────────────┘       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

## Auto-Detection (Default Behavior)

**Emulation mode is ON by default** and auto-detects at driver load:

1. Driver initializes RPC communication with GSP
2. Sends test `BAR_TRAP` RPC (non-destructive READ)
3. **If GSP responds** (ret=0) → vGPU handlers present → **emulation disabled**
4. **If GSP times out/errors** (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → **emulation enabled**

```
dmesg output:
  gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
  gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
  # or
  gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode
```

## Module Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `emulation_mode` | bool | 1 (ON) | Current state (read-only, shows auto-detected value) |
| `emulation_override` | bool | 0 (auto) | Override: 1=force emulation ON, 0=force emulation OFF |

```bash
# Default: auto-detect
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# Force emulation ON (testing with GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1

# Force emulation OFF (requires GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0
```

## Emulated RPC Handlers

### Kernel Emulation (`driver/`)

| RPC Type | Handler | Emulation |
|----------|---------|-----------|
| `GSP_RPC_ALLOC` (0x01) | `gsp_vf_mem_alloc()` | Synthetic handles/VAs from local counter |
| `GSP_RPC_PIN` (0x05) | `gsp_vf_mem_pin()` | Synthetic DMA addresses from local counter |
| `GSP_RPC_MAP` (0x03) | `gsp_vf_mem_map()` | Identity map (return requested VA) |
| `GSP_RPC_BAR_TRAP` (0x0A) | `gsp_bar_trap_handler()` | Local aperture/force_access state |
| `GSP_RPC_MAP` (WPR) | `gsp_vf_wpr_protect()` | Local WPR region list |
| `GSP_RPC_UNMAP` (WPR) | `gsp_vf_wpr_unprotect()` | Local WPR region removal |

### Userspace Emulation (`userspace/lib/rpc_emulator.c`)

| Function | Purpose |
|----------|---------|
| `rpc_emu_bar_config()` | Local aperture offset/size + backing store |
| `rpc_emu_bar_force_access()` | Local force_access state |
| `rpc_emu_bar_save/restore()` | Migration state serialization |
| `rpc_emu_wpr_protect_write/read()` | Local WPR region tracking |
| `rpc_emu_wpr_unprotect()` | Local WPR region removal |
| `rpc_emu_wpr_set_timeout()` | Local timeout tracking |
| `rpc_emu_wpr_check_access()` | Access violation detection |

## Emulation Data Structures

### BAR State (per VF)
```c
struct gsp_bar_state {
    uint64_t aperture_offset;   // BAR1 aperture base
    uint64_t aperture_size;     // BAR1 aperture size (max 256KB)
    bool force_access;          // ForceBarAccess enabled
    uint32_t force_vf;          // VF holding force access
    void *backing_store;        // Emulated BAR memory
    size_t backing_size;        // Backing store size
};
```

### WPR Region List (per VF)
```c
struct gsp_wpr_region {
    struct list_head list;
    uint64_t base;              // Region base address
    uint64_t size;              // Region size
    uint32_t flags;             // GSP_WPR_WRITE/READ/EXECUTE
    uint32_t vf_id;             // Owning VF
    void (*completion_cb)(...); // Write/read complete callback
    void (*timeout_cb)(...);    // Timeout callback
    uint64_t timeout_ns;        // Timeout in nanoseconds
};
```

### Userspace Emulator State
```c
struct emu_bar_state {
    uint64_t aperture_offset;
    uint64_t aperture_size;
    bool force_access;
    uint32_t force_vf;
    void *backing_store;
    size_t backing_size;
    pthread_mutex_t lock;
};

struct emu_wpr_state {
    struct emu_wpr_region *regions;  // Linked list
    pthread_mutex_t lock;
};
```

## Fallback Logic

Userspace library (`libiov.so`) implements fallback:

```c
vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
    // 1. Try kernel IOCTL (sends RPC to GSP)
    ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
    
    // 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
    if (ret < 0 && dev->emulation_enabled && 
        (errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
        return rpc_emu_bar_config(vf->id, offset, size);
    }
    
    return map_errno(errno);
}
```

## Limitations

| Feature | Emulation Status | Notes |
|---------|-----------------|-------|
| BAR Aperture Config | ✅ Full | Local state + backing store |
| ForceBarAccess | ✅ Full | Per-VF exclusive access |
| BAR Save/Restore | ✅ Full | Migration support |
| WPR Write Protect | ✅ Full | Region tracking + access check |
| WPR Read Protect | ✅ Full | Region tracking + access check |
| WPR Timeout | ✅ Partial | Timeout stored, callbacks not wired |
| Memory Alloc | ✅ Basic | Synthetic handles/VAs |
| Memory Pin | ✅ Basic | Synthetic DMA addresses |
| Memory Map | ✅ Basic | Identity mapping |
| Migration | ⚠️ Partial | BAR/WPR state only |
| Scheduling | ❌ No | Returns success, no actual scheduling |
| Notifications | ❌ No | No async event delivery |

## Testing Emulation

```bash
# 1. Load driver (auto-enables emulation on consumer GSP)
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# 2. Verify emulation enabled in dmesg
dmesg -T | grep -i emulation
# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode

# 3. Run example - should complete without timeouts
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu

# 4. Check debugfs for emulated state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions
```

## Future Work

- [ ] Wire WPR timeout callbacks to kernel timers
- [ ] Implement emulated scheduling with work queues
- [ ] Add async notification support via eventfd
- [ ] Complete migration state for WPR regions
- [ ] Performance optimization (batch RPC emulation)
- [ ] Conformance tests vs GRID GSP behavior

## Related Documentation

- [BAR Emulation Specification](BAR-Emulation-Specification.md)
- [WPR Management](WPR-Management.md)
- [RPC Message Format](RPC-Message-Format.md)
- [Integration Guide](vGPU-IO-Integration-Guide.md)