path: root/firmware/57X-GSP/570.144/API_REFERENCE.md

# GSP 570.144W RPC Client API Reference

## Overview

This document describes the kernel-space RPC client API for communicating with the NVIDIA GSP (GPU System Processor) firmware version 570.144W (TU102/TU104). The API is derived from full firmware decompilation and provides direct access to all 12 RPC types and their sub-commands.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Kernel Driver                            │
│  ┌─────────────────────────────────────────────────────┐   │
│  │            gsp_rpc_client.c/h                        │   │
│  │  - Message encoding/decoding                         │   │
│  │  - RPC dispatch via doorbell                         │   │
│  │  - Response handling                                 │   │
│  └─────────────────────────────────────────────────────┘   │
│                              │                              │
│                              ▼                              │
│  ┌─────────────────────────────────────────────────────┐   │
│  │            gsp_rpc_integration.c/h                   │   │
│  │  - Integration with existing gsp_comm.c              │   │
│  │  - Enhanced RPC operations with firmware addresses   │   │
│  │  - WPR management                                    │   │
│  └─────────────────────────────────────────────────────┘   │
│                              │                              │
│                              ▼                              │
│  ┌─────────────────────────────────────────────────────┐   │
│  │            gsp_comm.c (existing)                     │   │
│  │  - Firmware loading/boot                             │   │
│  │  - Doorbell/shared memory management                 │   │
│  │  - IRQ handling                                      │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                              │
                    PCIe BAR0 Doorbell
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    GSP Firmware (570.144W)                  │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Primary Dispatcher (0x057a1ac0)                    │   │
│  │  - Jump table for 6 RPC types                       │   │
│  │  - 32 ALLOC sub-commands at 0x057a3ad8              │   │
│  └─────────────────────────────────────────────────────┘   │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Secondary Dispatcher (0x04ed39dc)                  │   │
│  │  - Full implementation for all 11 RPC types         │   │
│  └─────────────────────────────────────────────────────┘   │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Specialized Handlers                                │   │
│  │  - ALLOC: 0x057e94ec                                │   │
│  │  - SCHEDULE: 0x051efbf4                             │   │
│  │  - BAR_TRAP: 0x057675b7 / 0x04cb8189                │   │
│  │  - MAP/PIN/UNPIN/SCHEDULE/YIELD/NOTIFY (jump table) │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
```

## Header Files

### `include/gsp_rpc_client.h`
Core RPC client API with all firmware addresses, message structures, and function prototypes.

### `include/gsp_rpc_integration.h`
Integration layer API that bridges the new RPC client with existing `gsp_comm.c`.

## RPC Types

| Type | Value | Description |
|------|-------|-------------|
| `GSP_RPC_ALLOC` | 0x01 | Memory allocation |
| `GSP_RPC_FREE` | 0x02 | Memory free |
| `GSP_RPC_MAP` | 0x03 | Map GPU VA to physical |
| `GSP_RPC_UNMAP` | 0x04 | Unmap GPU VA |
| `GSP_RPC_PIN` | 0x05 | Pin pages for DMA |
| `GSP_RPC_UNPIN` | 0x06 | Unpin pages |
| `GSP_RPC_SCHEDULE` | 0x07 | Schedule work |
| `GSP_RPC_YIELD` | 0x08 | Yield current work |
| `GSP_RPC_NOTIFY` | 0x09 | Notification/event |
| `GSP_RPC_BAR_TRAP` | 0x0A | BAR register trap emulation |
| `GSP_RPC_MIGRATE` | 0x0B | Live migration |
| `GSP_RPC_DEBUG` | 0xFF | Debug/diagnostic |

## Core API Functions

### Client Lifecycle

```c
struct gsp_client *gsp_client_init(struct device *dev, phys_addr_t fw_base, phys_addr_t doorbell);
void gsp_client_fini(struct gsp_client *client);
```

### Memory Management

```c
int gsp_rpc_alloc(struct gsp_client *client,
                   u64 size, u64 alignment, u32 mem_type,
                   u32 flags, u64 addr_hint, u32 pid,
                   u64 *gpu_va, u64 *handle, u64 *actual_size);

int gsp_rpc_free(struct gsp_client *client, u64 handle);

int gsp_rpc_map(struct gsp_client *client,
                 u64 handle, u64 gpu_va, u64 size,
                 u32 prot, u32 flags, u64 offset,
                 u64 *mapped_va);

int gsp_rpc_unmap(struct gsp_client *client, u64 handle, u64 gpu_va, u64 size);
```

### Page Pinning

```c
int gsp_rpc_pin(struct gsp_client *client,
                 u64 handle, u64 offset, u64 size,
                 u32 flags, u64 *dma_addr, u32 *num_pages);

int gsp_rpc_unpin(struct gsp_client *client, u64 handle, u64 offset, u64 size);
```

### Scheduling

```c
int gsp_rpc_schedule(struct gsp_client *client,
                      u64 work_handle, u64 callback, u64 arg,
                      u32 priority, u32 flags);

int gsp_rpc_yield(struct gsp_client *client);
```

### BAR Trap Emulation

```c
int gsp_rpc_bar_trap(struct gsp_client *client,
                      u64 bar_addr, u32 access_type, u32 size,
                      u64 value, u32 bar_num, u64 *ret_value);
```

### Migration

```c
int gsp_rpc_migrate_save(struct gsp_client *client,
                          u64 src_gpu_id, u64 dst_gpu_id,
                          u64 memory_bitmap, u64 page_table_ptr,
                          u64 *checkpoint);

int gsp_rpc_migrate_restore(struct gsp_client *client,
                             u64 src_gpu_id, u64 dst_gpu_id,
                             u64 memory_bitmap, u64 page_table_ptr,
                             u64 checkpoint);
```

### WPR Management

```c
int gsp_wpr_write_mutex_acquire(struct gsp_client *client, u32 wpr_id);
int gsp_wpr_write_mutex_release(struct gsp_client *client, u32 wpr_id);
int gsp_wpr_scratch_mutex_acquire(struct gsp_client *client, u32 wpr_id);
int gsp_wpr_scratch_mutex_release(struct gsp_client *client, u32 wpr_id);
```

## Integration API

### Initialization/Cleanup

```c
int nv_vgpu_gsp_rpc_integration_init(struct nv_vgpu_device *vgpu,
                                      struct nv_vgpu_gsp_comm_ctx *comm_ctx);

void nv_vgpu_gsp_rpc_integration_cleanup(struct nv_vgpu_device *vgpu);
```

### Enhanced RPC Operations

All functions follow the pattern `nv_vgpu_gsp_rpc_<op>_fw()` and provide full firmware parameter support:

```c
int nv_vgpu_gsp_rpc_alloc_fw(struct nv_vgpu_device *vgpu,
                              u64 size, u64 alignment, u32 mem_type,
                              u32 flags, u64 addr_hint, u32 pid,
                              u64 *gpu_va, u64 *handle, u64 *actual_size);

int nv_vgpu_gsp_rpc_map_fw(struct nv_vgpu_device *vgpu,
                            u64 handle, u64 gpu_va, u64 size,
                            u32 prot, u32 flags, u64 offset,
                            u64 *mapped_va);

int nv_vgpu_gsp_rpc_pin_fw(struct nv_vgpu_device *vgpu,
                            u64 handle, u64 offset, u64 size,
                            u32 flags, u64 *dma_addr, u32 *num_pages);

int nv_vgpu_gsp_rpc_schedule_fw(struct nv_vgpu_device *vgpu,
                                 u64 work_handle, u64 callback, u64 arg,
                                 u32 priority, u32 flags);

int nv_vgpu_gsp_rpc_bar_trap_fw(struct nv_vgpu_device *vgpu,
                                 u64 bar_addr, u32 access_type, u32 size,
                                 u64 value, u32 bar_num, u64 *ret_value);

int nv_vgpu_gsp_rpc_migrate_save_fw(struct nv_vgpu_device *vgpu,
                                     u64 src_gpu_id, u64 dst_gpu_id,
                                     u64 memory_bitmap, u64 page_table_ptr,
                                     u64 *checkpoint);

int nv_vgpu_gsp_rpc_migrate_restore_fw(struct nv_vgpu_device *vgpu,
                                        u64 src_gpu_id, u64 dst_gpu_id,
                                        u64 memory_bitmap, u64 page_table_ptr,
                                        u64 checkpoint);
```

### WPR Management

```c
int nv_vgpu_gsp_wpr_write_mutex_acquire(struct nv_vgpu_device *vgpu, u32 wpr_id);
int nv_vgpu_gsp_wpr_write_mutex_release(struct nv_vgpu_device *vgpu, u32 wpr_id);
int nv_vgpu_gsp_wpr_scratch_mutex_acquire(struct nv_vgpu_device *vgpu, u32 wpr_id);
int nv_vgpu_gsp_wpr_scratch_mutex_release(struct nv_vgpu_device *vgpu, u32 wpr_id);
```

### Firmware Address Accessors

```c
u64 nv_vgpu_gsp_get_primary_dispatcher(void);       // 0x057a1ac0
u64 nv_vgpu_gsp_get_secondary_dispatcher(void);     // 0x04ed39dc
u64 nv_vgpu_gsp_get_alloc_main(void);               // 0x057e94ec
u64 nv_vgpu_gsp_get_schedule_main(void);            // 0x051efbf4
u64 nv_vgpu_gsp_get_bar_trap_handler_jt(void);      // 0x057675b7
u64 nv_vgpu_gsp_get_bar_trap_handler_primary(void); // 0x04cb8189
u64 nv_vgpu_gsp_get_entry_point(void);              // 0x04009000
```

## Message Structures

All structures use little-endian encoding (`__le32`, `__le64`) for cross-architecture compatibility.

### `struct gsp_rpc_msg` (64 bytes)
```c
struct gsp_rpc_msg {
    __le32 type;           // RPC type (0x01-0x0B, 0xFF)
    __le32 subcmd;         // Sub-command (type-specific)
    __le64 payload[6];     // 48-byte payload for request data
    __le32 seq_id;         // Sequence ID for matching responses
    __le32 flags;          // Message flags
    __le32 reserved[2];    // Reserved for future use
} __packed;
```

### `struct gsp_alloc_req` (48 bytes)
```c
struct gsp_alloc_req {
    __le64 size;           // Allocation size in bytes
    __le64 alignment;      // Required alignment
    __le32 mem_type;       // Memory type (VRAM, sysmem, etc.)
    __le32 flags;          // Allocation flags
    __le64 addr_hint;      // Preferred address hint
    __le32 pid;            // Process ID
    __le32 reserved;       // Reserved
} __packed;
```

### `struct gsp_map_req` (56 bytes)
```c
struct gsp_map_req {
    __le64 handle;         // Allocation handle from ALLOC
    __le64 gpu_va;         // Target GPU virtual address
    __le64 size;           // Mapping size
    __le32 prot;           // Protection (R/W/X)
    __le32 flags;          // Mapping flags
    __le64 offset;         // Offset into allocation
    __le32 reserved[2];    // Reserved
} __packed;
```

### `struct gsp_pin_req` (48 bytes)
```c
struct gsp_pin_req {
    __le64 handle;         // Allocation handle
    __le64 offset;         // Offset into allocation
    __le64 size;           // Size to pin
    __le32 flags;          // Pin flags
    __le32 reserved[3];    // Reserved
} __packed;
```

### `struct gsp_bar_trap_req` (40 bytes)
```c
struct gsp_bar_trap_req {
    __le64 bar_addr;       // BAR register address
    __le32 access_type;    // 0=read, 1=write
    __le32 size;           // Access size (1/2/4/8)
    __le64 value;          // Value for write operations
    __le32 bar_num;        // BAR number (0-5)
    __le32 reserved;       // Reserved
} __packed;
```

### `struct gsp_migrate_req` (56 bytes)
```c
struct gsp_migrate_req {
    __le64 src_gpu_id;     // Source GPU ID
    __le64 dst_gpu_id;     // Destination GPU ID
    __le64 memory_bitmap;  // Pages to migrate
    __le64 page_table_ptr; // Page table pointer
    __le32 state;          // Migration state (save/restore)
    __le32 flags;          // Migration flags
    __le64 context;        // Checkpoint context
    __le32 reserved;       // Reserved
} __packed;
```

## ALLOC Sub-Commands (Primary Dispatcher Jump Table)

The primary dispatcher at `0x057a1ac0` uses a jump table at `0x057a3ad8` with 32 entries for ALLOC sub-commands:

| Index | Constant | Description |
|-------|----------|-------------|
| 0 | `GSP_ALLOC_SUBCMD_DEFAULT` | Default allocation |
| 1 | `GSP_ALLOC_SUBCMD_TYPE1` | Type 1 allocation |
| 3 | `GSP_ALLOC_SUBCMD_TYPE2` | Type 2 allocation |
| 4 | `GSP_ALLOC_SUBCMD_TYPE3` | Type 3 allocation |
| 5 | `GSP_ALLOC_SUBCMD_TYPE4` | Type 4 allocation |
| 8 | `GSP_ALLOC_SUBCMD_TYPE5` | Type 5 allocation |
| 10 | `GSP_ALLOC_SUBCMD_TYPE6` | Type 6 allocation |
| 12 | `GSP_ALLOC_SUBCMD_TYPE7` | Type 7 allocation |
| 17 | `GSP_ALLOC_SUBCMD_TYPE8` | Type 8 allocation |
| 19 | `GSP_ALLOC_SUBCMD_TYPE9` | Type 9 allocation |
| 20 | `GSP_ALLOC_SUBCMD_TYPE10` | Type 10 allocation |
| 21 | `GSP_ALLOC_SUBCMD_TYPE11` | Type 11 allocation |
| 22 | `GSP_ALLOC_SUBCMD_TYPE12` | Type 12 allocation |
| 23 | `GSP_ALLOC_SUBCMD_TYPE13` | Type 13 allocation |
| 24 | `GSP_ALLOC_SUBCMD_TYPE14` | Type 14 allocation |
| 25 | `GSP_ALLOC_SUBCMD_TYPE15` | Type 15 allocation |
| 26 | `GSP_ALLOC_SUBCMD_TYPE16` | Type 16 allocation |
| 27 | `GSP_ALLOC_SUBCMD_TYPE17` | Type 17 allocation |
| 28 | `GSP_ALLOC_SUBCMD_TYPE18` | Type 18 allocation |
| 29 | `GSP_ALLOC_SUBCMD_TYPE19` | Type 19 allocation |
| 30 | `GSP_ALLOC_SUBCMD_TYPE20` | Type 20 allocation |
| 31 | `GSP_ALLOC_SUBCMD_TYPE21` | Type 21 allocation |

## MIGRATE Sub-Commands

| Value | Constant | Description |
|-------|----------|-------------|
| 0x04 | `GSP_MIGRATE_SUBCMD_SAVE` | Save migration state |
| 0x05 | `GSP_MIGRATE_SUBCMD_RESTORE` | Restore migration state |

## Firmware Address Reference

All addresses are relative to `GSP_FW_BASE = 0x04000000`.

| Symbol | Address | Description |
|--------|---------|-------------|
| `GSP_RPC_DISPATCH_PRIMARY` | 0x057a1ac0 | Primary dispatcher (jump table) |
| `GSP_RPC_JUMP_TABLE` | 0x057a3ad8 | 32-entry ALLOC sub-command table |
| `GSP_RPC_DISPATCH_SECONDARY` | 0x04ed39dc | Secondary dispatcher (full 11 types) |
| `GSP_RPC_DISPATCH_TERTIARY` | 0x04cc58f8 | Tertiary dispatcher (stack=160) |
| `GSP_RPC_DISPATCH_QUATERNARY` | 0x05723ef4 | Quaternary dispatcher (stack=96) |
| `GSP_ALLOC_MAIN` | 0x057e94ec | Main ALLOC implementation |
| `GSP_SCHEDULE_MAIN` | 0x051efbf4 | Main SCHEDULE implementation |
| `GSP_BAR_TRAP_HANDLER_JT` | 0x057675b7 | BAR trap handler (jump table) |
| `GSP_BAR_TRAP_HANDLER_PRIM` | 0x04cb8189 | BAR trap handler (primary) |
| `GSP_PIN_HANDLER_SEC` | 0x04ed34b8 | PIN handler (secondary) |
| `GSP_MAP_HANDLER_A` | 0x05766ab7 | MAP handler A |
| `GSP_MAP_HANDLER_B` | 0x05778a37 | MAP handler B |
| `GSP_UNMAP_HANDLER_A` | 0x057669b7 | UNMAP handler A |
| `GSP_UNMAP_HANDLER_B` | 0x05779937 | UNMAP handler B |
| `GSP_PIN_HANDLER_A` | 0x057664b7 | PIN handler A |
| `GSP_PIN_HANDLER_B` | 0x0577a3b7 | PIN handler B |
| `GSP_UNPIN_HANDLER_A` | 0x057682b7 | UNPIN handler A |
| `GSP_UNPIN_HANDLER_B` | 0x05779fb7 | UNPIN handler B |
| `GSP_SCHEDULE_HANDLER_A` | 0x05767f37 | SCHEDULE handler A |
| `GSP_SCHEDULE_HANDLER_B` | 0x0575aeb7 | SCHEDULE handler B |
| `GSP_YIELD_HANDLER_A` | 0x0575ae37 | YIELD handler A |
| `GSP_YIELD_HANDLER_B` | 0x05767337 | YIELD handler B |
| `GSP_NOTIFY_HANDLER_A` | 0x0577c8b7 | NOTIFY handler A |
| `GSP_NOTIFY_HANDLER_B` | 0x05767837 | NOTIFY handler B |
| `GSP_ENTRY_POINT` | 0x04009000 | Firmware entry point |

## WPR Mutex Strings (from Firmware)

The firmware contains 60 references to WPR mutex strings:
- `VPR_WPR_WRITE_MUTEX_ACQUIRE`
- `VPR_WPR_WRITE_MUTEX_RELEASE`
- `VPR_WPR_SCRATCH_MUTEX_ACQUIRE`
- `VPR_WPR_SCRATCH_MUTEX_RELEASE`
- `ACQUIRE_VPR_WPR_WRITE_MUTEX`
- `RELEASE_VPR_WPR_WRITE_MUTEX`

These are used by functions in the `0x0407xxxx` region for WPR (Write-Protected Region) management.

## BAR Emulation Strings (from Firmware)

28 references to BAR-related strings:
- `BAR1`, `Bar1Rest`, `ForceBarAccess`
- `Bar1Mapping`, `Bar1Aperture`, `Bar1Size`
- `BAR0`, `BAR2`, `BAR3`

## Building

```bash
# Build main module with RPC client
cd $OPEN_IOV_ROOT/kernel
make

# Build test module
cd tests
make
```

## Testing

Load the test module to validate message encoding/decoding:

```bash
insmod tests/gsp_rpc_test.ko
dmesg | grep -i "gsp rpc test"
```

Expected output:
```
GSP RPC Client Test Suite Starting
=== Testing Structure Sizes ===
PASS: gsp_rpc_msg size
PASS: gsp_rpc_response size
...
=== Testing Firmware Address Constants ===
PASS: GSP_FW_BASE
PASS: Primary Dispatcher
...
Test Results: 85 total, 85 passed, 0 failed
TEST SUITE PASSED
```

## Integration with Existing Code

The integration layer (`gsp_rpc_integration.c`) connects to existing `gsp_comm.c`:

1. Call `nv_vgpu_gsp_rpc_integration_init(vgpu, comm_ctx)` during `nv_vgpu_gsp_init()`
2. Use `nv_vgpu_gsp_rpc_*_fw()` functions for enhanced RPC operations
3. Call `nv_vgpu_gsp_rpc_integration_cleanup(vgpu)` during `nv_vgpu_gsp_cleanup()`

## Error Codes

All functions return standard Linux kernel error codes:
- `0` - Success
- `-EINVAL` - Invalid parameter
- `-ENOMEM` - Out of memory
- `-ENODEV` - Device not available
- `-ETIMEDOUT` - RPC timeout
- `-EIO` - I/O error
- `-ENOTSUPP` - Operation not supported

## License

GPL v2 - Open-IOV Project