# GSP 570.144W RPC Client API Reference ## Overview This document describes the kernel-space RPC client API for communicating with the NVIDIA GSP (GPU System Processor) firmware version 570.144W (TU102/TU104). The API is derived from full firmware decompilation and provides direct access to all 12 RPC types and their sub-commands. ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Kernel Driver │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ gsp_rpc_client.c/h │ │ │ │ - Message encoding/decoding │ │ │ │ - RPC dispatch via doorbell │ │ │ │ - Response handling │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ gsp_rpc_integration.c/h │ │ │ │ - Integration with existing gsp_comm.c │ │ │ │ - Enhanced RPC operations with firmware addresses │ │ │ │ - WPR management │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ gsp_comm.c (existing) │ │ │ │ - Firmware loading/boot │ │ │ │ - Doorbell/shared memory management │ │ │ │ - IRQ handling │ │ │ └─────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ │ PCIe BAR0 Doorbell │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ GSP Firmware (570.144W) │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Primary Dispatcher (0x057a1ac0) │ │ │ │ - Jump table for 6 RPC types │ │ │ │ - 32 ALLOC sub-commands at 0x057a3ad8 │ │ │ └─────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Secondary Dispatcher (0x04ed39dc) │ │ │ │ - Full implementation for all 11 RPC types │ │ │ └─────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Specialized Handlers │ │ │ │ - ALLOC: 0x057e94ec │ │ │ │ - SCHEDULE: 0x051efbf4 │ │ │ │ - BAR_TRAP: 0x057675b7 / 0x04cb8189 │ │ │ │ - MAP/PIN/UNPIN/SCHEDULE/YIELD/NOTIFY (jump table) │ │ │ └─────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` ## Header Files ### `include/gsp_rpc_client.h` Core RPC client API with all firmware addresses, message structures, and function prototypes. ### `include/gsp_rpc_integration.h` Integration layer API that bridges the new RPC client with existing `gsp_comm.c`. ## RPC Types | Type | Value | Description | |------|-------|-------------| | `GSP_RPC_ALLOC` | 0x01 | Memory allocation | | `GSP_RPC_FREE` | 0x02 | Memory free | | `GSP_RPC_MAP` | 0x03 | Map GPU VA to physical | | `GSP_RPC_UNMAP` | 0x04 | Unmap GPU VA | | `GSP_RPC_PIN` | 0x05 | Pin pages for DMA | | `GSP_RPC_UNPIN` | 0x06 | Unpin pages | | `GSP_RPC_SCHEDULE` | 0x07 | Schedule work | | `GSP_RPC_YIELD` | 0x08 | Yield current work | | `GSP_RPC_NOTIFY` | 0x09 | Notification/event | | `GSP_RPC_BAR_TRAP` | 0x0A | BAR register trap emulation | | `GSP_RPC_MIGRATE` | 0x0B | Live migration | | `GSP_RPC_DEBUG` | 0xFF | Debug/diagnostic | ## Core API Functions ### Client Lifecycle ```c struct gsp_client *gsp_client_init(struct device *dev, phys_addr_t fw_base, phys_addr_t doorbell); void gsp_client_fini(struct gsp_client *client); ``` ### Memory Management ```c int gsp_rpc_alloc(struct gsp_client *client, u64 size, u64 alignment, u32 mem_type, u32 flags, u64 addr_hint, u32 pid, u64 *gpu_va, u64 *handle, u64 *actual_size); int gsp_rpc_free(struct gsp_client *client, u64 handle); int gsp_rpc_map(struct gsp_client *client, u64 handle, u64 gpu_va, u64 size, u32 prot, u32 flags, u64 offset, u64 *mapped_va); int gsp_rpc_unmap(struct gsp_client *client, u64 handle, u64 gpu_va, u64 size); ``` ### Page Pinning ```c int gsp_rpc_pin(struct gsp_client *client, u64 handle, u64 offset, u64 size, u32 flags, u64 *dma_addr, u32 *num_pages); int gsp_rpc_unpin(struct gsp_client *client, u64 handle, u64 offset, u64 size); ``` ### Scheduling ```c int gsp_rpc_schedule(struct gsp_client *client, u64 work_handle, u64 callback, u64 arg, u32 priority, u32 flags); int gsp_rpc_yield(struct gsp_client *client); ``` ### BAR Trap Emulation ```c int gsp_rpc_bar_trap(struct gsp_client *client, u64 bar_addr, u32 access_type, u32 size, u64 value, u32 bar_num, u64 *ret_value); ``` ### Migration ```c int gsp_rpc_migrate_save(struct gsp_client *client, u64 src_gpu_id, u64 dst_gpu_id, u64 memory_bitmap, u64 page_table_ptr, u64 *checkpoint); int gsp_rpc_migrate_restore(struct gsp_client *client, u64 src_gpu_id, u64 dst_gpu_id, u64 memory_bitmap, u64 page_table_ptr, u64 checkpoint); ``` ### WPR Management ```c int gsp_wpr_write_mutex_acquire(struct gsp_client *client, u32 wpr_id); int gsp_wpr_write_mutex_release(struct gsp_client *client, u32 wpr_id); int gsp_wpr_scratch_mutex_acquire(struct gsp_client *client, u32 wpr_id); int gsp_wpr_scratch_mutex_release(struct gsp_client *client, u32 wpr_id); ``` ## Integration API ### Initialization/Cleanup ```c int nv_vgpu_gsp_rpc_integration_init(struct nv_vgpu_device *vgpu, struct nv_vgpu_gsp_comm_ctx *comm_ctx); void nv_vgpu_gsp_rpc_integration_cleanup(struct nv_vgpu_device *vgpu); ``` ### Enhanced RPC Operations All functions follow the pattern `nv_vgpu_gsp_rpc__fw()` and provide full firmware parameter support: ```c int nv_vgpu_gsp_rpc_alloc_fw(struct nv_vgpu_device *vgpu, u64 size, u64 alignment, u32 mem_type, u32 flags, u64 addr_hint, u32 pid, u64 *gpu_va, u64 *handle, u64 *actual_size); int nv_vgpu_gsp_rpc_map_fw(struct nv_vgpu_device *vgpu, u64 handle, u64 gpu_va, u64 size, u32 prot, u32 flags, u64 offset, u64 *mapped_va); int nv_vgpu_gsp_rpc_pin_fw(struct nv_vgpu_device *vgpu, u64 handle, u64 offset, u64 size, u32 flags, u64 *dma_addr, u32 *num_pages); int nv_vgpu_gsp_rpc_schedule_fw(struct nv_vgpu_device *vgpu, u64 work_handle, u64 callback, u64 arg, u32 priority, u32 flags); int nv_vgpu_gsp_rpc_bar_trap_fw(struct nv_vgpu_device *vgpu, u64 bar_addr, u32 access_type, u32 size, u64 value, u32 bar_num, u64 *ret_value); int nv_vgpu_gsp_rpc_migrate_save_fw(struct nv_vgpu_device *vgpu, u64 src_gpu_id, u64 dst_gpu_id, u64 memory_bitmap, u64 page_table_ptr, u64 *checkpoint); int nv_vgpu_gsp_rpc_migrate_restore_fw(struct nv_vgpu_device *vgpu, u64 src_gpu_id, u64 dst_gpu_id, u64 memory_bitmap, u64 page_table_ptr, u64 checkpoint); ``` ### WPR Management ```c int nv_vgpu_gsp_wpr_write_mutex_acquire(struct nv_vgpu_device *vgpu, u32 wpr_id); int nv_vgpu_gsp_wpr_write_mutex_release(struct nv_vgpu_device *vgpu, u32 wpr_id); int nv_vgpu_gsp_wpr_scratch_mutex_acquire(struct nv_vgpu_device *vgpu, u32 wpr_id); int nv_vgpu_gsp_wpr_scratch_mutex_release(struct nv_vgpu_device *vgpu, u32 wpr_id); ``` ### Firmware Address Accessors ```c u64 nv_vgpu_gsp_get_primary_dispatcher(void); // 0x057a1ac0 u64 nv_vgpu_gsp_get_secondary_dispatcher(void); // 0x04ed39dc u64 nv_vgpu_gsp_get_alloc_main(void); // 0x057e94ec u64 nv_vgpu_gsp_get_schedule_main(void); // 0x051efbf4 u64 nv_vgpu_gsp_get_bar_trap_handler_jt(void); // 0x057675b7 u64 nv_vgpu_gsp_get_bar_trap_handler_primary(void); // 0x04cb8189 u64 nv_vgpu_gsp_get_entry_point(void); // 0x04009000 ``` ## Message Structures All structures use little-endian encoding (`__le32`, `__le64`) for cross-architecture compatibility. ### `struct gsp_rpc_msg` (64 bytes) ```c struct gsp_rpc_msg { __le32 type; // RPC type (0x01-0x0B, 0xFF) __le32 subcmd; // Sub-command (type-specific) __le64 payload[6]; // 48-byte payload for request data __le32 seq_id; // Sequence ID for matching responses __le32 flags; // Message flags __le32 reserved[2]; // Reserved for future use } __packed; ``` ### `struct gsp_alloc_req` (48 bytes) ```c struct gsp_alloc_req { __le64 size; // Allocation size in bytes __le64 alignment; // Required alignment __le32 mem_type; // Memory type (VRAM, sysmem, etc.) __le32 flags; // Allocation flags __le64 addr_hint; // Preferred address hint __le32 pid; // Process ID __le32 reserved; // Reserved } __packed; ``` ### `struct gsp_map_req` (56 bytes) ```c struct gsp_map_req { __le64 handle; // Allocation handle from ALLOC __le64 gpu_va; // Target GPU virtual address __le64 size; // Mapping size __le32 prot; // Protection (R/W/X) __le32 flags; // Mapping flags __le64 offset; // Offset into allocation __le32 reserved[2]; // Reserved } __packed; ``` ### `struct gsp_pin_req` (48 bytes) ```c struct gsp_pin_req { __le64 handle; // Allocation handle __le64 offset; // Offset into allocation __le64 size; // Size to pin __le32 flags; // Pin flags __le32 reserved[3]; // Reserved } __packed; ``` ### `struct gsp_bar_trap_req` (40 bytes) ```c struct gsp_bar_trap_req { __le64 bar_addr; // BAR register address __le32 access_type; // 0=read, 1=write __le32 size; // Access size (1/2/4/8) __le64 value; // Value for write operations __le32 bar_num; // BAR number (0-5) __le32 reserved; // Reserved } __packed; ``` ### `struct gsp_migrate_req` (56 bytes) ```c struct gsp_migrate_req { __le64 src_gpu_id; // Source GPU ID __le64 dst_gpu_id; // Destination GPU ID __le64 memory_bitmap; // Pages to migrate __le64 page_table_ptr; // Page table pointer __le32 state; // Migration state (save/restore) __le32 flags; // Migration flags __le64 context; // Checkpoint context __le32 reserved; // Reserved } __packed; ``` ## ALLOC Sub-Commands (Primary Dispatcher Jump Table) The primary dispatcher at `0x057a1ac0` uses a jump table at `0x057a3ad8` with 32 entries for ALLOC sub-commands: | Index | Constant | Description | |-------|----------|-------------| | 0 | `GSP_ALLOC_SUBCMD_DEFAULT` | Default allocation | | 1 | `GSP_ALLOC_SUBCMD_TYPE1` | Type 1 allocation | | 3 | `GSP_ALLOC_SUBCMD_TYPE2` | Type 2 allocation | | 4 | `GSP_ALLOC_SUBCMD_TYPE3` | Type 3 allocation | | 5 | `GSP_ALLOC_SUBCMD_TYPE4` | Type 4 allocation | | 8 | `GSP_ALLOC_SUBCMD_TYPE5` | Type 5 allocation | | 10 | `GSP_ALLOC_SUBCMD_TYPE6` | Type 6 allocation | | 12 | `GSP_ALLOC_SUBCMD_TYPE7` | Type 7 allocation | | 17 | `GSP_ALLOC_SUBCMD_TYPE8` | Type 8 allocation | | 19 | `GSP_ALLOC_SUBCMD_TYPE9` | Type 9 allocation | | 20 | `GSP_ALLOC_SUBCMD_TYPE10` | Type 10 allocation | | 21 | `GSP_ALLOC_SUBCMD_TYPE11` | Type 11 allocation | | 22 | `GSP_ALLOC_SUBCMD_TYPE12` | Type 12 allocation | | 23 | `GSP_ALLOC_SUBCMD_TYPE13` | Type 13 allocation | | 24 | `GSP_ALLOC_SUBCMD_TYPE14` | Type 14 allocation | | 25 | `GSP_ALLOC_SUBCMD_TYPE15` | Type 15 allocation | | 26 | `GSP_ALLOC_SUBCMD_TYPE16` | Type 16 allocation | | 27 | `GSP_ALLOC_SUBCMD_TYPE17` | Type 17 allocation | | 28 | `GSP_ALLOC_SUBCMD_TYPE18` | Type 18 allocation | | 29 | `GSP_ALLOC_SUBCMD_TYPE19` | Type 19 allocation | | 30 | `GSP_ALLOC_SUBCMD_TYPE20` | Type 20 allocation | | 31 | `GSP_ALLOC_SUBCMD_TYPE21` | Type 21 allocation | ## MIGRATE Sub-Commands | Value | Constant | Description | |-------|----------|-------------| | 0x04 | `GSP_MIGRATE_SUBCMD_SAVE` | Save migration state | | 0x05 | `GSP_MIGRATE_SUBCMD_RESTORE` | Restore migration state | ## Firmware Address Reference All addresses are relative to `GSP_FW_BASE = 0x04000000`. | Symbol | Address | Description | |--------|---------|-------------| | `GSP_RPC_DISPATCH_PRIMARY` | 0x057a1ac0 | Primary dispatcher (jump table) | | `GSP_RPC_JUMP_TABLE` | 0x057a3ad8 | 32-entry ALLOC sub-command table | | `GSP_RPC_DISPATCH_SECONDARY` | 0x04ed39dc | Secondary dispatcher (full 11 types) | | `GSP_RPC_DISPATCH_TERTIARY` | 0x04cc58f8 | Tertiary dispatcher (stack=160) | | `GSP_RPC_DISPATCH_QUATERNARY` | 0x05723ef4 | Quaternary dispatcher (stack=96) | | `GSP_ALLOC_MAIN` | 0x057e94ec | Main ALLOC implementation | | `GSP_SCHEDULE_MAIN` | 0x051efbf4 | Main SCHEDULE implementation | | `GSP_BAR_TRAP_HANDLER_JT` | 0x057675b7 | BAR trap handler (jump table) | | `GSP_BAR_TRAP_HANDLER_PRIM` | 0x04cb8189 | BAR trap handler (primary) | | `GSP_PIN_HANDLER_SEC` | 0x04ed34b8 | PIN handler (secondary) | | `GSP_MAP_HANDLER_A` | 0x05766ab7 | MAP handler A | | `GSP_MAP_HANDLER_B` | 0x05778a37 | MAP handler B | | `GSP_UNMAP_HANDLER_A` | 0x057669b7 | UNMAP handler A | | `GSP_UNMAP_HANDLER_B` | 0x05779937 | UNMAP handler B | | `GSP_PIN_HANDLER_A` | 0x057664b7 | PIN handler A | | `GSP_PIN_HANDLER_B` | 0x0577a3b7 | PIN handler B | | `GSP_UNPIN_HANDLER_A` | 0x057682b7 | UNPIN handler A | | `GSP_UNPIN_HANDLER_B` | 0x05779fb7 | UNPIN handler B | | `GSP_SCHEDULE_HANDLER_A` | 0x05767f37 | SCHEDULE handler A | | `GSP_SCHEDULE_HANDLER_B` | 0x0575aeb7 | SCHEDULE handler B | | `GSP_YIELD_HANDLER_A` | 0x0575ae37 | YIELD handler A | | `GSP_YIELD_HANDLER_B` | 0x05767337 | YIELD handler B | | `GSP_NOTIFY_HANDLER_A` | 0x0577c8b7 | NOTIFY handler A | | `GSP_NOTIFY_HANDLER_B` | 0x05767837 | NOTIFY handler B | | `GSP_ENTRY_POINT` | 0x04009000 | Firmware entry point | ## WPR Mutex Strings (from Firmware) The firmware contains 60 references to WPR mutex strings: - `VPR_WPR_WRITE_MUTEX_ACQUIRE` - `VPR_WPR_WRITE_MUTEX_RELEASE` - `VPR_WPR_SCRATCH_MUTEX_ACQUIRE` - `VPR_WPR_SCRATCH_MUTEX_RELEASE` - `ACQUIRE_VPR_WPR_WRITE_MUTEX` - `RELEASE_VPR_WPR_WRITE_MUTEX` These are used by functions in the `0x0407xxxx` region for WPR (Write-Protected Region) management. ## BAR Emulation Strings (from Firmware) 28 references to BAR-related strings: - `BAR1`, `Bar1Rest`, `ForceBarAccess` - `Bar1Mapping`, `Bar1Aperture`, `Bar1Size` - `BAR0`, `BAR2`, `BAR3` ## Building ```bash # Build main module with RPC client cd $OPEN_IOV_ROOT/kernel make # Build test module cd tests make ``` ## Testing Load the test module to validate message encoding/decoding: ```bash insmod tests/gsp_rpc_test.ko dmesg | grep -i "gsp rpc test" ``` Expected output: ``` GSP RPC Client Test Suite Starting === Testing Structure Sizes === PASS: gsp_rpc_msg size PASS: gsp_rpc_response size ... === Testing Firmware Address Constants === PASS: GSP_FW_BASE PASS: Primary Dispatcher ... Test Results: 85 total, 85 passed, 0 failed TEST SUITE PASSED ``` ## Integration with Existing Code The integration layer (`gsp_rpc_integration.c`) connects to existing `gsp_comm.c`: 1. Call `nv_vgpu_gsp_rpc_integration_init(vgpu, comm_ctx)` during `nv_vgpu_gsp_init()` 2. Use `nv_vgpu_gsp_rpc_*_fw()` functions for enhanced RPC operations 3. Call `nv_vgpu_gsp_rpc_integration_cleanup(vgpu)` during `nv_vgpu_gsp_cleanup()` ## Error Codes All functions return standard Linux kernel error codes: - `0` - Success - `-EINVAL` - Invalid parameter - `-ENOMEM` - Out of memory - `-ENODEV` - Device not available - `-ETIMEDOUT` - RPC timeout - `-EIO` - I/O error - `-ENOTSUPP` - Operation not supported ## License GPL v2 - Open-IOV Project