diff options
Diffstat (limited to 'firmware/57X-GSP/570.144/API_REFERENCE.md')
| -rw-r--r-- | firmware/57X-GSP/570.144/API_REFERENCE.md | 456 |
1 files changed, 456 insertions, 0 deletions
diff --git a/firmware/57X-GSP/570.144/API_REFERENCE.md b/firmware/57X-GSP/570.144/API_REFERENCE.md new file mode 100644 index 0000000..fbfbce2 --- /dev/null +++ b/firmware/57X-GSP/570.144/API_REFERENCE.md @@ -0,0 +1,456 @@ +# GSP 570.144W RPC Client API Reference + +## Overview + +This document describes the kernel-space RPC client API for communicating with the NVIDIA GSP (GPU System Processor) firmware version 570.144W (TU102/TU104). The API is derived from full firmware decompilation and provides direct access to all 12 RPC types and their sub-commands. + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Kernel Driver │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ gsp_rpc_client.c/h │ │ +│ │ - Message encoding/decoding │ │ +│ │ - RPC dispatch via doorbell │ │ +│ │ - Response handling │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ gsp_rpc_integration.c/h │ │ +│ │ - Integration with existing gsp_comm.c │ │ +│ │ - Enhanced RPC operations with firmware addresses │ │ +│ │ - WPR management │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ gsp_comm.c (existing) │ │ +│ │ - Firmware loading/boot │ │ +│ │ - Doorbell/shared memory management │ │ +│ │ - IRQ handling │ │ +│ └─────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ + │ + PCIe BAR0 Doorbell + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ GSP Firmware (570.144W) │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Primary Dispatcher (0x057a1ac0) │ │ +│ │ - Jump table for 6 RPC types │ │ +│ │ - 32 ALLOC sub-commands at 0x057a3ad8 │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Secondary Dispatcher (0x04ed39dc) │ │ +│ │ - Full implementation for all 11 RPC types │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Specialized Handlers │ │ +│ │ - ALLOC: 0x057e94ec │ │ +│ │ - SCHEDULE: 0x051efbf4 │ │ +│ │ - BAR_TRAP: 0x057675b7 / 0x04cb8189 │ │ +│ │ - MAP/PIN/UNPIN/SCHEDULE/YIELD/NOTIFY (jump table) │ │ +│ └─────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Header Files + +### `include/gsp_rpc_client.h` +Core RPC client API with all firmware addresses, message structures, and function prototypes. + +### `include/gsp_rpc_integration.h` +Integration layer API that bridges the new RPC client with existing `gsp_comm.c`. + +## RPC Types + +| Type | Value | Description | +|------|-------|-------------| +| `GSP_RPC_ALLOC` | 0x01 | Memory allocation | +| `GSP_RPC_FREE` | 0x02 | Memory free | +| `GSP_RPC_MAP` | 0x03 | Map GPU VA to physical | +| `GSP_RPC_UNMAP` | 0x04 | Unmap GPU VA | +| `GSP_RPC_PIN` | 0x05 | Pin pages for DMA | +| `GSP_RPC_UNPIN` | 0x06 | Unpin pages | +| `GSP_RPC_SCHEDULE` | 0x07 | Schedule work | +| `GSP_RPC_YIELD` | 0x08 | Yield current work | +| `GSP_RPC_NOTIFY` | 0x09 | Notification/event | +| `GSP_RPC_BAR_TRAP` | 0x0A | BAR register trap emulation | +| `GSP_RPC_MIGRATE` | 0x0B | Live migration | +| `GSP_RPC_DEBUG` | 0xFF | Debug/diagnostic | + +## Core API Functions + +### Client Lifecycle + +```c +struct gsp_client *gsp_client_init(struct device *dev, phys_addr_t fw_base, phys_addr_t doorbell); +void gsp_client_fini(struct gsp_client *client); +``` + +### Memory Management + +```c +int gsp_rpc_alloc(struct gsp_client *client, + u64 size, u64 alignment, u32 mem_type, + u32 flags, u64 addr_hint, u32 pid, + u64 *gpu_va, u64 *handle, u64 *actual_size); + +int gsp_rpc_free(struct gsp_client *client, u64 handle); + +int gsp_rpc_map(struct gsp_client *client, + u64 handle, u64 gpu_va, u64 size, + u32 prot, u32 flags, u64 offset, + u64 *mapped_va); + +int gsp_rpc_unmap(struct gsp_client *client, u64 handle, u64 gpu_va, u64 size); +``` + +### Page Pinning + +```c +int gsp_rpc_pin(struct gsp_client *client, + u64 handle, u64 offset, u64 size, + u32 flags, u64 *dma_addr, u32 *num_pages); + +int gsp_rpc_unpin(struct gsp_client *client, u64 handle, u64 offset, u64 size); +``` + +### Scheduling + +```c +int gsp_rpc_schedule(struct gsp_client *client, + u64 work_handle, u64 callback, u64 arg, + u32 priority, u32 flags); + +int gsp_rpc_yield(struct gsp_client *client); +``` + +### BAR Trap Emulation + +```c +int gsp_rpc_bar_trap(struct gsp_client *client, + u64 bar_addr, u32 access_type, u32 size, + u64 value, u32 bar_num, u64 *ret_value); +``` + +### Migration + +```c +int gsp_rpc_migrate_save(struct gsp_client *client, + u64 src_gpu_id, u64 dst_gpu_id, + u64 memory_bitmap, u64 page_table_ptr, + u64 *checkpoint); + +int gsp_rpc_migrate_restore(struct gsp_client *client, + u64 src_gpu_id, u64 dst_gpu_id, + u64 memory_bitmap, u64 page_table_ptr, + u64 checkpoint); +``` + +### WPR Management + +```c +int gsp_wpr_write_mutex_acquire(struct gsp_client *client, u32 wpr_id); +int gsp_wpr_write_mutex_release(struct gsp_client *client, u32 wpr_id); +int gsp_wpr_scratch_mutex_acquire(struct gsp_client *client, u32 wpr_id); +int gsp_wpr_scratch_mutex_release(struct gsp_client *client, u32 wpr_id); +``` + +## Integration API + +### Initialization/Cleanup + +```c +int nv_vgpu_gsp_rpc_integration_init(struct nv_vgpu_device *vgpu, + struct nv_vgpu_gsp_comm_ctx *comm_ctx); + +void nv_vgpu_gsp_rpc_integration_cleanup(struct nv_vgpu_device *vgpu); +``` + +### Enhanced RPC Operations + +All functions follow the pattern `nv_vgpu_gsp_rpc_<op>_fw()` and provide full firmware parameter support: + +```c +int nv_vgpu_gsp_rpc_alloc_fw(struct nv_vgpu_device *vgpu, + u64 size, u64 alignment, u32 mem_type, + u32 flags, u64 addr_hint, u32 pid, + u64 *gpu_va, u64 *handle, u64 *actual_size); + +int nv_vgpu_gsp_rpc_map_fw(struct nv_vgpu_device *vgpu, + u64 handle, u64 gpu_va, u64 size, + u32 prot, u32 flags, u64 offset, + u64 *mapped_va); + +int nv_vgpu_gsp_rpc_pin_fw(struct nv_vgpu_device *vgpu, + u64 handle, u64 offset, u64 size, + u32 flags, u64 *dma_addr, u32 *num_pages); + +int nv_vgpu_gsp_rpc_schedule_fw(struct nv_vgpu_device *vgpu, + u64 work_handle, u64 callback, u64 arg, + u32 priority, u32 flags); + +int nv_vgpu_gsp_rpc_bar_trap_fw(struct nv_vgpu_device *vgpu, + u64 bar_addr, u32 access_type, u32 size, + u64 value, u32 bar_num, u64 *ret_value); + +int nv_vgpu_gsp_rpc_migrate_save_fw(struct nv_vgpu_device *vgpu, + u64 src_gpu_id, u64 dst_gpu_id, + u64 memory_bitmap, u64 page_table_ptr, + u64 *checkpoint); + +int nv_vgpu_gsp_rpc_migrate_restore_fw(struct nv_vgpu_device *vgpu, + u64 src_gpu_id, u64 dst_gpu_id, + u64 memory_bitmap, u64 page_table_ptr, + u64 checkpoint); +``` + +### WPR Management + +```c +int nv_vgpu_gsp_wpr_write_mutex_acquire(struct nv_vgpu_device *vgpu, u32 wpr_id); +int nv_vgpu_gsp_wpr_write_mutex_release(struct nv_vgpu_device *vgpu, u32 wpr_id); +int nv_vgpu_gsp_wpr_scratch_mutex_acquire(struct nv_vgpu_device *vgpu, u32 wpr_id); +int nv_vgpu_gsp_wpr_scratch_mutex_release(struct nv_vgpu_device *vgpu, u32 wpr_id); +``` + +### Firmware Address Accessors + +```c +u64 nv_vgpu_gsp_get_primary_dispatcher(void); // 0x057a1ac0 +u64 nv_vgpu_gsp_get_secondary_dispatcher(void); // 0x04ed39dc +u64 nv_vgpu_gsp_get_alloc_main(void); // 0x057e94ec +u64 nv_vgpu_gsp_get_schedule_main(void); // 0x051efbf4 +u64 nv_vgpu_gsp_get_bar_trap_handler_jt(void); // 0x057675b7 +u64 nv_vgpu_gsp_get_bar_trap_handler_primary(void); // 0x04cb8189 +u64 nv_vgpu_gsp_get_entry_point(void); // 0x04009000 +``` + +## Message Structures + +All structures use little-endian encoding (`__le32`, `__le64`) for cross-architecture compatibility. + +### `struct gsp_rpc_msg` (64 bytes) +```c +struct gsp_rpc_msg { + __le32 type; // RPC type (0x01-0x0B, 0xFF) + __le32 subcmd; // Sub-command (type-specific) + __le64 payload[6]; // 48-byte payload for request data + __le32 seq_id; // Sequence ID for matching responses + __le32 flags; // Message flags + __le32 reserved[2]; // Reserved for future use +} __packed; +``` + +### `struct gsp_alloc_req` (48 bytes) +```c +struct gsp_alloc_req { + __le64 size; // Allocation size in bytes + __le64 alignment; // Required alignment + __le32 mem_type; // Memory type (VRAM, sysmem, etc.) + __le32 flags; // Allocation flags + __le64 addr_hint; // Preferred address hint + __le32 pid; // Process ID + __le32 reserved; // Reserved +} __packed; +``` + +### `struct gsp_map_req` (56 bytes) +```c +struct gsp_map_req { + __le64 handle; // Allocation handle from ALLOC + __le64 gpu_va; // Target GPU virtual address + __le64 size; // Mapping size + __le32 prot; // Protection (R/W/X) + __le32 flags; // Mapping flags + __le64 offset; // Offset into allocation + __le32 reserved[2]; // Reserved +} __packed; +``` + +### `struct gsp_pin_req` (48 bytes) +```c +struct gsp_pin_req { + __le64 handle; // Allocation handle + __le64 offset; // Offset into allocation + __le64 size; // Size to pin + __le32 flags; // Pin flags + __le32 reserved[3]; // Reserved +} __packed; +``` + +### `struct gsp_bar_trap_req` (40 bytes) +```c +struct gsp_bar_trap_req { + __le64 bar_addr; // BAR register address + __le32 access_type; // 0=read, 1=write + __le32 size; // Access size (1/2/4/8) + __le64 value; // Value for write operations + __le32 bar_num; // BAR number (0-5) + __le32 reserved; // Reserved +} __packed; +``` + +### `struct gsp_migrate_req` (56 bytes) +```c +struct gsp_migrate_req { + __le64 src_gpu_id; // Source GPU ID + __le64 dst_gpu_id; // Destination GPU ID + __le64 memory_bitmap; // Pages to migrate + __le64 page_table_ptr; // Page table pointer + __le32 state; // Migration state (save/restore) + __le32 flags; // Migration flags + __le64 context; // Checkpoint context + __le32 reserved; // Reserved +} __packed; +``` + +## ALLOC Sub-Commands (Primary Dispatcher Jump Table) + +The primary dispatcher at `0x057a1ac0` uses a jump table at `0x057a3ad8` with 32 entries for ALLOC sub-commands: + +| Index | Constant | Description | +|-------|----------|-------------| +| 0 | `GSP_ALLOC_SUBCMD_DEFAULT` | Default allocation | +| 1 | `GSP_ALLOC_SUBCMD_TYPE1` | Type 1 allocation | +| 3 | `GSP_ALLOC_SUBCMD_TYPE2` | Type 2 allocation | +| 4 | `GSP_ALLOC_SUBCMD_TYPE3` | Type 3 allocation | +| 5 | `GSP_ALLOC_SUBCMD_TYPE4` | Type 4 allocation | +| 8 | `GSP_ALLOC_SUBCMD_TYPE5` | Type 5 allocation | +| 10 | `GSP_ALLOC_SUBCMD_TYPE6` | Type 6 allocation | +| 12 | `GSP_ALLOC_SUBCMD_TYPE7` | Type 7 allocation | +| 17 | `GSP_ALLOC_SUBCMD_TYPE8` | Type 8 allocation | +| 19 | `GSP_ALLOC_SUBCMD_TYPE9` | Type 9 allocation | +| 20 | `GSP_ALLOC_SUBCMD_TYPE10` | Type 10 allocation | +| 21 | `GSP_ALLOC_SUBCMD_TYPE11` | Type 11 allocation | +| 22 | `GSP_ALLOC_SUBCMD_TYPE12` | Type 12 allocation | +| 23 | `GSP_ALLOC_SUBCMD_TYPE13` | Type 13 allocation | +| 24 | `GSP_ALLOC_SUBCMD_TYPE14` | Type 14 allocation | +| 25 | `GSP_ALLOC_SUBCMD_TYPE15` | Type 15 allocation | +| 26 | `GSP_ALLOC_SUBCMD_TYPE16` | Type 16 allocation | +| 27 | `GSP_ALLOC_SUBCMD_TYPE17` | Type 17 allocation | +| 28 | `GSP_ALLOC_SUBCMD_TYPE18` | Type 18 allocation | +| 29 | `GSP_ALLOC_SUBCMD_TYPE19` | Type 19 allocation | +| 30 | `GSP_ALLOC_SUBCMD_TYPE20` | Type 20 allocation | +| 31 | `GSP_ALLOC_SUBCMD_TYPE21` | Type 21 allocation | + +## MIGRATE Sub-Commands + +| Value | Constant | Description | +|-------|----------|-------------| +| 0x04 | `GSP_MIGRATE_SUBCMD_SAVE` | Save migration state | +| 0x05 | `GSP_MIGRATE_SUBCMD_RESTORE` | Restore migration state | + +## Firmware Address Reference + +All addresses are relative to `GSP_FW_BASE = 0x04000000`. + +| Symbol | Address | Description | +|--------|---------|-------------| +| `GSP_RPC_DISPATCH_PRIMARY` | 0x057a1ac0 | Primary dispatcher (jump table) | +| `GSP_RPC_JUMP_TABLE` | 0x057a3ad8 | 32-entry ALLOC sub-command table | +| `GSP_RPC_DISPATCH_SECONDARY` | 0x04ed39dc | Secondary dispatcher (full 11 types) | +| `GSP_RPC_DISPATCH_TERTIARY` | 0x04cc58f8 | Tertiary dispatcher (stack=160) | +| `GSP_RPC_DISPATCH_QUATERNARY` | 0x05723ef4 | Quaternary dispatcher (stack=96) | +| `GSP_ALLOC_MAIN` | 0x057e94ec | Main ALLOC implementation | +| `GSP_SCHEDULE_MAIN` | 0x051efbf4 | Main SCHEDULE implementation | +| `GSP_BAR_TRAP_HANDLER_JT` | 0x057675b7 | BAR trap handler (jump table) | +| `GSP_BAR_TRAP_HANDLER_PRIM` | 0x04cb8189 | BAR trap handler (primary) | +| `GSP_PIN_HANDLER_SEC` | 0x04ed34b8 | PIN handler (secondary) | +| `GSP_MAP_HANDLER_A` | 0x05766ab7 | MAP handler A | +| `GSP_MAP_HANDLER_B` | 0x05778a37 | MAP handler B | +| `GSP_UNMAP_HANDLER_A` | 0x057669b7 | UNMAP handler A | +| `GSP_UNMAP_HANDLER_B` | 0x05779937 | UNMAP handler B | +| `GSP_PIN_HANDLER_A` | 0x057664b7 | PIN handler A | +| `GSP_PIN_HANDLER_B` | 0x0577a3b7 | PIN handler B | +| `GSP_UNPIN_HANDLER_A` | 0x057682b7 | UNPIN handler A | +| `GSP_UNPIN_HANDLER_B` | 0x05779fb7 | UNPIN handler B | +| `GSP_SCHEDULE_HANDLER_A` | 0x05767f37 | SCHEDULE handler A | +| `GSP_SCHEDULE_HANDLER_B` | 0x0575aeb7 | SCHEDULE handler B | +| `GSP_YIELD_HANDLER_A` | 0x0575ae37 | YIELD handler A | +| `GSP_YIELD_HANDLER_B` | 0x05767337 | YIELD handler B | +| `GSP_NOTIFY_HANDLER_A` | 0x0577c8b7 | NOTIFY handler A | +| `GSP_NOTIFY_HANDLER_B` | 0x05767837 | NOTIFY handler B | +| `GSP_ENTRY_POINT` | 0x04009000 | Firmware entry point | + +## WPR Mutex Strings (from Firmware) + +The firmware contains 60 references to WPR mutex strings: +- `VPR_WPR_WRITE_MUTEX_ACQUIRE` +- `VPR_WPR_WRITE_MUTEX_RELEASE` +- `VPR_WPR_SCRATCH_MUTEX_ACQUIRE` +- `VPR_WPR_SCRATCH_MUTEX_RELEASE` +- `ACQUIRE_VPR_WPR_WRITE_MUTEX` +- `RELEASE_VPR_WPR_WRITE_MUTEX` + +These are used by functions in the `0x0407xxxx` region for WPR (Write-Protected Region) management. + +## BAR Emulation Strings (from Firmware) + +28 references to BAR-related strings: +- `BAR1`, `Bar1Rest`, `ForceBarAccess` +- `Bar1Mapping`, `Bar1Aperture`, `Bar1Size` +- `BAR0`, `BAR2`, `BAR3` + +## Building + +```bash +# Build main module with RPC client +cd $OPEN_IOV_ROOT/kernel +make + +# Build test module +cd tests +make +``` + +## Testing + +Load the test module to validate message encoding/decoding: + +```bash +insmod tests/gsp_rpc_test.ko +dmesg | grep -i "gsp rpc test" +``` + +Expected output: +``` +GSP RPC Client Test Suite Starting +=== Testing Structure Sizes === +PASS: gsp_rpc_msg size +PASS: gsp_rpc_response size +... +=== Testing Firmware Address Constants === +PASS: GSP_FW_BASE +PASS: Primary Dispatcher +... +Test Results: 85 total, 85 passed, 0 failed +TEST SUITE PASSED +``` + +## Integration with Existing Code + +The integration layer (`gsp_rpc_integration.c`) connects to existing `gsp_comm.c`: + +1. Call `nv_vgpu_gsp_rpc_integration_init(vgpu, comm_ctx)` during `nv_vgpu_gsp_init()` +2. Use `nv_vgpu_gsp_rpc_*_fw()` functions for enhanced RPC operations +3. Call `nv_vgpu_gsp_rpc_integration_cleanup(vgpu)` during `nv_vgpu_gsp_cleanup()` + +## Error Codes + +All functions return standard Linux kernel error codes: +- `0` - Success +- `-EINVAL` - Invalid parameter +- `-ENOMEM` - Out of memory +- `-ENODEV` - Device not available +- `-ETIMEDOUT` - RPC timeout +- `-EIO` - I/O error +- `-ENOTSUPP` - Operation not supported + +## License + +GPL v2 - Open-IOV Project |