path: root/firmware/57X-GSP/570.144/INTEGRATION_GUIDE.md

# GSP 570.144W RPC Client Integration Guide

## Overview

This guide describes how to integrate the firmware-decompiled GSP RPC client into the vGPU-IO kernel driver. The integration provides access to all 12 RPC types with full firmware parameter support, using addresses extracted from the 28.5MB GSP firmware binary.

## Prerequisites

- Linux kernel 5.10+
- NVIDIA GPU (TU102/TU104) with GSP firmware 570.144W
- Existing vGPU-IO kernel driver (`nvidia-vgpu-vfio.ko`)
- Kernel headers for current kernel

## Files Added

### Core RPC Client
| File | Purpose |
|------|---------|
| `kernel/gsp_rpc_client.c` | Core RPC client implementation |
| `kernel/include/gsp_rpc_client.h` | Core API header with all firmware addresses |

### Integration Layer
| File | Purpose |
|------|---------|
| `kernel/gsp_rpc_integration.c` | Bridge to existing `gsp_comm.c` |
| `kernel/include/gsp_rpc_integration.h` | Integration API header |

### Testing
| File | Purpose |
|------|---------|
| `kernel/tests/gsp_rpc_test.c` | Validation test module |
| `kernel/tests/Makefile` | Test build configuration |

### Documentation
| File | Purpose |
|------|---------|
| `firmware/570-GSP/API_REFERENCE.md` | Complete API documentation |
| `firmware/570-GSP/INTEGRATION_GUIDE.md` | This file |

## Build Instructions

### 1. Build Main Module

```bash
cd $OPEN_IOV_ROOT/kernel
make clean
make
```

This builds `nvidia-vgpu-vfio.ko` with the new RPC client integrated.

### 2. Build Test Module

```bash
cd $OPEN_IOV_ROOT/kernel/tests
make clean
make
```

This builds `gsp_rpc_test.ko` for validation.

### 3. Install

```bash
# Install main module
cd $OPEN_IOV_ROOT/kernel
sudo make install

# Install test module
cd tests
sudo make install
```

## Integration Steps

### Step 1: Update gsp_comm.c to Use Integration Layer

In `kernel/gsp_comm.c`, add the integration initialization:

```c
#include "gsp_rpc_integration.h"

/* In nv_vgpu_gsp_init(), after existing init: */
ret = nv_vgpu_gsp_rpc_integration_init(vgpu, ctx);
if (ret) {
    nv_vgpu_warn(vgpu, "GSP RPC integration init failed: %d", ret);
    /* Non-fatal - continue with legacy RPC */
}

/* In nv_vgpu_gsp_cleanup(), before existing cleanup: */
nv_vgpu_gsp_rpc_integration_cleanup(vgpu);
```

### Step 2: Replace Legacy RPC Calls

Replace existing RPC functions with enhanced versions:

```c
/* Old: */
ret = nv_vgpu_gsp_alloc(vgpu, size, align, handle);

/* New: */
ret = nv_vgpu_gsp_rpc_alloc_fw(vgpu, size, align, mem_type, flags,
                                addr_hint, pid, gpu_va, handle, actual_size);
```

### Step 3: Add New Capabilities

Use new RPC types not previously available:

```c
/* Page pinning for DMA */
ret = nv_vgpu_gsp_rpc_pin_fw(vgpu, handle, offset, size, flags,
                              &dma_addr, &num_pages);

/* BAR trap emulation */
ret = nv_vgpu_gsp_rpc_bar_trap_fw(vgpu, bar_addr, access_type, size,
                                   value, bar_num, &ret_value);

/* Live migration */
ret = nv_vgpu_gsp_rpc_migrate_save_fw(vgpu, src_gpu, dst_gpu,
                                       mem_bitmap, pt_ptr, &checkpoint);

ret = nv_vgpu_gsp_rpc_migrate_restore_fw(vgpu, src_gpu, dst_gpu,
                                          mem_bitmap, pt_ptr, checkpoint);

/* WPR management */
ret = nv_vgpu_gsp_wpr_write_mutex_acquire(vgpu, wpr_id);
ret = nv_vgpu_gsp_wpr_write_mutex_release(vgpu, wpr_id);
```

## Firmware Address Validation

The integration layer exports firmware address accessors for runtime validation:

```c
/* Verify firmware matches expected version */
u64 primary = nv_vgpu_gsp_get_primary_dispatcher();
u64 entry = nv_vgpu_gsp_get_entry_point();

if (primary != 0x057a1ac0 || entry != 0x04009000) {
    pr_warn("Firmware version mismatch! Expected 570.144W\n");
}
```

## Testing

### Run Validation Tests

```bash
# Load test module
sudo insmod tests/gsp_rpc_test.ko

# Check results
dmesg -T | grep -i "gsp rpc test"

# Unload
sudo rmmod gsp_rpc_test
```

Expected output:
```
GSP RPC Client Test Suite Starting
=== Testing Structure Sizes ===
PASS: gsp_rpc_msg size
PASS: gsp_rpc_response size
...
=== Testing Firmware Address Constants ===
PASS: GSP_FW_BASE
PASS: Primary Dispatcher
...
Test Results: 85 total, 85 passed, 0 failed
TEST SUITE PASSED
```

### Test with Real Hardware

```bash
# Load main module
sudo insmod nvidia-vgpu-vfio.ko

# Check GSP initialization
dmesg -T | grep -i "gsp"

# Create mdev device
echo "nvidia-vgpu-turing" | sudo tee /sys/bus/mdev/devices/.../create

# Check RPC integration
dmesg -T | grep -i "rpc integration"
```

## Debugging

### Enable Debug Logging

Add to `kernel/Makefile`:
```makefile
ccflags-y += -DDEBUG
```

Or dynamically:
```bash
echo 8 > /sys/module/nvidia_vgpu_vfio/parameters/debug
```

### Verify Firmware Addresses

```bash
# Check firmware symbols in kernel log
dmesg | grep -E "(0x057a1ac0|0x04ed39dc|0x057e94ec|0x051efbf4)"

# Verify jump table
dmesg | grep "0x057a3ad8"
```

### Common Issues

| Issue | Cause | Solution |
|-------|-------|----------|
| `gsp_client_init` returns `-ENODEV` | BAR0 not mapped | Ensure `nv_vgpu_bar_init()` called first |
| RPC timeout | Doorbell not working | Check IRQ registration in `gsp_comm.c` |
| Structure size mismatch | Kernel version difference | Verify `__packed` attribute on all structs |
| Migration fails | WPR mutex not available | Check `0x0407xxxx` region in firmware |

## Migration from Legacy RPC

### Before (Legacy)
```c
int nv_vgpu_gsp_alloc(struct nv_vgpu_device *vgpu, u64 size, u32 align, u64 *handle)
{
    struct { u64 size; u32 align; u64 handle; } msg = { .size = size, .align = align };
    return nv_vgpu_gsp_send_rpc(vgpu, &msg, sizeof(msg));
}
```

### After (Enhanced)
```c
int nv_vgpu_gsp_rpc_alloc_fw(struct nv_vgpu_device *vgpu,
                              u64 size, u64 alignment, u32 mem_type,
                              u32 flags, u64 addr_hint, u32 pid,
                              u64 *gpu_va, u64 *handle, u64 *actual_size)
{
    struct gsp_rpc_integration *integ = g_integration;
    return gsp_rpc_alloc(integ->rpc_client, size, alignment, mem_type,
                          flags, addr_hint, pid, gpu_va, handle, actual_size);
}
```

## Performance Considerations

1. **Message Batching**: The RPC client uses shared memory for message passing - batch multiple operations when possible.

2. **Doorbell Optimization**: Write sequence ID to doorbell register only once per batch.

3. **WPR Mutex**: Cache WPR mutex state to avoid redundant acquire/release cycles.

4. **Migration**: Use `GSP_MIGRATE_SUBCMD_SAVE`/`RESTORE` for checkpoint-based migration.

## Firmware Version Compatibility

| Firmware Version | Base Address | Primary Dispatcher | Compatible |
|------------------|--------------|-------------------|------------|
| 570.144W | 0x04000000 | 0x057a1ac0 | ✅ Yes |
| 535.x | 0x04000000 | Different | ❌ No |
| 550.x | 0x04000000 | Different | ⚠️ Partial |

**Note**: This implementation is specific to GSP firmware 570.144W. Addresses will differ for other versions.

## Troubleshooting Checklist

- [ ] Kernel module builds without errors
- [ ] Test module passes all 85 tests
- [ ] Main module loads without warnings
- [ ] GSP firmware loads (check `dmesg` for "GSP firmware loaded")
- [ ] GSP boot sequence initiates (check for "GSP boot sequence initiated")
- [ ] RPC integration initializes (check for "GSP RPC integration initialized")
- [ ] Doorbell IRQ fires on RPC completion
- [ ] ALLOC/MAP/PIN operations return success
- [ ] Migration save/restore works
- [ ] WPR mutex operations work

## Support

For issues or questions:
- Check `firmware/570-GSP/DECOMPILATION_COMPLETE.md` for full decompilation details
- Check `firmware/570-GSP/RPC_HANDLER_MAP.md` for complete handler address map
- Check `firmware/570-GSP/IMPLEMENTATION_MAP.md` for 7-phase integration plan
- Review `kernel/tests/gsp_rpc_test.c` for usage examples

## License

GPL v2 - Open-IOV Project