diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-14 06:43:11 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-14 06:43:11 -0400 |
| commit | 5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60 (patch) | |
| tree | 7523e29d8044d4d5b6dcd7c1c888a3491137fcd1 /docs | |
| parent | 9a7263fce4e3156349a526631805fce66b87fb1b (diff) | |
| download | open-iov-5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60.tar.gz open-iov-5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60.zip | |
VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl handlers
- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create()
- Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions
- Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP
- Added config space read/write handlers at offset 0x70000000000
- Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers)
- Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL)
- Added udev rules for vfio group permissions
- Created stack_manager.sh unified tool
- Fixed mdev remove crash with deferred cleanup in vfio_device_release callback
- Kernel 6.8+ IOMMUFD compatibility
- QEMU launches successfully with vGPU passthrough
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/BUILD_NVIDIA_OPENRM.md | 116 | ||||
| -rw-r--r-- | docs/DEVELOPMENT_WORKFLOW.md | 660 | ||||
| -rw-r--r-- | docs/PORTABLE_ARCHITECTURE.md | 742 | ||||
| -rw-r--r-- | docs/old/ARCHITECTURE.md (renamed from docs/ARCHITECTURE.md) | 0 | ||||
| -rw-r--r-- | docs/refactored-architecture.md | 595 | ||||
| -rw-r--r-- | docs/tools/README.md | 75 | ||||
| -rw-r--r-- | docs/tools/rmtool.md | 149 | ||||
| -rw-r--r-- | docs/tools/rmtop.md | 227 |
8 files changed, 2564 insertions, 0 deletions
diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md new file mode 100644 index 0000000..5699036 --- /dev/null +++ b/docs/BUILD_NVIDIA_OPENRM.md @@ -0,0 +1,116 @@ +# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols + +## Overview + +The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: +- `nvidia_vgpu_vfio_probe` +- `nvidia_vgpu_vfio_remove` +- `nvidia_vgpu_vfio_get_ops` +- `nvidia_vgpu_vfio_set_ops` +- `nv_vgpu_mdev_register_parent` +- `nv_vgpu_mdev_unregister_parent` +- And all GSP RPC integration symbols + +These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`. + +## Build Order (Critical) + +``` +1. Build open-iov-vfio.ko → Generates kernel/Module.symvers +2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers +3. Load nvidia.ko first +4. Load open-iov-vfio.ko +``` + +## Automated Build (Recommended) + +Use the helper script: + +```bash +cd open-iov +./scripts/build_nvidia_openrm.sh +``` + +This script: +1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`) +2. Cleans previous NVIDIA OpenRM build +3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set +4. Verifies vGPU symbols are resolved + +## Manual Build + +```bash +# Step 1: Build Open-IOV kernel module +cd $OPEN_IOV_ROOT +make kernel + +# Step 2: Build NVIDIA OpenRM with extra symbols +cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Step 3: Verify symbols resolved +grep -i vgpu Module.symvers +``` + +## Verification + +After building, verify the symbols are resolved: + +```bash +# Check Module.symvers has vGPU symbols +grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers + +# Check nvidia.ko modinfo +modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` | +| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first | +| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild | +| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config | + +## Module Loading Order + +```bash +# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko) +sudo insmod kernel/open-iov-vfio.ko + +# 3. Copy firmware +sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ + +# 4. Create mdev device +./scripts/create_mdev.sh + +# 5. Launch QEMU with vGPU +./scripts/launch_qemu_vgpu.sh +``` + +## Key Files + +| File | Purpose | +|------|---------| +| `scripts/build_nvidia_openrm.sh` | Automated build script | +| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko | +| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment | +| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild | + +## Troubleshooting + +If symbols are still unresolved after rebuild: + +```bash +# Full clean rebuild +cd nvidia-open-gpu-kernel-modules/kernel-open +make clean +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Check modpost output for unresolved symbols +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu" +```
\ No newline at end of file diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md new file mode 100644 index 0000000..97aa267 --- /dev/null +++ b/docs/DEVELOPMENT_WORKFLOW.md @@ -0,0 +1,660 @@ +# Open-IOV vGPU-IO Development Workflow & Tools Guide + +## Overview + +This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging. + +--- + +## 1. Repository Structure & Navigation + +### Primary Workspace +```bash +~/Git/open-iov/ # Main monorepo (git + LFS) +``` + +### Key Directories +| Directory | Purpose | +|-----------|---------| +| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) | +| `driver/` | Guest vGPU driver (`gsp-vgpu.ko`) - **to be removed** | +| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` | +| `userspace/examples/` | `example_vgpu` test app | +| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files | +| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) | +| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) | +| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent | +| `docs/` | Architecture, implementation guides, TODO lists | +| `wiki/` | 10 technical wiki pages | + +--- + +## 2. Shell Integration (zshrc) + +### Required Aliases & Functions +Add to `~/.zshrc` and run `source ~/.zshrc`: + +```bash +# Project navigation +alias oiv="cd ~/Git/open-iov" +alias oivk="cd ~/Git/open-iov/kernel" +alias oivu="cd ~/Git/open-iov/userspace" +alias oivf="cd ~/Git/open-iov/firmware/57X-GSP/570.144" +alias oivd="cd ~/Git/open-iov/docs" +alias oivs="cd ~/Git/open-iov/scripts" +alias oivt="cd ~/Git/open-iov/tools" + +# Build commands +alias oiv-build="cd ~/Git/open-iov && make -j$(nproc)" +alias oiv-build-kernel="cd ~/Git/open-iov && make kernel" +alias oiv-build-userspace="cd ~/Git/open-iov && make userspace" +alias oiv-clean="cd ~/Git/open-iov && make clean" + +# Module loading (requires reboot first!) +alias oiv-load="~/Git/open-iov/scripts/load_vgpu_stack.sh" +alias oiv-unload="~/Git/open-iov/scripts/unload_vgpu_stack.sh" +alias oiv-test="~/Git/open-iov/scripts/test_vgpu_stack.sh" +alias oiv-test-gsp="~/Git/open-iov/scripts/test_gsp_emu.sh" +alias oiv-test-guest="~/Git/open-iov/scripts/test_guest_driver.sh" +alias oiv-test-cuda="~/Git/open-iov/scripts/test_cuda_workloads.sh" +alias oiv-test-graphics="~/Git/open-iov/scripts/test_graphics_workloads.sh" +alias oiv-test-windows="~/Git/open-iov/scripts/test_windows_guest.sh" + +# mdev management +alias oiv-mdev-create="~/Git/open-iov/scripts/create_mdev.sh" +alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'" + +# QEMU +alias oiv-qemu="~/Git/open-iov/scripts/launch_qemu_vgpu.sh" +alias oiv-qemu-debug="~/Git/open-iov/scripts/debug_guest_init.sh" + +# Firmware analysis +alias oiv-ghidra="cd ~/Git/open-iov/tools/ghidra && ./ghidraRun" +alias oiv-fw-analyze="cd ~/Git/open-iov && python3 tools/firmware_analyzer.py" + +# Debugging tools (NEW) +alias oiv-guest-monitor="python3 ~/Git/open-iov/tools/guest_init_monitor.py" +alias oiv-failure-analyzer="python3 ~/Git/open-iov/tools/failure_analyzer.py" +alias oiv-benchmark="~/Git/open-iov/tools/benchmark_infrastructure.sh" +alias oiv-windows-test="~/Git/open-iov/scripts/windows_guest_automation.sh" + +# Kernel debugging +alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" +alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" +alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic" +alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu" + +# Git +alias oiv-git="cd ~/Git/open-iov && git status" +alias oiv-git-log="cd ~/Git/open-iov && git log --oneline -20" +alias oiv-git-diff="cd ~/Git/open-iov && git diff" + +# Restore point +alias oiv-restore="cd ~/Git/open-iov && git checkout f9ed5d8 -- . && git clean -fd" + +# Help +oiv-help() { + echo "╔═══════════════════════════════════════════════════════════════════════════════╗" + echo "║ Open-IOV vGPU-IO Development Commands ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ NAVIGATION ║" + echo "║ oiv - cd to open-iov root ║" + echo "║ oivk - cd to kernel/ ║" + echo "║ oivu - cd to userspace/ ║" + echo "║ oivf - cd to firmware/57X-GSP/570.144/ ║" + echo "║ oivd - cd to docs/ ║" + echo "║ oivs - cd to scripts/ ║" + echo "║ oivt - cd to tools/ ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ BUILD ║" + echo "║ oiv-build - Build everything (kernel + userspace + examples) ║" + echo "║ oiv-build-kernel - Build open-iov-vfio.ko only ║" + echo "║ oiv-build-userspace - Build libiov.so + example_vgpu ║" + echo "║ oiv-clean - Clean all build artifacts ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ MODULE OPERATIONS (require root) ║" + echo "║ oiv-load - Load nvidia.ko → open-iov-vfio.ko (refactored stack) ║" + echo "║ oiv-unload - Unload all modules + remove mdevs ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ TESTING ║" + echo "║ oiv-test - Run comprehensive test suite (test_vgpu_stack.sh) ║" + echo "║ oiv-test-gsp - Run GSP emulation unit tests (test_gsp_emu.sh) ║" + echo "║ oiv-test-guest - Test guest driver integration (test_guest_driver.sh)║" + echo "║ oiv-test-cuda - Test CUDA workloads (test_cuda_workloads.sh) ║" + echo "║ oiv-test-graphics - Test graphics workloads (test_graphics_workloads.sh)║" + echo "║ oiv-test-windows - Test Windows guest (test_windows_guest.sh) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ MDEV MANAGEMENT ║" + echo "║ oiv-mdev-create - Create mdev device for QEMU ║" + echo "║ oiv-mdev-list - List existing mdev devices ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ QEMU ║" + echo "║ oiv-qemu - Launch QEMU with vGPU ║" + echo "║ oiv-qemu-debug - Launch QEMU with GDB + guest monitor ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ DEBUGGING ║" + echo "║ oiv-dmesg - Filtered dmesg for NVIDIA/GSP/vGPU/mdev ║" + echo "║ oiv-dmesg-follow - Live filtered dmesg (tail -f) ║" + echo "║ oiv-dmesg-errors - Show only errors/warnings ║" + echo "║ oiv-trace - Live kernel trace_pipe filtered for gsp_emu ║" + echo "║ oiv-guest-monitor - Monitor guest init (correlates host traces + serial) ║" + echo "║ oiv-failure-analyzer - Analyze/classify guest init failures ║" + echo "║ oiv-benchmark - Run VM vs bare metal benchmarks ║" + echo "║ oiv-windows-test - Windows guest test automation (WinRM) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ GIT ║" + echo "║ oiv-git - Git status in open-iov ║" + echo "║ oiv-git-log - Git log (last 20 commits) ║" + echo "║ oiv-git-diff - Show uncommitted changes ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ FIRMWARE ANALYSIS ║" + echo "║ oiv-ghidra - Launch Ghidra GUI ║" + echo "║ oiv-fw-analyze - Run firmware analyzer (Python) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ EMERGENCY ║" + echo "║ oiv-restore - Restore to commit f9ed5d8 (restore point) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ oiv-help - This help ║" + echo "╚═══════════════════════════════════════════════════════════════════════════════╝" +} +``` + +### Usage +```bash +# After adding to ~/.zshrc: +source ~/.zshrc +oiv-help # Show all commands +oiv # Jump to project root +oiv-build # Build everything +oiv-load # Load kernel modules +oiv-mdev-create # Create mdev for QEMU +oiv-qemu # Launch QEMU guest +oiv-dmesg-follow # Live kernel logs +``` + +--- + +## 3. Build Workflow + +### Full Build +```bash +cd ~/Git/open-iov +make # Builds kernel + userspace + examples +# OR +oiv-build +``` + +### Incremental Builds +```bash +# Kernel only (open-iov-vfio.ko) +make kernel +# OR +cd kernel && make -j$(nproc) + +# Userspace only (libiov.so + libvgpu.so + example_vgpu) +make userspace +# OR +cd userspace && make -j$(nproc) + +# Examples only +make examples + +# Tests +make tests + +# Clean +make clean +# OR +oiv-clean +``` + +### Build Outputs +| Artifact | Location | Size | +|----------|----------|------| +| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB | +| `open-iov-vfio.ko` | `kernel/` | 2.8MB | +| `gsp-vgpu.ko` | `driver/` | 1.8MB | +| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB | +| `example_vgpu` | `userspace/examples/` | 27KB | + +--- + +## 4. Kernel Module Loading + +### Prerequisites +- **REBOOT REQUIRED** if kernel cmdline has `module_blacklist=nvidia,...` +- Check: `cat /proc/cmdline | grep module_blacklist` +- GRUB must be updated: `/etc/default/grub` → `update-grub` → `update-initramfs -u` → reboot + +### Load Order (Critical) +```bash +# 1. NVIDIA OpenRM host driver (exports vGPU VFIO symbols) +sudo insmod ~/Git/open-iov/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Open-IOV VFIO mdev driver (depends on nvidia.ko symbols) +sudo insmod ~/Git/open-iov/kernel/open-iov-vfio.ko + +# 3. Guest vGPU driver (standalone, works without above) +sudo insmod ~/Git/open-iov/driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin +``` + +### Using Helper Script +```bash +oiv-load # Runs all three in order +``` + +### Verify Load +```bash +lsmod | grep -E "nvidia|open_iov|gsp_vgpu" +dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov +ls -la /dev/gsp-vgpu /dev/nvidia* +``` + +### Unload +```bash +oiv-unload # Removes mdevs, unloads modules in reverse order +``` + +--- + +## 5. mdev Device Management + +### Create mdev (for QEMU) +```bash +oiv-mdev-create +# Interactive: selects GPU, vGPU type, generates UUID +# Creates: /sys/bus/mdev/devices/<UUID>/ +``` + +### Manual Creation +```bash +# List GPUs +lspci -D -d 10de: | grep -E "VGA|3D" + +# List supported vGPU types +ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/ + +# Create specific type +echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create +``` + +### List mdevs +```bash +oiv-mdev-list +# OR +ls -la /sys/bus/mdev/devices/ +``` + +### Remove mdev +```bash +echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove +``` + +--- + +## 6. QEMU Guest Launch + +### Using Helper Script +```bash +oiv-qemu +# Interactive: selects mdev, guest image, launches QEMU +``` + +### Manual Launch +```bash +qemu-system-x86_64 \ + -enable-kvm \ + -machine q35,accel=kvm,kernel_irqchip=split \ + -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \ + -smp 8,sockets=1,cores=8,threads=1 \ + -m 16G,slots=4,maxmem=64G \ + -mem-prealloc \ + -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \ + -numa node,memdev=mem \ + -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \ + -vga none -display none \ + -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \ + -netdev user,id=net0,hostfwd=tcp::2222-:22 \ + -device virtio-net-pci,netdev=net0 \ + -usb -device usb-tablet \ + -name "vGPU-Guest" \ + -pidfile /tmp/qemu-vgpu.pid \ + -daemonize +``` + +### Connect to Guest +```bash +# SSH (port 2222 forwarded) +ssh -p 2222 user@localhost + +# VNC (if enabled in launch script) +vncviewer localhost:1 +``` + +--- + +## 7. Guest Driver Setup + +### Inside Guest VM +```bash +# Run setup script (installs NVIDIA driver, firmware, loads module) +sudo ~/Git/open-iov/scripts/setup_guest_driver.sh + +# Or manual: +# 1. Install NVIDIA driver (OpenRM 570.144) +# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/ +# 3. modprobe nvidia +# 4. nvidia-smi +``` + +### Verify Guest +```bash +nvidia-smi # Should show vGPU (T4/A100/etc.) +lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU) +dmesg -T | grep -i nvidia # Check probe success +``` + +--- + +## 8. Testing Workflow + +### Host Test Suite +```bash +oiv-test +# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification +``` + +### Guest CUDA Tests +```bash +# Inside guest: +cd /tmp +git clone https://github.com/NVIDIA/cuda-samples.git +cd cuda-samples/Samples/1_Utilities/vectorAdd +make +./vectorAdd +``` + +### Guest Graphics Tests +```bash +# Inside guest: +vulkaninfo +vkcube +glxgears +glmark2 +``` + +### Guest Video Tests +```bash +# Inside guest: +ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4 +ffmpeg -hwaccel cuda -i test.mp4 -f null - +``` + +### Windows Guest Tests +```powershell +# Inside Windows guest: +# 1. Install NVIDIA driver (570.144) +# 2. nvidia-smi.exe +# 3. dxdiag +# 4. Run CUDA-Z / GPU-Z +# 5. Run 3DMark / FurMark +``` + +--- + +## 9. Firmware Reverse Engineering Workflow + +### Analyze New Firmware +```bash +# 1. Download firmware from NVIDIA OpenRM release +# 2. Place in firmware dir +cp new-gsp.bin ~/Git/open-iov/firmware/570-GSP/gsp-XXX.fwimage.bin + +# 3. Run Ghidra headless analysis +cd ~/Git/open-iov/tools/ghidra +./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \ + -import ~/Git/open-iov/firmware/570-GSP/gsp-XXX.fwimage.bin \ + -processor "RISCV:RV64:default" \ + -postScript DecompileFunctions.java \ + -deleteProject + +# 4. Run firmware analyzer (auto-discovers addresses) +cd ~/Git/open-iov +python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json + +# 5. Compare with previous version +python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json + +# 6. Generate config if new version +python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml +``` + +### Ghidra Scripts (in tools/ghidra/) +| Script | Purpose | +|--------|---------| +| `DecompileFunctions.java` | Batch decompile all functions | +| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table | +| `ExtractStructLayouts.java` | Extract struct definitions from prologues | +| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching | + +### Key Firmware Addresses (570.144W) +| Component | Address | Description | +|-----------|---------|-------------| +| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands | +| Secondary Dispatcher | 0x04ED39DC | Additional routing | +| BAR Trap Handler | 0x04CB8188 | 40KB | +| WPR Handler 1 | 0x04F00B0C | 62KB | +| WPR Handler 2 | 0x04F00C78 | 62KB | +| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers | +| ALLOC Main | 0x057E94EC | 40KB allocator | +| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler | +| Entry Point | 0x04009000 | Bootstrap | + +--- + +## 10. Debugging Workflows + +### Guest Driver Probe Issues +```bash +# Host: Watch live logs +oiv-dmesg-follow + +# Guest: Check PCI config +lspci -vvv -d 10de: +cat /proc/bus/pci/devices + +# Guest: Check BAR mapping +lspci -v -d 10de: | grep -A 10 "Memory at" + +# Guest: Check MSI-X +lspci -vvv -d 10de: | grep -A 20 "MSI-X" +``` + +### GSP Emulation Debugging +```bash +# Check emulation thread +ps aux | grep gsp_emu + +# Check RPC processing +dmesg -T | grep -i "gsp_emu\|rpc" + +# Debugfs stats +cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats + +# Add trace to gsp_emulation.c: +trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))); +cat /sys/kernel/debug/tracing/trace +``` + +### QEMU with GDB (Guest Kernel Debugging) +```bash +# Launch QEMU with GDB server +qemu-system-x86_64 -s -S ... # -s = gdbserver on :1234, -S = freeze + +# In another terminal: +gdb ./vmlinux +(gdb) target remote :1234 +(gdb) break nvidia_probe +(gdb) continue +``` + +### NEW: Comprehensive Debug Session (One Command) +```bash +# Launch debug session (QEMU + GDB + guest monitor) +./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux + +# Or manually: +# Terminal 1: QEMU with GDB +./scripts/launch_qemu_vgpu.sh --gdb --freeze + +# Terminal 2: Guest init monitor +python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300 + +# Terminal 3: GDB +gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux +(gdb) target remote localhost:1234 +``` + +### NEW: Failure Analysis +```bash +# After a failed guest init, analyze logs +python3 tools/failure_analyzer.py \ + --guest-log /tmp/guest_serial.log \ + --host-trace /tmp/host_trace.log \ + --qemu-log /tmp/qemu.log \ + --output /tmp/failure_report.json + +# This classifies the failure and provides specific recommendations +``` + +### NEW: Benchmarking +```bash +# Run VM vs bare metal benchmarks +./tools/benchmark_infrastructure.sh all + +# Or specific categories +./tools/benchmark_infrastructure.sh cuda +./tools/benchmark_infrastructure.sh graphics +./tools/benchmark_infrastructure.sh pytorch +``` + +### Performance Profiling +```bash +# GSP RPC latency +# Add to gsp_emulation.c: +trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))) + +# View trace +cat /sys/kernel/debug/tracing/trace + +# CUDA kernel launch overhead (in guest) +nvprof --profile-from-start off ./vectorAdd + +# MSI-X injection latency +# Measure in gsp_emu_inject_completion_interrupt() +``` + +--- + +## 11. Git Workflow + +### Restore Point +```bash +oiv-restore # git checkout f9ed5d8 -- . && git clean -fd +``` + +### Branch for Refactor +```bash +cd ~/Git/open-iov +git checkout -b refactor/host-gsp-emulation +# Implement changes per docs/refactored-implementation-to-do.md +``` + +### Commit Changes +```bash +git add . +git commit -m "Description of changes" +``` + +### LFS for Large Files +```bash +git lfs ls-files # Shows tracked large files +git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o" +``` + +--- + +## 12. Common Issues & Solutions + +| Issue | Solution | +|-------|----------| +| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist | +| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps | +| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap | +| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory | +| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` | +| Migration fails | State mismatch - checksum all migrated structures | +| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table | +| Build fails | `make clean && make` or check kernel headers match running kernel | + +--- + +## 13. Extending the Shell Environment + +### Adding New Helpers +When you create new scripts or workflows, add them to `~/.zshrc`: + +```bash +# Example: Add new test command +alias oiv-test-cuda="~/Git/open-iov/scripts/test_cuda_workloads.sh" +alias oiv-test-gfx="~/Git/open-iov/scripts/test_graphics_workloads.sh" + +# Example: Add firmware analysis shortcut +alias oiv-fw-new="cd ~/Git/open-iov && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin" + +# Reload +source ~/.zshrc +``` + +### Document New Helpers +Update `oiv-help()` function in `~/.zshrc` to include new commands. + +--- + +## 14. Key Files Reference + +| File | Purpose | +|------|---------| +| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps | +| `AGENTS.md` | Agent brain - workflow, tools, architecture | +| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design | +| `docs/refactored-architecture.md` | Host-only emulation design | +| `docs/refactored-implementation-to-do.md` | Granular implementation TODO | +| `scripts/load_vgpu_stack.sh` | Load all kernel modules | +| `scripts/create_mdev.sh` | Create mdev device | +| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU | +| `scripts/test_vgpu_stack.sh` | Comprehensive test suite | +| `scripts/debug_guest_init.sh` | NEW: One-command debug launch | +| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation | +| `tools/guest_init_monitor.py` | NEW: Guest init monitor | +| `tools/failure_analyzer.py` | NEW: Failure analyzer | +| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner | +| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent | +| `kernel/gsp_emulation.c/h` | GSP emulation engine | +| `kernel/firmware_analyzer.c/h` | Firmware analyzer | +| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher | +| `kernel/architecture/architecture.c/h` | Hardware abstraction | +| `kernel/version/version.c/h` | Version adaptation | +| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests | +| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers | +| `kernel/gsp_rpc_integration.c` | Integration layer | +| `kernel/main.c` | Module entry, mdev registration | +| `kernel/bar_emulation.c` | BAR + MSI-X emulation | +| `userspace/lib/libvgpu.c` | Userspace API | +| `userspace/lib/rpc_emulator.c` | Emulation fallbacks | + +--- + +*Development Workflow Guide Version: 1.0* +*Repository: ~/Git/open-iov* +*Last Updated: $(date)*
\ No newline at end of file diff --git a/docs/PORTABLE_ARCHITECTURE.md b/docs/PORTABLE_ARCHITECTURE.md new file mode 100644 index 0000000..47d0297 --- /dev/null +++ b/docs/PORTABLE_ARCHITECTURE.md @@ -0,0 +1,742 @@ +# Open-IOV vGPU-IO Portable Architecture Design + +## Design Goals + +### 1. Driver Version Portability +- **Support NVIDIA OpenRM 550+** without code changes per version +- **Automatic version detection** at runtime +- **Backward/forward compatibility** for RPC interfaces +- **Graceful degradation** when features unavailable + +### 2. Hardware Portability +- **Support multiple GPU architectures**: Turing (TU102/TU104), Ampere (GA100/GA102), Hopper (GH100), Blackwell (GB202/GB203/GB100), future +- **Support multiple form factors**: Consumer (RTX 3090/4090/5090), Data Center (A100/H100/B200), DGX Spark, DGX Station +- **Automatic hardware detection** via PCI device IDs and firmware signatures +- **Capability-based feature enablement** + +### 3. Zero Guest Modification +- **Guest runs stock NVIDIA driver** from NVIDIA website +- **No Open-IOV code in guest** kernel or userspace +- **Standard VFIO/mdev interface** only + +--- + +## Core Architecture: Three-Layer Abstraction + +``` +╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ PORTABLE ARCHITECTURE LAYERS ║ +╠═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣ +║ ║ +║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ +║ │ LAYER 1: HARDWARE ABSTRACTION LAYER (HAL) │ ║ +║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ +║ │ │ GpuArchitecture (abstract base) │ │ ║ +║ │ │ ├── detect(pci_dev) → GpuArchitecture* │ │ ║ +║ │ │ ├── get_pci_device_ids() → vector<DeviceID> │ │ ║ +║ │ │ ├── get_bar_layout() → BarLayout │ │ ║ +║ │ │ ├── get_gsp_firmware_info() → FirmwareInfo │ │ ║ +║ │ │ ├── get_rpc_dispatcher_addresses() → RpcAddresses │ │ ║ +║ │ │ ├── get_capabilities() → GpuCapabilities │ │ ║ +║ │ │ ├── create_rpc_handler(rpc_type) → RpcHandler* │ │ ║ +║ │ │ └── create_bar_emulator() → BarEmulator* │ │ ║ +║ │ │ │ │ ║ +║ │ │ Concrete Implementations (per architecture): │ │ ║ +║ │ │ ├── TuringArchitecture (TU102, TU104, TU106, TU116, TU117) │ │ ║ +║ │ │ ├── AmpereArchitecture (GA100, GA102, GA103, GA104, GA106, GA107) │ │ ║ +║ │ │ ├── HopperArchitecture (GH100, GH200) │ │ ║ +║ │ │ ├── BlackwellArchitecture (GB202, GB203, GB205, GB206, GB100) │ │ ║ +║ │ │ └── FutureArchitecture (auto-generated from firmware analysis) │ │ ║ +║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ +║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ +║ │ LAYER 2: VERSION ADAPTATION LAYER (VAL) │ ║ +║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ +║ │ │ DriverVersion (semantic: major.minor.patch) │ │ ║ +║ │ │ ├── parse(version_string) → DriverVersion │ │ ║ +║ │ │ ├── is_compatible(min_version, max_version) → bool │ │ ║ +║ │ │ ├── get_rpc_interface_version() → RpcInterfaceVersion │ │ ║ +║ │ │ ├── get_firmware_signature() → string (SHA256 of firmware) │ │ ║ +║ │ │ └── get_feature_flags() → FeatureFlags │ │ ║ +║ │ │ │ │ ║ +║ │ │ RpcInterfaceVersion (per driver version): │ │ ║ +║ │ │ ├── get_rpc_type_count() → int │ │ ║ +║ │ │ ├── get_rpc_struct_layout(rpc_type) → StructLayout │ │ ║ +║ │ │ ├── get_subcommand_mapping() → SubcommandMap │ │ ║ +║ │ │ ├── get_error_codes() → ErrorCodeMap │ │ ║ +║ │ │ └── get_capability_bits() → CapabilityBits │ │ ║ +║ │ │ │ │ ║ +║ │ │ Version Registry (loaded at runtime): │ │ ║ +║ │ │ ├── 550.x → RpcInterfaceV1 │ │ ║ +║ │ │ ├── 560.x → RpcInterfaceV2 (added MIGRATE subcommands) │ │ ║ +║ │ │ ├── 570.x → RpcInterfaceV3 (added DEBUG, NOTIFY enhancements) │ │ ║ +║ │ │ ├── 580.x → RpcInterfaceV4 (confidential computing, TEE) │ │ ║ +║ │ │ └── auto-detect from firmware signature │ │ ║ +║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ +║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ +║ │ LAYER 3: FIRMWARE-DRIVEN RPC ENGINE (FDRE) │ ║ +║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ +║ │ │ FirmwareAnalyzer (runs at module load) │ │ ║ +║ │ │ ├── load_firmware(firmware_path) → FirmwareImage │ │ ║ +║ │ │ ├── find_rpc_dispatcher() → address │ │ ║ +║ │ │ ├── find_jump_table() → address │ │ ║ +║ │ │ ├── find_handler_addresses() → map<rpc_type, address> │ │ ║ +║ │ │ ├── extract_struct_layouts() → map<rpc_type, StructLayout> │ │ ║ +║ │ │ ├── compute_firmware_signature() → SHA256 │ │ ║ +║ │ │ └── validate_against_known_versions() → DriverVersion │ │ ║ +║ │ │ │ │ ║ +║ │ │ DynamicRpcDispatcher: │ │ ║ +║ │ │ ├── dispatch(rpc_msg) → uses discovered handler addresses │ │ ║ +║ │ │ ├── fallback_to_emulation(rpc_type) → if handler not found │ │ ║ +║ │ │ └── hot_patch_handler(rpc_type, new_address) → for live updates │ │ ║ +║ │ │ │ │ ║ +║ │ │ EmulationFallbackRegistry: │ │ ║ +║ │ │ ├── register_emulation(rpc_type, EmulationHandler*) │ │ ║ +║ │ │ ├── get_emulation(rpc_type) → EmulationHandler* │ │ ║ +║ │ │ └── supports_emulation(rpc_type) → bool │ │ ║ +║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ +║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ +``` + +--- + +## Configuration-Driven Design + +### 1. Hardware Configuration Files + +```yaml +# config/hardware/turing.yaml +architecture: "turing" +pci_device_ids: + - vendor: 0x10de + device: 0x1e07 # TU102 + name: "RTX 2080 Ti" + - vendor: 0x10de + device: 0x1e08 # TU104 + name: "RTX 2080 / T4" + - vendor: 0x10de + device: 0x1ed3 # TU104 mobile + name: "RTX 2080 Super Mobile" +bar_layout: + bar0_size: 16MB + bar1_size: 256MB + bar2_size: 32MB + bar3_size: 32MB + bar0_prefetchable: true + bar1_prefetchable: true +gsp: + firmware_base: 0x04000000 + doorbell_offset: 0x110C00 + shared_mem_offset: 0x1000 + shared_mem_size: 4KB + rpc_dispatcher_primary: 0x057A1AC0 + rpc_dispatcher_secondary: 0x04ED39DC + rpc_jump_table: 0x059BFEB0 + handlers: + alloc: 0x057E94EC + schedule: 0x051EFBF4 + bar_trap: 0x04CB8188 + wpr_1: 0x04F00B0C + wpr_2: 0x04F00C78 + migration_table_1: 0x04179ED8 + migration_table_2: 0x04179FBC +capabilities: + max_vfs: 7 + supports_mig: true + supports_wpr: true + supports_bar_trap: true + supports_scheduling: true + supports_notifications: true + msi_x_vectors: 32 +``` + +```yaml +# config/hardware/ampere.yaml +architecture: "ampere" +pci_device_ids: + - vendor: 0x10de + device: 0x20b0 # GA100 + name: "A100" + - vendor: 0x10de + device: 0x2204 # GA102 + name: "RTX 3090" + - vendor: 0x10de + device: 0x2206 # GA102 + name: "RTX 3080" +bar_layout: + bar0_size: 16MB + bar1_size: 512MB # Larger FB on Ampere + bar2_size: 64MB + bar3_size: 64MB +gsp: + firmware_base: 0x04000000 + doorbell_offset: 0x110C00 + shared_mem_offset: 0x1000 + # Addresses discovered at runtime via FirmwareAnalyzer +capabilities: + max_vfs: 7 + supports_mig: true + supports_mig_mig: true # MIG (Multi-Instance GPU) + supports_confidential: false + supports_wpr: true + supports_bar_trap: true + msi_x_vectors: 64 +``` + +```yaml +# config/hardware/blackwell.yaml +architecture: "blackwell" +pci_device_ids: + - vendor: 0x10de + device: 0x2480 # GB202 (RTX 5090) + name: "RTX 5090" + - vendor: 0x10de + device: 0x2380 # GB100 (B200) + name: "B200" + - vendor: 0x10de + device: 0x2384 # GB205 + name: "RTX 5070" +bar_layout: + bar0_size: 16MB + bar1_size: 1GB # Blackwell unified memory + bar2_size: 128MB + bar3_size: 128MB +gsp: + firmware_base: 0x04000000 + doorbell_offset: 0x110C00 + shared_mem_offset: 0x1000 +capabilities: + max_vfs: 8 + supports_mig: true + supports_mig_mig: true + supports_confidential: true # TEE support + supports_wpr: true + supports_bar_trap: true + supports_nvlink: true + msi_x_vectors: 128 +``` + +### 2. Driver Version Configuration + +```yaml +# config/versions/550.yaml +version: "550" +rpc_interface_version: 1 +rpc_types: + - ALLOC: 0x01 + - FREE: 0x02 + - MAP: 0x03 + - UNMAP: 0x04 + - PIN: 0x05 + - UNPIN: 0x06 + - SCHEDULE: 0x07 + - YIELD: 0x08 + - NOTIFY: 0x09 + - BAR_TRAP: 0x0A + - MIGRATE: 0x0B + - DEBUG: 0x0C +struct_layouts: + ALLOC: + size: 48 + fields: + - {name: size, offset: 0, type: u64} + - {name: alignment, offset: 8, type: u64} + - {name: mem_type, offset: 16, type: u32} + - {name: flags, offset: 20, type: u32} + - {name: addr_hint, offset: 24, type: u64} + - {name: pid, offset: 32, type: u32} +capabilities: + supports_migrate: true + migrate_subcommands: + - SAVE: 0x04 + - RESTORE: 0x05 + supports_notifications: basic +``` + +```yaml +# config/versions/570.yaml +version: "570" +rpc_interface_version: 3 +extends: "550" +rpc_types: + - NOTIFY: 0x09 # Enhanced with subcommands + - MIGRATE: 0x0B # Enhanced with 4-phase +struct_layouts: + NOTIFY: + size: 32 + fields: + - {name: type, offset: 0, type: u32} + - {name: subcmd, offset: 4, type: u32} + - {name: event_fd, offset: 8, type: u64} + - {name: flags, offset: 16, type: u32} +capabilities: + supports_notifications: enhanced + supports_migrate_4phase: true + migrate_phases: [QUERY, PREPARE, TRANSFER, COMMIT, CLEANUP] +``` + +```yaml +# config/versions/580.yaml +version: "580" +rpc_interface_version: 4 +extends: "570" +rpc_types: + - TEE_ATTEST: 0x0D # New in 580 for confidential computing + - TEE_SEAL: 0x0E + - TEE_UNSEAL: 0x0F +capabilities: + supports_confidential: true + supports_tee: true + tee_attestation_types: [CPU, GPU, MEMORY] +``` + +--- + +## Runtime Detection & Adaptation Flow + +``` +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ MODULE LOAD: RUNTIME DETECTION FLOW │ +├─────────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. open-iov-vfio.ko loads │ +│ │ │ +│ ▼ │ +│ 2. Probe PCI device │ +│ │ │ +│ ▼ │ +│ 3. Match PCI Device ID → Hardware Config (turing.yaml, ampere.yaml, etc.) │ +│ │ │ +│ ▼ │ +│ 4. Load Hardware Config → Instantiate GpuArchitecture subclass │ +│ │ │ +│ ▼ │ +│ 5. Load GSP Firmware (from /lib/firmware or host nvidia.ko) │ +│ │ │ +│ ▼ │ +│ 6. FirmwareAnalyzer.analyze(firmware) │ +│ ├── Compute firmware SHA256 │ +│ ├── Find RPC dispatcher via pattern matching │ +│ ├── Find jump table via cross-references │ +│ ├── Extract handler addresses for all 12+ RPC types │ +│ ├── Extract struct layouts via pattern matching │ +│ └── Match firmware signature → DriverVersion (550, 560, 570, 580, etc.) │ +│ │ │ +│ ▼ │ +│ 7. Load Version Config → Instantiate RpcInterfaceVersion │ +│ │ │ +│ ▼ │ +│ 8. Validate: Hardware Config + Version Config compatible? │ +│ │ │ +│ ├── YES → Initialize DynamicRpcDispatcher with discovered addresses │ +│ ├── PARTIAL → Enable emulation for missing handlers │ +│ └── NO → Fail with clear error: "Unsupported GPU/driver combination" │ +│ │ │ +│ ▼ │ +│ 9. Register EmulationFallbackRegistry for all RPC types │ +│ │ │ +│ ▼ │ +│ 10. Start GSP Emulation Thread (per mdev) │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## FirmwareAnalyzer - Automatic Address Discovery + +```c +// kernel/firmware_analyzer.c + +struct firmware_signature { + uint8_t sha256[32]; + char version_string[64]; + uint32_t crc32; +}; + +struct discovered_rpc_handlers { + uint64_t primary_dispatcher; + uint64_t secondary_dispatcher; + uint64_t jump_table; + struct { + uint32_t rpc_type; + uint64_t handler_address; + bool found; + } handlers[16]; // Support up to 16 RPC types + uint32_t handler_count; +}; + +struct firmware_analyzer { + const uint8_t *firmware; + size_t firmware_size; + uint64_t firmware_base; + struct discovered_rpc_handlers discovered; + struct firmware_signature signature; +}; + +int firmware_analyzer_init(struct firmware_analyzer *fa, const char *firmware_path) { + // Load firmware file + fa->firmware = load_firmware_file(firmware_path, &fa->firmware_size); + if (!fa->firmware) + return -ENOENT; + + // Compute signature + sha256(fa->firmware, fa->firmware_size, fa->signature.sha256); + fa->signature.crc32 = crc32(0, fa->firmware, fa->firmware_size); + + // Set base address (from hardware config) + fa->firmware_base = get_hardware_config()->gsp.firmware_base; + + return 0; +} + +int firmware_analyzer_discover(struct firmware_analyzer *fa) { + // 1. Find primary dispatcher: search for known pattern + // Pattern: "RPC_DISPATCH" string reference or known instruction sequence + fa->discovered.primary_dispatcher = find_pattern(fa->firmware, fa->firmware_size, + "RPC_DISPATCH", fa->firmware_base); + + // 2. Find jump table: cross-reference from dispatcher + fa->discovered.jump_table = find_jump_table(fa->firmware, fa->firmware_size, + fa->discovered.primary_dispatcher); + + // 3. Parse jump table entries (32 entries for 12 RPC types) + parse_jump_table(fa->firmware, fa->discovered.jump_table, &fa->discovered); + + // 4. Find secondary dispatcher + fa->discovered.secondary_dispatcher = find_secondary_dispatcher(fa->firmware, + fa->firmware_size, fa->discovered.primary_dispatcher); + + // 5. Find specialized handlers via string references + find_handler_by_string(fa, "BAR_TRAP", "BAR1", "ForceBarAccess", &fa->discovered.handlers[GSP_RPC_BAR_TRAP]); + find_handler_by_string(fa, "WPR", "completion", "timeout", &fa->discovered.handlers[GSP_RPC_NOTIFY]); + find_handler_by_string(fa, "ALLOC", "size", "alignment", &fa->discovered.handlers[GSP_RPC_ALLOC]); + find_handler_by_string(fa, "SCHEDULE", "work", "callback", &fa->discovered.handlers[GSP_RPC_SCHEDULE]); + find_handler_by_string(fa, "MIGRATE", "checkpoint", "progress", &fa->discovered.handlers[GSP_RPC_MIGRATE]); + + // 6. Extract struct layouts via pattern matching on handler prologues + extract_struct_layouts(fa); + + // 7. Match signature against known versions + match_firmware_version(fa); + + return 0; +} + +void firmware_analyzer_match_version(struct firmware_analyzer *fa) { + // Compare signature against known database + static const struct { + const char *version; + uint8_t sha256[32]; + uint32_t crc32; + } known_versions[] = { + {"550.163.01", {0x...}, 0x...}, + {"560.28.03", {0x...}, 0x...}, + {"570.144.00", {0x...}, 0x...}, + {"570.144.01", {0x...}, 0x...}, + {"580.00.00", {0x...}, 0x...}, + }; + + for (int i = 0; i < ARRAY_SIZE(known_versions); i++) { + if (memcmp(fa->signature.sha256, known_versions[i].sha256, 32) == 0) { + strncpy(fa->signature.version_string, known_versions[i].version, 63); + return; + } + } + + // Unknown version - use heuristic based on handler count and features + if (fa->discovered.handler_count >= 12) + strcpy(fa->signature.version_string, "570+ (unknown)"); + else if (fa->discovered.handler_count >= 10) + strcpy(fa->signature.version_string, "560+ (unknown)"); + else + strcpy(fa->signature.version_string, "550+ (unknown)"); +} +``` + +--- + +## Dynamic RPC Dispatcher + +```c +// kernel/dynamic_rpc_dispatcher.c + +struct dynamic_rpc_dispatcher { + struct discovered_rpc_handlers *discovered; + struct rpc_interface_version *interface; + struct emulation_fallback_registry *fallbacks; + spinlock_t lock; + uint64_t dispatch_stats[16]; +}; + +int dynamic_rpc_dispatcher_dispatch(struct dynamic_rpc_dispatcher *dr, + struct gsp_rpc_msg *req, + struct gsp_rpc_response *rsp) { + // 1. Validate RPC type + if (req->type >= dr->interface->rpc_type_count) + return -ENOTTY; + + // 2. Check if we have a native handler address + uint64_t handler_addr = 0; + for (int i = 0; i < dr->discovered->handler_count; i++) { + if (dr->discovered->handlers[i].rpc_type == req->type) { + handler_addr = dr->discovered->handlers[i].handler_address; + break; + } + } + + // 3. Dispatch to native handler if available + if (handler_addr) { + return dynamic_rpc_call_native(dr, handler_addr, req, rsp); + } + + // 4. Fallback to emulation + struct emulation_handler *emu = emulation_fallback_get(dr->fallbacks, req->type); + if (emu) { + dr->dispatch_stats[req->type]++; // Track emulation usage + return emu->handle(req, rsp); + } + + // 5. No handler, no emulation + rsp->status = -ENOTTY; + return -ENOTTY; +} + +int dynamic_rpc_call_native(struct dynamic_rpc_dispatcher *dr, + uint64_t handler_addr, + struct gsp_rpc_msg *req, + struct gsp_rpc_response *rsp) { + // Use firmware-decompiled calling convention + // This calls the actual GSP firmware handler via the discovered address + // In emulation mode, we call our C reimplementation instead + + if (gsp_emulation_mode_enabled()) { + // Call our C reimplementation (from gsp_rpc_client.c) + return gsp_rpc_client_call(req->type, req, rsp); + } else { + // Native mode: would call into GSP firmware + // Not used in our architecture (we always emulate in host) + return -ENOTSUPP; + } +} +``` + +--- + +## Capability Negotiation with Guest + +``` +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ GUEST DRIVER CAPABILITY NEGOTIATION │ +├─────────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Guest nvidia.ko probes vGPU: │ +│ 1. Reads PCI Config Space → gets device ID, BAR sizes, capabilities │ +│ 2. Maps BAR0 → finds GSP doorbell + shared memory │ +│ 3. Sends RPC: QUERY_CAPABILITIES (new RPC type or subcommand) │ +│ │ +│ Host DynamicRpcDispatcher receives QUERY_CAPABILITIES: │ +│ 1. Constructs response from: │ +│ - Hardware Config capabilities │ +│ - Version Config capabilities │ +│ - EmulationFallbackRegistry (what we can emulate) │ +│ 2. Returns capability bitmap to guest │ +│ │ +│ Guest nvidia.ko enables features based on response: │ +│ - If MIG supported → enables MIG code paths │ +│ - If WPR supported → enables WPR code paths │ +│ - If CONFIDENTIAL supported → enables TEE code paths │ +│ - If NOTIFY enhanced → uses async eventfd │ +│ │ +│ This allows SAME guest driver to work across ALL hardware/versions! │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Update Strategy for New Driver Versions + +### 1. Adding New Driver Version (e.g., 590) + +```bash +# 1. Add version config +cp config/versions/580.yaml config/versions/590.yaml +# Edit: version, rpc_interface_version, new RPC types, capabilities + +# 2. Test with new firmware +# FirmwareAnalyzer auto-discovers addresses - NO CODE CHANGES NEEDED + +# 3. If new RPC types added: +# - Add to config/versions/590.yaml +# - Add emulation handler in kernel/emulation_fallbacks.c +# - Register in EmulationFallbackRegistry + +# 4. If struct layouts changed: +# - FirmwareAnalyzer extracts new layouts automatically +# - Version config specifies layout version +``` + +### 2. Adding New GPU Architecture (e.g., Rubin R100) + +```bash +# 1. Add hardware config +cp config/hardware/blackwell.yaml config/hardware/rubin.yaml +# Edit: PCI device IDs, BAR layout, capabilities + +# 2. Add GpuArchitecture subclass (if special handling needed) +# kernel/arch/rubin_architecture.c + +# 3. Register in architecture factory +# kernel/architecture_factory.c + +# 4. Test - FirmwareAnalyzer handles address discovery +``` + +### 3. Automated Firmware Analysis Pipeline + +```yaml +# .github/workflows/firmware-analysis.yml +name: Firmware Analysis +on: + schedule: + - cron: '0 0 * * 0' # Weekly + workflow_dispatch: + inputs: + firmware_url: + description: 'NVIDIA firmware download URL' + required: true + +jobs: + analyze: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Download firmware + run: wget -O firmware.bin "${{ inputs.firmware_url }}" + - name: Run FirmwareAnalyzer + run: | + ./tools/firmware_analyzer firmware.bin > analysis.json + - name: Compare with known versions + run: ./tools/compare_analysis.py analysis.json + - name: Generate config if new + run: | + if [ "$(cat analysis.json | jq -r '.is_new')" = "true" ]; then + ./tools/generate_config.py analysis.json > config/versions/NEW.yaml + git add config/versions/NEW.yaml + git commit -m "Add support for new firmware version" + git push + fi +``` + +--- + +## Testing Matrix for Portability + +| GPU Architecture | PCI Device IDs | Driver Versions | Test Status | +|------------------|----------------|-----------------|-------------| +| **Turing** | 10de:1e07, 10de:1e08, 10de:1ed3 | 550, 560, 570, 580 | ✅ Baseline | +| **Ampere** | 10de:20b0, 10de:2204, 10de:2206 | 550, 560, 570, 580 | 🔄 In Progress | +| **Hopper** | 10de:2330, 10de:2334 | 560, 570, 580 | 🔄 Planned | +| **Blackwell** | 10de:2480, 10de:2380, 10de:2384 | 570, 580 | 🔄 Planned | +| **Rubin (Future)** | TBD | 590+ | 📋 Designed | + +### Per-Combination Tests + +```bash +# Automated test matrix +for arch in turing ampere hopper blackwell; do + for version in 550 560 570 580; do + if [ -f "config/hardware/${arch}.yaml" ] && [ -f "config/versions/${version}.yaml" ]; then + echo "Testing ${arch} + ${version}..." + ./scripts/test_combination.sh ${arch} ${version} + fi + done +done +``` + +--- + +## Implementation File Structure (Portable) + +``` +open-iov/ +├── config/ +│ ├── hardware/ +│ │ ├── turing.yaml +│ │ ├── ampere.yaml +│ │ ├── hopper.yaml +│ │ ├── blackwell.yaml +│ │ └── rubin.yaml (future) +│ ├── versions/ +│ │ ├── 550.yaml +│ │ ├── 560.yaml +│ │ ├── 570.yaml +│ │ ├── 580.yaml +│ │ └── 590.yaml (future) +│ └── defaults.yaml +├── kernel/ +│ ├── architecture/ +│ │ ├── architecture.h # Abstract base class +│ │ ├── architecture_factory.c # Detection & instantiation +│ │ ├── turing_architecture.c +│ │ ├── ampere_architecture.c +│ │ ├── hopper_architecture.c +│ │ ├── blackwell_architecture.c +│ │ └── generic_architecture.c # Fallback +│ ├── version/ +│ │ ├── version.h # DriverVersion, RpcInterfaceVersion +│ │ ├── version_registry.c # Loads version configs +│ │ ├── interface_v1.c # 550 interface +│ │ ├── interface_v2.c # 560 interface +│ │ ├── interface_v3.c # 570 interface +│ │ └── interface_v4.c # 580 interface +│ ├── firmware/ +│ │ ├── firmware_analyzer.c # Auto-discovers addresses +│ │ ├── firmware_analyzer.h +│ │ ├── pattern_matcher.c # Binary pattern matching +│ │ └── struct_layout_extractor.c +│ ├── dispatch/ +│ │ ├── dynamic_rpc_dispatcher.c +│ │ ├── dynamic_rpc_dispatcher.h +│ │ ├── emulation_fallback_registry.c +│ │ └── emulation_fallback_registry.h +│ ├── gsp_emulation.c # Uses dynamic dispatcher +│ ├── main.c # Integrates all layers +│ └── Makefile +├── tools/ +│ ├── firmware_analyzer/ # Standalone analysis tool +│ ├── compare_analysis.py +│ ├── generate_config.py +│ └── test_combination.sh +└── docs/ + ├── PORTABLE_ARCHITECTURE.md + └── VERSION_COMPATIBILITY.md +``` + +--- + +## Key Design Principles Summary + +| Principle | Implementation | +|-----------|----------------| +| **No hardcoded addresses** | FirmwareAnalyzer discovers at runtime | +| **No hardcoded struct layouts** | Version config + FirmwareAnalyzer extraction | +| **No hardcoded device IDs** | Hardware config YAML files | +| **No hardcoded capabilities** | Capability negotiation via QUERY_CAPABILITIES RPC | +| **Graceful degradation** | EmulationFallbackRegistry for missing handlers | +| **Forward compatibility** | Version config `extends` mechanism | +| **Automated updates** | Firmware analysis pipeline + config generation | +| **Test matrix** | Automated CI across architecture × version combinations | + +--- + +*Document Version: 1.0* +*Repository: $OPEN_IOV_ROOT* +*Branch: refactor/host-gsp-emulation* +*Base Commit: f9ed5d8*
\ No newline at end of file diff --git a/docs/ARCHITECTURE.md b/docs/old/ARCHITECTURE.md index f9b7ae4..f9b7ae4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/old/ARCHITECTURE.md diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md new file mode 100644 index 0000000..fe17248 --- /dev/null +++ b/docs/refactored-architecture.md @@ -0,0 +1,595 @@ +# Open-IOV vGPU-IO Refactored Architecture + +## System Design from First Principles + +### 1. Problem Statement + +**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with full GPU functionality (CUDA, graphics, video encode/decode, AI/ML) on Turing GPUs (TU102/TU104) without requiring: +- Proprietary GRID host drivers +- Modified guest drivers +- Paravirtualization interfaces +- Guest-side Open-IOV code + +**Constraints**: +- Guest runs stock `nvidia.ko` + `libcuda.so` + `libnvidia-ml.so` + `libnvidia-opencl.so` +- Host runs OpenRM `nvidia.ko` (for physical GPU management) + Open-IOV hypervisor components +- No guest kernel modifications, no guest userspace Open-IOV libraries +- High performance: near-native GPU utilization for compute and graphics workloads +- Support both Linux and Windows guests + +### 2. Core Design Principle: Host-Side GSP Emulation + +The fundamental insight: **The GSP (GPU System Processor) is a RISC-V microcontroller running firmware that implements 12 RPC types. We can emulate this firmware entirely in the host kernel.** + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ FIRST PRINCIPLE: GSP IS JUST AN RPC SERVER │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Physical Reality: │ +│ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │ +│ │ Guest RM │ ─────────────────────────────► │ GSP Firmware │ │ +│ │ (nvidia.ko)│ ◄───────────────────────────── │ (570.144W, RV64IMAC) │ │ +│ └─────────────┘ Responses └─────────────────────────┘ │ +│ │ │ │ +│ │ 12 RPC Types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, │ │ +│ │ SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG │ │ +│ │ │ │ +│ ▼ │ │ +│ Virtualized Reality: │ │ +│ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │ +│ │ Guest RM │ ─────────────────────────────► │ Host Kernel │ │ +│ │ (nvidia.ko)│ ◄───────────────────────────── │ (open-iov-vfio.ko) │ │ +│ └─────────────┘ Responses │ gsp_rpc_client.c │ │ +│ │ │ (12 emulated handlers)│ │ +│ │ Guest sees identical └─────────────────────────┘ │ +│ │ interface - cannot tell │ +│ ▼ │ +│ No guest-side code required. │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +### 3. System Architecture Overview + +``` +╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ HOST SYSTEM ║ +║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ USERSPACE ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ QEMU/KVM + libvfio-user │ ║ ║ +║ ║ │ • -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID> │ ║ ║ +║ ║ │ • VFIO region mmap, ioctl, eventfd for IRQs │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ mdev management CLI / mdev management daemon (Management Tools) │ ║ ║ +║ ║ │ • Link against: libvgpu.so (host-side) │ ║ ║ +║ ║ │ • VF lifecycle: create/destroy, resource allocation, migration orchestration │ ║ ║ +║ ║ │ • Calls: vgpu_open(), vgpu_vf_create(), vgpu_mem_alloc(), vgpu_bar_config(), │ ║ ║ +║ ║ │ vgpu_wpr_protect(), vgpu_schedule(), vgpu_migrate_start(), etc. │ ║ ║ +║ ║ │ • Communicates via VFIO ioctls on /dev/vfio/<mdev> │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ libvgpu.so / libiov.so (HOST-SIDE ONLY) │ ║ ║ +║ ║ │ • Full vGPU management API: vgpu_*() functions │ ║ ║ +║ ║ │ • Implements: VF create/destroy, memory ops, BAR config, WPR, scheduling, │ ║ ║ +║ ║ │ notifications, migration │ ║ ║ +║ ║ │ • rpc_emulator.c: Host-side emulation fallbacks for management operations │ ║ ║ +║ ║ │ (rpc_emu_bar_config/save/restore, rpc_emu_wpr_*, rpc_emu_schedule/yield, │ ║ ║ +║ ║ │ rpc_emu_notify_register/unregister/send, rpc_emu_migrate_*) │ ║ ║ +║ ║ │ • NO GSP RPC emulation here - that lives in kernel │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +║ ║ +║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ KERNEL SPACE ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ nvidia.ko (Host RM - OpenRM 570.144) │ ║ ║ +║ ║ │ • Boots physical GSP firmware (570.144W) │ ║ ║ +║ ║ │ • Manages physical GPU resources (VRAM, engines, display) │ ║ ║ +║ ║ │ • Exports vGPU VFIO symbols: │ ║ ║ +║ ║ │ - nvidia_vgpu_vfio_get_ops() / nvidia_vgpu_vfio_set_ops() │ ║ ║ +║ ║ │ - nv_vgpu_mdev_register_parent() / nv_vgpu_mdev_unregister_parent() │ ║ ║ +║ ║ │ • GPC callbacks from open-iov-vfio.ko: vgpu_create, vgpu_delete, vgpu_bar_info, │ ║ ║ +║ ║ │ vgpu_start, get_name, get_description, get_instances, get_sparse_mmap, update_request │ ║ ║ +║ ║ │ • Owns physical MSI-X interrupt │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ open-iov-vfio.ko (HYPERVISOR - Core Component) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ mdev Parent Registration │ │ ║ ║ +║ ║ │ │ • Registers with nvidia.ko via nv_vgpu_mdev_register_parent() │ │ ║ ║ +║ ║ │ │ • Creates mediated device types: nvidia-vgpu-turing, nvidia-vgpu-ampere, etc. │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ PCI Config Space Emulation (256 bytes per mdev) │ │ ║ ║ +║ ║ │ │ • Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU), Class=0x030000 │ │ ║ ║ +║ ║ │ │ • BAR0=16MB (64-bit, prefetchable), BAR1=256MB (64-bit, prefetchable) │ │ ║ ║ +║ ║ │ │ • BAR2=32MB, BAR3=32MB (64-bit) │ │ ║ ║ +║ ║ │ │ • Capabilities: PM (0x40), MSI (0x50), MSI-X (0x70), PCIe (0x80) │ │ ║ ║ +║ ║ │ │ • MSI-X Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ │ ║ ║ +║ ║ │ │ • VFIO_CONFIG_REGION_INDEX mapped to this emulated config space │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ BAR Region Emulation (VFIO Region Indices) │ │ ║ ║ +║ ║ │ │ • VFIO_PCI_BAR0_REGION_INDEX → 16MB emulated config space │ │ ║ ║ +║ ║ │ │ • VFIO_PCI_BAR1_REGION_INDEX → 256MB emulated framebuffer (backed by host VRAM allocation) │ │ ║ ║ +║ ║ │ │ • VFIO_PCI_BAR2_REGION_INDEX → 32MB reserved │ │ ║ ║ +║ ║ │ │ • VFIO_PCI_BAR3_REGION_INDEX → 32MB reserved │ │ ║ ║ +║ ║ │ │ • VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table (0x2000) + PBA (0x3000) in BAR0 │ │ ║ ║ +║ ║ │ │ • BAR trap handling: 64 trapped regions covering all BAR0 registers (bar_emulation.c) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ GSP EMULATION ENGINE (NEW - Core Innovation) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ Per-mdev GSP Emulation Thread: │ │ ║ ║ +║ ║ │ │ ```c │ │ ║ ║ +║ ║ │ │ static int gsp_emu_thread(void *data) { │ │ ║ ║ +║ ║ │ │ struct nv_vgpu_device *vgpu = data; │ │ ║ ║ +║ ║ │ │ while (!kthread_should_stop()) { │ │ ║ ║ +║ ║ │ │ u32 doorbell = readl(vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); │ │ ║ ║ +║ ║ │ │ if (doorbell) { │ │ ║ ║ +║ ║ │ │ writel(0, vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); // ACK │ │ ║ ║ +║ ║ │ │ gsp_emu_process_rpc(vgpu); // Process via gsp_rpc_client.c │ │ ║ ║ +║ ║ │ │ } │ │ ║ ║ +║ ║ │ │ usleep_range(100, 200); // 1-2μs polling interval │ │ ║ ║ +║ ║ │ │ } │ │ ║ ║ +║ ║ │ │ return 0; │ │ ║ ║ +║ ║ │ │ } │ │ ║ ║ +║ ║ │ │ ``` │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ GSP Shared Memory Management: │ │ ║ ║ +║ ║ │ │ • Allocates contiguous host memory for GSP mailbox (4KB aligned) │ │ ║ ║ +�b ║ │ │ • Maps into guest BAR0 at GSP_SHARED_MEM_OFFSET (0x1000) │ │ ║ ║ +║ ║ │ │ • WC (Write-Combining) mapping for performance │ │ ║ ║ +║ ║ │ │ • Cache coherency: clflush on write, memory barriers │ │ ║ ║ +║ ║ │ │ • Matches firmware mailbox layout exactly (from decompilation) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ RPC Handler Suite (gsp_rpc_client.c - ALL 12 TYPES): │ │ ║ ║ +║ ║ │ │ ┌─────────────┬──────────────────────────────────────────────────────────────────────────────────┐ │ │ ║ +║ ║ │ │ │ RPC Type │ Emulated Implementation │ │ │ ║ +║ ║ │ │ ├─────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ │ ║ +║ ║ │ │ │ ALLOC (0x01)│ VRAM allocation via host nvidia.ko, returns GPU VA + handle │ │ │ ║ +║ ║ │ │ │ FREE (0x02) │ Free allocation, unmap pages │ │ │ ║ +║ ║ │ │ │ MAP (0x03) │ Map allocation to GPU VA, update page tables │ │ │ ║ +║ ║ │ │ │ UNMAP (0x04)│ Unmap GPU VA range │ │ │ ║ +║ ║ │ │ │ PIN (0x05) │ Pin pages for DMA, return DMA address │ │ │ ║ +║ ║ │ │ │ UNPIN (0x06)│ Unpin pages │ │ │ ║ +║ ║ │ │ │ SCHEDULE(0x07)│ Submit work to GSP scheduler (local workqueue) │ │ │ ║ +║ ║ │ │ │ YIELD (0x08)│ Timeslice yield (no-op success) │ │ │ ║ +║ ║ │ │ │ NOTIFY(0x09)│ Async notification registration, eventfd signaling │ │ │ ║ +║ ║ │ │ │ BAR_TRAP(0x0A)│ BAR register read/write emulation via shadow memory │ │ │ ║ +║ ║ │ │ │ MIGRATE(0x0B)│ 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP) │ │ │ ║ +║ ║ │ │ │ DEBUG (0x0C)│ Debug interface, firmware introspection │ │ │ ║ +║ ║ │ │ └─────────────┴──────────────────────────────────────────────────────────────────────────────────┘ │ │ ║ +║ ║ │ │ │ │ ║ +║ ║ │ │ Firmware Address Mapping (from decompilation): │ │ ║ +║ ║ │ │ • Primary Dispatcher: 0x057A1AC0 (32 sub-commands) │ │ ║ +║ ║ │ │ • Secondary Dispatcher: 0x04ED39DC │ │ ║ +║ ║ │ │ • BAR Trap Handler: 0x04CB8188 │ │ ║ +║ ║ │ │ • WPR Handlers: 0x04F00B0C, 0x04F00C78 │ │ ║ +║ ║ │ │ • ALLOC Main: 0x057E94EC │ │ ║ +║ ║ │ │ • SCHEDULE Main: 0x051EFBF4 │ │ ║ +║ ║ │ │ • Migration Tables: 0x04179ED8, 0x04179FBC (16 entries each) │ │ ║ +║ ║ │ │ • RPC Jump Table: 0x059BFEB0 (12 handlers) │ │ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ MSI-X Interrupt Injection │ │ ║ ║ +║ ║ │ │ • When emulated GSP completes RPC → write to MSI-X PBA → eventfd_signal() │ │ ║ ║ +║ ║ │ │ • QEMU/KVM delivers MSI-X interrupt to guest nvidia.ko ISR │ │ ║ ║ +║ ║ │ │ • Guest sees normal GSP completion interrupt │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ Memory Management (memory.c) │ │ ║ ║ +║ ║ │ │ • DMA mapping via IOMMU (VFIO DMA API) │ │ ║ ║ +║ ║ │ │ • Pin/unpin for device DMA │ │ ║ ║ +║ ║ │ │ • VRAM allocation via host nvidia.ko │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ Live Migration (migration.c) │ │ ║ ║ +║ ║ │ │ • 4-phase: QUERY → PREPARE → TRANSFER → COMMIT → CLEANUP │ │ ║ ║ +║ ║ │ │ • State save/restore: GSP context, page tables, VRAM contents, BAR state │ │ ║ ║ +║ ║ │ │ • Dual jump tables from firmware (0x04179ED8, 0x04179FBC) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +║ ║ +║ PHYSICAL GPU: TU102/TU104 (RTX 2080 Ti / T4 / RTX 2080 Super) ║ +║ • REAL GSP Firmware 570.144W running on FALCON RISC-V ║ +║ • Used by host nvidia.ko for physical GPU management ║ +║ • NOT used for guest vGPU RPCs (those are emulated in host kernel) ║ +║ ║ +╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ + │ VFIO MEDIATED DEVICE (mdev) │ + │ /sys/bus/mdev/devices/<UUID>/ │ + ▼ + +╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ GUEST VM ║ +║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ USERSPACE ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ Guest Applications │ ║ ║ +║ ║ │ • CUDA Applications (cuBLAS, cuDNN, custom kernels) │ ║ ║ +║ ║ │ • Graphics: OpenGL, Vulkan, DirectX (via WINE/Proton on Linux, native on Windows) │ ║ ║ +║ ║ │ • Video: NVENC/NVDEC (FFmpeg, GStreamer, Media Foundation) │ ║ ║ +║ ║ │ • AI/ML: TensorRT, PyTorch, TensorFlow │ ║ ║ +║ ║ │ • NO Open-IOV libraries, NO modified code │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ Standard NVIDIA Userspace Drivers (UNMODIFIED) │ ║ ║ +║ ║ │ • libcuda.so / libcuda.so.1 (CUDA Driver API) │ ║ ║ +║ ║ │ • libnvidia-ml.so / libnvidia-ml.so.1 (NVML) │ ║ ║ +║ ║ │ • libnvidia-opencl.so / libnvidia-opencl.so.1 (OpenCL) │ ║ ║ +║ ║ │ • libnvidia-vulkan.so / libGLX_nvidia.so (Vulkan/OpenGL) │ ║ ║ +║ ║ │ • libnvidia-encode.so / libnvidia-decode.so (Video) │ ║ ║ +║ ║ │ • nvidia-smi, nvidia-settings, nvidia-debugdump │ ║ ║ +║ ║ │ • ALL communicate with guest nvidia.ko via /dev/nvidia* ioctls │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +║ ║ +║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ KERNEL SPACE ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ nvidia.ko (Guest RM - UNMODIFIED OpenRM 570.144) │ ║ ║ +║ ║ │ • PCI Probe: detects 10de:1ed3 (TU104 vGPU) from emulated config space │ ║ ║ +║ ║ │ • BAR Mapping: │ ║ ║ +║ ║ │ - BAR0 (16MB): Config registers, GSP doorbell @ 0x110c00, shared mem @ 0x1000 │ ║ ║ +║ ║ │ - BAR1 (256MB): Framebuffer (VRAM) │ ║ ║ +║ ║ │ - BAR2 (32MB), BAR3 (32MB): Reserved │ ║ ║ +║ ║ │ • MSI-X Setup: Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ ║ ║ +║ ║ │ • GSP Client Initialization: │ ║ ║ +║ ║ │ - Writes to doorbell (queue head) to submit RPCs │ ║ ║ +║ ║ │ - Reads/writes shared memory for RPC messages │ ║ ║ +║ ║ │ - Waits for MSI-X interrupt on completion │ ║ ║ +║ ║ │ • RM Subsystems (all functional): │ ║ ║ +║ ║ │ - Memory: alloc, free, map, unmap, pin, unpin via GSP RPC │ ║ ║ +║ ║ │ - Scheduling: work submission, yield, priority via GSP RPC │ ║ ║ +║ ║ │ - BAR: register access via BAR_TRAP RPC │ ║ ║ +║ ║ │ - WPR: protect/unprotect via GSP RPC │ ║ ║ +║ ║ │ - Migration: save/restore via MIGRATE RPC │ ║ ║ +║ ║ │ - Notifications: async events via NOTIFY RPC │ ║ ║ +║ ║ │ • NO Open-IOV code in guest kernel │ ║ ║ +║ ║ │ • NO gsp-vgpu.ko, NO libvgpu.so in guest │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ vfio-pci.ko (Guest VFIO Core) │ ║ ║ +║ ║ │ • mmap of BAR regions → host VFIO regions │ ║ ║ +║ ║ │ • MSI-X interrupt handling → eventfd from host │ ║ ║ +║ ║ │ • Config space read/write → host emulated config space │ ║ ║ +║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +║ ║ +║ VIRTUAL GPU (mdev as seen by Guest): ║ +║ PCI Config (256B) │ BAR0(16MB) │ BAR1(256MB) │ BAR2(32MB) │ BAR3(32MB) │ MSI-X Table/PBA ║ +║ Vendor=10DE │ Config │ Framebuffer │ Reserved │ Reserved │ Table@0x2000 ║ +║ Device=1ED3 │ Doorbell │ (VRAM) │ │ │ PBA@0x3000 ║ +║ Class=030000 │ Queue Heads│ │ │ │ ║ +║ BARs: 64-bit │ Shared Mem │ │ │ │ ║ +║ Caps: PM,MSI, │ @0x110c00 │ │ │ │ ║ +║ MSI-X, PCIe │ @0x1000 │ │ │ │ ║ +╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ +``` + +### 4. Communication Pathways (Complete) + +| # | Pathway | Direction | Mechanism | Latency Target | +|---|---------|-----------|-----------|----------------| +| 1 | **GSP RPC (Guest→Host)** | Guest nvidia.ko → Host open-iov-vfio.ko | Doorbell write (MMIO) → host polling thread → gsp_rpc_client.c | < 5μs | +| 2 | **GSP Response (Host→Guest)** | Host open-iov-vfio.ko → Guest nvidia.ko | Write shared memory → MSI-X interrupt injection (eventfd) | < 3μs | +| 3 | **BAR Register Access** | Guest nvidia.ko → Host open-iov-vfio.ko | MMIO page fault → bar_emulation.c shadow memory | < 2μs | +| 4 | **PCI Config Space** | Guest kernel → Host open-iov-vfio.ko | VFIO_CONFIG_REGION_INDEX ioctl → emulated 256B config | < 1μs | +| 5 | **VFIO DMA (VRAM)** | Guest nvidia.ko ↔ Physical VRAM | IOMMU DMA mapping via VFIO DMA API | Near-native | +| 6 | **MSI-X Interrupts** | Physical GPU → Guest nvidia.ko | Host nvidia.ko ISR → nv_vgpu_vfio_inject_interrupt → eventfd → QEMU → Guest MSI-X | < 10μs | +| 7 | **Management Ops** | mdev management CLI/mgr → Host open-iov-vfio.ko | VFIO ioctls on mdev device → libvgpu.so → rpc_emulator.c | < 100μs | +| 8 | **Migration** | Host ↔ Host | Migration protocol over network, state via open-iov-vfio.ko | Seconds | + +### 5. GSP Emulation Engine - Detailed Design + +#### 5.1 Mailbox Layout (from Firmware Decompilation) + +``` +GSP Shared Memory Mailbox (at BAR0 + 0x1000, 4KB): +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Offset │ Size │ Field │ Description │ +├────────┼───────┼──────────────────────────┼─────────────────────────────────┤ +│ 0x0000 │ 64B │ RPC Request Header │ type, subcmd, seq_id, payload │ +│ 0x0040 │ 64B │ RPC Request Payload[6] │ u64 payload[6] per RPC type │ +│ 0x0080 │ 64B │ RPC Response Header │ status, type, seq_id │ +│ 0x00C0 │ 64B │ RPC Response Payload[6] │ u64 result[6] │ +│ 0x0100 │ 3KB │ Reserved / Extended │ Future use, migration state │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +#### 5.2 Doorbell Register (at BAR0 + 0x110C00) + +``` +Queue Head Register (0x110C00): +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Bits │ Field │ Description │ +├───────┼────────────────┼───────────────────────────────────────────────────┤ +│ 31:1 │ Reserved │ Must be zero │ +│ 0 │ Doorbell Valid │ 1 = New RPC request in mailbox, 0 = Idle │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +#### 5.3 Per-mdev GSP Emulation Thread + +```c +// In open-iov-vfio.ko - new file: gsp_emulation.c + +struct gsp_emu_context { + struct nv_vgpu_device *vgpu; + struct task_struct *thread; + void __iomem *bar0_vaddr; + void __iomem *doorbell_reg; + void __iomem *shared_mem; + struct gsp_client *rpc_client; // From gsp_rpc_client.c + spinlock_t lock; + bool running; + u64 processed_rpcs; + u64 failed_rpcs; +}; + +static int gsp_emu_thread_fn(void *data) { + struct gsp_emu_context *ctx = data; + + while (!kthread_should_stop()) { + // Read doorbell (atomic, no lock needed for single reader) + u32 doorbell = readl(ctx->doorbell_reg); + + if (doorbell & 0x1) { + // ACK immediately to prevent guest retry + writel(0, ctx->doorbell_reg); + + // Process RPC - this is the hot path + int ret = gsp_emu_process_rpc(ctx); + if (ret) + ctx->failed_rpcs++; + else + ctx->processed_rpcs++; + } + + // Adaptive polling: busy-wait for first 100μs, then sleep + if (ctx->processed_rpcs % 1000 == 0) + usleep_range(50, 100); + else + cpu_relax(); + } + return 0; +} + +static int gsp_emu_process_rpc(struct gsp_emu_context *ctx) { + // Copy request from shared memory (guest → host) + struct gsp_rpc_msg req; + memcpy_fromio(&req, ctx->shared_mem, sizeof(req)); + + // Validate sequence ID + if (req.seq_id == 0) + return -EINVAL; + + // Dispatch to firmware-decompiled handler + struct gsp_rpc_response rsp = {0}; + rsp.seq_id = req.seq_id; + rsp.type = req.type; + + int ret = 0; + switch (req.type) { + case GSP_RPC_ALLOC: + ret = gsp_rpc_alloc(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2], + req.payload[3], req.payload[4], req.payload[5], + &rsp.result[0], &rsp.result[1], &rsp.result[2]); + break; + case GSP_RPC_FREE: + ret = gsp_rpc_free(ctx->rpc_client, req.payload[0]); + break; + case GSP_RPC_MAP: + ret = gsp_rpc_map(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2], + req.payload[3], req.payload[4], req.payload[5], + &rsp.result[0]); + break; + case GSP_RPC_UNMAP: + ret = gsp_rpc_unmap(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2]); + break; + case GSP_RPC_PIN: + ret = gsp_rpc_pin(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2], + req.payload[3], &rsp.result[0], &rsp.result[1]); + break; + case GSP_RPC_UNPIN: + ret = gsp_rpc_unpin(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2]); + break; + case GSP_RPC_SCHEDULE: + ret = gsp_rpc_schedule(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2], + req.payload[3], req.payload[4]); + break; + case GSP_RPC_YIELD: + ret = gsp_rpc_yield(ctx->rpc_client); + break; + case GSP_RPC_NOTIFY: + ret = gsp_rpc_notify_register(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2]); + break; + case GSP_RPC_BAR_TRAP: + ret = gsp_rpc_bar_trap(ctx->rpc_client, + req.payload[0], req.payload[1], req.payload[2], + req.payload[3], req.payload[4], &rsp.result[0]); + break; + case GSP_RPC_MIGRATE: + // Handle migration sub-commands + ret = gsp_emu_handle_migrate(ctx, &req, &rsp); + break; + case GSP_RPC_DEBUG: + ret = gsp_emu_handle_debug(ctx, &req, &rsp); + break; + default: + rsp.status = -ENOTTY; + ret = -ENOTTY; + } + + // Write response to shared memory (host → guest) + rsp.status = ret ? -EIO : 0; + memcpy_toio(ctx->shared_mem + 0x80, &rsp, sizeof(rsp)); + + // Inject MSI-X completion interrupt + gsp_emu_inject_completion_interrupt(ctx, req.seq_id); + + return ret; +} +``` + +#### 5.4 MSI-X Completion Interrupt Injection + +```c +static void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id) { + // Write to MSI-X PBA (Pending Bit Array) at BAR0 + 0x3000 + // Vector 0 = GSP completion interrupt + void __iomem *pba = ctx->bar0_vaddr + 0x3000; + writel(0x1, pba); // Set pending bit for vector 0 + + // Trigger eventfd for VFIO interrupt injection + if (ctx->vgpu->msix_trigger[0]) + eventfd_signal(ctx->vgpu->msix_trigger[0], 1); +} +``` + +### 6. Performance Requirements + +| Metric | Target | Measurement Method | +|--------|--------|-------------------| +| GSP RPC round-trip (emulated) | < 10μs | ktime_get_ns() in gsp_emu_thread | +| BAR register access | < 2μs | Guest rdtsc around MMIO | +| VRAM allocation (ALLOC) | < 50μs | Guest nvidia.ko tracing | +| VRAM map (MAP) | < 20μs | Guest nvidia.ko tracing | +| DMA pin (PIN) | < 30μs | Guest nvidia.ko tracing | +| CUDA kernel launch overhead | < 5% vs native | CUDA events, nvprof | +| Graphics frame time (Vulkan) | < 2ms @ 60fps | PresentMon, vk_perf | +| Migration downtime | < 100ms | Stopwatch, ping during migration | +| MSI-X injection latency | < 5μs | Guest ISR timestamp vs host | + +### 7. Testing Strategy - Incremental Validation + +#### Phase 0: Host Infrastructure (Week 1) +- [ ] `open-iov-vfio.ko` loads, registers mdev parent with `nvidia.ko` +- [ ] mdev creation via sysfs succeeds +- [ ] PCI config space presented correctly (lspci in guest shows 10de:1ed3) +- [ ] BAR regions mmap'able in guest +- [ ] MSI-X table/PBA accessible in guest BAR0 + +#### Phase 1: GSP Emulation Core (Week 2) +- [ ] GSP emulation thread starts per mdev +- [ ] Doorbell polling detects guest writes +- [ ] ALLOC RPC: guest allocates VRAM, gets valid GPU VA + handle +- [ ] FREE RPC: guest frees allocation, no leaks +- [ ] MAP/UNMAP RPC: guest maps/unmaps GPU VA ranges +- [ ] PIN/UNPIN RPC: guest pins pages, gets valid DMA addresses + +#### Phase 2: Advanced RPCs (Week 3) +- [ ] SCHEDULE/YIELD: guest submits work, gets completion +- [ ] BAR_TRAP: guest reads/writes BAR registers, gets emulated values +- [ ] NOTIFY: guest registers for async events, receives them +- [ ] MIGRATE: 4-phase migration saves/restores state + +#### Phase 3: Guest Driver Integration (Week 4) +- [ ] Guest `nvidia.ko` loads, probes vGPU successfully +- [ ] `nvidia-smi` shows vGPU (T4 / A100 equivalent) +- [ ] CUDA context creation succeeds +- [ ] Simple CUDA kernel (vectorAdd) executes correctly +- [ ] `nvidia-ml` (NVML) queries work (temperature, utilization, memory) + +#### Phase 4: Compute Workloads (Week 5) +- [ ] cuBLAS SGEMM/DGEMM passes numerical verification +- [ ] cuDNN convolution forward/backward passes +- [ ] PyTorch/TensorFlow training step completes +- [ ] Multi-GPU (NCCL) across vGPUs works +- [ ] CUDA Graphs capture/replay works + +#### Phase 5: Graphics Workloads (Week 6) +- [ ] Vulkan: `vkCreateDevice` succeeds, render triangle +- [ ] OpenGL: `glxinfo` shows renderer, `glxgears` runs +- [ ] Video encode: FFmpeg + NVENC H.264/HEVC +- [ ] Video decode: FFmpeg + NVDEC H.264/HEVC +- [ ] Windows guest: D3D11/D3D12 device creation, render loop + +#### Phase 6: Stress & Reliability (Week 7) +- [ ] 24-hour CUDA stress test (memtest, compute) +- [ ] 24-hour graphics stress (Unigine, FurMark equivalent) +- [ ] Hot-plug VF create/destroy cycles (1000x) +- [ ] Migration under load (live migrate running CUDA kernel) +- [ ] Host reboot recovery (mdev persists, guest recovers) + +#### Phase 7: Windows Guest (Week 8) +- [ ] Windows 10/11 guest installs NVIDIA driver +- [ ] Device Manager shows "NVIDIA T4" (or equivalent) +- [ ] CUDA-Z / GPU-Z detect vGPU correctly +- [ ] DirectX 11/12: D3D11CreateDevice, D3D12CreateDevice +- [ ] Vulkan on Windows: vkCreateInstance, render +- [ ] Video encode/decode via Media Foundation + +### 8. Implementation File Map + +``` +open-iov/ +├── kernel/ +│ ├── gsp_emulation.c # NEW: GSP emulation thread + RPC processing +│ ├── gsp_emulation.h # NEW: Structures, prototypes +│ ├── gsp_rpc_client.c # EXISTING: Firmware-decompiled handlers +│ ├── gsp_rpc_integration.c # EXISTING: Integration layer +│ ├── bar_emulation.c # EXISTING: BAR trap handling +│ ├── memory.c # EXISTING: DMA, VRAM management +│ ├── irq.c # EXISTING: MSI-X injection +│ ├── migration.c # EXISTING: Live migration +│ ├── main.c # MODIFY: Add gsp_emu_context to nv_vgpu_device +│ └── Makefile # MODIFY: Add gsp_emulation.o +├── userspace/lib/ +│ ├── libvgpu.c # EXISTING: Management API +│ ├── rpc_emulator.c # EXISTING: Host-side emulation fallbacks +│ └── Makefile # UNCHANGED +├── scripts/ +│ ├── test_gsp_emu.sh # NEW: Automated GSP emulation tests +│ ├── test_guest_driver.sh # NEW: Guest driver validation +│ ├── test_cuda_workloads.sh # NEW: CUDA workload validation +│ ├── test_graphics_workloads.sh # NEW: Graphics validation +│ └── test_windows_guest.sh # NEW: Windows guest validation +└── docs/ + ├── refactored-architecture.md # THIS DOCUMENT + └── refactored-implementation-to-do.md # TODO LIST +``` + +### 9. Risk Mitigation + +| Risk | Likelihood | Impact | Mitigation | +|------|------------|--------|------------| +| GSP mailbox layout mismatch | Medium | High | Validate against firmware decompilation; add runtime checksum | +| MSI-X injection race | Medium | High | Use atomic PBA updates; test with concurrent RPCs | +| Cache coherency (WC memory) | High | Medium | clflush + mfence on every shared mem write; test on AMD/Intel | +| Guest driver version skew | Medium | Medium | Test against multiple OpenRM versions (550, 560, 570) | +| Windows guest driver issues | Medium | High | Separate Windows test CI; use WHQL driver | +| Performance regression vs native | Low | High | Continuous benchmarking; profile hot paths | +| Migration state corruption | Low | High | Checksum all migrated state; dry-run validation | + +--- + +*Document Version: 1.0 (Refactored Architecture)* +*Author: Open-IOV Team* +*Date: $(date)* +*Repository: $OPEN_IOV_ROOT* +*Git Commit: f9ed5d8 (restore point)*
\ No newline at end of file diff --git a/docs/tools/README.md b/docs/tools/README.md new file mode 100644 index 0000000..1c7c0b5 --- /dev/null +++ b/docs/tools/README.md @@ -0,0 +1,75 @@ +# Open-IOV Tools Documentation + +## Overview +This directory contains documentation for userspace tools built on top of the Open-IOV vGPU-IO stack, leveraging recycled NVIDIA RM API code from `integ/gpu_drv/ext-tree/`. + +## Tools + +### Debugging & Observability (Implemented) +| Tool | Location | Description | +|------|----------|-------------| +| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output | +| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations | +| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner | +| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) | +| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) | +| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM | + +### RM API Tooling (Planned) +| Tool | Documentation | Description | +|------|---------------|-------------| +| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control | +| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard | + +## Code Reuse Sources +All planned tools recycle existing NVIDIA driver code from `integ/gpu_drv/ext-tree/`: + +| Component | Path | Use Case | +|-----------|------|----------| +| nvidia-smi | `drivers/gpgpu/nvidia-smi/` | CLI framework, GPU enumeration, NV2080 ioctls | +| gpumon/perfmon | `drivers/gpgpu/gpumon_perfmon/` | Performance sampling, perfmon counters | +| NV2080 controls | `drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) | +| Resource Server | `drivers/gpgpu/resserve/` | Resource Server (resserve) API | +| CUDA Internal | `drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer | +| Resource Manager | `drivers/gpgpu/resman/` | Resman API | + +## Build Integration +Tools will be built under `open-iov/tools/`: +``` +tools/ +├── rmtool/ +│ ├── src/ +│ ├── lib/ +│ └── Makefile +├── rmtop/ +│ ├── src/ +│ ├── lib/ +│ └── Makefile +└── Makefile # Builds all tools +``` + +## Dependencies +- `libnvidia-rmapi.so` - Recycled RM API (from ext-tree) +- `libgpumon.so` - Recycled perfmon (from ext-tree) +- `libpciaccess` - PCI configuration access +- `libnuma` - NUMA topology +- `ncursesw` - rmtop UI (wide character support) +- `libjson-c` - JSON output formatting +- `libcurl` - Remote API (future) + +## Integration with Open-IOV Stack +Tools will integrate with the host VFIO driver via: +- `/dev/nvidia*` character devices (existing nvidia.ko) +- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`) +- VFIO ioctls for guest state introspection +- GSP-RM telemetry via firmware analyzer / dynamic dispatcher + +## Future Tool Ideas +| Tool | Purpose | +|------|---------| +| `rmmigrate` | Live migration coordinator via resserve | +| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) | +| `rmpolicy` | Policy engine for automated engine allocation | +| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) | +| `rmguest` | Guest introspection via vGPU VFIO interface | +| `rmfirmware` | GSP firmware analysis / update utility |
\ No newline at end of file diff --git a/docs/tools/rmtool.md b/docs/tools/rmtool.md new file mode 100644 index 0000000..e15306b --- /dev/null +++ b/docs/tools/rmtool.md @@ -0,0 +1,149 @@ +# rmtool - RM API Swiss Army Knife + +## Overview +`rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built by recycling existing code from the NVIDIA driver stack (`integ/gpu_drv/ext-tree/`). + +## Source Code Reuse +| Component | Source Location | Purpose | +|-----------|-----------------|---------| +| nvidia-smi core | `ext-tree/drivers/gpgpu/nvidia-smi/` | CLI framework, NV2080 ioctls, GPU enumeration | +| gpumon/perfmon | `ext-tree/drivers/gpgpu/gpumon_perfmon/` | Performance monitoring, sampling infrastructure | +| NV2080 controls | `ext-tree/drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) | +| Resource Server | `ext-tree/drivers/gpgpu/resserve/` | Resource Server (resserve) API | +| CUDA Internal | `ext-tree/drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer | +| Resource Manager | `ext-tree/drivers/gpgpu/resman/` | Resman API | + +## Core Commands + +### GPU Enumeration & Topology +```bash +rmtool gpu list # List all GPUs with PCI info +rmtool gpu topology # Show GPU topology (NVLink, PCIe, PCI IDs) +rmtool gpu info -d 0000:65:00.0 # Detailed GPU info (like nvidia-smi -q) +``` + +### Engine Management +```bash +rmtool engine list # List all engines (GR, CE, NVDEC, NVENC, NVJPG, OFA) +rmtool engine info -d 0 -e gr # Engine details (registers, capabilities) +rmtool engine allocate -d 0 -e gr -c 4 # Allocate 4 GR channels +rmtool engine start -d 0 -e gr -c 0 # Start engine channel +rmtool engine stop -d 0 -e gr -c 0 # Stop engine channel +rmtool engine config -d 0 -e nvenc --preset=h264 --bitrate=10M +``` + +### Channel & Context Control +```bash +rmtool channel list -d 0 # List all channels on GPU +rmtool channel create -d 0 -e gr -t compute # Create compute channel +rmtool channel submit -d 0 -c 0 --pushbuf=work.bin # Submit work +rmtool channel status -d 0 -c 0 # Channel status (busy, pending, error) +rmtool context list -d 0 # List all contexts +rmtool context info -d 0 -c 0 # Context details (page tables, memory) +``` + +### Memory Operations +```bash +rmtool mem alloc -d 0 --size=1G --type=video # Allocate VRAM +rmtool mem alloc -d 0 --size=512M --type=system # Allocate sysmem +rmtool mem map -d 0 -h 0x1234 --gpu-va=0x100000 # Map to GPU VA +rmtool mem unmap -d 0 -h 0x1234 # Unmap +rmtool mem pin -d 0 -h 0x1234 # Pin for DMA +rmtool mem info -d 0 -h 0x1234 # Allocation details +rmtool mem stats -d 0 # VRAM/sysmem usage breakdown +``` + +### Performance Sampling +```bash +rmtool sample engine -d 0 -e gr --interval=100ms --count=100 +rmtool sample clocks -d 0 --interval=1s +rmtool sample power -d 0 --interval=1s +rmtool sample thermal -d 0 --interval=1s +rmtool sample pcie -d 0 --interval=1s +``` + +### NV2080 Direct Control +```bash +rmtool nv2080 class list -d 0 # List all NV2080 classes +rmtool nv2080 method call -d 0 -c 0x906D -m 0x100 --params=... +rmtool nv2080 event subscribe -d 0 -c 0x906D -e 0x100 +rmtool nv2080 event wait -d 0 -c 0x906D -e 0x100 --timeout=5000 +``` + +### Resource Server (resserve) Integration +```bash +rmtool resserve status # Resource Server status +rmtool resserve reservation list # List all reservations +rmtool resserve reservation create --gpu=0 --engine=gr --size=4 +rmtool resserve reservation release --id=123 +``` + +### CUDA Internal Layer (CUDASW) +```bash +rmtool cudasw context list # CUDA contexts +rmtool cudasw module list # Loaded CUDA modules +rmtool cudasw stream list -c 0 # Streams in context +rmtool cudasw event list -c 0 # Events in context +``` + +### Resource Manager (Resman) +```bash +rmtool resman client list # Resman clients +rmtool resman resource list -c 0 # Resources owned by client +rmtool resman heap info -d 0 # Heap state +``` + +### GSP-RM Telemetry +```bash +rmtool gsp rpc-stats -d 0 # RPC queue depths, latency +rmtool gsp doorbell -d 0 # Doorbell rates +rmtool gsp firmware -d 0 # Firmware version, threads +rmtool gsp wpr -d 0 # WPR state +``` + +## Output Formats +```bash +rmtool gpu list --format=json +rmtool engine list --format=csv +rmtool sample engine --format=prometheus +rmtool mem stats --format=influx +``` + +## Build +```bash +cd open-iov/tools/rmtool +make +# Links against: libnvidia-rmapi.so (from ext-tree), libpciaccess, libnuma +``` + +## Architecture +``` +rmtool/ +├── src/ +│ ├── main.c # CLI parsing, command dispatch +│ ├── gpu.c # GPU enumeration, topology +│ ├── engine.c # Engine allocation/control +│ ├── channel.c # Channel/context management +│ ├── memory.c # Memory alloc/map/pin +│ ├── sample.c # Performance sampling +│ ├── nv2080.c # NV2080 direct control +│ ├── resserve.c # Resource Server client +│ ├── cudasw.c # CUDA Internal Layer +│ ├── resman.c # Resource Manager +│ ├── gsp.c # GSP-RM telemetry +│ └── output.c # JSON/CSV/Prometheus formatters +├── include/ +│ └── rmtool.h # Internal headers +├── lib/ +│ ├── librmapi.a # Recycled RM API code +│ └── libgpumon.a # Recycled perfmon code +└── Makefile +``` + +## Future Extensions +- **Remote gRPC/REST API** for fleet management +- **Policy Engine** for automated engine allocation +- **Migration Coordination** via resserve +- **Telemetry Export** (Prometheus, OpenTelemetry, InfluxDB) +- **Guest Introspection** via vGPU VFIO interface +- **Workload Profiling** with CUDA/GR engine correlation
\ No newline at end of file diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md new file mode 100644 index 0000000..6408268 --- /dev/null +++ b/docs/tools/rmtop.md @@ -0,0 +1,227 @@ +# rmtop - GPU htop Equivalent + +## Overview +`rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry. + +## Source Code Reuse +Same recycled components as `rmtool`: +- `gpumon_perfmon` - sampling infrastructure, perfmon counters +- `nvidia-smi` - GPU enumeration, NVML-style queries +- `nv2080` - direct RM control for real-time state +- `resserve` - resource server state +- `resman` - resource manager heaps/clients + +## UI Layout + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ rmtop 1.0 │ GPU 0: RTX 5090 (GB202) │ Driver: 610.XX │ 14:32:15 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ ENGINES │ UTIL │ FREQ │ POWER │ TEMP │ ACTIVE CHANNELS │ +│ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │ 320W │ 78°C │ ch: 0,3,7,12 │ +│ ├─ CE0 (Copy) │ ████░░░░░░ 41% │ 2100MHz │ │ │ ch: 1,4 │ +│ ├─ CE1 (Copy) │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVDEC0 │ ██████████ 100% │ 1800MHz │ │ │ ch: 2 │ +│ ├─ NVDEC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVENC0 │ ████████░░ 82% │ 1800MHz │ │ │ ch: 5 │ +│ ├─ NVENC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVJPG │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ └─ OFA │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ MEMORY │ USED │ TOTAL │ FREE │ FRAG │ BAR1 │ +│ ├─ VRAM │ 18.2G │ 24.0G │ 5.8G │ 12% │ 256M / 256M │ +│ ├─ System RAM │ 4.1G │ 64.0G │ 59.9G │ 3% │ │ +│ └─ BAR1 (CPU) │ 12M │ 256M │ 244M │ 1% │ │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ CHANNELS / CONTEXTS (PID │ ENG │ CTX │ VRAM │ SYSMEM │ UTIL │ STATE) │ +│ ├─ 1247 │ GR │ 0x3 │ 2.1G │ 512M │ 82% │ RUNNING │ +│ ├─ 1247 │ CE0 │ 0x3 │ 128M │ 64M │ 41% │ RUNNING │ +│ ├─ 1247 │ DEC │ 0x3 │ 4.0G │ 256M │ 100% │ RUNNING │ +│ ├─ 1247 │ ENC │ 0x3 │ 1.5G │ 128M │ 82% │ RUNNING │ +│ ├─ 3891 │ GR │ 0x7 │ 512M │ 128M │ 12% │ WAITING │ +│ └─ 3891 │ CE0 │ 0x7 │ 64M │ 32M │ 5% │ WAITING │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ POWER / CLOCKS / THERMALS │ +│ ├─ Power Draw: 320W / 450W (71%) │ GPU Clock: 2850 MHz (max) │ +│ ├─ Mem Clock: 1313 MHz (max) │ SM Clock: 2850 MHz │ +│ ├─ Temp: 78°C (slowdown: 83°C) │ Fan: 65% (auto) │ +│ └─ PState: P0 (Performance) │ Perf Cap: None │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ GSP-RM TELEMETRY │ +│ ├─ RPC Queue: GR=12 CE=3 DEC=0 ENC=1 (depth / max=64) │ +│ ├─ Doorbell Rate: 45K/s │ Firmware Threads: 8/16 active │ +│ ├─ WPR Regions: 3 active │ Migration: IDLE │ +│ └─ Errors: 0 (CRC) 0 (Timeout) 0 (Page Fault) │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ PCIe / NVLink │ +│ ├─ Link: Gen4 x16 │ Throughput: 12.5 GB/s / 31.5 GB/s (40%) │ +│ ├─ Replay Timer: 0 │ NAK: 0 │ L0→L1: 12 L1→L0: 12 │ +│ └─ NVLink: N/A (single GPU) │ +└─────────────────────────────────────────────────────────────────────────────┘ + [F1] Help [F2] GPU Select [F3] Engine Detail [F4] Memory Detail [F5] Processes [F6] GSP [F10] Quit +``` + +## Key Bindings +| Key | Action | +|-----|--------| +| `↑/↓` | Navigate process list | +| `←/→` | Switch GPU (multi-GPU) | +| `F1` | Help | +| `F2` | GPU Select menu | +| `F3` | Engine detail view | +| `F4` | Memory detail view | +| `F5` | Process list (expanded) | +| `F6` | GSP-RM telemetry | +| `F7` | PCIe/NVLink detail | +| `F8` | Power/thermal graphs | +| `F9` | Configuration | +| `F10` / `q` | Quit | +| `Space` | Pause/resume updates | +| `s` | Change sample interval (100ms-10s) | +| `u` | Sort by GPU util / Memory / PID | +| `k` | Kill process (send SIGTERM to guest) | +| `e` | Export snapshot (JSON) | + +## Engine Detail View (F3) +``` +┌─ GR Engine Detail ────────────────────────────────────────────────────────┐ +│ Channel 0 (PID 1247, CTX 0x3) │ Util: 82% │ Submitted: 1.2M ops │ +│ ├─ Pushbuffer: 0x7f8a... │ Fence: 0x123456 │ Pending: 47 │ +│ ├─ Context: 0x3 │ Page Tables: 0x100000 │ VRAM: 2.1G │ +│ └─ Perf: SM=82% TEX=45% L2=67% ROPS=34% │ +│ Channel 3 (PID 1247, CTX 0x3) │ Util: 12% │ Submitted: 45K ops │ +│ └─ ... │ +│ Channel 7 (PID 3891, CTX 0x7) │ Util: 5% │ Submitted: 12K ops │ +│ └─ ... │ +│ Channel 12 (PID 3891, CTX 0x7) │ Util: 0% │ Submitted: 0 ops │ +│ └─ IDLE │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Memory Detail View (F4) +``` +┌─ Memory Detail ────────────────────────────────────────────────────────────┐ +│ VRAM Allocations (by PID) │ +│ ├─ 1247: 18.2G ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │ +│ ├─ 3891: 0.6G ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% │ +│ └─ System: 0.1G ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │ +│ │ +│ Fragmentation: 12% │ Largest Free Block: 2.1G │ Allocation Count: 347 │ +│ │ +│ BAR1 Mappings (CPU→GPU) │ +│ ├─ 0x7f8a00000000 → GPU VA 0x100000 (256M) RW PID 1247 │ +│ └─ 0x7f8b00000000 → GPU VA 0x200000 (128M) RO PID 3891 │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## GSP-RM Telemetry View (F6) +``` +┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐ +│ RPC Queues (depth / max=64) │ +│ ├─ GR: ████████████░░░░░░░░░░ 12/64 │ Avg Latency: 2.3μs │ +│ ├─ CE0: ███░░░░░░░░░░░░░░░░░░░░░ 3/64 │ Avg Latency: 1.1μs │ +│ ├─ CE1: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ +│ ├─ NVDEC: ██████████████████████ 64/64 │ Avg Latency: 4.7μs ⚠ SATURATED│ +│ ├─ NVENC: ████████████░░░░░░░░░░░░░ 1/64 │ Avg Latency: 3.2μs │ +│ └─ OFA: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ +│ │ +│ Doorbell Statistics │ +│ ├─ Total Rate: 45,231/s │ Peak: 120,451/s │ +│ ├─ GR: 28,123/s │ CE: 8,456/s │ DEC: 5,234/s │ ENC: 3,418/s │ +│ └─ Errors: 0 (spurious) 0 (invalid) │ +│ │ +│ Firmware Threads (16 total) │ +│ ├─ Active: 8 │ Idle: 6 │ Blocked: 2 │ +│ ├─ Thread 0: GR Scheduler │ CPU: 45% │ RPC: 12,341/s │ +│ ├─ Thread 1: CE Manager │ CPU: 12% │ RPC: 3,456/s │ +│ ├─ Thread 2: DEC Manager │ CPU: 78% │ RPC: 5,234/s ⚠ HIGH │ +│ └─ ... │ +│ │ +│ WPR State │ +│ ├─ Region 0: 0x100000-0x200000 (1M) READ-ONLY PID 1247 │ +│ ├─ Region 1: 0x300000-0x400000 (1M) NO-ACCESS PID 1247 │ +│ └─ Region 2: 0x500000-0x600000 (1M) READ-WRITE PID 3891 │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Multi-GPU Support +``` +rmtop -g 0 # Start on GPU 0 +rmtop -g all # Cycle through all GPUs (tab) +rmtop --layout=grid # Show all GPUs in grid (requires wide terminal) +``` + +## Configuration (`~/.config/rmtop/config.toml`) +```toml +[general] +sample_interval_ms = 500 +theme = "dark" # dark, light, mono +show_gsp = true +show_pcie = true +show_nvlink = false + +[colors] +engine_high = "red" +engine_medium = "yellow" +engine_low = "green" +warning = "red" +critical = "bright_red" + +[columns] +# Show/hide columns in process list +pid = true +name = true +engine = true +context = true +vram = true +sysmem = true +util = true +state = true + +[alerts] +gsp_queue_warning = 48 # 75% of max +gsp_queue_critical = 56 # 87.5% of max +temp_warning = 80 +temp_critical = 85 +power_warning = 400 +``` + +## Build +```bash +cd open-iov/tools/rmtop +make +# Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess +``` + +## Architecture +``` +rmtop/ +├── src/ +│ ├── main.c # ncurses init, main loop +│ ├── ui.c # Layout, rendering, input handling +│ ├── gpu_panel.c # GPU summary panel +│ ├── engine_panel.c # Engine utilization bars +│ ├── memory_panel.c # Memory breakdown +│ ├── process_panel.c # Process/channel list +│ ├── power_panel.c # Power/clocks/thermals +│ ├── gsp_panel.c # GSP-RM telemetry +│ ├── pcie_panel.c # PCIe/NVLink +│ ├── detail_views.c # F3-F8 detail panels +│ ├── sampler.c # Background sampling thread +│ └── config.c # Config file parsing +├── include/ +│ └── rmtop.h +├── lib/ +│ ├── librmapi.a +│ └── libgpumon.a +└── Makefile +``` + +## Future Extensions +- **Remote Dashboard**: WebSocket server for browser-based monitoring +- **Historical Graphs**: In-terminal sparklines for utilization history +- **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO +- **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads +- **Policy Visualization**: Show resserve reservations, quotas +- **Migration Tracking**: Live migration progress bars +- **Telemetry Export**: Background Prometheus/OpenTelemetry push
\ No newline at end of file |