diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
| commit | 681494a79e6c0e084354eec0229386a696a314c0 (patch) | |
| tree | f0d69e9583cd6ed752da7e40c8167df24e228a9a /docs/DEVELOPMENT_WORKFLOW.md | |
| parent | bfcb856e09207d8edb714373d775537165d06b05 (diff) | |
| download | open-iov-681494a79e6c0e084354eec0229386a696a314c0.tar.gz open-iov-681494a79e6c0e084354eec0229386a696a314c0.zip | |
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful
material in the structured tree: debugging workflows and the EPT-livelock
case study (05-operations/02), performance microbenchmarks and A/B
methodology (05-operations/01), the version-upgrade runbook
(05-operations/02 section 4, with corrected host/guest framing), kernel-code
pitfalls (07-developers) and the OpenRM build/load order
(03-architecture/06 section 7). The remaining legacy files were verified
redundant and removed.
* docs/README now records where each migrated topic lives.
* Portability: the config YAML files are called out as design targets that no
code reads yet, with what adding support actually requires today.
* Getting started: the OpenRM step is marked optional (native vGPU-management
workflows only; the emulated path uses the distro driver).
* tools docs: rmtool/rmtop marked as planned, pointing at the implemented
tools/rmq subset.
* wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the
canonical userspace/cuda analysis) and index update.
* scripts/check-docs-links.py plus a CI step so relative links cannot rot.
* STATE.md references to removed documents repointed.
Diffstat (limited to 'docs/DEVELOPMENT_WORKFLOW.md')
| -rw-r--r-- | docs/DEVELOPMENT_WORKFLOW.md | 681 |
1 files changed, 0 insertions, 681 deletions
diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md deleted file mode 100644 index 5d43896..0000000 --- a/docs/DEVELOPMENT_WORKFLOW.md +++ /dev/null @@ -1,681 +0,0 @@ -# Open-IOV vGPU-IO Development Workflow & Tools Guide - -> **Historical document — do not follow as written (2026-09-21).** This guide -> predates the current workflow: it contains the old nvidia-first load order, -> reboot advice, and LFS instructions that no longer apply (LFS was removed; -> firmware and decompiled artifacts are local/gitignored). The current -> workflow is -> [`../scripts/stack_manager.sh`](../scripts/stack_manager.sh), -> [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) -> and -> [`02-getting-started/01-install-and-first-run.md`](02-getting-started/01-install-and-first-run.md). - -## Overview - -This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging. - ---- - -## 1. Repository Structure & Navigation - -### Primary Workspace -```bash -$OPEN_IOV_ROOT/ # Main monorepo (git + LFS) -``` - -### Key Directories -| Directory | Purpose | -|-----------|---------| -| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) | -| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) | -| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` | -| `userspace/examples/` | `example_vgpu` test app | -| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files | -| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) | -| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) | -| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent | -| `docs/` | Architecture, implementation guides, TODO lists | -| `wiki/` | 10 technical wiki pages | - ---- - -## 2. Shell Integration (zshrc) - -### Required Aliases & Functions -Add to `~/.zshrc` and run `source ~/.zshrc`: - -```bash -# Project navigation -alias oiv="cd $OPEN_IOV_ROOT" -alias oivk="cd $OPEN_IOV_ROOT/kernel" -alias oivu="cd $OPEN_IOV_ROOT/userspace" -alias oivf="cd $OPEN_IOV_ROOT/firmware/57X-GSP/570.144" -alias oivd="cd $OPEN_IOV_ROOT/docs" -alias oivs="cd $OPEN_IOV_ROOT/scripts" -alias oivt="cd $OPEN_IOV_ROOT/tools" - -# Build commands -alias oiv-build="cd $OPEN_IOV_ROOT && make -j$(nproc)" -alias oiv-build-kernel="cd $OPEN_IOV_ROOT && make kernel" -alias oiv-build-userspace="cd $OPEN_IOV_ROOT && make userspace" -alias oiv-clean="cd $OPEN_IOV_ROOT && make clean" - -# Module loading (requires reboot first!) -alias oiv-load="$OPEN_IOV_ROOT/scripts/load_vgpu_stack.sh" -alias oiv-unload="$OPEN_IOV_ROOT/scripts/unload_vgpu_stack.sh" -alias oiv-test="$OPEN_IOV_ROOT/scripts/test_vgpu_stack.sh" -alias oiv-test-gsp="$OPEN_IOV_ROOT/scripts/test_gsp_emu.sh" -alias oiv-test-guest="$OPEN_IOV_ROOT/scripts/test_guest_driver.sh" -alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh" -alias oiv-test-graphics="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh" -alias oiv-test-windows="$OPEN_IOV_ROOT/scripts/test_windows_guest.sh" - -# mdev management -alias oiv-mdev-create="$OPEN_IOV_ROOT/scripts/create_mdev.sh" -alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'" - -# QEMU -alias oiv-qemu="$OPEN_IOV_ROOT/scripts/launch_qemu_vgpu.sh" -alias oiv-qemu-debug="$OPEN_IOV_ROOT/scripts/debug_guest_init.sh" - -# Firmware analysis -alias oiv-ghidra="cd $OPEN_IOV_ROOT/tools/ghidra && ./ghidraRun" -alias oiv-fw-analyze="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py" - -# Debugging tools (NEW) -alias oiv-guest-monitor="python3 $OPEN_IOV_ROOT/tools/guest_init_monitor.py" -alias oiv-failure-analyzer="python3 $OPEN_IOV_ROOT/tools/failure_analyzer.py" -alias oiv-benchmark="$OPEN_IOV_ROOT/tools/benchmark_infrastructure.sh" -alias oiv-windows-test="$OPEN_IOV_ROOT/scripts/windows_guest_automation.sh" - -# Kernel debugging -alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" -alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" -alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic" -alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu" - -# Git -alias oiv-git="cd $OPEN_IOV_ROOT && git status" -alias oiv-git-log="cd $OPEN_IOV_ROOT && git log --oneline -20" -alias oiv-git-diff="cd $OPEN_IOV_ROOT && git diff" - -# Restore point -alias oiv-restore="cd $OPEN_IOV_ROOT && git checkout f9ed5d8 -- . && git clean -fd" - -# Help -oiv-help() { - echo "╔═══════════════════════════════════════════════════════════════════════════════╗" - echo "║ Open-IOV vGPU-IO Development Commands ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ NAVIGATION ║" - echo "║ oiv - cd to open-iov root ║" - echo "║ oivk - cd to kernel/ ║" - echo "║ oivu - cd to userspace/ ║" - echo "║ oivf - cd to firmware/57X-GSP/570.144/ ║" - echo "║ oivd - cd to docs/ ║" - echo "║ oivs - cd to scripts/ ║" - echo "║ oivt - cd to tools/ ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ BUILD ║" - echo "║ oiv-build - Build everything (kernel + userspace + examples) ║" - echo "║ oiv-build-kernel - Build open-iov-vfio.ko only ║" - echo "║ oiv-build-userspace - Build libiov.so + example_vgpu ║" - echo "║ oiv-clean - Clean all build artifacts ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ MODULE OPERATIONS (require root) ║" - echo "║ oiv-load - Load nvidia.ko → open-iov-vfio.ko (refactored stack) ║" - echo "║ oiv-unload - Unload all modules + remove mdevs ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ TESTING ║" - echo "║ oiv-test - Run comprehensive test suite (test_vgpu_stack.sh) ║" - echo "║ oiv-test-gsp - Run GSP emulation unit tests (test_gsp_emu.sh) ║" - echo "║ oiv-test-guest - Test guest driver integration (test_guest_driver.sh)║" - echo "║ oiv-test-cuda - Test CUDA workloads (test_cuda_workloads.sh) ║" - echo "║ oiv-test-graphics - Test graphics workloads (test_graphics_workloads.sh)║" - echo "║ oiv-test-windows - Test Windows guest (test_windows_guest.sh) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ MDEV MANAGEMENT ║" - echo "║ oiv-mdev-create - Create mdev device for QEMU ║" - echo "║ oiv-mdev-list - List existing mdev devices ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ QEMU ║" - echo "║ oiv-qemu - Launch QEMU with vGPU ║" - echo "║ oiv-qemu-debug - Launch QEMU with GDB + guest monitor ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ DEBUGGING ║" - echo "║ oiv-dmesg - Filtered dmesg for NVIDIA/GSP/vGPU/mdev ║" - echo "║ oiv-dmesg-follow - Live filtered dmesg (tail -f) ║" - echo "║ oiv-dmesg-errors - Show only errors/warnings ║" - echo "║ oiv-trace - Live kernel trace_pipe filtered for gsp_emu ║" - echo "║ oiv-guest-monitor - Monitor guest init (correlates host traces + serial) ║" - echo "║ oiv-failure-analyzer - Analyze/classify guest init failures ║" - echo "║ oiv-benchmark - Run VM vs bare metal benchmarks ║" - echo "║ oiv-windows-test - Windows guest test automation (WinRM) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ GIT ║" - echo "║ oiv-git - Git status in open-iov ║" - echo "║ oiv-git-log - Git log (last 20 commits) ║" - echo "║ oiv-git-diff - Show uncommitted changes ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ FIRMWARE ANALYSIS ║" - echo "║ oiv-ghidra - Launch Ghidra GUI ║" - echo "║ oiv-fw-analyze - Run firmware analyzer (Python) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ EMERGENCY ║" - echo "║ oiv-restore - Restore to commit f9ed5d8 (restore point) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ oiv-help - This help ║" - echo "╚═══════════════════════════════════════════════════════════════════════════════╝" -} -``` - -### Usage -```bash -# After adding to ~/.zshrc: -source ~/.zshrc -oiv-help # Show all commands -oiv # Jump to project root -oiv-build # Build everything -oiv-load # Load kernel modules -oiv-mdev-create # Create mdev for QEMU -oiv-qemu # Launch QEMU guest -oiv-dmesg-follow # Live kernel logs -``` - ---- - -## 3. Build Workflow - -### Full Build -```bash -cd $OPEN_IOV_ROOT -make # Builds kernel + userspace + examples -# OR -oiv-build -``` - -### Incremental Builds -```bash -# Kernel only (open-iov-vfio.ko) -make kernel -# OR -cd kernel && make -j$(nproc) - -# Userspace only (libiov.so + libvgpu.so + example_vgpu) -make userspace -# OR -cd userspace && make -j$(nproc) - -# Examples only -make examples - -# Tests -make tests - -# Clean -make clean -# OR -oiv-clean -``` - -### Build Outputs -| Artifact | Location | Size | -|----------|----------|------| -| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB | -| `open-iov-vfio.ko` | `kernel/` | ~3.8MB | -| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB | -| `example_vgpu` | `userspace/examples/` | 27KB | - ---- - -## 4. Kernel Module Loading - -### Prerequisites -- The **stock Ubuntu nvidia driver must not be loaded** when you want our - OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on - desktop boots (udev) and is held open by the GUI. - `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this - machine) prevents that auto-load after reboot; `insmod` is unaffected. -- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia - (compares `/sys/module/nvidia/version` with our build) and refuses with - guidance instead of silently skipping. - -### Load Order (Critical) -```bash -# 1. VFIO/mdev prerequisites -sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev - -# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia -# imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the -# mdev parent on the physical GPU. -sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko - -# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols -# (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers) -sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko - -# 4. GSP firmware for the host GPU -sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ -``` - -### Using Helper Script -```bash -./scripts/stack_manager.sh load-stack # prereqs → open-iov-vfio → nvidia → firmware -``` - -### Verify Load -```bash -lsmod | grep -E "nvidia|open_iov" -dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/ -cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances -``` - -### Unload -```bash -oiv-unload # Removes mdevs, unloads modules in reverse order -``` - ---- - -## 5. mdev Device Management - -### Create mdev (for QEMU) -```bash -oiv-mdev-create -# Interactive: selects GPU, vGPU type, generates UUID -# Creates: /sys/bus/mdev/devices/<UUID>/ -``` - -### Manual Creation -```bash -# List GPUs -lspci -D -d 10de: | grep -E "VGA|3D" - -# List supported vGPU types -ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/ - -# Create specific type -echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create -``` - -### List mdevs -```bash -oiv-mdev-list -# OR -ls -la /sys/bus/mdev/devices/ -``` - -### Remove mdev -```bash -echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove -``` - ---- - -## 6. QEMU Guest Launch - -### Using Helper Script -```bash -oiv-qemu -# Interactive: selects mdev, guest image, launches QEMU -``` - -### Manual Launch -```bash -qemu-system-x86_64 \ - -enable-kvm \ - -machine q35,accel=kvm,kernel_irqchip=split \ - -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \ - -smp 8,sockets=1,cores=8,threads=1 \ - -m 16G,slots=4,maxmem=64G \ - -mem-prealloc \ - -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \ - -numa node,memdev=mem \ - -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \ - -vga none -display none \ - -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \ - -netdev user,id=net0,hostfwd=tcp::2222-:22 \ - -device virtio-net-pci,netdev=net0 \ - -usb -device usb-tablet \ - -name "vGPU-Guest" \ - -pidfile /tmp/qemu-vgpu.pid \ - -daemonize -``` - -### Connect to Guest -```bash -# SSH (port 2222 forwarded) -ssh -p 2222 user@localhost - -# VNC (if enabled in launch script) -vncviewer localhost:1 -``` - ---- - -## 7. Guest Driver Setup - -### Inside Guest VM -```bash -# Run setup script (installs NVIDIA driver, firmware, loads module) -sudo $OPEN_IOV_ROOT/scripts/setup_guest_driver.sh - -# Or manual: -# 1. Install NVIDIA driver (OpenRM 570.144) -# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/ -# 3. modprobe nvidia -# 4. nvidia-smi -``` - -### Verify Guest -```bash -nvidia-smi # Should show vGPU (T4/A100/etc.) -lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU) -dmesg -T | grep -i nvidia # Check probe success -``` - ---- - -## 8. Testing Workflow - -### Host Test Suite -```bash -oiv-test -# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification -``` - -### Guest CUDA Tests -```bash -# Inside guest: -cd /tmp -git clone https://github.com/NVIDIA/cuda-samples.git -cd cuda-samples/Samples/1_Utilities/vectorAdd -make -./vectorAdd -``` - -### Guest Graphics Tests -```bash -# Inside guest: -vulkaninfo -vkcube -glxgears -glmark2 -``` - -### Guest Video Tests -```bash -# Inside guest: -ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4 -ffmpeg -hwaccel cuda -i test.mp4 -f null - -``` - -### Windows Guest Tests -```powershell -# Inside Windows guest: -# 1. Install NVIDIA driver (570.144) -# 2. nvidia-smi.exe -# 3. dxdiag -# 4. Run CUDA-Z / GPU-Z -# 5. Run 3DMark / FurMark -``` - ---- - -## 9. Firmware Reverse Engineering Workflow - -### Analyze New Firmware -```bash -# 1. Download firmware from NVIDIA OpenRM release -# 2. Place in firmware dir -cp new-gsp.bin $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin - -# 3. Run Ghidra headless analysis -cd $OPEN_IOV_ROOT/tools/ghidra -./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \ - -import $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin \ - -processor "RISCV:RV64:default" \ - -postScript DecompileFunctions.java \ - -deleteProject - -# 4. Run firmware analyzer (auto-discovers addresses) -cd $OPEN_IOV_ROOT -python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json - -# 5. Compare with previous version -python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json - -# 6. Generate config if new version -python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml -``` - -### Ghidra Scripts (in tools/ghidra/) -| Script | Purpose | -|--------|---------| -| `DecompileFunctions.java` | Batch decompile all functions | -| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table | -| `ExtractStructLayouts.java` | Extract struct definitions from prologues | -| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching | - -### Key Firmware Addresses (570.144W) -| Component | Address | Description | -|-----------|---------|-------------| -| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands | -| Secondary Dispatcher | 0x04ED39DC | Additional routing | -| BAR Trap Handler | 0x04CB8188 | 40KB | -| WPR Handler 1 | 0x04F00B0C | 62KB | -| WPR Handler 2 | 0x04F00C78 | 62KB | -| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers | -| ALLOC Main | 0x057E94EC | 40KB allocator | -| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler | -| Entry Point | 0x04009000 | Bootstrap | - ---- - -## 10. Debugging Workflows - -### Guest Driver Probe Issues -```bash -# Host: Watch live logs -oiv-dmesg-follow - -# Guest: Check PCI config -lspci -vvv -d 10de: -cat /proc/bus/pci/devices - -# Guest: Check BAR mapping -lspci -v -d 10de: | grep -A 10 "Memory at" - -# Guest: Check MSI-X -lspci -vvv -d 10de: | grep -A 20 "MSI-X" -``` - -### GSP Emulation Debugging -```bash -# Check emulation thread -ps aux | grep gsp_emu - -# Check RPC processing -dmesg -T | grep -i "gsp_emu\|rpc" - -# Debugfs stats -cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats - -# Add trace to gsp_emulation.c: -trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))); -cat /sys/kernel/debug/tracing/trace -``` - -### QEMU with GDB (Guest Kernel Debugging) -```bash -# Launch QEMU with GDB server -qemu-system-x86_64 -s -S ... # -s = gdbserver on :1234, -S = freeze - -# In another terminal: -gdb ./vmlinux -(gdb) target remote :1234 -(gdb) break nvidia_probe -(gdb) continue -``` - -### NEW: Comprehensive Debug Session (One Command) -```bash -# Launch debug session (QEMU + GDB + guest monitor) -./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux - -# Or manually: -# Terminal 1: QEMU with GDB -./scripts/launch_qemu_vgpu.sh --gdb --freeze - -# Terminal 2: Guest init monitor -python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300 - -# Terminal 3: GDB -gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux -(gdb) target remote localhost:1234 -``` - -### NEW: Failure Analysis -```bash -# After a failed guest init, analyze logs -python3 tools/failure_analyzer.py \ - --guest-log /tmp/guest_serial.log \ - --host-trace /tmp/host_trace.log \ - --qemu-log /tmp/qemu.log \ - --output /tmp/failure_report.json - -# This classifies the failure and provides specific recommendations -``` - -### NEW: Benchmarking -```bash -# Run VM vs bare metal benchmarks -./tools/benchmark_infrastructure.sh all - -# Or specific categories -./tools/benchmark_infrastructure.sh cuda -./tools/benchmark_infrastructure.sh graphics -./tools/benchmark_infrastructure.sh pytorch -``` - -### Performance Profiling -```bash -# GSP RPC latency -# Add to gsp_emulation.c: -trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))) - -# View trace -cat /sys/kernel/debug/tracing/trace - -# CUDA kernel launch overhead (in guest) -nvprof --profile-from-start off ./vectorAdd - -# MSI-X injection latency -# Measure in gsp_emu_inject_completion_interrupt() -``` - ---- - -## 11. Git Workflow - -### Restore Point -```bash -oiv-restore # git checkout f9ed5d8 -- . && git clean -fd -``` - -### Branch for Refactor -```bash -cd $OPEN_IOV_ROOT -git checkout -b refactor/host-gsp-emulation -# Implement changes per docs/refactored-implementation-to-do.md -``` - -### Commit Changes -```bash -git add . -git commit -m "Description of changes" -``` - -### LFS for Large Files -```bash -git lfs ls-files # Shows tracked large files -git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o" -``` - ---- - -## 12. Common Issues & Solutions - -| Issue | Solution | -|-------|----------| -| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist | -| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps | -| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap | -| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory | -| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` | -| Migration fails | State mismatch - checksum all migrated structures | -| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table | -| Build fails | `make clean && make` or check kernel headers match running kernel | - ---- - -## 13. Extending the Shell Environment - -### Adding New Helpers -When you create new scripts or workflows, add them to `~/.zshrc`: - -```bash -# Example: Add new test command -alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh" -alias oiv-test-gfx="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh" - -# Example: Add firmware analysis shortcut -alias oiv-fw-new="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin" - -# Reload -source ~/.zshrc -``` - -### Document New Helpers -Update `oiv-help()` function in `~/.zshrc` to include new commands. - ---- - -## 14. Key Files Reference - -| File | Purpose | -|------|---------| -| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps | -| `AGENTS.md` | Agent brain - workflow, tools, architecture | -| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design | -| `docs/refactored-architecture.md` | Host-only emulation design | -| `docs/refactored-implementation-to-do.md` | Granular implementation TODO | -| `scripts/load_vgpu_stack.sh` | Load all kernel modules | -| `scripts/create_mdev.sh` | Create mdev device | -| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU | -| `scripts/test_vgpu_stack.sh` | Comprehensive test suite | -| `scripts/debug_guest_init.sh` | NEW: One-command debug launch | -| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation | -| `tools/guest_init_monitor.py` | NEW: Guest init monitor | -| `tools/failure_analyzer.py` | NEW: Failure analyzer | -| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner | -| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent | -| `kernel/gsp_emulation.c/h` | GSP emulation engine | -| `kernel/firmware_analyzer.c/h` | Firmware analyzer | -| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher | -| `kernel/architecture/architecture.c/h` | Hardware abstraction | -| `kernel/version/version.c/h` | Version adaptation | -| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests | -| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers | -| `kernel/gsp_rpc_integration.c` | Integration layer | -| `kernel/main.c` | Module entry, mdev registration | -| `kernel/bar_emulation.c` | BAR + MSI-X emulation | -| `userspace/lib/libvgpu.c` | Userspace API | -| `userspace/lib/rpc_emulator.c` | Emulation fallbacks | - ---- - -*Development Workflow Guide Version: 1.0* -*Repository: $OPEN_IOV_ROOT* -*Last Updated: $(date)*
\ No newline at end of file |
