path: root/docs/DEVELOPMENT_WORKFLOW.md

# Open-IOV vGPU-IO Development Workflow & Tools Guide

> **Historical document — do not follow as written (2026-09-21).** This guide
> predates the current workflow: it contains the old nvidia-first load order,
> reboot advice, and LFS instructions that no longer apply (LFS was removed;
> firmware and decompiled artifacts are local/gitignored). The current
> workflow is
> [`../scripts/stack_manager.sh`](../scripts/stack_manager.sh),
> [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md)
> and
> [`02-getting-started/01-install-and-first-run.md`](02-getting-started/01-install-and-first-run.md).

## Overview

This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging.

---

## 1. Repository Structure & Navigation

### Primary Workspace
```bash
$OPEN_IOV_ROOT/          # Main monorepo (git + LFS)
```

### Key Directories
| Directory | Purpose |
|-----------|---------|
| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) |
| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) |
| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` |
| `userspace/examples/` | `example_vgpu` test app |
| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files |
| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) |
| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) |
| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent |
| `docs/` | Architecture, implementation guides, TODO lists |
| `wiki/` | 10 technical wiki pages |

---

## 2. Shell Integration (zshrc)

### Required Aliases & Functions
Add to `~/.zshrc` and run `source ~/.zshrc`:

```bash
# Project navigation
alias oiv="cd $OPEN_IOV_ROOT"
alias oivk="cd $OPEN_IOV_ROOT/kernel"
alias oivu="cd $OPEN_IOV_ROOT/userspace"
alias oivf="cd $OPEN_IOV_ROOT/firmware/57X-GSP/570.144"
alias oivd="cd $OPEN_IOV_ROOT/docs"
alias oivs="cd $OPEN_IOV_ROOT/scripts"
alias oivt="cd $OPEN_IOV_ROOT/tools"

# Build commands
alias oiv-build="cd $OPEN_IOV_ROOT && make -j$(nproc)"
alias oiv-build-kernel="cd $OPEN_IOV_ROOT && make kernel"
alias oiv-build-userspace="cd $OPEN_IOV_ROOT && make userspace"
alias oiv-clean="cd $OPEN_IOV_ROOT && make clean"

# Module loading (requires reboot first!)
alias oiv-load="$OPEN_IOV_ROOT/scripts/load_vgpu_stack.sh"
alias oiv-unload="$OPEN_IOV_ROOT/scripts/unload_vgpu_stack.sh"
alias oiv-test="$OPEN_IOV_ROOT/scripts/test_vgpu_stack.sh"
alias oiv-test-gsp="$OPEN_IOV_ROOT/scripts/test_gsp_emu.sh"
alias oiv-test-guest="$OPEN_IOV_ROOT/scripts/test_guest_driver.sh"
alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh"
alias oiv-test-graphics="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh"
alias oiv-test-windows="$OPEN_IOV_ROOT/scripts/test_windows_guest.sh"

# mdev management
alias oiv-mdev-create="$OPEN_IOV_ROOT/scripts/create_mdev.sh"
alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'"

# QEMU
alias oiv-qemu="$OPEN_IOV_ROOT/scripts/launch_qemu_vgpu.sh"
alias oiv-qemu-debug="$OPEN_IOV_ROOT/scripts/debug_guest_init.sh"

# Firmware analysis
alias oiv-ghidra="cd $OPEN_IOV_ROOT/tools/ghidra && ./ghidraRun"
alias oiv-fw-analyze="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py"

# Debugging tools (NEW)
alias oiv-guest-monitor="python3 $OPEN_IOV_ROOT/tools/guest_init_monitor.py"
alias oiv-failure-analyzer="python3 $OPEN_IOV_ROOT/tools/failure_analyzer.py"
alias oiv-benchmark="$OPEN_IOV_ROOT/tools/benchmark_infrastructure.sh"
alias oiv-windows-test="$OPEN_IOV_ROOT/scripts/windows_guest_automation.sh"

# Kernel debugging
alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev"
alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev"
alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic"
alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu"

# Git
alias oiv-git="cd $OPEN_IOV_ROOT && git status"
alias oiv-git-log="cd $OPEN_IOV_ROOT && git log --oneline -20"
alias oiv-git-diff="cd $OPEN_IOV_ROOT && git diff"

# Restore point
alias oiv-restore="cd $OPEN_IOV_ROOT && git checkout f9ed5d8 -- . && git clean -fd"

# Help
oiv-help() {
    echo "╔═══════════════════════════════════════════════════════════════════════════════╗"
    echo "║         Open-IOV vGPU-IO Development Commands                                ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  NAVIGATION                                                                  ║"
    echo "║    oiv           - cd to open-iov root                                       ║"
    echo "║    oivk          - cd to kernel/                                             ║"
    echo "║    oivu          - cd to userspace/                                          ║"
    echo "║    oivf          - cd to firmware/57X-GSP/570.144/                           ║"
    echo "║    oivd          - cd to docs/                                               ║"
    echo "║    oivs          - cd to scripts/                                            ║"
    echo "║    oivt          - cd to tools/                                              ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  BUILD                                                                       ║"
    echo "║    oiv-build           - Build everything (kernel + userspace + examples)    ║"
    echo "║    oiv-build-kernel    - Build open-iov-vfio.ko only                         ║"
    echo "║    oiv-build-userspace - Build libiov.so + example_vgpu                      ║"
    echo "║    oiv-clean           - Clean all build artifacts                           ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  MODULE OPERATIONS (require root)                                            ║"
    echo "║    oiv-load       - Load nvidia.ko → open-iov-vfio.ko (refactored stack)     ║"
    echo "║    oiv-unload     - Unload all modules + remove mdevs                        ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  TESTING                                                                     ║"
    echo "║    oiv-test            - Run comprehensive test suite (test_vgpu_stack.sh)   ║"
    echo "║    oiv-test-gsp        - Run GSP emulation unit tests (test_gsp_emu.sh)      ║"
    echo "║    oiv-test-guest      - Test guest driver integration (test_guest_driver.sh)║"
    echo "║    oiv-test-cuda       - Test CUDA workloads (test_cuda_workloads.sh)        ║"
    echo "║    oiv-test-graphics   - Test graphics workloads (test_graphics_workloads.sh)║"
    echo "║    oiv-test-windows    - Test Windows guest (test_windows_guest.sh)          ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  MDEV MANAGEMENT                                                             ║"
    echo "║    oiv-mdev-create - Create mdev device for QEMU                             ║"
    echo "║    oiv-mdev-list   - List existing mdev devices                              ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  QEMU                                                                        ║"
    echo "║    oiv-qemu        - Launch QEMU with vGPU                                   ║"
    echo "║    oiv-qemu-debug  - Launch QEMU with GDB + guest monitor      ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  DEBUGGING                                                                   ║"
    echo "║    oiv-dmesg          - Filtered dmesg for NVIDIA/GSP/vGPU/mdev              ║"
    echo "║    oiv-dmesg-follow   - Live filtered dmesg (tail -f)                        ║"
    echo "║    oiv-dmesg-errors   - Show only errors/warnings                            ║"
    echo "║    oiv-trace          - Live kernel trace_pipe filtered for gsp_emu          ║"
    echo "║    oiv-guest-monitor  - Monitor guest init (correlates host traces + serial) ║"
    echo "║    oiv-failure-analyzer - Analyze/classify guest init failures               ║"
    echo "║    oiv-benchmark      - Run VM vs bare metal benchmarks                      ║"
    echo "║    oiv-windows-test   - Windows guest test automation (WinRM)                ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  GIT                                                                         ║"
    echo "║    oiv-git        - Git status in open-iov                                   ║"
    echo "║    oiv-git-log    - Git log (last 20 commits)                                ║"
    echo "║    oiv-git-diff   - Show uncommitted changes                                 ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  FIRMWARE ANALYSIS                                                           ║"
    echo "║    oiv-ghidra       - Launch Ghidra GUI                                      ║"
    echo "║    oiv-fw-analyze   - Run firmware analyzer (Python)                         ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║  EMERGENCY                                                                   ║"
    echo "║    oiv-restore    - Restore to commit f9ed5d8 (restore point)                ║"
    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
    echo "║    oiv-help       - This help                                                ║"
    echo "╚═══════════════════════════════════════════════════════════════════════════════╝"
}
```

### Usage
```bash
# After adding to ~/.zshrc:
source ~/.zshrc
oiv-help          # Show all commands
oiv               # Jump to project root
oiv-build         # Build everything
oiv-load          # Load kernel modules
oiv-mdev-create   # Create mdev for QEMU
oiv-qemu          # Launch QEMU guest
oiv-dmesg-follow  # Live kernel logs
```

---

## 3. Build Workflow

### Full Build
```bash
cd $OPEN_IOV_ROOT
make                    # Builds kernel + userspace + examples
# OR
oiv-build
```

### Incremental Builds
```bash
# Kernel only (open-iov-vfio.ko)
make kernel
# OR
cd kernel && make -j$(nproc)

# Userspace only (libiov.so + libvgpu.so + example_vgpu)
make userspace
# OR
cd userspace && make -j$(nproc)

# Examples only
make examples

# Tests
make tests

# Clean
make clean
# OR
oiv-clean
```

### Build Outputs
| Artifact | Location | Size |
|----------|----------|------|
| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB |
| `open-iov-vfio.ko` | `kernel/` | ~3.8MB |
| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB |
| `example_vgpu` | `userspace/examples/` | 27KB |

---

## 4. Kernel Module Loading

### Prerequisites
- The **stock Ubuntu nvidia driver must not be loaded** when you want our
  OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on
  desktop boots (udev) and is held open by the GUI.
  `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this
  machine) prevents that auto-load after reboot; `insmod` is unaffected.
- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia
  (compares `/sys/module/nvidia/version` with our build) and refuses with
  guidance instead of silently skipping.

### Load Order (Critical)
```bash
# 1. VFIO/mdev prerequisites
sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev

# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia
#    imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the
#    mdev parent on the physical GPU.
sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko

# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols
#    (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers)
sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko

# 4. GSP firmware for the host GPU
sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/
```

### Using Helper Script
```bash
./scripts/stack_manager.sh load-stack  # prereqs → open-iov-vfio → nvidia → firmware
```

### Verify Load
```bash
lsmod | grep -E "nvidia|open_iov"
dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov
ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/
cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances
```

### Unload
```bash
oiv-unload  # Removes mdevs, unloads modules in reverse order
```

---

## 5. mdev Device Management

### Create mdev (for QEMU)
```bash
oiv-mdev-create
# Interactive: selects GPU, vGPU type, generates UUID
# Creates: /sys/bus/mdev/devices/<UUID>/
```

### Manual Creation
```bash
# List GPUs
lspci -D -d 10de: | grep -E "VGA|3D"

# List supported vGPU types
ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/

# Create specific type
echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create
```

### List mdevs
```bash
oiv-mdev-list
# OR
ls -la /sys/bus/mdev/devices/
```

### Remove mdev
```bash
echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove
```

---

## 6. QEMU Guest Launch

### Using Helper Script
```bash
oiv-qemu
# Interactive: selects mdev, guest image, launches QEMU
```

### Manual Launch
```bash
qemu-system-x86_64 \
  -enable-kvm \
  -machine q35,accel=kvm,kernel_irqchip=split \
  -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \
  -smp 8,sockets=1,cores=8,threads=1 \
  -m 16G,slots=4,maxmem=64G \
  -mem-prealloc \
  -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \
  -numa node,memdev=mem \
  -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \
  -vga none -display none \
  -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0 \
  -usb -device usb-tablet \
  -name "vGPU-Guest" \
  -pidfile /tmp/qemu-vgpu.pid \
  -daemonize
```

### Connect to Guest
```bash
# SSH (port 2222 forwarded)
ssh -p 2222 user@localhost

# VNC (if enabled in launch script)
vncviewer localhost:1
```

---

## 7. Guest Driver Setup

### Inside Guest VM
```bash
# Run setup script (installs NVIDIA driver, firmware, loads module)
sudo $OPEN_IOV_ROOT/scripts/setup_guest_driver.sh

# Or manual:
# 1. Install NVIDIA driver (OpenRM 570.144)
# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/
# 3. modprobe nvidia
# 4. nvidia-smi
```

### Verify Guest
```bash
nvidia-smi                    # Should show vGPU (T4/A100/etc.)
lspci -d 10de:                # Should show 10de:1ed3 (TU104 vGPU)
dmesg -T | grep -i nvidia     # Check probe success
```

---

## 8. Testing Workflow

### Host Test Suite
```bash
oiv-test
# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification
```

### Guest CUDA Tests
```bash
# Inside guest:
cd /tmp
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/vectorAdd
make
./vectorAdd
```

### Guest Graphics Tests
```bash
# Inside guest:
vulkaninfo
vkcube
glxgears
glmark2
```

### Guest Video Tests
```bash
# Inside guest:
ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4
ffmpeg -hwaccel cuda -i test.mp4 -f null -
```

### Windows Guest Tests
```powershell
# Inside Windows guest:
# 1. Install NVIDIA driver (570.144)
# 2. nvidia-smi.exe
# 3. dxdiag
# 4. Run CUDA-Z / GPU-Z
# 5. Run 3DMark / FurMark
```

---

## 9. Firmware Reverse Engineering Workflow

### Analyze New Firmware
```bash
# 1. Download firmware from NVIDIA OpenRM release
# 2. Place in firmware dir
cp new-gsp.bin $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin

# 3. Run Ghidra headless analysis
cd $OPEN_IOV_ROOT/tools/ghidra
./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \
  -import $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin \
  -processor "RISCV:RV64:default" \
  -postScript DecompileFunctions.java \
  -deleteProject

# 4. Run firmware analyzer (auto-discovers addresses)
cd $OPEN_IOV_ROOT
python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json

# 5. Compare with previous version
python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json

# 6. Generate config if new version
python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml
```

### Ghidra Scripts (in tools/ghidra/)
| Script | Purpose |
|--------|---------|
| `DecompileFunctions.java` | Batch decompile all functions |
| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table |
| `ExtractStructLayouts.java` | Extract struct definitions from prologues |
| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching |

### Key Firmware Addresses (570.144W)
| Component | Address | Description |
|-----------|---------|-------------|
| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands |
| Secondary Dispatcher | 0x04ED39DC | Additional routing |
| BAR Trap Handler | 0x04CB8188 | 40KB |
| WPR Handler 1 | 0x04F00B0C | 62KB |
| WPR Handler 2 | 0x04F00C78 | 62KB |
| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers |
| ALLOC Main | 0x057E94EC | 40KB allocator |
| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler |
| Entry Point | 0x04009000 | Bootstrap |

---

## 10. Debugging Workflows

### Guest Driver Probe Issues
```bash
# Host: Watch live logs
oiv-dmesg-follow

# Guest: Check PCI config
lspci -vvv -d 10de:
cat /proc/bus/pci/devices

# Guest: Check BAR mapping
lspci -v -d 10de: | grep -A 10 "Memory at"

# Guest: Check MSI-X
lspci -vvv -d 10de: | grep -A 20 "MSI-X"
```

### GSP Emulation Debugging
```bash
# Check emulation thread
ps aux | grep gsp_emu

# Check RPC processing
dmesg -T | grep -i "gsp_emu\|rpc"

# Debugfs stats
cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats

# Add trace to gsp_emulation.c:
trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start)));
cat /sys/kernel/debug/tracing/trace
```

### QEMU with GDB (Guest Kernel Debugging)
```bash
# Launch QEMU with GDB server
qemu-system-x86_64 -s -S ...  # -s = gdbserver on :1234, -S = freeze

# In another terminal:
gdb ./vmlinux
(gdb) target remote :1234
(gdb) break nvidia_probe
(gdb) continue
```

### NEW: Comprehensive Debug Session (One Command)
```bash
# Launch debug session (QEMU + GDB + guest monitor)
./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux

# Or manually:
# Terminal 1: QEMU with GDB
./scripts/launch_qemu_vgpu.sh --gdb --freeze

# Terminal 2: Guest init monitor
python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300

# Terminal 3: GDB
gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux
(gdb) target remote localhost:1234
```

### NEW: Failure Analysis
```bash
# After a failed guest init, analyze logs
python3 tools/failure_analyzer.py \
    --guest-log /tmp/guest_serial.log \
    --host-trace /tmp/host_trace.log \
    --qemu-log /tmp/qemu.log \
    --output /tmp/failure_report.json

# This classifies the failure and provides specific recommendations
```

### NEW: Benchmarking
```bash
# Run VM vs bare metal benchmarks
./tools/benchmark_infrastructure.sh all

# Or specific categories
./tools/benchmark_infrastructure.sh cuda
./tools/benchmark_infrastructure.sh graphics
./tools/benchmark_infrastructure.sh pytorch
```

### Performance Profiling
```bash
# GSP RPC latency
# Add to gsp_emulation.c:
trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start)))

# View trace
cat /sys/kernel/debug/tracing/trace

# CUDA kernel launch overhead (in guest)
nvprof --profile-from-start off ./vectorAdd

# MSI-X injection latency
# Measure in gsp_emu_inject_completion_interrupt()
```

---

## 11. Git Workflow

### Restore Point
```bash
oiv-restore  # git checkout f9ed5d8 -- . && git clean -fd
```

### Branch for Refactor
```bash
cd $OPEN_IOV_ROOT
git checkout -b refactor/host-gsp-emulation
# Implement changes per docs/refactored-implementation-to-do.md
```

### Commit Changes
```bash
git add .
git commit -m "Description of changes"
```

### LFS for Large Files
```bash
git lfs ls-files  # Shows tracked large files
git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o"
```

---

## 12. Common Issues & Solutions

| Issue | Solution |
|-------|----------|
| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist |
| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps |
| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap |
| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory |
| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` |
| Migration fails | State mismatch - checksum all migrated structures |
| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table |
| Build fails | `make clean && make` or check kernel headers match running kernel |

---

## 13. Extending the Shell Environment

### Adding New Helpers
When you create new scripts or workflows, add them to `~/.zshrc`:

```bash
# Example: Add new test command
alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh"
alias oiv-test-gfx="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh"

# Example: Add firmware analysis shortcut
alias oiv-fw-new="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin"

# Reload
source ~/.zshrc
```

### Document New Helpers
Update `oiv-help()` function in `~/.zshrc` to include new commands.

---

## 14. Key Files Reference

| File | Purpose |
|------|---------|
| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps |
| `AGENTS.md` | Agent brain - workflow, tools, architecture |
| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design |
| `docs/refactored-architecture.md` | Host-only emulation design |
| `docs/refactored-implementation-to-do.md` | Granular implementation TODO |
| `scripts/load_vgpu_stack.sh` | Load all kernel modules |
| `scripts/create_mdev.sh` | Create mdev device |
| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU |
| `scripts/test_vgpu_stack.sh` | Comprehensive test suite |
| `scripts/debug_guest_init.sh` | NEW: One-command debug launch |
| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation |
| `tools/guest_init_monitor.py` | NEW: Guest init monitor |
| `tools/failure_analyzer.py` | NEW: Failure analyzer |
| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner |
| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent |
| `kernel/gsp_emulation.c/h` | GSP emulation engine |
| `kernel/firmware_analyzer.c/h` | Firmware analyzer |
| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher |
| `kernel/architecture/architecture.c/h` | Hardware abstraction |
| `kernel/version/version.c/h` | Version adaptation |
| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests |
| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers |
| `kernel/gsp_rpc_integration.c` | Integration layer |
| `kernel/main.c` | Module entry, mdev registration |
| `kernel/bar_emulation.c` | BAR + MSI-X emulation |
| `userspace/lib/libvgpu.c` | Userspace API |
| `userspace/lib/rpc_emulator.c` | Emulation fallbacks |

---

*Development Workflow Guide Version: 1.0*
*Repository: $OPEN_IOV_ROOT*
*Last Updated: $(date)*