Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 671 insertions, 0 deletions

diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md
new file mode 100644
index 0000000..9019ac1
--- /dev/null
+++ b/docs/DEVELOPMENT_WORKFLOW.md
@@ -0,0 +1,671 @@
+# Open-IOV vGPU-IO Development Workflow & Tools Guide
+
+## Overview
+
+This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging.
+
+---
+
+## 1. Repository Structure & Navigation
+
+### Primary Workspace
+```bash
+$OPEN_IOV_ROOT/          # Main monorepo (git + LFS)
+```
+
+### Key Directories
+| Directory | Purpose |
+|-----------|---------|
+| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) |
+| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) |
+| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` |
+| `userspace/examples/` | `example_vgpu` test app |
+| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files |
+| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) |
+| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) |
+| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent |
+| `docs/` | Architecture, implementation guides, TODO lists |
+| `wiki/` | 10 technical wiki pages |
+
+---
+
+## 2. Shell Integration (zshrc)
+
+### Required Aliases & Functions
+Add to `~/.zshrc` and run `source ~/.zshrc`:
+
+```bash
+# Project navigation
+alias oiv="cd $OPEN_IOV_ROOT"
+alias oivk="cd $OPEN_IOV_ROOT/kernel"
+alias oivu="cd $OPEN_IOV_ROOT/userspace"
+alias oivf="cd $OPEN_IOV_ROOT/firmware/57X-GSP/570.144"
+alias oivd="cd $OPEN_IOV_ROOT/docs"
+alias oivs="cd $OPEN_IOV_ROOT/scripts"
+alias oivt="cd $OPEN_IOV_ROOT/tools"
+
+# Build commands
+alias oiv-build="cd $OPEN_IOV_ROOT && make -j$(nproc)"
+alias oiv-build-kernel="cd $OPEN_IOV_ROOT && make kernel"
+alias oiv-build-userspace="cd $OPEN_IOV_ROOT && make userspace"
+alias oiv-clean="cd $OPEN_IOV_ROOT && make clean"
+
+# Module loading (requires reboot first!)
+alias oiv-load="$OPEN_IOV_ROOT/scripts/load_vgpu_stack.sh"
+alias oiv-unload="$OPEN_IOV_ROOT/scripts/unload_vgpu_stack.sh"
+alias oiv-test="$OPEN_IOV_ROOT/scripts/test_vgpu_stack.sh"
+alias oiv-test-gsp="$OPEN_IOV_ROOT/scripts/test_gsp_emu.sh"
+alias oiv-test-guest="$OPEN_IOV_ROOT/scripts/test_guest_driver.sh"
+alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh"
+alias oiv-test-graphics="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh"
+alias oiv-test-windows="$OPEN_IOV_ROOT/scripts/test_windows_guest.sh"
+
+# mdev management
+alias oiv-mdev-create="$OPEN_IOV_ROOT/scripts/create_mdev.sh"
+alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'"
+
+# QEMU
+alias oiv-qemu="$OPEN_IOV_ROOT/scripts/launch_qemu_vgpu.sh"
+alias oiv-qemu-debug="$OPEN_IOV_ROOT/scripts/debug_guest_init.sh"
+
+# Firmware analysis
+alias oiv-ghidra="cd $OPEN_IOV_ROOT/tools/ghidra && ./ghidraRun"
+alias oiv-fw-analyze="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py"
+
+# Debugging tools (NEW)
+alias oiv-guest-monitor="python3 $OPEN_IOV_ROOT/tools/guest_init_monitor.py"
+alias oiv-failure-analyzer="python3 $OPEN_IOV_ROOT/tools/failure_analyzer.py"
+alias oiv-benchmark="$OPEN_IOV_ROOT/tools/benchmark_infrastructure.sh"
+alias oiv-windows-test="$OPEN_IOV_ROOT/scripts/windows_guest_automation.sh"
+
+# Kernel debugging
+alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev"
+alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev"
+alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic"
+alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu"
+
+# Git
+alias oiv-git="cd $OPEN_IOV_ROOT && git status"
+alias oiv-git-log="cd $OPEN_IOV_ROOT && git log --oneline -20"
+alias oiv-git-diff="cd $OPEN_IOV_ROOT && git diff"
+
+# Restore point
+alias oiv-restore="cd $OPEN_IOV_ROOT && git checkout f9ed5d8 -- . && git clean -fd"
+
+# Help
+oiv-help() {
+    echo "╔═══════════════════════════════════════════════════════════════════════════════╗"
+    echo "║         Open-IOV vGPU-IO Development Commands                                ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  NAVIGATION                                                                  ║"
+    echo "║    oiv           - cd to open-iov root                                       ║"
+    echo "║    oivk          - cd to kernel/                                             ║"
+    echo "║    oivu          - cd to userspace/                                          ║"
+    echo "║    oivf          - cd to firmware/57X-GSP/570.144/                           ║"
+    echo "║    oivd          - cd to docs/                                               ║"
+    echo "║    oivs          - cd to scripts/                                            ║"
+    echo "║    oivt          - cd to tools/                                              ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  BUILD                                                                       ║"
+    echo "║    oiv-build           - Build everything (kernel + userspace + examples)    ║"
+    echo "║    oiv-build-kernel    - Build open-iov-vfio.ko only                         ║"
+    echo "║    oiv-build-userspace - Build libiov.so + example_vgpu                      ║"
+    echo "║    oiv-clean           - Clean all build artifacts                           ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  MODULE OPERATIONS (require root)                                            ║"
+    echo "║    oiv-load       - Load nvidia.ko → open-iov-vfio.ko (refactored stack)     ║"
+    echo "║    oiv-unload     - Unload all modules + remove mdevs                        ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  TESTING                                                                     ║"
+    echo "║    oiv-test            - Run comprehensive test suite (test_vgpu_stack.sh)   ║"
+    echo "║    oiv-test-gsp        - Run GSP emulation unit tests (test_gsp_emu.sh)      ║"
+    echo "║    oiv-test-guest      - Test guest driver integration (test_guest_driver.sh)║"
+    echo "║    oiv-test-cuda       - Test CUDA workloads (test_cuda_workloads.sh)        ║"
+    echo "║    oiv-test-graphics   - Test graphics workloads (test_graphics_workloads.sh)║"
+    echo "║    oiv-test-windows    - Test Windows guest (test_windows_guest.sh)          ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  MDEV MANAGEMENT                                                             ║"
+    echo "║    oiv-mdev-create - Create mdev device for QEMU                             ║"
+    echo "║    oiv-mdev-list   - List existing mdev devices                              ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  QEMU                                                                        ║"
+    echo "║    oiv-qemu        - Launch QEMU with vGPU                                   ║"
+    echo "║    oiv-qemu-debug  - Launch QEMU with GDB + guest monitor      ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  DEBUGGING                                                                   ║"
+    echo "║    oiv-dmesg          - Filtered dmesg for NVIDIA/GSP/vGPU/mdev              ║"
+    echo "║    oiv-dmesg-follow   - Live filtered dmesg (tail -f)                        ║"
+    echo "║    oiv-dmesg-errors   - Show only errors/warnings                            ║"
+    echo "║    oiv-trace          - Live kernel trace_pipe filtered for gsp_emu          ║"
+    echo "║    oiv-guest-monitor  - Monitor guest init (correlates host traces + serial) ║"
+    echo "║    oiv-failure-analyzer - Analyze/classify guest init failures               ║"
+    echo "║    oiv-benchmark      - Run VM vs bare metal benchmarks                      ║"
+    echo "║    oiv-windows-test   - Windows guest test automation (WinRM)                ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  GIT                                                                         ║"
+    echo "║    oiv-git        - Git status in open-iov                                   ║"
+    echo "║    oiv-git-log    - Git log (last 20 commits)                                ║"
+    echo "║    oiv-git-diff   - Show uncommitted changes                                 ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  FIRMWARE ANALYSIS                                                           ║"
+    echo "║    oiv-ghidra       - Launch Ghidra GUI                                      ║"
+    echo "║    oiv-fw-analyze   - Run firmware analyzer (Python)                         ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║  EMERGENCY                                                                   ║"
+    echo "║    oiv-restore    - Restore to commit f9ed5d8 (restore point)                ║"
+    echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
+    echo "║    oiv-help       - This help                                                ║"
+    echo "╚═══════════════════════════════════════════════════════════════════════════════╝"
+}
+```
+
+### Usage
+```bash
+# After adding to ~/.zshrc:
+source ~/.zshrc
+oiv-help          # Show all commands
+oiv               # Jump to project root
+oiv-build         # Build everything
+oiv-load          # Load kernel modules
+oiv-mdev-create   # Create mdev for QEMU
+oiv-qemu          # Launch QEMU guest
+oiv-dmesg-follow  # Live kernel logs
+```
+
+---
+
+## 3. Build Workflow
+
+### Full Build
+```bash
+cd $OPEN_IOV_ROOT
+make                    # Builds kernel + userspace + examples
+# OR
+oiv-build
+```
+
+### Incremental Builds
+```bash
+# Kernel only (open-iov-vfio.ko)
+make kernel
+# OR
+cd kernel && make -j$(nproc)
+
+# Userspace only (libiov.so + libvgpu.so + example_vgpu)
+make userspace
+# OR
+cd userspace && make -j$(nproc)
+
+# Examples only
+make examples
+
+# Tests
+make tests
+
+# Clean
+make clean
+# OR
+oiv-clean
+```
+
+### Build Outputs
+| Artifact | Location | Size |
+|----------|----------|------|
+| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB |
+| `open-iov-vfio.ko` | `kernel/` | ~3.8MB |
+| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB |
+| `example_vgpu` | `userspace/examples/` | 27KB |
+
+---
+
+## 4. Kernel Module Loading
+
+### Prerequisites
+- The **stock Ubuntu nvidia driver must not be loaded** when you want our
+  OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on
+  desktop boots (udev) and is held open by the GUI.
+  `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this
+  machine) prevents that auto-load after reboot; `insmod` is unaffected.
+- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia
+  (compares `/sys/module/nvidia/version` with our build) and refuses with
+  guidance instead of silently skipping.
+
+### Load Order (Critical)
+```bash
+# 1. VFIO/mdev prerequisites
+sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev
+
+# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia
+#    imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the
+#    mdev parent on the physical GPU.
+sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko
+
+# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols
+#    (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers)
+sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
+
+# 4. GSP firmware for the host GPU
+sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/
+```
+
+### Using Helper Script
+```bash
+./scripts/stack_manager.sh load-stack  # prereqs → open-iov-vfio → nvidia → firmware
+```
+
+### Verify Load
+```bash
+lsmod | grep -E "nvidia|open_iov"
+dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov
+ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/
+cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances
+```
+
+### Unload
+```bash
+oiv-unload  # Removes mdevs, unloads modules in reverse order
+```
+
+---
+
+## 5. mdev Device Management
+
+### Create mdev (for QEMU)
+```bash
+oiv-mdev-create
+# Interactive: selects GPU, vGPU type, generates UUID
+# Creates: /sys/bus/mdev/devices/<UUID>/
+```
+
+### Manual Creation
+```bash
+# List GPUs
+lspci -D -d 10de: | grep -E "VGA|3D"
+
+# List supported vGPU types
+ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/
+
+# Create specific type
+echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create
+```
+
+### List mdevs
+```bash
+oiv-mdev-list
+# OR
+ls -la /sys/bus/mdev/devices/
+```
+
+### Remove mdev
+```bash
+echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove
+```
+
+---
+
+## 6. QEMU Guest Launch
+
+### Using Helper Script
+```bash
+oiv-qemu
+# Interactive: selects mdev, guest image, launches QEMU
+```
+
+### Manual Launch
+```bash
+qemu-system-x86_64 \
+  -enable-kvm \
+  -machine q35,accel=kvm,kernel_irqchip=split \
+  -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \
+  -smp 8,sockets=1,cores=8,threads=1 \
+  -m 16G,slots=4,maxmem=64G \
+  -mem-prealloc \
+  -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \
+  -numa node,memdev=mem \
+  -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \
+  -vga none -display none \
+  -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \
+  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
+  -device virtio-net-pci,netdev=net0 \
+  -usb -device usb-tablet \
+  -name "vGPU-Guest" \
+  -pidfile /tmp/qemu-vgpu.pid \
+  -daemonize
+```
+
+### Connect to Guest
+```bash
+# SSH (port 2222 forwarded)
+ssh -p 2222 user@localhost
+
+# VNC (if enabled in launch script)
+vncviewer localhost:1
+```
+
+---
+
+## 7. Guest Driver Setup
+
+### Inside Guest VM
+```bash
+# Run setup script (installs NVIDIA driver, firmware, loads module)
+sudo $OPEN_IOV_ROOT/scripts/setup_guest_driver.sh
+
+# Or manual:
+# 1. Install NVIDIA driver (OpenRM 570.144)
+# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/
+# 3. modprobe nvidia
+# 4. nvidia-smi
+```
+
+### Verify Guest
+```bash
+nvidia-smi                    # Should show vGPU (T4/A100/etc.)
+lspci -d 10de:                # Should show 10de:1ed3 (TU104 vGPU)
+dmesg -T | grep -i nvidia     # Check probe success
+```
+
+---
+
+## 8. Testing Workflow
+
+### Host Test Suite
+```bash
+oiv-test
+# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification
+```
+
+### Guest CUDA Tests
+```bash
+# Inside guest:
+cd /tmp
+git clone https://github.com/NVIDIA/cuda-samples.git
+cd cuda-samples/Samples/1_Utilities/vectorAdd
+make
+./vectorAdd
+```
+
+### Guest Graphics Tests
+```bash
+# Inside guest:
+vulkaninfo
+vkcube
+glxgears
+glmark2
+```
+
+### Guest Video Tests
+```bash
+# Inside guest:
+ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4
+ffmpeg -hwaccel cuda -i test.mp4 -f null -
+```
+
+### Windows Guest Tests
+```powershell
+# Inside Windows guest:
+# 1. Install NVIDIA driver (570.144)
+# 2. nvidia-smi.exe
+# 3. dxdiag
+# 4. Run CUDA-Z / GPU-Z
+# 5. Run 3DMark / FurMark
+```
+
+---
+
+## 9. Firmware Reverse Engineering Workflow
+
+### Analyze New Firmware
+```bash
+# 1. Download firmware from NVIDIA OpenRM release
+# 2. Place in firmware dir
+cp new-gsp.bin $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin
+
+# 3. Run Ghidra headless analysis
+cd $OPEN_IOV_ROOT/tools/ghidra
+./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \
+  -import $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin \
+  -processor "RISCV:RV64:default" \
+  -postScript DecompileFunctions.java \
+  -deleteProject
+
+# 4. Run firmware analyzer (auto-discovers addresses)
+cd $OPEN_IOV_ROOT
+python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json
+
+# 5. Compare with previous version
+python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json
+
+# 6. Generate config if new version
+python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml
+```
+
+### Ghidra Scripts (in tools/ghidra/)
+| Script | Purpose |
+|--------|---------|
+| `DecompileFunctions.java` | Batch decompile all functions |
+| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table |
+| `ExtractStructLayouts.java` | Extract struct definitions from prologues |
+| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching |
+
+### Key Firmware Addresses (570.144W)
+| Component | Address | Description |
+|-----------|---------|-------------|
+| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands |
+| Secondary Dispatcher | 0x04ED39DC | Additional routing |
+| BAR Trap Handler | 0x04CB8188 | 40KB |
+| WPR Handler 1 | 0x04F00B0C | 62KB |
+| WPR Handler 2 | 0x04F00C78 | 62KB |
+| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers |
+| ALLOC Main | 0x057E94EC | 40KB allocator |
+| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler |
+| Entry Point | 0x04009000 | Bootstrap |
+
+---
+
+## 10. Debugging Workflows
+
+### Guest Driver Probe Issues
+```bash
+# Host: Watch live logs
+oiv-dmesg-follow
+
+# Guest: Check PCI config
+lspci -vvv -d 10de:
+cat /proc/bus/pci/devices
+
+# Guest: Check BAR mapping
+lspci -v -d 10de: | grep -A 10 "Memory at"
+
+# Guest: Check MSI-X
+lspci -vvv -d 10de: | grep -A 20 "MSI-X"
+```
+
+### GSP Emulation Debugging
+```bash
+# Check emulation thread
+ps aux | grep gsp_emu
+
+# Check RPC processing
+dmesg -T | grep -i "gsp_emu\|rpc"
+
+# Debugfs stats
+cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats
+
+# Add trace to gsp_emulation.c:
+trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start)));
+cat /sys/kernel/debug/tracing/trace
+```
+
+### QEMU with GDB (Guest Kernel Debugging)
+```bash
+# Launch QEMU with GDB server
+qemu-system-x86_64 -s -S ...  # -s = gdbserver on :1234, -S = freeze
+
+# In another terminal:
+gdb ./vmlinux
+(gdb) target remote :1234
+(gdb) break nvidia_probe
+(gdb) continue
+```
+
+### NEW: Comprehensive Debug Session (One Command)
+```bash
+# Launch debug session (QEMU + GDB + guest monitor)
+./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux
+
+# Or manually:
+# Terminal 1: QEMU with GDB
+./scripts/launch_qemu_vgpu.sh --gdb --freeze
+
+# Terminal 2: Guest init monitor
+python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300
+
+# Terminal 3: GDB
+gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux
+(gdb) target remote localhost:1234
+```
+
+### NEW: Failure Analysis
+```bash
+# After a failed guest init, analyze logs
+python3 tools/failure_analyzer.py \
+    --guest-log /tmp/guest_serial.log \
+    --host-trace /tmp/host_trace.log \
+    --qemu-log /tmp/qemu.log \
+    --output /tmp/failure_report.json
+
+# This classifies the failure and provides specific recommendations
+```
+
+### NEW: Benchmarking
+```bash
+# Run VM vs bare metal benchmarks
+./tools/benchmark_infrastructure.sh all
+
+# Or specific categories
+./tools/benchmark_infrastructure.sh cuda
+./tools/benchmark_infrastructure.sh graphics
+./tools/benchmark_infrastructure.sh pytorch
+```
+
+### Performance Profiling
+```bash
+# GSP RPC latency
+# Add to gsp_emulation.c:
+trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start)))
+
+# View trace
+cat /sys/kernel/debug/tracing/trace
+
+# CUDA kernel launch overhead (in guest)
+nvprof --profile-from-start off ./vectorAdd
+
+# MSI-X injection latency
+# Measure in gsp_emu_inject_completion_interrupt()
+```
+
+---
+
+## 11. Git Workflow
+
+### Restore Point
+```bash
+oiv-restore  # git checkout f9ed5d8 -- . && git clean -fd
+```
+
+### Branch for Refactor
+```bash
+cd $OPEN_IOV_ROOT
+git checkout -b refactor/host-gsp-emulation
+# Implement changes per docs/refactored-implementation-to-do.md
+```
+
+### Commit Changes
+```bash
+git add .
+git commit -m "Description of changes"
+```
+
+### LFS for Large Files
+```bash
+git lfs ls-files  # Shows tracked large files
+git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o"
+```
+
+---
+
+## 12. Common Issues & Solutions
+
+| Issue | Solution |
+|-------|----------|
+| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist |
+| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps |
+| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap |
+| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory |
+| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` |
+| Migration fails | State mismatch - checksum all migrated structures |
+| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table |
+| Build fails | `make clean && make` or check kernel headers match running kernel |
+
+---
+
+## 13. Extending the Shell Environment
+
+### Adding New Helpers
+When you create new scripts or workflows, add them to `~/.zshrc`:
+
+```bash
+# Example: Add new test command
+alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh"
+alias oiv-test-gfx="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh"
+
+# Example: Add firmware analysis shortcut
+alias oiv-fw-new="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin"
+
+# Reload
+source ~/.zshrc
+```
+
+### Document New Helpers
+Update `oiv-help()` function in `~/.zshrc` to include new commands.
+
+---
+
+## 14. Key Files Reference
+
+| File | Purpose |
+|------|---------|
+| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps |
+| `AGENTS.md` | Agent brain - workflow, tools, architecture |
+| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design |
+| `docs/refactored-architecture.md` | Host-only emulation design |
+| `docs/refactored-implementation-to-do.md` | Granular implementation TODO |
+| `scripts/load_vgpu_stack.sh` | Load all kernel modules |
+| `scripts/create_mdev.sh` | Create mdev device |
+| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU |
+| `scripts/test_vgpu_stack.sh` | Comprehensive test suite |
+| `scripts/debug_guest_init.sh` | NEW: One-command debug launch |
+| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation |
+| `tools/guest_init_monitor.py` | NEW: Guest init monitor |
+| `tools/failure_analyzer.py` | NEW: Failure analyzer |
+| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner |
+| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent |
+| `kernel/gsp_emulation.c/h` | GSP emulation engine |
+| `kernel/firmware_analyzer.c/h` | Firmware analyzer |
+| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher |
+| `kernel/architecture/architecture.c/h` | Hardware abstraction |
+| `kernel/version/version.c/h` | Version adaptation |
+| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests |
+| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers |
+| `kernel/gsp_rpc_integration.c` | Integration layer |
+| `kernel/main.c` | Module entry, mdev registration |
+| `kernel/bar_emulation.c` | BAR + MSI-X emulation |
+| `userspace/lib/libvgpu.c` | Userspace API |
+| `userspace/lib/rpc_emulator.c` | Emulation fallbacks |
+
+---
+
+*Development Workflow Guide Version: 1.0*
+*Repository: $OPEN_IOV_ROOT*
+*Last Updated: $(date)*
\ No newline at end of file