summaryrefslogtreecommitdiffstats
path: root/docs/DEVELOPMENT_WORKFLOW.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:07 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:07 -0400
commit681494a79e6c0e084354eec0229386a696a314c0 (patch)
treef0d69e9583cd6ed752da7e40c8167df24e228a9a /docs/DEVELOPMENT_WORKFLOW.md
parentbfcb856e09207d8edb714373d775537165d06b05 (diff)
downloadopen-iov-681494a79e6c0e084354eec0229386a696a314c0.tar.gz
open-iov-681494a79e6c0e084354eec0229386a696a314c0.zip
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful material in the structured tree: debugging workflows and the EPT-livelock case study (05-operations/02), performance microbenchmarks and A/B methodology (05-operations/01), the version-upgrade runbook (05-operations/02 section 4, with corrected host/guest framing), kernel-code pitfalls (07-developers) and the OpenRM build/load order (03-architecture/06 section 7). The remaining legacy files were verified redundant and removed. * docs/README now records where each migrated topic lives. * Portability: the config YAML files are called out as design targets that no code reads yet, with what adding support actually requires today. * Getting started: the OpenRM step is marked optional (native vGPU-management workflows only; the emulated path uses the distro driver). * tools docs: rmtool/rmtop marked as planned, pointing at the implemented tools/rmq subset. * wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the canonical userspace/cuda analysis) and index update. * scripts/check-docs-links.py plus a CI step so relative links cannot rot. * STATE.md references to removed documents repointed.
Diffstat (limited to 'docs/DEVELOPMENT_WORKFLOW.md')
-rw-r--r--docs/DEVELOPMENT_WORKFLOW.md681
1 files changed, 0 insertions, 681 deletions
diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md
deleted file mode 100644
index 5d43896..0000000
--- a/docs/DEVELOPMENT_WORKFLOW.md
+++ /dev/null
@@ -1,681 +0,0 @@
-# Open-IOV vGPU-IO Development Workflow & Tools Guide
-
-> **Historical document — do not follow as written (2026-09-21).** This guide
-> predates the current workflow: it contains the old nvidia-first load order,
-> reboot advice, and LFS instructions that no longer apply (LFS was removed;
-> firmware and decompiled artifacts are local/gitignored). The current
-> workflow is
-> [`../scripts/stack_manager.sh`](../scripts/stack_manager.sh),
-> [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md)
-> and
-> [`02-getting-started/01-install-and-first-run.md`](02-getting-started/01-install-and-first-run.md).
-
-## Overview
-
-This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging.
-
----
-
-## 1. Repository Structure & Navigation
-
-### Primary Workspace
-```bash
-$OPEN_IOV_ROOT/ # Main monorepo (git + LFS)
-```
-
-### Key Directories
-| Directory | Purpose |
-|-----------|---------|
-| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) |
-| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) |
-| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` |
-| `userspace/examples/` | `example_vgpu` test app |
-| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files |
-| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) |
-| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) |
-| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent |
-| `docs/` | Architecture, implementation guides, TODO lists |
-| `wiki/` | 10 technical wiki pages |
-
----
-
-## 2. Shell Integration (zshrc)
-
-### Required Aliases & Functions
-Add to `~/.zshrc` and run `source ~/.zshrc`:
-
-```bash
-# Project navigation
-alias oiv="cd $OPEN_IOV_ROOT"
-alias oivk="cd $OPEN_IOV_ROOT/kernel"
-alias oivu="cd $OPEN_IOV_ROOT/userspace"
-alias oivf="cd $OPEN_IOV_ROOT/firmware/57X-GSP/570.144"
-alias oivd="cd $OPEN_IOV_ROOT/docs"
-alias oivs="cd $OPEN_IOV_ROOT/scripts"
-alias oivt="cd $OPEN_IOV_ROOT/tools"
-
-# Build commands
-alias oiv-build="cd $OPEN_IOV_ROOT && make -j$(nproc)"
-alias oiv-build-kernel="cd $OPEN_IOV_ROOT && make kernel"
-alias oiv-build-userspace="cd $OPEN_IOV_ROOT && make userspace"
-alias oiv-clean="cd $OPEN_IOV_ROOT && make clean"
-
-# Module loading (requires reboot first!)
-alias oiv-load="$OPEN_IOV_ROOT/scripts/load_vgpu_stack.sh"
-alias oiv-unload="$OPEN_IOV_ROOT/scripts/unload_vgpu_stack.sh"
-alias oiv-test="$OPEN_IOV_ROOT/scripts/test_vgpu_stack.sh"
-alias oiv-test-gsp="$OPEN_IOV_ROOT/scripts/test_gsp_emu.sh"
-alias oiv-test-guest="$OPEN_IOV_ROOT/scripts/test_guest_driver.sh"
-alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh"
-alias oiv-test-graphics="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh"
-alias oiv-test-windows="$OPEN_IOV_ROOT/scripts/test_windows_guest.sh"
-
-# mdev management
-alias oiv-mdev-create="$OPEN_IOV_ROOT/scripts/create_mdev.sh"
-alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'"
-
-# QEMU
-alias oiv-qemu="$OPEN_IOV_ROOT/scripts/launch_qemu_vgpu.sh"
-alias oiv-qemu-debug="$OPEN_IOV_ROOT/scripts/debug_guest_init.sh"
-
-# Firmware analysis
-alias oiv-ghidra="cd $OPEN_IOV_ROOT/tools/ghidra && ./ghidraRun"
-alias oiv-fw-analyze="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py"
-
-# Debugging tools (NEW)
-alias oiv-guest-monitor="python3 $OPEN_IOV_ROOT/tools/guest_init_monitor.py"
-alias oiv-failure-analyzer="python3 $OPEN_IOV_ROOT/tools/failure_analyzer.py"
-alias oiv-benchmark="$OPEN_IOV_ROOT/tools/benchmark_infrastructure.sh"
-alias oiv-windows-test="$OPEN_IOV_ROOT/scripts/windows_guest_automation.sh"
-
-# Kernel debugging
-alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev"
-alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev"
-alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic"
-alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu"
-
-# Git
-alias oiv-git="cd $OPEN_IOV_ROOT && git status"
-alias oiv-git-log="cd $OPEN_IOV_ROOT && git log --oneline -20"
-alias oiv-git-diff="cd $OPEN_IOV_ROOT && git diff"
-
-# Restore point
-alias oiv-restore="cd $OPEN_IOV_ROOT && git checkout f9ed5d8 -- . && git clean -fd"
-
-# Help
-oiv-help() {
- echo "╔═══════════════════════════════════════════════════════════════════════════════╗"
- echo "║ Open-IOV vGPU-IO Development Commands ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ NAVIGATION ║"
- echo "║ oiv - cd to open-iov root ║"
- echo "║ oivk - cd to kernel/ ║"
- echo "║ oivu - cd to userspace/ ║"
- echo "║ oivf - cd to firmware/57X-GSP/570.144/ ║"
- echo "║ oivd - cd to docs/ ║"
- echo "║ oivs - cd to scripts/ ║"
- echo "║ oivt - cd to tools/ ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ BUILD ║"
- echo "║ oiv-build - Build everything (kernel + userspace + examples) ║"
- echo "║ oiv-build-kernel - Build open-iov-vfio.ko only ║"
- echo "║ oiv-build-userspace - Build libiov.so + example_vgpu ║"
- echo "║ oiv-clean - Clean all build artifacts ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ MODULE OPERATIONS (require root) ║"
- echo "║ oiv-load - Load nvidia.ko → open-iov-vfio.ko (refactored stack) ║"
- echo "║ oiv-unload - Unload all modules + remove mdevs ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ TESTING ║"
- echo "║ oiv-test - Run comprehensive test suite (test_vgpu_stack.sh) ║"
- echo "║ oiv-test-gsp - Run GSP emulation unit tests (test_gsp_emu.sh) ║"
- echo "║ oiv-test-guest - Test guest driver integration (test_guest_driver.sh)║"
- echo "║ oiv-test-cuda - Test CUDA workloads (test_cuda_workloads.sh) ║"
- echo "║ oiv-test-graphics - Test graphics workloads (test_graphics_workloads.sh)║"
- echo "║ oiv-test-windows - Test Windows guest (test_windows_guest.sh) ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ MDEV MANAGEMENT ║"
- echo "║ oiv-mdev-create - Create mdev device for QEMU ║"
- echo "║ oiv-mdev-list - List existing mdev devices ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ QEMU ║"
- echo "║ oiv-qemu - Launch QEMU with vGPU ║"
- echo "║ oiv-qemu-debug - Launch QEMU with GDB + guest monitor ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ DEBUGGING ║"
- echo "║ oiv-dmesg - Filtered dmesg for NVIDIA/GSP/vGPU/mdev ║"
- echo "║ oiv-dmesg-follow - Live filtered dmesg (tail -f) ║"
- echo "║ oiv-dmesg-errors - Show only errors/warnings ║"
- echo "║ oiv-trace - Live kernel trace_pipe filtered for gsp_emu ║"
- echo "║ oiv-guest-monitor - Monitor guest init (correlates host traces + serial) ║"
- echo "║ oiv-failure-analyzer - Analyze/classify guest init failures ║"
- echo "║ oiv-benchmark - Run VM vs bare metal benchmarks ║"
- echo "║ oiv-windows-test - Windows guest test automation (WinRM) ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ GIT ║"
- echo "║ oiv-git - Git status in open-iov ║"
- echo "║ oiv-git-log - Git log (last 20 commits) ║"
- echo "║ oiv-git-diff - Show uncommitted changes ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ FIRMWARE ANALYSIS ║"
- echo "║ oiv-ghidra - Launch Ghidra GUI ║"
- echo "║ oiv-fw-analyze - Run firmware analyzer (Python) ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ EMERGENCY ║"
- echo "║ oiv-restore - Restore to commit f9ed5d8 (restore point) ║"
- echo "╠═══════════════════════════════════════════════════════════════════════════════╣"
- echo "║ oiv-help - This help ║"
- echo "╚═══════════════════════════════════════════════════════════════════════════════╝"
-}
-```
-
-### Usage
-```bash
-# After adding to ~/.zshrc:
-source ~/.zshrc
-oiv-help # Show all commands
-oiv # Jump to project root
-oiv-build # Build everything
-oiv-load # Load kernel modules
-oiv-mdev-create # Create mdev for QEMU
-oiv-qemu # Launch QEMU guest
-oiv-dmesg-follow # Live kernel logs
-```
-
----
-
-## 3. Build Workflow
-
-### Full Build
-```bash
-cd $OPEN_IOV_ROOT
-make # Builds kernel + userspace + examples
-# OR
-oiv-build
-```
-
-### Incremental Builds
-```bash
-# Kernel only (open-iov-vfio.ko)
-make kernel
-# OR
-cd kernel && make -j$(nproc)
-
-# Userspace only (libiov.so + libvgpu.so + example_vgpu)
-make userspace
-# OR
-cd userspace && make -j$(nproc)
-
-# Examples only
-make examples
-
-# Tests
-make tests
-
-# Clean
-make clean
-# OR
-oiv-clean
-```
-
-### Build Outputs
-| Artifact | Location | Size |
-|----------|----------|------|
-| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB |
-| `open-iov-vfio.ko` | `kernel/` | ~3.8MB |
-| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB |
-| `example_vgpu` | `userspace/examples/` | 27KB |
-
----
-
-## 4. Kernel Module Loading
-
-### Prerequisites
-- The **stock Ubuntu nvidia driver must not be loaded** when you want our
- OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on
- desktop boots (udev) and is held open by the GUI.
- `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this
- machine) prevents that auto-load after reboot; `insmod` is unaffected.
-- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia
- (compares `/sys/module/nvidia/version` with our build) and refuses with
- guidance instead of silently skipping.
-
-### Load Order (Critical)
-```bash
-# 1. VFIO/mdev prerequisites
-sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev
-
-# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia
-# imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the
-# mdev parent on the physical GPU.
-sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko
-
-# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols
-# (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers)
-sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
-
-# 4. GSP firmware for the host GPU
-sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/
-```
-
-### Using Helper Script
-```bash
-./scripts/stack_manager.sh load-stack # prereqs → open-iov-vfio → nvidia → firmware
-```
-
-### Verify Load
-```bash
-lsmod | grep -E "nvidia|open_iov"
-dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov
-ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/
-cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances
-```
-
-### Unload
-```bash
-oiv-unload # Removes mdevs, unloads modules in reverse order
-```
-
----
-
-## 5. mdev Device Management
-
-### Create mdev (for QEMU)
-```bash
-oiv-mdev-create
-# Interactive: selects GPU, vGPU type, generates UUID
-# Creates: /sys/bus/mdev/devices/<UUID>/
-```
-
-### Manual Creation
-```bash
-# List GPUs
-lspci -D -d 10de: | grep -E "VGA|3D"
-
-# List supported vGPU types
-ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/
-
-# Create specific type
-echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create
-```
-
-### List mdevs
-```bash
-oiv-mdev-list
-# OR
-ls -la /sys/bus/mdev/devices/
-```
-
-### Remove mdev
-```bash
-echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove
-```
-
----
-
-## 6. QEMU Guest Launch
-
-### Using Helper Script
-```bash
-oiv-qemu
-# Interactive: selects mdev, guest image, launches QEMU
-```
-
-### Manual Launch
-```bash
-qemu-system-x86_64 \
- -enable-kvm \
- -machine q35,accel=kvm,kernel_irqchip=split \
- -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \
- -smp 8,sockets=1,cores=8,threads=1 \
- -m 16G,slots=4,maxmem=64G \
- -mem-prealloc \
- -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \
- -numa node,memdev=mem \
- -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \
- -vga none -display none \
- -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \
- -netdev user,id=net0,hostfwd=tcp::2222-:22 \
- -device virtio-net-pci,netdev=net0 \
- -usb -device usb-tablet \
- -name "vGPU-Guest" \
- -pidfile /tmp/qemu-vgpu.pid \
- -daemonize
-```
-
-### Connect to Guest
-```bash
-# SSH (port 2222 forwarded)
-ssh -p 2222 user@localhost
-
-# VNC (if enabled in launch script)
-vncviewer localhost:1
-```
-
----
-
-## 7. Guest Driver Setup
-
-### Inside Guest VM
-```bash
-# Run setup script (installs NVIDIA driver, firmware, loads module)
-sudo $OPEN_IOV_ROOT/scripts/setup_guest_driver.sh
-
-# Or manual:
-# 1. Install NVIDIA driver (OpenRM 570.144)
-# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/
-# 3. modprobe nvidia
-# 4. nvidia-smi
-```
-
-### Verify Guest
-```bash
-nvidia-smi # Should show vGPU (T4/A100/etc.)
-lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU)
-dmesg -T | grep -i nvidia # Check probe success
-```
-
----
-
-## 8. Testing Workflow
-
-### Host Test Suite
-```bash
-oiv-test
-# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification
-```
-
-### Guest CUDA Tests
-```bash
-# Inside guest:
-cd /tmp
-git clone https://github.com/NVIDIA/cuda-samples.git
-cd cuda-samples/Samples/1_Utilities/vectorAdd
-make
-./vectorAdd
-```
-
-### Guest Graphics Tests
-```bash
-# Inside guest:
-vulkaninfo
-vkcube
-glxgears
-glmark2
-```
-
-### Guest Video Tests
-```bash
-# Inside guest:
-ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4
-ffmpeg -hwaccel cuda -i test.mp4 -f null -
-```
-
-### Windows Guest Tests
-```powershell
-# Inside Windows guest:
-# 1. Install NVIDIA driver (570.144)
-# 2. nvidia-smi.exe
-# 3. dxdiag
-# 4. Run CUDA-Z / GPU-Z
-# 5. Run 3DMark / FurMark
-```
-
----
-
-## 9. Firmware Reverse Engineering Workflow
-
-### Analyze New Firmware
-```bash
-# 1. Download firmware from NVIDIA OpenRM release
-# 2. Place in firmware dir
-cp new-gsp.bin $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin
-
-# 3. Run Ghidra headless analysis
-cd $OPEN_IOV_ROOT/tools/ghidra
-./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \
- -import $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin \
- -processor "RISCV:RV64:default" \
- -postScript DecompileFunctions.java \
- -deleteProject
-
-# 4. Run firmware analyzer (auto-discovers addresses)
-cd $OPEN_IOV_ROOT
-python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json
-
-# 5. Compare with previous version
-python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json
-
-# 6. Generate config if new version
-python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml
-```
-
-### Ghidra Scripts (in tools/ghidra/)
-| Script | Purpose |
-|--------|---------|
-| `DecompileFunctions.java` | Batch decompile all functions |
-| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table |
-| `ExtractStructLayouts.java` | Extract struct definitions from prologues |
-| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching |
-
-### Key Firmware Addresses (570.144W)
-| Component | Address | Description |
-|-----------|---------|-------------|
-| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands |
-| Secondary Dispatcher | 0x04ED39DC | Additional routing |
-| BAR Trap Handler | 0x04CB8188 | 40KB |
-| WPR Handler 1 | 0x04F00B0C | 62KB |
-| WPR Handler 2 | 0x04F00C78 | 62KB |
-| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers |
-| ALLOC Main | 0x057E94EC | 40KB allocator |
-| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler |
-| Entry Point | 0x04009000 | Bootstrap |
-
----
-
-## 10. Debugging Workflows
-
-### Guest Driver Probe Issues
-```bash
-# Host: Watch live logs
-oiv-dmesg-follow
-
-# Guest: Check PCI config
-lspci -vvv -d 10de:
-cat /proc/bus/pci/devices
-
-# Guest: Check BAR mapping
-lspci -v -d 10de: | grep -A 10 "Memory at"
-
-# Guest: Check MSI-X
-lspci -vvv -d 10de: | grep -A 20 "MSI-X"
-```
-
-### GSP Emulation Debugging
-```bash
-# Check emulation thread
-ps aux | grep gsp_emu
-
-# Check RPC processing
-dmesg -T | grep -i "gsp_emu\|rpc"
-
-# Debugfs stats
-cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats
-
-# Add trace to gsp_emulation.c:
-trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start)));
-cat /sys/kernel/debug/tracing/trace
-```
-
-### QEMU with GDB (Guest Kernel Debugging)
-```bash
-# Launch QEMU with GDB server
-qemu-system-x86_64 -s -S ... # -s = gdbserver on :1234, -S = freeze
-
-# In another terminal:
-gdb ./vmlinux
-(gdb) target remote :1234
-(gdb) break nvidia_probe
-(gdb) continue
-```
-
-### NEW: Comprehensive Debug Session (One Command)
-```bash
-# Launch debug session (QEMU + GDB + guest monitor)
-./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux
-
-# Or manually:
-# Terminal 1: QEMU with GDB
-./scripts/launch_qemu_vgpu.sh --gdb --freeze
-
-# Terminal 2: Guest init monitor
-python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300
-
-# Terminal 3: GDB
-gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux
-(gdb) target remote localhost:1234
-```
-
-### NEW: Failure Analysis
-```bash
-# After a failed guest init, analyze logs
-python3 tools/failure_analyzer.py \
- --guest-log /tmp/guest_serial.log \
- --host-trace /tmp/host_trace.log \
- --qemu-log /tmp/qemu.log \
- --output /tmp/failure_report.json
-
-# This classifies the failure and provides specific recommendations
-```
-
-### NEW: Benchmarking
-```bash
-# Run VM vs bare metal benchmarks
-./tools/benchmark_infrastructure.sh all
-
-# Or specific categories
-./tools/benchmark_infrastructure.sh cuda
-./tools/benchmark_infrastructure.sh graphics
-./tools/benchmark_infrastructure.sh pytorch
-```
-
-### Performance Profiling
-```bash
-# GSP RPC latency
-# Add to gsp_emulation.c:
-trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start)))
-
-# View trace
-cat /sys/kernel/debug/tracing/trace
-
-# CUDA kernel launch overhead (in guest)
-nvprof --profile-from-start off ./vectorAdd
-
-# MSI-X injection latency
-# Measure in gsp_emu_inject_completion_interrupt()
-```
-
----
-
-## 11. Git Workflow
-
-### Restore Point
-```bash
-oiv-restore # git checkout f9ed5d8 -- . && git clean -fd
-```
-
-### Branch for Refactor
-```bash
-cd $OPEN_IOV_ROOT
-git checkout -b refactor/host-gsp-emulation
-# Implement changes per docs/refactored-implementation-to-do.md
-```
-
-### Commit Changes
-```bash
-git add .
-git commit -m "Description of changes"
-```
-
-### LFS for Large Files
-```bash
-git lfs ls-files # Shows tracked large files
-git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o"
-```
-
----
-
-## 12. Common Issues & Solutions
-
-| Issue | Solution |
-|-------|----------|
-| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist |
-| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps |
-| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap |
-| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory |
-| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` |
-| Migration fails | State mismatch - checksum all migrated structures |
-| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table |
-| Build fails | `make clean && make` or check kernel headers match running kernel |
-
----
-
-## 13. Extending the Shell Environment
-
-### Adding New Helpers
-When you create new scripts or workflows, add them to `~/.zshrc`:
-
-```bash
-# Example: Add new test command
-alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh"
-alias oiv-test-gfx="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh"
-
-# Example: Add firmware analysis shortcut
-alias oiv-fw-new="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin"
-
-# Reload
-source ~/.zshrc
-```
-
-### Document New Helpers
-Update `oiv-help()` function in `~/.zshrc` to include new commands.
-
----
-
-## 14. Key Files Reference
-
-| File | Purpose |
-|------|---------|
-| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps |
-| `AGENTS.md` | Agent brain - workflow, tools, architecture |
-| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design |
-| `docs/refactored-architecture.md` | Host-only emulation design |
-| `docs/refactored-implementation-to-do.md` | Granular implementation TODO |
-| `scripts/load_vgpu_stack.sh` | Load all kernel modules |
-| `scripts/create_mdev.sh` | Create mdev device |
-| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU |
-| `scripts/test_vgpu_stack.sh` | Comprehensive test suite |
-| `scripts/debug_guest_init.sh` | NEW: One-command debug launch |
-| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation |
-| `tools/guest_init_monitor.py` | NEW: Guest init monitor |
-| `tools/failure_analyzer.py` | NEW: Failure analyzer |
-| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner |
-| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent |
-| `kernel/gsp_emulation.c/h` | GSP emulation engine |
-| `kernel/firmware_analyzer.c/h` | Firmware analyzer |
-| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher |
-| `kernel/architecture/architecture.c/h` | Hardware abstraction |
-| `kernel/version/version.c/h` | Version adaptation |
-| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests |
-| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers |
-| `kernel/gsp_rpc_integration.c` | Integration layer |
-| `kernel/main.c` | Module entry, mdev registration |
-| `kernel/bar_emulation.c` | BAR + MSI-X emulation |
-| `userspace/lib/libvgpu.c` | Userspace API |
-| `userspace/lib/rpc_emulator.c` | Emulation fallbacks |
-
----
-
-*Development Workflow Guide Version: 1.0*
-*Repository: $OPEN_IOV_ROOT*
-*Last Updated: $(date)* \ No newline at end of file