kernel: Fix GSP emulation shadow binding, eventfd wiring, RPC livelocks + docs suite + repo reorg

CRITICAL FIXES (guest driver bring-up):
- gsp_emulation.c: emulation engine bound to the WRONG BAR0 - it polled the
  host physical BAR0 (bars[0].vaddr ioremap) for the doorbell and read RPCs
  from a local dma_alloc_wc buffer, while the guest writes land in the
  vzalloc'd bar0_shadow. The RPC path could never work; PBA writes even
  corrupted the host GPU's physical BAR0+0x3000. gsp_emu_init now binds
  exclusively to bar0_shadow (doorbell/mailbox/PBA aliases;
  shared_mem_dma==0 marks the alias so cleanup never dma_free's it).
- main.c SET_IRQS: DATA_EVENTFD eventfds were DISCARDED ('TODO' + kfree) -
  the guest never received MSI-X completions. Full wiring now:
  eventfd_ctx_fdget per vector, fd -1 clears, dev_close releases refs.
  gsp_emu_inject_completion_interrupt looks up vgpu->irqs[0].trigger LIVE
  (init-time copy was permanently NULL since QEMU wires after gsp_emu_init).
- gsp_rpc_client.c: BAR_TRAP/MIGRATE in emulation mode called
  gsp_rpc_submit_raw which rings the SAME doorbell the thread polls -
  self-triggered RPC livelock + multi-second stalls. Now fail fast with
  -ENOTSUPP + pr_warn_ratelimited (guest BAR access is direct MMIO).

MORE FIXES:
- main.c: nv_vgpu_config_write returned SIZE_MAX (while(count--) wrap) and
  was dead code while the VFIO write handler did unfiltered byte copies
  (guest could rewrite vendor ID/BARs). Return fixed; VFIO write handler
  now routes through the per-register filter.
- kernel/Makefile: M=$(PWD) used the env var which is stale when invoked
  via the top-level make (pointed at repo root -> kbuild silently built
  NOTHING; explains every 'stale .ko' incident). Now M=$(CURDIR).
- Warnings sweep: header prototypes for mdev/type functions, format casts
  in gsp_comm.c DRY-RUN logs, unused variable.

ROBUSTNESS (harness/tooling):
- vgpu_vm.py: run had no timeout (start could block forever);
  guest_shell recv could block forever on a wedged guest. Both now bounded;
  guest_shell pre-flights guest-ping and reports wedge diagnostics.
- stack_manager.sh: qemu/qemu-debug now launch via the vgpu_vm.py harness
  (workflow rule: never raw QEMU).

REPO REORGANIZATION:
- driver/ (legacy guest-side gsp-vgpu.ko) moved to legacy-driver-guest/
  (reference only, not built; README explains). 3 legacy un-buildable test
  files moved to legacy-driver-guest/kernel-tests/. Top-level Makefile
  cleaned of the dead DRIVER_DIR reference.

DOCUMENTATION SUITE (new):
- docs/GSP_EMULATION_ARCHITECTURE.md - emulator deep-dive: shadow BAR0
  model, doorbell protocol, dispatch table, MSI-X path, lifecycle, memory-
  model caveats
- docs/DEBUGGING.md - QEMU GDB stub workflow, serial console, QMP NMI,
  failure taxonomy, full bring-up validation checklist, EPT-livelock case
  study, version-matrix discipline
- docs/ONBOARDING.md - repo tour, coding standards (incl. virt_to_phys /
  dma_addr_t / get-put rules), first build & test
- docs/VERSION_UPGRADE.md - end-to-end new-driver-version procedure
  (firmware fetch, Ghidra analysis, configs, signatures, validation)
- docs/PERFORMANCE.md - bare-metal A/B methodology + targets
- README.md doc index; STATE.md refreshed; DEVELOPMENT_WORKFLOW.md corrected
  (load order: open-iov-vfio BEFORE nvidia; stock-driver blacklist note)

6 files changed, 808 insertions, 14 deletions

diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md
new file mode 100644
index 0000000..dc95ed7
--- /dev/null
+++ b/docs/DEBUGGING.md
@@ -0,0 +1,190 @@
+# Open-IOV Debugging Guide
+
+> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest
+> driver failures, and validating a fully working setup. Built from real
+> debugging sessions (including the 2026-09-21 EPT-livelock hunt).
+
+---
+
+## 1. Debug surfaces (know them all)
+
+| Surface | Where | What it shows |
+|---------|-------|---------------|
+| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns |
+| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead |
+| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** |
+| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status |
+| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** |
+| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer |
+| GDB on host kernel | (optional) | host-side driver state |
+
+The  VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain:
+
+```yaml
+serialLog: "/tmp/vgpu-guest-serial.log"
+gdbPort: 1234
+gdbFreeze: false      # true = halt at reset vector (boot debugging)
+```
+
+---
+
+## 2. The golden rule of failure triage
+
+**When the guest agent dies mid-test, the guest is wedged in kernel mode.
+Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.**
+
+Triage order:
+1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages
+2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words
+3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck?
+
+---
+
+## 3. QEMU GDB stub workflow
+
+Attach (no symbols needed for triage):
+
+```bash
+gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \
+    -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \
+    -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit'
+```
+
+Reading the result:
+
+| Observation | Meaning |
+|-------------|---------|
+| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine |
+| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver |
+| RIPs moving between samples | guest alive and progressing |
+
+For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical
+`$rip` **and** `$rsp` every time = the *same instruction* faults/retries
+(EPT-violation livelock) rather than a loop making progress.
+
+With guest symbols (`vmlinux` + module `.ko` with debug info), add
+`add-symbol-file` and get real function names.
+
+### QMP NMI (soft alternative)
+
+```bash
+(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \
+ echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \
+ | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock
+```
+
+Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU
+(spinning with a faulting MMIO instruction) may not print anything — use the
+GDB stub instead.
+
+---
+
+## 4. Case study: the 2026-09-21 EPT livelock
+
+Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive.
+
+1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with
+   `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN.
+2. **GDB stub**: vCPU2 frozen at one address in module text
+   (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in
+   `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that
+   can never complete.
+3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0
+   shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM
+   retried forever.
+4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see
+   `docs/GSP_EMULATION_ARCHITECTURE.md` §2.
+
+Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from
+your VFIO mmap handler.** Always question `virt_to_phys()` on anything that
+isn't lowmem.
+
+---
+
+## 5. What a *successful* driver bring-up looks like (validation checklist)
+
+After `stack_manager.sh load-stack` on the host:
+
+- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers
+- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1**
+- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg
+
+After `vgpu_vm.py start vgpu-linux`:
+
+- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>`
+- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started`
+- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored)
+- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped)
+- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root`
+
+When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`):
+
+- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first)
+- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment)
+- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam
+- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found")
+- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done
+- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144**
+
+Teardown regression:
+
+- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1**
+- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread)
+- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone
+
+Kernel unit test:
+
+- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED`
+- [ ] `sudo rmmod gsp_emulation_test` → clean
+
+---
+
+## 6. Failure taxonomy (symptom → likely cause)
+
+| Symptom | Likely cause | First check |
+|---------|--------------|-------------|
+| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` |
+| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` |
+| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` |
+| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN |
+| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes |
+| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL |
+| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) |
+| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? |
+| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs |
+| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot |
+
+---
+
+## 7. Host-guest version matrix discipline
+
+Probe failures caused by **version mismatch** look identical to emulation bugs.
+Before deep debugging, verify the matrix:
+
+| Component | Must be | Check |
+|-----------|---------|-------|
+| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) |
+| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) |
+| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` |
+| Emulation target | 570.144W | decompiled firmware basis |
+
+Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs
+570.144 emulation cost a full session before the real mmap bug was found —
+the mismatch masked the signal.)
+
+---
+
+## 8. Quick reference
+
+```bash
+# Stack
+./scripts/stack_manager.sh status|load-stack|unload|dmesg
+# VM (harness ONLY - never raw /qemu)
+./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell
+# Debug
+tail -f /tmp/vgpu-guest-serial.log &
+gdb -ex 'target remote localhost:1234'
+sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov'
+# Recover wedged VM
+./scripts/vgpu_vm.py stop <uuid>   # or: sudo kill -9 <qemu pid>
+```
diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md
index 2263af6..9019ac1 100644
--- a/docs/DEVELOPMENT_WORKFLOW.md
+++ b/docs/DEVELOPMENT_WORKFLOW.md
@@ -17,7 +17,7 @@ $OPEN_IOV_ROOT/          # Main monorepo (git + LFS)
 | Directory | Purpose |
 |-----------|---------|
 | `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) |
-| `driver/` | Guest vGPU driver (`gsp-vgpu.ko`) - **to be removed** |
+| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) |
 | `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` |
 | `userspace/examples/` | `example_vgpu` test app |
 | `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files |
@@ -212,8 +212,7 @@ oiv-clean
 | Artifact | Location | Size |
 |----------|----------|------|
 | `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB |
-| `open-iov-vfio.ko` | `kernel/` | 2.8MB |
-| `gsp-vgpu.ko` | `driver/` | 1.8MB |
+| `open-iov-vfio.ko` | `kernel/` | ~3.8MB |
 | `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB |
 | `example_vgpu` | `userspace/examples/` | 27KB |
 
@@ -222,32 +221,44 @@ oiv-clean
 ## 4. Kernel Module Loading
 
 ### Prerequisites
-- **REBOOT REQUIRED** if kernel cmdline has `module_blacklist=nvidia,...`
-- Check: `cat /proc/cmdline | grep module_blacklist`
-- GRUB must be updated: `/etc/default/grub` → `update-grub` → `update-initramfs -u` → reboot
+- The **stock Ubuntu nvidia driver must not be loaded** when you want our
+  OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on
+  desktop boots (udev) and is held open by the GUI.
+  `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this
+  machine) prevents that auto-load after reboot; `insmod` is unaffected.
+- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia
+  (compares `/sys/module/nvidia/version` with our build) and refuses with
+  guidance instead of silently skipping.
 
 ### Load Order (Critical)
 ```bash
-# 1. NVIDIA OpenRM host driver (exports vGPU VFIO symbols)
-sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
+# 1. VFIO/mdev prerequisites
+sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev
 
-# 2. Open-IOV VFIO mdev driver (depends on nvidia.ko symbols)
+# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia
+#    imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the
+#    mdev parent on the physical GPU.
 sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko
 
-# 3. Guest vGPU driver (standalone, works without above)
-sudo insmod $OPEN_IOV_ROOT/driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin
+# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols
+#    (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers)
+sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
+
+# 4. GSP firmware for the host GPU
+sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/
 ```
 
 ### Using Helper Script
 ```bash
-oiv-load  # Runs all three in order
+./scripts/stack_manager.sh load-stack  # prereqs → open-iov-vfio → nvidia → firmware
 ```
 
 ### Verify Load
 ```bash
-lsmod | grep -E "nvidia|open_iov|gsp_vgpu"
+lsmod | grep -E "nvidia|open_iov"
 dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov
-ls -la /dev/gsp-vgpu /dev/nvidia*
+ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/
+cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances
 ```
 
 ### Unload
diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md
new file mode 100644
index 0000000..eb7f697
--- /dev/null
+++ b/docs/GSP_EMULATION_ARCHITECTURE.md
@@ -0,0 +1,192 @@
+# Open-IOV GSP Emulation Engine — Architecture
+
+> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`,
+> `kernel/bar_emulation.c`, or debugging guest driver bring-up.
+>
+> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21
+> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`;
+> for the original refactor plan see `refactored-architecture.md`.
+
+---
+
+## 1. Purpose
+
+An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects
+to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC
+messages to a mailbox in BAR0, rings a doorbell, and waits for a completion
+interrupt plus a response written back into the mailbox.
+
+Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the
+host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the
+host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes
+RPCs, performs them in software (or declines them), writes responses back, and
+injects MSI-X interrupts.
+
+```
+┌──────────────────────────── Guest VM ────────────────────────────┐
+│  stock nvidia.ko (570.144)                                       │
+│     │  writel(doorbell), memcpy(RPC → BAR0+0x1000)               │
+│     ▼                                                            │
+│  vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow        │
+└──────────────────────────────────────────────────────────────────┘
+                     │  (16 MB vzalloc'd shadow, uncached)
+┌──────────────────────────── Host ────────────────────────────────┐
+│  gsp_emu thread (open-iov-vfio.ko)                               │
+│     poll shadow+0x110C00 (doorbell)                              │
+│       → read  RPC msg   from shadow+0x1000                       │
+│       → dispatch (12 RPC types, emulation implementations)       │
+│       → write response  to   shadow+0x1080                       │
+│       → set   PBA bit     at   shadow+0x3000                     │
+│       → eventfd_signal(msix_trigger[0])  → guest MSI-X           │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. The single source of truth: `bar0_shadow`
+
+The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software
+shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()`
+(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler
+(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`).
+
+Everything the guest driver does with BAR0 lands in this shadow:
+
+| BAR0 offset      | Region                            | Guest writes go to | Emulation reads from |
+|------------------|-----------------------------------|--------------------|----------------------|
+| `0x0000-0x0FFF`  | (shadow register area)            | shadow             | shadow               |
+| `0x1000`         | RPC request mailbox (72 B msg)    | shadow             | `shared_mem_virt`    |
+| `0x1080`         | RPC response mailbox (60 B)       | shadow (written by emu) | guest reads   |
+| `0x2000`         | MSI-X table (32 × 16 B)           | shadow             | (guest-managed)      |
+| `0x3000`         | MSI-X PBA (pending bits)          | shadow (written by emu) | guest reads   |
+| `0x110C00`       | **GSP doorbell**                  | shadow             | thread polls this    |
+| rest             | emulated register space           | shadow             | shadow               |
+
+> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled
+> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real
+> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer —
+> it could never see guest traffic and even wrote PBA bits into the real GPU's
+> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`.
+> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so
+> cleanup never `dma_free`s it.
+
+### 2.1 Transport protocol (as emulated)
+
+1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000`
+   (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`).
+2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`.
+3. The emulation thread sees the doorbell, ACKs it (writes 0), validates
+   `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`.
+4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`,
+   `result[6]`, `seq_id`) written to `BAR0+0x1080`.
+5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired
+   eventfd → QEMU delivers MSI-X to the guest.
+6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and
+   accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the
+   public `gsp_emu_process_rpc()` entry does not bump counters).
+
+---
+
+## 3. RPC dispatch
+
+`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls
+the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation
+implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by
+`gsp_emu_init`).
+
+| Type (val)     | Handler path in emulation                              | Notes |
+|----------------|--------------------------------------------------------|-------|
+| ALLOC (0x01)   | `gsp_rpc_alloc_emul` — host pages + `dma_map_page`     | tracks `gsp_vram_alloc` list |
+| FREE (0x02)    | `gsp_rpc_free_emul`                                    |       |
+| MAP (0x03)     | `gsp_rpc_map_emul`                                     |       |
+| UNMAP (0x04)   | `gsp_rpc_unmap_emul`                                   |       |
+| PIN (0x05)     | `gsp_rpc_pin_emul`                                     |       |
+| UNPIN (0x06)   | `gsp_rpc_unpin_emul`                                   |       |
+| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission |
+| YIELD (0x08)   | no-op success                                          |       |
+| NOTIFY (0x09)  | log-only in dispatch                                   | no eventfd delivery yet |
+| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast)                            | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO |
+| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast)                            | 4-phase host protocol lives in `migration.c`; RPC wiring TODO |
+| DEBUG (0xFF)   | log-only in dispatch                                   |       |
+| unknown        | `-ENOTSUPP`                                            |       |
+
+> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to
+> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox
+> and **rings the same doorbell the thread polls** — a self-triggered infinite
+> RPC loop plus multi-second stalls. Emulation mode now fails these fast with
+> `pr_warn_ratelimited`.
+
+### Memory model caveat (design note)
+
+Guest VRAM (BAR1) is mapped **directly to the host GPU's physical BAR1** (gpa).
+`gsp_rpc_alloc_emul`, however, services ALLOC with *host system pages* DMA-mapped
+to the physical device. These two models are not yet unified: a guest allocation
+returning `gpu_va=0x100000000+` is not visible through the guest's BAR1 window.
+This is a known design gap to resolve before CUDA workloads; it does not affect
+driver probe/GSP boot, which is the current milestone.
+
+---
+
+## 4. Interrupt delivery (MSI-X)
+
+1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`.
+   Our ioctl handler takes `eventfd_ctx_fdget()` references into
+   `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs).
+2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`:
+   sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger
+   from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after*
+   `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a
+   real bug fixed 2026-09-21).
+
+INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator
+currently fires vector 0 for GSP completion.
+
+---
+
+## 5. Lifecycle
+
+```
+mdev create (nv_vgpu_mdev_create)
+  └─ schedule init_work
+       └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow)
+       └─ nv_vgpu_irq_init / nv_vgpu_memory_init
+       └─ gsp_emu_init
+            ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases)
+            ├─ gsp_client_init_emulation (RPC client, emulation_mode=true)
+            ├─ gsp_rpc_emulation_set(true)
+            └─ kthread_run(gsp_emu_thread_fn)
+
+mdev remove (nv_vgpu_mdev_remove)
+  └─ cancel init_work, wait for VFIO fds to close
+  └─ vfio_unregister_group_dev
+  └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion)
+                   → memory → irq → BAR (frees bar0_shadow)
+  └─ list_del, pci_dev_put, vfio_put_device
+```
+
+---
+
+## 6. Debugging the emulator
+
+- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'`
+- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on
+  the thread path only)
+- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub
+  `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation
+  checklist.
+- Unit test: `kernel/tests/gsp_emulation_test.ko` drives
+  `gsp_emu_process_rpc()` with crafted mailbox messages (requires
+  `open-iov-vfio.ko` loaded; `depends:` is set accordingly).
+
+---
+
+## 7. Key files
+
+| File | Role |
+|------|------|
+| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection |
+| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs |
+| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state |
+| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space |
+| `kernel/memory.c` | simple mapping/pin tracking (host management API) |
+| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) |
diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md
new file mode 100644
index 0000000..1d0a4f7
--- /dev/null
+++ b/docs/ONBOARDING.md
@@ -0,0 +1,166 @@
+# Open-IOV Developer Onboarding
+
+> Welcome. This guide gets you from zero to a productive contributor in ~30
+> minutes: what the project is, how the repo is laid out, how to build and
+> test, and the standards we hold the code to.
+
+---
+
+## 1. What this project is (60 seconds)
+
+**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver**
+runs in a VM with full GPU functionality — no GRID license. The trick: the
+host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the
+firmware processor the driver talks to) in software, so the guest's stock
+`nvidia.ko` can't tell it isn't running on real hardware.
+
+Everything guest-facing is **host-side only**. There is no Open-IOV code in
+the guest.
+
+How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W
+firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types
+mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md`
+for how the emulator works.
+
+## 2. Repository tour
+
+```
+open-iov/
+├── Makefile                 # top-level: all/kernel/userspace/examples/tests/clean
+├── install.sh               # fresh-machine reproduction (deps→nvidia→firmware→build→load)
+├── README.md                # project intro + quick start
+├── STATE.md                 # live project state (works/limitations/bugs)
+├── kernel/                  # ★ the host driver (open-iov-vfio.ko)
+│   ├── main.c               #   module init, mdev parent/driver, VFIO ops, PCI config
+│   ├── gsp_emulation.c/h    #   GSP emulation engine (thread, dispatch, MSI-X)
+│   ├── gsp_rpc_client.c/h   #   RPC API + emulation implementations + msg structs
+│   ├── bar_emulation.c/h    #   BAR0 shadow + MSI-X shadow state
+│   ├── bar_init.c           #   physical BAR mapping (host side)
+│   ├── memory.c             #   simple mapping/pin tracking
+│   ├── irq.c                #   IRQ/eventfd plumbing
+│   ├── ioctl.c              #   host management ioctls (libiov consumers)
+│   ├── migration.c          #   4-phase live-migration skeleton
+│   ├── firmware_analyzer.c/h    # runtime firmware signature/pattern analysis
+│   ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch
+│   ├── architecture/        #   hw abstraction (Turing/Ampere/Blackwell)
+│   ├── version/             #   driver-version adaptation (550/560/570/580)
+│   ├── include/             #   kernel headers (nvidia-vgpu-vfio.h is the main one)
+│   └── tests/               #   kernel unit tests (gsp_emulation_test)
+├── userspace/
+│   ├── lib/                 # libiov.so (+ libvgpu.so compat) management API
+│   ├── examples/            # example_vgpu test app
+│   └── tests/
+├── legacy-driver-guest/     # historical guest-side driver (reference only, NOT built)
+├── firmware/57X-GSP/570.144/    # GSP blob + analysis scripts (per-version dirs)
+├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko)
+├── patches/nvidia/          # our delta vs upstream OpenRM tag
+├── scripts/                 # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers
+├── tools/                   # guest monitor, failure analyzer, benchmarks, ghidra
+├── config/                  # declarative YAML: hardware/, versions/
+├── docs/                    # ★ all developer documentation (start here)
+└── wiki/                    # firmware analysis pages (RPC format, addresses, ...)
+```
+
+## 3. Your first build & test
+
+```bash
+cd open-iov
+./scripts/stack_manager.sh build-all      # kernel → nvidia → userspace (ORDER MATTERS)
+sudo ./scripts/stack_manager.sh load-stack
+sudo ./scripts/stack_manager.sh mdev-create
+./scripts/stack_manager.sh status
+```
+
+Rules that will save your week (also enforced in `stack_manager.sh`):
+
+1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`.
+   nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`.
+2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware.
+3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`.
+4. **Version matrix**: guest driver == host OpenRM == host GSP firmware ==
+   emulation target (all **570.144** today). Mismatches look exactly like
+   emulation bugs — check versions first (`docs/DEBUGGING.md` §7).
+5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness).
+   Never raw `` or `qemu-system-*` commands.
+
+Run the unit tests (host, needs the module loaded):
+
+```bash
+make tests
+sudo insmod kernel/tests/gsp_emulation_test.ko   # dmesg → ALL TESTS PASSED
+sudo rmmod gsp_emulation_test
+```
+
+## 4. Coding standards
+
+We follow the **Linux kernel coding style** for kernel code and common-sense
+C for userspace. The practical checklist:
+
+### Must
+- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the
+  surrounding file.
+- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate
+  `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order).
+- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use
+  `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT`
+  (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a
+  whole session — it is called out in code review.
+- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32`
+  (IOMMU IOVAs can exceed 32 bits).
+- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`,
+  `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`,
+  module refs. Unbalanced gets become leaks that block unload.
+- **Remove paths must fully unwind init paths** (we had a remove that forgot
+  `list_del` + kthread stop + VFIO unregister — leaked everything).
+- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation
+  (it rings the doorbell our own thread polls → self-trigger livelock).
+  Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists.
+- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant);
+  rate-limit anything a guest can spam.
+- Comments explain **why**, not what — especially around hardware/firmware
+  behavior discovered by reverse engineering (cite the wiki/address).
+- **Signed-off-by** style commit messages: short imperative summary line,
+  then the *why* (root cause), the *what*, and the user-visible effect.
+  Reference real dmesg/GDB evidence where relevant.
+
+### Must not
+- No guest-side Open-IOV code. Ever. The guest is stock.
+- No reboots/system-affecting commands from scripts without explicit operator
+  action (see the project's hard rule).
+- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone;
+  nvidia.ko imports *our* symbols, not vice versa.
+- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`.
+- Don't widen an API's hardware assumptions silently: if a function needs
+  `vgpu->pdev`, say so in the comment (unit tests can't call those paths).
+
+### Kernel module specifics
+- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`).
+- Exported symbols: only what other components genuinely need
+  (`EXPORT_SYMBOL` right after the definition).
+- New per-device resources belong in `struct nv_vgpu_device` with matching
+  init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain.
+
+## 5. Documentation expectations
+
+- New subsystem → new doc in `docs/` + a link from `README.md` and this file.
+- Changed behavior → update the affected doc in the **same commit**.
+- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table.
+- Firmware addresses/structures → `wiki/Address-Reference.md` and friends.
+
+## 6. Where to read next (in order)
+
+1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works
+2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist
+3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design
+4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end
+5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal
+6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference
+7. `wiki/` — firmware reverse-engineering canon
+
+## 7. Getting help / escalating
+
+- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`.
+- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first.
+- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test.
+- Found a new failure mode? Debug it per the guide, then **write it down** in
+  the taxonomy table so the next person doesn't rediscover it.
diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md
new file mode 100644
index 0000000..6077920
--- /dev/null
+++ b/docs/PERFORMANCE.md
@@ -0,0 +1,98 @@
+# Open-IOV Performance Testing & A/B Methodology
+
+> How we measure vGPU performance against bare metal, interpret the results,
+> and keep overhead on the critical path honest. **Run correctness first**
+> (`docs/DEBUGGING.md` §5) — numbers from a half-working stack are noise.
+
+---
+
+## 1. What overhead is acceptable (targets)
+
+| Path | Target | Why |
+|------|--------|-----|
+| GSP RPC round-trip (doorbell→response→MSI-X) | < 50 µs (stretch: < 10 µs) | every driver op goes through here |
+| MSI-X injection (eventfd → guest handler) | < 5 µs | completion latency dominates |
+| Guest BAR0 MMIO read/write | < 2 µs (shadow RAM) | hot path for register polling |
+| CUDA kernel launch (added overhead vs bare metal) | < 10% | user-visible |
+| Memory alloc/map RPCs | < 100 µs | setup-time, amortized |
+| CUDA compute throughput | ≥ 95% of bare metal | GPU-bound, should be ~free |
+| Graphics (frame rate) | ≥ 90% of bare metal | swapchain + submission overhead |
+
+## 2. The A/B rule
+
+Every measurement runs **twice on the same machine, same boot if possible**:
+
+- **A (bare metal)**: workload on the physical GPU via the host's stock
+  driver (or our OpenRM nvidia.ko directly on the hardware).
+- **B (vGPU)**: identical workload in the guest on the emulated vGPU.
+
+Same binary, same input sizes, same environment (governor, clocks, hugepages).
+Report **B/A ratio %** — never absolute numbers alone.
+
+One variable at a time. If B regresses, you changed the stack, the workload,
+or the machine — find which before optimizing.
+
+## 3. Microbenchmarks (build these first)
+
+1. **RPC round-trip**: from the guest, submit a DEBUG/NOOP-ish RPC in a loop;
+   time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()`
+   gives `total_latency_ns` and counts — divide for per-RPC host cost.
+   (Add `trace_printk` in `gsp_emu_dispatch_rpc` entry/exit for distribution;
+   read via `/sys/kernel/debug/tracing/trace`.)
+2. **MSI-X latency**: timestamp (guest TSC) at RPC submit → in the guest
+   driver's interrupt handler. Compare with the host eventfd_signal timestamp.
+3. **BAR0 shadow read latency**: guest loops reading a shadow register;
+   expect ~memory speed (uncached, so ~100-300 ns) — anything ≥ 1 µs means
+   you're trapping somewhere you shouldn't.
+4. **Doorbell polling cost**: host `top`/`perf` on `gsp_emu_N` — the adaptive
+   poll (busy → 50-100 µs sleep) should sit at ~0% CPU when idle, and react
+   within one sleep quantum under load. Tune `GSP_EMU_POLL_*` only with data.
+
+## 4. Workload benchmarks
+
+Runner: `tools/benchmark_infrastructure.sh` (extend it; keep the A/B pattern).
+In-guest execution goes through the harness (`vgpu_vm.py exec`) or the guest
+test agent (`tools/guest_agent/`).
+
+| Suite | What | Pass criterion |
+|-------|------|----------------|
+| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A |
+| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A |
+| `nvbandwidth` (if available) | H2D/D2H/device copy GB/s | report; DMA path validation |
+| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A |
+| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) |
+| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A |
+
+Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK`
+both sides — throttling skews everything), and the command lines.
+
+## 5. Where the overhead lives (profile in this order)
+
+1. **RPC dispatch** — the single biggest lever. Everything the guest driver
+   does funnels through the emulation thread.
+2. **Doorbell polling cadence** — too slow = latency; too hot = host CPU burn.
+3. **eventfd/MSI-X delivery** — measure, don't assume.
+4. **BAR0 uncached semantics** — shadow is UC for the guest; that's
+   MMIO-correct but slow for large reads. If the driver bulk-reads shadow,
+   consider caching-safe sub-regions.
+5. **Memory model gap** (ALLOC host-pages vs BAR1 direct VRAM) — see
+   `docs/GSP_EMULATION_ARCHITECTURE.md` §3. Until unified, DMA-heavy
+   workloads are not representative.
+
+## 6. Optimizing safely
+
+- Change one thing, re-run the A/B pair, keep the numbers with the commit.
+- Never optimize off bare-metal-only data: the goal is B/A, not A.
+- If a change makes B faster but breaks the §5 validation checklist in
+  `docs/DEBUGGING.md`, it doesn't count.
+- Record results in `STATE.md` (date, versions, B/A ratios) so regressions
+  are visible across commits.
+
+## 7. Reporting template
+
+```
+Date / commit / versions (guest=host=firmware=emulation):
+Machine / GPU / clocks locked?:
+Test, unit, A (bare), B (vGPU), B/A %:
+Notes (what changed vs last run):
+```
diff --git a/docs/VERSION_UPGRADE.md b/docs/VERSION_UPGRADE.md
new file mode 100644
index 0000000..94c7d68
--- /dev/null
+++ b/docs/VERSION_UPGRADE.md
@@ -0,0 +1,137 @@
+# Open-IOV Version Upgrade Guide
+
+> How to support a new NVIDIA driver version (e.g. moving from 570.144 to a
+> 610-series release) end-to-end: firmware acquisition → reverse engineering →
+> config → host driver → guest validation. The goal is that a version bump is
+> **configuration + analysis**, not a rewrite.
+
+---
+
+## 0. Version discipline (read first)
+
+Four components must stay in lock-step; a mismatch between any two produces
+failures that are indistinguishable from emulation bugs:
+
+```
+guest nvidia driver  ==  host OpenRM nvidia.ko  ==  host GSP firmware  ==  emulation target (decompiled GSP)
+```
+
+Today that anchor is **570.144** everywhere. When you bump one, you bump all
+four. Verify with `docs/DEBUGGING.md` §7 before any deep debugging.
+
+---
+
+## 1. Acquire the new firmware + matching OpenRM tag
+
+1. Download the Linux x86_64 runfile for the target version:
+   `https://us.download.nvidia.com/XFree86/Linux-x86_64/<VER>/NVIDIA-Linux-x86_64-<VER>.run`
+   (the `tesla/` path 404s for consumer releases).
+2. Extract the GSP firmware (`gsp_tu10x.bin` / arch-specific) — or fetch the
+   versioned blob from `linux-firmware.git`:
+   `nvidia/<VER>/gsp_tu10x.bin`.
+3. Extract the `.fwimage` section (host boot image) and record the MD5:
+   ```bash
+   objcopy --only-section=.fwimage gsp_tu10x.bin gsp-<VER>.fwimage.bin
+   md5sum gsp-<VER>.fwimage.bin
+   ```
+4. Note the matching OpenRM tag at
+   `github.com/NVIDIA/open-gpu-kernel-modules` (e.g. `570.144`).
+
+## 2. Lay down versioned directories (never overwrite a working version)
+
+```
+firmware/<SERIES>-GSP/<VER>/        # e.g. firmware/61X-GSP/610.XX/
+    gsp-<VER>.fwimage.bin
+    decompiled/                     # ghidra output lands here
+config/versions/<MAJOR>.yaml        # e.g. config/versions/610.yaml
+```
+
+Copy the analysis scripts from the previous version dir
+(`firmware/57X-GSP/570.144/*.py`) into the new one — they are script-relative.
+
+## 3. Reverse-engineer the new firmware (Ghidra headless)
+
+```bash
+cd tools/ghidra
+python3 analyze_fwimage.py \
+    ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \
+    /tmp/ghidra_project_<VER> "gsp_<VER>"
+cd ../../firmware/<SERIES>-GSP/<VER>
+python3 analyze_firmware.py gsp-<VER>.fwimage.bin
+```
+
+What to extract (compare against `wiki/Address-Reference.md` for 570.144W):
+
+| Artifact | Why it matters |
+|----------|----------------|
+| Primary RPC dispatcher + jump table | RPC types/subcommands may shift |
+| BAR trap / WPR handlers | BAR emulation behavior |
+| Entry/boot sequence | GSP boot RPC ordering the guest driver expects |
+| Migration tables | migration protocol (if enabled) |
+| RPC message layouts | struct changes break the emulator silently |
+| Version string/signature | runtime firmware matching |
+
+Sanity checks: still RV64IMAC? still the same 12 top-level RPC types? new
+subcommands? Confidential-computing/TEE additions?
+
+## 4. Produce the version config
+
+Update `config/versions/<MAJOR>.yaml` with:
+- `version.name/major/minor` and `firmware.gsp_version`
+- discovered dispatcher/handler addresses (if tracked in yaml)
+- `features` toggles per what the firmware actually has
+- `symbols` (the nvidia.ko vGPU VFIO interface — verify against the new
+  OpenRM tag; renamed exports break the nvidia build)
+
+## 5. Update the runtime signature database
+
+`kernel/firmware_analyzer.c` matches firmware by CRC32/SHA256 + patterns.
+Add the new entry to `known_signatures[]` (version string, hashes, discovered
+addresses). This is what lets one `open-iov-vfio.ko` serve multiple firmware
+versions safely.
+
+## 6. Rebuild host + guest against the new anchor
+
+1. Point `patches/nvidia/` at the new OpenRM tag (rebase our delta; verify
+   `modinfo nvidia.ko | grep depends` still shows `open-iov-vfio`).
+2. `./scripts/stack_manager.sh build-all` (kernel → nvidia → userspace).
+3. Install the new host firmware to `/lib/firmware/`
+   (`stack_manager.sh install-firmware`).
+4. **Guest**: install the *matching* stock driver in the VM (runfile for the
+   exact version), replacing the old one. Keep the guest able to boot without
+   nvidia auto-loading during the swap (temporarily move dkms-built modules
+   aside — see `docs/DEBUGGING.md`).
+
+## 7. Validate (same checklist every version)
+
+Run the full bring-up checklist in `docs/DEBUGGING.md` §5:
+module load → mdev → VM boot → guest probe → `nvidia-smi` → teardown
+regression → unit test. Then the workload tests (CUDA vectorAdd, graphics).
+Only after the checklist is green do performance comparisons
+(`docs/PERFORMANCE.md`).
+
+## 8. Rollback plan
+
+Keep the previous version's directory + config untouched. Rollback =
+`git checkout` the repo tag/commit for the old anchor + reinstall the old
+guest driver. Because versions live in parallel directories, the working
+setup is never destroyed by a forward attempt.
+
+## 9. Multi-version testing matrix (target state)
+
+For each supported anchor version × each supported GPU family
+(`config/hardware/*.yaml`), the CI matrix should run the §7 checklist. When
+adding a version, extend the matrix rather than replacing the old column —
+regressions across versions are the cheapest bugs to catch.
+
+## 10. Common pitfalls (learned the hard way)
+
+- **Runfile version drift**: Ubuntu's `nvidia-driver-XXX-open` metapackages
+  can resolve to a *newer* major (570-open pulled 580). Always check the
+  candidate version before installing; prefer the exact-version runfile.
+- **GSP firmware filename**: guest driver requests `nvidia/<VER>/gsp_tu10x.bin`;
+  a wrong-version firmware file = probe failure that looks like an emulation bug.
+- **Kernel text addresses differ per build**: re-derive Ghidra addresses per
+  firmware; never carry 570.144 addresses into a 610 config.
+- **Symbol renames across OpenRM tags**: `nvidia_vgpu_vfio_*` renames break
+  the nvidia build with `Unknown symbol` — check `kernel-open` diffs.