kernel: Fix GSP emulation shadow binding, eventfd wiring, RPC livelocks + docs suite + repo reorg

CRITICAL FIXES (guest driver bring-up):
- gsp_emulation.c: emulation engine bound to the WRONG BAR0 - it polled the
  host physical BAR0 (bars[0].vaddr ioremap) for the doorbell and read RPCs
  from a local dma_alloc_wc buffer, while the guest writes land in the
  vzalloc'd bar0_shadow. The RPC path could never work; PBA writes even
  corrupted the host GPU's physical BAR0+0x3000. gsp_emu_init now binds
  exclusively to bar0_shadow (doorbell/mailbox/PBA aliases;
  shared_mem_dma==0 marks the alias so cleanup never dma_free's it).
- main.c SET_IRQS: DATA_EVENTFD eventfds were DISCARDED ('TODO' + kfree) -
  the guest never received MSI-X completions. Full wiring now:
  eventfd_ctx_fdget per vector, fd -1 clears, dev_close releases refs.
  gsp_emu_inject_completion_interrupt looks up vgpu->irqs[0].trigger LIVE
  (init-time copy was permanently NULL since QEMU wires after gsp_emu_init).
- gsp_rpc_client.c: BAR_TRAP/MIGRATE in emulation mode called
  gsp_rpc_submit_raw which rings the SAME doorbell the thread polls -
  self-triggered RPC livelock + multi-second stalls. Now fail fast with
  -ENOTSUPP + pr_warn_ratelimited (guest BAR access is direct MMIO).

MORE FIXES:
- main.c: nv_vgpu_config_write returned SIZE_MAX (while(count--) wrap) and
  was dead code while the VFIO write handler did unfiltered byte copies
  (guest could rewrite vendor ID/BARs). Return fixed; VFIO write handler
  now routes through the per-register filter.
- kernel/Makefile: M=$(PWD) used the env var which is stale when invoked
  via the top-level make (pointed at repo root -> kbuild silently built
  NOTHING; explains every 'stale .ko' incident). Now M=$(CURDIR).
- Warnings sweep: header prototypes for mdev/type functions, format casts
  in gsp_comm.c DRY-RUN logs, unused variable.

ROBUSTNESS (harness/tooling):
- vgpu_vm.py: run had no timeout (start could block forever);
  guest_shell recv could block forever on a wedged guest. Both now bounded;
  guest_shell pre-flights guest-ping and reports wedge diagnostics.
- stack_manager.sh: qemu/qemu-debug now launch via the vgpu_vm.py harness
  (workflow rule: never raw QEMU).

REPO REORGANIZATION:
- driver/ (legacy guest-side gsp-vgpu.ko) moved to legacy-driver-guest/
  (reference only, not built; README explains). 3 legacy un-buildable test
  files moved to legacy-driver-guest/kernel-tests/. Top-level Makefile
  cleaned of the dead DRIVER_DIR reference.

DOCUMENTATION SUITE (new):
- docs/GSP_EMULATION_ARCHITECTURE.md - emulator deep-dive: shadow BAR0
  model, doorbell protocol, dispatch table, MSI-X path, lifecycle, memory-
  model caveats
- docs/DEBUGGING.md - QEMU GDB stub workflow, serial console, QMP NMI,
  failure taxonomy, full bring-up validation checklist, EPT-livelock case
  study, version-matrix discipline
- docs/ONBOARDING.md - repo tour, coding standards (incl. virt_to_phys /
  dma_addr_t / get-put rules), first build & test
- docs/VERSION_UPGRADE.md - end-to-end new-driver-version procedure
  (firmware fetch, Ghidra analysis, configs, signatures, validation)
- docs/PERFORMANCE.md - bare-metal A/B methodology + targets
- README.md doc index; STATE.md refreshed; DEVELOPMENT_WORKFLOW.md corrected
  (load order: open-iov-vfio BEFORE nvidia; stock-driver blacklist note)

1 files changed, 190 insertions, 0 deletions

diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md
new file mode 100644
index 0000000..dc95ed7
--- /dev/null
+++ b/docs/DEBUGGING.md
@@ -0,0 +1,190 @@
+# Open-IOV Debugging Guide
+
+> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest
+> driver failures, and validating a fully working setup. Built from real
+> debugging sessions (including the 2026-09-21 EPT-livelock hunt).
+
+---
+
+## 1. Debug surfaces (know them all)
+
+| Surface | Where | What it shows |
+|---------|-------|---------------|
+| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns |
+| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead |
+| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** |
+| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status |
+| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** |
+| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer |
+| GDB on host kernel | (optional) | host-side driver state |
+
+The  VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain:
+
+```yaml
+serialLog: "/tmp/vgpu-guest-serial.log"
+gdbPort: 1234
+gdbFreeze: false      # true = halt at reset vector (boot debugging)
+```
+
+---
+
+## 2. The golden rule of failure triage
+
+**When the guest agent dies mid-test, the guest is wedged in kernel mode.
+Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.**
+
+Triage order:
+1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages
+2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words
+3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck?
+
+---
+
+## 3. QEMU GDB stub workflow
+
+Attach (no symbols needed for triage):
+
+```bash
+gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \
+    -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \
+    -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit'
+```
+
+Reading the result:
+
+| Observation | Meaning |
+|-------------|---------|
+| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine |
+| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver |
+| RIPs moving between samples | guest alive and progressing |
+
+For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical
+`$rip` **and** `$rsp` every time = the *same instruction* faults/retries
+(EPT-violation livelock) rather than a loop making progress.
+
+With guest symbols (`vmlinux` + module `.ko` with debug info), add
+`add-symbol-file` and get real function names.
+
+### QMP NMI (soft alternative)
+
+```bash
+(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \
+ echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \
+ | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock
+```
+
+Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU
+(spinning with a faulting MMIO instruction) may not print anything — use the
+GDB stub instead.
+
+---
+
+## 4. Case study: the 2026-09-21 EPT livelock
+
+Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive.
+
+1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with
+   `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN.
+2. **GDB stub**: vCPU2 frozen at one address in module text
+   (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in
+   `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that
+   can never complete.
+3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0
+   shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM
+   retried forever.
+4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see
+   `docs/GSP_EMULATION_ARCHITECTURE.md` §2.
+
+Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from
+your VFIO mmap handler.** Always question `virt_to_phys()` on anything that
+isn't lowmem.
+
+---
+
+## 5. What a *successful* driver bring-up looks like (validation checklist)
+
+After `stack_manager.sh load-stack` on the host:
+
+- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers
+- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1**
+- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg
+
+After `vgpu_vm.py start vgpu-linux`:
+
+- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>`
+- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started`
+- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored)
+- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped)
+- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root`
+
+When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`):
+
+- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first)
+- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment)
+- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam
+- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found")
+- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done
+- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144**
+
+Teardown regression:
+
+- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1**
+- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread)
+- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone
+
+Kernel unit test:
+
+- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED`
+- [ ] `sudo rmmod gsp_emulation_test` → clean
+
+---
+
+## 6. Failure taxonomy (symptom → likely cause)
+
+| Symptom | Likely cause | First check |
+|---------|--------------|-------------|
+| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` |
+| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` |
+| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` |
+| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN |
+| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes |
+| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL |
+| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) |
+| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? |
+| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs |
+| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot |
+
+---
+
+## 7. Host-guest version matrix discipline
+
+Probe failures caused by **version mismatch** look identical to emulation bugs.
+Before deep debugging, verify the matrix:
+
+| Component | Must be | Check |
+|-----------|---------|-------|
+| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) |
+| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) |
+| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` |
+| Emulation target | 570.144W | decompiled firmware basis |
+
+Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs
+570.144 emulation cost a full session before the real mmap bug was found —
+the mismatch masked the signal.)
+
+---
+
+## 8. Quick reference
+
+```bash
+# Stack
+./scripts/stack_manager.sh status|load-stack|unload|dmesg
+# VM (harness ONLY - never raw /qemu)
+./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell
+# Debug
+tail -f /tmp/vgpu-guest-serial.log &
+gdb -ex 'target remote localhost:1234'
+sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov'
+# Recover wedged VM
+./scripts/vgpu_vm.py stop <uuid>   # or: sudo kill -9 <qemu pid>
+```