docs: merge the legacy flat documents, add the CUDA deep-dive and link checks

* Merge and remove the pre-reboot flat documents, preserving their still-useful
  material in the structured tree: debugging workflows and the EPT-livelock
  case study (05-operations/02), performance microbenchmarks and A/B
  methodology (05-operations/01), the version-upgrade runbook
  (05-operations/02 section 4, with corrected host/guest framing), kernel-code
  pitfalls (07-developers) and the OpenRM build/load order
  (03-architecture/06 section 7). The remaining legacy files were verified
  redundant and removed.
* docs/README now records where each migrated topic lives.
* Portability: the config YAML files are called out as design targets that no
  code reads yet, with what adding support actually requires today.
* Getting started: the OpenRM step is marked optional (native vGPU-management
  workflows only; the emulated path uses the distro driver).
* tools docs: rmtool/rmtop marked as planned, pointing at the implemented
  tools/rmq subset.
* wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the
  canonical userspace/cuda analysis) and index update.
* scripts/check-docs-links.py plus a CI step so relative links cannot rot.
* STATE.md references to removed documents repointed.

2 files changed, 267 insertions, 10 deletions

diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md
index 0a27709..583d6c8 100644
--- a/docs/05-operations/01-performance-correctness-security.md
+++ b/docs/05-operations/01-performance-correctness-security.md
@@ -101,6 +101,77 @@ handoff documents carry the current rung.
 * Bulk copies and sustained compute: as close to native as the forwarding path
   allows; the forwarding design (proxy vs direct) is decided by measurement.
 
+### 3.4 Microbenchmarks (build these first)
+
+1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop;
+   time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()`
+   gives `total_latency_ns` and counts — divide for per-RPC host cost. Add
+   `trace_printk` at dispatch entry/exit for a distribution; read via
+   `/sys/kernel/debug/tracing/trace`.
+2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest
+   driver's interrupt handler. Compare with the host `eventfd_signal`
+   timestamp.
+3. **BAR0 shadow read latency** — guest loops reading a shadow register;
+   expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs
+   means a trap where there should not be one.
+4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the
+   adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and
+   react within one sleep quantum under load. Tune the poll quantum only with
+   data.
+
+### 3.5 A/B methodology
+
+Every measurement runs **twice on the same machine, same boot if possible**:
+
+* **A (bare metal)**: the workload on the physical GPU via the host's stock
+  driver (or the OpenRM `nvidia.ko` directly on the hardware).
+* **B (vGPU)**: the identical workload in the guest on the emulated vGPU.
+
+Same binary, same input sizes, same environment (governor, clocks, hugepages).
+Report **B/A ratio %** — never absolute numbers alone. Change one variable at a
+time; if B regresses, you changed the stack, the workload, or the machine, and
+finding which comes before optimizing.
+
+The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B
+pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the
+guest test agent. Workload suites and their pass criteria:
+
+| Suite | What | Pass criterion |
+|---|---|---|
+| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A |
+| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A |
+| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation |
+| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A |
+| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) |
+| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A |
+
+Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK`
+on both sides — throttling skews everything), and the command lines. Record
+results with the release evidence (§5) so regressions are visible across
+commits.
+
+### 3.6 Where the overhead lives (profile in this order)
+
+1. **RPC dispatch** — the single biggest lever; everything the guest driver
+   does funnels through the emulation thread.
+2. **Doorbell polling cadence** — too slow costs latency, too hot burns host
+   CPU.
+3. **eventfd/MSI-X delivery** — measure, do not assume.
+4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is
+   MMIO-correct but slow for large reads; if the driver bulk-reads shadows,
+   consider cache-safe sub-regions.
+5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads
+   are not representative until the real backing path exists.
+
+### 3.7 Optimizing safely
+
+* Change one thing, re-run the A/B pair, keep the numbers with the change.
+* Never optimize off bare-metal-only data: the goal is B/A, not A.
+* A change that makes B faster but breaks the §2.1 corpus or the rung ladder
+  does not count.
+* Record results (date, versions, B/A ratios) where the next release can see
+  them.
+
 ## 4. Security and isolation
 
 ### 4.1 Threat model
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md
index 331597a..52cee94 100644
--- a/docs/05-operations/02-debugging-and-maintenance-workflows.md
+++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md
@@ -148,19 +148,166 @@ coherent. Start at
 and
 [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md).
 
+### 3.7 QEMU gdbstub hang triage
+
+**The golden rule of failure triage: when the guest agent dies mid-test, the
+guest is wedged in kernel mode. Stop retrying `exec` and go straight to the
+gdbstub and host dmesg.** Triage order:
+
+1. `sudo dmesg -T | tail -50` — KVM warnings (e.g. `make_spte`, EPT), our
+   driver's last messages.
+2. `tail -100 /tmp/vgpu-guest-serial.log` — the guest kernel's last words
+   (works even when guest userspace is dead).
+3. gdbstub vCPU sampling (below) — *where* is each vCPU stuck?
+
+Attach without symbols for triage:
+
+```bash
+gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \
+    -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \
+    -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit'
+```
+
+Reading the result:
+
+| Observation | Meaning |
+|-------------|---------|
+| vCPU RIP in kernel text, disassembly shows `hlt` | CPU idle — fine |
+| vCPU RIP in module range, **same RIP+RSP across samples** | hard spin in a kernel module (e.g. an `nvidia.ko` polling loop) — this is the hung driver |
+| RIPs moving between samples | guest alive and progressing |
+
+Sample the stuck vCPU three times a few seconds apart. Identical `$rip` **and**
+`$rsp` every time means the *same instruction* faults/retries (an
+EPT-violation livelock) rather than a loop making progress. With guest symbols
+(`vmlinux` plus the module `.ko` built with debug info) use `add-symbol-file`
+to recover real function names.
+
+### 3.8 QMP NMI (soft alternative)
+
+```bash
+(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \
+ echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \
+  | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock
+```
+
+Watch the serial log afterwards for backtraces. A truly hard-wedged vCPU
+(spinning on a faulting MMIO instruction) may print nothing — use the gdbstub
+instead.
+
 ## 4. Workflows for new inputs
 
-### 4.1 New firmware version
+### 4.1 New firmware / driver version (upgrade runbook)
+
+**Version discipline first.** The anchor that must stay coherent is the
+**guest driver ↔ emulation target** (the GSP firmware generation whose
+protocol the host emulates): a guest driver whose protocol differs from what
+the emulation implements produces failures indistinguishable from emulation
+bugs. The host's physical-GPU driver is *not* part of that lock-step — the
+reference host uses the distribution's driver, and the OpenRM host modules are
+optional (native vGPU-management workflows only; see
+[`../03-architecture/06-host-module-and-hybrid-model.md` §7](../03-architecture/06-host-module-and-hybrid-model.md)).
+Today's emulation target is **570.144**; a version bump moves the guest driver
+and the emulation, not the host distro driver.
+
+**1. Acquire the firmware and matching source reference.**
+
+* Extract the GSP firmware from the target driver runfile (or fetch the
+  versioned blob from `linux-firmware.git`), then extract the `.fwimage`
+  section (host boot image) and record its MD5:
 
-1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study
-   material; firmware blobs are gitignored and not distributed in the
-   repository).
-2. Compute signatures (CRC32/SHA256) with the firmware analyzer.
-3. Add a firmware profile entry (addresses/signatures) so runtime analysis can
-   identify it.
-4. Run the driver against a guest using that firmware's matching driver; diff
-   the RPC/control inventory against the baseline.
-5. Add/adjust handlers only where the diff requires it; re-run the ladder.
+  ```bash
+  objcopy --only-section=.fwimage <extracted-blob> gsp-<VER>.fwimage.bin
+  md5sum gsp-<VER>.fwimage.bin
+  ```
+
+* Note the matching NVIDIA OpenRM tag (e.g. `570.144`) — it is the
+  version-exact reference for headers and wire structures.
+
+**2. Lay down versioned directories (never overwrite a working version).**
+
+```
+firmware/<SERIES>-GSP/<VER>/
+    gsp-<VER>.fwimage.bin
+    decompiled/                 # Ghidra output lands here
+config/versions/<MAJOR>.yaml    # design target; see §6 of the portability doc
+```
+
+Copy the analysis scripts from the previous version directory — they are
+script-relative.
+
+**3. Reverse-engineer the new firmware (Ghidra headless).**
+
+```bash
+cd tools/ghidra
+python3 analyze_fwimage.py \
+    ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \
+    /tmp/ghidra_project_<VER> "gsp_<VER>"
+cd ../../firmware/<SERIES>-GSP/<VER>
+python3 analyze_firmware.py gsp-<VER>.fwimage.bin
+```
+
+Diff against the 570.144 record (`wiki/Address-Reference.md`). What to extract:
+
+| Artifact | Why it matters |
+|---|---|
+| Primary RPC dispatcher + jump table | RPC types/subcommands may shift |
+| BAR trap / WPR handlers | BAR emulation behaviour |
+| Entry/boot sequence | the boot RPC ordering the guest driver expects |
+| Migration tables | migration protocol (if enabled) |
+| RPC message layouts | struct changes break the emulator silently |
+| Version string/signature | runtime firmware matching |
+
+Sanity checks: still RV64IMAC? still the same twelve top-level RPC types? new
+subcommands? confidential-computing/TEE additions? Never carry 570.144
+addresses into a new configuration — re-derive per firmware.
+
+**4. Update the version surface (data first).**
+
+* Add a hardware/version profile entry (addresses, signatures, capability
+  toggles) so runtime analysis can identify the image; remember the YAML
+  files under `config/` are design targets today (see
+  [`../06-portability/hardware-software-matrix.md` §6](../06-portability/hardware-software-matrix.md)).
+* Add the new entry to the runtime signature database in
+  `kernel/firmware_analyzer.c` (`known_signatures[]`: version string, hashes,
+  discovered addresses). This is what lets one `open-iov-vfio.ko` identify
+  multiple firmware versions safely.
+
+**5. Validate (same checklist every version).**
+
+1. Host: module load/unload ×3 and mdev create/destroy
+   (`./scripts/stack_manager.sh test`).
+2. Guest: boot → `nvidia.ko` probe → `nvidia-smi` → teardown regression
+   (no leaked `gsp_emu` thread, mdev removable again).
+3. Inventory diff: capture the full init trace and diff function/control IDs
+   against the previous version; for each difference decode against the
+   exact-version reference headers, then add a version-keyed handler or a
+   capability-table entry only where the diff requires it.
+4. Then the evaluation ladder (rungs 1–2, then the CUDA corpus) and only then
+   performance comparisons
+   ([`01-performance-correctness-security.md`](01-performance-correctness-security.md)).
+
+**6. Rollback.** Keep the previous version's directory and configuration
+untouched. Rollback = check out the old anchor's commit/tag and reinstall the
+old guest driver. Because versions live in parallel directories, a forward
+attempt never destroys the working setup.
+
+**Common pitfalls (learned the hard way).**
+
+* **Runfile version drift**: distribution metapackages can resolve to a newer
+  major than requested. Check the candidate version before installing; prefer
+  the exact-version runfile for the guest.
+* **Firmware filename mismatch**: a wrong-version firmware file is a probe
+  failure that looks like an emulation bug.
+* **Address drift per build**: kernel text addresses differ per firmware
+  build; re-derive Ghidra addresses, never reuse them.
+* **Symbol renames across OpenRM tags**: renamed exports break the optional
+  OpenRM build with `Unknown symbol` — check the `kernel-open` diff when that
+  path is in use.
+
+**Multi-version testing matrix (target state).** For each supported anchor
+version × each hardware family, run the step-5 bring-up sequence. When adding
+a version, extend the matrix rather than replacing the old column —
+regressions across versions are the cheapest bugs to catch.
 
 ### 4.2 New guest driver version
 
@@ -188,6 +335,23 @@ and
 
 ## 5. Crash and hang recovery
 
+### 5.1 Failure taxonomy (symptom → likely cause)
+
+| Symptom | Likely cause | First check |
+|---------|--------------|-------------|
+| `insmod open-iov-vfio.ko: Unknown symbol` | missing prerequisite module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` |
+| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` |
+| Guest: no PCI device | mdev not attached | QEMU command line for `sysfsdev=` |
+| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN |
+| Guest: probe hangs, no KVM warning | doorbell/RPC never consumed | host dmesg for the emulation thread and doorbell writes |
+| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL |
+| Guest agent dead, QEMU alive | guest kernel wedge | gdbstub vCPU sampling (§3.7) |
+| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? |
+| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs |
+| `pr_warn_ratelimited(... not supported in emulation mode)` | guest used a BAR_TRAP/MIGRATE RPC | expected fail-fast; wire a real handler if hot |
+
+### 5.2 Recovery actions
+
 | Situation | Action |
 |---|---|
 | Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
@@ -195,6 +359,28 @@ and
 | Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
 | Kernel oops in the host module | capture the full oops + `dmesg` before reloading |
 
+### 5.3 Case study: the EPT livelock
+
+Symptoms: guest `modprobe nvidia` hangs; the guest agent dies; QEMU stays
+alive.
+
+1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte`
+   with `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN.
+2. **gdbstub**: one vCPU frozen at a single address in module text, identical
+   RIP/RSP across samples; the other vCPUs in `hlt`. The faulting instruction
+   was `mov 0x4(%rax),%r13d` — a memory read that can never complete.
+3. **Root cause**: the VFIO `mmap` handler used `virt_to_phys()` on the
+   `vzalloc`'d BAR0 shadow → garbage PFN in the EPT → every guest BAR0 read
+   faulted → KVM retried forever.
+4. **Fix**: per-page `vmalloc_to_pfn()` mapping.
+
+**Lesson: a KVM `make_spte` reserved-bits warning plus one frozen vCPU means a
+bad PFN from your VFIO mmap handler.** Always question `virt_to_phys()` on
+anything that is not lowmem; the module rule is recorded in
+[`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md)
+and the shadow mapping is described in
+[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md).
+
 ## 6. Release engineering
 
 * Versions are tagged; each tag carries the rung/matrix table and the