diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
| commit | fade84f8b5b781d29daad40699d07928dedea550 (patch) | |
| tree | e4156b3d0a6a1c794c90ef6834b949321cdfb5cb | |
| parent | c587eac377c1c5b720142edadebaace5cb1b17c0 (diff) | |
| download | open-iov-fade84f8b5b781d29daad40699d07928dedea550.tar.gz open-iov-fade84f8b5b781d29daad40699d07928dedea550.zip | |
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful
material in the structured tree: debugging workflows and the EPT-livelock
case study (05-operations/02), performance microbenchmarks and A/B
methodology (05-operations/01), the version-upgrade runbook
(05-operations/02 section 4, with corrected host/guest framing), kernel-code
pitfalls (07-developers) and the OpenRM build/load order
(03-architecture/06 section 7). The remaining legacy files were verified
redundant and removed.
* docs/README now records where each migrated topic lives.
* Portability: the config YAML files are called out as design targets that no
code reads yet, with what adding support actually requires today.
* Getting started: the OpenRM step is marked optional (native vGPU-management
workflows only; the emulated path uses the distro driver).
* tools docs: rmtool/rmtop marked as planned, pointing at the implemented
tools/rmq subset.
* wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the
canonical userspace/cuda analysis) and index update.
* scripts/check-docs-links.py plus a CI step so relative links cannot rot.
* STATE.md references to removed documents repointed.
24 files changed, 830 insertions, 3230 deletions
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f3f71a9..125a43f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -173,6 +173,9 @@ jobs: echo "Found: $f" done + - name: Check documentation links + run: python3 scripts/check-docs-links.py + - name: Validate markdown syntax run: | cd wiki @@ -30,7 +30,7 @@ Guest driver **570.144 stock runfile installed** in the VM. The fixed module is built and pushed; it loads after the next host reboot (a leaked `gsp_emu` kthread from the pre-fix remove path pins the old module — rmmod would oops). Post-reboot: guest `nvidia.ko` probe → `nvidia-smi` → teardown -regression → unit test → CUDA validation (`docs/DEBUGGING.md` §5 checklist). +regression → unit test → CUDA validation (`docs/05-operations/02-debugging-and-maintenance-workflows.md`; the historical checklist is summarised there). ## Recently solved bugs @@ -56,11 +56,11 @@ regression → unit test → CUDA validation (`docs/DEBUGGING.md` §5 checklist) ## Known open bugs / limitations - Guest `nvidia.ko` end-to-end probe not yet validated (fixed module loads - post-reboot; checklist in `docs/DEBUGGING.md` §5) + post-reboot; checklist in `docs/05-operations/02-debugging-and-maintenance-workflows.md`) - **Memory-model gap**: guest VRAM (BAR1) maps directly to physical BAR1 gpa, but `gsp_rpc_alloc_emul` services ALLOC with host system pages — the two are not unified; must be resolved before CUDA/DMA-heavy workloads - (`docs/GSP_EMULATION_ARCHITECTURE.md` §3) + (`docs/03-architecture/06-host-module-and-hybrid-model.md`) - BAR_TRAP / MIGRATE RPCs fail fast with `-ENOTSUPP` in emulation mode (guest's normal BAR access is direct MMIO; real handlers TODO) - `gsp_emu_process_rpc()` doesn't bump stats counters (only the kthread @@ -78,7 +78,7 @@ regression → unit test → CUDA validation (`docs/DEBUGGING.md` §5 checklist) - Windows guest not yet validated (automation scaffolding present) - MSI-X emulation has dual stores (guest-visible `bar0_shadow` table vs `bar_emul_ctx.msix_table[]` used by the currently-unused trap path) — - documented in `docs/GSP_EMULATION_ARCHITECTURE.md`; harmless with the + documented in `docs/03-architecture/06-host-module-and-hybrid-model.md`; harmless with the direct-mmap design ## Reproducibility notes diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md index f323dbb..d1c57c3 100644 --- a/docs/02-getting-started/01-install-and-first-run.md +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -30,7 +30,7 @@ git clone https://git.vgpu.io/open-iov.git cd open-iov ./install.sh deps # build/runtime dependencies -./install.sh nvidia # NVIDIA open kernel modules (submodule) + patches +./install.sh nvidia # optional: NVIDIA open kernel modules (submodule) + patches ./install.sh firmware # GSP firmware image (verified) ./install.sh build # kernel module + userspace ``` @@ -45,7 +45,8 @@ short and comments the order. # 1. Kernel module first: it generates Module.symvers used by the next step. make kernel -# 2. NVIDIA OpenRM modules (must see our symbols): +# 2. Optional: NVIDIA OpenRM modules (needed only for native vGPU-management +# workflows; see section 3). They must see our symbols: # KBUILD_EXTRA_SYMBOLS=<repo>/kernel/Module.symvers ./scripts/build_nvidia_openrm.sh @@ -62,11 +63,22 @@ make userspace ```bash ./scripts/stack_manager.sh status # check what is currently loaded -./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> open-iov -> nvidia (if used) +./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> open-iov -> nvidia (optional) ./scripts/stack_manager.sh mdev-create # create a vGPU mediated device ./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/ ``` +**The `nvidia` (OpenRM) stage is optional.** It is needed only when the host +must run NVIDIA's OpenRM modules built against the symbols `open-iov-vfio.ko` +exports — that is, the native vGPU-management/RM-callback workflows. The +QEMU/emulated path uses the distribution driver that already manages the +physical GPU; `load-stack` skips the OpenRM submodule when a distribution +driver is present, and the build steps for it (`./install.sh nvidia`, +`./scripts/build_nvidia_openrm.sh`) can be skipped. When the OpenRM path *is* +used, the build and load order in +[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md) +§7 applies. + The mdev device is the virtual GPU. Everything after this point treats it like any other VFIO device. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index 4f9efdb..5849071 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -237,7 +237,29 @@ mismatch. Their workflows are documented in * On typical single-GPU developer systems the distribution's NVIDIA driver manages the physical GPU and the Open-IOV module is used standalone; the OpenRM submodule path is only needed for the native-vGPU-management - workflows. + workflows (RM callbacks) and is unnecessary for the QEMU/emulated path. + +**Build and load order on the OpenRM path:** + +```bash +# 1. Kernel module first - generates kernel/Module.symvers. +make kernel # or: ./scripts/stack_manager.sh build-kernel + +# 2. OpenRM modules with our symbols resolved. +./scripts/build_nvidia_openrm.sh # wraps KBUILD_EXTRA_SYMBOLS=<repo>/kernel/Module.symvers +grep -i vgpu kernel/Module.symvers # symbols exported +modinfo .../kernel-open/nvidia.ko | grep -i vgpu # symbols consumed + +# 3. Load order: Open-IOV first (exports and registers the mdev parent), +# then the optional OpenRM modules, then firmware. +``` + +If a distribution driver already owns the physical GPU, skip step 2 and the +OpenRM part of step 3: the emulated QEMU path runs with the distribution +driver. Common failures on this path: `Unknown symbol nvidia_vgpu_vfio_*` +(`KBUILD_EXTRA_SYMBOLS` missing or wrong) and symbols present in +`Module.symvers` but absent from `nvidia.ko` (stale build cache — `make clean` +in `kernel-open` first). ## 8. What is implemented vs. designed diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 0a27709..583d6c8 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -101,6 +101,77 @@ handoff documents carry the current rung. * Bulk copies and sustained compute: as close to native as the forwarding path allows; the forwarding design (proxy vs direct) is decided by measurement. +### 3.4 Microbenchmarks (build these first) + +1. **RPC round-trip** — from the guest, submit a DEBUG/NOOP-ish RPC in a loop; + time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()` + gives `total_latency_ns` and counts — divide for per-RPC host cost. Add + `trace_printk` at dispatch entry/exit for a distribution; read via + `/sys/kernel/debug/tracing/trace`. +2. **MSI-X latency** — timestamp (guest TSC) at RPC submit → the guest + driver's interrupt handler. Compare with the host `eventfd_signal` + timestamp. +3. **BAR0 shadow read latency** — guest loops reading a shadow register; + expect memory speed (uncached, so roughly 100–300 ns). Anything ≥ 1 µs + means a trap where there should not be one. +4. **Doorbell polling cost** — host `top`/`perf` on the `gsp_emu` thread; the + adaptive poll (busy → 50–100 µs sleep) should sit at ~0% CPU when idle and + react within one sleep quantum under load. Tune the poll quantum only with + data. + +### 3.5 A/B methodology + +Every measurement runs **twice on the same machine, same boot if possible**: + +* **A (bare metal)**: the workload on the physical GPU via the host's stock + driver (or the OpenRM `nvidia.ko` directly on the hardware). +* **B (vGPU)**: the identical workload in the guest on the emulated vGPU. + +Same binary, same input sizes, same environment (governor, clocks, hugepages). +Report **B/A ratio %** — never absolute numbers alone. Change one variable at a +time; if B regresses, you changed the stack, the workload, or the machine, and +finding which comes before optimizing. + +The runner is `tools/benchmark_infrastructure.sh` (extend it; keep the A/B +pattern); in-guest execution goes through `scripts/vgpu_vm.py exec` or the +guest test agent. Workload suites and their pass criteria: + +| Suite | What | Pass criterion | +|---|---|---| +| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A | +| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A | +| `nvbandwidth` (if available) | H2D/D2H/D2D copy GB/s | report; DMA-path validation | +| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A | +| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) | +| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A | + +Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK` +on both sides — throttling skews everything), and the command lines. Record +results with the release evidence (§5) so regressions are visible across +commits. + +### 3.6 Where the overhead lives (profile in this order) + +1. **RPC dispatch** — the single biggest lever; everything the guest driver + does funnels through the emulation thread. +2. **Doorbell polling cadence** — too slow costs latency, too hot burns host + CPU. +3. **eventfd/MSI-X delivery** — measure, do not assume. +4. **BAR0 uncached semantics** — the shadow is uncached to the guest, which is + MMIO-correct but slow for large reads; if the driver bulk-reads shadows, + consider cache-safe sub-regions. +5. **Memory-model gap** — guest VRAM is host RAM today; DMA-heavy workloads + are not representative until the real backing path exists. + +### 3.7 Optimizing safely + +* Change one thing, re-run the A/B pair, keep the numbers with the change. +* Never optimize off bare-metal-only data: the goal is B/A, not A. +* A change that makes B faster but breaks the §2.1 corpus or the rung ladder + does not count. +* Record results (date, versions, B/A ratios) where the next release can see + them. + ## 4. Security and isolation ### 4.1 Threat model diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index 331597a..52cee94 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -148,19 +148,166 @@ coherent. Start at and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). +### 3.7 QEMU gdbstub hang triage + +**The golden rule of failure triage: when the guest agent dies mid-test, the +guest is wedged in kernel mode. Stop retrying `exec` and go straight to the +gdbstub and host dmesg.** Triage order: + +1. `sudo dmesg -T | tail -50` — KVM warnings (e.g. `make_spte`, EPT), our + driver's last messages. +2. `tail -100 /tmp/vgpu-guest-serial.log` — the guest kernel's last words + (works even when guest userspace is dead). +3. gdbstub vCPU sampling (below) — *where* is each vCPU stuck? + +Attach without symbols for triage: + +```bash +gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ + -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ + -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' +``` + +Reading the result: + +| Observation | Meaning | +|-------------|---------| +| vCPU RIP in kernel text, disassembly shows `hlt` | CPU idle — fine | +| vCPU RIP in module range, **same RIP+RSP across samples** | hard spin in a kernel module (e.g. an `nvidia.ko` polling loop) — this is the hung driver | +| RIPs moving between samples | guest alive and progressing | + +Sample the stuck vCPU three times a few seconds apart. Identical `$rip` **and** +`$rsp` every time means the *same instruction* faults/retries (an +EPT-violation livelock) rather than a loop making progress. With guest symbols +(`vmlinux` plus the module `.ko` built with debug info) use `add-symbol-file` +to recover real function names. + +### 3.8 QMP NMI (soft alternative) + +```bash +(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ + echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ + | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock +``` + +Watch the serial log afterwards for backtraces. A truly hard-wedged vCPU +(spinning on a faulting MMIO instruction) may print nothing — use the gdbstub +instead. + ## 4. Workflows for new inputs -### 4.1 New firmware version +### 4.1 New firmware / driver version (upgrade runbook) + +**Version discipline first.** The anchor that must stay coherent is the +**guest driver ↔ emulation target** (the GSP firmware generation whose +protocol the host emulates): a guest driver whose protocol differs from what +the emulation implements produces failures indistinguishable from emulation +bugs. The host's physical-GPU driver is *not* part of that lock-step — the +reference host uses the distribution's driver, and the OpenRM host modules are +optional (native vGPU-management workflows only; see +[`../03-architecture/06-host-module-and-hybrid-model.md` §7](../03-architecture/06-host-module-and-hybrid-model.md)). +Today's emulation target is **570.144**; a version bump moves the guest driver +and the emulation, not the host distro driver. + +**1. Acquire the firmware and matching source reference.** + +* Extract the GSP firmware from the target driver runfile (or fetch the + versioned blob from `linux-firmware.git`), then extract the `.fwimage` + section (host boot image) and record its MD5: -1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study - material; firmware blobs are gitignored and not distributed in the - repository). -2. Compute signatures (CRC32/SHA256) with the firmware analyzer. -3. Add a firmware profile entry (addresses/signatures) so runtime analysis can - identify it. -4. Run the driver against a guest using that firmware's matching driver; diff - the RPC/control inventory against the baseline. -5. Add/adjust handlers only where the diff requires it; re-run the ladder. + ```bash + objcopy --only-section=.fwimage <extracted-blob> gsp-<VER>.fwimage.bin + md5sum gsp-<VER>.fwimage.bin + ``` + +* Note the matching NVIDIA OpenRM tag (e.g. `570.144`) — it is the + version-exact reference for headers and wire structures. + +**2. Lay down versioned directories (never overwrite a working version).** + +``` +firmware/<SERIES>-GSP/<VER>/ + gsp-<VER>.fwimage.bin + decompiled/ # Ghidra output lands here +config/versions/<MAJOR>.yaml # design target; see §6 of the portability doc +``` + +Copy the analysis scripts from the previous version directory — they are +script-relative. + +**3. Reverse-engineer the new firmware (Ghidra headless).** + +```bash +cd tools/ghidra +python3 analyze_fwimage.py \ + ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \ + /tmp/ghidra_project_<VER> "gsp_<VER>" +cd ../../firmware/<SERIES>-GSP/<VER> +python3 analyze_firmware.py gsp-<VER>.fwimage.bin +``` + +Diff against the 570.144 record (`wiki/Address-Reference.md`). What to extract: + +| Artifact | Why it matters | +|---|---| +| Primary RPC dispatcher + jump table | RPC types/subcommands may shift | +| BAR trap / WPR handlers | BAR emulation behaviour | +| Entry/boot sequence | the boot RPC ordering the guest driver expects | +| Migration tables | migration protocol (if enabled) | +| RPC message layouts | struct changes break the emulator silently | +| Version string/signature | runtime firmware matching | + +Sanity checks: still RV64IMAC? still the same twelve top-level RPC types? new +subcommands? confidential-computing/TEE additions? Never carry 570.144 +addresses into a new configuration — re-derive per firmware. + +**4. Update the version surface (data first).** + +* Add a hardware/version profile entry (addresses, signatures, capability + toggles) so runtime analysis can identify the image; remember the YAML + files under `config/` are design targets today (see + [`../06-portability/hardware-software-matrix.md` §6](../06-portability/hardware-software-matrix.md)). +* Add the new entry to the runtime signature database in + `kernel/firmware_analyzer.c` (`known_signatures[]`: version string, hashes, + discovered addresses). This is what lets one `open-iov-vfio.ko` identify + multiple firmware versions safely. + +**5. Validate (same checklist every version).** + +1. Host: module load/unload ×3 and mdev create/destroy + (`./scripts/stack_manager.sh test`). +2. Guest: boot → `nvidia.ko` probe → `nvidia-smi` → teardown regression + (no leaked `gsp_emu` thread, mdev removable again). +3. Inventory diff: capture the full init trace and diff function/control IDs + against the previous version; for each difference decode against the + exact-version reference headers, then add a version-keyed handler or a + capability-table entry only where the diff requires it. +4. Then the evaluation ladder (rungs 1–2, then the CUDA corpus) and only then + performance comparisons + ([`01-performance-correctness-security.md`](01-performance-correctness-security.md)). + +**6. Rollback.** Keep the previous version's directory and configuration +untouched. Rollback = check out the old anchor's commit/tag and reinstall the +old guest driver. Because versions live in parallel directories, a forward +attempt never destroys the working setup. + +**Common pitfalls (learned the hard way).** + +* **Runfile version drift**: distribution metapackages can resolve to a newer + major than requested. Check the candidate version before installing; prefer + the exact-version runfile for the guest. +* **Firmware filename mismatch**: a wrong-version firmware file is a probe + failure that looks like an emulation bug. +* **Address drift per build**: kernel text addresses differ per firmware + build; re-derive Ghidra addresses, never reuse them. +* **Symbol renames across OpenRM tags**: renamed exports break the optional + OpenRM build with `Unknown symbol` — check the `kernel-open` diff when that + path is in use. + +**Multi-version testing matrix (target state).** For each supported anchor +version × each hardware family, run the step-5 bring-up sequence. When adding +a version, extend the matrix rather than replacing the old column — +regressions across versions are the cheapest bugs to catch. ### 4.2 New guest driver version @@ -188,6 +335,23 @@ and ## 5. Crash and hang recovery +### 5.1 Failure taxonomy (symptom → likely cause) + +| Symptom | Likely cause | First check | +|---------|--------------|-------------| +| `insmod open-iov-vfio.ko: Unknown symbol` | missing prerequisite module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | +| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | +| Guest: no PCI device | mdev not attached | QEMU command line for `sysfsdev=` | +| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | +| Guest: probe hangs, no KVM warning | doorbell/RPC never consumed | host dmesg for the emulation thread and doorbell writes | +| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | +| Guest agent dead, QEMU alive | guest kernel wedge | gdbstub vCPU sampling (§3.7) | +| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | +| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | +| `pr_warn_ratelimited(... not supported in emulation mode)` | guest used a BAR_TRAP/MIGRATE RPC | expected fail-fast; wire a real handler if hot | + +### 5.2 Recovery actions + | Situation | Action | |---|---| | Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) | @@ -195,6 +359,28 @@ and | Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first | | Kernel oops in the host module | capture the full oops + `dmesg` before reloading | +### 5.3 Case study: the EPT livelock + +Symptoms: guest `modprobe nvidia` hangs; the guest agent dies; QEMU stays +alive. + +1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` + with `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. +2. **gdbstub**: one vCPU frozen at a single address in module text, identical + RIP/RSP across samples; the other vCPUs in `hlt`. The faulting instruction + was `mov 0x4(%rax),%r13d` — a memory read that can never complete. +3. **Root cause**: the VFIO `mmap` handler used `virt_to_phys()` on the + `vzalloc`'d BAR0 shadow → garbage PFN in the EPT → every guest BAR0 read + faulted → KVM retried forever. +4. **Fix**: per-page `vmalloc_to_pfn()` mapping. + +**Lesson: a KVM `make_spte` reserved-bits warning plus one frozen vCPU means a +bad PFN from your VFIO mmap handler.** Always question `virt_to_phys()` on +anything that is not lowmem; the module rule is recorded in +[`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md) +and the shadow mapping is described in +[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md). + ## 6. Release engineering * Versions are tagged; each tag carries the rung/matrix table and the diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index b8393f4..49c4bbe 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -27,7 +27,9 @@ | Other NVIDIA generations | **Experimental** | bring-up path documented in §5 | The defining property we are building toward: **each new host platform should -require configuration + verification, not new code.** +require configuration + verification, not new code.** That is the design +direction, not the current state — the profile files are not read by any code +yet and selection is compile-time (see §6). ## 3. Guest operating systems @@ -90,13 +92,31 @@ config/ versions/<driver>.yaml # RPC/control inventories, capability tables ``` -There is no `profiles/` directory today. Note also that the code currently -hardcodes the TU104 profile, so the YAML surface is the intended interface -rather than the path the module reads at load time — treating it as -authoritative is aspirational until the profiles are wired in. - -A profile is the unit of support: "hardware X + driver family Y + firmware Z = -profile P, tested at rung R on date D." Releases publish the profile table. +> **The `config/hardware/*.yaml` and `config/versions/*.yaml` files are +> design targets, not live configuration.** No code currently reads them: +> hardware/version selection is compile-time and the module is built for the +> TU104 profile. Treating the YAML surface as authoritative — or claiming +> "configuration-driven portability" — is aspirational until the profiles are +> wired into the module's load path. There is no `profiles/` directory today; +> `config/` is the only profile surface. + +**What "adding support" requires today** (TU104/570.144 is the reference): + +1. New GPU: edit the compile-time hardware constants/table for the target + (PCI IDs, BAR layout and sizes, memory geometry, engine inventory, + interrupt topology, probe-time register values) and rebuild the module; + the `config/hardware/*.yaml` file is documentation until the plumbing + exists. +2. New driver/firmware family: capture the init trace, diff the + RPC/control inventory, add version-keyed handlers, and register the + firmware signature for runtime identification; the `config/versions/*.yaml` + file is documentation until the plumbing exists. +3. Re-run the evaluation ladder and record the combination as + "hardware X + driver family Y + firmware Z, tested at rung R on date D." + Releases publish the table. + +A profile is the *unit of support* — what is missing is the code that reads +the profile files, not the profile schema. ### 6.1 Userspace driver facts @@ -131,6 +151,10 @@ start at | Guest-independent design | Windows and Linux guests share the same host code | | Firmware analysis toolchain is version-generic | new firmware is analyzed with the same scripts | +These are design properties, not a description of the current build: today the +module is compiled for one profile (TU104) and the `config/` YAML is not on the +load path (see §6). + ## 8. Known portability risks | Risk | Mitigation | diff --git a/docs/07-developers/onboarding-and-standards.md b/docs/07-developers/onboarding-and-standards.md index 3d7ac78..b3ac1c5 100644 --- a/docs/07-developers/onboarding-and-standards.md +++ b/docs/07-developers/onboarding-and-standards.md @@ -85,6 +85,41 @@ open-iov/ interface only. * Error paths: fail closed, log once with context, and give up the device cleanly. +* **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate + `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order). + +The following rules each come from a real bug and are enforced in review: + +* **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use + `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys() >> PAGE_SHIFT` + (see `nv_vgpu_mmap_virt_to_pfn`). A bad PFN in a VFIO mmap puts a garbage + PFN in the guest's EPT and wedges the guest in an EPT-violation livelock — + case study in + [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §5.3. +* **`dma_addr_t` for every DMA address.** Never truncate it into `int`/`u32`: + IOMMU IOVAs can exceed 32 bits. +* **Every `get` needs a `put`**: `pci_dev_get`/`pci_dev_put`, + `eventfd_ctx_fdget`/`eventfd_ctx_put`, `vfio_alloc_device`/`vfio_put_device`, + module references. Unbalanced gets become leaks that block unload. +* **Remove paths must fully unwind init paths** (including `list_del` and + kthread stop); a remove that forgets one leaves the module unloadable. +* **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from the + emulation path — it writes into the same mailbox and rings the same + doorbell our own thread polls, a self-triggered RPC livelock. Return + `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists. +* **Log through the NULL-tolerant `nv_vgpu_*` wrappers and rate-limit** + anything a guest can spam. +* `open-iov-vfio.ko` is **standalone**: no `MODULE_SOFTDEP`/dependency on + `nvidia` — `nvidia.ko` imports our symbols, not the other way around. +* VM access in tooling goes through `scripts/vgpu_vm.py`; do not invoke + ``/QEMU directly. +* Scripts never reboot or otherwise change system state without explicit + operator action (project hard rule). +* Do not widen an API's hardware assumptions silently: if a function needs + `vgpu->pdev` (or similar), say so in the comment — unit tests cannot call + those paths. +* The guest is stock: **no Open-IOV code in the guest, ever** — no guest + agent, no driver patch, no extra kernel parameters. ### 3.2 Userspace and tools (`userspace/`, `tools/`, `scripts/`) @@ -98,6 +133,9 @@ open-iov/ * Every behaviour change updates the doc that describes it. Docs are part of the definition of done. +* Debugging discoveries go into the failure taxonomy in + [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) + so the next person does not rediscover them. * Prefer a table or a decision tree over prose for anything an operator will follow under pressure. * Protocol facts (IDs, offsets, structures) must cite their source (a live diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md deleted file mode 100644 index 309a64f..0000000 --- a/docs/BUILD_NVIDIA_OPENRM.md +++ /dev/null @@ -1,123 +0,0 @@ -# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols - -> **Status: historical build notes.** These notes predate the current workflow. -> For the current build/load model see -> [03-architecture/06-host-module-and-hybrid-model.md](03-architecture/06-host-module-and-hybrid-model.md) -> and [02-getting-started/01-install-and-first-run.md](02-getting-started/01-install-and-first-run.md). - -## Overview - -The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: -- `nvidia_vgpu_vfio_probe` -- `nvidia_vgpu_vfio_remove` -- `nvidia_vgpu_vfio_get_ops` -- `nvidia_vgpu_vfio_set_ops` -- `nv_vgpu_mdev_register_parent` -- `nv_vgpu_mdev_unregister_parent` -- And all GSP RPC integration symbols - -These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`. - -## Build Order (Critical) - -``` -1. Build open-iov-vfio.ko → Generates kernel/Module.symvers -2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers -3. Load open-iov-vfio.ko first (exports the vGPU VFIO symbols) -4. Optionally load nvidia.ko (imports those symbols) -``` - -## Automated Build (Recommended) - -Use the helper script: - -```bash -cd open-iov -./scripts/build_nvidia_openrm.sh -``` - -This script: -1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`) -2. Cleans previous NVIDIA OpenRM build -3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set -4. Verifies vGPU symbols are resolved - -## Manual Build - -```bash -# Step 1: Build Open-IOV kernel module -cd $OPEN_IOV_ROOT -make kernel - -# Step 2: Build NVIDIA OpenRM with extra symbols -cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules - -# Step 3: Verify symbols resolved -grep -i vgpu Module.symvers -``` - -## Verification - -After building, verify the symbols are resolved: - -```bash -# Check Module.symvers has vGPU symbols -grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers - -# Check nvidia.ko modinfo -modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu -``` - -## Common Issues - -| Issue | Cause | Fix | -|-------|-------|-----| -| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` | -| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first | -| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild | -| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config | - -## Module Loading Order - -```bash -# 1. Load Open-IOV VFIO driver FIRST (exports the vGPU VFIO symbols and -# registers the mdev parent on the physical GPU) -sudo insmod kernel/open-iov-vfio.ko - -# 2. Load the optional NVIDIA OpenRM (imports the vGPU VFIO symbols; build it -# with KBUILD_EXTRA_SYMBOLS as above) -sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko - -# 3. Copy firmware -sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ - -# 4. Create mdev device -./scripts/create_mdev.sh - -# 5. Launch QEMU with vGPU -./scripts/launch_qemu_vgpu.sh -``` - -## Key Files - -| File | Purpose | -|------|---------| -| `scripts/build_nvidia_openrm.sh` | Automated build script | -| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko | -| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment | -| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild | - -## Troubleshooting - -If symbols are still unresolved after rebuild: - -```bash -# Full clean rebuild -cd nvidia-open-gpu-kernel-modules/kernel-open -make clean -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules - -# Check modpost output for unresolved symbols -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu" -```
\ No newline at end of file diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md deleted file mode 100644 index e2d7fe0..0000000 --- a/docs/DEBUGGING.md +++ /dev/null @@ -1,196 +0,0 @@ -# Open-IOV Debugging Guide - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Debugging -> replacement: -> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md); -> the §3–§4 case studies here remain useful. - -> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest -> driver failures, and validating a fully working setup. Built from real -> debugging sessions (including the 2026-09-21 EPT-livelock hunt). - ---- - -## 1. Debug surfaces (know them all) - -| Surface | Where | What it shows | -|---------|-------|---------------| -| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns | -| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead | -| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** | -| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status | -| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** | -| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer | -| GDB on host kernel | (optional) | host-side driver state | - -The VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain: - -```yaml -serialLog: "/tmp/vgpu-guest-serial.log" -gdbPort: 1234 -gdbFreeze: false # true = halt at reset vector (boot debugging) -``` - ---- - -## 2. The golden rule of failure triage - -**When the guest agent dies mid-test, the guest is wedged in kernel mode. -Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.** - -Triage order: -1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages -2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words -3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck? - ---- - -## 3. QEMU GDB stub workflow - -Attach (no symbols needed for triage): - -```bash -gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ - -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ - -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' -``` - -Reading the result: - -| Observation | Meaning | -|-------------|---------| -| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine | -| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver | -| RIPs moving between samples | guest alive and progressing | - -For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical -`$rip` **and** `$rsp` every time = the *same instruction* faults/retries -(EPT-violation livelock) rather than a loop making progress. - -With guest symbols (`vmlinux` + module `.ko` with debug info), add -`add-symbol-file` and get real function names. - -### QMP NMI (soft alternative) - -```bash -(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ - echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ - | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock -``` - -Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU -(spinning with a faulting MMIO instruction) may not print anything — use the -GDB stub instead. - ---- - -## 4. Case study: the 2026-09-21 EPT livelock - -Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive. - -1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with - `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. -2. **GDB stub**: vCPU2 frozen at one address in module text - (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in - `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that - can never complete. -3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0 - shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM - retried forever. -4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see - `docs/GSP_EMULATION_ARCHITECTURE.md` §2. - -Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from -your VFIO mmap handler.** Always question `virt_to_phys()` on anything that -isn't lowmem. - ---- - -## 5. What a *successful* driver bring-up looks like (validation checklist) - -After `stack_manager.sh load-stack` on the host: - -- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers -- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1** -- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg - -After `vgpu_vm.py start vgpu-linux`: - -- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>` -- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started` -- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored) -- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped) -- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root` - -When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`): - -- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first) -- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment) -- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam -- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found") -- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done -- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144** - -Teardown regression: - -- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1** -- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread) -- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone - -Kernel unit test: - -- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED` -- [ ] `sudo rmmod gsp_emulation_test` → clean - ---- - -## 6. Failure taxonomy (symptom → likely cause) - -| Symptom | Likely cause | First check | -|---------|--------------|-------------| -| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | -| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | -| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` | -| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | -| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes | -| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | -| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) | -| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | -| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | -| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot | - ---- - -## 7. Host-guest version matrix discipline - -Probe failures caused by **version mismatch** look identical to emulation bugs. -Before deep debugging, verify the matrix: - -| Component | Must be | Check | -|-----------|---------|-------| -| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) | -| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) | -| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` | -| Emulation target | 570.144W | decompiled firmware basis | - -Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs -570.144 emulation cost a full session before the real mmap bug was found — -the mismatch masked the signal.) - ---- - -## 8. Quick reference - -```bash -# Stack -./scripts/stack_manager.sh status|load-stack|unload|dmesg -# VM (harness ONLY - never raw /qemu) -./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell -# Debug -tail -f /tmp/vgpu-guest-serial.log & -gdb -ex 'target remote localhost:1234' -sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov' -# Recover wedged VM -./scripts/vgpu_vm.py stop <uuid> # or: sudo kill -9 <qemu pid> -``` diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md deleted file mode 100644 index 5d43896..0000000 --- a/docs/DEVELOPMENT_WORKFLOW.md +++ /dev/null @@ -1,681 +0,0 @@ -# Open-IOV vGPU-IO Development Workflow & Tools Guide - -> **Historical document — do not follow as written (2026-09-21).** This guide -> predates the current workflow: it contains the old nvidia-first load order, -> reboot advice, and LFS instructions that no longer apply (LFS was removed; -> firmware and decompiled artifacts are local/gitignored). The current -> workflow is -> [`../scripts/stack_manager.sh`](../scripts/stack_manager.sh), -> [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) -> and -> [`02-getting-started/01-install-and-first-run.md`](02-getting-started/01-install-and-first-run.md). - -## Overview - -This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging. - ---- - -## 1. Repository Structure & Navigation - -### Primary Workspace -```bash -$OPEN_IOV_ROOT/ # Main monorepo (git + LFS) -``` - -### Key Directories -| Directory | Purpose | -|-----------|---------| -| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) | -| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) | -| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` | -| `userspace/examples/` | `example_vgpu` test app | -| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files | -| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) | -| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) | -| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent | -| `docs/` | Architecture, implementation guides, TODO lists | -| `wiki/` | 10 technical wiki pages | - ---- - -## 2. Shell Integration (zshrc) - -### Required Aliases & Functions -Add to `~/.zshrc` and run `source ~/.zshrc`: - -```bash -# Project navigation -alias oiv="cd $OPEN_IOV_ROOT" -alias oivk="cd $OPEN_IOV_ROOT/kernel" -alias oivu="cd $OPEN_IOV_ROOT/userspace" -alias oivf="cd $OPEN_IOV_ROOT/firmware/57X-GSP/570.144" -alias oivd="cd $OPEN_IOV_ROOT/docs" -alias oivs="cd $OPEN_IOV_ROOT/scripts" -alias oivt="cd $OPEN_IOV_ROOT/tools" - -# Build commands -alias oiv-build="cd $OPEN_IOV_ROOT && make -j$(nproc)" -alias oiv-build-kernel="cd $OPEN_IOV_ROOT && make kernel" -alias oiv-build-userspace="cd $OPEN_IOV_ROOT && make userspace" -alias oiv-clean="cd $OPEN_IOV_ROOT && make clean" - -# Module loading (requires reboot first!) -alias oiv-load="$OPEN_IOV_ROOT/scripts/load_vgpu_stack.sh" -alias oiv-unload="$OPEN_IOV_ROOT/scripts/unload_vgpu_stack.sh" -alias oiv-test="$OPEN_IOV_ROOT/scripts/test_vgpu_stack.sh" -alias oiv-test-gsp="$OPEN_IOV_ROOT/scripts/test_gsp_emu.sh" -alias oiv-test-guest="$OPEN_IOV_ROOT/scripts/test_guest_driver.sh" -alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh" -alias oiv-test-graphics="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh" -alias oiv-test-windows="$OPEN_IOV_ROOT/scripts/test_windows_guest.sh" - -# mdev management -alias oiv-mdev-create="$OPEN_IOV_ROOT/scripts/create_mdev.sh" -alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'" - -# QEMU -alias oiv-qemu="$OPEN_IOV_ROOT/scripts/launch_qemu_vgpu.sh" -alias oiv-qemu-debug="$OPEN_IOV_ROOT/scripts/debug_guest_init.sh" - -# Firmware analysis -alias oiv-ghidra="cd $OPEN_IOV_ROOT/tools/ghidra && ./ghidraRun" -alias oiv-fw-analyze="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py" - -# Debugging tools (NEW) -alias oiv-guest-monitor="python3 $OPEN_IOV_ROOT/tools/guest_init_monitor.py" -alias oiv-failure-analyzer="python3 $OPEN_IOV_ROOT/tools/failure_analyzer.py" -alias oiv-benchmark="$OPEN_IOV_ROOT/tools/benchmark_infrastructure.sh" -alias oiv-windows-test="$OPEN_IOV_ROOT/scripts/windows_guest_automation.sh" - -# Kernel debugging -alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" -alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" -alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic" -alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu" - -# Git -alias oiv-git="cd $OPEN_IOV_ROOT && git status" -alias oiv-git-log="cd $OPEN_IOV_ROOT && git log --oneline -20" -alias oiv-git-diff="cd $OPEN_IOV_ROOT && git diff" - -# Restore point -alias oiv-restore="cd $OPEN_IOV_ROOT && git checkout f9ed5d8 -- . && git clean -fd" - -# Help -oiv-help() { - echo "╔═══════════════════════════════════════════════════════════════════════════════╗" - echo "║ Open-IOV vGPU-IO Development Commands ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ NAVIGATION ║" - echo "║ oiv - cd to open-iov root ║" - echo "║ oivk - cd to kernel/ ║" - echo "║ oivu - cd to userspace/ ║" - echo "║ oivf - cd to firmware/57X-GSP/570.144/ ║" - echo "║ oivd - cd to docs/ ║" - echo "║ oivs - cd to scripts/ ║" - echo "║ oivt - cd to tools/ ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ BUILD ║" - echo "║ oiv-build - Build everything (kernel + userspace + examples) ║" - echo "║ oiv-build-kernel - Build open-iov-vfio.ko only ║" - echo "║ oiv-build-userspace - Build libiov.so + example_vgpu ║" - echo "║ oiv-clean - Clean all build artifacts ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ MODULE OPERATIONS (require root) ║" - echo "║ oiv-load - Load nvidia.ko → open-iov-vfio.ko (refactored stack) ║" - echo "║ oiv-unload - Unload all modules + remove mdevs ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ TESTING ║" - echo "║ oiv-test - Run comprehensive test suite (test_vgpu_stack.sh) ║" - echo "║ oiv-test-gsp - Run GSP emulation unit tests (test_gsp_emu.sh) ║" - echo "║ oiv-test-guest - Test guest driver integration (test_guest_driver.sh)║" - echo "║ oiv-test-cuda - Test CUDA workloads (test_cuda_workloads.sh) ║" - echo "║ oiv-test-graphics - Test graphics workloads (test_graphics_workloads.sh)║" - echo "║ oiv-test-windows - Test Windows guest (test_windows_guest.sh) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ MDEV MANAGEMENT ║" - echo "║ oiv-mdev-create - Create mdev device for QEMU ║" - echo "║ oiv-mdev-list - List existing mdev devices ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ QEMU ║" - echo "║ oiv-qemu - Launch QEMU with vGPU ║" - echo "║ oiv-qemu-debug - Launch QEMU with GDB + guest monitor ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ DEBUGGING ║" - echo "║ oiv-dmesg - Filtered dmesg for NVIDIA/GSP/vGPU/mdev ║" - echo "║ oiv-dmesg-follow - Live filtered dmesg (tail -f) ║" - echo "║ oiv-dmesg-errors - Show only errors/warnings ║" - echo "║ oiv-trace - Live kernel trace_pipe filtered for gsp_emu ║" - echo "║ oiv-guest-monitor - Monitor guest init (correlates host traces + serial) ║" - echo "║ oiv-failure-analyzer - Analyze/classify guest init failures ║" - echo "║ oiv-benchmark - Run VM vs bare metal benchmarks ║" - echo "║ oiv-windows-test - Windows guest test automation (WinRM) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ GIT ║" - echo "║ oiv-git - Git status in open-iov ║" - echo "║ oiv-git-log - Git log (last 20 commits) ║" - echo "║ oiv-git-diff - Show uncommitted changes ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ FIRMWARE ANALYSIS ║" - echo "║ oiv-ghidra - Launch Ghidra GUI ║" - echo "║ oiv-fw-analyze - Run firmware analyzer (Python) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ EMERGENCY ║" - echo "║ oiv-restore - Restore to commit f9ed5d8 (restore point) ║" - echo "╠═══════════════════════════════════════════════════════════════════════════════╣" - echo "║ oiv-help - This help ║" - echo "╚═══════════════════════════════════════════════════════════════════════════════╝" -} -``` - -### Usage -```bash -# After adding to ~/.zshrc: -source ~/.zshrc -oiv-help # Show all commands -oiv # Jump to project root -oiv-build # Build everything -oiv-load # Load kernel modules -oiv-mdev-create # Create mdev for QEMU -oiv-qemu # Launch QEMU guest -oiv-dmesg-follow # Live kernel logs -``` - ---- - -## 3. Build Workflow - -### Full Build -```bash -cd $OPEN_IOV_ROOT -make # Builds kernel + userspace + examples -# OR -oiv-build -``` - -### Incremental Builds -```bash -# Kernel only (open-iov-vfio.ko) -make kernel -# OR -cd kernel && make -j$(nproc) - -# Userspace only (libiov.so + libvgpu.so + example_vgpu) -make userspace -# OR -cd userspace && make -j$(nproc) - -# Examples only -make examples - -# Tests -make tests - -# Clean -make clean -# OR -oiv-clean -``` - -### Build Outputs -| Artifact | Location | Size | -|----------|----------|------| -| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB | -| `open-iov-vfio.ko` | `kernel/` | ~3.8MB | -| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB | -| `example_vgpu` | `userspace/examples/` | 27KB | - ---- - -## 4. Kernel Module Loading - -### Prerequisites -- The **stock Ubuntu nvidia driver must not be loaded** when you want our - OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on - desktop boots (udev) and is held open by the GUI. - `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this - machine) prevents that auto-load after reboot; `insmod` is unaffected. -- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia - (compares `/sys/module/nvidia/version` with our build) and refuses with - guidance instead of silently skipping. - -### Load Order (Critical) -```bash -# 1. VFIO/mdev prerequisites -sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev - -# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia -# imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the -# mdev parent on the physical GPU. -sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko - -# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols -# (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers) -sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko - -# 4. GSP firmware for the host GPU -sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ -``` - -### Using Helper Script -```bash -./scripts/stack_manager.sh load-stack # prereqs → open-iov-vfio → nvidia → firmware -``` - -### Verify Load -```bash -lsmod | grep -E "nvidia|open_iov" -dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/ -cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances -``` - -### Unload -```bash -oiv-unload # Removes mdevs, unloads modules in reverse order -``` - ---- - -## 5. mdev Device Management - -### Create mdev (for QEMU) -```bash -oiv-mdev-create -# Interactive: selects GPU, vGPU type, generates UUID -# Creates: /sys/bus/mdev/devices/<UUID>/ -``` - -### Manual Creation -```bash -# List GPUs -lspci -D -d 10de: | grep -E "VGA|3D" - -# List supported vGPU types -ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/ - -# Create specific type -echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create -``` - -### List mdevs -```bash -oiv-mdev-list -# OR -ls -la /sys/bus/mdev/devices/ -``` - -### Remove mdev -```bash -echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove -``` - ---- - -## 6. QEMU Guest Launch - -### Using Helper Script -```bash -oiv-qemu -# Interactive: selects mdev, guest image, launches QEMU -``` - -### Manual Launch -```bash -qemu-system-x86_64 \ - -enable-kvm \ - -machine q35,accel=kvm,kernel_irqchip=split \ - -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \ - -smp 8,sockets=1,cores=8,threads=1 \ - -m 16G,slots=4,maxmem=64G \ - -mem-prealloc \ - -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \ - -numa node,memdev=mem \ - -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \ - -vga none -display none \ - -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \ - -netdev user,id=net0,hostfwd=tcp::2222-:22 \ - -device virtio-net-pci,netdev=net0 \ - -usb -device usb-tablet \ - -name "vGPU-Guest" \ - -pidfile /tmp/qemu-vgpu.pid \ - -daemonize -``` - -### Connect to Guest -```bash -# SSH (port 2222 forwarded) -ssh -p 2222 user@localhost - -# VNC (if enabled in launch script) -vncviewer localhost:1 -``` - ---- - -## 7. Guest Driver Setup - -### Inside Guest VM -```bash -# Run setup script (installs NVIDIA driver, firmware, loads module) -sudo $OPEN_IOV_ROOT/scripts/setup_guest_driver.sh - -# Or manual: -# 1. Install NVIDIA driver (OpenRM 570.144) -# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/ -# 3. modprobe nvidia -# 4. nvidia-smi -``` - -### Verify Guest -```bash -nvidia-smi # Should show vGPU (T4/A100/etc.) -lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU) -dmesg -T | grep -i nvidia # Check probe success -``` - ---- - -## 8. Testing Workflow - -### Host Test Suite -```bash -oiv-test -# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification -``` - -### Guest CUDA Tests -```bash -# Inside guest: -cd /tmp -git clone https://github.com/NVIDIA/cuda-samples.git -cd cuda-samples/Samples/1_Utilities/vectorAdd -make -./vectorAdd -``` - -### Guest Graphics Tests -```bash -# Inside guest: -vulkaninfo -vkcube -glxgears -glmark2 -``` - -### Guest Video Tests -```bash -# Inside guest: -ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4 -ffmpeg -hwaccel cuda -i test.mp4 -f null - -``` - -### Windows Guest Tests -```powershell -# Inside Windows guest: -# 1. Install NVIDIA driver (570.144) -# 2. nvidia-smi.exe -# 3. dxdiag -# 4. Run CUDA-Z / GPU-Z -# 5. Run 3DMark / FurMark -``` - ---- - -## 9. Firmware Reverse Engineering Workflow - -### Analyze New Firmware -```bash -# 1. Download firmware from NVIDIA OpenRM release -# 2. Place in firmware dir -cp new-gsp.bin $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin - -# 3. Run Ghidra headless analysis -cd $OPEN_IOV_ROOT/tools/ghidra -./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \ - -import $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin \ - -processor "RISCV:RV64:default" \ - -postScript DecompileFunctions.java \ - -deleteProject - -# 4. Run firmware analyzer (auto-discovers addresses) -cd $OPEN_IOV_ROOT -python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json - -# 5. Compare with previous version -python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json - -# 6. Generate config if new version -python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml -``` - -### Ghidra Scripts (in tools/ghidra/) -| Script | Purpose | -|--------|---------| -| `DecompileFunctions.java` | Batch decompile all functions | -| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table | -| `ExtractStructLayouts.java` | Extract struct definitions from prologues | -| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching | - -### Key Firmware Addresses (570.144W) -| Component | Address | Description | -|-----------|---------|-------------| -| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands | -| Secondary Dispatcher | 0x04ED39DC | Additional routing | -| BAR Trap Handler | 0x04CB8188 | 40KB | -| WPR Handler 1 | 0x04F00B0C | 62KB | -| WPR Handler 2 | 0x04F00C78 | 62KB | -| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers | -| ALLOC Main | 0x057E94EC | 40KB allocator | -| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler | -| Entry Point | 0x04009000 | Bootstrap | - ---- - -## 10. Debugging Workflows - -### Guest Driver Probe Issues -```bash -# Host: Watch live logs -oiv-dmesg-follow - -# Guest: Check PCI config -lspci -vvv -d 10de: -cat /proc/bus/pci/devices - -# Guest: Check BAR mapping -lspci -v -d 10de: | grep -A 10 "Memory at" - -# Guest: Check MSI-X -lspci -vvv -d 10de: | grep -A 20 "MSI-X" -``` - -### GSP Emulation Debugging -```bash -# Check emulation thread -ps aux | grep gsp_emu - -# Check RPC processing -dmesg -T | grep -i "gsp_emu\|rpc" - -# Debugfs stats -cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats - -# Add trace to gsp_emulation.c: -trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))); -cat /sys/kernel/debug/tracing/trace -``` - -### QEMU with GDB (Guest Kernel Debugging) -```bash -# Launch QEMU with GDB server -qemu-system-x86_64 -s -S ... # -s = gdbserver on :1234, -S = freeze - -# In another terminal: -gdb ./vmlinux -(gdb) target remote :1234 -(gdb) break nvidia_probe -(gdb) continue -``` - -### NEW: Comprehensive Debug Session (One Command) -```bash -# Launch debug session (QEMU + GDB + guest monitor) -./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux - -# Or manually: -# Terminal 1: QEMU with GDB -./scripts/launch_qemu_vgpu.sh --gdb --freeze - -# Terminal 2: Guest init monitor -python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300 - -# Terminal 3: GDB -gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux -(gdb) target remote localhost:1234 -``` - -### NEW: Failure Analysis -```bash -# After a failed guest init, analyze logs -python3 tools/failure_analyzer.py \ - --guest-log /tmp/guest_serial.log \ - --host-trace /tmp/host_trace.log \ - --qemu-log /tmp/qemu.log \ - --output /tmp/failure_report.json - -# This classifies the failure and provides specific recommendations -``` - -### NEW: Benchmarking -```bash -# Run VM vs bare metal benchmarks -./tools/benchmark_infrastructure.sh all - -# Or specific categories -./tools/benchmark_infrastructure.sh cuda -./tools/benchmark_infrastructure.sh graphics -./tools/benchmark_infrastructure.sh pytorch -``` - -### Performance Profiling -```bash -# GSP RPC latency -# Add to gsp_emulation.c: -trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))) - -# View trace -cat /sys/kernel/debug/tracing/trace - -# CUDA kernel launch overhead (in guest) -nvprof --profile-from-start off ./vectorAdd - -# MSI-X injection latency -# Measure in gsp_emu_inject_completion_interrupt() -``` - ---- - -## 11. Git Workflow - -### Restore Point -```bash -oiv-restore # git checkout f9ed5d8 -- . && git clean -fd -``` - -### Branch for Refactor -```bash -cd $OPEN_IOV_ROOT -git checkout -b refactor/host-gsp-emulation -# Implement changes per docs/refactored-implementation-to-do.md -``` - -### Commit Changes -```bash -git add . -git commit -m "Description of changes" -``` - -### LFS for Large Files -```bash -git lfs ls-files # Shows tracked large files -git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o" -``` - ---- - -## 12. Common Issues & Solutions - -| Issue | Solution | -|-------|----------| -| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist | -| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps | -| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap | -| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory | -| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` | -| Migration fails | State mismatch - checksum all migrated structures | -| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table | -| Build fails | `make clean && make` or check kernel headers match running kernel | - ---- - -## 13. Extending the Shell Environment - -### Adding New Helpers -When you create new scripts or workflows, add them to `~/.zshrc`: - -```bash -# Example: Add new test command -alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh" -alias oiv-test-gfx="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh" - -# Example: Add firmware analysis shortcut -alias oiv-fw-new="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin" - -# Reload -source ~/.zshrc -``` - -### Document New Helpers -Update `oiv-help()` function in `~/.zshrc` to include new commands. - ---- - -## 14. Key Files Reference - -| File | Purpose | -|------|---------| -| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps | -| `AGENTS.md` | Agent brain - workflow, tools, architecture | -| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design | -| `docs/refactored-architecture.md` | Host-only emulation design | -| `docs/refactored-implementation-to-do.md` | Granular implementation TODO | -| `scripts/load_vgpu_stack.sh` | Load all kernel modules | -| `scripts/create_mdev.sh` | Create mdev device | -| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU | -| `scripts/test_vgpu_stack.sh` | Comprehensive test suite | -| `scripts/debug_guest_init.sh` | NEW: One-command debug launch | -| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation | -| `tools/guest_init_monitor.py` | NEW: Guest init monitor | -| `tools/failure_analyzer.py` | NEW: Failure analyzer | -| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner | -| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent | -| `kernel/gsp_emulation.c/h` | GSP emulation engine | -| `kernel/firmware_analyzer.c/h` | Firmware analyzer | -| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher | -| `kernel/architecture/architecture.c/h` | Hardware abstraction | -| `kernel/version/version.c/h` | Version adaptation | -| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests | -| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers | -| `kernel/gsp_rpc_integration.c` | Integration layer | -| `kernel/main.c` | Module entry, mdev registration | -| `kernel/bar_emulation.c` | BAR + MSI-X emulation | -| `userspace/lib/libvgpu.c` | Userspace API | -| `userspace/lib/rpc_emulator.c` | Emulation fallbacks | - ---- - -*Development Workflow Guide Version: 1.0* -*Repository: $OPEN_IOV_ROOT* -*Last Updated: $(date)*
\ No newline at end of file diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md deleted file mode 100644 index 70b4b76..0000000 --- a/docs/GSP_EMULATION_ARCHITECTURE.md +++ /dev/null @@ -1,210 +0,0 @@ -# Open-IOV GSP Emulation Engine — Architecture - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Architecture -> replacement: -> [03-architecture/01](03-architecture/01-system-overview.md) and -> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md); -> the transport section in particular is superseded by the msgq -> documentation in -> [03-architecture/03](03-architecture/03-gsp-and-gsp-rm.md). - -> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, -> `kernel/bar_emulation.c`, or debugging guest driver bring-up. -> -> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21 -> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`; -> for the original refactor plan see `refactored-architecture.md`. - ---- - -## 1. Purpose - -An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects -to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC -messages to a mailbox in BAR0, rings a doorbell, and waits for a completion -interrupt plus a response written back into the mailbox. - -Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the -host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the -host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes -RPCs, performs them in software (or declines them), writes responses back, and -injects MSI-X interrupts. - -``` -┌──────────────────────────── Guest VM ────────────────────────────┐ -│ stock nvidia.ko (570.144) │ -│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │ -│ ▼ │ -│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │ -└──────────────────────────────────────────────────────────────────┘ - │ (16 MB vzalloc'd shadow, uncached) -┌──────────────────────────── Host ────────────────────────────────┐ -│ gsp_emu thread (open-iov-vfio.ko) │ -│ poll shadow+0x110C00 (doorbell) │ -│ → read RPC msg from shadow+0x1000 │ -│ → dispatch (12 RPC types, emulation implementations) │ -│ → write response to shadow+0x1080 │ -│ → set PBA bit at shadow+0x3000 │ -│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │ -└──────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 2. The single source of truth: `bar0_shadow` - -The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software -shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()` -(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler -(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`). - -Everything the guest driver does with BAR0 lands in this shadow: - -| BAR0 offset | Region | Guest writes go to | Emulation reads from | -|------------------|-----------------------------------|--------------------|----------------------| -| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow | -| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` | -| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads | -| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) | -| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads | -| `0x110C00` | **GSP doorbell** | shadow | thread polls this | -| rest | emulated register space | shadow | shadow | - -> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled -> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real -> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer — -> it could never see guest traffic and even wrote PBA bits into the real GPU's -> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`. -> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so -> cleanup never `dma_free`s it. - -### 2.1 Transport protocol (as emulated) - -1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000` - (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`). -2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`. -3. The emulation thread sees the doorbell, ACKs it (writes 0), validates - `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`. -4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`, - `result[6]`, `seq_id`) written to `BAR0+0x1080`. -5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired - eventfd → QEMU delivers MSI-X to the guest. -6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and - accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the - public `gsp_emu_process_rpc()` entry does not bump counters). - ---- - -## 3. RPC dispatch - -`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls -the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation -implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by -`gsp_emu_init`). - -| Type (val) | Handler path in emulation | Notes | -|----------------|--------------------------------------------------------|-------| -| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list | -| FREE (0x02) | `gsp_rpc_free_emul` | | -| MAP (0x03) | `gsp_rpc_map_emul` | | -| UNMAP (0x04) | `gsp_rpc_unmap_emul` | | -| PIN (0x05) | `gsp_rpc_pin_emul` | | -| UNPIN (0x06) | `gsp_rpc_unpin_emul` | | -| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission | -| YIELD (0x08) | no-op success | | -| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet | -| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO | -| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO | -| DEBUG (0xFF) | log-only in dispatch | | -| unknown | `-ENOTSUPP` | | - -> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to -> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox -> and **rings the same doorbell the thread polls** — a self-triggered infinite -> RPC loop plus multi-second stalls. Emulation mode now fails these fast with -> `pr_warn_ratelimited`. - -### Memory model (corrected 2026-09-22) - -Guest BAR1 (VRAM aperture) is backed by a **dedicated host-RAM vzalloc buffer** -(256 MB, `bar_init.c`), NOT the host GPU's physical BAR1. Guest VRAM accesses -hit our clean host-RAM backing. This is a placeholder — for real CUDA compute, -the memory model must evolve (see `refactored-architecture.md` Phase 2): - -- **Current (Phase 1)**: Guest VRAM = host RAM (safe, no host GPU corruption). - Guest driver probes but cannot compute. -- **Phase 2**: Guest VRAM backed by DMA-coherent host memory or physical GPU - VRAM partition (required for CUDA). -- **Phase 3**: Guest channel/pushbuffer work forwarded to physical GPU - channels (required for compute). - -Guest BAR3 (instance memory window) is also a **vzalloc'd shadow** (32 MB, -`bar_init.c`), NOT the physical GPU's BAR3. Guest BAR3 writes must not reach -the host GPU's instance memory (root cause of host GSP crash, fixed `85d0668`). - ---- - -## 4. Interrupt delivery (MSI-X) - -1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`. - Our ioctl handler takes `eventfd_ctx_fdget()` references into - `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs). -2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`: - sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger - from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after* - `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a - real bug fixed 2026-09-21). - -INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator -currently fires vector 0 for GSP completion. - ---- - -## 5. Lifecycle - -``` -mdev create (nv_vgpu_mdev_create) - └─ schedule init_work - └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow) - └─ nv_vgpu_irq_init / nv_vgpu_memory_init - └─ gsp_emu_init - ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases) - ├─ gsp_client_init_emulation (RPC client, emulation_mode=true) - ├─ gsp_rpc_emulation_set(true) - └─ kthread_run(gsp_emu_thread_fn) - -mdev remove (nv_vgpu_mdev_remove) - └─ cancel init_work, wait for VFIO fds to close - └─ vfio_unregister_group_dev - └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion) - → memory → irq → BAR (frees bar0_shadow) - └─ list_del, pci_dev_put, vfio_put_device -``` - ---- - -## 6. Debugging the emulator - -- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'` -- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on - the thread path only) -- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub - `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation - checklist. -- Unit test: `kernel/tests/gsp_emulation_test.ko` drives - `gsp_emu_process_rpc()` with crafted mailbox messages (requires - `open-iov-vfio.ko` loaded; `depends:` is set accordingly). - ---- - -## 7. Key files - -| File | Role | -|------|------| -| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection | -| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs | -| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state | -| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space | -| `kernel/memory.c` | simple mapping/pin tracking (host management API) | -| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) | diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md deleted file mode 100644 index 615e016..0000000 --- a/docs/ONBOARDING.md +++ /dev/null @@ -1,171 +0,0 @@ -# Open-IOV Developer Onboarding - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Onboarding -> replacement: -> [07-developers/onboarding-and-standards.md](07-developers/onboarding-and-standards.md). - -> Welcome. This guide gets you from zero to a productive contributor in ~30 -> minutes: what the project is, how the repo is laid out, how to build and -> test, and the standards we hold the code to. - ---- - -## 1. What this project is (60 seconds) - -**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver** -runs in a VM with full GPU functionality — no GRID license. The trick: the -host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the -firmware processor the driver talks to) in software, so the guest's stock -`nvidia.ko` can't tell it isn't running on real hardware. - -Everything guest-facing is **host-side only**. There is no Open-IOV code in -the guest. - -How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W -firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types -mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md` -for how the emulator works. - -## 2. Repository tour - -``` -open-iov/ -├── Makefile # top-level: all/kernel/userspace/examples/tests/clean -├── install.sh # fresh-machine reproduction (deps→nvidia→firmware→build→load) -├── README.md # project intro + quick start -├── STATE.md # live project state (works/limitations/bugs) -├── kernel/ # ★ the host driver (open-iov-vfio.ko) -│ ├── main.c # module init, mdev parent/driver, VFIO ops, PCI config -│ ├── gsp_emulation.c/h # GSP emulation engine (thread, dispatch, MSI-X) -│ ├── gsp_rpc_client.c/h # RPC API + emulation implementations + msg structs -│ ├── bar_emulation.c/h # BAR0 shadow + MSI-X shadow state -│ ├── bar_init.c # physical BAR mapping (host side) -│ ├── memory.c # simple mapping/pin tracking -│ ├── irq.c # IRQ/eventfd plumbing -│ ├── ioctl.c # host management ioctls (libiov consumers) -│ ├── migration.c # 4-phase live-migration skeleton -│ ├── firmware_analyzer.c/h # runtime firmware signature/pattern analysis -│ ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch -│ ├── architecture/ # hw abstraction (Turing/Ampere/Blackwell) -│ ├── version/ # driver-version adaptation (550/560/570/580) -│ ├── include/ # kernel headers (nvidia-vgpu-vfio.h is the main one) -│ └── tests/ # kernel unit tests (gsp_emulation_test) -├── userspace/ -│ ├── lib/ # libiov.so (+ libvgpu.so compat) management API -│ ├── examples/ # example_vgpu test app -│ └── tests/ -├── legacy-driver-guest/ # historical guest-side driver (reference only, NOT built) -├── firmware/57X-GSP/570.144/ # GSP blob + analysis scripts (per-version dirs) -├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko) -├── legacy-patches/nvidia/ # our delta vs upstream OpenRM tag -├── scripts/ # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers -├── tools/ # guest monitor, failure analyzer, benchmarks, ghidra -├── config/ # declarative YAML: hardware/, versions/ -├── docs/ # ★ all developer documentation (start here) -└── wiki/ # firmware analysis pages (RPC format, addresses, ...) -``` - -## 3. Your first build & test - -```bash -cd open-iov -./scripts/stack_manager.sh build-all # kernel → nvidia → userspace (ORDER MATTERS) -sudo ./scripts/stack_manager.sh load-stack -sudo ./scripts/stack_manager.sh mdev-create -./scripts/stack_manager.sh status -``` - -Rules that will save your week (also enforced in `stack_manager.sh`): - -1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`. - nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`. -2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware. -3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`. -4. **Version matrix**: guest driver == host OpenRM == host GSP firmware == - emulation target (all **570.144** today). Mismatches look exactly like - emulation bugs — check versions first (`docs/DEBUGGING.md` §7). -5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness). - Never raw `` or `qemu-system-*` commands. - -Run the unit tests (host, needs the module loaded): - -```bash -make tests -sudo insmod kernel/tests/gsp_emulation_test.ko # dmesg → ALL TESTS PASSED -sudo rmmod gsp_emulation_test -``` - -## 4. Coding standards - -We follow the **Linux kernel coding style** for kernel code and common-sense -C for userspace. The practical checklist: - -### Must -- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the - surrounding file. -- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate - `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order). -- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use - `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT` - (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a - whole session — it is called out in code review. -- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32` - (IOMMU IOVAs can exceed 32 bits). -- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`, - `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`, - module refs. Unbalanced gets become leaks that block unload. -- **Remove paths must fully unwind init paths** (we had a remove that forgot - `list_del` + kthread stop + VFIO unregister — leaked everything). -- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation - (it rings the doorbell our own thread polls → self-trigger livelock). - Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists. -- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant); - rate-limit anything a guest can spam. -- Comments explain **why**, not what — especially around hardware/firmware - behavior discovered by reverse engineering (cite the wiki/address). -- **Signed-off-by** style commit messages: short imperative summary line, - then the *why* (root cause), the *what*, and the user-visible effect. - Reference real dmesg/GDB evidence where relevant. - -### Must not -- No guest-side Open-IOV code. Ever. The guest is stock. -- No reboots/system-affecting commands from scripts without explicit operator - action (see the project's hard rule). -- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone; - nvidia.ko imports *our* symbols, not vice versa. -- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`. -- Don't widen an API's hardware assumptions silently: if a function needs - `vgpu->pdev`, say so in the comment (unit tests can't call those paths). - -### Kernel module specifics -- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`). -- Exported symbols: only what other components genuinely need - (`EXPORT_SYMBOL` right after the definition). -- New per-device resources belong in `struct nv_vgpu_device` with matching - init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain. - -## 5. Documentation expectations - -- New subsystem → new doc in `docs/` + a link from `README.md` and this file. -- Changed behavior → update the affected doc in the **same commit**. -- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table. -- Firmware addresses/structures → `wiki/Address-Reference.md` and friends. - -## 6. Where to read next (in order) - -1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works -2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist -3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design -4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end -5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal -6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference -7. `wiki/` — firmware reverse-engineering canon - -## 7. Getting help / escalating - -- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`. -- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first. -- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test. -- Found a new failure mode? Debug it per the guide, then **write it down** in - the taxonomy table so the next person doesn't rediscover it. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md deleted file mode 100644 index 6ba8d1b..0000000 --- a/docs/PERFORMANCE.md +++ /dev/null @@ -1,103 +0,0 @@ -# Open-IOV Performance Testing & A/B Methodology - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Performance -> replacement: -> [05-operations/01-performance-correctness-security.md](05-operations/01-performance-correctness-security.md). - -> How we measure vGPU performance against bare metal, interpret the results, -> and keep overhead on the critical path honest. **Run correctness first** -> (`docs/DEBUGGING.md` §5) — numbers from a half-working stack are noise. - ---- - -## 1. What overhead is acceptable (targets) - -| Path | Target | Why | -|------|--------|-----| -| GSP RPC round-trip (doorbell→response→MSI-X) | < 50 µs (stretch: < 10 µs) | every driver op goes through here | -| MSI-X injection (eventfd → guest handler) | < 5 µs | completion latency dominates | -| Guest BAR0 MMIO read/write | < 2 µs (shadow RAM) | hot path for register polling | -| CUDA kernel launch (added overhead vs bare metal) | < 10% | user-visible | -| Memory alloc/map RPCs | < 100 µs | setup-time, amortized | -| CUDA compute throughput | ≥ 95% of bare metal | GPU-bound, should be ~free | -| Graphics (frame rate) | ≥ 90% of bare metal | swapchain + submission overhead | - -## 2. The A/B rule - -Every measurement runs **twice on the same machine, same boot if possible**: - -- **A (bare metal)**: workload on the physical GPU via the host's stock - driver (or our OpenRM nvidia.ko directly on the hardware). -- **B (vGPU)**: identical workload in the guest on the emulated vGPU. - -Same binary, same input sizes, same environment (governor, clocks, hugepages). -Report **B/A ratio %** — never absolute numbers alone. - -One variable at a time. If B regresses, you changed the stack, the workload, -or the machine — find which before optimizing. - -## 3. Microbenchmarks (build these first) - -1. **RPC round-trip**: from the guest, submit a DEBUG/NOOP-ish RPC in a loop; - time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()` - gives `total_latency_ns` and counts — divide for per-RPC host cost. - (Add `trace_printk` in `gsp_emu_dispatch_rpc` entry/exit for distribution; - read via `/sys/kernel/debug/tracing/trace`.) -2. **MSI-X latency**: timestamp (guest TSC) at RPC submit → in the guest - driver's interrupt handler. Compare with the host eventfd_signal timestamp. -3. **BAR0 shadow read latency**: guest loops reading a shadow register; - expect ~memory speed (uncached, so ~100-300 ns) — anything ≥ 1 µs means - you're trapping somewhere you shouldn't. -4. **Doorbell polling cost**: host `top`/`perf` on `gsp_emu_N` — the adaptive - poll (busy → 50-100 µs sleep) should sit at ~0% CPU when idle, and react - within one sleep quantum under load. Tune `GSP_EMU_POLL_*` only with data. - -## 4. Workload benchmarks - -Runner: `tools/benchmark_infrastructure.sh` (extend it; keep the A/B pattern). -In-guest execution goes through the harness (`vgpu_vm.py exec`) or the guest -test agent (`tools/guest_agent/`). - -| Suite | What | Pass criterion | -|-------|------|----------------| -| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A | -| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A | -| `nvbandwidth` (if available) | H2D/D2H/device copy GB/s | report; DMA path validation | -| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A | -| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) | -| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A | - -Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK` -both sides — throttling skews everything), and the command lines. - -## 5. Where the overhead lives (profile in this order) - -1. **RPC dispatch** — the single biggest lever. Everything the guest driver - does funnels through the emulation thread. -2. **Doorbell polling cadence** — too slow = latency; too hot = host CPU burn. -3. **eventfd/MSI-X delivery** — measure, don't assume. -4. **BAR0 uncached semantics** — shadow is UC for the guest; that's - MMIO-correct but slow for large reads. If the driver bulk-reads shadow, - consider caching-safe sub-regions. -5. **Memory model gap** (ALLOC host-pages vs BAR1 direct VRAM) — see - `docs/GSP_EMULATION_ARCHITECTURE.md` §3. Until unified, DMA-heavy - workloads are not representative. - -## 6. Optimizing safely - -- Change one thing, re-run the A/B pair, keep the numbers with the commit. -- Never optimize off bare-metal-only data: the goal is B/A, not A. -- If a change makes B faster but breaks the §5 validation checklist in - `docs/DEBUGGING.md`, it doesn't count. -- Record results in `STATE.md` (date, versions, B/A ratios) so regressions - are visible across commits. - -## 7. Reporting template - -``` -Date / commit / versions (guest=host=firmware=emulation): -Machine / GPU / clocks locked?: -Test, unit, A (bare), B (vGPU), B/A %: -Notes (what changed vs last run): -``` diff --git a/docs/PORTABLE_ARCHITECTURE.md b/docs/PORTABLE_ARCHITECTURE.md deleted file mode 100644 index 4842407..0000000 --- a/docs/PORTABLE_ARCHITECTURE.md +++ /dev/null @@ -1,747 +0,0 @@ -# Open-IOV vGPU-IO Portable Architecture Design - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Portability -> replacement: -> [06-portability/hardware-software-matrix.md](06-portability/hardware-software-matrix.md). - -## Design Goals - -### 1. Driver Version Portability -- **Support NVIDIA OpenRM 550+** without code changes per version -- **Automatic version detection** at runtime -- **Backward/forward compatibility** for RPC interfaces -- **Graceful degradation** when features unavailable - -### 2. Hardware Portability -- **Support multiple GPU architectures**: Turing (TU102/TU104), Ampere (GA100/GA102), Hopper (GH100), Blackwell (GB202/GB203/GB100), future -- **Support multiple form factors**: Consumer (RTX 3090/4090/5090), Data Center (A100/H100/B200), DGX Spark, DGX Station -- **Automatic hardware detection** via PCI device IDs and firmware signatures -- **Capability-based feature enablement** - -### 3. Zero Guest Modification -- **Guest runs stock NVIDIA driver** from NVIDIA website -- **No Open-IOV code in guest** kernel or userspace -- **Standard VFIO/mdev interface** only - ---- - -## Core Architecture: Three-Layer Abstraction - -``` -╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ -║ PORTABLE ARCHITECTURE LAYERS ║ -╠═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣ -║ ║ -║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ -║ │ LAYER 1: HARDWARE ABSTRACTION LAYER (HAL) │ ║ -║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ -║ │ │ GpuArchitecture (abstract base) │ │ ║ -║ │ │ ├── detect(pci_dev) → GpuArchitecture* │ │ ║ -║ │ │ ├── get_pci_device_ids() → vector<DeviceID> │ │ ║ -║ │ │ ├── get_bar_layout() → BarLayout │ │ ║ -║ │ │ ├── get_gsp_firmware_info() → FirmwareInfo │ │ ║ -║ │ │ ├── get_rpc_dispatcher_addresses() → RpcAddresses │ │ ║ -║ │ │ ├── get_capabilities() → GpuCapabilities │ │ ║ -║ │ │ ├── create_rpc_handler(rpc_type) → RpcHandler* │ │ ║ -║ │ │ └── create_bar_emulator() → BarEmulator* │ │ ║ -║ │ │ │ │ ║ -║ │ │ Concrete Implementations (per architecture): │ │ ║ -║ │ │ ├── TuringArchitecture (TU102, TU104, TU106, TU116, TU117) │ │ ║ -║ │ │ ├── AmpereArchitecture (GA100, GA102, GA103, GA104, GA106, GA107) │ │ ║ -║ │ │ ├── HopperArchitecture (GH100, GH200) │ │ ║ -║ │ │ ├── BlackwellArchitecture (GB202, GB203, GB205, GB206, GB100) │ │ ║ -║ │ │ └── FutureArchitecture (auto-generated from firmware analysis) │ │ ║ -║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ -║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ -║ ║ -║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ -║ │ LAYER 2: VERSION ADAPTATION LAYER (VAL) │ ║ -║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ -║ │ │ DriverVersion (semantic: major.minor.patch) │ │ ║ -║ │ │ ├── parse(version_string) → DriverVersion │ │ ║ -║ │ │ ├── is_compatible(min_version, max_version) → bool │ │ ║ -║ │ │ ├── get_rpc_interface_version() → RpcInterfaceVersion │ │ ║ -║ │ │ ├── get_firmware_signature() → string (SHA256 of firmware) │ │ ║ -║ │ │ └── get_feature_flags() → FeatureFlags │ │ ║ -║ │ │ │ │ ║ -║ │ │ RpcInterfaceVersion (per driver version): │ │ ║ -║ │ │ ├── get_rpc_type_count() → int │ │ ║ -║ │ │ ├── get_rpc_struct_layout(rpc_type) → StructLayout │ │ ║ -║ │ │ ├── get_subcommand_mapping() → SubcommandMap │ │ ║ -║ │ │ ├── get_error_codes() → ErrorCodeMap │ │ ║ -║ │ │ └── get_capability_bits() → CapabilityBits │ │ ║ -║ │ │ │ │ ║ -║ │ │ Version Registry (loaded at runtime): │ │ ║ -║ │ │ ├── 550.x → RpcInterfaceV1 │ │ ║ -║ │ │ ├── 560.x → RpcInterfaceV2 (added MIGRATE subcommands) │ │ ║ -║ │ │ ├── 570.x → RpcInterfaceV3 (added DEBUG, NOTIFY enhancements) │ │ ║ -║ │ │ ├── 580.x → RpcInterfaceV4 (confidential computing, TEE) │ │ ║ -║ │ │ └── auto-detect from firmware signature │ │ ║ -║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ -║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ -║ ║ -║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ -║ │ LAYER 3: FIRMWARE-DRIVEN RPC ENGINE (FDRE) │ ║ -║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ -║ │ │ FirmwareAnalyzer (runs at module load) │ │ ║ -║ │ │ ├── load_firmware(firmware_path) → FirmwareImage │ │ ║ -║ │ │ ├── find_rpc_dispatcher() → address │ │ ║ -║ │ │ ├── find_jump_table() → address │ │ ║ -║ │ │ ├── find_handler_addresses() → map<rpc_type, address> │ │ ║ -║ │ │ ├── extract_struct_layouts() → map<rpc_type, StructLayout> │ │ ║ -║ │ │ ├── compute_firmware_signature() → SHA256 │ │ ║ -║ │ │ └── validate_against_known_versions() → DriverVersion │ │ ║ -║ │ │ │ │ ║ -║ │ │ DynamicRpcDispatcher: │ │ ║ -║ │ │ ├── dispatch(rpc_msg) → uses discovered handler addresses │ │ ║ -║ │ │ ├── fallback_to_emulation(rpc_type) → if handler not found │ │ ║ -║ │ │ └── hot_patch_handler(rpc_type, new_address) → for live updates │ │ ║ -║ │ │ │ │ ║ -║ │ │ EmulationFallbackRegistry: │ │ ║ -║ │ │ ├── register_emulation(rpc_type, EmulationHandler*) │ │ ║ -║ │ │ ├── get_emulation(rpc_type) → EmulationHandler* │ │ ║ -║ │ │ └── supports_emulation(rpc_type) → bool │ │ ║ -║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ -║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ -║ ║ -╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ -``` - ---- - -## Configuration-Driven Design - -### 1. Hardware Configuration Files - -```yaml -# config/hardware/turing.yaml -architecture: "turing" -pci_device_ids: - - vendor: 0x10de - device: 0x1e07 # TU102 - name: "RTX 2080 Ti" - - vendor: 0x10de - device: 0x1e08 # TU104 - name: "RTX 2080 / T4" - - vendor: 0x10de - device: 0x1ed3 # TU104 mobile - name: "RTX 2080 Super Mobile" -bar_layout: - bar0_size: 16MB - bar1_size: 256MB - bar2_size: 32MB - bar3_size: 32MB - bar0_prefetchable: true - bar1_prefetchable: true -gsp: - firmware_base: 0x04000000 - doorbell_offset: 0x110C00 - shared_mem_offset: 0x1000 - shared_mem_size: 4KB - rpc_dispatcher_primary: 0x057A1AC0 - rpc_dispatcher_secondary: 0x04ED39DC - rpc_jump_table: 0x059BFEB0 - handlers: - alloc: 0x057E94EC - schedule: 0x051EFBF4 - bar_trap: 0x04CB8188 - wpr_1: 0x04F00B0C - wpr_2: 0x04F00C78 - migration_table_1: 0x04179ED8 - migration_table_2: 0x04179FBC -capabilities: - max_vfs: 7 - supports_mig: true - supports_wpr: true - supports_bar_trap: true - supports_scheduling: true - supports_notifications: true - msi_x_vectors: 32 -``` - -```yaml -# config/hardware/ampere.yaml -architecture: "ampere" -pci_device_ids: - - vendor: 0x10de - device: 0x20b0 # GA100 - name: "A100" - - vendor: 0x10de - device: 0x2204 # GA102 - name: "RTX 3090" - - vendor: 0x10de - device: 0x2206 # GA102 - name: "RTX 3080" -bar_layout: - bar0_size: 16MB - bar1_size: 512MB # Larger FB on Ampere - bar2_size: 64MB - bar3_size: 64MB -gsp: - firmware_base: 0x04000000 - doorbell_offset: 0x110C00 - shared_mem_offset: 0x1000 - # Addresses discovered at runtime via FirmwareAnalyzer -capabilities: - max_vfs: 7 - supports_mig: true - supports_mig_mig: true # MIG (Multi-Instance GPU) - supports_confidential: false - supports_wpr: true - supports_bar_trap: true - msi_x_vectors: 64 -``` - -```yaml -# config/hardware/blackwell.yaml -architecture: "blackwell" -pci_device_ids: - - vendor: 0x10de - device: 0x2480 # GB202 (RTX 5090) - name: "RTX 5090" - - vendor: 0x10de - device: 0x2380 # GB100 (B200) - name: "B200" - - vendor: 0x10de - device: 0x2384 # GB205 - name: "RTX 5070" -bar_layout: - bar0_size: 16MB - bar1_size: 1GB # Blackwell unified memory - bar2_size: 128MB - bar3_size: 128MB -gsp: - firmware_base: 0x04000000 - doorbell_offset: 0x110C00 - shared_mem_offset: 0x1000 -capabilities: - max_vfs: 8 - supports_mig: true - supports_mig_mig: true - supports_confidential: true # TEE support - supports_wpr: true - supports_bar_trap: true - supports_nvlink: true - msi_x_vectors: 128 -``` - -### 2. Driver Version Configuration - -```yaml -# config/versions/550.yaml -version: "550" -rpc_interface_version: 1 -rpc_types: - - ALLOC: 0x01 - - FREE: 0x02 - - MAP: 0x03 - - UNMAP: 0x04 - - PIN: 0x05 - - UNPIN: 0x06 - - SCHEDULE: 0x07 - - YIELD: 0x08 - - NOTIFY: 0x09 - - BAR_TRAP: 0x0A - - MIGRATE: 0x0B - - DEBUG: 0x0C -struct_layouts: - ALLOC: - size: 48 - fields: - - {name: size, offset: 0, type: u64} - - {name: alignment, offset: 8, type: u64} - - {name: mem_type, offset: 16, type: u32} - - {name: flags, offset: 20, type: u32} - - {name: addr_hint, offset: 24, type: u64} - - {name: pid, offset: 32, type: u32} -capabilities: - supports_migrate: true - migrate_subcommands: - - SAVE: 0x04 - - RESTORE: 0x05 - supports_notifications: basic -``` - -```yaml -# config/versions/570.yaml -version: "570" -rpc_interface_version: 3 -extends: "550" -rpc_types: - - NOTIFY: 0x09 # Enhanced with subcommands - - MIGRATE: 0x0B # Enhanced with 4-phase -struct_layouts: - NOTIFY: - size: 32 - fields: - - {name: type, offset: 0, type: u32} - - {name: subcmd, offset: 4, type: u32} - - {name: event_fd, offset: 8, type: u64} - - {name: flags, offset: 16, type: u32} -capabilities: - supports_notifications: enhanced - supports_migrate_4phase: true - migrate_phases: [QUERY, PREPARE, TRANSFER, COMMIT, CLEANUP] -``` - -```yaml -# config/versions/580.yaml -version: "580" -rpc_interface_version: 4 -extends: "570" -rpc_types: - - TEE_ATTEST: 0x0D # New in 580 for confidential computing - - TEE_SEAL: 0x0E - - TEE_UNSEAL: 0x0F -capabilities: - supports_confidential: true - supports_tee: true - tee_attestation_types: [CPU, GPU, MEMORY] -``` - ---- - -## Runtime Detection & Adaptation Flow - -``` -┌─────────────────────────────────────────────────────────────────────────────────────┐ -│ MODULE LOAD: RUNTIME DETECTION FLOW │ -├─────────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ 1. open-iov-vfio.ko loads │ -│ │ │ -│ ▼ │ -│ 2. Probe PCI device │ -│ │ │ -│ ▼ │ -│ 3. Match PCI Device ID → Hardware Config (turing.yaml, ampere.yaml, etc.) │ -│ │ │ -│ ▼ │ -│ 4. Load Hardware Config → Instantiate GpuArchitecture subclass │ -│ │ │ -│ ▼ │ -│ 5. Load GSP Firmware (from /lib/firmware or host nvidia.ko) │ -│ │ │ -│ ▼ │ -│ 6. FirmwareAnalyzer.analyze(firmware) │ -│ ├── Compute firmware SHA256 │ -│ ├── Find RPC dispatcher via pattern matching │ -│ ├── Find jump table via cross-references │ -│ ├── Extract handler addresses for all 12+ RPC types │ -│ ├── Extract struct layouts via pattern matching │ -│ └── Match firmware signature → DriverVersion (550, 560, 570, 580, etc.) │ -│ │ │ -│ ▼ │ -│ 7. Load Version Config → Instantiate RpcInterfaceVersion │ -│ │ │ -│ ▼ │ -│ 8. Validate: Hardware Config + Version Config compatible? │ -│ │ │ -│ ├── YES → Initialize DynamicRpcDispatcher with discovered addresses │ -│ ├── PARTIAL → Enable emulation for missing handlers │ -│ └── NO → Fail with clear error: "Unsupported GPU/driver combination" │ -│ │ │ -│ ▼ │ -│ 9. Register EmulationFallbackRegistry for all RPC types │ -│ │ │ -│ ▼ │ -│ 10. Start GSP Emulation Thread (per mdev) │ -│ │ -└─────────────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## FirmwareAnalyzer - Automatic Address Discovery - -```c -// kernel/firmware_analyzer.c - -struct firmware_signature { - uint8_t sha256[32]; - char version_string[64]; - uint32_t crc32; -}; - -struct discovered_rpc_handlers { - uint64_t primary_dispatcher; - uint64_t secondary_dispatcher; - uint64_t jump_table; - struct { - uint32_t rpc_type; - uint64_t handler_address; - bool found; - } handlers[16]; // Support up to 16 RPC types - uint32_t handler_count; -}; - -struct firmware_analyzer { - const uint8_t *firmware; - size_t firmware_size; - uint64_t firmware_base; - struct discovered_rpc_handlers discovered; - struct firmware_signature signature; -}; - -int firmware_analyzer_init(struct firmware_analyzer *fa, const char *firmware_path) { - // Load firmware file - fa->firmware = load_firmware_file(firmware_path, &fa->firmware_size); - if (!fa->firmware) - return -ENOENT; - - // Compute signature - sha256(fa->firmware, fa->firmware_size, fa->signature.sha256); - fa->signature.crc32 = crc32(0, fa->firmware, fa->firmware_size); - - // Set base address (from hardware config) - fa->firmware_base = get_hardware_config()->gsp.firmware_base; - - return 0; -} - -int firmware_analyzer_discover(struct firmware_analyzer *fa) { - // 1. Find primary dispatcher: search for known pattern - // Pattern: "RPC_DISPATCH" string reference or known instruction sequence - fa->discovered.primary_dispatcher = find_pattern(fa->firmware, fa->firmware_size, - "RPC_DISPATCH", fa->firmware_base); - - // 2. Find jump table: cross-reference from dispatcher - fa->discovered.jump_table = find_jump_table(fa->firmware, fa->firmware_size, - fa->discovered.primary_dispatcher); - - // 3. Parse jump table entries (32 entries for 12 RPC types) - parse_jump_table(fa->firmware, fa->discovered.jump_table, &fa->discovered); - - // 4. Find secondary dispatcher - fa->discovered.secondary_dispatcher = find_secondary_dispatcher(fa->firmware, - fa->firmware_size, fa->discovered.primary_dispatcher); - - // 5. Find specialized handlers via string references - find_handler_by_string(fa, "BAR_TRAP", "BAR1", "ForceBarAccess", &fa->discovered.handlers[GSP_RPC_BAR_TRAP]); - find_handler_by_string(fa, "WPR", "completion", "timeout", &fa->discovered.handlers[GSP_RPC_NOTIFY]); - find_handler_by_string(fa, "ALLOC", "size", "alignment", &fa->discovered.handlers[GSP_RPC_ALLOC]); - find_handler_by_string(fa, "SCHEDULE", "work", "callback", &fa->discovered.handlers[GSP_RPC_SCHEDULE]); - find_handler_by_string(fa, "MIGRATE", "checkpoint", "progress", &fa->discovered.handlers[GSP_RPC_MIGRATE]); - - // 6. Extract struct layouts via pattern matching on handler prologues - extract_struct_layouts(fa); - - // 7. Match signature against known versions - match_firmware_version(fa); - - return 0; -} - -void firmware_analyzer_match_version(struct firmware_analyzer *fa) { - // Compare signature against known database - static const struct { - const char *version; - uint8_t sha256[32]; - uint32_t crc32; - } known_versions[] = { - {"550.163.01", {0x...}, 0x...}, - {"560.28.03", {0x...}, 0x...}, - {"570.144.00", {0x...}, 0x...}, - {"570.144.01", {0x...}, 0x...}, - {"580.00.00", {0x...}, 0x...}, - }; - - for (int i = 0; i < ARRAY_SIZE(known_versions); i++) { - if (memcmp(fa->signature.sha256, known_versions[i].sha256, 32) == 0) { - strncpy(fa->signature.version_string, known_versions[i].version, 63); - return; - } - } - - // Unknown version - use heuristic based on handler count and features - if (fa->discovered.handler_count >= 12) - strcpy(fa->signature.version_string, "570+ (unknown)"); - else if (fa->discovered.handler_count >= 10) - strcpy(fa->signature.version_string, "560+ (unknown)"); - else - strcpy(fa->signature.version_string, "550+ (unknown)"); -} -``` - ---- - -## Dynamic RPC Dispatcher - -```c -// kernel/dynamic_rpc_dispatcher.c - -struct dynamic_rpc_dispatcher { - struct discovered_rpc_handlers *discovered; - struct rpc_interface_version *interface; - struct emulation_fallback_registry *fallbacks; - spinlock_t lock; - uint64_t dispatch_stats[16]; -}; - -int dynamic_rpc_dispatcher_dispatch(struct dynamic_rpc_dispatcher *dr, - struct gsp_rpc_msg *req, - struct gsp_rpc_response *rsp) { - // 1. Validate RPC type - if (req->type >= dr->interface->rpc_type_count) - return -ENOTTY; - - // 2. Check if we have a native handler address - uint64_t handler_addr = 0; - for (int i = 0; i < dr->discovered->handler_count; i++) { - if (dr->discovered->handlers[i].rpc_type == req->type) { - handler_addr = dr->discovered->handlers[i].handler_address; - break; - } - } - - // 3. Dispatch to native handler if available - if (handler_addr) { - return dynamic_rpc_call_native(dr, handler_addr, req, rsp); - } - - // 4. Fallback to emulation - struct emulation_handler *emu = emulation_fallback_get(dr->fallbacks, req->type); - if (emu) { - dr->dispatch_stats[req->type]++; // Track emulation usage - return emu->handle(req, rsp); - } - - // 5. No handler, no emulation - rsp->status = -ENOTTY; - return -ENOTTY; -} - -int dynamic_rpc_call_native(struct dynamic_rpc_dispatcher *dr, - uint64_t handler_addr, - struct gsp_rpc_msg *req, - struct gsp_rpc_response *rsp) { - // Use firmware-decompiled calling convention - // This calls the actual GSP firmware handler via the discovered address - // In emulation mode, we call our C reimplementation instead - - if (gsp_emulation_mode_enabled()) { - // Call our C reimplementation (from gsp_rpc_client.c) - return gsp_rpc_client_call(req->type, req, rsp); - } else { - // Native mode: would call into GSP firmware - // Not used in our architecture (we always emulate in host) - return -ENOTSUPP; - } -} -``` - ---- - -## Capability Negotiation with Guest - -``` -┌─────────────────────────────────────────────────────────────────────────────────────┐ -│ GUEST DRIVER CAPABILITY NEGOTIATION │ -├─────────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ Guest nvidia.ko probes vGPU: │ -│ 1. Reads PCI Config Space → gets device ID, BAR sizes, capabilities │ -│ 2. Maps BAR0 → finds GSP doorbell + shared memory │ -│ 3. Sends RPC: QUERY_CAPABILITIES (new RPC type or subcommand) │ -│ │ -│ Host DynamicRpcDispatcher receives QUERY_CAPABILITIES: │ -│ 1. Constructs response from: │ -│ - Hardware Config capabilities │ -│ - Version Config capabilities │ -│ - EmulationFallbackRegistry (what we can emulate) │ -│ 2. Returns capability bitmap to guest │ -│ │ -│ Guest nvidia.ko enables features based on response: │ -│ - If MIG supported → enables MIG code paths │ -│ - If WPR supported → enables WPR code paths │ -│ - If CONFIDENTIAL supported → enables TEE code paths │ -│ - If NOTIFY enhanced → uses async eventfd │ -│ │ -│ This allows SAME guest driver to work across ALL hardware/versions! │ -│ │ -└─────────────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## Update Strategy for New Driver Versions - -### 1. Adding New Driver Version (e.g., 590) - -```bash -# 1. Add version config -cp config/versions/580.yaml config/versions/590.yaml -# Edit: version, rpc_interface_version, new RPC types, capabilities - -# 2. Test with new firmware -# FirmwareAnalyzer auto-discovers addresses - NO CODE CHANGES NEEDED - -# 3. If new RPC types added: -# - Add to config/versions/590.yaml -# - Add emulation handler in kernel/emulation_fallbacks.c -# - Register in EmulationFallbackRegistry - -# 4. If struct layouts changed: -# - FirmwareAnalyzer extracts new layouts automatically -# - Version config specifies layout version -``` - -### 2. Adding New GPU Architecture (e.g., Rubin R100) - -```bash -# 1. Add hardware config -cp config/hardware/blackwell.yaml config/hardware/rubin.yaml -# Edit: PCI device IDs, BAR layout, capabilities - -# 2. Add GpuArchitecture subclass (if special handling needed) -# kernel/arch/rubin_architecture.c - -# 3. Register in architecture factory -# kernel/architecture_factory.c - -# 4. Test - FirmwareAnalyzer handles address discovery -``` - -### 3. Automated Firmware Analysis Pipeline - -```yaml -# .github/workflows/firmware-analysis.yml -name: Firmware Analysis -on: - schedule: - - cron: '0 0 * * 0' # Weekly - workflow_dispatch: - inputs: - firmware_url: - description: 'NVIDIA firmware download URL' - required: true - -jobs: - analyze: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Download firmware - run: wget -O firmware.bin "${{ inputs.firmware_url }}" - - name: Run FirmwareAnalyzer - run: | - ./tools/firmware_analyzer firmware.bin > analysis.json - - name: Compare with known versions - run: ./tools/compare_analysis.py analysis.json - - name: Generate config if new - run: | - if [ "$(cat analysis.json | jq -r '.is_new')" = "true" ]; then - ./tools/generate_config.py analysis.json > config/versions/NEW.yaml - git add config/versions/NEW.yaml - git commit -m "Add support for new firmware version" - git push - fi -``` - ---- - -## Testing Matrix for Portability - -| GPU Architecture | PCI Device IDs | Driver Versions | Test Status | -|------------------|----------------|-----------------|-------------| -| **Turing** | 10de:1e07, 10de:1e08, 10de:1ed3 | 550, 560, 570, 580 | ✅ Baseline | -| **Ampere** | 10de:20b0, 10de:2204, 10de:2206 | 550, 560, 570, 580 | 🔄 In Progress | -| **Hopper** | 10de:2330, 10de:2334 | 560, 570, 580 | 🔄 Planned | -| **Blackwell** | 10de:2480, 10de:2380, 10de:2384 | 570, 580 | 🔄 Planned | -| **Rubin (Future)** | TBD | 590+ | 📋 Designed | - -### Per-Combination Tests - -```bash -# Automated test matrix -for arch in turing ampere hopper blackwell; do - for version in 550 560 570 580; do - if [ -f "config/hardware/${arch}.yaml" ] && [ -f "config/versions/${version}.yaml" ]; then - echo "Testing ${arch} + ${version}..." - ./scripts/test_combination.sh ${arch} ${version} - fi - done -done -``` - ---- - -## Implementation File Structure (Portable) - -``` -open-iov/ -├── config/ -│ ├── hardware/ -│ │ ├── turing.yaml -│ │ ├── ampere.yaml -│ │ ├── hopper.yaml -│ │ ├── blackwell.yaml -│ │ └── rubin.yaml (future) -│ ├── versions/ -│ │ ├── 550.yaml -│ │ ├── 560.yaml -│ │ ├── 570.yaml -│ │ ├── 580.yaml -│ │ └── 590.yaml (future) -│ └── defaults.yaml -├── kernel/ -│ ├── architecture/ -│ │ ├── architecture.h # Abstract base class -│ │ ├── architecture_factory.c # Detection & instantiation -│ │ ├── turing_architecture.c -│ │ ├── ampere_architecture.c -│ │ ├── hopper_architecture.c -│ │ ├── blackwell_architecture.c -│ │ └── generic_architecture.c # Fallback -│ ├── version/ -│ │ ├── version.h # DriverVersion, RpcInterfaceVersion -│ │ ├── version_registry.c # Loads version configs -│ │ ├── interface_v1.c # 550 interface -│ │ ├── interface_v2.c # 560 interface -│ │ ├── interface_v3.c # 570 interface -│ │ └── interface_v4.c # 580 interface -│ ├── firmware/ -│ │ ├── firmware_analyzer.c # Auto-discovers addresses -│ │ ├── firmware_analyzer.h -│ │ ├── pattern_matcher.c # Binary pattern matching -│ │ └── struct_layout_extractor.c -│ ├── dispatch/ -│ │ ├── dynamic_rpc_dispatcher.c -│ │ ├── dynamic_rpc_dispatcher.h -│ │ ├── emulation_fallback_registry.c -│ │ └── emulation_fallback_registry.h -│ ├── gsp_emulation.c # Uses dynamic dispatcher -│ ├── main.c # Integrates all layers -│ └── Makefile -├── tools/ -│ ├── firmware_analyzer/ # Standalone analysis tool -│ ├── compare_analysis.py -│ ├── generate_config.py -│ └── test_combination.sh -└── docs/ - ├── PORTABLE_ARCHITECTURE.md - └── VERSION_COMPATIBILITY.md -``` - ---- - -## Key Design Principles Summary - -| Principle | Implementation | -|-----------|----------------| -| **No hardcoded addresses** | FirmwareAnalyzer discovers at runtime | -| **No hardcoded struct layouts** | Version config + FirmwareAnalyzer extraction | -| **No hardcoded device IDs** | Hardware config YAML files | -| **No hardcoded capabilities** | Capability negotiation via QUERY_CAPABILITIES RPC | -| **Graceful degradation** | EmulationFallbackRegistry for missing handlers | -| **Forward compatibility** | Version config `extends` mechanism | -| **Automated updates** | Firmware analysis pipeline + config generation | -| **Test matrix** | Automated CI across architecture × version combinations | - ---- - -*Document Version: 1.0* -*Repository: $OPEN_IOV_ROOT* -*Branch: refactor/host-gsp-emulation* -*Base Commit: f9ed5d8*
\ No newline at end of file diff --git a/docs/README.md b/docs/README.md index cee435e..ed0f206 100644 --- a/docs/README.md +++ b/docs/README.md @@ -153,16 +153,31 @@ decompilation record, address reference, historical protocol notes). Wiki pages that predate the current implementation carry a status banner naming the canonical document in this tree; use the banner, not the old body. -Historical material is kept but must not be read as current state. The -following carry a **superseded** banner and point at their replacements: -`refactored-architecture.md`, `refactored-implementation-to-do.md`, -`PORTABLE_ARCHITECTURE.md`, `GSP_EMULATION_ARCHITECTURE.md`, `ONBOARDING.md`, -`DEVELOPMENT_WORKFLOW.md`, `PERFORMANCE.md`, `DEBUGGING.md` (sections 3–4 -remain useful case studies), `VERSION_UPGRADE.md`, `BUILD_NVIDIA_OPENRM.md` -(historical build notes; its load order is corrected in place), and -[`../STATE.md`](../STATE.md) (2026-09-21 snapshot). The current versions of -those topics are `03-architecture/01`/`06`, `04-implementation/01`/`02`, -`05-operations/01`/`02`, `06-portability`, and `07-developers`. +The pre-reboot flat documents have been merged into this tree and removed +(2026-09-23). Where their material went: + +* debugging: gdbstub/NMI workflows, the EPT-livelock case study and the + failure taxonomy → + [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md) +* performance: the microbenchmark list and the A/B methodology → + [`05-operations/01-performance-correctness-security.md`](05-operations/01-performance-correctness-security.md) +* version upgrades: the firmware/driver upgrade runbook (with the corrected + host/guest framing) → + [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md) §4 +* kernel-code pitfalls and the must/must-not rules → + [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) +* the OpenRM build/symbol/load order → + [`03-architecture/06-host-module-and-hybrid-model.md`](03-architecture/06-host-module-and-hybrid-model.md) §7 +* architecture, portability and implementation-plan material is now covered by + [`03-architecture/01-system-overview.md`](03-architecture/01-system-overview.md), + [`03-architecture/06-host-module-and-hybrid-model.md`](03-architecture/06-host-module-and-hybrid-model.md), + [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md), + [`04-implementation/02-practical-challenges-and-adversarial-interop.md`](04-implementation/02-practical-challenges-and-adversarial-interop.md) + and [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md). + +What remains historical is the pre-supersession wiki material, `old/ARCHITECTURE.md`, +and [`../STATE.md`](../STATE.md) (a 2026-09-21 snapshot); their status banners +name the canonical document. ## 5. What makes Open-IOV different? @@ -190,7 +205,8 @@ the stack has to be modified or licensed. valuable thing you can send — especially non-Turing hardware, Windows guests, and non-KVM hypervisors. * **Extend it.** The architecture docs mark the interfaces that are expected to - grow (memory model, forwarding backends, config-driven portability). + grow (memory model, forwarding backends, profile-based portability — a design + target today, see [`06-portability/hardware-software-matrix.md` §6](06-portability/hardware-software-matrix.md)). * **Integrate it.** If you build a hypervisor, an edge platform or a security stack, see [`08-ecosystem/haaeep-and-collaboration.md`](08-ecosystem/haaeep-and-collaboration.md). * **Fund or co-develop it.** Commercial use is explicitly welcome under diff --git a/docs/VERSION_UPGRADE.md b/docs/VERSION_UPGRADE.md deleted file mode 100644 index 3fc5de4..0000000 --- a/docs/VERSION_UPGRADE.md +++ /dev/null @@ -1,144 +0,0 @@ -# Open-IOV Version Upgrade Guide - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). -> Version-upgrade replacements: -> [06-portability/hardware-software-matrix.md](06-portability/hardware-software-matrix.md) -> and -> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md). - -> How to support a new NVIDIA driver version (e.g. moving from 570.144 to a -> 610-series release) end-to-end: firmware acquisition → reverse engineering → -> config → host driver → guest validation. The goal is that a version bump is -> **configuration + analysis**, not a rewrite. - ---- - -## 0. Version discipline (read first) - -Four components must stay in lock-step; a mismatch between any two produces -failures that are indistinguishable from emulation bugs: - -``` -guest nvidia driver == host OpenRM nvidia.ko == host GSP firmware == emulation target (decompiled GSP) -``` - -Today that anchor is **570.144** everywhere. When you bump one, you bump all -four. Verify with `docs/DEBUGGING.md` §7 before any deep debugging. - ---- - -## 1. Acquire the new firmware + matching OpenRM tag - -1. Download the Linux x86_64 runfile for the target version: - `https://us.download.nvidia.com/XFree86/Linux-x86_64/<VER>/NVIDIA-Linux-x86_64-<VER>.run` - (the `tesla/` path 404s for consumer releases). -2. Extract the GSP firmware (`gsp_tu10x.bin` / arch-specific) — or fetch the - versioned blob from `linux-firmware.git`: - `nvidia/<VER>/gsp_tu10x.bin`. -3. Extract the `.fwimage` section (host boot image) and record the MD5: - ```bash - objcopy --only-section=.fwimage gsp_tu10x.bin gsp-<VER>.fwimage.bin - md5sum gsp-<VER>.fwimage.bin - ``` -4. Note the matching OpenRM tag at - `github.com/NVIDIA/open-gpu-kernel-modules` (e.g. `570.144`). - -## 2. Lay down versioned directories (never overwrite a working version) - -``` -firmware/<SERIES>-GSP/<VER>/ # e.g. firmware/61X-GSP/610.XX/ - gsp-<VER>.fwimage.bin - decompiled/ # ghidra output lands here -config/versions/<MAJOR>.yaml # e.g. config/versions/610.yaml -``` - -Copy the analysis scripts from the previous version dir -(`firmware/57X-GSP/570.144/*.py`) into the new one — they are script-relative. - -## 3. Reverse-engineer the new firmware (Ghidra headless) - -```bash -cd tools/ghidra -python3 analyze_fwimage.py \ - ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \ - /tmp/ghidra_project_<VER> "gsp_<VER>" -cd ../../firmware/<SERIES>-GSP/<VER> -python3 analyze_firmware.py gsp-<VER>.fwimage.bin -``` - -What to extract (compare against `wiki/Address-Reference.md` for 570.144W): - -| Artifact | Why it matters | -|----------|----------------| -| Primary RPC dispatcher + jump table | RPC types/subcommands may shift | -| BAR trap / WPR handlers | BAR emulation behavior | -| Entry/boot sequence | GSP boot RPC ordering the guest driver expects | -| Migration tables | migration protocol (if enabled) | -| RPC message layouts | struct changes break the emulator silently | -| Version string/signature | runtime firmware matching | - -Sanity checks: still RV64IMAC? still the same 12 top-level RPC types? new -subcommands? Confidential-computing/TEE additions? - -## 4. Produce the version config - -Update `config/versions/<MAJOR>.yaml` with: -- `version.name/major/minor` and `firmware.gsp_version` -- discovered dispatcher/handler addresses (if tracked in yaml) -- `features` toggles per what the firmware actually has -- `symbols` (the nvidia.ko vGPU VFIO interface — verify against the new - OpenRM tag; renamed exports break the nvidia build) - -## 5. Update the runtime signature database - -`kernel/firmware_analyzer.c` matches firmware by CRC32/SHA256 + patterns. -Add the new entry to `known_signatures[]` (version string, hashes, discovered -addresses). This is what lets one `open-iov-vfio.ko` serve multiple firmware -versions safely. - -## 6. Rebuild host + guest against the new anchor - -1. Point `legacy-patches/nvidia/` at the new OpenRM tag (rebase our delta; verify - `modinfo nvidia.ko | grep depends` still shows `open-iov-vfio`). -2. `./scripts/stack_manager.sh build-all` (kernel → nvidia → userspace). -3. Install the new host firmware to `/lib/firmware/` - (`stack_manager.sh install-firmware`). -4. **Guest**: install the *matching* stock driver in the VM (runfile for the - exact version), replacing the old one. Keep the guest able to boot without - nvidia auto-loading during the swap (temporarily move dkms-built modules - aside — see `docs/DEBUGGING.md`). - -## 7. Validate (same checklist every version) - -Run the full bring-up checklist in `docs/DEBUGGING.md` §5: -module load → mdev → VM boot → guest probe → `nvidia-smi` → teardown -regression → unit test. Then the workload tests (CUDA vectorAdd, graphics). -Only after the checklist is green do performance comparisons -(`docs/PERFORMANCE.md`). - -## 8. Rollback plan - -Keep the previous version's directory + config untouched. Rollback = -`git checkout` the repo tag/commit for the old anchor + reinstall the old -guest driver. Because versions live in parallel directories, the working -setup is never destroyed by a forward attempt. - -## 9. Multi-version testing matrix (target state) - -For each supported anchor version × each supported GPU family -(`config/hardware/*.yaml`), the CI matrix should run the §7 checklist. When -adding a version, extend the matrix rather than replacing the old column — -regressions across versions are the cheapest bugs to catch. - -## 10. Common pitfalls (learned the hard way) - -- **Runfile version drift**: Ubuntu's `nvidia-driver-XXX-open` metapackages - can resolve to a *newer* major (570-open pulled 580). Always check the - candidate version before installing; prefer the exact-version runfile. -- **GSP firmware filename**: guest driver requests `nvidia/<VER>/gsp_tu10x.bin`; - a wrong-version firmware file = probe failure that looks like an emulation bug. -- **Kernel text addresses differ per build**: re-derive Ghidra addresses per - firmware; never carry 570.144 addresses into a 610 config. -- **Symbol renames across OpenRM tags**: `nvidia_vgpu_vfio_*` renames break - the nvidia build with `Unknown symbol` — check `kernel-open` diffs. diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md deleted file mode 100644 index f45df2f..0000000 --- a/docs/refactored-architecture.md +++ /dev/null @@ -1,302 +0,0 @@ -# Open-IOV vGPU-IO Architecture - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). Architecture -> replacement: -> [03-architecture/01](03-architecture/01-system-overview.md) and -> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md). - -> **Status**: Living document. Updated 2026-09-22 post host-stability fix. -> **Supersedes**: Previous refactored-architecture.md (v1.0). - ---- - -## 1. Problem Statement - -**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with **full GPU functionality** (CUDA, graphics, video, AI/ML) on consumer and data-center GPUs without proprietary GRID licenses. - -**Correctness requirement**: Guest must see a fully functional NVIDIA GPU. `nvidia-smi`, CUDA, OpenGL/Vulkan, and compute must all work. - -**Performance requirement**: GPU compute must execute on the physical GPU hardware, not in software. Target: < 5% overhead vs native for CUDA kernel launches. - -**Constraints**: -- Guest runs **stock** `nvidia.ko` + standard NVIDIA userspace (no Open-IOV code in guest) -- Host runs **apt** `nvidia-driver-570` for physical GPU management (570.211.01) -- Host runs **open-iov-vfio.ko** for vGPU emulation (mediated device, mdev) -- No SR-IOV required (works on consumer GPUs like RTX 2080 Super) - ---- - -## 2. Core Architectural Principle: Emulate the Interface, Pass Through the Work - -A vGPU is a **mediated passthrough** device. Some functions must be emulated (the guest expects virtual hardware), others must be passed through to the physical GPU (for performance). - -``` -┌──────────────────────────────────────────────────────────────────────┐ -│ EMULATE (software — guest expects virtual hardware) │ -│ • PCI config space (guest sees virtual TU104 [10de:1ed3]) │ -│ • BAR0 register interface (shadow, not physical GPU BAR0) │ -│ • GSP boot sequence (FWSEC/Booter/RISCV auto-complete) │ -│ • GSP-RM protocol layer (command queue, reply format) │ -│ • Interrupt routing (eventfd → guest MSI-X) │ -│ • Display output (headless / virtual display) │ -│ • BAR3 instance memory window (shadow, not physical GPU) │ -└──────────────────────────────────────────────────────────────────────┘ - -┌──────────────────────────────────────────────────────────────────────┐ -│ PASS-THROUGH (hardware — for performance) │ -│ • GPU compute (CUDA kernels) → execute on physical GPU │ -│ • GPU memory (guest VRAM) → backed by real GPU VRAM or DMA-mapped │ -│ • DMA operations → use real GPU DMA engines │ -│ • Channel/pushbuffer work → physical GPU channels │ -│ • Graphics rendering → physical GPU render pipeline │ -│ • Video encode/decode → physical GPU NVENC/NVDEC engines │ -└──────────────────────────────────────────────────────────────────────┘ -``` - -**Why not full emulation?** Software emulation of CUDA kernels would be 100-1000x slower than native. The physical GPU must do the actual compute work. - -**Why not full passthrough?** The guest driver expects to manage the GSP, boot firmware, and own the GPU. We can't give it direct physical GPU access (the host driver already owns it). The GSP emulation layer translates guest commands into host GPU operations. - ---- - -## 3. Host Module Architecture (Corrected 2026-09-22) - -``` -╔══════════════════════════════════════════════════════════════════════╗ -║ HOST KERNEL SPACE ║ -║ ║ -║ ┌────────────────────────────────────────────────────────────────┐ ║ -║ │ nvidia.ko (apt 570.211.01) │ ║ -║ │ • Manages physical RTX 2080 Super (GSP firmware booted) │ ║ -║ │ • Provides /dev/nvidia0 for nvidia-smi on host │ ║ -║ │ • Owns physical GPU interrupts, BARs, VRAM │ ║ -║ │ • NOT modified, NOT rebuilt — stock Ubuntu package │ ║ -║ └────────────────────────────────────────────────────────────────┘ ║ -║ ║ -║ ┌────────────────────────────────────────────────────────────────┐ ║ -║ │ open-iov-vfio.ko (Open-IOV vGPU VFIO driver) │ ║ -║ │ • Registers mdev parent on the physical GPU's PCI device │ ║ -║ │ • Creates mdev types (nvidia-vgpu-turing, etc.) │ ║ -║ │ • Per-mdev: emulated PCI config, BAR0 shadow, BAR1 vidmem, │ ║ -║ │ BAR3 shadow, MSI-X emulation, GSP emulation thread │ ║ -║ │ • Exports VFIO device ops (read/write/ioctl/mmap) │ ║ -║ │ • Self-contained: depends only on vfio, mdev │ ║ -║ │ • NO dependency on nvidia submodule (removed 2026-09-22) │ ║ -║ └────────────────────────────────────────────────────────────────┘ ║ -║ ║ -║ ┌────────────────────────────────────────────────────────────────┐ ║ -║ │ vfio / mdev (kernel prerequisites) │ ║ -║ │ • vfio, vfio_iommu_type1, vfio_pci, mdev │ ║ -║ │ • IOMMU domain management, device isolation │ ║ -║ └────────────────────────────────────────────────────────────────┘ ║ -║ ║ -║ PHYSICAL GPU: RTX 2080 Super (TU104, 10de:1ed3) ║ -║ • Managed by apt nvidia driver (host nvidia-smi works) ║ -║ • open-iov-vfio registers mdev parent but does NOT drive the GPU ║ -║ • Guest vGPU is a software construct — physical GPU unaffected ║ -╚══════════════════════════════════════════════════════════════════════╝ - │ VFIO mdev - │ /sys/bus/mdev/devices/<UUID>/ - ▼ -╔══════════════════════════════════════════════════════════════════════╗ -║ GUEST VM (QEMU/KVM) ║ -║ ┌────────────────────────────────────────────────────────────────┐ ║ -║ │ nvidia.ko (stock 570.144) │ ║ -║ │ • PCI probe: sees TU104 [10de:1ed3] via emulated config │ ║ -║ │ • BAR0: shadow (vzalloc'd, mapped via VFIO) │ ║ -║ │ • BAR1: emulated vidmem (vzalloc'd host RAM) │ ║ -║ │ • BAR3: shadow (vzalloc'd) │ ║ -║ │ • MSI-X: eventfd → QEMU → guest ISR │ ║ -║ │ • GSP client: msgq RPCs via emulated GSP-RM channel │ ║ -║ │ • NO Open-IOV code in guest │ ║ -║ └────────────────────────────────────────────────────────────────┘ ║ -╚══════════════════════════════════════════════════════════════════════╝ -``` - -### Why no nvidia submodule on host? - -The **nvidia-open-gpu-kernel-modules submodule (610.57.04-based)** was originally intended for NVIDIA's *native* vGPU management (RM callbacks like `vgpu_create_callback`). For our QEMU emulated-VFIO path, it's **unnecessary and harmful**: - -1. **Unnecessary**: open-iov-vfio.ko handles all VFIO emulation independently -2. **Harmful**: Loading the 610-based submodule alongside apt 570.211.01 causes driver conflicts that crash the host GSP (WPR2 stuck, `unexpected WPR2 already up`) - -**Root cause fixed 2026-09-22** (`85d0668`): stack_manager skips submodule when apt nvidia is present. - ---- - -## 4. Emulated Component Details - -### 4.1 BAR0 Shadow (16 MB vzalloc) - -The guest's BAR0 is a software shadow, NOT the physical GPU's BAR0. - -| BAR0 offset | Region | Who writes | Who reads | -|-------------------|-------------------------------------|------------|-----------| -| `0x000-0xFFF` | Emulated registers (PMC boot, etc.) | shadow | shadow | -| `0x1000` | RPC request mailbox | guest | emu thread| -| `0x1080` | RPC response mailbox | emu thread | guest | -| `0x2000` | MSI-X table (32 entries) | guest | guest | -| `0x3000` | MSI-X PBA (pending bits) | emu thread | guest | -| `0x110040` | GSP MAILBOX0 (boot args) | guest | emu thread| -| `0x110C00` | GSP doorbell | guest | emu thread| -| `0x300000` | VBIOS (1MB PROM window) | host init | guest | -| rest | Emulated register space | shadow | shadow | - -**Critical**: The GSP emulation thread polls the **shadow** exclusively. Touching the physical GPU's BAR0 (even via `ioremap`) caused the original GSP crash (fixed `f60ae7f`). - -### 4.2 BAR1 Emulated Vidmem (256 MB vzalloc) - -Guest VRAM aperture backed by host RAM. This is a **placeholder** — for real compute, this must be backed by physical GPU memory (see Phase 3 below). - -### 4.3 BAR3 Shadow (32 MB vzalloc) - -Instance memory window. **Must be shadowed** — mapping to the physical GPU's BAR3 would let guest writes corrupt host GPU instance memory (fixed `85d0668`). - -### 4.4 GSP Boot Emulation - -The guest driver boots "GSP" through falcon stages: - -1. **FWSEC FRTS** (GSP falcon CPUCTL) → auto-complete: HALTED + VBIOS_SCRATCH[0x0E]=0 + WPR2 ADDR -2. **Booter Load** (SEC2 falcon CPUCTL) → auto-complete: SEC2_MAILBOX0=0 + HALTED + RISCV_STATUS=1 -3. **GSP-RM msgq init** (libos args via MAILBOX0/1) → our emulation parses the queue chain and processes RPCs - -### 4.5 GSP-RM Emulation - -The guest's physical-path driver uses the **real GSP-RM msgq protocol** (not the old PV doorbell/BAR0-shared-mem). Our emulation: -- Parses `LibosMemoryRegionInitArgument` chain from GSP MAILBOX0/1 -- Finds the shared queue buffer in guest sysmem (129 non-contiguous 4K pages) -- Processes commands from the command queue -- Writes replies to the status queue -- Injects MSI-X interrupts - -Key RPC types handled: GSP_INIT_DONE, SET_SYSTEM_INFO, SET_REGISTRY, SET_GUEST_SYSTEM_INFO, GET_GSP_STATIC_INFO, GSP_RM_CONTROL (multiple sub-commands). - ---- - -## 5. Phased Implementation Roadmap - -### Phase 1: Correctness — GSP-RM Emulation Complete (IN PROGRESS) - -**Goal**: `nvidia-smi` works in the guest. Guest driver fully initializes. - -**What's needed**: Complete GSP-RM reply content for the RM init sequence: -- `INTERNAL_GET_DEVICE_INFO_TABLE` (0x20800a40) — 11 device entries -- `FIFO_GET_DEVICE_INFO_TABLE` (0x20801112) — 12 engine entries -- `INTERNAL_INTR_GET_KERNEL_TABLE` (0x20800a5c) — 8 entries + subtreeMap -- `GspStaticConfigInfo` — full TU104 config -- All remaining `GSP_RM_CONTROL` sub-commands - -**Oracle**: The host apt nvidia driver (570.211.01) has a **real GSP** managing the physical GPU. Use `tools/rmq/rmq` or `nvidia-smi` on host to dump ground-truth struct layouts. Compare against `kernel/gsp_rm_tu104.h`. - -**Current blocker**: `RmInitNvDevice` fails with `Cannot initialize the device (0x24:0x72:1099)`. The fn-10 FREE timeout during cleanup is characterized but not fully resolved. - -### Phase 2: Memory Model — Guest VRAM Backed by Real GPU Memory - -**Goal**: Guest CUDA memory allocations land in physical GPU VRAM (or DMA-coherent host memory accessible by the GPU). - -**What's needed**: -- Replace emulated vidmem (BAR1 host RAM) with DMA-mapped region the physical GPU can access -- Implement `SET_PAGE_DIRECTORY`, `UPDATE_PDE_2`, `ALLOC` handlers that map guest allocations to real GPU address space -- Ensure cache coherency between guest CPU and GPU DMA - -**Key challenge**: The guest's GPU VA space must map to physical GPU VRAM addresses. The host driver's GSP manages the physical GPU's memory. We'd need to either: -- (a) Cooperate with host GSP (share VRAM regions) — complex -- (b) Use a dedicated VRAM partition for the guest — requires host driver cooperation -- (c) Use system memory with DMA mapping (slower but simpler) — guest sees "unified memory" - -### Phase 3: Compute Forwarding — Guest Work Executes on Physical GPU - -**Goal**: CUDA kernels launched in the guest execute on the physical GPU. - -**What's needed**: -- Guest channel/pushbuffer writes are intercepted and forwarded to host GPU channels -- GPU context switching between host and guest contexts -- Completion interrupts relayed from host GPU to guest - -**Approach options**: -1. **GSP-RM translation**: Translate guest GSP-RM compute commands to host GSP-RM commands. The host GSP firmware already knows how to schedule work on the GPU. We'd be a "GSP proxy". -2. **Channel forwarding**: Intercept guest channel doorbell writes, translate guest pushbuffer contents, submit to host GPU channels. Requires deep understanding of GPU channel format. -3. **UVM integration**: Use the host's Unified Virtual Memory (UVM) to share CUDA memory between host and guest. Launch kernels via host CUDA APIs on behalf of guest. - -**Most promising**: Option 1 (GSP-RM translation). The guest driver sends GSP-RM commands for compute (ALLOC, MAP, SCHEDULE). Our emulation translates these to equivalent host GSP-RM commands via the host driver. The host GSP firmware executes them on the physical GPU. Results are relayed back. - -### Phase 4: Performance Optimization - -**Goal**: < 5% overhead vs native for CUDA workloads. - -**What's needed**: -- Optimize GSP-RM translation latency (batch commands, avoid unnecessary copies) -- Direct DMA mapping for large memory transfers -- Interrupt coalescing for high-frequency completions -- Profile and eliminate hot-path bottlenecks (doorbell polling, shared memory access) - -### Phase 5: Graphics & Video - -**Goal**: OpenGL/Vulkan rendering, NVENC/NVDEC video. - -**What's needed**: -- Display surface management (virtual display or forwarded framebuffer) -- Graphics pipeline forwarding (similar to compute forwarding) -- Video engine access (NVENC/NVDEC on physical GPU) - ---- - -## 6. Testing Strategy - -### Correctness Testing (Phase 1) -- [x] Host nvidia-smi works with open-iov-vfio loaded -- [x] mdev create/remove lifecycle clean (no hang, no host GPU corruption) -- [x] QEMU guest boots with vGPU attached -- [x] Guest PCI probe detects vGPU (TU104 [10de:1ed3]) -- [x] Guest nvidia driver loads and creates /dev/nvidia0 -- [ ] Guest RmInitAdapter completes (RmInitNvDevice succeeds) -- [ ] Guest nvidia-smi shows GPU -- [ ] Guest CUDA context creation - -### Performance Testing (Phases 2-4) -- [ ] GSP-RM round-trip latency < 10μs -- [ ] CUDA kernel launch overhead < 5% vs native -- [ ] Memory bandwidth within 10% of native -- [ ] Graphics frame time within 5% of native - ---- - -## 7. Key Files - -| File | Role | -|------|------| -| `kernel/gsp_emulation.c/h` | GSP emulation thread, falcon boot auto-complete, MSI-X injection | -| `kernel/gsp_rm_emulation.c/h` | GSP-RM msgq protocol emulation (guest sysmem queues) | -| `kernel/gsp_rm_tu104.h` | TU104 hardware tables (engines, interrupts, falcons, device info) | -| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations | -| `kernel/bar_emulation.c/h` | BAR0 shadow allocation, MSI-X shadow state | -| `kernel/bar_init.c` | BAR init (shadow vidmem, BAR3 shadow, physical ioremap for reference) | -| `kernel/main.c` | VFIO ops, mdev create/remove, PCI config space, VFIO mmap | -| `kernel/memory.c` | DMA mapping, VRAM management | -| `kernel/irq.c` | MSI-X/MSI setup and injection | -| `tools/rmq/rmq.c` | Host RM API client (oracle for GSP-RM ground truth) | - ---- - -## 8. Host Stability Guarantees (2026-09-22) - -After the host-stability fix (`85d0668`), the following are guaranteed: - -1. **open-iov-vfio.ko loads/unloads cleanly** with apt nvidia driver present -2. **Host nvidia-smi works throughout** the full lifecycle (load, mdev, QEMU, unload) -3. **No host GPU corruption** — no WPR2 stuck, no GSP crash, no module hang -4. **Guest driver probe is safe** — even when RmInitNvDevice fails, host is unaffected -5. **mdev create/remove is clean** — gsp_emu thread starts/stops properly - -**Root causes that were fixed**: -- Submodule nvidia.ko conflict with apt nvidia → skip submodule -- BAR3 physical mapping to guest → vzalloc'd shadow -- GSP-RM teardown garbage declaring GSP death → quiet stop polling - ---- - -*Document Version: 2.0 (Emulate vs Pass-Through Architecture)* -*Updated: 2026-09-22 (post host-stability fix)* -*Repository: ~/Git/integdev_gpu_drv/open-iov* -*Git Commit: 85d0668 (host stability fix)* diff --git a/docs/refactored-implementation-to-do.md b/docs/refactored-implementation-to-do.md deleted file mode 100644 index e8ddd3c..0000000 --- a/docs/refactored-implementation-to-do.md +++ /dev/null @@ -1,516 +0,0 @@ -# Open-IOV vGPU-IO Refactored Implementation TODO List - -> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept -> as a record. Current entry point: [docs/README.md](README.md). For the -> current architecture and work list see -> [03-architecture/01](03-architecture/01-system-overview.md), -> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md) and -> [04-implementation/01](04-implementation/01-known-knowns-and-unknowns.md). - -## Overview -This document tracks the granular implementation steps to refactor Open-IOV from guest-side emulation to **host-side GSP emulation**, enabling unmodified guest NVIDIA drivers to run with full functionality. - -**Restore Point**: Git commit `f9ed5d8` - complete current stack before refactoring -**Target**: Host-only GSP emulation in `open-iov-vfio.ko`, guest runs pure unmodified `nvidia.ko` - ---- - -## Phase 0: Repository & Infrastructure Setup - -### 0.1 Git Branching -- [ ] `git checkout -b refactor/host-gsp-emulation` from `f9ed5d8` -- [ ] Create `docs/old/` with previous ARCHITECTURE.md (DONE) -- [ ] Create `docs/refactored-architecture.md` (DONE) -- [ ] Create this TODO document - -### 0.2 Build System Verification -- [ ] Verify current build works: `make clean && make` in open-iov root -- [ ] Verify all 3 kernel modules build: `nvidia.ko`, `open-iov-vfio.ko`, `gsp-vgpu.ko` -- [ ] Verify userspace builds: `libiov.so`, `example_vgpu` -- [ ] Document current build commands for regression testing - -### 0.3 Test Infrastructure -- [x] Create `scripts/test_gsp_emu.sh` - automated GSP emulation unit tests -- [x] Create `scripts/test_guest_driver.sh` - guest driver validation suite -- [x] Create `scripts/test_cuda_workloads.sh` - CUDA workload validation -- [x] Create `scripts/test_graphics_workloads.sh` - graphics validation -- [x] Create `scripts/test_windows_guest.sh` - Windows guest validation -- [x] Create `scripts/test_vgpu_stack.sh` - comprehensive test runner -- [ ] Set up CI pipeline for incremental testing - ---- - -## Phase 1: Host Kernel - GSP Emulation Engine (Core) - -### 1.1 New File: `kernel/gsp_emulation.h` -- [x] Define `struct gsp_emu_context` with: - - [x] `struct nv_vgpu_device *vgpu` - - [x] `struct task_struct *thread` - - [x] `void __iomem *bar0_vaddr`, `*doorbell_reg`, `*shared_mem` - - [x] `struct gsp_client *rpc_client` - - [x] `spinlock_t lock` - - [x] `bool running` - - [x] Statistics: `u64 processed_rpcs`, `failed_rpcs`, `total_latency_ns` -- [x] Function prototypes: - - [x] `int gsp_emu_init(struct nv_vgpu_device *vgpu)` - - [x] `void gsp_emu_cleanup(struct nv_vgpu_device *vgpu)` - - [x] `int gsp_emu_process_rpc(struct gsp_emu_context *ctx)` - - [x] `void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id)` - -### 1.2 New File: `kernel/gsp_emulation.c` -- [x] Implement `gsp_emu_thread_fn()` - per-mdev polling thread - - [x] Read doorbell at `bar0_vaddr + 0x110C00` - - [x] ACK doorbell immediately (write 0) - - [x] Call `gsp_emu_process_rpc()` - - [x] Adaptive polling: busy-wait first 100μs, then `usleep_range(50, 100)` - - [x] Handle `kthread_should_stop()` for clean shutdown -- [x] Implement `gsp_emu_init()` - - [x] Allocate `gsp_emu_context` - - [x] Map BAR0, locate doorbell reg + shared mem offset (0x1000) - - [x] Allocate shared memory (4KB, page-aligned, WC mapping) - - [x] Map shared memory into guest BAR0 at 0x1000 - - [x] Initialize `gsp_client` via `gsp_client_init()` (reuse existing) - - [x] Start kthread `gsp_emu_thread_fn` -- [x] Implement `gsp_emu_cleanup()` - - [x] Stop kthread - - [x] Free shared memory - - [x] Call `gsp_client_fini()` - - [x] Free context -- [x] Implement `gsp_emu_process_rpc()` - dispatch all 12 RPC types - - [x] Copy request from shared memory (offset 0x0000) - - [x] Validate seq_id != 0 - - [x] Switch on `req.type` → call appropriate `gsp_rpc_*` function - - [x] Write response to shared memory (offset 0x0080) - - [x] Call `gsp_emu_inject_completion_interrupt()` -- [x] Implement `gsp_emu_inject_completion_interrupt()` - - [x] Write to MSI-X PBA at `bar0_vaddr + 0x3000` (vector 0) - - [x] Signal eventfd: `eventfd_signal(vgpu->msix_trigger[0])` - -### 1.3 Modify `kernel/main.c` -- [x] Add `struct gsp_emu_context *gsp_emu` to `struct nv_vgpu_device` (reused `gsp_comm_ctx`) -- [x] In `nv_vgpu_device_init_work()`: call `gsp_emu_init(vgpu)` after `nv_vgpu_gsp_init()` -- [x] In `nv_vgpu_device_cleanup_work()`: call `gsp_emu_cleanup(vgpu)` before `nv_vgpu_gsp_cleanup()` -- [x] Include `gsp_emulation.h` - -### 1.4 Modify `kernel/Makefile` -- [x] Add `gsp_emulation.o` to `open-iov-vfio-y` or module objects -- [x] Verify build: `make clean && make` - -### 1.5 Unit Tests for GSP Emulation (kernel) -- [x] Create `kernel/tests/gsp_emulation_test.c` -- [x] Create `kernel/tests/Makefile` -- [x] Test ALLOC/FREE round-trip -- [x] Test MAP/UNMAP with various sizes/alignments -- [x] Test PIN/UNPIN returns valid DMA addresses -- [x] Test SCHEDULE/YIELD completes -- [x] Test BAR_TRAP read/write emulation -- [x] Test NOTIFY register/unregister -- [x] Test MIGRATE 4-phase state machine -- [x] Test concurrent RPCs from multiple threads -- [x] Test error handling (invalid seq_id, bad payload) -- [x] Add to `kernel/tests/Makefile` - ---- - -## Phase 2: PCI Config Space & BAR Enhancements - -### 2.1 Enhance PCI Config Space (`kernel/main.c`) -- [x] Verify MSI-X capability at 0x70 points to Table@BAR0+0x2000, PBA@BAR0+0x3000 -- [x] Ensure `VFIO_PCI_CONFIG_REGION_INDEX` returns full 256B config space -- [x] Add config space write handlers for: - - [x] Command register (0x04): Memory/IO/BusMaster enable - - [x] MSI-X Message Control (0x72): Enable/disable, table size - - [x] MSI-X Table Offset (0x74-0x77): Verify BAR0+0x2000 - - [x] MSI-X PBA Offset (0x78-0x7B): Verify BAR0+0x3000 - -### 2.2 BAR Region VFIO Mapping (`kernel/bar_emulation.c`) -- [x] Implement `VFIO_PCI_BAR0_REGION_INDEX` → 16MB emulated config space -- [x] Implement `VFIO_PCI_BAR1_REGION_INDEX` → 256MB VRAM (backed by host allocation) -- [x] Implement `VFIO_PCI_BAR2_REGION_INDEX` → 32MB reserved -- [x] Implement `VFIO_PCI_BAR3_REGION_INDEX` → 32MB reserved -- [x] Implement `VFIO_PCI_MSIX_REGION_INDEX` → MSI-X Table (0x2000) + PBA (0x3000) -- [x] Ensure BAR0 mmap includes doorbell @ 0x110C00 and shared mem @ 0x1000 - -### 2.3 MSI-X Table/PBA Emulation -- [x] Allocate MSI-X table (32 entries × 16 bytes = 512 bytes) in BAR0 at 0x2000 -- [x] Allocate PBA (32 bits = 4 bytes, aligned to 64 bytes) in BAR0 at 0x3000 -- [x] Guest writes to MSI-X table entries → host stores for interrupt injection -- [x] Guest reads PBA → host returns pending bits - ---- - -## Phase 3: GSP RPC Handler Completion - -### 3.1 Verify All 12 RPC Handlers in `gsp_rpc_client.c` -- [x] ALLOC (0x01): VRAM allocation via host nvidia.ko -- [x] FREE (0x02): Free allocation -- [x] MAP (0x03): Map to GPU VA -- [x] UNMAP (0x04): Unmap GPU VA -- [x] PIN (0x05): Pin for DMA, return DMA address -- [x] UNPIN (0x06): Unpin pages -- [x] SCHEDULE (0x07): Submit work to local workqueue -- [x] YIELD (0x08): No-op success -- [x] NOTIFY (0x09): Async notification registration -- [x] BAR_TRAP (0x0A): Shadow register read/write -- [x] MIGRATE (0x0B): 4-phase protocol -- [x] DEBUG (0xFF): Firmware introspection - -### 3.2 Enhance Handler Implementations -- [x] ALLOC: Use host page allocation + DMA mapping -- [x] MAP: GPU VA mapping management -- [x] PIN: Return DMA address from mapping -- [x] BAR_TRAP: Use bar_emulation.c shadow memory -- [x] MIGRATE: Use migration.c 4-phase state machine -- [x] NOTIFY: Use gsp_rpc_notify_register() + eventfd for guest delivery - -### 3.3 Add RPC Validation & Error Handling -- [x] Sequence ID tracking (detect duplicates, gaps) -- [x] Payload size validation per RPC type -- [x] Timeout handling (guest watchdog) -- [x] Proper error codes in response.status - ---- - -## Phase 4: Portable Architecture Implementation - -### 4.1 Firmware Analyzer (`kernel/firmware_analyzer.c`) -- [x] Compute firmware CRC32 and SHA256 signatures -- [x] Pattern matching for RPC dispatcher discovery -- [x] Pattern matching for jump table discovery -- [x] Pattern matching for ALLOC handler discovery -- [x] Known firmware signature database -- [x] Auto-discovery fallback for unknown firmware -- [x] `fw_analyzer_analyze()` - analyze firmware blob -- [x] `fw_analyzer_load_and_analyze()` - load and analyze from filesystem -- [x] `fw_analyzer_print_result()` - debug output - -### 4.2 Dynamic RPC Dispatcher (`kernel/dynamic_rpc_dispatcher.c`) -- [x] Version-specific interface registry (550, 560, 570, 580) -- [x] Native firmware handler function pointers -- [x] Emulation fallback interface -- [x] Automatic interface selection based on firmware analysis -- [x] Per-RPC-type emulation toggle -- [x] Dispatcher API for all 12 RPC types -- [x] `dynamic_dispatcher_init()` - initialize with firmware path -- [x] `dynamic_dispatcher_get_interface()` - get current interface -- [x] `dynamic_dispatcher_use_emulation()` - check if emulation used for type - -### 4.3 Hardware Architecture Layer (`kernel/architecture/`) -- [x] `architecture.h` - hardware abstraction API -- [x] `architecture.c` - architecture implementations -- [x] Turing architecture (TU102/TU104) config -- [x] Ampere architecture (GA100/GA102/GA104) config -- [x] Blackwell architecture (GB100/GB202) config -- [x] Architecture lookup by name and PCI ID -- [x] BAR configuration queries -- [x] Doorbell/shared memory offset queries -- [x] MSI-X vector count queries -- [x] Feature support queries (MIG, SR-IOV) - -### 4.4 Driver Version Adaptation Layer (`kernel/version/`) -- [x] `version.h` - version adaptation API -- [x] `version.c` - version implementations -- [x] Version 550 interface -- [x] Version 560 interface -- [x] Version 570 interface -- [x] Version 580 interface -- [x] Feature detection per version -- [x] Interface lookup by version string - -### 4.5 Declarative Configuration Files -- [x] `config/hardware/turing.yaml` - Turing hardware config -- [x] `config/hardware/ampere.yaml` - Ampere hardware config -- [x] `config/hardware/blackwell.yaml` - Blackwell hardware config -- [x] `config/versions/550.yaml` - Driver 550 config -- [x] `config/versions/560.yaml` - Driver 560 config -- [x] `config/versions/570.yaml` - Driver 570 config -- [x] `config/versions/580.yaml` - Driver 580 config - ---- - -## Phase 5: Guest Driver Integration Testing - -### 4.1 Host Module Loading Sequence -- [ ] `sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko` -- [ ] `sudo insmod kernel/open-iov-vfio.ko` -- [ ] Verify mdev types appear: `ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/` -- [ ] Create mdev: `echo <UUID> | sudo tee /sys/bus/pci/devices/.../mdev_supported_types/nvidia-vgpu-turing/create` - -### 4.2 QEMU Launch with mdev -- [ ] `qemu-system-x86_64 -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off` -- [ ] Verify guest sees PCI device: `lspci -d 10de:` → should show 10de:1ed3 -- [ ] Verify BARs: `lspci -v -d 10de:` → BAR0=16MB, BAR1=256MB, BAR2=32MB, BAR3=32MB -- [ ] Verify MSI-X: `lspci -v -d 10de:` → MSI-X capability present - -### 4.3 Guest NVIDIA Driver Installation -- [ ] Install OpenRM 570.144 in guest: `./NVIDIA-Linux-x86_64-570.144.run --silent --dkms` -- [ ] Install firmware: Copy `bootloader`, `booter_load`, `gsp` binaries to `/lib/firmware/nvidia/tu102/gsp/` -- [ ] Load driver: `sudo modprobe nvidia` -- [ ] Verify: `nvidia-smi` shows vGPU (T4 or equivalent) -- [ ] Verify: `dmesg | grep -i nvidia` shows successful probe - -### 4.4 Basic Functionality Tests -- [ ] `nvidia-smi -q` shows all fields populated -- [ ] `nvidia-smi dmon` shows utilization counters -- [ ] `nvidia-smi mig` shows MIG not supported (expected for vGPU) -- [ ] `cat /proc/driver/nvidia/gpus/0000:00:00.0/information` shows correct info - ---- - -## Phase 5: CUDA Workload Validation - -### 5.1 CUDA SDK Tests -- [ ] Compile and run `vectorAdd` sample -- [ ] Compile and run `matrixMul` sample -- [ ] Compile and run `simpleCUBLAS` sample -- [ ] Compile and run `simpleCUFFT` sample -- [ ] Verify numerical correctness vs CPU reference - -### 5.2 cuBLAS / cuDNN Tests -- [ ] SGEMM/DGEMM various sizes (64x64 to 4096x4096) -- [ ] Batched GEMM -- [ ] cuDNN convolution forward/backward -- [ ] cuDNN RNN/LSTM -- [ ] Performance within 10% of native - -### 5.3 PyTorch / TensorFlow Integration -- [ ] PyTorch: `torch.cuda.is_available()`, tensor ops, autograd -- [ ] PyTorch: Training loop (ResNet-50, BERT-small) -- [ ] TensorFlow: `tf.config.list_physical_devices('GPU')` -- [ ] TensorFlow: Keras model training -- [ ] Mixed precision (FP16/BF16) works - -### 5.4 Multi-GPU / NCCL Tests -- [ ] Create 2+ vGPUs on same physical GPU -- [ ] NCCL all-reduce across vGPUs -- [ ] NCCL broadcast/reduce -- [ ] Multi-process CUDA (MPS) if supported - -### 5.5 CUDA Graphs & Streams -- [ ] CUDA Graph capture/replay -- [ ] Multiple streams concurrent execution -- [ ] Stream priorities -- [ ] Event synchronization - ---- - -## Phase 6: Graphics Workload Validation - -### 6.1 Vulkan Tests -- [ ] `vulkaninfo` shows GPU properties -- [ ] `vkcube` renders correctly -- [ ] `vkcubepp` (Vulkan 1.1+) renders -- [ ] Sascha Willems Vulkan samples (deferred, compute, ray tracing if supported) -- [ ] VK_KHR_swapchain, VK_KHR_dedicated_allocation work - -### 6.2 OpenGL Tests -- [ ] `glxinfo | grep -i nvidia` shows correct renderer -- [ ] `glxgears` runs at expected FPS -- [ ] `glmark2` benchmark completes -- [ ] Piglit test suite (subset) passes - -### 6.3 Video Encode/Decode -- [ ] FFmpeg + NVENC: H.264 encode 1080p60 -- [ ] FFmpeg + NVENC: HEVC encode 4K30 -- [ ] FFmpeg + NVDEC: H.264 decode 1080p60 -- [ ] FFmpeg + NVDEC: HEVC decode 4K30 -- [ ] GStreamer nvh264enc/nvh265enc elements -- [ ] GStreamer nvh264dec/nvh265dec elements - -### 6.4 Windows Guest Graphics (Separate Track) -- [ ] Windows 11 guest with NVIDIA 570.144 driver -- [ ] Device Manager: "NVIDIA T4" (or equivalent) with no error codes -- [ ] GPU-Z / CUDA-Z detect vGPU correctly -- [ ] D3D11: `D3D11CreateDevice` succeeds, render triangle -- [ ] D3D12: `D3D12CreateDevice` succeeds, render triangle -- [ ] Vulkan on Windows: `vkCreateInstance`, render -- [ ] DXGI swapchain present works -- [ ] Video encode via Media Foundation (MFT) -- [ ] Video decode via Media Foundation (MFT) - ---- - -## Phase 7: Performance Benchmarking - -### 7.1 Microbenchmarks -- [ ] GSP RPC round-trip latency (target < 10μs) -- [ ] BAR register access latency (target < 2μs) -- [ ] VRAM alloc/free latency (target < 50μs) -- [ ] VRAM map/unmap latency (target < 20μs) -- [ ] DMA pin/unpin latency (target < 30μs) -- [ ] MSI-X injection latency (target < 5μs) - -### 7.2 Macrobenchmarks -- [ ] CUDA kernel launch overhead (target < 5% vs native) -- [ ] cuBLAS SGEMM 4096x4096 (target > 90% native GFLOPS) -- [ ] cuDNN ResNet-50 training step (target > 85% native) -- [ ] Vulkan frame time (target < 2ms @ 1080p) -- [ ] NVENC 1080p60 encode latency (target < 5ms) -- [ ] NVDEC 1080p60 decode latency (target < 3ms) - -### 7.3 Stress Tests -- [ ] 24-hour CUDA compute stress (no errors, no memory leaks) -- [ ] 24-hour graphics stress (Unigine Heaven/Valley loop) -- [ ] 1000x VF create/destroy cycles -- [ ] Concurrent CUDA + graphics workloads -- [ ] Memory pressure test (allocate 90% VRAM, verify OOM handling) - ---- - -## Phase 8: Live Migration Validation - -### 8.1 Migration Protocol Tests -- [ ] QUERY phase: source reports capabilities -- [ ] PREPARE phase: destination allocates resources -- [ ] TRANSFER phase: iterative VRAM + GSP state copy -- [ ] COMMIT phase: cutover, guest resumes on destination -- [ ] CLEANUP phase: source releases resources - -### 8.2 Migration Under Load -- [ ] Migrate idle vGPU (downtime < 50ms) -- [ ] Migrate running CUDA kernel (downtime < 100ms) -- [ ] Migrate running graphics (no visual artifacts) -- [ ] Migrate with pinned DMA buffers -- [ ] Verify post-migration CUDA/graphics work correctly - -### 8.3 Migration State Verification -- [ ] GSP context (registers, PC, stack) restored -- [ ] Page tables identical -- [ ] VRAM contents identical (checksum) -- [ ] BAR register state identical -- [ ] MSI-X table/PBA state identical - ---- - -## Phase 9: Windows Guest Validation - -### 9.1 Windows Guest Setup -- [ ] Windows 10 22H2 / Windows 11 23H2 guest -- [ ] VirtIO drivers for network/storage -- [ ] QEMU guest agent -- [ ] UEFI firmware (OVMF with TPM) - -### 9.2 NVIDIA Driver Installation -- [ ] Download 570.144 Windows driver -- [ ] Install with `setup.exe -s -noreboot` (silent) -- [ ] Reboot guest -- [ ] Verify in Device Manager: "NVIDIA T4" (Code 0 = working) - -### 9.3 Windows Compute Tests -- [ ] `nvidia-smi.exe` shows vGPU -- [ ] CUDA-Z detects GPU, runs device query -- [ ] `vectorAdd.exe` (compiled in guest) executes -- [ ] PyTorch/TensorFlow in Windows (WSL2 or native) - -### 9.4 Windows Graphics Tests -- [ ] `dxdiag` shows correct GPU info -- [ ] D3D11 tutorial samples render -- [ ] D3D12 tutorial samples render -- [ ] Vulkan SDK samples run -- [ ] FurMark / 3DMark stress test - -### 9.5 Windows Video Tests -- [ ] Media Foundation H.264/HEVC encode -- [ ] Media Foundation H.264/HEVC decode -- [ ] NVENC API direct usage -- [ ] NVDEC API direct usage - ---- - -## Phase 10: Reliability & Production Hardening - -### 10.1 Error Handling & Recovery -- [ ] Guest driver crash → host cleans up mdev state -- [ ] Host kernel oops in GSP emulation → watchdog restarts thread -- [ ] Physical GPU reset → host reinitializes GSP emulation -- [ ] IOMMU fault → proper error to guest -- [ ] VRAM OOM → proper CUDA_OUT_OF_MEMORY to guest - -### 10.2 Monitoring & Observability -- [ ] Debugfs interface: `/sys/kernel/debug/open-iov-vfio/` - - [ ] Per-mdev RPC statistics - - [ ] Latency histograms - - [ ] Error counters - - [ ] GSP emulation thread CPU usage -- [ ] Prometheus metrics exporter (optional) -- [ ] Tracepoints for RPC processing - -### 10.3 Security Hardening -- [ ] Validate all guest inputs (seq_id, payload sizes, pointers) -- [ ] Isolate mdev memory (no cross-VF access) -- [ ] Rate-limit doorbell writes (prevent DoS) -- [ ] Audit VFIO ioctl permissions - -### 10.4 Documentation & Handoff -- [ ] Update `docs/refactored-architecture.md` with as-built details -- [ ] Create `docs/OPERATIONS.md` for deployment -- [ ] Create `docs/TROUBLESHOOTING.md` for common issues -- [ ] Record all firmware addresses used -- [ ] Document test results for each phase - ---- - -## Success Criteria Checklist - -### Minimum Viable Product (Phases 0-4) -- [ ] Host loads `open-iov-vfio.ko` with GSP emulation thread -- [ ] mdev creates successfully with correct PCI config space -- [ ] Guest `nvidia.ko` loads, probes vGPU successfully -- [ ] `nvidia-smi` in guest shows vGPU -- [ ] Simple CUDA kernel (`vectorAdd`) executes correctly - -### Compute Ready (Phases 5-6) -- [ ] cuBLAS/cuDNN workloads pass numerical verification -- [ ] PyTorch/TensorFlow training works -- [ ] Vulkan/OpenGL rendering works -- [ ] Video encode/decode works -- [ ] Performance > 85% native for compute, > 90% for graphics - -### Production Ready (Phases 7-10) -- [ ] 24-hour stress tests pass -- [ ] Live migration works under load -- [ ] Windows guest fully functional -- [ ] Monitoring/observability in place -- [ ] Security audit passed -- [ ] Documentation complete - ---- - -## Dependencies & Prerequisites - -| Component | Version | Source | -|-----------|---------|--------| -| Linux Kernel | 6.8+ | Host distro | -| NVIDIA OpenRM | 570.144 | `nvidia-open-gpu-kernel-modules` submodule | -| QEMU | 8.0+ | Host distro / source | -| libvfio-user | Latest | `libvfio-user` repo | -| CUDA Toolkit | 12.4+ | Guest (for compilation) | -| Vulkan SDK | 1.3+ | Guest | -| Windows 10/11 | 22H2/23H2 | Guest ISO | - ---- - -## Tracking - -| Phase | Status | Start Date | End Date | Notes | -|-------|--------|------------|----------|-------| -| 0: Setup | ✅ DONE | - | - | Git repo, docs, restore point | -| 1: GSP Emulation Engine | ✅ DONE | | | Core implementation complete | -| 2: PCI/BAR Enhancements | ✅ DONE | | | Config space, MSI-X, BAR regions | -| 3: RPC Handler Completion | ✅ DONE | | | All 12 RPC types with emulation | -| 4: Portable Architecture | ✅ DONE | | | Firmware analyzer, dispatcher, arch/version layers, configs | -| 5: Guest Driver Integration | 🔄 IN PROGRESS | | | Requires reboot to test | -| 6: CUDA Validation | ⏳ PENDING | | | | -| 7: Graphics Validation | ⏳ PENDING | | | | -| 8: Performance Benchmarks | ⏳ PENDING | | | | -| 9: Migration Validation | ⏳ PENDING | | | | -| 10: Windows Guest | ⏳ PENDING | | | | -| 11: Production Hardening | ⏳ PENDING | | | | -| 10: Production Hardening | ⏳ PENDING | | | | - ---- - -*TODO List Version: 1.0* -*Repository: $OPEN_IOV_ROOT* -*Branch: refactor/host-gsp-emulation* -*Base Commit: f9ed5d8*
\ No newline at end of file diff --git a/docs/tools/rmtool.md b/docs/tools/rmtool.md index a7ec2c6..93140f1 100644 --- a/docs/tools/rmtool.md +++ b/docs/tools/rmtool.md @@ -1,5 +1,10 @@ # rmtool - RM API Swiss Army Knife +> **Status: planned / not implemented.** There is no `tools/rmtool/` directory +> or build target today. The implemented RM-API tooling is the `rmq` subset +> (`tools/rmq/`, the host RM-API oracle); treat this page as a design sketch, +> not a user guide. See [tools/README.md](README.md). + ## Overview `rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built on top of driver components studied through reverse engineering and decompilation (referenced by component). diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md index 6408268..6f9c047 100644 --- a/docs/tools/rmtop.md +++ b/docs/tools/rmtop.md @@ -1,5 +1,10 @@ # rmtop - GPU htop Equivalent +> **Status: planned / not implemented.** There is no `tools/rmtop/` directory +> or build target today. The implemented RM-API tooling is the `rmq` subset +> (`tools/rmq/`, the host RM-API oracle); treat this page as a design sketch, +> not a user guide. See [tools/README.md](README.md). + ## Overview `rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry. diff --git a/scripts/check-docs-links.py b/scripts/check-docs-links.py new file mode 100755 index 0000000..fcb39ca --- /dev/null +++ b/scripts/check-docs-links.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Check relative links in the project's tracked markdown documentation. + +Walks docs/, wiki/ and userspace/cuda/, extracts markdown links, and fails if +a relative target does not exist. External links (http/https/mailto/tel/ftp/ +data/javascript), pure anchors and links inside fenced code blocks are +skipped. + +The raw imported wiki HTML dumps and the archived wiki copy are excluded: +their markup is not ours and their link shape is not representative. + +Usage: + python3 scripts/check-docs-links.py +""" + +import os +import re +import subprocess +import sys +from urllib.parse import unquote + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +WATCHED_ROOTS = ("docs", "wiki", "userspace/cuda") + +EXCLUDED_FILES = { + "wiki/GPU_Driver_Internals.md", + "wiki/GPU_Firmware.md", + "wiki/GPU_SBOM.md", + "wiki/Virtual_IO_Internals.md", +} +EXCLUDED_PREFIXES = ("wiki/open-iov.org/",) + +SKIPPED_SCHEMES = ( + "http://", + "https://", + "mailto:", + "tel:", + "ftp://", + "ftps://", + "data:", + "javascript:", +) + +LINK_RE = re.compile(r"!?\[[^\]]*\]\(\s*([^)]+?)\s*\)") + + +def tracked_markdown_files(): + """Return repo-relative paths of tracked markdown files (git first).""" + try: + out = subprocess.run( + ["git", "ls-files", "--", *WATCHED_ROOTS], + cwd=REPO_ROOT, + check=True, + capture_output=True, + text=True, + ).stdout + except (OSError, subprocess.CalledProcessError): + out = "" + if out.strip(): + return [line for line in out.splitlines() if line.endswith(".md")] + + files = [] + for root in WATCHED_ROOTS: + for dirpath, _dirnames, filenames in os.walk(os.path.join(REPO_ROOT, root)): + for name in filenames: + if name.endswith(".md"): + full = os.path.join(dirpath, name) + files.append(os.path.relpath(full, REPO_ROOT)) + return files + + +def excluded(relpath): + if relpath in EXCLUDED_FILES: + return True + return relpath.startswith(EXCLUDED_PREFIXES) + + +def strip_code_blocks(text): + """Blank out fenced code blocks so example links are not checked.""" + lines = [] + fence = None + for line in text.splitlines(): + stripped = line.lstrip() + marker = stripped[:3] + if fence is None: + if marker in ("```", "~~~"): + fence = marker + lines.append("") + continue + lines.append(line) + else: + if stripped.startswith(fence): + fence = None + lines.append("") + return "\n".join(lines) + + +def normalize_target(raw): + target = raw.strip() + if target.startswith("<") and target.endswith(">"): + target = target[1:-1].strip() + if not target: + return None + if target.startswith("#"): + return None + if target.lower().startswith(SKIPPED_SCHEMES): + return None + # Drop an anchor and any markdown title, then percent-decode. + target = target.split()[0] if " " in target else target + target = target.split("#", 1)[0] + target = unquote(target) + if not target: + return None + return target + + +def check_file(relpath): + full = os.path.join(REPO_ROOT, relpath) + with open(full, "r", encoding="utf-8", errors="replace") as handle: + text = strip_code_blocks(handle.read()) + + problems = [] + in_fence = False + for lineno, line in enumerate(text.splitlines(), start=1): + if line.lstrip()[:3] in ("```", "~~~"): + in_fence = not in_fence + continue + if in_fence: + continue + for raw in LINK_RE.findall(line): + target = normalize_target(raw) + if target is None: + continue + if target.startswith("/"): + resolved = os.path.join(REPO_ROOT, target.lstrip("/")) + else: + resolved = os.path.normpath( + os.path.join(REPO_ROOT, os.path.dirname(relpath), target) + ) + if not os.path.exists(resolved): + problems.append((lineno, target, os.path.relpath(resolved, REPO_ROOT))) + return problems + + +def main(): + files = sorted( + relpath + for relpath in tracked_markdown_files() + if not excluded(relpath) + ) + failures = 0 + for relpath in files: + for lineno, target, resolved in check_file(relpath): + failures += 1 + print(f"{relpath}:{lineno}: broken link: {target} (resolved: {resolved})") + + if failures: + print(f"\n{failures} broken relative link(s) in {len(files)} file(s).") + return 1 + print(f"OK: {len(files)} markdown file(s) checked, no broken relative links.") + return 0 + + +if __name__ == "__main__": + sys.exit(main())
\ No newline at end of file diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md new file mode 100644 index 0000000..3fa5d41 --- /dev/null +++ b/wiki/CUDA-Userspace-Driver.md @@ -0,0 +1,245 @@ +# CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive + +> **Status (2026-09-23, honest).** The analysis covers the whole library for +> driver 570.144 and the tooling is version-agnostic. The *control plane* gets +> the stock guest stack through `cuInit`, device enumeration and into +> `cuCtxCreate`, which is blocked at `rc=999` because the emulated engine list +> is missing the async copy engine `COPY14`. The *data plane* is at the +> software memop bridge: the host executes the memsets, copies and semaphore +> releases it decodes from guest push buffers, and **no CUDA kernel has been +> forwarded yet**. Canonical detail: [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) +> through `08`; reading order +> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); +> front door [`../docs/README.md`](../docs/README.md). + +## 1. What it is + +`libcuda.so.1.1` is the userspace half of the CUDA driver: the library an +application or language runtime loads to talk to a GPU. For 570.144 it is a +**single, fully stripped, position-independent shared object of about 71 MB**: + +| Property | Value | +|---|---| +| Exports | 659 (public `cu*` API plus internal/legacy entry points) | +| Imports | 243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path | +| `.text` | `0x179ec0`–`0xe94c63` (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables) | +| `.rodata` | ~53 MB — error strings, dispatch/version tables, **embedded device code** | +| CUDA note | CUDA 12.8; supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` | + +Two entry surfaces coexist: the **public driver API** (`cuInit`, +`cuCtxCreate_v2`, `cuMemAlloc_v2`, `cuLaunchKernel`, …) and the +**entry-table API** — `cuGetExportTable`, a 128-bit-UUID-keyed lookup that +hands internal tables to the CUDA runtime (`libcudart`), tooling and +interposition libraries. Tables are versioned by a leading `struct_size` +field and are append-only. + +The library embeds **device code** — cubins for memcpy/memset kernels, the +graph scheduler, syscall trampolines and preemption-handler assembly — and +loads it through its own module machinery, so a stripped `libcuda.so` still +contains many small ELF images in `.rodata`. The CUDA *runtime* (`libcudart`, +the `cuda*` API) is a separate stock library that bottoms out in `libcuda`; it +also calls through the entry tables, so a guest failure in a `cuda*` call may +reach driver code through a different door than the public `cu*` entry point. + +**Open-IOV requires this library unmodified.** In the guest it links against +the guest's stock `nvidia.ko` / `nvidia-uvm.ko` exactly as on bare metal; there +is no `LD_PRELOAD` shim and no patch. Instrumentation uses `ptrace` or +kernel-side tracing, because the RM client issues raw `syscall(SYS_ioctl)` +rather than libc `ioctl`. + +## 2. Layer model + +The library is strictly layered, and the boundaries are where the emulation +interfaces live (condensed from `docs/01` and `docs/02`): + +``` +application / libcudart + | public cu* API or UUID entry tables + v +generated wrappers -> API layer -> CUDA internal engine + |-- driver-model layer (RM) -> /dev/nvidia* ioctls + |-- hardware layer (QMD/method encoding) + | -> GPU queues, doorbells + |-- UVM userspace (managed memory) -> /dev/nvidia-uvm + '-- JIT/ELF (PTX->SASS, cubin loading) +kernel driver + GSP-RM + GPU +``` + +* **Generated wrappers and API layer.** A master export list (name, platform + flags, CUDA version introduced) generates the exported symbols, wrappers, + tool callbacks and `cuGetProcAddress` tables (`_vN`/`_ptsz` variants are part + of that data). Below it, the hand-written layer reads the per-thread context + stack, validates state, takes one context lock, calls one engine function and + returns a `CUresult` — the semantic work is always one layer down. +* **Internal engine.** Owns the object graph: process globals (init state, + kernel-client handle, device list), device (chip/caps/limits, vtables), + context (memory/channel/stream managers, QMD pools, ISR thread), stream, + module/function, memory objects, events/graphs/libraries. +* **Driver-model layer.** Selected at initialization behind one function + table: the **RM backend** (the one Open-IOV cares about) plus SoC, Amodel, + MPS and WDDM. It turns engine operations into RM controls and object + allocations over `/dev/nvidia*`, and owns the **driver-ABI adaptation** + layer — one instance per supported kernel-driver ABI, chosen at `cuInit`. +* **Hardware layer.** Chip-specific method/QMD/semaphore/doorbell encoders + selected by compute capability. The userspace driver *builds* the GPU + command streams; the kernel driver allocates, maps, schedules and services + interrupts. +* **Synchronization model.** Three primitives explain most of it: the + **tracking semaphore** (per-channel counter of completed pushes), the + **marker** ("all work before this point") and the **QMD semaphore** + (released when a grid completes). Blocking waits use a per-context ISR + thread and a DPC queue. + +## 3. What it touches in a guest + +This is the emulation contract; the full checklist is `docs/03` §9. + +**Device nodes.** `/dev/nvidiactl` (process-wide control: version negotiation, +client creation, GPU attach, OS events, exported-object fds), `/dev/nvidiaN` +(per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation), +`/dev/nvidia-uvm` (managed memory and page migration) and `/dev/nvidia-vgpuN` +(vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths). + +**ioctls.** The control device uses `_IOWR('F', cmd, …)`: `200` +`NV_ESC_CARD_INFO`, `210` `NV_ESC_CHECK_VERSION_STR`, `212` +`NV_ESC_ATTACH_GPUS_TO_FD`. RM escapes include `0x2b` `NV_ESC_RM_ALLOC` +(**the main object-allocation path**), `0x2a` `NV_ESC_RM_CONTROL`, `0x29` +`NV_ESC_RM_FREE`, `0x4e`/`0x4f` `NV_ESC_RM_MAP_MEMORY`/`UNMAP_MEMORY`, +`0x57`/`0x58` `NV_ESC_RM_MAP_MEMORY_DMA`/`UNMAP_MEMORY_DMA` and `0x4a` +`NV_ESC_RM_VID_HEAP_CONTROL`. The request word `rsi=0xc030462b` in a crash +dump decodes to "allocating an RM object with a 48-byte payload". + +**RM objects/classes.** Every RM call names a class. Observed for 570.144 on +an emulated TU104: `NV01_ROOT` (`0x0`), `NV01_DEVICE_0` (`0x80`), +`NV20_SUBDEVICE_0` (`0x2080`), `NV01_EVENT` (`0x40`), +`KEPLER_CHANNEL_GROUP_A` (`0xa06c`), a channel-group/runlist object (`0x9067`), +`TURING_CHANNEL_GPFIFO_A` (`0xc46f`, the channel), `TURING_COMPUTE_A` +(`0xc5c0`, the compute engine object), plus memory (`0x3e`) and VA-space +classes. The observed `cuCtxCreate` order is root → device → subdevice → +events → channel group → channel-group object → memory → channel → control +`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` → compute object → engine registration. + +**Controls.** Device/GPU info (`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`, +`GPU_GET_VIRTUALIZATION_MODE`, `FB_GET_INFO`, `CE_GET_CAPS`, …), FIFO/channels +(`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` `0x80170d`, GPFIFO scheduling, +timeslices, `GET_CLASS_ENGINEID`), GR/context switches, memory, events and +interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates +unknown-but-successful controls; it only fails when a *specific reply field* +is consumed and wrong. + +**Memory.** VA reservation (`NV_ESC_RM_VID_HEAP_CONTROL`/`NVOS32`), backing +allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and +mapping (`NVOS33` into the process, `NVOS46` into a device VA space). All +guest memory CUDA uses is either an RM allocation backed by emulated +VRAM/BAR1 or pinned guest RAM the GPU must DMA to. + +**Channels, GPFIFO, USERD, doorbells.** A channel is a GPFIFO (array of +pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO, +error notifier and (Volta and later) the USERD itself, passing the handles to +the channel allocation; submission writes the GPFIFO put pointer into USERD +and rings a doorbell token returned by the channel allocation. Pushbuffers are +method streams terminated by a tracking-semaphore release and a new GPFIFO +entry; the host emulation decodes them as such. + +**Synchronization below the API.** Semaphores (plain GPU-visible memory +values), 4 KB per-channel error notifiers, OS events (kernel objects used to +wake blocked threads), interrupts routed through the kernel driver, and a +trap handler for debuggers and `device-side assert`. + +**UVM and GSP-RM.** When UVM is available the driver registers GPUs, VA +spaces, channels and managed allocations and pushes UVM semaphores into the +same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing +from userspace — it still issues RM ioctls; inside the kernel driver controls +become GSP-RM RPCs, which Open-IOV's host module answers. + +## 4. Why forwarding matters, and where we are + +The performance budget is dominated by **push-buffer execution**, +**synchronization latency** and (once, at setup) **control-plane latency** +(`docs/04` §4). Only forwarding *real work* to the physical GPU meets the +first two; software execution of state/memory ops is a correctness bridge, not +the data plane (`docs/08`). + +* **Stated architecture:** *emulate the interface, pass through the work, + never patch anyone.* Prototype the host-RM/GSP proxy first for correctness, + measure, then move the hot path to direct submission if the budgets demand + it — work path abstracted from day one. +* **Control plane today:** GSP-RM emulation, RPC handling, BAR/PCI/MSI-X + emulation and object/control replies are implemented; the stock guest stack + gets through `cuInit` and device enumeration into `cuCtxCreate`. +* **Current blocker:** `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) + because the class-filtered engine list is missing the async copy engine + `COPY14` (descriptor `0x793ceb0e`, instance 14, client type `0x38`, class + `0xc9b5`). The oracle list has 7 engines (GR, CE0..CE4, `COPY14`); the guest + list has 6. The driver silently skips engines with no per-(class, instance) + database entry, then reads a zeroed secondary record and rolls back; what + builds that database entry is the open question (`docs/05` §3.6–3.7). +* **Data plane today:** the host module decodes guest push buffers as method + streams and **executes the small work it understands itself** — memsets, + copies and semaphore releases. **No CUDA kernel has been forwarded yet.** + +## 5. The crash-path lesson: replies are consumed data + +The old deterministic `cuCtxCreate` segfault at `libcuda+0x453f90` shows why +reply coherence matters (`docs/05` §3.5, `docs/04` §6). The engine-registration +hook allocates a `TURING_COMPUTE_A` object with `pAllocParms` pointing at the +caller's own **8-byte stack local** and `paramsSize = 0`; the guest driver does +not honour `paramsSize`, so it reads **16-byte** class params starting there +(the upper 8 bytes are the hook's saved `rbp`) and copies 16 bytes of reply +params back over the same buffer. With the old zeroed reply that copy-back +zeroed the saved `rbp`; the function's epilogue then loaded `rbp = 0` and its +next access — reloading a pointer from `[rbp-0x38]` — faulted at +`0xffffffffffffffc8` (the store was the kernel's `copy_to_user`). + +**The fix:** echo the request's params back, forcing only `status` to `NV_OK`; +the copy-back then writes exactly the bytes it read, and the fault disappears — +the same thing the real GSP-RM does for these in/out params. + +**The general rule:** the userspace library is a *consumer of emulated data*. +A wrong reply is worse than a slow one. Work up the reply-fidelity ladder — +status-only → coherent zeroed params → coherent values for consumed fields → +exact values — and validate each rung with the A/B oracle. + +## 6. How to look things up + +The toolchain is version-agnostic (`docs/06`): + +``` +tools/cudatool.py info|exports|functions|nearest|crash|delta <lib> +tools/callwalk.py <callgraph.tsv> callers|callees|chain ... +``` + +* **`cudatool.py`** — build id/sections/exports/ABI note (`info`), symbol map + (`exports`), function ranges from `.eh_frame_hdr` (`functions`), containing + function and nearest export for an address (`nearest`), dmesg `ip` to file + vaddr (`crash`), and an interface-level diff (`delta`). +* **`callwalk.py`** — caller/callee trees and shortest call paths over the + callgraph index. +* **Annotated corpus** — `570.144/annotations/names.tsv` (our name map; exports + exact, internals grow over time) and `subsystems.tsv`; corpus headers carry + the recovered name, file address and direct callers/callees. The generated corpus + is not distributed. + +**Address convention (read this before citing an offset).** Public docs cite +**file virtual addresses** as `libcuda+0x...`. Ghidra loads these binaries at +image base `0x100000`, so `program address = file vaddr + 0x100000`; the +`tools/` wrappers translate automatically, raw Ghidra scripts do not. An +earlier analysis passed file vaddrs straight to Ghidra and decompiled functions +0x100000 too low, so its decompiled-code conclusions were about the wrong +functions (its live traces stayed valid). The same applies when transferring a +name map to a new driver version. + +## 7. Where the detail lives + +| Document | What it gives you | +|---|---| +| [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | the reading order, fast recipes, and the two findings that change how you debug | +| [`../userspace/cuda/docs/01-architecture.md`](../userspace/cuda/docs/01-architecture.md) | the full layer/object/sync/memory model | +| [`../userspace/cuda/docs/02-build-and-execution.md`](../userspace/cuda/docs/02-build-and-execution.md) | build model, embedded device code, init/threading, what it touches | +| [`../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../userspace/cuda/docs/03-driver-and-gpu-interface.md) | the emulation contract: nodes, ioctls, classes, controls, channels, UVM | +| [`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) | NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security | +| [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md) | the binary map, the crash resolution, and the current engine-topology blocker | +| [`../userspace/cuda/docs/06-tooling.md`](../userspace/cuda/docs/06-tooling.md) | every tool, its usage and its cost | +| [`../userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md) | re-running the analysis on a new driver version | +| [`../userspace/cuda/docs/08-virtualization-approaches.md`](../userspace/cuda/docs/08-virtualization-approaches.md) | forwarding options and the honest current stage | +| [`../docs/README.md`](../docs/README.md) | the project documentation front door | diff --git a/wiki/README.md b/wiki/README.md index bab1cdc..4c90b66 100644 --- a/wiki/README.md +++ b/wiki/README.md @@ -11,6 +11,7 @@ ### Core Analysis - [GSP 570.144W Firmware Analysis](GSP-570.144W-Firmware-Analysis.md) - Firmware decompilation and address record - [Address Reference](Address-Reference.md) - Firmware-side address tables +- [CUDA Userspace Driver](CUDA-Userspace-Driver.md) - `libcuda.so` deep dive: layer model, guest interface, forwarding status, crash-path lesson and lookup tooling - [RPC Message Format](RPC-Message-Format.md) - Legacy mailbox wire format (superseded) - [BAR Emulation Specification](BAR-Emulation-Specification.md) - BAR_TRAP RPC analysis (superseded) - [WPR Management](WPR-Management.md) - WPR handler analysis (superseded) |
