open-iov.git/scripts, branch archive/pre-squash

docs: architecture rewrite - emulate vs pass-through model, phased performance roadmap

2026-09-22T13:51:51+00:00

- Clear separation: EMULATE (GSP interface, BARs, config) vs PASS-THROUGH (compute, memory, DMA to physical GPU) - Phased roadmap: Phase 1 (GSP-RM correctness) → Phase 2 (memory model) → Phase 3 (compute forwarding) → Phase 4 (performance) → Phase 5 (graphics) - Host module architecture corrected: apt nvidia (570.211.01) for physical GPU, open-iov-vfio.ko for vGPU emulation, NO 610 submodule - Host stability guarantees documented (BAR3 shadow, submodule skip) - GSP_EMULATION_ARCHITECTURE.md: memory model corrected (BAR1/BAR3 shadows)

host stability: BAR3 shadow + skip submodule nvidia + teardown-safe garbage handler

2026-09-22T13:43:22+00:00

Root causes of host GSP crash (WPR2 stuck, 'unexpected WPR2 already up'): 1. stack_manager load-nvidia tried to load our submodule nvidia.ko (610.57.04-base) alongside/over apt nvidia 570.211.01 -> GSP crash during driver conflict. Fix: skip submodule when apt nvidia is already loaded (not needed for QEMU). 2. BAR3 (32MB instance memory window) was mapped to the physical GPU's real BAR3, so guest BAR3 writes could corrupt the host GPU's instance memory. Fix: vzalloc'd shadow buffer (like BAR1 emulated vidmem), per-page mmap. 3. GSP-RM garbage element handler declared GSP death on teardown queue garbage, preventing driver cleanup frees from completing. Fix: stop polling quietly, let Booter-unload kick handle reset. Verified: full lifecycle (load, mdev create, QEMU guest boot, guest probe, mdev remove, unload, reload) with host nvidia-smi (apt 570.211.01) stable throughout. No GSP crash, no WPR2 stuck, no module unload hang.

kernel: Present physical TU104 to guest + emulate GSP boot (FWSEC/Booter) + VBIOS serving

2026-09-21T18:11:45+00:00

Guest driver probe fixes (unmodified 570.144 open driver on physical path): - kernel/main.c: add show_description to the mdev driver. Userspace mdev management tools parse 'framebuffer=M' from every mdev type's description sysfs file before creating an mdev; the missing file made such tools fail to create the vGPU. Also fix create_mdev.sh for the real type dir name pci-nvidia-vgpu-* (mdev core names dirs dev_driver_string(parent)-; the parent registers before nvidia.ko binds so dev_driver_string == 'pci'). - kernel/bar_emulation.c: seed BAR0 shadow so guest sees a REAL, PHYSICAL, non-virtual TU104 (user directive: no PV/VF vGPU paths, unmodified driver): * PMC_BOOT_0/1/42 = physical TU104 values captured from host hardware (0x164000a1 / 0x0 / 0x164a1000). Without these, halmgrGetHalForGpu() fails and probe aborts with 'does not include the required GSP'. * Serve a valid VBIOS at the PROM window (BAR0+0x300000) loaded from /lib/firmware/tu104_vbios.bin (dump of host TU104 ROM). Guest's kgspExtractVbiosFromRom_TU102 needs it for the mandatory FWSEC ucode (kgspGetFrtsSize is hardwired to 1MB on Turing). * Pre-set GSP secure-boot (GFW) completion state: PGC6 PLM (0x118128)=1, GFW_BOOT progress (0x118234)=0xFF, GSP falcon CPUCTL HALTED (0x110100). The driver's post-FLR GFW wait only *waits* (never kicks), so the pre-set state satisfies it immediately. - kernel/gsp_emulation.c/h: physical GSP boot emulation. * Falcon boot-watch: poll GSP falcon/PGC6/WPR2/scratch registers and log changes (guest BAR0 is mmap-based, otherwise invisible to host). * gsp_emu_poll_boot: STARTCPU auto-complete state machine. Guest kicks FWSEC then Booter Load via GSP falcon CPUCTL; detect each kick and complete the stage (HALTED + FRTS err NONE + WPR2 + mailbox + RISCV-active) so the guest believes its ucodes ran. Diagnosis that led here (documented in Handoff-vGPU-IO.md): - Open nvidia.ko can never be an SR-IOV VF vGPU guest: os_is_grid_supported() is a stub returning FALSE in the open build (disassembled). PV path also rejected per user directive. Only the physical path with a fully emulated local GSP boot remains.

kernel: Fix GSP emulation shadow binding, eventfd wiring, RPC livelocks + docs suite + repo reorg

2026-09-21T17:06:51+00:00

CRITICAL FIXES (guest driver bring-up): - gsp_emulation.c: emulation engine bound to the WRONG BAR0 - it polled the host physical BAR0 (bars[0].vaddr ioremap) for the doorbell and read RPCs from a local dma_alloc_wc buffer, while the guest writes land in the vzalloc'd bar0_shadow. The RPC path could never work; PBA writes even corrupted the host GPU's physical BAR0+0x3000. gsp_emu_init now binds exclusively to bar0_shadow (doorbell/mailbox/PBA aliases; shared_mem_dma==0 marks the alias so cleanup never dma_free's it). - main.c SET_IRQS: DATA_EVENTFD eventfds were DISCARDED ('TODO' + kfree) - the guest never received MSI-X completions. Full wiring now: eventfd_ctx_fdget per vector, fd -1 clears, dev_close releases refs. gsp_emu_inject_completion_interrupt looks up vgpu->irqs[0].trigger LIVE (init-time copy was permanently NULL since QEMU wires after gsp_emu_init). - gsp_rpc_client.c: BAR_TRAP/MIGRATE in emulation mode called gsp_rpc_submit_raw which rings the SAME doorbell the thread polls - self-triggered RPC livelock + multi-second stalls. Now fail fast with -ENOTSUPP + pr_warn_ratelimited (guest BAR access is direct MMIO). MORE FIXES: - main.c: nv_vgpu_config_write returned SIZE_MAX (while(count--) wrap) and was dead code while the VFIO write handler did unfiltered byte copies (guest could rewrite vendor ID/BARs). Return fixed; VFIO write handler now routes through the per-register filter. - kernel/Makefile: M=$(PWD) used the env var which is stale when invoked via the top-level make (pointed at repo root -> kbuild silently built NOTHING; explains every 'stale .ko' incident). Now M=$(CURDIR). - Warnings sweep: header prototypes for mdev/type functions, format casts in gsp_comm.c DRY-RUN logs, unused variable. ROBUSTNESS (harness/tooling): - vgpu_vm.py: run had no timeout (start could block forever); guest_shell recv could block forever on a wedged guest. Both now bounded; guest_shell pre-flights guest-ping and reports wedge diagnostics. - stack_manager.sh: qemu/qemu-debug now launch via the vgpu_vm.py harness (workflow rule: never raw QEMU). REPO REORGANIZATION: - driver/ (legacy guest-side gsp-vgpu.ko) moved to legacy-driver-guest/ (reference only, not built; README explains). 3 legacy un-buildable test files moved to legacy-driver-guest/kernel-tests/. Top-level Makefile cleaned of the dead DRIVER_DIR reference. DOCUMENTATION SUITE (new): - docs/GSP_EMULATION_ARCHITECTURE.md - emulator deep-dive: shadow BAR0 model, doorbell protocol, dispatch table, MSI-X path, lifecycle, memory- model caveats - docs/DEBUGGING.md - QEMU GDB stub workflow, serial console, QMP NMI, failure taxonomy, full bring-up validation checklist, EPT-livelock case study, version-matrix discipline - docs/ONBOARDING.md - repo tour, coding standards (incl. virt_to_phys / dma_addr_t / get-put rules), first build & test - docs/VERSION_UPGRADE.md - end-to-end new-driver-version procedure (firmware fetch, Ghidra analysis, configs, signatures, validation) - docs/PERFORMANCE.md - bare-metal A/B methodology + targets - README.md doc index; STATE.md refreshed; DEVELOPMENT_WORKFLOW.md corrected (load order: open-iov-vfio BEFORE nvidia; stock-driver blacklist note)

kernel: Fix guest-hang root cause (VFIO mmap garbage PFN) + rebuild mdev remove path

2026-09-21T16:37:46+00:00

VFIO mmap (main.c): virt_to_phys() was used on vmalloc-family addresses (vzalloc'd BAR0 shadow, ioremap'd BAR2/3), producing garbage PFNs. Guest BAR0 access then hit a KVM make_spte reserved-bits WARN and an EPT-violation livelock, hard-wedging the guest on every nvidia driver probe (confirmed via QEMU GDB stub). BAR0 now maps per-page via vmalloc_to_pfn(); BAR2/3/4/5 map bars[N].gpa directly (like BAR1); config space uses new nv_vgpu_mmap_virt_to_pfn() helper. mdev remove path (main.c): old path never list_del'd the device (available_instances leaked), never ran cleanup (leaked gsp_emu kthread), never unregistered VFIO (leaked /dev/vfio/N + iommu group), never put pdev, never freed the struct. New kvmgt-style path: cancel init_work -> wait fds -> vfio_unregister_group_dev -> cleanup work -> list_del -> pci_dev_put -> vfio_put_device. gsp_rpc_client.c: fix dma_map_page dma_addr_t->int truncation in gsp_rpc_alloc_emul (corrupts >=4GB IOVAs); export gsp_client_init_emulation. gsp_emulation.c: export gsp_emu_process_rpc and gsp_emu_get_stats. main.c: remove wrong MODULE_SOFTDEP("pre: nvidia") - module has zero nvidia imports, loads standalone, and must load BEFORE nvidia.ko (it exports nvidia_vgpu_vfio_* for nvidia.ko to import). nvidia-vgpu-vfio.h: NULL-safe nv_vgpu_{dbg,info,warn,err} macros. tests/gsp_emulation_test.c: rewritten against current API - drives gsp_emu_process_rpc() with crafted shared-memory messages (protocol validation, SCHEDULE/YIELD/NOTIFY/DEBUG dispatch, stress). Builds with KBUILD_EXTRA_SYMBOLS; depends: open-iov-vfio. stack_manager.sh: vfio_mdev doesn't exist on >=6.4 (probe per-module); load-nvidia detects foreign nvidia via /sys/module/nvidia/version and errors with guidance instead of silently skipping.

Make repo fully path-portable and add reproducible installer

2026-09-21T15:09:36+00:00

- install.sh: one-command fresh-Ubuntu reproduction (deps, NVIDIA clone + patch, GSP firmware fetch from linux-firmware.git with objcopy .fwimage extraction + MD5 verify, ordered build, host config, load, mdev creation). Wrapper-neutral: works with libvirt/raw QEMU. - patches/nvidia: regenerated path-independent patch (KBUILD_EXTRA_SYMBOLS resolved via $(src)/$(CURDIR) relative to the NVIDIA tree). - scripts/stack_manager.sh and helper scripts auto-detect repo root (OPEN_IOV_ROOT override); no hardcoded developer paths anywhere. - firmware analysis scripts use script-relative paths (also fixes stale firmware/570-GSP references); docs/wiki updated to $OPEN_IOV_ROOT. - .github/workflows/ci.yml updated for standalone repo layout; firmware fetched from linux-firmware in CI. - Untrack remaining kbuild artifacts (Module.symvers, modules.order, *.o.d); gitignore them. - STATE.md: live project state, limitations, fixed/open bugs. - README.md: rewritten for clone-and-run reproducibility.

Fix create_mdev.sh type names and update launch_qemu_vgpu.sh with install/run modes

2026-09-21T08:42:08+00:00

VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl handlers

2026-09-14T10:43:11+00:00

- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create() - Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions - Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP - Added config space read/write handlers at offset 0x70000000000 - Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers) - Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL) - Added udev rules for vfio group permissions - Created stack_manager.sh unified tool - Fixed mdev remove crash with deferred cleanup in vfio_device_release callback - Kernel 6.8+ IOMMUFD compatibility - QEMU launches successfully with vGPU passthrough

Phase 5: Fix MDEV creation blocker - mdev parent registration & ioctl-based vGPU type registration

2026-09-14T02:17:09+00:00

- Implemented nv_vgpu_mdev_register_parent() using mdev_register_parent() API with nv_vgpu_mdev_types array for proper mdev parent registration - Added ioctl NV_VGPU_IOCTL_REGISTER_VGPU_TYPE for userspace vGPU type registration - Created separate header nv-vgpu-vfio-interface-nvidia.h for nvidia.ko with only extern declarations to avoid duplicate symbol exports - Used KBUILD_EXTRA_SYMBOLS to provide only needed symbols (nvidia_vgpu_vfio_*) - Updated create_mdev.sh to use correct mdev type names (pci-nvidia-vgpu-*) - Updated load_vgpu_stack.sh to load open-iov-vfio.ko before nvidia.ko Result: 4 mdev types now show available_instances=8, mdev creation successful

Phase 4: Portable Architecture Implementation

2026-09-13T23:28:03+00:00

- Firmware Analyzer: Runtime firmware signature computation (CRC32, SHA256) and pattern-based RPC handler address discovery - Dynamic RPC Dispatcher: Version-aware dispatch with native firmware handlers (550/560/570/580) and emulation fallback - Hardware Architecture Layer: Turing, Ampere, Blackwell configs with BAR layouts, doorbell offsets, MSI-X vectors - Driver Version Adaptation Layer: Feature detection per driver version - Declarative YAML configs for hardware (turing/ampere/blackwell) and versions (550/560/570/580) - Kernel unit tests for GSP emulation (gsp_emulation_test.c) - Test infrastructure scripts (test_gsp_emu.sh, test_guest_driver.sh, test_cuda_workloads.sh, test_graphics_workloads.sh, test_windows_guest.sh, test_vgpu_stack.sh) - Updated kernel Makefile with new source files