Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=cs-share 2026-09-30T18:40:01+00:00 2026-09-30T18:40:01+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-30T18:40:01+00:00 urn:sha1:50e54ed1ee3cd7827d01458f8b3b2b22eff833b8 The static CUDA runtime's device init failed with cudaErrorInitializationError even though every kernel ioctl returned 0: cuInit succeeded and then cuDeviceGetAttribute(ECC_ENABLED=36) returned CUDA_ERROR_NOT_FOUND (500). Tracing libcuda's clock-domain helper (libcuda+0xe8c410) shows it issues the CLK legacy controls 0x20809009/0x20809064/0x20809001 and bails with 500 when the replies do not carry the expected domain bits/entries; the dependent attribute getters (ECC_ENABLED, TOTAL_CONSTANT_MEMORY) then abort init. Reply to the three queries with the minimal fields the helper checks: bits 32|34 present in 0x20809009, non-zero domain entries at offsets 12/20/28/36 of 0x20809064, and a matching selector in 0x20809001. No frequencies are fabricated (all other fields stay zero). Verified in the guest with the stock 570.144 static-cudart vectorAdd client: cudaGetDeviceCount -> rc=0 (ndev=1), cudaSetDevice, cudaFree, three cudaMallocs and both HtoD copies all return rc=0; the client then parks in cudaDeviceSynchronize waiting for the (not yet forwarded) kernel launch. 2026-09-30T08:04:22+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-30T08:04:22+00:00 urn:sha1:e4c3f8554d70dcf22567621528a4601d7444666b 2026-09-26T07:21:37+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-26T07:21:37+00:00 urn:sha1:4b214ba45276a8d60161d74ffcb303a28d5f4dab 2026-09-25T06:53:50+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-25T06:53:50+00:00 urn:sha1:b968c287865b8860e7938f56db3fa358c5bef66e - Added TU104_CTRL_INTERNAL_MEMSYS_L2_INVALIDATE_EVICT (0x20800a6c) - Added TU104_CTRL_INTERNAL_GR_GET_FECS_TRACE_HW_ENABLE (0x20800a38) - Added TU104_CTRL_SYSTEM_MAP_SHARED_DATA (0x730151) - Fixed async RPCs (72 GSP_SET_SYSTEM_INFO, 73 SET_REGISTRY, 71 CONTINUATION_RECORD) to send SUCCESS replies instead of silent consume - Added explicit handlers for RPC 47 (UNLOADING_GUEST_DRIVER) and 70 (UPDATE_BAR_PDE) with proper reply semantics Guest driver was failing at GSP-RM level with unknown RPCs 47, 70, 72, 73. Channels registered then immediately unregistered. Now they should stay registered and complete initialization. 2026-09-23T18:03:27+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T18:03:27+00:00 urn:sha1:2a59dcb768541b759ada9cd6f24986e2c234bd37 The guest's CUDA stack was stuck inside context creation because the work it submits is never executed by the emulated GPU. Two forms of pushed work were missing, and the completion path had grown far too slow to be useful: - Channel tracking: on a channel allocation parse the GPFIFO GPU VA and entry count, the USERD address/aperture and the instance block, hand them to the progress service and drop them again when the channel is freed. - Host-class (C46F) semaphore releases: pushes on a compute channel signal progress with a five-dword method run (SEM_ADDR_LO/HI, PAYLOAD_LO/HI, EXECUTE) rather than the copy-engine group the service already understood, so those releases were never applied. Decode both forms. - Emulated vidmem: channel blocks (USERD/instance) and some client semaphores live in the emulated frame buffer, whose VAs are a window over the BAR1 backing rather than guest RAM; access them through the right aperture. - Latency: the service's full-RAM sweep took seconds per pass, so guests' progress waits took whole seconds and the driver tore its own setup back down in the meantime. Sweep guest RAM in bounded, chunked windows on every pass, remember the pages pushes are actually found in and revisit those first, and keep the VA->PA map complete (dedup, sort, binary search) so semaphores resolve reliably. Also add a vidmem poke parameter for bringing up and debugging emulated frame-buffer state. 2026-09-23T15:46:33+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T15:46:33+00:00 urn:sha1:534131933ef24d602314281573db4ef16cc08a93 - GSP_RM_ALLOC: echo the request payload back with status forced to NV_OK. The kernel driver copies the reply params over the caller's params buffer using the class-expected size, so a zeroed reply smashed libcuda's stack (SIGSEGV at the engine-registration continuation); echoing makes the copy-back idempotent. - CE_GET_PHYSICAL_CAPS / CE_GET_ALL(_PHYSICAL)_CAPS: report sysmem, P2P and BL copy caps and mark the GR-synchronous CE (GRCE, COPY0). Without a GRCE the client's context walk ends with an empty secondary-engine record and the engine RM alloc is issued with class 0 (rejected -> cuCtxCreate fails). - TU104 engine table: CE0 shares the GR PBDMA ids (the driver's GR/GRCE runqueue validation requires it), plus an entry for the async copy engine (COPY14) advertised in the GSP static info engine caps. - tools/guest-diag/ioctl_trace: decode the RM status fields of alloc, control and free so the first failing driver call is visible in traces. gitignore: keep the CUDA analysis scratch tree (incl. WORKING-HERE.md) local. 2026-09-23T09:46:43+00:00 Open-IOV dev@open-iov.org 2026-09-23T09:44:08+00:00 urn:sha1:68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.