Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=archive%2Fpre-squash 2026-09-23T05:26:25+00:00 2026-09-23T05:26:25+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T05:26:25+00:00 urn:sha1:5367ffa2108fd9204d98f1cb23bb718a9143a25d kceGetDeviceCaps (kernel_ce.c) copies the caps the client hands to UVM from NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS (0x20802a07), which reaches GSP-RM and was answered with zeros. With capsTbl = {0,0} no CE had the SYSMEM bit, so UVM's channel manager found no usable CE and UVM_REGISTER_GPU failed with NV_ERR_NOT_SUPPORTED ("Failed to initialize the channel manager", uvm_gpu.c:1574 - visible with uvm_debug_prints=1). New handlers: - CE_GET_PHYSICAL_CAPS (0x20802a07): {ceEngineType, capsTbl[2]} with SYSMEM_READ|SYSMEM_WRITE|SYSMEM|P2P|BL_GT_64K (0xEC) for COPY0..COPY4, NOT_SUPPORTED otherwise (matches subdeviceCtrlCmdCeGetCaps_IMPL). - CE_GET_ALL_CAPS (0x20802a0a) / CE_GET_ALL_PHYSICAL_CAPS (0x20802a0b): capsTbl[64][2] + present bitmap with the five CEs. - CE_GET_CE_PCE_MASK (0x20802a02): pceMask = BIT(ceIdx) (one PCE per LCE). Verified in-guest progress: UVM now gets past the channel manager and creates channels; the register path's new blocker is the first channel push (channel_init, uvm_channel.c:2535) whose semaphore release the emulated GPU must complete ("Tracker entry ... stuck waiting for 90s", sema VA 0x121000000 on channel 1:3 CE 0). 2026-09-23T05:10:40+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T05:10:40+00:00 urn:sha1:841da7540f4397b0cac5f47537d2a39aca9ec3cf 2026-09-23T05:04:38+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T05:04:38+00:00 urn:sha1:b800c713c393633d8c6c603ffa7ad32bccc0c7be 2026-09-23T04:51:55+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T04:51:55+00:00 urn:sha1:7897fe79ba6a00cc59b7a357903ecfc70480ce97 The ptrace tracer resolves the current CUDA blocker: libcuda issues raw syscall instructions, so LD_PRELOAD shims and ltrace see nothing; following the child with PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO does. It found UVM_REGISTER_GPU (ioctl 0x25) returning rmStatus=0x56 NV_ERR_NOT_SUPPORTED, with inputs identical to the working host run (rmCtrlFd=-1, hClient=0, hSmcPartRef=0). Also dumps NV_ESC_RM_CONTROL cmd/status and RM alloc/free statuses (all zero) and the UVM ioctl set (guest stops after 0x27,0x1,0x25,0x2 vs 141 ioctls on the host). launch_qemu_rootport.sh is the working QEMU command line with the pcie-root-port fix (see its header + the handoff PART 4); vmcycle.sh drives the module/mdev/VM reload cycle around it. 2026-09-23T04:41:46+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T04:41:46+00:00 urn:sha1:7aef82a5112ee72ee2fc5a430db072085933e927 Three fixes that clear all driver-visible errors in UVM's device create (cuInit still fails later in UVM_REGISTER_GPU): 1. GSP_RM_CONTROL 0x20800a59 INTERNAL_GMMU_GET_STATIC_INFO (new handler): kgmmu.pStaticInfo->replayableFaultBufferSize feeds kgmmuFaultBufferReplayableAllocate (called by faultbufConstruct when UVM opens the device). The generic zero reply made it a 0-page allocation: "faultbufConstruct_IMPL: Failed to setup Replayable Fault buffer (status=0x51)". Report 64KB replayable / 4KB legacy, mirroring kgmmuSetAndGetDefaultFaultBufferSize_TU102 defaults. 2. VF-window NV_VIRTUAL_FUNCTION_PRIV_MMU_FAULT_BUFFER_SIZE(0/1) (BAR0+0xB83010/0xB83030, plus plain aliases) pre-filled with _VAL in NVC369_BUF_SIZE units (0x800=64KB, 0x80=4KB) so the local (non-RPC) path reads the same values the control handler reports. 3. VF-window NV_VIRTUAL_FUNCTION_PRIV_ACCESS_COUNTER_NOTIFY_BUFFER_SIZE (BAR0+0xB83110, plus plain alias) = 0x1000 (the value the host's physical TU104 reports; x32B = 128KB). uvmGetAccessCounterBufferSize_ TU102 multiplied our zero and uvmInitializeAccessCntrBuffer failed with NV_ERR_NO_MEMORY at mem_desc.c:1353. Also note (companion finding, no code change): the guest must be placed behind a pcie-root-port; on the Q35 root bus QEMU rewrites the PCIe cap to Root Complex Integrated Endpoint and zeroes LNKCAP/LNKCTL/DeviceCaps (vfio_setup_pcie_cap: "On a Root Complex bus Endpoints become Root Complex Integrated Endpoints, which changes the type and clears the LNK & LNK2 fields"), which is what made getPCIELinkRateMBps read speed 0. 2026-09-23T04:09:13+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T04:09:13+00:00 urn:sha1:d28e5bb94752ba523b826945409e5ad3206dad5a The poison test (fill every dword of BAR0 0x88000-0x88FFF, 0xB80000- 0xB80EFF, 0xC08000-0xC08EFF with 0x103) did not change the CUDA "Unknown PCIe speed" failure, proving the failing BUS_GET_INFO_V2 index-3 data does not come from any BAR0 mirror read. Combined with the QEMU config-space behavior (link registers zeroed, type=0x92, 0x84 never read from us) the value must come from the VGPU_STATIC_INFO branch in getBusInfos (pVSI-> pcieGpuLinkCaps), i.e. the guest needs the host vGPU static-info payload. Cleanup: remove the poison loops and the per-write CFGWR log; keep the once-per-init PCIe cap print and add per-entry BUS_GET_INFO_V2 logging (index+data) for the ongoing CUDA/UVM work. 2026-09-23T04:04:14+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T04:04:14+00:00 urn:sha1:7d6c789a469de154ee6d26a7fb9161ef7279a34a Move the emulated PCIe capability to config offset 0x78 and shift MSI to 0x48 / MSI-X to 0x68, matching the real TU104 (host lspci: PM 0x60, MSI 0x68, Express 0x78). This matters because the driver's NV_PCFG mirror (BAR0+0x88000, size 0x1000) is read as PCI config space: osDevReadReg032 -> vgpuDevReadReg032 intercepts [NV_PCFG, +0x30) (or the full 0x1000 for SRIOV) and forwards to osPciReadDword, so NV_XVE_LINK_CAPABILITIES (0x84) must be the PCIe Link Capabilities register, i.e. cap+0x0C with the cap at 0x78. Link Caps = Gen3 x16 (0x103), Link Status = 0x0103. Also set MCFG-visible caps: the guest now reports MSI [48], MSI-X [68], Express [78] exactly like the host GPU. Fixing the capability chain: the first attempt set the PCIe cap's NEXT to 0x0C, which broke enumeration (device 00:06.0 disappeared from the guest) - NEXT must terminate at 0. Unknown still open: QEMU presents type=0x92 (Root Complex Integrated Endpoint) and zeroes 0x84-0x8B in its config cache (never read from us, not refreshed by writes), and poisoning all BAR0 mirror windows with 0x103 does not change the driver's "Unknown PCIe speed" - so the value comes from the VGPU_STATIC_INFO branch (pVSI->pcieGpuLinkCaps, IS_VIRTUAL true via hypervisor detection) rather than any register read. Debug helpers added (temporary): CFGWR config-write log, PCFG change watch, PCIe-cap print at init. 2026-09-23T03:41:13+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T03:41:13+00:00 urn:sha1:1ed5d98cbb2273e75674d050fb48e3c3422a3877 Logs changes to NV_PMC_BOOT_1 (0x4), the low PCFG alias, the BAR0 PCFG block (0x880000/0x880080/0x880084/0x880088/0x88008c) and the VF-window aliases. Observed during a CUDA run: the guest never writes any of them, so the link-cap values we pre-set (0x103 / 0x01030000) are not clobbered. Reads are not observable through the mmap'd BAR0. 2026-09-23T03:32:44+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T03:32:44+00:00 urn:sha1:14283fe73fb5e432a7af45de94961655714910f6 CUDA (cuInit) now reaches UVM's device create: libcuda opens nvidia-uvm, UVM calls nvGpuOpsDeviceCreate/gpuGetGpuInfo, and the driver fails at getPCIELinkRateMBps -> calculatePCIELinkRateMBps ("Unknown PCIe speed", NV_ERR_INVALID_STATE) -> nv_open_device aborts -> cuInit returns CUDA_ERROR_NOT_INITIALIZED (3). Implemented along the way: - FB_GET_INFO_V2 (0x20801303): echo the requested FB info indices and answer TU104 values (256MB RAM/heap/BAR1, GDDR6, 8 LTCs, 4MB L2, ...). Fixes the kern_mem_sys_ctrl.c:911 asserts and gives CUDA/UVM real memory sizes. - BUS_GET_INFO_V2 (0x20801823): echo indices; PCIe Gen3 x16 link caps / GEN info / link status, SYSMEM_CONNECTION_TYPE=PCIE. UVM's gpuGetSysmemLinkInfo reads SYSMEM_CONNECTION_TYPE and the nominal link bandwidth from here. - PCIe Link Capabilities in the emulated config space (0x8C/0x92) and in the BAR0 PCFG mirror (gpuReadBusConfigReg -> NV_PCFG+0x84 = BAR0+ 0x88084): MAX_SPEED=3 (8GT/s Gen3), MAX_WIDTH=0x10 (x16); link status mirror at 0x88088. Also mirrored under the VF-window/low aliases. - Temporary change-watch on PCFG[0x88084] to see if the guest overwrites it (it does not - stays 0x103). Verified: guest nvidia-smi -L still works and reports PCIe Generation Max/Current = 3. cuInit still fails; the remaining question is which address the driver's kbifGetGpuLinkCapabilities read sees as zero (the PCFG shadow holds 0x103). Guest-side kprobe tracing of that function panicked the VM (free_swap_cache page fault on a corrupted pointer) - possible guest memory corruption to investigate separately. 2026-09-23T02:43:43+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T02:43:43+00:00 urn:sha1:b34e4531d8f0b984dc15660d3ab406ae48822931 Documents the current CUDA blocker: cuInit returns CUDA_ERROR_NOT_INITIALIZED (3) on the emulated GPU even though every nvidia.ko ioctl succeeds and RmInitAdapter completes. libcuda opens /dev/nvidia-uvm before /dev/nvidia0, so the next milestone is making nvidia-uvm initialize against the emulated MMU/VAS model.