Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=archive%2Fpre-squash 2026-09-23T05:26:25+00:00 2026-09-23T05:26:25+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T05:26:25+00:00 urn:sha1:5367ffa2108fd9204d98f1cb23bb718a9143a25d kceGetDeviceCaps (kernel_ce.c) copies the caps the client hands to UVM from NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS (0x20802a07), which reaches GSP-RM and was answered with zeros. With capsTbl = {0,0} no CE had the SYSMEM bit, so UVM's channel manager found no usable CE and UVM_REGISTER_GPU failed with NV_ERR_NOT_SUPPORTED ("Failed to initialize the channel manager", uvm_gpu.c:1574 - visible with uvm_debug_prints=1). New handlers: - CE_GET_PHYSICAL_CAPS (0x20802a07): {ceEngineType, capsTbl[2]} with SYSMEM_READ|SYSMEM_WRITE|SYSMEM|P2P|BL_GT_64K (0xEC) for COPY0..COPY4, NOT_SUPPORTED otherwise (matches subdeviceCtrlCmdCeGetCaps_IMPL). - CE_GET_ALL_CAPS (0x20802a0a) / CE_GET_ALL_PHYSICAL_CAPS (0x20802a0b): capsTbl[64][2] + present bitmap with the five CEs. - CE_GET_CE_PCE_MASK (0x20802a02): pceMask = BIT(ceIdx) (one PCE per LCE). Verified in-guest progress: UVM now gets past the channel manager and creates channels; the register path's new blocker is the first channel push (channel_init, uvm_channel.c:2535) whose semaphore release the emulated GPU must complete ("Tracker entry ... stuck waiting for 90s", sema VA 0x121000000 on channel 1:3 CE 0). 2026-09-23T04:41:46+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T04:41:46+00:00 urn:sha1:7aef82a5112ee72ee2fc5a430db072085933e927 Three fixes that clear all driver-visible errors in UVM's device create (cuInit still fails later in UVM_REGISTER_GPU): 1. GSP_RM_CONTROL 0x20800a59 INTERNAL_GMMU_GET_STATIC_INFO (new handler): kgmmu.pStaticInfo->replayableFaultBufferSize feeds kgmmuFaultBufferReplayableAllocate (called by faultbufConstruct when UVM opens the device). The generic zero reply made it a 0-page allocation: "faultbufConstruct_IMPL: Failed to setup Replayable Fault buffer (status=0x51)". Report 64KB replayable / 4KB legacy, mirroring kgmmuSetAndGetDefaultFaultBufferSize_TU102 defaults. 2. VF-window NV_VIRTUAL_FUNCTION_PRIV_MMU_FAULT_BUFFER_SIZE(0/1) (BAR0+0xB83010/0xB83030, plus plain aliases) pre-filled with _VAL in NVC369_BUF_SIZE units (0x800=64KB, 0x80=4KB) so the local (non-RPC) path reads the same values the control handler reports. 3. VF-window NV_VIRTUAL_FUNCTION_PRIV_ACCESS_COUNTER_NOTIFY_BUFFER_SIZE (BAR0+0xB83110, plus plain alias) = 0x1000 (the value the host's physical TU104 reports; x32B = 128KB). uvmGetAccessCounterBufferSize_ TU102 multiplied our zero and uvmInitializeAccessCntrBuffer failed with NV_ERR_NO_MEMORY at mem_desc.c:1353. Also note (companion finding, no code change): the guest must be placed behind a pcie-root-port; on the Q35 root bus QEMU rewrites the PCIe cap to Root Complex Integrated Endpoint and zeroes LNKCAP/LNKCTL/DeviceCaps (vfio_setup_pcie_cap: "On a Root Complex bus Endpoints become Root Complex Integrated Endpoints, which changes the type and clears the LNK & LNK2 fields"), which is what made getPCIELinkRateMBps read speed 0. 2026-09-23T03:32:44+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T03:32:44+00:00 urn:sha1:14283fe73fb5e432a7af45de94961655714910f6 CUDA (cuInit) now reaches UVM's device create: libcuda opens nvidia-uvm, UVM calls nvGpuOpsDeviceCreate/gpuGetGpuInfo, and the driver fails at getPCIELinkRateMBps -> calculatePCIELinkRateMBps ("Unknown PCIe speed", NV_ERR_INVALID_STATE) -> nv_open_device aborts -> cuInit returns CUDA_ERROR_NOT_INITIALIZED (3). Implemented along the way: - FB_GET_INFO_V2 (0x20801303): echo the requested FB info indices and answer TU104 values (256MB RAM/heap/BAR1, GDDR6, 8 LTCs, 4MB L2, ...). Fixes the kern_mem_sys_ctrl.c:911 asserts and gives CUDA/UVM real memory sizes. - BUS_GET_INFO_V2 (0x20801823): echo indices; PCIe Gen3 x16 link caps / GEN info / link status, SYSMEM_CONNECTION_TYPE=PCIE. UVM's gpuGetSysmemLinkInfo reads SYSMEM_CONNECTION_TYPE and the nominal link bandwidth from here. - PCIe Link Capabilities in the emulated config space (0x8C/0x92) and in the BAR0 PCFG mirror (gpuReadBusConfigReg -> NV_PCFG+0x84 = BAR0+ 0x88084): MAX_SPEED=3 (8GT/s Gen3), MAX_WIDTH=0x10 (x16); link status mirror at 0x88088. Also mirrored under the VF-window/low aliases. - Temporary change-watch on PCFG[0x88084] to see if the guest overwrites it (it does not - stays 0x103). Verified: guest nvidia-smi -L still works and reports PCIe Generation Max/Current = 3. cuInit still fails; the remaining question is which address the driver's kbifGetGpuLinkCapabilities read sees as zero (the PCFG shadow holds 0x103). Guest-side kprobe tracing of that function panicked the VM (free_swap_cache page fault on a corrupted pointer) - possible guest memory corruption to investigate separately. 2026-09-23T02:40:26+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-23T02:40:26+00:00 urn:sha1:f6d450a8be075fdc19a874826e7649ae0149b099 Root cause of the golden-image NV_ERR_NO_MEMORY: STATIC_KGR_GET_CONTEXT_ BUFFERS_INFO (0x20800a32) fell through to the generic zeroed reply, so kgraphicsInitializeDeferredStaticData() stored a 0-byte main ctx buffer size and kgrctxAllocMainCtxBuffer() did a 0-page allocation (mem_desc.c:1353). Serve the TU104 per-engine ctx buffer table (26 engine IDs x 8 GR engines; NV_U32_MAX = unsupported, which the client skips). Second blocker after that: L2 cache-op and TLB-invalidate completion registers. Turing reaches them through the virtual-register window (GPU_VREG_* = BAR0 + sriovState.virtualRegPhysOffset + offset, 0xB80000 for a non-SRIOV virtual GPU): NV_VIRTUAL_FUNCTION_PRIV_MMU_INVALIDATE (0xB830B0, TRIGGER bit31) and the L2 invalidate/dirty-flush set (0x70000-0x70014 and 0xB80F00/0xB80F04). The driver soft-locked spinning in osDevReadReg032 on 0xB830B4 (kgmmuCheckPendingInvalidates_TU102) and timed out in kmemsysDoCacheOp_GM107; auto-clear all of them (both the plain and windowed aliases). Third: re-probe failed with GspStatusQueueInit msgqRxLink timeout because gsp_rm_emu_reset() was not called before a fresh boot, leaving rm->parsed set so the new libos args were never parsed. Reset GSP-RM state on the fresh FWSEC kick. Verified: guest nvidia-smi and nvidia-smi -L both exit 0 (full unload/reload cycle), GPU enumerated as "NVIDIA GeForce RTX 2080 SUPER" with 256MB, no NO_MEMORY / timeouts / soft lockups in the guest. Also add tools/rmq/grctx_oracle.c documenting that the host RM does not expose the ctx-buffer table to userspace (the real values are probed from the ctxsw ucode by GSP-RM), so the emulation serves synthesized sizes. 2026-09-22T06:16:18+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-22T06:16:18+00:00 urn:sha1:5eac589d9a7af8f8b321b00398575070b9a88513 - STATIC_KGR_GET_FLOORSWEEPING_MASKS(0x20800a26): TU104 gpcMask=0x3f (6 GPCs), 4 TPC/GPC. numGpcs==0 killed krcWatchdogInit -> RmInitAdapter bailed ('numGpcs > 0' assert at kernel_graphics_object.c:191). - CPU interrupt tree (osVerifyInterrupts): driver accesses VF window at virtualRegPhysOffset=NV_VIRTUAL_FUNCTION_FULL_PHYS_OFFSET(0xB80000)+off, NOT 0x1640 (was watching wrong offset). On LEAF_TRIGGER write, set LEAF/TOP pending bits + inject MSI-X so the sanity-test ISR runs - osVerifySystemEnvironment now passes. - WPR2 race: clear boot state BEFORE CPUCTL=HALTED on Booter-unload kick so WPR2 is down when driver checks kgspIsWpr2Up ('WPR2 is still up'). - Dedicated GSP_RM_FREE(10) handler: echo NVOS00_PARAMETERS + status=NV_OK (proper free_v03_00 form). fn-10 timeout persists despite this - characterized as driver-side reply-matching issue, not content. 2026-09-22T03:18:53+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-22T03:18:53+00:00 urn:sha1:38425e0891760d86d8a34cc84e8b8f8b39479a8d Driver caches the GR infoList as pGrInfo and reads MAX_SUBCONTEXT_COUNT (0x2c) - zero made channel-group numMax==0, asserting channel allocation (numMax == numFree && numMax != 0 -> NV_ERR_INVALID_STATE, blocking the per-device open -> NVML 'No devices'). Fill litter topology (6 GPC x 4 TPC x 2 SM = 48 SMs) and subcontext counts = 64 (NV_PGRAPH_PRI_FE_CHIP_DEF_INFO_MAX_VEID_COUNT_INIT). 2026-09-22T00:08:39+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-22T00:08:39+00:00 urn:sha1:1e73a2e00c76e25e4ce2980e5d04a533b0c6158d - gsp_emu_poll_boot generic SEC2 kick path: clear SEC2_MAILBOX0=0 (success) - Booter Unload during GPU shutdown read a stale 0xff, failing the unload and killing the subsequent re-init ('Booter failed with non-zero error code: 0xff' -> RmInitAdapter 0x40). A SEC2 kick after a completed boot is a GSP shutdown/reload: clear boot state. Driver unload/reload cycle (last close -> RmShutdown, next open -> RmInitAdapter) now works. - STATIC_KGR_GET_GLOBAL_SM_ORDER(0x20800a22): TU104 topology 6 GPC x 4 TPC x 2 SM = 48 SMs/24 TPCs on GR0. kgrStaticInit is init-critical (NV_CHECK_OK_OR_GOTO); zeros left numSm=0 and NVML spun on empty topology. 2026-09-21T23:53:12+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-21T23:53:12+00:00 urn:sha1:0630028527ae401e15a99cf1145829e9725a13ad CRITICAL protocol fix (msgq.c): writePtr/readPtr are slot indices that WRAP at msgCount ('writePtr += n; if (>= msgCount) -= msgCount'), NOT monotonic counters. '62 -> 0' is a legitimate wrap, not queue death. - gsp_rm_poll_commands: consume while (cmd_read != writePtr), advance cmd_read = (cmd_read + elem_count) % msgcount. Death detection now only: writePtr >= msgCount (garbage header) / bad element signature / insane elemCount. Removes false deaths on healthy wraps that killed the emulator mid-session (fn 10 FREE timeouts). - gsp_rm_send: published status writePtr now wraps too (driver's free-space/slot math breaks on unbounded values >= 2*msgCount). - gsp_rm_emu_poll: back off on_booter_done retries to 250ms (was every 100us poll -> thousands of 'will retry' lines/sec, dmesg unusable); retry log now pr_warn_ratelimited. - CE_GET_FAULT_METHOD_BUFFER_SIZE(0x20802a08) = 12288 (GV100 formula inherited by Turing: 135*(13*5+2) page-aligned, 5 PCEs). Zero reply failed fault method buffer allocation (NV_ERR_NO_MEMORY) killing channel-group creation. 2026-09-21T23:42:17+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-21T23:42:17+00:00 urn:sha1:9b44eb71ca2392606b74a15f7f330482be3e826a Driver's intrCacheIntrFields requires subtree(ACCESS_CNTR=72)=1 within subtreeMap[UVM_OWNED] and UVM_SHARED to be a single subtree. Zeroed map failed RmInitNvDevice state load ('unexpected subtree'). Map matches intrInitSubtreeMap_TU102 (what physical-path guests get from real GSP-RM): ESCHED{3,3} ESCHED_NOTIF{0,0} UVM_OWNED{1,1} UVM_SHARED{2,2}, DEFAULT/RUNLIST/RUNLIST_NOTIF=INVALID(0xFF). Struct retyped to 7x2B; out_params_size spans full struct (driver reads subtreeMap at offset 2052). Verified: probe passes state load, /dev/nvidia0 persists. 2026-09-21T23:28:01+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-09-21T23:28:01+00:00 urn:sha1:8ddb13c9d96be05b859056788740330b5653e72b - FIFO_GET_NUM_CHANNELS(0x20800a61)=64/runlist: zero tripped NV_ASSERT(numChannels>0), kdisp clientChannelTable alloc failed. - BUS_GET_PCI_BAR_INFO(0x20801803): report BAR0=16MB/BAR1=256MB/ BAR2=1MB instmem window@BAR0+0x700000/BAR3=32MB (was all zeros). - DISPLAY_GET_STATIC_INFO(0x20800a01): heads=1, numDispChannels=1 (36B wire layout; zero -> clientChannelTable[0] alloc failed). - DISPLAY_GET_IP_VERSION(0x20800a4b)=0x04000000 (Turing DISP v04.00; driver passes uninitialized input - must return a real version). - I2C_TRANSACTION(0x402c0105): echo request params + NOT_SUPPORTED. Variable-length FINN-serialized params: zeroed reply fails DeserializeUp and the calling ioctl never completes (nvidia-smi hang). - Default unknown-cmd policy back to zeroed params (echo is dangerous for queries with uninitialized input structs). Milestone: guest RmInitAdapter completed, /dev/nvidia0 created.