open-iov.git/kernel, branch archive/pre-squash

kernel: serve CE caps + CE->PCE masks (unblocks UVM channel manager)

2026-09-23T05:26:25+00:00

kceGetDeviceCaps (kernel_ce.c) copies the caps the client hands to UVM from NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS (0x20802a07), which reaches GSP-RM and was answered with zeros. With capsTbl = {0,0} no CE had the SYSMEM bit, so UVM's channel manager found no usable CE and UVM_REGISTER_GPU failed with NV_ERR_NOT_SUPPORTED ("Failed to initialize the channel manager", uvm_gpu.c:1574 - visible with uvm_debug_prints=1). New handlers: - CE_GET_PHYSICAL_CAPS (0x20802a07): {ceEngineType, capsTbl[2]} with SYSMEM_READ|SYSMEM_WRITE|SYSMEM|P2P|BL_GT_64K (0xEC) for COPY0..COPY4, NOT_SUPPORTED otherwise (matches subdeviceCtrlCmdCeGetCaps_IMPL). - CE_GET_ALL_CAPS (0x20802a0a) / CE_GET_ALL_PHYSICAL_CAPS (0x20802a0b): capsTbl[64][2] + present bitmap with the five CEs. - CE_GET_CE_PCE_MASK (0x20802a02): pceMask = BIT(ceIdx) (one PCE per LCE). Verified in-guest progress: UVM now gets past the channel manager and creates channels; the register path's new blocker is the first channel push (channel_init, uvm_channel.c:2535) whose semaphore release the emulated GPU must complete ("Tracker entry ... stuck waiting for 90s", sema VA 0x121000000 on channel 1:3 CE 0).

kernel: serve GMMU static info + per-VF fault/access-counter sizes for UVM

2026-09-23T04:41:46+00:00

Three fixes that clear all driver-visible errors in UVM's device create (cuInit still fails later in UVM_REGISTER_GPU): 1. GSP_RM_CONTROL 0x20800a59 INTERNAL_GMMU_GET_STATIC_INFO (new handler): kgmmu.pStaticInfo->replayableFaultBufferSize feeds kgmmuFaultBufferReplayableAllocate (called by faultbufConstruct when UVM opens the device). The generic zero reply made it a 0-page allocation: "faultbufConstruct_IMPL: Failed to setup Replayable Fault buffer (status=0x51)". Report 64KB replayable / 4KB legacy, mirroring kgmmuSetAndGetDefaultFaultBufferSize_TU102 defaults. 2. VF-window NV_VIRTUAL_FUNCTION_PRIV_MMU_FAULT_BUFFER_SIZE(0/1) (BAR0+0xB83010/0xB83030, plus plain aliases) pre-filled with _VAL in NVC369_BUF_SIZE units (0x800=64KB, 0x80=4KB) so the local (non-RPC) path reads the same values the control handler reports. 3. VF-window NV_VIRTUAL_FUNCTION_PRIV_ACCESS_COUNTER_NOTIFY_BUFFER_SIZE (BAR0+0xB83110, plus plain alias) = 0x1000 (the value the host's physical TU104 reports; x32B = 128KB). uvmGetAccessCounterBufferSize_ TU102 multiplied our zero and uvmInitializeAccessCntrBuffer failed with NV_ERR_NO_MEMORY at mem_desc.c:1353. Also note (companion finding, no code change): the guest must be placed behind a pcie-root-port; on the Q35 root bus QEMU rewrites the PCIe cap to Root Complex Integrated Endpoint and zeroes LNKCAP/LNKCTL/DeviceCaps (vfio_setup_pcie_cap: "On a Root Complex bus Endpoints become Root Complex Integrated Endpoints, which changes the type and clears the LNK & LNK2 fields"), which is what made getPCIELinkRateMBps read speed 0.

kernel: drop PCFG poison experiment and config-write spam, keep bus-info detail log

2026-09-23T04:09:13+00:00

The poison test (fill every dword of BAR0 0x88000-0x88FFF, 0xB80000- 0xB80EFF, 0xC08000-0xC08EFF with 0x103) did not change the CUDA "Unknown PCIe speed" failure, proving the failing BUS_GET_INFO_V2 index-3 data does not come from any BAR0 mirror read. Combined with the QEMU config-space behavior (link registers zeroed, type=0x92, 0x84 never read from us) the value must come from the VGPU_STATIC_INFO branch in getBusInfos (pVSI-> pcieGpuLinkCaps), i.e. the guest needs the host vGPU static-info payload. Cleanup: remove the poison loops and the per-write CFGWR log; keep the once-per-init PCIe cap print and add per-entry BUS_GET_INFO_V2 logging (index+data) for the ongoing CUDA/UVM work.

kernel: real TU104 config-space layout (PCIe cap at 0x78) + CUDA/UVM tracing

2026-09-23T04:04:14+00:00

Move the emulated PCIe capability to config offset 0x78 and shift MSI to 0x48 / MSI-X to 0x68, matching the real TU104 (host lspci: PM 0x60, MSI 0x68, Express 0x78). This matters because the driver's NV_PCFG mirror (BAR0+0x88000, size 0x1000) is read as PCI config space: osDevReadReg032 -> vgpuDevReadReg032 intercepts [NV_PCFG, +0x30) (or the full 0x1000 for SRIOV) and forwards to osPciReadDword, so NV_XVE_LINK_CAPABILITIES (0x84) must be the PCIe Link Capabilities register, i.e. cap+0x0C with the cap at 0x78. Link Caps = Gen3 x16 (0x103), Link Status = 0x0103. Also set MCFG-visible caps: the guest now reports MSI [48], MSI-X [68], Express [78] exactly like the host GPU. Fixing the capability chain: the first attempt set the PCIe cap's NEXT to 0x0C, which broke enumeration (device 00:06.0 disappeared from the guest) - NEXT must terminate at 0. Unknown still open: QEMU presents type=0x92 (Root Complex Integrated Endpoint) and zeroes 0x84-0x8B in its config cache (never read from us, not refreshed by writes), and poisoning all BAR0 mirror windows with 0x103 does not change the driver's "Unknown PCIe speed" - so the value comes from the VGPU_STATIC_INFO branch (pVSI->pcieGpuLinkCaps, IS_VIRTUAL true via hypervisor detection) rather than any register read. Debug helpers added (temporary): CFGWR config-write log, PCFG change watch, PCIe-cap print at init.

kernel: widen PCFG mirror watch to link-cap aliases (debug helper)

2026-09-23T03:41:13+00:00

Logs changes to NV_PMC_BOOT_1 (0x4), the low PCFG alias, the BAR0 PCFG block (0x880000/0x880080/0x880084/0x880088/0x88008c) and the VF-window aliases. Observed during a CUDA run: the guest never writes any of them, so the link-cap values we pre-set (0x103 / 0x01030000) are not clobbered. Reads are not observable through the mmap'd BAR0.

kernel: FB/BUS info-list controls + PCIe link caps for the UVM path

2026-09-23T03:32:44+00:00

CUDA (cuInit) now reaches UVM's device create: libcuda opens nvidia-uvm, UVM calls nvGpuOpsDeviceCreate/gpuGetGpuInfo, and the driver fails at getPCIELinkRateMBps -> calculatePCIELinkRateMBps ("Unknown PCIe speed", NV_ERR_INVALID_STATE) -> nv_open_device aborts -> cuInit returns CUDA_ERROR_NOT_INITIALIZED (3). Implemented along the way: - FB_GET_INFO_V2 (0x20801303): echo the requested FB info indices and answer TU104 values (256MB RAM/heap/BAR1, GDDR6, 8 LTCs, 4MB L2, ...). Fixes the kern_mem_sys_ctrl.c:911 asserts and gives CUDA/UVM real memory sizes. - BUS_GET_INFO_V2 (0x20801823): echo indices; PCIe Gen3 x16 link caps / GEN info / link status, SYSMEM_CONNECTION_TYPE=PCIE. UVM's gpuGetSysmemLinkInfo reads SYSMEM_CONNECTION_TYPE and the nominal link bandwidth from here. - PCIe Link Capabilities in the emulated config space (0x8C/0x92) and in the BAR0 PCFG mirror (gpuReadBusConfigReg -> NV_PCFG+0x84 = BAR0+ 0x88084): MAX_SPEED=3 (8GT/s Gen3), MAX_WIDTH=0x10 (x16); link status mirror at 0x88088. Also mirrored under the VF-window/low aliases. - Temporary change-watch on PCFG[0x88084] to see if the guest overwrites it (it does not - stays 0x103). Verified: guest nvidia-smi -L still works and reports PCIe Generation Max/Current = 3. cuInit still fails; the remaining question is which address the driver's kbifGetGpuLinkCapabilities read sees as zero (the PCFG shadow holds 0x103). Guest-side kprobe tracing of that function panicked the VM (free_swap_cache page fault on a corrupted pointer) - possible guest memory corruption to investigate separately.

kernel: guest probe completes - nvidia-smi enumerates the emulated TU104

2026-09-23T02:40:26+00:00

Root cause of the golden-image NV_ERR_NO_MEMORY: STATIC_KGR_GET_CONTEXT_ BUFFERS_INFO (0x20800a32) fell through to the generic zeroed reply, so kgraphicsInitializeDeferredStaticData() stored a 0-byte main ctx buffer size and kgrctxAllocMainCtxBuffer() did a 0-page allocation (mem_desc.c:1353). Serve the TU104 per-engine ctx buffer table (26 engine IDs x 8 GR engines; NV_U32_MAX = unsupported, which the client skips). Second blocker after that: L2 cache-op and TLB-invalidate completion registers. Turing reaches them through the virtual-register window (GPU_VREG_* = BAR0 + sriovState.virtualRegPhysOffset + offset, 0xB80000 for a non-SRIOV virtual GPU): NV_VIRTUAL_FUNCTION_PRIV_MMU_INVALIDATE (0xB830B0, TRIGGER bit31) and the L2 invalidate/dirty-flush set (0x70000-0x70014 and 0xB80F00/0xB80F04). The driver soft-locked spinning in osDevReadReg032 on 0xB830B4 (kgmmuCheckPendingInvalidates_TU102) and timed out in kmemsysDoCacheOp_GM107; auto-clear all of them (both the plain and windowed aliases). Third: re-probe failed with GspStatusQueueInit msgqRxLink timeout because gsp_rm_emu_reset() was not called before a fresh boot, leaving rm->parsed set so the new libos args were never parsed. Reset GSP-RM state on the fresh FWSEC kick. Verified: guest nvidia-smi and nvidia-smi -L both exit 0 (full unload/reload cycle), GPU enumerated as "NVIDIA GeForce RTX 2080 SUPER" with 256MB, no NO_MEMORY / timeouts / soft lockups in the guest. Also add tools/rmq/grctx_oracle.c documenting that the host RM does not expose the ctx-buffer table to userspace (the real values are probed from the ctxsw ucode by GSP-RM), so the emulation serves synthesized sizes.

BAR2 verify test fix: remove PRAMIN sync + contiguous ALLOC + vidmem VA allocator

2026-09-23T01:55:54+00:00

Fixes the BAR2 verify test (kbusVerifyBar2_GM107): 1. Removed PRAMIN window sync from emulated BAR1 - the sync was overwriting test data written by the driver through the vBar2 aperture. The shadow BAR0 is already consistent (guest writes/reads hit the same shadow location). 2. Added contiguous page allocation for golden image (alloc_pages with order for large allocations, falling back to individual pages). 3. Added vidmem VA allocator - ALLOC now returns GPU VAs within the emulated GPU's vidmem range (0x100000 - fb_size) instead of default 4GB, so the memmgr accepts them. Verified: driver probes past RmInitNvDevice bus init, reaches golden image channel allocation (progress from fn-10 FREE timeout).

PRAMIN window translation + queue state preservation on Booter-unload

2026-09-22T14:59:59+00:00

Fixes the BAR2 verify test failure (kbusVerifyBar2_GM107): - GSP emulation thread now watches NV_PBUS_BAR0_WINDOW (BAR0+0x1700) for base/target changes and syncs the PRAMIN window (BAR0+0x700000) from the emulated BAR1 (vidmem) at (base << 16). This makes PRAMIN reads return correct vidmem content for the BAR2 verify test. Fixes the fn-10 FREE timeout during driver cleanup: - gsp_emu_clear_boot_state no longer resets the GSP-RM queue state on Booter-unload kick. The driver sends cleanup FREEs AFTER the Booter-unload kick; resetting the queue here caused those frees to time out. - Queue state is now only reset on: ENGINE reset (full GPU reset) or genuine queue death (insane writePtr). Both call gsp_rm_emu_reset explicitly. Also fixes rmq tool ioctl encoding: - Use _IOWR('F', nr, type) encoding for NVIDIA ioctls (was raw numbers) - Open /dev/nvidiactl (not /dev/nvidia0) for RM_ALLOC - Add NVOS64_PARAMETERS (48 bytes) for device/subdevice allocs Verified: BAR2 verify test passes, INSTLOC regkey read, driver probes past RmInitNvDevice bus init, reaches fn-10 FREE timeout during cleanup (same as handoff's characterized blocker).

host stability: BAR3 shadow + skip submodule nvidia + teardown-safe garbage handler

2026-09-22T13:43:22+00:00

Root causes of host GSP crash (WPR2 stuck, 'unexpected WPR2 already up'): 1. stack_manager load-nvidia tried to load our submodule nvidia.ko (610.57.04-base) alongside/over apt nvidia 570.211.01 -> GSP crash during driver conflict. Fix: skip submodule when apt nvidia is already loaded (not needed for QEMU). 2. BAR3 (32MB instance memory window) was mapped to the physical GPU's real BAR3, so guest BAR3 writes could corrupt the host GPU's instance memory. Fix: vzalloc'd shadow buffer (like BAR1 emulated vidmem), per-page mmap. 3. GSP-RM garbage element handler declared GSP death on teardown queue garbage, preventing driver cleanup frees from completing. Fix: stop polling quietly, let Booter-unload kick handle reset. Verified: full lifecycle (load, mdev create, QEMU guest boot, guest probe, mdev remove, unload, reload) with host nvidia-smi (apt 570.211.01) stable throughout. No GSP crash, no WPR2 stuck, no module unload hang.