open-iov.git/kernel/gsp_rm_emulation.c, branch archive/pre-squash

kernel: serve CE caps + CE->PCE masks (unblocks UVM channel manager)

2026-09-23T05:26:25+00:00

kceGetDeviceCaps (kernel_ce.c) copies the caps the client hands to UVM from NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS (0x20802a07), which reaches GSP-RM and was answered with zeros. With capsTbl = {0,0} no CE had the SYSMEM bit, so UVM's channel manager found no usable CE and UVM_REGISTER_GPU failed with NV_ERR_NOT_SUPPORTED ("Failed to initialize the channel manager", uvm_gpu.c:1574 - visible with uvm_debug_prints=1). New handlers: - CE_GET_PHYSICAL_CAPS (0x20802a07): {ceEngineType, capsTbl[2]} with SYSMEM_READ|SYSMEM_WRITE|SYSMEM|P2P|BL_GT_64K (0xEC) for COPY0..COPY4, NOT_SUPPORTED otherwise (matches subdeviceCtrlCmdCeGetCaps_IMPL). - CE_GET_ALL_CAPS (0x20802a0a) / CE_GET_ALL_PHYSICAL_CAPS (0x20802a0b): capsTbl[64][2] + present bitmap with the five CEs. - CE_GET_CE_PCE_MASK (0x20802a02): pceMask = BIT(ceIdx) (one PCE per LCE). Verified in-guest progress: UVM now gets past the channel manager and creates channels; the register path's new blocker is the first channel push (channel_init, uvm_channel.c:2535) whose semaphore release the emulated GPU must complete ("Tracker entry ... stuck waiting for 90s", sema VA 0x121000000 on channel 1:3 CE 0).

kernel: serve GMMU static info + per-VF fault/access-counter sizes for UVM

2026-09-23T04:41:46+00:00

Three fixes that clear all driver-visible errors in UVM's device create (cuInit still fails later in UVM_REGISTER_GPU): 1. GSP_RM_CONTROL 0x20800a59 INTERNAL_GMMU_GET_STATIC_INFO (new handler): kgmmu.pStaticInfo->replayableFaultBufferSize feeds kgmmuFaultBufferReplayableAllocate (called by faultbufConstruct when UVM opens the device). The generic zero reply made it a 0-page allocation: "faultbufConstruct_IMPL: Failed to setup Replayable Fault buffer (status=0x51)". Report 64KB replayable / 4KB legacy, mirroring kgmmuSetAndGetDefaultFaultBufferSize_TU102 defaults. 2. VF-window NV_VIRTUAL_FUNCTION_PRIV_MMU_FAULT_BUFFER_SIZE(0/1) (BAR0+0xB83010/0xB83030, plus plain aliases) pre-filled with _VAL in NVC369_BUF_SIZE units (0x800=64KB, 0x80=4KB) so the local (non-RPC) path reads the same values the control handler reports. 3. VF-window NV_VIRTUAL_FUNCTION_PRIV_ACCESS_COUNTER_NOTIFY_BUFFER_SIZE (BAR0+0xB83110, plus plain alias) = 0x1000 (the value the host's physical TU104 reports; x32B = 128KB). uvmGetAccessCounterBufferSize_ TU102 multiplied our zero and uvmInitializeAccessCntrBuffer failed with NV_ERR_NO_MEMORY at mem_desc.c:1353. Also note (companion finding, no code change): the guest must be placed behind a pcie-root-port; on the Q35 root bus QEMU rewrites the PCIe cap to Root Complex Integrated Endpoint and zeroes LNKCAP/LNKCTL/DeviceCaps (vfio_setup_pcie_cap: "On a Root Complex bus Endpoints become Root Complex Integrated Endpoints, which changes the type and clears the LNK & LNK2 fields"), which is what made getPCIELinkRateMBps read speed 0.

kernel: drop PCFG poison experiment and config-write spam, keep bus-info detail log

2026-09-23T04:09:13+00:00

The poison test (fill every dword of BAR0 0x88000-0x88FFF, 0xB80000- 0xB80EFF, 0xC08000-0xC08EFF with 0x103) did not change the CUDA "Unknown PCIe speed" failure, proving the failing BUS_GET_INFO_V2 index-3 data does not come from any BAR0 mirror read. Combined with the QEMU config-space behavior (link registers zeroed, type=0x92, 0x84 never read from us) the value must come from the VGPU_STATIC_INFO branch in getBusInfos (pVSI-> pcieGpuLinkCaps), i.e. the guest needs the host vGPU static-info payload. Cleanup: remove the poison loops and the per-write CFGWR log; keep the once-per-init PCIe cap print and add per-entry BUS_GET_INFO_V2 logging (index+data) for the ongoing CUDA/UVM work.

kernel: FB/BUS info-list controls + PCIe link caps for the UVM path

2026-09-23T03:32:44+00:00

CUDA (cuInit) now reaches UVM's device create: libcuda opens nvidia-uvm, UVM calls nvGpuOpsDeviceCreate/gpuGetGpuInfo, and the driver fails at getPCIELinkRateMBps -> calculatePCIELinkRateMBps ("Unknown PCIe speed", NV_ERR_INVALID_STATE) -> nv_open_device aborts -> cuInit returns CUDA_ERROR_NOT_INITIALIZED (3). Implemented along the way: - FB_GET_INFO_V2 (0x20801303): echo the requested FB info indices and answer TU104 values (256MB RAM/heap/BAR1, GDDR6, 8 LTCs, 4MB L2, ...). Fixes the kern_mem_sys_ctrl.c:911 asserts and gives CUDA/UVM real memory sizes. - BUS_GET_INFO_V2 (0x20801823): echo indices; PCIe Gen3 x16 link caps / GEN info / link status, SYSMEM_CONNECTION_TYPE=PCIE. UVM's gpuGetSysmemLinkInfo reads SYSMEM_CONNECTION_TYPE and the nominal link bandwidth from here. - PCIe Link Capabilities in the emulated config space (0x8C/0x92) and in the BAR0 PCFG mirror (gpuReadBusConfigReg -> NV_PCFG+0x84 = BAR0+ 0x88084): MAX_SPEED=3 (8GT/s Gen3), MAX_WIDTH=0x10 (x16); link status mirror at 0x88088. Also mirrored under the VF-window/low aliases. - Temporary change-watch on PCFG[0x88084] to see if the guest overwrites it (it does not - stays 0x103). Verified: guest nvidia-smi -L still works and reports PCIe Generation Max/Current = 3. cuInit still fails; the remaining question is which address the driver's kbifGetGpuLinkCapabilities read sees as zero (the PCFG shadow holds 0x103). Guest-side kprobe tracing of that function panicked the VM (free_swap_cache page fault on a corrupted pointer) - possible guest memory corruption to investigate separately.

kernel: guest probe completes - nvidia-smi enumerates the emulated TU104

2026-09-23T02:40:26+00:00

Root cause of the golden-image NV_ERR_NO_MEMORY: STATIC_KGR_GET_CONTEXT_ BUFFERS_INFO (0x20800a32) fell through to the generic zeroed reply, so kgraphicsInitializeDeferredStaticData() stored a 0-byte main ctx buffer size and kgrctxAllocMainCtxBuffer() did a 0-page allocation (mem_desc.c:1353). Serve the TU104 per-engine ctx buffer table (26 engine IDs x 8 GR engines; NV_U32_MAX = unsupported, which the client skips). Second blocker after that: L2 cache-op and TLB-invalidate completion registers. Turing reaches them through the virtual-register window (GPU_VREG_* = BAR0 + sriovState.virtualRegPhysOffset + offset, 0xB80000 for a non-SRIOV virtual GPU): NV_VIRTUAL_FUNCTION_PRIV_MMU_INVALIDATE (0xB830B0, TRIGGER bit31) and the L2 invalidate/dirty-flush set (0x70000-0x70014 and 0xB80F00/0xB80F04). The driver soft-locked spinning in osDevReadReg032 on 0xB830B4 (kgmmuCheckPendingInvalidates_TU102) and timed out in kmemsysDoCacheOp_GM107; auto-clear all of them (both the plain and windowed aliases). Third: re-probe failed with GspStatusQueueInit msgqRxLink timeout because gsp_rm_emu_reset() was not called before a fresh boot, leaving rm->parsed set so the new libos args were never parsed. Reset GSP-RM state on the fresh FWSEC kick. Verified: guest nvidia-smi and nvidia-smi -L both exit 0 (full unload/reload cycle), GPU enumerated as "NVIDIA GeForce RTX 2080 SUPER" with 256MB, no NO_MEMORY / timeouts / soft lockups in the guest. Also add tools/rmq/grctx_oracle.c documenting that the host RM does not expose the ctx-buffer table to userspace (the real values are probed from the ctxsw ucode by GSP-RM), so the emulation serves synthesized sizes.

PRAMIN window translation + queue state preservation on Booter-unload

2026-09-22T14:59:59+00:00

Fixes the BAR2 verify test failure (kbusVerifyBar2_GM107): - GSP emulation thread now watches NV_PBUS_BAR0_WINDOW (BAR0+0x1700) for base/target changes and syncs the PRAMIN window (BAR0+0x700000) from the emulated BAR1 (vidmem) at (base << 16). This makes PRAMIN reads return correct vidmem content for the BAR2 verify test. Fixes the fn-10 FREE timeout during driver cleanup: - gsp_emu_clear_boot_state no longer resets the GSP-RM queue state on Booter-unload kick. The driver sends cleanup FREEs AFTER the Booter-unload kick; resetting the queue here caused those frees to time out. - Queue state is now only reset on: ENGINE reset (full GPU reset) or genuine queue death (insane writePtr). Both call gsp_rm_emu_reset explicitly. Also fixes rmq tool ioctl encoding: - Use _IOWR('F', nr, type) encoding for NVIDIA ioctls (was raw numbers) - Open /dev/nvidiactl (not /dev/nvidia0) for RM_ALLOC - Add NVOS64_PARAMETERS (48 bytes) for device/subdevice allocs Verified: BAR2 verify test passes, INSTLOC regkey read, driver probes past RmInitNvDevice bus init, reaches fn-10 FREE timeout during cleanup (same as handoff's characterized blocker).

host stability: BAR3 shadow + skip submodule nvidia + teardown-safe garbage handler

2026-09-22T13:43:22+00:00

Root causes of host GSP crash (WPR2 stuck, 'unexpected WPR2 already up'): 1. stack_manager load-nvidia tried to load our submodule nvidia.ko (610.57.04-base) alongside/over apt nvidia 570.211.01 -> GSP crash during driver conflict. Fix: skip submodule when apt nvidia is already loaded (not needed for QEMU). 2. BAR3 (32MB instance memory window) was mapped to the physical GPU's real BAR3, so guest BAR3 writes could corrupt the host GPU's instance memory. Fix: vzalloc'd shadow buffer (like BAR1 emulated vidmem), per-page mmap. 3. GSP-RM garbage element handler declared GSP death on teardown queue garbage, preventing driver cleanup frees from completing. Fix: stop polling quietly, let Booter-unload kick handle reset. Verified: full lifecycle (load, mdev create, QEMU guest boot, guest probe, mdev remove, unload, reload) with host nvidia-smi (apt 570.211.01) stable throughout. No GSP crash, no WPR2 stuck, no module unload hang.

kernel: revert fb to 256MB (512MB broke FWSEC WPR2-location check)

2026-09-22T07:58:30+00:00

The FB=512MB change made the FWSEC WPR2-location check fail ('WPR2 initialized at an unexpected location: 0x1fe00 (expected 0xfe00)') because NV_PFB_PRI_MMU_LOCAL_MEMORY_RANGE must stay consistent with GSCI fb_length. Reverted to 256MB everywhere. The golden-image GR context buffer NO_MEMORY is a heap free-list issue, not fb size.

kernel: floorsweeping masks + interrupt tree + WPR2 race + fn-10 handler

2026-09-22T06:16:18+00:00

- STATIC_KGR_GET_FLOORSWEEPING_MASKS(0x20800a26): TU104 gpcMask=0x3f (6 GPCs), 4 TPC/GPC. numGpcs==0 killed krcWatchdogInit -> RmInitAdapter bailed ('numGpcs > 0' assert at kernel_graphics_object.c:191). - CPU interrupt tree (osVerifyInterrupts): driver accesses VF window at virtualRegPhysOffset=NV_VIRTUAL_FUNCTION_FULL_PHYS_OFFSET(0xB80000)+off, NOT 0x1640 (was watching wrong offset). On LEAF_TRIGGER write, set LEAF/TOP pending bits + inject MSI-X so the sanity-test ISR runs - osVerifySystemEnvironment now passes. - WPR2 race: clear boot state BEFORE CPUCTL=HALTED on Booter-unload kick so WPR2 is down when driver checks kgspIsWpr2Up ('WPR2 is still up'). - Dedicated GSP_RM_FREE(10) handler: echo NVOS00_PARAMETERS + status=NV_OK (proper free_v03_00 form). fn-10 timeout persists despite this - characterized as driver-side reply-matching issue, not content.

kernel: fn-10 delay tuned to 2ms (partial mitigation, not full fix)

2026-09-22T04:05:39+00:00

2ms udelay before fn-10/47 replies. Mitigates but does not fully eliminate the intermittent fn-10 FREE timeout (driver-side reply- matching issue). Verified reply path correct: seqNum matches driver's rxSeqNum (poison experiment 'Expected 75 got 175'), checksum folds to 0, function=10, writePtr/readPtr land, driver drains 1:1. Root cause of the driver's intermittent non-acceptance of fn-10/47 replies during cleanup remains under investigation (needs real-GSP oracle or guest- kernel GDB inspection of the msgq reply-matching).