| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
kceGetDeviceCaps (kernel_ce.c) copies the caps the client hands to UVM from
NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS (0x20802a07), which reaches GSP-RM and
was answered with zeros. With capsTbl = {0,0} no CE had the SYSMEM bit, so
UVM's channel manager found no usable CE and UVM_REGISTER_GPU failed with
NV_ERR_NOT_SUPPORTED ("Failed to initialize the channel manager",
uvm_gpu.c:1574 - visible with uvm_debug_prints=1).
New handlers:
- CE_GET_PHYSICAL_CAPS (0x20802a07): {ceEngineType, capsTbl[2]} with
SYSMEM_READ|SYSMEM_WRITE|SYSMEM|P2P|BL_GT_64K (0xEC) for COPY0..COPY4,
NOT_SUPPORTED otherwise (matches subdeviceCtrlCmdCeGetCaps_IMPL).
- CE_GET_ALL_CAPS (0x20802a0a) / CE_GET_ALL_PHYSICAL_CAPS (0x20802a0b):
capsTbl[64][2] + present bitmap with the five CEs.
- CE_GET_CE_PCE_MASK (0x20802a02): pceMask = BIT(ceIdx) (one PCE per LCE).
Verified in-guest progress: UVM now gets past the channel manager and
creates channels; the register path's new blocker is the first channel
push (channel_init, uvm_channel.c:2535) whose semaphore release the
emulated GPU must complete ("Tracker entry ... stuck waiting for 90s",
sema VA 0x121000000 on channel 1:3 CE 0).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Three fixes that clear all driver-visible errors in UVM's device create
(cuInit still fails later in UVM_REGISTER_GPU):
1. GSP_RM_CONTROL 0x20800a59 INTERNAL_GMMU_GET_STATIC_INFO (new handler):
kgmmu.pStaticInfo->replayableFaultBufferSize feeds
kgmmuFaultBufferReplayableAllocate (called by faultbufConstruct when UVM
opens the device). The generic zero reply made it a 0-page allocation:
"faultbufConstruct_IMPL: Failed to setup Replayable Fault buffer
(status=0x51)". Report 64KB replayable / 4KB legacy, mirroring
kgmmuSetAndGetDefaultFaultBufferSize_TU102 defaults.
2. VF-window NV_VIRTUAL_FUNCTION_PRIV_MMU_FAULT_BUFFER_SIZE(0/1)
(BAR0+0xB83010/0xB83030, plus plain aliases) pre-filled with _VAL in
NVC369_BUF_SIZE units (0x800=64KB, 0x80=4KB) so the local (non-RPC)
path reads the same values the control handler reports.
3. VF-window NV_VIRTUAL_FUNCTION_PRIV_ACCESS_COUNTER_NOTIFY_BUFFER_SIZE
(BAR0+0xB83110, plus plain alias) = 0x1000 (the value the host's
physical TU104 reports; x32B = 128KB). uvmGetAccessCounterBufferSize_
TU102 multiplied our zero and uvmInitializeAccessCntrBuffer failed with
NV_ERR_NO_MEMORY at mem_desc.c:1353.
Also note (companion finding, no code change): the guest must be placed
behind a pcie-root-port; on the Q35 root bus QEMU rewrites the PCIe cap to
Root Complex Integrated Endpoint and zeroes LNKCAP/LNKCTL/DeviceCaps
(vfio_setup_pcie_cap: "On a Root Complex bus Endpoints become Root Complex
Integrated Endpoints, which changes the type and clears the LNK & LNK2
fields"), which is what made getPCIELinkRateMBps read speed 0.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
CUDA (cuInit) now reaches UVM's device create: libcuda opens nvidia-uvm,
UVM calls nvGpuOpsDeviceCreate/gpuGetGpuInfo, and the driver fails at
getPCIELinkRateMBps -> calculatePCIELinkRateMBps ("Unknown PCIe speed",
NV_ERR_INVALID_STATE) -> nv_open_device aborts -> cuInit returns
CUDA_ERROR_NOT_INITIALIZED (3).
Implemented along the way:
- FB_GET_INFO_V2 (0x20801303): echo the requested FB info indices and
answer TU104 values (256MB RAM/heap/BAR1, GDDR6, 8 LTCs, 4MB L2, ...).
Fixes the kern_mem_sys_ctrl.c:911 asserts and gives CUDA/UVM real
memory sizes.
- BUS_GET_INFO_V2 (0x20801823): echo indices; PCIe Gen3 x16 link caps /
GEN info / link status, SYSMEM_CONNECTION_TYPE=PCIE. UVM's
gpuGetSysmemLinkInfo reads SYSMEM_CONNECTION_TYPE and the nominal link
bandwidth from here.
- PCIe Link Capabilities in the emulated config space (0x8C/0x92) and in
the BAR0 PCFG mirror (gpuReadBusConfigReg -> NV_PCFG+0x84 = BAR0+
0x88084): MAX_SPEED=3 (8GT/s Gen3), MAX_WIDTH=0x10 (x16); link status
mirror at 0x88088. Also mirrored under the VF-window/low aliases.
- Temporary change-watch on PCFG[0x88084] to see if the guest overwrites
it (it does not - stays 0x103).
Verified: guest nvidia-smi -L still works and reports PCIe Generation
Max/Current = 3. cuInit still fails; the remaining question is which
address the driver's kbifGetGpuLinkCapabilities read sees as zero (the
PCFG shadow holds 0x103). Guest-side kprobe tracing of that function
panicked the VM (free_swap_cache page fault on a corrupted pointer) -
possible guest memory corruption to investigate separately.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Root cause of the golden-image NV_ERR_NO_MEMORY: STATIC_KGR_GET_CONTEXT_
BUFFERS_INFO (0x20800a32) fell through to the generic zeroed reply, so
kgraphicsInitializeDeferredStaticData() stored a 0-byte main ctx buffer
size and kgrctxAllocMainCtxBuffer() did a 0-page allocation
(mem_desc.c:1353). Serve the TU104 per-engine ctx buffer table (26 engine
IDs x 8 GR engines; NV_U32_MAX = unsupported, which the client skips).
Second blocker after that: L2 cache-op and TLB-invalidate completion
registers. Turing reaches them through the virtual-register window
(GPU_VREG_* = BAR0 + sriovState.virtualRegPhysOffset + offset, 0xB80000
for a non-SRIOV virtual GPU): NV_VIRTUAL_FUNCTION_PRIV_MMU_INVALIDATE
(0xB830B0, TRIGGER bit31) and the L2 invalidate/dirty-flush set
(0x70000-0x70014 and 0xB80F00/0xB80F04). The driver soft-locked spinning
in osDevReadReg032 on 0xB830B4 (kgmmuCheckPendingInvalidates_TU102) and
timed out in kmemsysDoCacheOp_GM107; auto-clear all of them (both the
plain and windowed aliases).
Third: re-probe failed with GspStatusQueueInit msgqRxLink timeout because
gsp_rm_emu_reset() was not called before a fresh boot, leaving rm->parsed
set so the new libos args were never parsed. Reset GSP-RM state on the
fresh FWSEC kick.
Verified: guest nvidia-smi and nvidia-smi -L both exit 0 (full
unload/reload cycle), GPU enumerated as "NVIDIA GeForce RTX 2080 SUPER"
with 256MB, no NO_MEMORY / timeouts / soft lockups in the guest.
Also add tools/rmq/grctx_oracle.c documenting that the host RM does not
expose the ctx-buffer table to userspace (the real values are probed from
the ctxsw ucode by GSP-RM), so the emulation serves synthesized sizes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- STATIC_KGR_GET_FLOORSWEEPING_MASKS(0x20800a26): TU104 gpcMask=0x3f
(6 GPCs), 4 TPC/GPC. numGpcs==0 killed krcWatchdogInit -> RmInitAdapter
bailed ('numGpcs > 0' assert at kernel_graphics_object.c:191).
- CPU interrupt tree (osVerifyInterrupts): driver accesses VF window at
virtualRegPhysOffset=NV_VIRTUAL_FUNCTION_FULL_PHYS_OFFSET(0xB80000)+off,
NOT 0x1640 (was watching wrong offset). On LEAF_TRIGGER write, set
LEAF/TOP pending bits + inject MSI-X so the sanity-test ISR runs -
osVerifySystemEnvironment now passes.
- WPR2 race: clear boot state BEFORE CPUCTL=HALTED on Booter-unload kick
so WPR2 is down when driver checks kgspIsWpr2Up ('WPR2 is still up').
- Dedicated GSP_RM_FREE(10) handler: echo NVOS00_PARAMETERS + status=NV_OK
(proper free_v03_00 form). fn-10 timeout persists despite this -
characterized as driver-side reply-matching issue, not content.
|
| |
|
|
|
|
|
|
|
| |
Driver caches the GR infoList as pGrInfo and reads MAX_SUBCONTEXT_COUNT
(0x2c) - zero made channel-group numMax==0, asserting channel allocation
(numMax == numFree && numMax != 0 -> NV_ERR_INVALID_STATE, blocking the
per-device open -> NVML 'No devices'). Fill litter topology (6 GPC x
4 TPC x 2 SM = 48 SMs) and subcontext counts = 64
(NV_PGRAPH_PRI_FE_CHIP_DEF_INFO_MAX_VEID_COUNT_INIT).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
- gsp_emu_poll_boot generic SEC2 kick path: clear SEC2_MAILBOX0=0
(success) - Booter Unload during GPU shutdown read a stale 0xff,
failing the unload and killing the subsequent re-init ('Booter failed
with non-zero error code: 0xff' -> RmInitAdapter 0x40). A SEC2 kick
after a completed boot is a GSP shutdown/reload: clear boot state.
Driver unload/reload cycle (last close -> RmShutdown, next open ->
RmInitAdapter) now works.
- STATIC_KGR_GET_GLOBAL_SM_ORDER(0x20800a22): TU104 topology 6 GPC x
4 TPC x 2 SM = 48 SMs/24 TPCs on GR0. kgrStaticInit is init-critical
(NV_CHECK_OK_OR_GOTO); zeros left numSm=0 and NVML spun on empty
topology.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
CRITICAL protocol fix (msgq.c): writePtr/readPtr are slot indices that
WRAP at msgCount ('writePtr += n; if (>= msgCount) -= msgCount'), NOT
monotonic counters. '62 -> 0' is a legitimate wrap, not queue death.
- gsp_rm_poll_commands: consume while (cmd_read != writePtr), advance
cmd_read = (cmd_read + elem_count) % msgcount. Death detection now
only: writePtr >= msgCount (garbage header) / bad element signature /
insane elemCount. Removes false deaths on healthy wraps that killed
the emulator mid-session (fn 10 FREE timeouts).
- gsp_rm_send: published status writePtr now wraps too (driver's
free-space/slot math breaks on unbounded values >= 2*msgCount).
- gsp_rm_emu_poll: back off on_booter_done retries to 250ms (was every
100us poll -> thousands of 'will retry' lines/sec, dmesg unusable);
retry log now pr_warn_ratelimited.
- CE_GET_FAULT_METHOD_BUFFER_SIZE(0x20802a08) = 12288 (GV100 formula
inherited by Turing: 135*(13*5+2) page-aligned, 5 PCEs). Zero reply
failed fault method buffer allocation (NV_ERR_NO_MEMORY) killing
channel-group creation.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Driver's intrCacheIntrFields requires subtree(ACCESS_CNTR=72)=1 within
subtreeMap[UVM_OWNED] and UVM_SHARED to be a single subtree. Zeroed map
failed RmInitNvDevice state load ('unexpected subtree').
Map matches intrInitSubtreeMap_TU102 (what physical-path guests get from
real GSP-RM): ESCHED{3,3} ESCHED_NOTIF{0,0} UVM_OWNED{1,1} UVM_SHARED{2,2},
DEFAULT/RUNLIST/RUNLIST_NOTIF=INVALID(0xFF). Struct retyped to 7x2B;
out_params_size spans full struct (driver reads subtreeMap at offset 2052).
Verified: probe passes state load, /dev/nvidia0 persists.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
NUM_CHANNELS/PCI_BAR_INFO/DISP_STATIC_INFO/DISP_IP_VERSION/I2C
- FIFO_GET_NUM_CHANNELS(0x20800a61)=64/runlist: zero tripped
NV_ASSERT(numChannels>0), kdisp clientChannelTable alloc failed.
- BUS_GET_PCI_BAR_INFO(0x20801803): report BAR0=16MB/BAR1=256MB/
BAR2=1MB instmem window@BAR0+0x700000/BAR3=32MB (was all zeros).
- DISPLAY_GET_STATIC_INFO(0x20800a01): heads=1, numDispChannels=1
(36B wire layout; zero -> clientChannelTable[0] alloc failed).
- DISPLAY_GET_IP_VERSION(0x20800a4b)=0x04000000 (Turing DISP v04.00;
driver passes uninitialized input - must return a real version).
- I2C_TRANSACTION(0x402c0105): echo request params + NOT_SUPPORTED.
Variable-length FINN-serialized params: zeroed reply fails
DeserializeUp and the calling ioctl never completes (nvidia-smi hang).
- Default unknown-cmd policy back to zeroed params (echo is dangerous
for queries with uninitialized input structs).
Milestone: guest RmInitAdapter completed, /dev/nvidia0 created.
|
|
|
GSP_RM_CONTROL)
Implements the GSP-RM reply content layer for the msgq RPC channel:
- kernel/gsp_rm_emulation.c/h: GSP-RM msgq emulation engine
- Queue chain discovery (libos args → RMARGS → GSP_ARGUMENTS_CACHED)
- Status queue link + GSP_INIT_DONE emission
- Command queue polling + RPC dispatch
- SET_GUEST_SYSTEM_INFO version negotiation (VGX 41.12)
- GET_GSP_STATIC_INFO(65) with 1656-byte GspStaticConfigInfo
- GSP_RM_CONTROL(76) sub-command dispatch:
- FIFO_GET_DEVICE_INFO_TABLE (0x20801112)
- INTERNAL_INTR_GET_KERNEL_TABLE (0x20800a5c)
- GPU_GET_CONSTRUCTED_FALCON_INFO (0x208001b0)
- INTERNAL_GET_DEVICE_INFO_TABLE (0x20800a40)
- kernel/gsp_rm_tu104.h: TU104 hardware tables
- 12-entry FIFO engine table (GR0, CE0-4, NVDEC0-1, NVENC0, SEC2, NVJPG, SW)
- 8-entry interrupt table (from intr_phys_tu102.c)
- 1-entry constructed falcon table (FBFLCN)
- 11-entry internal device info table
- GspStaticConfigInfo layout (1656 bytes, ABI-verified offsets)
- Data from NVIDIA driver material (ext-tree) + hwref/turing/tu104
- tools/rmq/: Userspace RM API client (oracle for ground-truth tables)
KNOWN ISSUE: gsp_rm_send crashes (fortify_panic) on multi-element messages
when reply data exceeds one 4096-byte queue element. Affects cmd 0x20800a41
(paramsSize=8204). Multi-element splitting is the next fix.
|
