diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-22 23:32:44 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-22 23:32:44 -0400 |
| commit | 14283fe73fb5e432a7af45de94961655714910f6 (patch) | |
| tree | e993bf56f3e2ef93195019ac0fd996d82204384a /kernel/bar_emulation.c | |
| parent | b34e4531d8f0b984dc15660d3ab406ae48822931 (diff) | |
| download | open-iov-14283fe73fb5e432a7af45de94961655714910f6.tar.gz open-iov-14283fe73fb5e432a7af45de94961655714910f6.zip | |
kernel: FB/BUS info-list controls + PCIe link caps for the UVM path
CUDA (cuInit) now reaches UVM's device create: libcuda opens nvidia-uvm,
UVM calls nvGpuOpsDeviceCreate/gpuGetGpuInfo, and the driver fails at
getPCIELinkRateMBps -> calculatePCIELinkRateMBps ("Unknown PCIe speed",
NV_ERR_INVALID_STATE) -> nv_open_device aborts -> cuInit returns
CUDA_ERROR_NOT_INITIALIZED (3).
Implemented along the way:
- FB_GET_INFO_V2 (0x20801303): echo the requested FB info indices and
answer TU104 values (256MB RAM/heap/BAR1, GDDR6, 8 LTCs, 4MB L2, ...).
Fixes the kern_mem_sys_ctrl.c:911 asserts and gives CUDA/UVM real
memory sizes.
- BUS_GET_INFO_V2 (0x20801823): echo indices; PCIe Gen3 x16 link caps /
GEN info / link status, SYSMEM_CONNECTION_TYPE=PCIE. UVM's
gpuGetSysmemLinkInfo reads SYSMEM_CONNECTION_TYPE and the nominal link
bandwidth from here.
- PCIe Link Capabilities in the emulated config space (0x8C/0x92) and in
the BAR0 PCFG mirror (gpuReadBusConfigReg -> NV_PCFG+0x84 = BAR0+
0x88084): MAX_SPEED=3 (8GT/s Gen3), MAX_WIDTH=0x10 (x16); link status
mirror at 0x88088. Also mirrored under the VF-window/low aliases.
- Temporary change-watch on PCFG[0x88084] to see if the guest overwrites
it (it does not - stays 0x103).
Verified: guest nvidia-smi -L still works and reports PCIe Generation
Max/Current = 3. cuInit still fails; the remaining question is which
address the driver's kbifGetGpuLinkCapabilities read sees as zero (the
PCFG shadow holds 0x103). Guest-side kprobe tracing of that function
panicked the VM (free_swap_cache page fault on a corrupted pointer) -
possible guest memory corruption to investigate separately.
Diffstat (limited to 'kernel/bar_emulation.c')
| -rw-r--r-- | kernel/bar_emulation.c | 30 |
1 files changed, 30 insertions, 0 deletions
diff --git a/kernel/bar_emulation.c b/kernel/bar_emulation.c index eaa006c..3448b8c 100644 --- a/kernel/bar_emulation.c +++ b/kernel/bar_emulation.c @@ -280,6 +280,36 @@ int nv_vgpu_bar_emul_init(struct nv_vgpu_device *vgpu) *(u32 *)((u8 *)ctx->bar0_shadow + 0x100ce0) = 0x00000018; /* + * NV_PCFG (BAR0+0x88000) - PCI config space mirror the driver reads + * via gpuReadBusConfigReg_HAL (GPU_BUS_CFG_RD32): + * 0x88084 NV_XVE_LINK_CAPABILITIES: MAX_LINK_SPEED 3:0 = 3 (8 GT/s, + * Gen3), MAX_LINK_WIDTH 9:4 = 0x10 (x16). + * 0x88088 NV_XVE_LINK_CONTROL_STATUS: negotiated speed 19:16 = 3, + * negotiated width 25:20 = 0x10 (x16). + * UVM's nvGpuOpsDeviceCreate -> getPCIELinkRateMBps -> + * kbifControlGetPCIEInfo -> kbifGetGpuLinkCapabilities reads 0x88084; + * zeros make calculatePCIELinkRateMBps log "Unknown PCIe speed" and + * return NV_ERR_INVALID_STATE, which fails cuInit + * (CUDA_ERROR_NOT_INITIALIZED in the guest). + * + * DEVICE_BASE(NV_PCFG) expands to 0 in this driver (nvmisc.h: + * "#define DEVICE_BASE(d) (0?d)"), so the read is actually + * gpuNvAddr + 0x84 - mirror the values at the low alias as well. + */ + *(u32 *)((u8 *)ctx->bar0_shadow + 0x88084) = 0x00000103; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x88088) = 0x01030000; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x00084) = 0x00000103; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x00088) = 0x01030000; + /* Same values under the VF register window aliases, in case this + * driver reaches the config mirror through virtualRegPhysOffset. */ + *(u32 *)((u8 *)ctx->bar0_shadow + 0xB80084) = 0x00000103; + *(u32 *)((u8 *)ctx->bar0_shadow + 0xB80088) = 0x01030000; + *(u32 *)((u8 *)ctx->bar0_shadow + 0xB88084) = 0x00000103; + *(u32 *)((u8 *)ctx->bar0_shadow + 0xB88088) = 0x01030000; + *(u32 *)((u8 *)ctx->bar0_shadow + 0xC08084) = 0x00000103; + *(u32 *)((u8 *)ctx->bar0_shadow + 0xC08088) = 0x01030000; + + /* * Serve a valid VBIOS at the PROM window (BAR0+0x300000). * * The guest driver's physical GSP boot (kgspInitRm -> TU102 path) |
