| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
kceGetDeviceCaps (kernel_ce.c) copies the caps the client hands to UVM from
NV2080_CTRL_CMD_CE_GET_PHYSICAL_CAPS (0x20802a07), which reaches GSP-RM and
was answered with zeros. With capsTbl = {0,0} no CE had the SYSMEM bit, so
UVM's channel manager found no usable CE and UVM_REGISTER_GPU failed with
NV_ERR_NOT_SUPPORTED ("Failed to initialize the channel manager",
uvm_gpu.c:1574 - visible with uvm_debug_prints=1).
New handlers:
- CE_GET_PHYSICAL_CAPS (0x20802a07): {ceEngineType, capsTbl[2]} with
SYSMEM_READ|SYSMEM_WRITE|SYSMEM|P2P|BL_GT_64K (0xEC) for COPY0..COPY4,
NOT_SUPPORTED otherwise (matches subdeviceCtrlCmdCeGetCaps_IMPL).
- CE_GET_ALL_CAPS (0x20802a0a) / CE_GET_ALL_PHYSICAL_CAPS (0x20802a0b):
capsTbl[64][2] + present bitmap with the five CEs.
- CE_GET_CE_PCE_MASK (0x20802a02): pceMask = BIT(ceIdx) (one PCE per LCE).
Verified in-guest progress: UVM now gets past the channel manager and
creates channels; the register path's new blocker is the first channel
push (channel_init, uvm_channel.c:2535) whose semaphore release the
emulated GPU must complete ("Tracker entry ... stuck waiting for 90s",
sema VA 0x121000000 on channel 1:3 CE 0).
|
| |
|
|
| |
script, README)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The ptrace tracer resolves the current CUDA blocker: libcuda issues raw
syscall instructions, so LD_PRELOAD shims and ltrace see nothing; following
the child with PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO does. It found
UVM_REGISTER_GPU (ioctl 0x25) returning rmStatus=0x56 NV_ERR_NOT_SUPPORTED,
with inputs identical to the working host run (rmCtrlFd=-1, hClient=0,
hSmcPartRef=0). Also dumps NV_ESC_RM_CONTROL cmd/status and RM alloc/free
statuses (all zero) and the UVM ioctl set (guest stops after
0x27,0x1,0x25,0x2 vs 141 ioctls on the host).
launch_qemu_rootport.sh is the working QEMU command line with the
pcie-root-port fix (see its header + the handoff PART 4); vmcycle.sh drives
the module/mdev/VM reload cycle around it.
|
| |
|
|
|
|
|
|
| |
Documents the current CUDA blocker: cuInit returns CUDA_ERROR_NOT_INITIALIZED
(3) on the emulated GPU even though every nvidia.ko ioctl succeeds and
RmInitAdapter completes. libcuda opens /dev/nvidia-uvm before /dev/nvidia0,
so the next milestone is making nvidia-uvm initialize against the emulated
MMU/VAS model.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Root cause of the golden-image NV_ERR_NO_MEMORY: STATIC_KGR_GET_CONTEXT_
BUFFERS_INFO (0x20800a32) fell through to the generic zeroed reply, so
kgraphicsInitializeDeferredStaticData() stored a 0-byte main ctx buffer
size and kgrctxAllocMainCtxBuffer() did a 0-page allocation
(mem_desc.c:1353). Serve the TU104 per-engine ctx buffer table (26 engine
IDs x 8 GR engines; NV_U32_MAX = unsupported, which the client skips).
Second blocker after that: L2 cache-op and TLB-invalidate completion
registers. Turing reaches them through the virtual-register window
(GPU_VREG_* = BAR0 + sriovState.virtualRegPhysOffset + offset, 0xB80000
for a non-SRIOV virtual GPU): NV_VIRTUAL_FUNCTION_PRIV_MMU_INVALIDATE
(0xB830B0, TRIGGER bit31) and the L2 invalidate/dirty-flush set
(0x70000-0x70014 and 0xB80F00/0xB80F04). The driver soft-locked spinning
in osDevReadReg032 on 0xB830B4 (kgmmuCheckPendingInvalidates_TU102) and
timed out in kmemsysDoCacheOp_GM107; auto-clear all of them (both the
plain and windowed aliases).
Third: re-probe failed with GspStatusQueueInit msgqRxLink timeout because
gsp_rm_emu_reset() was not called before a fresh boot, leaving rm->parsed
set so the new libos args were never parsed. Reset GSP-RM state on the
fresh FWSEC kick.
Verified: guest nvidia-smi and nvidia-smi -L both exit 0 (full
unload/reload cycle), GPU enumerated as "NVIDIA GeForce RTX 2080 SUPER"
with 256MB, no NO_MEMORY / timeouts / soft lockups in the guest.
Also add tools/rmq/grctx_oracle.c documenting that the host RM does not
expose the ctx-buffer table to userspace (the real values are probed from
the ctxsw ucode by GSP-RM), so the emulation serves synthesized sizes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes the BAR2 verify test failure (kbusVerifyBar2_GM107):
- GSP emulation thread now watches NV_PBUS_BAR0_WINDOW (BAR0+0x1700) for
base/target changes and syncs the PRAMIN window (BAR0+0x700000) from the
emulated BAR1 (vidmem) at (base << 16). This makes PRAMIN reads return
correct vidmem content for the BAR2 verify test.
Fixes the fn-10 FREE timeout during driver cleanup:
- gsp_emu_clear_boot_state no longer resets the GSP-RM queue state on
Booter-unload kick. The driver sends cleanup FREEs AFTER the Booter-unload
kick; resetting the queue here caused those frees to time out.
- Queue state is now only reset on: ENGINE reset (full GPU reset) or
genuine queue death (insane writePtr). Both call gsp_rm_emu_reset
explicitly.
Also fixes rmq tool ioctl encoding:
- Use _IOWR('F', nr, type) encoding for NVIDIA ioctls (was raw numbers)
- Open /dev/nvidiactl (not /dev/nvidia0) for RM_ALLOC
- Add NVOS64_PARAMETERS (48 bytes) for device/subdevice allocs
Verified: BAR2 verify test passes, INSTLOC regkey read, driver probes
past RmInitNvDevice bus init, reaches fn-10 FREE timeout during cleanup
(same as handoff's characterized blocker).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
GSP_RM_CONTROL)
Implements the GSP-RM reply content layer for the msgq RPC channel:
- kernel/gsp_rm_emulation.c/h: GSP-RM msgq emulation engine
- Queue chain discovery (libos args → RMARGS → GSP_ARGUMENTS_CACHED)
- Status queue link + GSP_INIT_DONE emission
- Command queue polling + RPC dispatch
- SET_GUEST_SYSTEM_INFO version negotiation (VGX 41.12)
- GET_GSP_STATIC_INFO(65) with 1656-byte GspStaticConfigInfo
- GSP_RM_CONTROL(76) sub-command dispatch:
- FIFO_GET_DEVICE_INFO_TABLE (0x20801112)
- INTERNAL_INTR_GET_KERNEL_TABLE (0x20800a5c)
- GPU_GET_CONSTRUCTED_FALCON_INFO (0x208001b0)
- INTERNAL_GET_DEVICE_INFO_TABLE (0x20800a40)
- kernel/gsp_rm_tu104.h: TU104 hardware tables
- 12-entry FIFO engine table (GR0, CE0-4, NVDEC0-1, NVENC0, SEC2, NVJPG, SW)
- 8-entry interrupt table (from intr_phys_tu102.c)
- 1-entry constructed falcon table (FBFLCN)
- 11-entry internal device info table
- GspStaticConfigInfo layout (1656 bytes, ABI-verified offsets)
- Data from NVIDIA driver material (ext-tree) + hwref/turing/tu104
- tools/rmq/: Userspace RM API client (oracle for ground-truth tables)
KNOWN ISSUE: gsp_rm_send crashes (fortify_panic) on multi-element messages
when reply data exceeds one 4096-byte queue element. Affects cmd 0x20800a41
(paramsSize=8204). Multi-element splitting is the next fix.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- install.sh: one-command fresh-Ubuntu reproduction (deps, NVIDIA clone
+ patch, GSP firmware fetch from linux-firmware.git with objcopy
.fwimage extraction + MD5 verify, ordered build, host config, load,
mdev creation). Wrapper-neutral: works with libvirt/raw QEMU.
- patches/nvidia: regenerated path-independent patch (KBUILD_EXTRA_SYMBOLS
resolved via $(src)/$(CURDIR) relative to the NVIDIA tree).
- scripts/stack_manager.sh and helper scripts auto-detect repo root
(OPEN_IOV_ROOT override); no hardcoded developer paths anywhere.
- firmware analysis scripts use script-relative paths (also fixes stale
firmware/570-GSP references); docs/wiki updated to $OPEN_IOV_ROOT.
- .github/workflows/ci.yml updated for standalone repo layout; firmware
fetched from linux-firmware in CI.
- Untrack remaining kbuild artifacts (Module.symvers, modules.order,
*.o.d); gitignore them.
- STATE.md: live project state, limitations, fixed/open bugs.
- README.md: rewritten for clone-and-run reproducibility.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Restructure for clean mirroring to git.vgpu.io/open-iov.git:
- Untrack GSP firmware blobs and decompiled output (firmware/**/*.bin,
firmware/**/decompiled/) - large, license-restricted NVIDIA binaries.
Analysis scripts, docs, and address maps remain tracked.
- Untrack nvidia-open-gpu-kernel-modules submodule; our delta against
upstream tag 610.57.04 is now tracked as a patch series in
patches/nvidia/ and applied by install.sh after cloning.
- Untrack kernel build artifacts (*.o, *.ko, Module.symvers, ...).
- Untrack Ghidra project databases (*.rep/, *.gpr) - scripts remain.
- Untrack VM images (*.iso, *.img, *.qcow2) and compiled helper binaries.
- Drop git-lfs (.gitattributes) - no large objects remain in git.
- Local QEMU-wrapper layers (scripts/vgpu_vm.py,
qemu_control.py, qemu_harness.sh) are gitignored: the repo is
hypervisor-management neutral (libvirt, raw QEMU all work).
All files remain on disk; only git tracking changes. No source code
modified.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create()
- Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions
- Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP
- Added config space read/write handlers at offset 0x70000000000
- Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers)
- Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL)
- Added udev rules for vfio group permissions
- Created stack_manager.sh unified tool
- Fixed mdev remove crash with deferred cleanup in vfio_device_release callback
- Kernel 6.8+ IOMMUFD compatibility
- QEMU launches successfully with vGPU passthrough
|
|
|
- Firmware decompilation: GSP 570.144W (28.5MB ELF, RV64IMAC), 132 decompiled C files
- Kernel modules: nvidia.ko (34MB, OpenRM), open-iov-vfio.ko (2.8MB, VFIO mdev), gsp-vgpu.ko (1.8MB, guest driver)
- Userspace: libiov.so/libvgpu.so (76KB), example_vgpu (28KB), rpc_emulator.c
- PCI config space emulation (256B with PM, MSI, MSI-X, PCIe caps)
- 12 RPC types fully mapped from firmware decompilation
- BAR trap, WPR, scheduling, notification, migration emulation
- Helper scripts: load_vgpu_stack, create_mdev, launch_qemu_vgpu, test_vgpu_stack, setup_guest_driver, unload_vgpu_stack
- Architecture documentation
- Wiki: 10 technical documents
- Git LFS for large binaries (.ko, .bin, .iso, .img, .o, ghidra repos)
- nvidia-open-gpu-kernel-modules as submodule
|
