| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
helper gates on
The static CUDA runtime's device init failed with cudaErrorInitializationError
even though every kernel ioctl returned 0: cuInit succeeded and then
cuDeviceGetAttribute(ECC_ENABLED=36) returned CUDA_ERROR_NOT_FOUND (500).
Tracing libcuda's clock-domain helper (libcuda+0xe8c410) shows it issues the
CLK legacy controls 0x20809009/0x20809064/0x20809001 and bails with 500 when
the replies do not carry the expected domain bits/entries; the dependent
attribute getters (ECC_ENABLED, TOTAL_CONSTANT_MEMORY) then abort init.
Reply to the three queries with the minimal fields the helper checks: bits
32|34 present in 0x20809009, non-zero domain entries at offsets 12/20/28/36
of 0x20809064, and a matching selector in 0x20809001. No frequencies are
fabricated (all other fields stay zero).
Verified in the guest with the stock 570.144 static-cudart vectorAdd client:
cudaGetDeviceCount -> rc=0 (ndev=1), cudaSetDevice, cudaFree, three
cudaMallocs and both HtoD copies all return rc=0; the client then parks in
cudaDeviceSynchronize waiting for the (not yet forwarded) kernel launch.
|
| |
|
|
|
|
|
|
|
|
| |
The control is cacheable in the guest kernel (RMCTRL_FLAGS_CACHEABLE)
and rmapiControlCacheSet validates paramsSize against the canonical
sizeof(NV2080_CTRL_BUS_GET_PCI_BAR_INFO_PARAMS) = 200. We replied only
8 + 4*24 = 104 bytes, so every reply tripped
NV_ERR_INVALID_PARAMETER in rmapiControlCacheSet (rpc.c:9998) and the
BAR-info cache entry was never populated. Zero-fill and report all
eight NV2080_CTRL_BUS_PCI_BAR_INFO entries (count still 4).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A healthy run left dmesg dominated by per-operation lines (sema-push,
sema-lda, sema-chan rel-apply, sema-grce, DOORBELL/BAR writes, GSPRM
per-RPC lines, the teardown garbage-element stream). Introduce the
module parameter sema_dbg (default 0) and route every high-frequency
diagnostic through it; the finer-grained watch knobs (sema_ops_log,
sema_copy_watch, sema_copy_pat, sema_copy_dst, sema_copy_dst2) only
take effect while it is on. Boot stats, channel registrations, pin
events and state changes stay unconditional.
Also carries the per-side translation provenance (op_tr_src, printed
as prov= in the destination-window watch) and the 16 KB destination
window used to observe the guest's page-table writes.
Verified: fresh boot + full cdt2 smoke test (64/64, ALL OK) leaves a
520-line dmesg with zero per-push/per-doorbell/per-RPC lines.
|
| |
|
|
| |
TPC count)
|
| |
|
|
| |
watcher reaping, wider CE stream window
|
| |
|
|
|
|
|
|
|
|
|
| |
NVA06C_CTRL_CMD_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS is sent at
channel-group allocation; it hands the GSP each runqueue's 12 KB CE fault
method buffer as {base (GPU physical), size, alignment, addressSpace,
cpuCacheAttrib} plus a bar2Addr. Parse and log them, and keep the last
set per channel-group object so the addresses (guest structures we cannot
otherwise enumerate) are available to the engine emulation. Live: two
12 KB buffers per group, base = guest PA, addressSpace = 1, buffers
zeroed (no faults posted yet).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
record
A freed client's notifier events (hSrcResource 0) stayed valid in the event
table, so the fallback wake-up nudge kept posting POST_EVENT records for a
(hClient, hEvent) pair the guest RM no longer knows; every one asserts in
CliGetEventInfo (kernel_gsp.c:466) and is dropped (171 failures in the
15:24 window of the last run, zero after this fix). FREE now retires the
matching event record (and a client's records when its root object is freed).
bp999.py now walks rdi -> ctx -> ctx[0x1291] -> +0x10 -> +0x10 and prints the
RM error notification record, and push_guest_tools.sh ships it to the guest.
|
| |
|
|
| |
shares GR's runlist/PBDMA
|
| | |
|
| |
|
|
|
|
| |
Guest driver was receiving SUCCESS replies as 'unexpected events' (0x48, 0x49).
These async init RPCs are fire-and-forget per GSP-RM protocol; driver doesn't
expect a reply on the status queue. CONTINUATION_RECORD still sends SUCCESS.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Added TU104_CTRL_INTERNAL_MEMSYS_L2_INVALIDATE_EVICT (0x20800a6c)
- Added TU104_CTRL_INTERNAL_GR_GET_FECS_TRACE_HW_ENABLE (0x20800a38)
- Added TU104_CTRL_SYSTEM_MAP_SHARED_DATA (0x730151)
- Fixed async RPCs (72 GSP_SET_SYSTEM_INFO, 73 SET_REGISTRY, 71 CONTINUATION_RECORD)
to send SUCCESS replies instead of silent consume
- Added explicit handlers for RPC 47 (UNLOADING_GUEST_DRIVER) and 70 (UPDATE_BAR_PDE)
with proper reply semantics
Guest driver was failing at GSP-RM level with unknown RPCs 47, 70, 72, 73.
Channels registered then immediately unregistered.
Now they should stay registered and complete initialization.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
- re-post the completion event for a bounded window after a channel
completes work (notifier consumed or semaphore release applied), so a
client that starts waiting after the edge-triggered wake-up still gets
a wake-up; the OS event classes carry no source object, so the hint
broadcasts to the registered events
- add rmstate (dump msgq cursors, guest consumption position, event
table) and rmpost (post an event record by index or to all) debug
parameters, both executing in kthreads because the IOMMU read path
only works with a NULL current->mm or the mapping owner's mm
- report the underlying error codes from the event post path
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- admit {GPU VA -> PA} pairs only when the preceding bytes form a valid
client memory descriptor header (type 0/1, CPU+GPU mapped bits), prefer
validated pairs over headerless candidates under first-wins dedup, and
invalidate the translation cache on every rebuild; fixes stale identity
entries shadowing live pool pages and lets the guest UVM channel
bring-up complete on its own
- capture OS event allocations (class 0x79) and post event records
(function 0x1003) into the guest status queue when a channel's busy
notification is consumed, with a broadcast fallback, serialized with
replies via a queue lock
- add BAR0 shadow introspection (dump_bar0/scan_bar0) and guest-side
diagnostic scripts for the VM cycle
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The guest's CUDA stack was stuck inside context creation because the work it
submits is never executed by the emulated GPU. Two forms of pushed work were
missing, and the completion path had grown far too slow to be useful:
- Channel tracking: on a channel allocation parse the GPFIFO GPU VA and entry
count, the USERD address/aperture and the instance block, hand them to the
progress service and drop them again when the channel is freed.
- Host-class (C46F) semaphore releases: pushes on a compute channel signal
progress with a five-dword method run (SEM_ADDR_LO/HI, PAYLOAD_LO/HI,
EXECUTE) rather than the copy-engine group the service already understood,
so those releases were never applied. Decode both forms.
- Emulated vidmem: channel blocks (USERD/instance) and some client semaphores
live in the emulated frame buffer, whose VAs are a window over the BAR1
backing rather than guest RAM; access them through the right aperture.
- Latency: the service's full-RAM sweep took seconds per pass, so guests'
progress waits took whole seconds and the driver tore its own setup back
down in the meantime. Sweep guest RAM in bounded, chunked windows on every
pass, remember the pages pushes are actually found in and revisit those
first, and keep the VA->PA map complete (dedup, sort, binary search) so
semaphores resolve reliably.
Also add a vidmem poke parameter for bringing up and debugging emulated
frame-buffer state.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- GSP_RM_ALLOC: echo the request payload back with status forced to NV_OK.
The kernel driver copies the reply params over the caller's params buffer
using the class-expected size, so a zeroed reply smashed libcuda's stack
(SIGSEGV at the engine-registration continuation); echoing makes the
copy-back idempotent.
- CE_GET_PHYSICAL_CAPS / CE_GET_ALL(_PHYSICAL)_CAPS: report sysmem, P2P and
BL copy caps and mark the GR-synchronous CE (GRCE, COPY0). Without a GRCE
the client's context walk ends with an empty secondary-engine record and
the engine RM alloc is issued with class 0 (rejected -> cuCtxCreate fails).
- TU104 engine table: CE0 shares the GR PBDMA ids (the driver's GR/GRCE
runqueue validation requires it), plus an entry for the async copy engine
(COPY14) advertised in the GSP static info engine caps.
- tools/guest-diag/ioctl_trace: decode the RM status fields of alloc,
control and free so the first failing driver call is visible in traces.
gitignore: keep the CUDA analysis scratch tree (incl. WORKING-HERE.md) local.
|
|
|
Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.
Core principle: emulate the interface, pass through the work.
kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
(PCI config space, BAR shadows, MSI-X), GSP boot emulation,
GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
guest-PA discovery) and the semaphore-completion service that
advances guest UVM progress.
userspace/ libiov.so management API and examples.
tools/ guest diagnostics, firmware analysis wrappers, RM probes.
scripts/ stack manager, VM harness, build/load/test automation.
config/ hardware/version profiles (the portability surface).
docs/ documentation tree - start at docs/README.md.
wiki/ long-form firmware and protocol deep dives.
Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.
License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.
|
