| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
| |
Sweep of the 155 tracked C/Python/shell files under userspace/ (excluding
the MIT-licensed vendored userspace/extern/), tools/ and scripts/ that had
no per-file license identifier. Each gains the minimal two-line header in
the area's existing form: 'SPDX-License-Identifier: BSD-3-Clause' plus
'Copyright (c) 2026 Open-IOV contributors'; shebang and coding lines stay
first. No other content touched. kernel/** is deliberately out of scope
(recorded as a follow-up in CODESTD-REPORT-2026-10-08.md): its SPDX
coverage and MODULE_LICENSE normalization need their own GPL-2.0 pass.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA
submits it on the compute channel as a compute-class DMA whose source is
the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>,
LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's
SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run
(opcode 3) and no GR work ran at all.
- Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT,
OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run.
- svc_grce_inline_write resolves the destination VA through trusted
sources (pins, map, published page directories, targeted rm_mem
resolve), pins the page and writes into vidmem or guest RAM; budgeted
sema-grce logs name the target or report UNRESOLVED.
- cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection
of the copy path; pushed by push_guest_tools.sh.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
call
|
| |
|
|
| |
validation
|
| | |
|
| |
|
|
| |
callback)
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Run #7 proved the membar-WAR descriptor lookup succeeds in the guest
(remap (7,5)->(7,0)); FUN_00e21850 then returns 1. bpz8 breaks after its
two inner calls (FUN_003e99d0 image create, FUN_003dfae0 image load) to name
which one fails.
|
| |
|
|
|
|
|
|
| |
Host oracle (real TU104, 570.211): FUN_0035a4d0(0x13,7,5) succeeds because
the 3-entry remap table at 0x3dc5b80 maps (7,5)->(7,0); the guest has the
same table. bpz7 breaks before/after the lookup and the FUN_00e21850 ucode
load in FUN_005d7e70 to see which one fails in the guest. Also fix invd.py's
lookup address (0x25a4d0, not 0x35a4d0).
|
| |
|
|
|
|
|
|
|
| |
Run #5 named the failing hook: idx=3 FUN_005d7e70 ('membar_war_bug2022401')
returned 999; its only source is FUN_0035a4d0(0x13, dev+0xc50, dev+0xc54)
returning NULL or the ucode load failing. bpz6 reads the live device state
arch/impl at the failure and evaluates the lookup in-process, scanning
(arch,rev) pairs to show which WAR-ucode descriptors exist; invd.py is the
standalone same-probe for a parked cdt.
|
| |
|
|
|
|
|
| |
Run #4 identified the +0x4580 target as file 0x4d1fa0: it runs four
descriptors 0x440f780/7c0/800/840 whose fns are 0x4d9350, 0x4c8a70,
0x4ad980, 0x4d7e70 and returns the first non-zero status. bpz5 breaks on
the hook call/return and prints the failing index, target and status.
|
| |
|
|
|
|
|
| |
Run #3 named the failing store: file 0x275cec, the indirect call at
0x275ce6 (slot +0x4580 of the table at [ctx+0x40], called with
(ctx, obj+0xb8)) returned 0x3e7. bpz4 prints the runtime target pointer so
it maps to the architecture vtable implementation that failed.
|
| |
|
|
|
|
|
| |
Run #2: all post-pump call sites up to 0x2755bf returned 0, but FUN_00374240
returned r14d=0x3e7 (999) 554 ms later. bpz3 arms every mov %eax,%r14d store
in FUN_00374240 plus the alternate epilogue and the single direct 999 store,
printing the first non-zero status with a backtrace.
|
| |
|
|
|
|
|
|
| |
bpz.py probes the four known rc=999 stores on the cuCtxCreate path plus the
scanner force sites; the live run eliminated all of them and showed the
engine event pump (FUN_0031a3b0) returning success. bpz2.py probes every
post-pump status store in the ctx bring-up (FUN_00374240) and the function's
final return, printing the first non-zero status with a backtrace.
|
| |
|
|
| |
watcher reaping, wider CE stream window
|
| |
|
|
|
|
| |
The failing wait can live in a libcuda worker thread (strace without -f
misses it); sample every /proc/<pid>/task/*/stack + wchan while the client
runs.
|
| |
|
|
|
|
|
|
|
|
| |
The guest-agent exec path is too slow under driver-init load (a 15k-page
window scan took 17-35 s to complete), so GR window pins landed after the
channels' entries had been consumed and after the client's cuCtxCreate
timeout. The guest now scans /proc/<pid>/pagemap locally and streams only
delta runs over vsock (no agent on the hot path); the host listener pins
them as they arrive. The scanner writes a pidfile so restarts do not kill
the new instance.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
injected-pin table
The GR channel GPFIFO entries point at pushes in the client GPU VA window
(observed 0x200400000 + stride, one allocation per channel) and the pump
cannot read them: pushes_nomap grows and the tracking-semaphore releases
they carry never land. The pushes are sysmem pages mapped into the client
process, so /proc/<pid>/pagemap in the guest gives the GPA.
New guest helper guest_pmap_win.py batch-reads the pagemap of a VA range
(512 entries per pread). New host tool pin_gr_window.py rescans the window
on an interval, follows cdt pid changes, and injects VA->PA runs through
sema_map_add (same mechanism as the counters and the CE pushbuffer pins).
The injected-pin table grows 8k -> 32k slots so the CE pushbuffer (4096)
plus the GR window pages fit without drops.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
chunk-read diagnostics
semfix_watch.py now resolves the CE pushbuffer from guest truth instead of
scanning a wide raw window: it walks uvm_channel_t +0=pool, pool+0=manager,
manager+8=pushbuffer, pushbuffer+8=rm_mem, rm_mem+0x10=vas[] and prefers the
rm_mem whose GPU VA is in the 0x120000000 CE pushbuffer range. Three entry
points: uvch probe on __uvm_push_begin_acquire_on_channel_with_info, uvbp
probe on uvm_pushbuffer_begin_push, and the uvmt tracking-sem kptr (channel
scan back from sem). Misses now print structured hop counters so a failed
chunk read is distinguishable from a failed chain. guest_kprobes.sh
(re)installs uvmt/uvch/uvbp/uvpbf after every guest boot (plain, module:
and bare spellings).
|
| |
|
|
| |
injector, wait sampler
|
| |
|
|
|
|
|
|
| |
The libcuda error scanner (FUN_0054fa00) walks a context list and returns
999 when its force flag is set and any context's RM error record has a
non-zero class. ctx999 only shows the first (zeroed) record; scanerr walks
the whole list and prints every non-zero record so the failing control or
alloc is named. Pushed by push_guest_tools.sh.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The counter enumeration/injection workflow lived in the guest's /tmp and was
lost on every restart. Make it durable under tools/guest-diag:
- sem_counters.py: translate a process's per-channel u32 completion counters
(16-byte stride in the /dev/nvidiactl counter page) to guest PAs through
/proc/<pid>/pagemap; --pairs emits handle,va,pa triples.
- sem_slots.py: read/write the counter slots via /proc/<pid>/mem (fallback
for the host counter publish).
- read_va.py: hex dump a process VA (UVM sysmem GPU VAs are usually valid
CPU VAs in the owning process); --u32 for method streams.
- inject_counters.sh: host-side one-shot: push the tool, take the GR handles
from the live chan_dump, enumerate, then register each counter with
sema_counter and pin its VA with sema_map_add.
- push_guest_tools.sh: push the new tools after every VM start.
- launch_qemu_direct.sh: parameterized mdev UUID/paths instead of a stale
hardcoded one; committed so it survives checkouts.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
record
A freed client's notifier events (hSrcResource 0) stayed valid in the event
table, so the fallback wake-up nudge kept posting POST_EVENT records for a
(hClient, hEvent) pair the guest RM no longer knows; every one asserts in
CliGetEventInfo (kernel_gsp.c:466) and is dropped (171 failures in the
15:24 window of the last run, zero after this fix). FREE now retires the
matching event record (and a client's records when its root object is freed).
bp999.py now walks rdi -> ctx -> ctx[0x1291] -> +0x10 -> +0x10 and prints the
RM error notification record, and push_guest_tools.sh ships it to the guest.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
the libcuda tracker wait
The vidmem window convention turns any VA above the window base into a
vidmem offset, but the CUDA client's GPU semaphores are sysmem objects
mapped into its address space at those same VAs. A release applied through
the convention therefore landed in the vidmem shadow instead of the page
the client polls, and the completion was silently lost: the libcuda
tracker it waits on never advanced. Semaphore reads and writes now use
trusted translations only (pins, learned cache, discovered map, page
walks), so an unresolved semaphore is counted as a miss and stays visible
instead of corrupting a shadow.
tools/guest-diag/trkinfo.py dumps the libcuda tracker the client spins on:
the tracker object behind p+0x9410, the semaphore object it references,
the u32 semaphore memory it polls (a /dev/nvidiactl mapping backed by
guest RAM) and the VMA that memory lives in.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Analysis tooling for debugging CUDA execution in Open-IOV guests:
- userspace/cuda/570.144/: fetch a library from a running guest (nc),
objdump helpers, crash ip -> file vaddr + symbol mapping, and Ghidra
headless import/analysis/decompile scripts (project kept so single
functions decompile in seconds). The purpose is debugging correctness
and performance of CUDA in Open-IOV VMs (not a reimplementation);
extracted binaries and decompiled output are gitignored.
- tools/guest-diag/ioctl_trace.c: ptrace tracker for the NVIDIA driver
ioctls of a child (libcuda issues raw syscalls, so LD_PRELOAD sees
nothing). Decodes RM_ALLOC (class) and RM_CONTROL (cmd) arguments;
this is how the cuCtxCreate crash sequence was captured.
- tools/test/cuda_drv_test.c: unbuffered + per-call markers so a crash
identifies the CUDA call it happened in.
|
|
|
Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.
Core principle: emulate the interface, pass through the work.
kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
(PCI config space, BAR shadows, MSI-X), GSP boot emulation,
GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
guest-PA discovery) and the semaphore-completion service that
advances guest UVM progress.
userspace/ libiov.so management API and examples.
tools/ guest diagnostics, firmware analysis wrappers, RM probes.
scripts/ stack manager, VM harness, build/load/test automation.
config/ hardware/version profiles (the portability surface).
docs/ documentation tree - start at docs/README.md.
wiki/ long-form firmware and protocol deep dives.
Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.
License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.
|
