| Commit message (Collapse) | Author | Age |
|---|
| ... | |
| |
|
|
|
|
|
|
| |
Live verification: the trackers waiting on counter slots idx2/idx3 belong
to the channels 2nd/3rd from the top of the ascending handle list, and the
only channel whose GPFIFO put advanced to 2 was the 4th from the top,
whose tracker polled idx3. Zip descending handles with ascending counter
indices by default; ORDER=asc restores creation order.
|
| |
|
|
|
|
|
| |
Restart the VM, push the guest tools, start cdt holding on failure, wait
for the 20 GR channels, register/pin the completion counters and arm
scanerr.py, so each iteration is a single command. scanerr now loops over
scanner hits until a hit carries a non-zero RM error record.
|
| |
|
|
|
|
|
|
| |
The libcuda error scanner (FUN_0054fa00) walks a context list and returns
999 when its force flag is set and any context's RM error record has a
non-zero class. ctx999 only shows the first (zeroed) record; scanerr walks
the whole list and prints every non-zero record so the failing control or
alloc is named. Pushed by push_guest_tools.sh.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The counter enumeration/injection workflow lived in the guest's /tmp and was
lost on every restart. Make it durable under tools/guest-diag:
- sem_counters.py: translate a process's per-channel u32 completion counters
(16-byte stride in the /dev/nvidiactl counter page) to guest PAs through
/proc/<pid>/pagemap; --pairs emits handle,va,pa triples.
- sem_slots.py: read/write the counter slots via /proc/<pid>/mem (fallback
for the host counter publish).
- read_va.py: hex dump a process VA (UVM sysmem GPU VAs are usually valid
CPU VAs in the owning process); --u32 for method streams.
- inject_counters.sh: host-side one-shot: push the tool, take the GR handles
from the live chan_dump, enumerate, then register each counter with
sema_counter and pin its VA with sema_map_add.
- push_guest_tools.sh: push the new tools after every VM start.
- launch_qemu_direct.sh: parameterized mdev UUID/paths instead of a stale
hardcoded one; committed so it survives checkouts.
|
| |
|
|
|
|
|
|
|
| |
The 15 s pinner loses the race against the client's internal deadline in
most boots: in the one boot where UVM's page-table write pushes were
submitted the pin had landed early, and later pins (60 s+) left the client
dead before submitting anything. Poll the guest pagemap every 2 s and pin
0x20400f000 once per cdt pid (re-pinning the same pid also stirred the
pump's rewind repeatedly, which is why the guard exists).
|
| |
|
|
|
|
|
|
|
|
|
| |
ctx999.py arms the scanner entry/returns plus cuCtxCreate and dumps the
ctx->mgr->reg->record chain at the stop. First live capture: the scanner is
called with param_2=1 during normal context creation and correctly returns 0
on a zero record (mgr[+0], the channel handle, and the record walk are now
confirmed live). pin_miss_pages.sh must match the exact VA pmapall printed:
guest_pmap_all.py returns the first nvidia page at-or-above va-min, so a page
that is not present resolves to an unrelated mapping (it pinned a UVM page
for 0x20400f000).
|
| |
|
|
|
|
|
|
|
|
| |
pin_miss_pages.sh watches the host journal for release-miss VAs, resolves
each page for the live cdt pid through the guest pagemap and injects it (the
injection rewinds the pump so releases consumed under a stale pin are
re-read); it re-pins when a new cdt pid appears. cuda_drv_test prints
cuGetErrorName/String on failure and, with CDT_HOLD=<seconds>, stays alive so
the RM error record and tracker state can be inspected after cuCtxCreate has
given up.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
record
A freed client's notifier events (hSrcResource 0) stayed valid in the event
table, so the fallback wake-up nudge kept posting POST_EVENT records for a
(hClient, hEvent) pair the guest RM no longer knows; every one asserts in
CliGetEventInfo (kernel_gsp.c:466) and is dropped (171 failures in the
15:24 window of the last run, zero after this fix). FREE now retires the
matching event record (and a client's records when its root object is freed).
bp999.py now walks rdi -> ctx -> ctx[0x1291] -> +0x10 -> +0x10 and prints the
RM error notification record, and push_guest_tools.sh ships it to the guest.
|
| |
|
|
|
|
|
|
|
|
| |
The guest RM's expiring waits are kernel busy-polls and all of their
functions are notrace, so ftrace/kprobes cannot name them. sysrq-l NMI
backtraces can: this sampler triggers them and keeps only the Comm and
[nvidia] frame lines per round, so the wait/teardown stacks survive dmesg
ring wraps. Caught the adapter-shutdown scrubber stack
(channelReadChannelMemdesc <- ceutilsUpdateProgress <- scrubberDestruct <-
memmgrDestroyInternalChannels <- gpuStatePreUnload <- nv_shutdown_adapter).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The content sweep reads arbitrary guest RAM and ran the full push decoder on
every page, so fabricated method streams in unrelated data executed
fills/copies: one boot scanned 65M pages, 'decoded' 124k pushes and ran
dozens of memsets from false positives - writes into live guest memory (a
guest kernel crashed in nvidia_uvm right after such a boot).
- Swept pages now only permit the bounded semaphore-release writes; fills
and copies execute only for pushes the pump read through a trusted ring
translation.
- Pushes read for a compute (GR) channel apply releases only: their method
space is not the copy engine's, so aliased encodings must never become
memory writes.
- sempages.py derives the pages to pin from the client's tracker objects
(sem object + sem memory) instead of matching bare sem-object shapes, which
pinned 140 pages where the client has ~53 sems; shape scan remains as a
fallback when no tracker resolves.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- The CE semaphore/memset method groups also arrive on subchannel 0 (CUDA's
push library binds the copy class on compute channels to carry the
work-submission releases), so the stream decoder accepts both 0 and 4.
- GR (compute) channel rings live in the same video-memory allocation as the
CE rings, 0x3000 bytes per channel after them: ring (0x2000) at
ce_end + 0x3000*rank, USERD at +0x2000. Discover and pin them once the CE
ranks are known; verified against ring entry 0 (1 MB aligned push VA in
the client window, sane length) and the USERD's put/get, so a layout
change costs a failed pin, not a wrong one. Gated by sema_gr_rings
(default on).
- memprobe: require the sem-VA triple in the sem-object scanner (a bare
qword in the window matches thousands of unrelated pointers once the
ring/pushbuffer windows are part of the range).
Live evidence: ring entries read through the guest's /proc/pid/mem decode to
the GR channels' first pushes; pinning the ring (fb 0x140000+0x3000*rank),
USERD (ring+0x2000) and first-push pages made all 20 GR channels execute
their entry 0 for the first time (get=1 sub=1).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The content sweep now:
- decodes a release whose method stream continues with any INC_METHOD,
not only the CE LAUNCH_DMA group, so the releases at the head of a
compute/GR channel's first push are applied too;
- remembers releases whose target VA had no translation (deferred list)
and retries them every service pass, so a VA->PA injection lands them
immediately instead of after the next full sweep;
- re-arms the fast sliced sweep for a few seconds after an injection;
- treats a channel whose USERD/ring cannot be resolved as stalled, so the
sweep keeps looking for the releases its unreadable pushes carry.
tools: add per-cdt client-pin helper (sem pages + /dev/nvidia0 windows),
sem-page scanner and memprobe fixes (short-read guard, --brief).
Live-verified: injecting a client sem page applied 12 deferred releases
within a second; the service kept applying (89 -> 202) while cdt ran.
|
| |
|
|
| |
replies in the ioctl tracer
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
- Add expected_sem_payload tracking to svc_channel for CE channels
- Implement svc_rescue_push_page() to scan guest RAM for push with
expected semaphore release when VA translation fails
- Add svc_init_expected_payload() to initialize expected payload from pool
- Add rate-limited logging of distinct stalled VAs (svc_log_stalled_va)
- Advance expected_sem_payload on successful push decode
- Pushes SEMA_MAP_MAX to 65536, SVC_PIN_SLOTS to 32768 for scale
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
mappings into the service
The pin table became the primary VA->PA store once push-buffer regions and
bridged client mappings started adding thousands of pages per boot; at
8192 slots with an 8-probe window, inserts were silently dropped and
lookups missed at half load. 32768 slots and a 32-probe window keep the
mapped set intact.
tools/guest-diag/{guest_pmap_all.py,bridge_sems.sh} translate the CUDA
client's /dev/nvidia* mappings through the guest's own pagemap and inject
each present page into the service as a sysmem pin. With the client's
semaphore pages pinned the service applies their releases into the page
the client polls (verified: applied 67 -> 109, misses stop growing for
those VAs), where before every write went to the vidmem shadow.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
the libcuda tracker wait
The vidmem window convention turns any VA above the window base into a
vidmem offset, but the CUDA client's GPU semaphores are sysmem objects
mapped into its address space at those same VAs. A release applied through
the convention therefore landed in the vidmem shadow instead of the page
the client polls, and the completion was silently lost: the libcuda
tracker it waits on never advanced. Semaphore reads and writes now use
trusted translations only (pins, learned cache, discovered map, page
walks), so an unresolved semaphore is counted as a miss and stays visible
instead of corrupting a shadow.
tools/guest-diag/trkinfo.py dumps the libcuda tracker the client spins on:
the tracker object behind p+0x9410, the semaphore object it references,
the u32 semaphore memory it polls (a /dev/nvidiactl mapping backed by
guest RAM) and the VMA that memory lives in.
|
| |
|
|
|
|
|
| |
The probe dumps {type, CPU VA, GPU VAs, size} on every
uvm_rm_mem_get_cpu_va call (and the first 64 bytes of the object's CPU
mapping with nested fetchargs), which is how the host-side learned map
can be checked against what the guest actually created.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- admit {GPU VA -> PA} pairs only when the preceding bytes form a valid
client memory descriptor header (type 0/1, CPU+GPU mapped bits), prefer
validated pairs over headerless candidates under first-wins dedup, and
invalidate the translation cache on every rebuild; fixes stale identity
entries shadowing live pool pages and lets the guest UVM channel
bring-up complete on its own
- capture OS event allocations (class 0x79) and post event records
(function 0x1003) into the guest status queue when a channel's busy
notification is consumed, with a broadcast fallback, serialized with
replies via a queue lock
- add BAR0 shadow introspection (dump_bar0/scan_bar0) and guest-side
diagnostic scripts for the VM cycle
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- GSP_RM_ALLOC: echo the request payload back with status forced to NV_OK.
The kernel driver copies the reply params over the caller's params buffer
using the class-expected size, so a zeroed reply smashed libcuda's stack
(SIGSEGV at the engine-registration continuation); echoing makes the
copy-back idempotent.
- CE_GET_PHYSICAL_CAPS / CE_GET_ALL(_PHYSICAL)_CAPS: report sysmem, P2P and
BL copy caps and mark the GR-synchronous CE (GRCE, COPY0). Without a GRCE
the client's context walk ends with an empty secondary-engine record and
the engine RM alloc is issued with class 0 (rejected -> cuCtxCreate fails).
- TU104 engine table: CE0 shares the GR PBDMA ids (the driver's GR/GRCE
runqueue validation requires it), plus an entry for the async copy engine
(COPY14) advertised in the GSP static info engine caps.
- tools/guest-diag/ioctl_trace: decode the RM status fields of alloc,
control and free so the first failing driver call is visible in traces.
gitignore: keep the CUDA analysis scratch tree (incl. WORKING-HERE.md) local.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Analysis tooling for debugging CUDA execution in Open-IOV guests:
- userspace/cuda/570.144/: fetch a library from a running guest (nc),
objdump helpers, crash ip -> file vaddr + symbol mapping, and Ghidra
headless import/analysis/decompile scripts (project kept so single
functions decompile in seconds). The purpose is debugging correctness
and performance of CUDA in Open-IOV VMs (not a reimplementation);
extracted binaries and decompiled output are gitignored.
- tools/guest-diag/ioctl_trace.c: ptrace tracker for the NVIDIA driver
ioctls of a child (libcuda issues raw syscalls, so LD_PRELOAD sees
nothing). Decodes RM_ALLOC (class) and RM_CONTROL (cmd) arguments;
this is how the cuCtxCreate crash sequence was captured.
- tools/test/cuda_drv_test.c: unbuffered + per-call markers so a crash
identifies the CUDA call it happened in.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The progress service now decodes push buffers as method streams and performs
the work the driver pushes, not just the tracking semaphore:
- C5B5 method decoder: SET_REMAP_CONST_A/B + COMPONENTS + OFFSET_OUT +
LINE_LENGTH_IN + LAUNCH_DMA = memset (into vidmem or guest RAM);
OFFSET_IN + OFFSET_OUT + LAUNCH_DMA = copy; SET_SEMAPHORE_A/B/PAYLOAD +
LAUNCH_DMA = release. Contiguous method runs (count 2/4) are applied in
full - OFFSET_IN/OUT arrive as one run.
- Pushes span page boundaries, so pages are decoded as method streams
without a start marker. Collected operations execute only when the
push's release is reached and still pending (payload ahead of the
semaphore), so recycled buffers replaying old pushes are skipped and
their stale memsets cannot overwrite newer state.
- VA->PA map is first-wins: stale copies of freed allocations and
unrelated adjacent u64 pairs produced bogus entries that hijacked a
good mapping and sent semaphore writes to the wrong page.
- dump_guest module parameter (hex dump of guest PA, kthread via
completion; kthread_stop before the thread runs loses the work).
With this the guest no longer stalls in uvm_tracker_wait and no longer
oopses in the UVM path; the failure moved to a userspace segfault inside
libcuda (NULL-0x38) after the driver's page allocations.
Also add tools/cuda-ab: run one deterministic CUDA workload on the host
(bare metal oracle) and in the guest, then compare the stable result
lines. The binary links the CUDA runtime statically so the guest only
needs the driver's libcuda.so.
|
|
|
Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.
Core principle: emulate the interface, pass through the work.
kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
(PCI config space, BAR shadows, MSI-X), GSP boot emulation,
GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
guest-PA discovery) and the semaphore-completion service that
advances guest UVM progress.
userspace/ libiov.so management API and examples.
tools/ guest diagnostics, firmware analysis wrappers, RM probes.
scripts/ stack manager, VM harness, build/load/test automation.
config/ hardware/version profiles (the portability surface).
docs/ documentation tree - start at docs/README.md.
wiki/ long-form firmware and protocol deep dives.
Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.
License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.
|
