| Commit message (Collapse) | Author | Age |
| |
|
|
| |
scanned map (push-data memcopies were rc=-ENOENT; UVM inline writes now execute)
|
| |
|
|
| |
TPC count)
|
| |
|
|
| |
(all 20 channel releases now land), queue payloads that arrive before the slot pin
|
| |
|
|
| |
watcher reaping, wider CE stream window
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The client's UVM_REGISTER_GPU blocks in uvm_push_end_and_wait on the first
CE channel's tracking semaphore (pool word 0x121000000) waiting for value 3.
The pool payload stayed at 1 while the client pushed: the pump processed
entries whose push bytes were unreadable or had no decodable release, so the
release bookkeeping advanced (expected_sem_payload) but nothing was written
to the pool page.
Treat the GPU semantics as normative: each consumed GPFIFO entry on an engine
channel releases the channel's tracking semaphore with the incrementing
queued value. When the decoded stream carries no release and sema_fake_rel is
enabled, deliver the expected payload through svc_fake_release() instead of
only bumping the counter. Also initialize expected_sem_payload for a fresh
pool page whose current payload is 0 (expected = 1) so the first push is
deliverable from the start.
|
| |
|
|
|
|
|
| |
The injection queue was only drained at the top of the service loop, which
can be inside a multi-second pump/discovery pass; live runs showed the GR
counter pins landing ~1 s after the client mapped the page, i.e. after its
context-init deadline. Waking on enqueue drains the batch immediately.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
through the exact slot
Live evidence: with the counter page pinned, counter_pa still resolved to a
stale self-discovered page, so every release write went to memory the client
never polls (host read back its own 1s; client's page stayed 0) and the
client's context-init wait expired. The generic release path could also write
through that stale translation.
- svc_counter_autoreg / svc_counter_learn trust only the injected-pin table
(svc_ipin_lookup); learn re-registers when the injected PA differs.
- Releases whose target is in the GR counter VA range skip the generic
translation write entirely and go through the injected-pin-backed slot.
|
| |
|
|
|
|
|
|
|
|
|
| |
counter-page delta
- rel-apply/publish now log the first 32/48 outcomes (written, skipped with
current value, no slot, read failure) so the live counter page can be
matched against host writes.
- guest pin scanner logs the counter page's pagemap state each loop and adds
it to the delta set explicitly, so the page is pinned the moment it is
mapped even if the window scan misses it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The fake-release fallback acknowledged unreadable CE pushes even when the
tracking-semaphore pool page had no translation yet, so the GPFIFO entry was
consumed without writing the semaphore the client waits on (observed: client
stuck in UVM_GPU_REGISTER while CE channel 2's release never landed, its
push-buffer pin arriving ~2 s later).
- svc_fake_release now reports failure when the pool page cannot be
translated/read/written instead of pretending success.
- CE channels with the fallback enabled leave the entry pending (no
multi-second RAM rescue) until the pool pin lands; the skip path only
consumes an entry when the release was actually delivered.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
queue 4k
- counter_pa autoreg result is written back to the channel table so later
passes cannot re-target a different slot when the live GR count changes
during teardown; re-registration of the same channel preserves the slot.
- counter slot layout uses the max GR count ever seen in the generation
instead of the shrinking live count.
- sema_map_add queue 64 -> 4096 entries so a full-window pin burst drains
without ENOSPC retries.
- pin_gr_stream: one batched sudo pass per flush (was one process per run,
~4 s for 400 runs), retrying only queue-rejected runs.
- semfix_uvmt: SEMFIX_SKIP_PB=1 skips the blocking push-buffer chain walk
when the vsock window stream already supplies those pages.
|
| | |
|
| | |
|
| |
|
|
| |
puts)
|
| |
|
|
| |
memsets)
|
| | |
|
| |
|
|
| |
on doorbell + tight pump loop while entries execute
|
| |
|
|
| |
last release VA; larger push log budget
|
| |
|
|
| |
corruption); tooling guards
|
| |
|
|
| |
semfix+uvmt watcher
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pin output
The first GR window pin landed ~2 s after the pump consumed the freshly
created channels' GPFIFO entries on the first failed read, so the pushes
(and the tracking-semaphore releases they carry) were lost. GR entries with
an unresolvable push page now stay pending for sema_gr_grace_ms (default
8 s) before the stale-entry consume; a pin landing in that window makes the
entry readable and it executes normally. The completion counter is still
published from put at the end of every pass, so the client's counter wait
is not delayed.
guest_pmap_win.py coalesces contiguous VA+PA pages into run lines, so a
15k-page window transfers as ~200 lines instead of ~15k and the pin batch
lands within a second of the pages appearing.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
injected-pin table
The GR channel GPFIFO entries point at pushes in the client GPU VA window
(observed 0x200400000 + stride, one allocation per channel) and the pump
cannot read them: pushes_nomap grows and the tracking-semaphore releases
they carry never land. The pushes are sysmem pages mapped into the client
process, so /proc/<pid>/pagemap in the guest gives the GPA.
New guest helper guest_pmap_win.py batch-reads the pagemap of a VA range
(512 entries per pread). New host tool pin_gr_window.py rescans the window
on an interval, follows cdt pid changes, and injects VA->PA runs through
sema_map_add (same mechanism as the counters and the CE pushbuffer pins).
The injected-pin table grows 8k -> 32k slots so the CE pushbuffer (4096)
plus the GR window pages fit without drops.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two fixes for the CE data path. (1) svc_handle_map_add rewound every
channel by 8 ring entries on every injected mapping; a 16 MB push-buffer pin
is thousands of page injections, so the pump's last_get was pinned in place
and it re-read the same entries for the whole batch (3645 submissions, zero
ops). The rewind now runs once per drained batch. (2) The push-start
heuristic (op=INC_METHOD at address 0) rejects UVM's real CE memop pushes,
which start with an IMM method at address 8 (observed 0x60020002, PTE data
words) or a release method at 0x700 (0x200381c0); a buffer read through a
host-injected (ipin) translation is now decoded fully trusted, ops included.
Result on the next boot: memset=2718/0/0 ringops=1794 (first sustained CE
work), false 12k->1.9k. semfix_watch: bulk-pin contiguous runs via
sema_map_add len, read rm_mem size, 16 MB (4096 page) cap, non-blocking QMP
read (the fixed 30 ms sleep made 4096 gva2gpa calls take two minutes), and
KEEP_TRACE=1 to preserve the guest trace for post-mortems.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
them
Real CE release pushes start with INC_METHOD addr 0x700 (0x200381c0, count
3, subch 4), and svc_method_stream_plausible requires method address 0, so
every readable release push was treated as a false pairing, rescued (fails)
and skipped - the client's tracking-semaphore release never applied and
cuCtxCreate waited forever (slot payload 74 vs queued 75). A buffer read
through a host-injected (ipin) translation is now decoded release-only
(allow_ops=false) when it fails the push-start heuristic; self-discovered
translations keep the strict rescue path.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The CE push-buffer page 0x120000000 resolved to a zero-filled page even after
sema_map_add injected the correct translation: the shared pin table silently
dropped inserts once a hash cluster exceeded the 32-probe window (the page
hashes to slot 0, a hot cluster), and lookups fell through to the
content-scanned map and answered with a stale page - so every CE push read as
implausible and memset stayed 0. Injected translations now live in a separate
authoritative table checked before every other source and are never written by
self-discovery; the general pin table is 4x larger with a 4x probe window and
counts drops (pin_drop/ipin_drop) in the stats line. guest_kprobes.sh disables
the kprobe group before removal (EBUSY) and matches p:kprobes/<name>; the
guest wait sampler can run counters-only with GDB=0.
|
| |
|
|
| |
scans)
|
| |
|
|
| |
fallback
|
| | |
|
| |
|
|
| |
injector, wait sampler
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- svc_handle_chan_dump()/svc_decode_stream(): static snapshot buffers. The
128-channel local array made an ~18 KB frame, which overflows the 16 KB
kernel stack in the function prologue and panicked the host 2-3 s after a
VM attached the vGPU (proven from EFI pstore: guard page hit at
svc_handle_chan_dump+0x20, sub rsp,0x4850).
- svc_pump_channels(): write-back now also persists expected_sem_payload,
rescue_last and rescue_fail. The pump works on a snapshot copy, so
without this the stale-entry skip and the rescue rate limit could never
fire and the payload expectation was reinitialized every pass.
- svc_targeted_resolve(): sliced scan (128 MB per call, one VA at a time,
failed-VA cache) instead of a full-RAM scan per miss; no longer appends
to the sorted map out of order.
- svc_rebuild(): merges new scan results into the previous map (fresh
translation wins in place, new VAs appended) instead of replacing it
wholesale, so live translations survive a scan-yield collapse.
- Warning fixes: misleading indentation, unused variable, jiffies
typecheck. Frame audit: largest frame now 872 B.
- Includes the previous session's uncommitted work: sema_counter /
shadow_scan / shadow_dump debug params, exit stage markers, pdev ref
fix, and the dynamic dispatcher request_firmware-out-of-spinlock fix.
|
| |
|
|
|
|
|
|
| |
Entries whose push buffer was overwritten (the rescue cannot validate any
candidate page) were retried forever, rescanning RAM every 5 s and
leaving the ring stuck behind them. After three real rescue failures
(rate-limited attempts do not count) the entry is consumed with a
'skipping stale entry' log and the pump moves on. New skip= stat.
|
| |
|
|
|
|
|
|
| |
Mid-slot entry VAs were rejected by the alignment guard, leaving their
false translations permanent. Candidate 2 (the group's page minus the
push length) is the PA of the entry VA's page regardless of alignment;
pin only the pages the push occupies for mid-slot entries and the whole
1 MB slot for slot-aligned ones. Validation by read-back is unchanged.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Kernel: the slot-aligned failing entries carry pushes whose release group
is not at the page's fixed +0x10 offset, so the shape matcher found
nothing. Scan every 4-byte offset for the exact group (hdr 0x20038090,
hi==1, lo&0xfff==slot, follower LAUNCH_DMA sem-release; payload may lag
the tracked expectation) with the header compared first. Each match
yields two candidate pin bases - the group's page and the page implied by
treating the group as the end of this entry's push - and each candidate
is validated by pinning and reading the entry VA back before it is
accepted. Live: 13 rescues pinned the MEMOPS slots; the 12700-byte push
then decoded to 29 op-runs with a real release, and memset=338 fills
(page-table clears) executed ~13 s before the client's rc=999.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The content rescue matched only the release group bytes and had matched
GSP firmware pages. It now requires the full known push shape around the
release (two SET_OBJECT class binds, the 3-word SET_SEMAPHORE_A run with
this channel's next payload, and the LAUNCH_DMA semaphore-release
follower), and it is also invoked when a trusted read resolves but the
buffer does not parse as a push (false content-scanned pair; observed
firmware/kernel-data pages), not only when the VA is unmapped. The rescue
pins the whole 1 MB push-buffer slot, which is what the UVM MEMOPS
channels' pushbuffer pool (0x1200_0000 + n*1MB) needs. New false= stat.
|
| |
|
|
|
|
| |
The long bogus pushes (RISC-V tails) could not be tied back to the ring
entry they came from; log the GPFIFO entry VA with each push so the
false translations are identifiable and can be compared against probes.
|
| |
|
|
|
|
|
|
|
|
| |
The discovered VA->PA map is content-scanned and can hand a ring entry's
push VA a false translation; observed: GSP firmware pages read as a
12700-byte method stream whose fills would have been executed into
vidmem. Work now requires the buffer to start with a plausible first
method (INC_METHOD, address 0, sane count that fits the length); releases
still apply from such buffers so a shifted stream is not lost. New
'denied' counter in the sweep stats.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Log one line per ring-decoded push (channel, engine, length, op-run
count, decoded release VA/payload, last 24 bytes) to make the
push->tracker path observable; it showed that the content rescue had
been pinning unrelated RAM pages: pushes of length 4196/8500/12700 whose
tails are RISC-V code (ecall/jal) - GSP firmware or other code pages whose
bytes matched the expected release pattern. Those pages decode as
plausible but bogus method streams, so work executed from them can write
garbage into vidmem. Rescued pages now apply the release only
(allow_ops=false); work requires a page read through a trusted
translation.
|
| |
|
|
|
|
|
|
|
|
|
| |
NVA06C_CTRL_CMD_INTERNAL_PROMOTE_FAULT_METHOD_BUFFERS is sent at
channel-group allocation; it hands the GSP each runqueue's 12 KB CE fault
method buffer as {base (GPU physical), size, alignment, addressSpace,
cpuCacheAttrib} plus a bar2Addr. Parse and log them, and keep the last
set per channel-group object so the addresses (guest structures we cannot
otherwise enumerate) are available to the engine emulation. Live: two
12 KB buffers per group, base = guest PA, addressSpace = 1, buffers
zeroed (no faults posted yet).
|
| |
|
|
|
|
|
|
|
|
| |
New module parameter walk_probe <va> runs the recorded-va-space walks and
the discovered-directory walk directly (pin/lat/pair map bypassed) with
per-level tracing, and prints the result of each. This separates 'the
page tables are not walkable' from 'the VA->PA pair map already answers'
when a translation is missing. Baseline on a live boot: root PDB entries
are zero (L0 e=0 idx=0) so the writes UVM issues into the level pages
have no parent entry yet.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The CE pushes UVM's MEMOPS channels submit release through SET_SEMAPHORE_A
plus a LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD. That path still
gated the push's collected fills/copies on the semaphore being readable
and behind the payload - the same bootstrap deadlock as the REPORT_SEM
path fixed before: the page-table writes that make the semaphores
translatable were dropped. Run the work unconditionally there as well
(trusted-ring pushes only).
Also log a one-line LAUNCH_DMA inventory (channel, engine, dma word,
modes, offsets, line length, collected op count) for the first pushes
read through rings, which is how the UVM page-table write sequence
(fill 512x8 zero + 16-byte PTE copies + release) was identified.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The pump gated collected fills/copies on the tracking semaphore being
translatable and behind the release payload. On real hardware the work
always executes and the release only orders observers, so the gate
dropped work exactly when it was needed to bootstrap translations
(UVM's CE memsets writing page tables into vidmem, whose tracking
semaphores live in the address space those writes create) and skipped
pushes as stale as soon as a tracking semaphore advanced past them.
Execute the work at the release method for every push read through a
trusted ring; swept-RAM decoding still never runs work. Count op runs
in the sweep stats (ringops).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
record
A freed client's notifier events (hSrcResource 0) stayed valid in the event
table, so the fallback wake-up nudge kept posting POST_EVENT records for a
(hClient, hEvent) pair the guest RM no longer knows; every one asserts in
CliGetEventInfo (kernel_gsp.c:466) and is dropped (171 failures in the
15:24 window of the last run, zero after this fix). FREE now retires the
matching event record (and a client's records when its root object is freed).
bp999.py now walks rdi -> ctx -> ctx[0x1291] -> +0x10 -> +0x10 and prints the
RM error notification record, and push_guest_tools.sh ships it to the guest.
|
| |
|
|
|
|
|
| |
The rewind used entries - (N - get) for get < N, which for an empty ring
(get=0, put=0) jumped last_get to entries-8, so the pump read ring slots the
producer had never reached and could execute stale/garbage entries. Replay
only recently consumed slots, clamped at 0.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A pin injected after the pump consumed pushes that release through that VA
resolved against the old (or bogus) translation, so the releases were
compared as stale against the old page's values and dropped - a new client
that maps fresh pages at the same VAs then waits forever. Rewind every
channel's last_get by 8 ring entries on injection so the pump re-reads those
pushes under the new translation; the release and fill application paths are
forward-progress guarded and idempotent, so re-execution is safe. The handler
runs in the service thread, which also runs the pump, so no pass can be
in flight during the rewind.
Diagnosed from cdt4: its tracker releases were consumed while the pin still
pointed at the previous cdt's page (value 1), so payload <= cur and every
release was dropped as stale; re-pinning afterwards could not resurrect them
because the ring entries had already been consumed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The content sweep reads arbitrary guest RAM and ran the full push decoder on
every page, so fabricated method streams in unrelated data executed
fills/copies: one boot scanned 65M pages, 'decoded' 124k pushes and ran
dozens of memsets from false positives - writes into live guest memory (a
guest kernel crashed in nvidia_uvm right after such a boot).
- Swept pages now only permit the bounded semaphore-release writes; fills
and copies execute only for pushes the pump read through a trusted ring
translation.
- Pushes read for a compute (GR) channel apply releases only: their method
space is not the copy engine's, so aliased encodings must never become
memory writes.
- sempages.py derives the pages to pin from the client's tracker objects
(sem object + sem memory) instead of matching bare sem-object shapes, which
pinned 140 pages where the client has ~53 sems; shape scan remains as a
fallback when no tracker resolves.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two failure modes kept every GR channel dark despite the ring layout being
known:
- The CE region size was latched on the first autopin attempt, which can run
before all CE channels are registered; a too-small base then points every
GR candidate at a chunk whose entry 0 is zeros, and it stays wrong for the
life of the boot. Recompute the base from the live CE ranks on every pass.
- An existing pin for a ring VA (e.g. from a convention guess) made the
autopin mark the rank done without validating the ring - a wrong USERD is
then never corrected. Validate the candidate chunk (entry 0 decode + USERD
put/get) first and only fall back to trusting an existing pin when the
chunk does not validate; on success overwrite the pin.
Also log one diagnostic line per GR rank when the chunk does not validate
yet, so a silent autopin failure is visible in dmesg.
Live: fresh boot, no manual ring pins - 20/20 GR channels auto-pinned
(correct USERD 0x142000 + 0x3000*i) and ran entry 0 (get=1 sub=1) once the
push pages were bridged.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- The CE semaphore/memset method groups also arrive on subchannel 0 (CUDA's
push library binds the copy class on compute channels to carry the
work-submission releases), so the stream decoder accepts both 0 and 4.
- GR (compute) channel rings live in the same video-memory allocation as the
CE rings, 0x3000 bytes per channel after them: ring (0x2000) at
ce_end + 0x3000*rank, USERD at +0x2000. Discover and pin them once the CE
ranks are known; verified against ring entry 0 (1 MB aligned push VA in
the client window, sane length) and the USERD's put/get, so a layout
change costs a failed pin, not a wrong one. Gated by sema_gr_rings
(default on).
- memprobe: require the sem-VA triple in the sem-object scanner (a bare
qword in the window matches thousands of unrelated pointers once the
ring/pushbuffer windows are part of the range).
Live evidence: ring entries read through the guest's /proc/pid/mem decode to
the GR channels' first pushes; pinning the ring (fb 0x140000+0x3000*rank),
USERD (ring+0x2000) and first-push pages made all 20 GR channels execute
their entry 0 for the first time (get=1 sub=1).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The content sweep now:
- decodes a release whose method stream continues with any INC_METHOD,
not only the CE LAUNCH_DMA group, so the releases at the head of a
compute/GR channel's first push are applied too;
- remembers releases whose target VA had no translation (deferred list)
and retries them every service pass, so a VA->PA injection lands them
immediately instead of after the next full sweep;
- re-arms the fast sliced sweep for a few seconds after an injection;
- treats a channel whose USERD/ring cannot be resolved as stalled, so the
sweep keeps looking for the releases its unreadable pushes carry.
tools: add per-cdt client-pin helper (sem pages + /dev/nvidia0 windows),
sem-page scanner and memprobe fixes (short-read guard, --brief).
Live-verified: injecting a client sem page applied 12 deferred releases
within a second; the service kept applying (89 -> 202) while cdt ran.
|
| |
|
|
| |
shares GR's runlist/PBDMA
|
| | |
|
| | |
|