summaryrefslogtreecommitdiffstats
path: root/kernel/gsp_sema_service.c
Commit message (Collapse)AuthorAge
...
* sema-svc: rewind replays may only be skipped when their release was deliveredArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | The per-put-epoch consumed-slot skip must not suppress a slot whose entry was consumed *without* delivering its release (the CE grace expiry in real mode): the driver-API smoke test's CE pushes are consumed unreadable before their ring pin lands and depend on the pin-batch rewind replaying them once the translation appears. The skip now applies only to slots whose consumption delivered a release: - tail consumption records a slot when the iteration decoded a release (svc_rel_count delta); the sweep-delivered entry_want consume and the fake-release consume always record - the grace-expiry consume records only when a recovery (fake or sema_rel_recover) delivered the release - svc_fake_release counts as a delivered release So real-mode replays still re-process undelivered entries, while the runtime-client case (readable pushes delivered immediately) keeps the fast path that removed the 8 s stalls.
* sema-svc: detect a decoded release by count, not by the (unchanging) VAArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | The pump decided "this push carried a release" by comparing the last decoded release VA before/after svc_decode_stream. Consecutive pushes on a channel release to the SAME tracking-semaphore VA (UVM's tracker pattern), so every push after the first read as release-less: the entry was held for the full CE grace window (observed: exactly one push consumed per 8 s, gating the whole runtime init; the release itself had already been applied by the decode, only the pump's bookkeeping stalled). svc_rel_count now increments on every decoded release (host sem block, REPORT_SEM block, CE launch-DMA release) and the pump compares counts at all three sites (initial decode, rescue re-route, rescued-page retry).
* sema-svc: never re-process a GPFIFO slot already consumed in the put epochArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | A pin-batch rewind replays recently consumed entries. An old entry whose rescue cannot validate any candidate then sits in the CE grace path - and because the client's put has advanced since, the replayed entry no longer matches any "new" expectation: it waits the full 8 s grace, blocking every later entry behind it on that channel. Observed live: the first push of rank 0 re-processed and skipped twice, 38 s apart, while eight later pushes queued behind it. The pump now marks each consumed slot in a per-channel bitmap valid for the current put epoch (cleared when put wraps, so a recycled slot in the next epoch is processed normally) and consumes a replayed slot immediately. Rewinds still refresh readable entries; they simply no longer resurrect entries that were already handled. Evidence: real-release watcher cycle, replay-after-put-advance no longer produces 'skipping stale entry ... after 8000 ms grace' for old slots.
* sema-svc: consume rewind replays and sweep-delivered CE entries at onceArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | Two stalls serialized the runtime client's init into one 8 s CE grace window per push: - a pin-batch rewind replays recently consumed GPFIFO slots; re-examining a consumed slot runs the rescue and then holds it for the full grace window again (observed: the same VA 'skipped after 8000 ms grace' twice, 38 s apart). The pump now records each consumed slot with the put value it was consumed at (32-entry per-channel replay ring, persisted across passes) and consumes a replayed slot whose producing put has not advanced immediately. - the content sweep applies releases from RAM pages before the pump reads their entries, so the pool advances past the entry's payload while it is pending. entry_want (recorded on first examination) now detects that on both the unreadable-rescue and the readable-no-release paths and consumes the entry instead of waiting the grace window. Evidence: real-release watcher cycle, pool semaphore advances no longer gated at 8 s intervals once the replay skip engages.
* sema-svc: validate pushes with the full method encoding; deliver releases ↵Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the sweep already applied The runtime client's second push on each UVM copy channel starts with a NON_INC method (0x60080002, count 8) and its release method sits 0x50 bytes in. The stream validator accepted only INC (op 1) headers, so the readable page failed the plausibility gate and every content-rescue candidate for that VA failed the read-back validation - the client's tracking semaphore for the push never advanced and the runtime init parked. - one shared parser (svc_stream_parse) accepts INC/NON_INC (op 1/3), IMMD_DATA (op 4, header-only), ONE_INC (op 5) and END_PB_SEGMENT (op 7), and still requires an exact method boundary over the whole entry plus trailing zero padding - the strict end condition is what keeps the content rescue safe against firmware/allocator pages - svc_method_stream_plausible and svc_stream_full_ok both use it, so a real readable push is decoded directly and a rescued page validates - the decoder skips NON_INC/ONE_INC methods whole (including their data words) instead of resyncing word-by-word through the payload, which could misread data as headers and queue bogus work Follow-on: the content sweep applies releases from RAM pages ahead of the pump, so the pump's expected payload was already satisfied when it examined the entry; the CE grace path then held each entry for the full 8 s window before consuming it - one grace period per push, serializing the client's runtime init. The pump now records the expected payload when it first examines an entry (entry_want) and consumes the entry immediately once the pool advances past it, on both the unreadable-rescue and the readable-no-release paths. Evidence (fresh boot, real releases, watchers + kprobes, no fakes): cudaGetDeviceCount rc=0 in the runtime client, device-attribute walk proceeds, pool semaphores advance; before this change the client sat in the 0x20801702 service-interrupt loop for >6 min and never left device init.
* sema-svc: deliver UVM CE tracking releases without host-side pinsArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The runtime client's registration stalled when no guest-side pin scanner was running: the content rescue refused to run before the pool semaphore was translated (its expected payload was unlearned), and unreadable CE entries were consumed after two failed rescues ~40 ms apart, dropping the client's wait. The push pages then could not be found within the entry's grace window. - rescue: drop the learned-payload gate (the matcher only needs the channel rank and the release shape); keep unreadable CE entries pending for sema_ce_grace_ms instead of consuming after a fixed attempt count - expected payload: re-learn from the pool whenever it advanced (a deferred release could land after the first read, leaving the expectation stale) - matcher: require the expected payload exactly and validate the read-back stream carries it, so a recycled page of an older push cannot be pinned - scan: descending full-range pass (the work allocations live in the high RAM block) plus a +/-512 MB near pass around the verified push-buffer base - releases decoded with an untranslated semaphore are queued and applied when the page appears (svc_pending_add on both decode paths) - a CE push that reads as a plausible stream but carries no release for the channel is re-routed through the rescue instead of silently consumed - new module param sema_rel_recover (default off): after grace + rescue, apply the channel's expected tracking value so the client's wait clears; counter-visible as releases_recovered, distinct from sema_fake_rel Evidence: fresh boot, real releases, no host watcher, no kprobes - the six UVM CE first pushes are found by content rescue, their releases applied, and the client reaches cudaGetDeviceCount rc=0 (it then waits for the third push, which this change routes to the rescue).
* sema-svc: gate per-push/per-doorbell/RPC tracing behind default-off sema_dbgArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | A healthy run left dmesg dominated by per-operation lines (sema-push, sema-lda, sema-chan rel-apply, sema-grce, DOORBELL/BAR writes, GSPRM per-RPC lines, the teardown garbage-element stream). Introduce the module parameter sema_dbg (default 0) and route every high-frequency diagnostic through it; the finer-grained watch knobs (sema_ops_log, sema_copy_watch, sema_copy_pat, sema_copy_dst, sema_copy_dst2) only take effect while it is on. Boot stats, channel registrations, pin events and state changes stay unconditional. Also carries the per-side translation provenance (op_tr_src, printed as prov= in the destination-window watch) and the 16 KB destination window used to observe the guest's page-table writes. Verified: fresh boot + full cdt2 smoke test (64/64, ALL OK) leaves a 520-line dmesg with zero per-push/per-doorbell/per-RPC lines.
* sema-svc: hold and retry a readable push whose operand is under a targeted ↵Arthur Hanson Rasmusson12 days
| | | | RAM scan (the DtoH dst staging VA needs multiple scan slices)
* sema-svc: operand resolver gains the targeted rm_mem fallback; per-side ↵Arthur Hanson Rasmusson12 days
| | | | translation-miss tracing (the DtoH copy operand is the failing side)
* sema-svc: targeted 256-byte op + compute-channel release-block tracing (the ↵Arthur Hanson Rasmusson12 days
| | | | DtoH op is still invisible)
* sema-svc: log the LAUNCH_DMA push VA and injected/allow flags; widen the ↵Arthur Hanson Rasmusson12 days
| | | | op-failure log budget (the DtoH work path is invisible at the old caps)
* sema-svc: apply UVM increment releases once per GPFIFO slotArthur Hanson Rasmusson12 days
| | | | | | | | | | | UVM CE tracking semaphores start at a base value and each completed push adds the release payload (observed base 1 + payload 1 -> the client waits for 2). The absolute comparison (payload > cur) could not see the second push and the release was silently dropped, leaving cuInit/UVM stuck. Track applied GPFIFO slots in a per-channel bitmap (preserved across re-registration and the pump's snapshot write-back) so a pin-batch rewind replays entries without double-incrementing; rescued pages stay release-only and never mark a slot.
* sema-svc: execute CE-class work on host-injected compute pushes (the DtoH copy)Arthur Hanson Rasmusson12 days
| | | | | | | | cuMemcpyDtoH submits the copy as C5B5 methods bound on subchannel 0 of a compute channel (eng=0). allow_ops was false for every eng=0 channel, so the LAUNCH_DMA copy was decoded (and its release applied) but never executed, and the client read zeros. A host-injected translation is verified truth: allow CE work on those pushes too.
* sema-svc: GRCE write log budget follows sema_ops_log when raised (catch late ↵Arthur Hanson Rasmusson12 days
| | | | HtoD writes)
* sema-svc: CE op translation falls back to the GMMU walk for virtual operandsArthur Hanson Rasmusson12 days
|
* sema-svc: decode GMMU PDE/PTE addresses with the real encoding (addr>>12 in ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | bits 32:8/53:8) The mask-and-keep decode dropped the left shift by 12, so every page-table child pointer and leaf resolved 16x too low and no GMMU walk ever produced a translation (walks_ok=0 on every boot). Verified against live page tables: directory entry 0x2020a -> table 0x202000, leaf 0x0600000000760001 (aperture VID, KIND=6) -> PA 0x7600000.
* sema-svc: destination-keyed copy watch (dump page-table writes)Arthur Hanson Rasmusson12 days
| | | | | | sema_copy_dst=dumps the source bytes of every copy whose destination falls in the given page, so the 32-byte page-directory updates the guest writes into vidmem can be read directly and their entry encoding learned.
* sema-svc: decode GRCE inline-data DMA (small HtoD path)Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA submits it on the compute channel as a compute-class DMA whose source is the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>, LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run (opcode 3) and no GR work ran at all. - Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT, OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run. - svc_grce_inline_write resolves the destination VA through trusted sources (pins, map, published page directories, targeted rm_mem resolve), pins the page and writes into vidmem or guest RAM; budgeted sema-grce logs name the target or report UNRESOLVED. - cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection of the copy path; pushed by push_guest_tools.sh.
* sema-svc: run CE work before its release, grace unreadable CE entries, ↵Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | payload-pattern audit - CE LAUNCH_DMA releases are now applied after the push's collected work runs: the client reads copied data as soon as it observes the release, so release-first let cuMemcpyDtoH observe a copy that had not executed. - Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms, default 8s) before the release fallback consumes them, so a translation that lands within a service pass lets the push be decoded and its work executed instead of being dropped. - sema_copy_pat matches an 8-byte value against every copy's src/dst first word and scans decoded pushes for it, naming the submission that carries a known payload regardless of size/encoding. - sema-dma-skip logs transfer-shaped launches that collect no op; op log lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG. - A real release application now records the release VA, so decoded pushes no longer trigger a spurious fake release.
* sema-svc: copy-watch also logs dst8_after (write-effect check)Arthur Hanson Rasmusson12 days
|
* sema-svc: sema_vidmem_find pattern search (CE data-path ground truth)Arthur Hanson Rasmusson12 days
|
* sema-svc: sema_copy_watch (size-keyed src/dst byte audit for the CE data path)Arthur Hanson Rasmusson12 days
|
* sema-svc: sema_vidmem_dump param to inspect emulated vidmem (CE data-path audit)Arthur Hanson Rasmusson12 days
|
* sema-svc: sema_ops_log param (default 8) to audit executed CE operands; ↵Arthur Hanson Rasmusson12 days
| | | | run_cycle OPS_LOG knob
* sema-svc: CE operand translation consults pinned translations before the ↵Arthur Hanson Rasmusson12 days
| | | | scanned map (push-data memcopies were rc=-ENOENT; UVM inline writes now execute)
* sema-svc: route 906f/report-semaphore releases to the exact GR counter slot ↵Arthur Hanson Rasmusson12 days
| | | | (all 20 channel releases now land), queue payloads that arrive before the slot pin
* sema-svc: deliver the tracking-sem payload for every consumed GPFIFO entryArthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | The client's UVM_REGISTER_GPU blocks in uvm_push_end_and_wait on the first CE channel's tracking semaphore (pool word 0x121000000) waiting for value 3. The pool payload stayed at 1 while the client pushed: the pump processed entries whose push bytes were unreadable or had no decodable release, so the release bookkeeping advanced (expected_sem_payload) but nothing was written to the pool page. Treat the GPU semantics as normative: each consumed GPFIFO entry on an engine channel releases the channel's tracking semaphore with the incrementing queued value. When the decoded stream carries no release and sema_fake_rel is enabled, deliver the expected payload through svc_fake_release() instead of only bumping the counter. Also initialize expected_sem_payload for a fresh pool page whose current payload is 0 (expected = 1) so the first push is deliverable from the start.
* sema-svc: wake the service thread on every injected pinArthur Hanson Rasmusson13 days
| | | | | | | The injection queue was only drained at the top of the service loop, which can be inside a multi-second pump/discovery pass; live runs showed the GR counter pins landing ~1 s after the client mapped the page, i.e. after its context-init deadline. Waking on enqueue drains the batch immediately.
* sema-svc: GR counter slots only from injected pins; route GR releases ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | through the exact slot Live evidence: with the counter page pinned, counter_pa still resolved to a stale self-discovered page, so every release write went to memory the client never polls (host read back its own 1s; client's page stayed 0) and the client's context-init wait expired. The generic release path could also write through that stale translation. - svc_counter_autoreg / svc_counter_learn trust only the injected-pin table (svc_ipin_lookup); learn re-registers when the injected PA differs. - Releases whose target is in the GR counter VA range skip the generic translation write entirely and go through the injected-pin-backed slot.
* sema-svc: bounded diagnostics for GR counter writes; scanner guarantees ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | counter-page delta - rel-apply/publish now log the first 32/48 outcomes (written, skipped with current value, no slot, read failure) so the live counter page can be matched against host writes. - guest pin scanner logs the counter page's pagemap state each loop and adds it to the delta set explicitly, so the page is pinned the moment it is mapped even if the window scan misses it.
* sema-svc: never consume a CE entry whose release was not deliveredArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | The fake-release fallback acknowledged unreadable CE pushes even when the tracking-semaphore pool page had no translation yet, so the GPFIFO entry was consumed without writing the semaphore the client waits on (observed: client stuck in UVM_GPU_REGISTER while CE channel 2's release never landed, its push-buffer pin arriving ~2 s later). - svc_fake_release now reports failure when the pool page cannot be translated/read/written instead of pretending success. - CE channels with the fallback enabled leave the entry pending (no multi-second RAM rescue) until the pool pin lands; the skip path only consumes an entry when the release was actually delivered.
* sema-svc: persist GR counter slot per channel; stable slot layout; bulk pin ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | queue 4k - counter_pa autoreg result is written back to the channel table so later passes cannot re-target a different slot when the live GR count changes during teardown; re-registration of the same channel preserves the slot. - counter slot layout uses the max GR count ever seen in the generation instead of the shrinking live count. - sema_map_add queue 64 -> 4096 entries so a full-window pin burst drains without ENOSPC retries. - pin_gr_stream: one batched sudo pass per flush (was one process per run, ~4 s for 400 runs), retrying only queue-rejected runs. - semfix_uvmt: SEMFIX_SKIP_PB=1 skips the blocking push-buffer chain walk when the vsock window stream already supplies those pages.
* sema-svc: auto-register GR counter slots from rank layout + pinned pageArthur Hanson Rasmusson13 days
|
* sema-svc: diagnostics for GR counter release/publish pathsArthur Hanson Rasmusson13 days
|
* sema-svc: learn GR counter PA from pinned release VA (auto-register, publish ↵Arthur Hanson Rasmusson13 days
| | | | puts)
* sema-svc: sema_skip_ops parameter (apply releases without running pushed ↵Arthur Hanson Rasmusson13 days
| | | | memsets)
* sema-svc: apply releases before push memsets; poller: bypass read cacheArthur Hanson Rasmusson13 days
|
* sema-svc: disable swept-page release writes (guest memory corruption); wake ↵Arthur Hanson Rasmusson13 days
| | | | on doorbell + tight pump loop while entries execute
* guest-diag: stream uvmt sems over vsock (no guest-agent race); dump channel ↵Arthur Hanson Rasmusson13 days
| | | | last release VA; larger push log budget
* sema-svc: never apply swept-page releases with no live channel (guest boot ↵Arthur Hanson Rasmusson13 days
| | | | corruption); tooling guards
* sema-svc: apply decoded releases to registered counter slots; single-QMP ↵Arthur Hanson Rasmusson13 days
| | | | semfix+uvmt watcher
* kernel: grace window for untranslatable GR entries; tools: coalesce window ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | pin output The first GR window pin landed ~2 s after the pump consumed the freshly created channels' GPFIFO entries on the first failed read, so the pushes (and the tracking-semaphore releases they carry) were lost. GR entries with an unresolvable push page now stay pending for sema_gr_grace_ms (default 8 s) before the stale-entry consume; a pin landing in that window makes the entry readable and it executes normally. The completion counter is still published from put at the end of every pass, so the client's counter wait is not delayed. guest_pmap_win.py coalesces contiguous VA+PA pages into run lines, so a 15k-page window transfers as ~200 lines instead of ~15k and the pin batch lands within a second of the pages appearing.
* tools: pin GR push-buffer window from guest pagemap per boot; widen ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | injected-pin table The GR channel GPFIFO entries point at pushes in the client GPU VA window (observed 0x200400000 + stride, one allocation per channel) and the pump cannot read them: pushes_nomap grows and the tracking-semaphore releases they carry never land. The pushes are sysmem pages mapped into the client process, so /proc/<pid>/pagemap in the guest gives the GPA. New guest helper guest_pmap_win.py batch-reads the pagemap of a VA range (512 entries per pread). New host tool pin_gr_window.py rescans the window on an interval, follows cdt pid changes, and injects VA->PA runs through sema_map_add (same mechanism as the counters and the CE pushbuffer pins). The injected-pin table grows 8k -> 32k slots so the CE pushbuffer (4096) plus the GR window pages fit without drops.
* kernel: batch the map-add channel rewind; trust injected streams for opsArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | Two fixes for the CE data path. (1) svc_handle_map_add rewound every channel by 8 ring entries on every injected mapping; a 16 MB push-buffer pin is thousands of page injections, so the pump's last_get was pinned in place and it re-read the same entries for the whole batch (3645 submissions, zero ops). The rewind now runs once per drained batch. (2) The push-start heuristic (op=INC_METHOD at address 0) rejects UVM's real CE memop pushes, which start with an IMM method at address 8 (observed 0x60020002, PTE data words) or a release method at 0x700 (0x200381c0); a buffer read through a host-injected (ipin) translation is now decoded fully trusted, ops included. Result on the next boot: memset=2718/0/0 ringops=1794 (first sustained CE work), false 12k->1.9k. semfix_watch: bulk-pin contiguous runs via sema_map_add len, read rm_mem size, 16 MB (4096 page) cap, non-blocking QMP read (the fixed 30 ms sleep made 4096 gva2gpa calls take two minutes), and KEEP_TRACE=1 to preserve the guest trace for post-mortems.
* kernel: decode release pushes read through injected pins instead of rescuing ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | them Real CE release pushes start with INC_METHOD addr 0x700 (0x200381c0, count 3, subch 4), and svc_method_stream_plausible requires method address 0, so every readable release push was treated as a false pairing, rescued (fails) and skipped - the client's tracking-semaphore release never applied and cuCtxCreate waited forever (slot payload 74 vs queued 75). A buffer read through a host-injected (ipin) translation is now decoded release-only (allow_ops=false) when it fails the push-start heuristic; self-discovered translations keep the strict rescue path.
* kernel: authoritative host-injected VA->PA pins (ipin table) + wider pin tableArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | The CE push-buffer page 0x120000000 resolved to a zero-filled page even after sema_map_add injected the correct translation: the shared pin table silently dropped inserts once a hash cluster exceeded the 32-probe window (the page hashes to slot 0, a hot cluster), and lookups fell through to the content-scanned map and answered with a stale page - so every CE push read as implausible and memset stayed 0. Injected translations now live in a separate authoritative table checked before every other source and are never written by self-discovery; the general pin table is 4x larger with a 4x probe window and counts drops (pin_drop/ipin_drop) in the stats line. guest_kprobes.sh disables the kprobe group before removal (EBUSY) and matches p:kprobes/<name>; the guest wait sampler can run counters-only with GDB=0.
* sema-svc: fake_rel applies releases immediately (skip multi-second CE rescue ↵Arthur Hanson Rasmusson13 days
| | | | scans)
* sema-svc: learn the tracking-sem VA from readable pushes for the release ↵Arthur Hanson Rasmusson13 days
| | | | fallback
* sema-svc: optional CE release fallback for unreadable pushes (sema_fake_rel)Arthur Hanson Rasmusson13 days
|
* sema-svc: async VA->PA injection queue; guest-diag: fast counter+pin ↵Arthur Hanson Rasmusson13 days
| | | | injector, wait sampler
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-12 12:18:56 +0000