summaryrefslogtreecommitdiffstats
path: root/kernel/gsp_sema_service.c
Commit message (Collapse)AuthorAge
* vgsp-sema: reachable GR counter publish + injected-pin override at autoreg ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (FORENSICS-v2 H3 + anomaly 6) FORENSICS-v2 H3: svc_counter_publish() was reachable only at the end of a fully pumped channel; when a GR channel's ring or USERD cannot be read the per-channel loop early-`continue`s above that call (USERD not resolvable, GP_PUT unreadable), so pub_calls stayed 0 even with all 20 GR counter slots registered and the client's tracker waiting on them. Duplicate the publish into the safety-net block at the top of the per-channel loop, next to svc_counter_autoreg/svc_counter_flush_pending: - the value is a real GPFIFO put read from the channel's registered USERD; when the USERD is not readable nothing is written, so a guessed put can never advance the client past work the pump cannot see; - the target is the proof-gated slot registration (injected pin or the read-back-validated derive), never an unproven page; - the same relay-hold guard as the end-of-pass call applies. The end-of-pass site now publishes for CE channels only, so a GR pass does not publish twice. Anomaly 6: an earlier-derived counter_pa could disagree with a later-arriving injected pin (r1: derived 0x123a43ff0 vs pin-backed 0x123a28ff0) and nothing re-checked the pin while counter_pa was already registered - a publish would then advance a page the client never polls. svc_counter_autoreg() now re-checks the rank's slot against the injected pin on every pass and re-registers (under chan_lock) on a mismatch, which also makes the new publish proof-gated through the pin. Evidence: FORENSICS-v2 H3 + anomaly 6; 11-hostdmesg-v2-r1.log:2429-2714 (20x "not at fb ... probe rc=-11", pub_calls=0, gr_rel=0)."
* vgsp-sema: kernel-native GR ring/USERD discovery (both apertures, trusted ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | walk, content fallback) (FORENSICS-v2 H1+H2) The v2 campaign confirmed the CE-side fixes work (89/89 releases applied, pool page resolved src=trusted) but the stall moved one phase later: the guest authors its eight GR context-init pushes and the host never reads a single GR GPFIFO entry (ring_auto=6 CE-only, pub_calls=0, gr_rel=0, work_gr req=0). The GR ring VA 0x200200000+R*0x3000 and its USERD are never resolved: svc_gr_ring_autopin only tried FB conventions, userd-derived FB candidates and existing pins (FORENSICS-v2 H1/H2). Three bounded, content-gated extensions: 1. Trusted walk (H1): try svc_lookup_trusted() for the ring VA - the driver-published VAS walk / self-discovered directory chain that resolved the CE pool page in v2 ("pool-pa ... src=trusted", 11-hostdmesg-v2-r1.log:2313). 2. Both apertures (H2): every ring-derived and registered USERD candidate (svc_chan_userd) and every userd-derived GR ring candidate (svc_gr_ring_autopin) is tried in the registered aperture first and the other one second; v2 read the GR USERD through FB and got 0/0 while the same boot's pushbuffer pins are sysmem. 3. Content fallback (H1 last resort): one self-gated, 2 s-rate-limited, 60-pass-capped 64 KB chunked scan of registered guest RAM locates, per rank 0..7, the release-group page ("200426c0 00000002 0400ffXX 00000001", group at push len-20, page-aligned push at VA 0x200400000+0x400000*rank, m104/handoff 2576-2597 geometry) and the ring page whose entry0 is the exact (push VA, len) pair. A pair is pinned only when the full gate passes: entry0 bytes exact, the push parses end-to-end as a method stream, the release group is exactly at len-20 with this rank's slot and payload 1, and the USERD at +0x2000 reads a plausible get/put pair (or a readable idle 0/0). Anything else pins nothing. All candidates share the existing rule: a wrong guess costs a failed probe, never a wrong pin. Evidence: FORENSICS-v2 H1/H2, m104 17-host-m104-r1.txt:169-171, v2 dmesg 11-hostdmesg-v2-r1.log:2313-2714.
* vgsp-sema: multi-page content rescue pins a whole CE push from RAM content ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (ISSUE-0001) svc_rescue_try() substitutes exactly one candidate page and requires every other page of the entry to resolve through svc_lookup_trusted(). For a multi-page push whose pages have no translations yet that precondition can never hold: the content scan finds the release-group page by content, but the pages that need a candidate are exactly the ones missing from every trusted source. That is the stalled-boot class named by SA-STALL (entry va=0x120021db8, len=12816, release #39; 17-host-m84-r1.txt:841-842), where the userspace pin stream was the only supplier of those translations. Add svc_rescue_run_try(): once content has anchored one page of the entry (release-group candidate from the matcher, or the verified window base), read every page of the entry directly from the same signed page delta in both physical directions, validate the assembled buffer exactly like the single-page rescue (full method-stream parse + the channel's exact next release), and only then pin every page of the run. A failed direction pins nothing; a passed validation is content proof for every page. Wire it into the svc_pb fast path, the hot-page scan and the descending RAM scan, ahead of the unchanged single-substitution try. This makes the pump able to consume a starved CE push with no watcher pins and no pre-existing per-page translations; the watchers become pure accelerators. New counter resc_run is printed when non-zero. Evidence: ISSUE-0001, SA-STALL 3.2/3.4; 17-host-m104-r1.txt entry[38] and 17-host-m84-r1.txt rescue rejects.
* vgsp-sema: keep push-buffer base discovery alive until the CE windows drain ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | (ISSUE-0001) svc_pb_discover() ran with a flat 12-pass budget for the whole boot and then retired, even when a registered CE channel had no content-discovered push-buffer base yet. A channel that appeared after the budget was spent (including one reappearing with a new client session) then had no paired window, so every push went through the slower per-page rescue and the userspace watcher pin stream. Stop the pass only after SVC_PB_IDLE_PASSES consecutive passes with no new base AND with every registered CE channel either paired or having a base recorded for its release slot, with a hard cap (SVC_PB_PASS_MAX) so a channel that genuinely has no base page cannot scan forever. The 3 s rate limit and chunked walk keep the service thread responsive; the per-pass log now carries the idle count. Evidence: ISSUE-0001; v2-r1 counter freeze (pb=6/12, see backups/evidence-20261002-victory/11-hostdmesg-v2-r1.log) and the SA-STALL class where the CE window was reached only after discovery had stopped.
* vgsp-sema: retryable page-directory discovery from fresh guest-truth pairs ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | (ISSUE-0001) svc_pd_discover() latched svc_pd_tried before its first scan, so a call that ran before the client's page tables existed (or before enough verified push-buffer pairs were known) disabled the one kernel-native source that translates pool/push windows the content-scanned map and the published directories miss, for the rest of the boot. That source is what svc_lookup_trusted() and svc_pool_pa_resolve() fall back to when the userspace watcher pins race. Make discovery retryable and evidence-driven: an attempt that cannot run does not latch, a scan runs only when the sample-set fingerprint changed, and the total is capped at SVC_PD_ATTEMPTS_MAX per boot - the same order as the old per-pairing re-arm, which this supersedes (the stale re-arm in svc_pb_pair is removed). Evidence: ISSUE-0001, SA-STALL 3.2/3.4 (the deferred pool release of the last CE push with no translation when watchers race).
* vgsp-sema: kernel-native pool release translation + starvation telemetry ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | (ISSUE-0001) svc_pool_pa_resolve() could translate an UVM pool/waiter page only from watcher pins, the read-back cache, the self-discovered directory and a targeted rm_mem scan. When the userspace watcher races a boot and the self-discovered directory has no entry yet, a starved completion release sits in svc_pending[] forever (p13-r4: applied frozen at 89/90, gr_rel=0), the guest tracker never wakes and CUDA blocks at DEVICE. Add: - svc_walk_vaspaces() (the driver-published page directories - the walker svc_lookup_trusted always used, previously absent here); - a final proof-gated svc_lookup_trusted() fallback (also the only path that triggers svc_pd_discover()). Both read-back proven like every other source. Plus per-entry starvation telemetry (tries + bounded logs) so a future stall names the exact va.
* sema: fix GR counter-slot mapping to fixed pool total (20)Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | The counter slot for GR rank r is 0x20400fec0 + (TOTAL-1-r)*0x10 with TOTAL the fixed pool total. The autoreg used a live high-water channel count instead; channels register progressively, so an early pass with a partial count (16) registered the ranks present at the wrong slots and every later rank's correct slot was then rejected as owned. Result: ranks 10..19 never got a counter, no counter was ever published, and the client's userspace tracker never advanced (cudaMalloc stall while the guest RM polled MC_SERVICE_INTERRUPTS). Validated on a live boot: 20/20 slots registered with 0 collisions (previously ranks 6..9 installed at the n_gr=16 mapping and ranks 10..19 rejected).
* vgsp+guest-diag: kcore-truth pin resolver (pool/pushbuffer/channel ↵Arthur Hanson Rasmusson5 days
| | | | | | | | | | | | | | | | | | | | | | | ring+userd) + GR ring userd-derived candidates guest_pin_stream.py now resolves translations entirely from guest truth via /proc/kcore + a manual page-table walk (no QMP races): the CE semaphore pool and pushbuffer memory from the UVM rm_mem structures, and each channel's GPFIFO ring + GPPUT from the tracking semaphore's embedding uvm_channel_t (channel_info located by its gpFifoGpuVa value). This removes the watcher pipeline's flakiness on the cudaMalloc stall class (missing pool/pushbuffer pins). gsp_sema_service.c: svc_gr_ring_autopin tries the guest-registered USERD memdesc base as a ring candidate (the fixed FB convention only matches when the guest's VRAM allocator happens to reproduce the old layout), plus an optional (gsp_sema_dbg) one-shot FB scan + raw-word diagnostics when every candidate fails. tools/cuda-ab/p2: self-checking arbitrary-kernel acceptance suite (10 programs: sequential launches, shapes, params, smem/GEMM, streams, malloc churn, printf+error, FMA benchmark, mixed, managed probe) with host oracle expectations and a guest runner.
* vgsp: m140 - resolve 33 pages/pointer (128KB buffers with 0x400 in-page ↵Arthur Hanson Rasmusson6 days
| | | | offset span 33 pages; tails read the next reloc record)
* vgsp: m137 - CE-hold liveness watchdog (bound deferred CE waits across ↵Arthur Hanson Rasmusson6 days
| | | | passes; targets the guest cudaMalloc stall class)
* vgsp: m121 - resolve param-page kernel-argument pointers (offsets >= 0x150) ↵Arthur Hanson Rasmusson6 days
| | | | before the bounce cap
* vgsp: m120 - data-fill 32 pages/ptr under the 100 cap + rewrite hit/coverage ↵Arthur Hanson Rasmusson6 days
| | | | logging + param 0x150 dump + main-PT slot dump
* vgsp: m118 - data fill candidates at page granularity (runs start at the ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | pointer page) m110 collected candidate windows at 2 MB granularity, so the 64-page runs started at the window base and never covered the pointer pages themselves - the bank/param pointers matched no record and were never rewritten (m117: only 2 relocations, trap unchanged). Collect candidate pages instead.
* vgsp: m114 - pre-commit bounce window for the kernel's data (native run w/ ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | | | | | copy-in/out) Decisive evidence: the SM walker resolves only the commit-time tree (every late write/install fails: m94, m99-m113; UNSET refused 0x1f). Plant a 112-page bounce window into the main small-PT at stage 3 (slots 0x110..0x17f - the proven win0 pattern), copy the launch's resolved data pages in, pointers are already rewritten to those VAs, and copy results back after the release. The mail message lives at bounce page 100 (c[0x4][0x0]=0x174000). Data cap 100 pages (440 KB), matching the shrunk victory_mail (3 x 128 KB).
* vgsp: m110 - relocate data into the engine-walkable main-PT window; shrink ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | | | | the demo All UPDATE_PDE_2 installs (post- or pre-commit) are invisible to the SM walker; the only window it honours is the commit-time main small-PT page (VA 0x100000+, the proven win0/const window). Relocate each candidate data pointer's first 64 pages into slots 0x110..0x1f0 of that window (fresh PTE lines, direct leaf writes), rewrite bank pointers per page record, and put the mail page at the reserved slot 0x1f0 (c[0x4][0x0]=0x1f0000). Shrink victory_mail's buffers (n=1<<15, 128 KB each) to fit the 2 MB window.
* vgsp: m106 - pre-install the data PT window pool in stage 3 (pre-commit)Arthur Hanson Rasmusson6 days
| | | | | | | | | | | | | Post-commit UPDATE_PDE_2 installs are accepted (status 0x0) but ineffective (m94's scratch windows, m101-m105's reloc windows: the SM never resolves them), while the stage-3 ctx-window installs are honoured. Add a fixed 32-window pool at GSP_PHYS_RMCALL_VA_GR_DATA_BASE (0x100000000 + k<<21, inside the server-managed region), each backed by a module PT page installed through UPDATE_PDE_2 during stage 3; the engine data map then only writes leaf PTEs into those fresh PT pages at launch (the P6b-proven 'direct leaf write into a walker-installed fresh table' pattern). The mail injection uses the reserved last window (k=31); service data windows cap at 31 and use the same fixed layout.
* vgsp: m102 - relocate guest data into the server-managed VA region and ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | | | | | rewrite bank pointers The server refuses UPDATE_PDE_2 (status 0x1f) for guest CUDA VAs: they are client-managed, while the golden VAS reserves the server-managed region [0x100000000, 0x11fffffff) (gsp_phys_booter.c:8455). Map each resolved data page there at an affine-relocated VA (identity HPA, no copies), rewrite every 8-byte pointer value inside the shadowed const-bank pages (banks 0..7) that falls inside a mapped guest window to the relocated VA, and raise the Tier-1 window pool to 32 windows x 512 PTEs so the relocated pages fit. The kernel now reaches its real data through the server-installed windows.
* vgsp: m100 - scan every const-bank page for data VAs; data PTEs kind 0 by ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | | | | default The +0x880 trap instruction disassembles (nvdisasm on the m98b prologue bytes) to LDG.E.U8.CONSTANT.SYS R5, [R2.64+UR4] with UR4 = c[0x4][0x0] - the first global load's base comes from const bank 4, not the c[0x0] param page the m99 scan covered. svc_gr_data_fill now scans every bank page the launch carries (QMD words 32..47, even LO words: banks 0..7), deduping up to 16 windows. Data PTEs use the gr_pte_kind param (default 0 = the kind proven SM-visible via the const reads) instead of a hardcoded 6.
* vgsp: m99 - identity-map the forwarded kernel's guest data pages into the ↵Arthur Hanson Rasmusson6 days
| | | | | | | | | | | | | | | | | | | | | | | | | engine tree The m97/m98 decode proved the SM instruction fetch and const reads work; the wall is the kernel's first global data access (prologue instruction at +0x80) to a guest-VA device pointer (param page words 6..9 carry the demo's two 16MB buffers at 0x763dXXXXXXXX), which is not mapped in the engine's page tables. The walker walks OUR root->PD2->PD1->PD0->PT tree (SC PDB == our VASPDB chunk, proven by matching the same-run INST base to the PDB-root DMA log), and fresh direct-planted tables are visible at first fetch. Service side (gsp_sema_service.c): svc_gr_data_fill() scans the param (c[0x0]) page for plausible device VAs, resolves the containing 2MB windows (-1..+16 windows, early stop when a window resolves nothing) via svc_resolve_va on the service thread, and carries {va, pa} in svc_fwd_req.gr_data; svc_fwd_gr_work() converts PA->HPA and attaches the list to the submit. Backend (gsp_phys_booter.c): gsp_phys_rmcall_gr_data_map() plants VA->HPA leaf PTEs (GENERIC_MEMORY kind 0x06, like UVM/RM on Turing) into lazily allocated PD1/PD0/PT chunk branches under root[0] (VAs < 2^47), caching branches per index; the submit maps all req->data pages before publish and logs the count.
* vgsp: TSG-sibling mode 30, UVM-stall Patch C, QMD shadow-window ensure + ↵Arthur Hanson Rasmusson7 days
| | | | launch diagnostics
* vgsp: PRIV VA-only booking bit + verify the exact slot before consuming ↵Arthur Hanson Rasmusson7 days
| | | | | | | | | | | | | unreadable CE entries - fwclient mask bit 15 (0x8000): book the PRIV_ACCESS_MAP VA (the FWCLIENT entry form is init-only; the golden9 form carries the VA at VA_GOLDEN9_PRIV). Accepted on V6 (status 0), completing the maximal promoted context. - gsp_sema_service: svc_entry_want_reached() verifies the exact tracking slot against entry_want before the three unreadable-CE consume shortcuts treat the entry as delivered; when the slot lags, re-seed the expectation and let svc_fake_release author or queue the exact slot (UVM registration stall pair with the pool-VA verified-resolver queue).
* vgsp: golden-exec shape bisection + fwclient per-entry promotion + pool-VA ↵Arthur Hanson Rasmusson7 days
| | | | | | | | | | | | | | | | | | | | | | | | | stall fix - Mode 28 golden execution lane behind gsp_phys_rmcall_gr_promote=28 with a shape bitmask (gsp_phys_rmcall_golden_exec_shape: ring / entries=8 / sysmem USERD / USERD size / hObjectBuffer=VAS / skip replay TURING_A) and a USERD control-offset helper. Finding: the RM-managed golden channel hangs any GR object construct (-110, no assert) as soon as client submission wiring (ring or sysmem USERD) is attached, so only the captured server-owned shape constructs; the machinery is kept for future wiring ideas. - fwclient mask bit 11 (0x800): send each non-MAIN FWCLIENT promote entry as its own one-entry control and continue past refusals. V6 acceptance matrix: PM, PATCH, PRIV_ACCESS_MAP, UNRESTRICTED_PRIV_ACCESS_MAP accepted, FECS_EVENT refused 0x1f. - fwclient mask bit 12 (0x1000): golden9 VA-only global bookings (BUNDLE_CB, PAGEPOOL, ATTRIBUTE_CB, RTV_CB_GLOBAL) accepted on V6; bits 13/14 (0x2000/0x4000): FECS PA-only (refused 0x1f) and FECS VA-only (accepted). - gsp_phys_rmcall_srv_main_ignore: ignore a server-reported MAIN in the srv-ctx commit and force the client INITIALIZE_CTX + VA-only fallback. - gsp_sema_service: pool semaphores never fall back to the learned map; queue the absolute release via svc_pending_add until svc_pool_pa_resolve succeeds, and judge pool slots through the verified resolver in svc_apply_pending and the release decode (UVM registration stall fix).
* vgsp: verbatim replay of the captured GR launch prelude+windowArthur Hanson Rasmusson7 days
|
* vgsp: dump the full captured guest push stream for method decodeArthur Hanson Rasmusson8 days
|
* vgsp: point inline-QMD launch at the guest QMD buffer + GR post-mortem loggingArthur Hanson Rasmusson8 days
|
* vgsp: GR forwarding static-review fixes (GR/CE slot bit, release VA->PA, ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | shadow refusal) Five defects found in a static review of the default-off GR forwarding path (no hardware test possible; GPU wedged): - clear the shared request slot's GR mode bit when a CE request is filled, so a prior GR request can never make a later CE push run the GR work function with stale fields; - resolve the QMD release0 guest VA to a guest PA on the service thread (counter pin/derive, else the trusted resolver) and only convert PA->HPA in the work item; the previous code fed the VA to the PA-based translator and would write release0 into an unrelated guest page; - take the refusal path (goto gr_page_unmapped) on every Tier-2 shadow failure instead of continuing to submit a half-built method stream; - consume the QMD run on replay passes when the GR forward is enabled so replayed QMD words are never reinterpreted as CE method headers; - only queue a captured GR candidate against the channel that captured it. Build clean (make -C kernel, W=1 only pre-existing warnings).
* vgsp: guest GR forwarding backend and capture (gsp_phys_gr_forward, default off)Arthur Hanson Rasmusson8 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a default-off guest GR (compute) forwarding path so a guest CUDA launch submitted on its GR channel can be executed by the physical GR engine on the ladder channel, with fail-closed fallback to the host CUDA bridge and the unchanged software release path. - New module parameter gsp_phys_gr_forward (default 0) and accessor. - New GR execution gate: boot_stage.rmcall_gr_exec_ok, latched only after a real GR submission logs REAL GR SEM OK; the backend refuses while it is unset. - Sema capture: a completed 64-word inline QMD on a trusted, non-replay GR decode becomes an svc_gr_cand (push identity, decoded release0, parameter/program/pointer fields, up to 8 captured inline-write records); the CUDA bridge stays authoritative while the parameter is off and is reached in order forward -> bridge -> software release. - Backend: gsp_phys_backend_guest_gr_submit() maps pages at their own guest VAs through fresh Tier-1 small-PT pages/PDE installs or relocates inline-captured pages to ladder shadow pages (Tier 2), builds SET_OBJECT + the inline-QMD method run + REPORT_SEMAPHORE on the compute subchannel, reuses the shared GPFIFO/USERD/doorbell publish tail (extracted from the CE submit path, CE bytes unchanged) and polls the QMD release and/or module semaphore, draining the epoch-gated fault queue on timeout. - Sema wiring: GR requests reuse the INFLIGHT/hold/deferral machinery so the guest channel release applies only on the submit's success/failure edge; hold shims preserve the channel engine type. - Counters and a work_gr state line plus a stats accessor. - Bounds are fixed: 16 pages, 8 inline records, 64 QMD words; every pointer/size checked, pool exhaustion and stale mappings fail closed. - New ABI layouts pinned with static_asserts. Compile-only change: no device test performed. (cherry picked from commit ce40bd872ea05776156a68afcc6d4571a0bbcad2)
* vgsp: hold every unreadable CE entry for the grace; delete the silent ↵Arthur Hanson Rasmusson8 days
| | | | | | | | | | pre-rescue fake An unreadable CE entry whose channel already had a decoded release VA was faked immediately and consumed with no log line, before any rescue attempt. A later pin-batch rewind then executed the skipped copy against a staging buffer the client had since refilled - the deterministic vadd input shift seen in runs 94/122/151/173b/215b, each with a silent pre-rescue fake. Now every unreadable CE entry with an expected payload is held for the bounded grace and the rescue is retried on each pass, so a translation that is still coming wins and the work runs before the release. Only an entry whose release was already delivered (the channel moved past entry_want) is consumed at once; the post-rescue fake remains the bounded, visible escape. The CE grace default moves 15000 -> 25000 ms (a translation arrived ~17 s after first examination in 215b). Under live campaign 229-236 (module 97eba5b7; green so far).
* vgsp: widen the CE pending-entry grace to 15 sArthur Hanson Rasmusson9 days
| | | | | | | | | | | A consumed entry's work is never run; if its release lands via the fallback before the entry's copies have executed, the client refills the staging slot the copies then read (observed as the deterministic one-chunk vadd input shift). Keep untranslatable CE entries pending longer so their copies run before any release reaches the client. 8-cycle pilot (141-148): zero corruption; a red-fingerprint cycle (held=1165, ring_auto=6) healed to ring_auto=26 with faked=0 and the exact checksum, where the 8 s window recorded 1 corruption in 36.
* vgsp: never run work for a slot consumed without it (epoch mark + ↵Arthur Hanson Rasmusson9 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | content-hash once gate) The grace-expiry fake path consumed a held CE entry without marking its GPFIFO slot as consumed for the put epoch, and the replay guard only ran while the channel was not deferring. A later pin-batch rewind then re-read the slot, found no recorded identity, and executed the push's copies against a staging buffer the client had refilled after the faked release - a deterministic vadd input shift (a[k] == h_a[k + 2 MB]). - grace-fake consume path now calls svc_replay_note() before advancing; - the svc_replay_seen() replay skip is evaluated unconditionally: a slot consumed in the current epoch is skipped on every replay, deferral or not, until a put wrap clears the mark; - the once-only identity gate gains a content witness (FNV-1a hash of the recorded push bytes): the same entry words with different push content is a new push (clients rewrite push buffers in place) and runs its ops with a cleared done bitmap; suppression requires identical content; - snapshot write-back persists the hash array; new drain counter reident= counts content-mismatch cancellations; - the grace window stays at 8 s: a faked entry's work is never run, so the hold is that work's only chance (a 2.5 s experiment made red-class runs complete with corrupt data instead of timing out). Verified: module 8e90fc7e..., three consecutive end-to-end acceptance runs (MAIL + VADD bad=0 checksum=392344090.500000 + STATUS OK).
* vgsp: per-op once-only execution bitmap for retried GPFIFO entriesArthur Hanson Rasmusson9 days
| | | | | | The entry-identity gate suppresses every operation on a replayed entry, but an entry held because one operand's targeted scan was still running is decoded again on retry, and the ops that already succeeded would re-run against a staging buffer the client has since refilled (observed: a retried HtoD wrote the second vadd input half into the first). Track a per-slot bitmap of the operations (0..63) already executed for the slot's current entry identity, plus a complete flag for the pass that finished with no operand pending. On replay, a complete entry suppresses all ops; otherwise the bitmap decides per operation. Successful fills/copies set their bit, and the skipped count is published as ops_once in the drain line.
* vgsp: end-to-end CUDA acceptance in the guestArthur Hanson Rasmusson9 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The no-claim emulation path now runs a stock CUDA program end to end: device properties, cudaMalloc, a GPU-authored string kernel read back byte-for-byte, two 4 MB host-to-device copies, a 1 M-element vectorAdd executed on the physical GPU through the host bridge, and the device-to-host copy-back verified element by element. Three consecutive supervised cycles report MAIL + VADD bad=0 checksum=392344090.500000 + STATUS OK with identical input/output hashes. Two data-path correctness fixes make the transfers sound: - Once-only op execution per GPFIFO entry. The pump replays recently consumed ring entries when translation batches are injected so missed releases can land; re-executing a CE copy is not idempotent because its source is a client staging buffer that may have been refilled. Slots now record their raw entry words and a replayed entry matching that identity suppresses operations while still processing releases. - Per-page translation of virtual operands. svc_copy resolved only the first page of a VA range and walked the PA linearly; client staging ranges are frequently split (observed 195 + 61 pages), so the tail of alternating chunks was read from unrelated memory. Multi-page copies now re-resolve each 4 KB page through the trusted resolver. Also carries the earlier uncommitted emulation fixes that got the client to the launch (QMD address-window gate, counter registration persistence, 512 MiB emulated FB, bridge slot hygiene) and bounded diagnostics (sema-bigop, sema-bigcopy, bridge-in/out hashes, daemon INCHECK/OUTCHECK/dumps).
* vgsp: host CUDA bridge + sema release recovery; greeneye simulator waveArthur Hanson Rasmusson10 days
| | | | | | | | - kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean - tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests - tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode - debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding - STATE-CURRENT/docs updates
* kernel: GR context-promote bisection modes 4-17 and server-managed-ctxbuf ↵Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | registry variant; complete vgsp rename; docs/tooling refresh - gsp_phys_booter: GR promote bisection modes 4-17 (two-phase zero-field VA, batched entries, hVirtMemory, bare VA, KMD SKIP_CTXBUFFER_ALLOC channel, old-interface GPU_INITIALIZE_CTX, FBMEM context buffer, UVM-faithful PA-only, PATCH/PM auxiliary promotes, server-managed no-promote path); PATCH/PM geometry queries and buffers; pre-boot SET_REGISTRY profile selection - gsp_phys_gpu: gsp_phys_rmcall_gr_promote clamp 0..17 and gsp_phys_srv_ctxbuf parameter; registry image variant with RMSetClientRMAllocatedCtxBuffer=0 - vgsp rename across kernel, scripts, tools and docs (module vgsp.ko, /sys/kernel/debug/vgsp/, vgsp-<subsystem>: log prefixes) - risc-v-gsp-rm-debugger user-facing rename and docs/CI refresh; guest-diag tool updates; README/SBOM/install/Makefile sync Live status: CE ladder and P6 guest-copy forwarding green; GR engine context commit still open (all promote shapes accepted or cleanly refused; the channel stays CTXNOTVALID) - see Handoff-vGPU-IO.md Addendum 21 (local).
* gsp-phys: relay verify hardening and counter exposure (default off)Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | - the work_relay state line now reports the real backend data-compare verify_fail plus a verify_transient count, distinct from the sema-service hold re-read verify_fail (previously the backend counter was dropped) - a relayed copy whose host-side compare fails is re-read and re-compared once after a short delay; a transient mismatch counts as success and only a persistent mismatch falls back to the software apply as before - the final-mismatch diagnostic now logs the semaphore target/payload/poll time, the first differing byte and 16-byte dumps of live dst, live src and a submit-time analysis material, flagging source mutation vs snapshot/dst - the fault no_event path logs the post reason/return All changes are behind the default-off relay path.
* gsp-phys: P6c C3 - relay forwarded-engine faults to the guest (default off)Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | New default-off gsp_phys_fault_relay: when a relayed copy times out, the GSP-RM status queue is drained and the first OS_ERROR_LOG (0x1006) recorded after the submission's doorbell is decoded (NOCAT corroboration), matched against the ladder channel's runlist/chid, and delivered to the guest by writing its channel notifier (info32=exceptType, info16=engine type, status=0xffff) and posting the channel event. A late-release recheck skips delivery if the engine actually completed, one delivery per hold, and the existing software fallback always runs first, so no release or data is lost. - pure decode helpers extracted from the status printers; prints unchanged - status-queue walks serialized with a mutex; unmatched 0x1006 records are no longer silently dropped - fault epoch snapshot at the doorbell prevents stale records from matching - fault counters and faults:/fault_relay: state lines - default-off gsp_phys_fault_inject debug knob forces the next relayed pair's source PTE to zero to produce a real engine fault - pump notifier latch prevents clobbering a delivered fault Live: injection run produced a real CE0 FAULT_PTE (0x1f @0x200000), decoded with errString, chid/runlist matched, notifier delivered, software fallback applied (applied+fallback==engaged), clean release; the guest RM reacted to the delivered fault by tearing down its channels (faithful semantics). Healthy control: faults all zero, relay requested=40 ok=40, smoke matched=64/64, clean release. Caveat: the auxiliary event post failed (no_event=1, no MSI-X) and the guest teardown means fault injection is not smoke-compatible; the parameter stays off.
* gsp-phys: P6c C0+C1 - engine-authored guest CE semaphore (default off)Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | New default-off gsp_phys_relay: for a forwarded guest CE copy whose push ends in a one-word release, the release page is resolved at forward time, mapped on a fresh scratch line, and the real engine writes the payload directly into the guest-visible semaphore page; the deferred-release hold then verifies that page and skips only that record's software apply. Any failure (unresolvable target, refused kind, pool exhaustion, poll timeout, data mismatch, channel teardown) falls back to applying every release in software, so a release is never lost or duplicated. - new exported gsp_phys_backend_guest_ce_copy_ex() + relay counters; gsp_phys_backend_guest_ce_copy() behavior unchanged (wrapper) - new work_relay: counters line in gsp-phys-state - relay limited to CE channels and one-word absolute releases; counter and reduction releases stay on the existing paths Live: param-off regression matched the wave-2 baseline exactly (req=101 ok=101 verify_ok=101, copy_skipped=119, smoke matched=64/64, relay zeros). With the parameter on: requested=68 ok=67 skipped_sw=67, verify_fail=0, resolve_fail=0, timeouts=0, smoke 64/64, and the engine-authored guest semaphore was observed directly (sem=0x00000003 / 0x0000007e at the guest page). One copy in 68 released its semaphore but failed the host-side data compare; it fell back to the software apply (fallback accounting applied+fallback==engaged) with the smoke green - residual to triage, which is why the parameter stays default-off. A forced-unresolvable run (requested=103, fallback=105, no relay submissions) kept the smoke green. Every cycle: 5 power-domain locks held, driver rebound, WPR2 down, no rail cycles, pstore empty, same boot.
* gsp-phys: P6b waves 1.5 and 2 - deferred release and hardware-authored guest ↵Arthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | | | | | | copies - new default-off gsp_phys_work_defer_release: a forwarded guest push's releases are recorded and its GPFIFO entry is held (no consume, no GP_GET write, no counter publish/stall) until the work item publishes completion; the service thread then delivers the releases, with a fallback on failure/timeout (500 ms cap enforced every pump pass)/ pause/quiesce/channel-delete - new default-off gsp_phys_work_hw_primary: the software copy is skipped for forwarded ops, so the physical engine authors the data; a once-only software copy runs before the releases on any hardware failure or when the request cannot be queued - release-apply helpers extracted; exactly-once latches; channel-delete and teardown handling; counters and a work_guard state line - live: deferred-release run engaged=75 applied=75 timeouts=0 with the full smoke green; hardware-primary run req=100 ok=100 verify_ok=100 fallback=0 copy_skipped=119 copy_fail=0, smoke ALL OK 64/64 (guest read data the engine authored), with 19 fail-closed inline software fallbacks under single-inflight contention; a range-unregistered run proved the fallback (121 skipped copies, all recovered, smoke green) - defaults off; the software path and all existing guards unchanged
* gsp-phys: P6b wave 1 - shadow-forward guest CE copies to the physical engineArthur Hanson Rasmusson11 days
| | | | | | | | | | | | | | | | | | | - the sema service captures the first eligible copy per push from the software resolver's own operands (the real push shape mixes guest RAM with emulated-VRAM shadow addresses, including physical destinations) - HPA resolution: guest RAM through the shared guest-RAM file page cache, emulated VRAM through the mapping of its kernel shadow - new scratch page-table pool: 16 zeroed page-table pages with their page-directory entries built before any engine access, handing out one whole 128-byte GMMU PTE fetch line per copy pair (16 PTEs, 512 pairs per boot, never reused; a PTE written into an already-fetched line is not observed and there is no invalidate path) - the backend submits the copy on the module-owned channel and verifies the result host-side; every miss falls back to the ladder shadow copy, and the software pump remains authoritative - live (full smoke, both parameters on): req=42 ok=42 verify_ok=42 verify_fail=0 fallback=0, guest payload roundtrip ALL OK 64/64, clean latched release; defaults stay off
* gsp-phys: freeze-safe release, boot-proof READY, and guest-wiring P2/P3/P5-wave1Arthur Hanson Rasmusson12 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Host-freeze fix bundle (validated 2/2 supervised cycles): - power unlock now latches D0/D3cold by default (new gsp_phys_repower, default 0); releasing the claim can no longer power-gate the board and hard-freeze the host on resume - bounded post-rebind readiness gate before dropping the locks; bounded D0 settle/verify before the unbind - READY means a live GSP-RM session: the async worker and the module-owned join only expose READY when init-done is seen and no unload was sent; otherwise the claim is released fail-open - FWSEC reads DMEM0/IMEM back before STARTCPU and fails fast when the staging is not visible - the claim harness leaves the board latched by default and takes --restore-power to opt back into the old restore Guest-wiring: - P2: generation-keyed fn65 static-config snapshot cache plus a default-off identity overlay (new gsp_phys_static_live) for name/short-name/VBIOS; the probe cycle runs once per backend generation at READY - P3: default-off read-only RM_CONTROL forwarding whitelist (new gsp_phys_ctrl_forward) with emulated-handle translation to the adopted server handles and full emulation fallback - P5 wave 1: default-off shadow execution of guest CE pushes on the ladder channel (new gsp_phys_work_forward) via a dedicated ordered work item; the ladder stage-4 submission is refactored into a shared bounded helper and the software release path stays authoritative Live: module-owned boot to init-done, guest smoke ALL OK 64/64 with P2/P3 enabled, 514 shadow copies observed on the physical CE; all cycles released cleanly with WPR2 down and no reboot.
* sema-svc: rewind replays may only be skipped when their release was deliveredArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | | | The per-put-epoch consumed-slot skip must not suppress a slot whose entry was consumed *without* delivering its release (the CE grace expiry in real mode): the driver-API smoke test's CE pushes are consumed unreadable before their ring pin lands and depend on the pin-batch rewind replaying them once the translation appears. The skip now applies only to slots whose consumption delivered a release: - tail consumption records a slot when the iteration decoded a release (svc_rel_count delta); the sweep-delivered entry_want consume and the fake-release consume always record - the grace-expiry consume records only when a recovery (fake or sema_rel_recover) delivered the release - svc_fake_release counts as a delivered release So real-mode replays still re-process undelivered entries, while the runtime-client case (readable pushes delivered immediately) keeps the fast path that removed the 8 s stalls.
* sema-svc: detect a decoded release by count, not by the (unchanging) VAArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | The pump decided "this push carried a release" by comparing the last decoded release VA before/after svc_decode_stream. Consecutive pushes on a channel release to the SAME tracking-semaphore VA (UVM's tracker pattern), so every push after the first read as release-less: the entry was held for the full CE grace window (observed: exactly one push consumed per 8 s, gating the whole runtime init; the release itself had already been applied by the decode, only the pump's bookkeeping stalled). svc_rel_count now increments on every decoded release (host sem block, REPORT_SEM block, CE launch-DMA release) and the pump compares counts at all three sites (initial decode, rescue re-route, rescued-page retry).
* sema-svc: never re-process a GPFIFO slot already consumed in the put epochArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | | A pin-batch rewind replays recently consumed entries. An old entry whose rescue cannot validate any candidate then sits in the CE grace path - and because the client's put has advanced since, the replayed entry no longer matches any "new" expectation: it waits the full 8 s grace, blocking every later entry behind it on that channel. Observed live: the first push of rank 0 re-processed and skipped twice, 38 s apart, while eight later pushes queued behind it. The pump now marks each consumed slot in a per-channel bitmap valid for the current put epoch (cleared when put wraps, so a recycled slot in the next epoch is processed normally) and consumes a replayed slot immediately. Rewinds still refresh readable entries; they simply no longer resurrect entries that were already handled. Evidence: real-release watcher cycle, replay-after-put-advance no longer produces 'skipping stale entry ... after 8000 ms grace' for old slots.
* sema-svc: consume rewind replays and sweep-delivered CE entries at onceArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | | | | Two stalls serialized the runtime client's init into one 8 s CE grace window per push: - a pin-batch rewind replays recently consumed GPFIFO slots; re-examining a consumed slot runs the rescue and then holds it for the full grace window again (observed: the same VA 'skipped after 8000 ms grace' twice, 38 s apart). The pump now records each consumed slot with the put value it was consumed at (32-entry per-channel replay ring, persisted across passes) and consumes a replayed slot whose producing put has not advanced immediately. - the content sweep applies releases from RAM pages before the pump reads their entries, so the pool advances past the entry's payload while it is pending. entry_want (recorded on first examination) now detects that on both the unreadable-rescue and the readable-no-release paths and consumes the entry instead of waiting the grace window. Evidence: real-release watcher cycle, pool semaphore advances no longer gated at 8 s intervals once the replay skip engages.
* sema-svc: validate pushes with the full method encoding; deliver releases ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the sweep already applied The runtime client's second push on each UVM copy channel starts with a NON_INC method (0x60080002, count 8) and its release method sits 0x50 bytes in. The stream validator accepted only INC (op 1) headers, so the readable page failed the plausibility gate and every content-rescue candidate for that VA failed the read-back validation - the client's tracking semaphore for the push never advanced and the runtime init parked. - one shared parser (svc_stream_parse) accepts INC/NON_INC (op 1/3), IMMD_DATA (op 4, header-only), ONE_INC (op 5) and END_PB_SEGMENT (op 7), and still requires an exact method boundary over the whole entry plus trailing zero padding - the strict end condition is what keeps the content rescue safe against firmware/allocator pages - svc_method_stream_plausible and svc_stream_full_ok both use it, so a real readable push is decoded directly and a rescued page validates - the decoder skips NON_INC/ONE_INC methods whole (including their data words) instead of resyncing word-by-word through the payload, which could misread data as headers and queue bogus work Follow-on: the content sweep applies releases from RAM pages ahead of the pump, so the pump's expected payload was already satisfied when it examined the entry; the CE grace path then held each entry for the full 8 s window before consuming it - one grace period per push, serializing the client's runtime init. The pump now records the expected payload when it first examines an entry (entry_want) and consumes the entry immediately once the pool advances past it, on both the unreadable-rescue and the readable-no-release paths. Evidence (fresh boot, real releases, watchers + kprobes, no fakes): cudaGetDeviceCount rc=0 in the runtime client, device-attribute walk proceeds, pool semaphores advance; before this change the client sat in the 0x20801702 service-interrupt loop for >6 min and never left device init.
* sema-svc: deliver UVM CE tracking releases without host-side pinsArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The runtime client's registration stalled when no guest-side pin scanner was running: the content rescue refused to run before the pool semaphore was translated (its expected payload was unlearned), and unreadable CE entries were consumed after two failed rescues ~40 ms apart, dropping the client's wait. The push pages then could not be found within the entry's grace window. - rescue: drop the learned-payload gate (the matcher only needs the channel rank and the release shape); keep unreadable CE entries pending for sema_ce_grace_ms instead of consuming after a fixed attempt count - expected payload: re-learn from the pool whenever it advanced (a deferred release could land after the first read, leaving the expectation stale) - matcher: require the expected payload exactly and validate the read-back stream carries it, so a recycled page of an older push cannot be pinned - scan: descending full-range pass (the work allocations live in the high RAM block) plus a +/-512 MB near pass around the verified push-buffer base - releases decoded with an untranslated semaphore are queued and applied when the page appears (svc_pending_add on both decode paths) - a CE push that reads as a plausible stream but carries no release for the channel is re-routed through the rescue instead of silently consumed - new module param sema_rel_recover (default off): after grace + rescue, apply the channel's expected tracking value so the client's wait clears; counter-visible as releases_recovered, distinct from sema_fake_rel Evidence: fresh boot, real releases, no host watcher, no kprobes - the six UVM CE first pushes are found by content rescue, their releases applied, and the client reaches cudaGetDeviceCount rc=0 (it then waits for the third push, which this change routes to the rescue).
* sema-svc: gate per-push/per-doorbell/RPC tracing behind default-off sema_dbgArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | | A healthy run left dmesg dominated by per-operation lines (sema-push, sema-lda, sema-chan rel-apply, sema-grce, DOORBELL/BAR writes, GSPRM per-RPC lines, the teardown garbage-element stream). Introduce the module parameter sema_dbg (default 0) and route every high-frequency diagnostic through it; the finer-grained watch knobs (sema_ops_log, sema_copy_watch, sema_copy_pat, sema_copy_dst, sema_copy_dst2) only take effect while it is on. Boot stats, channel registrations, pin events and state changes stay unconditional. Also carries the per-side translation provenance (op_tr_src, printed as prov= in the destination-window watch) and the 16 KB destination window used to observe the guest's page-table writes. Verified: fresh boot + full cdt2 smoke test (64/64, ALL OK) leaves a 520-line dmesg with zero per-push/per-doorbell/per-RPC lines.
* sema-svc: hold and retry a readable push whose operand is under a targeted ↵Arthur Hanson Rasmusson13 days
| | | | RAM scan (the DtoH dst staging VA needs multiple scan slices)
* sema-svc: operand resolver gains the targeted rm_mem fallback; per-side ↵Arthur Hanson Rasmusson13 days
| | | | translation-miss tracing (the DtoH copy operand is the failing side)
* sema-svc: targeted 256-byte op + compute-channel release-block tracing (the ↵Arthur Hanson Rasmusson13 days
| | | | DtoH op is still invisible)
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-13 19:51:41 +0000