summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* guest-diag: bpz3 - probe every r14d store in the ctx bring-up (85 sites)Arthur Hanson Rasmusson13 days
| | | | | | | Run #2: all post-pump call sites up to 0x2755bf returned 0, but FUN_00374240 returned r14d=0x3e7 (999) 554 ms later. bpz3 arms every mov %eax,%r14d store in FUN_00374240 plus the alternate epilogue and the single direct 999 store, printing the first non-zero status with a backtrace.
* guest-diag: post-pump ctx-init status probes (bpz/bpz2 gdb scripts)Arthur Hanson Rasmusson13 days
| | | | | | | | bpz.py probes the four known rc=999 stores on the cuCtxCreate path plus the scanner force sites; the live run eliminated all of them and showed the engine event pump (FUN_0031a3b0) returning success. bpz2.py probes every post-pump status store in the ctx bring-up (FUN_00374240) and the function's final return, printing the first non-zero status with a backtrace.
* docs: refresh current-state snapshot (20/20 GR releases land; failure not a ↵Arthur Hanson Rasmusson13 days
| | | | UVM wait)
* guest-diag: UVM API kretprobe installer (retval probes for the ↵Arthur Hanson Rasmusson13 days
| | | | failure-instant trace)
* sema-svc: route 906f/report-semaphore releases to the exact GR counter slot ↵Arthur Hanson Rasmusson13 days
| | | | (all 20 channel releases now land), queue payloads that arrive before the slot pin
* gsprm: reply-latency logging + param dump cap; guest-diag: CDT_PAUSE, ↵Arthur Hanson Rasmusson13 days
| | | | watcher reaping, wider CE stream window
* guest-diag: one-shot gdb/state samplers for timed post-control capturesArthur Hanson Rasmusson13 days
| | | | | | | gdb_now.sh attaches once at a host-chosen moment (final RM control) and dumps all threads; oneshot_gdb.sh triggers on user-running state; state_log.sh records state/syscall/wchan so kernel busy-waits can be told apart from real userspace spins.
* sema-svc: deliver the tracking-sem payload for every consumed GPFIFO entryArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | The client's UVM_REGISTER_GPU blocks in uvm_push_end_and_wait on the first CE channel's tracking semaphore (pool word 0x121000000) waiting for value 3. The pool payload stayed at 1 while the client pushed: the pump processed entries whose push bytes were unreadable or had no decodable release, so the release bookkeeping advanced (expected_sem_payload) but nothing was written to the pool page. Treat the GPU semantics as normative: each consumed GPFIFO entry on an engine channel releases the channel's tracking semaphore with the incrementing queued value. When the decoded stream carries no release and sema_fake_rel is enabled, deliver the expected payload through svc_fake_release() instead of only bumping the counter. Also initialize expected_sem_payload for a fresh pool page whose current payload is 0 (expected = 1) so the first push is deliverable from the start.
* guest-diag: semfix counter-pin retry backoff + shorter exec timeout (stops ↵Arthur Hanson Rasmusson13 days
| | | | guest-agent starvation)
* STATE-CURRENT: UVM registration milestone; frontier moved past itArthur Hanson Rasmusson13 days
| | | | | | | - registration wait named and unblocked (first CE channel tracker, value 3; early push-buffer pinning) - frontier now the post-registration window (external allocation burst) - note the recovered guest wedge on the fastest-pin configuration
* guest-diag: single-read counter sampler (kills phantom 4-of-20 view); ↵Arthur Hanson Rasmusson13 days
| | | | vmrestart reaps stale watchers
* guest-diag: enlarge and clear the trace ring at probe installArthur Hanson Rasmusson13 days
| | | | | | The client's spin floods the default ring with uvmt/uvsv events; a one-shot uvbp event was overwritten before the pin scanner read it (push-buffer pin landed ~50 s after the client began pushing). 64 MB ring + clear.
* guest-diag: stream uvbp pushbuffer kptrs for immediate pinningArthur Hanson Rasmusson13 days
| | | | | | | | | | The first CE channel's push-buffer was only pinned ~1 minute after the client's failure: uvbp fired at t+4s but semfix's trace read (guest agent, every 5 loops) picked it up at t+70s, so the tracker wait for value 3 spun for 16 s. The guest scanner now emits '#pb 0x<kptr>' from the uvbp trace lines as soon as they appear; the host mirrors them into the sem stream and semfix resolves/pins the push-buffer through the UVM chain on the next loop (sub-second).
* guest-diag: probe the tracker completion check and its resultArthur Hanson Rasmusson13 days
| | | | | | uvsv/uvsvr capture (sem, value) and the boolean return of uvm_gpu_tracking_semaphore_is_value_completed, so a wait that fails despite an advanced payload is visible as a false return.
* guest-diag: pin the CE push-buffer window tooArthur Hanson Rasmusson13 days
| | | | | | | | | | The failing wait (uvtw) is uvm_tracker_wait_for_entry for value 3 on a CE channel whose tracking sem stayed at payload 1 for 14 s: its pushes' release values 2..3 were never applied because the CE push-buffer pages (0x120000000 + 0x100000*rank) are outside the GR window and were only found by the content sweep, late and often wrong (false=3065). Scan both windows in the guest so the host gets guest-truth translations for the CE pushes as well.
* guest-diag: probe uvm_tracker_wait_for_entry for the exact (channel, value) waitArthur Hanson Rasmusson13 days
| | | | | | The failing wait is uvm_tracker_wait_for_entry inside channel_pool_add (UVM_GPU_REGISTER): capture the tracker entry's channel pointer and value so the host can tell which release value is still missing.
* guest-diag: per-thread kernel stack sampler for the CUDA clientArthur Hanson Rasmusson13 days
| | | | | | The failing wait can live in a libcuda worker thread (strace without -f misses it); sample every /proc/<pid>/task/*/stack + wchan while the client runs.
* scripts: add browse-project.py - read-only TUI project browserArthur Hanson Rasmusson13 days
| | | | | | | | | | | - docs reader with rendered Markdown and a built-in Mermaid renderer (flowchart TB/LR with subgraphs, stateDiagram-v2, sequenceDiagram) - syntax-highlighted code viewer with per-file added-line highlighting - foldable file tree; search via git grep - history: browse docs/code at any commit or branch without checkout - architecture explorer: zoom whole system -> module -> components, with calls-into and source-file jump - strictly read-only; non-interactive modes (--render/--diagrams/--tree/--log)
* guest-diag: non-blocking trace tail + 100ms scan loop + raw counter-page dumpArthur Hanson Rasmusson13 days
| | | | | | | | | | | - the trace tail (subprocess + 256KB read) ran inside the scan loop and blocked it for seconds exactly when the client's GR window appeared; moved to a background thread whose queue the loop drains. - scan interval 250ms -> 100ms. - track_waits sampler also dumps 512 raw bytes at page+0xe00 each sample: the per-slot reads disagreed with the bulk dump within one sample because host writes land between the two, and the raw dump is the only trustworthy view of the slot layout.
* sema-svc: wake the service thread on every injected pinArthur Hanson Rasmusson13 days
| | | | | | | The injection queue was only drained at the top of the service loop, which can be inside a multi-second pump/discovery pass; live runs showed the GR counter pins landing ~1 s after the client mapped the page, i.e. after its context-init deadline. Waking on enqueue drains the batch immediately.
* sema-svc: GR counter slots only from injected pins; route GR releases ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | through the exact slot Live evidence: with the counter page pinned, counter_pa still resolved to a stale self-discovered page, so every release write went to memory the client never polls (host read back its own 1s; client's page stayed 0) and the client's context-init wait expired. The generic release path could also write through that stale translation. - svc_counter_autoreg / svc_counter_learn trust only the injected-pin table (svc_ipin_lookup); learn re-registers when the injected PA differs. - Releases whose target is in the GR counter VA range skip the generic translation write entirely and go through the injected-pin-backed slot.
* sema-svc: bounded diagnostics for GR counter writes; scanner guarantees ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | counter-page delta - rel-apply/publish now log the first 32/48 outcomes (written, skipped with current value, no slot, read failure) so the live counter page can be matched against host writes. - guest pin scanner logs the counter page's pagemap state each loop and adds it to the delta set explicitly, so the page is pinned the moment it is mapped even if the window scan misses it.
* sema-svc: never consume a CE entry whose release was not deliveredArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | The fake-release fallback acknowledged unreadable CE pushes even when the tracking-semaphore pool page had no translation yet, so the GPFIFO entry was consumed without writing the semaphore the client waits on (observed: client stuck in UVM_GPU_REGISTER while CE channel 2's release never landed, its push-buffer pin arriving ~2 s later). - svc_fake_release now reports failure when the pool page cannot be translated/read/written instead of pretending success. - CE channels with the fallback enabled leave the entry pending (no multi-second RAM rescue) until the pool pin lands; the skip path only consumes an entry when the release was actually delivered.
* sema-svc: persist GR counter slot per channel; stable slot layout; bulk pin ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | queue 4k - counter_pa autoreg result is written back to the channel table so later passes cannot re-target a different slot when the live GR count changes during teardown; re-registration of the same channel preserves the slot. - counter slot layout uses the max GR count ever seen in the generation instead of the shrinking live count. - sema_map_add queue 64 -> 4096 entries so a full-window pin burst drains without ENOSPC retries. - pin_gr_stream: one batched sudo pass per flush (was one process per run, ~4 s for 400 runs), retrying only queue-rejected runs. - semfix_uvmt: SEMFIX_SKIP_PB=1 skips the blocking push-buffer chain walk when the vsock window stream already supplies those pages.
* docs: add the full-system picture and interaction/scheduling diagramsArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | - new 03-architecture/09-full-system-picture.md: end-state map, GSP boot, guest init, RPC lifecycle, memory model, work path (bridge and end state), scheduling, isolation levels, KVM/mdev/VFIO, BAR/engine/channel/GMMU, portability, performance model, implementation/test status - new 03-architecture/10-interaction-and-scheduling.md: interaction inventory, control-plane call path, msgq element lifecycle, channel pump, concurrency, timing budgets, multi-tenant scheduling, teardown ordering, versioning - README: mermaid system diagram replaces the ASCII block; reading list extended and renumbered - component docs: GSP boot sequence, mdev/VFIO lifecycle, end-state interaction sequence; STATE-CURRENT links
* sema-svc: auto-register GR counter slots from rank layout + pinned pageArthur Hanson Rasmusson13 days
|
* sema-svc: diagnostics for GR counter release/publish pathsArthur Hanson Rasmusson13 days
|
* sema-svc: learn GR counter PA from pinned release VA (auto-register, publish ↵Arthur Hanson Rasmusson13 days
| | | | puts)
* guest-diag: scan 512KB per 2MB region and start window at 0x200000000 (cover ↵Arthur Hanson Rasmusson13 days
| | | | GR GPFIFO rings)
* sema-svc: sema_skip_ops parameter (apply releases without running pushed ↵Arthur Hanson Rasmusson13 days
| | | | memsets)
* sema-svc: apply releases before push memsets; poller: bypass read cacheArthur Hanson Rasmusson13 days
|
* sema-svc: disable swept-page release writes (guest memory corruption); wake ↵Arthur Hanson Rasmusson13 days
| | | | on doorbell + tight pump loop while entries execute
* docs: record physical-GPU integration mechanism and refresh project statusArthur Hanson Rasmusson13 days
| | | | | | | | | | | - add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
* guest-diag: stream uvmt sems over vsock (no guest-agent race); dump channel ↵Arthur Hanson Rasmusson13 days
| | | | last release VA; larger push log budget
* sema-svc: never apply swept-page releases with no live channel (guest boot ↵Arthur Hanson Rasmusson13 days
| | | | corruption); tooling guards
* sema-svc: apply decoded releases to registered counter slots; single-QMP ↵Arthur Hanson Rasmusson13 days
| | | | semfix+uvmt watcher
* guest-diag: live tracking-sem poller (payload VA + raw fields)Arthur Hanson Rasmusson13 days
|
* guest-diag: print tracking-sem state (index, queued, payload) on changeArthur Hanson Rasmusson13 days
| | | | | | | The final cuCtxCreate stall is a UVM tracking semaphore whose completed value stays 0; the pool payload is the ground truth. semfix now prints idx/+0x10/+0x28/payload for every tracking sem it sees, whenever a field changes, so the last stall before a failure is visible in its log.
* guest-diag: sparse allocation-head scan in the pin streamArthur Hanson Rasmusson13 days
| | | | | | | | | The first deltas of a 15k-page window took tens of seconds to arrive while the client was mapping under load, so the pins still landed after the client's cuCtxCreate deadline. The scanner now reads only the first 64 KB of every 2 MB region - the pushes sit at the per-channel allocation heads - which cuts the per-loop page-table work ~50x, and it logs scan timing so the latency stays visible.
* guest-diag: vsock pin stream (guest pagemap -> host sema_map_add)Arthur Hanson Rasmusson13 days
| | | | | | | | | | The guest-agent exec path is too slow under driver-init load (a 15k-page window scan took 17-35 s to complete), so GR window pins landed after the channels' entries had been consumed and after the client's cuCtxCreate timeout. The guest now scans /proc/<pid>/pagemap locally and streams only delta runs over vsock (no agent on the hot path); the host listener pins them as they arrive. The scanner writes a pidfile so restarts do not kill the new instance.
* guest-diag: cheaper trace read, optional push sampler, bigger pagemap batchesArthur Hanson Rasmusson13 days
| | | | | | | | Under load the semfix poll grepped the whole (growing) trace file and the track_waits push block spawned three guest processes per sample, both of which slowed the pin path into the client's failure window. The trace read now tails 128 KB; track_waits.sh gates the push dump behind PUSH=0; the pagemap helper reads 32 KB chunks.
* kernel: grace window for untranslatable GR entries; tools: coalesce window ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | pin output The first GR window pin landed ~2 s after the pump consumed the freshly created channels' GPFIFO entries on the first failed read, so the pushes (and the tracking-semaphore releases they carry) were lost. GR entries with an unresolvable push page now stay pending for sema_gr_grace_ms (default 8 s) before the stale-entry consume; a pin landing in that window makes the entry readable and it executes normally. The completion counter is still published from put at the end of every pass, so the client's counter wait is not delayed. guest_pmap_win.py coalesces contiguous VA+PA pages into run lines, so a 15k-page window transfers as ~200 lines instead of ~15k and the pin batch lands within a second of the pages appearing.
* tools: pin GR push-buffer window from guest pagemap per boot; widen ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | injected-pin table The GR channel GPFIFO entries point at pushes in the client GPU VA window (observed 0x200400000 + stride, one allocation per channel) and the pump cannot read them: pushes_nomap grows and the tracking-semaphore releases they carry never land. The pushes are sysmem pages mapped into the client process, so /proc/<pid>/pagemap in the guest gives the GPA. New guest helper guest_pmap_win.py batch-reads the pagemap of a VA range (512 entries per pread). New host tool pin_gr_window.py rescans the window on an interval, follows cdt pid changes, and injects VA->PA runs through sema_map_add (same mechanism as the counters and the CE pushbuffer pins). The injected-pin table grows 8k -> 32k slots so the CE pushbuffer (4096) plus the GR window pages fit without drops.
* kernel: batch the map-add channel rewind; trust injected streams for opsArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | Two fixes for the CE data path. (1) svc_handle_map_add rewound every channel by 8 ring entries on every injected mapping; a 16 MB push-buffer pin is thousands of page injections, so the pump's last_get was pinned in place and it re-read the same entries for the whole batch (3645 submissions, zero ops). The rewind now runs once per drained batch. (2) The push-start heuristic (op=INC_METHOD at address 0) rejects UVM's real CE memop pushes, which start with an IMM method at address 8 (observed 0x60020002, PTE data words) or a release method at 0x700 (0x200381c0); a buffer read through a host-injected (ipin) translation is now decoded fully trusted, ops included. Result on the next boot: memset=2718/0/0 ringops=1794 (first sustained CE work), false 12k->1.9k. semfix_watch: bulk-pin contiguous runs via sema_map_add len, read rm_mem size, 16 MB (4096 page) cap, non-blocking QMP read (the fixed 30 ms sleep made 4096 gva2gpa calls take two minutes), and KEEP_TRACE=1 to preserve the guest trace for post-mortems.
* kernel: decode release pushes read through injected pins instead of rescuing ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | them Real CE release pushes start with INC_METHOD addr 0x700 (0x200381c0, count 3, subch 4), and svc_method_stream_plausible requires method address 0, so every readable release push was treated as a false pairing, rescued (fails) and skipped - the client's tracking-semaphore release never applied and cuCtxCreate waited forever (slot payload 74 vs queued 75). A buffer read through a host-injected (ipin) translation is now decoded release-only (allow_ops=false) when it fails the push-start heuristic; self-discovered translations keep the strict rescue path.
* kernel: authoritative host-injected VA->PA pins (ipin table) + wider pin tableArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | The CE push-buffer page 0x120000000 resolved to a zero-filled page even after sema_map_add injected the correct translation: the shared pin table silently dropped inserts once a hash cluster exceeded the 32-probe window (the page hashes to slot 0, a hot cluster), and lookups fell through to the content-scanned map and answered with a stale page - so every CE push read as implausible and memset stayed 0. Injected translations now live in a separate authoritative table checked before every other source and are never written by self-discovery; the general pin table is 4x larger with a 4x probe window and counts drops (pin_drop/ipin_drop) in the stats line. guest_kprobes.sh disables the kprobe group before removal (EBUSY) and matches p:kprobes/<name>; the guest wait sampler can run counters-only with GDB=0.
* guest-diag: deterministic channel->pool->manager->pushbuffer chain + ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | chunk-read diagnostics semfix_watch.py now resolves the CE pushbuffer from guest truth instead of scanning a wide raw window: it walks uvm_channel_t +0=pool, pool+0=manager, manager+8=pushbuffer, pushbuffer+8=rm_mem, rm_mem+0x10=vas[] and prefers the rm_mem whose GPU VA is in the 0x120000000 CE pushbuffer range. Three entry points: uvch probe on __uvm_push_begin_acquire_on_channel_with_info, uvbp probe on uvm_pushbuffer_begin_push, and the uvmt tracking-sem kptr (channel scan back from sem). Misses now print structured hop counters so a failed chunk read is distinguishable from a failed chain. guest_kprobes.sh (re)installs uvmt/uvch/uvbp/uvpbf after every guest boot (plain, module: and bare spellings).
* guest-diag: identify pushbuffer rm_mem by vmalloc CPU VA (any GPU VA)Arthur Hanson Rasmusson13 days
|
* guest-diag: widen pushbuffer chain scan (256KB, slab-prefix filter)Arthur Hanson Rasmusson13 days
|
* guest-diag: read trace before clearing (one-shot pushbuffer-create events)Arthur Hanson Rasmusson13 days
|
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-12 23:45:10 +0000