summaryrefslogtreecommitdiffstats
path: root/tools/guest-diag/semfix_watch.py
Commit message (Collapse)AuthorAge
* guest-diag: semfix_watch read64 uncached by default (stale-cache pool-pin ↵Arthur Hanson Rasmusson7 hours
| | | | | | | | | | | failure) The watcher's resolve_sem_pool read sem->page->mem->vas through a process-lifetime cache: a transient zero captured while UVM was still initialising the structures was served forever, the pool pin never landed, and the second-session client hung in cudaMallocManaged (r88 evidence: 'pool unresolved (retry)' from the watcher while the same walk succeeded from a fresh QMP client). Cache is now opt-in.
* spdx: add BSD-3-Clause identifiers to userspace/tools/scripts sourcesArthur Hanson Rasmusson4 days
| | | | | | | | | | | Sweep of the 155 tracked C/Python/shell files under userspace/ (excluding the MIT-licensed vendored userspace/extern/), tools/ and scripts/ that had no per-file license identifier. Each gains the minimal two-line header in the area's existing form: 'SPDX-License-Identifier: BSD-3-Clause' plus 'Copyright (c) 2026 Open-IOV contributors'; shebang and coding lines stay first. No other content touched. kernel/** is deliberately out of scope (recorded as a follow-up in CODESTD-REPORT-2026-10-08.md): its SPDX coverage and MODULE_LICENSE normalization need their own GPL-2.0 pass.
* tools: arm UVM kprobes deterministically (modprobe+wait+verify) and retry ↵Arthur Hanson Rasmusson4 days
| | | | | | | | | | | | | | | | | | | | | | pushbuffer pins Root cause of the cudaMalloc/UVM-init stall on our bring-up path: guest_kprobes.sh ran while nvidia_uvm was not yet loaded (it loads lazily with the first CUDA client), so every probe add failed silently (symbols not in kallsyms) and the host never got the #sem -> pushbuffer-pin pipeline. The CE push page (0x120021db8) then never translated and the guest spun forever at DEVICE. - guest_kprobes.sh: modprobe nvidia_uvm, wait (bounded 30 s) for the uvmt symbol, install, and print KP_ARMED/KP_FAIL for bring-up checks. - semfix_watch.pin_pushbuffer: retry transient/partial gva2gpa misses in-call (bounded passes with a short sleep) instead of aborting on the first all-miss block; report unresolved page counts. Validated: 2/2 cold boots (vmkeep6/7) arm the probes and run victory_mail green in 15-20 s (MAIL + VADD bad=0 checksum=12204516 + STATUS OK, oracle-matched); pre-fix boots with unarmed probes stalled 4/4.
* kernel: GR context-promote bisection modes 4-17 and server-managed-ctxbuf ↵Arthur Hanson Rasmusson10 days
| | | | | | | | | | | | | | | | | | | | registry variant; complete vgsp rename; docs/tooling refresh - gsp_phys_booter: GR promote bisection modes 4-17 (two-phase zero-field VA, batched entries, hVirtMemory, bare VA, KMD SKIP_CTXBUFFER_ALLOC channel, old-interface GPU_INITIALIZE_CTX, FBMEM context buffer, UVM-faithful PA-only, PATCH/PM auxiliary promotes, server-managed no-promote path); PATCH/PM geometry queries and buffers; pre-boot SET_REGISTRY profile selection - gsp_phys_gpu: gsp_phys_rmcall_gr_promote clamp 0..17 and gsp_phys_srv_ctxbuf parameter; registry image variant with RMSetClientRMAllocatedCtxBuffer=0 - vgsp rename across kernel, scripts, tools and docs (module vgsp.ko, /sys/kernel/debug/vgsp/, vgsp-<subsystem>: log prefixes) - risc-v-gsp-rm-debugger user-facing rename and docs/CI refresh; guest-diag tool updates; README/SBOM/install/Makefile sync Live status: CE ladder and P6 guest-copy forwarding green; GR engine context commit still open (all promote shapes accepted or cleanly refused; the channel stays CTXNOTVALID) - see Handoff-vGPU-IO.md Addendum 21 (local).
* guest-diag: cycle monitor CPUs on kernel-VA translation failureArthur Hanson Rasmusson12 days
| | | | | | | | | HMP x/gva2gpa translate with the selected vCPU's CR3; while that CPU runs user code under PTI the kernel half is hidden, so kernel-VA reads (the tracking-sem chain and the CE pushbuffer CPU alias) fail and the pin pass aborts. Try the other vCPUs before giving up: during an active CUDA client at least one CPU is in kernel mode, so the chain/pin work no longer require the client to be parked in sleep().
* guest-diag: pushbuffer pin survives PTI holes (full-range scan + bounded ↵Arthur Hanson Rasmusson12 days
| | | | retries until complete)
* sema-svc: apply releases before push memsets; poller: bypass read cacheArthur Hanson Rasmusson13 days
|
* guest-diag: print tracking-sem state (index, queued, payload) on changeArthur Hanson Rasmusson13 days
| | | | | | | The final cuCtxCreate stall is a UVM tracking semaphore whose completed value stays 0; the pool payload is the ground truth. semfix now prints idx/+0x10/+0x28/payload for every tracking sem it sees, whenever a field changes, so the last stall before a failure is visible in its log.
* guest-diag: cheaper trace read, optional push sampler, bigger pagemap batchesArthur Hanson Rasmusson13 days
| | | | | | | | Under load the semfix poll grepped the whole (growing) trace file and the track_waits push block spawned three guest processes per sample, both of which slowed the pin path into the client's failure window. The trace read now tails 128 KB; track_waits.sh gates the push dump behind PUSH=0; the pagemap helper reads 32 KB chunks.
* kernel: batch the map-add channel rewind; trust injected streams for opsArthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | | | Two fixes for the CE data path. (1) svc_handle_map_add rewound every channel by 8 ring entries on every injected mapping; a 16 MB push-buffer pin is thousands of page injections, so the pump's last_get was pinned in place and it re-read the same entries for the whole batch (3645 submissions, zero ops). The rewind now runs once per drained batch. (2) The push-start heuristic (op=INC_METHOD at address 0) rejects UVM's real CE memop pushes, which start with an IMM method at address 8 (observed 0x60020002, PTE data words) or a release method at 0x700 (0x200381c0); a buffer read through a host-injected (ipin) translation is now decoded fully trusted, ops included. Result on the next boot: memset=2718/0/0 ringops=1794 (first sustained CE work), false 12k->1.9k. semfix_watch: bulk-pin contiguous runs via sema_map_add len, read rm_mem size, 16 MB (4096 page) cap, non-blocking QMP read (the fixed 30 ms sleep made 4096 gva2gpa calls take two minutes), and KEEP_TRACE=1 to preserve the guest trace for post-mortems.
* guest-diag: deterministic channel->pool->manager->pushbuffer chain + ↵Arthur Hanson Rasmusson13 days
| | | | | | | | | | | | | | | chunk-read diagnostics semfix_watch.py now resolves the CE pushbuffer from guest truth instead of scanning a wide raw window: it walks uvm_channel_t +0=pool, pool+0=manager, manager+8=pushbuffer, pushbuffer+8=rm_mem, rm_mem+0x10=vas[] and prefers the rm_mem whose GPU VA is in the 0x120000000 CE pushbuffer range. Three entry points: uvch probe on __uvm_push_begin_acquire_on_channel_with_info, uvbp probe on uvm_pushbuffer_begin_push, and the uvmt tracking-sem kptr (channel scan back from sem). Misses now print structured hop counters so a failed chunk read is distinguishable from a failed chain. guest_kprobes.sh (re)installs uvmt/uvch/uvbp/uvpbf after every guest boot (plain, module: and bare spellings).
* guest-diag: identify pushbuffer rm_mem by vmalloc CPU VA (any GPU VA)Arthur Hanson Rasmusson13 days
|
* guest-diag: widen pushbuffer chain scan (256KB, slab-prefix filter)Arthur Hanson Rasmusson13 days
|
* guest-diag: read trace before clearing (one-shot pushbuffer-create events)Arthur Hanson Rasmusson13 days
|
* guest-diag: per-page pushbuffer translation, per-client re-pinArthur Hanson Rasmusson13 days
|
* guest-diag: resolve pushbuffer from uvpbf kprobe pointer (deterministic)Arthur Hanson Rasmusson13 days
|
* guest-diag: bound pushbuffer fallback scan (400 candidates / 30s)Arthur Hanson Rasmusson13 days
|
* guest-diag: pushbuffer resolution fallback (direct pushbuffer pointer ↵Arthur Hanson Rasmusson13 days
| | | | candidates)
* guest-diag: semfix also resolves+pins the CE pushbuffer range (descending ↵Arthur Hanson Rasmusson13 days
| | | | GPA layout)
* guest-diag: semfix watcher (stale sem-pool VA->PA from guest truth); ↵Arthur Hanson Rasmusson13 days
injector marker-based handle discovery