| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
| |
failure)
The watcher's resolve_sem_pool read sem->page->mem->vas through a
process-lifetime cache: a transient zero captured while UVM was still
initialising the structures was served forever, the pool pin never
landed, and the second-session client hung in cudaMallocManaged
(r88 evidence: 'pool unresolved (retry)' from the watcher while the
same walk succeeded from a fresh QMP client). Cache is now opt-in.
|
| |
|
|
|
|
|
|
|
| |
Two failure modes found when the scanner detects clients by device fd:
a transient boot nvidia-smi was picked and its exited pagemap read back
EOF, and scan_window advanced by got=0 forever (busy spin, no stream);
and non-client nvidia fd holders must never be committed to. Break on
got<=0 in scan_window (scan_range already did), skip nvidia-smi/
persistenced/Xorg, and prefer /dev/nvidia-uvm holders.
|
| |
|
|
|
|
|
|
|
|
| |
The guest pin scanner only tracked processes whose comm matched
CLIENT_PREFIXES (cdt,va,victory). An arbitrary CUDA binary (e.g.
/tmp/smiley) was invisible: no pages streamed, no VA->PA pins injected,
the service could not read its pushes and the client hung in
cudaMalloc/uvm_tracker_wait. Detect compute clients by the CUDA device
they hold open (/dev/nvidia-uvm preferred, then any /dev/nvidia*), with
the prefix match kept as the first-choice path.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The launch line carried both an explicit rtl8139 on netdev net0 and a
stray '-net user,hostfwd=tcp::2222-:22 -net nic' pair, creating a SECOND
slirp network with a second NIC and a second 2222 forward. The SSH
connection landed on one slirp instance while the guest's replies left
through the other interface, stalling the handshake before auth. The
explicit rtl8139 now owns the only user netdev with the 2222 forward.
The generated launch_qemu_rootport.run.sh is patched in place by the
harness from the same line.
Verified: sshpass -p ubuntu ssh -p 2222 ubuntu@127.0.0.1 -> nvidia-smi
green. Guest-side companion fix (persisted in the qcow2 image): the
cloud image's /etc/ssh/sshd_config.d/60-cloudimg-settings.conf set
PasswordAuthentication no; it is now yes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(ISSUE-0002)
svc_chan_userd now prefers the layout-derived CE USERD
(ring_pa + 0x10000 in the ring aperture) and accepts it while idle, so
a second session cannot latch onto the registered pool-relative offset
when that alias holds a DIFFERENT rank's stale USERD: r40 c2 rank0 read
rank3's stale 1/1 at FB 0xf0000 and the pump never followed rank0's own
GP_PUT (guest GPUT mapping for gpFifoGpuVa proves ring+0x10000).
guest_pin_stream.va_to_pa drops translations outside the registered
guest-RAM windows so BAR1-phys addresses (0x70000xxxxx) are no longer
injected as SYS pins; r40 rank2's ring resolved to the bogus
pa=0x70000c0000 ap=SYS and read 66/66.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
#put (ISSUE-0002 diag)
CE channels register gpfifo=0x121010000+0x30000*rank, outside the
GR-only window [0x200200000,0x200400000) the #put resolver searched, so
no #put line ever fired for the CE channels the second session stalls
on. Resolve channel_info from both windows with layout validation, keep
re-resolving the channel after the pool mem has resolved, reset the
#put map on client switch, and add bounded resolver diagnostics to the
guest log.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Sweep of the 155 tracked C/Python/shell files under userspace/ (excluding
the MIT-licensed vendored userspace/extern/), tools/ and scripts/ that had
no per-file license identifier. Each gains the minimal two-line header in
the area's existing form: 'SPDX-License-Identifier: BSD-3-Clause' plus
'Copyright (c) 2026 Open-IOV contributors'; shebang and coding lines stay
first. No other content touched. kernel/** is deliberately out of scope
(recorded as a follow-up in CODESTD-REPORT-2026-10-08.md): its SPDX
coverage and MODULE_LICENSE normalization need their own GPL-2.0 pass.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
window
The guest's first RmInitAdapter intermittently fails: GspStatusQueueInit
gives the status-queue link only 4 s (message_queue_cpu.c: timeoutUs =
4000000) and the first boot attempt can burn it, after which the RM tears
down, leaks the device and retries; the surviving retry leaves the GR/ctx
state degraded, which is the persistent cudaMalloc stall class.
Changes:
- gsp_rm_emu_poll: parse-retry cadence 250 ms -> 50 ms (the 250 ms tick
consumed a fifth of the guest's budget per attempt).
- gsp_rm_emu_reset: clear last_parse_retry so a new boot gets a fresh
retry budget; take stat_lock while clearing queue cursors so a sema
event post cannot race the reset.
- gsp_rm_emu_on_booter_done: on GSP_INIT_DONE send failure re-arm the
retry path (parsed/active were already set, so the poll loop would
never retry).
- gsp_rm_parse_queue_chain: free the previous ptes before re-alloc so
retries and post-reset re-parses do not leak.
- gsp_rm_post_channel_event: never post events across a queue reset
(parsed gate).
guest_kprobes.sh: add rpcs/rpcp transport probes (the direct RPC-wait
symbol is notrace in 570.144) and ccset entry/return probes for
rmapiControlCacheSet so a stall names the control whose reply breaks the
guest's control-cache bookkeeping.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ring+userd) + GR ring userd-derived candidates
guest_pin_stream.py now resolves translations entirely from guest truth via
/proc/kcore + a manual page-table walk (no QMP races): the CE semaphore pool
and pushbuffer memory from the UVM rm_mem structures, and each channel's
GPFIFO ring + GPPUT from the tracking semaphore's embedding uvm_channel_t
(channel_info located by its gpFifoGpuVa value). This removes the watcher
pipeline's flakiness on the cudaMalloc stall class (missing pool/pushbuffer
pins).
gsp_sema_service.c: svc_gr_ring_autopin tries the guest-registered USERD
memdesc base as a ring candidate (the fixed FB convention only matches when
the guest's VRAM allocator happens to reproduce the old layout), plus an
optional (gsp_sema_dbg) one-shot FB scan + raw-word diagnostics when every
candidate fails.
tools/cuda-ab/p2: self-checking arbitrary-kernel acceptance suite (10
programs: sequential launches, shapes, params, smem/GEMM, streams, malloc
churn, printf+error, FMA benchmark, mixed, managed probe) with host oracle
expectations and a guest runner.
|
| |
|
|
|
|
|
|
| |
The hostfwd tcp::2222-:22 sat on the rtl8139 NIC (net0, guest enp0s7,
DOWN); the guest networks over the legacy -net user NIC (enp0s2,
10.0.2.15), so SSH to the guest timed out during banner exchange. Move
the forward to the legacy NIC (net0 loses it) - ssh -p 2222 now reaches
the guest and key auth works.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
second-client stall race)
The guest scanner's TraceReader tailed the last 256 KB of the trace ring
every 0.3 s. A one-shot uvmt sem= event could age out of that window
before the next poll (the client's spin floods the ring), so a second
client in the same VM was never discovered: semfix re-pinned the previous
client's freed sem while the new client's push page stayed untranslated
(host nomap, drain held, guest spins forever).
- guest_pin_stream.py: consume /sys/kernel/tracing/trace_pipe instead
(each event delivered exactly once; no ring-window race), with
reconnect on read error. The old ring-tail read is kept only as
semfix's fallback.
- clear the trace ring on every client switch so the fallback cannot
re-learn the previous client's stale events.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pushbuffer pins
Root cause of the cudaMalloc/UVM-init stall on our bring-up path:
guest_kprobes.sh ran while nvidia_uvm was not yet loaded (it loads lazily
with the first CUDA client), so every probe add failed silently (symbols
not in kallsyms) and the host never got the #sem -> pushbuffer-pin
pipeline. The CE push page (0x120021db8) then never translated and the
guest spun forever at DEVICE.
- guest_kprobes.sh: modprobe nvidia_uvm, wait (bounded 30 s) for the
uvmt symbol, install, and print KP_ARMED/KP_FAIL for bring-up checks.
- semfix_watch.pin_pushbuffer: retry transient/partial gva2gpa misses
in-call (bounded passes with a short sleep) instead of aborting on the
first all-miss block; report unresolved page counts.
Validated: 2/2 cold boots (vmkeep6/7) arm the probes and run
victory_mail green in 15-20 s (MAIL + VADD bad=0 checksum=12204516 +
STATUS OK, oracle-matched); pre-fix boots with unarmed probes stalled
4/4.
|
| |
|
|
|
|
|
|
| |
- kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean
- tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests
- tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode
- debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding
- STATE-CURRENT/docs updates
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
registry variant; complete vgsp rename; docs/tooling refresh
- gsp_phys_booter: GR promote bisection modes 4-17 (two-phase zero-field VA,
batched entries, hVirtMemory, bare VA, KMD SKIP_CTXBUFFER_ALLOC channel,
old-interface GPU_INITIALIZE_CTX, FBMEM context buffer, UVM-faithful PA-only,
PATCH/PM auxiliary promotes, server-managed no-promote path); PATCH/PM
geometry queries and buffers; pre-boot SET_REGISTRY profile selection
- gsp_phys_gpu: gsp_phys_rmcall_gr_promote clamp 0..17 and gsp_phys_srv_ctxbuf
parameter; registry image variant with RMSetClientRMAllocatedCtxBuffer=0
- vgsp rename across kernel, scripts, tools and docs (module vgsp.ko,
/sys/kernel/debug/vgsp/, vgsp-<subsystem>: log prefixes)
- risc-v-gsp-rm-debugger user-facing rename and docs/CI refresh; guest-diag
tool updates; README/SBOM/install/Makefile sync
Live status: CE ladder and P6 guest-copy forwarding green; GR engine context
commit still open (all promote shapes accepted or cleanly refused; the channel
stays CTXNOTVALID) - see Handoff-vGPU-IO.md Addendum 21 (local).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- file-backed guest RAM (launcher memory-backend-file share=on) and
in-kernel GPA->HPA translation through the shared file page cache
(read_cache_page + page_to_phys + pfn_valid), opened O_LARGEFILE;
new default-off gsp_phys_guest_mem parameter
- debugfs gsp-phys-guest-ram (path + gpa/off/size ranges) and
gsp-phys-guest-gpa (one pinned GPA per boot); rm-call triggers 6/7 run
host->guest and guest->host 4 KiB CE copies; trigger 8 is a PTE
self-test through a fresh slot
- guest page-table slots are pre-built during ladder stage 3: runtime
PTE writes into an already-walked page-table line are not observed
without a TLB invalidate
- guest RAM layout (q35 4.2, 4 GiB): below-4G GPA == file offset,
above-4G offset = GPA - 0x80000000 (2 GiB PCI hole)
- tests: tools/test/gpage_test.c guest probe; docs + STATE-CURRENT
record the live green (G2H/W2G with semaphore release, guest observed
the write, smoke ALL OK 64/64, clean latched release)
|
| |
|
|
| |
third-party references
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Host-freeze fix bundle (validated 2/2 supervised cycles):
- power unlock now latches D0/D3cold by default (new gsp_phys_repower,
default 0); releasing the claim can no longer power-gate the board and
hard-freeze the host on resume
- bounded post-rebind readiness gate before dropping the locks; bounded D0
settle/verify before the unbind
- READY means a live GSP-RM session: the async worker and the module-owned
join only expose READY when init-done is seen and no unload was sent;
otherwise the claim is released fail-open
- FWSEC reads DMEM0/IMEM back before STARTCPU and fails fast when the
staging is not visible
- the claim harness leaves the board latched by default and takes
--restore-power to opt back into the old restore
Guest-wiring:
- P2: generation-keyed fn65 static-config snapshot cache plus a default-off
identity overlay (new gsp_phys_static_live) for name/short-name/VBIOS;
the probe cycle runs once per backend generation at READY
- P3: default-off read-only RM_CONTROL forwarding whitelist (new
gsp_phys_ctrl_forward) with emulated-handle translation to the adopted
server handles and full emulation fallback
- P5 wave 1: default-off shadow execution of guest CE pushes on the ladder
channel (new gsp_phys_work_forward) via a dedicated ordered work item;
the ladder stage-4 submission is refactored into a shared bounded helper
and the software release path stays authoritative
Live: module-owned boot to init-done, guest smoke ALL OK 64/64 with P2/P3
enabled, 514 shadow copies observed on the physical CE; all cycles released
cleanly with WPR2 down and no reboot.
|
| |
|
|
| |
harness
|
| | |
|
| | |
|
| |
|
|
| |
--boot is the only device-writing mode (full boot lifecycle: FWSEC, Booter Load, RISC-V start, FALCON_OS, RM-init poll). --boot-dryrun rehearses the host staging read-only. --keep leaves a green claimed session running. --fwsec is retired and refused at parse time: a FWSEC-only carve is not host-recoverable on this part.
|
| | |
|
| | |
|
| |
|
|
| |
abort checks)
|
| |
|
|
|
|
|
|
| |
Reproducible TASK 2 harness: fresh boot, sema_dbg=Y with
sema_copy_dst=0x200000 / sema_copy_dst2=0x208000, cdt2 parked at
CDT_CP_PAUSE so the root PD/PT pages can be dumped and walk_probe run
while the client is live. Prints the offending write data (prov= names
the translation layer) and the post-run walk state.
|
| |
|
|
|
|
|
|
|
| |
The DtoH staging buffer lives in the client's high-VA /dev/zero|nvidia
mappings (e.g. devptr+0x3c00100), outside the scanner's fixed CE/GR
windows; only the host's pagemap read can see it. The guest scanner
now walks those VMAs and streams their present pages so the module
pins them. run_cycle.sh learns the sema_dbg master switch (auto-on
when any watch knob is set).
|
| |
|
|
| |
failure left a boot without pins and the client hung)
|
| |
|
|
|
|
| |
A fresh boot can leave /sys/kernel/tracing as an unmounted mount point;
the probe adds then fail silently (probe count 0) and the whole sem/kptr
pipeline goes blind.
|
| |
|
|
|
|
|
| |
A new CUDA client allocates fresh channel pools at the same GPU VAs; the
watcher kept the previous client's sem kptrs and seen-pin marks, so it
pinned the previous (freed) pool pages and the new client's release writes
went to dead memory. Clear the learned sems/marks on each pid change.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
HMP x/gva2gpa translate with the selected vCPU's CR3; while that CPU runs
user code under PTI the kernel half is hidden, so kernel-VA reads (the
tracking-sem chain and the CE pushbuffer CPU alias) fail and the pin pass
aborts. Try the other vCPUs before giving up: during an active CUDA client
at least one CPU is in kernel mode, so the chain/pin work no longer require
the client to be parked in sleep().
|
| |
|
|
|
|
|
| |
The GR-window pin scanner matched comm=="cdt" exactly, so runs using the
cuda_drv_test2 binary (comm "cdt2") never streamed their mappings: the
GR push pages stayed unresolvable and the client stalled in context
creation. Match the cdt prefix instead.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA
submits it on the compute channel as a compute-class DMA whose source is
the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>,
LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's
SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run
(opcode 3) and no GR work ran at all.
- Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT,
OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run.
- svc_grce_inline_write resolves the destination VA through trusted
sources (pins, map, published page directories, targeted rm_mem
resolve), pins the page and writes into vidmem or guest RAM; budgeted
sema-grce logs name the target or report UNRESOLVED.
- cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection
of the copy path; pushed by push_guest_tools.sh.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
payload-pattern audit
- CE LAUNCH_DMA releases are now applied after the push's collected work
runs: the client reads copied data as soon as it observes the release,
so release-first let cuMemcpyDtoH observe a copy that had not executed.
- Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms,
default 8s) before the release fallback consumes them, so a translation
that lands within a service pass lets the push be decoded and its work
executed instead of being dropped.
- sema_copy_pat matches an 8-byte value against every copy's src/dst first
word and scans decoded pushes for it, naming the submission that carries
a known payload regardless of size/encoding.
- sema-dma-skip logs transfer-shaped launches that collect no op; op log
lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG.
- A real release application now records the release VA, so decoded pushes
no longer trigger a spurious fake release.
|
| |
|
|
| |
CDT_WAIT passthrough
|
| |
|
|
| |
retries until complete)
|
| | |
|
| |
|
|
| |
run_cycle OPS_LOG knob
|
| |
|
|
| |
window has no CPU alias; the vsock stream cannot supply it)
|
| |
|
|
| |
roundtrip diagnostics
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
call
|
| |
|
|
| |
validation
|
| | |
|
| | |
|
