| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
window
The guest's first RmInitAdapter intermittently fails: GspStatusQueueInit
gives the status-queue link only 4 s (message_queue_cpu.c: timeoutUs =
4000000) and the first boot attempt can burn it, after which the RM tears
down, leaks the device and retries; the surviving retry leaves the GR/ctx
state degraded, which is the persistent cudaMalloc stall class.
Changes:
- gsp_rm_emu_poll: parse-retry cadence 250 ms -> 50 ms (the 250 ms tick
consumed a fifth of the guest's budget per attempt).
- gsp_rm_emu_reset: clear last_parse_retry so a new boot gets a fresh
retry budget; take stat_lock while clearing queue cursors so a sema
event post cannot race the reset.
- gsp_rm_emu_on_booter_done: on GSP_INIT_DONE send failure re-arm the
retry path (parsed/active were already set, so the poll loop would
never retry).
- gsp_rm_parse_queue_chain: free the previous ptes before re-alloc so
retries and post-reset re-parses do not leak.
- gsp_rm_post_channel_event: never post events across a queue reset
(parsed gate).
guest_kprobes.sh: add rpcs/rpcp transport probes (the direct RPC-wait
symbol is notrace in 570.144) and ccset entry/return probes for
rmapiControlCacheSet so a stall names the control whose reply breaks the
guest's control-cache bookkeeping.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ring+userd) + GR ring userd-derived candidates
guest_pin_stream.py now resolves translations entirely from guest truth via
/proc/kcore + a manual page-table walk (no QMP races): the CE semaphore pool
and pushbuffer memory from the UVM rm_mem structures, and each channel's
GPFIFO ring + GPPUT from the tracking semaphore's embedding uvm_channel_t
(channel_info located by its gpFifoGpuVa value). This removes the watcher
pipeline's flakiness on the cudaMalloc stall class (missing pool/pushbuffer
pins).
gsp_sema_service.c: svc_gr_ring_autopin tries the guest-registered USERD
memdesc base as a ring candidate (the fixed FB convention only matches when
the guest's VRAM allocator happens to reproduce the old layout), plus an
optional (gsp_sema_dbg) one-shot FB scan + raw-word diagnostics when every
candidate fails.
tools/cuda-ab/p2: self-checking arbitrary-kernel acceptance suite (10
programs: sequential launches, shapes, params, smem/GEMM, streams, malloc
churn, printf+error, FMA benchmark, mixed, managed probe) with host oracle
expectations and a guest runner.
|
| |
|
|
|
|
|
|
| |
The hostfwd tcp::2222-:22 sat on the rtl8139 NIC (net0, guest enp0s7,
DOWN); the guest networks over the legacy -net user NIC (enp0s2,
10.0.2.15), so SSH to the guest timed out during banner exchange. Move
the forward to the legacy NIC (net0 loses it) - ssh -p 2222 now reaches
the guest and key auth works.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
second-client stall race)
The guest scanner's TraceReader tailed the last 256 KB of the trace ring
every 0.3 s. A one-shot uvmt sem= event could age out of that window
before the next poll (the client's spin floods the ring), so a second
client in the same VM was never discovered: semfix re-pinned the previous
client's freed sem while the new client's push page stayed untranslated
(host nomap, drain held, guest spins forever).
- guest_pin_stream.py: consume /sys/kernel/tracing/trace_pipe instead
(each event delivered exactly once; no ring-window race), with
reconnect on read error. The old ring-tail read is kept only as
semfix's fallback.
- clear the trace ring on every client switch so the fallback cannot
re-learn the previous client's stale events.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pushbuffer pins
Root cause of the cudaMalloc/UVM-init stall on our bring-up path:
guest_kprobes.sh ran while nvidia_uvm was not yet loaded (it loads lazily
with the first CUDA client), so every probe add failed silently (symbols
not in kallsyms) and the host never got the #sem -> pushbuffer-pin
pipeline. The CE push page (0x120021db8) then never translated and the
guest spun forever at DEVICE.
- guest_kprobes.sh: modprobe nvidia_uvm, wait (bounded 30 s) for the
uvmt symbol, install, and print KP_ARMED/KP_FAIL for bring-up checks.
- semfix_watch.pin_pushbuffer: retry transient/partial gva2gpa misses
in-call (bounded passes with a short sleep) instead of aborting on the
first all-miss block; report unresolved page counts.
Validated: 2/2 cold boots (vmkeep6/7) arm the probes and run
victory_mail green in 15-20 s (MAIL + VADD bad=0 checksum=12204516 +
STATUS OK, oracle-matched); pre-fix boots with unarmed probes stalled
4/4.
|
| |
|
|
|
|
|
|
| |
- kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean
- tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests
- tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode
- debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding
- STATE-CURRENT/docs updates
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
registry variant; complete vgsp rename; docs/tooling refresh
- gsp_phys_booter: GR promote bisection modes 4-17 (two-phase zero-field VA,
batched entries, hVirtMemory, bare VA, KMD SKIP_CTXBUFFER_ALLOC channel,
old-interface GPU_INITIALIZE_CTX, FBMEM context buffer, UVM-faithful PA-only,
PATCH/PM auxiliary promotes, server-managed no-promote path); PATCH/PM
geometry queries and buffers; pre-boot SET_REGISTRY profile selection
- gsp_phys_gpu: gsp_phys_rmcall_gr_promote clamp 0..17 and gsp_phys_srv_ctxbuf
parameter; registry image variant with RMSetClientRMAllocatedCtxBuffer=0
- vgsp rename across kernel, scripts, tools and docs (module vgsp.ko,
/sys/kernel/debug/vgsp/, vgsp-<subsystem>: log prefixes)
- risc-v-gsp-rm-debugger user-facing rename and docs/CI refresh; guest-diag
tool updates; README/SBOM/install/Makefile sync
Live status: CE ladder and P6 guest-copy forwarding green; GR engine context
commit still open (all promote shapes accepted or cleanly refused; the channel
stays CTXNOTVALID) - see Handoff-vGPU-IO.md Addendum 21 (local).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- file-backed guest RAM (launcher memory-backend-file share=on) and
in-kernel GPA->HPA translation through the shared file page cache
(read_cache_page + page_to_phys + pfn_valid), opened O_LARGEFILE;
new default-off gsp_phys_guest_mem parameter
- debugfs gsp-phys-guest-ram (path + gpa/off/size ranges) and
gsp-phys-guest-gpa (one pinned GPA per boot); rm-call triggers 6/7 run
host->guest and guest->host 4 KiB CE copies; trigger 8 is a PTE
self-test through a fresh slot
- guest page-table slots are pre-built during ladder stage 3: runtime
PTE writes into an already-walked page-table line are not observed
without a TLB invalidate
- guest RAM layout (q35 4.2, 4 GiB): below-4G GPA == file offset,
above-4G offset = GPA - 0x80000000 (2 GiB PCI hole)
- tests: tools/test/gpage_test.c guest probe; docs + STATE-CURRENT
record the live green (G2H/W2G with semaphore release, guest observed
the write, smoke ALL OK 64/64, clean latched release)
|
| |
|
|
| |
third-party references
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Host-freeze fix bundle (validated 2/2 supervised cycles):
- power unlock now latches D0/D3cold by default (new gsp_phys_repower,
default 0); releasing the claim can no longer power-gate the board and
hard-freeze the host on resume
- bounded post-rebind readiness gate before dropping the locks; bounded D0
settle/verify before the unbind
- READY means a live GSP-RM session: the async worker and the module-owned
join only expose READY when init-done is seen and no unload was sent;
otherwise the claim is released fail-open
- FWSEC reads DMEM0/IMEM back before STARTCPU and fails fast when the
staging is not visible
- the claim harness leaves the board latched by default and takes
--restore-power to opt back into the old restore
Guest-wiring:
- P2: generation-keyed fn65 static-config snapshot cache plus a default-off
identity overlay (new gsp_phys_static_live) for name/short-name/VBIOS;
the probe cycle runs once per backend generation at READY
- P3: default-off read-only RM_CONTROL forwarding whitelist (new
gsp_phys_ctrl_forward) with emulated-handle translation to the adopted
server handles and full emulation fallback
- P5 wave 1: default-off shadow execution of guest CE pushes on the ladder
channel (new gsp_phys_work_forward) via a dedicated ordered work item;
the ladder stage-4 submission is refactored into a shared bounded helper
and the software release path stays authoritative
Live: module-owned boot to init-done, guest smoke ALL OK 64/64 with P2/P3
enabled, 514 shadow copies observed on the physical CE; all cycles released
cleanly with WPR2 down and no reboot.
|
| |
|
|
| |
harness
|
| | |
|
| | |
|
| |
|
|
| |
--boot is the only device-writing mode (full boot lifecycle: FWSEC, Booter Load, RISC-V start, FALCON_OS, RM-init poll). --boot-dryrun rehearses the host staging read-only. --keep leaves a green claimed session running. --fwsec is retired and refused at parse time: a FWSEC-only carve is not host-recoverable on this part.
|
| | |
|
| | |
|
| |
|
|
| |
abort checks)
|
| |
|
|
|
|
|
|
| |
Reproducible TASK 2 harness: fresh boot, sema_dbg=Y with
sema_copy_dst=0x200000 / sema_copy_dst2=0x208000, cdt2 parked at
CDT_CP_PAUSE so the root PD/PT pages can be dumped and walk_probe run
while the client is live. Prints the offending write data (prov= names
the translation layer) and the post-run walk state.
|
| |
|
|
|
|
|
|
|
| |
The DtoH staging buffer lives in the client's high-VA /dev/zero|nvidia
mappings (e.g. devptr+0x3c00100), outside the scanner's fixed CE/GR
windows; only the host's pagemap read can see it. The guest scanner
now walks those VMAs and streams their present pages so the module
pins them. run_cycle.sh learns the sema_dbg master switch (auto-on
when any watch knob is set).
|
| |
|
|
| |
failure left a boot without pins and the client hung)
|
| |
|
|
|
|
| |
A fresh boot can leave /sys/kernel/tracing as an unmounted mount point;
the probe adds then fail silently (probe count 0) and the whole sem/kptr
pipeline goes blind.
|
| |
|
|
|
|
|
| |
A new CUDA client allocates fresh channel pools at the same GPU VAs; the
watcher kept the previous client's sem kptrs and seen-pin marks, so it
pinned the previous (freed) pool pages and the new client's release writes
went to dead memory. Clear the learned sems/marks on each pid change.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
HMP x/gva2gpa translate with the selected vCPU's CR3; while that CPU runs
user code under PTI the kernel half is hidden, so kernel-VA reads (the
tracking-sem chain and the CE pushbuffer CPU alias) fail and the pin pass
aborts. Try the other vCPUs before giving up: during an active CUDA client
at least one CPU is in kernel mode, so the chain/pin work no longer require
the client to be parked in sleep().
|
| |
|
|
|
|
|
| |
The GR-window pin scanner matched comm=="cdt" exactly, so runs using the
cuda_drv_test2 binary (comm "cdt2") never streamed their mappings: the
GR push pages stayed unresolvable and the client stalled in context
creation. Match the cdt prefix instead.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA
submits it on the compute channel as a compute-class DMA whose source is
the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>,
LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's
SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run
(opcode 3) and no GR work ran at all.
- Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT,
OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run.
- svc_grce_inline_write resolves the destination VA through trusted
sources (pins, map, published page directories, targeted rm_mem
resolve), pins the page and writes into vidmem or guest RAM; budgeted
sema-grce logs name the target or report UNRESOLVED.
- cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection
of the copy path; pushed by push_guest_tools.sh.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
payload-pattern audit
- CE LAUNCH_DMA releases are now applied after the push's collected work
runs: the client reads copied data as soon as it observes the release,
so release-first let cuMemcpyDtoH observe a copy that had not executed.
- Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms,
default 8s) before the release fallback consumes them, so a translation
that lands within a service pass lets the push be decoded and its work
executed instead of being dropped.
- sema_copy_pat matches an 8-byte value against every copy's src/dst first
word and scans decoded pushes for it, naming the submission that carries
a known payload regardless of size/encoding.
- sema-dma-skip logs transfer-shaped launches that collect no op; op log
lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG.
- A real release application now records the release VA, so decoded pushes
no longer trigger a spurious fake release.
|
| |
|
|
| |
CDT_WAIT passthrough
|
| |
|
|
| |
retries until complete)
|
| | |
|
| |
|
|
| |
run_cycle OPS_LOG knob
|
| |
|
|
| |
window has no CPU alias; the vsock stream cannot supply it)
|
| |
|
|
| |
roundtrip diagnostics
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
call
|
| |
|
|
| |
validation
|
| | |
|
| | |
|
| |
|
|
| |
callback)
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Run #7 proved the membar-WAR descriptor lookup succeeds in the guest
(remap (7,5)->(7,0)); FUN_00e21850 then returns 1. bpz8 breaks after its
two inner calls (FUN_003e99d0 image create, FUN_003dfae0 image load) to name
which one fails.
|
| |
|
|
|
|
|
|
| |
Host oracle (real TU104, 570.211): FUN_0035a4d0(0x13,7,5) succeeds because
the 3-entry remap table at 0x3dc5b80 maps (7,5)->(7,0); the guest has the
same table. bpz7 breaks before/after the lookup and the FUN_00e21850 ucode
load in FUN_005d7e70 to see which one fails in the guest. Also fix invd.py's
lookup address (0x25a4d0, not 0x35a4d0).
|
| |
|
|
|
|
|
|
|
| |
Run #5 named the failing hook: idx=3 FUN_005d7e70 ('membar_war_bug2022401')
returned 999; its only source is FUN_0035a4d0(0x13, dev+0xc50, dev+0xc54)
returning NULL or the ucode load failing. bpz6 reads the live device state
arch/impl at the failure and evaluates the lookup in-process, scanning
(arch,rev) pairs to show which WAR-ucode descriptors exist; invd.py is the
standalone same-probe for a parked cdt.
|
