| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
GSP-RM init
The pre-boot SET_REGISTRY (function 73) submitted an empty
PACKED_REGISTRY_TABLE. Replace it with the default registry profile the
host driver sends before boot: the 35 default dword parameters from its
parameter table plus RmDisableDisplay=1. The display key makes the GSP-RM
display-presence check treat display as absent, so the display engine is
removed during pre-init instead of being brought up on a headless boot.
The payload is generated into kernel/gsp_phys_registry_img.h as a
PACKED_REGISTRY_TABLE blob (8-byte header, 36 16-byte dword entries,
NUL-terminated names after the array; values inline; no value bytes in
the name blob, matching the reference packer): size 1342, numEntries 36,
rpc.length 1374, one 4 KB element. fn72 is unchanged.
The payload length is not 8-byte aligned (record 1390 -> 1392); the TX
producer already zero-pads the kvzalloc'd staging record and folds the
XOR checksum over the padded length, exactly like the reference producer
zero-pads before _checkSum32, so no producer change is needed. The blob
is static const data: no heap, no new stack frame, and a submit failure
still aborts the boot. The prepared log line now prints the payload size
and entry count.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The real-GSP boot path staged the message queue but never queued the two
fire-and-forget init RPCs the production client places in it before the
booter starts. GSP-RM therefore created its GPU object without any system
information and aborted RM init (live GSP_INIT_DONE rpc_result=0x59), with
the resource-manager's object-database lookups asserting from the first
journal record.
Submit function 72 (GSP_SET_SYSTEM_INFO, 928-byte v17_00 payload carrying
the BAR physical bases, domain/bus/slot, PCI IDs, the BAR0 config-mirror
offset, link/FLR capabilities and upstream-port flags read from the
pci_dev) and function 73 (SET_REGISTRY, header-only empty table, which the
server explicitly accepts) through the existing submit-only queue producer
after all stage-3d preconditions and before the GSP falcon reset/Booter
Load kick. The wire layouts are re-declared in the module with per-field
static asserts against the reference ABI; payloads are heap-allocated, so
no new stack frames are introduced.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
A zero REG_POLL timeout dword is the reference's GPU_TIMEOUT_DEFAULT
sentinel: the reference substitutes the platform default (microseconds;
4 s in graphics mode, the default platform) before waiting. The executor
clamped zero to the 1 us floor, i.e. one jiffy, so a poll that was not
already satisfied failed the boot in 1 ms.
The live 0x1002 sequencer element contains exactly one REG_POLL with
timeout=0 (addr 0x110040); substitute the 4 s graphics-mode default and
then keep the existing 1 us .. 10 s clamp.
|
| |
|
|
|
|
|
|
|
| |
The live 0x1002 element carries bufferSizeDWord=0x3FE2 (16354) - the size
of GSP-RM's own command buffer, not the message. The executor's 0x1000
sanity cap refused it, so the sequencer was never run and GSP_INIT_DONE
was never observed. The reference imposes no upper bound (only nonzero and
greater than cmdIndex); raise the cap to a sanity ceiling and document
that the real execution bound is cmdIndex against the element payload.
|
| |
|
|
|
|
| |
Applies the adversarial review fixes for the command-queue TX and unload RPC: the unload RPC is gated on GSP-RM init-done, debugfs fops pin the module, REG_POLL timeouts use microseconds, the host status readPtr is published, TX free space is checked, swap flags and barriers are corrected, and the legacy payload fallback is opt-in.
Generalizes the transport to multi-element records and a reply-matching status-queue reader, and adds a debugfs probe that round-trips GET_GSP_STATIC_INFO (with a SET_GUEST_SYSTEM_INFO fallback), decodes the server configuration, and a read-only state snapshot file. All gated by gsp_phys_boot; the normal load path is unchanged.
|
| |
|
|
| |
Adds a minimal msgq producer on the command queue (element layout, checksum, sequence, dma_wmb before writePtr, doorbell) and submits the UNLOADING_GUEST_DRIVER RPC so GSP-RM shuts down and sets the scratch14 handoff bit the Booter Unload gate needs. Release attempts the RPC once before quiescing when a boot ran; on-demand testing is available through the debugfs file gsp-phys-unload-rpc while the module owns the GPU. GSP-RM processor suspend is polled on the GSP mailbox.
|
| |
|
|
|
|
|
|
| |
Stage 3d stages the GSP-RM boot from host sysmem (FMC, radix3 over the GSP-RM ELF, signature, message queues, libos LOGINIT/LOGRM/RMARGS and GSP arguments) and runs Booter Load on SEC2: GSP RISC-V reset, libos mailbox, IMEM/DMEM direct load, BOOTVEC=0, M0==0, FALCON_OS, RISC-V active. Stage 3e links the status queue and polls for GSP-RM events; it now also executes the inline run-sequencer command buffer (event 0x1002) against BAR0 (REG_WRITE/MODIFY/POLL, DELAY, STORE, CORE_* including CORE_RESUME) and then waits for GSP_INIT_DONE 0x1001.
Gated by gsp_phys_boot / gsp_phys_boot_dryrun (default off; the normal load path writes nothing). gsp_phys_fwsec=1 without gsp_phys_boot=1 is refused: a FWSEC-only carve cannot pass the booter-unload handoff gate. Release quiesces the GSP, runs the existing FWSEC-SB/booter-unload teardown and frees all staging.
tools/gen_gsp_bindata.py decodes the public generated archives into the Booter Load image (prod signature patched at 0x8700) and the GSP-RM boot (FMC) image; both are self-checked at build time.
|
| | |
|
| |
|
|
| |
release
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
bind-event hardening)
|
| |
|
|
| |
strict board check)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stage 3b allocates the three coherent boot-argument pages and fills the
256-byte WPR metadata from the computed Turing FB layout; no device
registers are written (uncommitted work from the previous session,
committed as-is behind the new gate).
The physical claim unbinds the distribution NVIDIA driver; on this host
that power-gates the dGPU board and the re-bind can leave the GPU
unresponsive and hard-freeze the system (observed twice, 2026-09-30, no
crash record). The claim now refuses unless the new gsp_phys_hazard_ack
module parameter is also set, so a stale command line cannot re-trigger
it. See the hazard note at the top of gsp_phys_gpu.c.
|
| |
|
|
| |
(no device writes)
|
| |
|
|
|
|
|
| |
Logs both falcons' run/halt bits, the RISC-V active bit, WPR2 up/down and
the security-falcon mailboxes when the physical GPU is claimed. Verified
live: GSP and SEC2 halted, RISC-V inactive, WPR2 down (HI=0). This is the
read-only half of the real boot rung; no device state is written.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rung 1 (ownership): a default-off gsp_phys_gpu module param; on claim the
distribution driver is released from 10de:1ed3, our pci_driver binds it,
BAR0/1/3 are mapped and the falcon state is sampled; release unbinds ours
and rebinds the distribution driver. Verified live: claim reads
BOOT0=0x164000a1 with the falcon halted and mailboxes clear; rmmod rebinds
nvidia and nvidia-smi stays healthy.
Rung 2 (firmware): while the device is owned, gsp_phys_fw (default
nvidia/570.211.01/gsp_tu10x.bin) is fetched with request_firmware, checked
as ELF64 RISC-V, and its sections are logged (.fwimage, .fwversion,
.fwsignature_tu10x). The image is held for the boot rung.
The claim is default-off; the emulation path is unchanged.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The per-put-epoch consumed-slot skip must not suppress a slot whose entry
was consumed *without* delivering its release (the CE grace expiry in real
mode): the driver-API smoke test's CE pushes are consumed unreadable before
their ring pin lands and depend on the pin-batch rewind replaying them once
the translation appears. The skip now applies only to slots whose
consumption delivered a release:
- tail consumption records a slot when the iteration decoded a release
(svc_rel_count delta); the sweep-delivered entry_want consume and the
fake-release consume always record
- the grace-expiry consume records only when a recovery (fake or
sema_rel_recover) delivered the release
- svc_fake_release counts as a delivered release
So real-mode replays still re-process undelivered entries, while the
runtime-client case (readable pushes delivered immediately) keeps the
fast path that removed the 8 s stalls.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The pump decided "this push carried a release" by comparing the last
decoded release VA before/after svc_decode_stream. Consecutive pushes on a
channel release to the SAME tracking-semaphore VA (UVM's tracker pattern),
so every push after the first read as release-less: the entry was held for
the full CE grace window (observed: exactly one push consumed per 8 s,
gating the whole runtime init; the release itself had already been applied
by the decode, only the pump's bookkeeping stalled).
svc_rel_count now increments on every decoded release (host sem block,
REPORT_SEM block, CE launch-DMA release) and the pump compares counts at
all three sites (initial decode, rescue re-route, rescued-page retry).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A pin-batch rewind replays recently consumed entries. An old entry whose
rescue cannot validate any candidate then sits in the CE grace path - and
because the client's put has advanced since, the replayed entry no longer
matches any "new" expectation: it waits the full 8 s grace, blocking every
later entry behind it on that channel. Observed live: the first push of
rank 0 re-processed and skipped twice, 38 s apart, while eight later pushes
queued behind it.
The pump now marks each consumed slot in a per-channel bitmap valid for the
current put epoch (cleared when put wraps, so a recycled slot in the next
epoch is processed normally) and consumes a replayed slot immediately.
Rewinds still refresh readable entries; they simply no longer resurrect
entries that were already handled.
Evidence: real-release watcher cycle, replay-after-put-advance no longer
produces 'skipping stale entry ... after 8000 ms grace' for old slots.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two stalls serialized the runtime client's init into one 8 s CE grace window
per push:
- a pin-batch rewind replays recently consumed GPFIFO slots; re-examining a
consumed slot runs the rescue and then holds it for the full grace window
again (observed: the same VA 'skipped after 8000 ms grace' twice, 38 s
apart). The pump now records each consumed slot with the put value it was
consumed at (32-entry per-channel replay ring, persisted across passes)
and consumes a replayed slot whose producing put has not advanced
immediately.
- the content sweep applies releases from RAM pages before the pump reads
their entries, so the pool advances past the entry's payload while it is
pending. entry_want (recorded on first examination) now detects that on
both the unreadable-rescue and the readable-no-release paths and consumes
the entry instead of waiting the grace window.
Evidence: real-release watcher cycle, pool semaphore advances no longer
gated at 8 s intervals once the replay skip engages.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
the sweep already applied
The runtime client's second push on each UVM copy channel starts with a
NON_INC method (0x60080002, count 8) and its release method sits 0x50 bytes
in. The stream validator accepted only INC (op 1) headers, so the readable
page failed the plausibility gate and every content-rescue candidate for
that VA failed the read-back validation - the client's tracking semaphore
for the push never advanced and the runtime init parked.
- one shared parser (svc_stream_parse) accepts INC/NON_INC (op 1/3),
IMMD_DATA (op 4, header-only), ONE_INC (op 5) and END_PB_SEGMENT (op 7),
and still requires an exact method boundary over the whole entry plus
trailing zero padding - the strict end condition is what keeps the
content rescue safe against firmware/allocator pages
- svc_method_stream_plausible and svc_stream_full_ok both use it, so a real
readable push is decoded directly and a rescued page validates
- the decoder skips NON_INC/ONE_INC methods whole (including their data
words) instead of resyncing word-by-word through the payload, which could
misread data as headers and queue bogus work
Follow-on: the content sweep applies releases from RAM pages ahead of the
pump, so the pump's expected payload was already satisfied when it examined
the entry; the CE grace path then held each entry for the full 8 s window
before consuming it - one grace period per push, serializing the client's
runtime init. The pump now records the expected payload when it first
examines an entry (entry_want) and consumes the entry immediately once the
pool advances past it, on both the unreadable-rescue and the
readable-no-release paths.
Evidence (fresh boot, real releases, watchers + kprobes, no fakes):
cudaGetDeviceCount rc=0 in the runtime client, device-attribute walk
proceeds, pool semaphores advance; before this change the client sat in the
0x20801702 service-interrupt loop for >6 min and never left device init.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The runtime client's registration stalled when no guest-side pin scanner was
running: the content rescue refused to run before the pool semaphore was
translated (its expected payload was unlearned), and unreadable CE entries
were consumed after two failed rescues ~40 ms apart, dropping the client's
wait. The push pages then could not be found within the entry's grace window.
- rescue: drop the learned-payload gate (the matcher only needs the channel
rank and the release shape); keep unreadable CE entries pending for
sema_ce_grace_ms instead of consuming after a fixed attempt count
- expected payload: re-learn from the pool whenever it advanced (a deferred
release could land after the first read, leaving the expectation stale)
- matcher: require the expected payload exactly and validate the read-back
stream carries it, so a recycled page of an older push cannot be pinned
- scan: descending full-range pass (the work allocations live in the high
RAM block) plus a +/-512 MB near pass around the verified push-buffer base
- releases decoded with an untranslated semaphore are queued and applied
when the page appears (svc_pending_add on both decode paths)
- a CE push that reads as a plausible stream but carries no release for the
channel is re-routed through the rescue instead of silently consumed
- new module param sema_rel_recover (default off): after grace + rescue,
apply the channel's expected tracking value so the client's wait clears;
counter-visible as releases_recovered, distinct from sema_fake_rel
Evidence: fresh boot, real releases, no host watcher, no kprobes - the six
UVM CE first pushes are found by content rescue, their releases applied, and
the client reaches cudaGetDeviceCount rc=0 (it then waits for the third
push, which this change routes to the rescue).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
helper gates on
The static CUDA runtime's device init failed with cudaErrorInitializationError
even though every kernel ioctl returned 0: cuInit succeeded and then
cuDeviceGetAttribute(ECC_ENABLED=36) returned CUDA_ERROR_NOT_FOUND (500).
Tracing libcuda's clock-domain helper (libcuda+0xe8c410) shows it issues the
CLK legacy controls 0x20809009/0x20809064/0x20809001 and bails with 500 when
the replies do not carry the expected domain bits/entries; the dependent
attribute getters (ECC_ENABLED, TOTAL_CONSTANT_MEMORY) then abort init.
Reply to the three queries with the minimal fields the helper checks: bits
32|34 present in 0x20809009, non-zero domain entries at offsets 12/20/28/36
of 0x20809064, and a matching selector in 0x20809001. No frequencies are
fabricated (all other fields stay zero).
Verified in the guest with the stock 570.144 static-cudart vectorAdd client:
cudaGetDeviceCount -> rc=0 (ndev=1), cudaSetDevice, cudaFree, three
cudaMallocs and both HtoD copies all return rc=0; the client then parks in
cudaDeviceSynchronize waiting for the (not yet forwarded) kernel launch.
|
| |
|
|
|
|
|
|
|
|
| |
The control is cacheable in the guest kernel (RMCTRL_FLAGS_CACHEABLE)
and rmapiControlCacheSet validates paramsSize against the canonical
sizeof(NV2080_CTRL_BUS_GET_PCI_BAR_INFO_PARAMS) = 200. We replied only
8 + 4*24 = 104 bytes, so every reply tripped
NV_ERR_INVALID_PARAMETER in rmapiControlCacheSet (rpc.c:9998) and the
BAR-info cache entry was never populated. Zero-fill and report all
eight NV2080_CTRL_BUS_PCI_BAR_INFO entries (count still 4).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A healthy run left dmesg dominated by per-operation lines (sema-push,
sema-lda, sema-chan rel-apply, sema-grce, DOORBELL/BAR writes, GSPRM
per-RPC lines, the teardown garbage-element stream). Introduce the
module parameter sema_dbg (default 0) and route every high-frequency
diagnostic through it; the finer-grained watch knobs (sema_ops_log,
sema_copy_watch, sema_copy_pat, sema_copy_dst, sema_copy_dst2) only
take effect while it is on. Boot stats, channel registrations, pin
events and state changes stay unconditional.
Also carries the per-side translation provenance (op_tr_src, printed
as prov= in the destination-window watch) and the 16 KB destination
window used to observe the guest's page-table writes.
Verified: fresh boot + full cdt2 smoke test (64/64, ALL OK) leaves a
520-line dmesg with zero per-push/per-doorbell/per-RPC lines.
|
| |
|
|
| |
RAM scan (the DtoH dst staging VA needs multiple scan slices)
|
| |
|
|
| |
translation-miss tracing (the DtoH copy operand is the failing side)
|
| |
|
|
| |
DtoH op is still invisible)
|
| |
|
|
| |
op-failure log budget (the DtoH work path is invisible at the old caps)
|
| |
|
|
|
|
|
|
|
|
|
| |
UVM CE tracking semaphores start at a base value and each completed push
adds the release payload (observed base 1 + payload 1 -> the client waits
for 2). The absolute comparison (payload > cur) could not see the second
push and the release was silently dropped, leaving cuInit/UVM stuck.
Track applied GPFIFO slots in a per-channel bitmap (preserved across
re-registration and the pump's snapshot write-back) so a pin-batch rewind
replays entries without double-incrementing; rescued pages stay
release-only and never mark a slot.
|
| |
|
|
|
|
|
|
| |
cuMemcpyDtoH submits the copy as C5B5 methods bound on subchannel 0 of a
compute channel (eng=0). allow_ops was false for every eng=0 channel, so the
LAUNCH_DMA copy was decoded (and its release applied) but never executed, and
the client read zeros. A host-injected translation is verified truth: allow
CE work on those pushes too.
|
| |
|
|
| |
HtoD writes)
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
bits 32:8/53:8)
The mask-and-keep decode dropped the left shift by 12, so every page-table
child pointer and leaf resolved 16x too low and no GMMU walk ever produced a
translation (walks_ok=0 on every boot). Verified against live page tables:
directory entry 0x2020a -> table 0x202000, leaf 0x0600000000760001 (aperture
VID, KIND=6) -> PA 0x7600000.
|
| |
|
|
|
|
| |
sema_copy_dst=dumps the source bytes of every copy whose destination falls
in the given page, so the 32-byte page-directory updates the guest writes
into vidmem can be read directly and their entry encoding learned.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The 256-byte cuMemcpyHtoD payload never appears in any CE push: CUDA
submits it on the compute channel as a compute-class DMA whose source is
the push's own inline data (LINE_LENGTH_IN=256, OFFSET_OUT=<device VA>,
LAUNCH_DMA=0x41, LOAD_INLINE_DATA 64 dwords, then the channel's
SET_REPORT_SEMAPHORE release). The CE-shaped decoder skipped the data run
(opcode 3) and no GR work ran at all.
- Decode NVC5C0 GRCE methods on subchannel 1 (LINE_LENGTH_IN, LINE_COUNT,
OFFSET_OUT/UPPER, LAUNCH_DMA) and the opcode-3 LOAD_INLINE_DATA run.
- svc_grce_inline_write resolves the destination VA through trusted
sources (pins, map, published page directories, targeted rm_mem
resolve), pins the page and writes into vidmem or guest RAM; budgeted
sema-grce logs name the target or report UNRESOLVED.
- cdt2 test binary gains CDT_CP_PAUSE / CDT_HP_PAUSE for live inspection
of the copy path; pushed by push_guest_tools.sh.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
payload-pattern audit
- CE LAUNCH_DMA releases are now applied after the push's collected work
runs: the client reads copied data as soon as it observes the release,
so release-first let cuMemcpyDtoH observe a copy that had not executed.
- Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms,
default 8s) before the release fallback consumes them, so a translation
that lands within a service pass lets the push be decoded and its work
executed instead of being dropped.
- sema_copy_pat matches an 8-byte value against every copy's src/dst first
word and scans decoded pushes for it, naming the submission that carries
a known payload regardless of size/encoding.
- sema-dma-skip logs transfer-shaped launches that collect no op; op log
lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG.
- A real release application now records the release VA, so decoded pushes
no longer trigger a spurious fake release.
|
| | |
|
| | |
|
| | |
|
| | |
|
