| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
148 r873)
The unguarded 3 ms mirror loop kept writing the count law into its latched
page after the guest had reused it (r872 evidence: guest 'Bad page map ...
pmd:102656067' pointing at exactly the mirrored page's PA as another
process's PTE table). The loop now latches the progress word's base bits and
stops permanently when they change or are zeroed, and re-checks the counter
object's signature every 64th tick. New sema_xw_comp parameter + vgsp-xwcomp
kthread can clear a submit member's armed 0x8000 status (value 0 pattern
only); r874 proved an unverified clear into a false member kills Xorg within
seconds, so it stays uninjected until the member address is watched live.
|
| |
|
|
|
|
|
|
|
| |
r867 with member = counter + learned-delta: every check is a permanent
+0x70 mismatch (the mirror re-advances right after each counter change),
xorg.log shows '(2, ..., 0xa8c, 0xa1c)' timeouts and the GL client gets
nothing. Plain member = counter is the correct law; the driver's
re-probe cadence gives the mirror room to settle between the driver's
counter increment and its next probe.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
r866: the mirror tracked perfectly yet every wait still paid the driver's
~3 s re-check sleep (one '(0)' success print every 20-80 s; the GL client
hung minutes on its X socket). Cause: the driver increments its counter
and then enters the wait; a member that only tracks the counter always
loses that entry-check race (mirror wakes <=3 ms after the change; the
driver probes within microseconds), so the check fails and the driver
sleeps out its retry interval. Writing member = counter + last-observed-
delta (learned from each counter change, 0 until the first) makes the
driver's own increment land exactly on the pre-written member value, so
the first probe after submission succeeds and frames complete at the
driver's pace.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
r864 ground truth while wedged: both the RC-watchdog channel (0x31415900)
and the X/GL channel (0x1000013) sit with their class2d notification
ARMED (u16 status 0xFFFF at notifier_pa + 0x3E = slot-3 struct status)
and nothing ever clears them (X notified=0 after 459 submissions), so
the guest krcWatchdog fires 'GPU is probably locked' every ~8.2 s and
the whole X/GL stack wedges (glxgears poll() never returns; late XOOM
death). Under the mirrored flow the watchdog ring reads go unreadable
(rc=-2), so the decode-time ack and the notify2d_va path both miss.
svc_notify2d_keepalive(): every pass, every GR channel with a registered
notifier PA, read slots 0..3; for each slot armed exactly 0xFFFF write
the zeroed 16-byte NvNotification and post the channel event - the same
effects as svc_notify_complete without any ring read. Also bump the
demo harness guest RAM to 8GB (launch_qemu_rootport.sh) to survive the
observed emulation-path allocation growth during accelerated runs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
r861 evidence: the X server's present path spins in a 10s-timeout wait
((EE) NVIDIA(GPU-0): WAIT (2, 8, 0x8000, <stuck 0xd4..>, <driver counter
+0x70/frame>)) comparing (progress_word & 0x3ffff) << 2 against its own
submitted-byte counter; the progress word (lone u32 0x140235 in an
otherwise-zero guest page) never advances because the emulation does not
execute the 2D method streams -> glxgears 0.09 FPS. A gdb write of the
word to 0x140000 | (counter >> 2) made the wait complete in 3 ms
(WAIT (0, ... <equal>, <equal>) immediately after the timeout print) and
silenced all further timeouts. This adds a 3 ms kthread that mirrors the
driver's counter word into the progress word; both guest PAs are injected
per boot via the sema_xw_mirror parameter (0,0 stops). Demo-path tuning
aid; no product logic depends on it.
|
| |
|
|
|
|
|
|
|
| |
r859: the X channel never resolved its ring all boot (garbage entries
va=0x41c000/0x2017a20/0x10104, unreadable/skipped, ~11 s per frame).
Rule verified on the RC watchdog across r848/r853/r858: ring = notifier
page + 0x400. Widened to every GR channel with a registered notifier PA;
the standard entry0+USERD candidate probe still content-gates each
derivation, so a wrong guess costs a failed probe (the current state).
|
| |
|
|
|
|
|
|
|
|
| |
The RC-watchdog channel's ring page sits at (notifier_pa page + 0x400) in
every observed boot (r848/r853/r858 data points). The trusted walk resolved
the same ring 14 s late on r858, inside the driver's DMA-test window on the
failing boots, where the ring read zeros/garbage - 'Failed to initialize
DMA' plus guest vmap-lock livelocks followed. Deriving the ring from the
registered notifier PA (guest truth from the channel alloc) removes the
race; the standard entry0+USERD candidate probe still content-gates it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- sema_trace_h + sema-perf logs: per-second channel state (put/last/armed/
counter) and publish-gate reasons, plus maprescue attempt outcomes.
Instrument added after window tuning failed to move the 22.4 s/frame
glxgears cadence; it located the real blocker in one run.
- svc_rescue_push_page: content-verified full-chain map rescue (works for
GR channels, whose rescue was previously refused outright).
- gsp_phys_booter: RPC reply waits now usleep_range(150-350us) instead of
msleep(5) (termbin review suggestion, in-tree constant verified; no
FPS change measured, kept as correctness-neutral latency reduction).
Evidence: r843-r847 dmesg_live logs; measured constant 22.4 s/frame for
glxgears across all param windows (grace/poll/first-poll), showing the
pacer is elsewhere - traced to client pushbuffer pages (va=0x3f06270-class)
failing svc_lookup (-ENOENT) while the client is alive.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two changes from the r839/r840 multi-client forensics (session 147):
1. The guest-RAM content scan for a GR ring must not run for the kernel
RC-watchdog channel (h=0x31415900). Its ring is a plain kernel allocation
always resolvable through the driver-published VAS walk; on r840 the walk
lost a registration race, the scan latched a foreign garbage page, and
the bad latch poisoned the driver's DMA test ('Failed to initialize DMA',
Xorg died). Gated, the walk retries per pass; r841 Xorg#1 passes DMA
(DMAFAIL=0) where r840 failed.
2. sema_force_flush module parameter + gsp_sema_service_sess_flush_force():
execute the ISSUE-0002 client-session flush even with live channels. The
channel-close trigger never fires on real boots (driver-global channels
stay valid), so later client sessions inherited stale discovery state.
Verified on r841: flush executes mid-boot, Xorg survives it and
re-registers its channels.
Remaining wall (addendum 9): GR client channels' pushbuffer reads return
all-zero through the trusted translation (va=0x4000000-class, src=4), so
first entries never complete and GL clients never render.
|
| |
|
|
|
|
|
| |
A stale frame left in the kvmfr shm by a previous publisher run (xshm_pub
restarts its own seq at 1) was saved and then blocked the new run's frames
via the seq-collision rule. Seeding last_seq from the header's current seq
skips the stale frame and lets a fresh run's frames save normally.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every RM client maps different objects at the same numeric VAs (all
clients' windows sit at 0x1200xxxxx), so the global VA-keyed pin/lat/map
layers conflate them across clients. Live walk_probe evidence (r834):
the X server's ring VA 0x120090000 resolved through the GL client's page
tables (vas h=0xcaf00005 -> a zero page) while the X client's own VAS
(vas h=0x10005) holds the real page (pa=0x1294fa000). That conflation is
what wedged the X server's readback wait and starved the RC watchdog's
ring (its pushes decoded as empty).
* Parse NV_CHANNEL_ALLOC_PARAMS.hVASpace (@28; layout confirmed by the
struct's own known offsets) at channel registration and carry it on
svc_channel.
* svc_chanvas_lookup: resolve a VA through the serviced channel's own
VAS page tables first (svc_cur_vas, refreshed per channel turn and in
the decode path). The hook sits at the head of svc_lookup_trusted, so
ring, push, release and notifier resolution all inherit it; a failed
walk falls through to the old chain unchanged.
Live r835: X's XShmGetImage readback, previously never completing,
returned after its long wait and a frame reached the kvmfr BAR (first
frame content still black and readback latency still ~87 s; the
guest-side fbcopy_pub framebuffer reader tool is added for the
direct-VRAM capture path, find/pub/test modes).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
r834: the watchdog push's translation can go stale under heavy GL load
(the content-scanned map answering a zero page where an injected pin
previously delivered the real method stream), starving the notifier ack.
Pin the page that delivered the first real decode (svc_read_last_pa at
the SET_NOTIFY_A capture) and the page that accepted the first ack
(svc_notify2d_write), so the every-pass ack keeps resolving.
Still observed: under 150 s of sustained GZDOOM the guest's krcWatchdog
can still report 13-17 timeouts and xshm_pub's XShmGetImage never
completes (the X readback op stalls), so the capture path is not yet
usable; the X server itself stays up (4 WAIT lines / 150 s).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
notifier ack)
The guest kernel's krcWatchdog posts a notifier cycle on its own channel
(object 0x31415900) every second and declares 'GPU is probably locked'
when the zeroed notification never comes back within 7 s. Observed as
the sustained-GL-load wedge: once a GL client (GZDOOM) enabled robust
channels the errors started and Xorg eventually stalled on screen
readback.
- rm: register the Volta GPFIFO class 0xc36f (plus the later
generations) as a channel class. The watchdog's channel allocates
0xc36f on this device - the driver picks the first class the gpu
reports supported - and without it the channel never reached the
progress service. It now registers and latches (put visible, entries
walked).
- sema: decode the class2d notifier cycle on subchannel 3
(SET_NOTIFY_A/B name the notifier VA, NOTIFY WRITE_ONLY writes the
zeroed NvNotification) and author the write, like svc_notify_complete
does for the work-submit token. Proven live: 'sema-notify2d: va=...
zeroed (RC watchdog acked)'.
- sema: also ack on every pass that sees the channel's put != get, so
the ack does not depend on the ring reads (its VA is known from the
first decode); idempotent.
- sema: never consume a zero-length/zero-VA GPFIFO entry as empty work.
Two bogus zero entries consumed through a mis-resolved ring page
pushed the watchdog userd's GPGet past the driver's own GPPut and
every later userd probe then read get>put, permanently rejecting the
channel's real userd. Hold such entries (bounded, per-channel
counter persists through the pass snapshot).
- sema: a registered userd value that is a real guest PA (the
watchdog's own block) outranks the ring+0x2000 convention guess the
ring auto-pin applies for the CUDA GR layout.
- rm: log channel-class allocs whose params are too short for
registration instead of skipping silently.
Remaining (next): under heavy GL traffic the watchdog ring's
translation can go stale for stretches (content-scanned map answering
where a pin used to), which starves the ack; pin the push/notifier
pages when the first real decode delivers them. The frame capture
(xshm_pub -> XShmGetImage) still stalls while that starvation is
active.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
push-scoped release deferral
The X server now starts on the emulated vGPU and GLX runs on the real
NVIDIA driver: Xorg :0 passes its DMA sanity push and ScreenInit, and
DISPLAY=:0 glxinfo reports 'OpenGL 4.6.0 NVIDIA 570.144' with direct
rendering; GZDOOM 4.14.2 launches and renders. Before this, every X
start died at 'NVIDIA(GPU-0): Failed to initialize DMA' (the
__nvPushTestPushBuffer 10 s GP_GET wait) and no nvidia-push kernel
buffer could be resolved.
What each piece fixes (all observed live on the r817->r827 cycles):
- sema: register the driver's PROMOTE_CTX context-buffer statements as
a VA->PA registry, consulted as the LAST resort after every walk.
A promoted (va, pa) pair is a VA-space booking, not a mapping: its
span can swallow other buffers' VAs (the ctx MAIN booking covered the
channel GPFIFO VA and range arithmetic answered the entry validation
with MAIN.pa+offset, a zero page - the latch never fired, r823), and
even its base page can host a separately mapped buffer (the PATCH
booking's base VA hosts nvidia-push's 4 KB progress semaphore; the
release landed in the booking's page while the driver's own polled
page stayed zero, r826). The walks describe the real mappings and
must answer first.
- sema: an all-zero GR push of <= 16 bytes resolved by the trusted
chain (src=4) is accepted as the driver's NOP sanity push
(NVA16F_DMA_NOP is a single zero dword; real launches are hundreds of
bytes). The r496e acceptance required a content-scanned ring, which
the X/GL channel is not.
- sema: the wave-1.5 release deferral is push-scoped. A release whose
push owns no captured forward candidate has nothing to wait for; it
applies inline (the progress-tracker semaphore release used to be
recorded and dropped at every pass reset while an unrelated lingering
candidate armed the global defer).
- sema: the m164 no-capture hold no longer arms for the accepted zero
NOP - it held the NOP entry for its 6 s window 'forcing capture' of a
launch that cannot exist.
- sema: once the ring resolves, the USERD latch scan may retry past its
8-scan cap (the client's VAS directory registration lost the race
against the 10 s DMA window when a rebuild stretched the pass cadence,
r820).
- sema: dbg-gated USERD-scan decision witnesses (rejection reason, sweep
summary, per-call state), GR-only and budgeted.
Known next issue (not this change): sustained X/GL rendering traffic
wedges the guest RC watchdog ('NVRM: krcWatchdog_IMPL: GPU is probably
locked') after minutes of GZDOOM/GLX activity; the FIFO service itself
keeps consuming.
|
| |
|
|
|
|
|
|
|
| |
The nvidia-push DMA test in Xorg screen init fails with 'Failed to initialize
DMA / Failed to allocate push buffer' (observed r816) because the X server's
pages are never pinned - the scanner excluded Xorg/X alongside the
nvidia-smi/persistenced transients. Xorg is long-lived and a legitimate GL
client of the vGPU, so it is now a valid pin target; the transient exclusions
stay.
|
| |
|
|
|
|
|
|
|
|
|
| |
split_screen.sh composes two guest frame streams (KVMFR PPMs from
frame_reader.py) into one hstacked mp4 with last-frame padding for unequal
lengths; build_video.sh wraps a title card, the main segment and an end card
(drawtext, detected font) into the deliverable mp4. Both verified end-to-end
with synthetic PPM sequences (ffprobe: 1280x720 h264, padding pixel-exact).
These are the capture-side pieces for the 1.0.0 video; the guest-side
publishers are tools/guest-diag/xshm_pub.c (X screen) and the existing
frame_producer; the host reader is tools/lg/frame_reader.py.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
foreign ring
The GR ring autopin fallback (SESSION 133c userd-shape scan) and the CE
first-submission capture each guarded block ownership only within their own
engine class, so a channel whose own ring was not yet written latched another
channel's live block permanently: the GR channels drained the CE rank-0 ring
(or an unlatched CE rank's convention block), the real launch entry was never
read, and CUDA clients hung after their first device query (victory_mail
stalled in UVM channel_pool_add; every post-reboot boot r805-r811 showed the
same steal, e.g. 'auto-pinned GR ring h=0x5c000019 rank=0 ... -> fb 0x80000
(userd-shape scan ... entry0 va=0x120000000 len=40 put=35 get=35)' while the
CE handle decoded the identical entry[35] raw words from pa=0x10350641c).
Changes:
- shared FB block claim table (svc_fb_claim_add/_foreign/_reset), session
scoped, consulted by both GR scan passes and the CE fallback scan; every
live CE rank's convention block is reserved even before that rank latches
(r809: CE rank 1's block at fb 0xa0000 was stolen pre-latch);
- every GR autopin latch (userd-derived, FB convention, pin, trusted walk,
scan pass 1/2) and both CE latch paths record claims;
- CE first-submission capture: a rank's own convention block is latched even
when its stale content probes incoherent (the previous session's consumed
state), instead of falling through to a never-used zero block (r812/r813:
client 2's 0x90000 block read put=1/get=146 at the capture instant and the
zero 0x82000 block was latched, hanging the client).
No module parameters or defaults change; the guards are additive.
Validated (plain mdev, script defaults):
- victory_mail GREEN repeatedly: 'VADD n=32768 bad=0 checksum=12204516.000000'
/ 'STATUS OK', all six GR ranks latched at their own fb 0x140000+0x3000*rank
rings with 'launch captured h=0x5c000019 push=0x20043be94 len=840';
- a second sequential client on the same boot now progresses (vecadd_loop:
'VERIFY launch=10 bad=0' - the 100-launch loop needs a long budget).
|
| |
|
|
| |
FAULT_PDE at the program VA), Vulkan copy channels registered/swept/never pushed, no program bytes on any channel; frontier = upload copy + program-VA mapping + restore the black-frame baseline
|
| |
|
|
| |
channels are registered/swept/never pushed; no channel carries program bytes or a LAUNCH_DMA; the code page never fills; the refusal path no longer reaches the black-frame baseline (SYNTH-FAULT vs completion ordering)
|
| |
|
|
| |
lines for every channel + a 128-row per-pass cap, so a channel absent from the log can never be confused with a channel not in the table (the r802 copy-class channels 0xbeef0101/2 visibility gap)
|
| |
|
|
| |
min_data=0 reaches the real engine, which MMU-faults (FAULT_PDE) at the guest program VA; the wall is now the program page's mapping and content (SASS upload), not the data-set gate
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
no-external-data launch class), not a clamp to 1
The Vulkan compute launch's single QMD pointer pair names its own QMD
window and its params are scalars (0.2/640/480), so n_data=0 is correct
for it; the bare n_data>=min threshold refused it as degenerate. The
bypass is safe now that gsp_phys_chunk_page_va answers NULL for an
unallocated chunk (r800) - before that guard the empty-set post-mortem
print dereferenced a garbage pointer (Oops at
gsp_phys_backend_guest_gr_submit+0x4394). Default stays 16; 0 is opt-in
for the egg-break test that forwards the launch to the real engine so
the copy-out can carry pixels back. Proper launch-internal predicate
next.
|
| |
|
|
| |
unallocated chunk (no per-page array AND no base va) - it previously returned (u8*)NULL + page*PAGE_SIZE, a small non-NULL garbage pointer, so the caller's NULL guard passed and the empty-set post-mortem print dereferenced it (Oops #2 at gsp_phys_backend_guest_gr_submit+0x4394, CR2=0x1e93c); with the guard, min_data=0's no-external-data forward can proceed
|
| |
|
|
| |
(QMD-page note + qmd-ptr seed + pin-aware reads) reaches the launch's data set; the empty-set forward reaches the backend and oopses at gsp_phys_backend_guest_gr_submit+0x4394 - guard first
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
seed, pin-aware scan reads
Three corrections that take the GR fill from 'no candidates' to the
launch's real pointer set:
1. Note the QMD page itself into the request's page list (word 34) so the
32..48-even bank scan covers it - the pointer VALUES live inside the
QMD (r790: word 34 = 0xdff0febc00) and were never scanned.
2. Seed the fill's candidate set directly from the launch's own QMD
pointer pairs (gr_qmd words 34..47, decoded from the push's inline-QMD
run): r793 'sema-gr: qmd-ptr seed va=0xdff0feb000 (word 34)' - the
values need no page scan (the pages' shadows read empty; the invisible
-vidmem-write class).
3. Read the scanned pages pin-aware (svc_read_va first, host mapping of
pp->pa as fallback): the push-buffer discovery's pinned staging copy
holds the content the decode reads; a raw mapping of the vidmem pa
sees zeros.
Verified r793/r794: candidates now appear (fill cand n=1 with the QMD
window page seeded); the page is correctly classified launch-internal by
the R3 hygiene filter (n_data stays 0 - nothing external to relocate for
this launch). With the r564 degenerate-set guard bypassed (min_data=0),
the forward ENTERS the backend and oopses in
gsp_phys_backend_guest_gr_submit+0x4394 ('Oops: 0000 [#2]', workqueue
gsp-phys-work svc_fwd_work_fn, NULL-base table walk CR2=0x1e93c R14=0) -
the empty-set path needs that guard before min_data=0 can be used.
This launch's QMD declares exactly one pointer pair, and it names its own
QMD window: a launch with no external data is legitimate and the refusal
class must not treat it as degenerate.
|
| |
|
|
| |
480.0f; the float-shape filter empties the false candidate set; the wall is content (i2m never queued / forward refused first)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
deduped word-48 page
Two evidence-based corrections to the launch-data candidate scan:
1. Float-shape filter at all four scan sites (the three page scans and
svc_gr_scan_cand): a word whose f32 exponent byte is in ~[0x6c..0x8f]
(2^-19..2^16) is a plausible float scalar, not a device VA - device
VAs (2^28..2^47) as f32 land in subnormal/tiny exponents. Root
evidence (r789): the launch param page carries the kernel's scalar
arguments (0.2f, 640.0f, 480.0f at +0x630 - the 640x480 viewport) and
0x43f00000 = 480.0f was the fill's ONLY candidate - an unresolvable
"arg base=0x43f00000" that drove the fail-closed diagnostic and the
demand-loop noise for days. After the filter: 'fill cand n=0'
(r790) - no false candidates, no false demands.
2. The bank-page scan now covers word 48: the QMD pointer-region pair can
name a 4K page that also carries the program pair (r788: ptr[0]
0xdff0febc00 word 34 shares its page with prog 0xdff0feb900), and
svc_gr_page_note dedupes the first note (the program's, word 48), so
the old 32..47-even filter left that page unscanned.
The remaining wall is content, not wiring: with the false candidates
removed the launch still refuses (n_data=0 vs min 16) because no scanned
source holds real device pointers - the param page holds scalars, the
QMD window page reads zero (its content would arrive via the driver's
i2m/LOAD_INLINE_DATA writes whose payloads are not in the push), and the
inline payload is the driver's zero stub. This matches the SESSION 130
zero-buffer finding and leaves the never-queued i2m copies (or the real
GPU path) as the only content sources.
|
| |
|
|
| |
printed (param/prog pages, deduped QMD window, single inline candidate, n_data=0 min 16 refusal); the window content exists in no host-visible source
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(sema_dbg-gated, bounded)
The dstpend/n_data frontier needed the launch's actual wiring. Prints, once
per launch within a bounded budget: the candidate handle, param_va, prog_va,
n_pointer + the QMD pointer pairs, the inline-rec count, and the resolved
gr_pages list (va/pa/ap/qmd_word); then in the fill, the built candidate
set with the arg flag.
First capture (r788, Vulkan launch h=0xbeec0100): param=0x4b86000 ->
page pa=0x386000 ap=2 qmd_word=32; prog=0xdff0feb900 -> page
pa=0x56b000 ap=2 qmd_word=48; ptr[0]=0xdff0febc00 word=34 (DEDUPED into
the qmd_word=48 page - same 4K page - and that page is never scanned by
the fill's word-32/32..47-even passes); the fill's only candidate is
0x43f00000 (arg=1, from the inline bytes) which resolves in NO vaspace
(rc=-2 in all four) -> n_data=0 -> the backend's min_data=2 refusal. The
param/prog/QMD-window pages read all-zero in the vidmem shadow on this
boot, so the launch's data/code bytes exist in no host-visible source.
|
| |
|
|
| |
for the Vulkan client is disproven by direct write watches (channelFillGpFifo is notrace; the entry store never executes; the client's ring mapping watchpoint catches zero writes); the frontier moves to the GR launch's dstpend/n_data page dependency
|
| |
|
|
| |
latch on record was a stale CUDA block; the ring content is in no shadow, no RAM, and not at the PTE target (BAR1 write path verified working; client-side write watch is the frontier)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
USERD candidate (get >= put > 0)
At the first-submission instant the channel's own submission is by
definition unconsumed - the notifier clear this capture runs before is
what would let the driver treat it as consumed. A candidate whose USERD
reads a consumed state (put == get > 0) cannot be this channel's live
just-submitted work.
Observed r785 (CUDA-first boot, three Vulkan runs): the capture latched
fb 0x80000 / userd 0x90000 for beef0101 (rank 0) with put=1 get=1 - the
dead CUDA client's caf00012 block surviving in VRAM after its session
ended. The decoded entry (va=0x120000000 len=40) was the old session's,
the expectations seeded from the stale pool word (want=2/3 vs pay=1) and
the Vulkan client stalled into its fence timeout on every run.
All genuine first-submission latches on record read put=1 get=0 at the
capture instant (r779 caf00012 rank0, r785 caf00012 rank0, r781/run1
beef0101). Idle 0/0 stays acceptable per svc_userd_probe's own rule.
Session-scoped rank ownership (svc_ce_firstsub_off) does not cover this
class: the stale block belongs to a previous CLIENT, and its consumed
state is the only local signal that it is not this submission.
|
| |
|
|
|
|
|
|
|
|
|
| |
first-submission capture caused the CUDA stall regression (fixed, a674c21c); the Vulkan CE rings live at 0x4020000/0x4100000/0x4160000 (above the 64 MB capture bound) and the frame chain reduces to the un-latched ring (copy pend -> dstpend -> black)
Records the instrument, the regression A/B (r746-era module passes, SESSION
133/134 code spins), the rank-aware fix and its r781 verification, the
Vulkan client's actual ring allocations from its own vidheap ioctl log, the
failed 4 GB RAM entry search, and the live frame dependency (GR launch
inline=8 + copy pend src 0xdff0d60000 -> vidmem 0x570000, dst framebuffer
0x50b0000) with ranked next steps.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rank-blind first-hit latched rank 0's block for rank 1
The SESSION 134 first-submission capture scanned the shadow from offset 0
and took the first ring-shaped page for EVERY CE channel. On the r779 boot
that latched rank 0's block (fb 0x80000, userd 0x90000) for rank 1 as well
(both rows scan_hits=1 latch=1 off=0x80000), so caf0001d's first release was
decoded from rank 0's ring/page (sem word +0 instead of +4) and
stale-dropped (cur=1 already from rank 0's applied release). The client's
rank-1 tracker wait never completed: victory_mail spun at 100% CPU with
pub_calls=0 on three consecutive boots (r777/r778/r779), while the r746-era
module (25523f84) passed the same run on a fresh boot (VADDs OK, STATUS OK).
Fix: try the rank's convention offset (0x80000 + 0x20000 * rank) first,
scan the rest bounded as before, and never accept an offset another CE rank
already latched this session (svc_ce_firstsub_off[], reset per session). A
rank whose block is not in the shadow at the instant stays unlatched for
the regular per-pass CE latch instead of stealing a foreign block.
Verified on r781 (fresh boot, same params xlate=1/util_scan=0/vas_cand_ms
=5000/ce_grace=20000): caf00012 rank0 -> fb 0x80000 upa 0x90000;
caf00001d rank1 -> fb 0xa0000 upa 0xb0000 (correct block, sem +4 applied:
pool-wr va=0x121000004 cur=0 -> val=1 ok); ranks 2-5 -> 0xc0000/0xe0000/
0x100000/0x120000 each +0x10000; r780 (r746 module) control run: same
client PASSED (VADD n=32768 bad=0 checksum=12204516.000000, STATUS OK);
r781 client progressed with pub_calls=185132+ (no spin stall). Also the
Vulkan client's CE channel latched via the classic autopin on r781
(beef0101 -> fb 0x80000, live entry0) - first time in the SESSION 131-134
window - confirming the latch machinery works once the wrong-block
mis-latch no longer poisons the channel.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
hva truncated the 40-bit VA so no CE-window entry could ever match)
hva must hold the 40-bit push VA (bit 32 set for the CE window) but was
declared u32 in the scan, so ((h1 & 0xff) << 32) truncated to the low
dword and hva < 0x120000000ULL was always true - scan_hits stayed 0 by
construction. Same decode as svc_chan_userd/svc_gr_ring_candidate (u64).
Verified building; the FB-convention latch itself (auto-pinned ring
h=0xcaf00012 -> fb 0x80000, userd 0x90000, entry decode + push decode,
ops_runs=1) was observed independently on r776.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- the CE ring latch instrument
The CE-class ring (the SASS-delivery channel's GPFIFO, registered at
gpfifo=0x4100000 on every boot) only ever latched on one boot (r763, the
FB convention matched shadow 0x80000 and the sweep decoded the inline
LAUNCH_DMA entry). On r765-r775 every deterministic source reads a zero
page (the page tables genuinely map the ring VA to an all-zero RAM page -
proven by the per-level walk trace) and the ring never appears in the
shadow (scan_hits=0 at the submission instant; dump_vidmem map shows only
the USERD put words) nor in the guest RAM image (a full 4 GB scan for
consecutive GPFIFO entry pairs around the CE push window finds none).
New bounded instrument, default-inert:
- svc_ce_firstsub_capture: on a CE channel's notifier clear (the one
instant ring content and USERD are both live), before the clear write:
reads entry0 through the channel translation and the trusted walk, then
scans the shadow (0x100 step, 64 MB) for a CE-window ring page
(1MB-aligned push VA in [0x120000000,0x121000000)) and latches via the
standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0. Budget 4/channel,
16 witness rows/boot, reset by svc_session_flush.
- sema_ce_hold_ms (module param, default 0 = off): holds an unlatched CE
channel's notifier busy for up to N ms so the guest keeps the
submission in flight (and with it the ring content) while the VAS lands
and the regular per-pass latch path retries; expires into the old
clear behaviour.
r774/r775 evidence (sema_ce_hold_ms=8000): sema-cefirst at the first
submission shows rrc=-2 (no VAS yet at +0.2 s), wrc=-1 (walk unresolved),
scan_hits=0 (no shadow ring); 30 s later the walk answers a zero page
(rpa changing per boot, e0=0x0), the hold expires unlatched, and the
client still completes (VKFRAME_DONE) - the hold is safe but the ring's
write path is still not visible to the emulation. Remaining unknown is
named: the driver's ring writes reach neither the shadow nor the
PTE-mapped guest page.
|
| |
|
|
| |
USERD by its own shape (put@+0x8c, get@+0x88, the find_userd layout) and derive the ring at userd-{0x10000,0x2000,0x1000}, validating entry0 (shape + not provably dead) and the USERD probe; the USERD words appear at channel creation and stay while the ring entry may be written later, so this is the stabler trigger for the CE-family rings that the periodic ring-entry scan misses
|
| |
|
|
| |
is FOUND (the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push: in=0x120a00004 -> out=0x205000, 4096 B/run; the Vulkan CE channels carry the same-shaped entries) and the three gates that keep the copies from running are named: the CE ring/USERD boot-lottery (FB convention matched r763 only; the real USERDs sit at 0xfee00/0xff000/0xff200 so the rings are at unaligned offsets like 0xefe00), the VAS age-out vs CE grace race (sema_vas_cand_ms=5000 registers at +5s), and the volatile ring/USERD content that a periodic shadow scan misses; next steps: USERD-shape-derived ring resolution or first-submission latch, then the inline copies and the n_data wiring
|
| |
|
|
| |
unaligned: r767 USERD shape scan found the channel blocks at 0xfee00/0xff000/0xff200, so the rings sit at offsets like 0xefe00) and retry rate-limited (1.5s) instead of a consumed once-bit, because a first pass can run before the client has written the ring or its USERD
|
| |
|
|
| |
rings live in VIDMEM and the FB convention only matches when the guest VRAM allocator happens to place them at the conventional offsets (r763: matched fb 0x80000/0xa0000 and the CE channel decoded its inline SASS copies; r765: read zeros, the ring VA resolved to a stale RAM page whose entry0 was dead, the CE channel never latched - black frames). When every deterministic candidate fails, scan the first 64 MB of the shadow for a ring-shaped page (entry0 names a 1 MB aligned push VA in the client window with a sane length, not provably dead via svc_ring_entry_dead) whose USERD at the observed deltas (+0x10000 CE family, +0x2000 GR, +0x1000) reads a coherent put/get; one scan per rank per boot, re-armed per client session. A false candidate still costs only a failed probe.
|
| |
|
|
| |
PROVEN in guest RAM (walk-confirmed per-run pages 0x11623d000/0x104b55000 with the 14-entry content the sweep decodes; the vidmem 0xa0000 'ring' was a false-positive autopin whose entry0 push reads all-zero and which the pump skipped after 39 rescues); the fixes (dead-entry guard + cross-aperture USERD + per-client util witnesses + op-level copyent) and their verification (r760: ring rc=0 on the live RAM page, live entry reads, VKFRAME_DONE 3 frames); the two LID uploads are the driver's zero buffers by design (256 B zero buffer + 12288 B CE fault buffer) while the GR channel's LID carries the real param bank; the code VA (prog=0xdfd8feb900) maps to a ROTATING code pool (0x62b900 -> 0xc8b000) whose lat entry can go stale and whose pages are empty - the SASS delivery shape (C5C0 0x3800/0x39C8 runs) and the walk-first code-VA resolution are the next two gates
|
| |
|
|
| |
r759 false positive is fixed at three points: (1) a ring candidate whose entry0 push provably reads all-zero through the client's own page tables is rejected when the VAS walk resolves the ring VA to a page with a live entry (svc_ring_entry_dead + svc_ring_walk_live, applied at the userd-derived, FB-convention and existing-pin paths; the walk cannot vote for itself); (2) svc_gr_ring_candidate now rejects dead-entry candidates too and accepts the channel's known USERD cross-aperture (r759: ring in guest RAM 0x11623d000, USERD in vidmem 0xb0000, so the walk-resolved ring failed the same-aperture USERD probe); (3) the util sweep arbitrates a dead channel-path ring entry against the direct VAS walk per slot. Also: util-dec/util-sweep witness budgets and their dedup keys become per-client (reset in svc_session_flush) and the sema-copyent witness logs once per copy op instead of per row (a 480-row frame copy alone exhausted the 128/session budget and hid every later copy, including any SASS upload)
|
| |
|
|
| |
frontier: the CUDA client's uploads fill the emulated vidmem (~450k non-zero dwords across many 1MB blocks) while the Vulkan client's leave only the 14 baseline dwords, so the Vulkan (NVN/GL) upload path is not among the writes the emulation can see; next steps recorded
|
| |
|
|
| |
(nz count + first offset) to locate the live client's uploaded regions in the 256 MB aperture without 65536 range dumps
|
| |
|
|
| |
in vidmem (map PA 0xa0000, live entry 0x120100000/40) and the decoded 14-entry LID ring is a STALE page from a dead session, which is why its payload chunks read zero through every path (four reads + BO poller + gdb write-watchpoint); the CUDA contrast proves the LID arm works with real contiguous payloads; next step is reading the live ring through the map and decoding its entries
|
| |
|
|
| |
LOAD_INLINE_DATA arm fires (branch-order fix), the payloads are the driver's zero buffer (four-read proof + full-run BO poller + gdb write-watchpoint), the SASS delivery is the remaining non-black gate (prog page 0x62b000 empty; NVN download path targets the pitch-pool allocation), plus the xlate/util_scan regression bisect result
|
| |
|
|
| |
regressions were re-tested on a fresh boot: with the translator ON and the utility sweep OFF, victory_saxpy and victory_math PASS with checksums identical to the gate-off baseline (saxpy csum=981440890.500000), so the SESSION 129/130 sweep changes fixed the saxpy/math half; the remaining xlate-on regression is victory_multikernel, whose physical forward times out repeatedly (GR LAUNCH FAILED ... qmd_rel=0xc0ffee01, hold active), so the default stays 0 until that forward path is fixed. The UTILITY SWEEP is the other regression source (util_scan=24 broke saxpy bad=1047527 and hung math) and stays opt-in until its channel reads prefer the direct walk
|
| |
|
|
| |
a zero-headed tail-served payload is now applied under a variant once-key (h | 1) instead of being refused forever (the destination is empty, so authoring zeros is idempotent and correct for the driver's internal zero buffer, whose 256 B i2m equals GF100_ZERO_BUFFER_SIZE exactly), while a later non-zero payload for the same entry still matches its own (h) key and lands; the utildec witness now carries alt=success/tries so a walk-read that fails is distinguishable from one that returns zero
|
| |
|
|
| |
GRCE operand group was swallowed by the TURING-copy branch (its `subch >= 2` guard also matches subchannel 2, leaving have_out/line_len unset so the LID arm never fired: r739 witness `lidn=1 subch=2 saw=0 dst=0 ll=0`); the 0x180..0x18c group now parses FIRST for subch 1|2, plus tail-content arbitration (a zero-headed channel tail is re-read through the direct registered-VAS walk before the arm's zero-refusal can stall the upload) and the witness now carries the served tail's first two words + the alt-read counter
|
| |
|
|
| |
end-to-end (VKFRAME seq/DONE/IDLE, three PPMs); the full fix chain (push-parse shapes, fence fold sema_gr_ahead, VAS pdb anchoring, COMPR downgrade, scrub-author, 9096/ZCULL, C5C0 translator); non-black frames (MME/i2m LOAD_INLINE_DATA arm) is the frontier; CUDA 8/3/1 fresh-boot battery; video v1 on files.vgpu.io
|
