| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
| |
r859: the X channel never resolved its ring all boot (garbage entries
va=0x41c000/0x2017a20/0x10104, unreadable/skipped, ~11 s per frame).
Rule verified on the RC watchdog across r848/r853/r858: ring = notifier
page + 0x400. Widened to every GR channel with a registered notifier PA;
the standard entry0+USERD candidate probe still content-gates each
derivation, so a wrong guess costs a failed probe (the current state).
|
| |
|
|
|
|
|
|
|
|
| |
The RC-watchdog channel's ring page sits at (notifier_pa page + 0x400) in
every observed boot (r848/r853/r858 data points). The trusted walk resolved
the same ring 14 s late on r858, inside the driver's DMA-test window on the
failing boots, where the ring read zeros/garbage - 'Failed to initialize
DMA' plus guest vmap-lock livelocks followed. Deriving the ring from the
registered notifier PA (guest truth from the channel alloc) removes the
race; the standard entry0+USERD candidate probe still content-gates it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- sema_trace_h + sema-perf logs: per-second channel state (put/last/armed/
counter) and publish-gate reasons, plus maprescue attempt outcomes.
Instrument added after window tuning failed to move the 22.4 s/frame
glxgears cadence; it located the real blocker in one run.
- svc_rescue_push_page: content-verified full-chain map rescue (works for
GR channels, whose rescue was previously refused outright).
- gsp_phys_booter: RPC reply waits now usleep_range(150-350us) instead of
msleep(5) (termbin review suggestion, in-tree constant verified; no
FPS change measured, kept as correctness-neutral latency reduction).
Evidence: r843-r847 dmesg_live logs; measured constant 22.4 s/frame for
glxgears across all param windows (grace/poll/first-poll), showing the
pacer is elsewhere - traced to client pushbuffer pages (va=0x3f06270-class)
failing svc_lookup (-ENOENT) while the client is alive.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two changes from the r839/r840 multi-client forensics (session 147):
1. The guest-RAM content scan for a GR ring must not run for the kernel
RC-watchdog channel (h=0x31415900). Its ring is a plain kernel allocation
always resolvable through the driver-published VAS walk; on r840 the walk
lost a registration race, the scan latched a foreign garbage page, and
the bad latch poisoned the driver's DMA test ('Failed to initialize DMA',
Xorg died). Gated, the walk retries per pass; r841 Xorg#1 passes DMA
(DMAFAIL=0) where r840 failed.
2. sema_force_flush module parameter + gsp_sema_service_sess_flush_force():
execute the ISSUE-0002 client-session flush even with live channels. The
channel-close trigger never fires on real boots (driver-global channels
stay valid), so later client sessions inherited stale discovery state.
Verified on r841: flush executes mid-boot, Xorg survives it and
re-registers its channels.
Remaining wall (addendum 9): GR client channels' pushbuffer reads return
all-zero through the trusted translation (va=0x4000000-class, src=4), so
first entries never complete and GL clients never render.
|
| |
|
|
|
|
|
| |
A stale frame left in the kvmfr shm by a previous publisher run (xshm_pub
restarts its own seq at 1) was saved and then blocked the new run's frames
via the seq-collision rule. Seeding last_seq from the header's current seq
skips the stale frame and lets a fresh run's frames save normally.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every RM client maps different objects at the same numeric VAs (all
clients' windows sit at 0x1200xxxxx), so the global VA-keyed pin/lat/map
layers conflate them across clients. Live walk_probe evidence (r834):
the X server's ring VA 0x120090000 resolved through the GL client's page
tables (vas h=0xcaf00005 -> a zero page) while the X client's own VAS
(vas h=0x10005) holds the real page (pa=0x1294fa000). That conflation is
what wedged the X server's readback wait and starved the RC watchdog's
ring (its pushes decoded as empty).
* Parse NV_CHANNEL_ALLOC_PARAMS.hVASpace (@28; layout confirmed by the
struct's own known offsets) at channel registration and carry it on
svc_channel.
* svc_chanvas_lookup: resolve a VA through the serviced channel's own
VAS page tables first (svc_cur_vas, refreshed per channel turn and in
the decode path). The hook sits at the head of svc_lookup_trusted, so
ring, push, release and notifier resolution all inherit it; a failed
walk falls through to the old chain unchanged.
Live r835: X's XShmGetImage readback, previously never completing,
returned after its long wait and a frame reached the kvmfr BAR (first
frame content still black and readback latency still ~87 s; the
guest-side fbcopy_pub framebuffer reader tool is added for the
direct-VRAM capture path, find/pub/test modes).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
r834: the watchdog push's translation can go stale under heavy GL load
(the content-scanned map answering a zero page where an injected pin
previously delivered the real method stream), starving the notifier ack.
Pin the page that delivered the first real decode (svc_read_last_pa at
the SET_NOTIFY_A capture) and the page that accepted the first ack
(svc_notify2d_write), so the every-pass ack keeps resolving.
Still observed: under 150 s of sustained GZDOOM the guest's krcWatchdog
can still report 13-17 timeouts and xshm_pub's XShmGetImage never
completes (the X readback op stalls), so the capture path is not yet
usable; the X server itself stays up (4 WAIT lines / 150 s).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
notifier ack)
The guest kernel's krcWatchdog posts a notifier cycle on its own channel
(object 0x31415900) every second and declares 'GPU is probably locked'
when the zeroed notification never comes back within 7 s. Observed as
the sustained-GL-load wedge: once a GL client (GZDOOM) enabled robust
channels the errors started and Xorg eventually stalled on screen
readback.
- rm: register the Volta GPFIFO class 0xc36f (plus the later
generations) as a channel class. The watchdog's channel allocates
0xc36f on this device - the driver picks the first class the gpu
reports supported - and without it the channel never reached the
progress service. It now registers and latches (put visible, entries
walked).
- sema: decode the class2d notifier cycle on subchannel 3
(SET_NOTIFY_A/B name the notifier VA, NOTIFY WRITE_ONLY writes the
zeroed NvNotification) and author the write, like svc_notify_complete
does for the work-submit token. Proven live: 'sema-notify2d: va=...
zeroed (RC watchdog acked)'.
- sema: also ack on every pass that sees the channel's put != get, so
the ack does not depend on the ring reads (its VA is known from the
first decode); idempotent.
- sema: never consume a zero-length/zero-VA GPFIFO entry as empty work.
Two bogus zero entries consumed through a mis-resolved ring page
pushed the watchdog userd's GPGet past the driver's own GPPut and
every later userd probe then read get>put, permanently rejecting the
channel's real userd. Hold such entries (bounded, per-channel
counter persists through the pass snapshot).
- sema: a registered userd value that is a real guest PA (the
watchdog's own block) outranks the ring+0x2000 convention guess the
ring auto-pin applies for the CUDA GR layout.
- rm: log channel-class allocs whose params are too short for
registration instead of skipping silently.
Remaining (next): under heavy GL traffic the watchdog ring's
translation can go stale for stretches (content-scanned map answering
where a pin used to), which starves the ack; pin the push/notifier
pages when the first real decode delivers them. The frame capture
(xshm_pub -> XShmGetImage) still stalls while that starvation is
active.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
push-scoped release deferral
The X server now starts on the emulated vGPU and GLX runs on the real
NVIDIA driver: Xorg :0 passes its DMA sanity push and ScreenInit, and
DISPLAY=:0 glxinfo reports 'OpenGL 4.6.0 NVIDIA 570.144' with direct
rendering; GZDOOM 4.14.2 launches and renders. Before this, every X
start died at 'NVIDIA(GPU-0): Failed to initialize DMA' (the
__nvPushTestPushBuffer 10 s GP_GET wait) and no nvidia-push kernel
buffer could be resolved.
What each piece fixes (all observed live on the r817->r827 cycles):
- sema: register the driver's PROMOTE_CTX context-buffer statements as
a VA->PA registry, consulted as the LAST resort after every walk.
A promoted (va, pa) pair is a VA-space booking, not a mapping: its
span can swallow other buffers' VAs (the ctx MAIN booking covered the
channel GPFIFO VA and range arithmetic answered the entry validation
with MAIN.pa+offset, a zero page - the latch never fired, r823), and
even its base page can host a separately mapped buffer (the PATCH
booking's base VA hosts nvidia-push's 4 KB progress semaphore; the
release landed in the booking's page while the driver's own polled
page stayed zero, r826). The walks describe the real mappings and
must answer first.
- sema: an all-zero GR push of <= 16 bytes resolved by the trusted
chain (src=4) is accepted as the driver's NOP sanity push
(NVA16F_DMA_NOP is a single zero dword; real launches are hundreds of
bytes). The r496e acceptance required a content-scanned ring, which
the X/GL channel is not.
- sema: the wave-1.5 release deferral is push-scoped. A release whose
push owns no captured forward candidate has nothing to wait for; it
applies inline (the progress-tracker semaphore release used to be
recorded and dropped at every pass reset while an unrelated lingering
candidate armed the global defer).
- sema: the m164 no-capture hold no longer arms for the accepted zero
NOP - it held the NOP entry for its 6 s window 'forcing capture' of a
launch that cannot exist.
- sema: once the ring resolves, the USERD latch scan may retry past its
8-scan cap (the client's VAS directory registration lost the race
against the 10 s DMA window when a rebuild stretched the pass cadence,
r820).
- sema: dbg-gated USERD-scan decision witnesses (rejection reason, sweep
summary, per-call state), GR-only and budgeted.
Known next issue (not this change): sustained X/GL rendering traffic
wedges the guest RC watchdog ('NVRM: krcWatchdog_IMPL: GPU is probably
locked') after minutes of GZDOOM/GLX activity; the FIFO service itself
keeps consuming.
|
| |
|
|
|
|
|
|
|
| |
The nvidia-push DMA test in Xorg screen init fails with 'Failed to initialize
DMA / Failed to allocate push buffer' (observed r816) because the X server's
pages are never pinned - the scanner excluded Xorg/X alongside the
nvidia-smi/persistenced transients. Xorg is long-lived and a legitimate GL
client of the vGPU, so it is now a valid pin target; the transient exclusions
stay.
|
| |
|
|
|
|
|
|
|
|
|
| |
split_screen.sh composes two guest frame streams (KVMFR PPMs from
frame_reader.py) into one hstacked mp4 with last-frame padding for unequal
lengths; build_video.sh wraps a title card, the main segment and an end card
(drawtext, detected font) into the deliverable mp4. Both verified end-to-end
with synthetic PPM sequences (ffprobe: 1280x720 h264, padding pixel-exact).
These are the capture-side pieces for the 1.0.0 video; the guest-side
publishers are tools/guest-diag/xshm_pub.c (X screen) and the existing
frame_producer; the host reader is tools/lg/frame_reader.py.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
foreign ring
The GR ring autopin fallback (SESSION 133c userd-shape scan) and the CE
first-submission capture each guarded block ownership only within their own
engine class, so a channel whose own ring was not yet written latched another
channel's live block permanently: the GR channels drained the CE rank-0 ring
(or an unlatched CE rank's convention block), the real launch entry was never
read, and CUDA clients hung after their first device query (victory_mail
stalled in UVM channel_pool_add; every post-reboot boot r805-r811 showed the
same steal, e.g. 'auto-pinned GR ring h=0x5c000019 rank=0 ... -> fb 0x80000
(userd-shape scan ... entry0 va=0x120000000 len=40 put=35 get=35)' while the
CE handle decoded the identical entry[35] raw words from pa=0x10350641c).
Changes:
- shared FB block claim table (svc_fb_claim_add/_foreign/_reset), session
scoped, consulted by both GR scan passes and the CE fallback scan; every
live CE rank's convention block is reserved even before that rank latches
(r809: CE rank 1's block at fb 0xa0000 was stolen pre-latch);
- every GR autopin latch (userd-derived, FB convention, pin, trusted walk,
scan pass 1/2) and both CE latch paths record claims;
- CE first-submission capture: a rank's own convention block is latched even
when its stale content probes incoherent (the previous session's consumed
state), instead of falling through to a never-used zero block (r812/r813:
client 2's 0x90000 block read put=1/get=146 at the capture instant and the
zero 0x82000 block was latched, hanging the client).
No module parameters or defaults change; the guards are additive.
Validated (plain mdev, script defaults):
- victory_mail GREEN repeatedly: 'VADD n=32768 bad=0 checksum=12204516.000000'
/ 'STATUS OK', all six GR ranks latched at their own fb 0x140000+0x3000*rank
rings with 'launch captured h=0x5c000019 push=0x20043be94 len=840';
- a second sequential client on the same boot now progresses (vecadd_loop:
'VERIFY launch=10 bad=0' - the 100-launch loop needs a long budget).
|
| |
|
|
| |
FAULT_PDE at the program VA), Vulkan copy channels registered/swept/never pushed, no program bytes on any channel; frontier = upload copy + program-VA mapping + restore the black-frame baseline
|
| |
|
|
| |
channels are registered/swept/never pushed; no channel carries program bytes or a LAUNCH_DMA; the code page never fills; the refusal path no longer reaches the black-frame baseline (SYNTH-FAULT vs completion ordering)
|
| |
|
|
| |
lines for every channel + a 128-row per-pass cap, so a channel absent from the log can never be confused with a channel not in the table (the r802 copy-class channels 0xbeef0101/2 visibility gap)
|
| |
|
|
| |
min_data=0 reaches the real engine, which MMU-faults (FAULT_PDE) at the guest program VA; the wall is now the program page's mapping and content (SASS upload), not the data-set gate
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
no-external-data launch class), not a clamp to 1
The Vulkan compute launch's single QMD pointer pair names its own QMD
window and its params are scalars (0.2/640/480), so n_data=0 is correct
for it; the bare n_data>=min threshold refused it as degenerate. The
bypass is safe now that gsp_phys_chunk_page_va answers NULL for an
unallocated chunk (r800) - before that guard the empty-set post-mortem
print dereferenced a garbage pointer (Oops at
gsp_phys_backend_guest_gr_submit+0x4394). Default stays 16; 0 is opt-in
for the egg-break test that forwards the launch to the real engine so
the copy-out can carry pixels back. Proper launch-internal predicate
next.
|
| |
|
|
| |
unallocated chunk (no per-page array AND no base va) - it previously returned (u8*)NULL + page*PAGE_SIZE, a small non-NULL garbage pointer, so the caller's NULL guard passed and the empty-set post-mortem print dereferenced it (Oops #2 at gsp_phys_backend_guest_gr_submit+0x4394, CR2=0x1e93c); with the guard, min_data=0's no-external-data forward can proceed
|
| |
|
|
| |
(QMD-page note + qmd-ptr seed + pin-aware reads) reaches the launch's data set; the empty-set forward reaches the backend and oopses at gsp_phys_backend_guest_gr_submit+0x4394 - guard first
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
seed, pin-aware scan reads
Three corrections that take the GR fill from 'no candidates' to the
launch's real pointer set:
1. Note the QMD page itself into the request's page list (word 34) so the
32..48-even bank scan covers it - the pointer VALUES live inside the
QMD (r790: word 34 = 0xdff0febc00) and were never scanned.
2. Seed the fill's candidate set directly from the launch's own QMD
pointer pairs (gr_qmd words 34..47, decoded from the push's inline-QMD
run): r793 'sema-gr: qmd-ptr seed va=0xdff0feb000 (word 34)' - the
values need no page scan (the pages' shadows read empty; the invisible
-vidmem-write class).
3. Read the scanned pages pin-aware (svc_read_va first, host mapping of
pp->pa as fallback): the push-buffer discovery's pinned staging copy
holds the content the decode reads; a raw mapping of the vidmem pa
sees zeros.
Verified r793/r794: candidates now appear (fill cand n=1 with the QMD
window page seeded); the page is correctly classified launch-internal by
the R3 hygiene filter (n_data stays 0 - nothing external to relocate for
this launch). With the r564 degenerate-set guard bypassed (min_data=0),
the forward ENTERS the backend and oopses in
gsp_phys_backend_guest_gr_submit+0x4394 ('Oops: 0000 [#2]', workqueue
gsp-phys-work svc_fwd_work_fn, NULL-base table walk CR2=0x1e93c R14=0) -
the empty-set path needs that guard before min_data=0 can be used.
This launch's QMD declares exactly one pointer pair, and it names its own
QMD window: a launch with no external data is legitimate and the refusal
class must not treat it as degenerate.
|
| |
|
|
| |
480.0f; the float-shape filter empties the false candidate set; the wall is content (i2m never queued / forward refused first)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
deduped word-48 page
Two evidence-based corrections to the launch-data candidate scan:
1. Float-shape filter at all four scan sites (the three page scans and
svc_gr_scan_cand): a word whose f32 exponent byte is in ~[0x6c..0x8f]
(2^-19..2^16) is a plausible float scalar, not a device VA - device
VAs (2^28..2^47) as f32 land in subnormal/tiny exponents. Root
evidence (r789): the launch param page carries the kernel's scalar
arguments (0.2f, 640.0f, 480.0f at +0x630 - the 640x480 viewport) and
0x43f00000 = 480.0f was the fill's ONLY candidate - an unresolvable
"arg base=0x43f00000" that drove the fail-closed diagnostic and the
demand-loop noise for days. After the filter: 'fill cand n=0'
(r790) - no false candidates, no false demands.
2. The bank-page scan now covers word 48: the QMD pointer-region pair can
name a 4K page that also carries the program pair (r788: ptr[0]
0xdff0febc00 word 34 shares its page with prog 0xdff0feb900), and
svc_gr_page_note dedupes the first note (the program's, word 48), so
the old 32..47-even filter left that page unscanned.
The remaining wall is content, not wiring: with the false candidates
removed the launch still refuses (n_data=0 vs min 16) because no scanned
source holds real device pointers - the param page holds scalars, the
QMD window page reads zero (its content would arrive via the driver's
i2m/LOAD_INLINE_DATA writes whose payloads are not in the push), and the
inline payload is the driver's zero stub. This matches the SESSION 130
zero-buffer finding and leaves the never-queued i2m copies (or the real
GPU path) as the only content sources.
|
| |
|
|
| |
printed (param/prog pages, deduped QMD window, single inline candidate, n_data=0 min 16 refusal); the window content exists in no host-visible source
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(sema_dbg-gated, bounded)
The dstpend/n_data frontier needed the launch's actual wiring. Prints, once
per launch within a bounded budget: the candidate handle, param_va, prog_va,
n_pointer + the QMD pointer pairs, the inline-rec count, and the resolved
gr_pages list (va/pa/ap/qmd_word); then in the fill, the built candidate
set with the arg flag.
First capture (r788, Vulkan launch h=0xbeec0100): param=0x4b86000 ->
page pa=0x386000 ap=2 qmd_word=32; prog=0xdff0feb900 -> page
pa=0x56b000 ap=2 qmd_word=48; ptr[0]=0xdff0febc00 word=34 (DEDUPED into
the qmd_word=48 page - same 4K page - and that page is never scanned by
the fill's word-32/32..47-even passes); the fill's only candidate is
0x43f00000 (arg=1, from the inline bytes) which resolves in NO vaspace
(rc=-2 in all four) -> n_data=0 -> the backend's min_data=2 refusal. The
param/prog/QMD-window pages read all-zero in the vidmem shadow on this
boot, so the launch's data/code bytes exist in no host-visible source.
|
| |
|
|
| |
for the Vulkan client is disproven by direct write watches (channelFillGpFifo is notrace; the entry store never executes; the client's ring mapping watchpoint catches zero writes); the frontier moves to the GR launch's dstpend/n_data page dependency
|
| |
|
|
| |
latch on record was a stale CUDA block; the ring content is in no shadow, no RAM, and not at the PTE target (BAR1 write path verified working; client-side write watch is the frontier)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
USERD candidate (get >= put > 0)
At the first-submission instant the channel's own submission is by
definition unconsumed - the notifier clear this capture runs before is
what would let the driver treat it as consumed. A candidate whose USERD
reads a consumed state (put == get > 0) cannot be this channel's live
just-submitted work.
Observed r785 (CUDA-first boot, three Vulkan runs): the capture latched
fb 0x80000 / userd 0x90000 for beef0101 (rank 0) with put=1 get=1 - the
dead CUDA client's caf00012 block surviving in VRAM after its session
ended. The decoded entry (va=0x120000000 len=40) was the old session's,
the expectations seeded from the stale pool word (want=2/3 vs pay=1) and
the Vulkan client stalled into its fence timeout on every run.
All genuine first-submission latches on record read put=1 get=0 at the
capture instant (r779 caf00012 rank0, r785 caf00012 rank0, r781/run1
beef0101). Idle 0/0 stays acceptable per svc_userd_probe's own rule.
Session-scoped rank ownership (svc_ce_firstsub_off) does not cover this
class: the stale block belongs to a previous CLIENT, and its consumed
state is the only local signal that it is not this submission.
|
| |
|
|
|
|
|
|
|
|
|
| |
first-submission capture caused the CUDA stall regression (fixed, a674c21c); the Vulkan CE rings live at 0x4020000/0x4100000/0x4160000 (above the 64 MB capture bound) and the frame chain reduces to the un-latched ring (copy pend -> dstpend -> black)
Records the instrument, the regression A/B (r746-era module passes, SESSION
133/134 code spins), the rank-aware fix and its r781 verification, the
Vulkan client's actual ring allocations from its own vidheap ioctl log, the
failed 4 GB RAM entry search, and the live frame dependency (GR launch
inline=8 + copy pend src 0xdff0d60000 -> vidmem 0x570000, dst framebuffer
0x50b0000) with ranked next steps.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rank-blind first-hit latched rank 0's block for rank 1
The SESSION 134 first-submission capture scanned the shadow from offset 0
and took the first ring-shaped page for EVERY CE channel. On the r779 boot
that latched rank 0's block (fb 0x80000, userd 0x90000) for rank 1 as well
(both rows scan_hits=1 latch=1 off=0x80000), so caf0001d's first release was
decoded from rank 0's ring/page (sem word +0 instead of +4) and
stale-dropped (cur=1 already from rank 0's applied release). The client's
rank-1 tracker wait never completed: victory_mail spun at 100% CPU with
pub_calls=0 on three consecutive boots (r777/r778/r779), while the r746-era
module (25523f84) passed the same run on a fresh boot (VADDs OK, STATUS OK).
Fix: try the rank's convention offset (0x80000 + 0x20000 * rank) first,
scan the rest bounded as before, and never accept an offset another CE rank
already latched this session (svc_ce_firstsub_off[], reset per session). A
rank whose block is not in the shadow at the instant stays unlatched for
the regular per-pass CE latch instead of stealing a foreign block.
Verified on r781 (fresh boot, same params xlate=1/util_scan=0/vas_cand_ms
=5000/ce_grace=20000): caf00012 rank0 -> fb 0x80000 upa 0x90000;
caf00001d rank1 -> fb 0xa0000 upa 0xb0000 (correct block, sem +4 applied:
pool-wr va=0x121000004 cur=0 -> val=1 ok); ranks 2-5 -> 0xc0000/0xe0000/
0x100000/0x120000 each +0x10000; r780 (r746 module) control run: same
client PASSED (VADD n=32768 bad=0 checksum=12204516.000000, STATUS OK);
r781 client progressed with pub_calls=185132+ (no spin stall). Also the
Vulkan client's CE channel latched via the classic autopin on r781
(beef0101 -> fb 0x80000, live entry0) - first time in the SESSION 131-134
window - confirming the latch machinery works once the wrong-block
mis-latch no longer poisons the channel.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
hva truncated the 40-bit VA so no CE-window entry could ever match)
hva must hold the 40-bit push VA (bit 32 set for the CE window) but was
declared u32 in the scan, so ((h1 & 0xff) << 32) truncated to the low
dword and hva < 0x120000000ULL was always true - scan_hits stayed 0 by
construction. Same decode as svc_chan_userd/svc_gr_ring_candidate (u64).
Verified building; the FB-convention latch itself (auto-pinned ring
h=0xcaf00012 -> fb 0x80000, userd 0x90000, entry decode + push decode,
ops_runs=1) was observed independently on r776.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- the CE ring latch instrument
The CE-class ring (the SASS-delivery channel's GPFIFO, registered at
gpfifo=0x4100000 on every boot) only ever latched on one boot (r763, the
FB convention matched shadow 0x80000 and the sweep decoded the inline
LAUNCH_DMA entry). On r765-r775 every deterministic source reads a zero
page (the page tables genuinely map the ring VA to an all-zero RAM page -
proven by the per-level walk trace) and the ring never appears in the
shadow (scan_hits=0 at the submission instant; dump_vidmem map shows only
the USERD put words) nor in the guest RAM image (a full 4 GB scan for
consecutive GPFIFO entry pairs around the CE push window finds none).
New bounded instrument, default-inert:
- svc_ce_firstsub_capture: on a CE channel's notifier clear (the one
instant ring content and USERD are both live), before the clear write:
reads entry0 through the channel translation and the trusted walk, then
scans the shadow (0x100 step, 64 MB) for a CE-window ring page
(1MB-aligned push VA in [0x120000000,0x121000000)) and latches via the
standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0. Budget 4/channel,
16 witness rows/boot, reset by svc_session_flush.
- sema_ce_hold_ms (module param, default 0 = off): holds an unlatched CE
channel's notifier busy for up to N ms so the guest keeps the
submission in flight (and with it the ring content) while the VAS lands
and the regular per-pass latch path retries; expires into the old
clear behaviour.
r774/r775 evidence (sema_ce_hold_ms=8000): sema-cefirst at the first
submission shows rrc=-2 (no VAS yet at +0.2 s), wrc=-1 (walk unresolved),
scan_hits=0 (no shadow ring); 30 s later the walk answers a zero page
(rpa changing per boot, e0=0x0), the hold expires unlatched, and the
client still completes (VKFRAME_DONE) - the hold is safe but the ring's
write path is still not visible to the emulation. Remaining unknown is
named: the driver's ring writes reach neither the shadow nor the
PTE-mapped guest page.
|
| |
|
|
| |
USERD by its own shape (put@+0x8c, get@+0x88, the find_userd layout) and derive the ring at userd-{0x10000,0x2000,0x1000}, validating entry0 (shape + not provably dead) and the USERD probe; the USERD words appear at channel creation and stay while the ring entry may be written later, so this is the stabler trigger for the CE-family rings that the periodic ring-entry scan misses
|
| |
|
|
| |
is FOUND (the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push: in=0x120a00004 -> out=0x205000, 4096 B/run; the Vulkan CE channels carry the same-shaped entries) and the three gates that keep the copies from running are named: the CE ring/USERD boot-lottery (FB convention matched r763 only; the real USERDs sit at 0xfee00/0xff000/0xff200 so the rings are at unaligned offsets like 0xefe00), the VAS age-out vs CE grace race (sema_vas_cand_ms=5000 registers at +5s), and the volatile ring/USERD content that a periodic shadow scan misses; next steps: USERD-shape-derived ring resolution or first-submission latch, then the inline copies and the n_data wiring
|
| |
|
|
| |
unaligned: r767 USERD shape scan found the channel blocks at 0xfee00/0xff000/0xff200, so the rings sit at offsets like 0xefe00) and retry rate-limited (1.5s) instead of a consumed once-bit, because a first pass can run before the client has written the ring or its USERD
|
| |
|
|
| |
rings live in VIDMEM and the FB convention only matches when the guest VRAM allocator happens to place them at the conventional offsets (r763: matched fb 0x80000/0xa0000 and the CE channel decoded its inline SASS copies; r765: read zeros, the ring VA resolved to a stale RAM page whose entry0 was dead, the CE channel never latched - black frames). When every deterministic candidate fails, scan the first 64 MB of the shadow for a ring-shaped page (entry0 names a 1 MB aligned push VA in the client window with a sane length, not provably dead via svc_ring_entry_dead) whose USERD at the observed deltas (+0x10000 CE family, +0x2000 GR, +0x1000) reads a coherent put/get; one scan per rank per boot, re-armed per client session. A false candidate still costs only a failed probe.
|
| |
|
|
| |
PROVEN in guest RAM (walk-confirmed per-run pages 0x11623d000/0x104b55000 with the 14-entry content the sweep decodes; the vidmem 0xa0000 'ring' was a false-positive autopin whose entry0 push reads all-zero and which the pump skipped after 39 rescues); the fixes (dead-entry guard + cross-aperture USERD + per-client util witnesses + op-level copyent) and their verification (r760: ring rc=0 on the live RAM page, live entry reads, VKFRAME_DONE 3 frames); the two LID uploads are the driver's zero buffers by design (256 B zero buffer + 12288 B CE fault buffer) while the GR channel's LID carries the real param bank; the code VA (prog=0xdfd8feb900) maps to a ROTATING code pool (0x62b900 -> 0xc8b000) whose lat entry can go stale and whose pages are empty - the SASS delivery shape (C5C0 0x3800/0x39C8 runs) and the walk-first code-VA resolution are the next two gates
|
| |
|
|
| |
r759 false positive is fixed at three points: (1) a ring candidate whose entry0 push provably reads all-zero through the client's own page tables is rejected when the VAS walk resolves the ring VA to a page with a live entry (svc_ring_entry_dead + svc_ring_walk_live, applied at the userd-derived, FB-convention and existing-pin paths; the walk cannot vote for itself); (2) svc_gr_ring_candidate now rejects dead-entry candidates too and accepts the channel's known USERD cross-aperture (r759: ring in guest RAM 0x11623d000, USERD in vidmem 0xb0000, so the walk-resolved ring failed the same-aperture USERD probe); (3) the util sweep arbitrates a dead channel-path ring entry against the direct VAS walk per slot. Also: util-dec/util-sweep witness budgets and their dedup keys become per-client (reset in svc_session_flush) and the sema-copyent witness logs once per copy op instead of per row (a 480-row frame copy alone exhausted the 128/session budget and hid every later copy, including any SASS upload)
|
| |
|
|
| |
frontier: the CUDA client's uploads fill the emulated vidmem (~450k non-zero dwords across many 1MB blocks) while the Vulkan client's leave only the 14 baseline dwords, so the Vulkan (NVN/GL) upload path is not among the writes the emulation can see; next steps recorded
|
| |
|
|
| |
(nz count + first offset) to locate the live client's uploaded regions in the 256 MB aperture without 65536 range dumps
|
| |
|
|
| |
in vidmem (map PA 0xa0000, live entry 0x120100000/40) and the decoded 14-entry LID ring is a STALE page from a dead session, which is why its payload chunks read zero through every path (four reads + BO poller + gdb write-watchpoint); the CUDA contrast proves the LID arm works with real contiguous payloads; next step is reading the live ring through the map and decoding its entries
|
| |
|
|
| |
LOAD_INLINE_DATA arm fires (branch-order fix), the payloads are the driver's zero buffer (four-read proof + full-run BO poller + gdb write-watchpoint), the SASS delivery is the remaining non-black gate (prog page 0x62b000 empty; NVN download path targets the pitch-pool allocation), plus the xlate/util_scan regression bisect result
|
| |
|
|
| |
regressions were re-tested on a fresh boot: with the translator ON and the utility sweep OFF, victory_saxpy and victory_math PASS with checksums identical to the gate-off baseline (saxpy csum=981440890.500000), so the SESSION 129/130 sweep changes fixed the saxpy/math half; the remaining xlate-on regression is victory_multikernel, whose physical forward times out repeatedly (GR LAUNCH FAILED ... qmd_rel=0xc0ffee01, hold active), so the default stays 0 until that forward path is fixed. The UTILITY SWEEP is the other regression source (util_scan=24 broke saxpy bad=1047527 and hung math) and stays opt-in until its channel reads prefer the direct walk
|
| |
|
|
| |
a zero-headed tail-served payload is now applied under a variant once-key (h | 1) instead of being refused forever (the destination is empty, so authoring zeros is idempotent and correct for the driver's internal zero buffer, whose 256 B i2m equals GF100_ZERO_BUFFER_SIZE exactly), while a later non-zero payload for the same entry still matches its own (h) key and lands; the utildec witness now carries alt=success/tries so a walk-read that fails is distinguishable from one that returns zero
|
| |
|
|
| |
GRCE operand group was swallowed by the TURING-copy branch (its `subch >= 2` guard also matches subchannel 2, leaving have_out/line_len unset so the LID arm never fired: r739 witness `lidn=1 subch=2 saw=0 dst=0 ll=0`); the 0x180..0x18c group now parses FIRST for subch 1|2, plus tail-content arbitration (a zero-headed channel tail is re-read through the direct registered-VAS walk before the arm's zero-refusal can stall the upload) and the witness now carries the served tail's first two words + the alt-read counter
|
| |
|
|
| |
end-to-end (VKFRAME seq/DONE/IDLE, three PPMs); the full fix chain (push-parse shapes, fence fold sema_gr_ahead, VAS pdb anchoring, COMPR downgrade, scrub-author, 9096/ZCULL, C5C0 translator); non-black frames (MME/i2m LOAD_INLINE_DATA arm) is the frontier; CUDA 8/3/1 fresh-boot battery; video v1 on files.vgpu.io
|
| |
|
|
| |
the subchannel-2 arm never fired (r737 writes=0) because the utility channel's read path could not produce a parseable methods entry (the tail-served split shape); the util sweep now arbitrates attempt A (channel path, unchanged) against attempt B (direct registered-VAS walk, the walk_probe-proven path) and lets the strict parse choose the stream, a still-zero tail-served payload is refused (no once-store, retried every pass; in-push payloads untouched), and a bounded deduplicated sema-utildec witness names per distinct entry: winning attempt, both parses, tail VA/length, LID header count/subch and the arm's saw/fired/zero state
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(sema_gr_util_scan, default 0) - decode the driver's engine-side MME/i2m upload path and materialize LOAD_INLINE_DATA downloads and pitch copy-ins from channels the put-bounded entry loop never decodes
The Vulkan client's code/constant uploads leave the utility channels (the
MEM2MEM/A140 composite at h=0xbeef0100 and the eng=11/12 copy channels)
and were invisible to the emulation: with an unresolved USERD the
put-bounded entry loop skips those channels entirely (the !userd_ok branch
continues), and with a resolved USERD the main decode's operand group and
LOAD_INLINE_DATA arms only accept subchannel 1 while the uploads ride the
mem2mem/copy subchannels (2/4).
svc_gr_util_sweep scans up to sema_gr_util_scan (default 0 = off; requires
sema_gr_xlate) GPFIFO entries of every channel directly, reads each push
plus its successor as the tail (the driver's source-view GPFIFO call
carries the payload of a trailing LOAD_INLINE_DATA run - observed
beef0100 idx00->idx01 and idx08->idx09), and feeds them through
svc_gr_ahead_push in a new inline-only mode: the GRCE/INLINE_TO_MEMORY
operand group now also accepts the mem2mem subchannel, the QMD window,
release and copy-pend arms are skipped, and a TURING_DMA_COPY_A launch on
the copy subchannels materializes the engine copy-in (src from the
client's staging buffer through the trusted read, dst through the same
svc_grce_inline_write the GR sweep uses) instead of queueing a launch
copy-out. Every materialization is once-only keyed by (push_va, content
hash, dst, bytes); SVC_AHEAD_TAIL_INLINE_MAX rises 4 KB -> 32 KB so the
12 KB code-staging payload is not truncated.
Verified on r737 (gated): the sweep runs for all ten channels (utility,
CE and GR) with bounded reads and no release/QMD state changes; the
utility channel is swept 14 entries per pass. The LOAD_INLINE_DATA arm on
the utility channel's subchannel-2 runs still does not fire (payload/parse
witness is the next debug item, see Handoff SESSION 128).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sema_gr_xlate (default 0)
The r727 full battery with the translator active but ungated regressed three
clients (saxpy bad=1047527, math sums=0, multikernel hang) while r729 with
the translator's semantics gated off is clean on the same subset (mail,
saxpy, math OK; multikernel still stalls in its pre-existing flaky class).
All translator behaviour is now behind sema_gr_xlate (default 0 = exactly the
historical sweep):
- the tail-aware method parse (tail passed only when enabled),
- the sweep's inline-QMD accumulation, LOAD_INLINE_DATA materialization,
C5B5 copy capture and candidate-owned release withholding,
- the client-VAS walk fallbacks in svc_gr_page_note / svc_gr_req_fill,
- the inline-rec filter at capture (keeps resolvable or QMD-named recs when
enabled; historical keep-all when off),
- the pump's sweep-captured-launch request and copy fallback.
sema_gr_xlate=1 restores the full translator (r730 re-verified: launch
captured, page-notes, copy pend/done, submit refused n_data=0, release
authored, client completes). Flip the default once the two CUDA-side
hazards (stale-VAS resolution widening, rec-filter interaction) are
eliminated and the battery is proven clean with it on.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
captured, filled and forwarded (and its copy-out executed) through the CUDA QMD machinery
The Vulkan client's compute dispatch is an inline-QMD launch that lives in
the PRE-BUILT ring continuation beyond GP_PUT (the earlier analysis only
inspected the put-bounded pushes' idx7/idx12 and concluded 'no inline QMD'):
beec0100 idx23 carries SET_INLINE_QMD_ADDRESS_B (0x31c = 0x4b85f, the upper
half absent) plus the 64-word LOAD_INLINE_QMD_DATA (0x320) run - grid 40x30,
prog=0xdfd8feb900 (the SASS page), param=0x4b86000, release0=0x4b85e08/1 -
and idx24 carries the TURING_DMA_COPY_A (0xC5B5, subch 4) image->buffer
copy-out (src=0xdfd8d60000 dst=0x050b0000, 640x480 RGBA rows) plus the
SEND_PCAS schedule (0x2b4 = qmd>>8, 0x2bc = SCHEDULE).
Pieces landed (all service-thread, all bounded/logged):
- svc_gr_ahead_push (the sema_gr_ahead sweep parser) is now a strict
tail-aware translated decode of one continuation entry: it accepts a
method stream whose final inline run is completed from the NEXT ring entry
(the driver splits big LOAD_INLINE_DATA payloads into a dedicated data
slot entry: beef0100 idx00->idx01 256 B, idx08->idx09 12288 B), feeds the
C5C0 inline-QMD window to svc_qmd_accum (capture via the standard CUDA QMD
candidate), materializes GRCE/KEPLER_INLINE_TO_MEMORY_B LOAD_INLINE_DATA
writes into the VA shadow exactly like the main decode (once-only keyed;
recorded in the dst ring for the fill's inline-served pages), parses the
C5B5 copy operands (0x400/0x404/0x408/0x40c/0x410/0x414/0x418/0x41c,
0x300 LAUNCH_DMA, 0x700/0x708 remap) into a pending copy list, and queues
the release groups minus a release a live launch candidate owns (that word
belongs to the completion edge, so the client's fence can never clear on a
pre-render image).
- The parser reads the run data words correctly (w3 fix; the GRCE 0x18c
operand was being read from the next run's header word, which sent the
1568 B parameter-bank write to a bogus destination).
- svc_qmd_accum: accept the ADDRESS_B-only inline-QMD form (implicit zero
upper half) so qmd_dev_va is real for the Vulkan launch.
- svc_gr_page_note + svc_gr_req_fill: resolve a client-VAS page through the
registered-VAS walk when the trusted sources do not carry it (the Vulkan
QMD's program/parameter/descriptor pages are client-managed vidmem; the
fill refused the whole launch with -ENOENT before this).
- The QMD capture's inline recs are filtered to the pages the captured QMD
names (a stale session-wide rec destination made the fill refuse with
ENOENT va=0x80412000 qmd_word=NONE).
- Pending GRCE copy-out list executed on the launch completion edge
(svc_fwd_hold_gr_release and svc_fwd_outcome_check) before the guest
release is authored, with a bounded no-forward fallback at the sweep; the
copy itself runs through the standard svc_copy executor (observed:
'copy done h=0xbeec0100 dst=0x50b0000 bytes=0x12c000 rows=480 rc=0').
- The pump offers a sweep-captured launch every pass (svc_gr_request) - the
continuation launch's only dispatch path - and retires pending copies when
no forward is owed.
- gsp_phys_gr_min_data (module param, default 16): the degenerate-data-set
refusal threshold is now tunable; the Vulkan dispatch's operand set is a
handful of records, not the CUDA-era 968.
- Bounded witnesses for the request paths (first-N/periodic: no-candidate,
backend-not-ready, hold-busy, inflight-busy, fill-ENOENT) and the capture
tail dump is now sema_dbg-gated with a 4/boot budget (it flooded dmesg).
Live status (r726, cycles r719..r726): the client runs green through the
fence (VKFRAME seq/DONE/IDLE, 3 PPMs) with no in-flight client assist; the
sweep captures the launch ('launch captured ... qmd=0x4b85f00
prog=0xdfd8feb900 param=0x4b86000'); the fill resolves the QMD pages
(page-note src=walk) and now passes its ENOENT gate; the copy-out executes
for every frame. The physical launch itself is still refused: the fill
yields n_data=0 because the launch's operand CONTENT (the SASS at
prog=0xdfd8feb900, the descriptor data) is delivered by the driver through
the MME / inline-to-memory engine upload stream (beec0100 idx7/idx12,
beef0100 idx08), which the emulation does not execute - the pages the QMD
names stay zero in the emulated vidmem. That engine-side upload path is the
next work item; sessions r719..r726, evidence in the handoff.
No host reboots; 0 oopses; the fetch of the CUDA battery on a fresh boot
follows separately.
|
| |
|
|
| |
(sema_gr_ahead sweeps the pre-built GPFIFO continuation beyond GP_PUT and applies the submit pushes' INC-5 RELEASE groups forward-only through the pending ledger; VKFRAME seq/DONE with no host assist) and the compute-dispatch shape decoded: a direct C5C0 method stream (SET_OBJECT + 0x7b4/0x2a4/0x270 windows + 0x3800/0x39C8 code runs + a subchannel-0 data-port stream), NO inline-QMD write, so the existing launch capture/physical forward never fires - the translator that builds the vendor launch from the C5C0 section is the execution gate
|
