summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* tools/cuda-ab: memset_probe + wedge_recover forensics clients (session 10)Arthur Hanson Rasmusson7 hours
| | | | | | | | | | | | | | | | memset_probe: linear 8 MB cudaMemset + D2H with first-bad offset and a 4 KB page map (correct / other / malloc-prefill) - separates 'fill under-delivered' from 'readback delivered wrong pages'; standalone green, bisects the memcpy_roundtrip memset phase. wedge_recover: launch+verify loop that on failure sleeps N seconds and retries the same iteration (same context), reporting per-iteration attempts and launch times - tests whether an idle gap heals the per-boot no-execute wedge (r409 log shows a ~37 s client-side gap healed a wedged run). Also ships the missing inplace_scale run binary (its source landed in 7685d1b3).
* tools/cuda-ab: inplace_scale - minimal in-place scale repro for the r407 ↵Arthur Hanson Rasmusson10 hours
| | | | | | | | | drop-stale release race One H2D, one in-place scale launch (128 KB = 32 pages), one D2H, per-page diff. The client made the r406/r407 stale-release race reproducible in isolation: with the drop-stale fix the same binary runs 6/6 + 4/4 green across two fresh boots (was intermittent 2/3 before).
* sema: r408 acquire-wait gate defaults ON (the r408 streams2d readback race) ↵Arthur Hanson Rasmusson11 hours
| | | | | | | | | | | | | | + explicit param desc With the gate off (the mission cycle recipe omitted sema_wait_gate=1), the c2 readback's C56F ACQ_CIRC_GEQ pay=4 ran at 106617.279 with cur=2 - before the c2-add launch was even submitted (106617.411) - and read the not-yet-produced buffer (all zeros). With the gate on the same readback holds until the scale channel's progress counter reaches 4 (sema-wait: hold ... cur=2 waited_ms=353 -> pass ... cur=4) and the ordering is correct. The holds are bounded (sema_wait_hold_ms, default 4000) and fail open loudly, so the pre-r336 pass-through remains available with sema_wait_gate=0 for an A/B.
* sema: r407 drop-stale must not deliver a held entry's releases on a rewind ↵Arthur Hanson Rasmusson11 hours
| | | | | | | | | | | | | | | | | | | (only consumed entries are stale) The r406 inplace_scale repro root cause: the H2D copy's operand translation was unresolved (memcopy rc=-2) and the entry was held with its release withheld (correct); a pin-batch rewind then re-opened lower indices (83) while the held entry (91) sat ahead, and the unconditional svc_opdef_drop_stale treated 91 as an already-consumed entry and applied its withheld release at 104955.751. The copy itself only executed at 104957.418 - the client raced past its blocking cudaMemcpy, launched, the launch's copy-in read the stale buffer (previous run's data) and the copy landed after the launch's copy-out: wrong results with a healthy-looking 32/1032 changed and no classifier firing. An entry is only 'passed' if the ring consumed it (slot_done_epoch); gate the stale drop on svc_replay_seen so a rewind can never deliver a still-held entry's release early.
* gr: r405b late-write probes - retire LATE-PROBE (+500 ms re-scan) and a ↵Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | tunable settle window (gsp_phys_gr_settle_ms) The r405 streams2 forensics show a new shape of the no-write class: the in-place scale launch wrote 1 of its 32 expected pages and the partial-changed count was content-stable for 300 ms, so the retire wait declared 'settled' and the copy-out shipped one page. Two runtime knobs discriminate and test the fix in one cycle: gsp_phys_gr_late_probe=1 re-scans +500 ms after the retire wait and logs whether the engine was still writing; gsp_phys_gr_settle_ms raises the content-stable window (default 300 ms).
* gr: r405 restore the no-write classifier input - propagate the tracked page ↵Arthur Hanson Rasmusson12 hours
| | | | | | | | | | | | | | | | total to the retire-wait out-param ed91540a (r345) changed gsp_phys_gr_scan_pages to return the changed count and report the tracked total through an out-param, but the caller saved it into a local and never wrote *n_tracked. Every launch since reported n_tracked=0, so the caller gate (wrote == 0 && n_tracked) was never true: the first-attempt witness, the outcome re-execution retries (fresh qmdwin bank + release zeroing + re-publish), the NOWRITE accept with release-word restore, and the late-store check have all been dead code. The r404 forensics show the regression directly: 'RETIRE-WIT n_data=968 n_tracked=0' beside 'output retirement wait: 0/1032 pages changed (no writes)' for the same launch. Propagate n_all so the classifier fires again.
* docs: GPUBreach-class audit and threat model for the host write pathsArthur Hanson Rasmusson14 hours
| | | | | | | | | | | | | | | | Audit of every host-side write whose target derives from guest data, against the GPUBreach (IEEE S&P 2026) chain: GPU PTE corruption -> arbitrary device read/write -> driver bugs -> CPU root. Findings: the host is structurally off that path (guest GPU page tables host-emulated, physical GPU PTEs host-owned, every write aperture-bounded by vfio_dma_rw to guest RAM or the vidmem shadow, no guest DMA to host buffers), and the remaining wrong-but-valid write class is bounded by the per-page resolution + refusal guards (FILL-TRUNC, FILL-PT-REFUSED, LOWFILL-REFUSED, GRFILL-DROP, stalled-VA demand, sema_copy_dst_pin_guard, DST-DIVERGE). Documents residual risks, what is not claimed (no isolation or side-channel claim; physical Rowhammer is out of scope), and three follow-ups.
* sema: r402 pitched (2D) fills/copies + copy-dst divergence witnessArthur Hanson Rasmusson15 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two additions for the memset2d/memcpy_roundtrip and readback classes: 1. Pitched transfers. The CE decoder now captures LINE_COUNT (0x41c), PITCH_IN (0x410) and PITCH_OUT (0x414); a fill/copy whose LINE_COUNT is > 1 and whose destination pitch exceeds the row length builds a row-wise op (svc_fill/svc_copy are thin wrappers around the existing 1D bodies, advancing dst by dpitch and src by spitch per row). Each row keeps the whole per-page resolution/ledger/guard machinery, and a failed row's retry re-runs already-written rows with the same pattern (idempotent). A flat transfer carries pitch == row length (or no pitch method) and keeps the historical single-run path unchanged. Bounded 'sema-2d' row names every 2D op build. Note: the current victory_memset2d traffic reaches the engine as GR kernels plus per-row 4 KB CE copies, so this capability does not yet trigger for that client - it is the correct decoder for the CE-class 2D shape (driver channel_utils.c memset2d emits LINE_COUNT/PITCH). 2. DST-DIVERGE witness: when a copy destination resolves from the write ledger, a read-only resolver probe cross-checks the answer and logs (bounded, gsp_sema_dbg) when the writer's page differs from the live resolver's - the 'writer follows the ledger while the reader holds another page' shape behind the failed D2H staging class (streams2 c1 zeros, smiley pass failures). Zero rows fired on r403, so the ledger is not the active divergence for that boot; the witness stays for the next failing capture.
* sema: r401 acquire-wait hold bound + session-wide exactly-once capture memoryArthur Hanson Rasmusson15 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two fixes for the streams2 / multi-stream ordering class, both verified on fresh boot r401 with the acquire gate enabled: 1. sema_wait_hold_ms (new module param, default 4000 ms) replaces the hardcoded 100 ms acquire-wait bound in svc_run_ops. The old bound is shorter than one emulated GR launch completion (0.5-1.6 s observed), so a D2H readback's ACQ_CIRC_GEQ on the producer channel's progress counter (op=3 pay=4 on 0x20400ffd0) gave up and ran BEFORE the scale launch wrote d_c2; the client then verified pre-scale data (bad=32767). r401 log: the same entry now holds ~1.2 s across cur=1->4 and runs after GUEST GR LAUNCH OK (copy-back optrace 92315.594824 with dst16_after = the post-scale bytes). Fail-open is preserved: a genuinely unsatisfiable acquire still expires, just late enough that the dependency's work has landed (late-but-correct over early-and-wrong). 2. svc_gr_seen: a session-scoped open-addressed (push_va, hash) table behind svc_gr_cand_push_seen(). The 4-slot candidate ring + 8-VA history evicted a real capture within ~12 later captures, so a true replay of an already-forwarded launch was re-captured and re-executed (r399: the same nine launch pushes re-submitted every ~10-11 s for minutes). Cleared in svc_session_flush because a new client's identical-content pushes restart their release counters - a boot-scoped table would silently drop them (LAUNCH-SKIP). Also adds tools/cuda-ab/streams2d.cu: streams2 with per-array/per-4KB-block diff reporting plus the device pointer dump, which localised the remaining failure to the D2H staging write path (our staging-page resolution vs the guest's view).
* sema: r398 size the inline-once table to 8192 slots (256 KB)Arthur Hanson Rasmusson17 hours
| | | | | | | | r398 evidence: the memset's inline write executed twice for the same push (85634.101649 first, 85642.940808 replay 70 us before the D2H readback, zeroing the accumulator). 76 inline writes happened between the two - the 64-slot table had evicted the key. 8192 slots give a ~100x margin over the busiest observed burst.
* sema: r391f take 2 - once-only GRCE inline writes keyed by (push_va, push ↵Arthur Hanson Rasmusson17 hours
| | | | | | | | | | | | | | | | hash, dst, bytes) The first take gated on !replay, which broke pushes whose first pass was untrusted (their bytes are only authored on the later trusted pass) and hung the second client of the boot. Key the inline write by (push_va, push content hash, dst, bytes): a replay of the same push is a genuine duplicate and is skipped, a new push (its release payload makes the hash unique) always runs. Table is session-scoped (cleared by svc_session_flush), 64 slots. Root cause retained: victory_reduce's cudaMemset(d_sum,0,8) is a GRCE LOAD_INLINE_DATA write; every re-decode of that push re-ran it and re-zeroed the accumulator after the launch's copy-out (the sum=0 class).
* sema: r391f once-only GRCE inline writes - the sum=0 class root causeArthur Hanson Rasmusson17 hours
| | | | | | | | | | | | | | | | | | A REPLAYED ring entry must not re-run its LOAD_INLINE_DATA writes. The fill/copy ops were already gated by allow_ops (cleared for entry_replay), but the inline-write gate was 'trusted || allow_ops' and a replay is trusted by construction, so every re-decode of an already-processed push re-wrote its inline bytes into the vidmem shadow. victory_reduce's cudaMemset(d_sum, 0, 8) is exactly such an inline write. Observed on r396: 'sema-grce: inline write dst=0x7845ef000000 -> pa=0x7c00000 ap=vidmem bytes=8 data8=0000000000000000' fired 86 us before the D2H readback of the same VA - zeroing the accumulator the launch had just written (OUT-PAGE-POST val8=0000008000080000 = the correct sum) so the client read zeros (sum=0, bad=1). The same replay rewrote the page repeatedly (482 inline writes/boot). The r383 replay-capture path only needs to CAPTURE the launch; the inline bytes were authored on the first (non-replay) pass.
* sema/boot: r391e print the shadow base pointer in the copy-out and bigcopy ↵Arthur Hanson Rasmusson18 hours
| | | | | | | | | | witnesses Rule in/out a stale or duplicate vgpu->bars[1] shadow between the launch copy-out (svc_fwd_req.vgpu) and the D2H reader (svc_copy's vgpu): the sum=0 runs show the write landing (OUT-PAGE-POST val8 = the sum) and the reader of the same ledger page reading zeros 8 ms later with no logged writer in between. Diagnostic only.
* boot/gr: r391d post-write witness - read the VA's landing page immediately ↵Arthur Hanson Rasmusson18 hours
| | | | | | | | | | | | | after the copy-out write The sum=0 class: OUT-PAGE-WIT shows the changed page's bytes in the bounce (the correct sum) and the ledger resolution (pa=0x7c00000 ap=FB), the copy-out writes hpa=0x16f07a000, and the D2H read of the same ledger page 8-34 ms later reads zeros. sema-fbhpa proves __va(hpa) == base+pa at submit time. This witness reads the page right after the memcpy through the same path the D2H reader uses, so the next failing run names whether the write target and the reader's page differ (aperture/staleness) or the write vanished.
* sema/boot: r391c OUT-PAGE-WIT resolution witness for the sum=0 readback classArthur Hanson Rasmusson18 hours
| | | | | | | | | | The GR copy-out now prints, per changed page, the landing-table resolution of the VA (the exact source the D2H reader's ledger-first branch uses) and its first 8 bytes beside the write target hpa - a launch/readback disagreement (write landed on another page/aperture than the reader resolves) becomes one line instead of a forensics session. Diagnostic only (bounded by the existing copywit budget).
* sema: r391b fix svc_op_translate polarity in the per-page fill resolverArthur Hanson Rasmusson18 hours
| | | | | | | | | svc_op_translate returns 0 on success; the new svc_fill_dst_page_pa tested it without negation, so every page answerable only by the walk/scan returned -ENOENT and the fill truncated (r391 boot: FILL-TRUNC for the page-table writes at va=0x...a00200 -> the client stalled at init). Also demand the unresolved page through the pin channel before failing (stalled VA row -> the userspace stream pushes it to the guest scanner).
* sema: r391 per-page destination resolution for virtual fills - the r390 ↵Arthur Hanson Rasmusson18 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | guest panic root cause The 383620-byte context-init zero fill (FILL-NOREMAP class) resolved only its FIRST page through the pin (va=0x7c36c7400000 -> pa=0x131b63000, the pin stream's first run = 29 pages) and then wrote the whole 375 KiB contiguously past it. The pin stream shows the VA's pages are independent guest allocations with split runs (0x131b63000 / 0x131a80000 / 0x128340000), so the contiguous write covered unrelated memory - on r390 exactly the guest kernel's page-table pages 0x131ba1000/0x131ba2000/0x131bab000. The direct map grew three 4 KiB holes and the guest died with 'Fatal exception in interrupt' (unmap_page_range / cpuacct_account_field / psi_task_change faults on physmap addresses). svc_copy has always translated virtual operands per 4 KB page ('a linear PA walk past the first page reads the wrong memory'); svc_fill never did. New svc_fill_dst_page_pa() resolves every page of a virtual fill through the same authority order (fresh pin wins and repairs the landing table, then the landing table, then the resolver chain); the SYS write loop clamps each chunk to the page boundary and writes at that page's own landing; a page with no resolution fails the fill loudly (FILL-TRUNC) instead of writing through a contiguous guess. The >= 64 KiB landing-table pre-store now records only the contiguous VID (shadow) case; SYS landings are stored per page as they are written. Also extends the r384 page-table-shape refusal to every page of a virtual zero fill (the r390 fill's first page was the client's buffer, so the first-page-only check could not see the PTE pages it later covered). Evidence: /tmp/opencode/r390-live.log (FILL-EXEC + sema-opadd lines), /tmp/opencode/r390-guestram.raw (three zeroed PTE pages, CR3 walk), pin log 12-pin_gr_stream-r390.log line 789 (the split runs).
* sema: r384b narrow the fill pin-demand hold to SYS (guest RAM) resolutions + ↵Arthur Hanson Rasmusson22 hours
| | | | | | | | | | | | | | | FILL-EXEC witness - The r384 boot held a large fill whose walk resolution was VID (the emulated vidmem shadow) - the hold was pointless overhead there and the re-decode re-resolves (the resolver is non-deterministic), so the hold now applies only when the fill would land in guest RAM (SYS aperture), the class that zeroed the guest's page-table pages. Shadow resolutions stay immediate. - FILL-EXEC witness (bounded 64/boot): va, pa, aperture, provenance and pattern of every >= 4 KiB virtual fill execution - the decisive line for the boot-corruption forensics; the large zero fills were invisible until now.
* sema: r384 fill-destination guards - pin-demand hold for large unproven ↵Arthur Hanson Rasmusson22 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fills + page-table-shape refusal for zero fills (the boot-corruption lottery root-caused to zero writes landing on guest page-table pages) Forensics from the r383 boot (guest panic at t=216 during victory_strided): the guest kernel's direct map had TWO 2 MiB holes (phys 0x105000000 and 0x131800000) because their PTE pages at gpa 0x12d3c2000/0x3000 - two adjacent 4 KiB page-table pages - were found all-zero. Kernel objects (the kprobe nodes UVM registers at module load) lived in those regions; the first access after the zeroing faulted (Oops #1 get_kprobe from uvm_init_entry), cascading to the t=230 panic. The zeroing is an 8 KiB all-zero write, the exact shape of the guest's window page-table clear (LINE_LENGTH=8192, value=0), and the emulation's virtual fill resolver is demonstrably non-deterministic: four executions of that same 8 KiB fill resolved to four different guest pages (0x126af8000, 0x12d8e4000, 0x12f20c000, 0x1187fb000 - every one refused by the below-floor guard). The below-floor guard covers window VAs only; a >= 1 TiB VA fill that resolves through the walk/scan had no guard at all. Two nets in svc_fill, both default-on (sema_fill_guard=1) and both fail-open/histed: - net 1: a >= 64 KiB virtual fill whose only authority is the walk/scan (no pin, no landing-table entry) is held through a bounded 800 ms window (sema_fill_guard_ms) while its VA is demanded through the pin channel (the userspace pin stream injects the guest-pagemap truth), so the retried decode resolves through a real pin. Mirrors the m169/r378 copy guards; releases with the historical behaviour after the bound. - net 2: a ZERO fill about to land in guest RAM on a page whose first 64 qwords are all present+NX+in-range PTEs is REFUSED loudly (FILL-PT-REFUSED) - a page table page is never a legitimate zero-fill destination. Pin-backed resolutions are exempt (guest pagemap truth). New witnesses: FILL-HOLD (held fills) and FILL-PT-REFUSED (refused page-table targets), both bounded.
* sema/boot: r383 kill the blanket bounce copy-out (alias+staleness write ↵Arthur Hanson Rasmusson22 hours
| | | | | | | | | surface) + write guards; REPLAY-CAPTURE closes the silent launch-drop hole; CE operand misses feed the pin-demand channel - bounce copy-out now records staged_hpa[bi] per launch and skips: (a) recs whose hpa != staged slot content (alias/leftover), (b) pages the engine did not change (~955 of 968 per boot launch); the remaining changed pages re-validate via gsp_sema_gr_write_guard before the 4 KiB memcpy. Removes the blanket 968-page write-back that was the remaining boot-corruption lottery vector (guest slab freelist clobbered by a foreign page write). - new bounded WGUARD-SKIP guards: gr_write_guard (pin-first re-resolution) on copy-out, release-word zero/restore (rel_va plumbed through the submit), stale-recover-zero, counter release; svc_write_guard pin-agreement on 4/8-byte SYS/FB service writes; svc_copy SYS-dst hold on a fresh-pin contradiction. - svc_qmd_accum replay hole: a trusted replay of an unseen push now captures (REPLAY-CAPTURE) instead of silently dropping; known push -> LAUNCH-SKIP witness; capture history ring prevents double-capture. - svc_op_translate miss path emits the stalled-VA demand signal for CE operands.
* sema: r377 pin-demand for unresolved launch operand pages - every dropped ↵Arthur Hanson Rasmusson24 hours
| | | | data-fill page is logged as a stalled VA (the userspace pin stream watches dmesg for that line and pushes the VA to the in-guest scanner's demand file), so the bounded launch hold can resume on real pins instead of waiting out its bound (the r345 wedge)
* sema: r376 pin-agreement on the copy ledger-first branches - a fresh ↵Arthur Hanson Rasmusson24 hours
| | | | disagreeing pin beats a recorded landing on both dst (write) and src (read) at top level and per page, repairing the entry; closes the stale-entry write (guest-corruption) and read (wrong-result) classes
* sema: r375b drop the resolver-answer ledger store - a stale resolver page ↵Arthur Hanson Rasmusson24 hours
| | | | recorded as the write authority poisons every later write of the VA (r375 boot slab GPF in kmem_cache_alloc at guest t=77s before any client); the table is fed by writes and fresh pins only
* sema: r375 read-after-write authority - the landing table wins for device-VA ↵Arthur Hanson Rasmusson24 hours
| | | | copy sources (top-level and per page), and a resolver answer is recorded so launch/read paths converge; the engine output can no longer be read through a different page/aperture than it was written
* sema: two-path identity check for the FB shadow hpa (__va(hpa) vs base+pa) - ↵Arthur Hanson Rasmusson25 hours
| | | | the sum=0 class shows the engine output written through one path and read through the other
* sema: page-note resolution diagnostic (va/pa/ap/branch) for the launch-page ↵Arthur Hanson Rasmusson25 hours
| | | | vs readback disagreement (sum=0 class); budget 512
* sema: pin-wins wave - a fresh pin re-homes the landing table ↵Arthur Hanson Rasmusson25 hours
| | | | | | | | | | | | | | | | (svc_handle_map_add) and wins over the ledger in svc_gr_page_note, svc_gr_data_fill and svc_fill; gate the displaced-pin fan-out on a current pin match Root cause class (sub-agent audit + r371 log forensics): the landing table (ledger) moved only on host writes, so a VA re-pinned to a NEW page (CUDA reuses VA ranges across processes) kept resolving to the OLD, freed page. Every later write through that entry landed on memory the guest kernel had since reused - the r371 mail client's guest oops (NULL page array in nv_free_system_pages, 8 zero bytes) is this class. Fixes: (1) svc_handle_ map_add repairs a disagreeing ledger entry from the fresh pin (PIN-REPAIR); (2) svc_gr_page_note and svc_gr_data_fill keep a live pin over the ledger (PIN-WINS); (3) svc_fill checks the pin before the ledger fallback; (4) the pool alt fan-out writes the displaced page only while the current pin still names it (alt_skip). New diag: pinrep= alt_skip=.
* cuda-ab/soak: accept a terminal STATUS line as completion - the guest ↵Arthur Hanson Rasmusson26 hours
| | | | wrapper's EXIT= marker is lost when the guest's process reaping stalls (observed: victory_mail reached STATUS OK but the zombie was never reaped; soak reported a false still-running failure)
* sema: store the landing table for every launch page in svc_gr_page_note - ↵Arthur Hanson Rasmusson26 hours
| | | | indirect/bank operands (the kernel output buffer) resolved there while the client D2H readback used an older ledger landing, read stale zeros and the kernel looked wrong (victory_reduce sum=0 with the correct sum proven in the bounce page)
* sema: refuse below-floor fills that resolve outside vidmem (LOWFILL-REFUSED) ↵Arthur Hanson Rasmusson26 hours
| | | | - the 0x204020000 engine-window clear resolved to a different guest-RAM page on every execution and zeroed 8 KB of guest kernel memory (gmain NULL deref in __fdget, task_struct corruption)
* sema: opdef and GR hold bounds are never re-armed while active - the ↵Arthur Hanson Rasmusson27 hours
| | | | arm-on-VA-change rule moved the deadline on every held entry (streams2 wedge: opdef=955/13/0, drain held climbing, expired 0); entries held while a bound is active ride it and flush together
* sema: TABLE-COPY witness - dump carried bytes of page-table page/entry ↵Arthur Hanson Rasmusson27 hours
| | | | copies (bounded) so zero-walking tables trace to their building copy or prove the source was zero
* sema: CE fill byte-exact semantics per the driver's own memset push - ↵Arthur Hanson Rasmusson27 hours
| | | | LINE_LENGTH_IN is the size in BYTES (count=line_length/elem; the old count=line_length wrote elem x too much), value from DST_X mapping (CONST_A/CONST_B), r367
* sema: keep count=1 remapconst parts as flags only (NOT have_remap) - r365 ↵Arthur Hanson Rasmusson27 hours
| | | | proved below-floor window fills executing via unproven resolution zeroed guest RAM (slab GPF -> guest panic); add LOWFILL-WIT resolution witness for below-floor fills
* sema: decode count=1 SET_REMAP_CONST A/B/COMPONENTS (the CE page-table fill ↵Arthur Hanson Rasmusson27 hours
| | | | constants) + opt-in vidmem-window fill execution (gsp_sema_winfill, off) + FILL-RAW raw-push diagnostic; remapconst log budget 256
* sema: opdef absolute per-launch hold bound (GRFIX lesson) + physical ↵Arthur Hanson Rasmusson28 hours
| | | | | | | | | | | | | | | | guest-RAM fill branch + diagnostics The opdef hold used a slot-keyed deadline that a pin-batch rewind re-arms on every replay (observed opdef=3301/32/0 with the guest wedged); the bound is now keyed to the launch push VA and written back through the pump snapshot, like the GR hold bound (r213). The fill op builder also accepts physical SYS destinations below the VA floor (guest-RAM page-table clears; bounded by SEMA_PA_MAX). The vidmem-window zero-fill experiment (r362) was reverted: executing those fills with elem=1 wrote 4-byte partial zeros and broke results (no drops but sum=0); the window fills need their true element size and value semantics decoded first (see handoff).
* sema: revert the capture-site launch hold (opdef holds accumulated without ↵Arthur Hanson Rasmusson29 hours
| | | | expiry and wedged client init; keep the content-hash witness and the bound-expiry retry-forward)
* gr/sema: content-hash witness for the once-only op gate (rewritten fills now ↵Arthur Hanson Rasmusson30 hours
| | | | run - stale GPU page tables class); launch hold without capture trust + bound-expiry retry-forward (no silent launch drop); guarded zero-fill for client device VAs; walk/rehold diagnostics
* gr: record the launch's resolved operand pages in the ledger - fixes the D2H ↵Arthur Hanson Rasmusson30 hours
| | | | readback reading a stale page (victory_reduce sum=0 class); add GMMU walk trace + fwd-hpa diagnostics
* cuda-ab: add launch_count guest binary (single-launch execution-count repro, ↵Arthur Hanson Rasmusson31 hours
| | | | expect LAUNCH_COUNT=1)
* gr: single scheduling edge (pcas default 0) - fixes double engine execution; ↵Arthur Hanson Rasmusson32 hours
| | | | retire-wait tracked-total fix; sema acquire wait gate per-entry + C56F op semantics (opt-in); fill-shape decode diagnostic
* cuda-ab: launch_count - single-launch execution-count repro (each forwarded ↵Arthur Hanson Rasmusson33 hours
| | | | launch runs its engine work twice; the multi-pass class root cause)
* STATE-CURRENT: 2026-10-20 evening snapshot - pin-wins corruption fix ↵Arthur Hanson Rasmusson34 hours
| | | | (vector_add green), multi-tenant live, battery frontier (multi-pass content + streams2 ordering)
* vgsp: C56F acquire wait gate as an opt-in experiment (default off)Arthur Hanson Rasmusson35 hours
| | | | | | | | | | | | | | | | | | | | | | The streams2 readback-before-scale class is root-caused: core CUDA starts compute-channel pushes with a C56F SEM_EXECUTE acquire (ACQ_CIRC_GEQ on a tracking semaphore; e.g. the D2H readback push waits on 0x20400fef0 >= its sequence number) and the decoder ignored it, so the copy ran ahead of the scale launch it waits for. The gate implemented here (decode the acquire; svc_run_ops holds the push's work until the semaphore satisfies, with the driver's release(N)+acquire(N) same-push pattern treated as satisfied; op-pending bounded hold; kill switch at 2048 holds; diagnostics sema-wait held rows with a per-session budget) OVER-HOLDS on boots where a channel's first pushes wait on counter values only their own withheld release would install (wait counts reach thousands, the client stalls; r337/r338/r339). The tracking-semaphore value semantics (what the guest word should read vs the driver's sequence numbers) need a dedicated pass before this can be enabled. Default OFF so behaviour is the pre-r336 one; enable per-boot with sema_wait_gate=1 for the experiment. Everything else (pin-wins, vector_add, the battery) is unaffected.
* vgsp: honor the C56F host-semaphore acquire (fix the streams2 ↵Arthur Hanson Rasmusson36 hours
| | | | | | | | | | | | | | | | | | | | | | readback-before-scale class) Core CUDA starts each compute-channel push with a C56F SEM_EXECUTE acquire method (five words [ADDR_LO][ADDR_HI][PAYLOAD_LO][PAYLOAD_HI] [EXECUTE]; op 0/2/3/4/5 = acquire/equal, strict GEQ, circular GEQ, AND, NOR). The decoder handled only op 1 (release) and 6 (reduction), so every wait was ignored and a push's work could run ahead of the dependency it waits on. Observed on streams2: the D2H readback push carries ACQ_CIRC_GEQ on the channel progress counter (0x20400ffd0 >= 4, the 4th push = the scale launch); the readback executed before the scale, and the client verified the pre-scale buffer (bad=32767, c2 = a2+b2 without the x2). Now the decoder records the acquire and svc_run_ops honors it before running any of the push's work: satisfied (read + circular/equal/GEQ compare) -> proceed; unsatisfied -> set op_tr_pending, so the entry is held and its release withheld (bounded by SVC_OPDEF_HOLD_MS, loud) and the re-decode runs the work once the counter advances. Unreadable semaphore: fail-open (pre-r336 behaviour). New stat wait=held/pass.
* vgsp: pin-backed translation wins the write-side ledger divergence (r336)Arthur Hanson Rasmusson36 hours
| | | | | | | | | | | | | | | | | | | The m180 unification redirected a divergent write to the ledger's page (where an earlier write landed) even when the current translation was pin-backed (guest pagemap truth, op_tr_src == 1). When the guest remaps/reuses a VA (free + reallocate, or migration), the ledger page is freed and can be reallocated by the guest kernel: the redirected write then smears client copy data over kernel memory. Observed on the r335 vector_add run: an RM map/AVL node's pointer field held client c-data (floats 551.25/552.0), the free-path traversal faulted (mapFindLEQ_IMPL/_mapInsertBase), and the guest panicked (Attempted to kill init). 4448 write-side divergences in that run. Now a pin-backed translation wins: the write goes to the fresh page and the ledger store after the write re-homes the entry, so readers converge on the same page. Non-pin translations (2..5) keep the m180 redirect - they can carry stale PTEs (m169) and the write history stays the authority there. New counter sema-svc ledger pinwins.
* site: 2026-10-20 status refresh - guest CUDA kernel execution workingArthur Hanson Rasmusson36 hours
|
* STATE-CURRENT: 2026-10-20 snapshot - boot lottery root-caused and fixed ↵Arthur Hanson Rasmusson36 hours
| | | | (status-queue SWAP_RX cursor); guest CUDA kernels green end-to-end (mail/matmul/transformer); large-buffer class frontier
* vgsp: read the status-queue consumption cursor from the command queue (SWAP_RX)Arthur Hanson Rasmusson40 hours
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Under MSGQ_FLAGS_SWAP_RX (both sides set it; flags=1 in both headers) the guest driver's msgqRxMarkConsumed writes its STATUS readPtr to pOurRxHdr->readPtr, and the driver's pOurRxHdr is the COMMAND-queue buffer's RX header (msgq.c msgqRxLink swap branch: pReadOutgoing = &pOurRxHdr->readPtr with pOurRxHdr from msgqTxCreate's pBackingStore = pCommandQueue). The status queue's own +rxHdrOff is the mirror we publish - our command-queue read position. The status free-space check read the mirror, so it tracked our own writes: with the guest mid-drain the ring was declared full (free=0, peer=0, write=62/63), replies were written into it anyway, unread records were overwritten, and the guest RM read a newer record - GspMsgQueueReceiveStatus: Bad sequence number. Expected 447 got 510. Possible memory corruption. - after which every RPC failed (status 0x25) and the first client's cudaMalloc returned device busy or unavailable. That is the boot lottery: on a lost boot the first client (mail) FAILs and the boot is poisoned for every later client. Evidence (r330 vs r332): r330 mail FAILed at the first cudaMalloc and only one channel unregistered in the whole boot (the guest RM never tore its channels down after the status desync); with the fix r332 mail + matmul pass and every client's channels unregister (79 unregisters, one session flush per client). Also adds bounded diagnostics: sema-d2hmap (per-page destination resolution of >=1MB host-dest copies) and a raised DIVERGE witness budget.
* vgsp: fail4 - route pending-pass releases through opdef withholding ↵Arthur Hanson Rasmusson46 hours
| | | | | | | | | | | | | | | | | | (cold-run gap) The fwd-defer arm took priority over svc_opdef_release, so a pass whose op was still pending recorded its fence release in svc_fwd_rel (applied at the hold break / fallback) instead of the opdef slot - the fence fired while the copy was still owed, the driver refilled the ping-pong staging, and the late capture/read got refilled bytes (cold-run vector_add gap: one to four chunks per cold boot). svc_fwd_defer_arm() is now skipped when svc.op_tr_pending, so those releases go through svc_opdef_release and apply only when the retried pass has no pending op (the op has run by then). Measured cold-run vector_add bad across this iteration: 193536 -> 100352 (r323) -> 479232 (r324 wrong queue-pending experiment, reverted) -> 66560 (r325, ~1 chunk). All other acceptance items green on r325 (mail OK, smi OK, matmul OK, transformer OK).
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-21 21:41:56 +0000