From 0892871eda284756b232e1ab157b0b4e170930cc Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Sun, 18 Oct 2026 02:57:26 -0400 Subject: vgsp: m180 VA->PA unification - one landing authority for every data-bearing device VA (kills the write/read translation split of the always-failing class) Context (r209, fc2d1ae5): multikernel/memcpy_roundtrip/streams2/ victory_streams_events still fail with self-consistent, silent wrongness while every launch reports OK. Evidence: streams2's bad data is exactly the PREVIOUS client's h_in pattern (engine read stale inputs); the memcpy window logs 16x "sema-ledger: DIVERGE read ... wrote=0x7f83000 read=0x8783000" (D2H resolved a different page than the write landed on); the failing four's device operands are all sysmem-backed (ap=2, pa=0x7800000/0x8800000 in every svc-gr cand[] line), i.e. exactly the aperture the m171/m172 ledger never recorded. Root cause class: a data-bearing guest VA had NO single authoritative PA. Write paths (H2D svc_copy dst, fills, inline writes, engine write-back) each landed wherever their own resolution said; read paths (GR launch fill, D2H svc_copy src) resolved independently again. The ledger patched up some of that, but (a) it recorded only VID-aperture copies >=64 KiB, so the dominant sysmem case was invisible; (b) it was a 2048-entry RING - one launch's submission record set (2505-4553 pages in r209) evicted the H2D records of the same client before its own D2H read them, so the readback fell back to its own (different) resolution; (c) several resolution failures returned -ENOENT even when the landing was known; and (d) copy paths deferred to the forwarded-request machinery skipped the in-loop record entirely. Fix - one landing authority, consulted everywhere: 1. The ledger is now a keyed hash table (16384 slots, 16-probe, one entry per VA page, in-place rewrite). No cross-VA eviction: a client's whole working set (8 MB in + 8 MB out + engine records) survives until its readers have run. Rewrites are counted + logged (bounded) because a write path moving an existing page is the write-side half of DIVERGE. 2. Every guest-visible write records its landing page: - svc_copy dst_virt: SYS as well as VID (the sysmem half was the hole); - svc_fill: SYS as well as VID; - svc_grce_inline_write: unconditionally; - small copies that are deferred to the forwarded path: recorded at candidate capture (the software loop never runs for them); - engine launch data pages: unchanged (submission-time record). 3. Every read/write consults it: - GR fill: already diverted (m171); now actually has SYS records to find; - D2H svc_copy src: per-page divert (m171) + NEW fallback to the recorded landing when the resolver fails (previously -ENOENT); - svc_copy dst / svc_fill / svc_grce_inline_write: NEW - resolve, then land on the recorded page when one exists (one page per VA, readers already follow it), and fall back to it when resolution fails. This also spares the m169 stale-walk hold for destinations whose page is already known (no pin needed). Effect on the failing class: the H2D/fill write landing becomes the page the engine fill, the copy-out and the D2H all use; where a stale and a fresh resolution disagree, the readers follow the landing and the other translation is logged as DIVERGE (read/write) or rewrite - bounded, once. streams2's c1 (engine read stale) is healed by the new SYS records; streams2's c2 (readback of an engine output after later launches) is healed by the hash table, which no longer evicts the first launch's records. Zero-write lottery diagnostics (secondary, booter; no behavior change on the green path): - content-hash settle rule in gsp_phys_gr_output_retire_wait: the "settled" verdict now requires the per-page checksum FOLD to be stable for ~300 ms, not just the changed-page COUNT (a continuing store ramp with a constant count used to read as settled - the r209 "17/32 settled"/"2048/4552 settled" partial copy-outs); - zero-write accept now logs both qmdwin banks' sentinel words (ours intact + other bank overwritten = an older deposit was consumed, i.e. mis-targeted deposit; ours overwritten = this deposit was fetched) and re-scans the tracked pages once at +1 s: LATE-STORES (copy-out will ship them) vs NO-STORES (dispatch lost or read-only kernel). The +1 s is bounded to the rare zero-write accept path only. The sentinel constant itself is left intact (it is load-bearing for the m179 post-mortem classification); the dual-bank read answers the same mis-targeting question without changing the engine-visible contract. Untouched on purpose: mm2 publish arm/gate, GR fresh-put rewind, the release machinery and every semaphore path (no regression surface there). make kernel -j green (new srcversion); main tree and the running VM were not touched; nothing pushed. --- kernel/gsp_phys_booter.c | 165 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 116 insertions(+), 49 deletions(-) (limited to 'kernel/gsp_phys_booter.c') diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 535c8ad..f60f6c5 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -17034,34 +17034,24 @@ static u32 gsp_phys_gr_data_bi(u64 reloc_va) * noinline so this rarely-taken wait does not grow the submit frame, which * is already near the warning bound. */ +/* + * m180: scan the request's unique tracked bounce pages and return how many + * differ from the copy-in checksum baseline. `hash_out` (optional) receives + * an FNV-1a fold of the per-page checksums - the content fingerprint the + * settle rule needs so a store ramp with a constant changed-page COUNT can + * never read as settled again. Static scratch: the only caller is the + * single-inflight ordered gsp-phys-work item (same discipline as the other + * submit-scope statics). + */ static noinline u32 -gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, - const u32 *csum_in, u32 *n_tracked) +gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, + const u32 *csum_in, u32 *n_tracked, u64 *hash_out) { - /* - * B-2: one entry per page of the grown region (64 KB), static like the - * other submit-scope scratch: the only caller is the single-inflight - * ordered work item (see the static scratch note in - * gsp_phys_backend_guest_gr_submit). 64 KB must not sit on the kernel - * stack (this frame was already near the warning bound at 205 pages). - */ static u32 checked[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); - u32 n_all = 0, di, iter; - u32 changed = 0, prev = 0xffffffffu; - const char *why = " (TIMEOUT)"; - unsigned long wait_start = jiffies; + u32 n_all = 0, di, j, changed = 0; + u64 hash = 1469598103934665603ULL; - /* - * m179: report how many pages were actually trackable. "wrote == 0" - * from a launch with n_tracked == 0 (no data entry resolves to a - * bounce page) is not a no-execution signature and must not trigger - * the no-write retry/accept path. - */ - if (n_tracked) - *n_tracked = 0; - if (!req->n_data) - return 0; for (di = 0; di < req->n_data; di++) { u32 bi = gsp_phys_gr_data_bi(req->data[di].reloc_va); u32 k; @@ -17077,41 +17067,68 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, if (!dup) checked[n_all++] = bi; } - if (!n_all) - return 0; if (n_tracked) *n_tracked = n_all; - for (iter = 0; iter < 300u; iter++) { - u32 j; + for (j = 0; j < n_all; j++) { + const u8 *src = gsp_phys_chunk_page_va(dchunk, checked[j]); + u32 ci, csum = 0; - changed = 0; - for (j = 0; j < n_all; j++) { - const u8 *src = gsp_phys_chunk_page_va( - dchunk, checked[j]); - u32 ci, csum = 0; + if (!src) + continue; + for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) + csum += get_unaligned_le32(src + ci); + if (csum != csum_in[checked[j]]) + changed++; + hash ^= csum; + hash *= 1099511628211ULL; + } + if (hash_out) + *hash_out = hash; + return changed; +} - if (!src) - continue; - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != csum_in[checked[j]]) - changed++; - } +static noinline u32 +gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, + const u32 *csum_in, u32 *n_tracked) +{ + u32 n_all = 0, iter; + u32 changed = 0; + u64 prev_hash = 0; + const char *why = " (TIMEOUT)"; + unsigned long wait_start = jiffies; + + /* + * m179: report how many pages were actually trackable. "wrote == 0" + * from a launch with n_tracked == 0 (no data entry resolves to a + * bounce page) is not a no-execution signature and must not trigger + * the no-write retry/accept path. + */ + if (n_tracked) + *n_tracked = 0; + if (!req->n_data) + return 0; + n_all = gsp_phys_gr_scan_pages(req, csum_in, n_tracked, NULL); + if (!n_all) + return 0; + for (iter = 0; iter < 300u; iter++) { + u64 hash = 0; + + changed = gsp_phys_gr_scan_pages(req, csum_in, NULL, &hash); if (changed >= n_all) { why = ""; break; } /* - * m162: a partial count must be stable for ~300 ms before it - * counts as settled. The old 3-iteration (~18 ms) window - * accepted a stalled store ramp or an un-dispatched grid as - * "settled": observed 1024-page vecadd outputs copied out with - * only 2 pages written (the client then verified 32736 bad - * elements). A genuinely masked/short kernel costs at most - * 300 ms extra; a ramping engine gets the time its stores - * need. + * m180 (H3): a partial count must be CONTENT-stable for + * ~300 ms before it counts as settled. The m162 count-only + * rule accepted a continuing store ramp whose page count + * happened to be constant (the always-failing class's + * "17/32 settled" copy-outs), and accepted an + * un-dispatched grid as settled. A genuinely masked/short + * kernel costs at most 300 ms extra; a ramping engine gets + * the time its stores need. */ - if (changed > 0 && changed == prev && iter >= 60u) { + if (changed > 0 && hash == prev_hash && iter >= 60u) { why = " (settled)"; break; } @@ -17119,7 +17136,7 @@ gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, why = " (no writes)"; break; } - prev = changed; + prev_hash = hash; usleep_range(4000, 6000); } pr_info("vgsp-gr: output retirement wait: %u/%u pages changed after %u ms%s\n", @@ -19231,6 +19248,56 @@ gr_page_unmapped: n_tracked, nw_attempts, want_qmd, fetched ? "OK" : "NONE"); + /* + * m180 (lottery discriminators, agent-order H1/H2). + * + * (a) Deposit identity without changing the sentinel + * contract: read BOTH qmdwin banks' sentinel words. + * Ours intact + the other bank overwritten means an + * OLDER deposit was consumed instead of this one + * (stale/mis-targeted deposit, H2); ours overwritten + * means the engine really fetched THIS deposit + * (lost-dispatch vs read-only kernel, H1); both + * intact means no deposit ever reached the engine. + */ + if (qmdwin_on && boot_stage.rmcall_gr_qmdwin.pages && + qmd_dst_mem) { + u8 *om = gsp_phys_chunk_page_va( + &boot_stage.rmcall_gr_qmdwin, + qw_bank ^ 1u); + + pr_warn("vgsp-gr: zero-write deposit stamp: bank=%u sentinel=0x%08x other-bank=%u sentinel=%s\n", + qw_bank, + get_unaligned_le32( + qmd_dst_mem + + GSP_PHYS_GR_QMD_SENTINEL_WORD * 4), + qw_bank ^ 1u, + om ? (gsp_phys_gr_qmd_fetched(om) ? + "overwritten (older deposit consumed)" : + "intact") : "n/a"); + } + /* + * (b) Late-store check: a store ramp that retires after + * the host's window is indistinguishable from a lost + * dispatch at the accept moment. Re-scan once at +1 s + * and report whether the tracked pages changed in the + * meantime; the copy-out below picks up any stores + * that arrived late, so this both classifies and heals + * the late sub-case. + */ + { + u32 late_tracked = 0; + u32 late; + + msleep(1000); + late = gsp_phys_gr_scan_pages(req, bounce_csum_in, + &late_tracked, NULL); + pr_warn("vgsp-gr: zero-write late-check: %u/%u pages changed after +1000 ms (%s)\n", + late, late_tracked, + late ? "LATE-STORES - copy-out will ship them" : + "NO-STORES - dispatch lost or read-only kernel"); + } + if (qmd_rel_hpa) { u8 *rp = (u8 *)__va(qmd_rel_hpa) + qmd_rel_off; -- cgit v1.2.3