diff options
Diffstat (limited to 'BIGPATH-REPORT.md')
| -rw-r--r-- | BIGPATH-REPORT.md | 197 |
1 files changed, 197 insertions, 0 deletions
diff --git a/BIGPATH-REPORT.md b/BIGPATH-REPORT.md new file mode 100644 index 0000000..75b0273 --- /dev/null +++ b/BIGPATH-REPORT.md @@ -0,0 +1,197 @@ +# BIGPATH-REPORT — B-2 grown data-bounce region (offline implementation) + +* Branch: `bigpath`, based on `ecfb1278` (m155). Worktree: + `/home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath`. +* Scope: implement `AGENT-chunked-path-design.md` §3 (B-2) — make GR launches + with arbitrarily large operands (vector_add 3×4 MB, victory_math 320 pages, + memcpy_roundtrip 8 MB) execute correctly by replacing the 205-page / + 68-pages-per-pointer data window with a boot-planted 16384-page (64 MiB) + region, with fail-closed cap semantics. +* Constraints honoured: offline only — no module load/unload, no QEMU/mdev, + no guest, no dmesg writes, no reboot. Build: `kernel && make -j8`. + Commit local on `bigpath` only; no remote push. + +## 1. Build result + +Clean build `make -j8` from a `make clean` state: **exit 0**. +Warning delta vs the HEAD baseline (same toolchain): + +| Baseline (ecfb1278) | After B-2 | +|---|---| +| `gsp_phys_booter.c` frame 1120 B > 1024 (`gsp_phys_backend_guest_gr_submit`) | frame **1104 B** (same pre-existing warning, shrunk by the static scratch) | +| `gsp_sema_service.c:12518` unused variable `n_win` | removed (fixed in passing) | +| `main.c:402` frame 1304 B; `%x` vs `unsigned long` in `gsp_phys_rmcall_stage3` | unchanged (pre-existing) | + +No new warnings; the touched lines are `-Wextra` clean. + +## 2. Site → change table + +| # | Spec item | Site (post-change) | Change | +|---|---|---|---| +| 1 | Constants | `kernel/gsp_phys_booter.h:366-369` | `GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0` 0x0A000000, `_SLOT0` 80, `_PT_PAGES` 32, `_PAGES` 32*512=16384. Shared header constant: the SVC derives its reloc base and cap from here (no drift, design R5c). | +| 2 | Boot chunk fields | `kernel/gsp_phys_booter.c:2325-2330` | `struct gsp_phys_boot_chunk rmcall_gr_data_bounce;` (16384 pages) + `rmcall_gr_data_bounce_pt` (32 PT pages) added to `boot_stage`. | +| 3 | Stage-3 plant | `kernel/gsp_phys_booter.c:9770-9840` | One-shot (`!pages`) plant next to the 208-page bounce plant: allocate the data chunk (`gsp_phys_chunk_alloc`, one call, contiguous + per-page fallback) and the PT chunk, zero both, write PD0 dual entries 80..111 (same shape as ext pool 32..79: 0 at `slot*16`, `gsp_phys_rmcall_pde(dma)` at `slot*16+8`), then for every page i write `gsp_phys_rmcall_pte(page_dma(i))` at `PT[i/512] + (i%512)*8`; `dma_wmb()`; log `data-bounce region planted: …`. All at stage 3, before any engine fetch. Gated by `gsp_phys_gr_data_region`. | +| 4 | Old window untouched | `gsp_phys_booter.c:9727-9769` (unchanged), mail `9756`/marker `9765`, mail injection mva `17492-17493` | The 208-page window, mail slot 206, marker 207 and the bank-4 mail injection stay byte-identical; only its comment records the B-2 decommissioning as a data carrier. | +| 5 | `bi` helper | `gsp_phys_booter.c:16933-16953` (`gsp_phys_gr_data_chunk`), `16961-16982` (`gsp_phys_gr_data_bi`) | One helper returns `(reloc_va − VA0)>>12` bounded `< 16384` for the grown region (`GSP_PHYS_GR_DATA_BI_NONE` outside); with the parameter off it serves the legacy window bounded by `MAIL_SLOT`. A chunk selector names the active data chunk. | +| 6 | Five sites replaced | retire wait `:17025`, fill_bounce `:17091`, copy-in `:17972`, copy-out `:18653`, window diag `:18698` | All five inline `(reloc_va − 0x110000)>>12` + `MAIL_SLOT` bounds replaced by the helper + chunk selector. No other copy-path semantics changed. | +| 7 | Checksum/scratch arrays | `bounce_csum_in` `:17156` → 16384 entries (static); `checked[]` retire wait `:17015` → static 16384 entries | 64 KB each, off the kernel stack (design risk #4). Single-inflight caller documented in place. | +| 8 | SVC reloc base | `gsp_sema_service.c:12371-12372`, used `:12768` | `SVC_GR_DATA_RELOC_BASE` = shared grown constant; `_OLD` 0x110000 for the parameter-off path. | +| 9 | SVC per-ptr cap | `gsp_sema_service.c:12381-12382`, used `:12703` | 4096 (runaway guard); old 68 kept for legacy. Stop-on-miss still bounds real spans. | +| 10 | SVC total cap, fail-closed | `gsp_sema_service.c:12391-12392`, check `:12749-12767` | Cap = `GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES` (16384); on hit: `svc_gr_data_stalled = true` + `pr_err(… CAP-HIT: region full, launch held fail-closed …)` + return — no silent truncation, the existing bounded hold/retry path takes over. Old 205 kept for legacy. | +| 11 | SVC array headroom | `gsp_sema_service.c:3931` | `SVC_GR_DATA_MAX` 16384 → 32768 (gr_data[] and the backend hand-off array). | +| 12 | R3 hygiene | `gsp_sema_service.c:12671-12700` | Skip the launch's own param VA and the qmd_word 32..47 gr_pages (bank/param) as data candidates, next to the existing QMD-VA skip. Critical for B-2: without it the param-page candidate would consume up to a 4096-page span of resolvable guest memory. | +| 13 | Static asserts | `gsp_phys_booter.c:2365-2410` | Build-time disjointness/geometry contract: 32 PT pages × 512 = 16384 pages; VA0 == slot 80 × 2 MB; root/PD2/PD1 index 0 and PD0 80..111 ≤ 128; end ≤ VAS vaSize 0x10000000; region above ext pool end 0x0A000000; region disjoint from old bounce 0x110000..0x1dffff, win0 0x1e0000.., guest scratch 0x200000... | +| 14 | Module param | `kernel/gsp_phys_gpu.c:1080-1097`, decl `kernel/gsp_phys_gpu.h:357` | `gsp_phys_gr_data_region` (int, 0644, default 1). Read by the stage-3 plant, the SVC fill and the copy path. Boot-fixed (set at insmod). | +| 15 | Teardown | `gsp_phys_booter.c:19611-19613` | Free the two new chunks in `gsp_phys_rmcall_fini` (and, as a one-line in-passing fix, the old 208-page bounce chunk that was leaked there). | + +## 3. Deviations from the design + +1. **Shared header constants instead of per-file duplicates.** The four + region constants live in `gsp_phys_booter.h` and `gsp_sema_service.c` + derives `SVC_GR_DATA_RELOC_BASE`/`SVC_GR_DATA_BOUNCE_MAX` from them. The + design (§3.2, R5c) explicitly wanted the relocation base "comment-bound" + or shared to prevent drift; a shared constant is the stronger form. +2. **`bi` helper covers both modes.** The design's helper description assumes + the grown region only. Because the module parameter must flip the whole + data path back (spec item 7), the helper also serves the legacy window + (bounded by `MAIL_SLOT`), and a small `gsp_phys_gr_data_chunk()` selector + picks the chunk for the copy path. Behaviour with the parameter on is + exactly the design's. +3. **`checked[]` made static, not a bitmap.** The design allowed either; the + static array mirrors the existing single-inflight static scratch pattern + and keeps the wait logic byte-identical. The dedup loop remains O(n²) as + at HEAD; worst case (16384 records) is ~134M u32 compares once per + launch — accepted for bring-up (see residual risks). +4. **Old bounce chunk now freed in `rmcall_fini`.** Pre-existing leak spotted + next to the new frees; freeing it is safe (`gsp_sema_service_work_quiesce` + runs before `gsp_phys_boot_staging_free`). Reported here because it is a + teardown behaviour change beyond the B-2 allocation lifecycle. +5. **Memory-leak observation not fixed:** `rmcall_gr_shadow_win0` and + `rmcall_gr_mail` are also never freed in `rmcall_fini` (pre-existing, out + of scope). Likewise `rmcall_ctx_pt[]` last slot can collide with the ext + pool PD0[0] if the server MAIN context ever reaches the 32 MB maximum + (pre-existing latent; the live MAIN is ~1.2 MB). Neither is touched here. +6. **No F1-style permanent refusal on a persistent cap hit.** Per the spec, + the cap hit marks the launch stalled for the existing bounded hold. Note + honestly: that hold is one-shot per GPFIFO entry; if the region is + genuinely too small (negative control, > 16384 unique operand pages), the + entry can be consumed after the hold window expires, exactly like every + other stalled launch at this revision. The CAP-HIT `pr_err` makes it + visible, but the permanent fail-closed guarantee is `AGENT-bigcopy` + F1/§4 territory and was explicitly kept out of this change (design risk + #6). + +## 4. Risks + +**Design risk #1 — engine walker reach for PD0 80..111 (LOW).** The plant +shape is byte-for-byte the ext pool pattern (PD0 dual entry small half → +fresh 4 KB PT; leaves = `pte(dma)`), which is proven engine-readable at PD0 +32..79; entries 80..111 sit in the same PD0 page and same root/PD2/PD1 chain +(indices all zero, asserted at compile time). The only unproven ingredient is +the slot range itself. Fallback if the first big launch faults/times out with +`changed=0` and the data chunk otherwise healthy: move the region to the +reserved win1 data windows (`GSP_PHYS_RMCALL_VA_GR_DATA_BASE` 0x100000000, +prewin install) — the copy-path changes carry over unchanged. + +**Design risk #3 — 64 MB contiguous `dma_alloc_coherent` fails → per-page +fallback (MEDIUM).** `gsp_phys_chunk_alloc` warns and falls back to 16384 +per-page coherent allocations; boot-time stage-3 cost grows (seconds) and +memory is fragmented across pages (harmless to correctness: each PTE uses the +page's own DMA). Watch the boot log for `contiguous 16384-page coherent +allocation failed` and stage-3 wall time. Mitigation if needed: split the +chunk into a few smaller contiguous allocations — deferred until measured. + +**Other risks carried from the design:** +* *Per-pointer span overshoot.* 4096 pages is a runaway guard; stop-on-miss + still bounds a real operand. If a client's operands are laid out + contiguously with resolvable neighbours, one candidate can consume up to + 4096 pages and the cap can be reached before all operands are in. The + fail-closed stall + CAP-HIT log would expose this on the first big cycle; + the cap value (design open question #4) is the tuning knob. +* *Copy cost.* Up to 64 MB in + 64 MB out per launch (tens of ms), plus up to + 300 × 5 ms retire-wait polls over 16384 pages when nothing changes — all + bounded by the existing wait exits; correctness first (design risk #8). +* *Param flip consistency.* `gsp_phys_gr_data_region` is boot-fixed; a + runtime flip after the stage-3 plant would desync SVC vs plant. Documented + in the param description; the default cannot desync. + +## 5. Grep audit — remaining uses of the old bounce VA0 / 0x110000 + +`grep -rn "RMCALL_GR_BOUNCE_VA0|SVC_GR_DATA_RELOC_BASE_OLD|0x110000" +kernel/*.c kernel/*.h` (excluding unrelated hits `bar_emulation.c:62` LTC, +`gsp_emulation.h:40` firmware block): + +| Site | What it is | Indexes data pages? | +|---|---|---| +| `gsp_phys_booter.c:2338` | the legacy constant definition | no | +| `:2349-2351` | win0-vs-bounce overlap `static_assert` (layout) | no | +| `:2399` | B-2 disjointness assert | no | +| `:9744-9745` | old bounce plant `pr_info` | no | +| `:1011, 2306, 2320, 2371, 16952` | comments/history | no | +| `:16975, 16977` | `gsp_phys_gr_data_bi` legacy branch — reachable only when `gsp_phys_gr_data_region=0`, where operand records really are in that window; bound `MAIL_SLOT` excludes mail/marker | only in legacy mode, by design | +| `:17492-17493` | mail-injection `mva = VA0 + (MAIL_SLOT<<12)` — the mail message page, never a data record | no | +| `gsp_sema_service.c:12372` | the legacy reloc base constant (param-off path) | only in legacy mode | +| `gsp_sema_service.c:12367, 12478, 12646` | comments | no | + +The five data-page index derivations all now go through +`gsp_phys_gr_data_bi()`; with the grown region on (default), no code path can +index a data page through the old window. The mail injection and the old +window's own plant/diagnostics are the only remaining consumers, exactly as +the design requires. + +## 6. Sanity math (checked by computation) + +* `32 × 512 × 4096 = 67 108 864 B = 64 MiB` ✔ +* VA range `0x0A000000 .. 0x0DFFFFFF` (end-exclusive `0x0E000000`) `< + 0x10000000` (VAS vaSize) with 32 MB slack ✔ +* PD0 slot = `0x0A000000 >> 21` = 80; slots 80..111 ≤ 128 (non-shared VAS + ceiling) ✔ +* Disjointness: ext pool `[0x04000000, 0x0A000000)` meets the region exactly + at VA0; old bounce ends `0x1E0000`, win0 ends `0x1F0000`, guest scratch + ends `0x2200000` — all far below 0x0A000000 ✔ (all compile-time asserted) + +## 7. Live validation commands for the lead (supervised; NOT run here) + +```sh +# 1. Bring the branch build into the campaign tree (lead decides merge vs +# manual copy of vgsp.ko) and confirm the param exists: +git -C /home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath log --oneline -1 +cd /home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath/kernel && make -j8 +modinfo vgsp.ko | grep gsp_phys_gr_data_region + +# 2. One full supervised cycle with the B-2 default (grown region ON; +# module args unchanged, gsp_phys_gr_data_region defaults to 1): +cd /home/rmcore/Git/integdev_gpu_drv/open-iov +./backups/evidence-20261002-victory/victory_grfwd_cycle.sh b2-r1 + +# 3. Boot gate (host dmesg) — expect exactly one plant line, no alloc error: +# vgsp-boot: rm-call: data-bounce region planted: 16384 pages at VA +# 0xa000000..0xdffffff (PD0 slots 80..111, 32 PT pages, before first fetch) +# and no "data-bounce region allocation failed" / no +# "contiguous 16384-page coherent allocation failed" (or accept the +# fallback warning and time stage 3). + +# 4. Guest clients: push the battery and run the three big clients plus the +# regression set (the cycle already runs the runner; for manual runs): +./tools/cuda-ab/push_battery.sh +# in the guest: run /tmp/victory_math /tmp/vector_add /tmp/memcpy_roundtrip +# plus regressions (victory_matmul smiley mall victory_mail victory_saxpy +# victory_strided victory_reduce victory_atomics victory_memset2d +# victory_streams_events vecadd_loop). +# Expected host log: "svc-gr data fill: … pages resolved (reloc 0xa000000+)" +# with NO "CAP-HIT"; copy-out "changed by engine" > 0 naming output pages; +# guest: MATH badf=0 badd=0 badi=0 / RESULT bad=0 / ROUNDTRIP bad=0 csum=269221888. + +# 5. Rollback / A-B trial (legacy 205-page window, one flip at insmod): +./backups/evidence-20261002-victory/victory_grfwd_cycle.sh b2-legacy-r1 gsp_phys_gr_data_region=0 + +# 6. Negative control (over-region, > 16384 unique operand pages, e.g. +# vector_add n=2^23): expect the CAP-HIT pr_err + hold, no silent success. +``` + +## 8. Rollback parameter + +`gsp_phys_gr_data_region=0` at insmod restores the legacy 205-page / +68-pages-per-pointer path (reloc base 0x110000): the stage-3 plant is skipped, +the SVC keeps the old caps, and the copy path indexes the old bounce chunk. +Boot-fixed; do not flip at runtime. |