path: root/BIGPATH-REPORT.md

# BIGPATH-REPORT — B-2 grown data-bounce region (offline implementation)

* Branch: `bigpath`, based on `ecfb1278` (m155). Worktree:
  `/home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath`.
* Scope: implement `AGENT-chunked-path-design.md` §3 (B-2) — make GR launches
  with arbitrarily large operands (vector_add 3×4 MB, victory_math 320 pages,
  memcpy_roundtrip 8 MB) execute correctly by replacing the 205-page /
  68-pages-per-pointer data window with a boot-planted 16384-page (64 MiB)
  region, with fail-closed cap semantics.
* Constraints honoured: offline only — no module load/unload, no QEMU/mdev,
  no guest, no dmesg writes, no reboot. Build: `kernel && make -j8`.
  Commit local on `bigpath` only; no remote push.

## 1. Build result

Clean build `make -j8` from a `make clean` state: **exit 0**.
Warning delta vs the HEAD baseline (same toolchain):

| Baseline (ecfb1278) | After B-2 |
|---|---|
| `gsp_phys_booter.c` frame 1120 B > 1024 (`gsp_phys_backend_guest_gr_submit`) | frame **1104 B** (same pre-existing warning, shrunk by the static scratch) |
| `gsp_sema_service.c:12518` unused variable `n_win` | removed (fixed in passing) |
| `main.c:402` frame 1304 B; `%x` vs `unsigned long` in `gsp_phys_rmcall_stage3` | unchanged (pre-existing) |

No new warnings; the touched lines are `-Wextra` clean.

## 2. Site → change table

| # | Spec item | Site (post-change) | Change |
|---|---|---|---|
| 1 | Constants | `kernel/gsp_phys_booter.h:366-369` | `GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0` 0x0A000000, `_SLOT0` 80, `_PT_PAGES` 32, `_PAGES` 32*512=16384. Shared header constant: the SVC derives its reloc base and cap from here (no drift, design R5c). |
| 2 | Boot chunk fields | `kernel/gsp_phys_booter.c:2325-2330` | `struct gsp_phys_boot_chunk rmcall_gr_data_bounce;` (16384 pages) + `rmcall_gr_data_bounce_pt` (32 PT pages) added to `boot_stage`. |
| 3 | Stage-3 plant | `kernel/gsp_phys_booter.c:9770-9840` | One-shot (`!pages`) plant next to the 208-page bounce plant: allocate the data chunk (`gsp_phys_chunk_alloc`, one call, contiguous + per-page fallback) and the PT chunk, zero both, write PD0 dual entries 80..111 (same shape as ext pool 32..79: 0 at `slot*16`, `gsp_phys_rmcall_pde(dma)` at `slot*16+8`), then for every page i write `gsp_phys_rmcall_pte(page_dma(i))` at `PT[i/512] + (i%512)*8`; `dma_wmb()`; log `data-bounce region planted: …`. All at stage 3, before any engine fetch. Gated by `gsp_phys_gr_data_region`. |
| 4 | Old window untouched | `gsp_phys_booter.c:9727-9769` (unchanged), mail `9756`/marker `9765`, mail injection mva `17492-17493` | The 208-page window, mail slot 206, marker 207 and the bank-4 mail injection stay byte-identical; only its comment records the B-2 decommissioning as a data carrier. |
| 5 | `bi` helper | `gsp_phys_booter.c:16933-16953` (`gsp_phys_gr_data_chunk`), `16961-16982` (`gsp_phys_gr_data_bi`) | One helper returns `(reloc_va − VA0)>>12` bounded `< 16384` for the grown region (`GSP_PHYS_GR_DATA_BI_NONE` outside); with the parameter off it serves the legacy window bounded by `MAIL_SLOT`. A chunk selector names the active data chunk. |
| 6 | Five sites replaced | retire wait `:17025`, fill_bounce `:17091`, copy-in `:17972`, copy-out `:18653`, window diag `:18698` | All five inline `(reloc_va − 0x110000)>>12` + `MAIL_SLOT` bounds replaced by the helper + chunk selector. No other copy-path semantics changed. |
| 7 | Checksum/scratch arrays | `bounce_csum_in` `:17156` → 16384 entries (static); `checked[]` retire wait `:17015` → static 16384 entries | 64 KB each, off the kernel stack (design risk #4). Single-inflight caller documented in place. |
| 8 | SVC reloc base | `gsp_sema_service.c:12371-12372`, used `:12768` | `SVC_GR_DATA_RELOC_BASE` = shared grown constant; `_OLD` 0x110000 for the parameter-off path. |
| 9 | SVC per-ptr cap | `gsp_sema_service.c:12381-12382`, used `:12703` | 4096 (runaway guard); old 68 kept for legacy. Stop-on-miss still bounds real spans. |
| 10 | SVC total cap, fail-closed | `gsp_sema_service.c:12391-12392`, check `:12749-12767` | Cap = `GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES` (16384); on hit: `svc_gr_data_stalled = true` + `pr_err(… CAP-HIT: region full, launch held fail-closed …)` + return — no silent truncation, the existing bounded hold/retry path takes over. Old 205 kept for legacy. |
| 11 | SVC array headroom | `gsp_sema_service.c:3931` | `SVC_GR_DATA_MAX` 16384 → 32768 (gr_data[] and the backend hand-off array). |
| 12 | R3 hygiene | `gsp_sema_service.c:12671-12700` | Skip the launch's own param VA and the qmd_word 32..47 gr_pages (bank/param) as data candidates, next to the existing QMD-VA skip. Critical for B-2: without it the param-page candidate would consume up to a 4096-page span of resolvable guest memory. |
| 13 | Static asserts | `gsp_phys_booter.c:2365-2410` | Build-time disjointness/geometry contract: 32 PT pages × 512 = 16384 pages; VA0 == slot 80 × 2 MB; root/PD2/PD1 index 0 and PD0 80..111 ≤ 128; end ≤ VAS vaSize 0x10000000; region above ext pool end 0x0A000000; region disjoint from old bounce 0x110000..0x1dffff, win0 0x1e0000.., guest scratch 0x200000... |
| 14 | Module param | `kernel/gsp_phys_gpu.c:1080-1097`, decl `kernel/gsp_phys_gpu.h:357` | `gsp_phys_gr_data_region` (int, 0644, default 1). Read by the stage-3 plant, the SVC fill and the copy path. Boot-fixed (set at insmod). |
| 15 | Teardown | `gsp_phys_booter.c:19611-19613` | Free the two new chunks in `gsp_phys_rmcall_fini` (and, as a one-line in-passing fix, the old 208-page bounce chunk that was leaked there). |

## 3. Deviations from the design

1. **Shared header constants instead of per-file duplicates.** The four
   region constants live in `gsp_phys_booter.h` and `gsp_sema_service.c`
   derives `SVC_GR_DATA_RELOC_BASE`/`SVC_GR_DATA_BOUNCE_MAX` from them. The
   design (§3.2, R5c) explicitly wanted the relocation base "comment-bound"
   or shared to prevent drift; a shared constant is the stronger form.
2. **`bi` helper covers both modes.** The design's helper description assumes
   the grown region only. Because the module parameter must flip the whole
   data path back (spec item 7), the helper also serves the legacy window
   (bounded by `MAIL_SLOT`), and a small `gsp_phys_gr_data_chunk()` selector
   picks the chunk for the copy path. Behaviour with the parameter on is
   exactly the design's.
3. **`checked[]` made static, not a bitmap.** The design allowed either; the
   static array mirrors the existing single-inflight static scratch pattern
   and keeps the wait logic byte-identical. The dedup loop remains O(n²) as
   at HEAD; worst case (16384 records) is ~134M u32 compares once per
   launch — accepted for bring-up (see residual risks).
4. **Old bounce chunk now freed in `rmcall_fini`.** Pre-existing leak spotted
   next to the new frees; freeing it is safe (`gsp_sema_service_work_quiesce`
   runs before `gsp_phys_boot_staging_free`). Reported here because it is a
   teardown behaviour change beyond the B-2 allocation lifecycle.
5. **Memory-leak observation not fixed:** `rmcall_gr_shadow_win0` and
   `rmcall_gr_mail` are also never freed in `rmcall_fini` (pre-existing, out
   of scope). Likewise `rmcall_ctx_pt[]` last slot can collide with the ext
   pool PD0[0] if the server MAIN context ever reaches the 32 MB maximum
   (pre-existing latent; the live MAIN is ~1.2 MB). Neither is touched here.
6. **No F1-style permanent refusal on a persistent cap hit.** Per the spec,
   the cap hit marks the launch stalled for the existing bounded hold. Note
   honestly: that hold is one-shot per GPFIFO entry; if the region is
   genuinely too small (negative control, > 16384 unique operand pages), the
   entry can be consumed after the hold window expires, exactly like every
   other stalled launch at this revision. The CAP-HIT `pr_err` makes it
   visible, but the permanent fail-closed guarantee is `AGENT-bigcopy`
   F1/§4 territory and was explicitly kept out of this change (design risk
   #6).

## 4. Risks

**Design risk #1 — engine walker reach for PD0 80..111 (LOW).** The plant
shape is byte-for-byte the ext pool pattern (PD0 dual entry small half →
fresh 4 KB PT; leaves = `pte(dma)`), which is proven engine-readable at PD0
32..79; entries 80..111 sit in the same PD0 page and same root/PD2/PD1 chain
(indices all zero, asserted at compile time). The only unproven ingredient is
the slot range itself. Fallback if the first big launch faults/times out with
`changed=0` and the data chunk otherwise healthy: move the region to the
reserved win1 data windows (`GSP_PHYS_RMCALL_VA_GR_DATA_BASE` 0x100000000,
prewin install) — the copy-path changes carry over unchanged.

**Design risk #3 — 64 MB contiguous `dma_alloc_coherent` fails → per-page
fallback (MEDIUM).** `gsp_phys_chunk_alloc` warns and falls back to 16384
per-page coherent allocations; boot-time stage-3 cost grows (seconds) and
memory is fragmented across pages (harmless to correctness: each PTE uses the
page's own DMA). Watch the boot log for `contiguous 16384-page coherent
allocation failed` and stage-3 wall time. Mitigation if needed: split the
chunk into a few smaller contiguous allocations — deferred until measured.

**Other risks carried from the design:**
* *Per-pointer span overshoot.* 4096 pages is a runaway guard; stop-on-miss
  still bounds a real operand. If a client's operands are laid out
  contiguously with resolvable neighbours, one candidate can consume up to
  4096 pages and the cap can be reached before all operands are in. The
  fail-closed stall + CAP-HIT log would expose this on the first big cycle;
  the cap value (design open question #4) is the tuning knob.
* *Copy cost.* Up to 64 MB in + 64 MB out per launch (tens of ms), plus up to
  300 × 5 ms retire-wait polls over 16384 pages when nothing changes — all
  bounded by the existing wait exits; correctness first (design risk #8).
* *Param flip consistency.* `gsp_phys_gr_data_region` is boot-fixed; a
  runtime flip after the stage-3 plant would desync SVC vs plant. Documented
  in the param description; the default cannot desync.

## 5. Grep audit — remaining uses of the old bounce VA0 / 0x110000

`grep -rn "RMCALL_GR_BOUNCE_VA0|SVC_GR_DATA_RELOC_BASE_OLD|0x110000"
kernel/*.c kernel/*.h` (excluding unrelated hits `bar_emulation.c:62` LTC,
`gsp_emulation.h:40` firmware block):

| Site | What it is | Indexes data pages? |
|---|---|---|
| `gsp_phys_booter.c:2338` | the legacy constant definition | no |
| `:2349-2351` | win0-vs-bounce overlap `static_assert` (layout) | no |
| `:2399` | B-2 disjointness assert | no |
| `:9744-9745` | old bounce plant `pr_info` | no |
| `:1011, 2306, 2320, 2371, 16952` | comments/history | no |
| `:16975, 16977` | `gsp_phys_gr_data_bi` legacy branch — reachable only when `gsp_phys_gr_data_region=0`, where operand records really are in that window; bound `MAIL_SLOT` excludes mail/marker | only in legacy mode, by design |
| `:17492-17493` | mail-injection `mva = VA0 + (MAIL_SLOT<<12)` — the mail message page, never a data record | no |
| `gsp_sema_service.c:12372` | the legacy reloc base constant (param-off path) | only in legacy mode |
| `gsp_sema_service.c:12367, 12478, 12646` | comments | no |

The five data-page index derivations all now go through
`gsp_phys_gr_data_bi()`; with the grown region on (default), no code path can
index a data page through the old window. The mail injection and the old
window's own plant/diagnostics are the only remaining consumers, exactly as
the design requires.

## 6. Sanity math (checked by computation)

* `32 × 512 × 4096 = 67 108 864 B = 64 MiB` ✔
* VA range `0x0A000000 .. 0x0DFFFFFF` (end-exclusive `0x0E000000`) `<
  0x10000000` (VAS vaSize) with 32 MB slack ✔
* PD0 slot = `0x0A000000 >> 21` = 80; slots 80..111 ≤ 128 (non-shared VAS
  ceiling) ✔
* Disjointness: ext pool `[0x04000000, 0x0A000000)` meets the region exactly
  at VA0; old bounce ends `0x1E0000`, win0 ends `0x1F0000`, guest scratch
  ends `0x2200000` — all far below 0x0A000000 ✔ (all compile-time asserted)

## 7. Live validation commands for the lead (supervised; NOT run here)

```sh
# 1. Bring the branch build into the campaign tree (lead decides merge vs
#    manual copy of vgsp.ko) and confirm the param exists:
git -C /home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath log --oneline -1
cd /home/rmcore/Git/integdev_gpu_drv/open-iov-bigpath/kernel && make -j8
modinfo vgsp.ko | grep gsp_phys_gr_data_region

# 2. One full supervised cycle with the B-2 default (grown region ON;
#    module args unchanged, gsp_phys_gr_data_region defaults to 1):
cd /home/rmcore/Git/integdev_gpu_drv/open-iov
./backups/evidence-20261002-victory/victory_grfwd_cycle.sh b2-r1

# 3. Boot gate (host dmesg) — expect exactly one plant line, no alloc error:
#    vgsp-boot: rm-call: data-bounce region planted: 16384 pages at VA
#    0xa000000..0xdffffff (PD0 slots 80..111, 32 PT pages, before first fetch)
#    and no "data-bounce region allocation failed" / no
#    "contiguous 16384-page coherent allocation failed" (or accept the
#    fallback warning and time stage 3).

# 4. Guest clients: push the battery and run the three big clients plus the
#    regression set (the cycle already runs the runner; for manual runs):
./tools/cuda-ab/push_battery.sh
# in the guest: run /tmp/victory_math /tmp/vector_add /tmp/memcpy_roundtrip
# plus regressions (victory_matmul smiley mall victory_mail victory_saxpy
# victory_strided victory_reduce victory_atomics victory_memset2d
# victory_streams_events vecadd_loop).
# Expected host log: "svc-gr data fill: … pages resolved (reloc 0xa000000+)"
# with NO "CAP-HIT"; copy-out "changed by engine" > 0 naming output pages;
# guest: MATH badf=0 badd=0 badi=0 / RESULT bad=0 / ROUNDTRIP bad=0 csum=269221888.

# 5. Rollback / A-B trial (legacy 205-page window, one flip at insmod):
./backups/evidence-20261002-victory/victory_grfwd_cycle.sh b2-legacy-r1 gsp_phys_gr_data_region=0

# 6. Negative control (over-region, > 16384 unique operand pages, e.g.
#    vector_add n=2^23): expect the CAP-HIT pr_err + hold, no silent success.
```

## 8. Rollback parameter

`gsp_phys_gr_data_region=0` at insmod restores the legacy 205-page /
68-pages-per-pointer path (reloc base 0x110000): the stage-3 plant is skipped,
the SVC keeps the old caps, and the copy path indexes the old bounce chunk.
Boot-fixed; do not flip at runtime.