tools/cuda-ab: P2 arbitrary-kernel suite expansion (regs/streams/printf), smiley push wiring, near-native perf harness (offline readiness; guest validation pending)

1 files changed, 365 insertions, 0 deletions

diff --git a/tools/cuda-ab/perf/PLAN-AND-HARNESS.md b/tools/cuda-ab/perf/PLAN-AND-HARNESS.md
new file mode 100644
index 0000000..74eed59
--- /dev/null
+++ b/tools/cuda-ab/perf/PLAN-AND-HARNESS.md
@@ -0,0 +1,365 @@
+# Near-Native Performance: Plan and Harness
+
+> **Status: PREPARED 2026-10-08 (SA-CHK, offline performance workstream). NOT
+> YET RUN — every threshold below is provisional until the first calibration
+> run on a verified-green stack.** This document defines *how* the near-native
+> performance program measures and records results, so that the moment the
+> reproducibility gate is green the lead can run `perf.sh` and file numbers.
+> Prepared during a supervised physical campaign cycle: nothing here touched
+> the GPU, the module, QEMU or the guest.
+>
+> Companion contracts: `docs/05-operations/01-performance-correctness-security.md`
+> §3 (metrics, A/B method, targets) and §5 (reporting); `docs/eng-ops.md`
+> (admission rule: scenario + measurable outcome + baseline + threshold +
+> failure signal — this document supplies all five per metric);
+> `handoff-meta-plan.md` (live state, §2 deltas; step 6 of the next-session
+> protocol: "near-native performance numbers — the 189 ms r3 launch is the
+> baseline").
+
+---
+
+## 1. Baseline extraction — what `elapsed_ms=189` actually measured
+
+### 1.1 Evidence
+
+`m141-r3` (2026-10-07, module sha256 `83d78872…`, commit `895ab8b8`) is the
+first M1 win. The performance-relevant lines:
+
+| File | Line | Content |
+|---|---|---|
+| `backups/evidence-20261002-victory/run-m141-r3.log` | 128 | `vgsp-gr: GUEST GR LAUNCH OK h=0x5c000019 push=0x20043be94 qmd=0x204608000 prog/pages=7 qmd_rel=0x00000006 ch_rel=0xc0ffee02 elapsed_ms=189 ok=1` |
+| `backups/evidence-20261002-victory/summary-m141-r3.txt` | 23 | same line |
+| `backups/evidence-20261002-victory/11-hostdmesg-m141-r3.log` | 6600 | launch 1: `elapsed_ms=189 ok=1` (mail kernel) |
+| `backups/evidence-20261002-victory/11-hostdmesg-m141-r3.log` | 6757 | launch 2: `elapsed_ms=166 ok=2` (vadd kernel) |
+
+The run's acceptance section also shows the counter line
+`work_gr: req=2 fallback=0 ok=2 refused=0 timeout=0 fault=0 submitted=2
+backend_ok=2 …` — two forwarded launches, both green.
+
+### 1.2 What emitted it (source path)
+
+`elapsed_ms` is emitted by `gsp_phys_backend_guest_gr_submit()` —
+`kernel/gsp_phys_booter.c:16457` — the **host-side backend service time for
+one forwarded guest GR launch**:
+
+* `unsigned long start = jiffies;` at function entry (`:16475`);
+* `pr_info("vgsp-gr: GUEST GR LAUNCH OK … elapsed_ms=%u …",
+  jiffies_to_msecs(jiffies - start), …)` at `:17757`;
+* the same value is copied to the caller's `struct … { u32 elapsed_ms; }`
+  result at `:17776`.
+
+So the number is a **host-kernel wall time measured in jiffies** (this kernel:
+`CONFIG_HZ=1000`, so 1 ms granularity). It is *not* the guest-observed time,
+*not* the CUDA API time, and *not* pure kernel execution.
+
+### 1.3 Decomposition of the 189 ms (from the r3 dmesg timeline)
+
+The backend function entered at ~`17209.348` (host monotonic seconds) and
+returned at `17209.536501`. Inside that window, for launch 1:
+
+| Component | Evidence | Cost |
+|---|---|---|
+| 8 serialized `PDE_UPDATE` forwards to the real GSP (fn76, ctrl `0x80180f`, seqNum 113–120): 1 scratch-window install + 7 guest-window attempts (2 accepted, 6 refused `status=0x1f`) | `TX submitted …` → `rm-call: … reply` pairs, each ~22–23 ms | ~180 ms |
+| QMD relocation/pre-write + param/program shadow copies | `qmdwin: relocating …` → `bounce copy-in: 100 pages` | < 1 ms |
+| Doorbell / GPFIFO publish → completion poll | `GR LAUNCH submitted` `17209.528630` → `GUEST GR LAUNCH OK` `17209.536501` | **7.87 ms** |
+| 100-page bounce copy-out | `bounce copy-out: 100 pages written back` | < 0.1 ms |
+
+Launch 2 (vadd): `elapsed_ms=166`, doorbell→OK **7.95 ms**. Each PDE round
+trip is dominated by the 5 ms poll quantum (`GSP_PHYS_RMCALL_POLL_MS = 5u`,
+`gsp_phys_booter.c:1404`) plus firmware service time; the per-launch poll cap
+is 5000 ms (`GSP_PHYS_RMCALL_GR_POLL_MS`, `:1411`).
+
+**Reading for the program:** the r3 189/166 ms is a *forwarding-service*
+latency, of which ~4 % is the engine completion path. Reducing it means
+batched/eliminated PDE forwards and fewer RPC round trips — not engine work.
+The post-doorbell ~7.9 ms is the number to watch as the launch path gets
+optimized.
+
+### 1.4 Historical spread (other runs, read-only scan)
+
+* First forwarded launch (`ok=1`, mail): run/summary artifacts show
+  `183×4, 184×6, 185×6, 186×2, 189×2`; full-dmesg samples add `197, 198, 200,
+  201`.
+* Second launch (`ok=2`, vadd): `164 … 176` across sampled logs; m145-r3
+  shows two pairs (`198/171`, `197/174`).
+
+**Canonical baseline (to quote):** `elapsed_ms=189` (launch 1) and `166`
+(launch 2), run `m141-r3`, module `83d78872…`. Use the pair, not one number.
+
+### 1.5 What the baseline does NOT say
+
+* No guest-observed (`cudaEvent`) launch latency was recorded in r3.
+* No copy bandwidth, no malloc latency, no init time, no variance (n=2
+  launches is not a distribution).
+* No host oracle was measured with the *same new binary* in the same boot as
+  r3 (the `tools/cuda-ab/run/victory_mail` oracle exists, but not the perf
+  workload).
+* Module diagnostics (`sema_dbg`) were on in the campaign recipe, so the
+  absolute numbers include tracing/log overhead. The perf protocol measures
+  with the campaign's accepted recipe and says which parameters were set.
+
+---
+
+## 2. Measurement matrix
+
+ID legend: **G** = in-guest instrument, **H** = host (kernel trace / counter /
+dmesg), **X** = cross-check. All guest metrics come from
+`perf_workload.cu` `PERF` lines; all host metrics from the vgsp surfaces in
+§3. "Oracle" column is the same binary on bare metal.
+
+| ID | Metric | Instrument | Baseline (r3) | Near-native target + how calibrated |
+|---|---|---|---|---|
+| P1 | Process start → device ready (`init_ms`) | **G** CLOCK_MONOTONIC in `main()` → `cudaGetDeviceProperties`+`cudaFree(0)` (`PERF init_ms`) | none recorded | ≤ **2.0×** host oracle same binary (docs 3.3). Calibrate: 3 host + 10 guest runs; if host is tiny, additionally cap at max(50 ms, 25× host). |
+| P2 | `cudaMalloc` first + steady `p50/p95` | **G** wall clock around `cudaMalloc/cudaFree`, 50 steady iters (`PERF malloc_first_ms`, `malloc_ms`) | none | No-regression (≤1.2× prior artifact) in phase 1; working gate `p95 ≤ 5×` host p95 once calibrated. |
+| P3 | First kernel launch, guest end-to-end | **G** `cudaEventRecord`→launch→sync (`PERF launch_first_ms`) | none (r3 has no CUDA timing) | Phase 1: record. Phase 2: `≤ 5×` host same-binary first launch, tightened toward 2× after the forwarding cost is attacked. |
+| P3h | First launch, host backend service (the r3 metric) | **H** dmesg `GUEST GR LAUNCH OK … elapsed_ms=`; also `GR LAUNCH submitted`→OK delta | **189 ms** (mail), **166 ms** (vadd) | No-regression ≤ **1.2×** previous recorded p50/p95 for the same module+recipe; separately drive to ≤ `2×` host P3. |
+| P4 | Warm launch latency `p50/p95` | **G** per-launch event+sync, N=100 (`PERF launch_warm`) | none | No-regression ≤1.2×; working gate `p50 ≤ 5×` host p50 (phase 2), tighten to 2× when feasible. |
+| P5 | CE push latency (submit → release) | **H** kprobe `gsp_phys_backend_guest_ce_copy[_ex]` entry/return per forwarded copy (or `sema-optrace` with `sema_dbg=1`); **G** 4 KiB sync copy as the end-to-end proxy (`PERF copy dir=htod bytes=4096`) | P6b live runs recorded counters (`req=100 ok=100`), no latency | Report p50/p95; working gate: 4 KiB proxy `≤ 5×` host 4 KiB copy; no-regression. |
+| P6 | MSI-X injection latency | **H** trace delta `gsp_rm_post_channel_event` → `gsp_emu_inject_completion_interrupt` (kprobes or `trace_pipe` timestamps) | none | p95 **≤ 1 ms** provisional (pure virtualization artifact); stability over absolute. Calibrate on first run. |
+| P7 | HtoD bandwidth | **G** size sweep 4 KiB…16 MiB, best of 3 (`PERF copy dir=htod`) | none | **≥ 95 %** of host at ≥ 4 MiB (docs 3.5); small sizes report-only. |
+| P8 | DtoH bandwidth | **G** same (`PERF copy dir=dtoh`) | none | **≥ 95 %** at ≥ 4 MiB. |
+| P9 | Kernel execution vs bare-metal oracle | **G** bit-exact FMA chain, event-timed (`PERF kernel_fma ms/gflops`) | none | **≥ 95 %** host GFLOP/s (docs 3.5 `vectorAdd ≥95%`); absolute GFLOP/s also reported. |
+| P10 | Steady-state throughput | **G** 100 launches enqueued, one sync (`PERF throughput kps`) | none | No-regression; working gate `≥ 20 %` of host kps (i.e. ≤5×), tightened as P3h/P4 improve. |
+| P11 | Per-run variance (mean/median/p95) | **G/H** `collect.py` aggregate over R=10 process runs; report p95/median per metric | none | Warm unchanged stack: dispersion `p95/median ≤ 1.5` for every metric; above that, investigate before optimizing. |
+| P12 | Kernel cycles (cross-check) | **X** `clock64()` start/end per block, 10 k FMA chain (`PERF clock64`) | none | Report only. Never treat cycles as time without a calibrated SM clock; never compare cycles across host/guest as a time metric. |
+
+**Measurement-method classes (required by the plan):**
+
+* **Host trace / kprobes** — P3h, P5, P6. Exact recipes in §3.2. No kernel
+  change needed; module symbols are probeable at runtime. Arming kprobes is a
+  supervised host action and does not modify the module or the campaign
+  scripts.
+* **In-guest wall clock** — P1, P2, P3, P4, P7–P11: monotonic clock and CUDA
+  events inside the workload, exactly as `p2_bench` does.
+* **`clock64()`** — P12 only, as a cross-check; per-SM counter, report as
+  cycles.
+
+---
+
+## 3. Methods and exact commands
+
+### 3.1 Harness files
+
+| File | Purpose |
+|---|---|
+| `perf_workload.cu` | one binary, both roles; emits stable `PERF` lines; self-verifying (bad=0 required) |
+| `perf.sh` | `check/build/host/guest/all/compare/push`; host oracle, guest push+run via `scripts/vgpu_vm.py`, host evidence capture, ratio compare |
+| `collect.py` | stdlib-only parser/aggregator: `PERF` lines + dmesg + `gsp-phys-state` → `open-iov-perf/1` JSON; `--compare` ratio table |
+| `results-template.json` / `results-template.md` | result schemas for the JSON artifact and the public summary |
+| `bin/`, `results/` | generated (gitignored) |
+
+Style follows `tools/cuda-ab/ab.sh`: same guest transfer (base64 chunks via
+the guest agent), same static-CUDA-runtime build (`-O2 -arch=sm_75
+--cudart=static`), same oracle/measured/compare split. `perf/` supersedes the
+stub `tools/benchmark_infrastructure.sh` for CUDA metrics; the A/B pattern
+stays as docs §3.5 requires.
+
+### 3.2 Exact commands
+
+```bash
+cd ~/Git/integdev_gpu_drv/open-iov/tools/cuda-ab/perf
+
+./perf.sh check                    # nvcc, VM reachability, host-GPU/state readability
+
+# 1) Bare-metal oracle — run while the GPU is on the distribution driver
+#    (before the campaign claims it, or after release):
+./perf.sh host                     # -> results/<stamp>-host/results.json
+
+# 2) Green campaign cycle; verify the green gate (victory_mail MAIL/VADD/STATUS OK),
+#    then, with the VM up:
+PERF_STAMP=<same-or-new-tag> PERF_RUNS=10 PERF_N_WARM=100 PERF_N_THRU=100 \
+  ./perf.sh guest                  # -> results/<stamp>-guest/{results.json,compare...}
+
+# 3) Compare (host is the oracle; every ratio is guest/host):
+PERF_ORACLE=results/<stamp>-host/results.json ./perf.sh compare
+# or standalone:
+python3 collect.py --compare results/<h>/results.json results/<g>/results.json \
+  --out results/<g>/compare.json [--thresholds thresholds.json]
+```
+
+Bring-up shortcut (faster, not for the record): `PERF_RUNS=3 PERF_N_WARM=20`.
+
+What `guest` captures, besides stdout (all read-only; never `dmesg -C`):
+
+| Artifact | Source |
+|---|---|
+| `host-state-before.txt` / `host-state-after.txt` | `cat /sys/kernel/debug/vgsp/gsp-phys-state` (sudo -n fallback) |
+| `host-gr-dmesg.txt` | `dmesg \| grep -E 'vgsp-gr:\|work_gr:\|sema-gr:\|vgsp-sema: sema-svc:' \| tail -n 3000` |
+| `dmesg-marker.txt` | last `dmesg` line before the run; `collect.py` drops host-GR entries at or before this raw monotonic timestamp, scoping the aggregate to this run |
+| `meta.txt` | stamp, uname, `sha256sum kernel/vgsp.ko`, run config |
+| `meta-guest.txt` | guest `nvidia-smi --query-gpu` + `nvidia-smi -q -d CLOCK` |
+| `run-N.txt`, `run.txt` | per-run and merged workload stdout |
+
+### 3.3 Optional host kprobes (P3h/P5/P6 distributions)
+
+```bash
+TR=/sys/kernel/tracing   # or /sys/kernel/debug/tracing
+sudo bash -c "cd $TR
+  echo 'p:fgr gsp_phys_backend_guest_gr_submit'   >> kprobe_events
+  echo 'r:fgr_ret gsp_phys_backend_guest_gr_submit \$retval' >> kprobe_events
+  echo 'p:fce gsp_phys_backend_guest_ce_copy'     >> kprobe_events
+  echo 'r:fce_ret gsp_phys_backend_guest_ce_copy \$retval' >> kprobe_events
+  echo 'p:ev gsp_rm_post_channel_event src=%di'   >> kprobe_events
+  echo 'p:msix gsp_emu_inject_completion_interrupt seq=%si' >> kprobe_events
+  echo 1 > events/kprobes/enable"
+```
+
+Read `trace` / `trace_pipe` around a workload window; entry↔return deltas give
+the per-call distributions (P5/P6; P3h is already in dmesg). If a symbol is
+not present in `/proc/kallsyms` (inlined), skip it — the dmesg + guest-event
+path still yields P3h/P4.
+
+### 3.4 The green gate (precondition, not optional)
+
+A perf run is valid only if, in the same boot/VM:
+
+1. `victory_mail` printed `MAIL …` + `VADD … bad=0` + `STATUS OK`;
+2. `perf_workload` ended `PERF STATUS OK` with `bad=0` on **both** sides;
+3. `gsp-phys-state` `work_gr: req == ok` after the run and
+   `pending release starved` did not appear (ISSUE-0001 gate);
+4. host and guest GPU clocks were captured (`nvidia-smi -q -d CLOCK`).
+
+If any gate fails, the numbers are diagnostics, not results — label them so.
+
+---
+
+## 4. Thresholds and calibration procedure
+
+Everything is calibrated on the first run; the method is fixed now:
+
+1. **Oracle first.** Run `perf.sh host` while the GPU is unclaimed
+   (pre-claim or post-release). Same machine, same boot if possible; if the
+   oracle is from a different boot, record both stamps and treat cross-boot
+   as extra noise.
+2. **Formal guest run.** `PERF_RUNS=10 PERF_N_WARM=100`, one green cycle.
+3. **Freeze thresholds** in `perf/thresholds.json` (created at recording
+   time, owned by the lead):
+
+   | Metric class | Rule (`thresholds.json` semantics) |
+   |---|---|
+   | `init_ms` | `{"mode":"max_ratio","limit":2.0}` (docs 3.3) |
+   | `copy_*_gbps` ≥ 4 MiB, `kernel_fma_gflops` | `{"mode":"min_ratio","limit":0.95}` (docs 3.5) |
+   | `launch_warm_p50`, `throughput_kps` | phase 1: no-regression only; phase 2 `max_ratio 5.0`, tighten to 2 — tighten only with a dated run showing it met |
+   | `malloc_ms_p95`, small-copy proxy | `max_ratio 5.0` provisional |
+   | host GR `elapsed_ms` (P3h) | no-regression vs previous artifact (`max_ratio 1.2`) |
+   | MSI-X p95 | absolute cap 1 ms provisional |
+   | dispersion (P11) | `p95/median ≤ 1.5` per metric |
+
+4. **No-regression rule.** Until the launch path is optimized, the primary
+   acceptance for P3h/P4/P10 is: a new measurement must not be worse than
+   **1.2×** the previous recorded p50/p95 for the same module + recipe.
+5. **Update discipline.** Thresholds change only with a fresh dated
+   measurement and a note in the results document; never silently.
+
+Report **guest/host ratios**, never absolute-only (docs §3.5). A threshold
+miss is a finding, not a failure of the harness.
+
+---
+
+## 5. Harness walkthrough — the workload
+
+`perf_workload.cu` (one binary, both roles):
+
+* `PERF init_ms` — `main()` → device properties + `cudaFree(0)` (P1).
+* `PERF malloc_first_ms`, `PERF malloc_ms n=50 mean/p50/p95` (P2).
+* `PERF launch_first_ms` — first `pattern_kernel` launch, event → sync (P3).
+* `PERF launch_warm n=… mean/p50/p95` — per-launch event+sync, default 50,
+  protocol 100 (P4/P11).
+* `PERF kernel_fma ms/gflops` + bit-exact host chain verification (P9).
+* `PERF throughput n=… wall_ms kps` — 100-launch burst, one sync (P10).
+* `PERF copy dir=htod|dtoh bytes=… ms gbps` — 4 KiB/256 KiB/4 MiB/16 MiB,
+  best-of-3 after a warmup; 4 KiB round trip verified (P5/P7/P8).
+* `PERF clock64 available/min/median` — per-block cycles, cross-check (P12).
+* `PERF checksum … bad=0` + `PERF STATUS OK` — verification gate.
+
+Host-side pairing: `collect.py` reads the same run count from
+`gsp-phys-state` (`work_gr` counters, `work_relay`) before/after and parses
+`GUEST GR LAUNCH OK … elapsed_ms=` plus `GR LAUNCH submitted` → OK deltas
+from the dmesg capture (P3h). The capture is scoped by `dmesg-marker.txt`
+(the last dmesg line before the run) so entries from earlier work in the same
+boot cannot leak into the aggregate; raw monotonic timestamps are required
+for that scoping, so capture `dmesg` without `-T` (the human `-T` form is
+still accepted for the `elapsed_ms` values, just not for scoping).
+
+---
+
+## 6. Evidence discipline and recording
+
+1. **No claim without a fresh measurement file.** Every performance statement
+   cites `perf/results/<stamp>-*/results.json` (+ `compare.json`) produced by
+   this harness after this plan, on a green stack. Fresh = same module sha256
+   and recipe as the claim; re-used numbers are labelled as such.
+2. **Never edit campaign scripts or evidence.** `perf/` writes only under
+   `perf/results/` and `perf/bin/` (gitignored); the guest side only writes
+   `/tmp/perf_workload*`.
+3. **Durable copy.** The lead copies `results/<stamp>-{host,guest}/` to the
+   running campaign's evidence directory (`backups/evidence-…/perf-<stamp>/`)
+   as an immutable snapshot; raw results may then be pruned.
+4. **Public summary (docs/).** Create
+   `open-iov/docs/05-operations/05-performance-results.md` from
+   `results-template.md`, and link it from
+   `docs/05-operations/01-performance-correctness-security.md` §5 (reporting
+   list) and the `docs/README.md` reading-order table. Tracked docs stay
+   component/behaviour-only (no private paths, per AGENTS.md).
+5. **Handoff §2 delta.** Append to `handoff-meta-plan.md` §2 (the live-state
+   section; `Handoff-vGPU-IO.md` stays frozen), template:
+
+   ```markdown
+   ### 📈 SESSION <date> <time> — PERFORMANCE BASELINE (near-native program, run <stamp>)
+   Green gate: victory_mail OK; perf_workload bad=0 both sides; work_gr req==ok.
+   Module <sha256> recipe <exact insmod args> commit <rev>.
+   Oracle: results/<stamp>-host (clocks …). Measured: results/<stamp>-guest.
+   | metric | host | guest | g/h | threshold | verdict |
+   |---|---|---|---|---|---|
+   | P1 init_ms | … | … | … | ≤2.0× | … |
+   | P3h GR elapsed_ms | n/a | … (prior 189/166) | — | ≤1.2× | … |
+   | P4 launch_warm p50/p95 | … | … | … | ≤5× (p1) | … |
+   | P7 HtoD 16MiB | … | … | … | ≥95% | … |
+   | P9 fma GFLOP/s | … | … | … | ≥95% | … |
+   (full table in docs/05-operations/05-performance-results.md; compare.json)
+   Follow-ups: <limiting metric + next optimization candidate>.
+   ```
+6. **Clocks and configuration are part of the evidence.** Record host and
+   guest `nvidia-smi -q -d CLOCK` digests; note `sema_dbg` and other
+   diagnostic parameters from the recipe (they cost absolute time but
+   preserve comparability run-to-run while unchanged).
+7. **One physical writer.** The lead owns running `perf.sh` during a campaign;
+   offline roles prepare and review. Benchmarks never run concurrently with
+   another workload on the same GPU.
+
+---
+
+## 7. Limitations and risks
+
+* **Host oracle availability sequencing.** During a green cycle the module
+  owns the GPU, so `perf.sh host` cannot run then. The oracle must be captured
+  pre-claim/post-release; record both stamps. `perf.sh all` warns and
+  continues, but the named protocol is separate windows.
+* **Jiffies quantization and tracing overhead.** P3h is 1 ms-quantized
+  (`HZ=1000`) and includes `sema_dbg` logging when that parameter is on; the
+  matrix states this rather than correcting for it.
+* **Emulated VRAM.** Guest "VRAM" is host RAM today (docs §3.6 item 5): bulk
+  copy bandwidth is not representative of real VRAM until the backing path
+  exists — report, do not over-claim.
+* **Cross-boot comparisons are noise** (docs §3.2); prefer same boot.
+* **`clock64` is an SM cycle counter**; frequency may differ per SM and is
+  not a time source without calibration.
+* **Small N.** 10 process runs bound the p95 confidence; the protocol states
+  N, and dispersion (P11) is itself a metric.
+* **First-run effects.** `init_ms`/`malloc_first_ms` include one-time driver
+  initialization; they are process-start metrics by design and are reported
+  as such, not averaged into steady state.
+
+## 8. Open items for the measurement session
+
+1. Run `perf.sh check` before the first green cycle to confirm nvcc, VM
+   executor, debugfs readability and the oracle window.
+2. Capture the host oracle **before** the next claim if the GPU is free; if
+   not, schedule one host-oracle window.
+3. After the first formal run: freeze `perf/thresholds.json`, write
+   `docs/05-operations/05-performance-results.md`, append the §2 delta, and
+   list the limiting metric as the next optimization candidate (expected
+   first candidate from §1.3: the serialized per-window `PDE_UPDATE` forwards
+   inside `gsp_phys_backend_guest_gr_submit`).
+4. Fold the §2 protocol step 6 pointer to this file when the handoff is next
+   updated by the lead.