path: root/tools/cuda-ab/perf/PLAN-AND-HARNESS.md

# Near-Native Performance: Plan and Harness

> **Status: PREPARED 2026-10-08 (SA-CHK, offline performance workstream). NOT
> YET RUN — every threshold below is provisional until the first calibration
> run on a verified-green stack.** This document defines *how* the near-native
> performance program measures and records results, so that the moment the
> reproducibility gate is green the lead can run `perf.sh` and file numbers.
> Prepared during a supervised physical campaign cycle: nothing here touched
> the GPU, the module, QEMU or the guest.
>
> Companion contracts: `docs/05-operations/01-performance-correctness-security.md`
> §3 (metrics, A/B method, targets) and §5 (reporting); `docs/eng-ops.md`
> (admission rule: scenario + measurable outcome + baseline + threshold +
> failure signal — this document supplies all five per metric);
> `handoff-meta-plan.md` (live state, §2 deltas; step 6 of the next-session
> protocol: "near-native performance numbers — the 189 ms r3 launch is the
> baseline").

---

## 1. Baseline extraction — what `elapsed_ms=189` actually measured

### 1.1 Evidence

`m141-r3` (2026-10-07, module sha256 `83d78872…`, commit `895ab8b8`) is the
first M1 win. The performance-relevant lines:

| File | Line | Content |
|---|---|---|
| `backups/evidence-20261002-victory/run-m141-r3.log` | 128 | `vgsp-gr: GUEST GR LAUNCH OK h=0x5c000019 push=0x20043be94 qmd=0x204608000 prog/pages=7 qmd_rel=0x00000006 ch_rel=0xc0ffee02 elapsed_ms=189 ok=1` |
| `backups/evidence-20261002-victory/summary-m141-r3.txt` | 23 | same line |
| `backups/evidence-20261002-victory/11-hostdmesg-m141-r3.log` | 6600 | launch 1: `elapsed_ms=189 ok=1` (mail kernel) |
| `backups/evidence-20261002-victory/11-hostdmesg-m141-r3.log` | 6757 | launch 2: `elapsed_ms=166 ok=2` (vadd kernel) |

The run's acceptance section also shows the counter line
`work_gr: req=2 fallback=0 ok=2 refused=0 timeout=0 fault=0 submitted=2
backend_ok=2 …` — two forwarded launches, both green.

### 1.2 What emitted it (source path)

`elapsed_ms` is emitted by `gsp_phys_backend_guest_gr_submit()` —
`kernel/gsp_phys_booter.c:16457` — the **host-side backend service time for
one forwarded guest GR launch**:

* `unsigned long start = jiffies;` at function entry (`:16475`);
* `pr_info("vgsp-gr: GUEST GR LAUNCH OK … elapsed_ms=%u …",
  jiffies_to_msecs(jiffies - start), …)` at `:17757`;
* the same value is copied to the caller's `struct … { u32 elapsed_ms; }`
  result at `:17776`.

So the number is a **host-kernel wall time measured in jiffies** (this kernel:
`CONFIG_HZ=1000`, so 1 ms granularity). It is *not* the guest-observed time,
*not* the CUDA API time, and *not* pure kernel execution.

### 1.3 Decomposition of the 189 ms (from the r3 dmesg timeline)

The backend function entered at ~`17209.348` (host monotonic seconds) and
returned at `17209.536501`. Inside that window, for launch 1:

| Component | Evidence | Cost |
|---|---|---|
| 8 serialized `PDE_UPDATE` forwards to the real GSP (fn76, ctrl `0x80180f`, seqNum 113–120): 1 scratch-window install + 7 guest-window attempts (2 accepted, 6 refused `status=0x1f`) | `TX submitted …` → `rm-call: … reply` pairs, each ~22–23 ms | ~180 ms |
| QMD relocation/pre-write + param/program shadow copies | `qmdwin: relocating …` → `bounce copy-in: 100 pages` | < 1 ms |
| Doorbell / GPFIFO publish → completion poll | `GR LAUNCH submitted` `17209.528630` → `GUEST GR LAUNCH OK` `17209.536501` | **7.87 ms** |
| 100-page bounce copy-out | `bounce copy-out: 100 pages written back` | < 0.1 ms |

Launch 2 (vadd): `elapsed_ms=166`, doorbell→OK **7.95 ms**. Each PDE round
trip is dominated by the 5 ms poll quantum (`GSP_PHYS_RMCALL_POLL_MS = 5u`,
`gsp_phys_booter.c:1404`) plus firmware service time; the per-launch poll cap
is 5000 ms (`GSP_PHYS_RMCALL_GR_POLL_MS`, `:1411`).

**Reading for the program:** the r3 189/166 ms is a *forwarding-service*
latency, of which ~4 % is the engine completion path. Reducing it means
batched/eliminated PDE forwards and fewer RPC round trips — not engine work.
The post-doorbell ~7.9 ms is the number to watch as the launch path gets
optimized.

### 1.4 Historical spread (other runs, read-only scan)

* First forwarded launch (`ok=1`, mail): run/summary artifacts show
  `183×4, 184×6, 185×6, 186×2, 189×2`; full-dmesg samples add `197, 198, 200,
  201`.
* Second launch (`ok=2`, vadd): `164 … 176` across sampled logs; m145-r3
  shows two pairs (`198/171`, `197/174`).

**Canonical baseline (to quote):** `elapsed_ms=189` (launch 1) and `166`
(launch 2), run `m141-r3`, module `83d78872…`. Use the pair, not one number.

### 1.5 What the baseline does NOT say

* No guest-observed (`cudaEvent`) launch latency was recorded in r3.
* No copy bandwidth, no malloc latency, no init time, no variance (n=2
  launches is not a distribution).
* No host oracle was measured with the *same new binary* in the same boot as
  r3 (the `tools/cuda-ab/run/victory_mail` oracle exists, but not the perf
  workload).
* Module diagnostics (`sema_dbg`) were on in the campaign recipe, so the
  absolute numbers include tracing/log overhead. The perf protocol measures
  with the campaign's accepted recipe and says which parameters were set.

---

## 2. Measurement matrix

ID legend: **G** = in-guest instrument, **H** = host (kernel trace / counter /
dmesg), **X** = cross-check. All guest metrics come from
`perf_workload.cu` `PERF` lines; all host metrics from the vgsp surfaces in
§3. "Oracle" column is the same binary on bare metal.

| ID | Metric | Instrument | Baseline (r3) | Near-native target + how calibrated |
|---|---|---|---|---|
| P1 | Process start → device ready (`init_ms`) | **G** CLOCK_MONOTONIC in `main()` → `cudaGetDeviceProperties`+`cudaFree(0)` (`PERF init_ms`) | none recorded | ≤ **2.0×** host oracle same binary (docs 3.3). Calibrate: 3 host + 10 guest runs; if host is tiny, additionally cap at max(50 ms, 25× host). |
| P2 | `cudaMalloc` first + steady `p50/p95` | **G** wall clock around `cudaMalloc/cudaFree`, 50 steady iters (`PERF malloc_first_ms`, `malloc_ms`) | none | No-regression (≤1.2× prior artifact) in phase 1; working gate `p95 ≤ 5×` host p95 once calibrated. |
| P3 | First kernel launch, guest end-to-end | **G** `cudaEventRecord`→launch→sync (`PERF launch_first_ms`) | none (r3 has no CUDA timing) | Phase 1: record. Phase 2: `≤ 5×` host same-binary first launch, tightened toward 2× after the forwarding cost is attacked. |
| P3h | First launch, host backend service (the r3 metric) | **H** dmesg `GUEST GR LAUNCH OK … elapsed_ms=`; also `GR LAUNCH submitted`→OK delta | **189 ms** (mail), **166 ms** (vadd) | No-regression ≤ **1.2×** previous recorded p50/p95 for the same module+recipe; separately drive to ≤ `2×` host P3. |
| P4 | Warm launch latency `p50/p95` | **G** per-launch event+sync, N=100 (`PERF launch_warm`) | none | No-regression ≤1.2×; working gate `p50 ≤ 5×` host p50 (phase 2), tighten to 2× when feasible. |
| P5 | CE push latency (submit → release) | **H** kprobe `gsp_phys_backend_guest_ce_copy[_ex]` entry/return per forwarded copy (or `sema-optrace` with `sema_dbg=1`); **G** 4 KiB sync copy as the end-to-end proxy (`PERF copy dir=htod bytes=4096`) | P6b live runs recorded counters (`req=100 ok=100`), no latency | Report p50/p95; working gate: 4 KiB proxy `≤ 5×` host 4 KiB copy; no-regression. |
| P6 | MSI-X injection latency | **H** trace delta `gsp_rm_post_channel_event` → `gsp_emu_inject_completion_interrupt` (kprobes or `trace_pipe` timestamps) | none | p95 **≤ 1 ms** provisional (pure virtualization artifact); stability over absolute. Calibrate on first run. |
| P7 | HtoD bandwidth | **G** size sweep 4 KiB…16 MiB, best of 3 (`PERF copy dir=htod`) | none | **≥ 95 %** of host at ≥ 4 MiB (docs 3.5); small sizes report-only. |
| P8 | DtoH bandwidth | **G** same (`PERF copy dir=dtoh`) | none | **≥ 95 %** at ≥ 4 MiB. |
| P9 | Kernel execution vs bare-metal oracle | **G** bit-exact FMA chain, event-timed (`PERF kernel_fma ms/gflops`) | none | **≥ 95 %** host GFLOP/s (docs 3.5 `vectorAdd ≥95%`); absolute GFLOP/s also reported. |
| P10 | Steady-state throughput | **G** 100 launches enqueued, one sync (`PERF throughput kps`) | none | No-regression; working gate `≥ 20 %` of host kps (i.e. ≤5×), tightened as P3h/P4 improve. |
| P11 | Per-run variance (mean/median/p95) | **G/H** `collect.py` aggregate over R=10 process runs; report p95/median per metric | none | Warm unchanged stack: dispersion `p95/median ≤ 1.5` for every metric; above that, investigate before optimizing. |
| P12 | Kernel cycles (cross-check) | **X** `clock64()` start/end per block, 10 k FMA chain (`PERF clock64`) | none | Report only. Never treat cycles as time without a calibrated SM clock; never compare cycles across host/guest as a time metric. |

**Measurement-method classes (required by the plan):**

* **Host trace / kprobes** — P3h, P5, P6. Exact recipes in §3.2. No kernel
  change needed; module symbols are probeable at runtime. Arming kprobes is a
  supervised host action and does not modify the module or the campaign
  scripts.
* **In-guest wall clock** — P1, P2, P3, P4, P7–P11: monotonic clock and CUDA
  events inside the workload, exactly as `p2_bench` does.
* **`clock64()`** — P12 only, as a cross-check; per-SM counter, report as
  cycles.

---

## 3. Methods and exact commands

### 3.1 Harness files

| File | Purpose |
|---|---|
| `perf_workload.cu` | one binary, both roles; emits stable `PERF` lines; self-verifying (bad=0 required) |
| `perf.sh` | `check/build/host/guest/all/compare/push`; host oracle, guest push+run via `scripts/vgpu_vm.py`, host evidence capture, ratio compare |
| `collect.py` | stdlib-only parser/aggregator: `PERF` lines + dmesg + `gsp-phys-state` → `open-iov-perf/1` JSON; `--compare` ratio table |
| `results-template.json` / `results-template.md` | result schemas for the JSON artifact and the public summary |
| `bin/`, `results/` | generated (gitignored) |

Style follows `tools/cuda-ab/ab.sh`: same guest transfer (base64 chunks via
the guest agent), same static-CUDA-runtime build (`-O2 -arch=sm_75
--cudart=static`), same oracle/measured/compare split. `perf/` supersedes the
stub `tools/benchmark_infrastructure.sh` for CUDA metrics; the A/B pattern
stays as docs §3.5 requires.

### 3.2 Exact commands

```bash
cd ~/Git/integdev_gpu_drv/open-iov/tools/cuda-ab/perf

./perf.sh check                    # nvcc, VM reachability, host-GPU/state readability

# 1) Bare-metal oracle — run while the GPU is on the distribution driver
#    (before the campaign claims it, or after release):
./perf.sh host                     # -> results/<stamp>-host/results.json

# 2) Green campaign cycle; verify the green gate (victory_mail MAIL/VADD/STATUS OK),
#    then, with the VM up:
PERF_STAMP=<same-or-new-tag> PERF_RUNS=10 PERF_N_WARM=100 PERF_N_THRU=100 \
  ./perf.sh guest                  # -> results/<stamp>-guest/{results.json,compare...}

# 3) Compare (host is the oracle; every ratio is guest/host):
PERF_ORACLE=results/<stamp>-host/results.json ./perf.sh compare
# or standalone:
python3 collect.py --compare results/<h>/results.json results/<g>/results.json \
  --out results/<g>/compare.json [--thresholds thresholds.json]
```

Bring-up shortcut (faster, not for the record): `PERF_RUNS=3 PERF_N_WARM=20`.

What `guest` captures, besides stdout (all read-only; never `dmesg -C`):

| Artifact | Source |
|---|---|
| `host-state-before.txt` / `host-state-after.txt` | `cat /sys/kernel/debug/vgsp/gsp-phys-state` (sudo -n fallback) |
| `host-gr-dmesg.txt` | `dmesg \| grep -E 'vgsp-gr:\|work_gr:\|sema-gr:\|vgsp-sema: sema-svc:' \| tail -n 3000` |
| `dmesg-marker.txt` | last `dmesg` line before the run; `collect.py` drops host-GR entries at or before this raw monotonic timestamp, scoping the aggregate to this run |
| `meta.txt` | stamp, uname, `sha256sum kernel/vgsp.ko`, run config |
| `meta-guest.txt` | guest `nvidia-smi --query-gpu` + `nvidia-smi -q -d CLOCK` |
| `run-N.txt`, `run.txt` | per-run and merged workload stdout |

### 3.3 Optional host kprobes (P3h/P5/P6 distributions)

```bash
TR=/sys/kernel/tracing   # or /sys/kernel/debug/tracing
sudo bash -c "cd $TR
  echo 'p:fgr gsp_phys_backend_guest_gr_submit'   >> kprobe_events
  echo 'r:fgr_ret gsp_phys_backend_guest_gr_submit \$retval' >> kprobe_events
  echo 'p:fce gsp_phys_backend_guest_ce_copy'     >> kprobe_events
  echo 'r:fce_ret gsp_phys_backend_guest_ce_copy \$retval' >> kprobe_events
  echo 'p:ev gsp_rm_post_channel_event src=%di'   >> kprobe_events
  echo 'p:msix gsp_emu_inject_completion_interrupt seq=%si' >> kprobe_events
  echo 1 > events/kprobes/enable"
```

Read `trace` / `trace_pipe` around a workload window; entry↔return deltas give
the per-call distributions (P5/P6; P3h is already in dmesg). If a symbol is
not present in `/proc/kallsyms` (inlined), skip it — the dmesg + guest-event
path still yields P3h/P4.

### 3.4 The green gate (precondition, not optional)

A perf run is valid only if, in the same boot/VM:

1. `victory_mail` printed `MAIL …` + `VADD … bad=0` + `STATUS OK`;
2. `perf_workload` ended `PERF STATUS OK` with `bad=0` on **both** sides;
3. `gsp-phys-state` `work_gr: req == ok` after the run and
   `pending release starved` did not appear (ISSUE-0001 gate);
4. host and guest GPU clocks were captured (`nvidia-smi -q -d CLOCK`).

If any gate fails, the numbers are diagnostics, not results — label them so.

---

## 4. Thresholds and calibration procedure

Everything is calibrated on the first run; the method is fixed now:

1. **Oracle first.** Run `perf.sh host` while the GPU is unclaimed
   (pre-claim or post-release). Same machine, same boot if possible; if the
   oracle is from a different boot, record both stamps and treat cross-boot
   as extra noise.
2. **Formal guest run.** `PERF_RUNS=10 PERF_N_WARM=100`, one green cycle.
3. **Freeze thresholds** in `perf/thresholds.json` (created at recording
   time, owned by the lead):

   | Metric class | Rule (`thresholds.json` semantics) |
   |---|---|
   | `init_ms` | `{"mode":"max_ratio","limit":2.0}` (docs 3.3) |
   | `copy_*_gbps` ≥ 4 MiB, `kernel_fma_gflops` | `{"mode":"min_ratio","limit":0.95}` (docs 3.5) |
   | `launch_warm_p50`, `throughput_kps` | phase 1: no-regression only; phase 2 `max_ratio 5.0`, tighten to 2 — tighten only with a dated run showing it met |
   | `malloc_ms_p95`, small-copy proxy | `max_ratio 5.0` provisional |
   | host GR `elapsed_ms` (P3h) | no-regression vs previous artifact (`max_ratio 1.2`) |
   | MSI-X p95 | absolute cap 1 ms provisional |
   | dispersion (P11) | `p95/median ≤ 1.5` per metric |

4. **No-regression rule.** Until the launch path is optimized, the primary
   acceptance for P3h/P4/P10 is: a new measurement must not be worse than
   **1.2×** the previous recorded p50/p95 for the same module + recipe.
5. **Update discipline.** Thresholds change only with a fresh dated
   measurement and a note in the results document; never silently.

Report **guest/host ratios**, never absolute-only (docs §3.5). A threshold
miss is a finding, not a failure of the harness.

---

## 5. Harness walkthrough — the workload

`perf_workload.cu` (one binary, both roles):

* `PERF init_ms` — `main()` → device properties + `cudaFree(0)` (P1).
* `PERF malloc_first_ms`, `PERF malloc_ms n=50 mean/p50/p95` (P2).
* `PERF launch_first_ms` — first `pattern_kernel` launch, event → sync (P3).
* `PERF launch_warm n=… mean/p50/p95` — per-launch event+sync, default 50,
  protocol 100 (P4/P11).
* `PERF kernel_fma ms/gflops` + bit-exact host chain verification (P9).
* `PERF throughput n=… wall_ms kps` — 100-launch burst, one sync (P10).
* `PERF copy dir=htod|dtoh bytes=… ms gbps` — 4 KiB/256 KiB/4 MiB/16 MiB,
  best-of-3 after a warmup; 4 KiB round trip verified (P5/P7/P8).
* `PERF clock64 available/min/median` — per-block cycles, cross-check (P12).
* `PERF checksum … bad=0` + `PERF STATUS OK` — verification gate.

Host-side pairing: `collect.py` reads the same run count from
`gsp-phys-state` (`work_gr` counters, `work_relay`) before/after and parses
`GUEST GR LAUNCH OK … elapsed_ms=` plus `GR LAUNCH submitted` → OK deltas
from the dmesg capture (P3h). The capture is scoped by `dmesg-marker.txt`
(the last dmesg line before the run) so entries from earlier work in the same
boot cannot leak into the aggregate; raw monotonic timestamps are required
for that scoping, so capture `dmesg` without `-T` (the human `-T` form is
still accepted for the `elapsed_ms` values, just not for scoping).

---

## 6. Evidence discipline and recording

1. **No claim without a fresh measurement file.** Every performance statement
   cites `perf/results/<stamp>-*/results.json` (+ `compare.json`) produced by
   this harness after this plan, on a green stack. Fresh = same module sha256
   and recipe as the claim; re-used numbers are labelled as such.
2. **Never edit campaign scripts or evidence.** `perf/` writes only under
   `perf/results/` and `perf/bin/` (gitignored); the guest side only writes
   `/tmp/perf_workload*`.
3. **Durable copy.** The lead copies `results/<stamp>-{host,guest}/` to the
   running campaign's evidence directory (`backups/evidence-…/perf-<stamp>/`)
   as an immutable snapshot; raw results may then be pruned.
4. **Public summary (docs/).** Create
   `open-iov/docs/05-operations/05-performance-results.md` from
   `results-template.md`, and link it from
   `docs/05-operations/01-performance-correctness-security.md` §5 (reporting
   list) and the `docs/README.md` reading-order table. Tracked docs stay
   component/behaviour-only (no private paths, per AGENTS.md).
5. **Handoff §2 delta.** Append to `handoff-meta-plan.md` §2 (the live-state
   section; `Handoff-vGPU-IO.md` stays frozen), template:

   ```markdown
   ### 📈 SESSION <date> <time> — PERFORMANCE BASELINE (near-native program, run <stamp>)
   Green gate: victory_mail OK; perf_workload bad=0 both sides; work_gr req==ok.
   Module <sha256> recipe <exact insmod args> commit <rev>.
   Oracle: results/<stamp>-host (clocks …). Measured: results/<stamp>-guest.
   | metric | host | guest | g/h | threshold | verdict |
   |---|---|---|---|---|---|
   | P1 init_ms | … | … | … | ≤2.0× | … |
   | P3h GR elapsed_ms | n/a | … (prior 189/166) | — | ≤1.2× | … |
   | P4 launch_warm p50/p95 | … | … | … | ≤5× (p1) | … |
   | P7 HtoD 16MiB | … | … | … | ≥95% | … |
   | P9 fma GFLOP/s | … | … | … | ≥95% | … |
   (full table in docs/05-operations/05-performance-results.md; compare.json)
   Follow-ups: <limiting metric + next optimization candidate>.
   ```
6. **Clocks and configuration are part of the evidence.** Record host and
   guest `nvidia-smi -q -d CLOCK` digests; note `sema_dbg` and other
   diagnostic parameters from the recipe (they cost absolute time but
   preserve comparability run-to-run while unchanged).
7. **One physical writer.** The lead owns running `perf.sh` during a campaign;
   offline roles prepare and review. Benchmarks never run concurrently with
   another workload on the same GPU.

---

## 7. Limitations and risks

* **Host oracle availability sequencing.** During a green cycle the module
  owns the GPU, so `perf.sh host` cannot run then. The oracle must be captured
  pre-claim/post-release; record both stamps. `perf.sh all` warns and
  continues, but the named protocol is separate windows.
* **Jiffies quantization and tracing overhead.** P3h is 1 ms-quantized
  (`HZ=1000`) and includes `sema_dbg` logging when that parameter is on; the
  matrix states this rather than correcting for it.
* **Emulated VRAM.** Guest "VRAM" is host RAM today (docs §3.6 item 5): bulk
  copy bandwidth is not representative of real VRAM until the backing path
  exists — report, do not over-claim.
* **Cross-boot comparisons are noise** (docs §3.2); prefer same boot.
* **`clock64` is an SM cycle counter**; frequency may differ per SM and is
  not a time source without calibration.
* **Small N.** 10 process runs bound the p95 confidence; the protocol states
  N, and dispersion (P11) is itself a metric.
* **First-run effects.** `init_ms`/`malloc_first_ms` include one-time driver
  initialization; they are process-start metrics by design and are reported
  as such, not averaged into steady state.

## 8. Open items for the measurement session

1. Run `perf.sh check` before the first green cycle to confirm nvcc, VM
   executor, debugfs readability and the oracle window.
2. Capture the host oracle **before** the next claim if the GPU is free; if
   not, schedule one host-oracle window.
3. After the first formal run: freeze `perf/thresholds.json`, write
   `docs/05-operations/05-performance-results.md`, append the §2 delta, and
   list the limiting metric as the next optimization candidate (expected
   first candidate from §1.3: the serialized per-window `PDE_UPDATE` forwards
   inside `gsp_phys_backend_guest_gr_submit`).
4. Fold the §2 protocol step 6 pointer to this file when the handoff is next
   updated by the lead.