vgsp-dual: DUAL-REPORT.md - per-item status, one-mdev equivalence, risks, live plandual

1 files changed, 360 insertions, 0 deletions

diff --git a/DUAL-REPORT.md b/DUAL-REPORT.md
new file mode 100644
index 0000000..334a772
--- /dev/null
+++ b/DUAL-REPORT.md
@@ -0,0 +1,360 @@
+# DUAL-REPORT.md — Stage B+C (minimal) per-mdev separation for dual-guest CUDA
+
+**Date:** 2026-10-15
+**Worktree:** `/home/rmcore/Git/integdev_gpu_drv/open-iov-dual` (branch `dual`, base `ecfb1278`)
+**Commits:** `81c11ca0` (stage B+C separation), `5769d5c4` (CUDA-bridge owner gating)
+**Build:** `cd kernel && make -j8` — clean success, `vgsp.ko` linked. No module
+was loaded, no device touched, no guest run (offline implementation only).
+
+Implementation of stages B+C (minimal) of
+`analysis-2026-10-15/AGENT-dual-guest-inventory.md` §3.1/§3.2 (items G1–G8),
+against the user scope: G1, G2, G3, G5-minimal, G6, G7, G8(a).
+Explicitly **not** implemented: G4 remainder (per-guest learned tables),
+G10 per-mdev debugfs views, parallel forwarding, boot_stage split.
+
+---
+
+## 1. Per-item status table
+
+| # | Item | Status | Change site (file:line @ `5769d5c4`) | Verified by |
+|---|------|--------|--------------------------------------|-------------|
+| G1 | Sema-service thread services ALL open mdevs each pass | DONE | `kernel/gsp_sema_service.c:16034` (`svc_thread`): collects every open mdev into `vgpus[]` (:16060–:16075) and runs the pass body per guest; `svc_handle_map_add()` drains once per pass; the scans/hint loop is per guest (:16174–:16226). Per-guest rebuild freshness (`sess->last_build`, :16105–:16119). | build only (`make -j8`) |
+| G2 | Channel table keyed by `(vgpu, h_object)`; per-guest rank; `vgpu` field in `svc_channel` | DONE | struct field `kernel/gsp_sema_service.c:165`; match in `channel_add` :1154 + `same` :1168; rank loops :1204–:1221; `channel_del` :1328; pump snapshot :13653; pump writeback :15201; counter paths :5198/:5271/:5360/:5551/:5600/:5626/:5686; `svc_pb_discover` :7821; `svc_gr_content_discover` :9103; `svc_gr_ring_autopin` :9445; chan dump :15516. Registration carries the owner: `kernel/gsp_rm_emulation.c:2360`; free passes it: :3248; desc field `kernel/gsp_sema_service.h:30`. | build only |
+| G3 | Per-vgpu session active/flush flags; departing guest never flushes a live guest | DONE (staged semantics — see §3.1) | session records + helpers `kernel/gsp_sema_service.c:468–:520` (struct :473, `svc_guest_get` :489); `channel_add` marks active :1242; `channel_del` deactivates per guest and requests the flush only when **no** guest has a live channel :1356–:1378; `gsp_sema_service_vgpu_cleanup` :16364 (session end on mdev remove). | build only |
+| G4 (remainder) | Per-guest learned tables (map/pin/vas/pending/…) | NOT IN SCOPE (documented risk §5.1) | — | — |
+| G5 (minimal) | Forward request/hold carry the owning vgpu; single global in-flight kept | DONE | req already carried `vgpu` (verified, set at :12972 GR `svc_gr_req_fill`, :13304 CE); hold abort keyed `(vgpu, handle)` :11091; dead-owner guards on both resolution paths :11928 (`svc_vgpu_open`), :11956 (`gone_finish`), :12019 (`stop_flush`); operand translation through the request's vgpu :10922. Follow-on: CUDA-bridge fallback queue now carries its owner (`kernel/gsp_cuda_bridge.c:143`, :624, :985, :1353). | build only |
+| G6 | Per-mdev guest-RAM backing {file, path, ranges} + per-mdev selector | DONE | records `kernel/gsp_phys_gpu.c:1264–:1373`; `gsp_phys_guest_ram_configure_vgpu` :1494; `gsp_phys_guest_ram_vgpu_clear` :1586; clear-all :1564; active :1604; state (:1646, legacy format byte-identical); debugfs writer `kernel/gsp_phys_booter.c:20487` (4-token legacy or 5-token `<uuid>` selector); selector resolve :20542 via `gsp_phys_open_mdev_by_name` (`gsp_phys_gpu.c:1356`). | build only |
+| G7 | `gpa_to_hpa` takes the requesting vgpu; all call sites plumbed | DONE | `gsp_phys_gpa_to_hpa_vgpu` `kernel/gsp_phys_gpu.c:1737`; wrapper `gsp_phys_gpa_to_hpa` :1827. Call sites: `gsp_sema_service.c:10922` (forwarding operands — passes the request's vgpu); `gsp_phys_booter.c:9738`, :19125, :19212 (single-guest guest-GPA pin — explicit `NULL` = first open mdev, see §3.2); no others remain (`grep`). | build only |
+| G8(a) | Guest-GPA pin explicitly single-guest; refusal + log | DONE | `kernel/gsp_phys_booter.c:20430–:20436`: pin refused `-EBUSY` with a log when `gsp_phys_open_mdev_count() > 1` (`gsp_phys_gpu.c:1333`); existing different-GPA refusal kept. Boot-ladder trigger call sites documented as first-open. | build only |
+| G10 | Per-mdev debugfs views | NOT IN SCOPE | — | — |
+
+Diff stats (`git diff --stat ecfb1278..HEAD`):
+
+```
+ kernel/gsp_cuda_bridge.c  |  92 ++++++-
+ kernel/gsp_phys_booter.c  | 115 +++++++--
+ kernel/gsp_phys_gpu.c     | 432 ++++++++++++++++++++++++++-----
+ kernel/gsp_phys_gpu.h     |  35 ++-
+ kernel/gsp_rm_emulation.c |   3 +-
+ kernel/gsp_sema_service.c | 644 +++++++++++++++++++++++++++++++++-----------
+ kernel/gsp_sema_service.h |  20 +-
+ kernel/main.c             |  10 +
+ 8 files changed, 1098 insertions(+), 253 deletions(-)
+```
+
+### 1.1 One-mdev equivalence argument (explicit fast paths)
+
+With exactly ONE open mdev every changed path degenerates to the old one:
+
+1. **Thread (G1):** `vgpus[]` has one entry; the per-guest loop body is the
+   historical pass in the historical order (probe → walk_probe → chan_dump →
+   scrub → rebuild → pump → bridge → pending → sweep, then the scan loop).
+   `svc_handle_map_add()` runs once per pass (`gsp_sema_service.c:16085`),
+   exactly as before.
+2. **Channels (G2):** `vgpu` is constant, so `(vgpu, h_object)` matches the
+   same slots as `h_object` alone; rank loops count the same set; the pump
+   snapshot/writeback select the same channels.
+3. **Session/flush (G3):** the single guest's last-channel close empties the
+   global table → `svc_sess_active = 0; svc_sess_flush = 1` at the same
+   transition as before.
+4. **Guest RAM (G6/G7):** a 4-token `gsp-phys-guest-ram` write configures the
+   legacy record with byte-identical log lines; `gpa_to_hpa_vgpu(vgpu…)`
+   resolves the same first-open mdev and the same legacy record; the
+   `gsp-phys-state` `guest_ram:` field renders the byte-identical
+   `active=1 path=… ranges=… gpa=…` text (and `active=0` when unconfigured).
+5. **Forwarding (G5):** `svc_fwd_req`/`svc_fwd_hold` set and read the same
+   vgpu the single pump always used; the dead-owner guards never trigger while
+   the guest is open; the CUDA bridge's `head_owner == vgpu` is always true.
+6. **Log lines relied on by the campaign** are unchanged: `sema-svc:`,
+   `vgsp-gr:`, `GR LAUNCH submitted`, `GUEST GR LAUNCH OK`, `REAL GR SEM OK`,
+   `backend start: joined the live module-owned claim (READY)`, `guest RAM
+   configured:` / `guest RAM released:`. Debugfs file names are unchanged
+   (the selector is an extra token on the same file). Parameter defaults are
+   untouched.
+
+---
+
+## 2. What changed, by item
+
+### G6 — per-mdev guest-RAM backing (`gsp_phys_gpu.c`, `gsp_phys_booter.c`)
+
+* `guest_ram_file`/`guest_ram_path`/`guest_ram_ranges` globals replaced by
+  `guest_ram_legacy` (slot for the historical top-level write) + up to 4
+  `guest_ram_mdevs[]` records (vgpu-pointer keyed), all under the existing
+  `guest_ram_lock`.
+* New API: `gsp_phys_guest_ram_configure_vgpu()`,
+  `gsp_phys_guest_ram_vgpu_clear()`, `gsp_phys_open_mdev_count()`,
+  `gsp_phys_open_mdev_by_name()` (exports + prototypes in `gsp_phys_gpu.h`).
+* debugfs `gsp-phys-guest-ram` accepts both
+  `"<path> <gpa_base> <off> <size>"` (legacy: first/only mdev) and
+  `"<mdev-uuid> <path> <gpa_base> <off> <size>"` (per-mdev). Unknown selector
+  → `-ENODEV` with a log; `"-"` clears every record. Token count (4 vs 5)
+  discriminates; paths/UUIDs contain no spaces.
+* mdev removal releases the record (`main.c:1750`
+  `gsp_phys_guest_ram_vgpu_clear(vgpu)`); physical release clears all records
+  (`gsp_phys_guest_ram_clear()`).
+
+### G7 — vgpu-aware GPA→HPA (`gsp_phys_gpu.c` + call sites)
+
+* `gsp_phys_gpa_to_hpa_vgpu(vgpu, gpa, *hpa)`:
+  * `vgpu == NULL` → first open mdev (historical);
+  * named vgpu whose session is closed → `-ENODEV` (never retargeted);
+  * translation uses the vgpu's own record, else the legacy record when the
+    vgpu is the first open mdev.
+* `gsp_phys_gpa_to_hpa()` kept as a thin `NULL` wrapper.
+* Forwarding operands (`svc_fwd_operand_hpa`) pass the request's vgpu; the
+  boot-ladder guest-GPA pin paths pass `NULL` on purpose (§3.2).
+
+### G1 — sema-service thread (`gsp_sema_service.c`)
+
+* Per pass: bounded list walk collects all open mdevs (≤ `SVC_MAX_GUESTS`=8,
+  the driver's per-type ceiling) and runs the full per-guest body for each.
+* Per-guest `svc_rebuild` freshness so the first guest's 10 s tick cannot
+  starve the second guest of rebuilds.
+* `pumped` is an aggregate: while any guest is progressing, the pass loops
+  tightly and skips the heavy scans, as before.
+
+### G2 — channel identity (`gsp_sema_service.c`, `gsp_rm_emulation.c`)
+
+* `svc_channel.vgpu`; add/del match `(vgpu, h_object)`; `same` (re-register)
+  requires the same owner; ranks count only same-owner channels; counter,
+  scrubber-adjacent, opdef-drop, push-buffer discovery, GR content discovery,
+  GR ring autopin max-CE scan, pump snapshot and pump write-back are all
+  filtered by the owning vgpu.
+* Scrubber channel state (`svc_scrub` → `svc_scrubs[]`) made per guest: the
+  authored finish/USERD words are written through the owning guest's address
+  space, never through whichever guest opened first.
+
+### G3 — sessions (`gsp_sema_service.c`)
+
+* `svc_guests[]` records: `active` + `last_build`, pointer-keyed, freed on
+  mdev cleanup. `channel_add` marks active; `channel_del` deactivates the
+  owner when its last channel closes and requests the global flush only when
+  **no guest** has a live channel (staged semantics, §3.1).
+* New `gsp_sema_service_vgpu_cleanup(vgpu)` (called from
+  `main.c:1749`): quiesces forwarding, drops the guest's hold (pure clear —
+  never a resolution through a vanishing device), removes its channels,
+  drops its opdef release slots, frees its session and scrubber records.
+
+### G5 (minimal) — forwarding identity (`gsp_sema_service.c`, `gsp_cuda_bridge.c`)
+
+* The single-inflight request keeps ONE global slot (serialization kept) but
+  carries its owner: CE requests set `svc_fwd_req.vgpu` (:13304), GR requests
+  set it in `svc_gr_req_fill` (:12972), and the operand resolver translates
+  through it.
+* Hold abort keyed by `(vgpu, handle)`; both hold resolution paths
+  (`svc_fwd_hold_gone_finish`, `svc_fwd_hold_stop_flush`) and the pump's
+  cross-vgpu resolution check `svc_vgpu_open()` before ever dereferencing the
+  stored device and drop the hold when its owner is gone.
+* CUDA bridge (the GR fallback under the single-inflight slot): request and
+  pending records now carry the owning mdev, the owner is part of the dedup
+  identity, only the owner's pass marshals/finishes, and stale-owner records
+  are dropped. This was not itemised in the inventory, but without it the
+  expected dual steady state (guest B's forward refused while A's hold is
+  active) would marshal guest B's fallback job through guest A.
+
+### G8(a) — guest-GPA pin
+
+* Refused while more than one mdev is open, with an explicit log; the pin
+  remains a single-guest test aid translating through the first open mdev.
+
+---
+
+## 3. Deviations from the inventory plan (deliberate, minimal)
+
+### 3.1 G3 flush is deferred, not per-guest-selective
+
+The inventory asks to "flush only the departing vgpu's learned state". The
+learned tables (`svc.map`, `svc.pin`, `svc.ipin`, `svc.lat`, `svc.vas*`,
+`svc_pool_pa`, `svc_pending`, `svc_retry_q`, `svc_pb[]`, rank bitmaps) are
+still shared — that is the explicitly out-of-scope G4 remainder. With shared
+tables a selective flush is impossible; the only safe minimal semantic is:
+**deactivate the departing guest's session and defer the flush while any other
+guest still has live channels.** One guest: identical to today (last-channel
+close is the global empty transition). Consequence for T5: guest B's
+close→reopen while A is live does not wipe B's stale learned entries (they
+must survive for A), so B's *second* session may hit the ISSUE-0002
+contamination class for its own VAs. Closing that hole is the G4 minimum
+(per-entry owner tags + per-guest lookup filter).
+
+### 3.2 Boot-ladder guest-GPA call sites pass `NULL`
+
+`gsp_phys_rmcall_pdb_build` (booter :9738) and triggers 6/7 (:19125, :19212)
+have no request context; the only user of that GPA is the single-guest
+test-only pin, which is now refused while >1 mdev is open (G8a). `NULL` is
+the documented first-open resolution — identical behaviour for every
+single-guest run, and explicitly not per-mdev.
+
+### 3.3 Debugfs selector: extra token, not a subdir/per-uuid file
+
+The hard requirement says "same debugfs file names". The recommended subdir
+per UUID / `gsp-phys-guest-ram-<uuid>` file would add files; the optional
+first-token UUID keeps the tree and the historical 4-token write untouched
+while giving per-mdev targeting. The per-mdev form is additive: 5 tokens.
+
+### 3.4 Scrubber registration binds to the first open mdev
+
+`sema_scrubbuf_add` (debug tooling written by the guest scanner) has no
+selector; it binds to the first open mdev's scrubber record (single-guest
+convention, like the scanner itself) and is refused with no open mdev.
+
+### 3.5 CUDA bridge owner gating (follow-on commit)
+
+See §2/G5. Default-on (`cuda_bridge=1`); the change is inert for one guest.
+
+---
+
+## 4. Self-check: module-global session/pump state audit
+
+Every read of module-global session/pump state after this change, classified:
+
+| # | Global state | Sites | Status |
+|---|--------------|-------|--------|
+| 1 | channel table `svc.chan[]` | add :1154, del :1338, ranks :1204–:1221, counters :5198/:5271/:5360/:5551/:5600/:5626/:5686, pb says :7821, gr-content :9103, gr autopin :9445, pump :13653/:15201, chan dump :15516 | **fixed** per `(vgpu, h)` |
+| 2 | session flags `svc_sess_active/flush` | :468–:520, :1356–:1378, :16364 | **fixed** (per-guest records drive the global flush) |
+| 3 | scrubber `svc_scrub` | :535–:580, :1272–:1295, :15973, :15912 | **fixed** per guest |
+| 4 | hold `svc_fwd_hold` | :11010–:12100 (:11956/:12019 guards) | **fixed**: vgpu-keyed abort + dead-owner guards |
+| 5 | request `svc_fwd_req` | :10648, :12972/:13304, :13140–:13170 | **fixed**: owner carried; single-inflight kept |
+| 6 | bridge req/pend | `gsp_cuda_bridge.c` (:143, :624, :985, :1353) | **fixed**: owner carried, owner-only marshal/finish |
+| 7 | opdef slots | :11185–:11500 (drop_gone :11468) | pre-existing per `(vgpu,h)`; `drop_gone` filter **fixed** |
+| 8 | learned map `svc.map` / `svc.map_n` / `svc.map_hash` | lookup :3300+, rebuild :3366 | **shared (G4)** — risk §5.1 |
+| 9 | pins `svc.pin[]/ipin[]/svc_ipin_alt` | :2023–:2235, flush :7176 | **shared (G4)** — risk §5.1 |
+| 10 | `svc.lat[]`, `svc.vas[]`, VA shadow, `svc_hot` | :1818+, :1878+, :2011 | **shared (G4)** — risk §5.1 |
+| 11 | `svc_pending[]`, `svc_retry_q[]`, `svc_pool_pa*`, `svc_cnt_der` | :7081+, :885+, :2586+, :5335 | **shared (G4)** — risk §5.1 (pending is the top cross-guest write vector) |
+| 12 | rank latches `svc_ring_done`, `svc_pb_done`, `svc_gr_ring_done`, `svc_pb[]`, gr-content state | :355, :341–:347, :501–:511 | **shared latches (G4)** — re-arm is content-validated/idempotent; extra re-scans possible |
+| 13 | `svc_userd_sweep_at/n` (ce_index) | :13782+ | **shared (G4)** — equal ranks across guests share sweep slots |
+| 14 | per-pass capture `svc_fwd_cand`, `svc_gr_cand`, `svc_fwd_rel` | :3896+, :2587+, :7233 | serially reused by the single service thread; safe with one in-flight, listed for completeness |
+| 15 | debug surfaces reading the first open mdev | `add_map` rewind :15693, `dump_vidmem` :15783, `find_vidmem` :15819, `counter_add` :15868, scrubbuf :15912 | **debug tooling, single-guest by design** — do not use in dual runs |
+| 16 | `gsp_phys_gpa_to_hpa(NULL,…)` (boot-ladder pin) | booter :9738/:19125/:19212 | **by design** first-open (§3.2, G8a refuses dual) |
+| 17 | last-build freshness `svc.last_build` | :3570, :16105–:16114 | **fixed**: per-guest gate; `svc.last_build` kept as the published stamp |
+| 18 | bridge debugfs FIFO views (job/in/out) | `gsp_cuda_bridge.c` | one global daemon FIFO; single-guest tooling surface |
+
+No call site that carries a requesting/owning vgpu still matches session state
+by handle alone in the paths in scope.
+
+---
+
+## 5. Risks (ranked)
+
+1. **Shared learned tables (G4 remainder) — highest.** Cross-guest resolution
+   is still possible: a lookup for guest B's VA can hit guest A's pin/map
+   entry with the same VA; `svc_pending[]`/`svc_retry_q[]` entries carry no
+   owner, so a queued release decoded for B can be applied through A (or
+   vice-versa). Expected symptoms under T2/T4: wrong results, stalls,
+   `rel-apply` writes to the wrong guest. This is the next planned change
+   (G4 minimum: owner tag on pin/map/lat/pending + per-guest lookup filter).
+2. **mdev-removal lifetime race.** The pump can be mid-pass for a vgpu whose
+   fd just closed; the vgpu struct is freed after `vfio_put_device()` in the
+   remove path. Pre-existing class (single-guest too), reduced by
+   `gsp_sema_service_vgpu_cleanup` (channels/hold/records cleared before the
+   free), not fully closed (the pump pass itself is not synchronised).
+3. **G3 staged semantics (T5).** Guest B reopen while A is live inherits B's
+   previous learned entries (see §3.1). A's next run is protected (no flush
+   while A is live) — that is the T5 guard — but B's own reopen may hit the
+   ISSUE-0002 wedge until G4-minimum.
+4. **Shared rank latches/pairing (`svc_pb[]`, done bitmaps, sweep arrays).**
+   Content validation makes re-arms idempotent, but equal ranks across guests
+   can pair the wrong push-buffer base records and share USERD sweep slots;
+   possible slowdowns/mis-pairs under concurrent CE load.
+5. **Legacy guest-RAM write semantics.** With two guests, the 4-token write
+   configures the legacy record served by the first open mdev; a script that
+   writes it expecting "the new guest" will bind guest A. Use the 5-token
+   UUID form for the second guest.
+6. **Reducer scope:** no boot_stage split, no per-mdev debugfs views, no
+   parallel forwarding — per plan. The single physical GR channel stays safe
+   under `gsp_phys_rm_lock` (inventory §4), serialized at launch granularity.
+
+---
+
+## 6. Live-validation plan (offline work ended here)
+
+All of the following is for the LEAD under the supervised main-job rules
+(no unattended cycles; stock guests; evidence per run). The module was NOT
+loaded by this task.
+
+### 6.0 Build/load (from the dual worktree)
+
+```bash
+cd ~/Git/integdev_gpu_drv/open-iov-dual/kernel && make -j8
+# note: modpost needs
+#   ~/Git/integdev_gpu_drv/open-iov-dual/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers
+# (gitignored; copied into this worktree from the master tree — keep it present
+#  or the final link fails with "No such file or directory")
+```
+
+Load (after a clean/unloaded host; parameters per inventory §5):
+
+```bash
+sudo insmod ~/Git/integdev_gpu_drv/open-iov-dual/kernel/vgsp.ko \
+  gsp_phys_gpu=1 gsp_phys_mdev_owned=1 gsp_phys_guest_mem=1 \
+  gsp_phys_work_forward=1 gsp_phys_gr_forward=1
+```
+
+### 6.1 Setup — two turing mdevs, two QEMUs
+
+```bash
+GPU=0000:01:00.0   # substitute the campaign's GPU BDF (10de:…)
+T=nvidia-nvidia-vgpu-turing
+UUID_A=$(uuidgen); UUID_B=$(uuidgen)
+echo "$UUID_A" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create
+echo "$UUID_B" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create
+# available_instances must read 2 - 2 = 0 afterwards
+```
+
+Launch QEMU A and B, each bound to its own mdev
+(`-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID_A` resp. `$UUID_B`),
+each with `memory-backend-file,share=yes` pointing at a **different** file
+(e.g. `/dev/shm/vgpu-ram-a`, `/dev/shm/vgpu-ram-b`).
+
+Register each guest's RAM window per mdev (5-token selector form):
+
+```bash
+echo "$UUID_A /dev/shm/vgpu-ram-a 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram
+echo "$UUID_B /dev/shm/vgpu-ram-b 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram
+# guest A may equally be configured by the legacy campaign scripts (4-token form);
+# it serves the first open mdev. A second write of a different path to the same
+# record is refused -EBUSY until that mdev is removed.
+```
+
+In each guest: stock 570.144 driver + `tools/cuda-ab/run/vecadd_loop` (or the
+guest-side build from `tools/cuda-ab/`).
+
+### 6.2 Test matrix (inventory §5, T1–T5)
+
+| Test | Action | Pass criteria |
+|------|--------|---------------|
+| T1 | `nvidia-smi -L` in both guests simultaneously | no host error/stall (`dmesg -T | grep -i vgsp`) |
+| T2 | start `./vecadd_loop` in A and B within 1 s | each prints `LOOP launches=100 bad=0`, exit 0 within 600 s |
+| T3 | A vecadd, B started 10 s later | both green |
+| T4 | `smiley` in A + `vecadd_loop` in B (CE vs GR cross) | both green |
+| T5 | stop guest B → its mdev removed → recreate B (+ its RAM record) → rerun A and B | A green across B's cycle (B itself: see risk §5.3) |
+
+Evidence capture: per-guest client logs + host dmesg to
+`backups/evidence-*/dual-guest-<run>/`; require ≥200 lines each of
+`GUEST GR LAUNCH OK` and zero `fwd_gr_fault`/timeouts in the steady state.
+Per-mdev attribution in the log: `guest RAM configured for mdev <uuid>`,
+`channel 0x… registered` around each guest's `GR LAUNCH submitted` stream.
+
+Useful live checks:
+
+```bash
+dmesg -T | grep -E "sema-svc|vgsp-gr|GUEST GR LAUNCH OK|cuda-bridge" | tail -200
+cat /sys/kernel/debug/vgsp/gsp-phys-state | grep -E "guest_ram|work_guest|backend"
+sudo cat /sys/kernel/debug/vgsp/gsp-phys-state | grep guest_ram   # both mdev records visible
+```
+
+Baseline (pre-fix) for the regression control: T2 with the old module should
+show the second guest stalling or wrong results (inventory §5).
+
+### 6.3 Rollback
+
+Worktree-local: `git checkout ecfb1278 -- kernel/ && make -j8` in
+`open-iov-dual`; the master worktree is untouched.
+
+---
+
+## 7. Artifacts
+
+* Branch `dual` commits `81c11ca0`, `5769d5c4` (local only, not pushed).
+* `kernel/vgsp.ko` built at `5769d5c4` (offline; never loaded).
+* This report. No sessions/dumps generated (no device access).