path: root/DUAL-REPORT.md

# DUAL-REPORT.md — Stage B+C (minimal) per-mdev separation for dual-guest CUDA

**Date:** 2026-10-15
**Worktree:** `/home/rmcore/Git/integdev_gpu_drv/open-iov-dual` (branch `dual`, base `ecfb1278`)
**Commits:** `81c11ca0` (stage B+C separation), `5769d5c4` (CUDA-bridge owner gating)
**Build:** `cd kernel && make -j8` — clean success, `vgsp.ko` linked. No module
was loaded, no device touched, no guest run (offline implementation only).

Implementation of stages B+C (minimal) of
`analysis-2026-10-15/AGENT-dual-guest-inventory.md` §3.1/§3.2 (items G1–G8),
against the user scope: G1, G2, G3, G5-minimal, G6, G7, G8(a).
Explicitly **not** implemented: G4 remainder (per-guest learned tables),
G10 per-mdev debugfs views, parallel forwarding, boot_stage split.

---

## 1. Per-item status table

| # | Item | Status | Change site (file:line @ `5769d5c4`) | Verified by |
|---|------|--------|--------------------------------------|-------------|
| G1 | Sema-service thread services ALL open mdevs each pass | DONE | `kernel/gsp_sema_service.c:16034` (`svc_thread`): collects every open mdev into `vgpus[]` (:16060–:16075) and runs the pass body per guest; `svc_handle_map_add()` drains once per pass; the scans/hint loop is per guest (:16174–:16226). Per-guest rebuild freshness (`sess->last_build`, :16105–:16119). | build only (`make -j8`) |
| G2 | Channel table keyed by `(vgpu, h_object)`; per-guest rank; `vgpu` field in `svc_channel` | DONE | struct field `kernel/gsp_sema_service.c:165`; match in `channel_add` :1154 + `same` :1168; rank loops :1204–:1221; `channel_del` :1328; pump snapshot :13653; pump writeback :15201; counter paths :5198/:5271/:5360/:5551/:5600/:5626/:5686; `svc_pb_discover` :7821; `svc_gr_content_discover` :9103; `svc_gr_ring_autopin` :9445; chan dump :15516. Registration carries the owner: `kernel/gsp_rm_emulation.c:2360`; free passes it: :3248; desc field `kernel/gsp_sema_service.h:30`. | build only |
| G3 | Per-vgpu session active/flush flags; departing guest never flushes a live guest | DONE (staged semantics — see §3.1) | session records + helpers `kernel/gsp_sema_service.c:468–:520` (struct :473, `svc_guest_get` :489); `channel_add` marks active :1242; `channel_del` deactivates per guest and requests the flush only when **no** guest has a live channel :1356–:1378; `gsp_sema_service_vgpu_cleanup` :16364 (session end on mdev remove). | build only |
| G4 (remainder) | Per-guest learned tables (map/pin/vas/pending/…) | NOT IN SCOPE (documented risk §5.1) | — | — |
| G5 (minimal) | Forward request/hold carry the owning vgpu; single global in-flight kept | DONE | req already carried `vgpu` (verified, set at :12972 GR `svc_gr_req_fill`, :13304 CE); hold abort keyed `(vgpu, handle)` :11091; dead-owner guards on both resolution paths :11928 (`svc_vgpu_open`), :11956 (`gone_finish`), :12019 (`stop_flush`); operand translation through the request's vgpu :10922. Follow-on: CUDA-bridge fallback queue now carries its owner (`kernel/gsp_cuda_bridge.c:143`, :624, :985, :1353). | build only |
| G6 | Per-mdev guest-RAM backing {file, path, ranges} + per-mdev selector | DONE | records `kernel/gsp_phys_gpu.c:1264–:1373`; `gsp_phys_guest_ram_configure_vgpu` :1494; `gsp_phys_guest_ram_vgpu_clear` :1586; clear-all :1564; active :1604; state (:1646, legacy format byte-identical); debugfs writer `kernel/gsp_phys_booter.c:20487` (4-token legacy or 5-token `<uuid>` selector); selector resolve :20542 via `gsp_phys_open_mdev_by_name` (`gsp_phys_gpu.c:1356`). | build only |
| G7 | `gpa_to_hpa` takes the requesting vgpu; all call sites plumbed | DONE | `gsp_phys_gpa_to_hpa_vgpu` `kernel/gsp_phys_gpu.c:1737`; wrapper `gsp_phys_gpa_to_hpa` :1827. Call sites: `gsp_sema_service.c:10922` (forwarding operands — passes the request's vgpu); `gsp_phys_booter.c:9738`, :19125, :19212 (single-guest guest-GPA pin — explicit `NULL` = first open mdev, see §3.2); no others remain (`grep`). | build only |
| G8(a) | Guest-GPA pin explicitly single-guest; refusal + log | DONE | `kernel/gsp_phys_booter.c:20430–:20436`: pin refused `-EBUSY` with a log when `gsp_phys_open_mdev_count() > 1` (`gsp_phys_gpu.c:1333`); existing different-GPA refusal kept. Boot-ladder trigger call sites documented as first-open. | build only |
| G10 | Per-mdev debugfs views | NOT IN SCOPE | — | — |

Diff stats (`git diff --stat ecfb1278..HEAD`):

```
 kernel/gsp_cuda_bridge.c  |  92 ++++++-
 kernel/gsp_phys_booter.c  | 115 +++++++--
 kernel/gsp_phys_gpu.c     | 432 ++++++++++++++++++++++++++-----
 kernel/gsp_phys_gpu.h     |  35 ++-
 kernel/gsp_rm_emulation.c |   3 +-
 kernel/gsp_sema_service.c | 644 +++++++++++++++++++++++++++++++++-----------
 kernel/gsp_sema_service.h |  20 +-
 kernel/main.c             |  10 +
 8 files changed, 1098 insertions(+), 253 deletions(-)
```

### 1.1 One-mdev equivalence argument (explicit fast paths)

With exactly ONE open mdev every changed path degenerates to the old one:

1. **Thread (G1):** `vgpus[]` has one entry; the per-guest loop body is the
   historical pass in the historical order (probe → walk_probe → chan_dump →
   scrub → rebuild → pump → bridge → pending → sweep, then the scan loop).
   `svc_handle_map_add()` runs once per pass (`gsp_sema_service.c:16085`),
   exactly as before.
2. **Channels (G2):** `vgpu` is constant, so `(vgpu, h_object)` matches the
   same slots as `h_object` alone; rank loops count the same set; the pump
   snapshot/writeback select the same channels.
3. **Session/flush (G3):** the single guest's last-channel close empties the
   global table → `svc_sess_active = 0; svc_sess_flush = 1` at the same
   transition as before.
4. **Guest RAM (G6/G7):** a 4-token `gsp-phys-guest-ram` write configures the
   legacy record with byte-identical log lines; `gpa_to_hpa_vgpu(vgpu…)`
   resolves the same first-open mdev and the same legacy record; the
   `gsp-phys-state` `guest_ram:` field renders the byte-identical
   `active=1 path=… ranges=… gpa=…` text (and `active=0` when unconfigured).
5. **Forwarding (G5):** `svc_fwd_req`/`svc_fwd_hold` set and read the same
   vgpu the single pump always used; the dead-owner guards never trigger while
   the guest is open; the CUDA bridge's `head_owner == vgpu` is always true.
6. **Log lines relied on by the campaign** are unchanged: `sema-svc:`,
   `vgsp-gr:`, `GR LAUNCH submitted`, `GUEST GR LAUNCH OK`, `REAL GR SEM OK`,
   `backend start: joined the live module-owned claim (READY)`, `guest RAM
   configured:` / `guest RAM released:`. Debugfs file names are unchanged
   (the selector is an extra token on the same file). Parameter defaults are
   untouched.

---

## 2. What changed, by item

### G6 — per-mdev guest-RAM backing (`gsp_phys_gpu.c`, `gsp_phys_booter.c`)

* `guest_ram_file`/`guest_ram_path`/`guest_ram_ranges` globals replaced by
  `guest_ram_legacy` (slot for the historical top-level write) + up to 4
  `guest_ram_mdevs[]` records (vgpu-pointer keyed), all under the existing
  `guest_ram_lock`.
* New API: `gsp_phys_guest_ram_configure_vgpu()`,
  `gsp_phys_guest_ram_vgpu_clear()`, `gsp_phys_open_mdev_count()`,
  `gsp_phys_open_mdev_by_name()` (exports + prototypes in `gsp_phys_gpu.h`).
* debugfs `gsp-phys-guest-ram` accepts both
  `"<path> <gpa_base> <off> <size>"` (legacy: first/only mdev) and
  `"<mdev-uuid> <path> <gpa_base> <off> <size>"` (per-mdev). Unknown selector
  → `-ENODEV` with a log; `"-"` clears every record. Token count (4 vs 5)
  discriminates; paths/UUIDs contain no spaces.
* mdev removal releases the record (`main.c:1750`
  `gsp_phys_guest_ram_vgpu_clear(vgpu)`); physical release clears all records
  (`gsp_phys_guest_ram_clear()`).

### G7 — vgpu-aware GPA→HPA (`gsp_phys_gpu.c` + call sites)

* `gsp_phys_gpa_to_hpa_vgpu(vgpu, gpa, *hpa)`:
  * `vgpu == NULL` → first open mdev (historical);
  * named vgpu whose session is closed → `-ENODEV` (never retargeted);
  * translation uses the vgpu's own record, else the legacy record when the
    vgpu is the first open mdev.
* `gsp_phys_gpa_to_hpa()` kept as a thin `NULL` wrapper.
* Forwarding operands (`svc_fwd_operand_hpa`) pass the request's vgpu; the
  boot-ladder guest-GPA pin paths pass `NULL` on purpose (§3.2).

### G1 — sema-service thread (`gsp_sema_service.c`)

* Per pass: bounded list walk collects all open mdevs (≤ `SVC_MAX_GUESTS`=8,
  the driver's per-type ceiling) and runs the full per-guest body for each.
* Per-guest `svc_rebuild` freshness so the first guest's 10 s tick cannot
  starve the second guest of rebuilds.
* `pumped` is an aggregate: while any guest is progressing, the pass loops
  tightly and skips the heavy scans, as before.

### G2 — channel identity (`gsp_sema_service.c`, `gsp_rm_emulation.c`)

* `svc_channel.vgpu`; add/del match `(vgpu, h_object)`; `same` (re-register)
  requires the same owner; ranks count only same-owner channels; counter,
  scrubber-adjacent, opdef-drop, push-buffer discovery, GR content discovery,
  GR ring autopin max-CE scan, pump snapshot and pump write-back are all
  filtered by the owning vgpu.
* Scrubber channel state (`svc_scrub` → `svc_scrubs[]`) made per guest: the
  authored finish/USERD words are written through the owning guest's address
  space, never through whichever guest opened first.

### G3 — sessions (`gsp_sema_service.c`)

* `svc_guests[]` records: `active` + `last_build`, pointer-keyed, freed on
  mdev cleanup. `channel_add` marks active; `channel_del` deactivates the
  owner when its last channel closes and requests the global flush only when
  **no guest** has a live channel (staged semantics, §3.1).
* New `gsp_sema_service_vgpu_cleanup(vgpu)` (called from
  `main.c:1749`): quiesces forwarding, drops the guest's hold (pure clear —
  never a resolution through a vanishing device), removes its channels,
  drops its opdef release slots, frees its session and scrubber records.

### G5 (minimal) — forwarding identity (`gsp_sema_service.c`, `gsp_cuda_bridge.c`)

* The single-inflight request keeps ONE global slot (serialization kept) but
  carries its owner: CE requests set `svc_fwd_req.vgpu` (:13304), GR requests
  set it in `svc_gr_req_fill` (:12972), and the operand resolver translates
  through it.
* Hold abort keyed by `(vgpu, handle)`; both hold resolution paths
  (`svc_fwd_hold_gone_finish`, `svc_fwd_hold_stop_flush`) and the pump's
  cross-vgpu resolution check `svc_vgpu_open()` before ever dereferencing the
  stored device and drop the hold when its owner is gone.
* CUDA bridge (the GR fallback under the single-inflight slot): request and
  pending records now carry the owning mdev, the owner is part of the dedup
  identity, only the owner's pass marshals/finishes, and stale-owner records
  are dropped. This was not itemised in the inventory, but without it the
  expected dual steady state (guest B's forward refused while A's hold is
  active) would marshal guest B's fallback job through guest A.

### G8(a) — guest-GPA pin

* Refused while more than one mdev is open, with an explicit log; the pin
  remains a single-guest test aid translating through the first open mdev.

---

## 3. Deviations from the inventory plan (deliberate, minimal)

### 3.1 G3 flush is deferred, not per-guest-selective

The inventory asks to "flush only the departing vgpu's learned state". The
learned tables (`svc.map`, `svc.pin`, `svc.ipin`, `svc.lat`, `svc.vas*`,
`svc_pool_pa`, `svc_pending`, `svc_retry_q`, `svc_pb[]`, rank bitmaps) are
still shared — that is the explicitly out-of-scope G4 remainder. With shared
tables a selective flush is impossible; the only safe minimal semantic is:
**deactivate the departing guest's session and defer the flush while any other
guest still has live channels.** One guest: identical to today (last-channel
close is the global empty transition). Consequence for T5: guest B's
close→reopen while A is live does not wipe B's stale learned entries (they
must survive for A), so B's *second* session may hit the ISSUE-0002
contamination class for its own VAs. Closing that hole is the G4 minimum
(per-entry owner tags + per-guest lookup filter).

### 3.2 Boot-ladder guest-GPA call sites pass `NULL`

`gsp_phys_rmcall_pdb_build` (booter :9738) and triggers 6/7 (:19125, :19212)
have no request context; the only user of that GPA is the single-guest
test-only pin, which is now refused while >1 mdev is open (G8a). `NULL` is
the documented first-open resolution — identical behaviour for every
single-guest run, and explicitly not per-mdev.

### 3.3 Debugfs selector: extra token, not a subdir/per-uuid file

The hard requirement says "same debugfs file names". The recommended subdir
per UUID / `gsp-phys-guest-ram-<uuid>` file would add files; the optional
first-token UUID keeps the tree and the historical 4-token write untouched
while giving per-mdev targeting. The per-mdev form is additive: 5 tokens.

### 3.4 Scrubber registration binds to the first open mdev

`sema_scrubbuf_add` (debug tooling written by the guest scanner) has no
selector; it binds to the first open mdev's scrubber record (single-guest
convention, like the scanner itself) and is refused with no open mdev.

### 3.5 CUDA bridge owner gating (follow-on commit)

See §2/G5. Default-on (`cuda_bridge=1`); the change is inert for one guest.

---

## 4. Self-check: module-global session/pump state audit

Every read of module-global session/pump state after this change, classified:

| # | Global state | Sites | Status |
|---|--------------|-------|--------|
| 1 | channel table `svc.chan[]` | add :1154, del :1338, ranks :1204–:1221, counters :5198/:5271/:5360/:5551/:5600/:5626/:5686, pb says :7821, gr-content :9103, gr autopin :9445, pump :13653/:15201, chan dump :15516 | **fixed** per `(vgpu, h)` |
| 2 | session flags `svc_sess_active/flush` | :468–:520, :1356–:1378, :16364 | **fixed** (per-guest records drive the global flush) |
| 3 | scrubber `svc_scrub` | :535–:580, :1272–:1295, :15973, :15912 | **fixed** per guest |
| 4 | hold `svc_fwd_hold` | :11010–:12100 (:11956/:12019 guards) | **fixed**: vgpu-keyed abort + dead-owner guards |
| 5 | request `svc_fwd_req` | :10648, :12972/:13304, :13140–:13170 | **fixed**: owner carried; single-inflight kept |
| 6 | bridge req/pend | `gsp_cuda_bridge.c` (:143, :624, :985, :1353) | **fixed**: owner carried, owner-only marshal/finish |
| 7 | opdef slots | :11185–:11500 (drop_gone :11468) | pre-existing per `(vgpu,h)`; `drop_gone` filter **fixed** |
| 8 | learned map `svc.map` / `svc.map_n` / `svc.map_hash` | lookup :3300+, rebuild :3366 | **shared (G4)** — risk §5.1 |
| 9 | pins `svc.pin[]/ipin[]/svc_ipin_alt` | :2023–:2235, flush :7176 | **shared (G4)** — risk §5.1 |
| 10 | `svc.lat[]`, `svc.vas[]`, VA shadow, `svc_hot` | :1818+, :1878+, :2011 | **shared (G4)** — risk §5.1 |
| 11 | `svc_pending[]`, `svc_retry_q[]`, `svc_pool_pa*`, `svc_cnt_der` | :7081+, :885+, :2586+, :5335 | **shared (G4)** — risk §5.1 (pending is the top cross-guest write vector) |
| 12 | rank latches `svc_ring_done`, `svc_pb_done`, `svc_gr_ring_done`, `svc_pb[]`, gr-content state | :355, :341–:347, :501–:511 | **shared latches (G4)** — re-arm is content-validated/idempotent; extra re-scans possible |
| 13 | `svc_userd_sweep_at/n` (ce_index) | :13782+ | **shared (G4)** — equal ranks across guests share sweep slots |
| 14 | per-pass capture `svc_fwd_cand`, `svc_gr_cand`, `svc_fwd_rel` | :3896+, :2587+, :7233 | serially reused by the single service thread; safe with one in-flight, listed for completeness |
| 15 | debug surfaces reading the first open mdev | `add_map` rewind :15693, `dump_vidmem` :15783, `find_vidmem` :15819, `counter_add` :15868, scrubbuf :15912 | **debug tooling, single-guest by design** — do not use in dual runs |
| 16 | `gsp_phys_gpa_to_hpa(NULL,…)` (boot-ladder pin) | booter :9738/:19125/:19212 | **by design** first-open (§3.2, G8a refuses dual) |
| 17 | last-build freshness `svc.last_build` | :3570, :16105–:16114 | **fixed**: per-guest gate; `svc.last_build` kept as the published stamp |
| 18 | bridge debugfs FIFO views (job/in/out) | `gsp_cuda_bridge.c` | one global daemon FIFO; single-guest tooling surface |

No call site that carries a requesting/owning vgpu still matches session state
by handle alone in the paths in scope.

---

## 5. Risks (ranked)

1. **Shared learned tables (G4 remainder) — highest.** Cross-guest resolution
   is still possible: a lookup for guest B's VA can hit guest A's pin/map
   entry with the same VA; `svc_pending[]`/`svc_retry_q[]` entries carry no
   owner, so a queued release decoded for B can be applied through A (or
   vice-versa). Expected symptoms under T2/T4: wrong results, stalls,
   `rel-apply` writes to the wrong guest. This is the next planned change
   (G4 minimum: owner tag on pin/map/lat/pending + per-guest lookup filter).
2. **mdev-removal lifetime race.** The pump can be mid-pass for a vgpu whose
   fd just closed; the vgpu struct is freed after `vfio_put_device()` in the
   remove path. Pre-existing class (single-guest too), reduced by
   `gsp_sema_service_vgpu_cleanup` (channels/hold/records cleared before the
   free), not fully closed (the pump pass itself is not synchronised).
3. **G3 staged semantics (T5).** Guest B reopen while A is live inherits B's
   previous learned entries (see §3.1). A's next run is protected (no flush
   while A is live) — that is the T5 guard — but B's own reopen may hit the
   ISSUE-0002 wedge until G4-minimum.
4. **Shared rank latches/pairing (`svc_pb[]`, done bitmaps, sweep arrays).**
   Content validation makes re-arms idempotent, but equal ranks across guests
   can pair the wrong push-buffer base records and share USERD sweep slots;
   possible slowdowns/mis-pairs under concurrent CE load.
5. **Legacy guest-RAM write semantics.** With two guests, the 4-token write
   configures the legacy record served by the first open mdev; a script that
   writes it expecting "the new guest" will bind guest A. Use the 5-token
   UUID form for the second guest.
6. **Reducer scope:** no boot_stage split, no per-mdev debugfs views, no
   parallel forwarding — per plan. The single physical GR channel stays safe
   under `gsp_phys_rm_lock` (inventory §4), serialized at launch granularity.

---

## 6. Live-validation plan (offline work ended here)

All of the following is for the LEAD under the supervised main-job rules
(no unattended cycles; stock guests; evidence per run). The module was NOT
loaded by this task.

### 6.0 Build/load (from the dual worktree)

```bash
cd ~/Git/integdev_gpu_drv/open-iov-dual/kernel && make -j8
# note: modpost needs
#   ~/Git/integdev_gpu_drv/open-iov-dual/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers
# (gitignored; copied into this worktree from the master tree — keep it present
#  or the final link fails with "No such file or directory")
```

Load (after a clean/unloaded host; parameters per inventory §5):

```bash
sudo insmod ~/Git/integdev_gpu_drv/open-iov-dual/kernel/vgsp.ko \
  gsp_phys_gpu=1 gsp_phys_mdev_owned=1 gsp_phys_guest_mem=1 \
  gsp_phys_work_forward=1 gsp_phys_gr_forward=1
```

### 6.1 Setup — two turing mdevs, two QEMUs

```bash
GPU=0000:01:00.0   # substitute the campaign's GPU BDF (10de:…)
T=nvidia-nvidia-vgpu-turing
UUID_A=$(uuidgen); UUID_B=$(uuidgen)
echo "$UUID_A" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create
echo "$UUID_B" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create
# available_instances must read 2 - 2 = 0 afterwards
```

Launch QEMU A and B, each bound to its own mdev
(`-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID_A` resp. `$UUID_B`),
each with `memory-backend-file,share=yes` pointing at a **different** file
(e.g. `/dev/shm/vgpu-ram-a`, `/dev/shm/vgpu-ram-b`).

Register each guest's RAM window per mdev (5-token selector form):

```bash
echo "$UUID_A /dev/shm/vgpu-ram-a 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram
echo "$UUID_B /dev/shm/vgpu-ram-b 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram
# guest A may equally be configured by the legacy campaign scripts (4-token form);
# it serves the first open mdev. A second write of a different path to the same
# record is refused -EBUSY until that mdev is removed.
```

In each guest: stock 570.144 driver + `tools/cuda-ab/run/vecadd_loop` (or the
guest-side build from `tools/cuda-ab/`).

### 6.2 Test matrix (inventory §5, T1–T5)

| Test | Action | Pass criteria |
|------|--------|---------------|
| T1 | `nvidia-smi -L` in both guests simultaneously | no host error/stall (`dmesg -T | grep -i vgsp`) |
| T2 | start `./vecadd_loop` in A and B within 1 s | each prints `LOOP launches=100 bad=0`, exit 0 within 600 s |
| T3 | A vecadd, B started 10 s later | both green |
| T4 | `smiley` in A + `vecadd_loop` in B (CE vs GR cross) | both green |
| T5 | stop guest B → its mdev removed → recreate B (+ its RAM record) → rerun A and B | A green across B's cycle (B itself: see risk §5.3) |

Evidence capture: per-guest client logs + host dmesg to
`backups/evidence-*/dual-guest-<run>/`; require ≥200 lines each of
`GUEST GR LAUNCH OK` and zero `fwd_gr_fault`/timeouts in the steady state.
Per-mdev attribution in the log: `guest RAM configured for mdev <uuid>`,
`channel 0x… registered` around each guest's `GR LAUNCH submitted` stream.

Useful live checks:

```bash
dmesg -T | grep -E "sema-svc|vgsp-gr|GUEST GR LAUNCH OK|cuda-bridge" | tail -200
cat /sys/kernel/debug/vgsp/gsp-phys-state | grep -E "guest_ram|work_guest|backend"
sudo cat /sys/kernel/debug/vgsp/gsp-phys-state | grep guest_ram   # both mdev records visible
```

Baseline (pre-fix) for the regression control: T2 with the old module should
show the second guest stalling or wrong results (inventory §5).

### 6.3 Rollback

Worktree-local: `git checkout ecfb1278 -- kernel/ && make -j8` in
`open-iov-dual`; the master worktree is untouched.

---

## 7. Artifacts

* Branch `dual` commits `81c11ca0`, `5769d5c4` (local only, not pushed).
* `kernel/vgsp.ko` built at `5769d5c4` (offline; never loaded).
* This report. No sessions/dumps generated (no device access).