summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-16 13:33:02 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-16 13:33:02 -0400
commit06b9333cf64f613be5dfb3ea1ee40e087a182061 (patch)
treed327a143112ef3b6e2cd4194dc972771adc209c5
parentb09743033bb8c42fdeea672eaf260d1eebe84092 (diff)
parent15fdb36784b62e36a76a15e8072274fd8f2b55cf (diff)
downloadopen-iov-06b9333cf64f613be5dfb3ea1ee40e087a182061.tar.gz
open-iov-06b9333cf64f613be5dfb3ea1ee40e087a182061.zip
Merge branch 'dual'
-rw-r--r--DUAL-REPORT.md360
-rw-r--r--kernel/gsp_cuda_bridge.c92
-rw-r--r--kernel/gsp_phys_booter.c115
-rw-r--r--kernel/gsp_phys_gpu.c432
-rw-r--r--kernel/gsp_phys_gpu.h35
-rw-r--r--kernel/gsp_rm_emulation.c3
-rw-r--r--kernel/gsp_sema_service.c644
-rw-r--r--kernel/gsp_sema_service.h20
-rw-r--r--kernel/main.c10
9 files changed, 1458 insertions, 253 deletions
diff --git a/DUAL-REPORT.md b/DUAL-REPORT.md
new file mode 100644
index 0000000..334a772
--- /dev/null
+++ b/DUAL-REPORT.md
@@ -0,0 +1,360 @@
+# DUAL-REPORT.md — Stage B+C (minimal) per-mdev separation for dual-guest CUDA
+
+**Date:** 2026-10-15
+**Worktree:** `/home/rmcore/Git/integdev_gpu_drv/open-iov-dual` (branch `dual`, base `ecfb1278`)
+**Commits:** `81c11ca0` (stage B+C separation), `5769d5c4` (CUDA-bridge owner gating)
+**Build:** `cd kernel && make -j8` — clean success, `vgsp.ko` linked. No module
+was loaded, no device touched, no guest run (offline implementation only).
+
+Implementation of stages B+C (minimal) of
+`analysis-2026-10-15/AGENT-dual-guest-inventory.md` §3.1/§3.2 (items G1–G8),
+against the user scope: G1, G2, G3, G5-minimal, G6, G7, G8(a).
+Explicitly **not** implemented: G4 remainder (per-guest learned tables),
+G10 per-mdev debugfs views, parallel forwarding, boot_stage split.
+
+---
+
+## 1. Per-item status table
+
+| # | Item | Status | Change site (file:line @ `5769d5c4`) | Verified by |
+|---|------|--------|--------------------------------------|-------------|
+| G1 | Sema-service thread services ALL open mdevs each pass | DONE | `kernel/gsp_sema_service.c:16034` (`svc_thread`): collects every open mdev into `vgpus[]` (:16060–:16075) and runs the pass body per guest; `svc_handle_map_add()` drains once per pass; the scans/hint loop is per guest (:16174–:16226). Per-guest rebuild freshness (`sess->last_build`, :16105–:16119). | build only (`make -j8`) |
+| G2 | Channel table keyed by `(vgpu, h_object)`; per-guest rank; `vgpu` field in `svc_channel` | DONE | struct field `kernel/gsp_sema_service.c:165`; match in `channel_add` :1154 + `same` :1168; rank loops :1204–:1221; `channel_del` :1328; pump snapshot :13653; pump writeback :15201; counter paths :5198/:5271/:5360/:5551/:5600/:5626/:5686; `svc_pb_discover` :7821; `svc_gr_content_discover` :9103; `svc_gr_ring_autopin` :9445; chan dump :15516. Registration carries the owner: `kernel/gsp_rm_emulation.c:2360`; free passes it: :3248; desc field `kernel/gsp_sema_service.h:30`. | build only |
+| G3 | Per-vgpu session active/flush flags; departing guest never flushes a live guest | DONE (staged semantics — see §3.1) | session records + helpers `kernel/gsp_sema_service.c:468–:520` (struct :473, `svc_guest_get` :489); `channel_add` marks active :1242; `channel_del` deactivates per guest and requests the flush only when **no** guest has a live channel :1356–:1378; `gsp_sema_service_vgpu_cleanup` :16364 (session end on mdev remove). | build only |
+| G4 (remainder) | Per-guest learned tables (map/pin/vas/pending/…) | NOT IN SCOPE (documented risk §5.1) | — | — |
+| G5 (minimal) | Forward request/hold carry the owning vgpu; single global in-flight kept | DONE | req already carried `vgpu` (verified, set at :12972 GR `svc_gr_req_fill`, :13304 CE); hold abort keyed `(vgpu, handle)` :11091; dead-owner guards on both resolution paths :11928 (`svc_vgpu_open`), :11956 (`gone_finish`), :12019 (`stop_flush`); operand translation through the request's vgpu :10922. Follow-on: CUDA-bridge fallback queue now carries its owner (`kernel/gsp_cuda_bridge.c:143`, :624, :985, :1353). | build only |
+| G6 | Per-mdev guest-RAM backing {file, path, ranges} + per-mdev selector | DONE | records `kernel/gsp_phys_gpu.c:1264–:1373`; `gsp_phys_guest_ram_configure_vgpu` :1494; `gsp_phys_guest_ram_vgpu_clear` :1586; clear-all :1564; active :1604; state (:1646, legacy format byte-identical); debugfs writer `kernel/gsp_phys_booter.c:20487` (4-token legacy or 5-token `<uuid>` selector); selector resolve :20542 via `gsp_phys_open_mdev_by_name` (`gsp_phys_gpu.c:1356`). | build only |
+| G7 | `gpa_to_hpa` takes the requesting vgpu; all call sites plumbed | DONE | `gsp_phys_gpa_to_hpa_vgpu` `kernel/gsp_phys_gpu.c:1737`; wrapper `gsp_phys_gpa_to_hpa` :1827. Call sites: `gsp_sema_service.c:10922` (forwarding operands — passes the request's vgpu); `gsp_phys_booter.c:9738`, :19125, :19212 (single-guest guest-GPA pin — explicit `NULL` = first open mdev, see §3.2); no others remain (`grep`). | build only |
+| G8(a) | Guest-GPA pin explicitly single-guest; refusal + log | DONE | `kernel/gsp_phys_booter.c:20430–:20436`: pin refused `-EBUSY` with a log when `gsp_phys_open_mdev_count() > 1` (`gsp_phys_gpu.c:1333`); existing different-GPA refusal kept. Boot-ladder trigger call sites documented as first-open. | build only |
+| G10 | Per-mdev debugfs views | NOT IN SCOPE | — | — |
+
+Diff stats (`git diff --stat ecfb1278..HEAD`):
+
+```
+ kernel/gsp_cuda_bridge.c | 92 ++++++-
+ kernel/gsp_phys_booter.c | 115 +++++++--
+ kernel/gsp_phys_gpu.c | 432 ++++++++++++++++++++++++++-----
+ kernel/gsp_phys_gpu.h | 35 ++-
+ kernel/gsp_rm_emulation.c | 3 +-
+ kernel/gsp_sema_service.c | 644 +++++++++++++++++++++++++++++++++-----------
+ kernel/gsp_sema_service.h | 20 +-
+ kernel/main.c | 10 +
+ 8 files changed, 1098 insertions(+), 253 deletions(-)
+```
+
+### 1.1 One-mdev equivalence argument (explicit fast paths)
+
+With exactly ONE open mdev every changed path degenerates to the old one:
+
+1. **Thread (G1):** `vgpus[]` has one entry; the per-guest loop body is the
+ historical pass in the historical order (probe → walk_probe → chan_dump →
+ scrub → rebuild → pump → bridge → pending → sweep, then the scan loop).
+ `svc_handle_map_add()` runs once per pass (`gsp_sema_service.c:16085`),
+ exactly as before.
+2. **Channels (G2):** `vgpu` is constant, so `(vgpu, h_object)` matches the
+ same slots as `h_object` alone; rank loops count the same set; the pump
+ snapshot/writeback select the same channels.
+3. **Session/flush (G3):** the single guest's last-channel close empties the
+ global table → `svc_sess_active = 0; svc_sess_flush = 1` at the same
+ transition as before.
+4. **Guest RAM (G6/G7):** a 4-token `gsp-phys-guest-ram` write configures the
+ legacy record with byte-identical log lines; `gpa_to_hpa_vgpu(vgpu…)`
+ resolves the same first-open mdev and the same legacy record; the
+ `gsp-phys-state` `guest_ram:` field renders the byte-identical
+ `active=1 path=… ranges=… gpa=…` text (and `active=0` when unconfigured).
+5. **Forwarding (G5):** `svc_fwd_req`/`svc_fwd_hold` set and read the same
+ vgpu the single pump always used; the dead-owner guards never trigger while
+ the guest is open; the CUDA bridge's `head_owner == vgpu` is always true.
+6. **Log lines relied on by the campaign** are unchanged: `sema-svc:`,
+ `vgsp-gr:`, `GR LAUNCH submitted`, `GUEST GR LAUNCH OK`, `REAL GR SEM OK`,
+ `backend start: joined the live module-owned claim (READY)`, `guest RAM
+ configured:` / `guest RAM released:`. Debugfs file names are unchanged
+ (the selector is an extra token on the same file). Parameter defaults are
+ untouched.
+
+---
+
+## 2. What changed, by item
+
+### G6 — per-mdev guest-RAM backing (`gsp_phys_gpu.c`, `gsp_phys_booter.c`)
+
+* `guest_ram_file`/`guest_ram_path`/`guest_ram_ranges` globals replaced by
+ `guest_ram_legacy` (slot for the historical top-level write) + up to 4
+ `guest_ram_mdevs[]` records (vgpu-pointer keyed), all under the existing
+ `guest_ram_lock`.
+* New API: `gsp_phys_guest_ram_configure_vgpu()`,
+ `gsp_phys_guest_ram_vgpu_clear()`, `gsp_phys_open_mdev_count()`,
+ `gsp_phys_open_mdev_by_name()` (exports + prototypes in `gsp_phys_gpu.h`).
+* debugfs `gsp-phys-guest-ram` accepts both
+ `"<path> <gpa_base> <off> <size>"` (legacy: first/only mdev) and
+ `"<mdev-uuid> <path> <gpa_base> <off> <size>"` (per-mdev). Unknown selector
+ → `-ENODEV` with a log; `"-"` clears every record. Token count (4 vs 5)
+ discriminates; paths/UUIDs contain no spaces.
+* mdev removal releases the record (`main.c:1750`
+ `gsp_phys_guest_ram_vgpu_clear(vgpu)`); physical release clears all records
+ (`gsp_phys_guest_ram_clear()`).
+
+### G7 — vgpu-aware GPA→HPA (`gsp_phys_gpu.c` + call sites)
+
+* `gsp_phys_gpa_to_hpa_vgpu(vgpu, gpa, *hpa)`:
+ * `vgpu == NULL` → first open mdev (historical);
+ * named vgpu whose session is closed → `-ENODEV` (never retargeted);
+ * translation uses the vgpu's own record, else the legacy record when the
+ vgpu is the first open mdev.
+* `gsp_phys_gpa_to_hpa()` kept as a thin `NULL` wrapper.
+* Forwarding operands (`svc_fwd_operand_hpa`) pass the request's vgpu; the
+ boot-ladder guest-GPA pin paths pass `NULL` on purpose (§3.2).
+
+### G1 — sema-service thread (`gsp_sema_service.c`)
+
+* Per pass: bounded list walk collects all open mdevs (≤ `SVC_MAX_GUESTS`=8,
+ the driver's per-type ceiling) and runs the full per-guest body for each.
+* Per-guest `svc_rebuild` freshness so the first guest's 10 s tick cannot
+ starve the second guest of rebuilds.
+* `pumped` is an aggregate: while any guest is progressing, the pass loops
+ tightly and skips the heavy scans, as before.
+
+### G2 — channel identity (`gsp_sema_service.c`, `gsp_rm_emulation.c`)
+
+* `svc_channel.vgpu`; add/del match `(vgpu, h_object)`; `same` (re-register)
+ requires the same owner; ranks count only same-owner channels; counter,
+ scrubber-adjacent, opdef-drop, push-buffer discovery, GR content discovery,
+ GR ring autopin max-CE scan, pump snapshot and pump write-back are all
+ filtered by the owning vgpu.
+* Scrubber channel state (`svc_scrub` → `svc_scrubs[]`) made per guest: the
+ authored finish/USERD words are written through the owning guest's address
+ space, never through whichever guest opened first.
+
+### G3 — sessions (`gsp_sema_service.c`)
+
+* `svc_guests[]` records: `active` + `last_build`, pointer-keyed, freed on
+ mdev cleanup. `channel_add` marks active; `channel_del` deactivates the
+ owner when its last channel closes and requests the global flush only when
+ **no guest** has a live channel (staged semantics, §3.1).
+* New `gsp_sema_service_vgpu_cleanup(vgpu)` (called from
+ `main.c:1749`): quiesces forwarding, drops the guest's hold (pure clear —
+ never a resolution through a vanishing device), removes its channels,
+ drops its opdef release slots, frees its session and scrubber records.
+
+### G5 (minimal) — forwarding identity (`gsp_sema_service.c`, `gsp_cuda_bridge.c`)
+
+* The single-inflight request keeps ONE global slot (serialization kept) but
+ carries its owner: CE requests set `svc_fwd_req.vgpu` (:13304), GR requests
+ set it in `svc_gr_req_fill` (:12972), and the operand resolver translates
+ through it.
+* Hold abort keyed by `(vgpu, handle)`; both hold resolution paths
+ (`svc_fwd_hold_gone_finish`, `svc_fwd_hold_stop_flush`) and the pump's
+ cross-vgpu resolution check `svc_vgpu_open()` before ever dereferencing the
+ stored device and drop the hold when its owner is gone.
+* CUDA bridge (the GR fallback under the single-inflight slot): request and
+ pending records now carry the owning mdev, the owner is part of the dedup
+ identity, only the owner's pass marshals/finishes, and stale-owner records
+ are dropped. This was not itemised in the inventory, but without it the
+ expected dual steady state (guest B's forward refused while A's hold is
+ active) would marshal guest B's fallback job through guest A.
+
+### G8(a) — guest-GPA pin
+
+* Refused while more than one mdev is open, with an explicit log; the pin
+ remains a single-guest test aid translating through the first open mdev.
+
+---
+
+## 3. Deviations from the inventory plan (deliberate, minimal)
+
+### 3.1 G3 flush is deferred, not per-guest-selective
+
+The inventory asks to "flush only the departing vgpu's learned state". The
+learned tables (`svc.map`, `svc.pin`, `svc.ipin`, `svc.lat`, `svc.vas*`,
+`svc_pool_pa`, `svc_pending`, `svc_retry_q`, `svc_pb[]`, rank bitmaps) are
+still shared — that is the explicitly out-of-scope G4 remainder. With shared
+tables a selective flush is impossible; the only safe minimal semantic is:
+**deactivate the departing guest's session and defer the flush while any other
+guest still has live channels.** One guest: identical to today (last-channel
+close is the global empty transition). Consequence for T5: guest B's
+close→reopen while A is live does not wipe B's stale learned entries (they
+must survive for A), so B's *second* session may hit the ISSUE-0002
+contamination class for its own VAs. Closing that hole is the G4 minimum
+(per-entry owner tags + per-guest lookup filter).
+
+### 3.2 Boot-ladder guest-GPA call sites pass `NULL`
+
+`gsp_phys_rmcall_pdb_build` (booter :9738) and triggers 6/7 (:19125, :19212)
+have no request context; the only user of that GPA is the single-guest
+test-only pin, which is now refused while >1 mdev is open (G8a). `NULL` is
+the documented first-open resolution — identical behaviour for every
+single-guest run, and explicitly not per-mdev.
+
+### 3.3 Debugfs selector: extra token, not a subdir/per-uuid file
+
+The hard requirement says "same debugfs file names". The recommended subdir
+per UUID / `gsp-phys-guest-ram-<uuid>` file would add files; the optional
+first-token UUID keeps the tree and the historical 4-token write untouched
+while giving per-mdev targeting. The per-mdev form is additive: 5 tokens.
+
+### 3.4 Scrubber registration binds to the first open mdev
+
+`sema_scrubbuf_add` (debug tooling written by the guest scanner) has no
+selector; it binds to the first open mdev's scrubber record (single-guest
+convention, like the scanner itself) and is refused with no open mdev.
+
+### 3.5 CUDA bridge owner gating (follow-on commit)
+
+See §2/G5. Default-on (`cuda_bridge=1`); the change is inert for one guest.
+
+---
+
+## 4. Self-check: module-global session/pump state audit
+
+Every read of module-global session/pump state after this change, classified:
+
+| # | Global state | Sites | Status |
+|---|--------------|-------|--------|
+| 1 | channel table `svc.chan[]` | add :1154, del :1338, ranks :1204–:1221, counters :5198/:5271/:5360/:5551/:5600/:5626/:5686, pb says :7821, gr-content :9103, gr autopin :9445, pump :13653/:15201, chan dump :15516 | **fixed** per `(vgpu, h)` |
+| 2 | session flags `svc_sess_active/flush` | :468–:520, :1356–:1378, :16364 | **fixed** (per-guest records drive the global flush) |
+| 3 | scrubber `svc_scrub` | :535–:580, :1272–:1295, :15973, :15912 | **fixed** per guest |
+| 4 | hold `svc_fwd_hold` | :11010–:12100 (:11956/:12019 guards) | **fixed**: vgpu-keyed abort + dead-owner guards |
+| 5 | request `svc_fwd_req` | :10648, :12972/:13304, :13140–:13170 | **fixed**: owner carried; single-inflight kept |
+| 6 | bridge req/pend | `gsp_cuda_bridge.c` (:143, :624, :985, :1353) | **fixed**: owner carried, owner-only marshal/finish |
+| 7 | opdef slots | :11185–:11500 (drop_gone :11468) | pre-existing per `(vgpu,h)`; `drop_gone` filter **fixed** |
+| 8 | learned map `svc.map` / `svc.map_n` / `svc.map_hash` | lookup :3300+, rebuild :3366 | **shared (G4)** — risk §5.1 |
+| 9 | pins `svc.pin[]/ipin[]/svc_ipin_alt` | :2023–:2235, flush :7176 | **shared (G4)** — risk §5.1 |
+| 10 | `svc.lat[]`, `svc.vas[]`, VA shadow, `svc_hot` | :1818+, :1878+, :2011 | **shared (G4)** — risk §5.1 |
+| 11 | `svc_pending[]`, `svc_retry_q[]`, `svc_pool_pa*`, `svc_cnt_der` | :7081+, :885+, :2586+, :5335 | **shared (G4)** — risk §5.1 (pending is the top cross-guest write vector) |
+| 12 | rank latches `svc_ring_done`, `svc_pb_done`, `svc_gr_ring_done`, `svc_pb[]`, gr-content state | :355, :341–:347, :501–:511 | **shared latches (G4)** — re-arm is content-validated/idempotent; extra re-scans possible |
+| 13 | `svc_userd_sweep_at/n` (ce_index) | :13782+ | **shared (G4)** — equal ranks across guests share sweep slots |
+| 14 | per-pass capture `svc_fwd_cand`, `svc_gr_cand`, `svc_fwd_rel` | :3896+, :2587+, :7233 | serially reused by the single service thread; safe with one in-flight, listed for completeness |
+| 15 | debug surfaces reading the first open mdev | `add_map` rewind :15693, `dump_vidmem` :15783, `find_vidmem` :15819, `counter_add` :15868, scrubbuf :15912 | **debug tooling, single-guest by design** — do not use in dual runs |
+| 16 | `gsp_phys_gpa_to_hpa(NULL,…)` (boot-ladder pin) | booter :9738/:19125/:19212 | **by design** first-open (§3.2, G8a refuses dual) |
+| 17 | last-build freshness `svc.last_build` | :3570, :16105–:16114 | **fixed**: per-guest gate; `svc.last_build` kept as the published stamp |
+| 18 | bridge debugfs FIFO views (job/in/out) | `gsp_cuda_bridge.c` | one global daemon FIFO; single-guest tooling surface |
+
+No call site that carries a requesting/owning vgpu still matches session state
+by handle alone in the paths in scope.
+
+---
+
+## 5. Risks (ranked)
+
+1. **Shared learned tables (G4 remainder) — highest.** Cross-guest resolution
+ is still possible: a lookup for guest B's VA can hit guest A's pin/map
+ entry with the same VA; `svc_pending[]`/`svc_retry_q[]` entries carry no
+ owner, so a queued release decoded for B can be applied through A (or
+ vice-versa). Expected symptoms under T2/T4: wrong results, stalls,
+ `rel-apply` writes to the wrong guest. This is the next planned change
+ (G4 minimum: owner tag on pin/map/lat/pending + per-guest lookup filter).
+2. **mdev-removal lifetime race.** The pump can be mid-pass for a vgpu whose
+ fd just closed; the vgpu struct is freed after `vfio_put_device()` in the
+ remove path. Pre-existing class (single-guest too), reduced by
+ `gsp_sema_service_vgpu_cleanup` (channels/hold/records cleared before the
+ free), not fully closed (the pump pass itself is not synchronised).
+3. **G3 staged semantics (T5).** Guest B reopen while A is live inherits B's
+ previous learned entries (see §3.1). A's next run is protected (no flush
+ while A is live) — that is the T5 guard — but B's own reopen may hit the
+ ISSUE-0002 wedge until G4-minimum.
+4. **Shared rank latches/pairing (`svc_pb[]`, done bitmaps, sweep arrays).**
+ Content validation makes re-arms idempotent, but equal ranks across guests
+ can pair the wrong push-buffer base records and share USERD sweep slots;
+ possible slowdowns/mis-pairs under concurrent CE load.
+5. **Legacy guest-RAM write semantics.** With two guests, the 4-token write
+ configures the legacy record served by the first open mdev; a script that
+ writes it expecting "the new guest" will bind guest A. Use the 5-token
+ UUID form for the second guest.
+6. **Reducer scope:** no boot_stage split, no per-mdev debugfs views, no
+ parallel forwarding — per plan. The single physical GR channel stays safe
+ under `gsp_phys_rm_lock` (inventory §4), serialized at launch granularity.
+
+---
+
+## 6. Live-validation plan (offline work ended here)
+
+All of the following is for the LEAD under the supervised main-job rules
+(no unattended cycles; stock guests; evidence per run). The module was NOT
+loaded by this task.
+
+### 6.0 Build/load (from the dual worktree)
+
+```bash
+cd ~/Git/integdev_gpu_drv/open-iov-dual/kernel && make -j8
+# note: modpost needs
+# ~/Git/integdev_gpu_drv/open-iov-dual/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers
+# (gitignored; copied into this worktree from the master tree — keep it present
+# or the final link fails with "No such file or directory")
+```
+
+Load (after a clean/unloaded host; parameters per inventory §5):
+
+```bash
+sudo insmod ~/Git/integdev_gpu_drv/open-iov-dual/kernel/vgsp.ko \
+ gsp_phys_gpu=1 gsp_phys_mdev_owned=1 gsp_phys_guest_mem=1 \
+ gsp_phys_work_forward=1 gsp_phys_gr_forward=1
+```
+
+### 6.1 Setup — two turing mdevs, two QEMUs
+
+```bash
+GPU=0000:01:00.0 # substitute the campaign's GPU BDF (10de:…)
+T=nvidia-nvidia-vgpu-turing
+UUID_A=$(uuidgen); UUID_B=$(uuidgen)
+echo "$UUID_A" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create
+echo "$UUID_B" | sudo tee /sys/bus/pci/devices/$GPU/mdev_supported_types/$T/create
+# available_instances must read 2 - 2 = 0 afterwards
+```
+
+Launch QEMU A and B, each bound to its own mdev
+(`-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID_A` resp. `$UUID_B`),
+each with `memory-backend-file,share=yes` pointing at a **different** file
+(e.g. `/dev/shm/vgpu-ram-a`, `/dev/shm/vgpu-ram-b`).
+
+Register each guest's RAM window per mdev (5-token selector form):
+
+```bash
+echo "$UUID_A /dev/shm/vgpu-ram-a 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram
+echo "$UUID_B /dev/shm/vgpu-ram-b 0x100000000 0x0 0x40000000" | sudo tee /sys/kernel/debug/vgsp/gsp-phys-guest-ram
+# guest A may equally be configured by the legacy campaign scripts (4-token form);
+# it serves the first open mdev. A second write of a different path to the same
+# record is refused -EBUSY until that mdev is removed.
+```
+
+In each guest: stock 570.144 driver + `tools/cuda-ab/run/vecadd_loop` (or the
+guest-side build from `tools/cuda-ab/`).
+
+### 6.2 Test matrix (inventory §5, T1–T5)
+
+| Test | Action | Pass criteria |
+|------|--------|---------------|
+| T1 | `nvidia-smi -L` in both guests simultaneously | no host error/stall (`dmesg -T | grep -i vgsp`) |
+| T2 | start `./vecadd_loop` in A and B within 1 s | each prints `LOOP launches=100 bad=0`, exit 0 within 600 s |
+| T3 | A vecadd, B started 10 s later | both green |
+| T4 | `smiley` in A + `vecadd_loop` in B (CE vs GR cross) | both green |
+| T5 | stop guest B → its mdev removed → recreate B (+ its RAM record) → rerun A and B | A green across B's cycle (B itself: see risk §5.3) |
+
+Evidence capture: per-guest client logs + host dmesg to
+`backups/evidence-*/dual-guest-<run>/`; require ≥200 lines each of
+`GUEST GR LAUNCH OK` and zero `fwd_gr_fault`/timeouts in the steady state.
+Per-mdev attribution in the log: `guest RAM configured for mdev <uuid>`,
+`channel 0x… registered` around each guest's `GR LAUNCH submitted` stream.
+
+Useful live checks:
+
+```bash
+dmesg -T | grep -E "sema-svc|vgsp-gr|GUEST GR LAUNCH OK|cuda-bridge" | tail -200
+cat /sys/kernel/debug/vgsp/gsp-phys-state | grep -E "guest_ram|work_guest|backend"
+sudo cat /sys/kernel/debug/vgsp/gsp-phys-state | grep guest_ram # both mdev records visible
+```
+
+Baseline (pre-fix) for the regression control: T2 with the old module should
+show the second guest stalling or wrong results (inventory §5).
+
+### 6.3 Rollback
+
+Worktree-local: `git checkout ecfb1278 -- kernel/ && make -j8` in
+`open-iov-dual`; the master worktree is untouched.
+
+---
+
+## 7. Artifacts
+
+* Branch `dual` commits `81c11ca0`, `5769d5c4` (local only, not pushed).
+* `kernel/vgsp.ko` built at `5769d5c4` (offline; never loaded).
+* This report. No sessions/dumps generated (no device access).
diff --git a/kernel/gsp_cuda_bridge.c b/kernel/gsp_cuda_bridge.c
index 2fac2b2..7adc8e3 100644
--- a/kernel/gsp_cuda_bridge.c
+++ b/kernel/gsp_cuda_bridge.c
@@ -141,6 +141,9 @@ MODULE_PARM_DESC(cuda_bridge,
/* ------------------------------------------------------------------ */
struct cubb_req {
+ struct nv_vgpu_device *vgpu; /* owning mdev (guest session); the
+ * queue is FIFO, so only the owner's
+ * service pass may marshal this job */
u32 h_object;
u64 qmd_dev_va;
u64 push_va;
@@ -178,6 +181,9 @@ static u64 cubb_dedup_hits;
struct cubb_pend {
bool valid;
+ struct nv_vgpu_device *vgpu; /* owning mdev: the completion write-back
+ * must run through this guest's
+ * address space, never another's */
u64 seq;
u32 h_object;
u32 opcode;
@@ -576,12 +582,15 @@ void gsp_cuda_bridge_launch(struct nv_vgpu_device *vgpu, u32 h_object,
if (!gsp_cuda_bridge_enabled())
return;
- (void)vgpu;
for (i = 0; i < 64; i++) {
hash ^= qmd[i];
hash *= 1099511628211ULL;
}
+ /* The owner is part of the identity: two guests' identical launches
+ * must never dedup against each other. */
+ hash ^= (u64)(unsigned long)vgpu;
+ hash *= 1099511628211ULL;
hash ^= (u64)h_object << 32;
hash *= 1099511628211ULL;
hash ^= qmd_dev_va;
@@ -612,6 +621,7 @@ void gsp_cuda_bridge_launch(struct nv_vgpu_device *vgpu, u32 h_object,
struct cubb_req *r =
&cubb_req[cubb_req_head % CUBB2_REQ_MAX];
+ r->vgpu = vgpu;
r->h_object = h_object;
r->qmd_dev_va = qmd_dev_va;
r->push_va = push_va;
@@ -972,6 +982,7 @@ static int cubb_marshal_one(struct nv_vgpu_device *vgpu)
}
job.seq = ++cubb_seq;
cubb_pend[slot].valid = true;
+ cubb_pend[slot].vgpu = r.vgpu;
cubb_pend[slot].seq = job.seq;
cubb_pend[slot].h_object = r.h_object;
cubb_pend[slot].opcode = opcode;
@@ -1303,12 +1314,48 @@ static bool cubb_finish_try(struct nv_vgpu_device *vgpu, struct cubb_pend *p,
}
/*
+ * Is this mdev still on the driver list with an open VFIO device? Guards the
+ * bridge's per-request/per-pend owner pointers (dual-guest goal 1.0): a
+ * removed mdev must never be dereferenced and its queued work can never be
+ * observed by a live guest. Pointer comparisons only - safe to call with a
+ * possibly-stale pointer.
+ */
+static bool cubb_vgpu_open(const struct nv_vgpu_device *vgpu)
+{
+ struct nv_vgpu_device *d;
+ bool open = false;
+
+ if (!vgpu)
+ return false;
+ mutex_lock(&nv_vgpu_driver.lock);
+ list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
+ if (d == vgpu && atomic_read(&d->vfio_dev_open_count) > 0) {
+ open = true;
+ break;
+ }
+ }
+ mutex_unlock(&nv_vgpu_driver.lock);
+ return open;
+}
+
+/*
* Service-thread hook: marshal at most one queued launch into a daemon-visible
* job, stage any returned done records, then drive staged completions. Called
* every service pass; a no-op when no launch has been seen.
+ *
+ * Dual-guest (goal 1.0): request and pend records carry their owning mdev
+ * (the bridge falls back here when forwarding was refused under the
+ * single-inflight slot). Only the owning guest's pass marshals its job (the
+ * marshal resolves VAs and reads guest memory through that guest) and only
+ * the owning pass writes a completion back; a head/record whose owner is
+ * gone is dropped instead of ever being dereferenced.
*/
void gsp_cuda_bridge_service(struct nv_vgpu_device *vgpu)
{
+ struct nv_vgpu_device *head_owner = NULL;
+ unsigned long flags;
+ u32 head_h = 0;
+ bool have_head = false;
struct cubb2_done d;
int guard = 0;
int i;
@@ -1316,7 +1363,29 @@ void gsp_cuda_bridge_service(struct nv_vgpu_device *vgpu)
if (!gsp_cuda_bridge_enabled())
return;
- (void)cubb_marshal_one(vgpu);
+ spin_lock_irqsave(&cubb_req_lock, flags);
+ if (cubb_req_head != cubb_req_tail) {
+ struct cubb_req *head =
+ &cubb_req[cubb_req_tail % CUBB2_REQ_MAX];
+
+ head_owner = head->vgpu;
+ head_h = head->h_object;
+ have_head = true;
+ }
+ spin_unlock_irqrestore(&cubb_req_lock, flags);
+ if (have_head) {
+ if (head_owner == vgpu) {
+ (void)cubb_marshal_one(vgpu);
+ } else if (!cubb_vgpu_open(head_owner)) {
+ /* The owner's guest is gone: nobody can observe it. */
+ cubb_req_drop_head();
+ cubb_dropped++;
+ pr_info_ratelimited("cuda-bridge: dropped a queued launch whose mdev is gone (h=0x%x)\n",
+ head_h);
+ }
+ /* else: another live guest's head; its own pass marshals it. */
+ }
+
while (guard++ < CUBB2_DONE_RING && cubb_done_pop(&d)) {
if (d.magic != CUBB2_DONE_MAGIC ||
d.version != CUBB2_VERSION) {
@@ -1330,8 +1399,23 @@ void gsp_cuda_bridge_service(struct nv_vgpu_device *vgpu)
mutex_lock(&cubb_lock);
for (i = 0; i < CUBB2_PEND_MAX; i++) {
- if (cubb_pend[i].valid && cubb_pend[i].have_done &&
- cubb_finish_try(vgpu, &cubb_pend[i], i))
+ struct cubb_pend *p = &cubb_pend[i];
+
+ if (!p->valid || !p->have_done)
+ continue;
+ if (p->vgpu != vgpu) {
+ /*
+ * Another guest's staged completion: its own pass
+ * drives the write-back. A removed owner's record is
+ * released here so the slot frees.
+ */
+ if (!cubb_vgpu_open(p->vgpu)) {
+ cubb_pend_release(p, i);
+ i = -1; /* compaction shifted the entries */
+ }
+ continue;
+ }
+ if (cubb_finish_try(vgpu, p, i))
i = -1; /* compaction shifted the entries; rescan */
}
mutex_unlock(&cubb_lock);
diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c
index b2302bc..ec39d39 100644
--- a/kernel/gsp_phys_booter.c
+++ b/kernel/gsp_phys_booter.c
@@ -9861,8 +9861,16 @@ static void gsp_phys_rmcall_pdb_build(void)
u64 guest_pte;
int gpa_ret;
- gpa_ret = gsp_phys_gpa_to_hpa(boot_stage.rmcall_guest_gpa,
- &guest_hpa);
+ /*
+ * The pinned GPA is a single-guest test aid (the pin write is
+ * refused while more than one mdev is open; see
+ * gsp_phys_guest_gpa_write): resolve through the first open
+ * mdev exactly as before the per-mdev records, via the NULL
+ * selector (a per-mdev pin is not modelled).
+ */
+ gpa_ret = gsp_phys_gpa_to_hpa_vgpu(NULL,
+ boot_stage.rmcall_guest_gpa,
+ &guest_hpa);
if (!gpa_ret) {
guest_pte = gsp_phys_rmcall_pte(guest_hpa);
put_unaligned_le64(guest_pte,
@@ -19293,7 +19301,11 @@ static int gsp_phys_rmcall_guest_g2h(void __iomem *bar0)
return -EIO;
}
- ret = gsp_phys_gpa_to_hpa(gpa, &hpa);
+ /*
+ * Single-guest pin (see gsp_phys_guest_gpa_write): NULL selects the
+ * first open mdev, exactly as before the per-mdev records.
+ */
+ ret = gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, &hpa);
if (ret) {
pr_err("vgsp-boot: rm-call: GUEST G2H COPY FAILED: hpa=0x%llx pte=0x%016llx: GPA 0x%llx -> HPA translation failed (%d: no active mdev/VFIO domain, GPA unmapped, or not a RAM PFN)\n",
hpa, pte, gpa, ret);
@@ -19376,7 +19388,11 @@ static int gsp_phys_rmcall_guest_w2g(void __iomem *bar0)
return -EIO;
}
- ret = gsp_phys_gpa_to_hpa(gpa, &hpa);
+ /*
+ * Single-guest pin (see gsp_phys_guest_gpa_write): NULL selects the
+ * first open mdev, exactly as before the per-mdev records.
+ */
+ ret = gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, &hpa);
if (ret) {
pr_err("vgsp-boot: rm-call: GUEST W2G COPY FAILED: hpa=0x%llx pte=0x%016llx: GPA 0x%llx -> HPA translation failed (%d: no active mdev/VFIO domain, GPA unmapped, or not a RAM PFN)\n",
hpa, pte, gpa, ret);
@@ -20595,6 +20611,18 @@ static ssize_t gsp_phys_guest_gpa_write(struct file *file,
gpa);
return -EINVAL;
}
+ /*
+ * Dual-guest guard (goal 1.0): the pin is a single-guest test aid. The
+ * triggers translate it through the first open mdev (see
+ * gsp_phys_gpa_to_hpa / boot_stage.rmcall_guest_gpa), so with more than
+ * one open mdev the pin cannot name its owner and is refused instead of
+ * silently resolving through whichever guest opened first.
+ */
+ if (gsp_phys_open_mdev_count() > 1) {
+ pr_err("vgsp-boot: gsp-phys-guest-gpa: refused: %d mdevs are open; the guest-GPA pin is a single-guest test aid (translates through the first open mdev). Close all but one mdev, or use the per-mdev guest/GPA paths\n",
+ gsp_phys_open_mdev_count());
+ return -EBUSY;
+ }
if (boot_stage.rmcall_guest_gpa_valid &&
boot_stage.rmcall_guest_gpa != gpa) {
pr_err("vgsp-boot: gsp-phys-guest-gpa: refused: this boot already pinned GPA 0x%llx; one GPA per boot (stale GMMU TLB safety). Reuse that GPA, or release/reboot before changing it\n",
@@ -20617,18 +20645,26 @@ static const struct file_operations gsp_phys_guest_gpa_fops = {
};
/*
- * gsp-phys-guest-ram (write-only, mode 0200): configure the shared guest RAM
- * backing file that gsp_phys_gpa_to_hpa() translates through (QEMU
+ * gsp-phys-guest-ram (write-only, mode 0200): configure a guest RAM backing
+ * file that gsp_phys_gpa_to_hpa_vgpu() translates through (QEMU
* memory-backend-file,...,share=yes). One write is
*
* "<path> <gpa_base_hex> <file_offset_hex> <size_hex>"
*
* (path without spaces; all three numbers hex with 0x, e.g.
- * "/dev/shm/vgpu-ram 0x100000000 0x0 0x40000000"). The first write opens the
- * file and fixes the path for this module cycle; later writes may add ranges
- * (up to four) or replace a range with the same gpa_base. The ranges are
- * only consulted by gsp_phys_gpa_to_hpa(), and only while a live mdev
- * session exists. A single "-" clears the configuration and closes the file.
+ * "/dev/shm/vgpu-ram 0x100000000 0x0 0x40000000"). This historical form
+ * configures the legacy record and serves the FIRST/ONLY open mdev exactly
+ * as before the per-mdev records existed. With two live guests each writes
+ * its own record by naming its mdev UUID first:
+ *
+ * "<mdev-uuid> <path> <gpa_base_hex> <file_offset_hex> <size_hex>"
+ *
+ * Paths and UUIDs contain no spaces, so the token count (4 vs 5) decides.
+ * The first write for a record opens the file and fixes the path for that
+ * record; later writes may add ranges (up to four) or replace a range with
+ * the same gpa_base. The ranges are only consulted by the translators, and
+ * only while a live mdev session exists. A single "-" clears ALL records
+ * (legacy and per-mdev) and closes the files.
*/
static char *gsp_phys_guest_ram_token(char **cursor)
{
@@ -20643,9 +20679,11 @@ static ssize_t gsp_phys_guest_ram_write(struct file *file,
const char __user *ubuf, size_t len,
loff_t *ppos)
{
- char buf[384];
- char *p, *tok, *path;
+ char buf[448];
+ char *t[5];
+ char *path;
u64 gpa_base, file_offset, size;
+ int nt = 0, off;
int ret;
if (len == 0 || len >= sizeof(buf))
@@ -20654,30 +20692,63 @@ static ssize_t gsp_phys_guest_ram_write(struct file *file,
return -EFAULT;
buf[len] = '\0';
- p = strim(buf);
- if (!strcmp(p, "-")) {
+ if (!strcmp(strim(buf), "-")) {
gsp_phys_guest_ram_clear();
pr_info("vgsp-boot: gsp-phys-guest-ram: cleared the guest RAM file configuration\n");
return len;
}
- path = gsp_phys_guest_ram_token(&p);
- tok = gsp_phys_guest_ram_token(&p);
- if (!path || !tok || kstrtoull(tok, 0, &gpa_base)) {
+ {
+ char *p = buf;
+
+ while (nt < 5 && (t[nt] = gsp_phys_guest_ram_token(&p)))
+ nt++;
+ }
+ if (nt != 4 && nt != 5) {
+ pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected \"<path> <gpa_base_hex> <file_offset_hex> <size_hex>\", \"<mdev-uuid> <path> <gpa_base_hex> <file_offset_hex> <size_hex>\" or \"-\"\n");
+ return -EINVAL;
+ }
+ off = nt - 4; /* 0 = legacy form, 1 = per-mdev form */
+ path = t[off];
+ if (!path || kstrtoull(t[off + 1], 0, &gpa_base)) {
pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected \"<path> <gpa_base_hex> <file_offset_hex> <size_hex>\" or \"-\"\n");
return -EINVAL;
}
- tok = gsp_phys_guest_ram_token(&p);
- if (!tok || kstrtoull(tok, 0, &file_offset)) {
+ if (kstrtoull(t[off + 2], 0, &file_offset)) {
pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected a hex file offset after the GPA\n");
return -EINVAL;
}
- tok = gsp_phys_guest_ram_token(&p);
- if (!tok || kstrtoull(tok, 0, &size)) {
+ if (kstrtoull(t[off + 3], 0, &size)) {
pr_info("vgsp-boot: gsp-phys-guest-ram: write ignored: expected a hex size after the file offset\n");
return -EINVAL;
}
+ if (off) {
+ /*
+ * Per-mdev form: the selector must name an OPEN mdev (the mdev
+ * UUID is its device name). Refusing an unknown selector keeps
+ * a typo from silently configuring the legacy record.
+ */
+ struct nv_vgpu_device *mdev =
+ gsp_phys_open_mdev_by_name(t[0]);
+
+ if (!mdev) {
+ pr_err("vgsp-boot: gsp-phys-guest-ram: no open mdev named %s; write refused\n",
+ t[0]);
+ return -ENODEV;
+ }
+ ret = gsp_phys_guest_ram_configure_vgpu(mdev, path, gpa_base,
+ file_offset, size);
+ if (ret) {
+ pr_err("vgsp-boot: gsp-phys-guest-ram: configure failed: mdev=%s path=%s gpa=0x%llx off=0x%llx size=0x%llx ret=%d\n",
+ t[0], path, gpa_base, file_offset, size, ret);
+ return ret;
+ }
+ pr_info("vgsp-boot: gsp-phys-guest-ram: configured mdev=%s path=%s gpa=0x%llx off=0x%llx size=0x%llx\n",
+ t[0], path, gpa_base, file_offset, size);
+ return len;
+ }
+
ret = gsp_phys_guest_ram_configure(path, gpa_base, file_offset, size);
if (ret) {
pr_err("vgsp-boot: gsp-phys-guest-ram: configure failed: path=%s gpa=0x%llx off=0x%llx size=0x%llx ret=%d\n",
diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c
index 7932268..b1945ee 100644
--- a/kernel/gsp_phys_gpu.c
+++ b/kernel/gsp_phys_gpu.c
@@ -1278,16 +1278,28 @@ EXPORT_SYMBOL_GPL(gsp_phys_gpu_fw_crc32);
* of a shared file. A GPA is translated by reading the file page that holds
* it (read_cache_page() pulls it into the page cache and pins it for the
* read) and converting that page to a host PA with page_to_phys(). The
- * ranges map guest GPA windows to file offsets; up to four windows of one
- * file per module cycle.
+ * ranges map guest GPA windows to file offsets; up to four windows per file.
+ *
+ * Per-mdev records (dual-guest goal 1.0): a guest RAM backing file belongs
+ * to ONE guest session, so the record moves with the mdev. Slot 0 is the
+ * legacy record (vgpu == NULL): it is what the historical top-level
+ * gsp-phys-guest-ram debugfs write configures and it serves the first/only
+ * open mdev exactly as the pre-dual-guest code did (one guest: byte-identical
+ * behaviour). A per-mdev record is configured with the mdev UUID as an extra
+ * first token on the same debugfs file and serves only that mdev; the
+ * record is released when the mdev is removed (gsp_phys_guest_ram_vgpu_clear)
+ * or when the physical cycle ends (gsp_phys_guest_ram_clear).
*
* Locking: guest_ram_lock serialises configure/clear against each other and
* against the translators; it is held across read_cache_page() (which may
* sleep) so a clear cannot fput the file while a translation is using it.
- * One file per module cycle: a second path is refused -EBUSY because a fresh
- * cycle reloads the module; more ranges of the same file may be added.
+ * Within one record the historical rule is unchanged: the first configure
+ * opens the file, a different path is refused -EBUSY, more ranges of the
+ * same file may be added.
*/
#define GSP_PHYS_GUEST_RAM_RANGES 4
+/* Per-mdev guest-RAM records (a handful of guests; 4 is far beyond 1.0). */
+#define GSP_PHYS_GUEST_RAM_MDEVS 4
struct gsp_phys_guest_ram_range {
u64 gpa_base;
@@ -1295,35 +1307,121 @@ struct gsp_phys_guest_ram_range {
u64 size;
};
-static struct file *guest_ram_file;
-static char guest_ram_path[128];
-static struct gsp_phys_guest_ram_range guest_ram_ranges[GSP_PHYS_GUEST_RAM_RANGES];
-static unsigned int guest_ram_ranges_n;
+struct gsp_phys_guest_ram_rec {
+ struct nv_vgpu_device *vgpu; /* NULL = legacy (first/only open mdev) */
+ struct file *file;
+ char path[128];
+ struct gsp_phys_guest_ram_range ranges[GSP_PHYS_GUEST_RAM_RANGES];
+ unsigned int ranges_n;
+};
+
+static struct gsp_phys_guest_ram_rec guest_ram_legacy;
+static struct gsp_phys_guest_ram_rec guest_ram_mdevs[GSP_PHYS_GUEST_RAM_MDEVS];
static DEFINE_MUTEX(guest_ram_lock);
+/* Caller holds guest_ram_lock. NULL when this vgpu has no bound record. */
+static struct gsp_phys_guest_ram_rec *
+gsp_phys_guest_ram_rec_find_locked(const struct nv_vgpu_device *vgpu)
+{
+ unsigned int i;
+
+ if (!vgpu)
+ return NULL;
+ for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++)
+ if (guest_ram_mdevs[i].vgpu == vgpu)
+ return &guest_ram_mdevs[i];
+ return NULL;
+}
+
+/* Caller holds guest_ram_lock. NULL when every per-mdev slot is taken. */
+static struct gsp_phys_guest_ram_rec *
+gsp_phys_guest_ram_rec_alloc_locked(struct nv_vgpu_device *vgpu)
+{
+ unsigned int i;
+
+ for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++)
+ if (!guest_ram_mdevs[i].vgpu) {
+ guest_ram_mdevs[i].vgpu = vgpu;
+ return &guest_ram_mdevs[i];
+ }
+ return NULL;
+}
+
/*
- * Configure the guest RAM backing file and add (or replace, keyed by
- * gpa_base) one GPA window. The first call opens the file read-only and
- * fixes the path for this module cycle; later calls may only add ranges of
- * that same file. The file must be a regular file and every range must fit
- * inside its current size, so a translation can never index a page beyond
- * EOF. Returns 0, -EBUSY for a different path, -ENOSPC with all four ranges
- * in use, or the file/argument error.
+ * Number of mdevs whose VFIO device is currently open. Shared by the
+ * per-mdev debugfs selector (which mdev does this UUID name?) and the
+ * single-guest guard on the legacy test-only paths.
*/
-int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base,
- u64 file_offset, u64 size)
+int gsp_phys_open_mdev_count(void)
+{
+ struct nv_vgpu_device *d;
+ int n = 0;
+
+ mutex_lock(&nv_vgpu_driver.lock);
+ list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
+ if (atomic_read(&d->vfio_dev_open_count) > 0)
+ n++;
+ }
+ mutex_unlock(&nv_vgpu_driver.lock);
+ return n;
+}
+EXPORT_SYMBOL_GPL(gsp_phys_open_mdev_count);
+
+/*
+ * Resolve an OPEN mdev by its device name (the mdev UUID, e.g.
+ * "b6acdc5e-..."). Returns the device or NULL. The caller must use the
+ * result only while that mdev's VFIO fd is known to be open (the same
+ * lifetime rule as the first-open resolution below): the driver list is
+ * walked under nv_vgpu_driver.lock, but the returned pointer is not
+ * reference-counted.
+ */
+struct nv_vgpu_device *gsp_phys_open_mdev_by_name(const char *name)
+{
+ struct nv_vgpu_device *d, *found = NULL;
+
+ if (!name || !*name)
+ return NULL;
+ mutex_lock(&nv_vgpu_driver.lock);
+ list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
+ if (atomic_read(&d->vfio_dev_open_count) > 0 && d->mdev &&
+ !strcmp(dev_name(&d->mdev->dev), name)) {
+ found = d;
+ break;
+ }
+ }
+ mutex_unlock(&nv_vgpu_driver.lock);
+ return found;
+}
+EXPORT_SYMBOL_GPL(gsp_phys_open_mdev_by_name);
+
+/*
+ * Shared configure body for one record: open the record's file (first call)
+ * and add (or replace, keyed by gpa_base) one GPA window. The first call
+ * opens the file read-only and fixes the path for the record's lifetime;
+ * later calls may only add ranges of that same file. The file must be a
+ * regular file and every range must fit inside its current size, so a
+ * translation can never index a page beyond EOF. Returns 0, -EBUSY for a
+ * different path, -ENOSPC with all four ranges in use, or the
+ * file/argument error. `who` is "" for the legacy record and
+ * " for mdev <uuid>" for a per-mdev record; the legacy strings stay
+ * byte-identical.
+ */
+static int gsp_phys_guest_ram_rec_configure(struct gsp_phys_guest_ram_rec *rec,
+ const char *path, u64 gpa_base,
+ u64 file_offset, u64 size,
+ const char *who)
{
struct gsp_phys_guest_ram_range *range;
struct file *f = NULL;
unsigned int i;
int ret = 0;
- if (!path || !*path)
+ if (!rec || !path || !*path)
return -EINVAL;
mutex_lock(&guest_ram_lock);
- if (!guest_ram_file) {
+ if (!rec->file) {
/*
* O_LARGEFILE is required: the guest RAM file is multi-GiB and
* generic_file_open() (shmem's open) refuses a file beyond
@@ -1353,35 +1451,38 @@ int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base,
ret = -ERANGE;
goto out_fput;
}
- guest_ram_file = f;
- strscpy(guest_ram_path, path, sizeof(guest_ram_path));
- } else if (strcmp(guest_ram_path, path)) {
- pr_err("vgsp: guest RAM already configured from %s; one file per module cycle (reload to switch); refusing %s\n",
- guest_ram_path, path);
+ rec->file = f;
+ strscpy(rec->path, path, sizeof(rec->path));
+ } else if (strcmp(rec->path, path)) {
+ if (who[0])
+ pr_err("vgsp: guest RAM already configured from %s; one file per guest session (reload or mdev remove to switch); refusing %s\n",
+ rec->path, path);
+ else
+ pr_err("vgsp: guest RAM already configured from %s; one file per module cycle (reload to switch); refusing %s\n",
+ rec->path, path);
ret = -EBUSY;
goto out;
}
- for (i = 0, range = NULL; i < guest_ram_ranges_n; i++) {
- if (guest_ram_ranges[i].gpa_base == gpa_base) {
- range = &guest_ram_ranges[i];
+ for (i = 0, range = NULL; i < rec->ranges_n; i++) {
+ if (rec->ranges[i].gpa_base == gpa_base) {
+ range = &rec->ranges[i];
break;
}
}
if (!range) {
- if (guest_ram_ranges_n >= GSP_PHYS_GUEST_RAM_RANGES) {
+ if (rec->ranges_n >= GSP_PHYS_GUEST_RAM_RANGES) {
ret = -ENOSPC;
goto out;
}
- range = &guest_ram_ranges[guest_ram_ranges_n++];
+ range = &rec->ranges[rec->ranges_n++];
range->gpa_base = gpa_base;
}
range->file_offset = file_offset;
range->size = size;
- pr_info("vgsp: guest RAM configured: path=%s gpa=0x%llx off=0x%llx size=0x%llx (ranges=%u)\n",
- guest_ram_path, gpa_base, file_offset, size,
- guest_ram_ranges_n);
+ pr_info("vgsp: guest RAM configured%s: path=%s gpa=0x%llx off=0x%llx size=0x%llx (ranges=%u)\n",
+ who, rec->path, gpa_base, file_offset, size, rec->ranges_n);
out:
mutex_unlock(&guest_ram_lock);
@@ -1391,35 +1492,148 @@ out_fput:
fput(f);
goto out;
}
+
+/*
+ * Configure the legacy guest RAM record (the historical top-level
+ * gsp-phys-guest-ram write). Serves the first/only open mdev exactly as
+ * before the per-mdev records existed.
+ */
+int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base,
+ u64 file_offset, u64 size)
+{
+ return gsp_phys_guest_ram_rec_configure(&guest_ram_legacy, path,
+ gpa_base, file_offset, size,
+ "");
+}
EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_configure);
/*
- * Close the configured guest RAM file (if any) and drop every range. Called
- * from the physical-GPU release path so module unload and a backend release
- * end the file reference; a fresh cycle reconfigures from debugfs.
+ * Configure the guest RAM record bound to one mdev (per-mdev selector on the
+ * debugfs write). A record for this vgpu is created on first use and closed
+ * by gsp_phys_guest_ram_vgpu_clear() when the mdev is removed. A failed
+ * first configure leaves no bound record behind.
+ */
+int gsp_phys_guest_ram_configure_vgpu(struct nv_vgpu_device *vgpu,
+ const char *path, u64 gpa_base,
+ u64 file_offset, u64 size)
+{
+ struct gsp_phys_guest_ram_rec *rec;
+ char who[80];
+ const char *name = (vgpu && vgpu->mdev) ?
+ dev_name(&vgpu->mdev->dev) : "?";
+ bool fresh = false;
+ int ret;
+
+ if (!vgpu)
+ return -EINVAL;
+
+ mutex_lock(&guest_ram_lock);
+ rec = gsp_phys_guest_ram_rec_find_locked(vgpu);
+ if (!rec) {
+ rec = gsp_phys_guest_ram_rec_alloc_locked(vgpu);
+ if (!rec) {
+ mutex_unlock(&guest_ram_lock);
+ pr_err("vgsp: guest RAM: no free per-mdev record slot (max %u); refusing mdev %s\n",
+ GSP_PHYS_GUEST_RAM_MDEVS, name);
+ return -ENOSPC;
+ }
+ fresh = true;
+ }
+ mutex_unlock(&guest_ram_lock);
+
+ scnprintf(who, sizeof(who), " for mdev %s", name);
+ ret = gsp_phys_guest_ram_rec_configure(rec, path, gpa_base,
+ file_offset, size, who);
+
+ /*
+ * A fresh record whose first configure failed must not hold the slot
+ * (the vgpu pointer would otherwise be reserved with no file and the
+ * next attempt could not create a clean record).
+ */
+ if (ret && fresh) {
+ mutex_lock(&guest_ram_lock);
+ if (rec->file)
+ ret = 0; /* raced with another configure; keep it */
+ else
+ rec->vgpu = NULL;
+ mutex_unlock(&guest_ram_lock);
+ }
+ return ret;
+}
+EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_configure_vgpu);
+
+/* Lock held: close one record's file and drop its ranges. */
+static void gsp_phys_guest_ram_rec_close(struct gsp_phys_guest_ram_rec *rec,
+ const char *release_log)
+{
+ if (rec->file) {
+ pr_info("%s: path=%s ranges=%u\n", release_log,
+ rec->path, rec->ranges_n);
+ fput(rec->file);
+ rec->file = NULL;
+ }
+ rec->path[0] = '\0';
+ memset(rec->ranges, 0, sizeof(rec->ranges));
+ rec->ranges_n = 0;
+}
+
+/*
+ * Close every configured guest RAM record (legacy and per-mdev) and drop
+ * every range. Called from the physical-GPU release path so module unload
+ * and a backend release end the file references; a fresh cycle reconfigures
+ * from debugfs. The legacy release log line is unchanged.
*/
void gsp_phys_guest_ram_clear(void)
{
+ unsigned int i;
+
mutex_lock(&guest_ram_lock);
- if (guest_ram_file) {
- pr_info("vgsp: guest RAM released: path=%s ranges=%u\n",
- guest_ram_path, guest_ram_ranges_n);
- fput(guest_ram_file);
- guest_ram_file = NULL;
- }
- guest_ram_path[0] = '\0';
- memset(guest_ram_ranges, 0, sizeof(guest_ram_ranges));
- guest_ram_ranges_n = 0;
+ gsp_phys_guest_ram_rec_close(&guest_ram_legacy,
+ "vgsp: guest RAM released");
+ for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) {
+ gsp_phys_guest_ram_rec_close(&guest_ram_mdevs[i],
+ "vgsp: guest RAM released for mdev");
+ guest_ram_mdevs[i].vgpu = NULL;
+ }
mutex_unlock(&guest_ram_lock);
}
EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_clear);
+/*
+ * Release the guest RAM record bound to one mdev. Called from the mdev
+ * cleanup path so the file reference and the vgpu pointer never outlive the
+ * session (a removed mdev's translations must stop, and the record slot must
+ * free up for a restarted guest).
+ */
+void gsp_phys_guest_ram_vgpu_clear(struct nv_vgpu_device *vgpu)
+{
+ unsigned int i;
+
+ if (!vgpu)
+ return;
+ mutex_lock(&guest_ram_lock);
+ for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++) {
+ if (guest_ram_mdevs[i].vgpu == vgpu) {
+ gsp_phys_guest_ram_rec_close(&guest_ram_mdevs[i],
+ "vgsp: guest RAM released for mdev");
+ guest_ram_mdevs[i].vgpu = NULL;
+ }
+ }
+ mutex_unlock(&guest_ram_lock);
+}
+EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_vgpu_clear);
+
bool gsp_phys_guest_ram_active(void)
{
- bool active;
+ bool active = false;
+ unsigned int i;
mutex_lock(&guest_ram_lock);
- active = guest_ram_file != NULL;
+ if (guest_ram_legacy.file)
+ active = true;
+ for (i = 0; !active && i < GSP_PHYS_GUEST_RAM_MDEVS; i++)
+ if (guest_ram_mdevs[i].file)
+ active = true;
mutex_unlock(&guest_ram_lock);
return active;
}
@@ -1427,33 +1641,78 @@ EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_active);
/*
* Lock-safe one-line snapshot for the gsp-phys-state dump: active flag,
- * path, range count and every range (gpa/offset/size). The caller supplies
- * the buffer; nothing is written when the file is not configured beyond the
- * active=0 marker.
+ * path, range count and every range (gpa/offset/size). With only the legacy
+ * record configured (the one-mdev campaign) the output is byte-identical to
+ * the pre-dual-guest format; a per-mdev record adds its mdev UUID so a
+ * dual-guest operator can tell the records apart.
*/
+static void gsp_phys_guest_ram_rec_state(char *buf, size_t len,
+ unsigned int *n,
+ const struct gsp_phys_guest_ram_rec *rec,
+ const char *label)
+{
+ unsigned int i;
+
+ if (*n >= len)
+ return;
+ *n += scnprintf(buf + *n, len - *n, "%s path=%s ranges=%u",
+ label, rec->path, rec->ranges_n);
+ for (i = 0; i < rec->ranges_n && *n < len; i++)
+ *n += scnprintf(buf + *n, len - *n,
+ " gpa=0x%llx off=0x%llx size=0x%llx",
+ rec->ranges[i].gpa_base,
+ rec->ranges[i].file_offset,
+ rec->ranges[i].size);
+}
+
void gsp_phys_guest_ram_state(char *buf, size_t len)
{
unsigned int i, n = 0;
+ bool any_mdev = false;
if (!buf || !len)
return;
buf[0] = '\0';
mutex_lock(&guest_ram_lock);
- if (!guest_ram_file) {
- scnprintf(buf, len, "active=0");
+ for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS; i++)
+ if (guest_ram_mdevs[i].file) {
+ any_mdev = true;
+ break;
+ }
+
+ if (!any_mdev) {
+ /*
+ * Legacy-only (or nothing): keep the historical shape - the
+ * campaign's parser (if any) sees the same text as before.
+ */
+ if (!guest_ram_legacy.file) {
+ scnprintf(buf, len, "active=0");
+ mutex_unlock(&guest_ram_lock);
+ return;
+ }
+ n += scnprintf(buf + n, len - n, "active=1");
+ gsp_phys_guest_ram_rec_state(buf, len, &n,
+ &guest_ram_legacy, "");
mutex_unlock(&guest_ram_lock);
return;
}
- n += scnprintf(buf + n, len - n, "active=1 path=%s ranges=%u",
- guest_ram_path, guest_ram_ranges_n);
- for (i = 0; i < guest_ram_ranges_n && n < len; i++)
- n += scnprintf(buf + n, len - n,
- " gpa=0x%llx off=0x%llx size=0x%llx",
- guest_ram_ranges[i].gpa_base,
- guest_ram_ranges[i].file_offset,
- guest_ram_ranges[i].size);
+ n += scnprintf(buf + n, len - n, "active=1");
+ if (guest_ram_legacy.file)
+ gsp_phys_guest_ram_rec_state(buf, len, &n, &guest_ram_legacy,
+ " legacy");
+ for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS && n < len; i++) {
+ char label[64];
+
+ if (!guest_ram_mdevs[i].file)
+ continue;
+ scnprintf(label, sizeof(label), " mdev=%s",
+ (guest_ram_mdevs[i].vgpu && guest_ram_mdevs[i].vgpu->mdev) ?
+ dev_name(&guest_ram_mdevs[i].vgpu->mdev->dev) : "?");
+ gsp_phys_guest_ram_rec_state(buf, len, &n,
+ &guest_ram_mdevs[i], label);
+ }
mutex_unlock(&guest_ram_lock);
}
EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_state);
@@ -1474,12 +1733,20 @@ EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_state);
* an unmapped hole or an MMIO frame can never be handed to the engine as
* guest RAM.
*
- * Guards kept from the original path: the active-mdev walk
+ * Guards kept from the original path: the mdev walk
* (nv_vgpu_driver.lock for the bounded list scan) gates every translation -
- * the engine path must only run with a live guest session; no such session
- * returns -ENODEV. The IOMMU-domain query remains as a documented fallback
- * for a setup whose mdev has a real domain (it is inert with the
- * emulated-IOMMU mdev used here).
+ * the engine path must only run with a live guest session; a vgpu given by
+ * the caller must itself be open, and no open session at all returns
+ * -ENODEV. The IOMMU-domain query remains as a documented fallback for a
+ * setup whose mdev has a real domain (it is inert with the emulated-IOMMU
+ * mdev used here).
+ *
+ * Per-mdev resolution (dual-guest goal 1.0): the caller names the owning
+ * vgpu (a captured forwarding request, a held release, ...). The vgpu's own
+ * record resolves the translation; the legacy record (configured by the
+ * historical top-level debugfs write) still serves the FIRST open mdev so
+ * single-guest scripts are unchanged. vgpu == NULL keeps the historical
+ * resolution: the first open mdev.
*
* Locking: guest_ram_lock is held across the range walk and the
* read_cache_page() so a concurrent configure/clear cannot change the file
@@ -1489,13 +1756,15 @@ EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_state);
* own lifetime guarantees for an open fd. No guest or rm lock may be held by
* the caller.
*/
-int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out)
+int gsp_phys_gpa_to_hpa_vgpu(struct nv_vgpu_device *vgpu, u64 gpa, u64 *hpa_out)
{
- struct nv_vgpu_device *vgpu = NULL;
struct nv_vgpu_device *d;
+ struct nv_vgpu_device *first = NULL;
+ struct gsp_phys_guest_ram_rec *rec = NULL;
struct iommu_domain *dom;
phys_addr_t p;
unsigned int i;
+ bool vgpu_gone = false;
if (!hpa_out)
return -EINVAL;
@@ -1503,19 +1772,28 @@ int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out)
mutex_lock(&nv_vgpu_driver.lock);
list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
if (atomic_read(&d->vfio_dev_open_count) > 0) {
- vgpu = d;
+ first = d;
break;
}
}
+ if (vgpu && atomic_read(&vgpu->vfio_dev_open_count) <= 0)
+ vgpu_gone = true; /* named session is gone: never retarget */
mutex_unlock(&nv_vgpu_driver.lock);
+ if (vgpu_gone)
+ return -ENODEV;
+ if (!vgpu)
+ vgpu = first; /* historical resolution: first open mdev */
if (!vgpu)
return -ENODEV;
mutex_lock(&guest_ram_lock);
- if (guest_ram_file) {
- for (i = 0; i < guest_ram_ranges_n; i++) {
+ rec = gsp_phys_guest_ram_rec_find_locked(vgpu);
+ if (!rec && vgpu == first)
+ rec = &guest_ram_legacy; /* legacy file: first/only mdev */
+ if (rec && rec->file) {
+ for (i = 0; i < rec->ranges_n; i++) {
const struct gsp_phys_guest_ram_range *r =
- &guest_ram_ranges[i];
+ &rec->ranges[i];
struct page *page;
u64 file_off;
@@ -1524,7 +1802,7 @@ int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out)
continue;
file_off = r->file_offset + (gpa - r->gpa_base);
- page = read_cache_page(guest_ram_file->f_mapping,
+ page = read_cache_page(rec->file->f_mapping,
file_off >> PAGE_SHIFT,
NULL, NULL);
mutex_unlock(&guest_ram_lock);
@@ -1560,6 +1838,18 @@ int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out)
*hpa_out = (u64)p;
return 0;
}
+EXPORT_SYMBOL_GPL(gsp_phys_gpa_to_hpa_vgpu);
+
+/*
+ * Historical entry point: resolve through the first open mdev (and that
+ * mdev's own record, else the legacy record). Used by the single-guest
+ * boot-ladder guest-GPA pin (test-only; see gsp_phys_guest_gpa_write) and
+ * kept as the compatibility wrapper for callers without a request context.
+ */
+int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out)
+{
+ return gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, hpa_out);
+}
EXPORT_SYMBOL_GPL(gsp_phys_gpa_to_hpa);
/*
diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h
index 84b56d6..a89493b 100644
--- a/kernel/gsp_phys_gpu.h
+++ b/kernel/gsp_phys_gpu.h
@@ -3,6 +3,8 @@
#include <linux/types.h>
+struct nv_vgpu_device;
+
/*
* Physical GPU ownership (real-GSP execution backend, bring-up rung 1).
* Claim is disabled unless the gsp_phys_gpu module parameter is set.
@@ -433,21 +435,50 @@ bool gsp_phys_rmcall_vas_shared_value(void);
int gsp_phys_gpa_to_hpa(u64 gpa, u64 *hpa_out);
/*
+ * Per-mdev variant: translate the GPA for the named owning vgpu (a captured
+ * forwarding request, a held release). The vgpu's own guest-RAM record is
+ * authoritative; the legacy record still serves the FIRST open mdev so
+ * single-guest callers are unchanged. NULL keeps the historical resolution
+ * (first open mdev). A named vgpu whose session is closed returns -ENODEV
+ * rather than being retargeted at another guest.
+ */
+int gsp_phys_gpa_to_hpa_vgpu(struct nv_vgpu_device *vgpu, u64 gpa,
+ u64 *hpa_out);
+
+/*
* Guest RAM backing-file configuration (debugfs gsp-phys-guest-ram). One
- * shared file per module cycle, up to four GPA->file-offset ranges keyed by
+ * record per guest session, up to four GPA->file-offset ranges keyed by
* gpa_base. The first configure opens the file; the same path adds/replaces
* ranges, a different path is refused -EBUSY, and the ranges are dropped and
- * the file closed by gsp_phys_guest_ram_clear() on physical-GPU release. The
+ * the file closed by gsp_phys_guest_ram_clear() on physical-GPU release (or
+ * gsp_phys_guest_ram_vgpu_clear() when one mdev is removed). The
+ * gsp_phys_guest_ram_configure() entry point configures the legacy record
+ * (the historical top-level debugfs write: serves the first/only open mdev);
+ * gsp_phys_guest_ram_configure_vgpu() binds the record to one mdev. The
* state accessor is lock-safe and writes "active=0" or the active path,
* range count and per-range gpa/off/size into the caller's buffer.
*/
int gsp_phys_guest_ram_configure(const char *path, u64 gpa_base,
u64 file_offset, u64 size);
+int gsp_phys_guest_ram_configure_vgpu(struct nv_vgpu_device *vgpu,
+ const char *path, u64 gpa_base,
+ u64 file_offset, u64 size);
void gsp_phys_guest_ram_clear(void);
+void gsp_phys_guest_ram_vgpu_clear(struct nv_vgpu_device *vgpu);
bool gsp_phys_guest_ram_active(void);
void gsp_phys_guest_ram_state(char *buf, size_t len);
/*
+ * Open-mdev helpers for the per-mdev debugfs selector and the single-guest
+ * guards: count the mdevs whose VFIO device is open, or resolve one open
+ * mdev by its device name (the mdev UUID). The returned device is not
+ * reference-counted; the caller must only use it while that mdev's fd is
+ * known to be open.
+ */
+int gsp_phys_open_mdev_count(void);
+struct nv_vgpu_device *gsp_phys_open_mdev_by_name(const char *name);
+
+/*
* Domain shape of the physical (claimed) device, as the P6a engine path
* needs it: the module programs raw host PAs into the CE page tables, so the
* engine may only run when the physical device's DMA addresses ARE host PAs.
diff --git a/kernel/gsp_rm_emulation.c b/kernel/gsp_rm_emulation.c
index 8318f56..2ffc8ac 100644
--- a/kernel/gsp_rm_emulation.c
+++ b/kernel/gsp_rm_emulation.c
@@ -2357,6 +2357,7 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object,
return;
memset(&desc, 0, sizeof(desc));
+ desc.vgpu = ctx->vgpu;
desc.h_object = h_object;
memcpy(&desc.gpfifo_va, params + GSPRM_CHAN_GPFIFO_OFFSET, 8);
memcpy(&desc.gpfifo_entries, params + GSPRM_CHAN_GPFIFO_ENTRIES, 4);
@@ -3244,7 +3245,7 @@ static void gsp_rm_handle_free(struct gsp_emu_context *ctx, const u8 *elem,
memcpy(&hObjOld, params + 8, 4);
memset(params + 12, 0, 4); /* status = NV_OK */
- gsp_sema_service_channel_del(hObjOld);
+ gsp_sema_service_channel_del(ctx->vgpu, hObjOld);
/*
* Retire event records that refer to the object being freed. Posting an
diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c
index baac8d4..31d5d82 100644
--- a/kernel/gsp_sema_service.c
+++ b/kernel/gsp_sema_service.c
@@ -162,6 +162,10 @@ struct sema_map_entry {
struct svc_channel {
bool valid;
+ struct nv_vgpu_device *vgpu; /* owning mdev (guest session); the
+ * table is keyed by (vgpu, h_object)
+ * because two guests' stock allocators
+ * choose identical handles */
u32 h_object;
u32 engine_type;
u32 cid; /* RM channel id (doorbell token vector) */
@@ -449,12 +453,65 @@ static u32 svc_ring_n;
* client spins). One clean boot always works, so make the next session look
* like one: when the channel set empties, request a flush and let the next
* rebuild start from the same empty state as a fresh boot.
+ *
+ * Dual-guest staging (goal 1.0): the discovery tables are still shared
+ * between guests (per-guest learned tables are the staged G4 remainder), so
+ * the flush is requested only when the channel set is globally empty. The
+ * active decision is made per guest (svc_guests below): one guest closing
+ * its last channel never flushes while another guest still has live
+ * channels - otherwise the departing guest would wipe the live guest's
+ * pins/maps and stall it. Single-guest behaviour is unchanged: the one
+ * guest's last-channel close is also the global empty transition.
*/
static u8 svc_sess_flush;
static u8 svc_sess_active; /* a channel of a session has registered (until flush) */
static struct completion svc_sess_flush_done; /* a pending flush has executed */
static unsigned long svc_ring_done; /* CE ranks whose ring is pinned (session state) */
+/* Per-guest session records (one per open mdev; see SVC_MAX_CHANNELS for the channel cap). */
+#define SVC_MAX_GUESTS 8
+struct svc_guest_sess {
+ struct nv_vgpu_device *vgpu; /* NULL = free slot */
+ bool active; /* a channel of this guest is
+ * registered until this guest's
+ * last channel closes */
+ u64 last_build; /* per-guest svc_rebuild freshness */
+};
+static struct svc_guest_sess svc_guests[SVC_MAX_GUESTS];
+static DEFINE_SPINLOCK(svc_guest_lock);
+
+/*
+ * Find (or, with create, allocate) the session record for one mdev. The
+ * records are pointer-keyed; a removed mdev's record is freed by
+ * gsp_sema_service_vgpu_cleanup() so the pointer can never match a newly
+ * allocated device.
+ */
+static struct svc_guest_sess *svc_guest_get(struct nv_vgpu_device *vgpu,
+ bool create)
+{
+ struct svc_guest_sess *sess = NULL;
+ unsigned long flags;
+ int i;
+
+ if (!vgpu)
+ return NULL;
+ spin_lock_irqsave(&svc_guest_lock, flags);
+ for (i = 0; i < SVC_MAX_GUESTS; i++) {
+ if (svc_guests[i].vgpu == vgpu) {
+ sess = &svc_guests[i];
+ break;
+ }
+ if (create && !sess && !svc_guests[i].vgpu)
+ sess = &svc_guests[i];
+ }
+ if (sess && create && !sess->vgpu) {
+ memset(sess, 0, sizeof(*sess));
+ sess->vgpu = vgpu;
+ }
+ spin_unlock_irqrestore(&svc_guest_lock, flags);
+ return sess;
+}
+
/*
* CeUtils scrubber channel (the guest PMA's internal CE channel created at
* driver init). When the PMA must hand out scrubbed pages it waits on the
@@ -476,6 +533,7 @@ static unsigned long svc_ring_done; /* CE ranks whose ring is pinned (session st
* block count (so the scrub wait completes). Idempotent, forward-only.
*/
struct svc_scrub {
+ struct nv_vgpu_device *vgpu; /* owning mdev (NULL = free slot) */
bool userd_ok; /* a 4096-entry channel was registered */
u32 h_object; /* handle of the registered scrubber channel */
u64 userd_pa;
@@ -492,7 +550,34 @@ struct svc_scrub {
u32 pa_gen; /* scanner registration generation (logged) */
u32 log_budget;
};
-static struct svc_scrub svc_scrub;
+/*
+ * One scrubber record per guest: every guest's PMA has its own 4096-entry
+ * scrubber channel, and the authored words must be written through the
+ * owning guest's address space (writing guest A's PA through guest B's VFIO
+ * device would corrupt B). Keyed by vgpu pointer, freed by
+ * gsp_sema_service_vgpu_cleanup().
+ */
+static struct svc_scrub svc_scrubs[SVC_MAX_GUESTS];
+
+static struct svc_scrub *svc_scrub_get(struct nv_vgpu_device *vgpu, bool create)
+{
+ struct svc_scrub *s = NULL;
+ int i;
+
+ if (!vgpu)
+ return NULL;
+ for (i = 0; i < SVC_MAX_GUESTS; i++) {
+ if (svc_scrubs[i].vgpu == vgpu)
+ return &svc_scrubs[i];
+ if (create && !s && !svc_scrubs[i].vgpu)
+ s = &svc_scrubs[i];
+ }
+ if (s) {
+ memset(s, 0, sizeof(*s));
+ s->vgpu = vgpu;
+ }
+ return s;
+}
static u8 svc_pb_passes;
static u64 svc_pb_last;
/*
@@ -1066,7 +1151,8 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
}
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid && svc.chan[i].h_object == desc->h_object) {
+ if (svc.chan[i].valid && svc.chan[i].vgpu == desc->vgpu &&
+ svc.chan[i].h_object == desc->h_object) {
ch = &svc.chan[i];
break;
}
@@ -1079,7 +1165,8 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
u64 keep_release_pa = 0;
u8 keep_gr_index = 0, keep_ce_index = 0;
DECLARE_BITMAP(keep_rel_done, SVC_GPFIFO_MAX_BITS);
- bool same = ch->valid && ch->h_object == desc->h_object;
+ bool same = ch->valid && ch->vgpu == desc->vgpu &&
+ ch->h_object == desc->h_object;
/*
* A re-registration of the same channel must not lose the
@@ -1102,18 +1189,22 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
bitmap_copy(ch->rel_done, keep_rel_done,
SVC_GPFIFO_MAX_BITS);
/*
- * Rank among the live CE channels. The RM hands out object
- * handles in creation order and the guest's UVM allocates one
- * push buffer, ring and tracking semaphore per channel in that
- * same order, so the rank is what ties a channel to its ring
- * offset and its push-buffer base (see svc_ring_autopin /
- * svc_pb_pair).
+ * Rank among the live CE channels OF THIS GUEST. The RM hands
+ * out object handles in creation order and the guest's UVM
+ * allocates one push buffer, ring and tracking semaphore per
+ * channel in that same order, so the rank is what ties a
+ * channel to its ring offset and its push-buffer base (see
+ * svc_ring_autopin / svc_pb_pair). The rank must not count the
+ * other guest's channels: their handles are independent and
+ * their allocators interleave.
*/
if (desc->engine_type) {
u8 rank = 0;
for (k = 0; k < SVC_MAX_CHANNELS; k++)
- if (svc.chan[k].valid && svc.chan[k].engine_type &&
+ if (svc.chan[k].valid &&
+ svc.chan[k].vgpu == desc->vgpu &&
+ svc.chan[k].engine_type &&
svc.chan[k].h_object < desc->h_object)
rank++;
ch->ce_index = rank;
@@ -1121,13 +1212,16 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
u8 rank = 0;
for (k = 0; k < SVC_MAX_CHANNELS; k++)
- if (svc.chan[k].valid && !svc.chan[k].engine_type &&
+ if (svc.chan[k].valid &&
+ svc.chan[k].vgpu == desc->vgpu &&
+ !svc.chan[k].engine_type &&
svc.chan[k].h_object < desc->h_object)
rank++;
ch->gr_index = rank;
}
ch->valid = true;
svc_sess_active = 1;
+ ch->vgpu = desc->vgpu;
ch->h_object = desc->h_object;
ch->engine_type = desc->engine_type;
ch->cid = desc->cid;
@@ -1145,6 +1239,14 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
ch->gr_index = keep_gr_index;
ch->ce_index = keep_ce_index;
}
+ /* Per-guest session flag: this guest has a live channel. */
+ {
+ struct svc_guest_sess *sess =
+ svc_guest_get(desc->vgpu, true);
+
+ if (sess)
+ sess->active = true;
+ }
}
spin_unlock_irqrestore(&svc.chan_lock, flags);
@@ -1169,25 +1271,30 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
*/
if (desc->gpfifo_entries >= 4096 && desc->gpfifo_va &&
desc->userd_pa) {
+ struct svc_scrub *s = svc_scrub_get(desc->vgpu, true);
+
+ if (!s)
+ goto scrub_done; /* table full: max guests */
/* Re-seed the counting on a fresh scrubber object. */
- if (!svc_scrub.userd_ok ||
- svc_scrub.h_object != desc->h_object ||
- svc_scrub.gpfifo_va != desc->gpfifo_va) {
- svc_scrub.seeded = false;
- svc_scrub.submitted = 0;
+ if (!s->userd_ok ||
+ s->h_object != desc->h_object ||
+ s->gpfifo_va != desc->gpfifo_va) {
+ s->seeded = false;
+ s->submitted = 0;
}
- svc_scrub.userd_ok = true;
- svc_scrub.h_object = desc->h_object;
- svc_scrub.userd_pa = desc->userd_pa;
- svc_scrub.userd_as = desc->userd_as ? desc->userd_as
- : SVC_APERTURE_SYS;
- svc_scrub.gpfifo_va = desc->gpfifo_va;
- svc_scrub.entries = desc->gpfifo_entries;
+ s->userd_ok = true;
+ s->h_object = desc->h_object;
+ s->userd_pa = desc->userd_pa;
+ s->userd_as = desc->userd_as ? desc->userd_as
+ : SVC_APERTURE_SYS;
+ s->gpfifo_va = desc->gpfifo_va;
+ s->entries = desc->gpfifo_entries;
pr_info("sema-svc: scrubber channel 0x%x detected (gpfifo=0x%llx entries=%u userd=0x%llx as=%u)\n",
desc->h_object, desc->gpfifo_va,
desc->gpfifo_entries, desc->userd_pa,
- svc_scrub.userd_as);
+ s->userd_as);
}
+scrub_done:
pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx\n",
desc->h_object, desc->gpfifo_va, desc->gpfifo_entries,
desc->userd_pa, desc->engine_type, desc->cid,
@@ -1199,7 +1306,7 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc)
}
/* P6b wave-1.5 hold state is defined with the forwarding section below. */
-static void svc_fwd_hold_abort(u32 h_object);
+static void svc_fwd_hold_abort(struct nv_vgpu_device *vgpu, u32 h_object);
/*
* P6b wave-2 inline software fallback for a withheld (skipped) candidate;
* defined with the forwarding section below. Service-thread only.
@@ -1218,7 +1325,7 @@ static void svc_opdef_drop(struct nv_vgpu_device *vgpu,
struct svc_channel *ch, u32 entry_idx);
static void svc_opdef_reset_all(void);
-void gsp_sema_service_channel_del(u32 h_object)
+void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object)
{
unsigned long flags;
int i;
@@ -1228,7 +1335,8 @@ void gsp_sema_service_channel_del(u32 h_object)
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid && svc.chan[i].h_object == h_object) {
+ if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu &&
+ svc.chan[i].h_object == h_object) {
svc.chan[i].valid = false;
pr_info("sema-svc: channel 0x%x unregistered (submissions=%llu)\n",
h_object, svc.chan[i].submissions);
@@ -1238,19 +1346,27 @@ void gsp_sema_service_channel_del(u32 h_object)
/*
* ISSUE-0002: when the last channel closes, the next client is a new
* session in a new UVM VA space; its discovery state must not inherit
- * this session's map/pins/page-directory latch. Request the flush
- * here; it executes at the next rebuild (the only writer of that
- * state) just like a fresh boot would start empty. Only the transition
- * from "a session existed" to "empty" may request it: redundant
- * deletes for handles that never registered would otherwise re-flush
- * (and wipe) a session that started after the real one ended.
+ * this session's map/pins/page-directory latch. The active decision is
+ * per guest (svc_guests) and the flush request is raised only when NO
+ * guest has a live channel left: the discovery tables are still shared
+ * between guests (staged G4), so flushing while another guest is live
+ * would wipe that guest's pins and stall it. One guest: the last-channel
+ * close IS the global empty transition, so behaviour is unchanged.
*/
if (svc_sess_active) {
+ struct svc_guest_sess *sess = svc_guest_get(vgpu, false);
+ bool any = false, own = false;
+
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid)
- break;
+ if (!svc.chan[i].valid)
+ continue;
+ any = true;
+ if (svc.chan[i].vgpu == vgpu)
+ own = true;
}
- if (i == SVC_MAX_CHANNELS) {
+ if (sess && !own)
+ sess->active = false;
+ if (!any) {
svc_sess_active = 0;
svc_sess_flush = 1;
}
@@ -1265,9 +1381,10 @@ void gsp_sema_service_channel_del(u32 h_object)
* P6b wave 1.5: a deleted channel's held releases must not wait out
* the timeout; mark the hold aborted so the pump applies them (or
* drops them if the snapshot already lost the channel) on its next
- * pass.
+ * pass. The hold is matched per (vgpu, handle): the other guest's
+ * identical handle must not abort this guest's hold.
*/
- svc_fwd_hold_abort(h_object);
+ svc_fwd_hold_abort(vgpu, h_object);
}
/*
@@ -5078,7 +5195,8 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h,
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid && svc.chan[i].h_object == h) {
+ if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu &&
+ svc.chan[i].h_object == h) {
/*
* The release targets the slot the push itself named
* (release_pa learned from the release VA); the
@@ -5150,6 +5268,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h,
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
if (svc.chan[i].valid &&
+ svc.chan[i].vgpu == vgpu &&
svc.chan[i].h_object == h) {
if (payload >
svc.chan[i].pending_rel_pay) {
@@ -5229,7 +5348,7 @@ static void svc_counter_release_apply(struct nv_vgpu_device *vgpu, u32 h,
* harmless for an absolute release (its guard drops it), but an increment
* would advance the slot twice.
*/
-static void svc_counter_pending_clear(u32 h)
+static void svc_counter_pending_clear(struct nv_vgpu_device *vgpu, u32 h)
{
unsigned long flags;
int i;
@@ -5238,7 +5357,8 @@ static void svc_counter_pending_clear(u32 h)
return;
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid && svc.chan[i].h_object == h) {
+ if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu &&
+ svc.chan[i].h_object == h) {
svc.chan[i].pending_rel_pay = 0;
svc.chan[i].pending_rel_inc = 0;
break;
@@ -5272,13 +5392,13 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu,
if (payload > 0x100000) {
ch->pending_rel_pay = 0;
ch->pending_rel_inc = 0;
- svc_counter_pending_clear(ch->h_object);
+ svc_counter_pending_clear(ch->vgpu, ch->h_object);
return;
}
} else if (payload <= cur || payload - cur > 0x100000) {
ch->pending_rel_pay = 0;
ch->pending_rel_inc = 0;
- svc_counter_pending_clear(ch->h_object);
+ svc_counter_pending_clear(ch->vgpu, ch->h_object);
if (cur > ch->counter_last)
ch->counter_last = cur;
return;
@@ -5289,7 +5409,7 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu,
return;
ch->pending_rel_pay = 0;
ch->pending_rel_inc = 0;
- svc_counter_pending_clear(ch->h_object);
+ svc_counter_pending_clear(ch->vgpu, ch->h_object);
if (nv > ch->counter_last)
ch->counter_last = nv;
svc.releases_applied++;
@@ -5428,6 +5548,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu,
ch->counter_pa = ipa & ~0x3ULL;
for (j = 0; j < SVC_MAX_CHANNELS; j++) {
if (svc.chan[j].valid &&
+ svc.chan[j].vgpu == vgpu &&
svc.chan[j].h_object == ch->h_object) {
svc.chan[j].counter_pa =
ch->counter_pa;
@@ -5476,6 +5597,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu,
spin_lock_irqsave(&svc.chan_lock, flags);
for (j = 0; j < SVC_MAX_CHANNELS; j++) {
if (svc.chan[j].valid &&
+ svc.chan[j].vgpu == vgpu &&
svc.chan[j].h_object != ch->h_object &&
svc.chan[j].counter_pa == pa) {
collision = true;
@@ -5501,6 +5623,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu,
ch->counter_pa = pa;
for (j = 0; j < SVC_MAX_CHANNELS; j++) {
if (svc.chan[j].valid &&
+ svc.chan[j].vgpu == vgpu &&
svc.chan[j].h_object == ch->h_object) {
svc.chan[j].counter_pa = pa;
break;
@@ -5560,7 +5683,8 @@ static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va)
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid && svc.chan[i].h_object == h) {
+ if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu &&
+ svc.chan[i].h_object == h) {
if (svc.chan[i].release_pa != (pa & ~0x3ULL)) {
svc.chan[i].release_pa = pa & ~0x3ULL;
svc.chan[i].relearn_n++;
@@ -7694,7 +7818,8 @@ static void svc_pb_discover(struct nv_vgpu_device *vgpu)
u32 slot, k;
bool have = false;
- if (!svc.chan[i].valid || !svc.chan[i].engine_type)
+ if (!svc.chan[i].valid || svc.chan[i].vgpu != vgpu ||
+ !svc.chan[i].engine_type)
continue;
ce = true;
if (svc.chan[i].ce_index >= 32)
@@ -8975,7 +9100,8 @@ static void svc_gr_content_discover(struct nv_vgpu_device *vgpu)
int pap = 0;
u32 r = c->gr_index;
- if (!c->valid || c->engine_type || r >= SVC_GR_CONTENT_RANKS)
+ if (!c->valid || c->vgpu != vgpu || c->engine_type ||
+ r >= SVC_GR_CONTENT_RANKS)
continue;
if (test_bit(r, &svc_gr_ring_done))
continue;
@@ -9316,7 +9442,8 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu,
*/
spin_lock_irqsave(&svc.chan_lock, flags);
for (k = 0; k < SVC_MAX_CHANNELS; k++)
- if (svc.chan[k].valid && svc.chan[k].engine_type &&
+ if (svc.chan[k].valid && svc.chan[k].vgpu == vgpu &&
+ svc.chan[k].engine_type &&
svc.chan[k].ce_index + 1 > max_ce)
max_ce = svc.chan[k].ce_index + 1;
spin_unlock_irqrestore(&svc.chan_lock, flags);
@@ -10792,7 +10919,7 @@ static int svc_fwd_operand_hpa(struct nv_vgpu_device *vgpu, u8 kind,
}
if (kind != SVC_APERTURE_SYS)
return -EINVAL;
- ret = gsp_phys_gpa_to_hpa(page, hpa);
+ ret = gsp_phys_gpa_to_hpa_vgpu(vgpu, page, hpa);
if (ret)
return ret;
*out_off = page_off;
@@ -10961,9 +11088,10 @@ struct svc_fwd_hold {
static struct svc_fwd_hold svc_fwd_hold;
-static void svc_fwd_hold_abort(u32 h_object)
+static void svc_fwd_hold_abort(struct nv_vgpu_device *vgpu, u32 h_object)
{
if (READ_ONCE(svc_fwd_hold.active) &&
+ svc_fwd_hold.vgpu == vgpu &&
svc_fwd_hold.ch_handle == h_object)
WRITE_ONCE(svc_fwd_hold.aborted, true);
}
@@ -11334,18 +11462,22 @@ static void svc_opdef_drop_stale(struct nv_vgpu_device *vgpu,
}
}
-/* A slot whose channel is gone from this pass's snapshot: drop it. */
-static void svc_opdef_drop_gone(const struct svc_channel *local, int n)
+/* A slot of THIS pump's guest whose channel is gone from the pass snapshot:
+ * drop it. Slots of other guests are never touched (their channels are not in
+ * this snapshot by construction). */
+static void svc_opdef_drop_gone(struct nv_vgpu_device *vgpu,
+ const struct svc_channel *local, int n)
{
int i, k;
for (i = 0; i < SVC_OPDEF_SLOTS; i++) {
struct svc_opdef_ent *e = &svc_opdef[i];
- if (!e->active)
+ if (!e->active || e->vgpu != vgpu)
continue;
for (k = 0; k < n; k++)
- if (local[k].h_object == e->h_object)
+ if (local[k].vgpu == e->vgpu &&
+ local[k].h_object == e->h_object)
break;
if (k == n)
svc_opdef_clear(e);
@@ -11787,6 +11919,31 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu,
}
/*
+ * Is this mdev still on the driver list with an open VFIO device? Decides
+ * whether a hold that names it can still be resolved through it: a removed
+ * device must never be dereferenced (its struct can be freed before the next
+ * pump pass would see the hold). Pointer comparisons only - safe to call
+ * with a possibly-stale pointer.
+ */
+static bool svc_vgpu_open(const struct nv_vgpu_device *vgpu)
+{
+ struct nv_vgpu_device *d;
+ bool open = false;
+
+ if (!vgpu)
+ return false;
+ mutex_lock(&nv_vgpu_driver.lock);
+ list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
+ if (d == vgpu && atomic_read(&d->vfio_dev_open_count) > 0) {
+ open = true;
+ break;
+ }
+ }
+ mutex_unlock(&nv_vgpu_driver.lock);
+ return open;
+}
+
+/*
* Channel vanished while a hold was active. Any resolved hold (completed,
* aborted, failed or expired) applies the recorded releases through a shim
* channel - holds exist only for CE channels and the vanished channel object
@@ -11806,6 +11963,18 @@ static void svc_fwd_hold_gone_finish(struct nv_vgpu_device *vgpu)
bool done = smp_load_acquire(&svc_fwd_req.done);
bool aborted = READ_ONCE(svc_fwd_hold.aborted);
+ if (svc_fwd_hold.vgpu && !svc_vgpu_open(svc_fwd_hold.vgpu)) {
+ /*
+ * The owning mdev was removed mid-hold (the mdev cleanup path
+ * already released its session state): its guest can never
+ * observe the releases and dereferencing the device would be a
+ * use-after-free. Drop the hold; the request slot frees for the
+ * next guest.
+ */
+ svc.fwd_hold_canceled++;
+ svc_fwd_hold_clear();
+ return;
+ }
if (!done && !aborted && !svc_fwd_hold_expired())
return;
/*
@@ -11858,6 +12027,15 @@ static void svc_fwd_hold_stop_flush(struct nv_vgpu_device *vgpu)
if (!svc_fwd_hold.active)
return;
+ /*
+ * A hold whose owning mdev is already gone must never be dereferenced:
+ * drop it (the guest cannot observe the releases anyway).
+ */
+ if (svc_fwd_hold.vgpu && !svc_vgpu_open(svc_fwd_hold.vgpu)) {
+ svc.fwd_hold_canceled++;
+ svc_fwd_hold_clear();
+ return;
+ }
done = smp_load_acquire(&svc_fwd_req.done);
/*
* P6b wave 2: the backend is going away; a skipped candidate whose
@@ -13528,7 +13706,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu)
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++) {
- if (svc.chan[i].valid)
+ if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu)
local[n++] = svc.chan[i];
}
spin_unlock_irqrestore(&svc.chan_lock, flags);
@@ -13564,7 +13742,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu)
* pass's snapshot belongs to a vanished channel; its waiter is gone,
* so drop the set instead of resolving it later.
*/
- svc_opdef_drop_gone(local, n);
+ svc_opdef_drop_gone(vgpu, local, n);
for (i = 0; i < n; i++) {
struct svc_channel *ch = &local[i];
@@ -15077,7 +15255,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu)
spin_lock_irqsave(&svc.chan_lock, flags);
for (k = 0; k < SVC_MAX_CHANNELS; k++) {
- if (svc.chan[k].valid &&
+ if (svc.chan[k].valid && svc.chan[k].vgpu == vgpu &&
svc.chan[k].h_object == ch->h_object) {
svc.chan[k].userd_pa = ch->userd_pa;
svc.chan[k].userd_as = ch->userd_as;
@@ -15391,7 +15569,7 @@ static void svc_handle_chan_dump(struct nv_vgpu_device *vgpu)
spin_lock_irqsave(&svc.chan_lock, flags);
for (i = 0; i < SVC_MAX_CHANNELS; i++)
- if (svc.chan[i].valid)
+ if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu)
local[n++] = svc.chan[i];
spin_unlock_irqrestore(&svc.chan_lock, flags);
@@ -15782,9 +15960,18 @@ int gsp_sema_service_counter_add(u32 h_object, u64 pa)
* (vmallocinfo vmap area + kcore page-table walk). fin_pa is the finish
* payload the guest's scrub wait polls; hsem_pa is the channel's host
* semaphore (channelWaitForFreeEntry polls it). Plain state update.
+ *
+ * Dual-guest: the words are written through a specific guest's address space
+ * (see svc_scrub_pass), so the registration binds to the FIRST open mdev -
+ * the same single-guest convention the scanner tooling follows (it tracks
+ * one client). Refused with no open mdev rather than parked globally.
*/
int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa)
{
+ struct nv_vgpu_device *vgpu = NULL;
+ struct nv_vgpu_device *d;
+ struct svc_scrub *s;
+
if (!svc.task)
return -ENODEV;
/* Both words are u32: require 4-byte alignment and a RAM-shaped PA. */
@@ -15793,12 +15980,28 @@ int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa)
if (!hsem_pa || (hsem_pa & 0x3) || hsem_pa >= SEMA_PA_MAX)
return -EINVAL;
- svc_scrub.fin_pa = fin_pa;
- svc_scrub.hsem_pa = hsem_pa;
- svc_scrub.pa_ok = true;
- svc_scrub.pa_gen++;
+ mutex_lock(&nv_vgpu_driver.lock);
+ list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
+ if (atomic_read(&d->vfio_dev_open_count) > 0) {
+ vgpu = d;
+ break;
+ }
+ }
+ mutex_unlock(&nv_vgpu_driver.lock);
+ if (!vgpu) {
+ pr_warn_ratelimited("sema-svc: scrubber buffer registration ignored: no open mdev to bind it to\n");
+ return -ENODEV;
+ }
+ s = svc_scrub_get(vgpu, true);
+ if (!s)
+ return -ENOSPC;
+
+ s->fin_pa = fin_pa;
+ s->hsem_pa = hsem_pa;
+ s->pa_ok = true;
+ s->pa_gen++;
pr_info("sema-svc: scrubber buffer registered fin_pa=0x%llx hsem_pa=0x%llx (gen %u)\n",
- fin_pa, hsem_pa, svc_scrub.pa_gen);
+ fin_pa, hsem_pa, s->pa_gen);
return 0;
}
@@ -15826,11 +16029,11 @@ int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa)
*/
static void svc_scrub_pass(struct nv_vgpu_device *vgpu)
{
- struct svc_scrub *s = &svc_scrub;
+ struct svc_scrub *s = svc_scrub_get(vgpu, false);
u32 put = 0;
u64 delta;
- if (!s->userd_ok || !s->pa_ok || !s->entries)
+ if (!s || !s->userd_ok || !s->pa_ok || !s->entries)
return;
if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as,
&put, sizeof(put))) {
@@ -15887,7 +16090,9 @@ static void svc_scrub_pass(struct nv_vgpu_device *vgpu)
static int svc_thread(void *arg)
{
+ struct nv_vgpu_device *vgpus[SVC_MAX_GUESTS];
struct nv_vgpu_device *vgpu;
+ int n_vgpus, g;
pr_info("sema-svc: thread started\n");
@@ -15907,130 +16112,174 @@ static int svc_thread(void *arg)
complete_all(&svc_sess_flush_done);
}
- vgpu = NULL;
+ /*
+ * Service EVERY open mdev each pass, in list order. With one
+ * mdev this is exactly the historical single-vgpu pass; with
+ * two, both guests get a pump turn per iteration (hardware
+ * forwarding stays serialized by the single-inflight slot; see
+ * the forwarding section). The list walk is the historical
+ * bounded scan under nv_vgpu_driver.lock.
+ */
+ n_vgpus = 0;
mutex_lock(&nv_vgpu_driver.lock);
{
struct nv_vgpu_device *d;
list_for_each_entry(d, &nv_vgpu_driver.devices, list) {
- if (atomic_read(&d->vfio_dev_open_count) > 0) {
- vgpu = d;
- break;
- }
+ if (atomic_read(&d->vfio_dev_open_count) > 0 &&
+ n_vgpus < SVC_MAX_GUESTS)
+ vgpus[n_vgpus++] = d;
}
}
mutex_unlock(&nv_vgpu_driver.lock);
- if (!vgpu) {
+ if (!n_vgpus) {
msleep(500);
continue;
}
- svc_handle_probe(vgpu);
- svc_handle_walk_probe(vgpu);
+ /* Host-injection queue drain: once per pass, not per guest. */
svc_handle_map_add();
- svc_handle_chan_dump(vgpu);
- svc_scrub_pass(vgpu);
- if (svc_sess_flush || !svc.direct_base ||
- time_after(jiffies,
- (unsigned long)(svc.last_build + 10 * HZ)))
- svc_rebuild(vgpu);
+ for (g = 0; g < n_vgpus; g++) {
+ struct svc_guest_sess *sess;
+ u64 done_before;
- /*
- * Channel-driven execution: notice submissions as they are made
- * and run the pushed work immediately, so the guest's UVM
- * progress waits complete in milliseconds rather than in the
- * time it takes the fallback scan to find the push. A doorbell
- * token means a channel was just scheduled; run everything now.
- */
- {
- u64 done_before = svc.gpfifo_entries_done;
+ vgpu = vgpus[g];
+ svc_handle_probe(vgpu);
+ svc_handle_walk_probe(vgpu);
+ svc_handle_chan_dump(vgpu);
+ svc_scrub_pass(vgpu);
+
+ /*
+ * Per-guest rebuild freshness: the discovery map is
+ * still shared between guests (staged G4), but each
+ * guest's RAM must be scanned on its own clock, or the
+ * first guest's 10 s tick would starve the second
+ * guest of rebuilds. The freshness stamp is the one
+ * svc_rebuild() publishes (svc.last_build).
+ */
+ sess = svc_guest_get(vgpu, true);
+ if (svc_sess_flush || !svc.direct_base ||
+ !sess ||
+ time_after(jiffies,
+ (unsigned long)(sess->last_build +
+ 10 * HZ))) {
+ svc_rebuild(vgpu);
+ if (sess)
+ sess->last_build = svc.last_build;
+ }
+ /*
+ * Channel-driven execution: notice submissions as they
+ * are made and run the pushed work immediately, so the
+ * guest's UVM progress waits complete in milliseconds
+ * rather than in the time it takes the fallback scan
+ * to find the push. A doorbell token means a channel
+ * was just scheduled; run everything now.
+ */
+ done_before = svc.gpfifo_entries_done;
svc.doorbell_pending = false;
svc_pump_channels(vgpu);
- pumped = (svc.gpfifo_entries_done != done_before);
- }
+ if (svc.gpfifo_entries_done != done_before)
+ pumped = true;
- /*
- * Host CUDA bridge: marshal one queued launch into a daemon
- * job and resolve completions. Inert until a launch is seen.
- */
- gsp_cuda_bridge_service(vgpu);
+ /*
+ * Host CUDA bridge: marshal one queued launch into a
+ * daemon job and resolve completions. Inert until a
+ * launch is seen.
+ */
+ gsp_cuda_bridge_service(vgpu);
- /*
- * Releases whose target VA was not translatable when the scan
- * saw them: retry every pass, so a pin or a learned pair applies
- * them immediately instead of on the next full sweep.
- */
- svc_apply_pending(vgpu);
+ /*
+ * Releases whose target VA was not translatable when
+ * the scan saw them: retry every pass, so a pin or a
+ * learned pair applies them immediately instead of on
+ * the next full sweep.
+ */
+ svc_apply_pending(vgpu);
- /*
- * A fresh VA->PA injection changes which releases are
- * applicable: run the fast sliced sweep while the forced window
- * is open (the pump's stall path also uses it).
- */
- if (time_before(jiffies, (unsigned long)svc.fast_force_until))
- svc_apply_sweep_slice(vgpu);
+ /*
+ * A fresh VA->PA injection changes which releases are
+ * applicable: run the fast sliced sweep while the
+ * forced window is open (the pump's stall path also
+ * uses it).
+ */
+ if (time_before(jiffies,
+ (unsigned long)svc.fast_force_until))
+ svc_apply_sweep_slice(vgpu);
+ }
/*
* While the pump is making progress the client is streaming
* pushes and waiting on their releases: loop tightly and skip
- * the discovery/sweep passes, which otherwise dominate the pass
- * period (the pump cadence is what the client's context init
- * deadline sees). Scans resume the first pass without progress.
+ * the discovery/sweep passes, which otherwise dominate the
+ * pass period (the pump cadence is what the client's context
+ * init deadline sees). Scans resume the first pass without
+ * progress. (With two guests, any guest's progress keeps the
+ * loop tight; the next iteration pumps both.)
*/
if (pumped) {
msleep(1);
continue;
}
- /*
- * Push-buffer base discovery: a chunked walk of guest RAM
- * finds the pages whose first push names a channel, so a
- * channel that appears with work already waiting is pinned and
- * pumped without waiting for the rolling sweep.
- */
- svc_pb_discover(vgpu);
+ for (g = 0; g < n_vgpus; g++) {
+ vgpu = vgpus[g];
- /*
- * Completion hint: re-post the events for a few seconds after a
- * completion, so a client that starts waiting after the
- * edge-triggered wake-up still observes it.
- */
- if (time_before(jiffies, (unsigned long)svc.post_hint_until) &&
- time_after(jiffies, (unsigned long)svc.last_hint + HZ / 2)) {
- int rc;
+ /*
+ * Push-buffer base discovery: a chunked walk of guest
+ * RAM finds the pages whose first push names a
+ * channel, so a channel that appears with work
+ * already waiting is pinned and pumped without
+ * waiting for the rolling sweep.
+ */
+ svc_pb_discover(vgpu);
- svc.last_hint = jiffies;
- rc = gsp_rm_post_all_events(vgpu);
- if (gsp_sema_dbg)
- pr_info_ratelimited("sema-svc: completion hint rc=%d\n",
- rc);
- }
+ /*
+ * Completion hint: re-post the events for a few
+ * seconds after a completion, so a client that starts
+ * waiting after the edge-triggered wake-up still
+ * observes it.
+ */
+ if (time_before(jiffies,
+ (unsigned long)svc.post_hint_until) &&
+ time_after(jiffies,
+ (unsigned long)svc.last_hint + HZ / 2)) {
+ int rc;
+
+ svc.last_hint = jiffies;
+ rc = gsp_rm_post_all_events(vgpu);
+ if (gsp_sema_dbg)
+ pr_info_ratelimited("sema-svc: completion hint rc=%d\n",
+ rc);
+ }
- /*
- * Fast path for the pushes themselves: every mapped page is a
- * candidate pushbuffer page, and there are far fewer of those
- * than there are pages of guest RAM.
- */
- if (svc.direct_base && svc.map_n)
- svc_apply_mapped(vgpu);
+ /*
+ * Fast path for the pushes themselves: every mapped
+ * page is a candidate pushbuffer page, and there are
+ * far fewer of those than there are pages of guest
+ * RAM.
+ */
+ if (svc.direct_base && svc.map_n)
+ svc_apply_mapped(vgpu);
- /*
- * Rolling RAM scan: the pushbuffers of some clients are not
- * covered by the VA map, so sweep guest RAM in bounded windows
- * in parallel with the targeted pass. Work per pass is fixed,
- * which keeps the emulation responsive while a full sweep
- * completes every few seconds.
- */
- svc_apply_pushes(vgpu);
+ /*
+ * Rolling RAM scan: the pushbuffers of some clients
+ * are not covered by the VA map, so sweep guest RAM in
+ * bounded windows in parallel with the targeted pass.
+ * Work per pass is fixed, which keeps the emulation
+ * responsive while a full sweep completes every few
+ * seconds.
+ */
+ svc_apply_pushes(vgpu);
- /*
- * Deferred readability probes for faked CE pushes (observation
- * only; see svc_retry_poll).
- */
- svc_retry_poll(vgpu);
+ /*
+ * Deferred readability probes for faked CE pushes
+ * (observation only; see svc_retry_poll).
+ */
+ svc_retry_poll(vgpu);
+ }
msleep(10);
}
@@ -16153,6 +16402,97 @@ void gsp_sema_service_stop(void)
svc_vas_free_all();
}
+/*
+ * Release every piece of service state owned by one mdev (dual-guest goal
+ * 1.0). Called from the mdev cleanup path after that mdev's emulation thread
+ * stopped and before its struct can be freed:
+ * - quiesce forwarding so no in-flight work item can still be reading the
+ * request's vgpu, and abort a release hold that names it;
+ * - drop any hold that survives (a request that never queued leaves no work
+ * item to clear it) BEFORE the device can go away - resolution would
+ * dereference it;
+ * - drop the guest's channels; request the global session flush only when no
+ * guest has live channels left (the discovery tables are still shared;
+ * staged G4);
+ * - drop the guest's held-operand release slots (its waiters are gone);
+ * - free the per-guest session and scrubber records so the vgpu pointer can
+ * never match a newly allocated device.
+ */
+void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu)
+{
+ unsigned long flags;
+ int i;
+
+ if (!vgpu)
+ return;
+
+ /*
+ * The forward work item runs on its own kthread and may be using the
+ * request's vgpu; quiesce waits for it (bounded by the submit's
+ * 200 ms poll cap) and aborts an active hold. Resume afterwards so the
+ * other guests keep forwarding.
+ */
+ gsp_sema_service_work_quiesce();
+ if (READ_ONCE(svc_fwd_hold.active) && svc_fwd_hold.vgpu == vgpu)
+ WRITE_ONCE(svc_fwd_hold.aborted, true);
+ gsp_sema_service_work_resume();
+
+ if (READ_ONCE(svc_fwd_hold.active) && svc_fwd_hold.vgpu == vgpu) {
+ /*
+ * Pure state clear (never a resolution through the vanishing
+ * device): the releases belong to a guest that is gone.
+ */
+ svc.fwd_hold_canceled++;
+ svc_fwd_hold_clear();
+ }
+
+ spin_lock_irqsave(&svc.chan_lock, flags);
+ for (i = 0; i < SVC_MAX_CHANNELS; i++) {
+ struct svc_channel *ch = &svc.chan[i];
+
+ if (!ch->valid || ch->vgpu != vgpu)
+ continue;
+ pr_info("sema-svc: channel 0x%x dropped (mdev cleanup; submissions=%llu)\n",
+ ch->h_object, ch->submissions);
+ memset(ch, 0, sizeof(*ch));
+ }
+ if (svc_sess_active) {
+ struct svc_guest_sess *sess = svc_guest_get(vgpu, false);
+ bool any = false;
+
+ for (i = 0; i < SVC_MAX_CHANNELS; i++)
+ if (svc.chan[i].valid) {
+ any = true;
+ break;
+ }
+ if (sess)
+ sess->active = false;
+ if (!any) {
+ svc_sess_active = 0;
+ svc_sess_flush = 1;
+ }
+ }
+ spin_unlock_irqrestore(&svc.chan_lock, flags);
+
+ /* Held-operand release slots of this guest cannot be delivered. */
+ for (i = 0; i < SVC_OPDEF_SLOTS; i++)
+ if (svc_opdef[i].active && svc_opdef[i].vgpu == vgpu)
+ svc_opdef_clear(&svc_opdef[i]);
+
+ /* Free the per-guest session and scrubber records. */
+ spin_lock_irqsave(&svc_guest_lock, flags);
+ for (i = 0; i < SVC_MAX_GUESTS; i++)
+ if (svc_guests[i].vgpu == vgpu)
+ memset(&svc_guests[i], 0, sizeof(svc_guests[i]));
+ spin_unlock_irqrestore(&svc_guest_lock, flags);
+ for (i = 0; i < SVC_MAX_GUESTS; i++)
+ if (svc_scrubs[i].vgpu == vgpu)
+ memset(&svc_scrubs[i], 0, sizeof(svc_scrubs[i]));
+
+ if (svc_sess_flush && svc.task)
+ wake_up_process(svc.task);
+}
+
/* ------------------------------------------------------------------ */
/* host CUDA bridge exports (gsp_cuda_bridge.c) */
/* ------------------------------------------------------------------ */
diff --git a/kernel/gsp_sema_service.h b/kernel/gsp_sema_service.h
index 6822bd5..c1dd763 100644
--- a/kernel/gsp_sema_service.h
+++ b/kernel/gsp_sema_service.h
@@ -18,8 +18,16 @@
*
* All addresses are guest-physical except gpfifo_va, which is the GPU VA of
* the channel's GPFIFO ring (translated through the discovered VA->PA map).
+ *
+ * vgpu names the owning mdev (guest session). Two guests' stock allocators
+ * choose the same object handles, so the channel table is keyed by
+ * (vgpu, h_object): a registration can never overwrite another guest's slot,
+ * and ranks/session bookkeeping are computed per guest.
*/
+struct nv_vgpu_device;
+
struct sema_channel_desc {
+ struct nv_vgpu_device *vgpu; /* owning mdev (guest session) */
u32 h_object;
u32 engine_type;
u32 cid; /* RM channel id (the doorbell token vector) */
@@ -33,7 +41,17 @@ struct sema_channel_desc {
};
void gsp_sema_service_channel_add(const struct sema_channel_desc *desc);
-void gsp_sema_service_channel_del(u32 h_object);
+void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object);
+
+/*
+ * Release every piece of service state that belongs to one mdev before its
+ * guest session disappears: drop the mdev's channels (per-vgpu session end),
+ * clear a release hold that names it (never dereference a removed device),
+ * and free its per-guest session record. Called from the mdev cleanup path
+ * after the device's emulation thread stopped and before the mdev struct can
+ * be freed. No-op for a vgpu that never registered a channel.
+ */
+void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu);
/*
* Register the guest PMA scrubber's channel-buffer words (the CeUtils
diff --git a/kernel/main.c b/kernel/main.c
index d0308c4..257638a 100644
--- a/kernel/main.c
+++ b/kernel/main.c
@@ -1740,6 +1740,16 @@ static void nv_vgpu_device_cleanup_work(struct work_struct *work)
gsp_emu_cleanup(vgpu);
/*
+ * Dual-guest (goal 1.0): release everything the sema service still holds
+ * for this mdev (channel entries, session/scrubber records, a release
+ * hold that names it - quiescing any in-flight forwarding work) and drop
+ * its per-mdev guest-RAM record, BEFORE the vgpu struct can be freed: a
+ * stale pointer in the service's tables must never match a later device.
+ */
+ gsp_sema_service_vgpu_cleanup(vgpu);
+ gsp_phys_guest_ram_vgpu_clear(vgpu);
+
+ /*
* Last-reference backend teardown: gsp_emu_cleanup() above has stopped
* this mdev's emulation thread, so no forward can be in flight when the
* claim is released. The backend session stays up until the last mdev