docs(vexecution-plane): add the vGSP Execution Plane architecture doc, wiki deep dive and indexes

1 files changed, 1029 insertions, 0 deletions

diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md
new file mode 100644
index 0000000..3cc6adb
--- /dev/null
+++ b/docs/03-architecture/13-vexecution-plane.md
@@ -0,0 +1,1029 @@
+# The vGSP Execution Plane (vEP)
+
+> **What this document is.** The canonical architecture and design document for
+> the layer of `vgsp.ko` that turns guest-authored GPU work into physical
+> execution and guest-visible completion: from a guest CUDA kernel launch's
+> GPFIFO push, through ring discovery, VA/PA truth, semaphores, counters and
+> notifiers, the QMD and shadow windows, submission to the physical engines,
+> and the completion published back to the guest. It names that layer
+> **vGSP Execution Plane (vEP)**, defines its boundaries and its four work
+> domains, and maps every major mechanism to the code that implements it.
+>
+> **Audience.** Developers and architects working on guest execution, the
+> forwarding ladder, or the GR launch frontier. Read
+> [`01-system-overview.md`](01-system-overview.md) and
+> [`09-full-system-picture.md`](09-full-system-picture.md) first for the whole
+> machine; read [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)
+> for the physical-session lifecycle this document builds on.
+>
+> **Status: 2026-10-08, tree `master @ 66388560`.** Every empirical claim below
+> cites a commit, a code location, a tracked document, or a local campaign
+> evidence file (paths under `backups/evidence-…/` are local evidence
+> artifacts, not shipped in the repository). Where a mechanism is a leading
+> hypothesis rather than a proven fact, the text says so. Nothing in this
+> document is a stability or support claim.
+
+Status tags used throughout (same convention as docs 09/10/11):
+`[live]` proven with the stock driver; `[gated]` implemented but default-off or
+parameter-gated; `[wip]` partially implemented; `[wall]` the current blocker;
+`[target]` the named success latch; `[planned]` designed, not implemented.
+
+---
+
+## 1. What the vEP is: name, boundaries, domains
+
+### 1.1 The name and why
+
+**Canonical name: vGSP Execution Plane (vEP).**
+
+> The **vGSP Execution Plane** is the host-side subsystem of `vgsp.ko` that
+> turns guest-authored GPU work into physical execution and guest-visible
+> completion. It spans channel-ring and USERD discovery, GPU VA-to-PA truth,
+> semaphore-release application, completion-counter and notifier publication,
+> QMD handling for forwarded launches, and submission to the physical GPU's
+> engines under the module-owned physical-backend session.
+
+Two framing terms, used consistently from here on:
+
+* **vGSP (virtual GSP)** — our host-side reimplementation of the GSP/GSP-RM
+  interface: the virtual device plus the emulated firmware service that the
+  stock guest driver interacts with. `vgsp.ko` is vGSP's implementation. The
+  guest believes it is talking to its GPU's GSP; vGSP is the thing it is
+  actually talking to.
+* **vEP** — the execution half of vGSP. The *control plane* (boot handshake,
+  message queues, RPC, RM objects, BAR shadows) makes the guest driver run;
+  the *execution plane* makes the guest's work run. This document covers the
+  second half and only the seams of the first that shape execution.
+
+The name is **ours**, the name of a subsystem we implement. It is deliberately
+not a vendor term: the three-letter form reads as the sibling of the existing
+project vocabulary (`vGSP`, `vGPU`, `vgsp.ko`), and "execution plane" states
+the job (execute guest work) rather than the mechanism (forwarding), which
+leaves room for the fast paths that stay in software.
+
+**Alternates considered and rejected:**
+
+| Rejected name | Why not |
+|---|---|
+| "Work plane" / "work path" | Already used in [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and [`09-full-system-picture.md`](09-full-system-picture.md) for the software-executed bridge inside the pump; reusing it would silently change the documented model. |
+| "Submission plane" | Covers only the submit half; hides discovery, translation, counters and completion — over half the subsystem. |
+| "Forwarding plane" | Mechanism-centric: forwarding is one execution mode (some work is applied in software by design), and "forwarding" already means a specific ladder in docs 11. |
+| "Data plane" | Networking connotation; the plane's most latency-critical work is state operations (releases, fences), not bulk data. |
+| "vGPU Work Engine" | "Engine" is the hardware term for GR/CE; naming software an engine invites confusion in every review. |
+| "GPU Execution Plane (GEP)" | Reads as a hardware/PCIe concept, not as our software subsystem. |
+| "RAS plane" (Ring/Alloc/Sem) | Cryptic, and it omits the completion/publication half. |
+| "vGSP Runlist" | Collides with the hardware runlist/TSG concept it actually schedules onto. |
+
+### 1.2 Scope: in and out
+
+**In scope (implemented by the vEP):**
+
+* the service thread and its per-channel pump
+  (`svc_thread`, `svc_pump_channels`, both `kernel/gsp_sema_service.c`);
+* ring/USERD discovery, autopin and the VA-to-PA truth model
+  (`svc_ring_autopin`, `svc_gr_ring_autopin`, `svc_gr_content_discover`,
+  `svc_chan_userd`, `svc_lookup`, `svc_walk_vaspaces`, `svc_gmmu_walk`,
+  `svc_pd_discover`, `svc_targeted_resolve`, `svc_pool_pa_resolve`);
+* push decode and QMD capture (`svc_decode_stream`, `svc_qmd_accum`,
+  `svc_gr_qmd_decode`, `svc_gr_cand_capture`);
+* release application and completion publication (`svc_sem_write`,
+  `svc_counter_release_apply`, `svc_counter_derive`, `svc_counter_autoreg`,
+  `svc_counter_flush_pending`, `svc_counter_publish`);
+* notifier and channel-event delivery (`svc_notify_complete`,
+  `gsp_rm_post_channel_event`);
+* physical submission and its mode matrix (`svc_gr_req_fill`,
+  `svc_gr_request`, `svc_fwd_gr_work`, `gsp_phys_backend_guest_gr_submit`,
+  `gsp_phys_backend_guest_ce_copy`, the ladder channel and its VAs);
+* the pacing of all of the above (doorbell steering, discovery cadence, the
+  single-inflight ordered workqueue) and its sharing across mdevs.
+
+**Out of scope (context, not this layer):** the GSP boot handshake and
+message-queue transport ([`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md)),
+the RM control/object surface as a whole
+([`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §4, `kernel/gsp_rm_emulation.c`),
+BAR shadows and MSI-X injection
+([`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md)),
+the physical claim/boot lifecycle
+([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)), and the
+guest stack ([`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md)).
+The vEP *uses* those interfaces; it does not own them.
+
+### 1.3 The four work domains
+
+The vEP can be described as one pipeline, but it answers four distinct
+questions. This mapping is the document's organizing model.
+
+| Domain | Question | vEP components | State |
+|---|---|---|---|
+| **Scheduler** | Which guest channel's work runs next, and at what pace? | Doorbell token steering (`gsp_sema_service_doorbell`, called from `main.c:865`); per-pass channel pump (`svc_pump_channels`); discovery cadence (`svc_gr_content_discover`); single-inflight ordered workqueue `gsp-phys-work` (`svc_gr_request`, `svc_fwd_request`); deferred-release hold (`svc_fwd_hold`); the service thread loop (`svc_thread`) | `[live]` software pacing and ordering; `[planned]` tenant fairness |
+| **Allocator** | Where do rings, USERDs, counters, notifiers, scratch pages and PT windows live, and who owns them? | CE/GR ring autopin and pins (`svc_ring_autopin`, `svc_gr_ring_autopin`, `svc_pin_store`/`svc_ipin_store`); GR content-keyed ring/push pairing (`svc_gr_content_discover`, `svc_gr_content_pin`); counter slot derive/autoreg (`svc_counter_derive`); ladder scratch windows and PT pools (window-0 shadow pool, data-prewin pool, the 128-byte PTE fetch-line pool in `gsp_phys_booter.c`); the BAR1 vidmem shadow (`vgpu->bars[1].vaddr`) | `[live]` discovery, pins, scratch pools; `[wip]` protocol-derived allocation; `[planned]` real VRAM backing |
+| **Control** | Which RM objects and controls shape execution, and how are they emulated or forwarded? | Channel registration from `GSP_RM_ALLOC` (`gsp_rm_register_channel` → `gsp_sema_service_channel_add`); event registration; `SET_PAGE_DIRECTORY` recording; `PROMOTE_CTX`; engine/class topology; the mode matrix (§2.3) | `[live]` registration + emulate; `[gated]` read-only control forwarding; `[planned]` alloc/free forwarding |
+| **Share** | How do clients and mdevs share one physical GPU with isolation and fairness? | mdev backend refcount and ownership (`gsp_phys_backend_start/stop`); one real GSP session serialized by `gsp_phys_rm_lock`; one ordered workqueue and one in-flight request slot; per-mdev emulation context; planned per-mdev real clients and fairness policy | `[live]` single-tenant; `[wip]` per-client state reset (ISSUE-0002); `[planned]` multi-tenant |
+
+The four domains cut across one code path. The rest of this document follows
+that path: the architecture and its diagrams (§2), how each stage actually
+works today (§3), the end-to-end GR chain (§4), then the program-level goals,
+challenges, portability and future work.
+
+---
+
+## 2. Architecture
+
+### 2.1 Component map
+
+```mermaid
+flowchart TB
+  subgraph GUEST["Guest VM - stock software"]
+    APP["CUDA application"]
+    LCU["libcuda - builds push + QMD"]
+    GKO["nvidia.ko - RM + UVM"]
+    APP --> LCU --> GKO
+  end
+
+  subgraph VDEV["vgsp.ko - guest-facing"]
+    BAR["BAR0 shadow + doorbell 0x88704"]
+    RM["GSP-RM emulation - controls, allocs, events"]
+  end
+
+  subgraph VEP["vGSP Execution Plane"]
+    DB["Doorbell steering"]
+    PUMP["Channel pump svc_pump_channels"]
+    DISC["Ring/USERD discovery + autopin"]
+    TRUTH["VA-to-PA truth - walk, pins, rescue"]
+    DEC["Push decode + QMD capture"]
+    REL["Release apply + counters + notifiers"]
+    WQ["Ordered workqueue gsp-phys-work"]
+    SUB["Ladder submission gr/ce"]
+    MODES["Mode matrix"]
+    PUMP --> DISC --> TRUTH --> DEC --> REL
+    DB --> PUMP
+    DEC --> WQ --> SUB
+    MODES --> SUB
+  end
+
+  subgraph BE["Physical backend - real GSP session"]
+    LAD["Module-owned ladder channel"]
+    GSP["Real GSP-RM server"]
+    ENG["GR and CE engines"]
+  end
+
+  subgraph HW["GPU hardware"]
+    GMMU["GMMU page tables"]
+    VRAM["VRAM / FB"]
+  end
+
+  GKO -->|"MMIO + DMA + MSI-X"| BAR
+  BAR --> DB
+  RM -->|"channel/event/vas registration"| PUMP
+  REL -->|"release, counter, notifier, event"| RM
+  RM -->|"MSI-X"| GKO
+  TRUTH -->|"reads guest page tables"| GKO
+  SUB --> LAD --> GSP
+  LAD -->|"GPFIFO + doorbell"| ENG
+  GSP --> ENG
+  ENG -->|"release semaphore"| SUB
+  ENG --> GMMU
+  SUB -.->|"planted PTEs"| GMMU
+  VRAM -.-> GMMU
+
+  classDef done fill:#d5f5d5,stroke:#1e7a1e
+  classDef wip fill:#fff3cd,stroke:#9a7b1a
+  classDef plan fill:#f8d7da,stroke:#9a1a1a
+  classDef hw fill:#e2e8f0,stroke:#4a5568
+  class DB,PUMP,DISC,TRUTH,DEC,REL done
+  class WQ,SUB,MODES wip
+  class BAR,RM done
+  class LAD,GSP,ENG hw
+  class GMMU,VRAM hw
+```
+
+ASCII fallback (reading top to bottom: guest → vgsp.ko → physical GPU):
+
+```text
+Guest VM (stock)
+  CUDA app -> libcuda (push + QMD) -> nvidia.ko (RM, UVM)
+        | MMIO write to BAR0 doorbell 0x88704        ^ MSI-X (events)
+        v                                            |
+vgsp.ko  ============================================ | =====================
+  BAR0 shadow / doorbell  ->  Doorbell steering       |
+  GSP-RM emulation  <-------  release/counter/event --+
+        | channel, event, vaspace registration
+        v
+  +----------------- vGSP Execution Plane ----------------------+
+  | pump -> ring/USERD discovery -> VA-to-PA truth -> decode     |
+  |   -> release apply + counter publish + notifier              |
+  |   -> ordered workqueue -> ladder submission (mode matrix)    |
+  +-------------------------- | ---------------------------------+
+                               v
+Physical backend (module-owned GPU, real GSP session)
+  ladder channel -> real GSP-RM -> GR/CE engines -> GMMU/VRAM
+        ^ engine-authored release semaphore (polled by the submit)
+```
+
+### 2.2 Data flow of one CUDA kernel launch
+
+The full chain, from CUDA context creation to the guest waking on the
+completion. Stages 4–8 are shared by every channel type; stage 9 onward is the
+GR-specific chain of §4. Status per stage is in the table after the diagram.
+
+```mermaid
+sequenceDiagram
+  autonumber
+  participant CU as Guest CUDA app
+  participant LC as libcuda / nvidia.ko
+  participant DEV as BAR0 shadow + doorbell
+  participant PU as vEP pump (service kthread)
+  participant TR as vEP truth (walk/pins/rescue)
+  participant DC as vEP decode + capture
+  participant WQ as Ordered workqueue
+  participant LB as Ladder submission
+  participant GP as Physical GSP + GR engine
+  participant CM as Completion publication
+
+  CU->>LC: cuCtxCreate, cuMemAlloc, cuLaunchKernel
+  Note over LC: builds GR push: ctx-init methods,<br/>QMD install, SASS pointers, params
+  LC->>DEV: GPFIFO entry appended; doorbell 0x88704
+  DEV->>PU: doorbell token steering (main.c:865)
+  PU->>TR: resolve ring VA and USERD (discovery/autopin)
+  PU->>TR: read GP_PUT, fetch GPFIFO entry, decode method stream
+  TR-->>PU: translated pages (or bounded rescue)
+  PU->>DC: accumulate QMD window + launch candidate
+  DC->>WQ: one GR request (single in-flight)
+  WQ->>LB: guest GR submit (QMD relocate, PTEs, method block)
+  LB->>GP: ladder GPFIFO publish + doorbell
+  GP->>GP: front-end fetch, inline-QMD install [wall],<br/>ctx switch, SASS dispatch
+  GP-->>LB: engine authors release semaphore in module/guest page
+  LB-->>PU: completion observed (poll)
+  PU->>CM: apply deferred release once; publish counter; notify
+  CM->>DEV: channel event -> MSI-X (gsp_rm_post_channel_event)
+  DEV->>LC: guest wakes; CUDA continues
+```
+
+ASCII fallback of the same chain:
+
+```text
+cuLaunchKernel
+  -> push built (ctx init, QMD, SASS ptrs, params) + GPFIFO entry + doorbell
+  -> pump reads GP_PUT / entry; auth: ring + USERD discovered, page translated
+  -> decode stream; capture QMD window and launch candidate
+  -> ordered workqueue: one in-flight GR request
+  -> ladder submission: relocate QMD, plant PTEs (pre-built fetch lines),
+     build launch methods, GPFIFO publish + doorbell on the real channel
+  -> physical front-end fetch -> inline-QMD install [wall] -> GR engine
+  -> engine authors release -> poll observes
+  -> apply deferred release exactly once + counter publish + notifier
+  -> channel event -> MSI-X -> guest tracker wakes
+```
+
+| Stage | What happens | State (2026-10-08) |
+|---|---|---|
+| 1 | Guest CUDA builds the launch (push methods, QMD, params, SASS refs) | `[live]` observed with the stock driver |
+| 2 | GPFIFO entry + doorbell written into the BAR0 shadow | `[live]` |
+| 3 | Pump steering, ring/USERD discovery, entry fetch, translation | `[live]` for CE; `[wip]` for GR (discovery now succeeds; see §3.2) |
+| 4 | Push decode; QMD window accumulation; launch candidate capture | `[live]` observed (ring entry consumed; methods execute) |
+| 5 | Physical submit on the ladder channel (mode-matrix shaped) | `[gated]` default-off; latched on `rmcall_gr_exec_ok` |
+| 6 | Front-end fetch + method execution on the real channel | `[live]` observed |
+| 7 | **Inline-QMD install / engine reads the descriptor** | **`[wall]` not observed** |
+| 8 | Context switch + SASS dispatch to SMs | `[wall]` not observed for forwarded launches |
+| 9 | Engine-authored release + channel release | `[wall]` absent for forwarded launches |
+| 10 | Completion application, counter publish, notifier, event, MSI-X | `[wip]` CE path live default-off; GR wake-up is the open question (ISSUE-0001) |
+
+The stage table's authority is
+[`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)
+(tracked) and the 2026-10-08 campaigns below it. Stage 1–4 status is
+re-verified by the `grv3` campaign: ring entries *were* consumed for ranks
+0–2, releases *were* applied, and the guest still did not proceed (local
+evidence `backups/evidence-20261002-victory/analysis-2026-10-08-stall-v2/FORENSICS-grv3.md`
+§0 and §2).
+
+### 2.3 The mode matrix
+
+The physical submit path is parameterized by a mode matrix: a set of
+module parameters, almost all default-off, that select rebuild-free A/B
+variants of the GR context commit, the launch method stream, the memory
+windows and the completion relay. The matrix exists because the rules the front-end and engine
+enforce are still being recovered; each knob isolates one variable so a
+supervised cycle can falsify one hypothesis at a time. Every non-neutral knob
+is documented in `kernel/gsp_phys_gpu.c` (and, for the work-forwarding waves,
+`main.c`) with its rationale and rollback; the parameter is the experiment's
+record.
+
+```mermaid
+flowchart LR
+  subgraph CTX["GR context commit"]
+    A1["gr_promote 0..35<br/>promote shape bisection"]
+    A2["srv_ctxbuf / srv_ctxcommit<br/>server-managed ctx"]
+    A3["channel variants V5/V6<br/>golden replay"]
+    A4["vas_shared / ctxshare_force<br/>VAS ownership"]
+  end
+  subgraph LAUNCH["GR launch shape"]
+    B1["prelude / qmdvirt / pcas<br/>method-stream form"]
+    B2["lmem_en - SKEDCHECK05 fix"]
+    B3["lane_skip / lane_tail<br/>schedule-state controls"]
+    B4["r1_first / recommit<br/>ordering"]
+  end
+  subgraph MEM["GR memory windows"]
+    C1["qmdwin / shadow_win0<br/>win0 PT line"]
+    C2["data_prewin / ptr_guest<br/>PTE kind"]
+    C3["mail_patch / inst_sys<br/>instance aperture"]
+    C4["gpc_mmu_init<br/>GPC walker"]
+  end
+  subgraph COMP["Completion + diagnostics"]
+    D1["rel_mod - engine-completion discriminant"]
+    D2["skedrec / img_probe<br/>notifier_sys / faultbuf_sys"]
+  end
+  subgraph WAVES["CE forwarding waves"]
+    E1["work_forward -> work_defer_release"]
+    E2["work_hw_primary -> relay"]
+    E3["fault_relay / fault_inject"]
+  end
+  LAUNCH --> MEM --> CTX
+  CTX --> COMP
+  WAVES --> COMP
+```
+
+ASCII fallback:
+
+```text
+CTX COMMIT      gr_promote(0..35) srv_ctxbuf srv_ctxcommit srv_main_ignore
+                channel(V5/V6) golden golden_exec_shape ctxshare_force vas_shared
+LAUNCH SHAPE    prelude qmdvirt pcas pcas_two_step lmem_en lmem_base lmem_tpc_size
+                lane_skip lane_tail r1_first recommit override_hex
+MEM WINDOWS     qmdwin shadow_win0 data_prewin ptr_guest pte_kind mail_patch
+                inst_sys gpc_mmu_init
+COMPLETION      rel_mod skedrec img_probe notifier_sys faultbuf_sys
+CE WAVES        work_forward -> work_defer_release -> work_hw_primary -> relay
+                fault_relay fault_inject
+GATES           gsp_phys_gpu mdev_owned gr_forward guest_mem static_live ctrl_forward
+```
+
+Not all of these are part of the vEP's steady state: `gsp_phys_gpu`,
+`gsp_phys_hazard_ack` and `mdev_owned` gate the physical session itself
+(docs 08/11); `static_live` and `ctrl_forward` are control-plane overlays.
+Everything else in the matrix shapes execution. The single most important
+gate is `rmcall_gr_exec_ok`: the latched proof, set only once a real GR
+submission has executed on the ladder (`gsp_phys_booter.c:14614`), that
+`gsp_phys_backend_gr_ready()` (`:14865`) requires before any guest GR launch
+may forward. A parameter alone can never make the module pretend GR works.
+
+---
+
+## 3. Design principles and implementation reality
+
+### 3.1 Principles the implementation enforces
+
+1. **Emulate the interface, execute the work.** The vEP never modifies the
+   guest driver; it satisfies the interfaces the guest observes and arranges
+   real execution underneath (project principle, doc 01 §1).
+2. **Guest truth over guesses.** Translations come from driver-published
+   structures (VAS page directories, channel descriptors) or validated
+   content, with a read-back proof before a pin is latched; rank-derived and
+   convention-derived addresses are candidates, never facts. The pinned
+   truth model is explicit in `gsp_sema_service.h` (`struct sema_channel_desc`,
+   `struct sema_vaspace_desc`).
+3. **One writer per structure.** Channel/vaspace state is copied out under
+   `svc.chan_lock`/`svc.vas_lock`; injected pins never overwrite authoritative
+   discovered data; guest-status-queue writes are single-writer in
+   `gsp_rm_send` (doc 10 §5, doc 11 §1.3).
+4. **Bounded budgets everywhere.** Bounded scans, retry counts, grace windows
+   (`sema_ce_grace_ms=25000`, `sema_gr_grace_ms=8000`, `sema_push_hold_ms=2500`),
+   a 5→60-pass discovery epoch with cooldown, bounded poll quanta (5 ms in the
+   ladder, `gsp_phys_booter.c:1404`). A miss must degrade, never freeze; the
+   starvation telemetry from ISSUE-0001 fix #2 exists to make a bound miss
+   visible.
+5. **Default-off, fail-closed, one-line rollback.** Every execution-changing
+   behavior is behind a parameter, and every parameter has a fallback story
+   (docs 11 §5). Forwarding refuses when the backend is not READY; a held
+   release is applied exactly once by software when the hardware path misses
+   (P6b wave 1.5 / P6c).
+6. **Evidence gates the latch.** A capability is enabled only when a named
+   acceptance signal has been observed (`REAL GR SEM OK` → `rmcall_gr_exec_ok`);
+   "silence is not health" (doc 03-execution-frontier §8).
+
+### 3.2 Ring discovery: kernel-native, both apertures, content-keyed
+
+The pump cannot execute anything until it can name two addresses per channel:
+the GPFIFO ring VA's translation, and the USERD's translation (PUT/GET live at
+USERD+0x8c/0x88, `SVC_USERD_GP_PUT`/`SVC_USERD_GP_GET`,
+`kernel/gsp_sema_service.c:143-144`).
+
+Today discovery has three cooperating sources, in order of preference:
+
+1. **Injected pins** (debug tooling and, historically, the userspace watcher
+   pipeline) — authoritative but external; `svc_pin_lookup_any`.
+2. **The kernel-native trusted walk** — driver-published page directories
+   recorded from `SET_PAGE_DIRECTORY` (`svc_walk_vaspaces`), a self-discovered
+   directory (`svc_pd_discover`, `svc_walk_discovered`), and per-VA targeted
+   resolution (`svc_targeted_resolve`), all through the shared four-level
+   `svc_gmmu_walk` with read-back validation (`svc_lookup_trusted`).
+3. **Content-keyed discovery** — when no translation source can name a GR
+   ring, `svc_gr_content_discover` scans for the *content* of the ring and the
+   staged ctx-init push, pairs them per rank, and validates the pair in
+   `svc_gr_content_pin` before pinning. The scan covers **both apertures**:
+   the emulated vidmem aperture (`vgpu->bars[1].vaddr`) and guest system RAM
+   through `svc_read()` = `vfio_dma_rw` (`gsp_sema_service.c:754`).
+
+The history matters because it explains the current code shape:
+
+* CE channels were discovered first and stably: the CE ring convention
+  `FB 0x80000 + 0x20000*rank` pinned 6/6 CE rings (`ring_auto=6`) in the
+  stalled v2/grv1 boots (FORENSICS-v2 §a; FORENSICS-grv1 §0.1).
+* GR rings follow `VA 0x200200000 + 0x3000*rank` with the USERD at
+  `+0x2000` (`SVC_GR_RING_VA_BASE/STRIDE`, `SVC_GR_USERD_OFF`,
+  `gsp_sema_service.c:7334-7431`), but *where the ring physically lives* is
+  boot-dependent: green m104 found all 20 GR rings at
+  `FB 0x140000 + 0x3000*rank`; grv1's guest never wrote them to FB;
+  grv3's guest wrote only ranks 0–2 (FORENSICS-grv3 §1).
+* FORENSICS-grv1 proved the ring was **not in guest RAM** and recommended an
+  FB-aperture scan; FORENSICS-grv3 identified the 60-pass epoch death and
+  recommended periodic re-arm plus an FB-first fast path. Those three changes
+  are in the tree as commits `ebcfc1c2` (both-aperture kernel-native
+  walk + content fallback), `7abd8f87` (FB-aperture discovery + per-channel
+  re-arm, FORENSICS-grv1 R1) and `66388560` (periodic re-arm + FB-first fast
+  path, FORENSICS-grv3 R1) — the current HEAD.
+
+In the current code (`svc_gr_content_discover`, `gsp_sema_service.c:7650`):
+the FB scan runs first at a page stride (milliseconds), push candidates
+persist across passes so a late ring can pair without re-walking 6 GiB, a
+registration re-arms the epoch immediately, and hitting the pass cap starts a
+30-second cooldown and then a fresh epoch while any registered rank still
+lacks a ring — never the old permanent retirement.
+
+**Known limits.** The content path is gated to ranks 0–7
+(`SVC_GR_CONTENT_RANKS=8`) and to the documented VA layout
+(`gpfifo_va == BASE + STRIDE*rank`); a layout change deliberately falls back
+to the safe paths. The walk depends on the guest publishing the owning VAS:
+grv1/grv3 saw only two `SET_PAGE_DIRECTORY` publications, and the ring VA was
+not covered by either (FORENSICS-grv1 §2). Content discovery is the fallback
+for exactly that hole; a protocol-derived source for the ring VA remains the
+durable direction (ISSUE-0001 "durable direction", `issue-number-1.md:107`).
+
+### 3.3 The semaphore service: release application
+
+`gsp_sema_service.c` is the vEP's engine room. Its service thread
+(`svc_thread`, `:13287`) loops over the open mdev, runs the pump, retries
+pending releases, and paces discovery. The release path:
+
+1. **Decode** (`svc_decode_stream`, `:5170`) turns a consumed GPFIFO entry's
+   method stream into operations; releases are recognized as
+   increment-method writes to a semaphore VA with a payload.
+2. **Translate** the target VA: `svc_lookup_trusted` first, then the pool
+   resolver `svc_pool_pa_resolve` (`:2284`) — the resolver ISSUE-0001 fix #1
+   completed by adding the driver-published VAS walk
+   (`svc_walk_vaspaces`, `:1274`); a miss is queued in `svc_pending_add`
+   (`:2472`) and retried every pass by `svc_apply_pending` (`:6201`), with the
+   per-entry `tries` telemetry added by the same fix (`7f21daa7`).
+3. **Apply** (`svc_sem_write`, `:2474`) reads the current word, writes only
+   forward (never backwards), and is the sole software author of a release;
+   a forwarded operation's releases can instead be deferred and then applied
+   exactly once by the hold's finish path (P6b wave 1.5), or authored by the
+   engine itself under P6c's relay (default off, docs 11 §P6c).
+
+The complementary **counter** path handles GR completion *counters* (distinct
+from CE tracking semaphores): `svc_counter_derive` (`:4658`) derives the
+counter slot from structural translations; `svc_counter_autoreg` registers it
+per GR channel; `svc_counter_release_apply` (`:4433`) applies decoded releases
+to the slot; `svc_counter_flush_pending` (`:4575`) lands payloads recorded
+before the slot was known; `svc_counter_publish` (`:9077`) writes the GPFIFO
+PUT into the slot when it is ahead of the current value. Two facts learned the
+hard way are now encoded: the counter pool total is fixed at 20
+(`SVC_GR_COUNTER_TOTAL`, commit `fcb96912`) because a partial live channel
+count poisons every later rank's slot; and the early publish is guarded so a
+guessed PUT is never written (the H3 finding, `gsp_sema_service.c:11448-11477`,
+FORENSICS-v2 §b/H3).
+
+### 3.4 Notifiers, GPFIFO consume, event delivery
+
+* **GPFIFO consume.** The pump reads `GP_PUT` from the USERD, walks entries
+  from its own GET, decodes, and advances; a replay-suppression path keeps
+  pin batches from re-consuming old slots (`gpfifo_skipped`), and a per-entry
+  grace window holds an untranslatable GR entry before it is consumed as
+  stale (`sema_gr_grace_ms`, default 8 s). `svc_userd_probe` (`:7099`)
+  distinguishes "unreadable" from "readable but idle (0/0)" so a zero page is
+  never mistaken for a live channel.
+* **Notifiers.** `svc_notify_complete` (`:7207`) services the channel's error
+  notifier token and posts the channel event. The notifier page address and
+  aperture come from the channel allocation descriptor
+  (`sema_channel_desc.notifier_pa/as`, `gsp_sema_service.h:31-32`), recorded
+  at channel registration.
+* **Events.** `gsp_rm_post_channel_event` (`gsp_rm_emulation.c:602`) finds the
+  guest's registered event objects (`rm->events[]`, registered from the event
+  alloc in `gsp_rm_handle_rm_alloc`, `:2443-2489`) and sends the RPC event;
+  the reply path injects MSI-X through the live eventfd
+  (`gsp_emu_inject_completion_interrupt`). A green boot shows the sequence
+  `notifier cleared … status=0xffff` (per GR channel) followed by the event
+  post; grv3 shows neither, and its three completed channels re-queued the
+  identical push and gave up — the leading reading is "counter moved, wake-up
+  never arrived" (FORENSICS-grv3 §3, hypothesis (d); not yet proven live
+  because the relevant traces are `sema_dbg`-gated).
+
+### 3.5 Physical submit
+
+Forwarding is a two-stage ladder. The sema service captures a candidate and
+queues one request on the single ordered workqueue `gsp-phys-work`
+(`svc_gr_request`, `:10991`; `svc_fwd_hold` is the single in-flight slot).
+The work item calls into the backend: `gsp_phys_backend_guest_ce_copy` for CE
+(`gsp_phys_booter.c:15621`) or `gsp_phys_backend_guest_gr_submit` (`:16457`)
+for GR, both gated by `gsp_phys_backend_work_ready()/
+gsp_phys_backend_gr_ready()` (`:14865`), which require READY, a live claim,
+an unload not committed, a client-memory ladder channel, a work token, every
+ladder VA, `gsp_phys_guest_mem` and an active guest RAM backing file.
+
+GR submission (`gsp_phys_backend_guest_gr_submit`) is the vEP's most
+parameterized function: it relocates and pre-writes the QMD (inline-QMD
+window, `gsp_phys_rmcall_gr_qmdwin`), copies program/param pages into shadow
+windows whose PTEs are planted at stage 3 before the engine's first fetch
+(`gsp_phys_rmcall_gr_shadow_win0`, `data_prewin`), builds the launch method
+block (`gsp_phys_rmcall_build_gr_launch`, `:15862`) from the recovered method
+vocabulary (compute init, inline-QMD window, PCAS pairs, report semaphore),
+publishes it to the ladder channel's GPFIFO and rings the doorbell
+(`gsp_phys_rmcall_publish_locked`, `:14345`), then polls for the release with
+`rel_mod`/`skedrec` diagnostics. The PTE fetch-line rule is enforced: the
+engine caches PTE fetches at 128-byte granularity with no invalidate path, so
+each scratch pair consumes a never-reused fetch line from pools built before
+first engine access (docs 11 §P6; `gsp_phys_booter.c:1044-1070`).
+
+Measured cost: the m141-r3 baseline (`895ab8b8`) is `elapsed_ms=189` for the
+first forwarded launch and `166` for the second, of which only ~7.9 ms is the
+post-doorbell engine path; ~180 ms is serialized per-window `PDE_UPDATE`
+forwards (local evidence and the prepared harness:
+`tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1.3). The baseline is a
+forwarding-service cost, not engine time — the number to drive down is the
+7.9 ms and the per-window round trips.
+
+### 3.6 Citation map for this section
+
+| Claim | Authority |
+|---|---|
+| Both-aperture kernel-native GR discovery + content fallback | commit `ebcfc1c2`; `gsp_sema_service.c:7334-7529` |
+| FB-aperture discovery + re-arm; periodic re-arm + FB-first | commits `7abd8f87`, `66388560`; `gsp_sema_service.c:7650-7800` |
+| Ring not in guest RAM; RAM-only scan limitation | `FORENSICS-grv1.md` §0/§1/§A (local evidence) |
+| 60-pass cap dead-end; wake-up hypothesis | `FORENSICS-grv3.md` §0/§3/§4/§5 (local evidence) |
+| CE clean consumption at 89 releases; no starvation | `FORENSICS-v2.md` §b/§c (local evidence) |
+| Pool resolver + starvation telemetry | commit `7f21daa7`; ISSUE-0001 fix #1/#2 |
+| Counter slot fixed-total 20 | commit `fcb96912` |
+| PROMOTE_CTX `bInitialize` must stay set | commit `88d47c1f`; ISSUE-0001 fix #3 |
+| GR exec latch + readiness gate | `gsp_phys_booter.c:14614`, `:14865`; `main.c:2609` |
+| PTE fetch-line rule, scratch pools | docs 11 §P6; `gsp_phys_booter.c:1044-1070` |
+| m141-r3 baseline 189/166 ms decomposition | `tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1 |
+
+---
+
+## 4. End-to-end GR execution, in our words
+
+This section defines the vocabulary as the code uses it, then walks the GR
+chain. The concepts are standard NVIDIA terminology; the definitions below
+are **ours**, written against the interfaces we implement.
+
+| Term | Our definition |
+|---|---|
+| **GR engine** | The physical engine that executes shader work: both compute grids and graphics pipelines. CUDA kernels are GR work. It is the one engine that can run arbitrary guest-authored code, which is why it is the frontier. |
+| **Channel** | A hardware execution context: a GPFIFO ring, a push-buffer pointer, a USERD control block, an instance/RAMFC block, and the engine object bound to it. The guest creates channels through RM allocs; the vEP registers them (`gsp_rm_register_channel` → `gsp_sema_service_channel_add`). |
+| **GPFIFO** | The ring of entries a channel consumes. Each entry points at a push buffer (or carries inline data); the front-end fetches it, then executes the method stream it references. |
+| **USERD** | The channel's control block in GPU memory. `GP_PUT` (offset 0x8c) is where the guest declares submissions; `GP_GET` (0x88) is the channel's consumption cursor. The pump reads these through `svc_read_ap` with the channel's aperture. |
+| **Push buffer / method stream** | The command stream: incrementing-method headers plus payload, addressing engine registers and subchannels. Decoding it is how the vEP learns what the guest asked for (`svc_decode_stream`). |
+| **QMD** | The launch descriptor: the fixed-layout structure GR reads to dispatch a grid (program address, grid/block dimensions, parameter pointers, constant-bank pointers, release targets). CUDA builds it; the forwarded path must install the inline copy into the channel's descriptor buffer in the form the engine reads (§6, the wall). |
+| **Context / ctx-init** | The per-channel engine state block (main + per-VEID subcontext images) that must be initialized and committed before GR can switch to the channel. The guest drives this with `GPU_PROMOTE_CTX`; the server-side rules are what the `gr_promote`/`srv_ctxcommit` matrix is bisecting. |
+| **Golden context** | The reference channel/3D object set the stock driver creates right after load so the server-side GR instance state exists (the `gsp_phys_rmcall_golden` replay). A context committed without that state does not dispatch. |
+| **SASS** | The compiled shader machine code the GR executes. Its pointers reach GR through QMD constant banks; for forwarded launches the vEP relocates referenced pages into engine-walkable windows rather than trusting guest pointers directly. |
+| **Semaphore release** | The completion primitive: a method that writes a payload word into a semaphore VA. CE work releases tracking semaphores; GR work releases the channel semaphore and the QMD release word. The vEP applies releases in software or, gated, lets the engine author them. |
+| **Counter** | The GR completion counter slot: the u32 the guest's userspace tracker polls for channel progress; the vEP derives, registers and publishes it (`svc_counter_*`). |
+| **Notifier** | The channel error/completion notification block. `svc_notify_complete` services it and the RM event path delivers the channel event. |
+| **Shadow window / scratch PT** | Module-owned pages mapped into the engine's page-table tree at VAs the ladder controls, used to place QMD/program/parameter/data content where GR can walk it. Window-0 PTEs are pre-planted before the channel commit because late writes into already-fetched PTE lines are invisible. |
+
+**The chain the vEP implements for GR:**
+
+1. **Context creation.** The guest RM allocates VRAM objects, a VAS, a TSG,
+   GR channels; the vEP registers each channel at alloc time and records its
+   descriptor. The guest's `SET_PAGE_DIRECTORY` publications become the
+   trusted translation sources. The guest then promotes its GR context
+   (`PROMOTE_CTX`); the emulation acknowledges and keeps `bInitialize` set
+   (fix `88d47c1f`) so the client marks its buffers initialized.
+2. **Golden context / server state.** On the physical ladder, the
+   `golden` replay allocates the stock client's post-load channel set so the
+   server-side GR instance state exists; the context commit variants
+   (`gr_promote`, `srv_ctxcommit`, `vas_shared`) are the ladder's bisection of
+   what the real server requires. The vEP's guest side does not depend on the
+   ladder shape; the gate is the latch.
+3. **QMD capture.** The pump decodes the guest's launch stream, accumulates
+   the inline QMD window (`svc_qmd_accum`), and captures the launch candidate
+   (`svc_gr_cand_capture`), including the release target and the referenced
+   program/param pages.
+4. **Shadow windows.** The submit relocates the QMD and copies referenced
+   pages into window-0 shadow pages whose PTEs were planted pre-commit;
+   data pages go through the pre-installed data pool; `mail_patch` injects a
+   module message page when the guest's constant bank is empty. `ptr_guest`
+   is the A/B for keeping the guest's original pointers (the SM walk appears
+   to resolve through the guest subcontext PDB; m91).
+5. **Physical submit.** The submit builds the launch method block, publishes
+   it to the ladder channel's GPFIFO, rings the doorbell, and polls for the
+   engine-authored release.
+6. **SASS on GR.** Between publish and release, the front-end executes the
+   methods; the inline-QMD install and the engine's read of the descriptor
+   are the stages the current campaigns have not yet observed. Once they
+   fire, GR switches to the context and dispatches the grid to the SMs.
+7. **Completion.** The engine authors the QMD release word and the channel
+   release; the poll observes them; the deferred release is applied once
+   (P6b wave 1.5 semantics); the counter is published; the notifier is
+   cleared; the channel event is posted; MSI-X wakes the guest. For GR the
+   last step is the open wake-up question (§6.1).
+
+---
+
+## 5. Enabling arbitrary CUDA code execution
+
+"Arbitrary CUDA" is the 1.0 acceptance bar: stock guest CUDA works for
+real workloads, not one demo, and every time. The program is staged:
+
+* **P1 — single-workload A/B.** `tools/cuda-ab/ab.sh` runs one deterministic
+  workload on the host (bare-metal oracle) and in the guest, then compares
+  the stable result lines. Workloads: `victory_mail` (the campaign mail
+  kernel), `vector_add`, `smiley` (an ASCII-art deliverable). This is the
+  rung that produced the m141-r3 first green forwarded launches in 189/166 ms
+  (`895ab8b8`).
+* **P2 — the "arbitrary CUDA kernels" acceptance suite.**
+  `tools/cuda-ab/p2/` is eleven self-checking programs covering launch shape
+  (1D/2D/3D grids, odd sizes), parameter counts 0–12 with mixed types,
+  register/launch-bounds variants, shared memory (reduction + tiled GEMM),
+  streams including eight independent kernels in flight, allocator churn,
+  device printf/assert, error reporting and recovery, a correctly-rounded
+  compute benchmark, and a combined end-to-end program. References are
+  deterministic (EMIT mode) until the host GPU is free for capture; the
+  suite currently runs `[planned]` pending the reproducibility gate
+  (`tools/cuda-ab/READINESS-P2-2026-10-08.md`).
+* **P3 — near-native performance.** `tools/cuda-ab/perf/` is the prepared
+  measurement program: host-oracle vs guest ratios for init, malloc, launch
+  (first/warm/throughput), HtoD/DtoH bandwidth, compute GFLOP/s, plus host
+  metrics (the backend `elapsed_ms`, injected-latency proxies). Status:
+  prepared, not yet run; every threshold provisional until a green
+  calibration (`tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1/§4).
+
+**Proven vs pending.** Proven today: single forwarded launches of two small
+kernels (mail, vadd) with correct release and clean teardown, on a lucky
+green boot; CE-class copies forwarded end-to-end with engine-authored release
+and fault relay (default off); compiled kernels of the P2 complexity are not
+yet observed executing through the stack. Pending: reproducibility
+(ISSUE-0001), the GR descriptor-install wall (§6.1), completion wake-up
+(§6.1), multiple kernels/clients per VM (ISSUE-0002), and P2/P3 in a green
+boot. The gating order is deliberate: reproducibility first, then the wall,
+then breadth, then performance.
+
+---
+
+## 6. Challenges
+
+### 6.1 Current
+
+* **The GR completion wall — two faces.** The older face is the
+  launch-descriptor install: the forwarded launch reaches and runs
+  front-end methods, the descriptor destination window is accepted, the
+  context is valid at dispatch, and yet the inline-QMD install is never
+  observed and no grid dispatches
+  ([`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)
+  §1–§5). The exonerated list is long (context validity, destination mapping,
+  entry fetch/alignment, TSG parenting); the live issues are the stream's
+  verbatim prelude, the absent descriptor-virtualization control method, and
+  the relocation transform, each with a one-cycle falsifier. The current code
+  carries many of those falsifiers as mode-matrix knobs (m26 falsified the
+  nonzero `qmdvirt` hypothesis; m72 found the SKED `LOCAL_MEMORY_TOTAL_SIZE`
+  cancel and added the `lmem_en` block; m91 showed the SM I-fetch MMU_NACK at
+  shadow VAs and added `ptr_guest`).
+* **The second face — completion wake-up.** FORENSICS-grv3 shows a boot where
+  discovery succeeded (three GR rings pinned), the ctx-init releases landed
+  (`gr_rel=3`, counter words = 1), and the guest *still* stopped: it
+  re-queued the identical push and gave up. The best-supported reading is
+  that the client waits for the completion *wake-up* (notifier clear + event),
+  not the counter value; green boots show `notifier cleared` ×N and the event
+  post, grv3 shows neither (FORENSICS-grv3 §3). The one-variable experiment is
+  a `sema_dbg=Y` cycle to un-gate the notifier/publish/post_event traces.
+* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The
+  first fix wave landed (kernel-native pool translation `7f21daa7`,
+  PROMOTE_CTX reply `88d47c1f`, issue register `a1407659`), and the
+  earlier stall mode — the client freezing at DEVICE because its last CE
+  completion could not be written — is fixed: the v2 campaign consumed all 89
+  CE pushes with zero holds and zero starvation. The stall then moved one
+  phase later, to GR ring discovery, where the tree now carries the
+  kernel-native discovery lineage (`1ba8ba65`, `ebcfc1c2`) plus the fixed
+  re-arm of `7abd8f87`/`66388560`. Acceptance is 10/10 consecutive cold boots
+  green (`issue-number-1.md` validation plan); that count is pending.
+* **Multiple kernels/clients per VM (ISSUE-0002).** A second CUDA process in
+  one boot wedges in guest UVM because the host's per-client sema/GR state is
+  not reset when the guest re-runs the GSP boot handshake per client; the
+  guest's UVM holds a global lock across the registration RM exchange, so a
+  host-side quiet path freezes every later client (`issue-number-2.md`). The
+  durable direction is per-client/per-channel state reset plus per-mdev sema
+  contexts and P0 isolation work, with the ordered hunks already recorded
+  (validation plan in the issue).
+
+### 6.2 Historical (fixed or retired — do not relitigate)
+
+* **CE starvation at the boot-critical release.** The old stall was one
+  deferred CE release whose semaphore page had no translation, waiting on a
+  racy userspace watcher. Fixed kernel-natively by the pool resolver's VAS
+  walk (`7f21daa7`) and the GR ring/USERD discovery lineage; `pushes_nomap`
+  now reaches 0 on consumed boots (FORENSICS-v2 §0/§c).
+* **The ring-location class.** Early discovery tried only FB conventions
+  (`0x140000 + 0x3000*rank`) and RAM scans; grv1 proved the ring was not in
+  guest RAM and the RAM-only content scan burned 60 passes. The fix is the
+  both-aperture, FB-first, re-armable discovery now in the tree
+  (`7abd8f87`, `66388560`).
+* **The notifier gate as a discovery symptom.** The `notifier cleared` line
+  historically correlated with ring resolution; it is a publication-side
+  signal and is now understood as the wake-up face of §6.1, not a discovery
+  fix in itself.
+
+### 6.3 What is working well, and what is not
+
+**Working well:** the CE path end-to-end (copies, deferred exactly-once
+release, hardware-primary data, engine-authored relay, fault relay) — all
+live-verified default-off; kernel-native discovery and the trusted-walk
+truth model; deterministic counter semantics with fixed pool total; the
+mode-matrix discipline (one variable per supervised cycle, one-line
+rollback); the bounded-budget culture that turns potential freezes into
+logged, counted degradations.
+
+**Not working:** GR dispatch (no grid observed); completion wake-up for GR;
+reproducibility at the required 10/10 bar; second-client state isolation;
+and the observability gaps that made both bugs slow to find — counter
+`pub_calls` counting calls rather than writes (H3; the increment still sits
+before the no-op guards at `gsp_sema_service.c:9085` and correction is
+planned, FORENSICS-grv1 §5 R3),
+`sema_dbg`-gated traces hiding the wake-up path, and the absence of a GR
+error-reporting object on the forwarded path ("silence is not health",
+doc 03-execution-frontier §8).
+
+---
+
+## 7. Goals, gaps, and the iteration plan
+
+### 7.1 Goals
+
+1. **Correctness:** a stock guest runs arbitrary CUDA in one boot, repeatedly.
+2. **Reproducibility:** 10/10 cold boots green (ISSUE-0001 exit criterion).
+3. **Breadth:** P2 suite green in a single boot; multi-client and multi-stream
+   (ISSUE-0002); then multi-mdev.
+4. **Performance:** near-native P3 numbers with no-regression discipline.
+5. **Portability:** the vEP survives driver-version and GPU-generation churn by
+   re-deriving facts, not by patching snapshots.
+
+### 7.2 The iteration loop
+
+Every change follows the same evidence loop. This is the operating system of
+the vEP program, not an appendix:
+
+1. **Question as a discriminating sentence + named acceptance signal.** E.g.
+   "does the notifier clear and the event post fire for a GR ctx-init
+   channel?" with the signal `notifier cleared` + `post_event` + guest
+   advancement past the re-queue.
+2. **Offline first.** Use the risc-v-gsp-rm-debugger (`debug/risc-v-gsp-rm-vm/`)
+   and its validation tools — `harness/valprobe.py` for one firmware
+   validation branch, `harness/rehost.py` for one whole function, RPC
+   capture/decode/compare (`debug/risc-v-gsp-rm-vm/docs/45-47`) — to test a
+   firmware-side rule without a supervised physical cycle. The entry points
+   are [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md)
+   and [`../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md`](../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md).
+3. **One supervised cycle, one variable.** A mode-matrix knob, a trace flag,
+   or a single code change; never two.
+4. **Evidence pair.** Session/counter pair (green vs this run) with the
+   discriminators named before the run; campaign evidence goes to
+   `backups/evidence-…/` (local), the public claim cites the behavior only.
+5. **Land or revert.** A landed change updates the doc and (if needed) the
+   issue; a falsified hypothesis is recorded in the knob comment so no future
+   session re-runs it.
+
+### 7.3 Tooling gaps and the plan for them
+
+| Gap | Tool / practice | Plan |
+|---|---|---|
+| GR rule recovery without burning cycles | risc-v-gsp-rm-debugger valprobe/rehost, RPC capture/replay (docs 45–47) | probe candidate front-end/engine rules offline; queue only falsifiable physical experiments |
+| Arbitrary-kernel acceptance | `tools/cuda-ab/p2/` (11 programs) | run in the first green boot after ISSUE-0001 acceptance; add a multi-process/multi-stream variant for ISSUE-0002 |
+| Performance measurement | `tools/cuda-ab/perf/` + `collect.py` | freeze thresholds on first green run; make the serialized `PDE_UPDATE` forwards the first optimization candidate |
+| Wake-up observability | unconditional or counted `notified/posted` telemetry in `svc_notify_complete`/`gsp_rm_post_channel_event` (FORENSICS-grv3 §5B) | land as telemetry, not a behavior change; treat `posted==0` after a real clear as a first-class failure |
+| Counter telemetry honesty | move the `pub_calls` increment after the write (FORENSICS-grv1 §5 R3) | land with the telemetry wave |
+| Multi-client state | ISSUE-0002 fix plan (per-client reset, per-mdev sema context, P0 hunks) | stage A1–A6 per the recorded plan; graduation criteria sequential/concurrent/simultaneous |
+| Evidence discipline | one evidence bundle per claim; counters as first-class discriminators; "no claim without a pair" | already the campaign norm (FORENSICS reports); keep counters stable enough to be quotable (see counter telemetry fix) |
+| Sub-agent workflows | read-only analysis sessions (`SA-ANALYSIS`, `SA-CHK`, `SA-DOC`) producing FORENSICS/readiness docs while the lead runs cycles | continue; keep analysis strictly read-only and cite line numbers; never touch the rig in analysis sessions |
+
+---
+
+## 8. Portability
+
+### 8.1 What is version-portable
+
+* The **architecture** of the vEP: observe pushes → resolve truth → decode →
+  execute/forward → publish completion. This shape does not depend on a
+  driver version or a GPU generation; only the facts it consumes do.
+* The **RPC/RM object model** the vEP registers against (channel alloc
+  descriptors, page-directory publications, event objects, `PROMOTE_CTX`)
+  is stable in structure across the 5xx/6xx families; new fields appear, but
+  the wire structures are versioned and compiled into the driver.
+* The **mode-matrix method**: parameterized, default-off, rollbackable
+  experiments and an evidence latch (`rmcall_gr_exec_ok`) port as a practice.
+* The **counter/notifier/event completion model** (a polled u32, a notifier
+  block, an RM event, MSI-X) is the same completion vocabulary across
+  versions.
+* The **QMD vocabulary at a coarse level** (a launch descriptor with program,
+  grid, parameters, release targets) is stable; exact field offsets are not.
+
+### 8.2 What must be re-derived per version or architecture
+
+| Item | Why | Re-derivation source |
+|---|---|---|
+| Firmware/driver identity, RPC inventory, control IDs | versions add/retire controls | measurement: `firmware_analyzer` signatures, live RPC captures (`docs/45`), host-oracle A/B (docs 07 P1) |
+| Ring/USERD VA and FB layout conventions | the guest's memory allocator changes with the driver | first-touch probes + content discovery; never hardcode beyond the documented candidate list |
+| QMD layout and launch method stream | the launch vocabulary changes (Turing → Ampere → Hopper → Blackwell) | reference capture of a working launch; differential against the forwarded stream |
+| Context image layout, promote entry sets, golden-context shape | server-side RM rules and image formats change | ladder bisection + offline valprobe/rehost |
+| PTE encodings, page-table geometry, fetch-line rule | GMMU changes per architecture | hardware observation; document the encoding proven in each profile |
+| Engine inventory and classes | missing/zeroed records roll whole contexts back (doc 05 §6) | bare-metal oracle capture; profile tables |
+| Scratch-window VA layout and pools | ours, but coupled to the ladder channel's VAS geometry | re-lay per profile; keep the pre-commit planting rule |
+
+Portability discipline: **protocol-level facts port; address-level facts do
+not.** Any new version or architecture enters as a profile plus a measured
+delta, never as a guess (docs 06 §5, `RUNBOOK.md` S-scenarios).
+
+### 8.3 Concrete paths
+
+* **Driver 610.** Hard gate first: RM 610 requires explicit named-human
+  approval — no acquisition, analysis, emulation edit, physical work or
+  support claim before it (`docs/23-RM610-READINESS.md`). Once approved:
+  acquire the image, run one-command triage
+  (`risc-v-gsp-rm-debugger.sh triage`, `docs/40`), diff the RPC inventory
+  with `harness/version_diff.py`/`ci_release.py` (`docs/21`), then port the
+  vEP facts in the order of §8.2. This is speculative until the gate opens;
+  nothing in the current tree pre-adapts to 610.
+* **Grace Blackwell (GB10/GB2xx, aarch64 hosts).** A different problem class:
+  different host CPU architecture (the module's guest-RAM translation,
+  `read_cache_page`-based HPA mapping, must remain architecture-neutral), an
+  engine inventory and GMMU generation with no current profile, and a
+  firmware family the debugger would need to re-host/port (docs 07 P3). The
+  vEP plan is: profile capture (PCI IDs, BAR geometry, engine list), GMMU
+  encoding work, then the same discovery/decode/submit ladder on a supervised
+  device, with the support ladder (docs 06 §1) moving only on evidence.
+* **Compute profiles.** The near-term "profile" is a small, per-version file
+  of measured facts (ring conventions, VA windows, QMD offsets, method
+  encodings, PTE kind) consumed by the vEP at runtime; today those facts are
+  in code comments and constants with commit provenance, which is the
+  pre-profile state.
+
+---
+
+## 9. Mapping vEP concepts to CUDA and to architecture
+
+### 9.1 CUDA high-level concepts
+
+| CUDA concept | What the guest does | vEP counterpart |
+|---|---|---|
+| Context | `cuCtxCreate`; RM client/device/subdevice + a GR engine context | guest context is emulated (objects, VAS, promote); the physical ladder commits its own context (`gr_promote`/`srv_ctxcommit`) |
+| Module / function | loads SASS/cubin into GPU memory; `cuModuleGetFunction` | SASS reaches GR through QMD program pointers; the vEP relocates referenced pages into engine-walkable windows (`shadow_win0`, `data_prewin`, `ptr_guest`) |
+| Stream | ordered work queue on a channel (or a graph) | channel + TSG; the guest's push order on the channel is the stream order the pump preserves |
+| Kernel launch | builds push (ctx init, QMD install, params), appends GPFIFO entry, rings doorbell | capture → one in-flight request → ladder submission; the QMD install is the wall (§6.1) |
+| Memory | `cuMemAlloc`/copies; VA objects mapped into the VAS | emulated VRAM + guest RAM with the trusted truth model; forwarded operands resolve VA→PA→HPA via the guest RAM backing |
+| Synchronisation | events, stream sync, semaphores, `cudaDeviceSynchronize` | releases, counters, notifiers, channel events; the vEP is the sole publication authority (or relays engine-authored releases) |
+
+### 9.2 Architecture details
+
+| Architecture concept | vEP relationship |
+|---|---|
+| Channel / GPFIFO / USERD | the unit of scheduling and consumption (§4); registration from the alloc descriptor |
+| TSG / runlist | the guest's scheduling vocabulary; the ladder's channel sits in a TSG on the real server; guest-side pacing is the pump today, tenant fairness is planned on top (doc 09 §5) |
+| Engine types / classes | GR is engine type 1, COPY0 is 0x9 (doc 05 §6); the ladder selects GR0 for GR work and refuses CE paths on a GR ladder (`gsp_phys_backend_ladder_ready`) |
+| GMMU | the translation ground truth; the trusted walk reads guest page tables, and the engine's view is fed by planted PTEs in pre-built fetch lines |
+| VRAM / FB apertures | emulated as a host buffer today; apertures appear in channel descriptors and pins (`SVC_APERTURE_SYS/FB`); real VRAM backing is future work |
+| MSI-X / interrupts | completion delivery: `gsp_rm_post_channel_event` → eventfd → irqfd; physical-engine faults are observed and relayed (C3, default off) |
+
+---
+
+## 10. Future challenges the current tools do not address
+
+These are the known gaps for which no tool or design exists yet. Each gets a
+proactive design direction so the architecture is not surprised later.
+
+1. **Multi-tenant fairness and performance isolation.** Today one mdev, one
+   real session, one in-flight request. The planned per-mdev real clients
+   need a policy layer (time slices, quotas, accounting) and a measurement
+   method that can fail (doc 09 §5; docs 06 §2.8). Proactive design: keep
+   every per-client datum inside a container keyed by the mdev/session, and
+   build the reset path (ISSUE-0002) as the container's lifecycle, not as
+   scattered clears.
+2. **Fault recovery semantics.** Fault relay exists (C3); fault *replay*,
+   *cancel*, and channel recovery after a fault do not. Proactive design:
+   model faults as first-class completion outcomes in the request record, so
+   a fault can trigger the same exactly-once release discipline as success.
+3. **Context switching and preemption between tenants.** The ladder leaves a
+   context loaded; the client preempts its TSG for a clean first dispatch
+   (`lane_tail`). A multi-tenant scheduler must own load/unload lifecycles.
+   Proactive design: the mode matrix's lane controls are the seed of a
+   documented context-switch protocol, not a launcher-only trick.
+4. **Hostile-guest safety.** A malformed push, a forged VA, or a wedged ring
+   must not corrupt the host or another tenant. Guards exist (bounds checks,
+   read-back proofs, `gsp_phys_seq_addr_valid`), but there is no fuzzing
+   harness for guest-authored streams. Proactive design: a synthetic-push
+   generator against the decoder with the same evidence discipline as the
+   physical ladders.
+5. **Observability as a product property.** The vEP's bugs were found by
+   counter archaeology; counters that count the wrong thing (H3) cost days.
+   Proactive design: every counter must answer one named question, and
+   telemetry for the wake-up path lands before it is needed again.
+6. **Version drift under an adversarial schedule.** New images will arrive
+   without notice; the triage/porting tooling exists but is unexercised on a
+   real new version. Proactive design: golden sessions per supported version
+   and a one-command triage that can fail loudly.
+7. **Windows guests and non-KVM transports.** They change the RAM/IRQ
+   plumbing under the vEP, not its pipeline. Proactive design: keep the
+   guest-memory accessor and the event injection behind interfaces (they
+   already are: guest-RAM backing file, eventfd), so the pipeline is
+   transport-agnostic.
+
+---
+
+## 11. 3D graphics interaction
+
+Graphics workloads (OpenGL/Vulkan) are GR work too, but they differ from
+compute in ways the vEP will feel directly:
+
+* **Engines:** more than GR — a 3D context still uses GR (graphics
+  pipelines), but video (NVDEC/NVENC) and image (NVJPG/OFA) engines join
+  later; CE remains for buffer moves; the engine inventory is a correctness
+  surface (doc 05 §6).
+* **Channels and sync:** drivers use multiple channels per context, TSGs for
+  scheduling groups, and inter-engine semaphores/fences (a copy waits on a
+  render; a present waits on a copy). The vEP's release/counter vocabulary
+  covers the primitives; the *volume and cross-engine ordering* of
+  synchronisation is higher.
+* **Pushes:** the graphics method vocabulary is much larger (pipeline state,
+  shader binding, render targets, viewports), and state lives in big
+  per-context image blocks. The current capture/decode path is compute-first;
+  a graphics capture would need the same treatment the CUDA launch got
+  (reference method stream, differential against the forwarded stream).
+* **Memory:** surfaces, tiled/swizzled layouts, render targets in VRAM; the
+  GMMU and aperture handling are the same, but the emulated host-RAM VRAM
+  becomes a real fidelity problem for tiling/swizzle expectations.
+* **Interaction with the current wall:** the descriptor-install wall is
+  shared — a graphics draw also dispatches through GR descriptors. Ceiling
+  note: the current docs mark OpenGL/Vulkan `not yet` (docs README status
+  table); the 3D plan is to reuse the vEP pipeline once compute dispatch is
+  proven, then add a graphics-capture workstream (mode-matrix discipline
+  applies: one class/method family at a time).
+
+---
+
+## 12. Overarching framing: why the vEP is a core problem
+
+**What vGSP is.** vGSP is the virtual GPU System Processor: the host-side
+reimplementation of the GSP/GSP-RM interface that lets a stock guest driver
+believe it owns a real NVIDIA GPU. `vgsp.ko` is vGSP: the virtual device, the
+emulated boot and message transport, the RM object/control service, and the
+execution plane. The guest never learns it is virtualized for these paths,
+and no component in the stack — host driver, guest driver, firmware, or
+hypervisor — is modified (docs 01, 09 §13).
+
+**Why the execution plane is one of vGSP's core problems.** Everything else
+in vGSP is necessary but not sufficient: the control plane makes the driver
+run, but only the execution plane makes applications *work*. It is where the
+project's hardest constraints collide:
+
+* **Correctness without cooperation.** The guest does not help; the vEP must
+  recover VA/PA truth, ring locations and completion semantics from
+  observation, and satisfy contracts (completion, ordering) that hardware
+  would otherwise enforce.
+* **Performance without emulation.** Compute cannot be emulated at useful
+  speed; the plan of record is real execution on real engines, which means
+  every semantic the emulation would have provided for free (memory
+  visibility, ordering, faults) must be provided by the forwarding path.
+* **Isolation across tenants.** The same plane is where the multi-tenant
+  promise is kept or broken: one physical GPU's engines shared by mdevs
+  requires per-client state, scheduling and fault containment — the
+  ISSUE-0002 class today, the fairness policy tomorrow.
+
+**Fit with HAAEEP and the ecosystem.** HAAEEP's premise is compute at the
+edge — local inference, full isolation, no cloud dependency, no license
+(`docs/08-ecosystem/haaeep-and-collaboration.md` §1/§3). That premise is only
+real if a guest can run its
+own CUDA workloads on hardware it owns, under vGSP, with isolation that a
+security stack can trust. The vEP is the subsystem that turns that premise
+from an interface emulation into execution; its progress metrics
+(reproducibility, arbitrary-kernel acceptance, near-native performance,
+per-tenant fairness) are the ecosystem's prerequisites, not internal
+milestones. When this document says "the wall", it is naming the last
+technical barrier between the project's architecture and its promise.
+
+---
+
+## See also
+
+* [`01-system-overview.md`](01-system-overview.md) — principles and layer map
+* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) — GMMU, channels, GPFIFO, GR, QMD concepts
+* [`09-full-system-picture.md`](09-full-system-picture.md) — the end-to-end map
+* [`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md) — interaction, locks and scheduling detail
+* [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) — the physical session and the P0–P6 ladder
+* [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) — the GR launch chain and its leading edge
+* [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) — the execution gate
+* [`../../debug/issue/issue-number-1.md`](../../debug/issue/issue-number-1.md) and [`issue-number-2.md`](../../debug/issue/issue-number-2.md) — reproducibility and multi-kernel tracking
+* [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md) — the offline firmware debugger used for offline-first work
+* [`../../wiki/Execution-Plane.md`](../../wiki/Execution-Plane.md) — the condensed public wiki version of this document
+
+## Change log
+
+| Date | Change |
+|---|---|
+| 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. |
\ No newline at end of file