path: root/docs/03-architecture/13-vexecution-plane.md

# The vGSP Execution Plane (vEP)

> **What this document is.** The canonical architecture and design document for
> the layer of `vgsp.ko` that turns guest-authored GPU work into physical
> execution and guest-visible completion: from a guest CUDA kernel launch's
> GPFIFO push, through ring discovery, VA/PA truth, semaphores, counters and
> notifiers, the QMD and shadow windows, submission to the physical engines,
> and the completion published back to the guest. It names that layer
> **vGSP Execution Plane (vEP)**, defines its boundaries and its four work
> domains, and maps every major mechanism to the code that implements it.
>
> **Audience.** Developers and architects working on guest execution, the
> forwarding ladder, or the GR launch frontier. Read
> [`01-system-overview.md`](01-system-overview.md) and
> [`09-full-system-picture.md`](09-full-system-picture.md) first for the whole
> machine; read [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)
> for the physical-session lifecycle this document builds on.
>
> **Status: 2026-10-08, tree `master @ 1f571dda`.** Every empirical claim below
> cites a commit, a code location, a tracked document, or a local campaign
> evidence file (paths under `backups/evidence-…/` are local evidence
> artifacts, not shipped in the repository). Where a mechanism is a leading
> hypothesis rather than a proven fact, the text says so. Nothing in this
> document is a stability or support claim.

Status tags used throughout (same convention as docs 09/10/11):
`[live]` proven with the stock driver; `[gated]` implemented but default-off or
parameter-gated; `[wip]` partially implemented; `[wall]` the current blocker;
`[target]` the named success latch; `[planned]` designed, not implemented.

---

## 1. What the vEP is: name, boundaries, domains

### 1.1 The name and why

**Canonical name: vGSP Execution Plane (vEP).**

> The **vGSP Execution Plane** is the host-side subsystem of `vgsp.ko` that
> turns guest-authored GPU work into physical execution and guest-visible
> completion. It spans channel-ring and USERD discovery, GPU VA-to-PA truth,
> semaphore-release application, completion-counter and notifier publication,
> QMD handling for forwarded launches, and submission to the physical GPU's
> engines under the module-owned physical-backend session.

Two framing terms, used consistently from here on:

* **vGSP (virtual GSP)** — our host-side reimplementation of the GSP/GSP-RM
  interface: the virtual device plus the emulated firmware service that the
  stock guest driver interacts with. `vgsp.ko` is vGSP's implementation. The
  guest believes it is talking to its GPU's GSP; vGSP is the thing it is
  actually talking to.
* **vEP** — the execution half of vGSP. The *control plane* (boot handshake,
  message queues, RPC, RM objects, BAR shadows) makes the guest driver run;
  the *execution plane* makes the guest's work run. This document covers the
  second half and only the seams of the first that shape execution.

The name is **ours**, the name of a subsystem we implement. It is deliberately
not a vendor term: the three-letter form reads as the sibling of the existing
project vocabulary (`vGSP`, `vGPU`, `vgsp.ko`), and "execution plane" states
the job (execute guest work) rather than the mechanism (forwarding), which
leaves room for the fast paths that stay in software.

**Alternates considered and rejected:**

| Rejected name | Why not |
|---|---|
| "Work plane" / "work path" | Already used in [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and [`09-full-system-picture.md`](09-full-system-picture.md) for the software-executed bridge inside the pump; reusing it would silently change the documented model. |
| "Submission plane" | Covers only the submit half; hides discovery, translation, counters and completion — over half the subsystem. |
| "Forwarding plane" | Mechanism-centric: forwarding is one execution mode (some work is applied in software by design), and "forwarding" already means a specific ladder in docs 11. |
| "Data plane" | Networking connotation; the plane's most latency-critical work is state operations (releases, fences), not bulk data. |
| "vGPU Work Engine" | "Engine" is the hardware term for GR/CE; naming software an engine invites confusion in every review. |
| "GPU Execution Plane (GEP)" | Reads as a hardware/PCIe concept, not as our software subsystem. |
| "RAS plane" (Ring/Alloc/Sem) | Cryptic, and it omits the completion/publication half. |
| "vGSP Runlist" | Collides with the hardware runlist/TSG concept it actually schedules onto. |

### 1.2 Scope: in and out

**In scope (implemented by the vEP):**

* the service thread and its per-channel pump
  (`svc_thread`, `svc_pump_channels`, both `kernel/gsp_sema_service.c`);
* ring/USERD discovery, autopin and the VA-to-PA truth model
  (`svc_ring_autopin`, `svc_gr_ring_autopin`, `svc_gr_content_discover`,
  `svc_chan_userd`, `svc_lookup`, `svc_walk_vaspaces`, `svc_gmmu_walk`,
  `svc_pd_discover`, `svc_targeted_resolve`, `svc_pool_pa_resolve`);
* push decode and QMD capture (`svc_decode_stream`, `svc_qmd_accum`,
  `svc_gr_qmd_decode`, `svc_gr_cand_capture`);
* release application and completion publication (`svc_sem_write`,
  `svc_counter_release_apply`, `svc_counter_derive`, `svc_counter_autoreg`,
  `svc_counter_flush_pending`, `svc_counter_publish`);
* notifier and channel-event delivery (`svc_notify_complete`,
  `gsp_rm_post_channel_event`);
* physical submission and its mode matrix (`svc_gr_req_fill`,
  `svc_gr_request`, `svc_fwd_gr_work`, `gsp_phys_backend_guest_gr_submit`,
  `gsp_phys_backend_guest_ce_copy`, the ladder channel and its VAs);
* the pacing of all of the above (doorbell steering, discovery cadence, the
  single-inflight ordered workqueue) and its sharing across mdevs.

**Out of scope (context, not this layer):** the GSP boot handshake and
message-queue transport ([`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md)),
the RM control/object surface as a whole
([`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §4, `kernel/gsp_rm_emulation.c`),
BAR shadows and MSI-X injection
([`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md)),
the physical claim/boot lifecycle
([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)), and the
guest stack ([`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md)).
The vEP *uses* those interfaces; it does not own them.

### 1.3 The four work domains

The vEP can be described as one pipeline, but it answers four distinct
questions. This mapping is the document's organizing model.

| Domain | Question | vEP components | State |
|---|---|---|---|
| **Scheduler** | Which guest channel's work runs next, and at what pace? | Doorbell token steering (`gsp_sema_service_doorbell`, called from `main.c:865`); per-pass channel pump (`svc_pump_channels`); discovery cadence (`svc_gr_content_discover`); single-inflight ordered workqueue `gsp-phys-work` (`svc_gr_request`, `svc_fwd_request`); deferred-release hold (`svc_fwd_hold`); the service thread loop (`svc_thread`) | `[live]` software pacing and ordering; `[planned]` tenant fairness |
| **Allocator** | Where do rings, USERDs, counters, notifiers, scratch pages and PT windows live, and who owns them? | CE/GR ring autopin and pins (`svc_ring_autopin`, `svc_gr_ring_autopin`, `svc_pin_store`/`svc_ipin_store`); GR content-keyed ring/push pairing (`svc_gr_content_discover`, `svc_gr_content_pin`); counter slot derive/autoreg (`svc_counter_derive`); ladder scratch windows and PT pools (window-0 shadow pool, data-prewin pool, the 128-byte PTE fetch-line pool in `gsp_phys_booter.c`); the BAR1 vidmem shadow (`vgpu->bars[1].vaddr`) | `[live]` discovery, pins, scratch pools; `[wip]` protocol-derived allocation; `[planned]` real VRAM backing |
| **Control** | Which RM objects and controls shape execution, and how are they emulated or forwarded? | Channel registration from `GSP_RM_ALLOC` (`gsp_rm_register_channel` → `gsp_sema_service_channel_add`); event registration; `SET_PAGE_DIRECTORY` recording; `PROMOTE_CTX`; engine/class topology; the mode matrix (§2.3) | `[live]` registration + emulate; `[gated]` read-only control forwarding; `[planned]` alloc/free forwarding |
| **Share** | How do clients and mdevs share one physical GPU with isolation and fairness? | mdev backend refcount and ownership (`gsp_phys_backend_start/stop`); one real GSP session serialized by `gsp_phys_rm_lock`; one ordered workqueue and one in-flight request slot; per-mdev emulation context; planned per-mdev real clients and fairness policy | `[live]` single-tenant; `[wip]` per-client state reset (ISSUE-0002); `[planned]` multi-tenant |

The four domains cut across one code path. The rest of this document follows
that path: the architecture and its diagrams (§2), how each stage actually
works today (§3), the end-to-end GR chain (§4), then the program-level goals,
challenges, portability and future work.

---

## 2. Architecture

### 2.1 Component map

```mermaid
flowchart TB
  subgraph GUEST["Guest VM - stock software"]
    APP["CUDA application"]
    LCU["libcuda - builds push + QMD"]
    GKO["nvidia.ko - RM + UVM"]
    APP --> LCU --> GKO
  end

  subgraph VDEV["vgsp.ko - guest-facing"]
    BAR["BAR0 shadow + doorbell 0x88704"]
    RM["GSP-RM emulation - controls, allocs, events"]
  end

  subgraph VEP["vGSP Execution Plane"]
    DB["Doorbell steering"]
    PUMP["Channel pump svc_pump_channels"]
    DISC["Ring/USERD discovery + autopin"]
    TRUTH["VA-to-PA truth - walk, pins, rescue"]
    DEC["Push decode + QMD capture"]
    REL["Release apply + counters + notifiers"]
    WQ["Ordered workqueue gsp-phys-work"]
    SUB["Ladder submission gr/ce"]
    MODES["Mode matrix"]
    PUMP --> DISC --> TRUTH --> DEC --> REL
    DB --> PUMP
    DEC --> WQ --> SUB
    MODES --> SUB
  end

  subgraph BE["Physical backend - real GSP session"]
    LAD["Module-owned ladder channel"]
    GSP["Real GSP-RM server"]
    ENG["GR and CE engines"]
  end

  subgraph HW["GPU hardware"]
    GMMU["GMMU page tables"]
    VRAM["VRAM / FB"]
  end

  GKO -->|"MMIO + DMA + MSI-X"| BAR
  BAR --> DB
  RM -->|"channel/event/vas registration"| PUMP
  REL -->|"release, counter, notifier, event"| RM
  RM -->|"MSI-X"| GKO
  TRUTH -->|"reads guest page tables"| GKO
  SUB --> LAD --> GSP
  LAD -->|"GPFIFO + doorbell"| ENG
  GSP --> ENG
  ENG -->|"release semaphore"| SUB
  ENG --> GMMU
  SUB -.->|"planted PTEs"| GMMU
  VRAM -.-> GMMU

  classDef done fill:#d5f5d5,stroke:#1e7a1e
  classDef wip fill:#fff3cd,stroke:#9a7b1a
  classDef plan fill:#f8d7da,stroke:#9a1a1a
  classDef hw fill:#e2e8f0,stroke:#4a5568
  class DB,PUMP,DISC,TRUTH,DEC,REL done
  class WQ,SUB,MODES wip
  class BAR,RM done
  class LAD,GSP,ENG hw
  class GMMU,VRAM hw
```

ASCII fallback (reading top to bottom: guest → vgsp.ko → physical GPU):

```text
Guest VM (stock)
  CUDA app -> libcuda (push + QMD) -> nvidia.ko (RM, UVM)
        | MMIO write to BAR0 doorbell 0x88704        ^ MSI-X (events)
        v                                            |
vgsp.ko  ============================================ | =====================
  BAR0 shadow / doorbell  ->  Doorbell steering       |
  GSP-RM emulation  <-------  release/counter/event --+
        | channel, event, vaspace registration
        v
  +----------------- vGSP Execution Plane ----------------------+
  | pump -> ring/USERD discovery -> VA-to-PA truth -> decode     |
  |   -> release apply + counter publish + notifier              |
  |   -> ordered workqueue -> ladder submission (mode matrix)    |
  +-------------------------- | ---------------------------------+
                               v
Physical backend (module-owned GPU, real GSP session)
  ladder channel -> real GSP-RM -> GR/CE engines -> GMMU/VRAM
        ^ engine-authored release semaphore (polled by the submit)
```

### 2.2 Data flow of one CUDA kernel launch

The full chain, from CUDA context creation to the guest waking on the
completion. Stages 4–8 are shared by every channel type; stage 9 onward is the
GR-specific chain of §4. Status per stage is in the table after the diagram.

```mermaid
sequenceDiagram
  autonumber
  participant CU as Guest CUDA app
  participant LC as libcuda / nvidia.ko
  participant DEV as BAR0 shadow + doorbell
  participant PU as vEP pump (service kthread)
  participant TR as vEP truth (walk/pins/rescue)
  participant DC as vEP decode + capture
  participant WQ as Ordered workqueue
  participant LB as Ladder submission
  participant GP as Physical GSP + GR engine
  participant CM as Completion publication

  CU->>LC: cuCtxCreate, cuMemAlloc, cuLaunchKernel
  Note over LC: builds GR push: ctx-init methods,<br/>QMD install, SASS pointers, params
  LC->>DEV: GPFIFO entry appended; doorbell 0x88704
  DEV->>PU: doorbell token steering (main.c:865)
  PU->>TR: resolve ring VA and USERD (discovery/autopin)
  PU->>TR: read GP_PUT, fetch GPFIFO entry, decode method stream
  TR-->>PU: translated pages (or bounded rescue)
  PU->>DC: accumulate QMD window + launch candidate
  DC->>WQ: one GR request (single in-flight)
  WQ->>LB: guest GR submit (QMD relocate, PTEs, method block)
  LB->>GP: ladder GPFIFO publish + doorbell
  GP->>GP: front-end fetch, inline-QMD install [wall],<br/>ctx switch, SASS dispatch
  GP-->>LB: engine authors release semaphore in module/guest page
  LB-->>PU: completion observed (poll)
  PU->>CM: apply deferred release once; publish counter; notify
  CM->>DEV: channel event -> MSI-X (gsp_rm_post_channel_event)
  DEV->>LC: guest wakes; CUDA continues
```

ASCII fallback of the same chain:

```text
cuLaunchKernel
  -> push built (ctx init, QMD, SASS ptrs, params) + GPFIFO entry + doorbell
  -> pump reads GP_PUT / entry; auth: ring + USERD discovered, page translated
  -> decode stream; capture QMD window and launch candidate
  -> ordered workqueue: one in-flight GR request
  -> ladder submission: relocate QMD, plant PTEs (pre-built fetch lines),
     build launch methods, GPFIFO publish + doorbell on the real channel
  -> physical front-end fetch -> inline-QMD install [wall] -> GR engine
  -> engine authors release -> poll observes
  -> apply deferred release exactly once + counter publish + notifier
  -> channel event -> MSI-X -> guest tracker wakes
```

| Stage | What happens | State (2026-10-08) |
|---|---|---|
| 1 | Guest CUDA builds the launch (push methods, QMD, params, SASS refs) | `[live]` observed with the stock driver |
| 2 | GPFIFO entry + doorbell written into the BAR0 shadow | `[live]` |
| 3 | Pump steering, ring/USERD discovery, entry fetch, translation | `[live]` for CE; `[wip]` for GR (discovery now succeeds; see §3.2) |
| 4 | Push decode; QMD window accumulation; launch candidate capture | `[live]` observed (ring entry consumed; methods execute) |
| 5 | Physical submit on the ladder channel (mode-matrix shaped) | `[gated]` default-off; latched on `rmcall_gr_exec_ok` |
| 6 | Front-end fetch + method execution on the real channel | `[live]` observed |
| 7 | **Inline-QMD install / engine reads the descriptor** | **`[wall]` not observed** |
| 8 | Context switch + SASS dispatch to SMs | `[wall]` not observed for forwarded launches |
| 9 | Engine-authored release + channel release | `[wall]` absent for forwarded launches |
| 10 | Completion application, counter publish, notifier, event, MSI-X | `[wip]` CE path live default-off; for GR the counter-slot publication and the GP_GET write-back are the open question (ISSUE-0001, FORENSICS-grv5) |

The stage table's authority is
[`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)
(tracked) and the 2026-10-08 campaigns below it. Stage 1–4 status is
re-verified by the `grv3` campaign: ring entries *were* consumed for ranks
0–2, releases *were* applied, and the guest still did not proceed (local
evidence `backups/evidence-20261002-victory/analysis-2026-10-08-stall-v2/FORENSICS-grv3.md`
§0 and §2). The later `grv5` campaign (same evidence directory,
`FORENSICS-grv5.md` §1) un-gated the completion traces with `sema_dbg=Y` and
moved the second face from the wake-up to the counter publication (§6.1).

### 2.3 The mode matrix

The physical submit path is parameterized by a mode matrix: a set of
module parameters, almost all default-off, that select rebuild-free A/B
variants of the GR context commit, the launch method stream, the memory
windows and the completion relay. The matrix exists because the rules the front-end and engine
enforce are still being recovered; each knob isolates one variable so a
supervised cycle can falsify one hypothesis at a time. Every non-neutral knob
is documented in `kernel/gsp_phys_gpu.c` with its rationale and rollback
(the execution gates registered in `main.c`, e.g. `gsp_phys_gr_forward` and
the `sema_*` cadence knobs, follow the same rule); the parameter is the
experiment's record.

```mermaid
flowchart LR
  subgraph CTX["GR context commit"]
    A1["gr_promote 0..35<br/>promote shape bisection"]
    A2["srv_ctxbuf / srv_ctxcommit<br/>server-managed ctx"]
    A3["channel variants V5/V6<br/>golden replay"]
    A4["vas_shared / ctxshare_force<br/>VAS ownership"]
  end
  subgraph LAUNCH["GR launch shape"]
    B1["prelude / qmdvirt / pcas<br/>method-stream form"]
    B2["lmem_en - SKEDCHECK05 fix"]
    B3["lane_skip / lane_tail<br/>schedule-state controls"]
    B4["r1_first / recommit<br/>ordering"]
  end
  subgraph MEM["GR memory windows"]
    C1["qmdwin / shadow_win0<br/>win0 PT line"]
    C2["data_prewin / ptr_guest<br/>PTE kind"]
    C3["mail_patch / inst_sys<br/>instance aperture"]
    C4["gpc_mmu_init<br/>GPC walker"]
  end
  subgraph COMP["Completion + diagnostics"]
    D1["rel_mod - engine-completion discriminant"]
    D2["skedrec / img_probe<br/>notifier_sys / faultbuf_sys"]
  end
  subgraph WAVES["CE forwarding waves"]
    E1["work_forward -> work_defer_release"]
    E2["work_hw_primary -> relay"]
    E3["fault_relay / fault_inject"]
  end
  LAUNCH --> MEM --> CTX
  CTX --> COMP
  WAVES --> COMP
```

ASCII fallback:

```text
CTX COMMIT      gr_promote(0..35) srv_ctxbuf srv_ctxcommit srv_main_ignore
                channel(V5/V6) golden golden_exec_shape ctxshare_force vas_shared
LAUNCH SHAPE    prelude qmdvirt pcas pcas_two_step lmem_en lmem_base lmem_tpc_size
                lane_skip lane_tail r1_first recommit override_hex
MEM WINDOWS     qmdwin shadow_win0 data_prewin ptr_guest pte_kind mail_patch
                inst_sys gpc_mmu_init
COMPLETION      rel_mod skedrec img_probe notifier_sys faultbuf_sys
CE WAVES        work_forward -> work_defer_release -> work_hw_primary -> relay
                fault_relay fault_inject
GATES           gsp_phys_gpu mdev_owned gr_forward guest_mem static_live ctrl_forward
```

Not all of these are part of the vEP's steady state: `gsp_phys_gpu`,
`gsp_phys_hazard_ack` and `mdev_owned` gate the physical session itself
(docs 08/11); `static_live` and `ctrl_forward` are control-plane overlays.
Everything else in the matrix shapes execution. The single most important
gate is `rmcall_gr_exec_ok`: the latched proof, set only once a real GR
submission has executed on the ladder (`gsp_phys_booter.c:14614`), that
`gsp_phys_backend_gr_ready()` (`:14865`) requires before any guest GR launch
may forward. A parameter alone can never make the module pretend GR works.

---

## 3. Design principles and implementation reality

### 3.1 Principles the implementation enforces

1. **Emulate the interface, execute the work.** The vEP never modifies the
   guest driver; it satisfies the interfaces the guest observes and arranges
   real execution underneath (project principle, doc 01 §1).
2. **Guest truth over guesses.** Translations come from driver-published
   structures (VAS page directories, channel descriptors) or validated
   content, with a read-back proof before a pin is latched; rank-derived and
   convention-derived addresses are candidates, never facts. The pinned
   truth model is explicit in `gsp_sema_service.h` (`struct sema_channel_desc`,
   `struct sema_vaspace_desc`).
3. **One writer per structure.** Channel/vaspace state is copied out under
   `svc.chan_lock`/`svc.vas_lock`; injected pins never overwrite authoritative
   discovered data; guest-status-queue writes are single-writer in
   `gsp_rm_send` (doc 10 §5, doc 11 §1.3).
4. **Bounded budgets everywhere.** Bounded scans, retry counts, grace windows
   (`sema_ce_grace_ms=25000`, `sema_gr_grace_ms=8000`, `sema_push_hold_ms=2500`),
   a 5→60-pass discovery epoch with cooldown, bounded poll quanta (5 ms in the
   ladder, `gsp_phys_booter.c:1404`). A miss must degrade, never freeze; the
   starvation telemetry from ISSUE-0001 fix #2 exists to make a bound miss
   visible.
5. **Default-off, fail-closed, one-line rollback.** Every execution-changing
   behavior is behind a parameter, and every parameter has a fallback story
   (docs 11 §5). Forwarding refuses when the backend is not READY; a held
   release is applied exactly once by software when the hardware path misses
   (P6b wave 1.5 / P6c).
6. **Evidence gates the latch.** A capability is enabled only when a named
   acceptance signal has been observed (`REAL GR SEM OK` → `rmcall_gr_exec_ok`);
   "silence is not health" (doc 03-execution-frontier §8).

### 3.2 Ring discovery: kernel-native, both apertures, content-keyed

The pump cannot execute anything until it can name two addresses per channel:
the GPFIFO ring VA's translation, and the USERD's translation (PUT/GET live at
USERD+0x8c/0x88, `SVC_USERD_GP_PUT`/`SVC_USERD_GP_GET`,
`kernel/gsp_sema_service.c:143-144`).

Today discovery has three cooperating sources, in order of preference:

1. **Injected pins** (debug tooling and, historically, the userspace watcher
   pipeline) — authoritative but external; `svc_pin_lookup_any`.
2. **The kernel-native trusted walk** — driver-published page directories
   recorded from `SET_PAGE_DIRECTORY` (`svc_walk_vaspaces`), a self-discovered
   directory (`svc_pd_discover`, `svc_walk_discovered`), and per-VA targeted
   resolution (`svc_targeted_resolve`), all through the shared four-level
   `svc_gmmu_walk` with read-back validation (`svc_lookup_trusted`).
3. **Content-keyed discovery** — when no translation source can name a GR
   ring, `svc_gr_content_discover` scans for the *content* of the ring and the
   staged ctx-init push, pairs them per rank, and validates the pair in
   `svc_gr_content_pin` before pinning. The scan covers **both apertures**:
   the emulated vidmem aperture (`vgpu->bars[1].vaddr`) and guest system RAM
   through `svc_read()` = `vfio_dma_rw` (`gsp_sema_service.c:754`).

The history matters because it explains the current code shape:

* CE channels were discovered first and stably: the CE ring convention
  `FB 0x80000 + 0x20000*rank` pinned 6/6 CE rings (`ring_auto=6`) in the
  stalled v2/grv1 boots (FORENSICS-v2 §a; FORENSICS-grv1 §0.1).
* GR rings follow `VA 0x200200000 + 0x3000*rank` with the USERD at
  `+0x2000` (`SVC_GR_RING_VA_BASE/STRIDE`, `SVC_GR_USERD_OFF`,
  `gsp_sema_service.c:7334-7431`), but *where the ring physically lives* is
  boot-dependent: green m104 found all 20 GR rings at
  `FB 0x140000 + 0x3000*rank`; grv1's guest never wrote them to FB;
  grv3's guest wrote only ranks 0–2 (FORENSICS-grv3 §1).
* FORENSICS-grv1 proved the ring was **not in guest RAM** and recommended an
  FB-aperture scan; FORENSICS-grv3 identified the 60-pass epoch death and
  recommended periodic re-arm plus an FB-first fast path. Those three changes
  are in the tree as commits `ebcfc1c2` (both-aperture kernel-native
  walk + content fallback), `7abd8f87` (FB-aperture discovery + per-channel
  re-arm, FORENSICS-grv1 R1) and `66388560` (periodic re-arm + FB-first fast
  path, FORENSICS-grv3 R1).

In the current code (`svc_gr_content_discover`, `gsp_sema_service.c:7650`):
the FB scan runs first and fast (the aperture is host RAM, so a page-stride
pass is milliseconds), passes are paced at one per 2 s (`:7731`), push
candidates persist across passes so a late ring can pair without re-walking
6 GiB, a registration re-arms the epoch immediately, and hitting the 60-pass
cap starts a 30-second cooldown and then a fresh epoch while any registered
rank still lacks a ring — never the old permanent retirement (`:7712-7734`).

**Known limits.** The content path is gated to ranks 0–7
(`SVC_GR_CONTENT_RANKS=8`) and to the documented VA layout
(`gpfifo_va == BASE + STRIDE*rank`); a layout change deliberately falls back
to the safe paths. The walk depends on the guest publishing the owning VAS:
grv1/grv3 saw only two `SET_PAGE_DIRECTORY` publications, and the ring VA was
not covered by either (FORENSICS-grv1 §2). Content discovery is the fallback
for exactly that hole; a protocol-derived source for the ring VA remains the
durable direction (ISSUE-0001 "durable direction", `issue-number-1.md:107`).

### 3.3 The semaphore service: release application

`gsp_sema_service.c` is the vEP's engine room. Its service thread
(`svc_thread`, `:13343`) loops over the open mdev, runs the pump, retries
pending releases, and paces discovery. The release path:

1. **Decode** (`svc_decode_stream`, `:5170`) turns a consumed GPFIFO entry's
   method stream into operations; releases are recognized as
   increment-method writes to a semaphore VA with a payload.
2. **Translate** the target VA: `svc_lookup_trusted` first, then the pool
   resolver `svc_pool_pa_resolve` (`:2284`) — the resolver ISSUE-0001 fix #1
   completed by adding the driver-published VAS walk
   (`svc_walk_vaspaces`, `:1274`); a miss is queued in `svc_pending_add`
   (`:6166`) and retried every pass by `svc_apply_pending` (`:6201`), with the
   per-entry `tries` telemetry added by the same fix (`7f21daa7`).
3. **Apply** (`svc_sem_write`, `:2474`) reads the current word, writes only
   forward (never backwards), and is the sole software author of a release;
   a forwarded operation's releases can instead be deferred and then applied
   exactly once by the hold's finish path (P6b wave 1.5), or authored by the
   engine itself under P6c's relay (default off, docs 11 §P6c).

The complementary **counter** path handles GR completion *counters* (distinct
from CE tracking semaphores): `svc_counter_derive` (`:4658`) derives the
counter slot from structural translations; `svc_counter_autoreg` (`:4710`)
registers it per GR channel by the fixed-total formula rank r →
`0x20400fec0 + (SVC_GR_COUNTER_TOTAL-1-r)*0x10` (`:4721`);
`svc_counter_release_apply` (`:4433`) applies decoded releases
to the slot; `svc_counter_flush_pending` (`:4575`) lands payloads recorded
before the slot was known; `svc_counter_publish` (`:9077`) writes the GPFIFO
PUT into the slot when it is ahead of the current value. Three facts learned
the hard way are now encoded: the counter pool total is fixed at 20
(`SVC_GR_COUNTER_TOTAL`, commit `fcb96912`) because a partial live channel
count poisons every later rank's slot; the early publish is guarded so a
guessed PUT is never written (the H3 finding, `gsp_sema_service.c:11489-11512`,
FORENSICS-v2 §b/H3); and the fixed-total mapping points each rank's slot at
the same word the ctx-init push releases to, so the publish becomes a
structural no-op (`put == counter_last`, `val <= cur`; `:9096-9109`) — the
grv5 stall ran with `counter publish` ×0 while `pub_calls=180142` and the
release, notifier clear and event post all fired (FORENSICS-grv5 §1,
verdict). The E1/E2 debug mirrors and GP_GET read-back for that finding
landed debug-gated as commit `c39b2a1c`.

### 3.4 Notifiers, GPFIFO consume, event delivery

* **GPFIFO consume.** The pump reads `GP_PUT` from the USERD, walks entries
  from its own GET, decodes, and advances (the GP_GET write-back is
  `gsp_sema_service.c:12682-12684`, the end-of-pass publish `:12731-12732`);
  a replay-suppression path keeps
  pin batches from re-consuming old slots (`gpfifo_skipped`), and a per-entry
  grace window holds an untranslatable GR entry before it is consumed as
  stale (`sema_gr_grace_ms`, default 8 s). `svc_userd_probe` (`:7099`)
  distinguishes "unreadable" from "readable but idle (0/0)" so a zero page is
  never mistaken for a live channel.
* **Notifiers.** `svc_notify_complete` (`:7207`; the clear and event post are
  `:7240-7277`) services the channel's error
  notifier token and posts the channel event. The notifier page address and
  aperture come from the channel allocation descriptor
  (`sema_channel_desc.notifier_pa/as`, `gsp_sema_service.h:31-32`), recorded
  at channel registration.
* **Events.** `gsp_rm_post_channel_event` (`gsp_rm_emulation.c:602`) finds the
  guest's registered event objects (`rm->events[]`, registered from the event
  alloc in `gsp_rm_handle_rm_alloc`, `:2443-2489`) and sends the RPC event;
  the reply path injects MSI-X through the live eventfd
  (`gsp_emu_inject_completion_interrupt`). A green boot shows the sequence
  `notifier cleared … status=0xffff` (per GR channel) followed by the event
  post. grv3 (traces off) showed neither and re-queued the identical push;
  the grv5 repeat with `sema_dbg=Y` settled the question — the notifier
  cleared ×34 and the event posted, while the counter publish never wrote.
  The leading reading is therefore the counter publication, not a missing
  wake-up (FORENSICS-grv5 §1; §6.1).

### 3.5 Physical submit

Forwarding is a two-stage ladder. The sema service captures a candidate and
queues one request on the single ordered workqueue `gsp-phys-work`
(`svc_gr_request`, `:10998`; `svc_fwd_hold` is the single in-flight slot).
The work item calls into the backend: `gsp_phys_backend_guest_ce_copy` for CE
(`gsp_phys_booter.c:15621`) or `gsp_phys_backend_guest_gr_submit` (`:16457`)
for GR, gated by `gsp_phys_backend_work_ready()`/`gsp_phys_backend_gr_ready()`
(`:14849`/`:14865`), which require READY, a live claim, an unload not
committed, a client-memory ladder channel, a work token, and every ladder VA;
the GR submit additionally re-checks `gsp_phys_guest_mem` and the active
guest RAM backing at `:16509-16510`.

GR submission (`gsp_phys_backend_guest_gr_submit`) is the vEP's most
parameterized function: it relocates and pre-writes the QMD (inline-QMD
window, `gsp_phys_rmcall_gr_qmdwin`), copies program/param pages into shadow
windows whose PTEs are planted at stage 3 before the engine's first fetch
(`gsp_phys_rmcall_gr_shadow_win0`, `data_prewin`), builds the launch method
block (`gsp_phys_rmcall_build_gr_launch`, `:15862`) from the recovered method
vocabulary (compute init, inline-QMD window, PCAS pairs, report semaphore),
publishes it to the ladder channel's GPFIFO and rings the doorbell
(`gsp_phys_rmcall_publish_locked`, `:14345`), then polls for the release with
`rel_mod`/`skedrec` diagnostics. The PTE fetch-line rule is enforced: the
engine caches PTE fetches at 128-byte granularity with no invalidate path, so
each scratch pair consumes a never-reused fetch line from pools built before
first engine access (docs 11 §P6; `gsp_phys_booter.c:1044-1070`).

Measured cost: the m141-r3 baseline (`895ab8b8`) is `elapsed_ms=189` for the
first forwarded launch and `166` for the second, of which only ~7.9 ms is the
post-doorbell engine path; ~180 ms is serialized per-window `PDE_UPDATE`
forwards (local evidence and the prepared harness:
`tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1.3). The baseline is a
forwarding-service cost, not engine time — the number to drive down is the
7.9 ms and the per-window round trips.

### 3.6 Citation map for this section

| Claim | Authority |
|---|---|
| Both-aperture kernel-native GR discovery + content fallback | commit `ebcfc1c2`; `gsp_sema_service.c:7334-7529` |
| FB-aperture discovery + re-arm; periodic re-arm + FB-first | commits `7abd8f87`, `66388560`; `gsp_sema_service.c:7650-7800` |
| Ring not in guest RAM; RAM-only scan limitation | `FORENSICS-grv1.md` §0/§1/§A (local evidence) |
| 60-pass cap dead-end; completion-chain questions | `FORENSICS-grv3.md` §0/§3/§4/§5; wake-up refuted and counter publication leading in `FORENSICS-grv5.md` §1/§4 (local evidence) |
| CE clean consumption at 89 releases; no starvation | `FORENSICS-v2.md` §b/§c (local evidence) |
| Pool resolver + starvation telemetry | commit `7f21daa7`; ISSUE-0001 fix #1/#2 |
| Counter slot fixed-total 20 | commit `fcb96912`; the fixed-total publish no-op is `FORENSICS-grv5.md` §1/§4 (local evidence) |
| Release written + notifier cleared + event posted while `counter publish` ×0 | `FORENSICS-grv5.md` §1/§4 (local evidence); commit `c39b2a1c` (E1/E2 debug mirrors) |
| PROMOTE_CTX `bInitialize` must stay set | commit `88d47c1f`; ISSUE-0001 fix #3 |
| GR exec latch + readiness gate | `gsp_phys_booter.c:14614`, `:14865`; `main.c:2609` |
| PTE fetch-line rule, scratch pools | docs 11 §P6; `gsp_phys_booter.c:1044-1070` |
| m141-r3 baseline 189/166 ms decomposition | `tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1 |

---

## 4. End-to-end GR execution, in our words

This section defines the vocabulary as the code uses it, then walks the GR
chain. The concepts are standard NVIDIA terminology; the definitions below
are **ours**, written against the interfaces we implement.

| Term | Our definition |
|---|---|
| **GR engine** | The physical engine that executes shader work: both compute grids and graphics pipelines. CUDA kernels are GR work. It is the one engine that can run arbitrary guest-authored code, which is why it is the frontier. |
| **Channel** | A hardware execution context: a GPFIFO ring, a push-buffer pointer, a USERD control block, an instance/RAMFC block, and the engine object bound to it. The guest creates channels through RM allocs; the vEP registers them (`gsp_rm_register_channel` → `gsp_sema_service_channel_add`). |
| **GPFIFO** | The ring of entries a channel consumes. Each entry points at a push buffer (or carries inline data); the front-end fetches it, then executes the method stream it references. |
| **USERD** | The channel's control block in GPU memory. `GP_PUT` (offset 0x8c) is where the guest declares submissions; `GP_GET` (0x88) is the channel's consumption cursor. The pump reads these through `svc_read_ap` with the channel's aperture. |
| **Push buffer / method stream** | The command stream: incrementing-method headers plus payload, addressing engine registers and subchannels. Decoding it is how the vEP learns what the guest asked for (`svc_decode_stream`). |
| **QMD** | The launch descriptor: the fixed-layout structure GR reads to dispatch a grid (program address, grid/block dimensions, parameter pointers, constant-bank pointers, release targets). CUDA builds it; the forwarded path must install the inline copy into the channel's descriptor buffer in the form the engine reads (§6, the wall). |
| **Context / ctx-init** | The per-channel engine state block (main + per-VEID subcontext images) that must be initialized and committed before GR can switch to the channel. The guest drives this with `GPU_PROMOTE_CTX`; the server-side rules are what the `gr_promote`/`srv_ctxcommit` matrix is bisecting. |
| **Golden context** | The reference channel/3D object set the stock driver creates right after load so the server-side GR instance state exists (the `gsp_phys_rmcall_golden` replay). A context committed without that state does not dispatch. |
| **SASS** | The compiled shader machine code the GR executes. Its pointers reach GR through QMD constant banks; for forwarded launches the vEP relocates referenced pages into engine-walkable windows rather than trusting guest pointers directly. |
| **Semaphore release** | The completion primitive: a method that writes a payload word into a semaphore VA. CE work releases tracking semaphores; GR work releases the channel semaphore and the QMD release word. The vEP applies releases in software or, gated, lets the engine author them. |
| **Counter** | The GR completion counter slot: the u32 the guest's userspace tracker polls for channel progress; the vEP derives, registers and publishes it (`svc_counter_*`). |
| **Notifier** | The channel error/completion notification block. `svc_notify_complete` services it and the RM event path delivers the channel event. |
| **Shadow window / scratch PT** | Module-owned pages mapped into the engine's page-table tree at VAs the ladder controls, used to place QMD/program/parameter/data content where GR can walk it. Window-0 PTEs are pre-planted before the channel commit because late writes into already-fetched PTE lines are invisible. |

**The chain the vEP implements for GR:**

1. **Context creation.** The guest RM allocates VRAM objects, a VAS, a TSG,
   GR channels; the vEP registers each channel at alloc time and records its
   descriptor. The guest's `SET_PAGE_DIRECTORY` publications become the
   trusted translation sources. The guest then promotes its GR context
   (`PROMOTE_CTX`); the emulation acknowledges and keeps `bInitialize` set
   (fix `88d47c1f`) so the client marks its buffers initialized.
2. **Golden context / server state.** On the physical ladder, the
   `golden` replay allocates the stock client's post-load channel set so the
   server-side GR instance state exists; the context commit variants
   (`gr_promote`, `srv_ctxcommit`, `vas_shared`) are the ladder's bisection of
   what the real server requires. The vEP's guest side does not depend on the
   ladder shape; the gate is the latch.
3. **QMD capture.** The pump decodes the guest's launch stream, accumulates
   the inline QMD window (`svc_qmd_accum`), and captures the launch candidate
   (`svc_gr_cand_capture`), including the release target and the referenced
   program/param pages.
4. **Shadow windows.** The submit relocates the QMD and copies referenced
   pages into window-0 shadow pages whose PTEs were planted pre-commit;
   data pages go through the pre-installed data pool; `mail_patch` injects a
   module message page when the guest's constant bank is empty. `ptr_guest`
   is the A/B for keeping the guest's original pointers (the SM walk appears
   to resolve through the guest subcontext PDB; m91).
5. **Physical submit.** The submit builds the launch method block, publishes
   it to the ladder channel's GPFIFO, rings the doorbell, and polls for the
   engine-authored release.
6. **SASS on GR.** Between publish and release, the front-end executes the
   methods; the inline-QMD install and the engine's read of the descriptor
   are the stages the current campaigns have not yet observed. Once they
   fire, GR switches to the context and dispatches the grid to the SMs.
7. **Completion.** The engine authors the QMD release word and the channel
   release; the poll observes them; the deferred release is applied once
   (P6b wave 1.5 semantics); the counter is published; the notifier is
   cleared; the channel event is posted; MSI-X wakes the guest. For GR the
   last step is the open counter-publication question (§6.1).

---

## 5. Enabling arbitrary CUDA code execution

"Arbitrary CUDA" is the 1.0 acceptance bar: stock guest CUDA works for
real workloads, not one demo, and every time. The program is staged:

* **P1 — single-workload A/B.** `tools/cuda-ab/ab.sh` runs one deterministic
  workload on the host (bare-metal oracle) and in the guest, then compares
  the stable result lines. Workloads: `victory_mail` (the campaign mail
  kernel), `vector_add`, `smiley` (an ASCII-art deliverable). This is the
  rung that produced the m141-r3 first green forwarded launches in 189/166 ms
  (`895ab8b8`).
* **P2 — the "arbitrary CUDA kernels" acceptance suite.**
  `tools/cuda-ab/p2/` is eleven self-checking programs covering launch shape
  (1D/2D/3D grids, odd sizes), parameter counts 0–12 with mixed types,
  register/launch-bounds variants, shared memory (reduction + tiled GEMM),
  streams including eight independent kernels in flight, allocator churn,
  device printf/assert, error reporting and recovery, a correctly-rounded
  compute benchmark, and a combined end-to-end program. References are
  deterministic (EMIT mode) until the host GPU is free for capture; the
  suite currently runs `[planned]` pending the reproducibility gate
  (`tools/cuda-ab/READINESS-P2-2026-10-08.md`).
* **P3 — near-native performance.** `tools/cuda-ab/perf/` is the prepared
  measurement program: host-oracle vs guest ratios for init, malloc, launch
  (first/warm/throughput), HtoD/DtoH bandwidth, compute GFLOP/s, plus host
  metrics (the backend `elapsed_ms`, injected-latency proxies). Status:
  prepared, not yet run; every threshold provisional until a green
  calibration (`tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1/§4).

**Proven vs pending.** Proven today: single forwarded launches of two small
kernels (mail, vadd) with correct release and clean teardown, on a lucky
green boot; CE-class copies forwarded end-to-end with engine-authored release
and fault relay (default off); compiled kernels of the P2 complexity are not
yet observed executing through the stack. Pending: reproducibility
(ISSUE-0001), the GR descriptor-install wall (§6.1), the GR counter-slot
publication / GP_GET write-back (§6.1; FORENSICS-grv5), multiple
kernels/clients per VM (ISSUE-0002), and P2/P3 in a green
boot. The gating order is deliberate: reproducibility first, then the wall,
then breadth, then performance.

---

## 6. Challenges

### 6.1 Current

* **The GR completion wall — two faces.** The older face is the
  launch-descriptor install: the forwarded launch reaches and runs
  front-end methods, the descriptor destination window is accepted, the
  context is valid at dispatch, and yet the inline-QMD install is never
  observed and no grid dispatches
  ([`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)
  §1–§5). The exonerated list is long (context validity, destination mapping,
  entry fetch/alignment, TSG parenting); the live issues are the stream's
  verbatim prelude, the absent descriptor-virtualization control method, and
  the relocation transform, each with a one-cycle falsifier. The current code
  carries many of those falsifiers as mode-matrix knobs (m26 falsified the
  nonzero `qmdvirt` hypothesis; m72 found the SKED `LOCAL_MEMORY_TOTAL_SIZE`
  cancel and added the `lmem_en` block; m91 showed the SM I-fetch MMU_NACK at
  shadow VAs and added `ptr_guest`).
* **The second face — the completion-counter publication.** grv3 showed a
  boot where discovery succeeded (three GR rings pinned), the ctx-init
  releases landed (`gr_rel=3`, counter words = 1), and the guest *still*
  stopped, re-queueing the identical push (FORENSICS-grv3 §0/§3). The grv5
  repeat ran with `sema_dbg=Y` and settled it: the release was written to the
  exact pin-backed slot, `notifier cleared` fired ×34 and the channel event
  posted (`posted=4`) — and `counter publish` still never wrote while
  `pub_calls=180142`, because the fixed-total autoreg mapping makes the
  publish a structural no-op (§3.3; FORENSICS-grv5 §1, verdict). The leading
  reading is the counter-slot publication, with the GP_GET write-back as the
  secondary, still-open candidate (the host writes `last_get` at
  `gsp_sema_service.c:12682-12684` but nothing has read it back;
  FORENSICS-grv5 §4/H2). Both observability surfaces are debug-gated today
  (commit `c39b2a1c`).
* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The
  first fix wave landed (kernel-native pool translation `7f21daa7`,
  PROMOTE_CTX reply `88d47c1f`, issue register `a1407659`), and the
  earlier stall mode — the client freezing at DEVICE because its last CE
  completion could not be written — is fixed: the v2 campaign consumed all 89
  CE pushes with zero holds and zero starvation. The stall then moved one
  phase later, to GR ring discovery, where the tree now carries the
  kernel-native discovery lineage (`1ba8ba65`, `ebcfc1c2`) plus the fixed
  re-arm of `7abd8f87`/`66388560`. Acceptance is 10/10 consecutive cold boots
  green (`issue-number-1.md` validation plan); that count is pending.
* **Multiple kernels/clients per VM (ISSUE-0002).** A second CUDA process in
  one boot wedges in guest UVM because the host's per-client sema/GR state is
  not reset when the guest re-runs the GSP boot handshake per client; the
  guest's UVM holds a global lock across the registration RM exchange, so a
  host-side quiet path freezes every later client (`issue-number-2.md`). The
  durable direction is per-client/per-channel state reset plus per-mdev sema
  contexts and P0 isolation work, with the ordered hunks already recorded
  (validation plan in the issue).

### 6.2 Historical (fixed or retired — do not relitigate)

* **CE starvation at the boot-critical release.** The old stall was one
  deferred CE release whose semaphore page had no translation, waiting on a
  racy userspace watcher. Fixed kernel-natively by the pool resolver's VAS
  walk (`7f21daa7`) and the GR ring/USERD discovery lineage; `pushes_nomap`
  now reaches 0 on consumed boots (FORENSICS-v2 §0/§c).
* **The ring-location class.** Early discovery tried only FB conventions
  (`0x140000 + 0x3000*rank`) and RAM scans; grv1 proved the ring was not in
  guest RAM and the RAM-only content scan burned 60 passes. The fix is the
  both-aperture, FB-first, re-armable discovery now in the tree
  (`7abd8f87`, `66388560`).
* **The notifier gate as a discovery symptom.** The `notifier cleared` line
  historically correlated with ring resolution; it is a publication-side
  signal and is now understood as part of the completion-publication face of
  §6.1 (grv5: the notifier/event wake-up fires; the counter write is what is
  missing), not a discovery fix in itself.

### 6.3 What is working well, and what is not

**Working well:** the CE path end-to-end (copies, deferred exactly-once
release, hardware-primary data, engine-authored relay, fault relay) — all
live-verified default-off; kernel-native discovery and the trusted-walk
truth model; deterministic counter semantics with fixed pool total; the
mode-matrix discipline (one variable per supervised cycle, one-line
rollback); the bounded-budget culture that turns potential freezes into
logged, counted degradations.

**Not working:** GR dispatch (no grid observed); the GR counter-slot
publication (structurally dead under the fixed-total mapping; FORENSICS-grv5
§1); reproducibility at the required 10/10 bar; second-client state
isolation; and the observability gaps that made both bugs slow to find —
counter `pub_calls` counting calls rather than writes (H3; the increment
still sits before the no-op guards at `gsp_sema_service.c:9085` and correction
is planned, FORENSICS-grv1 §5 R3), completion traces still behind `sema_dbg`
(grv5 captured them, but the telemetry must become unconditional), and the
absence of a GR error-reporting object on the forwarded path ("silence is not
health", doc 03-execution-frontier §8).

---

## 7. Goals, gaps, and the iteration plan

### 7.1 Goals

1. **Correctness:** a stock guest runs arbitrary CUDA in one boot, repeatedly.
2. **Reproducibility:** 10/10 cold boots green (ISSUE-0001 exit criterion).
3. **Breadth:** P2 suite green in a single boot; multi-client and multi-stream
   (ISSUE-0002); then multi-mdev.
4. **Performance:** near-native P3 numbers with no-regression discipline.
5. **Portability:** the vEP survives driver-version and GPU-generation churn by
   re-deriving facts, not by patching snapshots.

### 7.2 The iteration loop

Every change follows the same evidence loop. This is the operating system of
the vEP program, not an appendix:

1. **Question as a discriminating sentence + named acceptance signal.** E.g.
   "does the counter slot get published for a GR ctx-init channel?" with the
   signal `counter publish … written` + guest advancement past the re-queue
   (the earlier notifier/event question was answered by grv5: they fire;
   FORENSICS-grv5 §1).
2. **Offline first.** Use the risc-v-gsp-rm-debugger (`debug/risc-v-gsp-rm-vm/`)
   and its validation tools — `harness/valprobe.py` for one firmware
   validation branch, `harness/rehost.py` for one whole function, RPC
   capture/decode/compare (`debug/risc-v-gsp-rm-vm/docs/45-47`) — to test a
   firmware-side rule without a supervised physical cycle. The entry points
   are [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md)
   and [`../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md`](../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md).
3. **One supervised cycle, one variable.** A mode-matrix knob, a trace flag,
   or a single code change; never two.
4. **Evidence pair.** Session/counter pair (green vs this run) with the
   discriminators named before the run; campaign evidence goes to
   `backups/evidence-…/` (local), the public claim cites the behavior only.
5. **Land or revert.** A landed change updates the doc and (if needed) the
   issue; a falsified hypothesis is recorded in the knob comment so no future
   session re-runs it.

### 7.3 Tooling gaps and the plan for them

| Gap | Tool / practice | Plan |
|---|---|---|
| GR rule recovery without burning cycles | risc-v-gsp-rm-debugger valprobe/rehost, RPC capture/replay (docs 45–47) | probe candidate front-end/engine rules offline; queue only falsifiable physical experiments |
| Arbitrary-kernel acceptance | `tools/cuda-ab/p2/` (11 programs) | run in the first green boot after ISSUE-0001 acceptance; add a multi-process/multi-stream variant for ISSUE-0002 |
| Performance measurement | `tools/cuda-ab/perf/` + `collect.py` | freeze thresholds on first green run; make the serialized `PDE_UPDATE` forwards the first optimization candidate |
| Completion observability | unconditional or counted `notified`/`posted`/`publish-written` telemetry in `svc_notify_complete`, `gsp_rm_post_channel_event` and `svc_counter_publish` (FORENSICS-grv3 §5B, FORENSICS-grv5 §5) | land as telemetry, not a behavior change; treat `posted==0` after a real clear, or `publish written==0` with a registered slot, as a first-class failure; the E1/E2 debug mirrors (`c39b2a1c`) are the interim form |
| Counter telemetry honesty | move the `pub_calls` increment after the write (FORENSICS-grv1 §5 R3) | land with the telemetry wave |
| Multi-client state | ISSUE-0002 fix plan (per-client reset, per-mdev sema context, P0 hunks) | stage A1–A6 per the recorded plan; graduation criteria sequential/concurrent/simultaneous |
| Evidence discipline | one evidence bundle per claim; counters as first-class discriminators; "no claim without a pair" | already the campaign norm (FORENSICS reports); keep counters stable enough to be quotable (see counter telemetry fix) |
| Sub-agent workflows | read-only analysis sessions (`SA-ANALYSIS`, `SA-CHK`, `SA-DOC`) producing FORENSICS/readiness docs while the lead runs cycles | continue; keep analysis strictly read-only and cite line numbers; never touch the rig in analysis sessions |

---

## 8. Portability

### 8.1 What is version-portable

* The **architecture** of the vEP: observe pushes → resolve truth → decode →
  execute/forward → publish completion. This shape does not depend on a
  driver version or a GPU generation; only the facts it consumes do.
* The **RPC/RM object model** the vEP registers against (channel alloc
  descriptors, page-directory publications, event objects, `PROMOTE_CTX`)
  is stable in structure across the 5xx/6xx families; new fields appear, but
  the wire structures are versioned and compiled into the driver.
* The **mode-matrix method**: parameterized, default-off, rollbackable
  experiments and an evidence latch (`rmcall_gr_exec_ok`) port as a practice.
* The **counter/notifier/event completion model** (a polled u32, a notifier
  block, an RM event, MSI-X) is the same completion vocabulary across
  versions.
* The **QMD vocabulary at a coarse level** (a launch descriptor with program,
  grid, parameters, release targets) is stable; exact field offsets are not.

### 8.2 What must be re-derived per version or architecture

| Item | Why | Re-derivation source |
|---|---|---|
| Firmware/driver identity, RPC inventory, control IDs | versions add/retire controls | measurement: `firmware_analyzer` signatures, live RPC captures (`docs/45`), host-oracle A/B (docs 07 P1) |
| Ring/USERD VA and FB layout conventions | the guest's memory allocator changes with the driver | first-touch probes + content discovery; never hardcode beyond the documented candidate list |
| QMD layout and launch method stream | the launch vocabulary changes (Turing → Ampere → Hopper → Blackwell) | reference capture of a working launch; differential against the forwarded stream |
| Context image layout, promote entry sets, golden-context shape | server-side RM rules and image formats change | ladder bisection + offline valprobe/rehost |
| PTE encodings, page-table geometry, fetch-line rule | GMMU changes per architecture | hardware observation; document the encoding proven in each profile |
| Engine inventory and classes | missing/zeroed records roll whole contexts back (doc 05 §6) | bare-metal oracle capture; profile tables |
| Scratch-window VA layout and pools | ours, but coupled to the ladder channel's VAS geometry | re-lay per profile; keep the pre-commit planting rule |

Portability discipline: **protocol-level facts port; address-level facts do
not.** Any new version or architecture enters as a profile plus a measured
delta, never as a guess (docs 06 §5, `RUNBOOK.md` S-scenarios).

### 8.3 Concrete paths

* **Driver 610.** Hard gate first: RM 610 requires explicit named-human
  approval — no acquisition, analysis, emulation edit, physical work or
  support claim before it (`docs/23-RM610-READINESS.md`). Once approved:
  acquire the image, run one-command triage
  (`risc-v-gsp-rm-debugger.sh triage`, `docs/40`), diff the RPC inventory
  with `harness/version_diff.py`/`ci_release.py` (`docs/21`), then port the
  vEP facts in the order of §8.2. This is speculative until the gate opens;
  nothing in the current tree pre-adapts to 610.
* **Grace Blackwell (GB10/GB2xx, aarch64 hosts).** A different problem class:
  different host CPU architecture (the module's guest-RAM translation,
  `read_cache_page`-based HPA mapping, must remain architecture-neutral), an
  engine inventory and GMMU generation with no current profile, and a
  firmware family the debugger would need to re-host/port (docs 07 P3). The
  vEP plan is: profile capture (PCI IDs, BAR geometry, engine list), GMMU
  encoding work, then the same discovery/decode/submit ladder on a supervised
  device, with the support ladder (docs 06 §1) moving only on evidence.
* **Compute profiles.** The near-term "profile" is a small, per-version file
  of measured facts (ring conventions, VA windows, QMD offsets, method
  encodings, PTE kind) consumed by the vEP at runtime; today those facts are
  in code comments and constants with commit provenance, which is the
  pre-profile state.

---

## 9. Mapping vEP concepts to CUDA and to architecture

### 9.1 CUDA high-level concepts

| CUDA concept | What the guest does | vEP counterpart |
|---|---|---|
| Context | `cuCtxCreate`; RM client/device/subdevice + a GR engine context | guest context is emulated (objects, VAS, promote); the physical ladder commits its own context (`gr_promote`/`srv_ctxcommit`) |
| Module / function | loads SASS/cubin into GPU memory; `cuModuleGetFunction` | SASS reaches GR through QMD program pointers; the vEP relocates referenced pages into engine-walkable windows (`shadow_win0`, `data_prewin`, `ptr_guest`) |
| Stream | ordered work queue on a channel (or a graph) | channel + TSG; the guest's push order on the channel is the stream order the pump preserves |
| Kernel launch | builds push (ctx init, QMD install, params), appends GPFIFO entry, rings doorbell | capture → one in-flight request → ladder submission; the QMD install is the wall (§6.1) |
| Memory | `cuMemAlloc`/copies; VA objects mapped into the VAS | emulated VRAM + guest RAM with the trusted truth model; forwarded operands resolve VA→PA→HPA via the guest RAM backing |
| Synchronisation | events, stream sync, semaphores, `cudaDeviceSynchronize` | releases, counters, notifiers, channel events; the vEP is the sole publication authority (or relays engine-authored releases) |

### 9.2 Architecture details

| Architecture concept | vEP relationship |
|---|---|
| Channel / GPFIFO / USERD | the unit of scheduling and consumption (§4); registration from the alloc descriptor |
| TSG / runlist | the guest's scheduling vocabulary; the ladder's channel sits in a TSG on the real server; guest-side pacing is the pump today, tenant fairness is planned on top (doc 09 §5) |
| Engine types / classes | GR is engine type 1, COPY0 is 0x9 (doc 05 §6); the ladder selects GR0 for GR work and refuses CE paths on a GR ladder (`gsp_phys_backend_ladder_ready`) |
| GMMU | the translation ground truth; the trusted walk reads guest page tables, and the engine's view is fed by planted PTEs in pre-built fetch lines |
| VRAM / FB apertures | emulated as a host buffer today; apertures appear in channel descriptors and pins (`SVC_APERTURE_SYS/FB`); real VRAM backing is future work |
| MSI-X / interrupts | completion delivery: `gsp_rm_post_channel_event` → eventfd → irqfd; physical-engine faults are observed and relayed (C3, default off) |

---

## 10. Future challenges the current tools do not address

These are the known gaps for which no tool or design exists yet. Each gets a
proactive design direction so the architecture is not surprised later.

1. **Multi-tenant fairness and performance isolation.** Today one mdev, one
   real session, one in-flight request. The planned per-mdev real clients
   need a policy layer (time slices, quotas, accounting) and a measurement
   method that can fail (doc 09 §5; docs 06 §2.8). Proactive design: keep
   every per-client datum inside a container keyed by the mdev/session, and
   build the reset path (ISSUE-0002) as the container's lifecycle, not as
   scattered clears.
2. **Fault recovery semantics.** Fault relay exists (C3); fault *replay*,
   *cancel*, and channel recovery after a fault do not. Proactive design:
   model faults as first-class completion outcomes in the request record, so
   a fault can trigger the same exactly-once release discipline as success.
3. **Context switching and preemption between tenants.** The ladder leaves a
   context loaded; the client preempts its TSG for a clean first dispatch
   (`lane_tail`). A multi-tenant scheduler must own load/unload lifecycles.
   Proactive design: the mode matrix's lane controls are the seed of a
   documented context-switch protocol, not a launcher-only trick.
4. **Hostile-guest safety.** A malformed push, a forged VA, or a wedged ring
   must not corrupt the host or another tenant. Guards exist (bounds checks,
   read-back proofs, `gsp_phys_seq_addr_valid`), but there is no fuzzing
   harness for guest-authored streams. Proactive design: a synthetic-push
   generator against the decoder with the same evidence discipline as the
   physical ladders.
5. **Observability as a product property.** The vEP's bugs were found by
   counter archaeology; counters that count the wrong thing (H3) cost days.
   Proactive design: every counter must answer one named question, and
   telemetry for the completion path (notifier, event, publish) lands before
   it is needed again.
6. **Version drift under an adversarial schedule.** New images will arrive
   without notice; the triage/porting tooling exists but is unexercised on a
   real new version. Proactive design: golden sessions per supported version
   and a one-command triage that can fail loudly.
7. **Windows guests and non-KVM transports.** They change the RAM/IRQ
   plumbing under the vEP, not its pipeline. Proactive design: keep the
   guest-memory accessor and the event injection behind interfaces (they
   already are: guest-RAM backing file, eventfd), so the pipeline is
   transport-agnostic.

---

## 11. 3D graphics interaction

Graphics workloads (OpenGL/Vulkan) are GR work too, but they differ from
compute in ways the vEP will feel directly:

* **Engines:** more than GR — a 3D context still uses GR (graphics
  pipelines), but video (NVDEC/NVENC) and image (NVJPG/OFA) engines join
  later; CE remains for buffer moves; the engine inventory is a correctness
  surface (doc 05 §6).
* **Channels and sync:** drivers use multiple channels per context, TSGs for
  scheduling groups, and inter-engine semaphores/fences (a copy waits on a
  render; a present waits on a copy). The vEP's release/counter vocabulary
  covers the primitives; the *volume and cross-engine ordering* of
  synchronisation is higher.
* **Pushes:** the graphics method vocabulary is much larger (pipeline state,
  shader binding, render targets, viewports), and state lives in big
  per-context image blocks. The current capture/decode path is compute-first;
  a graphics capture would need the same treatment the CUDA launch got
  (reference method stream, differential against the forwarded stream).
* **Memory:** surfaces, tiled/swizzled layouts, render targets in VRAM; the
  GMMU and aperture handling are the same, but the emulated host-RAM VRAM
  becomes a real fidelity problem for tiling/swizzle expectations.
* **Interaction with the current wall:** the descriptor-install wall is
  shared — a graphics draw also dispatches through GR descriptors. Ceiling
  note: the current docs mark OpenGL/Vulkan `not yet` (docs README status
  table); the 3D plan is to reuse the vEP pipeline once compute dispatch is
  proven, then add a graphics-capture workstream (mode-matrix discipline
  applies: one class/method family at a time).

---

## 12. Overarching framing: why the vEP is a core problem

**What vGSP is.** vGSP is the virtual GPU System Processor: the host-side
reimplementation of the GSP/GSP-RM interface that lets a stock guest driver
believe it owns a real NVIDIA GPU. `vgsp.ko` is vGSP: the virtual device, the
emulated boot and message transport, the RM object/control service, and the
execution plane. The guest never learns it is virtualized for these paths,
and no component in the stack — host driver, guest driver, firmware, or
hypervisor — is modified (docs 01, 09 §13).

**Why the execution plane is one of vGSP's core problems.** Everything else
in vGSP is necessary but not sufficient: the control plane makes the driver
run, but only the execution plane makes applications *work*. It is where the
project's hardest constraints collide:

* **Correctness without cooperation.** The guest does not help; the vEP must
  recover VA/PA truth, ring locations and completion semantics from
  observation, and satisfy contracts (completion, ordering) that hardware
  would otherwise enforce.
* **Performance without emulation.** Compute cannot be emulated at useful
  speed; the plan of record is real execution on real engines, which means
  every semantic the emulation would have provided for free (memory
  visibility, ordering, faults) must be provided by the forwarding path.
* **Isolation across tenants.** The same plane is where the multi-tenant
  promise is kept or broken: one physical GPU's engines shared by mdevs
  requires per-client state, scheduling and fault containment — the
  ISSUE-0002 class today, the fairness policy tomorrow.

**Fit with HAAEEP and the ecosystem.** HAAEEP's premise is compute at the
edge — local inference, full isolation, no cloud dependency, no license
(`docs/08-ecosystem/haaeep-and-collaboration.md` §1/§3). That premise is only
real if a guest can run its
own CUDA workloads on hardware it owns, under vGSP, with isolation that a
security stack can trust. The vEP is the subsystem that turns that premise
from an interface emulation into execution; its progress metrics
(reproducibility, arbitrary-kernel acceptance, near-native performance,
per-tenant fairness) are the ecosystem's prerequisites, not internal
milestones. When this document says "the wall", it is naming the last
technical barrier between the project's architecture and its promise.

---

## See also

* [`01-system-overview.md`](01-system-overview.md) — principles and layer map
* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) — GMMU, channels, GPFIFO, GR, QMD concepts
* [`09-full-system-picture.md`](09-full-system-picture.md) — the end-to-end map
* [`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md) — interaction, locks and scheduling detail
* [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) — the physical session and the P0–P6 ladder
* [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) — the GR launch chain and its leading edge
* [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) — the execution gate
* [`../../debug/issue/issue-number-1.md`](../../debug/issue/issue-number-1.md) and [`issue-number-2.md`](../../debug/issue/issue-number-2.md) — reproducibility and multi-kernel tracking
* [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md) — the offline firmware debugger used for offline-first work
* [`../../wiki/Execution-Plane.md`](../../wiki/Execution-Plane.md) — the condensed public wiki version of this document

## Change log

| Date | Change |
|---|---|
| 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. |
| 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (`svc_thread`, `svc_gr_request`, `svc_pending_add`, H3 guard, readiness gate); GR completion §3.4/§6.1 reconciled with FORENSICS-grv5 (wake-up fires; the counter publication is the leading reading); discovery cadence wording corrected (2-second paced passes, not "milliseconds"); mode-matrix source note corrected. |