# The vGSP Execution Plane (vEP) > **What this document is.** The canonical architecture and design document for > the layer of `vgsp.ko` that turns guest-authored GPU work into physical > execution and guest-visible completion: from a guest CUDA kernel launch's > GPFIFO push, through ring discovery, VA/PA truth, semaphores, counters and > notifiers, the QMD and shadow windows, submission to the physical engines, > and the completion published back to the guest. It names that layer > **vGSP Execution Plane (vEP)**, defines its boundaries and its four work > domains, and maps every major mechanism to the code that implements it. > > **Audience.** Developers and architects working on guest execution, the > forwarding ladder, or the GR launch frontier. Read > [`01-system-overview.md`](01-system-overview.md) and > [`09-full-system-picture.md`](09-full-system-picture.md) first for the whole > machine; read [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) > for the physical-session lifecycle this document builds on. > > **Status: 2026-10-08, tree `master @ 1f571dda`.** Every empirical claim below > cites a commit, a code location, a tracked document, or a local campaign > evidence file (paths under `backups/evidence-…/` are local evidence > artifacts, not shipped in the repository). Where a mechanism is a leading > hypothesis rather than a proven fact, the text says so. Nothing in this > document is a stability or support claim. Status tags used throughout (same convention as docs 09/10/11): `[live]` proven with the stock driver; `[gated]` implemented but default-off or parameter-gated; `[wip]` partially implemented; `[wall]` the current blocker; `[target]` the named success latch; `[planned]` designed, not implemented. --- ## 1. What the vEP is: name, boundaries, domains ### 1.1 The name and why **Canonical name: vGSP Execution Plane (vEP).** > The **vGSP Execution Plane** is the host-side subsystem of `vgsp.ko` that > turns guest-authored GPU work into physical execution and guest-visible > completion. It spans channel-ring and USERD discovery, GPU VA-to-PA truth, > semaphore-release application, completion-counter and notifier publication, > QMD handling for forwarded launches, and submission to the physical GPU's > engines under the module-owned physical-backend session. Two framing terms, used consistently from here on: * **vGSP (virtual GSP)** — our host-side reimplementation of the GSP/GSP-RM interface: the virtual device plus the emulated firmware service that the stock guest driver interacts with. `vgsp.ko` is vGSP's implementation. The guest believes it is talking to its GPU's GSP; vGSP is the thing it is actually talking to. * **vEP** — the execution half of vGSP. The *control plane* (boot handshake, message queues, RPC, RM objects, BAR shadows) makes the guest driver run; the *execution plane* makes the guest's work run. This document covers the second half and only the seams of the first that shape execution. The name is **ours**, the name of a subsystem we implement. It is deliberately not a vendor term: the three-letter form reads as the sibling of the existing project vocabulary (`vGSP`, `vGPU`, `vgsp.ko`), and "execution plane" states the job (execute guest work) rather than the mechanism (forwarding), which leaves room for the fast paths that stay in software. **Alternates considered and rejected:** | Rejected name | Why not | |---|---| | "Work plane" / "work path" | Already used in [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and [`09-full-system-picture.md`](09-full-system-picture.md) for the software-executed bridge inside the pump; reusing it would silently change the documented model. | | "Submission plane" | Covers only the submit half; hides discovery, translation, counters and completion — over half the subsystem. | | "Forwarding plane" | Mechanism-centric: forwarding is one execution mode (some work is applied in software by design), and "forwarding" already means a specific ladder in docs 11. | | "Data plane" | Networking connotation; the plane's most latency-critical work is state operations (releases, fences), not bulk data. | | "vGPU Work Engine" | "Engine" is the hardware term for GR/CE; naming software an engine invites confusion in every review. | | "GPU Execution Plane (GEP)" | Reads as a hardware/PCIe concept, not as our software subsystem. | | "RAS plane" (Ring/Alloc/Sem) | Cryptic, and it omits the completion/publication half. | | "vGSP Runlist" | Collides with the hardware runlist/TSG concept it actually schedules onto. | ### 1.2 Scope: in and out **In scope (implemented by the vEP):** * the service thread and its per-channel pump (`svc_thread`, `svc_pump_channels`, both `kernel/gsp_sema_service.c`); * ring/USERD discovery, autopin and the VA-to-PA truth model (`svc_ring_autopin`, `svc_gr_ring_autopin`, `svc_gr_content_discover`, `svc_chan_userd`, `svc_lookup`, `svc_walk_vaspaces`, `svc_gmmu_walk`, `svc_pd_discover`, `svc_targeted_resolve`, `svc_pool_pa_resolve`); * push decode and QMD capture (`svc_decode_stream`, `svc_qmd_accum`, `svc_gr_qmd_decode`, `svc_gr_cand_capture`); * release application and completion publication (`svc_sem_write`, `svc_counter_release_apply`, `svc_counter_derive`, `svc_counter_autoreg`, `svc_counter_flush_pending`, `svc_counter_publish`); * notifier and channel-event delivery (`svc_notify_complete`, `gsp_rm_post_channel_event`); * physical submission and its mode matrix (`svc_gr_req_fill`, `svc_gr_request`, `svc_fwd_gr_work`, `gsp_phys_backend_guest_gr_submit`, `gsp_phys_backend_guest_ce_copy`, the ladder channel and its VAs); * the pacing of all of the above (doorbell steering, discovery cadence, the single-inflight ordered workqueue) and its sharing across mdevs. **Out of scope (context, not this layer):** the GSP boot handshake and message-queue transport ([`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md)), the RM control/object surface as a whole ([`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §4, `kernel/gsp_rm_emulation.c`), BAR shadows and MSI-X injection ([`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md)), the physical claim/boot lifecycle ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)), and the guest stack ([`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md)). The vEP *uses* those interfaces; it does not own them. ### 1.3 The four work domains The vEP can be described as one pipeline, but it answers four distinct questions. This mapping is the document's organizing model. | Domain | Question | vEP components | State | |---|---|---|---| | **Scheduler** | Which guest channel's work runs next, and at what pace? | Doorbell token steering (`gsp_sema_service_doorbell`, called from `main.c:865`); per-pass channel pump (`svc_pump_channels`); discovery cadence (`svc_gr_content_discover`); single-inflight ordered workqueue `gsp-phys-work` (`svc_gr_request`, `svc_fwd_request`); deferred-release hold (`svc_fwd_hold`); the service thread loop (`svc_thread`) | `[live]` software pacing and ordering; `[planned]` tenant fairness | | **Allocator** | Where do rings, USERDs, counters, notifiers, scratch pages and PT windows live, and who owns them? | CE/GR ring autopin and pins (`svc_ring_autopin`, `svc_gr_ring_autopin`, `svc_pin_store`/`svc_ipin_store`); GR content-keyed ring/push pairing (`svc_gr_content_discover`, `svc_gr_content_pin`); counter slot derive/autoreg (`svc_counter_derive`); ladder scratch windows and PT pools (window-0 shadow pool, data-prewin pool, the 128-byte PTE fetch-line pool in `gsp_phys_booter.c`); the BAR1 vidmem shadow (`vgpu->bars[1].vaddr`) | `[live]` discovery, pins, scratch pools; `[wip]` protocol-derived allocation; `[planned]` real VRAM backing | | **Control** | Which RM objects and controls shape execution, and how are they emulated or forwarded? | Channel registration from `GSP_RM_ALLOC` (`gsp_rm_register_channel` → `gsp_sema_service_channel_add`); event registration; `SET_PAGE_DIRECTORY` recording; `PROMOTE_CTX`; engine/class topology; the mode matrix (§2.3) | `[live]` registration + emulate; `[gated]` read-only control forwarding; `[planned]` alloc/free forwarding | | **Share** | How do clients and mdevs share one physical GPU with isolation and fairness? | mdev backend refcount and ownership (`gsp_phys_backend_start/stop`); one real GSP session serialized by `gsp_phys_rm_lock`; one ordered workqueue and one in-flight request slot; per-mdev emulation context; planned per-mdev real clients and fairness policy | `[live]` single-tenant; `[wip]` per-client state reset (ISSUE-0002); `[planned]` multi-tenant | The four domains cut across one code path. The rest of this document follows that path: the architecture and its diagrams (§2), how each stage actually works today (§3), the end-to-end GR chain (§4), then the program-level goals, challenges, portability and future work. --- ## 2. Architecture ### 2.1 Component map ```mermaid flowchart TB subgraph GUEST["Guest VM - stock software"] APP["CUDA application"] LCU["libcuda - builds push + QMD"] GKO["nvidia.ko - RM + UVM"] APP --> LCU --> GKO end subgraph VDEV["vgsp.ko - guest-facing"] BAR["BAR0 shadow + doorbell 0x88704"] RM["GSP-RM emulation - controls, allocs, events"] end subgraph VEP["vGSP Execution Plane"] DB["Doorbell steering"] PUMP["Channel pump svc_pump_channels"] DISC["Ring/USERD discovery + autopin"] TRUTH["VA-to-PA truth - walk, pins, rescue"] DEC["Push decode + QMD capture"] REL["Release apply + counters + notifiers"] WQ["Ordered workqueue gsp-phys-work"] SUB["Ladder submission gr/ce"] MODES["Mode matrix"] PUMP --> DISC --> TRUTH --> DEC --> REL DB --> PUMP DEC --> WQ --> SUB MODES --> SUB end subgraph BE["Physical backend - real GSP session"] LAD["Module-owned ladder channel"] GSP["Real GSP-RM server"] ENG["GR and CE engines"] end subgraph HW["GPU hardware"] GMMU["GMMU page tables"] VRAM["VRAM / FB"] end GKO -->|"MMIO + DMA + MSI-X"| BAR BAR --> DB RM -->|"channel/event/vas registration"| PUMP REL -->|"release, counter, notifier, event"| RM RM -->|"MSI-X"| GKO TRUTH -->|"reads guest page tables"| GKO SUB --> LAD --> GSP LAD -->|"GPFIFO + doorbell"| ENG GSP --> ENG ENG -->|"release semaphore"| SUB ENG --> GMMU SUB -.->|"planted PTEs"| GMMU VRAM -.-> GMMU classDef done fill:#d5f5d5,stroke:#1e7a1e classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a classDef hw fill:#e2e8f0,stroke:#4a5568 class DB,PUMP,DISC,TRUTH,DEC,REL done class WQ,SUB,MODES wip class BAR,RM done class LAD,GSP,ENG hw class GMMU,VRAM hw ``` ASCII fallback (reading top to bottom: guest → vgsp.ko → physical GPU): ```text Guest VM (stock) CUDA app -> libcuda (push + QMD) -> nvidia.ko (RM, UVM) | MMIO write to BAR0 doorbell 0x88704 ^ MSI-X (events) v | vgsp.ko ============================================ | ===================== BAR0 shadow / doorbell -> Doorbell steering | GSP-RM emulation <------- release/counter/event --+ | channel, event, vaspace registration v +----------------- vGSP Execution Plane ----------------------+ | pump -> ring/USERD discovery -> VA-to-PA truth -> decode | | -> release apply + counter publish + notifier | | -> ordered workqueue -> ladder submission (mode matrix) | +-------------------------- | ---------------------------------+ v Physical backend (module-owned GPU, real GSP session) ladder channel -> real GSP-RM -> GR/CE engines -> GMMU/VRAM ^ engine-authored release semaphore (polled by the submit) ``` ### 2.2 Data flow of one CUDA kernel launch The full chain, from CUDA context creation to the guest waking on the completion. Stages 4–8 are shared by every channel type; stage 9 onward is the GR-specific chain of §4. Status per stage is in the table after the diagram. ```mermaid sequenceDiagram autonumber participant CU as Guest CUDA app participant LC as libcuda / nvidia.ko participant DEV as BAR0 shadow + doorbell participant PU as vEP pump (service kthread) participant TR as vEP truth (walk/pins/rescue) participant DC as vEP decode + capture participant WQ as Ordered workqueue participant LB as Ladder submission participant GP as Physical GSP + GR engine participant CM as Completion publication CU->>LC: cuCtxCreate, cuMemAlloc, cuLaunchKernel Note over LC: builds GR push: ctx-init methods,
QMD install, SASS pointers, params LC->>DEV: GPFIFO entry appended; doorbell 0x88704 DEV->>PU: doorbell token steering (main.c:865) PU->>TR: resolve ring VA and USERD (discovery/autopin) PU->>TR: read GP_PUT, fetch GPFIFO entry, decode method stream TR-->>PU: translated pages (or bounded rescue) PU->>DC: accumulate QMD window + launch candidate DC->>WQ: one GR request (single in-flight) WQ->>LB: guest GR submit (QMD relocate, PTEs, method block) LB->>GP: ladder GPFIFO publish + doorbell GP->>GP: front-end fetch, inline-QMD install [wall],
ctx switch, SASS dispatch GP-->>LB: engine authors release semaphore in module/guest page LB-->>PU: completion observed (poll) PU->>CM: apply deferred release once; publish counter; notify CM->>DEV: channel event -> MSI-X (gsp_rm_post_channel_event) DEV->>LC: guest wakes; CUDA continues ``` ASCII fallback of the same chain: ```text cuLaunchKernel -> push built (ctx init, QMD, SASS ptrs, params) + GPFIFO entry + doorbell -> pump reads GP_PUT / entry; auth: ring + USERD discovered, page translated -> decode stream; capture QMD window and launch candidate -> ordered workqueue: one in-flight GR request -> ladder submission: relocate QMD, plant PTEs (pre-built fetch lines), build launch methods, GPFIFO publish + doorbell on the real channel -> physical front-end fetch -> inline-QMD install [wall] -> GR engine -> engine authors release -> poll observes -> apply deferred release exactly once + counter publish + notifier -> channel event -> MSI-X -> guest tracker wakes ``` | Stage | What happens | State (2026-10-08) | |---|---|---| | 1 | Guest CUDA builds the launch (push methods, QMD, params, SASS refs) | `[live]` observed with the stock driver | | 2 | GPFIFO entry + doorbell written into the BAR0 shadow | `[live]` | | 3 | Pump steering, ring/USERD discovery, entry fetch, translation | `[live]` for CE; `[wip]` for GR (discovery now succeeds; see §3.2) | | 4 | Push decode; QMD window accumulation; launch candidate capture | `[live]` observed (ring entry consumed; methods execute) | | 5 | Physical submit on the ladder channel (mode-matrix shaped) | `[gated]` default-off; latched on `rmcall_gr_exec_ok` | | 6 | Front-end fetch + method execution on the real channel | `[live]` observed | | 7 | **Inline-QMD install / engine reads the descriptor** | **`[wall]` not observed** | | 8 | Context switch + SASS dispatch to SMs | `[wall]` not observed for forwarded launches | | 9 | Engine-authored release + channel release | `[wall]` absent for forwarded launches | | 10 | Completion application, counter publish, notifier, event, MSI-X | `[wip]` CE path live default-off; for GR the counter-slot publication and the GP_GET write-back are the open question (ISSUE-0001, FORENSICS-grv5) | The stage table's authority is [`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) (tracked) and the 2026-10-08 campaigns below it. Stage 1–4 status is re-verified by the `grv3` campaign: ring entries *were* consumed for ranks 0–2, releases *were* applied, and the guest still did not proceed (local evidence `backups/evidence-20261002-victory/analysis-2026-10-08-stall-v2/FORENSICS-grv3.md` §0 and §2). The later `grv5` campaign (same evidence directory, `FORENSICS-grv5.md` §1) un-gated the completion traces with `sema_dbg=Y` and moved the second face from the wake-up to the counter publication (§6.1). ### 2.3 The mode matrix The physical submit path is parameterized by a mode matrix: a set of module parameters, almost all default-off, that select rebuild-free A/B variants of the GR context commit, the launch method stream, the memory windows and the completion relay. The matrix exists because the rules the front-end and engine enforce are still being recovered; each knob isolates one variable so a supervised cycle can falsify one hypothesis at a time. Every non-neutral knob is documented in `kernel/gsp_phys_gpu.c` with its rationale and rollback (the execution gates registered in `main.c`, e.g. `gsp_phys_gr_forward` and the `sema_*` cadence knobs, follow the same rule); the parameter is the experiment's record. ```mermaid flowchart LR subgraph CTX["GR context commit"] A1["gr_promote 0..35
promote shape bisection"] A2["srv_ctxbuf / srv_ctxcommit
server-managed ctx"] A3["channel variants V5/V6
golden replay"] A4["vas_shared / ctxshare_force
VAS ownership"] end subgraph LAUNCH["GR launch shape"] B1["prelude / qmdvirt / pcas
method-stream form"] B2["lmem_en - SKEDCHECK05 fix"] B3["lane_skip / lane_tail
schedule-state controls"] B4["r1_first / recommit
ordering"] end subgraph MEM["GR memory windows"] C1["qmdwin / shadow_win0
win0 PT line"] C2["data_prewin / ptr_guest
PTE kind"] C3["mail_patch / inst_sys
instance aperture"] C4["gpc_mmu_init
GPC walker"] end subgraph COMP["Completion + diagnostics"] D1["rel_mod - engine-completion discriminant"] D2["skedrec / img_probe
notifier_sys / faultbuf_sys"] end subgraph WAVES["CE forwarding waves"] E1["work_forward -> work_defer_release"] E2["work_hw_primary -> relay"] E3["fault_relay / fault_inject"] end LAUNCH --> MEM --> CTX CTX --> COMP WAVES --> COMP ``` ASCII fallback: ```text CTX COMMIT gr_promote(0..35) srv_ctxbuf srv_ctxcommit srv_main_ignore channel(V5/V6) golden golden_exec_shape ctxshare_force vas_shared LAUNCH SHAPE prelude qmdvirt pcas pcas_two_step lmem_en lmem_base lmem_tpc_size lane_skip lane_tail r1_first recommit override_hex MEM WINDOWS qmdwin shadow_win0 data_prewin ptr_guest pte_kind mail_patch inst_sys gpc_mmu_init COMPLETION rel_mod skedrec img_probe notifier_sys faultbuf_sys CE WAVES work_forward -> work_defer_release -> work_hw_primary -> relay fault_relay fault_inject GATES gsp_phys_gpu mdev_owned gr_forward guest_mem static_live ctrl_forward ``` Not all of these are part of the vEP's steady state: `gsp_phys_gpu`, `gsp_phys_hazard_ack` and `mdev_owned` gate the physical session itself (docs 08/11); `static_live` and `ctrl_forward` are control-plane overlays. Everything else in the matrix shapes execution. The single most important gate is `rmcall_gr_exec_ok`: the latched proof, set only once a real GR submission has executed on the ladder (`gsp_phys_booter.c:14614`), that `gsp_phys_backend_gr_ready()` (`:14865`) requires before any guest GR launch may forward. A parameter alone can never make the module pretend GR works. --- ## 3. Design principles and implementation reality ### 3.1 Principles the implementation enforces 1. **Emulate the interface, execute the work.** The vEP never modifies the guest driver; it satisfies the interfaces the guest observes and arranges real execution underneath (project principle, doc 01 §1). 2. **Guest truth over guesses.** Translations come from driver-published structures (VAS page directories, channel descriptors) or validated content, with a read-back proof before a pin is latched; rank-derived and convention-derived addresses are candidates, never facts. The pinned truth model is explicit in `gsp_sema_service.h` (`struct sema_channel_desc`, `struct sema_vaspace_desc`). 3. **One writer per structure.** Channel/vaspace state is copied out under `svc.chan_lock`/`svc.vas_lock`; injected pins never overwrite authoritative discovered data; guest-status-queue writes are single-writer in `gsp_rm_send` (doc 10 §5, doc 11 §1.3). 4. **Bounded budgets everywhere.** Bounded scans, retry counts, grace windows (`sema_ce_grace_ms=25000`, `sema_gr_grace_ms=8000`, `sema_push_hold_ms=2500`), a 5→60-pass discovery epoch with cooldown, bounded poll quanta (5 ms in the ladder, `gsp_phys_booter.c:1404`). A miss must degrade, never freeze; the starvation telemetry from ISSUE-0001 fix #2 exists to make a bound miss visible. 5. **Default-off, fail-closed, one-line rollback.** Every execution-changing behavior is behind a parameter, and every parameter has a fallback story (docs 11 §5). Forwarding refuses when the backend is not READY; a held release is applied exactly once by software when the hardware path misses (P6b wave 1.5 / P6c). 6. **Evidence gates the latch.** A capability is enabled only when a named acceptance signal has been observed (`REAL GR SEM OK` → `rmcall_gr_exec_ok`); "silence is not health" (doc 03-execution-frontier §8). ### 3.2 Ring discovery: kernel-native, both apertures, content-keyed The pump cannot execute anything until it can name two addresses per channel: the GPFIFO ring VA's translation, and the USERD's translation (PUT/GET live at USERD+0x8c/0x88, `SVC_USERD_GP_PUT`/`SVC_USERD_GP_GET`, `kernel/gsp_sema_service.c:143-144`). Today discovery has three cooperating sources, in order of preference: 1. **Injected pins** (debug tooling and, historically, the userspace watcher pipeline) — authoritative but external; `svc_pin_lookup_any`. 2. **The kernel-native trusted walk** — driver-published page directories recorded from `SET_PAGE_DIRECTORY` (`svc_walk_vaspaces`), a self-discovered directory (`svc_pd_discover`, `svc_walk_discovered`), and per-VA targeted resolution (`svc_targeted_resolve`), all through the shared four-level `svc_gmmu_walk` with read-back validation (`svc_lookup_trusted`). 3. **Content-keyed discovery** — when no translation source can name a GR ring, `svc_gr_content_discover` scans for the *content* of the ring and the staged ctx-init push, pairs them per rank, and validates the pair in `svc_gr_content_pin` before pinning. The scan covers **both apertures**: the emulated vidmem aperture (`vgpu->bars[1].vaddr`) and guest system RAM through `svc_read()` = `vfio_dma_rw` (`gsp_sema_service.c:754`). The history matters because it explains the current code shape: * CE channels were discovered first and stably: the CE ring convention `FB 0x80000 + 0x20000*rank` pinned 6/6 CE rings (`ring_auto=6`) in the stalled v2/grv1 boots (FORENSICS-v2 §a; FORENSICS-grv1 §0.1). * GR rings follow `VA 0x200200000 + 0x3000*rank` with the USERD at `+0x2000` (`SVC_GR_RING_VA_BASE/STRIDE`, `SVC_GR_USERD_OFF`, `gsp_sema_service.c:7334-7431`), but *where the ring physically lives* is boot-dependent: green m104 found all 20 GR rings at `FB 0x140000 + 0x3000*rank`; grv1's guest never wrote them to FB; grv3's guest wrote only ranks 0–2 (FORENSICS-grv3 §1). * FORENSICS-grv1 proved the ring was **not in guest RAM** and recommended an FB-aperture scan; FORENSICS-grv3 identified the 60-pass epoch death and recommended periodic re-arm plus an FB-first fast path. Those three changes are in the tree as commits `ebcfc1c2` (both-aperture kernel-native walk + content fallback), `7abd8f87` (FB-aperture discovery + per-channel re-arm, FORENSICS-grv1 R1) and `66388560` (periodic re-arm + FB-first fast path, FORENSICS-grv3 R1). In the current code (`svc_gr_content_discover`, `gsp_sema_service.c:7650`): the FB scan runs first and fast (the aperture is host RAM, so a page-stride pass is milliseconds), passes are paced at one per 2 s (`:7731`), push candidates persist across passes so a late ring can pair without re-walking 6 GiB, a registration re-arms the epoch immediately, and hitting the 60-pass cap starts a 30-second cooldown and then a fresh epoch while any registered rank still lacks a ring — never the old permanent retirement (`:7712-7734`). **Known limits.** The content path is gated to ranks 0–7 (`SVC_GR_CONTENT_RANKS=8`) and to the documented VA layout (`gpfifo_va == BASE + STRIDE*rank`); a layout change deliberately falls back to the safe paths. The walk depends on the guest publishing the owning VAS: grv1/grv3 saw only two `SET_PAGE_DIRECTORY` publications, and the ring VA was not covered by either (FORENSICS-grv1 §2). Content discovery is the fallback for exactly that hole; a protocol-derived source for the ring VA remains the durable direction (ISSUE-0001 "durable direction", `issue-number-1.md:107`). ### 3.3 The semaphore service: release application `gsp_sema_service.c` is the vEP's engine room. Its service thread (`svc_thread`, `:13343`) loops over the open mdev, runs the pump, retries pending releases, and paces discovery. The release path: 1. **Decode** (`svc_decode_stream`, `:5170`) turns a consumed GPFIFO entry's method stream into operations; releases are recognized as increment-method writes to a semaphore VA with a payload. 2. **Translate** the target VA: `svc_lookup_trusted` first, then the pool resolver `svc_pool_pa_resolve` (`:2284`) — the resolver ISSUE-0001 fix #1 completed by adding the driver-published VAS walk (`svc_walk_vaspaces`, `:1274`); a miss is queued in `svc_pending_add` (`:6166`) and retried every pass by `svc_apply_pending` (`:6201`), with the per-entry `tries` telemetry added by the same fix (`7f21daa7`). 3. **Apply** (`svc_sem_write`, `:2474`) reads the current word, writes only forward (never backwards), and is the sole software author of a release; a forwarded operation's releases can instead be deferred and then applied exactly once by the hold's finish path (P6b wave 1.5), or authored by the engine itself under P6c's relay (default off, docs 11 §P6c). The complementary **counter** path handles GR completion *counters* (distinct from CE tracking semaphores): `svc_counter_derive` (`:4658`) derives the counter slot from structural translations; `svc_counter_autoreg` (`:4710`) registers it per GR channel by the fixed-total formula rank r → `0x20400fec0 + (SVC_GR_COUNTER_TOTAL-1-r)*0x10` (`:4721`); `svc_counter_release_apply` (`:4433`) applies decoded releases to the slot; `svc_counter_flush_pending` (`:4575`) lands payloads recorded before the slot was known; `svc_counter_publish` (`:9077`) writes the GPFIFO PUT into the slot when it is ahead of the current value. Three facts learned the hard way are now encoded: the counter pool total is fixed at 20 (`SVC_GR_COUNTER_TOTAL`, commit `fcb96912`) because a partial live channel count poisons every later rank's slot; the early publish is guarded so a guessed PUT is never written (the H3 finding, `gsp_sema_service.c:11489-11512`, FORENSICS-v2 §b/H3); and the fixed-total mapping points each rank's slot at the same word the ctx-init push releases to, so the publish becomes a structural no-op (`put == counter_last`, `val <= cur`; `:9096-9109`) — the grv5 stall ran with `counter publish` ×0 while `pub_calls=180142` and the release, notifier clear and event post all fired (FORENSICS-grv5 §1, verdict). The E1/E2 debug mirrors and GP_GET read-back for that finding landed debug-gated as commit `c39b2a1c`. ### 3.4 Notifiers, GPFIFO consume, event delivery * **GPFIFO consume.** The pump reads `GP_PUT` from the USERD, walks entries from its own GET, decodes, and advances (the GP_GET write-back is `gsp_sema_service.c:12682-12684`, the end-of-pass publish `:12731-12732`); a replay-suppression path keeps pin batches from re-consuming old slots (`gpfifo_skipped`), and a per-entry grace window holds an untranslatable GR entry before it is consumed as stale (`sema_gr_grace_ms`, default 8 s). `svc_userd_probe` (`:7099`) distinguishes "unreadable" from "readable but idle (0/0)" so a zero page is never mistaken for a live channel. * **Notifiers.** `svc_notify_complete` (`:7207`; the clear and event post are `:7240-7277`) services the channel's error notifier token and posts the channel event. The notifier page address and aperture come from the channel allocation descriptor (`sema_channel_desc.notifier_pa/as`, `gsp_sema_service.h:31-32`), recorded at channel registration. * **Events.** `gsp_rm_post_channel_event` (`gsp_rm_emulation.c:602`) finds the guest's registered event objects (`rm->events[]`, registered from the event alloc in `gsp_rm_handle_rm_alloc`, `:2443-2489`) and sends the RPC event; the reply path injects MSI-X through the live eventfd (`gsp_emu_inject_completion_interrupt`). A green boot shows the sequence `notifier cleared … status=0xffff` (per GR channel) followed by the event post. grv3 (traces off) showed neither and re-queued the identical push; the grv5 repeat with `sema_dbg=Y` settled the question — the notifier cleared ×34 and the event posted, while the counter publish never wrote. The leading reading is therefore the counter publication, not a missing wake-up (FORENSICS-grv5 §1; §6.1). ### 3.5 Physical submit Forwarding is a two-stage ladder. The sema service captures a candidate and queues one request on the single ordered workqueue `gsp-phys-work` (`svc_gr_request`, `:10998`; `svc_fwd_hold` is the single in-flight slot). The work item calls into the backend: `gsp_phys_backend_guest_ce_copy` for CE (`gsp_phys_booter.c:15621`) or `gsp_phys_backend_guest_gr_submit` (`:16457`) for GR, gated by `gsp_phys_backend_work_ready()`/`gsp_phys_backend_gr_ready()` (`:14849`/`:14865`), which require READY, a live claim, an unload not committed, a client-memory ladder channel, a work token, and every ladder VA; the GR submit additionally re-checks `gsp_phys_guest_mem` and the active guest RAM backing at `:16509-16510`. GR submission (`gsp_phys_backend_guest_gr_submit`) is the vEP's most parameterized function: it relocates and pre-writes the QMD (inline-QMD window, `gsp_phys_rmcall_gr_qmdwin`), copies program/param pages into shadow windows whose PTEs are planted at stage 3 before the engine's first fetch (`gsp_phys_rmcall_gr_shadow_win0`, `data_prewin`), builds the launch method block (`gsp_phys_rmcall_build_gr_launch`, `:15862`) from the recovered method vocabulary (compute init, inline-QMD window, PCAS pairs, report semaphore), publishes it to the ladder channel's GPFIFO and rings the doorbell (`gsp_phys_rmcall_publish_locked`, `:14345`), then polls for the release with `rel_mod`/`skedrec` diagnostics. The PTE fetch-line rule is enforced: the engine caches PTE fetches at 128-byte granularity with no invalidate path, so each scratch pair consumes a never-reused fetch line from pools built before first engine access (docs 11 §P6; `gsp_phys_booter.c:1044-1070`). Measured cost: the m141-r3 baseline (`895ab8b8`) is `elapsed_ms=189` for the first forwarded launch and `166` for the second, of which only ~7.9 ms is the post-doorbell engine path; ~180 ms is serialized per-window `PDE_UPDATE` forwards (local evidence and the prepared harness: `tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1.3). The baseline is a forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. ### 3.6 Citation map for this section | Claim | Authority | |---|---| | Both-aperture kernel-native GR discovery + content fallback | commit `ebcfc1c2`; `gsp_sema_service.c:7334-7529` | | FB-aperture discovery + re-arm; periodic re-arm + FB-first | commits `7abd8f87`, `66388560`; `gsp_sema_service.c:7650-7800` | | Ring not in guest RAM; RAM-only scan limitation | `FORENSICS-grv1.md` §0/§1/§A (local evidence) | | 60-pass cap dead-end; completion-chain questions | `FORENSICS-grv3.md` §0/§3/§4/§5; wake-up refuted and counter publication leading in `FORENSICS-grv5.md` §1/§4 (local evidence) | | CE clean consumption at 89 releases; no starvation | `FORENSICS-v2.md` §b/§c (local evidence) | | Pool resolver + starvation telemetry | commit `7f21daa7`; ISSUE-0001 fix #1/#2 | | Counter slot fixed-total 20 | commit `fcb96912`; the fixed-total publish no-op is `FORENSICS-grv5.md` §1/§4 (local evidence) | | Release written + notifier cleared + event posted while `counter publish` ×0 | `FORENSICS-grv5.md` §1/§4 (local evidence); commit `c39b2a1c` (E1/E2 debug mirrors) | | PROMOTE_CTX `bInitialize` must stay set | commit `88d47c1f`; ISSUE-0001 fix #3 | | GR exec latch + readiness gate | `gsp_phys_booter.c:14614`, `:14865`; `main.c:2609` | | PTE fetch-line rule, scratch pools | docs 11 §P6; `gsp_phys_booter.c:1044-1070` | | m141-r3 baseline 189/166 ms decomposition | `tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1 | --- ## 4. End-to-end GR execution, in our words This section defines the vocabulary as the code uses it, then walks the GR chain. The concepts are standard NVIDIA terminology; the definitions below are **ours**, written against the interfaces we implement. | Term | Our definition | |---|---| | **GR engine** | The physical engine that executes shader work: both compute grids and graphics pipelines. CUDA kernels are GR work. It is the one engine that can run arbitrary guest-authored code, which is why it is the frontier. | | **Channel** | A hardware execution context: a GPFIFO ring, a push-buffer pointer, a USERD control block, an instance/RAMFC block, and the engine object bound to it. The guest creates channels through RM allocs; the vEP registers them (`gsp_rm_register_channel` → `gsp_sema_service_channel_add`). | | **GPFIFO** | The ring of entries a channel consumes. Each entry points at a push buffer (or carries inline data); the front-end fetches it, then executes the method stream it references. | | **USERD** | The channel's control block in GPU memory. `GP_PUT` (offset 0x8c) is where the guest declares submissions; `GP_GET` (0x88) is the channel's consumption cursor. The pump reads these through `svc_read_ap` with the channel's aperture. | | **Push buffer / method stream** | The command stream: incrementing-method headers plus payload, addressing engine registers and subchannels. Decoding it is how the vEP learns what the guest asked for (`svc_decode_stream`). | | **QMD** | The launch descriptor: the fixed-layout structure GR reads to dispatch a grid (program address, grid/block dimensions, parameter pointers, constant-bank pointers, release targets). CUDA builds it; the forwarded path must install the inline copy into the channel's descriptor buffer in the form the engine reads (§6, the wall). | | **Context / ctx-init** | The per-channel engine state block (main + per-VEID subcontext images) that must be initialized and committed before GR can switch to the channel. The guest drives this with `GPU_PROMOTE_CTX`; the server-side rules are what the `gr_promote`/`srv_ctxcommit` matrix is bisecting. | | **Golden context** | The reference channel/3D object set the stock driver creates right after load so the server-side GR instance state exists (the `gsp_phys_rmcall_golden` replay). A context committed without that state does not dispatch. | | **SASS** | The compiled shader machine code the GR executes. Its pointers reach GR through QMD constant banks; for forwarded launches the vEP relocates referenced pages into engine-walkable windows rather than trusting guest pointers directly. | | **Semaphore release** | The completion primitive: a method that writes a payload word into a semaphore VA. CE work releases tracking semaphores; GR work releases the channel semaphore and the QMD release word. The vEP applies releases in software or, gated, lets the engine author them. | | **Counter** | The GR completion counter slot: the u32 the guest's userspace tracker polls for channel progress; the vEP derives, registers and publishes it (`svc_counter_*`). | | **Notifier** | The channel error/completion notification block. `svc_notify_complete` services it and the RM event path delivers the channel event. | | **Shadow window / scratch PT** | Module-owned pages mapped into the engine's page-table tree at VAs the ladder controls, used to place QMD/program/parameter/data content where GR can walk it. Window-0 PTEs are pre-planted before the channel commit because late writes into already-fetched PTE lines are invisible. | **The chain the vEP implements for GR:** 1. **Context creation.** The guest RM allocates VRAM objects, a VAS, a TSG, GR channels; the vEP registers each channel at alloc time and records its descriptor. The guest's `SET_PAGE_DIRECTORY` publications become the trusted translation sources. The guest then promotes its GR context (`PROMOTE_CTX`); the emulation acknowledges and keeps `bInitialize` set (fix `88d47c1f`) so the client marks its buffers initialized. 2. **Golden context / server state.** On the physical ladder, the `golden` replay allocates the stock client's post-load channel set so the server-side GR instance state exists; the context commit variants (`gr_promote`, `srv_ctxcommit`, `vas_shared`) are the ladder's bisection of what the real server requires. The vEP's guest side does not depend on the ladder shape; the gate is the latch. 3. **QMD capture.** The pump decodes the guest's launch stream, accumulates the inline QMD window (`svc_qmd_accum`), and captures the launch candidate (`svc_gr_cand_capture`), including the release target and the referenced program/param pages. 4. **Shadow windows.** The submit relocates the QMD and copies referenced pages into window-0 shadow pages whose PTEs were planted pre-commit; data pages go through the pre-installed data pool; `mail_patch` injects a module message page when the guest's constant bank is empty. `ptr_guest` is the A/B for keeping the guest's original pointers (the SM walk appears to resolve through the guest subcontext PDB; m91). 5. **Physical submit.** The submit builds the launch method block, publishes it to the ladder channel's GPFIFO, rings the doorbell, and polls for the engine-authored release. 6. **SASS on GR.** Between publish and release, the front-end executes the methods; the inline-QMD install and the engine's read of the descriptor are the stages the current campaigns have not yet observed. Once they fire, GR switches to the context and dispatches the grid to the SMs. 7. **Completion.** The engine authors the QMD release word and the channel release; the poll observes them; the deferred release is applied once (P6b wave 1.5 semantics); the counter is published; the notifier is cleared; the channel event is posted; MSI-X wakes the guest. For GR the last step is the open counter-publication question (§6.1). --- ## 5. Enabling arbitrary CUDA code execution "Arbitrary CUDA" is the 1.0 acceptance bar: stock guest CUDA works for real workloads, not one demo, and every time. The program is staged: * **P1 — single-workload A/B.** `tools/cuda-ab/ab.sh` runs one deterministic workload on the host (bare-metal oracle) and in the guest, then compares the stable result lines. Workloads: `victory_mail` (the campaign mail kernel), `vector_add`, `smiley` (an ASCII-art deliverable). This is the rung that produced the m141-r3 first green forwarded launches in 189/166 ms (`895ab8b8`). * **P2 — the "arbitrary CUDA kernels" acceptance suite.** `tools/cuda-ab/p2/` is eleven self-checking programs covering launch shape (1D/2D/3D grids, odd sizes), parameter counts 0–12 with mixed types, register/launch-bounds variants, shared memory (reduction + tiled GEMM), streams including eight independent kernels in flight, allocator churn, device printf/assert, error reporting and recovery, a correctly-rounded compute benchmark, and a combined end-to-end program. References are deterministic (EMIT mode) until the host GPU is free for capture; the suite currently runs `[planned]` pending the reproducibility gate (`tools/cuda-ab/READINESS-P2-2026-10-08.md`). * **P3 — near-native performance.** `tools/cuda-ab/perf/` is the prepared measurement program: host-oracle vs guest ratios for init, malloc, launch (first/warm/throughput), HtoD/DtoH bandwidth, compute GFLOP/s, plus host metrics (the backend `elapsed_ms`, injected-latency proxies). Status: prepared, not yet run; every threshold provisional until a green calibration (`tools/cuda-ab/perf/PLAN-AND-HARNESS.md` §1/§4). **Proven vs pending.** Proven today: single forwarded launches of two small kernels (mail, vadd) with correct release and clean teardown, on a lucky green boot; CE-class copies forwarded end-to-end with engine-authored release and fault relay (default off); compiled kernels of the P2 complexity are not yet observed executing through the stack. Pending: reproducibility (ISSUE-0001), the GR descriptor-install wall (§6.1), the GR counter-slot publication / GP_GET write-back (§6.1; FORENSICS-grv5), multiple kernels/clients per VM (ISSUE-0002), and P2/P3 in a green boot. The gating order is deliberate: reproducibility first, then the wall, then breadth, then performance. --- ## 6. Challenges ### 6.1 Current * **The GR completion wall — two faces.** The older face is the launch-descriptor install: the forwarded launch reaches and runs front-end methods, the descriptor destination window is accepted, the context is valid at dispatch, and yet the inline-QMD install is never observed and no grid dispatches ([`04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) §1–§5). The exonerated list is long (context validity, destination mapping, entry fetch/alignment, TSG parenting); the live issues are the stream's verbatim prelude, the absent descriptor-virtualization control method, and the relocation transform, each with a one-cycle falsifier. The current code carries many of those falsifiers as mode-matrix knobs (m26 falsified the nonzero `qmdvirt` hypothesis; m72 found the SKED `LOCAL_MEMORY_TOTAL_SIZE` cancel and added the `lmem_en` block; m91 showed the SM I-fetch MMU_NACK at shadow VAs and added `ptr_guest`). * **The second face — the completion-counter publication.** grv3 showed a boot where discovery succeeded (three GR rings pinned), the ctx-init releases landed (`gr_rel=3`, counter words = 1), and the guest *still* stopped, re-queueing the identical push (FORENSICS-grv3 §0/§3). The grv5 repeat ran with `sema_dbg=Y` and settled it: the release was written to the exact pin-backed slot, `notifier cleared` fired ×34 and the channel event posted (`posted=4`) — and `counter publish` still never wrote while `pub_calls=180142`, because the fixed-total autoreg mapping makes the publish a structural no-op (§3.3; FORENSICS-grv5 §1, verdict). The leading reading is the counter-slot publication, with the GP_GET write-back as the secondary, still-open candidate (the host writes `last_get` at `gsp_sema_service.c:12682-12684` but nothing has read it back; FORENSICS-grv5 §4/H2). Both observability surfaces are debug-gated today (commit `c39b2a1c`). * **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The first fix wave landed (kernel-native pool translation `7f21daa7`, PROMOTE_CTX reply `88d47c1f`, issue register `a1407659`), and the earlier stall mode — the client freezing at DEVICE because its last CE completion could not be written — is fixed: the v2 campaign consumed all 89 CE pushes with zero holds and zero starvation. The stall then moved one phase later, to GR ring discovery, where the tree now carries the kernel-native discovery lineage (`1ba8ba65`, `ebcfc1c2`) plus the fixed re-arm of `7abd8f87`/`66388560`. Acceptance is 10/10 consecutive cold boots green (`issue-number-1.md` validation plan); that count is pending. * **Multiple kernels/clients per VM (ISSUE-0002).** A second CUDA process in one boot wedges in guest UVM because the host's per-client sema/GR state is not reset when the guest re-runs the GSP boot handshake per client; the guest's UVM holds a global lock across the registration RM exchange, so a host-side quiet path freezes every later client (`issue-number-2.md`). The durable direction is per-client/per-channel state reset plus per-mdev sema contexts and P0 isolation work, with the ordered hunks already recorded (validation plan in the issue). ### 6.2 Historical (fixed or retired — do not relitigate) * **CE starvation at the boot-critical release.** The old stall was one deferred CE release whose semaphore page had no translation, waiting on a racy userspace watcher. Fixed kernel-natively by the pool resolver's VAS walk (`7f21daa7`) and the GR ring/USERD discovery lineage; `pushes_nomap` now reaches 0 on consumed boots (FORENSICS-v2 §0/§c). * **The ring-location class.** Early discovery tried only FB conventions (`0x140000 + 0x3000*rank`) and RAM scans; grv1 proved the ring was not in guest RAM and the RAM-only content scan burned 60 passes. The fix is the both-aperture, FB-first, re-armable discovery now in the tree (`7abd8f87`, `66388560`). * **The notifier gate as a discovery symptom.** The `notifier cleared` line historically correlated with ring resolution; it is a publication-side signal and is now understood as part of the completion-publication face of §6.1 (grv5: the notifier/event wake-up fires; the counter write is what is missing), not a discovery fix in itself. ### 6.3 What is working well, and what is not **Working well:** the CE path end-to-end (copies, deferred exactly-once release, hardware-primary data, engine-authored relay, fault relay) — all live-verified default-off; kernel-native discovery and the trusted-walk truth model; deterministic counter semantics with fixed pool total; the mode-matrix discipline (one variable per supervised cycle, one-line rollback); the bounded-budget culture that turns potential freezes into logged, counted degradations. **Not working:** GR dispatch (no grid observed); the GR counter-slot publication (structurally dead under the fixed-total mapping; FORENSICS-grv5 §1); reproducibility at the required 10/10 bar; second-client state isolation; and the observability gaps that made both bugs slow to find — counter `pub_calls` counting calls rather than writes (H3; the increment still sits before the no-op guards at `gsp_sema_service.c:9085` and correction is planned, FORENSICS-grv1 §5 R3), completion traces still behind `sema_dbg` (grv5 captured them, but the telemetry must become unconditional), and the absence of a GR error-reporting object on the forwarded path ("silence is not health", doc 03-execution-frontier §8). --- ## 7. Goals, gaps, and the iteration plan ### 7.1 Goals 1. **Correctness:** a stock guest runs arbitrary CUDA in one boot, repeatedly. 2. **Reproducibility:** 10/10 cold boots green (ISSUE-0001 exit criterion). 3. **Breadth:** P2 suite green in a single boot; multi-client and multi-stream (ISSUE-0002); then multi-mdev. 4. **Performance:** near-native P3 numbers with no-regression discipline. 5. **Portability:** the vEP survives driver-version and GPU-generation churn by re-deriving facts, not by patching snapshots. ### 7.2 The iteration loop Every change follows the same evidence loop. This is the operating system of the vEP program, not an appendix: 1. **Question as a discriminating sentence + named acceptance signal.** E.g. "does the counter slot get published for a GR ctx-init channel?" with the signal `counter publish … written` + guest advancement past the re-queue (the earlier notifier/event question was answered by grv5: they fire; FORENSICS-grv5 §1). 2. **Offline first.** Use the risc-v-gsp-rm-debugger (`debug/risc-v-gsp-rm-vm/`) and its validation tools — `harness/valprobe.py` for one firmware validation branch, `harness/rehost.py` for one whole function, RPC capture/decode/compare (`debug/risc-v-gsp-rm-vm/docs/45-47`) — to test a firmware-side rule without a supervised physical cycle. The entry points are [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md) and [`../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md`](../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md). 3. **One supervised cycle, one variable.** A mode-matrix knob, a trace flag, or a single code change; never two. 4. **Evidence pair.** Session/counter pair (green vs this run) with the discriminators named before the run; campaign evidence goes to `backups/evidence-…/` (local), the public claim cites the behavior only. 5. **Land or revert.** A landed change updates the doc and (if needed) the issue; a falsified hypothesis is recorded in the knob comment so no future session re-runs it. ### 7.3 Tooling gaps and the plan for them | Gap | Tool / practice | Plan | |---|---|---| | GR rule recovery without burning cycles | risc-v-gsp-rm-debugger valprobe/rehost, RPC capture/replay (docs 45–47) | probe candidate front-end/engine rules offline; queue only falsifiable physical experiments | | Arbitrary-kernel acceptance | `tools/cuda-ab/p2/` (11 programs) | run in the first green boot after ISSUE-0001 acceptance; add a multi-process/multi-stream variant for ISSUE-0002 | | Performance measurement | `tools/cuda-ab/perf/` + `collect.py` | freeze thresholds on first green run; make the serialized `PDE_UPDATE` forwards the first optimization candidate | | Completion observability | unconditional or counted `notified`/`posted`/`publish-written` telemetry in `svc_notify_complete`, `gsp_rm_post_channel_event` and `svc_counter_publish` (FORENSICS-grv3 §5B, FORENSICS-grv5 §5) | land as telemetry, not a behavior change; treat `posted==0` after a real clear, or `publish written==0` with a registered slot, as a first-class failure; the E1/E2 debug mirrors (`c39b2a1c`) are the interim form | | Counter telemetry honesty | move the `pub_calls` increment after the write (FORENSICS-grv1 §5 R3) | land with the telemetry wave | | Multi-client state | ISSUE-0002 fix plan (per-client reset, per-mdev sema context, P0 hunks) | stage A1–A6 per the recorded plan; graduation criteria sequential/concurrent/simultaneous | | Evidence discipline | one evidence bundle per claim; counters as first-class discriminators; "no claim without a pair" | already the campaign norm (FORENSICS reports); keep counters stable enough to be quotable (see counter telemetry fix) | | Sub-agent workflows | read-only analysis sessions (`SA-ANALYSIS`, `SA-CHK`, `SA-DOC`) producing FORENSICS/readiness docs while the lead runs cycles | continue; keep analysis strictly read-only and cite line numbers; never touch the rig in analysis sessions | --- ## 8. Portability ### 8.1 What is version-portable * The **architecture** of the vEP: observe pushes → resolve truth → decode → execute/forward → publish completion. This shape does not depend on a driver version or a GPU generation; only the facts it consumes do. * The **RPC/RM object model** the vEP registers against (channel alloc descriptors, page-directory publications, event objects, `PROMOTE_CTX`) is stable in structure across the 5xx/6xx families; new fields appear, but the wire structures are versioned and compiled into the driver. * The **mode-matrix method**: parameterized, default-off, rollbackable experiments and an evidence latch (`rmcall_gr_exec_ok`) port as a practice. * The **counter/notifier/event completion model** (a polled u32, a notifier block, an RM event, MSI-X) is the same completion vocabulary across versions. * The **QMD vocabulary at a coarse level** (a launch descriptor with program, grid, parameters, release targets) is stable; exact field offsets are not. ### 8.2 What must be re-derived per version or architecture | Item | Why | Re-derivation source | |---|---|---| | Firmware/driver identity, RPC inventory, control IDs | versions add/retire controls | measurement: `firmware_analyzer` signatures, live RPC captures (`docs/45`), host-oracle A/B (docs 07 P1) | | Ring/USERD VA and FB layout conventions | the guest's memory allocator changes with the driver | first-touch probes + content discovery; never hardcode beyond the documented candidate list | | QMD layout and launch method stream | the launch vocabulary changes (Turing → Ampere → Hopper → Blackwell) | reference capture of a working launch; differential against the forwarded stream | | Context image layout, promote entry sets, golden-context shape | server-side RM rules and image formats change | ladder bisection + offline valprobe/rehost | | PTE encodings, page-table geometry, fetch-line rule | GMMU changes per architecture | hardware observation; document the encoding proven in each profile | | Engine inventory and classes | missing/zeroed records roll whole contexts back (doc 05 §6) | bare-metal oracle capture; profile tables | | Scratch-window VA layout and pools | ours, but coupled to the ladder channel's VAS geometry | re-lay per profile; keep the pre-commit planting rule | Portability discipline: **protocol-level facts port; address-level facts do not.** Any new version or architecture enters as a profile plus a measured delta, never as a guess (docs 06 §5, `RUNBOOK.md` S-scenarios). ### 8.3 Concrete paths * **Driver 610.** Hard gate first: RM 610 requires explicit named-human approval — no acquisition, analysis, emulation edit, physical work or support claim before it (`docs/23-RM610-READINESS.md`). Once approved: acquire the image, run one-command triage (`risc-v-gsp-rm-debugger.sh triage`, `docs/40`), diff the RPC inventory with `harness/version_diff.py`/`ci_release.py` (`docs/21`), then port the vEP facts in the order of §8.2. This is speculative until the gate opens; nothing in the current tree pre-adapts to 610. * **Grace Blackwell (GB10/GB2xx, aarch64 hosts).** A different problem class: different host CPU architecture (the module's guest-RAM translation, `read_cache_page`-based HPA mapping, must remain architecture-neutral), an engine inventory and GMMU generation with no current profile, and a firmware family the debugger would need to re-host/port (docs 07 P3). The vEP plan is: profile capture (PCI IDs, BAR geometry, engine list), GMMU encoding work, then the same discovery/decode/submit ladder on a supervised device, with the support ladder (docs 06 §1) moving only on evidence. * **Compute profiles.** The near-term "profile" is a small, per-version file of measured facts (ring conventions, VA windows, QMD offsets, method encodings, PTE kind) consumed by the vEP at runtime; today those facts are in code comments and constants with commit provenance, which is the pre-profile state. --- ## 9. Mapping vEP concepts to CUDA and to architecture ### 9.1 CUDA high-level concepts | CUDA concept | What the guest does | vEP counterpart | |---|---|---| | Context | `cuCtxCreate`; RM client/device/subdevice + a GR engine context | guest context is emulated (objects, VAS, promote); the physical ladder commits its own context (`gr_promote`/`srv_ctxcommit`) | | Module / function | loads SASS/cubin into GPU memory; `cuModuleGetFunction` | SASS reaches GR through QMD program pointers; the vEP relocates referenced pages into engine-walkable windows (`shadow_win0`, `data_prewin`, `ptr_guest`) | | Stream | ordered work queue on a channel (or a graph) | channel + TSG; the guest's push order on the channel is the stream order the pump preserves | | Kernel launch | builds push (ctx init, QMD install, params), appends GPFIFO entry, rings doorbell | capture → one in-flight request → ladder submission; the QMD install is the wall (§6.1) | | Memory | `cuMemAlloc`/copies; VA objects mapped into the VAS | emulated VRAM + guest RAM with the trusted truth model; forwarded operands resolve VA→PA→HPA via the guest RAM backing | | Synchronisation | events, stream sync, semaphores, `cudaDeviceSynchronize` | releases, counters, notifiers, channel events; the vEP is the sole publication authority (or relays engine-authored releases) | ### 9.2 Architecture details | Architecture concept | vEP relationship | |---|---| | Channel / GPFIFO / USERD | the unit of scheduling and consumption (§4); registration from the alloc descriptor | | TSG / runlist | the guest's scheduling vocabulary; the ladder's channel sits in a TSG on the real server; guest-side pacing is the pump today, tenant fairness is planned on top (doc 09 §5) | | Engine types / classes | GR is engine type 1, COPY0 is 0x9 (doc 05 §6); the ladder selects GR0 for GR work and refuses CE paths on a GR ladder (`gsp_phys_backend_ladder_ready`) | | GMMU | the translation ground truth; the trusted walk reads guest page tables, and the engine's view is fed by planted PTEs in pre-built fetch lines | | VRAM / FB apertures | emulated as a host buffer today; apertures appear in channel descriptors and pins (`SVC_APERTURE_SYS/FB`); real VRAM backing is future work | | MSI-X / interrupts | completion delivery: `gsp_rm_post_channel_event` → eventfd → irqfd; physical-engine faults are observed and relayed (C3, default off) | --- ## 10. Future challenges the current tools do not address These are the known gaps for which no tool or design exists yet. Each gets a proactive design direction so the architecture is not surprised later. 1. **Multi-tenant fairness and performance isolation.** Today one mdev, one real session, one in-flight request. The planned per-mdev real clients need a policy layer (time slices, quotas, accounting) and a measurement method that can fail (doc 09 §5; docs 06 §2.8). Proactive design: keep every per-client datum inside a container keyed by the mdev/session, and build the reset path (ISSUE-0002) as the container's lifecycle, not as scattered clears. 2. **Fault recovery semantics.** Fault relay exists (C3); fault *replay*, *cancel*, and channel recovery after a fault do not. Proactive design: model faults as first-class completion outcomes in the request record, so a fault can trigger the same exactly-once release discipline as success. 3. **Context switching and preemption between tenants.** The ladder leaves a context loaded; the client preempts its TSG for a clean first dispatch (`lane_tail`). A multi-tenant scheduler must own load/unload lifecycles. Proactive design: the mode matrix's lane controls are the seed of a documented context-switch protocol, not a launcher-only trick. 4. **Hostile-guest safety.** A malformed push, a forged VA, or a wedged ring must not corrupt the host or another tenant. Guards exist (bounds checks, read-back proofs, `gsp_phys_seq_addr_valid`), but there is no fuzzing harness for guest-authored streams. Proactive design: a synthetic-push generator against the decoder with the same evidence discipline as the physical ladders. 5. **Observability as a product property.** The vEP's bugs were found by counter archaeology; counters that count the wrong thing (H3) cost days. Proactive design: every counter must answer one named question, and telemetry for the completion path (notifier, event, publish) lands before it is needed again. 6. **Version drift under an adversarial schedule.** New images will arrive without notice; the triage/porting tooling exists but is unexercised on a real new version. Proactive design: golden sessions per supported version and a one-command triage that can fail loudly. 7. **Windows guests and non-KVM transports.** They change the RAM/IRQ plumbing under the vEP, not its pipeline. Proactive design: keep the guest-memory accessor and the event injection behind interfaces (they already are: guest-RAM backing file, eventfd), so the pipeline is transport-agnostic. --- ## 11. 3D graphics interaction Graphics workloads (OpenGL/Vulkan) are GR work too, but they differ from compute in ways the vEP will feel directly: * **Engines:** more than GR — a 3D context still uses GR (graphics pipelines), but video (NVDEC/NVENC) and image (NVJPG/OFA) engines join later; CE remains for buffer moves; the engine inventory is a correctness surface (doc 05 §6). * **Channels and sync:** drivers use multiple channels per context, TSGs for scheduling groups, and inter-engine semaphores/fences (a copy waits on a render; a present waits on a copy). The vEP's release/counter vocabulary covers the primitives; the *volume and cross-engine ordering* of synchronisation is higher. * **Pushes:** the graphics method vocabulary is much larger (pipeline state, shader binding, render targets, viewports), and state lives in big per-context image blocks. The current capture/decode path is compute-first; a graphics capture would need the same treatment the CUDA launch got (reference method stream, differential against the forwarded stream). * **Memory:** surfaces, tiled/swizzled layouts, render targets in VRAM; the GMMU and aperture handling are the same, but the emulated host-RAM VRAM becomes a real fidelity problem for tiling/swizzle expectations. * **Interaction with the current wall:** the descriptor-install wall is shared — a graphics draw also dispatches through GR descriptors. Ceiling note: the current docs mark OpenGL/Vulkan `not yet` (docs README status table); the 3D plan is to reuse the vEP pipeline once compute dispatch is proven, then add a graphics-capture workstream (mode-matrix discipline applies: one class/method family at a time). --- ## 12. Overarching framing: why the vEP is a core problem **What vGSP is.** vGSP is the virtual GPU System Processor: the host-side reimplementation of the GSP/GSP-RM interface that lets a stock guest driver believe it owns a real NVIDIA GPU. `vgsp.ko` is vGSP: the virtual device, the emulated boot and message transport, the RM object/control service, and the execution plane. The guest never learns it is virtualized for these paths, and no component in the stack — host driver, guest driver, firmware, or hypervisor — is modified (docs 01, 09 §13). **Why the execution plane is one of vGSP's core problems.** Everything else in vGSP is necessary but not sufficient: the control plane makes the driver run, but only the execution plane makes applications *work*. It is where the project's hardest constraints collide: * **Correctness without cooperation.** The guest does not help; the vEP must recover VA/PA truth, ring locations and completion semantics from observation, and satisfy contracts (completion, ordering) that hardware would otherwise enforce. * **Performance without emulation.** Compute cannot be emulated at useful speed; the plan of record is real execution on real engines, which means every semantic the emulation would have provided for free (memory visibility, ordering, faults) must be provided by the forwarding path. * **Isolation across tenants.** The same plane is where the multi-tenant promise is kept or broken: one physical GPU's engines shared by mdevs requires per-client state, scheduling and fault containment — the ISSUE-0002 class today, the fairness policy tomorrow. **Fit with HAAEEP and the ecosystem.** HAAEEP's premise is compute at the edge — local inference, full isolation, no cloud dependency, no license (`docs/08-ecosystem/haaeep-and-collaboration.md` §1/§3). That premise is only real if a guest can run its own CUDA workloads on hardware it owns, under vGSP, with isolation that a security stack can trust. The vEP is the subsystem that turns that premise from an interface emulation into execution; its progress metrics (reproducibility, arbitrary-kernel acceptance, near-native performance, per-tenant fairness) are the ecosystem's prerequisites, not internal milestones. When this document says "the wall", it is naming the last technical barrier between the project's architecture and its promise. --- ## See also * [`01-system-overview.md`](01-system-overview.md) — principles and layer map * [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) — GMMU, channels, GPFIFO, GR, QMD concepts * [`09-full-system-picture.md`](09-full-system-picture.md) — the end-to-end map * [`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md) — interaction, locks and scheduling detail * [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) — the physical session and the P0–P6 ladder * [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) — the GR launch chain and its leading edge * [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) — the execution gate * [`../../debug/issue/issue-number-1.md`](../../debug/issue/issue-number-1.md) and [`issue-number-2.md`](../../debug/issue/issue-number-2.md) — reproducibility and multi-kernel tracking * [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md) — the offline firmware debugger used for offline-first work * [`../../wiki/Execution-Plane.md`](../../wiki/Execution-Plane.md) — the condensed public wiki version of this document ## Change log | Date | Change | |---|---| | 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. | | 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (`svc_thread`, `svc_gr_request`, `svc_pending_add`, H3 guard, readiness gate); GR completion §3.4/§6.1 reconciled with FORENSICS-grv5 (wake-up fires; the counter publication is the leading reading); discovery cadence wording corrected (2-second paced passes, not "milliseconds"); mode-matrix source note corrected. |