path: root/docs/03-architecture/11-physical-backend-guest-wiring.md

# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session

> **Status: P0-P5(w1) implemented; P6a guest-page reachability (trigger
> residual fixed) and P6b waves
> 1/1.5/2 (guest-copy forwarding, deferred release, hardware-primary copies)
> proven live (2026-10-02); P6c C0/C1 (engine-authored guest CE semaphore
> relay) and C3 (forwarded-engine fault observation/relay) implemented
> default-off; H1 host GR ladder open.** The backend API skeleton
> (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that
> path is an open defect off the critical path), the static-config identity
> overlay (P2, verified live in an A/B run), the read-only control
> forwarding whitelist (P3, default off, verified live: ok=22/fallback=2,
> smoke `ALL OK` 64/64), the P5 wave-1 guest-submission shadow execution
> (default off, verified live: 514 real CE copies during a green guest smoke
> `ALL OK` 64/64), the P6b wave-1 guest-copy forwarding through
> module-built GMMU scratch mappings (default off, verified live: req=42
> ok=42 verify_ok=42, 503 ladder shadow copies also run), the wave-1.5
> deferred release (default off, verified live: engaged=75 applied=75,
> releases exactly once) and the wave-2 hardware-primary copies (default off,
> verified live: copy_skipped=119, queued req=100 ok=100 verify_ok=100
> verify_fail=0 fallback=0 timeouts=0, smoke `matched=64/64` `ALL OK`, with
> the software fallback for unqueued/failed copies) are in the tree. The P6c
> relay is live-verified default-off — the physical engine authored the
> guest CE copy's semaphore release (`relays=64`), failed/refused copies fall
> back to the software apply exactly once, and one relayed-copy data mismatch
> is under triage (mismatch logging plus software fallback in charge); the C3
> forwarded-engine fault relay is live green, default off. The P6a trigger
> residual is fixed: triggers 6/7 re-translate at trigger time and use fresh
> scratch lines.
> The physical backend itself is live: the module-owned
> GSP-RM completes initialization on the real TU104, serves the full
> static-config reply, and shuts down gracefully. The host resource-manager
> client ladder has since allocated memory, a virtual-address-space object with
> a committed page directory, a channel group and a real copy-engine channel,
> and has executed the **first real work on the physical engine** — a 4 KiB
> copy in virtual-address mode with the semaphore released and destination
> equal to source (cycle 25). The H1 host GR ladder then allocated a GR0
> channel with a compute-class object, but its first semaphore push faulted
> with an uncommitted context (`CTXNOTVALID`), and the follow-up
> `GPU_PROMOTE_CTX` attempt wedged GSP-RM; the GR execution wall is open and
> the device sits in the designed locked fail-safe. What this document
> specifies is the remaining
> bridge: wiring that live session into the guest-facing mdev/VFIO path so that
> guest submissions execute on the physical GPU. It extends
> [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which
> chose the module-owned-GPU mechanism, and
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md),
> which describes the emulation plane it must not disturb.
>
> The design was written against the tree at commit `f8e5d3b`; line numbers are
> advisory and have moved with later commits — anchor by function name when
> re-grepping. Behaviour claims below are grounded in live evidence from the
> 2026-10-01 and 2026-10-02 cycles; anything not yet exercised is labelled
> "not yet run".

---

## 0. Ground truth: the two planes that exist today

### 0.1 Guest-facing emulation plane (works, green)

* One emulation context per mdev: `nv_vgpu_device.gsp_comm_ctx`
  (`kernel/include/nvidia-vgpu-vfio.h:115`), created by
  `gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from
  `nv_vgpu_device_init_work()` (`kernel/main.c:1553`).
* The emulation thread (`gsp_emu_thread_fn`, `kernel/gsp_emulation.c:651`)
  does four things: samples the guest's BAR0 *shadow* falcon registers
  (`gsp_emu_poll_boot`, `:193`), polls the guest's sysmem msgq
  (`gsp_rm_emu_poll`, `kernel/gsp_rm_emulation.c:3386`), serves the legacy
  doorbell/shared-memory RPC class (`gsp_emu_dispatch_rpc`, `:458`), and
  injects MSI-X through the live eventfd (`gsp_emu_inject_completion_interrupt`,
  `:626`).
* The guest's real 570.144 physical-path protocol (msgq over *guest* system
  memory, vfio_dma_rw transport) is handled entirely in
  `gsp_rm_emulation.c`: queue-chain parse (`:806`), status-queue link
  (`:917`), command poll (`:3194`), per-function dispatch (`:3084`),
  reply construction/transmission (`gsp_rm_send`, `:452`).
* `GSP_INIT_DONE` is emitted from `gsp_rm_emu_on_booter_done()` (`:3339`)
  when the *emulated* Booter kick completes (`kernel/gsp_emulation.c:416-431`).
* Static configuration is a hand-built TU104 model:
  `gsp_rm_build_static_info()` (`:1025`) with 256 MB FB, fake
  `TU104_H_INTERNAL_{CLIENT,DEVICE,SUBDEVICE}` handles and hand-tuned engine
  caps; `gsp_rm_handle_get_static_info()` (`:1170`) sends it.
* Work is executed in software by the sema service: doorbell entry
  `gsp_sema_service_doorbell()` (`kernel/gsp_sema_service.c:6369`) called from
  the VFIO write path (`kernel/main.c:826`), pump
  `svc_pump_channels()` (`gsp_sema_service.c:5019`), decode
  `svc_decode_stream()` (`:2802`), release apply `svc_sem_write()` (`:1514`),
  guest event post `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`).

### 0.2 Physical backend plane (alive, queryable, not wired to the guest)

* Ownership: `gsp_phys_gpu_claim()` (`kernel/gsp_phys_gpu.c:662`) requires
  both `gsp_phys_gpu=1` (`kernel/main.c:2457`) and `gsp_phys_hazard_ack=1`
  (`kernel/gsp_phys_gpu.c:670`), locks the whole board + root port into D0
  (`gsp_phys_power_lock()`, `:523`; runtime-PM ref `:740`), unbinds the
  distribution driver (`:706`), binds `open-iov-gsp-phys` (`:718`) and verifies
  config-space liveness (`gsp_phys_board_check()`, `:591`).
* Boot: claim runs FWSEC (`:792`), staging `gsp_phys_boot_prime()`
  (`booter:1349`), Booter Load + RISC-V `gsp_phys_boot_run()` (`booter:1524`,
  pre-boot fn72/73 at `:1559-1568`), then init-done poll
  `gsp_phys_boot_rmwait()` (`booter:2764`, called `gsp_phys_gpu.c:811`).
  A full boot has run live multiple times; init-done with `rpc_result=0` and
  the full 1656-byte static-config reply were observed (cycle 13).
* Transport: `gsp_phys_queue_tx_submit()` (`booter:2935`) for TX,
  `gsp_phys_status_queue_*` (`booter:2467/2490/2554`) +
  `gsp_phys_status_wait()` (`booter:2670`) for RX. All real-GSP traffic is
  serialized by the single static mutex `gsp_phys_rm_lock`
  (`booter:3671`), taken by `gsp_phys_boot_rm_unload()` (`:3685`),
  `gsp_phys_rm_probe()` (`:4090`) and `gsp_phys_rmcall_run()` (`:5117`).
  `gsp_phys_queue_tx_submit()` and `gsp_phys_status_wait()` themselves take
  no lock and are not safe to call concurrently.
* Host RM-client ladder (debugfs only, `booter:5754` write handler):
  stage 1 root/device/subdevice (`:4575`), stage 2 memory registration
  (`:4650`), stage 3 TSG + channel variants (`:4761`; variant matrix `:4714`),
  stage 4 real CE submission (`:4923`), stage 5 adopt static-config handles
  (`:5068`). Stage 4 now runs end to end: the channel allocates, the page
  directory is committed, and a 4 KiB virtual-address-mode copy executes on
  the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25).
  `gsp_phys_rmcall_engine=1` selects a GR0 ladder variant (stage 3 builds the
  channel and a `TURING_COMPUTE_A` compute object with `engineType=1`, stage 4
  pushes a GR semaphore release). Its first live push faulted with an
  uncommitted context (`CTXNOTVALID`). `gsp_phys_rmcall_gr_promote=1..3`
  selects opt-in `GPU_PROMOTE_CTX` shapes to commit that context; the first
  attempt wedged GSP-RM (internal assert, no reply) and teardown could not
  clear WPR2, so the fail-safe held the device. Debugfs trigger 9 re-runs the
  GR semaphore push on an already-built GR ladder without touching WPR2, for
  use once a promote shape is viable.
* Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends
  `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2
  (`:911`), frees staging (`:914`), and only rebinds the distribution driver
  if the board answers (`:955`) and WPR2 is gone (`:939`); otherwise the D0
  locks intentionally stay held (fail-safe).

### 0.3 The gap

The two planes now meet at the lifecycle seam (P0/P1, mdev-owned
claim/release), the static-config identity overlay (P2), the read-only control
whitelist (P3) and, for eligible CE copies, the P6b scratch-mapped work path
(waves 1/1.5/2: guest operands, deferred exactly-once release,
hardware-primary copies) plus the P6c relay (C0/C1 engine-authored guest
release, C3 forwarded-engine fault observation/relay), all default off;
objects, FB geometry, engine caps and real guest page tables are still
emulated. The remaining bridge must be added at the reply point
`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the
work-plane seams (see §2).

### 0.4 The lifecycle at a glance (init -> work -> completion -> teardown)

One page for the whole physical-backed path, with the module's function names
for grepping. Status tags: `[live]` proven with the stock driver; `[gated]`
implemented but default-off, awaiting the ladder gate; `[wall]` the current
blocker; `[target]` the success latch that arms the guest work path.

```text
GUEST (QEMU)              HOST: vgsp.ko (module-owned GPU)           PHYSICAL TU104
────────────────────────────────────────────────────────────────────────────────────

PHASE I - INIT
 I0 preflight  nvidia bound + green | no vgsp/mdev/QEMU | pstore empty      [live]
 I1 claim      insmod: D0 lock (board + root port) -> unbind distribution
               driver -> bind open-iov-gsp-phys -> board check              [live]
 I2 boot       FWSEC -> Booter Load -> RISC-V active / WPR2 up (stage 3d)
               -> GSP-RM init event 0x1001 rpc_result=0 (stage 3e, ~950 ms) [live]
 I3 mdev       create -> nv_vgpu_device + gsp_emu_init(); backend start:
               "joined the live module-owned claim (READY)"                 [live]
 I4 ladder     debugfs gsp-phys-rm-call triggers 5/3/4:
               stage 1 root/device/subdevice + stage 5 adopt                [live]
               stage 3 GR ctx prep -> channel -> GPU_PROMOTE_CTX            [wall]
               stage 4 BIND -> GPFIFO_SCHEDULE -> TSG schedule -> token
               -> push; latch "REAL GR SEM OK" => rmcall_gr_exec_ok = 1     [target]
 I5 guest      QEMU with mdev -> nvidia.ko probes; emulation plane serves
               boot handshake, static-config overlay, RM battery, UVM
               registration -> CUDA ready                                   [live]
 I6 M1 prep    guest RAM registered (gsp-phys-guest-ram); watchers
               (pin_gr_stream, semfix_uvmt)                                 [gated]

PHASE II - RUNNING WORK (native GR path; gsp_phys_gr_forward=1,
            gsp_phys_guest_mem=1, cuda_bridge=0)
 guest  CUDA app -> libcuda -> nvidia.ko -> GR channel GPFIFO push -> BAR doorbell
 host   gsp_sema_service_doorbell() -> svc_pump_channels() -> svc_decode_stream()
        -> svc_qmd_accum() -> svc_gr_qmd_decode() / svc_gr_cand_capture()
        -> svc_gr_request() / svc_gr_req_fill() -> svc_fwd_gr_work()
        -> gate gsp_phys_backend_gr_ready()          [needs rmcall_gr_exec_ok]
        -> gsp_phys_backend_guest_gr_submit()
             Tier 1: guest VA -> PA -> HPA, fresh 128 B PT at the guest VA,
                     DMA_UPDATE_PDE_2 per 2 MiB window
             Tier 2: captured inline bytes -> shadow page (else refused)
        -> gsp_phys_rmcall_build_gr_launch()
        -> gsp_phys_rmcall_publish_locked() -> ladder channel GPFIFO + doorbell
 GPU   real GSP scheduler -> GR engine executes the QMD
 (scaffolding, not the destination: cuda_bridge replays the kernel on host CUDA)

PHASE III - COMPLETION
 GPU   engine authors the guest-visible QMD release word (5 -> 6 in the
       acceptance program) and the channel release
 host  poll (200 ms) -> "GUEST GR LAUNCH OK ... qmd_rel=0x00000006"
       -> guest release/event: gsp_rm_post_channel_event() -> MSI-X injection
       -> counters work_gr: req/ok/fallback/refused/timeout/fault
 guest semaphore observed -> CUDA continues -> verified output
 fault recType=4 faultingEngine="GR" -> fwd_gr_fault; releases applied exactly
       once, failed launch never retried

PHASE IV - TEARDOWN
 T1 drain  mdev remove -> QEMU stop -> watchers stopped -> rmmod vgsp
 T2 release gsp_phys_gpu_release(): UNLOADING_GUEST_DRIVER -> quiesce ->
           WPR2 down -> free staging -> rebind distribution driver (only if
           the board answers and WPR2 is gone) -> release D0 locks           [live]
 T3 verify rmmod rc=0 | "rebound to nvidia (rc=1)" | WPR2 down | 5 power
           domains locked | pstore empty | nvidia-smi green | 0 Xid          [live]
 FAIL      RmInitAdapter failed (no Xid) -> PCI function-level reset;
           WPR2-locked wedge -> stop; reboot only per the reboot-approval
           policy (written reason + post-reboot plan first)
```

The single open step on this path is I4's stage-3 GR context commit: the
server must accept a real engine-context construct (the reference client's
channel shape plus its atomic promote entry set) before stage 4 can latch
`REAL GR SEM OK`. Everything from I4-stage-4 onward is implemented and
fail-closed; until the latch is set, every forward refuses with
`exec_ok=0` and the guest work path behaves exactly as it does today
(see §1.1 and the M1 gate in §5 P6).

---

## 1. Target architecture: runtime state machine

### 1.1 Backend session state (module-global; one physical GPU)

```
        BACKEND_OFF
             |  first mdev create (gsp_phys_gpu=1 + hazard_ack; no other mdev)
             v
        BACKEND_CLAIMING      power-lock D0 -> unbind dist -> bind phys -> board check
             |  claim ok                            \  abort signature / lock fail
             v                                        v
        BACKEND_BOOTING       FWSEC -> prime -> Booter -> RISC-V -> rmwait (init-done)
             |  init-done rpc_result=0                \ init failure / timeout
             v                                         v
        BACKEND_READY         static-config snapshot cached; RPC/control forwarding legal
             |  last mdev remove | rmmod | board signature
             v
        BACKEND_STOPPING      stop emulation forwarding; UNLOADING_GUEST_DRIVER;
             |                quiesce; WPR2 teardown; staging free
             v
        BACKEND_OFF           power unlock only when gsp_phys_repower=1 (default
                              0 latches the D0/D3cold locks); distribution driver
                              rebound and settled
```

`BACKEND_READY` is the single gate for every live answer. In every other
state the guest-facing handlers must behave exactly as they do today.

### 1.2 Ownership: who starts/stops what

| Trigger | Action | Where |
|---|---|---|
| first `mdev` create | backend refcount++ ; if 0 -> claim + boot asynchronously from `init_work` | `kernel/main.c:535`, `:1553`; new `gsp_phys_backend_start()` |
| last `mdev` remove | backend refcount-- ; if 0 -> `gsp_phys_gpu_release()` after the emulation thread is stopped | `kernel/main.c:614`, `:1614`; new `gsp_phys_backend_stop()` |
| `rmmod` with live mdevs | forbidden by policy, but code path must drain: STOPPING first, then stop emu threads, then release | `kernel/main.c:1487` (`nv_vgpu_exit`) |
| load-time `gsp_phys_gpu=1` | keep today's supervised behavior: claim (and full boot) at module init, owned by the module, released only at `rmmod` (debugfs harness unchanged) | `kernel/main.c:1450` |
| QEMU close/open only | no backend action; the session persists across VM restarts until mdev remove | `nv_vgpu_vfio_dev_close`, `kernel/main.c:698` |
| power-loss signature anywhere | STOPPING immediately (no new forwards); release fail-safe keeps D0 locks; no retry in this boot | `gsp_phys_board_check`, `gsp_phys_gpu.c:591` |

The module-owned claim (load-time `gsp_phys_gpu=1`) and the mdev-owned claim
share the same `phys` singleton. `gsp_phys_gpu_claim()` is idempotent today
(`gsp_phys_gpu.c:667` returns 0 if already claimed); the refcount/ownership
flag is new state and decides **who** releases.

### 1.3 Locks and ownership

| Lock | Protects | Taken by | Rule |
|---|---|---|---|
| `gsp_phys_session_lock` (new) | backend state, refcount, owner | `backend_start/stop/state`; `nv_vgpu_exit` | outermost; never held while waiting on a guest lock; may be held while calling `rm_lock` |
| `gsp_phys_rm_lock` (existing, `booter:3671`) | all real-GSP TX/RX: queue cursors, `boot_stage.status_read_ptr`, `tx_seq_num`, probe/ladder/unload | `gsp_phys_boot_rm_unload`, `gsp_phys_rm_probe`, `gsp_phys_rmcall_run`, new `gsp_phys_backend_rpc()` | one real RPC at a time, always; **never** taken while holding `rm->stat_lock` or any guest-memory path |
| `rm->stat_lock` (`gsp_rm_emulation.c:186`) | guest status-queue writes | `gsp_rm_send` (`:452`) | build the full reply before taking it; release immediately after write; never hold across a backend wait |
| `vgpu->lock` (`nvidia-vgpu-vfio.h:99`) | per-mdev state transitions | `init_work`/`cleanup_work` | never held across claim/release or RPC waits |
| `nv_vgpu_driver.lock` | device list | create/remove, sema thread scan | never taken under session/rm locks |
| `svc.chan_lock` / `svc.vas_lock` | software work pipeline | sema service | unchanged; forwarding reads channel state copied out under these locks |

Lock order: `nv_vgpu_driver.lock` (shortest, leaf) < `gsp_phys_session_lock`
< `gsp_phys_rm_lock`. The emulation thread must take
`session_lock -> check state -> drop session_lock -> rm_lock -> re-check a
generation counter` so a release cannot deadlock against a forward in flight.

### 1.4 Failure paths

* **Guest crash / QEMU killed:** `open_count` drops; the backend stays READY
  and claimed (device owned, host headless). No device writes. The next
  `mdev` remove or `rmmod` drives the graceful path. This is the same
  `--keep` semantics the supervised harness already uses.
* **Guest RPC while STOPPING/unload already submitted:** every forwarding
  seam returns "not ready" and the emulation answer is used. This mirrors the
  defensive re-checks in `gsp_phys_rmcall_run()` (`booter:5119`).
* **Module unload order (must change):** today `nv_vgpu_exit()` calls
  `gsp_phys_gpu_release()` at `kernel/main.c:1500` *before*
  `gsp_sema_service_stop()` (`:1525`) and before mdev unregistration
  (`:1530`) which is what stops the emulation threads (`:1619`). Once
  handlers forward to the backend, release must not race live forwards:
  new order = set STOPPING (session lock) -> `gsp_sema_service_stop()` ->
  `mdev_unregister_driver()` (drains emulation) -> `gsp_phys_gpu_release()` ->
  parent/class teardown.
* **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out,
  the existing release still quiesces, runs the FWSEC teardown and refuses to
  rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then follows the
  project reboot-approval policy (a written reason and post-reboot plan before
  approval).
* **Static-config served before READY:** bounded wait (2 s, implemented) then
  pure emulation fallback, with a stat counter; never block the emulation
  thread longer than the guest's own RPC poll can tolerate.

---

## 2. Seam table

Legend: **L** = live backend, **E** = existing emulation, **H** = hybrid.
"Change point" is the exact function that must learn the new behavior.

### 2.1 Guest GSP-RM RPC class (msgq function IDs)

| # | Guest message | Today | Target | Change point (file:line) |
|---|---|---|---|---|
| 1 | `GSP_SET_SYSTEM_INFO` (72) | consumed, no reply (`gsp_rm_emulation.c:3095`) | **E** - describes the guest client at guest boot; the backend already sent its own fn72 at `booter:1559` | `gsp_rm_handle_element` `gsp_rm_emulation.c:3095` |
| 2 | `SET_REGISTRY` (73) | consumed, no reply (`:3096`) | **E** - same reason | `:3096` |
| 3 | `CONTINUATION_RECORD` (71) | SUCCESS reply (`:3105`) | **E** | `:3105` |
| 4 | `SET_GUEST_SYSTEM_INFO` (1) | echoes guest vgx version (`:982`) | **E** - this is the guest's negotiation, deliberately decoupled from the host session (see §4.3) | `gsp_rm_handle_set_guest_system_info` `:982` |
| 5 | `GET_GSP_STATIC_INFO` (65) | hard-coded TU104 model (`:1025`), 256 MB FB, fake handles, hand-tuned caps | **H** - overlay live identity/capability fields proven safe; keep model-consistent FB geometry, PDE bases and handles until phases P4/P6 (see §4) | `gsp_rm_build_static_info` `:1025`, `gsp_rm_handle_get_static_info` `:1170` |
| 6 | `GSP_RM_ALLOC` (103) | echo request + `NV_OK`; channels/events registered locally (`:2322`) | **H** - echo for emulation-only classes; forward selected classes (root/device/subdevice, then channel/memory) once a guest<->server handle map exists; always keep local side effects | `gsp_rm_handle_rm_alloc` `:2322` |
| 7 | `GSP_RM_CONTROL` (76) | ~25 static per-cmd handlers; unknown -> zeroed success (`:2611`, default `:2807`) | **H** - whitelist of side-effect-free queries goes live with emulation fallback; vGPU-specific controls stay **E** (`COPY_SERVER_RESERVED_PDES` `:2647`, `PROMOTE_FAULT_METHOD_BUFFERS` `:2654`) | `gsp_rm_handle_rm_control` `:2611`; new forward helper before the switch |
| 8 | `GSP_RM_FREE` (10) | echo + retire channel/events (`:2882`) | **H** - forward when the object was backend-allocated; always retire local bookkeeping | `gsp_rm_handle_free` `:2882` |
| 9 | `SET_PAGE_DIRECTORY` (54) | recorded for translation + SUCCESS (`:2975`) | **H** - recording stays **E**; forwarding to the real MMU is P6c and guest pages are GPU-DMA-reachable on the reference host (P6a status below; page-table pre-build required). P6b waves 1/1.5/2 execute eligible copies without forwarding page-directory state, through module-built scratch page tables, with deferred exactly-once release and hardware-authored data | `gsp_rm_handle_set_page_directory` `:2975` |
| 10 | `UNSET_PAGE_DIRECTORY` (79) | record remove + SUCCESS (`:3063`) | **H** - same gate as #9 (real page-directory forwarding is P6c; reachability resolved, pre-build required) | `:3063` |
| 11 | `UPDATE_BAR_PDE` (70) | SUCCESS ack (`:3144`) | **E** now; **L** only when BAR1 is backed by real VRAM | `:3144` |
| 12 | `UNLOADING_GUEST_DRIVER` (47) | SUCCESS ack (`:3155`) | **E** - the guest's own shutdown; the real session's shutdown is host-driven on backend stop (`gsp_phys_gpu.c:909`) | `:3155` |
| 13 | default/unknown | generic SUCCESS (`:3169`) | **E** (fail-safe: never invent a live forward for an unknown function) | `:3169` |

### 2.2 Outbound events and replies

| # | Record | Today | Target | Change point |
|---|---|---|---|---|
| 14 | `GSP_INIT_DONE` (0x1001) | emitted as soon as the emulated Booter finishes (`gsp_rm_emulation.c:3374`) | **H** - emit immediately as today, but live answers gate themselves on READY; alternatively delay emission on a bounded backend wait so the guest's first RM RPC sees READY | `gsp_rm_emu_on_booter_done` `:3339` |
| 15 | `POST_EVENT` (0x1003) | software pump completion (`gsp_rm_post_channel_event` `:599`) | **E** as the relay mechanism; its *trigger* becomes the observation of a real hardware semaphore (P5) instead of a software op | `gsp_rm_post_channel_event` `:599`, `svc_notify_complete` `gsp_sema_service.c:4151` |
| 16 | all replies | single writer `gsp_rm_send` (`:452`) | **E** - keep as the only guest-queue writer; live payloads are copied into it | `gsp_rm_send` `:452` |

### 2.3 BAR / register and work-plane classes

| # | Guest-visible thing | Today | Target | Change point |
|---|---|---|---|---|
| 17 | GSP falcon boot regs (FWSEC/Booter/RISCV/WPR2), SW interrupt tree, MMU invalidate, L2 flushes | emulated auto-complete (`gsp_emulation.c:193-452`) | **E always** - the two GSP sessions are intentionally independent; guest boot must never depend on the physical session | `gsp_emu_poll_boot` `:193` |
| 18 | Guest VF doorbell `0x88704` | software pump (`main.c:826` -> `gsp_sema_service_doorbell` `:6369`) | **H** - pump classifies each push: pure state/release stays software; eligible CE copies are forwarded to hardware (P5 host-buffer mode; P6b waves 1/1.5/2 guest operands, deferred release, hardware-primary copies with a software fallback) | `svc_pump_channels` `gsp_sema_service.c:5019`, `svc_decode_stream` `:2802` |
| 19 | BAR0/BAR1/BAR3 shadows, PRAMIN window | host-RAM shadows (`bar_emulation.c:180`, `main.c:1205`) | **E** until real VRAM forwarding (P6) | `nv_vgpu_bar_emul_*`, `nv_vgpu_vfio_dev_mmap` |
| 20 | legacy doorbell/shared-mem RPC class (`GSP_RPC_*`) | internal emulation client thread (`gsp_emulation.c:458-620`, `:674-694`) | **E / disable when READY** - not part of the guest physical-path protocol; gate it off in backend mode to avoid intercepting stray writes | `gsp_emu_thread_fn` `:674` |

**Highest-leverage seams:** #5/#7 (one reply point, one dispatch point; all
live answers plug in there) and #6 (whatever alloc/free semantics the guest
gets determine whether real channels/page tables are possible).

---

## 3. Lifecycle changes in `main.c` / `gsp_emulation.c`

1. **New backend API** (add to `gsp_phys_booter.[ch]`, used by both debugfs
   and the guest plane; keeps `gsp_phys_rm_lock` private):
   * `int gsp_phys_backend_state(void)` - OFF/CLAIMING/BOOTING/READY/STOPPING.
   * `int gsp_phys_backend_rpc(u32 function, const void *req, u32 req_len,
     void **reply, u32 *reply_len, u32 timeout_ms)` - checks
     `claimed && rm_init_done && !rm_unload_submitted`, takes
     `gsp_phys_rm_lock`, calls `gsp_phys_queue_tx_submit()` +
     `gsp_phys_status_wait()` (both at `booter:2935`/`:2670`), returns a copy
     of the payload; caller frees. Unmatched records (events, NOCAT) are
     relayed or summarized like `gsp_phys_status_wait` already does.
   * `int gsp_phys_backend_static_snapshot(u8 *out, u32 len)` - cached full
     1656-byte fn65 payload; filled once at READY by the existing
     `gsp_phys_rm_probe_cycle()` (`booter:4056`) under the same lock.
   * `void gsp_phys_backend_begin_stop(void)` / `gsp_phys_backend_end_stop()` -
     state transitions only (no hardware).
2. **New session state + refcount** in `gsp_phys_gpu.c` (new mutex, owner
   flag, state enum). `gsp_phys_gpu_claim()`/`_release()` stay the only
   hardware entry points; add `gsp_phys_backend_start()/stop()` that call
   them exactly once.
3. **`nv_vgpu_mdev_create()` (`main.c:535`)**: after
   `vfio_register_emulated_iommu_dev()` (`:591`) and before/around
   `schedule_work(&vgpu->init_work)` (`:598`), call
   `gsp_phys_backend_start()`. Claiming must happen while no VM holds a live
   mdev: with `available_instances` computed from per-type max (1) in
   `nv_vgpu_get_available()` (`main.c:81`) there is at most one mdev of the
   type in flight, so this holds.
4. **`nv_vgpu_device_init_work()` (`main.c:1553`)**: keep BAR/IRQ/memory/
   `gsp_emu_init()` order. The backend boot runs in its own kworker so
   `mdev` create returns quickly; `init_work` only observes READY when it
   initializes forwarding tables. Do **not** run the 60 s `rmwait`
   (`booter:2764`, timeout `GSP_PHYS_RM_INIT_TIMEOUT_MS` 60000 at
   `booter:446`) inline in `init_work` if it can be avoided: run claim+boot
   in a dedicated work item so `mdev->remove` is never stuck behind it.
5. **`nv_vgpu_device_cleanup_work()` (`main.c:1614`)**: unchanged per-mdev,
   but the last-cleanup path calls `gsp_phys_backend_stop()` **after**
   `gsp_emu_cleanup()` (`:1619`) so no emulation thread can forward while the
   backend is being torn down.
6. **`nv_vgpu_exit()` (`main.c:1487`)**: reorder as in §1.4; the load-time
   claim path (owner = module) keeps the existing `gsp_phys_gpu_release()`
   position relative to debugfs fini, but mdev-owned sessions are released in
   the new order.
7. **`gsp_emu_init()` (`gsp_emulation.c:724`)**: no structural change; the
   `gsp_rm_emu` context gains a reference to the module backend (not a new
   per-mdev session). `gsp_emu_cleanup()` (`:831`) unchanged except it must
   not be the thing that releases the backend (done by `main.c`, step 5).
8. **`gsp_rm_emu_on_booter_done()` (`gsp_rm_emulation.c:3339`)**: after
   linking the queues, optionally wait (bounded) for READY before emitting
   INIT_DONE, or emit immediately and let each live handler wait. Recommended:
   emit immediately (the guest's early RPCs are all emulatable), and let
   `GET_GSP_STATIC_INFO` be the single bounded-wait point.
9. **`gsp_emu_thread_fn()` (`gsp_emulation.c:651`)**: no new blocking calls in
   the loop; live forwards happen inside `gsp_rm_handle_element()` on that
   thread, with the 2 s per-RPC bound. If a longer wait is ever needed, move
   forwarding to a dedicated backend kthread and keep this loop responsive.

---

## 4. Static-config flow and identity/version negotiation

### 4.1 Current guest model (must stay internally consistent)

`gsp_rm_build_static_info()` (`gsp_rm_emulation.c:1025`) hard-codes:
`fb_length = 0x10000000` (256 MB, offset 1224), `fbio_mask=0xff`,
`fbp_mask=0x3f`, `gpuNameString="NVIDIA GeForce RTX 2080 SUPER"` (1260),
`gpuShortNameString="TU104"` (1324), `bar1PdeBase=bar2PdeBase=0`
(1536/1544), fake internal handles (1600/1604/1608), engine caps
`0x60803E02 / 0x00362020 / 0` (1204). The 256 MB figure is coupled to the
BAR0 shadow's `LOCAL_MEMORY_RANGE` (`bar_emulation.c:280`,
`0x100ce0=0x18` -> 256 MB) and to the emulated WPR2/FRTS location the guest
driver checks. Changing it without changing the shadow breaks the guest boot
(this is why 512 MB was reverted, per the comment at `:1096`).

### 4.2 Live values observed (cycle 13)

```
static-info internal: hClient=0xc2000006 hDevice=0xabcd0080 hSubdevice=0xabcd2080
static-info fb: bar1PdeBase=0x1f7650000 bar2PdeBase=0x1f76c2000
               fb_length=0x200000000 fbio_mask=0xf fbp_mask=0x11
               busWidth=0 ramType=256 l2CacheSize=15
static-info gpu: name='NVIDIA GeForce RTX 2080 Super with Max-Q Design'
                short='TU104-B' vbiosSubVendor=0x4d50 vbiosSubDevice=0x4449
static-info engineCaps=[0x08083e02,0x00000044,0x00000000]
```

Two grounded warnings:

* The decoded `busWidth=0`, `ramType=256` are not plausible TU104 values
  (256-bit, GDDR6). Either the decode offsets for those two fields do not
  match the real struct, or the emulated values were hand-derived rather than
  observed. **A/B every field before overlaying it.**
* Engine caps differ materially (`0x60803E02` vs `0x08083E02`, and the second
  word). The emulation deliberately sets extra bits (COPY14 etc.) to make
  `libcuda` accept the engine inventory. Overlaying live caps would change
  the guest's engine/class topology and can regress the whole CUDA init.
  Keep emulated caps until the backend can actually service the resulting
  demands (P4+).

**P2 status (2026-10-02):** the name, short-name and VBIOS identity fields are
now overlaid live behind `gsp_phys_static_live` (A/B verified); the FB
geometry, masks and engine caps above remain emulated, per these warnings,
until P4/P6.

### 4.3 Proposed flow

1. At READY, cache the full fn65 payload (new bytes in the backend session,
   keyed by the loaded firmware's `phys.fw_crc32`, `gsp_phys_gpu.c:272`), via
   the existing probe cycle (`booter:4056`) under `gsp_phys_rm_lock`.
2. `gsp_rm_handle_get_static_info()` (`:1170`) keeps building the emulated
   1656-byte model, then applies an allow-listed overlay from the snapshot
   (start: name/short-name/VBIOS identity/L2 if A/B passes; **not** FB
   geometry, masks, PDE bases, handles, engine caps).
3. Any overlay field is independently switchable (`gsp_phys_static_live`
   module parameter, default off) so a regression is a one-line rollback.
4. Handles move in P4 and PDE bases/geometry in P6, each behind its own
   switch and its own A/B.
5. Version negotiation stays decoupled: the guest sees the emulation's
   version answer (`SET_GUEST_SYSTEM_INFO`, `gsp_rm_emulation.c:982`); the
   real server is negotiated by the host as the host client
   (`gsp_phys_rm_probe_fn1`, `booter:3984`, vgx 0x29/0x0C). The guest must
   never learn the physical firmware's version; the emulation target remains
   570.144 while the host may boot a different, matching TU10x GSP image
   (`gsp_phys_gpu.c:75`). The wire structures are the same family and the
   live snapshot is keyed by firmware CRC.

---

## 5. Phased implementation plan (smallest first)

Each phase is independently revertible. "Supervised" means the documented
harness discipline: `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`, human
go-ahead, recovery plan, pstore, never unattended
(`docs/03-architecture/08-physical-gpu-integration.md` §7/§8).

### P0 - Backend API skeleton (inert)
* **Status (2026-10-02): implemented.** Session state/refcount plus the locked
  backend RPC and static-snapshot helpers are in the tree and refuse when the
  backend is OFF/not-init-done/unloading.
* **Scope:** new state/refcount/lock in `gsp_phys_gpu.c`; locked
  `backend_rpc()`/`static_snapshot()` in `gsp_phys_booter.c`; no caller
  changes; no behavior change.
* **Acceptance:** `make kernel` clean; load/unload with no params; all
  existing debugfs triggers behave identically; new API logs and refuses when
  OFF/not-init-done/unloading.
* **Risk:** none (never sends unless a live init-done session exists).
* **Rollback:** trivial.

### P1 - Lifecycle wiring (claim on mdev create, release on last remove)
* **Status (2026-10-02): implemented and exercised live; fail-open.** The
  first mdev create starts the claim/boot asynchronously and the last remove
  releases it. The async boot currently fails at FWSEC on this path
  (`staging not visible before kick`) and the READY gate releases the claim
  fail-open; the module-owned live session is the proven path (and what P2
  uses).
* **Scope:** `main.c` create/remove/init_work/cleanup_work/exit ordering
  (§3 items 3-6); no live answers yet.
* **Acceptance (supervised, harness `phys_claim_cycle.sh` style):** fresh
  boot -> insmod params -> create mdev -> one real boot log sequence (FWSEC,
  Booter M0=0, RISC-V active, init-done after sequencer) -> guest boots and
  the CUDA driver-API smoke test still prints `ALL OK` 64/64 -> stop VM ->
  remove mdev -> `UNLOADING_GUEST_DRIVER` suspend, scratch14 bits 24-27 DONE,
  WPR2 down, distribution driver rebound; repeat twice; zero power-domain
  cycles; no reboot.
* **Risk:** **highest phase.** Claim is now triggered by a sysfs action, not
  by module load. The two-param gate and the D0 lock must already be active
  inside claim (they are, `gsp_phys_gpu.c:670/700`), the claim must refuse a
  second owner, and the release must never run while a forward is in flight
  (STOPPING gate). Any power-loss signature -> hold locks, no retry.
* **Rollback:** `gsp_phys_gpu=0` (no claim) or load-time `gsp_phys_gpu=1`
  only; `mdev` emits `-EOPNOTSUPP` from `backend_start()`.

### P2 - Static-config identity overlay
* **Status (2026-10-02): implemented and verified live (A/B).** The READY-time
  snapshot cache and the `gsp_phys_static_live` parameter (bool, default 0)
  are in the tree. With the overlay on, the guest `nvidia-smi -q` Product
  Name is the live `NVIDIA GeForce RTX 2080 Super with Max-Q Design`; with it
  off it is the emulated `NVIDIA GeForce RTX 2080 SUPER`; the driver-API
  smoke stays green (`ALL OK`, payload 64/64). Only name, short name and
  VBIOS identity are overlaid; FB geometry, masks, engine caps, PDE bases and
  handles stay emulated.
* **Scope:** cache snapshot at READY; overlay allow-list in
  `gsp_rm_build_static_info()`/`gsp_rm_handle_get_static_info()`; module
  parameter; stats. Bounded wait (2 s) then emulation fallback.
* **Acceptance:** guest still `nvidia-smi -L` + smoke `ALL OK`; guest
  `nvidia-smi -q` name changes to the live `... with Max-Q Design`; dmesg
  shows which fields came live; backend absent -> byte-identical guest reply.
* **Risk:** a consumed identity field changes a driver decision (VBIOS
  validation is the likeliest). Mitigation: field-by-field A/B, default-off,
  instant fallback; never touch geometry/caps/handles in P2.
* **Rollback:** parameter off.

### P3 - Read-only control forwarding (whitelist)
* **Status (2026-10-02): implemented and verified live (default off).** With
  `gsp_phys_ctrl_forward=1` the whitelist (`BUS_GET_PCI_BAR_INFO`,
  `BUS_GET_INFO_V2`, `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) is
  forwarded over fn76 with the guest's emulated internal handles translated
  to the snapshot's adopted server handles; the server's echoed params become
  the guest reply, and everything else falls back to the emulated handler
  (fail-open; counters forwarded/fallback/timeout). Live acceptance: guest
  smoke GREEN `ALL OK` 64/64, counters ok=22 fallback=2 timeout=0, guest
  `nvidia-smi -q` worked.
* **Scope:** in `gsp_rm_handle_rm_control()` (`:2611`), before the per-cmd
  switch: for an allow-list of side-effect-free queries (FB/BUS/KGR/GMMU/
  CE/display statics), call `backend_rpc()` with the **adopted internal
  handles** from the snapshot (the guest's emulated handles are not valid on
  the server yet), copy reply params (payload+24) on `rpc_result==0 &&
  status==0`, else fall back to the existing handler. Add per-cmd
  forwarded/fallback counters.
* **Acceptance:** guest boot and smoke green with forwarding on; dmesg shows
  N forwarded controls; forced backend timeout -> same behavior as today;
  repeat calls yield identical params (no server-side state change).
* **Risk:** a whitelist entry is not actually read-only (display/KGR can
  allocate on first call). Start with FB/BUS/GMMU static info only, verify
  against the server's log ring (`gsp-phys-log-dump`), then grow.
* **Rollback:** parameter/whitelist off.

### P4 - Alloc/free forwarding for the object ladder
* **Scope:** `gsp_rm_handle_rm_alloc()` (`:2322`) forwards
  root/device/subdevice (and only those, first) via `backend_rpc(fn103)`;
  returns the server's chosen handle to the guest; maintains a
  guest-handle -> server-handle map used by P3 and `gsp_rm_handle_free()`.
  Any failure falls back to today's echo (never half-forward), and local
  channel/event registration still runs.
* **Acceptance:** guest boot + smoke green; dmesg shows adopted handles
  (`0xc2.../0xabcd0080/0xabcd2080`) matching cycle13; guest FREE returns
  success; no server-side wedge (no RPC timeout after alloc).
* **Risk:** the guest's alloc params may not satisfy the server's per-class
  validation; a bad forward can wedge the RM task (see the `0x40` GSP-owned
  channel incident). Guard: one class at a time, abort on first failure,
  `UNLOADING_GUEST_DRIVER` + rmmod remains the only fail-safe; never retry a
  mutating RPC.
* **Rollback:** parameter off (guest returns to echo path).

### P5 - Work forwarding + completion relay (interim, host buffers)
* **Status (2026-10-02): wave 1 implemented and verified live (default off).**
  With `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE
  pushes on the physical engine through one ordered work item running the
  module-owned ladder copy; the software release path stays authoritative and
  no guest memory/VA is touched. Live: guest smoke GREEN `ALL OK` 64/64 with
  514 real CE copies (194 requests coalesced/dropped while one was in flight,
  0 refusals, 0 timeouts). The P6b waves 1/1.5/2 (guest-copy forwarding,
  deferred exactly-once release, hardware-primary copies with a software
  fallback) have since landed (P6 status below), and P5 wave 2 (hardware-
  primary release relay) is subsumed by the P6b wave-2 hardware-primary mode
  and the P6c engine-authored relay; the real guest-page-table path and GR
  execution remain.
* **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25:
  channel allocates, page directory commits, `REAL COPY OK` on the physical
  CE in VA mode). The remaining work: bind each guest CE/GR channel to a real
  channel; in `svc_pump_channels()` (`gsp_sema_service.c:5019`) classify the
  decoded push and, for forwarded ops, submit an equivalent real method block
  built by the existing `gsp_phys_rmcall_build_methods()` (`booter:4879`) +
  GPFIFO/USERD/doorbell (`:4991-5032`). Observe the real semaphore, then
  mirror the release into guest memory with `svc_sem_write()`
  (`gsp_sema_service.c:1514`) and post the guest event with
  `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`). All hardware
  operands are module-owned DMA buffers (as in the ladder), so no guest memory
  is touched by the engine yet.
* **Acceptance:** ladder: `echo 4 > gsp-phys-rm-call` logs
  `REAL COPY OK` (`booter:5052`) and src==dst after a hardware copy - met
  (cycle 25, 4 KiB, semaphore released, dst==src). Guest: a submitted copy
  completes with the semaphore observed in host memory and the guest tracker
  advancing; wrong-operand/refused submissions fall back to software and are
  counted.
* **Risk:** doorbell to a wedged channel; wrong GPFIFO/USERD (guards:
  `rmcall_channel_server_mem` refusal `:4944`, work-submit token after
  `GPFIFO_SCHEDULE` `:4977`, `GP_PUT` bounds `:4995`); hardware writes only
  host DMA memory, so no cross-tenant exposure yet.
* **Rollback:** per-channel `forward=0`; software pump is untouched.

### P6 - Guest memory into the real MMU (end state: guest submission on hardware)
* **Status (2026-10-02): P6a guest-page reachability and P6b waves 1/1.5/2
  (guest-copy forwarding, deferred release, hardware-primary copies) proven
  live.** P6a: the physical GPU's
  IOMMU group is an identity domain on the reference host and the mediated
  device registers as an emulated-IOMMU VFIO device, so its type-1 container
  keeps guest DMA mappings privately and no kernel `iommu_domain` exists for
  it: `iommu_iova_to_phys` cannot translate guest GPAs. Guest RAM is therefore
  backed by a QEMU shared file (`memory-backend-file,share=yes,prealloc=on`);
  the module opens it and translates GPA->HPA through the file page cache
  (`read_cache_page` + `page_to_phys` + `pfn_valid`) behind the active-mdev
  gate, under the default-off `gsp_phys_guest_mem` parameter and write-only
  `gsp-phys-guest-ram`/`gsp-phys-guest-gpa` debugfs controls (ladder trigger
  cases 6 host->guest write, 7 guest->host read, 8 fresh-slot PTE self-test).
  On the reference q35 4 GiB guest, GPA equals file offset below 4 GiB and
  offset = GPA - 0x80000000 above it (2 GiB 32-bit PCI hole). **Pre-build
  requirement:** channel page-table slots for the guest window are built in
  ladder stage 3 before any engine access; a runtime PTE write into a line
  the GMMU had already walked was accepted but never completed (no
  TLB-invalidate path yet). **Live:** the physical copy engine copied 4 KiB
  from a real guest RAM page into a module buffer (G2H) and back into the
  same page (W2G), semaphore released; the guest process observed the write
  in its own page; the driver-API smoke stayed green (`ALL OK` 64/64); the
  informational PTE self-test passed. **P6b wave 1:** with
  `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service captures
  the first eligible copy per push from the software path's own resolved
  operands (the real push shapes: a physical emulated-VRAM destination plus a
  virtual source and vice versa, over a mix of sysmem guest RAM and
  emulated-VRAM shadow pages), resolves each side to a host physical address
  (guest RAM through the shared-file page cache; shadow pages through the
  vmalloc-to-physical mapping of the emulated VRAM shadow), maps both onto
  fresh module-built GMMU scratch slots and submits a real copy-engine copy
  on the module-owned channel; any miss falls back to the existing
  ladder-buffer shadow copy and the software pump stays authoritative. **PTE
  fetch-line rule (learned live):** the engine caches PTE fetches at 128-byte
  granularity and there is no invalidate path, so a scratch copy pair
  consumes one whole 128-byte line (16 PTEs) from a pool of 16 zeroed
  page-table pages built before the first engine access: capacity 512 pairs
  per boot, and a line is never reused within a boot. **Live:** `req=42 ok=42
  verify_ok=42 verify_fail=0 fallback=0 resolve_fail=0 pool_exhausted=0
  refused=0`, with 503 ladder shadow copies also run; the guest smoke stayed
  green (`ALL OK` 64/64) and release stayed latched (distribution driver
  rebound, protected region down, exactly 5 power-domain locks intentionally
  held, 0 rail cycles, no reboot). Shadow-mode limits: the software copy
  already applied the same bytes, so the hard evidence is acceptance of the
  translated-HPA submission, the semaphore release and host-side
  data-consistency verification (`verify_ok`); a late hardware write can
  theoretically race guest reuse inside the submit window. **P6b wave 1.5
  (deferred release, verified live):** with `gsp_phys_work_defer_release=1` a
  forwarded push's releases are recorded instead of applied and its GPFIFO
  entry is held until the hardware copy completes; the service thread then
  delivers the releases once. Live: `engaged=75 applied=75 timeouts=0
  canceled=0 overflow=0 pending=0`, the CUDA driver-API smoke stayed green
  (`ALL OK` 64/64), releases exactly once, clean release. **P6b wave 2
  (hardware-primary, verified live):** with `gsp_phys_work_hw_primary=1`
  (plus the forwarding, guest-memory and deferred-release parameters) the
  software copy is skipped for forwarded operations, so the physical engine
  authors the data; the guest read it back correctly. Live:
  `copy_skipped=119`, queued guest copies
  `req=100 ok=100 verify_ok=100 verify_fail=0 fallback=0 timeouts=0`, full
  smoke `matched=64/64` `ALL OK`. Nineteen skipped copies were
  software-authored inline fallbacks when the single forward slot was busy
  (`copy_soft=19 copy_fail=0`), the designed fail-closed path; a separate run
  with the guest-RAM ranges unregistered proved that fallback again
  (`copy_skipped=121`, recovered once each by the software copy,
  `copy_fail=0`, smoke still green), so the software fallback remains for any
  unqueued or failed copy. **P6a trigger residual — fixed (2026-10-02):**
  the debug triggers now re-translate the guest page's host physical address
  at trigger time through the fresh-scratch-line path (triggers 6/7), retiring
  the stale-PTE data-check failure; live `GUEST G2H COPY OK` and
  `GUEST W2G COPY OK verify_ok=1`, with the PTE self-test and ladder
  `REAL COPY OK` on a clean release. Forwarding was never affected (HPAs are
  resolved at submit time). **P6c C0/C1 (engine-authored guest CE semaphore
  relay, default off):** with `gsp_phys_relay=1` the guest-visible semaphore
  page is mapped into the CE channel and the engine authors one selected
  release; the deferred hold verifies the guest word and skips the software
  apply of that record, and any refusal, failure, timeout or failed
  verification leaves the unchanged software apply in charge, so the release
  stays exactly-once. Live: the engine-authored release was observed in the
  guest page (`relays=64`), the relay-on run stayed green
  (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64), and a
  forced-unresolvable control proved the fallback recovers every copy once;
  one relayed copy released its semaphore but failed the host-side data
  compare, so the parameter stays default-off pending triage, with the
  mismatch logged and the software apply still in charge. **P6c C3
  (forwarded-engine fault observation/relay, default off):** with
  `gsp_phys_fault_relay=1` a timed-out forwarded copy drains the GSP-RM
  status queue for the engine's fault record, matches it to the ladder
  channel and writes the guest channel notifier/event through the existing
  MSI-X path (live green, default off). **Remaining:** real guest page tables
  and channels, real page-directory shadowing, GR context validity/execution
  and `vectorAdd` correct results; guest-submission forwarding through the
  real guest page tables is not implemented yet, and the GR promote attempt
  wedged the server (§0.2). All new parameters default off; with them off the
  software path is unchanged.
* **Scope:** map guest page-table pages/records so the physical GPU can DMA
  them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real
  server; allocate real memory objects for guest allocations (fn4 with real
  PFNs); per-guest-channel real channels with client rings (module or guest
  memory); pump forwards real push streams referencing guest VAs; completion
  and faults relayed through real MSI-X -> module IRQ -> existing eventfd
  injection (`gsp_emu_inject_completion_interrupt`, `gsp_emulation.c:626`;
  the physical driver does not `request_irq()` yet, `gsp_phys_gpu.c:380-461`).
* **Acceptance:** `vectorAdd` correct results through a forwarded launch;
  copy with semaphore; second mdev is refused/emulated (single-session
  isolation until per-mdev clients exist); fault test.
* **Risk:** wrong page tables -> engine writes to arbitrary host memory;
  guest-page reachability is **resolved on the reference host** (P6a status
  above; §2.2 of `docs/04-implementation/01-known-knowns-and-unknowns.md` now
  carries the resolution note), so wrong-page-table writes remain the top
  risk. Keep `gsp_phys_seq_addr_valid()` (`booter:1800`) and all queue
  validations.
* **Rollback:** per-mdev isolation not yet available; disable forwarding and
  fall back to P5 host-buffer mode.

---

## 6. Top risks and the guards that must be preserved

1. **Host hard-freeze on unbind/rebind.** Unbinding the distribution driver
   can power-gate the board; re-bind intermittently fails and the host
   freezes with no crash record. Guards: two-param gate
   (`gsp_phys_gpu.c:670`), refuse if GPU/root port already
   runtime-suspended (`:688`), D0 lock on GPU + siblings + root port
   (`:499-549`), runtime-PM usage ref (`:188-204`), config-space liveness
   checks after claim and before rebind (`:591`), no retry after a
   power-loss signature, human approval for any reboot.
2. **WPR2 held / one-way protected region.** Always run the release sequence
   in order: `UNLOADING_GUEST_DRIVER` (only if init-done was seen)
   -> quiesce GSP falcon -> FWSEC teardown -> free staging
   (`gsp_phys_gpu.c:909-914`); if WPR2 survives, do **not** rebind and keep
   the locks (`:939`); a FWSEC-only stop is forbidden
   (`gsp_phys_gpu.c:770`). The debugfs `gsp-phys-unload-rpc` remains
   write-only and inert outside a live claim.
3. **RM-task wedge (live 0x40 incident).** Setting the GSP-owned channel
   `internalFlags` bit (0x40) wedged GSP-RM: the alloc RPC never replied and
   every later RPC timed out; the fail-safe held the device until reboot.
   The working tree now bans that bit and refuses the submission when the
   winning variant used server-allocated memdescs (`booter:281`, `:4714`,
   `:4942`). Never re-enable it; abort the ladder at the first failure with
   no object teardown (`:5123` comments); `UNLOADING_GUEST_DRIVER` + rmmod is
   the only recovery.
4. **Wrong-page-table / wrong-doorbell writes.** No forwarding seam may
   program guest-derived page tables into the real MMU until the P6c
   page-table population path is proven end to end; the P6b waves 1/1.5/2
   path submits engine work only through module-built scratch page tables
   mapping already-resolved host pages and never reuses a fetch line in a
   boot.
   Guest-page reachability itself is resolved on the reference host (P6a),
   so wrong-page-table writes remain the top risk. Preserve: queue-header
   validation in `gsp_phys_queue_tx_submit()` (`booter:2983`), free-space
   check (`:3032`), status-record validation (`booter:2554`), sequencer
   address validation (`booter:1800`), stage-4 token/bounds guards
   (`booter:4969-4999`).
5. **Guest status-queue invariants.** Strict sequence/wrap/checksum and
   multi-element records; `gsp_rm_send()` (`:452`) is the only writer and
   must not be called while holding `gsp_phys_rm_lock` or waiting on the
   backend; the event-burst cap (`:488`) and parse retry backoff (`:3397`)
   stay.
6. **Two planes, two BARs.** The guest face must only ever touch
   `gsp_emu_context.bar0_vaddr` (the vzalloc shadow,
   `gsp_emulation.c:761-778`); the backend must only ever touch `phys.bar0`
   (real ioremap, `gsp_phys_gpu.c:420`). Never ioremap a guest GPA (the
   fixed `gsp_client_init_emulation()` contract).
7. **Ordering and context rules.** No long scans or backend waits in VFIO
   callbacks; `vfio_dma_rw` only from kthread context (mm == NULL);
   never `rmmod` with a live VM; mdev remove before module unload; stop
   emulation before backend release once P2+ lands (new `nv_vgpu_exit`
   order).
8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX
   seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only
   `gsp-phys-state`/`gsp-phys-log-dump` files, and attach the evidence
   artifacts to the change on every live cycle.

---

## Appendix A - line anchor map (function -> file:line)

> **Historical anchors.** The line numbers below were taken from the
> working-tree revision used for the original design and are stale; they are
> kept as a historical map only. Anchor by function name when re-grepping and
> treat the numbers as untrusted.

`kernel/gsp_phys_booter.c` (working-tree revision used for this design):
`gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init`
2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670,
`gsp_phys_boot_rmwait` 2764, `gsp_phys_queue_tx_submit` 2935, `gsp_phys_rm_lock`
3671, `gsp_phys_boot_rm_unload` 3673, `gsp_phys_sci_decode` 3820,
`gsp_phys_rm_probe_fn65` 3918, `fn1` 3984, `_cycle` 4056, `gsp_phys_rm_probe`
4086, `gsp_phys_rmcall_rpc` 4329, `_alloc` 4385, `_register_mem` 4454,
`_control` 4510, `_stage1` 4575, `_stage2` 4650, `_channel_variants` 4714,
`_stage3` 4761, `_build_methods` 4879, `_stage4` 4923, `_stage5` 5068,
`_run` 5108, `_fini` 5197, debugfs write handlers 5613/5681/5754,
`gsp_phys_boot_debugfs_init` 5827, `gsp_phys_boot_quiesce` 5919,
`gsp_phys_boot_staging_free` 5944.

`kernel/gsp_emulation.c`: thread 651, init 724, cleanup 831, poll_boot 193,
rm poll call 672, legacy RPC dispatch 458, MSI-X inject 626.

`kernel/gsp_rm_emulation.c`: `gsp_rm_send` 452, parse 806, link 917,
`set_guest_system_info` 982, `build_static_info` 1025, `get_static_info` 1170,
`rm_alloc` 2322, `rm_control` 2611 (default 2807), `free` 2882,
`set_page_directory` 2975, `unset` 3063, `handle_element` 3084,
`poll_commands` 3194, `on_booter_done` 3339, `poll` 3386.

`kernel/main.c`: `nv_vgpu_mdev_create` 535, `remove` 614, init_work 1553,
cleanup_work 1614, `nv_vgpu_init` 1388, claim 1450, debugfs init 1461,
sema start 1468, `nv_vgpu_exit` 1487, release 1500, sema stop 1525,
mdev unregister 1530, param 2457.

`kernel/gsp_phys_gpu.c`: claim 662, hazard gate 670, suspend refusal 688,
power lock 523/499, unbind 706, driver register 718, board check 591,
runtime ref 188, boot 745-848 (FWSEC 792, prime 799, run 805, rmwait 811),
release 876 (unload 909, quiesce 910, fwsec 911, staging free 914, WPR2
fail-safe 939, board check 955, rebind 972).

`kernel/gsp_sema_service.c`: doorbell 6369, start 6392, stop 6416, thread
6232, pump 5019, decode 2802, sem write 1514, notify 4151.