# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session

> **Status: design for implementation (2026-10-01). No code from this design is
> in the tree yet.** The physical backend itself is live: the module-owned
> GSP-RM completes initialization on the real TU104, serves the full
> static-config reply, and shuts down gracefully. The host resource-manager
> client ladder has since allocated memory, a virtual-address-space object with
> a committed page directory, a channel group and a real copy-engine channel,
> and has executed the **first real work on the physical engine** — a 4 KiB
> copy in virtual-address mode with the semaphore released and destination
> equal to source (cycle 25). What this document specifies is the remaining
> bridge: wiring that live session into the guest-facing mdev/VFIO path so that
> guest submissions execute on the physical GPU. It extends
> [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which
> chose the module-owned-GPU mechanism, and
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md),
> which describes the emulation plane it must not disturb.
>
> The design was written against the tree at commit `f8e5d3b`; line numbers are
> advisory and have moved with later commits — anchor by function name when
> re-grepping. Behaviour claims below are grounded in live evidence from the
> 2026-10-01 cycles; anything not yet exercised is labelled "not yet run".

---

## 0. Ground truth: the two planes that exist today

### 0.1 Guest-facing emulation plane (works, green)

* One emulation context per mdev: `nv_vgpu_device.gsp_comm_ctx`
  (`kernel/include/nvidia-vgpu-vfio.h:115`), created by
  `gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from
  `nv_vgpu_device_init_work()` (`kernel/main.c:1553`).
* The emulation thread (`gsp_emu_thread_fn`, `kernel/gsp_emulation.c:651`)
  does four things: samples the guest's BAR0 *shadow* falcon registers
  (`gsp_emu_poll_boot`, `:193`), polls the guest's sysmem msgq
  (`gsp_rm_emu_poll`, `kernel/gsp_rm_emulation.c:3386`), serves the legacy
  doorbell/shared-memory RPC class (`gsp_emu_dispatch_rpc`, `:458`), and
  injects MSI-X through the live eventfd (`gsp_emu_inject_completion_interrupt`,
  `:626`).
* The guest's real 570.144 physical-path protocol (msgq over *guest* system
  memory, vfio_dma_rw transport) is handled entirely in
  `gsp_rm_emulation.c`: queue-chain parse (`:806`), status-queue link
  (`:917`), command poll (`:3194`), per-function dispatch (`:3084`),
  reply construction/transmission (`gsp_rm_send`, `:452`).
* `GSP_INIT_DONE` is emitted from `gsp_rm_emu_on_booter_done()` (`:3339`)
  when the *emulated* Booter kick completes (`kernel/gsp_emulation.c:416-431`).
* Static configuration is a hand-built TU104 model:
  `gsp_rm_build_static_info()` (`:1025`) with 256 MB FB, fake
  `TU104_H_INTERNAL_{CLIENT,DEVICE,SUBDEVICE}` handles and hand-tuned engine
  caps; `gsp_rm_handle_get_static_info()` (`:1170`) sends it.
* Work is executed in software by the sema service: doorbell entry
  `gsp_sema_service_doorbell()` (`kernel/gsp_sema_service.c:6369`) called from
  the VFIO write path (`kernel/main.c:826`), pump
  `svc_pump_channels()` (`gsp_sema_service.c:5019`), decode
  `svc_decode_stream()` (`:2802`), release apply `svc_sem_write()` (`:1514`),
  guest event post `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`).

### 0.2 Physical backend plane (alive, queryable, not wired to the guest)

* Ownership: `gsp_phys_gpu_claim()` (`kernel/gsp_phys_gpu.c:662`) requires
  both `gsp_phys_gpu=1` (`kernel/main.c:2457`) and `gsp_phys_hazard_ack=1`
  (`kernel/gsp_phys_gpu.c:670`), locks the whole board + root port into D0
  (`gsp_phys_power_lock()`, `:523`; runtime-PM ref `:740`), unbinds the
  distribution driver (`:706`), binds `open-iov-gsp-phys` (`:718`) and verifies
  config-space liveness (`gsp_phys_board_check()`, `:591`).
* Boot: claim runs FWSEC (`:792`), staging `gsp_phys_boot_prime()`
  (`booter:1349`), Booter Load + RISC-V `gsp_phys_boot_run()` (`booter:1524`,
  pre-boot fn72/73 at `:1559-1568`), then init-done poll
  `gsp_phys_boot_rmwait()` (`booter:2764`, called `gsp_phys_gpu.c:811`).
  A full boot has run live multiple times; init-done with `rpc_result=0` and
  the full 1656-byte static-config reply were observed (cycle 13).
* Transport: `gsp_phys_queue_tx_submit()` (`booter:2935`) for TX,
  `gsp_phys_status_queue_*` (`booter:2467/2490/2554`) +
  `gsp_phys_status_wait()` (`booter:2670`) for RX. All real-GSP traffic is
  serialized by the single static mutex `gsp_phys_rm_lock`
  (`booter:3671`), taken by `gsp_phys_boot_rm_unload()` (`:3685`),
  `gsp_phys_rm_probe()` (`:4090`) and `gsp_phys_rmcall_run()` (`:5117`).
  `gsp_phys_queue_tx_submit()` and `gsp_phys_status_wait()` themselves take
  no lock and are not safe to call concurrently.
* Host RM-client ladder (debugfs only, `booter:5754` write handler):
  stage 1 root/device/subdevice (`:4575`), stage 2 memory registration
  (`:4650`), stage 3 TSG + channel variants (`:4761`; variant matrix `:4714`),
  stage 4 real CE submission (`:4923`), stage 5 adopt static-config handles
  (`:5068`). Stage 4 now runs end to end: the channel allocates, the page
  directory is committed, and a 4 KiB virtual-address-mode copy executes on
  the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25).
* Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends
  `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2
  (`:911`), frees staging (`:914`), and only rebinds the distribution driver
  if the board answers (`:955`) and WPR2 is gone (`:939`); otherwise the D0
  locks intentionally stay held (fail-safe).

### 0.3 The gap

The two planes never touch. The guest's static answers, controls and objects
are emulated; the physical session's real handles, real FB geometry, real
engine caps and real work capability are only visible to debugfs. The bridge
must be added at exactly three places (see §2): the reply point
`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the
lifecycle points in `main.c` / `gsp_phys_gpu.c`.

---

## 1. Target architecture: runtime state machine

### 1.1 Backend session state (module-global; one physical GPU)

```
        BACKEND_OFF
             |  first mdev create (gsp_phys_gpu=1 + hazard_ack; no other mdev)
             v
        BACKEND_CLAIMING      power-lock D0 -> unbind dist -> bind phys -> board check
             |  claim ok                            \  abort signature / lock fail
             v                                        v
        BACKEND_BOOTING       FWSEC -> prime -> Booter -> RISC-V -> rmwait (init-done)
             |  init-done rpc_result=0                \ init failure / timeout
             v                                         v
        BACKEND_READY         static-config snapshot cached; RPC/control forwarding legal
             |  last mdev remove | rmmod | board signature
             v
        BACKEND_STOPPING      stop emulation forwarding; UNLOADING_GUEST_DRIVER;
             |                quiesce; WPR2 teardown; staging free
             v
        BACKEND_OFF           power unlock; distribution driver rebound and settled
```

`BACKEND_READY` is the single gate for every live answer. In every other
state the guest-facing handlers must behave exactly as they do today.

### 1.2 Ownership: who starts/stops what

| Trigger | Action | Where |
|---|---|---|
| first `mdev` create | backend refcount++ ; if 0 -> claim + boot asynchronously from `init_work` | `kernel/main.c:535`, `:1553`; new `gsp_phys_backend_start()` |
| last `mdev` remove | backend refcount-- ; if 0 -> `gsp_phys_gpu_release()` after the emulation thread is stopped | `kernel/main.c:614`, `:1614`; new `gsp_phys_backend_stop()` |
| `rmmod` with live mdevs | forbidden by policy, but code path must drain: STOPPING first, then stop emu threads, then release | `kernel/main.c:1487` (`nv_vgpu_exit`) |
| load-time `gsp_phys_gpu=1` | keep today's supervised behavior: claim (and full boot) at module init, owned by the module, released only at `rmmod` (debugfs harness unchanged) | `kernel/main.c:1450` |
| QEMU close/open only | no backend action; the session persists across VM restarts until mdev remove | `nv_vgpu_vfio_dev_close`, `kernel/main.c:698` |
| power-loss signature anywhere | STOPPING immediately (no new forwards); release fail-safe keeps D0 locks; no retry in this boot | `gsp_phys_board_check`, `gsp_phys_gpu.c:591` |

The module-owned claim (load-time `gsp_phys_gpu=1`) and the mdev-owned claim
share the same `phys` singleton. `gsp_phys_gpu_claim()` is idempotent today
(`gsp_phys_gpu.c:667` returns 0 if already claimed); the refcount/ownership
flag is new state and decides **who** releases.

### 1.3 Locks and ownership

| Lock | Protects | Taken by | Rule |
|---|---|---|---|
| `gsp_phys_session_lock` (new) | backend state, refcount, owner | `backend_start/stop/state`; `nv_vgpu_exit` | outermost; never held while waiting on a guest lock; may be held while calling `rm_lock` |
| `gsp_phys_rm_lock` (existing, `booter:3671`) | all real-GSP TX/RX: queue cursors, `boot_stage.status_read_ptr`, `tx_seq_num`, probe/ladder/unload | `gsp_phys_boot_rm_unload`, `gsp_phys_rm_probe`, `gsp_phys_rmcall_run`, new `gsp_phys_backend_rpc()` | one real RPC at a time, always; **never** taken while holding `rm->stat_lock` or any guest-memory path |
| `rm->stat_lock` (`gsp_rm_emulation.c:186`) | guest status-queue writes | `gsp_rm_send` (`:452`) | build the full reply before taking it; release immediately after write; never hold across a backend wait |
| `vgpu->lock` (`nvidia-vgpu-vfio.h:99`) | per-mdev state transitions | `init_work`/`cleanup_work` | never held across claim/release or RPC waits |
| `nv_vgpu_driver.lock` | device list | create/remove, sema thread scan | never taken under session/rm locks |
| `svc.chan_lock` / `svc.vas_lock` | software work pipeline | sema service | unchanged; forwarding reads channel state copied out under these locks |

Lock order: `nv_vgpu_driver.lock` (shortest, leaf) < `gsp_phys_session_lock`
< `gsp_phys_rm_lock`. The emulation thread must take
`session_lock -> check state -> drop session_lock -> rm_lock -> re-check a
generation counter` so a release cannot deadlock against a forward in flight.

### 1.4 Failure paths

* **Guest crash / QEMU killed:** `open_count` drops; the backend stays READY
  and claimed (device owned, host headless). No device writes. The next
  `mdev` remove or `rmmod` drives the graceful path. This is the same
  `--keep` semantics the supervised harness already uses.
* **Guest RPC while STOPPING/unload already submitted:** every forwarding
  seam returns "not ready" and the emulation answer is used. This mirrors the
  defensive re-checks in `gsp_phys_rmcall_run()` (`booter:5119`).
* **Module unload order (must change):** today `nv_vgpu_exit()` calls
  `gsp_phys_gpu_release()` at `kernel/main.c:1500` *before*
  `gsp_sema_service_stop()` (`:1525`) and before mdev unregistration
  (`:1530`) which is what stops the emulation threads (`:1619`). Once
  handlers forward to the backend, release must not race live forwards:
  new order = set STOPPING (session lock) -> `gsp_sema_service_stop()` ->
  `mdev_unregister_driver()` (drains emulation) -> `gsp_phys_gpu_release()` ->
  parent/class teardown.
* **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out,
  the existing release still quiesces, runs the FWSEC teardown and refuses to
  rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit
  human approval plus a handoff update.
* **Static-config served before READY:** bounded wait (recommend 2 s) then
  pure emulation fallback, with a stat counter; never block the emulation
  thread longer than the guest's own RPC poll can tolerate.

---

## 2. Seam table

Legend: **L** = live backend, **E** = existing emulation, **H** = hybrid.
"Change point" is the exact function that must learn the new behavior.

### 2.1 Guest GSP-RM RPC class (msgq function IDs)

| # | Guest message | Today | Target | Change point (file:line) |
|---|---|---|---|---|
| 1 | `GSP_SET_SYSTEM_INFO` (72) | consumed, no reply (`gsp_rm_emulation.c:3095`) | **E** - describes the guest client at guest boot; the backend already sent its own fn72 at `booter:1559` | `gsp_rm_handle_element` `gsp_rm_emulation.c:3095` |
| 2 | `SET_REGISTRY` (73) | consumed, no reply (`:3096`) | **E** - same reason | `:3096` |
| 3 | `CONTINUATION_RECORD` (71) | SUCCESS reply (`:3105`) | **E** | `:3105` |
| 4 | `SET_GUEST_SYSTEM_INFO` (1) | echoes guest vgx version (`:982`) | **E** - this is the guest's negotiation, deliberately decoupled from the host session (see §4.3) | `gsp_rm_handle_set_guest_system_info` `:982` |
| 5 | `GET_GSP_STATIC_INFO` (65) | hard-coded TU104 model (`:1025`), 256 MB FB, fake handles, hand-tuned caps | **H** - overlay live identity/capability fields proven safe; keep model-consistent FB geometry, PDE bases and handles until phases P4/P6 (see §4) | `gsp_rm_build_static_info` `:1025`, `gsp_rm_handle_get_static_info` `:1170` |
| 6 | `GSP_RM_ALLOC` (103) | echo request + `NV_OK`; channels/events registered locally (`:2322`) | **H** - echo for emulation-only classes; forward selected classes (root/device/subdevice, then channel/memory) once a guest<->server handle map exists; always keep local side effects | `gsp_rm_handle_rm_alloc` `:2322` |
| 7 | `GSP_RM_CONTROL` (76) | ~25 static per-cmd handlers; unknown -> zeroed success (`:2611`, default `:2807`) | **H** - whitelist of side-effect-free queries goes live with emulation fallback; vGPU-specific controls stay **E** (`COPY_SERVER_RESERVED_PDES` `:2647`, `PROMOTE_FAULT_METHOD_BUFFERS` `:2654`) | `gsp_rm_handle_rm_control` `:2611`; new forward helper before the switch |
| 8 | `GSP_RM_FREE` (10) | echo + retire channel/events (`:2882`) | **H** - forward when the object was backend-allocated; always retire local bookkeeping | `gsp_rm_handle_free` `:2882` |
| 9 | `SET_PAGE_DIRECTORY` (54) | recorded for translation + SUCCESS (`:2975`) | **H** - recording stays **E**; forwarding to the real MMU is P6 and gated on guest pages being GPU-DMA-reachable | `gsp_rm_handle_set_page_directory` `:2975` |
| 10 | `UNSET_PAGE_DIRECTORY` (79) | record remove + SUCCESS (`:3063`) | **H** - same gate as #9 | `:3063` |
| 11 | `UPDATE_BAR_PDE` (70) | SUCCESS ack (`:3144`) | **E** now; **L** only when BAR1 is backed by real VRAM | `:3144` |
| 12 | `UNLOADING_GUEST_DRIVER` (47) | SUCCESS ack (`:3155`) | **E** - the guest's own shutdown; the real session's shutdown is host-driven on backend stop (`gsp_phys_gpu.c:909`) | `:3155` |
| 13 | default/unknown | generic SUCCESS (`:3169`) | **E** (fail-safe: never invent a live forward for an unknown function) | `:3169` |

### 2.2 Outbound events and replies

| # | Record | Today | Target | Change point |
|---|---|---|---|---|
| 14 | `GSP_INIT_DONE` (0x1001) | emitted as soon as the emulated Booter finishes (`gsp_rm_emulation.c:3374`) | **H** - emit immediately as today, but live answers gate themselves on READY; alternatively delay emission on a bounded backend wait so the guest's first RM RPC sees READY | `gsp_rm_emu_on_booter_done` `:3339` |
| 15 | `POST_EVENT` (0x1003) | software pump completion (`gsp_rm_post_channel_event` `:599`) | **E** as the relay mechanism; its *trigger* becomes the observation of a real hardware semaphore (P5) instead of a software op | `gsp_rm_post_channel_event` `:599`, `svc_notify_complete` `gsp_sema_service.c:4151` |
| 16 | all replies | single writer `gsp_rm_send` (`:452`) | **E** - keep as the only guest-queue writer; live payloads are copied into it | `gsp_rm_send` `:452` |

### 2.3 BAR / register and work-plane classes

| # | Guest-visible thing | Today | Target | Change point |
|---|---|---|---|---|
| 17 | GSP falcon boot regs (FWSEC/Booter/RISCV/WPR2), SW interrupt tree, MMU invalidate, L2 flushes | emulated auto-complete (`gsp_emulation.c:193-452`) | **E always** - the two GSP sessions are intentionally independent; guest boot must never depend on the physical session | `gsp_emu_poll_boot` `:193` |
| 18 | Guest VF doorbell `0x88704` | software pump (`main.c:826` -> `gsp_sema_service_doorbell` `:6369`) | **H** - pump classifies each push: pure state/release stays software; engine work is forwarded (P5) | `svc_pump_channels` `gsp_sema_service.c:5019`, `svc_decode_stream` `:2802` |
| 19 | BAR0/BAR1/BAR3 shadows, PRAMIN window | host-RAM shadows (`bar_emulation.c:180`, `main.c:1205`) | **E** until real VRAM forwarding (P6) | `nv_vgpu_bar_emul_*`, `nv_vgpu_vfio_dev_mmap` |
| 20 | legacy doorbell/shared-mem RPC class (`GSP_RPC_*`) | internal emulation client thread (`gsp_emulation.c:458-620`, `:674-694`) | **E / disable when READY** - not part of the guest physical-path protocol; gate it off in backend mode to avoid intercepting stray writes | `gsp_emu_thread_fn` `:674` |

**Highest-leverage seams:** #5/#7 (one reply point, one dispatch point; all
live answers plug in there) and #6 (whatever alloc/free semantics the guest
gets determine whether real channels/page tables are possible).

---

## 3. Lifecycle changes in `main.c` / `gsp_emulation.c`

1. **New backend API** (add to `gsp_phys_booter.[ch]`, used by both debugfs
   and the guest plane; keeps `gsp_phys_rm_lock` private):
   * `int gsp_phys_backend_state(void)` - OFF/CLAIMING/BOOTING/READY/STOPPING.
   * `int gsp_phys_backend_rpc(u32 function, const void *req, u32 req_len,
     void **reply, u32 *reply_len, u32 timeout_ms)` - checks
     `claimed && rm_init_done && !rm_unload_submitted`, takes
     `gsp_phys_rm_lock`, calls `gsp_phys_queue_tx_submit()` +
     `gsp_phys_status_wait()` (both at `booter:2935`/`:2670`), returns a copy
     of the payload; caller frees. Unmatched records (events, NOCAT) are
     relayed or summarized like `gsp_phys_status_wait` already does.
   * `int gsp_phys_backend_static_snapshot(u8 *out, u32 len)` - cached full
     1656-byte fn65 payload; filled once at READY by the existing
     `gsp_phys_rm_probe_cycle()` (`booter:4056`) under the same lock.
   * `void gsp_phys_backend_begin_stop(void)` / `gsp_phys_backend_end_stop()` -
     state transitions only (no hardware).
2. **New session state + refcount** in `gsp_phys_gpu.c` (new mutex, owner
   flag, state enum). `gsp_phys_gpu_claim()`/`_release()` stay the only
   hardware entry points; add `gsp_phys_backend_start()/stop()` that call
   them exactly once.
3. **`nv_vgpu_mdev_create()` (`main.c:535`)**: after
   `vfio_register_emulated_iommu_dev()` (`:591`) and before/around
   `schedule_work(&vgpu->init_work)` (`:598`), call
   `gsp_phys_backend_start()`. Claiming must happen while no VM holds a live
   mdev: with `available_instances` computed from per-type max (1) in
   `nv_vgpu_get_available()` (`main.c:81`) there is at most one mdev of the
   type in flight, so this holds.
4. **`nv_vgpu_device_init_work()` (`main.c:1553`)**: keep BAR/IRQ/memory/
   `gsp_emu_init()` order. The backend boot runs in its own kworker so
   `mdev` create returns quickly; `init_work` only observes READY when it
   initializes forwarding tables. Do **not** run the 60 s `rmwait`
   (`booter:2764`, timeout `GSP_PHYS_RM_INIT_TIMEOUT_MS` 60000 at
   `booter:446`) inline in `init_work` if it can be avoided: run claim+boot
   in a dedicated work item so `mdev->remove` is never stuck behind it.
5. **`nv_vgpu_device_cleanup_work()` (`main.c:1614`)**: unchanged per-mdev,
   but the last-cleanup path calls `gsp_phys_backend_stop()` **after**
   `gsp_emu_cleanup()` (`:1619`) so no emulation thread can forward while the
   backend is being torn down.
6. **`nv_vgpu_exit()` (`main.c:1487`)**: reorder as in §1.4; the load-time
   claim path (owner = module) keeps the existing `gsp_phys_gpu_release()`
   position relative to debugfs fini, but mdev-owned sessions are released in
   the new order.
7. **`gsp_emu_init()` (`gsp_emulation.c:724`)**: no structural change; the
   `gsp_rm_emu` context gains a reference to the module backend (not a new
   per-mdev session). `gsp_emu_cleanup()` (`:831`) unchanged except it must
   not be the thing that releases the backend (done by `main.c`, step 5).
8. **`gsp_rm_emu_on_booter_done()` (`gsp_rm_emulation.c:3339`)**: after
   linking the queues, optionally wait (bounded) for READY before emitting
   INIT_DONE, or emit immediately and let each live handler wait. Recommended:
   emit immediately (the guest's early RPCs are all emulatable), and let
   `GET_GSP_STATIC_INFO` be the single bounded-wait point.
9. **`gsp_emu_thread_fn()` (`gsp_emulation.c:651`)**: no new blocking calls in
   the loop; live forwards happen inside `gsp_rm_handle_element()` on that
   thread, with the 2 s per-RPC bound. If a longer wait is ever needed, move
   forwarding to a dedicated backend kthread and keep this loop responsive.

---

## 4. Static-config flow and identity/version negotiation

### 4.1 Current guest model (must stay internally consistent)

`gsp_rm_build_static_info()` (`gsp_rm_emulation.c:1025`) hard-codes:
`fb_length = 0x10000000` (256 MB, offset 1224), `fbio_mask=0xff`,
`fbp_mask=0x3f`, `gpuNameString="NVIDIA GeForce RTX 2080 SUPER"` (1260),
`gpuShortNameString="TU104"` (1324), `bar1PdeBase=bar2PdeBase=0`
(1536/1544), fake internal handles (1600/1604/1608), engine caps
`0x60803E02 / 0x00362020 / 0` (1204). The 256 MB figure is coupled to the
BAR0 shadow's `LOCAL_MEMORY_RANGE` (`bar_emulation.c:280`,
`0x100ce0=0x18` -> 256 MB) and to the emulated WPR2/FRTS location the guest
driver checks. Changing it without changing the shadow breaks the guest boot
(this is why 512 MB was reverted, per the comment at `:1096`).

### 4.2 Live values observed (cycle 13)

```
static-info internal: hClient=0xc2000006 hDevice=0xabcd0080 hSubdevice=0xabcd2080
static-info fb: bar1PdeBase=0x1f7650000 bar2PdeBase=0x1f76c2000
               fb_length=0x200000000 fbio_mask=0xf fbp_mask=0x11
               busWidth=0 ramType=256 l2CacheSize=15
static-info gpu: name='NVIDIA GeForce RTX 2080 Super with Max-Q Design'
                short='TU104-B' vbiosSubVendor=0x4d50 vbiosSubDevice=0x4449
static-info engineCaps=[0x08083e02,0x00000044,0x00000000]
```

Two grounded warnings:

* The decoded `busWidth=0`, `ramType=256` are not plausible TU104 values
  (256-bit, GDDR6). Either the decode offsets for those two fields do not
  match the real struct, or the emulated values were hand-derived rather than
  observed. **A/B every field before overlaying it.**
* Engine caps differ materially (`0x60803E02` vs `0x08083E02`, and the second
  word). The emulation deliberately sets extra bits (COPY14 etc.) to make
  `libcuda` accept the engine inventory. Overlaying live caps would change
  the guest's engine/class topology and can regress the whole CUDA init.
  Keep emulated caps until the backend can actually service the resulting
  demands (P4+).

### 4.3 Proposed flow

1. At READY, cache the full fn65 payload (new bytes in the backend session,
   keyed by the loaded firmware's `phys.fw_crc32`, `gsp_phys_gpu.c:272`), via
   the existing probe cycle (`booter:4056`) under `gsp_phys_rm_lock`.
2. `gsp_rm_handle_get_static_info()` (`:1170`) keeps building the emulated
   1656-byte model, then applies an allow-listed overlay from the snapshot
   (start: name/short-name/VBIOS identity/L2 if A/B passes; **not** FB
   geometry, masks, PDE bases, handles, engine caps).
3. Any overlay field is independently switchable (`gsp_phys_static_live`
   module parameter, default off) so a regression is a one-line rollback.
4. Handles move in P4 and PDE bases/geometry in P6, each behind its own
   switch and its own A/B.
5. Version negotiation stays decoupled: the guest sees the emulation's
   version answer (`SET_GUEST_SYSTEM_INFO`, `gsp_rm_emulation.c:982`); the
   real server is negotiated by the host as the host client
   (`gsp_phys_rm_probe_fn1`, `booter:3984`, vgx 0x29/0x0C). The guest must
   never learn the physical firmware's version; the emulation target remains
   570.144 while the host may boot a different, matching TU10x GSP image
   (`gsp_phys_gpu.c:75`). The wire structures are the same family and the
   live snapshot is keyed by firmware CRC.

---

## 5. Phased implementation plan (smallest first)

Each phase is independently revertible. "Supervised" means the documented
harness discipline: `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`, human
go-ahead, recovery plan, pstore, never unattended
(`docs/03-architecture/08-physical-gpu-integration.md` §7/§8).

### P0 - Backend API skeleton (inert)
* **Scope:** new state/refcount/lock in `gsp_phys_gpu.c`; locked
  `backend_rpc()`/`static_snapshot()` in `gsp_phys_booter.c`; no caller
  changes; no behavior change.
* **Acceptance:** `make kernel` clean; load/unload with no params; all
  existing debugfs triggers behave identically; new API logs and refuses when
  OFF/not-init-done/unloading.
* **Risk:** none (never sends unless a live init-done session exists).
* **Rollback:** trivial.

### P1 - Lifecycle wiring (claim on mdev create, release on last remove)
* **Scope:** `main.c` create/remove/init_work/cleanup_work/exit ordering
  (§3 items 3-6); no live answers yet.
* **Acceptance (supervised, harness `phys_claim_cycle.sh` style):** fresh
  boot -> insmod params -> create mdev -> one real boot log sequence (FWSEC,
  Booter M0=0, RISC-V active, init-done after sequencer) -> guest boots and
  the CUDA driver-API smoke test still prints `ALL OK` 64/64 -> stop VM ->
  remove mdev -> `UNLOADING_GUEST_DRIVER` suspend, scratch14 bits 24-27 DONE,
  WPR2 down, distribution driver rebound; repeat twice; zero power-domain
  cycles; no reboot.
* **Risk:** **highest phase.** Claim is now triggered by a sysfs action, not
  by module load. The two-param gate and the D0 lock must already be active
  inside claim (they are, `gsp_phys_gpu.c:670/700`), the claim must refuse a
  second owner, and the release must never run while a forward is in flight
  (STOPPING gate). Any power-loss signature -> hold locks, no retry.
* **Rollback:** `gsp_phys_gpu=0` (no claim) or load-time `gsp_phys_gpu=1`
  only; `mdev` emits `-EOPNOTSUPP` from `backend_start()`.

### P2 - Static-config identity overlay
* **Scope:** cache snapshot at READY; overlay allow-list in
  `gsp_rm_build_static_info()`/`gsp_rm_handle_get_static_info()`; module
  parameter; stats. Bounded wait (2 s) then emulation fallback.
* **Acceptance:** guest still `nvidia-smi -L` + smoke `ALL OK`; guest
  `nvidia-smi -q` name changes to the live `... with Max-Q Design`; dmesg
  shows which fields came live; backend absent -> byte-identical guest reply.
* **Risk:** a consumed identity field changes a driver decision (VBIOS
  validation is the likeliest). Mitigation: field-by-field A/B, default-off,
  instant fallback; never touch geometry/caps/handles in P2.
* **Rollback:** parameter off.

### P3 - Read-only control forwarding (whitelist)
* **Scope:** in `gsp_rm_handle_rm_control()` (`:2611`), before the per-cmd
  switch: for an allow-list of side-effect-free queries (FB/BUS/KGR/GMMU/
  CE/display statics), call `backend_rpc()` with the **adopted internal
  handles** from the snapshot (the guest's emulated handles are not valid on
  the server yet), copy reply params (payload+24) on `rpc_result==0 &&
  status==0`, else fall back to the existing handler. Add per-cmd
  forwarded/fallback counters.
* **Acceptance:** guest boot and smoke green with forwarding on; dmesg shows
  N forwarded controls; forced backend timeout -> same behavior as today;
  repeat calls yield identical params (no server-side state change).
* **Risk:** a whitelist entry is not actually read-only (display/KGR can
  allocate on first call). Start with FB/BUS/GMMU static info only, verify
  against the server's log ring (`gsp-phys-log-dump`), then grow.
* **Rollback:** parameter/whitelist off.

### P4 - Alloc/free forwarding for the object ladder
* **Scope:** `gsp_rm_handle_rm_alloc()` (`:2322`) forwards
  root/device/subdevice (and only those, first) via `backend_rpc(fn103)`;
  returns the server's chosen handle to the guest; maintains a
  guest-handle -> server-handle map used by P3 and `gsp_rm_handle_free()`.
  Any failure falls back to today's echo (never half-forward), and local
  channel/event registration still runs.
* **Acceptance:** guest boot + smoke green; dmesg shows adopted handles
  (`0xc2.../0xabcd0080/0xabcd2080`) matching cycle13; guest FREE returns
  success; no server-side wedge (no RPC timeout after alloc).
* **Risk:** the guest's alloc params may not satisfy the server's per-class
  validation; a bad forward can wedge the RM task (see the `0x40` GSP-owned
  channel incident). Guard: one class at a time, abort on first failure,
  `UNLOADING_GUEST_DRIVER` + rmmod remains the only fail-safe; never retry a
  mutating RPC.
* **Rollback:** parameter off (guest returns to echo path).

### P5 - Work forwarding + completion relay (interim, host buffers)
* **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25:
  channel allocates, page directory commits, `REAL COPY OK` on the physical
  CE in VA mode). The remaining work: bind each guest CE/GR channel to a real
  channel; in `svc_pump_channels()` (`gsp_sema_service.c:5019`) classify the
  decoded push and, for forwarded ops, submit an equivalent real method block
  built by the existing `gsp_phys_rmcall_build_methods()` (`booter:4879`) +
  GPFIFO/USERD/doorbell (`:4991-5032`). Observe the real semaphore, then
  mirror the release into guest memory with `svc_sem_write()`
  (`gsp_sema_service.c:1514`) and post the guest event with
  `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`). All hardware
  operands are module-owned DMA buffers (as in the ladder), so no guest memory
  is touched by the engine yet.
* **Acceptance:** ladder: `echo 4 > gsp-phys-rm-call` logs
  `REAL COPY OK` (`booter:5052`) and src==dst after a hardware copy - met
  (cycle 25, 4 KiB, semaphore released, dst==src). Guest: a submitted copy
  completes with the semaphore observed in host memory and the guest tracker
  advancing; wrong-operand/refused submissions fall back to software and are
  counted.
* **Risk:** doorbell to a wedged channel; wrong GPFIFO/USERD (guards:
  `rmcall_channel_server_mem` refusal `:4944`, work-submit token after
  `GPFIFO_SCHEDULE` `:4977`, `GP_PUT` bounds `:4995`); hardware writes only
  host DMA memory, so no cross-tenant exposure yet.
* **Rollback:** per-channel `forward=0`; software pump is untouched.

### P6 - Guest memory into the real MMU (end state: guest submission on hardware)
* **Scope:** map guest page-table pages/records so the physical GPU can DMA
  them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real
  server; allocate real memory objects for guest allocations (fn4 with real
  PFNs); per-guest-channel real channels with client rings (module or guest
  memory); pump forwards real push streams referencing guest VAs; completion
  and faults relayed through real MSI-X -> module IRQ -> existing eventfd
  injection (`gsp_emu_inject_completion_interrupt`, `gsp_emulation.c:626`;
  the physical driver does not `request_irq()` yet, `gsp_phys_gpu.c:380-461`).
* **Acceptance:** `vectorAdd` correct results through a forwarded launch;
  copy with semaphore; second mdev is refused/emulated (single-session
  isolation until per-mdev clients exist); fault test.
* **Risk:** wrong page tables -> engine writes to arbitrary host memory;
  IOMMU/DMA reachability of guest pages is the project's existing
  known-unknown (`docs/04-implementation/01-known-knowns-and-unknowns.md`
  §2.2) and is a prerequisite, not solved by this design. Keep
  `gsp_phys_seq_addr_valid()` (`booter:1800`) and all queue validations.
* **Rollback:** per-mdev isolation not yet available; disable forwarding and
  fall back to P5 host-buffer mode.

---

## 6. Top risks and the guards that must be preserved

1. **Host hard-freeze on unbind/rebind.** Unbinding the distribution driver
   can power-gate the board; re-bind intermittently fails and the host
   freezes with no crash record. Guards: two-param gate
   (`gsp_phys_gpu.c:670`), refuse if GPU/root port already
   runtime-suspended (`:688`), D0 lock on GPU + siblings + root port
   (`:499-549`), runtime-PM usage ref (`:188-204`), config-space liveness
   checks after claim and before rebind (`:591`), no retry after a
   power-loss signature, human approval for any reboot.
2. **WPR2 held / one-way protected region.** Always run the release sequence
   in order: `UNLOADING_GUEST_DRIVER` (only if init-done was seen)
   -> quiesce GSP falcon -> FWSEC teardown -> free staging
   (`gsp_phys_gpu.c:909-914`); if WPR2 survives, do **not** rebind and keep
   the locks (`:939`); a FWSEC-only stop is forbidden
   (`gsp_phys_gpu.c:770`). The debugfs `gsp-phys-unload-rpc` remains
   write-only and inert outside a live claim.
3. **RM-task wedge (live 0x40 incident).** Setting the GSP-owned channel
   `internalFlags` bit (0x40) wedged GSP-RM: the alloc RPC never replied and
   every later RPC timed out; the fail-safe held the device until reboot.
   The working tree now bans that bit and refuses the submission when the
   winning variant used server-allocated memdescs (`booter:281`, `:4714`,
   `:4942`). Never re-enable it; abort the ladder at the first failure with
   no object teardown (`:5123` comments); `UNLOADING_GUEST_DRIVER` + rmmod is
   the only recovery.
4. **Wrong-page-table / wrong-doorbell writes.** No forwarding seam may
   submit engine work or program page tables until P6's reachability
   prerequisite holds. Preserve: queue header validation in
   `gsp_phys_queue_tx_submit()` (`booter:2983`), free-space check (`:3032`),
   status-record validation (`booter:2554`), sequencer address validation
   (`booter:1800`), stage-4 token/bounds guards (`booter:4969-4999`).
5. **Guest status-queue invariants.** Strict sequence/wrap/checksum and
   multi-element records; `gsp_rm_send()` (`:452`) is the only writer and
   must not be called while holding `gsp_phys_rm_lock` or waiting on the
   backend; the event-burst cap (`:488`) and parse retry backoff (`:3397`)
   stay.
6. **Two planes, two BARs.** The guest face must only ever touch
   `gsp_emu_context.bar0_vaddr` (the vzalloc shadow,
   `gsp_emulation.c:761-778`); the backend must only ever touch `phys.bar0`
   (real ioremap, `gsp_phys_gpu.c:420`). Never ioremap a guest GPA (the
   fixed `gsp_client_init_emulation()` contract).
7. **Ordering and context rules.** No long scans or backend waits in VFIO
   callbacks; `vfio_dma_rw` only from kthread context (mm == NULL);
   never `rmmod` with a live VM; mdev remove before module unload; stop
   emulation before backend release once P2+ lands (new `nv_vgpu_exit`
   order).
8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX
   seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only
   `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree
   (`backups/evidence-*`) current on every live cycle.

---

## Appendix A - line anchor map (function -> file:line)

`kernel/gsp_phys_booter.c` (working-tree revision used for this design):
`gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init`
2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670,
`gsp_phys_boot_rmwait` 2764, `gsp_phys_queue_tx_submit` 2935, `gsp_phys_rm_lock`
3671, `gsp_phys_boot_rm_unload` 3673, `gsp_phys_sci_decode` 3820,
`gsp_phys_rm_probe_fn65` 3918, `fn1` 3984, `_cycle` 4056, `gsp_phys_rm_probe`
4086, `gsp_phys_rmcall_rpc` 4329, `_alloc` 4385, `_register_mem` 4454,
`_control` 4510, `_stage1` 4575, `_stage2` 4650, `_channel_variants` 4714,
`_stage3` 4761, `_build_methods` 4879, `_stage4` 4923, `_stage5` 5068,
`_run` 5108, `_fini` 5197, debugfs write handlers 5613/5681/5754,
`gsp_phys_boot_debugfs_init` 5827, `gsp_phys_boot_quiesce` 5919,
`gsp_phys_boot_staging_free` 5944.

`kernel/gsp_emulation.c`: thread 651, init 724, cleanup 831, poll_boot 193,
rm poll call 672, legacy RPC dispatch 458, MSI-X inject 626.

`kernel/gsp_rm_emulation.c`: `gsp_rm_send` 452, parse 806, link 917,
`set_guest_system_info` 982, `build_static_info` 1025, `get_static_info` 1170,
`rm_alloc` 2322, `rm_control` 2611 (default 2807), `free` 2882,
`set_page_directory` 2975, `unset` 3063, `handle_element` 3084,
`poll_commands` 3194, `on_booter_done` 3339, `poll` 3386.

`kernel/main.c`: `nv_vgpu_mdev_create` 535, `remove` 614, init_work 1553,
cleanup_work 1614, `nv_vgpu_init` 1388, claim 1450, debugfs init 1461,
sema start 1468, `nv_vgpu_exit` 1487, release 1500, sema stop 1525,
mdev unregister 1530, param 2457.

`kernel/gsp_phys_gpu.c`: claim 662, hazard gate 670, suspend refusal 688,
power lock 523/499, unbind 706, driver register 718, board check 591,
runtime ref 188, boot 745-848 (FWSEC 792, prime 799, run 805, rmwait 811),
release 876 (unload 909, quiesce 910, fwsec 911, staging free 914, WPR2
fail-safe 939, board check 955, rebind 972).

`kernel/gsp_sema_service.c`: doorbell 6369, start 6392, stop 6416, thread
6232, pump 5019, decode 2802, sem write 1514, notify 4151.