path: root/docs/03-architecture/11-physical-backend-guest-wiring.md

# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session

> **Status: P0-P5(w1) implemented; P6a guest-page reachability proven live
> (2026-10-02).** The backend API skeleton
> (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that
> path is an open defect off the critical path), the static-config identity
> overlay (P2, verified live in an A/B run), the read-only control
> forwarding whitelist (P3, default off, verified live: ok=22/fallback=2,
> smoke `ALL OK` 64/64) and the P5 wave-1 guest-submission shadow execution
> (default off, verified live: 514 real CE copies during a green guest smoke
> `ALL OK` 64/64) are in the tree. The physical
> backend itself is live: the module-owned
> GSP-RM completes initialization on the real TU104, serves the full
> static-config reply, and shuts down gracefully. The host resource-manager
> client ladder has since allocated memory, a virtual-address-space object with
> a committed page directory, a channel group and a real copy-engine channel,
> and has executed the **first real work on the physical engine** — a 4 KiB
> copy in virtual-address mode with the semaphore released and destination
> equal to source (cycle 25). What this document specifies is the remaining
> bridge: wiring that live session into the guest-facing mdev/VFIO path so that
> guest submissions execute on the physical GPU. It extends
> [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which
> chose the module-owned-GPU mechanism, and
> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md),
> which describes the emulation plane it must not disturb.
>
> The design was written against the tree at commit `f8e5d3b`; line numbers are
> advisory and have moved with later commits — anchor by function name when
> re-grepping. Behaviour claims below are grounded in live evidence from the
> 2026-10-01 and 2026-10-02 cycles; anything not yet exercised is labelled
> "not yet run".

---

## 0. Ground truth: the two planes that exist today

### 0.1 Guest-facing emulation plane (works, green)

* One emulation context per mdev: `nv_vgpu_device.gsp_comm_ctx`
  (`kernel/include/nvidia-vgpu-vfio.h:115`), created by
  `gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from
  `nv_vgpu_device_init_work()` (`kernel/main.c:1553`).
* The emulation thread (`gsp_emu_thread_fn`, `kernel/gsp_emulation.c:651`)
  does four things: samples the guest's BAR0 *shadow* falcon registers
  (`gsp_emu_poll_boot`, `:193`), polls the guest's sysmem msgq
  (`gsp_rm_emu_poll`, `kernel/gsp_rm_emulation.c:3386`), serves the legacy
  doorbell/shared-memory RPC class (`gsp_emu_dispatch_rpc`, `:458`), and
  injects MSI-X through the live eventfd (`gsp_emu_inject_completion_interrupt`,
  `:626`).
* The guest's real 570.144 physical-path protocol (msgq over *guest* system
  memory, vfio_dma_rw transport) is handled entirely in
  `gsp_rm_emulation.c`: queue-chain parse (`:806`), status-queue link
  (`:917`), command poll (`:3194`), per-function dispatch (`:3084`),
  reply construction/transmission (`gsp_rm_send`, `:452`).
* `GSP_INIT_DONE` is emitted from `gsp_rm_emu_on_booter_done()` (`:3339`)
  when the *emulated* Booter kick completes (`kernel/gsp_emulation.c:416-431`).
* Static configuration is a hand-built TU104 model:
  `gsp_rm_build_static_info()` (`:1025`) with 256 MB FB, fake
  `TU104_H_INTERNAL_{CLIENT,DEVICE,SUBDEVICE}` handles and hand-tuned engine
  caps; `gsp_rm_handle_get_static_info()` (`:1170`) sends it.
* Work is executed in software by the sema service: doorbell entry
  `gsp_sema_service_doorbell()` (`kernel/gsp_sema_service.c:6369`) called from
  the VFIO write path (`kernel/main.c:826`), pump
  `svc_pump_channels()` (`gsp_sema_service.c:5019`), decode
  `svc_decode_stream()` (`:2802`), release apply `svc_sem_write()` (`:1514`),
  guest event post `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`).

### 0.2 Physical backend plane (alive, queryable, not wired to the guest)

* Ownership: `gsp_phys_gpu_claim()` (`kernel/gsp_phys_gpu.c:662`) requires
  both `gsp_phys_gpu=1` (`kernel/main.c:2457`) and `gsp_phys_hazard_ack=1`
  (`kernel/gsp_phys_gpu.c:670`), locks the whole board + root port into D0
  (`gsp_phys_power_lock()`, `:523`; runtime-PM ref `:740`), unbinds the
  distribution driver (`:706`), binds `open-iov-gsp-phys` (`:718`) and verifies
  config-space liveness (`gsp_phys_board_check()`, `:591`).
* Boot: claim runs FWSEC (`:792`), staging `gsp_phys_boot_prime()`
  (`booter:1349`), Booter Load + RISC-V `gsp_phys_boot_run()` (`booter:1524`,
  pre-boot fn72/73 at `:1559-1568`), then init-done poll
  `gsp_phys_boot_rmwait()` (`booter:2764`, called `gsp_phys_gpu.c:811`).
  A full boot has run live multiple times; init-done with `rpc_result=0` and
  the full 1656-byte static-config reply were observed (cycle 13).
* Transport: `gsp_phys_queue_tx_submit()` (`booter:2935`) for TX,
  `gsp_phys_status_queue_*` (`booter:2467/2490/2554`) +
  `gsp_phys_status_wait()` (`booter:2670`) for RX. All real-GSP traffic is
  serialized by the single static mutex `gsp_phys_rm_lock`
  (`booter:3671`), taken by `gsp_phys_boot_rm_unload()` (`:3685`),
  `gsp_phys_rm_probe()` (`:4090`) and `gsp_phys_rmcall_run()` (`:5117`).
  `gsp_phys_queue_tx_submit()` and `gsp_phys_status_wait()` themselves take
  no lock and are not safe to call concurrently.
* Host RM-client ladder (debugfs only, `booter:5754` write handler):
  stage 1 root/device/subdevice (`:4575`), stage 2 memory registration
  (`:4650`), stage 3 TSG + channel variants (`:4761`; variant matrix `:4714`),
  stage 4 real CE submission (`:4923`), stage 5 adopt static-config handles
  (`:5068`). Stage 4 now runs end to end: the channel allocates, the page
  directory is committed, and a 4 KiB virtual-address-mode copy executes on
  the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25).
* Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends
  `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2
  (`:911`), frees staging (`:914`), and only rebinds the distribution driver
  if the board answers (`:955`) and WPR2 is gone (`:939`); otherwise the D0
  locks intentionally stay held (fail-safe).

### 0.3 The gap

The two planes now meet only at the lifecycle seam (P0/P1, mdev-owned
claim/release) and the static-config identity overlay (P2); controls, objects,
FB geometry, engine caps and work are still emulated. The remaining bridge
must be added at the reply point `gsp_rm_send()`, the inbound dispatch point
`gsp_rm_handle_element()`, and the work-plane seams (see §2).

---

## 1. Target architecture: runtime state machine

### 1.1 Backend session state (module-global; one physical GPU)

```
        BACKEND_OFF
             |  first mdev create (gsp_phys_gpu=1 + hazard_ack; no other mdev)
             v
        BACKEND_CLAIMING      power-lock D0 -> unbind dist -> bind phys -> board check
             |  claim ok                            \  abort signature / lock fail
             v                                        v
        BACKEND_BOOTING       FWSEC -> prime -> Booter -> RISC-V -> rmwait (init-done)
             |  init-done rpc_result=0                \ init failure / timeout
             v                                         v
        BACKEND_READY         static-config snapshot cached; RPC/control forwarding legal
             |  last mdev remove | rmmod | board signature
             v
        BACKEND_STOPPING      stop emulation forwarding; UNLOADING_GUEST_DRIVER;
             |                quiesce; WPR2 teardown; staging free
             v
        BACKEND_OFF           power unlock; distribution driver rebound and settled
```

`BACKEND_READY` is the single gate for every live answer. In every other
state the guest-facing handlers must behave exactly as they do today.

### 1.2 Ownership: who starts/stops what

| Trigger | Action | Where |
|---|---|---|
| first `mdev` create | backend refcount++ ; if 0 -> claim + boot asynchronously from `init_work` | `kernel/main.c:535`, `:1553`; new `gsp_phys_backend_start()` |
| last `mdev` remove | backend refcount-- ; if 0 -> `gsp_phys_gpu_release()` after the emulation thread is stopped | `kernel/main.c:614`, `:1614`; new `gsp_phys_backend_stop()` |
| `rmmod` with live mdevs | forbidden by policy, but code path must drain: STOPPING first, then stop emu threads, then release | `kernel/main.c:1487` (`nv_vgpu_exit`) |
| load-time `gsp_phys_gpu=1` | keep today's supervised behavior: claim (and full boot) at module init, owned by the module, released only at `rmmod` (debugfs harness unchanged) | `kernel/main.c:1450` |
| QEMU close/open only | no backend action; the session persists across VM restarts until mdev remove | `nv_vgpu_vfio_dev_close`, `kernel/main.c:698` |
| power-loss signature anywhere | STOPPING immediately (no new forwards); release fail-safe keeps D0 locks; no retry in this boot | `gsp_phys_board_check`, `gsp_phys_gpu.c:591` |

The module-owned claim (load-time `gsp_phys_gpu=1`) and the mdev-owned claim
share the same `phys` singleton. `gsp_phys_gpu_claim()` is idempotent today
(`gsp_phys_gpu.c:667` returns 0 if already claimed); the refcount/ownership
flag is new state and decides **who** releases.

### 1.3 Locks and ownership

| Lock | Protects | Taken by | Rule |
|---|---|---|---|
| `gsp_phys_session_lock` (new) | backend state, refcount, owner | `backend_start/stop/state`; `nv_vgpu_exit` | outermost; never held while waiting on a guest lock; may be held while calling `rm_lock` |
| `gsp_phys_rm_lock` (existing, `booter:3671`) | all real-GSP TX/RX: queue cursors, `boot_stage.status_read_ptr`, `tx_seq_num`, probe/ladder/unload | `gsp_phys_boot_rm_unload`, `gsp_phys_rm_probe`, `gsp_phys_rmcall_run`, new `gsp_phys_backend_rpc()` | one real RPC at a time, always; **never** taken while holding `rm->stat_lock` or any guest-memory path |
| `rm->stat_lock` (`gsp_rm_emulation.c:186`) | guest status-queue writes | `gsp_rm_send` (`:452`) | build the full reply before taking it; release immediately after write; never hold across a backend wait |
| `vgpu->lock` (`nvidia-vgpu-vfio.h:99`) | per-mdev state transitions | `init_work`/`cleanup_work` | never held across claim/release or RPC waits |
| `nv_vgpu_driver.lock` | device list | create/remove, sema thread scan | never taken under session/rm locks |
| `svc.chan_lock` / `svc.vas_lock` | software work pipeline | sema service | unchanged; forwarding reads channel state copied out under these locks |

Lock order: `nv_vgpu_driver.lock` (shortest, leaf) < `gsp_phys_session_lock`
< `gsp_phys_rm_lock`. The emulation thread must take
`session_lock -> check state -> drop session_lock -> rm_lock -> re-check a
generation counter` so a release cannot deadlock against a forward in flight.

### 1.4 Failure paths

* **Guest crash / QEMU killed:** `open_count` drops; the backend stays READY
  and claimed (device owned, host headless). No device writes. The next
  `mdev` remove or `rmmod` drives the graceful path. This is the same
  `--keep` semantics the supervised harness already uses.
* **Guest RPC while STOPPING/unload already submitted:** every forwarding
  seam returns "not ready" and the emulation answer is used. This mirrors the
  defensive re-checks in `gsp_phys_rmcall_run()` (`booter:5119`).
* **Module unload order (must change):** today `nv_vgpu_exit()` calls
  `gsp_phys_gpu_release()` at `kernel/main.c:1500` *before*
  `gsp_sema_service_stop()` (`:1525`) and before mdev unregistration
  (`:1530`) which is what stops the emulation threads (`:1619`). Once
  handlers forward to the backend, release must not race live forwards:
  new order = set STOPPING (session lock) -> `gsp_sema_service_stop()` ->
  `mdev_unregister_driver()` (drains emulation) -> `gsp_phys_gpu_release()` ->
  parent/class teardown.
* **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out,
  the existing release still quiesces, runs the FWSEC teardown and refuses to
  rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit
  human approval plus a handoff update.
* **Static-config served before READY:** bounded wait (2 s, implemented) then
  pure emulation fallback, with a stat counter; never block the emulation
  thread longer than the guest's own RPC poll can tolerate.

---

## 2. Seam table

Legend: **L** = live backend, **E** = existing emulation, **H** = hybrid.
"Change point" is the exact function that must learn the new behavior.

### 2.1 Guest GSP-RM RPC class (msgq function IDs)

| # | Guest message | Today | Target | Change point (file:line) |
|---|---|---|---|---|
| 1 | `GSP_SET_SYSTEM_INFO` (72) | consumed, no reply (`gsp_rm_emulation.c:3095`) | **E** - describes the guest client at guest boot; the backend already sent its own fn72 at `booter:1559` | `gsp_rm_handle_element` `gsp_rm_emulation.c:3095` |
| 2 | `SET_REGISTRY` (73) | consumed, no reply (`:3096`) | **E** - same reason | `:3096` |
| 3 | `CONTINUATION_RECORD` (71) | SUCCESS reply (`:3105`) | **E** | `:3105` |
| 4 | `SET_GUEST_SYSTEM_INFO` (1) | echoes guest vgx version (`:982`) | **E** - this is the guest's negotiation, deliberately decoupled from the host session (see §4.3) | `gsp_rm_handle_set_guest_system_info` `:982` |
| 5 | `GET_GSP_STATIC_INFO` (65) | hard-coded TU104 model (`:1025`), 256 MB FB, fake handles, hand-tuned caps | **H** - overlay live identity/capability fields proven safe; keep model-consistent FB geometry, PDE bases and handles until phases P4/P6 (see §4) | `gsp_rm_build_static_info` `:1025`, `gsp_rm_handle_get_static_info` `:1170` |
| 6 | `GSP_RM_ALLOC` (103) | echo request + `NV_OK`; channels/events registered locally (`:2322`) | **H** - echo for emulation-only classes; forward selected classes (root/device/subdevice, then channel/memory) once a guest<->server handle map exists; always keep local side effects | `gsp_rm_handle_rm_alloc` `:2322` |
| 7 | `GSP_RM_CONTROL` (76) | ~25 static per-cmd handlers; unknown -> zeroed success (`:2611`, default `:2807`) | **H** - whitelist of side-effect-free queries goes live with emulation fallback; vGPU-specific controls stay **E** (`COPY_SERVER_RESERVED_PDES` `:2647`, `PROMOTE_FAULT_METHOD_BUFFERS` `:2654`) | `gsp_rm_handle_rm_control` `:2611`; new forward helper before the switch |
| 8 | `GSP_RM_FREE` (10) | echo + retire channel/events (`:2882`) | **H** - forward when the object was backend-allocated; always retire local bookkeeping | `gsp_rm_handle_free` `:2882` |
| 9 | `SET_PAGE_DIRECTORY` (54) | recorded for translation + SUCCESS (`:2975`) | **H** - recording stays **E**; forwarding to the real MMU is P6b and guest pages are GPU-DMA-reachable on the reference host (P6a status below; page-table pre-build required) | `gsp_rm_handle_set_page_directory` `:2975` |
| 10 | `UNSET_PAGE_DIRECTORY` (79) | record remove + SUCCESS (`:3063`) | **H** - same gate as #9 (reachability resolved; pre-build required) | `:3063` |
| 11 | `UPDATE_BAR_PDE` (70) | SUCCESS ack (`:3144`) | **E** now; **L** only when BAR1 is backed by real VRAM | `:3144` |
| 12 | `UNLOADING_GUEST_DRIVER` (47) | SUCCESS ack (`:3155`) | **E** - the guest's own shutdown; the real session's shutdown is host-driven on backend stop (`gsp_phys_gpu.c:909`) | `:3155` |
| 13 | default/unknown | generic SUCCESS (`:3169`) | **E** (fail-safe: never invent a live forward for an unknown function) | `:3169` |

### 2.2 Outbound events and replies

| # | Record | Today | Target | Change point |
|---|---|---|---|---|
| 14 | `GSP_INIT_DONE` (0x1001) | emitted as soon as the emulated Booter finishes (`gsp_rm_emulation.c:3374`) | **H** - emit immediately as today, but live answers gate themselves on READY; alternatively delay emission on a bounded backend wait so the guest's first RM RPC sees READY | `gsp_rm_emu_on_booter_done` `:3339` |
| 15 | `POST_EVENT` (0x1003) | software pump completion (`gsp_rm_post_channel_event` `:599`) | **E** as the relay mechanism; its *trigger* becomes the observation of a real hardware semaphore (P5) instead of a software op | `gsp_rm_post_channel_event` `:599`, `svc_notify_complete` `gsp_sema_service.c:4151` |
| 16 | all replies | single writer `gsp_rm_send` (`:452`) | **E** - keep as the only guest-queue writer; live payloads are copied into it | `gsp_rm_send` `:452` |

### 2.3 BAR / register and work-plane classes

| # | Guest-visible thing | Today | Target | Change point |
|---|---|---|---|---|
| 17 | GSP falcon boot regs (FWSEC/Booter/RISCV/WPR2), SW interrupt tree, MMU invalidate, L2 flushes | emulated auto-complete (`gsp_emulation.c:193-452`) | **E always** - the two GSP sessions are intentionally independent; guest boot must never depend on the physical session | `gsp_emu_poll_boot` `:193` |
| 18 | Guest VF doorbell `0x88704` | software pump (`main.c:826` -> `gsp_sema_service_doorbell` `:6369`) | **H** - pump classifies each push: pure state/release stays software; engine work is forwarded (P5) | `svc_pump_channels` `gsp_sema_service.c:5019`, `svc_decode_stream` `:2802` |
| 19 | BAR0/BAR1/BAR3 shadows, PRAMIN window | host-RAM shadows (`bar_emulation.c:180`, `main.c:1205`) | **E** until real VRAM forwarding (P6) | `nv_vgpu_bar_emul_*`, `nv_vgpu_vfio_dev_mmap` |
| 20 | legacy doorbell/shared-mem RPC class (`GSP_RPC_*`) | internal emulation client thread (`gsp_emulation.c:458-620`, `:674-694`) | **E / disable when READY** - not part of the guest physical-path protocol; gate it off in backend mode to avoid intercepting stray writes | `gsp_emu_thread_fn` `:674` |

**Highest-leverage seams:** #5/#7 (one reply point, one dispatch point; all
live answers plug in there) and #6 (whatever alloc/free semantics the guest
gets determine whether real channels/page tables are possible).

---

## 3. Lifecycle changes in `main.c` / `gsp_emulation.c`

1. **New backend API** (add to `gsp_phys_booter.[ch]`, used by both debugfs
   and the guest plane; keeps `gsp_phys_rm_lock` private):
   * `int gsp_phys_backend_state(void)` - OFF/CLAIMING/BOOTING/READY/STOPPING.
   * `int gsp_phys_backend_rpc(u32 function, const void *req, u32 req_len,
     void **reply, u32 *reply_len, u32 timeout_ms)` - checks
     `claimed && rm_init_done && !rm_unload_submitted`, takes
     `gsp_phys_rm_lock`, calls `gsp_phys_queue_tx_submit()` +
     `gsp_phys_status_wait()` (both at `booter:2935`/`:2670`), returns a copy
     of the payload; caller frees. Unmatched records (events, NOCAT) are
     relayed or summarized like `gsp_phys_status_wait` already does.
   * `int gsp_phys_backend_static_snapshot(u8 *out, u32 len)` - cached full
     1656-byte fn65 payload; filled once at READY by the existing
     `gsp_phys_rm_probe_cycle()` (`booter:4056`) under the same lock.
   * `void gsp_phys_backend_begin_stop(void)` / `gsp_phys_backend_end_stop()` -
     state transitions only (no hardware).
2. **New session state + refcount** in `gsp_phys_gpu.c` (new mutex, owner
   flag, state enum). `gsp_phys_gpu_claim()`/`_release()` stay the only
   hardware entry points; add `gsp_phys_backend_start()/stop()` that call
   them exactly once.
3. **`nv_vgpu_mdev_create()` (`main.c:535`)**: after
   `vfio_register_emulated_iommu_dev()` (`:591`) and before/around
   `schedule_work(&vgpu->init_work)` (`:598`), call
   `gsp_phys_backend_start()`. Claiming must happen while no VM holds a live
   mdev: with `available_instances` computed from per-type max (1) in
   `nv_vgpu_get_available()` (`main.c:81`) there is at most one mdev of the
   type in flight, so this holds.
4. **`nv_vgpu_device_init_work()` (`main.c:1553`)**: keep BAR/IRQ/memory/
   `gsp_emu_init()` order. The backend boot runs in its own kworker so
   `mdev` create returns quickly; `init_work` only observes READY when it
   initializes forwarding tables. Do **not** run the 60 s `rmwait`
   (`booter:2764`, timeout `GSP_PHYS_RM_INIT_TIMEOUT_MS` 60000 at
   `booter:446`) inline in `init_work` if it can be avoided: run claim+boot
   in a dedicated work item so `mdev->remove` is never stuck behind it.
5. **`nv_vgpu_device_cleanup_work()` (`main.c:1614`)**: unchanged per-mdev,
   but the last-cleanup path calls `gsp_phys_backend_stop()` **after**
   `gsp_emu_cleanup()` (`:1619`) so no emulation thread can forward while the
   backend is being torn down.
6. **`nv_vgpu_exit()` (`main.c:1487`)**: reorder as in §1.4; the load-time
   claim path (owner = module) keeps the existing `gsp_phys_gpu_release()`
   position relative to debugfs fini, but mdev-owned sessions are released in
   the new order.
7. **`gsp_emu_init()` (`gsp_emulation.c:724`)**: no structural change; the
   `gsp_rm_emu` context gains a reference to the module backend (not a new
   per-mdev session). `gsp_emu_cleanup()` (`:831`) unchanged except it must
   not be the thing that releases the backend (done by `main.c`, step 5).
8. **`gsp_rm_emu_on_booter_done()` (`gsp_rm_emulation.c:3339`)**: after
   linking the queues, optionally wait (bounded) for READY before emitting
   INIT_DONE, or emit immediately and let each live handler wait. Recommended:
   emit immediately (the guest's early RPCs are all emulatable), and let
   `GET_GSP_STATIC_INFO` be the single bounded-wait point.
9. **`gsp_emu_thread_fn()` (`gsp_emulation.c:651`)**: no new blocking calls in
   the loop; live forwards happen inside `gsp_rm_handle_element()` on that
   thread, with the 2 s per-RPC bound. If a longer wait is ever needed, move
   forwarding to a dedicated backend kthread and keep this loop responsive.

---

## 4. Static-config flow and identity/version negotiation

### 4.1 Current guest model (must stay internally consistent)

`gsp_rm_build_static_info()` (`gsp_rm_emulation.c:1025`) hard-codes:
`fb_length = 0x10000000` (256 MB, offset 1224), `fbio_mask=0xff`,
`fbp_mask=0x3f`, `gpuNameString="NVIDIA GeForce RTX 2080 SUPER"` (1260),
`gpuShortNameString="TU104"` (1324), `bar1PdeBase=bar2PdeBase=0`
(1536/1544), fake internal handles (1600/1604/1608), engine caps
`0x60803E02 / 0x00362020 / 0` (1204). The 256 MB figure is coupled to the
BAR0 shadow's `LOCAL_MEMORY_RANGE` (`bar_emulation.c:280`,
`0x100ce0=0x18` -> 256 MB) and to the emulated WPR2/FRTS location the guest
driver checks. Changing it without changing the shadow breaks the guest boot
(this is why 512 MB was reverted, per the comment at `:1096`).

### 4.2 Live values observed (cycle 13)

```
static-info internal: hClient=0xc2000006 hDevice=0xabcd0080 hSubdevice=0xabcd2080
static-info fb: bar1PdeBase=0x1f7650000 bar2PdeBase=0x1f76c2000
               fb_length=0x200000000 fbio_mask=0xf fbp_mask=0x11
               busWidth=0 ramType=256 l2CacheSize=15
static-info gpu: name='NVIDIA GeForce RTX 2080 Super with Max-Q Design'
                short='TU104-B' vbiosSubVendor=0x4d50 vbiosSubDevice=0x4449
static-info engineCaps=[0x08083e02,0x00000044,0x00000000]
```

Two grounded warnings:

* The decoded `busWidth=0`, `ramType=256` are not plausible TU104 values
  (256-bit, GDDR6). Either the decode offsets for those two fields do not
  match the real struct, or the emulated values were hand-derived rather than
  observed. **A/B every field before overlaying it.**
* Engine caps differ materially (`0x60803E02` vs `0x08083E02`, and the second
  word). The emulation deliberately sets extra bits (COPY14 etc.) to make
  `libcuda` accept the engine inventory. Overlaying live caps would change
  the guest's engine/class topology and can regress the whole CUDA init.
  Keep emulated caps until the backend can actually service the resulting
  demands (P4+).

**P2 status (2026-10-02):** the name, short-name and VBIOS identity fields are
now overlaid live behind `gsp_phys_static_live` (A/B verified); the FB
geometry, masks and engine caps above remain emulated, per these warnings,
until P4/P6.

### 4.3 Proposed flow

1. At READY, cache the full fn65 payload (new bytes in the backend session,
   keyed by the loaded firmware's `phys.fw_crc32`, `gsp_phys_gpu.c:272`), via
   the existing probe cycle (`booter:4056`) under `gsp_phys_rm_lock`.
2. `gsp_rm_handle_get_static_info()` (`:1170`) keeps building the emulated
   1656-byte model, then applies an allow-listed overlay from the snapshot
   (start: name/short-name/VBIOS identity/L2 if A/B passes; **not** FB
   geometry, masks, PDE bases, handles, engine caps).
3. Any overlay field is independently switchable (`gsp_phys_static_live`
   module parameter, default off) so a regression is a one-line rollback.
4. Handles move in P4 and PDE bases/geometry in P6, each behind its own
   switch and its own A/B.
5. Version negotiation stays decoupled: the guest sees the emulation's
   version answer (`SET_GUEST_SYSTEM_INFO`, `gsp_rm_emulation.c:982`); the
   real server is negotiated by the host as the host client
   (`gsp_phys_rm_probe_fn1`, `booter:3984`, vgx 0x29/0x0C). The guest must
   never learn the physical firmware's version; the emulation target remains
   570.144 while the host may boot a different, matching TU10x GSP image
   (`gsp_phys_gpu.c:75`). The wire structures are the same family and the
   live snapshot is keyed by firmware CRC.

---

## 5. Phased implementation plan (smallest first)

Each phase is independently revertible. "Supervised" means the documented
harness discipline: `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`, human
go-ahead, recovery plan, pstore, never unattended
(`docs/03-architecture/08-physical-gpu-integration.md` §7/§8).

### P0 - Backend API skeleton (inert)
* **Status (2026-10-02): implemented.** Session state/refcount plus the locked
  backend RPC and static-snapshot helpers are in the tree and refuse when the
  backend is OFF/not-init-done/unloading.
* **Scope:** new state/refcount/lock in `gsp_phys_gpu.c`; locked
  `backend_rpc()`/`static_snapshot()` in `gsp_phys_booter.c`; no caller
  changes; no behavior change.
* **Acceptance:** `make kernel` clean; load/unload with no params; all
  existing debugfs triggers behave identically; new API logs and refuses when
  OFF/not-init-done/unloading.
* **Risk:** none (never sends unless a live init-done session exists).
* **Rollback:** trivial.

### P1 - Lifecycle wiring (claim on mdev create, release on last remove)
* **Status (2026-10-02): implemented and exercised live; fail-open.** The
  first mdev create starts the claim/boot asynchronously and the last remove
  releases it. The async boot currently fails at FWSEC on this path
  (`staging not visible before kick`) and the READY gate releases the claim
  fail-open; the module-owned live session is the proven path (and what P2
  uses).
* **Scope:** `main.c` create/remove/init_work/cleanup_work/exit ordering
  (§3 items 3-6); no live answers yet.
* **Acceptance (supervised, harness `phys_claim_cycle.sh` style):** fresh
  boot -> insmod params -> create mdev -> one real boot log sequence (FWSEC,
  Booter M0=0, RISC-V active, init-done after sequencer) -> guest boots and
  the CUDA driver-API smoke test still prints `ALL OK` 64/64 -> stop VM ->
  remove mdev -> `UNLOADING_GUEST_DRIVER` suspend, scratch14 bits 24-27 DONE,
  WPR2 down, distribution driver rebound; repeat twice; zero power-domain
  cycles; no reboot.
* **Risk:** **highest phase.** Claim is now triggered by a sysfs action, not
  by module load. The two-param gate and the D0 lock must already be active
  inside claim (they are, `gsp_phys_gpu.c:670/700`), the claim must refuse a
  second owner, and the release must never run while a forward is in flight
  (STOPPING gate). Any power-loss signature -> hold locks, no retry.
* **Rollback:** `gsp_phys_gpu=0` (no claim) or load-time `gsp_phys_gpu=1`
  only; `mdev` emits `-EOPNOTSUPP` from `backend_start()`.

### P2 - Static-config identity overlay
* **Status (2026-10-02): implemented and verified live (A/B).** The READY-time
  snapshot cache and the `gsp_phys_static_live` parameter (bool, default 0)
  are in the tree. With the overlay on, the guest `nvidia-smi -q` Product
  Name is the live `NVIDIA GeForce RTX 2080 Super with Max-Q Design`; with it
  off it is the emulated `NVIDIA GeForce RTX 2080 SUPER`; the driver-API
  smoke stays green (`ALL OK`, payload 64/64). Only name, short name and
  VBIOS identity are overlaid; FB geometry, masks, engine caps, PDE bases and
  handles stay emulated.
* **Scope:** cache snapshot at READY; overlay allow-list in
  `gsp_rm_build_static_info()`/`gsp_rm_handle_get_static_info()`; module
  parameter; stats. Bounded wait (2 s) then emulation fallback.
* **Acceptance:** guest still `nvidia-smi -L` + smoke `ALL OK`; guest
  `nvidia-smi -q` name changes to the live `... with Max-Q Design`; dmesg
  shows which fields came live; backend absent -> byte-identical guest reply.
* **Risk:** a consumed identity field changes a driver decision (VBIOS
  validation is the likeliest). Mitigation: field-by-field A/B, default-off,
  instant fallback; never touch geometry/caps/handles in P2.
* **Rollback:** parameter off.

### P3 - Read-only control forwarding (whitelist)
* **Status (2026-10-02): implemented and verified live (default off).** With
  `gsp_phys_ctrl_forward=1` the whitelist (`BUS_GET_PCI_BAR_INFO`,
  `BUS_GET_INFO_V2`, `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) is
  forwarded over fn76 with the guest's emulated internal handles translated
  to the snapshot's adopted server handles; the server's echoed params become
  the guest reply, and everything else falls back to the emulated handler
  (fail-open; counters forwarded/fallback/timeout). Live acceptance: guest
  smoke GREEN `ALL OK` 64/64, counters ok=22 fallback=2 timeout=0, guest
  `nvidia-smi -q` worked.
* **Scope:** in `gsp_rm_handle_rm_control()` (`:2611`), before the per-cmd
  switch: for an allow-list of side-effect-free queries (FB/BUS/KGR/GMMU/
  CE/display statics), call `backend_rpc()` with the **adopted internal
  handles** from the snapshot (the guest's emulated handles are not valid on
  the server yet), copy reply params (payload+24) on `rpc_result==0 &&
  status==0`, else fall back to the existing handler. Add per-cmd
  forwarded/fallback counters.
* **Acceptance:** guest boot and smoke green with forwarding on; dmesg shows
  N forwarded controls; forced backend timeout -> same behavior as today;
  repeat calls yield identical params (no server-side state change).
* **Risk:** a whitelist entry is not actually read-only (display/KGR can
  allocate on first call). Start with FB/BUS/GMMU static info only, verify
  against the server's log ring (`gsp-phys-log-dump`), then grow.
* **Rollback:** parameter/whitelist off.

### P4 - Alloc/free forwarding for the object ladder
* **Scope:** `gsp_rm_handle_rm_alloc()` (`:2322`) forwards
  root/device/subdevice (and only those, first) via `backend_rpc(fn103)`;
  returns the server's chosen handle to the guest; maintains a
  guest-handle -> server-handle map used by P3 and `gsp_rm_handle_free()`.
  Any failure falls back to today's echo (never half-forward), and local
  channel/event registration still runs.
* **Acceptance:** guest boot + smoke green; dmesg shows adopted handles
  (`0xc2.../0xabcd0080/0xabcd2080`) matching cycle13; guest FREE returns
  success; no server-side wedge (no RPC timeout after alloc).
* **Risk:** the guest's alloc params may not satisfy the server's per-class
  validation; a bad forward can wedge the RM task (see the `0x40` GSP-owned
  channel incident). Guard: one class at a time, abort on first failure,
  `UNLOADING_GUEST_DRIVER` + rmmod remains the only fail-safe; never retry a
  mutating RPC.
* **Rollback:** parameter off (guest returns to echo path).

### P5 - Work forwarding + completion relay (interim, host buffers)
* **Status (2026-10-02): wave 1 implemented and verified live (default off).**
  With `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE
  pushes on the physical engine through one ordered work item running the
  module-owned ladder copy; the software release path stays authoritative and
  no guest memory/VA is touched. Live: guest smoke GREEN `ALL OK` 64/64 with
  514 real CE copies (194 requests coalesced/dropped while one was in flight,
  0 refusals, 0 timeouts). Wave 2 (hardware-primary release relay) and P6
  remain.
* **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25:
  channel allocates, page directory commits, `REAL COPY OK` on the physical
  CE in VA mode). The remaining work: bind each guest CE/GR channel to a real
  channel; in `svc_pump_channels()` (`gsp_sema_service.c:5019`) classify the
  decoded push and, for forwarded ops, submit an equivalent real method block
  built by the existing `gsp_phys_rmcall_build_methods()` (`booter:4879`) +
  GPFIFO/USERD/doorbell (`:4991-5032`). Observe the real semaphore, then
  mirror the release into guest memory with `svc_sem_write()`
  (`gsp_sema_service.c:1514`) and post the guest event with
  `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`). All hardware
  operands are module-owned DMA buffers (as in the ladder), so no guest memory
  is touched by the engine yet.
* **Acceptance:** ladder: `echo 4 > gsp-phys-rm-call` logs
  `REAL COPY OK` (`booter:5052`) and src==dst after a hardware copy - met
  (cycle 25, 4 KiB, semaphore released, dst==src). Guest: a submitted copy
  completes with the semaphore observed in host memory and the guest tracker
  advancing; wrong-operand/refused submissions fall back to software and are
  counted.
* **Risk:** doorbell to a wedged channel; wrong GPFIFO/USERD (guards:
  `rmcall_channel_server_mem` refusal `:4944`, work-submit token after
  `GPFIFO_SCHEDULE` `:4977`, `GP_PUT` bounds `:4995`); hardware writes only
  host DMA memory, so no cross-tenant exposure yet.
* **Rollback:** per-channel `forward=0`; software pump is untouched.

### P6 - Guest memory into the real MMU (end state: guest submission on hardware)
* **Status (2026-10-02): P6a - guest-page reachability proven live.** The
  physical GPU's IOMMU group is an identity domain on the reference host and
  the mediated device registers as an emulated-IOMMU VFIO device, so its
  type-1 container keeps guest DMA mappings privately and no kernel
  `iommu_domain` exists for it: `iommu_iova_to_phys` cannot translate guest
  GPAs. Guest RAM is therefore backed by a QEMU shared file
  (`memory-backend-file,share=yes,prealloc=on`); the module opens it and
  translates GPA->HPA through the file page cache (`read_cache_page` +
  `page_to_phys` + `pfn_valid`) behind the active-mdev gate, under the new
  default-off `gsp_phys_guest_mem` parameter and write-only
  `gsp-phys-guest-ram`/`gsp-phys-guest-gpa` debugfs controls (ladder trigger
  cases 6 host->guest write, 7 guest->host read, 8 fresh-slot PTE self-test).
  On the reference q35 4 GiB guest, GPA equals file offset below 4 GiB and
  offset = GPA - 0x80000000 above it (2 GiB 32-bit PCI hole). **Pre-build
  requirement:** channel page-table slots for the guest window are built in
  ladder stage 3 before any engine access; a runtime PTE write into a line
  the GMMU had already walked was accepted but never completed (no
  TLB-invalidate path yet). **Live:** the physical copy engine copied 4 KiB
  from a real guest RAM page into a module buffer (G2H) and back into the
  same page (W2G), semaphore released; the guest process observed the write
  in its own page; the driver-API smoke stayed green (`ALL OK` 64/64) and
  release stayed latched (distribution driver rebound, protected region down,
  exactly 5 power-domain locks intentionally held, 0 rail cycles, no reboot);
  the informational PTE self-test passed. **Remaining (P6b):** page-directory
  shadowing, real guest channels and push forwarding - guest-submission
  forwarding is not implemented yet.
* **Scope:** map guest page-table pages/records so the physical GPU can DMA
  them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real
  server; allocate real memory objects for guest allocations (fn4 with real
  PFNs); per-guest-channel real channels with client rings (module or guest
  memory); pump forwards real push streams referencing guest VAs; completion
  and faults relayed through real MSI-X -> module IRQ -> existing eventfd
  injection (`gsp_emu_inject_completion_interrupt`, `gsp_emulation.c:626`;
  the physical driver does not `request_irq()` yet, `gsp_phys_gpu.c:380-461`).
* **Acceptance:** `vectorAdd` correct results through a forwarded launch;
  copy with semaphore; second mdev is refused/emulated (single-session
  isolation until per-mdev clients exist); fault test.
* **Risk:** wrong page tables -> engine writes to arbitrary host memory;
  guest-page reachability is **resolved on the reference host** (P6a status
  above; §2.2 of `docs/04-implementation/01-known-knowns-and-unknowns.md` now
  carries the resolution note), so wrong-page-table writes remain the top
  risk. Keep `gsp_phys_seq_addr_valid()` (`booter:1800`) and all queue
  validations.
* **Rollback:** per-mdev isolation not yet available; disable forwarding and
  fall back to P5 host-buffer mode.

---

## 6. Top risks and the guards that must be preserved

1. **Host hard-freeze on unbind/rebind.** Unbinding the distribution driver
   can power-gate the board; re-bind intermittently fails and the host
   freezes with no crash record. Guards: two-param gate
   (`gsp_phys_gpu.c:670`), refuse if GPU/root port already
   runtime-suspended (`:688`), D0 lock on GPU + siblings + root port
   (`:499-549`), runtime-PM usage ref (`:188-204`), config-space liveness
   checks after claim and before rebind (`:591`), no retry after a
   power-loss signature, human approval for any reboot.
2. **WPR2 held / one-way protected region.** Always run the release sequence
   in order: `UNLOADING_GUEST_DRIVER` (only if init-done was seen)
   -> quiesce GSP falcon -> FWSEC teardown -> free staging
   (`gsp_phys_gpu.c:909-914`); if WPR2 survives, do **not** rebind and keep
   the locks (`:939`); a FWSEC-only stop is forbidden
   (`gsp_phys_gpu.c:770`). The debugfs `gsp-phys-unload-rpc` remains
   write-only and inert outside a live claim.
3. **RM-task wedge (live 0x40 incident).** Setting the GSP-owned channel
   `internalFlags` bit (0x40) wedged GSP-RM: the alloc RPC never replied and
   every later RPC timed out; the fail-safe held the device until reboot.
   The working tree now bans that bit and refuses the submission when the
   winning variant used server-allocated memdescs (`booter:281`, `:4714`,
   `:4942`). Never re-enable it; abort the ladder at the first failure with
   no object teardown (`:5123` comments); `UNLOADING_GUEST_DRIVER` + rmmod is
   the only recovery.
4. **Wrong-page-table / wrong-doorbell writes.** No forwarding seam may
   submit engine work or program page tables until the P6 page-table
   population path is proven end to end; guest-page reachability itself is
   resolved on the reference host (P6a), so wrong-page-table writes remain
   the top risk. Preserve: queue header validation in
   `gsp_phys_queue_tx_submit()` (`booter:2983`), free-space check (`:3032`),
   status-record validation (`booter:2554`), sequencer address validation
   (`booter:1800`), stage-4 token/bounds guards (`booter:4969-4999`).
5. **Guest status-queue invariants.** Strict sequence/wrap/checksum and
   multi-element records; `gsp_rm_send()` (`:452`) is the only writer and
   must not be called while holding `gsp_phys_rm_lock` or waiting on the
   backend; the event-burst cap (`:488`) and parse retry backoff (`:3397`)
   stay.
6. **Two planes, two BARs.** The guest face must only ever touch
   `gsp_emu_context.bar0_vaddr` (the vzalloc shadow,
   `gsp_emulation.c:761-778`); the backend must only ever touch `phys.bar0`
   (real ioremap, `gsp_phys_gpu.c:420`). Never ioremap a guest GPA (the
   fixed `gsp_client_init_emulation()` contract).
7. **Ordering and context rules.** No long scans or backend waits in VFIO
   callbacks; `vfio_dma_rw` only from kthread context (mm == NULL);
   never `rmmod` with a live VM; mdev remove before module unload; stop
   emulation before backend release once P2+ lands (new `nv_vgpu_exit`
   order).
8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX
   seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only
   `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree
   (`backups/evidence-*`) current on every live cycle.

---

## Appendix A - line anchor map (function -> file:line)

`kernel/gsp_phys_booter.c` (working-tree revision used for this design):
`gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init`
2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670,
`gsp_phys_boot_rmwait` 2764, `gsp_phys_queue_tx_submit` 2935, `gsp_phys_rm_lock`
3671, `gsp_phys_boot_rm_unload` 3673, `gsp_phys_sci_decode` 3820,
`gsp_phys_rm_probe_fn65` 3918, `fn1` 3984, `_cycle` 4056, `gsp_phys_rm_probe`
4086, `gsp_phys_rmcall_rpc` 4329, `_alloc` 4385, `_register_mem` 4454,
`_control` 4510, `_stage1` 4575, `_stage2` 4650, `_channel_variants` 4714,
`_stage3` 4761, `_build_methods` 4879, `_stage4` 4923, `_stage5` 5068,
`_run` 5108, `_fini` 5197, debugfs write handlers 5613/5681/5754,
`gsp_phys_boot_debugfs_init` 5827, `gsp_phys_boot_quiesce` 5919,
`gsp_phys_boot_staging_free` 5944.

`kernel/gsp_emulation.c`: thread 651, init 724, cleanup 831, poll_boot 193,
rm poll call 672, legacy RPC dispatch 458, MSI-X inject 626.

`kernel/gsp_rm_emulation.c`: `gsp_rm_send` 452, parse 806, link 917,
`set_guest_system_info` 982, `build_static_info` 1025, `get_static_info` 1170,
`rm_alloc` 2322, `rm_control` 2611 (default 2807), `free` 2882,
`set_page_directory` 2975, `unset` 3063, `handle_element` 3084,
`poll_commands` 3194, `on_booter_done` 3339, `poll` 3386.

`kernel/main.c`: `nv_vgpu_mdev_create` 535, `remove` 614, init_work 1553,
cleanup_work 1614, `nv_vgpu_init` 1388, claim 1450, debugfs init 1461,
sema start 1468, `nv_vgpu_exit` 1487, release 1500, sema stop 1525,
mdev unregister 1530, param 2457.

`kernel/gsp_phys_gpu.c`: claim 662, hazard gate 670, suspend refusal 688,
power lock 523/499, unbind 706, driver register 718, board check 591,
runtime ref 188, boot 745-848 (FWSEC 792, prime 799, run 805, rmwait 811),
release 876 (unload 909, quiesce 910, fwsec 911, staging free 914, WPR2
fail-safe 939, board check 955, rebind 972).

`kernel/gsp_sema_service.c`: doorbell 6369, start 6392, stop 6416, thread
6232, pump 5019, decode 2802, sem write 1514, notify 4151.