summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-01 20:58:32 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-01 20:58:32 -0400
commitc50eacb25fdc43826aa631aa1fc20ef66774882f (patch)
tree0624a2e4ef21beb72cf0aee92ab3f733f758f993
parente3feaa8f237150735caef56254568e0a0563021a (diff)
downloadopen-iov-c50eacb25fdc43826aa631aa1fc20ef66774882f.tar.gz
open-iov-c50eacb25fdc43826aa631aa1fc20ef66774882f.zip
STATE-CURRENT + docs: first real copy executed by the physical CE; add the physical-backend guest-wiring design
- STATE-CURRENT snapshot 2026-10-01 21:05: the host RM-client ladder now allocates VAS+PDB+TSG+channel and executes a real 4 KiB VA-mode CE copy (semaphore released, dst==src) on the physical engine; next is wiring the live session into the guest-facing mdev/vfio path. - docs/03-architecture/11-physical-backend-guest-wiring.md: phased design (P0-P6) for bridging the physical backend into the guest-facing emulation, with the seam table, lifecycle changes and static-config flow. - docs/README.md: reading order updated for the new document; execution status refreshed.
-rw-r--r--STATE-CURRENT.md66
-rw-r--r--docs/03-architecture/11-physical-backend-guest-wiring.md575
-rw-r--r--docs/README.md38
3 files changed, 630 insertions, 49 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index f68fcc0..9c417e3 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
# Open-IOV Current State
-> **Snapshot: 2026-10-01 20:12.** This is a dated public status page, not live
+> **Snapshot: 2026-10-01 21:05.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
@@ -60,14 +60,16 @@ meant to execute on the physical GPU.
cleanly with no reboot. Since 2026-10-01 17:24 the next rung has both code
and live results: a host resource-manager client creates memory and
channel-group objects on the booted server, allocates and programs the
- page-directory base of a virtual-address-space object, and **allocates a real
- channel** on the live server. It has since filed the **first real submission
- to the physical engine**: a copy work item was placed in the channel's GPFIFO
- and the doorbell was rung, but the copy engine did not execute it within the
- timeout, so no completion was observed - getting the physical channel to run
- submitted work is the current wall. One unsafe channel variant hung the server
- task earlier, so the fail-safe held the device and an approved reboot restored
- it; the variant is removed from the tree. |
+ page-directory base of a virtual-address-space object, and **allocates a real
+ channel** on the live server. In cycles 18-25 it went all the way to
+ execution: the channel was bound and scheduled, a work-submit token was
+ obtained, a copy work item was placed in its GPFIFO and the doorbell rung,
+ and after the engine's fetch and method-path faults were diagnosed and fixed
+ the physical copy engine ran it - a 4 KiB copy with the semaphore released
+ and destination equal to source. Every cycle released the device cleanly
+ with no reboot; the next rung is wiring the live physical session into the
+ guest-facing mdev/vfio path
+ ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
@@ -79,26 +81,24 @@ chain succeeds.
## Right now (frontier)
-- **Submission-rung bring-up (2026-10-01 17:24-20:12): the first real work
- submitted to the physical engine.** A host resource-manager client now talks
- to the booted server and the channel-allocation wall has fallen: ten
- system-memory page-table registrations and a channel-group object are created
- under the server's own internal handles, a virtual-address-space object is
- allocated under the device handle, its page-directory base is programmed (the
- earlier rejection is fixed), and a real copy-engine channel is allocated. In
- the latest cycle the client bound the channel, scheduled it, obtained a
- work-submit token, placed a 30-word copy work item in the channel's GPFIFO and
- **rang the doorbell** - the first real submission to the physical engine.
- The copy engine did **not** execute the work within the timeout (no semaphore
- release, the channel made no progress), so completion is not yet observed;
- making the physical channel run submitted work is the new wall. The
- root-client allocation path remains parked after an invalid-client result. One
- channel variant hung the server's resource-manager task; the fail-safe held the
- device instead of handing it back in a wedged state, and an approved reboot
- restored the distribution driver (~18:37). The unsafe variant has been removed
- from the tree, and a decoded resource-manager log dump makes server-side
- failures readable. Evidence: `backups/evidence-20261001-cycle19/` through
- `...-cycle21/`.
+- **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes
+ on the physical engine.** The host resource-manager client allocated a
+ virtual-address-space object with a committed page directory, a channel-group
+ object and a real copy-engine channel, bound and scheduled it, obtained a
+ work-submit token, and filed a copy work item in the channel's GPFIFO. The
+ engine's first fetches faulted; draining and decoding the status queue then
+ exposed the wrong virtual address in the fetch, and with that fixed the
+ physical copy engine executed the block: `REAL COPY OK sem=0xc0ffee01
+ src=0x1170d5000 dst=0x11ccbd000 bytes=4096 GPGet=1 GPPut=1 words=26
+ token=0x00000001` - a 4 KiB copy with the semaphore released and destination
+ equal to source. Every cycle ran under the supervised harness and released the
+ device cleanly (server shutdown, protected region down, distribution driver
+ rebound); no reboot was needed and the host stayed healthy. The next rung is
+ wiring this live session into the guest-facing mdev/vfio path so guest
+ submissions execute on the physical engine (phased plan:
+ [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)).
+ Evidence: `backups/evidence-20261001-cycle18/` through `...-cycle25/`, with
+ per-wall reports in `backups/evidence-20261001-reports/next/`.
- **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM
completes initialization on the real TU104 and serves the static-config
RPC.** Three supervised cycles, no reboot, clean release each time:
@@ -423,9 +423,11 @@ Work execution (host service vs client synchronization):
launch; the queued compute pushes decode but are not executed
[partial] GPU VA -> guest PA translation: page-table walk fixed and succeeding;
server-owned page-table population still in progress
- [partial] real work submitted to the physical GPU: a copy work item was placed
- in a real channel's GPFIFO and the doorbell rung; the engine did not
- yet execute it (no completion)
+ [done] real work executes on the physical GPU: the host RM client's copy
+ engine block runs on the physical CE - 4 KiB copied, semaphore
+ released, destination equals source
+ [not yet] guest-submitted work relayed to the physical engine (next rung:
+ wire the physical session into the mdev/vfio path)
[not yet] correct results (vectorAdd)
```
diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md
new file mode 100644
index 0000000..da440eb
--- /dev/null
+++ b/docs/03-architecture/11-physical-backend-guest-wiring.md
@@ -0,0 +1,575 @@
+# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session
+
+> **Status: design for implementation (2026-10-01). No code from this design is
+> in the tree yet.** The physical backend itself is live: the module-owned
+> GSP-RM completes initialization on the real TU104, serves the full
+> static-config reply, and shuts down gracefully. The host resource-manager
+> client ladder has since allocated memory, a virtual-address-space object with
+> a committed page directory, a channel group and a real copy-engine channel,
+> and has executed the **first real work on the physical engine** — a 4 KiB
+> copy in virtual-address mode with the semaphore released and destination
+> equal to source (cycle 25). What this document specifies is the remaining
+> bridge: wiring that live session into the guest-facing mdev/VFIO path so that
+> guest submissions execute on the physical GPU. It extends
+> [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which
+> chose the module-owned-GPU mechanism, and
+> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md),
+> which describes the emulation plane it must not disturb.
+>
+> The design was written against the tree at commit `f8e5d3b`; line numbers are
+> advisory and have moved with later commits — anchor by function name when
+> re-grepping. Behaviour claims below are grounded in live evidence from the
+> 2026-10-01 cycles; anything not yet exercised is labelled "not yet run".
+
+---
+
+## 0. Ground truth: the two planes that exist today
+
+### 0.1 Guest-facing emulation plane (works, green)
+
+* One emulation context per mdev: `nv_vgpu_device.gsp_comm_ctx`
+ (`kernel/include/nvidia-vgpu-vfio.h:115`), created by
+ `gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from
+ `nv_vgpu_device_init_work()` (`kernel/main.c:1553`).
+* The emulation thread (`gsp_emu_thread_fn`, `kernel/gsp_emulation.c:651`)
+ does four things: samples the guest's BAR0 *shadow* falcon registers
+ (`gsp_emu_poll_boot`, `:193`), polls the guest's sysmem msgq
+ (`gsp_rm_emu_poll`, `kernel/gsp_rm_emulation.c:3386`), serves the legacy
+ doorbell/shared-memory RPC class (`gsp_emu_dispatch_rpc`, `:458`), and
+ injects MSI-X through the live eventfd (`gsp_emu_inject_completion_interrupt`,
+ `:626`).
+* The guest's real 570.144 physical-path protocol (msgq over *guest* system
+ memory, vfio_dma_rw transport) is handled entirely in
+ `gsp_rm_emulation.c`: queue-chain parse (`:806`), status-queue link
+ (`:917`), command poll (`:3194`), per-function dispatch (`:3084`),
+ reply construction/transmission (`gsp_rm_send`, `:452`).
+* `GSP_INIT_DONE` is emitted from `gsp_rm_emu_on_booter_done()` (`:3339`)
+ when the *emulated* Booter kick completes (`kernel/gsp_emulation.c:416-431`).
+* Static configuration is a hand-built TU104 model:
+ `gsp_rm_build_static_info()` (`:1025`) with 256 MB FB, fake
+ `TU104_H_INTERNAL_{CLIENT,DEVICE,SUBDEVICE}` handles and hand-tuned engine
+ caps; `gsp_rm_handle_get_static_info()` (`:1170`) sends it.
+* Work is executed in software by the sema service: doorbell entry
+ `gsp_sema_service_doorbell()` (`kernel/gsp_sema_service.c:6369`) called from
+ the VFIO write path (`kernel/main.c:826`), pump
+ `svc_pump_channels()` (`gsp_sema_service.c:5019`), decode
+ `svc_decode_stream()` (`:2802`), release apply `svc_sem_write()` (`:1514`),
+ guest event post `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`).
+
+### 0.2 Physical backend plane (alive, queryable, not wired to the guest)
+
+* Ownership: `gsp_phys_gpu_claim()` (`kernel/gsp_phys_gpu.c:662`) requires
+ both `gsp_phys_gpu=1` (`kernel/main.c:2457`) and `gsp_phys_hazard_ack=1`
+ (`kernel/gsp_phys_gpu.c:670`), locks the whole board + root port into D0
+ (`gsp_phys_power_lock()`, `:523`; runtime-PM ref `:740`), unbinds the
+ distribution driver (`:706`), binds `open-iov-gsp-phys` (`:718`) and verifies
+ config-space liveness (`gsp_phys_board_check()`, `:591`).
+* Boot: claim runs FWSEC (`:792`), staging `gsp_phys_boot_prime()`
+ (`booter:1349`), Booter Load + RISC-V `gsp_phys_boot_run()` (`booter:1524`,
+ pre-boot fn72/73 at `:1559-1568`), then init-done poll
+ `gsp_phys_boot_rmwait()` (`booter:2764`, called `gsp_phys_gpu.c:811`).
+ A full boot has run live multiple times; init-done with `rpc_result=0` and
+ the full 1656-byte static-config reply were observed (cycle 13).
+* Transport: `gsp_phys_queue_tx_submit()` (`booter:2935`) for TX,
+ `gsp_phys_status_queue_*` (`booter:2467/2490/2554`) +
+ `gsp_phys_status_wait()` (`booter:2670`) for RX. All real-GSP traffic is
+ serialized by the single static mutex `gsp_phys_rm_lock`
+ (`booter:3671`), taken by `gsp_phys_boot_rm_unload()` (`:3685`),
+ `gsp_phys_rm_probe()` (`:4090`) and `gsp_phys_rmcall_run()` (`:5117`).
+ `gsp_phys_queue_tx_submit()` and `gsp_phys_status_wait()` themselves take
+ no lock and are not safe to call concurrently.
+* Host RM-client ladder (debugfs only, `booter:5754` write handler):
+ stage 1 root/device/subdevice (`:4575`), stage 2 memory registration
+ (`:4650`), stage 3 TSG + channel variants (`:4761`; variant matrix `:4714`),
+ stage 4 real CE submission (`:4923`), stage 5 adopt static-config handles
+ (`:5068`). Stage 4 now runs end to end: the channel allocates, the page
+ directory is committed, and a 4 KiB virtual-address-mode copy executes on
+ the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25).
+* Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends
+ `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2
+ (`:911`), frees staging (`:914`), and only rebinds the distribution driver
+ if the board answers (`:955`) and WPR2 is gone (`:939`); otherwise the D0
+ locks intentionally stay held (fail-safe).
+
+### 0.3 The gap
+
+The two planes never touch. The guest's static answers, controls and objects
+are emulated; the physical session's real handles, real FB geometry, real
+engine caps and real work capability are only visible to debugfs. The bridge
+must be added at exactly three places (see §2): the reply point
+`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the
+lifecycle points in `main.c` / `gsp_phys_gpu.c`.
+
+---
+
+## 1. Target architecture: runtime state machine
+
+### 1.1 Backend session state (module-global; one physical GPU)
+
+```
+ BACKEND_OFF
+ | first mdev create (gsp_phys_gpu=1 + hazard_ack; no other mdev)
+ v
+ BACKEND_CLAIMING power-lock D0 -> unbind dist -> bind phys -> board check
+ | claim ok \ abort signature / lock fail
+ v v
+ BACKEND_BOOTING FWSEC -> prime -> Booter -> RISC-V -> rmwait (init-done)
+ | init-done rpc_result=0 \ init failure / timeout
+ v v
+ BACKEND_READY static-config snapshot cached; RPC/control forwarding legal
+ | last mdev remove | rmmod | board signature
+ v
+ BACKEND_STOPPING stop emulation forwarding; UNLOADING_GUEST_DRIVER;
+ | quiesce; WPR2 teardown; staging free
+ v
+ BACKEND_OFF power unlock; distribution driver rebound and settled
+```
+
+`BACKEND_READY` is the single gate for every live answer. In every other
+state the guest-facing handlers must behave exactly as they do today.
+
+### 1.2 Ownership: who starts/stops what
+
+| Trigger | Action | Where |
+|---|---|---|
+| first `mdev` create | backend refcount++ ; if 0 -> claim + boot asynchronously from `init_work` | `kernel/main.c:535`, `:1553`; new `gsp_phys_backend_start()` |
+| last `mdev` remove | backend refcount-- ; if 0 -> `gsp_phys_gpu_release()` after the emulation thread is stopped | `kernel/main.c:614`, `:1614`; new `gsp_phys_backend_stop()` |
+| `rmmod` with live mdevs | forbidden by policy, but code path must drain: STOPPING first, then stop emu threads, then release | `kernel/main.c:1487` (`nv_vgpu_exit`) |
+| load-time `gsp_phys_gpu=1` | keep today's supervised behavior: claim (and full boot) at module init, owned by the module, released only at `rmmod` (debugfs harness unchanged) | `kernel/main.c:1450` |
+| QEMU close/open only | no backend action; the session persists across VM restarts until mdev remove | `nv_vgpu_vfio_dev_close`, `kernel/main.c:698` |
+| power-loss signature anywhere | STOPPING immediately (no new forwards); release fail-safe keeps D0 locks; no retry in this boot | `gsp_phys_board_check`, `gsp_phys_gpu.c:591` |
+
+The module-owned claim (load-time `gsp_phys_gpu=1`) and the mdev-owned claim
+share the same `phys` singleton. `gsp_phys_gpu_claim()` is idempotent today
+(`gsp_phys_gpu.c:667` returns 0 if already claimed); the refcount/ownership
+flag is new state and decides **who** releases.
+
+### 1.3 Locks and ownership
+
+| Lock | Protects | Taken by | Rule |
+|---|---|---|---|
+| `gsp_phys_session_lock` (new) | backend state, refcount, owner | `backend_start/stop/state`; `nv_vgpu_exit` | outermost; never held while waiting on a guest lock; may be held while calling `rm_lock` |
+| `gsp_phys_rm_lock` (existing, `booter:3671`) | all real-GSP TX/RX: queue cursors, `boot_stage.status_read_ptr`, `tx_seq_num`, probe/ladder/unload | `gsp_phys_boot_rm_unload`, `gsp_phys_rm_probe`, `gsp_phys_rmcall_run`, new `gsp_phys_backend_rpc()` | one real RPC at a time, always; **never** taken while holding `rm->stat_lock` or any guest-memory path |
+| `rm->stat_lock` (`gsp_rm_emulation.c:186`) | guest status-queue writes | `gsp_rm_send` (`:452`) | build the full reply before taking it; release immediately after write; never hold across a backend wait |
+| `vgpu->lock` (`nvidia-vgpu-vfio.h:99`) | per-mdev state transitions | `init_work`/`cleanup_work` | never held across claim/release or RPC waits |
+| `nv_vgpu_driver.lock` | device list | create/remove, sema thread scan | never taken under session/rm locks |
+| `svc.chan_lock` / `svc.vas_lock` | software work pipeline | sema service | unchanged; forwarding reads channel state copied out under these locks |
+
+Lock order: `nv_vgpu_driver.lock` (shortest, leaf) < `gsp_phys_session_lock`
+< `gsp_phys_rm_lock`. The emulation thread must take
+`session_lock -> check state -> drop session_lock -> rm_lock -> re-check a
+generation counter` so a release cannot deadlock against a forward in flight.
+
+### 1.4 Failure paths
+
+* **Guest crash / QEMU killed:** `open_count` drops; the backend stays READY
+ and claimed (device owned, host headless). No device writes. The next
+ `mdev` remove or `rmmod` drives the graceful path. This is the same
+ `--keep` semantics the supervised harness already uses.
+* **Guest RPC while STOPPING/unload already submitted:** every forwarding
+ seam returns "not ready" and the emulation answer is used. This mirrors the
+ defensive re-checks in `gsp_phys_rmcall_run()` (`booter:5119`).
+* **Module unload order (must change):** today `nv_vgpu_exit()` calls
+ `gsp_phys_gpu_release()` at `kernel/main.c:1500` *before*
+ `gsp_sema_service_stop()` (`:1525`) and before mdev unregistration
+ (`:1530`) which is what stops the emulation threads (`:1619`). Once
+ handlers forward to the backend, release must not race live forwards:
+ new order = set STOPPING (session lock) -> `gsp_sema_service_stop()` ->
+ `mdev_unregister_driver()` (drains emulation) -> `gsp_phys_gpu_release()` ->
+ parent/class teardown.
+* **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out,
+ the existing release still quiesces, runs the FWSEC teardown and refuses to
+ rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit
+ human approval plus a handoff update.
+* **Static-config served before READY:** bounded wait (recommend 2 s) then
+ pure emulation fallback, with a stat counter; never block the emulation
+ thread longer than the guest's own RPC poll can tolerate.
+
+---
+
+## 2. Seam table
+
+Legend: **L** = live backend, **E** = existing emulation, **H** = hybrid.
+"Change point" is the exact function that must learn the new behavior.
+
+### 2.1 Guest GSP-RM RPC class (msgq function IDs)
+
+| # | Guest message | Today | Target | Change point (file:line) |
+|---|---|---|---|---|
+| 1 | `GSP_SET_SYSTEM_INFO` (72) | consumed, no reply (`gsp_rm_emulation.c:3095`) | **E** - describes the guest client at guest boot; the backend already sent its own fn72 at `booter:1559` | `gsp_rm_handle_element` `gsp_rm_emulation.c:3095` |
+| 2 | `SET_REGISTRY` (73) | consumed, no reply (`:3096`) | **E** - same reason | `:3096` |
+| 3 | `CONTINUATION_RECORD` (71) | SUCCESS reply (`:3105`) | **E** | `:3105` |
+| 4 | `SET_GUEST_SYSTEM_INFO` (1) | echoes guest vgx version (`:982`) | **E** - this is the guest's negotiation, deliberately decoupled from the host session (see §4.3) | `gsp_rm_handle_set_guest_system_info` `:982` |
+| 5 | `GET_GSP_STATIC_INFO` (65) | hard-coded TU104 model (`:1025`), 256 MB FB, fake handles, hand-tuned caps | **H** - overlay live identity/capability fields proven safe; keep model-consistent FB geometry, PDE bases and handles until phases P4/P6 (see §4) | `gsp_rm_build_static_info` `:1025`, `gsp_rm_handle_get_static_info` `:1170` |
+| 6 | `GSP_RM_ALLOC` (103) | echo request + `NV_OK`; channels/events registered locally (`:2322`) | **H** - echo for emulation-only classes; forward selected classes (root/device/subdevice, then channel/memory) once a guest<->server handle map exists; always keep local side effects | `gsp_rm_handle_rm_alloc` `:2322` |
+| 7 | `GSP_RM_CONTROL` (76) | ~25 static per-cmd handlers; unknown -> zeroed success (`:2611`, default `:2807`) | **H** - whitelist of side-effect-free queries goes live with emulation fallback; vGPU-specific controls stay **E** (`COPY_SERVER_RESERVED_PDES` `:2647`, `PROMOTE_FAULT_METHOD_BUFFERS` `:2654`) | `gsp_rm_handle_rm_control` `:2611`; new forward helper before the switch |
+| 8 | `GSP_RM_FREE` (10) | echo + retire channel/events (`:2882`) | **H** - forward when the object was backend-allocated; always retire local bookkeeping | `gsp_rm_handle_free` `:2882` |
+| 9 | `SET_PAGE_DIRECTORY` (54) | recorded for translation + SUCCESS (`:2975`) | **H** - recording stays **E**; forwarding to the real MMU is P6 and gated on guest pages being GPU-DMA-reachable | `gsp_rm_handle_set_page_directory` `:2975` |
+| 10 | `UNSET_PAGE_DIRECTORY` (79) | record remove + SUCCESS (`:3063`) | **H** - same gate as #9 | `:3063` |
+| 11 | `UPDATE_BAR_PDE` (70) | SUCCESS ack (`:3144`) | **E** now; **L** only when BAR1 is backed by real VRAM | `:3144` |
+| 12 | `UNLOADING_GUEST_DRIVER` (47) | SUCCESS ack (`:3155`) | **E** - the guest's own shutdown; the real session's shutdown is host-driven on backend stop (`gsp_phys_gpu.c:909`) | `:3155` |
+| 13 | default/unknown | generic SUCCESS (`:3169`) | **E** (fail-safe: never invent a live forward for an unknown function) | `:3169` |
+
+### 2.2 Outbound events and replies
+
+| # | Record | Today | Target | Change point |
+|---|---|---|---|---|
+| 14 | `GSP_INIT_DONE` (0x1001) | emitted as soon as the emulated Booter finishes (`gsp_rm_emulation.c:3374`) | **H** - emit immediately as today, but live answers gate themselves on READY; alternatively delay emission on a bounded backend wait so the guest's first RM RPC sees READY | `gsp_rm_emu_on_booter_done` `:3339` |
+| 15 | `POST_EVENT` (0x1003) | software pump completion (`gsp_rm_post_channel_event` `:599`) | **E** as the relay mechanism; its *trigger* becomes the observation of a real hardware semaphore (P5) instead of a software op | `gsp_rm_post_channel_event` `:599`, `svc_notify_complete` `gsp_sema_service.c:4151` |
+| 16 | all replies | single writer `gsp_rm_send` (`:452`) | **E** - keep as the only guest-queue writer; live payloads are copied into it | `gsp_rm_send` `:452` |
+
+### 2.3 BAR / register and work-plane classes
+
+| # | Guest-visible thing | Today | Target | Change point |
+|---|---|---|---|---|
+| 17 | GSP falcon boot regs (FWSEC/Booter/RISCV/WPR2), SW interrupt tree, MMU invalidate, L2 flushes | emulated auto-complete (`gsp_emulation.c:193-452`) | **E always** - the two GSP sessions are intentionally independent; guest boot must never depend on the physical session | `gsp_emu_poll_boot` `:193` |
+| 18 | Guest VF doorbell `0x88704` | software pump (`main.c:826` -> `gsp_sema_service_doorbell` `:6369`) | **H** - pump classifies each push: pure state/release stays software; engine work is forwarded (P5) | `svc_pump_channels` `gsp_sema_service.c:5019`, `svc_decode_stream` `:2802` |
+| 19 | BAR0/BAR1/BAR3 shadows, PRAMIN window | host-RAM shadows (`bar_emulation.c:180`, `main.c:1205`) | **E** until real VRAM forwarding (P6) | `nv_vgpu_bar_emul_*`, `nv_vgpu_vfio_dev_mmap` |
+| 20 | legacy doorbell/shared-mem RPC class (`GSP_RPC_*`) | internal emulation client thread (`gsp_emulation.c:458-620`, `:674-694`) | **E / disable when READY** - not part of the guest physical-path protocol; gate it off in backend mode to avoid intercepting stray writes | `gsp_emu_thread_fn` `:674` |
+
+**Highest-leverage seams:** #5/#7 (one reply point, one dispatch point; all
+live answers plug in there) and #6 (whatever alloc/free semantics the guest
+gets determine whether real channels/page tables are possible).
+
+---
+
+## 3. Lifecycle changes in `main.c` / `gsp_emulation.c`
+
+1. **New backend API** (add to `gsp_phys_booter.[ch]`, used by both debugfs
+ and the guest plane; keeps `gsp_phys_rm_lock` private):
+ * `int gsp_phys_backend_state(void)` - OFF/CLAIMING/BOOTING/READY/STOPPING.
+ * `int gsp_phys_backend_rpc(u32 function, const void *req, u32 req_len,
+ void **reply, u32 *reply_len, u32 timeout_ms)` - checks
+ `claimed && rm_init_done && !rm_unload_submitted`, takes
+ `gsp_phys_rm_lock`, calls `gsp_phys_queue_tx_submit()` +
+ `gsp_phys_status_wait()` (both at `booter:2935`/`:2670`), returns a copy
+ of the payload; caller frees. Unmatched records (events, NOCAT) are
+ relayed or summarized like `gsp_phys_status_wait` already does.
+ * `int gsp_phys_backend_static_snapshot(u8 *out, u32 len)` - cached full
+ 1656-byte fn65 payload; filled once at READY by the existing
+ `gsp_phys_rm_probe_cycle()` (`booter:4056`) under the same lock.
+ * `void gsp_phys_backend_begin_stop(void)` / `gsp_phys_backend_end_stop()` -
+ state transitions only (no hardware).
+2. **New session state + refcount** in `gsp_phys_gpu.c` (new mutex, owner
+ flag, state enum). `gsp_phys_gpu_claim()`/`_release()` stay the only
+ hardware entry points; add `gsp_phys_backend_start()/stop()` that call
+ them exactly once.
+3. **`nv_vgpu_mdev_create()` (`main.c:535`)**: after
+ `vfio_register_emulated_iommu_dev()` (`:591`) and before/around
+ `schedule_work(&vgpu->init_work)` (`:598`), call
+ `gsp_phys_backend_start()`. Claiming must happen while no VM holds a live
+ mdev: with `available_instances` computed from per-type max (1) in
+ `nv_vgpu_get_available()` (`main.c:81`) there is at most one mdev of the
+ type in flight, so this holds.
+4. **`nv_vgpu_device_init_work()` (`main.c:1553`)**: keep BAR/IRQ/memory/
+ `gsp_emu_init()` order. The backend boot runs in its own kworker so
+ `mdev` create returns quickly; `init_work` only observes READY when it
+ initializes forwarding tables. Do **not** run the 60 s `rmwait`
+ (`booter:2764`, timeout `GSP_PHYS_RM_INIT_TIMEOUT_MS` 60000 at
+ `booter:446`) inline in `init_work` if it can be avoided: run claim+boot
+ in a dedicated work item so `mdev->remove` is never stuck behind it.
+5. **`nv_vgpu_device_cleanup_work()` (`main.c:1614`)**: unchanged per-mdev,
+ but the last-cleanup path calls `gsp_phys_backend_stop()` **after**
+ `gsp_emu_cleanup()` (`:1619`) so no emulation thread can forward while the
+ backend is being torn down.
+6. **`nv_vgpu_exit()` (`main.c:1487`)**: reorder as in §1.4; the load-time
+ claim path (owner = module) keeps the existing `gsp_phys_gpu_release()`
+ position relative to debugfs fini, but mdev-owned sessions are released in
+ the new order.
+7. **`gsp_emu_init()` (`gsp_emulation.c:724`)**: no structural change; the
+ `gsp_rm_emu` context gains a reference to the module backend (not a new
+ per-mdev session). `gsp_emu_cleanup()` (`:831`) unchanged except it must
+ not be the thing that releases the backend (done by `main.c`, step 5).
+8. **`gsp_rm_emu_on_booter_done()` (`gsp_rm_emulation.c:3339`)**: after
+ linking the queues, optionally wait (bounded) for READY before emitting
+ INIT_DONE, or emit immediately and let each live handler wait. Recommended:
+ emit immediately (the guest's early RPCs are all emulatable), and let
+ `GET_GSP_STATIC_INFO` be the single bounded-wait point.
+9. **`gsp_emu_thread_fn()` (`gsp_emulation.c:651`)**: no new blocking calls in
+ the loop; live forwards happen inside `gsp_rm_handle_element()` on that
+ thread, with the 2 s per-RPC bound. If a longer wait is ever needed, move
+ forwarding to a dedicated backend kthread and keep this loop responsive.
+
+---
+
+## 4. Static-config flow and identity/version negotiation
+
+### 4.1 Current guest model (must stay internally consistent)
+
+`gsp_rm_build_static_info()` (`gsp_rm_emulation.c:1025`) hard-codes:
+`fb_length = 0x10000000` (256 MB, offset 1224), `fbio_mask=0xff`,
+`fbp_mask=0x3f`, `gpuNameString="NVIDIA GeForce RTX 2080 SUPER"` (1260),
+`gpuShortNameString="TU104"` (1324), `bar1PdeBase=bar2PdeBase=0`
+(1536/1544), fake internal handles (1600/1604/1608), engine caps
+`0x60803E02 / 0x00362020 / 0` (1204). The 256 MB figure is coupled to the
+BAR0 shadow's `LOCAL_MEMORY_RANGE` (`bar_emulation.c:280`,
+`0x100ce0=0x18` -> 256 MB) and to the emulated WPR2/FRTS location the guest
+driver checks. Changing it without changing the shadow breaks the guest boot
+(this is why 512 MB was reverted, per the comment at `:1096`).
+
+### 4.2 Live values observed (cycle 13)
+
+```
+static-info internal: hClient=0xc2000006 hDevice=0xabcd0080 hSubdevice=0xabcd2080
+static-info fb: bar1PdeBase=0x1f7650000 bar2PdeBase=0x1f76c2000
+ fb_length=0x200000000 fbio_mask=0xf fbp_mask=0x11
+ busWidth=0 ramType=256 l2CacheSize=15
+static-info gpu: name='NVIDIA GeForce RTX 2080 Super with Max-Q Design'
+ short='TU104-B' vbiosSubVendor=0x4d50 vbiosSubDevice=0x4449
+static-info engineCaps=[0x08083e02,0x00000044,0x00000000]
+```
+
+Two grounded warnings:
+
+* The decoded `busWidth=0`, `ramType=256` are not plausible TU104 values
+ (256-bit, GDDR6). Either the decode offsets for those two fields do not
+ match the real struct, or the emulated values were hand-derived rather than
+ observed. **A/B every field before overlaying it.**
+* Engine caps differ materially (`0x60803E02` vs `0x08083E02`, and the second
+ word). The emulation deliberately sets extra bits (COPY14 etc.) to make
+ `libcuda` accept the engine inventory. Overlaying live caps would change
+ the guest's engine/class topology and can regress the whole CUDA init.
+ Keep emulated caps until the backend can actually service the resulting
+ demands (P4+).
+
+### 4.3 Proposed flow
+
+1. At READY, cache the full fn65 payload (new bytes in the backend session,
+ keyed by the loaded firmware's `phys.fw_crc32`, `gsp_phys_gpu.c:272`), via
+ the existing probe cycle (`booter:4056`) under `gsp_phys_rm_lock`.
+2. `gsp_rm_handle_get_static_info()` (`:1170`) keeps building the emulated
+ 1656-byte model, then applies an allow-listed overlay from the snapshot
+ (start: name/short-name/VBIOS identity/L2 if A/B passes; **not** FB
+ geometry, masks, PDE bases, handles, engine caps).
+3. Any overlay field is independently switchable (`gsp_phys_static_live`
+ module parameter, default off) so a regression is a one-line rollback.
+4. Handles move in P4 and PDE bases/geometry in P6, each behind its own
+ switch and its own A/B.
+5. Version negotiation stays decoupled: the guest sees the emulation's
+ version answer (`SET_GUEST_SYSTEM_INFO`, `gsp_rm_emulation.c:982`); the
+ real server is negotiated by the host as the host client
+ (`gsp_phys_rm_probe_fn1`, `booter:3984`, vgx 0x29/0x0C). The guest must
+ never learn the physical firmware's version; the emulation target remains
+ 570.144 while the host may boot a different, matching TU10x GSP image
+ (`gsp_phys_gpu.c:75`). The wire structures are the same family and the
+ live snapshot is keyed by firmware CRC.
+
+---
+
+## 5. Phased implementation plan (smallest first)
+
+Each phase is independently revertible. "Supervised" means the documented
+harness discipline: `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`, human
+go-ahead, recovery plan, pstore, never unattended
+(`docs/03-architecture/08-physical-gpu-integration.md` §7/§8).
+
+### P0 - Backend API skeleton (inert)
+* **Scope:** new state/refcount/lock in `gsp_phys_gpu.c`; locked
+ `backend_rpc()`/`static_snapshot()` in `gsp_phys_booter.c`; no caller
+ changes; no behavior change.
+* **Acceptance:** `make kernel` clean; load/unload with no params; all
+ existing debugfs triggers behave identically; new API logs and refuses when
+ OFF/not-init-done/unloading.
+* **Risk:** none (never sends unless a live init-done session exists).
+* **Rollback:** trivial.
+
+### P1 - Lifecycle wiring (claim on mdev create, release on last remove)
+* **Scope:** `main.c` create/remove/init_work/cleanup_work/exit ordering
+ (§3 items 3-6); no live answers yet.
+* **Acceptance (supervised, harness `phys_claim_cycle.sh` style):** fresh
+ boot -> insmod params -> create mdev -> one real boot log sequence (FWSEC,
+ Booter M0=0, RISC-V active, init-done after sequencer) -> guest boots and
+ the CUDA driver-API smoke test still prints `ALL OK` 64/64 -> stop VM ->
+ remove mdev -> `UNLOADING_GUEST_DRIVER` suspend, scratch14 bits 24-27 DONE,
+ WPR2 down, distribution driver rebound; repeat twice; zero power-domain
+ cycles; no reboot.
+* **Risk:** **highest phase.** Claim is now triggered by a sysfs action, not
+ by module load. The two-param gate and the D0 lock must already be active
+ inside claim (they are, `gsp_phys_gpu.c:670/700`), the claim must refuse a
+ second owner, and the release must never run while a forward is in flight
+ (STOPPING gate). Any power-loss signature -> hold locks, no retry.
+* **Rollback:** `gsp_phys_gpu=0` (no claim) or load-time `gsp_phys_gpu=1`
+ only; `mdev` emits `-EOPNOTSUPP` from `backend_start()`.
+
+### P2 - Static-config identity overlay
+* **Scope:** cache snapshot at READY; overlay allow-list in
+ `gsp_rm_build_static_info()`/`gsp_rm_handle_get_static_info()`; module
+ parameter; stats. Bounded wait (2 s) then emulation fallback.
+* **Acceptance:** guest still `nvidia-smi -L` + smoke `ALL OK`; guest
+ `nvidia-smi -q` name changes to the live `... with Max-Q Design`; dmesg
+ shows which fields came live; backend absent -> byte-identical guest reply.
+* **Risk:** a consumed identity field changes a driver decision (VBIOS
+ validation is the likeliest). Mitigation: field-by-field A/B, default-off,
+ instant fallback; never touch geometry/caps/handles in P2.
+* **Rollback:** parameter off.
+
+### P3 - Read-only control forwarding (whitelist)
+* **Scope:** in `gsp_rm_handle_rm_control()` (`:2611`), before the per-cmd
+ switch: for an allow-list of side-effect-free queries (FB/BUS/KGR/GMMU/
+ CE/display statics), call `backend_rpc()` with the **adopted internal
+ handles** from the snapshot (the guest's emulated handles are not valid on
+ the server yet), copy reply params (payload+24) on `rpc_result==0 &&
+ status==0`, else fall back to the existing handler. Add per-cmd
+ forwarded/fallback counters.
+* **Acceptance:** guest boot and smoke green with forwarding on; dmesg shows
+ N forwarded controls; forced backend timeout -> same behavior as today;
+ repeat calls yield identical params (no server-side state change).
+* **Risk:** a whitelist entry is not actually read-only (display/KGR can
+ allocate on first call). Start with FB/BUS/GMMU static info only, verify
+ against the server's log ring (`gsp-phys-log-dump`), then grow.
+* **Rollback:** parameter/whitelist off.
+
+### P4 - Alloc/free forwarding for the object ladder
+* **Scope:** `gsp_rm_handle_rm_alloc()` (`:2322`) forwards
+ root/device/subdevice (and only those, first) via `backend_rpc(fn103)`;
+ returns the server's chosen handle to the guest; maintains a
+ guest-handle -> server-handle map used by P3 and `gsp_rm_handle_free()`.
+ Any failure falls back to today's echo (never half-forward), and local
+ channel/event registration still runs.
+* **Acceptance:** guest boot + smoke green; dmesg shows adopted handles
+ (`0xc2.../0xabcd0080/0xabcd2080`) matching cycle13; guest FREE returns
+ success; no server-side wedge (no RPC timeout after alloc).
+* **Risk:** the guest's alloc params may not satisfy the server's per-class
+ validation; a bad forward can wedge the RM task (see the `0x40` GSP-owned
+ channel incident). Guard: one class at a time, abort on first failure,
+ `UNLOADING_GUEST_DRIVER` + rmmod remains the only fail-safe; never retry a
+ mutating RPC.
+* **Rollback:** parameter off (guest returns to echo path).
+
+### P5 - Work forwarding + completion relay (interim, host buffers)
+* **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25:
+ channel allocates, page directory commits, `REAL COPY OK` on the physical
+ CE in VA mode). The remaining work: bind each guest CE/GR channel to a real
+ channel; in `svc_pump_channels()` (`gsp_sema_service.c:5019`) classify the
+ decoded push and, for forwarded ops, submit an equivalent real method block
+ built by the existing `gsp_phys_rmcall_build_methods()` (`booter:4879`) +
+ GPFIFO/USERD/doorbell (`:4991-5032`). Observe the real semaphore, then
+ mirror the release into guest memory with `svc_sem_write()`
+ (`gsp_sema_service.c:1514`) and post the guest event with
+ `gsp_rm_post_channel_event()` (`gsp_rm_emulation.c:599`). All hardware
+ operands are module-owned DMA buffers (as in the ladder), so no guest memory
+ is touched by the engine yet.
+* **Acceptance:** ladder: `echo 4 > gsp-phys-rm-call` logs
+ `REAL COPY OK` (`booter:5052`) and src==dst after a hardware copy - met
+ (cycle 25, 4 KiB, semaphore released, dst==src). Guest: a submitted copy
+ completes with the semaphore observed in host memory and the guest tracker
+ advancing; wrong-operand/refused submissions fall back to software and are
+ counted.
+* **Risk:** doorbell to a wedged channel; wrong GPFIFO/USERD (guards:
+ `rmcall_channel_server_mem` refusal `:4944`, work-submit token after
+ `GPFIFO_SCHEDULE` `:4977`, `GP_PUT` bounds `:4995`); hardware writes only
+ host DMA memory, so no cross-tenant exposure yet.
+* **Rollback:** per-channel `forward=0`; software pump is untouched.
+
+### P6 - Guest memory into the real MMU (end state: guest submission on hardware)
+* **Scope:** map guest page-table pages/records so the physical GPU can DMA
+ them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real
+ server; allocate real memory objects for guest allocations (fn4 with real
+ PFNs); per-guest-channel real channels with client rings (module or guest
+ memory); pump forwards real push streams referencing guest VAs; completion
+ and faults relayed through real MSI-X -> module IRQ -> existing eventfd
+ injection (`gsp_emu_inject_completion_interrupt`, `gsp_emulation.c:626`;
+ the physical driver does not `request_irq()` yet, `gsp_phys_gpu.c:380-461`).
+* **Acceptance:** `vectorAdd` correct results through a forwarded launch;
+ copy with semaphore; second mdev is refused/emulated (single-session
+ isolation until per-mdev clients exist); fault test.
+* **Risk:** wrong page tables -> engine writes to arbitrary host memory;
+ IOMMU/DMA reachability of guest pages is the project's existing
+ known-unknown (`docs/04-implementation/01-known-knowns-and-unknowns.md`
+ §2.2) and is a prerequisite, not solved by this design. Keep
+ `gsp_phys_seq_addr_valid()` (`booter:1800`) and all queue validations.
+* **Rollback:** per-mdev isolation not yet available; disable forwarding and
+ fall back to P5 host-buffer mode.
+
+---
+
+## 6. Top risks and the guards that must be preserved
+
+1. **Host hard-freeze on unbind/rebind.** Unbinding the distribution driver
+ can power-gate the board; re-bind intermittently fails and the host
+ freezes with no crash record. Guards: two-param gate
+ (`gsp_phys_gpu.c:670`), refuse if GPU/root port already
+ runtime-suspended (`:688`), D0 lock on GPU + siblings + root port
+ (`:499-549`), runtime-PM usage ref (`:188-204`), config-space liveness
+ checks after claim and before rebind (`:591`), no retry after a
+ power-loss signature, human approval for any reboot.
+2. **WPR2 held / one-way protected region.** Always run the release sequence
+ in order: `UNLOADING_GUEST_DRIVER` (only if init-done was seen)
+ -> quiesce GSP falcon -> FWSEC teardown -> free staging
+ (`gsp_phys_gpu.c:909-914`); if WPR2 survives, do **not** rebind and keep
+ the locks (`:939`); a FWSEC-only stop is forbidden
+ (`gsp_phys_gpu.c:770`). The debugfs `gsp-phys-unload-rpc` remains
+ write-only and inert outside a live claim.
+3. **RM-task wedge (live 0x40 incident).** Setting the GSP-owned channel
+ `internalFlags` bit (0x40) wedged GSP-RM: the alloc RPC never replied and
+ every later RPC timed out; the fail-safe held the device until reboot.
+ The working tree now bans that bit and refuses the submission when the
+ winning variant used server-allocated memdescs (`booter:281`, `:4714`,
+ `:4942`). Never re-enable it; abort the ladder at the first failure with
+ no object teardown (`:5123` comments); `UNLOADING_GUEST_DRIVER` + rmmod is
+ the only recovery.
+4. **Wrong-page-table / wrong-doorbell writes.** No forwarding seam may
+ submit engine work or program page tables until P6's reachability
+ prerequisite holds. Preserve: queue header validation in
+ `gsp_phys_queue_tx_submit()` (`booter:2983`), free-space check (`:3032`),
+ status-record validation (`booter:2554`), sequencer address validation
+ (`booter:1800`), stage-4 token/bounds guards (`booter:4969-4999`).
+5. **Guest status-queue invariants.** Strict sequence/wrap/checksum and
+ multi-element records; `gsp_rm_send()` (`:452`) is the only writer and
+ must not be called while holding `gsp_phys_rm_lock` or waiting on the
+ backend; the event-burst cap (`:488`) and parse retry backoff (`:3397`)
+ stay.
+6. **Two planes, two BARs.** The guest face must only ever touch
+ `gsp_emu_context.bar0_vaddr` (the vzalloc shadow,
+ `gsp_emulation.c:761-778`); the backend must only ever touch `phys.bar0`
+ (real ioremap, `gsp_phys_gpu.c:420`). Never ioremap a guest GPA (the
+ fixed `gsp_client_init_emulation()` contract).
+7. **Ordering and context rules.** No long scans or backend waits in VFIO
+ callbacks; `vfio_dma_rw` only from kthread context (mm == NULL);
+ never `rmmod` with a live VM; mdev remove before module unload; stop
+ emulation before backend release once P2+ lands (new `nv_vgpu_exit`
+ order).
+8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX
+ seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only
+ `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree
+ (`backups/evidence-*`) current on every live cycle.
+
+---
+
+## Appendix A - line anchor map (function -> file:line)
+
+`kernel/gsp_phys_booter.c` (working-tree revision used for this design):
+`gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init`
+2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670,
+`gsp_phys_boot_rmwait` 2764, `gsp_phys_queue_tx_submit` 2935, `gsp_phys_rm_lock`
+3671, `gsp_phys_boot_rm_unload` 3673, `gsp_phys_sci_decode` 3820,
+`gsp_phys_rm_probe_fn65` 3918, `fn1` 3984, `_cycle` 4056, `gsp_phys_rm_probe`
+4086, `gsp_phys_rmcall_rpc` 4329, `_alloc` 4385, `_register_mem` 4454,
+`_control` 4510, `_stage1` 4575, `_stage2` 4650, `_channel_variants` 4714,
+`_stage3` 4761, `_build_methods` 4879, `_stage4` 4923, `_stage5` 5068,
+`_run` 5108, `_fini` 5197, debugfs write handlers 5613/5681/5754,
+`gsp_phys_boot_debugfs_init` 5827, `gsp_phys_boot_quiesce` 5919,
+`gsp_phys_boot_staging_free` 5944.
+
+`kernel/gsp_emulation.c`: thread 651, init 724, cleanup 831, poll_boot 193,
+rm poll call 672, legacy RPC dispatch 458, MSI-X inject 626.
+
+`kernel/gsp_rm_emulation.c`: `gsp_rm_send` 452, parse 806, link 917,
+`set_guest_system_info` 982, `build_static_info` 1025, `get_static_info` 1170,
+`rm_alloc` 2322, `rm_control` 2611 (default 2807), `free` 2882,
+`set_page_directory` 2975, `unset` 3063, `handle_element` 3084,
+`poll_commands` 3194, `on_booter_done` 3339, `poll` 3386.
+
+`kernel/main.c`: `nv_vgpu_mdev_create` 535, `remove` 614, init_work 1553,
+cleanup_work 1614, `nv_vgpu_init` 1388, claim 1450, debugfs init 1461,
+sema start 1468, `nv_vgpu_exit` 1487, release 1500, sema stop 1525,
+mdev unregister 1530, param 2457.
+
+`kernel/gsp_phys_gpu.c`: claim 662, hazard gate 670, suspend refusal 688,
+power lock 523/499, unbind 706, driver register 718, board check 591,
+runtime ref 188, boot 745-848 (FWSEC 792, prime 799, run 805, rmwait 811),
+release 876 (unload 909, quiesce 910, fwsec 911, staging free 914, WPR2
+fail-safe 939, board check 955, rebind 972).
+
+`kernel/gsp_sema_service.c`: doorbell 6369, start 6392, stop 6416, thread
+6232, pump 5019, decode 2802, sem write 1514, notify 4151.
diff --git a/docs/README.md b/docs/README.md
index 5f9a32f..5057c9f 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -57,7 +57,7 @@ flowchart TB
classDef hw fill:#e2e8f0,stroke:#4a5568
class E,F done
class G wip
- class H plan
+ class H wip
class I hw
```
@@ -108,7 +108,7 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai-
| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened |
| Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) |
| Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-01 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
-| CUDA kernel execution | 🔴 Not yet — the runtime client initializes and the kernel launch is queued, but execution awaits the backend; the mechanism is chosen (module-owned GPU + real GSP session) with ownership + firmware staging verified and the power-domain lock implemented pending supervised verification ([`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md)); the real GSP boot has not started, and today only decoded memsets/copies/semaphore releases execute, in the host |
+| CUDA kernel execution | 🟡 The backend is live: the module owns the physical GPU and its real GSP-RM initialized, serves the full static-config reply and shuts down cleanly; a host resource-manager client now allocates memory/VAS/page-directory/TSG/channel and has executed the first real copy on the physical copy engine (4 KiB, semaphore released, dst==src, 2026-10-01). What remains is wiring that session into the guest-facing path so guest submissions execute there ([`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md)); see also [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md) |
| OpenGL / Vulkan | 🔴 Not yet |
| Windows guest | 🔴 Not yet |
| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented |
@@ -121,13 +121,16 @@ returns and the 64-byte payload round-trips exactly (64/64, re-verified
2026-10-01 on the current build). The full
mechanism, with offsets and the decompiled call path, is in
[`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md).
-The remaining frontier is real execution on the physical GPU: the runtime client
-initializes and queues a kernel launch, but does not execute it. The chosen
-backend — a module-owned GPU running its real GSP firmware as the
-resource-manager server — has verified ownership + firmware staging and an
-implemented (pending supervised verification) power-domain lock; the real GSP
-boot has not started. It is specified in
-[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md).
+The remaining frontier is guest-facing execution: the runtime client initializes
+and queues a kernel launch, and the physical backend now executes real work (the
+first copy-engine transfer completed on hardware — 4 KiB with the semaphore
+released and dst==src, 2026-10-01) — but the two planes are not yet wired
+together. The backend, a module-owned GPU running its real GSP firmware as the
+resource-manager server, is specified in
+[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md);
+the bridge from the live session to the guest-facing mdev/VFIO path is specified
+in
+[`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md).
For the visual end-state map and the cross-component call paths, see
[`03-architecture/09-full-system-picture.md`](03-architecture/09-full-system-picture.md)
@@ -160,14 +163,15 @@ Read in order if you are new. Jump directly if you know what you need.
| 11 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min |
| 12 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min |
| 13 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min |
-| 14 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
-| 15 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
-| 16 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
-| 17 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
-| 18 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
-| 19 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
-| 20 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
-| 21 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
+| 14 | [Physical backend guest wiring](03-architecture/11-physical-backend-guest-wiring.md) | developers, architects | 25 min |
+| 15 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
+| 16 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
+| 17 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
+| 18 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
+| 19 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
+| 20 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
+| 21 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
+| 22 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware
decompilation record, address reference, historical protocol notes). Wiki