diff options
Diffstat (limited to 'docs/03-architecture/11-physical-backend-guest-wiring.md')
| -rw-r--r-- | docs/03-architecture/11-physical-backend-guest-wiring.md | 109 |
1 files changed, 80 insertions, 29 deletions
diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md index cbb4b33..2229365 100644 --- a/docs/03-architecture/11-physical-backend-guest-wiring.md +++ b/docs/03-architecture/11-physical-backend-guest-wiring.md @@ -1,8 +1,11 @@ # Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session -> **Status: P0-P5(w1) implemented; P6a guest-page reachability and P6b waves +> **Status: P0-P5(w1) implemented; P6a guest-page reachability (trigger +> residual fixed) and P6b waves > 1/1.5/2 (guest-copy forwarding, deferred release, hardware-primary copies) -> proven live (2026-10-02).** The backend API skeleton +> proven live (2026-10-02); P6c C0/C1 (engine-authored guest CE semaphore +> relay) and C3 (forwarded-engine fault observation/relay) implemented +> default-off; H1 host GR ladder open.** The backend API skeleton > (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that > path is an open defect off the critical path), the static-config identity > overlay (P2, verified live in an A/B run), the read-only control @@ -16,7 +19,14 @@ > releases exactly once) and the wave-2 hardware-primary copies (default off, > verified live: copy_skipped=119, queued req=100 ok=100 verify_ok=100 > verify_fail=0 fallback=0 timeouts=0, smoke `matched=64/64` `ALL OK`, with -> the software fallback for unqueued/failed copies) are in the tree. +> the software fallback for unqueued/failed copies) are in the tree. The P6c +> relay is live-verified default-off — the physical engine authored the +> guest CE copy's semaphore release (`relays=64`), failed/refused copies fall +> back to the software apply exactly once, and one relayed-copy data mismatch +> is under triage (mismatch logging plus software fallback in charge); the C3 +> forwarded-engine fault relay is live green, default off. The P6a trigger +> residual is fixed: triggers 6/7 re-translate at trigger time and use fresh +> scratch lines. > The physical backend itself is live: the module-owned > GSP-RM completes initialization on the real TU104, serves the full > static-config reply, and shuts down gracefully. The host resource-manager @@ -24,7 +34,12 @@ > a committed page directory, a channel group and a real copy-engine channel, > and has executed the **first real work on the physical engine** — a 4 KiB > copy in virtual-address mode with the semaphore released and destination -> equal to source (cycle 25). What this document specifies is the remaining +> equal to source (cycle 25). The H1 host GR ladder then allocated a GR0 +> channel with a compute-class object, but its first semaphore push faulted +> with an uncommitted context (`CTXNOTVALID`), and the follow-up +> `GPU_PROMOTE_CTX` attempt wedged GSP-RM; the GR execution wall is open and +> the device sits in the designed locked fail-safe. What this document +> specifies is the remaining > bridge: wiring that live session into the guest-facing mdev/VFIO path so that > guest submissions execute on the physical GPU. It extends > [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which @@ -102,6 +117,15 @@ (`:5068`). Stage 4 now runs end to end: the channel allocates, the page directory is committed, and a 4 KiB virtual-address-mode copy executes on the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25). + `gsp_phys_rmcall_engine=1` selects a GR0 ladder variant (stage 3 builds the + channel and a `TURING_COMPUTE_A` compute object with `engineType=1`, stage 4 + pushes a GR semaphore release). Its first live push faulted with an + uncommitted context (`CTXNOTVALID`). `gsp_phys_rmcall_gr_promote=1..3` + selects opt-in `GPU_PROMOTE_CTX` shapes to commit that context; the first + attempt wedged GSP-RM (internal assert, no reply) and teardown could not + clear WPR2, so the fail-safe held the device. Debugfs trigger 9 re-runs the + GR semaphore push on an already-built GR ladder without touching WPR2, for + use once a promote shape is viable. * Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2 (`:911`), frees staging (`:914`), and only rebinds the distribution driver @@ -114,10 +138,12 @@ The two planes now meet at the lifecycle seam (P0/P1, mdev-owned claim/release), the static-config identity overlay (P2), the read-only control whitelist (P3) and, for eligible CE copies, the P6b scratch-mapped work path (waves 1/1.5/2: guest operands, deferred exactly-once release, -hardware-primary copies); objects, FB geometry, engine caps, real guest -page tables and completion/fault relay are still emulated. The remaining -bridge must be added at the reply point `gsp_rm_send()`, the inbound dispatch -point `gsp_rm_handle_element()`, and the work-plane seams (see §2). +hardware-primary copies) plus the P6c relay (C0/C1 engine-authored guest +release, C3 forwarded-engine fault observation/relay), all default off; +objects, FB geometry, engine caps and real guest page tables are still +emulated. The remaining bridge must be added at the reply point +`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the +work-plane seams (see §2). --- @@ -141,7 +167,9 @@ point `gsp_rm_handle_element()`, and the work-plane seams (see §2). BACKEND_STOPPING stop emulation forwarding; UNLOADING_GUEST_DRIVER; | quiesce; WPR2 teardown; staging free v - BACKEND_OFF power unlock; distribution driver rebound and settled + BACKEND_OFF power unlock only when gsp_phys_repower=1 (default + 0 latches the D0/D3cold locks); distribution driver + rebound and settled ``` `BACKEND_READY` is the single gate for every live answer. In every other @@ -198,8 +226,9 @@ generation counter` so a release cannot deadlock against a forward in flight. parent/class teardown. * **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out, the existing release still quiesces, runs the FWSEC teardown and refuses to - rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit - human approval plus a handoff update. + rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then follows the + project reboot-approval policy (a written reason and post-reboot plan before + approval). * **Static-config served before READY:** bounded wait (2 s, implemented) then pure emulation fallback, with a stat counter; never block the emulation thread longer than the guest's own RPC poll can tolerate. @@ -495,8 +524,10 @@ go-ahead, recovery plan, pstore, never unattended 514 real CE copies (194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts). The P6b waves 1/1.5/2 (guest-copy forwarding, deferred exactly-once release, hardware-primary copies with a software - fallback) have since landed (P6 status below); the real guest-page-table - path (P6c) remains. + fallback) have since landed (P6 status below), and P5 wave 2 (hardware- + primary release relay) is subsumed by the P6b wave-2 hardware-primary mode + and the P6c engine-authored relay; the real guest-page-table path and GR + execution remain. * **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25: channel allocates, page directory commits, `REAL COPY OK` on the physical CE in VA mode). The remaining work: bind each guest CE/GR channel to a real @@ -587,20 +618,35 @@ go-ahead, recovery plan, pstore, never unattended with the guest-RAM ranges unregistered proved that fallback again (`copy_skipped=121`, recovered once each by the software copy, `copy_fail=0`, smoke still green), so the software fallback remains for any - unqueued or failed copy. **Residual (P6a trigger path):** - the debug-trigger path pre-builds its page-table slots once during ladder - stage 3; a guest page's host physical address can change between - translations (observed `0x13a591000` then `0x3b8318000` for the same GPA), - and a runtime PTE update into an already-fetched 128-byte line is invisible, - so the later write-direction trigger failed its data check. Forwarding is - unaffected (it resolves HPAs at submit time); the follow-up is to give the - debug triggers fresh scratch lines (the forwarding path) or to pin the - translated page. **Remaining:** P6c - completion/fault relay, real - page-directory shadowing, real guest channels and push forwarding with - guest page tables in the real MMU, and `vectorAdd` correct results; - guest-submission forwarding through the real guest page tables is not - implemented yet. All new parameters default off; with them off the software - path is unchanged. + unqueued or failed copy. **P6a trigger residual — fixed (2026-10-02):** + the debug triggers now re-translate the guest page's host physical address + at trigger time through the fresh-scratch-line path (triggers 6/7), retiring + the stale-PTE data-check failure; live `GUEST G2H COPY OK` and + `GUEST W2G COPY OK verify_ok=1`, with the PTE self-test and ladder + `REAL COPY OK` on a clean release. Forwarding was never affected (HPAs are + resolved at submit time). **P6c C0/C1 (engine-authored guest CE semaphore + relay, default off):** with `gsp_phys_relay=1` the guest-visible semaphore + page is mapped into the CE channel and the engine authors one selected + release; the deferred hold verifies the guest word and skips the software + apply of that record, and any refusal, failure, timeout or failed + verification leaves the unchanged software apply in charge, so the release + stays exactly-once. Live: the engine-authored release was observed in the + guest page (`relays=64`), the relay-on run stayed green + (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64), and a + forced-unresolvable control proved the fallback recovers every copy once; + one relayed copy released its semaphore but failed the host-side data + compare, so the parameter stays default-off pending triage, with the + mismatch logged and the software apply still in charge. **P6c C3 + (forwarded-engine fault observation/relay, default off):** with + `gsp_phys_fault_relay=1` a timed-out forwarded copy drains the GSP-RM + status queue for the engine's fault record, matches it to the ladder + channel and writes the guest channel notifier/event through the existing + MSI-X path (live green, default off). **Remaining:** real guest page tables + and channels, real page-directory shadowing, GR context validity/execution + and `vectorAdd` correct results; guest-submission forwarding through the + real guest page tables is not implemented yet, and the GR promote attempt + wedged the server (§0.2). All new parameters default off; with them off the + software path is unchanged. * **Scope:** map guest page-table pages/records so the physical GPU can DMA them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real server; allocate real memory objects for guest allocations (fn4 with real @@ -677,13 +723,18 @@ go-ahead, recovery plan, pstore, never unattended order). 8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only - `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree - (`backups/evidence-*`) current on every live cycle. + `gsp-phys-state`/`gsp-phys-log-dump` files, and attach the evidence + artifacts to the change on every live cycle. --- ## Appendix A - line anchor map (function -> file:line) +> **Historical anchors.** The line numbers below were taken from the +> working-tree revision used for the original design and are stale; they are +> kept as a historical map only. Anchor by function name when re-grepping and +> treat the numbers as untrusted. + `kernel/gsp_phys_booter.c` (working-tree revision used for this design): `gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init` 2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670, |