From d2de7bf1d6bb7d0931085f87da6c3013cd7f1aca Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 03:24:35 -0400 Subject: gsp-phys: P6a - real copy engine reads/writes guest RAM pages - file-backed guest RAM (launcher memory-backend-file share=on) and in-kernel GPA->HPA translation through the shared file page cache (read_cache_page + page_to_phys + pfn_valid), opened O_LARGEFILE; new default-off gsp_phys_guest_mem parameter - debugfs gsp-phys-guest-ram (path + gpa/off/size ranges) and gsp-phys-guest-gpa (one pinned GPA per boot); rm-call triggers 6/7 run host->guest and guest->host 4 KiB CE copies; trigger 8 is a PTE self-test through a fresh slot - guest page-table slots are pre-built during ladder stage 3: runtime PTE writes into an already-walked page-table line are not observed without a TLB invalidate - guest RAM layout (q35 4.2, 4 GiB): below-4G GPA == file offset, above-4G offset = GPA - 0x80000000 (2 GiB PCI hole) - tests: tools/test/gpage_test.c guest probe; docs + STATE-CURRENT record the live green (G2H/W2G with semaphore release, guest observed the write, smoke ALL OK 64/64, clean latched release) --- STATE-CURRENT.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) (limited to 'STATE-CURRENT.md') diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index f15a403..5a68054 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-02 02:25.** This is a dated public status page, not live +> **Snapshot: 2026-10-02 03:07.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -69,7 +69,7 @@ meant to execute on the physical GPU. and destination equal to source. Every cycle released the device cleanly with no reboot; the next rung is wiring the live physical session into the guest-facing mdev/vfio path - ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 (guest page tables/memory into the real MMU, including the IOMMU/DMA reachability known-unknown) is the next end-to-end rung, P5 wave 2 (hardware-primary release relay) is optional, and P4 (alloc/free handle adoption) remains deliberately deferred. | + ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot), so forwarding guest submissions (page-directory shadowing, real guest channels, push forwarding) is the next end-to-end rung; P5 wave 2 (hardware-primary release relay) is optional, and P4 (alloc/free handle adoption) remains deliberately deferred. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -81,6 +81,51 @@ chain succeeds. ## Right now (frontier) +- **Guest-page DMA proven on the physical engine (P6a, 2026-10-02).** The + IOMMU/DMA reachability known-unknown is closed on the reference host: the + physical GPU's IOMMU group is an identity domain (all-RAM identity) and the + mediated device registers as an emulated-IOMMU VFIO device, so its type-1 + container keeps guest DMA mappings privately and no kernel `iommu_domain` + exists for guest GPAs - `iommu_iova_to_phys` cannot translate them. Guest + RAM is instead a QEMU file-backed shared-memory region + (`memory-backend-file,share=yes,prealloc=on` under `/dev/shm`); the module + opens that file and translates GPA -> HPA through its page cache + (`read_cache_page` + `page_to_phys` + `pfn_valid`, behind the active-mdev + gate), under the new default-off `gsp_phys_guest_mem` parameter and + write-only `gsp-phys-guest-ram` / `gsp-phys-guest-gpa` debugfs controls + (the ladder trigger gained host->guest write, guest->host read and a + fresh-slot PTE self-test). On the reference q35 4 GiB guest, GPA equals file + offset below 4 GiB and offset = GPA - 0x80000000 above it (the 2 GiB 32-bit + PCI hole). Channel page-table slots for the guest window are pre-built during + ladder stage 3 before any engine access: a PTE written into a page-table line + the GMMU had already walked was accepted but never completed, and no + TLB-invalidate path exists yet. Live: the physical copy engine copied 4 KiB + from a real guest RAM page into a module buffer (G2H) and back into the same + page (W2G) with the semaphore released; the guest process observed the write + in its own page; the driver-API smoke stayed GREEN `ALL OK` 64/64 and the + informational PTE self-test passed. Release stayed latched as designed: + distribution driver rebound, protected region down, exactly 5 power-domain + locks intentionally held, 0 in-window rail cycles, no reboot. Module sha256 + `e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory + shadowing, real guest channels, push forwarding) - not implemented yet. +- **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved + by the P6a result above).** The + rung puts guest memory into the real execution path: guest RAM is now a + QEMU file-backed `pc.ram` region (`/dev/shm/vgpu-guest-ram`, 4096M, + `share=yes`, `prealloc=on`) that the host module can map, under a new + `gsp_phys_guest_mem` parameter plus guest-GPA and guest-RAM debugfs + interfaces, with a small guest pagemap probe (`tools/test/gpage_test.c`) + that publishes its own guest-physical address for a host-side + guest-to-host / host-to-guest copy check. The full boot + RM ladder + (`REAL COPY OK`) and the latched clean release remain green on every + attempt, but the data path is not yet green: the first attempts exposed + two concrete host-side defects - the guest physical address resolved to + no host translation through the active mediated device's VFIO domain, and + the guest-RAM file setup failed with EOVERFLOW because the host-side file + open omitted the large-file flag on the 4 GiB shared-memory file (the + latter was root-caused in-run). The P6 code was uncommitted at the time of + that snapshot; the follow-up attempt resolved both defects (P6a result + above). - **Guest-submission shadow execution (P5 wave 1) landed and verified live (2026-10-02 ~02:00-02:20).** Under the new `gsp_phys_work_forward` parameter (bool, default off) the sema pump requests ONE shadow copy for each eligible -- cgit v1.2.3