docs: record physical-GPU integration mechanism and refresh project status

- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that
  exist, mechanism options A-E, chosen path (module-owned GPU driven through
  its real GSP firmware), sharing policy, decision record
- update 06-host-module-and-hybrid-model, 02-practical-challenges, system
  overview, known-knowns, docs/README reading order and CUDA-side docs to
  match
- correct stale engine-topology/COPY14 blocker claims across the tree
- add STATE-CURRENT.md as the dated public status snapshot

1 files changed, 241 insertions, 0 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
new file mode 100644
index 0000000..c00231b
--- /dev/null
+++ b/STATE-CURRENT.md
@@ -0,0 +1,241 @@
+# Open-IOV Current State
+
+> **Snapshot: 2026-09-29 15:25.** This is a dated public status page, not live
+> state; it is refreshed on every progress check. Live working notes are
+> maintained outside this repository; the public known-knowns/unknowns document
+> is authoritative here:
+> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
+> The 2026-09-21 snapshot remains in [`STATE.md`](STATE.md) as a historical
+> record.
+
+## Mission
+
+Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
+(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
+proprietary GRID licenses - via host-side GSP emulation in
+`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
+Open-IOV code.
+
+Architecture stance: *emulate the interface, pass the work through to real
+hardware, never patch anyone.* The control plane is emulated; the work is
+meant to execute on the physical GPU.
+
+## Milestone ladder (this snapshot)
+
+| Milestone | Status |
+|---|---|
+| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
+| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
+| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
+| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
+| `cuCtxCreate` returns | ⏳ **frontier** |
+| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ❌ not reached |
+| Full `nvidia-smi` output | ❌ (faults on the CUDA path) |
+| CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) |
+| Graphics (3D), video | ❌ |
+| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
+
+Test status is tracked with the CUDA driver-API smoke test
+([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
+`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
+DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
+chain succeeds.
+
+## Right now (frontier)
+
+- The stock guest driver initializes fully; the CUDA client reaches
+  `cuCtxCreate` within seconds and runs there for minutes before failing with
+  a forced `rc=999` (a timeout path, not an RM error record).
+- The per-channel completion counters in guest memory are enumerated,
+  registered and published; the client's per-channel waits clear, and the
+  per-channel GPFIFO work on copy channels is consumed.
+- Both engine push-buffer streams are now reachable: the copy-engine buffer is
+  resolved deterministically (function-trace pointer chain, all pages pinned),
+  and the compute-engine mapping window is being pinned from guest page tables
+  on every boot.
+- **The copy-engine data path executes**: decoded method streams run in
+  sustained fashion (memsets, copies, releases climbing; thousands of push
+  entries consumed).
+- **Frontier: context creation still times out on one completion.** The final
+  wait is a tracking semaphore whose release was decoded from a readable
+  compute push but could not be applied in time because the counter page had
+  no translation at that moment. The fix in progress is to pin the
+  compute-channel mapping window from guest truth at boot so the release
+  lands before the client's deadline.
+- Active work: the per-boot compute-window pin stream and re-running the full
+  cycle.
+- Execution backend (chosen, not started): the module will own the physical
+  GPU and run its real GSP firmware as the resource-manager server, so real
+  work executes with an unmodified host driver and no host daemon; see
+  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
+- Host stability: the host hang class observed on 2026-09-28 (kernel stack
+  overflow in a channel snapshot path) is fixed and frame-audited; repeated
+  VM load/attach/teardown cycles run without panics or pstore records.
+
+## Initialization, work and teardown path
+
+Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
+`[not yet]`.
+
+```
+        HOST SIDE                                        GUEST (stock driver, unmodified)
+        =========                                        ================================
+
+ [done] build + load open-iov-vfio.ko
+        |  (module first: exports the symbols the NVIDIA open modules link against)
+        v
+ [done] mdev create ------------------------------------>  /
+        |  BAR/PCI/MSI-X emulation; GSP emulation thread
+        v
+ [done] VM attach (QEMU q35 + pcie-root-port) ---------->  config space read; BAR mmap
+        |
+        v
+ [done] GSP boot handshake (emulated) ------------------>  nvidia.ko probe / RmInitAdapter
+        |
+        v
+ [done] GSP-RM message queues / RPC / controls --------->  object + client setup
+        |
+        v
+ [done] engine + class lists --------------------------->  GPU_GET_ENGINES, class DB
+        |                                                  (two-GRCE fix; async copy engine)
+        v
+ [done] ------------------------------------------------>  nvidia-smi -L enumerates TU104
+        |
+        v
+ [done] ------------------------------------------------>  cuInit / DeviceGetCount / DeviceGet
+        |
+        v
+ [frontier] -------------------------------------------->  cuCtxCreate
+        |                                                  UVM register GPU; channel manager
+        |                                                  GR + CE channels; push buffers
+        |                                                  semaphore pool; completion counters
+        v
+ [not yet] --------------------------------------------->  cuMemAlloc + cuMemcpy roundtrip
+ [not yet] --------------------------------------------->  full nvidia-smi output
+```
+
+Work execution (host service vs client synchronization):
+
+```
+ [partial] host decodes guest pushes (method streams): memsets, copies, releases
+           - CE memop pushes now execute (memset/ringops climbing, misses at zero)
+           - GR/compute pushes not yet consumed
+ [partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress
+ [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission)
+ [not yet] correct results (vectorAdd)
+```
+
+Teardown and recovery:
+
+```
+ [partial] failure teardown exercised (context rollback, channel stop, notifier status)
+ [done]    VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
+ [done]    crash recovery: pstore inspection; the host hang class is fixed and frame-audited
+```
+
+Recent chain of walls removed (2026-09-28 to 2026-09-29):
+
+```
+ host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
+   -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
+   -> stale semaphore-pool translation -> fake-only proven insufficient
+   -> push buffer found and validated -> per-page pins -> authoritative pin table
+   -> deterministic chain solved -> pin-drop and release-push fixes
+   -> copy-engine data path executes -> compute-window pinning
+```
+
+## Tools, code paths and fixes behind this progress
+
+```
+ KERNEL MODULE (open-iov-vfio.ko)                GUEST-SIDE TOOLS (tracked)
+ ================================                ==========================
+ gsp_emulation.c      boot, doorbells, MSI-X      tools/test/cuda_drv_test.c   smoke test
+ gsp_rm_emulation.c   msgq, RPC, controls         tools/guest-diag/vmrestart.sh
+   gsp_rm_tu104.h       two-GRCE engine fix       tools/guest-diag/cuda_cycle.sh
+ gsp_sema_service.c   THE WORKHORSE               tools/guest-diag/push_guest_tools.sh
+   channel pump, method decode, rescue,           tools/guest-diag/inject_counters.sh
+   map rebuild, pins, counters, releases          tools/guest-diag/semfix_watch.py
+ bar_emulation.c      BAR shadows                 tools/guest-diag/track_waits.sh
+ main.c               mdev/VFIO lifecycle         tools/guest-diag/trkinfo.py
+   stack-frame audit (-Wframe-larger-than)        tools/guest-diag/bp999.py / scanerr.py
+ dynamic_rpc_dispatcher.c  firmware load fix      tools/guest-diag/read_va.py
+                                                  tools/guest-diag/guest_kprobes.sh
+
+ DATA STRUCTURES                                 DEBUG PLANES
+ ===============                                 ============
+ channel table (handle, GPFIFO, USERD,           module stats: applied / miss / memset /
+   notifier, ring)                                 ringops / nomap / skip / faked /
+ VA->PA map (merge rebuild, hot pages, cache)      pins / ipins
+ pin table + authoritative injected pin table    debug params: sema_probe, sema_counter,
+ completion counter table (per channel)            sema_map_add, chan_dump, shadow_dump
+ semaphore pool page (GPU VA 0x121000000)        QMP gva2gpa + guest gdbstub reads
+ push buffer allocation (GPU VA 0x120000000)     guest kprobes; /proc/<pid>/pagemap
+ client trackers / RM error record chain         marker-based /dev/kmsg parsing
+```
+
+| Wall | Fix |
+|---|---|
+| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
+| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
+| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
+| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
+| Four-word semaphore releases ignored | decode both release encodings |
+| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
+| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
+| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
+| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
+| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
+| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
+| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |
+
+## What "CUDA works" means (open gates)
+
+Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
+section 4, a defensible claim needs **all** of:
+
+1. every guest-to-device message for the CUDA path answered or forwarded,
+   with coherent replies for consumed fields;
+2. every work VA translatable and server-owned page-table ranges populated;
+3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
+   `vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
+   GPU's compute engine - not merely acknowledged - and copies executing on
+   real hardware with correct results;
+4. completion (semaphores, events, faults) behaving as the guest expects;
+5. at least two driver families passing;
+6. tenant isolation;
+7. measured performance within an agreed factor of native.
+
+Until then the honest statement is: **initialization works; execution is in
+progress.**
+
+## Environment
+
+- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
+- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
+  the physical GPU; `open-iov-vfio.ko` loads as an independent module.
+- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
+  `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
+- Firmware: TU104 GSP 570.144 (public release).
+
+## Build and load order
+
+Build: host module first (it generates the symbol versions the NVIDIA open
+kernel modules link against), then the NVIDIA open kernel modules, then
+userspace.
+
+Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
+attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
+device.
+
+## See also
+
+- [`docs/README.md`](docs/README.md) - documentation front door
+- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
+- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
+- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
+- [`STATE.md`](STATE.md) - 2026-09-21 historical snapshot
+
+---
+
+*Refresh this page on every progress check (snapshot time in the header);
+prefer moving detail into the `docs/` tree and keeping this page short.*