diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 241 |
1 files changed, 241 insertions, 0 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md new file mode 100644 index 0000000..c00231b --- /dev/null +++ b/STATE-CURRENT.md @@ -0,0 +1,241 @@ +# Open-IOV Current State + +> **Snapshot: 2026-09-29 15:25.** This is a dated public status page, not live +> state; it is refreshed on every progress check. Live working notes are +> maintained outside this repository; the public known-knowns/unknowns document +> is authoritative here: +> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md). +> The 2026-09-21 snapshot remains in [`STATE.md`](STATE.md) as a historical +> record. + +## Mission + +Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality +(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without +proprietary GRID licenses - via host-side GSP emulation in +`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side +Open-IOV code. + +Architecture stance: *emulate the interface, pass the work through to real +hardware, never patch anyone.* The control plane is emulated; the work is +meant to execute on the physical GPU. + +## Milestone ladder (this snapshot) + +| Milestone | Status | +|---|---| +| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ | +| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ | +| `nvidia-smi -L` enumerates the emulated TU104 | ✅ | +| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | +| `cuCtxCreate` returns | ⏳ **frontier** | +| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ❌ not reached | +| Full `nvidia-smi` output | ❌ (faults on the CUDA path) | +| CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) | +| Graphics (3D), video | ❌ | +| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | + +Test status is tracked with the CUDA driver-API smoke test +([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises +`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/ +DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole +chain succeeds. + +## Right now (frontier) + +- The stock guest driver initializes fully; the CUDA client reaches + `cuCtxCreate` within seconds and runs there for minutes before failing with + a forced `rc=999` (a timeout path, not an RM error record). +- The per-channel completion counters in guest memory are enumerated, + registered and published; the client's per-channel waits clear, and the + per-channel GPFIFO work on copy channels is consumed. +- Both engine push-buffer streams are now reachable: the copy-engine buffer is + resolved deterministically (function-trace pointer chain, all pages pinned), + and the compute-engine mapping window is being pinned from guest page tables + on every boot. +- **The copy-engine data path executes**: decoded method streams run in + sustained fashion (memsets, copies, releases climbing; thousands of push + entries consumed). +- **Frontier: context creation still times out on one completion.** The final + wait is a tracking semaphore whose release was decoded from a readable + compute push but could not be applied in time because the counter page had + no translation at that moment. The fix in progress is to pin the + compute-channel mapping window from guest truth at boot so the release + lands before the client's deadline. +- Active work: the per-boot compute-window pin stream and re-running the full + cycle. +- Execution backend (chosen, not started): the module will own the physical + GPU and run its real GSP firmware as the resource-manager server, so real + work executes with an unmodified host driver and no host daemon; see + [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). +- Host stability: the host hang class observed on 2026-09-28 (kernel stack + overflow in a channel snapshot path) is fixed and frame-audited; repeated + VM load/attach/teardown cycles run without panics or pstore records. + +## Initialization, work and teardown path + +Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, +`[not yet]`. + +``` + HOST SIDE GUEST (stock driver, unmodified) + ========= ================================ + + [done] build + load open-iov-vfio.ko + | (module first: exports the symbols the NVIDIA open modules link against) + v + [done] mdev create ------------------------------------> / + | BAR/PCI/MSI-X emulation; GSP emulation thread + v + [done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap + | + v + [done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter + | + v + [done] GSP-RM message queues / RPC / controls ---------> object + client setup + | + v + [done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB + | (two-GRCE fix; async copy engine) + v + [done] ------------------------------------------------> nvidia-smi -L enumerates TU104 + | + v + [done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet + | + v + [frontier] --------------------------------------------> cuCtxCreate + | UVM register GPU; channel manager + | GR + CE channels; push buffers + | semaphore pool; completion counters + v + [not yet] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip + [not yet] ---------------------------------------------> full nvidia-smi output +``` + +Work execution (host service vs client synchronization): + +``` + [partial] host decodes guest pushes (method streams): memsets, copies, releases + - CE memop pushes now execute (memset/ringops climbing, misses at zero) + - GR/compute pushes not yet consumed + [partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress + [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission) + [not yet] correct results (vectorAdd) +``` + +Teardown and recovery: + +``` + [partial] failure teardown exercised (context rollback, channel stop, notifier status) + [done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached + [done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited +``` + +Recent chain of walls removed (2026-09-28 to 2026-09-29): + +``` + host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild) + -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20) + -> stale semaphore-pool translation -> fake-only proven insufficient + -> push buffer found and validated -> per-page pins -> authoritative pin table + -> deterministic chain solved -> pin-drop and release-push fixes + -> copy-engine data path executes -> compute-window pinning +``` + +## Tools, code paths and fixes behind this progress + +``` + KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked) + ================================ ========================== + gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test + gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh + gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh + gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh + channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh + map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py + bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh + main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py + stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py + dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py + tools/guest-diag/guest_kprobes.sh + + DATA STRUCTURES DEBUG PLANES + =============== ============ + channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset / + notifier, ring) ringops / nomap / skip / faked / + VA->PA map (merge rebuild, hot pages, cache) pins / ipins + pin table + authoritative injected pin table debug params: sema_probe, sema_counter, + completion counter table (per channel) sema_map_add, chan_dump, shadow_dump + semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads + push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc/<pid>/pagemap + client trackers / RM error record chain marker-based /dev/kmsg parsing +``` + +| Wall | Fix | +|---|---| +| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build | +| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels | +| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace | +| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits | +| Four-word semaphore releases ignored | decode both release encodings | +| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed | +| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot | +| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed | +| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table | +| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe | +| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins | +| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot | + +## What "CUDA works" means (open gates) + +Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) +section 4, a defensible claim needs **all** of: + +1. every guest-to-device message for the CUDA path answered or forwarded, + with coherent replies for consumed fields; +2. every work VA translatable and server-owned page-table ranges populated; +3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g. + `vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical + GPU's compute engine - not merely acknowledged - and copies executing on + real hardware with correct results; +4. completion (semaphores, events, faults) behaving as the guest expects; +5. at least two driver families passing; +6. tenant isolation; +7. measured performance within an agreed factor of native. + +Until then the honest statement is: **initialization works; execution is in +progress.** + +## Environment + +- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. +- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages + the physical GPU; `open-iov-vfio.ko` loads as an independent module. +- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a + `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. +- Firmware: TU104 GSP 570.144 (public release). + +## Build and load order + +Build: host module first (it generates the symbol versions the NVIDIA open +kernel modules link against), then the NVIDIA open kernel modules, then +userspace. + +Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and +attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the +device. + +## See also + +- [`docs/README.md`](docs/README.md) - documentation front door +- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done +- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point +- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session) +- [`STATE.md`](STATE.md) - 2026-09-21 historical snapshot + +--- + +*Refresh this page on every progress check (snapshot time in the header); +prefer moving detail into the `docs/` tree and keeping this page short.* |