From 06a424b52bf345cd463a2a41d1ac8213df987c02 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Wed, 30 Sep 2026 05:58:38 -0400 Subject: sema-svc: run CE work before its release, grace unreadable CE entries, payload-pattern audit - CE LAUNCH_DMA releases are now applied after the push's collected work runs: the client reads copied data as soon as it observes the release, so release-first let cuMemcpyDtoH observe a copy that had not executed. - Unreadable CE entries stay pending for a grace window (sema_ce_grace_ms, default 8s) before the release fallback consumes them, so a translation that lands within a service pass lets the push be decoded and its work executed instead of being dropped. - sema_copy_pat matches an 8-byte value against every copy's src/dst first word and scans decoded pushes for it, naming the submission that carries a known payload regardless of size/encoding. - sema-dma-skip logs transfer-shaped launches that collect no op; op log lines carry the channel handle; LAUNCH_DMA inventory follows OPS_LOG. - A real release application now records the release VA, so decoded pushes no longer trigger a spurious fake release. --- STATE-CURRENT.md | 96 ++++++++++++++++++++++++-------------------------------- 1 file changed, 41 insertions(+), 55 deletions(-) (limited to 'STATE-CURRENT.md') diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 9543100..8d597aa 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-09-30 05:05.** This is a dated public status page, not live +> **Snapshot: 2026-09-30 05:54.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -28,8 +28,8 @@ meant to execute on the physical GPU. | Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ | | `nvidia-smi -L` enumerates the emulated TU104 | ✅ | | CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | -| `cuCtxCreate` returns | ⏳ **frontier** - the previously fatal check now passes; the final result is being captured | -| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ❌ not reached | +| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | +| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | 🟡 all calls succeed and the test prints `ALL OK`; the 64-byte data roundtrip is not yet correct | | Full `nvidia-smi` output | ❌ (faults on the CUDA path) | | CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) | | Graphics (3D), video | ❌ | @@ -43,52 +43,35 @@ chain succeeds. ## Right now (frontier) -- The stock guest driver initializes fully; the CUDA client reaches - `cuCtxCreate` within seconds and - with the latest fix - passes the check - that previously forced a `rc=999` timeout, continuing into UVM for minutes. - The final result is being captured on fresh boots. -- The per-channel completion counters in guest memory are enumerated, - registered and published; the client's per-channel waits clear, and the - per-channel GPFIFO work on copy channels is consumed. -- Both engine push-buffer streams are reachable: the copy-engine buffer is - resolved deterministically, and the compute-engine mapping window plus - GPFIFO rings are pinned from guest page tables on every boot (a guest-side - scanner streams page-map deltas to the host). -- **The copy-engine data path executes**: decoded method streams run in - sustained fashion (memsets, copies, releases; thousands of push entries - consumed), and compute-channel releases decode. -- **Both counter walls are falling**: live raw dumps of the client's counter - page show all 20 compute-channel counter slots written at the correct - 16-byte stride (host releases now land exactly where the client polls), and - auto-registration walks the full channel rank range. -- **The registration wall is root-caused and fixed**: the host consumed - channel submissions without delivering their tracking-semaphore release; - every consumed engine submission now delivers the incrementing release, and - the pool value advances within a second (previously it stalled until the - deadline). Registration no longer dominates the run. -- **A second release-path drop is fixed**: some compute-channel releases were - decoded through the generic translation map and silently dropped; routing - them through the exact counter slot makes **all 20/20 channel releases - land** (verified on live boots). The failure is unchanged by that fix, so - the counter path was not the gate. -- **The context-creation error is root-caused and fixed**: the emulated GR - information table left four indices zero (the shader-pipe count and - related clock fields), so the client's clock-table validation rejected the - context. Filling them with the real-device values makes the client sail - past the previously fatal check and **run inside UVM for minutes** - - channels advancing, memsets and releases executing - instead of failing in - seconds. -- **Next wall is the push-data path**: with context creation cleared, the - copy-engine pushes need their operand translations; routing those through - pinned translations (previously a missing-entry failure) now lets the - inline writes execute (`memset=860/1653`, `ringops=1687` live). -- Active work: capture the final context-creation result on a fresh boot, - then move to the allocation/copy roundtrip; runs that end with the - guest-agent unresponsive are the known trace-flood starvation, not a panic. -- Guest health note: agent/SSH timeouts during a run are usually guest-agent - starvation and recover when the client stops; one hard wedge occurred only - when pushed memsets were re-enabled, so bulk pushed operations stay disabled - while stale translations can exist. +- **The smoke test now completes end-to-end**: `cuInit -> cuDeviceGet -> + cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> + cuCtxDestroy` all succeed and the test prints `ALL OK`. The context-creation + error that stalled this for days is root-caused and fixed: the emulated GR + information table left the shader-pipe count and related clock fields zero, + which the client's clock-table validation rejected. +- **The remaining wall is the user payload**: the memory roundtrip returns + zeros - the 64-byte payload does not reach the emulated VRAM through the + path this allocation uses. The copies the host observes carry UVM + bookkeeping, not the payload; a full scan of the pinned copy-engine push + buffer and of the 256 MB VRAM shadow finds no trace of it. The next suspect + is the external-allocation path the client uses (RM-allocated memory mapped + into UVM), including a possible windowed CPU copy. +- Recap of the fixes behind this point: all 20/20 compute-channel releases + land (three decode paths routed to the exact counter slot); registration + delivers the incrementing tracking release for every consumed submission; + the copy-engine push buffer is pinned from guest truth on every boot; and + the copy-engine decoder consults pinned translations before the scanned + map, so every push-data operation executes (`memset=19473`, `ringops=10551` + live). +- Active work: find the payload's host-to-device path for the external + allocation - pattern-match the copy's source, audit the RM + allocation/descriptor path, and check where windowed writes land. +- Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an + operation log are now module parameters, and the test's output survives + guest-agent stalls via the serial console. +- Guest health note: agent/SSH timeouts during long runs are the guest-side + trace reader flooding the ring; they recover on their own, so a first + timeout is not a reason to restart the VM. - Execution backend (chosen, not started): the module will own the physical GPU and run its real GSP firmware as the resource-manager server, so real work executes with an unmodified host driver and no host daemon; see @@ -130,12 +113,13 @@ Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, [done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet | v - [frontier] --------------------------------------------> cuCtxCreate + [done] ------------------------------------------------> cuCtxCreate | UVM register GPU; channel manager | GR + CE channels; push buffers | semaphore pool; completion counters v - [not yet] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip + [partial] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip + | (all calls succeed; payload data path pending) [not yet] ---------------------------------------------> full nvidia-smi output ``` @@ -143,8 +127,8 @@ Work execution (host service vs client synchronization): ``` [partial] host decodes guest pushes (method streams): memsets, copies, releases - - CE memop pushes now execute (memset/ringops climbing, misses at zero) - - GR/compute pushes not yet consumed + - CE pushes execute at scale (memset=19473, ringops=10551; misses at zero) + - the 64-byte test payload's HtoD path is not decoded yet [partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission) [not yet] correct results (vectorAdd) @@ -153,12 +137,12 @@ Work execution (host service vs client synchronization): Teardown and recovery: ``` - [partial] failure teardown exercised (context rollback, channel stop, notifier status) + [done] happy-path teardown exercised end-to-end (cuCtxDestroy OK) [done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached [done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited ``` -Recent chain of walls removed (2026-09-28 to 2026-09-29): +Recent chain of walls removed (2026-09-28 to 2026-09-30): ``` host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild) @@ -167,6 +151,8 @@ Recent chain of walls removed (2026-09-28 to 2026-09-29): -> push buffer found and validated -> per-page pins -> authoritative pin table -> deterministic chain solved -> pin-drop and release-push fixes -> copy-engine data path executes -> compute-window pinning + -> registration release delivery -> 20/20 counter slots -> GR-info clock fix + -> cuCtxCreate OK and the full smoke test ALL OK; payload data path remains ``` ## Tools, code paths and fixes behind this progress -- cgit v1.2.3