diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 39 |
1 files changed, 23 insertions, 16 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 7a83f54..67b3d11 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-09-29 17:59.** This is a dated public status page, not live +> **Snapshot: 2026-09-29 23:57.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -44,8 +44,9 @@ chain succeeds. ## Right now (frontier) - The stock guest driver initializes fully; the CUDA client reaches - `cuCtxCreate` within seconds and runs there for minutes before failing with - a forced `rc=999` (a timeout path, not an RM error record). + `cuCtxCreate` within seconds and now fails with a forced `rc=999` a few + seconds after the compute-channel phase (a timeout path, not an RM error + record). - The per-channel completion counters in guest memory are enumerated, registered and published; the client's per-channel waits clear, and the per-channel GPFIFO work on copy channels is consumed. @@ -60,19 +61,25 @@ chain succeeds. page show all 20 compute-channel counter slots written at the correct 16-byte stride (host releases now land exactly where the client polls), and auto-registration walks the full channel rank range. -- **The registration wait is named and unblocked**: the ~16 s stall is a - single tracker wait (value 3) on the first copy-engine channel created - during UVM registration; pinning the copy-engine push buffer early (about a - second after it appears, instead of tens of seconds) is what lets - registration complete. -- **The frontier has moved past registration** to the ~1.6 s that follow it - (a burst of external-allocation operations and rollback) — that is the - failure now being captured with the stack sampler and ioctl trace. -- Active work: delivering the copy-engine push-buffer translation before the - client's first tracker wait, then diagnosing the post-registration failure. -- One guest-side wedge occurred on the fastest-pin configuration (enlarged - trace buffer); it recovered by VM restart and is being retried with adjusted - probe settings. The host stayed stable throughout. +- **The registration wall is root-caused and fixed**: the host consumed + channel submissions without delivering their tracking-semaphore release; + every consumed engine submission now delivers the incrementing release, and + the pool value advances within a second (previously it stalled until the + deadline). Registration no longer dominates the run. +- **New frontier: the guest's own GSP health check**: a second after the + compute-channel phase the guest driver runs a GSP health check while one of + its resource-manager replies is pending, then rolls the context back. Two + suspects under test: reply-path latency now that releases are applied at a + high rate, and a reply whose parameter size is not canonical (the guest + logs `NV_ERR_INVALID_PARAMETER` on its reply-cache path). +- Active work: an extended debug loop on the health-check failure - more + one-shot breakpoint tooling and fresh failure-instant guest/host snapshots + are being produced, with kernel and tool edits still in the tree awaiting + the next build/commit. +- Guest health note: agent/SSH timeouts during a run are usually guest-agent + starvation and recover when the client stops; one hard wedge occurred only + when pushed memsets were re-enabled, so bulk pushed operations stay disabled + while stale translations can exist. - Execution backend (chosen, not started): the module will own the physical GPU and run its real GSP firmware as the resource-manager server, so real work executes with an unmodified host driver and no host daemon; see |