gsprm: reply-latency logging + param dump cap; guest-diag: CDT_PAUSE, watcher reaping, wider CE stream window

1 files changed, 23 insertions, 16 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 7a83f54..67b3d11 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
 # Open-IOV Current State
 
-> **Snapshot: 2026-09-29 17:59.** This is a dated public status page, not live
+> **Snapshot: 2026-09-29 23:57.** This is a dated public status page, not live
 > state; it is refreshed on every progress check. Live working notes are
 > maintained outside this repository; the public known-knowns/unknowns document
 > is authoritative here:
@@ -44,8 +44,9 @@ chain succeeds.
 ## Right now (frontier)
 
 - The stock guest driver initializes fully; the CUDA client reaches
-  `cuCtxCreate` within seconds and runs there for minutes before failing with
-  a forced `rc=999` (a timeout path, not an RM error record).
+  `cuCtxCreate` within seconds and now fails with a forced `rc=999` a few
+  seconds after the compute-channel phase (a timeout path, not an RM error
+  record).
 - The per-channel completion counters in guest memory are enumerated,
   registered and published; the client's per-channel waits clear, and the
   per-channel GPFIFO work on copy channels is consumed.
@@ -60,19 +61,25 @@ chain succeeds.
   page show all 20 compute-channel counter slots written at the correct
   16-byte stride (host releases now land exactly where the client polls), and
   auto-registration walks the full channel rank range.
-- **The registration wait is named and unblocked**: the ~16 s stall is a
-  single tracker wait (value 3) on the first copy-engine channel created
-  during UVM registration; pinning the copy-engine push buffer early (about a
-  second after it appears, instead of tens of seconds) is what lets
-  registration complete.
-- **The frontier has moved past registration** to the ~1.6 s that follow it
-  (a burst of external-allocation operations and rollback) — that is the
-  failure now being captured with the stack sampler and ioctl trace.
-- Active work: delivering the copy-engine push-buffer translation before the
-  client's first tracker wait, then diagnosing the post-registration failure.
-- One guest-side wedge occurred on the fastest-pin configuration (enlarged
-  trace buffer); it recovered by VM restart and is being retried with adjusted
-  probe settings. The host stayed stable throughout.
+- **The registration wall is root-caused and fixed**: the host consumed
+  channel submissions without delivering their tracking-semaphore release;
+  every consumed engine submission now delivers the incrementing release, and
+  the pool value advances within a second (previously it stalled until the
+  deadline). Registration no longer dominates the run.
+- **New frontier: the guest's own GSP health check**: a second after the
+  compute-channel phase the guest driver runs a GSP health check while one of
+  its resource-manager replies is pending, then rolls the context back. Two
+  suspects under test: reply-path latency now that releases are applied at a
+  high rate, and a reply whose parameter size is not canonical (the guest
+  logs `NV_ERR_INVALID_PARAMETER` on its reply-cache path).
+- Active work: an extended debug loop on the health-check failure - more
+  one-shot breakpoint tooling and fresh failure-instant guest/host snapshots
+  are being produced, with kernel and tool edits still in the tree awaiting
+  the next build/commit.
+- Guest health note: agent/SSH timeouts during a run are usually guest-agent
+  starvation and recover when the client stops; one hard wedge occurred only
+  when pushed memsets were re-enabled, so bulk pushed operations stay disabled
+  while stale translations can exist.
 - Execution backend (chosen, not started): the module will own the physical
   GPU and run its real GSP firmware as the resource-manager server, so real
   work executes with an unmodified host driver and no host daemon; see