STATE-CURRENT: 06:50 check-back snapshot + debugging-options catalogue

Refresh the status page: the acceptance is holding under hardening (per-op once bitmap, epoch/content-hash gate, 15 s CE grace; first-attempt 86% -> ~94-95%, fakes/run 3.39 -> 0.61), the hold-all-unreadable experiment is live, and the residual classes are named (late-decode corruption, early UVM channel-manager stall, guest init panic). Add an options section listing candidate techniques that would accelerate debugging toward end-to-end CUDA (deterministic fault injection, offline pump model, structured host tracing, guest-memory forensics, GR-wall metadata/golden reference/static-analysis throughput, cheaper supervised cycles, concurrency checks, tooling automation) - explicitly not directives, each requiring an eng-ops admission record.

1 files changed, 159 insertions, 25 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 38017d5..d65eab9 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,7 +1,8 @@
 # Open-IOV Current State
 
-> **Snapshot: 2026-10-04 (reorganized; simulator status current 2026-10-04,
-> physical/guest campaign data as of 2026-10-04 ~18:45).** This is a dated public
+> **Snapshot: 2026-10-05 (reorganized; simulator status current 2026-10-04,
+> physical/guest campaign data as of 2026-10-05 ~06:50; debugging-options
+> catalogue added 2026-10-05).** This is a dated public
 > status page, not live state; it is refreshed on every progress check. Live
 > working notes are maintained outside this repository; the public
 > known-knowns/unknowns document is authoritative here:
@@ -31,22 +32,26 @@ isolation, performance, portability).
   and relay paths (all parameters default off; one relayed copy's data
   mismatch is under triage), and the host GR ladder allocates a GR0 channel +
   compute object.
-- **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA (2026-10-04 ~16:20).**
-  The emulation-only ("no-claim") campaign now runs an unmodified guest CUDA
-  program to completion with verified results: `MAIL the little gpu man says
-  you've got mail`, `VADD n=1048576 bad=0 checksum=392344090.500000`,
-  `STATUS OK`. Three consecutive supervised cycles (`noclaim38/39/40`)
-  accepted, deterministic (identical IN/OUT hashes; `noclaim39/40` on the same
-  module `582bf1da...`), pixel-verified by the host bridge daemon
-  (`CUBB2 seq=2 op=2 ... status=ok bad=0`), and committed (`cf6ff542`). Two
-  data-path defects were root-caused to get there: a replayed GPFIFO entry
-  re-executed its CE copies (fixed with per-slot raw-entry identity - replays
-  re-apply releases but never re-run operations) and a multi-page virtual
-  operand was copied linearly from its first page (fixed with page-granular
-  re-resolution). A repeat-run stability campaign since is at 14 of 15 green
-  (`noclaim42`-`55` OK; `noclaim41` failed on the guest-side flake); `noclaim56`
-  live at check. The vadd executes via the host CUDA bridge daemon; forwarding
-  guest engine work to the physical GPU remains the open backend wall.
+- **Guest frontier: FIRST GREEN END-TO-END GUEST CUDA, NOW UNDER HARDENING
+  (2026-10-04 ~16:20; hardening 2026-10-05 overnight).** The emulation-only
+  ("no-claim") campaign runs an unmodified guest CUDA program to completion
+  with verified results: `MAIL the little gpu man says you've got mail`,
+  `VADD n=1048576 bad=0 checksum=392344090.500000`, `STATUS OK`, deterministic
+  (identical IN/OUT hashes), pixel-verified by the host bridge daemon, and
+  committed (`cf6ff542`). Three data-path/hazard classes were then closed:
+  replayed GPFIFO entries never re-run work (per-slot entry identity, then a
+  per-op executed bitmap); a slot consumed by a grace fake is epoch-marked so
+  its work can never run later against a refilled staging buffer (plus an FNV
+  content witness in the once gate); and the CE pending-entry grace was widened
+  8 s -> 15 s so a copy's first run precedes its fallback release. Measured
+  effect: first-attempt acceptance moved from 86% to ~94-95% across campaigns
+  and fakes/run fell 3.39 -> 0.61. A further experiment (hold every unreadable
+  CE entry for the bounded grace, delete the silent pre-rescue fake, grace
+  25 s; module `97eba5b7`) is live at check with campaign 229-236 at 3/3 green
+  so far. Residual classes: late-decode corruption (~3-5%), an early UVM
+  channel-manager stall (~5-10%), and a guest init panic (~1/10, retried).
+  The vadd executes via the host CUDA bridge daemon; forwarding guest engine
+  work to the physical GPU remains the open backend wall.
 - **Hardware incident + recovery (2026-10-03).** The first emulation-only
   (no-claim) acceptance run exposed real GPU BAR2 PFNs to the guest while the
   host driver owned the device and triggered an uncorrectable ECC Xid, after
@@ -78,13 +83,18 @@ isolation, performance, portability).
    root-cause the shared-management `TURING_COMPUTE_A` wedge; physical cycles
    only with a named hypothesis and supervision.
 2. **Hardening the accepted no-claim path.** The first green guest CUDA
-   acceptance has landed and is holding across repeats; what remains is the
-   intermittent guest-side corruption flake (intermittent guest boot
-   double-faults/UVM teardown oopses; `noclaim32`/`41`; host unaffected),
-   trimming/re-running with the bounded diagnostics, and extending acceptance
-   to a second workload. Guest-truth kernel probes (installed per guest boot)
-   are load-bearing for the GPU-only pool/push-buffer translations; without
-   them the host falls back to content heuristics.
+   acceptance has landed and is holding; the residual classes are (a) corruption
+   via a late first decode behind pump backlog (~3-5% of runs; an experiment
+   deleting the silent pre-rescue fake and holding every unreadable CE entry
+   for a 25 s bounded grace is under live validation), (b) an early stall
+   before any push reaches the service (client waits in
+   `uvm_channel_manager_create`/`channel_pool_add`; `ring_auto=6` vs 26 is the
+   discriminator; ~5-10%, retried by the campaign), and (c) the guest init
+   panic class (~1/10, retried; host unaffected). Remaining work: settle or
+   fail-loud the residual classes, trim/re-run with the bounded diagnostics,
+   and extend acceptance to a second workload. Guest-truth kernel probes
+   (installed per guest boot) are load-bearing for the GPU-only pool/push-buffer
+   translations; without them the host falls back to content heuristics.
 3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side
    data compare; handled fail-closed, parameter default-off).
 4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect
@@ -98,6 +108,113 @@ isolation, performance, portability).
    panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup -
    guest-side only, host stays healthy.
 
+## Options to accelerate debugging toward end-to-end CUDA (candidate techniques; not directives)
+
+> Added 2026-10-05. These are options the project can pick up when it fits the
+> current campaign; they do not redirect the work in progress and are not
+> commitments. Each is a tool/workflow introduction and would need a
+> `docs/eng-ops.md` admission record (scenario + measurable outcome + baseline +
+> evaluation threshold + failure signal) before adoption.
+
+The present issues they target: (a) the residual no-claim failure classes - a
+late-decode corruption (~3-5% of runs: a queued copy's first run can land after
+the client has refilled its staging buffer), an early stall before pushes reach
+the service (~5-10%; the client waits in the UVM channel-manager path), and a
+guest-kernel init panic (~1/10, retried); (b) the physical GR engine-context
+commit wall (external-VAS booking; the shared-management `TURING_COMPUTE_A`
+wedge; the rec-677 soft-0x40 message); (c) the cost of a supervised cycle (a
+wedge currently ends in a reboot). The end-to-end objective is an unmodified
+guest CUDA workload executing on the physical GPU; the nearer objective is a
+settled, repeatable acceptance on the no-claim bridge.
+
+1. **Deterministic fault injection for the service pump.** The pump's hostile
+   interleavings (translation latency, replay/rewind, grace fakes, staging
+   refill) are reachable today only by waiting for a low-probability schedule.
+   Debug knobs that force each one would turn the 3-5% classes into 100%
+   reproductions and let a hostile-interleaving suite run every build.
+   *Helps now:* residual corruption/stall fixes become fast and testable
+   instead of probabilistic. *End-to-end:* removes the replay/fake hazard class
+   from the acceptance path and gives a deterministic regression gate for every
+   future workload. *Status:* not in use (threshold parameters exist; no
+   forced-interleaving surface).
+
+2. **Offline replay and an invariant model of the pump.** The decision logic
+   (hold/retry, once-gate, epoch marks, fakes) is instrumented and mostly
+   isolated; replaying captured push bytes plus translation/release timings
+   through a userspace mirror, and property-checking the invariants
+   (exactly-once work per slot/push generation; no work after the client has
+   observed a release; fakes only after bounded retries and marked consumed),
+   would find holes in minutes instead of supervised cycles. *Helps now:* the
+   residual classes are exactly invariant violations. *End-to-end:* a checked
+   model of the bridge's safety contract is reusable for GR forwarding and
+   future workloads. *Status:* not in use; the simulator workstream is
+   firmware-focused and this would be its host-side sibling.
+
+3. **Structured host tracing instead of printk floods.** Pump decisions are
+   traced as text today; runs have produced hundreds of MB of kernel log, slow
+   to collect and hard to order. ftrace trace events (or `trace_printk`) with a
+   binary ring and a per-run timeline, plus eBPF/bpftrace kprobes on the
+   service functions for latency histograms and stack traces without rebuilds,
+   would give ordered evidence at low overhead. *Helps now:* the residual bugs
+   are ordering bugs. *End-to-end:* a permanent low-overhead tracing surface
+   for every future performance/porting question. *Status:* not in use on the
+   host (guest kprobes are in use).
+
+4. **Guest-memory forensics for the panic/corruption class.** The init panic
+   has no identified writer. The QEMU gdbstub is already listening, so hardware
+   watchpoints on the guest address mapping the corrupted page plus a
+   `dump-guest-memory` at panic would name a CPU writer; a KVM dirty-log view
+   would attribute writes per guest frame; a diagnostic guest kernel with
+   KASAN/KFENCE/slub_debug (product guests stay stock) would catch a
+   guest-software wild write; and because device DMA bypasses CPU watchpoints,
+   host-side checks on every module/bridge write to guest page-table frames are
+   the complement. *Helps now:* converts an unexplained ~1/10 flake into a
+   located bug. *End-to-end:* guest-memory integrity is a prerequisite for
+   multi-tenancy/isolation and for trusting any physical-backend result.
+   *Status:* named as a next step but not executed.
+
+5. **Physical GR wall: runtime metadata, a golden reference, static-analysis
+   throughput.** Three independent moves: (a) snapshot the runtime-inflated
+   logging metadata from a live boot to name the rec-677 soft-0x40 message (the
+   format text is not in the shipped image); (b) use a host RM client as a
+   golden reference - the userspace oracle showed the engine-context controls
+   are not reachable from userspace, so a kernel-side RM client is the plausible
+   path to run the real promote/allocate sequence and diff shapes against our
+   ladder; (c) restore Ghidra headless scripting (batch decompile and xrefs) or
+   use an alternate static-analysis engine for targeted "which check returns
+   0x40" queries - the promote path has multi-KB functions that are impractical
+   to read by hand. *Helps now:* each shortens the GR commit investigation.
+   *End-to-end:* GR execution is the remaining gate between the host-bridge
+   acceptance and real guest compute on the GPU. *Status:* (a) identified, not
+   done; (b) oracle exists, userspace route proven blocked; (c) headless
+   scripting currently unavailable.
+
+6. **Cut the supervised-cycle cost.** A wedged GSP boot currently ends in a
+   supervised reboot, which caps experiment throughput. The tree already has a
+   WPR2-clear utility (BAR0 MMIO) and a designed fail-safe; extending the
+   supervised recovery path to use them, or restarting only the GSP boot ladder
+   instead of the host, would multiply cycles per hour. *Helps now:* every
+   residual-class fix and every GR hypothesis gets more attempts. *End-to-end:*
+   shortens the critical path to physical GR proof and to multi-version/
+   multi-device porting. *Status:* utilities exist; not part of the routine
+   recovery flow.
+
+7. **Concurrency and robustness checks on the host module.** KCSAN (data
+   races), lockdep (the stuck-daemon class seen after the service thread oops),
+   KFENCE (the slot-reuse use-after-free class), and sparse/smatch over the
+   large service file would catch hazard classes before they surface as oopses
+   in supervised runs. *Helps now:* the module has had both a use-after-free
+   and a lock-hold failure. *End-to-end:* hardening needed for long-running
+   multi-VM operation. *Status:* not in use.
+
+8. **Wire the existing analysis tools into the loop.** The failure classifier
+   and guest-init monitor exist; running them automatically per cycle and
+   tracking acceptance rates and hazard-branch coverage per commit would turn
+   the campaign into a regression gate and make sure the once-gate/fake/epoch
+   paths are exercised deliberately rather than by luck. *Helps now:* rates are
+   currently hand-tallied. *End-to-end:* the acceptance suite becomes CI for
+   future workloads and versions. *Status:* tools exist; not automated.
+
 ## Simulator workstream (risc-v-gsp-rm-debugger)
 
 Development-only tooling for simulating and inspecting GSP firmware runtime
@@ -156,6 +273,7 @@ chain succeeds.
 
 | ID | Milestone | When | State |
 |---|---|---|---|
+| MS-20261005-01 | Hardening the accepted no-claim path (hazard classes closed; rates up) | 2026-10-05 overnight | per-op once bitmap + epoch/content-hash gate + grace 15 s (`b645cf6b`, `3b094c2d`, `c3cd4631`); first-attempt 86% -> ~94-95%; fakes/run 3.39 -> 0.61; live experiment `97eba5b7` campaign 229-236 at 3/3 |
 | MS-20261004-04 | **First green end-to-end guest CUDA acceptance** | 2026-10-04 ~16:20 | `noclaim38/39/40` `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, deterministic, committed `cf6ff542`; 14/15 green in the repeat campaign |
 | MS-20261004-01 | No-claim bridge reaches the guest `vectorAdd` launch | 2026-10-04 ~14:45 | guest-truth pins + DtoH mail read-back fix + FB 512 MiB; `cudaMalloc` unblocked; no `STATUS OK` yet |
 | MS-20261004-02 | Latent bridge pending-slot oops found, fixed; reboot | 2026-10-04 ~14:45-15:00 | `cubb_pend_release` slot reuse -> NULL deref; fixed `85539844...`; approved reboot restored the GPU |
@@ -187,6 +305,22 @@ chain succeeds.
 
 ## Milestone log - current frontier and recent milestones (granular IDs in each entry)
 
+- `MS-20261005-01` **Hardening the accepted no-claim path (2026-10-05
+  overnight).** Three kernel-side fixes closed the replay/fake hazard classes
+  after the first acceptance: (1) a held-entry retry must not re-run completed
+  ops - per-slot executed-op bitmap plus a complete flag, persisted in the
+  snapshot (`b645cf6b`); (2) a slot consumed by a grace fake is epoch-marked so
+  its work can never run later against a refilled staging buffer, and the
+  once gate gained an FNV content witness so a rewritten push is treated as
+  new (`3b094c2d`); (3) the CE pending-entry grace widened 8 s -> 15 s so a
+  copy's first run precedes its fallback release (`c3cd4631`). Measured:
+  first-attempt acceptance 86% -> ~94-95% across campaigns, fakes/run
+  3.39 -> 0.61. Two further experiments were tested and reverted (a
+  defer-the-release branch missed the late-first-decode path); a third (hold
+  every unreadable CE entry, delete the silent pre-rescue fake, 25 s grace,
+  module `97eba5b7`) is live at check with campaign 229-236 at 3/3 green.
+  Residual classes: late-decode corruption (~3-5%), early UVM channel-manager
+  stall (~5-10%), guest init panic (~1/10, retried).
 - `MS-20261004-04` **First green end-to-end guest CUDA acceptance (2026-10-04
   ~16:20).** After `noclaim32` (guest double-fault) and `noclaim33` (vadd data
   corrupt), two data-path defects were root-caused with bounded