open-iov.git, branch stallfree

vgsp: stale-replay leg is now strictly non-harmful (r235 second-client death)

2026-10-18T20:01:10+00:00

r235-2 mechanism: at R2's ctx-init one pass fired 8 'stale-replay rewind=1' lines (waited_ms=3491, incl. fresh channels with sub=6 put=1) and set each channel's pin-batch rewind cooldown to +24 s. The rewind is the retry path for a fresh context (its init pushes are decoded before the pins land; the rewind re-decodes them once the translations exist), so disabling it killed R2 exactly like it would every next client (r233 P2+, r235 R2-R4). R1 (working) fired zero tokens. The 3 s window misread a normal init oscillation (rewind replaying a just-consumed entry) as a rewind livelock. - Removed the rewind cooldown entirely: wd_rewind_off_until field, write-back, watchdog clear/set and the svc_handle_map_add skip are gone. The pin-batch rewind is never manipulated by any recovery. - stale-replay now requires SVC_GR_STALE_REPLAY_MS (10 s) of stalled position with consumption happening (a genuine livelock; r225 wedge was 42 s+), keeps its window accumulating until then, and clamps last_get = put at most once per put epoch (wd_epoch_done). - Token updated: '... reason=stale-replay reset=0 clamp=1 ...' (no rewind manipulation; clamp is bookkeeping, a rewind may undo it). Epoch legs (residual/dead-range), stale-entry, newbind window, pending-map/rebuild path, kill-recovery, r220 guard and the m181 rule are unchanged. make kernel green.

vgsp: bounded newbind/rebuild diagnostic tokens (r234 discrimination)

2026-10-18T19:48:15+00:00

r233: ctaid_probe attempt-1 completed (newbind fix works) but the NEXT client (P2, fresh process after P1's CLEAN teardown+flush) stalled with zero tokens. witness-r233.log shows why: after the last FREE (180966.07) and rebuild #40 (180967.07) the guest sends ZERO traffic for 9+ s (only 'completion hint rc=-2'); no channel binding ever exists, so no entry-level recovery (pending-map/chanswitch/stale-entry) can apply. Pattern across boots: a CLEAN client exit is followed by a pre-registration stall; a KILLED client is followed by a working client. Diagnostics added (bounded 16/session each): - 'GR-STALL-newbind h=... put=... get=... action=window reason=stale-entry-newbind' when a binding's freshness window is armed (proves the reconcile runs per binding, not just once). - 'GR-STALL-rebuild consumed=1 by=svc_thread ...' when the service loop consumes svc.rebuild_now (proves the forced rebuild is taken; rules out the stale-flag hypothesis). wd_chanswitch/wd_newbind_until are per-channel/per-binding (armed on every !same_gen registration) - no global-state stomping between clients. make kernel green.

vgsp: channel-reuse reconciliation + first-attempt readiness (fresh on 54a41db2)

2026-10-18T19:41:21+00:00

Net port of the stallfree work onto master without the reverted hint-stop: - Channel-reuse reconciliation, provably safe: svc_chan_registration marks a genuinely new binding (!same_gen) with wd_chanswitch; the first pump pass drops opdef slots bound to the reused handle ONLY when their first-pending stamp predates the reconcile instant (time_before), so a slot the fresh binding creates (after registration, at/after the reconcile) can never be touched; logs reason=stale-entry-chanswitch when the ring already carries content. Captures stay enabled (the fresh client's own entries may already be at slot 0). - First-attempt readiness (r229/r231: attempt 1 stalls, attempt 2 works once the 10 s rebuild cadence caught up): a genuinely new binding gets a 3 s freshness window (wd_newbind_until, SVC_GR_NEWBIND_MS) in which the pump (a) does NOT silently consume a GR entry that decoded nothing (no release/QMD/ops, non-replay) - it holds it and requests an immediate rebuild (svc.rebuild_now, consumed at the top of the service loop before the next pump), and (b) does the same for a GR entry whose page is unmapped. The entry re-evaluates with resolved pages inside the window, so the launch executes on the FIRST attempt; after the window entries consume as before (<=3 s extra on first contact). - Token: 'vgsp-gr: GR-STALL-pending-map h=... va=... entry=... action=rebuild reason=stale-entry-newbind' (bounded 16/session). - No hint-stop (master's revert already removed it; kept absent - the re-post window is the delivery bridge for late-registered events). - f0789007 kill-recovery, the r220 hold guard and the m181 arm rule are untouched (already present on this base). make kernel green (0 errors, pre-existing warnings only).

vgsp: fail4 r231 - READ-ORDER guard/witness + 32KB copy-resolution witness

2026-10-18T19:30:38+00:00

r231 probe: RAWA (fresh 64 KB upload -> straight D2H readback, no kernel) returned foreign deterministic content (0x437F0000-family ramps) where the uploaded ints 0..16383 must be, and PROBE2 == PROBE2B on that same content. The upload/readback round trip inside the emulation is therefore not content-preserving - before any engine execution. Two candidate mechanisms, both now witnessed: - READ-ORDER: a device-src copy reading a VA whose device-dst write is still inside the m169 pin-guard window (the writer's bytes land only when the guard resolves) - the read executes first. READ-ORDER-WIT logs the coincidence; with sema_read_order_guard=1 (new param, default 0) the read is deferred through the existing bounded-hold re-decode so it runs after the write. Opt-in pending an A/B proving no green regression. - RESOLUTION DIVERGENCE: the dst (H2D) resolution consults the ledger before svc_op_translate while the src (D2H) resolution does the opposite, so a stale ledger landing can make the writer and reader touch different pages. The sema-bigcopy proof (va->pa/aperture/provenance for BOTH sides) now fires from 32 KB (was 1 MB), so the probe-class copies are on record. Build green; module param registered and visible. (cherry picked from commit bfe5d295dd3922c0dbcd0bd1a29912a02acb58ec)

Revert "vgsp: channel-reuse reconciliation + hint-flood stop (GR-STALL-stale-entry-chanswitch)"

2026-10-18T19:22:36+00:00

This reverts commit 21eb5bb0582aaa2409601a1a1466f7bc72841647.

vgsp: channel-reuse reconciliation + hint-flood stop (GR-STALL-stale-entry-chanswitch)

2026-10-18T19:12:27+00:00

r229a evidence: victory_mail OK -> ctaid_probe attempt 1 printed DEVICE and never completed (killed at 50 s); attempt 2 (fresh context) worked. The retained window shows mail's teardown + session flush (gen 3) then, for the whole first-attempt window, no channel registration and only the host 'completion hint rc=-2' flood (~0.5 s cadence, window reopened by the completion-apply sites). No GR lines, no stale-* lines: the first launch never reached the sema walk for the new binding. - svc_thread hint: on rc == -2 (no event context to post to) stop reopening svc.post_hint_until; the next real completion reopens it. The flood was pure noise and obscured the real state. - Channel-reuse reconciliation: svc_chan_registration marks a genuinely new binding (!same_gen) with wd_chanswitch; the channel's first pump pass drops every opdef slot bound to that handle (a dead context's withheld sets must not ride the reused channel - the r228 opdef-hold class) and, when the ring already carries content (put > 1), logs 'GR-STALL-stale-wedge ... reason=stale-entry-chanswitch'. Captures stay enabled: the fresh client's own entries may already be at slot 0, and suppressing them would lose exactly the first launch this protects; a stale launch re-walk stays bounded by gsp_phys_gr_wait_max_ms and the m181 put-drop recovery. r220 hold guard and the m181 arm rule untouched. Note: the phase2 witness for r229 was not readable at commit time (/tmp/opencode/witness-r229-phase2.log absent); r229a is boot-side and post-flush, so the attempt-1 window shows the guest sent no RM/channel traffic for ~9.5 s - the reconciliation removes the host-side reused- handle state classes; r230 will show whether the first-attempt launch now reaches the walk. make kernel green. (cherry picked from commit 21eb5bb0582aaa2409601a1a1466f7bc72841647)

vgsp: fail4 r229 - REC-DUMP per-rec VA->HPA->reloc census at submit

2026-10-18T19:12:26+00:00

The r229 ctaid_probe came back as: PROBE1 (coordinate map) perfect, PROBE2 (flat A read) wrong at every element - band 0 read A's own values at wildly wrong offsets (16382/16383 = A's last elements), bands >= 1 read float-typed data. Writes are flawless; the engine's READ of a freshly H2D'd 64 KB buffer returned foreign content, so before anything else the next capture needs the exact (va, hpa, reloc_va) triples the engine will read for that launch. REC-DUMP logs the submit's rec table at launch build time: all recs for small launches (n_data <= 64, the probe class), first/last 8 for larger ones, gated by gsp_phys_gr_copyout_dump. Comparing the A buffer's recs against the H2D landing (and against each other: distinct VAs collapsing onto one HPA, or HPAs outside the uploaded range) names the wrong-read source directly. Logging-only; greens safe. (cherry picked from commit f83ef995c17fba896950cc7f1818c65ac8839ac4)

vgsp: stalled-entry detector + opdef-hold deadline fix (GR-STALL-stale-entry)

2026-10-18T19:04:50+00:00

r228 witness: entry 82 of channel 0x5c000059 stayed pending with put=84 get=82 last=82 frozen for ~10 s (its 1 MiB op operand never resolved, sema-optrace rc=-2) while 'sema-opdef: hold entry=82' and 'opdef-clear caller=4 entry=82' repeated: svc_opdef_drop_stale cleared the opdef slot on every rewind replay that processed a different entry of the channel, so the 10 s deadline restarted and the hold re-armed forever (opdef_held climbing, no EXPIRED). No GR-STALL line fired because this is below the GR-publish layer: the launch never reached a capture/submit. - svc_opdef_drop_stale: only a slot whose entry was CONSUMED this put epoch (slot_done_epoch bit set by svc_replay_note) is delivered-late and dropped. A rewind re-decoding an unconsumed entry no longer resets its hold deadline, so the bounded 10 s flush/expiry is reachable again. - svc_gr_stale_wedge_check: new stale-entry leg (reason=stale-entry, entry=1) for a genuinely stalled pending entry: GR channel, put > get, position AND submissions static for the 3 s window, and no bounded owner active (no defer_since grace, no no-capture window, no live m150 hold, no opdef slot for the stuck entry). Action is the canonical unstuck: last_get = 0 + wd_nocap_put, decode releases only (forward-only, no re-capture/re-forward), once per put epoch. No residual publish/arm (m181-only rule); r220 hold guard untouched. Green safety: a green pending entry either advances submissions (consume within ms) or sets one of the bounded owners the leg gates on; the stale-entry leg therefore needs all of {put > get, 3 s position stall, 0 consumption, no owner} to fire. make kernel green. (cherry picked from commit f0789007c49c7c6e01da5643d56ceff0b2b83819)

vgsp: stale-wedge recovery round 2 - dead-range reset, no residual publish

2026-10-18T18:53:46+00:00

r227 field result: the watchdog fired 10x (reason=stale-epoch, arm+publish) but the wedge survived, and victory_mail after the stalled rerun also hung. Root causes fixed here: 1. Do NOT arm/publish on a residual epoch. Arming let the next pass's svc_counter_publish write the residual put (35/101) into the fresh client's completion word - the r207/mm2 far-jump wedge (UVM_GPU_SEMAPHORE_MAX_JUMP), the exact poison gr_pub_armed exists to prevent. The epoch reset now only zeroes a word the HOST published this session (counter_last != 0, m181 semantics); the arm stays at m181's put-drop and the pristine put <= 1 discovery. 2. The stale-replay leg could never fire: its predicate required wd_since (set only when a candidate armed the window) - a circular dependency. The window now arms on either candidate and the action is decided at fire time with proof of the livelock (submissions moved during the stalled window). 3. Dead-range reset (reason=stale-dead-range): when the host caught up but the whole consumed range delivered nothing (counter_last == 0, submissions >= 8) and the epoch was never armed, rewind last_get to slot 0 so the fresh session's own entries (which start at slot 0) can be walked, and force replay for those dead slots (wd_nocap_put) so the re-walk decodes releases only (idempotent, forward-only) and never re-captures/re-forwards a dead launch. Once per put epoch (wd_epoch_done), cleared when new producer work arrives. 4. Log extended: reason=stale-epoch|stale-dead-range|stale-replay plus reset=/rewind=/cpa=/rpa= so the next run shows exactly which recovery ran and whether a publish target even exists. Green safety: both epoch legs require !gr_pub_armed (a channel that never proved a fresh epoch); a green mid-stream-discovered unarmed channel can at worst pay one forced-replay re-read of its consumed range per put epoch (no capture, no word writes). The r220 hold-storm guard and the rewind cooldown are untouched. make kernel green. (cherry picked from commit d2ba3cadecf1ca0e72358024f80dbe7a8c468842)

vgsp: fail4 r227 - INPUT-ALIAS-WIT rec census + RETIRE-WIT rec-array witness

2026-10-18T18:53:45+00:00

r227 verdict inputs: GRID-WIT == the guest's own geometry for every kernel (gemm_nn w12=0x10 w13=0x4, gemm_nt w12=0x4, softmax w12=0x40, block dims in w18/w19) - no QMD geometry emission bug; the old consecutive-pair IN-ALIAS stayed silent. Two new witnesses for the next capture: - INPUT-ALIAS-WIT: per-launch census of the copy-in recs (the exact VA -> HPA -> content triples the engine reads, inputs included; the fill already covers them as recs). Every non-zero rec is compared against the last 64: distinct VA -> same HPA is the resolver-aliasing smoking gun; distinct HPA with equal content is the upstream-write-clone class. Zero pages excluded on both sides. Replaces the consecutive-only check. - RETIRE-WIT: one line per launch at the retirement wait with n_data, n_tracked and the first/last recs' va/reloc_va/bi. r227's numbers are impossible under this source (retire n_tracked 4/4/4/1/1/16/16 vs copy-in/copy-out 1482/1418/1354/913/864/882/1290 recs for the same launch), so either the rec array genuinely aliases reloc VAs at run time or the deployed module differs from this tree; RETIRE-WIT shows which in one line (sequential in-region rels + n_data=1482 with n_tracked=4 would prove the former). Logging-only; no behaviour change; greens safe. Build green. (cherry picked from commit 1dbde7967bda4ba51dadf8a96e8aeaacc44c5575)