open-iov.git/kernel/gsp_sema_service.c, branch stallfree

vgsp: stale-replay leg is now strictly non-harmful (r235 second-client death)

2026-10-18T20:01:10+00:00

r235-2 mechanism: at R2's ctx-init one pass fired 8 'stale-replay rewind=1' lines (waited_ms=3491, incl. fresh channels with sub=6 put=1) and set each channel's pin-batch rewind cooldown to +24 s. The rewind is the retry path for a fresh context (its init pushes are decoded before the pins land; the rewind re-decodes them once the translations exist), so disabling it killed R2 exactly like it would every next client (r233 P2+, r235 R2-R4). R1 (working) fired zero tokens. The 3 s window misread a normal init oscillation (rewind replaying a just-consumed entry) as a rewind livelock. - Removed the rewind cooldown entirely: wd_rewind_off_until field, write-back, watchdog clear/set and the svc_handle_map_add skip are gone. The pin-batch rewind is never manipulated by any recovery. - stale-replay now requires SVC_GR_STALE_REPLAY_MS (10 s) of stalled position with consumption happening (a genuine livelock; r225 wedge was 42 s+), keeps its window accumulating until then, and clamps last_get = put at most once per put epoch (wd_epoch_done). - Token updated: '... reason=stale-replay reset=0 clamp=1 ...' (no rewind manipulation; clamp is bookkeeping, a rewind may undo it). Epoch legs (residual/dead-range), stale-entry, newbind window, pending-map/rebuild path, kill-recovery, r220 guard and the m181 rule are unchanged. make kernel green.

vgsp: bounded newbind/rebuild diagnostic tokens (r234 discrimination)

2026-10-18T19:48:15+00:00

r233: ctaid_probe attempt-1 completed (newbind fix works) but the NEXT client (P2, fresh process after P1's CLEAN teardown+flush) stalled with zero tokens. witness-r233.log shows why: after the last FREE (180966.07) and rebuild #40 (180967.07) the guest sends ZERO traffic for 9+ s (only 'completion hint rc=-2'); no channel binding ever exists, so no entry-level recovery (pending-map/chanswitch/stale-entry) can apply. Pattern across boots: a CLEAN client exit is followed by a pre-registration stall; a KILLED client is followed by a working client. Diagnostics added (bounded 16/session each): - 'GR-STALL-newbind h=... put=... get=... action=window reason=stale-entry-newbind' when a binding's freshness window is armed (proves the reconcile runs per binding, not just once). - 'GR-STALL-rebuild consumed=1 by=svc_thread ...' when the service loop consumes svc.rebuild_now (proves the forced rebuild is taken; rules out the stale-flag hypothesis). wd_chanswitch/wd_newbind_until are per-channel/per-binding (armed on every !same_gen registration) - no global-state stomping between clients. make kernel green.

vgsp: channel-reuse reconciliation + first-attempt readiness (fresh on 54a41db2)

2026-10-18T19:41:21+00:00

Net port of the stallfree work onto master without the reverted hint-stop: - Channel-reuse reconciliation, provably safe: svc_chan_registration marks a genuinely new binding (!same_gen) with wd_chanswitch; the first pump pass drops opdef slots bound to the reused handle ONLY when their first-pending stamp predates the reconcile instant (time_before), so a slot the fresh binding creates (after registration, at/after the reconcile) can never be touched; logs reason=stale-entry-chanswitch when the ring already carries content. Captures stay enabled (the fresh client's own entries may already be at slot 0). - First-attempt readiness (r229/r231: attempt 1 stalls, attempt 2 works once the 10 s rebuild cadence caught up): a genuinely new binding gets a 3 s freshness window (wd_newbind_until, SVC_GR_NEWBIND_MS) in which the pump (a) does NOT silently consume a GR entry that decoded nothing (no release/QMD/ops, non-replay) - it holds it and requests an immediate rebuild (svc.rebuild_now, consumed at the top of the service loop before the next pump), and (b) does the same for a GR entry whose page is unmapped. The entry re-evaluates with resolved pages inside the window, so the launch executes on the FIRST attempt; after the window entries consume as before (<=3 s extra on first contact). - Token: 'vgsp-gr: GR-STALL-pending-map h=... va=... entry=... action=rebuild reason=stale-entry-newbind' (bounded 16/session). - No hint-stop (master's revert already removed it; kept absent - the re-post window is the delivery bridge for late-registered events). - f0789007 kill-recovery, the r220 hold guard and the m181 arm rule are untouched (already present on this base). make kernel green (0 errors, pre-existing warnings only).

vgsp: fail4 r231 - READ-ORDER guard/witness + 32KB copy-resolution witness

2026-10-18T19:30:38+00:00

r231 probe: RAWA (fresh 64 KB upload -> straight D2H readback, no kernel) returned foreign deterministic content (0x437F0000-family ramps) where the uploaded ints 0..16383 must be, and PROBE2 == PROBE2B on that same content. The upload/readback round trip inside the emulation is therefore not content-preserving - before any engine execution. Two candidate mechanisms, both now witnessed: - READ-ORDER: a device-src copy reading a VA whose device-dst write is still inside the m169 pin-guard window (the writer's bytes land only when the guard resolves) - the read executes first. READ-ORDER-WIT logs the coincidence; with sema_read_order_guard=1 (new param, default 0) the read is deferred through the existing bounded-hold re-decode so it runs after the write. Opt-in pending an A/B proving no green regression. - RESOLUTION DIVERGENCE: the dst (H2D) resolution consults the ledger before svc_op_translate while the src (D2H) resolution does the opposite, so a stale ledger landing can make the writer and reader touch different pages. The sema-bigcopy proof (va->pa/aperture/provenance for BOTH sides) now fires from 32 KB (was 1 MB), so the probe-class copies are on record. Build green; module param registered and visible. (cherry picked from commit bfe5d295dd3922c0dbcd0bd1a29912a02acb58ec)

Revert "vgsp: channel-reuse reconciliation + hint-flood stop (GR-STALL-stale-entry-chanswitch)"

2026-10-18T19:22:36+00:00

This reverts commit 21eb5bb0582aaa2409601a1a1466f7bc72841647.

vgsp: channel-reuse reconciliation + hint-flood stop (GR-STALL-stale-entry-chanswitch)

2026-10-18T19:12:27+00:00

r229a evidence: victory_mail OK -> ctaid_probe attempt 1 printed DEVICE and never completed (killed at 50 s); attempt 2 (fresh context) worked. The retained window shows mail's teardown + session flush (gen 3) then, for the whole first-attempt window, no channel registration and only the host 'completion hint rc=-2' flood (~0.5 s cadence, window reopened by the completion-apply sites). No GR lines, no stale-* lines: the first launch never reached the sema walk for the new binding. - svc_thread hint: on rc == -2 (no event context to post to) stop reopening svc.post_hint_until; the next real completion reopens it. The flood was pure noise and obscured the real state. - Channel-reuse reconciliation: svc_chan_registration marks a genuinely new binding (!same_gen) with wd_chanswitch; the channel's first pump pass drops every opdef slot bound to that handle (a dead context's withheld sets must not ride the reused channel - the r228 opdef-hold class) and, when the ring already carries content (put > 1), logs 'GR-STALL-stale-wedge ... reason=stale-entry-chanswitch'. Captures stay enabled: the fresh client's own entries may already be at slot 0, and suppressing them would lose exactly the first launch this protects; a stale launch re-walk stays bounded by gsp_phys_gr_wait_max_ms and the m181 put-drop recovery. r220 hold guard and the m181 arm rule untouched. Note: the phase2 witness for r229 was not readable at commit time (/tmp/opencode/witness-r229-phase2.log absent); r229a is boot-side and post-flush, so the attempt-1 window shows the guest sent no RM/channel traffic for ~9.5 s - the reconciliation removes the host-side reused- handle state classes; r230 will show whether the first-attempt launch now reaches the walk. make kernel green. (cherry picked from commit 21eb5bb0582aaa2409601a1a1466f7bc72841647)

vgsp: stalled-entry detector + opdef-hold deadline fix (GR-STALL-stale-entry)

2026-10-18T19:04:50+00:00

r228 witness: entry 82 of channel 0x5c000059 stayed pending with put=84 get=82 last=82 frozen for ~10 s (its 1 MiB op operand never resolved, sema-optrace rc=-2) while 'sema-opdef: hold entry=82' and 'opdef-clear caller=4 entry=82' repeated: svc_opdef_drop_stale cleared the opdef slot on every rewind replay that processed a different entry of the channel, so the 10 s deadline restarted and the hold re-armed forever (opdef_held climbing, no EXPIRED). No GR-STALL line fired because this is below the GR-publish layer: the launch never reached a capture/submit. - svc_opdef_drop_stale: only a slot whose entry was CONSUMED this put epoch (slot_done_epoch bit set by svc_replay_note) is delivered-late and dropped. A rewind re-decoding an unconsumed entry no longer resets its hold deadline, so the bounded 10 s flush/expiry is reachable again. - svc_gr_stale_wedge_check: new stale-entry leg (reason=stale-entry, entry=1) for a genuinely stalled pending entry: GR channel, put > get, position AND submissions static for the 3 s window, and no bounded owner active (no defer_since grace, no no-capture window, no live m150 hold, no opdef slot for the stuck entry). Action is the canonical unstuck: last_get = 0 + wd_nocap_put, decode releases only (forward-only, no re-capture/re-forward), once per put epoch. No residual publish/arm (m181-only rule); r220 hold guard untouched. Green safety: a green pending entry either advances submissions (consume within ms) or sets one of the bounded owners the leg gates on; the stale-entry leg therefore needs all of {put > get, 3 s position stall, 0 consumption, no owner} to fire. make kernel green. (cherry picked from commit f0789007c49c7c6e01da5643d56ceff0b2b83819)

vgsp: stale-wedge recovery round 2 - dead-range reset, no residual publish

2026-10-18T18:53:46+00:00

r227 field result: the watchdog fired 10x (reason=stale-epoch, arm+publish) but the wedge survived, and victory_mail after the stalled rerun also hung. Root causes fixed here: 1. Do NOT arm/publish on a residual epoch. Arming let the next pass's svc_counter_publish write the residual put (35/101) into the fresh client's completion word - the r207/mm2 far-jump wedge (UVM_GPU_SEMAPHORE_MAX_JUMP), the exact poison gr_pub_armed exists to prevent. The epoch reset now only zeroes a word the HOST published this session (counter_last != 0, m181 semantics); the arm stays at m181's put-drop and the pristine put <= 1 discovery. 2. The stale-replay leg could never fire: its predicate required wd_since (set only when a candidate armed the window) - a circular dependency. The window now arms on either candidate and the action is decided at fire time with proof of the livelock (submissions moved during the stalled window). 3. Dead-range reset (reason=stale-dead-range): when the host caught up but the whole consumed range delivered nothing (counter_last == 0, submissions >= 8) and the epoch was never armed, rewind last_get to slot 0 so the fresh session's own entries (which start at slot 0) can be walked, and force replay for those dead slots (wd_nocap_put) so the re-walk decodes releases only (idempotent, forward-only) and never re-captures/re-forwards a dead launch. Once per put epoch (wd_epoch_done), cleared when new producer work arrives. 4. Log extended: reason=stale-epoch|stale-dead-range|stale-replay plus reset=/rewind=/cpa=/rpa= so the next run shows exactly which recovery ran and whether a publish target even exists. Green safety: both epoch legs require !gr_pub_armed (a channel that never proved a fresh epoch); a green mid-stream-discovered unarmed channel can at worst pay one forced-replay re-read of its consumed range per put epoch (no capture, no word writes). The r220 hold-storm guard and the rewind cooldown are untouched. make kernel green. (cherry picked from commit d2ba3cadecf1ca0e72358024f80dbe7a8c468842)

vgsp: stale-epoch/stale-replay wedge watchdog (GR-STALL-stale-wedge)

2026-10-18T17:10:07+00:00

r225 evidence: after a client was killed mid-run, a fresh client's GR channel (0x5c000059) was registered with a residual USERD put=101 from the dead session. The m181 fresh-put-drop recovery never fired (the fresh session never wrote a put - it was stuck at ctx-init, polling MC_SERVICE_INTERRUPTS once a second), the completion-counter publish stayed withheld (gr_pub_armed false), and the pin-batch rewind (SVC_PIN_REWIND=8) pinned last_get at 93 on every injection batch: the entry loop re-walked slots 93..100 forever, decoding nothing (rel=0, no capture) and consuming with zero delivery. No capture -> no request -> no bounded submit -> zero GR-STALL lines; the launch never reached the bounded path because the session epoch itself was wedged. - svc_gr_stale_wedge_check (pump pass, before the entry loop, GR only): tracks (last_get, put) per channel and fires after a 3 s no-progress window with two signatures: 1. stale-epoch: residual put (put > 1, publish withheld, host believes the ring fully consumed): arm gr_pub_armed and void the withheld publish state (m181 semantics, counter_last-gated zeroing; no last_get rewind - the proven m181 rewind stays with its put-drop signature), so the fresh client's ctx-init wait gets its host edge. 2. stale-replay: last_get pinned behind put while put == get and the loop consumed entries during the window (the rewind re-opening an already-consumed window): drop the window (last_get = put) and put the pin-batch rewind on a bounded cooldown (wd_rewind_off_until, cleared by new work) so the futile retry cannot pin the pump. - svc_handle_map_add: channels under the rewind cooldown are not rewound. - New per-channel watchdog state written back in the snapshot list (GRFIX NOTE rule); new diag token gr_wedge (drain line); new greppable line 'vgsp-gr: GR-STALL-stale-wedge h=... va=... last=... put=... get=... sub=... waited_ms=... action=recover reason=...'. Green safety: fresh channels have put <= 1 (signature 1 inert) and consume their first entries within milliseconds (submissions > 0); signature 2 needs a stalled (last_get, put) pair for 3 s with no pending producer work AND entries consumed during the window, which only the rewind re-opening of a consumed window produces. make kernel green; only pre-existing warnings. (cherry picked from commit 6aa5709f57cc1d91a01205042d973d5b203c9bec)

vgsp: fail4 discrimination tooling - copy-out page witnesses + full-boot bigcopy resolution budget

2026-10-18T16:10:03+00:00

Four guest CUDA clients fail all-bad while four pass; the aggregate counters cannot separate 'engine result never produced' from 'produced but written to a different guest page than the client's read resolves'. Add the two per-page witnesses that correlate the write side with the read side: - gsp_phys_gr_copyout_dump (booter, default 1): per-changed-page OUT-PAGE-WIT va= hpa= first8/last8=, bounded 12 pages per launch, plus one RELOC-CENSUS line per launch (gaps vs late-resolved words vs late pages). - sema_bigcopy_log (sema service, default 24, was a hardcoded per-boot budget): the svc_copy large-copy proof already logs both operands' resolved VA -> pa/aperture/provenance and the first 16 bytes each side; raising the budget past the boot's head makes the failing clients' H2D/D2H resolutions visible. Discrimination test for the next cycle: same device VA on OUT-PAGE-WIT (copy-out HPA) and sema-bigcopy (D2H source pa) with different PAs proves the produced-but-misplaced class; equal PAs redirects to the readback content/engine-input path. (cherry picked from commit 78ea8b7807231ac418a17b85c29ed1845e1b0062)