Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=lottery-m156-gate-attempt 2026-10-15T21:50:57+00:00 2026-10-15T21:50:57+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T21:50:57+00:00 urn:sha1:eee9223116692285dece42568defbecb2e4e0cdf Companion to the backend witness gate. A data-bearing forwarded GR launch's completion release is now authored only when an executor was witnessed (backend success = retirement wait observed writes, or the CUDA bridge owns it, or an engine fault was delivered as the loud edge). When the hold resolves with no witness, the release is withheld and the channel's error notifier + event are delivered so the client observes a launch failure instead of stale/zero output (r150 big-copy, r155 streams2 STREAMS bad=32768). - svc_gr_rel_witnessed() gate at the two hold-resolution sites (svc_fwd_hold_expire_check, svc_fwd_hold_poll failure branch); - svc_gr_failclosed_deliver() reuses the fault-deliver notifier tail (record shape + notifier_err latch) with the host-detected marker; - a queued data-bearing GR launch now takes the hold even when the pass recorded no release of its own, so the deferred completion release always has an author (its success edge) and a withholder (its failure edge); - the GR hold bound is raised to 40 s under the gate so the cold-context recovery chain (5 s + re-arm + 5 s + 3x2 s retries) can publish its outcome before the watchdog fires; - new gr_failclosed counter in the work_guard summary line. gsp_phys_rmcall_gr_failclosed=0 restores the historical behavior with one switch. Build: make -j8 (kernel), green. 2026-10-15T21:50:52+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T21:50:52+00:00 urn:sha1:de91d640757afb40b1f78d659399a5cff3e540c6 The engine's QMD/channel completion ack is a front-end retirement, not an execution witness. r155 live evidence: streams2's launch logged qmd_rel=6/ch_rel=0xc0ffee02 while the retirement wait saw '0/205 pages changed' on the initial attempt and all three outcome retries; the backend then returned success and the client read all-zero output (STREAMS bad=32768). The r152 matmul/smiley boots show the same signature. For a data-bearing launch (resolved operand pages + working bounce window): - new param gsp_phys_rmcall_gr_failclosed (default 1) forces release0 to the module semaphore (the m64 rel-mod shape), so the engine's ack cannot reach the guest's release word before an executor is witnessed; - when the retirement wait still saw zero writes after the bounded retries, the backend returns -EIO instead of success and logs 'FAIL-CLOSED: ... success withheld', so the sema service can withhold the guest release and fail loudly (companion commit). Set gsp_phys_rmcall_gr_failclosed=0 to restore the historical fail-open behavior. Build: make -j8 (kernel), green. 2026-10-15T16:41:14+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T16:41:14+00:00 urn:sha1:ecfb1278086f98161402f93bca8d3df602fbcb45 r154: after the second client's idle period the FECS saved/unloaded the context (SKED rec: cwd0=0x12486 fecs_cur=b0633ee3 -> cwd0=0x2086 fecs_cur=0 and no change ever after) and every later forwarded launch acked (qmd_rel/ch_rel) while dispatching nothing - including all three outcome retries with fresh ring entries (0/205 changed, 720 ms no-writes, x4). Re-issuing only GPFIFO_SCHEDULE+TSG_GPFIFO_SCHEDULE+doorbell did not revive dispatch. The extended re-arm now replays the mode-35 lane-enable tail first (GR_SET_CTXSW_PREEMPTION_MODE CILP + SET_TIMESLICE + DEVICE_SCHED - the proven-accepted stage-4 sequence, HF-3 design), then the schedules and the doorbell. Poison variants (explicit PREEMPT/tail, BIND, PROMOTE) remain excluded. 2026-10-15T16:08:53+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T16:08:53+00:00 urn:sha1:e07c62b1fab2f2dbc9981458c0697829b287b01e r153 root cause (3rd-client userspace spin for the rest of the boot): the guest scanner's merged CE-window run (va=0x121000000 len=0x40000; the merged length varies per session) covered a CE channel's GPFIFO ring VA 0x121010000. svc_ipin_store took precedence in the ring read path, so the ring read the guest page at the wrong PA (0x102e02000): an all-zero entry was silently consumed, the pool release was never authored (pushes frozen at 1643, crel frozen), and victory_matmul spun in userspace for 50+ min. The ring VA translation must come from the channel's own discovery path; an injected pin inside any live channel's ring range is always wrong - drop it (counted + logged, first 8 per session). Validated on the next cycle: 'injected pin ... DROPPED' lines plus 'entry[1] raw!=0', advancing pushes, MATMUL bad=0 STATUS OK. 2026-10-15T14:44:19+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:44:19+00:00 urn:sha1:02249afd91f4ee02589a07657d130b9b9411debd Live r153 evidence: the highest-reloc data entry (bi=204) is a UVM-internal buffer that correct launches never write, so the 'top operand untouched' condition fired on correct executions and re-executed them up to 3x - a correctness hazard for side-effecting kernels (atomics/printf) and pure waste otherwise. The honest execution witness is the retirement wait's changed count; only zero-writes retries now (safe by construction: a kernel that wrote nothing has nothing to duplicate). Read-only kernels pay the bounded extra polls. 2026-10-15T14:26:45+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:26:45+00:00 urn:sha1:5a446c10c3ffa9180328180318163b46cb00a16d The server refuses multi-page client memdescs for these buffers (a 48-page chunk aborted stage 2 with rpc_result 0x1f + firmware ASSERT 0x5ab0214 at ALLOC_MEMORY buffer 30); the only accepted shape is the existing single-page-per-PD0-slot pattern. GUEST_PT_EXT0..47 are 48 separate single-page chunks; BUF_COUNT 33 -> 81 (handle range still inside the meta assert bound). 2026-10-15T14:20:13+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:20:13+00:00 urn:sha1:48a8089ed3c4892c2577db2da815e6731b882045 m152: the Tier-2 shadow alias pool consumed one never-reused 128 B fetch line per mapping per launch (512 lines/boot), so a normal launch mix exhausted it permanently: every later launch that needed a new mapping was refused (the deterministic r152 'launch refused: no Tier-1 mapping' loop). Two fixes: (a) an HPA-keyed alias cache in map_shadow_locked that returns the existing never-rewritten VA for a page already mapped, so repeat launches of the same buffers/sem pages consume nothing; a hit reports line_out = ~0 so contiguity bookkeeping never extends an old line; (b) a 48-window extension pool at VA 0x04000000 (PD0 indices 32..79, one 48-page PT chunk) raising capacity to 2048 lines/boot. m153: the intermittent per-boot mis-execution classes (engine wrote nothing; or wrote into a lower operand window while the output operand stayed untouched - both complete with qmd_rel/ch_rel satisfied, live evidence in r152 smiley: passes wrote b into the a window / nothing / correct-after-10s-retry) are now detected by the m151 retirement wait and retried: refill the bounce window from the guest inputs, re-publish the same method block on a fresh ring entry, re-issue schedule, re-poll, re-wait - bounded by the timeout path's attempt budget. Gate param gsp_phys_rmcall_gr_nowrite_retry (default 1). 2026-10-15T06:33:03+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:33:03+00:00 urn:sha1:7e36e2909280414916c821e29e00b341750c8a8f Footprint-by-inference (max-reloc contiguous run, launch-page exclusion) failed twice: the fill's candidate list ends with the launch's own bank/ param pages, so the inferred output run was 3-4 pages (r150/r151: '0/3 pages changed after 72 ms (no writes)') and the wait exited before any real output landed. Replace with a whole-window count over every resolved data page plus cheap adaptive exits: - all pages changed -> done (one poll in the normal case), - partial count stable for 3 polls -> stores settled, - no change for 600 ms -> no output for this launch, - hard cap 1.5 s. No inference, no exclusion list, bounded cost. 2026-10-15T06:16:23+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:16:23+00:00 urn:sha1:4806384848c2903e921dfdefdc8096854e40480f r150's first cut excluded only QMD-word pages; the fill also relocates the 0xffffffff (non-QMD) forwarded pages (pushbuffer/indirect/bank pages) as high-VA candidates, so out_base landed on the bank-page group, out_pages came out 3 and the wait exited after 72 ms - losing the late-write capture it exists for (smiley r150: '0/3 pages changed after 72 ms (no writes)' followed by mixed 8-changed/0-changed/32-changed copy-outs and one wrong pass). Now every forwarded page VA is excluded, so the footprint is the real output operand (32 pages for smiley, 64 for matmul). 2026-10-15T06:06:03+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:06:03+00:00 urn:sha1:b6ab91d098328f940f51d02c4e9941cb1db8d74c The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).