Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=lottery 2026-10-15T22:02:17+00:00 2026-10-15T22:02:17+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T22:02:17+00:00 urn:sha1:8ca6ee526a330b8549c67cb96f90ea736dbd62f5 2026-10-15T22:01:46+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T22:01:46+00:00 urn:sha1:99e5a61e87b34416259254c22241a072fbd412a4 LOTTERY-REPORT.md: live r155 evidence (streams2 bad=32768 with qmd_rel=6 and 0/205 on all attempts; 98 never-attempted captures soft-released during the vecadd burst), the exact retry/failure chain, the proven unsoundness of the F1 fail-closed release (idempotent repeats show 0/205 with correct data), ranked fix proposals (defer busy refusals, 4-bank win0 + qmdwin double-buffer, warm sem push, sound-discriminator hunt), and the stall class characterization. No kernel changes; the rejected gate attempt is preserved on branch lottery-m156-gate-attempt. 2026-10-15T16:41:14+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T16:41:14+00:00 urn:sha1:ecfb1278086f98161402f93bca8d3df602fbcb45 r154: after the second client's idle period the FECS saved/unloaded the context (SKED rec: cwd0=0x12486 fecs_cur=b0633ee3 -> cwd0=0x2086 fecs_cur=0 and no change ever after) and every later forwarded launch acked (qmd_rel/ch_rel) while dispatching nothing - including all three outcome retries with fresh ring entries (0/205 changed, 720 ms no-writes, x4). Re-issuing only GPFIFO_SCHEDULE+TSG_GPFIFO_SCHEDULE+doorbell did not revive dispatch. The extended re-arm now replays the mode-35 lane-enable tail first (GR_SET_CTXSW_PREEMPTION_MODE CILP + SET_TIMESLICE + DEVICE_SCHED - the proven-accepted stage-4 sequence, HF-3 design), then the schedules and the doorbell. Poison variants (explicit PREEMPT/tail, BIND, PROMOTE) remain excluded. 2026-10-15T16:08:53+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T16:08:53+00:00 urn:sha1:e07c62b1fab2f2dbc9981458c0697829b287b01e r153 root cause (3rd-client userspace spin for the rest of the boot): the guest scanner's merged CE-window run (va=0x121000000 len=0x40000; the merged length varies per session) covered a CE channel's GPFIFO ring VA 0x121010000. svc_ipin_store took precedence in the ring read path, so the ring read the guest page at the wrong PA (0x102e02000): an all-zero entry was silently consumed, the pool release was never authored (pushes frozen at 1643, crel frozen), and victory_matmul spun in userspace for 50+ min. The ring VA translation must come from the channel's own discovery path; an injected pin inside any live channel's ring range is always wrong - drop it (counted + logged, first 8 per session). Validated on the next cycle: 'injected pin ... DROPPED' lines plus 'entry[1] raw!=0', advancing pushes, MATMUL bad=0 STATUS OK. 2026-10-15T14:44:19+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:44:19+00:00 urn:sha1:02249afd91f4ee02589a07657d130b9b9411debd Live r153 evidence: the highest-reloc data entry (bi=204) is a UVM-internal buffer that correct launches never write, so the 'top operand untouched' condition fired on correct executions and re-executed them up to 3x - a correctness hazard for side-effecting kernels (atomics/printf) and pure waste otherwise. The honest execution witness is the retirement wait's changed count; only zero-writes retries now (safe by construction: a kernel that wrote nothing has nothing to duplicate). Read-only kernels pay the bounded extra polls. 2026-10-15T14:26:45+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:26:45+00:00 urn:sha1:5a446c10c3ffa9180328180318163b46cb00a16d The server refuses multi-page client memdescs for these buffers (a 48-page chunk aborted stage 2 with rpc_result 0x1f + firmware ASSERT 0x5ab0214 at ALLOC_MEMORY buffer 30); the only accepted shape is the existing single-page-per-PD0-slot pattern. GUEST_PT_EXT0..47 are 48 separate single-page chunks; BUF_COUNT 33 -> 81 (handle range still inside the meta assert bound). 2026-10-15T14:20:13+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:20:13+00:00 urn:sha1:48a8089ed3c4892c2577db2da815e6731b882045 m152: the Tier-2 shadow alias pool consumed one never-reused 128 B fetch line per mapping per launch (512 lines/boot), so a normal launch mix exhausted it permanently: every later launch that needed a new mapping was refused (the deterministic r152 'launch refused: no Tier-1 mapping' loop). Two fixes: (a) an HPA-keyed alias cache in map_shadow_locked that returns the existing never-rewritten VA for a page already mapped, so repeat launches of the same buffers/sem pages consume nothing; a hit reports line_out = ~0 so contiguity bookkeeping never extends an old line; (b) a 48-window extension pool at VA 0x04000000 (PD0 indices 32..79, one 48-page PT chunk) raising capacity to 2048 lines/boot. m153: the intermittent per-boot mis-execution classes (engine wrote nothing; or wrote into a lower operand window while the output operand stayed untouched - both complete with qmd_rel/ch_rel satisfied, live evidence in r152 smiley: passes wrote b into the a window / nothing / correct-after-10s-retry) are now detected by the m151 retirement wait and retried: refill the bounce window from the guest inputs, re-publish the same method block on a fresh ring entry, re-issue schedule, re-poll, re-wait - bounded by the timeout path's attempt budget. Gate param gsp_phys_rmcall_gr_nowrite_retry (default 1). 2026-10-15T06:33:03+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:33:03+00:00 urn:sha1:7e36e2909280414916c821e29e00b341750c8a8f Footprint-by-inference (max-reloc contiguous run, launch-page exclusion) failed twice: the fill's candidate list ends with the launch's own bank/ param pages, so the inferred output run was 3-4 pages (r150/r151: '0/3 pages changed after 72 ms (no writes)') and the wait exited before any real output landed. Replace with a whole-window count over every resolved data page plus cheap adaptive exits: - all pages changed -> done (one poll in the normal case), - partial count stable for 3 polls -> stores settled, - no change for 600 ms -> no output for this launch, - hard cap 1.5 s. No inference, no exclusion list, bounded cost. 2026-10-15T06:16:23+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:16:23+00:00 urn:sha1:4806384848c2903e921dfdefdc8096854e40480f r150's first cut excluded only QMD-word pages; the fill also relocates the 0xffffffff (non-QMD) forwarded pages (pushbuffer/indirect/bank pages) as high-VA candidates, so out_base landed on the bank-page group, out_pages came out 3 and the wait exited after 72 ms - losing the late-write capture it exists for (smiley r150: '0/3 pages changed after 72 ms (no writes)' followed by mixed 8-changed/0-changed/32-changed copy-outs and one wrong pass). Now every forwarded page VA is excluded, so the footprint is the real output operand (32 pages for smiley, 64 for matmul). 2026-10-15T06:06:03+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:06:03+00:00 urn:sha1:b6ab91d098328f940f51d02c4e9941cb1db8d74c The one-shot #sem kprobe event that drives the CE push-buffer chain resolution can be lost (trace rotation, scanner restart, slab kptr reuse), and the chain resolver then never learns a client - every GPFIFO entry of its CE channels is grace-skipped and the client spins on its tracker semaphore (the copy-heavy stall class: streams2/memcpy/multikernel/saxpy in r148; smiley as a later client in r147 took ~90 s to recover). New demand channel, both ends here: - pin_gr_stream: _demand_loop scans the host dmesg for 'stalled VA #N: 0x...' (the service logs each distinct unreadable push VA once), tracks each sighting, and re-sends every 20 s for up to 5 min to /tmp/guest_want_vas.txt in the guest (via the guest agent). - guest_pin_stream: the main loop reads new VAs from that file and resolves each by CONTENT against the nv_vm_map_pages vmap areas (resolve_va_by_content: candidate leaf page at the VA's in-window offset, +1/-1 neighbours for the header-page offset ambiguity, host-plausible method-stream prefix). On a match the exact pair is emitted; only the requested VA is pinned, so a wrong candidate cannot poison other VAs. Also fix the scanner start: the kill-first pattern must be anchored (^python3 /tmp/guest_pin_...) - the unanchored pattern matched the deploy shell's own cmdline and every start attempt killed its own shell before launching the scanner (r149: no scanner for the whole boot).