Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=dual 2026-10-15T22:08:27+00:00 2026-10-15T22:08:27+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T22:08:27+00:00 urn:sha1:15fdb36784b62e36a76a15e8072274fd8f2b55cf 2026-10-15T22:02:49+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T22:02:49+00:00 urn:sha1:5769d5c43f97434fa4b82b4dab2f83ebee7ed73c The host CUDA bridge is the GR fallback path when a forward request is refused under the single-inflight slot (the expected steady state with two guests). Its FIFO request and pending records dropped the owning vgpu, so guest B's fallback job could be marshalled (VAs resolved, guest memory read/written) through guest A's address space. - cubb_req/cubb_pend gain the owning mdev; the launch stores it and it is part of the dedup identity. - Only the owning guest's service pass marshals its job or drives its staged completion; a head/record whose mdev is gone is dropped instead of ever being dereferenced. - One guest: head_owner is always the pumped vgpu, so behaviour is unchanged. 2026-10-15T21:53:52+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T21:53:52+00:00 urn:sha1:81c11ca09adf85bd314f7406a8e0d7d637726a80 Goal 1.0: two mdevs + two QEMU guests running CUDA simultaneously. Single-mdev behaviour is kept identical: the legacy guest-RAM record still serves the first/only open mdev, the channel table degenerates to the historical key, and the session flush fires at the same last-channel-closed transition. - G6: guest-RAM backing {file, path, ranges} per mdev (legacy record + up to 4 vgpu-bound records); debugfs gsp-phys-guest-ram accepts an mdev UUID as an extra first token and keeps the 4-token form; records released on mdev cleanup and physical release. - G7: gsp_phys_gpa_to_hpa_vgpu(vgpu, gpa, *hpa); NULL keeps the historical first-open resolution. Forwarding operands pass the request's vgpu; the single-guest boot-ladder pin passes NULL. - G1: the sema-service thread services every open mdev each pass. - G2: channel table keyed by (vgpu, h_object); per-guest ranks; counter/scrub/opdef/pump snapshot+writeback matches per vgpu; scrubber records per guest. - G3: per-guest session records drive the flush; the shared learned tables (staged G4) are only flushed when no guest has live channels. - G5: hold/abort keyed by (vgpu, handle); dead-owner guards on the hold resolution paths; single-inflight serialization kept. - G8a: guest-GPA pin refused while more than one mdev is open. Build: make -j8 clean success (vgsp.ko). No module load. See DUAL-REPORT.md for the per-item status and live-validation plan. 2026-10-15T16:41:14+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T16:41:14+00:00 urn:sha1:ecfb1278086f98161402f93bca8d3df602fbcb45 r154: after the second client's idle period the FECS saved/unloaded the context (SKED rec: cwd0=0x12486 fecs_cur=b0633ee3 -> cwd0=0x2086 fecs_cur=0 and no change ever after) and every later forwarded launch acked (qmd_rel/ch_rel) while dispatching nothing - including all three outcome retries with fresh ring entries (0/205 changed, 720 ms no-writes, x4). Re-issuing only GPFIFO_SCHEDULE+TSG_GPFIFO_SCHEDULE+doorbell did not revive dispatch. The extended re-arm now replays the mode-35 lane-enable tail first (GR_SET_CTXSW_PREEMPTION_MODE CILP + SET_TIMESLICE + DEVICE_SCHED - the proven-accepted stage-4 sequence, HF-3 design), then the schedules and the doorbell. Poison variants (explicit PREEMPT/tail, BIND, PROMOTE) remain excluded. 2026-10-15T16:08:53+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T16:08:53+00:00 urn:sha1:e07c62b1fab2f2dbc9981458c0697829b287b01e r153 root cause (3rd-client userspace spin for the rest of the boot): the guest scanner's merged CE-window run (va=0x121000000 len=0x40000; the merged length varies per session) covered a CE channel's GPFIFO ring VA 0x121010000. svc_ipin_store took precedence in the ring read path, so the ring read the guest page at the wrong PA (0x102e02000): an all-zero entry was silently consumed, the pool release was never authored (pushes frozen at 1643, crel frozen), and victory_matmul spun in userspace for 50+ min. The ring VA translation must come from the channel's own discovery path; an injected pin inside any live channel's ring range is always wrong - drop it (counted + logged, first 8 per session). Validated on the next cycle: 'injected pin ... DROPPED' lines plus 'entry[1] raw!=0', advancing pushes, MATMUL bad=0 STATUS OK. 2026-10-15T14:44:19+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:44:19+00:00 urn:sha1:02249afd91f4ee02589a07657d130b9b9411debd Live r153 evidence: the highest-reloc data entry (bi=204) is a UVM-internal buffer that correct launches never write, so the 'top operand untouched' condition fired on correct executions and re-executed them up to 3x - a correctness hazard for side-effecting kernels (atomics/printf) and pure waste otherwise. The honest execution witness is the retirement wait's changed count; only zero-writes retries now (safe by construction: a kernel that wrote nothing has nothing to duplicate). Read-only kernels pay the bounded extra polls. 2026-10-15T14:26:45+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:26:45+00:00 urn:sha1:5a446c10c3ffa9180328180318163b46cb00a16d The server refuses multi-page client memdescs for these buffers (a 48-page chunk aborted stage 2 with rpc_result 0x1f + firmware ASSERT 0x5ab0214 at ALLOC_MEMORY buffer 30); the only accepted shape is the existing single-page-per-PD0-slot pattern. GUEST_PT_EXT0..47 are 48 separate single-page chunks; BUF_COUNT 33 -> 81 (handle range still inside the meta assert bound). 2026-10-15T14:20:13+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T14:20:13+00:00 urn:sha1:48a8089ed3c4892c2577db2da815e6731b882045 m152: the Tier-2 shadow alias pool consumed one never-reused 128 B fetch line per mapping per launch (512 lines/boot), so a normal launch mix exhausted it permanently: every later launch that needed a new mapping was refused (the deterministic r152 'launch refused: no Tier-1 mapping' loop). Two fixes: (a) an HPA-keyed alias cache in map_shadow_locked that returns the existing never-rewritten VA for a page already mapped, so repeat launches of the same buffers/sem pages consume nothing; a hit reports line_out = ~0 so contiguity bookkeeping never extends an old line; (b) a 48-window extension pool at VA 0x04000000 (PD0 indices 32..79, one 48-page PT chunk) raising capacity to 2048 lines/boot. m153: the intermittent per-boot mis-execution classes (engine wrote nothing; or wrote into a lower operand window while the output operand stayed untouched - both complete with qmd_rel/ch_rel satisfied, live evidence in r152 smiley: passes wrote b into the a window / nothing / correct-after-10s-retry) are now detected by the m151 retirement wait and retried: refill the bounce window from the guest inputs, re-publish the same method block on a fresh ring entry, re-issue schedule, re-poll, re-wait - bounded by the timeout path's attempt budget. Gate param gsp_phys_rmcall_gr_nowrite_retry (default 1). 2026-10-15T06:33:03+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:33:03+00:00 urn:sha1:7e36e2909280414916c821e29e00b341750c8a8f Footprint-by-inference (max-reloc contiguous run, launch-page exclusion) failed twice: the fill's candidate list ends with the launch's own bank/ param pages, so the inferred output run was 3-4 pages (r150/r151: '0/3 pages changed after 72 ms (no writes)') and the wait exited before any real output landed. Replace with a whole-window count over every resolved data page plus cheap adaptive exits: - all pages changed -> done (one poll in the normal case), - partial count stable for 3 polls -> stores settled, - no change for 600 ms -> no output for this launch, - hard cap 1.5 s. No inference, no exclusion list, bounded cost. 2026-10-15T06:16:23+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T06:16:23+00:00 urn:sha1:4806384848c2903e921dfdefdc8096854e40480f r150's first cut excluded only QMD-word pages; the fill also relocates the 0xffffffff (non-QMD) forwarded pages (pushbuffer/indirect/bank pages) as high-VA candidates, so out_base landed on the bank-page group, out_pages came out 3 and the wait exited after 72 ms - losing the late-write capture it exists for (smiley r150: '0/3 pages changed after 72 ms (no writes)' followed by mixed 8-changed/0-changed/32-changed copy-outs and one wrong pass). Now every forwarded page VA is excluded, so the footprint is the real output operand (32 pages for smiley, 64 for matmul).