Open-IOV vGPU-IO: host-side GSP emulation for unmodified NVIDIA guest drivers https://git.vgpu.io/open-iov.git/atom?h=stallfree 2026-10-17T21:51:19+00:00 2026-10-17T21:51:19+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-17T21:51:19+00:00 urn:sha1:78ac3864039e066354181cb6b53c92e96d86ff3e (1) svc_session_flush keeps svc.direct_base (per-boot constant; the pin-free targeted scanner's gate/input) and a new gsp_sema_service_guest_reset() hook wipes boot-scoped discovery state on guest RM reload; (2) counter-page derive now takes the first READABLE source (vaspaces -> discovered -> targeted) instead of failing -EIO on a walked-but-garbage page, with one-shot diagnostics; (3) new m177c UVM ring-space directory discovery: content-verified (push_va -> push_pa) samples from the GR content discovery and CE push pairing feed a second discovered-directory slot (walks the 0x2xxxxxxx family), consulted by svc_walk_discovered; (4) GR channels now deliver withheld opdef sets at entry consume (normal tail and the stale-skip path) instead of waiting for the 10 s sweep; (5) host tools hardened: semfix_uvmt QMP reconnect + pid-generation dedupe + stale-sem eviction + post-success pin latch + heartbeats; pin_gr_stream accept timeout + guest-scanner reinstall backoff + restart loop + subprocess timeouts; new pin_supervisor.sh (heartbeat watchdog, capped restarts) wired into start_vm_keep.sh. Evidence: r194-r196 cycles; pool resolves src=trusted pin-free; ring dir 0x10801c000 found 4/4; row: 'rel-apply ... written' with tools up. 2026-10-17T18:09:04+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-17T18:09:04+00:00 urn:sha1:1c8ee4942f0377e5304446c691e2e597588d0cac 2026-10-17T05:15:00+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-17T05:15:00+00:00 urn:sha1:0bb52e0e3cdc86b31dce2daa23eafd099461604a - gsp_rm_emulation: the event burst budget becomes a time-windowed rate (1 s) with completion posts allowed headroom over hint re-posts; the periodic hint dedups events posted within 250 ms (strictly below its own 500 ms cadence - an equal window races the boundary and skips forever); force_post skips retired slots (rc=22 spam was an artifact). Verified: the minimal cudaEventCreate/Record/Synchronize repro and a second repro with cross-stream waits and elapsed-time now complete on a live boot; previously cudaEventSynchronize hung forever. - gsp_sema_service: write/read consistency ledger (AGENT-F R2). Large device-destination writes record {va_page -> pa, aperture}; the launch data fill and device-source reads prefer the recorded page whenever their own resolution disagrees (sema-ledger DIVERGE log). Session scoped, zero cost while empty. 2026-10-16T20:02:00+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-16T20:02:00+00:00 urn:sha1:da9a755bd8841fca0f01199c99aeaa495a108da7 2026-10-15T21:53:52+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-15T21:53:52+00:00 urn:sha1:81c11ca09adf85bd314f7406a8e0d7d637726a80 Goal 1.0: two mdevs + two QEMU guests running CUDA simultaneously. Single-mdev behaviour is kept identical: the legacy guest-RAM record still serves the first/only open mdev, the channel table degenerates to the historical key, and the session flush fires at the same last-channel-closed transition. - G6: guest-RAM backing {file, path, ranges} per mdev (legacy record + up to 4 vgpu-bound records); debugfs gsp-phys-guest-ram accepts an mdev UUID as an extra first token and keeps the 4-token form; records released on mdev cleanup and physical release. - G7: gsp_phys_gpa_to_hpa_vgpu(vgpu, gpa, *hpa); NULL keeps the historical first-open resolution. Forwarding operands pass the request's vgpu; the single-guest boot-ladder pin passes NULL. - G1: the sema-service thread services every open mdev each pass. - G2: channel table keyed by (vgpu, h_object); per-guest ranks; counter/scrub/opdef/pump snapshot+writeback matches per vgpu; scrubber records per guest. - G3: per-guest session records drive the flush; the shared learned tables (staged G4) are only flushed when no guest has live channels. - G5: hold/abort keyed by (vgpu, handle); dead-owner guards on the hold resolution paths; single-inflight serialization kept. - G8a: guest-GPA pin refused while more than one mdev is open. Build: make -j8 clean success (vgsp.ko). No module load. See DUAL-REPORT.md for the per-item status and live-validation plan. 2026-10-11T21:50:53+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-11T21:50:53+00:00 urn:sha1:74ca378aeb5c160ca173697213861f0dbacdb434 This reverts commit f5961087aa17111c6cdb7f6641626c0c877eee6b. 2026-10-11T20:43:33+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-11T20:43:33+00:00 urn:sha1:f5961087aa17111c6cdb7f6641626c0c877eee6b Handles are per-client, but the FREE path unregistered any channel whose numeric handle matched the freed object, with no client check. r77 root cause of the third-client pma scrub hang: the boot session's CeUtils scrubber channel (h=0x2, gpfifo=0x120064000, entries=4096) was unregistered when an unrelated class-0x2080 object with handle 0x2 in client 0xc1d00011 was freed. The guest kept using the live channel for the scrub; the service no longer pumped it, its pushes were never decoded and the channel-buffer finish payload (pbGpuVA+0x6c004) was never written, so scrubCheckAndWaitForSize polled forever (sysrq stack: ceutilsUpdateProgress -> channelReadChannelMemdesc). Channel registration now records h_client (the RM alloc header's client) and deletion matches (h_object, h_client); a missing client on either side keeps the handle-only behaviour. Re-registration keeps the counter/release slots only for the same (handle, client) pair. Live proof from the hung VM: sema_probe translates the finish-payload VA 0x12006c004 to pa=0x10f962004 aperture=ram - the word the guest polls is reachable; the channel was simply invisible to the pump. 2026-10-11T19:43:30+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-11T19:43:30+00:00 urn:sha1:428686fde01802e3a8c7d11393f26ac6f6cb6df8 The pma scrub in a second/third CUDA session memsets a legitimate vidmem offset at 262M (inside the 512M turing shadow) and the copy path rejected it with -EINVAL because every FB bounds check compared against vgpu->bars[1].size, the physical BAR1 aperture exposed to the guest (256M on TU104). The op was held and retried forever (opdef), the GR scrub waited in scrubCheckAndWaitForSize and the client froze. All bounds checks that guard a dereference of the emulated vidmem shadow at vgpu->bars[1].vaddr now use nv_vgpu_fb_emul_size() via svc_fb_shadow_size(): the copy/fill paths, svc_read_ap/svc_write_ap, svc_ap_ok, the page-directory discovery scans, svc_lookup's vidmem VA translation, the GRCE inline write, svc_fwd_operand_hpa and the debug scans. The 2 MB leaf path of svc_gmmu_walk now validates through svc_ap_ok like every other leaf path so an out-of-shadow translation falls through to the shadow fallback instead of reaching the copy path. The RM page-directory debug peek is bounded the same way. Evidence: r75b live log: sema-bigcopy dst_va=0x76aa08e02200 -> pa=0x10602200 ap=0 prov=3; sema-bigop rc=-22 repeated; guest sysrq stack scrubCheckAndWaitForSize -> pmaAllocatePages. 2026-10-08T15:07:42+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-08T15:07:42+00:00 urn:sha1:32d8bc7167060e7afd2256bac266d172abfc6029 kgrobjPromoteContext_IMPL marks each ctx buffer initialized when the REPLY's promoteEntry[i].bInitialize is set: if (!params.promoteEntry[i].bInitialize) continue; kgrctxMarkCtxBufferInitialized(...); The flag is the server's initialized-ACK, so the correct emulation reply is a verbatim echo (the physical backend owns residency; the control answers SUCCESS). 0c75c0cb cleared the flag believing the check inverted - that skipped the mark and left the RM re-promoting the same buffers forever. Restore echo semantics with the source-cited comment. 2026-10-08T14:03:19+00:00 Arthur Hanson Rasmusson arthur@vgpu.io 2026-10-08T14:03:19+00:00 urn:sha1:0c75c0cbbf51c71dd70396082beca8cb0c102332 kgrobjPromoteContext_IMPL marks a context buffer initialized only when the REPLY's promoteEntry[i].bInitialize is clear; echoing the request back kept the flag set so the RM re-promoted the same contexts indefinitely. Clear the flag on the echoed reply (per-entry, 5 entries observed per call). Note: this aligns the reply with the caller contract but did NOT by itself unblock the observed cudaMalloc stall (client still reaches DEVICE then spins; zero GR ring content in VRAM) - kept because the old behavior was wrong and the flag gates kgrctxMarkCtxBufferInitialized.